2024-09-18T07:14:40.439 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-18T07:14:40.459 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T07:14:40.547 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910293 branch: wip-guits-main-2024-09-17-1213 description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} email: gabrioux@ibm.com first_in_suite: false job_id: '7910293' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-main-2024-09-17-1213 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) log-only-match: - CEPHADM_ mon_bind_msgr2: false sha1: e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: wip-guits-main-2024-09-17-1213 sha1: e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 8982 sha1: e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-main-2024-09-17-1213 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e targets: smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFPNF+cMMwtWTf+ecwkYNCoVm7lvcLn7uvL40JAtkB9th8z3biYX+bpgZC8NplTK2ShLrE2SWUTnMXuyzP9UKwY= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPGjK4DzMvmrzMT/RGXeh0mKq1mYTIjWlSu5LtQEcZBYAmE9tYIs03ykFDHSoV9qISReWx/Tbo+FJYEVblLcWvw= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install nvmetcli nvme-cli -y - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-18_06:59:44 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-18T07:14:40.547 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/qa; will attempt to use it 2024-09-18T07:14:40.547 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/qa/tasks 2024-09-18T07:14:40.548 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-18T07:14:40.549 INFO:teuthology.task.internal:Checking packages... 2024-09-18T07:14:40.570 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e' 2024-09-18T07:14:40.571 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-18T07:14:40.571 INFO:teuthology.packaging:ref: None 2024-09-18T07:14:40.571 INFO:teuthology.packaging:tag: None 2024-09-18T07:14:40.571 INFO:teuthology.packaging:branch: wip-guits-main-2024-09-17-1213 2024-09-18T07:14:40.571 INFO:teuthology.packaging:sha1: e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:14:40.571 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-main-2024-09-17-1213 2024-09-18T07:14:40.714 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4983.ge7fb7b56 2024-09-18T07:14:40.716 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-18T07:14:40.722 INFO:teuthology.task.internal:no buildpackages task found 2024-09-18T07:14:40.723 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-18T07:14:40.773 INFO:teuthology.task.internal:Saving configuration 2024-09-18T07:14:40.788 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-18T07:14:40.807 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-18T07:14:40.830 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910293', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 07:09:36.614593', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFPNF+cMMwtWTf+ecwkYNCoVm7lvcLn7uvL40JAtkB9th8z3biYX+bpgZC8NplTK2ShLrE2SWUTnMXuyzP9UKwY='} 2024-09-18T07:14:40.854 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910293', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 07:09:36.613467', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPGjK4DzMvmrzMT/RGXeh0mKq1mYTIjWlSu5LtQEcZBYAmE9tYIs03ykFDHSoV9qISReWx/Tbo+FJYEVblLcWvw='} 2024-09-18T07:14:40.854 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-18T07:14:40.860 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-18T07:14:40.861 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-18T07:14:40.861 INFO:teuthology.run_tasks:Running task console_log... 2024-09-18T07:14:40.940 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fae5950bbe0>, signals=[15]) 2024-09-18T07:14:40.941 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-18T07:14:40.949 INFO:teuthology.task.internal:Opening connections... 2024-09-18T07:14:40.950 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-09-18T07:14:40.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:14:41.031 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-09-18T07:14:41.032 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:14:41.110 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-18T07:14:41.116 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-09-18T07:14:41.135 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-09-18T07:14:41.135 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-09-18T07:14:41.193 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream" 2024-09-18T07:14:41.193 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="9" 2024-09-18T07:14:41.193 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2024-09-18T07:14:41.193 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2024-09-18T07:14:41.194 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="9" 2024-09-18T07:14:41.194 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el9" 2024-09-18T07:14:41.194 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-18T07:14:41.194 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2024-09-18T07:14:41.194 INFO:teuthology.orchestra.run.smithi137.stdout:LOGO="fedora-logo-icon" 2024-09-18T07:14:41.194 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-18T07:14:41.194 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2024-09-18T07:14:41.194 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-18T07:14:41.194 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-18T07:14:41.194 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-18T07:14:41.195 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-09-18T07:14:41.212 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-09-18T07:14:41.232 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-09-18T07:14:41.232 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-09-18T07:14:41.290 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2024-09-18T07:14:41.290 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9" 2024-09-18T07:14:41.291 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2024-09-18T07:14:41.291 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2024-09-18T07:14:41.291 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9" 2024-09-18T07:14:41.291 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9" 2024-09-18T07:14:41.291 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-18T07:14:41.291 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2024-09-18T07:14:41.291 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon" 2024-09-18T07:14:41.291 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-18T07:14:41.291 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2024-09-18T07:14:41.291 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-18T07:14:41.291 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-18T07:14:41.291 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-18T07:14:41.292 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-09-18T07:14:41.312 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-18T07:14:41.322 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-18T07:14:41.364 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-18T07:14:41.365 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-09-18T07:14:41.367 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-09-18T07:14:41.384 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-18T07:14:41.398 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-18T07:14:41.398 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-09-18T07:14:41.425 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-09-18T07:14:41.540 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-18T07:14:41.592 INFO:teuthology.run_tasks:Running task kernel... 2024-09-18T07:14:41.608 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-18T07:14:41.608 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-18T07:14:41.608 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T07:14:41.609 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T07:14:41.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:14:41.626 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-09-18T07:14:41.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:14:41.627 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-09-18T07:14:41.684 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-503.el9.x86_64 2024-09-18T07:14:41.684 INFO:teuthology.task.kernel:Running kernel on smithi137: 5.14.0-503.el9.x86_64 2024-09-18T07:14:41.684 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum install -y kernel 2024-09-18T07:14:41.686 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-503.el9.x86_64 2024-09-18T07:14:41.686 INFO:teuthology.task.kernel:Running kernel on smithi203: 5.14.0-503.el9.x86_64 2024-09-18T07:14:41.686 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum install -y kernel 2024-09-18T07:14:43.889 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.3 MB 00:00 2024-09-18T07:14:45.316 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.3 MB 00:01 2024-09-18T07:14:46.665 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 12 MB/s | 20 MB 00:01 2024-09-18T07:14:49.500 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 6.6 MB/s | 20 MB 00:03 2024-09-18T07:14:50.882 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-18T07:14:52.749 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 19 kB 00:00 2024-09-18T07:14:53.867 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-18T07:14:54.078 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-18T07:14:55.351 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 19 kB 00:00 2024-09-18T07:14:57.047 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 14 MB/s | 23 MB 00:01 2024-09-18T07:14:59.534 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-18T07:15:01.416 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-18T07:15:01.416 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-18T07:15:01.496 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:15:01.501 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:15:01.501 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-09-18T07:15:01.501 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:15:01.501 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:15:01.502 INFO:teuthology.orchestra.run.smithi137.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-18T07:15:01.502 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-18T07:15:01.502 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-18T07:15:01.502 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-18T07:15:01.502 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-18T07:15:01.502 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:15:01.502 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:15:01.502 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:15:01.502 INFO:teuthology.orchestra.run.smithi137.stdout:Install 4 Packages 2024-09-18T07:15:01.502 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:15:01.503 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 79 M 2024-09-18T07:15:01.503 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 126 M 2024-09-18T07:15:01.503 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:15:01.861 INFO:teuthology.orchestra.run.smithi137.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 325 kB/s | 36 kB 00:00 2024-09-18T07:15:02.236 INFO:teuthology.orchestra.run.smithi137.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-09-18T07:15:02.577 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-18T07:15:02.611 INFO:teuthology.orchestra.run.smithi137.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 38 MB/s | 29 MB 00:00 2024-09-18T07:15:02.920 INFO:teuthology.orchestra.run.smithi137.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 30 MB/s | 35 MB 00:01 2024-09-18T07:15:02.920 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:15:02.920 INFO:teuthology.orchestra.run.smithi137.stdout:Total 56 MB/s | 79 MB 00:01 2024-09-18T07:15:03.605 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:15:03.950 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:15:03.950 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:15:04.450 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-18T07:15:04.451 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-18T07:15:04.540 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:15:04.553 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:15:04.553 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-09-18T07:15:04.553 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:15:04.553 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:15:04.553 INFO:teuthology.orchestra.run.smithi203.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-18T07:15:04.553 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-18T07:15:04.554 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-18T07:15:04.554 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-18T07:15:04.554 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-18T07:15:04.554 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:15:04.554 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:15:04.554 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:15:04.555 INFO:teuthology.orchestra.run.smithi203.stdout:Install 4 Packages 2024-09-18T07:15:04.555 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:15:04.555 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 79 M 2024-09-18T07:15:04.555 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 126 M 2024-09-18T07:15:04.555 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:15:04.845 INFO:teuthology.orchestra.run.smithi203.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 475 kB/s | 36 kB 00:00 2024-09-18T07:15:04.904 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:15:04.904 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:15:05.295 INFO:teuthology.orchestra.run.smithi203.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-09-18T07:15:05.970 INFO:teuthology.orchestra.run.smithi203.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 25 MB/s | 29 MB 00:01 2024-09-18T07:15:06.271 INFO:teuthology.orchestra.run.smithi203.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 23 MB/s | 35 MB 00:01 2024-09-18T07:15:06.271 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:15:06.271 INFO:teuthology.orchestra.run.smithi203.stdout:Total 46 MB/s | 79 MB 00:01 2024-09-18T07:15:06.951 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:15:07.288 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:15:07.288 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:15:07.335 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:15:07.964 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-18T07:15:08.045 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T07:15:08.285 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:15:08.285 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:15:10.729 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:15:10.777 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T07:15:10.905 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T07:15:11.352 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-18T07:15:11.440 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T07:15:14.192 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T07:15:14.323 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T07:15:15.726 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T07:15:15.763 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:15:19.130 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T07:15:19.165 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:15:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:15:24.031 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:15:58.819 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:15:58.858 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:15:59.202 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:15:59.203 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-18T07:15:59.203 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T07:15:59.203 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T07:16:00.072 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:16:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:16:00.073 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:16:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-18T07:16:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-18T07:16:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-18T07:16:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-18T07:16:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:16:00.073 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:16:00.440 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T07:16:00.520 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T07:16:00.521 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T07:16:00.521 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-18T07:16:00.521 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T07:16:00.522 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T07:16:00.522 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-18T07:16:00.522 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T07:16:00.522 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T07:16:00.580 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T07:16:00.581 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T07:16:00.582 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -qi grub2-tools 2024-09-18T07:16:00.613 INFO:teuthology.orchestra.run.smithi137.stdout:Name : grub2-tools 2024-09-18T07:16:00.613 INFO:teuthology.orchestra.run.smithi137.stdout:Epoch : 1 2024-09-18T07:16:00.613 INFO:teuthology.orchestra.run.smithi137.stdout:Version : 2.06 2024-09-18T07:16:00.613 INFO:teuthology.orchestra.run.smithi137.stdout:Release : 82.el9 2024-09-18T07:16:00.614 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: x86_64 2024-09-18T07:16:00.614 INFO:teuthology.orchestra.run.smithi137.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-18T07:16:00.614 INFO:teuthology.orchestra.run.smithi137.stdout:Group : Unspecified 2024-09-18T07:16:00.614 INFO:teuthology.orchestra.run.smithi137.stdout:Size : 8274670 2024-09-18T07:16:00.614 INFO:teuthology.orchestra.run.smithi137.stdout:License : GPLv3+ 2024-09-18T07:16:00.614 INFO:teuthology.orchestra.run.smithi137.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-18T07:16:00.614 INFO:teuthology.orchestra.run.smithi137.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-18T07:16:00.614 INFO:teuthology.orchestra.run.smithi137.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-18T07:16:00.614 INFO:teuthology.orchestra.run.smithi137.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-18T07:16:00.614 INFO:teuthology.orchestra.run.smithi137.stdout:Packager : builder@centos.org 2024-09-18T07:16:00.614 INFO:teuthology.orchestra.run.smithi137.stdout:Vendor : CentOS 2024-09-18T07:16:00.614 INFO:teuthology.orchestra.run.smithi137.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-18T07:16:00.614 INFO:teuthology.orchestra.run.smithi137.stdout:Summary : Support tools for GRUB. 2024-09-18T07:16:00.615 INFO:teuthology.orchestra.run.smithi137.stdout:Description : 2024-09-18T07:16:00.615 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:16:00.615 INFO:teuthology.orchestra.run.smithi137.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-18T07:16:00.615 INFO:teuthology.orchestra.run.smithi137.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-18T07:16:00.615 INFO:teuthology.orchestra.run.smithi137.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-18T07:16:00.615 INFO:teuthology.orchestra.run.smithi137.stdout:hardware devices. 2024-09-18T07:16:00.615 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:16:00.615 INFO:teuthology.orchestra.run.smithi137.stdout:This subpackage provides tools for support of all platforms. 2024-09-18T07:16:00.616 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-18T07:16:00.616 INFO:teuthology.task.kernel:Updating grub on smithi137 to boot 5.14.0-508.el9.x86_64 2024-09-18T07:16:00.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-18T07:16:01.261 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2024-09-18T07:16:02.503 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:16:02.543 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:16:02.586 INFO:teuthology.orchestra.run.smithi137.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-18T07:16:02.607 INFO:teuthology.orchestra.run.smithi137.stderr:done 2024-09-18T07:16:02.610 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-09-18T07:16:02.626 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.vFmWRDIyR3 2024-09-18T07:16:02.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.vFmWRDIyR3 2024-09-18T07:16:02.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.vFmWRDIyR3 2024-09-18T07:16:02.815 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:16:02.816 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-18T07:16:02.816 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-18T07:16:02.816 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-18T07:16:02.842 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.vFmWRDIyR3 is 6KB 2024-09-18T07:16:02.892 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.vFmWRDIyR3 2024-09-18T07:16:02.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo /bin/ls /boot/loader/entries || true 2024-09-18T07:16:02.976 INFO:teuthology.orchestra.run.smithi137.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-18T07:16:02.976 INFO:teuthology.orchestra.run.smithi137.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-18T07:16:02.976 INFO:teuthology.orchestra.run.smithi137.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-18T07:16:02.976 INFO:teuthology.orchestra.run.smithi137.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-18T07:16:02.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-18T07:16:03.140 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2024-09-18T07:16:03.709 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-18T07:16:03.709 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:16:03.709 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:16:03.709 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-18T07:16:03.709 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-18T07:16:03.709 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-18T07:16:03.710 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-18T07:16:03.710 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:16:03.710 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:16:04.067 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T07:16:04.158 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T07:16:04.158 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T07:16:04.159 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-18T07:16:04.159 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T07:16:04.159 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T07:16:04.159 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-18T07:16:04.159 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi203.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T07:16:04.159 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-09-18T07:16:04.218 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-18T07:16:04.218 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-18T07:16:04.218 DEBUG:teuthology.orchestra.run.smithi203:> sudo rpm -qi grub2-tools 2024-09-18T07:16:04.257 INFO:teuthology.orchestra.run.smithi203.stdout:Name : grub2-tools 2024-09-18T07:16:04.257 INFO:teuthology.orchestra.run.smithi203.stdout:Epoch : 1 2024-09-18T07:16:04.257 INFO:teuthology.orchestra.run.smithi203.stdout:Version : 2.06 2024-09-18T07:16:04.257 INFO:teuthology.orchestra.run.smithi203.stdout:Release : 82.el9 2024-09-18T07:16:04.257 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: x86_64 2024-09-18T07:16:04.257 INFO:teuthology.orchestra.run.smithi203.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-18T07:16:04.257 INFO:teuthology.orchestra.run.smithi203.stdout:Group : Unspecified 2024-09-18T07:16:04.258 INFO:teuthology.orchestra.run.smithi203.stdout:Size : 8274670 2024-09-18T07:16:04.258 INFO:teuthology.orchestra.run.smithi203.stdout:License : GPLv3+ 2024-09-18T07:16:04.258 INFO:teuthology.orchestra.run.smithi203.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-18T07:16:04.258 INFO:teuthology.orchestra.run.smithi203.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-18T07:16:04.258 INFO:teuthology.orchestra.run.smithi203.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-18T07:16:04.258 INFO:teuthology.orchestra.run.smithi203.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-18T07:16:04.258 INFO:teuthology.orchestra.run.smithi203.stdout:Packager : builder@centos.org 2024-09-18T07:16:04.258 INFO:teuthology.orchestra.run.smithi203.stdout:Vendor : CentOS 2024-09-18T07:16:04.258 INFO:teuthology.orchestra.run.smithi203.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-18T07:16:04.258 INFO:teuthology.orchestra.run.smithi203.stdout:Summary : Support tools for GRUB. 2024-09-18T07:16:04.258 INFO:teuthology.orchestra.run.smithi203.stdout:Description : 2024-09-18T07:16:04.258 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:16:04.258 INFO:teuthology.orchestra.run.smithi203.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-18T07:16:04.258 INFO:teuthology.orchestra.run.smithi203.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-18T07:16:04.259 INFO:teuthology.orchestra.run.smithi203.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-18T07:16:04.259 INFO:teuthology.orchestra.run.smithi203.stdout:hardware devices. 2024-09-18T07:16:04.259 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:16:04.259 INFO:teuthology.orchestra.run.smithi203.stdout:This subpackage provides tools for support of all platforms. 2024-09-18T07:16:04.260 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-18T07:16:04.260 INFO:teuthology.task.kernel:Updating grub on smithi203 to boot 5.14.0-508.el9.x86_64 2024-09-18T07:16:04.260 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-18T07:16:04.914 INFO:teuthology.orchestra.run.smithi203.stderr:Generating grub configuration file ... 2024-09-18T07:16:06.217 INFO:teuthology.orchestra.run.smithi203.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-18T07:16:06.253 INFO:teuthology.orchestra.run.smithi203.stderr:done 2024-09-18T07:16:06.254 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-09-18T07:16:06.273 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.432MpxX4QI 2024-09-18T07:16:06.273 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.432MpxX4QI 2024-09-18T07:16:06.345 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.432MpxX4QI 2024-09-18T07:16:06.484 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.432MpxX4QI is 6KB 2024-09-18T07:16:06.533 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.432MpxX4QI 2024-09-18T07:16:06.548 DEBUG:teuthology.orchestra.run.smithi203:> sudo /bin/ls /boot/loader/entries || true 2024-09-18T07:16:06.611 INFO:teuthology.orchestra.run.smithi203.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-18T07:16:06.611 INFO:teuthology.orchestra.run.smithi203.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-18T07:16:06.611 INFO:teuthology.orchestra.run.smithi203.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-18T07:16:06.611 INFO:teuthology.orchestra.run.smithi203.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-18T07:16:06.613 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-18T07:16:06.747 DEBUG:teuthology.orchestra.run.smithi203:> sudo shutdown -r now 2024-09-18T07:16:33.169 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-18T07:16:33.170 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-18T07:16:33.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:16:36.753 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-18T07:16:36.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-18T07:16:36.754 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:16:51.645 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-09-18T07:16:55.229 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-09-18T07:17:00.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-18T07:17:00.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:17:03.709 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-09-18T07:17:04.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-18T07:17:04.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:17:07.297 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-09-18T07:17:15.713 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-18T07:17:15.714 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:17:19.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-18T07:17:19.302 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:17:37.725 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-09-18T07:17:47.693 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-09-18T07:17:48.413 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-18T07:17:48.414 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-18T07:17:48.414 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-09-18T07:17:48.429 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-508.el9.x86_64 2024-09-18T07:17:48.429 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-18T07:17:48.429 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T07:17:48.429 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-18T07:17:49.430 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-18T07:17:49.430 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T07:17:49.503 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-09-18T07:17:49.528 DEBUG:teuthology.parallel:result is None 2024-09-18T07:17:52.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-18T07:17:52.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:17:53.179 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-09-18T07:17:53.600 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-18T07:17:53.601 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-18T07:17:53.601 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-09-18T07:17:53.617 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-508.el9.x86_64 2024-09-18T07:17:53.617 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-18T07:17:53.617 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T07:17:53.618 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-18T07:17:54.618 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-18T07:17:54.619 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T07:17:54.739 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2024-09-18T07:17:54.772 DEBUG:teuthology.parallel:result is None 2024-09-18T07:17:54.773 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-18T07:17:54.783 INFO:teuthology.task.internal:Creating test directory... 2024-09-18T07:17:54.784 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T07:17:54.787 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T07:17:54.803 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-18T07:17:54.835 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-18T07:17:54.885 INFO:teuthology.task.internal:Creating archive directory... 2024-09-18T07:17:54.885 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T07:17:54.888 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T07:17:54.992 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-18T07:17:55.004 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-18T07:17:55.005 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T07:17:55.007 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T07:17:55.042 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T07:17:55.046 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T07:17:55.054 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T07:17:55.057 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T07:17:55.058 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-18T07:17:55.066 INFO:teuthology.task.internal:Configuring sudo... 2024-09-18T07:17:55.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T07:17:55.098 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T07:17:55.127 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-18T07:17:55.139 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-18T07:17:55.139 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T07:17:55.168 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T07:17:55.186 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T07:17:55.261 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T07:17:55.334 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T07:17:55.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T07:17:55.420 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-18T07:17:55.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T07:17:55.487 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T07:17:55.526 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T07:17:55.582 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T07:17:55.668 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T07:17:55.712 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-18T07:17:55.712 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T07:17:55.780 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-09-18T07:17:55.782 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-09-18T07:17:55.827 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T07:17:55.874 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T07:17:56.348 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-18T07:17:56.356 INFO:teuthology.task.internal:Starting timer... 2024-09-18T07:17:56.356 INFO:teuthology.run_tasks:Running task pcp... 2024-09-18T07:17:56.406 INFO:teuthology.run_tasks:Running task selinux... 2024-09-18T07:17:56.455 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-18T07:17:56.474 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2024-09-18T07:17:56.525 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2024-09-18T07:17:56.527 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2024-09-18T07:17:56.616 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2024-09-18T07:17:56.618 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-18T07:17:56.618 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2024-09-18T07:17:56.675 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2024-09-18T07:17:56.676 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2024-09-18T07:17:56.698 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2024-09-18T07:17:56.699 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi137.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2024-09-18T07:17:56.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-18T07:17:56.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:17:56.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-18T07:17:56.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:17:56.768 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-18T07:17:56.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2024-09-18T07:17:56.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2024-09-18T07:17:56.856 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-18T07:17:56.867 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-18T07:17:56.875 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-18T07:17:57.003 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-18T07:17:57.015 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-18T07:17:57.016 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi137.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-18T07:21:40.081 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi137.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-09-18T07:21:40.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-18T07:21:40.084 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:21:40.164 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-09-18T07:21:40.240 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-18T07:21:40.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-18T07:21:40.241 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T07:21:40.319 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-09-18T07:21:40.400 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-18T07:21:40.401 INFO:teuthology.run_tasks:Running task clock... 2024-09-18T07:21:40.412 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-18T07:21:40.412 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T07:21:40.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T07:21:40.415 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T07:21:40.415 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T07:21:40.442 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-18T07:21:40.456 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-18T07:21:40.473 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-18T07:21:40.480 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: ntpd: command not found 2024-09-18T07:21:40.489 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-18T07:21:40.490 INFO:teuthology.orchestra.run.smithi137.stdout:506 Cannot talk to daemon 2024-09-18T07:21:40.504 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-18T07:21:40.514 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2024-09-18T07:21:40.518 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-18T07:21:40.526 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2024-09-18T07:21:40.541 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-18T07:21:40.555 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-18T07:21:40.575 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-09-18T07:21:40.579 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T07:21:40.579 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-09-18T07:21:40.579 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T07:21:40.579 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T07:21:40.579 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T07:21:40.579 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T07:21:40.611 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-09-18T07:21:40.615 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T07:21:40.615 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-09-18T07:21:40.615 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T07:21:40.615 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T07:21:40.615 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T07:21:40.615 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T07:21:40.616 INFO:teuthology.run_tasks:Running task pexec... 2024-09-18T07:21:40.626 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-18T07:21:40.626 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi137.front.sepia.ceph.com 2024-09-18T07:21:40.626 DEBUG:teuthology.orchestra.run.smithi137:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-18T07:21:40.627 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi203.front.sepia.ceph.com 2024-09-18T07:21:40.627 DEBUG:teuthology.orchestra.run.smithi203:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-18T07:21:40.980 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:21:40.994 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:21:40.994 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-09-18T07:21:40.994 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:21:40.994 INFO:teuthology.orchestra.run.smithi137.stdout:Removing: 2024-09-18T07:21:40.994 INFO:teuthology.orchestra.run.smithi137.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-18T07:21:40.994 INFO:teuthology.orchestra.run.smithi137.stdout:Removing unused dependencies: 2024-09-18T07:21:40.994 INFO:teuthology.orchestra.run.smithi137.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-18T07:21:40.994 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:21:40.995 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:21:40.995 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:21:40.995 INFO:teuthology.orchestra.run.smithi137.stdout:Remove 2 Packages 2024-09-18T07:21:40.995 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:21:40.995 INFO:teuthology.orchestra.run.smithi137.stdout:Freed space: 5.5 M 2024-09-18T07:21:40.995 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:21:40.998 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:21:40.998 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:21:41.005 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:21:41.013 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:21:41.013 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-09-18T07:21:41.013 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:21:41.013 INFO:teuthology.orchestra.run.smithi203.stdout:Removing: 2024-09-18T07:21:41.013 INFO:teuthology.orchestra.run.smithi203.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-18T07:21:41.013 INFO:teuthology.orchestra.run.smithi203.stdout:Removing unused dependencies: 2024-09-18T07:21:41.013 INFO:teuthology.orchestra.run.smithi203.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-18T07:21:41.013 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:21:41.013 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:21:41.013 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:21:41.013 INFO:teuthology.orchestra.run.smithi203.stdout:Remove 2 Packages 2024-09-18T07:21:41.014 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:21:41.014 INFO:teuthology.orchestra.run.smithi203.stdout:Freed space: 5.5 M 2024-09-18T07:21:41.014 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:21:41.017 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:21:41.017 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:21:41.061 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:21:41.061 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:21:41.082 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:21:41.082 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:21:41.198 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:21:41.218 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:21:41.276 INFO:teuthology.orchestra.run.smithi137.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-18T07:21:41.291 INFO:teuthology.orchestra.run.smithi203.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-18T07:21:41.336 INFO:teuthology.orchestra.run.smithi137.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-18T07:21:41.354 INFO:teuthology.orchestra.run.smithi203.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-18T07:21:41.785 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-18T07:21:41.785 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-18T07:21:41.816 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-18T07:21:41.816 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-18T07:21:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-18T07:21:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:21:42.028 INFO:teuthology.orchestra.run.smithi137.stdout:Removed: 2024-09-18T07:21:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-18T07:21:42.028 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:21:42.028 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:21:42.047 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-18T07:21:42.047 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:21:42.047 INFO:teuthology.orchestra.run.smithi203.stdout:Removed: 2024-09-18T07:21:42.047 INFO:teuthology.orchestra.run.smithi203.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-18T07:21:42.047 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:21:42.047 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:21:42.867 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:47 ago on Wed 18 Sep 2024 07:20:55 AM UTC. 2024-09-18T07:21:42.893 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:47 ago on Wed 18 Sep 2024 07:20:55 AM UTC. 2024-09-18T07:21:42.999 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:21:43.000 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:21:43.000 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-09-18T07:21:43.000 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:21:43.000 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:21:43.000 INFO:teuthology.orchestra.run.smithi203.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-18T07:21:43.000 INFO:teuthology.orchestra.run.smithi203.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-18T07:21:43.000 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-18T07:21:43.000 INFO:teuthology.orchestra.run.smithi203.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-18T07:21:43.000 INFO:teuthology.orchestra.run.smithi203.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-18T07:21:43.001 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-18T07:21:43.001 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-18T07:21:43.001 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:21:43.001 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:21:43.001 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:21:43.001 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2024-09-18T07:21:43.001 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:21:43.007 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 2.0 M 2024-09-18T07:21:43.007 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 9.2 M 2024-09-18T07:21:43.007 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:21:43.026 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:21:43.026 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:21:43.027 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-09-18T07:21:43.027 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:21:43.027 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:21:43.027 INFO:teuthology.orchestra.run.smithi137.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-18T07:21:43.027 INFO:teuthology.orchestra.run.smithi137.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-18T07:21:43.028 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-18T07:21:43.028 INFO:teuthology.orchestra.run.smithi137.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-18T07:21:43.028 INFO:teuthology.orchestra.run.smithi137.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-18T07:21:43.028 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-18T07:21:43.028 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-18T07:21:43.028 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:21:43.028 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:21:43.028 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:21:43.028 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2024-09-18T07:21:43.028 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:21:43.036 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 2.0 M 2024-09-18T07:21:43.036 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 9.2 M 2024-09-18T07:21:43.036 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:21:43.252 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 378 kB/s | 44 kB 00:00 2024-09-18T07:21:43.267 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 397 kB/s | 44 kB 00:00 2024-09-18T07:21:43.277 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 741 kB/s | 106 kB 00:00 2024-09-18T07:21:43.292 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 773 kB/s | 106 kB 00:00 2024-09-18T07:21:43.303 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 1.4 MB/s | 72 kB 00:00 2024-09-18T07:21:43.337 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.4 MB/s | 84 kB 00:00 2024-09-18T07:21:43.343 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.6 MB/s | 84 kB 00:00 2024-09-18T07:21:43.368 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): python3-configshell-1.1.30-1.el9.noarch. 713 kB/s | 72 kB 00:00 2024-09-18T07:21:43.378 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 3.6 MB/s | 906 kB 00:00 2024-09-18T07:21:43.429 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 6.5 MB/s | 837 kB 00:00 2024-09-18T07:21:43.429 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:21:43.429 INFO:teuthology.orchestra.run.smithi203.stdout:Total 4.8 MB/s | 2.0 MB 00:00 2024-09-18T07:21:43.503 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:21:43.517 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:21:43.517 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:21:43.543 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 2.3 MB/s | 906 kB 00:00 2024-09-18T07:21:43.602 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 3.2 MB/s | 837 kB 00:00 2024-09-18T07:21:43.602 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:21:43.602 INFO:teuthology.orchestra.run.smithi137.stdout:Total 3.5 MB/s | 2.0 MB 00:00 2024-09-18T07:21:43.675 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:21:43.689 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:21:43.689 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:21:43.810 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:21:43.810 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:21:43.972 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:21:43.972 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:21:44.225 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:21:44.332 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-18T07:21:44.369 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-18T07:21:44.392 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:21:44.403 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-18T07:21:44.479 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-18T07:21:44.518 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-18T07:21:44.551 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-18T07:21:44.614 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-18T07:21:44.643 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-18T07:21:44.759 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-18T07:21:44.790 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-18T07:21:45.073 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-18T07:21:45.104 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-18T07:21:45.253 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-18T07:21:45.287 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-18T07:21:46.063 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-18T07:21:46.063 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-18T07:21:46.063 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-18T07:21:46.063 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-18T07:21:46.063 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-18T07:21:46.064 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-18T07:21:46.249 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-18T07:21:46.249 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-18T07:21:46.249 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-18T07:21:46.249 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-18T07:21:46.249 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-18T07:21:46.249 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-18T07:21:46.409 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-18T07:21:46.409 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:21:46.409 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:21:46.409 INFO:teuthology.orchestra.run.smithi203.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-18T07:21:46.409 INFO:teuthology.orchestra.run.smithi203.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-18T07:21:46.409 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-18T07:21:46.409 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:21:46.409 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:21:46.610 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-18T07:21:46.610 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:21:46.610 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:21:46.610 INFO:teuthology.orchestra.run.smithi137.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-18T07:21:46.610 INFO:teuthology.orchestra.run.smithi137.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-18T07:21:46.611 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-18T07:21:46.611 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:21:46.611 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:21:46.734 DEBUG:teuthology.parallel:result is None 2024-09-18T07:21:46.923 DEBUG:teuthology.parallel:result is None 2024-09-18T07:21:46.923 INFO:teuthology.run_tasks:Running task install... 2024-09-18T07:21:46.931 DEBUG:teuthology.task.install:project ceph 2024-09-18T07:21:46.931 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e'}} 2024-09-18T07:21:46.931 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e'} 2024-09-18T07:21:46.931 INFO:teuthology.task.install:Using flavor: default 2024-09-18T07:21:46.937 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-18T07:21:46.937 INFO:teuthology.task.install:extra packages: [] 2024-09-18T07:21:46.937 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e', 'tag': None, 'wait_for_package': False} 2024-09-18T07:21:46.938 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:21:46.940 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e', 'tag': None, 'wait_for_package': False} 2024-09-18T07:21:46.940 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:21:47.087 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/ 2024-09-18T07:21:47.171 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4983.ge7fb7b56 2024-09-18T07:21:47.174 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/ 2024-09-18T07:21:47.175 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4983.ge7fb7b56 2024-09-18T07:21:47.281 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-18T07:21:47.281 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-18T07:21:47.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-18T07:21:47.287 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-18T07:21:47.288 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-18T07:21:47.288 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-18T07:21:47.309 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-18T07:21:47.309 DEBUG:teuthology.orchestra.run.smithi137:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-18T07:21:47.317 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-18T07:21:47.317 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-18T07:21:47.375 DEBUG:teuthology.orchestra.run.smithi137:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-18T07:21:47.384 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-18T07:21:47.457 DEBUG:teuthology.orchestra.run.smithi137:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-18T07:21:47.459 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-18T07:21:47.521 INFO:teuthology.orchestra.run.smithi137.stdout:check_obsoletes = 1 2024-09-18T07:21:47.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum clean all 2024-09-18T07:21:47.523 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2024-09-18T07:21:47.525 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2024-09-18T07:21:47.743 INFO:teuthology.orchestra.run.smithi203.stdout:45 files removed 2024-09-18T07:21:47.747 INFO:teuthology.orchestra.run.smithi137.stdout:45 files removed 2024-09-18T07:21:47.770 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw 2024-09-18T07:21:47.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-radosgw 2024-09-18T07:21:48.325 INFO:teuthology.orchestra.run.smithi137.stdout:ceph packages for x86_64 319 kB/s | 84 kB 00:00 2024-09-18T07:21:48.327 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 311 kB/s | 84 kB 00:00 2024-09-18T07:21:48.618 INFO:teuthology.orchestra.run.smithi137.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-18T07:21:48.620 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-18T07:21:48.893 INFO:teuthology.orchestra.run.smithi137.stdout:ceph source packages 8.2 kB/s | 1.9 kB 00:00 2024-09-18T07:21:48.895 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-18T07:21:49.387 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.3 MB 00:00 2024-09-18T07:21:49.585 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.3 MB 00:00 2024-09-18T07:21:51.247 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-09-18T07:21:51.703 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 19 MB/s | 20 MB 00:01 2024-09-18T07:21:55.531 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-18T07:21:56.587 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 5.9 MB/s | 6.5 MB 00:01 2024-09-18T07:21:57.207 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 19 kB 00:00 2024-09-18T07:21:58.629 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-18T07:21:58.904 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 19 kB 00:01 2024-09-18T07:22:00.392 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-18T07:22:04.068 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-18T07:22:05.731 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-18T07:22:06.289 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:22:06.290 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:22:06.290 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-18T07:22:06.290 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:22:06.290 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:22:06.290 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 11 M 2024-09-18T07:22:06.290 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2024-09-18T07:22:06.291 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 3.5 M 2024-09-18T07:22:06.291 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 3.2 M 2024-09-18T07:22:06.291 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-18T07:22:06.291 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-18T07:22:06.291 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 5.5 M 2024-09-18T07:22:06.291 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 22 M 2024-09-18T07:22:06.291 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 25 k 2024-09-18T07:22:06.291 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-18T07:22:06.291 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-18T07:22:06.291 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-18T07:22:06.291 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 755 k 2024-09-18T07:22:06.292 INFO:teuthology.orchestra.run.smithi203.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-18T07:22:06.292 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-18T07:22:06.292 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-18T07:22:06.292 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-18T07:22:06.292 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 509 k 2024-09-18T07:22:06.292 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-18T07:22:06.292 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 5.5 M 2024-09-18T07:22:06.292 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-18T07:22:06.292 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-18T07:22:06.292 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-18T07:22:06.293 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 45 k 2024-09-18T07:22:06.293 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 143 k 2024-09-18T07:22:06.293 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 161 k 2024-09-18T07:22:06.293 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-18T07:22:06.293 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 320 k 2024-09-18T07:22:06.293 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 306 k 2024-09-18T07:22:06.293 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 99 k 2024-09-18T07:22:06.293 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-18T07:22:06.293 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-18T07:22:06.293 INFO:teuthology.orchestra.run.smithi203.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-18T07:22:06.293 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-18T07:22:06.293 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2024-09-18T07:22:06.294 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-18T07:22:06.294 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:06.294 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:22:06.294 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:22:06.294 INFO:teuthology.orchestra.run.smithi203.stdout:Install 31 Packages 2024-09-18T07:22:06.294 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 2 Packages 2024-09-18T07:22:06.294 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:06.294 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 62 M 2024-09-18T07:22:06.295 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:22:06.911 INFO:teuthology.orchestra.run.smithi203.stdout:(1/33): ceph-base-19.3.0-4983.ge7fb7b56.el9.x86 10 MB/s | 5.5 MB 00:00 2024-09-18T07:22:06.961 INFO:teuthology.orchestra.run.smithi203.stdout:(2/33): ceph-selinux-19.3.0-4983.ge7fb7b56.el9. 503 kB/s | 25 kB 00:00 2024-09-18T07:22:07.070 INFO:teuthology.orchestra.run.smithi203.stdout:(3/33): libcephfs2-19.3.0-4983.ge7fb7b56.el9.x8 6.8 MB/s | 755 kB 00:00 2024-09-18T07:22:07.212 INFO:teuthology.orchestra.run.smithi203.stdout:(4/33): ceph-radosgw-19.3.0-4983.ge7fb7b56.el9. 13 MB/s | 11 MB 00:00 2024-09-18T07:22:07.245 INFO:teuthology.orchestra.run.smithi203.stdout:(5/33): libradosstriper1-19.3.0-4983.ge7fb7b56. 2.8 MB/s | 509 kB 00:00 2024-09-18T07:22:07.312 INFO:teuthology.orchestra.run.smithi203.stdout:(6/33): python3-ceph-argparse-19.3.0-4983.ge7fb 676 kB/s | 45 kB 00:00 2024-09-18T07:22:07.379 INFO:teuthology.orchestra.run.smithi203.stdout:(7/33): python3-ceph-common-19.3.0-4983.ge7fb7b 2.1 MB/s | 143 kB 00:00 2024-09-18T07:22:07.454 INFO:teuthology.orchestra.run.smithi203.stdout:(8/33): python3-cephfs-19.3.0-4983.ge7fb7b56.el 2.1 MB/s | 161 kB 00:00 2024-09-18T07:22:07.530 INFO:teuthology.orchestra.run.smithi203.stdout:(9/33): librgw2-19.3.0-4983.ge7fb7b56.el9.x86_6 17 MB/s | 5.5 MB 00:00 2024-09-18T07:22:07.555 INFO:teuthology.orchestra.run.smithi203.stdout:(10/33): python3-rados-19.3.0-4983.ge7fb7b56.el 3.1 MB/s | 320 kB 00:00 2024-09-18T07:22:07.653 INFO:teuthology.orchestra.run.smithi203.stdout:(11/33): python3-rbd-19.3.0-4983.ge7fb7b56.el9. 6.0 MB/s | 306 kB 00:00 2024-09-18T07:22:07.653 INFO:teuthology.orchestra.run.smithi203.stdout:(12/33): python3-rgw-19.3.0-4983.ge7fb7b56.el9. 1.9 MB/s | 99 kB 00:00 2024-09-18T07:22:07.823 INFO:teuthology.orchestra.run.smithi203.stdout:(13/33): ceph-common-19.3.0-4983.ge7fb7b56.el9. 15 MB/s | 22 MB 00:01 2024-09-18T07:22:07.865 INFO:teuthology.orchestra.run.smithi203.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 376 kB/s | 107 kB 00:00 2024-09-18T07:22:07.899 INFO:teuthology.orchestra.run.smithi203.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-09-18T07:22:07.923 INFO:teuthology.orchestra.run.smithi203.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 537 kB/s | 171 kB 00:00 2024-09-18T07:22:07.949 INFO:teuthology.orchestra.run.smithi203.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-09-18T07:22:07.963 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:22:07.965 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:22:07.965 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-18T07:22:07.965 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:22:07.965 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:22:07.965 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 11 M 2024-09-18T07:22:07.965 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrading: 2024-09-18T07:22:07.965 INFO:teuthology.orchestra.run.smithi137.stdout: librados2 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 3.5 M 2024-09-18T07:22:07.966 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 3.2 M 2024-09-18T07:22:07.966 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-18T07:22:07.966 INFO:teuthology.orchestra.run.smithi137.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-18T07:22:07.966 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 5.5 M 2024-09-18T07:22:07.966 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 22 M 2024-09-18T07:22:07.966 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 25 k 2024-09-18T07:22:07.966 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-18T07:22:07.966 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-18T07:22:07.966 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-18T07:22:07.966 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 755 k 2024-09-18T07:22:07.966 INFO:teuthology.orchestra.run.smithi137.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-18T07:22:07.966 INFO:teuthology.orchestra.run.smithi137.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-18T07:22:07.966 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-18T07:22:07.967 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-18T07:22:07.967 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 509 k 2024-09-18T07:22:07.967 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-18T07:22:07.967 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2 x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 5.5 M 2024-09-18T07:22:07.967 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-18T07:22:07.967 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-18T07:22:07.967 INFO:teuthology.orchestra.run.smithi137.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-18T07:22:07.967 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 45 k 2024-09-18T07:22:07.967 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 143 k 2024-09-18T07:22:07.967 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 161 k 2024-09-18T07:22:07.967 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-18T07:22:07.967 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 320 k 2024-09-18T07:22:07.968 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 306 k 2024-09-18T07:22:07.968 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 99 k 2024-09-18T07:22:07.968 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-18T07:22:07.968 INFO:teuthology.orchestra.run.smithi137.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-18T07:22:07.968 INFO:teuthology.orchestra.run.smithi137.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-18T07:22:07.968 INFO:teuthology.orchestra.run.smithi137.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-18T07:22:07.968 INFO:teuthology.orchestra.run.smithi137.stdout:Installing weak dependencies: 2024-09-18T07:22:07.968 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-18T07:22:07.968 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:07.968 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:22:07.968 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:22:07.968 INFO:teuthology.orchestra.run.smithi137.stdout:Install 31 Packages 2024-09-18T07:22:07.969 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrade 2 Packages 2024-09-18T07:22:07.969 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:07.969 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 62 M 2024-09-18T07:22:07.969 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:22:07.991 INFO:teuthology.orchestra.run.smithi203.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-09-18T07:22:08.017 INFO:teuthology.orchestra.run.smithi203.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-09-18T07:22:08.033 INFO:teuthology.orchestra.run.smithi203.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 502 kB/s | 42 kB 00:00 2024-09-18T07:22:08.050 INFO:teuthology.orchestra.run.smithi203.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 36 kB 00:00 2024-09-18T07:22:08.067 INFO:teuthology.orchestra.run.smithi203.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-09-18T07:22:08.093 INFO:teuthology.orchestra.run.smithi203.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-09-18T07:22:08.151 INFO:teuthology.orchestra.run.smithi203.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 592 kB/s | 26 kB 00:00 2024-09-18T07:22:08.177 INFO:teuthology.orchestra.run.smithi203.stdout:(25/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.4 MB/s | 308 kB 00:00 2024-09-18T07:22:08.202 INFO:teuthology.orchestra.run.smithi203.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 967 kB/s | 49 kB 00:00 2024-09-18T07:22:08.227 INFO:teuthology.orchestra.run.smithi203.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-18T07:22:08.277 INFO:teuthology.orchestra.run.smithi203.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-18T07:22:08.337 INFO:teuthology.orchestra.run.smithi203.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.1 MB/s | 839 kB 00:00 2024-09-18T07:22:08.528 INFO:teuthology.orchestra.run.smithi203.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-09-18T07:22:08.629 INFO:teuthology.orchestra.run.smithi203.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 7.9 MB/s | 4.4 MB 00:00 2024-09-18T07:22:08.687 INFO:teuthology.orchestra.run.smithi203.stdout:(32/33): librados2-19.3.0-4983.ge7fb7b56.el9.x8 9.9 MB/s | 3.5 MB 00:00 2024-09-18T07:22:08.790 INFO:teuthology.orchestra.run.smithi137.stdout:(1/33): ceph-base-19.3.0-4983.ge7fb7b56.el9.x86 8.7 MB/s | 5.5 MB 00:00 2024-09-18T07:22:08.821 INFO:teuthology.orchestra.run.smithi203.stdout:(33/33): librbd1-19.3.0-4983.ge7fb7b56.el9.x86_ 11 MB/s | 3.2 MB 00:00 2024-09-18T07:22:08.825 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:22:08.825 INFO:teuthology.orchestra.run.smithi203.stdout:Total 24 MB/s | 62 MB 00:02 2024-09-18T07:22:08.857 INFO:teuthology.orchestra.run.smithi137.stdout:(2/33): ceph-selinux-19.3.0-4983.ge7fb7b56.el9. 377 kB/s | 25 kB 00:00 2024-09-18T07:22:08.938 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:22:08.966 INFO:teuthology.orchestra.run.smithi137.stdout:(3/33): libcephfs2-19.3.0-4983.ge7fb7b56.el9.x8 6.8 MB/s | 755 kB 00:00 2024-09-18T07:22:08.974 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:22:08.974 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:22:09.082 INFO:teuthology.orchestra.run.smithi137.stdout:(4/33): ceph-radosgw-19.3.0-4983.ge7fb7b56.el9. 12 MB/s | 11 MB 00:00 2024-09-18T07:22:09.108 INFO:teuthology.orchestra.run.smithi137.stdout:(5/33): libradosstriper1-19.3.0-4983.ge7fb7b56. 3.5 MB/s | 509 kB 00:00 2024-09-18T07:22:09.167 INFO:teuthology.orchestra.run.smithi137.stdout:(6/33): python3-ceph-argparse-19.3.0-4983.ge7fb 774 kB/s | 45 kB 00:00 2024-09-18T07:22:09.233 INFO:teuthology.orchestra.run.smithi137.stdout:(7/33): python3-ceph-common-19.3.0-4983.ge7fb7b 2.1 MB/s | 143 kB 00:00 2024-09-18T07:22:09.300 INFO:teuthology.orchestra.run.smithi137.stdout:(8/33): python3-cephfs-19.3.0-4983.ge7fb7b56.el 2.4 MB/s | 161 kB 00:00 2024-09-18T07:22:09.384 INFO:teuthology.orchestra.run.smithi137.stdout:(9/33): librgw2-19.3.0-4983.ge7fb7b56.el9.x86_6 18 MB/s | 5.5 MB 00:00 2024-09-18T07:22:09.409 INFO:teuthology.orchestra.run.smithi137.stdout:(10/33): python3-rados-19.3.0-4983.ge7fb7b56.el 2.9 MB/s | 320 kB 00:00 2024-09-18T07:22:09.434 INFO:teuthology.orchestra.run.smithi137.stdout:(11/33): python3-rbd-19.3.0-4983.ge7fb7b56.el9. 5.9 MB/s | 306 kB 00:00 2024-09-18T07:22:09.459 INFO:teuthology.orchestra.run.smithi137.stdout:(12/33): python3-rgw-19.3.0-4983.ge7fb7b56.el9. 1.9 MB/s | 99 kB 00:00 2024-09-18T07:22:09.471 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:22:09.471 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:22:09.660 INFO:teuthology.orchestra.run.smithi137.stdout:(13/33): ceph-common-19.3.0-4983.ge7fb7b56.el9. 14 MB/s | 22 MB 00:01 2024-09-18T07:22:09.793 INFO:teuthology.orchestra.run.smithi137.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 298 kB/s | 107 kB 00:00 2024-09-18T07:22:09.819 INFO:teuthology.orchestra.run.smithi137.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 475 kB/s | 171 kB 00:00 2024-09-18T07:22:09.836 INFO:teuthology.orchestra.run.smithi137.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-18T07:22:09.861 INFO:teuthology.orchestra.run.smithi137.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 799 kB/s | 160 kB 00:00 2024-09-18T07:22:09.903 INFO:teuthology.orchestra.run.smithi137.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-18T07:22:09.936 INFO:teuthology.orchestra.run.smithi137.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-09-18T07:22:09.962 INFO:teuthology.orchestra.run.smithi137.stdout:(20/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-18T07:22:09.987 INFO:teuthology.orchestra.run.smithi137.stdout:(21/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-18T07:22:10.004 INFO:teuthology.orchestra.run.smithi137.stdout:(22/33): qatlib-service-24.02.0-1.el9.x86_64.rp 535 kB/s | 36 kB 00:00 2024-09-18T07:22:10.021 INFO:teuthology.orchestra.run.smithi137.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 805 kB/s | 47 kB 00:00 2024-09-18T07:22:10.079 INFO:teuthology.orchestra.run.smithi137.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.3 MB/s | 308 kB 00:00 2024-09-18T07:22:10.105 INFO:teuthology.orchestra.run.smithi137.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.0 MB/s | 49 kB 00:00 2024-09-18T07:22:10.130 INFO:teuthology.orchestra.run.smithi137.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 276 kB/s | 26 kB 00:00 2024-09-18T07:22:10.155 INFO:teuthology.orchestra.run.smithi137.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-18T07:22:10.205 INFO:teuthology.orchestra.run.smithi137.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 3.8 MB/s | 191 kB 00:00 2024-09-18T07:22:10.289 INFO:teuthology.orchestra.run.smithi137.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.2 MB/s | 839 kB 00:00 2024-09-18T07:22:10.406 INFO:teuthology.orchestra.run.smithi137.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-09-18T07:22:10.548 INFO:teuthology.orchestra.run.smithi137.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 8.1 MB/s | 4.4 MB 00:00 2024-09-18T07:22:10.606 INFO:teuthology.orchestra.run.smithi137.stdout:(32/33): librados2-19.3.0-4983.ge7fb7b56.el9.x8 11 MB/s | 3.5 MB 00:00 2024-09-18T07:22:10.673 INFO:teuthology.orchestra.run.smithi137.stdout:(33/33): librbd1-19.3.0-4983.ge7fb7b56.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-09-18T07:22:10.678 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:22:10.678 INFO:teuthology.orchestra.run.smithi137.stdout:Total 23 MB/s | 62 MB 00:02 2024-09-18T07:22:10.791 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:22:10.828 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:22:10.828 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:22:10.957 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:22:11.040 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-18T07:22:11.076 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-18T07:22:11.101 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-18T07:22:11.124 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-18T07:22:11.144 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-18T07:22:11.333 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-18T07:22:11.340 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:22:11.340 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:22:11.386 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-18T07:22:11.783 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-18T07:22:11.852 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 7/35 2024-09-18T07:22:11.945 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 7/35 2024-09-18T07:22:12.035 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:19.3.0-4983.ge7fb7b56.el9.x86_64 8/35 2024-09-18T07:22:12.062 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 9/35 2024-09-18T07:22:12.117 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 9/35 2024-09-18T07:22:12.154 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-18T07:22:12.206 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:19.3.0-4983.ge7fb7b56.el9. 11/35 2024-09-18T07:22:12.277 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:19.3.0-4983.ge7fb7b56.el9.x86_64 12/35 2024-09-18T07:22:12.306 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:19.3.0-4983.ge7fb7b56.el9.x86_6 13/35 2024-09-18T07:22:12.370 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4983.ge7fb7b56.el9.x86_6 13/35 2024-09-18T07:22:12.405 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-18T07:22:12.529 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-18T07:22:12.564 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-18T07:22:12.613 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-18T07:22:12.646 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-18T07:22:12.675 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-18T07:22:12.709 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-18T07:22:12.746 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-18T07:22:12.815 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:22:12.891 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-18T07:22:12.929 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-18T07:22:12.963 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-18T07:22:12.985 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-18T07:22:13.005 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-18T07:22:13.077 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-18T07:22:13.122 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 23/35 2024-09-18T07:22:13.196 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-18T07:22:13.218 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 23/35 2024-09-18T07:22:13.250 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-18T07:22:13.262 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 24/35 2024-09-18T07:22:13.305 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-18T07:22:13.600 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-18T07:22:13.713 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 7/35 2024-09-18T07:22:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-18T07:22:13.806 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 7/35 2024-09-18T07:22:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 27/35 2024-09-18T07:22:13.888 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 27/35 2024-09-18T07:22:13.904 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rados-2:19.3.0-4983.ge7fb7b56.el9.x86_64 8/35 2024-09-18T07:22:13.932 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 9/35 2024-09-18T07:22:13.975 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 28/35 2024-09-18T07:22:14.003 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x8 29/35 2024-09-18T07:22:14.003 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 9/35 2024-09-18T07:22:14.032 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-18T07:22:14.079 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-argparse-2:19.3.0-4983.ge7fb7b56.el9. 11/35 2024-09-18T07:22:14.146 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cephfs-2:19.3.0-4983.ge7fb7b56.el9.x86_64 12/35 2024-09-18T07:22:14.168 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libradosstriper1-2:19.3.0-4983.ge7fb7b56.el9.x86_6 13/35 2024-09-18T07:22:14.232 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4983.ge7fb7b56.el9.x86_6 13/35 2024-09-18T07:22:14.266 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-18T07:22:14.376 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-18T07:22:14.409 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-18T07:22:14.454 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-18T07:22:14.482 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-18T07:22:14.511 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-18T07:22:14.544 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-18T07:22:14.581 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-18T07:22:14.897 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-18T07:22:14.941 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 23/35 2024-09-18T07:22:15.023 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 23/35 2024-09-18T07:22:15.054 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 24/35 2024-09-18T07:22:15.090 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-18T07:22:15.580 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-18T07:22:15.610 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librgw2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 27/35 2024-09-18T07:22:15.681 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librgw2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 27/35 2024-09-18T07:22:15.751 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 28/35 2024-09-18T07:22:15.787 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x8 29/35 2024-09-18T07:22:16.349 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 30/35 2024-09-18T07:22:16.391 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 30/35 2024-09-18T07:22:16.972 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 30/35 2024-09-18T07:22:17.011 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 31/35 2024-09-18T07:22:17.104 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 31/35 2024-09-18T07:22:17.104 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-18T07:22:17.104 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-18T07:22:17.104 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:17.130 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:19.3.0-4983.ge7fb7b56.el9.x86_64 32/35 2024-09-18T07:22:18.094 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 30/35 2024-09-18T07:22:18.134 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 30/35 2024-09-18T07:22:18.701 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 30/35 2024-09-18T07:22:18.729 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 31/35 2024-09-18T07:22:18.822 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 31/35 2024-09-18T07:22:18.823 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-18T07:22:18.823 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-18T07:22:18.823 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:18.855 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-selinux-2:19.3.0-4983.ge7fb7b56.el9.x86_64 32/35 2024-09-18T07:22:27.367 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4983.ge7fb7b56.el9.x86_64 32/35 2024-09-18T07:22:27.367 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2024-09-18T07:22:27.367 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2024-09-18T07:22:27.367 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2024-09-18T07:22:27.368 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2024-09-18T07:22:27.368 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2024-09-18T07:22:27.368 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /root 2024-09-18T07:22:27.368 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2024-09-18T07:22:27.368 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:28.400 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 33/35 2024-09-18T07:22:28.429 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 33/35 2024-09-18T07:22:28.429 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:22:28.429 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-18T07:22:28.429 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T07:22:28.429 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T07:22:28.429 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:28.429 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-18T07:22:28.502 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-18T07:22:28.502 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-18T07:22:28.878 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4983.ge7fb7b56.el9.x86_64 32/35 2024-09-18T07:22:28.879 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /sys 2024-09-18T07:22:28.879 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /proc 2024-09-18T07:22:28.879 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /mnt 2024-09-18T07:22:28.879 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /var/tmp 2024-09-18T07:22:28.879 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /home 2024-09-18T07:22:28.879 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /root 2024-09-18T07:22:28.879 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /tmp 2024-09-18T07:22:28.879 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:29.689 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-18T07:22:29.689 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/35 2024-09-18T07:22:29.690 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/35 2024-09-18T07:22:29.690 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 3/35 2024-09-18T07:22:29.690 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:19.3.0-4983.ge7fb7b56.el9.x86_64 4/35 2024-09-18T07:22:29.690 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 5/35 2024-09-18T07:22:29.690 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:19.3.0-4983.ge7fb7b56.el9.x86_6 6/35 2024-09-18T07:22:29.690 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 7/35 2024-09-18T07:22:29.690 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4983.ge7fb7b56.el9. 8/35 2024-09-18T07:22:29.690 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x8 9/35 2024-09-18T07:22:29.690 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:19.3.0-4983.ge7fb7b56.el9.x86_64 10/35 2024-09-18T07:22:29.690 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:19.3.0-4983.ge7fb7b56.el9.x86_64 11/35 2024-09-18T07:22:29.690 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 12/35 2024-09-18T07:22:29.691 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 13/35 2024-09-18T07:22:29.691 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-18T07:22:29.691 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-18T07:22:29.691 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-18T07:22:29.691 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-18T07:22:29.691 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-18T07:22:29.691 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-18T07:22:29.691 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-18T07:22:29.691 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-18T07:22:29.691 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-18T07:22:29.692 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-18T07:22:29.692 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-18T07:22:29.692 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-18T07:22:29.692 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-18T07:22:29.692 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-18T07:22:29.693 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-18T07:22:29.693 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-18T07:22:29.693 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-18T07:22:29.693 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-18T07:22:29.694 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 32/35 2024-09-18T07:22:29.694 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-18T07:22:29.694 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 34/35 2024-09-18T07:22:29.912 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-radosgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 33/35 2024-09-18T07:22:29.938 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 33/35 2024-09-18T07:22:29.939 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:22:29.939 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-18T07:22:29.939 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T07:22:29.939 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-18T07:22:29.939 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:29.940 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-18T07:22:30.032 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-18T07:22:30.032 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-18T07:22:30.616 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-18T07:22:30.616 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:30.616 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2024-09-18T07:22:30.616 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:30.616 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:30.617 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:22:30.617 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-18T07:22:30.617 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:30.617 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:30.617 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:30.617 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:30.617 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-18T07:22:30.617 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-18T07:22:30.617 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-18T07:22:30.617 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:30.617 INFO:teuthology.orchestra.run.smithi203.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-18T07:22:30.617 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-18T07:22:30.617 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-18T07:22:30.618 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-18T07:22:30.618 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:30.618 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-18T07:22:30.618 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:30.618 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-18T07:22:30.618 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-18T07:22:30.618 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-18T07:22:30.618 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:30.618 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:30.618 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:30.618 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-18T07:22:30.618 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:30.619 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:30.619 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:30.619 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-18T07:22:30.619 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-18T07:22:30.619 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-18T07:22:30.619 INFO:teuthology.orchestra.run.smithi203.stdout: re2-1:20211101-20.el9.x86_64 2024-09-18T07:22:30.619 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-18T07:22:30.619 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:30.619 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:22:31.004 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-test 2024-09-18T07:22:31.264 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-18T07:22:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/35 2024-09-18T07:22:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/35 2024-09-18T07:22:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-radosgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 3/35 2024-09-18T07:22:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-selinux-2:19.3.0-4983.ge7fb7b56.el9.x86_64 4/35 2024-09-18T07:22:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 5/35 2024-09-18T07:22:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libradosstriper1-2:19.3.0-4983.ge7fb7b56.el9.x86_6 6/35 2024-09-18T07:22:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librgw2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 7/35 2024-09-18T07:22:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4983.ge7fb7b56.el9. 8/35 2024-09-18T07:22:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x8 9/35 2024-09-18T07:22:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cephfs-2:19.3.0-4983.ge7fb7b56.el9.x86_64 10/35 2024-09-18T07:22:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rados-2:19.3.0-4983.ge7fb7b56.el9.x86_64 11/35 2024-09-18T07:22:31.265 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 12/35 2024-09-18T07:22:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 13/35 2024-09-18T07:22:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-18T07:22:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-18T07:22:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-18T07:22:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-18T07:22:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-18T07:22:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-18T07:22:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-18T07:22:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-18T07:22:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-18T07:22:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-18T07:22:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-18T07:22:31.266 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-18T07:22:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-18T07:22:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-18T07:22:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-18T07:22:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-18T07:22:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-18T07:22:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-18T07:22:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 32/35 2024-09-18T07:22:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-18T07:22:31.267 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 34/35 2024-09-18T07:22:31.593 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:27 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:22:31.725 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:22:31.726 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:22:31.726 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-18T07:22:31.726 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:22:31.726 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:22:31.726 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 46 M 2024-09-18T07:22:31.726 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-18T07:22:31.726 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 161 k 2024-09-18T07:22:31.726 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-18T07:22:31.726 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-18T07:22:31.727 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:31.727 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:22:31.727 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:22:31.727 INFO:teuthology.orchestra.run.smithi203.stdout:Install 4 Packages 2024-09-18T07:22:31.727 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:31.727 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 46 M 2024-09-18T07:22:31.728 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 201 M 2024-09-18T07:22:31.728 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:22:32.004 INFO:teuthology.orchestra.run.smithi203.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.0 MB/s | 304 kB 00:00 2024-09-18T07:22:32.045 INFO:teuthology.orchestra.run.smithi203.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-18T07:22:32.087 INFO:teuthology.orchestra.run.smithi203.stdout:(3/4): libcephsqlite-19.3.0-4983.ge7fb7b56.el9. 689 kB/s | 161 kB 00:00 2024-09-18T07:22:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-18T07:22:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:32.258 INFO:teuthology.orchestra.run.smithi137.stdout:Upgraded: 2024-09-18T07:22:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:32.258 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:22:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-18T07:22:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-18T07:22:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-18T07:22:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-18T07:22:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-18T07:22:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-18T07:22:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-18T07:22:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-18T07:22:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-18T07:22:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-18T07:22:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-18T07:22:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-18T07:22:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-18T07:22:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-18T07:22:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-18T07:22:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-18T07:22:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: re2-1:20211101-20.el9.x86_64 2024-09-18T07:22:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-18T07:22:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:32.261 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:22:32.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-test 2024-09-18T07:22:33.229 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:28 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:22:33.271 INFO:teuthology.orchestra.run.smithi203.stdout:(4/4): ceph-test-19.3.0-4983.ge7fb7b56.el9.x86_ 32 MB/s | 46 MB 00:01 2024-09-18T07:22:33.273 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:22:33.274 INFO:teuthology.orchestra.run.smithi203.stdout:Total 30 MB/s | 46 MB 00:01 2024-09-18T07:22:33.302 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:22:33.323 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:22:33.323 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:22:33.360 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:22:33.361 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:22:33.361 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-18T07:22:33.361 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:22:33.361 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:22:33.361 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 46 M 2024-09-18T07:22:33.361 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-18T07:22:33.361 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 161 k 2024-09-18T07:22:33.361 INFO:teuthology.orchestra.run.smithi137.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-18T07:22:33.361 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-18T07:22:33.362 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:33.362 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:22:33.362 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:22:33.362 INFO:teuthology.orchestra.run.smithi137.stdout:Install 4 Packages 2024-09-18T07:22:33.362 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:33.362 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 46 M 2024-09-18T07:22:33.362 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 201 M 2024-09-18T07:22:33.362 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:22:33.630 INFO:teuthology.orchestra.run.smithi137.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-09-18T07:22:33.672 INFO:teuthology.orchestra.run.smithi137.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-18T07:22:33.689 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:22:33.690 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:22:33.714 INFO:teuthology.orchestra.run.smithi137.stdout:(3/4): libcephsqlite-19.3.0-4983.ge7fb7b56.el9. 708 kB/s | 161 kB 00:00 2024-09-18T07:22:34.195 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:22:34.283 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-18T07:22:34.337 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-18T07:22:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:19.3.0-4983.ge7fb7b56.el9.x86_64 3/4 2024-09-18T07:22:34.931 INFO:teuthology.orchestra.run.smithi137.stdout:(4/4): ceph-test-19.3.0-4983.ge7fb7b56.el9.x86_ 31 MB/s | 46 MB 00:01 2024-09-18T07:22:34.933 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:22:34.933 INFO:teuthology.orchestra.run.smithi137.stdout:Total 29 MB/s | 46 MB 00:01 2024-09-18T07:22:34.961 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:22:34.982 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:22:34.982 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:22:35.346 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:22:35.347 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:22:35.859 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:22:35.941 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-18T07:22:35.980 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-18T07:22:36.007 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephsqlite-2:19.3.0-4983.ge7fb7b56.el9.x86_64 3/4 2024-09-18T07:22:38.732 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4983.ge7fb7b56.el9.x86_64 3/4 2024-09-18T07:22:38.805 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:19.3.0-4983.ge7fb7b56.el9.x86_64 4/4 2024-09-18T07:22:40.354 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4983.ge7fb7b56.el9.x86_64 3/4 2024-09-18T07:22:40.447 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-test-2:19.3.0-4983.ge7fb7b56.el9.x86_64 4/4 2024-09-18T07:22:41.271 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-test-2:19.3.0-4983.ge7fb7b56.el9.x86_64 4/4 2024-09-18T07:22:41.271 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/4 2024-09-18T07:22:41.271 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/4 2024-09-18T07:22:41.271 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-18T07:22:41.709 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-18T07:22:41.709 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:41.709 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:22:41.709 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:41.709 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:41.709 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-18T07:22:41.709 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-18T07:22:41.710 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:41.710 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:22:41.977 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph 2024-09-18T07:22:42.591 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:38 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:22:42.733 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:22:42.736 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:22:42.736 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-18T07:22:42.736 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:22:42.736 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:22:42.736 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 6.6 k 2024-09-18T07:22:42.736 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-18T07:22:42.736 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 2.4 M 2024-09-18T07:22:42.736 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 1.5 M 2024-09-18T07:22:42.736 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 291 k 2024-09-18T07:22:42.736 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 4.8 M 2024-09-18T07:22:42.737 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 17 M 2024-09-18T07:22:42.737 INFO:teuthology.orchestra.run.smithi203.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-18T07:22:42.737 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-18T07:22:42.737 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-18T07:22:42.737 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-18T07:22:42.737 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-18T07:22:42.737 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-18T07:22:42.737 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-18T07:22:42.737 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-18T07:22:42.738 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-18T07:22:42.738 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-18T07:22:42.738 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections 2024-09-18T07:22:42.738 INFO:teuthology.orchestra.run.smithi203.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-18T07:22:42.738 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-18T07:22:42.738 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-18T07:22:42.738 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-18T07:22:42.738 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-18T07:22:42.739 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-18T07:22:42.739 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-18T07:22:42.739 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-18T07:22:42.739 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-18T07:22:42.739 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-18T07:22:42.739 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-18T07:22:42.739 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-18T07:22:42.739 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-18T07:22:42.739 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-18T07:22:42.739 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-18T07:22:42.739 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-18T07:22:42.739 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-18T07:22:42.740 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-18T07:22:42.740 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-18T07:22:42.740 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-18T07:22:42.740 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-18T07:22:42.740 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:42.740 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:22:42.740 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:22:42.740 INFO:teuthology.orchestra.run.smithi203.stdout:Install 37 Packages 2024-09-18T07:22:42.741 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:42.742 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 31 M 2024-09-18T07:22:42.742 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 109 M 2024-09-18T07:22:42.742 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:22:42.839 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-test-2:19.3.0-4983.ge7fb7b56.el9.x86_64 4/4 2024-09-18T07:22:42.839 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-test-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/4 2024-09-18T07:22:42.839 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephsqlite-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/4 2024-09-18T07:22:42.839 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-18T07:22:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-18T07:22:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:43.210 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:22:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-18T07:22:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-18T07:22:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:43.211 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:22:43.325 INFO:teuthology.orchestra.run.smithi203.stdout:(1/37): ceph-19.3.0-4983.ge7fb7b56.el9.x86_64.r 49 kB/s | 6.6 kB 00:00 2024-09-18T07:22:43.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph 2024-09-18T07:22:43.525 INFO:teuthology.orchestra.run.smithi203.stdout:(2/37): ceph-mgr-19.3.0-4983.ge7fb7b56.el9.x86_ 4.6 MB/s | 1.5 MB 00:00 2024-09-18T07:22:43.567 INFO:teuthology.orchestra.run.smithi203.stdout:(3/37): ceph-mds-19.3.0-4983.ge7fb7b56.el9.x86_ 6.3 MB/s | 2.4 MB 00:00 2024-09-18T07:22:43.642 INFO:teuthology.orchestra.run.smithi203.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4983.ge7fb 3.8 MB/s | 291 kB 00:00 2024-09-18T07:22:43.768 INFO:teuthology.orchestra.run.smithi203.stdout:(5/37): ceph-mon-19.3.0-4983.ge7fb7b56.el9.x86_ 11 MB/s | 4.8 MB 00:00 2024-09-18T07:22:43.826 INFO:teuthology.orchestra.run.smithi203.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-18T07:22:43.910 INFO:teuthology.orchestra.run.smithi203.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 106 kB 00:00 2024-09-18T07:22:43.961 INFO:teuthology.orchestra.run.smithi203.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 6.4 MB/s | 1.2 MB 00:00 2024-09-18T07:22:44.107 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:39 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:22:44.136 INFO:teuthology.orchestra.run.smithi203.stdout:(9/37): ceph-osd-19.3.0-4983.ge7fb7b56.el9.x86_ 28 MB/s | 17 MB 00:00 2024-09-18T07:22:44.161 INFO:teuthology.orchestra.run.smithi203.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 539 kB/s | 135 kB 00:00 2024-09-18T07:22:44.186 INFO:teuthology.orchestra.run.smithi203.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 553 kB/s | 125 kB 00:00 2024-09-18T07:22:44.249 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:22:44.251 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:22:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-18T07:22:44.252 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:22:44.252 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:22:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: ceph x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 6.6 k 2024-09-18T07:22:44.252 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-18T07:22:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 2.4 M 2024-09-18T07:22:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 1.5 M 2024-09-18T07:22:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 291 k 2024-09-18T07:22:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 4.8 M 2024-09-18T07:22:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 17 M 2024-09-18T07:22:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-18T07:22:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-18T07:22:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-18T07:22:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-18T07:22:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-18T07:22:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-18T07:22:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-18T07:22:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-18T07:22:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-18T07:22:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-18T07:22:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections 2024-09-18T07:22:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-18T07:22:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-18T07:22:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-18T07:22:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-18T07:22:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-18T07:22:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-18T07:22:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-18T07:22:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-18T07:22:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-18T07:22:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-18T07:22:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-18T07:22:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-18T07:22:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-18T07:22:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-18T07:22:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-18T07:22:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-18T07:22:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-18T07:22:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-18T07:22:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-18T07:22:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-18T07:22:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-18T07:22:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:44.255 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:22:44.255 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:22:44.255 INFO:teuthology.orchestra.run.smithi137.stdout:Install 37 Packages 2024-09-18T07:22:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:44.256 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 31 M 2024-09-18T07:22:44.256 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 109 M 2024-09-18T07:22:44.256 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:22:44.286 INFO:teuthology.orchestra.run.smithi203.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.4 MB/s | 215 kB 00:00 2024-09-18T07:22:44.353 INFO:teuthology.orchestra.run.smithi203.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-18T07:22:44.379 INFO:teuthology.orchestra.run.smithi203.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 867 kB/s | 188 kB 00:00 2024-09-18T07:22:44.404 INFO:teuthology.orchestra.run.smithi203.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 296 kB/s | 35 kB 00:00 2024-09-18T07:22:44.429 INFO:teuthology.orchestra.run.smithi203.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-18T07:22:44.455 INFO:teuthology.orchestra.run.smithi203.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 674 kB/s | 51 kB 00:00 2024-09-18T07:22:44.480 INFO:teuthology.orchestra.run.smithi203.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 550 kB/s | 42 kB 00:00 2024-09-18T07:22:44.539 INFO:teuthology.orchestra.run.smithi203.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 746 kB/s | 43 kB 00:00 2024-09-18T07:22:44.581 INFO:teuthology.orchestra.run.smithi203.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-09-18T07:22:44.606 INFO:teuthology.orchestra.run.smithi203.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-09-18T07:22:44.631 INFO:teuthology.orchestra.run.smithi203.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 426 kB/s | 11 kB 00:00 2024-09-18T07:22:44.657 INFO:teuthology.orchestra.run.smithi203.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 714 kB/s | 18 kB 00:00 2024-09-18T07:22:44.682 INFO:teuthology.orchestra.run.smithi203.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-09-18T07:22:44.707 INFO:teuthology.orchestra.run.smithi203.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-18T07:22:44.724 INFO:teuthology.orchestra.run.smithi137.stdout:(1/37): ceph-19.3.0-4983.ge7fb7b56.el9.x86_64.r 57 kB/s | 6.6 kB 00:00 2024-09-18T07:22:44.732 INFO:teuthology.orchestra.run.smithi203.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-18T07:22:44.758 INFO:teuthology.orchestra.run.smithi203.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 454 kB/s | 20 kB 00:00 2024-09-18T07:22:44.783 INFO:teuthology.orchestra.run.smithi203.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-09-18T07:22:44.808 INFO:teuthology.orchestra.run.smithi203.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-18T07:22:44.825 INFO:teuthology.orchestra.run.smithi203.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 1.0 MB/s | 16 kB 00:00 2024-09-18T07:22:44.851 INFO:teuthology.orchestra.run.smithi203.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-09-18T07:22:44.876 INFO:teuthology.orchestra.run.smithi203.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-18T07:22:44.893 INFO:teuthology.orchestra.run.smithi203.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 852 kB/s | 36 kB 00:00 2024-09-18T07:22:44.918 INFO:teuthology.orchestra.run.smithi203.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-09-18T07:22:44.925 INFO:teuthology.orchestra.run.smithi137.stdout:(2/37): ceph-mgr-19.3.0-4983.ge7fb7b56.el9.x86_ 4.8 MB/s | 1.5 MB 00:00 2024-09-18T07:22:44.944 INFO:teuthology.orchestra.run.smithi203.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 791 kB/s | 20 kB 00:00 2024-09-18T07:22:44.975 INFO:teuthology.orchestra.run.smithi137.stdout:(3/37): ceph-mds-19.3.0-4983.ge7fb7b56.el9.x86_ 6.5 MB/s | 2.4 MB 00:00 2024-09-18T07:22:44.977 INFO:teuthology.orchestra.run.smithi203.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-09-18T07:22:45.050 INFO:teuthology.orchestra.run.smithi137.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4983.ge7fb 3.8 MB/s | 291 kB 00:00 2024-09-18T07:22:45.061 INFO:teuthology.orchestra.run.smithi203.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 35 kB/s | 22 kB 00:00 2024-09-18T07:22:45.065 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:22:45.066 INFO:teuthology.orchestra.run.smithi203.stdout:Total 13 MB/s | 31 MB 00:02 2024-09-18T07:22:45.117 INFO:teuthology.orchestra.run.smithi137.stdout:(5/37): ceph-mon-19.3.0-4983.ge7fb7b56.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-18T07:22:45.219 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:22:45.260 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:22:45.260 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:22:45.284 INFO:teuthology.orchestra.run.smithi137.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-18T07:22:45.467 INFO:teuthology.orchestra.run.smithi137.stdout:(7/37): ceph-osd-19.3.0-4983.ge7fb7b56.el9.x86_ 32 MB/s | 17 MB 00:00 2024-09-18T07:22:45.493 INFO:teuthology.orchestra.run.smithi137.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 511 kB/s | 106 kB 00:00 2024-09-18T07:22:45.526 INFO:teuthology.orchestra.run.smithi137.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 3.0 MB/s | 1.2 MB 00:00 2024-09-18T07:22:45.551 INFO:teuthology.orchestra.run.smithi137.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-09-18T07:22:45.585 INFO:teuthology.orchestra.run.smithi137.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-18T07:22:45.669 INFO:teuthology.orchestra.run.smithi137.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-09-18T07:22:45.694 INFO:teuthology.orchestra.run.smithi137.stdout:(13/37): python3-pycparser-2.20-6.el9.noarch.rp 598 kB/s | 135 kB 00:00 2024-09-18T07:22:45.719 INFO:teuthology.orchestra.run.smithi137.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-18T07:22:45.733 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:22:45.733 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:22:45.744 INFO:teuthology.orchestra.run.smithi137.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 461 kB/s | 35 kB 00:00 2024-09-18T07:22:45.778 INFO:teuthology.orchestra.run.smithi137.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 922 kB/s | 77 kB 00:00 2024-09-18T07:22:45.803 INFO:teuthology.orchestra.run.smithi137.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 609 kB/s | 51 kB 00:00 2024-09-18T07:22:45.828 INFO:teuthology.orchestra.run.smithi137.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 497 kB/s | 42 kB 00:00 2024-09-18T07:22:45.879 INFO:teuthology.orchestra.run.smithi137.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-09-18T07:22:45.904 INFO:teuthology.orchestra.run.smithi137.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 578 kB/s | 43 kB 00:00 2024-09-18T07:22:45.929 INFO:teuthology.orchestra.run.smithi137.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-09-18T07:22:45.954 INFO:teuthology.orchestra.run.smithi137.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 430 kB/s | 11 kB 00:00 2024-09-18T07:22:45.988 INFO:teuthology.orchestra.run.smithi137.stdout:(23/37): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2024-09-18T07:22:46.013 INFO:teuthology.orchestra.run.smithi137.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 301 kB/s | 18 kB 00:00 2024-09-18T07:22:46.038 INFO:teuthology.orchestra.run.smithi137.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-09-18T07:22:46.064 INFO:teuthology.orchestra.run.smithi137.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-18T07:22:46.089 INFO:teuthology.orchestra.run.smithi137.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-18T07:22:46.114 INFO:teuthology.orchestra.run.smithi137.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-18T07:22:46.140 INFO:teuthology.orchestra.run.smithi137.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-18T07:22:46.165 INFO:teuthology.orchestra.run.smithi137.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 648 kB/s | 16 kB 00:00 2024-09-18T07:22:46.190 INFO:teuthology.orchestra.run.smithi137.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 3.9 MB/s | 272 kB 00:00 2024-09-18T07:22:46.216 INFO:teuthology.orchestra.run.smithi137.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-18T07:22:46.241 INFO:teuthology.orchestra.run.smithi137.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-18T07:22:46.266 INFO:teuthology.orchestra.run.smithi137.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-18T07:22:46.283 INFO:teuthology.orchestra.run.smithi137.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-18T07:22:46.334 INFO:teuthology.orchestra.run.smithi137.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-18T07:22:46.500 INFO:teuthology.orchestra.run.smithi137.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 31 kB/s | 22 kB 00:00 2024-09-18T07:22:46.506 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:22:46.506 INFO:teuthology.orchestra.run.smithi137.stdout:Total 14 MB/s | 31 MB 00:02 2024-09-18T07:22:46.660 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:22:46.701 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:22:46.702 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:22:47.043 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:22:47.105 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-18T07:22:47.155 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-18T07:22:47.161 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:22:47.161 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:22:47.182 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-18T07:22:47.253 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-18T07:22:47.290 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-18T07:22:47.323 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-18T07:22:47.415 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-18T07:22:47.502 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-18T07:22:47.543 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-18T07:22:47.586 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-18T07:22:47.616 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-18T07:22:47.651 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-18T07:22:47.692 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-18T07:22:47.748 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-18T07:22:47.793 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-18T07:22:47.827 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-18T07:22:47.876 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-18T07:22:47.908 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-18T07:22:47.981 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-18T07:22:48.120 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-18T07:22:48.208 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-18T07:22:48.270 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-18T07:22:48.308 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-18T07:22:48.357 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-18T07:22:48.413 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-18T07:22:48.441 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:22:48.495 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-18T07:22:48.537 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-18T07:22:48.572 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-18T07:22:48.601 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-18T07:22:48.643 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-18T07:22:48.655 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-18T07:22:48.688 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-18T07:22:48.729 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-18T07:22:48.732 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-18T07:22:48.819 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-18T07:22:48.822 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-18T07:22:48.871 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-18T07:22:48.899 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-18T07:22:48.948 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-18T07:22:48.983 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-18T07:22:49.000 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-18T07:22:49.013 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-18T07:22:49.047 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-18T07:22:49.081 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-18T07:22:49.121 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-18T07:22:49.160 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4983.ge7fb7b56.el9. 32/37 2024-09-18T07:22:49.164 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-18T07:22:49.181 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 33/37 2024-09-18T07:22:49.199 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-18T07:22:49.208 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 33/37 2024-09-18T07:22:49.208 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:22:49.209 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-18T07:22:49.209 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T07:22:49.209 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T07:22:49.209 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:49.248 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-18T07:22:49.280 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-18T07:22:49.353 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-18T07:22:49.491 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-18T07:22:49.579 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-18T07:22:49.633 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-18T07:22:49.679 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-18T07:22:49.727 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-18T07:22:49.784 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-18T07:22:49.980 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-18T07:22:50.024 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-18T07:22:50.103 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-18T07:22:50.191 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-18T07:22:50.224 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-18T07:22:50.352 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-18T07:22:50.513 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4983.ge7fb7b56.el9. 32/37 2024-09-18T07:22:50.543 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 33/37 2024-09-18T07:22:50.569 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 33/37 2024-09-18T07:22:50.569 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:22:50.569 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-18T07:22:50.569 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T07:22:50.569 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-18T07:22:50.569 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:50.674 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 34/37 2024-09-18T07:22:50.701 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 34/37 2024-09-18T07:22:50.702 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:22:50.702 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-18T07:22:50.702 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T07:22:50.702 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T07:22:50.702 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:51.158 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:19.3.0-4983.ge7fb7b56.el9.x86_64 35/37 2024-09-18T07:22:51.182 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:19.3.0-4983.ge7fb7b56.el9.x86_64 35/37 2024-09-18T07:22:51.182 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:22:51.182 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-18T07:22:51.182 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T07:22:51.182 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T07:22:51.182 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:51.409 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:19.3.0-4983.ge7fb7b56.el9.x86_64 36/37 2024-09-18T07:22:51.433 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:19.3.0-4983.ge7fb7b56.el9.x86_64 36/37 2024-09-18T07:22:51.433 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:22:51.433 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-18T07:22:51.433 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T07:22:51.433 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T07:22:51.433 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:51.497 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:19.3.0-4983.ge7fb7b56.el9.x86_64 37/37 2024-09-18T07:22:52.027 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-osd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 34/37 2024-09-18T07:22:52.055 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-osd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 34/37 2024-09-18T07:22:52.055 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:22:52.055 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-18T07:22:52.055 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T07:22:52.055 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-18T07:22:52.055 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:52.502 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mon-2:19.3.0-4983.ge7fb7b56.el9.x86_64 35/37 2024-09-18T07:22:52.526 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mon-2:19.3.0-4983.ge7fb7b56.el9.x86_64 35/37 2024-09-18T07:22:52.526 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:22:52.526 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-18T07:22:52.527 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T07:22:52.527 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-18T07:22:52.527 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:52.753 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mds-2:19.3.0-4983.ge7fb7b56.el9.x86_64 36/37 2024-09-18T07:22:52.777 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mds-2:19.3.0-4983.ge7fb7b56.el9.x86_64 36/37 2024-09-18T07:22:52.777 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:22:52.777 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-18T07:22:52.777 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T07:22:52.777 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-18T07:22:52.777 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:52.851 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-2:19.3.0-4983.ge7fb7b56.el9.x86_64 37/37 2024-09-18T07:22:53.150 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-2:19.3.0-4983.ge7fb7b56.el9.x86_64 37/37 2024-09-18T07:22:53.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/37 2024-09-18T07:22:53.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/37 2024-09-18T07:22:53.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 3/37 2024-09-18T07:22:53.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:19.3.0-4983.ge7fb7b56.el9.x86_64 4/37 2024-09-18T07:22:53.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 5/37 2024-09-18T07:22:53.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4983.ge7fb7b56.el9. 6/37 2024-09-18T07:22:53.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-18T07:22:53.151 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-18T07:22:53.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-18T07:22:53.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-18T07:22:53.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-18T07:22:53.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-18T07:22:53.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-18T07:22:53.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-18T07:22:53.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-18T07:22:53.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-18T07:22:53.153 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-18T07:22:53.153 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-18T07:22:53.153 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-18T07:22:53.153 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-18T07:22:53.153 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-18T07:22:53.153 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-18T07:22:53.153 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-18T07:22:53.153 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-18T07:22:53.153 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-18T07:22:53.154 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-18T07:22:53.154 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-18T07:22:53.154 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-18T07:22:53.154 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-18T07:22:53.154 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-18T07:22:53.154 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-18T07:22:53.154 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-18T07:22:53.154 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-18T07:22:53.154 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-18T07:22:53.155 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-18T07:22:53.155 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-18T07:22:54.188 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-18T07:22:54.188 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:54.188 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:22:54.188 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:54.188 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:54.188 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:54.188 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:22:54.188 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:54.188 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:54.188 INFO:teuthology.orchestra.run.smithi203.stdout: lua-5.4.4-4.el9.x86_64 2024-09-18T07:22:54.189 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-18T07:22:54.189 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-18T07:22:54.189 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-18T07:22:54.189 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-18T07:22:54.189 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-18T07:22:54.189 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-18T07:22:54.189 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-18T07:22:54.189 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-18T07:22:54.189 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-18T07:22:54.189 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-18T07:22:54.189 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-18T07:22:54.189 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-18T07:22:54.189 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-18T07:22:54.190 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-18T07:22:54.190 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-18T07:22:54.190 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-18T07:22:54.190 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-18T07:22:54.190 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-18T07:22:54.190 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply-3.11-14.el9.noarch 2024-09-18T07:22:54.190 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-18T07:22:54.190 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-18T07:22:54.190 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-18T07:22:54.190 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-18T07:22:54.190 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-18T07:22:54.190 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-18T07:22:54.190 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-18T07:22:54.191 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-18T07:22:54.191 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-18T07:22:54.191 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-18T07:22:54.191 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-18T07:22:54.191 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:54.191 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:22:54.589 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-2:19.3.0-4983.ge7fb7b56.el9.x86_64 37/37 2024-09-18T07:22:54.590 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/37 2024-09-18T07:22:54.590 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mds-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/37 2024-09-18T07:22:54.590 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 3/37 2024-09-18T07:22:54.590 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mon-2:19.3.0-4983.ge7fb7b56.el9.x86_64 4/37 2024-09-18T07:22:54.590 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-osd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 5/37 2024-09-18T07:22:54.590 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4983.ge7fb7b56.el9. 6/37 2024-09-18T07:22:54.590 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-18T07:22:54.590 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-18T07:22:54.590 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-18T07:22:54.590 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-18T07:22:54.591 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-18T07:22:54.591 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-18T07:22:54.591 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-18T07:22:54.591 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-18T07:22:54.591 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-18T07:22:54.591 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-18T07:22:54.591 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-18T07:22:54.591 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-18T07:22:54.591 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-18T07:22:54.591 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-18T07:22:54.591 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-18T07:22:54.591 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-18T07:22:54.591 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-18T07:22:54.592 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-18T07:22:54.592 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-18T07:22:54.593 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-18T07:22:54.593 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-18T07:22:54.593 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-18T07:22:54.593 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-18T07:22:54.593 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-18T07:22:54.593 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-18T07:22:54.593 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-18T07:22:54.594 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-18T07:22:54.594 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-18T07:22:54.594 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-18T07:22:54.594 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-18T07:22:54.597 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-base 2024-09-18T07:22:55.194 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:51 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:22:55.285 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:22:55.327 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:22:55.328 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-18T07:22:55.328 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:22:55.407 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install cephadm 2024-09-18T07:22:55.641 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-18T07:22:55.642 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:55.642 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:22:55.642 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:55.642 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:55.642 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:55.642 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:22:55.642 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:55.642 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:55.642 INFO:teuthology.orchestra.run.smithi137.stdout: lua-5.4.4-4.el9.x86_64 2024-09-18T07:22:55.642 INFO:teuthology.orchestra.run.smithi137.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-18T07:22:55.643 INFO:teuthology.orchestra.run.smithi137.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-18T07:22:55.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-18T07:22:55.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-18T07:22:55.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-18T07:22:55.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-18T07:22:55.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-18T07:22:55.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-18T07:22:55.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-18T07:22:55.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-18T07:22:55.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-18T07:22:55.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-18T07:22:55.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-18T07:22:55.643 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-18T07:22:55.644 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-18T07:22:55.644 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-18T07:22:55.644 INFO:teuthology.orchestra.run.smithi137.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-18T07:22:55.644 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-18T07:22:55.644 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply-3.11-14.el9.noarch 2024-09-18T07:22:55.644 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-18T07:22:55.644 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-18T07:22:55.644 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-18T07:22:55.644 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-18T07:22:55.644 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-18T07:22:55.644 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-18T07:22:55.644 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-18T07:22:55.644 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-18T07:22:55.645 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-18T07:22:55.645 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-18T07:22:55.645 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-18T07:22:55.645 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:55.645 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:22:55.985 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:51 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:22:56.108 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-base 2024-09-18T07:22:56.116 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:22:56.117 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:22:56.117 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-18T07:22:56.117 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:22:56.117 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:22:56.117 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 784 k 2024-09-18T07:22:56.117 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:56.117 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:22:56.117 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:22:56.118 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-09-18T07:22:56.118 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:56.118 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 784 k 2024-09-18T07:22:56.118 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 791 k 2024-09-18T07:22:56.118 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:22:56.376 INFO:teuthology.orchestra.run.smithi203.stdout:cephadm-19.3.0-4983.ge7fb7b56.el9.noarch.rpm 3.0 MB/s | 784 kB 00:00 2024-09-18T07:22:56.377 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:22:56.377 INFO:teuthology.orchestra.run.smithi203.stdout:Total 3.0 MB/s | 784 kB 00:00 2024-09-18T07:22:56.377 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:22:56.382 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:22:56.382 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:22:56.388 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:22:56.389 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:22:56.462 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:22:56.705 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:51 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:22:56.796 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-base-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:22:56.822 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:22:56.837 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:22:56.838 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-18T07:22:56.838 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:22:56.911 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install cephadm 2024-09-18T07:22:56.914 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:22:57.292 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:22:57.495 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:52 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:22:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:22:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:57.550 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:22:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:22:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:57.550 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:22:57.626 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:22:57.627 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:22:57.627 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-18T07:22:57.627 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:22:57.627 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:22:57.628 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 784 k 2024-09-18T07:22:57.628 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:57.628 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:22:57.628 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:22:57.628 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-18T07:22:57.628 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:57.628 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 784 k 2024-09-18T07:22:57.628 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 791 k 2024-09-18T07:22:57.628 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:22:57.790 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-immutable-object-cache 2024-09-18T07:22:57.888 INFO:teuthology.orchestra.run.smithi137.stdout:cephadm-19.3.0-4983.ge7fb7b56.el9.noarch.rpm 3.0 MB/s | 784 kB 00:00 2024-09-18T07:22:57.889 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:22:57.889 INFO:teuthology.orchestra.run.smithi137.stdout:Total 2.9 MB/s | 784 kB 00:00 2024-09-18T07:22:57.889 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:22:57.894 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:22:57.894 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:22:57.900 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:22:57.900 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:22:57.971 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:22:58.382 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:54 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:22:58.450 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:22:58.519 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:22:58.520 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:22:58.520 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-09-18T07:22:58.520 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:22:58.520 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:22:58.520 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 145 k 2024-09-18T07:22:58.521 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:58.521 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:22:58.521 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:22:58.521 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-09-18T07:22:58.521 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:58.521 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 145 k 2024-09-18T07:22:58.521 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 432 k 2024-09-18T07:22:58.521 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:22:58.541 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:22:58.726 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-immutable-object-cache-19.3.0-4983.ge7fb7b 709 kB/s | 145 kB 00:00 2024-09-18T07:22:58.726 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:22:58.727 INFO:teuthology.orchestra.run.smithi203.stdout:Total 704 kB/s | 145 kB 00:00 2024-09-18T07:22:58.727 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:22:58.734 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:22:58.734 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:22:58.792 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:22:58.879 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:22:58.912 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:22:58.938 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:22:59.010 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4983.ge7fb7b56. 1/1 2024-09-18T07:22:59.035 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4983.ge7fb7b56. 1/1 2024-09-18T07:22:59.036 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:22:59.036 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-18T07:22:59.036 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:59.157 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:22:59.157 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:59.157 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:22:59.157 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:22:59.157 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:22:59.157 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:22:59.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-immutable-object-cache 2024-09-18T07:22:59.976 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:54 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:22:59.976 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4983.ge7fb7b56. 1/1 2024-09-18T07:22:59.976 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:59.976 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:22:59.976 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:22:59.977 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:22:59.977 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:00.113 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:00.113 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:00.113 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-09-18T07:23:00.114 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:00.114 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:23:00.114 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 145 k 2024-09-18T07:23:00.114 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:00.114 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:23:00.114 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:00.114 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-18T07:23:00.114 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:00.114 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 145 k 2024-09-18T07:23:00.114 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 432 k 2024-09-18T07:23:00.114 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:23:00.215 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr 2024-09-18T07:23:00.319 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-immutable-object-cache-19.3.0-4983.ge7fb7b 711 kB/s | 145 kB 00:00 2024-09-18T07:23:00.319 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:00.319 INFO:teuthology.orchestra.run.smithi137.stdout:Total 706 kB/s | 145 kB 00:00 2024-09-18T07:23:00.320 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:23:00.327 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:23:00.327 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:23:00.384 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:23:00.385 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:23:00.531 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:23:00.594 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4983.ge7fb7b56. 1/1 2024-09-18T07:23:00.621 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4983.ge7fb7b56. 1/1 2024-09-18T07:23:00.621 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:23:00.621 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-18T07:23:00.621 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:00.810 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:56 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:00.899 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:23:00.940 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:00.941 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-18T07:23:00.942 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:01.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-dashboard 2024-09-18T07:23:01.553 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4983.ge7fb7b56. 1/1 2024-09-18T07:23:01.553 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:01.553 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:23:01.553 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:23:01.554 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:01.554 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:01.609 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:57 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:01.742 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:01.743 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:01.743 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-18T07:23:01.743 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:01.743 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:23:01.743 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 14 M 2024-09-18T07:23:01.743 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-18T07:23:01.743 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-18T07:23:01.743 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 34 k 2024-09-18T07:23:01.744 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 16 k 2024-09-18T07:23:01.744 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-18T07:23:01.744 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-18T07:23:01.744 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-18T07:23:01.744 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-18T07:23:01.744 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-18T07:23:01.744 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-18T07:23:01.744 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-18T07:23:01.744 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-18T07:23:01.744 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:01.744 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:23:01.744 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:01.744 INFO:teuthology.orchestra.run.smithi203.stdout:Install 12 Packages 2024-09-18T07:23:01.745 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:01.745 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 19 M 2024-09-18T07:23:01.745 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 159 M 2024-09-18T07:23:01.745 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:23:01.773 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr 2024-09-18T07:23:02.222 INFO:teuthology.orchestra.run.smithi203.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4983.ge7 234 kB/s | 34 kB 00:00 2024-09-18T07:23:02.256 INFO:teuthology.orchestra.run.smithi203.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4983.ge7f 91 kB/s | 16 kB 00:00 2024-09-18T07:23:02.372 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:57 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:02.462 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-mgr-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:23:02.504 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:02.505 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-18T07:23:02.505 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:02.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-dashboard 2024-09-18T07:23:02.748 INFO:teuthology.orchestra.run.smithi203.stdout:(3/12): ceph-mgr-dashboard-19.3.0-4983.ge7fb7b5 21 MB/s | 14 MB 00:00 2024-09-18T07:23:02.898 INFO:teuthology.orchestra.run.smithi203.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.6 MB/s | 863 kB 00:00 2024-09-18T07:23:02.973 INFO:teuthology.orchestra.run.smithi203.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 7.2 MB/s | 548 kB 00:00 2024-09-18T07:23:02.999 INFO:teuthology.orchestra.run.smithi203.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 772 kB/s | 19 kB 00:00 2024-09-18T07:23:03.158 INFO:teuthology.orchestra.run.smithi203.stdout:(7/12): python3-protobuf-3.14.0-13.el9.noarch.r 299 kB/s | 269 kB 00:00 2024-09-18T07:23:03.162 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:58 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:03.216 INFO:teuthology.orchestra.run.smithi203.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-09-18T07:23:03.241 INFO:teuthology.orchestra.run.smithi203.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-18T07:23:03.295 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:03.296 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:03.296 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-18T07:23:03.296 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:03.296 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:23:03.296 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 14 M 2024-09-18T07:23:03.296 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-18T07:23:03.296 INFO:teuthology.orchestra.run.smithi137.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-18T07:23:03.296 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 34 k 2024-09-18T07:23:03.296 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 16 k 2024-09-18T07:23:03.297 INFO:teuthology.orchestra.run.smithi137.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-18T07:23:03.297 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-18T07:23:03.297 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-18T07:23:03.297 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-18T07:23:03.297 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-18T07:23:03.297 INFO:teuthology.orchestra.run.smithi137.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-18T07:23:03.297 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-18T07:23:03.297 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-18T07:23:03.297 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:03.297 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:23:03.298 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:03.298 INFO:teuthology.orchestra.run.smithi137.stdout:Install 12 Packages 2024-09-18T07:23:03.298 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:03.298 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 19 M 2024-09-18T07:23:03.298 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 159 M 2024-09-18T07:23:03.299 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:23:03.300 INFO:teuthology.orchestra.run.smithi203.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.8 MB/s | 2.0 MB 00:00 2024-09-18T07:23:03.334 INFO:teuthology.orchestra.run.smithi203.stdout:(11/12): protobuf-3.14.0-13.el9.x86_64.rpm 930 kB/s | 1.0 MB 00:01 2024-09-18T07:23:03.359 INFO:teuthology.orchestra.run.smithi203.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-09-18T07:23:03.363 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:03.363 INFO:teuthology.orchestra.run.smithi203.stdout:Total 12 MB/s | 19 MB 00:01 2024-09-18T07:23:03.411 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:23:03.430 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:23:03.430 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:23:03.704 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:23:03.705 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:23:04.048 INFO:teuthology.orchestra.run.smithi137.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4983.ge7f 112 kB/s | 16 kB 00:00 2024-09-18T07:23:04.072 INFO:teuthology.orchestra.run.smithi137.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4983.ge7 199 kB/s | 34 kB 00:00 2024-09-18T07:23:04.300 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:23:04.377 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-18T07:23:04.412 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-18T07:23:04.459 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-18T07:23:04.497 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-18T07:23:04.531 INFO:teuthology.orchestra.run.smithi137.stdout:(3/12): ceph-mgr-dashboard-19.3.0-4983.ge7fb7b5 22 MB/s | 14 MB 00:00 2024-09-18T07:23:04.600 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-18T07:23:04.706 INFO:teuthology.orchestra.run.smithi137.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 425 kB/s | 269 kB 00:00 2024-09-18T07:23:04.725 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-18T07:23:04.765 INFO:teuthology.orchestra.run.smithi137.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-18T07:23:04.790 INFO:teuthology.orchestra.run.smithi137.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 781 kB/s | 19 kB 00:00 2024-09-18T07:23:04.819 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-18T07:23:04.870 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-18T07:23:04.891 INFO:teuthology.orchestra.run.smithi137.stdout:(7/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.2 MB/s | 1.0 MB 00:00 2024-09-18T07:23:04.898 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-18T07:23:04.931 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4983.ge7fb7b56.el9 10/12 2024-09-18T07:23:04.949 INFO:teuthology.orchestra.run.smithi137.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.5 MB/s | 144 kB 00:00 2024-09-18T07:23:04.975 INFO:teuthology.orchestra.run.smithi137.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-18T07:23:05.017 INFO:teuthology.orchestra.run.smithi137.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.3 MB/s | 2.0 MB 00:00 2024-09-18T07:23:05.050 INFO:teuthology.orchestra.run.smithi137.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-09-18T07:23:05.500 INFO:teuthology.orchestra.run.smithi137.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 891 kB/s | 863 kB 00:00 2024-09-18T07:23:05.505 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:05.505 INFO:teuthology.orchestra.run.smithi137.stdout:Total 8.7 MB/s | 19 MB 00:02 2024-09-18T07:23:05.554 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:23:05.574 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:23:05.574 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:23:05.865 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:23:05.865 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:23:06.467 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:23:06.559 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-18T07:23:06.594 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-18T07:23:06.652 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-18T07:23:06.688 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-18T07:23:06.794 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-18T07:23:06.932 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-18T07:23:07.018 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-18T07:23:07.068 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-18T07:23:07.096 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-18T07:23:07.128 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4983.ge7fb7b56.el9 10/12 2024-09-18T07:23:08.147 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4983.ge7fb7b56.el 11/12 2024-09-18T07:23:08.202 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4983.ge7fb7b56.el9.noa 12/12 2024-09-18T07:23:10.223 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4983.ge7fb7b56.el9.noa 12/12 2024-09-18T07:23:10.223 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4983.ge7fb7b56.el 1/12 2024-09-18T07:23:10.223 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4983.ge7fb7b56.el9.noa 2/12 2024-09-18T07:23:10.224 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4983.ge7fb7b56.el9 3/12 2024-09-18T07:23:10.224 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-18T07:23:10.224 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-18T07:23:10.224 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-18T07:23:10.224 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-18T07:23:10.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-18T07:23:10.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-18T07:23:10.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-18T07:23:10.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-18T07:23:10.367 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4983.ge7fb7b56.el 11/12 2024-09-18T07:23:10.425 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4983.ge7fb7b56.el9.noa 12/12 2024-09-18T07:23:10.541 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-18T07:23:10.541 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:10.541 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:23:10.541 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-18T07:23:10.541 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:23:10.542 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:23:10.542 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:23:10.542 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-18T07:23:10.542 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-18T07:23:10.542 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-18T07:23:10.542 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-18T07:23:10.542 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-18T07:23:10.542 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-18T07:23:10.543 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-18T07:23:10.543 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-18T07:23:10.543 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:10.543 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:10.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-18T07:23:11.422 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:07 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:11.554 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:11.555 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-09-18T07:23:11.556 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-18T07:23:11.556 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-09-18T07:23:11.556 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:23:11.556 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 7.4 M 2024-09-18T07:23:11.556 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-18T07:23:11.556 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-18T07:23:11.556 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-18T07:23:11.556 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-18T07:23:11.556 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-18T07:23:11.557 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-18T07:23:11.557 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-18T07:23:11.557 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-18T07:23:11.557 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-18T07:23:11.557 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-18T07:23:11.557 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-18T07:23:11.557 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-18T07:23:11.557 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:11.557 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:23:11.558 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-09-18T07:23:11.558 INFO:teuthology.orchestra.run.smithi203.stdout:Install 12 Packages 2024-09-18T07:23:11.558 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:11.558 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 38 M 2024-09-18T07:23:11.558 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 212 M 2024-09-18T07:23:11.558 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:23:12.227 INFO:teuthology.orchestra.run.smithi203.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-18T07:23:12.286 INFO:teuthology.orchestra.run.smithi203.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.0 MB/s | 801 kB 00:00 2024-09-18T07:23:12.386 INFO:teuthology.orchestra.run.smithi203.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 206 kB/s | 33 kB 00:00 2024-09-18T07:23:12.404 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4983.ge7fb7b56.el9.noa 12/12 2024-09-18T07:23:12.407 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4983.ge7fb7b56.el 1/12 2024-09-18T07:23:12.408 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4983.ge7fb7b56.el9.noa 2/12 2024-09-18T07:23:12.408 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4983.ge7fb7b56.el9 3/12 2024-09-18T07:23:12.408 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-18T07:23:12.408 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-18T07:23:12.408 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-18T07:23:12.408 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-18T07:23:12.408 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-18T07:23:12.408 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-18T07:23:12.408 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-18T07:23:12.408 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-18T07:23:12.462 INFO:teuthology.orchestra.run.smithi203.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 242 kB/s | 18 kB 00:00 2024-09-18T07:23:12.553 INFO:teuthology.orchestra.run.smithi203.stdout:(5/12): ceph-mgr-diskprediction-local-19.3.0-49 16 MB/s | 7.4 MB 00:00 2024-09-18T07:23:12.579 INFO:teuthology.orchestra.run.smithi203.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 329 kB/s | 38 kB 00:00 2024-09-18T07:23:12.732 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-18T07:23:12.733 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:12.733 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:23:12.733 INFO:teuthology.orchestra.run.smithi137.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-18T07:23:12.733 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:23:12.733 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:23:12.733 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:23:12.733 INFO:teuthology.orchestra.run.smithi137.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-18T07:23:12.733 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-18T07:23:12.733 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-18T07:23:12.733 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-18T07:23:12.734 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-18T07:23:12.734 INFO:teuthology.orchestra.run.smithi137.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-18T07:23:12.734 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-18T07:23:12.734 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-18T07:23:12.734 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:12.734 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:12.737 INFO:teuthology.orchestra.run.smithi203.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-09-18T07:23:12.904 INFO:teuthology.orchestra.run.smithi203.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.8 MB/s | 3.0 MB 00:00 2024-09-18T07:23:12.995 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-18T07:23:13.013 INFO:teuthology.orchestra.run.smithi203.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 11 MB/s | 4.9 MB 00:00 2024-09-18T07:23:13.038 INFO:teuthology.orchestra.run.smithi203.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.8 MB/s | 250 kB 00:00 2024-09-18T07:23:13.105 INFO:teuthology.orchestra.run.smithi203.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 5.1 MB 00:00 2024-09-18T07:23:13.413 INFO:teuthology.orchestra.run.smithi203.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-09-18T07:23:13.414 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:13.415 INFO:teuthology.orchestra.run.smithi203.stdout:Total 21 MB/s | 38 MB 00:01 2024-09-18T07:23:13.604 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:08 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:13.746 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:13.747 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================= 2024-09-18T07:23:13.747 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-18T07:23:13.747 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================= 2024-09-18T07:23:13.747 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:23:13.747 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 7.4 M 2024-09-18T07:23:13.747 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-18T07:23:13.748 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-18T07:23:13.748 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-18T07:23:13.748 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-18T07:23:13.748 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-18T07:23:13.748 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-18T07:23:13.748 INFO:teuthology.orchestra.run.smithi137.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-18T07:23:13.748 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-18T07:23:13.748 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-18T07:23:13.748 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-18T07:23:13.748 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-18T07:23:13.749 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-18T07:23:13.749 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:13.749 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:23:13.749 INFO:teuthology.orchestra.run.smithi137.stdout:======================================================================================= 2024-09-18T07:23:13.749 INFO:teuthology.orchestra.run.smithi137.stdout:Install 12 Packages 2024-09-18T07:23:13.749 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:13.750 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 38 M 2024-09-18T07:23:13.750 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 212 M 2024-09-18T07:23:13.750 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:23:13.769 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:23:13.795 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:23:13.795 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:23:14.090 INFO:teuthology.orchestra.run.smithi137.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 2.0 MB/s | 191 kB 00:00 2024-09-18T07:23:14.124 INFO:teuthology.orchestra.run.smithi137.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 6.1 MB/s | 801 kB 00:00 2024-09-18T07:23:14.207 INFO:teuthology.orchestra.run.smithi137.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2024-09-18T07:23:14.231 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:23:14.231 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:23:14.258 INFO:teuthology.orchestra.run.smithi137.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-18T07:23:14.308 INFO:teuthology.orchestra.run.smithi137.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 770 kB/s | 38 kB 00:00 2024-09-18T07:23:14.450 INFO:teuthology.orchestra.run.smithi137.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-49 16 MB/s | 7.4 MB 00:00 2024-09-18T07:23:14.575 INFO:teuthology.orchestra.run.smithi137.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.6 MB/s | 3.0 MB 00:00 2024-09-18T07:23:14.684 INFO:teuthology.orchestra.run.smithi137.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-09-18T07:23:14.776 INFO:teuthology.orchestra.run.smithi137.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.7 MB/s | 250 kB 00:00 2024-09-18T07:23:14.851 INFO:teuthology.orchestra.run.smithi137.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 9.1 MB/s | 4.9 MB 00:00 2024-09-18T07:23:14.959 INFO:teuthology.orchestra.run.smithi137.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-09-18T07:23:14.995 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:23:15.068 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-18T07:23:15.111 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-18T07:23:15.214 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-18T07:23:15.254 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-18T07:23:15.443 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-18T07:23:15.470 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-18T07:23:15.556 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-18T07:23:15.560 INFO:teuthology.orchestra.run.smithi137.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-09-18T07:23:15.561 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:15.561 INFO:teuthology.orchestra.run.smithi137.stdout:Total 21 MB/s | 38 MB 00:01 2024-09-18T07:23:15.915 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:23:15.942 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:23:15.942 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:23:16.215 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-18T07:23:16.333 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-18T07:23:16.382 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:23:16.383 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:23:17.139 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:23:17.204 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-18T07:23:17.249 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-18T07:23:17.349 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-18T07:23:17.384 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-18T07:23:17.563 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T07:23:17.568 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-18T07:23:17.591 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-18T07:23:17.668 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-18T07:23:18.325 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-18T07:23:18.465 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-18T07:23:18.614 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-18T07:23:18.647 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4983.ge7fb7 12/12 2024-09-18T07:23:19.715 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T07:23:20.768 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-18T07:23:20.799 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4983.ge7fb7 12/12 2024-09-18T07:23:21.348 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4983.ge7fb7 12/12 2024-09-18T07:23:21.348 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4983.ge7fb7 1/12 2024-09-18T07:23:21.349 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-18T07:23:21.349 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-18T07:23:21.349 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-18T07:23:21.349 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-18T07:23:21.349 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-18T07:23:21.349 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-18T07:23:21.349 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-18T07:23:21.349 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-18T07:23:21.349 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T07:23:21.349 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-18T07:23:21.822 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-18T07:23:21.823 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:21.823 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:23:21.823 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:23:21.823 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-18T07:23:21.823 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-18T07:23:21.823 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-18T07:23:21.823 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-18T07:23:21.823 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-18T07:23:21.824 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-18T07:23:21.824 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-18T07:23:21.824 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-18T07:23:21.824 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-18T07:23:21.824 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-18T07:23:21.824 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-18T07:23:21.824 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:21.824 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:22.237 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-rook 2024-09-18T07:23:22.826 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:18 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:22.961 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:22.962 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:22.962 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-18T07:23:22.962 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:22.962 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:23:22.962 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 49 k 2024-09-18T07:23:22.963 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-18T07:23:22.963 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-18T07:23:22.963 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-18T07:23:22.963 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-18T07:23:22.963 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-18T07:23:22.963 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-18T07:23:22.964 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-18T07:23:22.964 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-18T07:23:22.964 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-18T07:23:22.964 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-18T07:23:22.964 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-18T07:23:22.964 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-18T07:23:22.965 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-18T07:23:22.965 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:22.965 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:23:22.965 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:22.965 INFO:teuthology.orchestra.run.smithi203.stdout:Install 13 Packages 2024-09-18T07:23:22.965 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:22.966 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 2.2 M 2024-09-18T07:23:22.966 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 26 M 2024-09-18T07:23:22.967 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:23:23.342 INFO:teuthology.orchestra.run.smithi203.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 402 kB/s | 26 kB 00:00 2024-09-18T07:23:23.368 INFO:teuthology.orchestra.run.smithi203.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 215 kB/s | 19 kB 00:00 2024-09-18T07:23:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4983.ge7fb7 12/12 2024-09-18T07:23:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4983.ge7fb7 1/12 2024-09-18T07:23:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-18T07:23:23.388 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-18T07:23:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-18T07:23:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-18T07:23:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-18T07:23:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-18T07:23:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-18T07:23:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-18T07:23:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-18T07:23:23.389 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-18T07:23:23.443 INFO:teuthology.orchestra.run.smithi203.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-09-18T07:23:23.468 INFO:teuthology.orchestra.run.smithi203.stdout:(4/13): ceph-mgr-rook-19.3.0-4983.ge7fb7b56.el9 259 kB/s | 49 kB 00:00 2024-09-18T07:23:23.493 INFO:teuthology.orchestra.run.smithi203.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-18T07:23:23.544 INFO:teuthology.orchestra.run.smithi203.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 647 kB/s | 32 kB 00:00 2024-09-18T07:23:23.569 INFO:teuthology.orchestra.run.smithi203.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 533 kB/s | 54 kB 00:00 2024-09-18T07:23:23.594 INFO:teuthology.orchestra.run.smithi203.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 280 kB/s | 14 kB 00:00 2024-09-18T07:23:23.628 INFO:teuthology.orchestra.run.smithi203.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-09-18T07:23:23.678 INFO:teuthology.orchestra.run.smithi203.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-18T07:23:23.703 INFO:teuthology.orchestra.run.smithi203.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 804 kB/s | 59 kB 00:00 2024-09-18T07:23:23.729 INFO:teuthology.orchestra.run.smithi203.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-18T07:23:23.820 INFO:teuthology.orchestra.run.smithi203.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.7 MB/s | 1.0 MB 00:00 2024-09-18T07:23:23.825 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:23.825 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.6 MB/s | 2.2 MB 00:00 2024-09-18T07:23:23.832 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-18T07:23:23.832 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:23.832 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:23:23.832 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:23:23.832 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-18T07:23:23.832 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-18T07:23:23.832 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-18T07:23:23.832 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-18T07:23:23.832 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-18T07:23:23.832 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-18T07:23:23.833 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-18T07:23:23.833 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-18T07:23:23.833 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-18T07:23:23.833 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-18T07:23:23.833 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-18T07:23:23.833 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:23.833 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:23.895 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:23:23.911 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:23:23.912 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:23:24.005 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:23:24.005 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:23:24.278 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-rook 2024-09-18T07:23:24.519 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:23:24.754 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-18T07:23:24.805 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-18T07:23:24.861 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-18T07:23:24.872 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:19 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:24.898 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-18T07:23:24.940 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-18T07:23:25.005 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:25.006 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:25.006 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-18T07:23:25.006 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:25.006 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:23:25.006 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 49 k 2024-09-18T07:23:25.006 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-18T07:23:25.006 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-18T07:23:25.006 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-18T07:23:25.006 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-18T07:23:25.006 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-18T07:23:25.006 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-18T07:23:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-18T07:23:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-18T07:23:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-18T07:23:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-18T07:23:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-18T07:23:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-18T07:23:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-18T07:23:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:25.007 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:23:25.007 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:25.007 INFO:teuthology.orchestra.run.smithi137.stdout:Install 13 Packages 2024-09-18T07:23:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:25.008 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 2.2 M 2024-09-18T07:23:25.008 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 26 M 2024-09-18T07:23:25.008 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:23:25.042 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-18T07:23:25.153 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-18T07:23:25.209 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-18T07:23:25.507 INFO:teuthology.orchestra.run.smithi137.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 225 kB/s | 26 kB 00:00 2024-09-18T07:23:25.524 INFO:teuthology.orchestra.run.smithi137.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 146 kB/s | 19 kB 00:00 2024-09-18T07:23:25.541 INFO:teuthology.orchestra.run.smithi137.stdout:(3/13): ceph-mgr-rook-19.3.0-4983.ge7fb7b56.el9 333 kB/s | 49 kB 00:00 2024-09-18T07:23:25.658 INFO:teuthology.orchestra.run.smithi137.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-09-18T07:23:25.675 INFO:teuthology.orchestra.run.smithi137.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-09-18T07:23:25.709 INFO:teuthology.orchestra.run.smithi137.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-18T07:23:25.742 INFO:teuthology.orchestra.run.smithi137.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 279 kB 00:00 2024-09-18T07:23:25.776 INFO:teuthology.orchestra.run.smithi137.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-09-18T07:23:25.809 INFO:teuthology.orchestra.run.smithi137.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 143 kB/s | 14 kB 00:00 2024-09-18T07:23:25.843 INFO:teuthology.orchestra.run.smithi137.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-09-18T07:23:25.876 INFO:teuthology.orchestra.run.smithi137.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-09-18T07:23:25.902 INFO:teuthology.orchestra.run.smithi137.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.4 MB/s | 220 kB 00:00 2024-09-18T07:23:25.926 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-18T07:23:25.952 INFO:teuthology.orchestra.run.smithi137.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-09-18T07:23:25.954 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:25.955 INFO:teuthology.orchestra.run.smithi137.stdout:Total 2.3 MB/s | 2.2 MB 00:00 2024-09-18T07:23:25.973 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-18T07:23:26.006 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-18T07:23:26.021 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:23:26.037 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:23:26.037 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:23:26.055 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-18T07:23:26.077 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:19.3.0-4983.ge7fb7b56.el9.noarch 13/13 2024-09-18T07:23:26.126 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:23:26.126 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:23:26.628 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:23:26.774 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4983.ge7fb7b56.el9.noarch 13/13 2024-09-18T07:23:26.774 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/13 2024-09-18T07:23:26.774 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-18T07:23:26.774 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-18T07:23:26.774 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-18T07:23:26.774 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-18T07:23:26.774 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-18T07:23:26.774 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-18T07:23:26.774 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-18T07:23:26.774 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-18T07:23:26.774 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-18T07:23:26.775 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-18T07:23:26.775 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-18T07:23:26.790 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-18T07:23:26.845 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-18T07:23:26.900 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-18T07:23:26.937 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-18T07:23:26.979 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-18T07:23:27.088 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-18T07:23:27.112 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-18T07:23:27.112 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:27.112 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:23:27.112 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:23:27.112 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-18T07:23:27.113 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-18T07:23:27.113 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-18T07:23:27.113 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-18T07:23:27.113 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-18T07:23:27.113 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-18T07:23:27.113 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-18T07:23:27.113 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-18T07:23:27.113 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-18T07:23:27.113 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-18T07:23:27.113 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-18T07:23:27.113 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-18T07:23:27.113 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:27.113 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:27.190 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-18T07:23:27.247 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-18T07:23:27.399 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-cephadm 2024-09-18T07:23:27.964 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-18T07:23:27.997 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:23 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:28.011 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-18T07:23:28.044 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-18T07:23:28.092 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-18T07:23:28.114 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-rook-2:19.3.0-4983.ge7fb7b56.el9.noarch 13/13 2024-09-18T07:23:28.131 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:28.132 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:28.132 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-18T07:23:28.132 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:28.133 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:23:28.133 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 154 k 2024-09-18T07:23:28.133 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-18T07:23:28.133 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-18T07:23:28.133 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-18T07:23:28.133 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-18T07:23:28.133 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-18T07:23:28.133 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-18T07:23:28.133 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:28.133 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:23:28.133 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:28.133 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2024-09-18T07:23:28.133 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:28.134 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 7.0 M 2024-09-18T07:23:28.134 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 33 M 2024-09-18T07:23:28.134 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:23:28.624 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-18T07:23:28.650 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4983.ge7fb7b56.e 644 kB/s | 154 kB 00:00 2024-09-18T07:23:28.708 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 992 kB/s | 58 kB 00:00 2024-09-18T07:23:28.742 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-18T07:23:28.775 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.5 MB/s | 548 kB 00:00 2024-09-18T07:23:28.920 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4983.ge7fb7b56.el9.noarch 13/13 2024-09-18T07:23:28.920 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/13 2024-09-18T07:23:28.920 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-18T07:23:28.920 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-18T07:23:28.920 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-18T07:23:28.920 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-18T07:23:28.920 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-18T07:23:28.920 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-18T07:23:28.920 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-18T07:23:28.921 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-18T07:23:28.921 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-18T07:23:28.921 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-18T07:23:28.921 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-18T07:23:28.959 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-09-18T07:23:28.962 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:28.962 INFO:teuthology.orchestra.run.smithi203.stdout:Total 8.5 MB/s | 7.0 MB 00:00 2024-09-18T07:23:29.033 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:23:29.044 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:23:29.044 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:23:29.145 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:23:29.146 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:23:29.245 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-18T07:23:29.245 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:29.245 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:23:29.245 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:23:29.245 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-18T07:23:29.246 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-18T07:23:29.246 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-18T07:23:29.246 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-18T07:23:29.246 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-18T07:23:29.246 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-18T07:23:29.246 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-18T07:23:29.246 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-18T07:23:29.246 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-18T07:23:29.246 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-18T07:23:29.246 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-18T07:23:29.246 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-18T07:23:29.246 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:29.247 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:29.477 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:23:29.509 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-cephadm 2024-09-18T07:23:29.658 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-18T07:23:29.703 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-18T07:23:30.092 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:25 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:30.226 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:30.226 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-18T07:23:30.227 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:30.227 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-18T07:23:30.227 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:30.227 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:23:30.227 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 154 k 2024-09-18T07:23:30.227 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-18T07:23:30.227 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-18T07:23:30.227 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-18T07:23:30.227 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-18T07:23:30.227 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-18T07:23:30.227 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-18T07:23:30.227 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:30.228 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:23:30.228 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:30.228 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2024-09-18T07:23:30.228 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:30.228 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 7.0 M 2024-09-18T07:23:30.228 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 33 M 2024-09-18T07:23:30.228 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:23:30.315 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-18T07:23:30.400 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-18T07:23:30.424 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 6/6 2024-09-18T07:23:30.770 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.9 MB/s | 248 kB 00:00 2024-09-18T07:23:30.837 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4983.ge7fb7b56.e 781 kB/s | 154 kB 00:00 2024-09-18T07:23:30.887 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-18T07:23:30.921 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-18T07:23:30.946 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.1 MB/s | 548 kB 00:00 2024-09-18T07:23:31.063 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-09-18T07:23:31.066 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:31.066 INFO:teuthology.orchestra.run.smithi137.stdout:Total 8.4 MB/s | 7.0 MB 00:00 2024-09-18T07:23:31.138 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:23:31.149 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:23:31.150 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:23:31.225 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 6/6 2024-09-18T07:23:31.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/6 2024-09-18T07:23:31.226 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-18T07:23:31.226 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-18T07:23:31.226 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-18T07:23:31.226 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-18T07:23:31.249 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:23:31.251 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:23:31.512 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-18T07:23:31.512 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:31.512 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:23:31.512 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:23:31.512 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-18T07:23:31.512 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-18T07:23:31.512 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-18T07:23:31.512 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-18T07:23:31.512 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-18T07:23:31.512 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:31.512 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:31.550 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:23:31.718 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-18T07:23:31.765 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-18T07:23:31.805 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-fuse 2024-09-18T07:23:32.289 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-18T07:23:32.376 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-18T07:23:32.389 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:28 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:32.444 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-18T07:23:32.468 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 6/6 2024-09-18T07:23:32.523 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:32.523 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:32.523 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-18T07:23:32.524 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:32.524 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:23:32.524 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 901 k 2024-09-18T07:23:32.524 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-09-18T07:23:32.524 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-18T07:23:32.524 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:32.524 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:23:32.524 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:32.524 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2024-09-18T07:23:32.524 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:32.524 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 980 k 2024-09-18T07:23:32.524 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 2.9 M 2024-09-18T07:23:32.524 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:23:32.788 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 549 kB/s | 80 kB 00:00 2024-09-18T07:23:32.930 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): ceph-fuse-19.3.0-4983.ge7fb7b56.el9.x86_ 3.1 MB/s | 901 kB 00:00 2024-09-18T07:23:32.930 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:32.930 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.4 MB/s | 980 kB 00:00 2024-09-18T07:23:32.942 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:23:32.965 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:23:32.966 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:23:33.048 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:23:33.048 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:23:33.238 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:23:33.302 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 6/6 2024-09-18T07:23:33.302 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/6 2024-09-18T07:23:33.302 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-18T07:23:33.302 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-18T07:23:33.302 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-18T07:23:33.303 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-18T07:23:33.374 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-18T07:23:33.437 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/2 2024-09-18T07:23:33.569 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-18T07:23:33.569 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:33.569 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:23:33.569 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:23:33.569 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-18T07:23:33.569 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-18T07:23:33.569 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-18T07:23:33.569 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-18T07:23:33.569 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-18T07:23:33.569 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:33.569 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:33.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-fuse 2024-09-18T07:23:34.110 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/2 2024-09-18T07:23:34.110 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/2 2024-09-18T07:23:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-18T07:23:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:34.405 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:23:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-18T07:23:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:34.405 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:34.493 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:29 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:34.624 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:34.624 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:34.625 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-18T07:23:34.625 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:34.625 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:23:34.625 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 901 k 2024-09-18T07:23:34.625 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-18T07:23:34.625 INFO:teuthology.orchestra.run.smithi137.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-18T07:23:34.625 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:34.625 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:23:34.625 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:34.625 INFO:teuthology.orchestra.run.smithi137.stdout:Install 2 Packages 2024-09-18T07:23:34.625 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:34.625 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 980 k 2024-09-18T07:23:34.626 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 2.9 M 2024-09-18T07:23:34.626 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:23:34.711 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-volume 2024-09-18T07:23:34.815 INFO:teuthology.orchestra.run.smithi137.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 568 kB/s | 80 kB 00:00 2024-09-18T07:23:34.940 INFO:teuthology.orchestra.run.smithi137.stdout:(2/2): ceph-fuse-19.3.0-4983.ge7fb7b56.el9.x86_ 3.3 MB/s | 901 kB 00:00 2024-09-18T07:23:34.940 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:34.940 INFO:teuthology.orchestra.run.smithi137.stdout:Total 3.0 MB/s | 980 kB 00:00 2024-09-18T07:23:34.952 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:23:34.974 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:23:34.974 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:23:35.055 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:23:35.055 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:23:35.243 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:23:35.303 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:31 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:35.384 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-18T07:23:35.437 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:35.438 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:35.438 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-09-18T07:23:35.438 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:35.438 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:23:35.438 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 285 k 2024-09-18T07:23:35.439 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:35.439 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:23:35.439 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:35.439 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-09-18T07:23:35.439 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:35.439 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 285 k 2024-09-18T07:23:35.439 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 1.3 M 2024-09-18T07:23:35.439 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:23:35.464 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/2 2024-09-18T07:23:35.671 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-volume-19.3.0-4983.ge7fb7b56.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-18T07:23:35.671 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:35.672 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-18T07:23:35.672 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:23:35.680 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:23:35.680 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:23:35.708 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:23:35.708 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:23:35.931 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:23:36.006 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-volume-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:23:36.023 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-volume-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:23:36.023 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:23:36.023 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-18T07:23:36.023 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:36.221 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2/2 2024-09-18T07:23:36.221 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/2 2024-09-18T07:23:36.484 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-18T07:23:36.485 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:36.485 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:23:36.485 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-18T07:23:36.485 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:36.485 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:36.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-volume 2024-09-18T07:23:37.012 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-volume-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:23:37.013 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:37.013 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:23:37.013 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:23:37.013 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:37.013 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:37.342 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados-devel 2024-09-18T07:23:37.382 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:32 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:37.517 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:37.518 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:37.518 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-18T07:23:37.518 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:37.518 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:23:37.519 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume noarch 2:19.3.0-4983.ge7fb7b56.el9 ceph-noarch 285 k 2024-09-18T07:23:37.519 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:37.519 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:23:37.519 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:37.519 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-18T07:23:37.519 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:37.519 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 285 k 2024-09-18T07:23:37.519 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 1.3 M 2024-09-18T07:23:37.519 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:23:37.752 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-volume-19.3.0-4983.ge7fb7b56.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-18T07:23:37.753 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:37.753 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-18T07:23:37.753 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:23:37.762 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:23:37.762 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:23:37.787 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:23:37.787 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:23:37.946 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:33 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:38.011 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:23:38.062 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-volume-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:23:38.079 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-volume-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:23:38.079 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:23:38.079 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-18T07:23:38.079 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:38.080 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:38.080 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:38.081 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-09-18T07:23:38.081 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:38.081 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:23:38.081 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 127 k 2024-09-18T07:23:38.081 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:38.081 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:23:38.081 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:38.081 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-09-18T07:23:38.081 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:38.082 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 127 k 2024-09-18T07:23:38.082 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 456 k 2024-09-18T07:23:38.082 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:23:38.292 INFO:teuthology.orchestra.run.smithi203.stdout:librados-devel-19.3.0-4983.ge7fb7b56.el9.x86_64 601 kB/s | 127 kB 00:00 2024-09-18T07:23:38.293 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:38.293 INFO:teuthology.orchestra.run.smithi203.stdout:Total 598 kB/s | 127 kB 00:00 2024-09-18T07:23:38.293 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:23:38.299 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:23:38.299 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:23:38.364 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:23:38.364 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:23:38.510 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:23:38.625 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:39.068 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:39.076 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-volume-2:19.3.0-4983.ge7fb7b56.el9.noarch 1/1 2024-09-18T07:23:39.077 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:39.077 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:23:39.077 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume-2:19.3.0-4983.ge7fb7b56.el9.noarch 2024-09-18T07:23:39.077 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:39.077 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:39.336 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:39.336 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:39.336 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:23:39.336 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:23:39.336 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:39.336 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:39.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados-devel 2024-09-18T07:23:39.628 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs2 2024-09-18T07:23:39.989 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:34 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:40.126 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:40.126 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:40.127 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-09-18T07:23:40.127 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:40.127 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:23:40.127 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 127 k 2024-09-18T07:23:40.127 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:40.127 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:23:40.127 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:40.127 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-18T07:23:40.127 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:40.127 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 127 k 2024-09-18T07:23:40.127 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 456 k 2024-09-18T07:23:40.127 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:23:40.224 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:36 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:40.314 INFO:teuthology.orchestra.run.smithi203.stdout:Package libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:23:40.333 INFO:teuthology.orchestra.run.smithi137.stdout:librados-devel-19.3.0-4983.ge7fb7b56.el9.x86_64 618 kB/s | 127 kB 00:00 2024-09-18T07:23:40.333 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:40.333 INFO:teuthology.orchestra.run.smithi137.stdout:Total 613 kB/s | 127 kB 00:00 2024-09-18T07:23:40.333 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:23:40.339 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:23:40.339 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:23:40.360 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:40.362 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-18T07:23:40.362 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:40.403 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:23:40.403 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:23:40.449 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs-devel 2024-09-18T07:23:40.560 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:23:40.663 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librados-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:41.023 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:37 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:41.113 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:41.156 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:41.157 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:41.157 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-09-18T07:23:41.157 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:41.157 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:23:41.158 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 32 k 2024-09-18T07:23:41.158 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:41.158 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:23:41.158 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:41.158 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-09-18T07:23:41.158 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:41.158 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 32 k 2024-09-18T07:23:41.158 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 142 k 2024-09-18T07:23:41.158 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:23:41.304 INFO:teuthology.orchestra.run.smithi203.stdout:libcephfs-devel-19.3.0-4983.ge7fb7b56.el9.x86_6 217 kB/s | 32 kB 00:00 2024-09-18T07:23:41.305 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:41.305 INFO:teuthology.orchestra.run.smithi203.stdout:Total 215 kB/s | 32 kB 00:00 2024-09-18T07:23:41.305 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:23:41.309 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:23:41.309 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:23:41.326 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:41.326 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:41.326 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:23:41.326 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:23:41.326 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:41.327 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:41.334 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:23:41.334 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:23:41.426 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:23:41.545 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:41.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs2 2024-09-18T07:23:41.912 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:42.152 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:42.152 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:42.152 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:23:42.152 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:23:42.152 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:42.152 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:42.215 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:37 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:42.305 INFO:teuthology.orchestra.run.smithi137.stdout:Package libcephfs2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:23:42.350 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:42.351 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-18T07:23:42.351 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:42.353 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados2 2024-09-18T07:23:42.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs-devel 2024-09-18T07:23:42.938 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:38 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:43.000 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:37 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:43.028 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:23:43.071 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:43.072 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-18T07:23:43.072 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:43.135 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:43.136 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:43.136 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-09-18T07:23:43.136 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:43.136 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:23:43.136 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 32 k 2024-09-18T07:23:43.136 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:43.136 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:23:43.137 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:43.137 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-18T07:23:43.137 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:43.137 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 32 k 2024-09-18T07:23:43.137 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 142 k 2024-09-18T07:23:43.137 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:23:43.161 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librbd1 2024-09-18T07:23:43.278 INFO:teuthology.orchestra.run.smithi137.stdout:libcephfs-devel-19.3.0-4983.ge7fb7b56.el9.x86_6 225 kB/s | 32 kB 00:00 2024-09-18T07:23:43.278 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:43.278 INFO:teuthology.orchestra.run.smithi137.stdout:Total 223 kB/s | 32 kB 00:00 2024-09-18T07:23:43.278 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:23:43.282 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:23:43.282 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:23:43.299 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:23:43.299 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:23:43.397 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:23:43.541 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:43.767 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:39 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:43.866 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:23:43.910 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:43.911 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-18T07:23:43.911 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:43.934 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:43.996 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rados 2024-09-18T07:23:44.165 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:44.165 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:44.165 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:23:44.165 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:23:44.165 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:44.165 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:44.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados2 2024-09-18T07:23:44.575 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:40 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:44.666 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rados-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:23:44.709 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:44.710 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-18T07:23:44.710 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:44.796 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rgw 2024-09-18T07:23:44.980 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:39 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:45.076 INFO:teuthology.orchestra.run.smithi137.stdout:Package librados2-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:23:45.121 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:45.123 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-18T07:23:45.123 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:45.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librbd1 2024-09-18T07:23:45.400 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:41 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:45.503 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:23:45.547 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:45.548 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-18T07:23:45.548 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:45.639 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-cephfs 2024-09-18T07:23:45.815 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:40 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:45.902 INFO:teuthology.orchestra.run.smithi137.stdout:Package librbd1-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:23:45.946 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:45.947 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-18T07:23:45.948 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:46.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rados 2024-09-18T07:23:46.225 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:42 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:46.313 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-cephfs-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:23:46.357 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:46.358 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-18T07:23:46.358 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:46.446 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rbd 2024-09-18T07:23:46.609 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:41 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:46.698 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rados-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:23:46.742 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:46.743 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-18T07:23:46.743 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:46.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rgw 2024-09-18T07:23:47.041 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:43 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:47.137 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:23:47.181 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:47.182 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-09-18T07:23:47.182 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:47.272 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-fuse 2024-09-18T07:23:47.384 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:42 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:47.473 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rgw-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:23:47.516 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:47.517 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-18T07:23:47.517 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:47.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-cephfs 2024-09-18T07:23:47.871 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:43 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:48.005 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:48.005 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:48.006 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-09-18T07:23:48.006 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:48.006 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:23:48.006 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 85 k 2024-09-18T07:23:48.006 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:48.006 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:23:48.006 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:48.006 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-09-18T07:23:48.006 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:48.006 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 85 k 2024-09-18T07:23:48.006 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 231 k 2024-09-18T07:23:48.006 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:23:48.162 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:43 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:48.216 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-fuse-19.3.0-4983.ge7fb7b56.el9.x86_64.rpm 409 kB/s | 85 kB 00:00 2024-09-18T07:23:48.217 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:48.217 INFO:teuthology.orchestra.run.smithi203.stdout:Total 406 kB/s | 85 kB 00:00 2024-09-18T07:23:48.217 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:23:48.223 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:23:48.223 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:23:48.252 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-cephfs-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:23:48.293 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:23:48.294 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:23:48.295 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:48.296 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-18T07:23:48.296 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:48.375 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rbd 2024-09-18T07:23:48.453 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:23:48.572 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:48.936 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:48.951 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:43 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:49.041 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 is already installed. 2024-09-18T07:23:49.084 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:49.086 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-18T07:23:49.086 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:49.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-fuse 2024-09-18T07:23:49.195 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:49.196 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:49.196 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:23:49.196 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:23:49.196 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:49.197 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:49.472 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-mirror 2024-09-18T07:23:49.742 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:44 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:49.877 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:49.878 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:49.878 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-09-18T07:23:49.878 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:49.878 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:23:49.878 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 85 k 2024-09-18T07:23:49.879 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:49.879 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:23:49.879 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:49.879 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-18T07:23:49.879 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:49.879 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 85 k 2024-09-18T07:23:49.879 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 231 k 2024-09-18T07:23:49.879 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:23:50.056 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-fuse-19.3.0-4983.ge7fb7b56.el9.x86_64.rpm 483 kB/s | 85 kB 00:00 2024-09-18T07:23:50.057 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:50.057 INFO:teuthology.orchestra.run.smithi137.stdout:Total 479 kB/s | 85 kB 00:00 2024-09-18T07:23:50.057 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:23:50.064 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:23:50.064 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:23:50.084 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:46 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:50.132 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:23:50.133 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:23:50.225 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:50.226 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:50.226 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-09-18T07:23:50.226 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:50.226 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:23:50.226 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 3.2 M 2024-09-18T07:23:50.227 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:50.227 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:23:50.227 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:50.227 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-09-18T07:23:50.227 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:50.227 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 3.2 M 2024-09-18T07:23:50.227 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 14 M 2024-09-18T07:23:50.227 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:23:50.294 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:23:50.395 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:50.561 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-mirror-19.3.0-4983.ge7fb7b56.el9.x86_64.rpm 9.6 MB/s | 3.2 MB 00:00 2024-09-18T07:23:50.562 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:50.562 INFO:teuthology.orchestra.run.smithi203.stdout:Total 9.5 MB/s | 3.2 MB 00:00 2024-09-18T07:23:50.562 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:23:50.571 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:23:50.572 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:23:50.657 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:23:50.657 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:23:50.806 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:51.101 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:51.101 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:51.101 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:23:51.101 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:23:51.101 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:51.101 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:51.159 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:23:51.228 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:51.257 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:51.257 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:23:51.257 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-18T07:23:51.257 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T07:23:51.258 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T07:23:51.258 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:51.352 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-mirror 2024-09-18T07:23:51.937 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:46 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:52.075 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:52.076 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:52.076 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-09-18T07:23:52.076 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:52.076 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:23:52.076 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 3.2 M 2024-09-18T07:23:52.076 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:52.076 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:23:52.076 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:52.076 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-18T07:23:52.076 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:52.076 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 3.2 M 2024-09-18T07:23:52.076 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 14 M 2024-09-18T07:23:52.077 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:23:52.350 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:52.350 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:52.350 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:23:52.350 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:23:52.350 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:52.350 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:52.418 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-mirror-19.3.0-4983.ge7fb7b56.el9.x86_64.rpm 9.3 MB/s | 3.2 MB 00:00 2024-09-18T07:23:52.418 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:52.419 INFO:teuthology.orchestra.run.smithi137.stdout:Total 9.3 MB/s | 3.2 MB 00:00 2024-09-18T07:23:52.419 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:23:52.428 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:23:52.428 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:23:52.510 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:23:52.510 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:23:52.573 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-nbd 2024-09-18T07:23:52.985 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:23:53.052 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-mirror-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:53.077 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:53.077 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-18T07:23:53.077 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-18T07:23:53.077 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T07:23:53.078 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-18T07:23:53.078 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:53.175 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:49 ago on Wed 18 Sep 2024 07:22:04 AM UTC. 2024-09-18T07:23:53.306 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-09-18T07:23:53.307 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:53.307 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-09-18T07:23:53.307 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:53.307 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-09-18T07:23:53.307 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 170 k 2024-09-18T07:23:53.307 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:53.307 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-09-18T07:23:53.307 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-09-18T07:23:53.307 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-09-18T07:23:53.308 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:53.308 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 170 k 2024-09-18T07:23:53.308 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 490 k 2024-09-18T07:23:53.308 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-09-18T07:23:53.523 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-nbd-19.3.0-4983.ge7fb7b56.el9.x86_64.rpm 794 kB/s | 170 kB 00:00 2024-09-18T07:23:53.523 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:53.523 INFO:teuthology.orchestra.run.smithi203.stdout:Total 789 kB/s | 170 kB 00:00 2024-09-18T07:23:53.524 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-09-18T07:23:53.531 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-09-18T07:23:53.531 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-09-18T07:23:53.595 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-09-18T07:23:53.595 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-09-18T07:23:53.758 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-09-18T07:23:53.854 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:54.148 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-mirror-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:54.149 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:54.149 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:23:54.149 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:23:54.149 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:54.149 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:54.251 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:54.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-nbd 2024-09-18T07:23:54.499 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:54.499 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:54.499 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-09-18T07:23:54.499 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:23:54.499 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:23:54.499 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-09-18T07:23:54.736 DEBUG:teuthology.parallel:result is None 2024-09-18T07:23:54.992 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:49 ago on Wed 18 Sep 2024 07:22:05 AM UTC. 2024-09-18T07:23:55.126 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-18T07:23:55.127 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:55.127 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-09-18T07:23:55.127 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:55.127 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-18T07:23:55.127 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd x86_64 2:19.3.0-4983.ge7fb7b56.el9 ceph 170 k 2024-09-18T07:23:55.127 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:55.128 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-18T07:23:55.128 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-18T07:23:55.128 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-18T07:23:55.128 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:55.128 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 170 k 2024-09-18T07:23:55.128 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 490 k 2024-09-18T07:23:55.128 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-18T07:23:55.341 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-nbd-19.3.0-4983.ge7fb7b56.el9.x86_64.rpm 812 kB/s | 170 kB 00:00 2024-09-18T07:23:55.342 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:23:55.342 INFO:teuthology.orchestra.run.smithi137.stdout:Total 805 kB/s | 170 kB 00:00 2024-09-18T07:23:55.342 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-18T07:23:55.347 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-18T07:23:55.347 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-18T07:23:55.411 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-18T07:23:55.411 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-18T07:23:55.575 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-18T07:23:55.686 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-nbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:56.116 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:56.398 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-nbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 1/1 2024-09-18T07:23:56.399 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:56.399 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-18T07:23:56.399 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd-2:19.3.0-4983.ge7fb7b56.el9.x86_64 2024-09-18T07:23:56.399 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:23:56.399 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-18T07:23:56.607 DEBUG:teuthology.parallel:result is None 2024-09-18T07:23:56.609 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:23:56.768 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-18T07:23:56.790 INFO:teuthology.orchestra.run.smithi137.stdout:19.3.0-4983.ge7fb7b56.el9 2024-09-18T07:23:56.791 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4983.ge7fb7b56.el9 2024-09-18T07:23:56.791 INFO:teuthology.task.install:The correct ceph version 19.3.0-4983.ge7fb7b56 is installed. 2024-09-18T07:23:56.793 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:23:56.946 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-18T07:23:56.974 INFO:teuthology.orchestra.run.smithi203.stdout:19.3.0-4983.ge7fb7b56.el9 2024-09-18T07:23:56.974 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4983.ge7fb7b56.el9 2024-09-18T07:23:56.974 INFO:teuthology.task.install:The correct ceph version 19.3.0-4983.ge7fb7b56 is installed. 2024-09-18T07:23:56.976 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-18T07:23:56.977 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-18T07:23:56.977 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T07:23:57.002 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-18T07:23:57.002 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T07:23:57.045 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-18T07:23:57.045 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-18T07:23:57.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T07:23:57.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T07:23:57.134 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-18T07:23:57.134 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T07:23:57.162 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T07:23:57.230 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-18T07:23:57.231 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-18T07:23:57.231 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T07:23:57.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T07:23:57.320 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-18T07:23:57.320 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T07:23:57.347 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T07:23:57.414 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-18T07:23:57.415 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-18T07:23:57.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T07:23:57.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T07:23:57.504 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-18T07:23:57.504 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T07:23:57.530 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T07:23:57.596 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-18T07:23:57.721 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'mon_bind_msgr2': False, 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e', 'cephadm_mode': 'root'} 2024-09-18T07:23:57.721 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:23:57.722 INFO:tasks.cephadm:Cluster fsid is f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:23:57.722 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-18T07:23:57.722 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.137:6789]', 'mon.c': '[v1:172.21.15.137:6790]', 'mon.b': '[v1:172.21.15.203:6789]'} 2024-09-18T07:23:57.722 INFO:tasks.cephadm:First mon is mon.a on smithi137 2024-09-18T07:23:57.722 INFO:tasks.cephadm:First mgr is y 2024-09-18T07:23:57.722 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-18T07:23:57.722 DEBUG:teuthology.orchestra.run.smithi137:> sudo hostname $(hostname -s) 2024-09-18T07:23:57.748 DEBUG:teuthology.orchestra.run.smithi203:> sudo hostname $(hostname -s) 2024-09-18T07:23:57.774 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-18T07:23:57.774 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:23:57.915 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83115/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-4983-ge7fb7b56', 'node_name': '172.21.2.6+braggi06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-4983.ge7fb7b56'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-17 17:21:57.325397', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-main-2024-09-17-1213', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-18T07:23:58.055 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-guits-main-2024-09-17-1213, sha1 e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:23:58.057 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/x86_64/flavors/default/cephadm 2024-09-18T07:23:58.057 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/x86_64/flavors/default/cephadm 2024-09-18T07:23:58.057 DEBUG:teuthology.orchestra.run.smithi137:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-18T07:23:58.606 INFO:teuthology.orchestra.run.smithi137.stdout:-rw-r--r--. 1 ubuntu ubuntu 804135 Sep 18 07:23 /home/ubuntu/cephtest/cephadm 2024-09-18T07:23:58.607 DEBUG:teuthology.orchestra.run.smithi203:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-17-1213/e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-18T07:23:59.129 INFO:teuthology.orchestra.run.smithi203.stdout:-rw-r--r--. 1 ubuntu ubuntu 804135 Sep 18 07:23 /home/ubuntu/cephtest/cephadm 2024-09-18T07:23:59.130 DEBUG:teuthology.orchestra.run.smithi137:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-18T07:23:59.149 DEBUG:teuthology.orchestra.run.smithi203:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-18T07:23:59.182 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e on all hosts... 2024-09-18T07:23:59.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e pull 2024-09-18T07:23:59.192 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e pull 2024-09-18T07:24:00.877 INFO:teuthology.orchestra.run.smithi137.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e... 2024-09-18T07:24:00.901 INFO:teuthology.orchestra.run.smithi203.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e... 2024-09-18T07:24:56.149 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:24:56.149 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 19.3.0-4983-ge7fb7b56 (e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e) squid (dev)", 2024-09-18T07:24:56.150 INFO:teuthology.orchestra.run.smithi137.stdout: "image_id": "b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a", 2024-09-18T07:24:56.150 INFO:teuthology.orchestra.run.smithi137.stdout: "repo_digests": [ 2024-09-18T07:24:56.150 INFO:teuthology.orchestra.run.smithi137.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9" 2024-09-18T07:24:56.150 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:24:56.150 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:25:01.815 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:25:01.816 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_version": "ceph version 19.3.0-4983-ge7fb7b56 (e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e) squid (dev)", 2024-09-18T07:25:01.816 INFO:teuthology.orchestra.run.smithi203.stdout: "image_id": "b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a", 2024-09-18T07:25:01.816 INFO:teuthology.orchestra.run.smithi203.stdout: "repo_digests": [ 2024-09-18T07:25:01.816 INFO:teuthology.orchestra.run.smithi203.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9" 2024-09-18T07:25:01.816 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:25:01.816 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:25:01.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph 2024-09-18T07:25:01.878 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph 2024-09-18T07:25:01.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /etc/ceph 2024-09-18T07:25:01.947 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /etc/ceph 2024-09-18T07:25:01.973 INFO:tasks.cephadm:Writing seed config... 2024-09-18T07:25:01.975 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-18T07:25:01.975 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-18T07:25:01.975 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-09-18T07:25:01.975 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-09-18T07:25:01.975 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-09-18T07:25:01.975 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-09-18T07:25:01.976 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-18T07:25:01.976 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-18T07:25:01.976 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-18T07:25:01.976 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-18T07:25:01.976 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-18T07:25:01.976 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-18T07:25:01.976 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-18T07:25:01.976 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-18T07:25:01.976 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-18T07:25:01.976 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-18T07:25:01.976 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-18T07:25:01.977 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-18T07:25:01.977 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-09-18T07:25:01.977 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-18T07:25:01.977 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-18T07:25:01.977 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-18T07:25:01.977 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-18T07:25:01.977 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-18T07:25:01.978 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-18T07:25:01.978 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-18T07:25:02.004 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = f7ef099c-758e-11ef-bceb-c7b262605968 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-09-18T07:25:02.005 DEBUG:teuthology.orchestra.run.smithi137:mon.a> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mon.a.service 2024-09-18T07:25:02.047 DEBUG:teuthology.orchestra.run.smithi137:mgr.y> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mgr.y.service 2024-09-18T07:25:02.090 INFO:tasks.cephadm:Bootstrapping... 2024-09-18T07:25:02.091 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e -v bootstrap --fsid f7ef099c-758e-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.137:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-18T07:25:02.296 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-18T07:25:02.296 INFO:teuthology.orchestra.run.smithi137.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e', '-v', 'bootstrap', '--fsid', 'f7ef099c-758e-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.137:6789]', '--skip-admin-label'] 2024-09-18T07:25:02.296 INFO:teuthology.orchestra.run.smithi137.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-18T07:25:02.297 INFO:teuthology.orchestra.run.smithi137.stdout:Verifying podman|docker is present... 2024-09-18T07:25:02.319 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stdout 5.2.2 2024-09-18T07:25:02.320 INFO:teuthology.orchestra.run.smithi137.stdout:Verifying lvm2 is present... 2024-09-18T07:25:02.320 INFO:teuthology.orchestra.run.smithi137.stdout:Verifying time synchronization is in place... 2024-09-18T07:25:02.327 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-18T07:25:02.327 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-18T07:25:02.333 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-18T07:25:02.334 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout inactive 2024-09-18T07:25:02.340 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout enabled 2024-09-18T07:25:02.346 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout active 2024-09-18T07:25:02.346 INFO:teuthology.orchestra.run.smithi137.stdout:Unit chronyd.service is enabled and running 2024-09-18T07:25:02.346 INFO:teuthology.orchestra.run.smithi137.stdout:Repeating the final host check... 2024-09-18T07:25:02.369 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stdout 5.2.2 2024-09-18T07:25:02.369 INFO:teuthology.orchestra.run.smithi137.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-18T07:25:02.369 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl is present 2024-09-18T07:25:02.369 INFO:teuthology.orchestra.run.smithi137.stdout:lvcreate is present 2024-09-18T07:25:02.375 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-18T07:25:02.376 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-18T07:25:02.381 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-18T07:25:02.382 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout inactive 2024-09-18T07:25:02.388 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout enabled 2024-09-18T07:25:02.394 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout active 2024-09-18T07:25:02.394 INFO:teuthology.orchestra.run.smithi137.stdout:Unit chronyd.service is enabled and running 2024-09-18T07:25:02.394 INFO:teuthology.orchestra.run.smithi137.stdout:Host looks OK 2024-09-18T07:25:02.394 INFO:teuthology.orchestra.run.smithi137.stdout:Cluster fsid: f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:25:02.395 INFO:teuthology.orchestra.run.smithi137.stdout:Acquiring lock 139917789448560 on /run/cephadm/f7ef099c-758e-11ef-bceb-c7b262605968.lock 2024-09-18T07:25:02.395 INFO:teuthology.orchestra.run.smithi137.stdout:Lock 139917789448560 acquired on /run/cephadm/f7ef099c-758e-11ef-bceb-c7b262605968.lock 2024-09-18T07:25:02.395 INFO:teuthology.orchestra.run.smithi137.stdout:Verifying IP 172.21.15.137 port 6789 ... 2024-09-18T07:25:02.395 INFO:teuthology.orchestra.run.smithi137.stdout:Base mon IP(s) is [172.21.15.137:6789], mon addrv is [v1:172.21.15.137:6789] 2024-09-18T07:25:02.398 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.137 metric 100 2024-09-18T07:25:02.398 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.137 metric 100 2024-09-18T07:25:02.401 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-18T07:25:02.401 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-18T07:25:02.401 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-18T07:25:02.405 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-18T07:25:02.405 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-18T07:25:02.405 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-18T07:25:02.405 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-18T07:25:02.405 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:8137/64 scope link noprefixroute 2024-09-18T07:25:02.405 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-18T07:25:02.406 INFO:teuthology.orchestra.run.smithi137.stdout:Mon IP `172.21.15.137` is in CIDR network `172.21.0.0/20` 2024-09-18T07:25:02.406 INFO:teuthology.orchestra.run.smithi137.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-09-18T07:25:02.406 INFO:teuthology.orchestra.run.smithi137.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-18T07:25:02.406 INFO:teuthology.orchestra.run.smithi137.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e... 2024-09-18T07:25:05.597 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stdout b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a 2024-09-18T07:25:05.597 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e... 2024-09-18T07:25:05.597 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stderr Getting image source signatures 2024-09-18T07:25:05.597 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-18T07:25:05.597 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stderr Copying blob sha256:dd5fdbcdb37f4ac53174664d5bf8ec5e6874a4a56dc34dfea3eb7684bf1294c4 2024-09-18T07:25:05.597 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stderr Copying config sha256:b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a 2024-09-18T07:25:05.597 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-18T07:25:11.294 INFO:teuthology.orchestra.run.smithi137.stdout:ceph: stdout ceph version 19.3.0-4983-ge7fb7b56 (e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e) squid (dev) 2024-09-18T07:25:11.294 INFO:teuthology.orchestra.run.smithi137.stdout:Ceph version: ceph version 19.3.0-4983-ge7fb7b56 (e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e) squid (dev) 2024-09-18T07:25:11.294 INFO:teuthology.orchestra.run.smithi137.stdout:Extracting ceph user uid/gid from container image... 2024-09-18T07:25:12.240 INFO:teuthology.orchestra.run.smithi137.stdout:stat: stdout 167 167 2024-09-18T07:25:12.240 INFO:teuthology.orchestra.run.smithi137.stdout:Creating initial keys... 2024-09-18T07:25:13.521 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph-authtool: stdout AQBYgOpmll6BIRAALKIpJTNsTMW0o/w5yic8Pg== 2024-09-18T07:25:14.469 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph-authtool: stdout AQBZgOpmf1Q7MhAAlDEnj+hCz/NOPqsazeZqjQ== 2024-09-18T07:25:15.474 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph-authtool: stdout AQBagOpmDjYpLhAAlmiJHlBftmmiCxO843V29w== 2024-09-18T07:25:15.475 INFO:teuthology.orchestra.run.smithi137.stdout:Creating initial monmap... 2024-09-18T07:25:17.234 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-18T07:25:17.235 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-18T07:25:17.235 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:25:17.235 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-18T07:25:17.235 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool for a [v1:172.21.15.137:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-18T07:25:17.235 INFO:teuthology.orchestra.run.smithi137.stdout:setting min_mon_release = quincy 2024-09-18T07:25:17.235 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/monmaptool: set fsid to f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:25:17.236 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-18T07:25:17.236 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:25:17.236 INFO:teuthology.orchestra.run.smithi137.stdout:Creating mon... 2024-09-18T07:25:18.215 INFO:teuthology.orchestra.run.smithi137.stdout:create mon.a on 2024-09-18T07:25:18.445 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-18T07:25:18.595 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-18T07:25:18.788 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-f7ef099c-758e-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-f7ef099c-758e-11ef-bceb-c7b262605968.target. 2024-09-18T07:25:18.788 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-f7ef099c-758e-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-f7ef099c-758e-11ef-bceb-c7b262605968.target. 2024-09-18T07:25:19.032 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mon.a 2024-09-18T07:25:19.032 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Failed to reset failed state of unit ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-18T07:25:19.192 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-f7ef099c-758e-11ef-bceb-c7b262605968.target.wants/ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-f7ef099c-758e-11ef-bceb-c7b262605968@.service. 2024-09-18T07:25:19.713 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-18T07:25:19.714 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout disabled 2024-09-18T07:25:19.720 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-18T07:25:19.721 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout inactive 2024-09-18T07:25:19.721 INFO:teuthology.orchestra.run.smithi137.stdout:firewalld.service is not enabled 2024-09-18T07:25:19.721 INFO:teuthology.orchestra.run.smithi137.stdout:Not possible to enable service . firewalld.service is not available 2024-09-18T07:25:19.721 INFO:teuthology.orchestra.run.smithi137.stdout:Waiting for mon to start... 2024-09-18T07:25:19.721 INFO:teuthology.orchestra.run.smithi137.stdout:Waiting for mon... 2024-09-18T07:25:19.843 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:19 smithi137 systemd[1]: Started Ceph mon.a for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:25:20.773 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:20 smithi137 ceph-mon[29041]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-18T07:25:20.773 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:20 smithi137 ceph-mon[29041]: monmap epoch 1 2024-09-18T07:25:20.773 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:20 smithi137 ceph-mon[29041]: fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:25:20.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:20 smithi137 ceph-mon[29041]: last_changed 2024-09-18T07:25:15.819392+0000 2024-09-18T07:25:20.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:20 smithi137 ceph-mon[29041]: created 2024-09-18T07:25:15.819392+0000 2024-09-18T07:25:20.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:20 smithi137 ceph-mon[29041]: min_mon_release 19 (squid) 2024-09-18T07:25:20.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:20 smithi137 ceph-mon[29041]: election_strategy: 1 2024-09-18T07:25:20.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:20 smithi137 ceph-mon[29041]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-18T07:25:20.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:20 smithi137 ceph-mon[29041]: fsmap 2024-09-18T07:25:20.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:20 smithi137 ceph-mon[29041]: osdmap e1: 0 total, 0 up, 0 in 2024-09-18T07:25:20.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:20 smithi137 ceph-mon[29041]: mgrmap e1: no daemons active 2024-09-18T07:25:20.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:20 smithi137 ceph-mon[29041]: from='client.? v1:172.21.15.137:0/3966297470' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-18T07:25:20.777 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout cluster: 2024-09-18T07:25:20.777 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout id: f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:25:20.777 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-18T07:25:20.777 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-18T07:25:20.777 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout services: 2024-09-18T07:25:20.777 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.550645s) 2024-09-18T07:25:20.778 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-18T07:25:20.778 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-18T07:25:20.778 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-18T07:25:20.778 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout data: 2024-09-18T07:25:20.778 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-18T07:25:20.778 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-18T07:25:20.778 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-18T07:25:20.778 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout pgs: 2024-09-18T07:25:20.778 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-18T07:25:20.779 INFO:teuthology.orchestra.run.smithi137.stdout:mon is available 2024-09-18T07:25:20.779 INFO:teuthology.orchestra.run.smithi137.stdout:Assimilating anything we can from ceph.conf... 2024-09-18T07:25:21.871 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:21 smithi137 ceph-mon[29041]: from='client.? v1:172.21.15.137:0/1965257564' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-18T07:25:21.872 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:21 smithi137 ceph-mon[29041]: from='client.? v1:172.21.15.137:0/1965257564' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-18T07:25:21.874 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-18T07:25:21.875 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout [global] 2024-09-18T07:25:21.876 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout fsid = f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:25:21.876 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-18T07:25:21.876 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.137:6789] 2024-09-18T07:25:21.876 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-18T07:25:21.876 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-18T07:25:21.876 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-18T07:25:21.876 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-18T07:25:21.876 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-18T07:25:21.876 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-18T07:25:21.876 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-18T07:25:21.877 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-18T07:25:21.877 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout [osd] 2024-09-18T07:25:21.877 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-18T07:25:21.877 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-18T07:25:21.877 INFO:teuthology.orchestra.run.smithi137.stdout:Generating new minimal ceph.conf... 2024-09-18T07:25:22.911 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:22 smithi137 ceph-mon[29041]: from='client.? v1:172.21.15.137:0/3267420986' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:25:22.914 INFO:teuthology.orchestra.run.smithi137.stdout:Restarting the monitor... 2024-09-18T07:25:23.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:22 smithi137 systemd[1]: Stopping Ceph mon.a for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:25:23.445 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:23 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a[29037]: 2024-09-18T07:25:23.186+0000 7f48cb94d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T07:25:23.445 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:23 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a[29037]: 2024-09-18T07:25:23.186+0000 7f48cb94d640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-18T07:25:23.763 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:23 smithi137 podman[29283]: 2024-09-18 07:25:23.44300452 +0000 UTC m=+0.369342588 container died c9061d8c1b775af5ad11dc79a6e294dbeffb5c4fc9e2bc69b6c51baaf036d8b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0) 2024-09-18T07:25:23.763 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:23 smithi137 podman[29283]: 2024-09-18 07:25:23.761546613 +0000 UTC m=+0.687884683 container cleanup c9061d8c1b775af5ad11dc79a6e294dbeffb5c4fc9e2bc69b6c51baaf036d8b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-18T07:25:24.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:23 smithi137 bash[29283]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a 2024-09-18T07:25:24.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:23 smithi137 podman[29295]: 2024-09-18 07:25:23.954537742 +0000 UTC m=+0.512796591 container remove c9061d8c1b775af5ad11dc79a6e294dbeffb5c4fc9e2bc69b6c51baaf036d8b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-09-18T07:25:24.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-18T07:25:24.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 systemd[1]: Stopped Ceph mon.a for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:25:24.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 systemd[1]: Starting Ceph mon.a for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:25:24.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 podman[29389]: 2024-09-18 07:25:24.287665303 +0000 UTC m=+0.015431266 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e 2024-09-18T07:25:24.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 podman[29389]: 2024-09-18 07:25:24.421916345 +0000 UTC m=+0.149682317 container create f2b0cbaedbeb412e09afc64593eb785cc0967881b3cb091f002c9e314a866268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T07:25:24.714 INFO:teuthology.orchestra.run.smithi137.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-18T07:25:24.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 podman[29389]: 2024-09-18 07:25:24.624230392 +0000 UTC m=+0.351996355 container init f2b0cbaedbeb412e09afc64593eb785cc0967881b3cb091f002c9e314a866268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-18T07:25:24.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 podman[29389]: 2024-09-18 07:25:24.627412319 +0000 UTC m=+0.355178282 container start f2b0cbaedbeb412e09afc64593eb785cc0967881b3cb091f002c9e314a866268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-18T07:25:24.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: set uid:gid to 167:167 (ceph:ceph) 2024-09-18T07:25:24.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: ceph version 19.3.0-4983-ge7fb7b56 (e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e) squid (dev), process ceph-mon, pid 2 2024-09-18T07:25:24.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: pidfile_write: ignore empty --pid-file 2024-09-18T07:25:24.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: load: jerasure load: lrc 2024-09-18T07:25:24.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: RocksDB version: 7.9.2 2024-09-18T07:25:24.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Git sha 0 2024-09-18T07:25:24.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Compile date 2024-09-17 15:59:30 2024-09-18T07:25:24.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: DB SUMMARY 2024-09-18T07:25:24.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: DB Session ID: WK5P9TOM7U46KZKON6M9 2024-09-18T07:25:24.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: CURRENT file: CURRENT 2024-09-18T07:25:24.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: IDENTITY file: IDENTITY 2024-09-18T07:25:24.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-18T07:25:24.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-18T07:25:24.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 92918 ; 2024-09-18T07:25:24.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.error_if_exists: 0 2024-09-18T07:25:24.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.create_if_missing: 0 2024-09-18T07:25:24.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.paranoid_checks: 1 2024-09-18T07:25:24.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-18T07:25:24.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-18T07:25:24.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-18T07:25:24.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.env: 0x557e8dc06120 2024-09-18T07:25:24.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.fs: PosixFileSystem 2024-09-18T07:25:24.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.info_log: 0x557e8f4436a0 2024-09-18T07:25:24.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_file_opening_threads: 16 2024-09-18T07:25:24.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.statistics: (nil) 2024-09-18T07:25:24.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.use_fsync: 0 2024-09-18T07:25:24.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_log_file_size: 0 2024-09-18T07:25:24.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-18T07:25:24.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-18T07:25:24.923 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.keep_log_file_num: 1000 2024-09-18T07:25:24.923 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.recycle_log_file_num: 0 2024-09-18T07:25:24.923 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.allow_fallocate: 1 2024-09-18T07:25:24.923 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.allow_mmap_reads: 0 2024-09-18T07:25:24.923 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.allow_mmap_writes: 0 2024-09-18T07:25:24.923 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.use_direct_reads: 0 2024-09-18T07:25:24.923 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-18T07:25:24.923 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.create_missing_column_families: 0 2024-09-18T07:25:24.923 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.db_log_dir: 2024-09-18T07:25:24.923 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.wal_dir: 2024-09-18T07:25:24.923 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-18T07:25:24.923 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-18T07:25:24.923 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-18T07:25:24.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-18T07:25:24.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-18T07:25:24.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-18T07:25:24.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.advise_random_on_open: 1 2024-09-18T07:25:24.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.db_write_buffer_size: 0 2024-09-18T07:25:24.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.write_buffer_manager: 0x557e8f447b80 2024-09-18T07:25:24.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-18T07:25:24.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-18T07:25:24.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-18T07:25:24.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.rate_limiter: (nil) 2024-09-18T07:25:24.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-18T07:25:24.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.wal_recovery_mode: 2 2024-09-18T07:25:24.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.enable_thread_tracking: 0 2024-09-18T07:25:24.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.enable_pipelined_write: 0 2024-09-18T07:25:24.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.unordered_write: 0 2024-09-18T07:25:24.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-18T07:25:24.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-18T07:25:24.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-18T07:25:24.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-18T07:25:24.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.row_cache: None 2024-09-18T07:25:24.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.wal_filter: None 2024-09-18T07:25:24.926 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-18T07:25:24.926 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.allow_ingest_behind: 0 2024-09-18T07:25:24.927 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.two_write_queues: 0 2024-09-18T07:25:24.927 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.manual_wal_flush: 0 2024-09-18T07:25:24.927 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.wal_compression: 0 2024-09-18T07:25:24.927 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.atomic_flush: 0 2024-09-18T07:25:24.927 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-18T07:25:24.927 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-18T07:25:24.927 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-18T07:25:24.927 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.log_readahead_size: 0 2024-09-18T07:25:24.927 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-18T07:25:24.927 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.best_efforts_recovery: 0 2024-09-18T07:25:24.927 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-18T07:25:24.928 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-18T07:25:24.928 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.allow_data_in_errors: 0 2024-09-18T07:25:24.928 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.db_host_id: __hostname__ 2024-09-18T07:25:24.928 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-18T07:25:24.928 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_background_jobs: 2 2024-09-18T07:25:24.928 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_background_compactions: -1 2024-09-18T07:25:24.928 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_subcompactions: 1 2024-09-18T07:25:24.928 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-18T07:25:24.928 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-18T07:25:24.928 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-18T07:25:24.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_total_wal_size: 0 2024-09-18T07:25:24.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-18T07:25:24.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-18T07:25:24.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-18T07:25:24.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-18T07:25:24.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_open_files: -1 2024-09-18T07:25:24.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.bytes_per_sync: 0 2024-09-18T07:25:24.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-18T07:25:24.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-18T07:25:24.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compaction_readahead_size: 0 2024-09-18T07:25:24.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_background_flushes: -1 2024-09-18T07:25:24.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Compression algorithms supported: 2024-09-18T07:25:24.930 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: kZSTD supported: 0 2024-09-18T07:25:24.930 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: kXpressCompression supported: 0 2024-09-18T07:25:24.930 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: kBZip2Compression supported: 0 2024-09-18T07:25:24.930 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-18T07:25:24.930 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: kLZ4Compression supported: 1 2024-09-18T07:25:24.930 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: kZlibCompression supported: 1 2024-09-18T07:25:24.930 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: kLZ4HCCompression supported: 1 2024-09-18T07:25:24.930 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: kSnappyCompression supported: 1 2024-09-18T07:25:24.930 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-18T07:25:24.930 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-18T07:25:24.930 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-18T07:25:24.930 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-18T07:25:24.930 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-18T07:25:24.931 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.merge_operator: 2024-09-18T07:25:24.931 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compaction_filter: None 2024-09-18T07:25:24.931 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compaction_filter_factory: None 2024-09-18T07:25:24.931 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.sst_partitioner_factory: None 2024-09-18T07:25:24.931 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-18T07:25:24.931 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-18T07:25:24.931 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557e8f4432c0) 2024-09-18T07:25:24.931 INFO:journalctl@ceph.mon.a.smithi137.stdout: cache_index_and_filter_blocks: 1 2024-09-18T07:25:24.931 INFO:journalctl@ceph.mon.a.smithi137.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-18T07:25:24.931 INFO:journalctl@ceph.mon.a.smithi137.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-18T07:25:24.931 INFO:journalctl@ceph.mon.a.smithi137.stdout: pin_top_level_index_and_filter: 1 2024-09-18T07:25:24.931 INFO:journalctl@ceph.mon.a.smithi137.stdout: index_type: 0 2024-09-18T07:25:24.931 INFO:journalctl@ceph.mon.a.smithi137.stdout: data_block_index_type: 0 2024-09-18T07:25:24.932 INFO:journalctl@ceph.mon.a.smithi137.stdout: index_shortening: 1 2024-09-18T07:25:24.932 INFO:journalctl@ceph.mon.a.smithi137.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-18T07:25:24.932 INFO:journalctl@ceph.mon.a.smithi137.stdout: checksum: 4 2024-09-18T07:25:24.932 INFO:journalctl@ceph.mon.a.smithi137.stdout: no_block_cache: 0 2024-09-18T07:25:24.932 INFO:journalctl@ceph.mon.a.smithi137.stdout: block_cache: 0x557e8f437610 2024-09-18T07:25:24.932 INFO:journalctl@ceph.mon.a.smithi137.stdout: block_cache_name: BinnedLRUCache 2024-09-18T07:25:24.932 INFO:journalctl@ceph.mon.a.smithi137.stdout: block_cache_options: 2024-09-18T07:25:24.932 INFO:journalctl@ceph.mon.a.smithi137.stdout: capacity : 536870912 2024-09-18T07:25:24.932 INFO:journalctl@ceph.mon.a.smithi137.stdout: num_shard_bits : 4 2024-09-18T07:25:24.932 INFO:journalctl@ceph.mon.a.smithi137.stdout: strict_capacity_limit : 0 2024-09-18T07:25:24.932 INFO:journalctl@ceph.mon.a.smithi137.stdout: high_pri_pool_ratio: 0.000 2024-09-18T07:25:24.932 INFO:journalctl@ceph.mon.a.smithi137.stdout: block_cache_compressed: (nil) 2024-09-18T07:25:24.932 INFO:journalctl@ceph.mon.a.smithi137.stdout: persistent_cache: (nil) 2024-09-18T07:25:24.933 INFO:journalctl@ceph.mon.a.smithi137.stdout: block_size: 4096 2024-09-18T07:25:24.933 INFO:journalctl@ceph.mon.a.smithi137.stdout: block_size_deviation: 10 2024-09-18T07:25:24.933 INFO:journalctl@ceph.mon.a.smithi137.stdout: block_restart_interval: 16 2024-09-18T07:25:24.933 INFO:journalctl@ceph.mon.a.smithi137.stdout: index_block_restart_interval: 1 2024-09-18T07:25:24.933 INFO:journalctl@ceph.mon.a.smithi137.stdout: metadata_block_size: 4096 2024-09-18T07:25:24.933 INFO:journalctl@ceph.mon.a.smithi137.stdout: partition_filters: 0 2024-09-18T07:25:24.933 INFO:journalctl@ceph.mon.a.smithi137.stdout: use_delta_encoding: 1 2024-09-18T07:25:24.933 INFO:journalctl@ceph.mon.a.smithi137.stdout: filter_policy: bloomfilter 2024-09-18T07:25:24.933 INFO:journalctl@ceph.mon.a.smithi137.stdout: whole_key_filtering: 1 2024-09-18T07:25:24.933 INFO:journalctl@ceph.mon.a.smithi137.stdout: verify_compression: 0 2024-09-18T07:25:24.933 INFO:journalctl@ceph.mon.a.smithi137.stdout: read_amp_bytes_per_bit: 0 2024-09-18T07:25:24.933 INFO:journalctl@ceph.mon.a.smithi137.stdout: format_version: 5 2024-09-18T07:25:24.933 INFO:journalctl@ceph.mon.a.smithi137.stdout: enable_index_compression: 1 2024-09-18T07:25:24.934 INFO:journalctl@ceph.mon.a.smithi137.stdout: block_align: 0 2024-09-18T07:25:24.934 INFO:journalctl@ceph.mon.a.smithi137.stdout: max_auto_readahead_size: 262144 2024-09-18T07:25:24.934 INFO:journalctl@ceph.mon.a.smithi137.stdout: prepopulate_block_cache: 0 2024-09-18T07:25:24.934 INFO:journalctl@ceph.mon.a.smithi137.stdout: initial_auto_readahead_size: 8192 2024-09-18T07:25:24.934 INFO:journalctl@ceph.mon.a.smithi137.stdout: num_file_reads_for_auto_readahead: 2 2024-09-18T07:25:24.934 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.write_buffer_size: 33554432 2024-09-18T07:25:24.934 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_write_buffer_number: 2 2024-09-18T07:25:24.934 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compression: NoCompression 2024-09-18T07:25:24.934 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.bottommost_compression: Disabled 2024-09-18T07:25:24.934 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.prefix_extractor: nullptr 2024-09-18T07:25:24.934 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-18T07:25:24.934 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.num_levels: 7 2024-09-18T07:25:24.935 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-18T07:25:24.935 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-18T07:25:24.935 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-18T07:25:24.935 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-18T07:25:24.935 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-18T07:25:24.935 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-18T07:25:24.935 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-18T07:25:24.935 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-18T07:25:24.935 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-18T07:25:24.935 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-18T07:25:24.935 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-18T07:25:24.935 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-18T07:25:24.935 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-18T07:25:24.935 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compression_opts.level: 32767 2024-09-18T07:25:24.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compression_opts.strategy: 0 2024-09-18T07:25:24.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-18T07:25:24.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-18T07:25:24.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-18T07:25:24.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-18T07:25:24.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compression_opts.enabled: false 2024-09-18T07:25:24.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-18T07:25:24.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-18T07:25:24.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-18T07:25:24.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-18T07:25:24.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.target_file_size_base: 67108864 2024-09-18T07:25:24.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-18T07:25:24.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-18T07:25:24.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-18T07:25:24.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-18T07:25:24.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-18T07:25:24.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-18T07:25:24.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-18T07:25:24.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-18T07:25:24.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-18T07:25:24.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-18T07:25:24.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-18T07:25:24.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-18T07:25:24.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-18T07:25:24.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-18T07:25:24.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.arena_block_size: 1048576 2024-09-18T07:25:24.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-18T07:25:24.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-18T07:25:24.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.disable_auto_compactions: 0 2024-09-18T07:25:24.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-18T07:25:24.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-18T07:25:24.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-18T07:25:24.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-18T07:25:24.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-18T07:25:24.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-18T07:25:24.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-18T07:25:24.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-18T07:25:24.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-18T07:25:24.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-18T07:25:24.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-18T07:25:24.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.inplace_update_support: 0 2024-09-18T07:25:24.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-18T07:25:24.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-18T07:25:24.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-18T07:25:24.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-18T07:25:24.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.bloom_locality: 0 2024-09-18T07:25:24.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.max_successive_merges: 0 2024-09-18T07:25:24.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-18T07:25:24.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.paranoid_file_checks: 0 2024-09-18T07:25:24.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.force_consistency_checks: 1 2024-09-18T07:25:24.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.report_bg_io_stats: 0 2024-09-18T07:25:24.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.ttl: 2592000 2024-09-18T07:25:24.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-18T07:25:24.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-18T07:25:24.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-18T07:25:24.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.enable_blob_files: false 2024-09-18T07:25:24.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.min_blob_size: 0 2024-09-18T07:25:24.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.blob_file_size: 268435456 2024-09-18T07:25:24.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-18T07:25:24.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-18T07:25:24.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-18T07:25:24.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-18T07:25:24.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-18T07:25:24.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.blob_file_starting_level: 0 2024-09-18T07:25:24.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-18T07:25:24.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-18T07:25:24.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-18T07:25:24.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9edc16a9-ab56-4d20-aba7-0453cc740a27 2024-09-18T07:25:24.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726644324653658, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-18T07:25:24.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-18T07:25:24.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726644324654546, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 88904, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 287, "table_properties": {"data_size": 86960, "index_size": 264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11809, "raw_average_key_size": 48, "raw_value_size": 80432, "raw_average_value_size": 326, "num_data_blocks": 11, "num_entries": 246, "num_filter_entries": 246, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726644324, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9edc16a9-ab56-4d20-aba7-0453cc740a27", "db_session_id": "WK5P9TOM7U46KZKON6M9", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-18T07:25:24.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726644324654645, "job": 1, "event": "recovery_finished"} 2024-09-18T07:25:24.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-18T07:25:24.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-18T07:25:24.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557e8f466e00 2024-09-18T07:25:24.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: DB pointer 0x557e8f568000 2024-09-18T07:25:24.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-18T07:25:24.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-18T07:25:24.942 INFO:journalctl@ceph.mon.a.smithi137.stdout: ** DB Stats ** 2024-09-18T07:25:24.942 INFO:journalctl@ceph.mon.a.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-18T07:25:24.942 INFO:journalctl@ceph.mon.a.smithi137.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-18T07:25:24.942 INFO:journalctl@ceph.mon.a.smithi137.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-18T07:25:24.942 INFO:journalctl@ceph.mon.a.smithi137.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-18T07:25:24.942 INFO:journalctl@ceph.mon.a.smithi137.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-18T07:25:24.942 INFO:journalctl@ceph.mon.a.smithi137.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-18T07:25:24.942 INFO:journalctl@ceph.mon.a.smithi137.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-18T07:25:24.942 INFO:journalctl@ceph.mon.a.smithi137.stdout: 2024-09-18T07:25:24.942 INFO:journalctl@ceph.mon.a.smithi137.stdout: ** Compaction Stats [default] ** 2024-09-18T07:25:24.942 INFO:journalctl@ceph.mon.a.smithi137.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-18T07:25:24.942 INFO:journalctl@ceph.mon.a.smithi137.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-18T07:25:24.942 INFO:journalctl@ceph.mon.a.smithi137.stdout: L0 2/0 88.65 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-18T07:25:24.943 INFO:journalctl@ceph.mon.a.smithi137.stdout: Sum 2/0 88.65 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-18T07:25:24.943 INFO:journalctl@ceph.mon.a.smithi137.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-18T07:25:24.943 INFO:journalctl@ceph.mon.a.smithi137.stdout: 2024-09-18T07:25:24.943 INFO:journalctl@ceph.mon.a.smithi137.stdout: ** Compaction Stats [default] ** 2024-09-18T07:25:24.943 INFO:journalctl@ceph.mon.a.smithi137.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-18T07:25:24.943 INFO:journalctl@ceph.mon.a.smithi137.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-18T07:25:24.943 INFO:journalctl@ceph.mon.a.smithi137.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 151.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-18T07:25:24.943 INFO:journalctl@ceph.mon.a.smithi137.stdout: 2024-09-18T07:25:24.943 INFO:journalctl@ceph.mon.a.smithi137.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-18T07:25:24.943 INFO:journalctl@ceph.mon.a.smithi137.stdout: 2024-09-18T07:25:24.943 INFO:journalctl@ceph.mon.a.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-18T07:25:24.943 INFO:journalctl@ceph.mon.a.smithi137.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-18T07:25:24.943 INFO:journalctl@ceph.mon.a.smithi137.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-18T07:25:24.944 INFO:journalctl@ceph.mon.a.smithi137.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-18T07:25:24.944 INFO:journalctl@ceph.mon.a.smithi137.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-18T07:25:24.944 INFO:journalctl@ceph.mon.a.smithi137.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-18T07:25:24.944 INFO:journalctl@ceph.mon.a.smithi137.stdout: Cumulative compaction: 0.00 GB write, 28.90 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-18T07:25:24.944 INFO:journalctl@ceph.mon.a.smithi137.stdout: Interval compaction: 0.00 GB write, 28.90 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-18T07:25:24.944 INFO:journalctl@ceph.mon.a.smithi137.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-18T07:25:24.944 INFO:journalctl@ceph.mon.a.smithi137.stdout: Block cache BinnedLRUCache@0x557e8f437610#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-18T07:25:24.944 INFO:journalctl@ceph.mon.a.smithi137.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-18T07:25:24.944 INFO:journalctl@ceph.mon.a.smithi137.stdout: 2024-09-18T07:25:24.944 INFO:journalctl@ceph.mon.a.smithi137.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-18T07:25:24.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: starting mon.a rank 0 at public addrs v1:172.21.15.137:6789/0 at bind addrs v1:172.21.15.137:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:25:24.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: mon.a@-1(???) e1 preinit fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:25:24.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: mon.a@-1(???).mds e1 new map 2024-09-18T07:25:24.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: mon.a@-1(???).mds e1 print_map 2024-09-18T07:25:24.945 INFO:journalctl@ceph.mon.a.smithi137.stdout: e1 2024-09-18T07:25:24.945 INFO:journalctl@ceph.mon.a.smithi137.stdout: btime 2024-09-18T07:25:19:662155+0000 2024-09-18T07:25:24.945 INFO:journalctl@ceph.mon.a.smithi137.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-18T07:25:24.945 INFO:journalctl@ceph.mon.a.smithi137.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-18T07:25:24.945 INFO:journalctl@ceph.mon.a.smithi137.stdout: legacy client fscid: -1 2024-09-18T07:25:24.945 INFO:journalctl@ceph.mon.a.smithi137.stdout: 2024-09-18T07:25:24.945 INFO:journalctl@ceph.mon.a.smithi137.stdout: No filesystems configured 2024-09-18T07:25:24.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-18T07:25:24.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-18T07:25:24.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-18T07:25:24.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-18T07:25:24.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-18T07:25:24.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 bash[29389]: f2b0cbaedbeb412e09afc64593eb785cc0967881b3cb091f002c9e314a866268 2024-09-18T07:25:24.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 systemd[1]: Started Ceph mon.a for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:25:24.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-18T07:25:24.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: monmap epoch 1 2024-09-18T07:25:24.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:25:24.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: last_changed 2024-09-18T07:25:15.819392+0000 2024-09-18T07:25:24.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: created 2024-09-18T07:25:15.819392+0000 2024-09-18T07:25:24.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: min_mon_release 19 (squid) 2024-09-18T07:25:24.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: election_strategy: 1 2024-09-18T07:25:24.947 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-18T07:25:24.947 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: fsmap 2024-09-18T07:25:24.947 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: osdmap e1: 0 total, 0 up, 0 in 2024-09-18T07:25:24.947 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:24 smithi137 ceph-mon[29408]: mgrmap e1: no daemons active 2024-09-18T07:25:25.811 INFO:teuthology.orchestra.run.smithi137.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-18T07:25:25.824 INFO:teuthology.orchestra.run.smithi137.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-18T07:25:25.824 INFO:teuthology.orchestra.run.smithi137.stdout:Creating mgr... 2024-09-18T07:25:25.825 INFO:teuthology.orchestra.run.smithi137.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-18T07:25:25.825 INFO:teuthology.orchestra.run.smithi137.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-18T07:25:26.061 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mgr.y 2024-09-18T07:25:26.061 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Failed to reset failed state of unit ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-18T07:25:26.211 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-f7ef099c-758e-11ef-bceb-c7b262605968.target.wants/ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-f7ef099c-758e-11ef-bceb-c7b262605968@.service. 2024-09-18T07:25:26.487 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:26 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2949200203' entity='client.admin' 2024-09-18T07:25:26.487 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:26 smithi137 podman[29620]: 2024-09-18 07:25:26.485215925 +0000 UTC m=+0.160437253 container create a576bfc627b59d86cef5ef079d5e240e4528fb684c8f4c9f22e9c0461f2ecd66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=) 2024-09-18T07:25:26.737 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-18T07:25:26.737 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout disabled 2024-09-18T07:25:26.744 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-18T07:25:26.745 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout inactive 2024-09-18T07:25:26.745 INFO:teuthology.orchestra.run.smithi137.stdout:firewalld.service is not enabled 2024-09-18T07:25:26.745 INFO:teuthology.orchestra.run.smithi137.stdout:Not possible to enable service . firewalld.service is not available 2024-09-18T07:25:26.753 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-18T07:25:26.753 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout disabled 2024-09-18T07:25:26.761 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-18T07:25:26.761 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout inactive 2024-09-18T07:25:26.761 INFO:teuthology.orchestra.run.smithi137.stdout:firewalld.service is not enabled 2024-09-18T07:25:26.761 INFO:teuthology.orchestra.run.smithi137.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-18T07:25:26.761 INFO:teuthology.orchestra.run.smithi137.stdout:Waiting for mgr to start... 2024-09-18T07:25:26.761 INFO:teuthology.orchestra.run.smithi137.stdout:Waiting for mgr... 2024-09-18T07:25:26.885 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:26 smithi137 podman[29620]: 2024-09-18 07:25:26.670783134 +0000 UTC m=+0.346004472 container init a576bfc627b59d86cef5ef079d5e240e4528fb684c8f4c9f22e9c0461f2ecd66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-18T07:25:26.885 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:26 smithi137 podman[29620]: 2024-09-18 07:25:26.673484762 +0000 UTC m=+0.348706091 container start a576bfc627b59d86cef5ef079d5e240e4528fb684c8f4c9f22e9c0461f2ecd66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-18T07:25:26.885 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:26 smithi137 bash[29620]: a576bfc627b59d86cef5ef079d5e240e4528fb684c8f4c9f22e9c0461f2ecd66 2024-09-18T07:25:26.885 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:26 smithi137 systemd[1]: Started Ceph mgr.y for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:25:27.728 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:27 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2988473997' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-18T07:25:27.967 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-18T07:25:27.967 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout { 2024-09-18T07:25:27.967 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "fsid": "f7ef099c-758e-11ef-bceb-c7b262605968", 2024-09-18T07:25:27.967 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "health": { 2024-09-18T07:25:27.967 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-18T07:25:27.967 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-18T07:25:27.967 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-18T07:25:27.968 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:27.968 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-18T07:25:27.968 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-18T07:25:27.968 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 0 2024-09-18T07:25:27.968 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:25:27.968 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-18T07:25:27.968 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "a" 2024-09-18T07:25:27.968 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:25:27.968 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-18T07:25:27.968 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-18T07:25:27.968 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:25:27.968 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-18T07:25:27.968 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-18T07:25:27.968 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:27.969 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-18T07:25:27.969 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:25:27.969 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-18T07:25:27.969 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-18T07:25:27.969 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-18T07:25:27.973 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-18T07:25:27.974 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-18T07:25:27.974 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-18T07:25:27.974 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:27.974 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-18T07:25:27.974 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-18T07:25:27.974 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-18T07:25:27.974 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-18T07:25:27.974 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-18T07:25:27.974 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-18T07:25:27.974 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-18T07:25:27.974 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-18T07:25:27.974 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-18T07:25:27.974 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:27.975 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-18T07:25:27.975 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:25:27.975 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "btime": "2024-09-18T07:25:19:662155+0000", 2024-09-18T07:25:27.975 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-18T07:25:27.975 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-18T07:25:27.975 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:27.975 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-18T07:25:27.975 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-18T07:25:27.975 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-18T07:25:27.975 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-18T07:25:27.975 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-18T07:25:27.975 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-18T07:25:27.975 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "restful" 2024-09-18T07:25:27.975 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:25:27.976 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-18T07:25:27.976 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:27.976 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-18T07:25:27.976 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:25:27.976 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "modified": "2024-09-18T07:25:19.662933+0000", 2024-09-18T07:25:27.976 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-18T07:25:27.976 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:27.976 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-18T07:25:27.976 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout } 2024-09-18T07:25:27.976 INFO:teuthology.orchestra.run.smithi137.stdout:mgr not available, waiting (1/15)... 2024-09-18T07:25:30.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:30 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1012698547' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-18T07:25:31.132 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-18T07:25:31.132 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout { 2024-09-18T07:25:31.132 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "fsid": "f7ef099c-758e-11ef-bceb-c7b262605968", 2024-09-18T07:25:31.133 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "health": { 2024-09-18T07:25:31.133 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-18T07:25:31.133 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-18T07:25:31.133 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-18T07:25:31.133 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:31.133 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-18T07:25:31.133 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-18T07:25:31.133 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 0 2024-09-18T07:25:31.133 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:25:31.133 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-18T07:25:31.133 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "a" 2024-09-18T07:25:31.133 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:25:31.134 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-18T07:25:31.134 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-18T07:25:31.134 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:25:31.134 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-18T07:25:31.134 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-18T07:25:31.134 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:31.134 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-18T07:25:31.134 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:25:31.134 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-18T07:25:31.134 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-18T07:25:31.134 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-18T07:25:31.139 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-18T07:25:31.139 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-18T07:25:31.139 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-18T07:25:31.139 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:31.139 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-18T07:25:31.139 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-18T07:25:31.139 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-18T07:25:31.140 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-18T07:25:31.140 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-18T07:25:31.140 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-18T07:25:31.140 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-18T07:25:31.140 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-18T07:25:31.140 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-18T07:25:31.140 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:31.140 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-18T07:25:31.140 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:25:31.140 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "btime": "2024-09-18T07:25:19:662155+0000", 2024-09-18T07:25:31.140 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-18T07:25:31.141 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-18T07:25:31.141 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:31.141 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-18T07:25:31.141 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-18T07:25:31.141 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-18T07:25:31.141 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-18T07:25:31.141 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-18T07:25:31.141 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-18T07:25:31.141 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "restful" 2024-09-18T07:25:31.141 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:25:31.141 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-18T07:25:31.141 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:31.142 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-18T07:25:31.142 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:25:31.142 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "modified": "2024-09-18T07:25:19.662933+0000", 2024-09-18T07:25:31.142 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-18T07:25:31.142 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:31.142 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-18T07:25:31.142 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout } 2024-09-18T07:25:31.142 INFO:teuthology.orchestra.run.smithi137.stdout:mgr not available, waiting (2/15)... 2024-09-18T07:25:32.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:31 smithi137 ceph-mon[29408]: Activating manager daemon y 2024-09-18T07:25:32.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:31 smithi137 ceph-mon[29408]: mgrmap e2: y(active, starting, since 0.00350221s) 2024-09-18T07:25:32.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:31 smithi137 ceph-mon[29408]: from='mgr.14100 v1:172.21.15.137:0/1961812847' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-18T07:25:32.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:31 smithi137 ceph-mon[29408]: from='mgr.14100 v1:172.21.15.137:0/1961812847' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-18T07:25:32.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:31 smithi137 ceph-mon[29408]: from='mgr.14100 v1:172.21.15.137:0/1961812847' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-18T07:25:32.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:31 smithi137 ceph-mon[29408]: from='mgr.14100 v1:172.21.15.137:0/1961812847' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-18T07:25:32.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:31 smithi137 ceph-mon[29408]: from='mgr.14100 v1:172.21.15.137:0/1961812847' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-18T07:25:32.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:31 smithi137 ceph-mon[29408]: from='mgr.14100 v1:172.21.15.137:0/1961812847' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-18T07:25:32.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:31 smithi137 ceph-mon[29408]: from='mgr.14100 v1:172.21.15.137:0/1961812847' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:25:32.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:31 smithi137 ceph-mon[29408]: from='mgr.14100 v1:172.21.15.137:0/1961812847' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-18T07:25:32.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:31 smithi137 ceph-mon[29408]: Manager daemon y is now available 2024-09-18T07:25:32.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:31 smithi137 ceph-mon[29408]: from='mgr.14100 v1:172.21.15.137:0/1961812847' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-18T07:25:32.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:31 smithi137 ceph-mon[29408]: from='mgr.14100 v1:172.21.15.137:0/1961812847' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-18T07:25:32.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:31 smithi137 ceph-mon[29408]: from='mgr.14100 v1:172.21.15.137:0/1961812847' entity='mgr.y' 2024-09-18T07:25:32.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:31 smithi137 ceph-mon[29408]: from='mgr.14100 v1:172.21.15.137:0/1961812847' entity='mgr.y' 2024-09-18T07:25:32.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:31 smithi137 ceph-mon[29408]: from='mgr.14100 v1:172.21.15.137:0/1961812847' entity='mgr.y' 2024-09-18T07:25:33.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:33 smithi137 ceph-mon[29408]: mgrmap e3: y(active, since 1.00749s) 2024-09-18T07:25:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-18T07:25:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout { 2024-09-18T07:25:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "fsid": "f7ef099c-758e-11ef-bceb-c7b262605968", 2024-09-18T07:25:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "health": { 2024-09-18T07:25:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-18T07:25:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-18T07:25:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-18T07:25:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-18T07:25:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-18T07:25:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 0 2024-09-18T07:25:34.314 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:25:34.314 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-18T07:25:34.314 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "a" 2024-09-18T07:25:34.314 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:25:34.314 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-18T07:25:34.314 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-18T07:25:34.314 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:25:34.314 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-18T07:25:34.314 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-18T07:25:34.314 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:34.314 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-18T07:25:34.314 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:25:34.315 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-18T07:25:34.315 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-18T07:25:34.315 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-18T07:25:34.319 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-18T07:25:34.319 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-18T07:25:34.319 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-18T07:25:34.319 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:34.319 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-18T07:25:34.319 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-18T07:25:34.319 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-18T07:25:34.319 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-18T07:25:34.319 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-18T07:25:34.319 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-18T07:25:34.320 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-18T07:25:34.320 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-18T07:25:34.320 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-18T07:25:34.320 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:34.320 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-18T07:25:34.320 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:25:34.320 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "btime": "2024-09-18T07:25:19:662155+0000", 2024-09-18T07:25:34.320 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-18T07:25:34.320 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-18T07:25:34.320 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:34.320 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-18T07:25:34.320 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-18T07:25:34.320 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-18T07:25:34.321 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-18T07:25:34.321 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-18T07:25:34.321 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-18T07:25:34.321 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "restful" 2024-09-18T07:25:34.321 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout ], 2024-09-18T07:25:34.321 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-18T07:25:34.321 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:34.321 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-18T07:25:34.321 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-18T07:25:34.321 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "modified": "2024-09-18T07:25:19.662933+0000", 2024-09-18T07:25:34.321 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-18T07:25:34.321 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-18T07:25:34.321 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-18T07:25:34.322 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout } 2024-09-18T07:25:34.322 INFO:teuthology.orchestra.run.smithi137.stdout:mgr is available 2024-09-18T07:25:34.890 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:34 smithi137 ceph-mon[29408]: mgrmap e4: y(active, since 2s) 2024-09-18T07:25:34.890 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:34 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3381379182' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-18T07:25:35.477 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-18T07:25:35.477 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout [global] 2024-09-18T07:25:35.477 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout fsid = f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:25:35.477 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-18T07:25:35.477 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.137:6789] 2024-09-18T07:25:35.477 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-18T07:25:35.477 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-18T07:25:35.477 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-18T07:25:35.478 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-18T07:25:35.478 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-18T07:25:35.478 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-18T07:25:35.478 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-18T07:25:35.478 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-18T07:25:35.478 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout [osd] 2024-09-18T07:25:35.478 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-18T07:25:35.478 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-18T07:25:35.478 INFO:teuthology.orchestra.run.smithi137.stdout:Enabling cephadm module... 2024-09-18T07:25:35.909 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:35 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2979660971' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-18T07:25:36.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:36 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4199608061' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-18T07:25:36.919 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:36 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ignoring --setuser ceph since I am not root 2024-09-18T07:25:36.919 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:36 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ignoring --setgroup ceph since I am not root 2024-09-18T07:25:37.891 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:37 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4199608061' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-18T07:25:37.891 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:37 smithi137 ceph-mon[29408]: mgrmap e5: y(active, since 5s) 2024-09-18T07:25:38.473 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout { 2024-09-18T07:25:38.473 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-18T07:25:38.473 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-18T07:25:38.473 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-18T07:25:38.474 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-18T07:25:38.474 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout } 2024-09-18T07:25:38.474 INFO:teuthology.orchestra.run.smithi137.stdout:Waiting for the mgr to restart... 2024-09-18T07:25:38.474 INFO:teuthology.orchestra.run.smithi137.stdout:Waiting for mgr epoch 5... 2024-09-18T07:25:38.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:38 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/889090047' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-18T07:25:42.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:41 smithi137 ceph-mon[29408]: Active manager daemon y restarted 2024-09-18T07:25:42.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:41 smithi137 ceph-mon[29408]: Activating manager daemon y 2024-09-18T07:25:42.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:41 smithi137 ceph-mon[29408]: osdmap e2: 0 total, 0 up, 0 in 2024-09-18T07:25:42.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:41 smithi137 ceph-mon[29408]: mgrmap e6: y(active, starting, since 0.00310265s) 2024-09-18T07:25:42.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:41 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:25:42.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:41 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-18T07:25:42.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:41 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-18T07:25:42.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:41 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-18T07:25:42.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:41 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-18T07:25:42.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:41 smithi137 ceph-mon[29408]: Manager daemon y is now available 2024-09-18T07:25:43.202 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout { 2024-09-18T07:25:43.203 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-18T07:25:43.203 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-18T07:25:43.203 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout } 2024-09-18T07:25:43.203 INFO:teuthology.orchestra.run.smithi137.stdout:mgr epoch 5 is available 2024-09-18T07:25:43.203 INFO:teuthology.orchestra.run.smithi137.stdout:Verifying orchestrator module is enabled... 2024-09-18T07:25:43.536 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:43 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' 2024-09-18T07:25:43.536 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:43 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' 2024-09-18T07:25:43.536 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:43 smithi137 ceph-mon[29408]: Found migration_current of "None". Setting to last migration. 2024-09-18T07:25:43.536 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:43 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' 2024-09-18T07:25:43.536 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:43 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' 2024-09-18T07:25:43.537 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:43 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:25:43.537 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:43 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:25:43.537 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:43 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-18T07:25:43.537 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:43 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-18T07:25:43.537 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:43 smithi137 ceph-mon[29408]: from='client.14122 v1:172.21.15.137:0/2078661900' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-18T07:25:43.537 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:43 smithi137 ceph-mon[29408]: mgrmap e7: y(active, since 1.00589s) 2024-09-18T07:25:43.537 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:43 smithi137 ceph-mon[29408]: from='client.14122 v1:172.21.15.137:0/2078661900' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-18T07:25:44.538 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3736499801' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "orchestrator"} : dispatch 2024-09-18T07:25:44.848 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: 2024-09-18T07:25:44.545+0000 7f12fe1de640 -1 log_channel(cephadm) log [ERR] : [18/Sep/2024:07:25:44] ENGINE Error in HTTPServer.serve 2024-09-18T07:25:44.848 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: Traceback (most recent call last): 2024-09-18T07:25:44.848 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-18T07:25:44.849 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: self._connections.run(self.expiration_interval) 2024-09-18T07:25:44.849 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-18T07:25:44.849 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: self._run(expiration_interval) 2024-09-18T07:25:44.849 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-18T07:25:44.849 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: new_conn = self._from_server_socket(self.server.socket) 2024-09-18T07:25:44.849 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-18T07:25:44.849 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-18T07:25:44.849 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-18T07:25:44.849 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: s = self.context.wrap_socket( 2024-09-18T07:25:44.849 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-18T07:25:44.849 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: return self.sslsocket_class._create( 2024-09-18T07:25:44.849 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-18T07:25:44.850 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: self.do_handshake() 2024-09-18T07:25:44.850 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-18T07:25:44.850 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: self._sslobj.do_handshake() 2024-09-18T07:25:44.850 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-18T07:25:44.850 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:44 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: 2024-09-18T07:25:44.850 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stderr module 'orchestrator' is already enabled (always-on) 2024-09-18T07:25:44.851 INFO:teuthology.orchestra.run.smithi137.stdout:Setting orchestrator backend to cephadm... 2024-09-18T07:25:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:45 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3736499801' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "orchestrator"}]': finished 2024-09-18T07:25:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:45 smithi137 ceph-mon[29408]: mgrmap e8: y(active, since 2s) 2024-09-18T07:25:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:45 smithi137 ceph-mon[29408]: [18/Sep/2024:07:25:44] ENGINE Bus STARTING 2024-09-18T07:25:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:45 smithi137 ceph-mon[29408]: [18/Sep/2024:07:25:44] ENGINE Serving on https://172.21.15.137:7150 2024-09-18T07:25:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:45 smithi137 ceph-mon[29408]: [18/Sep/2024:07:25:44] ENGINE Error in HTTPServer.serve 2024-09-18T07:25:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout: Traceback (most recent call last): 2024-09-18T07:25:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-18T07:25:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._connections.run(self.expiration_interval) 2024-09-18T07:25:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-18T07:25:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._run(expiration_interval) 2024-09-18T07:25:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-18T07:25:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-18T07:25:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-18T07:25:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-18T07:25:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-18T07:25:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout: s = self.context.wrap_socket( 2024-09-18T07:25:45.421 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-18T07:25:45.421 INFO:journalctl@ceph.mon.a.smithi137.stdout: return self.sslsocket_class._create( 2024-09-18T07:25:45.421 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-18T07:25:45.421 INFO:journalctl@ceph.mon.a.smithi137.stdout: self.do_handshake() 2024-09-18T07:25:45.421 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-18T07:25:45.421 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._sslobj.do_handshake() 2024-09-18T07:25:45.421 INFO:journalctl@ceph.mon.a.smithi137.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-18T07:25:45.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:45 smithi137 ceph-mon[29408]: [18/Sep/2024:07:25:44] ENGINE Serving on http://172.21.15.137:8765 2024-09-18T07:25:45.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:45 smithi137 ceph-mon[29408]: [18/Sep/2024:07:25:44] ENGINE Bus STARTED 2024-09-18T07:25:45.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:45 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:25:46.628 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:46 smithi137 ceph-mon[29408]: from='client.14132 v1:172.21.15.137:0/2888679108' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:25:46.628 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:46 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' 2024-09-18T07:25:46.628 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:46 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:25:47.180 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-18T07:25:47.180 INFO:teuthology.orchestra.run.smithi137.stdout:Generating ssh key... 2024-09-18T07:25:47.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:47 smithi137 ceph-mon[29408]: from='client.14134 v1:172.21.15.137:0/4070630447' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:25:48.366 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: Generating public/private rsa key pair. 2024-09-18T07:25:48.366 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: Your identification has been saved in /tmp/tmpoalq2l25/key 2024-09-18T07:25:48.366 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: Your public key has been saved in /tmp/tmpoalq2l25/key.pub 2024-09-18T07:25:48.367 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: The key fingerprint is: 2024-09-18T07:25:48.367 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: SHA256:2WOowLPkulXubb5PSLz9m+LpG6XJjAkp8cCxFFKuV6A ceph-f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:25:48.367 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: The key's randomart image is: 2024-09-18T07:25:48.367 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: +---[RSA 3072]----+ 2024-09-18T07:25:48.367 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: | ..*. | 2024-09-18T07:25:48.367 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: | * + | 2024-09-18T07:25:48.367 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: | E * . | 2024-09-18T07:25:48.367 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: | o = o + | 2024-09-18T07:25:48.367 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: | . B = S + . | 2024-09-18T07:25:48.367 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: | + B + X = | 2024-09-18T07:25:48.367 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: | + o = O | 2024-09-18T07:25:48.367 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: | o . ....+ . | 2024-09-18T07:25:48.368 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: | o. .o+=*o+. | 2024-09-18T07:25:48.368 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:25:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: +----[SHA256]-----+ 2024-09-18T07:25:49.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:49 smithi137 ceph-mon[29408]: from='client.14136 v1:172.21.15.137:0/629145073' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:25:49.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:49 smithi137 ceph-mon[29408]: Generating ssh key... 2024-09-18T07:25:49.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:49 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' 2024-09-18T07:25:49.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:49 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' 2024-09-18T07:25:49.800 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrnBmUXppQys/mDLvo745XwnIWnPB+pJ5DH68xXXslB0zIW+d66UJVOf1KWGg0ZSgxBo6KO0uCjdMJl7wTotePjxr7Gf6x8YZZH7hUmxvAp3byb/CJxF1Syr1oSIc7BtCr0MuEi8gJUGVfGAHG7OUl0Z8r3Ez6r6gbOE5fhx4J0vLmAfMc31apwIsr1ABqwI9/HSBmXg9qCAMyCzPqBNGT/wBVwDnklQjd9HqLViTpM1Wo66jVuHUUYhrHzZmdmlstp48LPliQacadyXM1CA8liWiiofv2iAUzFMshdti/AMxedb6Ld8cmJxEjJlcykMVkr44QWnpiEXZ68hHjXUDGpr3/j9oMttELXDZtbKHDMvM2m2M703MeOQarv2xcqVPDwl2i8iio9xqBfvFmdeuLyn87nkcVFQ1w6ukBv1kBVY4C6POnydN9v87YCvXBkg5m3CLeL1J0c838vQjCjjdVxM+Vz5Il0m1+x7An9bs47z1Cv3Jvpyo4zdDWxCIMAn8= ceph-f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:25:49.801 INFO:teuthology.orchestra.run.smithi137.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-18T07:25:49.801 INFO:teuthology.orchestra.run.smithi137.stdout:Adding key to root@localhost authorized_keys... 2024-09-18T07:25:49.806 INFO:teuthology.orchestra.run.smithi137.stdout:Adding host smithi137... 2024-09-18T07:25:50.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:50 smithi137 ceph-mon[29408]: from='client.14138 v1:172.21.15.137:0/1194053417' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:25:51.158 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:51 smithi137 ceph-mon[29408]: from='client.14140 v1:172.21.15.137:0/557260154' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi137", "addr": "172.21.15.137", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:25:52.083 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:52 smithi137 ceph-mon[29408]: Deploying cephadm binary to smithi137 2024-09-18T07:25:54.279 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout Added host 'smithi137' with addr '172.21.15.137' 2024-09-18T07:25:54.279 INFO:teuthology.orchestra.run.smithi137.stdout:Deploying unmanaged mon service... 2024-09-18T07:25:54.551 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:54 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' 2024-09-18T07:25:54.551 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:54 smithi137 ceph-mon[29408]: Added host smithi137 2024-09-18T07:25:54.551 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:54 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:25:56.321 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:55 smithi137 ceph-mon[29408]: from='client.14142 v1:172.21.15.137:0/74658146' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:25:56.321 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:55 smithi137 ceph-mon[29408]: Saving service mon spec with placement count:5 2024-09-18T07:25:56.321 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:55 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' 2024-09-18T07:25:56.321 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:55 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' 2024-09-18T07:25:56.323 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-18T07:25:56.323 INFO:teuthology.orchestra.run.smithi137.stdout:Deploying unmanaged mgr service... 2024-09-18T07:25:57.343 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:56 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' 2024-09-18T07:25:57.343 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:56 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' 2024-09-18T07:25:57.690 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-18T07:25:58.250 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:57 smithi137 ceph-mon[29408]: from='client.14144 v1:172.21.15.137:0/586805319' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:25:58.250 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:57 smithi137 ceph-mon[29408]: Saving service mgr spec with placement count:2 2024-09-18T07:25:59.390 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:59 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1144905126' entity='client.admin' 2024-09-18T07:25:59.391 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:25:59 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' 2024-09-18T07:26:00.155 INFO:teuthology.orchestra.run.smithi137.stdout:Enabling the dashboard module... 2024-09-18T07:26:00.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:00 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1632465826' entity='client.admin' 2024-09-18T07:26:00.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:00 smithi137 ceph-mon[29408]: from='mgr.14118 v1:172.21.15.137:0/118435946' entity='mgr.y' 2024-09-18T07:26:01.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:01 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/784824257' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-18T07:26:01.919 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:01 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ignoring --setuser ceph since I am not root 2024-09-18T07:26:01.920 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:01 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ignoring --setgroup ceph since I am not root 2024-09-18T07:26:02.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:02 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/784824257' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-18T07:26:02.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:02 smithi137 ceph-mon[29408]: mgrmap e9: y(active, since 19s) 2024-09-18T07:26:03.566 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout { 2024-09-18T07:26:03.566 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-18T07:26:03.566 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-18T07:26:03.566 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-18T07:26:03.566 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-18T07:26:03.566 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout } 2024-09-18T07:26:03.567 INFO:teuthology.orchestra.run.smithi137.stdout:Waiting for the mgr to restart... 2024-09-18T07:26:03.567 INFO:teuthology.orchestra.run.smithi137.stdout:Waiting for mgr epoch 9... 2024-09-18T07:26:03.882 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:03 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2461307236' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-18T07:26:06.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:06 smithi137 ceph-mon[29408]: Active manager daemon y restarted 2024-09-18T07:26:06.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:06 smithi137 ceph-mon[29408]: Activating manager daemon y 2024-09-18T07:26:06.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:06 smithi137 ceph-mon[29408]: osdmap e3: 0 total, 0 up, 0 in 2024-09-18T07:26:06.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:06 smithi137 ceph-mon[29408]: mgrmap e10: y(active, starting, since 0.00373412s) 2024-09-18T07:26:06.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:06 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:26:06.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:06 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-18T07:26:06.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:06 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-18T07:26:06.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:06 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-18T07:26:06.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:06 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-18T07:26:06.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:06 smithi137 ceph-mon[29408]: Manager daemon y is now available 2024-09-18T07:26:07.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:07 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:26:07.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:07 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-18T07:26:07.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:07 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-18T07:26:07.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:07 smithi137 ceph-mon[29408]: from='client.14156 v1:172.21.15.137:0/66805450' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-18T07:26:07.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:07 smithi137 ceph-mon[29408]: mgrmap e11: y(active, since 1.00625s) 2024-09-18T07:26:07.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:07 smithi137 ceph-mon[29408]: from='client.14156 v1:172.21.15.137:0/66805450' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-18T07:26:08.316 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout { 2024-09-18T07:26:08.316 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-18T07:26:08.316 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-18T07:26:08.316 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout } 2024-09-18T07:26:08.316 INFO:teuthology.orchestra.run.smithi137.stdout:mgr epoch 9 is available 2024-09-18T07:26:08.316 INFO:teuthology.orchestra.run.smithi137.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-18T07:26:08.599 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: 2024-09-18T07:26:08.344+0000 7f171cc0e640 -1 log_channel(cephadm) log [ERR] : [18/Sep/2024:07:26:08] ENGINE Error in HTTPServer.serve 2024-09-18T07:26:08.599 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: Traceback (most recent call last): 2024-09-18T07:26:08.600 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-18T07:26:08.600 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: self._connections.run(self.expiration_interval) 2024-09-18T07:26:08.600 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-18T07:26:08.600 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: self._run(expiration_interval) 2024-09-18T07:26:08.600 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-18T07:26:08.600 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: new_conn = self._from_server_socket(self.server.socket) 2024-09-18T07:26:08.600 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-18T07:26:08.600 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-18T07:26:08.600 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-18T07:26:08.600 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: s = self.context.wrap_socket( 2024-09-18T07:26:08.600 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-18T07:26:08.601 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: return self.sslsocket_class._create( 2024-09-18T07:26:08.601 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-18T07:26:08.601 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: self.do_handshake() 2024-09-18T07:26:08.601 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-18T07:26:08.601 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: self._sslobj.do_handshake() 2024-09-18T07:26:08.601 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-18T07:26:08.601 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: 2024-09-18T07:26:08.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-mon[29408]: [18/Sep/2024:07:26:08] ENGINE Bus STARTING 2024-09-18T07:26:08.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-mon[29408]: [18/Sep/2024:07:26:08] ENGINE Serving on http://172.21.15.137:8765 2024-09-18T07:26:08.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-mon[29408]: [18/Sep/2024:07:26:08] ENGINE Serving on https://172.21.15.137:7150 2024-09-18T07:26:08.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-mon[29408]: [18/Sep/2024:07:26:08] ENGINE Bus STARTED 2024-09-18T07:26:08.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:08 smithi137 ceph-mon[29408]: [18/Sep/2024:07:26:08] ENGINE Error in HTTPServer.serve 2024-09-18T07:26:08.920 INFO:journalctl@ceph.mon.a.smithi137.stdout: Traceback (most recent call last): 2024-09-18T07:26:08.920 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-18T07:26:08.920 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._connections.run(self.expiration_interval) 2024-09-18T07:26:08.920 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-18T07:26:08.920 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._run(expiration_interval) 2024-09-18T07:26:08.920 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-18T07:26:08.920 INFO:journalctl@ceph.mon.a.smithi137.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-18T07:26:08.920 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-18T07:26:08.920 INFO:journalctl@ceph.mon.a.smithi137.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-18T07:26:08.920 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-18T07:26:08.920 INFO:journalctl@ceph.mon.a.smithi137.stdout: s = self.context.wrap_socket( 2024-09-18T07:26:08.920 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-18T07:26:08.920 INFO:journalctl@ceph.mon.a.smithi137.stdout: return self.sslsocket_class._create( 2024-09-18T07:26:08.921 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-18T07:26:08.921 INFO:journalctl@ceph.mon.a.smithi137.stdout: self.do_handshake() 2024-09-18T07:26:08.921 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-18T07:26:08.921 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._sslobj.do_handshake() 2024-09-18T07:26:08.921 INFO:journalctl@ceph.mon.a.smithi137.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-18T07:26:09.829 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:09 smithi137 ceph-mon[29408]: mgrmap e12: y(active, since 2s) 2024-09-18T07:26:09.829 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:09 smithi137 ceph-mon[29408]: from='client.14164 v1:172.21.15.137:0/1731030587' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:26:09.829 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:09 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:10.073 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIE/DCCAuSgAwIBAgIUL5T65D90JZYAqZD+YtiuxVaXS9QwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkxODA3MjYwOVoXDTM0\nMDkxOTA3MjYwOVowGDEWMBQGA1UEAwwNMTcyLjIxLjE1LjEzNzCCAiIwDQYJKoZI\nhvcNAQEBBQADggIPADCCAgoCggIBAIXhBWQ63WQg2dXI2NVxd4K0vuwPwufUwz8t\nCGaf/jEtY8a5sKpsukrx6qvkf8bBEELXdIc6A72iLEu3ShVAUfRqBU7CGHWRC86z\nMDLRmDyXeS6KmSBZsOtzMB+CRtZ1Y7EdZlj8+OAtG4nsC2iYWMsaFIAwU4+k8Ssd\nQFIVcd9B2tksuW8TuTxAcZ2CxkWyylaNfmCgkAKJRvh0Nkn+JjmVa0fewx5IjivZ\n6G5bOvWZu26NklYlKypJ1RjylKvI2i+IKZIDsEB8LsOBxNjd9rvo1ciTmiy6Vtoi\n8sHFTOJZIV2TqSVErZOkMCg0f9oVE4MCS1OI9mI5rbktU1mAb8cHyvc7My6rgCk2\ntXdyEHFlTFSwazH14NZIlQGMRt3UF+2s+lhnXHo3Dxukkin7Wm67n1QZ5jThg0pQ\nF+gIv1Pg1znh+W27vsWVmzyaN9a83Yh0X3r8hxtC1q+S3fN/uVO+W9e37fYnAmvF\ncCHaTDWQnmTAw1pW7BR7o5LbUDnFkn2bXa9fVkDgD2fcU7noDKIUJYFLJwEzoJ2C\nvPY4lcvseY8tEF6it/gpmq6FBYH7vjTG6L9iLp8Ro+/4W0nu87WHTebY0etpR3C4\n/uXaULxxJQxOpB6REBeaVXzWRyqyxAgEqsSbEfsIO73MG6YWHMOchLYYZr6oh6FH\nhw9QBXIfAgMBAAGjPzA9MC0GA1UdEQQmMCSCCXNtaXRoaTEzN4IRZGFzaGJvYXJk\nX3NlcnZlcnOHBKwVD4kwDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQsFAAOCAgEA\nf7w4xHFt0WYhTORSYFqaHHFMSTpge/aPFL/Zn3pLq0YWB2rlUV980DSVMo9P4pEM\nS+4MBqKIb4MUPXNqTEeoGwmnGj9oaUo67+CoMPT5jVgk/gQywy7ccuA57NPBAODo\nOGZ1jJResbqBW0wor+d3VAh7798Dtg6If9lBmFMuvkDWgBUhEhFcnIE5LMW2y4e+\nnL811DesQp7LfxDjCzHkan2JiO/HivAXMWAsxHV7rwAcsh/Rk2XPMmSoJcxEI5i3\nv9wbSkxm/D3vnC9S2uePy5NEktllAfnAfVc6iPjVzSNznbKByGQgyQ4y8Js2L57/\nCjgHdYaphEvFicHBUWNex7yXqdRpWFwHIdEjNfLM11bBRIkgIcQ676s5iHMmw+5S\nO2M8WTXYuqSvWEUlPtLR/eEKpqyXR8FG6HJqUEO7iua+QcoEFAgvpiHvM++8T9uT\n7YeQJYvC2l3VRbb0m13xU3Hi01tsjYhuwgWlncUsyT0fhscGs0M3a9liwcEwSxpD\nVN3hvAiBnM0JWkDnse88LbBr5F1kK4r4La4/IFFAAe6+khnVul5sl7LlIKdw3kOV\n0eBEcnBipbs+tYRKXul0WIBqwUCNQKJKsYvdG2TGObqp+/Ei7q22yevqlwBvVM5s\nPYImSnZ2SlXujlkgzpsFHJrFIJz0oAf/m/e1XTEs8pA=\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKQIBAAKCAgEAheEFZDrdZCDZ1cjY1XF3grS+7A/C59TDPy0IZp/+MS1jxrmw\nqmy6SvHqq+R/xsEQQtd0hzoDvaIsS7dKFUBR9GoFTsIYdZELzrMwMtGYPJd5LoqZ\nIFmw63MwH4JG1nVjsR1mWPz44C0biewLaJhYyxoUgDBTj6TxKx1AUhVx30Ha2Sy5\nbxO5PEBxnYLGRbLKVo1+YKCQAolG+HQ2Sf4mOZVrR97DHkiOK9nobls69Zm7bo2S\nViUrKknVGPKUq8jaL4gpkgOwQHwuw4HE2N32u+jVyJOaLLpW2iLywcVM4lkhXZOp\nJUStk6QwKDR/2hUTgwJLU4j2YjmtuS1TWYBvxwfK9zszLquAKTa1d3IQcWVMVLBr\nMfXg1kiVAYxG3dQX7az6WGdcejcPG6SSKftabrufVBnmNOGDSlAX6Ai/U+DXOeH5\nbbu+xZWbPJo31rzdiHRfevyHG0LWr5Ld83+5U75b17ft9icCa8VwIdpMNZCeZMDD\nWlbsFHujkttQOcWSfZtdr19WQOAPZ9xTuegMohQlgUsnATOgnYK89jiVy+x5jy0Q\nXqK3+CmaroUFgfu+NMbov2IunxGj7/hbSe7ztYdN5tjR62lHcLj+5dpQvHElDE6k\nHpEQF5pVfNZHKrLECASqxJsR+wg7vcwbphYcw5yEthhmvqiHoUeHD1AFch8CAwEA\nAQKCAgAnO6WuVTfK33um1EiC6FELCzfGCuplsOqUyXgkRu+hbrlGI+WbU+iWDOXm\nALzXPH2s8OHTvRMaj/tmJuz9ZE+ADk/G4Ed1sI9iwA04X0a7PvKCypgMLGyV2zg3\ni6CBuX4BCeoT2vJ+E4n19af0O20N+/MVC9R5DG+mYFiYlXM851ogeaRP4lpOx13k\nOE4tX7M/AREPdQ4uyBb1uWyta169UkXRlnzck4uiWyXrrS2OisIsKC3Bs/u91Uwv\nFuutmdsiwcbY5N7STvIPViifGxDyyU1ZDbRJNkQ8gbBdS5n33QHTn/GaG9G4Kg34\nwkXTXYjNxC7Z6JG3gEqiMR+zY1md7+jdfhjcXTAdL1GwZRPEQXwnTzQMHC3Wk19/\nqk15qOW86YL+qZKw/sOg18xLdGXVmJ/5/t7u/W9g+73NljEdssxE1y9/m7n9oqSq\nGJVW2T6k/dh4NxsPGZVDPK6Bc0QNDDv6gDRJebwGsvBJapHwQOBMMF0KqaqBjaV0\n4vIcTgTyibbOLzBr96j01FwDclk8SpvuyMQFy0rsHs8c44slXkV1+V6DszAar4rA\nsKYCfwVCv4KD8EQMded+5VJvnOMaa/hSD9sgP/8Bga67Zy+NVGrnvTm+efPZC4O3\nvwcEpj9MXssjQUn0PvXWMb3Xox3e4IqqykPUIhxyUCiR/ujr7QKCAQEAvU5S9yTT\nzPJtBN7a/8fL8yxlfi9Oke3QRBCy4RH4P1qXDHF8zRAbDB2lKy8a8gCevqRkRojc\nWyE/pH1+rvQKXIyONA70t5Ds/f5TXjb8fHtVQRp1Sv6ahtNNfVyLcd9SiToFbW/z\nEdXIOIv8pUbuUC8ds618ZJmyMseLQq7ExSsmS5+goYa+OHCDkGCcT0ijdpw5hZxX\nwXbO07POfsUusLC/qM995O/hT/x/pI1vsn9RVggVjm5Z6UVXIl2RGHPJdwKv5U/I\nlsSfxUVmygSQBw0egIeYU7zPuVJTelEtdWpvsdzlzolB1gGHcAmvo5C7XhHACnK1\nwKCgZ9pXtJ3aswKCAQEAtQuwS9X3B9eVFIInB6OK3rweeKRw1xjWAH2FpiVtq57/\nDIFB90DvIj56s1rsoxwOkGx2cvki6KFCC9M7GqSPSo/ec7wWuLMLB4Kdi80kP8pf\nqTdipVb3t+xCQrIrtTC3N+CtTkmNJJOcOev8Kt2PCJpzcxfW4sVLdK4dn+HJ4den\noFdm1GH5Fh+U6//O0jagZdRHwXO6SJlX3jZklhkWg2Lc+1AUTBVctUaa6D5/0+Hs\nwTDqF3cU2HJCvjTyc04jkRf7c9sRsuxdrrwI00aPwnwqL6MVSFnmH0OsRZO93hrI\n5tJZL1RkKxoGsuy/kE/L1D7db5BPtorHCs9/n5Pw5QKCAQEAqXM9snnlmLN7wHtX\nXtNSDUa1RqFD3D4aSYMFC5BRPSxyvrGrO8dp3ot+uEv++5ZwgkeWdsi1LmFxnWS0\nDBjoWXV7KMAmb1YrvSN1LoVbtnWTl6X+TfXfDkaAO3GmZHS5ZZppF0+5LULGaU9t\nA+idE2tJU/c7vLUTHH8feb4E5lS8f77QQyXgiZa1PXwtDo7/uwEWQDH1H3igpEFz\n2mfYZlKgDJRGpbHPZhCnlP0Bea6beL3dxdOjabomx90nLtMeCAiAuGuVKtewCB4a\nZ0R0Q8uLfRNi17nP2pwFmxyk61At0PDhR+H264AM4s3YyTmZbhfHZdLOPDHLw4pf\nLY6LHwKCAQEApSiWmXgFtUe7nkvlYxiUdQDUI2aP8gmcIDJiG+GFHM0dCsFX9kFs\nZabttHS770iHGMV5gQU6DVIeuD7oajO47GI2ownxCGl9JErJgmGQMgliHzKBcSfX\nMBf9km6LZO2eFwGF2CdM9RMbr9NR5aDc/RI5Q8N5qsIlRboZGGOYEcj0CFfzXMhL\nvRzudCSW/joLNRnko9v5EoWhHPwt4qfcW9tykGjLO2U5f717T2bEofw2ST9gLivZ\n/mKt9GrEyt97nr9sCpLMoJb0F5U9z89NEjTHH1H89lXckqTfoVFYIif5Imi+J2kr\nQA6sEcncL8A7x9HZnA7gSTxsu7XzTRZe6QKCAQBGgwZvwIvEZvhEU/fnlMCNd/9A\nKjfUU540311A7y6ah+TFPbVc8YW/WCGOTbKBp07/5f4Gym9x59L5fVE6V+L+fTTE\nL6YGD3dTnwUjDhAeALfhZ9A08pBB75uQUv6nLlW+453/32LXnHItnorenyk6oLMp\nX2soYIY0UEPF1eZ0UZTqqvq/8tNUCd8tlUDo3Uq5cZ1vUtfF9iJ0q3lkIHtV6On5\n+S1snOWT67YnybNUtb59WJOTh602fcEnorMca97YfQDwgyqsby+jTToMyzDRmxnU\nNKg+OEO4khZOZM36dZWmDwvPUtuTBCd4gIfiQVMv+ry9YXsB8aKbgOjG4kd7\n-----END RSA PRIVATE KEY-----\n"} 2024-09-18T07:26:11.315 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-18T07:26:11.413 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:11 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:11.413 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:11 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:12.395 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:12 smithi137 ceph-mon[29408]: from='client.14166 v1:172.21.15.137:0/2335904786' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:26:13.217 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-18T07:26:13.217 INFO:teuthology.orchestra.run.smithi137.stdout:Creating initial admin user... 2024-09-18T07:26:13.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:13 smithi137 ceph-mon[29408]: from='client.14168 v1:172.21.15.137:0/163676162' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:26:13.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:13 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:14.582 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$v/ar3JC5IgZPCPaWkcjRH.UskuH1WV3n0rj6ulQiYF5pSaBtkOGcS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726644374, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-18T07:26:14.582 INFO:teuthology.orchestra.run.smithi137.stdout:Fetching dashboard port number... 2024-09-18T07:26:15.144 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:15 smithi137 ceph-mon[29408]: from='client.14170 v1:172.21.15.137:0/3766461267' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:26:15.165 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:15 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:15.663 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 8443 2024-09-18T07:26:15.673 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-18T07:26:15.673 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout disabled 2024-09-18T07:26:15.680 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-18T07:26:15.680 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout inactive 2024-09-18T07:26:15.680 INFO:teuthology.orchestra.run.smithi137.stdout:firewalld.service is not enabled 2024-09-18T07:26:15.680 INFO:teuthology.orchestra.run.smithi137.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-18T07:26:15.682 INFO:teuthology.orchestra.run.smithi137.stdout:Ceph Dashboard is now available at: 2024-09-18T07:26:15.682 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:26:15.682 INFO:teuthology.orchestra.run.smithi137.stdout: URL: https://smithi137.front.sepia.ceph.com:8443/ 2024-09-18T07:26:15.682 INFO:teuthology.orchestra.run.smithi137.stdout: User: admin 2024-09-18T07:26:15.682 INFO:teuthology.orchestra.run.smithi137.stdout: Password: n9y0xmnt89 2024-09-18T07:26:15.682 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:26:15.683 INFO:teuthology.orchestra.run.smithi137.stdout:Saving cluster configuration to /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config directory 2024-09-18T07:26:16.364 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:16 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1149835468' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-18T07:26:16.962 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-18T07:26:16.962 INFO:teuthology.orchestra.run.smithi137.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-18T07:26:16.962 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:26:16.962 INFO:teuthology.orchestra.run.smithi137.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-18T07:26:16.962 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:26:16.962 INFO:teuthology.orchestra.run.smithi137.stdout:Or, if you are only running a single cluster on this host: 2024-09-18T07:26:16.962 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:26:16.962 INFO:teuthology.orchestra.run.smithi137.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-18T07:26:16.962 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:26:16.963 INFO:teuthology.orchestra.run.smithi137.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-18T07:26:16.963 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:26:16.963 INFO:teuthology.orchestra.run.smithi137.stdout: ceph telemetry on 2024-09-18T07:26:16.963 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:26:16.963 INFO:teuthology.orchestra.run.smithi137.stdout:For more information see: 2024-09-18T07:26:16.963 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:26:16.963 INFO:teuthology.orchestra.run.smithi137.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-18T07:26:16.963 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:26:16.963 INFO:teuthology.orchestra.run.smithi137.stdout:Bootstrap complete. 2024-09-18T07:26:17.002 INFO:tasks.cephadm:Fetching config... 2024-09-18T07:26:17.003 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-18T07:26:17.003 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-18T07:26:17.019 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-18T07:26:17.020 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-18T07:26:17.020 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-18T07:26:17.075 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-18T07:26:17.075 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-18T07:26:17.075 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-18T07:26:17.140 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-18T07:26:17.140 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-18T07:26:17.140 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-18T07:26:17.196 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-18T07:26:17.196 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrnBmUXppQys/mDLvo745XwnIWnPB+pJ5DH68xXXslB0zIW+d66UJVOf1KWGg0ZSgxBo6KO0uCjdMJl7wTotePjxr7Gf6x8YZZH7hUmxvAp3byb/CJxF1Syr1oSIc7BtCr0MuEi8gJUGVfGAHG7OUl0Z8r3Ez6r6gbOE5fhx4J0vLmAfMc31apwIsr1ABqwI9/HSBmXg9qCAMyCzPqBNGT/wBVwDnklQjd9HqLViTpM1Wo66jVuHUUYhrHzZmdmlstp48LPliQacadyXM1CA8liWiiofv2iAUzFMshdti/AMxedb6Ld8cmJxEjJlcykMVkr44QWnpiEXZ68hHjXUDGpr3/j9oMttELXDZtbKHDMvM2m2M703MeOQarv2xcqVPDwl2i8iio9xqBfvFmdeuLyn87nkcVFQ1w6ukBv1kBVY4C6POnydN9v87YCvXBkg5m3CLeL1J0c838vQjCjjdVxM+Vz5Il0m1+x7An9bs47z1Cv3Jvpyo4zdDWxCIMAn8= ceph-f7ef099c-758e-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-18T07:26:17.269 INFO:teuthology.orchestra.run.smithi137.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrnBmUXppQys/mDLvo745XwnIWnPB+pJ5DH68xXXslB0zIW+d66UJVOf1KWGg0ZSgxBo6KO0uCjdMJl7wTotePjxr7Gf6x8YZZH7hUmxvAp3byb/CJxF1Syr1oSIc7BtCr0MuEi8gJUGVfGAHG7OUl0Z8r3Ez6r6gbOE5fhx4J0vLmAfMc31apwIsr1ABqwI9/HSBmXg9qCAMyCzPqBNGT/wBVwDnklQjd9HqLViTpM1Wo66jVuHUUYhrHzZmdmlstp48LPliQacadyXM1CA8liWiiofv2iAUzFMshdti/AMxedb6Ld8cmJxEjJlcykMVkr44QWnpiEXZ68hHjXUDGpr3/j9oMttELXDZtbKHDMvM2m2M703MeOQarv2xcqVPDwl2i8iio9xqBfvFmdeuLyn87nkcVFQ1w6ukBv1kBVY4C6POnydN9v87YCvXBkg5m3CLeL1J0c838vQjCjjdVxM+Vz5Il0m1+x7An9bs47z1Cv3Jvpyo4zdDWxCIMAn8= ceph-f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:26:17.282 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrnBmUXppQys/mDLvo745XwnIWnPB+pJ5DH68xXXslB0zIW+d66UJVOf1KWGg0ZSgxBo6KO0uCjdMJl7wTotePjxr7Gf6x8YZZH7hUmxvAp3byb/CJxF1Syr1oSIc7BtCr0MuEi8gJUGVfGAHG7OUl0Z8r3Ez6r6gbOE5fhx4J0vLmAfMc31apwIsr1ABqwI9/HSBmXg9qCAMyCzPqBNGT/wBVwDnklQjd9HqLViTpM1Wo66jVuHUUYhrHzZmdmlstp48LPliQacadyXM1CA8liWiiofv2iAUzFMshdti/AMxedb6Ld8cmJxEjJlcykMVkr44QWnpiEXZ68hHjXUDGpr3/j9oMttELXDZtbKHDMvM2m2M703MeOQarv2xcqVPDwl2i8iio9xqBfvFmdeuLyn87nkcVFQ1w6ukBv1kBVY4C6POnydN9v87YCvXBkg5m3CLeL1J0c838vQjCjjdVxM+Vz5Il0m1+x7An9bs47z1Cv3Jvpyo4zdDWxCIMAn8= ceph-f7ef099c-758e-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-18T07:26:17.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:17 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2783379920' entity='client.admin' 2024-09-18T07:26:17.446 INFO:teuthology.orchestra.run.smithi203.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrnBmUXppQys/mDLvo745XwnIWnPB+pJ5DH68xXXslB0zIW+d66UJVOf1KWGg0ZSgxBo6KO0uCjdMJl7wTotePjxr7Gf6x8YZZH7hUmxvAp3byb/CJxF1Syr1oSIc7BtCr0MuEi8gJUGVfGAHG7OUl0Z8r3Ez6r6gbOE5fhx4J0vLmAfMc31apwIsr1ABqwI9/HSBmXg9qCAMyCzPqBNGT/wBVwDnklQjd9HqLViTpM1Wo66jVuHUUYhrHzZmdmlstp48LPliQacadyXM1CA8liWiiofv2iAUzFMshdti/AMxedb6Ld8cmJxEjJlcykMVkr44QWnpiEXZ68hHjXUDGpr3/j9oMttELXDZtbKHDMvM2m2M703MeOQarv2xcqVPDwl2i8iio9xqBfvFmdeuLyn87nkcVFQ1w6ukBv1kBVY4C6POnydN9v87YCvXBkg5m3CLeL1J0c838vQjCjjdVxM+Vz5Il0m1+x7An9bs47z1Cv3Jvpyo4zdDWxCIMAn8= ceph-f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:26:17.460 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-18T07:26:17.711 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:26:18.967 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-18T07:26:18.967 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-18T07:26:19.459 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:26:19.556 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:19 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1668953773' entity='client.admin' 2024-09-18T07:26:20.542 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:20.542 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:20 smithi137 ceph-mon[29408]: mgrmap e13: y(active, since 12s) 2024-09-18T07:26:20.542 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:20.542 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi137", "name": "osd_memory_target"} : dispatch 2024-09-18T07:26:20.543 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:20.543 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:26:20.543 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:21.799 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:21 smithi137 ceph-mon[29408]: from='client.14178 v1:172.21.15.137:0/1183139131' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:26:21.799 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:21 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:21.799 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:21 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:26:21.799 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:21 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:21.800 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:21 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:26:21.800 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:21 smithi137 ceph-mon[29408]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-18T07:26:21.800 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:21 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:21.800 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:21 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:21.800 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:21 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:21.828 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi203 2024-09-18T07:26:21.828 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-18T07:26:21.828 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.conf 2024-09-18T07:26:21.849 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-18T07:26:21.849 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:26:21.908 INFO:tasks.cephadm:Adding host smithi203 to orchestrator... 2024-09-18T07:26:21.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch host add smithi203 2024-09-18T07:26:22.151 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:26:22.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:22 smithi137 ceph-mon[29408]: Updating smithi137:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:26:22.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:22 smithi137 ceph-mon[29408]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:26:22.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:22 smithi137 ceph-mon[29408]: Updating smithi137:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T07:26:24.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:23 smithi137 ceph-mon[29408]: from='client.14180 v1:172.21.15.137:0/2915356307' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:26:24.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:23 smithi137 ceph-mon[29408]: Deploying cephadm binary to smithi203 2024-09-18T07:26:25.844 INFO:teuthology.orchestra.run.smithi137.stdout:Added host 'smithi203' with addr '172.21.15.203' 2024-09-18T07:26:26.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-18T07:26:26.736 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:26:27.100 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:26 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:27.100 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:26 smithi137 ceph-mon[29408]: Added host smithi203 2024-09-18T07:26:27.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:26 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:26:27.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:26 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:27.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:26 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:27.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:26 smithi137 ceph-mon[29408]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:27.371 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:26:27.371 INFO:teuthology.orchestra.run.smithi137.stdout:[{"addr": "172.21.15.137", "hostname": "smithi137", "labels": [], "status": ""}, {"addr": "172.21.15.203", "hostname": "smithi203", "labels": [], "status": ""}] 2024-09-18T07:26:27.966 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-18T07:26:27.967 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-18T07:26:28.210 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:26:28.330 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:28 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:28.330 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:28 smithi137 ceph-mon[29408]: from='client.14182 v1:172.21.15.137:0/1328863449' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:26:29.157 INFO:teuthology.orchestra.run.smithi137.stderr:adjusted tunables profile to default 2024-09-18T07:26:29.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:29 smithi137 ceph-mon[29408]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:29.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:29 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4239655369' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-18T07:26:29.731 INFO:tasks.cephadm:Adding mon.a on smithi137 2024-09-18T07:26:29.731 INFO:tasks.cephadm:Adding mon.c on smithi137 2024-09-18T07:26:29.731 INFO:tasks.cephadm:Adding mon.b on smithi203 2024-09-18T07:26:29.732 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi137:[v1:172.21.15.137:6789]=a;smithi137:[v1:172.21.15.137:6790]=c;smithi203:[v1:172.21.15.203:6789]=b' 2024-09-18T07:26:29.991 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-18T07:26:30.038 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-18T07:26:30.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:30 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4239655369' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-18T07:26:30.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:30 smithi137 ceph-mon[29408]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:26:30.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:30 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:30.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:30 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:30.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:30 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:30.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:30 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:30.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:30 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"} : dispatch 2024-09-18T07:26:30.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:30 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:30.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:30 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:26:30.729 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mon update... 2024-09-18T07:26:31.349 DEBUG:teuthology.orchestra.run.smithi137:mon.c> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mon.c.service 2024-09-18T07:26:31.352 DEBUG:teuthology.orchestra.run.smithi203:mon.b> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mon.b.service 2024-09-18T07:26:31.354 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-18T07:26:31.355 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-18T07:26:31.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:31 smithi137 ceph-mon[29408]: Updating smithi203:/etc/ceph/ceph.conf 2024-09-18T07:26:31.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:31 smithi137 ceph-mon[29408]: Updating smithi203:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:26:31.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:31 smithi137 ceph-mon[29408]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:26:31.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:31 smithi137 ceph-mon[29408]: Updating smithi203:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T07:26:31.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:31 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:31.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:31 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:31.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:31 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:31.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:31 smithi137 ceph-mon[29408]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:31.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:31 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:31.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:31 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:26:31.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:31 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:31.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:31 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:26:31.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:31 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:31.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:31 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:26:31.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:31 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:32.035 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:26:32.684 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:26:32.685 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"f7ef099c-758e-11ef-bceb-c7b262605968","modified":"2024-09-18T07:25:15.819392Z","created":"2024-09-18T07:25:15.819392Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-18T07:26:32.685 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-09-18T07:26:32.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:32 smithi137 ceph-mon[29408]: from='client.14186 v1:172.21.15.203:0/823080327' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi137:[v1:172.21.15.137:6789]=a;smithi137:[v1:172.21.15.137:6790]=c;smithi203:[v1:172.21.15.203:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:26:32.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:32 smithi137 ceph-mon[29408]: Saving service mon spec with placement smithi137:[v1:172.21.15.137:6789]=a;smithi137:[v1:172.21.15.137:6790]=c;smithi203:[v1:172.21.15.203:6789]=b;count:3 2024-09-18T07:26:32.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:32 smithi137 ceph-mon[29408]: Deploying daemon mon.b on smithi203 2024-09-18T07:26:33.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:33 smithi137 ceph-mon[29408]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:33.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:33 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.203:0/1352026863' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:26:34.724 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-18T07:26:34.724 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-18T07:26:35.094 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:26:35.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:34 smithi137 ceph-mon[29408]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:35.632 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:35 smithi203 systemd[1]: Starting Ceph mon.b for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:26:35.978 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:26:35.978 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"f7ef099c-758e-11ef-bceb-c7b262605968","modified":"2024-09-18T07:25:15.819392Z","created":"2024-09-18T07:25:15.819392Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-18T07:26:35.978 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-09-18T07:26:36.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:35 smithi203 podman[32205]: 2024-09-18 07:26:35.955845475 +0000 UTC m=+0.110875920 container create b63ad8ee863c70070b7130c5119f6279c2f73077d55481306af42c31719b243d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-18T07:26:36.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:35 smithi203 podman[32205]: 2024-09-18 07:26:35.860699173 +0000 UTC m=+0.015729619 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:26:36.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:36 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.203:0/2421693283' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:26:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 podman[32205]: 2024-09-18 07:26:36.335171139 +0000 UTC m=+0.490201593 container init b63ad8ee863c70070b7130c5119f6279c2f73077d55481306af42c31719b243d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-18T07:26:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 podman[32205]: 2024-09-18 07:26:36.33853074 +0000 UTC m=+0.493561186 container start b63ad8ee863c70070b7130c5119f6279c2f73077d55481306af42c31719b243d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-18T07:26:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: set uid:gid to 167:167 (ceph:ceph) 2024-09-18T07:26:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: ceph version 19.3.0-4983-ge7fb7b56 (e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e) squid (dev), process ceph-mon, pid 2 2024-09-18T07:26:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: pidfile_write: ignore empty --pid-file 2024-09-18T07:26:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: load: jerasure load: lrc 2024-09-18T07:26:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: RocksDB version: 7.9.2 2024-09-18T07:26:36.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Git sha 0 2024-09-18T07:26:36.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Compile date 2024-09-17 15:59:30 2024-09-18T07:26:36.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: DB SUMMARY 2024-09-18T07:26:36.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: DB Session ID: 2OI65DZ60F8U8YH68DP3 2024-09-18T07:26:36.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: CURRENT file: CURRENT 2024-09-18T07:26:36.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: IDENTITY file: IDENTITY 2024-09-18T07:26:36.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-18T07:26:36.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-18T07:26:36.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 476 ; 2024-09-18T07:26:36.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.error_if_exists: 0 2024-09-18T07:26:36.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.create_if_missing: 0 2024-09-18T07:26:36.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.paranoid_checks: 1 2024-09-18T07:26:36.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-18T07:26:36.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-18T07:26:36.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-18T07:26:36.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.env: 0x55e492178120 2024-09-18T07:26:36.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.fs: PosixFileSystem 2024-09-18T07:26:36.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.info_log: 0x55e49395d680 2024-09-18T07:26:36.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_file_opening_threads: 16 2024-09-18T07:26:36.714 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.statistics: (nil) 2024-09-18T07:26:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.use_fsync: 0 2024-09-18T07:26:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_log_file_size: 0 2024-09-18T07:26:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-18T07:26:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-18T07:26:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.keep_log_file_num: 1000 2024-09-18T07:26:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.recycle_log_file_num: 0 2024-09-18T07:26:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.allow_fallocate: 1 2024-09-18T07:26:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.allow_mmap_reads: 0 2024-09-18T07:26:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.allow_mmap_writes: 0 2024-09-18T07:26:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.use_direct_reads: 0 2024-09-18T07:26:36.715 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-18T07:26:36.716 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.create_missing_column_families: 0 2024-09-18T07:26:36.716 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.db_log_dir: 2024-09-18T07:26:36.716 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.wal_dir: 2024-09-18T07:26:36.716 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-18T07:26:36.716 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-18T07:26:36.716 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-18T07:26:36.716 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-18T07:26:36.716 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-18T07:26:36.716 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-18T07:26:36.716 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.advise_random_on_open: 1 2024-09-18T07:26:36.716 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.db_write_buffer_size: 0 2024-09-18T07:26:36.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.write_buffer_manager: 0x55e493961b80 2024-09-18T07:26:36.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-18T07:26:36.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-18T07:26:36.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-18T07:26:36.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.rate_limiter: (nil) 2024-09-18T07:26:36.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-18T07:26:36.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.wal_recovery_mode: 2 2024-09-18T07:26:36.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.enable_thread_tracking: 0 2024-09-18T07:26:36.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.enable_pipelined_write: 0 2024-09-18T07:26:36.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.unordered_write: 0 2024-09-18T07:26:36.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-18T07:26:36.717 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-18T07:26:36.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-18T07:26:36.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-18T07:26:36.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.row_cache: None 2024-09-18T07:26:36.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.wal_filter: None 2024-09-18T07:26:36.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-18T07:26:36.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.allow_ingest_behind: 0 2024-09-18T07:26:36.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.two_write_queues: 0 2024-09-18T07:26:36.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.manual_wal_flush: 0 2024-09-18T07:26:36.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.wal_compression: 0 2024-09-18T07:26:36.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.atomic_flush: 0 2024-09-18T07:26:36.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-18T07:26:36.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-18T07:26:36.718 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-18T07:26:36.719 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.log_readahead_size: 0 2024-09-18T07:26:36.719 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-18T07:26:36.719 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.best_efforts_recovery: 0 2024-09-18T07:26:36.719 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-18T07:26:36.719 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-18T07:26:36.719 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.allow_data_in_errors: 0 2024-09-18T07:26:36.719 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.db_host_id: __hostname__ 2024-09-18T07:26:36.719 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-18T07:26:36.719 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_background_jobs: 2 2024-09-18T07:26:36.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_background_compactions: -1 2024-09-18T07:26:36.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_subcompactions: 1 2024-09-18T07:26:36.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-18T07:26:36.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-18T07:26:36.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-18T07:26:36.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_total_wal_size: 0 2024-09-18T07:26:36.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-18T07:26:36.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-18T07:26:36.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-18T07:26:36.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-18T07:26:36.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_open_files: -1 2024-09-18T07:26:36.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.bytes_per_sync: 0 2024-09-18T07:26:36.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-18T07:26:36.720 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-18T07:26:36.721 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compaction_readahead_size: 0 2024-09-18T07:26:36.721 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_background_flushes: -1 2024-09-18T07:26:36.721 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Compression algorithms supported: 2024-09-18T07:26:36.721 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: kZSTD supported: 0 2024-09-18T07:26:36.721 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: kXpressCompression supported: 0 2024-09-18T07:26:36.721 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: kBZip2Compression supported: 0 2024-09-18T07:26:36.721 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-18T07:26:36.721 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: kLZ4Compression supported: 1 2024-09-18T07:26:36.721 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: kZlibCompression supported: 1 2024-09-18T07:26:36.721 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: kLZ4HCCompression supported: 1 2024-09-18T07:26:36.721 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: kSnappyCompression supported: 1 2024-09-18T07:26:36.721 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-18T07:26:36.721 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-18T07:26:36.722 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-18T07:26:36.722 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-18T07:26:36.722 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-18T07:26:36.722 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.merge_operator: 2024-09-18T07:26:36.722 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compaction_filter: None 2024-09-18T07:26:36.722 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compaction_filter_factory: None 2024-09-18T07:26:36.722 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.sst_partitioner_factory: None 2024-09-18T07:26:36.722 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-18T07:26:36.722 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-18T07:26:36.722 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e49395d2a0) 2024-09-18T07:26:36.722 INFO:journalctl@ceph.mon.b.smithi203.stdout: cache_index_and_filter_blocks: 1 2024-09-18T07:26:36.722 INFO:journalctl@ceph.mon.b.smithi203.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-18T07:26:36.722 INFO:journalctl@ceph.mon.b.smithi203.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-18T07:26:36.722 INFO:journalctl@ceph.mon.b.smithi203.stdout: pin_top_level_index_and_filter: 1 2024-09-18T07:26:36.723 INFO:journalctl@ceph.mon.b.smithi203.stdout: index_type: 0 2024-09-18T07:26:36.723 INFO:journalctl@ceph.mon.b.smithi203.stdout: data_block_index_type: 0 2024-09-18T07:26:36.723 INFO:journalctl@ceph.mon.b.smithi203.stdout: index_shortening: 1 2024-09-18T07:26:36.723 INFO:journalctl@ceph.mon.b.smithi203.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-18T07:26:36.723 INFO:journalctl@ceph.mon.b.smithi203.stdout: checksum: 4 2024-09-18T07:26:36.723 INFO:journalctl@ceph.mon.b.smithi203.stdout: no_block_cache: 0 2024-09-18T07:26:36.723 INFO:journalctl@ceph.mon.b.smithi203.stdout: block_cache: 0x55e493951610 2024-09-18T07:26:36.723 INFO:journalctl@ceph.mon.b.smithi203.stdout: block_cache_name: BinnedLRUCache 2024-09-18T07:26:36.723 INFO:journalctl@ceph.mon.b.smithi203.stdout: block_cache_options: 2024-09-18T07:26:36.723 INFO:journalctl@ceph.mon.b.smithi203.stdout: capacity : 536870912 2024-09-18T07:26:36.723 INFO:journalctl@ceph.mon.b.smithi203.stdout: num_shard_bits : 4 2024-09-18T07:26:36.723 INFO:journalctl@ceph.mon.b.smithi203.stdout: strict_capacity_limit : 0 2024-09-18T07:26:36.723 INFO:journalctl@ceph.mon.b.smithi203.stdout: high_pri_pool_ratio: 0.000 2024-09-18T07:26:36.724 INFO:journalctl@ceph.mon.b.smithi203.stdout: block_cache_compressed: (nil) 2024-09-18T07:26:36.724 INFO:journalctl@ceph.mon.b.smithi203.stdout: persistent_cache: (nil) 2024-09-18T07:26:36.724 INFO:journalctl@ceph.mon.b.smithi203.stdout: block_size: 4096 2024-09-18T07:26:36.724 INFO:journalctl@ceph.mon.b.smithi203.stdout: block_size_deviation: 10 2024-09-18T07:26:36.724 INFO:journalctl@ceph.mon.b.smithi203.stdout: block_restart_interval: 16 2024-09-18T07:26:36.724 INFO:journalctl@ceph.mon.b.smithi203.stdout: index_block_restart_interval: 1 2024-09-18T07:26:36.724 INFO:journalctl@ceph.mon.b.smithi203.stdout: metadata_block_size: 4096 2024-09-18T07:26:36.724 INFO:journalctl@ceph.mon.b.smithi203.stdout: partition_filters: 0 2024-09-18T07:26:36.724 INFO:journalctl@ceph.mon.b.smithi203.stdout: use_delta_encoding: 1 2024-09-18T07:26:36.724 INFO:journalctl@ceph.mon.b.smithi203.stdout: filter_policy: bloomfilter 2024-09-18T07:26:36.724 INFO:journalctl@ceph.mon.b.smithi203.stdout: whole_key_filtering: 1 2024-09-18T07:26:36.724 INFO:journalctl@ceph.mon.b.smithi203.stdout: verify_compression: 0 2024-09-18T07:26:36.724 INFO:journalctl@ceph.mon.b.smithi203.stdout: read_amp_bytes_per_bit: 0 2024-09-18T07:26:36.724 INFO:journalctl@ceph.mon.b.smithi203.stdout: format_version: 5 2024-09-18T07:26:36.725 INFO:journalctl@ceph.mon.b.smithi203.stdout: enable_index_compression: 1 2024-09-18T07:26:36.725 INFO:journalctl@ceph.mon.b.smithi203.stdout: block_align: 0 2024-09-18T07:26:36.725 INFO:journalctl@ceph.mon.b.smithi203.stdout: max_auto_readahead_size: 262144 2024-09-18T07:26:36.725 INFO:journalctl@ceph.mon.b.smithi203.stdout: prepopulate_block_cache: 0 2024-09-18T07:26:36.725 INFO:journalctl@ceph.mon.b.smithi203.stdout: initial_auto_readahead_size: 8192 2024-09-18T07:26:36.725 INFO:journalctl@ceph.mon.b.smithi203.stdout: num_file_reads_for_auto_readahead: 2 2024-09-18T07:26:36.725 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.write_buffer_size: 33554432 2024-09-18T07:26:36.725 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_write_buffer_number: 2 2024-09-18T07:26:36.725 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compression: NoCompression 2024-09-18T07:26:36.725 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.bottommost_compression: Disabled 2024-09-18T07:26:36.725 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.prefix_extractor: nullptr 2024-09-18T07:26:36.725 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-18T07:26:36.725 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.num_levels: 7 2024-09-18T07:26:36.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-18T07:26:36.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-18T07:26:36.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-18T07:26:36.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-18T07:26:36.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-18T07:26:36.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-18T07:26:36.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-18T07:26:36.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-18T07:26:36.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-18T07:26:36.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-18T07:26:36.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-18T07:26:36.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-18T07:26:36.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-18T07:26:36.726 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compression_opts.level: 32767 2024-09-18T07:26:36.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compression_opts.strategy: 0 2024-09-18T07:26:36.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-18T07:26:36.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-18T07:26:36.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-18T07:26:36.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-18T07:26:36.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compression_opts.enabled: false 2024-09-18T07:26:36.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-18T07:26:36.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-18T07:26:36.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-18T07:26:36.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-18T07:26:36.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.target_file_size_base: 67108864 2024-09-18T07:26:36.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-18T07:26:36.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-18T07:26:36.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-18T07:26:36.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-18T07:26:36.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-18T07:26:36.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-18T07:26:36.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-18T07:26:36.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-18T07:26:36.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-18T07:26:36.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-18T07:26:36.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-18T07:26:36.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-18T07:26:36.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-18T07:26:36.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-18T07:26:36.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.arena_block_size: 1048576 2024-09-18T07:26:36.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-18T07:26:36.729 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-18T07:26:36.729 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.disable_auto_compactions: 0 2024-09-18T07:26:36.729 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-18T07:26:36.729 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-18T07:26:36.729 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-18T07:26:36.729 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-18T07:26:36.729 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-18T07:26:36.729 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-18T07:26:36.729 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-18T07:26:36.729 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-18T07:26:36.729 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-18T07:26:36.729 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-18T07:26:36.729 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-18T07:26:36.730 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.inplace_update_support: 0 2024-09-18T07:26:36.730 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-18T07:26:36.730 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-18T07:26:36.730 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-18T07:26:36.730 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-18T07:26:36.730 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.bloom_locality: 0 2024-09-18T07:26:36.730 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.max_successive_merges: 0 2024-09-18T07:26:36.730 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-18T07:26:36.730 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.paranoid_file_checks: 0 2024-09-18T07:26:36.730 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.force_consistency_checks: 1 2024-09-18T07:26:36.730 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.report_bg_io_stats: 0 2024-09-18T07:26:36.730 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.ttl: 2592000 2024-09-18T07:26:36.730 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-18T07:26:36.730 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-18T07:26:36.731 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-18T07:26:36.731 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.enable_blob_files: false 2024-09-18T07:26:36.731 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.min_blob_size: 0 2024-09-18T07:26:36.731 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.blob_file_size: 268435456 2024-09-18T07:26:36.731 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-18T07:26:36.731 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-18T07:26:36.731 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-18T07:26:36.731 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-18T07:26:36.731 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-18T07:26:36.731 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.blob_file_starting_level: 0 2024-09-18T07:26:36.732 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-18T07:26:36.732 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-18T07:26:36.732 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-18T07:26:36.732 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 72e2c698-9821-43ac-95bf-1d79f8e48510 2024-09-18T07:26:36.732 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726644396365715, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-18T07:26:36.732 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-18T07:26:36.732 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726644396366141, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1613, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726644396, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "72e2c698-9821-43ac-95bf-1d79f8e48510", "db_session_id": "2OI65DZ60F8U8YH68DP3", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-18T07:26:36.732 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726644396366238, "job": 1, "event": "recovery_finished"} 2024-09-18T07:26:36.732 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-18T07:26:36.732 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-18T07:26:36.732 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e493980e00 2024-09-18T07:26:36.732 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: DB pointer 0x55e493a82000 2024-09-18T07:26:36.732 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-18T07:26:36.733 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-18T07:26:36.733 INFO:journalctl@ceph.mon.b.smithi203.stdout: ** DB Stats ** 2024-09-18T07:26:36.733 INFO:journalctl@ceph.mon.b.smithi203.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-18T07:26:36.733 INFO:journalctl@ceph.mon.b.smithi203.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-18T07:26:36.733 INFO:journalctl@ceph.mon.b.smithi203.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-18T07:26:36.733 INFO:journalctl@ceph.mon.b.smithi203.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-18T07:26:36.733 INFO:journalctl@ceph.mon.b.smithi203.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-18T07:26:36.733 INFO:journalctl@ceph.mon.b.smithi203.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-18T07:26:36.733 INFO:journalctl@ceph.mon.b.smithi203.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-18T07:26:36.733 INFO:journalctl@ceph.mon.b.smithi203.stdout: 2024-09-18T07:26:36.733 INFO:journalctl@ceph.mon.b.smithi203.stdout: ** Compaction Stats [default] ** 2024-09-18T07:26:36.733 INFO:journalctl@ceph.mon.b.smithi203.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-18T07:26:36.733 INFO:journalctl@ceph.mon.b.smithi203.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-18T07:26:36.733 INFO:journalctl@ceph.mon.b.smithi203.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-18T07:26:36.734 INFO:journalctl@ceph.mon.b.smithi203.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-18T07:26:36.734 INFO:journalctl@ceph.mon.b.smithi203.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-18T07:26:36.734 INFO:journalctl@ceph.mon.b.smithi203.stdout: 2024-09-18T07:26:36.734 INFO:journalctl@ceph.mon.b.smithi203.stdout: ** Compaction Stats [default] ** 2024-09-18T07:26:36.734 INFO:journalctl@ceph.mon.b.smithi203.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-18T07:26:36.734 INFO:journalctl@ceph.mon.b.smithi203.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-18T07:26:36.734 INFO:journalctl@ceph.mon.b.smithi203.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-18T07:26:36.734 INFO:journalctl@ceph.mon.b.smithi203.stdout: 2024-09-18T07:26:36.734 INFO:journalctl@ceph.mon.b.smithi203.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-18T07:26:36.734 INFO:journalctl@ceph.mon.b.smithi203.stdout: 2024-09-18T07:26:36.734 INFO:journalctl@ceph.mon.b.smithi203.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-18T07:26:36.734 INFO:journalctl@ceph.mon.b.smithi203.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-18T07:26:36.734 INFO:journalctl@ceph.mon.b.smithi203.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-18T07:26:36.735 INFO:journalctl@ceph.mon.b.smithi203.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-18T07:26:36.735 INFO:journalctl@ceph.mon.b.smithi203.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-18T07:26:36.735 INFO:journalctl@ceph.mon.b.smithi203.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-18T07:26:36.735 INFO:journalctl@ceph.mon.b.smithi203.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-18T07:26:36.735 INFO:journalctl@ceph.mon.b.smithi203.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-18T07:26:36.735 INFO:journalctl@ceph.mon.b.smithi203.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-18T07:26:36.735 INFO:journalctl@ceph.mon.b.smithi203.stdout: Block cache BinnedLRUCache@0x55e493951610#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-18T07:26:36.735 INFO:journalctl@ceph.mon.b.smithi203.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-18T07:26:36.735 INFO:journalctl@ceph.mon.b.smithi203.stdout: 2024-09-18T07:26:36.735 INFO:journalctl@ceph.mon.b.smithi203.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-18T07:26:36.735 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-18T07:26:36.735 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: using public_addrv v1:172.21.15.203:6789/0 2024-09-18T07:26:36.735 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: starting mon.b rank -1 at public addrs v1:172.21.15.203:6789/0 at bind addrs v1:172.21.15.203:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:26:36.736 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: mon.b@-1(???) e0 preinit fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:26:36.736 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: mon.b@-1(synchronizing).mds e1 new map 2024-09-18T07:26:36.736 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-18T07:26:36.736 INFO:journalctl@ceph.mon.b.smithi203.stdout: e1 2024-09-18T07:26:36.736 INFO:journalctl@ceph.mon.b.smithi203.stdout: btime 2024-09-18T07:25:19:662155+0000 2024-09-18T07:26:36.736 INFO:journalctl@ceph.mon.b.smithi203.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-18T07:26:36.736 INFO:journalctl@ceph.mon.b.smithi203.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-18T07:26:36.736 INFO:journalctl@ceph.mon.b.smithi203.stdout: legacy client fscid: -1 2024-09-18T07:26:36.736 INFO:journalctl@ceph.mon.b.smithi203.stdout: 2024-09-18T07:26:36.736 INFO:journalctl@ceph.mon.b.smithi203.stdout: No filesystems configured 2024-09-18T07:26:36.736 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-18T07:26:36.736 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-18T07:26:36.736 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-18T07:26:36.736 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-18T07:26:36.737 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-18T07:26:36.737 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-18T07:26:36.737 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-18T07:26:36.737 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-18T07:26:36.737 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-18T07:26:36.737 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-18T07:26:36.737 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:36.737 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='client.14182 v1:172.21.15.137:0/1328863449' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:26:36.737 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:36.737 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4239655369' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-18T07:26:36.737 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4239655369' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-18T07:26:36.737 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:26:36.737 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:36.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:36.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:36.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:36.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"} : dispatch 2024-09-18T07:26:36.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:36.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:26:36.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: Updating smithi203:/etc/ceph/ceph.conf 2024-09-18T07:26:36.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: Updating smithi203:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:26:36.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:26:36.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: Updating smithi203:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T07:26:36.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:36.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:36.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:36.738 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:36.739 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:36.739 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:26:36.739 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:36.739 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:26:36.739 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:36.739 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:26:36.739 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:36.739 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='client.14186 v1:172.21.15.203:0/823080327' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi137:[v1:172.21.15.137:6789]=a;smithi137:[v1:172.21.15.137:6790]=c;smithi203:[v1:172.21.15.203:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:26:36.739 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: Saving service mon spec with placement smithi137:[v1:172.21.15.137:6789]=a;smithi137:[v1:172.21.15.137:6790]=c;smithi203:[v1:172.21.15.203:6789]=b;count:3 2024-09-18T07:26:36.739 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: Deploying daemon mon.b on smithi203 2024-09-18T07:26:36.739 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:36.739 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.203:0/1352026863' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:26:36.739 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:36.739 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.203:0/2421693283' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:26:36.740 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 ceph-mon[32237]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-18T07:26:36.740 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 bash[32205]: b63ad8ee863c70070b7130c5119f6279c2f73077d55481306af42c31719b243d 2024-09-18T07:26:36.740 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:36 smithi203 systemd[1]: Started Ceph mon.b for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:26:37.926 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-18T07:26:37.926 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-18T07:26:38.184 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:26:41.667 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:26:41.668 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":2,"fsid":"f7ef099c-758e-11ef-bceb-c7b262605968","modified":"2024-09-18T07:26:36.387788Z","created":"2024-09-18T07:25:15.819392Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-18T07:26:41.668 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 2 2024-09-18T07:26:41.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:26:41.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: mon.a calling monitor election 2024-09-18T07:26:41.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:41.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:41.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:41.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:41.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: mon.b calling monitor election 2024-09-18T07:26:41.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:41.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:41.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:41.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:41.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:41.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-18T07:26:41.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: monmap epoch 2 2024-09-18T07:26:41.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:26:41.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: last_changed 2024-09-18T07:26:36.387788+0000 2024-09-18T07:26:41.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: created 2024-09-18T07:25:15.819392+0000 2024-09-18T07:26:41.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: min_mon_release 19 (squid) 2024-09-18T07:26:41.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: election_strategy: 1 2024-09-18T07:26:41.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-18T07:26:41.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: 1: v1:172.21.15.203:6789/0 mon.b 2024-09-18T07:26:41.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: fsmap 2024-09-18T07:26:41.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:26:41.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: mgrmap e13: y(active, since 34s) 2024-09-18T07:26:41.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: overall HEALTH_OK 2024-09-18T07:26:41.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:41.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:41.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:41.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:26:41.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:41 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:41.809 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:26:41.810 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: mon.a calling monitor election 2024-09-18T07:26:41.810 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:41.810 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:41.810 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:41.810 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:41.810 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: mon.b calling monitor election 2024-09-18T07:26:41.810 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:41.810 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:41.810 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:41.810 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:41.810 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:41.810 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-18T07:26:41.810 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: monmap epoch 2 2024-09-18T07:26:41.811 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:26:41.811 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: last_changed 2024-09-18T07:26:36.387788+0000 2024-09-18T07:26:41.811 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: created 2024-09-18T07:25:15.819392+0000 2024-09-18T07:26:41.811 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: min_mon_release 19 (squid) 2024-09-18T07:26:41.811 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: election_strategy: 1 2024-09-18T07:26:41.811 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-18T07:26:41.811 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: 1: v1:172.21.15.203:6789/0 mon.b 2024-09-18T07:26:41.811 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: fsmap 2024-09-18T07:26:41.811 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:26:41.811 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: mgrmap e13: y(active, since 34s) 2024-09-18T07:26:41.811 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: overall HEALTH_OK 2024-09-18T07:26:41.811 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:41.811 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:41.812 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:41.812 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:26:41.812 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:41 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:42.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:42 smithi203 ceph-mon[32237]: Deploying daemon mon.c on smithi137 2024-09-18T07:26:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:42 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.203:0/1013688470' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:26:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:42 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:42.808 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:42 smithi137 ceph-mon[29408]: Deploying daemon mon.c on smithi137 2024-09-18T07:26:42.809 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:42 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.203:0/1013688470' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:26:42.809 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:42 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:43.225 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-18T07:26:43.225 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-18T07:26:43.482 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:26:43.602 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:43 smithi203 ceph-mon[32237]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:43.748 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:43 smithi137 ceph-mon[29408]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:44.167 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:26:44.167 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":2,"fsid":"f7ef099c-758e-11ef-bceb-c7b262605968","modified":"2024-09-18T07:26:36.387788Z","created":"2024-09-18T07:25:15.819392Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-18T07:26:44.167 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 2 2024-09-18T07:26:44.393 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 systemd[1]: Starting Ceph mon.c for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:26:44.662 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.203:0/3582725168' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:26:44.663 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 podman[39156]: 2024-09-18 07:26:44.391325636 +0000 UTC m=+0.098355179 container create 46d7d7550bc469526b0b5bd656c23b4b5dbd40c2b9c4e05aa6b09069b70ab5bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-18T07:26:44.663 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 podman[39156]: 2024-09-18 07:26:44.308635579 +0000 UTC m=+0.015665127 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:26:44.663 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 podman[39156]: 2024-09-18 07:26:44.635486438 +0000 UTC m=+0.342515986 container init 46d7d7550bc469526b0b5bd656c23b4b5dbd40c2b9c4e05aa6b09069b70ab5bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-c, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-18T07:26:44.663 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 podman[39156]: 2024-09-18 07:26:44.639127983 +0000 UTC m=+0.346157523 container start 46d7d7550bc469526b0b5bd656c23b4b5dbd40c2b9c4e05aa6b09069b70ab5bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5) 2024-09-18T07:26:44.663 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: set uid:gid to 167:167 (ceph:ceph) 2024-09-18T07:26:44.663 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: ceph version 19.3.0-4983-ge7fb7b56 (e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e) squid (dev), process ceph-mon, pid 2 2024-09-18T07:26:44.663 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: pidfile_write: ignore empty --pid-file 2024-09-18T07:26:44.704 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:44 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.203:0/3582725168' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:26:44.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: load: jerasure load: lrc 2024-09-18T07:26:44.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: RocksDB version: 7.9.2 2024-09-18T07:26:44.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Git sha 0 2024-09-18T07:26:44.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Compile date 2024-09-17 15:59:30 2024-09-18T07:26:44.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: DB SUMMARY 2024-09-18T07:26:44.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: DB Session ID: Z4QDG728C30MXY409GK6 2024-09-18T07:26:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: CURRENT file: CURRENT 2024-09-18T07:26:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: IDENTITY file: IDENTITY 2024-09-18T07:26:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-18T07:26:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-18T07:26:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 566 ; 2024-09-18T07:26:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.error_if_exists: 0 2024-09-18T07:26:44.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.create_if_missing: 0 2024-09-18T07:26:44.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.paranoid_checks: 1 2024-09-18T07:26:44.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-18T07:26:44.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-18T07:26:44.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-18T07:26:44.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.env: 0x55803d43e120 2024-09-18T07:26:44.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.fs: PosixFileSystem 2024-09-18T07:26:44.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.info_log: 0x55803e2156e0 2024-09-18T07:26:44.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_file_opening_threads: 16 2024-09-18T07:26:44.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.statistics: (nil) 2024-09-18T07:26:44.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.use_fsync: 0 2024-09-18T07:26:44.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_log_file_size: 0 2024-09-18T07:26:44.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-18T07:26:44.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-18T07:26:44.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.keep_log_file_num: 1000 2024-09-18T07:26:44.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.recycle_log_file_num: 0 2024-09-18T07:26:44.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.allow_fallocate: 1 2024-09-18T07:26:44.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.allow_mmap_reads: 0 2024-09-18T07:26:44.926 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.allow_mmap_writes: 0 2024-09-18T07:26:44.926 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.use_direct_reads: 0 2024-09-18T07:26:44.926 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-18T07:26:44.926 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.create_missing_column_families: 0 2024-09-18T07:26:44.926 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.db_log_dir: 2024-09-18T07:26:44.926 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.wal_dir: 2024-09-18T07:26:44.927 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-18T07:26:44.927 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-18T07:26:44.927 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-18T07:26:44.927 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-18T07:26:44.930 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-18T07:26:44.930 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-18T07:26:44.931 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.advise_random_on_open: 1 2024-09-18T07:26:44.931 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.db_write_buffer_size: 0 2024-09-18T07:26:44.931 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.write_buffer_manager: 0x55803e219b80 2024-09-18T07:26:44.931 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-18T07:26:44.931 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-18T07:26:44.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-18T07:26:44.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.rate_limiter: (nil) 2024-09-18T07:26:44.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-18T07:26:44.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.wal_recovery_mode: 2 2024-09-18T07:26:44.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.enable_thread_tracking: 0 2024-09-18T07:26:44.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.enable_pipelined_write: 0 2024-09-18T07:26:44.933 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.unordered_write: 0 2024-09-18T07:26:44.933 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-18T07:26:44.933 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-18T07:26:44.933 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-18T07:26:44.933 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-18T07:26:44.933 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.row_cache: None 2024-09-18T07:26:44.934 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.wal_filter: None 2024-09-18T07:26:44.934 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-18T07:26:44.934 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.allow_ingest_behind: 0 2024-09-18T07:26:44.934 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.two_write_queues: 0 2024-09-18T07:26:44.934 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.manual_wal_flush: 0 2024-09-18T07:26:44.934 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.wal_compression: 0 2024-09-18T07:26:44.935 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.atomic_flush: 0 2024-09-18T07:26:44.935 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-18T07:26:44.935 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-18T07:26:44.935 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-18T07:26:44.935 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.log_readahead_size: 0 2024-09-18T07:26:44.935 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-18T07:26:44.936 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.best_efforts_recovery: 0 2024-09-18T07:26:44.936 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-18T07:26:44.936 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-18T07:26:44.936 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.allow_data_in_errors: 0 2024-09-18T07:26:44.936 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.db_host_id: __hostname__ 2024-09-18T07:26:44.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-18T07:26:44.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_background_jobs: 2 2024-09-18T07:26:44.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_background_compactions: -1 2024-09-18T07:26:44.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_subcompactions: 1 2024-09-18T07:26:44.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-18T07:26:44.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-18T07:26:44.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-18T07:26:44.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_total_wal_size: 0 2024-09-18T07:26:44.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-18T07:26:44.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-18T07:26:44.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-18T07:26:44.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-18T07:26:44.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_open_files: -1 2024-09-18T07:26:44.939 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.bytes_per_sync: 0 2024-09-18T07:26:44.939 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-18T07:26:44.939 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-18T07:26:44.939 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compaction_readahead_size: 0 2024-09-18T07:26:44.939 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_background_flushes: -1 2024-09-18T07:26:44.939 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Compression algorithms supported: 2024-09-18T07:26:44.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: kZSTD supported: 0 2024-09-18T07:26:44.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: kXpressCompression supported: 0 2024-09-18T07:26:44.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: kBZip2Compression supported: 0 2024-09-18T07:26:44.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-18T07:26:44.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: kLZ4Compression supported: 1 2024-09-18T07:26:44.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: kZlibCompression supported: 1 2024-09-18T07:26:44.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: kLZ4HCCompression supported: 1 2024-09-18T07:26:44.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: kSnappyCompression supported: 1 2024-09-18T07:26:44.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-18T07:26:44.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-18T07:26:44.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-18T07:26:44.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-18T07:26:44.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-18T07:26:44.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.merge_operator: 2024-09-18T07:26:44.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compaction_filter: None 2024-09-18T07:26:44.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compaction_filter_factory: None 2024-09-18T07:26:44.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.sst_partitioner_factory: None 2024-09-18T07:26:44.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-18T07:26:44.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-18T07:26:44.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55803e215300) 2024-09-18T07:26:44.943 INFO:journalctl@ceph.mon.c.smithi137.stdout: cache_index_and_filter_blocks: 1 2024-09-18T07:26:44.943 INFO:journalctl@ceph.mon.c.smithi137.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-18T07:26:44.943 INFO:journalctl@ceph.mon.c.smithi137.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-18T07:26:44.943 INFO:journalctl@ceph.mon.c.smithi137.stdout: pin_top_level_index_and_filter: 1 2024-09-18T07:26:44.943 INFO:journalctl@ceph.mon.c.smithi137.stdout: index_type: 0 2024-09-18T07:26:44.944 INFO:journalctl@ceph.mon.c.smithi137.stdout: data_block_index_type: 0 2024-09-18T07:26:44.944 INFO:journalctl@ceph.mon.c.smithi137.stdout: index_shortening: 1 2024-09-18T07:26:44.944 INFO:journalctl@ceph.mon.c.smithi137.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-18T07:26:44.944 INFO:journalctl@ceph.mon.c.smithi137.stdout: checksum: 4 2024-09-18T07:26:44.944 INFO:journalctl@ceph.mon.c.smithi137.stdout: no_block_cache: 0 2024-09-18T07:26:44.944 INFO:journalctl@ceph.mon.c.smithi137.stdout: block_cache: 0x55803e209610 2024-09-18T07:26:44.945 INFO:journalctl@ceph.mon.c.smithi137.stdout: block_cache_name: BinnedLRUCache 2024-09-18T07:26:44.945 INFO:journalctl@ceph.mon.c.smithi137.stdout: block_cache_options: 2024-09-18T07:26:44.945 INFO:journalctl@ceph.mon.c.smithi137.stdout: capacity : 536870912 2024-09-18T07:26:44.945 INFO:journalctl@ceph.mon.c.smithi137.stdout: num_shard_bits : 4 2024-09-18T07:26:44.945 INFO:journalctl@ceph.mon.c.smithi137.stdout: strict_capacity_limit : 0 2024-09-18T07:26:44.945 INFO:journalctl@ceph.mon.c.smithi137.stdout: high_pri_pool_ratio: 0.000 2024-09-18T07:26:44.946 INFO:journalctl@ceph.mon.c.smithi137.stdout: block_cache_compressed: (nil) 2024-09-18T07:26:44.946 INFO:journalctl@ceph.mon.c.smithi137.stdout: persistent_cache: (nil) 2024-09-18T07:26:44.946 INFO:journalctl@ceph.mon.c.smithi137.stdout: block_size: 4096 2024-09-18T07:26:44.946 INFO:journalctl@ceph.mon.c.smithi137.stdout: block_size_deviation: 10 2024-09-18T07:26:44.946 INFO:journalctl@ceph.mon.c.smithi137.stdout: block_restart_interval: 16 2024-09-18T07:26:44.946 INFO:journalctl@ceph.mon.c.smithi137.stdout: index_block_restart_interval: 1 2024-09-18T07:26:44.946 INFO:journalctl@ceph.mon.c.smithi137.stdout: metadata_block_size: 4096 2024-09-18T07:26:44.947 INFO:journalctl@ceph.mon.c.smithi137.stdout: partition_filters: 0 2024-09-18T07:26:44.947 INFO:journalctl@ceph.mon.c.smithi137.stdout: use_delta_encoding: 1 2024-09-18T07:26:44.947 INFO:journalctl@ceph.mon.c.smithi137.stdout: filter_policy: bloomfilter 2024-09-18T07:26:44.947 INFO:journalctl@ceph.mon.c.smithi137.stdout: whole_key_filtering: 1 2024-09-18T07:26:44.947 INFO:journalctl@ceph.mon.c.smithi137.stdout: verify_compression: 0 2024-09-18T07:26:44.948 INFO:journalctl@ceph.mon.c.smithi137.stdout: read_amp_bytes_per_bit: 0 2024-09-18T07:26:44.948 INFO:journalctl@ceph.mon.c.smithi137.stdout: format_version: 5 2024-09-18T07:26:44.948 INFO:journalctl@ceph.mon.c.smithi137.stdout: enable_index_compression: 1 2024-09-18T07:26:44.948 INFO:journalctl@ceph.mon.c.smithi137.stdout: block_align: 0 2024-09-18T07:26:44.948 INFO:journalctl@ceph.mon.c.smithi137.stdout: max_auto_readahead_size: 262144 2024-09-18T07:26:44.948 INFO:journalctl@ceph.mon.c.smithi137.stdout: prepopulate_block_cache: 0 2024-09-18T07:26:44.948 INFO:journalctl@ceph.mon.c.smithi137.stdout: initial_auto_readahead_size: 8192 2024-09-18T07:26:44.949 INFO:journalctl@ceph.mon.c.smithi137.stdout: num_file_reads_for_auto_readahead: 2 2024-09-18T07:26:44.949 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.write_buffer_size: 33554432 2024-09-18T07:26:44.949 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_write_buffer_number: 2 2024-09-18T07:26:44.949 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compression: NoCompression 2024-09-18T07:26:44.949 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.bottommost_compression: Disabled 2024-09-18T07:26:44.949 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.prefix_extractor: nullptr 2024-09-18T07:26:44.950 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-18T07:26:44.950 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.num_levels: 7 2024-09-18T07:26:44.950 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-18T07:26:44.950 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-18T07:26:44.950 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-18T07:26:44.950 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-18T07:26:44.950 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-18T07:26:44.950 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-18T07:26:44.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-18T07:26:44.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-18T07:26:44.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-18T07:26:44.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-18T07:26:44.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-18T07:26:44.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-18T07:26:44.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-18T07:26:44.952 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compression_opts.level: 32767 2024-09-18T07:26:44.952 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compression_opts.strategy: 0 2024-09-18T07:26:44.952 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-18T07:26:44.952 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-18T07:26:44.952 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-18T07:26:44.952 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-18T07:26:44.953 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compression_opts.enabled: false 2024-09-18T07:26:44.953 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-18T07:26:44.953 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-18T07:26:44.953 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-18T07:26:44.953 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-18T07:26:44.953 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.target_file_size_base: 67108864 2024-09-18T07:26:44.953 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-18T07:26:44.954 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-18T07:26:44.954 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-18T07:26:44.954 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-18T07:26:44.954 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-18T07:26:44.954 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-18T07:26:44.954 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-18T07:26:44.954 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-18T07:26:44.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-18T07:26:44.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-18T07:26:44.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-18T07:26:44.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-18T07:26:44.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-18T07:26:44.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-18T07:26:44.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.arena_block_size: 1048576 2024-09-18T07:26:44.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-18T07:26:44.956 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-18T07:26:44.956 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.disable_auto_compactions: 0 2024-09-18T07:26:44.956 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-18T07:26:44.956 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-18T07:26:44.956 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-18T07:26:44.956 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-18T07:26:44.956 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-18T07:26:44.957 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-18T07:26:44.957 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-18T07:26:44.957 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-18T07:26:44.957 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-18T07:26:44.957 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-18T07:26:44.957 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-18T07:26:44.958 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.inplace_update_support: 0 2024-09-18T07:26:44.958 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-18T07:26:44.958 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-18T07:26:44.958 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-18T07:26:44.958 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-18T07:26:44.958 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.bloom_locality: 0 2024-09-18T07:26:44.958 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.max_successive_merges: 0 2024-09-18T07:26:44.959 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-18T07:26:44.959 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.paranoid_file_checks: 0 2024-09-18T07:26:44.959 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.force_consistency_checks: 1 2024-09-18T07:26:44.959 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.report_bg_io_stats: 0 2024-09-18T07:26:44.959 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.ttl: 2592000 2024-09-18T07:26:44.959 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-18T07:26:44.959 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-18T07:26:44.960 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-18T07:26:44.960 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.enable_blob_files: false 2024-09-18T07:26:44.960 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.min_blob_size: 0 2024-09-18T07:26:44.960 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.blob_file_size: 268435456 2024-09-18T07:26:44.960 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-18T07:26:44.960 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-18T07:26:44.960 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-18T07:26:44.961 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-18T07:26:44.961 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-18T07:26:44.961 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.blob_file_starting_level: 0 2024-09-18T07:26:44.961 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-18T07:26:44.961 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-18T07:26:44.961 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-18T07:26:44.961 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7641da1c-fc63-42b5-b6e7-38283d9681c6 2024-09-18T07:26:44.962 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726644404666379, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-18T07:26:44.962 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-18T07:26:44.962 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726644404666918, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1703, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726644404, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7641da1c-fc63-42b5-b6e7-38283d9681c6", "db_session_id": "Z4QDG728C30MXY409GK6", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-18T07:26:44.962 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726644404667052, "job": 1, "event": "recovery_finished"} 2024-09-18T07:26:44.962 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-18T07:26:44.962 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-18T07:26:44.963 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55803e238e00 2024-09-18T07:26:44.963 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: DB pointer 0x55803e33a000 2024-09-18T07:26:44.963 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-18T07:26:44.963 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-18T07:26:44.963 INFO:journalctl@ceph.mon.c.smithi137.stdout: ** DB Stats ** 2024-09-18T07:26:44.963 INFO:journalctl@ceph.mon.c.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-18T07:26:44.963 INFO:journalctl@ceph.mon.c.smithi137.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-18T07:26:44.964 INFO:journalctl@ceph.mon.c.smithi137.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-18T07:26:44.964 INFO:journalctl@ceph.mon.c.smithi137.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-18T07:26:44.964 INFO:journalctl@ceph.mon.c.smithi137.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-18T07:26:44.964 INFO:journalctl@ceph.mon.c.smithi137.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-18T07:26:44.964 INFO:journalctl@ceph.mon.c.smithi137.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-18T07:26:44.964 INFO:journalctl@ceph.mon.c.smithi137.stdout: 2024-09-18T07:26:44.964 INFO:journalctl@ceph.mon.c.smithi137.stdout: ** Compaction Stats [default] ** 2024-09-18T07:26:44.965 INFO:journalctl@ceph.mon.c.smithi137.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-18T07:26:44.965 INFO:journalctl@ceph.mon.c.smithi137.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-18T07:26:44.965 INFO:journalctl@ceph.mon.c.smithi137.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-18T07:26:44.965 INFO:journalctl@ceph.mon.c.smithi137.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-18T07:26:44.965 INFO:journalctl@ceph.mon.c.smithi137.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-18T07:26:44.965 INFO:journalctl@ceph.mon.c.smithi137.stdout: 2024-09-18T07:26:44.965 INFO:journalctl@ceph.mon.c.smithi137.stdout: ** Compaction Stats [default] ** 2024-09-18T07:26:44.966 INFO:journalctl@ceph.mon.c.smithi137.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-18T07:26:44.966 INFO:journalctl@ceph.mon.c.smithi137.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-18T07:26:44.966 INFO:journalctl@ceph.mon.c.smithi137.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-18T07:26:44.966 INFO:journalctl@ceph.mon.c.smithi137.stdout: 2024-09-18T07:26:44.966 INFO:journalctl@ceph.mon.c.smithi137.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-18T07:26:44.966 INFO:journalctl@ceph.mon.c.smithi137.stdout: 2024-09-18T07:26:44.966 INFO:journalctl@ceph.mon.c.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-18T07:26:44.967 INFO:journalctl@ceph.mon.c.smithi137.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-18T07:26:44.967 INFO:journalctl@ceph.mon.c.smithi137.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-18T07:26:44.967 INFO:journalctl@ceph.mon.c.smithi137.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-18T07:26:44.967 INFO:journalctl@ceph.mon.c.smithi137.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-18T07:26:44.967 INFO:journalctl@ceph.mon.c.smithi137.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-18T07:26:44.967 INFO:journalctl@ceph.mon.c.smithi137.stdout: Cumulative compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-18T07:26:44.967 INFO:journalctl@ceph.mon.c.smithi137.stdout: Interval compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-18T07:26:44.968 INFO:journalctl@ceph.mon.c.smithi137.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-18T07:26:44.968 INFO:journalctl@ceph.mon.c.smithi137.stdout: Block cache BinnedLRUCache@0x55803e209610#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-18T07:26:44.968 INFO:journalctl@ceph.mon.c.smithi137.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-18T07:26:44.968 INFO:journalctl@ceph.mon.c.smithi137.stdout: 2024-09-18T07:26:44.968 INFO:journalctl@ceph.mon.c.smithi137.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-18T07:26:44.968 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-18T07:26:44.968 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: using public_addrv v1:172.21.15.137:6790/0 2024-09-18T07:26:44.969 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: starting mon.c rank -1 at public addrs v1:172.21.15.137:6790/0 at bind addrs v1:172.21.15.137:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:26:44.969 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.c@-1(???) e0 preinit fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:26:44.969 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.c@-1(synchronizing).mds e1 new map 2024-09-18T07:26:44.969 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-18T07:26:44.969 INFO:journalctl@ceph.mon.c.smithi137.stdout: e1 2024-09-18T07:26:44.969 INFO:journalctl@ceph.mon.c.smithi137.stdout: btime 2024-09-18T07:25:19:662155+0000 2024-09-18T07:26:44.969 INFO:journalctl@ceph.mon.c.smithi137.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-18T07:26:44.970 INFO:journalctl@ceph.mon.c.smithi137.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-18T07:26:44.970 INFO:journalctl@ceph.mon.c.smithi137.stdout: legacy client fscid: -1 2024-09-18T07:26:44.970 INFO:journalctl@ceph.mon.c.smithi137.stdout: 2024-09-18T07:26:44.970 INFO:journalctl@ceph.mon.c.smithi137.stdout: No filesystems configured 2024-09-18T07:26:44.970 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-18T07:26:44.970 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-18T07:26:44.970 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-18T07:26:44.971 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-18T07:26:44.971 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-18T07:26:44.971 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-18T07:26:44.971 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-18T07:26:44.971 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-18T07:26:44.971 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-18T07:26:44.971 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-18T07:26:44.972 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:44.972 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='client.14182 v1:172.21.15.137:0/1328863449' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:26:44.972 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:44.972 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4239655369' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-18T07:26:44.972 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4239655369' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-18T07:26:44.972 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:26:44.972 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:44.972 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:44.972 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:44.972 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:44.972 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"} : dispatch 2024-09-18T07:26:44.972 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:44.972 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:26:44.973 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: Updating smithi203:/etc/ceph/ceph.conf 2024-09-18T07:26:44.973 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: Updating smithi203:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:26:44.973 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:26:44.973 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: Updating smithi203:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T07:26:44.973 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:44.973 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:44.973 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:44.973 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:44.973 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:44.973 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:26:44.973 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:44.973 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:26:44.973 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:44.974 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:26:44.974 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:44.974 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='client.14186 v1:172.21.15.203:0/823080327' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi137:[v1:172.21.15.137:6789]=a;smithi137:[v1:172.21.15.137:6790]=c;smithi203:[v1:172.21.15.203:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:26:44.974 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: Saving service mon spec with placement smithi137:[v1:172.21.15.137:6789]=a;smithi137:[v1:172.21.15.137:6790]=c;smithi203:[v1:172.21.15.203:6789]=b;count:3 2024-09-18T07:26:44.974 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: Deploying daemon mon.b on smithi203 2024-09-18T07:26:44.974 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:44.974 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.203:0/1352026863' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:26:44.974 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:44.974 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.203:0/2421693283' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:26:44.974 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:26:44.974 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.a calling monitor election 2024-09-18T07:26:44.974 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:44.974 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:44.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:44.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:44.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.b calling monitor election 2024-09-18T07:26:44.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:44.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:44.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:44.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:44.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:44.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-18T07:26:44.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: monmap epoch 2 2024-09-18T07:26:44.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:26:44.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: last_changed 2024-09-18T07:26:36.387788+0000 2024-09-18T07:26:44.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: created 2024-09-18T07:25:15.819392+0000 2024-09-18T07:26:44.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: min_mon_release 19 (squid) 2024-09-18T07:26:44.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: election_strategy: 1 2024-09-18T07:26:44.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-18T07:26:44.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: 1: v1:172.21.15.203:6789/0 mon.b 2024-09-18T07:26:44.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: fsmap 2024-09-18T07:26:44.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:26:44.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mgrmap e13: y(active, since 34s) 2024-09-18T07:26:44.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: overall HEALTH_OK 2024-09-18T07:26:44.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:44.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:44.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:44.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:26:44.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:44.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: Deploying daemon mon.c on smithi137 2024-09-18T07:26:44.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.203:0/1013688470' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:26:44.977 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:44.977 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:44.977 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.203:0/3582725168' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:26:44.977 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 ceph-mon[39175]: mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-09-18T07:26:44.977 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 bash[39156]: 46d7d7550bc469526b0b5bd656c23b4b5dbd40c2b9c4e05aa6b09069b70ab5bf 2024-09-18T07:26:44.977 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:44 smithi137 systemd[1]: Started Ceph mon.c for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:26:45.732 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-18T07:26:45.732 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-18T07:26:45.988 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:26:49.975 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:26:49.975 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":3,"fsid":"f7ef099c-758e-11ef-bceb-c7b262605968","modified":"2024-09-18T07:26:44.690052Z","created":"2024-09-18T07:25:15.819392Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6790","nonce":0}]},"addr":"172.21.15.137:6790/0","public_addr":"172.21.15.137:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-18T07:26:49.976 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 3 2024-09-18T07:26:50.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:26:50.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: mon.a calling monitor election 2024-09-18T07:26:50.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: mon.b calling monitor election 2024-09-18T07:26:50.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:50.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:50.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:50.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:50.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:50.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:50.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:50.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:50.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:50.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-18T07:26:50.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: monmap epoch 3 2024-09-18T07:26:50.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:26:50.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: last_changed 2024-09-18T07:26:44.690052+0000 2024-09-18T07:26:50.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: created 2024-09-18T07:25:15.819392+0000 2024-09-18T07:26:50.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: min_mon_release 19 (squid) 2024-09-18T07:26:50.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: election_strategy: 1 2024-09-18T07:26:50.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-18T07:26:50.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: 1: v1:172.21.15.203:6789/0 mon.b 2024-09-18T07:26:50.172 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: 2: v1:172.21.15.137:6790/0 mon.c 2024-09-18T07:26:50.172 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: fsmap 2024-09-18T07:26:50.172 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:26:50.172 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: mgrmap e13: y(active, since 43s) 2024-09-18T07:26:50.172 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-18T07:26:50.172 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-18T07:26:50.172 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-18T07:26:50.172 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: mon.c (rank 2) addr v1:172.21.15.137:6790/0 is down (out of quorum) 2024-09-18T07:26:50.172 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:50.172 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:50.172 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:50.172 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:49 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:26:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: mon.a calling monitor election 2024-09-18T07:26:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: mon.b calling monitor election 2024-09-18T07:26:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:50.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:50.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:50.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:50.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:50.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:50.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:50.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:50.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:50.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-18T07:26:50.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: monmap epoch 3 2024-09-18T07:26:50.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:26:50.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: last_changed 2024-09-18T07:26:44.690052+0000 2024-09-18T07:26:50.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: created 2024-09-18T07:25:15.819392+0000 2024-09-18T07:26:50.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: min_mon_release 19 (squid) 2024-09-18T07:26:50.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: election_strategy: 1 2024-09-18T07:26:50.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-18T07:26:50.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: 1: v1:172.21.15.203:6789/0 mon.b 2024-09-18T07:26:50.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: 2: v1:172.21.15.137:6790/0 mon.c 2024-09-18T07:26:50.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: fsmap 2024-09-18T07:26:50.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:26:50.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: mgrmap e13: y(active, since 43s) 2024-09-18T07:26:50.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-18T07:26:50.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-18T07:26:50.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-18T07:26:50.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: mon.c (rank 2) addr v1:172.21.15.137:6790/0 is down (out of quorum) 2024-09-18T07:26:50.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:50.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:50.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:50.214 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:49 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:50.514 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-18T07:26:50.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-18T07:26:50.871 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:26:51.025 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:50 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:26:51.025 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:50 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.203:0/4198590274' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:26:51.025 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:50 smithi137 ceph-mon[29408]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:51.025 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:50 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:51.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:50 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:26:51.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:50 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.203:0/4198590274' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:26:51.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:50 smithi203 ceph-mon[32237]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:51.031 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:50 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:51.549 INFO:teuthology.orchestra.run.smithi137.stdout:# minimal ceph.conf for f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:26:51.549 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2024-09-18T07:26:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:26:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: mon_host = 172.21.15.137:6789/0 172.21.15.203:6789/0 v1:172.21.15.137:6790/0 2024-09-18T07:26:51.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:26:51.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: mon.a calling monitor election 2024-09-18T07:26:51.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: mon.b calling monitor election 2024-09-18T07:26:51.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:26:51.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:51.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:51.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:51.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:51.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:51.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:51.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:51.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:51.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-18T07:26:51.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: monmap epoch 3 2024-09-18T07:26:51.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:26:51.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: last_changed 2024-09-18T07:26:44.690052+0000 2024-09-18T07:26:51.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: created 2024-09-18T07:25:15.819392+0000 2024-09-18T07:26:51.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: min_mon_release 19 (squid) 2024-09-18T07:26:51.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: election_strategy: 1 2024-09-18T07:26:51.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-18T07:26:51.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: 1: v1:172.21.15.203:6789/0 mon.b 2024-09-18T07:26:51.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: 2: v1:172.21.15.137:6790/0 mon.c 2024-09-18T07:26:51.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: fsmap 2024-09-18T07:26:51.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:26:51.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: mgrmap e13: y(active, since 43s) 2024-09-18T07:26:51.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-18T07:26:51.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-18T07:26:51.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-18T07:26:51.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: mon.c (rank 2) addr v1:172.21.15.137:6790/0 is down (out of quorum) 2024-09-18T07:26:51.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:51.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:51.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:51.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:51.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:26:51.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.203:0/4198590274' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:26:51.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:51.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:52.215 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-18T07:26:52.215 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-18T07:26:52.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.conf 2024-09-18T07:26:52.242 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-18T07:26:52.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:26:52.307 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-18T07:26:52.308 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.conf 2024-09-18T07:26:52.335 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-18T07:26:52.335 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:26:52.401 INFO:tasks.cephadm:Adding mgr.y on smithi137 2024-09-18T07:26:52.401 INFO:tasks.cephadm:Adding mgr.x on smithi203 2024-09-18T07:26:52.402 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi137=y;smithi203=x' 2024-09-18T07:26:52.692 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:26:52.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: mon.c calling monitor election 2024-09-18T07:26:52.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: mon.c calling monitor election 2024-09-18T07:26:52.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: mon.b calling monitor election 2024-09-18T07:26:52.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: mon.a calling monitor election 2024-09-18T07:26:52.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-18T07:26:52.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: monmap epoch 3 2024-09-18T07:26:52.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:26:52.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: last_changed 2024-09-18T07:26:44.690052+0000 2024-09-18T07:26:52.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: created 2024-09-18T07:25:15.819392+0000 2024-09-18T07:26:52.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: min_mon_release 19 (squid) 2024-09-18T07:26:52.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: election_strategy: 1 2024-09-18T07:26:52.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-18T07:26:52.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: 1: v1:172.21.15.203:6789/0 mon.b 2024-09-18T07:26:52.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: 2: v1:172.21.15.137:6790/0 mon.c 2024-09-18T07:26:52.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: fsmap 2024-09-18T07:26:52.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:26:52.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: mgrmap e13: y(active, since 45s) 2024-09-18T07:26:52.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-18T07:26:52.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:26:52.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: overall HEALTH_OK 2024-09-18T07:26:52.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:52.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: mon.c calling monitor election 2024-09-18T07:26:52.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: mon.c calling monitor election 2024-09-18T07:26:52.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: mon.b calling monitor election 2024-09-18T07:26:52.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: mon.a calling monitor election 2024-09-18T07:26:52.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-18T07:26:52.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: monmap epoch 3 2024-09-18T07:26:52.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:26:52.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: last_changed 2024-09-18T07:26:44.690052+0000 2024-09-18T07:26:52.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: created 2024-09-18T07:25:15.819392+0000 2024-09-18T07:26:52.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: min_mon_release 19 (squid) 2024-09-18T07:26:52.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: election_strategy: 1 2024-09-18T07:26:52.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-18T07:26:52.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: 1: v1:172.21.15.203:6789/0 mon.b 2024-09-18T07:26:52.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: 2: v1:172.21.15.137:6790/0 mon.c 2024-09-18T07:26:52.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: fsmap 2024-09-18T07:26:52.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:26:52.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: mgrmap e13: y(active, since 45s) 2024-09-18T07:26:52.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-18T07:26:52.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:26:52.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: overall HEALTH_OK 2024-09-18T07:26:52.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:52 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:52.931 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: mon.c calling monitor election 2024-09-18T07:26:52.931 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: mon.c calling monitor election 2024-09-18T07:26:52.931 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: mon.b calling monitor election 2024-09-18T07:26:52.931 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: mon.a calling monitor election 2024-09-18T07:26:52.931 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-18T07:26:52.931 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: monmap epoch 3 2024-09-18T07:26:52.931 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:26:52.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: last_changed 2024-09-18T07:26:44.690052+0000 2024-09-18T07:26:52.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: created 2024-09-18T07:25:15.819392+0000 2024-09-18T07:26:52.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: min_mon_release 19 (squid) 2024-09-18T07:26:52.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: election_strategy: 1 2024-09-18T07:26:52.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-18T07:26:52.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: 1: v1:172.21.15.203:6789/0 mon.b 2024-09-18T07:26:52.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: 2: v1:172.21.15.137:6790/0 mon.c 2024-09-18T07:26:52.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: fsmap 2024-09-18T07:26:52.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: osdmap e4: 0 total, 0 up, 0 in 2024-09-18T07:26:52.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: mgrmap e13: y(active, since 45s) 2024-09-18T07:26:52.932 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-18T07:26:52.933 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:26:52.933 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: overall HEALTH_OK 2024-09-18T07:26:52.933 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:52 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:53.338 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mgr update... 2024-09-18T07:26:53.898 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:53 smithi203 ceph-mon[32237]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:53.899 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:53.899 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:53.899 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:53.899 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:53.899 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-18T07:26:53.900 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-18T07:26:53.900 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:53.908 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:53 smithi137 ceph-mon[29408]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:53.908 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:53.908 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:53.908 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:53.908 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:53.908 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-18T07:26:53.909 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-18T07:26:53.909 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:53.909 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:53 smithi137 ceph-mon[39175]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:53.909 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:26:53.909 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:53.909 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:53.909 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:53.909 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-18T07:26:53.909 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-18T07:26:53.909 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:53.926 DEBUG:teuthology.orchestra.run.smithi203:mgr.x> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mgr.x.service 2024-09-18T07:26:53.928 DEBUG:tasks.cephadm:set 0 configs 2024-09-18T07:26:53.929 INFO:tasks.cephadm:Deploying OSDs... 2024-09-18T07:26:53.929 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-18T07:26:53.929 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T07:26:53.945 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T07:26:53.945 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_1 2024-09-18T07:26:54.001 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T07:26:54.001 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T07:26:54.001 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 977 Links: 1 2024-09-18T07:26:54.001 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T07:26:54.001 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T07:26:54.001 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-18 07:26:44.130660388 +0000 2024-09-18T07:26:54.002 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-18 07:21:45.353715874 +0000 2024-09-18T07:26:54.002 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-18 07:21:45.353715874 +0000 2024-09-18T07:26:54.002 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-09-18 07:21:45.353715874 +0000 2024-09-18T07:26:54.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T07:26:54.067 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-18T07:26:54.067 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-18T07:26:54.067 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000274847 s, 1.9 MB/s 2024-09-18T07:26:54.068 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T07:26:54.125 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_2 2024-09-18T07:26:54.181 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T07:26:54.181 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T07:26:54.181 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 984 Links: 1 2024-09-18T07:26:54.181 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T07:26:54.181 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T07:26:54.181 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-18 07:26:44.131660371 +0000 2024-09-18T07:26:54.181 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-18 07:21:45.354716012 +0000 2024-09-18T07:26:54.181 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-18 07:21:45.354716012 +0000 2024-09-18T07:26:54.182 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-09-18 07:21:45.354716012 +0000 2024-09-18T07:26:54.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T07:26:54.244 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-18T07:26:54.245 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-18T07:26:54.245 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000244905 s, 2.1 MB/s 2024-09-18T07:26:54.246 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T07:26:54.302 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_3 2024-09-18T07:26:54.358 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T07:26:54.358 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T07:26:54.358 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 971 Links: 1 2024-09-18T07:26:54.358 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T07:26:54.358 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T07:26:54.358 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-18 07:26:44.131660371 +0000 2024-09-18T07:26:54.358 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-18 07:21:45.353715874 +0000 2024-09-18T07:26:54.358 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-18 07:21:45.353715874 +0000 2024-09-18T07:26:54.358 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-09-18 07:21:45.353715874 +0000 2024-09-18T07:26:54.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T07:26:54.421 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-18T07:26:54.421 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-18T07:26:54.421 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000167523 s, 3.1 MB/s 2024-09-18T07:26:54.422 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T07:26:54.479 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_4 2024-09-18T07:26:54.534 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T07:26:54.534 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T07:26:54.534 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 968 Links: 1 2024-09-18T07:26:54.534 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T07:26:54.534 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T07:26:54.534 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-18 07:26:44.131660371 +0000 2024-09-18T07:26:54.534 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-18 07:21:45.352715736 +0000 2024-09-18T07:26:54.534 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-18 07:21:45.352715736 +0000 2024-09-18T07:26:54.534 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-09-18 07:21:45.352715736 +0000 2024-09-18T07:26:54.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T07:26:54.597 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-18T07:26:54.597 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-18T07:26:54.597 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000167306 s, 3.1 MB/s 2024-09-18T07:26:54.598 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T07:26:54.655 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-18T07:26:54.655 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T07:26:54.672 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T07:26:54.672 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-09-18T07:26:54.729 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T07:26:54.729 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T07:26:54.729 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 969 Links: 1 2024-09-18T07:26:54.729 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T07:26:54.730 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T07:26:54.730 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-18 07:26:35.427625300 +0000 2024-09-18T07:26:54.730 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-18 07:21:45.183163503 +0000 2024-09-18T07:26:54.730 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-18 07:21:45.183163503 +0000 2024-09-18T07:26:54.730 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-09-18 07:21:45.183163503 +0000 2024-09-18T07:26:54.730 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T07:26:54.794 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-18T07:26:54.794 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-18T07:26:54.794 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000196215 s, 2.6 MB/s 2024-09-18T07:26:54.795 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T07:26:54.852 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-09-18T07:26:54.907 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T07:26:54.908 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T07:26:54.908 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 982 Links: 1 2024-09-18T07:26:54.908 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T07:26:54.908 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T07:26:54.908 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-18 07:26:35.428625276 +0000 2024-09-18T07:26:54.908 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-18 07:21:45.184163477 +0000 2024-09-18T07:26:54.908 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-18 07:21:45.184163477 +0000 2024-09-18T07:26:54.908 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-09-18 07:21:45.184163477 +0000 2024-09-18T07:26:54.908 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T07:26:54.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:54 smithi137 ceph-mon[39175]: from='client.24109 v1:172.21.15.203:0/1432288192' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi137=y;smithi203=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:26:54.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:54 smithi137 ceph-mon[39175]: Saving service mgr spec with placement smithi137=y;smithi203=x;count:2 2024-09-18T07:26:54.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:54 smithi137 ceph-mon[39175]: Reconfiguring mgr.y (unknown last config time)... 2024-09-18T07:26:54.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:54 smithi137 ceph-mon[39175]: Reconfiguring daemon mgr.y on smithi137 2024-09-18T07:26:54.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:54 smithi137 ceph-mon[29408]: from='client.24109 v1:172.21.15.203:0/1432288192' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi137=y;smithi203=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:26:54.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:54 smithi137 ceph-mon[29408]: Saving service mgr spec with placement smithi137=y;smithi203=x;count:2 2024-09-18T07:26:54.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:54 smithi137 ceph-mon[29408]: Reconfiguring mgr.y (unknown last config time)... 2024-09-18T07:26:54.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:54 smithi137 ceph-mon[29408]: Reconfiguring daemon mgr.y on smithi137 2024-09-18T07:26:54.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:54 smithi203 ceph-mon[32237]: from='client.24109 v1:172.21.15.203:0/1432288192' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi137=y;smithi203=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:26:54.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:54 smithi203 ceph-mon[32237]: Saving service mgr spec with placement smithi137=y;smithi203=x;count:2 2024-09-18T07:26:54.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:54 smithi203 ceph-mon[32237]: Reconfiguring mgr.y (unknown last config time)... 2024-09-18T07:26:54.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:54 smithi203 ceph-mon[32237]: Reconfiguring daemon mgr.y on smithi137 2024-09-18T07:26:54.970 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-18T07:26:54.971 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-18T07:26:54.971 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000207536 s, 2.5 MB/s 2024-09-18T07:26:54.972 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T07:26:55.029 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-09-18T07:26:55.085 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T07:26:55.085 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T07:26:55.086 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 966 Links: 1 2024-09-18T07:26:55.086 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T07:26:55.086 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T07:26:55.086 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-18 07:26:35.428625276 +0000 2024-09-18T07:26:55.086 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-18 07:21:45.183163503 +0000 2024-09-18T07:26:55.086 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-18 07:21:45.183163503 +0000 2024-09-18T07:26:55.086 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-09-18 07:21:45.183163503 +0000 2024-09-18T07:26:55.086 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T07:26:55.148 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-18T07:26:55.149 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-18T07:26:55.149 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000179365 s, 2.9 MB/s 2024-09-18T07:26:55.151 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T07:26:55.206 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-09-18T07:26:55.262 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T07:26:55.262 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T07:26:55.262 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 967 Links: 1 2024-09-18T07:26:55.262 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T07:26:55.262 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-09-18T07:26:55.262 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-18 07:26:35.429625251 +0000 2024-09-18T07:26:55.262 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-18 07:21:45.183163503 +0000 2024-09-18T07:26:55.262 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-18 07:21:45.183163503 +0000 2024-09-18T07:26:55.262 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-09-18 07:21:45.183163503 +0000 2024-09-18T07:26:55.263 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T07:26:55.326 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-18T07:26:55.326 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-18T07:26:55.326 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000188406 s, 2.7 MB/s 2024-09-18T07:26:55.327 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T07:26:55.384 INFO:tasks.cephadm:Deploying osd.0 on smithi137 with /dev/vg_nvme/lv_4... 2024-09-18T07:26:55.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-18T07:26:55.625 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:26:55.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:55 smithi137 ceph-mon[29408]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:55.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:55 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:55.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:55 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:55.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:55 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:26:55.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:55 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-18T07:26:55.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:55 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:55.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:55 smithi137 ceph-mon[39175]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:55.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:55 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:55.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:55 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:55.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:55 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:26:55.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:55 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-18T07:26:55.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:55 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:55.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:55 smithi203 ceph-mon[32237]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:55.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:55 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:55.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:55 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:55.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:55 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:26:55.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:55 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-18T07:26:55.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:55 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:56.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:56 smithi203 ceph-mon[32237]: Reconfiguring mon.c (monmap changed)... 2024-09-18T07:26:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:56 smithi203 ceph-mon[32237]: Reconfiguring daemon mon.c on smithi137 2024-09-18T07:26:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:56 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:56 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:56 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:26:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:56 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-18T07:26:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:56 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:56.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:56 smithi137 ceph-mon[39175]: Reconfiguring mon.c (monmap changed)... 2024-09-18T07:26:56.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:56 smithi137 ceph-mon[39175]: Reconfiguring daemon mon.c on smithi137 2024-09-18T07:26:56.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:56 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:56.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:56 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:56.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:56 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:26:56.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:56 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-18T07:26:56.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:56 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:56.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:56 smithi137 ceph-mon[29408]: Reconfiguring mon.c (monmap changed)... 2024-09-18T07:26:56.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:56 smithi137 ceph-mon[29408]: Reconfiguring daemon mon.c on smithi137 2024-09-18T07:26:56.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:56 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:56.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:56 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:56.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:56 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-18T07:26:56.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:56 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-18T07:26:56.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:56 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:57 smithi203 ceph-mon[32237]: Reconfiguring mon.b (monmap changed)... 2024-09-18T07:26:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:57 smithi203 ceph-mon[32237]: Reconfiguring daemon mon.b on smithi203 2024-09-18T07:26:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:57 smithi203 ceph-mon[32237]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:57 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:57 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:57 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:26:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:57 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:57 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:26:57.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:57 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:57.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:57 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-18T07:26:57.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:57 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-18T07:26:57.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:57 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-18T07:26:57.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:57 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:57.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[29408]: Reconfiguring mon.b (monmap changed)... 2024-09-18T07:26:57.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[29408]: Reconfiguring daemon mon.b on smithi203 2024-09-18T07:26:57.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[29408]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:57.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:57.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:57.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:26:57.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:57.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:26:57.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:57.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-18T07:26:57.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-18T07:26:57.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-18T07:26:57.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:57.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[39175]: Reconfiguring mon.b (monmap changed)... 2024-09-18T07:26:57.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[39175]: Reconfiguring daemon mon.b on smithi203 2024-09-18T07:26:57.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[39175]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:57.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:57.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:57.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:26:57.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:57.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:26:57.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:26:57.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-18T07:26:57.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-18T07:26:57.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-18T07:26:57.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:57 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:58.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:58 smithi137 ceph-mon[29408]: Deploying daemon mgr.x on smithi203 2024-09-18T07:26:58.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:58 smithi137 ceph-mon[39175]: Deploying daemon mgr.x on smithi203 2024-09-18T07:26:58.703 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:26:58.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi137:vg_nvme/lv_4 2024-09-18T07:26:58.788 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:58 smithi203 ceph-mon[32237]: Deploying daemon mgr.x on smithi203 2024-09-18T07:26:58.983 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:26:59.432 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:26:59 smithi203 systemd[1]: Starting Ceph mgr.x for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:26:59.685 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:59 smithi203 ceph-mon[32237]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:59.685 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:59 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:26:59.685 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:59 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:26:59.685 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:26:59 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:59.685 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:26:59 smithi203 podman[33728]: 2024-09-18 07:26:59.430319884 +0000 UTC m=+0.110678821 container create 1be2f12a125f85ae0c5740cdfc25bf253fd90a2f19fadda22684fbf8913df4a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-18T07:26:59.686 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:26:59 smithi203 podman[33728]: 2024-09-18 07:26:59.335642106 +0000 UTC m=+0.016001044 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:26:59.686 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:26:59 smithi203 podman[33728]: 2024-09-18 07:26:59.633304781 +0000 UTC m=+0.313663739 container init 1be2f12a125f85ae0c5740cdfc25bf253fd90a2f19fadda22684fbf8913df4a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-x, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-18T07:26:59.686 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:26:59 smithi203 podman[33728]: 2024-09-18 07:26:59.636199031 +0000 UTC m=+0.316557970 container start 1be2f12a125f85ae0c5740cdfc25bf253fd90a2f19fadda22684fbf8913df4a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-x, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T07:26:59.686 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:26:59 smithi203 ceph-mgr[33747]: -- 172.21.15.203:0/1428552919 <== mon.1 v1:172.21.15.203:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (unknown 170631757 0 0) 0x5646b8e9cd20 con 0x5646b8e7b400 2024-09-18T07:26:59.686 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:26:59 smithi203 bash[33728]: 1be2f12a125f85ae0c5740cdfc25bf253fd90a2f19fadda22684fbf8913df4a5 2024-09-18T07:26:59.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:59 smithi137 ceph-mon[29408]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:59.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:59 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:26:59.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:59 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:26:59.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:26:59 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:59.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:59 smithi137 ceph-mon[39175]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:26:59.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:59 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:26:59.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:59 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:26:59.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:26:59 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:26:59.960 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:26:59 smithi203 systemd[1]: Started Ceph mgr.x for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:27:00.883 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:00 smithi137 ceph-mon[29408]: from='client.34103 v1:172.21.15.137:0/4125828342' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:27:00.883 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:00 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:00.883 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:00 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:00.884 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:00 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:00.884 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:00 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:00.884 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:00 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:00.884 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:00 smithi137 ceph-mon[39175]: from='client.34103 v1:172.21.15.137:0/4125828342' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:27:00.884 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:00 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:00.885 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:00 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:00.885 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:00 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:00.885 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:00 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:00.885 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:00 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:00.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:00 smithi203 ceph-mon[32237]: from='client.34103 v1:172.21.15.137:0/4125828342' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:27:00.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:00 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:00.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:00 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:00.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:00 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:00.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:00 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:00.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:00 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:01.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[29408]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:01.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:01.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:01.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:01.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:01.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:01.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:01.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:01.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:01.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:01.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[39175]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:01.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:01.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:01.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:01.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:01.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:01.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:01.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:01.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:01.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:01 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:01.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:01 smithi203 ceph-mon[32237]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:01.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:01 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:01.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:01 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:01.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:01 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:01.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:01 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:01.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:01 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:01.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:01 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:01.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:01 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:01.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:01 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:01.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:01 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:03.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:02 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:03.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:02 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3423197973' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "268cd580-c931-4992-bb93-d6f63c2bfe88"} : dispatch 2024-09-18T07:27:03.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:02 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3423197973' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "268cd580-c931-4992-bb93-d6f63c2bfe88"}]': finished 2024-09-18T07:27:03.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:02 smithi137 ceph-mon[39175]: osdmap e5: 1 total, 0 up, 1 in 2024-09-18T07:27:03.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:02 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:03.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:02 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3157897450' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:27:03.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:02 smithi137 ceph-mon[39175]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:03.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:02 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:03.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:02 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3423197973' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "268cd580-c931-4992-bb93-d6f63c2bfe88"} : dispatch 2024-09-18T07:27:03.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:02 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3423197973' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "268cd580-c931-4992-bb93-d6f63c2bfe88"}]': finished 2024-09-18T07:27:03.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:02 smithi137 ceph-mon[29408]: osdmap e5: 1 total, 0 up, 1 in 2024-09-18T07:27:03.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:02 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:03.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:02 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3157897450' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:27:03.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:02 smithi137 ceph-mon[29408]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:03.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:02 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:03.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:02 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3423197973' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "268cd580-c931-4992-bb93-d6f63c2bfe88"} : dispatch 2024-09-18T07:27:03.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:02 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3423197973' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "268cd580-c931-4992-bb93-d6f63c2bfe88"}]': finished 2024-09-18T07:27:03.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:02 smithi203 ceph-mon[32237]: osdmap e5: 1 total, 0 up, 1 in 2024-09-18T07:27:03.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:02 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:03.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:02 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3157897450' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:27:03.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:02 smithi203 ceph-mon[32237]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:04 smithi203 ceph-mon[32237]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:04 smithi203 ceph-mon[32237]: Standby manager daemon x started 2024-09-18T07:27:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:04 smithi203 ceph-mon[32237]: from='mgr.? v1:172.21.15.203:0/166638251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-18T07:27:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:04 smithi203 ceph-mon[32237]: from='mgr.? v1:172.21.15.203:0/166638251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-18T07:27:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:04 smithi203 ceph-mon[32237]: from='mgr.? v1:172.21.15.203:0/166638251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-18T07:27:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:04 smithi203 ceph-mon[32237]: from='mgr.? v1:172.21.15.203:0/166638251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-18T07:27:05.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:04 smithi137 ceph-mon[29408]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:05.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:04 smithi137 ceph-mon[29408]: Standby manager daemon x started 2024-09-18T07:27:05.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:04 smithi137 ceph-mon[29408]: from='mgr.? v1:172.21.15.203:0/166638251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-18T07:27:05.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:04 smithi137 ceph-mon[29408]: from='mgr.? v1:172.21.15.203:0/166638251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-18T07:27:05.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:04 smithi137 ceph-mon[29408]: from='mgr.? v1:172.21.15.203:0/166638251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-18T07:27:05.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:04 smithi137 ceph-mon[29408]: from='mgr.? v1:172.21.15.203:0/166638251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-18T07:27:05.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:04 smithi137 ceph-mon[39175]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:05.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:04 smithi137 ceph-mon[39175]: Standby manager daemon x started 2024-09-18T07:27:05.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:04 smithi137 ceph-mon[39175]: from='mgr.? v1:172.21.15.203:0/166638251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-18T07:27:05.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:04 smithi137 ceph-mon[39175]: from='mgr.? v1:172.21.15.203:0/166638251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-18T07:27:05.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:04 smithi137 ceph-mon[39175]: from='mgr.? v1:172.21.15.203:0/166638251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-18T07:27:05.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:04 smithi137 ceph-mon[39175]: from='mgr.? v1:172.21.15.203:0/166638251' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-18T07:27:05.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:05 smithi137 ceph-mon[29408]: mgrmap e14: y(active, since 58s), standbys: x 2024-09-18T07:27:05.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:05 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-18T07:27:05.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:05 smithi137 ceph-mon[39175]: mgrmap e14: y(active, since 58s), standbys: x 2024-09-18T07:27:05.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:05 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-18T07:27:05.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:05 smithi203 ceph-mon[32237]: mgrmap e14: y(active, since 58s), standbys: x 2024-09-18T07:27:05.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:05 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-18T07:27:06.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:06 smithi137 ceph-mon[29408]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:06.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:06 smithi137 ceph-mon[39175]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:07.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:06 smithi203 ceph-mon[32237]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:08.862 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:08 smithi137 ceph-mon[29408]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:08.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:08 smithi137 ceph-mon[39175]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:08.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:08 smithi203 ceph-mon[32237]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:09.839 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:09 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-18T07:27:09.839 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:09 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:09.840 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:09 smithi137 ceph-mon[29408]: Deploying daemon osd.0 on smithi137 2024-09-18T07:27:09.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:09 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-18T07:27:09.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:09 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:09.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:09 smithi137 ceph-mon[39175]: Deploying daemon osd.0 on smithi137 2024-09-18T07:27:09.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:09 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-18T07:27:09.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:09 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:09.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:09 smithi203 ceph-mon[32237]: Deploying daemon osd.0 on smithi137 2024-09-18T07:27:10.828 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:10 smithi137 ceph-mon[29408]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:10.828 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:10 smithi137 ceph-mon[39175]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:10.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:10 smithi203 ceph-mon[32237]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:12.772 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:12 smithi137 ceph-mon[29408]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:12.773 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:12 smithi137 ceph-mon[39175]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:12.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:12 smithi203 ceph-mon[32237]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:14.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:14 smithi203 ceph-mon[32237]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:15.088 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:14 smithi137 ceph-mon[29408]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:15.088 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:14 smithi137 ceph-mon[39175]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:16.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:16 smithi203 ceph-mon[32237]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:17.111 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:16 smithi137 ceph-mon[29408]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:17.111 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:16 smithi137 ceph-mon[39175]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:17.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:17 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:17.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:17 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:17.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:17 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:18.049 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:17 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:18.049 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:17 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:18.049 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:17 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:18.049 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:17 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:18.050 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:17 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:18.050 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:17 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:18.970 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:18 smithi137 ceph-mon[29408]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:18.970 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:18 smithi137 ceph-mon[39175]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:19.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:18 smithi203 ceph-mon[32237]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:20.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:20.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:20.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:20.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:20.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:20.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:20 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:20.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:20 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:20.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:20 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:20.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:20 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:20.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:20 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:20.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:20 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:20.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:20 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:20.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:20 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:20.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:20 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:20.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:20 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:21.560 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:21 smithi137 ceph-mon[29408]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:21.560 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:21 smithi137 ceph-mon[39175]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:21.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:21 smithi203 ceph-mon[32237]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:22.221 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 0 on host 'smithi137' 2024-09-18T07:27:22.520 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:22 smithi137 ceph-mon[29408]: from='osd.0 v1:172.21.15.137:6801/2811444477' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-18T07:27:22.520 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:22 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:22.520 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:22 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:22.520 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:22 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:22.521 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:22 smithi137 ceph-mon[39175]: from='osd.0 v1:172.21.15.137:6801/2811444477' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-18T07:27:22.521 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:22 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:22.521 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:22 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:22.521 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:22 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:22.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:22 smithi203 ceph-mon[32237]: from='osd.0 v1:172.21.15.137:6801/2811444477' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-18T07:27:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:22 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:22 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:22 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:22.870 DEBUG:teuthology.orchestra.run.smithi137:osd.0> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.0.service 2024-09-18T07:27:22.873 INFO:tasks.cephadm:Deploying osd.1 on smithi137 with /dev/vg_nvme/lv_3... 2024-09-18T07:27:22.873 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-18T07:27:23.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:23 smithi137 ceph-mon[29408]: from='osd.0 v1:172.21.15.137:6801/2811444477' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-18T07:27:23.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:23 smithi137 ceph-mon[29408]: osdmap e6: 1 total, 0 up, 1 in 2024-09-18T07:27:23.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:23 smithi137 ceph-mon[29408]: from='osd.0 v1:172.21.15.137:6801/2811444477' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:27:23.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:23.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:23 smithi137 ceph-mon[29408]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:23.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:23 smithi137 ceph-mon[39175]: from='osd.0 v1:172.21.15.137:6801/2811444477' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-18T07:27:23.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:23 smithi137 ceph-mon[39175]: osdmap e6: 1 total, 0 up, 1 in 2024-09-18T07:27:23.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:23 smithi137 ceph-mon[39175]: from='osd.0 v1:172.21.15.137:6801/2811444477' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:27:23.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:23.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:23 smithi137 ceph-mon[39175]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:23.421 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:27:23 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-0[42480]: 2024-09-18T07:27:23.280+0000 7fb0e6c39640 -1 osd.0 0 waiting for initial osdmap 2024-09-18T07:27:23.421 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:27:23 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-0[42480]: 2024-09-18T07:27:23.283+0000 7fb0e2a60640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T07:27:23.589 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:27:23.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:23 smithi203 ceph-mon[32237]: from='osd.0 v1:172.21.15.137:6801/2811444477' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-18T07:27:23.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:23 smithi203 ceph-mon[32237]: osdmap e6: 1 total, 0 up, 1 in 2024-09-18T07:27:23.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:23 smithi203 ceph-mon[32237]: from='osd.0 v1:172.21.15.137:6801/2811444477' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:27:23.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:23.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:23 smithi203 ceph-mon[32237]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:24.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:24 smithi137 ceph-mon[29408]: from='osd.0 v1:172.21.15.137:6801/2811444477' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-18T07:27:24.656 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:24 smithi137 ceph-mon[29408]: osdmap e7: 1 total, 0 up, 1 in 2024-09-18T07:27:24.656 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:24 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:24.656 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:24 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:24.656 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:24 smithi137 ceph-mon[39175]: from='osd.0 v1:172.21.15.137:6801/2811444477' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-18T07:27:24.656 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:24 smithi137 ceph-mon[39175]: osdmap e7: 1 total, 0 up, 1 in 2024-09-18T07:27:24.656 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:24 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:24.656 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:24 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:24.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:24 smithi203 ceph-mon[32237]: from='osd.0 v1:172.21.15.137:6801/2811444477' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-18T07:27:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:24 smithi203 ceph-mon[32237]: osdmap e7: 1 total, 0 up, 1 in 2024-09-18T07:27:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:24 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:24 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:25.562 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:25 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:27:25.562 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:25 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:27:25.563 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:25 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:25.563 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:25 smithi137 ceph-mon[29408]: osd.0 v1:172.21.15.137:6801/2811444477 boot 2024-09-18T07:27:25.563 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:25 smithi137 ceph-mon[29408]: osdmap e8: 1 total, 1 up, 1 in 2024-09-18T07:27:25.563 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:25 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:25.563 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:25 smithi137 ceph-mon[29408]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:25.563 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:25 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:27:25.563 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:25 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:27:25.563 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:25 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:25.563 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:25 smithi137 ceph-mon[39175]: osd.0 v1:172.21.15.137:6801/2811444477 boot 2024-09-18T07:27:25.563 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:25 smithi137 ceph-mon[39175]: osdmap e8: 1 total, 1 up, 1 in 2024-09-18T07:27:25.564 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:25 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:25.564 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:25 smithi137 ceph-mon[39175]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:25.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:25 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:27:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:25 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:27:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:25 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:25 smithi203 ceph-mon[32237]: osd.0 v1:172.21.15.137:6801/2811444477 boot 2024-09-18T07:27:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:25 smithi203 ceph-mon[32237]: osdmap e8: 1 total, 1 up, 1 in 2024-09-18T07:27:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:25 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:27:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:25 smithi203 ceph-mon[32237]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-18T07:27:26.301 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:27:26.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi137:vg_nvme/lv_3 2024-09-18T07:27:26.608 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:26 smithi137 ceph-mon[29408]: osdmap e9: 1 total, 1 up, 1 in 2024-09-18T07:27:26.608 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:26 smithi137 ceph-mon[39175]: osdmap e9: 1 total, 1 up, 1 in 2024-09-18T07:27:26.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:26 smithi203 ceph-mon[32237]: osdmap e9: 1 total, 1 up, 1 in 2024-09-18T07:27:26.894 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:27:27.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:27 smithi137 ceph-mon[29408]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:27.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:27 smithi137 ceph-mon[39175]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:27.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:27 smithi203 ceph-mon[32237]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:28.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[29408]: from='client.14253 v1:172.21.15.137:0/1680347912' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:27:28.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:27:28.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:27:28.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:28.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:28.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:28.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:27:28.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:28.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:28.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:28.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:28.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[39175]: from='client.14253 v1:172.21.15.137:0/1680347912' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:27:28.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:27:28.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:27:28.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:28.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:28.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:28.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:27:28.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:28.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:28.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:28.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:28 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:28 smithi203 ceph-mon[32237]: from='client.14253 v1:172.21.15.137:0/1680347912' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:27:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:28 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:27:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:28 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:27:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:28 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:28 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:28 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:28.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:28 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:27:28.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:28 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:28.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:28 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:28.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:28 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:28.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:28 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:29.561 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:29 smithi137 ceph-mon[39175]: Detected new or changed devices on smithi137 2024-09-18T07:27:29.561 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:29 smithi137 ceph-mon[39175]: Adjusting osd_memory_target on smithi137 to 16064M 2024-09-18T07:27:29.561 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:29 smithi137 ceph-mon[39175]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:29.562 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:29 smithi137 ceph-mon[29408]: Detected new or changed devices on smithi137 2024-09-18T07:27:29.562 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:29 smithi137 ceph-mon[29408]: Adjusting osd_memory_target on smithi137 to 16064M 2024-09-18T07:27:29.562 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:29 smithi137 ceph-mon[29408]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:29.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:29 smithi203 ceph-mon[32237]: Detected new or changed devices on smithi137 2024-09-18T07:27:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:29 smithi203 ceph-mon[32237]: Adjusting osd_memory_target on smithi137 to 16064M 2024-09-18T07:27:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:29 smithi203 ceph-mon[32237]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:30.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:30 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3179795725' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575"} : dispatch 2024-09-18T07:27:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:30 smithi137 ceph-mon[29408]: from='client.34142 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575"} : dispatch 2024-09-18T07:27:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:30 smithi137 ceph-mon[29408]: from='client.34142 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575"}]': finished 2024-09-18T07:27:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:30 smithi137 ceph-mon[29408]: osdmap e10: 2 total, 1 up, 2 in 2024-09-18T07:27:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:30 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:30 smithi137 ceph-mon[29408]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:30 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/849784840' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:27:30.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:30 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3179795725' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575"} : dispatch 2024-09-18T07:27:30.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:30 smithi137 ceph-mon[39175]: from='client.34142 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575"} : dispatch 2024-09-18T07:27:30.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:30 smithi137 ceph-mon[39175]: from='client.34142 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575"}]': finished 2024-09-18T07:27:30.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:30 smithi137 ceph-mon[39175]: osdmap e10: 2 total, 1 up, 2 in 2024-09-18T07:27:30.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:30 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:30.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:30 smithi137 ceph-mon[39175]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:30.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:30 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/849784840' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:27:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:30 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3179795725' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575"} : dispatch 2024-09-18T07:27:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:30 smithi203 ceph-mon[32237]: from='client.34142 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575"} : dispatch 2024-09-18T07:27:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:30 smithi203 ceph-mon[32237]: from='client.34142 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575"}]': finished 2024-09-18T07:27:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:30 smithi203 ceph-mon[32237]: osdmap e10: 2 total, 1 up, 2 in 2024-09-18T07:27:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:30 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:30 smithi203 ceph-mon[32237]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:30 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/849784840' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:27:32.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:32 smithi203 ceph-mon[32237]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:33.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:32 smithi137 ceph-mon[29408]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:33.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:32 smithi137 ceph-mon[39175]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:34.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:34 smithi137 ceph-mon[29408]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:34.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:34 smithi137 ceph-mon[39175]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:34.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:34 smithi203 ceph-mon[32237]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:36.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:36 smithi137 ceph-mon[29408]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:36.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:36 smithi137 ceph-mon[39175]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:36.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:36 smithi203 ceph-mon[32237]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:37.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:37 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-18T07:27:37.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:37 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:37.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:37 smithi203 ceph-mon[32237]: Deploying daemon osd.1 on smithi137 2024-09-18T07:27:37.963 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:37 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-18T07:27:37.963 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:37 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:37.963 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:37 smithi137 ceph-mon[29408]: Deploying daemon osd.1 on smithi137 2024-09-18T07:27:37.963 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:37 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-18T07:27:37.964 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:37 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:37.964 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:37 smithi137 ceph-mon[39175]: Deploying daemon osd.1 on smithi137 2024-09-18T07:27:38.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:38 smithi137 ceph-mon[29408]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:38.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:38 smithi137 ceph-mon[39175]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:39.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:38 smithi203 ceph-mon[32237]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:40.871 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:40 smithi137 ceph-mon[29408]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:40.871 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:40 smithi137 ceph-mon[39175]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:41.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:40 smithi203 ceph-mon[32237]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:42.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:42 smithi203 ceph-mon[32237]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:42 smithi137 ceph-mon[29408]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:43.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:42 smithi137 ceph-mon[39175]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:44.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:44 smithi137 ceph-mon[29408]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:44.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:44 smithi137 ceph-mon[39175]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:44.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:44 smithi203 ceph-mon[32237]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:45.815 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:45 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:45.815 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:45 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:45.815 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:45 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:45.815 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:45 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:45.815 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:45 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:45.816 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:45 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:45.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:45 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:45.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:45 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:45.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:45 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:47.021 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:46 smithi137 ceph-mon[29408]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:47.022 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:46 smithi137 ceph-mon[39175]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:47.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:46 smithi203 ceph-mon[32237]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:48.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:48 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:48.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:48 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:48.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:48 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:48.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:48 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:48.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:48 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:48.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:48 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:48.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:48 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:48.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:48 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:48.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:48 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:48.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:48 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:48.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:48 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:48.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:48 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:48.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:48 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:48.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:48 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:48.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:48 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:49.733 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:49 smithi137 ceph-mon[29408]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:49.734 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:49 smithi137 ceph-mon[29408]: from='osd.1 v1:172.21.15.137:6805/874102487' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-18T07:27:49.734 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:49 smithi137 ceph-mon[39175]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:49.734 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:49 smithi137 ceph-mon[39175]: from='osd.1 v1:172.21.15.137:6805/874102487' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-18T07:27:49.775 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 1 on host 'smithi137' 2024-09-18T07:27:49.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:49 smithi203 ceph-mon[32237]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:49.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:49 smithi203 ceph-mon[32237]: from='osd.1 v1:172.21.15.137:6805/874102487' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-18T07:27:50.407 DEBUG:teuthology.orchestra.run.smithi137:osd.1> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.1.service 2024-09-18T07:27:50.410 INFO:tasks.cephadm:Deploying osd.2 on smithi137 with /dev/vg_nvme/lv_2... 2024-09-18T07:27:50.410 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-18T07:27:51.084 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[39175]: from='osd.1 v1:172.21.15.137:6805/874102487' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-18T07:27:51.084 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[39175]: osdmap e11: 2 total, 1 up, 2 in 2024-09-18T07:27:51.084 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[39175]: from='osd.1 v1:172.21.15.137:6805/874102487' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:27:51.084 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:51.084 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:51.084 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:51.084 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:51.084 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[39175]: from='osd.1 v1:172.21.15.137:6805/874102487' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-18T07:27:51.084 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[39175]: osdmap e12: 2 total, 1 up, 2 in 2024-09-18T07:27:51.084 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:51.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[29408]: from='osd.1 v1:172.21.15.137:6805/874102487' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-18T07:27:51.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[29408]: osdmap e11: 2 total, 1 up, 2 in 2024-09-18T07:27:51.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[29408]: from='osd.1 v1:172.21.15.137:6805/874102487' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:27:51.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:51.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:51.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:51.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:51.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[29408]: from='osd.1 v1:172.21.15.137:6805/874102487' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-18T07:27:51.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[29408]: osdmap e12: 2 total, 1 up, 2 in 2024-09-18T07:27:51.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:50 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:51.165 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:27:51.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:50 smithi203 ceph-mon[32237]: from='osd.1 v1:172.21.15.137:6805/874102487' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-18T07:27:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:50 smithi203 ceph-mon[32237]: osdmap e11: 2 total, 1 up, 2 in 2024-09-18T07:27:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:50 smithi203 ceph-mon[32237]: from='osd.1 v1:172.21.15.137:6805/874102487' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:27:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:50 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:50 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:27:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:50 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:50 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:50 smithi203 ceph-mon[32237]: from='osd.1 v1:172.21.15.137:6805/874102487' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-18T07:27:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:50 smithi203 ceph-mon[32237]: osdmap e12: 2 total, 1 up, 2 in 2024-09-18T07:27:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:50 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:51.381 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:27:51 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-1[48136]: 2024-09-18T07:27:51.079+0000 7fbad90ee640 -1 osd.1 0 waiting for initial osdmap 2024-09-18T07:27:51.381 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:27:51 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-1[48136]: 2024-09-18T07:27:51.082+0000 7fbad06f9640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T07:27:52.132 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:51 smithi203 ceph-mon[32237]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:52.133 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:51 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:52.133 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:51 smithi203 ceph-mon[32237]: from='osd.1 v1:172.21.15.137:6805/874102487' entity='osd.1' 2024-09-18T07:27:52.133 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:51 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:52.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:51 smithi137 ceph-mon[29408]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:52.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:51 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:52.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:51 smithi137 ceph-mon[29408]: from='osd.1 v1:172.21.15.137:6805/874102487' entity='osd.1' 2024-09-18T07:27:52.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:51 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:52.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:51 smithi137 ceph-mon[39175]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:52.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:52.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:51 smithi137 ceph-mon[39175]: from='osd.1 v1:172.21.15.137:6805/874102487' entity='osd.1' 2024-09-18T07:27:52.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:53 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:27:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:53 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:27:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:53 smithi137 ceph-mon[29408]: osd.1 v1:172.21.15.137:6805/874102487 boot 2024-09-18T07:27:53.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:53 smithi137 ceph-mon[29408]: osdmap e13: 2 total, 2 up, 2 in 2024-09-18T07:27:53.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:53.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:53 smithi137 ceph-mon[29408]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:53 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:27:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:53 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:27:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:53 smithi137 ceph-mon[39175]: osd.1 v1:172.21.15.137:6805/874102487 boot 2024-09-18T07:27:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:53 smithi137 ceph-mon[39175]: osdmap e13: 2 total, 2 up, 2 in 2024-09-18T07:27:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:53 smithi137 ceph-mon[39175]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:53 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:27:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:53 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:27:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:53 smithi203 ceph-mon[32237]: osd.1 v1:172.21.15.137:6805/874102487 boot 2024-09-18T07:27:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:53 smithi203 ceph-mon[32237]: osdmap e13: 2 total, 2 up, 2 in 2024-09-18T07:27:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:27:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:53 smithi203 ceph-mon[32237]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-18T07:27:54.356 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[29408]: osdmap e14: 2 total, 2 up, 2 in 2024-09-18T07:27:54.356 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:54.356 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:54.356 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:27:54.356 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:27:54.357 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:54.357 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:54.357 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:54.357 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:54.357 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[39175]: osdmap e14: 2 total, 2 up, 2 in 2024-09-18T07:27:54.357 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:54.357 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:54.357 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:27:54.357 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:27:54.357 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:54.358 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:54.358 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:54.358 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:54 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:54.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:54 smithi203 ceph-mon[32237]: osdmap e14: 2 total, 2 up, 2 in 2024-09-18T07:27:54.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:54 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:54.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:54 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:54.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:54 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:27:54.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:54 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:27:54.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:54 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:54.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:54 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:54.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:54 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:27:54.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:54 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:27:55.140 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:55 smithi137 ceph-mon[29408]: Detected new or changed devices on smithi137 2024-09-18T07:27:55.141 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:55 smithi137 ceph-mon[29408]: Adjusting osd_memory_target on smithi137 to 8032M 2024-09-18T07:27:55.141 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:55 smithi137 ceph-mon[29408]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:27:55.141 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:55 smithi137 ceph-mon[39175]: Detected new or changed devices on smithi137 2024-09-18T07:27:55.141 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:55 smithi137 ceph-mon[39175]: Adjusting osd_memory_target on smithi137 to 8032M 2024-09-18T07:27:55.141 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:55 smithi137 ceph-mon[39175]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:27:55.427 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:27:55.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi137:vg_nvme/lv_2 2024-09-18T07:27:55.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:55 smithi203 ceph-mon[32237]: Detected new or changed devices on smithi137 2024-09-18T07:27:55.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:55 smithi203 ceph-mon[32237]: Adjusting osd_memory_target on smithi137 to 8032M 2024-09-18T07:27:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:55 smithi203 ceph-mon[32237]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:27:55.697 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:27:56.664 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:56 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:27:56.664 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:56 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:27:56.664 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:56 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:56.664 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:56 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:27:56.664 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:56 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:27:56.664 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:56 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:56.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:56 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:27:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:56 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:27:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:56 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:27:57.642 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:57 smithi137 ceph-mon[29408]: from='client.34163 v1:172.21.15.137:0/576891371' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:27:57.642 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:57 smithi137 ceph-mon[29408]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:27:57.642 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:57 smithi137 ceph-mon[39175]: from='client.34163 v1:172.21.15.137:0/576891371' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:27:57.642 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:57 smithi137 ceph-mon[39175]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:27:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:57 smithi203 ceph-mon[32237]: from='client.34163 v1:172.21.15.137:0/576891371' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:27:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:57 smithi203 ceph-mon[32237]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:27:58.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:58 smithi203 ceph-mon[32237]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:27:58.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:58 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1294686413' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "480ad6c5-0e0b-4405-bbe2-381682d00bae"} : dispatch 2024-09-18T07:27:58.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:58 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1294686413' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "480ad6c5-0e0b-4405-bbe2-381682d00bae"}]': finished 2024-09-18T07:27:58.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:58 smithi203 ceph-mon[32237]: osdmap e15: 3 total, 2 up, 3 in 2024-09-18T07:27:58.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:58 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:27:59.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:58 smithi137 ceph-mon[39175]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:27:59.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:58 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1294686413' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "480ad6c5-0e0b-4405-bbe2-381682d00bae"} : dispatch 2024-09-18T07:27:59.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:58 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1294686413' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "480ad6c5-0e0b-4405-bbe2-381682d00bae"}]': finished 2024-09-18T07:27:59.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:58 smithi137 ceph-mon[39175]: osdmap e15: 3 total, 2 up, 3 in 2024-09-18T07:27:59.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:58 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:27:59.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:58 smithi137 ceph-mon[29408]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:27:59.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:58 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1294686413' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "480ad6c5-0e0b-4405-bbe2-381682d00bae"} : dispatch 2024-09-18T07:27:59.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:58 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1294686413' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "480ad6c5-0e0b-4405-bbe2-381682d00bae"}]': finished 2024-09-18T07:27:59.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:58 smithi137 ceph-mon[29408]: osdmap e15: 3 total, 2 up, 3 in 2024-09-18T07:27:59.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:58 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:27:59.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:27:59 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/558000183' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:28:00.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:27:59 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/558000183' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:28:00.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:27:59 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/558000183' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:28:00.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:00 smithi203 ceph-mon[32237]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:01.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:00 smithi137 ceph-mon[29408]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:01.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:00 smithi137 ceph-mon[39175]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:02.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:02 smithi137 ceph-mon[29408]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:02.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:02 smithi137 ceph-mon[39175]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:02.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:02 smithi203 ceph-mon[32237]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:04.995 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:04 smithi137 ceph-mon[29408]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:04.996 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:04 smithi137 ceph-mon[39175]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:05.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:04 smithi203 ceph-mon[32237]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:06.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:05 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-18T07:28:06.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:05 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:06.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:05 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-18T07:28:06.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:05 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:06.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:05 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-18T07:28:06.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:05 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:07.252 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:06 smithi137 ceph-mon[39175]: Deploying daemon osd.2 on smithi137 2024-09-18T07:28:07.252 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:06 smithi137 ceph-mon[39175]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:07.253 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:06 smithi137 ceph-mon[29408]: Deploying daemon osd.2 on smithi137 2024-09-18T07:28:07.253 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:06 smithi137 ceph-mon[29408]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:07.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:06 smithi203 ceph-mon[32237]: Deploying daemon osd.2 on smithi137 2024-09-18T07:28:07.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:06 smithi203 ceph-mon[32237]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:08.961 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:08 smithi137 ceph-mon[29408]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:08.961 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:08 smithi137 ceph-mon[39175]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:09.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:08 smithi203 ceph-mon[32237]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:10.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:10 smithi203 ceph-mon[32237]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:11.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:10 smithi137 ceph-mon[29408]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:11.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:10 smithi137 ceph-mon[39175]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:12.854 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:12 smithi137 ceph-mon[29408]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:12.855 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:12 smithi137 ceph-mon[39175]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:13.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:12 smithi203 ceph-mon[32237]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:13.977 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:13 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:28:13.977 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:13 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:13.977 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:13 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:13.977 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:13 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:28:13.978 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:13 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:13.978 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:13 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:14.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:13 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:28:14.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:13 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:14.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:13 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:15.045 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:14 smithi137 ceph-mon[29408]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:15.045 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:14 smithi137 ceph-mon[39175]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:15.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:14 smithi203 ceph-mon[32237]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:17.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:17 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:17.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:17 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:17.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:17 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:17 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:28:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:17 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:17 smithi137 ceph-mon[39175]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:17.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:17 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:17.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:17 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:17.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:17 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:17.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:17 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:28:17.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:17 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:17.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:17 smithi137 ceph-mon[29408]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:17.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:17 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:17 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:17 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:17 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:28:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:17 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:17 smithi203 ceph-mon[32237]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:18.902 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:18 smithi137 ceph-mon[29408]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:18.902 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:18 smithi137 ceph-mon[29408]: from='osd.2 v1:172.21.15.137:6809/1754684686' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-18T07:28:18.902 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:18 smithi137 ceph-mon[29408]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-18T07:28:18.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:18 smithi137 ceph-mon[39175]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:18.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:18 smithi137 ceph-mon[39175]: from='osd.2 v1:172.21.15.137:6809/1754684686' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-18T07:28:18.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:18 smithi137 ceph-mon[39175]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-18T07:28:18.944 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 2 on host 'smithi137' 2024-09-18T07:28:19.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:18 smithi203 ceph-mon[32237]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:19.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:18 smithi203 ceph-mon[32237]: from='osd.2 v1:172.21.15.137:6809/1754684686' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-18T07:28:19.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:18 smithi203 ceph-mon[32237]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-18T07:28:19.558 DEBUG:teuthology.orchestra.run.smithi137:osd.2> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.2.service 2024-09-18T07:28:19.561 INFO:tasks.cephadm:Deploying osd.3 on smithi137 with /dev/vg_nvme/lv_1... 2024-09-18T07:28:19.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-18T07:28:19.900 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-mon[29408]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-18T07:28:19.900 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-mon[29408]: osdmap e16: 3 total, 2 up, 3 in 2024-09-18T07:28:19.900 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:28:19.900 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-mon[29408]: from='osd.2 v1:172.21.15.137:6809/1754684686' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:28:19.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-mon[29408]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:28:19.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:28:19.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:19.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:19.901 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-mon[39175]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-18T07:28:19.901 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-mon[39175]: osdmap e16: 3 total, 2 up, 3 in 2024-09-18T07:28:19.901 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:28:19.901 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-mon[39175]: from='osd.2 v1:172.21.15.137:6809/1754684686' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:28:19.901 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-mon[39175]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:28:19.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:28:19.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:19.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:19.902 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-2[53968]: 2024-09-18T07:28:19.736+0000 7f8d41dea640 -1 osd.2 0 waiting for initial osdmap 2024-09-18T07:28:19.902 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:28:19 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-2[53968]: 2024-09-18T07:28:19.740+0000 7f8d3dc11640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T07:28:20.112 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:19 smithi203 ceph-mon[32237]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-18T07:28:20.112 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:19 smithi203 ceph-mon[32237]: osdmap e16: 3 total, 2 up, 3 in 2024-09-18T07:28:20.112 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:19 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:28:20.112 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:19 smithi203 ceph-mon[32237]: from='osd.2 v1:172.21.15.137:6809/1754684686' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:28:20.112 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:19 smithi203 ceph-mon[32237]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:28:20.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:19 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:28:20.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:19 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:20.113 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:19 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:20.609 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:28:20.861 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:20 smithi137 ceph-mon[29408]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-18T07:28:20.861 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:20 smithi137 ceph-mon[29408]: osdmap e17: 3 total, 2 up, 3 in 2024-09-18T07:28:20.861 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:28:20.861 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:28:20.861 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:20 smithi137 ceph-mon[29408]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:20.861 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:28:20.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:20 smithi137 ceph-mon[39175]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-18T07:28:20.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:20 smithi137 ceph-mon[39175]: osdmap e17: 3 total, 2 up, 3 in 2024-09-18T07:28:20.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:20 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:28:20.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:20 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:28:20.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:20 smithi137 ceph-mon[39175]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:20.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:20 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:28:21.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:20 smithi203 ceph-mon[32237]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-18T07:28:21.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:20 smithi203 ceph-mon[32237]: osdmap e17: 3 total, 2 up, 3 in 2024-09-18T07:28:21.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:20 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:28:21.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:20 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:28:21.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:20 smithi203 ceph-mon[32237]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-18T07:28:21.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:20 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:28:22.029 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:21 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:28:22.029 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:21 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:28:22.029 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:21 smithi137 ceph-mon[29408]: osd.2 v1:172.21.15.137:6809/1754684686 boot 2024-09-18T07:28:22.029 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:21 smithi137 ceph-mon[29408]: osdmap e18: 3 total, 3 up, 3 in 2024-09-18T07:28:22.029 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:21 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:28:22.029 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:21 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:28:22.030 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:21 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:28:22.030 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:21 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:28:22.030 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:21 smithi137 ceph-mon[39175]: osd.2 v1:172.21.15.137:6809/1754684686 boot 2024-09-18T07:28:22.030 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:21 smithi137 ceph-mon[39175]: osdmap e18: 3 total, 3 up, 3 in 2024-09-18T07:28:22.030 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:21 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:28:22.030 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:21 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:28:22.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:21 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:28:22.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:21 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:28:22.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:21 smithi203 ceph-mon[32237]: osd.2 v1:172.21.15.137:6809/1754684686 boot 2024-09-18T07:28:22.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:21 smithi203 ceph-mon[32237]: osdmap e18: 3 total, 3 up, 3 in 2024-09-18T07:28:22.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:21 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:28:22.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:21 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:28:22.884 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57746]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-18T07:28:22.884 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57746]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T07:28:22.885 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57746]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T07:28:22.885 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57746]: pam_unix(sudo:session): session closed for user root 2024-09-18T07:28:22.885 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57749]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-18T07:28:22.885 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57749]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T07:28:22.885 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57749]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T07:28:22.885 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57749]: pam_unix(sudo:session): session closed for user root 2024-09-18T07:28:22.885 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:22 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-18T07:28:22.885 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:22 smithi137 ceph-mon[29408]: osdmap e19: 3 total, 3 up, 3 in 2024-09-18T07:28:22.886 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:22 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:28:22.886 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:22 smithi137 ceph-mon[29408]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:22.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:22 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-18T07:28:22.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:22 smithi137 ceph-mon[39175]: osdmap e19: 3 total, 3 up, 3 in 2024-09-18T07:28:22.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:22 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:28:22.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:22 smithi137 ceph-mon[39175]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:22.886 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57752]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-18T07:28:22.887 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57752]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T07:28:22.887 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57752]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T07:28:22.887 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57752]: pam_unix(sudo:session): session closed for user root 2024-09-18T07:28:22.887 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57755]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-18T07:28:22.887 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57755]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T07:28:23.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:22 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-18T07:28:23.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:22 smithi203 ceph-mon[32237]: osdmap e19: 3 total, 3 up, 3 in 2024-09-18T07:28:23.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:22 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:28:23.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:22 smithi203 ceph-mon[32237]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:23.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:22 smithi203 sudo[34176]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-18T07:28:23.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:22 smithi203 sudo[34176]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T07:28:23.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:22 smithi203 sudo[34176]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T07:28:23.008 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:22 smithi203 sudo[34176]: pam_unix(sudo:session): session closed for user root 2024-09-18T07:28:23.169 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57755]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T07:28:23.169 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57755]: pam_unix(sudo:session): session closed for user root 2024-09-18T07:28:23.170 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57758]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-18T07:28:23.170 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57758]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T07:28:23.170 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57758]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T07:28:23.170 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57758]: pam_unix(sudo:session): session closed for user root 2024-09-18T07:28:23.170 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57762]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-18T07:28:23.170 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57762]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T07:28:23.170 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57762]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T07:28:23.170 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57762]: pam_unix(sudo:session): session closed for user root 2024-09-18T07:28:23.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57765]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-18T07:28:23.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57765]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T07:28:23.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57765]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T07:28:23.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57765]: pam_unix(sudo:session): session closed for user root 2024-09-18T07:28:23.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57768]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-18T07:28:23.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57768]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T07:28:23.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57768]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T07:28:23.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:22 smithi137 sudo[57768]: pam_unix(sudo:session): session closed for user root 2024-09-18T07:28:23.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 sudo[57771]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-18T07:28:23.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 sudo[57771]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T07:28:23.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 sudo[57771]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T07:28:23.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 sudo[57771]: pam_unix(sudo:session): session closed for user root 2024-09-18T07:28:23.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 sudo[57774]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-18T07:28:23.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 sudo[57774]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T07:28:23.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 sudo[57774]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T07:28:23.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 sudo[57774]: pam_unix(sudo:session): session closed for user root 2024-09-18T07:28:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 sudo[34179]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-18T07:28:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 sudo[34179]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-18T07:28:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 sudo[34179]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-18T07:28:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 sudo[34179]: pam_unix(sudo:session): session closed for user root 2024-09-18T07:28:23.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:28:23.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:28:23.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: osdmap e20: 3 total, 3 up, 3 in 2024-09-18T07:28:23.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:23.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:28:23.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:28:23.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:28:23.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T07:28:23.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T07:28:23.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:28:23.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:28:23.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T07:28:23.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:28:23.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T07:28:23.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:23.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:28:23.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T07:28:23.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:28:23.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:28:23.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T07:28:23.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: Detected new or changed devices on smithi137 2024-09-18T07:28:23.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:23.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:23.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:23.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:23.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:23.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: Adjusting osd_memory_target on smithi137 to 5354M 2024-09-18T07:28:23.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:23.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:23.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:28:23.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:23.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:28:23.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:28:23.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: osdmap e20: 3 total, 3 up, 3 in 2024-09-18T07:28:23.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:23.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:28:23.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:28:23.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:28:23.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T07:28:23.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T07:28:23.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:28:23.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:28:23.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T07:28:23.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:28:23.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T07:28:23.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:23.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:28:23.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T07:28:23.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:28:23.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:28:23.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T07:28:23.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: Detected new or changed devices on smithi137 2024-09-18T07:28:23.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:23.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:23.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:23.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:23.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:23.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: Adjusting osd_memory_target on smithi137 to 5354M 2024-09-18T07:28:23.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:23.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:23.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:28:23.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:24.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:28:24.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:28:24.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: osdmap e20: 3 total, 3 up, 3 in 2024-09-18T07:28:24.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:24.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:28:24.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:28:24.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:28:24.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T07:28:24.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T07:28:24.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:28:24.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:28:24.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T07:28:24.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:28:24.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T07:28:24.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:24.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:28:24.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-18T07:28:24.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:28:24.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:28:24.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-18T07:28:24.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: Detected new or changed devices on smithi137 2024-09-18T07:28:24.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:24.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:24.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:24.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:24.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:24.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: Adjusting osd_memory_target on smithi137 to 5354M 2024-09-18T07:28:24.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:24.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:24.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:28:24.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:24.991 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:24 smithi137 ceph-mon[29408]: osdmap e21: 3 total, 3 up, 3 in 2024-09-18T07:28:24.991 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:24 smithi137 ceph-mon[29408]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:24.992 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:24 smithi137 ceph-mon[39175]: osdmap e21: 3 total, 3 up, 3 in 2024-09-18T07:28:24.992 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:24 smithi137 ceph-mon[39175]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:24.994 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:28:25.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi137:vg_nvme/lv_1 2024-09-18T07:28:25.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:24 smithi203 ceph-mon[32237]: osdmap e21: 3 total, 3 up, 3 in 2024-09-18T07:28:25.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:24 smithi203 ceph-mon[32237]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:25.266 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:28:26.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:25 smithi137 ceph-mon[29408]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:28:26.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:25 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:28:26.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:25 smithi137 ceph-mon[29408]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-18T07:28:26.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:25 smithi137 ceph-mon[39175]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:28:26.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:25 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:28:26.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:25 smithi137 ceph-mon[39175]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-18T07:28:26.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:25 smithi203 ceph-mon[32237]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:28:26.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:25 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:28:26.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:25 smithi203 ceph-mon[32237]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-18T07:28:26.858 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:26 smithi137 ceph-mon[29408]: from='client.14316 v1:172.21.15.137:0/1914967582' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:28:26.858 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:26 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:28:26.859 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:26 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:28:26.859 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:26 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:26.859 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:26 smithi137 ceph-mon[29408]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:26.859 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:26 smithi137 ceph-mon[39175]: from='client.14316 v1:172.21.15.137:0/1914967582' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:28:26.859 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:26 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:28:26.859 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:26 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:28:26.859 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:26 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:26.859 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:26 smithi137 ceph-mon[39175]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:26 smithi203 ceph-mon[32237]: from='client.14316 v1:172.21.15.137:0/1914967582' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:28:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:26 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:28:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:26 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:28:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:26 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:26 smithi203 ceph-mon[32237]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:28.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:28 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2634443962' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14"} : dispatch 2024-09-18T07:28:28.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:28 smithi137 ceph-mon[29408]: from='client.34205 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14"} : dispatch 2024-09-18T07:28:28.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:28 smithi137 ceph-mon[29408]: from='client.34205 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14"}]': finished 2024-09-18T07:28:28.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:28 smithi137 ceph-mon[29408]: osdmap e22: 4 total, 3 up, 4 in 2024-09-18T07:28:28.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:28 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:28.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:28 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2634443962' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14"} : dispatch 2024-09-18T07:28:28.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:28 smithi137 ceph-mon[39175]: from='client.34205 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14"} : dispatch 2024-09-18T07:28:28.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:28 smithi137 ceph-mon[39175]: from='client.34205 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14"}]': finished 2024-09-18T07:28:28.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:28 smithi137 ceph-mon[39175]: osdmap e22: 4 total, 3 up, 4 in 2024-09-18T07:28:28.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:28 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:28.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:28 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2634443962' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14"} : dispatch 2024-09-18T07:28:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:28 smithi203 ceph-mon[32237]: from='client.34205 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14"} : dispatch 2024-09-18T07:28:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:28 smithi203 ceph-mon[32237]: from='client.34205 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14"}]': finished 2024-09-18T07:28:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:28 smithi203 ceph-mon[32237]: osdmap e22: 4 total, 3 up, 4 in 2024-09-18T07:28:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:28 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:29.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:29 smithi137 ceph-mon[29408]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:29 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4024446178' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:28:29.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:29 smithi137 ceph-mon[39175]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:29.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:29 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4024446178' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:28:29.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:29 smithi203 ceph-mon[32237]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:29 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4024446178' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:28:30.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:30 smithi203 ceph-mon[32237]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:31.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:30 smithi137 ceph-mon[29408]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:31.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:30 smithi137 ceph-mon[39175]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:32.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:32 smithi137 ceph-mon[29408]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:32.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:32 smithi137 ceph-mon[39175]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:33.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:32 smithi203 ceph-mon[32237]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:35.054 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:34 smithi137 ceph-mon[29408]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:35.054 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:34 smithi137 ceph-mon[39175]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:35.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:34 smithi203 ceph-mon[32237]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:36.025 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:35 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-18T07:28:36.025 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:35 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:36.025 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:35 smithi137 ceph-mon[29408]: Deploying daemon osd.3 on smithi137 2024-09-18T07:28:36.025 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:35 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-18T07:28:36.025 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:35 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:36.025 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:35 smithi137 ceph-mon[39175]: Deploying daemon osd.3 on smithi137 2024-09-18T07:28:36.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:35 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-18T07:28:36.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:35 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:36.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:35 smithi203 ceph-mon[32237]: Deploying daemon osd.3 on smithi137 2024-09-18T07:28:36.970 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:36 smithi137 ceph-mon[29408]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:36.971 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:36 smithi137 ceph-mon[39175]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:37.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:36 smithi203 ceph-mon[32237]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:38.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:38 smithi137 ceph-mon[29408]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:38.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:38 smithi137 ceph-mon[39175]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:39.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:38 smithi203 ceph-mon[32237]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:41.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:40 smithi137 ceph-mon[29408]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:41.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:40 smithi137 ceph-mon[39175]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:41.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:40 smithi203 ceph-mon[32237]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:43.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:42 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:28:43.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:42 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:28:43.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:42 smithi137 ceph-mon[29408]: osdmap e23: 4 total, 3 up, 4 in 2024-09-18T07:28:43.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:42 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:43.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:42 smithi137 ceph-mon[29408]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:43.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:42 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:28:43.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:42 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:43.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:42 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:43.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:42 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:28:43.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:42 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:28:43.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:42 smithi137 ceph-mon[39175]: osdmap e23: 4 total, 3 up, 4 in 2024-09-18T07:28:43.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:42 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:43.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:42 smithi137 ceph-mon[39175]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:43.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:42 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:28:43.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:42 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:43.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:42 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:42 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:28:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:42 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:28:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:42 smithi203 ceph-mon[32237]: osdmap e23: 4 total, 3 up, 4 in 2024-09-18T07:28:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:42 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:42 smithi203 ceph-mon[32237]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:42 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:28:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:42 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:42 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:44.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:44 smithi137 ceph-mon[29408]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:44.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:44 smithi137 ceph-mon[39175]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:45.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:44 smithi203 ceph-mon[32237]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:47.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:46 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:47.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:46 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:47.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:46 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:47.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:46 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:28:47.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:46 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:47.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:46 smithi137 ceph-mon[29408]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:47.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:46 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:47.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:46 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:47.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:46 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:47.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:46 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:28:47.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:46 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:47.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:46 smithi137 ceph-mon[39175]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:47.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:46 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:47.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:46 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:46 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:46 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:28:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:46 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:46 smithi203 ceph-mon[32237]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:48.263 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:48 smithi137 ceph-mon[29408]: from='osd.3 v1:172.21.15.137:6813/116058392' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-18T07:28:48.263 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:48 smithi137 ceph-mon[29408]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-18T07:28:48.263 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:48 smithi137 ceph-mon[39175]: from='osd.3 v1:172.21.15.137:6813/116058392' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-18T07:28:48.263 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:48 smithi137 ceph-mon[39175]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-18T07:28:48.311 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 3 on host 'smithi137' 2024-09-18T07:28:48.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:48 smithi203 ceph-mon[32237]: from='osd.3 v1:172.21.15.137:6813/116058392' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-18T07:28:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:48 smithi203 ceph-mon[32237]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-18T07:28:48.927 DEBUG:teuthology.orchestra.run.smithi137:osd.3> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.3.service 2024-09-18T07:28:48.929 INFO:tasks.cephadm:Deploying osd.4 on smithi203 with /dev/vg_nvme/lv_4... 2024-09-18T07:28:48.930 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-18T07:28:49.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[29408]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-18T07:28:49.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[29408]: osdmap e24: 4 total, 3 up, 4 in 2024-09-18T07:28:49.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:49.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[29408]: from='osd.3 v1:172.21.15.137:6813/116058392' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:28:49.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[29408]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:28:49.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:28:49.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:49.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:49.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[29408]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:49.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[39175]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-18T07:28:49.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[39175]: osdmap e24: 4 total, 3 up, 4 in 2024-09-18T07:28:49.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:49.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[39175]: from='osd.3 v1:172.21.15.137:6813/116058392' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:28:49.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[39175]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:28:49.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:28:49.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:49.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:49.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-mon[39175]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:49.171 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-3[59869]: 2024-09-18T07:28:49.078+0000 7f4fec96f640 -1 osd.3 0 waiting for initial osdmap 2024-09-18T07:28:49.171 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:28:49 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-3[59869]: 2024-09-18T07:28:49.081+0000 7f4fe8796640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T07:28:49.172 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:28:49.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:49 smithi203 ceph-mon[32237]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-18T07:28:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:49 smithi203 ceph-mon[32237]: osdmap e24: 4 total, 3 up, 4 in 2024-09-18T07:28:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:49 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:49 smithi203 ceph-mon[32237]: from='osd.3 v1:172.21.15.137:6813/116058392' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:28:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:49 smithi203 ceph-mon[32237]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]} : dispatch 2024-09-18T07:28:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:49 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:28:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:49 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:49 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:49 smithi203 ceph-mon[32237]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:50.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:50 smithi203 ceph-mon[32237]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-18T07:28:50.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:50 smithi203 ceph-mon[32237]: osdmap e25: 4 total, 3 up, 4 in 2024-09-18T07:28:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:50 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:50 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:50.398 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:50 smithi137 ceph-mon[29408]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-18T07:28:50.398 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:50 smithi137 ceph-mon[29408]: osdmap e25: 4 total, 3 up, 4 in 2024-09-18T07:28:50.398 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:50 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:50.399 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:50 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:50.399 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:50 smithi137 ceph-mon[39175]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-18T07:28:50.399 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:50 smithi137 ceph-mon[39175]: osdmap e25: 4 total, 3 up, 4 in 2024-09-18T07:28:50.399 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:50 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:50.399 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:50 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:51.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:51 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:28:51.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:51 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:28:51.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:51.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:51 smithi137 ceph-mon[39175]: osd.3 v1:172.21.15.137:6813/116058392 boot 2024-09-18T07:28:51.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:51 smithi137 ceph-mon[39175]: osdmap e26: 4 total, 4 up, 4 in 2024-09-18T07:28:51.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:51 smithi137 ceph-mon[39175]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:51.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:51 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:28:51.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:51 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:28:51.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:51 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:51.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:51 smithi137 ceph-mon[29408]: osd.3 v1:172.21.15.137:6813/116058392 boot 2024-09-18T07:28:51.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:51 smithi137 ceph-mon[29408]: osdmap e26: 4 total, 4 up, 4 in 2024-09-18T07:28:51.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:51 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:51.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:51 smithi137 ceph-mon[29408]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:51.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:51 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:28:51.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:51 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:28:51.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:51 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:51.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:51 smithi203 ceph-mon[32237]: osd.3 v1:172.21.15.137:6813/116058392 boot 2024-09-18T07:28:51.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:51 smithi203 ceph-mon[32237]: osdmap e26: 4 total, 4 up, 4 in 2024-09-18T07:28:51.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:51 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:28:51.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:51 smithi203 ceph-mon[32237]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-18T07:28:52.196 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:52 smithi203 ceph-mon[32237]: osdmap e27: 4 total, 4 up, 4 in 2024-09-18T07:28:52.196 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:52 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:52.196 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:52 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:52.196 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:52 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:52.196 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:52 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:52.196 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:52 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:52.196 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:52 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:52.196 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:52 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:52.196 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:52 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:52.196 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:52 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:28:52.196 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:52 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:52.199 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:28:52.225 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_4 2024-09-18T07:28:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[29408]: osdmap e27: 4 total, 4 up, 4 in 2024-09-18T07:28:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:28:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[39175]: osdmap e27: 4 total, 4 up, 4 in 2024-09-18T07:28:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T07:28:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:28:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:52 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:28:52.473 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:28:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:53 smithi137 ceph-mon[29408]: Detected new or changed devices on smithi137 2024-09-18T07:28:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:53 smithi137 ceph-mon[29408]: Adjusting osd_memory_target on smithi137 to 4016M 2024-09-18T07:28:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:53 smithi137 ceph-mon[29408]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:28:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:53 smithi137 ceph-mon[39175]: Detected new or changed devices on smithi137 2024-09-18T07:28:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:53 smithi137 ceph-mon[39175]: Adjusting osd_memory_target on smithi137 to 4016M 2024-09-18T07:28:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:53 smithi137 ceph-mon[39175]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:28:53.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:53 smithi203 ceph-mon[32237]: Detected new or changed devices on smithi137 2024-09-18T07:28:53.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:53 smithi203 ceph-mon[32237]: Adjusting osd_memory_target on smithi137 to 4016M 2024-09-18T07:28:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:53 smithi203 ceph-mon[32237]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:28:54.414 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:54 smithi203 ceph-mon[32237]: from='client.24238 v1:172.21.15.203:0/1984557266' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:28:54.414 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:54 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:28:54.414 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:54 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:28:54.414 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:54 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:54.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:54 smithi137 ceph-mon[29408]: from='client.24238 v1:172.21.15.203:0/1984557266' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:28:54.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:54 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:28:54.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:54 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:28:54.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:54 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:54.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:54 smithi137 ceph-mon[39175]: from='client.24238 v1:172.21.15.203:0/1984557266' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:28:54.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:54 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:28:54.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:54 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:28:54.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:54 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:28:55.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:55 smithi137 ceph-mon[29408]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:28:55.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:55 smithi137 ceph-mon[39175]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:28:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:55 smithi203 ceph-mon[32237]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:28:56.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:56 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.203:0/545979222' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de"} : dispatch 2024-09-18T07:28:56.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:56 smithi203 ceph-mon[32237]: from='client.24244 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de"} : dispatch 2024-09-18T07:28:56.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:56 smithi203 ceph-mon[32237]: from='client.24244 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de"}]': finished 2024-09-18T07:28:56.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:56 smithi203 ceph-mon[32237]: osdmap e28: 5 total, 4 up, 5 in 2024-09-18T07:28:56.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:56 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:28:56.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:56 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.203:0/651633804' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:28:56.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:56 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.203:0/545979222' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de"} : dispatch 2024-09-18T07:28:56.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:56 smithi137 ceph-mon[29408]: from='client.24244 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de"} : dispatch 2024-09-18T07:28:56.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:56 smithi137 ceph-mon[29408]: from='client.24244 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de"}]': finished 2024-09-18T07:28:56.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:56 smithi137 ceph-mon[29408]: osdmap e28: 5 total, 4 up, 5 in 2024-09-18T07:28:56.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:56 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:28:56.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:56 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.203:0/651633804' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:28:56.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:56 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.203:0/545979222' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de"} : dispatch 2024-09-18T07:28:56.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:56 smithi137 ceph-mon[39175]: from='client.24244 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de"} : dispatch 2024-09-18T07:28:56.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:56 smithi137 ceph-mon[39175]: from='client.24244 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de"}]': finished 2024-09-18T07:28:56.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:56 smithi137 ceph-mon[39175]: osdmap e28: 5 total, 4 up, 5 in 2024-09-18T07:28:56.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:56 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:28:56.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:56 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.203:0/651633804' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:28:57.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:57 smithi137 ceph-mon[29408]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:28:57.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:57 smithi137 ceph-mon[39175]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:28:57.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:57 smithi203 ceph-mon[32237]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:28:58.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:28:58 smithi137 ceph-mon[29408]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:28:58.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:28:58 smithi137 ceph-mon[39175]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:28:59.141 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:28:58 smithi203 ceph-mon[32237]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:00.769 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:00 smithi203 ceph-mon[32237]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:01.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:00 smithi137 ceph-mon[29408]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:01.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:00 smithi137 ceph-mon[39175]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:02.934 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:02 smithi203 ceph-mon[32237]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:02.935 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:02 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-18T07:29:02.935 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:02 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:03.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:02 smithi137 ceph-mon[29408]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:03.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:02 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-18T07:29:03.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:02 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:03.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:02 smithi137 ceph-mon[39175]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:03.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:02 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-18T07:29:03.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:02 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:04.004 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:03 smithi203 ceph-mon[32237]: Deploying daemon osd.4 on smithi203 2024-09-18T07:29:04.004 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:03 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:29:04.004 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:03 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:29:04.004 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:03 smithi203 ceph-mon[32237]: osdmap e29: 5 total, 4 up, 5 in 2024-09-18T07:29:04.004 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:03 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:29:04.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:03 smithi137 ceph-mon[29408]: Deploying daemon osd.4 on smithi203 2024-09-18T07:29:04.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:03 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:29:04.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:03 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:29:04.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:03 smithi137 ceph-mon[29408]: osdmap e29: 5 total, 4 up, 5 in 2024-09-18T07:29:04.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:03 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:29:04.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:03 smithi137 ceph-mon[39175]: Deploying daemon osd.4 on smithi203 2024-09-18T07:29:04.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:03 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:29:04.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:03 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:29:04.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:03 smithi137 ceph-mon[39175]: osdmap e29: 5 total, 4 up, 5 in 2024-09-18T07:29:04.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:03 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:29:04.911 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:04 smithi203 ceph-mon[32237]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:05.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:04 smithi137 ceph-mon[39175]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:05.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:04 smithi137 ceph-mon[29408]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:06.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:06 smithi137 ceph-mon[29408]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:06.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:06 smithi137 ceph-mon[39175]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:07.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:06 smithi203 ceph-mon[32237]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:08.869 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:08 smithi203 ceph-mon[32237]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:09.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:08 smithi137 ceph-mon[29408]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:09.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:08 smithi137 ceph-mon[39175]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:10.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:10 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:29:10.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:10 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:10.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:10 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:10.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:10 smithi203 ceph-mon[32237]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:11.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:10 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:29:11.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:10 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:11.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:10 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:11.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:10 smithi137 ceph-mon[29408]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:11.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:10 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:29:11.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:10 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:11.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:10 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:11.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:10 smithi137 ceph-mon[39175]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:12.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:12 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:12.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:12 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:12.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:12 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:12.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:12 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:29:12.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:12 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:12.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:12 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:12.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:12 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:12.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:12 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:12.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:12 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:29:12.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:12 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:12.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:12 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:12.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:12 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:12.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:12 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:12.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:12 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:29:12.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:12 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:13.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:13 smithi137 ceph-mon[29408]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:13.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:13 smithi137 ceph-mon[39175]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:13.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:13 smithi203 ceph-mon[32237]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:14.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:14 smithi137 ceph-mon[39175]: from='osd.4 v1:172.21.15.203:6800/3654297954' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T07:29:14.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:14 smithi137 ceph-mon[39175]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T07:29:14.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:14 smithi137 ceph-mon[29408]: from='osd.4 v1:172.21.15.203:6800/3654297954' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T07:29:14.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:14 smithi137 ceph-mon[29408]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T07:29:14.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:14 smithi203 ceph-mon[32237]: from='osd.4 v1:172.21.15.203:6800/3654297954' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T07:29:14.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:14 smithi203 ceph-mon[32237]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T07:29:15.448 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 4 on host 'smithi203' 2024-09-18T07:29:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:15 smithi203 ceph-mon[32237]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:15 smithi203 ceph-mon[32237]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-18T07:29:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:15 smithi203 ceph-mon[32237]: from='osd.4 v1:172.21.15.203:6800/3654297954' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:29:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:15 smithi203 ceph-mon[32237]: osdmap e30: 5 total, 4 up, 5 in 2024-09-18T07:29:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:15 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:29:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:15 smithi203 ceph-mon[32237]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:29:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:15 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:29:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:15 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:15 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:16.089 DEBUG:teuthology.orchestra.run.smithi203:osd.4> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.4.service 2024-09-18T07:29:16.092 INFO:tasks.cephadm:Deploying osd.5 on smithi203 with /dev/vg_nvme/lv_3... 2024-09-18T07:29:16.092 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-18T07:29:16.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[29408]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:16.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[29408]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-18T07:29:16.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[29408]: from='osd.4 v1:172.21.15.203:6800/3654297954' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:29:16.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[29408]: osdmap e30: 5 total, 4 up, 5 in 2024-09-18T07:29:16.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:29:16.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[29408]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:29:16.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:29:16.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:16.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:16.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[39175]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:16.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[39175]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-18T07:29:16.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[39175]: from='osd.4 v1:172.21.15.203:6800/3654297954' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:29:16.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[39175]: osdmap e30: 5 total, 4 up, 5 in 2024-09-18T07:29:16.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:29:16.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[39175]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:29:16.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:29:16.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:16.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:15 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:16.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:16 smithi203 ceph-mon[32237]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-18T07:29:16.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:16 smithi203 ceph-mon[32237]: osdmap e31: 5 total, 4 up, 5 in 2024-09-18T07:29:16.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:16 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:29:16.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:16 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:29:16.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:16 smithi203 ceph-mon[32237]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:17.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:16 smithi137 ceph-mon[39175]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-18T07:29:17.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:16 smithi137 ceph-mon[39175]: osdmap e31: 5 total, 4 up, 5 in 2024-09-18T07:29:17.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:16 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:29:17.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:16 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:29:17.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:16 smithi137 ceph-mon[39175]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:17.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:16 smithi137 ceph-mon[29408]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-18T07:29:17.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:16 smithi137 ceph-mon[29408]: osdmap e31: 5 total, 4 up, 5 in 2024-09-18T07:29:17.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:16 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:29:17.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:16 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:29:17.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:16 smithi137 ceph-mon[29408]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-18T07:29:17.171 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:29:17.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:17 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:29:17.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:17 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:29:17.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:17 smithi203 ceph-mon[32237]: osd.4 v1:172.21.15.203:6800/3654297954 boot 2024-09-18T07:29:17.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:17 smithi203 ceph-mon[32237]: osdmap e32: 5 total, 5 up, 5 in 2024-09-18T07:29:17.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:17 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:29:17.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:17 smithi203 ceph-mon[32237]: osdmap e33: 5 total, 5 up, 5 in 2024-09-18T07:29:18.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:17 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:29:18.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:17 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:29:18.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:17 smithi137 ceph-mon[39175]: osd.4 v1:172.21.15.203:6800/3654297954 boot 2024-09-18T07:29:18.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:17 smithi137 ceph-mon[39175]: osdmap e32: 5 total, 5 up, 5 in 2024-09-18T07:29:18.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:17 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:29:18.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:17 smithi137 ceph-mon[39175]: osdmap e33: 5 total, 5 up, 5 in 2024-09-18T07:29:18.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:17 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:29:18.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:17 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:29:18.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:17 smithi137 ceph-mon[29408]: osd.4 v1:172.21.15.203:6800/3654297954 boot 2024-09-18T07:29:18.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:17 smithi137 ceph-mon[29408]: osdmap e32: 5 total, 5 up, 5 in 2024-09-18T07:29:18.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:17 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:29:18.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:17 smithi137 ceph-mon[29408]: osdmap e33: 5 total, 5 up, 5 in 2024-09-18T07:29:18.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:18 smithi203 ceph-mon[32237]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:18.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:18 smithi203 ceph-mon[32237]: osdmap e34: 5 total, 5 up, 5 in 2024-09-18T07:29:19.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:18 smithi137 ceph-mon[39175]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:19.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:18 smithi137 ceph-mon[39175]: osdmap e34: 5 total, 5 up, 5 in 2024-09-18T07:29:19.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:18 smithi137 ceph-mon[29408]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:19.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:18 smithi137 ceph-mon[29408]: osdmap e34: 5 total, 5 up, 5 in 2024-09-18T07:29:21.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[39175]: Detected new or changed devices on smithi203 2024-09-18T07:29:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:29:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[39175]: Adjusting osd_memory_target on smithi203 to 17088M 2024-09-18T07:29:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:29:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[39175]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:21.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[29408]: Detected new or changed devices on smithi203 2024-09-18T07:29:21.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:21.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:21.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:29:21.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[29408]: Adjusting osd_memory_target on smithi203 to 17088M 2024-09-18T07:29:21.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:21.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:21.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:29:21.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:21.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:20 smithi137 ceph-mon[29408]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:21.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:20 smithi203 ceph-mon[32237]: Detected new or changed devices on smithi203 2024-09-18T07:29:21.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:20 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:21.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:20 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:21.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:20 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:29:21.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:20 smithi203 ceph-mon[32237]: Adjusting osd_memory_target on smithi203 to 17088M 2024-09-18T07:29:21.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:20 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:21.174 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:20 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:21.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:20 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:29:21.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:20 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:21.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:20 smithi203 ceph-mon[32237]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:21.488 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:29:21.514 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_3 2024-09-18T07:29:21.765 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:29:22.799 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:22 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:29:22.800 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:22 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:29:22.800 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:22 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:22.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:22 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:29:22.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:22 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:29:22.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:22 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:22.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:22 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:29:22.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:22 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:29:22.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:22 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:23.762 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:23 smithi203 ceph-mon[32237]: from='client.24265 v1:172.21.15.203:0/2512802569' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:29:23.762 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:23 smithi203 ceph-mon[32237]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:23.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:23 smithi137 ceph-mon[29408]: from='client.24265 v1:172.21.15.203:0/2512802569' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:29:23.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:23 smithi137 ceph-mon[29408]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:23.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:23 smithi137 ceph-mon[39175]: from='client.24265 v1:172.21.15.203:0/2512802569' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:29:23.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:23 smithi137 ceph-mon[39175]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:25.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:24 smithi203 ceph-mon[32237]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T07:29:25.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:24 smithi137 ceph-mon[39175]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T07:29:25.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:24 smithi137 ceph-mon[29408]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T07:29:26.071 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:25 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.203:0/4057321604' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c"} : dispatch 2024-09-18T07:29:26.071 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:25 smithi137 ceph-mon[29408]: from='client.24271 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c"} : dispatch 2024-09-18T07:29:26.071 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:25 smithi137 ceph-mon[29408]: from='client.24271 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c"}]': finished 2024-09-18T07:29:26.071 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:25 smithi137 ceph-mon[29408]: osdmap e35: 6 total, 5 up, 6 in 2024-09-18T07:29:26.071 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:25 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:29:26.071 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:25 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.203:0/1007513004' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:29:26.072 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:25 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.203:0/4057321604' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c"} : dispatch 2024-09-18T07:29:26.072 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:25 smithi137 ceph-mon[39175]: from='client.24271 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c"} : dispatch 2024-09-18T07:29:26.072 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:25 smithi137 ceph-mon[39175]: from='client.24271 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c"}]': finished 2024-09-18T07:29:26.072 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:25 smithi137 ceph-mon[39175]: osdmap e35: 6 total, 5 up, 6 in 2024-09-18T07:29:26.072 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:25 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:29:26.072 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:25 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.203:0/1007513004' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:29:26.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:25 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.203:0/4057321604' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c"} : dispatch 2024-09-18T07:29:26.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:25 smithi203 ceph-mon[32237]: from='client.24271 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c"} : dispatch 2024-09-18T07:29:26.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:25 smithi203 ceph-mon[32237]: from='client.24271 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c"}]': finished 2024-09-18T07:29:26.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:25 smithi203 ceph-mon[32237]: osdmap e35: 6 total, 5 up, 6 in 2024-09-18T07:29:26.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:25 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:29:26.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:25 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.203:0/1007513004' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:29:27.084 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:26 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:29:27.084 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:26 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:29:27.084 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:26 smithi137 ceph-mon[29408]: osdmap e36: 6 total, 5 up, 6 in 2024-09-18T07:29:27.084 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:26 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:29:27.084 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:26 smithi137 ceph-mon[29408]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T07:29:27.085 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:26 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:29:27.085 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:26 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:29:27.085 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:26 smithi137 ceph-mon[39175]: osdmap e36: 6 total, 5 up, 6 in 2024-09-18T07:29:27.085 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:26 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:29:27.085 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:26 smithi137 ceph-mon[39175]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T07:29:27.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:26 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:29:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:26 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:29:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:26 smithi203 ceph-mon[32237]: osdmap e36: 6 total, 5 up, 6 in 2024-09-18T07:29:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:26 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:29:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:26 smithi203 ceph-mon[32237]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T07:29:28.896 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:28 smithi203 ceph-mon[32237]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T07:29:29.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:28 smithi137 ceph-mon[29408]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T07:29:29.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:28 smithi137 ceph-mon[39175]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T07:29:29.915 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:29 smithi203 ceph-mon[32237]: 1.0 scrub starts 2024-09-18T07:29:29.916 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:29 smithi203 ceph-mon[32237]: 1.0 scrub ok 2024-09-18T07:29:30.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:29 smithi137 ceph-mon[29408]: 1.0 scrub starts 2024-09-18T07:29:30.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:29 smithi137 ceph-mon[29408]: 1.0 scrub ok 2024-09-18T07:29:30.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:29 smithi137 ceph-mon[39175]: 1.0 scrub starts 2024-09-18T07:29:30.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:29 smithi137 ceph-mon[39175]: 1.0 scrub ok 2024-09-18T07:29:30.830 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:30 smithi203 ceph-mon[32237]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T07:29:31.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:30 smithi137 ceph-mon[29408]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T07:29:31.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:30 smithi137 ceph-mon[39175]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T07:29:32.413 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:32 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-18T07:29:32.414 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:32 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:32.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:32 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-18T07:29:32.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:32 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:32.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:32 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-18T07:29:32.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:32 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:33.361 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:33 smithi203 ceph-mon[32237]: Deploying daemon osd.5 on smithi203 2024-09-18T07:29:33.361 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:33 smithi203 ceph-mon[32237]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:33.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:33 smithi137 ceph-mon[29408]: Deploying daemon osd.5 on smithi203 2024-09-18T07:29:33.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:33 smithi137 ceph-mon[29408]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:33.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:33 smithi137 ceph-mon[39175]: Deploying daemon osd.5 on smithi203 2024-09-18T07:29:33.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:33 smithi137 ceph-mon[39175]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:34.881 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:34 smithi203 ceph-mon[32237]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:35.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:34 smithi137 ceph-mon[29408]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:35.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:34 smithi137 ceph-mon[39175]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:37.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:36 smithi137 ceph-mon[29408]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:37.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:36 smithi137 ceph-mon[39175]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:37.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:36 smithi203 ceph-mon[32237]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:38.845 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:38 smithi203 ceph-mon[32237]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:39.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:38 smithi137 ceph-mon[29408]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:39.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:38 smithi137 ceph-mon[39175]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:39.981 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:39 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:29:39.981 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:39 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:39.981 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:39 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:40.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:39 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:29:40.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:39 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:40.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:39 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:40.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:39 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:29:40.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:39 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:40.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:39 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:40.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:40 smithi203 ceph-mon[32237]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:41.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:40 smithi137 ceph-mon[29408]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:41.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:40 smithi137 ceph-mon[39175]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:43.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:42 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:43.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:42 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:42 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:42 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:29:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:42 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:42 smithi203 ceph-mon[32237]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:42 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:42 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:42 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:42 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:29:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:42 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:43.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:42 smithi137 ceph-mon[29408]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:42 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:42 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:42 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:42 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:29:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:42 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:42 smithi137 ceph-mon[39175]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:44.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:43 smithi203 ceph-mon[32237]: from='osd.5 v1:172.21.15.203:6804/837994222' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T07:29:44.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:43 smithi203 ceph-mon[32237]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T07:29:44.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:43 smithi137 ceph-mon[29408]: from='osd.5 v1:172.21.15.203:6804/837994222' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T07:29:44.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:43 smithi137 ceph-mon[29408]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T07:29:44.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:43 smithi137 ceph-mon[39175]: from='osd.5 v1:172.21.15.203:6804/837994222' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T07:29:44.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:43 smithi137 ceph-mon[39175]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T07:29:44.785 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 5 on host 'smithi203' 2024-09-18T07:29:45.404 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:45 smithi203 ceph-mon[32237]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-18T07:29:45.404 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:45 smithi203 ceph-mon[32237]: osdmap e37: 6 total, 5 up, 6 in 2024-09-18T07:29:45.404 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:45 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:29:45.404 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:45 smithi203 ceph-mon[32237]: from='osd.5 v1:172.21.15.203:6804/837994222' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:29:45.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:45 smithi203 ceph-mon[32237]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:29:45.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:45 smithi203 ceph-mon[32237]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:45.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:45 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:29:45.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:45 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:45.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:45 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[29408]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-18T07:29:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[29408]: osdmap e37: 6 total, 5 up, 6 in 2024-09-18T07:29:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:29:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[29408]: from='osd.5 v1:172.21.15.203:6804/837994222' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:29:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[29408]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:29:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[29408]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:29:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[39175]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-18T07:29:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[39175]: osdmap e37: 6 total, 5 up, 6 in 2024-09-18T07:29:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:29:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[39175]: from='osd.5 v1:172.21.15.203:6804/837994222' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:29:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[39175]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:29:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[39175]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-18T07:29:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:29:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:45.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:45.433 DEBUG:teuthology.orchestra.run.smithi203:osd.5> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.5.service 2024-09-18T07:29:45.435 INFO:tasks.cephadm:Deploying osd.6 on smithi203 with /dev/vg_nvme/lv_2... 2024-09-18T07:29:45.436 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-18T07:29:46.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:45 smithi137 ceph-mon[29408]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-18T07:29:46.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:45 smithi137 ceph-mon[29408]: osdmap e38: 6 total, 5 up, 6 in 2024-09-18T07:29:46.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:45 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:29:46.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:45 smithi137 ceph-mon[29408]: from='osd.5 ' entity='osd.5' 2024-09-18T07:29:46.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:45 smithi137 ceph-mon[29408]: osd.5 v1:172.21.15.203:6804/837994222 boot 2024-09-18T07:29:46.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:45 smithi137 ceph-mon[29408]: osdmap e39: 6 total, 6 up, 6 in 2024-09-18T07:29:46.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:45 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:29:46.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:45 smithi137 ceph-mon[39175]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-18T07:29:46.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:45 smithi137 ceph-mon[39175]: osdmap e38: 6 total, 5 up, 6 in 2024-09-18T07:29:46.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:45 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:29:46.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:45 smithi137 ceph-mon[39175]: from='osd.5 ' entity='osd.5' 2024-09-18T07:29:46.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:45 smithi137 ceph-mon[39175]: osd.5 v1:172.21.15.203:6804/837994222 boot 2024-09-18T07:29:46.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:45 smithi137 ceph-mon[39175]: osdmap e39: 6 total, 6 up, 6 in 2024-09-18T07:29:46.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:45 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:29:46.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:45 smithi203 ceph-mon[32237]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-18T07:29:46.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:45 smithi203 ceph-mon[32237]: osdmap e38: 6 total, 5 up, 6 in 2024-09-18T07:29:46.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:45 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:29:46.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:45 smithi203 ceph-mon[32237]: from='osd.5 ' entity='osd.5' 2024-09-18T07:29:46.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:45 smithi203 ceph-mon[32237]: osd.5 v1:172.21.15.203:6804/837994222 boot 2024-09-18T07:29:46.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:45 smithi203 ceph-mon[32237]: osdmap e39: 6 total, 6 up, 6 in 2024-09-18T07:29:46.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:45 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:29:46.607 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:29:47.109 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:46 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:29:47.109 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:46 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:29:47.109 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:46 smithi203 ceph-mon[32237]: pgmap v139: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:29:47.109 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:46 smithi203 ceph-mon[32237]: osdmap e40: 6 total, 6 up, 6 in 2024-09-18T07:29:47.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:46 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:29:47.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:46 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:29:47.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:46 smithi137 ceph-mon[39175]: pgmap v139: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:29:47.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:46 smithi137 ceph-mon[39175]: osdmap e40: 6 total, 6 up, 6 in 2024-09-18T07:29:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:46 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:29:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:46 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:29:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:46 smithi137 ceph-mon[29408]: pgmap v139: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:29:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:46 smithi137 ceph-mon[29408]: osdmap e40: 6 total, 6 up, 6 in 2024-09-18T07:29:49.290 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:48 smithi203 ceph-mon[32237]: osdmap e41: 6 total, 6 up, 6 in 2024-09-18T07:29:49.290 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:48 smithi203 ceph-mon[32237]: pgmap v142: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:29:49.291 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:48 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:49.291 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:48 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:49.291 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:48 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:29:49.291 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:48 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:29:49.291 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:48 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:49.291 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:48 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:49.291 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:48 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:29:49.291 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:48 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:49.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[39175]: osdmap e41: 6 total, 6 up, 6 in 2024-09-18T07:29:49.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[39175]: pgmap v142: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:29:49.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:49.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:49.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:29:49.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:29:49.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:49.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:49.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:29:49.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:49.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[29408]: osdmap e41: 6 total, 6 up, 6 in 2024-09-18T07:29:49.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[29408]: pgmap v142: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:29:49.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:49.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:49.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:29:49.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:29:49.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:49.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:49.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:29:49.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:48 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:29:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:49 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:29:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:49 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:29:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:49 smithi203 ceph-mon[32237]: Detected new or changed devices on smithi203 2024-09-18T07:29:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:49 smithi203 ceph-mon[32237]: Adjusting osd_memory_target on smithi203 to 8544M 2024-09-18T07:29:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:49 smithi203 ceph-mon[32237]: osdmap e42: 6 total, 6 up, 6 in 2024-09-18T07:29:50.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:49 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:29:50.494 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:49 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:29:50.494 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:49 smithi137 ceph-mon[29408]: Detected new or changed devices on smithi203 2024-09-18T07:29:50.494 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:49 smithi137 ceph-mon[29408]: Adjusting osd_memory_target on smithi203 to 8544M 2024-09-18T07:29:50.494 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:49 smithi137 ceph-mon[29408]: osdmap e42: 6 total, 6 up, 6 in 2024-09-18T07:29:50.495 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:49 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:29:50.495 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:49 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:29:50.495 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:49 smithi137 ceph-mon[39175]: Detected new or changed devices on smithi203 2024-09-18T07:29:50.495 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:49 smithi137 ceph-mon[39175]: Adjusting osd_memory_target on smithi203 to 8544M 2024-09-18T07:29:50.495 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:49 smithi137 ceph-mon[39175]: osdmap e42: 6 total, 6 up, 6 in 2024-09-18T07:29:50.495 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:29:50.508 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_2 2024-09-18T07:29:50.752 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:29:51.132 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:50 smithi203 ceph-mon[32237]: pgmap v144: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:29:51.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:50 smithi137 ceph-mon[29408]: pgmap v144: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:29:51.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:50 smithi137 ceph-mon[39175]: pgmap v144: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:29:52.127 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:51 smithi203 ceph-mon[32237]: from='client.24292 v1:172.21.15.203:0/1342903226' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:29:52.127 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:51 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:29:52.127 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:51 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:29:52.127 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:51 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:51 smithi137 ceph-mon[29408]: from='client.24292 v1:172.21.15.203:0/1342903226' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:29:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:51 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:29:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:51 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:29:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:51 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:51 smithi137 ceph-mon[39175]: from='client.24292 v1:172.21.15.203:0/1342903226' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:29:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:29:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:29:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:51 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:29:53.040 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:52 smithi203 ceph-mon[32237]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-18T07:29:53.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:52 smithi137 ceph-mon[39175]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-18T07:29:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:52 smithi137 ceph-mon[29408]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-18T07:29:54.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:53 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.203:0/1116881742' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6"} : dispatch 2024-09-18T07:29:54.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:53 smithi203 ceph-mon[32237]: from='client.24298 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6"} : dispatch 2024-09-18T07:29:54.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:53 smithi203 ceph-mon[32237]: from='client.24298 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6"}]': finished 2024-09-18T07:29:54.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:53 smithi203 ceph-mon[32237]: osdmap e43: 7 total, 6 up, 7 in 2024-09-18T07:29:54.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:29:54.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:53 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.203:0/1116881742' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6"} : dispatch 2024-09-18T07:29:54.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:53 smithi137 ceph-mon[39175]: from='client.24298 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6"} : dispatch 2024-09-18T07:29:54.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:53 smithi137 ceph-mon[39175]: from='client.24298 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6"}]': finished 2024-09-18T07:29:54.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:53 smithi137 ceph-mon[39175]: osdmap e43: 7 total, 6 up, 7 in 2024-09-18T07:29:54.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:29:54.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:53 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.203:0/1116881742' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6"} : dispatch 2024-09-18T07:29:54.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:53 smithi137 ceph-mon[29408]: from='client.24298 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6"} : dispatch 2024-09-18T07:29:54.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:53 smithi137 ceph-mon[29408]: from='client.24298 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6"}]': finished 2024-09-18T07:29:54.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:53 smithi137 ceph-mon[29408]: osdmap e43: 7 total, 6 up, 7 in 2024-09-18T07:29:54.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:29:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:54 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.203:0/3217664452' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:29:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:54 smithi137 ceph-mon[39175]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-18T07:29:55.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:54 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.203:0/3217664452' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:29:55.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:54 smithi137 ceph-mon[29408]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-18T07:29:55.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:54 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.203:0/3217664452' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:29:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:54 smithi203 ceph-mon[32237]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-18T07:29:56.321 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:55 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:29:56.321 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:55 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:29:56.321 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:55 smithi137 ceph-mon[29408]: osdmap e44: 7 total, 6 up, 7 in 2024-09-18T07:29:56.321 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:55 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:29:56.321 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:55 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:29:56.321 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:55 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:29:56.321 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:55 smithi137 ceph-mon[39175]: osdmap e44: 7 total, 6 up, 7 in 2024-09-18T07:29:56.321 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:55 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:29:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:55 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:29:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:55 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:29:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:55 smithi203 ceph-mon[32237]: osdmap e44: 7 total, 6 up, 7 in 2024-09-18T07:29:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:55 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:29:57.356 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:56 smithi203 ceph-mon[32237]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-18T07:29:57.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:56 smithi137 ceph-mon[39175]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-18T07:29:57.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:56 smithi137 ceph-mon[29408]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-18T07:29:58.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:29:58 smithi203 ceph-mon[32237]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T07:29:59.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:29:58 smithi137 ceph-mon[29408]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T07:29:59.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:29:58 smithi137 ceph-mon[39175]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-18T07:30:00.271 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:00 smithi203 ceph-mon[32237]: overall HEALTH_OK 2024-09-18T07:30:00.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:00 smithi137 ceph-mon[29408]: overall HEALTH_OK 2024-09-18T07:30:00.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:00 smithi137 ceph-mon[39175]: overall HEALTH_OK 2024-09-18T07:30:01.351 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:01 smithi203 ceph-mon[32237]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:01.351 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:01 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-18T07:30:01.351 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:01 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:01.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:01 smithi137 ceph-mon[29408]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:01.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:01 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-18T07:30:01.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:01 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:01.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:01 smithi137 ceph-mon[39175]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:01.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:01 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-18T07:30:01.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:01 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:02.364 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:02 smithi203 ceph-mon[32237]: Deploying daemon osd.6 on smithi203 2024-09-18T07:30:02.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:02 smithi137 ceph-mon[29408]: Deploying daemon osd.6 on smithi203 2024-09-18T07:30:02.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:02 smithi137 ceph-mon[39175]: Deploying daemon osd.6 on smithi203 2024-09-18T07:30:03.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:03 smithi137 ceph-mon[29408]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:03.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:03 smithi137 ceph-mon[39175]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:03.453 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:03 smithi203 ceph-mon[32237]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:05.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:04 smithi137 ceph-mon[29408]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:05.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:04 smithi137 ceph-mon[39175]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:04 smithi203 ceph-mon[32237]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:07.082 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:06 smithi203 ceph-mon[32237]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:07.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:06 smithi137 ceph-mon[29408]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:07.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:06 smithi137 ceph-mon[39175]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:08.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:08 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:08.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:08 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:08.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:08 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:08.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:08 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:08.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:08 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:08.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:08 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:08.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:08 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:08.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:08 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:08.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:08 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:09.799 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:09 smithi203 ceph-mon[32237]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:09.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:09 smithi137 ceph-mon[29408]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:09.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:09 smithi137 ceph-mon[39175]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:10.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:10 smithi203 ceph-mon[32237]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:11.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:10 smithi137 ceph-mon[29408]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:11.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:10 smithi137 ceph-mon[39175]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:12 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:12 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:12 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:12 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:12 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:12 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:12 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:12 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:12 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:12 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:12.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:12 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:12.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:12 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:12 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:12 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:12 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:13.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:13 smithi137 ceph-mon[29408]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:13.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:13 smithi137 ceph-mon[39175]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:13.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:13 smithi203 ceph-mon[32237]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:13.935 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 6 on host 'smithi203' 2024-09-18T07:30:14.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:14 smithi203 ceph-mon[32237]: from='osd.6 v1:172.21.15.203:6808/1494175466' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-18T07:30:14.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:14 smithi203 ceph-mon[32237]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-18T07:30:14.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:14 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:14.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:14 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:14.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:14 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:14.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:14 smithi137 ceph-mon[29408]: from='osd.6 v1:172.21.15.203:6808/1494175466' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-18T07:30:14.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:14 smithi137 ceph-mon[29408]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-18T07:30:14.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:14 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:14.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:14 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:14.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:14 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:14.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:14 smithi137 ceph-mon[39175]: from='osd.6 v1:172.21.15.203:6808/1494175466' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-18T07:30:14.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:14 smithi137 ceph-mon[39175]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-18T07:30:14.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:14 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:14.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:14 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:14.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:14 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:14.586 DEBUG:teuthology.orchestra.run.smithi203:osd.6> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.6.service 2024-09-18T07:30:14.589 INFO:tasks.cephadm:Deploying osd.7 on smithi203 with /dev/vg_nvme/lv_1... 2024-09-18T07:30:14.589 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-18T07:30:15.110 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:15 smithi203 ceph-mon[32237]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-18T07:30:15.110 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:15 smithi203 ceph-mon[32237]: from='osd.6 v1:172.21.15.203:6808/1494175466' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:30:15.110 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:15 smithi203 ceph-mon[32237]: osdmap e45: 7 total, 6 up, 7 in 2024-09-18T07:30:15.111 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:15 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:30:15.111 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:15 smithi203 ceph-mon[32237]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:30:15.111 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:15 smithi203 ceph-mon[32237]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:15.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:15 smithi137 ceph-mon[29408]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-18T07:30:15.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:15 smithi137 ceph-mon[29408]: from='osd.6 v1:172.21.15.203:6808/1494175466' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:30:15.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:15 smithi137 ceph-mon[29408]: osdmap e45: 7 total, 6 up, 7 in 2024-09-18T07:30:15.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:15 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:30:15.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:15 smithi137 ceph-mon[29408]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:30:15.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:15 smithi137 ceph-mon[29408]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:15 smithi137 ceph-mon[39175]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-18T07:30:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:15 smithi137 ceph-mon[39175]: from='osd.6 v1:172.21.15.203:6808/1494175466' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:30:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:15 smithi137 ceph-mon[39175]: osdmap e45: 7 total, 6 up, 7 in 2024-09-18T07:30:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:15 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:30:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:15 smithi137 ceph-mon[39175]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:30:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:15 smithi137 ceph-mon[39175]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-18T07:30:15.755 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:30:15 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:30:15.466+0000 7fe3e0f78640 -1 osd.6 0 waiting for initial osdmap 2024-09-18T07:30:15.756 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:30:15 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:30:15.473+0000 7fe3d8d97640 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T07:30:15.805 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:30:16.353 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:16 smithi203 ceph-mon[32237]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-18T07:30:16.353 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:16 smithi203 ceph-mon[32237]: osdmap e46: 7 total, 6 up, 7 in 2024-09-18T07:30:16.353 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:16 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:30:16.353 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:16 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:30:16.353 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:16 smithi203 ceph-mon[32237]: from='osd.6 ' entity='osd.6' 2024-09-18T07:30:16.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:16 smithi137 ceph-mon[29408]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-18T07:30:16.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:16 smithi137 ceph-mon[29408]: osdmap e46: 7 total, 6 up, 7 in 2024-09-18T07:30:16.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:16 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:30:16.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:16 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:30:16.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:16 smithi137 ceph-mon[29408]: from='osd.6 ' entity='osd.6' 2024-09-18T07:30:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:16 smithi137 ceph-mon[39175]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-18T07:30:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:16 smithi137 ceph-mon[39175]: osdmap e46: 7 total, 6 up, 7 in 2024-09-18T07:30:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:16 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:30:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:16 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:30:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:16 smithi137 ceph-mon[39175]: from='osd.6 ' entity='osd.6' 2024-09-18T07:30:17.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:17 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:30:17.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:17 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:30:17.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:17 smithi203 ceph-mon[32237]: osd.6 v1:172.21.15.203:6808/1494175466 boot 2024-09-18T07:30:17.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:17 smithi203 ceph-mon[32237]: osdmap e47: 7 total, 7 up, 7 in 2024-09-18T07:30:17.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:17 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:30:17.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:17 smithi203 ceph-mon[32237]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 625 GiB / 626 GiB avail 2024-09-18T07:30:17.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:17 smithi203 ceph-mon[32237]: osdmap e48: 7 total, 7 up, 7 in 2024-09-18T07:30:17.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:17 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:30:17.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:17 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:30:17.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:17 smithi137 ceph-mon[29408]: osd.6 v1:172.21.15.203:6808/1494175466 boot 2024-09-18T07:30:17.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:17 smithi137 ceph-mon[29408]: osdmap e47: 7 total, 7 up, 7 in 2024-09-18T07:30:17.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:17 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:30:17.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:17 smithi137 ceph-mon[29408]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 625 GiB / 626 GiB avail 2024-09-18T07:30:17.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:17 smithi137 ceph-mon[29408]: osdmap e48: 7 total, 7 up, 7 in 2024-09-18T07:30:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:17 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:30:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:17 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:30:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:17 smithi137 ceph-mon[39175]: osd.6 v1:172.21.15.203:6808/1494175466 boot 2024-09-18T07:30:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:17 smithi137 ceph-mon[39175]: osdmap e47: 7 total, 7 up, 7 in 2024-09-18T07:30:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:17 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:30:17.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:17 smithi137 ceph-mon[39175]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 625 GiB / 626 GiB avail 2024-09-18T07:30:17.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:17 smithi137 ceph-mon[39175]: osdmap e48: 7 total, 7 up, 7 in 2024-09-18T07:30:19.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[39175]: osdmap e49: 7 total, 7 up, 7 in 2024-09-18T07:30:19.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[39175]: Detected new or changed devices on smithi203 2024-09-18T07:30:19.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:19.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:19.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:19.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:19.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:19.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[39175]: Adjusting osd_memory_target on smithi203 to 5696M 2024-09-18T07:30:19.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:19.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:19.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:19.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:19.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[39175]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 625 GiB / 626 GiB avail 2024-09-18T07:30:19.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[29408]: osdmap e49: 7 total, 7 up, 7 in 2024-09-18T07:30:19.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[29408]: Detected new or changed devices on smithi203 2024-09-18T07:30:19.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:19.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:19.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:19.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:19.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:19.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[29408]: Adjusting osd_memory_target on smithi203 to 5696M 2024-09-18T07:30:19.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:19.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:19.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:19.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:19.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:19 smithi137 ceph-mon[29408]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 625 GiB / 626 GiB avail 2024-09-18T07:30:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:19 smithi203 ceph-mon[32237]: osdmap e49: 7 total, 7 up, 7 in 2024-09-18T07:30:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:19 smithi203 ceph-mon[32237]: Detected new or changed devices on smithi203 2024-09-18T07:30:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:19 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:19 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:19 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:19 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:19 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:19.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:19 smithi203 ceph-mon[32237]: Adjusting osd_memory_target on smithi203 to 5696M 2024-09-18T07:30:19.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:19 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:19.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:19 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:19.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:19 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:19.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:19 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:19.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:19 smithi203 ceph-mon[32237]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 625 GiB / 626 GiB avail 2024-09-18T07:30:19.883 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-18T07:30:19.911 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_1 2024-09-18T07:30:20.159 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:30:20.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:20 smithi203 ceph-mon[32237]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 625 GiB / 626 GiB avail 2024-09-18T07:30:21.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:20 smithi137 ceph-mon[29408]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 625 GiB / 626 GiB avail 2024-09-18T07:30:21.174 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:20 smithi137 ceph-mon[39175]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 625 GiB / 626 GiB avail 2024-09-18T07:30:22.083 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:21 smithi203 ceph-mon[32237]: from='client.24319 v1:172.21.15.203:0/2955210931' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:30:22.083 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:21 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:30:22.083 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:21 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:30:22.083 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:21 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:22.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:21 smithi137 ceph-mon[29408]: from='client.24319 v1:172.21.15.203:0/2955210931' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:30:22.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:21 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:30:22.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:21 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:30:22.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:21 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:22.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:21 smithi137 ceph-mon[39175]: from='client.24319 v1:172.21.15.203:0/2955210931' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:30:22.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:21 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-18T07:30:22.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:21 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-18T07:30:22.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:21 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:23.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:22 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:30:23.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:22 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:30:23.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:22 smithi137 ceph-mon[29408]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:23.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:22 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:30:23.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:22 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:30:23.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:22 smithi137 ceph-mon[39175]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:23.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:22 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:30:23.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:22 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:30:23.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:22 smithi203 ceph-mon[32237]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:23.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:23 smithi203 ceph-mon[32237]: osdmap e50: 7 total, 7 up, 7 in 2024-09-18T07:30:23.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:23 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.203:0/1662642447' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a"} : dispatch 2024-09-18T07:30:23.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:23 smithi203 ceph-mon[32237]: from='client.24325 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a"} : dispatch 2024-09-18T07:30:23.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:23 smithi203 ceph-mon[32237]: from='client.24325 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a"}]': finished 2024-09-18T07:30:23.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:23 smithi203 ceph-mon[32237]: osdmap e51: 8 total, 7 up, 8 in 2024-09-18T07:30:23.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:23 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:23.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:23 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:30:23.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:23 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:30:23.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:23 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.203:0/4093066521' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:30:24.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[29408]: osdmap e50: 7 total, 7 up, 7 in 2024-09-18T07:30:24.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.203:0/1662642447' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a"} : dispatch 2024-09-18T07:30:24.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[29408]: from='client.24325 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a"} : dispatch 2024-09-18T07:30:24.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[29408]: from='client.24325 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a"}]': finished 2024-09-18T07:30:24.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[29408]: osdmap e51: 8 total, 7 up, 8 in 2024-09-18T07:30:24.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:24.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:30:24.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:30:24.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.203:0/4093066521' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:30:24.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[39175]: osdmap e50: 7 total, 7 up, 7 in 2024-09-18T07:30:24.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.203:0/1662642447' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a"} : dispatch 2024-09-18T07:30:24.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[39175]: from='client.24325 ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a"} : dispatch 2024-09-18T07:30:24.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[39175]: from='client.24325 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a"}]': finished 2024-09-18T07:30:24.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[39175]: osdmap e51: 8 total, 7 up, 8 in 2024-09-18T07:30:24.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:24.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:30:24.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:30:24.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:23 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.203:0/4093066521' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-18T07:30:25.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:25 smithi137 ceph-mon[29408]: osdmap e52: 8 total, 7 up, 8 in 2024-09-18T07:30:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:25 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:25 smithi137 ceph-mon[29408]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:25.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:25 smithi137 ceph-mon[39175]: osdmap e52: 8 total, 7 up, 8 in 2024-09-18T07:30:25.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:25 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:25.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:25 smithi137 ceph-mon[39175]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:25.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:25 smithi203 ceph-mon[32237]: osdmap e52: 8 total, 7 up, 8 in 2024-09-18T07:30:25.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:25 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:25.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:25 smithi203 ceph-mon[32237]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:27.125 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:26 smithi203 ceph-mon[32237]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:27.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:26 smithi137 ceph-mon[29408]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:27.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:26 smithi137 ceph-mon[39175]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:29.141 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:28 smithi203 ceph-mon[32237]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:29.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:28 smithi137 ceph-mon[29408]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:29.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:28 smithi137 ceph-mon[39175]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:30.965 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:30 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-18T07:30:30.966 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:30 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:30.966 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:30 smithi203 ceph-mon[32237]: Deploying daemon osd.7 on smithi203 2024-09-18T07:30:30.966 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:30 smithi203 ceph-mon[32237]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:31.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:30 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-18T07:30:31.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:30 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:31.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:30 smithi137 ceph-mon[29408]: Deploying daemon osd.7 on smithi203 2024-09-18T07:30:31.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:30 smithi137 ceph-mon[29408]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:31.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:30 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-18T07:30:31.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:30 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:31.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:30 smithi137 ceph-mon[39175]: Deploying daemon osd.7 on smithi203 2024-09-18T07:30:31.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:30 smithi137 ceph-mon[39175]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:32.922 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:32 smithi203 ceph-mon[32237]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:33.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:32 smithi137 ceph-mon[29408]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:33.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:32 smithi137 ceph-mon[39175]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:35.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:34 smithi137 ceph-mon[39175]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:35.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:34 smithi137 ceph-mon[29408]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:35.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:34 smithi203 ceph-mon[32237]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:37.065 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:36 smithi203 ceph-mon[32237]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:37.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:36 smithi137 ceph-mon[29408]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:37.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:36 smithi137 ceph-mon[39175]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:38.406 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:38 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:38.406 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:38 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:38.406 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:38 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:38.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:38 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:38.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:38 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:38.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:38 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:38.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:38 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:38.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:38 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:38.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:38 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:39.393 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:39 smithi203 ceph-mon[32237]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:39.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:39 smithi137 ceph-mon[29408]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:39.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:39 smithi137 ceph-mon[39175]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:41.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:41 smithi137 ceph-mon[29408]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:41.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:41 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:41.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:41 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:41.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:41 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:41.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:41 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:41.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:41 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:41.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:41 smithi137 ceph-mon[39175]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:41.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:41 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:41.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:41 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:41.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:41 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:41.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:41 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:41.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:41 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:41.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:41 smithi203 ceph-mon[32237]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:41.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:41 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:41.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:41 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:41.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:41 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:41.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:41 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:41.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:41 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:43.080 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 7 on host 'smithi203' 2024-09-18T07:30:43.707 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:43 smithi203 ceph-mon[32237]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:43.707 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:43 smithi203 ceph-mon[32237]: from='osd.7 v1:172.21.15.203:6812/2670809315' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-18T07:30:43.707 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:43 smithi203 ceph-mon[32237]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-18T07:30:43.707 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:43 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:43.707 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:43 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:43.707 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:43 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:43.734 DEBUG:teuthology.orchestra.run.smithi203:osd.7> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.7.service 2024-09-18T07:30:43.737 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-18T07:30:43.737 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-18T07:30:43.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:43 smithi137 ceph-mon[29408]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:43.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:43 smithi137 ceph-mon[29408]: from='osd.7 v1:172.21.15.203:6812/2670809315' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-18T07:30:43.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:43 smithi137 ceph-mon[29408]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-18T07:30:43.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:43 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:43.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:43 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:43.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:43 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:43.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:43 smithi137 ceph-mon[39175]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:43.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:43 smithi137 ceph-mon[39175]: from='osd.7 v1:172.21.15.203:6812/2670809315' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-18T07:30:43.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:43 smithi137 ceph-mon[39175]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-18T07:30:43.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:43 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:43.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:43 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:43.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:43 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:43.998 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:30:44.621 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:30:44.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:44 smithi203 ceph-mon[32237]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-18T07:30:44.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:44 smithi203 ceph-mon[32237]: osdmap e53: 8 total, 7 up, 8 in 2024-09-18T07:30:44.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:44 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:44.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:44 smithi203 ceph-mon[32237]: from='osd.7 v1:172.21.15.203:6812/2670809315' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:30:44.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:44 smithi203 ceph-mon[32237]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:30:44.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:44 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3587594344' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-18T07:30:44.961 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:30:44 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:30:44.682+0000 7f9b8bca5640 -1 osd.7 0 waiting for initial osdmap 2024-09-18T07:30:44.961 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:30:44 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:30:44.688+0000 7f9b87acc640 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T07:30:45.127 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:44 smithi137 ceph-mon[29408]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-18T07:30:45.128 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:44 smithi137 ceph-mon[29408]: osdmap e53: 8 total, 7 up, 8 in 2024-09-18T07:30:45.128 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:44 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:45.128 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:44 smithi137 ceph-mon[29408]: from='osd.7 v1:172.21.15.203:6812/2670809315' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:30:45.128 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:44 smithi137 ceph-mon[29408]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:30:45.128 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:44 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3587594344' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-18T07:30:45.128 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:44 smithi137 ceph-mon[39175]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-18T07:30:45.128 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:44 smithi137 ceph-mon[39175]: osdmap e53: 8 total, 7 up, 8 in 2024-09-18T07:30:45.128 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:44 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:45.128 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:44 smithi137 ceph-mon[39175]: from='osd.7 v1:172.21.15.203:6812/2670809315' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:30:45.129 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:44 smithi137 ceph-mon[39175]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:30:45.129 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3587594344' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-18T07:30:45.435 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":53,"num_osds":8,"num_up_osds":7,"osd_up_since":1726644616,"num_in_osds":8,"osd_in_since":1726644623,"num_remapped_pgs":0} 2024-09-18T07:30:45.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:45 smithi137 ceph-mon[39175]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:45.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:45 smithi137 ceph-mon[39175]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-18T07:30:45.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:45 smithi137 ceph-mon[39175]: osdmap e54: 8 total, 7 up, 8 in 2024-09-18T07:30:45.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:45 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:45.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:45 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:45.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:45 smithi137 ceph-mon[39175]: osd.7 v1:172.21.15.203:6812/2670809315 boot 2024-09-18T07:30:45.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:45 smithi137 ceph-mon[39175]: osdmap e55: 8 total, 8 up, 8 in 2024-09-18T07:30:45.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:45 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:45.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:45 smithi137 ceph-mon[29408]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:45.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:45 smithi137 ceph-mon[29408]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-18T07:30:45.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:45 smithi137 ceph-mon[29408]: osdmap e54: 8 total, 7 up, 8 in 2024-09-18T07:30:45.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:45 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:45.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:45 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:45.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:45 smithi137 ceph-mon[29408]: osd.7 v1:172.21.15.203:6812/2670809315 boot 2024-09-18T07:30:45.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:45 smithi137 ceph-mon[29408]: osdmap e55: 8 total, 8 up, 8 in 2024-09-18T07:30:45.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:45 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:45.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:45 smithi203 ceph-mon[32237]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-18T07:30:45.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:45 smithi203 ceph-mon[32237]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-09-18T07:30:45.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:45 smithi203 ceph-mon[32237]: osdmap e54: 8 total, 7 up, 8 in 2024-09-18T07:30:45.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:45 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:45.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:45 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:45.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:45 smithi203 ceph-mon[32237]: osd.7 v1:172.21.15.203:6812/2670809315 boot 2024-09-18T07:30:45.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:45 smithi203 ceph-mon[32237]: osdmap e55: 8 total, 8 up, 8 in 2024-09-18T07:30:45.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:45 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:30:46.436 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-18T07:30:46.682 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:30:47.010 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:30:47.010 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:30:47.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[29408]: osdmap e56: 8 total, 8 up, 8 in 2024-09-18T07:30:47.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[29408]: Detected new or changed devices on smithi203 2024-09-18T07:30:47.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:47.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:47.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:47.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:47.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:47.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:47.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[29408]: Adjusting osd_memory_target on smithi203 to 4272M 2024-09-18T07:30:47.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:47.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:47.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:47.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:47.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[29408]: pgmap v186: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-18T07:30:47.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:30:47.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:30:47.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[39175]: osdmap e56: 8 total, 8 up, 8 in 2024-09-18T07:30:47.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[39175]: Detected new or changed devices on smithi203 2024-09-18T07:30:47.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:47.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:47.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:47.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:47.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:47.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:47.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[39175]: Adjusting osd_memory_target on smithi203 to 4272M 2024-09-18T07:30:47.013 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:47.013 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:47.013 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:47.013 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:47.013 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:46 smithi137 ceph-mon[39175]: pgmap v186: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-18T07:30:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:46 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:30:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:46 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:30:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:46 smithi203 ceph-mon[32237]: osdmap e56: 8 total, 8 up, 8 in 2024-09-18T07:30:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:46 smithi203 ceph-mon[32237]: Detected new or changed devices on smithi203 2024-09-18T07:30:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:46 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:46 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:46 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:46 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:46 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:46 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T07:30:47.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:46 smithi203 ceph-mon[32237]: Adjusting osd_memory_target on smithi203 to 4272M 2024-09-18T07:30:47.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:46 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:47.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:46 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:47.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:46 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:47.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:46 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:47.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:46 smithi203 ceph-mon[32237]: pgmap v186: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-18T07:30:47.277 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:30:48.065 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:47 smithi137 ceph-mon[29408]: osdmap e57: 8 total, 8 up, 8 in 2024-09-18T07:30:48.065 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:47 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/367178244' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-18T07:30:48.065 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:47 smithi137 ceph-mon[29408]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-18T07:30:48.065 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:47 smithi137 ceph-mon[39175]: osdmap e57: 8 total, 8 up, 8 in 2024-09-18T07:30:48.065 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:47 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/367178244' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-18T07:30:48.065 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:47 smithi137 ceph-mon[39175]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-18T07:30:48.094 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1726644644,"num_in_osds":8,"osd_in_since":1726644623,"num_remapped_pgs":0} 2024-09-18T07:30:48.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-18T07:30:48.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:47 smithi203 ceph-mon[32237]: osdmap e57: 8 total, 8 up, 8 in 2024-09-18T07:30:48.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:47 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/367178244' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-18T07:30:48.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:47 smithi203 ceph-mon[32237]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-18T07:30:48.339 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:30:48.952 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:30:48.952 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":57,"fsid":"f7ef099c-758e-11ef-bceb-c7b262605968","created":"2024-09-18T07:25:19.662367+0000","modified":"2024-09-18T07:30:46.712637+0000","last_up_change":"2024-09-18T07:30:44.708496+0000","last_in_change":"2024-09-18T07:30:23.052551+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T07:28:20.899655+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268cd580-c931-4992-bb93-d6f63c2bfe88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6801","nonce":2811444477}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6802","nonce":2811444477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6804","nonce":2811444477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6803","nonce":2811444477}]},"public_addr":"172.21.15.137:6801/2811444477","cluster_addr":"172.21.15.137:6802/2811444477","heartbeat_back_addr":"172.21.15.137:6804/2811444477","heartbeat_front_addr":"172.21.15.137:6803/2811444477","state":["exists","up"]},{"osd":1,"uuid":"b06907bc-b499-46c0-8667-c5bdc0d85575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6805","nonce":874102487}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6806","nonce":874102487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6808","nonce":874102487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6807","nonce":874102487}]},"public_addr":"172.21.15.137:6805/874102487","cluster_addr":"172.21.15.137:6806/874102487","heartbeat_back_addr":"172.21.15.137:6808/874102487","heartbeat_front_addr":"172.21.15.137:6807/874102487","state":["exists","up"]},{"osd":2,"uuid":"480ad6c5-0e0b-4405-bbe2-381682d00bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6809","nonce":1754684686}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6810","nonce":1754684686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6812","nonce":1754684686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6811","nonce":1754684686}]},"public_addr":"172.21.15.137:6809/1754684686","cluster_addr":"172.21.15.137:6810/1754684686","heartbeat_back_addr":"172.21.15.137:6812/1754684686","heartbeat_front_addr":"172.21.15.137:6811/1754684686","state":["exists","up"]},{"osd":3,"uuid":"178be3e0-f07b-4856-807b-2a1181e48a14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6813","nonce":116058392}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6814","nonce":116058392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6816","nonce":116058392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6815","nonce":116058392}]},"public_addr":"172.21.15.137:6813/116058392","cluster_addr":"172.21.15.137:6814/116058392","heartbeat_back_addr":"172.21.15.137:6816/116058392","heartbeat_front_addr":"172.21.15.137:6815/116058392","state":["exists","up"]},{"osd":4,"uuid":"7547102c-951a-4077-a094-fb8a65bfe6de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":3654297954}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":3654297954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":3654297954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":3654297954}]},"public_addr":"172.21.15.203:6800/3654297954","cluster_addr":"172.21.15.203:6801/3654297954","heartbeat_back_addr":"172.21.15.203:6803/3654297954","heartbeat_front_addr":"172.21.15.203:6802/3654297954","state":["exists","up"]},{"osd":5,"uuid":"49c17901-8102-48a4-9348-fa1bdb63032c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":837994222}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":837994222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":837994222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":837994222}]},"public_addr":"172.21.15.203:6804/837994222","cluster_addr":"172.21.15.203:6805/837994222","heartbeat_back_addr":"172.21.15.203:6807/837994222","heartbeat_front_addr":"172.21.15.203:6806/837994222","state":["exists","up"]},{"osd":6,"uuid":"75f0ead4-e981-4a15-b1c1-d83dce871ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":1494175466}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":1494175466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":1494175466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":1494175466}]},"public_addr":"172.21.15.203:6808/1494175466","cluster_addr":"172.21.15.203:6809/1494175466","heartbeat_back_addr":"172.21.15.203:6811/1494175466","heartbeat_front_addr":"172.21.15.203:6810/1494175466","state":["exists","up"]},{"osd":7,"uuid":"c0dbf299-6798-4b99-9a85-62b123971f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6812","nonce":2670809315}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6813","nonce":2670809315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6815","nonce":2670809315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6814","nonce":2670809315}]},"public_addr":"172.21.15.203:6812/2670809315","cluster_addr":"172.21.15.203:6813/2670809315","heartbeat_back_addr":"172.21.15.203:6815/2670809315","heartbeat_front_addr":"172.21.15.203:6814/2670809315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:29:48.818217+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:30:21.954128+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:29:25.677218+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:29:54.936280+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:30:23.581624+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:29:44.944010+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:30:14.117573+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:30:43.753862+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:0/4028304023":"2024-09-19T07:26:06.595443+0000","172.21.15.137:6800/1482147392":"2024-09-19T07:26:06.595443+0000","172.21.15.137:0/3387637412":"2024-09-19T07:26:06.595443+0000","172.21.15.137:0/2101286111":"2024-09-19T07:25:41.661328+0000","172.21.15.137:0/1821482181":"2024-09-19T07:25:41.661328+0000","172.21.15.137:0/3490817671":"2024-09-19T07:26:06.595443+0000","172.21.15.137:0/334967443":"2024-09-19T07:25:41.661328+0000","172.21.15.137:6800/3636177939":"2024-09-19T07:25:41.661328+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T07:30:49.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:48 smithi137 ceph-mon[29408]: pgmap v188: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-18T07:30:49.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:48 smithi137 ceph-mon[39175]: pgmap v188: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-18T07:30:49.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:48 smithi203 ceph-mon[32237]: pgmap v188: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-18T07:30:49.736 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T07:28:20.899655+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-18T07:30:49.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-18T07:30:49.981 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:30:50.101 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:49 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1803727951' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T07:30:50.101 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:49 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:30:50.101 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:49 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:30:50.102 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:49 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1803727951' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T07:30:50.102 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:49 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:30:50.102 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:49 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:30:50.112 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:49 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1803727951' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T07:30:50.112 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:49 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:30:50.112 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:49 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:30:50.597 INFO:teuthology.orchestra.run.smithi137.stdout:pg_num: 1 2024-09-18T07:30:51.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:50 smithi137 ceph-mon[29408]: osdmap e58: 8 total, 8 up, 8 in 2024-09-18T07:30:51.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:50 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2833274034' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-18T07:30:51.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:50 smithi137 ceph-mon[29408]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-18T07:30:51.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:50 smithi137 ceph-mon[39175]: osdmap e58: 8 total, 8 up, 8 in 2024-09-18T07:30:51.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:50 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2833274034' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-18T07:30:51.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:50 smithi137 ceph-mon[39175]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-18T07:30:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:50 smithi203 ceph-mon[32237]: osdmap e58: 8 total, 8 up, 8 in 2024-09-18T07:30:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:50 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2833274034' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-18T07:30:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:50 smithi203 ceph-mon[32237]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-18T07:30:51.324 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi137 2024-09-18T07:30:51.325 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi137=foo.a' 2024-09-18T07:30:51.575 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:30:51.956 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:51 smithi203 ceph-mon[32237]: 1.0 scrub starts 2024-09-18T07:30:51.956 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:51 smithi203 ceph-mon[32237]: 1.0 scrub ok 2024-09-18T07:30:52.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:51 smithi137 ceph-mon[29408]: 1.0 scrub starts 2024-09-18T07:30:52.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:51 smithi137 ceph-mon[29408]: 1.0 scrub ok 2024-09-18T07:30:52.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:51 smithi137 ceph-mon[39175]: 1.0 scrub starts 2024-09-18T07:30:52.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:51 smithi137 ceph-mon[39175]: 1.0 scrub ok 2024-09-18T07:30:52.248 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled rgw.foo.a update... 2024-09-18T07:30:52.845 DEBUG:teuthology.orchestra.run.smithi137:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@rgw.foo.a.service 2024-09-18T07:30:52.848 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi203 2024-09-18T07:30:52.848 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-09-18T07:30:53.090 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:30:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:53 smithi203 ceph-mon[32237]: from='client.24370 v1:172.21.15.203:0/3761536070' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi137=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:30:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:53 smithi203 ceph-mon[32237]: Saving service rgw.foo.a spec with placement smithi137=foo.a;count:1 2024-09-18T07:30:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:53.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-18T07:30:53.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-18T07:30:53.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:53.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:53 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:53.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:53 smithi203 ceph-mon[32237]: Deploying daemon rgw.foo.a on smithi137 2024-09-18T07:30:53.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:53 smithi203 ceph-mon[32237]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-18T07:30:53.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[39175]: from='client.24370 v1:172.21.15.203:0/3761536070' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi137=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:30:53.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[39175]: Saving service rgw.foo.a spec with placement smithi137=foo.a;count:1 2024-09-18T07:30:53.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:53.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:53.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:53.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:53.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:53.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-18T07:30:53.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-18T07:30:53.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:53.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:53.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[39175]: Deploying daemon rgw.foo.a on smithi137 2024-09-18T07:30:53.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[39175]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-18T07:30:53.662 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[29408]: from='client.24370 v1:172.21.15.203:0/3761536070' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi137=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:30:53.662 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[29408]: Saving service rgw.foo.a spec with placement smithi137=foo.a;count:1 2024-09-18T07:30:53.662 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:53.662 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:53.662 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:53.662 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:53.663 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:53.663 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-18T07:30:53.663 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-18T07:30:53.663 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:53.663 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:53.663 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[29408]: Deploying daemon rgw.foo.a on smithi137 2024-09-18T07:30:53.663 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:53 smithi137 ceph-mon[29408]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-18T07:30:54.248 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 18 07:30:54 smithi137 systemd[1]: Starting Ceph rgw.foo.a for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:30:54.276 INFO:teuthology.orchestra.run.smithi203.stderr:pool 'datapool' created 2024-09-18T07:30:54.513 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:54 smithi137 ceph-mon[29408]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-18T07:30:54.513 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:54 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:30:54.513 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:54 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.203:0/2616451221' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-18T07:30:54.513 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:54 smithi137 ceph-mon[29408]: from='client.24376 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-18T07:30:54.513 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:54 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:30:54.513 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:54 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:30:54.514 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:54 smithi137 ceph-mon[39175]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-18T07:30:54.514 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:54 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:30:54.514 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:54 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.203:0/2616451221' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-18T07:30:54.514 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:54 smithi137 ceph-mon[39175]: from='client.24376 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-18T07:30:54.514 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:54 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:30:54.514 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:54 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:30:54.514 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 18 07:30:54 smithi137 podman[64588]: 2024-09-18 07:30:54.241125132 +0000 UTC m=+0.015984003 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:30:54.514 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 18 07:30:54 smithi137 podman[64588]: 2024-09-18 07:30:54.361225774 +0000 UTC m=+0.136084644 container create cb85a1a7c9de62e094f9bd54956ca7d943c0f85562d0ad2abce5b3ceff0b26a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-rgw-foo-a, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-18T07:30:54.574 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:54 smithi203 ceph-mon[32237]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-18T07:30:54.574 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:54 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:30:54.574 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:54 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.203:0/2616451221' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-18T07:30:54.574 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:54 smithi203 ceph-mon[32237]: from='client.24376 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-18T07:30:54.574 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:54 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:30:54.574 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:54 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:30:54.907 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- rbd pool init datapool 2024-09-18T07:30:54.919 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 18 07:30:54 smithi137 podman[64588]: 2024-09-18 07:30:54.630302601 +0000 UTC m=+0.405161481 container init cb85a1a7c9de62e094f9bd54956ca7d943c0f85562d0ad2abce5b3ceff0b26a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-rgw-foo-a, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-18T07:30:54.919 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 18 07:30:54 smithi137 podman[64588]: 2024-09-18 07:30:54.633331786 +0000 UTC m=+0.408190660 container start cb85a1a7c9de62e094f9bd54956ca7d943c0f85562d0ad2abce5b3ceff0b26a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-rgw-foo-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731) 2024-09-18T07:30:54.919 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 18 07:30:54 smithi137 bash[64588]: cb85a1a7c9de62e094f9bd54956ca7d943c0f85562d0ad2abce5b3ceff0b26a9 2024-09-18T07:30:54.919 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 18 07:30:54 smithi137 systemd[1]: Started Ceph rgw.foo.a for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:30:55.152 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:30:55.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[29408]: from='client.24376 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-18T07:30:55.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[29408]: osdmap e59: 8 total, 8 up, 8 in 2024-09-18T07:30:55.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[29408]: pgmap v193: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-18T07:30:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[39175]: from='client.24376 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-18T07:30:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[39175]: osdmap e59: 8 total, 8 up, 8 in 2024-09-18T07:30:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[39175]: pgmap v193: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-18T07:30:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:55 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:55.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:55 smithi203 ceph-mon[32237]: from='client.24376 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-18T07:30:55.523 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:55 smithi203 ceph-mon[32237]: osdmap e59: 8 total, 8 up, 8 in 2024-09-18T07:30:55.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:55 smithi203 ceph-mon[32237]: pgmap v193: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-18T07:30:55.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:55 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:55.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:55 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:55.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:55 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:55.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:55 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:55.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:55 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:55.524 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:55 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:30:56.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[29408]: Saving service rgw.foo.a spec with placement smithi137=foo.a;count:1 2024-09-18T07:30:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[29408]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:30:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[29408]: osdmap e60: 8 total, 8 up, 8 in 2024-09-18T07:30:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/56998061' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-18T07:30:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[29408]: from='client.34364 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-18T07:30:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.203:0/521323987' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-18T07:30:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[29408]: from='client.24391 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-18T07:30:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[29408]: from='client.34364 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-18T07:30:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[29408]: from='client.24391 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-18T07:30:56.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[29408]: osdmap e61: 8 total, 8 up, 8 in 2024-09-18T07:30:56.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[39175]: Saving service rgw.foo.a spec with placement smithi137=foo.a;count:1 2024-09-18T07:30:56.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[39175]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:30:56.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[39175]: osdmap e60: 8 total, 8 up, 8 in 2024-09-18T07:30:56.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/56998061' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-18T07:30:56.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[39175]: from='client.34364 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-18T07:30:56.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.203:0/521323987' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-18T07:30:56.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[39175]: from='client.24391 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-18T07:30:56.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:56.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:56.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:56.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:56.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[39175]: from='client.34364 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-18T07:30:56.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[39175]: from='client.24391 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-18T07:30:56.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:56 smithi137 ceph-mon[39175]: osdmap e61: 8 total, 8 up, 8 in 2024-09-18T07:30:56.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:56 smithi203 ceph-mon[32237]: Saving service rgw.foo.a spec with placement smithi137=foo.a;count:1 2024-09-18T07:30:56.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:56 smithi203 ceph-mon[32237]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:30:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:56 smithi203 ceph-mon[32237]: osdmap e60: 8 total, 8 up, 8 in 2024-09-18T07:30:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:56 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/56998061' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-18T07:30:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:56 smithi203 ceph-mon[32237]: from='client.34364 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-18T07:30:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:56 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.203:0/521323987' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-18T07:30:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:56 smithi203 ceph-mon[32237]: from='client.24391 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-18T07:30:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:56 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:56 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:56 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:30:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:56 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:30:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:56 smithi203 ceph-mon[32237]: from='client.34364 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-18T07:30:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:56 smithi203 ceph-mon[32237]: from='client.24391 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-18T07:30:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:56 smithi203 ceph-mon[32237]: osdmap e61: 8 total, 8 up, 8 in 2024-09-18T07:30:57.573 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:57 smithi137 ceph-mon[29408]: Checking dashboard <-> RGW credentials 2024-09-18T07:30:57.573 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:57 smithi137 ceph-mon[29408]: pgmap v196: 36 pgs: 25 creating+peering, 3 active+clean, 8 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-18T07:30:57.573 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:57 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:57.574 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:57 smithi137 ceph-mon[39175]: Checking dashboard <-> RGW credentials 2024-09-18T07:30:57.574 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:57 smithi137 ceph-mon[39175]: pgmap v196: 36 pgs: 25 creating+peering, 3 active+clean, 8 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-18T07:30:57.574 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:57 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:57.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:57 smithi203 ceph-mon[32237]: Checking dashboard <-> RGW credentials 2024-09-18T07:30:57.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:57 smithi203 ceph-mon[32237]: pgmap v196: 36 pgs: 25 creating+peering, 3 active+clean, 8 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-18T07:30:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:57 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:30:58.632 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:58 smithi203 ceph-mon[32237]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:30:58.633 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:58 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:30:58.633 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:58 smithi203 ceph-mon[32237]: osdmap e62: 8 total, 8 up, 8 in 2024-09-18T07:30:58.633 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:58 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4233491205' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T07:30:58.633 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:58 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3887064292' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T07:30:58.633 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:58 smithi203 ceph-mon[32237]: from='client.24412 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T07:30:58.633 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:58 smithi203 ceph-mon[32237]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T07:30:58.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:58 smithi137 ceph-mon[29408]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:30:58.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:58 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:30:58.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:58 smithi137 ceph-mon[29408]: osdmap e62: 8 total, 8 up, 8 in 2024-09-18T07:30:58.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:58 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4233491205' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T07:30:58.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:58 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3887064292' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T07:30:58.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:58 smithi137 ceph-mon[29408]: from='client.24412 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T07:30:58.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:58 smithi137 ceph-mon[29408]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T07:30:58.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:58 smithi137 ceph-mon[39175]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:30:58.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:58 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:30:58.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:58 smithi137 ceph-mon[39175]: osdmap e62: 8 total, 8 up, 8 in 2024-09-18T07:30:58.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:58 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4233491205' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T07:30:58.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:58 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3887064292' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T07:30:58.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:58 smithi137 ceph-mon[39175]: from='client.24412 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T07:30:58.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:58 smithi137 ceph-mon[39175]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-18T07:30:59.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.203 --placement '1;smithi203=iscsi.a' 2024-09-18T07:30:59.385 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:30:59.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:59 smithi203 ceph-mon[32237]: from='client.24412 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-18T07:30:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:59 smithi203 ceph-mon[32237]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-18T07:30:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:59 smithi203 ceph-mon[32237]: osdmap e63: 8 total, 8 up, 8 in 2024-09-18T07:30:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:30:59 smithi203 ceph-mon[32237]: pgmap v199: 68 pgs: 30 creating+peering, 11 active+clean, 27 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-18T07:30:59.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:59 smithi137 ceph-mon[29408]: from='client.24412 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-18T07:30:59.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:59 smithi137 ceph-mon[29408]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-18T07:30:59.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:59 smithi137 ceph-mon[29408]: osdmap e63: 8 total, 8 up, 8 in 2024-09-18T07:30:59.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:30:59 smithi137 ceph-mon[29408]: pgmap v199: 68 pgs: 30 creating+peering, 11 active+clean, 27 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-18T07:30:59.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:59 smithi137 ceph-mon[39175]: from='client.24412 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-18T07:30:59.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:59 smithi137 ceph-mon[39175]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-18T07:30:59.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:59 smithi137 ceph-mon[39175]: osdmap e63: 8 total, 8 up, 8 in 2024-09-18T07:30:59.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:30:59 smithi137 ceph-mon[39175]: pgmap v199: 68 pgs: 30 creating+peering, 11 active+clean, 27 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-18T07:31:00.153 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled iscsi.datapool update... 2024-09-18T07:31:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:00 smithi203 ceph-mon[32237]: osdmap e64: 8 total, 8 up, 8 in 2024-09-18T07:31:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:00 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3887064292' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T07:31:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:00 smithi203 ceph-mon[32237]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T07:31:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:00 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4233491205' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T07:31:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:00 smithi203 ceph-mon[32237]: from='client.24412 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T07:31:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:00 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:00.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:00 smithi137 ceph-mon[29408]: osdmap e64: 8 total, 8 up, 8 in 2024-09-18T07:31:00.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:00 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3887064292' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T07:31:00.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:00 smithi137 ceph-mon[29408]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T07:31:00.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:00 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4233491205' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T07:31:00.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:00 smithi137 ceph-mon[29408]: from='client.24412 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T07:31:00.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:00 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:00.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:00 smithi137 ceph-mon[39175]: osdmap e64: 8 total, 8 up, 8 in 2024-09-18T07:31:00.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:00 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3887064292' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T07:31:00.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:00 smithi137 ceph-mon[39175]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T07:31:00.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:00 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4233491205' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T07:31:00.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:00 smithi137 ceph-mon[39175]: from='client.24412 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-18T07:31:00.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:00 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:00.776 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-18T07:31:00.776 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-18T07:31:00.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-18T07:31:00.809 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-18T07:31:00.809 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-18T07:31:00.835 DEBUG:teuthology.orchestra.run.smithi203:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@iscsi.iscsi.a.service 2024-09-18T07:31:00.878 INFO:tasks.cephadm:Adding prometheus.a on smithi203 2024-09-18T07:31:00.878 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch apply prometheus '1;smithi203=a' 2024-09-18T07:31:01.163 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:31:01.569 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:01 smithi203 ceph-mon[32237]: from='client.24433 v1:172.21.15.203:0/1252389393' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.203", "placement": "1;smithi203=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:31:01.569 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:01 smithi203 ceph-mon[32237]: Saving service iscsi.datapool spec with placement smithi203=iscsi.a;count:1 2024-09-18T07:31:01.569 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:01 smithi203 ceph-mon[32237]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-18T07:31:01.569 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:01 smithi203 ceph-mon[32237]: from='client.24412 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-18T07:31:01.569 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:01 smithi203 ceph-mon[32237]: osdmap e65: 8 total, 8 up, 8 in 2024-09-18T07:31:01.569 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:01 smithi203 ceph-mon[32237]: pgmap v202: 100 pgs: 27 creating+peering, 22 active+clean, 51 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-09-18T07:31:01.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:01 smithi137 ceph-mon[29408]: from='client.24433 v1:172.21.15.203:0/1252389393' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.203", "placement": "1;smithi203=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:31:01.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:01 smithi137 ceph-mon[29408]: Saving service iscsi.datapool spec with placement smithi203=iscsi.a;count:1 2024-09-18T07:31:01.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:01 smithi137 ceph-mon[29408]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-18T07:31:01.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:01 smithi137 ceph-mon[29408]: from='client.24412 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-18T07:31:01.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:01 smithi137 ceph-mon[29408]: osdmap e65: 8 total, 8 up, 8 in 2024-09-18T07:31:01.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:01 smithi137 ceph-mon[29408]: pgmap v202: 100 pgs: 27 creating+peering, 22 active+clean, 51 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-09-18T07:31:01.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:01 smithi137 ceph-mon[39175]: from='client.24433 v1:172.21.15.203:0/1252389393' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.203", "placement": "1;smithi203=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:31:01.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:01 smithi137 ceph-mon[39175]: Saving service iscsi.datapool spec with placement smithi203=iscsi.a;count:1 2024-09-18T07:31:01.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:01 smithi137 ceph-mon[39175]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-18T07:31:01.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:01 smithi137 ceph-mon[39175]: from='client.24412 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-18T07:31:01.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:01 smithi137 ceph-mon[39175]: osdmap e65: 8 total, 8 up, 8 in 2024-09-18T07:31:01.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:01 smithi137 ceph-mon[39175]: pgmap v202: 100 pgs: 27 creating+peering, 22 active+clean, 51 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-09-18T07:31:01.852 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled prometheus update... 2024-09-18T07:31:02.456 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:02 smithi203 ceph-mon[32237]: osdmap e66: 8 total, 8 up, 8 in 2024-09-18T07:31:02.456 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:02 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4233491205' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T07:31:02.456 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:02 smithi203 ceph-mon[32237]: from='client.24412 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T07:31:02.456 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:02 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3887064292' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T07:31:02.456 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:02 smithi203 ceph-mon[32237]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T07:31:02.456 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:02 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:02.482 DEBUG:teuthology.orchestra.run.smithi203:prometheus.a> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@prometheus.a.service 2024-09-18T07:31:02.485 INFO:tasks.cephadm:Adding node-exporter.a on smithi137 2024-09-18T07:31:02.485 INFO:tasks.cephadm:Adding node-exporter.b on smithi203 2024-09-18T07:31:02.485 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch apply node-exporter '2;smithi137=a;smithi203=b' 2024-09-18T07:31:02.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:02 smithi137 ceph-mon[29408]: osdmap e66: 8 total, 8 up, 8 in 2024-09-18T07:31:02.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:02 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4233491205' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T07:31:02.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:02 smithi137 ceph-mon[29408]: from='client.24412 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T07:31:02.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:02 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3887064292' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T07:31:02.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:02 smithi137 ceph-mon[29408]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T07:31:02.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:02 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:02.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:02 smithi137 ceph-mon[39175]: osdmap e66: 8 total, 8 up, 8 in 2024-09-18T07:31:02.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:02 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4233491205' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T07:31:02.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:02 smithi137 ceph-mon[39175]: from='client.24412 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T07:31:02.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:02 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3887064292' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T07:31:02.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:02 smithi137 ceph-mon[39175]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-18T07:31:02.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:02 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:02.768 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:31:03.443 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled node-exporter update... 2024-09-18T07:31:03.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:03 smithi203 ceph-mon[32237]: from='client.24439 v1:172.21.15.203:0/1174876116' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi203=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:31:03.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:03 smithi203 ceph-mon[32237]: Saving service prometheus spec with placement smithi203=a;count:1 2024-09-18T07:31:03.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:03 smithi203 ceph-mon[32237]: from='client.24412 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-18T07:31:03.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:03 smithi203 ceph-mon[32237]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-18T07:31:03.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:03 smithi203 ceph-mon[32237]: osdmap e67: 8 total, 8 up, 8 in 2024-09-18T07:31:03.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:03 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4233491205' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T07:31:03.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:03 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3887064292' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T07:31:03.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:03 smithi203 ceph-mon[32237]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T07:31:03.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:03 smithi203 ceph-mon[32237]: from='client.24412 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T07:31:03.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:03 smithi203 ceph-mon[32237]: pgmap v205: 132 pgs: 15 creating+peering, 88 active+clean, 29 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-09-18T07:31:03.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[29408]: from='client.24439 v1:172.21.15.203:0/1174876116' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi203=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:31:03.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[29408]: Saving service prometheus spec with placement smithi203=a;count:1 2024-09-18T07:31:03.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[29408]: from='client.24412 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-18T07:31:03.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[29408]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-18T07:31:03.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[29408]: osdmap e67: 8 total, 8 up, 8 in 2024-09-18T07:31:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4233491205' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T07:31:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3887064292' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T07:31:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[29408]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T07:31:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[29408]: from='client.24412 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T07:31:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[29408]: pgmap v205: 132 pgs: 15 creating+peering, 88 active+clean, 29 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-09-18T07:31:03.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[39175]: from='client.24439 v1:172.21.15.203:0/1174876116' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi203=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:31:03.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[39175]: Saving service prometheus spec with placement smithi203=a;count:1 2024-09-18T07:31:03.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[39175]: from='client.24412 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-18T07:31:03.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[39175]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-18T07:31:03.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[39175]: osdmap e67: 8 total, 8 up, 8 in 2024-09-18T07:31:03.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4233491205' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T07:31:03.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3887064292' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T07:31:03.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[39175]: from='client.34409 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T07:31:03.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[39175]: from='client.24412 ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-18T07:31:03.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:03 smithi137 ceph-mon[39175]: pgmap v205: 132 pgs: 15 creating+peering, 88 active+clean, 29 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-09-18T07:31:04.106 DEBUG:teuthology.orchestra.run.smithi137:node-exporter.a> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@node-exporter.a.service 2024-09-18T07:31:04.109 DEBUG:teuthology.orchestra.run.smithi203:node-exporter.b> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@node-exporter.b.service 2024-09-18T07:31:04.111 INFO:tasks.cephadm:Adding alertmanager.a on smithi137 2024-09-18T07:31:04.111 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch apply alertmanager '1;smithi137=a' 2024-09-18T07:31:04.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:31:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:31:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[29408]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-18T07:31:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[29408]: from='client.24412 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-18T07:31:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[29408]: osdmap e68: 8 total, 8 up, 8 in 2024-09-18T07:31:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[29408]: from='client.24445 v1:172.21.15.203:0/2517267185' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi137=a;smithi203=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:31:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[29408]: Saving service node-exporter spec with placement smithi137=a;smithi203=b;count:2 2024-09-18T07:31:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:04.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-18T07:31:04.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-18T07:31:04.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:31:04.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[29408]: Deploying daemon iscsi.iscsi.a on smithi203 2024-09-18T07:31:04.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:31:04.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:31:04.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[39175]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-18T07:31:04.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[39175]: from='client.24412 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-18T07:31:04.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[39175]: osdmap e68: 8 total, 8 up, 8 in 2024-09-18T07:31:04.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[39175]: from='client.24445 v1:172.21.15.203:0/2517267185' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi137=a;smithi203=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:31:04.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[39175]: Saving service node-exporter spec with placement smithi137=a;smithi203=b;count:2 2024-09-18T07:31:04.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:04.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:04.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:04.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:04.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-18T07:31:04.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-18T07:31:04.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:31:04.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:04 smithi137 ceph-mon[39175]: Deploying daemon iscsi.iscsi.a on smithi203 2024-09-18T07:31:04.426 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:04 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:31:04.426 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:04 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:31:04.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:04 smithi203 ceph-mon[32237]: from='client.34409 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-18T07:31:04.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:04 smithi203 ceph-mon[32237]: from='client.24412 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-18T07:31:04.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:04 smithi203 ceph-mon[32237]: osdmap e68: 8 total, 8 up, 8 in 2024-09-18T07:31:04.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:04 smithi203 ceph-mon[32237]: from='client.24445 v1:172.21.15.203:0/2517267185' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi137=a;smithi203=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:31:04.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:04 smithi203 ceph-mon[32237]: Saving service node-exporter spec with placement smithi137=a;smithi203=b;count:2 2024-09-18T07:31:04.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:04 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:04.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:04 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:04.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:04 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:04.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:04 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:04.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:04 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-18T07:31:04.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:04 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-18T07:31:04.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:04 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:31:04.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:04 smithi203 ceph-mon[32237]: Deploying daemon iscsi.iscsi.a on smithi203 2024-09-18T07:31:05.325 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:31:05.620 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:05 smithi203 ceph-mon[32237]: pgmap v207: 132 pgs: 10 creating+peering, 100 active+clean, 22 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-09-18T07:31:05.621 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:05 smithi203 ceph-mon[32237]: osdmap e69: 8 total, 8 up, 8 in 2024-09-18T07:31:05.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:05 smithi137 ceph-mon[29408]: pgmap v207: 132 pgs: 10 creating+peering, 100 active+clean, 22 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-09-18T07:31:05.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:05 smithi137 ceph-mon[29408]: osdmap e69: 8 total, 8 up, 8 in 2024-09-18T07:31:05.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:05 smithi137 ceph-mon[39175]: pgmap v207: 132 pgs: 10 creating+peering, 100 active+clean, 22 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-09-18T07:31:05.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:05 smithi137 ceph-mon[39175]: osdmap e69: 8 total, 8 up, 8 in 2024-09-18T07:31:05.961 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:05 smithi203 systemd[1]: Starting Ceph iscsi.iscsi.a for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:31:06.064 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled alertmanager update... 2024-09-18T07:31:06.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:06 smithi203 podman[57237]: 2024-09-18 07:31:06.057710829 +0000 UTC m=+0.106323180 container create 4924466ab0b425557b7062c993691dd30bc4bf2d2aae4ba8ea31f3b7f25b3fef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-18T07:31:06.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:06 smithi203 podman[57237]: 2024-09-18 07:31:05.967434071 +0000 UTC m=+0.016046423 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:31:06.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:06 smithi203 podman[57237]: 2024-09-18 07:31:06.494938155 +0000 UTC m=+0.543550506 container init 4924466ab0b425557b7062c993691dd30bc4bf2d2aae4ba8ea31f3b7f25b3fef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-18T07:31:06.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:06 smithi203 podman[57237]: 2024-09-18 07:31:06.49883477 +0000 UTC m=+0.547447123 container start 4924466ab0b425557b7062c993691dd30bc4bf2d2aae4ba8ea31f3b7f25b3fef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-18T07:31:06.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:06 smithi203 bash[57237]: 4924466ab0b425557b7062c993691dd30bc4bf2d2aae4ba8ea31f3b7f25b3fef 2024-09-18T07:31:06.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:06 smithi203 systemd[1]: Started Ceph iscsi.iscsi.a for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:31:07.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:06 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug Started the configuration object watcher 2024-09-18T07:31:07.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:06 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug Checking for config object changes every 1s 2024-09-18T07:31:07.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:06 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug Processing osd blocklist entries for this node 2024-09-18T07:31:07.055 DEBUG:teuthology.orchestra.run.smithi137:alertmanager.a> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@alertmanager.a.service 2024-09-18T07:31:07.057 INFO:tasks.cephadm:Adding grafana.a on smithi203 2024-09-18T07:31:07.058 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph orch apply grafana '1;smithi203=a' 2024-09-18T07:31:07.364 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-mon[32237]: from='client.24466 v1:172.21.15.203:0/918578641' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi137=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:31:07.364 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-mon[32237]: Saving service alertmanager spec with placement smithi137=a;count:1 2024-09-18T07:31:07.364 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.364 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-mon[32237]: pgmap v209: 132 pgs: 7 creating+peering, 125 active+clean; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.4 KiB/s wr, 110 op/s 2024-09-18T07:31:07.364 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.364 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.365 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.365 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-mon[32237]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-18T07:31:07.365 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.365 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.365 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug Reading the configuration object to update local LIO configuration 2024-09-18T07:31:07.365 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug Configuration does not have an entry for this host(smithi203.front.sepia.ceph.com) - nothing to define to LIO 2024-09-18T07:31:07.365 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-18T07:31:07.365 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: * Environment: production 2024-09-18T07:31:07.365 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-18T07:31:07.365 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: Use a production WSGI server instead. 2024-09-18T07:31:07.365 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: * Debug mode: off 2024-09-18T07:31:07.365 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug * Running on all addresses. 2024-09-18T07:31:07.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-18T07:31:07.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: * Running on all addresses. 2024-09-18T07:31:07.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-18T07:31:07.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug * Running on http://[fe80::ec4:7aff:fe8f:4a23]:5000/ (Press CTRL+C to quit) 2024-09-18T07:31:07.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: * Running on http://[fe80::ec4:7aff:fe8f:4a23]:5000/ (Press CTRL+C to quit) 2024-09-18T07:31:07.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[29408]: from='client.24466 v1:172.21.15.203:0/918578641' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi137=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:31:07.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[29408]: Saving service alertmanager spec with placement smithi137=a;count:1 2024-09-18T07:31:07.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[29408]: pgmap v209: 132 pgs: 7 creating+peering, 125 active+clean; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.4 KiB/s wr, 110 op/s 2024-09-18T07:31:07.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[29408]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-18T07:31:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[39175]: from='client.24466 v1:172.21.15.203:0/918578641' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi137=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:31:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[39175]: Saving service alertmanager spec with placement smithi137=a;count:1 2024-09-18T07:31:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[39175]: pgmap v209: 132 pgs: 7 creating+peering, 125 active+clean; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.4 KiB/s wr, 110 op/s 2024-09-18T07:31:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[39175]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-18T07:31:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:07 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:07.622 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:31:08.295 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled grafana update... 2024-09-18T07:31:08.295 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:08 smithi203 ceph-mon[32237]: Deploying daemon prometheus.a on smithi203 2024-09-18T07:31:08.295 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:08 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.203:0/3460519847' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-18T07:31:08.295 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:08 smithi203 ceph-mon[32237]: mgrmap e16: y(active, since 5m), standbys: x 2024-09-18T07:31:08.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:08 smithi137 ceph-mon[29408]: Deploying daemon prometheus.a on smithi203 2024-09-18T07:31:08.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:08 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.203:0/3460519847' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-18T07:31:08.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:08 smithi137 ceph-mon[29408]: mgrmap e16: y(active, since 5m), standbys: x 2024-09-18T07:31:08.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:08 smithi137 ceph-mon[39175]: Deploying daemon prometheus.a on smithi203 2024-09-18T07:31:08.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:08 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.203:0/3460519847' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-18T07:31:08.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:08 smithi137 ceph-mon[39175]: mgrmap e16: y(active, since 5m), standbys: x 2024-09-18T07:31:09.324 DEBUG:teuthology.orchestra.run.smithi203:grafana.a> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@grafana.a.service 2024-09-18T07:31:09.327 INFO:tasks.cephadm:Setting up client nodes... 2024-09-18T07:31:09.327 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-18T07:31:09.584 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:09.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:09 smithi137 ceph-mon[29408]: from='client.24502 v1:172.21.15.203:0/3350221839' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi203=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:31:09.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:09 smithi137 ceph-mon[29408]: Saving service grafana spec with placement smithi203=a;count:1 2024-09-18T07:31:09.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:09 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:09.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:09 smithi137 ceph-mon[29408]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.5 KiB/s wr, 177 op/s 2024-09-18T07:31:09.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:09 smithi137 ceph-mon[39175]: from='client.24502 v1:172.21.15.203:0/3350221839' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi203=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:31:09.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:09 smithi137 ceph-mon[39175]: Saving service grafana spec with placement smithi203=a;count:1 2024-09-18T07:31:09.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:09 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:09.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:09 smithi137 ceph-mon[39175]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.5 KiB/s wr, 177 op/s 2024-09-18T07:31:09.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:09 smithi203 ceph-mon[32237]: from='client.24502 v1:172.21.15.203:0/3350221839' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi203=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:31:09.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:09 smithi203 ceph-mon[32237]: Saving service grafana spec with placement smithi203=a;count:1 2024-09-18T07:31:09.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:09 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:09.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:09 smithi203 ceph-mon[32237]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.5 KiB/s wr, 177 op/s 2024-09-18T07:31:10.407 INFO:teuthology.orchestra.run.smithi137.stdout:[client.0] 2024-09-18T07:31:10.407 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQC+gepmRfMFGBAAGn+JriHWgxKxo84ZOJHOFQ== 2024-09-18T07:31:11.163 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:10 smithi137 ceph-mon[29408]: osdmap e70: 8 total, 8 up, 8 in 2024-09-18T07:31:11.163 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:10 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2455285574' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T07:31:11.163 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:10 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2455285574' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-18T07:31:11.163 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:10 smithi137 ceph-mon[29408]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 4.6 KiB/s wr, 152 op/s 2024-09-18T07:31:11.163 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:10 smithi137 ceph-mon[39175]: osdmap e70: 8 total, 8 up, 8 in 2024-09-18T07:31:11.163 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:10 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2455285574' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T07:31:11.163 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:10 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2455285574' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-18T07:31:11.164 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:10 smithi137 ceph-mon[39175]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 4.6 KiB/s wr, 152 op/s 2024-09-18T07:31:11.193 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-18T07:31:11.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-18T07:31:11.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-18T07:31:11.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:10 smithi203 ceph-mon[32237]: osdmap e70: 8 total, 8 up, 8 in 2024-09-18T07:31:11.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:10 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2455285574' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T07:31:11.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:10 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2455285574' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-18T07:31:11.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:10 smithi203 ceph-mon[32237]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 4.6 KiB/s wr, 152 op/s 2024-09-18T07:31:11.232 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-18T07:31:13.019 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:31:13.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:12 smithi137 ceph-mon[29408]: pgmap v213: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-09-18T07:31:13.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:12 smithi137 ceph-mon[39175]: pgmap v213: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-09-18T07:31:13.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:12 smithi203 ceph-mon[32237]: pgmap v213: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-09-18T07:31:14.647 INFO:teuthology.orchestra.run.smithi203.stdout:[client.1] 2024-09-18T07:31:14.647 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQDCgepmwNwxJhAAgs4UCm+RQukR3wklc+jJvA== 2024-09-18T07:31:14.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:14 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.203:0/1068867027' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T07:31:14.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:14 smithi203 ceph-mon[32237]: from='client.24514 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T07:31:14.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:14 smithi203 ceph-mon[32237]: from='client.24514 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-18T07:31:14.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:14 smithi203 ceph-mon[32237]: pgmap v214: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 113 op/s 2024-09-18T07:31:15.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:14 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.203:0/1068867027' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T07:31:15.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:14 smithi137 ceph-mon[29408]: from='client.24514 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T07:31:15.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:14 smithi137 ceph-mon[29408]: from='client.24514 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-18T07:31:15.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:14 smithi137 ceph-mon[29408]: pgmap v214: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 113 op/s 2024-09-18T07:31:15.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:14 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.203:0/1068867027' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T07:31:15.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:14 smithi137 ceph-mon[39175]: from='client.24514 ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-18T07:31:15.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:14 smithi137 ceph-mon[39175]: from='client.24514 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-18T07:31:15.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:14 smithi137 ceph-mon[39175]: pgmap v214: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 113 op/s 2024-09-18T07:31:16.083 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:15 smithi203 ceph-mon[32237]: osdmap e71: 8 total, 8 up, 8 in 2024-09-18T07:31:16.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:15 smithi137 ceph-mon[29408]: osdmap e71: 8 total, 8 up, 8 in 2024-09-18T07:31:16.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:15 smithi137 ceph-mon[39175]: osdmap e71: 8 total, 8 up, 8 in 2024-09-18T07:31:16.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:16 smithi203 ceph-mon[32237]: pgmap v216: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-09-18T07:31:16.961 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:16 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:31:17.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:16 smithi137 ceph-mon[29408]: pgmap v216: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-09-18T07:31:17.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:16 smithi137 ceph-mon[39175]: pgmap v216: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-09-18T07:31:17.332 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-18T07:31:17.332 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-18T07:31:17.333 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-18T07:31:17.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph config log 1 --format=json 2024-09-18T07:31:17.632 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:17.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:17 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:31:17.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:17 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:18.025 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:17 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:31:18.025 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:17 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:18.025 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:17 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:31:18.025 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:17 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:18.300 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:31:18.461 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 systemd[1]: Starting Ceph prometheus.a for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:31:18.765 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 podman[58162]: 2024-09-18 07:31:18.393791606 +0000 UTC m=+0.017079872 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-18T07:31:18.766 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 podman[58162]: 2024-09-18 07:31:18.511249312 +0000 UTC m=+0.134537572 container create ff9b936dfcead466a78ab3046b8e8dbe1612c03cfb5b38fc9a348ba8030d69fb (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T07:31:18.766 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 podman[58162]: 2024-09-18 07:31:18.76348763 +0000 UTC m=+0.386775895 container init ff9b936dfcead466a78ab3046b8e8dbe1612c03cfb5b38fc9a348ba8030d69fb (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T07:31:19.029 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:18 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4079951719' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-18T07:31:19.029 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:18 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:31:19.029 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:18 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:31:19.029 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:18 smithi137 ceph-mon[29408]: pgmap v217: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 114 B/s wr, 2 op/s 2024-09-18T07:31:19.029 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:18 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4079951719' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-18T07:31:19.029 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:18 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:31:19.030 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:18 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:31:19.030 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:18 smithi137 ceph-mon[39175]: pgmap v217: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 114 B/s wr, 2 op/s 2024-09-18T07:31:19.056 INFO:teuthology.orchestra.run.smithi137.stdout:[{"version":23,"timestamp":"2024-09-18T07:31:03.562498+0000","name":"","changes":[{"name":"mgr/mgr/dashboard/RGW_API_SECRET_KEY","new_value":"oUSq3rBjern81A3JV4EydKqqMJ2rXbqfBe59scBB"}]}] 2024-09-18T07:31:19.056 INFO:tasks.ceph_manager:config epoch is 23 2024-09-18T07:31:19.056 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-18T07:31:19.057 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-18T07:31:19.057 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-18T07:31:19.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4079951719' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-18T07:31:19.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:31:19.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:31:19.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-mon[32237]: pgmap v217: 132 pgs: 132 active+clean; 583 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 114 B/s wr, 2 op/s 2024-09-18T07:31:19.211 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 podman[58162]: 2024-09-18 07:31:18.766759832 +0000 UTC m=+0.390048101 container start ff9b936dfcead466a78ab3046b8e8dbe1612c03cfb5b38fc9a348ba8030d69fb (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T07:31:19.211 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.786Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-18T07:31:19.211 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.786Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-18T07:31:19.211 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.786Z caller=main.go:623 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi203 (none))" 2024-09-18T07:31:19.212 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.786Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-18T07:31:19.212 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.787Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-18T07:31:19.212 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.788Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-18T07:31:19.212 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.788Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-18T07:31:19.212 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.790Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-18T07:31:19.212 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.790Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-18T07:31:19.212 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.792Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-18T07:31:19.212 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.792Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.919µs 2024-09-18T07:31:19.212 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.792Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-18T07:31:19.212 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.792Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-18T07:31:19.212 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.792Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=22.414µs wal_replay_duration=175.637µs wbl_replay_duration=126ns total_replay_duration=216.703µs 2024-09-18T07:31:19.213 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.794Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-18T07:31:19.213 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.794Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-18T07:31:19.213 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.794Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-18T07:31:19.213 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.809Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.232526ms db_storage=727ns remote_storage=1.597µs web_handler=313ns query_engine=541ns scrape=446.374µs scrape_sd=100.648µs notify=977ns notify_sd=751ns rules=14.389792ms tracing=5.591µs 2024-09-18T07:31:19.213 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.809Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-18T07:31:19.213 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:31:18.809Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-18T07:31:19.213 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 bash[58162]: ff9b936dfcead466a78ab3046b8e8dbe1612c03cfb5b38fc9a348ba8030d69fb 2024-09-18T07:31:19.213 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:31:18 smithi203 systemd[1]: Started Ceph prometheus.a for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:31:19.308 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:20.011 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:31:20.162 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:19 smithi203 ceph-mon[32237]: osdmap e72: 8 total, 8 up, 8 in 2024-09-18T07:31:20.162 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:19 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:20.162 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:19 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:20.163 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:19 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:20.163 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:19 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-18T07:31:20.163 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:31:19 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-x[33743]: ignoring --setuser ceph since I am not root 2024-09-18T07:31:20.163 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:31:19 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-x[33743]: ignoring --setgroup ceph since I am not root 2024-09-18T07:31:20.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:19 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ignoring --setuser ceph since I am not root 2024-09-18T07:31:20.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:19 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ignoring --setgroup ceph since I am not root 2024-09-18T07:31:20.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:19 smithi137 ceph-mon[39175]: osdmap e72: 8 total, 8 up, 8 in 2024-09-18T07:31:20.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:19 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:20.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:19 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:20.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:19 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:20.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:19 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-18T07:31:20.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:19 smithi137 ceph-mon[29408]: osdmap e72: 8 total, 8 up, 8 in 2024-09-18T07:31:20.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:19 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:20.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:19 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:20.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:19 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' 2024-09-18T07:31:20.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:19 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-18T07:31:21.033 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6800","nonce":2096039404}]},"active_addr":"172.21.15.137:6800/2096039404","active_change":"2024-09-18T07:26:06.595547+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24121,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.137:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":1762989503}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":3160704935}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":2357293499}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":1762352860}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":621434521}]}]} 2024-09-18T07:31:21.040 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-18T07:31:21.040 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-18T07:31:21.040 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-18T07:31:21.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:20 smithi203 ceph-mon[32237]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-18T07:31:21.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:20 smithi203 ceph-mon[32237]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-18T07:31:21.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:20 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1974005170' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-18T07:31:21.299 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:21.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:20 smithi137 ceph-mon[39175]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-18T07:31:21.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:20 smithi137 ceph-mon[39175]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-18T07:31:21.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:20 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1974005170' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-18T07:31:21.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:20 smithi137 ceph-mon[29408]: from='mgr.14152 v1:172.21.15.137:0/87207476' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-18T07:31:21.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:20 smithi137 ceph-mon[29408]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-18T07:31:21.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:20 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1974005170' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-18T07:31:21.958 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:31:21.959 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":72,"fsid":"f7ef099c-758e-11ef-bceb-c7b262605968","created":"2024-09-18T07:25:19.662367+0000","modified":"2024-09-18T07:31:18.900866+0000","last_up_change":"2024-09-18T07:30:44.708496+0000","last_in_change":"2024-09-18T07:30:23.052551+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T07:28:20.899655+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-18T07:30:53.715373+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-18T07:30:54.672127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-18T07:30:56.351343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-18T07:30:58.322401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-18T07:31:00.390499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268cd580-c931-4992-bb93-d6f63c2bfe88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6801","nonce":2811444477}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6802","nonce":2811444477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6804","nonce":2811444477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6803","nonce":2811444477}]},"public_addr":"172.21.15.137:6801/2811444477","cluster_addr":"172.21.15.137:6802/2811444477","heartbeat_back_addr":"172.21.15.137:6804/2811444477","heartbeat_front_addr":"172.21.15.137:6803/2811444477","state":["exists","up"]},{"osd":1,"uuid":"b06907bc-b499-46c0-8667-c5bdc0d85575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6805","nonce":874102487}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6806","nonce":874102487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6808","nonce":874102487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6807","nonce":874102487}]},"public_addr":"172.21.15.137:6805/874102487","cluster_addr":"172.21.15.137:6806/874102487","heartbeat_back_addr":"172.21.15.137:6808/874102487","heartbeat_front_addr":"172.21.15.137:6807/874102487","state":["exists","up"]},{"osd":2,"uuid":"480ad6c5-0e0b-4405-bbe2-381682d00bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6809","nonce":1754684686}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6810","nonce":1754684686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6812","nonce":1754684686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6811","nonce":1754684686}]},"public_addr":"172.21.15.137:6809/1754684686","cluster_addr":"172.21.15.137:6810/1754684686","heartbeat_back_addr":"172.21.15.137:6812/1754684686","heartbeat_front_addr":"172.21.15.137:6811/1754684686","state":["exists","up"]},{"osd":3,"uuid":"178be3e0-f07b-4856-807b-2a1181e48a14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6813","nonce":116058392}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6814","nonce":116058392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6816","nonce":116058392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6815","nonce":116058392}]},"public_addr":"172.21.15.137:6813/116058392","cluster_addr":"172.21.15.137:6814/116058392","heartbeat_back_addr":"172.21.15.137:6816/116058392","heartbeat_front_addr":"172.21.15.137:6815/116058392","state":["exists","up"]},{"osd":4,"uuid":"7547102c-951a-4077-a094-fb8a65bfe6de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":3654297954}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":3654297954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":3654297954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":3654297954}]},"public_addr":"172.21.15.203:6800/3654297954","cluster_addr":"172.21.15.203:6801/3654297954","heartbeat_back_addr":"172.21.15.203:6803/3654297954","heartbeat_front_addr":"172.21.15.203:6802/3654297954","state":["exists","up"]},{"osd":5,"uuid":"49c17901-8102-48a4-9348-fa1bdb63032c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":837994222}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":837994222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":837994222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":837994222}]},"public_addr":"172.21.15.203:6804/837994222","cluster_addr":"172.21.15.203:6805/837994222","heartbeat_back_addr":"172.21.15.203:6807/837994222","heartbeat_front_addr":"172.21.15.203:6806/837994222","state":["exists","up"]},{"osd":6,"uuid":"75f0ead4-e981-4a15-b1c1-d83dce871ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":1494175466}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":1494175466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":1494175466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":1494175466}]},"public_addr":"172.21.15.203:6808/1494175466","cluster_addr":"172.21.15.203:6809/1494175466","heartbeat_back_addr":"172.21.15.203:6811/1494175466","heartbeat_front_addr":"172.21.15.203:6810/1494175466","state":["exists","up"]},{"osd":7,"uuid":"c0dbf299-6798-4b99-9a85-62b123971f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6812","nonce":2670809315}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6813","nonce":2670809315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6815","nonce":2670809315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6814","nonce":2670809315}]},"public_addr":"172.21.15.203:6812/2670809315","cluster_addr":"172.21.15.203:6813/2670809315","heartbeat_back_addr":"172.21.15.203:6815/2670809315","heartbeat_front_addr":"172.21.15.203:6814/2670809315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:31:18.629314+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:30:21.954128+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:30:49.591564+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:31:02.992252+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:30:23.581624+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:30:53.837709+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:30:14.117573+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:30:43.753862+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:0/4028304023":"2024-09-19T07:26:06.595443+0000","172.21.15.137:6800/1482147392":"2024-09-19T07:26:06.595443+0000","172.21.15.137:0/3387637412":"2024-09-19T07:26:06.595443+0000","172.21.15.137:0/2101286111":"2024-09-19T07:25:41.661328+0000","172.21.15.137:0/1821482181":"2024-09-19T07:25:41.661328+0000","172.21.15.137:0/3490817671":"2024-09-19T07:26:06.595443+0000","172.21.15.137:0/334967443":"2024-09-19T07:25:41.661328+0000","172.21.15.137:6800/3636177939":"2024-09-19T07:25:41.661328+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T07:31:22.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:22 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4105303504' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T07:31:22.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:22 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4105303504' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T07:31:22.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:22 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4105303504' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T07:31:22.591 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-18T07:31:22.591 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-18T07:31:22.845 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:23.510 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:31:23.510 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":72,"fsid":"f7ef099c-758e-11ef-bceb-c7b262605968","created":"2024-09-18T07:25:19.662367+0000","modified":"2024-09-18T07:31:18.900866+0000","last_up_change":"2024-09-18T07:30:44.708496+0000","last_in_change":"2024-09-18T07:30:23.052551+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T07:28:20.899655+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-18T07:30:53.715373+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-18T07:30:54.672127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-18T07:30:56.351343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-18T07:30:58.322401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-18T07:31:00.390499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268cd580-c931-4992-bb93-d6f63c2bfe88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6801","nonce":2811444477}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6802","nonce":2811444477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6804","nonce":2811444477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6803","nonce":2811444477}]},"public_addr":"172.21.15.137:6801/2811444477","cluster_addr":"172.21.15.137:6802/2811444477","heartbeat_back_addr":"172.21.15.137:6804/2811444477","heartbeat_front_addr":"172.21.15.137:6803/2811444477","state":["exists","up"]},{"osd":1,"uuid":"b06907bc-b499-46c0-8667-c5bdc0d85575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6805","nonce":874102487}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6806","nonce":874102487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6808","nonce":874102487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6807","nonce":874102487}]},"public_addr":"172.21.15.137:6805/874102487","cluster_addr":"172.21.15.137:6806/874102487","heartbeat_back_addr":"172.21.15.137:6808/874102487","heartbeat_front_addr":"172.21.15.137:6807/874102487","state":["exists","up"]},{"osd":2,"uuid":"480ad6c5-0e0b-4405-bbe2-381682d00bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6809","nonce":1754684686}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6810","nonce":1754684686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6812","nonce":1754684686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6811","nonce":1754684686}]},"public_addr":"172.21.15.137:6809/1754684686","cluster_addr":"172.21.15.137:6810/1754684686","heartbeat_back_addr":"172.21.15.137:6812/1754684686","heartbeat_front_addr":"172.21.15.137:6811/1754684686","state":["exists","up"]},{"osd":3,"uuid":"178be3e0-f07b-4856-807b-2a1181e48a14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6813","nonce":116058392}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6814","nonce":116058392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6816","nonce":116058392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6815","nonce":116058392}]},"public_addr":"172.21.15.137:6813/116058392","cluster_addr":"172.21.15.137:6814/116058392","heartbeat_back_addr":"172.21.15.137:6816/116058392","heartbeat_front_addr":"172.21.15.137:6815/116058392","state":["exists","up"]},{"osd":4,"uuid":"7547102c-951a-4077-a094-fb8a65bfe6de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":3654297954}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":3654297954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":3654297954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":3654297954}]},"public_addr":"172.21.15.203:6800/3654297954","cluster_addr":"172.21.15.203:6801/3654297954","heartbeat_back_addr":"172.21.15.203:6803/3654297954","heartbeat_front_addr":"172.21.15.203:6802/3654297954","state":["exists","up"]},{"osd":5,"uuid":"49c17901-8102-48a4-9348-fa1bdb63032c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":837994222}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":837994222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":837994222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":837994222}]},"public_addr":"172.21.15.203:6804/837994222","cluster_addr":"172.21.15.203:6805/837994222","heartbeat_back_addr":"172.21.15.203:6807/837994222","heartbeat_front_addr":"172.21.15.203:6806/837994222","state":["exists","up"]},{"osd":6,"uuid":"75f0ead4-e981-4a15-b1c1-d83dce871ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":1494175466}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":1494175466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":1494175466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":1494175466}]},"public_addr":"172.21.15.203:6808/1494175466","cluster_addr":"172.21.15.203:6809/1494175466","heartbeat_back_addr":"172.21.15.203:6811/1494175466","heartbeat_front_addr":"172.21.15.203:6810/1494175466","state":["exists","up"]},{"osd":7,"uuid":"c0dbf299-6798-4b99-9a85-62b123971f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6812","nonce":2670809315}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6813","nonce":2670809315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6815","nonce":2670809315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6814","nonce":2670809315}]},"public_addr":"172.21.15.203:6812/2670809315","cluster_addr":"172.21.15.203:6813/2670809315","heartbeat_back_addr":"172.21.15.203:6815/2670809315","heartbeat_front_addr":"172.21.15.203:6814/2670809315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:31:18.629314+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:30:21.954128+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:30:49.591564+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:31:02.992252+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:30:23.581624+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:30:53.837709+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:30:14.117573+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:30:43.753862+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:0/4028304023":"2024-09-19T07:26:06.595443+0000","172.21.15.137:6800/1482147392":"2024-09-19T07:26:06.595443+0000","172.21.15.137:0/3387637412":"2024-09-19T07:26:06.595443+0000","172.21.15.137:0/2101286111":"2024-09-19T07:25:41.661328+0000","172.21.15.137:0/1821482181":"2024-09-19T07:25:41.661328+0000","172.21.15.137:0/3490817671":"2024-09-19T07:26:06.595443+0000","172.21.15.137:0/334967443":"2024-09-19T07:25:41.661328+0000","172.21.15.137:6800/3636177939":"2024-09-19T07:25:41.661328+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T07:31:23.826 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:23 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3654072453' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T07:31:23.826 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:23 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3654072453' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T07:31:23.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:23 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3654072453' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T07:31:24.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-18T07:31:24.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-18T07:31:24.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-18T07:31:24.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-18T07:31:24.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-18T07:31:24.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-18T07:31:24.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-18T07:31:24.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-18T07:31:24.690 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:24.700 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:24.701 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:24.705 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:24.706 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:24.708 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:24.710 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:24.718 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:25.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:25 smithi137 ceph-mon[29408]: Standby manager daemon x restarted 2024-09-18T07:31:25.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:25 smithi137 ceph-mon[29408]: Standby manager daemon x started 2024-09-18T07:31:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:25 smithi137 ceph-mon[29408]: from='mgr.? v1:172.21.15.203:0/1597739793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-18T07:31:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:25 smithi137 ceph-mon[29408]: from='mgr.? v1:172.21.15.203:0/1597739793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-18T07:31:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:25 smithi137 ceph-mon[29408]: from='mgr.? v1:172.21.15.203:0/1597739793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-18T07:31:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:25 smithi137 ceph-mon[29408]: from='mgr.? v1:172.21.15.203:0/1597739793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-18T07:31:25.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:25 smithi137 ceph-mon[39175]: Standby manager daemon x restarted 2024-09-18T07:31:25.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:25 smithi137 ceph-mon[39175]: Standby manager daemon x started 2024-09-18T07:31:25.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:25 smithi137 ceph-mon[39175]: from='mgr.? v1:172.21.15.203:0/1597739793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-18T07:31:25.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:25 smithi137 ceph-mon[39175]: from='mgr.? v1:172.21.15.203:0/1597739793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-18T07:31:25.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:25 smithi137 ceph-mon[39175]: from='mgr.? v1:172.21.15.203:0/1597739793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-18T07:31:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:25 smithi137 ceph-mon[39175]: from='mgr.? v1:172.21.15.203:0/1597739793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-18T07:31:25.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:25 smithi203 ceph-mon[32237]: Standby manager daemon x restarted 2024-09-18T07:31:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:25 smithi203 ceph-mon[32237]: Standby manager daemon x started 2024-09-18T07:31:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:25 smithi203 ceph-mon[32237]: from='mgr.? v1:172.21.15.203:0/1597739793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-18T07:31:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:25 smithi203 ceph-mon[32237]: from='mgr.? v1:172.21.15.203:0/1597739793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-18T07:31:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:25 smithi203 ceph-mon[32237]: from='mgr.? v1:172.21.15.203:0/1597739793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-18T07:31:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:25 smithi203 ceph-mon[32237]: from='mgr.? v1:172.21.15.203:0/1597739793' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-18T07:31:25.461 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:31:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-x[33743]: [18/Sep/2024:07:31:25] ENGINE Bus STARTING 2024-09-18T07:31:25.461 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:31:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-x[33743]: CherryPy Checker: 2024-09-18T07:31:25.461 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:31:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-x[33743]: The Application mounted at '' has an empty config. 2024-09-18T07:31:25.461 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:31:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-x[33743]: 2024-09-18T07:31:25.461 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:31:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-x[33743]: [18/Sep/2024:07:31:25] ENGINE Serving on http://:::9283 2024-09-18T07:31:25.461 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:31:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-x[33743]: [18/Sep/2024:07:31:25] ENGINE Bus STARTED 2024-09-18T07:31:26.412 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-18T07:31:26.413 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: Active manager daemon y restarted 2024-09-18T07:31:26.413 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: Activating manager daemon y 2024-09-18T07:31:26.413 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: osdmap e73: 8 total, 8 up, 8 in 2024-09-18T07:31:26.413 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: mgrmap e19: y(active, starting, since 0.0103769s), standbys: x 2024-09-18T07:31:26.413 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:31:26.413 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:31:26.413 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:31:26.413 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-18T07:31:26.413 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-18T07:31:26.413 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:31:26.413 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:31:26.414 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:31:26.414 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:31:26.414 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:31:26.414 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:31:26.414 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:31:26.414 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:31:26.414 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-18T07:31:26.414 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-18T07:31:26.414 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-18T07:31:26.414 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: Manager daemon y is now available 2024-09-18T07:31:26.414 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:31:26.414 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:26.414 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:31:26.414 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:31:26.414 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-18T07:31:26.415 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-18T07:31:26.415 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-18T07:31:26.415 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-18T07:31:26.415 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-18T07:31:26.415 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: Active manager daemon y restarted 2024-09-18T07:31:26.415 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: Activating manager daemon y 2024-09-18T07:31:26.415 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: osdmap e73: 8 total, 8 up, 8 in 2024-09-18T07:31:26.415 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: mgrmap e19: y(active, starting, since 0.0103769s), standbys: x 2024-09-18T07:31:26.415 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:31:26.415 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:31:26.416 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:31:26.416 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-18T07:31:26.416 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-18T07:31:26.416 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:31:26.416 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:31:26.416 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:31:26.416 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:31:26.416 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:31:26.416 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:31:26.416 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:31:26.416 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:31:26.416 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-18T07:31:26.416 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-18T07:31:26.416 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-18T07:31:26.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: Manager daemon y is now available 2024-09-18T07:31:26.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:31:26.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:26.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:31:26.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:31:26.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-18T07:31:26.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-18T07:31:26.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-18T07:31:26.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-18T07:31:26.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-18T07:31:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: Active manager daemon y restarted 2024-09-18T07:31:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: Activating manager daemon y 2024-09-18T07:31:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: osdmap e73: 8 total, 8 up, 8 in 2024-09-18T07:31:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: mgrmap e19: y(active, starting, since 0.0103769s), standbys: x 2024-09-18T07:31:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-18T07:31:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-18T07:31:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-18T07:31:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-18T07:31:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-18T07:31:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-18T07:31:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-18T07:31:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-18T07:31:26.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-18T07:31:26.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:31:26.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:31:26.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-18T07:31:26.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-18T07:31:26.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-18T07:31:26.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-18T07:31:26.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-18T07:31:26.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: Manager daemon y is now available 2024-09-18T07:31:26.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:31:26.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:26.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:31:26.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:31:26.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-18T07:31:26.463 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-18T07:31:26.463 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-18T07:31:26.463 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-18T07:31:26.669 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:31:26] ENGINE Bus STARTING 2024-09-18T07:31:26.669 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: CherryPy Checker: 2024-09-18T07:31:26.669 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: The Application mounted at '' has an empty config. 2024-09-18T07:31:26.669 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: 2024-09-18T07:31:26.669 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:31:26] ENGINE Serving on http://:::9283 2024-09-18T07:31:26.669 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:26 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:31:26] ENGINE Bus STARTED 2024-09-18T07:31:27.100 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: 2024-09-18T07:31:27.016+0000 7f45c2aaf640 -1 log_channel(cephadm) log [ERR] : [18/Sep/2024:07:31:27] ENGINE Error in HTTPServer.serve 2024-09-18T07:31:27.101 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: Traceback (most recent call last): 2024-09-18T07:31:27.101 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-18T07:31:27.101 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: self._connections.run(self.expiration_interval) 2024-09-18T07:31:27.101 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-18T07:31:27.101 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: self._run(expiration_interval) 2024-09-18T07:31:27.101 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-18T07:31:27.101 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: new_conn = self._from_server_socket(self.server.socket) 2024-09-18T07:31:27.101 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-18T07:31:27.101 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-18T07:31:27.101 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-18T07:31:27.101 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: s = self.context.wrap_socket( 2024-09-18T07:31:27.101 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-18T07:31:27.101 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: return self.sslsocket_class._create( 2024-09-18T07:31:27.102 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-18T07:31:27.102 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: self.do_handshake() 2024-09-18T07:31:27.102 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-18T07:31:27.102 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: self._sslobj.do_handshake() 2024-09-18T07:31:27.102 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-18T07:31:27.102 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: 2024-09-18T07:31:27.102 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:31:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:31:27.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:26 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:31:27.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-mon[39175]: mgrmap e20: y(active, since 1.01076s), standbys: x 2024-09-18T07:31:27.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-mon[39175]: [18/Sep/2024:07:31:26] ENGINE Bus STARTING 2024-09-18T07:31:27.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:31:27.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-mon[39175]: [18/Sep/2024:07:31:26] ENGINE Serving on http://172.21.15.137:8765 2024-09-18T07:31:27.660 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-mon[29408]: mgrmap e20: y(active, since 1.01076s), standbys: x 2024-09-18T07:31:27.660 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-mon[29408]: [18/Sep/2024:07:31:26] ENGINE Bus STARTING 2024-09-18T07:31:27.660 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:31:27.660 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:27 smithi137 ceph-mon[29408]: [18/Sep/2024:07:31:26] ENGINE Serving on http://172.21.15.137:8765 2024-09-18T07:31:27.661 INFO:teuthology.orchestra.run.smithi137.stdout:77309411365 2024-09-18T07:31:27.661 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-18T07:31:27.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:27 smithi203 ceph-mon[32237]: mgrmap e20: y(active, since 1.01076s), standbys: x 2024-09-18T07:31:27.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:27 smithi203 ceph-mon[32237]: [18/Sep/2024:07:31:26] ENGINE Bus STARTING 2024-09-18T07:31:27.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:27 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:31:27.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:27 smithi203 ceph-mon[32237]: [18/Sep/2024:07:31:26] ENGINE Serving on http://172.21.15.137:8765 2024-09-18T07:31:28.607 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:28 smithi137 ceph-mon[29408]: [18/Sep/2024:07:31:27] ENGINE Serving on https://172.21.15.137:7150 2024-09-18T07:31:28.607 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:28 smithi137 ceph-mon[29408]: [18/Sep/2024:07:31:27] ENGINE Bus STARTED 2024-09-18T07:31:28.607 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:28 smithi137 ceph-mon[29408]: [18/Sep/2024:07:31:27] ENGINE Error in HTTPServer.serve 2024-09-18T07:31:28.607 INFO:journalctl@ceph.mon.a.smithi137.stdout: Traceback (most recent call last): 2024-09-18T07:31:28.607 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-18T07:31:28.607 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._connections.run(self.expiration_interval) 2024-09-18T07:31:28.608 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-18T07:31:28.608 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._run(expiration_interval) 2024-09-18T07:31:28.608 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-18T07:31:28.608 INFO:journalctl@ceph.mon.a.smithi137.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-18T07:31:28.608 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-18T07:31:28.608 INFO:journalctl@ceph.mon.a.smithi137.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-18T07:31:28.608 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-18T07:31:28.608 INFO:journalctl@ceph.mon.a.smithi137.stdout: s = self.context.wrap_socket( 2024-09-18T07:31:28.608 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-18T07:31:28.608 INFO:journalctl@ceph.mon.a.smithi137.stdout: return self.sslsocket_class._create( 2024-09-18T07:31:28.608 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-18T07:31:28.608 INFO:journalctl@ceph.mon.a.smithi137.stdout: self.do_handshake() 2024-09-18T07:31:28.608 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-18T07:31:28.608 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._sslobj.do_handshake() 2024-09-18T07:31:28.609 INFO:journalctl@ceph.mon.a.smithi137.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-18T07:31:28.609 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:28 smithi137 ceph-mon[29408]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-18T07:31:28.609 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:28 smithi137 ceph-mon[29408]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-18T07:31:28.609 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:28 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:31:28.609 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:28 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:31:28.609 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:28 smithi137 ceph-mon[39175]: [18/Sep/2024:07:31:27] ENGINE Serving on https://172.21.15.137:7150 2024-09-18T07:31:28.609 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:28 smithi137 ceph-mon[39175]: [18/Sep/2024:07:31:27] ENGINE Bus STARTED 2024-09-18T07:31:28.609 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:28 smithi137 ceph-mon[39175]: [18/Sep/2024:07:31:27] ENGINE Error in HTTPServer.serve 2024-09-18T07:31:28.609 INFO:journalctl@ceph.mon.c.smithi137.stdout: Traceback (most recent call last): 2024-09-18T07:31:28.609 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-18T07:31:28.609 INFO:journalctl@ceph.mon.c.smithi137.stdout: self._connections.run(self.expiration_interval) 2024-09-18T07:31:28.610 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-18T07:31:28.610 INFO:journalctl@ceph.mon.c.smithi137.stdout: self._run(expiration_interval) 2024-09-18T07:31:28.610 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-18T07:31:28.610 INFO:journalctl@ceph.mon.c.smithi137.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-18T07:31:28.610 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-18T07:31:28.610 INFO:journalctl@ceph.mon.c.smithi137.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-18T07:31:28.610 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-18T07:31:28.610 INFO:journalctl@ceph.mon.c.smithi137.stdout: s = self.context.wrap_socket( 2024-09-18T07:31:28.610 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-18T07:31:28.610 INFO:journalctl@ceph.mon.c.smithi137.stdout: return self.sslsocket_class._create( 2024-09-18T07:31:28.610 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-18T07:31:28.610 INFO:journalctl@ceph.mon.c.smithi137.stdout: self.do_handshake() 2024-09-18T07:31:28.610 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-18T07:31:28.611 INFO:journalctl@ceph.mon.c.smithi137.stdout: self._sslobj.do_handshake() 2024-09-18T07:31:28.611 INFO:journalctl@ceph.mon.c.smithi137.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-18T07:31:28.611 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:28 smithi137 ceph-mon[39175]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-18T07:31:28.611 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:28 smithi137 ceph-mon[39175]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-18T07:31:28.611 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:28 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:31:28.611 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:28 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:31:28.637 INFO:teuthology.orchestra.run.smithi137.stdout:236223201290 2024-09-18T07:31:28.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-18T07:31:28.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:28 smithi203 ceph-mon[32237]: [18/Sep/2024:07:31:27] ENGINE Serving on https://172.21.15.137:7150 2024-09-18T07:31:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:28 smithi203 ceph-mon[32237]: [18/Sep/2024:07:31:27] ENGINE Bus STARTED 2024-09-18T07:31:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:28 smithi203 ceph-mon[32237]: [18/Sep/2024:07:31:27] ENGINE Error in HTTPServer.serve 2024-09-18T07:31:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout: Traceback (most recent call last): 2024-09-18T07:31:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-18T07:31:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout: self._connections.run(self.expiration_interval) 2024-09-18T07:31:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-18T07:31:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout: self._run(expiration_interval) 2024-09-18T07:31:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-18T07:31:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-18T07:31:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-18T07:31:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-18T07:31:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-18T07:31:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout: s = self.context.wrap_socket( 2024-09-18T07:31:28.712 INFO:journalctl@ceph.mon.b.smithi203.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-18T07:31:28.712 INFO:journalctl@ceph.mon.b.smithi203.stdout: return self.sslsocket_class._create( 2024-09-18T07:31:28.712 INFO:journalctl@ceph.mon.b.smithi203.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-18T07:31:28.712 INFO:journalctl@ceph.mon.b.smithi203.stdout: self.do_handshake() 2024-09-18T07:31:28.712 INFO:journalctl@ceph.mon.b.smithi203.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-18T07:31:28.712 INFO:journalctl@ceph.mon.b.smithi203.stdout: self._sslobj.do_handshake() 2024-09-18T07:31:28.712 INFO:journalctl@ceph.mon.b.smithi203.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-18T07:31:28.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:28 smithi203 ceph-mon[32237]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-18T07:31:28.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:28 smithi203 ceph-mon[32237]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-18T07:31:28.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:28 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:31:28.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:28 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:31:29.547 INFO:teuthology.orchestra.run.smithi137.stdout:111669149727 2024-09-18T07:31:29.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-18T07:31:29.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[29408]: osdmap e74: 8 total, 8 up, 8 in 2024-09-18T07:31:29.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:29.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:29.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:29.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[39175]: osdmap e74: 8 total, 8 up, 8 in 2024-09-18T07:31:29.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:29 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:29.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:29 smithi203 ceph-mon[32237]: osdmap e74: 8 total, 8 up, 8 in 2024-09-18T07:31:29.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:29 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:29 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:29 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:29 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:29 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:29 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:29 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:29 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:29 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:29 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:29 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:29 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:29 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:30.647 INFO:teuthology.orchestra.run.smithi137.stdout:167503724565 2024-09-18T07:31:30.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-18T07:31:30.886 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:30 smithi137 ceph-mon[29408]: Adjusting osd_memory_target on smithi203 to 3760M 2024-09-18T07:31:30.886 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:30 smithi137 ceph-mon[29408]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-18T07:31:30.886 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:30 smithi137 ceph-mon[29408]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-18T07:31:30.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:30 smithi137 ceph-mon[39175]: Adjusting osd_memory_target on smithi203 to 3760M 2024-09-18T07:31:30.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:30 smithi137 ceph-mon[39175]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-18T07:31:30.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:30 smithi137 ceph-mon[39175]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-18T07:31:31.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:30 smithi203 ceph-mon[32237]: Adjusting osd_memory_target on smithi203 to 3760M 2024-09-18T07:31:31.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:30 smithi203 ceph-mon[32237]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-18T07:31:31.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:30 smithi203 ceph-mon[32237]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-18T07:31:31.643 INFO:teuthology.orchestra.run.smithi137.stdout:34359738418 2024-09-18T07:31:31.644 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-18T07:31:32.476 INFO:teuthology.orchestra.run.smithi137.stdout:201863462929 2024-09-18T07:31:32.476 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-18T07:31:32.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:32 smithi203 ceph-mon[32237]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-18T07:31:33.052 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:32 smithi137 ceph-mon[29408]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-18T07:31:33.053 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:32 smithi137 ceph-mon[39175]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-18T07:31:33.265 INFO:teuthology.orchestra.run.smithi137.stdout:55834574893 2024-09-18T07:31:33.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-18T07:31:33.882 INFO:teuthology.orchestra.run.smithi137.stdout:137438953501 2024-09-18T07:31:33.883 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-18T07:31:33.960 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:33.961 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:33.963 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:34.008 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:34.814 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:34 smithi137 ceph-mon[29408]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-18T07:31:34.814 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:34 smithi137 ceph-mon[39175]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-18T07:31:35.024 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:35.204 INFO:teuthology.orchestra.run.smithi137.stdout:111669149729 2024-09-18T07:31:35.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:34 smithi203 ceph-mon[32237]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-18T07:31:35.468 INFO:teuthology.orchestra.run.smithi137.stdout:55834574894 2024-09-18T07:31:36.135 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:35 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3969115465' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-18T07:31:36.135 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:35 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2214402218' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-18T07:31:36.135 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:35 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3969115465' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-18T07:31:36.135 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:35 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2214402218' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-18T07:31:36.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:35 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3969115465' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-18T07:31:36.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:35 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2214402218' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-18T07:31:36.372 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149727 got 111669149729 for osd.3 2024-09-18T07:31:36.372 DEBUG:teuthology.parallel:result is None 2024-09-18T07:31:36.725 INFO:teuthology.orchestra.run.smithi137.stdout:167503724566 2024-09-18T07:31:37.064 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:36 smithi137 ceph-mon[39175]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-18T07:31:37.064 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:36 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:31:37.064 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:36 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:31:37.064 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:36 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2757556804' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-18T07:31:37.064 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:36 smithi137 ceph-mon[29408]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-18T07:31:37.064 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:36 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:31:37.065 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:36 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:31:37.065 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:36 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2757556804' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-18T07:31:37.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:36 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:31:37.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:36 smithi203 ceph-mon[32237]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-18T07:31:37.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:36 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:31:37.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:36 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:31:37.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:36 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2757556804' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-18T07:31:37.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:31:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:31:37.448 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574894 for osd.1 2024-09-18T07:31:37.448 DEBUG:teuthology.parallel:result is None 2024-09-18T07:31:37.846 INFO:teuthology.orchestra.run.smithi137.stdout:201863462930 2024-09-18T07:31:38.109 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:37 smithi137 ceph-mon[29408]: osdmap e75: 8 total, 8 up, 8 in 2024-09-18T07:31:38.109 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:37 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:31:38.109 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:37 smithi137 ceph-mon[39175]: osdmap e75: 8 total, 8 up, 8 in 2024-09-18T07:31:38.109 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:37 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:31:38.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:37 smithi203 ceph-mon[32237]: osdmap e75: 8 total, 8 up, 8 in 2024-09-18T07:31:38.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:37 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:31:38.402 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724565 got 167503724566 for osd.5 2024-09-18T07:31:38.403 DEBUG:teuthology.parallel:result is None 2024-09-18T07:31:38.970 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:38 smithi137 ceph-mon[29408]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-09-18T07:31:38.971 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:38 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3108111317' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-18T07:31:38.971 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:38 smithi137 ceph-mon[39175]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-09-18T07:31:38.971 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:38 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3108111317' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-18T07:31:38.997 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462929 got 201863462930 for osd.6 2024-09-18T07:31:38.998 DEBUG:teuthology.parallel:result is None 2024-09-18T07:31:39.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:38 smithi203 ceph-mon[32237]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-09-18T07:31:39.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:38 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3108111317' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-18T07:31:39.549 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:39.550 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:39.550 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:39.696 INFO:teuthology.orchestra.run.smithi137.stdout:77309411368 2024-09-18T07:31:40.024 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:39 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1709759350' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-18T07:31:40.024 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:39 smithi137 ceph-mon[29408]: osdmap e76: 8 total, 8 up, 8 in 2024-09-18T07:31:40.025 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:39 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1709759350' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-18T07:31:40.025 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:39 smithi137 ceph-mon[39175]: osdmap e76: 8 total, 8 up, 8 in 2024-09-18T07:31:40.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:39 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1709759350' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-18T07:31:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:39 smithi203 ceph-mon[32237]: osdmap e76: 8 total, 8 up, 8 in 2024-09-18T07:31:40.724 INFO:teuthology.orchestra.run.smithi137.stdout:137438953502 2024-09-18T07:31:41.084 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:40 smithi137 ceph-mon[29408]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-18T07:31:41.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:40 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/590085654' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-18T07:31:41.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:40 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:31:41.085 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:40 smithi137 ceph-mon[39175]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-18T07:31:41.085 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:40 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/590085654' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-18T07:31:41.085 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:40 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:31:41.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:40 smithi203 ceph-mon[32237]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-18T07:31:41.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:40 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/590085654' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-18T07:31:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:40 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:31:41.352 INFO:teuthology.orchestra.run.smithi137.stdout:34359738420 2024-09-18T07:31:41.624 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953501 got 137438953502 for osd.4 2024-09-18T07:31:41.624 DEBUG:teuthology.parallel:result is None 2024-09-18T07:31:41.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:41 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3535158470' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-18T07:31:41.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:41 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3535158470' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-18T07:31:42.042 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411365 got 77309411368 for osd.2 2024-09-18T07:31:42.043 DEBUG:teuthology.parallel:result is None 2024-09-18T07:31:42.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:41 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3535158470' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-18T07:31:42.881 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738420 for osd.0 2024-09-18T07:31:42.881 DEBUG:teuthology.parallel:result is None 2024-09-18T07:31:43.020 INFO:teuthology.orchestra.run.smithi137.stdout:236223201293 2024-09-18T07:31:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:42 smithi137 ceph-mon[29408]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:31:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:42 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2385043976' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-18T07:31:43.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:42 smithi137 ceph-mon[39175]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:31:43.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:42 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2385043976' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-18T07:31:43.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:42 smithi203 ceph-mon[32237]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:31:43.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:42 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2385043976' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-18T07:31:43.696 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201290 got 236223201293 for osd.7 2024-09-18T07:31:43.696 DEBUG:teuthology.parallel:result is None 2024-09-18T07:31:43.696 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-18T07:31:43.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:31:43.972 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:44.640 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:31:44.642 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:31:44.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:31:44.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:44.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:44.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:31:44.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:31:44.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:44.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:44.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:44.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:44.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:44.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:44.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:44.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:44.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:44.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:44.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:44.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:31:44.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:31:44.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:31:44.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:31:44.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:31:44.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:44.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:31:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:31:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:44.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:31:44.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:31:44.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:31:44.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:44 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:31:45.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:31:45.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:45.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:45.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:31:45.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:31:45.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:45.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:45.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:45.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:45.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:45.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:45.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:45.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:45.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:45.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T07:31:45.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:45.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:31:45.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:31:45.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:31:45.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:44 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:31:45.423 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-18T07:31:43.407632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":888,"num_read_kb":745,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236480,"kb_used_data":7772,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724736,"statfs":{"total":767960285184,"available":767718129664,"internally_reserved":0,"allocated":7958528,"data_stored":4215769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001596"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193595+0000","last_change":"2024-09-18T07:30:56.284087+0000","last_active":"2024-09-18T07:31:40.193595+0000","last_peered":"2024-09-18T07:31:40.193595+0000","last_clean":"2024-09-18T07:31:40.193595+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:31:40.193595+0000","last_undegraded":"2024-09-18T07:31:40.193595+0000","last_fullsized":"2024-09-18T07:31:40.193595+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.049077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724596+0000","last_change":"2024-09-18T07:30:58.309315+0000","last_active":"2024-09-18T07:31:39.724596+0000","last_peered":"2024-09-18T07:31:39.724596+0000","last_clean":"2024-09-18T07:31:39.724596+0000","last_became_active":"2024-09-18T07:30:58.308981+0000","last_became_peered":"2024-09-18T07:30:58.308981+0000","last_unstale":"2024-09-18T07:31:39.724596+0000","last_undegraded":"2024-09-18T07:31:39.724596+0000","last_fullsized":"2024-09-18T07:31:39.724596+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:08.376023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723770+0000","last_change":"2024-09-18T07:31:00.336284+0000","last_active":"2024-09-18T07:31:39.723770+0000","last_peered":"2024-09-18T07:31:39.723770+0000","last_clean":"2024-09-18T07:31:39.723770+0000","last_became_active":"2024-09-18T07:31:00.335958+0000","last_became_peered":"2024-09-18T07:31:00.335958+0000","last_unstale":"2024-09-18T07:31:39.723770+0000","last_undegraded":"2024-09-18T07:31:39.723770+0000","last_fullsized":"2024-09-18T07:31:39.723770+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:08.645781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724775+0000","last_change":"2024-09-18T07:31:02.342510+0000","last_active":"2024-09-18T07:31:39.724775+0000","last_peered":"2024-09-18T07:31:39.724775+0000","last_clean":"2024-09-18T07:31:39.724775+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:31:39.724775+0000","last_undegraded":"2024-09-18T07:31:39.724775+0000","last_fullsized":"2024-09-18T07:31:39.724775+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.002947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193793+0000","last_change":"2024-09-18T07:31:02.492941+0000","last_active":"2024-09-18T07:31:40.193793+0000","last_peered":"2024-09-18T07:31:40.193793+0000","last_clean":"2024-09-18T07:31:40.193793+0000","last_became_active":"2024-09-18T07:31:02.492405+0000","last_became_peered":"2024-09-18T07:31:02.492405+0000","last_unstale":"2024-09-18T07:31:40.193793+0000","last_undegraded":"2024-09-18T07:31:40.193793+0000","last_fullsized":"2024-09-18T07:31:40.193793+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:22.665514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194149+0000","last_change":"2024-09-18T07:30:56.286313+0000","last_active":"2024-09-18T07:31:40.194149+0000","last_peered":"2024-09-18T07:31:40.194149+0000","last_clean":"2024-09-18T07:31:40.194149+0000","last_became_active":"2024-09-18T07:30:56.286040+0000","last_became_peered":"2024-09-18T07:30:56.286040+0000","last_unstale":"2024-09-18T07:31:40.194149+0000","last_undegraded":"2024-09-18T07:31:40.194149+0000","last_fullsized":"2024-09-18T07:31:40.194149+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:09.154352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":63,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194399+0000","last_change":"2024-09-18T07:30:58.314153+0000","last_active":"2024-09-18T07:31:40.194399+0000","last_peered":"2024-09-18T07:31:40.194399+0000","last_clean":"2024-09-18T07:31:40.194399+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:31:40.194399+0000","last_undegraded":"2024-09-18T07:31:40.194399+0000","last_fullsized":"2024-09-18T07:31:40.194399+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.980863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724596+0000","last_change":"2024-09-18T07:31:00.331498+0000","last_active":"2024-09-18T07:31:39.724596+0000","last_peered":"2024-09-18T07:31:39.724596+0000","last_clean":"2024-09-18T07:31:39.724596+0000","last_became_active":"2024-09-18T07:31:00.331224+0000","last_became_peered":"2024-09-18T07:31:00.331224+0000","last_unstale":"2024-09-18T07:31:39.724596+0000","last_undegraded":"2024-09-18T07:31:39.724596+0000","last_fullsized":"2024-09-18T07:31:39.724596+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:15.385510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195908+0000","last_change":"2024-09-18T07:30:56.286407+0000","last_active":"2024-09-18T07:31:40.195908+0000","last_peered":"2024-09-18T07:31:40.195908+0000","last_clean":"2024-09-18T07:31:40.195908+0000","last_became_active":"2024-09-18T07:30:56.285882+0000","last_became_peered":"2024-09-18T07:30:56.285882+0000","last_unstale":"2024-09-18T07:31:40.195908+0000","last_undegraded":"2024-09-18T07:31:40.195908+0000","last_fullsized":"2024-09-18T07:31:40.195908+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.252387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724970+0000","last_change":"2024-09-18T07:30:58.317018+0000","last_active":"2024-09-18T07:31:39.724970+0000","last_peered":"2024-09-18T07:31:39.724970+0000","last_clean":"2024-09-18T07:31:39.724970+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:31:39.724970+0000","last_undegraded":"2024-09-18T07:31:39.724970+0000","last_fullsized":"2024-09-18T07:31:39.724970+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:59.135054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195286+0000","last_change":"2024-09-18T07:31:00.330533+0000","last_active":"2024-09-18T07:31:40.195286+0000","last_peered":"2024-09-18T07:31:40.195286+0000","last_clean":"2024-09-18T07:31:40.195286+0000","last_became_active":"2024-09-18T07:31:00.330272+0000","last_became_peered":"2024-09-18T07:31:00.330272+0000","last_unstale":"2024-09-18T07:31:40.195286+0000","last_undegraded":"2024-09-18T07:31:40.195286+0000","last_fullsized":"2024-09-18T07:31:40.195286+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.245066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193105+0000","last_change":"2024-09-18T07:31:02.353810+0000","last_active":"2024-09-18T07:31:40.193105+0000","last_peered":"2024-09-18T07:31:40.193105+0000","last_clean":"2024-09-18T07:31:40.193105+0000","last_became_active":"2024-09-18T07:31:02.353494+0000","last_became_peered":"2024-09-18T07:31:02.353494+0000","last_unstale":"2024-09-18T07:31:40.193105+0000","last_undegraded":"2024-09-18T07:31:40.193105+0000","last_fullsized":"2024-09-18T07:31:40.193105+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.436492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195548+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:31:40.195548+0000","last_peered":"2024-09-18T07:31:40.195548+0000","last_clean":"2024-09-18T07:31:40.195548+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:31:40.195548+0000","last_undegraded":"2024-09-18T07:31:40.195548+0000","last_fullsized":"2024-09-18T07:31:40.195548+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.419105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725044+0000","last_change":"2024-09-18T07:30:58.317041+0000","last_active":"2024-09-18T07:31:39.725044+0000","last_peered":"2024-09-18T07:31:39.725044+0000","last_clean":"2024-09-18T07:31:39.725044+0000","last_became_active":"2024-09-18T07:30:58.316479+0000","last_became_peered":"2024-09-18T07:30:58.316479+0000","last_unstale":"2024-09-18T07:31:39.725044+0000","last_undegraded":"2024-09-18T07:31:39.725044+0000","last_fullsized":"2024-09-18T07:31:39.725044+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.191081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194451+0000","last_change":"2024-09-18T07:31:00.334023+0000","last_active":"2024-09-18T07:31:40.194451+0000","last_peered":"2024-09-18T07:31:40.194451+0000","last_clean":"2024-09-18T07:31:40.194451+0000","last_became_active":"2024-09-18T07:31:00.333685+0000","last_became_peered":"2024-09-18T07:31:00.333685+0000","last_unstale":"2024-09-18T07:31:40.194451+0000","last_undegraded":"2024-09-18T07:31:40.194451+0000","last_fullsized":"2024-09-18T07:31:40.194451+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.774482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194831+0000","last_change":"2024-09-18T07:31:02.349964+0000","last_active":"2024-09-18T07:31:40.194831+0000","last_peered":"2024-09-18T07:31:40.194831+0000","last_clean":"2024-09-18T07:31:40.194831+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:31:40.194831+0000","last_undegraded":"2024-09-18T07:31:40.194831+0000","last_fullsized":"2024-09-18T07:31:40.194831+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.932342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725534+0000","last_change":"2024-09-18T07:31:02.490397+0000","last_active":"2024-09-18T07:31:39.725534+0000","last_peered":"2024-09-18T07:31:39.725534+0000","last_clean":"2024-09-18T07:31:39.725534+0000","last_became_active":"2024-09-18T07:31:02.490014+0000","last_became_peered":"2024-09-18T07:31:02.490014+0000","last_unstale":"2024-09-18T07:31:39.725534+0000","last_undegraded":"2024-09-18T07:31:39.725534+0000","last_fullsized":"2024-09-18T07:31:39.725534+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.792505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193956+0000","last_change":"2024-09-18T07:30:56.288891+0000","last_active":"2024-09-18T07:31:40.193956+0000","last_peered":"2024-09-18T07:31:40.193956+0000","last_clean":"2024-09-18T07:31:40.193956+0000","last_became_active":"2024-09-18T07:30:56.288589+0000","last_became_peered":"2024-09-18T07:30:56.288589+0000","last_unstale":"2024-09-18T07:31:40.193956+0000","last_undegraded":"2024-09-18T07:31:40.193956+0000","last_fullsized":"2024-09-18T07:31:40.193956+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.974097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":63,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725072+0000","last_change":"2024-09-18T07:30:58.316556+0000","last_active":"2024-09-18T07:31:39.725072+0000","last_peered":"2024-09-18T07:31:39.725072+0000","last_clean":"2024-09-18T07:31:39.725072+0000","last_became_active":"2024-09-18T07:30:58.316266+0000","last_became_peered":"2024-09-18T07:30:58.316266+0000","last_unstale":"2024-09-18T07:31:39.725072+0000","last_undegraded":"2024-09-18T07:31:39.725072+0000","last_fullsized":"2024-09-18T07:31:39.725072+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:22.570802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725127+0000","last_change":"2024-09-18T07:31:00.338287+0000","last_active":"2024-09-18T07:31:39.725127+0000","last_peered":"2024-09-18T07:31:39.725127+0000","last_clean":"2024-09-18T07:31:39.725127+0000","last_became_active":"2024-09-18T07:31:00.337474+0000","last_became_peered":"2024-09-18T07:31:00.337474+0000","last_unstale":"2024-09-18T07:31:39.725127+0000","last_undegraded":"2024-09-18T07:31:39.725127+0000","last_fullsized":"2024-09-18T07:31:39.725127+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.609543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194017+0000","last_change":"2024-09-18T07:31:02.492790+0000","last_active":"2024-09-18T07:31:40.194017+0000","last_peered":"2024-09-18T07:31:40.194017+0000","last_clean":"2024-09-18T07:31:40.194017+0000","last_became_active":"2024-09-18T07:31:02.492348+0000","last_became_peered":"2024-09-18T07:31:02.492348+0000","last_unstale":"2024-09-18T07:31:40.194017+0000","last_undegraded":"2024-09-18T07:31:40.194017+0000","last_fullsized":"2024-09-18T07:31:40.194017+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:08.542586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725108+0000","last_change":"2024-09-18T07:30:56.289489+0000","last_active":"2024-09-18T07:31:39.725108+0000","last_peered":"2024-09-18T07:31:39.725108+0000","last_clean":"2024-09-18T07:31:39.725108+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:31:39.725108+0000","last_undegraded":"2024-09-18T07:31:39.725108+0000","last_fullsized":"2024-09-18T07:31:39.725108+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.331217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":61,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194047+0000","last_change":"2024-09-18T07:30:58.319115+0000","last_active":"2024-09-18T07:31:40.194047+0000","last_peered":"2024-09-18T07:31:40.194047+0000","last_clean":"2024-09-18T07:31:40.194047+0000","last_became_active":"2024-09-18T07:30:58.316380+0000","last_became_peered":"2024-09-18T07:30:58.316380+0000","last_unstale":"2024-09-18T07:31:40.194047+0000","last_undegraded":"2024-09-18T07:31:40.194047+0000","last_fullsized":"2024-09-18T07:31:40.194047+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.995221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725226+0000","last_change":"2024-09-18T07:31:00.334313+0000","last_active":"2024-09-18T07:31:39.725226+0000","last_peered":"2024-09-18T07:31:39.725226+0000","last_clean":"2024-09-18T07:31:39.725226+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:31:39.725226+0000","last_undegraded":"2024-09-18T07:31:39.725226+0000","last_fullsized":"2024-09-18T07:31:39.725226+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:09.373920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194653+0000","last_change":"2024-09-18T07:31:02.348855+0000","last_active":"2024-09-18T07:31:40.194653+0000","last_peered":"2024-09-18T07:31:40.194653+0000","last_clean":"2024-09-18T07:31:40.194653+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:31:40.194653+0000","last_undegraded":"2024-09-18T07:31:40.194653+0000","last_fullsized":"2024-09-18T07:31:40.194653+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.101140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723909+0000","last_change":"2024-09-18T07:30:56.283995+0000","last_active":"2024-09-18T07:31:39.723909+0000","last_peered":"2024-09-18T07:31:39.723909+0000","last_clean":"2024-09-18T07:31:39.723909+0000","last_became_active":"2024-09-18T07:30:56.283705+0000","last_became_peered":"2024-09-18T07:30:56.283705+0000","last_unstale":"2024-09-18T07:31:39.723909+0000","last_undegraded":"2024-09-18T07:31:39.723909+0000","last_fullsized":"2024-09-18T07:31:39.723909+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:55.938900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193741+0000","last_change":"2024-09-18T07:30:58.318003+0000","last_active":"2024-09-18T07:31:40.193741+0000","last_peered":"2024-09-18T07:31:40.193741+0000","last_clean":"2024-09-18T07:31:40.193741+0000","last_became_active":"2024-09-18T07:30:58.317355+0000","last_became_peered":"2024-09-18T07:30:58.317355+0000","last_unstale":"2024-09-18T07:31:40.193741+0000","last_undegraded":"2024-09-18T07:31:40.193741+0000","last_fullsized":"2024-09-18T07:31:40.193741+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.431023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194985+0000","last_change":"2024-09-18T07:31:00.331318+0000","last_active":"2024-09-18T07:31:40.194985+0000","last_peered":"2024-09-18T07:31:40.194985+0000","last_clean":"2024-09-18T07:31:40.194985+0000","last_became_active":"2024-09-18T07:31:00.331014+0000","last_became_peered":"2024-09-18T07:31:00.331014+0000","last_unstale":"2024-09-18T07:31:40.194985+0000","last_undegraded":"2024-09-18T07:31:40.194985+0000","last_fullsized":"2024-09-18T07:31:40.194985+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.342355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":63,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725256+0000","last_change":"2024-09-18T07:30:58.312916+0000","last_active":"2024-09-18T07:31:39.725256+0000","last_peered":"2024-09-18T07:31:39.725256+0000","last_clean":"2024-09-18T07:31:39.725256+0000","last_became_active":"2024-09-18T07:30:58.312627+0000","last_became_peered":"2024-09-18T07:30:58.312627+0000","last_unstale":"2024-09-18T07:31:39.725256+0000","last_undegraded":"2024-09-18T07:31:39.725256+0000","last_fullsized":"2024-09-18T07:31:39.725256+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.188948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193595+0000","last_change":"2024-09-18T07:30:56.289331+0000","last_active":"2024-09-18T07:31:40.193595+0000","last_peered":"2024-09-18T07:31:40.193595+0000","last_clean":"2024-09-18T07:31:40.193595+0000","last_became_active":"2024-09-18T07:30:56.289081+0000","last_became_peered":"2024-09-18T07:30:56.289081+0000","last_unstale":"2024-09-18T07:31:40.193595+0000","last_undegraded":"2024-09-18T07:31:40.193595+0000","last_fullsized":"2024-09-18T07:31:40.193595+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:56.283698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725726+0000","last_change":"2024-09-18T07:31:00.333026+0000","last_active":"2024-09-18T07:31:39.725726+0000","last_peered":"2024-09-18T07:31:39.725726+0000","last_clean":"2024-09-18T07:31:39.725726+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:31:39.725726+0000","last_undegraded":"2024-09-18T07:31:39.725726+0000","last_fullsized":"2024-09-18T07:31:39.725726+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:09.647541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195545+0000","last_change":"2024-09-18T07:31:02.351412+0000","last_active":"2024-09-18T07:31:40.195545+0000","last_peered":"2024-09-18T07:31:40.195545+0000","last_clean":"2024-09-18T07:31:40.195545+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:31:40.195545+0000","last_undegraded":"2024-09-18T07:31:40.195545+0000","last_fullsized":"2024-09-18T07:31:40.195545+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.243657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":70,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193495+0000","last_change":"2024-09-18T07:30:58.321857+0000","last_active":"2024-09-18T07:31:40.193495+0000","last_peered":"2024-09-18T07:31:40.193495+0000","last_clean":"2024-09-18T07:31:40.193495+0000","last_became_active":"2024-09-18T07:30:58.321244+0000","last_became_peered":"2024-09-18T07:30:58.321244+0000","last_unstale":"2024-09-18T07:31:40.193495+0000","last_undegraded":"2024-09-18T07:31:40.193495+0000","last_fullsized":"2024-09-18T07:31:40.193495+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:58.899213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194441+0000","last_change":"2024-09-18T07:30:56.289862+0000","last_active":"2024-09-18T07:31:40.194441+0000","last_peered":"2024-09-18T07:31:40.194441+0000","last_clean":"2024-09-18T07:31:40.194441+0000","last_became_active":"2024-09-18T07:30:56.289556+0000","last_became_peered":"2024-09-18T07:30:56.289556+0000","last_unstale":"2024-09-18T07:31:40.194441+0000","last_undegraded":"2024-09-18T07:31:40.194441+0000","last_fullsized":"2024-09-18T07:31:40.194441+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:59.535940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724628+0000","last_change":"2024-09-18T07:31:00.333320+0000","last_active":"2024-09-18T07:31:39.724628+0000","last_peered":"2024-09-18T07:31:39.724628+0000","last_clean":"2024-09-18T07:31:39.724628+0000","last_became_active":"2024-09-18T07:31:00.332854+0000","last_became_peered":"2024-09-18T07:31:00.332854+0000","last_unstale":"2024-09-18T07:31:39.724628+0000","last_undegraded":"2024-09-18T07:31:39.724628+0000","last_fullsized":"2024-09-18T07:31:39.724628+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:20.691030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724958+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:31:39.724958+0000","last_peered":"2024-09-18T07:31:39.724958+0000","last_clean":"2024-09-18T07:31:39.724958+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:31:39.724958+0000","last_undegraded":"2024-09-18T07:31:39.724958+0000","last_fullsized":"2024-09-18T07:31:39.724958+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.039810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724630+0000","last_change":"2024-09-18T07:30:58.309938+0000","last_active":"2024-09-18T07:31:39.724630+0000","last_peered":"2024-09-18T07:31:39.724630+0000","last_clean":"2024-09-18T07:31:39.724630+0000","last_became_active":"2024-09-18T07:30:58.309649+0000","last_became_peered":"2024-09-18T07:30:58.309649+0000","last_unstale":"2024-09-18T07:31:39.724630+0000","last_undegraded":"2024-09-18T07:31:39.724630+0000","last_fullsized":"2024-09-18T07:31:39.724630+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:20.121712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193999+0000","last_change":"2024-09-18T07:30:56.287909+0000","last_active":"2024-09-18T07:31:40.193999+0000","last_peered":"2024-09-18T07:31:40.193999+0000","last_clean":"2024-09-18T07:31:40.193999+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:31:40.193999+0000","last_undegraded":"2024-09-18T07:31:40.193999+0000","last_fullsized":"2024-09-18T07:31:40.193999+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:05.243933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193351+0000","last_change":"2024-09-18T07:31:00.336059+0000","last_active":"2024-09-18T07:31:40.193351+0000","last_peered":"2024-09-18T07:31:40.193351+0000","last_clean":"2024-09-18T07:31:40.193351+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:31:40.193351+0000","last_undegraded":"2024-09-18T07:31:40.193351+0000","last_fullsized":"2024-09-18T07:31:40.193351+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.046222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195090+0000","last_change":"2024-09-18T07:31:02.492797+0000","last_active":"2024-09-18T07:31:40.195090+0000","last_peered":"2024-09-18T07:31:40.195090+0000","last_clean":"2024-09-18T07:31:40.195090+0000","last_became_active":"2024-09-18T07:31:02.492483+0000","last_became_peered":"2024-09-18T07:31:02.492483+0000","last_unstale":"2024-09-18T07:31:40.195090+0000","last_undegraded":"2024-09-18T07:31:40.195090+0000","last_fullsized":"2024-09-18T07:31:40.195090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.277226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724110+0000","last_change":"2024-09-18T07:30:58.307470+0000","last_active":"2024-09-18T07:31:39.724110+0000","last_peered":"2024-09-18T07:31:39.724110+0000","last_clean":"2024-09-18T07:31:39.724110+0000","last_became_active":"2024-09-18T07:30:58.307035+0000","last_became_peered":"2024-09-18T07:30:58.307035+0000","last_unstale":"2024-09-18T07:31:39.724110+0000","last_undegraded":"2024-09-18T07:31:39.724110+0000","last_fullsized":"2024-09-18T07:31:39.724110+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:06.964257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.196031+0000","last_change":"2024-09-18T07:30:56.290040+0000","last_active":"2024-09-18T07:31:40.196031+0000","last_peered":"2024-09-18T07:31:40.196031+0000","last_clean":"2024-09-18T07:31:40.196031+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:31:40.196031+0000","last_undegraded":"2024-09-18T07:31:40.196031+0000","last_fullsized":"2024-09-18T07:31:40.196031+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.332869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193654+0000","last_change":"2024-09-18T07:31:00.337241+0000","last_active":"2024-09-18T07:31:40.193654+0000","last_peered":"2024-09-18T07:31:40.193654+0000","last_clean":"2024-09-18T07:31:40.193654+0000","last_became_active":"2024-09-18T07:31:00.336905+0000","last_became_peered":"2024-09-18T07:31:00.336905+0000","last_unstale":"2024-09-18T07:31:40.193654+0000","last_undegraded":"2024-09-18T07:31:40.193654+0000","last_fullsized":"2024-09-18T07:31:40.193654+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.826427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193237+0000","last_change":"2024-09-18T07:31:02.348538+0000","last_active":"2024-09-18T07:31:40.193237+0000","last_peered":"2024-09-18T07:31:40.193237+0000","last_clean":"2024-09-18T07:31:40.193237+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:31:40.193237+0000","last_undegraded":"2024-09-18T07:31:40.193237+0000","last_fullsized":"2024-09-18T07:31:40.193237+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.088828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":74,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193703+0000","last_change":"2024-09-18T07:30:58.318096+0000","last_active":"2024-09-18T07:31:40.193703+0000","last_peered":"2024-09-18T07:31:40.193703+0000","last_clean":"2024-09-18T07:31:40.193703+0000","last_became_active":"2024-09-18T07:30:58.317767+0000","last_became_peered":"2024-09-18T07:30:58.317767+0000","last_unstale":"2024-09-18T07:31:40.193703+0000","last_undegraded":"2024-09-18T07:31:40.193703+0000","last_fullsized":"2024-09-18T07:31:40.193703+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:26.992100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724928+0000","last_change":"2024-09-18T07:30:56.284440+0000","last_active":"2024-09-18T07:31:39.724928+0000","last_peered":"2024-09-18T07:31:39.724928+0000","last_clean":"2024-09-18T07:31:39.724928+0000","last_became_active":"2024-09-18T07:30:56.284120+0000","last_became_peered":"2024-09-18T07:30:56.284120+0000","last_unstale":"2024-09-18T07:31:39.724928+0000","last_undegraded":"2024-09-18T07:31:39.724928+0000","last_fullsized":"2024-09-18T07:31:39.724928+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.025187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193979+0000","last_change":"2024-09-18T07:31:00.335100+0000","last_active":"2024-09-18T07:31:40.193979+0000","last_peered":"2024-09-18T07:31:40.193979+0000","last_clean":"2024-09-18T07:31:40.193979+0000","last_became_active":"2024-09-18T07:31:00.334776+0000","last_became_peered":"2024-09-18T07:31:00.334776+0000","last_unstale":"2024-09-18T07:31:40.193979+0000","last_undegraded":"2024-09-18T07:31:40.193979+0000","last_fullsized":"2024-09-18T07:31:40.193979+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.405293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723750+0000","last_change":"2024-09-18T07:31:02.490534+0000","last_active":"2024-09-18T07:31:39.723750+0000","last_peered":"2024-09-18T07:31:39.723750+0000","last_clean":"2024-09-18T07:31:39.723750+0000","last_became_active":"2024-09-18T07:31:02.490186+0000","last_became_peered":"2024-09-18T07:31:02.490186+0000","last_unstale":"2024-09-18T07:31:39.723750+0000","last_undegraded":"2024-09-18T07:31:39.723750+0000","last_fullsized":"2024-09-18T07:31:39.723750+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.881705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":96,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724845+0000","last_change":"2024-09-18T07:30:58.317477+0000","last_active":"2024-09-18T07:31:39.724845+0000","last_peered":"2024-09-18T07:31:39.724845+0000","last_clean":"2024-09-18T07:31:39.724845+0000","last_became_active":"2024-09-18T07:30:58.316812+0000","last_became_peered":"2024-09-18T07:30:58.316812+0000","last_unstale":"2024-09-18T07:31:39.724845+0000","last_undegraded":"2024-09-18T07:31:39.724845+0000","last_fullsized":"2024-09-18T07:31:39.724845+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.769678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725296+0000","last_change":"2024-09-18T07:30:56.285353+0000","last_active":"2024-09-18T07:31:39.725296+0000","last_peered":"2024-09-18T07:31:39.725296+0000","last_clean":"2024-09-18T07:31:39.725296+0000","last_became_active":"2024-09-18T07:30:56.285032+0000","last_became_peered":"2024-09-18T07:30:56.285032+0000","last_unstale":"2024-09-18T07:31:39.725296+0000","last_undegraded":"2024-09-18T07:31:39.725296+0000","last_fullsized":"2024-09-18T07:31:39.725296+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:23.187833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195617+0000","last_change":"2024-09-18T07:30:58.299308+0000","last_active":"2024-09-18T07:31:40.195617+0000","last_peered":"2024-09-18T07:31:40.195617+0000","last_clean":"2024-09-18T07:31:40.195617+0000","last_became_active":"2024-09-18T07:30:55.283292+0000","last_became_peered":"2024-09-18T07:30:55.283292+0000","last_unstale":"2024-09-18T07:31:40.195617+0000","last_undegraded":"2024-09-18T07:31:40.195617+0000","last_fullsized":"2024-09-18T07:31:40.195617+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:13.332916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193537+0000","last_change":"2024-09-18T07:31:00.332738+0000","last_active":"2024-09-18T07:31:40.193537+0000","last_peered":"2024-09-18T07:31:40.193537+0000","last_clean":"2024-09-18T07:31:40.193537+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:31:40.193537+0000","last_undegraded":"2024-09-18T07:31:40.193537+0000","last_fullsized":"2024-09-18T07:31:40.193537+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:23.578215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193936+0000","last_change":"2024-09-18T07:31:02.351778+0000","last_active":"2024-09-18T07:31:40.193936+0000","last_peered":"2024-09-18T07:31:40.193936+0000","last_clean":"2024-09-18T07:31:40.193936+0000","last_became_active":"2024-09-18T07:31:02.351403+0000","last_became_peered":"2024-09-18T07:31:02.351403+0000","last_unstale":"2024-09-18T07:31:40.193936+0000","last_undegraded":"2024-09-18T07:31:40.193936+0000","last_fullsized":"2024-09-18T07:31:40.193936+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:27.871124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":65,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725277+0000","last_change":"2024-09-18T07:30:58.310789+0000","last_active":"2024-09-18T07:31:39.725277+0000","last_peered":"2024-09-18T07:31:39.725277+0000","last_clean":"2024-09-18T07:31:39.725277+0000","last_became_active":"2024-09-18T07:30:58.310490+0000","last_became_peered":"2024-09-18T07:30:58.310490+0000","last_unstale":"2024-09-18T07:31:39.725277+0000","last_undegraded":"2024-09-18T07:31:39.725277+0000","last_fullsized":"2024-09-18T07:31:39.725277+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:22.258389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723760+0000","last_change":"2024-09-18T07:30:56.285054+0000","last_active":"2024-09-18T07:31:39.723760+0000","last_peered":"2024-09-18T07:31:39.723760+0000","last_clean":"2024-09-18T07:31:39.723760+0000","last_became_active":"2024-09-18T07:30:56.284803+0000","last_became_peered":"2024-09-18T07:30:56.284803+0000","last_unstale":"2024-09-18T07:31:39.723760+0000","last_undegraded":"2024-09-18T07:31:39.723760+0000","last_fullsized":"2024-09-18T07:31:39.723760+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.764106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":51,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723587+0000","last_change":"2024-09-18T07:30:58.301149+0000","last_active":"2024-09-18T07:31:39.723587+0000","last_peered":"2024-09-18T07:31:39.723587+0000","last_clean":"2024-09-18T07:31:39.723587+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:31:39.723587+0000","last_undegraded":"2024-09-18T07:31:39.723587+0000","last_fullsized":"2024-09-18T07:31:39.723587+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:01.827803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724199+0000","last_change":"2024-09-18T07:31:00.331226+0000","last_active":"2024-09-18T07:31:39.724199+0000","last_peered":"2024-09-18T07:31:39.724199+0000","last_clean":"2024-09-18T07:31:39.724199+0000","last_became_active":"2024-09-18T07:31:00.330883+0000","last_became_peered":"2024-09-18T07:31:00.330883+0000","last_unstale":"2024-09-18T07:31:39.724199+0000","last_undegraded":"2024-09-18T07:31:39.724199+0000","last_fullsized":"2024-09-18T07:31:39.724199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.695331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194820+0000","last_change":"2024-09-18T07:31:02.491512+0000","last_active":"2024-09-18T07:31:40.194820+0000","last_peered":"2024-09-18T07:31:40.194820+0000","last_clean":"2024-09-18T07:31:40.194820+0000","last_became_active":"2024-09-18T07:31:02.491233+0000","last_became_peered":"2024-09-18T07:31:02.491233+0000","last_unstale":"2024-09-18T07:31:40.194820+0000","last_undegraded":"2024-09-18T07:31:40.194820+0000","last_fullsized":"2024-09-18T07:31:40.194820+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:26.099794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":56,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193830+0000","last_change":"2024-09-18T07:30:58.311778+0000","last_active":"2024-09-18T07:31:40.193830+0000","last_peered":"2024-09-18T07:31:40.193830+0000","last_clean":"2024-09-18T07:31:40.193830+0000","last_became_active":"2024-09-18T07:30:58.311491+0000","last_became_peered":"2024-09-18T07:30:58.311491+0000","last_unstale":"2024-09-18T07:31:40.193830+0000","last_undegraded":"2024-09-18T07:31:40.193830+0000","last_fullsized":"2024-09-18T07:31:40.193830+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:05.875741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193169+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:31:40.193169+0000","last_peered":"2024-09-18T07:31:40.193169+0000","last_clean":"2024-09-18T07:31:40.193169+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:31:40.193169+0000","last_undegraded":"2024-09-18T07:31:40.193169+0000","last_fullsized":"2024-09-18T07:31:40.193169+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:20.989190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":93,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194586+0000","last_change":"2024-09-18T07:30:58.319969+0000","last_active":"2024-09-18T07:31:40.194586+0000","last_peered":"2024-09-18T07:31:40.194586+0000","last_clean":"2024-09-18T07:31:40.194586+0000","last_became_active":"2024-09-18T07:30:55.286980+0000","last_became_peered":"2024-09-18T07:30:55.286980+0000","last_unstale":"2024-09-18T07:31:40.194586+0000","last_undegraded":"2024-09-18T07:31:40.194586+0000","last_fullsized":"2024-09-18T07:31:40.194586+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.742429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.196240+0000","last_change":"2024-09-18T07:31:00.331929+0000","last_active":"2024-09-18T07:31:40.196240+0000","last_peered":"2024-09-18T07:31:40.196240+0000","last_clean":"2024-09-18T07:31:40.196240+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:31:40.196240+0000","last_undegraded":"2024-09-18T07:31:40.196240+0000","last_fullsized":"2024-09-18T07:31:40.196240+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.504554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724544+0000","last_change":"2024-09-18T07:31:02.347659+0000","last_active":"2024-09-18T07:31:39.724544+0000","last_peered":"2024-09-18T07:31:39.724544+0000","last_clean":"2024-09-18T07:31:39.724544+0000","last_became_active":"2024-09-18T07:31:02.347390+0000","last_became_peered":"2024-09-18T07:31:02.347390+0000","last_unstale":"2024-09-18T07:31:39.724544+0000","last_undegraded":"2024-09-18T07:31:39.724544+0000","last_fullsized":"2024-09-18T07:31:39.724544+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.353794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":70,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194489+0000","last_change":"2024-09-18T07:30:58.318426+0000","last_active":"2024-09-18T07:31:40.194489+0000","last_peered":"2024-09-18T07:31:40.194489+0000","last_clean":"2024-09-18T07:31:40.194489+0000","last_became_active":"2024-09-18T07:30:58.318068+0000","last_became_peered":"2024-09-18T07:30:58.318068+0000","last_unstale":"2024-09-18T07:31:40.194489+0000","last_undegraded":"2024-09-18T07:31:40.194489+0000","last_fullsized":"2024-09-18T07:31:40.194489+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:59.863882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193876+0000","last_change":"2024-09-18T07:30:56.288420+0000","last_active":"2024-09-18T07:31:40.193876+0000","last_peered":"2024-09-18T07:31:40.193876+0000","last_clean":"2024-09-18T07:31:40.193876+0000","last_became_active":"2024-09-18T07:30:56.288149+0000","last_became_peered":"2024-09-18T07:30:56.288149+0000","last_unstale":"2024-09-18T07:31:40.193876+0000","last_undegraded":"2024-09-18T07:31:40.193876+0000","last_fullsized":"2024-09-18T07:31:40.193876+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:56.385307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":89,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194761+0000","last_change":"2024-09-18T07:30:50.813696+0000","last_active":"2024-09-18T07:31:40.194761+0000","last_peered":"2024-09-18T07:31:40.194761+0000","last_clean":"2024-09-18T07:31:40.194761+0000","last_became_active":"2024-09-18T07:30:46.719588+0000","last_became_peered":"2024-09-18T07:30:46.719588+0000","last_unstale":"2024-09-18T07:31:40.194761+0000","last_undegraded":"2024-09-18T07:31:40.194761+0000","last_fullsized":"2024-09-18T07:31:40.194761+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-18T07:30:50.813645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:30:50.813645+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:06.405856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195451+0000","last_change":"2024-09-18T07:31:00.331602+0000","last_active":"2024-09-18T07:31:40.195451+0000","last_peered":"2024-09-18T07:31:40.195451+0000","last_clean":"2024-09-18T07:31:40.195451+0000","last_became_active":"2024-09-18T07:31:00.331327+0000","last_became_peered":"2024-09-18T07:31:00.331327+0000","last_unstale":"2024-09-18T07:31:40.195451+0000","last_undegraded":"2024-09-18T07:31:40.195451+0000","last_fullsized":"2024-09-18T07:31:40.195451+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.402435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.196137+0000","last_change":"2024-09-18T07:31:02.348225+0000","last_active":"2024-09-18T07:31:40.196137+0000","last_peered":"2024-09-18T07:31:40.196137+0000","last_clean":"2024-09-18T07:31:40.196137+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:31:40.196137+0000","last_undegraded":"2024-09-18T07:31:40.196137+0000","last_fullsized":"2024-09-18T07:31:40.196137+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.749034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":57,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725729+0000","last_change":"2024-09-18T07:30:58.313677+0000","last_active":"2024-09-18T07:31:39.725729+0000","last_peered":"2024-09-18T07:31:39.725729+0000","last_clean":"2024-09-18T07:31:39.725729+0000","last_became_active":"2024-09-18T07:30:58.313382+0000","last_became_peered":"2024-09-18T07:30:58.313382+0000","last_unstale":"2024-09-18T07:31:39.725729+0000","last_undegraded":"2024-09-18T07:31:39.725729+0000","last_fullsized":"2024-09-18T07:31:39.725729+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.250207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724904+0000","last_change":"2024-09-18T07:30:56.291117+0000","last_active":"2024-09-18T07:31:39.724904+0000","last_peered":"2024-09-18T07:31:39.724904+0000","last_clean":"2024-09-18T07:31:39.724904+0000","last_became_active":"2024-09-18T07:30:56.290836+0000","last_became_peered":"2024-09-18T07:30:56.290836+0000","last_unstale":"2024-09-18T07:31:39.724904+0000","last_undegraded":"2024-09-18T07:31:39.724904+0000","last_fullsized":"2024-09-18T07:31:39.724904+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.125102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.196078+0000","last_change":"2024-09-18T07:31:00.336459+0000","last_active":"2024-09-18T07:31:40.196078+0000","last_peered":"2024-09-18T07:31:40.196078+0000","last_clean":"2024-09-18T07:31:40.196078+0000","last_became_active":"2024-09-18T07:31:00.336172+0000","last_became_peered":"2024-09-18T07:31:00.336172+0000","last_unstale":"2024-09-18T07:31:40.196078+0000","last_undegraded":"2024-09-18T07:31:40.196078+0000","last_fullsized":"2024-09-18T07:31:40.196078+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:28.160928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193866+0000","last_change":"2024-09-18T07:31:02.491383+0000","last_active":"2024-09-18T07:31:40.193866+0000","last_peered":"2024-09-18T07:31:40.193866+0000","last_clean":"2024-09-18T07:31:40.193866+0000","last_became_active":"2024-09-18T07:31:02.490953+0000","last_became_peered":"2024-09-18T07:31:02.490953+0000","last_unstale":"2024-09-18T07:31:40.193866+0000","last_undegraded":"2024-09-18T07:31:40.193866+0000","last_fullsized":"2024-09-18T07:31:40.193866+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.829440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":66,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725540+0000","last_change":"2024-09-18T07:30:58.312636+0000","last_active":"2024-09-18T07:31:39.725540+0000","last_peered":"2024-09-18T07:31:39.725540+0000","last_clean":"2024-09-18T07:31:39.725540+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:31:39.725540+0000","last_undegraded":"2024-09-18T07:31:39.725540+0000","last_fullsized":"2024-09-18T07:31:39.725540+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:13.709878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194249+0000","last_change":"2024-09-18T07:30:56.284303+0000","last_active":"2024-09-18T07:31:40.194249+0000","last_peered":"2024-09-18T07:31:40.194249+0000","last_clean":"2024-09-18T07:31:40.194249+0000","last_became_active":"2024-09-18T07:30:56.284020+0000","last_became_peered":"2024-09-18T07:30:56.284020+0000","last_unstale":"2024-09-18T07:31:40.194249+0000","last_undegraded":"2024-09-18T07:31:40.194249+0000","last_fullsized":"2024-09-18T07:31:40.194249+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:15.982099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724628+0000","last_change":"2024-09-18T07:31:00.333155+0000","last_active":"2024-09-18T07:31:39.724628+0000","last_peered":"2024-09-18T07:31:39.724628+0000","last_clean":"2024-09-18T07:31:39.724628+0000","last_became_active":"2024-09-18T07:31:00.332836+0000","last_became_peered":"2024-09-18T07:31:00.332836+0000","last_unstale":"2024-09-18T07:31:39.724628+0000","last_undegraded":"2024-09-18T07:31:39.724628+0000","last_fullsized":"2024-09-18T07:31:39.724628+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.946427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724573+0000","last_change":"2024-09-18T07:31:02.347665+0000","last_active":"2024-09-18T07:31:39.724573+0000","last_peered":"2024-09-18T07:31:39.724573+0000","last_clean":"2024-09-18T07:31:39.724573+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:31:39.724573+0000","last_undegraded":"2024-09-18T07:31:39.724573+0000","last_fullsized":"2024-09-18T07:31:39.724573+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.617366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725236+0000","last_change":"2024-09-18T07:30:58.317008+0000","last_active":"2024-09-18T07:31:39.725236+0000","last_peered":"2024-09-18T07:31:39.725236+0000","last_clean":"2024-09-18T07:31:39.725236+0000","last_became_active":"2024-09-18T07:30:58.316423+0000","last_became_peered":"2024-09-18T07:30:58.316423+0000","last_unstale":"2024-09-18T07:31:39.725236+0000","last_undegraded":"2024-09-18T07:31:39.725236+0000","last_fullsized":"2024-09-18T07:31:39.725236+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.580907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194222+0000","last_change":"2024-09-18T07:30:56.287284+0000","last_active":"2024-09-18T07:31:40.194222+0000","last_peered":"2024-09-18T07:31:40.194222+0000","last_clean":"2024-09-18T07:31:40.194222+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:31:40.194222+0000","last_undegraded":"2024-09-18T07:31:40.194222+0000","last_fullsized":"2024-09-18T07:31:40.194222+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:00.720049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723893+0000","last_change":"2024-09-18T07:31:00.333376+0000","last_active":"2024-09-18T07:31:39.723893+0000","last_peered":"2024-09-18T07:31:39.723893+0000","last_clean":"2024-09-18T07:31:39.723893+0000","last_became_active":"2024-09-18T07:31:00.333085+0000","last_became_peered":"2024-09-18T07:31:00.333085+0000","last_unstale":"2024-09-18T07:31:39.723893+0000","last_undegraded":"2024-09-18T07:31:39.723893+0000","last_fullsized":"2024-09-18T07:31:39.723893+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:22.590817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725525+0000","last_change":"2024-09-18T07:31:02.347721+0000","last_active":"2024-09-18T07:31:39.725525+0000","last_peered":"2024-09-18T07:31:39.725525+0000","last_clean":"2024-09-18T07:31:39.725525+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:31:39.725525+0000","last_undegraded":"2024-09-18T07:31:39.725525+0000","last_fullsized":"2024-09-18T07:31:39.725525+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.723859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193700+0000","last_change":"2024-09-18T07:30:58.314424+0000","last_active":"2024-09-18T07:31:40.193700+0000","last_peered":"2024-09-18T07:31:40.193700+0000","last_clean":"2024-09-18T07:31:40.193700+0000","last_became_active":"2024-09-18T07:30:58.314127+0000","last_became_peered":"2024-09-18T07:30:58.314127+0000","last_unstale":"2024-09-18T07:31:40.193700+0000","last_undegraded":"2024-09-18T07:31:40.193700+0000","last_fullsized":"2024-09-18T07:31:40.193700+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:13.636020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195735+0000","last_change":"2024-09-18T07:30:56.289177+0000","last_active":"2024-09-18T07:31:40.195735+0000","last_peered":"2024-09-18T07:31:40.195735+0000","last_clean":"2024-09-18T07:31:40.195735+0000","last_became_active":"2024-09-18T07:30:56.288868+0000","last_became_peered":"2024-09-18T07:30:56.288868+0000","last_unstale":"2024-09-18T07:31:40.195735+0000","last_undegraded":"2024-09-18T07:31:40.195735+0000","last_fullsized":"2024-09-18T07:31:40.195735+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.629409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723663+0000","last_change":"2024-09-18T07:31:00.336020+0000","last_active":"2024-09-18T07:31:39.723663+0000","last_peered":"2024-09-18T07:31:39.723663+0000","last_clean":"2024-09-18T07:31:39.723663+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:31:39.723663+0000","last_undegraded":"2024-09-18T07:31:39.723663+0000","last_fullsized":"2024-09-18T07:31:39.723663+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:28.112892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193110+0000","last_change":"2024-09-18T07:31:02.349757+0000","last_active":"2024-09-18T07:31:40.193110+0000","last_peered":"2024-09-18T07:31:40.193110+0000","last_clean":"2024-09-18T07:31:40.193110+0000","last_became_active":"2024-09-18T07:31:02.349454+0000","last_became_peered":"2024-09-18T07:31:02.349454+0000","last_unstale":"2024-09-18T07:31:40.193110+0000","last_undegraded":"2024-09-18T07:31:40.193110+0000","last_fullsized":"2024-09-18T07:31:40.193110+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.874882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":69,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194032+0000","last_change":"2024-09-18T07:30:58.318421+0000","last_active":"2024-09-18T07:31:40.194032+0000","last_peered":"2024-09-18T07:31:40.194032+0000","last_clean":"2024-09-18T07:31:40.194032+0000","last_became_active":"2024-09-18T07:30:58.318063+0000","last_became_peered":"2024-09-18T07:30:58.318063+0000","last_unstale":"2024-09-18T07:31:40.194032+0000","last_undegraded":"2024-09-18T07:31:40.194032+0000","last_fullsized":"2024-09-18T07:31:40.194032+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:59.591866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195532+0000","last_change":"2024-09-18T07:30:56.293678+0000","last_active":"2024-09-18T07:31:40.195532+0000","last_peered":"2024-09-18T07:31:40.195532+0000","last_clean":"2024-09-18T07:31:40.195532+0000","last_became_active":"2024-09-18T07:30:56.293371+0000","last_became_peered":"2024-09-18T07:30:56.293371+0000","last_unstale":"2024-09-18T07:31:40.195532+0000","last_undegraded":"2024-09-18T07:31:40.195532+0000","last_fullsized":"2024-09-18T07:31:40.195532+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.829476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724381+0000","last_change":"2024-09-18T07:31:00.332322+0000","last_active":"2024-09-18T07:31:39.724381+0000","last_peered":"2024-09-18T07:31:39.724381+0000","last_clean":"2024-09-18T07:31:39.724381+0000","last_became_active":"2024-09-18T07:31:00.331914+0000","last_became_peered":"2024-09-18T07:31:00.331914+0000","last_unstale":"2024-09-18T07:31:39.724381+0000","last_undegraded":"2024-09-18T07:31:39.724381+0000","last_fullsized":"2024-09-18T07:31:39.724381+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.543866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194651+0000","last_change":"2024-09-18T07:31:02.351941+0000","last_active":"2024-09-18T07:31:40.194651+0000","last_peered":"2024-09-18T07:31:40.194651+0000","last_clean":"2024-09-18T07:31:40.194651+0000","last_became_active":"2024-09-18T07:31:02.351678+0000","last_became_peered":"2024-09-18T07:31:02.351678+0000","last_unstale":"2024-09-18T07:31:40.194651+0000","last_undegraded":"2024-09-18T07:31:40.194651+0000","last_fullsized":"2024-09-18T07:31:40.194651+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.816291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":61,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724785+0000","last_change":"2024-09-18T07:30:58.320305+0000","last_active":"2024-09-18T07:31:39.724785+0000","last_peered":"2024-09-18T07:31:39.724785+0000","last_clean":"2024-09-18T07:31:39.724785+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:31:39.724785+0000","last_undegraded":"2024-09-18T07:31:39.724785+0000","last_fullsized":"2024-09-18T07:31:39.724785+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.490659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195144+0000","last_change":"2024-09-18T07:30:56.290367+0000","last_active":"2024-09-18T07:31:40.195144+0000","last_peered":"2024-09-18T07:31:40.195144+0000","last_clean":"2024-09-18T07:31:40.195144+0000","last_became_active":"2024-09-18T07:30:56.289855+0000","last_became_peered":"2024-09-18T07:30:56.289855+0000","last_unstale":"2024-09-18T07:31:40.195144+0000","last_undegraded":"2024-09-18T07:31:40.195144+0000","last_fullsized":"2024-09-18T07:31:40.195144+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.319326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723264+0000","last_change":"2024-09-18T07:31:00.329175+0000","last_active":"2024-09-18T07:31:39.723264+0000","last_peered":"2024-09-18T07:31:39.723264+0000","last_clean":"2024-09-18T07:31:39.723264+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:31:39.723264+0000","last_undegraded":"2024-09-18T07:31:39.723264+0000","last_fullsized":"2024-09-18T07:31:39.723264+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.309875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193965+0000","last_change":"2024-09-18T07:31:02.353267+0000","last_active":"2024-09-18T07:31:40.193965+0000","last_peered":"2024-09-18T07:31:40.193965+0000","last_clean":"2024-09-18T07:31:40.193965+0000","last_became_active":"2024-09-18T07:31:02.352922+0000","last_became_peered":"2024-09-18T07:31:02.352922+0000","last_unstale":"2024-09-18T07:31:40.193965+0000","last_undegraded":"2024-09-18T07:31:40.193965+0000","last_fullsized":"2024-09-18T07:31:40.193965+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:26.271964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":63,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193728+0000","last_change":"2024-09-18T07:30:58.319585+0000","last_active":"2024-09-18T07:31:40.193728+0000","last_peered":"2024-09-18T07:31:40.193728+0000","last_clean":"2024-09-18T07:31:40.193728+0000","last_became_active":"2024-09-18T07:30:58.319300+0000","last_became_peered":"2024-09-18T07:30:58.319300+0000","last_unstale":"2024-09-18T07:31:40.193728+0000","last_undegraded":"2024-09-18T07:31:40.193728+0000","last_fullsized":"2024-09-18T07:31:40.193728+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.143713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195339+0000","last_change":"2024-09-18T07:30:56.292832+0000","last_active":"2024-09-18T07:31:40.195339+0000","last_peered":"2024-09-18T07:31:40.195339+0000","last_clean":"2024-09-18T07:31:40.195339+0000","last_became_active":"2024-09-18T07:30:56.292531+0000","last_became_peered":"2024-09-18T07:30:56.292531+0000","last_unstale":"2024-09-18T07:31:40.195339+0000","last_undegraded":"2024-09-18T07:31:40.195339+0000","last_fullsized":"2024-09-18T07:31:40.195339+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.359238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194103+0000","last_change":"2024-09-18T07:31:00.337915+0000","last_active":"2024-09-18T07:31:40.194103+0000","last_peered":"2024-09-18T07:31:40.194103+0000","last_clean":"2024-09-18T07:31:40.194103+0000","last_became_active":"2024-09-18T07:31:00.337483+0000","last_became_peered":"2024-09-18T07:31:00.337483+0000","last_unstale":"2024-09-18T07:31:40.194103+0000","last_undegraded":"2024-09-18T07:31:40.194103+0000","last_fullsized":"2024-09-18T07:31:40.194103+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:20.556343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193984+0000","last_change":"2024-09-18T07:31:02.489914+0000","last_active":"2024-09-18T07:31:40.193984+0000","last_peered":"2024-09-18T07:31:40.193984+0000","last_clean":"2024-09-18T07:31:40.193984+0000","last_became_active":"2024-09-18T07:31:02.489619+0000","last_became_peered":"2024-09-18T07:31:02.489619+0000","last_unstale":"2024-09-18T07:31:40.193984+0000","last_undegraded":"2024-09-18T07:31:40.193984+0000","last_fullsized":"2024-09-18T07:31:40.193984+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:17.709609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193918+0000","last_change":"2024-09-18T07:30:56.282629+0000","last_active":"2024-09-18T07:31:40.193918+0000","last_peered":"2024-09-18T07:31:40.193918+0000","last_clean":"2024-09-18T07:31:40.193918+0000","last_became_active":"2024-09-18T07:30:56.282357+0000","last_became_peered":"2024-09-18T07:30:56.282357+0000","last_unstale":"2024-09-18T07:31:40.193918+0000","last_undegraded":"2024-09-18T07:31:40.193918+0000","last_fullsized":"2024-09-18T07:31:40.193918+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.549682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194391+0000","last_change":"2024-09-18T07:30:58.320781+0000","last_active":"2024-09-18T07:31:40.194391+0000","last_peered":"2024-09-18T07:31:40.194391+0000","last_clean":"2024-09-18T07:31:40.194391+0000","last_became_active":"2024-09-18T07:30:58.320563+0000","last_became_peered":"2024-09-18T07:30:58.320563+0000","last_unstale":"2024-09-18T07:31:40.194391+0000","last_undegraded":"2024-09-18T07:31:40.194391+0000","last_fullsized":"2024-09-18T07:31:40.194391+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.842501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195911+0000","last_change":"2024-09-18T07:31:00.334155+0000","last_active":"2024-09-18T07:31:40.195911+0000","last_peered":"2024-09-18T07:31:40.195911+0000","last_clean":"2024-09-18T07:31:40.195911+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:31:40.195911+0000","last_undegraded":"2024-09-18T07:31:40.195911+0000","last_fullsized":"2024-09-18T07:31:40.195911+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.652219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194724+0000","last_change":"2024-09-18T07:31:02.492999+0000","last_active":"2024-09-18T07:31:40.194724+0000","last_peered":"2024-09-18T07:31:40.194724+0000","last_clean":"2024-09-18T07:31:40.194724+0000","last_became_active":"2024-09-18T07:31:02.492489+0000","last_became_peered":"2024-09-18T07:31:02.492489+0000","last_unstale":"2024-09-18T07:31:40.194724+0000","last_undegraded":"2024-09-18T07:31:40.194724+0000","last_fullsized":"2024-09-18T07:31:40.194724+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.972057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193917+0000","last_change":"2024-09-18T07:30:58.318263+0000","last_active":"2024-09-18T07:31:40.193917+0000","last_peered":"2024-09-18T07:31:40.193917+0000","last_clean":"2024-09-18T07:31:40.193917+0000","last_became_active":"2024-09-18T07:30:58.317732+0000","last_became_peered":"2024-09-18T07:30:58.317732+0000","last_unstale":"2024-09-18T07:31:40.193917+0000","last_undegraded":"2024-09-18T07:31:40.193917+0000","last_fullsized":"2024-09-18T07:31:40.193917+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.024721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195239+0000","last_change":"2024-09-18T07:30:56.292272+0000","last_active":"2024-09-18T07:31:40.195239+0000","last_peered":"2024-09-18T07:31:40.195239+0000","last_clean":"2024-09-18T07:31:40.195239+0000","last_became_active":"2024-09-18T07:30:56.290788+0000","last_became_peered":"2024-09-18T07:30:56.290788+0000","last_unstale":"2024-09-18T07:31:40.195239+0000","last_undegraded":"2024-09-18T07:31:40.195239+0000","last_fullsized":"2024-09-18T07:31:40.195239+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.447855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194323+0000","last_change":"2024-09-18T07:31:00.334848+0000","last_active":"2024-09-18T07:31:40.194323+0000","last_peered":"2024-09-18T07:31:40.194323+0000","last_clean":"2024-09-18T07:31:40.194323+0000","last_became_active":"2024-09-18T07:31:00.334478+0000","last_became_peered":"2024-09-18T07:31:00.334478+0000","last_unstale":"2024-09-18T07:31:40.194323+0000","last_undegraded":"2024-09-18T07:31:40.194323+0000","last_fullsized":"2024-09-18T07:31:40.194323+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.087953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724734+0000","last_change":"2024-09-18T07:31:02.344766+0000","last_active":"2024-09-18T07:31:39.724734+0000","last_peered":"2024-09-18T07:31:39.724734+0000","last_clean":"2024-09-18T07:31:39.724734+0000","last_became_active":"2024-09-18T07:31:02.344011+0000","last_became_peered":"2024-09-18T07:31:02.344011+0000","last_unstale":"2024-09-18T07:31:39.724734+0000","last_undegraded":"2024-09-18T07:31:39.724734+0000","last_fullsized":"2024-09-18T07:31:39.724734+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.862393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195887+0000","last_change":"2024-09-18T07:30:58.318363+0000","last_active":"2024-09-18T07:31:40.195887+0000","last_peered":"2024-09-18T07:31:40.195887+0000","last_clean":"2024-09-18T07:31:40.195887+0000","last_became_active":"2024-09-18T07:30:58.318057+0000","last_became_peered":"2024-09-18T07:30:58.318057+0000","last_unstale":"2024-09-18T07:31:40.195887+0000","last_undegraded":"2024-09-18T07:31:40.195887+0000","last_fullsized":"2024-09-18T07:31:40.195887+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.599726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193524+0000","last_change":"2024-09-18T07:30:56.287385+0000","last_active":"2024-09-18T07:31:40.193524+0000","last_peered":"2024-09-18T07:31:40.193524+0000","last_clean":"2024-09-18T07:31:40.193524+0000","last_became_active":"2024-09-18T07:30:56.287068+0000","last_became_peered":"2024-09-18T07:30:56.287068+0000","last_unstale":"2024-09-18T07:31:40.193524+0000","last_undegraded":"2024-09-18T07:31:40.193524+0000","last_fullsized":"2024-09-18T07:31:40.193524+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.184849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194082+0000","last_change":"2024-09-18T07:31:00.337716+0000","last_active":"2024-09-18T07:31:40.194082+0000","last_peered":"2024-09-18T07:31:40.194082+0000","last_clean":"2024-09-18T07:31:40.194082+0000","last_became_active":"2024-09-18T07:31:00.337444+0000","last_became_peered":"2024-09-18T07:31:00.337444+0000","last_unstale":"2024-09-18T07:31:40.194082+0000","last_undegraded":"2024-09-18T07:31:40.194082+0000","last_fullsized":"2024-09-18T07:31:40.194082+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.512126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725668+0000","last_change":"2024-09-18T07:31:02.343206+0000","last_active":"2024-09-18T07:31:39.725668+0000","last_peered":"2024-09-18T07:31:39.725668+0000","last_clean":"2024-09-18T07:31:39.725668+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:31:39.725668+0000","last_undegraded":"2024-09-18T07:31:39.725668+0000","last_fullsized":"2024-09-18T07:31:39.725668+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:23.329902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194303+0000","last_change":"2024-09-18T07:30:58.320351+0000","last_active":"2024-09-18T07:31:40.194303+0000","last_peered":"2024-09-18T07:31:40.194303+0000","last_clean":"2024-09-18T07:31:40.194303+0000","last_became_active":"2024-09-18T07:30:58.320093+0000","last_became_peered":"2024-09-18T07:30:58.320093+0000","last_unstale":"2024-09-18T07:31:40.194303+0000","last_undegraded":"2024-09-18T07:31:40.194303+0000","last_fullsized":"2024-09-18T07:31:40.194303+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.687274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195252+0000","last_change":"2024-09-18T07:30:56.291807+0000","last_active":"2024-09-18T07:31:40.195252+0000","last_peered":"2024-09-18T07:31:40.195252+0000","last_clean":"2024-09-18T07:31:40.195252+0000","last_became_active":"2024-09-18T07:30:56.291238+0000","last_became_peered":"2024-09-18T07:30:56.291238+0000","last_unstale":"2024-09-18T07:31:40.195252+0000","last_undegraded":"2024-09-18T07:31:40.195252+0000","last_fullsized":"2024-09-18T07:31:40.195252+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:17.785800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195794+0000","last_change":"2024-09-18T07:31:00.332729+0000","last_active":"2024-09-18T07:31:40.195794+0000","last_peered":"2024-09-18T07:31:40.195794+0000","last_clean":"2024-09-18T07:31:40.195794+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:31:40.195794+0000","last_undegraded":"2024-09-18T07:31:40.195794+0000","last_fullsized":"2024-09-18T07:31:40.195794+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:08.863683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193268+0000","last_change":"2024-09-18T07:31:02.348726+0000","last_active":"2024-09-18T07:31:40.193268+0000","last_peered":"2024-09-18T07:31:40.193268+0000","last_clean":"2024-09-18T07:31:40.193268+0000","last_became_active":"2024-09-18T07:31:02.348423+0000","last_became_peered":"2024-09-18T07:31:02.348423+0000","last_unstale":"2024-09-18T07:31:40.193268+0000","last_undegraded":"2024-09-18T07:31:40.193268+0000","last_fullsized":"2024-09-18T07:31:40.193268+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.760836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":57,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725116+0000","last_change":"2024-09-18T07:30:58.314730+0000","last_active":"2024-09-18T07:31:39.725116+0000","last_peered":"2024-09-18T07:31:39.725116+0000","last_clean":"2024-09-18T07:31:39.725116+0000","last_became_active":"2024-09-18T07:30:58.314428+0000","last_became_peered":"2024-09-18T07:30:58.314428+0000","last_unstale":"2024-09-18T07:31:39.725116+0000","last_undegraded":"2024-09-18T07:31:39.725116+0000","last_fullsized":"2024-09-18T07:31:39.725116+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.641591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725433+0000","last_change":"2024-09-18T07:30:56.291115+0000","last_active":"2024-09-18T07:31:39.725433+0000","last_peered":"2024-09-18T07:31:39.725433+0000","last_clean":"2024-09-18T07:31:39.725433+0000","last_became_active":"2024-09-18T07:30:56.290816+0000","last_became_peered":"2024-09-18T07:30:56.290816+0000","last_unstale":"2024-09-18T07:31:39.725433+0000","last_undegraded":"2024-09-18T07:31:39.725433+0000","last_fullsized":"2024-09-18T07:31:39.725433+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.554722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723958+0000","last_change":"2024-09-18T07:31:00.337826+0000","last_active":"2024-09-18T07:31:39.723958+0000","last_peered":"2024-09-18T07:31:39.723958+0000","last_clean":"2024-09-18T07:31:39.723958+0000","last_became_active":"2024-09-18T07:31:00.334158+0000","last_became_peered":"2024-09-18T07:31:00.334158+0000","last_unstale":"2024-09-18T07:31:39.723958+0000","last_undegraded":"2024-09-18T07:31:39.723958+0000","last_fullsized":"2024-09-18T07:31:39.723958+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:28.224028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193699+0000","last_change":"2024-09-18T07:31:02.351247+0000","last_active":"2024-09-18T07:31:40.193699+0000","last_peered":"2024-09-18T07:31:40.193699+0000","last_clean":"2024-09-18T07:31:40.193699+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:31:40.193699+0000","last_undegraded":"2024-09-18T07:31:40.193699+0000","last_fullsized":"2024-09-18T07:31:40.193699+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:25.935005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724310+0000","last_change":"2024-09-18T07:30:58.307904+0000","last_active":"2024-09-18T07:31:39.724310+0000","last_peered":"2024-09-18T07:31:39.724310+0000","last_clean":"2024-09-18T07:31:39.724310+0000","last_became_active":"2024-09-18T07:30:58.307448+0000","last_became_peered":"2024-09-18T07:30:58.307448+0000","last_unstale":"2024-09-18T07:31:39.724310+0000","last_undegraded":"2024-09-18T07:31:39.724310+0000","last_fullsized":"2024-09-18T07:31:39.724310+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:01.900229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195450+0000","last_change":"2024-09-18T07:30:56.293467+0000","last_active":"2024-09-18T07:31:40.195450+0000","last_peered":"2024-09-18T07:31:40.195450+0000","last_clean":"2024-09-18T07:31:40.195450+0000","last_became_active":"2024-09-18T07:30:56.292980+0000","last_became_peered":"2024-09-18T07:30:56.292980+0000","last_unstale":"2024-09-18T07:31:40.195450+0000","last_undegraded":"2024-09-18T07:31:40.195450+0000","last_fullsized":"2024-09-18T07:31:40.195450+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.161233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193884+0000","last_change":"2024-09-18T07:31:00.333689+0000","last_active":"2024-09-18T07:31:40.193884+0000","last_peered":"2024-09-18T07:31:40.193884+0000","last_clean":"2024-09-18T07:31:40.193884+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:31:40.193884+0000","last_undegraded":"2024-09-18T07:31:40.193884+0000","last_fullsized":"2024-09-18T07:31:40.193884+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.075146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193043+0000","last_change":"2024-09-18T07:31:02.352805+0000","last_active":"2024-09-18T07:31:40.193043+0000","last_peered":"2024-09-18T07:31:40.193043+0000","last_clean":"2024-09-18T07:31:40.193043+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:31:40.193043+0000","last_undegraded":"2024-09-18T07:31:40.193043+0000","last_fullsized":"2024-09-18T07:31:40.193043+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.115071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":57,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194091+0000","last_change":"2024-09-18T07:30:58.320419+0000","last_active":"2024-09-18T07:31:40.194091+0000","last_peered":"2024-09-18T07:31:40.194091+0000","last_clean":"2024-09-18T07:31:40.194091+0000","last_became_active":"2024-09-18T07:30:58.320108+0000","last_became_peered":"2024-09-18T07:30:58.320108+0000","last_unstale":"2024-09-18T07:31:40.194091+0000","last_undegraded":"2024-09-18T07:31:40.194091+0000","last_fullsized":"2024-09-18T07:31:40.194091+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.449011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.196053+0000","last_change":"2024-09-18T07:30:56.289671+0000","last_active":"2024-09-18T07:31:40.196053+0000","last_peered":"2024-09-18T07:31:40.196053+0000","last_clean":"2024-09-18T07:31:40.196053+0000","last_became_active":"2024-09-18T07:30:56.289328+0000","last_became_peered":"2024-09-18T07:30:56.289328+0000","last_unstale":"2024-09-18T07:31:40.196053+0000","last_undegraded":"2024-09-18T07:31:40.196053+0000","last_fullsized":"2024-09-18T07:31:40.196053+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:58.531067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193590+0000","last_change":"2024-09-18T07:31:00.336053+0000","last_active":"2024-09-18T07:31:40.193590+0000","last_peered":"2024-09-18T07:31:40.193590+0000","last_clean":"2024-09-18T07:31:40.193590+0000","last_became_active":"2024-09-18T07:31:00.335749+0000","last_became_peered":"2024-09-18T07:31:00.335749+0000","last_unstale":"2024-09-18T07:31:40.193590+0000","last_undegraded":"2024-09-18T07:31:40.193590+0000","last_fullsized":"2024-09-18T07:31:40.193590+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:26.183381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194246+0000","last_change":"2024-09-18T07:31:02.491487+0000","last_active":"2024-09-18T07:31:40.194246+0000","last_peered":"2024-09-18T07:31:40.194246+0000","last_clean":"2024-09-18T07:31:40.194246+0000","last_became_active":"2024-09-18T07:31:02.490994+0000","last_became_peered":"2024-09-18T07:31:02.490994+0000","last_unstale":"2024-09-18T07:31:40.194246+0000","last_undegraded":"2024-09-18T07:31:40.194246+0000","last_fullsized":"2024-09-18T07:31:40.194246+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.401313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193399+0000","last_change":"2024-09-18T07:30:58.316173+0000","last_active":"2024-09-18T07:31:40.193399+0000","last_peered":"2024-09-18T07:31:40.193399+0000","last_clean":"2024-09-18T07:31:40.193399+0000","last_became_active":"2024-09-18T07:30:58.315435+0000","last_became_peered":"2024-09-18T07:30:58.315435+0000","last_unstale":"2024-09-18T07:31:40.193399+0000","last_undegraded":"2024-09-18T07:31:40.193399+0000","last_fullsized":"2024-09-18T07:31:40.193399+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:05.325393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193617+0000","last_change":"2024-09-18T07:30:56.283643+0000","last_active":"2024-09-18T07:31:40.193617+0000","last_peered":"2024-09-18T07:31:40.193617+0000","last_clean":"2024-09-18T07:31:40.193617+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:31:40.193617+0000","last_undegraded":"2024-09-18T07:31:40.193617+0000","last_fullsized":"2024-09-18T07:31:40.193617+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:17.971275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193929+0000","last_change":"2024-09-18T07:31:00.333126+0000","last_active":"2024-09-18T07:31:40.193929+0000","last_peered":"2024-09-18T07:31:40.193929+0000","last_clean":"2024-09-18T07:31:40.193929+0000","last_became_active":"2024-09-18T07:31:00.332855+0000","last_became_peered":"2024-09-18T07:31:00.332855+0000","last_unstale":"2024-09-18T07:31:40.193929+0000","last_undegraded":"2024-09-18T07:31:40.193929+0000","last_fullsized":"2024-09-18T07:31:40.193929+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:29.188004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194334+0000","last_change":"2024-09-18T07:31:02.350978+0000","last_active":"2024-09-18T07:31:40.194334+0000","last_peered":"2024-09-18T07:31:40.194334+0000","last_clean":"2024-09-18T07:31:40.194334+0000","last_became_active":"2024-09-18T07:31:02.350692+0000","last_became_peered":"2024-09-18T07:31:02.350692+0000","last_unstale":"2024-09-18T07:31:40.194334+0000","last_undegraded":"2024-09-18T07:31:40.194334+0000","last_fullsized":"2024-09-18T07:31:40.194334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.916468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724958+0000","last_change":"2024-09-18T07:31:02.347827+0000","last_active":"2024-09-18T07:31:39.724958+0000","last_peered":"2024-09-18T07:31:39.724958+0000","last_clean":"2024-09-18T07:31:39.724958+0000","last_became_active":"2024-09-18T07:31:02.347537+0000","last_became_peered":"2024-09-18T07:31:02.347537+0000","last_unstale":"2024-09-18T07:31:39.724958+0000","last_undegraded":"2024-09-18T07:31:39.724958+0000","last_fullsized":"2024-09-18T07:31:39.724958+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:28.722445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193580+0000","last_change":"2024-09-18T07:30:56.292657+0000","last_active":"2024-09-18T07:31:40.193580+0000","last_peered":"2024-09-18T07:31:40.193580+0000","last_clean":"2024-09-18T07:31:40.193580+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:31:40.193580+0000","last_undegraded":"2024-09-18T07:31:40.193580+0000","last_fullsized":"2024-09-18T07:31:40.193580+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:13.946464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":57,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194832+0000","last_change":"2024-09-18T07:30:58.314191+0000","last_active":"2024-09-18T07:31:40.194832+0000","last_peered":"2024-09-18T07:31:40.194832+0000","last_clean":"2024-09-18T07:31:40.194832+0000","last_became_active":"2024-09-18T07:30:58.313556+0000","last_became_peered":"2024-09-18T07:30:58.313556+0000","last_unstale":"2024-09-18T07:31:40.194832+0000","last_undegraded":"2024-09-18T07:31:40.194832+0000","last_fullsized":"2024-09-18T07:31:40.194832+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.554527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193642+0000","last_change":"2024-09-18T07:31:00.331995+0000","last_active":"2024-09-18T07:31:40.193642+0000","last_peered":"2024-09-18T07:31:40.193642+0000","last_clean":"2024-09-18T07:31:40.193642+0000","last_became_active":"2024-09-18T07:31:00.331564+0000","last_became_peered":"2024-09-18T07:31:00.331564+0000","last_unstale":"2024-09-18T07:31:40.193642+0000","last_undegraded":"2024-09-18T07:31:40.193642+0000","last_fullsized":"2024-09-18T07:31:40.193642+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.738488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201293,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28360,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716792,"statfs":{"total":95995035648,"available":95965995008,"internally_reserved":0,"allocated":1363968,"data_stored":895800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462931,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32500,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712652,"statfs":{"total":95995035648,"available":95961755648,"internally_reserved":0,"allocated":1343488,"data_stored":894715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724568,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31924,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713228,"statfs":{"total":95995035648,"available":95962345472,"internally_reserved":0,"allocated":749568,"data_stored":304693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953503,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27788,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717364,"statfs":{"total":95995035648,"available":95966580736,"internally_reserved":0,"allocated":778240,"data_stored":305375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27804,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717348,"statfs":{"total":95995035648,"available":95966564352,"internally_reserved":0,"allocated":794624,"data_stored":306667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27752,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":741376,"data_stored":304482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":815104,"data_stored":307112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1372160,"data_stored":896925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:31:45.429 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:31:45.685 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:46.128 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[29408]: Adjusting osd_memory_target on smithi137 to 3760M 2024-09-18T07:31:46.129 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[29408]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-18T07:31:46.129 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[29408]: Updating smithi203:/etc/ceph/ceph.conf 2024-09-18T07:31:46.129 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[29408]: from='client.34649 v1:172.21.15.137:0/987293575' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:31:46.129 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[29408]: Updating smithi137:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:31:46.129 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[29408]: Updating smithi203:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:31:46.129 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[29408]: osdmap e77: 8 total, 8 up, 8 in 2024-09-18T07:31:46.129 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[29408]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:31:46.129 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[29408]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:31:46.129 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:46.129 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:46.130 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:46.130 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:46.130 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:46.130 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[39175]: Adjusting osd_memory_target on smithi137 to 3760M 2024-09-18T07:31:46.130 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[39175]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-18T07:31:46.130 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[39175]: Updating smithi203:/etc/ceph/ceph.conf 2024-09-18T07:31:46.130 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[39175]: from='client.34649 v1:172.21.15.137:0/987293575' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:31:46.130 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[39175]: Updating smithi137:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:31:46.130 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[39175]: Updating smithi203:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:31:46.130 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[39175]: osdmap e77: 8 total, 8 up, 8 in 2024-09-18T07:31:46.130 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[39175]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:31:46.131 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[39175]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:31:46.131 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:46.131 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:46.131 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:46.131 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:46.131 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:45 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:45 smithi203 ceph-mon[32237]: Adjusting osd_memory_target on smithi137 to 3760M 2024-09-18T07:31:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:45 smithi203 ceph-mon[32237]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-18T07:31:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:45 smithi203 ceph-mon[32237]: Updating smithi203:/etc/ceph/ceph.conf 2024-09-18T07:31:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:45 smithi203 ceph-mon[32237]: from='client.34649 v1:172.21.15.137:0/987293575' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:31:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:45 smithi203 ceph-mon[32237]: Updating smithi137:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:31:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:45 smithi203 ceph-mon[32237]: Updating smithi203:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-18T07:31:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:45 smithi203 ceph-mon[32237]: osdmap e77: 8 total, 8 up, 8 in 2024-09-18T07:31:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:45 smithi203 ceph-mon[32237]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:31:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:45 smithi203 ceph-mon[32237]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-09-18T07:31:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:45 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:45 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:46.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:45 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:46.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:45 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:46.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:45 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:46.414 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:31:46.416 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:31:47.074 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:46 smithi137 ceph-mon[29408]: Updating smithi137:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T07:31:47.075 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:46 smithi137 ceph-mon[29408]: Updating smithi203:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T07:31:47.075 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:46 smithi137 ceph-mon[29408]: Deploying daemon alertmanager.a on smithi137 2024-09-18T07:31:47.075 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:46 smithi137 ceph-mon[29408]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-18T07:31:47.075 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:46 smithi137 ceph-mon[39175]: Updating smithi137:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T07:31:47.075 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:46 smithi137 ceph-mon[39175]: Updating smithi203:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T07:31:47.075 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:46 smithi137 ceph-mon[39175]: Deploying daemon alertmanager.a on smithi137 2024-09-18T07:31:47.075 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:46 smithi137 ceph-mon[39175]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-18T07:31:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:46 smithi203 ceph-mon[32237]: Updating smithi137:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T07:31:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:46 smithi203 ceph-mon[32237]: Updating smithi203:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-18T07:31:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:46 smithi203 ceph-mon[32237]: Deploying daemon alertmanager.a on smithi137 2024-09-18T07:31:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:46 smithi203 ceph-mon[32237]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-18T07:31:47.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:46 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:31:47.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:31:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:31:47.850 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-18T07:31:45.408239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":888,"num_read_kb":745,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236480,"kb_used_data":7772,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724736,"statfs":{"total":767960285184,"available":767718129664,"internally_reserved":0,"allocated":7958528,"data_stored":4215769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001857"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193595+0000","last_change":"2024-09-18T07:30:56.284087+0000","last_active":"2024-09-18T07:31:40.193595+0000","last_peered":"2024-09-18T07:31:40.193595+0000","last_clean":"2024-09-18T07:31:40.193595+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:31:40.193595+0000","last_undegraded":"2024-09-18T07:31:40.193595+0000","last_fullsized":"2024-09-18T07:31:40.193595+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.049077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724596+0000","last_change":"2024-09-18T07:30:58.309315+0000","last_active":"2024-09-18T07:31:39.724596+0000","last_peered":"2024-09-18T07:31:39.724596+0000","last_clean":"2024-09-18T07:31:39.724596+0000","last_became_active":"2024-09-18T07:30:58.308981+0000","last_became_peered":"2024-09-18T07:30:58.308981+0000","last_unstale":"2024-09-18T07:31:39.724596+0000","last_undegraded":"2024-09-18T07:31:39.724596+0000","last_fullsized":"2024-09-18T07:31:39.724596+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:08.376023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723770+0000","last_change":"2024-09-18T07:31:00.336284+0000","last_active":"2024-09-18T07:31:39.723770+0000","last_peered":"2024-09-18T07:31:39.723770+0000","last_clean":"2024-09-18T07:31:39.723770+0000","last_became_active":"2024-09-18T07:31:00.335958+0000","last_became_peered":"2024-09-18T07:31:00.335958+0000","last_unstale":"2024-09-18T07:31:39.723770+0000","last_undegraded":"2024-09-18T07:31:39.723770+0000","last_fullsized":"2024-09-18T07:31:39.723770+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:08.645781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724775+0000","last_change":"2024-09-18T07:31:02.342510+0000","last_active":"2024-09-18T07:31:39.724775+0000","last_peered":"2024-09-18T07:31:39.724775+0000","last_clean":"2024-09-18T07:31:39.724775+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:31:39.724775+0000","last_undegraded":"2024-09-18T07:31:39.724775+0000","last_fullsized":"2024-09-18T07:31:39.724775+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.002947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193793+0000","last_change":"2024-09-18T07:31:02.492941+0000","last_active":"2024-09-18T07:31:40.193793+0000","last_peered":"2024-09-18T07:31:40.193793+0000","last_clean":"2024-09-18T07:31:40.193793+0000","last_became_active":"2024-09-18T07:31:02.492405+0000","last_became_peered":"2024-09-18T07:31:02.492405+0000","last_unstale":"2024-09-18T07:31:40.193793+0000","last_undegraded":"2024-09-18T07:31:40.193793+0000","last_fullsized":"2024-09-18T07:31:40.193793+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:22.665514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194149+0000","last_change":"2024-09-18T07:30:56.286313+0000","last_active":"2024-09-18T07:31:40.194149+0000","last_peered":"2024-09-18T07:31:40.194149+0000","last_clean":"2024-09-18T07:31:40.194149+0000","last_became_active":"2024-09-18T07:30:56.286040+0000","last_became_peered":"2024-09-18T07:30:56.286040+0000","last_unstale":"2024-09-18T07:31:40.194149+0000","last_undegraded":"2024-09-18T07:31:40.194149+0000","last_fullsized":"2024-09-18T07:31:40.194149+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:09.154352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":63,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194399+0000","last_change":"2024-09-18T07:30:58.314153+0000","last_active":"2024-09-18T07:31:40.194399+0000","last_peered":"2024-09-18T07:31:40.194399+0000","last_clean":"2024-09-18T07:31:40.194399+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:31:40.194399+0000","last_undegraded":"2024-09-18T07:31:40.194399+0000","last_fullsized":"2024-09-18T07:31:40.194399+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.980863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724596+0000","last_change":"2024-09-18T07:31:00.331498+0000","last_active":"2024-09-18T07:31:39.724596+0000","last_peered":"2024-09-18T07:31:39.724596+0000","last_clean":"2024-09-18T07:31:39.724596+0000","last_became_active":"2024-09-18T07:31:00.331224+0000","last_became_peered":"2024-09-18T07:31:00.331224+0000","last_unstale":"2024-09-18T07:31:39.724596+0000","last_undegraded":"2024-09-18T07:31:39.724596+0000","last_fullsized":"2024-09-18T07:31:39.724596+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:15.385510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195908+0000","last_change":"2024-09-18T07:30:56.286407+0000","last_active":"2024-09-18T07:31:40.195908+0000","last_peered":"2024-09-18T07:31:40.195908+0000","last_clean":"2024-09-18T07:31:40.195908+0000","last_became_active":"2024-09-18T07:30:56.285882+0000","last_became_peered":"2024-09-18T07:30:56.285882+0000","last_unstale":"2024-09-18T07:31:40.195908+0000","last_undegraded":"2024-09-18T07:31:40.195908+0000","last_fullsized":"2024-09-18T07:31:40.195908+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.252387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724970+0000","last_change":"2024-09-18T07:30:58.317018+0000","last_active":"2024-09-18T07:31:39.724970+0000","last_peered":"2024-09-18T07:31:39.724970+0000","last_clean":"2024-09-18T07:31:39.724970+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:31:39.724970+0000","last_undegraded":"2024-09-18T07:31:39.724970+0000","last_fullsized":"2024-09-18T07:31:39.724970+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:59.135054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195286+0000","last_change":"2024-09-18T07:31:00.330533+0000","last_active":"2024-09-18T07:31:40.195286+0000","last_peered":"2024-09-18T07:31:40.195286+0000","last_clean":"2024-09-18T07:31:40.195286+0000","last_became_active":"2024-09-18T07:31:00.330272+0000","last_became_peered":"2024-09-18T07:31:00.330272+0000","last_unstale":"2024-09-18T07:31:40.195286+0000","last_undegraded":"2024-09-18T07:31:40.195286+0000","last_fullsized":"2024-09-18T07:31:40.195286+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.245066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193105+0000","last_change":"2024-09-18T07:31:02.353810+0000","last_active":"2024-09-18T07:31:40.193105+0000","last_peered":"2024-09-18T07:31:40.193105+0000","last_clean":"2024-09-18T07:31:40.193105+0000","last_became_active":"2024-09-18T07:31:02.353494+0000","last_became_peered":"2024-09-18T07:31:02.353494+0000","last_unstale":"2024-09-18T07:31:40.193105+0000","last_undegraded":"2024-09-18T07:31:40.193105+0000","last_fullsized":"2024-09-18T07:31:40.193105+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.436492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195548+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:31:40.195548+0000","last_peered":"2024-09-18T07:31:40.195548+0000","last_clean":"2024-09-18T07:31:40.195548+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:31:40.195548+0000","last_undegraded":"2024-09-18T07:31:40.195548+0000","last_fullsized":"2024-09-18T07:31:40.195548+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.419105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725044+0000","last_change":"2024-09-18T07:30:58.317041+0000","last_active":"2024-09-18T07:31:39.725044+0000","last_peered":"2024-09-18T07:31:39.725044+0000","last_clean":"2024-09-18T07:31:39.725044+0000","last_became_active":"2024-09-18T07:30:58.316479+0000","last_became_peered":"2024-09-18T07:30:58.316479+0000","last_unstale":"2024-09-18T07:31:39.725044+0000","last_undegraded":"2024-09-18T07:31:39.725044+0000","last_fullsized":"2024-09-18T07:31:39.725044+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.191081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194451+0000","last_change":"2024-09-18T07:31:00.334023+0000","last_active":"2024-09-18T07:31:40.194451+0000","last_peered":"2024-09-18T07:31:40.194451+0000","last_clean":"2024-09-18T07:31:40.194451+0000","last_became_active":"2024-09-18T07:31:00.333685+0000","last_became_peered":"2024-09-18T07:31:00.333685+0000","last_unstale":"2024-09-18T07:31:40.194451+0000","last_undegraded":"2024-09-18T07:31:40.194451+0000","last_fullsized":"2024-09-18T07:31:40.194451+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.774482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194831+0000","last_change":"2024-09-18T07:31:02.349964+0000","last_active":"2024-09-18T07:31:40.194831+0000","last_peered":"2024-09-18T07:31:40.194831+0000","last_clean":"2024-09-18T07:31:40.194831+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:31:40.194831+0000","last_undegraded":"2024-09-18T07:31:40.194831+0000","last_fullsized":"2024-09-18T07:31:40.194831+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.932342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725534+0000","last_change":"2024-09-18T07:31:02.490397+0000","last_active":"2024-09-18T07:31:39.725534+0000","last_peered":"2024-09-18T07:31:39.725534+0000","last_clean":"2024-09-18T07:31:39.725534+0000","last_became_active":"2024-09-18T07:31:02.490014+0000","last_became_peered":"2024-09-18T07:31:02.490014+0000","last_unstale":"2024-09-18T07:31:39.725534+0000","last_undegraded":"2024-09-18T07:31:39.725534+0000","last_fullsized":"2024-09-18T07:31:39.725534+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.792505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193956+0000","last_change":"2024-09-18T07:30:56.288891+0000","last_active":"2024-09-18T07:31:40.193956+0000","last_peered":"2024-09-18T07:31:40.193956+0000","last_clean":"2024-09-18T07:31:40.193956+0000","last_became_active":"2024-09-18T07:30:56.288589+0000","last_became_peered":"2024-09-18T07:30:56.288589+0000","last_unstale":"2024-09-18T07:31:40.193956+0000","last_undegraded":"2024-09-18T07:31:40.193956+0000","last_fullsized":"2024-09-18T07:31:40.193956+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.974097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":63,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725072+0000","last_change":"2024-09-18T07:30:58.316556+0000","last_active":"2024-09-18T07:31:39.725072+0000","last_peered":"2024-09-18T07:31:39.725072+0000","last_clean":"2024-09-18T07:31:39.725072+0000","last_became_active":"2024-09-18T07:30:58.316266+0000","last_became_peered":"2024-09-18T07:30:58.316266+0000","last_unstale":"2024-09-18T07:31:39.725072+0000","last_undegraded":"2024-09-18T07:31:39.725072+0000","last_fullsized":"2024-09-18T07:31:39.725072+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:22.570802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725127+0000","last_change":"2024-09-18T07:31:00.338287+0000","last_active":"2024-09-18T07:31:39.725127+0000","last_peered":"2024-09-18T07:31:39.725127+0000","last_clean":"2024-09-18T07:31:39.725127+0000","last_became_active":"2024-09-18T07:31:00.337474+0000","last_became_peered":"2024-09-18T07:31:00.337474+0000","last_unstale":"2024-09-18T07:31:39.725127+0000","last_undegraded":"2024-09-18T07:31:39.725127+0000","last_fullsized":"2024-09-18T07:31:39.725127+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.609543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194017+0000","last_change":"2024-09-18T07:31:02.492790+0000","last_active":"2024-09-18T07:31:40.194017+0000","last_peered":"2024-09-18T07:31:40.194017+0000","last_clean":"2024-09-18T07:31:40.194017+0000","last_became_active":"2024-09-18T07:31:02.492348+0000","last_became_peered":"2024-09-18T07:31:02.492348+0000","last_unstale":"2024-09-18T07:31:40.194017+0000","last_undegraded":"2024-09-18T07:31:40.194017+0000","last_fullsized":"2024-09-18T07:31:40.194017+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:08.542586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725108+0000","last_change":"2024-09-18T07:30:56.289489+0000","last_active":"2024-09-18T07:31:39.725108+0000","last_peered":"2024-09-18T07:31:39.725108+0000","last_clean":"2024-09-18T07:31:39.725108+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:31:39.725108+0000","last_undegraded":"2024-09-18T07:31:39.725108+0000","last_fullsized":"2024-09-18T07:31:39.725108+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.331217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":61,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194047+0000","last_change":"2024-09-18T07:30:58.319115+0000","last_active":"2024-09-18T07:31:40.194047+0000","last_peered":"2024-09-18T07:31:40.194047+0000","last_clean":"2024-09-18T07:31:40.194047+0000","last_became_active":"2024-09-18T07:30:58.316380+0000","last_became_peered":"2024-09-18T07:30:58.316380+0000","last_unstale":"2024-09-18T07:31:40.194047+0000","last_undegraded":"2024-09-18T07:31:40.194047+0000","last_fullsized":"2024-09-18T07:31:40.194047+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.995221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725226+0000","last_change":"2024-09-18T07:31:00.334313+0000","last_active":"2024-09-18T07:31:39.725226+0000","last_peered":"2024-09-18T07:31:39.725226+0000","last_clean":"2024-09-18T07:31:39.725226+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:31:39.725226+0000","last_undegraded":"2024-09-18T07:31:39.725226+0000","last_fullsized":"2024-09-18T07:31:39.725226+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:09.373920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194653+0000","last_change":"2024-09-18T07:31:02.348855+0000","last_active":"2024-09-18T07:31:40.194653+0000","last_peered":"2024-09-18T07:31:40.194653+0000","last_clean":"2024-09-18T07:31:40.194653+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:31:40.194653+0000","last_undegraded":"2024-09-18T07:31:40.194653+0000","last_fullsized":"2024-09-18T07:31:40.194653+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.101140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723909+0000","last_change":"2024-09-18T07:30:56.283995+0000","last_active":"2024-09-18T07:31:39.723909+0000","last_peered":"2024-09-18T07:31:39.723909+0000","last_clean":"2024-09-18T07:31:39.723909+0000","last_became_active":"2024-09-18T07:30:56.283705+0000","last_became_peered":"2024-09-18T07:30:56.283705+0000","last_unstale":"2024-09-18T07:31:39.723909+0000","last_undegraded":"2024-09-18T07:31:39.723909+0000","last_fullsized":"2024-09-18T07:31:39.723909+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:55.938900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193741+0000","last_change":"2024-09-18T07:30:58.318003+0000","last_active":"2024-09-18T07:31:40.193741+0000","last_peered":"2024-09-18T07:31:40.193741+0000","last_clean":"2024-09-18T07:31:40.193741+0000","last_became_active":"2024-09-18T07:30:58.317355+0000","last_became_peered":"2024-09-18T07:30:58.317355+0000","last_unstale":"2024-09-18T07:31:40.193741+0000","last_undegraded":"2024-09-18T07:31:40.193741+0000","last_fullsized":"2024-09-18T07:31:40.193741+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.431023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194985+0000","last_change":"2024-09-18T07:31:00.331318+0000","last_active":"2024-09-18T07:31:40.194985+0000","last_peered":"2024-09-18T07:31:40.194985+0000","last_clean":"2024-09-18T07:31:40.194985+0000","last_became_active":"2024-09-18T07:31:00.331014+0000","last_became_peered":"2024-09-18T07:31:00.331014+0000","last_unstale":"2024-09-18T07:31:40.194985+0000","last_undegraded":"2024-09-18T07:31:40.194985+0000","last_fullsized":"2024-09-18T07:31:40.194985+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.342355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":63,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725256+0000","last_change":"2024-09-18T07:30:58.312916+0000","last_active":"2024-09-18T07:31:39.725256+0000","last_peered":"2024-09-18T07:31:39.725256+0000","last_clean":"2024-09-18T07:31:39.725256+0000","last_became_active":"2024-09-18T07:30:58.312627+0000","last_became_peered":"2024-09-18T07:30:58.312627+0000","last_unstale":"2024-09-18T07:31:39.725256+0000","last_undegraded":"2024-09-18T07:31:39.725256+0000","last_fullsized":"2024-09-18T07:31:39.725256+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.188948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193595+0000","last_change":"2024-09-18T07:30:56.289331+0000","last_active":"2024-09-18T07:31:40.193595+0000","last_peered":"2024-09-18T07:31:40.193595+0000","last_clean":"2024-09-18T07:31:40.193595+0000","last_became_active":"2024-09-18T07:30:56.289081+0000","last_became_peered":"2024-09-18T07:30:56.289081+0000","last_unstale":"2024-09-18T07:31:40.193595+0000","last_undegraded":"2024-09-18T07:31:40.193595+0000","last_fullsized":"2024-09-18T07:31:40.193595+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:56.283698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725726+0000","last_change":"2024-09-18T07:31:00.333026+0000","last_active":"2024-09-18T07:31:39.725726+0000","last_peered":"2024-09-18T07:31:39.725726+0000","last_clean":"2024-09-18T07:31:39.725726+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:31:39.725726+0000","last_undegraded":"2024-09-18T07:31:39.725726+0000","last_fullsized":"2024-09-18T07:31:39.725726+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:09.647541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195545+0000","last_change":"2024-09-18T07:31:02.351412+0000","last_active":"2024-09-18T07:31:40.195545+0000","last_peered":"2024-09-18T07:31:40.195545+0000","last_clean":"2024-09-18T07:31:40.195545+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:31:40.195545+0000","last_undegraded":"2024-09-18T07:31:40.195545+0000","last_fullsized":"2024-09-18T07:31:40.195545+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.243657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":70,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193495+0000","last_change":"2024-09-18T07:30:58.321857+0000","last_active":"2024-09-18T07:31:40.193495+0000","last_peered":"2024-09-18T07:31:40.193495+0000","last_clean":"2024-09-18T07:31:40.193495+0000","last_became_active":"2024-09-18T07:30:58.321244+0000","last_became_peered":"2024-09-18T07:30:58.321244+0000","last_unstale":"2024-09-18T07:31:40.193495+0000","last_undegraded":"2024-09-18T07:31:40.193495+0000","last_fullsized":"2024-09-18T07:31:40.193495+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:58.899213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194441+0000","last_change":"2024-09-18T07:30:56.289862+0000","last_active":"2024-09-18T07:31:40.194441+0000","last_peered":"2024-09-18T07:31:40.194441+0000","last_clean":"2024-09-18T07:31:40.194441+0000","last_became_active":"2024-09-18T07:30:56.289556+0000","last_became_peered":"2024-09-18T07:30:56.289556+0000","last_unstale":"2024-09-18T07:31:40.194441+0000","last_undegraded":"2024-09-18T07:31:40.194441+0000","last_fullsized":"2024-09-18T07:31:40.194441+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:59.535940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724628+0000","last_change":"2024-09-18T07:31:00.333320+0000","last_active":"2024-09-18T07:31:39.724628+0000","last_peered":"2024-09-18T07:31:39.724628+0000","last_clean":"2024-09-18T07:31:39.724628+0000","last_became_active":"2024-09-18T07:31:00.332854+0000","last_became_peered":"2024-09-18T07:31:00.332854+0000","last_unstale":"2024-09-18T07:31:39.724628+0000","last_undegraded":"2024-09-18T07:31:39.724628+0000","last_fullsized":"2024-09-18T07:31:39.724628+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:20.691030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724958+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:31:39.724958+0000","last_peered":"2024-09-18T07:31:39.724958+0000","last_clean":"2024-09-18T07:31:39.724958+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:31:39.724958+0000","last_undegraded":"2024-09-18T07:31:39.724958+0000","last_fullsized":"2024-09-18T07:31:39.724958+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.039810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724630+0000","last_change":"2024-09-18T07:30:58.309938+0000","last_active":"2024-09-18T07:31:39.724630+0000","last_peered":"2024-09-18T07:31:39.724630+0000","last_clean":"2024-09-18T07:31:39.724630+0000","last_became_active":"2024-09-18T07:30:58.309649+0000","last_became_peered":"2024-09-18T07:30:58.309649+0000","last_unstale":"2024-09-18T07:31:39.724630+0000","last_undegraded":"2024-09-18T07:31:39.724630+0000","last_fullsized":"2024-09-18T07:31:39.724630+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:20.121712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193999+0000","last_change":"2024-09-18T07:30:56.287909+0000","last_active":"2024-09-18T07:31:40.193999+0000","last_peered":"2024-09-18T07:31:40.193999+0000","last_clean":"2024-09-18T07:31:40.193999+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:31:40.193999+0000","last_undegraded":"2024-09-18T07:31:40.193999+0000","last_fullsized":"2024-09-18T07:31:40.193999+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:05.243933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193351+0000","last_change":"2024-09-18T07:31:00.336059+0000","last_active":"2024-09-18T07:31:40.193351+0000","last_peered":"2024-09-18T07:31:40.193351+0000","last_clean":"2024-09-18T07:31:40.193351+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:31:40.193351+0000","last_undegraded":"2024-09-18T07:31:40.193351+0000","last_fullsized":"2024-09-18T07:31:40.193351+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.046222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195090+0000","last_change":"2024-09-18T07:31:02.492797+0000","last_active":"2024-09-18T07:31:40.195090+0000","last_peered":"2024-09-18T07:31:40.195090+0000","last_clean":"2024-09-18T07:31:40.195090+0000","last_became_active":"2024-09-18T07:31:02.492483+0000","last_became_peered":"2024-09-18T07:31:02.492483+0000","last_unstale":"2024-09-18T07:31:40.195090+0000","last_undegraded":"2024-09-18T07:31:40.195090+0000","last_fullsized":"2024-09-18T07:31:40.195090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.277226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724110+0000","last_change":"2024-09-18T07:30:58.307470+0000","last_active":"2024-09-18T07:31:39.724110+0000","last_peered":"2024-09-18T07:31:39.724110+0000","last_clean":"2024-09-18T07:31:39.724110+0000","last_became_active":"2024-09-18T07:30:58.307035+0000","last_became_peered":"2024-09-18T07:30:58.307035+0000","last_unstale":"2024-09-18T07:31:39.724110+0000","last_undegraded":"2024-09-18T07:31:39.724110+0000","last_fullsized":"2024-09-18T07:31:39.724110+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:06.964257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.196031+0000","last_change":"2024-09-18T07:30:56.290040+0000","last_active":"2024-09-18T07:31:40.196031+0000","last_peered":"2024-09-18T07:31:40.196031+0000","last_clean":"2024-09-18T07:31:40.196031+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:31:40.196031+0000","last_undegraded":"2024-09-18T07:31:40.196031+0000","last_fullsized":"2024-09-18T07:31:40.196031+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.332869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193654+0000","last_change":"2024-09-18T07:31:00.337241+0000","last_active":"2024-09-18T07:31:40.193654+0000","last_peered":"2024-09-18T07:31:40.193654+0000","last_clean":"2024-09-18T07:31:40.193654+0000","last_became_active":"2024-09-18T07:31:00.336905+0000","last_became_peered":"2024-09-18T07:31:00.336905+0000","last_unstale":"2024-09-18T07:31:40.193654+0000","last_undegraded":"2024-09-18T07:31:40.193654+0000","last_fullsized":"2024-09-18T07:31:40.193654+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.826427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193237+0000","last_change":"2024-09-18T07:31:02.348538+0000","last_active":"2024-09-18T07:31:40.193237+0000","last_peered":"2024-09-18T07:31:40.193237+0000","last_clean":"2024-09-18T07:31:40.193237+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:31:40.193237+0000","last_undegraded":"2024-09-18T07:31:40.193237+0000","last_fullsized":"2024-09-18T07:31:40.193237+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.088828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":74,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193703+0000","last_change":"2024-09-18T07:30:58.318096+0000","last_active":"2024-09-18T07:31:40.193703+0000","last_peered":"2024-09-18T07:31:40.193703+0000","last_clean":"2024-09-18T07:31:40.193703+0000","last_became_active":"2024-09-18T07:30:58.317767+0000","last_became_peered":"2024-09-18T07:30:58.317767+0000","last_unstale":"2024-09-18T07:31:40.193703+0000","last_undegraded":"2024-09-18T07:31:40.193703+0000","last_fullsized":"2024-09-18T07:31:40.193703+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:26.992100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724928+0000","last_change":"2024-09-18T07:30:56.284440+0000","last_active":"2024-09-18T07:31:39.724928+0000","last_peered":"2024-09-18T07:31:39.724928+0000","last_clean":"2024-09-18T07:31:39.724928+0000","last_became_active":"2024-09-18T07:30:56.284120+0000","last_became_peered":"2024-09-18T07:30:56.284120+0000","last_unstale":"2024-09-18T07:31:39.724928+0000","last_undegraded":"2024-09-18T07:31:39.724928+0000","last_fullsized":"2024-09-18T07:31:39.724928+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.025187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193979+0000","last_change":"2024-09-18T07:31:00.335100+0000","last_active":"2024-09-18T07:31:40.193979+0000","last_peered":"2024-09-18T07:31:40.193979+0000","last_clean":"2024-09-18T07:31:40.193979+0000","last_became_active":"2024-09-18T07:31:00.334776+0000","last_became_peered":"2024-09-18T07:31:00.334776+0000","last_unstale":"2024-09-18T07:31:40.193979+0000","last_undegraded":"2024-09-18T07:31:40.193979+0000","last_fullsized":"2024-09-18T07:31:40.193979+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.405293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723750+0000","last_change":"2024-09-18T07:31:02.490534+0000","last_active":"2024-09-18T07:31:39.723750+0000","last_peered":"2024-09-18T07:31:39.723750+0000","last_clean":"2024-09-18T07:31:39.723750+0000","last_became_active":"2024-09-18T07:31:02.490186+0000","last_became_peered":"2024-09-18T07:31:02.490186+0000","last_unstale":"2024-09-18T07:31:39.723750+0000","last_undegraded":"2024-09-18T07:31:39.723750+0000","last_fullsized":"2024-09-18T07:31:39.723750+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.881705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":96,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724845+0000","last_change":"2024-09-18T07:30:58.317477+0000","last_active":"2024-09-18T07:31:39.724845+0000","last_peered":"2024-09-18T07:31:39.724845+0000","last_clean":"2024-09-18T07:31:39.724845+0000","last_became_active":"2024-09-18T07:30:58.316812+0000","last_became_peered":"2024-09-18T07:30:58.316812+0000","last_unstale":"2024-09-18T07:31:39.724845+0000","last_undegraded":"2024-09-18T07:31:39.724845+0000","last_fullsized":"2024-09-18T07:31:39.724845+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.769678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725296+0000","last_change":"2024-09-18T07:30:56.285353+0000","last_active":"2024-09-18T07:31:39.725296+0000","last_peered":"2024-09-18T07:31:39.725296+0000","last_clean":"2024-09-18T07:31:39.725296+0000","last_became_active":"2024-09-18T07:30:56.285032+0000","last_became_peered":"2024-09-18T07:30:56.285032+0000","last_unstale":"2024-09-18T07:31:39.725296+0000","last_undegraded":"2024-09-18T07:31:39.725296+0000","last_fullsized":"2024-09-18T07:31:39.725296+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:23.187833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195617+0000","last_change":"2024-09-18T07:30:58.299308+0000","last_active":"2024-09-18T07:31:40.195617+0000","last_peered":"2024-09-18T07:31:40.195617+0000","last_clean":"2024-09-18T07:31:40.195617+0000","last_became_active":"2024-09-18T07:30:55.283292+0000","last_became_peered":"2024-09-18T07:30:55.283292+0000","last_unstale":"2024-09-18T07:31:40.195617+0000","last_undegraded":"2024-09-18T07:31:40.195617+0000","last_fullsized":"2024-09-18T07:31:40.195617+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:13.332916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193537+0000","last_change":"2024-09-18T07:31:00.332738+0000","last_active":"2024-09-18T07:31:40.193537+0000","last_peered":"2024-09-18T07:31:40.193537+0000","last_clean":"2024-09-18T07:31:40.193537+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:31:40.193537+0000","last_undegraded":"2024-09-18T07:31:40.193537+0000","last_fullsized":"2024-09-18T07:31:40.193537+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:23.578215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193936+0000","last_change":"2024-09-18T07:31:02.351778+0000","last_active":"2024-09-18T07:31:40.193936+0000","last_peered":"2024-09-18T07:31:40.193936+0000","last_clean":"2024-09-18T07:31:40.193936+0000","last_became_active":"2024-09-18T07:31:02.351403+0000","last_became_peered":"2024-09-18T07:31:02.351403+0000","last_unstale":"2024-09-18T07:31:40.193936+0000","last_undegraded":"2024-09-18T07:31:40.193936+0000","last_fullsized":"2024-09-18T07:31:40.193936+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:27.871124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":65,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725277+0000","last_change":"2024-09-18T07:30:58.310789+0000","last_active":"2024-09-18T07:31:39.725277+0000","last_peered":"2024-09-18T07:31:39.725277+0000","last_clean":"2024-09-18T07:31:39.725277+0000","last_became_active":"2024-09-18T07:30:58.310490+0000","last_became_peered":"2024-09-18T07:30:58.310490+0000","last_unstale":"2024-09-18T07:31:39.725277+0000","last_undegraded":"2024-09-18T07:31:39.725277+0000","last_fullsized":"2024-09-18T07:31:39.725277+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:22.258389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723760+0000","last_change":"2024-09-18T07:30:56.285054+0000","last_active":"2024-09-18T07:31:39.723760+0000","last_peered":"2024-09-18T07:31:39.723760+0000","last_clean":"2024-09-18T07:31:39.723760+0000","last_became_active":"2024-09-18T07:30:56.284803+0000","last_became_peered":"2024-09-18T07:30:56.284803+0000","last_unstale":"2024-09-18T07:31:39.723760+0000","last_undegraded":"2024-09-18T07:31:39.723760+0000","last_fullsized":"2024-09-18T07:31:39.723760+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.764106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":51,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723587+0000","last_change":"2024-09-18T07:30:58.301149+0000","last_active":"2024-09-18T07:31:39.723587+0000","last_peered":"2024-09-18T07:31:39.723587+0000","last_clean":"2024-09-18T07:31:39.723587+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:31:39.723587+0000","last_undegraded":"2024-09-18T07:31:39.723587+0000","last_fullsized":"2024-09-18T07:31:39.723587+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:01.827803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724199+0000","last_change":"2024-09-18T07:31:00.331226+0000","last_active":"2024-09-18T07:31:39.724199+0000","last_peered":"2024-09-18T07:31:39.724199+0000","last_clean":"2024-09-18T07:31:39.724199+0000","last_became_active":"2024-09-18T07:31:00.330883+0000","last_became_peered":"2024-09-18T07:31:00.330883+0000","last_unstale":"2024-09-18T07:31:39.724199+0000","last_undegraded":"2024-09-18T07:31:39.724199+0000","last_fullsized":"2024-09-18T07:31:39.724199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.695331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194820+0000","last_change":"2024-09-18T07:31:02.491512+0000","last_active":"2024-09-18T07:31:40.194820+0000","last_peered":"2024-09-18T07:31:40.194820+0000","last_clean":"2024-09-18T07:31:40.194820+0000","last_became_active":"2024-09-18T07:31:02.491233+0000","last_became_peered":"2024-09-18T07:31:02.491233+0000","last_unstale":"2024-09-18T07:31:40.194820+0000","last_undegraded":"2024-09-18T07:31:40.194820+0000","last_fullsized":"2024-09-18T07:31:40.194820+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:26.099794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":56,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193830+0000","last_change":"2024-09-18T07:30:58.311778+0000","last_active":"2024-09-18T07:31:40.193830+0000","last_peered":"2024-09-18T07:31:40.193830+0000","last_clean":"2024-09-18T07:31:40.193830+0000","last_became_active":"2024-09-18T07:30:58.311491+0000","last_became_peered":"2024-09-18T07:30:58.311491+0000","last_unstale":"2024-09-18T07:31:40.193830+0000","last_undegraded":"2024-09-18T07:31:40.193830+0000","last_fullsized":"2024-09-18T07:31:40.193830+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:05.875741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193169+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:31:40.193169+0000","last_peered":"2024-09-18T07:31:40.193169+0000","last_clean":"2024-09-18T07:31:40.193169+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:31:40.193169+0000","last_undegraded":"2024-09-18T07:31:40.193169+0000","last_fullsized":"2024-09-18T07:31:40.193169+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:20.989190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":93,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194586+0000","last_change":"2024-09-18T07:30:58.319969+0000","last_active":"2024-09-18T07:31:40.194586+0000","last_peered":"2024-09-18T07:31:40.194586+0000","last_clean":"2024-09-18T07:31:40.194586+0000","last_became_active":"2024-09-18T07:30:55.286980+0000","last_became_peered":"2024-09-18T07:30:55.286980+0000","last_unstale":"2024-09-18T07:31:40.194586+0000","last_undegraded":"2024-09-18T07:31:40.194586+0000","last_fullsized":"2024-09-18T07:31:40.194586+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.742429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.196240+0000","last_change":"2024-09-18T07:31:00.331929+0000","last_active":"2024-09-18T07:31:40.196240+0000","last_peered":"2024-09-18T07:31:40.196240+0000","last_clean":"2024-09-18T07:31:40.196240+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:31:40.196240+0000","last_undegraded":"2024-09-18T07:31:40.196240+0000","last_fullsized":"2024-09-18T07:31:40.196240+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.504554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724544+0000","last_change":"2024-09-18T07:31:02.347659+0000","last_active":"2024-09-18T07:31:39.724544+0000","last_peered":"2024-09-18T07:31:39.724544+0000","last_clean":"2024-09-18T07:31:39.724544+0000","last_became_active":"2024-09-18T07:31:02.347390+0000","last_became_peered":"2024-09-18T07:31:02.347390+0000","last_unstale":"2024-09-18T07:31:39.724544+0000","last_undegraded":"2024-09-18T07:31:39.724544+0000","last_fullsized":"2024-09-18T07:31:39.724544+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.353794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":70,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194489+0000","last_change":"2024-09-18T07:30:58.318426+0000","last_active":"2024-09-18T07:31:40.194489+0000","last_peered":"2024-09-18T07:31:40.194489+0000","last_clean":"2024-09-18T07:31:40.194489+0000","last_became_active":"2024-09-18T07:30:58.318068+0000","last_became_peered":"2024-09-18T07:30:58.318068+0000","last_unstale":"2024-09-18T07:31:40.194489+0000","last_undegraded":"2024-09-18T07:31:40.194489+0000","last_fullsized":"2024-09-18T07:31:40.194489+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:59.863882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193876+0000","last_change":"2024-09-18T07:30:56.288420+0000","last_active":"2024-09-18T07:31:40.193876+0000","last_peered":"2024-09-18T07:31:40.193876+0000","last_clean":"2024-09-18T07:31:40.193876+0000","last_became_active":"2024-09-18T07:30:56.288149+0000","last_became_peered":"2024-09-18T07:30:56.288149+0000","last_unstale":"2024-09-18T07:31:40.193876+0000","last_undegraded":"2024-09-18T07:31:40.193876+0000","last_fullsized":"2024-09-18T07:31:40.193876+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:56.385307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":89,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194761+0000","last_change":"2024-09-18T07:30:50.813696+0000","last_active":"2024-09-18T07:31:40.194761+0000","last_peered":"2024-09-18T07:31:40.194761+0000","last_clean":"2024-09-18T07:31:40.194761+0000","last_became_active":"2024-09-18T07:30:46.719588+0000","last_became_peered":"2024-09-18T07:30:46.719588+0000","last_unstale":"2024-09-18T07:31:40.194761+0000","last_undegraded":"2024-09-18T07:31:40.194761+0000","last_fullsized":"2024-09-18T07:31:40.194761+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-18T07:30:50.813645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:30:50.813645+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:06.405856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195451+0000","last_change":"2024-09-18T07:31:00.331602+0000","last_active":"2024-09-18T07:31:40.195451+0000","last_peered":"2024-09-18T07:31:40.195451+0000","last_clean":"2024-09-18T07:31:40.195451+0000","last_became_active":"2024-09-18T07:31:00.331327+0000","last_became_peered":"2024-09-18T07:31:00.331327+0000","last_unstale":"2024-09-18T07:31:40.195451+0000","last_undegraded":"2024-09-18T07:31:40.195451+0000","last_fullsized":"2024-09-18T07:31:40.195451+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.402435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.196137+0000","last_change":"2024-09-18T07:31:02.348225+0000","last_active":"2024-09-18T07:31:40.196137+0000","last_peered":"2024-09-18T07:31:40.196137+0000","last_clean":"2024-09-18T07:31:40.196137+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:31:40.196137+0000","last_undegraded":"2024-09-18T07:31:40.196137+0000","last_fullsized":"2024-09-18T07:31:40.196137+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.749034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":57,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725729+0000","last_change":"2024-09-18T07:30:58.313677+0000","last_active":"2024-09-18T07:31:39.725729+0000","last_peered":"2024-09-18T07:31:39.725729+0000","last_clean":"2024-09-18T07:31:39.725729+0000","last_became_active":"2024-09-18T07:30:58.313382+0000","last_became_peered":"2024-09-18T07:30:58.313382+0000","last_unstale":"2024-09-18T07:31:39.725729+0000","last_undegraded":"2024-09-18T07:31:39.725729+0000","last_fullsized":"2024-09-18T07:31:39.725729+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.250207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724904+0000","last_change":"2024-09-18T07:30:56.291117+0000","last_active":"2024-09-18T07:31:39.724904+0000","last_peered":"2024-09-18T07:31:39.724904+0000","last_clean":"2024-09-18T07:31:39.724904+0000","last_became_active":"2024-09-18T07:30:56.290836+0000","last_became_peered":"2024-09-18T07:30:56.290836+0000","last_unstale":"2024-09-18T07:31:39.724904+0000","last_undegraded":"2024-09-18T07:31:39.724904+0000","last_fullsized":"2024-09-18T07:31:39.724904+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.125102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.196078+0000","last_change":"2024-09-18T07:31:00.336459+0000","last_active":"2024-09-18T07:31:40.196078+0000","last_peered":"2024-09-18T07:31:40.196078+0000","last_clean":"2024-09-18T07:31:40.196078+0000","last_became_active":"2024-09-18T07:31:00.336172+0000","last_became_peered":"2024-09-18T07:31:00.336172+0000","last_unstale":"2024-09-18T07:31:40.196078+0000","last_undegraded":"2024-09-18T07:31:40.196078+0000","last_fullsized":"2024-09-18T07:31:40.196078+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:28.160928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193866+0000","last_change":"2024-09-18T07:31:02.491383+0000","last_active":"2024-09-18T07:31:40.193866+0000","last_peered":"2024-09-18T07:31:40.193866+0000","last_clean":"2024-09-18T07:31:40.193866+0000","last_became_active":"2024-09-18T07:31:02.490953+0000","last_became_peered":"2024-09-18T07:31:02.490953+0000","last_unstale":"2024-09-18T07:31:40.193866+0000","last_undegraded":"2024-09-18T07:31:40.193866+0000","last_fullsized":"2024-09-18T07:31:40.193866+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.829440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":66,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725540+0000","last_change":"2024-09-18T07:30:58.312636+0000","last_active":"2024-09-18T07:31:39.725540+0000","last_peered":"2024-09-18T07:31:39.725540+0000","last_clean":"2024-09-18T07:31:39.725540+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:31:39.725540+0000","last_undegraded":"2024-09-18T07:31:39.725540+0000","last_fullsized":"2024-09-18T07:31:39.725540+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:13.709878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194249+0000","last_change":"2024-09-18T07:30:56.284303+0000","last_active":"2024-09-18T07:31:40.194249+0000","last_peered":"2024-09-18T07:31:40.194249+0000","last_clean":"2024-09-18T07:31:40.194249+0000","last_became_active":"2024-09-18T07:30:56.284020+0000","last_became_peered":"2024-09-18T07:30:56.284020+0000","last_unstale":"2024-09-18T07:31:40.194249+0000","last_undegraded":"2024-09-18T07:31:40.194249+0000","last_fullsized":"2024-09-18T07:31:40.194249+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:15.982099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724628+0000","last_change":"2024-09-18T07:31:00.333155+0000","last_active":"2024-09-18T07:31:39.724628+0000","last_peered":"2024-09-18T07:31:39.724628+0000","last_clean":"2024-09-18T07:31:39.724628+0000","last_became_active":"2024-09-18T07:31:00.332836+0000","last_became_peered":"2024-09-18T07:31:00.332836+0000","last_unstale":"2024-09-18T07:31:39.724628+0000","last_undegraded":"2024-09-18T07:31:39.724628+0000","last_fullsized":"2024-09-18T07:31:39.724628+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.946427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724573+0000","last_change":"2024-09-18T07:31:02.347665+0000","last_active":"2024-09-18T07:31:39.724573+0000","last_peered":"2024-09-18T07:31:39.724573+0000","last_clean":"2024-09-18T07:31:39.724573+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:31:39.724573+0000","last_undegraded":"2024-09-18T07:31:39.724573+0000","last_fullsized":"2024-09-18T07:31:39.724573+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.617366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725236+0000","last_change":"2024-09-18T07:30:58.317008+0000","last_active":"2024-09-18T07:31:39.725236+0000","last_peered":"2024-09-18T07:31:39.725236+0000","last_clean":"2024-09-18T07:31:39.725236+0000","last_became_active":"2024-09-18T07:30:58.316423+0000","last_became_peered":"2024-09-18T07:30:58.316423+0000","last_unstale":"2024-09-18T07:31:39.725236+0000","last_undegraded":"2024-09-18T07:31:39.725236+0000","last_fullsized":"2024-09-18T07:31:39.725236+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.580907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194222+0000","last_change":"2024-09-18T07:30:56.287284+0000","last_active":"2024-09-18T07:31:40.194222+0000","last_peered":"2024-09-18T07:31:40.194222+0000","last_clean":"2024-09-18T07:31:40.194222+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:31:40.194222+0000","last_undegraded":"2024-09-18T07:31:40.194222+0000","last_fullsized":"2024-09-18T07:31:40.194222+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:00.720049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723893+0000","last_change":"2024-09-18T07:31:00.333376+0000","last_active":"2024-09-18T07:31:39.723893+0000","last_peered":"2024-09-18T07:31:39.723893+0000","last_clean":"2024-09-18T07:31:39.723893+0000","last_became_active":"2024-09-18T07:31:00.333085+0000","last_became_peered":"2024-09-18T07:31:00.333085+0000","last_unstale":"2024-09-18T07:31:39.723893+0000","last_undegraded":"2024-09-18T07:31:39.723893+0000","last_fullsized":"2024-09-18T07:31:39.723893+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:22.590817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725525+0000","last_change":"2024-09-18T07:31:02.347721+0000","last_active":"2024-09-18T07:31:39.725525+0000","last_peered":"2024-09-18T07:31:39.725525+0000","last_clean":"2024-09-18T07:31:39.725525+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:31:39.725525+0000","last_undegraded":"2024-09-18T07:31:39.725525+0000","last_fullsized":"2024-09-18T07:31:39.725525+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.723859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193700+0000","last_change":"2024-09-18T07:30:58.314424+0000","last_active":"2024-09-18T07:31:40.193700+0000","last_peered":"2024-09-18T07:31:40.193700+0000","last_clean":"2024-09-18T07:31:40.193700+0000","last_became_active":"2024-09-18T07:30:58.314127+0000","last_became_peered":"2024-09-18T07:30:58.314127+0000","last_unstale":"2024-09-18T07:31:40.193700+0000","last_undegraded":"2024-09-18T07:31:40.193700+0000","last_fullsized":"2024-09-18T07:31:40.193700+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:13.636020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195735+0000","last_change":"2024-09-18T07:30:56.289177+0000","last_active":"2024-09-18T07:31:40.195735+0000","last_peered":"2024-09-18T07:31:40.195735+0000","last_clean":"2024-09-18T07:31:40.195735+0000","last_became_active":"2024-09-18T07:30:56.288868+0000","last_became_peered":"2024-09-18T07:30:56.288868+0000","last_unstale":"2024-09-18T07:31:40.195735+0000","last_undegraded":"2024-09-18T07:31:40.195735+0000","last_fullsized":"2024-09-18T07:31:40.195735+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.629409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723663+0000","last_change":"2024-09-18T07:31:00.336020+0000","last_active":"2024-09-18T07:31:39.723663+0000","last_peered":"2024-09-18T07:31:39.723663+0000","last_clean":"2024-09-18T07:31:39.723663+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:31:39.723663+0000","last_undegraded":"2024-09-18T07:31:39.723663+0000","last_fullsized":"2024-09-18T07:31:39.723663+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:28.112892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193110+0000","last_change":"2024-09-18T07:31:02.349757+0000","last_active":"2024-09-18T07:31:40.193110+0000","last_peered":"2024-09-18T07:31:40.193110+0000","last_clean":"2024-09-18T07:31:40.193110+0000","last_became_active":"2024-09-18T07:31:02.349454+0000","last_became_peered":"2024-09-18T07:31:02.349454+0000","last_unstale":"2024-09-18T07:31:40.193110+0000","last_undegraded":"2024-09-18T07:31:40.193110+0000","last_fullsized":"2024-09-18T07:31:40.193110+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.874882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":69,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194032+0000","last_change":"2024-09-18T07:30:58.318421+0000","last_active":"2024-09-18T07:31:40.194032+0000","last_peered":"2024-09-18T07:31:40.194032+0000","last_clean":"2024-09-18T07:31:40.194032+0000","last_became_active":"2024-09-18T07:30:58.318063+0000","last_became_peered":"2024-09-18T07:30:58.318063+0000","last_unstale":"2024-09-18T07:31:40.194032+0000","last_undegraded":"2024-09-18T07:31:40.194032+0000","last_fullsized":"2024-09-18T07:31:40.194032+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:59.591866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195532+0000","last_change":"2024-09-18T07:30:56.293678+0000","last_active":"2024-09-18T07:31:40.195532+0000","last_peered":"2024-09-18T07:31:40.195532+0000","last_clean":"2024-09-18T07:31:40.195532+0000","last_became_active":"2024-09-18T07:30:56.293371+0000","last_became_peered":"2024-09-18T07:30:56.293371+0000","last_unstale":"2024-09-18T07:31:40.195532+0000","last_undegraded":"2024-09-18T07:31:40.195532+0000","last_fullsized":"2024-09-18T07:31:40.195532+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.829476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724381+0000","last_change":"2024-09-18T07:31:00.332322+0000","last_active":"2024-09-18T07:31:39.724381+0000","last_peered":"2024-09-18T07:31:39.724381+0000","last_clean":"2024-09-18T07:31:39.724381+0000","last_became_active":"2024-09-18T07:31:00.331914+0000","last_became_peered":"2024-09-18T07:31:00.331914+0000","last_unstale":"2024-09-18T07:31:39.724381+0000","last_undegraded":"2024-09-18T07:31:39.724381+0000","last_fullsized":"2024-09-18T07:31:39.724381+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.543866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194651+0000","last_change":"2024-09-18T07:31:02.351941+0000","last_active":"2024-09-18T07:31:40.194651+0000","last_peered":"2024-09-18T07:31:40.194651+0000","last_clean":"2024-09-18T07:31:40.194651+0000","last_became_active":"2024-09-18T07:31:02.351678+0000","last_became_peered":"2024-09-18T07:31:02.351678+0000","last_unstale":"2024-09-18T07:31:40.194651+0000","last_undegraded":"2024-09-18T07:31:40.194651+0000","last_fullsized":"2024-09-18T07:31:40.194651+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.816291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":61,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724785+0000","last_change":"2024-09-18T07:30:58.320305+0000","last_active":"2024-09-18T07:31:39.724785+0000","last_peered":"2024-09-18T07:31:39.724785+0000","last_clean":"2024-09-18T07:31:39.724785+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:31:39.724785+0000","last_undegraded":"2024-09-18T07:31:39.724785+0000","last_fullsized":"2024-09-18T07:31:39.724785+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.490659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195144+0000","last_change":"2024-09-18T07:30:56.290367+0000","last_active":"2024-09-18T07:31:40.195144+0000","last_peered":"2024-09-18T07:31:40.195144+0000","last_clean":"2024-09-18T07:31:40.195144+0000","last_became_active":"2024-09-18T07:30:56.289855+0000","last_became_peered":"2024-09-18T07:30:56.289855+0000","last_unstale":"2024-09-18T07:31:40.195144+0000","last_undegraded":"2024-09-18T07:31:40.195144+0000","last_fullsized":"2024-09-18T07:31:40.195144+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.319326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723264+0000","last_change":"2024-09-18T07:31:00.329175+0000","last_active":"2024-09-18T07:31:39.723264+0000","last_peered":"2024-09-18T07:31:39.723264+0000","last_clean":"2024-09-18T07:31:39.723264+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:31:39.723264+0000","last_undegraded":"2024-09-18T07:31:39.723264+0000","last_fullsized":"2024-09-18T07:31:39.723264+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.309875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193965+0000","last_change":"2024-09-18T07:31:02.353267+0000","last_active":"2024-09-18T07:31:40.193965+0000","last_peered":"2024-09-18T07:31:40.193965+0000","last_clean":"2024-09-18T07:31:40.193965+0000","last_became_active":"2024-09-18T07:31:02.352922+0000","last_became_peered":"2024-09-18T07:31:02.352922+0000","last_unstale":"2024-09-18T07:31:40.193965+0000","last_undegraded":"2024-09-18T07:31:40.193965+0000","last_fullsized":"2024-09-18T07:31:40.193965+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:26.271964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":63,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193728+0000","last_change":"2024-09-18T07:30:58.319585+0000","last_active":"2024-09-18T07:31:40.193728+0000","last_peered":"2024-09-18T07:31:40.193728+0000","last_clean":"2024-09-18T07:31:40.193728+0000","last_became_active":"2024-09-18T07:30:58.319300+0000","last_became_peered":"2024-09-18T07:30:58.319300+0000","last_unstale":"2024-09-18T07:31:40.193728+0000","last_undegraded":"2024-09-18T07:31:40.193728+0000","last_fullsized":"2024-09-18T07:31:40.193728+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.143713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195339+0000","last_change":"2024-09-18T07:30:56.292832+0000","last_active":"2024-09-18T07:31:40.195339+0000","last_peered":"2024-09-18T07:31:40.195339+0000","last_clean":"2024-09-18T07:31:40.195339+0000","last_became_active":"2024-09-18T07:30:56.292531+0000","last_became_peered":"2024-09-18T07:30:56.292531+0000","last_unstale":"2024-09-18T07:31:40.195339+0000","last_undegraded":"2024-09-18T07:31:40.195339+0000","last_fullsized":"2024-09-18T07:31:40.195339+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.359238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194103+0000","last_change":"2024-09-18T07:31:00.337915+0000","last_active":"2024-09-18T07:31:40.194103+0000","last_peered":"2024-09-18T07:31:40.194103+0000","last_clean":"2024-09-18T07:31:40.194103+0000","last_became_active":"2024-09-18T07:31:00.337483+0000","last_became_peered":"2024-09-18T07:31:00.337483+0000","last_unstale":"2024-09-18T07:31:40.194103+0000","last_undegraded":"2024-09-18T07:31:40.194103+0000","last_fullsized":"2024-09-18T07:31:40.194103+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:20.556343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193984+0000","last_change":"2024-09-18T07:31:02.489914+0000","last_active":"2024-09-18T07:31:40.193984+0000","last_peered":"2024-09-18T07:31:40.193984+0000","last_clean":"2024-09-18T07:31:40.193984+0000","last_became_active":"2024-09-18T07:31:02.489619+0000","last_became_peered":"2024-09-18T07:31:02.489619+0000","last_unstale":"2024-09-18T07:31:40.193984+0000","last_undegraded":"2024-09-18T07:31:40.193984+0000","last_fullsized":"2024-09-18T07:31:40.193984+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:17.709609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193918+0000","last_change":"2024-09-18T07:30:56.282629+0000","last_active":"2024-09-18T07:31:40.193918+0000","last_peered":"2024-09-18T07:31:40.193918+0000","last_clean":"2024-09-18T07:31:40.193918+0000","last_became_active":"2024-09-18T07:30:56.282357+0000","last_became_peered":"2024-09-18T07:30:56.282357+0000","last_unstale":"2024-09-18T07:31:40.193918+0000","last_undegraded":"2024-09-18T07:31:40.193918+0000","last_fullsized":"2024-09-18T07:31:40.193918+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.549682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":47,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194391+0000","last_change":"2024-09-18T07:30:58.320781+0000","last_active":"2024-09-18T07:31:40.194391+0000","last_peered":"2024-09-18T07:31:40.194391+0000","last_clean":"2024-09-18T07:31:40.194391+0000","last_became_active":"2024-09-18T07:30:58.320563+0000","last_became_peered":"2024-09-18T07:30:58.320563+0000","last_unstale":"2024-09-18T07:31:40.194391+0000","last_undegraded":"2024-09-18T07:31:40.194391+0000","last_fullsized":"2024-09-18T07:31:40.194391+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.842501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195911+0000","last_change":"2024-09-18T07:31:00.334155+0000","last_active":"2024-09-18T07:31:40.195911+0000","last_peered":"2024-09-18T07:31:40.195911+0000","last_clean":"2024-09-18T07:31:40.195911+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:31:40.195911+0000","last_undegraded":"2024-09-18T07:31:40.195911+0000","last_fullsized":"2024-09-18T07:31:40.195911+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.652219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194724+0000","last_change":"2024-09-18T07:31:02.492999+0000","last_active":"2024-09-18T07:31:40.194724+0000","last_peered":"2024-09-18T07:31:40.194724+0000","last_clean":"2024-09-18T07:31:40.194724+0000","last_became_active":"2024-09-18T07:31:02.492489+0000","last_became_peered":"2024-09-18T07:31:02.492489+0000","last_unstale":"2024-09-18T07:31:40.194724+0000","last_undegraded":"2024-09-18T07:31:40.194724+0000","last_fullsized":"2024-09-18T07:31:40.194724+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.972057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193917+0000","last_change":"2024-09-18T07:30:58.318263+0000","last_active":"2024-09-18T07:31:40.193917+0000","last_peered":"2024-09-18T07:31:40.193917+0000","last_clean":"2024-09-18T07:31:40.193917+0000","last_became_active":"2024-09-18T07:30:58.317732+0000","last_became_peered":"2024-09-18T07:30:58.317732+0000","last_unstale":"2024-09-18T07:31:40.193917+0000","last_undegraded":"2024-09-18T07:31:40.193917+0000","last_fullsized":"2024-09-18T07:31:40.193917+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.024721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195239+0000","last_change":"2024-09-18T07:30:56.292272+0000","last_active":"2024-09-18T07:31:40.195239+0000","last_peered":"2024-09-18T07:31:40.195239+0000","last_clean":"2024-09-18T07:31:40.195239+0000","last_became_active":"2024-09-18T07:30:56.290788+0000","last_became_peered":"2024-09-18T07:30:56.290788+0000","last_unstale":"2024-09-18T07:31:40.195239+0000","last_undegraded":"2024-09-18T07:31:40.195239+0000","last_fullsized":"2024-09-18T07:31:40.195239+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.447855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194323+0000","last_change":"2024-09-18T07:31:00.334848+0000","last_active":"2024-09-18T07:31:40.194323+0000","last_peered":"2024-09-18T07:31:40.194323+0000","last_clean":"2024-09-18T07:31:40.194323+0000","last_became_active":"2024-09-18T07:31:00.334478+0000","last_became_peered":"2024-09-18T07:31:00.334478+0000","last_unstale":"2024-09-18T07:31:40.194323+0000","last_undegraded":"2024-09-18T07:31:40.194323+0000","last_fullsized":"2024-09-18T07:31:40.194323+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.087953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724734+0000","last_change":"2024-09-18T07:31:02.344766+0000","last_active":"2024-09-18T07:31:39.724734+0000","last_peered":"2024-09-18T07:31:39.724734+0000","last_clean":"2024-09-18T07:31:39.724734+0000","last_became_active":"2024-09-18T07:31:02.344011+0000","last_became_peered":"2024-09-18T07:31:02.344011+0000","last_unstale":"2024-09-18T07:31:39.724734+0000","last_undegraded":"2024-09-18T07:31:39.724734+0000","last_fullsized":"2024-09-18T07:31:39.724734+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.862393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195887+0000","last_change":"2024-09-18T07:30:58.318363+0000","last_active":"2024-09-18T07:31:40.195887+0000","last_peered":"2024-09-18T07:31:40.195887+0000","last_clean":"2024-09-18T07:31:40.195887+0000","last_became_active":"2024-09-18T07:30:58.318057+0000","last_became_peered":"2024-09-18T07:30:58.318057+0000","last_unstale":"2024-09-18T07:31:40.195887+0000","last_undegraded":"2024-09-18T07:31:40.195887+0000","last_fullsized":"2024-09-18T07:31:40.195887+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.599726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193524+0000","last_change":"2024-09-18T07:30:56.287385+0000","last_active":"2024-09-18T07:31:40.193524+0000","last_peered":"2024-09-18T07:31:40.193524+0000","last_clean":"2024-09-18T07:31:40.193524+0000","last_became_active":"2024-09-18T07:30:56.287068+0000","last_became_peered":"2024-09-18T07:30:56.287068+0000","last_unstale":"2024-09-18T07:31:40.193524+0000","last_undegraded":"2024-09-18T07:31:40.193524+0000","last_fullsized":"2024-09-18T07:31:40.193524+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.184849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194082+0000","last_change":"2024-09-18T07:31:00.337716+0000","last_active":"2024-09-18T07:31:40.194082+0000","last_peered":"2024-09-18T07:31:40.194082+0000","last_clean":"2024-09-18T07:31:40.194082+0000","last_became_active":"2024-09-18T07:31:00.337444+0000","last_became_peered":"2024-09-18T07:31:00.337444+0000","last_unstale":"2024-09-18T07:31:40.194082+0000","last_undegraded":"2024-09-18T07:31:40.194082+0000","last_fullsized":"2024-09-18T07:31:40.194082+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.512126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725668+0000","last_change":"2024-09-18T07:31:02.343206+0000","last_active":"2024-09-18T07:31:39.725668+0000","last_peered":"2024-09-18T07:31:39.725668+0000","last_clean":"2024-09-18T07:31:39.725668+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:31:39.725668+0000","last_undegraded":"2024-09-18T07:31:39.725668+0000","last_fullsized":"2024-09-18T07:31:39.725668+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:23.329902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":53,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194303+0000","last_change":"2024-09-18T07:30:58.320351+0000","last_active":"2024-09-18T07:31:40.194303+0000","last_peered":"2024-09-18T07:31:40.194303+0000","last_clean":"2024-09-18T07:31:40.194303+0000","last_became_active":"2024-09-18T07:30:58.320093+0000","last_became_peered":"2024-09-18T07:30:58.320093+0000","last_unstale":"2024-09-18T07:31:40.194303+0000","last_undegraded":"2024-09-18T07:31:40.194303+0000","last_fullsized":"2024-09-18T07:31:40.194303+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.687274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195252+0000","last_change":"2024-09-18T07:30:56.291807+0000","last_active":"2024-09-18T07:31:40.195252+0000","last_peered":"2024-09-18T07:31:40.195252+0000","last_clean":"2024-09-18T07:31:40.195252+0000","last_became_active":"2024-09-18T07:30:56.291238+0000","last_became_peered":"2024-09-18T07:30:56.291238+0000","last_unstale":"2024-09-18T07:31:40.195252+0000","last_undegraded":"2024-09-18T07:31:40.195252+0000","last_fullsized":"2024-09-18T07:31:40.195252+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:17.785800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195794+0000","last_change":"2024-09-18T07:31:00.332729+0000","last_active":"2024-09-18T07:31:40.195794+0000","last_peered":"2024-09-18T07:31:40.195794+0000","last_clean":"2024-09-18T07:31:40.195794+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:31:40.195794+0000","last_undegraded":"2024-09-18T07:31:40.195794+0000","last_fullsized":"2024-09-18T07:31:40.195794+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:08.863683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193268+0000","last_change":"2024-09-18T07:31:02.348726+0000","last_active":"2024-09-18T07:31:40.193268+0000","last_peered":"2024-09-18T07:31:40.193268+0000","last_clean":"2024-09-18T07:31:40.193268+0000","last_became_active":"2024-09-18T07:31:02.348423+0000","last_became_peered":"2024-09-18T07:31:02.348423+0000","last_unstale":"2024-09-18T07:31:40.193268+0000","last_undegraded":"2024-09-18T07:31:40.193268+0000","last_fullsized":"2024-09-18T07:31:40.193268+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.760836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":57,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725116+0000","last_change":"2024-09-18T07:30:58.314730+0000","last_active":"2024-09-18T07:31:39.725116+0000","last_peered":"2024-09-18T07:31:39.725116+0000","last_clean":"2024-09-18T07:31:39.725116+0000","last_became_active":"2024-09-18T07:30:58.314428+0000","last_became_peered":"2024-09-18T07:30:58.314428+0000","last_unstale":"2024-09-18T07:31:39.725116+0000","last_undegraded":"2024-09-18T07:31:39.725116+0000","last_fullsized":"2024-09-18T07:31:39.725116+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.641591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.725433+0000","last_change":"2024-09-18T07:30:56.291115+0000","last_active":"2024-09-18T07:31:39.725433+0000","last_peered":"2024-09-18T07:31:39.725433+0000","last_clean":"2024-09-18T07:31:39.725433+0000","last_became_active":"2024-09-18T07:30:56.290816+0000","last_became_peered":"2024-09-18T07:30:56.290816+0000","last_unstale":"2024-09-18T07:31:39.725433+0000","last_undegraded":"2024-09-18T07:31:39.725433+0000","last_fullsized":"2024-09-18T07:31:39.725433+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.554722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.723958+0000","last_change":"2024-09-18T07:31:00.337826+0000","last_active":"2024-09-18T07:31:39.723958+0000","last_peered":"2024-09-18T07:31:39.723958+0000","last_clean":"2024-09-18T07:31:39.723958+0000","last_became_active":"2024-09-18T07:31:00.334158+0000","last_became_peered":"2024-09-18T07:31:00.334158+0000","last_unstale":"2024-09-18T07:31:39.723958+0000","last_undegraded":"2024-09-18T07:31:39.723958+0000","last_fullsized":"2024-09-18T07:31:39.723958+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:28.224028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193699+0000","last_change":"2024-09-18T07:31:02.351247+0000","last_active":"2024-09-18T07:31:40.193699+0000","last_peered":"2024-09-18T07:31:40.193699+0000","last_clean":"2024-09-18T07:31:40.193699+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:31:40.193699+0000","last_undegraded":"2024-09-18T07:31:40.193699+0000","last_fullsized":"2024-09-18T07:31:40.193699+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:25.935005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724310+0000","last_change":"2024-09-18T07:30:58.307904+0000","last_active":"2024-09-18T07:31:39.724310+0000","last_peered":"2024-09-18T07:31:39.724310+0000","last_clean":"2024-09-18T07:31:39.724310+0000","last_became_active":"2024-09-18T07:30:58.307448+0000","last_became_peered":"2024-09-18T07:30:58.307448+0000","last_unstale":"2024-09-18T07:31:39.724310+0000","last_undegraded":"2024-09-18T07:31:39.724310+0000","last_fullsized":"2024-09-18T07:31:39.724310+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:01.900229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.195450+0000","last_change":"2024-09-18T07:30:56.293467+0000","last_active":"2024-09-18T07:31:40.195450+0000","last_peered":"2024-09-18T07:31:40.195450+0000","last_clean":"2024-09-18T07:31:40.195450+0000","last_became_active":"2024-09-18T07:30:56.292980+0000","last_became_peered":"2024-09-18T07:30:56.292980+0000","last_unstale":"2024-09-18T07:31:40.195450+0000","last_undegraded":"2024-09-18T07:31:40.195450+0000","last_fullsized":"2024-09-18T07:31:40.195450+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.161233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193884+0000","last_change":"2024-09-18T07:31:00.333689+0000","last_active":"2024-09-18T07:31:40.193884+0000","last_peered":"2024-09-18T07:31:40.193884+0000","last_clean":"2024-09-18T07:31:40.193884+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:31:40.193884+0000","last_undegraded":"2024-09-18T07:31:40.193884+0000","last_fullsized":"2024-09-18T07:31:40.193884+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.075146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193043+0000","last_change":"2024-09-18T07:31:02.352805+0000","last_active":"2024-09-18T07:31:40.193043+0000","last_peered":"2024-09-18T07:31:40.193043+0000","last_clean":"2024-09-18T07:31:40.193043+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:31:40.193043+0000","last_undegraded":"2024-09-18T07:31:40.193043+0000","last_fullsized":"2024-09-18T07:31:40.193043+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.115071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":57,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194091+0000","last_change":"2024-09-18T07:30:58.320419+0000","last_active":"2024-09-18T07:31:40.194091+0000","last_peered":"2024-09-18T07:31:40.194091+0000","last_clean":"2024-09-18T07:31:40.194091+0000","last_became_active":"2024-09-18T07:30:58.320108+0000","last_became_peered":"2024-09-18T07:30:58.320108+0000","last_unstale":"2024-09-18T07:31:40.194091+0000","last_undegraded":"2024-09-18T07:31:40.194091+0000","last_fullsized":"2024-09-18T07:31:40.194091+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.449011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.196053+0000","last_change":"2024-09-18T07:30:56.289671+0000","last_active":"2024-09-18T07:31:40.196053+0000","last_peered":"2024-09-18T07:31:40.196053+0000","last_clean":"2024-09-18T07:31:40.196053+0000","last_became_active":"2024-09-18T07:30:56.289328+0000","last_became_peered":"2024-09-18T07:30:56.289328+0000","last_unstale":"2024-09-18T07:31:40.196053+0000","last_undegraded":"2024-09-18T07:31:40.196053+0000","last_fullsized":"2024-09-18T07:31:40.196053+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:58.531067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193590+0000","last_change":"2024-09-18T07:31:00.336053+0000","last_active":"2024-09-18T07:31:40.193590+0000","last_peered":"2024-09-18T07:31:40.193590+0000","last_clean":"2024-09-18T07:31:40.193590+0000","last_became_active":"2024-09-18T07:31:00.335749+0000","last_became_peered":"2024-09-18T07:31:00.335749+0000","last_unstale":"2024-09-18T07:31:40.193590+0000","last_undegraded":"2024-09-18T07:31:40.193590+0000","last_fullsized":"2024-09-18T07:31:40.193590+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:26.183381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194246+0000","last_change":"2024-09-18T07:31:02.491487+0000","last_active":"2024-09-18T07:31:40.194246+0000","last_peered":"2024-09-18T07:31:40.194246+0000","last_clean":"2024-09-18T07:31:40.194246+0000","last_became_active":"2024-09-18T07:31:02.490994+0000","last_became_peered":"2024-09-18T07:31:02.490994+0000","last_unstale":"2024-09-18T07:31:40.194246+0000","last_undegraded":"2024-09-18T07:31:40.194246+0000","last_fullsized":"2024-09-18T07:31:40.194246+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.401313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":44,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193399+0000","last_change":"2024-09-18T07:30:58.316173+0000","last_active":"2024-09-18T07:31:40.193399+0000","last_peered":"2024-09-18T07:31:40.193399+0000","last_clean":"2024-09-18T07:31:40.193399+0000","last_became_active":"2024-09-18T07:30:58.315435+0000","last_became_peered":"2024-09-18T07:30:58.315435+0000","last_unstale":"2024-09-18T07:31:40.193399+0000","last_undegraded":"2024-09-18T07:31:40.193399+0000","last_fullsized":"2024-09-18T07:31:40.193399+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:05.325393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193617+0000","last_change":"2024-09-18T07:30:56.283643+0000","last_active":"2024-09-18T07:31:40.193617+0000","last_peered":"2024-09-18T07:31:40.193617+0000","last_clean":"2024-09-18T07:31:40.193617+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:31:40.193617+0000","last_undegraded":"2024-09-18T07:31:40.193617+0000","last_fullsized":"2024-09-18T07:31:40.193617+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:17.971275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193929+0000","last_change":"2024-09-18T07:31:00.333126+0000","last_active":"2024-09-18T07:31:40.193929+0000","last_peered":"2024-09-18T07:31:40.193929+0000","last_clean":"2024-09-18T07:31:40.193929+0000","last_became_active":"2024-09-18T07:31:00.332855+0000","last_became_peered":"2024-09-18T07:31:00.332855+0000","last_unstale":"2024-09-18T07:31:40.193929+0000","last_undegraded":"2024-09-18T07:31:40.193929+0000","last_fullsized":"2024-09-18T07:31:40.193929+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:29.188004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194334+0000","last_change":"2024-09-18T07:31:02.350978+0000","last_active":"2024-09-18T07:31:40.194334+0000","last_peered":"2024-09-18T07:31:40.194334+0000","last_clean":"2024-09-18T07:31:40.194334+0000","last_became_active":"2024-09-18T07:31:02.350692+0000","last_became_peered":"2024-09-18T07:31:02.350692+0000","last_unstale":"2024-09-18T07:31:40.194334+0000","last_undegraded":"2024-09-18T07:31:40.194334+0000","last_fullsized":"2024-09-18T07:31:40.194334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.916468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":30,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:39.724958+0000","last_change":"2024-09-18T07:31:02.347827+0000","last_active":"2024-09-18T07:31:39.724958+0000","last_peered":"2024-09-18T07:31:39.724958+0000","last_clean":"2024-09-18T07:31:39.724958+0000","last_became_active":"2024-09-18T07:31:02.347537+0000","last_became_peered":"2024-09-18T07:31:02.347537+0000","last_unstale":"2024-09-18T07:31:39.724958+0000","last_undegraded":"2024-09-18T07:31:39.724958+0000","last_fullsized":"2024-09-18T07:31:39.724958+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:28.722445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193580+0000","last_change":"2024-09-18T07:30:56.292657+0000","last_active":"2024-09-18T07:31:40.193580+0000","last_peered":"2024-09-18T07:31:40.193580+0000","last_clean":"2024-09-18T07:31:40.193580+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:31:40.193580+0000","last_undegraded":"2024-09-18T07:31:40.193580+0000","last_fullsized":"2024-09-18T07:31:40.193580+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:13.946464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":57,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.194832+0000","last_change":"2024-09-18T07:30:58.314191+0000","last_active":"2024-09-18T07:31:40.194832+0000","last_peered":"2024-09-18T07:31:40.194832+0000","last_clean":"2024-09-18T07:31:40.194832+0000","last_became_active":"2024-09-18T07:30:58.313556+0000","last_became_peered":"2024-09-18T07:30:58.313556+0000","last_unstale":"2024-09-18T07:31:40.194832+0000","last_undegraded":"2024-09-18T07:31:40.194832+0000","last_fullsized":"2024-09-18T07:31:40.194832+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.554527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":34,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-18T07:31:40.193642+0000","last_change":"2024-09-18T07:31:00.331995+0000","last_active":"2024-09-18T07:31:40.193642+0000","last_peered":"2024-09-18T07:31:40.193642+0000","last_clean":"2024-09-18T07:31:40.193642+0000","last_became_active":"2024-09-18T07:31:00.331564+0000","last_became_peered":"2024-09-18T07:31:00.331564+0000","last_unstale":"2024-09-18T07:31:40.193642+0000","last_undegraded":"2024-09-18T07:31:40.193642+0000","last_fullsized":"2024-09-18T07:31:40.193642+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.738488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201293,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28360,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716792,"statfs":{"total":95995035648,"available":95965995008,"internally_reserved":0,"allocated":1363968,"data_stored":895800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462931,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32500,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712652,"statfs":{"total":95995035648,"available":95961755648,"internally_reserved":0,"allocated":1343488,"data_stored":894715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724568,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31924,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713228,"statfs":{"total":95995035648,"available":95962345472,"internally_reserved":0,"allocated":749568,"data_stored":304693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953503,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27788,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717364,"statfs":{"total":95995035648,"available":95966580736,"internally_reserved":0,"allocated":778240,"data_stored":305375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27804,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717348,"statfs":{"total":95995035648,"available":95966564352,"internally_reserved":0,"allocated":794624,"data_stored":306667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27752,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":741376,"data_stored":304482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":815104,"data_stored":307112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1372160,"data_stored":896925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:31:47.856 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-18T07:31:47.856 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-18T07:31:47.856 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-18T07:31:47.856 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-18T07:31:48.109 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:48.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:47 smithi137 ceph-mon[39175]: from='client.14784 v1:172.21.15.137:0/4228590248' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:31:48.240 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:47 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:31:48.240 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:47 smithi137 ceph-mon[29408]: from='client.14784 v1:172.21.15.137:0/4228590248' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:31:48.240 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:47 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:31:48.240 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:47 smithi203 ceph-mon[32237]: from='client.14784 v1:172.21.15.137:0/4228590248' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:31:48.240 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:47 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:31:48.967 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:31:48.967 INFO:teuthology.orchestra.run.smithi137.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-18T07:31:49.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:48 smithi137 ceph-mon[29408]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:31:49.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:48 smithi137 ceph-mon[39175]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:31:49.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:48 smithi203 ceph-mon[32237]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:31:50.112 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:49 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2735979869' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-18T07:31:50.112 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:49 smithi203 ceph-mon[32237]: osdmap e78: 8 total, 8 up, 8 in 2024-09-18T07:31:50.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:49 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2735979869' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-18T07:31:50.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:49 smithi137 ceph-mon[29408]: osdmap e78: 8 total, 8 up, 8 in 2024-09-18T07:31:50.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:49 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2735979869' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-18T07:31:50.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:49 smithi137 ceph-mon[39175]: osdmap e78: 8 total, 8 up, 8 in 2024-09-18T07:31:50.776 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-18T07:31:50.777 INFO:tasks.cephadm:Setup complete, yielding 2024-09-18T07:31:50.777 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-18T07:31:50.794 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-18T07:31:50.794 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-guits-main-2024-09-17-1213'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'mon_bind_msgr2': False, 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'wip-guits-main-2024-09-17-1213', 'sha1': 'e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e'}} 2024-09-18T07:31:50.794 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-18T07:31:50.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-18T07:31:51.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:50 smithi137 ceph-mon[29408]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-18T07:31:51.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:50 smithi137 ceph-mon[39175]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-18T07:31:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:50 smithi203 ceph-mon[32237]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-18T07:31:53.143 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:52 smithi137 ceph-mon[29408]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:31:53.143 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:52 smithi137 ceph-mon[39175]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:31:53.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:52 smithi203 ceph-mon[32237]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:31:53.615 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:54.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:54 smithi137 ceph-mon[29408]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:31:54.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:54 smithi137 ceph-mon[39175]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:31:55.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:54 smithi203 ceph-mon[32237]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:31:55.293 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:54 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:31:54] ENGINE Bus STOPPING 2024-09-18T07:31:55.293 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:55 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:31:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-18T07:31:55.293 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:55 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:31:55] ENGINE Bus STOPPED 2024-09-18T07:31:55.293 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:55 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:31:55] ENGINE Bus STARTING 2024-09-18T07:31:55.293 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:55 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:31:55] ENGINE Serving on http://:::9283 2024-09-18T07:31:55.293 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:55 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:31:55] ENGINE Bus STARTED 2024-09-18T07:31:56.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:55 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2728130851' entity='client.admin' 2024-09-18T07:31:56.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:55 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:31:56.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:55 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:31:56.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:55 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2728130851' entity='client.admin' 2024-09-18T07:31:56.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:55 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:31:56.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:55 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:31:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:55 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2728130851' entity='client.admin' 2024-09-18T07:31:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:55 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:31:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:55 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:31:56.474 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-18T07:31:56.474 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-18T07:31:56.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:31:56.744 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:56.866 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:31:56 smithi137 systemd[1]: Started Ceph alertmanager.a for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:31:57.139 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:31:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:31:57.139 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[29408]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T07:31:57.140 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[29408]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:31:57.140 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[29408]: 3.19 scrub starts 2024-09-18T07:31:57.140 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[29408]: 3.19 scrub ok 2024-09-18T07:31:57.140 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[29408]: 3.8 scrub starts 2024-09-18T07:31:57.140 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[29408]: 3.8 scrub ok 2024-09-18T07:31:57.140 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:57.140 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:57.140 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:57.140 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:57.140 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[39175]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T07:31:57.141 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[39175]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:31:57.141 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[39175]: 3.19 scrub starts 2024-09-18T07:31:57.141 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[39175]: 3.19 scrub ok 2024-09-18T07:31:57.141 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[39175]: 3.8 scrub starts 2024-09-18T07:31:57.141 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[39175]: 3.8 scrub ok 2024-09-18T07:31:57.141 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:57.141 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:57.141 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:57.141 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:56 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:57.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:31:56 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:31:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:56 smithi203 ceph-mon[32237]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T07:31:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:56 smithi203 ceph-mon[32237]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:31:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:56 smithi203 ceph-mon[32237]: 3.19 scrub starts 2024-09-18T07:31:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:56 smithi203 ceph-mon[32237]: 3.19 scrub ok 2024-09-18T07:31:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:56 smithi203 ceph-mon[32237]: 3.8 scrub starts 2024-09-18T07:31:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:56 smithi203 ceph-mon[32237]: 3.8 scrub ok 2024-09-18T07:31:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:56 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:56 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:56 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:56 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:57.416 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:31:57.418 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:31:58.037 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-18T07:31:55.410620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":898,"num_read_kb":755,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236668,"kb_used_data":7964,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724548,"statfs":{"total":767960285184,"available":767717937152,"internally_reserved":0,"allocated":8155136,"data_stored":4318521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002381"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728698+0000","last_change":"2024-09-18T07:30:56.284087+0000","last_active":"2024-09-18T07:31:49.728698+0000","last_peered":"2024-09-18T07:31:49.728698+0000","last_clean":"2024-09-18T07:31:49.728698+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:31:49.728698+0000","last_undegraded":"2024-09-18T07:31:49.728698+0000","last_fullsized":"2024-09-18T07:31:49.728698+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.049077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725514+0000","last_change":"2024-09-18T07:30:58.309315+0000","last_active":"2024-09-18T07:31:49.725514+0000","last_peered":"2024-09-18T07:31:49.725514+0000","last_clean":"2024-09-18T07:31:49.725514+0000","last_became_active":"2024-09-18T07:30:58.308981+0000","last_became_peered":"2024-09-18T07:30:58.308981+0000","last_unstale":"2024-09-18T07:31:49.725514+0000","last_undegraded":"2024-09-18T07:31:49.725514+0000","last_fullsized":"2024-09-18T07:31:49.725514+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:08.376023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212174+0000","last_change":"2024-09-18T07:31:00.336284+0000","last_active":"2024-09-18T07:31:50.212174+0000","last_peered":"2024-09-18T07:31:50.212174+0000","last_clean":"2024-09-18T07:31:50.212174+0000","last_became_active":"2024-09-18T07:31:00.335958+0000","last_became_peered":"2024-09-18T07:31:00.335958+0000","last_unstale":"2024-09-18T07:31:50.212174+0000","last_undegraded":"2024-09-18T07:31:50.212174+0000","last_fullsized":"2024-09-18T07:31:50.212174+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:08.645781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725723+0000","last_change":"2024-09-18T07:31:02.342510+0000","last_active":"2024-09-18T07:31:49.725723+0000","last_peered":"2024-09-18T07:31:49.725723+0000","last_clean":"2024-09-18T07:31:49.725723+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:31:49.725723+0000","last_undegraded":"2024-09-18T07:31:49.725723+0000","last_fullsized":"2024-09-18T07:31:49.725723+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.002947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212748+0000","last_change":"2024-09-18T07:31:02.492941+0000","last_active":"2024-09-18T07:31:50.212748+0000","last_peered":"2024-09-18T07:31:50.212748+0000","last_clean":"2024-09-18T07:31:50.212748+0000","last_became_active":"2024-09-18T07:31:02.492405+0000","last_became_peered":"2024-09-18T07:31:02.492405+0000","last_unstale":"2024-09-18T07:31:50.212748+0000","last_undegraded":"2024-09-18T07:31:50.212748+0000","last_fullsized":"2024-09-18T07:31:50.212748+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:22.665514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213765+0000","last_change":"2024-09-18T07:30:56.286313+0000","last_active":"2024-09-18T07:31:50.213765+0000","last_peered":"2024-09-18T07:31:50.213765+0000","last_clean":"2024-09-18T07:31:50.213765+0000","last_became_active":"2024-09-18T07:30:56.286040+0000","last_became_peered":"2024-09-18T07:30:56.286040+0000","last_unstale":"2024-09-18T07:31:50.213765+0000","last_undegraded":"2024-09-18T07:31:50.213765+0000","last_fullsized":"2024-09-18T07:31:50.213765+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:09.154352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212813+0000","last_change":"2024-09-18T07:30:58.314153+0000","last_active":"2024-09-18T07:31:50.212813+0000","last_peered":"2024-09-18T07:31:50.212813+0000","last_clean":"2024-09-18T07:31:50.212813+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:31:50.212813+0000","last_undegraded":"2024-09-18T07:31:50.212813+0000","last_fullsized":"2024-09-18T07:31:50.212813+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.980863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725505+0000","last_change":"2024-09-18T07:31:00.331498+0000","last_active":"2024-09-18T07:31:49.725505+0000","last_peered":"2024-09-18T07:31:49.725505+0000","last_clean":"2024-09-18T07:31:49.725505+0000","last_became_active":"2024-09-18T07:31:00.331224+0000","last_became_peered":"2024-09-18T07:31:00.331224+0000","last_unstale":"2024-09-18T07:31:49.725505+0000","last_undegraded":"2024-09-18T07:31:49.725505+0000","last_fullsized":"2024-09-18T07:31:49.725505+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:15.385510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214428+0000","last_change":"2024-09-18T07:30:56.286407+0000","last_active":"2024-09-18T07:31:50.214428+0000","last_peered":"2024-09-18T07:31:50.214428+0000","last_clean":"2024-09-18T07:31:50.214428+0000","last_became_active":"2024-09-18T07:30:56.285882+0000","last_became_peered":"2024-09-18T07:30:56.285882+0000","last_unstale":"2024-09-18T07:31:50.214428+0000","last_undegraded":"2024-09-18T07:31:50.214428+0000","last_fullsized":"2024-09-18T07:31:50.214428+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.252387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725892+0000","last_change":"2024-09-18T07:30:58.317018+0000","last_active":"2024-09-18T07:31:49.725892+0000","last_peered":"2024-09-18T07:31:49.725892+0000","last_clean":"2024-09-18T07:31:49.725892+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:31:49.725892+0000","last_undegraded":"2024-09-18T07:31:49.725892+0000","last_fullsized":"2024-09-18T07:31:49.725892+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:59.135054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214036+0000","last_change":"2024-09-18T07:31:00.330533+0000","last_active":"2024-09-18T07:31:50.214036+0000","last_peered":"2024-09-18T07:31:50.214036+0000","last_clean":"2024-09-18T07:31:50.214036+0000","last_became_active":"2024-09-18T07:31:00.330272+0000","last_became_peered":"2024-09-18T07:31:00.330272+0000","last_unstale":"2024-09-18T07:31:50.214036+0000","last_undegraded":"2024-09-18T07:31:50.214036+0000","last_fullsized":"2024-09-18T07:31:50.214036+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.245066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.727443+0000","last_change":"2024-09-18T07:31:02.353810+0000","last_active":"2024-09-18T07:31:49.727443+0000","last_peered":"2024-09-18T07:31:49.727443+0000","last_clean":"2024-09-18T07:31:49.727443+0000","last_became_active":"2024-09-18T07:31:02.353494+0000","last_became_peered":"2024-09-18T07:31:02.353494+0000","last_unstale":"2024-09-18T07:31:49.727443+0000","last_undegraded":"2024-09-18T07:31:49.727443+0000","last_fullsized":"2024-09-18T07:31:49.727443+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.436492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214151+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:31:50.214151+0000","last_peered":"2024-09-18T07:31:50.214151+0000","last_clean":"2024-09-18T07:31:50.214151+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:31:50.214151+0000","last_undegraded":"2024-09-18T07:31:50.214151+0000","last_fullsized":"2024-09-18T07:31:50.214151+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.419105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":52,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725843+0000","last_change":"2024-09-18T07:30:58.317041+0000","last_active":"2024-09-18T07:31:49.725843+0000","last_peered":"2024-09-18T07:31:49.725843+0000","last_clean":"2024-09-18T07:31:49.725843+0000","last_became_active":"2024-09-18T07:30:58.316479+0000","last_became_peered":"2024-09-18T07:30:58.316479+0000","last_unstale":"2024-09-18T07:31:49.725843+0000","last_undegraded":"2024-09-18T07:31:49.725843+0000","last_fullsized":"2024-09-18T07:31:49.725843+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.191081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213747+0000","last_change":"2024-09-18T07:31:00.334023+0000","last_active":"2024-09-18T07:31:50.213747+0000","last_peered":"2024-09-18T07:31:50.213747+0000","last_clean":"2024-09-18T07:31:50.213747+0000","last_became_active":"2024-09-18T07:31:00.333685+0000","last_became_peered":"2024-09-18T07:31:00.333685+0000","last_unstale":"2024-09-18T07:31:50.213747+0000","last_undegraded":"2024-09-18T07:31:50.213747+0000","last_fullsized":"2024-09-18T07:31:50.213747+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.774482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212600+0000","last_change":"2024-09-18T07:31:02.349964+0000","last_active":"2024-09-18T07:31:50.212600+0000","last_peered":"2024-09-18T07:31:50.212600+0000","last_clean":"2024-09-18T07:31:50.212600+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:31:50.212600+0000","last_undegraded":"2024-09-18T07:31:50.212600+0000","last_fullsized":"2024-09-18T07:31:50.212600+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.932342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726399+0000","last_change":"2024-09-18T07:31:02.490397+0000","last_active":"2024-09-18T07:31:49.726399+0000","last_peered":"2024-09-18T07:31:49.726399+0000","last_clean":"2024-09-18T07:31:49.726399+0000","last_became_active":"2024-09-18T07:31:02.490014+0000","last_became_peered":"2024-09-18T07:31:02.490014+0000","last_unstale":"2024-09-18T07:31:49.726399+0000","last_undegraded":"2024-09-18T07:31:49.726399+0000","last_fullsized":"2024-09-18T07:31:49.726399+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.792505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.729429+0000","last_change":"2024-09-18T07:30:56.288891+0000","last_active":"2024-09-18T07:31:49.729429+0000","last_peered":"2024-09-18T07:31:49.729429+0000","last_clean":"2024-09-18T07:31:49.729429+0000","last_became_active":"2024-09-18T07:30:56.288589+0000","last_became_peered":"2024-09-18T07:30:56.288589+0000","last_unstale":"2024-09-18T07:31:49.729429+0000","last_undegraded":"2024-09-18T07:31:49.729429+0000","last_fullsized":"2024-09-18T07:31:49.729429+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.974097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728460+0000","last_change":"2024-09-18T07:30:58.316556+0000","last_active":"2024-09-18T07:31:49.728460+0000","last_peered":"2024-09-18T07:31:49.728460+0000","last_clean":"2024-09-18T07:31:49.728460+0000","last_became_active":"2024-09-18T07:30:58.316266+0000","last_became_peered":"2024-09-18T07:30:58.316266+0000","last_unstale":"2024-09-18T07:31:49.728460+0000","last_undegraded":"2024-09-18T07:31:49.728460+0000","last_fullsized":"2024-09-18T07:31:49.728460+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:22.570802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213206+0000","last_change":"2024-09-18T07:31:00.338287+0000","last_active":"2024-09-18T07:31:50.213206+0000","last_peered":"2024-09-18T07:31:50.213206+0000","last_clean":"2024-09-18T07:31:50.213206+0000","last_became_active":"2024-09-18T07:31:00.337474+0000","last_became_peered":"2024-09-18T07:31:00.337474+0000","last_unstale":"2024-09-18T07:31:50.213206+0000","last_undegraded":"2024-09-18T07:31:50.213206+0000","last_fullsized":"2024-09-18T07:31:50.213206+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.609543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213754+0000","last_change":"2024-09-18T07:31:02.492790+0000","last_active":"2024-09-18T07:31:50.213754+0000","last_peered":"2024-09-18T07:31:50.213754+0000","last_clean":"2024-09-18T07:31:50.213754+0000","last_became_active":"2024-09-18T07:31:02.492348+0000","last_became_peered":"2024-09-18T07:31:02.492348+0000","last_unstale":"2024-09-18T07:31:50.213754+0000","last_undegraded":"2024-09-18T07:31:50.213754+0000","last_fullsized":"2024-09-18T07:31:50.213754+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:08.542586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726056+0000","last_change":"2024-09-18T07:30:56.289489+0000","last_active":"2024-09-18T07:31:49.726056+0000","last_peered":"2024-09-18T07:31:49.726056+0000","last_clean":"2024-09-18T07:31:49.726056+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:31:49.726056+0000","last_undegraded":"2024-09-18T07:31:49.726056+0000","last_fullsized":"2024-09-18T07:31:49.726056+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.331217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213566+0000","last_change":"2024-09-18T07:30:58.319115+0000","last_active":"2024-09-18T07:31:50.213566+0000","last_peered":"2024-09-18T07:31:50.213566+0000","last_clean":"2024-09-18T07:31:50.213566+0000","last_became_active":"2024-09-18T07:30:58.316380+0000","last_became_peered":"2024-09-18T07:30:58.316380+0000","last_unstale":"2024-09-18T07:31:50.213566+0000","last_undegraded":"2024-09-18T07:31:50.213566+0000","last_fullsized":"2024-09-18T07:31:50.213566+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.995221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726092+0000","last_change":"2024-09-18T07:31:00.334313+0000","last_active":"2024-09-18T07:31:49.726092+0000","last_peered":"2024-09-18T07:31:49.726092+0000","last_clean":"2024-09-18T07:31:49.726092+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:31:49.726092+0000","last_undegraded":"2024-09-18T07:31:49.726092+0000","last_fullsized":"2024-09-18T07:31:49.726092+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:09.373920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214016+0000","last_change":"2024-09-18T07:31:02.348855+0000","last_active":"2024-09-18T07:31:50.214016+0000","last_peered":"2024-09-18T07:31:50.214016+0000","last_clean":"2024-09-18T07:31:50.214016+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:31:50.214016+0000","last_undegraded":"2024-09-18T07:31:50.214016+0000","last_fullsized":"2024-09-18T07:31:50.214016+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.101140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212298+0000","last_change":"2024-09-18T07:30:56.283995+0000","last_active":"2024-09-18T07:31:50.212298+0000","last_peered":"2024-09-18T07:31:50.212298+0000","last_clean":"2024-09-18T07:31:50.212298+0000","last_became_active":"2024-09-18T07:30:56.283705+0000","last_became_peered":"2024-09-18T07:30:56.283705+0000","last_unstale":"2024-09-18T07:31:50.212298+0000","last_undegraded":"2024-09-18T07:31:50.212298+0000","last_fullsized":"2024-09-18T07:31:50.212298+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:55.938900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.729111+0000","last_change":"2024-09-18T07:30:58.318003+0000","last_active":"2024-09-18T07:31:49.729111+0000","last_peered":"2024-09-18T07:31:49.729111+0000","last_clean":"2024-09-18T07:31:49.729111+0000","last_became_active":"2024-09-18T07:30:58.317355+0000","last_became_peered":"2024-09-18T07:30:58.317355+0000","last_unstale":"2024-09-18T07:31:49.729111+0000","last_undegraded":"2024-09-18T07:31:49.729111+0000","last_fullsized":"2024-09-18T07:31:49.729111+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.431023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213749+0000","last_change":"2024-09-18T07:31:00.331318+0000","last_active":"2024-09-18T07:31:50.213749+0000","last_peered":"2024-09-18T07:31:50.213749+0000","last_clean":"2024-09-18T07:31:50.213749+0000","last_became_active":"2024-09-18T07:31:00.331014+0000","last_became_peered":"2024-09-18T07:31:00.331014+0000","last_unstale":"2024-09-18T07:31:50.213749+0000","last_undegraded":"2024-09-18T07:31:50.213749+0000","last_fullsized":"2024-09-18T07:31:50.213749+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.342355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212788+0000","last_change":"2024-09-18T07:30:58.312916+0000","last_active":"2024-09-18T07:31:50.212788+0000","last_peered":"2024-09-18T07:31:50.212788+0000","last_clean":"2024-09-18T07:31:50.212788+0000","last_became_active":"2024-09-18T07:30:58.312627+0000","last_became_peered":"2024-09-18T07:30:58.312627+0000","last_unstale":"2024-09-18T07:31:50.212788+0000","last_undegraded":"2024-09-18T07:31:50.212788+0000","last_fullsized":"2024-09-18T07:31:50.212788+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.188948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212555+0000","last_change":"2024-09-18T07:30:56.289331+0000","last_active":"2024-09-18T07:31:50.212555+0000","last_peered":"2024-09-18T07:31:50.212555+0000","last_clean":"2024-09-18T07:31:50.212555+0000","last_became_active":"2024-09-18T07:30:56.289081+0000","last_became_peered":"2024-09-18T07:30:56.289081+0000","last_unstale":"2024-09-18T07:31:50.212555+0000","last_undegraded":"2024-09-18T07:31:50.212555+0000","last_fullsized":"2024-09-18T07:31:50.212555+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:56.283698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726532+0000","last_change":"2024-09-18T07:31:00.333026+0000","last_active":"2024-09-18T07:31:49.726532+0000","last_peered":"2024-09-18T07:31:49.726532+0000","last_clean":"2024-09-18T07:31:49.726532+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:31:49.726532+0000","last_undegraded":"2024-09-18T07:31:49.726532+0000","last_fullsized":"2024-09-18T07:31:49.726532+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:09.647541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214053+0000","last_change":"2024-09-18T07:31:02.351412+0000","last_active":"2024-09-18T07:31:50.214053+0000","last_peered":"2024-09-18T07:31:50.214053+0000","last_clean":"2024-09-18T07:31:50.214053+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:31:50.214053+0000","last_undegraded":"2024-09-18T07:31:50.214053+0000","last_fullsized":"2024-09-18T07:31:50.214053+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.243657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":74,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212420+0000","last_change":"2024-09-18T07:30:58.321857+0000","last_active":"2024-09-18T07:31:50.212420+0000","last_peered":"2024-09-18T07:31:50.212420+0000","last_clean":"2024-09-18T07:31:50.212420+0000","last_became_active":"2024-09-18T07:30:58.321244+0000","last_became_peered":"2024-09-18T07:30:58.321244+0000","last_unstale":"2024-09-18T07:31:50.212420+0000","last_undegraded":"2024-09-18T07:31:50.212420+0000","last_fullsized":"2024-09-18T07:31:50.212420+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:58.899213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213919+0000","last_change":"2024-09-18T07:30:56.289862+0000","last_active":"2024-09-18T07:31:50.213919+0000","last_peered":"2024-09-18T07:31:50.213919+0000","last_clean":"2024-09-18T07:31:50.213919+0000","last_became_active":"2024-09-18T07:30:56.289556+0000","last_became_peered":"2024-09-18T07:30:56.289556+0000","last_unstale":"2024-09-18T07:31:50.213919+0000","last_undegraded":"2024-09-18T07:31:50.213919+0000","last_fullsized":"2024-09-18T07:31:50.213919+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:59.535940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725580+0000","last_change":"2024-09-18T07:31:00.333320+0000","last_active":"2024-09-18T07:31:49.725580+0000","last_peered":"2024-09-18T07:31:49.725580+0000","last_clean":"2024-09-18T07:31:49.725580+0000","last_became_active":"2024-09-18T07:31:00.332854+0000","last_became_peered":"2024-09-18T07:31:00.332854+0000","last_unstale":"2024-09-18T07:31:49.725580+0000","last_undegraded":"2024-09-18T07:31:49.725580+0000","last_fullsized":"2024-09-18T07:31:49.725580+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:20.691030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725889+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:31:49.725889+0000","last_peered":"2024-09-18T07:31:49.725889+0000","last_clean":"2024-09-18T07:31:49.725889+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:31:49.725889+0000","last_undegraded":"2024-09-18T07:31:49.725889+0000","last_fullsized":"2024-09-18T07:31:49.725889+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.039810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":63,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725593+0000","last_change":"2024-09-18T07:30:58.309938+0000","last_active":"2024-09-18T07:31:49.725593+0000","last_peered":"2024-09-18T07:31:49.725593+0000","last_clean":"2024-09-18T07:31:49.725593+0000","last_became_active":"2024-09-18T07:30:58.309649+0000","last_became_peered":"2024-09-18T07:30:58.309649+0000","last_unstale":"2024-09-18T07:31:49.725593+0000","last_undegraded":"2024-09-18T07:31:49.725593+0000","last_fullsized":"2024-09-18T07:31:49.725593+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:20.121712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.729501+0000","last_change":"2024-09-18T07:30:56.287909+0000","last_active":"2024-09-18T07:31:49.729501+0000","last_peered":"2024-09-18T07:31:49.729501+0000","last_clean":"2024-09-18T07:31:49.729501+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:31:49.729501+0000","last_undegraded":"2024-09-18T07:31:49.729501+0000","last_fullsized":"2024-09-18T07:31:49.729501+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:05.243933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212450+0000","last_change":"2024-09-18T07:31:00.336059+0000","last_active":"2024-09-18T07:31:50.212450+0000","last_peered":"2024-09-18T07:31:50.212450+0000","last_clean":"2024-09-18T07:31:50.212450+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:31:50.212450+0000","last_undegraded":"2024-09-18T07:31:50.212450+0000","last_fullsized":"2024-09-18T07:31:50.212450+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.046222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213223+0000","last_change":"2024-09-18T07:31:02.492797+0000","last_active":"2024-09-18T07:31:50.213223+0000","last_peered":"2024-09-18T07:31:50.213223+0000","last_clean":"2024-09-18T07:31:50.213223+0000","last_became_active":"2024-09-18T07:31:02.492483+0000","last_became_peered":"2024-09-18T07:31:02.492483+0000","last_unstale":"2024-09-18T07:31:50.213223+0000","last_undegraded":"2024-09-18T07:31:50.213223+0000","last_fullsized":"2024-09-18T07:31:50.213223+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.277226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212350+0000","last_change":"2024-09-18T07:30:58.307470+0000","last_active":"2024-09-18T07:31:50.212350+0000","last_peered":"2024-09-18T07:31:50.212350+0000","last_clean":"2024-09-18T07:31:50.212350+0000","last_became_active":"2024-09-18T07:30:58.307035+0000","last_became_peered":"2024-09-18T07:30:58.307035+0000","last_unstale":"2024-09-18T07:31:50.212350+0000","last_undegraded":"2024-09-18T07:31:50.212350+0000","last_fullsized":"2024-09-18T07:31:50.212350+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:06.964257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214484+0000","last_change":"2024-09-18T07:30:56.290040+0000","last_active":"2024-09-18T07:31:50.214484+0000","last_peered":"2024-09-18T07:31:50.214484+0000","last_clean":"2024-09-18T07:31:50.214484+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:31:50.214484+0000","last_undegraded":"2024-09-18T07:31:50.214484+0000","last_fullsized":"2024-09-18T07:31:50.214484+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.332869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.729169+0000","last_change":"2024-09-18T07:31:00.337241+0000","last_active":"2024-09-18T07:31:49.729169+0000","last_peered":"2024-09-18T07:31:49.729169+0000","last_clean":"2024-09-18T07:31:49.729169+0000","last_became_active":"2024-09-18T07:31:00.336905+0000","last_became_peered":"2024-09-18T07:31:00.336905+0000","last_unstale":"2024-09-18T07:31:49.729169+0000","last_undegraded":"2024-09-18T07:31:49.729169+0000","last_fullsized":"2024-09-18T07:31:49.729169+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.826427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.727559+0000","last_change":"2024-09-18T07:31:02.348538+0000","last_active":"2024-09-18T07:31:49.727559+0000","last_peered":"2024-09-18T07:31:49.727559+0000","last_clean":"2024-09-18T07:31:49.727559+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:31:49.727559+0000","last_undegraded":"2024-09-18T07:31:49.727559+0000","last_fullsized":"2024-09-18T07:31:49.727559+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.088828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":78,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.729264+0000","last_change":"2024-09-18T07:30:58.318096+0000","last_active":"2024-09-18T07:31:49.729264+0000","last_peered":"2024-09-18T07:31:49.729264+0000","last_clean":"2024-09-18T07:31:49.729264+0000","last_became_active":"2024-09-18T07:30:58.317767+0000","last_became_peered":"2024-09-18T07:30:58.317767+0000","last_unstale":"2024-09-18T07:31:49.729264+0000","last_undegraded":"2024-09-18T07:31:49.729264+0000","last_fullsized":"2024-09-18T07:31:49.729264+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:26.992100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212961+0000","last_change":"2024-09-18T07:30:56.284440+0000","last_active":"2024-09-18T07:31:50.212961+0000","last_peered":"2024-09-18T07:31:50.212961+0000","last_clean":"2024-09-18T07:31:50.212961+0000","last_became_active":"2024-09-18T07:30:56.284120+0000","last_became_peered":"2024-09-18T07:30:56.284120+0000","last_unstale":"2024-09-18T07:31:50.212961+0000","last_undegraded":"2024-09-18T07:31:50.212961+0000","last_fullsized":"2024-09-18T07:31:50.212961+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.025187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212835+0000","last_change":"2024-09-18T07:31:00.335100+0000","last_active":"2024-09-18T07:31:50.212835+0000","last_peered":"2024-09-18T07:31:50.212835+0000","last_clean":"2024-09-18T07:31:50.212835+0000","last_became_active":"2024-09-18T07:31:00.334776+0000","last_became_peered":"2024-09-18T07:31:00.334776+0000","last_unstale":"2024-09-18T07:31:50.212835+0000","last_undegraded":"2024-09-18T07:31:50.212835+0000","last_fullsized":"2024-09-18T07:31:50.212835+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.405293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212161+0000","last_change":"2024-09-18T07:31:02.490534+0000","last_active":"2024-09-18T07:31:50.212161+0000","last_peered":"2024-09-18T07:31:50.212161+0000","last_clean":"2024-09-18T07:31:50.212161+0000","last_became_active":"2024-09-18T07:31:02.490186+0000","last_became_peered":"2024-09-18T07:31:02.490186+0000","last_unstale":"2024-09-18T07:31:50.212161+0000","last_undegraded":"2024-09-18T07:31:50.212161+0000","last_fullsized":"2024-09-18T07:31:50.212161+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.881705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":102,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213384+0000","last_change":"2024-09-18T07:30:58.317477+0000","last_active":"2024-09-18T07:31:50.213384+0000","last_peered":"2024-09-18T07:31:50.213384+0000","last_clean":"2024-09-18T07:31:50.213384+0000","last_became_active":"2024-09-18T07:30:58.316812+0000","last_became_peered":"2024-09-18T07:30:58.316812+0000","last_unstale":"2024-09-18T07:31:50.213384+0000","last_undegraded":"2024-09-18T07:31:50.213384+0000","last_fullsized":"2024-09-18T07:31:50.213384+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.769678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726147+0000","last_change":"2024-09-18T07:30:56.285353+0000","last_active":"2024-09-18T07:31:49.726147+0000","last_peered":"2024-09-18T07:31:49.726147+0000","last_clean":"2024-09-18T07:31:49.726147+0000","last_became_active":"2024-09-18T07:30:56.285032+0000","last_became_peered":"2024-09-18T07:30:56.285032+0000","last_unstale":"2024-09-18T07:31:49.726147+0000","last_undegraded":"2024-09-18T07:31:49.726147+0000","last_fullsized":"2024-09-18T07:31:49.726147+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:23.187833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212993+0000","last_change":"2024-09-18T07:30:58.299308+0000","last_active":"2024-09-18T07:31:50.212993+0000","last_peered":"2024-09-18T07:31:50.212993+0000","last_clean":"2024-09-18T07:31:50.212993+0000","last_became_active":"2024-09-18T07:30:55.283292+0000","last_became_peered":"2024-09-18T07:30:55.283292+0000","last_unstale":"2024-09-18T07:31:50.212993+0000","last_undegraded":"2024-09-18T07:31:50.212993+0000","last_fullsized":"2024-09-18T07:31:50.212993+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:13.332916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728454+0000","last_change":"2024-09-18T07:31:00.332738+0000","last_active":"2024-09-18T07:31:49.728454+0000","last_peered":"2024-09-18T07:31:49.728454+0000","last_clean":"2024-09-18T07:31:49.728454+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:31:49.728454+0000","last_undegraded":"2024-09-18T07:31:49.728454+0000","last_fullsized":"2024-09-18T07:31:49.728454+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:23.578215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213578+0000","last_change":"2024-09-18T07:31:02.351778+0000","last_active":"2024-09-18T07:31:50.213578+0000","last_peered":"2024-09-18T07:31:50.213578+0000","last_clean":"2024-09-18T07:31:50.213578+0000","last_became_active":"2024-09-18T07:31:02.351403+0000","last_became_peered":"2024-09-18T07:31:02.351403+0000","last_unstale":"2024-09-18T07:31:50.213578+0000","last_undegraded":"2024-09-18T07:31:50.213578+0000","last_fullsized":"2024-09-18T07:31:50.213578+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:27.871124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":69,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212883+0000","last_change":"2024-09-18T07:30:58.310789+0000","last_active":"2024-09-18T07:31:50.212883+0000","last_peered":"2024-09-18T07:31:50.212883+0000","last_clean":"2024-09-18T07:31:50.212883+0000","last_became_active":"2024-09-18T07:30:58.310490+0000","last_became_peered":"2024-09-18T07:30:58.310490+0000","last_unstale":"2024-09-18T07:31:50.212883+0000","last_undegraded":"2024-09-18T07:31:50.212883+0000","last_fullsized":"2024-09-18T07:31:50.212883+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:22.258389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212394+0000","last_change":"2024-09-18T07:30:56.285054+0000","last_active":"2024-09-18T07:31:50.212394+0000","last_peered":"2024-09-18T07:31:50.212394+0000","last_clean":"2024-09-18T07:31:50.212394+0000","last_became_active":"2024-09-18T07:30:56.284803+0000","last_became_peered":"2024-09-18T07:30:56.284803+0000","last_unstale":"2024-09-18T07:31:50.212394+0000","last_undegraded":"2024-09-18T07:31:50.212394+0000","last_fullsized":"2024-09-18T07:31:50.212394+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.764106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.727693+0000","last_change":"2024-09-18T07:30:58.301149+0000","last_active":"2024-09-18T07:31:49.727693+0000","last_peered":"2024-09-18T07:31:49.727693+0000","last_clean":"2024-09-18T07:31:49.727693+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:31:49.727693+0000","last_undegraded":"2024-09-18T07:31:49.727693+0000","last_fullsized":"2024-09-18T07:31:49.727693+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:01.827803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728591+0000","last_change":"2024-09-18T07:31:00.331226+0000","last_active":"2024-09-18T07:31:49.728591+0000","last_peered":"2024-09-18T07:31:49.728591+0000","last_clean":"2024-09-18T07:31:49.728591+0000","last_became_active":"2024-09-18T07:31:00.330883+0000","last_became_peered":"2024-09-18T07:31:00.330883+0000","last_unstale":"2024-09-18T07:31:49.728591+0000","last_undegraded":"2024-09-18T07:31:49.728591+0000","last_fullsized":"2024-09-18T07:31:49.728591+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.695331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213817+0000","last_change":"2024-09-18T07:31:02.491512+0000","last_active":"2024-09-18T07:31:50.213817+0000","last_peered":"2024-09-18T07:31:50.213817+0000","last_clean":"2024-09-18T07:31:50.213817+0000","last_became_active":"2024-09-18T07:31:02.491233+0000","last_became_peered":"2024-09-18T07:31:02.491233+0000","last_unstale":"2024-09-18T07:31:50.213817+0000","last_undegraded":"2024-09-18T07:31:50.213817+0000","last_fullsized":"2024-09-18T07:31:50.213817+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:26.099794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.729187+0000","last_change":"2024-09-18T07:30:58.311778+0000","last_active":"2024-09-18T07:31:49.729187+0000","last_peered":"2024-09-18T07:31:49.729187+0000","last_clean":"2024-09-18T07:31:49.729187+0000","last_became_active":"2024-09-18T07:30:58.311491+0000","last_became_peered":"2024-09-18T07:30:58.311491+0000","last_unstale":"2024-09-18T07:31:49.729187+0000","last_undegraded":"2024-09-18T07:31:49.729187+0000","last_fullsized":"2024-09-18T07:31:49.729187+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:05.875741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":52,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728777+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:31:49.728777+0000","last_peered":"2024-09-18T07:31:49.728777+0000","last_clean":"2024-09-18T07:31:49.728777+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:31:49.728777+0000","last_undegraded":"2024-09-18T07:31:49.728777+0000","last_fullsized":"2024-09-18T07:31:49.728777+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:20.989190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":107,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214817+0000","last_change":"2024-09-18T07:30:58.319969+0000","last_active":"2024-09-18T07:31:50.214817+0000","last_peered":"2024-09-18T07:31:50.214817+0000","last_clean":"2024-09-18T07:31:50.214817+0000","last_became_active":"2024-09-18T07:30:55.286980+0000","last_became_peered":"2024-09-18T07:30:55.286980+0000","last_unstale":"2024-09-18T07:31:50.214817+0000","last_undegraded":"2024-09-18T07:31:50.214817+0000","last_fullsized":"2024-09-18T07:31:50.214817+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.742429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":50,"num_read_kb":45,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.215030+0000","last_change":"2024-09-18T07:31:00.331929+0000","last_active":"2024-09-18T07:31:50.215030+0000","last_peered":"2024-09-18T07:31:50.215030+0000","last_clean":"2024-09-18T07:31:50.215030+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:31:50.215030+0000","last_undegraded":"2024-09-18T07:31:50.215030+0000","last_fullsized":"2024-09-18T07:31:50.215030+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.504554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212644+0000","last_change":"2024-09-18T07:31:02.347659+0000","last_active":"2024-09-18T07:31:50.212644+0000","last_peered":"2024-09-18T07:31:50.212644+0000","last_clean":"2024-09-18T07:31:50.212644+0000","last_became_active":"2024-09-18T07:31:02.347390+0000","last_became_peered":"2024-09-18T07:31:02.347390+0000","last_unstale":"2024-09-18T07:31:50.212644+0000","last_undegraded":"2024-09-18T07:31:50.212644+0000","last_fullsized":"2024-09-18T07:31:50.212644+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.353794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":76,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213473+0000","last_change":"2024-09-18T07:30:58.318426+0000","last_active":"2024-09-18T07:31:50.213473+0000","last_peered":"2024-09-18T07:31:50.213473+0000","last_clean":"2024-09-18T07:31:50.213473+0000","last_became_active":"2024-09-18T07:30:58.318068+0000","last_became_peered":"2024-09-18T07:30:58.318068+0000","last_unstale":"2024-09-18T07:31:50.213473+0000","last_undegraded":"2024-09-18T07:31:50.213473+0000","last_fullsized":"2024-09-18T07:31:50.213473+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:59.863882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213362+0000","last_change":"2024-09-18T07:30:56.288420+0000","last_active":"2024-09-18T07:31:50.213362+0000","last_peered":"2024-09-18T07:31:50.213362+0000","last_clean":"2024-09-18T07:31:50.213362+0000","last_became_active":"2024-09-18T07:30:56.288149+0000","last_became_peered":"2024-09-18T07:30:56.288149+0000","last_unstale":"2024-09-18T07:31:50.213362+0000","last_undegraded":"2024-09-18T07:31:50.213362+0000","last_fullsized":"2024-09-18T07:31:50.213362+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:56.385307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":93,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214868+0000","last_change":"2024-09-18T07:30:50.813696+0000","last_active":"2024-09-18T07:31:50.214868+0000","last_peered":"2024-09-18T07:31:50.214868+0000","last_clean":"2024-09-18T07:31:50.214868+0000","last_became_active":"2024-09-18T07:30:46.719588+0000","last_became_peered":"2024-09-18T07:30:46.719588+0000","last_unstale":"2024-09-18T07:31:50.214868+0000","last_undegraded":"2024-09-18T07:31:50.214868+0000","last_fullsized":"2024-09-18T07:31:50.214868+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-18T07:30:50.813645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:30:50.813645+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:06.405856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214166+0000","last_change":"2024-09-18T07:31:00.331602+0000","last_active":"2024-09-18T07:31:50.214166+0000","last_peered":"2024-09-18T07:31:50.214166+0000","last_clean":"2024-09-18T07:31:50.214166+0000","last_became_active":"2024-09-18T07:31:00.331327+0000","last_became_peered":"2024-09-18T07:31:00.331327+0000","last_unstale":"2024-09-18T07:31:50.214166+0000","last_undegraded":"2024-09-18T07:31:50.214166+0000","last_fullsized":"2024-09-18T07:31:50.214166+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.402435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214934+0000","last_change":"2024-09-18T07:31:02.348225+0000","last_active":"2024-09-18T07:31:50.214934+0000","last_peered":"2024-09-18T07:31:50.214934+0000","last_clean":"2024-09-18T07:31:50.214934+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:31:50.214934+0000","last_undegraded":"2024-09-18T07:31:50.214934+0000","last_fullsized":"2024-09-18T07:31:50.214934+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.749034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726559+0000","last_change":"2024-09-18T07:30:58.313677+0000","last_active":"2024-09-18T07:31:49.726559+0000","last_peered":"2024-09-18T07:31:49.726559+0000","last_clean":"2024-09-18T07:31:49.726559+0000","last_became_active":"2024-09-18T07:30:58.313382+0000","last_became_peered":"2024-09-18T07:30:58.313382+0000","last_unstale":"2024-09-18T07:31:49.726559+0000","last_undegraded":"2024-09-18T07:31:49.726559+0000","last_fullsized":"2024-09-18T07:31:49.726559+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.250207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725860+0000","last_change":"2024-09-18T07:30:56.291117+0000","last_active":"2024-09-18T07:31:49.725860+0000","last_peered":"2024-09-18T07:31:49.725860+0000","last_clean":"2024-09-18T07:31:49.725860+0000","last_became_active":"2024-09-18T07:30:56.290836+0000","last_became_peered":"2024-09-18T07:30:56.290836+0000","last_unstale":"2024-09-18T07:31:49.725860+0000","last_undegraded":"2024-09-18T07:31:49.725860+0000","last_fullsized":"2024-09-18T07:31:49.725860+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.125102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214586+0000","last_change":"2024-09-18T07:31:00.336459+0000","last_active":"2024-09-18T07:31:50.214586+0000","last_peered":"2024-09-18T07:31:50.214586+0000","last_clean":"2024-09-18T07:31:50.214586+0000","last_became_active":"2024-09-18T07:31:00.336172+0000","last_became_peered":"2024-09-18T07:31:00.336172+0000","last_unstale":"2024-09-18T07:31:50.214586+0000","last_undegraded":"2024-09-18T07:31:50.214586+0000","last_fullsized":"2024-09-18T07:31:50.214586+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:28.160928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213176+0000","last_change":"2024-09-18T07:31:02.491383+0000","last_active":"2024-09-18T07:31:50.213176+0000","last_peered":"2024-09-18T07:31:50.213176+0000","last_clean":"2024-09-18T07:31:50.213176+0000","last_became_active":"2024-09-18T07:31:02.490953+0000","last_became_peered":"2024-09-18T07:31:02.490953+0000","last_unstale":"2024-09-18T07:31:50.213176+0000","last_undegraded":"2024-09-18T07:31:50.213176+0000","last_fullsized":"2024-09-18T07:31:50.213176+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.829440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":70,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726449+0000","last_change":"2024-09-18T07:30:58.312636+0000","last_active":"2024-09-18T07:31:49.726449+0000","last_peered":"2024-09-18T07:31:49.726449+0000","last_clean":"2024-09-18T07:31:49.726449+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:31:49.726449+0000","last_undegraded":"2024-09-18T07:31:49.726449+0000","last_fullsized":"2024-09-18T07:31:49.726449+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:13.709878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213218+0000","last_change":"2024-09-18T07:30:56.284303+0000","last_active":"2024-09-18T07:31:50.213218+0000","last_peered":"2024-09-18T07:31:50.213218+0000","last_clean":"2024-09-18T07:31:50.213218+0000","last_became_active":"2024-09-18T07:30:56.284020+0000","last_became_peered":"2024-09-18T07:30:56.284020+0000","last_unstale":"2024-09-18T07:31:50.213218+0000","last_undegraded":"2024-09-18T07:31:50.213218+0000","last_fullsized":"2024-09-18T07:31:50.213218+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:15.982099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212742+0000","last_change":"2024-09-18T07:31:00.333155+0000","last_active":"2024-09-18T07:31:50.212742+0000","last_peered":"2024-09-18T07:31:50.212742+0000","last_clean":"2024-09-18T07:31:50.212742+0000","last_became_active":"2024-09-18T07:31:00.332836+0000","last_became_peered":"2024-09-18T07:31:00.332836+0000","last_unstale":"2024-09-18T07:31:50.212742+0000","last_undegraded":"2024-09-18T07:31:50.212742+0000","last_fullsized":"2024-09-18T07:31:50.212742+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.946427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728567+0000","last_change":"2024-09-18T07:31:02.347665+0000","last_active":"2024-09-18T07:31:49.728567+0000","last_peered":"2024-09-18T07:31:49.728567+0000","last_clean":"2024-09-18T07:31:49.728567+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:31:49.728567+0000","last_undegraded":"2024-09-18T07:31:49.728567+0000","last_fullsized":"2024-09-18T07:31:49.728567+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.617366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726141+0000","last_change":"2024-09-18T07:30:58.317008+0000","last_active":"2024-09-18T07:31:49.726141+0000","last_peered":"2024-09-18T07:31:49.726141+0000","last_clean":"2024-09-18T07:31:49.726141+0000","last_became_active":"2024-09-18T07:30:58.316423+0000","last_became_peered":"2024-09-18T07:30:58.316423+0000","last_unstale":"2024-09-18T07:31:49.726141+0000","last_undegraded":"2024-09-18T07:31:49.726141+0000","last_fullsized":"2024-09-18T07:31:49.726141+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.580907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213156+0000","last_change":"2024-09-18T07:30:56.287284+0000","last_active":"2024-09-18T07:31:50.213156+0000","last_peered":"2024-09-18T07:31:50.213156+0000","last_clean":"2024-09-18T07:31:50.213156+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:31:50.213156+0000","last_undegraded":"2024-09-18T07:31:50.213156+0000","last_fullsized":"2024-09-18T07:31:50.213156+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:00.720049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728127+0000","last_change":"2024-09-18T07:31:00.333376+0000","last_active":"2024-09-18T07:31:49.728127+0000","last_peered":"2024-09-18T07:31:49.728127+0000","last_clean":"2024-09-18T07:31:49.728127+0000","last_became_active":"2024-09-18T07:31:00.333085+0000","last_became_peered":"2024-09-18T07:31:00.333085+0000","last_unstale":"2024-09-18T07:31:49.728127+0000","last_undegraded":"2024-09-18T07:31:49.728127+0000","last_fullsized":"2024-09-18T07:31:49.728127+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:22.590817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726384+0000","last_change":"2024-09-18T07:31:02.347721+0000","last_active":"2024-09-18T07:31:49.726384+0000","last_peered":"2024-09-18T07:31:49.726384+0000","last_clean":"2024-09-18T07:31:49.726384+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:31:49.726384+0000","last_undegraded":"2024-09-18T07:31:49.726384+0000","last_fullsized":"2024-09-18T07:31:49.726384+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.723859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.729220+0000","last_change":"2024-09-18T07:30:58.314424+0000","last_active":"2024-09-18T07:31:49.729220+0000","last_peered":"2024-09-18T07:31:49.729220+0000","last_clean":"2024-09-18T07:31:49.729220+0000","last_became_active":"2024-09-18T07:30:58.314127+0000","last_became_peered":"2024-09-18T07:30:58.314127+0000","last_unstale":"2024-09-18T07:31:49.729220+0000","last_undegraded":"2024-09-18T07:31:49.729220+0000","last_fullsized":"2024-09-18T07:31:49.729220+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:13.636020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214284+0000","last_change":"2024-09-18T07:30:56.289177+0000","last_active":"2024-09-18T07:31:50.214284+0000","last_peered":"2024-09-18T07:31:50.214284+0000","last_clean":"2024-09-18T07:31:50.214284+0000","last_became_active":"2024-09-18T07:30:56.288868+0000","last_became_peered":"2024-09-18T07:30:56.288868+0000","last_unstale":"2024-09-18T07:31:50.214284+0000","last_undegraded":"2024-09-18T07:31:50.214284+0000","last_fullsized":"2024-09-18T07:31:50.214284+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.629409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.727916+0000","last_change":"2024-09-18T07:31:00.336020+0000","last_active":"2024-09-18T07:31:49.727916+0000","last_peered":"2024-09-18T07:31:49.727916+0000","last_clean":"2024-09-18T07:31:49.727916+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:31:49.727916+0000","last_undegraded":"2024-09-18T07:31:49.727916+0000","last_fullsized":"2024-09-18T07:31:49.727916+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:28.112892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.727371+0000","last_change":"2024-09-18T07:31:02.349757+0000","last_active":"2024-09-18T07:31:49.727371+0000","last_peered":"2024-09-18T07:31:49.727371+0000","last_clean":"2024-09-18T07:31:49.727371+0000","last_became_active":"2024-09-18T07:31:02.349454+0000","last_became_peered":"2024-09-18T07:31:02.349454+0000","last_unstale":"2024-09-18T07:31:49.727371+0000","last_undegraded":"2024-09-18T07:31:49.727371+0000","last_fullsized":"2024-09-18T07:31:49.727371+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.874882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":73,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212932+0000","last_change":"2024-09-18T07:30:58.318421+0000","last_active":"2024-09-18T07:31:50.212932+0000","last_peered":"2024-09-18T07:31:50.212932+0000","last_clean":"2024-09-18T07:31:50.212932+0000","last_became_active":"2024-09-18T07:30:58.318063+0000","last_became_peered":"2024-09-18T07:30:58.318063+0000","last_unstale":"2024-09-18T07:31:50.212932+0000","last_undegraded":"2024-09-18T07:31:50.212932+0000","last_fullsized":"2024-09-18T07:31:50.212932+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:59.591866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214181+0000","last_change":"2024-09-18T07:30:56.293678+0000","last_active":"2024-09-18T07:31:50.214181+0000","last_peered":"2024-09-18T07:31:50.214181+0000","last_clean":"2024-09-18T07:31:50.214181+0000","last_became_active":"2024-09-18T07:30:56.293371+0000","last_became_peered":"2024-09-18T07:30:56.293371+0000","last_unstale":"2024-09-18T07:31:50.214181+0000","last_undegraded":"2024-09-18T07:31:50.214181+0000","last_fullsized":"2024-09-18T07:31:50.214181+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.829476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728083+0000","last_change":"2024-09-18T07:31:00.332322+0000","last_active":"2024-09-18T07:31:49.728083+0000","last_peered":"2024-09-18T07:31:49.728083+0000","last_clean":"2024-09-18T07:31:49.728083+0000","last_became_active":"2024-09-18T07:31:00.331914+0000","last_became_peered":"2024-09-18T07:31:00.331914+0000","last_unstale":"2024-09-18T07:31:49.728083+0000","last_undegraded":"2024-09-18T07:31:49.728083+0000","last_fullsized":"2024-09-18T07:31:49.728083+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.543866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214811+0000","last_change":"2024-09-18T07:31:02.351941+0000","last_active":"2024-09-18T07:31:50.214811+0000","last_peered":"2024-09-18T07:31:50.214811+0000","last_clean":"2024-09-18T07:31:50.214811+0000","last_became_active":"2024-09-18T07:31:02.351678+0000","last_became_peered":"2024-09-18T07:31:02.351678+0000","last_unstale":"2024-09-18T07:31:50.214811+0000","last_undegraded":"2024-09-18T07:31:50.214811+0000","last_fullsized":"2024-09-18T07:31:50.214811+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.816291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725733+0000","last_change":"2024-09-18T07:30:58.320305+0000","last_active":"2024-09-18T07:31:49.725733+0000","last_peered":"2024-09-18T07:31:49.725733+0000","last_clean":"2024-09-18T07:31:49.725733+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:31:49.725733+0000","last_undegraded":"2024-09-18T07:31:49.725733+0000","last_fullsized":"2024-09-18T07:31:49.725733+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.490659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213738+0000","last_change":"2024-09-18T07:30:56.290367+0000","last_active":"2024-09-18T07:31:50.213738+0000","last_peered":"2024-09-18T07:31:50.213738+0000","last_clean":"2024-09-18T07:31:50.213738+0000","last_became_active":"2024-09-18T07:30:56.289855+0000","last_became_peered":"2024-09-18T07:30:56.289855+0000","last_unstale":"2024-09-18T07:31:50.213738+0000","last_undegraded":"2024-09-18T07:31:50.213738+0000","last_fullsized":"2024-09-18T07:31:50.213738+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.319326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.727869+0000","last_change":"2024-09-18T07:31:00.329175+0000","last_active":"2024-09-18T07:31:49.727869+0000","last_peered":"2024-09-18T07:31:49.727869+0000","last_clean":"2024-09-18T07:31:49.727869+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:31:49.727869+0000","last_undegraded":"2024-09-18T07:31:49.727869+0000","last_fullsized":"2024-09-18T07:31:49.727869+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.309875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212847+0000","last_change":"2024-09-18T07:31:02.353267+0000","last_active":"2024-09-18T07:31:50.212847+0000","last_peered":"2024-09-18T07:31:50.212847+0000","last_clean":"2024-09-18T07:31:50.212847+0000","last_became_active":"2024-09-18T07:31:02.352922+0000","last_became_peered":"2024-09-18T07:31:02.352922+0000","last_unstale":"2024-09-18T07:31:50.212847+0000","last_undegraded":"2024-09-18T07:31:50.212847+0000","last_fullsized":"2024-09-18T07:31:50.212847+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:26.271964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212469+0000","last_change":"2024-09-18T07:30:58.319585+0000","last_active":"2024-09-18T07:31:50.212469+0000","last_peered":"2024-09-18T07:31:50.212469+0000","last_clean":"2024-09-18T07:31:50.212469+0000","last_became_active":"2024-09-18T07:30:58.319300+0000","last_became_peered":"2024-09-18T07:30:58.319300+0000","last_unstale":"2024-09-18T07:31:50.212469+0000","last_undegraded":"2024-09-18T07:31:50.212469+0000","last_fullsized":"2024-09-18T07:31:50.212469+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.143713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214578+0000","last_change":"2024-09-18T07:30:56.292832+0000","last_active":"2024-09-18T07:31:50.214578+0000","last_peered":"2024-09-18T07:31:50.214578+0000","last_clean":"2024-09-18T07:31:50.214578+0000","last_became_active":"2024-09-18T07:30:56.292531+0000","last_became_peered":"2024-09-18T07:30:56.292531+0000","last_unstale":"2024-09-18T07:31:50.214578+0000","last_undegraded":"2024-09-18T07:31:50.214578+0000","last_fullsized":"2024-09-18T07:31:50.214578+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.359238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213404+0000","last_change":"2024-09-18T07:31:00.337915+0000","last_active":"2024-09-18T07:31:50.213404+0000","last_peered":"2024-09-18T07:31:50.213404+0000","last_clean":"2024-09-18T07:31:50.213404+0000","last_became_active":"2024-09-18T07:31:00.337483+0000","last_became_peered":"2024-09-18T07:31:00.337483+0000","last_unstale":"2024-09-18T07:31:50.213404+0000","last_undegraded":"2024-09-18T07:31:50.213404+0000","last_fullsized":"2024-09-18T07:31:50.213404+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:20.556343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212683+0000","last_change":"2024-09-18T07:31:02.489914+0000","last_active":"2024-09-18T07:31:50.212683+0000","last_peered":"2024-09-18T07:31:50.212683+0000","last_clean":"2024-09-18T07:31:50.212683+0000","last_became_active":"2024-09-18T07:31:02.489619+0000","last_became_peered":"2024-09-18T07:31:02.489619+0000","last_unstale":"2024-09-18T07:31:50.212683+0000","last_undegraded":"2024-09-18T07:31:50.212683+0000","last_fullsized":"2024-09-18T07:31:50.212683+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:17.709609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213098+0000","last_change":"2024-09-18T07:30:56.282629+0000","last_active":"2024-09-18T07:31:50.213098+0000","last_peered":"2024-09-18T07:31:50.213098+0000","last_clean":"2024-09-18T07:31:50.213098+0000","last_became_active":"2024-09-18T07:30:56.282357+0000","last_became_peered":"2024-09-18T07:30:56.282357+0000","last_unstale":"2024-09-18T07:31:50.213098+0000","last_undegraded":"2024-09-18T07:31:50.213098+0000","last_fullsized":"2024-09-18T07:31:50.213098+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.549682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":51,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213873+0000","last_change":"2024-09-18T07:30:58.320781+0000","last_active":"2024-09-18T07:31:50.213873+0000","last_peered":"2024-09-18T07:31:50.213873+0000","last_clean":"2024-09-18T07:31:50.213873+0000","last_became_active":"2024-09-18T07:30:58.320563+0000","last_became_peered":"2024-09-18T07:30:58.320563+0000","last_unstale":"2024-09-18T07:31:50.213873+0000","last_undegraded":"2024-09-18T07:31:50.213873+0000","last_fullsized":"2024-09-18T07:31:50.213873+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.842501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214119+0000","last_change":"2024-09-18T07:31:00.334155+0000","last_active":"2024-09-18T07:31:50.214119+0000","last_peered":"2024-09-18T07:31:50.214119+0000","last_clean":"2024-09-18T07:31:50.214119+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:31:50.214119+0000","last_undegraded":"2024-09-18T07:31:50.214119+0000","last_fullsized":"2024-09-18T07:31:50.214119+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.652219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213731+0000","last_change":"2024-09-18T07:31:02.492999+0000","last_active":"2024-09-18T07:31:50.213731+0000","last_peered":"2024-09-18T07:31:50.213731+0000","last_clean":"2024-09-18T07:31:50.213731+0000","last_became_active":"2024-09-18T07:31:02.492489+0000","last_became_peered":"2024-09-18T07:31:02.492489+0000","last_unstale":"2024-09-18T07:31:50.213731+0000","last_undegraded":"2024-09-18T07:31:50.213731+0000","last_fullsized":"2024-09-18T07:31:50.213731+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.972057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.729302+0000","last_change":"2024-09-18T07:30:58.318263+0000","last_active":"2024-09-18T07:31:49.729302+0000","last_peered":"2024-09-18T07:31:49.729302+0000","last_clean":"2024-09-18T07:31:49.729302+0000","last_became_active":"2024-09-18T07:30:58.317732+0000","last_became_peered":"2024-09-18T07:30:58.317732+0000","last_unstale":"2024-09-18T07:31:49.729302+0000","last_undegraded":"2024-09-18T07:31:49.729302+0000","last_fullsized":"2024-09-18T07:31:49.729302+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.024721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213564+0000","last_change":"2024-09-18T07:30:56.292272+0000","last_active":"2024-09-18T07:31:50.213564+0000","last_peered":"2024-09-18T07:31:50.213564+0000","last_clean":"2024-09-18T07:31:50.213564+0000","last_became_active":"2024-09-18T07:30:56.290788+0000","last_became_peered":"2024-09-18T07:30:56.290788+0000","last_unstale":"2024-09-18T07:31:50.213564+0000","last_undegraded":"2024-09-18T07:31:50.213564+0000","last_fullsized":"2024-09-18T07:31:50.213564+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.447855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213829+0000","last_change":"2024-09-18T07:31:00.334848+0000","last_active":"2024-09-18T07:31:50.213829+0000","last_peered":"2024-09-18T07:31:50.213829+0000","last_clean":"2024-09-18T07:31:50.213829+0000","last_became_active":"2024-09-18T07:31:00.334478+0000","last_became_peered":"2024-09-18T07:31:00.334478+0000","last_unstale":"2024-09-18T07:31:50.213829+0000","last_undegraded":"2024-09-18T07:31:50.213829+0000","last_fullsized":"2024-09-18T07:31:50.213829+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.087953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728225+0000","last_change":"2024-09-18T07:31:02.344766+0000","last_active":"2024-09-18T07:31:49.728225+0000","last_peered":"2024-09-18T07:31:49.728225+0000","last_clean":"2024-09-18T07:31:49.728225+0000","last_became_active":"2024-09-18T07:31:02.344011+0000","last_became_peered":"2024-09-18T07:31:02.344011+0000","last_unstale":"2024-09-18T07:31:49.728225+0000","last_undegraded":"2024-09-18T07:31:49.728225+0000","last_fullsized":"2024-09-18T07:31:49.728225+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.862393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214407+0000","last_change":"2024-09-18T07:30:58.318363+0000","last_active":"2024-09-18T07:31:50.214407+0000","last_peered":"2024-09-18T07:31:50.214407+0000","last_clean":"2024-09-18T07:31:50.214407+0000","last_became_active":"2024-09-18T07:30:58.318057+0000","last_became_peered":"2024-09-18T07:30:58.318057+0000","last_unstale":"2024-09-18T07:31:50.214407+0000","last_undegraded":"2024-09-18T07:31:50.214407+0000","last_fullsized":"2024-09-18T07:31:50.214407+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.599726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728179+0000","last_change":"2024-09-18T07:30:56.287385+0000","last_active":"2024-09-18T07:31:49.728179+0000","last_peered":"2024-09-18T07:31:49.728179+0000","last_clean":"2024-09-18T07:31:49.728179+0000","last_became_active":"2024-09-18T07:30:56.287068+0000","last_became_peered":"2024-09-18T07:30:56.287068+0000","last_unstale":"2024-09-18T07:31:49.728179+0000","last_undegraded":"2024-09-18T07:31:49.728179+0000","last_fullsized":"2024-09-18T07:31:49.728179+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.184849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213435+0000","last_change":"2024-09-18T07:31:00.337716+0000","last_active":"2024-09-18T07:31:50.213435+0000","last_peered":"2024-09-18T07:31:50.213435+0000","last_clean":"2024-09-18T07:31:50.213435+0000","last_became_active":"2024-09-18T07:31:00.337444+0000","last_became_peered":"2024-09-18T07:31:00.337444+0000","last_unstale":"2024-09-18T07:31:50.213435+0000","last_undegraded":"2024-09-18T07:31:50.213435+0000","last_fullsized":"2024-09-18T07:31:50.213435+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.512126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726637+0000","last_change":"2024-09-18T07:31:02.343206+0000","last_active":"2024-09-18T07:31:49.726637+0000","last_peered":"2024-09-18T07:31:49.726637+0000","last_clean":"2024-09-18T07:31:49.726637+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:31:49.726637+0000","last_undegraded":"2024-09-18T07:31:49.726637+0000","last_fullsized":"2024-09-18T07:31:49.726637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:23.329902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213498+0000","last_change":"2024-09-18T07:30:58.320351+0000","last_active":"2024-09-18T07:31:50.213498+0000","last_peered":"2024-09-18T07:31:50.213498+0000","last_clean":"2024-09-18T07:31:50.213498+0000","last_became_active":"2024-09-18T07:30:58.320093+0000","last_became_peered":"2024-09-18T07:30:58.320093+0000","last_unstale":"2024-09-18T07:31:50.213498+0000","last_undegraded":"2024-09-18T07:31:50.213498+0000","last_fullsized":"2024-09-18T07:31:50.213498+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.687274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213549+0000","last_change":"2024-09-18T07:30:56.291807+0000","last_active":"2024-09-18T07:31:50.213549+0000","last_peered":"2024-09-18T07:31:50.213549+0000","last_clean":"2024-09-18T07:31:50.213549+0000","last_became_active":"2024-09-18T07:30:56.291238+0000","last_became_peered":"2024-09-18T07:30:56.291238+0000","last_unstale":"2024-09-18T07:31:50.213549+0000","last_undegraded":"2024-09-18T07:31:50.213549+0000","last_fullsized":"2024-09-18T07:31:50.213549+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:17.785800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214349+0000","last_change":"2024-09-18T07:31:00.332729+0000","last_active":"2024-09-18T07:31:50.214349+0000","last_peered":"2024-09-18T07:31:50.214349+0000","last_clean":"2024-09-18T07:31:50.214349+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:31:50.214349+0000","last_undegraded":"2024-09-18T07:31:50.214349+0000","last_fullsized":"2024-09-18T07:31:50.214349+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:08.863683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728909+0000","last_change":"2024-09-18T07:31:02.348726+0000","last_active":"2024-09-18T07:31:49.728909+0000","last_peered":"2024-09-18T07:31:49.728909+0000","last_clean":"2024-09-18T07:31:49.728909+0000","last_became_active":"2024-09-18T07:31:02.348423+0000","last_became_peered":"2024-09-18T07:31:02.348423+0000","last_unstale":"2024-09-18T07:31:49.728909+0000","last_undegraded":"2024-09-18T07:31:49.728909+0000","last_fullsized":"2024-09-18T07:31:49.728909+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.760836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725914+0000","last_change":"2024-09-18T07:30:58.314730+0000","last_active":"2024-09-18T07:31:49.725914+0000","last_peered":"2024-09-18T07:31:49.725914+0000","last_clean":"2024-09-18T07:31:49.725914+0000","last_became_active":"2024-09-18T07:30:58.314428+0000","last_became_peered":"2024-09-18T07:30:58.314428+0000","last_unstale":"2024-09-18T07:31:49.725914+0000","last_undegraded":"2024-09-18T07:31:49.725914+0000","last_fullsized":"2024-09-18T07:31:49.725914+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.641591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726270+0000","last_change":"2024-09-18T07:30:56.291115+0000","last_active":"2024-09-18T07:31:49.726270+0000","last_peered":"2024-09-18T07:31:49.726270+0000","last_clean":"2024-09-18T07:31:49.726270+0000","last_became_active":"2024-09-18T07:30:56.290816+0000","last_became_peered":"2024-09-18T07:30:56.290816+0000","last_unstale":"2024-09-18T07:31:49.726270+0000","last_undegraded":"2024-09-18T07:31:49.726270+0000","last_fullsized":"2024-09-18T07:31:49.726270+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.554722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212261+0000","last_change":"2024-09-18T07:31:00.337826+0000","last_active":"2024-09-18T07:31:50.212261+0000","last_peered":"2024-09-18T07:31:50.212261+0000","last_clean":"2024-09-18T07:31:50.212261+0000","last_became_active":"2024-09-18T07:31:00.334158+0000","last_became_peered":"2024-09-18T07:31:00.334158+0000","last_unstale":"2024-09-18T07:31:50.212261+0000","last_undegraded":"2024-09-18T07:31:50.212261+0000","last_fullsized":"2024-09-18T07:31:50.212261+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:28.224028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212433+0000","last_change":"2024-09-18T07:31:02.351247+0000","last_active":"2024-09-18T07:31:50.212433+0000","last_peered":"2024-09-18T07:31:50.212433+0000","last_clean":"2024-09-18T07:31:50.212433+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:31:50.212433+0000","last_undegraded":"2024-09-18T07:31:50.212433+0000","last_fullsized":"2024-09-18T07:31:50.212433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:25.935005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212555+0000","last_change":"2024-09-18T07:30:58.307904+0000","last_active":"2024-09-18T07:31:50.212555+0000","last_peered":"2024-09-18T07:31:50.212555+0000","last_clean":"2024-09-18T07:31:50.212555+0000","last_became_active":"2024-09-18T07:30:58.307448+0000","last_became_peered":"2024-09-18T07:30:58.307448+0000","last_unstale":"2024-09-18T07:31:50.212555+0000","last_undegraded":"2024-09-18T07:31:50.212555+0000","last_fullsized":"2024-09-18T07:31:50.212555+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:01.900229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214035+0000","last_change":"2024-09-18T07:30:56.293467+0000","last_active":"2024-09-18T07:31:50.214035+0000","last_peered":"2024-09-18T07:31:50.214035+0000","last_clean":"2024-09-18T07:31:50.214035+0000","last_became_active":"2024-09-18T07:30:56.292980+0000","last_became_peered":"2024-09-18T07:30:56.292980+0000","last_unstale":"2024-09-18T07:31:50.214035+0000","last_undegraded":"2024-09-18T07:31:50.214035+0000","last_fullsized":"2024-09-18T07:31:50.214035+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.161233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212934+0000","last_change":"2024-09-18T07:31:00.333689+0000","last_active":"2024-09-18T07:31:50.212934+0000","last_peered":"2024-09-18T07:31:50.212934+0000","last_clean":"2024-09-18T07:31:50.212934+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:31:50.212934+0000","last_undegraded":"2024-09-18T07:31:50.212934+0000","last_fullsized":"2024-09-18T07:31:50.212934+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.075146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.727305+0000","last_change":"2024-09-18T07:31:02.352805+0000","last_active":"2024-09-18T07:31:49.727305+0000","last_peered":"2024-09-18T07:31:49.727305+0000","last_clean":"2024-09-18T07:31:49.727305+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:31:49.727305+0000","last_undegraded":"2024-09-18T07:31:49.727305+0000","last_fullsized":"2024-09-18T07:31:49.727305+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.115071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212634+0000","last_change":"2024-09-18T07:30:58.320419+0000","last_active":"2024-09-18T07:31:50.212634+0000","last_peered":"2024-09-18T07:31:50.212634+0000","last_clean":"2024-09-18T07:31:50.212634+0000","last_became_active":"2024-09-18T07:30:58.320108+0000","last_became_peered":"2024-09-18T07:30:58.320108+0000","last_unstale":"2024-09-18T07:31:50.212634+0000","last_undegraded":"2024-09-18T07:31:50.212634+0000","last_fullsized":"2024-09-18T07:31:50.212634+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.449011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214420+0000","last_change":"2024-09-18T07:30:56.289671+0000","last_active":"2024-09-18T07:31:50.214420+0000","last_peered":"2024-09-18T07:31:50.214420+0000","last_clean":"2024-09-18T07:31:50.214420+0000","last_became_active":"2024-09-18T07:30:56.289328+0000","last_became_peered":"2024-09-18T07:30:56.289328+0000","last_unstale":"2024-09-18T07:31:50.214420+0000","last_undegraded":"2024-09-18T07:31:50.214420+0000","last_fullsized":"2024-09-18T07:31:50.214420+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:58.531067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214601+0000","last_change":"2024-09-18T07:31:00.336053+0000","last_active":"2024-09-18T07:31:50.214601+0000","last_peered":"2024-09-18T07:31:50.214601+0000","last_clean":"2024-09-18T07:31:50.214601+0000","last_became_active":"2024-09-18T07:31:00.335749+0000","last_became_peered":"2024-09-18T07:31:00.335749+0000","last_unstale":"2024-09-18T07:31:50.214601+0000","last_undegraded":"2024-09-18T07:31:50.214601+0000","last_fullsized":"2024-09-18T07:31:50.214601+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:26.183381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213076+0000","last_change":"2024-09-18T07:31:02.491487+0000","last_active":"2024-09-18T07:31:50.213076+0000","last_peered":"2024-09-18T07:31:50.213076+0000","last_clean":"2024-09-18T07:31:50.213076+0000","last_became_active":"2024-09-18T07:31:02.490994+0000","last_became_peered":"2024-09-18T07:31:02.490994+0000","last_unstale":"2024-09-18T07:31:50.213076+0000","last_undegraded":"2024-09-18T07:31:50.213076+0000","last_fullsized":"2024-09-18T07:31:50.213076+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.401313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212321+0000","last_change":"2024-09-18T07:30:58.316173+0000","last_active":"2024-09-18T07:31:50.212321+0000","last_peered":"2024-09-18T07:31:50.212321+0000","last_clean":"2024-09-18T07:31:50.212321+0000","last_became_active":"2024-09-18T07:30:58.315435+0000","last_became_peered":"2024-09-18T07:30:58.315435+0000","last_unstale":"2024-09-18T07:31:50.212321+0000","last_undegraded":"2024-09-18T07:31:50.212321+0000","last_fullsized":"2024-09-18T07:31:50.212321+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:05.325393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728765+0000","last_change":"2024-09-18T07:30:56.283643+0000","last_active":"2024-09-18T07:31:49.728765+0000","last_peered":"2024-09-18T07:31:49.728765+0000","last_clean":"2024-09-18T07:31:49.728765+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:31:49.728765+0000","last_undegraded":"2024-09-18T07:31:49.728765+0000","last_fullsized":"2024-09-18T07:31:49.728765+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:17.971275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212610+0000","last_change":"2024-09-18T07:31:00.333126+0000","last_active":"2024-09-18T07:31:50.212610+0000","last_peered":"2024-09-18T07:31:50.212610+0000","last_clean":"2024-09-18T07:31:50.212610+0000","last_became_active":"2024-09-18T07:31:00.332855+0000","last_became_peered":"2024-09-18T07:31:00.332855+0000","last_unstale":"2024-09-18T07:31:50.212610+0000","last_undegraded":"2024-09-18T07:31:50.212610+0000","last_fullsized":"2024-09-18T07:31:50.212610+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:29.188004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213659+0000","last_change":"2024-09-18T07:31:02.350978+0000","last_active":"2024-09-18T07:31:50.213659+0000","last_peered":"2024-09-18T07:31:50.213659+0000","last_clean":"2024-09-18T07:31:50.213659+0000","last_became_active":"2024-09-18T07:31:02.350692+0000","last_became_peered":"2024-09-18T07:31:02.350692+0000","last_unstale":"2024-09-18T07:31:50.213659+0000","last_undegraded":"2024-09-18T07:31:50.213659+0000","last_fullsized":"2024-09-18T07:31:50.213659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.916468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212923+0000","last_change":"2024-09-18T07:31:02.347827+0000","last_active":"2024-09-18T07:31:50.212923+0000","last_peered":"2024-09-18T07:31:50.212923+0000","last_clean":"2024-09-18T07:31:50.212923+0000","last_became_active":"2024-09-18T07:31:02.347537+0000","last_became_peered":"2024-09-18T07:31:02.347537+0000","last_unstale":"2024-09-18T07:31:50.212923+0000","last_undegraded":"2024-09-18T07:31:50.212923+0000","last_fullsized":"2024-09-18T07:31:50.212923+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:28.722445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":52,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212504+0000","last_change":"2024-09-18T07:30:56.292657+0000","last_active":"2024-09-18T07:31:50.212504+0000","last_peered":"2024-09-18T07:31:50.212504+0000","last_clean":"2024-09-18T07:31:50.212504+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:31:50.212504+0000","last_undegraded":"2024-09-18T07:31:50.212504+0000","last_fullsized":"2024-09-18T07:31:50.212504+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:13.946464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213764+0000","last_change":"2024-09-18T07:30:58.314191+0000","last_active":"2024-09-18T07:31:50.213764+0000","last_peered":"2024-09-18T07:31:50.213764+0000","last_clean":"2024-09-18T07:31:50.213764+0000","last_became_active":"2024-09-18T07:30:58.313556+0000","last_became_peered":"2024-09-18T07:30:58.313556+0000","last_unstale":"2024-09-18T07:31:50.213764+0000","last_undegraded":"2024-09-18T07:31:50.213764+0000","last_fullsized":"2024-09-18T07:31:50.213764+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.554527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.729038+0000","last_change":"2024-09-18T07:31:00.331995+0000","last_active":"2024-09-18T07:31:49.729038+0000","last_peered":"2024-09-18T07:31:49.729038+0000","last_clean":"2024-09-18T07:31:49.729038+0000","last_became_active":"2024-09-18T07:31:00.331564+0000","last_became_peered":"2024-09-18T07:31:00.331564+0000","last_unstale":"2024-09-18T07:31:49.729038+0000","last_undegraded":"2024-09-18T07:31:49.729038+0000","last_fullsized":"2024-09-18T07:31:49.729038+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.738488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":45,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201295,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28384,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716768,"statfs":{"total":95995035648,"available":95965970432,"internally_reserved":0,"allocated":1388544,"data_stored":908644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462933,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32524,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712628,"statfs":{"total":95995035648,"available":95961731072,"internally_reserved":0,"allocated":1368064,"data_stored":907559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724570,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31944,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713208,"statfs":{"total":95995035648,"available":95962324992,"internally_reserved":0,"allocated":774144,"data_stored":317537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953505,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":802816,"data_stored":318219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27828,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717324,"statfs":{"total":95995035648,"available":95966539776,"internally_reserved":0,"allocated":819200,"data_stored":319511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27776,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717376,"statfs":{"total":95995035648,"available":95966593024,"internally_reserved":0,"allocated":765952,"data_stored":317326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32008,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713144,"statfs":{"total":95995035648,"available":95962259456,"internally_reserved":0,"allocated":839680,"data_stored":319956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28392,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716760,"statfs":{"total":95995035648,"available":95965962240,"internally_reserved":0,"allocated":1396736,"data_stored":909769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:31:58.043 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:31:58.309 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:31:58.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:57 smithi137 ceph-mon[29408]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-18T07:31:58.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:57 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:31:58.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:31:58] ENGINE Bus STOPPING 2024-09-18T07:31:58.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:57 smithi137 ceph-mon[39175]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-18T07:31:58.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:57 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:31:58.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:57 smithi203 ceph-mon[32237]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-18T07:31:58.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:57 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:31:58.705 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:31:58] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-18T07:31:58.705 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:31:58] ENGINE Bus STOPPED 2024-09-18T07:31:58.705 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:31:58] ENGINE Bus STARTING 2024-09-18T07:31:58.705 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:31:58] ENGINE Serving on http://:::9283 2024-09-18T07:31:58.706 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:31:58] ENGINE Bus STARTED 2024-09-18T07:31:58.706 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a[71635]: ts=2024-09-18T07:31:58.564Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000851721s 2024-09-18T07:31:59.005 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:31:59.006 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:31:59.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[29408]: from='client.34673 v1:172.21.15.137:0/995015315' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:31:59.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[29408]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:31:59.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:59.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:59.348 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-18T07:31:59.348 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[29408]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-18T07:31:59.348 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:59.348 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[29408]: Deploying daemon grafana.a on smithi203 2024-09-18T07:31:59.348 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:31:59.348 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[29408]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T07:31:59.348 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[29408]: 3.16 scrub starts 2024-09-18T07:31:59.349 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[29408]: 3.16 scrub ok 2024-09-18T07:31:59.349 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[39175]: from='client.34673 v1:172.21.15.137:0/995015315' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:31:59.349 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[39175]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:31:59.349 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:59.349 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:59.349 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-18T07:31:59.349 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[39175]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-18T07:31:59.349 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:59.350 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[39175]: Deploying daemon grafana.a on smithi203 2024-09-18T07:31:59.350 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:31:59.350 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[39175]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T07:31:59.350 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[39175]: 3.16 scrub starts 2024-09-18T07:31:59.350 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:31:58 smithi137 ceph-mon[39175]: 3.16 scrub ok 2024-09-18T07:31:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:58 smithi203 ceph-mon[32237]: from='client.34673 v1:172.21.15.137:0/995015315' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:31:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:58 smithi203 ceph-mon[32237]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:31:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:58 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:58 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:58 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-18T07:31:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:58 smithi203 ceph-mon[32237]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-18T07:31:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:58 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:31:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:58 smithi203 ceph-mon[32237]: Deploying daemon grafana.a on smithi203 2024-09-18T07:31:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:58 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:31:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:58 smithi203 ceph-mon[32237]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T07:31:59.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:58 smithi203 ceph-mon[32237]: 3.16 scrub starts 2024-09-18T07:31:59.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:31:58 smithi203 ceph-mon[32237]: 3.16 scrub ok 2024-09-18T07:31:59.785 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-18T07:31:57.410933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":903,"num_read_kb":760,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236668,"kb_used_data":7964,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724548,"statfs":{"total":767960285184,"available":767717937152,"internally_reserved":0,"allocated":8155136,"data_stored":4318521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002291"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728698+0000","last_change":"2024-09-18T07:30:56.284087+0000","last_active":"2024-09-18T07:31:49.728698+0000","last_peered":"2024-09-18T07:31:49.728698+0000","last_clean":"2024-09-18T07:31:49.728698+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:31:49.728698+0000","last_undegraded":"2024-09-18T07:31:49.728698+0000","last_fullsized":"2024-09-18T07:31:49.728698+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.049077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725514+0000","last_change":"2024-09-18T07:30:58.309315+0000","last_active":"2024-09-18T07:31:49.725514+0000","last_peered":"2024-09-18T07:31:49.725514+0000","last_clean":"2024-09-18T07:31:49.725514+0000","last_became_active":"2024-09-18T07:30:58.308981+0000","last_became_peered":"2024-09-18T07:30:58.308981+0000","last_unstale":"2024-09-18T07:31:49.725514+0000","last_undegraded":"2024-09-18T07:31:49.725514+0000","last_fullsized":"2024-09-18T07:31:49.725514+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:08.376023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212174+0000","last_change":"2024-09-18T07:31:00.336284+0000","last_active":"2024-09-18T07:31:50.212174+0000","last_peered":"2024-09-18T07:31:50.212174+0000","last_clean":"2024-09-18T07:31:50.212174+0000","last_became_active":"2024-09-18T07:31:00.335958+0000","last_became_peered":"2024-09-18T07:31:00.335958+0000","last_unstale":"2024-09-18T07:31:50.212174+0000","last_undegraded":"2024-09-18T07:31:50.212174+0000","last_fullsized":"2024-09-18T07:31:50.212174+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:08.645781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725723+0000","last_change":"2024-09-18T07:31:02.342510+0000","last_active":"2024-09-18T07:31:49.725723+0000","last_peered":"2024-09-18T07:31:49.725723+0000","last_clean":"2024-09-18T07:31:49.725723+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:31:49.725723+0000","last_undegraded":"2024-09-18T07:31:49.725723+0000","last_fullsized":"2024-09-18T07:31:49.725723+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.002947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212748+0000","last_change":"2024-09-18T07:31:02.492941+0000","last_active":"2024-09-18T07:31:50.212748+0000","last_peered":"2024-09-18T07:31:50.212748+0000","last_clean":"2024-09-18T07:31:50.212748+0000","last_became_active":"2024-09-18T07:31:02.492405+0000","last_became_peered":"2024-09-18T07:31:02.492405+0000","last_unstale":"2024-09-18T07:31:50.212748+0000","last_undegraded":"2024-09-18T07:31:50.212748+0000","last_fullsized":"2024-09-18T07:31:50.212748+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:22.665514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213765+0000","last_change":"2024-09-18T07:30:56.286313+0000","last_active":"2024-09-18T07:31:50.213765+0000","last_peered":"2024-09-18T07:31:50.213765+0000","last_clean":"2024-09-18T07:31:50.213765+0000","last_became_active":"2024-09-18T07:30:56.286040+0000","last_became_peered":"2024-09-18T07:30:56.286040+0000","last_unstale":"2024-09-18T07:31:50.213765+0000","last_undegraded":"2024-09-18T07:31:50.213765+0000","last_fullsized":"2024-09-18T07:31:50.213765+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:09.154352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212813+0000","last_change":"2024-09-18T07:30:58.314153+0000","last_active":"2024-09-18T07:31:50.212813+0000","last_peered":"2024-09-18T07:31:50.212813+0000","last_clean":"2024-09-18T07:31:50.212813+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:31:50.212813+0000","last_undegraded":"2024-09-18T07:31:50.212813+0000","last_fullsized":"2024-09-18T07:31:50.212813+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.980863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725505+0000","last_change":"2024-09-18T07:31:00.331498+0000","last_active":"2024-09-18T07:31:49.725505+0000","last_peered":"2024-09-18T07:31:49.725505+0000","last_clean":"2024-09-18T07:31:49.725505+0000","last_became_active":"2024-09-18T07:31:00.331224+0000","last_became_peered":"2024-09-18T07:31:00.331224+0000","last_unstale":"2024-09-18T07:31:49.725505+0000","last_undegraded":"2024-09-18T07:31:49.725505+0000","last_fullsized":"2024-09-18T07:31:49.725505+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:15.385510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214428+0000","last_change":"2024-09-18T07:30:56.286407+0000","last_active":"2024-09-18T07:31:50.214428+0000","last_peered":"2024-09-18T07:31:50.214428+0000","last_clean":"2024-09-18T07:31:50.214428+0000","last_became_active":"2024-09-18T07:30:56.285882+0000","last_became_peered":"2024-09-18T07:30:56.285882+0000","last_unstale":"2024-09-18T07:31:50.214428+0000","last_undegraded":"2024-09-18T07:31:50.214428+0000","last_fullsized":"2024-09-18T07:31:50.214428+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.252387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725892+0000","last_change":"2024-09-18T07:30:58.317018+0000","last_active":"2024-09-18T07:31:49.725892+0000","last_peered":"2024-09-18T07:31:49.725892+0000","last_clean":"2024-09-18T07:31:49.725892+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:31:49.725892+0000","last_undegraded":"2024-09-18T07:31:49.725892+0000","last_fullsized":"2024-09-18T07:31:49.725892+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:59.135054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214036+0000","last_change":"2024-09-18T07:31:00.330533+0000","last_active":"2024-09-18T07:31:50.214036+0000","last_peered":"2024-09-18T07:31:50.214036+0000","last_clean":"2024-09-18T07:31:50.214036+0000","last_became_active":"2024-09-18T07:31:00.330272+0000","last_became_peered":"2024-09-18T07:31:00.330272+0000","last_unstale":"2024-09-18T07:31:50.214036+0000","last_undegraded":"2024-09-18T07:31:50.214036+0000","last_fullsized":"2024-09-18T07:31:50.214036+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.245066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.727443+0000","last_change":"2024-09-18T07:31:02.353810+0000","last_active":"2024-09-18T07:31:49.727443+0000","last_peered":"2024-09-18T07:31:49.727443+0000","last_clean":"2024-09-18T07:31:49.727443+0000","last_became_active":"2024-09-18T07:31:02.353494+0000","last_became_peered":"2024-09-18T07:31:02.353494+0000","last_unstale":"2024-09-18T07:31:49.727443+0000","last_undegraded":"2024-09-18T07:31:49.727443+0000","last_fullsized":"2024-09-18T07:31:49.727443+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.436492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214151+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:31:50.214151+0000","last_peered":"2024-09-18T07:31:50.214151+0000","last_clean":"2024-09-18T07:31:50.214151+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:31:50.214151+0000","last_undegraded":"2024-09-18T07:31:50.214151+0000","last_fullsized":"2024-09-18T07:31:50.214151+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.419105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":52,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725843+0000","last_change":"2024-09-18T07:30:58.317041+0000","last_active":"2024-09-18T07:31:49.725843+0000","last_peered":"2024-09-18T07:31:49.725843+0000","last_clean":"2024-09-18T07:31:49.725843+0000","last_became_active":"2024-09-18T07:30:58.316479+0000","last_became_peered":"2024-09-18T07:30:58.316479+0000","last_unstale":"2024-09-18T07:31:49.725843+0000","last_undegraded":"2024-09-18T07:31:49.725843+0000","last_fullsized":"2024-09-18T07:31:49.725843+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.191081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213747+0000","last_change":"2024-09-18T07:31:00.334023+0000","last_active":"2024-09-18T07:31:50.213747+0000","last_peered":"2024-09-18T07:31:50.213747+0000","last_clean":"2024-09-18T07:31:50.213747+0000","last_became_active":"2024-09-18T07:31:00.333685+0000","last_became_peered":"2024-09-18T07:31:00.333685+0000","last_unstale":"2024-09-18T07:31:50.213747+0000","last_undegraded":"2024-09-18T07:31:50.213747+0000","last_fullsized":"2024-09-18T07:31:50.213747+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.774482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212600+0000","last_change":"2024-09-18T07:31:02.349964+0000","last_active":"2024-09-18T07:31:50.212600+0000","last_peered":"2024-09-18T07:31:50.212600+0000","last_clean":"2024-09-18T07:31:50.212600+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:31:50.212600+0000","last_undegraded":"2024-09-18T07:31:50.212600+0000","last_fullsized":"2024-09-18T07:31:50.212600+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.932342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726399+0000","last_change":"2024-09-18T07:31:02.490397+0000","last_active":"2024-09-18T07:31:49.726399+0000","last_peered":"2024-09-18T07:31:49.726399+0000","last_clean":"2024-09-18T07:31:49.726399+0000","last_became_active":"2024-09-18T07:31:02.490014+0000","last_became_peered":"2024-09-18T07:31:02.490014+0000","last_unstale":"2024-09-18T07:31:49.726399+0000","last_undegraded":"2024-09-18T07:31:49.726399+0000","last_fullsized":"2024-09-18T07:31:49.726399+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.792505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.729429+0000","last_change":"2024-09-18T07:30:56.288891+0000","last_active":"2024-09-18T07:31:49.729429+0000","last_peered":"2024-09-18T07:31:49.729429+0000","last_clean":"2024-09-18T07:31:49.729429+0000","last_became_active":"2024-09-18T07:30:56.288589+0000","last_became_peered":"2024-09-18T07:30:56.288589+0000","last_unstale":"2024-09-18T07:31:49.729429+0000","last_undegraded":"2024-09-18T07:31:49.729429+0000","last_fullsized":"2024-09-18T07:31:49.729429+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.974097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728460+0000","last_change":"2024-09-18T07:30:58.316556+0000","last_active":"2024-09-18T07:31:49.728460+0000","last_peered":"2024-09-18T07:31:49.728460+0000","last_clean":"2024-09-18T07:31:49.728460+0000","last_became_active":"2024-09-18T07:30:58.316266+0000","last_became_peered":"2024-09-18T07:30:58.316266+0000","last_unstale":"2024-09-18T07:31:49.728460+0000","last_undegraded":"2024-09-18T07:31:49.728460+0000","last_fullsized":"2024-09-18T07:31:49.728460+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:22.570802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213206+0000","last_change":"2024-09-18T07:31:00.338287+0000","last_active":"2024-09-18T07:31:50.213206+0000","last_peered":"2024-09-18T07:31:50.213206+0000","last_clean":"2024-09-18T07:31:50.213206+0000","last_became_active":"2024-09-18T07:31:00.337474+0000","last_became_peered":"2024-09-18T07:31:00.337474+0000","last_unstale":"2024-09-18T07:31:50.213206+0000","last_undegraded":"2024-09-18T07:31:50.213206+0000","last_fullsized":"2024-09-18T07:31:50.213206+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.609543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213754+0000","last_change":"2024-09-18T07:31:02.492790+0000","last_active":"2024-09-18T07:31:50.213754+0000","last_peered":"2024-09-18T07:31:50.213754+0000","last_clean":"2024-09-18T07:31:50.213754+0000","last_became_active":"2024-09-18T07:31:02.492348+0000","last_became_peered":"2024-09-18T07:31:02.492348+0000","last_unstale":"2024-09-18T07:31:50.213754+0000","last_undegraded":"2024-09-18T07:31:50.213754+0000","last_fullsized":"2024-09-18T07:31:50.213754+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:08.542586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726056+0000","last_change":"2024-09-18T07:30:56.289489+0000","last_active":"2024-09-18T07:31:49.726056+0000","last_peered":"2024-09-18T07:31:49.726056+0000","last_clean":"2024-09-18T07:31:49.726056+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:31:49.726056+0000","last_undegraded":"2024-09-18T07:31:49.726056+0000","last_fullsized":"2024-09-18T07:31:49.726056+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.331217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213566+0000","last_change":"2024-09-18T07:30:58.319115+0000","last_active":"2024-09-18T07:31:50.213566+0000","last_peered":"2024-09-18T07:31:50.213566+0000","last_clean":"2024-09-18T07:31:50.213566+0000","last_became_active":"2024-09-18T07:30:58.316380+0000","last_became_peered":"2024-09-18T07:30:58.316380+0000","last_unstale":"2024-09-18T07:31:50.213566+0000","last_undegraded":"2024-09-18T07:31:50.213566+0000","last_fullsized":"2024-09-18T07:31:50.213566+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.995221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726092+0000","last_change":"2024-09-18T07:31:00.334313+0000","last_active":"2024-09-18T07:31:49.726092+0000","last_peered":"2024-09-18T07:31:49.726092+0000","last_clean":"2024-09-18T07:31:49.726092+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:31:49.726092+0000","last_undegraded":"2024-09-18T07:31:49.726092+0000","last_fullsized":"2024-09-18T07:31:49.726092+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:09.373920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214016+0000","last_change":"2024-09-18T07:31:02.348855+0000","last_active":"2024-09-18T07:31:50.214016+0000","last_peered":"2024-09-18T07:31:50.214016+0000","last_clean":"2024-09-18T07:31:50.214016+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:31:50.214016+0000","last_undegraded":"2024-09-18T07:31:50.214016+0000","last_fullsized":"2024-09-18T07:31:50.214016+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.101140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:56.262559+0000","last_change":"2024-09-18T07:31:56.262559+0000","last_active":"2024-09-18T07:31:56.262559+0000","last_peered":"2024-09-18T07:31:56.262559+0000","last_clean":"2024-09-18T07:31:56.262559+0000","last_became_active":"2024-09-18T07:30:56.283705+0000","last_became_peered":"2024-09-18T07:30:56.283705+0000","last_unstale":"2024-09-18T07:31:56.262559+0000","last_undegraded":"2024-09-18T07:31:56.262559+0000","last_fullsized":"2024-09-18T07:31:56.262559+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:31:56.262487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:56.262487+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:15.704852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.729111+0000","last_change":"2024-09-18T07:30:58.318003+0000","last_active":"2024-09-18T07:31:49.729111+0000","last_peered":"2024-09-18T07:31:49.729111+0000","last_clean":"2024-09-18T07:31:49.729111+0000","last_became_active":"2024-09-18T07:30:58.317355+0000","last_became_peered":"2024-09-18T07:30:58.317355+0000","last_unstale":"2024-09-18T07:31:49.729111+0000","last_undegraded":"2024-09-18T07:31:49.729111+0000","last_fullsized":"2024-09-18T07:31:49.729111+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.431023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":62,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:51.345698+0000","last_change":"2024-09-18T07:31:00.331318+0000","last_active":"2024-09-18T07:31:51.345698+0000","last_peered":"2024-09-18T07:31:51.345698+0000","last_clean":"2024-09-18T07:31:51.345698+0000","last_became_active":"2024-09-18T07:31:00.331014+0000","last_became_peered":"2024-09-18T07:31:00.331014+0000","last_unstale":"2024-09-18T07:31:51.345698+0000","last_undegraded":"2024-09-18T07:31:51.345698+0000","last_fullsized":"2024-09-18T07:31:51.345698+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.342355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212788+0000","last_change":"2024-09-18T07:30:58.312916+0000","last_active":"2024-09-18T07:31:50.212788+0000","last_peered":"2024-09-18T07:31:50.212788+0000","last_clean":"2024-09-18T07:31:50.212788+0000","last_became_active":"2024-09-18T07:30:58.312627+0000","last_became_peered":"2024-09-18T07:30:58.312627+0000","last_unstale":"2024-09-18T07:31:50.212788+0000","last_undegraded":"2024-09-18T07:31:50.212788+0000","last_fullsized":"2024-09-18T07:31:50.212788+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.188948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212555+0000","last_change":"2024-09-18T07:30:56.289331+0000","last_active":"2024-09-18T07:31:50.212555+0000","last_peered":"2024-09-18T07:31:50.212555+0000","last_clean":"2024-09-18T07:31:50.212555+0000","last_became_active":"2024-09-18T07:30:56.289081+0000","last_became_peered":"2024-09-18T07:30:56.289081+0000","last_unstale":"2024-09-18T07:31:50.212555+0000","last_undegraded":"2024-09-18T07:31:50.212555+0000","last_fullsized":"2024-09-18T07:31:50.212555+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:56.283698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:51.345704+0000","last_change":"2024-09-18T07:31:00.333026+0000","last_active":"2024-09-18T07:31:51.345704+0000","last_peered":"2024-09-18T07:31:51.345704+0000","last_clean":"2024-09-18T07:31:51.345704+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:31:51.345704+0000","last_undegraded":"2024-09-18T07:31:51.345704+0000","last_fullsized":"2024-09-18T07:31:51.345704+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:09.647541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214053+0000","last_change":"2024-09-18T07:31:02.351412+0000","last_active":"2024-09-18T07:31:50.214053+0000","last_peered":"2024-09-18T07:31:50.214053+0000","last_clean":"2024-09-18T07:31:50.214053+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:31:50.214053+0000","last_undegraded":"2024-09-18T07:31:50.214053+0000","last_fullsized":"2024-09-18T07:31:50.214053+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.243657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":74,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212420+0000","last_change":"2024-09-18T07:30:58.321857+0000","last_active":"2024-09-18T07:31:50.212420+0000","last_peered":"2024-09-18T07:31:50.212420+0000","last_clean":"2024-09-18T07:31:50.212420+0000","last_became_active":"2024-09-18T07:30:58.321244+0000","last_became_peered":"2024-09-18T07:30:58.321244+0000","last_unstale":"2024-09-18T07:31:50.212420+0000","last_undegraded":"2024-09-18T07:31:50.212420+0000","last_fullsized":"2024-09-18T07:31:50.212420+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:58.899213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213919+0000","last_change":"2024-09-18T07:30:56.289862+0000","last_active":"2024-09-18T07:31:50.213919+0000","last_peered":"2024-09-18T07:31:50.213919+0000","last_clean":"2024-09-18T07:31:50.213919+0000","last_became_active":"2024-09-18T07:30:56.289556+0000","last_became_peered":"2024-09-18T07:30:56.289556+0000","last_unstale":"2024-09-18T07:31:50.213919+0000","last_undegraded":"2024-09-18T07:31:50.213919+0000","last_fullsized":"2024-09-18T07:31:50.213919+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:59.535940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725580+0000","last_change":"2024-09-18T07:31:00.333320+0000","last_active":"2024-09-18T07:31:49.725580+0000","last_peered":"2024-09-18T07:31:49.725580+0000","last_clean":"2024-09-18T07:31:49.725580+0000","last_became_active":"2024-09-18T07:31:00.332854+0000","last_became_peered":"2024-09-18T07:31:00.332854+0000","last_unstale":"2024-09-18T07:31:49.725580+0000","last_undegraded":"2024-09-18T07:31:49.725580+0000","last_fullsized":"2024-09-18T07:31:49.725580+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:20.691030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725889+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:31:49.725889+0000","last_peered":"2024-09-18T07:31:49.725889+0000","last_clean":"2024-09-18T07:31:49.725889+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:31:49.725889+0000","last_undegraded":"2024-09-18T07:31:49.725889+0000","last_fullsized":"2024-09-18T07:31:49.725889+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.039810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":63,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725593+0000","last_change":"2024-09-18T07:30:58.309938+0000","last_active":"2024-09-18T07:31:49.725593+0000","last_peered":"2024-09-18T07:31:49.725593+0000","last_clean":"2024-09-18T07:31:49.725593+0000","last_became_active":"2024-09-18T07:30:58.309649+0000","last_became_peered":"2024-09-18T07:30:58.309649+0000","last_unstale":"2024-09-18T07:31:49.725593+0000","last_undegraded":"2024-09-18T07:31:49.725593+0000","last_fullsized":"2024-09-18T07:31:49.725593+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:20.121712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.729501+0000","last_change":"2024-09-18T07:30:56.287909+0000","last_active":"2024-09-18T07:31:49.729501+0000","last_peered":"2024-09-18T07:31:49.729501+0000","last_clean":"2024-09-18T07:31:49.729501+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:31:49.729501+0000","last_undegraded":"2024-09-18T07:31:49.729501+0000","last_fullsized":"2024-09-18T07:31:49.729501+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:05.243933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:51.345487+0000","last_change":"2024-09-18T07:31:00.336059+0000","last_active":"2024-09-18T07:31:51.345487+0000","last_peered":"2024-09-18T07:31:51.345487+0000","last_clean":"2024-09-18T07:31:51.345487+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:31:51.345487+0000","last_undegraded":"2024-09-18T07:31:51.345487+0000","last_fullsized":"2024-09-18T07:31:51.345487+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.046222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213223+0000","last_change":"2024-09-18T07:31:02.492797+0000","last_active":"2024-09-18T07:31:50.213223+0000","last_peered":"2024-09-18T07:31:50.213223+0000","last_clean":"2024-09-18T07:31:50.213223+0000","last_became_active":"2024-09-18T07:31:02.492483+0000","last_became_peered":"2024-09-18T07:31:02.492483+0000","last_unstale":"2024-09-18T07:31:50.213223+0000","last_undegraded":"2024-09-18T07:31:50.213223+0000","last_fullsized":"2024-09-18T07:31:50.213223+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.277226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212350+0000","last_change":"2024-09-18T07:30:58.307470+0000","last_active":"2024-09-18T07:31:50.212350+0000","last_peered":"2024-09-18T07:31:50.212350+0000","last_clean":"2024-09-18T07:31:50.212350+0000","last_became_active":"2024-09-18T07:30:58.307035+0000","last_became_peered":"2024-09-18T07:30:58.307035+0000","last_unstale":"2024-09-18T07:31:50.212350+0000","last_undegraded":"2024-09-18T07:31:50.212350+0000","last_fullsized":"2024-09-18T07:31:50.212350+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:06.964257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214484+0000","last_change":"2024-09-18T07:30:56.290040+0000","last_active":"2024-09-18T07:31:50.214484+0000","last_peered":"2024-09-18T07:31:50.214484+0000","last_clean":"2024-09-18T07:31:50.214484+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:31:50.214484+0000","last_undegraded":"2024-09-18T07:31:50.214484+0000","last_fullsized":"2024-09-18T07:31:50.214484+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.332869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:51.345490+0000","last_change":"2024-09-18T07:31:00.337241+0000","last_active":"2024-09-18T07:31:51.345490+0000","last_peered":"2024-09-18T07:31:51.345490+0000","last_clean":"2024-09-18T07:31:51.345490+0000","last_became_active":"2024-09-18T07:31:00.336905+0000","last_became_peered":"2024-09-18T07:31:00.336905+0000","last_unstale":"2024-09-18T07:31:51.345490+0000","last_undegraded":"2024-09-18T07:31:51.345490+0000","last_fullsized":"2024-09-18T07:31:51.345490+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.826427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.727559+0000","last_change":"2024-09-18T07:31:02.348538+0000","last_active":"2024-09-18T07:31:49.727559+0000","last_peered":"2024-09-18T07:31:49.727559+0000","last_clean":"2024-09-18T07:31:49.727559+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:31:49.727559+0000","last_undegraded":"2024-09-18T07:31:49.727559+0000","last_fullsized":"2024-09-18T07:31:49.727559+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.088828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":78,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.729264+0000","last_change":"2024-09-18T07:30:58.318096+0000","last_active":"2024-09-18T07:31:49.729264+0000","last_peered":"2024-09-18T07:31:49.729264+0000","last_clean":"2024-09-18T07:31:49.729264+0000","last_became_active":"2024-09-18T07:30:58.317767+0000","last_became_peered":"2024-09-18T07:30:58.317767+0000","last_unstale":"2024-09-18T07:31:49.729264+0000","last_undegraded":"2024-09-18T07:31:49.729264+0000","last_fullsized":"2024-09-18T07:31:49.729264+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:26.992100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212961+0000","last_change":"2024-09-18T07:30:56.284440+0000","last_active":"2024-09-18T07:31:50.212961+0000","last_peered":"2024-09-18T07:31:50.212961+0000","last_clean":"2024-09-18T07:31:50.212961+0000","last_became_active":"2024-09-18T07:30:56.284120+0000","last_became_peered":"2024-09-18T07:30:56.284120+0000","last_unstale":"2024-09-18T07:31:50.212961+0000","last_undegraded":"2024-09-18T07:31:50.212961+0000","last_fullsized":"2024-09-18T07:31:50.212961+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.025187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212835+0000","last_change":"2024-09-18T07:31:00.335100+0000","last_active":"2024-09-18T07:31:50.212835+0000","last_peered":"2024-09-18T07:31:50.212835+0000","last_clean":"2024-09-18T07:31:50.212835+0000","last_became_active":"2024-09-18T07:31:00.334776+0000","last_became_peered":"2024-09-18T07:31:00.334776+0000","last_unstale":"2024-09-18T07:31:50.212835+0000","last_undegraded":"2024-09-18T07:31:50.212835+0000","last_fullsized":"2024-09-18T07:31:50.212835+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.405293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212161+0000","last_change":"2024-09-18T07:31:02.490534+0000","last_active":"2024-09-18T07:31:50.212161+0000","last_peered":"2024-09-18T07:31:50.212161+0000","last_clean":"2024-09-18T07:31:50.212161+0000","last_became_active":"2024-09-18T07:31:02.490186+0000","last_became_peered":"2024-09-18T07:31:02.490186+0000","last_unstale":"2024-09-18T07:31:50.212161+0000","last_undegraded":"2024-09-18T07:31:50.212161+0000","last_fullsized":"2024-09-18T07:31:50.212161+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.881705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":103,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:51.345498+0000","last_change":"2024-09-18T07:30:58.317477+0000","last_active":"2024-09-18T07:31:51.345498+0000","last_peered":"2024-09-18T07:31:51.345498+0000","last_clean":"2024-09-18T07:31:51.345498+0000","last_became_active":"2024-09-18T07:30:58.316812+0000","last_became_peered":"2024-09-18T07:30:58.316812+0000","last_unstale":"2024-09-18T07:31:51.345498+0000","last_undegraded":"2024-09-18T07:31:51.345498+0000","last_fullsized":"2024-09-18T07:31:51.345498+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.769678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726147+0000","last_change":"2024-09-18T07:30:56.285353+0000","last_active":"2024-09-18T07:31:49.726147+0000","last_peered":"2024-09-18T07:31:49.726147+0000","last_clean":"2024-09-18T07:31:49.726147+0000","last_became_active":"2024-09-18T07:30:56.285032+0000","last_became_peered":"2024-09-18T07:30:56.285032+0000","last_unstale":"2024-09-18T07:31:49.726147+0000","last_undegraded":"2024-09-18T07:31:49.726147+0000","last_fullsized":"2024-09-18T07:31:49.726147+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:23.187833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212993+0000","last_change":"2024-09-18T07:30:58.299308+0000","last_active":"2024-09-18T07:31:50.212993+0000","last_peered":"2024-09-18T07:31:50.212993+0000","last_clean":"2024-09-18T07:31:50.212993+0000","last_became_active":"2024-09-18T07:30:55.283292+0000","last_became_peered":"2024-09-18T07:30:55.283292+0000","last_unstale":"2024-09-18T07:31:50.212993+0000","last_undegraded":"2024-09-18T07:31:50.212993+0000","last_fullsized":"2024-09-18T07:31:50.212993+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:13.332916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728454+0000","last_change":"2024-09-18T07:31:00.332738+0000","last_active":"2024-09-18T07:31:49.728454+0000","last_peered":"2024-09-18T07:31:49.728454+0000","last_clean":"2024-09-18T07:31:49.728454+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:31:49.728454+0000","last_undegraded":"2024-09-18T07:31:49.728454+0000","last_fullsized":"2024-09-18T07:31:49.728454+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:23.578215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213578+0000","last_change":"2024-09-18T07:31:02.351778+0000","last_active":"2024-09-18T07:31:50.213578+0000","last_peered":"2024-09-18T07:31:50.213578+0000","last_clean":"2024-09-18T07:31:50.213578+0000","last_became_active":"2024-09-18T07:31:02.351403+0000","last_became_peered":"2024-09-18T07:31:02.351403+0000","last_unstale":"2024-09-18T07:31:50.213578+0000","last_undegraded":"2024-09-18T07:31:50.213578+0000","last_fullsized":"2024-09-18T07:31:50.213578+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:27.871124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":69,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212883+0000","last_change":"2024-09-18T07:30:58.310789+0000","last_active":"2024-09-18T07:31:50.212883+0000","last_peered":"2024-09-18T07:31:50.212883+0000","last_clean":"2024-09-18T07:31:50.212883+0000","last_became_active":"2024-09-18T07:30:58.310490+0000","last_became_peered":"2024-09-18T07:30:58.310490+0000","last_unstale":"2024-09-18T07:31:50.212883+0000","last_undegraded":"2024-09-18T07:31:50.212883+0000","last_fullsized":"2024-09-18T07:31:50.212883+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:22.258389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212394+0000","last_change":"2024-09-18T07:30:56.285054+0000","last_active":"2024-09-18T07:31:50.212394+0000","last_peered":"2024-09-18T07:31:50.212394+0000","last_clean":"2024-09-18T07:31:50.212394+0000","last_became_active":"2024-09-18T07:30:56.284803+0000","last_became_peered":"2024-09-18T07:30:56.284803+0000","last_unstale":"2024-09-18T07:31:50.212394+0000","last_undegraded":"2024-09-18T07:31:50.212394+0000","last_fullsized":"2024-09-18T07:31:50.212394+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.764106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.727693+0000","last_change":"2024-09-18T07:30:58.301149+0000","last_active":"2024-09-18T07:31:49.727693+0000","last_peered":"2024-09-18T07:31:49.727693+0000","last_clean":"2024-09-18T07:31:49.727693+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:31:49.727693+0000","last_undegraded":"2024-09-18T07:31:49.727693+0000","last_fullsized":"2024-09-18T07:31:49.727693+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:01.827803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728591+0000","last_change":"2024-09-18T07:31:00.331226+0000","last_active":"2024-09-18T07:31:49.728591+0000","last_peered":"2024-09-18T07:31:49.728591+0000","last_clean":"2024-09-18T07:31:49.728591+0000","last_became_active":"2024-09-18T07:31:00.330883+0000","last_became_peered":"2024-09-18T07:31:00.330883+0000","last_unstale":"2024-09-18T07:31:49.728591+0000","last_undegraded":"2024-09-18T07:31:49.728591+0000","last_fullsized":"2024-09-18T07:31:49.728591+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.695331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213817+0000","last_change":"2024-09-18T07:31:02.491512+0000","last_active":"2024-09-18T07:31:50.213817+0000","last_peered":"2024-09-18T07:31:50.213817+0000","last_clean":"2024-09-18T07:31:50.213817+0000","last_became_active":"2024-09-18T07:31:02.491233+0000","last_became_peered":"2024-09-18T07:31:02.491233+0000","last_unstale":"2024-09-18T07:31:50.213817+0000","last_undegraded":"2024-09-18T07:31:50.213817+0000","last_fullsized":"2024-09-18T07:31:50.213817+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:26.099794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.729187+0000","last_change":"2024-09-18T07:30:58.311778+0000","last_active":"2024-09-18T07:31:49.729187+0000","last_peered":"2024-09-18T07:31:49.729187+0000","last_clean":"2024-09-18T07:31:49.729187+0000","last_became_active":"2024-09-18T07:30:58.311491+0000","last_became_peered":"2024-09-18T07:30:58.311491+0000","last_unstale":"2024-09-18T07:31:49.729187+0000","last_undegraded":"2024-09-18T07:31:49.729187+0000","last_fullsized":"2024-09-18T07:31:49.729187+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:05.875741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":52,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728777+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:31:49.728777+0000","last_peered":"2024-09-18T07:31:49.728777+0000","last_clean":"2024-09-18T07:31:49.728777+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:31:49.728777+0000","last_undegraded":"2024-09-18T07:31:49.728777+0000","last_fullsized":"2024-09-18T07:31:49.728777+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:20.989190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":112,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:54.879580+0000","last_change":"2024-09-18T07:30:58.319969+0000","last_active":"2024-09-18T07:31:54.879580+0000","last_peered":"2024-09-18T07:31:54.879580+0000","last_clean":"2024-09-18T07:31:54.879580+0000","last_became_active":"2024-09-18T07:30:55.286980+0000","last_became_peered":"2024-09-18T07:30:55.286980+0000","last_unstale":"2024-09-18T07:31:54.879580+0000","last_undegraded":"2024-09-18T07:31:54.879580+0000","last_fullsized":"2024-09-18T07:31:54.879580+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.742429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":55,"num_read_kb":50,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.215030+0000","last_change":"2024-09-18T07:31:00.331929+0000","last_active":"2024-09-18T07:31:50.215030+0000","last_peered":"2024-09-18T07:31:50.215030+0000","last_clean":"2024-09-18T07:31:50.215030+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:31:50.215030+0000","last_undegraded":"2024-09-18T07:31:50.215030+0000","last_fullsized":"2024-09-18T07:31:50.215030+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.504554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212644+0000","last_change":"2024-09-18T07:31:02.347659+0000","last_active":"2024-09-18T07:31:50.212644+0000","last_peered":"2024-09-18T07:31:50.212644+0000","last_clean":"2024-09-18T07:31:50.212644+0000","last_became_active":"2024-09-18T07:31:02.347390+0000","last_became_peered":"2024-09-18T07:31:02.347390+0000","last_unstale":"2024-09-18T07:31:50.212644+0000","last_undegraded":"2024-09-18T07:31:50.212644+0000","last_fullsized":"2024-09-18T07:31:50.212644+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.353794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":77,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:51.345840+0000","last_change":"2024-09-18T07:30:58.318426+0000","last_active":"2024-09-18T07:31:51.345840+0000","last_peered":"2024-09-18T07:31:51.345840+0000","last_clean":"2024-09-18T07:31:51.345840+0000","last_became_active":"2024-09-18T07:30:58.318068+0000","last_became_peered":"2024-09-18T07:30:58.318068+0000","last_unstale":"2024-09-18T07:31:51.345840+0000","last_undegraded":"2024-09-18T07:31:51.345840+0000","last_fullsized":"2024-09-18T07:31:51.345840+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:59.863882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213362+0000","last_change":"2024-09-18T07:30:56.288420+0000","last_active":"2024-09-18T07:31:50.213362+0000","last_peered":"2024-09-18T07:31:50.213362+0000","last_clean":"2024-09-18T07:31:50.213362+0000","last_became_active":"2024-09-18T07:30:56.288149+0000","last_became_peered":"2024-09-18T07:30:56.288149+0000","last_unstale":"2024-09-18T07:31:50.213362+0000","last_undegraded":"2024-09-18T07:31:50.213362+0000","last_fullsized":"2024-09-18T07:31:50.213362+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:56.385307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":93,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214868+0000","last_change":"2024-09-18T07:30:50.813696+0000","last_active":"2024-09-18T07:31:50.214868+0000","last_peered":"2024-09-18T07:31:50.214868+0000","last_clean":"2024-09-18T07:31:50.214868+0000","last_became_active":"2024-09-18T07:30:46.719588+0000","last_became_peered":"2024-09-18T07:30:46.719588+0000","last_unstale":"2024-09-18T07:31:50.214868+0000","last_undegraded":"2024-09-18T07:31:50.214868+0000","last_fullsized":"2024-09-18T07:31:50.214868+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-18T07:30:50.813645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:30:50.813645+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:06.405856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214166+0000","last_change":"2024-09-18T07:31:00.331602+0000","last_active":"2024-09-18T07:31:50.214166+0000","last_peered":"2024-09-18T07:31:50.214166+0000","last_clean":"2024-09-18T07:31:50.214166+0000","last_became_active":"2024-09-18T07:31:00.331327+0000","last_became_peered":"2024-09-18T07:31:00.331327+0000","last_unstale":"2024-09-18T07:31:50.214166+0000","last_undegraded":"2024-09-18T07:31:50.214166+0000","last_fullsized":"2024-09-18T07:31:50.214166+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.402435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214934+0000","last_change":"2024-09-18T07:31:02.348225+0000","last_active":"2024-09-18T07:31:50.214934+0000","last_peered":"2024-09-18T07:31:50.214934+0000","last_clean":"2024-09-18T07:31:50.214934+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:31:50.214934+0000","last_undegraded":"2024-09-18T07:31:50.214934+0000","last_fullsized":"2024-09-18T07:31:50.214934+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.749034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726559+0000","last_change":"2024-09-18T07:30:58.313677+0000","last_active":"2024-09-18T07:31:49.726559+0000","last_peered":"2024-09-18T07:31:49.726559+0000","last_clean":"2024-09-18T07:31:49.726559+0000","last_became_active":"2024-09-18T07:30:58.313382+0000","last_became_peered":"2024-09-18T07:30:58.313382+0000","last_unstale":"2024-09-18T07:31:49.726559+0000","last_undegraded":"2024-09-18T07:31:49.726559+0000","last_fullsized":"2024-09-18T07:31:49.726559+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.250207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725860+0000","last_change":"2024-09-18T07:30:56.291117+0000","last_active":"2024-09-18T07:31:49.725860+0000","last_peered":"2024-09-18T07:31:49.725860+0000","last_clean":"2024-09-18T07:31:49.725860+0000","last_became_active":"2024-09-18T07:30:56.290836+0000","last_became_peered":"2024-09-18T07:30:56.290836+0000","last_unstale":"2024-09-18T07:31:49.725860+0000","last_undegraded":"2024-09-18T07:31:49.725860+0000","last_fullsized":"2024-09-18T07:31:49.725860+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.125102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214586+0000","last_change":"2024-09-18T07:31:00.336459+0000","last_active":"2024-09-18T07:31:50.214586+0000","last_peered":"2024-09-18T07:31:50.214586+0000","last_clean":"2024-09-18T07:31:50.214586+0000","last_became_active":"2024-09-18T07:31:00.336172+0000","last_became_peered":"2024-09-18T07:31:00.336172+0000","last_unstale":"2024-09-18T07:31:50.214586+0000","last_undegraded":"2024-09-18T07:31:50.214586+0000","last_fullsized":"2024-09-18T07:31:50.214586+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:28.160928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213176+0000","last_change":"2024-09-18T07:31:02.491383+0000","last_active":"2024-09-18T07:31:50.213176+0000","last_peered":"2024-09-18T07:31:50.213176+0000","last_clean":"2024-09-18T07:31:50.213176+0000","last_became_active":"2024-09-18T07:31:02.490953+0000","last_became_peered":"2024-09-18T07:31:02.490953+0000","last_unstale":"2024-09-18T07:31:50.213176+0000","last_undegraded":"2024-09-18T07:31:50.213176+0000","last_fullsized":"2024-09-18T07:31:50.213176+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.829440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":70,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726449+0000","last_change":"2024-09-18T07:30:58.312636+0000","last_active":"2024-09-18T07:31:49.726449+0000","last_peered":"2024-09-18T07:31:49.726449+0000","last_clean":"2024-09-18T07:31:49.726449+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:31:49.726449+0000","last_undegraded":"2024-09-18T07:31:49.726449+0000","last_fullsized":"2024-09-18T07:31:49.726449+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:13.709878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213218+0000","last_change":"2024-09-18T07:30:56.284303+0000","last_active":"2024-09-18T07:31:50.213218+0000","last_peered":"2024-09-18T07:31:50.213218+0000","last_clean":"2024-09-18T07:31:50.213218+0000","last_became_active":"2024-09-18T07:30:56.284020+0000","last_became_peered":"2024-09-18T07:30:56.284020+0000","last_unstale":"2024-09-18T07:31:50.213218+0000","last_undegraded":"2024-09-18T07:31:50.213218+0000","last_fullsized":"2024-09-18T07:31:50.213218+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:15.982099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212742+0000","last_change":"2024-09-18T07:31:00.333155+0000","last_active":"2024-09-18T07:31:50.212742+0000","last_peered":"2024-09-18T07:31:50.212742+0000","last_clean":"2024-09-18T07:31:50.212742+0000","last_became_active":"2024-09-18T07:31:00.332836+0000","last_became_peered":"2024-09-18T07:31:00.332836+0000","last_unstale":"2024-09-18T07:31:50.212742+0000","last_undegraded":"2024-09-18T07:31:50.212742+0000","last_fullsized":"2024-09-18T07:31:50.212742+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.946427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728567+0000","last_change":"2024-09-18T07:31:02.347665+0000","last_active":"2024-09-18T07:31:49.728567+0000","last_peered":"2024-09-18T07:31:49.728567+0000","last_clean":"2024-09-18T07:31:49.728567+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:31:49.728567+0000","last_undegraded":"2024-09-18T07:31:49.728567+0000","last_fullsized":"2024-09-18T07:31:49.728567+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.617366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726141+0000","last_change":"2024-09-18T07:30:58.317008+0000","last_active":"2024-09-18T07:31:49.726141+0000","last_peered":"2024-09-18T07:31:49.726141+0000","last_clean":"2024-09-18T07:31:49.726141+0000","last_became_active":"2024-09-18T07:30:58.316423+0000","last_became_peered":"2024-09-18T07:30:58.316423+0000","last_unstale":"2024-09-18T07:31:49.726141+0000","last_undegraded":"2024-09-18T07:31:49.726141+0000","last_fullsized":"2024-09-18T07:31:49.726141+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.580907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213156+0000","last_change":"2024-09-18T07:30:56.287284+0000","last_active":"2024-09-18T07:31:50.213156+0000","last_peered":"2024-09-18T07:31:50.213156+0000","last_clean":"2024-09-18T07:31:50.213156+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:31:50.213156+0000","last_undegraded":"2024-09-18T07:31:50.213156+0000","last_fullsized":"2024-09-18T07:31:50.213156+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:00.720049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:51.345486+0000","last_change":"2024-09-18T07:31:00.333376+0000","last_active":"2024-09-18T07:31:51.345486+0000","last_peered":"2024-09-18T07:31:51.345486+0000","last_clean":"2024-09-18T07:31:51.345486+0000","last_became_active":"2024-09-18T07:31:00.333085+0000","last_became_peered":"2024-09-18T07:31:00.333085+0000","last_unstale":"2024-09-18T07:31:51.345486+0000","last_undegraded":"2024-09-18T07:31:51.345486+0000","last_fullsized":"2024-09-18T07:31:51.345486+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:22.590817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726384+0000","last_change":"2024-09-18T07:31:02.347721+0000","last_active":"2024-09-18T07:31:49.726384+0000","last_peered":"2024-09-18T07:31:49.726384+0000","last_clean":"2024-09-18T07:31:49.726384+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:31:49.726384+0000","last_undegraded":"2024-09-18T07:31:49.726384+0000","last_fullsized":"2024-09-18T07:31:49.726384+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.723859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.729220+0000","last_change":"2024-09-18T07:30:58.314424+0000","last_active":"2024-09-18T07:31:49.729220+0000","last_peered":"2024-09-18T07:31:49.729220+0000","last_clean":"2024-09-18T07:31:49.729220+0000","last_became_active":"2024-09-18T07:30:58.314127+0000","last_became_peered":"2024-09-18T07:30:58.314127+0000","last_unstale":"2024-09-18T07:31:49.729220+0000","last_undegraded":"2024-09-18T07:31:49.729220+0000","last_fullsized":"2024-09-18T07:31:49.729220+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:13.636020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214284+0000","last_change":"2024-09-18T07:30:56.289177+0000","last_active":"2024-09-18T07:31:50.214284+0000","last_peered":"2024-09-18T07:31:50.214284+0000","last_clean":"2024-09-18T07:31:50.214284+0000","last_became_active":"2024-09-18T07:30:56.288868+0000","last_became_peered":"2024-09-18T07:30:56.288868+0000","last_unstale":"2024-09-18T07:31:50.214284+0000","last_undegraded":"2024-09-18T07:31:50.214284+0000","last_fullsized":"2024-09-18T07:31:50.214284+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.629409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.727916+0000","last_change":"2024-09-18T07:31:00.336020+0000","last_active":"2024-09-18T07:31:49.727916+0000","last_peered":"2024-09-18T07:31:49.727916+0000","last_clean":"2024-09-18T07:31:49.727916+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:31:49.727916+0000","last_undegraded":"2024-09-18T07:31:49.727916+0000","last_fullsized":"2024-09-18T07:31:49.727916+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:28.112892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.727371+0000","last_change":"2024-09-18T07:31:02.349757+0000","last_active":"2024-09-18T07:31:49.727371+0000","last_peered":"2024-09-18T07:31:49.727371+0000","last_clean":"2024-09-18T07:31:49.727371+0000","last_became_active":"2024-09-18T07:31:02.349454+0000","last_became_peered":"2024-09-18T07:31:02.349454+0000","last_unstale":"2024-09-18T07:31:49.727371+0000","last_undegraded":"2024-09-18T07:31:49.727371+0000","last_fullsized":"2024-09-18T07:31:49.727371+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.874882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":73,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212932+0000","last_change":"2024-09-18T07:30:58.318421+0000","last_active":"2024-09-18T07:31:50.212932+0000","last_peered":"2024-09-18T07:31:50.212932+0000","last_clean":"2024-09-18T07:31:50.212932+0000","last_became_active":"2024-09-18T07:30:58.318063+0000","last_became_peered":"2024-09-18T07:30:58.318063+0000","last_unstale":"2024-09-18T07:31:50.212932+0000","last_undegraded":"2024-09-18T07:31:50.212932+0000","last_fullsized":"2024-09-18T07:31:50.212932+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:59.591866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214181+0000","last_change":"2024-09-18T07:30:56.293678+0000","last_active":"2024-09-18T07:31:50.214181+0000","last_peered":"2024-09-18T07:31:50.214181+0000","last_clean":"2024-09-18T07:31:50.214181+0000","last_became_active":"2024-09-18T07:30:56.293371+0000","last_became_peered":"2024-09-18T07:30:56.293371+0000","last_unstale":"2024-09-18T07:31:50.214181+0000","last_undegraded":"2024-09-18T07:31:50.214181+0000","last_fullsized":"2024-09-18T07:31:50.214181+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.829476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728083+0000","last_change":"2024-09-18T07:31:00.332322+0000","last_active":"2024-09-18T07:31:49.728083+0000","last_peered":"2024-09-18T07:31:49.728083+0000","last_clean":"2024-09-18T07:31:49.728083+0000","last_became_active":"2024-09-18T07:31:00.331914+0000","last_became_peered":"2024-09-18T07:31:00.331914+0000","last_unstale":"2024-09-18T07:31:49.728083+0000","last_undegraded":"2024-09-18T07:31:49.728083+0000","last_fullsized":"2024-09-18T07:31:49.728083+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.543866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214811+0000","last_change":"2024-09-18T07:31:02.351941+0000","last_active":"2024-09-18T07:31:50.214811+0000","last_peered":"2024-09-18T07:31:50.214811+0000","last_clean":"2024-09-18T07:31:50.214811+0000","last_became_active":"2024-09-18T07:31:02.351678+0000","last_became_peered":"2024-09-18T07:31:02.351678+0000","last_unstale":"2024-09-18T07:31:50.214811+0000","last_undegraded":"2024-09-18T07:31:50.214811+0000","last_fullsized":"2024-09-18T07:31:50.214811+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.816291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725733+0000","last_change":"2024-09-18T07:30:58.320305+0000","last_active":"2024-09-18T07:31:49.725733+0000","last_peered":"2024-09-18T07:31:49.725733+0000","last_clean":"2024-09-18T07:31:49.725733+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:31:49.725733+0000","last_undegraded":"2024-09-18T07:31:49.725733+0000","last_fullsized":"2024-09-18T07:31:49.725733+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.490659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213738+0000","last_change":"2024-09-18T07:30:56.290367+0000","last_active":"2024-09-18T07:31:50.213738+0000","last_peered":"2024-09-18T07:31:50.213738+0000","last_clean":"2024-09-18T07:31:50.213738+0000","last_became_active":"2024-09-18T07:30:56.289855+0000","last_became_peered":"2024-09-18T07:30:56.289855+0000","last_unstale":"2024-09-18T07:31:50.213738+0000","last_undegraded":"2024-09-18T07:31:50.213738+0000","last_fullsized":"2024-09-18T07:31:50.213738+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.319326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.727869+0000","last_change":"2024-09-18T07:31:00.329175+0000","last_active":"2024-09-18T07:31:49.727869+0000","last_peered":"2024-09-18T07:31:49.727869+0000","last_clean":"2024-09-18T07:31:49.727869+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:31:49.727869+0000","last_undegraded":"2024-09-18T07:31:49.727869+0000","last_fullsized":"2024-09-18T07:31:49.727869+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.309875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212847+0000","last_change":"2024-09-18T07:31:02.353267+0000","last_active":"2024-09-18T07:31:50.212847+0000","last_peered":"2024-09-18T07:31:50.212847+0000","last_clean":"2024-09-18T07:31:50.212847+0000","last_became_active":"2024-09-18T07:31:02.352922+0000","last_became_peered":"2024-09-18T07:31:02.352922+0000","last_unstale":"2024-09-18T07:31:50.212847+0000","last_undegraded":"2024-09-18T07:31:50.212847+0000","last_fullsized":"2024-09-18T07:31:50.212847+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:26.271964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212469+0000","last_change":"2024-09-18T07:30:58.319585+0000","last_active":"2024-09-18T07:31:50.212469+0000","last_peered":"2024-09-18T07:31:50.212469+0000","last_clean":"2024-09-18T07:31:50.212469+0000","last_became_active":"2024-09-18T07:30:58.319300+0000","last_became_peered":"2024-09-18T07:30:58.319300+0000","last_unstale":"2024-09-18T07:31:50.212469+0000","last_undegraded":"2024-09-18T07:31:50.212469+0000","last_fullsized":"2024-09-18T07:31:50.212469+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.143713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214578+0000","last_change":"2024-09-18T07:30:56.292832+0000","last_active":"2024-09-18T07:31:50.214578+0000","last_peered":"2024-09-18T07:31:50.214578+0000","last_clean":"2024-09-18T07:31:50.214578+0000","last_became_active":"2024-09-18T07:30:56.292531+0000","last_became_peered":"2024-09-18T07:30:56.292531+0000","last_unstale":"2024-09-18T07:31:50.214578+0000","last_undegraded":"2024-09-18T07:31:50.214578+0000","last_fullsized":"2024-09-18T07:31:50.214578+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.359238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:51.345698+0000","last_change":"2024-09-18T07:31:00.337915+0000","last_active":"2024-09-18T07:31:51.345698+0000","last_peered":"2024-09-18T07:31:51.345698+0000","last_clean":"2024-09-18T07:31:51.345698+0000","last_became_active":"2024-09-18T07:31:00.337483+0000","last_became_peered":"2024-09-18T07:31:00.337483+0000","last_unstale":"2024-09-18T07:31:51.345698+0000","last_undegraded":"2024-09-18T07:31:51.345698+0000","last_fullsized":"2024-09-18T07:31:51.345698+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:20.556343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212683+0000","last_change":"2024-09-18T07:31:02.489914+0000","last_active":"2024-09-18T07:31:50.212683+0000","last_peered":"2024-09-18T07:31:50.212683+0000","last_clean":"2024-09-18T07:31:50.212683+0000","last_became_active":"2024-09-18T07:31:02.489619+0000","last_became_peered":"2024-09-18T07:31:02.489619+0000","last_unstale":"2024-09-18T07:31:50.212683+0000","last_undegraded":"2024-09-18T07:31:50.212683+0000","last_fullsized":"2024-09-18T07:31:50.212683+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:17.709609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213098+0000","last_change":"2024-09-18T07:30:56.282629+0000","last_active":"2024-09-18T07:31:50.213098+0000","last_peered":"2024-09-18T07:31:50.213098+0000","last_clean":"2024-09-18T07:31:50.213098+0000","last_became_active":"2024-09-18T07:30:56.282357+0000","last_became_peered":"2024-09-18T07:30:56.282357+0000","last_unstale":"2024-09-18T07:31:50.213098+0000","last_undegraded":"2024-09-18T07:31:50.213098+0000","last_fullsized":"2024-09-18T07:31:50.213098+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.549682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":51,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213873+0000","last_change":"2024-09-18T07:30:58.320781+0000","last_active":"2024-09-18T07:31:50.213873+0000","last_peered":"2024-09-18T07:31:50.213873+0000","last_clean":"2024-09-18T07:31:50.213873+0000","last_became_active":"2024-09-18T07:30:58.320563+0000","last_became_peered":"2024-09-18T07:30:58.320563+0000","last_unstale":"2024-09-18T07:31:50.213873+0000","last_undegraded":"2024-09-18T07:31:50.213873+0000","last_fullsized":"2024-09-18T07:31:50.213873+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.842501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214119+0000","last_change":"2024-09-18T07:31:00.334155+0000","last_active":"2024-09-18T07:31:50.214119+0000","last_peered":"2024-09-18T07:31:50.214119+0000","last_clean":"2024-09-18T07:31:50.214119+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:31:50.214119+0000","last_undegraded":"2024-09-18T07:31:50.214119+0000","last_fullsized":"2024-09-18T07:31:50.214119+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.652219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213731+0000","last_change":"2024-09-18T07:31:02.492999+0000","last_active":"2024-09-18T07:31:50.213731+0000","last_peered":"2024-09-18T07:31:50.213731+0000","last_clean":"2024-09-18T07:31:50.213731+0000","last_became_active":"2024-09-18T07:31:02.492489+0000","last_became_peered":"2024-09-18T07:31:02.492489+0000","last_unstale":"2024-09-18T07:31:50.213731+0000","last_undegraded":"2024-09-18T07:31:50.213731+0000","last_fullsized":"2024-09-18T07:31:50.213731+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:11.972057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.729302+0000","last_change":"2024-09-18T07:30:58.318263+0000","last_active":"2024-09-18T07:31:49.729302+0000","last_peered":"2024-09-18T07:31:49.729302+0000","last_clean":"2024-09-18T07:31:49.729302+0000","last_became_active":"2024-09-18T07:30:58.317732+0000","last_became_peered":"2024-09-18T07:30:58.317732+0000","last_unstale":"2024-09-18T07:31:49.729302+0000","last_undegraded":"2024-09-18T07:31:49.729302+0000","last_fullsized":"2024-09-18T07:31:49.729302+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.024721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213564+0000","last_change":"2024-09-18T07:30:56.292272+0000","last_active":"2024-09-18T07:31:50.213564+0000","last_peered":"2024-09-18T07:31:50.213564+0000","last_clean":"2024-09-18T07:31:50.213564+0000","last_became_active":"2024-09-18T07:30:56.290788+0000","last_became_peered":"2024-09-18T07:30:56.290788+0000","last_unstale":"2024-09-18T07:31:50.213564+0000","last_undegraded":"2024-09-18T07:31:50.213564+0000","last_fullsized":"2024-09-18T07:31:50.213564+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.447855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213829+0000","last_change":"2024-09-18T07:31:00.334848+0000","last_active":"2024-09-18T07:31:50.213829+0000","last_peered":"2024-09-18T07:31:50.213829+0000","last_clean":"2024-09-18T07:31:50.213829+0000","last_became_active":"2024-09-18T07:31:00.334478+0000","last_became_peered":"2024-09-18T07:31:00.334478+0000","last_unstale":"2024-09-18T07:31:50.213829+0000","last_undegraded":"2024-09-18T07:31:50.213829+0000","last_fullsized":"2024-09-18T07:31:50.213829+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:16.087953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728225+0000","last_change":"2024-09-18T07:31:02.344766+0000","last_active":"2024-09-18T07:31:49.728225+0000","last_peered":"2024-09-18T07:31:49.728225+0000","last_clean":"2024-09-18T07:31:49.728225+0000","last_became_active":"2024-09-18T07:31:02.344011+0000","last_became_peered":"2024-09-18T07:31:02.344011+0000","last_unstale":"2024-09-18T07:31:49.728225+0000","last_undegraded":"2024-09-18T07:31:49.728225+0000","last_fullsized":"2024-09-18T07:31:49.728225+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.862393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214407+0000","last_change":"2024-09-18T07:30:58.318363+0000","last_active":"2024-09-18T07:31:50.214407+0000","last_peered":"2024-09-18T07:31:50.214407+0000","last_clean":"2024-09-18T07:31:50.214407+0000","last_became_active":"2024-09-18T07:30:58.318057+0000","last_became_peered":"2024-09-18T07:30:58.318057+0000","last_unstale":"2024-09-18T07:31:50.214407+0000","last_undegraded":"2024-09-18T07:31:50.214407+0000","last_fullsized":"2024-09-18T07:31:50.214407+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.599726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728179+0000","last_change":"2024-09-18T07:30:56.287385+0000","last_active":"2024-09-18T07:31:49.728179+0000","last_peered":"2024-09-18T07:31:49.728179+0000","last_clean":"2024-09-18T07:31:49.728179+0000","last_became_active":"2024-09-18T07:30:56.287068+0000","last_became_peered":"2024-09-18T07:30:56.287068+0000","last_unstale":"2024-09-18T07:31:49.728179+0000","last_undegraded":"2024-09-18T07:31:49.728179+0000","last_fullsized":"2024-09-18T07:31:49.728179+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.184849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:51.345493+0000","last_change":"2024-09-18T07:31:00.337716+0000","last_active":"2024-09-18T07:31:51.345493+0000","last_peered":"2024-09-18T07:31:51.345493+0000","last_clean":"2024-09-18T07:31:51.345493+0000","last_became_active":"2024-09-18T07:31:00.337444+0000","last_became_peered":"2024-09-18T07:31:00.337444+0000","last_unstale":"2024-09-18T07:31:51.345493+0000","last_undegraded":"2024-09-18T07:31:51.345493+0000","last_fullsized":"2024-09-18T07:31:51.345493+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.512126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726637+0000","last_change":"2024-09-18T07:31:02.343206+0000","last_active":"2024-09-18T07:31:49.726637+0000","last_peered":"2024-09-18T07:31:49.726637+0000","last_clean":"2024-09-18T07:31:49.726637+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:31:49.726637+0000","last_undegraded":"2024-09-18T07:31:49.726637+0000","last_fullsized":"2024-09-18T07:31:49.726637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:23.329902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213498+0000","last_change":"2024-09-18T07:30:58.320351+0000","last_active":"2024-09-18T07:31:50.213498+0000","last_peered":"2024-09-18T07:31:50.213498+0000","last_clean":"2024-09-18T07:31:50.213498+0000","last_became_active":"2024-09-18T07:30:58.320093+0000","last_became_peered":"2024-09-18T07:30:58.320093+0000","last_unstale":"2024-09-18T07:31:50.213498+0000","last_undegraded":"2024-09-18T07:31:50.213498+0000","last_fullsized":"2024-09-18T07:31:50.213498+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:04.687274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213549+0000","last_change":"2024-09-18T07:30:56.291807+0000","last_active":"2024-09-18T07:31:50.213549+0000","last_peered":"2024-09-18T07:31:50.213549+0000","last_clean":"2024-09-18T07:31:50.213549+0000","last_became_active":"2024-09-18T07:30:56.291238+0000","last_became_peered":"2024-09-18T07:30:56.291238+0000","last_unstale":"2024-09-18T07:31:50.213549+0000","last_undegraded":"2024-09-18T07:31:50.213549+0000","last_fullsized":"2024-09-18T07:31:50.213549+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:17.785800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":62,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:51.345714+0000","last_change":"2024-09-18T07:31:00.332729+0000","last_active":"2024-09-18T07:31:51.345714+0000","last_peered":"2024-09-18T07:31:51.345714+0000","last_clean":"2024-09-18T07:31:51.345714+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:31:51.345714+0000","last_undegraded":"2024-09-18T07:31:51.345714+0000","last_fullsized":"2024-09-18T07:31:51.345714+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:08.863683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728909+0000","last_change":"2024-09-18T07:31:02.348726+0000","last_active":"2024-09-18T07:31:49.728909+0000","last_peered":"2024-09-18T07:31:49.728909+0000","last_clean":"2024-09-18T07:31:49.728909+0000","last_became_active":"2024-09-18T07:31:02.348423+0000","last_became_peered":"2024-09-18T07:31:02.348423+0000","last_unstale":"2024-09-18T07:31:49.728909+0000","last_undegraded":"2024-09-18T07:31:49.728909+0000","last_fullsized":"2024-09-18T07:31:49.728909+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.760836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.725914+0000","last_change":"2024-09-18T07:30:58.314730+0000","last_active":"2024-09-18T07:31:49.725914+0000","last_peered":"2024-09-18T07:31:49.725914+0000","last_clean":"2024-09-18T07:31:49.725914+0000","last_became_active":"2024-09-18T07:30:58.314428+0000","last_became_peered":"2024-09-18T07:30:58.314428+0000","last_unstale":"2024-09-18T07:31:49.725914+0000","last_undegraded":"2024-09-18T07:31:49.725914+0000","last_fullsized":"2024-09-18T07:31:49.725914+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.641591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.726270+0000","last_change":"2024-09-18T07:30:56.291115+0000","last_active":"2024-09-18T07:31:49.726270+0000","last_peered":"2024-09-18T07:31:49.726270+0000","last_clean":"2024-09-18T07:31:49.726270+0000","last_became_active":"2024-09-18T07:30:56.290816+0000","last_became_peered":"2024-09-18T07:30:56.290816+0000","last_unstale":"2024-09-18T07:31:49.726270+0000","last_undegraded":"2024-09-18T07:31:49.726270+0000","last_fullsized":"2024-09-18T07:31:49.726270+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:12.554722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212261+0000","last_change":"2024-09-18T07:31:00.337826+0000","last_active":"2024-09-18T07:31:50.212261+0000","last_peered":"2024-09-18T07:31:50.212261+0000","last_clean":"2024-09-18T07:31:50.212261+0000","last_became_active":"2024-09-18T07:31:00.334158+0000","last_became_peered":"2024-09-18T07:31:00.334158+0000","last_unstale":"2024-09-18T07:31:50.212261+0000","last_undegraded":"2024-09-18T07:31:50.212261+0000","last_fullsized":"2024-09-18T07:31:50.212261+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:28.224028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212433+0000","last_change":"2024-09-18T07:31:02.351247+0000","last_active":"2024-09-18T07:31:50.212433+0000","last_peered":"2024-09-18T07:31:50.212433+0000","last_clean":"2024-09-18T07:31:50.212433+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:31:50.212433+0000","last_undegraded":"2024-09-18T07:31:50.212433+0000","last_fullsized":"2024-09-18T07:31:50.212433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:25.935005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212555+0000","last_change":"2024-09-18T07:30:58.307904+0000","last_active":"2024-09-18T07:31:50.212555+0000","last_peered":"2024-09-18T07:31:50.212555+0000","last_clean":"2024-09-18T07:31:50.212555+0000","last_became_active":"2024-09-18T07:30:58.307448+0000","last_became_peered":"2024-09-18T07:30:58.307448+0000","last_unstale":"2024-09-18T07:31:50.212555+0000","last_undegraded":"2024-09-18T07:31:50.212555+0000","last_fullsized":"2024-09-18T07:31:50.212555+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:01.900229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214035+0000","last_change":"2024-09-18T07:30:56.293467+0000","last_active":"2024-09-18T07:31:50.214035+0000","last_peered":"2024-09-18T07:31:50.214035+0000","last_clean":"2024-09-18T07:31:50.214035+0000","last_became_active":"2024-09-18T07:30:56.292980+0000","last_became_peered":"2024-09-18T07:30:56.292980+0000","last_unstale":"2024-09-18T07:31:50.214035+0000","last_undegraded":"2024-09-18T07:31:50.214035+0000","last_fullsized":"2024-09-18T07:31:50.214035+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:14.161233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212934+0000","last_change":"2024-09-18T07:31:00.333689+0000","last_active":"2024-09-18T07:31:50.212934+0000","last_peered":"2024-09-18T07:31:50.212934+0000","last_clean":"2024-09-18T07:31:50.212934+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:31:50.212934+0000","last_undegraded":"2024-09-18T07:31:50.212934+0000","last_fullsized":"2024-09-18T07:31:50.212934+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:19.075146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.727305+0000","last_change":"2024-09-18T07:31:02.352805+0000","last_active":"2024-09-18T07:31:49.727305+0000","last_peered":"2024-09-18T07:31:49.727305+0000","last_clean":"2024-09-18T07:31:49.727305+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:31:49.727305+0000","last_undegraded":"2024-09-18T07:31:49.727305+0000","last_fullsized":"2024-09-18T07:31:49.727305+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.115071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212634+0000","last_change":"2024-09-18T07:30:58.320419+0000","last_active":"2024-09-18T07:31:50.212634+0000","last_peered":"2024-09-18T07:31:50.212634+0000","last_clean":"2024-09-18T07:31:50.212634+0000","last_became_active":"2024-09-18T07:30:58.320108+0000","last_became_peered":"2024-09-18T07:30:58.320108+0000","last_unstale":"2024-09-18T07:31:50.212634+0000","last_undegraded":"2024-09-18T07:31:50.212634+0000","last_fullsized":"2024-09-18T07:31:50.212634+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:07.449011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214420+0000","last_change":"2024-09-18T07:30:56.289671+0000","last_active":"2024-09-18T07:31:50.214420+0000","last_peered":"2024-09-18T07:31:50.214420+0000","last_clean":"2024-09-18T07:31:50.214420+0000","last_became_active":"2024-09-18T07:30:56.289328+0000","last_became_peered":"2024-09-18T07:30:56.289328+0000","last_unstale":"2024-09-18T07:31:50.214420+0000","last_undegraded":"2024-09-18T07:31:50.214420+0000","last_fullsized":"2024-09-18T07:31:50.214420+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:58.531067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.214601+0000","last_change":"2024-09-18T07:31:00.336053+0000","last_active":"2024-09-18T07:31:50.214601+0000","last_peered":"2024-09-18T07:31:50.214601+0000","last_clean":"2024-09-18T07:31:50.214601+0000","last_became_active":"2024-09-18T07:31:00.335749+0000","last_became_peered":"2024-09-18T07:31:00.335749+0000","last_unstale":"2024-09-18T07:31:50.214601+0000","last_undegraded":"2024-09-18T07:31:50.214601+0000","last_fullsized":"2024-09-18T07:31:50.214601+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:26.183381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213076+0000","last_change":"2024-09-18T07:31:02.491487+0000","last_active":"2024-09-18T07:31:50.213076+0000","last_peered":"2024-09-18T07:31:50.213076+0000","last_clean":"2024-09-18T07:31:50.213076+0000","last_became_active":"2024-09-18T07:31:02.490994+0000","last_became_peered":"2024-09-18T07:31:02.490994+0000","last_unstale":"2024-09-18T07:31:50.213076+0000","last_undegraded":"2024-09-18T07:31:50.213076+0000","last_fullsized":"2024-09-18T07:31:50.213076+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:10.401313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212321+0000","last_change":"2024-09-18T07:30:58.316173+0000","last_active":"2024-09-18T07:31:50.212321+0000","last_peered":"2024-09-18T07:31:50.212321+0000","last_clean":"2024-09-18T07:31:50.212321+0000","last_became_active":"2024-09-18T07:30:58.315435+0000","last_became_peered":"2024-09-18T07:30:58.315435+0000","last_unstale":"2024-09-18T07:31:50.212321+0000","last_undegraded":"2024-09-18T07:31:50.212321+0000","last_fullsized":"2024-09-18T07:31:50.212321+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:05.325393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.728765+0000","last_change":"2024-09-18T07:30:56.283643+0000","last_active":"2024-09-18T07:31:49.728765+0000","last_peered":"2024-09-18T07:31:49.728765+0000","last_clean":"2024-09-18T07:31:49.728765+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:31:49.728765+0000","last_undegraded":"2024-09-18T07:31:49.728765+0000","last_fullsized":"2024-09-18T07:31:49.728765+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:17.971275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212610+0000","last_change":"2024-09-18T07:31:00.333126+0000","last_active":"2024-09-18T07:31:50.212610+0000","last_peered":"2024-09-18T07:31:50.212610+0000","last_clean":"2024-09-18T07:31:50.212610+0000","last_became_active":"2024-09-18T07:31:00.332855+0000","last_became_peered":"2024-09-18T07:31:00.332855+0000","last_unstale":"2024-09-18T07:31:50.212610+0000","last_undegraded":"2024-09-18T07:31:50.212610+0000","last_fullsized":"2024-09-18T07:31:50.212610+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:29.188004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213659+0000","last_change":"2024-09-18T07:31:02.350978+0000","last_active":"2024-09-18T07:31:50.213659+0000","last_peered":"2024-09-18T07:31:50.213659+0000","last_clean":"2024-09-18T07:31:50.213659+0000","last_became_active":"2024-09-18T07:31:02.350692+0000","last_became_peered":"2024-09-18T07:31:02.350692+0000","last_unstale":"2024-09-18T07:31:50.213659+0000","last_undegraded":"2024-09-18T07:31:50.213659+0000","last_fullsized":"2024-09-18T07:31:50.213659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:02.916468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212923+0000","last_change":"2024-09-18T07:31:02.347827+0000","last_active":"2024-09-18T07:31:50.212923+0000","last_peered":"2024-09-18T07:31:50.212923+0000","last_clean":"2024-09-18T07:31:50.212923+0000","last_became_active":"2024-09-18T07:31:02.347537+0000","last_became_peered":"2024-09-18T07:31:02.347537+0000","last_unstale":"2024-09-18T07:31:50.212923+0000","last_undegraded":"2024-09-18T07:31:50.212923+0000","last_fullsized":"2024-09-18T07:31:50.212923+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:28.722445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":52,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.212504+0000","last_change":"2024-09-18T07:30:56.292657+0000","last_active":"2024-09-18T07:31:50.212504+0000","last_peered":"2024-09-18T07:31:50.212504+0000","last_clean":"2024-09-18T07:31:50.212504+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:31:50.212504+0000","last_undegraded":"2024-09-18T07:31:50.212504+0000","last_fullsized":"2024-09-18T07:31:50.212504+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:13.946464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:50.213764+0000","last_change":"2024-09-18T07:30:58.314191+0000","last_active":"2024-09-18T07:31:50.213764+0000","last_peered":"2024-09-18T07:31:50.213764+0000","last_clean":"2024-09-18T07:31:50.213764+0000","last_became_active":"2024-09-18T07:30:58.313556+0000","last_became_peered":"2024-09-18T07:30:58.313556+0000","last_unstale":"2024-09-18T07:31:50.213764+0000","last_undegraded":"2024-09-18T07:31:50.213764+0000","last_fullsized":"2024-09-18T07:31:50.213764+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:24.554527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-18T07:31:49.729038+0000","last_change":"2024-09-18T07:31:00.331995+0000","last_active":"2024-09-18T07:31:49.729038+0000","last_peered":"2024-09-18T07:31:49.729038+0000","last_clean":"2024-09-18T07:31:49.729038+0000","last_became_active":"2024-09-18T07:31:00.331564+0000","last_became_peered":"2024-09-18T07:31:00.331564+0000","last_unstale":"2024-09-18T07:31:49.729038+0000","last_undegraded":"2024-09-18T07:31:49.729038+0000","last_fullsized":"2024-09-18T07:31:49.729038+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:21.738488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":50,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201296,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28384,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716768,"statfs":{"total":95995035648,"available":95965970432,"internally_reserved":0,"allocated":1388544,"data_stored":908644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462934,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32524,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712628,"statfs":{"total":95995035648,"available":95961731072,"internally_reserved":0,"allocated":1368064,"data_stored":907559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724571,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31944,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713208,"statfs":{"total":95995035648,"available":95962324992,"internally_reserved":0,"allocated":774144,"data_stored":317537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953506,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":802816,"data_stored":318219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27828,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717324,"statfs":{"total":95995035648,"available":95966539776,"internally_reserved":0,"allocated":819200,"data_stored":319511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27776,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717376,"statfs":{"total":95995035648,"available":95966593024,"internally_reserved":0,"allocated":765952,"data_stored":317326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32008,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713144,"statfs":{"total":95995035648,"available":95962259456,"internally_reserved":0,"allocated":839680,"data_stored":319956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28392,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716760,"statfs":{"total":95995035648,"available":95965962240,"internally_reserved":0,"allocated":1396736,"data_stored":909769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:31:59.790 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-18T07:31:59.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd dump 2024-09-18T07:32:00.038 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:00.158 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:00 smithi137 ceph-mon[29408]: from='client.14808 v1:172.21.15.137:0/1326749394' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:32:00.158 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:00 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:32:00.158 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:00 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:32:00.158 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:00 smithi137 ceph-mon[29408]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:32:00.159 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:00 smithi137 ceph-mon[29408]: 3.2 scrub starts 2024-09-18T07:32:00.159 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:00 smithi137 ceph-mon[29408]: 3.2 scrub ok 2024-09-18T07:32:00.159 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:00 smithi137 ceph-mon[39175]: from='client.14808 v1:172.21.15.137:0/1326749394' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:32:00.159 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:00 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:32:00.159 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:00 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:32:00.159 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:00 smithi137 ceph-mon[39175]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:32:00.159 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:00 smithi137 ceph-mon[39175]: 3.2 scrub starts 2024-09-18T07:32:00.159 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:00 smithi137 ceph-mon[39175]: 3.2 scrub ok 2024-09-18T07:32:00.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:00 smithi203 ceph-mon[32237]: from='client.14808 v1:172.21.15.137:0/1326749394' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:32:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:00 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:32:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:00 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:32:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:00 smithi203 ceph-mon[32237]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:32:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:00 smithi203 ceph-mon[32237]: 3.2 scrub starts 2024-09-18T07:32:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:00 smithi203 ceph-mon[32237]: 3.2 scrub ok 2024-09-18T07:32:00.745 INFO:teuthology.orchestra.run.smithi137.stdout:epoch 79 2024-09-18T07:32:00.745 INFO:teuthology.orchestra.run.smithi137.stdout:fsid f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:32:00.745 INFO:teuthology.orchestra.run.smithi137.stdout:created 2024-09-18T07:25:19.662367+0000 2024-09-18T07:32:00.745 INFO:teuthology.orchestra.run.smithi137.stdout:modified 2024-09-18T07:31:59.996817+0000 2024-09-18T07:32:00.745 INFO:teuthology.orchestra.run.smithi137.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-18T07:32:00.745 INFO:teuthology.orchestra.run.smithi137.stdout:crush_version 18 2024-09-18T07:32:00.745 INFO:teuthology.orchestra.run.smithi137.stdout:full_ratio 0.95 2024-09-18T07:32:00.745 INFO:teuthology.orchestra.run.smithi137.stdout:backfillfull_ratio 0.9 2024-09-18T07:32:00.745 INFO:teuthology.orchestra.run.smithi137.stdout:nearfull_ratio 0.85 2024-09-18T07:32:00.746 INFO:teuthology.orchestra.run.smithi137.stdout:require_min_compat_client luminous 2024-09-18T07:32:00.746 INFO:teuthology.orchestra.run.smithi137.stdout:min_compat_client jewel 2024-09-18T07:32:00.746 INFO:teuthology.orchestra.run.smithi137.stdout:require_osd_release squid 2024-09-18T07:32:00.746 INFO:teuthology.orchestra.run.smithi137.stdout:stretch_mode_enabled false 2024-09-18T07:32:00.746 INFO:teuthology.orchestra.run.smithi137.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-18T07:32:00.746 INFO:teuthology.orchestra.run.smithi137.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 63 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-09-18T07:32:00.746 INFO:teuthology.orchestra.run.smithi137.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 62 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-09-18T07:32:00.746 INFO:teuthology.orchestra.run.smithi137.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 64 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-09-18T07:32:00.746 INFO:teuthology.orchestra.run.smithi137.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 66 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-09-18T07:32:00.746 INFO:teuthology.orchestra.run.smithi137.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 68 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-09-18T07:32:00.746 INFO:teuthology.orchestra.run.smithi137.stdout:max_osd 8 2024-09-18T07:32:00.746 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 up in weight 1 up_from 8 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6801/2811444477 v1:172.21.15.137:6802/2811444477 exists,up 268cd580-c931-4992-bb93-d6f63c2bfe88 2024-09-18T07:32:00.747 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 up in weight 1 up_from 13 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6805/874102487 v1:172.21.15.137:6806/874102487 exists,up b06907bc-b499-46c0-8667-c5bdc0d85575 2024-09-18T07:32:00.747 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 up in weight 1 up_from 18 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6809/1754684686 v1:172.21.15.137:6810/1754684686 exists,up 480ad6c5-0e0b-4405-bbe2-381682d00bae 2024-09-18T07:32:00.747 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 up in weight 1 up_from 26 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6813/116058392 v1:172.21.15.137:6814/116058392 exists,up 178be3e0-f07b-4856-807b-2a1181e48a14 2024-09-18T07:32:00.747 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 up in weight 1 up_from 32 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.203:6800/3654297954 v1:172.21.15.203:6801/3654297954 exists,up 7547102c-951a-4077-a094-fb8a65bfe6de 2024-09-18T07:32:00.747 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 up in weight 1 up_from 39 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.203:6804/837994222 v1:172.21.15.203:6805/837994222 exists,up 49c17901-8102-48a4-9348-fa1bdb63032c 2024-09-18T07:32:00.747 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 up in weight 1 up_from 47 up_thru 64 down_at 0 last_clean_interval [0,0) v1:172.21.15.203:6808/1494175466 v1:172.21.15.203:6809/1494175466 exists,up 75f0ead4-e981-4a15-b1c1-d83dce871ab6 2024-09-18T07:32:00.747 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 up in weight 1 up_from 55 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.203:6812/2670809315 v1:172.21.15.203:6813/2670809315 exists,up c0dbf299-6798-4b99-9a85-62b123971f2a 2024-09-18T07:32:00.747 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:6800/2096039404 expires 2024-09-19T07:31:25.389124+0000 2024-09-18T07:32:00.747 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:6800/3636177939 expires 2024-09-19T07:25:41.661328+0000 2024-09-18T07:32:00.747 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/334967443 expires 2024-09-19T07:25:41.661328+0000 2024-09-18T07:32:00.747 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/2101286111 expires 2024-09-19T07:25:41.661328+0000 2024-09-18T07:32:00.747 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/3387637412 expires 2024-09-19T07:26:06.595443+0000 2024-09-18T07:32:00.747 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/1762989503 expires 2024-09-19T07:31:25.389124+0000 2024-09-18T07:32:00.747 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/3490817671 expires 2024-09-19T07:26:06.595443+0000 2024-09-18T07:32:00.748 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:6800/1482147392 expires 2024-09-19T07:26:06.595443+0000 2024-09-18T07:32:00.748 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/1821482181 expires 2024-09-19T07:25:41.661328+0000 2024-09-18T07:32:00.748 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/4028304023 expires 2024-09-19T07:26:06.595443+0000 2024-09-18T07:32:00.748 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/621434521 expires 2024-09-19T07:31:25.389124+0000 2024-09-18T07:32:00.748 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/3160704935 expires 2024-09-19T07:31:25.389124+0000 2024-09-18T07:32:00.748 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/2357293499 expires 2024-09-19T07:31:25.389124+0000 2024-09-18T07:32:00.748 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/1762352860 expires 2024-09-19T07:31:25.389124+0000 2024-09-18T07:32:01.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:01 smithi203 ceph-mon[32237]: osdmap e79: 8 total, 8 up, 8 in 2024-09-18T07:32:01.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:01 smithi203 ceph-mon[32237]: 4.0 scrub starts 2024-09-18T07:32:01.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:01 smithi203 ceph-mon[32237]: 4.0 scrub ok 2024-09-18T07:32:01.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:01 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:01.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:01 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3986555468' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-18T07:32:01.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:01 smithi137 ceph-mon[39175]: osdmap e79: 8 total, 8 up, 8 in 2024-09-18T07:32:01.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:01 smithi137 ceph-mon[39175]: 4.0 scrub starts 2024-09-18T07:32:01.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:01 smithi137 ceph-mon[39175]: 4.0 scrub ok 2024-09-18T07:32:01.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:01 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:01.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:01 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3986555468' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-18T07:32:01.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:01 smithi137 ceph-mon[29408]: osdmap e79: 8 total, 8 up, 8 in 2024-09-18T07:32:01.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:01 smithi137 ceph-mon[29408]: 4.0 scrub starts 2024-09-18T07:32:01.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:01 smithi137 ceph-mon[29408]: 4.0 scrub ok 2024-09-18T07:32:01.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:01 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:01.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:01 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3986555468' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-18T07:32:01.474 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6801/2811444477 v1:172.21.15.137:6802/2811444477 exists,up 268cd580-c931-4992-bb93-d6f63c2bfe88', 'osd.1 up in weight 1 up_from 13 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6805/874102487 v1:172.21.15.137:6806/874102487 exists,up b06907bc-b499-46c0-8667-c5bdc0d85575', 'osd.2 up in weight 1 up_from 18 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6809/1754684686 v1:172.21.15.137:6810/1754684686 exists,up 480ad6c5-0e0b-4405-bbe2-381682d00bae', 'osd.3 up in weight 1 up_from 26 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6813/116058392 v1:172.21.15.137:6814/116058392 exists,up 178be3e0-f07b-4856-807b-2a1181e48a14', 'osd.4 up in weight 1 up_from 32 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.203:6800/3654297954 v1:172.21.15.203:6801/3654297954 exists,up 7547102c-951a-4077-a094-fb8a65bfe6de', 'osd.5 up in weight 1 up_from 39 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.203:6804/837994222 v1:172.21.15.203:6805/837994222 exists,up 49c17901-8102-48a4-9348-fa1bdb63032c', 'osd.6 up in weight 1 up_from 47 up_thru 64 down_at 0 last_clean_interval [0,0) v1:172.21.15.203:6808/1494175466 v1:172.21.15.203:6809/1494175466 exists,up 75f0ead4-e981-4a15-b1c1-d83dce871ab6', 'osd.7 up in weight 1 up_from 55 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.203:6812/2670809315 v1:172.21.15.203:6813/2670809315 exists,up c0dbf299-6798-4b99-9a85-62b123971f2a'] 2024-09-18T07:32:01.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon mon.a config show 2024-09-18T07:32:01.719 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:02.110 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:02 smithi137 ceph-mon[29408]: 4.a scrub starts 2024-09-18T07:32:02.110 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:02 smithi137 ceph-mon[29408]: 4.a scrub ok 2024-09-18T07:32:02.110 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:02 smithi137 ceph-mon[29408]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:32:02.110 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:02 smithi137 ceph-mon[39175]: 4.a scrub starts 2024-09-18T07:32:02.110 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:02 smithi137 ceph-mon[39175]: 4.a scrub ok 2024-09-18T07:32:02.110 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:02 smithi137 ceph-mon[39175]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:32:02.204 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:32:02.204 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "mon.a", 2024-09-18T07:32:02.204 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster": "ceph", 2024-09-18T07:32:02.205 INFO:teuthology.orchestra.run.smithi137.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-18T07:32:02.205 INFO:teuthology.orchestra.run.smithi137.stdout: "admin_socket_mode": "", 2024-09-18T07:32:02.205 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-18T07:32:02.205 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_client_required": "cephx, none", 2024-09-18T07:32:02.205 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_cluster_required": "cephx", 2024-09-18T07:32:02.205 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_debug": "true", 2024-09-18T07:32:02.205 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-18T07:32:02.205 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-18T07:32:02.205 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_service_required": "cephx", 2024-09-18T07:32:02.205 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-18T07:32:02.205 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_supported": "", 2024-09-18T07:32:02.206 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_aio": "true", 2024-09-18T07:32:02.206 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-18T07:32:02.206 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_aio_poll_ms": "250", 2024-09-18T07:32:02.206 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_aio_reap_max": "16", 2024-09-18T07:32:02.206 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_async_discard_threads": "0", 2024-09-18T07:32:02.206 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_block_size": "4096", 2024-09-18T07:32:02.206 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_debug_aio": "false", 2024-09-18T07:32:02.206 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-18T07:32:02.206 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-18T07:32:02.206 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_debug_inflight_ios": "false", 2024-09-18T07:32:02.206 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_enable_discard": "false", 2024-09-18T07:32:02.207 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_flock_retry": "3", 2024-09-18T07:32:02.207 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-18T07:32:02.207 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_inject_crash": "0", 2024-09-18T07:32:02.207 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-18T07:32:02.207 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_ioring": "false", 2024-09-18T07:32:02.207 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_ioring_hipri": "false", 2024-09-18T07:32:02.207 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-18T07:32:02.207 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-18T07:32:02.207 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-18T07:32:02.207 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-18T07:32:02.207 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-18T07:32:02.207 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-18T07:32:02.208 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_type": "", 2024-09-18T07:32:02.208 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_alloc_size": "1048576", 2024-09-18T07:32:02.208 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_allocator": "hybrid", 2024-09-18T07:32:02.208 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_buffered_io": "true", 2024-09-18T07:32:02.208 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_check_for_zeros": "false", 2024-09-18T07:32:02.208 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-18T07:32:02.208 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-18T07:32:02.208 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_compact_log_sync": "false", 2024-09-18T07:32:02.208 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-18T07:32:02.208 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-18T07:32:02.208 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-18T07:32:02.209 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-18T07:32:02.209 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_max_log_runway": "4194304", 2024-09-18T07:32:02.209 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_max_prefetch": "1048576", 2024-09-18T07:32:02.209 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_min_flush_size": "524288", 2024-09-18T07:32:02.209 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_min_log_runway": "1048576", 2024-09-18T07:32:02.209 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_replay_recovery": "false", 2024-09-18T07:32:02.209 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-18T07:32:02.209 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-18T07:32:02.209 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_sync_write": "false", 2024-09-18T07:32:02.209 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-18T07:32:02.209 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-18T07:32:02.209 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-18T07:32:02.210 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_allocation_from_file": "true", 2024-09-18T07:32:02.210 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_allocator": "hybrid", 2024-09-18T07:32:02.210 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_async_db_compaction": "true", 2024-09-18T07:32:02.210 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-18T07:32:02.210 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-18T07:32:02.210 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-18T07:32:02.210 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-18T07:32:02.210 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bdev_label_multi": "true", 2024-09-18T07:32:02.210 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-18T07:32:02.210 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-18T07:32:02.210 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-18T07:32:02.210 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-18T07:32:02.210 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-18T07:32:02.210 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_create": "true", 2024-09-18T07:32:02.211 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_db_create": "false", 2024-09-18T07:32:02.211 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_db_path": "", 2024-09-18T07:32:02.211 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_db_size": "0", 2024-09-18T07:32:02.211 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_path": "", 2024-09-18T07:32:02.211 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_preallocate_file": "false", 2024-09-18T07:32:02.211 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_size": "107374182400", 2024-09-18T07:32:02.211 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_wal_create": "false", 2024-09-18T07:32:02.211 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_wal_path": "", 2024-09-18T07:32:02.211 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_wal_size": "100663296", 2024-09-18T07:32:02.211 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bluefs": "true", 2024-09-18T07:32:02.211 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-18T07:32:02.211 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-18T07:32:02.212 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-18T07:32:02.212 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-18T07:32:02.212 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-18T07:32:02.212 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-18T07:32:02.212 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-18T07:32:02.212 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-18T07:32:02.212 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-18T07:32:02.212 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_autotune": "true", 2024-09-18T07:32:02.212 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-18T07:32:02.212 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-18T07:32:02.212 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-18T07:32:02.212 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-18T07:32:02.212 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_size": "0", 2024-09-18T07:32:02.213 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-18T07:32:02.213 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-18T07:32:02.213 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-18T07:32:02.213 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-18T07:32:02.213 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_type": "2q", 2024-09-18T07:32:02.213 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-18T07:32:02.213 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_clone_cow": "true", 2024-09-18T07:32:02.213 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-18T07:32:02.213 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-18T07:32:02.213 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-18T07:32:02.213 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-18T07:32:02.213 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-18T07:32:02.214 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-18T07:32:02.214 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-18T07:32:02.214 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_mode": "none", 2024-09-18T07:32:02.214 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-18T07:32:02.214 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_csum_type": "crc32c", 2024-09-18T07:32:02.214 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-18T07:32:02.214 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-18T07:32:02.214 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_freelist": "false", 2024-09-18T07:32:02.214 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-18T07:32:02.214 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-18T07:32:02.214 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-18T07:32:02.214 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-18T07:32:02.214 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-18T07:32:02.215 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-18T07:32:02.215 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-18T07:32:02.215 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-18T07:32:02.215 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-18T07:32:02.215 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-18T07:32:02.215 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-18T07:32:02.215 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-18T07:32:02.215 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_small_allocations": "0", 2024-09-18T07:32:02.215 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-18T07:32:02.215 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_default_buffered_read": "true", 2024-09-18T07:32:02.215 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_default_buffered_write": "false", 2024-09-18T07:32:02.215 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-18T07:32:02.216 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-18T07:32:02.216 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-18T07:32:02.216 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-18T07:32:02.216 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-18T07:32:02.216 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-18T07:32:02.216 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-18T07:32:02.216 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-18T07:32:02.216 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-18T07:32:02.216 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fail_eio": "false", 2024-09-18T07:32:02.216 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-18T07:32:02.216 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-18T07:32:02.217 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-18T07:32:02.217 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-18T07:32:02.217 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-18T07:32:02.217 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-18T07:32:02.217 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_on_mount": "false", 2024-09-18T07:32:02.217 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-18T07:32:02.217 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_on_umount": "false", 2024-09-18T07:32:02.217 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-18T07:32:02.217 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-18T07:32:02.217 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-18T07:32:02.217 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-18T07:32:02.217 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-18T07:32:02.218 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-18T07:32:02.218 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-18T07:32:02.218 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-18T07:32:02.218 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_ignore_data_csum": "false", 2024-09-18T07:32:02.218 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-18T07:32:02.218 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-18T07:32:02.218 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-18T07:32:02.218 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-18T07:32:02.218 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_log_op_age": "5.000000", 2024-09-18T07:32:02.218 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_max_alloc_size": "0", 2024-09-18T07:32:02.218 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_max_blob_size": "0", 2024-09-18T07:32:02.218 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-18T07:32:02.218 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-18T07:32:02.219 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-18T07:32:02.219 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_max_deferred_txc": "32", 2024-09-18T07:32:02.219 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_min_alloc_size": "0", 2024-09-18T07:32:02.219 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-18T07:32:02.219 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-18T07:32:02.219 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_nid_prealloc": "1024", 2024-09-18T07:32:02.219 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-18T07:32:02.219 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-18T07:32:02.219 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-18T07:32:02.219 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-18T07:32:02.219 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_retry_disk_reads": "3", 2024-09-18T07:32:02.219 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_rocksdb_cf": "true", 2024-09-18T07:32:02.219 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-18T07:32:02.220 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-18T07:32:02.220 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-18T07:32:02.220 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-18T07:32:02.220 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-18T07:32:02.220 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-18T07:32:02.220 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-18T07:32:02.220 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-18T07:32:02.220 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_spdk_mem": "512", 2024-09-18T07:32:02.220 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-18T07:32:02.220 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-18T07:32:02.220 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-18T07:32:02.220 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-18T07:32:02.220 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-18T07:32:02.220 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-18T07:32:02.221 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-18T07:32:02.221 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_tracing": "false", 2024-09-18T07:32:02.221 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-18T07:32:02.221 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-18T07:32:02.221 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-18T07:32:02.221 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-18T07:32:02.221 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-18T07:32:02.221 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-18T07:32:02.221 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-18T07:32:02.221 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-18T07:32:02.221 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-18T07:32:02.221 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_write_v2": "false", 2024-09-18T07:32:02.221 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_write_v2_random": "false", 2024-09-18T07:32:02.221 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_zero_block_detection": "false", 2024-09-18T07:32:02.222 INFO:teuthology.orchestra.run.smithi137.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-18T07:32:02.222 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-18T07:32:02.222 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-18T07:32:02.222 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-18T07:32:02.222 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-18T07:32:02.222 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-18T07:32:02.222 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-18T07:32:02.222 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-18T07:32:02.222 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-18T07:32:02.222 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-18T07:32:02.222 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-18T07:32:02.222 INFO:teuthology.orchestra.run.smithi137.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-18T07:32:02.223 INFO:teuthology.orchestra.run.smithi137.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-18T07:32:02.223 INFO:teuthology.orchestra.run.smithi137.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-18T07:32:02.223 INFO:teuthology.orchestra.run.smithi137.stdout: "cephx_cluster_require_signatures": "false", 2024-09-18T07:32:02.223 INFO:teuthology.orchestra.run.smithi137.stdout: "cephx_cluster_require_version": "2", 2024-09-18T07:32:02.223 INFO:teuthology.orchestra.run.smithi137.stdout: "cephx_require_signatures": "false", 2024-09-18T07:32:02.223 INFO:teuthology.orchestra.run.smithi137.stdout: "cephx_require_version": "2", 2024-09-18T07:32:02.223 INFO:teuthology.orchestra.run.smithi137.stdout: "cephx_service_require_signatures": "false", 2024-09-18T07:32:02.223 INFO:teuthology.orchestra.run.smithi137.stdout: "cephx_service_require_version": "2", 2024-09-18T07:32:02.223 INFO:teuthology.orchestra.run.smithi137.stdout: "cephx_sign_messages": "true", 2024-09-18T07:32:02.223 INFO:teuthology.orchestra.run.smithi137.stdout: "chdir": "", 2024-09-18T07:32:02.223 INFO:teuthology.orchestra.run.smithi137.stdout: "client_acl_type": "", 2024-09-18T07:32:02.223 INFO:teuthology.orchestra.run.smithi137.stdout: "client_asio_thread_count": "2", 2024-09-18T07:32:02.224 INFO:teuthology.orchestra.run.smithi137.stdout: "client_cache_mid": "0.750000", 2024-09-18T07:32:02.224 INFO:teuthology.orchestra.run.smithi137.stdout: "client_cache_size": "16384", 2024-09-18T07:32:02.224 INFO:teuthology.orchestra.run.smithi137.stdout: "client_caps_release_delay": "5", 2024-09-18T07:32:02.224 INFO:teuthology.orchestra.run.smithi137.stdout: "client_check_pool_perm": "true", 2024-09-18T07:32:02.224 INFO:teuthology.orchestra.run.smithi137.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-18T07:32:02.224 INFO:teuthology.orchestra.run.smithi137.stdout: "client_debug_force_sync_read": "false", 2024-09-18T07:32:02.224 INFO:teuthology.orchestra.run.smithi137.stdout: "client_debug_getattr_caps": "false", 2024-09-18T07:32:02.224 INFO:teuthology.orchestra.run.smithi137.stdout: "client_debug_inject_features": "", 2024-09-18T07:32:02.224 INFO:teuthology.orchestra.run.smithi137.stdout: "client_debug_inject_tick_delay": "0", 2024-09-18T07:32:02.224 INFO:teuthology.orchestra.run.smithi137.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-18T07:32:02.224 INFO:teuthology.orchestra.run.smithi137.stdout: "client_die_on_failed_remount": "false", 2024-09-18T07:32:02.224 INFO:teuthology.orchestra.run.smithi137.stdout: "client_dirsize_rbytes": "true", 2024-09-18T07:32:02.225 INFO:teuthology.orchestra.run.smithi137.stdout: "client_force_lazyio": "false", 2024-09-18T07:32:02.225 INFO:teuthology.orchestra.run.smithi137.stdout: "client_fs": "", 2024-09-18T07:32:02.225 INFO:teuthology.orchestra.run.smithi137.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-18T07:32:02.225 INFO:teuthology.orchestra.run.smithi137.stdout: "client_inject_release_failure": "false", 2024-09-18T07:32:02.225 INFO:teuthology.orchestra.run.smithi137.stdout: "client_max_inline_size": "4096", 2024-09-18T07:32:02.225 INFO:teuthology.orchestra.run.smithi137.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-18T07:32:02.225 INFO:teuthology.orchestra.run.smithi137.stdout: "client_mds_namespace": "", 2024-09-18T07:32:02.225 INFO:teuthology.orchestra.run.smithi137.stdout: "client_metadata": "", 2024-09-18T07:32:02.225 INFO:teuthology.orchestra.run.smithi137.stdout: "client_mount_gid": "-1", 2024-09-18T07:32:02.225 INFO:teuthology.orchestra.run.smithi137.stdout: "client_mount_timeout": "300", 2024-09-18T07:32:02.225 INFO:teuthology.orchestra.run.smithi137.stdout: "client_mount_uid": "-1", 2024-09-18T07:32:02.225 INFO:teuthology.orchestra.run.smithi137.stdout: "client_mountpoint": "/", 2024-09-18T07:32:02.225 INFO:teuthology.orchestra.run.smithi137.stdout: "client_notify_timeout": "10", 2024-09-18T07:32:02.226 INFO:teuthology.orchestra.run.smithi137.stdout: "client_oc": "true", 2024-09-18T07:32:02.226 INFO:teuthology.orchestra.run.smithi137.stdout: "client_oc_max_dirty": "104857600", 2024-09-18T07:32:02.226 INFO:teuthology.orchestra.run.smithi137.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-18T07:32:02.226 INFO:teuthology.orchestra.run.smithi137.stdout: "client_oc_max_objects": "1000", 2024-09-18T07:32:02.226 INFO:teuthology.orchestra.run.smithi137.stdout: "client_oc_size": "209715200", 2024-09-18T07:32:02.226 INFO:teuthology.orchestra.run.smithi137.stdout: "client_oc_target_dirty": "8388608", 2024-09-18T07:32:02.226 INFO:teuthology.orchestra.run.smithi137.stdout: "client_permissions": "true", 2024-09-18T07:32:02.226 INFO:teuthology.orchestra.run.smithi137.stdout: "client_quota": "true", 2024-09-18T07:32:02.226 INFO:teuthology.orchestra.run.smithi137.stdout: "client_quota_df": "true", 2024-09-18T07:32:02.226 INFO:teuthology.orchestra.run.smithi137.stdout: "client_readahead_max_bytes": "0", 2024-09-18T07:32:02.226 INFO:teuthology.orchestra.run.smithi137.stdout: "client_readahead_max_periods": "4", 2024-09-18T07:32:02.226 INFO:teuthology.orchestra.run.smithi137.stdout: "client_readahead_min": "131072", 2024-09-18T07:32:02.226 INFO:teuthology.orchestra.run.smithi137.stdout: "client_reconnect_stale": "false", 2024-09-18T07:32:02.226 INFO:teuthology.orchestra.run.smithi137.stdout: "client_shutdown_timeout": "30", 2024-09-18T07:32:02.227 INFO:teuthology.orchestra.run.smithi137.stdout: "client_snapdir": ".snap", 2024-09-18T07:32:02.227 INFO:teuthology.orchestra.run.smithi137.stdout: "client_tick_interval": "1", 2024-09-18T07:32:02.227 INFO:teuthology.orchestra.run.smithi137.stdout: "client_trace": "", 2024-09-18T07:32:02.227 INFO:teuthology.orchestra.run.smithi137.stdout: "client_try_dentry_invalidate": "false", 2024-09-18T07:32:02.227 INFO:teuthology.orchestra.run.smithi137.stdout: "client_use_faked_inos": "false", 2024-09-18T07:32:02.227 INFO:teuthology.orchestra.run.smithi137.stdout: "client_use_random_mds": "false", 2024-09-18T07:32:02.227 INFO:teuthology.orchestra.run.smithi137.stdout: "clog_to_graylog": "false", 2024-09-18T07:32:02.227 INFO:teuthology.orchestra.run.smithi137.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-18T07:32:02.227 INFO:teuthology.orchestra.run.smithi137.stdout: "clog_to_graylog_port": "12201", 2024-09-18T07:32:02.227 INFO:teuthology.orchestra.run.smithi137.stdout: "clog_to_monitors": "default=true", 2024-09-18T07:32:02.227 INFO:teuthology.orchestra.run.smithi137.stdout: "clog_to_syslog": "false", 2024-09-18T07:32:02.227 INFO:teuthology.orchestra.run.smithi137.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-18T07:32:02.228 INFO:teuthology.orchestra.run.smithi137.stdout: "clog_to_syslog_level": "info", 2024-09-18T07:32:02.228 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "-", 2024-09-18T07:32:02.228 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_network": "", 2024-09-18T07:32:02.228 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_network_interface": "", 2024-09-18T07:32:02.228 INFO:teuthology.orchestra.run.smithi137.stdout: "compressor_zlib_isal": "false", 2024-09-18T07:32:02.228 INFO:teuthology.orchestra.run.smithi137.stdout: "compressor_zlib_level": "5", 2024-09-18T07:32:02.228 INFO:teuthology.orchestra.run.smithi137.stdout: "compressor_zlib_winsize": "-15", 2024-09-18T07:32:02.228 INFO:teuthology.orchestra.run.smithi137.stdout: "compressor_zstd_level": "1", 2024-09-18T07:32:02.228 INFO:teuthology.orchestra.run.smithi137.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9", 2024-09-18T07:32:02.228 INFO:teuthology.orchestra.run.smithi137.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-18T07:32:02.228 INFO:teuthology.orchestra.run.smithi137.stdout: "crimson_alien_op_num_threads": "6", 2024-09-18T07:32:02.228 INFO:teuthology.orchestra.run.smithi137.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-18T07:32:02.228 INFO:teuthology.orchestra.run.smithi137.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-18T07:32:02.229 INFO:teuthology.orchestra.run.smithi137.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-18T07:32:02.229 INFO:teuthology.orchestra.run.smithi137.stdout: "crimson_osd_stat_interval": "0", 2024-09-18T07:32:02.229 INFO:teuthology.orchestra.run.smithi137.stdout: "crimson_seastar_cpu_cores": "", 2024-09-18T07:32:02.229 INFO:teuthology.orchestra.run.smithi137.stdout: "crimson_seastar_num_threads": "0", 2024-09-18T07:32:02.229 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_location": "", 2024-09-18T07:32:02.229 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_location_hook": "", 2024-09-18T07:32:02.229 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_location_hook_timeout": "10", 2024-09-18T07:32:02.229 INFO:teuthology.orchestra.run.smithi137.stdout: "daemonize": "false", 2024-09-18T07:32:02.229 INFO:teuthology.orchestra.run.smithi137.stdout: "daos_pool": "tank", 2024-09-18T07:32:02.229 INFO:teuthology.orchestra.run.smithi137.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-18T07:32:02.229 INFO:teuthology.orchestra.run.smithi137.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-18T07:32:02.229 INFO:teuthology.orchestra.run.smithi137.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-18T07:32:02.229 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_alienstore": "0/5", 2024-09-18T07:32:02.230 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_allow_any_pool_priority": "false", 2024-09-18T07:32:02.230 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_asok": "1/5", 2024-09-18T07:32:02.230 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_asok_assert_abort": "false", 2024-09-18T07:32:02.230 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_asserts_on_shutdown": "true", 2024-09-18T07:32:02.230 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_auth": "1/5", 2024-09-18T07:32:02.230 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_bdev": "1/3", 2024-09-18T07:32:02.230 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_bluefs": "1/5", 2024-09-18T07:32:02.230 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_bluestore": "1/5", 2024-09-18T07:32:02.230 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_buffer": "0/1", 2024-09-18T07:32:02.230 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_ceph_dedup": "0/5", 2024-09-18T07:32:02.230 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_ceph_exporter": "1/5", 2024-09-18T07:32:02.230 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_cephfs_mirror": "0/5", 2024-09-18T07:32:02.230 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_cephsqlite": "0/5", 2024-09-18T07:32:02.230 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_client": "0/5", 2024-09-18T07:32:02.231 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_compressor": "1/5", 2024-09-18T07:32:02.231 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_context": "0/1", 2024-09-18T07:32:02.231 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_crimson_interrupt": "0/5", 2024-09-18T07:32:02.231 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_crush": "1/1", 2024-09-18T07:32:02.231 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_crypto": "1/5", 2024-09-18T07:32:02.231 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_cyanstore": "0/5", 2024-09-18T07:32:02.231 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_deliberately_leak_memory": "false", 2024-09-18T07:32:02.231 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_disable_randomized_ping": "false", 2024-09-18T07:32:02.231 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_dpdk": "1/5", 2024-09-18T07:32:02.231 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_eventtrace": "1/5", 2024-09-18T07:32:02.231 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_filer": "0/1", 2024-09-18T07:32:02.231 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_filestore": "1/3", 2024-09-18T07:32:02.231 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_finisher": "1/1", 2024-09-18T07:32:02.232 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_fuse": "1/5", 2024-09-18T07:32:02.232 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_heartbeat_testing_span": "0", 2024-09-18T07:32:02.232 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_heartbeatmap": "1/5", 2024-09-18T07:32:02.232 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-18T07:32:02.232 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_javaclient": "1/5", 2024-09-18T07:32:02.232 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_journal": "1/3", 2024-09-18T07:32:02.232 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_journaler": "0/5", 2024-09-18T07:32:02.232 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_kstore": "1/5", 2024-09-18T07:32:02.232 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_lockdep": "0/1", 2024-09-18T07:32:02.232 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mclock": "1/5", 2024-09-18T07:32:02.232 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mds": "1/5", 2024-09-18T07:32:02.232 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mds_balancer": "1/5", 2024-09-18T07:32:02.232 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mds_locker": "1/5", 2024-09-18T07:32:02.232 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mds_log": "1/5", 2024-09-18T07:32:02.233 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mds_log_expire": "1/5", 2024-09-18T07:32:02.233 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mds_migrator": "1/5", 2024-09-18T07:32:02.233 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mds_quiesce": "3/5", 2024-09-18T07:32:02.233 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_memstore": "1/5", 2024-09-18T07:32:02.233 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mgr": "2/5", 2024-09-18T07:32:02.233 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mgrc": "1/5", 2024-09-18T07:32:02.233 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mon": "20/20", 2024-09-18T07:32:02.233 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_monc": "0/10", 2024-09-18T07:32:02.233 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_ms": "1/1", 2024-09-18T07:32:02.233 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_none": "0/5", 2024-09-18T07:32:02.233 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_objclass": "0/5", 2024-09-18T07:32:02.233 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_objectcacher": "0/5", 2024-09-18T07:32:02.233 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_objecter": "0/1", 2024-09-18T07:32:02.234 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_optracker": "0/5", 2024-09-18T07:32:02.234 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_osd": "1/5", 2024-09-18T07:32:02.234 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_paxos": "20/20", 2024-09-18T07:32:02.234 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_perfcounter": "1/5", 2024-09-18T07:32:02.234 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_prioritycache": "1/5", 2024-09-18T07:32:02.234 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rados": "0/5", 2024-09-18T07:32:02.234 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rbd": "0/5", 2024-09-18T07:32:02.234 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rbd_mirror": "0/5", 2024-09-18T07:32:02.234 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rbd_pwl": "0/5", 2024-09-18T07:32:02.234 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rbd_replay": "0/5", 2024-09-18T07:32:02.234 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_refs": "0/0", 2024-09-18T07:32:02.234 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_reserver": "1/1", 2024-09-18T07:32:02.234 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rgw": "1/5", 2024-09-18T07:32:02.234 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rgw_access": "1/5", 2024-09-18T07:32:02.234 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rgw_datacache": "1/5", 2024-09-18T07:32:02.235 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rgw_dbstore": "1/5", 2024-09-18T07:32:02.235 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rgw_flight": "1/5", 2024-09-18T07:32:02.236 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-18T07:32:02.236 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rgw_notification": "1/5", 2024-09-18T07:32:02.236 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rgw_sync": "1/5", 2024-09-18T07:32:02.236 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rocksdb": "4/5", 2024-09-18T07:32:02.236 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore": "0/5", 2024-09-18T07:32:02.236 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_backref": "0/5", 2024-09-18T07:32:02.236 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_cache": "0/5", 2024-09-18T07:32:02.237 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_cleaner": "0/5", 2024-09-18T07:32:02.237 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_device": "0/5", 2024-09-18T07:32:02.237 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_epm": "0/5", 2024-09-18T07:32:02.237 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-18T07:32:02.237 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_journal": "0/5", 2024-09-18T07:32:02.237 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_lba": "0/5", 2024-09-18T07:32:02.237 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_odata": "0/5", 2024-09-18T07:32:02.237 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_omap": "0/5", 2024-09-18T07:32:02.237 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_onode": "0/5", 2024-09-18T07:32:02.237 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_t": "0/5", 2024-09-18T07:32:02.237 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_tm": "0/5", 2024-09-18T07:32:02.238 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_striper": "0/1", 2024-09-18T07:32:02.238 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_test": "0/5", 2024-09-18T07:32:02.238 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_throttle": "1/1", 2024-09-18T07:32:02.238 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_timer": "0/1", 2024-09-18T07:32:02.238 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_tp": "0/5", 2024-09-18T07:32:02.238 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_trace": "1/5", 2024-09-18T07:32:02.238 INFO:teuthology.orchestra.run.smithi137.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-18T07:32:02.238 INFO:teuthology.orchestra.run.smithi137.stdout: "device_failure_prediction_mode": "none", 2024-09-18T07:32:02.238 INFO:teuthology.orchestra.run.smithi137.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-18T07:32:02.238 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-18T07:32:02.239 INFO:teuthology.orchestra.run.smithi137.stdout: "err_to_graylog": "false", 2024-09-18T07:32:02.239 INFO:teuthology.orchestra.run.smithi137.stdout: "err_to_journald": "false", 2024-09-18T07:32:02.239 INFO:teuthology.orchestra.run.smithi137.stdout: "err_to_stderr": "true", 2024-09-18T07:32:02.239 INFO:teuthology.orchestra.run.smithi137.stdout: "err_to_syslog": "false", 2024-09-18T07:32:02.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event_tracing": "false", 2024-09-18T07:32:02.239 INFO:teuthology.orchestra.run.smithi137.stdout: "exporter_addr": "0.0.0.0", 2024-09-18T07:32:02.239 INFO:teuthology.orchestra.run.smithi137.stdout: "exporter_cert_file": "", 2024-09-18T07:32:02.239 INFO:teuthology.orchestra.run.smithi137.stdout: "exporter_http_port": "9926", 2024-09-18T07:32:02.239 INFO:teuthology.orchestra.run.smithi137.stdout: "exporter_key_file": "", 2024-09-18T07:32:02.239 INFO:teuthology.orchestra.run.smithi137.stdout: "exporter_prio_limit": "5", 2024-09-18T07:32:02.239 INFO:teuthology.orchestra.run.smithi137.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-18T07:32:02.240 INFO:teuthology.orchestra.run.smithi137.stdout: "exporter_sort_metrics": "true", 2024-09-18T07:32:02.240 INFO:teuthology.orchestra.run.smithi137.stdout: "exporter_stats_period": "5", 2024-09-18T07:32:02.240 INFO:teuthology.orchestra.run.smithi137.stdout: "fake_statfs_for_testing": "0", 2024-09-18T07:32:02.240 INFO:teuthology.orchestra.run.smithi137.stdout: "fatal_signal_handlers": "true", 2024-09-18T07:32:02.240 INFO:teuthology.orchestra.run.smithi137.stdout: "filer_max_purge_ops": "10", 2024-09-18T07:32:02.240 INFO:teuthology.orchestra.run.smithi137.stdout: "filer_max_truncate_ops": "128", 2024-09-18T07:32:02.240 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_apply_finisher_threads": "1", 2024-09-18T07:32:02.240 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_blackhole": "false", 2024-09-18T07:32:02.240 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_btrfs_clone_range": "true", 2024-09-18T07:32:02.240 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_btrfs_snap": "true", 2024-09-18T07:32:02.240 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_caller_concurrency": "10", 2024-09-18T07:32:02.240 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_collect_device_partition_information": "true", 2024-09-18T07:32:02.241 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_commit_timeout": "600.000000", 2024-09-18T07:32:02.241 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_debug_inject_read_err": "false", 2024-09-18T07:32:02.241 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_debug_omap_check": "false", 2024-09-18T07:32:02.241 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_debug_verify_split": "false", 2024-09-18T07:32:02.241 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_dump_file": "", 2024-09-18T07:32:02.241 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-18T07:32:02.241 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-18T07:32:02.241 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_fadvise": "true", 2024-09-18T07:32:02.241 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_fail_eio": "true", 2024-09-18T07:32:02.241 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_fd_cache_shards": "16", 2024-09-18T07:32:02.241 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_fd_cache_size": "128", 2024-09-18T07:32:02.241 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_fiemap": "false", 2024-09-18T07:32:02.241 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_fiemap_threshold": "4096", 2024-09-18T07:32:02.242 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-18T07:32:02.242 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-18T07:32:02.242 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_inject_stall": "0", 2024-09-18T07:32:02.242 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_journal_parallel": "false", 2024-09-18T07:32:02.242 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_journal_trailing": "false", 2024-09-18T07:32:02.242 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_journal_writeahead": "false", 2024-09-18T07:32:02.242 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_kill_at": "0", 2024-09-18T07:32:02.242 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-18T07:32:02.242 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-18T07:32:02.242 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-18T07:32:02.243 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-18T07:32:02.243 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-18T07:32:02.243 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_inline_xattrs": "0", 2024-09-18T07:32:02.243 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-18T07:32:02.243 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-18T07:32:02.243 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-18T07:32:02.243 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-18T07:32:02.243 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_xattr_value_size": "0", 2024-09-18T07:32:02.243 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-18T07:32:02.243 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-18T07:32:02.243 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-18T07:32:02.243 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_merge_threshold": "-10", 2024-09-18T07:32:02.244 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-18T07:32:02.244 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_odsync_write": "false", 2024-09-18T07:32:02.244 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_omap_backend": "rocksdb", 2024-09-18T07:32:02.244 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_omap_backend_path": "", 2024-09-18T07:32:02.244 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-18T07:32:02.244 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-18T07:32:02.244 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-18T07:32:02.244 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_op_thread_timeout": "60", 2024-09-18T07:32:02.244 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_op_threads": "2", 2024-09-18T07:32:02.244 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_punch_hole": "false", 2024-09-18T07:32:02.244 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-18T07:32:02.244 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-18T07:32:02.244 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-18T07:32:02.245 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-18T07:32:02.245 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-18T07:32:02.245 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-18T07:32:02.245 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-18T07:32:02.245 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-18T07:32:02.245 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-18T07:32:02.245 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_max_ops": "50", 2024-09-18T07:32:02.245 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-18T07:32:02.245 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_seek_data_hole": "false", 2024-09-18T07:32:02.245 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_sloppy_crc": "false", 2024-09-18T07:32:02.245 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-18T07:32:02.245 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_splice": "false", 2024-09-18T07:32:02.246 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_split_multiple": "2", 2024-09-18T07:32:02.246 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_split_rand_factor": "20", 2024-09-18T07:32:02.246 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_update_to": "1000", 2024-09-18T07:32:02.246 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-18T07:32:02.246 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-18T07:32:02.246 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-18T07:32:02.246 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-18T07:32:02.246 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-18T07:32:02.246 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-18T07:32:02.246 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_enable": "true", 2024-09-18T07:32:02.246 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-18T07:32:02.246 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-18T07:32:02.246 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-18T07:32:02.247 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-18T07:32:02.247 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-18T07:32:02.247 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-18T07:32:02.247 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_xfs_extsize": "false", 2024-09-18T07:32:02.247 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_zfs_snap": "false", 2024-09-18T07:32:02.247 INFO:teuthology.orchestra.run.smithi137.stdout: "fio_dir": "/tmp/fio", 2024-09-18T07:32:02.247 INFO:teuthology.orchestra.run.smithi137.stdout: "fsid": "f7ef099c-758e-11ef-bceb-c7b262605968", 2024-09-18T07:32:02.247 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_allow_other": "true", 2024-09-18T07:32:02.247 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_atomic_o_trunc": "true", 2024-09-18T07:32:02.247 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_big_writes": "true", 2024-09-18T07:32:02.247 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_debug": "false", 2024-09-18T07:32:02.247 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_default_permissions": "false", 2024-09-18T07:32:02.248 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_disable_pagecache": "false", 2024-09-18T07:32:02.248 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_max_write": "0", 2024-09-18T07:32:02.248 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_multithreaded": "true", 2024-09-18T07:32:02.248 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_require_active_mds": "true", 2024-09-18T07:32:02.248 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_set_user_groups": "true", 2024-09-18T07:32:02.248 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_splice_move": "true", 2024-09-18T07:32:02.248 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_splice_read": "true", 2024-09-18T07:32:02.248 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_splice_write": "true", 2024-09-18T07:32:02.248 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-18T07:32:02.248 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_use_invalidate_cb": "true", 2024-09-18T07:32:02.248 INFO:teuthology.orchestra.run.smithi137.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-18T07:32:02.248 INFO:teuthology.orchestra.run.smithi137.stdout: "gss_target_name": "ceph", 2024-09-18T07:32:02.249 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_file": "", 2024-09-18T07:32:02.249 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_inject_failure": "0", 2024-09-18T07:32:02.249 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_interval": "5", 2024-09-18T07:32:02.249 INFO:teuthology.orchestra.run.smithi137.stdout: "host": "smithi137", 2024-09-18T07:32:02.249 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-18T07:32:02.249 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-18T07:32:02.249 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-18T07:32:02.249 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-18T07:32:02.249 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-18T07:32:02.249 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-18T07:32:02.249 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-18T07:32:02.249 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-18T07:32:02.249 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-18T07:32:02.249 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-18T07:32:02.250 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-18T07:32:02.250 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-18T07:32:02.250 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-18T07:32:02.250 INFO:teuthology.orchestra.run.smithi137.stdout: "inject_early_sigterm": "false", 2024-09-18T07:32:02.250 INFO:teuthology.orchestra.run.smithi137.stdout: "jaeger_agent_port": "6799", 2024-09-18T07:32:02.250 INFO:teuthology.orchestra.run.smithi137.stdout: "jaeger_tracing_enable": "false", 2024-09-18T07:32:02.250 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_aio": "true", 2024-09-18T07:32:02.250 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_align_min_size": "65536", 2024-09-18T07:32:02.250 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_block_align": "true", 2024-09-18T07:32:02.250 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_block_size": "4096", 2024-09-18T07:32:02.250 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_dio": "true", 2024-09-18T07:32:02.250 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_discard": "false", 2024-09-18T07:32:02.250 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_force_aio": "false", 2024-09-18T07:32:02.251 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_ignore_corruption": "false", 2024-09-18T07:32:02.251 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_max_write_bytes": "10485760", 2024-09-18T07:32:02.251 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_max_write_entries": "100", 2024-09-18T07:32:02.251 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_replay_from": "0", 2024-09-18T07:32:02.251 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-18T07:32:02.251 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-18T07:32:02.251 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-18T07:32:02.251 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-18T07:32:02.251 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_write_header_frequency": "0", 2024-09-18T07:32:02.251 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_zero_on_create": "false", 2024-09-18T07:32:02.251 INFO:teuthology.orchestra.run.smithi137.stdout: "journaler_prefetch_periods": "10", 2024-09-18T07:32:02.251 INFO:teuthology.orchestra.run.smithi137.stdout: "journaler_prezero_periods": "5", 2024-09-18T07:32:02.251 INFO:teuthology.orchestra.run.smithi137.stdout: "journaler_write_head_interval": "15", 2024-09-18T07:32:02.251 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "", 2024-09-18T07:32:02.252 INFO:teuthology.orchestra.run.smithi137.stdout: "keyfile": "", 2024-09-18T07:32:02.252 INFO:teuthology.orchestra.run.smithi137.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-18T07:32:02.252 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_backend": "rocksdb", 2024-09-18T07:32:02.252 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_default_stripe_size": "65536", 2024-09-18T07:32:02.252 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_fsck_on_mount": "false", 2024-09-18T07:32:02.252 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-18T07:32:02.252 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_max_bytes": "67108864", 2024-09-18T07:32:02.252 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_max_ops": "512", 2024-09-18T07:32:02.252 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_nid_prealloc": "1024", 2024-09-18T07:32:02.252 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_onode_map_size": "1024", 2024-09-18T07:32:02.252 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-18T07:32:02.252 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_sync_submit_transaction": "false", 2024-09-18T07:32:02.252 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_sync_transaction": "false", 2024-09-18T07:32:02.252 INFO:teuthology.orchestra.run.smithi137.stdout: "librados_thread_count": "2", 2024-09-18T07:32:02.253 INFO:teuthology.orchestra.run.smithi137.stdout: "lockdep": "false", 2024-09-18T07:32:02.253 INFO:teuthology.orchestra.run.smithi137.stdout: "lockdep_force_backtrace": "false", 2024-09-18T07:32:02.253 INFO:teuthology.orchestra.run.smithi137.stdout: "log_coarse_timestamps": "true", 2024-09-18T07:32:02.253 INFO:teuthology.orchestra.run.smithi137.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-18T07:32:02.253 INFO:teuthology.orchestra.run.smithi137.stdout: "log_flush_on_exit": "false", 2024-09-18T07:32:02.253 INFO:teuthology.orchestra.run.smithi137.stdout: "log_graylog_host": "127.0.0.1", 2024-09-18T07:32:02.253 INFO:teuthology.orchestra.run.smithi137.stdout: "log_graylog_port": "12201", 2024-09-18T07:32:02.253 INFO:teuthology.orchestra.run.smithi137.stdout: "log_max_new": "1000", 2024-09-18T07:32:02.253 INFO:teuthology.orchestra.run.smithi137.stdout: "log_max_recent": "10000", 2024-09-18T07:32:02.253 INFO:teuthology.orchestra.run.smithi137.stdout: "log_stderr_prefix": "", 2024-09-18T07:32:02.253 INFO:teuthology.orchestra.run.smithi137.stdout: "log_stop_at_utilization": "0.970000", 2024-09-18T07:32:02.253 INFO:teuthology.orchestra.run.smithi137.stdout: "log_to_file": "true", 2024-09-18T07:32:02.253 INFO:teuthology.orchestra.run.smithi137.stdout: "log_to_graylog": "false", 2024-09-18T07:32:02.254 INFO:teuthology.orchestra.run.smithi137.stdout: "log_to_journald": "false", 2024-09-18T07:32:02.254 INFO:teuthology.orchestra.run.smithi137.stdout: "log_to_stderr": "false", 2024-09-18T07:32:02.254 INFO:teuthology.orchestra.run.smithi137.stdout: "log_to_syslog": "false", 2024-09-18T07:32:02.254 INFO:teuthology.orchestra.run.smithi137.stdout: "max_rotating_auth_attempts": "10", 2024-09-18T07:32:02.254 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-18T07:32:02.254 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_action_on_write_error": "1", 2024-09-18T07:32:02.254 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_alternate_name_max": "8192", 2024-09-18T07:32:02.254 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_asio_thread_count": "2", 2024-09-18T07:32:02.254 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_export_pin": "true", 2024-09-18T07:32:02.254 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_fragment_dirs": "true", 2024-09-18T07:32:02.254 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-18T07:32:02.254 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_fragment_interval": "5", 2024-09-18T07:32:02.254 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-18T07:32:02.255 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-18T07:32:02.255 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_interval": "10", 2024-09-18T07:32:02.255 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_max": "-1", 2024-09-18T07:32:02.255 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_max_until": "-1", 2024-09-18T07:32:02.255 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_merge_size": "50", 2024-09-18T07:32:02.255 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_midchunk": "0.300000", 2024-09-18T07:32:02.255 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-18T07:32:02.255 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_min_start": "0.200000", 2024-09-18T07:32:02.255 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_minchunk": "0.001000", 2024-09-18T07:32:02.255 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_mode": "0", 2024-09-18T07:32:02.255 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_need_max": "1.200000", 2024-09-18T07:32:02.255 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_need_min": "0.800000", 2024-09-18T07:32:02.255 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_overload_epochs": "2", 2024-09-18T07:32:02.256 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-18T07:32:02.256 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-18T07:32:02.256 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_split_bits": "3", 2024-09-18T07:32:02.256 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-18T07:32:02.256 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_split_size": "10000", 2024-09-18T07:32:02.256 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-18T07:32:02.256 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_target_decay": "10.000000", 2024-09-18T07:32:02.256 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-18T07:32:02.256 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_beacon_grace": "15.000000", 2024-09-18T07:32:02.256 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_beacon_interval": "4.000000", 2024-09-18T07:32:02.256 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-18T07:32:02.256 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-18T07:32:02.256 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_mid": "0.700000", 2024-09-18T07:32:02.257 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-18T07:32:02.257 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_quiesce_delay": "0", 2024-09-18T07:32:02.257 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-18T07:32:02.257 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-18T07:32:02.257 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-18T07:32:02.257 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_release_free_interval": "10", 2024-09-18T07:32:02.257 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_reservation": "0.050000", 2024-09-18T07:32:02.257 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-18T07:32:02.257 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_trim_interval": "1", 2024-09-18T07:32:02.257 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_trim_threshold": "262144", 2024-09-18T07:32:02.257 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-18T07:32:02.257 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-18T07:32:02.257 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-18T07:32:02.258 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_client_prealloc_inos": "1000", 2024-09-18T07:32:02.258 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-18T07:32:02.258 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_connect_bootstrapping": "false", 2024-09-18T07:32:02.258 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_damage_table_max_entries": "10000", 2024-09-18T07:32:02.258 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-18T07:32:02.258 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_debug_auth_pins": "false", 2024-09-18T07:32:02.258 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_debug_frag": "false", 2024-09-18T07:32:02.258 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_debug_scatterstat": "false", 2024-09-18T07:32:02.258 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_debug_subtrees": "false", 2024-09-18T07:32:02.258 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_decay_halflife": "5.000000", 2024-09-18T07:32:02.258 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_default_dir_hash": "2", 2024-09-18T07:32:02.258 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_defer_session_stale": "true", 2024-09-18T07:32:02.258 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_deny_all_reconnect": "false", 2024-09-18T07:32:02.258 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dir_keys_per_op": "16384", 2024-09-18T07:32:02.259 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dir_max_commit_size": "10", 2024-09-18T07:32:02.259 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dir_max_entries": "0", 2024-09-18T07:32:02.259 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dir_prefetch": "true", 2024-09-18T07:32:02.259 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-18T07:32:02.259 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-18T07:32:02.259 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dump_cache_on_map": "false", 2024-09-18T07:32:02.259 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-18T07:32:02.259 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-18T07:32:02.259 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_early_reply": "true", 2024-09-18T07:32:02.259 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_enable_op_tracker": "true", 2024-09-18T07:32:02.259 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_enforce_unique_name": "true", 2024-09-18T07:32:02.259 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-18T07:32:02.259 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-18T07:32:02.260 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_export_ephemeral_random": "true", 2024-09-18T07:32:02.260 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-18T07:32:02.260 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-18T07:32:02.260 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-18T07:32:02.260 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-18T07:32:02.260 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-18T07:32:02.260 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-18T07:32:02.260 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-18T07:32:02.260 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-18T07:32:02.260 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_health_summarize_threshold": "10", 2024-09-18T07:32:02.260 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-18T07:32:02.260 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-18T07:32:02.261 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_inject_health_dummy": "false", 2024-09-18T07:32:02.261 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-18T07:32:02.261 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_inject_migrator_session_race": "false", 2024-09-18T07:32:02.261 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-18T07:32:02.261 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-18T07:32:02.261 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-18T07:32:02.261 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_join_fs": "", 2024-09-18T07:32:02.261 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_journal_format": "1", 2024-09-18T07:32:02.261 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-18T07:32:02.261 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_create_at": "0", 2024-09-18T07:32:02.261 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_dirfrag_at": "0", 2024-09-18T07:32:02.261 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_export_at": "0", 2024-09-18T07:32:02.262 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_import_at": "0", 2024-09-18T07:32:02.262 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_journal_at": "0", 2024-09-18T07:32:02.262 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_journal_expire_at": "0", 2024-09-18T07:32:02.262 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_journal_replay_at": "0", 2024-09-18T07:32:02.262 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_link_at": "0", 2024-09-18T07:32:02.262 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_mdstable_at": "0", 2024-09-18T07:32:02.262 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_openc_at": "0", 2024-09-18T07:32:02.262 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_rename_at": "0", 2024-09-18T07:32:02.262 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_shutdown_at": "0", 2024-09-18T07:32:02.262 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_event_large_threshold": "524288", 2024-09-18T07:32:02.262 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_events_per_segment": "1024", 2024-09-18T07:32:02.262 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-18T07:32:02.262 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_max_events": "-1", 2024-09-18T07:32:02.263 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_max_segments": "128", 2024-09-18T07:32:02.263 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_pause": "false", 2024-09-18T07:32:02.263 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_segment_size": "0", 2024-09-18T07:32:02.263 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-18T07:32:02.263 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-18T07:32:02.263 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-18T07:32:02.263 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_trim_threshold": "128", 2024-09-18T07:32:02.263 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-18T07:32:02.263 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_warn_factor": "2.000000", 2024-09-18T07:32:02.263 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_caps_per_client": "1048576", 2024-09-18T07:32:02.263 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_completed_flushes": "100000", 2024-09-18T07:32:02.263 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_completed_requests": "100000", 2024-09-18T07:32:02.263 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_export_size": "20971520", 2024-09-18T07:32:02.264 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_file_recover": "32", 2024-09-18T07:32:02.264 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_purge_files": "64", 2024-09-18T07:32:02.264 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_purge_ops": "8192", 2024-09-18T07:32:02.264 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-18T07:32:02.264 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-18T07:32:02.264 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_snaps_per_dir": "100", 2024-09-18T07:32:02.264 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_metrics_update_interval": "2", 2024-09-18T07:32:02.264 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_min_caps_per_client": "100", 2024-09-18T07:32:02.264 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_min_caps_working_set": "10000", 2024-09-18T07:32:02.264 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-18T07:32:02.264 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_numa_node": "-1", 2024-09-18T07:32:02.264 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-18T07:32:02.264 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_op_complaint_time": "30.000000", 2024-09-18T07:32:02.264 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_op_history_duration": "600", 2024-09-18T07:32:02.264 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_op_history_size": "20", 2024-09-18T07:32:02.265 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_op_history_slow_op_size": "20", 2024-09-18T07:32:02.265 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-18T07:32:02.265 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_op_log_threshold": "5", 2024-09-18T07:32:02.265 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_ping_grace": "15", 2024-09-18T07:32:02.265 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_ping_interval": "5", 2024-09-18T07:32:02.265 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-18T07:32:02.265 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-18T07:32:02.265 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_recall_max_caps": "30000", 2024-09-18T07:32:02.265 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-18T07:32:02.265 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-18T07:32:02.265 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-18T07:32:02.265 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_recall_warning_threshold": "262144", 2024-09-18T07:32:02.265 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-18T07:32:02.266 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_replay_interval": "1.000000", 2024-09-18T07:32:02.266 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-18T07:32:02.266 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-18T07:32:02.266 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_root_ino_gid": "0", 2024-09-18T07:32:02.266 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_root_ino_uid": "0", 2024-09-18T07:32:02.266 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-18T07:32:02.266 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_scrub_stats_review_period": "1", 2024-09-18T07:32:02.266 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-18T07:32:02.266 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-18T07:32:02.266 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-18T07:32:02.266 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-18T07:32:02.266 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-18T07:32:02.266 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-18T07:32:02.266 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-18T07:32:02.267 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-18T07:32:02.267 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-18T07:32:02.267 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-18T07:32:02.267 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-18T07:32:02.267 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_shutdown_check": "0", 2024-09-18T07:32:02.267 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_skip_ino": "0", 2024-09-18T07:32:02.267 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-18T07:32:02.267 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_snap_max_uid": "4294967294", 2024-09-18T07:32:02.267 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_snap_min_uid": "0", 2024-09-18T07:32:02.267 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_snap_rstat": "false", 2024-09-18T07:32:02.267 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_standby_replay_damaged": "false", 2024-09-18T07:32:02.267 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_symlink_recovery": "true", 2024-09-18T07:32:02.267 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-18T07:32:02.267 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_thrash_exports": "0", 2024-09-18T07:32:02.267 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_thrash_fragments": "0", 2024-09-18T07:32:02.268 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_tick_interval": "5.000000", 2024-09-18T07:32:02.268 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_valgrind_exit": "false", 2024-09-18T07:32:02.268 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_verify_backtrace": "1", 2024-09-18T07:32:02.268 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_verify_scatter": "false", 2024-09-18T07:32:02.268 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-18T07:32:02.268 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_wipe_sessions": "false", 2024-09-18T07:32:02.268 INFO:teuthology.orchestra.run.smithi137.stdout: "mempool_debug": "false", 2024-09-18T07:32:02.268 INFO:teuthology.orchestra.run.smithi137.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-18T07:32:02.268 INFO:teuthology.orchestra.run.smithi137.stdout: "memstore_device_bytes": "1073741824", 2024-09-18T07:32:02.268 INFO:teuthology.orchestra.run.smithi137.stdout: "memstore_page_set": "false", 2024-09-18T07:32:02.268 INFO:teuthology.orchestra.run.smithi137.stdout: "memstore_page_size": "65536", 2024-09-18T07:32:02.268 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_client_bytes": "134217728", 2024-09-18T07:32:02.268 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_client_messages": "512", 2024-09-18T07:32:02.269 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-18T07:32:02.269 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-18T07:32:02.269 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-18T07:32:02.269 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-18T07:32:02.269 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_disabled_modules": "", 2024-09-18T07:32:02.269 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_enable_op_tracker": "true", 2024-09-18T07:32:02.269 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-18T07:32:02.269 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_max_pg_creating": "1024", 2024-09-18T07:32:02.269 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_max_pg_num_change": "128", 2024-09-18T07:32:02.269 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_mds_bytes": "134217728", 2024-09-18T07:32:02.269 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_mds_messages": "128", 2024-09-18T07:32:02.269 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-18T07:32:02.269 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_mon_bytes": "134217728", 2024-09-18T07:32:02.269 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_mon_messages": "128", 2024-09-18T07:32:02.270 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-18T07:32:02.270 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-18T07:32:02.270 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_op_history_duration": "600", 2024-09-18T07:32:02.270 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_op_history_size": "20", 2024-09-18T07:32:02.270 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-18T07:32:02.270 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-18T07:32:02.270 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_op_log_threshold": "5", 2024-09-18T07:32:02.270 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_osd_bytes": "536870912", 2024-09-18T07:32:02.270 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_osd_messages": "8192", 2024-09-18T07:32:02.270 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_pool": "true", 2024-09-18T07:32:02.270 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-18T07:32:02.270 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_standby_modules": "true", 2024-09-18T07:32:02.270 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_stats_period": "5", 2024-09-18T07:32:02.271 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_stats_threshold": "5", 2024-09-18T07:32:02.271 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_tick_period": "2", 2024-09-18T07:32:02.271 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-18T07:32:02.271 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-18T07:32:02.271 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_allow_pool_delete": "true", 2024-09-18T07:32:02.271 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_allow_pool_size_one": "false", 2024-09-18T07:32:02.271 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_auth_validate_all_caps": "true", 2024-09-18T07:32:02.271 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-18T07:32:02.271 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-18T07:32:02.271 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_bytes": "104857600", 2024-09-18T07:32:02.271 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_directed_command_retry": "2", 2024-09-18T07:32:02.271 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-18T07:32:02.271 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-18T07:32:02.272 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-18T07:32:02.272 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-18T07:32:02.272 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_hunt_parallel": "3", 2024-09-18T07:32:02.272 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_log_interval": "1.000000", 2024-09-18T07:32:02.272 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-18T07:32:02.272 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_ping_interval": "10.000000", 2024-09-18T07:32:02.272 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-18T07:32:02.272 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_target_rank": "-1", 2024-09-18T07:32:02.272 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-18T07:32:02.272 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-18T07:32:02.272 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-18T07:32:02.272 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_level": "debug", 2024-09-18T07:32:02.272 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_to_file": "true", 2024-09-18T07:32:02.273 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-18T07:32:02.273 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-18T07:32:02.273 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-18T07:32:02.273 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_to_journald": "true", 2024-09-18T07:32:02.273 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-18T07:32:02.273 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-18T07:32:02.273 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-18T07:32:02.273 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_compact_on_bootstrap": "false", 2024-09-18T07:32:02.273 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_compact_on_start": "false", 2024-09-18T07:32:02.273 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_compact_on_trim": "true", 2024-09-18T07:32:02.273 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-18T07:32:02.273 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-18T07:32:02.273 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-18T07:32:02.274 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cpu_threads": "4", 2024-09-18T07:32:02.274 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_crush_min_required_version": "hammer", 2024-09-18T07:32:02.274 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_daemon_bytes": "419430400", 2024-09-18T07:32:02.274 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-18T07:32:02.274 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_data_avail_crit": "5", 2024-09-18T07:32:02.274 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_data_avail_warn": "5", 2024-09-18T07:32:02.274 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_data_size_warn": "16106127360", 2024-09-18T07:32:02.274 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-18T07:32:02.274 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-18T07:32:02.274 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_dump_json": "false", 2024-09-18T07:32:02.274 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-18T07:32:02.274 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_dump_transactions": "false", 2024-09-18T07:32:02.274 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_extra_checks": "false", 2024-09-18T07:32:02.274 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-18T07:32:02.275 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-18T07:32:02.275 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_no_require_reef": "false", 2024-09-18T07:32:02.275 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_no_require_squid": "false", 2024-09-18T07:32:02.275 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-18T07:32:02.275 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-18T07:32:02.275 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-18T07:32:02.275 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_down_mkfs_grace": "60", 2024-09-18T07:32:02.275 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_down_uptime_grace": "60", 2024-09-18T07:32:02.275 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_election_default_strategy": "1", 2024-09-18T07:32:02.275 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_election_timeout": "5.000000", 2024-09-18T07:32:02.275 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-18T07:32:02.275 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_elector_ping_divisor": "2", 2024-09-18T07:32:02.275 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-18T07:32:02.275 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_enable_op_tracker": "true", 2024-09-18T07:32:02.275 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_fake_pool_delete": "false", 2024-09-18T07:32:02.276 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_force_quorum_join": "false", 2024-09-18T07:32:02.276 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-18T07:32:02.276 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_globalid_prealloc": "10000", 2024-09-18T07:32:02.276 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_health_detail_to_clog": "true", 2024-09-18T07:32:02.276 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_health_log_update_period": "5", 2024-09-18T07:32:02.276 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_health_max_detail": "50", 2024-09-18T07:32:02.276 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_health_to_clog": "true", 2024-09-18T07:32:02.276 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_health_to_clog_interval": "600", 2024-09-18T07:32:02.276 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-18T07:32:02.276 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_host": "172.21.15.137:6789/0", 2024-09-18T07:32:02.276 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_host_override": "", 2024-09-18T07:32:02.276 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_initial_members": "", 2024-09-18T07:32:02.276 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-18T07:32:02.276 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-18T07:32:02.277 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-18T07:32:02.277 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-18T07:32:02.277 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-18T07:32:02.277 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_lease": "5.000000", 2024-09-18T07:32:02.277 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-18T07:32:02.277 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-18T07:32:02.277 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_log_full_interval": "50", 2024-09-18T07:32:02.277 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_log_max": "10000", 2024-09-18T07:32:02.277 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_log_max_summary": "50", 2024-09-18T07:32:02.277 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-18T07:32:02.277 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_max_log_epochs": "500", 2024-09-18T07:32:02.277 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-18T07:32:02.277 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-18T07:32:02.277 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-18T07:32:02.278 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_max_osd": "10000", 2024-09-18T07:32:02.278 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_max_pg_per_osd": "10000", 2024-09-18T07:32:02.278 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_max_pool_pg_num": "65536", 2024-09-18T07:32:02.278 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-18T07:32:02.278 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-18T07:32:02.278 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mds_force_trim_to": "0", 2024-09-18T07:32:02.278 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mds_skip_sanity": "false", 2024-09-18T07:32:02.278 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_memory_autotune": "true", 2024-09-18T07:32:02.278 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_memory_target": "2147483648", 2024-09-18T07:32:02.278 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mgr_beacon_grace": "30", 2024-09-18T07:32:02.278 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-18T07:32:02.278 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mgr_digest_period": "5", 2024-09-18T07:32:02.279 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mgr_inactive_grace": "60", 2024-09-18T07:32:02.279 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-18T07:32:02.279 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-18T07:32:02.279 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_min_osdmap_epochs": "50", 2024-09-18T07:32:02.279 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-18T07:32:02.279 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-18T07:32:02.279 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_op_complaint_time": "30", 2024-09-18T07:32:02.279 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_op_history_duration": "600", 2024-09-18T07:32:02.279 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_op_history_size": "20", 2024-09-18T07:32:02.279 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_op_history_slow_op_size": "20", 2024-09-18T07:32:02.279 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-18T07:32:02.279 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_op_log_threshold": "5", 2024-09-18T07:32:02.279 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-18T07:32:02.280 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-18T07:32:02.280 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-18T07:32:02.280 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_auto_mark_in": "false", 2024-09-18T07:32:02.280 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-18T07:32:02.280 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-18T07:32:02.280 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-18T07:32:02.280 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_cache_size": "500", 2024-09-18T07:32:02.280 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-18T07:32:02.280 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-18T07:32:02.280 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-18T07:32:02.280 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_down_out_interval": "600", 2024-09-18T07:32:02.280 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-18T07:32:02.281 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-18T07:32:02.281 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_force_trim_to": "0", 2024-09-18T07:32:02.281 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-18T07:32:02.281 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-18T07:32:02.281 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-18T07:32:02.281 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-18T07:32:02.281 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-18T07:32:02.281 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-18T07:32:02.281 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-18T07:32:02.281 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_min_down_reporters": "2", 2024-09-18T07:32:02.281 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-18T07:32:02.281 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-18T07:32:02.281 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-18T07:32:02.281 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-18T07:32:02.282 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-18T07:32:02.282 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-18T07:32:02.282 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_report_timeout": "900", 2024-09-18T07:32:02.282 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-18T07:32:02.282 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-18T07:32:02.282 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-18T07:32:02.282 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-18T07:32:02.282 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-18T07:32:02.282 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-18T07:32:02.282 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-18T07:32:02.282 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-18T07:32:02.282 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-18T07:32:02.282 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_pg_stuck_threshold": "60", 2024-09-18T07:32:02.283 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-18T07:32:02.283 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-18T07:32:02.283 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-18T07:32:02.283 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-18T07:32:02.283 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-18T07:32:02.283 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-18T07:32:02.283 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_probe_timeout": "2.000000", 2024-09-18T07:32:02.283 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_reweight_max_change": "0.050000", 2024-09-18T07:32:02.283 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_reweight_max_osds": "4", 2024-09-18T07:32:02.283 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-18T07:32:02.283 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-18T07:32:02.283 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-18T07:32:02.283 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-18T07:32:02.284 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-18T07:32:02.284 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_scrub_interval": "86400", 2024-09-18T07:32:02.284 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_scrub_max_keys": "100", 2024-09-18T07:32:02.284 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_scrub_timeout": "300", 2024-09-18T07:32:02.284 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_session_timeout": "300", 2024-09-18T07:32:02.284 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_smart_report_timeout": "5", 2024-09-18T07:32:02.284 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_stat_smooth_intervals": "6", 2024-09-18T07:32:02.284 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-18T07:32:02.284 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_stretch_pool_min_size": "2", 2024-09-18T07:32:02.284 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_stretch_pool_size": "4", 2024-09-18T07:32:02.284 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-18T07:32:02.284 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-18T07:32:02.285 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_sync_debug": "false", 2024-09-18T07:32:02.285 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-18T07:32:02.285 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-18T07:32:02.285 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_sync_provider_kill_at": "0", 2024-09-18T07:32:02.285 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_sync_requester_kill_at": "0", 2024-09-18T07:32:02.285 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_sync_timeout": "60.000000", 2024-09-18T07:32:02.285 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_target_pg_per_osd": "100", 2024-09-18T07:32:02.285 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_tick_interval": "5", 2024-09-18T07:32:02.285 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_timecheck_interval": "300.000000", 2024-09-18T07:32:02.285 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-18T07:32:02.285 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_use_min_delay_socket": "false", 2024-09-18T07:32:02.285 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_older_version_delay": "604800", 2024-09-18T07:32:02.285 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-18T07:32:02.286 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-18T07:32:02.286 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-18T07:32:02.286 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-18T07:32:02.286 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-18T07:32:02.286 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-18T07:32:02.286 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-18T07:32:02.286 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_misplaced": "false", 2024-09-18T07:32:02.286 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-18T07:32:02.286 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_older_version": "true", 2024-09-18T07:32:02.286 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-18T07:32:02.286 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-18T07:32:02.286 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-18T07:32:02.286 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-18T07:32:02.287 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-18T07:32:02.287 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-18T07:32:02.287 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-18T07:32:02.287 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-18T07:32:02.287 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-18T07:32:02.287 INFO:teuthology.orchestra.run.smithi137.stdout: "monmap": "", 2024-09-18T07:32:02.287 INFO:teuthology.orchestra.run.smithi137.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-18T07:32:02.287 INFO:teuthology.orchestra.run.smithi137.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-18T07:32:02.287 INFO:teuthology.orchestra.run.smithi137.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-18T07:32:02.287 INFO:teuthology.orchestra.run.smithi137.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-18T07:32:02.287 INFO:teuthology.orchestra.run.smithi137.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-18T07:32:02.287 INFO:teuthology.orchestra.run.smithi137.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-18T07:32:02.287 INFO:teuthology.orchestra.run.smithi137.stdout: "motr_tracing_enabled": "false", 2024-09-18T07:32:02.288 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_op_threads": "3", 2024-09-18T07:32:02.288 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-18T07:32:02.288 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_cm": "false", 2024-09-18T07:32:02.288 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_device_name": "", 2024-09-18T07:32:02.288 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_dscp": "96", 2024-09-18T07:32:02.288 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-18T07:32:02.288 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-18T07:32:02.288 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_local_gid": "", 2024-09-18T07:32:02.288 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-18T07:32:02.288 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_port_num": "1", 2024-09-18T07:32:02.288 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-18T07:32:02.288 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-18T07:32:02.288 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-18T07:32:02.289 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-18T07:32:02.289 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_sl": "3", 2024-09-18T07:32:02.289 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_support_srq": "true", 2024-09-18T07:32:02.289 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_type": "ib", 2024-09-18T07:32:02.289 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_reap_threshold": "5", 2024-09-18T07:32:02.289 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_before_connect": "false", 2024-09-18T07:32:02.289 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_ipv4": "true", 2024-09-18T07:32:02.289 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_ipv6": "false", 2024-09-18T07:32:02.289 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_msgr1": "true", 2024-09-18T07:32:02.289 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_msgr2": "false", 2024-09-18T07:32:02.289 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_port_max": "7568", 2024-09-18T07:32:02.289 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_port_min": "6800", 2024-09-18T07:32:02.289 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-18T07:32:02.290 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_retry_count": "3", 2024-09-18T07:32:02.290 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_retry_delay": "5", 2024-09-18T07:32:02.290 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_blackhole_client": "false", 2024-09-18T07:32:02.290 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_blackhole_mds": "false", 2024-09-18T07:32:02.290 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_blackhole_mgr": "false", 2024-09-18T07:32:02.290 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_blackhole_mon": "false", 2024-09-18T07:32:02.290 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_blackhole_osd": "false", 2024-09-18T07:32:02.290 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_client_mode": "crc secure", 2024-09-18T07:32:02.290 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-18T07:32:02.290 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_cluster_mode": "crc secure", 2024-09-18T07:32:02.290 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_cluster_type": "", 2024-09-18T07:32:02.290 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_compress_secure": "false", 2024-09-18T07:32:02.290 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_connection_idle_timeout": "900", 2024-09-18T07:32:02.291 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_connection_ready_timeout": "10", 2024-09-18T07:32:02.291 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_crc_data": "true", 2024-09-18T07:32:02.291 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_crc_header": "true", 2024-09-18T07:32:02.291 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_die_on_bad_msg": "false", 2024-09-18T07:32:02.291 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_die_on_bug": "true", 2024-09-18T07:32:02.291 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_die_on_old_message": "true", 2024-09-18T07:32:02.291 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_die_on_skipped_message": "false", 2024-09-18T07:32:02.291 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-18T07:32:02.291 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-18T07:32:02.291 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_coremask": "0xF", 2024-09-18T07:32:02.291 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-18T07:32:02.291 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-18T07:32:02.291 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_enable_tso": "true", 2024-09-18T07:32:02.292 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-18T07:32:02.292 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-18T07:32:02.292 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_hugepages": "", 2024-09-18T07:32:02.292 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-18T07:32:02.292 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-18T07:32:02.292 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_lro": "true", 2024-09-18T07:32:02.292 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_memory_channel": "4", 2024-09-18T07:32:02.292 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-18T07:32:02.292 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_pmd": "", 2024-09-18T07:32:02.292 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_port_id": "0", 2024-09-18T07:32:02.292 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-18T07:32:02.292 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-18T07:32:02.292 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dump_on_send": "false", 2024-09-18T07:32:02.293 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_initial_backoff": "0.200000", 2024-09-18T07:32:02.293 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_inject_delay_max": "1.000000", 2024-09-18T07:32:02.293 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-18T07:32:02.293 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_inject_delay_type": "", 2024-09-18T07:32:02.293 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-18T07:32:02.293 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_inject_network_congestion": "0", 2024-09-18T07:32:02.293 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_inject_socket_failures": "0", 2024-09-18T07:32:02.293 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_learn_addr_from_peer": "true", 2024-09-18T07:32:02.293 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_max_accept_failures": "4", 2024-09-18T07:32:02.293 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_max_backoff": "15.000000", 2024-09-18T07:32:02.293 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_mon_client_mode": "secure crc", 2024-09-18T07:32:02.293 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-18T07:32:02.293 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_mon_service_mode": "secure crc", 2024-09-18T07:32:02.294 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_osd_compress_min_size": "1024", 2024-09-18T07:32:02.294 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_osd_compress_mode": "none", 2024-09-18T07:32:02.294 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-18T07:32:02.294 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-18T07:32:02.294 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_pq_min_cost": "65536", 2024-09-18T07:32:02.294 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_public_type": "", 2024-09-18T07:32:02.294 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_service_mode": "crc secure", 2024-09-18T07:32:02.294 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_tcp_listen_backlog": "512", 2024-09-18T07:32:02.294 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_tcp_nodelay": "true", 2024-09-18T07:32:02.294 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-18T07:32:02.294 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_tcp_rcvbuf": "0", 2024-09-18T07:32:02.294 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-18T07:32:02.294 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_type": "async+posix", 2024-09-18T07:32:02.294 INFO:teuthology.orchestra.run.smithi137.stdout: "no_config_file": "false", 2024-09-18T07:32:02.295 INFO:teuthology.orchestra.run.smithi137.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-18T07:32:02.295 INFO:teuthology.orchestra.run.smithi137.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-18T07:32:02.295 INFO:teuthology.orchestra.run.smithi137.stdout: "objecter_completion_locks_per_session": "32", 2024-09-18T07:32:02.295 INFO:teuthology.orchestra.run.smithi137.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-18T07:32:02.295 INFO:teuthology.orchestra.run.smithi137.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-18T07:32:02.295 INFO:teuthology.orchestra.run.smithi137.stdout: "objecter_inflight_ops": "1024", 2024-09-18T07:32:02.295 INFO:teuthology.orchestra.run.smithi137.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-18T07:32:02.295 INFO:teuthology.orchestra.run.smithi137.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-18T07:32:02.295 INFO:teuthology.orchestra.run.smithi137.stdout: "objecter_tick_interval": "5.000000", 2024-09-18T07:32:02.295 INFO:teuthology.orchestra.run.smithi137.stdout: "objecter_timeout": "10.000000", 2024-09-18T07:32:02.295 INFO:teuthology.orchestra.run.smithi137.stdout: "objectstore_blackhole": "false", 2024-09-18T07:32:02.295 INFO:teuthology.orchestra.run.smithi137.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-18T07:32:02.295 INFO:teuthology.orchestra.run.smithi137.stdout: "openssl_engine_opts": "", 2024-09-18T07:32:02.295 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_agent_delay_time": "5.000000", 2024-09-18T07:32:02.295 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_agent_hist_halflife": "1000", 2024-09-18T07:32:02.296 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_agent_max_low_ops": "2", 2024-09-18T07:32:02.296 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_agent_max_ops": "4", 2024-09-18T07:32:02.296 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-18T07:32:02.296 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-18T07:32:02.296 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_agent_slop": "0.020000", 2024-09-18T07:32:02.296 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-18T07:32:02.296 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-18T07:32:02.296 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_asio_thread_count": "2", 2024-09-18T07:32:02.296 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_async_recovery_min_cost": "100", 2024-09-18T07:32:02.296 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-18T07:32:02.296 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-18T07:32:02.296 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_backfill_scan_max": "512", 2024-09-18T07:32:02.296 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_backfill_scan_min": "64", 2024-09-18T07:32:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_backoff_on_degraded": "false", 2024-09-18T07:32:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_backoff_on_peering": "false", 2024-09-18T07:32:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_backoff_on_unfound": "true", 2024-09-18T07:32:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_beacon_report_interval": "300", 2024-09-18T07:32:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_bench_duration": "30", 2024-09-18T07:32:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-18T07:32:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_bench_max_block_size": "67108864", 2024-09-18T07:32:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-18T07:32:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_blkin_trace_all": "false", 2024-09-18T07:32:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-18T07:32:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-18T07:32:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-18T07:32:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-18T07:32:02.297 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_check_for_log_corruption": "false", 2024-09-18T07:32:02.298 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-18T07:32:02.298 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-18T07:32:02.298 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-18T07:32:02.298 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-18T07:32:02.298 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_class_update_on_start": "true", 2024-09-18T07:32:02.298 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_client_message_cap": "256", 2024-09-18T07:32:02.298 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_client_message_size_cap": "524288000", 2024-09-18T07:32:02.298 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_client_op_priority": "63", 2024-09-18T07:32:02.298 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_client_watch_timeout": "30", 2024-09-18T07:32:02.298 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_command_max_records": "256", 2024-09-18T07:32:02.298 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_compact_on_start": "false", 2024-09-18T07:32:02.298 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-18T07:32:02.299 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-18T07:32:02.299 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-18T07:32:02.299 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_crush_update_on_start": "true", 2024-09-18T07:32:02.299 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_crush_update_weight_set": "true", 2024-09-18T07:32:02.299 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-18T07:32:02.299 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-18T07:32:02.299 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-18T07:32:02.299 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-18T07:32:02.299 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-18T07:32:02.299 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_feed_pullee": "-1", 2024-09-18T07:32:02.299 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-18T07:32:02.299 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-18T07:32:02.299 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-18T07:32:02.299 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_misdirected_ops": "false", 2024-09-18T07:32:02.300 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_no_acting_change": "false", 2024-09-18T07:32:02.300 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_no_purge_strays": "false", 2024-09-18T07:32:02.300 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_op_order": "false", 2024-09-18T07:32:02.300 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-18T07:32:02.300 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-18T07:32:02.300 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-18T07:32:02.300 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-18T07:32:02.300 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_shutdown": "false", 2024-09-18T07:32:02.300 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-18T07:32:02.300 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-18T07:32:02.300 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_trim_objects": "false", 2024-09-18T07:32:02.300 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-18T07:32:02.300 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-18T07:32:02.300 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_verify_snaps": "false", 2024-09-18T07:32:02.300 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-18T07:32:02.301 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-18T07:32:02.301 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_deep_scrub_keys": "1024", 2024-09-18T07:32:02.301 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-18T07:32:02.301 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-18T07:32:02.301 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-18T07:32:02.301 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_deep_scrub_stride": "524288", 2024-09-18T07:32:02.301 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-18T07:32:02.301 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-18T07:32:02.301 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_default_notify_timeout": "30", 2024-09-18T07:32:02.301 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_delete_sleep": "0.000000", 2024-09-18T07:32:02.301 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-18T07:32:02.301 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-18T07:32:02.302 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-18T07:32:02.302 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_discard_disconnected_ops": "true", 2024-09-18T07:32:02.302 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_ec_partial_reads": "true", 2024-09-18T07:32:02.302 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_enable_op_tracker": "true", 2024-09-18T07:32:02.302 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-18T07:32:02.302 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-18T07:32:02.302 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-18T07:32:02.302 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-18T07:32:02.302 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_fast_info": "true", 2024-09-18T07:32:02.302 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_fast_shutdown": "true", 2024-09-18T07:32:02.302 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-18T07:32:02.302 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-18T07:32:02.302 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-18T07:32:02.302 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-18T07:32:02.303 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-18T07:32:02.303 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_function_tracing": "false", 2024-09-18T07:32:02.303 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_heartbeat_grace": "20", 2024-09-18T07:32:02.303 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_heartbeat_interval": "6", 2024-09-18T07:32:02.303 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-18T07:32:02.303 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_heartbeat_min_peers": "10", 2024-09-18T07:32:02.303 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_heartbeat_min_size": "2000", 2024-09-18T07:32:02.303 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_heartbeat_stale": "600", 2024-09-18T07:32:02.303 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-18T07:32:02.303 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_hit_set_max_size": "100000", 2024-09-18T07:32:02.303 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_hit_set_min_size": "1000", 2024-09-18T07:32:02.303 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-18T07:32:02.303 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-18T07:32:02.304 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-18T07:32:02.304 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-18T07:32:02.304 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-18T07:32:02.304 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-18T07:32:02.304 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_journal_size": "5120", 2024-09-18T07:32:02.304 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_kill_backfill_at": "0", 2024-09-18T07:32:02.304 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-18T07:32:02.304 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_map_cache_size": "50", 2024-09-18T07:32:02.304 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_map_dedup": "true", 2024-09-18T07:32:02.304 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_map_message_max": "40", 2024-09-18T07:32:02.304 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-18T07:32:02.304 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_map_share_max_epochs": "40", 2024-09-18T07:32:02.304 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_attr_name_len": "100", 2024-09-18T07:32:02.304 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_attr_size": "0", 2024-09-18T07:32:02.305 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_backfills": "1", 2024-09-18T07:32:02.305 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_markdown_count": "5", 2024-09-18T07:32:02.305 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_markdown_period": "600", 2024-09-18T07:32:02.305 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_object_name_len": "2048", 2024-09-18T07:32:02.305 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_object_namespace_len": "256", 2024-09-18T07:32:02.305 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_object_size": "134217728", 2024-09-18T07:32:02.305 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-18T07:32:02.305 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-18T07:32:02.305 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_pg_blocked_by": "16", 2024-09-18T07:32:02.305 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_pg_log_entries": "10000", 2024-09-18T07:32:02.305 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-18T07:32:02.305 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_pgls": "1024", 2024-09-18T07:32:02.306 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_push_cost": "8388608", 2024-09-18T07:32:02.306 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_push_objects": "10", 2024-09-18T07:32:02.306 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_scrubs": "3", 2024-09-18T07:32:02.306 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-18T07:32:02.306 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_trimming_pgs": "2", 2024-09-18T07:32:02.306 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_write_op_reply_len": "64", 2024-09-18T07:32:02.306 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_write_size": "90", 2024-09-18T07:32:02.306 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-18T07:32:02.306 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-18T07:32:02.306 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-18T07:32:02.306 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-18T07:32:02.306 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-18T07:32:02.307 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-18T07:32:02.307 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-18T07:32:02.307 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-18T07:32:02.307 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_profile": "balanced", 2024-09-18T07:32:02.307 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-18T07:32:02.307 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-18T07:32:02.307 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-18T07:32:02.307 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-18T07:32:02.307 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-18T07:32:02.307 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-18T07:32:02.307 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-18T07:32:02.307 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-18T07:32:02.307 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-18T07:32:02.308 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-18T07:32:02.308 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-18T07:32:02.308 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_memory_base": "805306368", 2024-09-18T07:32:02.308 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_memory_cache_min": "134217728", 2024-09-18T07:32:02.308 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-18T07:32:02.308 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-18T07:32:02.308 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_memory_target": "4294967296", 2024-09-18T07:32:02.308 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_memory_target_autotune": "false", 2024-09-18T07:32:02.308 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-18T07:32:02.308 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_min_pg_log_entries": "250", 2024-09-18T07:32:02.308 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_min_recovery_priority": "0", 2024-09-18T07:32:02.308 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-18T07:32:02.309 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-18T07:32:02.309 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mon_report_interval": "5", 2024-09-18T07:32:02.309 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-18T07:32:02.309 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-18T07:32:02.309 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_num_cache_shards": "32", 2024-09-18T07:32:02.309 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_num_op_tracker_shard": "32", 2024-09-18T07:32:02.309 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_numa_auto_affinity": "true", 2024-09-18T07:32:02.309 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_numa_node": "-1", 2024-09-18T07:32:02.309 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_numa_prefer_iface": "true", 2024-09-18T07:32:02.309 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-18T07:32:02.309 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_objecter_finishers": "1", 2024-09-18T07:32:02.309 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_objectstore": "bluestore", 2024-09-18T07:32:02.309 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_objectstore_fuse": "false", 2024-09-18T07:32:02.310 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-18T07:32:02.310 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_objectstore_tracing": "false", 2024-09-18T07:32:02.310 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_complaint_time": "30.000000", 2024-09-18T07:32:02.310 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_history_duration": "600", 2024-09-18T07:32:02.310 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_history_size": "20", 2024-09-18T07:32:02.310 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_history_slow_op_size": "20", 2024-09-18T07:32:02.310 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-18T07:32:02.310 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_log_threshold": "5", 2024-09-18T07:32:02.310 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_num_shards": "0", 2024-09-18T07:32:02.310 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_num_shards_hdd": "5", 2024-09-18T07:32:02.310 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_num_shards_ssd": "8", 2024-09-18T07:32:02.310 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-18T07:32:02.310 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-18T07:32:02.310 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-18T07:32:02.310 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-18T07:32:02.311 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_pq_min_cost": "65536", 2024-09-18T07:32:02.311 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-18T07:32:02.311 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_queue_cut_off": "high", 2024-09-18T07:32:02.311 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-18T07:32:02.311 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_thread_timeout": "15", 2024-09-18T07:32:02.311 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_open_classes_on_start": "true", 2024-09-18T07:32:02.311 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_os_flags": "0", 2024-09-18T07:32:02.311 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_peering_op_priority": "255", 2024-09-18T07:32:02.311 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_delete_cost": "1048576", 2024-09-18T07:32:02.311 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_delete_priority": "5", 2024-09-18T07:32:02.311 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-18T07:32:02.311 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-18T07:32:02.311 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-18T07:32:02.311 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_log_trim_max": "10000", 2024-09-18T07:32:02.311 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_log_trim_min": "100", 2024-09-18T07:32:02.312 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-18T07:32:02.312 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-18T07:32:02.312 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-18T07:32:02.312 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-18T07:32:02.312 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-18T07:32:02.312 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-18T07:32:02.312 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-18T07:32:02.312 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-18T07:32:02.312 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-18T07:32:02.312 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-18T07:32:02.312 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_crimson": "false", 2024-09-18T07:32:02.312 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-18T07:32:02.312 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-18T07:32:02.313 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-18T07:32:02.313 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-18T07:32:02.313 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-18T07:32:02.313 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-18T07:32:02.313 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-18T07:32:02.313 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-18T07:32:02.313 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_flags": "0", 2024-09-18T07:32:02.313 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-18T07:32:02.313 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_min_size": "0", 2024-09-18T07:32:02.313 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-18T07:32:02.313 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_pg_num": "32", 2024-09-18T07:32:02.313 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_pgp_num": "0", 2024-09-18T07:32:02.313 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-18T07:32:02.313 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_read_ratio": "70", 2024-09-18T07:32:02.314 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_size": "3", 2024-09-18T07:32:02.314 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_type": "replicated", 2024-09-18T07:32:02.314 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-18T07:32:02.314 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-18T07:32:02.314 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_push_per_object_cost": "1000", 2024-09-18T07:32:02.314 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-18T07:32:02.314 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recover_clone_overlap": "true", 2024-09-18T07:32:02.314 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-18T07:32:02.314 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_cost": "20971520", 2024-09-18T07:32:02.314 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-18T07:32:02.314 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_max_active": "0", 2024-09-18T07:32:02.314 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-18T07:32:02.314 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-18T07:32:02.314 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-18T07:32:02.315 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-18T07:32:02.315 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_max_single_start": "1", 2024-09-18T07:32:02.315 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_op_priority": "3", 2024-09-18T07:32:02.315 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-18T07:32:02.315 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_priority": "5", 2024-09-18T07:32:02.315 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-18T07:32:02.315 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_sleep": "0.000000", 2024-09-18T07:32:02.315 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-18T07:32:02.315 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-18T07:32:02.315 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-18T07:32:02.315 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_repair_during_recovery": "false", 2024-09-18T07:32:02.315 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_requested_scrub_priority": "120", 2024-09-18T07:32:02.315 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-18T07:32:02.315 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-18T07:32:02.315 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_auto_repair": "false", 2024-09-18T07:32:02.316 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-18T07:32:02.316 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-18T07:32:02.316 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_begin_hour": "0", 2024-09-18T07:32:02.316 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_begin_week_day": "0", 2024-09-18T07:32:02.316 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_chunk_max": "15", 2024-09-18T07:32:02.316 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_chunk_min": "5", 2024-09-18T07:32:02.316 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_cost": "52428800", 2024-09-18T07:32:02.316 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-18T07:32:02.316 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_during_recovery": "false", 2024-09-18T07:32:02.316 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_end_hour": "0", 2024-09-18T07:32:02.316 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_end_week_day": "0", 2024-09-18T07:32:02.316 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_event_cost": "4096", 2024-09-18T07:32:02.317 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-18T07:32:02.317 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-18T07:32:02.317 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_invalid_stats": "true", 2024-09-18T07:32:02.317 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-18T07:32:02.317 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-18T07:32:02.317 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_max_preemptions": "5", 2024-09-18T07:32:02.317 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-18T07:32:02.317 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_priority": "5", 2024-09-18T07:32:02.317 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-18T07:32:02.317 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_retry_delay": "30", 2024-09-18T07:32:02.317 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-18T07:32:02.317 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-18T07:32:02.318 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_retry_trimming": "10", 2024-09-18T07:32:02.318 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_sleep": "0.000000", 2024-09-18T07:32:02.318 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-18T07:32:02.318 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-18T07:32:02.318 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-18T07:32:02.318 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-18T07:32:02.318 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_skip_data_digest": "false", 2024-09-18T07:32:02.318 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_smart_report_timeout": "5", 2024-09-18T07:32:02.318 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_snap_trim_cost": "1048576", 2024-09-18T07:32:02.318 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_snap_trim_priority": "5", 2024-09-18T07:32:02.318 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-18T07:32:02.318 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-18T07:32:02.318 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-18T07:32:02.319 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-18T07:32:02.319 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-18T07:32:02.319 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-18T07:32:02.319 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-18T07:32:02.319 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_target_transaction_size": "30", 2024-09-18T07:32:02.319 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-18T07:32:02.319 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-18T07:32:02.319 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-18T07:32:02.319 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-18T07:32:02.319 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-18T07:32:02.319 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-18T07:32:02.319 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-18T07:32:02.320 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-18T07:32:02.320 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-18T07:32:02.320 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-18T07:32:02.320 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tracing": "false", 2024-09-18T07:32:02.320 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_use_stale_snap": "false", 2024-09-18T07:32:02.320 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-18T07:32:02.320 INFO:teuthology.orchestra.run.smithi137.stdout: "osdc_blkin_trace_all": "false", 2024-09-18T07:32:02.320 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_kill_at": "0", 2024-09-18T07:32:02.320 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_max_join_drift": "10", 2024-09-18T07:32:02.320 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_min": "500", 2024-09-18T07:32:02.320 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_min_wait": "0.050000", 2024-09-18T07:32:02.320 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_propose_interval": "1.000000", 2024-09-18T07:32:02.320 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_service_trim_max": "500", 2024-09-18T07:32:02.321 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-18T07:32:02.321 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_service_trim_min": "10", 2024-09-18T07:32:02.321 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_stash_full_interval": "25", 2024-09-18T07:32:02.321 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_trim_max": "500", 2024-09-18T07:32:02.321 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_trim_min": "250", 2024-09-18T07:32:02.321 INFO:teuthology.orchestra.run.smithi137.stdout: "perf": "true", 2024-09-18T07:32:02.321 INFO:teuthology.orchestra.run.smithi137.stdout: "pid_file": "", 2024-09-18T07:32:02.321 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-18T07:32:02.321 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-18T07:32:02.321 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "-", 2024-09-18T07:32:02.321 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrv": "", 2024-09-18T07:32:02.321 INFO:teuthology.orchestra.run.smithi137.stdout: "public_bind_addr": "-", 2024-09-18T07:32:02.321 INFO:teuthology.orchestra.run.smithi137.stdout: "public_network": "", 2024-09-18T07:32:02.322 INFO:teuthology.orchestra.run.smithi137.stdout: "public_network_interface": "", 2024-09-18T07:32:02.322 INFO:teuthology.orchestra.run.smithi137.stdout: "qat_compressor_busy_polling": "false", 2024-09-18T07:32:02.322 INFO:teuthology.orchestra.run.smithi137.stdout: "qat_compressor_enabled": "false", 2024-09-18T07:32:02.322 INFO:teuthology.orchestra.run.smithi137.stdout: "qat_compressor_session_max_number": "256", 2024-09-18T07:32:02.322 INFO:teuthology.orchestra.run.smithi137.stdout: "rados_mon_op_timeout": "0", 2024-09-18T07:32:02.322 INFO:teuthology.orchestra.run.smithi137.stdout: "rados_osd_op_timeout": "0", 2024-09-18T07:32:02.322 INFO:teuthology.orchestra.run.smithi137.stdout: "rados_tracing": "false", 2024-09-18T07:32:02.322 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_atime_update_interval": "60", 2024-09-18T07:32:02.322 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-18T07:32:02.322 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_balance_parent_reads": "false", 2024-09-18T07:32:02.322 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_balance_snap_reads": "false", 2024-09-18T07:32:02.322 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_blkin_trace_all": "false", 2024-09-18T07:32:02.322 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-18T07:32:02.322 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-18T07:32:02.323 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache": "true", 2024-09-18T07:32:02.323 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-18T07:32:02.323 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-18T07:32:02.323 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-18T07:32:02.323 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-18T07:32:02.323 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache_policy": "writearound", 2024-09-18T07:32:02.323 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache_size": "33554432", 2024-09-18T07:32:02.323 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-18T07:32:02.323 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-18T07:32:02.323 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_clone_copy_on_read": "false", 2024-09-18T07:32:02.323 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_compression_hint": "none", 2024-09-18T07:32:02.323 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_concurrent_management_ops": "10", 2024-09-18T07:32:02.323 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-18T07:32:02.324 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_clone_format": "auto", 2024-09-18T07:32:02.324 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_data_pool": "", 2024-09-18T07:32:02.324 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_features": "61", 2024-09-18T07:32:02.324 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_format": "2", 2024-09-18T07:32:02.324 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_map_options": "", 2024-09-18T07:32:02.324 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_order": "22", 2024-09-18T07:32:02.324 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_pool": "rbd", 2024-09-18T07:32:02.324 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-18T07:32:02.324 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_stripe_count": "0", 2024-09-18T07:32:02.324 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_stripe_unit": "0", 2024-09-18T07:32:02.324 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-18T07:32:02.324 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-18T07:32:02.324 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-18T07:32:02.325 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_enable_alloc_hint": "true", 2024-09-18T07:32:02.325 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-18T07:32:02.325 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_io_scheduler": "simple", 2024-09-18T07:32:02.325 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-18T07:32:02.325 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-18T07:32:02.325 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-18T07:32:02.325 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-18T07:32:02.325 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-18T07:32:02.325 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-18T07:32:02.325 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-18T07:32:02.325 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-18T07:32:02.325 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-18T07:32:02.325 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_order": "24", 2024-09-18T07:32:02.325 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_pool": "", 2024-09-18T07:32:02.326 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_splay_width": "4", 2024-09-18T07:32:02.326 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_localize_parent_reads": "false", 2024-09-18T07:32:02.326 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_localize_snap_reads": "false", 2024-09-18T07:32:02.326 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-18T07:32:02.327 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-18T07:32:02.327 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-18T07:32:02.327 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-18T07:32:02.327 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-18T07:32:02.327 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-18T07:32:02.327 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-18T07:32:02.327 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-18T07:32:02.327 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-18T07:32:02.327 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-18T07:32:02.327 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-18T07:32:02.327 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-18T07:32:02.328 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-18T07:32:02.328 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-18T07:32:02.328 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-18T07:32:02.328 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-18T07:32:02.328 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-18T07:32:02.328 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-18T07:32:02.328 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-18T07:32:02.328 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-18T07:32:02.328 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-18T07:32:02.328 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-18T07:32:02.328 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-18T07:32:02.328 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-18T07:32:02.329 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-18T07:32:02.329 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-18T07:32:02.329 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-18T07:32:02.329 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-18T07:32:02.329 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-18T07:32:02.329 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-18T07:32:02.329 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-18T07:32:02.329 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mtime_update_interval": "60", 2024-09-18T07:32:02.329 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_non_blocking_aio": "true", 2024-09-18T07:32:02.329 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_op_thread_timeout": "60", 2024-09-18T07:32:02.329 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_op_threads": "1", 2024-09-18T07:32:02.329 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_parent_cache_enabled": "false", 2024-09-18T07:32:02.330 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-18T07:32:02.330 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-18T07:32:02.330 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-18T07:32:02.330 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_plugins": "", 2024-09-18T07:32:02.330 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_bps_burst": "0", 2024-09-18T07:32:02.330 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-18T07:32:02.330 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_bps_limit": "0", 2024-09-18T07:32:02.330 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_exclude_ops": "0", 2024-09-18T07:32:02.330 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_iops_burst": "0", 2024-09-18T07:32:02.330 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-18T07:32:02.330 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_iops_limit": "0", 2024-09-18T07:32:02.330 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-18T07:32:02.330 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-18T07:32:02.331 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-18T07:32:02.331 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-18T07:32:02.331 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-18T07:32:02.331 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-18T07:32:02.331 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-18T07:32:02.331 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-18T07:32:02.331 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-18T07:32:02.331 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-18T07:32:02.331 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-18T07:32:02.331 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-18T07:32:02.331 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-18T07:32:02.331 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-18T07:32:02.331 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_read_from_replica_policy": "default", 2024-09-18T07:32:02.332 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-18T07:32:02.332 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-18T07:32:02.332 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-18T07:32:02.332 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-18T07:32:02.332 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_skip_partial_discard": "true", 2024-09-18T07:32:02.332 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-18T07:32:02.332 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_tracing": "false", 2024-09-18T07:32:02.332 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_validate_names": "true", 2024-09-18T07:32:02.332 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_validate_pool": "true", 2024-09-18T07:32:02.332 INFO:teuthology.orchestra.run.smithi137.stdout: "restapi_base_url": "", 2024-09-18T07:32:02.332 INFO:teuthology.orchestra.run.smithi137.stdout: "restapi_log_level": "", 2024-09-18T07:32:02.332 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-18T07:32:02.332 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-18T07:32:02.333 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_acl_grants_max_num": "100", 2024-09-18T07:32:02.333 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_admin_entry": "admin", 2024-09-18T07:32:02.333 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-18T07:32:02.333 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_asio_assert_yielding": "false", 2024-09-18T07:32:02.333 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_backend_store": "rados", 2024-09-18T07:32:02.333 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_barbican_url": "", 2024-09-18T07:32:02.333 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_beast_enable_async": "true", 2024-09-18T07:32:02.333 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_counters_cache": "false", 2024-09-18T07:32:02.333 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-18T07:32:02.333 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-18T07:32:02.333 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-18T07:32:02.333 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-18T07:32:02.333 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-18T07:32:02.334 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-18T07:32:02.334 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-18T07:32:02.334 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-18T07:32:02.334 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_cache_enabled": "true", 2024-09-18T07:32:02.334 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_cache_expiry_interval": "900", 2024-09-18T07:32:02.334 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_cache_lru_size": "25000", 2024-09-18T07:32:02.334 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_config_store": "rados", 2024-09-18T07:32:02.334 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_content_length_compat": "false", 2024-09-18T07:32:02.334 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_copy_obj_progress": "true", 2024-09-18T07:32:02.334 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-18T07:32:02.334 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_cors_rules_max_num": "100", 2024-09-18T07:32:02.334 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_cross_domain_policy": "", 2024-09-18T07:32:02.334 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-18T07:32:02.335 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_kmip_addr": "", 2024-09-18T07:32:02.335 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-18T07:32:02.335 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-18T07:32:02.335 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-18T07:32:02.335 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-18T07:32:02.335 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_kmip_password": "", 2024-09-18T07:32:02.335 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-18T07:32:02.335 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_kmip_username": "", 2024-09-18T07:32:02.335 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_require_ssl": "true", 2024-09-18T07:32:02.335 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-18T07:32:02.335 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-18T07:32:02.335 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-18T07:32:02.336 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-18T07:32:02.336 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-18T07:32:02.336 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-18T07:32:02.336 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-18T07:32:02.336 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-18T07:32:02.336 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-18T07:32:02.336 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-18T07:32:02.336 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-18T07:32:02.336 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-18T07:32:02.336 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-18T07:32:02.336 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-18T07:32:02.336 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-18T07:32:02.336 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_addr": "", 2024-09-18T07:32:02.337 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_auth": "token", 2024-09-18T07:32:02.337 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_namespace": "", 2024-09-18T07:32:02.337 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_prefix": "", 2024-09-18T07:32:02.337 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-18T07:32:02.337 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-18T07:32:02.337 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-18T07:32:02.337 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-18T07:32:02.337 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_token_file": "", 2024-09-18T07:32:02.337 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-18T07:32:02.337 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_curl_buffersize": "524288", 2024-09-18T07:32:02.337 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-18T07:32:02.337 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_curl_low_speed_time": "300", 2024-09-18T07:32:02.338 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-18T07:32:02.338 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-18T07:32:02.338 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-18T07:32:02.338 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-18T07:32:02.338 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-18T07:32:02.338 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-18T07:32:02.338 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-18T07:32:02.338 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-18T07:32:02.338 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-18T07:32:02.338 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-18T07:32:02.338 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-18T07:32:02.338 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-18T07:32:02.338 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-18T07:32:02.339 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-18T07:32:02.339 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-18T07:32:02.339 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-18T07:32:02.339 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-18T07:32:02.339 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-18T07:32:02.339 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-18T07:32:02.339 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_data_log_changes_size": "1000", 2024-09-18T07:32:02.339 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_data_log_num_shards": "128", 2024-09-18T07:32:02.339 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-18T07:32:02.339 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_data_log_window": "30", 2024-09-18T07:32:02.339 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-18T07:32:02.339 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-18T07:32:02.339 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-18T07:32:02.339 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-18T07:32:02.340 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-18T07:32:02.340 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-18T07:32:02.340 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-18T07:32:02.340 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-18T07:32:02.340 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-18T07:32:02.340 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-18T07:32:02.340 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-18T07:32:02.340 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-18T07:32:02.340 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-18T07:32:02.340 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_disable_s3select": "false", 2024-09-18T07:32:02.340 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-18T07:32:02.340 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-18T07:32:02.340 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-18T07:32:02.340 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-18T07:32:02.341 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-18T07:32:02.341 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-18T07:32:02.341 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-18T07:32:02.341 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-18T07:32:02.341 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-18T07:32:02.341 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-18T07:32:02.341 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-18T07:32:02.341 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-18T07:32:02.341 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dns_name": "", 2024-09-18T07:32:02.341 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dns_s3website_name": "", 2024-09-18T07:32:02.341 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dynamic_resharding": "true", 2024-09-18T07:32:02.341 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-18T07:32:02.341 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-18T07:32:02.342 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-18T07:32:02.342 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_enable_gc_threads": "true", 2024-09-18T07:32:02.342 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_enable_lc_threads": "true", 2024-09-18T07:32:02.342 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_enable_mdsearch": "true", 2024-09-18T07:32:02.342 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_enable_ops_log": "false", 2024-09-18T07:32:02.342 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_enable_quota_threads": "true", 2024-09-18T07:32:02.342 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_enable_static_website": "false", 2024-09-18T07:32:02.342 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_enable_usage_log": "false", 2024-09-18T07:32:02.342 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_enforce_swift_acls": "true", 2024-09-18T07:32:02.342 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_exit_timeout_secs": "120", 2024-09-18T07:32:02.342 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_expose_bucket": "false", 2024-09-18T07:32:02.342 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_extended_http_attrs": "", 2024-09-18T07:32:02.342 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_filter": "none", 2024-09-18T07:32:02.343 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-18T07:32:02.343 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_frontends": "beast port=7480", 2024-09-18T07:32:02.343 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-18T07:32:02.343 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_max_deferred": "50", 2024-09-18T07:32:02.343 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-18T07:32:02.343 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_max_objs": "32", 2024-09-18T07:32:02.343 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-18T07:32:02.343 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-18T07:32:02.343 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-18T07:32:02.343 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-18T07:32:02.343 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_processor_period": "3600", 2024-09-18T07:32:02.343 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-18T07:32:02.343 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-18T07:32:02.344 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-18T07:32:02.344 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-18T07:32:02.344 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_init_timeout": "300", 2024-09-18T07:32:02.344 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-18T07:32:02.344 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-18T07:32:02.344 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_kafka_connection_idle": "300", 2024-09-18T07:32:02.344 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-18T07:32:02.344 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-18T07:32:02.344 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-18T07:32:02.344 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-18T07:32:02.344 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-18T07:32:02.344 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_admin_domain": "", 2024-09-18T07:32:02.344 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_admin_password": "", 2024-09-18T07:32:02.345 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_admin_password_path": "", 2024-09-18T07:32:02.345 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_admin_project": "", 2024-09-18T07:32:02.345 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_admin_tenant": "", 2024-09-18T07:32:02.345 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_admin_token": "", 2024-09-18T07:32:02.345 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_admin_token_path": "", 2024-09-18T07:32:02.345 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_admin_user": "", 2024-09-18T07:32:02.345 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_barbican_domain": "", 2024-09-18T07:32:02.345 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_barbican_password": "", 2024-09-18T07:32:02.345 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_barbican_project": "", 2024-09-18T07:32:02.345 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-18T07:32:02.345 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_barbican_user": "", 2024-09-18T07:32:02.345 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-18T07:32:02.346 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-18T07:32:02.346 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-18T07:32:02.346 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-18T07:32:02.346 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-18T07:32:02.346 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_url": "", 2024-09-18T07:32:02.346 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-18T07:32:02.346 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lc_debug_interval": "-1", 2024-09-18T07:32:02.346 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lc_lock_max_time": "90", 2024-09-18T07:32:02.346 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lc_max_objs": "32", 2024-09-18T07:32:02.346 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lc_max_rules": "1000", 2024-09-18T07:32:02.346 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lc_max_worker": "3", 2024-09-18T07:32:02.346 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-18T07:32:02.346 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lc_thread_delay": "0", 2024-09-18T07:32:02.346 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-18T07:32:02.346 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ldap_dnattr": "uid", 2024-09-18T07:32:02.347 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-18T07:32:02.347 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ldap_searchfilter": "", 2024-09-18T07:32:02.347 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-18T07:32:02.347 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ldap_uri": "", 2024-09-18T07:32:02.347 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-18T07:32:02.347 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-18T07:32:02.347 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-18T07:32:02.347 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-18T07:32:02.347 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_log_http_headers": "", 2024-09-18T07:32:02.347 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-18T07:32:02.347 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-18T07:32:02.347 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_log_object_name_utc": "false", 2024-09-18T07:32:02.347 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-18T07:32:02.348 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-18T07:32:02.348 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_attr_name_len": "0", 2024-09-18T07:32:02.348 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_attr_size": "0", 2024-09-18T07:32:02.348 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-18T07:32:02.348 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_chunk_size": "4194304", 2024-09-18T07:32:02.348 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-18T07:32:02.348 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-18T07:32:02.348 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-18T07:32:02.348 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_listing_results": "5000", 2024-09-18T07:32:02.348 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_notify_retries": "10", 2024-09-18T07:32:02.348 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-18T07:32:02.348 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_put_param_size": "1048576", 2024-09-18T07:32:02.348 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_put_size": "5368709120", 2024-09-18T07:32:02.349 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_slo_entries": "1000", 2024-09-18T07:32:02.349 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_md_log_max_shards": "64", 2024-09-18T07:32:02.349 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-18T07:32:02.349 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-18T07:32:02.349 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-18T07:32:02.349 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-18T07:32:02.349 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_mp_lock_max_time": "600", 2024-09-18T07:32:02.349 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-18T07:32:02.349 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-18T07:32:02.349 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-18T07:32:02.349 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-18T07:32:02.349 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-18T07:32:02.349 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-18T07:32:02.349 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-18T07:32:02.350 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-18T07:32:02.350 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_max_gc": "300", 2024-09-18T07:32:02.350 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-18T07:32:02.350 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-18T07:32:02.350 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-18T07:32:02.350 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-18T07:32:02.350 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-18T07:32:02.350 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-18T07:32:02.350 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-18T07:32:02.350 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_num_async_rados_threads": "32", 2024-09-18T07:32:02.350 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_num_control_oids": "8", 2024-09-18T07:32:02.350 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_numa_node": "-1", 2024-09-18T07:32:02.350 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-18T07:32:02.351 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-18T07:32:02.351 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_objexp_chunk_size": "100", 2024-09-18T07:32:02.351 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_objexp_gc_interval": "600", 2024-09-18T07:32:02.351 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-18T07:32:02.351 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-18T07:32:02.351 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-18T07:32:02.351 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_op_thread_timeout": "600", 2024-09-18T07:32:02.351 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_op_tracing": "false", 2024-09-18T07:32:02.351 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_opa_token": "", 2024-09-18T07:32:02.351 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_opa_url": "", 2024-09-18T07:32:02.351 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_opa_verify_ssl": "true", 2024-09-18T07:32:02.351 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-18T07:32:02.351 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-18T07:32:02.352 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ops_log_rados": "false", 2024-09-18T07:32:02.352 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ops_log_socket_path": "", 2024-09-18T07:32:02.352 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-18T07:32:02.352 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-18T07:32:02.352 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-18T07:32:02.352 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_period_push_interval": "2.000000", 2024-09-18T07:32:02.352 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-18T07:32:02.352 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-18T07:32:02.352 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-18T07:32:02.352 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-18T07:32:02.352 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_posix_cache_lanes": "3", 2024-09-18T07:32:02.352 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-18T07:32:02.353 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-18T07:32:02.353 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_posix_cache_partitions": "3", 2024-09-18T07:32:02.353 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-18T07:32:02.353 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_print_continue": "true", 2024-09-18T07:32:02.353 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-18T07:32:02.353 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-18T07:32:02.353 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-18T07:32:02.353 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-18T07:32:02.353 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-18T07:32:02.353 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_rados_tracing": "false", 2024-09-18T07:32:02.353 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_realm": "", 2024-09-18T07:32:02.353 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_realm_id": "", 2024-09-18T07:32:02.354 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-18T07:32:02.354 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_region": "", 2024-09-18T07:32:02.354 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-18T07:32:02.354 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-18T07:32:02.354 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-18T07:32:02.354 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-18T07:32:02.354 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_request_uri": "", 2024-09-18T07:32:02.354 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_reshard_batch_size": "64", 2024-09-18T07:32:02.354 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-18T07:32:02.354 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-18T07:32:02.354 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_reshard_max_aio": "128", 2024-09-18T07:32:02.354 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_reshard_num_logs": "16", 2024-09-18T07:32:02.354 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-18T07:32:02.354 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-18T07:32:02.355 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_reshard_thread_interval": "600", 2024-09-18T07:32:02.355 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-18T07:32:02.355 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_resolve_cname": "false", 2024-09-18T07:32:02.355 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-18T07:32:02.355 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_run_sync_thread": "true", 2024-09-18T07:32:02.355 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-18T07:32:02.355 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-18T07:32:02.355 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-18T07:32:02.355 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-18T07:32:02.355 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-18T07:32:02.355 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-18T07:32:02.355 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-18T07:32:02.355 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-18T07:32:02.355 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-18T07:32:02.355 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_scheduler_type": "throttler", 2024-09-18T07:32:02.356 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_script_uri": "", 2024-09-18T07:32:02.356 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_service_provider_name": "", 2024-09-18T07:32:02.356 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-18T07:32:02.356 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sts_client_id": "", 2024-09-18T07:32:02.356 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sts_client_secret": "", 2024-09-18T07:32:02.356 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sts_entry": "sts", 2024-09-18T07:32:02.356 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sts_key": "sts", 2024-09-18T07:32:02.356 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-18T07:32:02.356 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sts_min_session_duration": "900", 2024-09-18T07:32:02.356 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sts_token_introspection_url": "", 2024-09-18T07:32:02.356 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_account_in_url": "false", 2024-09-18T07:32:02.356 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_auth_entry": "auth", 2024-09-18T07:32:02.356 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_auth_url": "", 2024-09-18T07:32:02.356 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_custom_header": "", 2024-09-18T07:32:02.357 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-18T07:32:02.357 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_need_stats": "true", 2024-09-18T07:32:02.357 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_tenant_name": "", 2024-09-18T07:32:02.357 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_token_expiration": "86400", 2024-09-18T07:32:02.357 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_url": "", 2024-09-18T07:32:02.357 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_url_prefix": "swift", 2024-09-18T07:32:02.357 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-18T07:32:02.357 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-18T07:32:02.357 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-18T07:32:02.357 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_lease_period": "120", 2024-09-18T07:32:02.357 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-18T07:32:02.357 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-18T07:32:02.358 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-18T07:32:02.358 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-18T07:32:02.358 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-18T07:32:02.358 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-18T07:32:02.358 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-18T07:32:02.358 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-18T07:32:02.358 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-18T07:32:02.358 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_thread_pool_size": "512", 2024-09-18T07:32:02.358 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-18T07:32:02.358 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-18T07:32:02.358 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-18T07:32:02.358 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-18T07:32:02.358 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_torrent_comment": "", 2024-09-18T07:32:02.358 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_torrent_createby": "", 2024-09-18T07:32:02.359 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_torrent_encoding": "", 2024-09-18T07:32:02.359 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_torrent_flag": "false", 2024-09-18T07:32:02.359 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-18T07:32:02.359 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_torrent_origin": "", 2024-09-18T07:32:02.359 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-18T07:32:02.359 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_torrent_tracker": "", 2024-09-18T07:32:02.359 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_trust_forwarded_https": "false", 2024-09-18T07:32:02.359 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-18T07:32:02.359 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-18T07:32:02.359 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_usage_max_shards": "32", 2024-09-18T07:32:02.359 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_usage_max_user_shards": "1", 2024-09-18T07:32:02.359 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_use_opa_authz": "false", 2024-09-18T07:32:02.360 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_counters_cache": "false", 2024-09-18T07:32:02.360 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-18T07:32:02.360 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-18T07:32:02.360 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-18T07:32:02.360 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_max_buckets": "1000", 2024-09-18T07:32:02.360 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_policies_max_num": "100", 2024-09-18T07:32:02.360 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-18T07:32:02.360 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-18T07:32:02.360 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-18T07:32:02.360 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-18T07:32:02.360 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_unique_email": "true", 2024-09-18T07:32:02.360 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_verify_ssl": "true", 2024-09-18T07:32:02.361 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-18T07:32:02.361 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_zone": "", 2024-09-18T07:32:02.361 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_zone_id": "", 2024-09-18T07:32:02.361 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-18T07:32:02.361 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_zonegroup": "", 2024-09-18T07:32:02.361 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_zonegroup_id": "", 2024-09-18T07:32:02.361 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-18T07:32:02.361 INFO:teuthology.orchestra.run.smithi137.stdout: "rgwlc_auto_session_clear": "true", 2024-09-18T07:32:02.361 INFO:teuthology.orchestra.run.smithi137.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-18T07:32:02.361 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_block_size": "4096", 2024-09-18T07:32:02.361 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-18T07:32:02.361 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-18T07:32:02.361 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-18T07:32:02.361 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-18T07:32:02.362 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-18T07:32:02.362 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cache_size": "536870912", 2024-09-18T07:32:02.362 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-18T07:32:02.362 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-18T07:32:02.362 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-18T07:32:02.362 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-18T07:32:02.362 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-18T07:32:02.362 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-18T07:32:02.362 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-18T07:32:02.362 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-18T07:32:02.362 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_index_type": "binary_search", 2024-09-18T07:32:02.362 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-18T07:32:02.362 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-18T07:32:02.362 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_partition_filters": "false", 2024-09-18T07:32:02.363 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_perf": "false", 2024-09-18T07:32:02.363 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-18T07:32:02.363 INFO:teuthology.orchestra.run.smithi137.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-18T07:32:02.363 INFO:teuthology.orchestra.run.smithi137.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-18T07:32:02.363 INFO:teuthology.orchestra.run.smithi137.stdout: "run_dir": "/var/run/ceph", 2024-09-18T07:32:02.363 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_block_create": "true", 2024-09-18T07:32:02.363 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_cache_lru_size": "67108864", 2024-09-18T07:32:02.363 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-18T07:32:02.363 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-18T07:32:02.363 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_default_max_object_size": "16777216", 2024-09-18T07:32:02.363 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-18T07:32:02.363 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_device_size": "53687091200", 2024-09-18T07:32:02.363 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-18T07:32:02.363 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_full_integrity_check": "false", 2024-09-18T07:32:02.364 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_journal_batch_capacity": "16", 2024-09-18T07:32:02.364 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-18T07:32:02.364 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-18T07:32:02.364 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-18T07:32:02.364 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_main_device_type": "SSD", 2024-09-18T07:32:02.364 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-18T07:32:02.364 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-18T07:32:02.364 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-18T07:32:02.364 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-18T07:32:02.364 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-18T07:32:02.364 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-18T07:32:02.364 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_segment_size": "67108864", 2024-09-18T07:32:02.364 INFO:teuthology.orchestra.run.smithi137.stdout: "set_keepcaps": "false", 2024-09-18T07:32:02.365 INFO:teuthology.orchestra.run.smithi137.stdout: "setgroup": "ceph", 2024-09-18T07:32:02.365 INFO:teuthology.orchestra.run.smithi137.stdout: "setuser": "ceph", 2024-09-18T07:32:02.365 INFO:teuthology.orchestra.run.smithi137.stdout: "setuser_match_path": "", 2024-09-18T07:32:02.365 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-18T07:32:02.365 INFO:teuthology.orchestra.run.smithi137.stdout: "thp": "false", 2024-09-18T07:32:02.365 INFO:teuthology.orchestra.run.smithi137.stdout: "threadpool_default_timeout": "60", 2024-09-18T07:32:02.365 INFO:teuthology.orchestra.run.smithi137.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-18T07:32:02.365 INFO:teuthology.orchestra.run.smithi137.stdout: "throttler_perf_counter": "true", 2024-09-18T07:32:02.365 INFO:teuthology.orchestra.run.smithi137.stdout: "tmp_dir": "/tmp", 2024-09-18T07:32:02.365 INFO:teuthology.orchestra.run.smithi137.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-18T07:32:02.365 INFO:teuthology.orchestra.run.smithi137.stdout: "uadk_compressor_enabled": "false", 2024-09-18T07:32:02.365 INFO:teuthology.orchestra.run.smithi137.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-18T07:32:02.365 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:32:02.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:02 smithi203 ceph-mon[32237]: 4.a scrub starts 2024-09-18T07:32:02.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:02 smithi203 ceph-mon[32237]: 4.a scrub ok 2024-09-18T07:32:02.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:02 smithi203 ceph-mon[32237]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:32:03.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-18T07:32:03.363 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:03.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:03 smithi137 ceph-mon[29408]: 4.1a scrub starts 2024-09-18T07:32:03.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:03 smithi137 ceph-mon[29408]: 4.1a scrub ok 2024-09-18T07:32:03.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:03 smithi137 ceph-mon[29408]: 4.12 scrub starts 2024-09-18T07:32:03.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:03 smithi137 ceph-mon[29408]: 4.12 scrub ok 2024-09-18T07:32:03.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:03 smithi137 ceph-mon[29408]: 2.1 scrub starts 2024-09-18T07:32:03.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:03 smithi137 ceph-mon[29408]: 2.1 scrub ok 2024-09-18T07:32:03.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:03 smithi137 ceph-mon[29408]: 5.4 scrub starts 2024-09-18T07:32:03.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:03 smithi137 ceph-mon[29408]: 5.4 scrub ok 2024-09-18T07:32:03.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:03 smithi137 ceph-mon[39175]: 4.1a scrub starts 2024-09-18T07:32:03.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:03 smithi137 ceph-mon[39175]: 4.1a scrub ok 2024-09-18T07:32:03.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:03 smithi137 ceph-mon[39175]: 4.12 scrub starts 2024-09-18T07:32:03.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:03 smithi137 ceph-mon[39175]: 4.12 scrub ok 2024-09-18T07:32:03.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:03 smithi137 ceph-mon[39175]: 2.1 scrub starts 2024-09-18T07:32:03.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:03 smithi137 ceph-mon[39175]: 2.1 scrub ok 2024-09-18T07:32:03.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:03 smithi137 ceph-mon[39175]: 5.4 scrub starts 2024-09-18T07:32:03.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:03 smithi137 ceph-mon[39175]: 5.4 scrub ok 2024-09-18T07:32:03.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:03 smithi203 ceph-mon[32237]: 4.1a scrub starts 2024-09-18T07:32:03.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:03 smithi203 ceph-mon[32237]: 4.1a scrub ok 2024-09-18T07:32:03.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:03 smithi203 ceph-mon[32237]: 4.12 scrub starts 2024-09-18T07:32:03.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:03 smithi203 ceph-mon[32237]: 4.12 scrub ok 2024-09-18T07:32:03.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:03 smithi203 ceph-mon[32237]: 2.1 scrub starts 2024-09-18T07:32:03.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:03 smithi203 ceph-mon[32237]: 2.1 scrub ok 2024-09-18T07:32:03.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:03 smithi203 ceph-mon[32237]: 5.4 scrub starts 2024-09-18T07:32:03.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:03 smithi203 ceph-mon[32237]: 5.4 scrub ok 2024-09-18T07:32:03.845 INFO:teuthology.orchestra.run.smithi137.stdout:mon.a: {} 2024-09-18T07:32:03.846 INFO:teuthology.orchestra.run.smithi137.stdout:mon.b: {} 2024-09-18T07:32:03.846 INFO:teuthology.orchestra.run.smithi137.stdout:mon.c: {} 2024-09-18T07:32:03.846 INFO:teuthology.orchestra.run.smithi137.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T07:32:03.846 INFO:teuthology.orchestra.run.smithi137.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T07:32:03.846 INFO:teuthology.orchestra.run.smithi137.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T07:32:04.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:04 smithi137 ceph-mon[29408]: 4.5 scrub starts 2024-09-18T07:32:04.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:04 smithi137 ceph-mon[29408]: 4.5 scrub ok 2024-09-18T07:32:04.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:04 smithi137 ceph-mon[29408]: 4.9 scrub starts 2024-09-18T07:32:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:04 smithi137 ceph-mon[29408]: 4.9 scrub ok 2024-09-18T07:32:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:04 smithi137 ceph-mon[29408]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:32:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:04 smithi137 ceph-mon[29408]: 3.d scrub starts 2024-09-18T07:32:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:04 smithi137 ceph-mon[29408]: 3.d scrub ok 2024-09-18T07:32:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:04 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3433489706' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:32:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:04 smithi137 ceph-mon[39175]: 4.5 scrub starts 2024-09-18T07:32:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:04 smithi137 ceph-mon[39175]: 4.5 scrub ok 2024-09-18T07:32:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:04 smithi137 ceph-mon[39175]: 4.9 scrub starts 2024-09-18T07:32:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:04 smithi137 ceph-mon[39175]: 4.9 scrub ok 2024-09-18T07:32:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:04 smithi137 ceph-mon[39175]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:32:04.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:04 smithi137 ceph-mon[39175]: 3.d scrub starts 2024-09-18T07:32:04.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:04 smithi137 ceph-mon[39175]: 3.d scrub ok 2024-09-18T07:32:04.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:04 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3433489706' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:32:04.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:04 smithi203 ceph-mon[32237]: 4.5 scrub starts 2024-09-18T07:32:04.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:04 smithi203 ceph-mon[32237]: 4.5 scrub ok 2024-09-18T07:32:04.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:04 smithi203 ceph-mon[32237]: 4.9 scrub starts 2024-09-18T07:32:04.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:04 smithi203 ceph-mon[32237]: 4.9 scrub ok 2024-09-18T07:32:04.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:04 smithi203 ceph-mon[32237]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:32:04.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:04 smithi203 ceph-mon[32237]: 3.d scrub starts 2024-09-18T07:32:04.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:04 smithi203 ceph-mon[32237]: 3.d scrub ok 2024-09-18T07:32:04.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:04 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3433489706' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:32:04.477 INFO:teuthology.run_tasks:Running task full_sequential... 2024-09-18T07:32:04.486 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-18T07:32:04.490 INFO:tasks.radosbench:Beginning radosbench... 2024-09-18T07:32:04.490 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-18T07:32:04.490 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-18T07:32:04.491 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-18T07:32:04.491 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:32:04.491 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:32:04.491 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-18T07:32:04.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-18T07:32:04.492 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-18T07:32:04.492 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:04.492 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-18T07:32:04.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:32:04.493 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-18T07:32:04.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T07:32:04.494 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-18T07:32:04.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:32:04.520 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:04.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:04.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:04.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:04.653 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:04.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:04.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:04.755 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:04.779 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:04.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:04.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:04.789 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:04.791 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:04.804 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:04.880 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:04.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:04.905 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:04.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:05.006 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:05.027 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:05.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:05.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:05.151 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:05.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:05.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:05.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[29408]: 6.1a scrub starts 2024-09-18T07:32:05.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[29408]: 6.1a scrub ok 2024-09-18T07:32:05.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[29408]: 6.4 scrub starts 2024-09-18T07:32:05.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[29408]: 6.4 scrub ok 2024-09-18T07:32:05.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[29408]: 5.8 scrub starts 2024-09-18T07:32:05.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[29408]: 5.8 scrub ok 2024-09-18T07:32:05.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[29408]: 6.12 scrub starts 2024-09-18T07:32:05.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[29408]: 6.12 scrub ok 2024-09-18T07:32:05.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[29408]: 6.10 scrub starts 2024-09-18T07:32:05.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[29408]: 6.10 scrub ok 2024-09-18T07:32:05.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[29408]: 3.7 scrub starts 2024-09-18T07:32:05.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[29408]: 3.7 scrub ok 2024-09-18T07:32:05.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[29408]: osdmap e80: 8 total, 8 up, 8 in 2024-09-18T07:32:05.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[39175]: 6.1a scrub starts 2024-09-18T07:32:05.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[39175]: 6.1a scrub ok 2024-09-18T07:32:05.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[39175]: 6.4 scrub starts 2024-09-18T07:32:05.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[39175]: 6.4 scrub ok 2024-09-18T07:32:05.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[39175]: 5.8 scrub starts 2024-09-18T07:32:05.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[39175]: 5.8 scrub ok 2024-09-18T07:32:05.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[39175]: 6.12 scrub starts 2024-09-18T07:32:05.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[39175]: 6.12 scrub ok 2024-09-18T07:32:05.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[39175]: 6.10 scrub starts 2024-09-18T07:32:05.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[39175]: 6.10 scrub ok 2024-09-18T07:32:05.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[39175]: 3.7 scrub starts 2024-09-18T07:32:05.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[39175]: 3.7 scrub ok 2024-09-18T07:32:05.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:05 smithi137 ceph-mon[39175]: osdmap e80: 8 total, 8 up, 8 in 2024-09-18T07:32:05.253 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:05.275 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:05.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:05.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:05.377 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:05.399 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:05.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:05.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:05 smithi203 ceph-mon[32237]: 6.1a scrub starts 2024-09-18T07:32:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:05 smithi203 ceph-mon[32237]: 6.1a scrub ok 2024-09-18T07:32:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:05 smithi203 ceph-mon[32237]: 6.4 scrub starts 2024-09-18T07:32:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:05 smithi203 ceph-mon[32237]: 6.4 scrub ok 2024-09-18T07:32:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:05 smithi203 ceph-mon[32237]: 5.8 scrub starts 2024-09-18T07:32:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:05 smithi203 ceph-mon[32237]: 5.8 scrub ok 2024-09-18T07:32:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:05 smithi203 ceph-mon[32237]: 6.12 scrub starts 2024-09-18T07:32:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:05 smithi203 ceph-mon[32237]: 6.12 scrub ok 2024-09-18T07:32:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:05 smithi203 ceph-mon[32237]: 6.10 scrub starts 2024-09-18T07:32:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:05 smithi203 ceph-mon[32237]: 6.10 scrub ok 2024-09-18T07:32:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:05 smithi203 ceph-mon[32237]: 3.7 scrub starts 2024-09-18T07:32:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:05 smithi203 ceph-mon[32237]: 3.7 scrub ok 2024-09-18T07:32:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:05 smithi203 ceph-mon[32237]: osdmap e80: 8 total, 8 up, 8 in 2024-09-18T07:32:05.500 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:05.521 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:05.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:05.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:05.548 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:32:05.548 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:32:05.548 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-18T07:32:05.548 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:32:05.623 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:05.645 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:05.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:05.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:05.746 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:05.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:05.768 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:05.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:05.869 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:05.893 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:05.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:05.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:06.017 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:06.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:06.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:06.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:06.141 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:06.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:06.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:06.156 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T07:32:06.219 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:06 smithi137 ceph-mon[29408]: 3.10 scrub starts 2024-09-18T07:32:06.219 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:06 smithi137 ceph-mon[29408]: 3.10 scrub ok 2024-09-18T07:32:06.219 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:06 smithi137 ceph-mon[29408]: 4.c scrub starts 2024-09-18T07:32:06.219 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:06 smithi137 ceph-mon[29408]: 4.c scrub ok 2024-09-18T07:32:06.219 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:06 smithi137 ceph-mon[29408]: 5.c scrub starts 2024-09-18T07:32:06.219 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:06 smithi137 ceph-mon[29408]: 5.c scrub ok 2024-09-18T07:32:06.219 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:06 smithi137 ceph-mon[29408]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-18T07:32:06.219 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:06 smithi137 ceph-mon[39175]: 3.10 scrub starts 2024-09-18T07:32:06.220 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:06 smithi137 ceph-mon[39175]: 3.10 scrub ok 2024-09-18T07:32:06.220 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:06 smithi137 ceph-mon[39175]: 4.c scrub starts 2024-09-18T07:32:06.220 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:06 smithi137 ceph-mon[39175]: 4.c scrub ok 2024-09-18T07:32:06.220 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:06 smithi137 ceph-mon[39175]: 5.c scrub starts 2024-09-18T07:32:06.220 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:06 smithi137 ceph-mon[39175]: 5.c scrub ok 2024-09-18T07:32:06.220 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:06 smithi137 ceph-mon[39175]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-18T07:32:06.243 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:06.265 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:06.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:06.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:06.306 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:06 smithi203 ceph-mon[32237]: 3.10 scrub starts 2024-09-18T07:32:06.306 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:06 smithi203 ceph-mon[32237]: 3.10 scrub ok 2024-09-18T07:32:06.306 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:06 smithi203 ceph-mon[32237]: 4.c scrub starts 2024-09-18T07:32:06.306 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:06 smithi203 ceph-mon[32237]: 4.c scrub ok 2024-09-18T07:32:06.306 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:06 smithi203 ceph-mon[32237]: 5.c scrub starts 2024-09-18T07:32:06.306 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:06 smithi203 ceph-mon[32237]: 5.c scrub ok 2024-09-18T07:32:06.306 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:06 smithi203 ceph-mon[32237]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-18T07:32:06.367 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:06.389 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:06.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:06.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:06.412 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:06.491 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:06.513 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:06.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:06.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:06.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:06.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:06.639 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:06.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:06.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:06.761 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:06.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:06.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:06.858 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:32:06 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a[71635]: ts=2024-09-18T07:32:06.566Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002562267s 2024-09-18T07:32:06.863 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:06.884 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:06.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:06.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:06.986 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:07.008 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:07.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:07.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:07.046 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:32:07.055 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'unique_pool_0' created 2024-09-18T07:32:07.109 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:07.110 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:32:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:32:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:32:07.110 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:07 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3333322112' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-18T07:32:07.111 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:07 smithi137 ceph-mon[29408]: from='client.24721 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-18T07:32:07.111 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:07 smithi137 ceph-mon[29408]: 3.b scrub starts 2024-09-18T07:32:07.111 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:07 smithi137 ceph-mon[29408]: 3.b scrub ok 2024-09-18T07:32:07.111 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:07 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:07.111 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:07 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2335810599' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:32:07.111 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:07 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3333322112' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-18T07:32:07.111 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:07 smithi137 ceph-mon[39175]: from='client.24721 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-18T07:32:07.111 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:07 smithi137 ceph-mon[39175]: 3.b scrub starts 2024-09-18T07:32:07.112 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:07 smithi137 ceph-mon[39175]: 3.b scrub ok 2024-09-18T07:32:07.112 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:07 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:07.112 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:07 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2335810599' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:32:07.128 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:32:06 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:32:07.129 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:07 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3333322112' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-18T07:32:07.129 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:07 smithi203 ceph-mon[32237]: from='client.24721 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-18T07:32:07.129 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:07 smithi203 ceph-mon[32237]: 3.b scrub starts 2024-09-18T07:32:07.129 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:07 smithi203 ceph-mon[32237]: 3.b scrub ok 2024-09-18T07:32:07.129 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:07 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:07.129 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:07 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2335810599' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:32:07.133 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:07.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:07.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:07.234 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:07.256 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:07.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:07.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:07.358 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:07.379 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:07.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:07.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:07.481 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:07.502 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:07.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:07.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:07.604 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:07.626 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:07.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:07.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:07.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:07.749 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:07.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:07.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:07.851 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:07.855 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-18T07:32:07.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:07.873 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:07.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:07.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:07.998 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:07.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:07.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:08.095 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:08 smithi137 ceph-mon[29408]: 6.1e scrub starts 2024-09-18T07:32:08.095 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:08 smithi137 ceph-mon[29408]: 6.1e scrub ok 2024-09-18T07:32:08.095 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:08 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:32:08.096 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:08 smithi137 ceph-mon[29408]: from='client.24721 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-18T07:32:08.096 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:08 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2335810599' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:32:08.096 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:08 smithi137 ceph-mon[29408]: osdmap e81: 8 total, 8 up, 8 in 2024-09-18T07:32:08.096 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:08 smithi137 ceph-mon[29408]: pgmap v32: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-18T07:32:08.096 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:08 smithi137 ceph-mon[39175]: 6.1e scrub starts 2024-09-18T07:32:08.096 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:08 smithi137 ceph-mon[39175]: 6.1e scrub ok 2024-09-18T07:32:08.096 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:08 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:32:08.096 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:08 smithi137 ceph-mon[39175]: from='client.24721 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-18T07:32:08.096 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:08 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2335810599' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:32:08.096 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:08 smithi137 ceph-mon[39175]: osdmap e81: 8 total, 8 up, 8 in 2024-09-18T07:32:08.097 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:08 smithi137 ceph-mon[39175]: pgmap v32: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-18T07:32:08.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:08.121 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:08.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:08.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:08.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:08.238 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:32:08.239 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:32:08.239 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 82, 2024-09-18T07:32:08.239 INFO:teuthology.orchestra.run.smithi137.stdout: "fsid": "f7ef099c-758e-11ef-bceb-c7b262605968", 2024-09-18T07:32:08.239 INFO:teuthology.orchestra.run.smithi137.stdout: "created": "2024-09-18T07:25:19.662367+0000", 2024-09-18T07:32:08.239 INFO:teuthology.orchestra.run.smithi137.stdout: "modified": "2024-09-18T07:32:08.044846+0000", 2024-09-18T07:32:08.239 INFO:teuthology.orchestra.run.smithi137.stdout: "last_up_change": "2024-09-18T07:30:44.708496+0000", 2024-09-18T07:32:08.239 INFO:teuthology.orchestra.run.smithi137.stdout: "last_in_change": "2024-09-18T07:30:23.052551+0000", 2024-09-18T07:32:08.239 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T07:32:08.239 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_num": 5801984, 2024-09-18T07:32:08.239 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_set": [ 2024-09-18T07:32:08.239 INFO:teuthology.orchestra.run.smithi137.stdout: "noscrub", 2024-09-18T07:32:08.239 INFO:teuthology.orchestra.run.smithi137.stdout: "pglog_hardlimit", 2024-09-18T07:32:08.240 INFO:teuthology.orchestra.run.smithi137.stdout: "purged_snapdirs", 2024-09-18T07:32:08.240 INFO:teuthology.orchestra.run.smithi137.stdout: "recovery_deletes", 2024-09-18T07:32:08.240 INFO:teuthology.orchestra.run.smithi137.stdout: "sortbitwise" 2024-09-18T07:32:08.240 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:32:08.240 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_version": 18, 2024-09-18T07:32:08.240 INFO:teuthology.orchestra.run.smithi137.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T07:32:08.240 INFO:teuthology.orchestra.run.smithi137.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T07:32:08.240 INFO:teuthology.orchestra.run.smithi137.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T07:32:08.240 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_snapshot": "", 2024-09-18T07:32:08.240 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_max": 7, 2024-09-18T07:32:08.241 INFO:teuthology.orchestra.run.smithi137.stdout: "max_osd": 8, 2024-09-18T07:32:08.241 INFO:teuthology.orchestra.run.smithi137.stdout: "require_min_compat_client": "luminous", 2024-09-18T07:32:08.241 INFO:teuthology.orchestra.run.smithi137.stdout: "min_compat_client": "jewel", 2024-09-18T07:32:08.241 INFO:teuthology.orchestra.run.smithi137.stdout: "require_osd_release": "squid", 2024-09-18T07:32:08.241 INFO:teuthology.orchestra.run.smithi137.stdout: "allow_crimson": false, 2024-09-18T07:32:08.241 INFO:teuthology.orchestra.run.smithi137.stdout: "pools": [ 2024-09-18T07:32:08.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.241 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 1, 2024-09-18T07:32:08.241 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".mgr", 2024-09-18T07:32:08.242 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:28:20.899655+0000", 2024-09-18T07:32:08.242 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:32:08.242 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:32:08.242 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:32:08.242 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:32:08.242 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:32:08.242 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:32:08.242 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:32:08.242 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:32:08.242 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:32:08.242 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:32:08.242 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:32:08.242 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:32:08.243 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:32:08.243 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 1, 2024-09-18T07:32:08.243 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 1, 2024-09-18T07:32:08.243 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 1, 2024-09-18T07:32:08.243 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 1, 2024-09-18T07:32:08.243 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 1, 2024-09-18T07:32:08.243 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:32:08.243 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:32:08.243 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:32:08.243 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:32:08.243 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:32:08.243 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:32:08.243 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:32:08.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.244 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "21", 2024-09-18T07:32:08.244 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:32:08.244 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:32:08.244 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:32:08.244 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:32:08.244 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:32:08.244 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:32:08.244 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:32:08.244 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:32:08.244 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:32:08.244 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:32:08.244 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:32:08.245 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:32:08.245 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:32:08.245 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:32:08.245 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:32:08.245 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:32:08.245 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:32:08.245 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:32:08.245 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:32:08.245 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:32:08.245 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:32:08.245 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:32:08.245 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:32:08.245 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:32:08.246 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:32:08.246 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:32:08.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.246 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:32:08.246 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:32:08.246 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:32:08.246 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:32:08.246 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:32:08.246 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:32:08.246 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:32:08.246 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:32:08.246 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:32:08.246 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:32:08.246 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:32:08.247 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:32:08.247 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_max": 32, 2024-09-18T07:32:08.247 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_min": 1 2024-09-18T07:32:08.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.247 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:32:08.247 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": {} 2024-09-18T07:32:08.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.247 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:32:08.247 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:32:08.247 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 7.8899998664855957, 2024-09-18T07:32:08.247 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 7.8899998664855957, 2024-09-18T07:32:08.247 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.37999999523162842, 2024-09-18T07:32:08.247 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 3, 2024-09-18T07:32:08.247 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 3, 2024-09-18T07:32:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:32:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:32:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:32:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 2, 2024-09-18T07:32:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "datapool", 2024-09-18T07:32:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:53.715373+0000", 2024-09-18T07:32:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-18T07:32:08.249 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T07:32:08.249 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:32:08.249 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:32:08.249 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:32:08.249 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:32:08.249 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:32:08.249 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:32:08.249 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:32:08.249 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:32:08.249 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:32:08.249 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:32:08.249 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:32:08.249 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 3, 2024-09-18T07:32:08.250 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 3, 2024-09-18T07:32:08.250 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 3, 2024-09-18T07:32:08.250 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 3, 2024-09-18T07:32:08.250 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 3, 2024-09-18T07:32:08.250 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:32:08.250 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:32:08.250 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:32:08.250 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:32:08.250 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:32:08.250 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:32:08.251 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:32:08.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.251 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "63", 2024-09-18T07:32:08.251 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:32:08.251 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:32:08.251 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:32:08.251 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:32:08.251 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:32:08.251 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 2, 2024-09-18T07:32:08.252 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 63, 2024-09-18T07:32:08.252 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:32:08.252 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:32:08.252 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:32:08.252 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:32:08.252 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:32:08.252 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:32:08.252 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:32:08.252 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:32:08.252 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:32:08.252 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:32:08.252 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:32:08.253 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:32:08.253 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:32:08.253 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:32:08.253 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:32:08.253 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:32:08.253 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:32:08.253 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:32:08.253 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:32:08.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.253 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:32:08.253 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:32:08.253 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:32:08.254 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:32:08.254 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:32:08.254 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:32:08.254 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:32:08.254 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:32:08.254 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:32:08.254 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:32:08.254 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:32:08.254 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:32:08.254 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:32:08.254 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd": {} 2024-09-18T07:32:08.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.254 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:32:08.254 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:32:08.255 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.6500000953674316, 2024-09-18T07:32:08.255 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.6500000953674316, 2024-09-18T07:32:08.255 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.87999999523162842, 2024-09-18T07:32:08.255 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-18T07:32:08.255 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-18T07:32:08.255 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:32:08.255 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:32:08.255 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:32:08.255 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.255 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 3, 2024-09-18T07:32:08.255 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".rgw.root", 2024-09-18T07:32:08.255 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:54.672127+0000", 2024-09-18T07:32:08.256 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:32:08.256 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:32:08.256 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:32:08.256 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:32:08.256 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:32:08.256 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:32:08.256 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:32:08.256 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:32:08.256 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:32:08.256 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:32:08.257 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:32:08.257 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:32:08.257 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:32:08.257 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:32:08.257 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:32:08.257 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:32:08.257 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:32:08.257 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:32:08.257 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:32:08.257 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:32:08.257 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:32:08.257 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:32:08.257 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:32:08.258 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:32:08.258 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:32:08.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.258 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "62", 2024-09-18T07:32:08.258 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:32:08.259 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:32:08.259 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:32:08.259 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:32:08.259 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:32:08.259 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:32:08.259 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:32:08.259 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:32:08.259 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:32:08.260 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:32:08.260 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:32:08.260 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:32:08.260 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:32:08.260 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:32:08.260 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:32:08.260 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:32:08.260 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:32:08.260 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:32:08.260 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:32:08.260 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:32:08.260 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:32:08.261 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:32:08.261 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:32:08.261 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:32:08.261 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:32:08.261 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:32:08.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.261 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:32:08.261 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:32:08.261 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:32:08.261 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:32:08.261 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:32:08.261 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:32:08.262 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:32:08.262 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:32:08.262 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:32:08.262 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:32:08.262 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:32:08.262 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:32:08.262 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:32:08.263 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:32:08.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.263 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:32:08.263 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:32:08.263 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.5, 2024-09-18T07:32:08.263 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.5, 2024-09-18T07:32:08.263 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:32:08.263 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.5, 2024-09-18T07:32:08.263 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.5, 2024-09-18T07:32:08.263 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:32:08.263 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:32:08.263 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:32:08.264 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.264 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 4, 2024-09-18T07:32:08.264 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.log", 2024-09-18T07:32:08.264 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:56.351343+0000", 2024-09-18T07:32:08.264 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:32:08.264 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:32:08.264 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:32:08.264 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:32:08.264 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:32:08.264 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:32:08.264 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:32:08.265 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:32:08.265 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:32:08.265 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:32:08.265 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:32:08.265 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:32:08.265 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:32:08.265 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:32:08.265 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:32:08.265 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:32:08.265 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:32:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:32:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:32:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:32:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:32:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:32:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:32:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:32:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:32:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "64", 2024-09-18T07:32:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:32:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:32:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:32:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:32:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:32:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:32:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:32:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:32:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:32:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:32:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:32:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:32:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:32:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:32:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:32:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:32:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:32:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:32:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:32:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:32:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:32:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:32:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:32:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:32:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:32:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:32:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:32:08.269 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:32:08.269 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:32:08.269 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:32:08.269 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:32:08.269 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:32:08.269 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:32:08.269 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:32:08.269 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:32:08.269 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:32:08.269 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:32:08.269 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:32:08.269 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:32:08.269 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:32:08.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.270 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:32:08.270 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:32:08.270 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.25, 2024-09-18T07:32:08.270 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.25, 2024-09-18T07:32:08.270 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:32:08.270 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.25, 2024-09-18T07:32:08.270 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.25, 2024-09-18T07:32:08.270 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:32:08.270 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:32:08.270 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:32:08.270 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.271 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 5, 2024-09-18T07:32:08.271 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.control", 2024-09-18T07:32:08.271 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:58.322401+0000", 2024-09-18T07:32:08.271 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:32:08.271 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:32:08.271 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:32:08.271 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:32:08.271 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:32:08.271 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:32:08.272 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:32:08.272 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:32:08.272 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:32:08.272 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:32:08.272 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:32:08.272 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:32:08.272 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:32:08.272 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:32:08.272 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:32:08.272 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:32:08.273 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:32:08.273 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:32:08.273 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:32:08.273 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:32:08.273 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:32:08.273 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:32:08.273 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:32:08.273 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:32:08.273 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:32:08.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.273 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "66", 2024-09-18T07:32:08.274 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:32:08.274 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:32:08.274 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:32:08.274 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:32:08.274 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:32:08.274 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:32:08.274 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:32:08.274 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:32:08.274 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:32:08.274 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:32:08.274 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:32:08.274 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:32:08.274 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:32:08.275 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:32:08.275 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:32:08.275 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:32:08.275 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:32:08.275 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:32:08.275 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:32:08.275 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:32:08.275 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:32:08.275 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:32:08.275 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:32:08.275 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:32:08.275 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:32:08.275 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:32:08.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.276 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:32:08.276 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:32:08.276 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:32:08.276 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:32:08.276 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:32:08.276 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:32:08.276 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:32:08.276 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:32:08.276 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:32:08.276 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:32:08.276 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:32:08.276 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:32:08.276 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:32:08.277 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:32:08.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.277 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:32:08.277 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:32:08.277 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.25, 2024-09-18T07:32:08.277 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.25, 2024-09-18T07:32:08.277 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:32:08.277 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.25, 2024-09-18T07:32:08.277 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.25, 2024-09-18T07:32:08.277 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:32:08.277 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:32:08.277 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:32:08.277 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:08.278 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:08.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:08.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.279 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 6, 2024-09-18T07:32:08.280 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.meta", 2024-09-18T07:32:08.280 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:31:00.390499+0000", 2024-09-18T07:32:08.280 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:32:08.280 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:32:08.280 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:32:08.280 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:32:08.280 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:32:08.280 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:32:08.280 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:32:08.280 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:32:08.280 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:32:08.280 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:32:08.281 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:32:08.281 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:32:08.281 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:32:08.281 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:32:08.281 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:32:08.281 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:32:08.281 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:32:08.281 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:32:08.281 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:32:08.281 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:32:08.281 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:32:08.281 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:32:08.282 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:32:08.282 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:32:08.282 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:32:08.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.282 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "68", 2024-09-18T07:32:08.282 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:32:08.282 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:32:08.282 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:32:08.282 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:32:08.282 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:32:08.282 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:32:08.282 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:32:08.282 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:32:08.282 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:32:08.283 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:32:08.283 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:32:08.283 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:32:08.283 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:32:08.283 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:32:08.283 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:32:08.283 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:32:08.283 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:32:08.284 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:32:08.284 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:32:08.284 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:32:08.284 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:32:08.284 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:32:08.284 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:32:08.284 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:32:08.284 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:32:08.284 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:32:08.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.284 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:32:08.284 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:32:08.285 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:32:08.285 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:32:08.285 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:32:08.285 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:32:08.285 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:32:08.285 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:32:08.285 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:32:08.285 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:32:08.285 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:32:08.285 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:32:08.286 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_bias": 4 2024-09-18T07:32:08.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.286 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:32:08.286 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:32:08.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.286 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:32:08.286 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:32:08.286 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.75, 2024-09-18T07:32:08.286 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.75, 2024-09-18T07:32:08.286 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:32:08.287 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.75, 2024-09-18T07:32:08.287 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.75, 2024-09-18T07:32:08.287 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:32:08.287 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:32:08.287 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:32:08.287 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.287 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-18T07:32:08.287 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "unique_pool_0", 2024-09-18T07:32:08.287 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:32:06.616722+0000", 2024-09-18T07:32:08.287 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 32769, 2024-09-18T07:32:08.288 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,creating", 2024-09-18T07:32:08.288 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:32:08.288 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:32:08.288 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:32:08.288 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:32:08.288 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:32:08.288 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:32:08.288 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:32:08.288 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:32:08.288 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:32:08.288 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:32:08.288 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:32:08.289 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 16, 2024-09-18T07:32:08.289 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 16, 2024-09-18T07:32:08.289 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 16, 2024-09-18T07:32:08.289 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 16, 2024-09-18T07:32:08.289 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 16, 2024-09-18T07:32:08.289 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:32:08.289 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:32:08.289 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:32:08.289 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:32:08.289 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:32:08.289 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:32:08.289 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:32:08.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.290 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "81", 2024-09-18T07:32:08.290 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:32:08.290 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:32:08.290 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:32:08.290 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:32:08.290 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:32:08.290 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:32:08.290 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:32:08.290 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:32:08.290 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:32:08.290 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:32:08.290 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:32:08.290 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:32:08.291 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:32:08.291 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:32:08.291 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:32:08.291 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:32:08.291 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:32:08.291 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:32:08.291 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:32:08.291 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:32:08.291 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:32:08.291 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:32:08.291 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:32:08.291 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:32:08.291 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:32:08.292 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:32:08.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.292 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:32:08.292 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:32:08.292 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:32:08.292 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:32:08.292 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:32:08.292 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:32:08.292 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:32:08.292 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:32:08.292 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:32:08.292 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:32:08.292 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:32:08.293 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:32:08.293 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": {}, 2024-09-18T07:32:08.293 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:32:08.293 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:32:08.293 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2, 2024-09-18T07:32:08.293 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2, 2024-09-18T07:32:08.293 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:32:08.293 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2, 2024-09-18T07:32:08.293 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2, 2024-09-18T07:32:08.293 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:32:08.293 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:32:08.293 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:32:08.293 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.294 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.294 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:32:08.294 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-18T07:32:08.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.294 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:32:08.294 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "268cd580-c931-4992-bb93-d6f63c2bfe88", 2024-09-18T07:32:08.294 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:32:08.294 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:32:08.294 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:32:08.294 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:32:08.295 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:32:08.295 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:32:08.295 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 8, 2024-09-18T07:32:08.295 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 81, 2024-09-18T07:32:08.295 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:32:08.295 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:32:08.295 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:32:08.295 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.295 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.295 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6801", 2024-09-18T07:32:08.295 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:32:08.295 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.296 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.296 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:32:08.296 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.296 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.296 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6802", 2024-09-18T07:32:08.296 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:32:08.296 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.296 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.296 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:32:08.296 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.297 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.297 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6804", 2024-09-18T07:32:08.297 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:32:08.297 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.297 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.297 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:32:08.297 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.297 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.297 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6803", 2024-09-18T07:32:08.297 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:32:08.298 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.298 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.298 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6801/2811444477", 2024-09-18T07:32:08.298 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6802/2811444477", 2024-09-18T07:32:08.298 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6804/2811444477", 2024-09-18T07:32:08.298 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6803/2811444477", 2024-09-18T07:32:08.298 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:32:08.298 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:32:08.298 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:32:08.298 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.298 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:32:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575", 2024-09-18T07:32:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:32:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:32:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:32:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:32:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:32:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:32:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 13, 2024-09-18T07:32:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 81, 2024-09-18T07:32:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:32:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:32:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:32:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6805", 2024-09-18T07:32:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:32:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:32:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6806", 2024-09-18T07:32:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:32:08.301 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.301 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.301 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:32:08.301 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.301 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.301 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6808", 2024-09-18T07:32:08.301 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:32:08.301 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.301 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.302 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:32:08.302 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.302 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.302 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6807", 2024-09-18T07:32:08.302 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:32:08.302 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.302 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.302 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6805/874102487", 2024-09-18T07:32:08.302 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6806/874102487", 2024-09-18T07:32:08.302 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6808/874102487", 2024-09-18T07:32:08.302 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6807/874102487", 2024-09-18T07:32:08.303 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:32:08.303 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:32:08.303 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:32:08.303 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.303 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:32:08.303 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "480ad6c5-0e0b-4405-bbe2-381682d00bae", 2024-09-18T07:32:08.303 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:32:08.303 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:32:08.303 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:32:08.303 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:32:08.303 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:32:08.304 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:32:08.304 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 18, 2024-09-18T07:32:08.304 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 81, 2024-09-18T07:32:08.304 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:32:08.304 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:32:08.304 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:32:08.304 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.304 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.305 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6809", 2024-09-18T07:32:08.305 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:32:08.305 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.305 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.305 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:32:08.305 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.305 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.305 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6810", 2024-09-18T07:32:08.305 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:32:08.305 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.305 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.306 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:32:08.306 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.306 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.306 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6812", 2024-09-18T07:32:08.306 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:32:08.306 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.306 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.306 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:32:08.306 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.307 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.307 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6811", 2024-09-18T07:32:08.307 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:32:08.307 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.307 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.307 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6809/1754684686", 2024-09-18T07:32:08.307 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6810/1754684686", 2024-09-18T07:32:08.307 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6812/1754684686", 2024-09-18T07:32:08.307 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6811/1754684686", 2024-09-18T07:32:08.307 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:32:08.307 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:32:08.307 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:32:08.308 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.308 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:32:08.308 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14", 2024-09-18T07:32:08.308 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:32:08.308 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:32:08.308 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:32:08.308 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:32:08.308 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:32:08.308 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:32:08.308 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 26, 2024-09-18T07:32:08.309 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 81, 2024-09-18T07:32:08.309 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:32:08.309 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:32:08.309 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:32:08.309 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.309 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.309 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6813", 2024-09-18T07:32:08.309 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:32:08.309 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.309 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.309 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:32:08.309 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.310 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.310 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6814", 2024-09-18T07:32:08.310 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:32:08.310 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.310 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.310 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:32:08.310 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.310 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.310 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6816", 2024-09-18T07:32:08.310 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:32:08.310 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.311 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.311 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:32:08.311 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.311 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.311 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6815", 2024-09-18T07:32:08.311 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:32:08.311 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.311 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.311 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6813/116058392", 2024-09-18T07:32:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6814/116058392", 2024-09-18T07:32:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6816/116058392", 2024-09-18T07:32:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6815/116058392", 2024-09-18T07:32:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:32:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:32:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:32:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:32:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de", 2024-09-18T07:32:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:32:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:32:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:32:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:32:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:32:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:32:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 32, 2024-09-18T07:32:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 81, 2024-09-18T07:32:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:32:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:32:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:32:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6800", 2024-09-18T07:32:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:32:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:32:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6801", 2024-09-18T07:32:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:32:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:32:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6803", 2024-09-18T07:32:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:32:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:32:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6802", 2024-09-18T07:32:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:32:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6800/3654297954", 2024-09-18T07:32:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6801/3654297954", 2024-09-18T07:32:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6803/3654297954", 2024-09-18T07:32:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6802/3654297954", 2024-09-18T07:32:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:32:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:32:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:32:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:32:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c", 2024-09-18T07:32:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:32:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:32:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:32:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:32:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:32:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:32:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 39, 2024-09-18T07:32:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 81, 2024-09-18T07:32:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:32:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:32:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:32:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6804", 2024-09-18T07:32:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:32:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:32:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6805", 2024-09-18T07:32:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:32:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:32:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.321 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6807", 2024-09-18T07:32:08.321 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:32:08.321 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.321 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.321 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:32:08.321 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.321 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.321 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6806", 2024-09-18T07:32:08.321 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:32:08.321 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.321 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.322 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6804/837994222", 2024-09-18T07:32:08.322 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6805/837994222", 2024-09-18T07:32:08.322 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6807/837994222", 2024-09-18T07:32:08.322 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6806/837994222", 2024-09-18T07:32:08.322 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:32:08.322 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:32:08.322 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:32:08.322 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.323 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:32:08.323 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6", 2024-09-18T07:32:08.323 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:32:08.323 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:32:08.323 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:32:08.323 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:32:08.324 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:32:08.324 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:32:08.324 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 47, 2024-09-18T07:32:08.324 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 81, 2024-09-18T07:32:08.324 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:32:08.324 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:32:08.325 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:32:08.325 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.325 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.325 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6808", 2024-09-18T07:32:08.325 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:32:08.325 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.326 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.326 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:32:08.326 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.327 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.328 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6809", 2024-09-18T07:32:08.328 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:32:08.328 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.328 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.329 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:32:08.329 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.329 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.329 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6811", 2024-09-18T07:32:08.329 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:32:08.329 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.329 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.329 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:32:08.330 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.330 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.330 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6810", 2024-09-18T07:32:08.330 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:32:08.330 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.330 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.330 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6808/1494175466", 2024-09-18T07:32:08.330 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6809/1494175466", 2024-09-18T07:32:08.330 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6811/1494175466", 2024-09-18T07:32:08.330 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6810/1494175466", 2024-09-18T07:32:08.330 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:32:08.331 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:32:08.331 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:32:08.331 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.331 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:32:08.331 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a", 2024-09-18T07:32:08.331 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:32:08.331 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:32:08.331 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:32:08.331 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:32:08.331 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:32:08.331 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:32:08.332 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 55, 2024-09-18T07:32:08.332 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 81, 2024-09-18T07:32:08.332 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:32:08.332 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:32:08.332 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:32:08.332 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.332 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.332 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6812", 2024-09-18T07:32:08.332 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:32:08.332 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.332 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.333 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:32:08.333 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.333 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.333 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6813", 2024-09-18T07:32:08.333 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:32:08.333 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.333 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.333 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:32:08.333 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.334 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.334 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6815", 2024-09-18T07:32:08.334 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:32:08.334 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.334 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.334 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:32:08.334 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:08.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.334 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:08.334 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6814", 2024-09-18T07:32:08.335 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:32:08.335 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.335 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.335 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6812/2670809315", 2024-09-18T07:32:08.335 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6813/2670809315", 2024-09-18T07:32:08.335 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6815/2670809315", 2024-09-18T07:32:08.335 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6814/2670809315", 2024-09-18T07:32:08.335 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:32:08.335 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:32:08.335 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:32:08.335 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:08.335 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.336 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:32:08.336 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_xinfo": [ 2024-09-18T07:32:08.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.336 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:32:08.336 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:32:08.336 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:32:08.336 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:32:08.336 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:32:08.336 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:32:08.336 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:31:18.629314+0000", 2024-09-18T07:32:08.336 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:32:08.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.337 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:32:08.337 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:32:08.337 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:32:08.337 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:32:08.337 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:32:08.337 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:32:08.337 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:31:43.978575+0000", 2024-09-18T07:32:08.337 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:32:08.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.337 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:32:08.338 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:32:08.338 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:32:08.338 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:32:08.338 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:32:08.338 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:32:08.338 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:31:59.214139+0000", 2024-09-18T07:32:08.338 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:32:08.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.338 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:32:08.338 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:32:08.338 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:32:08.338 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:32:08.338 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:32:08.339 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:32:08.339 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:31:02.992252+0000", 2024-09-18T07:32:08.339 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:32:08.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.339 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:32:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:32:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:32:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:32:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:32:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:32:08.340 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:31:36.464352+0000", 2024-09-18T07:32:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:32:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:32:08.341 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:32:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:32:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:32:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:32:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:32:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:30:53.837709+0000", 2024-09-18T07:32:08.342 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:32:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:32:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:32:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:32:08.343 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:32:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:32:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:32:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:31:27.885377+0000", 2024-09-18T07:32:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:32:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:32:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:32:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:32:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:32:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:32:08.345 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:32:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:31:44.706496+0000", 2024-09-18T07:32:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:32:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:32:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap": [], 2024-09-18T07:32:08.346 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_items": [], 2024-09-18T07:32:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_primaries": [], 2024-09-18T07:32:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_temp": [], 2024-09-18T07:32:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_temp": [], 2024-09-18T07:32:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklist": { 2024-09-18T07:32:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/2096039404": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:32:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/3636177939": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:32:08.347 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/334967443": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:32:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2101286111": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:32:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3387637412": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:32:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762989503": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:32:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3490817671": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:32:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1482147392": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:32:08.348 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1821482181": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:32:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/4028304023": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:32:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/621434521": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:32:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3160704935": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:32:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2357293499": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:32:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762352860": "2024-09-19T07:31:25.389124+0000" 2024-09-18T07:32:08.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "range_blocklist": {}, 2024-09-18T07:32:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profiles": { 2024-09-18T07:32:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "default": { 2024-09-18T07:32:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "crush-failure-domain": "osd", 2024-09-18T07:32:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "k": "2", 2024-09-18T07:32:08.350 INFO:teuthology.orchestra.run.smithi137.stdout: "m": "1", 2024-09-18T07:32:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin": "jerasure", 2024-09-18T07:32:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "technique": "reed_sol_van" 2024-09-18T07:32:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps_queue": [], 2024-09-18T07:32:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "new_removed_snaps": [], 2024-09-18T07:32:08.351 INFO:teuthology.orchestra.run.smithi137.stdout: "new_purged_snaps": [], 2024-09-18T07:32:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_node_flags": {}, 2024-09-18T07:32:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "device_class_flags": {}, 2024-09-18T07:32:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode": { 2024-09-18T07:32:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_enabled": false, 2024-09-18T07:32:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_bucket_count": 0, 2024-09-18T07:32:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "degraded_stretch_mode": 0, 2024-09-18T07:32:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "recovering_stretch_mode": 0, 2024-09-18T07:32:08.352 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_bucket": 0 2024-09-18T07:32:08.353 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:08.353 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:32:08.354 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:08 smithi203 ceph-mon[32237]: 6.1e scrub starts 2024-09-18T07:32:08.354 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:08 smithi203 ceph-mon[32237]: 6.1e scrub ok 2024-09-18T07:32:08.354 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:08 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:32:08.354 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:08 smithi203 ceph-mon[32237]: from='client.24721 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-18T07:32:08.354 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:08 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2335810599' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:32:08.354 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:08 smithi203 ceph-mon[32237]: osdmap e81: 8 total, 8 up, 8 in 2024-09-18T07:32:08.354 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:08 smithi203 ceph-mon[32237]: pgmap v32: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-18T07:32:08.380 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:08.403 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:08.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:08.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:08.505 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:08.528 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:08.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:08.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:08.552 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:08.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:08.651 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:08.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:08.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:08.699 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:08 smithi203 systemd[1]: Starting Ceph grafana.a for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:32:08.753 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:08.774 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:08.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:08.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:08.876 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:08.897 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:08.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:08.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:08.960 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:08 smithi203 podman[59952]: 2024-09-18 07:32:08.600474276 +0000 UTC m=+0.016365629 image pull c8b91775d855b99270fc5d22f3c6737e8cca01ef4c25c8b0362295e0746fa39b quay.io/ceph/grafana:10.4.0 2024-09-18T07:32:08.961 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:08 smithi203 podman[59952]: 2024-09-18 07:32:08.703418586 +0000 UTC m=+0.119309930 container create 9d9dd8054cbc2930b1dc715438c326408ff78c2a949101b453685de4f05ee7f8 (image=quay.io/ceph/grafana:10.4.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-18T07:32:08.961 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:08 smithi203 podman[59952]: 2024-09-18 07:32:08.906032572 +0000 UTC m=+0.321923926 container init 9d9dd8054cbc2930b1dc715438c326408ff78c2a949101b453685de4f05ee7f8 (image=quay.io/ceph/grafana:10.4.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-18T07:32:08.961 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:08 smithi203 podman[59952]: 2024-09-18 07:32:08.908768082 +0000 UTC m=+0.324659423 container start 9d9dd8054cbc2930b1dc715438c326408ff78c2a949101b453685de4f05ee7f8 (image=quay.io/ceph/grafana:10.4.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-18T07:32:08.999 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:09.021 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:09.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:09.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:09.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:09.143 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:09.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:09.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:09.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-mon[32237]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:32:09.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-mon[32237]: osdmap e82: 8 total, 8 up, 8 in 2024-09-18T07:32:09.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/441247796' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:32:09.214 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 bash[59952]: 9d9dd8054cbc2930b1dc715438c326408ff78c2a949101b453685de4f05ee7f8 2024-09-18T07:32:09.214 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 systemd[1]: Started Ceph grafana.a for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:32:09.214 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.036366384Z level=info msg="Starting Grafana" version=10.4.0 commit=03f502a94d17f7dc4e6c34acdf8428aedd986e4c branch=HEAD compiled=2024-09-18T07:32:09Z 2024-09-18T07:32:09.214 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.036614658Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-18T07:32:09.214 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.036622804Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-18T07:32:09.214 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.036626048Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-09-18T07:32:09.214 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.03662892Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-09-18T07:32:09.214 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.03663152Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-09-18T07:32:09.214 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.036634081Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-09-18T07:32:09.214 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.036639177Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-09-18T07:32:09.214 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.036643697Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-18T07:32:09.214 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.03664793Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-18T07:32:09.215 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.036650807Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-18T07:32:09.215 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.036653306Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-18T07:32:09.215 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.036655958Z level=info msg=Target target=[all] 2024-09-18T07:32:09.215 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.036663163Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-18T07:32:09.215 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.036665613Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-18T07:32:09.215 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.036668046Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-18T07:32:09.215 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.03667054Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-18T07:32:09.215 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.036672999Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-18T07:32:09.215 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=settings t=2024-09-18T07:32:09.036675557Z level=info msg="App mode production" 2024-09-18T07:32:09.215 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=sqlstore t=2024-09-18T07:32:09.0369257Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-18T07:32:09.215 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=sqlstore t=2024-09-18T07:32:09.036938745Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-18T07:32:09.215 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.03738305Z level=info msg="Starting DB migrations" 2024-09-18T07:32:09.215 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.038248588Z level=info msg="Executing migration" id="create migration_log table" 2024-09-18T07:32:09.215 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.0390219Z level=info msg="Migration successfully executed" id="create migration_log table" duration=772.997µs 2024-09-18T07:32:09.216 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.039483294Z level=info msg="Executing migration" id="create user table" 2024-09-18T07:32:09.216 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.040030839Z level=info msg="Migration successfully executed" id="create user table" duration=547.982µs 2024-09-18T07:32:09.216 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.040486237Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-18T07:32:09.216 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.041209952Z level=info msg="Migration successfully executed" id="add unique index user.login" duration=723.456µs 2024-09-18T07:32:09.216 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.041710581Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-18T07:32:09.216 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.042292528Z level=info msg="Migration successfully executed" id="add unique index user.email" duration=581.366µs 2024-09-18T07:32:09.216 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.04278608Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-18T07:32:09.216 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.043456394Z level=info msg="Migration successfully executed" id="drop index UQE_user_login - v1" duration=670.275µs 2024-09-18T07:32:09.216 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.044240469Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-18T07:32:09.216 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.044936795Z level=info msg="Migration successfully executed" id="drop index UQE_user_email - v1" duration=696.577µs 2024-09-18T07:32:09.216 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.045416714Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-18T07:32:09.216 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.047858546Z level=info msg="Migration successfully executed" id="Rename table user to user_v1 - v1" duration=2.440941ms 2024-09-18T07:32:09.218 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.048406429Z level=info msg="Executing migration" id="create user table v2" 2024-09-18T07:32:09.218 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.049167992Z level=info msg="Migration successfully executed" id="create user table v2" duration=761.4µs 2024-09-18T07:32:09.218 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.049614623Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-18T07:32:09.218 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.050409355Z level=info msg="Migration successfully executed" id="create index UQE_user_login - v2" duration=794.515µs 2024-09-18T07:32:09.218 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.050861222Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-18T07:32:09.218 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.051561351Z level=info msg="Migration successfully executed" id="create index UQE_user_email - v2" duration=699.663µs 2024-09-18T07:32:09.218 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.052041635Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-18T07:32:09.219 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.052414282Z level=info msg="Migration successfully executed" id="copy data_source v1 to v2" duration=372.817µs 2024-09-18T07:32:09.219 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.052926865Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-18T07:32:09.219 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.053560013Z level=info msg="Migration successfully executed" id="Drop old table user_v1" duration=636.094µs 2024-09-18T07:32:09.219 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.056053876Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-18T07:32:09.219 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.057254447Z level=info msg="Migration successfully executed" id="Add column help_flags1 to user table" duration=1.200878ms 2024-09-18T07:32:09.219 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.05891218Z level=info msg="Executing migration" id="Update user table charset" 2024-09-18T07:32:09.219 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.058941878Z level=info msg="Migration successfully executed" id="Update user table charset" duration=28.168µs 2024-09-18T07:32:09.219 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.061752504Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-18T07:32:09.219 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.06251731Z level=info msg="Migration successfully executed" id="Add last_seen_at column to user" duration=764.565µs 2024-09-18T07:32:09.219 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.064798116Z level=info msg="Executing migration" id="Add missing user data" 2024-09-18T07:32:09.219 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.064975627Z level=info msg="Migration successfully executed" id="Add missing user data" duration=177.423µs 2024-09-18T07:32:09.219 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.067625524Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-18T07:32:09.220 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.068425754Z level=info msg="Migration successfully executed" id="Add is_disabled column to user" duration=799.844µs 2024-09-18T07:32:09.220 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.070475656Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-18T07:32:09.220 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.070935595Z level=info msg="Migration successfully executed" id="Add index user.login/user.email" duration=459.987µs 2024-09-18T07:32:09.220 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.073267161Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-18T07:32:09.220 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.074014417Z level=info msg="Migration successfully executed" id="Add is_service_account column to user" duration=746.814µs 2024-09-18T07:32:09.220 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.076092819Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-18T07:32:09.220 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.081406843Z level=info msg="Migration successfully executed" id="Update is_service_account column to nullable" duration=5.308934ms 2024-09-18T07:32:09.220 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.081798113Z level=info msg="Executing migration" id="Add uid column to user" 2024-09-18T07:32:09.220 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.082635713Z level=info msg="Migration successfully executed" id="Add uid column to user" duration=837.296µs 2024-09-18T07:32:09.220 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.084686334Z level=info msg="Executing migration" id="Update uid column values for users" 2024-09-18T07:32:09.220 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.08483853Z level=info msg="Migration successfully executed" id="Update uid column values for users" duration=152.124µs 2024-09-18T07:32:09.220 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.087538491Z level=info msg="Executing migration" id="Add unique index user_uid" 2024-09-18T07:32:09.220 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.088026825Z level=info msg="Migration successfully executed" id="Add unique index user_uid" duration=488.214µs 2024-09-18T07:32:09.221 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.090535855Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-18T07:32:09.221 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.091093596Z level=info msg="Migration successfully executed" id="create temp user table v1-7" duration=557.906µs 2024-09-18T07:32:09.221 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.093617399Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-18T07:32:09.221 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.094092145Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v1-7" duration=479.414µs 2024-09-18T07:32:09.221 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.09755625Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-18T07:32:09.221 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.098129925Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v1-7" duration=573.341µs 2024-09-18T07:32:09.221 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.100429908Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-18T07:32:09.221 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.100888959Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v1-7" duration=480.076µs 2024-09-18T07:32:09.221 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.103262362Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-18T07:32:09.221 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.103713789Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v1-7" duration=451.274µs 2024-09-18T07:32:09.221 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.107286004Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-18T07:32:09.222 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.107302063Z level=info msg="Migration successfully executed" id="Update temp_user table charset" duration=16.773µs 2024-09-18T07:32:09.222 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.110072434Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-18T07:32:09.222 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.110542932Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_email - v1" duration=470.38µs 2024-09-18T07:32:09.222 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.110897461Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-18T07:32:09.222 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.111343972Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_org_id - v1" duration=446.602µs 2024-09-18T07:32:09.222 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.113676038Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-18T07:32:09.222 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.1141523Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_code - v1" duration=476.037µs 2024-09-18T07:32:09.222 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.116694355Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-18T07:32:09.222 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.117178593Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_status - v1" duration=484.067µs 2024-09-18T07:32:09.222 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.119378388Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-18T07:32:09.222 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.121446605Z level=info msg="Migration successfully executed" id="Rename table temp_user to temp_user_tmp_qwerty - v1" duration=2.06254ms 2024-09-18T07:32:09.222 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.122060346Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-18T07:32:09.222 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.122702212Z level=info msg="Migration successfully executed" id="create temp_user v2" duration=643.968µs 2024-09-18T07:32:09.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.124627144Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-18T07:32:09.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.125127851Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v2" duration=500.421µs 2024-09-18T07:32:09.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.127542414Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-18T07:32:09.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.128033276Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v2" duration=490.654µs 2024-09-18T07:32:09.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.130168162Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-18T07:32:09.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.130653242Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v2" duration=484.834µs 2024-09-18T07:32:09.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.131003314Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-18T07:32:09.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.131468398Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v2" duration=464.779µs 2024-09-18T07:32:09.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.133773327Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-18T07:32:09.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.134061871Z level=info msg="Migration successfully executed" id="copy temp_user v1 to v2" duration=288.909µs 2024-09-18T07:32:09.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.136672687Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-18T07:32:09.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.137047027Z level=info msg="Migration successfully executed" id="drop temp_user_tmp_qwerty" duration=372.257µs 2024-09-18T07:32:09.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.139390435Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-18T07:32:09.223 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.139639368Z level=info msg="Migration successfully executed" id="Set created for temp users that will otherwise prematurely expire" duration=249.401µs 2024-09-18T07:32:09.224 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.142109917Z level=info msg="Executing migration" id="create star table" 2024-09-18T07:32:09.224 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.142538019Z level=info msg="Migration successfully executed" id="create star table" duration=428.495µs 2024-09-18T07:32:09.224 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.144787013Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-18T07:32:09.224 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.145279856Z level=info msg="Migration successfully executed" id="add unique index star.user_id_dashboard_id" duration=491.472µs 2024-09-18T07:32:09.224 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.147513183Z level=info msg="Executing migration" id="create org table v1" 2024-09-18T07:32:09.224 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.148040188Z level=info msg="Migration successfully executed" id="create org table v1" duration=526.712µs 2024-09-18T07:32:09.224 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.150425963Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-18T07:32:09.224 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.151025708Z level=info msg="Migration successfully executed" id="create index UQE_org_name - v1" duration=605.559µs 2024-09-18T07:32:09.224 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.153065134Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-18T07:32:09.224 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.153506224Z level=info msg="Migration successfully executed" id="create org_user table v1" duration=440.897µs 2024-09-18T07:32:09.224 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.1538633Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-18T07:32:09.224 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.154374542Z level=info msg="Migration successfully executed" id="create index IDX_org_user_org_id - v1" duration=510.878µs 2024-09-18T07:32:09.225 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.15474061Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-18T07:32:09.225 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.155253021Z level=info msg="Migration successfully executed" id="create index UQE_org_user_org_id_user_id - v1" duration=512.1µs 2024-09-18T07:32:09.225 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.155612775Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-18T07:32:09.225 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.156106564Z level=info msg="Migration successfully executed" id="create index IDX_org_user_user_id - v1" duration=493.535µs 2024-09-18T07:32:09.225 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.156483926Z level=info msg="Executing migration" id="Update org table charset" 2024-09-18T07:32:09.225 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.156499179Z level=info msg="Migration successfully executed" id="Update org table charset" duration=15.956µs 2024-09-18T07:32:09.225 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.157347908Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-18T07:32:09.225 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.157366613Z level=info msg="Migration successfully executed" id="Update org_user table charset" duration=19.664µs 2024-09-18T07:32:09.225 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.158306059Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-18T07:32:09.225 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.158580608Z level=info msg="Migration successfully executed" id="Migrate all Read Only Viewers to Viewers" duration=278.084µs 2024-09-18T07:32:09.225 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.158997219Z level=info msg="Executing migration" id="create dashboard table" 2024-09-18T07:32:09.225 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.159524212Z level=info msg="Migration successfully executed" id="create dashboard table" duration=527.163µs 2024-09-18T07:32:09.226 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.160012629Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-18T07:32:09.226 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.160550187Z level=info msg="Migration successfully executed" id="add index dashboard.account_id" duration=537.341µs 2024-09-18T07:32:09.226 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.160915038Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-18T07:32:09.226 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.161451261Z level=info msg="Migration successfully executed" id="add unique index dashboard_account_id_slug" duration=535.946µs 2024-09-18T07:32:09.226 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.161819968Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-18T07:32:09.226 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.162285939Z level=info msg="Migration successfully executed" id="create dashboard_tag table" duration=466.346µs 2024-09-18T07:32:09.226 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.162649349Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-18T07:32:09.226 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.163155225Z level=info msg="Migration successfully executed" id="add unique index dashboard_tag.dasboard_id_term" duration=504.051µs 2024-09-18T07:32:09.226 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.163736081Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-18T07:32:09.226 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.164490754Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" duration=754.646µs 2024-09-18T07:32:09.226 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.165210569Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-18T07:32:09.226 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.169073308Z level=info msg="Migration successfully executed" id="Rename table dashboard to dashboard_v1 - v1" duration=3.863291ms 2024-09-18T07:32:09.226 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.169498755Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-18T07:32:09.226 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.17000931Z level=info msg="Migration successfully executed" id="create dashboard v2" duration=510.371µs 2024-09-18T07:32:09.227 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.170349707Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-18T07:32:09.227 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.170811785Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_org_id - v2" duration=461.938µs 2024-09-18T07:32:09.227 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.171215382Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-18T07:32:09.227 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.171694422Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_org_id_slug - v2" duration=478.796µs 2024-09-18T07:32:09.227 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.172060555Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-18T07:32:09.227 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.172301681Z level=info msg="Migration successfully executed" id="copy dashboard v1 to v2" duration=240.973µs 2024-09-18T07:32:09.227 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.173087884Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-18T07:32:09.227 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.173815013Z level=info msg="Migration successfully executed" id="drop table dashboard_v1" duration=728.985µs 2024-09-18T07:32:09.227 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.174171637Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-18T07:32:09.227 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.174210503Z level=info msg="Migration successfully executed" id="alter dashboard.data to mediumtext v1" duration=39.139µs 2024-09-18T07:32:09.227 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.17503561Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-18T07:32:09.227 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.176137554Z level=info msg="Migration successfully executed" id="Add column updated_by in dashboard - v2" duration=1.10154ms 2024-09-18T07:32:09.227 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.176482809Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-18T07:32:09.228 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.177553857Z level=info msg="Migration successfully executed" id="Add column created_by in dashboard - v2" duration=1.070745ms 2024-09-18T07:32:09.228 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.177911112Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-18T07:32:09.228 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.178999509Z level=info msg="Migration successfully executed" id="Add column gnetId in dashboard" duration=1.088131ms 2024-09-18T07:32:09.228 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.179343295Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-18T07:32:09.228 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.179814003Z level=info msg="Migration successfully executed" id="Add index for gnetId in dashboard" duration=470.536µs 2024-09-18T07:32:09.228 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.180186912Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-18T07:32:09.228 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.18129008Z level=info msg="Migration successfully executed" id="Add column plugin_id in dashboard" duration=1.10284ms 2024-09-18T07:32:09.228 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.181648815Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-18T07:32:09.228 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.182145098Z level=info msg="Migration successfully executed" id="Add index for plugin_id in dashboard" duration=494.804µs 2024-09-18T07:32:09.228 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.182496526Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-18T07:32:09.228 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.182978713Z level=info msg="Migration successfully executed" id="Add index for dashboard_id in dashboard_tag" duration=482.01µs 2024-09-18T07:32:09.228 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.183335996Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-18T07:32:09.228 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.183351718Z level=info msg="Migration successfully executed" id="Update dashboard table charset" duration=16.401µs 2024-09-18T07:32:09.229 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.184176548Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-18T07:32:09.229 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.184193607Z level=info msg="Migration successfully executed" id="Update dashboard_tag table charset" duration=17.554µs 2024-09-18T07:32:09.229 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.185230121Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-18T07:32:09.229 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.186420132Z level=info msg="Migration successfully executed" id="Add column folder_id in dashboard" duration=1.189672ms 2024-09-18T07:32:09.229 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.186785161Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-18T07:32:09.229 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.187924357Z level=info msg="Migration successfully executed" id="Add column isFolder in dashboard" duration=1.138942ms 2024-09-18T07:32:09.229 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.188299684Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-18T07:32:09.229 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.189483625Z level=info msg="Migration successfully executed" id="Add column has_acl in dashboard" duration=1.183698ms 2024-09-18T07:32:09.229 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.189850649Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-18T07:32:09.229 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.19100892Z level=info msg="Migration successfully executed" id="Add column uid in dashboard" duration=1.157983ms 2024-09-18T07:32:09.229 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.191354002Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-18T07:32:09.229 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.191487419Z level=info msg="Migration successfully executed" id="Update uid column values in dashboard" duration=133.559µs 2024-09-18T07:32:09.229 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.192195373Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-18T07:32:09.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.192677845Z level=info msg="Migration successfully executed" id="Add unique index dashboard_org_id_uid" duration=466.78µs 2024-09-18T07:32:09.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.193048068Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-18T07:32:09.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.193517145Z level=info msg="Migration successfully executed" id="Remove unique index org_id_slug" duration=468.106µs 2024-09-18T07:32:09.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.195863258Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-18T07:32:09.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.1958961Z level=info msg="Migration successfully executed" id="Update dashboard title length" duration=33.523µs 2024-09-18T07:32:09.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.198699853Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-18T07:32:09.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.199228391Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_title_folder_id" duration=528.138µs 2024-09-18T07:32:09.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.201457984Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-18T07:32:09.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.201902655Z level=info msg="Migration successfully executed" id="create dashboard_provisioning" duration=444.98µs 2024-09-18T07:32:09.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.202282088Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-18T07:32:09.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.205345782Z level=info msg="Migration successfully executed" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" duration=3.063568ms 2024-09-18T07:32:09.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.205714053Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-18T07:32:09.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.206171239Z level=info msg="Migration successfully executed" id="create dashboard_provisioning v2" duration=456.993µs 2024-09-18T07:32:09.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.208549118Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-18T07:32:09.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.209068706Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id - v2" duration=519.434µs 2024-09-18T07:32:09.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.211359121Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-18T07:32:09.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.211884534Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" duration=525.049µs 2024-09-18T07:32:09.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.212265611Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-18T07:32:09.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.212477201Z level=info msg="Migration successfully executed" id="copy dashboard_provisioning v1 to v2" duration=211.534µs 2024-09-18T07:32:09.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.21308132Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-18T07:32:09.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.213454152Z level=info msg="Migration successfully executed" id="drop dashboard_provisioning_tmp_qwerty" duration=372.588µs 2024-09-18T07:32:09.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.21392841Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-18T07:32:09.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.215536329Z level=info msg="Migration successfully executed" id="Add check_sum column" duration=1.607095ms 2024-09-18T07:32:09.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.215931666Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-18T07:32:09.245 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:09.266 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:09.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:09.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:09.367 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:09 smithi137 ceph-mon[29408]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:32:09.367 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:09 smithi137 ceph-mon[29408]: osdmap e82: 8 total, 8 up, 8 in 2024-09-18T07:32:09.367 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:09 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/441247796' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:32:09.368 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:09 smithi137 ceph-mon[39175]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:32:09.368 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:09 smithi137 ceph-mon[39175]: osdmap e82: 8 total, 8 up, 8 in 2024-09-18T07:32:09.368 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:09 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/441247796' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:32:09.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:09.389 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:09.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:09.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:09.401 INFO:tasks.thrashosds.thrasher:j is {'epoch': 82, 'fsid': 'f7ef099c-758e-11ef-bceb-c7b262605968', 'created': '2024-09-18T07:25:19.662367+0000', 'modified': '2024-09-18T07:32:08.044846+0000', 'last_up_change': '2024-09-18T07:30:44.708496+0000', 'last_in_change': '2024-09-18T07:30:23.052551+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T07:28:20.899655+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-18T07:30:53.715373+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-18T07:30:54.672127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-18T07:30:56.351343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-18T07:30:58.322401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-18T07:31:00.390499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-18T07:32:06.616722+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '268cd580-c931-4992-bb93-d6f63c2bfe88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6801', 'nonce': 2811444477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6802', 'nonce': 2811444477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6804', 'nonce': 2811444477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6803', 'nonce': 2811444477}]}, 'public_addr': '172.21.15.137:6801/2811444477', 'cluster_addr': '172.21.15.137:6802/2811444477', 'heartbeat_back_addr': '172.21.15.137:6804/2811444477', 'heartbeat_front_addr': '172.21.15.137:6803/2811444477', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b06907bc-b499-46c0-8667-c5bdc0d85575', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6805', 'nonce': 874102487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6806', 'nonce': 874102487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6808', 'nonce': 874102487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6807', 'nonce': 874102487}]}, 'public_addr': '172.21.15.137:6805/874102487', 'cluster_addr': '172.21.15.137:6806/874102487', 'heartbeat_back_addr': '172.21.15.137:6808/874102487', 'heartbeat_front_addr': '172.21.15.137:6807/874102487', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '480ad6c5-0e0b-4405-bbe2-381682d00bae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6809', 'nonce': 1754684686}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6810', 'nonce': 1754684686}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6812', 'nonce': 1754684686}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6811', 'nonce': 1754684686}]}, 'public_addr': '172.21.15.137:6809/1754684686', 'cluster_addr': '172.21.15.137:6810/1754684686', 'heartbeat_back_addr': '172.21.15.137:6812/1754684686', 'heartbeat_front_addr': '172.21.15.137:6811/1754684686', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '178be3e0-f07b-4856-807b-2a1181e48a14', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6813', 'nonce': 116058392}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6814', 'nonce': 116058392}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6816', 'nonce': 116058392}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6815', 'nonce': 116058392}]}, 'public_addr': '172.21.15.137:6813/116058392', 'cluster_addr': '172.21.15.137:6814/116058392', 'heartbeat_back_addr': '172.21.15.137:6816/116058392', 'heartbeat_front_addr': '172.21.15.137:6815/116058392', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7547102c-951a-4077-a094-fb8a65bfe6de', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6800', 'nonce': 3654297954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6801', 'nonce': 3654297954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6803', 'nonce': 3654297954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6802', 'nonce': 3654297954}]}, 'public_addr': '172.21.15.203:6800/3654297954', 'cluster_addr': '172.21.15.203:6801/3654297954', 'heartbeat_back_addr': '172.21.15.203:6803/3654297954', 'heartbeat_front_addr': '172.21.15.203:6802/3654297954', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '49c17901-8102-48a4-9348-fa1bdb63032c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6804', 'nonce': 837994222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6805', 'nonce': 837994222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6807', 'nonce': 837994222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6806', 'nonce': 837994222}]}, 'public_addr': '172.21.15.203:6804/837994222', 'cluster_addr': '172.21.15.203:6805/837994222', 'heartbeat_back_addr': '172.21.15.203:6807/837994222', 'heartbeat_front_addr': '172.21.15.203:6806/837994222', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '75f0ead4-e981-4a15-b1c1-d83dce871ab6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6808', 'nonce': 1494175466}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6809', 'nonce': 1494175466}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6811', 'nonce': 1494175466}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6810', 'nonce': 1494175466}]}, 'public_addr': '172.21.15.203:6808/1494175466', 'cluster_addr': '172.21.15.203:6809/1494175466', 'heartbeat_back_addr': '172.21.15.203:6811/1494175466', 'heartbeat_front_addr': '172.21.15.203:6810/1494175466', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c0dbf299-6798-4b99-9a85-62b123971f2a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 55, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6812', 'nonce': 2670809315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6813', 'nonce': 2670809315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6815', 'nonce': 2670809315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6814', 'nonce': 2670809315}]}, 'public_addr': '172.21.15.203:6812/2670809315', 'cluster_addr': '172.21.15.203:6813/2670809315', 'heartbeat_back_addr': '172.21.15.203:6815/2670809315', 'heartbeat_front_addr': '172.21.15.203:6814/2670809315', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:31:18.629314+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:31:43.978575+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:31:59.214139+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:31:02.992252+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:31:36.464352+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:30:53.837709+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:31:27.885377+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:31:44.706496+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.137:6800/2096039404': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:6800/3636177939': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/334967443': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/2101286111': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/3387637412': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1762989503': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3490817671': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:6800/1482147392': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1821482181': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/4028304023': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/621434521': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3160704935': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/2357293499': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/1762352860': '2024-09-19T07:31:25.389124+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T07:32:09.402 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-18T07:32:09.464 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.21645963Z level=info msg="Migration successfully executed" id="Add index for dashboard_title" duration=527.73µs 2024-09-18T07:32:09.465 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.216831431Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-18T07:32:09.465 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.216967583Z level=info msg="Migration successfully executed" id="delete tags for deleted dashboards" duration=136.147µs 2024-09-18T07:32:09.465 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.217708056Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-18T07:32:09.465 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.217834014Z level=info msg="Migration successfully executed" id="delete stars for deleted dashboards" duration=126.255µs 2024-09-18T07:32:09.465 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.218548325Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-18T07:32:09.465 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.219064497Z level=info msg="Migration successfully executed" id="Add index for dashboard_is_folder" duration=515.95µs 2024-09-18T07:32:09.465 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.219447747Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-18T07:32:09.465 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.220743945Z level=info msg="Migration successfully executed" id="Add isPublic for dashboard" duration=1.295787ms 2024-09-18T07:32:09.465 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.221142735Z level=info msg="Executing migration" id="create data_source table" 2024-09-18T07:32:09.465 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.221771027Z level=info msg="Migration successfully executed" id="create data_source table" duration=628.061µs 2024-09-18T07:32:09.465 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.222223057Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-18T07:32:09.466 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.222733719Z level=info msg="Migration successfully executed" id="add index data_source.account_id" duration=509.863µs 2024-09-18T07:32:09.466 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.223106848Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-18T07:32:09.466 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.223616833Z level=info msg="Migration successfully executed" id="add unique index data_source.account_id_name" duration=509.741µs 2024-09-18T07:32:09.466 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.224035026Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-18T07:32:09.466 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.224529637Z level=info msg="Migration successfully executed" id="drop index IDX_data_source_account_id - v1" duration=494.474µs 2024-09-18T07:32:09.467 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.224892873Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-18T07:32:09.467 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.225399504Z level=info msg="Migration successfully executed" id="drop index UQE_data_source_account_id_name - v1" duration=506.569µs 2024-09-18T07:32:09.467 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.225780744Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-18T07:32:09.467 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.22923693Z level=info msg="Migration successfully executed" id="Rename table data_source to data_source_v1 - v1" duration=3.455874ms 2024-09-18T07:32:09.467 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.229627887Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-18T07:32:09.467 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.230250016Z level=info msg="Migration successfully executed" id="create data_source table v2" duration=621.836µs 2024-09-18T07:32:09.467 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.232583024Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-18T07:32:09.467 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.233135917Z level=info msg="Migration successfully executed" id="create index IDX_data_source_org_id - v2" duration=557.64µs 2024-09-18T07:32:09.467 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.233506043Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-18T07:32:09.467 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.2340473Z level=info msg="Migration successfully executed" id="create index UQE_data_source_org_id_name - v2" duration=541.03µs 2024-09-18T07:32:09.467 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.237673774Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-18T07:32:09.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.238113472Z level=info msg="Migration successfully executed" id="Drop old table data_source_v1 #2" duration=439.374µs 2024-09-18T07:32:09.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.240560474Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-18T07:32:09.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.242119125Z level=info msg="Migration successfully executed" id="Add column with_credentials" duration=1.562012ms 2024-09-18T07:32:09.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.24316117Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-18T07:32:09.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.244574051Z level=info msg="Migration successfully executed" id="Add secure json data column" duration=1.412569ms 2024-09-18T07:32:09.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.244971827Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-18T07:32:09.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.24498846Z level=info msg="Migration successfully executed" id="Update data_source table charset" duration=17.508µs 2024-09-18T07:32:09.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.248053422Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-18T07:32:09.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.24833803Z level=info msg="Migration successfully executed" id="Update initial version to 1" duration=287.699µs 2024-09-18T07:32:09.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.250586575Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-18T07:32:09.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.252014344Z level=info msg="Migration successfully executed" id="Add read_only data column" duration=1.427313ms 2024-09-18T07:32:09.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.253545388Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-18T07:32:09.468 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.253844797Z level=info msg="Migration successfully executed" id="Migrate logging ds to loki ds" duration=302.231µs 2024-09-18T07:32:09.469 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.254273874Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-18T07:32:09.469 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.254410965Z level=info msg="Migration successfully executed" id="Update json_data with nulls" duration=137.166µs 2024-09-18T07:32:09.469 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.256934855Z level=info msg="Executing migration" id="Add uid column" 2024-09-18T07:32:09.469 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.258398784Z level=info msg="Migration successfully executed" id="Add uid column" duration=1.463591ms 2024-09-18T07:32:09.469 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.260103901Z level=info msg="Executing migration" id="Update uid value" 2024-09-18T07:32:09.469 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.260299143Z level=info msg="Migration successfully executed" id="Update uid value" duration=197.723µs 2024-09-18T07:32:09.469 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.262710276Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-18T07:32:09.469 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.263328627Z level=info msg="Migration successfully executed" id="Add unique index datasource_org_id_uid" duration=618.14µs 2024-09-18T07:32:09.469 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.263709695Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-18T07:32:09.469 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.264227761Z level=info msg="Migration successfully executed" id="add unique index datasource_org_id_is_default" duration=517.896µs 2024-09-18T07:32:09.469 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.266457365Z level=info msg="Executing migration" id="create api_key table" 2024-09-18T07:32:09.469 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.267031772Z level=info msg="Migration successfully executed" id="create api_key table" duration=573.943µs 2024-09-18T07:32:09.469 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.269289909Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-18T07:32:09.470 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.26981599Z level=info msg="Migration successfully executed" id="add index api_key.account_id" duration=525.628µs 2024-09-18T07:32:09.470 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.272123194Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-18T07:32:09.470 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.272619584Z level=info msg="Migration successfully executed" id="add index api_key.key" duration=496.063µs 2024-09-18T07:32:09.470 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.27302349Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-18T07:32:09.470 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.273541268Z level=info msg="Migration successfully executed" id="add index api_key.account_id_name" duration=523.742µs 2024-09-18T07:32:09.470 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.275703497Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-18T07:32:09.470 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.276210655Z level=info msg="Migration successfully executed" id="drop index IDX_api_key_account_id - v1" duration=506.854µs 2024-09-18T07:32:09.470 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.276571339Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-18T07:32:09.470 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.277058753Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_key - v1" duration=487.298µs 2024-09-18T07:32:09.470 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.277389867Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-18T07:32:09.470 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.277861275Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_account_id_name - v1" duration=471.222µs 2024-09-18T07:32:09.470 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.27820142Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-18T07:32:09.470 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.281941865Z level=info msg="Migration successfully executed" id="Rename table api_key to api_key_v1 - v1" duration=3.740098ms 2024-09-18T07:32:09.471 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.282321519Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-18T07:32:09.471 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.282806389Z level=info msg="Migration successfully executed" id="create api_key table v2" duration=484.786µs 2024-09-18T07:32:09.471 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.283168338Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-18T07:32:09.471 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.283660029Z level=info msg="Migration successfully executed" id="create index IDX_api_key_org_id - v2" duration=490.624µs 2024-09-18T07:32:09.471 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.283999248Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-18T07:32:09.471 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.28447283Z level=info msg="Migration successfully executed" id="create index UQE_api_key_key - v2" duration=473.431µs 2024-09-18T07:32:09.471 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.28480088Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-18T07:32:09.471 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.285306625Z level=info msg="Migration successfully executed" id="create index UQE_api_key_org_id_name - v2" duration=505.451µs 2024-09-18T07:32:09.471 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.285661708Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-18T07:32:09.471 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.285897037Z level=info msg="Migration successfully executed" id="copy api_key v1 to v2" duration=236.126µs 2024-09-18T07:32:09.471 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.286240658Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-18T07:32:09.471 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.286616204Z level=info msg="Migration successfully executed" id="Drop old table api_key_v1" duration=375.577µs 2024-09-18T07:32:09.471 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.286968257Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-18T07:32:09.472 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.286985745Z level=info msg="Migration successfully executed" id="Update api_key table charset" duration=17.704µs 2024-09-18T07:32:09.472 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.287385146Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-18T07:32:09.472 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.288844632Z level=info msg="Migration successfully executed" id="Add expires to api_key table" duration=1.459256ms 2024-09-18T07:32:09.472 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.289206993Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-18T07:32:09.472 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.290732311Z level=info msg="Migration successfully executed" id="Add service account foreign key" duration=1.509579ms 2024-09-18T07:32:09.472 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.291089883Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-18T07:32:09.472 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.2911918Z level=info msg="Migration successfully executed" id="set service account foreign key to nil if 0" duration=102.1µs 2024-09-18T07:32:09.472 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.291599795Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-18T07:32:09.472 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.293145864Z level=info msg="Migration successfully executed" id="Add last_used_at to api_key table" duration=1.545588ms 2024-09-18T07:32:09.472 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.293482776Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-18T07:32:09.474 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.295041886Z level=info msg="Migration successfully executed" id="Add is_revoked column to api_key table" duration=1.557798ms 2024-09-18T07:32:09.474 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.295388611Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-18T07:32:09.474 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.295885135Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v4" duration=495.958µs 2024-09-18T07:32:09.474 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.296240107Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-18T07:32:09.474 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.296614772Z level=info msg="Migration successfully executed" id="drop table dashboard_snapshot_v4 #1" duration=374.529µs 2024-09-18T07:32:09.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.296970848Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-18T07:32:09.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.297516077Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v5 #2" duration=545.167µs 2024-09-18T07:32:09.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.297861762Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-18T07:32:09.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.298382447Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_key - v5" duration=520.437µs 2024-09-18T07:32:09.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.298767509Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-18T07:32:09.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.299304387Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_delete_key - v5" duration=536.616µs 2024-09-18T07:32:09.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.299634293Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-18T07:32:09.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.300137515Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_snapshot_user_id - v5" duration=502.94µs 2024-09-18T07:32:09.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.300467857Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-18T07:32:09.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.300508105Z level=info msg="Migration successfully executed" id="alter dashboard_snapshot to mediumtext v2" duration=39.393µs 2024-09-18T07:32:09.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.300886323Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-18T07:32:09.475 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.300901988Z level=info msg="Migration successfully executed" id="Update dashboard_snapshot table charset" duration=16.248µs 2024-09-18T07:32:09.476 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.301296737Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-18T07:32:09.476 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.302886991Z level=info msg="Migration successfully executed" id="Add column external_delete_url to dashboard_snapshots table" duration=1.589858ms 2024-09-18T07:32:09.476 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.303246971Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-18T07:32:09.476 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.304833052Z level=info msg="Migration successfully executed" id="Add encrypted dashboard json column" duration=1.585888ms 2024-09-18T07:32:09.476 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.305217453Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-18T07:32:09.476 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.305257923Z level=info msg="Migration successfully executed" id="Change dashboard_encrypted column to MEDIUMBLOB" duration=39.819µs 2024-09-18T07:32:09.476 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.305648913Z level=info msg="Executing migration" id="create quota table v1" 2024-09-18T07:32:09.476 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.3061199Z level=info msg="Migration successfully executed" id="create quota table v1" duration=470.665µs 2024-09-18T07:32:09.476 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.306457117Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-18T07:32:09.476 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.306944794Z level=info msg="Migration successfully executed" id="create index UQE_quota_org_id_user_id_target - v1" duration=487.265µs 2024-09-18T07:32:09.476 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.307312294Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-18T07:32:09.476 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.307334971Z level=info msg="Migration successfully executed" id="Update quota table charset" duration=22.233µs 2024-09-18T07:32:09.476 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.30773281Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-18T07:32:09.476 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.308263182Z level=info msg="Migration successfully executed" id="create plugin_setting table" duration=530.039µs 2024-09-18T07:32:09.477 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.308626656Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-18T07:32:09.477 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.309163346Z level=info msg="Migration successfully executed" id="create index UQE_plugin_setting_org_id_plugin_id - v1" duration=535.284µs 2024-09-18T07:32:09.477 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.309505704Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-18T07:32:09.477 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.311178154Z level=info msg="Migration successfully executed" id="Add column plugin_version to plugin_settings" duration=1.67231ms 2024-09-18T07:32:09.477 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.31153048Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-18T07:32:09.477 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.311550979Z level=info msg="Migration successfully executed" id="Update plugin_setting table charset" duration=20.739µs 2024-09-18T07:32:09.477 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.311963084Z level=info msg="Executing migration" id="create session table" 2024-09-18T07:32:09.477 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.312523412Z level=info msg="Migration successfully executed" id="create session table" duration=560.99µs 2024-09-18T07:32:09.477 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.312891179Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-18T07:32:09.477 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.312969352Z level=info msg="Migration successfully executed" id="Drop old table playlist table" duration=78.342µs 2024-09-18T07:32:09.477 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.315573161Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-18T07:32:09.477 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.315659682Z level=info msg="Migration successfully executed" id="Drop old table playlist_item table" duration=87.161µs 2024-09-18T07:32:09.478 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.316050205Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-18T07:32:09.478 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.316513046Z level=info msg="Migration successfully executed" id="create playlist table v2" duration=462.613µs 2024-09-18T07:32:09.478 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.316875078Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-18T07:32:09.478 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.317366267Z level=info msg="Migration successfully executed" id="create playlist item table v2" duration=489.809µs 2024-09-18T07:32:09.478 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.317756013Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-18T07:32:09.478 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.317776031Z level=info msg="Migration successfully executed" id="Update playlist table charset" duration=20.189µs 2024-09-18T07:32:09.478 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.318175948Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-18T07:32:09.478 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.318195978Z level=info msg="Migration successfully executed" id="Update playlist_item table charset" duration=20.246µs 2024-09-18T07:32:09.478 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.318591048Z level=info msg="Executing migration" id="Add playlist column created_at" 2024-09-18T07:32:09.478 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.320388747Z level=info msg="Migration successfully executed" id="Add playlist column created_at" duration=1.797428ms 2024-09-18T07:32:09.478 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.320752182Z level=info msg="Executing migration" id="Add playlist column updated_at" 2024-09-18T07:32:09.478 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.322562118Z level=info msg="Migration successfully executed" id="Add playlist column updated_at" duration=1.809573ms 2024-09-18T07:32:09.479 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.322921924Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-18T07:32:09.479 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.323001047Z level=info msg="Migration successfully executed" id="drop preferences table v2" duration=78.428µs 2024-09-18T07:32:09.479 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.323382413Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-18T07:32:09.479 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.323446133Z level=info msg="Migration successfully executed" id="drop preferences table v3" duration=63.809µs 2024-09-18T07:32:09.479 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.323807643Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-18T07:32:09.479 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.324333781Z level=info msg="Migration successfully executed" id="create preferences table v3" duration=526.023µs 2024-09-18T07:32:09.479 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.324703963Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-18T07:32:09.479 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.324722806Z level=info msg="Migration successfully executed" id="Update preferences table charset" duration=19.231µs 2024-09-18T07:32:09.479 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.325131342Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-18T07:32:09.479 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.326935152Z level=info msg="Migration successfully executed" id="Add column team_id in preferences" duration=1.803604ms 2024-09-18T07:32:09.479 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.327300115Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-18T07:32:09.479 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.327403975Z level=info msg="Migration successfully executed" id="Update team_id column values in preferences" duration=104.066µs 2024-09-18T07:32:09.479 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.327776589Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-18T07:32:09.480 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.329694627Z level=info msg="Migration successfully executed" id="Add column week_start in preferences" duration=1.917798ms 2024-09-18T07:32:09.480 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.330060124Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-18T07:32:09.480 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.33186088Z level=info msg="Migration successfully executed" id="Add column preferences.json_data" duration=1.799538ms 2024-09-18T07:32:09.480 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.332237867Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-18T07:32:09.480 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.332287871Z level=info msg="Migration successfully executed" id="alter preferences.json_data to mediumtext v1" duration=50.366µs 2024-09-18T07:32:09.480 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.332754886Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-18T07:32:09.480 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.333408295Z level=info msg="Migration successfully executed" id="Add preferences index org_id" duration=654.186µs 2024-09-18T07:32:09.480 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.333790381Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-18T07:32:09.480 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.334388852Z level=info msg="Migration successfully executed" id="Add preferences index user_id" duration=598.21µs 2024-09-18T07:32:09.480 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.336841421Z level=info msg="Executing migration" id="create alert table v1" 2024-09-18T07:32:09.480 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.337756907Z level=info msg="Migration successfully executed" id="create alert table v1" duration=917.694µs 2024-09-18T07:32:09.480 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.338155197Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-18T07:32:09.481 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.338779558Z level=info msg="Migration successfully executed" id="add index alert org_id & id " duration=624.133µs 2024-09-18T07:32:09.481 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.341014918Z level=info msg="Executing migration" id="add index alert state" 2024-09-18T07:32:09.481 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.341694175Z level=info msg="Migration successfully executed" id="add index alert state" duration=681.812µs 2024-09-18T07:32:09.481 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.343486083Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-18T07:32:09.481 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.344067798Z level=info msg="Migration successfully executed" id="add index alert dashboard_id" duration=581.586µs 2024-09-18T07:32:09.481 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.344429371Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-18T07:32:09.481 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.344880167Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v1" duration=450.608µs 2024-09-18T07:32:09.481 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.345248644Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-18T07:32:09.481 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.345828459Z level=info msg="Migration successfully executed" id="Add unique index alert_rule_tag.alert_id_tag_id" duration=579.618µs 2024-09-18T07:32:09.481 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.346202597Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-18T07:32:09.481 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.346763907Z level=info msg="Migration successfully executed" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" duration=561.198µs 2024-09-18T07:32:09.481 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.347101316Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-18T07:32:09.481 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.352650767Z level=info msg="Migration successfully executed" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" duration=5.549263ms 2024-09-18T07:32:09.482 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.353024326Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-18T07:32:09.482 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.353485658Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v2" duration=461.427µs 2024-09-18T07:32:09.482 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.353832645Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-18T07:32:09.482 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.354369302Z level=info msg="Migration successfully executed" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" duration=536.462µs 2024-09-18T07:32:09.482 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.354723913Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-18T07:32:09.482 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.354913369Z level=info msg="Migration successfully executed" id="copy alert_rule_tag v1 to v2" duration=189.325µs 2024-09-18T07:32:09.482 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.355243971Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-18T07:32:09.482 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.355616335Z level=info msg="Migration successfully executed" id="drop table alert_rule_tag_v1" duration=372.312µs 2024-09-18T07:32:09.482 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.355994941Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-18T07:32:09.482 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.356476164Z level=info msg="Migration successfully executed" id="create alert_notification table v1" duration=481.175µs 2024-09-18T07:32:09.482 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.356826741Z level=info msg="Executing migration" id="Add column is_default" 2024-09-18T07:32:09.482 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.358995194Z level=info msg="Migration successfully executed" id="Add column is_default" duration=2.168142ms 2024-09-18T07:32:09.482 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.359328303Z level=info msg="Executing migration" id="Add column frequency" 2024-09-18T07:32:09.483 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.361591791Z level=info msg="Migration successfully executed" id="Add column frequency" duration=2.26318ms 2024-09-18T07:32:09.483 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.361942046Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-18T07:32:09.483 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.364274739Z level=info msg="Migration successfully executed" id="Add column send_reminder" duration=2.332507ms 2024-09-18T07:32:09.483 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.364629977Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-18T07:32:09.483 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.366647181Z level=info msg="Migration successfully executed" id="Add column disable_resolve_message" duration=2.01707ms 2024-09-18T07:32:09.483 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.367011233Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-18T07:32:09.483 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.367581159Z level=info msg="Migration successfully executed" id="add index alert_notification org_id & name" duration=569.648µs 2024-09-18T07:32:09.483 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.367929623Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-18T07:32:09.483 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.367944753Z level=info msg="Migration successfully executed" id="Update alert table charset" duration=15.858µs 2024-09-18T07:32:09.483 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.368383317Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-18T07:32:09.483 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.36839789Z level=info msg="Migration successfully executed" id="Update alert_notification table charset" duration=15.303µs 2024-09-18T07:32:09.483 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.36876211Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-18T07:32:09.483 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.369277621Z level=info msg="Migration successfully executed" id="create notification_journal table v1" duration=515.218µs 2024-09-18T07:32:09.484 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.369639802Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-18T07:32:09.484 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.370188606Z level=info msg="Migration successfully executed" id="add index notification_journal org_id & alert_id & notifier_id" duration=547.523µs 2024-09-18T07:32:09.484 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.370517136Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-18T07:32:09.484 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.371034677Z level=info msg="Migration successfully executed" id="drop alert_notification_journal" duration=517.298µs 2024-09-18T07:32:09.484 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.373141527Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-18T07:32:09.484 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.373656631Z level=info msg="Migration successfully executed" id="create alert_notification_state table v1" duration=514.827µs 2024-09-18T07:32:09.484 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.376012591Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-18T07:32:09.484 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.376558386Z level=info msg="Migration successfully executed" id="add index alert_notification_state org_id & alert_id & notifier_id" duration=544.613µs 2024-09-18T07:32:09.484 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.378711943Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-18T07:32:09.484 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.380860684Z level=info msg="Migration successfully executed" id="Add for to alert table" duration=2.148476ms 2024-09-18T07:32:09.484 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.38136439Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-18T07:32:09.484 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.383522662Z level=info msg="Migration successfully executed" id="Add column uid in alert_notification" duration=2.157907ms 2024-09-18T07:32:09.485 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.384085284Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-18T07:32:09.485 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.384201895Z level=info msg="Migration successfully executed" id="Update uid column values in alert_notification" duration=116.838µs 2024-09-18T07:32:09.485 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.386818026Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-18T07:32:09.485 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.38738381Z level=info msg="Migration successfully executed" id="Add unique index alert_notification_org_id_uid" duration=563.945µs 2024-09-18T07:32:09.485 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.389431362Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-18T07:32:09.485 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.389988294Z level=info msg="Migration successfully executed" id="Remove unique index org_id_name" duration=556.761µs 2024-09-18T07:32:09.485 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.39034422Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-18T07:32:09.485 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.392463993Z level=info msg="Migration successfully executed" id="Add column secure_settings in alert_notification" duration=2.120274ms 2024-09-18T07:32:09.485 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.393145587Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-18T07:32:09.485 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.393186769Z level=info msg="Migration successfully executed" id="alter alert.settings to mediumtext" duration=41.701µs 2024-09-18T07:32:09.485 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.395725182Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-18T07:32:09.485 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.396277284Z level=info msg="Migration successfully executed" id="Add non-unique index alert_notification_state_alert_id" duration=552.746µs 2024-09-18T07:32:09.485 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.398480973Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-18T07:32:09.486 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.399082902Z level=info msg="Migration successfully executed" id="Add non-unique index alert_rule_tag_alert_id" duration=601.729µs 2024-09-18T07:32:09.486 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.399450506Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-18T07:32:09.486 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.399508439Z level=info msg="Migration successfully executed" id="Drop old annotation table v4" duration=58.089µs 2024-09-18T07:32:09.486 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.400313817Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-18T07:32:09.486 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.400891468Z level=info msg="Migration successfully executed" id="create annotation table v5" duration=577.52µs 2024-09-18T07:32:09.486 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.401272876Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-18T07:32:09.486 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.401794624Z level=info msg="Migration successfully executed" id="add index annotation 0 v3" duration=521.621µs 2024-09-18T07:32:09.486 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.402162623Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-18T07:32:09.486 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.402669672Z level=info msg="Migration successfully executed" id="add index annotation 1 v3" duration=506.803µs 2024-09-18T07:32:09.486 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.403036616Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-18T07:32:09.486 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.403735215Z level=info msg="Migration successfully executed" id="add index annotation 2 v3" duration=697.817µs 2024-09-18T07:32:09.486 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.404153346Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-18T07:32:09.487 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.40476547Z level=info msg="Migration successfully executed" id="add index annotation 3 v3" duration=611.772µs 2024-09-18T07:32:09.487 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.405125791Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-18T07:32:09.487 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.405718224Z level=info msg="Migration successfully executed" id="add index annotation 4 v3" duration=593.177µs 2024-09-18T07:32:09.487 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.406073502Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-18T07:32:09.487 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.406088825Z level=info msg="Migration successfully executed" id="Update annotation table charset" duration=16.115µs 2024-09-18T07:32:09.487 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.406475762Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-18T07:32:09.487 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.408909265Z level=info msg="Migration successfully executed" id="Add column region_id to annotation table" duration=2.433072ms 2024-09-18T07:32:09.487 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.409289278Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-18T07:32:09.487 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.409801105Z level=info msg="Migration successfully executed" id="Drop category_id index" duration=511.816µs 2024-09-18T07:32:09.487 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.410144171Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-18T07:32:09.487 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.412403054Z level=info msg="Migration successfully executed" id="Add column tags to annotation table" duration=2.258931ms 2024-09-18T07:32:09.487 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.412773779Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-18T07:32:09.487 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.413226221Z level=info msg="Migration successfully executed" id="Create annotation_tag table v2" duration=452.293µs 2024-09-18T07:32:09.488 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.413579163Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-18T07:32:09.488 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.414157979Z level=info msg="Migration successfully executed" id="Add unique index annotation_tag.annotation_id_tag_id" duration=578.312µs 2024-09-18T07:32:09.488 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.416434283Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-18T07:32:09.488 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.416988872Z level=info msg="Migration successfully executed" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" duration=554.534µs 2024-09-18T07:32:09.488 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.419256725Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-18T07:32:09.488 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.425488481Z level=info msg="Migration successfully executed" id="Rename table annotation_tag to annotation_tag_v2 - v2" duration=6.23127ms 2024-09-18T07:32:09.488 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.425844167Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-18T07:32:09.488 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.426314868Z level=info msg="Migration successfully executed" id="Create annotation_tag table v3" duration=470.522µs 2024-09-18T07:32:09.488 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.426665808Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-18T07:32:09.488 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.427191436Z level=info msg="Migration successfully executed" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" duration=525.259µs 2024-09-18T07:32:09.488 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.429804734Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-18T07:32:09.488 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.430111317Z level=info msg="Migration successfully executed" id="copy annotation_tag v2 to v3" duration=308.71µs 2024-09-18T07:32:09.488 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.432579881Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-18T07:32:09.488 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.433017583Z level=info msg="Migration successfully executed" id="drop table annotation_tag_v2" duration=439.91µs 2024-09-18T07:32:09.489 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.435126349Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-18T07:32:09.489 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.435233506Z level=info msg="Migration successfully executed" id="Update alert annotations and set TEXT to empty" duration=107.374µs 2024-09-18T07:32:09.489 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.435606182Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-18T07:32:09.489 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.437889632Z level=info msg="Migration successfully executed" id="Add created time to annotation table" duration=2.283115ms 2024-09-18T07:32:09.489 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.438340861Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-18T07:32:09.489 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.440725703Z level=info msg="Migration successfully executed" id="Add updated time to annotation table" duration=2.384571ms 2024-09-18T07:32:09.489 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.441128544Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-18T07:32:09.489 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.441680605Z level=info msg="Migration successfully executed" id="Add index for created in annotation table" duration=551.799µs 2024-09-18T07:32:09.489 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.444177758Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-18T07:32:09.489 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.444760823Z level=info msg="Migration successfully executed" id="Add index for updated in annotation table" duration=582.956µs 2024-09-18T07:32:09.489 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.445160887Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-18T07:32:09.489 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.445311034Z level=info msg="Migration successfully executed" id="Convert existing annotations from seconds to milliseconds" duration=149.597µs 2024-09-18T07:32:09.490 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.447992976Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-18T07:32:09.490 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.450406906Z level=info msg="Migration successfully executed" id="Add epoch_end column" duration=2.4145ms 2024-09-18T07:32:09.490 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.450933724Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-18T07:32:09.490 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.451664688Z level=info msg="Migration successfully executed" id="Add index for epoch_end" duration=733.408µs 2024-09-18T07:32:09.490 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.453526796Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-18T07:32:09.490 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.453657426Z level=info msg="Migration successfully executed" id="Make epoch_end the same as epoch" duration=130.51µs 2024-09-18T07:32:09.490 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.454047656Z level=info msg="Executing migration" id="Move region to single row" 2024-09-18T07:32:09.490 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.454272116Z level=info msg="Migration successfully executed" id="Move region to single row" duration=224.501µs 2024-09-18T07:32:09.490 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.456885042Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-18T07:32:09.490 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.457464069Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch from annotation table" duration=579.001µs 2024-09-18T07:32:09.490 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.459463095Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-18T07:32:09.490 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.460027184Z level=info msg="Migration successfully executed" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" duration=562.873µs 2024-09-18T07:32:09.491 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.462335459Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-18T07:32:09.491 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.463160024Z level=info msg="Migration successfully executed" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" duration=825.774µs 2024-09-18T07:32:09.491 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.463626098Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-18T07:32:09.491 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.464428192Z level=info msg="Migration successfully executed" id="Add index for org_id_epoch_end_epoch on annotation table" duration=801.092µs 2024-09-18T07:32:09.491 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.46626661Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-18T07:32:09.491 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.467031903Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch_epoch_end from annotation table" duration=765.304µs 2024-09-18T07:32:09.491 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.468932542Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-18T07:32:09.491 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.469703033Z level=info msg="Migration successfully executed" id="Add index for alert_id on annotation table" duration=770.299µs 2024-09-18T07:32:09.491 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:09.512 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:09.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:09.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:09.527 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:09.614 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:09.635 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:09.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:09.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:09.715 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.471652493Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-18T07:32:09.715 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.47169308Z level=info msg="Migration successfully executed" id="Increase tags column to length 4096" duration=41µs 2024-09-18T07:32:09.716 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.472127062Z level=info msg="Executing migration" id="create test_data table" 2024-09-18T07:32:09.716 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.472651894Z level=info msg="Migration successfully executed" id="create test_data table" duration=524.711µs 2024-09-18T07:32:09.716 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.474807565Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-18T07:32:09.716 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.475323741Z level=info msg="Migration successfully executed" id="create dashboard_version table v1" duration=516.114µs 2024-09-18T07:32:09.716 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.477527917Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-18T07:32:09.716 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.478049391Z level=info msg="Migration successfully executed" id="add index dashboard_version.dashboard_id" duration=521.61µs 2024-09-18T07:32:09.716 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.480370817Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-18T07:32:09.716 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.480933771Z level=info msg="Migration successfully executed" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" duration=563.58µs 2024-09-18T07:32:09.716 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.481291849Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-18T07:32:09.716 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.481403695Z level=info msg="Migration successfully executed" id="Set dashboard version to 1 where 0" duration=110.908µs 2024-09-18T07:32:09.716 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.484055842Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-18T07:32:09.717 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.484363745Z level=info msg="Migration successfully executed" id="save existing dashboard data in dashboard_version table v1" duration=310.354µs 2024-09-18T07:32:09.717 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.486462688Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-18T07:32:09.717 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.486501065Z level=info msg="Migration successfully executed" id="alter dashboard_version.data to mediumtext v1" duration=38.508µs 2024-09-18T07:32:09.717 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.489105283Z level=info msg="Executing migration" id="create team table" 2024-09-18T07:32:09.717 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.489778872Z level=info msg="Migration successfully executed" id="create team table" duration=673.736µs 2024-09-18T07:32:09.717 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.490324403Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-18T07:32:09.717 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.491035658Z level=info msg="Migration successfully executed" id="add index team.org_id" duration=710.972µs 2024-09-18T07:32:09.717 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.493045481Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-18T07:32:09.717 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.493643648Z level=info msg="Migration successfully executed" id="add unique index team_org_id_name" duration=598.01µs 2024-09-18T07:32:09.717 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.495665768Z level=info msg="Executing migration" id="Add column uid in team" 2024-09-18T07:32:09.717 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.498275157Z level=info msg="Migration successfully executed" id="Add column uid in team" duration=2.611103ms 2024-09-18T07:32:09.717 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.498643657Z level=info msg="Executing migration" id="Update uid column values in team" 2024-09-18T07:32:09.718 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.498765697Z level=info msg="Migration successfully executed" id="Update uid column values in team" duration=121.038µs 2024-09-18T07:32:09.718 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.499163563Z level=info msg="Executing migration" id="Add unique index team_org_id_uid" 2024-09-18T07:32:09.718 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.499739732Z level=info msg="Migration successfully executed" id="Add unique index team_org_id_uid" duration=575.876µs 2024-09-18T07:32:09.718 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.50009341Z level=info msg="Executing migration" id="create team member table" 2024-09-18T07:32:09.718 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.500620633Z level=info msg="Migration successfully executed" id="create team member table" duration=526.956µs 2024-09-18T07:32:09.718 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.501004773Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-18T07:32:09.718 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.501556289Z level=info msg="Migration successfully executed" id="add index team_member.org_id" duration=551.501µs 2024-09-18T07:32:09.718 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.501929491Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-18T07:32:09.718 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.502509908Z level=info msg="Migration successfully executed" id="add unique index team_member_org_id_team_id_user_id" duration=580.1µs 2024-09-18T07:32:09.718 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.502885955Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-18T07:32:09.720 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.503480954Z level=info msg="Migration successfully executed" id="add index team_member.team_id" duration=595.17µs 2024-09-18T07:32:09.720 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.503867233Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-18T07:32:09.720 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.506704503Z level=info msg="Migration successfully executed" id="Add column email to team table" duration=2.836925ms 2024-09-18T07:32:09.720 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.507105973Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-18T07:32:09.720 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.509995893Z level=info msg="Migration successfully executed" id="Add column external to team_member table" duration=2.889308ms 2024-09-18T07:32:09.720 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.510375678Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-18T07:32:09.720 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.513030224Z level=info msg="Migration successfully executed" id="Add column permission to team_member table" duration=2.654297ms 2024-09-18T07:32:09.720 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.513541703Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-18T07:32:09.720 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.514284747Z level=info msg="Migration successfully executed" id="create dashboard acl table" duration=742.814µs 2024-09-18T07:32:09.721 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.514681983Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-18T07:32:09.721 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.515302385Z level=info msg="Migration successfully executed" id="add index dashboard_acl_dashboard_id" duration=620.2µs 2024-09-18T07:32:09.721 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.515706422Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-18T07:32:09.721 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.516371936Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_user_id" duration=665.21µs 2024-09-18T07:32:09.721 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.516768594Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-18T07:32:09.721 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.517378212Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_team_id" duration=609.407µs 2024-09-18T07:32:09.721 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.517739524Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-18T07:32:09.721 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.518323721Z level=info msg="Migration successfully executed" id="add index dashboard_acl_user_id" duration=584.038µs 2024-09-18T07:32:09.721 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.518706131Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-18T07:32:09.721 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.519313473Z level=info msg="Migration successfully executed" id="add index dashboard_acl_team_id" duration=607.162µs 2024-09-18T07:32:09.721 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.519683633Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-18T07:32:09.722 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.520286703Z level=info msg="Migration successfully executed" id="add index dashboard_acl_org_id_role" duration=604.388µs 2024-09-18T07:32:09.722 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.520657244Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-18T07:32:09.722 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.521286148Z level=info msg="Migration successfully executed" id="add index dashboard_permission" duration=628.673µs 2024-09-18T07:32:09.722 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.521664402Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-18T07:32:09.722 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.522035703Z level=info msg="Migration successfully executed" id="save default acl rules in dashboard_acl table" duration=371.108µs 2024-09-18T07:32:09.722 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.522519007Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-18T07:32:09.722 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.522681065Z level=info msg="Migration successfully executed" id="delete acl rules for deleted dashboards and folders" duration=163.007µs 2024-09-18T07:32:09.722 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.52335867Z level=info msg="Executing migration" id="create tag table" 2024-09-18T07:32:09.722 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.523881436Z level=info msg="Migration successfully executed" id="create tag table" duration=522.383µs 2024-09-18T07:32:09.722 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.524262416Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-18T07:32:09.722 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.524868123Z level=info msg="Migration successfully executed" id="add index tag.key_value" duration=605.517µs 2024-09-18T07:32:09.722 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.525253133Z level=info msg="Executing migration" id="create login attempt table" 2024-09-18T07:32:09.722 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.525726321Z level=info msg="Migration successfully executed" id="create login attempt table" duration=472.945µs 2024-09-18T07:32:09.722 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.526103232Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-18T07:32:09.723 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.526673628Z level=info msg="Migration successfully executed" id="add index login_attempt.username" duration=570.139µs 2024-09-18T07:32:09.723 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.527045512Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-18T07:32:09.723 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.527624357Z level=info msg="Migration successfully executed" id="drop index IDX_login_attempt_username - v1" duration=578.414µs 2024-09-18T07:32:09.723 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.527992728Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-18T07:32:09.723 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.536112527Z level=info msg="Migration successfully executed" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" duration=8.120573ms 2024-09-18T07:32:09.723 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.536525175Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-18T07:32:09.723 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.537034133Z level=info msg="Migration successfully executed" id="create login_attempt v2" duration=508.952µs 2024-09-18T07:32:09.723 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.537401521Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-18T07:32:09.723 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.538014189Z level=info msg="Migration successfully executed" id="create index IDX_login_attempt_username - v2" duration=612.327µs 2024-09-18T07:32:09.723 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.538385722Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-18T07:32:09.723 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.538636614Z level=info msg="Migration successfully executed" id="copy login_attempt v1 to v2" duration=251.822µs 2024-09-18T07:32:09.724 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.53923661Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-18T07:32:09.724 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.539689408Z level=info msg="Migration successfully executed" id="drop login_attempt_tmp_qwerty" duration=452.657µs 2024-09-18T07:32:09.724 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.540120738Z level=info msg="Executing migration" id="create user auth table" 2024-09-18T07:32:09.724 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.540612098Z level=info msg="Migration successfully executed" id="create user auth table" duration=491.27µs 2024-09-18T07:32:09.724 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.540996329Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-18T07:32:09.724 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.541592488Z level=info msg="Migration successfully executed" id="create index IDX_user_auth_auth_module_auth_id - v1" duration=595.871µs 2024-09-18T07:32:09.724 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.541981805Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-18T07:32:09.724 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.542027074Z level=info msg="Migration successfully executed" id="alter user_auth.auth_id to length 190" duration=44.787µs 2024-09-18T07:32:09.724 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.542878176Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-18T07:32:09.724 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.545905882Z level=info msg="Migration successfully executed" id="Add OAuth access token to user_auth" duration=3.027772ms 2024-09-18T07:32:09.724 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.546277881Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-18T07:32:09.724 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.549164073Z level=info msg="Migration successfully executed" id="Add OAuth refresh token to user_auth" duration=2.885868ms 2024-09-18T07:32:09.724 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.549502468Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-18T07:32:09.725 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.552411629Z level=info msg="Migration successfully executed" id="Add OAuth token type to user_auth" duration=2.909101ms 2024-09-18T07:32:09.725 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.552773555Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-18T07:32:09.725 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.555701092Z level=info msg="Migration successfully executed" id="Add OAuth expiry to user_auth" duration=2.927264ms 2024-09-18T07:32:09.725 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.556060012Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-18T07:32:09.725 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.556643502Z level=info msg="Migration successfully executed" id="Add index to user_id column in user_auth" duration=583.407µs 2024-09-18T07:32:09.725 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.558831326Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-18T07:32:09.725 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.561779456Z level=info msg="Migration successfully executed" id="Add OAuth ID token to user_auth" duration=2.947709ms 2024-09-18T07:32:09.725 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.562140938Z level=info msg="Executing migration" id="create server_lock table" 2024-09-18T07:32:09.725 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.562651645Z level=info msg="Migration successfully executed" id="create server_lock table" duration=510.575µs 2024-09-18T07:32:09.725 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.565072055Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-18T07:32:09.725 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.565682154Z level=info msg="Migration successfully executed" id="add index server_lock.operation_uid" duration=609.928µs 2024-09-18T07:32:09.725 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.567792964Z level=info msg="Executing migration" id="create user auth token table" 2024-09-18T07:32:09.725 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.568383579Z level=info msg="Migration successfully executed" id="create user auth token table" duration=588.963µs 2024-09-18T07:32:09.726 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.570672299Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-18T07:32:09.726 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.571300059Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.auth_token" duration=630.096µs 2024-09-18T07:32:09.726 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.573153655Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-18T07:32:09.726 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.573765658Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.prev_auth_token" duration=611.711µs 2024-09-18T07:32:09.726 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.576056537Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-18T07:32:09.726 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.576750544Z level=info msg="Migration successfully executed" id="add index user_auth_token.user_id" duration=693.871µs 2024-09-18T07:32:09.726 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.578748191Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-18T07:32:09.726 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.581902646Z level=info msg="Migration successfully executed" id="Add revoked_at to the user auth token" duration=3.154884ms 2024-09-18T07:32:09.726 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.582264453Z level=info msg="Executing migration" id="add index user_auth_token.revoked_at" 2024-09-18T07:32:09.726 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.582863014Z level=info msg="Migration successfully executed" id="add index user_auth_token.revoked_at" duration=598.329µs 2024-09-18T07:32:09.726 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.585112377Z level=info msg="Executing migration" id="create cache_data table" 2024-09-18T07:32:09.726 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.585665298Z level=info msg="Migration successfully executed" id="create cache_data table" duration=552.655µs 2024-09-18T07:32:09.727 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.588053668Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-18T07:32:09.727 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.588709425Z level=info msg="Migration successfully executed" id="add unique index cache_data.cache_key" duration=657.748µs 2024-09-18T07:32:09.727 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.59060374Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-18T07:32:09.727 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.591278663Z level=info msg="Migration successfully executed" id="create short_url table v1" duration=674.768µs 2024-09-18T07:32:09.727 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.593611616Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-18T07:32:09.727 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.594314589Z level=info msg="Migration successfully executed" id="add index short_url.org_id-uid" duration=705.278µs 2024-09-18T07:32:09.727 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.596319598Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-18T07:32:09.727 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.596418891Z level=info msg="Migration successfully executed" id="alter table short_url alter column created_by type to bigint" duration=102.1µs 2024-09-18T07:32:09.727 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.599091958Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-18T07:32:09.727 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.599150136Z level=info msg="Migration successfully executed" id="delete alert_definition table" duration=58.425µs 2024-09-18T07:32:09.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.601980484Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-18T07:32:09.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.602530456Z level=info msg="Migration successfully executed" id="recreate alert_definition table" duration=549.863µs 2024-09-18T07:32:09.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.604719068Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-18T07:32:09.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.605361806Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and title columns" duration=648.17µs 2024-09-18T07:32:09.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.607301179Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-18T07:32:09.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.607970215Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and uid columns" duration=670.9µs 2024-09-18T07:32:09.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.61019816Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-18T07:32:09.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.610238816Z level=info msg="Migration successfully executed" id="alter alert_definition table data column to mediumtext in mysql" duration=40.998µs 2024-09-18T07:32:09.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.612881605Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-18T07:32:09.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.613500274Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and title columns" duration=618.927µs 2024-09-18T07:32:09.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.61556927Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-18T07:32:09.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.616191141Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and uid columns" duration=621.796µs 2024-09-18T07:32:09.728 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.618278316Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-18T07:32:09.729 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.618913854Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and title columns" duration=635.231µs 2024-09-18T07:32:09.729 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.620992753Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-18T07:32:09.729 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.621665544Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and uid columns" duration=672.839µs 2024-09-18T07:32:09.729 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.623620123Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-18T07:32:09.729 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.627069846Z level=info msg="Migration successfully executed" id="Add column paused in alert_definition" duration=3.45006ms 2024-09-18T07:32:09.729 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.6274462Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-18T07:32:09.729 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.628140673Z level=info msg="Migration successfully executed" id="drop alert_definition table" duration=693.582µs 2024-09-18T07:32:09.729 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.630204573Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-18T07:32:09.729 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.630278949Z level=info msg="Migration successfully executed" id="delete alert_definition_version table" duration=73.656µs 2024-09-18T07:32:09.729 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.632916309Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-18T07:32:09.729 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.633592767Z level=info msg="Migration successfully executed" id="recreate alert_definition_version table" duration=676.416µs 2024-09-18T07:32:09.729 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.635765461Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-18T07:32:09.729 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.636442433Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_id and version columns" duration=675.651µs 2024-09-18T07:32:09.730 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.638411907Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-18T07:32:09.730 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.639071104Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_uid and version columns" duration=659.743µs 2024-09-18T07:32:09.730 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.641215439Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-18T07:32:09.730 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.641262392Z level=info msg="Migration successfully executed" id="alter alert_definition_version table data column to mediumtext in mysql" duration=46.483µs 2024-09-18T07:32:09.730 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.643630398Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-18T07:32:09.730 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.64441641Z level=info msg="Migration successfully executed" id="drop alert_definition_version table" duration=785.692µs 2024-09-18T07:32:09.730 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.646556455Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-18T07:32:09.730 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.64728072Z level=info msg="Migration successfully executed" id="create alert_instance table" duration=726.605µs 2024-09-18T07:32:09.730 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.649029365Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-18T07:32:09.730 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.649663414Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" duration=633.832µs 2024-09-18T07:32:09.730 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.650053839Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-18T07:32:09.730 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.650680249Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, current_state columns" duration=626.199µs 2024-09-18T07:32:09.730 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.651056317Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-18T07:32:09.731 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.654496072Z level=info msg="Migration successfully executed" id="add column current_state_end to alert_instance" duration=3.439511ms 2024-09-18T07:32:09.731 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.654878167Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-18T07:32:09.731 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.655476562Z level=info msg="Migration successfully executed" id="remove index def_org_id, def_uid, current_state on alert_instance" duration=598.216µs 2024-09-18T07:32:09.731 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.655840883Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-18T07:32:09.731 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.656436411Z level=info msg="Migration successfully executed" id="remove index def_org_id, current_state on alert_instance" duration=595.295µs 2024-09-18T07:32:09.731 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.656801771Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-18T07:32:09.731 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.674438342Z level=info msg="Migration successfully executed" id="rename def_org_id to rule_org_id in alert_instance" duration=17.637432ms 2024-09-18T07:32:09.731 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.674887213Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-18T07:32:09.731 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.691199445Z level=info msg="Migration successfully executed" id="rename def_uid to rule_uid in alert_instance" duration=16.312003ms 2024-09-18T07:32:09.731 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.691580076Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-18T07:32:09.731 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.692202568Z level=info msg="Migration successfully executed" id="add index rule_org_id, rule_uid, current_state on alert_instance" duration=622.118µs 2024-09-18T07:32:09.731 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.692591935Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-18T07:32:09.732 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.693204744Z level=info msg="Migration successfully executed" id="add index rule_org_id, current_state on alert_instance" duration=612.713µs 2024-09-18T07:32:09.732 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.693587887Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-18T07:32:09.732 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.696882973Z level=info msg="Migration successfully executed" id="add current_reason column related to current_state" duration=3.294527ms 2024-09-18T07:32:09.732 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.69726099Z level=info msg="Executing migration" id="add result_fingerprint column to alert_instance" 2024-09-18T07:32:09.732 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.700527221Z level=info msg="Migration successfully executed" id="add result_fingerprint column to alert_instance" duration=3.265785ms 2024-09-18T07:32:09.732 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.700926929Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-18T07:32:09.732 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.701576887Z level=info msg="Migration successfully executed" id="create alert_rule table" duration=650.487µs 2024-09-18T07:32:09.732 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.701976265Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-18T07:32:09.732 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.70260974Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and title columns" duration=633.328µs 2024-09-18T07:32:09.732 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.703005488Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-18T07:32:09.732 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.703636422Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and uid columns" duration=630.801µs 2024-09-18T07:32:09.732 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.704031227Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-18T07:32:09.732 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.704718521Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" duration=687.087µs 2024-09-18T07:32:09.733 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.705122031Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-18T07:32:09.733 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.705167846Z level=info msg="Migration successfully executed" id="alter alert_rule table data column to mediumtext in mysql" duration=46.637µs 2024-09-18T07:32:09.733 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.705984129Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-18T07:32:09.733 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.709479585Z level=info msg="Migration successfully executed" id="add column for to alert_rule" duration=3.494977ms 2024-09-18T07:32:09.733 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.709866038Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-18T07:32:09.733 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.713268111Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule" duration=3.401791ms 2024-09-18T07:32:09.733 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.713678608Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-18T07:32:09.733 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.717428875Z level=info msg="Migration successfully executed" id="add column labels to alert_rule" duration=3.748249ms 2024-09-18T07:32:09.737 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:09.758 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:09.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:09.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:09.860 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:09.883 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:09.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:09.968 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.717821699Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-18T07:32:09.968 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.718448913Z level=info msg="Migration successfully executed" id="remove unique index from alert_rule on org_id, title columns" duration=627.019µs 2024-09-18T07:32:09.968 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.718809486Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-18T07:32:09.968 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.719452865Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespase_uid and title columns" duration=641.884µs 2024-09-18T07:32:09.968 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.719815526Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-18T07:32:09.968 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.723235252Z level=info msg="Migration successfully executed" id="add dashboard_uid column to alert_rule" duration=3.42028ms 2024-09-18T07:32:09.968 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.723607758Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-18T07:32:09.968 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.726985572Z level=info msg="Migration successfully executed" id="add panel_id column to alert_rule" duration=3.377386ms 2024-09-18T07:32:09.969 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.72735377Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-18T07:32:09.969 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.727984488Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" duration=630.356µs 2024-09-18T07:32:09.969 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.731381898Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-18T07:32:09.969 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.734879559Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule" duration=3.497104ms 2024-09-18T07:32:09.969 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.735297045Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-18T07:32:09.969 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.739672149Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule table" duration=4.372251ms 2024-09-18T07:32:09.969 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.741460483Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-18T07:32:09.969 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.741518833Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule table" duration=58.68µs 2024-09-18T07:32:09.969 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.744336504Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-18T07:32:09.970 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.745366551Z level=info msg="Migration successfully executed" id="create alert_rule_version table" duration=1.031564ms 2024-09-18T07:32:09.970 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.745807414Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-18T07:32:09.970 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.74657117Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" duration=764.02µs 2024-09-18T07:32:09.970 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.747104247Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-18T07:32:09.970 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.747909028Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" duration=804.587µs 2024-09-18T07:32:09.971 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.750127777Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-18T07:32:09.971 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.750202296Z level=info msg="Migration successfully executed" id="alter alert_rule_version table data column to mediumtext in mysql" duration=75.829µs 2024-09-18T07:32:09.971 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.756172797Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-18T07:32:09.971 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.760279589Z level=info msg="Migration successfully executed" id="add column for to alert_rule_version" duration=4.106915ms 2024-09-18T07:32:09.971 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.760686406Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-18T07:32:09.971 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.76433882Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule_version" duration=3.652187ms 2024-09-18T07:32:09.971 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.764719828Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-18T07:32:09.971 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.768269579Z level=info msg="Migration successfully executed" id="add column labels to alert_rule_version" duration=3.549412ms 2024-09-18T07:32:09.971 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.768645685Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-18T07:32:09.971 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.772170021Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule_version" duration=3.525635ms 2024-09-18T07:32:09.971 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.772532853Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-18T07:32:09.972 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.776046513Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule_versions table" duration=3.512682ms 2024-09-18T07:32:09.972 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.776402544Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-18T07:32:09.972 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.776451554Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule_version table" duration=49.214µs 2024-09-18T07:32:09.972 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.777250313Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-18T07:32:09.972 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.777753109Z level=info msg="Migration successfully executed" id=create_alert_configuration_table duration=502.929µs 2024-09-18T07:32:09.972 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.77812341Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-18T07:32:09.972 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.781746855Z level=info msg="Migration successfully executed" id="Add column default in alert_configuration" duration=3.623011ms 2024-09-18T07:32:09.972 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.782132678Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-18T07:32:09.972 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.782181327Z level=info msg="Migration successfully executed" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" duration=48.99µs 2024-09-18T07:32:09.972 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.783154442Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-18T07:32:09.973 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.786912682Z level=info msg="Migration successfully executed" id="add column org_id in alert_configuration" duration=3.760242ms 2024-09-18T07:32:09.973 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.78730086Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-18T07:32:09.973 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.787917645Z level=info msg="Migration successfully executed" id="add index in alert_configuration table on org_id column" duration=616.515µs 2024-09-18T07:32:09.973 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.788291346Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-18T07:32:09.973 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.791915198Z level=info msg="Migration successfully executed" id="add configuration_hash column to alert_configuration" duration=3.62335ms 2024-09-18T07:32:09.973 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.792296253Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-18T07:32:09.973 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.792795393Z level=info msg="Migration successfully executed" id=create_ngalert_configuration_table duration=498.908µs 2024-09-18T07:32:09.973 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.795449809Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-18T07:32:09.973 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.79624159Z level=info msg="Migration successfully executed" id="add index in ngalert_configuration on org_id column" duration=793.967µs 2024-09-18T07:32:09.973 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.798267965Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-18T07:32:09.974 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.801989354Z level=info msg="Migration successfully executed" id="add column send_alerts_to in ngalert_configuration" duration=3.723444ms 2024-09-18T07:32:09.974 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.802354758Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-18T07:32:09.974 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.802853825Z level=info msg="Migration successfully executed" id="create provenance_type table" duration=499.038µs 2024-09-18T07:32:09.974 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.803249123Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-18T07:32:09.974 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.803865442Z level=info msg="Migration successfully executed" id="add index to uniquify (record_key, record_type, org_id) columns" duration=616.048µs 2024-09-18T07:32:09.974 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.806077554Z level=info msg="Executing migration" id="create alert_image table" 2024-09-18T07:32:09.974 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.806599605Z level=info msg="Migration successfully executed" id="create alert_image table" duration=520.822µs 2024-09-18T07:32:09.974 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.809114783Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-18T07:32:09.974 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.809807746Z level=info msg="Migration successfully executed" id="add unique index on token to alert_image table" duration=696.399µs 2024-09-18T07:32:09.974 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.811704552Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-18T07:32:09.974 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.811771509Z level=info msg="Migration successfully executed" id="support longer URLs in alert_image table" duration=67.001µs 2024-09-18T07:32:09.974 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.812195703Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-18T07:32:09.974 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.812850857Z level=info msg="Migration successfully executed" id=create_alert_configuration_history_table duration=655.08µs 2024-09-18T07:32:09.974 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.815167039Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-18T07:32:09.975 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.815765961Z level=info msg="Migration successfully executed" id="drop non-unique orgID index on alert_configuration" duration=598.698µs 2024-09-18T07:32:09.975 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.818077696Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-18T07:32:09.976 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.818317967Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-18T07:32:09.976 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.8206308Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-18T07:32:09.976 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.820920794Z level=info msg="Migration successfully executed" id="extract alertmanager configuration history to separate table" duration=289.789µs 2024-09-18T07:32:09.976 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.821275633Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-18T07:32:09.976 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.8218851Z level=info msg="Migration successfully executed" id="add unique index on orgID to alert_configuration" duration=609.266µs 2024-09-18T07:32:09.976 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.824088174Z level=info msg="Executing migration" id="add last_applied column to alert_configuration_history" 2024-09-18T07:32:09.976 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.827941214Z level=info msg="Migration successfully executed" id="add last_applied column to alert_configuration_history" duration=3.852779ms 2024-09-18T07:32:09.976 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.828323114Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-18T07:32:09.976 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.829029201Z level=info msg="Migration successfully executed" id="create library_element table v1" duration=705.346µs 2024-09-18T07:32:09.976 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.83109092Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-18T07:32:09.976 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.832140315Z level=info msg="Migration successfully executed" id="add index library_element org_id-folder_id-name-kind" duration=1.048772ms 2024-09-18T07:32:09.976 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.833164743Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-18T07:32:09.977 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.834069107Z level=info msg="Migration successfully executed" id="create library_element_connection table v1" duration=904.297µs 2024-09-18T07:32:09.977 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.83454823Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-18T07:32:09.977 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.835303589Z level=info msg="Migration successfully executed" id="add index library_element_connection element_id-kind-connection_id" duration=755.739µs 2024-09-18T07:32:09.977 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.837292365Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-18T07:32:09.977 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.83792853Z level=info msg="Migration successfully executed" id="add unique index library_element org_id_uid" duration=637.485µs 2024-09-18T07:32:09.977 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.840056906Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-18T07:32:09.977 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.840101334Z level=info msg="Migration successfully executed" id="increase max description length to 2048" duration=42.889µs 2024-09-18T07:32:09.977 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.840695714Z level=info msg="Executing migration" id="alter library_element model to mediumtext" 2024-09-18T07:32:09.977 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.840824608Z level=info msg="Migration successfully executed" id="alter library_element model to mediumtext" duration=127.577µs 2024-09-18T07:32:09.978 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.843443109Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-18T07:32:09.978 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.843787514Z level=info msg="Migration successfully executed" id="clone move dashboard alerts to unified alerting" duration=346.795µs 2024-09-18T07:32:09.978 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.846139889Z level=info msg="Executing migration" id="create data_keys table" 2024-09-18T07:32:09.978 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.846972648Z level=info msg="Migration successfully executed" id="create data_keys table" duration=834.885µs 2024-09-18T07:32:09.978 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.847381213Z level=info msg="Executing migration" id="create secrets table" 2024-09-18T07:32:09.978 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.84795733Z level=info msg="Migration successfully executed" id="create secrets table" duration=563.394µs 2024-09-18T07:32:09.979 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.84868673Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-18T07:32:09.979 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.869930437Z level=info msg="Migration successfully executed" id="rename data_keys name column to id" duration=21.241536ms 2024-09-18T07:32:09.979 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.870377168Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-18T07:32:09.979 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.874532477Z level=info msg="Migration successfully executed" id="add name column into data_keys" duration=4.154562ms 2024-09-18T07:32:09.979 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.874885149Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-18T07:32:09.979 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.875000022Z level=info msg="Migration successfully executed" id="copy data_keys id column values into name" duration=115.576µs 2024-09-18T07:32:09.979 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.877657611Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-18T07:32:09.980 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.899524281Z level=info msg="Migration successfully executed" id="rename data_keys name column to label" duration=21.864184ms 2024-09-18T07:32:09.980 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.899927179Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-18T07:32:09.980 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.921648931Z level=info msg="Migration successfully executed" id="rename data_keys id column back to name" duration=21.722011ms 2024-09-18T07:32:09.980 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.922053315Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-18T07:32:09.980 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.922647704Z level=info msg="Migration successfully executed" id="create kv_store table v1" duration=594.205µs 2024-09-18T07:32:09.981 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.924733041Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-18T07:32:09.981 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.925457784Z level=info msg="Migration successfully executed" id="add index kv_store.org_id-namespace-key" duration=724.434µs 2024-09-18T07:32:09.981 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.927512744Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-18T07:32:09.981 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.927745669Z level=info msg="Migration successfully executed" id="update dashboard_uid and panel_id from existing annotations" duration=235.542µs 2024-09-18T07:32:09.981 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.928145176Z level=info msg="Executing migration" id="create permission table" 2024-09-18T07:32:09.981 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.9287355Z level=info msg="Migration successfully executed" id="create permission table" duration=590.278µs 2024-09-18T07:32:09.982 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.930938469Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-18T07:32:09.982 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.931579274Z level=info msg="Migration successfully executed" id="add unique index permission.role_id" duration=640.748µs 2024-09-18T07:32:09.982 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.933740429Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-18T07:32:09.982 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.934410124Z level=info msg="Migration successfully executed" id="add unique index role_id_action_scope" duration=669.416µs 2024-09-18T07:32:09.982 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.936463391Z level=info msg="Executing migration" id="create role table" 2024-09-18T07:32:09.982 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.93719406Z level=info msg="Migration successfully executed" id="create role table" duration=732.755µs 2024-09-18T07:32:09.983 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.937569514Z level=info msg="Executing migration" id="add column display_name" 2024-09-18T07:32:09.983 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.941845279Z level=info msg="Migration successfully executed" id="add column display_name" duration=4.275551ms 2024-09-18T07:32:09.983 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.942193515Z level=info msg="Executing migration" id="add column group_name" 2024-09-18T07:32:09.983 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.946448463Z level=info msg="Migration successfully executed" id="add column group_name" duration=4.254886ms 2024-09-18T07:32:09.983 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.946832207Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-18T07:32:09.983 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.947501197Z level=info msg="Migration successfully executed" id="add index role.org_id" duration=668.981µs 2024-09-18T07:32:09.983 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.94786897Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-18T07:32:09.983 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.948579662Z level=info msg="Migration successfully executed" id="add unique index role_org_id_name" duration=709.991µs 2024-09-18T07:32:09.983 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.94894269Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-18T07:32:09.983 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.94962704Z level=info msg="Migration successfully executed" id="add index role_org_id_uid" duration=684.122µs 2024-09-18T07:32:09.984 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.949987595Z level=info msg="Executing migration" id="create team role table" 2024-09-18T07:32:09.984 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.950550499Z level=info msg="Migration successfully executed" id="create team role table" duration=562.929µs 2024-09-18T07:32:09.984 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.950917946Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-18T07:32:09.984 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.951608373Z level=info msg="Migration successfully executed" id="add index team_role.org_id" duration=690.169µs 2024-09-18T07:32:09.984 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.951976348Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-18T07:32:09.984 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.952680195Z level=info msg="Migration successfully executed" id="add unique index team_role_org_id_team_id_role_id" duration=702.302µs 2024-09-18T07:32:09.984 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.953058856Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-18T07:32:09.984 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.953688171Z level=info msg="Migration successfully executed" id="add index team_role.team_id" duration=629.215µs 2024-09-18T07:32:09.984 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.954043652Z level=info msg="Executing migration" id="create user role table" 2024-09-18T07:32:09.984 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.954575399Z level=info msg="Migration successfully executed" id="create user role table" duration=530.611µs 2024-09-18T07:32:09.984 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.954950098Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-18T07:32:09.984 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.955589437Z level=info msg="Migration successfully executed" id="add index user_role.org_id" duration=643.369µs 2024-09-18T07:32:09.985 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.955934503Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-18T07:32:09.985 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.956588199Z level=info msg="Migration successfully executed" id="add unique index user_role_org_id_user_id_role_id" duration=653.352µs 2024-09-18T07:32:09.985 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.956932826Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-18T07:32:09.985 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.957567527Z level=info msg="Migration successfully executed" id="add index user_role.user_id" duration=634.476µs 2024-09-18T07:32:09.985 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.957921986Z level=info msg="Executing migration" id="create builtin role table" 2024-09-18T07:32:09.985 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.958476485Z level=info msg="Migration successfully executed" id="create builtin role table" duration=554.459µs 2024-09-18T07:32:09.985 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.958822325Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-18T07:32:09.985 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.95947984Z level=info msg="Migration successfully executed" id="add index builtin_role.role_id" duration=657.367µs 2024-09-18T07:32:09.985 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.959826015Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-18T07:32:09.985 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.960476459Z level=info msg="Migration successfully executed" id="add index builtin_role.name" duration=650.241µs 2024-09-18T07:32:09.985 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.960838388Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-18T07:32:09.985 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.965653833Z level=info msg="Migration successfully executed" id="Add column org_id to builtin_role table" duration=4.815157ms 2024-09-18T07:32:09.986 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.966040966Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-18T07:32:09.986 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.966697291Z level=info msg="Migration successfully executed" id="add index builtin_role.org_id" duration=656.176µs 2024-09-18T07:32:09.986 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.967079654Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-18T07:32:09.986 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.967771047Z level=info msg="Migration successfully executed" id="add unique index builtin_role_org_id_role_id_role" duration=691.079µs 2024-09-18T07:32:09.986 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.9682384Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-18T07:32:09.986 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.968907181Z level=info msg="Migration successfully executed" id="Remove unique index role_org_id_uid" duration=669.266µs 2024-09-18T07:32:09.986 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.969272802Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-18T07:32:09.986 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.969935167Z level=info msg="Migration successfully executed" id="add unique index role.uid" duration=662.262µs 2024-09-18T07:32:09.986 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.970315374Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-18T07:32:09.986 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.970820606Z level=info msg="Migration successfully executed" id="create seed assignment table" duration=505.02µs 2024-09-18T07:32:09.986 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.971201866Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-18T07:32:09.986 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.971907285Z level=info msg="Migration successfully executed" id="add unique index builtin_role_role_name" duration=704.646µs 2024-09-18T07:32:09.986 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.972284136Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-18T07:32:09.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:10.009 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:10.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:10.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:10.045 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:10.111 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:10.132 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:10.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:10.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:10.228 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-mon[32237]: osdmap e83: 8 total, 8 up, 8 in 2024-09-18T07:32:10.228 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:10.228 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:10.228 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:10.228 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:10.228 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-mon[32237]: Deploying daemon node-exporter.a on smithi137 2024-09-18T07:32:10.228 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-mon[32237]: pgmap v35: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:32:10.228 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-mon[32237]: osdmap e84: 8 total, 8 up, 8 in 2024-09-18T07:32:10.229 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.976975626Z level=info msg="Migration successfully executed" id="add column hidden to role table" duration=4.69078ms 2024-09-18T07:32:10.229 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.97733782Z level=info msg="Executing migration" id="permission kind migration" 2024-09-18T07:32:10.229 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.981940193Z level=info msg="Migration successfully executed" id="permission kind migration" duration=4.602042ms 2024-09-18T07:32:10.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.982327407Z level=info msg="Executing migration" id="permission attribute migration" 2024-09-18T07:32:10.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.98684031Z level=info msg="Migration successfully executed" id="permission attribute migration" duration=4.512644ms 2024-09-18T07:32:10.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.987224294Z level=info msg="Executing migration" id="permission identifier migration" 2024-09-18T07:32:10.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.991765423Z level=info msg="Migration successfully executed" id="permission identifier migration" duration=4.540786ms 2024-09-18T07:32:10.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.992152324Z level=info msg="Executing migration" id="add permission identifier index" 2024-09-18T07:32:10.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.992889376Z level=info msg="Migration successfully executed" id="add permission identifier index" duration=736.615µs 2024-09-18T07:32:10.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.993268538Z level=info msg="Executing migration" id="add permission action scope role_id index" 2024-09-18T07:32:10.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.994010177Z level=info msg="Migration successfully executed" id="add permission action scope role_id index" duration=741.197µs 2024-09-18T07:32:10.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.994385249Z level=info msg="Executing migration" id="remove permission role_id action scope index" 2024-09-18T07:32:10.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.995062888Z level=info msg="Migration successfully executed" id="remove permission role_id action scope index" duration=677.371µs 2024-09-18T07:32:10.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.995429227Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-18T07:32:10.230 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.99603502Z level=info msg="Migration successfully executed" id="create query_history table v1" duration=606.242µs 2024-09-18T07:32:10.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.996400955Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-18T07:32:10.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.997107263Z level=info msg="Migration successfully executed" id="add index query_history.org_id-created_by-datasource_uid" duration=705.471µs 2024-09-18T07:32:10.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.997471947Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-18T07:32:10.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.997519009Z level=info msg="Migration successfully executed" id="alter table query_history alter column created_by type to bigint" duration=46.671µs 2024-09-18T07:32:10.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.998005442Z level=info msg="Executing migration" id="rbac disabled migrator" 2024-09-18T07:32:10.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.998028027Z level=info msg="Migration successfully executed" id="rbac disabled migrator" duration=23.58µs 2024-09-18T07:32:10.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.998421077Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-18T07:32:10.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.998676498Z level=info msg="Migration successfully executed" id="teams permissions migration" duration=256.438µs 2024-09-18T07:32:10.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.999024633Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-18T07:32:10.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.999355706Z level=info msg="Migration successfully executed" id="dashboard permissions" duration=331.621µs 2024-09-18T07:32:10.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:09.999741143Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-18T07:32:10.231 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.00017217Z level=info msg="Migration successfully executed" id="dashboard permissions uid scopes" duration=431.113µs 2024-09-18T07:32:10.232 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.000666141Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-18T07:32:10.232 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.000824366Z level=info msg="Migration successfully executed" id="drop managed folder create actions" duration=158.973µs 2024-09-18T07:32:10.232 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.001230771Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-18T07:32:10.232 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.001546623Z level=info msg="Migration successfully executed" id="alerting notification permissions" duration=315.904µs 2024-09-18T07:32:10.232 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.001935484Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-18T07:32:10.232 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.002466438Z level=info msg="Migration successfully executed" id="create query_history_star table v1" duration=531.268µs 2024-09-18T07:32:10.232 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.002834363Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-18T07:32:10.232 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.003507121Z level=info msg="Migration successfully executed" id="add index query_history.user_id-query_uid" duration=672.574µs 2024-09-18T07:32:10.232 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.003876446Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-18T07:32:10.232 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.008484914Z level=info msg="Migration successfully executed" id="add column org_id in query_history_star" duration=4.608141ms 2024-09-18T07:32:10.232 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.008860876Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-18T07:32:10.232 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.008902677Z level=info msg="Migration successfully executed" id="alter table query_history_star_mig column user_id type to bigint" duration=41.053µs 2024-09-18T07:32:10.232 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.009360872Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-18T07:32:10.233 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.010064196Z level=info msg="Migration successfully executed" id="create correlation table v1" duration=703.032µs 2024-09-18T07:32:10.233 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.010469839Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-18T07:32:10.233 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.011156113Z level=info msg="Migration successfully executed" id="add index correlations.uid" duration=686.292µs 2024-09-18T07:32:10.233 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.011496865Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-18T07:32:10.233 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.01215388Z level=info msg="Migration successfully executed" id="add index correlations.source_uid" duration=656.6µs 2024-09-18T07:32:10.233 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.012486722Z level=info msg="Executing migration" id="add correlation config column" 2024-09-18T07:32:10.233 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.017330375Z level=info msg="Migration successfully executed" id="add correlation config column" duration=4.843293ms 2024-09-18T07:32:10.233 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.017682385Z level=info msg="Executing migration" id="drop index IDX_correlation_uid - v1" 2024-09-18T07:32:10.233 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.018333572Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_uid - v1" duration=650.976µs 2024-09-18T07:32:10.233 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.018685925Z level=info msg="Executing migration" id="drop index IDX_correlation_source_uid - v1" 2024-09-18T07:32:10.233 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.019344224Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_source_uid - v1" duration=658.16µs 2024-09-18T07:32:10.233 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.019704156Z level=info msg="Executing migration" id="Rename table correlation to correlation_tmp_qwerty - v1" 2024-09-18T07:32:10.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.033124829Z level=info msg="Migration successfully executed" id="Rename table correlation to correlation_tmp_qwerty - v1" duration=13.419169ms 2024-09-18T07:32:10.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.033477731Z level=info msg="Executing migration" id="create correlation v2" 2024-09-18T07:32:10.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.034206844Z level=info msg="Migration successfully executed" id="create correlation v2" duration=728.455µs 2024-09-18T07:32:10.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.036301913Z level=info msg="Executing migration" id="create index IDX_correlation_uid - v2" 2024-09-18T07:32:10.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.036972649Z level=info msg="Migration successfully executed" id="create index IDX_correlation_uid - v2" duration=672.804µs 2024-09-18T07:32:10.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.039014028Z level=info msg="Executing migration" id="create index IDX_correlation_source_uid - v2" 2024-09-18T07:32:10.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.039717541Z level=info msg="Migration successfully executed" id="create index IDX_correlation_source_uid - v2" duration=703.344µs 2024-09-18T07:32:10.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.040068325Z level=info msg="Executing migration" id="create index IDX_correlation_org_id - v2" 2024-09-18T07:32:10.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.040707215Z level=info msg="Migration successfully executed" id="create index IDX_correlation_org_id - v2" duration=638.671µs 2024-09-18T07:32:10.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.042864838Z level=info msg="Executing migration" id="copy correlation v1 to v2" 2024-09-18T07:32:10.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.043046611Z level=info msg="Migration successfully executed" id="copy correlation v1 to v2" duration=181.598µs 2024-09-18T07:32:10.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.045593914Z level=info msg="Executing migration" id="drop correlation_tmp_qwerty" 2024-09-18T07:32:10.234 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.046143985Z level=info msg="Migration successfully executed" id="drop correlation_tmp_qwerty" duration=549.788µs 2024-09-18T07:32:10.235 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.048179682Z level=info msg="Executing migration" id="add provisioning column" 2024-09-18T07:32:10.235 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.053469457Z level=info msg="Migration successfully executed" id="add provisioning column" duration=5.289007ms 2024-09-18T07:32:10.235 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.053840782Z level=info msg="Executing migration" id="create entity_events table" 2024-09-18T07:32:10.235 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.054421181Z level=info msg="Migration successfully executed" id="create entity_events table" duration=580.227µs 2024-09-18T07:32:10.235 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.056809095Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-18T07:32:10.235 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.057528039Z level=info msg="Migration successfully executed" id="create dashboard public config v1" duration=718.808µs 2024-09-18T07:32:10.235 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:10.236 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.059603848Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-18T07:32:10.236 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.059994625Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-18T07:32:10.236 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.062254171Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-18T07:32:10.236 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.062532881Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-18T07:32:10.236 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.065373184Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-18T07:32:10.236 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.066038621Z level=info msg="Migration successfully executed" id="Drop old dashboard public config table" duration=667.673µs 2024-09-18T07:32:10.237 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.068151523Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-18T07:32:10.237 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.068823082Z level=info msg="Migration successfully executed" id="recreate dashboard public config v1" duration=671.436µs 2024-09-18T07:32:10.237 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.069224312Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-18T07:32:10.237 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.069909522Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v1" duration=685.118µs 2024-09-18T07:32:10.237 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.072015511Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-18T07:32:10.237 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.072736108Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" duration=720.282µs 2024-09-18T07:32:10.237 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.074989682Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-18T07:32:10.237 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.075766104Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_public_config_uid - v2" duration=779.044µs 2024-09-18T07:32:10.237 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.077787992Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-18T07:32:10.237 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.078514813Z level=info msg="Migration successfully executed" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=729.474µs 2024-09-18T07:32:10.238 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.078895147Z level=info msg="Executing migration" id="Drop public config table" 2024-09-18T07:32:10.238 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.079467173Z level=info msg="Migration successfully executed" id="Drop public config table" duration=571.853µs 2024-09-18T07:32:10.238 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.081626296Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-18T07:32:10.238 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.082374069Z level=info msg="Migration successfully executed" id="Recreate dashboard public config v2" duration=748.517µs 2024-09-18T07:32:10.238 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.084477093Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-18T07:32:10.238 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.085290409Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v2" duration=815.532µs 2024-09-18T07:32:10.238 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.086938486Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-18T07:32:10.238 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.087656172Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=717.503µs 2024-09-18T07:32:10.238 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.088025908Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-18T07:32:10.238 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.088715163Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_access_token - v2" duration=687.267µs 2024-09-18T07:32:10.238 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.090808164Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-18T07:32:10.238 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.10703993Z level=info msg="Migration successfully executed" id="Rename table dashboard_public_config to dashboard_public - v2" duration=16.23035ms 2024-09-18T07:32:10.238 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.10747077Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-18T07:32:10.238 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.112894171Z level=info msg="Migration successfully executed" id="add annotations_enabled column" duration=5.422235ms 2024-09-18T07:32:10.239 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.113369217Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-18T07:32:10.239 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.118334457Z level=info msg="Migration successfully executed" id="add time_selection_enabled column" duration=4.963759ms 2024-09-18T07:32:10.239 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.118746971Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-18T07:32:10.239 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.118901207Z level=info msg="Migration successfully executed" id="delete orphaned public dashboards" duration=154.706µs 2024-09-18T07:32:10.239 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.121506881Z level=info msg="Executing migration" id="add share column" 2024-09-18T07:32:10.239 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.126340794Z level=info msg="Migration successfully executed" id="add share column" duration=4.83248ms 2024-09-18T07:32:10.239 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.126745791Z level=info msg="Executing migration" id="backfill empty share column fields with default of public" 2024-09-18T07:32:10.239 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.126890242Z level=info msg="Migration successfully executed" id="backfill empty share column fields with default of public" duration=144.741µs 2024-09-18T07:32:10.239 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.129484226Z level=info msg="Executing migration" id="create file table" 2024-09-18T07:32:10.239 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.130229044Z level=info msg="Migration successfully executed" id="create file table" duration=743.292µs 2024-09-18T07:32:10.239 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.132105278Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-18T07:32:10.240 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.133021627Z level=info msg="Migration successfully executed" id="file table idx: path natural pk" duration=919.188µs 2024-09-18T07:32:10.240 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.134706462Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-18T07:32:10.240 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.135382693Z level=info msg="Migration successfully executed" id="file table idx: parent_folder_path_hash fast folder retrieval" duration=676.071µs 2024-09-18T07:32:10.240 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.13722713Z level=info msg="Executing migration" id="create file_meta table" 2024-09-18T07:32:10.240 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.137744409Z level=info msg="Migration successfully executed" id="create file_meta table" duration=524.214µs 2024-09-18T07:32:10.240 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.139931703Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-18T07:32:10.240 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.140624561Z level=info msg="Migration successfully executed" id="file table idx: path key" duration=692.582µs 2024-09-18T07:32:10.240 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.142628071Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-18T07:32:10.240 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.142670842Z level=info msg="Migration successfully executed" id="set path collation in file table" duration=43.088µs 2024-09-18T07:32:10.240 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.143429801Z level=info msg="Executing migration" id="migrate contents column to mediumblob for MySQL" 2024-09-18T07:32:10.240 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.143468868Z level=info msg="Migration successfully executed" id="migrate contents column to mediumblob for MySQL" duration=39.382µs 2024-09-18T07:32:10.240 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.144283668Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-18T07:32:10.241 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.144605276Z level=info msg="Migration successfully executed" id="managed permissions migration" duration=321.471µs 2024-09-18T07:32:10.241 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.145108824Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-18T07:32:10.241 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.145232879Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions migration" duration=124.079µs 2024-09-18T07:32:10.241 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.145930352Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-18T07:32:10.241 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.146761792Z level=info msg="Migration successfully executed" id="RBAC action name migrator" duration=829.622µs 2024-09-18T07:32:10.241 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.147162366Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-18T07:32:10.241 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.152149254Z level=info msg="Migration successfully executed" id="Add UID column to playlist" duration=4.986164ms 2024-09-18T07:32:10.241 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.152498865Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-18T07:32:10.241 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.152605762Z level=info msg="Migration successfully executed" id="Update uid column values in playlist" duration=105.413µs 2024-09-18T07:32:10.241 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.153374615Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-18T07:32:10.241 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.154129984Z level=info msg="Migration successfully executed" id="Add index for uid in playlist" duration=755.157µs 2024-09-18T07:32:10.242 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.154494088Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-18T07:32:10.242 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.154736755Z level=info msg="Migration successfully executed" id="update group index for alert rules" duration=243.344µs 2024-09-18T07:32:10.242 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.155290471Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-18T07:32:10.242 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.155418593Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated migration" duration=129.809µs 2024-09-18T07:32:10.242 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.156143957Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-18T07:32:10.242 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.156429554Z level=info msg="Migration successfully executed" id="admin only folder/dashboard permission" duration=286.565µs 2024-09-18T07:32:10.242 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.157004979Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-18T07:32:10.242 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.162009928Z level=info msg="Migration successfully executed" id="add action column to seed_assignment" duration=5.003802ms 2024-09-18T07:32:10.242 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.162370301Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-18T07:32:10.242 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.167361582Z level=info msg="Migration successfully executed" id="add scope column to seed_assignment" duration=4.989473ms 2024-09-18T07:32:10.242 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.167738015Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-18T07:32:10.243 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.168424698Z level=info msg="Migration successfully executed" id="remove unique index builtin_role_role_name before nullable update" duration=686.714µs 2024-09-18T07:32:10.243 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.168762627Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-18T07:32:10.243 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.225986226Z level=info msg="Migration successfully executed" id="update seed_assignment role_name column to nullable" duration=57.220247ms 2024-09-18T07:32:10.243 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.226547519Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-18T07:32:10.243 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.22740056Z level=info msg="Migration successfully executed" id="add unique index builtin_role_name back" duration=853.57µs 2024-09-18T07:32:10.243 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.227764602Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-18T07:32:10.243 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.22858602Z level=info msg="Migration successfully executed" id="add unique index builtin_role_action_scope" duration=820.919µs 2024-09-18T07:32:10.243 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.229056394Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-18T07:32:10.264 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:10.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:10.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:10.282 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:10 smithi137 ceph-mon[39175]: osdmap e83: 8 total, 8 up, 8 in 2024-09-18T07:32:10.283 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:10 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:10.283 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:10 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:10.283 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:10 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:10.283 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:10 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:10.283 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:10 smithi137 ceph-mon[39175]: Deploying daemon node-exporter.a on smithi137 2024-09-18T07:32:10.283 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:10 smithi137 ceph-mon[39175]: pgmap v35: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:32:10.283 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:10 smithi137 ceph-mon[39175]: osdmap e84: 8 total, 8 up, 8 in 2024-09-18T07:32:10.283 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:10 smithi137 ceph-mon[29408]: osdmap e83: 8 total, 8 up, 8 in 2024-09-18T07:32:10.283 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:10 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:10.283 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:10 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:10.284 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:10 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:10.284 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:10 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:10.284 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:10 smithi137 ceph-mon[29408]: Deploying daemon node-exporter.a on smithi137 2024-09-18T07:32:10.284 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:10 smithi137 ceph-mon[29408]: pgmap v35: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:32:10.284 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:10 smithi137 ceph-mon[29408]: osdmap e84: 8 total, 8 up, 8 in 2024-09-18T07:32:10.367 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:10.389 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:10.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:10.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:10.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:10.496 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.247389395Z level=info msg="Migration successfully executed" id="add primary key to seed_assigment" duration=18.332006ms 2024-09-18T07:32:10.496 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.247841722Z level=info msg="Executing migration" id="add origin column to seed_assignment" 2024-09-18T07:32:10.496 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.252940054Z level=info msg="Migration successfully executed" id="add origin column to seed_assignment" duration=5.096788ms 2024-09-18T07:32:10.496 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.253295773Z level=info msg="Executing migration" id="add origin to plugin seed_assignment" 2024-09-18T07:32:10.497 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.2534842Z level=info msg="Migration successfully executed" id="add origin to plugin seed_assignment" duration=188.652µs 2024-09-18T07:32:10.497 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.254127565Z level=info msg="Executing migration" id="prevent seeding OnCall access" 2024-09-18T07:32:10.497 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.254240874Z level=info msg="Migration successfully executed" id="prevent seeding OnCall access" duration=111.967µs 2024-09-18T07:32:10.497 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.254619311Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-18T07:32:10.497 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.254753598Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated fixed migration" duration=133.816µs 2024-09-18T07:32:10.497 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.255489483Z level=info msg="Executing migration" id="managed folder permissions library panel actions migration" 2024-09-18T07:32:10.497 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.255629967Z level=info msg="Migration successfully executed" id="managed folder permissions library panel actions migration" duration=140.409µs 2024-09-18T07:32:10.497 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.256334994Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-18T07:32:10.497 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.256477918Z level=info msg="Migration successfully executed" id="migrate external alertmanagers to datsourcse" duration=142.967µs 2024-09-18T07:32:10.497 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.257169704Z level=info msg="Executing migration" id="create folder table" 2024-09-18T07:32:10.497 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.257759431Z level=info msg="Migration successfully executed" id="create folder table" duration=589.752µs 2024-09-18T07:32:10.497 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.258114799Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-18T07:32:10.497 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.258911048Z level=info msg="Migration successfully executed" id="Add index for parent_uid" duration=795.865µs 2024-09-18T07:32:10.498 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.259291713Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-18T07:32:10.498 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.260023774Z level=info msg="Migration successfully executed" id="Add unique index for folder.uid and folder.org_id" duration=732.932µs 2024-09-18T07:32:10.498 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.260374534Z level=info msg="Executing migration" id="Update folder title length" 2024-09-18T07:32:10.498 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.260390336Z level=info msg="Migration successfully executed" id="Update folder title length" duration=16.622µs 2024-09-18T07:32:10.498 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.261202503Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-18T07:32:10.499 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.261926438Z level=info msg="Migration successfully executed" id="Add unique index for folder.title and folder.parent_uid" duration=723.757µs 2024-09-18T07:32:10.499 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.262292094Z level=info msg="Executing migration" id="Remove unique index for folder.title and folder.parent_uid" 2024-09-18T07:32:10.499 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.262977825Z level=info msg="Migration successfully executed" id="Remove unique index for folder.title and folder.parent_uid" duration=685.666µs 2024-09-18T07:32:10.499 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.263324503Z level=info msg="Executing migration" id="Add unique index for title, parent_uid, and org_id" 2024-09-18T07:32:10.499 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.26406613Z level=info msg="Migration successfully executed" id="Add unique index for title, parent_uid, and org_id" duration=741.221µs 2024-09-18T07:32:10.499 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.264432228Z level=info msg="Executing migration" id="Sync dashboard and folder table" 2024-09-18T07:32:10.499 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.264728503Z level=info msg="Migration successfully executed" id="Sync dashboard and folder table" duration=296.11µs 2024-09-18T07:32:10.499 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.265263957Z level=info msg="Executing migration" id="Remove ghost folders from the folder table" 2024-09-18T07:32:10.499 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.265449124Z level=info msg="Migration successfully executed" id="Remove ghost folders from the folder table" duration=185.071µs 2024-09-18T07:32:10.499 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.265838054Z level=info msg="Executing migration" id="Remove unique index UQE_folder_uid_org_id" 2024-09-18T07:32:10.499 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.266520659Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_uid_org_id" duration=682.503µs 2024-09-18T07:32:10.499 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.266881448Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_uid" 2024-09-18T07:32:10.499 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.267877956Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_uid" duration=995.319µs 2024-09-18T07:32:10.499 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.268439418Z level=info msg="Executing migration" id="Remove unique index UQE_folder_title_parent_uid_org_id" 2024-09-18T07:32:10.500 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.269509759Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_title_parent_uid_org_id" duration=1.070208ms 2024-09-18T07:32:10.500 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.271215849Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_parent_uid_title" 2024-09-18T07:32:10.500 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.272332304Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_parent_uid_title" duration=1.132951ms 2024-09-18T07:32:10.500 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.272795767Z level=info msg="Executing migration" id="Remove index IDX_folder_parent_uid_org_id" 2024-09-18T07:32:10.500 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.273836529Z level=info msg="Migration successfully executed" id="Remove index IDX_folder_parent_uid_org_id" duration=1.040831ms 2024-09-18T07:32:10.500 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.275564254Z level=info msg="Executing migration" id="create anon_device table" 2024-09-18T07:32:10.500 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.276450381Z level=info msg="Migration successfully executed" id="create anon_device table" duration=886.047µs 2024-09-18T07:32:10.500 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.278438869Z level=info msg="Executing migration" id="add unique index anon_device.device_id" 2024-09-18T07:32:10.500 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.279589365Z level=info msg="Migration successfully executed" id="add unique index anon_device.device_id" duration=1.150434ms 2024-09-18T07:32:10.500 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.281127889Z level=info msg="Executing migration" id="add index anon_device.updated_at" 2024-09-18T07:32:10.500 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.282202734Z level=info msg="Migration successfully executed" id="add index anon_device.updated_at" duration=1.074446ms 2024-09-18T07:32:10.500 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.282713065Z level=info msg="Executing migration" id="create signing_key table" 2024-09-18T07:32:10.500 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.283654311Z level=info msg="Migration successfully executed" id="create signing_key table" duration=941.436µs 2024-09-18T07:32:10.501 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.28544691Z level=info msg="Executing migration" id="add unique index signing_key.key_id" 2024-09-18T07:32:10.501 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.286525015Z level=info msg="Migration successfully executed" id="add unique index signing_key.key_id" duration=1.078003ms 2024-09-18T07:32:10.501 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.288171043Z level=info msg="Executing migration" id="set legacy alert migration status in kvstore" 2024-09-18T07:32:10.501 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.289308639Z level=info msg="Migration successfully executed" id="set legacy alert migration status in kvstore" duration=1.137571ms 2024-09-18T07:32:10.501 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.291000085Z level=info msg="Executing migration" id="migrate record of created folders during legacy migration to kvstore" 2024-09-18T07:32:10.501 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.291244775Z level=info msg="Migration successfully executed" id="migrate record of created folders during legacy migration to kvstore" duration=247.531µs 2024-09-18T07:32:10.501 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.291772466Z level=info msg="Executing migration" id="Add folder_uid for dashboard" 2024-09-18T07:32:10.501 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.301245483Z level=info msg="Migration successfully executed" id="Add folder_uid for dashboard" duration=9.455428ms 2024-09-18T07:32:10.501 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.30173022Z level=info msg="Executing migration" id="Populate dashboard folder_uid column" 2024-09-18T07:32:10.501 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.30243336Z level=info msg="Migration successfully executed" id="Populate dashboard folder_uid column" duration=703.946µs 2024-09-18T07:32:10.501 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.304555032Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title" 2024-09-18T07:32:10.501 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.305654061Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title" duration=1.098783ms 2024-09-18T07:32:10.502 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.307294477Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_id_title" 2024-09-18T07:32:10.502 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.308381472Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_id_title" duration=1.08692ms 2024-09-18T07:32:10.502 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.308858581Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_uid_title" 2024-09-18T07:32:10.502 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.309941903Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_uid_title" duration=1.083171ms 2024-09-18T07:32:10.502 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.311571994Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" 2024-09-18T07:32:10.502 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.312748041Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" duration=1.175905ms 2024-09-18T07:32:10.502 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.314304564Z level=info msg="Executing migration" id="Restore index for dashboard_org_id_folder_id_title" 2024-09-18T07:32:10.502 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.315401528Z level=info msg="Migration successfully executed" id="Restore index for dashboard_org_id_folder_id_title" duration=1.096602ms 2024-09-18T07:32:10.502 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.317101436Z level=info msg="Executing migration" id="create sso_setting table" 2024-09-18T07:32:10.502 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.318190516Z level=info msg="Migration successfully executed" id="create sso_setting table" duration=1.088656ms 2024-09-18T07:32:10.502 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.318680681Z level=info msg="Executing migration" id="copy kvstore migration status to each org" 2024-09-18T07:32:10.502 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.319408774Z level=info msg="Migration successfully executed" id="copy kvstore migration status to each org" duration=729.203µs 2024-09-18T07:32:10.503 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.321392343Z level=info msg="Executing migration" id="add back entry for orgid=0 migrated status" 2024-09-18T07:32:10.503 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.321671678Z level=info msg="Migration successfully executed" id="add back entry for orgid=0 migrated status" duration=280.396µs 2024-09-18T07:32:10.503 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.324111717Z level=info msg="Executing migration" id="alter kv_store.value to longtext" 2024-09-18T07:32:10.503 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.324162097Z level=info msg="Migration successfully executed" id="alter kv_store.value to longtext" duration=53.193µs 2024-09-18T07:32:10.503 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.326866793Z level=info msg="Executing migration" id="add notification_settings column to alert_rule table" 2024-09-18T07:32:10.503 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.332161094Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule table" duration=5.294172ms 2024-09-18T07:32:10.503 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.332543185Z level=info msg="Executing migration" id="add notification_settings column to alert_rule_version table" 2024-09-18T07:32:10.503 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.337891882Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule_version table" duration=5.346042ms 2024-09-18T07:32:10.503 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.338261273Z level=info msg="Executing migration" id="removing scope from alert.instances:read action migration" 2024-09-18T07:32:10.503 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.338505344Z level=info msg="Migration successfully executed" id="removing scope from alert.instances:read action migration" duration=244.543µs 2024-09-18T07:32:10.503 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=migrator t=2024-09-18T07:32:10.340967655Z level=info msg="migrations completed" performed=547 skipped=0 duration=1.302751544s 2024-09-18T07:32:10.503 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=sqlstore t=2024-09-18T07:32:10.341873788Z level=info msg="Created default organization" 2024-09-18T07:32:10.503 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=secrets t=2024-09-18T07:32:10.344037854Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-18T07:32:10.504 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=plugin.store t=2024-09-18T07:32:10.358681382Z level=info msg="Loading plugins..." 2024-09-18T07:32:10.504 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=local.finder t=2024-09-18T07:32:10.405115099Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-09-18T07:32:10.504 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=plugin.store t=2024-09-18T07:32:10.405296839Z level=info msg="Plugins loaded" count=55 duration=46.616771ms 2024-09-18T07:32:10.504 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=query_data t=2024-09-18T07:32:10.40750515Z level=info msg="Query Service initialization" 2024-09-18T07:32:10.504 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=live.push_http t=2024-09-18T07:32:10.413335252Z level=info msg="Live Push Gateway initialization" 2024-09-18T07:32:10.504 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=ngalert.migration t=2024-09-18T07:32:10.414439725Z level=info msg=Starting 2024-09-18T07:32:10.504 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=ngalert.migration t=2024-09-18T07:32:10.414711398Z level=info msg="Applying transition" currentType=Legacy desiredType=UnifiedAlerting cleanOnDowngrade=false cleanOnUpgrade=false 2024-09-18T07:32:10.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=ngalert.migration orgID=1 t=2024-09-18T07:32:10.415015616Z level=info msg="Migrating alerts for organisation" 2024-09-18T07:32:10.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=ngalert.migration orgID=1 t=2024-09-18T07:32:10.415464786Z level=info msg="Alerts found to migrate" alerts=0 2024-09-18T07:32:10.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=ngalert.migration t=2024-09-18T07:32:10.416651516Z level=info msg="Completed alerting migration" 2024-09-18T07:32:10.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=ngalert.state.manager t=2024-09-18T07:32:10.430175314Z level=info msg="Running in alternative execution of Error/NoData mode" 2024-09-18T07:32:10.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=infra.usagestats.collector t=2024-09-18T07:32:10.431522711Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-18T07:32:10.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=provisioning.datasources t=2024-09-18T07:32:10.433255658Z level=info msg="inserting datasource from configuration" name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-18T07:32:10.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=provisioning.datasources t=2024-09-18T07:32:10.445647424Z level=info msg="inserting datasource from configuration" name=Loki uid=P8E80F9AEF21F6940 2024-09-18T07:32:10.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=provisioning.alerting t=2024-09-18T07:32:10.458586796Z level=info msg="starting to provision alerting" 2024-09-18T07:32:10.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=provisioning.alerting t=2024-09-18T07:32:10.458604827Z level=info msg="finished to provision alerting" 2024-09-18T07:32:10.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=grafanaStorageLogger t=2024-09-18T07:32:10.458716371Z level=info msg="Storage starting" 2024-09-18T07:32:10.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=ngalert.state.manager t=2024-09-18T07:32:10.458876072Z level=info msg="Warming state cache for startup" 2024-09-18T07:32:10.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=ngalert.state.manager t=2024-09-18T07:32:10.459237903Z level=info msg="State cache has been initialized" states=0 duration=360.788µs 2024-09-18T07:32:10.505 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=ngalert.multiorg.alertmanager t=2024-09-18T07:32:10.459269719Z level=info msg="Starting MultiOrg Alertmanager" 2024-09-18T07:32:10.506 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=ngalert.scheduler t=2024-09-18T07:32:10.459290683Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1 2024-09-18T07:32:10.506 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=ticker t=2024-09-18T07:32:10.459326399Z level=info msg=starting first_tick=2024-09-18T07:32:20Z 2024-09-18T07:32:10.506 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=http.server t=2024-09-18T07:32:10.462640909Z level=info msg="HTTP Server TLS settings" MinTLSVersion=TLS1.2 configuredciphers=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA 2024-09-18T07:32:10.506 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=http.server t=2024-09-18T07:32:10.462963145Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-18T07:32:10.506 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=provisioning.dashboard t=2024-09-18T07:32:10.494318688Z level=info msg="starting to provision dashboards" 2024-09-18T07:32:10.514 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:10.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:10.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:10.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:10.638 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:10.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:10.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:10.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:10.761 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:10.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:10.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:10.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:10.888 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:10.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:10.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:10.961 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=sqlstore.transactions t=2024-09-18T07:32:10.497849073Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-18T07:32:10.961 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=sqlstore.transactions t=2024-09-18T07:32:10.508984488Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-18T07:32:10.961 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=sqlstore.transactions t=2024-09-18T07:32:10.519708131Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-09-18T07:32:10.961 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=sqlstore.transactions t=2024-09-18T07:32:10.530102329Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-09-18T07:32:10.961 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=grafana.update.checker t=2024-09-18T07:32:10.536465422Z level=info msg="Update check succeeded" duration=76.575726ms 2024-09-18T07:32:10.961 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=plugins.update.checker t=2024-09-18T07:32:10.540030443Z level=info msg="Update check succeeded" duration=81.027798ms 2024-09-18T07:32:10.961 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=provisioning.dashboard t=2024-09-18T07:32:10.719400523Z level=info msg="finished to provision dashboards" 2024-09-18T07:32:10.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:11.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:11.012 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:11.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:11.113 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:11.122 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:32:11.123 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:32:11.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:11.136 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:11.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:11.137 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:32:11.137 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:32:11.151 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:32:11.151 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:32:11.167 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:32:11.167 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:32:11.183 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:32:11.184 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:32:11.201 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:32:11.201 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:32:11.216 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:32:11.217 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:32:11.234 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:32:11.234 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:32:11.236 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:11.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:11.258 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:11.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:11.359 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:11.380 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:11.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:11.433 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=grafana-apiserver t=2024-09-18T07:32:10.964987128Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager" 2024-09-18T07:32:11.433 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:32:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=grafana-apiserver t=2024-09-18T07:32:10.965356175Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager" 2024-09-18T07:32:11.482 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:11.504 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:11.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:11.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:11.606 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:11.627 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:11.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:11.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:11.669 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:11 smithi137 bash[73511]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-18T07:32:11.729 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:11.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:11.752 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:11.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:11.754 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-18T07:32:11.853 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:11.875 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:11.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:11.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:11 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:11.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:11 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:11.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:11 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:32:11.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:11 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3778542777' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:32:11.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:11 smithi203 ceph-mon[32237]: from='client.34721 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:32:11.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:11 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2188158292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:32:11.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:11.999 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:12.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:12.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:12.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:32:12.092 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:11 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:12.092 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:11 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:12.092 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:11 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:32:12.092 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:11 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3778542777' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:32:12.092 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:11 smithi137 ceph-mon[39175]: from='client.34721 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:32:12.092 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:11 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2188158292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:32:12.093 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:11 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:12.093 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:11 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:12.093 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:11 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:32:12.093 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:11 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3778542777' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:32:12.093 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:11 smithi137 ceph-mon[29408]: from='client.34721 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:32:12.093 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:11 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2188158292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:32:12.101 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:12.123 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:12.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:12.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:12.225 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:12.246 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:12.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:12.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:12.347 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:12.368 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:12.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:12.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:12.470 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:12.492 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:12.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:12.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:12.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:12.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:12.615 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:12.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:12.716 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:12.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:12.739 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:12.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:12.840 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:12.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:12.863 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:12.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:12.964 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:12.986 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:12.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:12.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:12.998 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:32:12.998 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:32:13.087 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:13.110 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:13.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:13.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:13.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:12 smithi137 ceph-mon[29408]: pgmap v37: 148 pgs: 1 unknown, 147 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:32:13.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:12 smithi137 ceph-mon[29408]: from='client.34721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:32:13.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:12 smithi137 ceph-mon[29408]: osdmap e85: 8 total, 8 up, 8 in 2024-09-18T07:32:13.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:12 smithi137 ceph-mon[39175]: pgmap v37: 148 pgs: 1 unknown, 147 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:32:13.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:12 smithi137 ceph-mon[39175]: from='client.34721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:32:13.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:12 smithi137 ceph-mon[39175]: osdmap e85: 8 total, 8 up, 8 in 2024-09-18T07:32:13.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:12 smithi203 ceph-mon[32237]: pgmap v37: 148 pgs: 1 unknown, 147 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:32:13.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:12 smithi203 ceph-mon[32237]: from='client.34721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:32:13.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:12 smithi203 ceph-mon[32237]: osdmap e85: 8 total, 8 up, 8 in 2024-09-18T07:32:13.212 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:13.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:13.235 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:13.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:13.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:13.356 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:13.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:13.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:13.458 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:13.481 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:13.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:13.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:13.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:13.606 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:13.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:13.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:13.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:13.729 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:13.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:13.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:13.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:13.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:13.855 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:13.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:13.956 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:13.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:13.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:13.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:14.079 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:14.102 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:14.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:14.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:13 smithi137 ceph-mon[29408]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:32:14.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:13 smithi137 ceph-mon[39175]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:32:14.170 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:13 smithi137 bash[73511]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-18T07:32:14.204 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:14.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:13 smithi203 ceph-mon[32237]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:32:14.226 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:14.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:14.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:14.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:32:14.327 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:14.354 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:14.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:14.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:14.402 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:32:14.403 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:32:14.403 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-18T07:32:14.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-18T07:32:14.456 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:14.477 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:14.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:14.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:14.579 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:14.600 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:14.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:14.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:14.654 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:14 smithi137 bash[73511]: Writing manifest to image destination 2024-09-18T07:32:14.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:14.723 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:14.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:14.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:14.819 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:14.825 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:14.846 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T07:32:14.849 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:14.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:14.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:14.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:14 smithi137 ceph-mon[29408]: pgmap v39: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 954 B/s rd, 0 op/s 2024-09-18T07:32:14.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:14 smithi137 ceph-mon[29408]: osdmap e86: 8 total, 8 up, 8 in 2024-09-18T07:32:14.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:14 smithi137 ceph-mon[39175]: pgmap v39: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 954 B/s rd, 0 op/s 2024-09-18T07:32:14.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:14 smithi137 ceph-mon[39175]: osdmap e86: 8 total, 8 up, 8 in 2024-09-18T07:32:14.920 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:14 smithi137 podman[73511]: 2024-09-18 07:32:14.687308389 +0000 UTC m=+3.762688131 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-18T07:32:14.951 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:14.977 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:14.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:14.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:15.079 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:15.104 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:15.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:15.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:15.206 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:15.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:14 smithi203 ceph-mon[32237]: pgmap v39: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 954 B/s rd, 0 op/s 2024-09-18T07:32:15.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:14 smithi203 ceph-mon[32237]: osdmap e86: 8 total, 8 up, 8 in 2024-09-18T07:32:15.232 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:15.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:15.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:15.249 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:14 smithi137 podman[73511]: 2024-09-18 07:32:14.952776473 +0000 UTC m=+4.028156203 container create 99cfcc69738e4b7bf82b21c6b9b23e60dc2e1afa8f0ca597e377a13f6390add7 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-18T07:32:15.249 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 podman[73511]: 2024-09-18 07:32:15.16352874 +0000 UTC m=+4.238908476 container init 99cfcc69738e4b7bf82b21c6b9b23e60dc2e1afa8f0ca597e377a13f6390add7 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-18T07:32:15.249 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 podman[73511]: 2024-09-18 07:32:15.166551332 +0000 UTC m=+4.241931061 container start 99cfcc69738e4b7bf82b21c6b9b23e60dc2e1afa8f0ca597e377a13f6390add7 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-18T07:32:15.249 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.169Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-18T07:32:15.249 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.169Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-18T07:32:15.249 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.170Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-18T07:32:15.250 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.170Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-18T07:32:15.250 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.170Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-18T07:32:15.250 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.170Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-18T07:32:15.250 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-18T07:32:15.250 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=arp 2024-09-18T07:32:15.250 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-18T07:32:15.250 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-18T07:32:15.250 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-18T07:32:15.250 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-18T07:32:15.251 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-18T07:32:15.251 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-18T07:32:15.251 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-18T07:32:15.251 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-18T07:32:15.251 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=edac 2024-09-18T07:32:15.251 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-18T07:32:15.251 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-18T07:32:15.251 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-18T07:32:15.251 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-18T07:32:15.251 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-18T07:32:15.251 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-18T07:32:15.252 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-18T07:32:15.252 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-18T07:32:15.252 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-18T07:32:15.252 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-18T07:32:15.252 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-18T07:32:15.252 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-18T07:32:15.252 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-18T07:32:15.252 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-18T07:32:15.252 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-18T07:32:15.252 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-18T07:32:15.252 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=os 2024-09-18T07:32:15.252 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-18T07:32:15.252 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-18T07:32:15.253 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-18T07:32:15.253 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-18T07:32:15.253 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-18T07:32:15.253 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-18T07:32:15.253 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-18T07:32:15.253 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=stat 2024-09-18T07:32:15.253 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-18T07:32:15.253 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-18T07:32:15.253 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-18T07:32:15.253 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=time 2024-09-18T07:32:15.253 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-18T07:32:15.253 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=uname 2024-09-18T07:32:15.253 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-18T07:32:15.254 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-18T07:32:15.254 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-18T07:32:15.254 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-18T07:32:15.254 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a[74095]: ts=2024-09-18T07:32:15.171Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-18T07:32:15.254 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 bash[73511]: 99cfcc69738e4b7bf82b21c6b9b23e60dc2e1afa8f0ca597e377a13f6390add7 2024-09-18T07:32:15.334 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:15.358 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:15.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:15.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:15.460 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:15.482 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:15.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:15.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:15.520 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:15.527 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:32:15 smithi137 systemd[1]: Started Ceph node-exporter.a for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:32:15.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:15.605 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:15.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:15.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:15.689 DEBUG:teuthology.orchestra.run.smithi137:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-18T07:32:15.691 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-18T07:32:15.692 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-18T07:32:15.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:15.728 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:15.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:15.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:15.768 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:hints = 1 2024-09-18T07:32:15.768 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-18T07:32:15.768 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Object prefix: benchmark_data_smithi137_74180 2024-09-18T07:32:15.768 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:32:15.768 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 0 0 0 0 0 0 - 0 2024-09-18T07:32:15.830 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:15.866 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:15.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:15.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:15.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:16.014 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:16.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:16.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:16.116 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:16.145 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:16.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:16.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:16.247 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:16.276 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:16.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:16.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:16.280 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:16.288 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:16.301 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:32:16.301 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:32:16.378 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:16.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:16 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:16.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:16 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:16.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:16 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:16.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:16 smithi137 ceph-mon[29408]: Deploying daemon node-exporter.b on smithi203 2024-09-18T07:32:16.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:16 smithi137 ceph-mon[29408]: pgmap v41: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:32:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:16 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:16 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:16 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:16 smithi137 ceph-mon[39175]: Deploying daemon node-exporter.b on smithi203 2024-09-18T07:32:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:16 smithi137 ceph-mon[39175]: pgmap v41: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:32:16.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:16.422 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:16.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:16.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:16 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:16.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:16 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:16.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:16 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:16.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:16 smithi203 ceph-mon[32237]: Deploying daemon node-exporter.b on smithi203 2024-09-18T07:32:16.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:16 smithi203 ceph-mon[32237]: pgmap v41: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:32:16.523 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:16.557 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:16.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:16.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:16.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:16.698 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:16.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:16.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:16.768 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 1 15 2928 2913 182.053 182.062 0.00678738 0.00547232 2024-09-18T07:32:16.800 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:16.832 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:16.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:16.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:16.836 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:16 smithi203 bash[60789]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-18T07:32:16.837 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:16 smithi203 bash[60789]: Getting image source signatures 2024-09-18T07:32:16.935 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:16.974 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:16.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:16.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:17.076 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:17.110 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:17.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:17.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:32:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:32:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:32:17.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:32:16 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:32:17.211 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:16 smithi203 bash[60789]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-18T07:32:17.211 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:16 smithi203 bash[60789]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-18T07:32:17.211 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:16 smithi203 bash[60789]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-18T07:32:17.213 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:17.245 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:17.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:17.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:17.335 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:32:17.347 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:17.375 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:17.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:17.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:17.477 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:17.515 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:17.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:17.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:17.617 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:17.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:17.640 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:17.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:17.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:17 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3355894004' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:32:17.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:17 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:17.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:17 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3355894004' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:32:17.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:17 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:17.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:17 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3355894004' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:32:17.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:17 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:17.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:17.768 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 2 16 6101 6085 190.121 198.25 0.00667165 0.00524893 2024-09-18T07:32:17.780 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:17.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:17.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:17.882 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:17.914 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:17.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:17.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:18.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:18.054 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:18.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:18.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:18.156 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:18.190 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:18.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:18.292 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:18.318 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:18.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:18.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:18.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:18.461 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:18.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:18.498 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:18 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:32:18.498 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:18 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3355894004' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:32:18.498 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:18 smithi137 ceph-mon[29408]: osdmap e87: 8 total, 8 up, 8 in 2024-09-18T07:32:18.498 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:18 smithi137 ceph-mon[29408]: pgmap v43: 148 pgs: 148 active+clean; 115 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 19 MiB/s wr, 305 op/s 2024-09-18T07:32:18.498 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:18 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:32:18.498 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:18 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3355894004' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:32:18.498 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:18 smithi137 ceph-mon[39175]: osdmap e87: 8 total, 8 up, 8 in 2024-09-18T07:32:18.499 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:18 smithi137 ceph-mon[39175]: pgmap v43: 148 pgs: 148 active+clean; 115 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 19 MiB/s wr, 305 op/s 2024-09-18T07:32:18.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:18.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:18.601 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:18.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:18.702 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:18.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:18 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:32:18.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:18 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3355894004' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:32:18.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:18 smithi203 ceph-mon[32237]: osdmap e87: 8 total, 8 up, 8 in 2024-09-18T07:32:18.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:18 smithi203 ceph-mon[32237]: pgmap v43: 148 pgs: 148 active+clean; 115 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 19 MiB/s wr, 305 op/s 2024-09-18T07:32:18.734 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:18.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:18.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:18.744 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:32:18.745 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:32:18.745 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:32:18.745 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:32:18.768 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 3 16 9042 9026 188.013 183.812 0.00742909 0.00530714 2024-09-18T07:32:18.769 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:32:18.769 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:32:18.805 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:32:18.805 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:32:18.837 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:18.847 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:32:18.847 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:32:18.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:18.864 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:18.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:18.879 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:32:18.879 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:32:18.914 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:32:18.915 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:32:18.927 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:32:18.928 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:32:18.928 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 87, 2024-09-18T07:32:18.928 INFO:teuthology.orchestra.run.smithi137.stdout: "fsid": "f7ef099c-758e-11ef-bceb-c7b262605968", 2024-09-18T07:32:18.928 INFO:teuthology.orchestra.run.smithi137.stdout: "created": "2024-09-18T07:25:19.662367+0000", 2024-09-18T07:32:18.928 INFO:teuthology.orchestra.run.smithi137.stdout: "modified": "2024-09-18T07:32:17.319195+0000", 2024-09-18T07:32:18.928 INFO:teuthology.orchestra.run.smithi137.stdout: "last_up_change": "2024-09-18T07:30:44.708496+0000", 2024-09-18T07:32:18.928 INFO:teuthology.orchestra.run.smithi137.stdout: "last_in_change": "2024-09-18T07:30:23.052551+0000", 2024-09-18T07:32:18.928 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T07:32:18.928 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_num": 5806080, 2024-09-18T07:32:18.929 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_set": [ 2024-09-18T07:32:18.929 INFO:teuthology.orchestra.run.smithi137.stdout: "nodeep-scrub", 2024-09-18T07:32:18.929 INFO:teuthology.orchestra.run.smithi137.stdout: "noscrub", 2024-09-18T07:32:18.929 INFO:teuthology.orchestra.run.smithi137.stdout: "pglog_hardlimit", 2024-09-18T07:32:18.929 INFO:teuthology.orchestra.run.smithi137.stdout: "purged_snapdirs", 2024-09-18T07:32:18.929 INFO:teuthology.orchestra.run.smithi137.stdout: "recovery_deletes", 2024-09-18T07:32:18.929 INFO:teuthology.orchestra.run.smithi137.stdout: "sortbitwise" 2024-09-18T07:32:18.929 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:32:18.930 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_version": 18, 2024-09-18T07:32:18.930 INFO:teuthology.orchestra.run.smithi137.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T07:32:18.930 INFO:teuthology.orchestra.run.smithi137.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T07:32:18.930 INFO:teuthology.orchestra.run.smithi137.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T07:32:18.930 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_snapshot": "", 2024-09-18T07:32:18.930 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_max": 7, 2024-09-18T07:32:18.930 INFO:teuthology.orchestra.run.smithi137.stdout: "max_osd": 8, 2024-09-18T07:32:18.930 INFO:teuthology.orchestra.run.smithi137.stdout: "require_min_compat_client": "luminous", 2024-09-18T07:32:18.931 INFO:teuthology.orchestra.run.smithi137.stdout: "min_compat_client": "jewel", 2024-09-18T07:32:18.931 INFO:teuthology.orchestra.run.smithi137.stdout: "require_osd_release": "squid", 2024-09-18T07:32:18.931 INFO:teuthology.orchestra.run.smithi137.stdout: "allow_crimson": false, 2024-09-18T07:32:18.931 INFO:teuthology.orchestra.run.smithi137.stdout: "pools": [ 2024-09-18T07:32:18.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.931 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 1, 2024-09-18T07:32:18.931 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".mgr", 2024-09-18T07:32:18.931 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:28:20.899655+0000", 2024-09-18T07:32:18.931 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:32:18.931 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:32:18.931 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:32:18.931 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:32:18.932 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:32:18.932 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:32:18.932 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:32:18.932 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:32:18.932 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:32:18.932 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:32:18.932 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:32:18.932 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:32:18.932 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:32:18.932 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 1, 2024-09-18T07:32:18.932 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 1, 2024-09-18T07:32:18.932 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 1, 2024-09-18T07:32:18.933 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 1, 2024-09-18T07:32:18.933 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 1, 2024-09-18T07:32:18.933 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:32:18.933 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:32:18.933 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:32:18.933 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:32:18.933 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:32:18.933 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:32:18.933 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:32:18.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.933 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "21", 2024-09-18T07:32:18.934 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:32:18.934 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:32:18.934 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:32:18.934 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:32:18.934 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:32:18.934 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:32:18.934 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:32:18.934 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:32:18.934 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:32:18.934 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:32:18.934 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:32:18.934 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:32:18.934 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:32:18.935 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:32:18.935 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:32:18.935 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:32:18.935 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:32:18.935 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:32:18.935 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:32:18.935 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:32:18.935 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:32:18.935 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:32:18.935 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:32:18.935 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:32:18.936 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:32:18.936 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:32:18.936 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.936 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:32:18.936 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:32:18.936 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:32:18.936 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:32:18.936 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:32:18.936 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:32:18.937 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:32:18.937 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:32:18.937 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:32:18.937 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:32:18.937 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:32:18.937 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:32:18.937 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_max": 32, 2024-09-18T07:32:18.937 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_min": 1 2024-09-18T07:32:18.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.937 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:32:18.937 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": {} 2024-09-18T07:32:18.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.938 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:32:18.938 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:32:18.938 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 7.8899998664855957, 2024-09-18T07:32:18.938 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 7.8899998664855957, 2024-09-18T07:32:18.938 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.37999999523162842, 2024-09-18T07:32:18.938 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 3, 2024-09-18T07:32:18.938 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 3, 2024-09-18T07:32:18.938 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:32:18.938 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:32:18.938 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:32:18.938 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:18.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.939 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 2, 2024-09-18T07:32:18.939 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "datapool", 2024-09-18T07:32:18.939 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:53.715373+0000", 2024-09-18T07:32:18.939 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-18T07:32:18.939 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T07:32:18.939 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:32:18.939 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:32:18.939 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:32:18.939 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:32:18.939 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:32:18.939 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:32:18.939 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:32:18.939 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:32:18.940 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:32:18.940 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:32:18.940 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:32:18.940 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 3, 2024-09-18T07:32:18.940 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 3, 2024-09-18T07:32:18.940 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 3, 2024-09-18T07:32:18.940 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 3, 2024-09-18T07:32:18.940 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 3, 2024-09-18T07:32:18.940 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:32:18.940 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:32:18.940 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:32:18.940 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:32:18.940 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:32:18.941 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:32:18.941 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:32:18.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.941 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "63", 2024-09-18T07:32:18.941 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:32:18.941 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:32:18.941 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:32:18.941 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:32:18.941 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:32:18.941 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 2, 2024-09-18T07:32:18.941 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 63, 2024-09-18T07:32:18.941 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:32:18.942 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:32:18.942 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:32:18.942 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:32:18.942 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:32:18.942 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:32:18.942 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:32:18.942 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:32:18.942 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:32:18.942 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:32:18.942 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:32:18.942 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:32:18.942 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:32:18.942 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:32:18.943 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:32:18.943 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:32:18.943 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:32:18.943 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:32:18.943 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:32:18.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.943 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:32:18.943 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:32:18.943 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:32:18.943 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:32:18.943 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:32:18.944 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:32:18.944 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:32:18.944 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:32:18.944 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:32:18.944 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:32:18.944 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:32:18.944 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:32:18.944 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:32:18.944 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd": {} 2024-09-18T07:32:18.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.944 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:32:18.944 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:32:18.945 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.6500000953674316, 2024-09-18T07:32:18.945 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.6500000953674316, 2024-09-18T07:32:18.945 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.87999999523162842, 2024-09-18T07:32:18.945 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-18T07:32:18.945 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-18T07:32:18.945 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:32:18.945 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:32:18.945 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:32:18.945 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:18.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.945 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 3, 2024-09-18T07:32:18.945 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".rgw.root", 2024-09-18T07:32:18.945 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:54.672127+0000", 2024-09-18T07:32:18.946 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:32:18.946 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:32:18.946 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:32:18.946 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:32:18.946 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:32:18.946 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:32:18.946 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:32:18.946 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:32:18.946 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:32:18.946 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:32:18.946 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:32:18.946 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:32:18.946 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:32:18.947 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:32:18.947 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:32:18.947 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:32:18.947 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:32:18.947 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:32:18.947 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:32:18.947 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:32:18.947 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:32:18.947 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:32:18.947 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:32:18.947 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:32:18.947 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:32:18.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.947 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "62", 2024-09-18T07:32:18.948 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:32:18.948 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:32:18.948 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:32:18.948 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:32:18.948 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:32:18.948 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:32:18.948 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:32:18.948 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:32:18.948 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:32:18.948 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:32:18.948 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:32:18.948 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:32:18.948 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:32:18.949 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:32:18.949 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:32:18.949 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:32:18.949 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:32:18.949 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:32:18.949 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:32:18.949 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:32:18.949 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:32:18.949 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:32:18.949 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:32:18.950 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:32:18.950 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:32:18.950 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:32:18.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.950 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:32:18.950 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:32:18.950 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:32:18.950 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:32:18.950 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:32:18.950 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:32:18.950 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:32:18.950 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:32:18.950 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:32:18.950 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:32:18.951 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:32:18.951 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:32:18.951 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:32:18.951 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:32:18.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.951 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:32:18.951 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:32:18.951 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.5, 2024-09-18T07:32:18.951 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.5, 2024-09-18T07:32:18.951 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:32:18.951 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.5, 2024-09-18T07:32:18.951 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.5, 2024-09-18T07:32:18.951 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:32:18.952 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:32:18.952 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:32:18.952 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:18.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.952 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 4, 2024-09-18T07:32:18.952 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.log", 2024-09-18T07:32:18.952 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:56.351343+0000", 2024-09-18T07:32:18.952 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:32:18.952 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:32:18.952 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:32:18.952 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:32:18.952 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:32:18.953 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:32:18.953 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:32:18.953 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:32:18.953 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:32:18.953 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:32:18.953 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:32:18.953 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:32:18.953 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:32:18.953 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:32:18.953 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:32:18.953 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:32:18.954 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:32:18.954 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:32:18.954 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:32:18.954 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:32:18.954 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:32:18.954 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:32:18.954 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:32:18.954 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:32:18.954 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:32:18.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.954 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "64", 2024-09-18T07:32:18.955 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:32:18.955 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:32:18.955 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:32:18.955 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:32:18.955 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:32:18.955 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:32:18.955 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:32:18.955 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:32:18.955 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:32:18.955 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:32:18.955 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:32:18.955 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:32:18.955 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:32:18.956 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:32:18.956 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:32:18.956 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:32:18.956 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:32:18.956 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:32:18.956 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:32:18.956 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:32:18.956 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:32:18.956 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:32:18.956 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:32:18.956 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:32:18.956 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:32:18.957 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:32:18.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.957 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:32:18.957 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:32:18.957 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:32:18.957 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:32:18.957 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:32:18.957 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:32:18.957 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:32:18.957 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:32:18.957 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:32:18.958 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:32:18.958 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:32:18.958 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:32:18.958 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:32:18.958 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:32:18.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.958 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:32:18.958 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:32:18.958 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.25, 2024-09-18T07:32:18.958 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.25, 2024-09-18T07:32:18.959 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:32:18.959 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.25, 2024-09-18T07:32:18.959 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.25, 2024-09-18T07:32:18.959 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:32:18.959 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:32:18.959 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:32:18.959 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:18.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.959 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 5, 2024-09-18T07:32:18.959 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.control", 2024-09-18T07:32:18.959 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:58.322401+0000", 2024-09-18T07:32:18.959 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:32:18.960 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:32:18.960 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:32:18.960 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:32:18.960 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:32:18.960 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:32:18.960 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:32:18.960 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:32:18.960 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:32:18.960 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:32:18.960 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:32:18.960 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:32:18.960 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:32:18.960 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:32:18.961 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:32:18.961 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:32:18.961 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:32:18.961 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:32:18.961 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:32:18.961 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:32:18.961 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:32:18.961 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:32:18.961 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:32:18.961 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:32:18.961 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:32:18.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.962 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "66", 2024-09-18T07:32:18.962 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:32:18.962 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:32:18.962 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:32:18.962 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:32:18.962 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:32:18.962 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:32:18.962 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:32:18.962 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:32:18.962 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:32:18.962 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:32:18.963 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:32:18.963 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:32:18.963 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:32:18.963 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:32:18.963 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:32:18.963 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:32:18.963 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:32:18.963 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:32:18.963 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:32:18.963 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:32:18.963 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:32:18.963 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:32:18.963 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:32:18.963 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:32:18.964 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:32:18.964 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:32:18.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.964 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:32:18.964 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:32:18.964 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:32:18.964 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:32:18.964 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:32:18.964 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:32:18.964 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:32:18.964 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:32:18.964 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:32:18.964 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:32:18.964 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:32:18.965 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:32:18.965 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:32:18.965 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:32:18.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.965 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:32:18.965 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:32:18.965 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.25, 2024-09-18T07:32:18.965 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.25, 2024-09-18T07:32:18.965 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:32:18.965 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.25, 2024-09-18T07:32:18.965 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.25, 2024-09-18T07:32:18.965 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:32:18.965 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:32:18.966 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:18.967 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:32:18.967 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:18.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.967 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 6, 2024-09-18T07:32:18.967 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.meta", 2024-09-18T07:32:18.967 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:31:00.390499+0000", 2024-09-18T07:32:18.967 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:32:18.968 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:32:18.968 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:32:18.968 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:32:18.968 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:32:18.968 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:32:18.968 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:32:18.968 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:32:18.968 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:32:18.968 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:32:18.968 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:32:18.969 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:32:18.969 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:32:18.969 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:32:18.969 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:32:18.969 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:32:18.969 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:32:18.969 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:32:18.969 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:32:18.969 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:32:18.969 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:32:18.969 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:32:18.969 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:32:18.970 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:32:18.970 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:32:18.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.970 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "68", 2024-09-18T07:32:18.970 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:32:18.970 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:32:18.970 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:32:18.970 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:32:18.970 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:32:18.970 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:32:18.970 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:32:18.970 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:32:18.971 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:32:18.971 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:32:18.971 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:32:18.971 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:32:18.971 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:32:18.971 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:32:18.971 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:32:18.971 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:32:18.971 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:32:18.971 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:32:18.971 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:32:18.971 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:32:18.972 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:32:18.972 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:32:18.972 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:32:18.972 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:32:18.972 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:32:18.972 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:32:18.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.972 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:32:18.972 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:32:18.972 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:32:18.972 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:32:18.973 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:32:18.973 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:32:18.973 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:32:18.973 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:32:18.973 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:32:18.973 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:32:18.973 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:32:18.973 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:32:18.973 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_bias": 4 2024-09-18T07:32:18.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.973 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:32:18.974 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:32:18.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.974 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:32:18.974 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:32:18.974 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.75, 2024-09-18T07:32:18.974 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.75, 2024-09-18T07:32:18.974 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:32:18.974 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.75, 2024-09-18T07:32:18.974 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.75, 2024-09-18T07:32:18.974 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:32:18.975 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:32:18.975 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:32:18.975 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:18.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.975 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-18T07:32:18.975 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "unique_pool_0", 2024-09-18T07:32:18.975 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:32:06.616722+0000", 2024-09-18T07:32:18.975 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:32:18.975 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:32:18.975 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:32:18.976 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:32:18.976 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:32:18.976 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:32:18.976 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:32:18.976 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:32:18.976 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:32:18.976 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:32:18.977 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:32:18.977 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:32:18.977 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:32:18.977 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 16, 2024-09-18T07:32:18.977 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 16, 2024-09-18T07:32:18.977 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 16, 2024-09-18T07:32:18.977 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 16, 2024-09-18T07:32:18.977 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 16, 2024-09-18T07:32:18.977 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:32:18.977 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:32:18.978 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:32:18.978 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:32:18.978 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:32:18.978 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:32:18.978 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:32:18.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.978 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "85", 2024-09-18T07:32:18.978 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:32:18.978 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:32:18.979 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:32:18.979 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:32:18.979 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:32:18.979 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:32:18.979 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:32:18.979 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:32:18.979 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:32:18.979 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:32:18.979 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:32:18.979 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:32:18.979 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:32:18.979 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:32:18.980 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:32:18.980 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:32:18.980 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:32:18.980 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:32:18.980 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:32:18.980 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:32:18.980 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:32:18.980 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:32:18.980 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:32:18.980 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:32:18.980 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:32:18.980 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:32:18.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.981 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:32:18.981 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:32:18.981 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:32:18.981 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:32:18.984 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:32:18.985 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:32:18.985 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:32:18.985 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:32:18.985 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:32:18.985 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:32:18.985 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:32:18.985 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:32:18.985 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:32:18.985 INFO:teuthology.orchestra.run.smithi137.stdout: "rados": {} 2024-09-18T07:32:18.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.985 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:32:18.986 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:32:18.986 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2, 2024-09-18T07:32:18.986 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2, 2024-09-18T07:32:18.986 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:32:18.986 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2, 2024-09-18T07:32:18.986 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2, 2024-09-18T07:32:18.986 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:32:18.986 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:32:18.986 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:32:18.986 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:18.986 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:18.986 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:32:18.986 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-18T07:32:18.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.987 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:32:18.987 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "268cd580-c931-4992-bb93-d6f63c2bfe88", 2024-09-18T07:32:18.987 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:32:18.987 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:32:18.987 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:32:18.987 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:32:18.987 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:32:18.987 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:32:18.987 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 8, 2024-09-18T07:32:18.987 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 81, 2024-09-18T07:32:18.987 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:32:18.987 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:32:18.987 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:32:18.988 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:18.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.988 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:18.988 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6801", 2024-09-18T07:32:18.988 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:32:18.988 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:18.988 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:18.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.988 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:32:18.988 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:18.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.988 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:18.988 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6802", 2024-09-18T07:32:18.989 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:32:18.989 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:18.989 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:18.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.989 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:32:18.989 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:18.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.989 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:18.989 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6804", 2024-09-18T07:32:18.989 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:32:18.989 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:18.989 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:18.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.990 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:32:18.990 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:18.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.990 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:18.990 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6803", 2024-09-18T07:32:18.990 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:32:18.990 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:18.990 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:18.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.990 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6801/2811444477", 2024-09-18T07:32:18.990 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6802/2811444477", 2024-09-18T07:32:18.990 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6804/2811444477", 2024-09-18T07:32:18.991 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6803/2811444477", 2024-09-18T07:32:18.991 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:32:18.991 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:32:18.991 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:32:18.991 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:18.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.991 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:32:18.991 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575", 2024-09-18T07:32:18.991 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:32:18.991 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:32:18.992 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:32:18.992 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:32:18.992 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:32:18.993 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:32:18.993 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:32:18.993 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:32:18.993 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 13, 2024-09-18T07:32:18.994 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 81, 2024-09-18T07:32:18.994 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:32:18.994 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:32:18.994 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:32:18.994 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:18.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.994 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:18.994 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6805", 2024-09-18T07:32:18.994 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:32:18.994 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:18.994 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:18.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.994 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:32:18.995 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:18.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.995 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:18.995 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6806", 2024-09-18T07:32:18.995 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:32:18.995 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:18.995 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:18.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.995 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:32:18.995 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:18.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.995 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:18.995 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6808", 2024-09-18T07:32:18.996 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:32:18.996 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:18.996 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:18.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.996 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:32:18.996 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:18.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.996 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:18.996 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6807", 2024-09-18T07:32:18.996 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:32:18.996 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:18.997 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:18.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.997 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6805/874102487", 2024-09-18T07:32:18.997 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6806/874102487", 2024-09-18T07:32:18.997 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6808/874102487", 2024-09-18T07:32:18.997 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6807/874102487", 2024-09-18T07:32:18.997 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:32:18.997 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:32:18.997 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:32:18.997 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:18.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.998 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:32:18.998 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "480ad6c5-0e0b-4405-bbe2-381682d00bae", 2024-09-18T07:32:18.998 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:32:18.998 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:32:18.998 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:32:18.998 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:32:18.998 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:32:18.998 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:32:18.998 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 18, 2024-09-18T07:32:18.998 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 81, 2024-09-18T07:32:18.998 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:32:18.998 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:32:18.998 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:32:18.999 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:18.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.999 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:18.999 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6809", 2024-09-18T07:32:18.999 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:32:18.999 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:18.999 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:18.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:18.999 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:32:18.999 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:18.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:18.999 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:18.999 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6810", 2024-09-18T07:32:18.999 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:32:18.999 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.000 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.000 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:32:19.000 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.000 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.000 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6812", 2024-09-18T07:32:19.000 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:32:19.000 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.000 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.000 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:32:19.000 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.001 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.001 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6811", 2024-09-18T07:32:19.001 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:32:19.001 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.001 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.001 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6809/1754684686", 2024-09-18T07:32:19.001 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6810/1754684686", 2024-09-18T07:32:19.001 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6812/1754684686", 2024-09-18T07:32:19.001 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6811/1754684686", 2024-09-18T07:32:19.001 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:32:19.001 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:32:19.002 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:32:19.002 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.002 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:32:19.002 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14", 2024-09-18T07:32:19.002 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:32:19.002 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:32:19.002 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:32:19.002 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:32:19.002 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:32:19.002 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:32:19.002 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 26, 2024-09-18T07:32:19.002 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 81, 2024-09-18T07:32:19.002 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:32:19.003 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:32:19.003 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:32:19.003 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.003 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.003 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6813", 2024-09-18T07:32:19.003 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:32:19.003 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.003 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.003 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:32:19.003 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6814", 2024-09-18T07:32:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:32:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:32:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6816", 2024-09-18T07:32:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:32:19.005 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.005 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.005 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:32:19.005 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.005 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.005 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6815", 2024-09-18T07:32:19.005 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:32:19.005 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.005 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6813/116058392", 2024-09-18T07:32:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6814/116058392", 2024-09-18T07:32:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6816/116058392", 2024-09-18T07:32:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6815/116058392", 2024-09-18T07:32:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:32:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:32:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:32:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:32:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de", 2024-09-18T07:32:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:32:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:32:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:32:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:32:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:32:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:32:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 32, 2024-09-18T07:32:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 81, 2024-09-18T07:32:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:32:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:32:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:32:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6800", 2024-09-18T07:32:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:32:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:32:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6801", 2024-09-18T07:32:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:32:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:32:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6803", 2024-09-18T07:32:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:32:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:32:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6802", 2024-09-18T07:32:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:32:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6800/3654297954", 2024-09-18T07:32:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6801/3654297954", 2024-09-18T07:32:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6803/3654297954", 2024-09-18T07:32:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6802/3654297954", 2024-09-18T07:32:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:32:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:32:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:32:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:32:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c", 2024-09-18T07:32:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:32:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:32:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:32:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:32:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:32:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:32:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 39, 2024-09-18T07:32:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 81, 2024-09-18T07:32:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:32:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:32:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:32:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6804", 2024-09-18T07:32:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:32:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:32:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6805", 2024-09-18T07:32:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:32:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:32:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6807", 2024-09-18T07:32:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:32:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:32:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6806", 2024-09-18T07:32:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:32:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6804/837994222", 2024-09-18T07:32:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6805/837994222", 2024-09-18T07:32:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6807/837994222", 2024-09-18T07:32:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6806/837994222", 2024-09-18T07:32:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:32:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:32:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:32:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:32:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6", 2024-09-18T07:32:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:32:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:32:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:32:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:32:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:32:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:32:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 47, 2024-09-18T07:32:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 81, 2024-09-18T07:32:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:32:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:32:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:32:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6808", 2024-09-18T07:32:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:32:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:32:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6809", 2024-09-18T07:32:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:32:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:32:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6811", 2024-09-18T07:32:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:32:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:32:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6810", 2024-09-18T07:32:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:32:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6808/1494175466", 2024-09-18T07:32:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6809/1494175466", 2024-09-18T07:32:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6811/1494175466", 2024-09-18T07:32:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6810/1494175466", 2024-09-18T07:32:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:32:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:32:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:32:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:32:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a", 2024-09-18T07:32:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:32:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:32:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:32:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:32:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:32:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:32:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 55, 2024-09-18T07:32:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 81, 2024-09-18T07:32:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:32:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:32:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:32:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6812", 2024-09-18T07:32:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:32:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:32:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6813", 2024-09-18T07:32:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:32:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:32:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6815", 2024-09-18T07:32:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:32:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:32:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:32:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:32:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6814", 2024-09-18T07:32:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:32:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6812/2670809315", 2024-09-18T07:32:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6813/2670809315", 2024-09-18T07:32:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6815/2670809315", 2024-09-18T07:32:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6814/2670809315", 2024-09-18T07:32:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:32:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:32:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:32:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:32:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_xinfo": [ 2024-09-18T07:32:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:32:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:32:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:32:19.026 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:32:19.026 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:32:19.026 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:32:19.026 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:31:18.629314+0000", 2024-09-18T07:32:19.026 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:32:19.026 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.026 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:32:19.026 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:32:19.026 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:32:19.026 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:32:19.026 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:32:19.026 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:32:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:31:43.978575+0000", 2024-09-18T07:32:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:32:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:32:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:32:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:32:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:32:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:32:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:32:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:31:59.214139+0000", 2024-09-18T07:32:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:32:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:32:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:32:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:32:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:32:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:32:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:32:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:31:02.992252+0000", 2024-09-18T07:32:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:32:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:32:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:32:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:32:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:32:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:32:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:32:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:31:36.464352+0000", 2024-09-18T07:32:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:32:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:32:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:32:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:32:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:32:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:32:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:32:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:30:53.837709+0000", 2024-09-18T07:32:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:32:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:32:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:32:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:32:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:32:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:32:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:32:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:31:27.885377+0000", 2024-09-18T07:32:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:32:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:32:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:32:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:32:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:32:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:32:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:32:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:31:44.706496+0000", 2024-09-18T07:32:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:32:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:32:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap": [], 2024-09-18T07:32:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_items": [], 2024-09-18T07:32:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_primaries": [], 2024-09-18T07:32:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_temp": [], 2024-09-18T07:32:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_temp": [], 2024-09-18T07:32:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklist": { 2024-09-18T07:32:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/2096039404": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:32:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/3636177939": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:32:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/334967443": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:32:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2101286111": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:32:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3387637412": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:32:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762989503": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:32:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3490817671": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:32:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1482147392": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:32:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1821482181": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:32:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/4028304023": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:32:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/621434521": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:32:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3160704935": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:32:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2357293499": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:32:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762352860": "2024-09-19T07:31:25.389124+0000" 2024-09-18T07:32:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "range_blocklist": {}, 2024-09-18T07:32:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profiles": { 2024-09-18T07:32:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "default": { 2024-09-18T07:32:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "crush-failure-domain": "osd", 2024-09-18T07:32:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "k": "2", 2024-09-18T07:32:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "m": "1", 2024-09-18T07:32:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin": "jerasure", 2024-09-18T07:32:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "technique": "reed_sol_van" 2024-09-18T07:32:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps_queue": [], 2024-09-18T07:32:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "new_removed_snaps": [], 2024-09-18T07:32:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "new_purged_snaps": [], 2024-09-18T07:32:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_node_flags": {}, 2024-09-18T07:32:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "device_class_flags": {}, 2024-09-18T07:32:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode": { 2024-09-18T07:32:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_enabled": false, 2024-09-18T07:32:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_bucket_count": 0, 2024-09-18T07:32:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: "degraded_stretch_mode": 0, 2024-09-18T07:32:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: "recovering_stretch_mode": 0, 2024-09-18T07:32:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_bucket": 0 2024-09-18T07:32:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:19.036 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:32:19.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:19.038 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:19.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:19.139 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:19.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:19.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:19.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:19.210 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:18 smithi203 bash[60789]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-18T07:32:19.211 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:19 smithi203 bash[60789]: Writing manifest to image destination 2024-09-18T07:32:19.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:19.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T07:32:19.317 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:19.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:19.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:19.410 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:19 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2836330857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:32:19.410 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:19 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2772536532' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:32:19.410 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:19 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2836330857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:32:19.411 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:19 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2772536532' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:32:19.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:19.445 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:19.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:19.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:19.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:19.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:19.597 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:19.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:19.698 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:19.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:19 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2836330857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:32:19.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:19 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2772536532' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:32:19.728 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:19.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:19.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:19.769 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 4 16 11525 11509 179.773 155.188 0.00442206 0.00555399 2024-09-18T07:32:19.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:19.874 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:19.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:19.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:20.015 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:20.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:20.117 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:20.147 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:20.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:20.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:20.248 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:20.289 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:20.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:20.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:20.391 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:20.421 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:20.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:20.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:20.523 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:20.549 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:20.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:20.651 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:20.681 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:20.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:20.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:20.770 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 5 16 14108 14092 176.104 161.438 0.0141705 0.00566521 2024-09-18T07:32:20.782 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:20.807 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:20.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:20.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:20.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:20.949 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:20.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:20.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:21.051 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:21.081 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:21.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:21.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:21.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:20 smithi137 ceph-mon[29408]: pgmap v44: 148 pgs: 148 active+clean; 115 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 667 B/s rd, 15 MiB/s wr, 238 op/s 2024-09-18T07:32:21.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:20 smithi137 ceph-mon[29408]: osdmap e88: 8 total, 8 up, 8 in 2024-09-18T07:32:21.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:20 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:20 smithi137 ceph-mon[39175]: pgmap v44: 148 pgs: 148 active+clean; 115 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 667 B/s rd, 15 MiB/s wr, 238 op/s 2024-09-18T07:32:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:20 smithi137 ceph-mon[39175]: osdmap e88: 8 total, 8 up, 8 in 2024-09-18T07:32:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:20 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:21.183 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:21.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:20 smithi203 ceph-mon[32237]: pgmap v44: 148 pgs: 148 active+clean; 115 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 667 B/s rd, 15 MiB/s wr, 238 op/s 2024-09-18T07:32:21.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:20 smithi203 ceph-mon[32237]: osdmap e88: 8 total, 8 up, 8 in 2024-09-18T07:32:21.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:20 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:21.213 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:21.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:21.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:21.314 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:21.348 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:21.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:21.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:21.415 INFO:tasks.thrashosds.thrasher:j is {'epoch': 87, 'fsid': 'f7ef099c-758e-11ef-bceb-c7b262605968', 'created': '2024-09-18T07:25:19.662367+0000', 'modified': '2024-09-18T07:32:17.319195+0000', 'last_up_change': '2024-09-18T07:30:44.708496+0000', 'last_in_change': '2024-09-18T07:30:23.052551+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T07:28:20.899655+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-18T07:30:53.715373+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-18T07:30:54.672127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-18T07:30:56.351343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-18T07:30:58.322401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-18T07:31:00.390499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-18T07:32:06.616722+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '268cd580-c931-4992-bb93-d6f63c2bfe88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6801', 'nonce': 2811444477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6802', 'nonce': 2811444477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6804', 'nonce': 2811444477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6803', 'nonce': 2811444477}]}, 'public_addr': '172.21.15.137:6801/2811444477', 'cluster_addr': '172.21.15.137:6802/2811444477', 'heartbeat_back_addr': '172.21.15.137:6804/2811444477', 'heartbeat_front_addr': '172.21.15.137:6803/2811444477', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b06907bc-b499-46c0-8667-c5bdc0d85575', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6805', 'nonce': 874102487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6806', 'nonce': 874102487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6808', 'nonce': 874102487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6807', 'nonce': 874102487}]}, 'public_addr': '172.21.15.137:6805/874102487', 'cluster_addr': '172.21.15.137:6806/874102487', 'heartbeat_back_addr': '172.21.15.137:6808/874102487', 'heartbeat_front_addr': '172.21.15.137:6807/874102487', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '480ad6c5-0e0b-4405-bbe2-381682d00bae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6809', 'nonce': 1754684686}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6810', 'nonce': 1754684686}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6812', 'nonce': 1754684686}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6811', 'nonce': 1754684686}]}, 'public_addr': '172.21.15.137:6809/1754684686', 'cluster_addr': '172.21.15.137:6810/1754684686', 'heartbeat_back_addr': '172.21.15.137:6812/1754684686', 'heartbeat_front_addr': '172.21.15.137:6811/1754684686', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '178be3e0-f07b-4856-807b-2a1181e48a14', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6813', 'nonce': 116058392}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6814', 'nonce': 116058392}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6816', 'nonce': 116058392}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6815', 'nonce': 116058392}]}, 'public_addr': '172.21.15.137:6813/116058392', 'cluster_addr': '172.21.15.137:6814/116058392', 'heartbeat_back_addr': '172.21.15.137:6816/116058392', 'heartbeat_front_addr': '172.21.15.137:6815/116058392', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7547102c-951a-4077-a094-fb8a65bfe6de', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6800', 'nonce': 3654297954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6801', 'nonce': 3654297954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6803', 'nonce': 3654297954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6802', 'nonce': 3654297954}]}, 'public_addr': '172.21.15.203:6800/3654297954', 'cluster_addr': '172.21.15.203:6801/3654297954', 'heartbeat_back_addr': '172.21.15.203:6803/3654297954', 'heartbeat_front_addr': '172.21.15.203:6802/3654297954', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '49c17901-8102-48a4-9348-fa1bdb63032c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6804', 'nonce': 837994222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6805', 'nonce': 837994222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6807', 'nonce': 837994222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6806', 'nonce': 837994222}]}, 'public_addr': '172.21.15.203:6804/837994222', 'cluster_addr': '172.21.15.203:6805/837994222', 'heartbeat_back_addr': '172.21.15.203:6807/837994222', 'heartbeat_front_addr': '172.21.15.203:6806/837994222', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '75f0ead4-e981-4a15-b1c1-d83dce871ab6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6808', 'nonce': 1494175466}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6809', 'nonce': 1494175466}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6811', 'nonce': 1494175466}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6810', 'nonce': 1494175466}]}, 'public_addr': '172.21.15.203:6808/1494175466', 'cluster_addr': '172.21.15.203:6809/1494175466', 'heartbeat_back_addr': '172.21.15.203:6811/1494175466', 'heartbeat_front_addr': '172.21.15.203:6810/1494175466', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c0dbf299-6798-4b99-9a85-62b123971f2a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 55, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6812', 'nonce': 2670809315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6813', 'nonce': 2670809315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6815', 'nonce': 2670809315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6814', 'nonce': 2670809315}]}, 'public_addr': '172.21.15.203:6812/2670809315', 'cluster_addr': '172.21.15.203:6813/2670809315', 'heartbeat_back_addr': '172.21.15.203:6815/2670809315', 'heartbeat_front_addr': '172.21.15.203:6814/2670809315', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:31:18.629314+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:31:43.978575+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:31:59.214139+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:31:02.992252+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:31:36.464352+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:30:53.837709+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:31:27.885377+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:31:44.706496+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.137:6800/2096039404': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:6800/3636177939': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/334967443': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/2101286111': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/3387637412': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1762989503': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3490817671': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:6800/1482147392': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1821482181': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/4028304023': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/621434521': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3160704935': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/2357293499': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/1762352860': '2024-09-19T07:31:25.389124+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T07:32:21.416 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-18T07:32:21.449 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:21.498 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:21.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:21.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:21.514 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:21.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:21.640 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:21.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:21.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:21.742 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:21.769 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 6 16 16650 16634 173.232 158.875 0.00367588 0.00576579 2024-09-18T07:32:21.774 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:21.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:21.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:21.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:21.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:21.919 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:21.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:21.960 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:21 smithi203 podman[60789]: 2024-09-18 07:32:21.514835533 +0000 UTC m=+5.197851283 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-18T07:32:22.021 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:22.050 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:22.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:22.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:22.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:22.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:22.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:22.219 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:32:22.219 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:32:22.219 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.219 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:17769 7.b 7:d4eb51ac:::benchmark_data_smithi137_74180_object17768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T07:32:22.219 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:22.212345+0000", 2024-09-18T07:32:22.219 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0033062260000000002, 2024-09-18T07:32:22.219 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033445430000000002, 2024-09-18T07:32:22.219 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:22.220 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:22.220 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:32:22.220 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:22.220 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:22.220 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:22.220 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 17769 2024-09-18T07:32:22.220 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.220 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:22.220 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.220 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:22.220 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.212345+0000", 2024-09-18T07:32:22.220 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:22.220 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.221 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:22.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.212345+0000", 2024-09-18T07:32:22.221 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:22.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.221 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:22.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.212347+0000", 2024-09-18T07:32:22.221 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.12e-06 2024-09-18T07:32:22.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.221 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:22.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.212372+0000", 2024-09-18T07:32:22.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4088999999999999e-05 2024-09-18T07:32:22.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:22.222 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.212386+0000", 2024-09-18T07:32:22.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4714e-05 2024-09-18T07:32:22.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:22.222 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.212395+0000", 2024-09-18T07:32:22.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3240000000000003e-06 2024-09-18T07:32:22.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.223 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:22.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.213234+0000", 2024-09-18T07:32:22.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00083898600000000001 2024-09-18T07:32:22.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.223 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:22.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.213346+0000", 2024-09-18T07:32:22.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011246100000000001 2024-09-18T07:32:22.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.223 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:22.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.213577+0000", 2024-09-18T07:32:22.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023105900000000001 2024-09-18T07:32:22.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.224 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:22.224 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.214751+0000", 2024-09-18T07:32:22.224 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011740609999999999 2024-09-18T07:32:22.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.224 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:22.224 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.214874+0000", 2024-09-18T07:32:22.224 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000122793 2024-09-18T07:32:22.224 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:22.224 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:22.224 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:22.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.225 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:17771 7.9 e88/81 7:9a8d973f:::benchmark_data_smithi137_74180_object17770:head v 88'1152, mlcod=88'1149)", 2024-09-18T07:32:22.225 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:22.214939+0000", 2024-09-18T07:32:22.225 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00071237099999999995, 2024-09-18T07:32:22.225 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00082998800000000004, 2024-09-18T07:32:22.225 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:22.225 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:22.225 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:32:22.225 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:22.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:22.225 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.214939+0000", 2024-09-18T07:32:22.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:22.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:22.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.214939+0000", 2024-09-18T07:32:22.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:22.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:22.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.214940+0000", 2024-09-18T07:32:22.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2750000000000001e-06 2024-09-18T07:32:22.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:22.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.214963+0000", 2024-09-18T07:32:22.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2898999999999999e-05 2024-09-18T07:32:22.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.228 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:22.228 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.214971+0000", 2024-09-18T07:32:22.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8460000000000008e-06 2024-09-18T07:32:22.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.228 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:22.228 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.214977+0000", 2024-09-18T07:32:22.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8760000000000002e-06 2024-09-18T07:32:22.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.228 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:22.228 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.215053+0000", 2024-09-18T07:32:22.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5895000000000004e-05 2024-09-18T07:32:22.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.229 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:22.229 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.215072+0000", 2024-09-18T07:32:22.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8536999999999999e-05 2024-09-18T07:32:22.229 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:22.229 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:22.229 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:22.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.229 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.14865.0:17769 7.b e88/81 ondisk, result = 0)", 2024-09-18T07:32:22.230 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:22.214965+0000", 2024-09-18T07:32:22.230 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00068701999999999999, 2024-09-18T07:32:22.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00084860999999999997, 2024-09-18T07:32:22.230 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:22.230 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:22.230 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:32:22.230 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:22.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:22.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.214965+0000", 2024-09-18T07:32:22.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:22.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:22.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.214965+0000", 2024-09-18T07:32:22.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:22.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:22.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.214965+0000", 2024-09-18T07:32:22.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3799999999999996e-07 2024-09-18T07:32:22.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:22.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.214969+0000", 2024-09-18T07:32:22.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7919999999999999e-06 2024-09-18T07:32:22.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:22.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.214986+0000", 2024-09-18T07:32:22.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6640999999999999e-05 2024-09-18T07:32:22.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:22.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.214992+0000", 2024-09-18T07:32:22.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8420000000000001e-06 2024-09-18T07:32:22.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:22.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.215733+0000", 2024-09-18T07:32:22.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000741619 2024-09-18T07:32:22.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:22.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.215739+0000", 2024-09-18T07:32:22.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0100000000000001e-06 2024-09-18T07:32:22.234 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:22.234 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:22.234 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:22.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.234 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:17774 7.d e88/81 7:b7ff102f:::benchmark_data_smithi137_74180_object17773:head v 88'1164, mlcod=88'1161)", 2024-09-18T07:32:22.234 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:22.215258+0000", 2024-09-18T07:32:22.234 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00039368699999999999, 2024-09-18T07:32:22.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00059924200000000005, 2024-09-18T07:32:22.235 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:22.235 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:22.235 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:32:22.235 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:22.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:22.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.215258+0000", 2024-09-18T07:32:22.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:22.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:22.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.215258+0000", 2024-09-18T07:32:22.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:22.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:22.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.215259+0000", 2024-09-18T07:32:22.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0210000000000001e-06 2024-09-18T07:32:22.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:22.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.215323+0000", 2024-09-18T07:32:22.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3800000000000006e-05 2024-09-18T07:32:22.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:22.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.215335+0000", 2024-09-18T07:32:22.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2524000000000001e-05 2024-09-18T07:32:22.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:22.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.215343+0000", 2024-09-18T07:32:22.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4410000000000004e-06 2024-09-18T07:32:22.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.238 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.238 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:22.238 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.215392+0000", 2024-09-18T07:32:22.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9274000000000003e-05 2024-09-18T07:32:22.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:22.238 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:22.238 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:22.238 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.215412+0000", 2024-09-18T07:32:22.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9862000000000001e-05 2024-09-18T07:32:22.238 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:22.238 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:22.238 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:22.239 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:22.239 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:32:22.239 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 4 2024-09-18T07:32:22.239 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:32:22.292 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:22.325 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:22.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:22.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:22.469 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:22.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:22.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:22.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:22.610 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:22.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:22.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:22.712 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:22.742 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:22.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:22.769 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 7 16 19158 19142 170.875 156.75 0.0365849 0.00583645 2024-09-18T07:32:22.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:32:22.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:22.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:22.889 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:22.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:22.990 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:23.012 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:23.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:23.114 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:23.155 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:23.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:23.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:23.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:22 smithi137 ceph-mon[29408]: pgmap v46: 148 pgs: 148 active+clean; 930 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 139 MiB/s wr, 2.22k op/s 2024-09-18T07:32:23.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:22 smithi137 ceph-mon[39175]: pgmap v46: 148 pgs: 148 active+clean; 930 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 139 MiB/s wr, 2.22k op/s 2024-09-18T07:32:23.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:22 smithi203 ceph-mon[32237]: pgmap v46: 148 pgs: 148 active+clean; 930 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 139 MiB/s wr, 2.22k op/s 2024-09-18T07:32:23.210 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:22 smithi203 podman[60789]: 2024-09-18 07:32:22.896713269 +0000 UTC m=+6.579729006 container create f2e1665435ce9d4ebbc72b082a8b99eedebbf498e8bf930711fbf0306505ef68 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-18T07:32:23.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:23.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:32:23.298 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:23.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:23.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:23.400 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:23.429 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:23.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:23.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:23.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:23.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:23.557 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:23.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:23.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:23.685 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:23.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:23.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:23.769 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 8 16 20978 20962 163.734 113.75 0.0101972 0.0061019 2024-09-18T07:32:23.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:23.826 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:23.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:23.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:23.928 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:23.951 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:23.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:23.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:24.052 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:24.085 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:24.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:24.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:24.186 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:24.207 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:24.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:24.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:24.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:24.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:24.331 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:24.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:24.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:24.452 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:24.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:24.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:24.554 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:24.587 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:24.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:24.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:24.689 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:24.710 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:24.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:24.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:24.770 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 9 16 22353 22337 155.09 85.9375 0.00263081 0.00636462 2024-09-18T07:32:24.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:24.834 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:24.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:24.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:24.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:24.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:24.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:25.062 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:25.085 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:25.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:25.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:25.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:24 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:32:25.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:24 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:32:25.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:24 smithi137 ceph-mon[39175]: pgmap v47: 148 pgs: 148 active+clean; 983 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.96k op/s 2024-09-18T07:32:25.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:24 smithi137 ceph-mon[39175]: osdmap e89: 8 total, 8 up, 8 in 2024-09-18T07:32:25.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:24 smithi137 ceph-mon[39175]: osdmap e90: 8 total, 8 up, 8 in 2024-09-18T07:32:25.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:24 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:32:25.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:24 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:32:25.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:24 smithi137 ceph-mon[29408]: pgmap v47: 148 pgs: 148 active+clean; 983 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.96k op/s 2024-09-18T07:32:25.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:24 smithi137 ceph-mon[29408]: osdmap e89: 8 total, 8 up, 8 in 2024-09-18T07:32:25.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:24 smithi137 ceph-mon[29408]: osdmap e90: 8 total, 8 up, 8 in 2024-09-18T07:32:25.186 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:25.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:24 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:32:25.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:24 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:32:25.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:24 smithi203 ceph-mon[32237]: pgmap v47: 148 pgs: 148 active+clean; 983 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.96k op/s 2024-09-18T07:32:25.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:24 smithi203 ceph-mon[32237]: osdmap e89: 8 total, 8 up, 8 in 2024-09-18T07:32:25.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:24 smithi203 ceph-mon[32237]: osdmap e90: 8 total, 8 up, 8 in 2024-09-18T07:32:25.216 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:25.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:25.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:25.317 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:25.343 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:25.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:25.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:25.445 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:25.472 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:25.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:25.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:25.574 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:25.595 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:25.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:25.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:25.696 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:25.722 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:25.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:25.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:25.770 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 10 16 23451 23435 146.441 68.625 0.008345 0.00682086 2024-09-18T07:32:25.823 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:25.858 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:25.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:25.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:25.959 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:25.993 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:25.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:25.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:26.095 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:26.126 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:26.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:26.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:26.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:25 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-18T07:32:26.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:25 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-18T07:32:26.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:25 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-18T07:32:26.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:25 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-18T07:32:26.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:25 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-18T07:32:26.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:25 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-18T07:32:26.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:25 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-18T07:32:26.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:25 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-18T07:32:26.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:25 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-18T07:32:26.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:25 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-18T07:32:26.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:25 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-18T07:32:26.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:25 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-18T07:32:26.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-18T07:32:26.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-18T07:32:26.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-18T07:32:26.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-18T07:32:26.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-18T07:32:26.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-18T07:32:26.211 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 podman[60789]: 2024-09-18 07:32:25.810076705 +0000 UTC m=+9.493092442 container init f2e1665435ce9d4ebbc72b082a8b99eedebbf498e8bf930711fbf0306505ef68 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-18T07:32:26.211 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 podman[60789]: 2024-09-18 07:32:25.818653304 +0000 UTC m=+9.501669039 container start f2e1665435ce9d4ebbc72b082a8b99eedebbf498e8bf930711fbf0306505ef68 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-18T07:32:26.211 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.821Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-18T07:32:26.212 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.821Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-18T07:32:26.212 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-18T07:32:26.212 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-18T07:32:26.212 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-18T07:32:26.212 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-18T07:32:26.212 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-18T07:32:26.212 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=arp 2024-09-18T07:32:26.212 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-18T07:32:26.212 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-18T07:32:26.212 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-18T07:32:26.212 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-18T07:32:26.213 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-18T07:32:26.213 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-18T07:32:26.213 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-18T07:32:26.213 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-18T07:32:26.213 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=edac 2024-09-18T07:32:26.213 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-18T07:32:26.214 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-18T07:32:26.214 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-18T07:32:26.214 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-18T07:32:26.214 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-18T07:32:26.214 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-18T07:32:26.214 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-18T07:32:26.214 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-18T07:32:26.214 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-18T07:32:26.214 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-18T07:32:26.214 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-18T07:32:26.214 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-18T07:32:26.214 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-18T07:32:26.214 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-18T07:32:26.215 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.823Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-18T07:32:26.215 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-18T07:32:26.215 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=os 2024-09-18T07:32:26.215 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-18T07:32:26.215 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-18T07:32:26.215 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-18T07:32:26.215 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-18T07:32:26.215 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-18T07:32:26.215 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-18T07:32:26.215 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-18T07:32:26.215 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=stat 2024-09-18T07:32:26.215 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-18T07:32:26.215 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-18T07:32:26.216 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-18T07:32:26.216 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=time 2024-09-18T07:32:26.216 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-18T07:32:26.216 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=uname 2024-09-18T07:32:26.216 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-18T07:32:26.216 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-18T07:32:26.216 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-18T07:32:26.216 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-18T07:32:26.216 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b[61358]: ts=2024-09-18T07:32:25.824Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-18T07:32:26.216 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 bash[60789]: f2e1665435ce9d4ebbc72b082a8b99eedebbf498e8bf930711fbf0306505ef68 2024-09-18T07:32:26.216 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:32:25 smithi203 systemd[1]: Started Ceph node-exporter.b for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:32:26.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:26.248 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:26.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:26.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:26.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:26.391 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:26.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:26.417 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:32:26.417 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:32:26.417 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:32:26.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd out 7 2024-09-18T07:32:26.492 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:26.525 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:26.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:26.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:26.627 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:26.652 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:26.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:26.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:26.755 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:26.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[39175]: pgmap v50: 148 pgs: 148 active+clean; 983 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 145 MiB/s wr, 2.31k op/s 2024-09-18T07:32:26.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-09-18T07:32:26.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-18T07:32:26.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-18T07:32:26.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[39175]: osdmap e91: 8 total, 8 up, 8 in 2024-09-18T07:32:26.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:26.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:32:26.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:26.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:26.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:26.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:26.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:26.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:32:26.770 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 11 16 25491 25475 144.718 127.5 0.00653139 0.00690237 2024-09-18T07:32:26.784 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:26.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:26.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:26.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:26.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:26.920 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:26.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:27.021 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:27.051 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:27.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:27.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:27.100 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[29408]: pgmap v50: 148 pgs: 148 active+clean; 983 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 145 MiB/s wr, 2.31k op/s 2024-09-18T07:32:27.100 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-09-18T07:32:27.100 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-18T07:32:27.100 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-18T07:32:27.100 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[29408]: osdmap e91: 8 total, 8 up, 8 in 2024-09-18T07:32:27.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:27.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:32:27.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:27.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:27.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:27.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:27.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:27.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:32:27.153 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:27.182 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:27.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:27.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:27.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:32:26 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:32:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:26 smithi203 ceph-mon[32237]: pgmap v50: 148 pgs: 148 active+clean; 983 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 145 MiB/s wr, 2.31k op/s 2024-09-18T07:32:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-09-18T07:32:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-18T07:32:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-18T07:32:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:26 smithi203 ceph-mon[32237]: osdmap e91: 8 total, 8 up, 8 in 2024-09-18T07:32:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:32:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:32:27.285 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:27.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:27.325 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:27.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:27.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:32:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:32:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:32:27.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:27.448 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:27.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:27.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:27.458 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T07:32:27.551 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:27.579 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:27.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:27.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:27.668 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:27.674 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:27.681 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:27.759 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:27.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:27.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:27.767 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:27.770 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 12 16 27300 27284 142.079 113.062 0.0123409 0.00703059 2024-09-18T07:32:27.861 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:27.889 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:27.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:27.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:27.991 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:28.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:28.020 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:28.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:28.041 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:27 smithi137 ceph-mon[29408]: osdmap e92: 8 total, 8 up, 8 in 2024-09-18T07:32:28.041 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:27 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:28.041 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:27 smithi137 ceph-mon[39175]: osdmap e92: 8 total, 8 up, 8 in 2024-09-18T07:32:28.041 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:27 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:28.121 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:28.133 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:27 smithi203 ceph-mon[32237]: osdmap e92: 8 total, 8 up, 8 in 2024-09-18T07:32:28.133 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:27 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:28.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:28.161 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:28.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:28.262 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:28.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:28.288 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:28.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:28.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:28.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:28.429 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:28.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:28.530 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:28.561 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:28.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:28.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:28.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:28.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:28.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:28.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:28.770 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 13 15 29324 29309 140.885 126.562 0.0113346 0.00709315 2024-09-18T07:32:28.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:28.845 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:28.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:28.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:28.897 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:28 smithi137 ceph-mon[29408]: pgmap v53: 148 pgs: 3 peering, 145 active+clean; 1.5 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 141 MiB/s wr, 2.26k op/s 2024-09-18T07:32:28.898 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:28 smithi137 ceph-mon[39175]: pgmap v53: 148 pgs: 3 peering, 145 active+clean; 1.5 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 141 MiB/s wr, 2.26k op/s 2024-09-18T07:32:28.947 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:28.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:28 smithi203 ceph-mon[32237]: pgmap v53: 148 pgs: 3 peering, 145 active+clean; 1.5 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 141 MiB/s wr, 2.26k op/s 2024-09-18T07:32:28.976 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:28.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:28.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:29.077 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:29.104 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:29.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:29.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:29.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:29.246 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:29.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:29.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:29.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:29.352 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:32:29.353 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:32:29.369 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:32:29.369 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:32:29.391 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:29.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:29.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:29.406 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:32:29.406 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:32:29.441 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:32:29.442 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:32:29.442 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:29 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a[29404]: 2024-09-18T07:32:29.181+0000 7fe5973a1640 -1 mon.a@0(leader).osd e92 definitely_dead 0 2024-09-18T07:32:29.475 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:32:29.475 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:32:29.492 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:29.510 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:32:29.510 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:32:29.519 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:29.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:29.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:29.553 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:32:29.553 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:32:29.590 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:32:29.590 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:32:29.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:29.662 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:29.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:29.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:29.764 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:29.770 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 14 16 31051 31035 138.526 107.875 0.00853245 0.00720968 2024-09-18T07:32:29.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:29.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:29.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:29.824 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:32:29.835 INFO:teuthology.orchestra.run.smithi137.stderr:marked out osd.7. 2024-09-18T07:32:29.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:29.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:29.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:29.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:30.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:30.056 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:30.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:30.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:30.104 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:29 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1218524781' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-18T07:32:30.104 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:29 smithi203 ceph-mon[32237]: from='client.34760 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-18T07:32:30.104 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:29 smithi203 ceph-mon[32237]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-18T07:32:30.105 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:29 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1538633320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:32:30.105 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:29 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1674514007' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:32:30.105 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:29 smithi203 ceph-mon[32237]: from='client.24787 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:32:30.157 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:30.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:29 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1218524781' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-18T07:32:30.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:29 smithi137 ceph-mon[29408]: from='client.34760 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-18T07:32:30.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:29 smithi137 ceph-mon[29408]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-18T07:32:30.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:29 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1538633320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:32:30.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:29 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1674514007' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:32:30.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:29 smithi137 ceph-mon[29408]: from='client.24787 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:32:30.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:29 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1218524781' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-18T07:32:30.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:29 smithi137 ceph-mon[39175]: from='client.34760 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-18T07:32:30.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:29 smithi137 ceph-mon[39175]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-18T07:32:30.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:29 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1538633320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:32:30.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:29 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1674514007' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:32:30.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:29 smithi137 ceph-mon[39175]: from='client.24787 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:32:30.179 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:30.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:30.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:30.281 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:30.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:30.302 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:30.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:30.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:30.425 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:30.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:30.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:30.522 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:30.527 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:30.549 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:30.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:30.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:30.651 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:30.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:30.674 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:30.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:30.770 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 15 16 31172 31156 129.796 7.5625 0.00289428 0.00722666 2024-09-18T07:32:30.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:30.796 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:30.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:30.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:30.898 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:30.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:30 smithi137 ceph-mon[39175]: pgmap v54: 148 pgs: 3 peering, 145 active+clean; 1.5 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 903 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-18T07:32:30.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:30 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:32:30.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:30 smithi137 ceph-mon[39175]: from='client.34760 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-18T07:32:30.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:30 smithi137 ceph-mon[39175]: from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:32:30.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:30 smithi137 ceph-mon[39175]: osdmap e93: 8 total, 8 up, 7 in 2024-09-18T07:32:30.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:30 smithi137 ceph-mon[39175]: 4.1c scrub starts 2024-09-18T07:32:30.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:30 smithi137 ceph-mon[39175]: 4.1c scrub ok 2024-09-18T07:32:30.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:30 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:30 smithi137 ceph-mon[29408]: pgmap v54: 148 pgs: 3 peering, 145 active+clean; 1.5 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 903 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-18T07:32:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:30 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:32:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:30 smithi137 ceph-mon[29408]: from='client.34760 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-18T07:32:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:30 smithi137 ceph-mon[29408]: from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:32:30.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:30 smithi137 ceph-mon[29408]: osdmap e93: 8 total, 8 up, 7 in 2024-09-18T07:32:30.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:30 smithi137 ceph-mon[29408]: 4.1c scrub starts 2024-09-18T07:32:30.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:30 smithi137 ceph-mon[29408]: 4.1c scrub ok 2024-09-18T07:32:30.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:30 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:30.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:30.922 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:30.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:31.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:31.045 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:31.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:31.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:31.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:31.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:31.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:31.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:31.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:30 smithi203 ceph-mon[32237]: pgmap v54: 148 pgs: 3 peering, 145 active+clean; 1.5 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 903 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-18T07:32:31.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:30 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:32:31.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:30 smithi203 ceph-mon[32237]: from='client.34760 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-18T07:32:31.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:30 smithi203 ceph-mon[32237]: from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:32:31.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:30 smithi203 ceph-mon[32237]: osdmap e93: 8 total, 8 up, 7 in 2024-09-18T07:32:31.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:30 smithi203 ceph-mon[32237]: 4.1c scrub starts 2024-09-18T07:32:31.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:30 smithi203 ceph-mon[32237]: 4.1c scrub ok 2024-09-18T07:32:31.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:30 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:31.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:31.291 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:31.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:31.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:31.393 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:31.419 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:31.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:31.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:31.509 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:32:31.510 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:32:31.521 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:31.547 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:31.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:31.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:31.649 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:31.670 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:31.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:31.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:31.770 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 16 16 31172 31156 121.684 0 - 0.00722666 2024-09-18T07:32:31.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:31.793 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:31.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:31.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:31.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:31.927 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:31.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:31.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:31 smithi203 ceph-mon[32237]: 6.19 scrub starts 2024-09-18T07:32:31.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:31 smithi203 ceph-mon[32237]: 6.19 scrub ok 2024-09-18T07:32:31.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:31 smithi203 ceph-mon[32237]: osdmap e94: 8 total, 8 up, 7 in 2024-09-18T07:32:31.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:31 smithi203 ceph-mon[32237]: 5.19 scrub starts 2024-09-18T07:32:31.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:31 smithi203 ceph-mon[32237]: 5.19 scrub ok 2024-09-18T07:32:32.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:32.053 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:32.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:32.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:32.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:32.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:31 smithi137 ceph-mon[29408]: 6.19 scrub starts 2024-09-18T07:32:32.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:31 smithi137 ceph-mon[29408]: 6.19 scrub ok 2024-09-18T07:32:32.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:31 smithi137 ceph-mon[29408]: osdmap e94: 8 total, 8 up, 7 in 2024-09-18T07:32:32.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:31 smithi137 ceph-mon[29408]: 5.19 scrub starts 2024-09-18T07:32:32.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:31 smithi137 ceph-mon[29408]: 5.19 scrub ok 2024-09-18T07:32:32.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:31 smithi137 ceph-mon[39175]: 6.19 scrub starts 2024-09-18T07:32:32.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:31 smithi137 ceph-mon[39175]: 6.19 scrub ok 2024-09-18T07:32:32.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:31 smithi137 ceph-mon[39175]: osdmap e94: 8 total, 8 up, 7 in 2024-09-18T07:32:32.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:31 smithi137 ceph-mon[39175]: 5.19 scrub starts 2024-09-18T07:32:32.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:31 smithi137 ceph-mon[39175]: 5.19 scrub ok 2024-09-18T07:32:32.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:32.199 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:32.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:32.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:32.341 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:32.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:32.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:32.443 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:32.471 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:32.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:32.571 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:32.597 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:32.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:32.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:32.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:32.739 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:32.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:32.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:32.770 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 17 16 32097 32081 117.927 28.9062 0.006931 0.0084729 2024-09-18T07:32:32.841 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:32.884 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:32.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:32.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:33.029 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:33.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:33.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:33.077 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:32:33.114 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:32 smithi137 ceph-mon[29408]: pgmap v57: 148 pgs: 3 remapped+peering, 145 active+clean; 1.8 GiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 149 MiB/s wr, 2.38k op/s; 272 B/s, 12 objects/s recovering 2024-09-18T07:32:33.114 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:32 smithi137 ceph-mon[29408]: 5.11 scrub starts 2024-09-18T07:32:33.115 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:32 smithi137 ceph-mon[29408]: 5.11 scrub ok 2024-09-18T07:32:33.115 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:32 smithi137 ceph-mon[29408]: osdmap e95: 8 total, 8 up, 7 in 2024-09-18T07:32:33.115 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:32 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:32:33.115 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:32 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:32:33.115 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:32 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:33.115 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:32 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:33.115 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:32 smithi137 ceph-mon[39175]: pgmap v57: 148 pgs: 3 remapped+peering, 145 active+clean; 1.8 GiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 149 MiB/s wr, 2.38k op/s; 272 B/s, 12 objects/s recovering 2024-09-18T07:32:33.115 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:32 smithi137 ceph-mon[39175]: 5.11 scrub starts 2024-09-18T07:32:33.115 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:32 smithi137 ceph-mon[39175]: 5.11 scrub ok 2024-09-18T07:32:33.115 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:32 smithi137 ceph-mon[39175]: osdmap e95: 8 total, 8 up, 7 in 2024-09-18T07:32:33.115 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:32 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:32:33.116 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:32 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:32:33.116 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:32 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:33.116 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:32 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:33.130 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:33.157 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:33.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:33.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:33.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:32 smithi203 ceph-mon[32237]: pgmap v57: 148 pgs: 3 remapped+peering, 145 active+clean; 1.8 GiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 149 MiB/s wr, 2.38k op/s; 272 B/s, 12 objects/s recovering 2024-09-18T07:32:33.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:32 smithi203 ceph-mon[32237]: 5.11 scrub starts 2024-09-18T07:32:33.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:32 smithi203 ceph-mon[32237]: 5.11 scrub ok 2024-09-18T07:32:33.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:32 smithi203 ceph-mon[32237]: osdmap e95: 8 total, 8 up, 7 in 2024-09-18T07:32:33.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:32 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:32:33.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:32 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:32:33.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:32 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:33.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:32 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:33.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:33.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:33.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:33.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T07:32:33.406 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:33.461 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:33.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:33.563 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:33.588 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:33.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:33.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:33.689 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:33.716 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:33.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:33.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:33.771 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 18 16 33297 33281 115.538 75 0.0111605 0.00864893 2024-09-18T07:32:33.818 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:33.842 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:33.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:33.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:33.944 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:33.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:33.969 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:33.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:34.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:34.106 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:34.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:34.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:34.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:33 smithi137 ceph-mon[29408]: osdmap e96: 8 total, 8 up, 7 in 2024-09-18T07:32:34.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:33 smithi137 ceph-mon[39175]: osdmap e96: 8 total, 8 up, 7 in 2024-09-18T07:32:34.208 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:34.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:33 smithi203 ceph-mon[32237]: osdmap e96: 8 total, 8 up, 7 in 2024-09-18T07:32:34.234 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:34.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:34.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:34.336 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:34.361 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:34.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:34.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:34.463 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:34.502 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:34.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:34.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:34.604 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:34.679 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:34.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:34.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:34.709 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:34.741 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:34.776 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 19 16 34396 34380 113.052 68.6875 0.00999962 0.00883242 2024-09-18T07:32:34.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:34.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:34.825 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:34.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:34.862 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:34 smithi137 ceph-mon[29408]: pgmap v60: 148 pgs: 5 remapped+peering, 143 active+clean; 1.9 GiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s; 145 KiB/s, 21 objects/s recovering 2024-09-18T07:32:34.926 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:34.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:34.953 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:34.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:35.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:35.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:35.106 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:35.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:35.133 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:34 smithi137 ceph-mon[39175]: pgmap v60: 148 pgs: 5 remapped+peering, 143 active+clean; 1.9 GiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s; 145 KiB/s, 21 objects/s recovering 2024-09-18T07:32:35.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:35.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:34 smithi203 ceph-mon[32237]: pgmap v60: 148 pgs: 5 remapped+peering, 143 active+clean; 1.9 GiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s; 145 KiB/s, 21 objects/s recovering 2024-09-18T07:32:35.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:35.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:35.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:35.350 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:35.374 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:35.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:35.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:35.476 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:35.530 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:35.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:35.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:35.632 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:35.672 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:35.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:35.738 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:32:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:32:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:32:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:32:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:19139 7.b 7:d685af6f:::benchmark_data_smithi137_74180_object19138:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T07:32:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:22.731330+0000", 2024-09-18T07:32:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.995159176, 2024-09-18T07:32:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.034871070999999997, 2024-09-18T07:32:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 19139 2024-09-18T07:32:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.731330+0000", 2024-09-18T07:32:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.731330+0000", 2024-09-18T07:32:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.731333+0000", 2024-09-18T07:32:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0779999999999998e-06 2024-09-18T07:32:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.731355+0000", 2024-09-18T07:32:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2021000000000001e-05 2024-09-18T07:32:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.731369+0000", 2024-09-18T07:32:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4426999999999999e-05 2024-09-18T07:32:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.731376+0000", 2024-09-18T07:32:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9029999999999998e-06 2024-09-18T07:32:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.731472+0000", 2024-09-18T07:32:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5808000000000002e-05 2024-09-18T07:32:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.731573+0000", 2024-09-18T07:32:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101197 2024-09-18T07:32:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.731692+0000", 2024-09-18T07:32:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000119256 2024-09-18T07:32:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.734047+0000", 2024-09-18T07:32:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023546230000000001 2024-09-18T07:32:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.734058+0000", 2024-09-18T07:32:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0773e-05 2024-09-18T07:32:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.766112+0000", 2024-09-18T07:32:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032054605 2024-09-18T07:32:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.766154+0000", 2024-09-18T07:32:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1346e-05 2024-09-18T07:32:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.766202+0000", 2024-09-18T07:32:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8034000000000001e-05 2024-09-18T07:32:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:19140 7.b 7:d2c56615:::benchmark_data_smithi137_74180_object19139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T07:32:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:22.731397+0000", 2024-09-18T07:32:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.995092940999999, 2024-09-18T07:32:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.036294370999999999, 2024-09-18T07:32:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 19140 2024-09-18T07:32:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.731397+0000", 2024-09-18T07:32:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.731397+0000", 2024-09-18T07:32:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.731397+0000", 2024-09-18T07:32:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0399999999999995e-07 2024-09-18T07:32:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.731417+0000", 2024-09-18T07:32:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9111e-05 2024-09-18T07:32:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.731423+0000", 2024-09-18T07:32:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0249999999999999e-06 2024-09-18T07:32:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.731427+0000", 2024-09-18T07:32:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8600000000000001e-06 2024-09-18T07:32:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.732884+0000", 2024-09-18T07:32:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014566480000000001 2024-09-18T07:32:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.732992+0000", 2024-09-18T07:32:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107879 2024-09-18T07:32:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.733116+0000", 2024-09-18T07:32:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012402100000000001 2024-09-18T07:32:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.734132+0000", 2024-09-18T07:32:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010162599999999999 2024-09-18T07:32:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.766415+0000", 2024-09-18T07:32:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032282475999999997 2024-09-18T07:32:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.767588+0000", 2024-09-18T07:32:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001173477 2024-09-18T07:32:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.767643+0000", 2024-09-18T07:32:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4701999999999999e-05 2024-09-18T07:32:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:22.767691+0000", 2024-09-18T07:32:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8208000000000003e-05 2024-09-18T07:32:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:20747 7.b 7:d187519a:::benchmark_data_smithi137_74180_object20746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T07:32:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:23.526769+0000", 2024-09-18T07:32:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.199720718, 2024-09-18T07:32:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10768604800000001, 2024-09-18T07:32:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 20747 2024-09-18T07:32:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526769+0000", 2024-09-18T07:32:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526769+0000", 2024-09-18T07:32:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526771+0000", 2024-09-18T07:32:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2510000000000001e-06 2024-09-18T07:32:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526805+0000", 2024-09-18T07:32:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3915000000000003e-05 2024-09-18T07:32:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526821+0000", 2024-09-18T07:32:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6308e-05 2024-09-18T07:32:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526829+0000", 2024-09-18T07:32:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.481e-06 2024-09-18T07:32:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526888+0000", 2024-09-18T07:32:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9434000000000001e-05 2024-09-18T07:32:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.527009+0000", 2024-09-18T07:32:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012099699999999999 2024-09-18T07:32:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.527125+0000", 2024-09-18T07:32:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116107 2024-09-18T07:32:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.528610+0000", 2024-09-18T07:32:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014843829999999999 2024-09-18T07:32:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.529010+0000", 2024-09-18T07:32:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040065400000000002 2024-09-18T07:32:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.634285+0000", 2024-09-18T07:32:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10527474100000001 2024-09-18T07:32:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.634407+0000", 2024-09-18T07:32:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012228099999999999 2024-09-18T07:32:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.634455+0000", 2024-09-18T07:32:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7496000000000001e-05 2024-09-18T07:32:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:20767 7.b 7:d7fc5ec8:::benchmark_data_smithi137_74180_object20766:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T07:32:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:23.539452+0000", 2024-09-18T07:32:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.187037909000001, 2024-09-18T07:32:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.095136148000000004, 2024-09-18T07:32:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 20767 2024-09-18T07:32:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539452+0000", 2024-09-18T07:32:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539452+0000", 2024-09-18T07:32:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539454+0000", 2024-09-18T07:32:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.903e-06 2024-09-18T07:32:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539480+0000", 2024-09-18T07:32:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6180999999999999e-05 2024-09-18T07:32:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539493+0000", 2024-09-18T07:32:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.323e-05 2024-09-18T07:32:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539501+0000", 2024-09-18T07:32:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2139999999999996e-06 2024-09-18T07:32:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539554+0000", 2024-09-18T07:32:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2920000000000002e-05 2024-09-18T07:32:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539632+0000", 2024-09-18T07:32:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8258999999999997e-05 2024-09-18T07:32:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539713+0000", 2024-09-18T07:32:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0840999999999994e-05 2024-09-18T07:32:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.540592+0000", 2024-09-18T07:32:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00087853499999999995 2024-09-18T07:32:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.540750+0000", 2024-09-18T07:32:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000158517 2024-09-18T07:32:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.634503+0000", 2024-09-18T07:32:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.093752775999999996 2024-09-18T07:32:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.634552+0000", 2024-09-18T07:32:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8874e-05 2024-09-18T07:32:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.634588+0000", 2024-09-18T07:32:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5898000000000001e-05 2024-09-18T07:32:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:20776 7.b 7:dc6a0efb:::benchmark_data_smithi137_74180_object20775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T07:32:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:23.636412+0000", 2024-09-18T07:32:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.09007716, 2024-09-18T07:32:35.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031110209999999999, 2024-09-18T07:32:35.765 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.765 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.765 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.765 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 20776 2024-09-18T07:32:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.636412+0000", 2024-09-18T07:32:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.636412+0000", 2024-09-18T07:32:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.636414+0000", 2024-09-18T07:32:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.858e-06 2024-09-18T07:32:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.636437+0000", 2024-09-18T07:32:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2609999999999999e-05 2024-09-18T07:32:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.636448+0000", 2024-09-18T07:32:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0601000000000001e-05 2024-09-18T07:32:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.636455+0000", 2024-09-18T07:32:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7300000000000005e-06 2024-09-18T07:32:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.636516+0000", 2024-09-18T07:32:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0849e-05 2024-09-18T07:32:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.636638+0000", 2024-09-18T07:32:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012148500000000001 2024-09-18T07:32:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.636756+0000", 2024-09-18T07:32:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118322 2024-09-18T07:32:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.637852+0000", 2024-09-18T07:32:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010956729999999999 2024-09-18T07:32:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.638016+0000", 2024-09-18T07:32:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016420700000000001 2024-09-18T07:32:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.667413+0000", 2024-09-18T07:32:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.029397039999999999 2024-09-18T07:32:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.667470+0000", 2024-09-18T07:32:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7306000000000003e-05 2024-09-18T07:32:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.667523+0000", 2024-09-18T07:32:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2528999999999998e-05 2024-09-18T07:32:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:20779 7.b 7:d9ccc9ba:::benchmark_data_smithi137_74180_object20778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T07:32:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:23.637898+0000", 2024-09-18T07:32:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.088591766, 2024-09-18T07:32:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031979516999999999, 2024-09-18T07:32:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 20779 2024-09-18T07:32:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.637898+0000", 2024-09-18T07:32:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.637898+0000", 2024-09-18T07:32:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.637900+0000", 2024-09-18T07:32:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2469999999999998e-06 2024-09-18T07:32:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.637922+0000", 2024-09-18T07:32:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2010000000000001e-05 2024-09-18T07:32:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.637934+0000", 2024-09-18T07:32:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1955e-05 2024-09-18T07:32:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.637942+0000", 2024-09-18T07:32:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4159999999999998e-06 2024-09-18T07:32:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.638056+0000", 2024-09-18T07:32:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114221 2024-09-18T07:32:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.638160+0000", 2024-09-18T07:32:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104227 2024-09-18T07:32:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.638280+0000", 2024-09-18T07:32:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000119731 2024-09-18T07:32:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.639947+0000", 2024-09-18T07:32:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001667591 2024-09-18T07:32:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.665967+0000", 2024-09-18T07:32:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026019614 2024-09-18T07:32:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.669732+0000", 2024-09-18T07:32:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0037647420000000002 2024-09-18T07:32:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.669828+0000", 2024-09-18T07:32:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6626999999999995e-05 2024-09-18T07:32:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.669877+0000", 2024-09-18T07:32:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9135999999999998e-05 2024-09-18T07:32:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:20782 7.b 7:d5af0914:::benchmark_data_smithi137_74180_object20781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T07:32:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:23.639063+0000", 2024-09-18T07:32:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.087426885999999, 2024-09-18T07:32:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.030994573000000001, 2024-09-18T07:32:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 20782 2024-09-18T07:32:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.639063+0000", 2024-09-18T07:32:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.639063+0000", 2024-09-18T07:32:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.639064+0000", 2024-09-18T07:32:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6029999999999999e-06 2024-09-18T07:32:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.639086+0000", 2024-09-18T07:32:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.207e-05 2024-09-18T07:32:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.639098+0000", 2024-09-18T07:32:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1368e-05 2024-09-18T07:32:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.783 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.639105+0000", 2024-09-18T07:32:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0210000000000002e-06 2024-09-18T07:32:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.639179+0000", 2024-09-18T07:32:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4463999999999997e-05 2024-09-18T07:32:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.639287+0000", 2024-09-18T07:32:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107494 2024-09-18T07:32:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.639377+0000", 2024-09-18T07:32:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0124999999999995e-05 2024-09-18T07:32:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.640113+0000", 2024-09-18T07:32:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00073598200000000004 2024-09-18T07:32:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.667674+0000", 2024-09-18T07:32:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027560628 2024-09-18T07:32:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.669919+0000", 2024-09-18T07:32:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022452489999999999 2024-09-18T07:32:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.670017+0000", 2024-09-18T07:32:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8145000000000003e-05 2024-09-18T07:32:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.670057+0000", 2024-09-18T07:32:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0423999999999998e-05 2024-09-18T07:32:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:21882 7.b 7:ddfde259:::benchmark_data_smithi137_74180_object21881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:32:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:24.308089+0000", 2024-09-18T07:32:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 11.418400881, 2024-09-18T07:32:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.077958042000000005, 2024-09-18T07:32:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 21882 2024-09-18T07:32:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308089+0000", 2024-09-18T07:32:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308089+0000", 2024-09-18T07:32:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308091+0000", 2024-09-18T07:32:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2469999999999998e-06 2024-09-18T07:32:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308115+0000", 2024-09-18T07:32:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3669e-05 2024-09-18T07:32:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308131+0000", 2024-09-18T07:32:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5891000000000001e-05 2024-09-18T07:32:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308139+0000", 2024-09-18T07:32:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1470000000000001e-06 2024-09-18T07:32:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308196+0000", 2024-09-18T07:32:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7092000000000002e-05 2024-09-18T07:32:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308300+0000", 2024-09-18T07:32:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104026 2024-09-18T07:32:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308448+0000", 2024-09-18T07:32:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000148271 2024-09-18T07:32:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.309303+0000", 2024-09-18T07:32:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00085516499999999996 2024-09-18T07:32:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.309558+0000", 2024-09-18T07:32:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00025495900000000002 2024-09-18T07:32:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.385821+0000", 2024-09-18T07:32:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.076263132999999997 2024-09-18T07:32:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.385898+0000", 2024-09-18T07:32:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7080000000000001e-05 2024-09-18T07:32:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.386047+0000", 2024-09-18T07:32:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000148362 2024-09-18T07:32:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:21894 7.b 7:da78b007:::benchmark_data_smithi137_74180_object21893:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:32:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:24.315487+0000", 2024-09-18T07:32:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 11.411002312000001, 2024-09-18T07:32:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.071918722000000004, 2024-09-18T07:32:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 21894 2024-09-18T07:32:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315487+0000", 2024-09-18T07:32:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315487+0000", 2024-09-18T07:32:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315489+0000", 2024-09-18T07:32:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1179999999999999e-06 2024-09-18T07:32:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315509+0000", 2024-09-18T07:32:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9448000000000001e-05 2024-09-18T07:32:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315520+0000", 2024-09-18T07:32:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0771000000000001e-05 2024-09-18T07:32:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315527+0000", 2024-09-18T07:32:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4930000000000001e-06 2024-09-18T07:32:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315587+0000", 2024-09-18T07:32:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0297000000000002e-05 2024-09-18T07:32:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315696+0000", 2024-09-18T07:32:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010875899999999999 2024-09-18T07:32:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315787+0000", 2024-09-18T07:32:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0686000000000006e-05 2024-09-18T07:32:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.316570+0000", 2024-09-18T07:32:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00078330100000000003 2024-09-18T07:32:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.316853+0000", 2024-09-18T07:32:35.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028248100000000001 2024-09-18T07:32:35.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.386216+0000", 2024-09-18T07:32:35.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.069363707999999996 2024-09-18T07:32:35.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.387354+0000", 2024-09-18T07:32:35.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001137742 2024-09-18T07:32:35.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.387406+0000", 2024-09-18T07:32:35.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1918e-05 2024-09-18T07:32:35.801 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.801 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.801 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.801 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:22340 7.b 7:dcab8849:::benchmark_data_smithi137_74180_object22339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:32:35.801 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:24.658442+0000", 2024-09-18T07:32:35.801 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 11.068047564, 2024-09-18T07:32:35.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.20618298299999999, 2024-09-18T07:32:35.801 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.801 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.802 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.802 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.802 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.802 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.802 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 22340 2024-09-18T07:32:35.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.802 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.802 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.802 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658442+0000", 2024-09-18T07:32:35.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658442+0000", 2024-09-18T07:32:35.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658445+0000", 2024-09-18T07:32:35.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4609999999999998e-06 2024-09-18T07:32:35.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658479+0000", 2024-09-18T07:32:35.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4393000000000001e-05 2024-09-18T07:32:35.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658497+0000", 2024-09-18T07:32:35.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7649999999999999e-05 2024-09-18T07:32:35.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658505+0000", 2024-09-18T07:32:35.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5350000000000004e-06 2024-09-18T07:32:35.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658567+0000", 2024-09-18T07:32:35.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1580000000000003e-05 2024-09-18T07:32:35.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658673+0000", 2024-09-18T07:32:35.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105907 2024-09-18T07:32:35.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658790+0000", 2024-09-18T07:32:35.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117494 2024-09-18T07:32:35.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.660870+0000", 2024-09-18T07:32:35.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020794849999999998 2024-09-18T07:32:35.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.661103+0000", 2024-09-18T07:32:35.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023344199999999999 2024-09-18T07:32:35.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.864502+0000", 2024-09-18T07:32:35.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.20339850100000001 2024-09-18T07:32:35.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.864571+0000", 2024-09-18T07:32:35.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9566000000000004e-05 2024-09-18T07:32:35.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.864625+0000", 2024-09-18T07:32:35.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3968999999999998e-05 2024-09-18T07:32:35.807 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.807 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.807 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.807 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:23002 7.b 7:d755eca4:::benchmark_data_smithi137_74180_object23001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:32:35.808 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:25.332555+0000", 2024-09-18T07:32:35.808 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 10.393934481000001, 2024-09-18T07:32:35.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.098253845000000006, 2024-09-18T07:32:35.808 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.808 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.808 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.808 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.808 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.808 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.808 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 23002 2024-09-18T07:32:35.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.808 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.332555+0000", 2024-09-18T07:32:35.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.332555+0000", 2024-09-18T07:32:35.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.332564+0000", 2024-09-18T07:32:35.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2699999999999993e-06 2024-09-18T07:32:35.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.332589+0000", 2024-09-18T07:32:35.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4139999999999999e-05 2024-09-18T07:32:35.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.332605+0000", 2024-09-18T07:32:35.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6874000000000001e-05 2024-09-18T07:32:35.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.332613+0000", 2024-09-18T07:32:35.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4390000000000003e-06 2024-09-18T07:32:35.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.332686+0000", 2024-09-18T07:32:35.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2898000000000006e-05 2024-09-18T07:32:35.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.332821+0000", 2024-09-18T07:32:35.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013513699999999999 2024-09-18T07:32:35.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.333031+0000", 2024-09-18T07:32:35.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020990000000000001 2024-09-18T07:32:35.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.334458+0000", 2024-09-18T07:32:35.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001427357 2024-09-18T07:32:35.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.334923+0000", 2024-09-18T07:32:35.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00046475999999999999 2024-09-18T07:32:35.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.430660+0000", 2024-09-18T07:32:35.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.095736694999999997 2024-09-18T07:32:35.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.430719+0000", 2024-09-18T07:32:35.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9698999999999999e-05 2024-09-18T07:32:35.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.430809+0000", 2024-09-18T07:32:35.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9676e-05 2024-09-18T07:32:35.814 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.814 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.814 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.814 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:23243 7.b 7:da651bd5:::benchmark_data_smithi137_74180_object23242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:32:35.814 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:25.633172+0000", 2024-09-18T07:32:35.814 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 10.093317647999999, 2024-09-18T07:32:35.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.049728649999999999, 2024-09-18T07:32:35.814 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.814 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.815 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.815 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.815 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.815 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.815 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 23243 2024-09-18T07:32:35.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.815 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.633172+0000", 2024-09-18T07:32:35.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.633172+0000", 2024-09-18T07:32:35.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.633173+0000", 2024-09-18T07:32:35.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.173e-06 2024-09-18T07:32:35.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.633196+0000", 2024-09-18T07:32:35.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3e-05 2024-09-18T07:32:35.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.633207+0000", 2024-09-18T07:32:35.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0487e-05 2024-09-18T07:32:35.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.633214+0000", 2024-09-18T07:32:35.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4449999999999999e-06 2024-09-18T07:32:35.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.634329+0000", 2024-09-18T07:32:35.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001114704 2024-09-18T07:32:35.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.634453+0000", 2024-09-18T07:32:35.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012430899999999999 2024-09-18T07:32:35.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.634659+0000", 2024-09-18T07:32:35.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000206376 2024-09-18T07:32:35.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.635703+0000", 2024-09-18T07:32:35.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010438839999999999 2024-09-18T07:32:35.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.635770+0000", 2024-09-18T07:32:35.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6230999999999997e-05 2024-09-18T07:32:35.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.682785+0000", 2024-09-18T07:32:35.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.047015873999999999 2024-09-18T07:32:35.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.682846+0000", 2024-09-18T07:32:35.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0179000000000001e-05 2024-09-18T07:32:35.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.682901+0000", 2024-09-18T07:32:35.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4988000000000003e-05 2024-09-18T07:32:35.820 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.820 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.820 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.821 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:23306 7.b 7:d652fc90:::benchmark_data_smithi137_74180_object23305:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:32:35.821 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:25.698938+0000", 2024-09-18T07:32:35.821 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 10.027551811, 2024-09-18T07:32:35.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015276858000000001, 2024-09-18T07:32:35.821 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.821 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.821 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.821 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.821 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.821 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.821 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 23306 2024-09-18T07:32:35.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.822 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.698938+0000", 2024-09-18T07:32:35.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.698938+0000", 2024-09-18T07:32:35.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.698940+0000", 2024-09-18T07:32:35.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9999999999999999e-06 2024-09-18T07:32:35.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.698962+0000", 2024-09-18T07:32:35.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2548000000000002e-05 2024-09-18T07:32:35.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.698974+0000", 2024-09-18T07:32:35.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1429e-05 2024-09-18T07:32:35.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.698991+0000", 2024-09-18T07:32:35.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6977e-05 2024-09-18T07:32:35.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.699050+0000", 2024-09-18T07:32:35.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8835999999999997e-05 2024-09-18T07:32:35.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.699173+0000", 2024-09-18T07:32:35.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000123681 2024-09-18T07:32:35.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.699310+0000", 2024-09-18T07:32:35.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013659800000000001 2024-09-18T07:32:35.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.700708+0000", 2024-09-18T07:32:35.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001398481 2024-09-18T07:32:35.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.701607+0000", 2024-09-18T07:32:35.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00089826800000000003 2024-09-18T07:32:35.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.714098+0000", 2024-09-18T07:32:35.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012491175 2024-09-18T07:32:35.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.714159+0000", 2024-09-18T07:32:35.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1482999999999994e-05 2024-09-18T07:32:35.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.827 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.827 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:32:35.773411+0000 min lat: 0.000980073 max lat: 2.1848 avg lat: 0.00900497 2024-09-18T07:32:35.827 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:32:35.827 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 20 16 35531 35515 110.947 70.9375 0.00659708 0.00900497 2024-09-18T07:32:35.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.714215+0000", 2024-09-18T07:32:35.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5381999999999999e-05 2024-09-18T07:32:35.828 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.828 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.828 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.828 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:23313 7.b 7:da8131d1:::benchmark_data_smithi137_74180_object23312:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:32:35.828 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:25.702216+0000", 2024-09-18T07:32:35.828 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 10.024274113000001, 2024-09-18T07:32:35.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012350257999999999, 2024-09-18T07:32:35.828 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.828 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.828 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.828 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.829 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.829 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.829 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 23313 2024-09-18T07:32:35.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.829 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.702216+0000", 2024-09-18T07:32:35.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.702216+0000", 2024-09-18T07:32:35.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.702218+0000", 2024-09-18T07:32:35.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.317e-06 2024-09-18T07:32:35.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.702243+0000", 2024-09-18T07:32:35.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5588e-05 2024-09-18T07:32:35.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.702260+0000", 2024-09-18T07:32:35.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6898e-05 2024-09-18T07:32:35.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.702268+0000", 2024-09-18T07:32:35.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.216e-06 2024-09-18T07:32:35.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.702333+0000", 2024-09-18T07:32:35.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5028000000000001e-05 2024-09-18T07:32:35.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.702466+0000", 2024-09-18T07:32:35.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000133428 2024-09-18T07:32:35.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.702633+0000", 2024-09-18T07:32:35.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016699799999999999 2024-09-18T07:32:35.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.704111+0000", 2024-09-18T07:32:35.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014776629999999999 2024-09-18T07:32:35.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.705906+0000", 2024-09-18T07:32:35.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017949509999999999 2024-09-18T07:32:35.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.714464+0000", 2024-09-18T07:32:35.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0085583919999999997 2024-09-18T07:32:35.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.714525+0000", 2024-09-18T07:32:35.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0680000000000002e-05 2024-09-18T07:32:35.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.714566+0000", 2024-09-18T07:32:35.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1099000000000001e-05 2024-09-18T07:32:35.834 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.834 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.834 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.834 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:23654 7.b 7:da1f32ad:::benchmark_data_smithi137_74180_object23653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-18T07:32:35.834 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:25.862233+0000", 2024-09-18T07:32:35.834 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 9.8642567789999998, 2024-09-18T07:32:35.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013511497000000001, 2024-09-18T07:32:35.834 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.834 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.835 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.835 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.835 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.835 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.835 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 23654 2024-09-18T07:32:35.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.835 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.862233+0000", 2024-09-18T07:32:35.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.862233+0000", 2024-09-18T07:32:35.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.862233+0000", 2024-09-18T07:32:35.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2200000000000004e-07 2024-09-18T07:32:35.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.862274+0000", 2024-09-18T07:32:35.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0117999999999998e-05 2024-09-18T07:32:35.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.862280+0000", 2024-09-18T07:32:35.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0480000000000004e-06 2024-09-18T07:32:35.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.862285+0000", 2024-09-18T07:32:35.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0730000000000004e-06 2024-09-18T07:32:35.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.866508+0000", 2024-09-18T07:32:35.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042234769999999998 2024-09-18T07:32:35.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.866605+0000", 2024-09-18T07:32:35.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6658999999999997e-05 2024-09-18T07:32:35.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.867099+0000", 2024-09-18T07:32:35.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00049410899999999995 2024-09-18T07:32:35.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.869792+0000", 2024-09-18T07:32:35.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026932219999999999 2024-09-18T07:32:35.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.870154+0000", 2024-09-18T07:32:35.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00036224100000000002 2024-09-18T07:32:35.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.874038+0000", 2024-09-18T07:32:35.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038837950000000002 2024-09-18T07:32:35.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.875693+0000", 2024-09-18T07:32:35.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001654406 2024-09-18T07:32:35.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.875744+0000", 2024-09-18T07:32:35.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1727000000000001e-05 2024-09-18T07:32:35.840 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.840 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.840 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.840 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:28410 7.d e92/81 7:b111a08f:::benchmark_data_smithi137_74180_object28409:head v 92'1816, mlcod=92'1812)", 2024-09-18T07:32:35.840 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:28.347204+0000", 2024-09-18T07:32:35.840 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 7.3792860520000003, 2024-09-18T07:32:35.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012848099999999999, 2024-09-18T07:32:35.841 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.841 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.841 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.841 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.347204+0000", 2024-09-18T07:32:35.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.347204+0000", 2024-09-18T07:32:35.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.347205+0000", 2024-09-18T07:32:35.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1969999999999999e-06 2024-09-18T07:32:35.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.347304+0000", 2024-09-18T07:32:35.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9222000000000006e-05 2024-09-18T07:32:35.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.347313+0000", 2024-09-18T07:32:35.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5790000000000004e-06 2024-09-18T07:32:35.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.347319+0000", 2024-09-18T07:32:35.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7900000000000002e-06 2024-09-18T07:32:35.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.347359+0000", 2024-09-18T07:32:35.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9422000000000003e-05 2024-09-18T07:32:35.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.347377+0000", 2024-09-18T07:32:35.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8036000000000001e-05 2024-09-18T07:32:35.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.359963+0000", 2024-09-18T07:32:35.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012585785 2024-09-18T07:32:35.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.360052+0000", 2024-09-18T07:32:35.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9068999999999997e-05 2024-09-18T07:32:35.845 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.845 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.845 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.845 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:28413 7.b 7:dbc39c9c:::benchmark_data_smithi137_74180_object28412:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-18T07:32:35.845 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:28.348526+0000", 2024-09-18T07:32:35.845 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 7.3779641299999996, 2024-09-18T07:32:35.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012701591999999999, 2024-09-18T07:32:35.845 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.845 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.845 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.845 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.845 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.846 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.846 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 28413 2024-09-18T07:32:35.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.846 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.348526+0000", 2024-09-18T07:32:35.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.348526+0000", 2024-09-18T07:32:35.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.348528+0000", 2024-09-18T07:32:35.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1059999999999998e-06 2024-09-18T07:32:35.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.348552+0000", 2024-09-18T07:32:35.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.406e-05 2024-09-18T07:32:35.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.355212+0000", 2024-09-18T07:32:35.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066599500000000004 2024-09-18T07:32:35.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.355222+0000", 2024-09-18T07:32:35.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0278000000000001e-05 2024-09-18T07:32:35.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.356273+0000", 2024-09-18T07:32:35.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010506669999999999 2024-09-18T07:32:35.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.356406+0000", 2024-09-18T07:32:35.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013390799999999999 2024-09-18T07:32:35.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.356577+0000", 2024-09-18T07:32:35.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017004800000000001 2024-09-18T07:32:35.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.361045+0000", 2024-09-18T07:32:35.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044685669999999997 2024-09-18T07:32:35.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.361057+0000", 2024-09-18T07:32:35.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2216e-05 2024-09-18T07:32:35.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.361106+0000", 2024-09-18T07:32:35.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8192999999999998e-05 2024-09-18T07:32:35.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.361168+0000", 2024-09-18T07:32:35.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2020999999999995e-05 2024-09-18T07:32:35.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.361227+0000", 2024-09-18T07:32:35.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9577999999999998e-05 2024-09-18T07:32:35.850 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.851 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.851 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.851 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:29112 7.b 7:db2e97d2:::benchmark_data_smithi137_74180_object29111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-18T07:32:35.851 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:28.659614+0000", 2024-09-18T07:32:35.851 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 7.0668752939999999, 2024-09-18T07:32:35.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019359711000000002, 2024-09-18T07:32:35.851 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.851 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.851 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.851 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.851 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.851 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.851 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29112 2024-09-18T07:32:35.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.852 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.659614+0000", 2024-09-18T07:32:35.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.659614+0000", 2024-09-18T07:32:35.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.659616+0000", 2024-09-18T07:32:35.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5820000000000001e-06 2024-09-18T07:32:35.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.659640+0000", 2024-09-18T07:32:35.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4315e-05 2024-09-18T07:32:35.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.659651+0000", 2024-09-18T07:32:35.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0597999999999999e-05 2024-09-18T07:32:35.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.659660+0000", 2024-09-18T07:32:35.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3109999999999995e-06 2024-09-18T07:32:35.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.667685+0000", 2024-09-18T07:32:35.854 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:32:35.854 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:32:35.854 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:32:35.855 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:32:35.855 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:32:35.855 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:32:35.855 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:32:35.855 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:32:35.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0080248880000000009 2024-09-18T07:32:35.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.671600+0000", 2024-09-18T07:32:35.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039151680000000001 2024-09-18T07:32:35.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.671733+0000", 2024-09-18T07:32:35.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013316000000000001 2024-09-18T07:32:35.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.674904+0000", 2024-09-18T07:32:35.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031703159999999998 2024-09-18T07:32:35.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.674938+0000", 2024-09-18T07:32:35.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4010000000000001e-05 2024-09-18T07:32:35.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.678881+0000", 2024-09-18T07:32:35.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039434170000000003 2024-09-18T07:32:35.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.678932+0000", 2024-09-18T07:32:35.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0912000000000003e-05 2024-09-18T07:32:35.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.678974+0000", 2024-09-18T07:32:35.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2033999999999998e-05 2024-09-18T07:32:35.858 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.858 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.858 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.859 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:29365 7.b 7:ded7e5e0:::benchmark_data_smithi137_74180_object29364:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-18T07:32:35.859 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:28.788669+0000", 2024-09-18T07:32:35.859 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 6.9378202450000002, 2024-09-18T07:32:35.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012781481000000001, 2024-09-18T07:32:35.859 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.859 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.859 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.859 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:35.859 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:35.859 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:35.859 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29365 2024-09-18T07:32:35.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.859 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.788669+0000", 2024-09-18T07:32:35.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.788669+0000", 2024-09-18T07:32:35.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.788671+0000", 2024-09-18T07:32:35.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2720000000000001e-06 2024-09-18T07:32:35.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.788693+0000", 2024-09-18T07:32:35.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1841999999999999e-05 2024-09-18T07:32:35.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.788702+0000", 2024-09-18T07:32:35.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6979999999999993e-06 2024-09-18T07:32:35.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.788768+0000", 2024-09-18T07:32:35.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5407000000000006e-05 2024-09-18T07:32:35.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.794105+0000", 2024-09-18T07:32:35.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053370249999999996 2024-09-18T07:32:35.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.794229+0000", 2024-09-18T07:32:35.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012463599999999999 2024-09-18T07:32:35.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:32:35.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.794365+0000", 2024-09-18T07:32:35.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013585600000000001 2024-09-18T07:32:35.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.796098+0000", 2024-09-18T07:32:35.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001732957 2024-09-18T07:32:35.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:35.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.796136+0000", 2024-09-18T07:32:35.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8068999999999998e-05 2024-09-18T07:32:35.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:35.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.801206+0000", 2024-09-18T07:32:35.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050699300000000003 2024-09-18T07:32:35.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.801402+0000", 2024-09-18T07:32:35.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000196057 2024-09-18T07:32:35.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:28.801451+0000", 2024-09-18T07:32:35.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8732e-05 2024-09-18T07:32:35.865 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.865 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.865 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.865 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:29785 7.4 e92/81 7:2c039360:::benchmark_data_smithi137_74180_object29784:head v 92'1900, mlcod=92'1896)", 2024-09-18T07:32:35.865 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:29.009495+0000", 2024-09-18T07:32:35.865 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 6.7169950370000002, 2024-09-18T07:32:35.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012554920000000001, 2024-09-18T07:32:35.865 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:35.865 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:35.866 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:32:35.866 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:35.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:35.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:29.009495+0000", 2024-09-18T07:32:35.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:35.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:29.009495+0000", 2024-09-18T07:32:35.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:35.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:35.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:29.009496+0000", 2024-09-18T07:32:35.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7430000000000001e-06 2024-09-18T07:32:35.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:35.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:29.009549+0000", 2024-09-18T07:32:35.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2485999999999999e-05 2024-09-18T07:32:35.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:35.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:29.009559+0000", 2024-09-18T07:32:35.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0529e-05 2024-09-18T07:32:35.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:35.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:29.009567+0000", 2024-09-18T07:32:35.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7109999999999995e-06 2024-09-18T07:32:35.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:35.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:29.009622+0000", 2024-09-18T07:32:35.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5211999999999999e-05 2024-09-18T07:32:35.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:35.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:29.009641+0000", 2024-09-18T07:32:35.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8632000000000001e-05 2024-09-18T07:32:35.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:32:35.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:29.021764+0000", 2024-09-18T07:32:35.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012123073 2024-09-18T07:32:35.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:35.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:35.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:32:35.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:29.022050+0000", 2024-09-18T07:32:35.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028553399999999999 2024-09-18T07:32:35.869 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.870 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.870 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.870 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:35.870 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:35.870 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:32:35.870 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:32:35.870 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:32:35.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:35.872 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:35.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:35.878 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:32:35.879 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:32:35.912 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:32:35.912 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:32:35.950 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:32:35.950 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:32:35.972 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:35 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2544291593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:32:35.973 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:35 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2544291593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:32:35.973 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:35.998 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:35.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:35.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:36.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:36.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:36.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:36.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:36.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:35 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2544291593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:32:36.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:36.290 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:36.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:36.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:36.392 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:36.418 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:36.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:36.520 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:36.547 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:36.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:36.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:36.649 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:36.687 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:36.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:36.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:36.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:32:36.776 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 21 16 36791 36775 109.411 78.75 0.0213763 0.00913277 2024-09-18T07:32:36.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:36.838 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:36.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:36.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:36.940 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:36.963 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:36.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:36.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:37.065 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:37.091 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:37.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:37.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:36 smithi137 ceph-mon[29408]: pgmap v61: 148 pgs: 5 remapped+peering, 143 active+clean; 1.9 GiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s; 103 KiB/s, 15 objects/s recovering 2024-09-18T07:32:37.101 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:36 smithi137 ceph-mon[39175]: pgmap v61: 148 pgs: 5 remapped+peering, 143 active+clean; 1.9 GiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s; 103 KiB/s, 15 objects/s recovering 2024-09-18T07:32:37.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:37.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:36 smithi203 ceph-mon[32237]: pgmap v61: 148 pgs: 5 remapped+peering, 143 active+clean; 1.9 GiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s; 103 KiB/s, 15 objects/s recovering 2024-09-18T07:32:37.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:32:36 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:32:37.243 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:37.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:37.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:37.345 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:37.357 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:32:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:32:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:32:37.369 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:37.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:37.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:37.471 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:37.495 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:37.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:37.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:37.597 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:37.623 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:37.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:37.725 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:37.765 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:37.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:37.775 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 22 16 37985 37969 107.83 74.625 0.0158769 0.00926615 2024-09-18T07:32:37.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T07:32:37.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:37.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:37.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:38.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:38.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:38.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:38.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:38.096 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:37 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:38.097 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:37 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:38.190 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:38.206 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:38.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:37 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:38.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:38.216 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:38.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:38.276 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:32:38.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-18T07:32:38.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:38.373 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:38.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:38.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:38.523 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:38.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:38.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:38.625 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:38.676 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:38.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:38.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:38.776 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 23 16 39042 39026 106.013 66.0625 0.0158884 0.00942636 2024-09-18T07:32:38.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:38.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:38.837 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:38.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:38.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:38.984 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:38.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:39.013 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:38 smithi137 ceph-mon[29408]: pgmap v62: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.2 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 110 MiB/s wr, 1.75k op/s; 1912/109542 objects degraded (1.745%); 6760/109542 objects misplaced (6.171%); 9.8 MiB/s, 168 objects/s recovering 2024-09-18T07:32:39.013 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:38 smithi137 ceph-mon[29408]: Health check failed: Degraded data redundancy: 1912/109542 objects degraded (1.745%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:32:39.013 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:38 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:32:39.013 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:38 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:32:39.013 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:38 smithi137 ceph-mon[39175]: pgmap v62: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.2 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 110 MiB/s wr, 1.75k op/s; 1912/109542 objects degraded (1.745%); 6760/109542 objects misplaced (6.171%); 9.8 MiB/s, 168 objects/s recovering 2024-09-18T07:32:39.013 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:38 smithi137 ceph-mon[39175]: Health check failed: Degraded data redundancy: 1912/109542 objects degraded (1.745%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:32:39.013 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:38 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:32:39.014 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:38 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:32:39.087 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:39.113 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:39.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:39.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:39.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:38 smithi203 ceph-mon[32237]: pgmap v62: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.2 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 110 MiB/s wr, 1.75k op/s; 1912/109542 objects degraded (1.745%); 6760/109542 objects misplaced (6.171%); 9.8 MiB/s, 168 objects/s recovering 2024-09-18T07:32:39.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:38 smithi203 ceph-mon[32237]: Health check failed: Degraded data redundancy: 1912/109542 objects degraded (1.745%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:32:39.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:38 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:32:39.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:38 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:32:39.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:39.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:39.279 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:39.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:39.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:32:39.379 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:39.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:39.463 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:39.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:39.564 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:39.626 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:39.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:39.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:39.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:39.749 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:32:39.755 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:39.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:39.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:39.778 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 24 16 40037 40021 104.185 62.1875 0.0301675 0.00959186 2024-09-18T07:32:39.857 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:39.921 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:39.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:40.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:40.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:40.068 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:40.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:40.125 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:40.169 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:39 smithi137 ceph-mon[29408]: osdmap e97: 8 total, 8 up, 7 in 2024-09-18T07:32:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:39 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2987279853' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:32:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:39 smithi137 ceph-mon[29408]: from='client.34778 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:32:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:39 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:32:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:39 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:32:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:39 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:32:40.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:39 smithi137 ceph-mon[29408]: from='client.34778 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:32:40.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:39 smithi137 ceph-mon[29408]: osdmap e98: 8 total, 8 up, 7 in 2024-09-18T07:32:40.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:39 smithi137 ceph-mon[39175]: osdmap e97: 8 total, 8 up, 7 in 2024-09-18T07:32:40.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:39 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2987279853' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:32:40.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:39 smithi137 ceph-mon[39175]: from='client.34778 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:32:40.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:39 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:32:40.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:39 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:32:40.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:39 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:32:40.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:39 smithi137 ceph-mon[39175]: from='client.34778 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:32:40.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:39 smithi137 ceph-mon[39175]: osdmap e98: 8 total, 8 up, 7 in 2024-09-18T07:32:40.195 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:40.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:40.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:40.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:39 smithi203 ceph-mon[32237]: osdmap e97: 8 total, 8 up, 7 in 2024-09-18T07:32:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:39 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2987279853' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:32:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:39 smithi203 ceph-mon[32237]: from='client.34778 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:32:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:39 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:32:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:39 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:32:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:39 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:32:40.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:39 smithi203 ceph-mon[32237]: from='client.34778 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:32:40.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:39 smithi203 ceph-mon[32237]: osdmap e98: 8 total, 8 up, 7 in 2024-09-18T07:32:40.297 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:40.322 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:40.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:40.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:40.424 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:40.450 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:40.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:40.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:40.552 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:40.576 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:40.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:40.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:40.678 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:40.701 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:40.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:40.725 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:40.776 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 25 16 41161 41145 102.828 70.25 0.0246925 0.00971746 2024-09-18T07:32:40.803 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:40.853 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:40.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:40.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:40.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:40 smithi137 ceph-mon[29408]: pgmap v64: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.2 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 674 B/s rd, 42 MiB/s wr, 679 op/s; 1912/109542 objects degraded (1.745%); 6760/109542 objects misplaced (6.171%); 8.4 MiB/s, 134 objects/s recovering 2024-09-18T07:32:40.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:40.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:40.997 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:40.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:41.098 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:41.124 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:41.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:41.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:41.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:40 smithi203 ceph-mon[32237]: pgmap v64: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.2 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 674 B/s rd, 42 MiB/s wr, 679 op/s; 1912/109542 objects degraded (1.745%); 6760/109542 objects misplaced (6.171%); 8.4 MiB/s, 134 objects/s recovering 2024-09-18T07:32:41.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:40 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:40 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:40 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:32:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:40 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:32:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:40 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:40 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:40 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:32:41.226 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:41.262 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:41.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:41.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:40 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:41.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:40 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:41.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:40 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:32:41.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:40 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:32:41.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:40 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:41.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:40 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:41.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:40 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:32:41.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:40 smithi137 ceph-mon[39175]: pgmap v64: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.2 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 674 B/s rd, 42 MiB/s wr, 679 op/s; 1912/109542 objects degraded (1.745%); 6760/109542 objects misplaced (6.171%); 8.4 MiB/s, 134 objects/s recovering 2024-09-18T07:32:41.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:40 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:41.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:40 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:41.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:40 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:32:41.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:40 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:32:41.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:40 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:41.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:40 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:32:41.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:40 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:32:41.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:41.412 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:41.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:41.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:41.514 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:41.538 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:41.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:41.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:41.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:41.682 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:41.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:41.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:41.776 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 26 15 42260 42245 101.517 68.75 0.0217194 0.00984281 2024-09-18T07:32:41.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:41.829 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:41.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:41.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:41.931 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:41.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:41.972 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:41.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:42.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:42.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:42.111 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:42.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:42.164 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:41 smithi137 ceph-mon[29408]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-18T07:32:42.164 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:41 smithi137 ceph-mon[29408]: Reconfiguring daemon alertmanager.a on smithi137 2024-09-18T07:32:42.165 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:41 smithi137 ceph-mon[39175]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-18T07:32:42.165 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:41 smithi137 ceph-mon[39175]: Reconfiguring daemon alertmanager.a on smithi137 2024-09-18T07:32:42.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:41 smithi203 ceph-mon[32237]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-18T07:32:42.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:41 smithi203 ceph-mon[32237]: Reconfiguring daemon alertmanager.a on smithi137 2024-09-18T07:32:42.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:42.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:42.244 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:42.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:42.345 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:42.372 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:42.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:42.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:42.474 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:42.516 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:42.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:42.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:42.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:42.658 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:42.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:42.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:42.760 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:42.776 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 27 15 43463 43448 100.542 75.1875 0.00724765 0.00993944 2024-09-18T07:32:42.788 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:32:42.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:42.803 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:42.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:42.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:42.962 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:32:42.962 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:32:42.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.962 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:42068 7.8 7:14618195:::benchmark_data_smithi137_74180_object42067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-18T07:32:42.962 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:41.576320+0000", 2024-09-18T07:32:42.962 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.014557616000000001, 2024-09-18T07:32:42.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014597598999999999, 2024-09-18T07:32:42.962 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:42.962 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:42.962 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:32:42.963 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:42.963 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:42.963 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:42.963 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 42068 2024-09-18T07:32:42.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.963 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:42.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:42.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576320+0000", 2024-09-18T07:32:42.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:42.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:42.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576320+0000", 2024-09-18T07:32:42.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:42.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:42.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576322+0000", 2024-09-18T07:32:42.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2010000000000002e-06 2024-09-18T07:32:42.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:42.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576348+0000", 2024-09-18T07:32:42.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6480000000000001e-05 2024-09-18T07:32:42.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:42.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576365+0000", 2024-09-18T07:32:42.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6307000000000002e-05 2024-09-18T07:32:42.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:42.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576374+0000", 2024-09-18T07:32:42.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0310000000000004e-06 2024-09-18T07:32:42.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:42.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.577373+0000", 2024-09-18T07:32:42.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00099895499999999994 2024-09-18T07:32:42.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:42.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.577492+0000", 2024-09-18T07:32:42.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011945100000000001 2024-09-18T07:32:42.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:32:42.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.577622+0000", 2024-09-18T07:32:42.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000130006 2024-09-18T07:32:42.967 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:42.967 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:42.967 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:42.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.967 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:42069 7.e 7:7360dadd:::benchmark_data_smithi137_74180_object42068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-18T07:32:42.967 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:41.576498+0000", 2024-09-18T07:32:42.967 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.014379584000000001, 2024-09-18T07:32:42.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01449685, 2024-09-18T07:32:42.967 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:42.967 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:42.967 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:32:42.967 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:42.968 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:42.968 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:42.968 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 42069 2024-09-18T07:32:42.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.968 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:42.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:42.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576498+0000", 2024-09-18T07:32:42.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:42.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:42.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576498+0000", 2024-09-18T07:32:42.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:42.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:42.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576499+0000", 2024-09-18T07:32:42.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.44e-06 2024-09-18T07:32:42.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:42.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576521+0000", 2024-09-18T07:32:42.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1746000000000002e-05 2024-09-18T07:32:42.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:42.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576535+0000", 2024-09-18T07:32:42.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.395e-05 2024-09-18T07:32:42.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:42.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576542+0000", 2024-09-18T07:32:42.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7570000000000004e-06 2024-09-18T07:32:42.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:42.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.577086+0000", 2024-09-18T07:32:42.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00054337500000000002 2024-09-18T07:32:42.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:42.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.577210+0000", 2024-09-18T07:32:42.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012417400000000001 2024-09-18T07:32:42.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:32:42.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.577358+0000", 2024-09-18T07:32:42.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014785899999999999 2024-09-18T07:32:42.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:42.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.579145+0000", 2024-09-18T07:32:42.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001787224 2024-09-18T07:32:42.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:42.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.580162+0000", 2024-09-18T07:32:42.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010167259999999999 2024-09-18T07:32:42.972 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:42.972 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:42.972 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:42.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.973 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:42065 7.c e98/94 7:356624d2:::benchmark_data_smithi137_74180_object42064:head v 98'2632, mlcod=88'824)", 2024-09-18T07:32:42.973 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:41.576731+0000", 2024-09-18T07:32:42.973 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.014146271, 2024-09-18T07:32:42.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014332018, 2024-09-18T07:32:42.973 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:42.973 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:42.973 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:32:42.973 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:42.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:42.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576731+0000", 2024-09-18T07:32:42.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:42.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:42.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576731+0000", 2024-09-18T07:32:42.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:42.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:42.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576732+0000", 2024-09-18T07:32:42.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3519999999999999e-06 2024-09-18T07:32:42.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:42.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576766+0000", 2024-09-18T07:32:42.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3739999999999999e-05 2024-09-18T07:32:42.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:42.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576778+0000", 2024-09-18T07:32:42.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1586000000000001e-05 2024-09-18T07:32:42.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:42.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.576785+0000", 2024-09-18T07:32:42.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0539999999999997e-06 2024-09-18T07:32:42.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:42.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.582040+0000", 2024-09-18T07:32:42.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052552800000000002 2024-09-18T07:32:42.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:42.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.582061+0000", 2024-09-18T07:32:42.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1171000000000001e-05 2024-09-18T07:32:42.976 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:42.977 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:42.977 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:42.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.977 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:42074 7.e 7:71bd0b85:::benchmark_data_smithi137_74180_object42073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-18T07:32:42.977 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:41.586025+0000", 2024-09-18T07:32:42.977 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0048521049999999998, 2024-09-18T07:32:42.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050891110000000003, 2024-09-18T07:32:42.977 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:42.977 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:42.977 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:32:42.977 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:42.978 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:42.978 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:42.978 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 42074 2024-09-18T07:32:42.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.978 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:42.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:42.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.586025+0000", 2024-09-18T07:32:42.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:42.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:42.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.586025+0000", 2024-09-18T07:32:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.586027+0000", 2024-09-18T07:32:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3630000000000001e-06 2024-09-18T07:32:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.586057+0000", 2024-09-18T07:32:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9697e-05 2024-09-18T07:32:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:42.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.586071+0000", 2024-09-18T07:32:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3572e-05 2024-09-18T07:32:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.586080+0000", 2024-09-18T07:32:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8929999999999994e-06 2024-09-18T07:32:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.586144+0000", 2024-09-18T07:32:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4178999999999997e-05 2024-09-18T07:32:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:42.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.586279+0000", 2024-09-18T07:32:42.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013569 2024-09-18T07:32:42.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:32:42.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.586424+0000", 2024-09-18T07:32:42.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014462899999999999 2024-09-18T07:32:42.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:32:42.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589667+0000", 2024-09-18T07:32:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003243227 2024-09-18T07:32:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:32:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.590844+0000", 2024-09-18T07:32:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011764740000000001 2024-09-18T07:32:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:42075 7.c e98/94 7:315cb188:::benchmark_data_smithi137_74180_object42074:head v 98'2633, mlcod=88'824)", 2024-09-18T07:32:42.982 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:41.586525+0000", 2024-09-18T07:32:42.983 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.004351761, 2024-09-18T07:32:42.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046525810000000003, 2024-09-18T07:32:42.983 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:42.983 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:42.983 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:32:42.983 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:42.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:42.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.586525+0000", 2024-09-18T07:32:42.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:42.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.984 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:42.984 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.586525+0000", 2024-09-18T07:32:42.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:42.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.984 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:42.984 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.586534+0000", 2024-09-18T07:32:42.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2470000000000008e-06 2024-09-18T07:32:42.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.984 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:42.984 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.586580+0000", 2024-09-18T07:32:42.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6459999999999999e-05 2024-09-18T07:32:42.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:42.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.586593+0000", 2024-09-18T07:32:42.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2928999999999999e-05 2024-09-18T07:32:42.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:42.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.586600+0000", 2024-09-18T07:32:42.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3889999999999999e-06 2024-09-18T07:32:42.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:42.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.587021+0000", 2024-09-18T07:32:42.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00042047800000000001 2024-09-18T07:32:42.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:42.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.587046+0000", 2024-09-18T07:32:42.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5236999999999999e-05 2024-09-18T07:32:42.986 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:42.986 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:42.986 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:42.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.986 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:42077 7.8 7:170abed5:::benchmark_data_smithi137_74180_object42076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-18T07:32:42.986 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:41.589109+0000", 2024-09-18T07:32:42.986 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0017682240000000001, 2024-09-18T07:32:42.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0021163649999999998, 2024-09-18T07:32:42.986 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:42.987 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:42.987 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:32:42.987 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:42.987 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:42.987 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:42.987 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 42077 2024-09-18T07:32:42.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.987 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:42.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:42.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589109+0000", 2024-09-18T07:32:42.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:42.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:42.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589109+0000", 2024-09-18T07:32:42.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:42.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:42.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589111+0000", 2024-09-18T07:32:42.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1280000000000002e-06 2024-09-18T07:32:42.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:42.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589136+0000", 2024-09-18T07:32:42.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4712999999999999e-05 2024-09-18T07:32:42.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:42.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589147+0000", 2024-09-18T07:32:42.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1270999999999999e-05 2024-09-18T07:32:42.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:42.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589155+0000", 2024-09-18T07:32:42.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7549999999999994e-06 2024-09-18T07:32:42.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.990 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:42.990 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.591120+0000", 2024-09-18T07:32:42.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019648310000000002 2024-09-18T07:32:42.990 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:42.990 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:42.990 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:42.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.990 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:42076 7.c e98/94 7:3bcec612:::benchmark_data_smithi137_74180_object42075:head v 98'2634, mlcod=88'824)", 2024-09-18T07:32:42.990 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:41.589218+0000", 2024-09-18T07:32:42.990 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0016588289999999999, 2024-09-18T07:32:42.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002060442, 2024-09-18T07:32:42.990 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:42.991 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:42.991 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:32:42.991 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:42.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.991 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:42.991 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589218+0000", 2024-09-18T07:32:42.991 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:42.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.991 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:42.991 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589218+0000", 2024-09-18T07:32:42.991 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:42.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:42.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589220+0000", 2024-09-18T07:32:42.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4139999999999999e-06 2024-09-18T07:32:42.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:42.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589278+0000", 2024-09-18T07:32:42.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8498000000000001e-05 2024-09-18T07:32:42.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:42.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589291+0000", 2024-09-18T07:32:42.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2551000000000001e-05 2024-09-18T07:32:42.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:42.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589298+0000", 2024-09-18T07:32:42.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0090000000000001e-06 2024-09-18T07:32:42.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:32:42.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589538+0000", 2024-09-18T07:32:42.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000240289 2024-09-18T07:32:42.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:32:42.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589561+0000", 2024-09-18T07:32:42.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2688999999999999e-05 2024-09-18T07:32:42.993 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:42.993 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:42.994 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:42.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.994 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:42079 7.8 7.c50cee08 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-18T07:32:42.994 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:41.589624+0000", 2024-09-18T07:32:42.994 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.001252904, 2024-09-18T07:32:42.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017043220000000001, 2024-09-18T07:32:42.994 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:32:42.994 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:32:42.994 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:32:42.994 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:32:42.994 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:32:42.994 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:32:42.995 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 42079 2024-09-18T07:32:42.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.995 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:32:42.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:32:42.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589624+0000", 2024-09-18T07:32:42.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:42.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:32:42.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589624+0000", 2024-09-18T07:32:42.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:32:42.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:32:42.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589625+0000", 2024-09-18T07:32:42.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8999999999999995e-07 2024-09-18T07:32:42.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:32:42.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589646+0000", 2024-09-18T07:32:42.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0335000000000001e-05 2024-09-18T07:32:42.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:32:42.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589653+0000", 2024-09-18T07:32:42.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0890000000000002e-06 2024-09-18T07:32:42.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:32:42.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:32:42.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:32:42.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:41.589658+0000", 2024-09-18T07:32:42.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1580000000000004e-06 2024-09-18T07:32:42.997 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:42.997 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:32:42.997 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:42.997 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:32:42.997 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:32:42.997 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 8 2024-09-18T07:32:42.997 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:32:42.998 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.0 to 1 (10000) 2024-09-18T07:32:42.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:42.999 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:42.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:43.100 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:43.124 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:43.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:43.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:42 smithi137 ceph-mon[29408]: pgmap v66: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.3 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 861 op/s; 1912/114744 objects degraded (1.666%); 6315/114744 objects misplaced (5.504%); 11 MiB/s, 183 objects/s recovering 2024-09-18T07:32:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:42 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/42632445' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:32:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:42 smithi137 ceph-mon[29408]: from='client.24805 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:32:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:42 smithi137 ceph-mon[39175]: pgmap v66: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.3 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 861 op/s; 1912/114744 objects degraded (1.666%); 6315/114744 objects misplaced (5.504%); 11 MiB/s, 183 objects/s recovering 2024-09-18T07:32:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:42 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/42632445' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:32:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:42 smithi137 ceph-mon[39175]: from='client.24805 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:32:43.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:43.246 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:43.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:43.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:43.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:43.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:43.397 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:43.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:43.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:42 smithi203 ceph-mon[32237]: pgmap v66: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.3 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 861 op/s; 1912/114744 objects degraded (1.666%); 6315/114744 objects misplaced (5.504%); 11 MiB/s, 183 objects/s recovering 2024-09-18T07:32:43.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:42 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/42632445' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:32:43.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:42 smithi203 ceph-mon[32237]: from='client.24805 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:32:43.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:43.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:43.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:43.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:43.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:43.664 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:43.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:43.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:43.777 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 28 16 44473 44457 99.2017 63.0625 0.00640885 0.0100739 2024-09-18T07:32:43.810 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:43.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:43.912 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:43.950 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:43.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:44.051 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:44.077 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:44.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:44.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:44.179 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:44.206 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:44.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:44.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:44.308 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:44.329 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:44.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:44.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:44.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:43 smithi137 ceph-mon[29408]: from='client.24805 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:32:44.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:44 smithi137 ceph-mon[29408]: osdmap e99: 8 total, 8 up, 7 in 2024-09-18T07:32:44.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:43 smithi137 ceph-mon[39175]: from='client.24805 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:32:44.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:44 smithi137 ceph-mon[39175]: osdmap e99: 8 total, 8 up, 7 in 2024-09-18T07:32:44.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:44.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:43 smithi203 ceph-mon[32237]: from='client.24805 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:32:44.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:43 smithi203 ceph-mon[32237]: osdmap e99: 8 total, 8 up, 7 in 2024-09-18T07:32:44.472 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:44.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:44.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:44.574 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:44.595 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:44.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:44.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:44.697 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:44.723 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:44.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:44.777 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 29 16 45562 45546 98.128 68.0625 0.0137813 0.0101834 2024-09-18T07:32:44.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:44.864 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:44.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:44.965 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:45.003 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:45.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:45.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:45.105 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:45.135 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:45.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:45.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:45.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:45.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:45.275 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:45.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:45.316 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:44 smithi137 ceph-mon[29408]: pgmap v68: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.6 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 931 op/s; 1912/126297 objects degraded (1.514%); 5525/126297 objects misplaced (4.375%); 13 MiB/s, 205 objects/s recovering 2024-09-18T07:32:45.316 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:44 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 1912/126297 objects degraded (1.514%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:32:45.316 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:44 smithi137 ceph-mon[29408]: osdmap e100: 8 total, 8 up, 7 in 2024-09-18T07:32:45.316 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:44 smithi137 ceph-mon[39175]: pgmap v68: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.6 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 931 op/s; 1912/126297 objects degraded (1.514%); 5525/126297 objects misplaced (4.375%); 13 MiB/s, 205 objects/s recovering 2024-09-18T07:32:45.316 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:44 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 1912/126297 objects degraded (1.514%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:32:45.316 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:44 smithi137 ceph-mon[39175]: osdmap e100: 8 total, 8 up, 7 in 2024-09-18T07:32:45.317 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:45.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:45.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:45.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:45.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:45.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:44 smithi203 ceph-mon[32237]: pgmap v68: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.6 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 931 op/s; 1912/126297 objects degraded (1.514%); 5525/126297 objects misplaced (4.375%); 13 MiB/s, 205 objects/s recovering 2024-09-18T07:32:45.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:44 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 1912/126297 objects degraded (1.514%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:32:45.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:44 smithi203 ceph-mon[32237]: osdmap e100: 8 total, 8 up, 7 in 2024-09-18T07:32:45.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:45.556 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:45.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:45.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:45.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:45.681 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:45.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:45.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:45.777 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 30 16 46642 46626 97.1072 67.5 0.0124314 0.0102915 2024-09-18T07:32:45.783 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:45.825 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:45.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:45.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:45.927 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:45.952 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:45.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:45.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:46.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:46.095 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:46.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:46.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:46.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:46.223 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:46.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:46.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:46.325 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:46.351 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:46.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:46.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:46.452 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:46.478 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:46.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:46.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:46.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:46.608 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:46.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:46.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:46.709 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:46.736 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:46.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:46.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:46.777 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 31 16 47713 47697 96.134 66.9375 0.0104016 0.0103963 2024-09-18T07:32:46.838 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:46.860 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:46.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:46.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:46.962 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:47.005 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:47.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:47.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:47.107 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:47.132 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:47.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:47.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:47.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:46 smithi203 ceph-mon[32237]: pgmap v70: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.6 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 931 op/s; 1912/126297 objects degraded (1.514%); 5525/126297 objects misplaced (4.375%); 13 MiB/s, 205 objects/s recovering 2024-09-18T07:32:47.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:32:46 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:32:47.234 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:47.272 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:47.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:47.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:47.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:47.400 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:47.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:47.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:47.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:32:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:32:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:32:47.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:46 smithi137 ceph-mon[29408]: pgmap v70: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.6 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 931 op/s; 1912/126297 objects degraded (1.514%); 5525/126297 objects misplaced (4.375%); 13 MiB/s, 205 objects/s recovering 2024-09-18T07:32:47.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:46 smithi137 ceph-mon[39175]: pgmap v70: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.6 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 931 op/s; 1912/126297 objects degraded (1.514%); 5525/126297 objects misplaced (4.375%); 13 MiB/s, 205 objects/s recovering 2024-09-18T07:32:47.502 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:47.526 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:47.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:47.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:47.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:47.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:47.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:47.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:47.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:47.778 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 32 16 48751 48735 95.1536 64.875 0.0077568 0.0105027 2024-09-18T07:32:47.796 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:47.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:47.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:47.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:47.941 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:47.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:47.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:48.043 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:48.081 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:48.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:48.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:48.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:48.226 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:48.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:48.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:48.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:48.374 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:48.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:48.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:48.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:47 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:48.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:47 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:48.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:47 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:48.476 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:48.501 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:48.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:48.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:48.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:48.647 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:48.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:48.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:48.748 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:48.773 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:48.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:48.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:48.779 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 33 16 49884 49868 94.416 70.8125 0.00941884 0.0105861 2024-09-18T07:32:48.875 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:48.912 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:48.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:48.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:49.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:49.053 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:49.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:49.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:49.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:49.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:48 smithi137 ceph-mon[29408]: pgmap v71: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.9 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.3 KiB/s rd, 90 MiB/s wr, 1.43k op/s; 1912/142557 objects degraded (1.341%); 4309/142557 objects misplaced (3.023%); 20 MiB/s, 319 objects/s recovering 2024-09-18T07:32:49.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:49 smithi137 ceph-mon[39175]: pgmap v71: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.9 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.3 KiB/s rd, 90 MiB/s wr, 1.43k op/s; 1912/142557 objects degraded (1.341%); 4309/142557 objects misplaced (3.023%); 20 MiB/s, 319 objects/s recovering 2024-09-18T07:32:49.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:49.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:49.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:49.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:49.305 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:49.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:49.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:49.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:49.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:49.450 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:49.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:49.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:48 smithi203 ceph-mon[32237]: pgmap v71: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.9 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.3 KiB/s rd, 90 MiB/s wr, 1.43k op/s; 1912/142557 objects degraded (1.341%); 4309/142557 objects misplaced (3.023%); 20 MiB/s, 319 objects/s recovering 2024-09-18T07:32:49.552 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:49.578 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:49.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:49.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:49.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:49.720 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:49.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:49.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:49.779 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 34 16 50959 50943 93.6152 67.1875 0.00496581 0.0106756 2024-09-18T07:32:49.823 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:49.861 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:49.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:49.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:49.965 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:49.992 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:49.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:50.094 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:50.115 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:50.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:50.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:50.217 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:50.246 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:50.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:50.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:50.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:50.389 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:50.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:50.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:50.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:50 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 1912/142557 objects degraded (1.341%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:32:50.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:50 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 1912/142557 objects degraded (1.341%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:32:50.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:50 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 1912/142557 objects degraded (1.341%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:32:50.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:50.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:50.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:50.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:50.631 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:50.699 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:50.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:50.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:50.779 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 35 16 51795 51779 92.4334 52.25 0.00815487 0.0107801 2024-09-18T07:32:50.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:50.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:50.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:50.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:50.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:50.976 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:50.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:51.077 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:51.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:51.106 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:51.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:51.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:51.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:51.246 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:51.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:51.347 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:51.361 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:51 smithi203 ceph-mon[32237]: pgmap v72: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.9 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 1912/142557 objects degraded (1.341%); 4309/142557 objects misplaced (3.023%); 16 MiB/s, 250 objects/s recovering 2024-09-18T07:32:51.372 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:51.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:51.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:51.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:51 smithi137 ceph-mon[29408]: pgmap v72: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.9 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 1912/142557 objects degraded (1.341%); 4309/142557 objects misplaced (3.023%); 16 MiB/s, 250 objects/s recovering 2024-09-18T07:32:51.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:51 smithi137 ceph-mon[39175]: pgmap v72: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 2.9 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 1912/142557 objects degraded (1.341%); 4309/142557 objects misplaced (3.023%); 16 MiB/s, 250 objects/s recovering 2024-09-18T07:32:51.474 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:51.499 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:51.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:51.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:51.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:51.645 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:51.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:51.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:51.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:51.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:51.778 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:51.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:51.779 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 36 16 52693 52677 91.4249 56.125 0.00303483 0.0109287 2024-09-18T07:32:51.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:51.916 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:51.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:51.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:52.018 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:52.043 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:52.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:52.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:52.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:52.189 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:52.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:52.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:52.291 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:52.317 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:52.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:52.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:52.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:52 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:32:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:52 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:32:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:52 smithi137 ceph-mon[29408]: osdmap e101: 8 total, 8 up, 7 in 2024-09-18T07:32:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:52 smithi137 ceph-mon[29408]: pgmap v74: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 3.0 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 866 op/s; 1912/147096 objects degraded (1.300%); 3791/147096 objects misplaced (2.577%); 14 MiB/s, 216 objects/s recovering 2024-09-18T07:32:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:52 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:32:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:52 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:32:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:52 smithi137 ceph-mon[39175]: osdmap e101: 8 total, 8 up, 7 in 2024-09-18T07:32:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:52 smithi137 ceph-mon[39175]: pgmap v74: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 3.0 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 866 op/s; 1912/147096 objects degraded (1.300%); 3791/147096 objects misplaced (2.577%); 14 MiB/s, 216 objects/s recovering 2024-09-18T07:32:52.445 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:52.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:52.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:52 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:32:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:52 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:32:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:52 smithi203 ceph-mon[32237]: osdmap e101: 8 total, 8 up, 7 in 2024-09-18T07:32:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:52 smithi203 ceph-mon[32237]: pgmap v74: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 3.0 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 866 op/s; 1912/147096 objects degraded (1.300%); 3791/147096 objects misplaced (2.577%); 14 MiB/s, 216 objects/s recovering 2024-09-18T07:32:52.547 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:52.572 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:52.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:52.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:52.727 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:52.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:52.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:52.779 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 37 16 53741 53725 90.7242 65.5 0.0109288 0.0110163 2024-09-18T07:32:52.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:52.867 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:52.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:52.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:52.968 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:52.994 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:52.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:52.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:53.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:53.122 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:53.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:53.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:53.224 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:53.248 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:53.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:53.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:53.349 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:53.380 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:53.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:53.481 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:53.507 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:53.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:53.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:53.608 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:53.630 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:53.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:53.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:53.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:53.761 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:53.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:53.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:53.780 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 38 16 54388 54372 89.4009 40.4375 0.00285277 0.0110304 2024-09-18T07:32:53.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:53.891 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:53.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:53.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:53.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:54.020 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:54.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:54.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:54.122 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:54.144 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:54.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:54.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:54.247 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:54.272 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:54.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:54.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:54.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:54.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:54.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:54.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:54.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:54 smithi137 ceph-mon[29408]: 6.15 scrub starts 2024-09-18T07:32:54.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:54 smithi137 ceph-mon[29408]: 6.15 scrub ok 2024-09-18T07:32:54.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:54 smithi137 ceph-mon[29408]: osdmap e102: 8 total, 8 up, 7 in 2024-09-18T07:32:54.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:54 smithi137 ceph-mon[29408]: pgmap v76: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 3.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 1912/157563 objects degraded (1.213%); 3044/157563 objects misplaced (1.932%); 19 MiB/s, 310 objects/s recovering 2024-09-18T07:32:54.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:54 smithi137 ceph-mon[39175]: 6.15 scrub starts 2024-09-18T07:32:54.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:54 smithi137 ceph-mon[39175]: 6.15 scrub ok 2024-09-18T07:32:54.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:54 smithi137 ceph-mon[39175]: osdmap e102: 8 total, 8 up, 7 in 2024-09-18T07:32:54.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:54 smithi137 ceph-mon[39175]: pgmap v76: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 3.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 1912/157563 objects degraded (1.213%); 3044/157563 objects misplaced (1.932%); 19 MiB/s, 310 objects/s recovering 2024-09-18T07:32:54.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:54 smithi203 ceph-mon[32237]: 6.15 scrub starts 2024-09-18T07:32:54.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:54 smithi203 ceph-mon[32237]: 6.15 scrub ok 2024-09-18T07:32:54.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:54 smithi203 ceph-mon[32237]: osdmap e102: 8 total, 8 up, 7 in 2024-09-18T07:32:54.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:54 smithi203 ceph-mon[32237]: pgmap v76: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 3.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 1912/157563 objects degraded (1.213%); 3044/157563 objects misplaced (1.932%); 19 MiB/s, 310 objects/s recovering 2024-09-18T07:32:54.520 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:54.546 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:54.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:54.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:54.647 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:54.675 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:54.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:54.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:54.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:54.785 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 39 16 55068 55052 88.1941 42.5 0.0368727 0.011329 2024-09-18T07:32:54.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-18T07:32:54.839 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:54.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:54.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:54.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:54.989 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:54.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:54.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:55.091 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:55.146 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:55.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:55.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:55.248 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:55.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:55.300 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:55.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:55.401 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:55.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:55 smithi137 ceph-mon[29408]: osdmap e103: 8 total, 8 up, 7 in 2024-09-18T07:32:55.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:55 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 1912/157563 objects degraded (1.213%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:32:55.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:55 smithi137 ceph-mon[29408]: osdmap e104: 8 total, 8 up, 7 in 2024-09-18T07:32:55.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:55 smithi137 ceph-mon[39175]: osdmap e103: 8 total, 8 up, 7 in 2024-09-18T07:32:55.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:55 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 1912/157563 objects degraded (1.213%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:32:55.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:55 smithi137 ceph-mon[39175]: osdmap e104: 8 total, 8 up, 7 in 2024-09-18T07:32:55.425 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:55.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:55.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:55.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:55 smithi203 ceph-mon[32237]: osdmap e103: 8 total, 8 up, 7 in 2024-09-18T07:32:55.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:55 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 1912/157563 objects degraded (1.213%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:32:55.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:55 smithi203 ceph-mon[32237]: osdmap e104: 8 total, 8 up, 7 in 2024-09-18T07:32:55.527 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:55.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:55.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:55.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:55.673 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:55.698 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:55.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:55.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:55.781 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:32:55.780085+0000 min lat: 0.000980073 max lat: 2.1848 avg lat: 0.0114011 2024-09-18T07:32:55.781 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:32:55.781 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 40 16 56140 56124 87.6643 67 0.00352643 0.0114011 2024-09-18T07:32:55.800 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:55.823 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:55.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:55.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:55.846 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T07:32:55.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:55.978 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:55.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:55.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:56.079 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:56.080 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:56.149 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:56.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:56.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:56.251 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:56.275 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:56.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:56.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:56.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:56.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:56 smithi137 ceph-mon[29408]: 6.3 scrub starts 2024-09-18T07:32:56.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:56 smithi137 ceph-mon[29408]: 6.3 scrub ok 2024-09-18T07:32:56.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:56 smithi137 ceph-mon[29408]: pgmap v79: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 3.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s; 1912/157563 objects degraded (1.213%); 3044/157563 objects misplaced (1.932%); 18 MiB/s, 294 objects/s recovering 2024-09-18T07:32:56.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:56 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:32:56.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:56 smithi137 ceph-mon[39175]: 6.3 scrub starts 2024-09-18T07:32:56.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:56 smithi137 ceph-mon[39175]: 6.3 scrub ok 2024-09-18T07:32:56.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:56 smithi137 ceph-mon[39175]: pgmap v79: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 3.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s; 1912/157563 objects degraded (1.213%); 3044/157563 objects misplaced (1.932%); 18 MiB/s, 294 objects/s recovering 2024-09-18T07:32:56.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:56 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:32:56.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:56.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:56.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:56.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:56 smithi203 ceph-mon[32237]: 6.3 scrub starts 2024-09-18T07:32:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:56 smithi203 ceph-mon[32237]: 6.3 scrub ok 2024-09-18T07:32:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:56 smithi203 ceph-mon[32237]: pgmap v79: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 143 active+clean; 3.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s; 1912/157563 objects degraded (1.213%); 3044/157563 objects misplaced (1.932%); 18 MiB/s, 294 objects/s recovering 2024-09-18T07:32:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:56 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:32:56.525 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:56.550 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:56.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:56.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:56.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:56.677 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:56.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:56.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:56.779 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:56.781 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 41 16 57075 57059 86.9508 58.4375 0.0153264 0.0114935 2024-09-18T07:32:56.802 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:56.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:56.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:56.883 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:32:56 smithi137 systemd[1]: Stopping Ceph alertmanager.a for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:32:56.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:56.953 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:56.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:56.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:57.055 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:32:57.078 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:57.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:57.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:32:57.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:32:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:32:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:32:57.179 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:57.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:32:56 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:32:57.239 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:57.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:57.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:57.341 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:57.364 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:57.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:57.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:57.417 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:32:57.418 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:32:57.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:57 smithi203 ceph-mon[32237]: 6.5 scrub starts 2024-09-18T07:32:57.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:57 smithi203 ceph-mon[32237]: 6.5 scrub ok 2024-09-18T07:32:57.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:57 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:57.466 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:57.473 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:32:57.473 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:32:57.473 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:57 smithi137 ceph-mon[29408]: 6.5 scrub starts 2024-09-18T07:32:57.473 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:57 smithi137 ceph-mon[29408]: 6.5 scrub ok 2024-09-18T07:32:57.473 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:57 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:57.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:57 smithi137 ceph-mon[39175]: 6.5 scrub starts 2024-09-18T07:32:57.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:57 smithi137 ceph-mon[39175]: 6.5 scrub ok 2024-09-18T07:32:57.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:57 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:32:57.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:57.489 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:57.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:57.517 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:32:57.518 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:32:57.563 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:32:57.563 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:32:57.591 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:57.603 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:32:57.603 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:32:57.613 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:57.619 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:32:57.645 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:32:57.645 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:32:57.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:57.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:57.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:57.684 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:32:57.684 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:32:57.722 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:32:57.723 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:32:57.757 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:57.782 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 42 16 57801 57785 85.9611 45.375 0.0254047 0.0116247 2024-09-18T07:32:57.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:57.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:57.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:57.914 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:57.939 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:57.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:57.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:58.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:58.077 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:58.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:58.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:58.179 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:58.202 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:58.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:58.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:58.204 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:32:58.232 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:58 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/905704778' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:32:58.232 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:58 smithi137 ceph-mon[29408]: pgmap v80: 148 pgs: 4 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.7 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 3023/170826 objects misplaced (1.770%); 28 MiB/s, 446 objects/s recovering 2024-09-18T07:32:58.232 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:58 smithi137 ceph-mon[29408]: 5.9 scrub starts 2024-09-18T07:32:58.232 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:58 smithi137 ceph-mon[29408]: 5.9 scrub ok 2024-09-18T07:32:58.232 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:58 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3709005878' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:32:58.233 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:58 smithi137 ceph-mon[29408]: from='client.24814 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:32:58.233 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:58 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/905704778' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:32:58.233 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:58 smithi137 ceph-mon[39175]: pgmap v80: 148 pgs: 4 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.7 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 3023/170826 objects misplaced (1.770%); 28 MiB/s, 446 objects/s recovering 2024-09-18T07:32:58.233 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:58 smithi137 ceph-mon[39175]: 5.9 scrub starts 2024-09-18T07:32:58.233 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:58 smithi137 ceph-mon[39175]: 5.9 scrub ok 2024-09-18T07:32:58.233 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:58 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3709005878' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:32:58.233 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:58 smithi137 ceph-mon[39175]: from='client.24814 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:32:58.305 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:58.339 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:58.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:58.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:58.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:58.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:58 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/905704778' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:32:58.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:58 smithi203 ceph-mon[32237]: pgmap v80: 148 pgs: 4 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.7 KiB/s rd, 82 MiB/s wr, 1.32k op/s; 3023/170826 objects misplaced (1.770%); 28 MiB/s, 446 objects/s recovering 2024-09-18T07:32:58.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:58 smithi203 ceph-mon[32237]: 5.9 scrub starts 2024-09-18T07:32:58.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:58 smithi203 ceph-mon[32237]: 5.9 scrub ok 2024-09-18T07:32:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:58 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3709005878' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:32:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:58 smithi203 ceph-mon[32237]: from='client.24814 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:32:58.470 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:58.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:58.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:58.573 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:58.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:58.599 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:58.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:58.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:58.725 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:58.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:58.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:58.782 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 43 16 58465 58449 84.9273 41.5 0.00287248 0.0116545 2024-09-18T07:32:58.827 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:32:58.852 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:58.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:58.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:32:58.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:58.976 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:58.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:58.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:59.077 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:32:59.102 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:59.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:59.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:32:59.204 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:32:59.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:59.242 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:59.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:32:59.340 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:32:59.340 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:32:59.343 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:32:59.381 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:32:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:59.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:32:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:59 smithi137 ceph-mon[29408]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1912/157563 objects degraded (1.213%), 1 pg degraded) 2024-09-18T07:32:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:59 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:32:59.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:59 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:32:59.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:59 smithi137 ceph-mon[29408]: from='client.24814 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:32:59.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:32:59 smithi137 ceph-mon[29408]: osdmap e105: 8 total, 8 up, 7 in 2024-09-18T07:32:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:59 smithi137 ceph-mon[39175]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1912/157563 objects degraded (1.213%), 1 pg degraded) 2024-09-18T07:32:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:59 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:32:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:59 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:32:59.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:59 smithi137 ceph-mon[39175]: from='client.24814 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:32:59.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:32:59 smithi137 ceph-mon[39175]: osdmap e105: 8 total, 8 up, 7 in 2024-09-18T07:32:59.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:59 smithi203 ceph-mon[32237]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1912/157563 objects degraded (1.213%), 1 pg degraded) 2024-09-18T07:32:59.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:59 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:32:59.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:59 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:32:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:59 smithi203 ceph-mon[32237]: from='client.24814 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:32:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:32:59 smithi203 ceph-mon[32237]: osdmap e105: 8 total, 8 up, 7 in 2024-09-18T07:32:59.484 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:59.508 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:59.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:59.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:59.610 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:32:59.634 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:59.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:59.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:32:59.737 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:32:59.760 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:59.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:32:59.782 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 44 16 59104 59088 83.9049 39.9375 0.0112864 0.0119107 2024-09-18T07:32:59.862 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:32:59.886 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:32:59.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:32:59.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:32:59.988 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:00.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:00.013 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:00.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:00.114 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:00.140 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:00.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:00.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:00.211 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.1 to 1 (10000) 2024-09-18T07:33:00.243 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:00.265 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:00 smithi137 ceph-mon[39175]: osdmap e106: 8 total, 8 up, 7 in 2024-09-18T07:33:00.265 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:00 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/990134036' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:33:00.265 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:00 smithi137 ceph-mon[39175]: from='client.24820 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:33:00.265 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:00 smithi137 ceph-mon[39175]: pgmap v83: 148 pgs: 4 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 970 B/s rd, 52 MiB/s wr, 839 op/s; 3023/170826 objects misplaced (1.770%); 23 MiB/s, 366 objects/s recovering 2024-09-18T07:33:00.266 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:00.266 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:00 smithi137 ceph-mon[29408]: osdmap e106: 8 total, 8 up, 7 in 2024-09-18T07:33:00.266 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:00 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/990134036' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:33:00.266 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:00 smithi137 ceph-mon[29408]: from='client.24820 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:33:00.266 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:00 smithi137 ceph-mon[29408]: pgmap v83: 148 pgs: 4 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 970 B/s rd, 52 MiB/s wr, 839 op/s; 3023/170826 objects misplaced (1.770%); 23 MiB/s, 366 objects/s recovering 2024-09-18T07:33:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:00.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:00.368 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:00.394 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:00.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:00.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:00.496 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:00.520 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:00.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:00.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:00.602 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:00 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a[71635]: ts=2024-09-18T07:33:00.262Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-18T07:33:00.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:00.645 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:00.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:00.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:00.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:00 smithi203 ceph-mon[32237]: osdmap e106: 8 total, 8 up, 7 in 2024-09-18T07:33:00.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:00 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/990134036' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:33:00.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:00 smithi203 ceph-mon[32237]: from='client.24820 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:33:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:00 smithi203 ceph-mon[32237]: pgmap v83: 148 pgs: 4 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 970 B/s rd, 52 MiB/s wr, 839 op/s; 3023/170826 objects misplaced (1.770%); 23 MiB/s, 366 objects/s recovering 2024-09-18T07:33:00.747 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:00.768 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:00.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:00.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:00.782 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 45 16 59878 59862 83.1154 48.375 0.00256025 0.0119154 2024-09-18T07:33:00.870 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:00.892 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:00.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:00.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:00.993 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:01.015 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:01.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:01.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:01.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:01.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:01.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:01.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:01.219 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:00 smithi137 podman[79071]: 2024-09-18 07:33:00.945734954 +0000 UTC m=+3.277313114 container died a66752344df259ce394b225889590c10faca1d9147a1da5b7450e10c7b602518 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T07:33:01.242 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:01.266 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:01.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:01.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:01.367 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:01.392 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:01.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:01.493 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:01.511 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:01 smithi137 ceph-mon[39175]: from='client.24820 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:33:01.511 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:01 smithi137 ceph-mon[39175]: osdmap e107: 8 total, 8 up, 7 in 2024-09-18T07:33:01.512 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:01 smithi137 podman[79071]: 2024-09-18 07:33:01.329912818 +0000 UTC m=+3.661490973 container cleanup a66752344df259ce394b225889590c10faca1d9147a1da5b7450e10c7b602518 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T07:33:01.512 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:01 smithi137 bash[79071]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a 2024-09-18T07:33:01.512 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:01 smithi137 ceph-mon[29408]: from='client.24820 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:33:01.512 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:01 smithi137 ceph-mon[29408]: osdmap e107: 8 total, 8 up, 7 in 2024-09-18T07:33:01.517 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:01.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:01.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:01.619 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:01.663 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:01.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:01.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:01 smithi203 ceph-mon[32237]: from='client.24820 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:33:01.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:01 smithi203 ceph-mon[32237]: osdmap e107: 8 total, 8 up, 7 in 2024-09-18T07:33:01.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:01.782 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 46 16 60519 60503 82.1796 40.0625 0.00735854 0.0121634 2024-09-18T07:33:01.808 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:01.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:01.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:01.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T07:33:01.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:01.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:01.995 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:01.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:02.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:33:02.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:02.121 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:02.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:02.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:02.148 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:02 smithi137 podman[79343]: 2024-09-18 07:33:02.144831538 +0000 UTC m=+1.824621250 container remove a66752344df259ce394b225889590c10faca1d9147a1da5b7450e10c7b602518 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T07:33:02.223 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:02.251 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:02.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:02.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:02.354 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:02.379 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:02.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:02.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:02.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:02 smithi137 ceph-mon[39175]: osdmap e108: 8 total, 8 up, 7 in 2024-09-18T07:33:02.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:02 smithi137 ceph-mon[39175]: pgmap v86: 148 pgs: 4 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 17 MiB/s wr, 269 op/s; 3023/174060 objects misplaced (1.737%) 2024-09-18T07:33:02.420 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:02 smithi137 podman[79343]: 2024-09-18 07:33:02.245842067 +0000 UTC m=+1.925631783 volume remove 90f5062a21b4a6eb041c68bdf9f4fa45eda53f9d029963a28296ceb0ab190c45 2024-09-18T07:33:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:02 smithi137 ceph-mon[29408]: osdmap e108: 8 total, 8 up, 7 in 2024-09-18T07:33:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:02 smithi137 ceph-mon[29408]: pgmap v86: 148 pgs: 4 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 17 MiB/s wr, 269 op/s; 3023/174060 objects misplaced (1.737%) 2024-09-18T07:33:02.481 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:02.526 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:02.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:02.628 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:02.653 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:02.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:02.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:02.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:02 smithi203 ceph-mon[32237]: osdmap e108: 8 total, 8 up, 7 in 2024-09-18T07:33:02.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:02 smithi203 ceph-mon[32237]: pgmap v86: 148 pgs: 4 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 17 MiB/s wr, 269 op/s; 3023/174060 objects misplaced (1.737%) 2024-09-18T07:33:02.755 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:02.779 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:02.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:02.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:02.782 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 47 16 61642 61626 81.9244 70.1875 0.012708 0.0122003 2024-09-18T07:33:02.882 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:02.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-18T07:33:02.933 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:02.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:02.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:02.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:02.963 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:33:03.040 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:03.092 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:03.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:03.194 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:03.219 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:03.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:03.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:03.374 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:03.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:03.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:03.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:03 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:33:03.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:03 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:33:03.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:03 smithi137 ceph-mon[39175]: osdmap e109: 8 total, 8 up, 7 in 2024-09-18T07:33:03.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:03 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:33:03.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:03 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:33:03.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:03 smithi137 ceph-mon[29408]: osdmap e109: 8 total, 8 up, 7 in 2024-09-18T07:33:03.477 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:03.501 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:03.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:03.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:03.603 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:03.620 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:33:03.620 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:33:03.625 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:03.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:03.629 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:03.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:03 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:33:03.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:03 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:33:03.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:03 smithi203 ceph-mon[32237]: osdmap e109: 8 total, 8 up, 7 in 2024-09-18T07:33:03.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:03.751 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:03.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:03.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:03.782 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 48 16 62416 62400 81.2256 48.375 0.0023988 0.0122279 2024-09-18T07:33:03.853 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:03.875 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:03.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:03.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:03.919 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:03 smithi137 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-18T07:33:03.919 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:03 smithi137 systemd[1]: Stopped Ceph alertmanager.a for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:33:03.919 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:03 smithi137 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.223s CPU time. 2024-09-18T07:33:03.919 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:03 smithi137 systemd[1]: Starting Ceph alertmanager.a for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:33:03.977 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:03.998 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:03.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:04.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:04.100 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:04.122 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:04.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:04.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:04.224 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:04.245 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:04.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:04.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:04.347 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:04.373 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:04.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:04.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:04.393 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:04 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:33:04.393 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:04 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:33:04.393 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:04 smithi137 ceph-mon[29408]: osdmap e110: 8 total, 8 up, 7 in 2024-09-18T07:33:04.393 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:04 smithi137 ceph-mon[29408]: pgmap v89: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 843 op/s; 1224/180936 objects misplaced (0.676%); 28 MiB/s, 449 objects/s recovering 2024-09-18T07:33:04.393 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:04 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:33:04.394 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:04 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:33:04.394 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:04 smithi137 ceph-mon[39175]: osdmap e110: 8 total, 8 up, 7 in 2024-09-18T07:33:04.394 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:04 smithi137 ceph-mon[39175]: pgmap v89: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 843 op/s; 1224/180936 objects misplaced (0.676%); 28 MiB/s, 449 objects/s recovering 2024-09-18T07:33:04.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:04.516 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:04.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:04.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:04.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:04.653 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:04.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:04.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:04.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:04 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:33:04.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:04 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:33:04.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:04 smithi203 ceph-mon[32237]: osdmap e110: 8 total, 8 up, 7 in 2024-09-18T07:33:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:04 smithi203 ceph-mon[32237]: pgmap v89: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 843 op/s; 1224/180936 objects misplaced (0.676%); 28 MiB/s, 449 objects/s recovering 2024-09-18T07:33:04.755 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:04.780 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:04.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:04.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:04.782 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 49 16 63038 63022 80.3614 38.875 0.00982731 0.0124371 2024-09-18T07:33:04.883 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:04.892 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:04.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T07:33:04.952 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:04.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:04.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:05.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:05.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:05.150 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:05.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:05.233 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:05.251 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:05.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:05.276 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:05.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:05.377 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:05.400 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:05.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:05.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:05.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:05 smithi137 ceph-mon[39175]: osdmap e111: 8 total, 8 up, 7 in 2024-09-18T07:33:05.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:05 smithi137 ceph-mon[39175]: osdmap e112: 8 total, 8 up, 7 in 2024-09-18T07:33:05.420 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:05 smithi137 podman[79908]: 2024-09-18 07:33:05.092072957 +0000 UTC m=+0.180753710 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-18T07:33:05.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:05 smithi137 ceph-mon[29408]: osdmap e111: 8 total, 8 up, 7 in 2024-09-18T07:33:05.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:05 smithi137 ceph-mon[29408]: osdmap e112: 8 total, 8 up, 7 in 2024-09-18T07:33:05.502 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:05.526 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:05.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:05.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:05.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:05.674 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:05.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:05.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:05 smithi203 ceph-mon[32237]: osdmap e111: 8 total, 8 up, 7 in 2024-09-18T07:33:05.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:05 smithi203 ceph-mon[32237]: osdmap e112: 8 total, 8 up, 7 in 2024-09-18T07:33:05.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:05.783 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 50 16 64071 64055 80.0454 64.5625 0.0121518 0.0124869 2024-09-18T07:33:05.818 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:05.819 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:05 smithi137 podman[79908]: 2024-09-18 07:33:05.600852507 +0000 UTC m=+0.689533234 volume create 5d6bfb21fbab9079fde370126939652fba56fcf9d542827c00ee40995f6d77bc 2024-09-18T07:33:05.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:05.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:05.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:05.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:05.973 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:05.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:06.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:06.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:06.114 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:06.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:06.169 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:05 smithi137 podman[79908]: 2024-09-18 07:33:05.942880157 +0000 UTC m=+1.031560887 container create b9bd9f49497c2c68d11bab42772f084db06529c5afd0bb679e6d6d5c4a7e1d13 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T07:33:06.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:06.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:06.255 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:06.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:06.295 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.2 to 1 (10000) 2024-09-18T07:33:06.296 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:33:06.356 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:06.381 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:06.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:06.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:06.473 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-mon[29408]: pgmap v92: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 574 op/s; 1224/180936 objects misplaced (0.676%); 28 MiB/s, 449 objects/s recovering 2024-09-18T07:33:06.473 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3344274456' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:06.473 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-mon[29408]: from='client.34805 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:06.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3384739930' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:33:06.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-mon[29408]: from='client.34811 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:33:06.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-mon[39175]: pgmap v92: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 574 op/s; 1224/180936 objects misplaced (0.676%); 28 MiB/s, 449 objects/s recovering 2024-09-18T07:33:06.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3344274456' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:06.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-mon[39175]: from='client.34805 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:06.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3384739930' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:33:06.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-mon[39175]: from='client.34811 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:33:06.475 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:06 smithi137 podman[79908]: 2024-09-18 07:33:06.398677217 +0000 UTC m=+1.487357960 container init b9bd9f49497c2c68d11bab42772f084db06529c5afd0bb679e6d6d5c4a7e1d13 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T07:33:06.475 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:06 smithi137 podman[79908]: 2024-09-18 07:33:06.4096203 +0000 UTC m=+1.498301029 container start b9bd9f49497c2c68d11bab42772f084db06529c5afd0bb679e6d6d5c4a7e1d13 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T07:33:06.475 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a[80142]: ts=2024-09-18T07:33:06.429Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-18T07:33:06.475 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a[80142]: ts=2024-09-18T07:33:06.430Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-18T07:33:06.475 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a[80142]: ts=2024-09-18T07:33:06.431Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.137 port=9094 2024-09-18T07:33:06.475 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a[80142]: ts=2024-09-18T07:33:06.434Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-18T07:33:06.475 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a[80142]: ts=2024-09-18T07:33:06.471Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-18T07:33:06.475 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a[80142]: ts=2024-09-18T07:33:06.471Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-18T07:33:06.484 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:06.509 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:06.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:06.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:06.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:06.663 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:06.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:06.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:06.710 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:33:06 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=infra.usagestats t=2024-09-18T07:33:06.466618307Z level=info msg="Usage stats are ready to report" 2024-09-18T07:33:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:06 smithi203 ceph-mon[32237]: pgmap v92: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 36 MiB/s wr, 574 op/s; 1224/180936 objects misplaced (0.676%); 28 MiB/s, 449 objects/s recovering 2024-09-18T07:33:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:06 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3344274456' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:06 smithi203 ceph-mon[32237]: from='client.34805 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:06 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3384739930' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:33:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:06 smithi203 ceph-mon[32237]: from='client.34811 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:33:06.765 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:06.782 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 51 16 65130 65114 79.7736 66.1875 0.0285822 0.0125298 2024-09-18T07:33:06.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:06.791 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:06.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:06.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:06.919 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a[80142]: ts=2024-09-18T07:33:06.475Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-18T07:33:06.919 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:06 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a[80142]: ts=2024-09-18T07:33:06.475Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-18T07:33:06.920 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:06 smithi137 bash[79908]: b9bd9f49497c2c68d11bab42772f084db06529c5afd0bb679e6d6d5c4a7e1d13 2024-09-18T07:33:06.920 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:06 smithi137 systemd[1]: Started Ceph alertmanager.a for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:33:06.938 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:06.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:07.040 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:07.066 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:07.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:07.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:07.168 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:07.198 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:07.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:07.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:07.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:33:06 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:33:07.300 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:07.324 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:07.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:07.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:07.404 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:33:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:33:07.404 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:33:07.404 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[29408]: from='client.34805 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:33:07.405 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[29408]: from='client.34811 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:33:07.405 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[29408]: osdmap e113: 8 total, 8 up, 7 in 2024-09-18T07:33:07.405 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:07.405 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:07.405 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[29408]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-18T07:33:07.405 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[29408]: Reconfiguring daemon prometheus.a on smithi203 2024-09-18T07:33:07.405 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:07.405 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:33:07.405 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[39175]: from='client.34805 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:33:07.405 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[39175]: from='client.34811 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:33:07.405 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[39175]: osdmap e113: 8 total, 8 up, 7 in 2024-09-18T07:33:07.406 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:07.406 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:07.406 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[39175]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-18T07:33:07.406 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[39175]: Reconfiguring daemon prometheus.a on smithi203 2024-09-18T07:33:07.406 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:07 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:07.425 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:07.451 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:07.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:07.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:07.552 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:07.580 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:07.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:07.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:07.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:07.685 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:07 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:33:07.686 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:07 smithi203 ceph-mon[32237]: from='client.34805 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:33:07.686 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:07 smithi203 ceph-mon[32237]: from='client.34811 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:33:07.686 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:07 smithi203 ceph-mon[32237]: osdmap e113: 8 total, 8 up, 7 in 2024-09-18T07:33:07.686 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:07 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:07.686 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:07 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:07.686 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:07 smithi203 ceph-mon[32237]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-18T07:33:07.686 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:07 smithi203 ceph-mon[32237]: Reconfiguring daemon prometheus.a on smithi203 2024-09-18T07:33:07.686 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:07 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:07.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:07.714 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:07.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:07.782 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 52 16 66397 66381 79.7622 79.1875 0.011443 0.0125318 2024-09-18T07:33:07.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:07.852 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:07.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:07.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:07.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:07.980 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:07.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:07.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:08.082 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:08.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:08.141 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:08.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:08.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-18T07:33:08.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:08.265 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:08.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:08.367 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:08.389 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:08 smithi137 ceph-mon[29408]: pgmap v94: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 591/194955 objects misplaced (0.303%); 9.5 MiB/s, 152 objects/s recovering 2024-09-18T07:33:08.389 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:08 smithi137 ceph-mon[39175]: pgmap v94: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 591/194955 objects misplaced (0.303%); 9.5 MiB/s, 152 objects/s recovering 2024-09-18T07:33:08.391 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:08.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:08.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:08.471 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:08.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:08.529 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:33:08.531 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:33:08.552 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:08.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:08.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:08.571 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:33:08.571 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:33:08.604 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:33:08.605 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:33:08.608 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:08 smithi203 ceph-mon[32237]: pgmap v94: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 591/194955 objects misplaced (0.303%); 9.5 MiB/s, 152 objects/s recovering 2024-09-18T07:33:08.640 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:33:08.640 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:33:08.656 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:08.669 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:08 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a[80142]: ts=2024-09-18T07:33:08.435Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001231449s 2024-09-18T07:33:08.671 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:33:08.671 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:33:08.680 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:08.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:08.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:08.702 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:33:08.702 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:33:08.736 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:33:08.737 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:33:08.771 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:33:08.771 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:33:08.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:08.783 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 53 16 67492 67476 79.5484 68.4375 0.020063 0.0125649 2024-09-18T07:33:08.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:08.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:08.923 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:08.950 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:08.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:09.052 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:09.075 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:09.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:09.177 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:09.203 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:09.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:09.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:09.305 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:09.330 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:09.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:09.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:09.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:09.459 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:09.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:09.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:09.561 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:09.587 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:09.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:09.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:09.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:09 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3254136810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:09.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:09 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3254136810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:09.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:09.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:09 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3254136810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:09.730 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:09.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:09.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:09.783 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 54 16 68630 68614 79.3923 71.125 0.00725527 0.0125892 2024-09-18T07:33:09.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:09.855 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:09.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:09.957 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:09.982 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:09.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:09.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:10.084 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:10.123 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:10.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:10.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:10.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:10.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:10.247 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:10.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:10.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:10.388 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:10.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:10.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:10.461 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:10 smithi203 systemd[1]: Stopping Ceph prometheus.a for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:33:10.490 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:10.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:10.534 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:10.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:10.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:10.677 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:10.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:10.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:10.779 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:10.783 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 55 16 69733 69717 79.2022 68.9375 0.0486839 0.0126158 2024-09-18T07:33:10.806 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:10.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:10.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:10.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:10 smithi203 ceph-mon[32237]: pgmap v95: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 49 MiB/s wr, 779 op/s; 591/194955 objects misplaced (0.303%); 6.6 MiB/s, 105 objects/s recovering 2024-09-18T07:33:10.863 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:10 smithi203 ceph-mon[32237]: osdmap e114: 8 total, 8 up, 7 in 2024-09-18T07:33:10.864 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:33:10.542Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-18T07:33:10.864 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:33:10.542Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-18T07:33:10.864 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:33:10.542Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-18T07:33:10.864 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:33:10.542Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-18T07:33:10.864 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:33:10.542Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-18T07:33:10.865 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:33:10.542Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-18T07:33:10.865 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:33:10.542Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-18T07:33:10.865 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:33:10.542Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-18T07:33:10.865 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:33:10.542Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-18T07:33:10.865 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:33:10.543Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-18T07:33:10.865 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:33:10.543Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-18T07:33:10.865 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[58178]: ts=2024-09-18T07:33:10.543Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-18T07:33:10.865 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:10 smithi203 podman[64880]: 2024-09-18 07:33:10.551842165 +0000 UTC m=+0.138089381 container died ff9b936dfcead466a78ab3046b8e8dbe1612c03cfb5b38fc9a348ba8030d69fb (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T07:33:10.908 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:10.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:33:10.934 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:10.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:10.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:11.036 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:11.061 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:11.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:11.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:11.163 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:11.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:10 smithi137 ceph-mon[29408]: pgmap v95: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 49 MiB/s wr, 779 op/s; 591/194955 objects misplaced (0.303%); 6.6 MiB/s, 105 objects/s recovering 2024-09-18T07:33:11.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:10 smithi137 ceph-mon[29408]: osdmap e114: 8 total, 8 up, 7 in 2024-09-18T07:33:11.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:10 smithi137 ceph-mon[39175]: pgmap v95: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 49 MiB/s wr, 779 op/s; 591/194955 objects misplaced (0.303%); 6.6 MiB/s, 105 objects/s recovering 2024-09-18T07:33:11.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:10 smithi137 ceph-mon[39175]: osdmap e114: 8 total, 8 up, 7 in 2024-09-18T07:33:11.204 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:10 smithi203 podman[64880]: 2024-09-18 07:33:10.992494043 +0000 UTC m=+0.578741250 container cleanup ff9b936dfcead466a78ab3046b8e8dbe1612c03cfb5b38fc9a348ba8030d69fb (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T07:33:11.204 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:10 smithi203 bash[64880]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a 2024-09-18T07:33:11.222 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:11.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:11.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:11.324 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:11.351 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:11.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:11.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:11.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:11.459 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:11 smithi203 podman[64892]: 2024-09-18 07:33:11.202807865 +0000 UTC m=+0.650020422 container remove ff9b936dfcead466a78ab3046b8e8dbe1612c03cfb5b38fc9a348ba8030d69fb (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T07:33:11.487 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:11.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:11.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:11.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:11.628 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:11.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:11.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:11.714 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:11 smithi203 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-18T07:33:11.714 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:11 smithi203 systemd[1]: Stopped Ceph prometheus.a for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:33:11.714 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:11 smithi203 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.153s CPU time. 2024-09-18T07:33:11.714 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:11 smithi203 systemd[1]: Starting Ceph prometheus.a for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:33:11.714 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:11 smithi203 podman[65065]: 2024-09-18 07:33:11.610839774 +0000 UTC m=+0.021149086 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-18T07:33:11.714 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:11 smithi203 podman[65065]: 2024-09-18 07:33:11.712344033 +0000 UTC m=+0.122653340 container create 40a774eaf7fecea80fb1af75d5688351fbad311e6b076215cac95284558d16bd (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T07:33:11.730 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:11.756 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:11.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:11.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:11.783 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 56 16 70820 70804 79.001 67.9375 0.0206547 0.012653 2024-09-18T07:33:11.857 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:11.885 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:11.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:11.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:11.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:12.028 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:12.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:12.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:12.130 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:12.151 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:12.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:12.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:12.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:11 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:12.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:11 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:33:12.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:11 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:12.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:11 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:33:12.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:11 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:12.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:11 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:33:12.254 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:12.280 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:12.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:12.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:12.382 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:12.411 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:12.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:12.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:12.456 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:12.457 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:12.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:12.558 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:12.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:12.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:12.615 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 podman[65065]: 2024-09-18 07:33:12.283133962 +0000 UTC m=+0.693443282 container init 40a774eaf7fecea80fb1af75d5688351fbad311e6b076215cac95284558d16bd (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T07:33:12.615 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 podman[65065]: 2024-09-18 07:33:12.287403438 +0000 UTC m=+0.697712747 container start 40a774eaf7fecea80fb1af75d5688351fbad311e6b076215cac95284558d16bd (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T07:33:12.616 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.316Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-18T07:33:12.616 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.317Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-18T07:33:12.616 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.317Z caller=main.go:623 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi203 (none))" 2024-09-18T07:33:12.616 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.317Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-18T07:33:12.616 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.318Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-18T07:33:12.616 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.321Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-18T07:33:12.616 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.322Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-18T07:33:12.617 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.323Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-18T07:33:12.617 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.324Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-18T07:33:12.617 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.328Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-18T07:33:12.617 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.328Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=4.04µs 2024-09-18T07:33:12.617 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.328Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-18T07:33:12.617 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.328Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-18T07:33:12.617 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.328Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-18T07:33:12.617 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.329Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=38.568µs wal_replay_duration=605.748µs wbl_replay_duration=131ns total_replay_duration=722.383µs 2024-09-18T07:33:12.620 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.330Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-18T07:33:12.620 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.330Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-18T07:33:12.620 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.331Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-18T07:33:12.620 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.364Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=33.333381ms db_storage=1.158µs remote_storage=1.524µs web_handler=571ns query_engine=706ns scrape=993.926µs scrape_sd=386.806µs notify=20.822µs notify_sd=37.195µs rules=31.27077ms tracing=10.237µs 2024-09-18T07:33:12.620 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.364Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-18T07:33:12.620 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:33:12.364Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-18T07:33:12.660 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:12.689 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:12.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:12.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:12.783 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 57 16 71894 71878 78.7925 67.125 0.016674 0.0126861 2024-09-18T07:33:12.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:12.834 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:12.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:12.867 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 bash[65065]: 40a774eaf7fecea80fb1af75d5688351fbad311e6b076215cac95284558d16bd 2024-09-18T07:33:12.867 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:33:12 smithi203 systemd[1]: Started Ceph prometheus.a for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:33:12.935 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:12.964 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:12.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:13.060 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:12] ENGINE Bus STOPPING 2024-09-18T07:33:13.061 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[29408]: pgmap v97: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 766 B/s rd, 60 MiB/s wr, 958 op/s; 591/200136 objects misplaced (0.295%); 5.9 MiB/s, 94 objects/s recovering 2024-09-18T07:33:13.061 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:13.061 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:13.061 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-18T07:33:13.061 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi137.front.sepia.ceph.com:9093"} : dispatch 2024-09-18T07:33:13.061 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:13.061 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-18T07:33:13.061 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi203.front.sepia.ceph.com:9095"} : dispatch 2024-09-18T07:33:13.061 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:13.061 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-18T07:33:13.062 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi203.front.sepia.ceph.com:3000"} : dispatch 2024-09-18T07:33:13.062 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:13.062 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[39175]: pgmap v97: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 766 B/s rd, 60 MiB/s wr, 958 op/s; 591/200136 objects misplaced (0.295%); 5.9 MiB/s, 94 objects/s recovering 2024-09-18T07:33:13.062 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:13.062 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:13.062 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-18T07:33:13.062 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi137.front.sepia.ceph.com:9093"} : dispatch 2024-09-18T07:33:13.062 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:13.062 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-18T07:33:13.062 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi203.front.sepia.ceph.com:9095"} : dispatch 2024-09-18T07:33:13.062 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:13.062 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-18T07:33:13.063 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi203.front.sepia.ceph.com:3000"} : dispatch 2024-09-18T07:33:13.063 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:12 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:13.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:13.109 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:13.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:13.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:13.210 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:13.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-mon[32237]: pgmap v97: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 766 B/s rd, 60 MiB/s wr, 958 op/s; 591/200136 objects misplaced (0.295%); 5.9 MiB/s, 94 objects/s recovering 2024-09-18T07:33:13.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:13.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:13.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-18T07:33:13.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi137.front.sepia.ceph.com:9093"} : dispatch 2024-09-18T07:33:13.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:13.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-18T07:33:13.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi203.front.sepia.ceph.com:9095"} : dispatch 2024-09-18T07:33:13.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:13.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-18T07:33:13.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi203.front.sepia.ceph.com:3000"} : dispatch 2024-09-18T07:33:13.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:12 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:13.240 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:13.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:13.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:13.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:13.363 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:13.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:13.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:13.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:13] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-18T07:33:13.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:13] ENGINE Bus STOPPED 2024-09-18T07:33:13.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:13] ENGINE Bus STARTING 2024-09-18T07:33:13.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:13] ENGINE Serving on http://:::9283 2024-09-18T07:33:13.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:13] ENGINE Bus STARTED 2024-09-18T07:33:13.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:13] ENGINE Bus STOPPING 2024-09-18T07:33:13.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:13] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-18T07:33:13.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:13] ENGINE Bus STOPPED 2024-09-18T07:33:13.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:13] ENGINE Bus STARTING 2024-09-18T07:33:13.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:13] ENGINE Serving on http://:::9283 2024-09-18T07:33:13.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:13] ENGINE Bus STARTED 2024-09-18T07:33:13.421 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:13] ENGINE Bus STOPPING 2024-09-18T07:33:13.464 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:13.495 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:13.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:13.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:13.597 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:13.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:13.628 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:13.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:13.729 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:13.758 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:13.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:13.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:13.783 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 58 16 72842 72826 78.4556 59.25 0.010411 0.0127404 2024-09-18T07:33:13.860 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:13.890 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:13.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:13.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:13.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:14.014 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:14.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:14.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:14.115 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:14.147 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:14.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:14.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:14.151 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:13 smithi203 ceph-mon[32237]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-18T07:33:14.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:13 smithi203 ceph-mon[32237]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi137.front.sepia.ceph.com:9093"}]: dispatch 2024-09-18T07:33:14.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:13 smithi203 ceph-mon[32237]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-18T07:33:14.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:13 smithi203 ceph-mon[32237]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi203.front.sepia.ceph.com:9095"}]: dispatch 2024-09-18T07:33:14.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:13 smithi203 ceph-mon[32237]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-18T07:33:14.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:13 smithi203 ceph-mon[32237]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi203.front.sepia.ceph.com:3000"}]: dispatch 2024-09-18T07:33:14.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:13 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:33:14.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:13 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:33:14.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:13 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:33:14.152 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:13 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:33:14.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:13] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-18T07:33:14.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:13] ENGINE Bus STOPPED 2024-09-18T07:33:14.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:13] ENGINE Bus STARTING 2024-09-18T07:33:14.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:13] ENGINE Serving on http://:::9283 2024-09-18T07:33:14.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:33:13] ENGINE Bus STARTED 2024-09-18T07:33:14.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[29408]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-18T07:33:14.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[29408]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi137.front.sepia.ceph.com:9093"}]: dispatch 2024-09-18T07:33:14.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[29408]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-18T07:33:14.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[29408]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi203.front.sepia.ceph.com:9095"}]: dispatch 2024-09-18T07:33:14.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[29408]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-18T07:33:14.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[29408]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi203.front.sepia.ceph.com:3000"}]: dispatch 2024-09-18T07:33:14.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:33:14.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:33:14.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:33:14.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:33:14.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[39175]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-18T07:33:14.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[39175]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi137.front.sepia.ceph.com:9093"}]: dispatch 2024-09-18T07:33:14.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[39175]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-18T07:33:14.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[39175]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi203.front.sepia.ceph.com:9095"}]: dispatch 2024-09-18T07:33:14.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[39175]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-18T07:33:14.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[39175]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi203.front.sepia.ceph.com:3000"}]: dispatch 2024-09-18T07:33:14.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:33:14.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:33:14.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:33:14.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:13 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:33:14.249 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:14.270 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:14.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:14.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:14.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:33:14.372 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:14.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:14.394 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:14.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:14.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:14.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:14.516 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:14.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:14.617 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:14.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:14.639 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:14.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:14.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:14.763 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:14.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:14.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:14.783 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 59 16 73211 73195 77.5168 23.0625 0.854043 0.0127761 2024-09-18T07:33:14.866 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:14.897 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:14.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:14.999 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:15.024 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:15.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:15.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:15.126 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:15.139 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:14 smithi137 ceph-mon[29408]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T07:33:15.139 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:14 smithi137 ceph-mon[29408]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T07:33:15.139 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:14 smithi137 ceph-mon[29408]: pgmap v98: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 164/211866 objects misplaced (0.077%); 8.3 MiB/s, 132 objects/s recovering 2024-09-18T07:33:15.139 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:14 smithi137 ceph-mon[29408]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T07:33:15.140 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:14 smithi137 ceph-mon[29408]: osdmap e115: 8 total, 8 up, 7 in 2024-09-18T07:33:15.140 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:14 smithi137 ceph-mon[29408]: osdmap e116: 8 total, 8 up, 7 in 2024-09-18T07:33:15.140 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:14 smithi137 ceph-mon[39175]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T07:33:15.140 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:14 smithi137 ceph-mon[39175]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T07:33:15.140 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:14 smithi137 ceph-mon[39175]: pgmap v98: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 164/211866 objects misplaced (0.077%); 8.3 MiB/s, 132 objects/s recovering 2024-09-18T07:33:15.140 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:14 smithi137 ceph-mon[39175]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T07:33:15.140 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:14 smithi137 ceph-mon[39175]: osdmap e115: 8 total, 8 up, 7 in 2024-09-18T07:33:15.140 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:14 smithi137 ceph-mon[39175]: osdmap e116: 8 total, 8 up, 7 in 2024-09-18T07:33:15.151 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:15.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:15.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:14 smithi203 ceph-mon[32237]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T07:33:15.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:14 smithi203 ceph-mon[32237]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T07:33:15.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:14 smithi203 ceph-mon[32237]: pgmap v98: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 164/211866 objects misplaced (0.077%); 8.3 MiB/s, 132 objects/s recovering 2024-09-18T07:33:15.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:14 smithi203 ceph-mon[32237]: from='mon.? v1:172.21.15.203:0/1570182824' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-18T07:33:15.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:14 smithi203 ceph-mon[32237]: osdmap e115: 8 total, 8 up, 7 in 2024-09-18T07:33:15.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:14 smithi203 ceph-mon[32237]: osdmap e116: 8 total, 8 up, 7 in 2024-09-18T07:33:15.253 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:15.310 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:15.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:15.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:15.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:15.437 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:15.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:15.538 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:15.564 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:15.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:15.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:15.666 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:15.692 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:15.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:15.783 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:33:15.782124+0000 min lat: 0.000980073 max lat: 2.1848 avg lat: 0.0129147 2024-09-18T07:33:15.783 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:33:15.783 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 60 16 74342 74326 77.4029 70.6875 0.00806374 0.0129147 2024-09-18T07:33:15.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:15.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:15.837 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:15.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:15.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:15.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:15.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:15.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:16.060 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:16.090 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:16.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:16.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:16.192 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:16.213 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:16.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:16.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:16.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:16.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:16.351 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:16.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:16.452 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:16.478 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:16.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:16.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:16.602 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:16.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:16.703 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:16.729 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:16.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:16.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:16.784 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 61 16 75171 75155 76.9834 51.8125 0.0111749 0.0129827 2024-09-18T07:33:16.831 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:16.868 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:16.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:16.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:16.882 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:33:16 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a[80142]: ts=2024-09-18T07:33:16.439Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004855514s 2024-09-18T07:33:16.949 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:16 smithi203 ceph-mon[32237]: pgmap v101: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 59 MiB/s wr, 940 op/s; 164/211866 objects misplaced (0.077%); 4.4 MiB/s, 71 objects/s recovering 2024-09-18T07:33:16.970 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:17.010 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:17.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:17.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:17.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:17.159 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:17.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:17.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:17.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:16 smithi137 ceph-mon[29408]: pgmap v101: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 59 MiB/s wr, 940 op/s; 164/211866 objects misplaced (0.077%); 4.4 MiB/s, 71 objects/s recovering 2024-09-18T07:33:17.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:16 smithi137 ceph-mon[39175]: pgmap v101: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 59 MiB/s wr, 940 op/s; 164/211866 objects misplaced (0.077%); 4.4 MiB/s, 71 objects/s recovering 2024-09-18T07:33:17.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:33:16 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:33:17.262 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:17.289 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:17.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:17.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:17.390 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:17.431 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:17.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:17.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:17.459 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:33:17.459 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:33:17.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.459 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:74938 7.2 7:4b223bb7:::benchmark_data_smithi137_74180_object74937:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:33:17.460 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:16.535414+0000", 2024-09-18T07:33:17.460 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.053344865999999998, 2024-09-18T07:33:17.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.053372233999999998, 2024-09-18T07:33:17.460 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:33:17.460 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:33:17.460 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:33:17.460 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:33:17.460 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:33:17.460 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:17.460 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 74938 2024-09-18T07:33:17.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.461 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:33:17.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:33:17.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.535414+0000", 2024-09-18T07:33:17.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:33:17.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:33:17.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.535414+0000", 2024-09-18T07:33:17.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:33:17.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:33:17.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.535415+0000", 2024-09-18T07:33:17.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3829999999999999e-06 2024-09-18T07:33:17.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:33:17.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.535437+0000", 2024-09-18T07:33:17.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1375000000000001e-05 2024-09-18T07:33:17.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:33:17.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.535447+0000", 2024-09-18T07:33:17.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0249e-05 2024-09-18T07:33:17.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:33:17.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.535454+0000", 2024-09-18T07:33:17.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3610000000000003e-06 2024-09-18T07:33:17.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:33:17.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.535503+0000", 2024-09-18T07:33:17.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8312000000000001e-05 2024-09-18T07:33:17.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:33:17.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.535620+0000", 2024-09-18T07:33:17.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011744900000000001 2024-09-18T07:33:17.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-18T07:33:17.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.535750+0000", 2024-09-18T07:33:17.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012945400000000001 2024-09-18T07:33:17.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:33:17.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.538841+0000", 2024-09-18T07:33:17.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030911599999999999 2024-09-18T07:33:17.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:17.465 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.541787+0000", 2024-09-18T07:33:17.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002946003 2024-09-18T07:33:17.465 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:33:17.465 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:33:17.465 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:33:17.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.465 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.14865.0:74938 7.2 e116/81)", 2024-09-18T07:33:17.466 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:16.542325+0000", 2024-09-18T07:33:17.466 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.046434077999999997, 2024-09-18T07:33:17.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.046503876, 2024-09-18T07:33:17.466 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:33:17.466 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:33:17.466 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:33:17.466 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:33:17.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:33:17.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542325+0000", 2024-09-18T07:33:17.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:33:17.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:33:17.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542325+0000", 2024-09-18T07:33:17.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:33:17.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:33:17.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542325+0000", 2024-09-18T07:33:17.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8000000000000001e-07 2024-09-18T07:33:17.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:33:17.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542328+0000", 2024-09-18T07:33:17.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7819999999999998e-06 2024-09-18T07:33:17.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:33:17.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542333+0000", 2024-09-18T07:33:17.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8189999999999999e-06 2024-09-18T07:33:17.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:33:17.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542337+0000", 2024-09-18T07:33:17.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3560000000000003e-06 2024-09-18T07:33:17.468 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:33:17.468 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:33:17.469 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:33:17.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.469 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.14865.0:74937 7.a e116/81 ondisk, result = 0)", 2024-09-18T07:33:17.469 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:16.542343+0000", 2024-09-18T07:33:17.469 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.046416331999999998, 2024-09-18T07:33:17.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.046506522000000002, 2024-09-18T07:33:17.469 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:33:17.469 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:33:17.469 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:33:17.469 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:33:17.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:33:17.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542343+0000", 2024-09-18T07:33:17.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:33:17.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.470 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:33:17.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542343+0000", 2024-09-18T07:33:17.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:33:17.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.470 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:33:17.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542343+0000", 2024-09-18T07:33:17.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6199999999999999e-07 2024-09-18T07:33:17.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:33:17.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542345+0000", 2024-09-18T07:33:17.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3130000000000001e-06 2024-09-18T07:33:17.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:33:17.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542349+0000", 2024-09-18T07:33:17.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2459999999999997e-06 2024-09-18T07:33:17.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:33:17.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542353+0000", 2024-09-18T07:33:17.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7730000000000001e-06 2024-09-18T07:33:17.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:33:17.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542826+0000", 2024-09-18T07:33:17.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00047320800000000003 2024-09-18T07:33:17.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:33:17.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542833+0000", 2024-09-18T07:33:17.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1109999999999996e-06 2024-09-18T07:33:17.472 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:33:17.472 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:33:17.472 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:33:17.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.473 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:74941 7.3 7:cc9da09e:::benchmark_data_smithi137_74180_object74940:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:33:17.473 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:16.542363+0000", 2024-09-18T07:33:17.473 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.046395369999999998, 2024-09-18T07:33:17.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.046513337000000002, 2024-09-18T07:33:17.473 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:33:17.473 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:33:17.473 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:33:17.473 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:33:17.473 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:33:17.473 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:17.473 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 74941 2024-09-18T07:33:17.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.474 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:33:17.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:33:17.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542363+0000", 2024-09-18T07:33:17.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:33:17.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:33:17.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542363+0000", 2024-09-18T07:33:17.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:33:17.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:33:17.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542365+0000", 2024-09-18T07:33:17.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6589999999999999e-06 2024-09-18T07:33:17.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:33:17.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542388+0000", 2024-09-18T07:33:17.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3247999999999998e-05 2024-09-18T07:33:17.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:33:17.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542396+0000", 2024-09-18T07:33:17.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3250000000000002e-06 2024-09-18T07:33:17.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:33:17.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542402+0000", 2024-09-18T07:33:17.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.784e-06 2024-09-18T07:33:17.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:33:17.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542487+0000", 2024-09-18T07:33:17.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5159000000000002e-05 2024-09-18T07:33:17.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:33:17.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542700+0000", 2024-09-18T07:33:17.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021377000000000001 2024-09-18T07:33:17.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,4", 2024-09-18T07:33:17.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542851+0000", 2024-09-18T07:33:17.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000150166 2024-09-18T07:33:17.477 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:33:17.477 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:33:17.477 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:33:17.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.477 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:74937 7.a 7:5dd48c69:::benchmark_data_smithi137_74180_object74936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:33:17.477 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:16.534804+0000", 2024-09-18T07:33:17.478 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.053954505999999999, 2024-09-18T07:33:17.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.054106952999999999, 2024-09-18T07:33:17.478 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:33:17.478 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:33:17.478 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:17.478 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:33:17.478 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:33:17.478 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:17.478 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 74937 2024-09-18T07:33:17.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.479 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:33:17.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:33:17.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.534804+0000", 2024-09-18T07:33:17.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:33:17.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:33:17.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.534804+0000", 2024-09-18T07:33:17.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:33:17.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:33:17.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.534806+0000", 2024-09-18T07:33:17.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9580000000000001e-06 2024-09-18T07:33:17.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:33:17.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.534830+0000", 2024-09-18T07:33:17.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3501000000000001e-05 2024-09-18T07:33:17.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:33:17.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.534841+0000", 2024-09-18T07:33:17.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1022999999999999e-05 2024-09-18T07:33:17.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:33:17.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.534848+0000", 2024-09-18T07:33:17.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1189999999999999e-06 2024-09-18T07:33:17.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:33:17.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.534898+0000", 2024-09-18T07:33:17.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0176999999999997e-05 2024-09-18T07:33:17.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:33:17.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.535602+0000", 2024-09-18T07:33:17.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00070408000000000005 2024-09-18T07:33:17.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-18T07:33:17.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.535733+0000", 2024-09-18T07:33:17.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000131023 2024-09-18T07:33:17.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:17.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.538369+0000", 2024-09-18T07:33:17.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026355089999999999 2024-09-18T07:33:17.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:33:17.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.538845+0000", 2024-09-18T07:33:17.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00047623300000000001 2024-09-18T07:33:17.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:17.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542837+0000", 2024-09-18T07:33:17.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039924879999999998 2024-09-18T07:33:17.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:33:17.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:33:17.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:33:17.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.542881+0000", 2024-09-18T07:33:17.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3906000000000003e-05 2024-09-18T07:33:17.483 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:33:17.483 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:33:17.484 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:33:17.484 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:33:17.484 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:33:17.484 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 5 2024-09-18T07:33:17.484 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:33:17.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:17.559 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:17.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:17.661 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:17.686 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:17.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:17.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:17.783 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 62 16 76201 76185 76.78 64.375 0.00624557 0.0130187 2024-09-18T07:33:17.788 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:17.826 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:17.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:17.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:17.928 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:17.965 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:17.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:17.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:18.066 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:18.087 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:18.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:18.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:18.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:17 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:18.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:17 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:18.189 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:18.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:17 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:18.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:18.230 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:18.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:18.331 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:18.357 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:18.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:18.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:18.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:18.479 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:18.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:18.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:18.581 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:18.608 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:18.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:18.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:18.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:18.731 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:18.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:18.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:18.784 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 63 15 77146 77131 76.4997 59.125 0.0167008 0.0130667 2024-09-18T07:33:18.833 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:18.855 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:18.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:18.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:18.956 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:18.983 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:18.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:18.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:19.084 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:19.124 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:19.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:19.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:19.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:18 smithi137 ceph-mon[29408]: pgmap v102: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 4.8 MiB/s, 76 objects/s recovering 2024-09-18T07:33:19.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:18 smithi137 ceph-mon[39175]: pgmap v102: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 4.8 MiB/s, 76 objects/s recovering 2024-09-18T07:33:19.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:18 smithi203 ceph-mon[32237]: pgmap v102: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 4.8 MiB/s, 76 objects/s recovering 2024-09-18T07:33:19.226 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:19.256 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:19.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:19.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:19.358 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:19.384 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:19.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:19.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:19.486 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:19.512 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:19.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:19.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:19.614 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:19.638 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:19.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:19.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:19.766 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:19.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:19.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:19.784 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 64 16 78155 78139 76.2887 63 0.00597962 0.0131022 2024-09-18T07:33:19.868 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:19.895 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:19.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:19.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:19.997 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:20.021 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:20.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:20.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:20.124 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:20.149 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:20.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:20.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:20.251 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:20.301 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:20.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:20.402 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:20.426 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:20.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:20.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:20.528 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:20.551 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:20.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:20.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:20.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:20.703 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:20.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:20.785 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 65 15 79173 79158 76.0948 63.6875 0.0109408 0.0131363 2024-09-18T07:33:20.805 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:20.830 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:20.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:20.923 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:33:20.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:20.955 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:20.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:20.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:20.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T07:33:21.056 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:21.081 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:21.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:21.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:21.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:20 smithi137 ceph-mon[29408]: pgmap v103: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 4.6 MiB/s, 73 objects/s recovering 2024-09-18T07:33:21.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:20 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:21.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:20 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:21.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:20 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1033560499' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:33:21.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:20 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3824409948' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:33:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:20 smithi137 ceph-mon[39175]: pgmap v103: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 4.6 MiB/s, 73 objects/s recovering 2024-09-18T07:33:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:20 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:20 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:20 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1033560499' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:33:21.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:20 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3824409948' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:33:21.183 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:21.207 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:21.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:21.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:21.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:20 smithi203 ceph-mon[32237]: pgmap v103: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 4.6 MiB/s, 73 objects/s recovering 2024-09-18T07:33:21.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:20 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:21.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:20 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:21.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:20 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1033560499' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:33:21.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:20 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3824409948' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:33:21.277 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.3 to 1 (10000) 2024-09-18T07:33:21.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:21.363 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:21.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:21.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:21.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:21.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:21.507 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:21.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:21.608 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:21.656 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:21.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:21.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:21.758 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:21.782 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:21.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:21.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:21.784 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 66 16 80248 80232 75.9588 67.125 0.0170159 0.0131596 2024-09-18T07:33:21.884 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:21.909 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:21 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:33:21.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:21.910 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:21.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:22.011 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:22.038 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:22.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:22.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:22.140 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:22.162 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:21 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1033560499' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:33:22.163 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:21 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3824409948' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T07:33:22.163 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:21 smithi137 ceph-mon[29408]: osdmap e117: 8 total, 8 up, 7 in 2024-09-18T07:33:22.163 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:21 smithi137 ceph-mon[29408]: 6.c scrub starts 2024-09-18T07:33:22.163 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:21 smithi137 ceph-mon[29408]: 6.c scrub ok 2024-09-18T07:33:22.163 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:21 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:33:22.163 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:21 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1033560499' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:33:22.163 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:21 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3824409948' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T07:33:22.163 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:21 smithi137 ceph-mon[39175]: osdmap e117: 8 total, 8 up, 7 in 2024-09-18T07:33:22.163 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:21 smithi137 ceph-mon[39175]: 6.c scrub starts 2024-09-18T07:33:22.164 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:21 smithi137 ceph-mon[39175]: 6.c scrub ok 2024-09-18T07:33:22.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:22.197 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:22.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:22.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:21 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:33:22.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:21 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1033560499' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:33:22.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:21 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3824409948' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T07:33:22.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:21 smithi203 ceph-mon[32237]: osdmap e117: 8 total, 8 up, 7 in 2024-09-18T07:33:22.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:21 smithi203 ceph-mon[32237]: 6.c scrub starts 2024-09-18T07:33:22.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:21 smithi203 ceph-mon[32237]: 6.c scrub ok 2024-09-18T07:33:22.298 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:22.341 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:22.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:22.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:22.443 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:22.469 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:22.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:22.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:22.571 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:22.597 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:22.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:22.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:22.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:22.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:22.746 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:22.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:22.784 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 67 16 81430 81414 75.9276 73.875 0.0135397 0.0131657 2024-09-18T07:33:22.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:22.886 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:22.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:22.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:22.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:23.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:23.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:23.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:23.171 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:23.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:23.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:23.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:22 smithi203 ceph-mon[32237]: 3.1d scrub starts 2024-09-18T07:33:23.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:22 smithi203 ceph-mon[32237]: 3.1d scrub ok 2024-09-18T07:33:23.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:22 smithi203 ceph-mon[32237]: pgmap v105: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 677 B/s rd, 50 MiB/s wr, 793 op/s; 1.4 MiB/s, 21 objects/s recovering 2024-09-18T07:33:23.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:22 smithi203 ceph-mon[32237]: 5.2 scrub starts 2024-09-18T07:33:23.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:22 smithi203 ceph-mon[32237]: 5.2 scrub ok 2024-09-18T07:33:23.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:22 smithi203 ceph-mon[32237]: 6.f scrub starts 2024-09-18T07:33:23.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:22 smithi203 ceph-mon[32237]: 6.f scrub ok 2024-09-18T07:33:23.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:22 smithi203 ceph-mon[32237]: 4.19 scrub starts 2024-09-18T07:33:23.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:22 smithi203 ceph-mon[32237]: 4.19 scrub ok 2024-09-18T07:33:23.273 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:23.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:23.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:23.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:23.415 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:23.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[29408]: 3.1d scrub starts 2024-09-18T07:33:23.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[29408]: 3.1d scrub ok 2024-09-18T07:33:23.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[29408]: pgmap v105: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 677 B/s rd, 50 MiB/s wr, 793 op/s; 1.4 MiB/s, 21 objects/s recovering 2024-09-18T07:33:23.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[29408]: 5.2 scrub starts 2024-09-18T07:33:23.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[29408]: 5.2 scrub ok 2024-09-18T07:33:23.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[29408]: 6.f scrub starts 2024-09-18T07:33:23.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[29408]: 6.f scrub ok 2024-09-18T07:33:23.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[29408]: 4.19 scrub starts 2024-09-18T07:33:23.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[29408]: 4.19 scrub ok 2024-09-18T07:33:23.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[39175]: 3.1d scrub starts 2024-09-18T07:33:23.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[39175]: 3.1d scrub ok 2024-09-18T07:33:23.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[39175]: pgmap v105: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 677 B/s rd, 50 MiB/s wr, 793 op/s; 1.4 MiB/s, 21 objects/s recovering 2024-09-18T07:33:23.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[39175]: 5.2 scrub starts 2024-09-18T07:33:23.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[39175]: 5.2 scrub ok 2024-09-18T07:33:23.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[39175]: 6.f scrub starts 2024-09-18T07:33:23.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[39175]: 6.f scrub ok 2024-09-18T07:33:23.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[39175]: 4.19 scrub starts 2024-09-18T07:33:23.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:22 smithi137 ceph-mon[39175]: 4.19 scrub ok 2024-09-18T07:33:23.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:23.441 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:23.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:23.459 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:23.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-18T07:33:23.525 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:23.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:23.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:23.633 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:23.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:23.734 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:23.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:23.778 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:23.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:23.784 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 68 16 82488 82472 75.7832 66.125 0.0248812 0.0131894 2024-09-18T07:33:23.879 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:23.925 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:23.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:24.028 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:24.051 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:24.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:24.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:24.070 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[29408]: 5.1e scrub starts 2024-09-18T07:33:24.070 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[29408]: 5.1e scrub ok 2024-09-18T07:33:24.071 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[29408]: 6.7 scrub starts 2024-09-18T07:33:24.071 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[29408]: 6.7 scrub ok 2024-09-18T07:33:24.071 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:33:24.071 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:33:24.071 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[29408]: osdmap e118: 8 total, 8 up, 7 in 2024-09-18T07:33:24.071 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[29408]: 3.0 scrub starts 2024-09-18T07:33:24.071 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[29408]: 3.0 scrub ok 2024-09-18T07:33:24.071 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[39175]: 5.1e scrub starts 2024-09-18T07:33:24.071 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[39175]: 5.1e scrub ok 2024-09-18T07:33:24.071 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[39175]: 6.7 scrub starts 2024-09-18T07:33:24.071 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[39175]: 6.7 scrub ok 2024-09-18T07:33:24.072 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:33:24.072 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:33:24.072 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[39175]: osdmap e118: 8 total, 8 up, 7 in 2024-09-18T07:33:24.072 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[39175]: 3.0 scrub starts 2024-09-18T07:33:24.072 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:23 smithi137 ceph-mon[39175]: 3.0 scrub ok 2024-09-18T07:33:24.153 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:24.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:33:24.176 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:24.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:24.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:24.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:23 smithi203 ceph-mon[32237]: 5.1e scrub starts 2024-09-18T07:33:24.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:23 smithi203 ceph-mon[32237]: 5.1e scrub ok 2024-09-18T07:33:24.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:23 smithi203 ceph-mon[32237]: 6.7 scrub starts 2024-09-18T07:33:24.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:23 smithi203 ceph-mon[32237]: 6.7 scrub ok 2024-09-18T07:33:24.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:23 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:33:24.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:23 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:33:24.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:23 smithi203 ceph-mon[32237]: osdmap e118: 8 total, 8 up, 7 in 2024-09-18T07:33:24.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:23 smithi203 ceph-mon[32237]: 3.0 scrub starts 2024-09-18T07:33:24.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:23 smithi203 ceph-mon[32237]: 3.0 scrub ok 2024-09-18T07:33:24.279 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:24.301 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:24.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:24.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:24.373 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:33:24.373 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:33:24.373 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:33:24.373 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:33:24.373 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:33:24.373 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:33:24.373 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:33:24.374 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:33:24.374 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:33:24.374 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:33:24.374 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:33:24.374 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:33:24.375 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:33:24.375 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:33:24.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:24.404 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:33:24.405 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:33:24.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:24.444 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:24.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:24.545 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:24.592 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:24.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:24.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:24.682 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:33:24.682 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:33:24.693 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:24.730 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:24.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:24.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:24.784 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 69 16 83407 83391 75.5173 57.4375 0.0041855 0.0132316 2024-09-18T07:33:24.832 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:24.857 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:24.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:24.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:24.959 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:24.992 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:24.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:24.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:25.094 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:25.119 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:25.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:25.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:25.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:25.261 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:24 smithi137 ceph-mon[29408]: 1.0 scrub starts 2024-09-18T07:33:25.261 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:24 smithi137 ceph-mon[29408]: 1.0 scrub ok 2024-09-18T07:33:25.261 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:24 smithi137 ceph-mon[29408]: pgmap v107: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 1.3 MiB/s, 20 objects/s recovering 2024-09-18T07:33:25.261 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:24 smithi137 ceph-mon[29408]: 6.1d scrub starts 2024-09-18T07:33:25.261 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:24 smithi137 ceph-mon[29408]: 6.1d scrub ok 2024-09-18T07:33:25.261 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:24 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2308149953' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:25.262 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:24 smithi137 ceph-mon[39175]: 1.0 scrub starts 2024-09-18T07:33:25.262 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:24 smithi137 ceph-mon[39175]: 1.0 scrub ok 2024-09-18T07:33:25.262 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:24 smithi137 ceph-mon[39175]: pgmap v107: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 1.3 MiB/s, 20 objects/s recovering 2024-09-18T07:33:25.262 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:24 smithi137 ceph-mon[39175]: 6.1d scrub starts 2024-09-18T07:33:25.262 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:24 smithi137 ceph-mon[39175]: 6.1d scrub ok 2024-09-18T07:33:25.262 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:24 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2308149953' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:25.274 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:25.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:25.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:25.379 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:25.428 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:25.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:25.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:25.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:24 smithi203 ceph-mon[32237]: 1.0 scrub starts 2024-09-18T07:33:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:24 smithi203 ceph-mon[32237]: 1.0 scrub ok 2024-09-18T07:33:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:24 smithi203 ceph-mon[32237]: pgmap v107: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 1.3 MiB/s, 20 objects/s recovering 2024-09-18T07:33:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:24 smithi203 ceph-mon[32237]: 6.1d scrub starts 2024-09-18T07:33:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:24 smithi203 ceph-mon[32237]: 6.1d scrub ok 2024-09-18T07:33:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:24 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2308149953' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:25.530 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:25.555 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:25.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:25.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:25.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:25.695 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:25.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:25.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:25.784 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 70 16 84520 84504 75.4322 69.5625 0.0113431 0.0132513 2024-09-18T07:33:25.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:25.836 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:25.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:25.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:25.937 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:25.963 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:25.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:25.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:26.065 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:26.090 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:26.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:26.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:26.192 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:26.215 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:26.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:26.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:26.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:26.366 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:26.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:26.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[29408]: 3.6 scrub starts 2024-09-18T07:33:26.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[29408]: 3.6 scrub ok 2024-09-18T07:33:26.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[29408]: 6.4 scrub starts 2024-09-18T07:33:26.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[29408]: 6.4 scrub ok 2024-09-18T07:33:26.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-09-18T07:33:26.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]} : dispatch 2024-09-18T07:33:26.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.b", "id": [0, 3]} : dispatch 2024-09-18T07:33:26.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [0, 6]} : dispatch 2024-09-18T07:33:26.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]} : dispatch 2024-09-18T07:33:26.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-09-18T07:33:26.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]} : dispatch 2024-09-18T07:33:26.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.b", "id": [0, 3]} : dispatch 2024-09-18T07:33:26.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [0, 6]} : dispatch 2024-09-18T07:33:26.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]} : dispatch 2024-09-18T07:33:26.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:33:26.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[39175]: 3.6 scrub starts 2024-09-18T07:33:26.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[39175]: 3.6 scrub ok 2024-09-18T07:33:26.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[39175]: 6.4 scrub starts 2024-09-18T07:33:26.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[39175]: 6.4 scrub ok 2024-09-18T07:33:26.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-09-18T07:33:26.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]} : dispatch 2024-09-18T07:33:26.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.b", "id": [0, 3]} : dispatch 2024-09-18T07:33:26.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [0, 6]} : dispatch 2024-09-18T07:33:26.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]} : dispatch 2024-09-18T07:33:26.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-09-18T07:33:26.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]} : dispatch 2024-09-18T07:33:26.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.b", "id": [0, 3]} : dispatch 2024-09-18T07:33:26.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [0, 6]} : dispatch 2024-09-18T07:33:26.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]} : dispatch 2024-09-18T07:33:26.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:25 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:33:26.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:25 smithi203 ceph-mon[32237]: 3.6 scrub starts 2024-09-18T07:33:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:25 smithi203 ceph-mon[32237]: 3.6 scrub ok 2024-09-18T07:33:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:25 smithi203 ceph-mon[32237]: 6.4 scrub starts 2024-09-18T07:33:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:25 smithi203 ceph-mon[32237]: 6.4 scrub ok 2024-09-18T07:33:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:25 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-09-18T07:33:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:25 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]} : dispatch 2024-09-18T07:33:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:25 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.b", "id": [0, 3]} : dispatch 2024-09-18T07:33:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:25 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [0, 6]} : dispatch 2024-09-18T07:33:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:25 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]} : dispatch 2024-09-18T07:33:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:25 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-09-18T07:33:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:25 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]} : dispatch 2024-09-18T07:33:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:25 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.b", "id": [0, 3]} : dispatch 2024-09-18T07:33:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:25 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [0, 6]} : dispatch 2024-09-18T07:33:26.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:25 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]} : dispatch 2024-09-18T07:33:26.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:25 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:33:26.468 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:26.495 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:26.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:26.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:26.636 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:26.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:26.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:26.693 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T07:33:26.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:26.785 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 71 15 85613 85598 75.3327 68.375 0.00646358 0.0132682 2024-09-18T07:33:26.799 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:26.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:26.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:26.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:26.926 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:26.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:26.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:27.028 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:27.052 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:27.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:27.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:27.154 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:27.178 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:27.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:27.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:27.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:33:26 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:33:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:26 smithi203 ceph-mon[32237]: pgmap v108: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 39 MiB/s wr, 630 op/s 2024-09-18T07:33:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-18T07:33:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]': finished 2024-09-18T07:33:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.b", "id": [0, 3]}]': finished 2024-09-18T07:33:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [0, 6]}]': finished 2024-09-18T07:33:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]}]': finished 2024-09-18T07:33:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:26 smithi203 ceph-mon[32237]: osdmap e119: 8 total, 8 up, 7 in 2024-09-18T07:33:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:26 smithi203 ceph-mon[32237]: 6.1f scrub starts 2024-09-18T07:33:27.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:26 smithi203 ceph-mon[32237]: 6.1f scrub ok 2024-09-18T07:33:27.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:27.340 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:27.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[29408]: pgmap v108: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 39 MiB/s wr, 630 op/s 2024-09-18T07:33:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-18T07:33:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]': finished 2024-09-18T07:33:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.b", "id": [0, 3]}]': finished 2024-09-18T07:33:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [0, 6]}]': finished 2024-09-18T07:33:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]}]': finished 2024-09-18T07:33:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[29408]: osdmap e119: 8 total, 8 up, 7 in 2024-09-18T07:33:27.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[29408]: 6.1f scrub starts 2024-09-18T07:33:27.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[29408]: 6.1f scrub ok 2024-09-18T07:33:27.421 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:33:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:33:27.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[39175]: pgmap v108: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 39 MiB/s wr, 630 op/s 2024-09-18T07:33:27.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-18T07:33:27.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]': finished 2024-09-18T07:33:27.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.b", "id": [0, 3]}]': finished 2024-09-18T07:33:27.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [0, 6]}]': finished 2024-09-18T07:33:27.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]}]': finished 2024-09-18T07:33:27.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[39175]: osdmap e119: 8 total, 8 up, 7 in 2024-09-18T07:33:27.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[39175]: 6.1f scrub starts 2024-09-18T07:33:27.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:26 smithi137 ceph-mon[39175]: 6.1f scrub ok 2024-09-18T07:33:27.440 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:27.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:27.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:27.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:27.567 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:27.584 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:27.588 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:27.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:27.688 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:27.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:27.785 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 72 16 86551 86535 75.0998 58.5625 0.0266825 0.0133109 2024-09-18T07:33:27.790 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:27.814 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:27.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:27.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:27.916 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:27.939 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:27.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:27.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:28.041 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:28.066 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:28.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:28.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:28.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:28.205 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:28.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:28.306 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:28.333 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:28.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:28.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:28.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:28 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:28.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:28 smithi137 ceph-mon[29408]: osdmap e120: 8 total, 8 up, 7 in 2024-09-18T07:33:28.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:28 smithi137 ceph-mon[29408]: 3.1e scrub starts 2024-09-18T07:33:28.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:28 smithi137 ceph-mon[29408]: 3.1e scrub ok 2024-09-18T07:33:28.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:28 smithi137 ceph-mon[29408]: 5.7 scrub starts 2024-09-18T07:33:28.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:28 smithi137 ceph-mon[29408]: 5.7 scrub ok 2024-09-18T07:33:28.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:28 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:33:28.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:28 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:33:28.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:28 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:28.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:28 smithi137 ceph-mon[39175]: osdmap e120: 8 total, 8 up, 7 in 2024-09-18T07:33:28.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:28 smithi137 ceph-mon[39175]: 3.1e scrub starts 2024-09-18T07:33:28.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:28 smithi137 ceph-mon[39175]: 3.1e scrub ok 2024-09-18T07:33:28.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:28 smithi137 ceph-mon[39175]: 5.7 scrub starts 2024-09-18T07:33:28.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:28 smithi137 ceph-mon[39175]: 5.7 scrub ok 2024-09-18T07:33:28.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:28 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:33:28.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:28 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:33:28.435 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:28.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:28.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:28.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:28.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:28 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:28.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:28 smithi203 ceph-mon[32237]: osdmap e120: 8 total, 8 up, 7 in 2024-09-18T07:33:28.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:28 smithi203 ceph-mon[32237]: 3.1e scrub starts 2024-09-18T07:33:28.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:28 smithi203 ceph-mon[32237]: 3.1e scrub ok 2024-09-18T07:33:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:28 smithi203 ceph-mon[32237]: 5.7 scrub starts 2024-09-18T07:33:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:28 smithi203 ceph-mon[32237]: 5.7 scrub ok 2024-09-18T07:33:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:28 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:33:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:28 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:33:28.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:28.602 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:28.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:28.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:28.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:28.750 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:33:28.750 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:33:28.756 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:28.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:28.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:28.784 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 73 16 87446 87430 74.8372 55.9375 0.0323669 0.013356 2024-09-18T07:33:28.858 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:28.882 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:28.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:28.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:28.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:29.006 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:29.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:29.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:29.084 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:29 smithi137 ceph-mon[29408]: pgmap v111: 148 pgs: 5 peering, 143 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.7 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-18T07:33:29.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:29 smithi137 ceph-mon[29408]: 4.1f scrub starts 2024-09-18T07:33:29.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:29 smithi137 ceph-mon[29408]: 4.1f scrub ok 2024-09-18T07:33:29.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:29 smithi137 ceph-mon[29408]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-18T07:33:29.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:29 smithi137 ceph-mon[29408]: osdmap e121: 8 total, 8 up, 7 in 2024-09-18T07:33:29.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:29 smithi137 ceph-mon[29408]: 3.c scrub starts 2024-09-18T07:33:29.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:29 smithi137 ceph-mon[29408]: 3.c scrub ok 2024-09-18T07:33:29.085 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:29 smithi137 ceph-mon[39175]: pgmap v111: 148 pgs: 5 peering, 143 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.7 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-18T07:33:29.085 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:29 smithi137 ceph-mon[39175]: 4.1f scrub starts 2024-09-18T07:33:29.085 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:29 smithi137 ceph-mon[39175]: 4.1f scrub ok 2024-09-18T07:33:29.085 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:29 smithi137 ceph-mon[39175]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-18T07:33:29.086 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:29 smithi137 ceph-mon[39175]: osdmap e121: 8 total, 8 up, 7 in 2024-09-18T07:33:29.086 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:29 smithi137 ceph-mon[39175]: 3.c scrub starts 2024-09-18T07:33:29.086 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:29 smithi137 ceph-mon[39175]: 3.c scrub ok 2024-09-18T07:33:29.108 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:29.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:29.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:29.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:29.259 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:29.283 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:29.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:29.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:29.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:29.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:29.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:29.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:29.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:29 smithi203 ceph-mon[32237]: pgmap v111: 148 pgs: 5 peering, 143 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.7 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-18T07:33:29.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:29 smithi203 ceph-mon[32237]: 4.1f scrub starts 2024-09-18T07:33:29.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:29 smithi203 ceph-mon[32237]: 4.1f scrub ok 2024-09-18T07:33:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:29 smithi203 ceph-mon[32237]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-18T07:33:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:29 smithi203 ceph-mon[32237]: osdmap e121: 8 total, 8 up, 7 in 2024-09-18T07:33:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:29 smithi203 ceph-mon[32237]: 3.c scrub starts 2024-09-18T07:33:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:29 smithi203 ceph-mon[32237]: 3.c scrub ok 2024-09-18T07:33:29.522 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:29.546 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:29.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:29.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:29.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:33:29.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:29.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:29.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:29.785 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 74 16 88277 88261 74.5278 51.9375 0.0161202 0.0134125 2024-09-18T07:33:29.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:29.823 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:29.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:29.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:29.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:29.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:29.982 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:29.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:30.023 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.4 to 1 (10000) 2024-09-18T07:33:30.023 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:33:30.043 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:30 smithi137 ceph-mon[29408]: 3.5 scrub starts 2024-09-18T07:33:30.043 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:30 smithi137 ceph-mon[29408]: 3.5 scrub ok 2024-09-18T07:33:30.043 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:30 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3333309503' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:33:30.043 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:30 smithi137 ceph-mon[29408]: from='client.34835 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:33:30.043 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:30 smithi137 ceph-mon[29408]: 4.18 scrub starts 2024-09-18T07:33:30.044 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:30 smithi137 ceph-mon[29408]: 4.18 scrub ok 2024-09-18T07:33:30.044 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:30 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/858364807' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:30.044 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:30 smithi137 ceph-mon[29408]: from='client.24862 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:30.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:30 smithi137 ceph-mon[39175]: 3.5 scrub starts 2024-09-18T07:33:30.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:30 smithi137 ceph-mon[39175]: 3.5 scrub ok 2024-09-18T07:33:30.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:30 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3333309503' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:33:30.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:30 smithi137 ceph-mon[39175]: from='client.34835 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:33:30.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:30 smithi137 ceph-mon[39175]: 4.18 scrub starts 2024-09-18T07:33:30.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:30 smithi137 ceph-mon[39175]: 4.18 scrub ok 2024-09-18T07:33:30.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:30 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/858364807' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:30.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:30 smithi137 ceph-mon[39175]: from='client.24862 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:30.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:30.122 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:30.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:30.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:30.152 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T07:33:30.224 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:30.248 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:30.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:30.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:30.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:30.387 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:30.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:30.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:30 smithi203 ceph-mon[32237]: 3.5 scrub starts 2024-09-18T07:33:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:30 smithi203 ceph-mon[32237]: 3.5 scrub ok 2024-09-18T07:33:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:30 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3333309503' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:33:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:30 smithi203 ceph-mon[32237]: from='client.34835 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:33:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:30 smithi203 ceph-mon[32237]: 4.18 scrub starts 2024-09-18T07:33:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:30 smithi203 ceph-mon[32237]: 4.18 scrub ok 2024-09-18T07:33:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:30 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/858364807' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:30 smithi203 ceph-mon[32237]: from='client.24862 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:30.482 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:33:30.490 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:30.522 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:30.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:30.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:30.677 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:30.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:30.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:30.778 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:30.785 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 75 16 89252 89236 74.3465 60.9375 0.0202201 0.0134451 2024-09-18T07:33:30.808 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:30.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:30.909 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:30.945 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:30.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:30.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:31.047 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:31.088 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:33:31.088 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:33:31.093 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:31.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:31.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:31.196 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:31.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:31 smithi203 ceph-mon[32237]: pgmap v113: 148 pgs: 5 peering, 143 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 56 MiB/s wr, 898 op/s 2024-09-18T07:33:31.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:31 smithi203 ceph-mon[32237]: 7.7 scrub starts 2024-09-18T07:33:31.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:31 smithi203 ceph-mon[32237]: 4.10 scrub starts 2024-09-18T07:33:31.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:31 smithi203 ceph-mon[32237]: 4.10 scrub ok 2024-09-18T07:33:31.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:31 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:33:31.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:31 smithi203 ceph-mon[32237]: from='client.34835 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:33:31.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:31 smithi203 ceph-mon[32237]: from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:33:31.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:31 smithi203 ceph-mon[32237]: osdmap e122: 8 total, 8 up, 7 in 2024-09-18T07:33:31.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:31 smithi203 ceph-mon[32237]: 5.10 scrub starts 2024-09-18T07:33:31.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:31 smithi203 ceph-mon[32237]: 5.10 scrub ok 2024-09-18T07:33:31.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:31.236 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:31.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:31.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:31.374 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:31.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:31.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:31.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[29408]: pgmap v113: 148 pgs: 5 peering, 143 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 56 MiB/s wr, 898 op/s 2024-09-18T07:33:31.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[29408]: 7.7 scrub starts 2024-09-18T07:33:31.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[29408]: 4.10 scrub starts 2024-09-18T07:33:31.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[29408]: 4.10 scrub ok 2024-09-18T07:33:31.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:33:31.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[29408]: from='client.34835 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:33:31.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[29408]: from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:33:31.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[29408]: osdmap e122: 8 total, 8 up, 7 in 2024-09-18T07:33:31.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[29408]: 5.10 scrub starts 2024-09-18T07:33:31.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[29408]: 5.10 scrub ok 2024-09-18T07:33:31.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[39175]: pgmap v113: 148 pgs: 5 peering, 143 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 56 MiB/s wr, 898 op/s 2024-09-18T07:33:31.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[39175]: 7.7 scrub starts 2024-09-18T07:33:31.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[39175]: 4.10 scrub starts 2024-09-18T07:33:31.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[39175]: 4.10 scrub ok 2024-09-18T07:33:31.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:33:31.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[39175]: from='client.34835 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:33:31.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[39175]: from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:33:31.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[39175]: osdmap e122: 8 total, 8 up, 7 in 2024-09-18T07:33:31.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[39175]: 5.10 scrub starts 2024-09-18T07:33:31.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:31 smithi137 ceph-mon[39175]: 5.10 scrub ok 2024-09-18T07:33:31.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:31.514 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:31.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:31.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:31.616 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:31.642 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:31.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:31.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:31.744 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:31.774 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:31.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:31.779 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T07:33:31.786 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 76 16 90322 90306 74.2467 66.875 0.0166943 0.0134626 2024-09-18T07:33:31.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-18T07:33:31.877 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:31.902 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:31.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:31.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:31.905 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:32.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:32.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:32.066 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:32.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:32.130 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:33:32.168 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:32.195 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:32.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:32.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[29408]: 7.c scrub starts 2024-09-18T07:33:32.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[29408]: 5.1d scrub starts 2024-09-18T07:33:32.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[29408]: 5.1d scrub ok 2024-09-18T07:33:32.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[29408]: 6.13 scrub starts 2024-09-18T07:33:32.291 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[29408]: 6.13 scrub ok 2024-09-18T07:33:32.291 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[29408]: 3.1f scrub starts 2024-09-18T07:33:32.291 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[29408]: 7.7 scrub ok 2024-09-18T07:33:32.291 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[29408]: 3.1f scrub ok 2024-09-18T07:33:32.291 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[29408]: pgmap v115: 148 pgs: 1 active+clean+scrubbing, 5 peering, 142 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 935 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-18T07:33:32.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[39175]: 7.c scrub starts 2024-09-18T07:33:32.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[39175]: 5.1d scrub starts 2024-09-18T07:33:32.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[39175]: 5.1d scrub ok 2024-09-18T07:33:32.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[39175]: 6.13 scrub starts 2024-09-18T07:33:32.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[39175]: 6.13 scrub ok 2024-09-18T07:33:32.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[39175]: 3.1f scrub starts 2024-09-18T07:33:32.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[39175]: 7.7 scrub ok 2024-09-18T07:33:32.292 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[39175]: 3.1f scrub ok 2024-09-18T07:33:32.292 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:32 smithi137 ceph-mon[39175]: pgmap v115: 148 pgs: 1 active+clean+scrubbing, 5 peering, 142 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 935 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-18T07:33:32.293 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:32 smithi203 ceph-mon[32237]: 7.c scrub starts 2024-09-18T07:33:32.293 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:32 smithi203 ceph-mon[32237]: 5.1d scrub starts 2024-09-18T07:33:32.293 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:32 smithi203 ceph-mon[32237]: 5.1d scrub ok 2024-09-18T07:33:32.293 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:32 smithi203 ceph-mon[32237]: 6.13 scrub starts 2024-09-18T07:33:32.293 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:32 smithi203 ceph-mon[32237]: 6.13 scrub ok 2024-09-18T07:33:32.294 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:32 smithi203 ceph-mon[32237]: 3.1f scrub starts 2024-09-18T07:33:32.294 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:32 smithi203 ceph-mon[32237]: 7.7 scrub ok 2024-09-18T07:33:32.294 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:32 smithi203 ceph-mon[32237]: 3.1f scrub ok 2024-09-18T07:33:32.294 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:32 smithi203 ceph-mon[32237]: pgmap v115: 148 pgs: 1 active+clean+scrubbing, 5 peering, 142 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 935 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-18T07:33:32.297 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:32.321 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:32.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:32.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:32.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:32.470 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:32.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:32.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:32.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:32.629 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:32.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:32.632 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:33:32.632 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:33:32.664 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:33:32.664 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:33:32.684 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:33:32.684 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:33:32.706 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:33:32.706 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:33:32.730 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:32.738 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:33:32.738 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:33:32.769 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:32.770 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:33:32.770 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:33:32.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:32.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:32.786 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 77 16 91224 91208 74.0145 56.375 0.017494 0.0135043 2024-09-18T07:33:32.801 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:33:32.802 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:33:32.821 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:33:32.821 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:33:32.838 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:33:32.838 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:33:32.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:32.917 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:32.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:32.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:33.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:33.062 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:33.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:33.135 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:33:33.163 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:33.187 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:33.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:33.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:33.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:33.313 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:33.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:33.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:33.326 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:33 smithi137 ceph-mon[29408]: 7.c scrub ok 2024-09-18T07:33:33.326 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:33 smithi137 ceph-mon[29408]: 6.16 scrub starts 2024-09-18T07:33:33.326 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:33 smithi137 ceph-mon[29408]: 6.16 scrub ok 2024-09-18T07:33:33.327 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:33 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2590523932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:33.327 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:33 smithi137 ceph-mon[29408]: 5.16 scrub starts 2024-09-18T07:33:33.327 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:33 smithi137 ceph-mon[29408]: 5.16 scrub ok 2024-09-18T07:33:33.327 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:33 smithi137 ceph-mon[39175]: 7.c scrub ok 2024-09-18T07:33:33.327 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:33 smithi137 ceph-mon[39175]: 6.16 scrub starts 2024-09-18T07:33:33.327 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:33 smithi137 ceph-mon[39175]: 6.16 scrub ok 2024-09-18T07:33:33.327 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:33 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2590523932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:33.328 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:33 smithi137 ceph-mon[39175]: 5.16 scrub starts 2024-09-18T07:33:33.328 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:33 smithi137 ceph-mon[39175]: 5.16 scrub ok 2024-09-18T07:33:33.337 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:33 smithi203 ceph-mon[32237]: 7.c scrub ok 2024-09-18T07:33:33.337 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:33 smithi203 ceph-mon[32237]: 6.16 scrub starts 2024-09-18T07:33:33.337 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:33 smithi203 ceph-mon[32237]: 6.16 scrub ok 2024-09-18T07:33:33.337 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:33 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2590523932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:33.337 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:33 smithi203 ceph-mon[32237]: 5.16 scrub starts 2024-09-18T07:33:33.337 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:33 smithi203 ceph-mon[32237]: 5.16 scrub ok 2024-09-18T07:33:33.372 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T07:33:33.415 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:33.481 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:33.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:33.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:33.583 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:33.610 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:33.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:33.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:33.711 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:33.732 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:33:33.783 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:33.786 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:33.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:33.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:33.788 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 78 16 92214 92198 73.8575 61.875 0.0215613 0.0135318 2024-09-18T07:33:33.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:33.934 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:33.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:34.036 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:34.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:34.076 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:34.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:34.166 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:34 smithi203 ceph-mon[32237]: 4.a scrub starts 2024-09-18T07:33:34.166 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:34 smithi203 ceph-mon[32237]: 4.a scrub ok 2024-09-18T07:33:34.166 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:34 smithi203 ceph-mon[32237]: 3.19 scrub starts 2024-09-18T07:33:34.166 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:34 smithi203 ceph-mon[32237]: 3.19 scrub ok 2024-09-18T07:33:34.166 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:34 smithi203 ceph-mon[32237]: 5.d scrub starts 2024-09-18T07:33:34.167 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:34 smithi203 ceph-mon[32237]: 5.d scrub ok 2024-09-18T07:33:34.167 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:34 smithi203 ceph-mon[32237]: 3.18 scrub starts 2024-09-18T07:33:34.167 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:34 smithi203 ceph-mon[32237]: 3.18 scrub ok 2024-09-18T07:33:34.167 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:34 smithi203 ceph-mon[32237]: pgmap v116: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 793 B/s rd, 46 MiB/s wr, 730 op/s; 17 B/s, 1 objects/s recovering 2024-09-18T07:33:34.167 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:34 smithi203 ceph-mon[32237]: 6.1c scrub starts 2024-09-18T07:33:34.167 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:34 smithi203 ceph-mon[32237]: 6.1c scrub ok 2024-09-18T07:33:34.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:34.215 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:34.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:34.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:34.301 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:33:34.301 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:33:34.301 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:33:34.301 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:33:34.301 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.301 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:20749 7.3 e88/81 7:c04d08df:::benchmark_data_smithi137_74180_object20748:head v 88'1239, mlcod=88'1237)", 2024-09-18T07:33:34.301 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.530035+0000", 2024-09-18T07:33:34.301 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 70.765055875000002, 2024-09-18T07:33:34.301 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13453020299999999, 2024-09-18T07:33:34.301 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.302 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.302 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.302 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.302 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.302 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.302 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530035+0000", 2024-09-18T07:33:34.302 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.302 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.302 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.302 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.302 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530035+0000", 2024-09-18T07:33:34.302 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.302 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.303 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.303 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.303 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530036+0000", 2024-09-18T07:33:34.303 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3200000000000004e-07 2024-09-18T07:33:34.303 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.303 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.303 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.303 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530106+0000", 2024-09-18T07:33:34.303 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0099999999999996e-05 2024-09-18T07:33:34.303 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.303 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.303 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.303 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530115+0000", 2024-09-18T07:33:34.304 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1419999999999999e-06 2024-09-18T07:33:34.304 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.304 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.304 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.304 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530122+0000", 2024-09-18T07:33:34.304 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3269999999999997e-06 2024-09-18T07:33:34.304 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.304 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.304 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.304 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530167+0000", 2024-09-18T07:33:34.304 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5247999999999998e-05 2024-09-18T07:33:34.304 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.304 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.305 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.305 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530185+0000", 2024-09-18T07:33:34.305 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8111e-05 2024-09-18T07:33:34.305 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.305 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.305 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.305 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530789+0000", 2024-09-18T07:33:34.305 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00060407400000000002 2024-09-18T07:33:34.305 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.305 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.305 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.305 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664566+0000", 2024-09-18T07:33:34.306 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13377636900000001 2024-09-18T07:33:34.306 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.306 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.306 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.306 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.306 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.306 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:20756 7.4 e88/81 7:206b3dce:::benchmark_data_smithi137_74180_object20755:head v 88'1303, mlcod=88'1301)", 2024-09-18T07:33:34.306 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.532983+0000", 2024-09-18T07:33:34.306 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 70.762108084999994, 2024-09-18T07:33:34.306 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13536040199999999, 2024-09-18T07:33:34.306 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.306 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.306 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.307 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.307 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.307 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.307 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.532983+0000", 2024-09-18T07:33:34.307 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.307 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.307 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.307 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.307 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.532983+0000", 2024-09-18T07:33:34.307 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.307 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.307 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.307 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.308 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.532984+0000", 2024-09-18T07:33:34.308 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7999999999999993e-07 2024-09-18T07:33:34.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.308 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.308 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.308 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533015+0000", 2024-09-18T07:33:34.308 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1195999999999997e-05 2024-09-18T07:33:34.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.308 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.308 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.308 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665392+0000", 2024-09-18T07:33:34.308 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13237665800000001 2024-09-18T07:33:34.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.309 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.309 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.309 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665400+0000", 2024-09-18T07:33:34.309 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7079999999999997e-06 2024-09-18T07:33:34.309 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.309 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.309 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.309 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665454+0000", 2024-09-18T07:33:34.309 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4116999999999997e-05 2024-09-18T07:33:34.309 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.309 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.309 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.310 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665474+0000", 2024-09-18T07:33:34.310 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0092999999999999e-05 2024-09-18T07:33:34.310 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.310 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.310 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.310 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.668302+0000", 2024-09-18T07:33:34.310 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0028280520000000002 2024-09-18T07:33:34.310 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.310 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.310 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.310 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.668343+0000", 2024-09-18T07:33:34.310 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1597999999999997e-05 2024-09-18T07:33:34.310 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.311 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.311 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.311 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.311 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.311 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:20750 7.3 e88/81 7:c7abdaa5:::benchmark_data_smithi137_74180_object20749:head v 88'1240, mlcod=88'1237)", 2024-09-18T07:33:34.311 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.533452+0000", 2024-09-18T07:33:34.311 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 70.761638899999994, 2024-09-18T07:33:34.311 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13358589900000001, 2024-09-18T07:33:34.311 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.311 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.311 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.311 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.311 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.312 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.312 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533452+0000", 2024-09-18T07:33:34.312 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.312 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.312 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.312 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.312 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533452+0000", 2024-09-18T07:33:34.312 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.312 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.312 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.312 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.312 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533453+0000", 2024-09-18T07:33:34.312 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.057e-06 2024-09-18T07:33:34.313 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.313 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.313 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.313 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533515+0000", 2024-09-18T07:33:34.313 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1557999999999996e-05 2024-09-18T07:33:34.313 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.313 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.313 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.313 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664524+0000", 2024-09-18T07:33:34.313 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13100931900000001 2024-09-18T07:33:34.313 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.313 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.313 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.314 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664540+0000", 2024-09-18T07:33:34.314 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5562000000000001e-05 2024-09-18T07:33:34.314 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.314 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.314 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.314 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664600+0000", 2024-09-18T07:33:34.314 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0124000000000002e-05 2024-09-18T07:33:34.314 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.314 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.314 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.314 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664620+0000", 2024-09-18T07:33:34.314 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9983000000000002e-05 2024-09-18T07:33:34.315 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.315 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.315 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.315 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.666988+0000", 2024-09-18T07:33:34.315 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023680559999999999 2024-09-18T07:33:34.315 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.315 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.315 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.315 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.667038+0000", 2024-09-18T07:33:34.315 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.024e-05 2024-09-18T07:33:34.315 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.315 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.315 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.315 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.316 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.316 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21270 7.1 7:801f54ca:::benchmark_data_smithi137_74180_object21269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:33:34.316 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.882531+0000", 2024-09-18T07:33:34.316 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 70.412560477, 2024-09-18T07:33:34.316 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16466234699999999, 2024-09-18T07:33:34.316 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.316 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.316 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.316 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:34.316 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:34.316 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:34.316 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21270 2024-09-18T07:33:34.317 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.317 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.317 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.317 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.317 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882531+0000", 2024-09-18T07:33:34.317 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.317 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.317 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.317 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.317 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882531+0000", 2024-09-18T07:33:34.317 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.317 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.317 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.318 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.318 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882533+0000", 2024-09-18T07:33:34.318 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8190000000000001e-06 2024-09-18T07:33:34.318 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.318 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.318 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.318 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882584+0000", 2024-09-18T07:33:34.318 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1919999999999998e-05 2024-09-18T07:33:34.318 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.318 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.318 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.318 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882593+0000", 2024-09-18T07:33:34.318 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8829999999999995e-06 2024-09-18T07:33:34.318 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.319 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.319 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.319 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882599+0000", 2024-09-18T07:33:34.319 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5849999999999999e-06 2024-09-18T07:33:34.319 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.319 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.319 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.319 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882634+0000", 2024-09-18T07:33:34.319 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5132999999999997e-05 2024-09-18T07:33:34.319 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.319 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.319 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.319 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882717+0000", 2024-09-18T07:33:34.320 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2819999999999996e-05 2024-09-18T07:33:34.320 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.320 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.320 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:33:34.320 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882824+0000", 2024-09-18T07:33:34.320 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010708400000000001 2024-09-18T07:33:34.320 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.320 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.320 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:34.320 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.883645+0000", 2024-09-18T07:33:34.320 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00082133399999999995 2024-09-18T07:33:34.320 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.320 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.321 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.321 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.940766+0000", 2024-09-18T07:33:34.321 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.057121166000000001 2024-09-18T07:33:34.321 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.321 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.321 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.321 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.046997+0000", 2024-09-18T07:33:34.321 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10623094900000001 2024-09-18T07:33:34.321 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.321 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.321 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.322 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047057+0000", 2024-09-18T07:33:34.322 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9580000000000002e-05 2024-09-18T07:33:34.322 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.322 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.322 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.322 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047193+0000", 2024-09-18T07:33:34.322 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000136074 2024-09-18T07:33:34.322 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.322 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.322 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.322 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.322 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.322 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21285 7.1 7:8dbe9f10:::benchmark_data_smithi137_74180_object21284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:33:34.322 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.944262+0000", 2024-09-18T07:33:34.323 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 70.350829433000001, 2024-09-18T07:33:34.323 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10313246500000001, 2024-09-18T07:33:34.323 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.323 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.323 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.323 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:34.323 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:34.323 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:34.323 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21285 2024-09-18T07:33:34.323 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.323 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.323 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.323 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.323 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944262+0000", 2024-09-18T07:33:34.324 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.324 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.324 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.324 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.324 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944262+0000", 2024-09-18T07:33:34.324 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.324 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.324 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.324 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.324 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944264+0000", 2024-09-18T07:33:34.324 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8679999999999999e-06 2024-09-18T07:33:34.324 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.324 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.325 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.325 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944324+0000", 2024-09-18T07:33:34.325 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0225e-05 2024-09-18T07:33:34.325 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.325 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.325 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.325 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944333+0000", 2024-09-18T07:33:34.325 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.064e-06 2024-09-18T07:33:34.325 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.325 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.325 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.325 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944338+0000", 2024-09-18T07:33:34.325 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9749999999999998e-06 2024-09-18T07:33:34.326 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.326 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.326 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.326 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944380+0000", 2024-09-18T07:33:34.326 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1773000000000001e-05 2024-09-18T07:33:34.326 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.326 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.326 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.326 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944476+0000", 2024-09-18T07:33:34.326 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6497999999999997e-05 2024-09-18T07:33:34.326 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.326 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.326 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:33:34.327 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944565+0000", 2024-09-18T07:33:34.327 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9288000000000003e-05 2024-09-18T07:33:34.327 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.327 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.327 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:34.327 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.945376+0000", 2024-09-18T07:33:34.327 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00081097899999999998 2024-09-18T07:33:34.327 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.327 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.327 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.327 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.946088+0000", 2024-09-18T07:33:34.327 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00071115600000000001 2024-09-18T07:33:34.327 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.328 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.328 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.328 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047252+0000", 2024-09-18T07:33:34.328 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.101164449 2024-09-18T07:33:34.328 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.328 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.328 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.328 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047340+0000", 2024-09-18T07:33:34.328 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8268000000000003e-05 2024-09-18T07:33:34.328 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.328 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.328 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.328 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047394+0000", 2024-09-18T07:33:34.329 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3922000000000003e-05 2024-09-18T07:33:34.329 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.329 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.329 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.329 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.329 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.329 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22331 7.1 7:8575e846:::benchmark_data_smithi137_74180_object22330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:33:34.329 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.655146+0000", 2024-09-18T07:33:34.329 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 69.639945311999995, 2024-09-18T07:33:34.329 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.29868304200000001, 2024-09-18T07:33:34.329 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.329 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.330 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.330 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:34.330 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:34.330 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:34.330 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22331 2024-09-18T07:33:34.330 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.330 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.330 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.330 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.330 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:34.332 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655146+0000", 2024-09-18T07:33:34.332 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.332 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.332 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.332 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.332 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655146+0000", 2024-09-18T07:33:34.332 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.332 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655148+0000", 2024-09-18T07:33:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.779e-06 2024-09-18T07:33:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655192+0000", 2024-09-18T07:33:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4023000000000002e-05 2024-09-18T07:33:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.333 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655201+0000", 2024-09-18T07:33:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3559999999999999e-06 2024-09-18T07:33:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655206+0000", 2024-09-18T07:33:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4539999999999998e-06 2024-09-18T07:33:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655250+0000", 2024-09-18T07:33:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3630000000000001e-05 2024-09-18T07:33:34.334 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655323+0000", 2024-09-18T07:33:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3382999999999999e-05 2024-09-18T07:33:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:33:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655411+0000", 2024-09-18T07:33:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7776999999999997e-05 2024-09-18T07:33:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:34.335 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.656141+0000", 2024-09-18T07:33:34.336 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00072986100000000005 2024-09-18T07:33:34.336 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.336 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.336 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.336 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.658652+0000", 2024-09-18T07:33:34.336 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025112569999999998 2024-09-18T07:33:34.336 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.336 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.336 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.336 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953705+0000", 2024-09-18T07:33:34.336 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.29505283900000001 2024-09-18T07:33:34.336 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.337 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.337 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.337 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953782+0000", 2024-09-18T07:33:34.337 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6793999999999997e-05 2024-09-18T07:33:34.337 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.338 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.338 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.338 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953829+0000", 2024-09-18T07:33:34.338 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6888999999999998e-05 2024-09-18T07:33:34.338 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.338 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.338 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.338 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.338 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.338 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22350 7.1 7:851589b5:::benchmark_data_smithi137_74180_object22349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:33:34.339 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.662419+0000", 2024-09-18T07:33:34.339 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 69.632672030999998, 2024-09-18T07:33:34.339 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.29155694799999998, 2024-09-18T07:33:34.339 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.339 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.340 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.340 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:34.340 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:34.340 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:34.340 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22350 2024-09-18T07:33:34.341 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.341 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.341 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.341 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.341 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662419+0000", 2024-09-18T07:33:34.342 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.342 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.342 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.342 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.342 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662419+0000", 2024-09-18T07:33:34.343 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.343 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.343 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.343 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.343 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662421+0000", 2024-09-18T07:33:34.344 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.39e-06 2024-09-18T07:33:34.344 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.344 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.344 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.344 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662473+0000", 2024-09-18T07:33:34.344 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2094e-05 2024-09-18T07:33:34.345 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.345 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.345 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.345 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662480+0000", 2024-09-18T07:33:34.345 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5449999999999998e-06 2024-09-18T07:33:34.346 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.346 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.346 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.346 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662485+0000", 2024-09-18T07:33:34.346 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9200000000000003e-06 2024-09-18T07:33:34.346 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.346 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.347 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.347 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662521+0000", 2024-09-18T07:33:34.347 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5559000000000003e-05 2024-09-18T07:33:34.347 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.347 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.347 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.347 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662589+0000", 2024-09-18T07:33:34.347 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8480999999999997e-05 2024-09-18T07:33:34.348 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.348 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.348 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:33:34.348 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662666+0000", 2024-09-18T07:33:34.348 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7224999999999993e-05 2024-09-18T07:33:34.348 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.349 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.349 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:34.349 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.663390+0000", 2024-09-18T07:33:34.349 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00072354199999999998 2024-09-18T07:33:34.349 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.349 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.349 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.349 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.663928+0000", 2024-09-18T07:33:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00053822699999999995 2024-09-18T07:33:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953879+0000", 2024-09-18T07:33:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.28995123699999997 2024-09-18T07:33:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953934+0000", 2024-09-18T07:33:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5047000000000003e-05 2024-09-18T07:33:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.350 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953976+0000", 2024-09-18T07:33:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1681000000000003e-05 2024-09-18T07:33:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22352 7.1 7:83b03301:::benchmark_data_smithi137_74180_object22351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:33:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.664495+0000", 2024-09-18T07:33:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 69.630595955000004, 2024-09-18T07:33:34.351 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.28962424599999997, 2024-09-18T07:33:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22352 2024-09-18T07:33:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664495+0000", 2024-09-18T07:33:34.352 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.353 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.353 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.353 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.353 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664495+0000", 2024-09-18T07:33:34.353 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.353 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.353 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.354 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.354 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664496+0000", 2024-09-18T07:33:34.354 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6000000000000003e-07 2024-09-18T07:33:34.354 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.354 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.354 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.354 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664568+0000", 2024-09-18T07:33:34.354 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2021999999999996e-05 2024-09-18T07:33:34.355 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.355 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.355 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.355 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664574+0000", 2024-09-18T07:33:34.355 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4620000000000001e-06 2024-09-18T07:33:34.355 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.355 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.355 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.356 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664579+0000", 2024-09-18T07:33:34.356 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0620000000000002e-06 2024-09-18T07:33:34.356 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.356 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.356 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.356 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664605+0000", 2024-09-18T07:33:34.356 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6126e-05 2024-09-18T07:33:34.356 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.357 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.357 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.357 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664676+0000", 2024-09-18T07:33:34.357 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1521999999999997e-05 2024-09-18T07:33:34.357 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.357 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.357 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:33:34.357 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664749+0000", 2024-09-18T07:33:34.358 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3232999999999995e-05 2024-09-18T07:33:34.358 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.358 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.358 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:34.358 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.665309+0000", 2024-09-18T07:33:34.358 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00055992900000000003 2024-09-18T07:33:34.358 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.359 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.359 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.359 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.665857+0000", 2024-09-18T07:33:34.359 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00054812600000000002 2024-09-18T07:33:34.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.359 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.359 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.360 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.954036+0000", 2024-09-18T07:33:34.360 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.288178926 2024-09-18T07:33:34.360 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.360 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.360 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.360 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.954085+0000", 2024-09-18T07:33:34.361 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8534e-05 2024-09-18T07:33:34.361 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.361 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.361 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.361 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.954119+0000", 2024-09-18T07:33:34.361 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4544e-05 2024-09-18T07:33:34.361 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.362 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.362 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.362 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.362 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.362 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:22994 7.9 e90/81 7:9ec32d99:::benchmark_data_smithi137_74180_object22993:head v 90'1458, mlcod=90'1455)", 2024-09-18T07:33:34.362 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.331557+0000", 2024-09-18T07:33:34.362 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 68.963534182999993, 2024-09-18T07:33:34.363 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.098100452000000005, 2024-09-18T07:33:34.363 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.363 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.363 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.363 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.363 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.363 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.363 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331557+0000", 2024-09-18T07:33:34.363 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331557+0000", 2024-09-18T07:33:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331558+0000", 2024-09-18T07:33:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4099999999999997e-07 2024-09-18T07:33:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.364 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331619+0000", 2024-09-18T07:33:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1216000000000005e-05 2024-09-18T07:33:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331626+0000", 2024-09-18T07:33:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3119999999999996e-06 2024-09-18T07:33:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331631+0000", 2024-09-18T07:33:34.365 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3370000000000001e-06 2024-09-18T07:33:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331670+0000", 2024-09-18T07:33:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9103999999999997e-05 2024-09-18T07:33:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331682+0000", 2024-09-18T07:33:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1926000000000001e-05 2024-09-18T07:33:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.366 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.429623+0000", 2024-09-18T07:33:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.097941316 2024-09-18T07:33:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.429657+0000", 2024-09-18T07:33:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4400000000000003e-05 2024-09-18T07:33:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.367 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:22989 7.3 e90/81 7:c023e9ac:::benchmark_data_smithi137_74180_object22988:head v 90'1383, mlcod=90'1381)", 2024-09-18T07:33:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.331705+0000", 2024-09-18T07:33:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 68.963385955999996, 2024-09-18T07:33:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.096971521000000005, 2024-09-18T07:33:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331705+0000", 2024-09-18T07:33:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.368 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.369 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.369 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.369 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331705+0000", 2024-09-18T07:33:34.369 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.369 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.369 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.369 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331706+0000", 2024-09-18T07:33:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7599999999999997e-07 2024-09-18T07:33:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331745+0000", 2024-09-18T07:33:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9201e-05 2024-09-18T07:33:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331749+0000", 2024-09-18T07:33:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5789999999999997e-06 2024-09-18T07:33:34.370 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331753+0000", 2024-09-18T07:33:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4139999999999998e-06 2024-09-18T07:33:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331801+0000", 2024-09-18T07:33:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7824000000000002e-05 2024-09-18T07:33:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.371 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331818+0000", 2024-09-18T07:33:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7402999999999999e-05 2024-09-18T07:33:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.428631+0000", 2024-09-18T07:33:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.096812980000000007 2024-09-18T07:33:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.428677+0000", 2024-09-18T07:33:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5643999999999999e-05 2024-09-18T07:33:34.372 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:23001 7.1 7:8979af80:::benchmark_data_smithi137_74180_object23000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:33:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.332415+0000", 2024-09-18T07:33:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 68.962676587999994, 2024-09-18T07:33:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.100079928, 2024-09-18T07:33:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.373 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 23001 2024-09-18T07:33:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332415+0000", 2024-09-18T07:33:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.374 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.375 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332415+0000", 2024-09-18T07:33:34.375 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.375 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.375 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.375 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.375 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332416+0000", 2024-09-18T07:33:34.375 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8369999999999999e-06 2024-09-18T07:33:34.375 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332477+0000", 2024-09-18T07:33:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0333e-05 2024-09-18T07:33:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332487+0000", 2024-09-18T07:33:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0203e-05 2024-09-18T07:33:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.376 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332494+0000", 2024-09-18T07:33:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.515e-06 2024-09-18T07:33:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332560+0000", 2024-09-18T07:33:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5874e-05 2024-09-18T07:33:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.428551+0000", 2024-09-18T07:33:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.095990961 2024-09-18T07:33:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.377 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:33:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.428728+0000", 2024-09-18T07:33:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017673299999999999 2024-09-18T07:33:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.430995+0000", 2024-09-18T07:33:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022672819999999998 2024-09-18T07:33:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.432117+0000", 2024-09-18T07:33:34.378 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0011216939999999999 2024-09-18T07:33:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.432392+0000", 2024-09-18T07:33:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00027448799999999998 2024-09-18T07:33:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.432451+0000", 2024-09-18T07:33:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9233e-05 2024-09-18T07:33:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.379 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.432495+0000", 2024-09-18T07:33:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3775e-05 2024-09-18T07:33:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:44367 7.1 7:854d3320:::benchmark_data_smithi137_74180_object44366:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-18T07:33:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:43.591378+0000", 2024-09-18T07:33:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 50.703713391000001, 2024-09-18T07:33:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.113341394, 2024-09-18T07:33:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.380 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.381 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.381 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:34.381 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:34.381 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:34.381 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 44367 2024-09-18T07:33:34.381 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.381 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.381 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.381 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.381 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591378+0000", 2024-09-18T07:33:34.381 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.381 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.381 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591378+0000", 2024-09-18T07:33:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591379+0000", 2024-09-18T07:33:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7099999999999999e-06 2024-09-18T07:33:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591420+0000", 2024-09-18T07:33:34.382 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0559000000000002e-05 2024-09-18T07:33:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591432+0000", 2024-09-18T07:33:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.226e-05 2024-09-18T07:33:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591440+0000", 2024-09-18T07:33:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5830000000000001e-06 2024-09-18T07:33:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.383 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.384 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591497+0000", 2024-09-18T07:33:34.384 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7114999999999998e-05 2024-09-18T07:33:34.384 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.384 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.384 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.384 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591664+0000", 2024-09-18T07:33:34.384 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000167188 2024-09-18T07:33:34.384 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.384 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.384 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:33:34.384 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591884+0000", 2024-09-18T07:33:34.384 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000220223 2024-09-18T07:33:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.593492+0000", 2024-09-18T07:33:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016074139999999999 2024-09-18T07:33:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.593595+0000", 2024-09-18T07:33:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010319699999999999 2024-09-18T07:33:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.385 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.704655+0000", 2024-09-18T07:33:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.111060176 2024-09-18T07:33:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.704690+0000", 2024-09-18T07:33:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4916999999999998e-05 2024-09-18T07:33:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.704719+0000", 2024-09-18T07:33:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9051999999999999e-05 2024-09-18T07:33:34.386 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:51788 7.1 7:81b6d503:::benchmark_data_smithi137_74180_object51787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:33:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:33:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 43.628147216000002, 2024-09-18T07:33:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14889203200000001, 2024-09-18T07:33:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:34.387 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 51788 2024-09-18T07:33:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:33:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:33:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.388 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.389 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.389 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666966+0000", 2024-09-18T07:33:34.389 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1860000000000001e-05 2024-09-18T07:33:34.389 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.389 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.389 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.389 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666998+0000", 2024-09-18T07:33:34.389 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1856000000000001e-05 2024-09-18T07:33:34.389 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.389 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.389 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.389 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667012+0000", 2024-09-18T07:33:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4592e-05 2024-09-18T07:33:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667019+0000", 2024-09-18T07:33:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1940000000000001e-06 2024-09-18T07:33:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667083+0000", 2024-09-18T07:33:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3553999999999995e-05 2024-09-18T07:33:34.390 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667217+0000", 2024-09-18T07:33:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000133712 2024-09-18T07:33:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:33:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667345+0000", 2024-09-18T07:33:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012796199999999999 2024-09-18T07:33:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.668905+0000", 2024-09-18T07:33:34.391 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001560344 2024-09-18T07:33:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669080+0000", 2024-09-18T07:33:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000174964 2024-09-18T07:33:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815763+0000", 2024-09-18T07:33:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.146683217 2024-09-18T07:33:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.392 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815804+0000", 2024-09-18T07:33:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0535999999999999e-05 2024-09-18T07:33:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815836+0000", 2024-09-18T07:33:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2240999999999998e-05 2024-09-18T07:33:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.393 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:51794 7.1 7:8f789502:::benchmark_data_smithi137_74180_object51793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:33:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:33:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 43.625974106000001, 2024-09-18T07:33:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15046784899999999, 2024-09-18T07:33:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 51794 2024-09-18T07:33:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.394 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:33:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:33:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.395 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669118+0000", 2024-09-18T07:33:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4079999999999999e-06 2024-09-18T07:33:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669172+0000", 2024-09-18T07:33:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3808999999999999e-05 2024-09-18T07:33:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669184+0000", 2024-09-18T07:33:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1888e-05 2024-09-18T07:33:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.396 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669192+0000", 2024-09-18T07:33:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7549999999999994e-06 2024-09-18T07:33:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669229+0000", 2024-09-18T07:33:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7234000000000003e-05 2024-09-18T07:33:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.397 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669310+0000", 2024-09-18T07:33:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1248999999999994e-05 2024-09-18T07:33:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:33:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669391+0000", 2024-09-18T07:33:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0401000000000005e-05 2024-09-18T07:33:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.670397+0000", 2024-09-18T07:33:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010061619999999999 2024-09-18T07:33:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.398 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.670664+0000", 2024-09-18T07:33:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00026735700000000002 2024-09-18T07:33:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819479+0000", 2024-09-18T07:33:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14881438499999999 2024-09-18T07:33:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819537+0000", 2024-09-18T07:33:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8192999999999997e-05 2024-09-18T07:33:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.399 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819585+0000", 2024-09-18T07:33:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8007999999999998e-05 2024-09-18T07:33:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:74680 7.1 7:8999b183:::benchmark_data_smithi137_74180_object74679:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:33:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:33:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 18.11712606, 2024-09-18T07:33:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13647764200000001, 2024-09-18T07:33:34.400 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 74680 2024-09-18T07:33:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:33:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.401 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:33:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177967+0000", 2024-09-18T07:33:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.559e-06 2024-09-18T07:33:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.402 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178020+0000", 2024-09-18T07:33:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3810999999999997e-05 2024-09-18T07:33:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178030+0000", 2024-09-18T07:33:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5200000000000003e-06 2024-09-18T07:33:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.403 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178035+0000", 2024-09-18T07:33:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0429999999999998e-06 2024-09-18T07:33:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.404 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178074+0000", 2024-09-18T07:33:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9014999999999998e-05 2024-09-18T07:33:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178151+0000", 2024-09-18T07:33:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6526000000000006e-05 2024-09-18T07:33:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:33:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178233+0000", 2024-09-18T07:33:34.405 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2174000000000003e-05 2024-09-18T07:33:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.180222+0000", 2024-09-18T07:33:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0019888459999999998 2024-09-18T07:33:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.180453+0000", 2024-09-18T07:33:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023154 2024-09-18T07:33:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.406 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314370+0000", 2024-09-18T07:33:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.133916497 2024-09-18T07:33:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314411+0000", 2024-09-18T07:33:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1421000000000002e-05 2024-09-18T07:33:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314443+0000", 2024-09-18T07:33:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1690000000000003e-05 2024-09-18T07:33:34.407 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:76535 7.1 7:86123a56:::benchmark_data_smithi137_74180_object76534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:33:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.065654+0000", 2024-09-18T07:33:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 16.229437020999999, 2024-09-18T07:33:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.097319778999999995, 2024-09-18T07:33:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.408 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 76535 2024-09-18T07:33:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.409 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065654+0000", 2024-09-18T07:33:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065654+0000", 2024-09-18T07:33:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.410 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065656+0000", 2024-09-18T07:33:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8220000000000001e-06 2024-09-18T07:33:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.411 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065751+0000", 2024-09-18T07:33:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5267999999999997e-05 2024-09-18T07:33:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065761+0000", 2024-09-18T07:33:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2799999999999992e-06 2024-09-18T07:33:34.412 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065766+0000", 2024-09-18T07:33:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6699999999999999e-06 2024-09-18T07:33:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.413 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065808+0000", 2024-09-18T07:33:34.414 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2150999999999997e-05 2024-09-18T07:33:34.414 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.414 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.414 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.414 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065904+0000", 2024-09-18T07:33:34.414 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5330999999999999e-05 2024-09-18T07:33:34.414 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.415 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.415 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:33:34.415 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.066005+0000", 2024-09-18T07:33:34.415 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010100800000000001 2024-09-18T07:33:34.415 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.415 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.415 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:34.416 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.067931+0000", 2024-09-18T07:33:34.416 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0019264239999999999 2024-09-18T07:33:34.416 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.416 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.416 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.416 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.070368+0000", 2024-09-18T07:33:34.417 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0024367659999999999 2024-09-18T07:33:34.417 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.417 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.417 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.417 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.162870+0000", 2024-09-18T07:33:34.417 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.092502134999999999 2024-09-18T07:33:34.417 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.418 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.418 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.418 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.162928+0000", 2024-09-18T07:33:34.418 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8029999999999998e-05 2024-09-18T07:33:34.418 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.418 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.419 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.419 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.162974+0000", 2024-09-18T07:33:34.419 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5893999999999998e-05 2024-09-18T07:33:34.419 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.419 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.419 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.419 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.419 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.419 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:76538 7.1 7:826fb340:::benchmark_data_smithi137_74180_object76537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:33:34.420 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.068561+0000", 2024-09-18T07:33:34.420 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 16.226529699, 2024-09-18T07:33:34.420 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.097070977000000003, 2024-09-18T07:33:34.420 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.420 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.420 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.420 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:34.420 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:34.420 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:34.420 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 76538 2024-09-18T07:33:34.420 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.421 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.421 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.421 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.421 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068561+0000", 2024-09-18T07:33:34.421 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.421 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.421 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.421 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.422 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068561+0000", 2024-09-18T07:33:34.422 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.422 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.422 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.422 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.422 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068563+0000", 2024-09-18T07:33:34.422 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5629999999999999e-06 2024-09-18T07:33:34.422 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.422 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.422 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.422 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068611+0000", 2024-09-18T07:33:34.422 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8412999999999999e-05 2024-09-18T07:33:34.422 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.423 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.423 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.423 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068620+0000", 2024-09-18T07:33:34.423 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7530000000000007e-06 2024-09-18T07:33:34.423 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.423 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.423 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.423 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068626+0000", 2024-09-18T07:33:34.423 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.818e-06 2024-09-18T07:33:34.423 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.423 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.423 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.423 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068663+0000", 2024-09-18T07:33:34.424 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7086999999999999e-05 2024-09-18T07:33:34.424 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.424 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.424 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.424 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068734+0000", 2024-09-18T07:33:34.424 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0605000000000006e-05 2024-09-18T07:33:34.424 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.424 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.424 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:33:34.424 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068829+0000", 2024-09-18T07:33:34.424 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5533999999999997e-05 2024-09-18T07:33:34.425 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.425 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.425 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:34.425 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.070471+0000", 2024-09-18T07:33:34.425 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016418800000000001 2024-09-18T07:33:34.425 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.425 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.425 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.425 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.071217+0000", 2024-09-18T07:33:34.425 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00074625099999999997 2024-09-18T07:33:34.425 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.425 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.426 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.426 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.165516+0000", 2024-09-18T07:33:34.426 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.094298471999999994 2024-09-18T07:33:34.426 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.426 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.426 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.426 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.165590+0000", 2024-09-18T07:33:34.426 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4103999999999994e-05 2024-09-18T07:33:34.426 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.426 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.426 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.426 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.165632+0000", 2024-09-18T07:33:34.426 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2497000000000003e-05 2024-09-18T07:33:34.427 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.427 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.427 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.427 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.427 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.427 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:76881 7.1 7:8430cf73:::benchmark_data_smithi137_74180_object76880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:33:34.427 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.426831+0000", 2024-09-18T07:33:34.427 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 15.868260301999999, 2024-09-18T07:33:34.427 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.115187357, 2024-09-18T07:33:34.427 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.427 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.427 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.427 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:34.427 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:34.428 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:34.428 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 76881 2024-09-18T07:33:34.428 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.428 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.428 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.428 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.428 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426831+0000", 2024-09-18T07:33:34.428 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.428 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.428 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.428 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.428 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426831+0000", 2024-09-18T07:33:34.429 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.429 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.429 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.429 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.429 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426833+0000", 2024-09-18T07:33:34.429 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7710000000000001e-06 2024-09-18T07:33:34.429 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.429 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.429 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.429 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426869+0000", 2024-09-18T07:33:34.429 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5886999999999997e-05 2024-09-18T07:33:34.429 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.429 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.430 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.430 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426879+0000", 2024-09-18T07:33:34.430 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0667e-05 2024-09-18T07:33:34.430 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.430 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.430 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.430 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426886+0000", 2024-09-18T07:33:34.430 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3799999999999999e-06 2024-09-18T07:33:34.430 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.430 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.430 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.430 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426932+0000", 2024-09-18T07:33:34.430 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6143999999999997e-05 2024-09-18T07:33:34.431 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.431 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.431 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.431 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.427047+0000", 2024-09-18T07:33:34.431 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011568700000000001 2024-09-18T07:33:34.431 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.431 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.431 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:33:34.431 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.427168+0000", 2024-09-18T07:33:34.431 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000120118 2024-09-18T07:33:34.431 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.432 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.432 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:34.432 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.429449+0000", 2024-09-18T07:33:34.432 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002281394 2024-09-18T07:33:34.432 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.432 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.432 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.432 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.431262+0000", 2024-09-18T07:33:34.433 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0018125940000000001 2024-09-18T07:33:34.433 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.433 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.433 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:34.433 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541358+0000", 2024-09-18T07:33:34.433 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.110095996 2024-09-18T07:33:34.433 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.434 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.434 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.434 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541963+0000", 2024-09-18T07:33:34.434 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00060516200000000002 2024-09-18T07:33:34.434 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.434 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.434 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.435 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.542018+0000", 2024-09-18T07:33:34.435 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5556999999999997e-05 2024-09-18T07:33:34.435 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.435 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.435 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.435 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.435 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.436 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:76891 7.f e116/115 7:f77e09af:::benchmark_data_smithi137_74180_object76890:head v 116'4811, mlcod=116'4809)", 2024-09-18T07:33:34.436 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.436137+0000", 2024-09-18T07:33:34.436 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 15.858954177999999, 2024-09-18T07:33:34.436 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.105530608, 2024-09-18T07:33:34.436 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.436 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.436 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.437 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.437 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.437 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.437 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436137+0000", 2024-09-18T07:33:34.437 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.437 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.438 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.438 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.438 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436137+0000", 2024-09-18T07:33:34.438 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.438 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.438 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.438 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.438 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436139+0000", 2024-09-18T07:33:34.439 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3259999999999998e-06 2024-09-18T07:33:34.439 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.439 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.439 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.439 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436192+0000", 2024-09-18T07:33:34.439 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2339000000000002e-05 2024-09-18T07:33:34.439 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.440 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.440 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.440 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436209+0000", 2024-09-18T07:33:34.440 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6935999999999998e-05 2024-09-18T07:33:34.440 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.440 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.440 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.440 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436219+0000", 2024-09-18T07:33:34.440 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0593e-05 2024-09-18T07:33:34.440 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.440 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.440 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.441 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436314+0000", 2024-09-18T07:33:34.441 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5035000000000001e-05 2024-09-18T07:33:34.441 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.441 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.441 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.441 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436347+0000", 2024-09-18T07:33:34.441 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2558999999999998e-05 2024-09-18T07:33:34.441 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.441 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.442 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.442 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.437824+0000", 2024-09-18T07:33:34.442 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001477405 2024-09-18T07:33:34.442 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.442 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.442 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.442 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541668+0000", 2024-09-18T07:33:34.442 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10384341499999999 2024-09-18T07:33:34.442 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.442 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.442 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.442 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.442 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.443 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:76893 7.4 e116/81 7:2dcd0014:::benchmark_data_smithi137_74180_object76892:head v 116'4832, mlcod=116'4830)", 2024-09-18T07:33:34.443 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.438822+0000", 2024-09-18T07:33:34.443 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 15.856269201, 2024-09-18T07:33:34.443 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.106356858, 2024-09-18T07:33:34.443 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:34.443 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:34.443 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:34.443 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:34.443 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.443 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:34.443 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.438822+0000", 2024-09-18T07:33:34.443 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.443 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.444 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.444 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:34.444 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.438822+0000", 2024-09-18T07:33:34.444 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:34.444 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.444 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.444 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:34.444 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.438823+0000", 2024-09-18T07:33:34.444 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2979999999999999e-06 2024-09-18T07:33:34.444 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.444 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.444 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:34.444 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.438847+0000", 2024-09-18T07:33:34.444 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3370000000000002e-05 2024-09-18T07:33:34.445 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.445 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.445 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:34.445 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541862+0000", 2024-09-18T07:33:34.445 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10301571299999999 2024-09-18T07:33:34.445 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.445 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.445 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:34.445 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541872+0000", 2024-09-18T07:33:34.445 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0017000000000001e-05 2024-09-18T07:33:34.445 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.445 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.445 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:34.446 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541931+0000", 2024-09-18T07:33:34.446 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8990000000000003e-05 2024-09-18T07:33:34.446 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.446 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.446 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:34.446 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541958+0000", 2024-09-18T07:33:34.446 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6996e-05 2024-09-18T07:33:34.446 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.446 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.446 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:34.446 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.545137+0000", 2024-09-18T07:33:34.446 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003178859 2024-09-18T07:33:34.446 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:34.447 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:34.447 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:34.447 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.545179+0000", 2024-09-18T07:33:34.447 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1615000000000001e-05 2024-09-18T07:33:34.447 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.447 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.447 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.447 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:34.447 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:34.447 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:33:34.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[29408]: 4.a scrub starts 2024-09-18T07:33:34.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[29408]: 4.a scrub ok 2024-09-18T07:33:34.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[29408]: 3.19 scrub starts 2024-09-18T07:33:34.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[29408]: 3.19 scrub ok 2024-09-18T07:33:34.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[29408]: 5.d scrub starts 2024-09-18T07:33:34.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[29408]: 5.d scrub ok 2024-09-18T07:33:34.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[29408]: 3.18 scrub starts 2024-09-18T07:33:34.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[29408]: 3.18 scrub ok 2024-09-18T07:33:34.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[29408]: pgmap v116: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 793 B/s rd, 46 MiB/s wr, 730 op/s; 17 B/s, 1 objects/s recovering 2024-09-18T07:33:34.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[29408]: 6.1c scrub starts 2024-09-18T07:33:34.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[29408]: 6.1c scrub ok 2024-09-18T07:33:34.449 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[39175]: 4.a scrub starts 2024-09-18T07:33:34.450 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[39175]: 4.a scrub ok 2024-09-18T07:33:34.450 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[39175]: 3.19 scrub starts 2024-09-18T07:33:34.450 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[39175]: 3.19 scrub ok 2024-09-18T07:33:34.450 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[39175]: 5.d scrub starts 2024-09-18T07:33:34.450 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[39175]: 5.d scrub ok 2024-09-18T07:33:34.450 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[39175]: 3.18 scrub starts 2024-09-18T07:33:34.450 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[39175]: 3.18 scrub ok 2024-09-18T07:33:34.450 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[39175]: pgmap v116: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 793 B/s rd, 46 MiB/s wr, 730 op/s; 17 B/s, 1 objects/s recovering 2024-09-18T07:33:34.450 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[39175]: 6.1c scrub starts 2024-09-18T07:33:34.450 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:34 smithi137 ceph-mon[39175]: 6.1c scrub ok 2024-09-18T07:33:34.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:34.451 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:34.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:34.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:34.589 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:34.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:34.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:34.691 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:34.716 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:34.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:34.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:34.742 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:34.788 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 79 16 93295 93279 73.7777 67.5625 0.0194776 0.0135489 2024-09-18T07:33:34.818 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:34.860 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:34.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:34.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:34.962 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:35.003 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:35.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:35.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:35.050 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:35 smithi137 ceph-mon[39175]: 5.e scrub starts 2024-09-18T07:33:35.050 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:35 smithi137 ceph-mon[39175]: 5.e scrub ok 2024-09-18T07:33:35.050 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:35 smithi137 ceph-mon[39175]: 4.5 scrub starts 2024-09-18T07:33:35.050 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:35 smithi137 ceph-mon[39175]: 4.5 scrub ok 2024-09-18T07:33:35.050 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:35 smithi137 ceph-mon[39175]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-18T07:33:35.050 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:35 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:33:35.106 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:35.129 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:35.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:35.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:35 smithi203 ceph-mon[32237]: 5.e scrub starts 2024-09-18T07:33:35.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:35 smithi203 ceph-mon[32237]: 5.e scrub ok 2024-09-18T07:33:35.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:35 smithi203 ceph-mon[32237]: 4.5 scrub starts 2024-09-18T07:33:35.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:35 smithi203 ceph-mon[32237]: 4.5 scrub ok 2024-09-18T07:33:35.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:35 smithi203 ceph-mon[32237]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-18T07:33:35.205 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:35 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:33:35.231 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:35.249 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T07:33:35.272 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:35.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:35.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:35.310 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:35 smithi137 ceph-mon[29408]: 5.e scrub starts 2024-09-18T07:33:35.310 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:35 smithi137 ceph-mon[29408]: 5.e scrub ok 2024-09-18T07:33:35.310 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:35 smithi137 ceph-mon[29408]: 4.5 scrub starts 2024-09-18T07:33:35.310 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:35 smithi137 ceph-mon[29408]: 4.5 scrub ok 2024-09-18T07:33:35.310 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:35 smithi137 ceph-mon[29408]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-18T07:33:35.311 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:35 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:33:35.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:35.414 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:35.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:35.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:35.516 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:35.547 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:35.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:35.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:35.564 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:33:35.649 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:35.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:35.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:35.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:35.788 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:33:35.786991+0000 min lat: 0.000980073 max lat: 2.1848 avg lat: 0.0135657 2024-09-18T07:33:35.788 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:33:35.788 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 80 16 94359 94343 73.6867 66.5 0.00947711 0.0135657 2024-09-18T07:33:35.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:35.816 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:35.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:35.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:35.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:35.955 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:35.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:35.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:36.057 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:36.083 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:36.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:36.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:36.115 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:33:36.115 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:33:36.115 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-09-18T07:33:36.116 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:33:36.185 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:36.210 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:36.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:36.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:36.312 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:36.333 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:36 smithi137 ceph-mon[29408]: 5.b scrub starts 2024-09-18T07:33:36.334 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:36 smithi137 ceph-mon[29408]: 5.b scrub ok 2024-09-18T07:33:36.334 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:36 smithi137 ceph-mon[29408]: pgmap v117: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 37 MiB/s wr, 588 op/s; 13 B/s, 1 objects/s recovering 2024-09-18T07:33:36.334 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:36 smithi137 ceph-mon[39175]: 5.b scrub starts 2024-09-18T07:33:36.334 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:36 smithi137 ceph-mon[39175]: 5.b scrub ok 2024-09-18T07:33:36.334 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:36 smithi137 ceph-mon[39175]: pgmap v117: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 37 MiB/s wr, 588 op/s; 13 B/s, 1 objects/s recovering 2024-09-18T07:33:36.355 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:36.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:36.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:36.430 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:36 smithi203 ceph-mon[32237]: 5.b scrub starts 2024-09-18T07:33:36.430 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:36 smithi203 ceph-mon[32237]: 5.b scrub ok 2024-09-18T07:33:36.430 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:36 smithi203 ceph-mon[32237]: pgmap v117: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 37 MiB/s wr, 588 op/s; 13 B/s, 1 objects/s recovering 2024-09-18T07:33:36.457 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:36.481 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:36.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:36.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:36.583 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:36.607 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:36.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:36.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:36.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:36.749 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:36.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:36.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:36.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:33:36.784 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T07:33:36.788 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 81 16 95359 95343 73.5486 62.5 0.0110473 0.0135909 2024-09-18T07:33:36.852 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:36.882 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:36.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:36.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:36.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:37.012 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:37.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:37.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:37.063 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.5 to 1 (10000) 2024-09-18T07:33:37.071 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:33:37.078 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:33:36 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:33:37.092 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:33:37.114 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:37.164 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:37.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:37.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:37.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:37.308 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:37.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:37.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:37.333 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[29408]: 6.12 scrub starts 2024-09-18T07:33:37.333 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[29408]: 6.12 scrub ok 2024-09-18T07:33:37.333 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[29408]: 4.1e scrub starts 2024-09-18T07:33:37.334 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[29408]: 4.1e scrub ok 2024-09-18T07:33:37.334 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/161345449' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:33:37.334 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[29408]: 5.f scrub starts 2024-09-18T07:33:37.334 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[29408]: 5.f scrub ok 2024-09-18T07:33:37.334 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/462290376' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:33:37.334 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[29408]: from='client.24877 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:33:37.334 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:37.334 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:33:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:33:37.338 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[39175]: 6.12 scrub starts 2024-09-18T07:33:37.338 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[39175]: 6.12 scrub ok 2024-09-18T07:33:37.338 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[39175]: 4.1e scrub starts 2024-09-18T07:33:37.338 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[39175]: 4.1e scrub ok 2024-09-18T07:33:37.338 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/161345449' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:33:37.338 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[39175]: 5.f scrub starts 2024-09-18T07:33:37.338 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[39175]: 5.f scrub ok 2024-09-18T07:33:37.338 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/462290376' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:33:37.338 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[39175]: from='client.24877 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:33:37.338 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:37 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:37.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:37 smithi203 ceph-mon[32237]: 6.12 scrub starts 2024-09-18T07:33:37.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:37 smithi203 ceph-mon[32237]: 6.12 scrub ok 2024-09-18T07:33:37.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:37 smithi203 ceph-mon[32237]: 4.1e scrub starts 2024-09-18T07:33:37.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:37 smithi203 ceph-mon[32237]: 4.1e scrub ok 2024-09-18T07:33:37.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:37 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/161345449' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:33:37.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:37 smithi203 ceph-mon[32237]: 5.f scrub starts 2024-09-18T07:33:37.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:37 smithi203 ceph-mon[32237]: 5.f scrub ok 2024-09-18T07:33:37.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:37 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/462290376' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:33:37.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:37 smithi203 ceph-mon[32237]: from='client.24877 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:33:37.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:37 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:37.410 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:37.435 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:37.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:37.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:37.564 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:37.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:37.639 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:33:37.639 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:33:37.639 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:33:37.639 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:33:37.639 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:33:37.666 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:37.712 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:37.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:37.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:37.788 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 82 15 96294 96279 73.365 58.5 0.0110454 0.0136248 2024-09-18T07:33:37.814 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:37.840 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:37.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:37.942 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:37.966 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:37.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:37.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:38.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:38.092 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:38.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:38.196 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:38.219 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:38.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:38.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:38.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:38.350 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:38.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:38.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:38.361 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:38 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:33:38.361 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:38 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/161345449' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:33:38.361 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:38 smithi203 ceph-mon[32237]: from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:33:38.361 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:38 smithi203 ceph-mon[32237]: osdmap e123: 8 total, 8 up, 7 in 2024-09-18T07:33:38.361 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:38 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:38.361 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:38 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:38.361 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:38 smithi203 ceph-mon[32237]: pgmap v119: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 13 B/s, 1 objects/s recovering 2024-09-18T07:33:38.361 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:38 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:33:38.362 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:38 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:33:38.362 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:38 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:38.363 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T07:33:38.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:33:38.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/161345449' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:33:38.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[29408]: from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:33:38.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[29408]: osdmap e123: 8 total, 8 up, 7 in 2024-09-18T07:33:38.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:38.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:38.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[29408]: pgmap v119: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 13 B/s, 1 objects/s recovering 2024-09-18T07:33:38.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:33:38.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:33:38.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:38.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:33:38.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/161345449' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:33:38.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[39175]: from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:33:38.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[39175]: osdmap e123: 8 total, 8 up, 7 in 2024-09-18T07:33:38.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:38.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:38.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[39175]: pgmap v119: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 13 B/s, 1 objects/s recovering 2024-09-18T07:33:38.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:33:38.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:33:38.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:38 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:33:38.451 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:38.480 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:38.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:38.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:38.582 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:38.609 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:38.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:38.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:38.711 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:38.722 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:33:38.745 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:38.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:38.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:38.788 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 83 16 97328 97312 73.2589 64.5625 0.00746698 0.0136453 2024-09-18T07:33:38.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:38.883 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:38.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:38.985 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:39.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:39.029 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:39.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:39.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:39.170 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:39.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:39.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:39.257 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:33:39.258 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:33:39.258 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:33:39.258 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:33:39.258 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.258 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21284 7.0 7:05208463:::benchmark_data_smithi137_74180_object21283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:33:39.258 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.944136+0000", 2024-09-18T07:33:39.258 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 75.309343112999997, 2024-09-18T07:33:39.258 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.103616704, 2024-09-18T07:33:39.258 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.258 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.259 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.259 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:39.259 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:39.259 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:39.259 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21284 2024-09-18T07:33:39.259 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.259 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.259 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.259 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.259 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944136+0000", 2024-09-18T07:33:39.259 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.259 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.259 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.260 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.260 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944136+0000", 2024-09-18T07:33:39.260 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.260 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.260 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.260 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.260 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944139+0000", 2024-09-18T07:33:39.260 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3539999999999998e-06 2024-09-18T07:33:39.260 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.260 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.260 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.260 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944205+0000", 2024-09-18T07:33:39.261 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5979e-05 2024-09-18T07:33:39.261 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.261 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.261 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.261 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944218+0000", 2024-09-18T07:33:39.261 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2965999999999999e-05 2024-09-18T07:33:39.261 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.261 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.261 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.261 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944226+0000", 2024-09-18T07:33:39.261 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3219999999999993e-06 2024-09-18T07:33:39.261 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.261 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.262 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.262 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944278+0000", 2024-09-18T07:33:39.262 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2148000000000003e-05 2024-09-18T07:33:39.262 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.262 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.262 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.262 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.043757+0000", 2024-09-18T07:33:39.262 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.099479174000000004 2024-09-18T07:33:39.262 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.262 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.262 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:33:39.262 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.043893+0000", 2024-09-18T07:33:39.262 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000135661 2024-09-18T07:33:39.263 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.263 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.263 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.263 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047550+0000", 2024-09-18T07:33:39.263 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003656898 2024-09-18T07:33:39.263 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.263 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.263 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:39.263 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047565+0000", 2024-09-18T07:33:39.263 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5488000000000001e-05 2024-09-18T07:33:39.263 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.263 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.263 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.264 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047660+0000", 2024-09-18T07:33:39.264 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4287e-05 2024-09-18T07:33:39.264 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.264 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.264 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.264 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047708+0000", 2024-09-18T07:33:39.264 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8522000000000001e-05 2024-09-18T07:33:39.264 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.264 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.264 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.264 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047753+0000", 2024-09-18T07:33:39.264 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4904999999999998e-05 2024-09-18T07:33:39.264 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.265 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.265 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.265 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.265 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.265 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:21283 7.e e89/81 7:72ad1dc4:::benchmark_data_smithi137_74180_object21282:head v 89'1356, mlcod=89'1354)", 2024-09-18T07:33:39.265 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.944608+0000", 2024-09-18T07:33:39.265 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 75.308871963000001, 2024-09-18T07:33:39.265 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.101997011, 2024-09-18T07:33:39.265 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.265 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.265 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.265 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.265 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.266 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.266 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944608+0000", 2024-09-18T07:33:39.266 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.266 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.266 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.266 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.266 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944608+0000", 2024-09-18T07:33:39.266 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.266 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.266 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.266 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.266 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944609+0000", 2024-09-18T07:33:39.266 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.305e-06 2024-09-18T07:33:39.267 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.267 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.267 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.267 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944648+0000", 2024-09-18T07:33:39.267 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9477000000000002e-05 2024-09-18T07:33:39.267 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.267 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.267 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.267 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.044974+0000", 2024-09-18T07:33:39.267 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.100325449 2024-09-18T07:33:39.267 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.267 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.267 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.268 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.044984+0000", 2024-09-18T07:33:39.268 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0251999999999999e-05 2024-09-18T07:33:39.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.268 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.268 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.268 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.045046+0000", 2024-09-18T07:33:39.268 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1744999999999999e-05 2024-09-18T07:33:39.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.268 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.268 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.268 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.045066+0000", 2024-09-18T07:33:39.268 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9763e-05 2024-09-18T07:33:39.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.269 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.269 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.269 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.046547+0000", 2024-09-18T07:33:39.269 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014817229999999999 2024-09-18T07:33:39.269 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.269 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.269 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.269 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.046605+0000", 2024-09-18T07:33:39.269 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7296999999999997e-05 2024-09-18T07:33:39.269 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.269 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.269 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.269 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.270 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.270 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22001 7.0 7:090cc9b7:::benchmark_data_smithi137_74180_object22000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:33:39.270 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.425471+0000", 2024-09-18T07:33:39.270 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 74.828008896, 2024-09-18T07:33:39.270 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.11142413700000001, 2024-09-18T07:33:39.270 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.270 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.270 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.270 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:39.270 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:39.270 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:39.270 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22001 2024-09-18T07:33:39.270 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.271 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.271 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.271 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.271 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425471+0000", 2024-09-18T07:33:39.271 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.271 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.271 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.271 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.271 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425471+0000", 2024-09-18T07:33:39.271 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.271 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.271 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.271 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.272 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425472+0000", 2024-09-18T07:33:39.272 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5650000000000001e-06 2024-09-18T07:33:39.272 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.272 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.272 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.272 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425519+0000", 2024-09-18T07:33:39.272 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7207999999999999e-05 2024-09-18T07:33:39.272 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.272 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.272 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.272 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425531+0000", 2024-09-18T07:33:39.273 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1249e-05 2024-09-18T07:33:39.273 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.273 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.273 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.273 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425538+0000", 2024-09-18T07:33:39.273 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.357e-06 2024-09-18T07:33:39.273 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.273 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.273 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.273 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425599+0000", 2024-09-18T07:33:39.273 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0588999999999999e-05 2024-09-18T07:33:39.273 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.274 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.274 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.274 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425719+0000", 2024-09-18T07:33:39.274 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000120269 2024-09-18T07:33:39.274 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.274 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.274 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:33:39.274 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425862+0000", 2024-09-18T07:33:39.274 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014290700000000001 2024-09-18T07:33:39.274 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.274 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.274 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:39.274 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.426665+0000", 2024-09-18T07:33:39.275 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00080358199999999995 2024-09-18T07:33:39.275 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.275 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.275 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.275 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.427231+0000", 2024-09-18T07:33:39.275 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00056515199999999997 2024-09-18T07:33:39.275 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.275 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.275 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.275 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536788+0000", 2024-09-18T07:33:39.275 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.109557486 2024-09-18T07:33:39.275 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.275 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.276 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.276 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536850+0000", 2024-09-18T07:33:39.276 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2207999999999998e-05 2024-09-18T07:33:39.276 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.276 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.276 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.276 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536895+0000", 2024-09-18T07:33:39.276 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4564999999999998e-05 2024-09-18T07:33:39.276 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.276 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.276 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.276 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.276 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.276 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22346 7.0 7:0d479b9b:::benchmark_data_smithi137_74180_object22345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:33:39.277 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.661502+0000", 2024-09-18T07:33:39.277 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 74.591977486999994, 2024-09-18T07:33:39.277 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19330275799999999, 2024-09-18T07:33:39.277 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.277 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.277 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.277 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:39.277 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:39.277 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:39.277 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22346 2024-09-18T07:33:39.277 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.277 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.278 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.278 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.278 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661502+0000", 2024-09-18T07:33:39.278 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.278 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.278 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.278 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.278 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661502+0000", 2024-09-18T07:33:39.278 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.278 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.278 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.278 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.278 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661503+0000", 2024-09-18T07:33:39.278 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7400000000000002e-07 2024-09-18T07:33:39.279 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.279 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.279 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.279 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661554+0000", 2024-09-18T07:33:39.279 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1014000000000003e-05 2024-09-18T07:33:39.279 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.279 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.279 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.279 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661561+0000", 2024-09-18T07:33:39.279 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3370000000000002e-06 2024-09-18T07:33:39.279 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.279 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.279 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.279 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661567+0000", 2024-09-18T07:33:39.280 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2580000000000003e-06 2024-09-18T07:33:39.280 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.280 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.280 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.280 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661598+0000", 2024-09-18T07:33:39.280 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1810999999999997e-05 2024-09-18T07:33:39.280 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.280 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.280 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.280 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852170+0000", 2024-09-18T07:33:39.280 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.190571617 2024-09-18T07:33:39.280 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.281 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.281 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:33:39.281 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852341+0000", 2024-09-18T07:33:39.281 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017095599999999999 2024-09-18T07:33:39.281 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.281 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.281 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:39.281 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.854512+0000", 2024-09-18T07:33:39.281 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021714299999999998 2024-09-18T07:33:39.281 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.281 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.281 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.282 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.854589+0000", 2024-09-18T07:33:39.282 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6168000000000007e-05 2024-09-18T07:33:39.282 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.282 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.282 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.282 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.854709+0000", 2024-09-18T07:33:39.282 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000120951 2024-09-18T07:33:39.282 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.282 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.282 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.282 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.854765+0000", 2024-09-18T07:33:39.282 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5761999999999999e-05 2024-09-18T07:33:39.282 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.283 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.283 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.283 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.854805+0000", 2024-09-18T07:33:39.283 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9579999999999997e-05 2024-09-18T07:33:39.283 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.283 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.283 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.283 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.283 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.283 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22349 7.0 7:0d8b0d01:::benchmark_data_smithi137_74180_object22348:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:33:39.283 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.662222+0000", 2024-09-18T07:33:39.283 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 74.591257382999999, 2024-09-18T07:33:39.283 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19509826899999999, 2024-09-18T07:33:39.284 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.284 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.284 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.284 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:39.284 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:39.284 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:39.284 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22349 2024-09-18T07:33:39.284 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.284 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.284 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.284 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.284 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662222+0000", 2024-09-18T07:33:39.284 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.285 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.285 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.285 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.285 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662222+0000", 2024-09-18T07:33:39.285 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.285 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.285 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.285 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.285 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662223+0000", 2024-09-18T07:33:39.285 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9399999999999993e-07 2024-09-18T07:33:39.285 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.286 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.286 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.286 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662284+0000", 2024-09-18T07:33:39.286 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0887000000000002e-05 2024-09-18T07:33:39.286 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.286 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.286 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.286 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852186+0000", 2024-09-18T07:33:39.286 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18990221400000001 2024-09-18T07:33:39.286 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.286 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.286 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.286 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852200+0000", 2024-09-18T07:33:39.287 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3703000000000001e-05 2024-09-18T07:33:39.287 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.287 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.287 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.287 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.853205+0000", 2024-09-18T07:33:39.287 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010050949999999999 2024-09-18T07:33:39.287 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.287 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.287 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.287 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:39.289 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.853308+0000", 2024-09-18T07:33:39.289 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010267700000000001 2024-09-18T07:33:39.289 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.289 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.289 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:33:39.289 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.853390+0000", 2024-09-18T07:33:39.289 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2711000000000001e-05 2024-09-18T07:33:39.289 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.289 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.289 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:39.290 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.854896+0000", 2024-09-18T07:33:39.290 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00150526 2024-09-18T07:33:39.290 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.290 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.290 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.290 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.855065+0000", 2024-09-18T07:33:39.290 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000169457 2024-09-18T07:33:39.290 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.290 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.290 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.290 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.857250+0000", 2024-09-18T07:33:39.290 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002184746 2024-09-18T07:33:39.290 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.291 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.291 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.291 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.857282+0000", 2024-09-18T07:33:39.291 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2005000000000003e-05 2024-09-18T07:33:39.291 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.291 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.291 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.291 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.857320+0000", 2024-09-18T07:33:39.291 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8519999999999997e-05 2024-09-18T07:33:39.291 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.291 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.291 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.291 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.292 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.292 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:44361 7.0 7:0e498b2d:::benchmark_data_smithi137_74180_object44360:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-18T07:33:39.292 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:43.589008+0000", 2024-09-18T07:33:39.292 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 55.664471382000002, 2024-09-18T07:33:39.292 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.11329566200000001, 2024-09-18T07:33:39.292 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.292 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.292 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.293 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:39.293 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:39.293 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:39.293 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 44361 2024-09-18T07:33:39.293 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.293 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.293 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.293 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.293 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.589008+0000", 2024-09-18T07:33:39.293 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.293 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.293 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.294 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.294 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.589008+0000", 2024-09-18T07:33:39.294 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.294 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.294 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.294 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.294 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.589010+0000", 2024-09-18T07:33:39.294 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5349999999999999e-06 2024-09-18T07:33:39.294 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.294 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.294 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.294 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.589058+0000", 2024-09-18T07:33:39.294 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8016999999999998e-05 2024-09-18T07:33:39.294 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.295 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.295 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.295 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.589068+0000", 2024-09-18T07:33:39.295 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0352e-05 2024-09-18T07:33:39.295 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.295 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.295 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.295 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.589076+0000", 2024-09-18T07:33:39.295 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1510000000000004e-06 2024-09-18T07:33:39.295 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.295 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.295 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.295 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.589133+0000", 2024-09-18T07:33:39.295 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6786000000000001e-05 2024-09-18T07:33:39.296 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.296 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.296 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.296 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.589237+0000", 2024-09-18T07:33:39.296 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000103997 2024-09-18T07:33:39.296 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.296 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.296 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:33:39.296 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.589349+0000", 2024-09-18T07:33:39.296 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011226900000000001 2024-09-18T07:33:39.296 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.296 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.296 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:39.297 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.590688+0000", 2024-09-18T07:33:39.297 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001338278 2024-09-18T07:33:39.297 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.297 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.297 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.297 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.592834+0000", 2024-09-18T07:33:39.297 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021467779999999998 2024-09-18T07:33:39.297 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.297 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.297 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.297 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.702185+0000", 2024-09-18T07:33:39.297 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.109351133 2024-09-18T07:33:39.298 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.298 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.298 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.298 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.702255+0000", 2024-09-18T07:33:39.298 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9592999999999996e-05 2024-09-18T07:33:39.298 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.298 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.298 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.298 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.702304+0000", 2024-09-18T07:33:39.298 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8773000000000002e-05 2024-09-18T07:33:39.298 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.298 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.298 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.299 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.299 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.299 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:44366 7.0 7:06248f44:::benchmark_data_smithi137_74180_object44365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-18T07:33:39.299 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:43.591174+0000", 2024-09-18T07:33:39.299 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 55.662305377999999, 2024-09-18T07:33:39.299 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.117567228, 2024-09-18T07:33:39.299 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.299 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.299 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.299 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:39.299 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:39.299 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:39.300 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 44366 2024-09-18T07:33:39.300 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.300 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.300 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.300 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.300 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591174+0000", 2024-09-18T07:33:39.300 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.300 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.300 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.300 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.300 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591174+0000", 2024-09-18T07:33:39.300 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.300 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.301 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.301 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.301 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591175+0000", 2024-09-18T07:33:39.301 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.17e-06 2024-09-18T07:33:39.301 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.301 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.301 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.301 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591232+0000", 2024-09-18T07:33:39.301 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6512999999999999e-05 2024-09-18T07:33:39.301 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.301 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.302 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.302 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591241+0000", 2024-09-18T07:33:39.302 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6750000000000008e-06 2024-09-18T07:33:39.302 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.302 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.302 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.302 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591248+0000", 2024-09-18T07:33:39.302 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0559999999999999e-06 2024-09-18T07:33:39.302 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.302 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.302 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.302 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591298+0000", 2024-09-18T07:33:39.302 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0107e-05 2024-09-18T07:33:39.303 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.303 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.303 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.303 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591450+0000", 2024-09-18T07:33:39.303 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015223999999999999 2024-09-18T07:33:39.303 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.303 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.303 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:33:39.303 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591627+0000", 2024-09-18T07:33:39.303 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017697599999999999 2024-09-18T07:33:39.303 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.303 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.303 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:39.304 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.593303+0000", 2024-09-18T07:33:39.304 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016765809999999999 2024-09-18T07:33:39.304 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.304 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.304 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.304 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.593444+0000", 2024-09-18T07:33:39.304 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000140562 2024-09-18T07:33:39.304 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.304 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.304 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.304 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.708663+0000", 2024-09-18T07:33:39.304 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.115219341 2024-09-18T07:33:39.304 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.305 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.305 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.305 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.708701+0000", 2024-09-18T07:33:39.305 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7540000000000003e-05 2024-09-18T07:33:39.305 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.305 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.305 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.305 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.708741+0000", 2024-09-18T07:33:39.305 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0466999999999997e-05 2024-09-18T07:33:39.305 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.305 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.305 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.306 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.306 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.306 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "MOSDPGPush(7.6 100/94 [PushOp(7:6b9338fa:::benchmark_data_smithi137_74180_object18095:head, version: 88'1110, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:6b9338fa:::benchmark_data_smithi137_74180_object18095:head@88'1110, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-18T07:33:39.306 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:50.954756+0000", 2024-09-18T07:33:39.306 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 48.298723488, 2024-09-18T07:33:39.306 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.12398751600000001, 2024-09-18T07:33:39.306 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.306 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.306 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:33:39.306 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.306 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.306 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.306 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.954756+0000", 2024-09-18T07:33:39.306 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.307 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.307 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.307 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.307 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.954756+0000", 2024-09-18T07:33:39.307 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.307 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.307 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.307 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.307 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.954759+0000", 2024-09-18T07:33:39.307 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5040000000000001e-06 2024-09-18T07:33:39.307 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.307 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.307 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.308 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.954824+0000", 2024-09-18T07:33:39.308 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5883999999999995e-05 2024-09-18T07:33:39.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.308 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.308 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.308 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.954864+0000", 2024-09-18T07:33:39.308 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9066000000000002e-05 2024-09-18T07:33:39.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.308 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.308 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.308 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.954893+0000", 2024-09-18T07:33:39.308 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9136e-05 2024-09-18T07:33:39.309 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.309 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.309 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.309 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.955058+0000", 2024-09-18T07:33:39.309 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016571199999999999 2024-09-18T07:33:39.309 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.309 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.309 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.309 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.955074+0000", 2024-09-18T07:33:39.309 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5704000000000001e-05 2024-09-18T07:33:39.309 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.309 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.309 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.310 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.078744+0000", 2024-09-18T07:33:39.310 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.12366951 2024-09-18T07:33:39.310 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.310 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.310 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.310 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.310 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.310 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:51954 7.0 7:0986551b:::benchmark_data_smithi137_74180_object51953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:33:39.310 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:50.954983+0000", 2024-09-18T07:33:39.310 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 48.298496256, 2024-09-18T07:33:39.310 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.126439316, 2024-09-18T07:33:39.310 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.310 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.311 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.311 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:39.311 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:39.311 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:39.311 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 51954 2024-09-18T07:33:39.311 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.311 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.311 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.311 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.311 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.954983+0000", 2024-09-18T07:33:39.311 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.311 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.311 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.312 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.312 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.954983+0000", 2024-09-18T07:33:39.312 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.312 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.312 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.312 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.312 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.954985+0000", 2024-09-18T07:33:39.312 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.282e-06 2024-09-18T07:33:39.312 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.312 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.312 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.312 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.955009+0000", 2024-09-18T07:33:39.312 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4545999999999998e-05 2024-09-18T07:33:39.313 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.313 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.313 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.313 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.955016+0000", 2024-09-18T07:33:39.313 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7519999999999999e-06 2024-09-18T07:33:39.313 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.313 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.313 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.313 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.955020+0000", 2024-09-18T07:33:39.313 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3359999999999997e-06 2024-09-18T07:33:39.313 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.313 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.313 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.313 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.955070+0000", 2024-09-18T07:33:39.314 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9339999999999999e-05 2024-09-18T07:33:39.314 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.314 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.314 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.314 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.077581+0000", 2024-09-18T07:33:39.314 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.122511359 2024-09-18T07:33:39.314 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.314 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.314 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:33:39.314 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.077732+0000", 2024-09-18T07:33:39.314 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000151543 2024-09-18T07:33:39.314 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.314 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.315 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:39.315 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.080809+0000", 2024-09-18T07:33:39.315 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0030769360000000002 2024-09-18T07:33:39.315 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.315 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.315 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.315 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.081242+0000", 2024-09-18T07:33:39.315 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00043269400000000001 2024-09-18T07:33:39.315 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.315 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.315 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.315 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.081355+0000", 2024-09-18T07:33:39.316 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011283200000000001 2024-09-18T07:33:39.316 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.316 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.316 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.316 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.081390+0000", 2024-09-18T07:33:39.316 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5327000000000002e-05 2024-09-18T07:33:39.316 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.316 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.316 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.316 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.081423+0000", 2024-09-18T07:33:39.316 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2369000000000001e-05 2024-09-18T07:33:39.316 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.316 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.316 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.317 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.317 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.317 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:51962 7.0 7:0989bbb3:::benchmark_data_smithi137_74180_object51961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:33:39.317 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:50.955033+0000", 2024-09-18T07:33:39.317 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 48.298446259000002, 2024-09-18T07:33:39.317 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.126734821, 2024-09-18T07:33:39.317 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.317 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.317 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.317 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:39.317 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:39.317 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:39.317 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 51962 2024-09-18T07:33:39.317 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.318 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.318 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.318 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.318 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.955033+0000", 2024-09-18T07:33:39.318 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.318 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.318 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.318 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.318 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.955033+0000", 2024-09-18T07:33:39.318 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.318 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.318 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.318 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.319 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.955034+0000", 2024-09-18T07:33:39.319 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1799999999999995e-07 2024-09-18T07:33:39.319 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.319 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.319 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.319 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.955102+0000", 2024-09-18T07:33:39.319 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8521000000000003e-05 2024-09-18T07:33:39.319 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.319 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.319 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.319 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.955110+0000", 2024-09-18T07:33:39.319 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3869999999999997e-06 2024-09-18T07:33:39.319 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.319 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.320 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.320 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.955115+0000", 2024-09-18T07:33:39.320 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7219999999999996e-06 2024-09-18T07:33:39.320 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.320 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.320 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.320 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.079342+0000", 2024-09-18T07:33:39.320 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.124226318 2024-09-18T07:33:39.320 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.320 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.320 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.320 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.079460+0000", 2024-09-18T07:33:39.321 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000118003 2024-09-18T07:33:39.321 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.321 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.321 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:33:39.321 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.079589+0000", 2024-09-18T07:33:39.321 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012966400000000001 2024-09-18T07:33:39.321 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.321 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.321 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:39.321 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.081024+0000", 2024-09-18T07:33:39.321 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014349829999999999 2024-09-18T07:33:39.321 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.322 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.322 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.322 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.081538+0000", 2024-09-18T07:33:39.322 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000513592 2024-09-18T07:33:39.322 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.322 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.322 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.322 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.081719+0000", 2024-09-18T07:33:39.322 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018109800000000001 2024-09-18T07:33:39.322 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.322 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.322 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.323 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.081747+0000", 2024-09-18T07:33:39.323 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7529e-05 2024-09-18T07:33:39.323 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.323 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.323 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.323 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.081768+0000", 2024-09-18T07:33:39.323 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1486000000000001e-05 2024-09-18T07:33:39.323 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.323 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.323 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.323 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.323 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.324 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:51957 7.b e100/81 7:dff3716a:::benchmark_data_smithi137_74180_object51956:head v 100'3197, mlcod=100'3195)", 2024-09-18T07:33:39.324 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:50.956549+0000", 2024-09-18T07:33:39.324 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 48.296930772000003, 2024-09-18T07:33:39.324 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.123588503, 2024-09-18T07:33:39.324 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.324 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.324 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.324 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.324 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.324 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.324 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.956549+0000", 2024-09-18T07:33:39.324 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.325 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.325 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.325 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.325 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.956549+0000", 2024-09-18T07:33:39.325 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.325 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.325 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.325 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.325 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.956550+0000", 2024-09-18T07:33:39.325 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4050000000000001e-06 2024-09-18T07:33:39.325 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.325 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.326 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.326 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.956590+0000", 2024-09-18T07:33:39.326 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9669999999999998e-05 2024-09-18T07:33:39.326 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.326 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.326 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.326 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.077306+0000", 2024-09-18T07:33:39.326 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.12071649600000001 2024-09-18T07:33:39.326 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.326 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.327 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.327 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.077322+0000", 2024-09-18T07:33:39.327 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5775999999999999e-05 2024-09-18T07:33:39.327 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.327 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.327 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.327 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.077387+0000", 2024-09-18T07:33:39.327 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5042000000000005e-05 2024-09-18T07:33:39.327 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.327 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.327 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.328 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.077412+0000", 2024-09-18T07:33:39.328 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4708000000000001e-05 2024-09-18T07:33:39.328 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.328 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.328 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.328 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.080096+0000", 2024-09-18T07:33:39.328 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026842390000000002 2024-09-18T07:33:39.328 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.328 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.328 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.328 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.080137+0000", 2024-09-18T07:33:39.328 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1167000000000001e-05 2024-09-18T07:33:39.329 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.329 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.329 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.329 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.329 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.329 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:51976 7.a e100/81 7:53a8d4b1:::benchmark_data_smithi137_74180_object51975:head v 100'3275, mlcod=100'3273)", 2024-09-18T07:33:39.329 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:50.969306+0000", 2024-09-18T07:33:39.329 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 48.284173387999999, 2024-09-18T07:33:39.329 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.110851926, 2024-09-18T07:33:39.329 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.329 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.329 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.329 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.330 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.330 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.330 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.969306+0000", 2024-09-18T07:33:39.330 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.330 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.330 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.330 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.330 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.969306+0000", 2024-09-18T07:33:39.330 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.330 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.331 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.331 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.331 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.969308+0000", 2024-09-18T07:33:39.331 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.012e-06 2024-09-18T07:33:39.331 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.331 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.331 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.331 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.969362+0000", 2024-09-18T07:33:39.331 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4308000000000002e-05 2024-09-18T07:33:39.331 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.331 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.331 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.331 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.077330+0000", 2024-09-18T07:33:39.331 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.107967226 2024-09-18T07:33:39.332 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.332 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.332 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.332 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.077340+0000", 2024-09-18T07:33:39.332 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0213e-05 2024-09-18T07:33:39.332 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.332 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.332 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.332 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.077400+0000", 2024-09-18T07:33:39.332 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0306000000000002e-05 2024-09-18T07:33:39.332 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.332 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.332 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.333 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.077419+0000", 2024-09-18T07:33:39.333 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9225e-05 2024-09-18T07:33:39.333 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.333 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.333 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.333 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.080118+0000", 2024-09-18T07:33:39.333 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026983770000000001 2024-09-18T07:33:39.333 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.333 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.333 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.334 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:51.080158+0000", 2024-09-18T07:33:39.334 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0259000000000002e-05 2024-09-18T07:33:39.335 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.335 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.335 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.335 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.335 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.335 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:72386 7.0 7:0967b059:::benchmark_data_smithi137_74180_object72385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-18T07:33:39.335 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:13.284634+0000", 2024-09-18T07:33:39.335 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 25.968845939000001, 2024-09-18T07:33:39.335 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10015186600000001, 2024-09-18T07:33:39.335 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.335 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.335 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.336 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:39.336 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:39.336 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:39.336 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 72386 2024-09-18T07:33:39.336 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.336 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.336 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.336 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.336 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:13.284634+0000", 2024-09-18T07:33:39.336 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.336 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.336 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.337 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.337 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:13.284634+0000", 2024-09-18T07:33:39.337 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.337 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.337 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.337 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.337 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:13.284635+0000", 2024-09-18T07:33:39.337 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.42e-06 2024-09-18T07:33:39.337 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.337 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.337 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.337 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:13.284695+0000", 2024-09-18T07:33:39.337 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0460000000000001e-05 2024-09-18T07:33:39.338 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.338 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.338 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.338 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:13.284706+0000", 2024-09-18T07:33:39.338 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0385e-05 2024-09-18T07:33:39.338 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.338 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.338 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.338 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:13.284714+0000", 2024-09-18T07:33:39.338 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0360000000000006e-06 2024-09-18T07:33:39.338 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.338 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.338 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.339 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:13.284767+0000", 2024-09-18T07:33:39.339 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3258999999999999e-05 2024-09-18T07:33:39.339 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.339 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.339 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.339 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:13.284895+0000", 2024-09-18T07:33:39.339 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000128097 2024-09-18T07:33:39.339 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.339 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.339 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:33:39.339 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:13.285047+0000", 2024-09-18T07:33:39.339 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000152108 2024-09-18T07:33:39.339 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.340 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.340 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:39.340 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:13.287079+0000", 2024-09-18T07:33:39.340 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0020313810000000001 2024-09-18T07:33:39.340 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.340 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.340 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.340 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:13.293575+0000", 2024-09-18T07:33:39.340 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0064965400000000003 2024-09-18T07:33:39.340 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.340 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.340 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.340 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:13.384674+0000", 2024-09-18T07:33:39.341 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.091098215999999996 2024-09-18T07:33:39.341 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.341 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.341 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.341 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:13.384734+0000", 2024-09-18T07:33:39.341 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0000999999999997e-05 2024-09-18T07:33:39.341 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.341 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.341 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.341 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:13.384785+0000", 2024-09-18T07:33:39.341 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1962999999999997e-05 2024-09-18T07:33:39.341 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.341 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.342 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.342 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.342 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.342 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:74407 7.0 7:0a7af25b:::benchmark_data_smithi137_74180_object74406:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:33:39.342 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:15.839101+0000", 2024-09-18T07:33:39.342 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 23.414379023999999, 2024-09-18T07:33:39.342 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13619060199999999, 2024-09-18T07:33:39.342 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.342 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.342 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.342 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:39.342 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:39.342 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:39.343 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 74407 2024-09-18T07:33:39.343 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.343 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.343 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.343 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.343 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:15.839101+0000", 2024-09-18T07:33:39.343 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.343 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.343 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.343 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.343 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:15.839101+0000", 2024-09-18T07:33:39.343 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.343 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.343 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.344 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.344 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:15.839102+0000", 2024-09-18T07:33:39.344 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.44e-06 2024-09-18T07:33:39.344 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.344 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.344 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.344 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:15.839154+0000", 2024-09-18T07:33:39.344 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2429000000000002e-05 2024-09-18T07:33:39.344 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.344 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.344 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.344 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:15.839165+0000", 2024-09-18T07:33:39.344 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0241000000000001e-05 2024-09-18T07:33:39.345 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.345 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.345 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.345 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:15.839170+0000", 2024-09-18T07:33:39.345 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1599999999999997e-06 2024-09-18T07:33:39.345 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.345 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.345 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.345 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:15.839213+0000", 2024-09-18T07:33:39.345 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2922e-05 2024-09-18T07:33:39.345 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.345 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.346 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.346 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:15.839308+0000", 2024-09-18T07:33:39.346 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5367999999999999e-05 2024-09-18T07:33:39.346 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.346 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.346 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:33:39.346 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:15.839404+0000", 2024-09-18T07:33:39.346 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5523e-05 2024-09-18T07:33:39.346 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.346 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.346 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:39.346 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:15.840952+0000", 2024-09-18T07:33:39.346 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0015487879999999999 2024-09-18T07:33:39.347 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.347 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.347 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.347 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:15.841525+0000", 2024-09-18T07:33:39.347 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00057216799999999996 2024-09-18T07:33:39.347 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.347 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.347 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.347 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:15.975227+0000", 2024-09-18T07:33:39.347 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.133702343 2024-09-18T07:33:39.347 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.347 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.347 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.348 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:15.975259+0000", 2024-09-18T07:33:39.348 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2019e-05 2024-09-18T07:33:39.348 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.348 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.348 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.348 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:15.975291+0000", 2024-09-18T07:33:39.348 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2200999999999999e-05 2024-09-18T07:33:39.348 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.348 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.348 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.348 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.348 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.348 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:74666 7.0 7:058d7705:::benchmark_data_smithi137_74180_object74665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:33:39.348 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:16.166293+0000", 2024-09-18T07:33:39.349 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 23.087186332999998, 2024-09-18T07:33:39.349 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15084240900000001, 2024-09-18T07:33:39.349 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.349 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.349 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.349 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:39.349 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:39.349 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:39.349 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 74666 2024-09-18T07:33:39.349 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.349 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.349 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.349 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.350 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166293+0000", 2024-09-18T07:33:39.350 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.350 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.350 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.350 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.350 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166293+0000", 2024-09-18T07:33:39.350 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.350 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.350 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.350 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.350 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166295+0000", 2024-09-18T07:33:39.350 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.53e-06 2024-09-18T07:33:39.350 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.351 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.351 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.351 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166353+0000", 2024-09-18T07:33:39.351 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8069000000000003e-05 2024-09-18T07:33:39.351 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.351 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.351 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.351 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166363+0000", 2024-09-18T07:33:39.351 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0195e-05 2024-09-18T07:33:39.351 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.351 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.351 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.352 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166370+0000", 2024-09-18T07:33:39.352 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4329999999999998e-06 2024-09-18T07:33:39.352 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.352 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.352 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.352 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166412+0000", 2024-09-18T07:33:39.352 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1375999999999998e-05 2024-09-18T07:33:39.352 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.352 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.352 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.352 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166498+0000", 2024-09-18T07:33:39.352 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6080000000000003e-05 2024-09-18T07:33:39.352 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.353 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.353 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:33:39.353 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166594+0000", 2024-09-18T07:33:39.353 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6346000000000002e-05 2024-09-18T07:33:39.353 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.353 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.353 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:39.353 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.168253+0000", 2024-09-18T07:33:39.353 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001658423 2024-09-18T07:33:39.353 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.353 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.353 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.353 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.175204+0000", 2024-09-18T07:33:39.353 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069512180000000003 2024-09-18T07:33:39.354 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.354 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.354 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.354 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.317012+0000", 2024-09-18T07:33:39.354 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.141807866 2024-09-18T07:33:39.354 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.354 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.354 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.354 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.317080+0000", 2024-09-18T07:33:39.354 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7762000000000006e-05 2024-09-18T07:33:39.355 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.355 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.355 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.355 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.317136+0000", 2024-09-18T07:33:39.355 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6110999999999999e-05 2024-09-18T07:33:39.355 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.355 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.355 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.355 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.355 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.355 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:74672 7.0 7:0c3f6e43:::benchmark_data_smithi137_74180_object74671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:33:39.355 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:16.172698+0000", 2024-09-18T07:33:39.355 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 23.080781269999999, 2024-09-18T07:33:39.355 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.144571955, 2024-09-18T07:33:39.356 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.356 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.356 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.356 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:39.356 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:39.356 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:39.356 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 74672 2024-09-18T07:33:39.356 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.356 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.356 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.356 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.356 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.172698+0000", 2024-09-18T07:33:39.356 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.357 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.357 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.357 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.357 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.172698+0000", 2024-09-18T07:33:39.357 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.357 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.357 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.357 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.357 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.172700+0000", 2024-09-18T07:33:39.357 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8589999999999999e-06 2024-09-18T07:33:39.357 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.357 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.357 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.358 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.172755+0000", 2024-09-18T07:33:39.358 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5133000000000002e-05 2024-09-18T07:33:39.358 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.358 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.358 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.358 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.172766+0000", 2024-09-18T07:33:39.358 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0627e-05 2024-09-18T07:33:39.358 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.358 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.358 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.359 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.172773+0000", 2024-09-18T07:33:39.359 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6769999999999998e-06 2024-09-18T07:33:39.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.359 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.359 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.359 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.172852+0000", 2024-09-18T07:33:39.359 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9454000000000002e-05 2024-09-18T07:33:39.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.359 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.359 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.359 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.172954+0000", 2024-09-18T07:33:39.359 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000101702 2024-09-18T07:33:39.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.359 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.360 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:33:39.360 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.173051+0000", 2024-09-18T07:33:39.360 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7293000000000005e-05 2024-09-18T07:33:39.360 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.360 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.360 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:39.360 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.176801+0000", 2024-09-18T07:33:39.360 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003750087 2024-09-18T07:33:39.360 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.360 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.360 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.360 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.179269+0000", 2024-09-18T07:33:39.361 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0024674789999999999 2024-09-18T07:33:39.361 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.361 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.361 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.361 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.317180+0000", 2024-09-18T07:33:39.361 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.137911229 2024-09-18T07:33:39.361 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.361 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.361 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.361 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.317233+0000", 2024-09-18T07:33:39.361 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3597999999999997e-05 2024-09-18T07:33:39.361 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.362 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.362 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.362 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.317270+0000", 2024-09-18T07:33:39.362 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6816999999999997e-05 2024-09-18T07:33:39.362 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.362 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.362 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.362 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.362 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.362 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:74675 7.0 7:0421ef5f:::benchmark_data_smithi137_74180_object74674:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:33:39.362 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:16.173394+0000", 2024-09-18T07:33:39.362 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 23.080085426, 2024-09-18T07:33:39.362 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14466180100000001, 2024-09-18T07:33:39.363 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.363 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.363 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.363 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:39.363 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:39.363 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:39.363 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 74675 2024-09-18T07:33:39.363 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.363 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.363 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.363 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.363 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.173394+0000", 2024-09-18T07:33:39.363 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.363 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.364 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.364 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.364 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.173394+0000", 2024-09-18T07:33:39.364 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.364 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.364 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.364 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.364 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.173395+0000", 2024-09-18T07:33:39.364 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9600000000000008e-07 2024-09-18T07:33:39.364 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.364 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.364 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.364 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.173460+0000", 2024-09-18T07:33:39.365 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5114e-05 2024-09-18T07:33:39.365 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.365 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.365 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.365 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.173471+0000", 2024-09-18T07:33:39.365 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0506000000000001e-05 2024-09-18T07:33:39.365 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.365 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.365 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.365 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.173476+0000", 2024-09-18T07:33:39.365 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.254e-06 2024-09-18T07:33:39.365 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.365 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.366 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.366 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.175237+0000", 2024-09-18T07:33:39.366 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001761475 2024-09-18T07:33:39.366 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.366 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.366 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.366 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.175349+0000", 2024-09-18T07:33:39.366 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011191799999999999 2024-09-18T07:33:39.366 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.366 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.366 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:33:39.366 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.175484+0000", 2024-09-18T07:33:39.367 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000134864 2024-09-18T07:33:39.367 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.367 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.367 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:39.367 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.176926+0000", 2024-09-18T07:33:39.367 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014422390000000001 2024-09-18T07:33:39.367 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.367 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.367 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.367 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.179509+0000", 2024-09-18T07:33:39.367 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025827990000000002 2024-09-18T07:33:39.367 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.367 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.368 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:39.368 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.317964+0000", 2024-09-18T07:33:39.368 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13845500499999999 2024-09-18T07:33:39.368 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.368 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.368 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.368 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.318020+0000", 2024-09-18T07:33:39.368 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5599000000000001e-05 2024-09-18T07:33:39.368 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.368 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.368 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.368 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.318056+0000", 2024-09-18T07:33:39.368 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6032000000000003e-05 2024-09-18T07:33:39.369 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.369 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.369 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.369 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.369 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.369 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:76877 7.6 e116/105 7:6f53810b:::benchmark_data_smithi137_74180_object76876:head v 116'4848, mlcod=116'4846)", 2024-09-18T07:33:39.369 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.430906+0000", 2024-09-18T07:33:39.369 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 20.82257311, 2024-09-18T07:33:39.369 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.117211378, 2024-09-18T07:33:39.369 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.369 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.369 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.369 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.370 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.370 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.370 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.430906+0000", 2024-09-18T07:33:39.370 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.370 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.370 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.370 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.370 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.430906+0000", 2024-09-18T07:33:39.370 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.370 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.370 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.370 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.370 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.430908+0000", 2024-09-18T07:33:39.371 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.08e-06 2024-09-18T07:33:39.371 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.371 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.371 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.371 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.430974+0000", 2024-09-18T07:33:39.371 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6369000000000001e-05 2024-09-18T07:33:39.371 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.371 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.371 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.371 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.430981+0000", 2024-09-18T07:33:39.371 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8689999999999998e-06 2024-09-18T07:33:39.371 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.372 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.372 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.372 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.430985+0000", 2024-09-18T07:33:39.372 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1760000000000003e-06 2024-09-18T07:33:39.372 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.372 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.372 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.372 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.431024+0000", 2024-09-18T07:33:39.372 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9317999999999999e-05 2024-09-18T07:33:39.372 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.372 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.372 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.372 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.431037+0000", 2024-09-18T07:33:39.372 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2435e-05 2024-09-18T07:33:39.373 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.373 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.373 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.373 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.548065+0000", 2024-09-18T07:33:39.373 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.11702828799999999 2024-09-18T07:33:39.373 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.373 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.373 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.373 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.548118+0000", 2024-09-18T07:33:39.373 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2843000000000002e-05 2024-09-18T07:33:39.373 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.373 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.373 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.374 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.374 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.374 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:76884 7.e e116/81 7:7e610b0a:::benchmark_data_smithi137_74180_object76883:head v 116'4796, mlcod=116'4790)", 2024-09-18T07:33:39.374 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.431901+0000", 2024-09-18T07:33:39.374 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 20.821578938999998, 2024-09-18T07:33:39.374 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.118586064, 2024-09-18T07:33:39.374 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.374 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.374 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.374 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.374 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.374 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.374 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.431901+0000", 2024-09-18T07:33:39.374 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.375 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.375 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.375 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.375 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.431901+0000", 2024-09-18T07:33:39.375 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.375 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.375 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.375 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.375 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.431901+0000", 2024-09-18T07:33:39.375 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2099999999999995e-07 2024-09-18T07:33:39.375 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.376 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.376 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.376 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.431966+0000", 2024-09-18T07:33:39.376 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4169000000000002e-05 2024-09-18T07:33:39.376 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.376 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.376 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.376 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.431972+0000", 2024-09-18T07:33:39.376 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1399999999999997e-06 2024-09-18T07:33:39.376 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.376 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.377 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.377 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.431975+0000", 2024-09-18T07:33:39.377 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0649999999999999e-06 2024-09-18T07:33:39.377 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.377 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.377 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.377 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.432006+0000", 2024-09-18T07:33:39.377 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0889000000000002e-05 2024-09-18T07:33:39.377 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.377 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.377 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.377 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.432018+0000", 2024-09-18T07:33:39.377 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2265e-05 2024-09-18T07:33:39.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.378 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.378 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.378 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.550379+0000", 2024-09-18T07:33:39.378 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.11836073900000001 2024-09-18T07:33:39.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.378 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.378 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.378 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.550487+0000", 2024-09-18T07:33:39.378 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000107976 2024-09-18T07:33:39.378 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.378 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.378 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.379 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.379 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:76879 7.6 e116/105 7:6fa0a2ea:::benchmark_data_smithi137_74180_object76878:head v 116'4849, mlcod=116'4846)", 2024-09-18T07:33:39.379 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.435035+0000", 2024-09-18T07:33:39.379 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 20.818444875000001, 2024-09-18T07:33:39.379 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.115484169, 2024-09-18T07:33:39.379 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:39.379 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:39.379 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:39.379 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:39.379 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.379 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:39.379 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.435035+0000", 2024-09-18T07:33:39.379 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.380 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.380 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.380 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:39.380 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.435035+0000", 2024-09-18T07:33:39.380 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:39.380 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.380 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.380 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:39.380 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.435035+0000", 2024-09-18T07:33:39.380 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3200000000000004e-07 2024-09-18T07:33:39.380 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.381 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.381 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:39.381 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.435146+0000", 2024-09-18T07:33:39.381 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000110714 2024-09-18T07:33:39.381 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.381 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.381 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:39.381 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.543253+0000", 2024-09-18T07:33:39.381 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10810718 2024-09-18T07:33:39.381 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.381 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.381 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:39.381 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.543280+0000", 2024-09-18T07:33:39.382 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6882000000000001e-05 2024-09-18T07:33:39.382 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.382 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.382 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:39.382 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.543600+0000", 2024-09-18T07:33:39.382 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000319365 2024-09-18T07:33:39.382 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.382 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.382 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:39.382 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.543760+0000", 2024-09-18T07:33:39.382 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016031500000000001 2024-09-18T07:33:39.382 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.382 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.383 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:39.383 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.550490+0000", 2024-09-18T07:33:39.383 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0067298039999999998 2024-09-18T07:33:39.383 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:39.383 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:39.383 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:39.383 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.550519+0000", 2024-09-18T07:33:39.383 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9077e-05 2024-09-18T07:33:39.383 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.383 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.383 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.383 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:39.383 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:39.384 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:33:39.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:39.385 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:39.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:39.486 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:39.509 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:39.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:39.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:39.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:39.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:39.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:39.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:39.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:39.788 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 84 16 98354 98338 73.1501 64.125 0.00537825 0.0136648 2024-09-18T07:33:39.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:39.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:39.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:39.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:39.933 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:39.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:40.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:40.083 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:40.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:40.185 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:40.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-18T07:33:40.235 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:40.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:40.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:40.260 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T07:33:40.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:40.338 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:40.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:40.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:40.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:40.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:40 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:33:40.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:40 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:33:40.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:40 smithi137 ceph-mon[39175]: osdmap e124: 8 total, 8 up, 7 in 2024-09-18T07:33:40.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:40 smithi137 ceph-mon[39175]: pgmap v121: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 13 B/s, 1 objects/s recovering 2024-09-18T07:33:40.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:40 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:33:40.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:40 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:33:40.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:40 smithi137 ceph-mon[29408]: osdmap e124: 8 total, 8 up, 7 in 2024-09-18T07:33:40.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:40 smithi137 ceph-mon[29408]: pgmap v121: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 13 B/s, 1 objects/s recovering 2024-09-18T07:33:40.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:40.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:40.548 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:40.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:40.564 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:33:40.649 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:40.674 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:40.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:40.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:40.686 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:40 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:33:40.687 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:40 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:33:40.687 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:40 smithi203 ceph-mon[32237]: osdmap e124: 8 total, 8 up, 7 in 2024-09-18T07:33:40.687 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:40 smithi203 ceph-mon[32237]: pgmap v121: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 13 B/s, 1 objects/s recovering 2024-09-18T07:33:40.777 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:40.788 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 85 16 99349 99333 73.0211 62.1875 0.0234573 0.0136894 2024-09-18T07:33:40.803 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:40.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:40.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:40.905 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:40.932 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:40.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:40.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:41.034 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:41.075 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:41.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:41.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:41.177 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:41.201 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:41.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:41.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:41.211 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:33:41.212 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:33:41.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:41.212 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:99682 7.d 7:b012e8eb:::benchmark_data_smithi137_74180_object99681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-18T07:33:41.212 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:41.098960+0000", 2024-09-18T07:33:41.212 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0034184189999999998, 2024-09-18T07:33:41.212 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0034401950000000001, 2024-09-18T07:33:41.212 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:41.212 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:41.213 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:33:41.213 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:41.213 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:41.214 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:41.214 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 99682 2024-09-18T07:33:41.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:41.214 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:41.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:41.214 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:41.214 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:41.098960+0000", 2024-09-18T07:33:41.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:41.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:41.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:41.214 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:41.214 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:41.098960+0000", 2024-09-18T07:33:41.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:41.215 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:41.215 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:41.215 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:41.215 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:41.098962+0000", 2024-09-18T07:33:41.215 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.04e-06 2024-09-18T07:33:41.215 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:41.215 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:41.215 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:41.215 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:41.099015+0000", 2024-09-18T07:33:41.215 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3665999999999997e-05 2024-09-18T07:33:41.215 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:41.215 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:41.215 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:41.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:41.099028+0000", 2024-09-18T07:33:41.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3071000000000001e-05 2024-09-18T07:33:41.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:41.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:41.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:41.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:41.099037+0000", 2024-09-18T07:33:41.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5730000000000008e-06 2024-09-18T07:33:41.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:41.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:41.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:41.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:41.099102+0000", 2024-09-18T07:33:41.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5160999999999995e-05 2024-09-18T07:33:41.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:41.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:41.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:41.217 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:41.099193+0000", 2024-09-18T07:33:41.217 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1057999999999995e-05 2024-09-18T07:33:41.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:41.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:41.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:33:41.217 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:41.099301+0000", 2024-09-18T07:33:41.217 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000107803 2024-09-18T07:33:41.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:41.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:41.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:41.218 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:41.101607+0000", 2024-09-18T07:33:41.218 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023054429999999999 2024-09-18T07:33:41.218 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:41.218 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:41.218 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:41.218 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:41.218 INFO:teuthology.orchestra.run.smithi203.stdout: ], 2024-09-18T07:33:41.218 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 1 2024-09-18T07:33:41.218 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:33:41.295 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:33:41.295 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:33:41.303 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:41.325 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:41.327 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:41.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:41.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:41.337 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:33:41.338 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:33:41.373 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:33:41.374 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:33:41.374 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:41 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:33:41.374 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:41 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4240025913' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:41.376 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:41 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:33:41.376 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:41 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4240025913' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:41.410 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:33:41.410 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:33:41.429 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:41.445 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:33:41.445 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:33:41.455 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:41.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:41.479 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:33:41.479 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:33:41.516 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:33:41.516 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:33:41.548 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:33:41.548 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:33:41.557 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:41.580 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:41.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:41.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:41.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:41.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:41 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:33:41.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:41 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4240025913' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:41.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:41.731 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:41.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:41.789 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 86 16 100361 100345 72.9074 63.25 0.0165551 0.0137109 2024-09-18T07:33:41.832 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:41.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:33:41.858 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:41.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:41.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:41.960 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:41.986 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:41.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:41.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:42.088 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:42.125 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T07:33:42.134 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:42.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:42.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:42.235 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:42.259 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:42.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:42.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:42.361 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:42.389 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:42.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:42.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:42.439 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:33:42.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:42 smithi203 ceph-mon[32237]: pgmap v122: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 831 op/s 2024-09-18T07:33:42.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:42 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:33:42.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:42 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:33:42.491 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:42.516 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:42.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:42.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:42.618 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:42.642 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:42.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:42.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:42 smithi137 ceph-mon[39175]: pgmap v122: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 831 op/s 2024-09-18T07:33:42.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:42 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:33:42.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:42 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:33:42.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:42 smithi137 ceph-mon[29408]: pgmap v122: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 831 op/s 2024-09-18T07:33:42.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:42 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:33:42.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:42 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:33:42.744 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:42.770 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:42.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:42.789 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 87 16 101367 101351 72.7921 62.875 0.00950689 0.0137309 2024-09-18T07:33:42.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:42.925 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:42.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:42.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:42.948 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:42.959 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:33:42.959 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:33:43.027 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:43.054 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:43.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:43.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:43.156 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:43.197 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:43.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:43.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:43.299 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:43.326 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:43.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:43.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:43.338 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.6 to 1 (10000) 2024-09-18T07:33:43.428 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:43.454 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:43.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:43.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:43.556 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:43.602 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:43.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:43.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:43.630 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:43 smithi203 ceph-mon[32237]: osdmap e125: 8 total, 8 up, 7 in 2024-09-18T07:33:43.630 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:43 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4073966233' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:33:43.631 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:43 smithi203 ceph-mon[32237]: from='client.24886 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:33:43.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:43 smithi137 ceph-mon[29408]: osdmap e125: 8 total, 8 up, 7 in 2024-09-18T07:33:43.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:43 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4073966233' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:33:43.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:43 smithi137 ceph-mon[29408]: from='client.24886 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:33:43.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:43 smithi137 ceph-mon[39175]: osdmap e125: 8 total, 8 up, 7 in 2024-09-18T07:33:43.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:43 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4073966233' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:33:43.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:43 smithi137 ceph-mon[39175]: from='client.24886 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:33:43.673 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T07:33:43.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:43.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:43.750 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:43.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:43.789 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 88 16 102378 102362 72.6829 63.1875 0.0190777 0.0137533 2024-09-18T07:33:43.850 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:43.894 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:43.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:43.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:43.996 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:44.013 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:33:44.029 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:44.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:44.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:44.132 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:44.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:44.162 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:44.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:44.204 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-18T07:33:44.204 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:33:44.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:44.321 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:44.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:44.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:44.362 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:33:44.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:44 smithi137 ceph-mon[29408]: from='client.24886 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:33:44.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:44 smithi137 ceph-mon[29408]: osdmap e126: 8 total, 8 up, 7 in 2024-09-18T07:33:44.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:44 smithi137 ceph-mon[29408]: pgmap v125: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 52 MiB/s wr, 830 op/s 2024-09-18T07:33:44.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:44 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4280044981' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:44.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:44 smithi137 ceph-mon[39175]: from='client.24886 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:33:44.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:44 smithi137 ceph-mon[39175]: osdmap e126: 8 total, 8 up, 7 in 2024-09-18T07:33:44.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:44 smithi137 ceph-mon[39175]: pgmap v125: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 52 MiB/s wr, 830 op/s 2024-09-18T07:33:44.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4280044981' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:44.423 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:44.450 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:44.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:44.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:44.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:44 smithi203 ceph-mon[32237]: from='client.24886 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:33:44.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:44 smithi203 ceph-mon[32237]: osdmap e126: 8 total, 8 up, 7 in 2024-09-18T07:33:44.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:44 smithi203 ceph-mon[32237]: pgmap v125: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 52 MiB/s wr, 830 op/s 2024-09-18T07:33:44.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:44 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4280044981' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:44.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:44.593 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:44.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:44.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:44.628 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:33:44.629 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:33:44.695 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:44.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:44.734 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:44.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:44.789 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 89 16 103418 103402 72.5965 65 0.0109459 0.0137695 2024-09-18T07:33:44.835 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:44.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:33:44.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:44.864 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:44.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:44.965 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:45.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:45.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:45.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:45.113 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:45.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:45.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:45.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:45.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:45.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:45.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:45.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T07:33:45.421 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:45.447 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:45.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:45.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:45.549 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:45.572 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:45.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:45.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:45.620 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:45.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:45 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:33:45.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:45 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4280044981' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:33:45.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:45 smithi137 ceph-mon[29408]: osdmap e127: 8 total, 8 up, 7 in 2024-09-18T07:33:45.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:45 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:33:45.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:45 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4280044981' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:33:45.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:45 smithi137 ceph-mon[39175]: osdmap e127: 8 total, 8 up, 7 in 2024-09-18T07:33:45.674 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:45.677 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:45.681 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:33:45.701 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:45.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:45.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:45.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:45 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:33:45.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:45 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4280044981' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:33:45.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:45 smithi203 ceph-mon[32237]: osdmap e127: 8 total, 8 up, 7 in 2024-09-18T07:33:45.789 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Total time run: 90.0069 2024-09-18T07:33:45.789 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Total writes made: 104413 2024-09-18T07:33:45.790 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Write size: 65536 2024-09-18T07:33:45.790 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Object size: 65536 2024-09-18T07:33:45.790 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Bandwidth (MB/sec): 72.5035 2024-09-18T07:33:45.790 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Stddev Bandwidth: 34.9471 2024-09-18T07:33:45.790 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Max bandwidth (MB/sec): 198.25 2024-09-18T07:33:45.790 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Min bandwidth (MB/sec): 0 2024-09-18T07:33:45.790 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Average IOPS: 1160 2024-09-18T07:33:45.790 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Stddev IOPS: 559.161 2024-09-18T07:33:45.790 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Max IOPS: 3172 2024-09-18T07:33:45.790 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Min IOPS: 0 2024-09-18T07:33:45.790 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Average Latency(s): 0.0137884 2024-09-18T07:33:45.790 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Stddev Latency(s): 0.036729 2024-09-18T07:33:45.790 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Max latency(s): 2.1848 2024-09-18T07:33:45.790 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Min latency(s): 0.000980073 2024-09-18T07:33:45.795 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Cleaning up (deleting benchmark objects) 2024-09-18T07:33:45.803 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:45.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:45.845 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:45.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:45.947 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:45.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:45.973 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:45.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:46.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:46.109 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:46.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:46.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:46.210 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:46.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:46.250 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:46.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:46.351 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:46.377 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:46.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:46.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:46 smithi203 ceph-mon[32237]: pgmap v127: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 52 MiB/s wr, 830 op/s 2024-09-18T07:33:46.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:46.517 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:46.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:46.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:46.619 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:46.643 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:46.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:46.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:46 smithi137 ceph-mon[29408]: pgmap v127: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 52 MiB/s wr, 830 op/s 2024-09-18T07:33:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:46 smithi137 ceph-mon[39175]: pgmap v127: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 52 MiB/s wr, 830 op/s 2024-09-18T07:33:46.745 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:46.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:46.774 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:46.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:46.875 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:46.918 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:46.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:46.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:46.979 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:33:46.979 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:33:47.021 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:47.044 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:47.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:47.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:47.146 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:47.169 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:47.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:47.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:47.266 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:33:47.268 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:33:47.270 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:47.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:47.292 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:47.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:47.377 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:33:46 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:33:47.393 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:47.424 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:47.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:47.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:47.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:33:47.526 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:47.541 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:33:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:33:47.541 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:47 smithi137 ceph-mon[29408]: from='client.24898 v1:172.21.15.137:0/1804193995' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:33:47.541 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:47 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:47.541 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:47 smithi137 ceph-mon[39175]: from='client.24898 v1:172.21.15.137:0/1804193995' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:33:47.541 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:47 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:47.552 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:47.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:47.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:47.654 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:47.677 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:47.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:47.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:47.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:47 smithi203 ceph-mon[32237]: from='client.24898 v1:172.21.15.137:0/1804193995' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:33:47.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:47 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:47.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:47.801 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:47.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:47.902 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:47.927 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:47.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:47.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:48.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:48.054 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:48.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:48.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:48.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:48.186 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:48.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:48.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:48.288 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:48.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:48.314 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:48.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:48.415 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:48.438 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:48.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:48.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:48.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:48.564 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:48.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:48.665 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:48.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:48 smithi137 ceph-mon[29408]: pgmap v128: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.8 KiB/s rd, 81 MiB/s wr, 1.42k op/s 2024-09-18T07:33:48.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:48 smithi137 ceph-mon[39175]: pgmap v128: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.8 KiB/s rd, 81 MiB/s wr, 1.42k op/s 2024-09-18T07:33:48.688 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:48.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:48.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:48.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:48 smithi203 ceph-mon[32237]: pgmap v128: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.8 KiB/s rd, 81 MiB/s wr, 1.42k op/s 2024-09-18T07:33:48.790 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:48.816 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:48.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:48.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:48.918 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:48.943 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:48.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:48.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:49.045 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:49.070 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:49.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:49.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:49.166 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-09-18T07:33:45.443785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6558194203,"num_objects":100260,"num_object_clones":0,"num_object_copies":300780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100260,"num_whiteouts":0,"num_read":901,"num_read_kb":760,"num_write":100610,"num_write_kb":6405577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9794,"num_bytes_recovered":636423665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79232,"ondisk_log_size":79232,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19702268,"kb_used_data":19111944,"kb_used_omap":10,"kb_used_meta":590197,"kb_avail":636513796,"statfs":{"total":671965249536,"available":651790127104,"internally_reserved":0,"allocated":19570630656,"data_stored":19567861483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11162,"internal_metadata":604361830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,7,5,10,15,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":326369280,"num_objects":4980,"num_object_clones":0,"num_object_copies":14940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4980,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4980,"num_write_kb":318720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001661"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":146,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.319236+0000","last_change":"2024-09-18T07:33:31.183001+0000","last_active":"2024-09-18T07:33:39.319236+0000","last_peered":"2024-09-18T07:33:39.319236+0000","last_clean":"2024-09-18T07:33:39.319236+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:33:39.319236+0000","last_undegraded":"2024-09-18T07:33:39.319236+0000","last_fullsized":"2024-09-18T07:33:39.319236+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:31.182913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:31.182913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:44.795464+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":176,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.302422+0000","last_change":"2024-09-18T07:33:29.685825+0000","last_active":"2024-09-18T07:33:39.302422+0000","last_peered":"2024-09-18T07:33:39.302422+0000","last_clean":"2024-09-18T07:33:39.302422+0000","last_became_active":"2024-09-18T07:32:26.785727+0000","last_became_peered":"2024-09-18T07:32:26.785727+0000","last_unstale":"2024-09-18T07:33:39.302422+0000","last_undegraded":"2024-09-18T07:33:39.302422+0000","last_fullsized":"2024-09-18T07:33:39.302422+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:33:29.685767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.685767+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:33.086673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":146,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.308400+0000","last_change":"2024-09-18T07:32:31.030879+0000","last_active":"2024-09-18T07:33:39.308400+0000","last_peered":"2024-09-18T07:33:39.308400+0000","last_clean":"2024-09-18T07:33:39.308400+0000","last_became_active":"2024-09-18T07:32:30.831949+0000","last_became_peered":"2024-09-18T07:32:30.831949+0000","last_unstale":"2024-09-18T07:33:39.308400+0000","last_undegraded":"2024-09-18T07:33:39.308400+0000","last_fullsized":"2024-09-18T07:33:39.308400+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:31.030835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:31.030835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:55.282052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":135,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.302970+0000","last_change":"2024-09-18T07:32:03.953881+0000","last_active":"2024-09-18T07:33:39.302970+0000","last_peered":"2024-09-18T07:33:39.302970+0000","last_clean":"2024-09-18T07:33:39.302970+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:33:39.302970+0000","last_undegraded":"2024-09-18T07:33:39.302970+0000","last_fullsized":"2024-09-18T07:33:39.302970+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.953827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.953827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":20,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.313775+0000","last_change":"2024-09-18T07:33:27.004713+0000","last_active":"2024-09-18T07:33:39.313775+0000","last_peered":"2024-09-18T07:33:39.313775+0000","last_clean":"2024-09-18T07:33:39.313775+0000","last_became_active":"2024-09-18T07:33:27.004385+0000","last_became_peered":"2024-09-18T07:33:27.004385+0000","last_unstale":"2024-09-18T07:33:39.313775+0000","last_undegraded":"2024-09-18T07:33:39.313775+0000","last_fullsized":"2024-09-18T07:33:39.313775+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:39.137803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":146,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.318729+0000","last_change":"2024-09-18T07:33:27.006545+0000","last_active":"2024-09-18T07:33:39.318729+0000","last_peered":"2024-09-18T07:33:39.318729+0000","last_clean":"2024-09-18T07:33:39.318729+0000","last_became_active":"2024-09-18T07:30:56.286040+0000","last_became_peered":"2024-09-18T07:30:56.286040+0000","last_unstale":"2024-09-18T07:33:39.318729+0000","last_undegraded":"2024-09-18T07:33:39.318729+0000","last_fullsized":"2024-09-18T07:33:39.318729+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.006488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.006488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.614689+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":167,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315461+0000","last_change":"2024-09-18T07:33:22.032321+0000","last_active":"2024-09-18T07:33:39.315461+0000","last_peered":"2024-09-18T07:33:39.315461+0000","last_clean":"2024-09-18T07:33:39.315461+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:33:39.315461+0000","last_undegraded":"2024-09-18T07:33:39.315461+0000","last_fullsized":"2024-09-18T07:33:39.315461+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:22.032110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.032110+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.263345+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":131,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.302371+0000","last_change":"2024-09-18T07:31:00.331498+0000","last_active":"2024-09-18T07:33:39.302371+0000","last_peered":"2024-09-18T07:33:39.302371+0000","last_clean":"2024-09-18T07:33:39.302371+0000","last_became_active":"2024-09-18T07:31:00.331224+0000","last_became_peered":"2024-09-18T07:31:00.331224+0000","last_unstale":"2024-09-18T07:33:39.302371+0000","last_undegraded":"2024-09-18T07:33:39.302371+0000","last_fullsized":"2024-09-18T07:33:39.302371+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":146,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315240+0000","last_change":"2024-09-18T07:33:21.338716+0000","last_active":"2024-09-18T07:33:39.315240+0000","last_peered":"2024-09-18T07:33:39.315240+0000","last_clean":"2024-09-18T07:33:39.315240+0000","last_became_active":"2024-09-18T07:30:56.285882+0000","last_became_peered":"2024-09-18T07:30:56.285882+0000","last_unstale":"2024-09-18T07:33:39.315240+0000","last_undegraded":"2024-09-18T07:33:39.315240+0000","last_fullsized":"2024-09-18T07:33:39.315240+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.338669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.338669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:26.027193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":159,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.303232+0000","last_change":"2024-09-18T07:32:02.002919+0000","last_active":"2024-09-18T07:33:39.303232+0000","last_peered":"2024-09-18T07:33:39.303232+0000","last_clean":"2024-09-18T07:33:39.303232+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:33:39.303232+0000","last_undegraded":"2024-09-18T07:33:39.303232+0000","last_fullsized":"2024-09-18T07:33:39.303232+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.002811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.002811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":137,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314213+0000","last_change":"2024-09-18T07:32:30.842423+0000","last_active":"2024-09-18T07:33:39.314213+0000","last_peered":"2024-09-18T07:33:39.314213+0000","last_clean":"2024-09-18T07:33:39.314213+0000","last_became_active":"2024-09-18T07:32:30.841963+0000","last_became_peered":"2024-09-18T07:32:30.841963+0000","last_unstale":"2024-09-18T07:33:39.314213+0000","last_undegraded":"2024-09-18T07:33:39.314213+0000","last_fullsized":"2024-09-18T07:33:39.314213+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.385503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":133,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.313448+0000","last_change":"2024-09-18T07:32:30.840094+0000","last_active":"2024-09-18T07:33:39.313448+0000","last_peered":"2024-09-18T07:33:39.313448+0000","last_clean":"2024-09-18T07:33:39.313448+0000","last_became_active":"2024-09-18T07:32:30.839724+0000","last_became_peered":"2024-09-18T07:32:30.839724+0000","last_unstale":"2024-09-18T07:33:39.313448+0000","last_undegraded":"2024-09-18T07:33:39.313448+0000","last_fullsized":"2024-09-18T07:33:39.313448+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":138,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314369+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:33:39.314369+0000","last_peered":"2024-09-18T07:33:39.314369+0000","last_clean":"2024-09-18T07:33:39.314369+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:33:39.314369+0000","last_undegraded":"2024-09-18T07:33:39.314369+0000","last_fullsized":"2024-09-18T07:33:39.314369+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:15.347368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":145,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.303919+0000","last_change":"2024-09-18T07:30:58.317041+0000","last_active":"2024-09-18T07:33:39.303919+0000","last_peered":"2024-09-18T07:33:39.303919+0000","last_clean":"2024-09-18T07:33:39.303919+0000","last_became_active":"2024-09-18T07:30:58.316479+0000","last_became_peered":"2024-09-18T07:30:58.316479+0000","last_unstale":"2024-09-18T07:33:39.303919+0000","last_undegraded":"2024-09-18T07:33:39.303919+0000","last_fullsized":"2024-09-18T07:33:39.303919+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":73,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.302874+0000","last_change":"2024-09-18T07:32:30.834493+0000","last_active":"2024-09-18T07:33:39.302874+0000","last_peered":"2024-09-18T07:33:39.302874+0000","last_clean":"2024-09-18T07:33:39.302874+0000","last_became_active":"2024-09-18T07:32:30.834192+0000","last_became_peered":"2024-09-18T07:32:30.834192+0000","last_unstale":"2024-09-18T07:33:39.302874+0000","last_undegraded":"2024-09-18T07:33:39.302874+0000","last_fullsized":"2024-09-18T07:33:39.302874+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314852+0000","last_change":"2024-09-18T07:32:30.107805+0000","last_active":"2024-09-18T07:33:39.314852+0000","last_peered":"2024-09-18T07:33:39.314852+0000","last_clean":"2024-09-18T07:33:39.314852+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:33:39.314852+0000","last_undegraded":"2024-09-18T07:33:39.314852+0000","last_fullsized":"2024-09-18T07:33:39.314852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:30.107745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:30.107745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":135,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.303522+0000","last_change":"2024-09-18T07:32:06.976159+0000","last_active":"2024-09-18T07:33:39.303522+0000","last_peered":"2024-09-18T07:33:39.303522+0000","last_clean":"2024-09-18T07:33:39.303522+0000","last_became_active":"2024-09-18T07:31:02.490014+0000","last_became_peered":"2024-09-18T07:31:02.490014+0000","last_unstale":"2024-09-18T07:33:39.303522+0000","last_undegraded":"2024-09-18T07:33:39.303522+0000","last_fullsized":"2024-09-18T07:33:39.303522+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:06.976113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:06.976113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":145,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.316876+0000","last_change":"2024-09-18T07:32:30.840267+0000","last_active":"2024-09-18T07:33:39.316876+0000","last_peered":"2024-09-18T07:33:39.316876+0000","last_clean":"2024-09-18T07:33:39.316876+0000","last_became_active":"2024-09-18T07:32:30.839952+0000","last_became_peered":"2024-09-18T07:32:30.839952+0000","last_unstale":"2024-09-18T07:33:39.316876+0000","last_undegraded":"2024-09-18T07:33:39.316876+0000","last_fullsized":"2024-09-18T07:33:39.316876+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":168,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.309780+0000","last_change":"2024-09-18T07:32:30.452884+0000","last_active":"2024-09-18T07:33:39.309780+0000","last_peered":"2024-09-18T07:33:39.309780+0000","last_clean":"2024-09-18T07:33:39.309780+0000","last_became_active":"2024-09-18T07:30:58.316266+0000","last_became_peered":"2024-09-18T07:30:58.316266+0000","last_unstale":"2024-09-18T07:33:39.309780+0000","last_undegraded":"2024-09-18T07:33:39.309780+0000","last_fullsized":"2024-09-18T07:33:39.309780+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:32:30.452816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:30.452816+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:43.553962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":139,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.309794+0000","last_change":"2024-09-18T07:33:30.955441+0000","last_active":"2024-09-18T07:33:39.309794+0000","last_peered":"2024-09-18T07:33:39.309794+0000","last_clean":"2024-09-18T07:33:39.309794+0000","last_became_active":"2024-09-18T07:31:00.337474+0000","last_became_peered":"2024-09-18T07:31:00.337474+0000","last_unstale":"2024-09-18T07:33:39.309794+0000","last_undegraded":"2024-09-18T07:33:39.309794+0000","last_fullsized":"2024-09-18T07:33:39.309794+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.955213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.955213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.470164+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.321270+0000","last_change":"2024-09-18T07:33:26.004195+0000","last_active":"2024-09-18T07:33:39.321270+0000","last_peered":"2024-09-18T07:33:39.321270+0000","last_clean":"2024-09-18T07:33:39.321270+0000","last_became_active":"2024-09-18T07:31:02.492348+0000","last_became_peered":"2024-09-18T07:31:02.492348+0000","last_unstale":"2024-09-18T07:33:39.321270+0000","last_undegraded":"2024-09-18T07:33:39.321270+0000","last_fullsized":"2024-09-18T07:33:39.321270+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:26.003425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:26.003425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:38.529586+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":139,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.303350+0000","last_change":"2024-09-18T07:30:56.289489+0000","last_active":"2024-09-18T07:33:39.303350+0000","last_peered":"2024-09-18T07:33:39.303350+0000","last_clean":"2024-09-18T07:33:39.303350+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:33:39.303350+0000","last_undegraded":"2024-09-18T07:33:39.303350+0000","last_fullsized":"2024-09-18T07:33:39.303350+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":157,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315312+0000","last_change":"2024-09-18T07:30:58.319115+0000","last_active":"2024-09-18T07:33:39.315312+0000","last_peered":"2024-09-18T07:33:39.315312+0000","last_clean":"2024-09-18T07:33:39.315312+0000","last_became_active":"2024-09-18T07:30:58.316380+0000","last_became_peered":"2024-09-18T07:30:58.316380+0000","last_unstale":"2024-09-18T07:33:39.315312+0000","last_undegraded":"2024-09-18T07:33:39.315312+0000","last_fullsized":"2024-09-18T07:33:39.315312+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.013089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":131,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.303706+0000","last_change":"2024-09-18T07:31:00.334313+0000","last_active":"2024-09-18T07:33:39.303706+0000","last_peered":"2024-09-18T07:33:39.303706+0000","last_clean":"2024-09-18T07:33:39.303706+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:33:39.303706+0000","last_undegraded":"2024-09-18T07:33:39.303706+0000","last_fullsized":"2024-09-18T07:33:39.303706+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.867220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":97,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314170+0000","last_change":"2024-09-18T07:33:33.565391+0000","last_active":"2024-09-18T07:33:39.314170+0000","last_peered":"2024-09-18T07:33:39.314170+0000","last_clean":"2024-09-18T07:33:39.314170+0000","last_became_active":"2024-09-18T07:32:30.840386+0000","last_became_peered":"2024-09-18T07:32:30.840386+0000","last_unstale":"2024-09-18T07:33:39.314170+0000","last_undegraded":"2024-09-18T07:33:39.314170+0000","last_fullsized":"2024-09-18T07:33:39.314170+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:33.565334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.565334+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:02.676072+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":156,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.308534+0000","last_change":"2024-09-18T07:33:32.976203+0000","last_active":"2024-09-18T07:33:39.308534+0000","last_peered":"2024-09-18T07:33:39.308534+0000","last_clean":"2024-09-18T07:33:39.308534+0000","last_became_active":"2024-09-18T07:30:56.283705+0000","last_became_peered":"2024-09-18T07:30:56.283705+0000","last_unstale":"2024-09-18T07:33:39.308534+0000","last_undegraded":"2024-09-18T07:33:39.308534+0000","last_fullsized":"2024-09-18T07:33:39.308534+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:32.976129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.976129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:54.706133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":176,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.320849+0000","last_change":"2024-09-18T07:33:36.118423+0000","last_active":"2024-09-18T07:33:39.320849+0000","last_peered":"2024-09-18T07:33:39.320849+0000","last_clean":"2024-09-18T07:33:39.320849+0000","last_became_active":"2024-09-18T07:32:30.860930+0000","last_became_peered":"2024-09-18T07:32:30.860930+0000","last_unstale":"2024-09-18T07:33:39.320849+0000","last_undegraded":"2024-09-18T07:33:39.320849+0000","last_fullsized":"2024-09-18T07:33:39.320849+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:33:36.118340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.118340+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:36.957408+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"94'9","reported_seq":186,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314968+0000","last_change":"2024-09-18T07:32:30.853596+0000","last_active":"2024-09-18T07:33:39.314968+0000","last_peered":"2024-09-18T07:33:39.314968+0000","last_clean":"2024-09-18T07:33:39.314968+0000","last_became_active":"2024-09-18T07:32:30.839828+0000","last_became_peered":"2024-09-18T07:32:30.839828+0000","last_unstale":"2024-09-18T07:33:39.314968+0000","last_undegraded":"2024-09-18T07:33:39.314968+0000","last_fullsized":"2024-09-18T07:33:39.314968+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:58.766416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"124'6226","reported_seq":10276,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:41.418499+0000","last_change":"2024-09-18T07:33:32.018562+0000","last_active":"2024-09-18T07:33:41.418499+0000","last_peered":"2024-09-18T07:33:41.418499+0000","last_clean":"2024-09-18T07:33:41.418499+0000","last_became_active":"2024-09-18T07:32:54.163740+0000","last_became_peered":"2024-09-18T07:32:54.163740+0000","last_unstale":"2024-09-18T07:33:41.418499+0000","last_undegraded":"2024-09-18T07:33:41.418499+0000","last_fullsized":"2024-09-18T07:33:41.418499+0000","mapping_epoch":102,"log_start":"87'600","ondisk_log_start":"87'600","created":81,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'5666","last_scrub_stamp":"2024-09-18T07:33:32.018370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.018370+0000","objects_scrubbed":5641,"log_size":5626,"log_dups_size":0,"ondisk_log_size":5626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:57.391560+0000","scrub_duration":1289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408027136,"num_objects":6226,"num_object_clones":0,"num_object_copies":18678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6226,"num_write_kb":398464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":127074304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":160,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.310325+0000","last_change":"2024-09-18T07:30:58.312916+0000","last_active":"2024-09-18T07:33:39.310325+0000","last_peered":"2024-09-18T07:33:39.310325+0000","last_clean":"2024-09-18T07:33:39.310325+0000","last_became_active":"2024-09-18T07:30:58.312627+0000","last_became_peered":"2024-09-18T07:30:58.312627+0000","last_unstale":"2024-09-18T07:33:39.310325+0000","last_undegraded":"2024-09-18T07:33:39.310325+0000","last_fullsized":"2024-09-18T07:33:39.310325+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:15.056322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":153,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.316497+0000","last_change":"2024-09-18T07:32:30.838798+0000","last_active":"2024-09-18T07:33:39.316497+0000","last_peered":"2024-09-18T07:33:39.316497+0000","last_clean":"2024-09-18T07:33:39.316497+0000","last_became_active":"2024-09-18T07:32:30.838348+0000","last_became_peered":"2024-09-18T07:32:30.838348+0000","last_unstale":"2024-09-18T07:33:39.316497+0000","last_undegraded":"2024-09-18T07:33:39.316497+0000","last_fullsized":"2024-09-18T07:33:39.316497+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:56.565412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:56.565412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":181,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.303628+0000","last_change":"2024-09-18T07:33:33.706253+0000","last_active":"2024-09-18T07:33:39.303628+0000","last_peered":"2024-09-18T07:33:39.303628+0000","last_clean":"2024-09-18T07:33:39.303628+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:33:39.303628+0000","last_undegraded":"2024-09-18T07:33:39.303628+0000","last_fullsized":"2024-09-18T07:33:39.303628+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:33:33.706201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.706201+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.605129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314556+0000","last_change":"2024-09-18T07:31:02.351412+0000","last_active":"2024-09-18T07:33:39.314556+0000","last_peered":"2024-09-18T07:33:39.314556+0000","last_clean":"2024-09-18T07:33:39.314556+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:33:39.314556+0000","last_undegraded":"2024-09-18T07:33:39.314556+0000","last_fullsized":"2024-09-18T07:33:39.314556+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.333040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"124'6173","reported_seq":6269,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:41.448161+0000","last_change":"2024-09-18T07:32:08.065129+0000","last_active":"2024-09-18T07:33:41.448161+0000","last_peered":"2024-09-18T07:33:41.448161+0000","last_clean":"2024-09-18T07:33:41.448161+0000","last_became_active":"2024-09-18T07:32:08.064876+0000","last_became_peered":"2024-09-18T07:32:08.064876+0000","last_unstale":"2024-09-18T07:33:41.448161+0000","last_undegraded":"2024-09-18T07:33:41.448161+0000","last_fullsized":"2024-09-18T07:33:41.448161+0000","mapping_epoch":81,"log_start":"96'2000","ondisk_log_start":"96'2000","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4173,"log_dups_size":0,"ondisk_log_size":4173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404553728,"num_objects":6173,"num_object_clones":0,"num_object_copies":18519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6173,"num_write_kb":395072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":199,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.316281+0000","last_change":"2024-09-18T07:32:30.876643+0000","last_active":"2024-09-18T07:33:39.316281+0000","last_peered":"2024-09-18T07:33:39.316281+0000","last_clean":"2024-09-18T07:33:39.316281+0000","last_became_active":"2024-09-18T07:32:30.852501+0000","last_became_peered":"2024-09-18T07:32:30.852501+0000","last_unstale":"2024-09-18T07:33:39.316281+0000","last_undegraded":"2024-09-18T07:33:39.316281+0000","last_fullsized":"2024-09-18T07:33:39.316281+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:32:00.660377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:00.660377+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":153,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.321590+0000","last_change":"2024-09-18T07:32:30.858438+0000","last_active":"2024-09-18T07:33:39.321590+0000","last_peered":"2024-09-18T07:33:39.321590+0000","last_clean":"2024-09-18T07:33:39.321590+0000","last_became_active":"2024-09-18T07:32:30.858176+0000","last_became_peered":"2024-09-18T07:32:30.858176+0000","last_unstale":"2024-09-18T07:33:39.321590+0000","last_undegraded":"2024-09-18T07:33:39.321590+0000","last_fullsized":"2024-09-18T07:33:39.321590+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.665591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.665591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":138,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.302722+0000","last_change":"2024-09-18T07:32:30.851490+0000","last_active":"2024-09-18T07:33:39.302722+0000","last_peered":"2024-09-18T07:33:39.302722+0000","last_clean":"2024-09-18T07:33:39.302722+0000","last_became_active":"2024-09-18T07:32:30.851161+0000","last_became_peered":"2024-09-18T07:32:30.851161+0000","last_unstale":"2024-09-18T07:33:39.302722+0000","last_undegraded":"2024-09-18T07:33:39.302722+0000","last_fullsized":"2024-09-18T07:33:39.302722+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":127,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.303153+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:33:39.303153+0000","last_peered":"2024-09-18T07:33:39.303153+0000","last_clean":"2024-09-18T07:33:39.303153+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:33:39.303153+0000","last_undegraded":"2024-09-18T07:33:39.303153+0000","last_fullsized":"2024-09-18T07:33:39.303153+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"124'6242","reported_seq":6338,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:41.514809+0000","last_change":"2024-09-18T07:32:08.066116+0000","last_active":"2024-09-18T07:33:41.514809+0000","last_peered":"2024-09-18T07:33:41.514809+0000","last_clean":"2024-09-18T07:33:41.514809+0000","last_became_active":"2024-09-18T07:32:08.065844+0000","last_became_peered":"2024-09-18T07:32:08.065844+0000","last_unstale":"2024-09-18T07:33:41.514809+0000","last_undegraded":"2024-09-18T07:33:41.514809+0000","last_fullsized":"2024-09-18T07:33:41.514809+0000","mapping_epoch":81,"log_start":"96'2000","ondisk_log_start":"96'2000","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409075712,"num_objects":6242,"num_object_clones":0,"num_object_copies":18726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6242,"num_write_kb":399488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":156,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.302750+0000","last_change":"2024-09-18T07:30:58.309938+0000","last_active":"2024-09-18T07:33:39.302750+0000","last_peered":"2024-09-18T07:33:39.302750+0000","last_clean":"2024-09-18T07:33:39.302750+0000","last_became_active":"2024-09-18T07:30:58.309649+0000","last_became_peered":"2024-09-18T07:30:58.309649+0000","last_unstale":"2024-09-18T07:33:39.302750+0000","last_undegraded":"2024-09-18T07:33:39.302750+0000","last_fullsized":"2024-09-18T07:33:39.302750+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":147,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.321100+0000","last_change":"2024-09-18T07:33:24.213432+0000","last_active":"2024-09-18T07:33:39.321100+0000","last_peered":"2024-09-18T07:33:39.321100+0000","last_clean":"2024-09-18T07:33:39.321100+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:33:39.321100+0000","last_undegraded":"2024-09-18T07:33:39.321100+0000","last_fullsized":"2024-09-18T07:33:39.321100+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:24.213356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:24.213356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:52.949578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":172,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315667+0000","last_change":"2024-09-18T07:31:00.336059+0000","last_active":"2024-09-18T07:33:39.315667+0000","last_peered":"2024-09-18T07:33:39.315667+0000","last_clean":"2024-09-18T07:33:39.315667+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:33:39.315667+0000","last_undegraded":"2024-09-18T07:33:39.315667+0000","last_fullsized":"2024-09-18T07:33:39.315667+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:17.963381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":81,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.313915+0000","last_change":"2024-09-18T07:32:54.740291+0000","last_active":"2024-09-18T07:33:39.313915+0000","last_peered":"2024-09-18T07:33:39.313915+0000","last_clean":"2024-09-18T07:33:39.313915+0000","last_became_active":"2024-09-18T07:32:30.830346+0000","last_became_peered":"2024-09-18T07:32:30.830346+0000","last_unstale":"2024-09-18T07:33:39.313915+0000","last_undegraded":"2024-09-18T07:33:39.313915+0000","last_fullsized":"2024-09-18T07:33:39.313915+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:54.740249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:54.740249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.965276+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"124'6316","reported_seq":6413,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:41.402329+0000","last_change":"2024-09-18T07:32:08.062282+0000","last_active":"2024-09-18T07:33:41.402329+0000","last_peered":"2024-09-18T07:33:41.402329+0000","last_clean":"2024-09-18T07:33:41.402329+0000","last_became_active":"2024-09-18T07:32:08.061934+0000","last_became_peered":"2024-09-18T07:32:08.061934+0000","last_unstale":"2024-09-18T07:33:41.402329+0000","last_undegraded":"2024-09-18T07:33:41.402329+0000","last_fullsized":"2024-09-18T07:33:41.402329+0000","mapping_epoch":81,"log_start":"92'1900","ondisk_log_start":"92'1900","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4416,"log_dups_size":0,"ondisk_log_size":4416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413925376,"num_objects":6316,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6316,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":150,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.309319+0000","last_change":"2024-09-18T07:30:58.307470+0000","last_active":"2024-09-18T07:33:39.309319+0000","last_peered":"2024-09-18T07:33:39.309319+0000","last_clean":"2024-09-18T07:33:39.309319+0000","last_became_active":"2024-09-18T07:30:58.307035+0000","last_became_peered":"2024-09-18T07:30:58.307035+0000","last_unstale":"2024-09-18T07:33:39.309319+0000","last_undegraded":"2024-09-18T07:33:39.309319+0000","last_fullsized":"2024-09-18T07:33:39.309319+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:14.091632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":146,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315331+0000","last_change":"2024-09-18T07:33:29.471394+0000","last_active":"2024-09-18T07:33:39.315331+0000","last_peered":"2024-09-18T07:33:39.315331+0000","last_clean":"2024-09-18T07:33:39.315331+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:33:39.315331+0000","last_undegraded":"2024-09-18T07:33:39.315331+0000","last_fullsized":"2024-09-18T07:33:39.315331+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:29.471330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.471330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:58.117832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":172,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314033+0000","last_change":"2024-09-18T07:31:00.337241+0000","last_active":"2024-09-18T07:33:39.314033+0000","last_peered":"2024-09-18T07:33:39.314033+0000","last_clean":"2024-09-18T07:33:39.314033+0000","last_became_active":"2024-09-18T07:31:00.336905+0000","last_became_peered":"2024-09-18T07:31:00.336905+0000","last_unstale":"2024-09-18T07:33:39.314033+0000","last_undegraded":"2024-09-18T07:33:39.314033+0000","last_fullsized":"2024-09-18T07:33:39.314033+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.313570+0000","last_change":"2024-09-18T07:31:02.348538+0000","last_active":"2024-09-18T07:33:39.313570+0000","last_peered":"2024-09-18T07:33:39.313570+0000","last_clean":"2024-09-18T07:33:39.313570+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:33:39.313570+0000","last_undegraded":"2024-09-18T07:33:39.313570+0000","last_fullsized":"2024-09-18T07:33:39.313570+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.077245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"124'6205","reported_seq":6299,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:41.406144+0000","last_change":"2024-09-18T07:32:08.062584+0000","last_active":"2024-09-18T07:33:41.406144+0000","last_peered":"2024-09-18T07:33:41.406144+0000","last_clean":"2024-09-18T07:33:41.406144+0000","last_became_active":"2024-09-18T07:32:08.062129+0000","last_became_peered":"2024-09-18T07:32:08.062129+0000","last_unstale":"2024-09-18T07:33:41.406144+0000","last_undegraded":"2024-09-18T07:33:41.406144+0000","last_fullsized":"2024-09-18T07:33:41.406144+0000","mapping_epoch":81,"log_start":"88'956","ondisk_log_start":"88'956","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5249,"log_dups_size":0,"ondisk_log_size":5249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406650880,"num_objects":6205,"num_object_clones":0,"num_object_copies":18615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6205,"num_write_kb":397120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":195,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314183+0000","last_change":"2024-09-18T07:32:30.859920+0000","last_active":"2024-09-18T07:33:39.314183+0000","last_peered":"2024-09-18T07:33:39.314183+0000","last_clean":"2024-09-18T07:33:39.314183+0000","last_became_active":"2024-09-18T07:32:30.842195+0000","last_became_peered":"2024-09-18T07:32:30.842195+0000","last_unstale":"2024-09-18T07:33:39.314183+0000","last_undegraded":"2024-09-18T07:33:39.314183+0000","last_fullsized":"2024-09-18T07:33:39.314183+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:18.104967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":139,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.311976+0000","last_change":"2024-09-18T07:30:56.284440+0000","last_active":"2024-09-18T07:33:39.311976+0000","last_peered":"2024-09-18T07:33:39.311976+0000","last_clean":"2024-09-18T07:33:39.311976+0000","last_became_active":"2024-09-18T07:30:56.284120+0000","last_became_peered":"2024-09-18T07:30:56.284120+0000","last_unstale":"2024-09-18T07:33:39.311976+0000","last_undegraded":"2024-09-18T07:33:39.311976+0000","last_fullsized":"2024-09-18T07:33:39.311976+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:06.088370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":138,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.313336+0000","last_change":"2024-09-18T07:33:21.834910+0000","last_active":"2024-09-18T07:33:39.313336+0000","last_peered":"2024-09-18T07:33:39.313336+0000","last_clean":"2024-09-18T07:33:39.313336+0000","last_became_active":"2024-09-18T07:31:00.334776+0000","last_became_peered":"2024-09-18T07:31:00.334776+0000","last_unstale":"2024-09-18T07:33:39.313336+0000","last_undegraded":"2024-09-18T07:33:39.313336+0000","last_fullsized":"2024-09-18T07:33:39.313336+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:24.513163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":127,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.307928+0000","last_change":"2024-09-18T07:31:02.490534+0000","last_active":"2024-09-18T07:33:39.307928+0000","last_peered":"2024-09-18T07:33:39.307928+0000","last_clean":"2024-09-18T07:33:39.307928+0000","last_became_active":"2024-09-18T07:31:02.490186+0000","last_became_peered":"2024-09-18T07:31:02.490186+0000","last_unstale":"2024-09-18T07:33:39.307928+0000","last_undegraded":"2024-09-18T07:33:39.307928+0000","last_fullsized":"2024-09-18T07:33:39.307928+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:19.074713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"124'6089","reported_seq":6247,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:41.400314+0000","last_change":"2024-09-18T07:33:31.181076+0000","last_active":"2024-09-18T07:33:41.400314+0000","last_peered":"2024-09-18T07:33:41.400314+0000","last_clean":"2024-09-18T07:33:41.400314+0000","last_became_active":"2024-09-18T07:32:08.064184+0000","last_became_peered":"2024-09-18T07:32:08.064184+0000","last_unstale":"2024-09-18T07:33:41.400314+0000","last_undegraded":"2024-09-18T07:33:41.400314+0000","last_fullsized":"2024-09-18T07:33:41.400314+0000","mapping_epoch":81,"log_start":"87'500","ondisk_log_start":"87'500","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'5435","last_scrub_stamp":"2024-09-18T07:33:31.180958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:33:31.180958+0000","objects_scrubbed":5388,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.901559+0000","scrub_duration":1433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399048704,"num_objects":6089,"num_object_clones":0,"num_object_copies":18267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6089,"num_write_kb":389696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":217,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.311522+0000","last_change":"2024-09-18T07:30:58.317477+0000","last_active":"2024-09-18T07:33:39.311522+0000","last_peered":"2024-09-18T07:33:39.311522+0000","last_clean":"2024-09-18T07:33:39.311522+0000","last_became_active":"2024-09-18T07:30:58.316812+0000","last_became_peered":"2024-09-18T07:30:58.316812+0000","last_unstale":"2024-09-18T07:33:39.311522+0000","last_undegraded":"2024-09-18T07:33:39.311522+0000","last_fullsized":"2024-09-18T07:33:39.311522+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:09.123487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":139,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.304275+0000","last_change":"2024-09-18T07:30:56.285353+0000","last_active":"2024-09-18T07:33:39.304275+0000","last_peered":"2024-09-18T07:33:39.304275+0000","last_clean":"2024-09-18T07:33:39.304275+0000","last_became_active":"2024-09-18T07:30:56.285032+0000","last_became_peered":"2024-09-18T07:30:56.285032+0000","last_unstale":"2024-09-18T07:33:39.304275+0000","last_undegraded":"2024-09-18T07:33:39.304275+0000","last_fullsized":"2024-09-18T07:33:39.304275+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":148,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314775+0000","last_change":"2024-09-18T07:30:58.299308+0000","last_active":"2024-09-18T07:33:39.314775+0000","last_peered":"2024-09-18T07:33:39.314775+0000","last_clean":"2024-09-18T07:33:39.314775+0000","last_became_active":"2024-09-18T07:30:55.283292+0000","last_became_peered":"2024-09-18T07:30:55.283292+0000","last_unstale":"2024-09-18T07:33:39.314775+0000","last_undegraded":"2024-09-18T07:33:39.314775+0000","last_fullsized":"2024-09-18T07:33:39.314775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:19.297747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":130,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.317450+0000","last_change":"2024-09-18T07:31:00.332738+0000","last_active":"2024-09-18T07:33:39.317450+0000","last_peered":"2024-09-18T07:33:39.317450+0000","last_clean":"2024-09-18T07:33:39.317450+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:33:39.317450+0000","last_undegraded":"2024-09-18T07:33:39.317450+0000","last_fullsized":"2024-09-18T07:33:39.317450+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":133,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.318512+0000","last_change":"2024-09-18T07:32:30.854643+0000","last_active":"2024-09-18T07:33:39.318512+0000","last_peered":"2024-09-18T07:33:39.318512+0000","last_clean":"2024-09-18T07:33:39.318512+0000","last_became_active":"2024-09-18T07:32:30.853087+0000","last_became_peered":"2024-09-18T07:32:30.853087+0000","last_unstale":"2024-09-18T07:33:39.318512+0000","last_undegraded":"2024-09-18T07:33:39.318512+0000","last_fullsized":"2024-09-18T07:33:39.318512+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"124'6316","reported_seq":6411,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:41.597548+0000","last_change":"2024-09-18T07:32:08.063002+0000","last_active":"2024-09-18T07:33:41.597548+0000","last_peered":"2024-09-18T07:33:41.597548+0000","last_clean":"2024-09-18T07:33:41.597548+0000","last_became_active":"2024-09-18T07:32:08.062715+0000","last_became_peered":"2024-09-18T07:32:08.062715+0000","last_unstale":"2024-09-18T07:33:41.597548+0000","last_undegraded":"2024-09-18T07:33:41.597548+0000","last_fullsized":"2024-09-18T07:33:41.597548+0000","mapping_epoch":81,"log_start":"92'1709","ondisk_log_start":"92'1709","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4607,"log_dups_size":0,"ondisk_log_size":4607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413925376,"num_objects":6316,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6316,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":162,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.310405+0000","last_change":"2024-09-18T07:30:58.310789+0000","last_active":"2024-09-18T07:33:39.310405+0000","last_peered":"2024-09-18T07:33:39.310405+0000","last_clean":"2024-09-18T07:33:39.310405+0000","last_became_active":"2024-09-18T07:30:58.310490+0000","last_became_peered":"2024-09-18T07:30:58.310490+0000","last_unstale":"2024-09-18T07:33:39.310405+0000","last_undegraded":"2024-09-18T07:33:39.310405+0000","last_fullsized":"2024-09-18T07:33:39.310405+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.090903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":147,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.308136+0000","last_change":"2024-09-18T07:33:23.038159+0000","last_active":"2024-09-18T07:33:39.308136+0000","last_peered":"2024-09-18T07:33:39.308136+0000","last_clean":"2024-09-18T07:33:39.308136+0000","last_became_active":"2024-09-18T07:30:56.284803+0000","last_became_peered":"2024-09-18T07:30:56.284803+0000","last_unstale":"2024-09-18T07:33:39.308136+0000","last_undegraded":"2024-09-18T07:33:39.308136+0000","last_fullsized":"2024-09-18T07:33:39.308136+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:23.037963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:23.037963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:45.045160+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":156,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.305180+0000","last_change":"2024-09-18T07:32:02.418590+0000","last_active":"2024-09-18T07:33:39.305180+0000","last_peered":"2024-09-18T07:33:39.305180+0000","last_clean":"2024-09-18T07:33:39.305180+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:33:39.305180+0000","last_undegraded":"2024-09-18T07:33:39.305180+0000","last_fullsized":"2024-09-18T07:33:39.305180+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:32:02.418507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.418507+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":138,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.307033+0000","last_change":"2024-09-18T07:32:30.837706+0000","last_active":"2024-09-18T07:33:39.307033+0000","last_peered":"2024-09-18T07:33:39.307033+0000","last_clean":"2024-09-18T07:33:39.307033+0000","last_became_active":"2024-09-18T07:32:30.837243+0000","last_became_peered":"2024-09-18T07:32:30.837243+0000","last_unstale":"2024-09-18T07:33:39.307033+0000","last_undegraded":"2024-09-18T07:33:39.307033+0000","last_fullsized":"2024-09-18T07:33:39.307033+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:09.666733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":97,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315717+0000","last_change":"2024-09-18T07:32:55.796838+0000","last_active":"2024-09-18T07:33:39.315717+0000","last_peered":"2024-09-18T07:33:39.315717+0000","last_clean":"2024-09-18T07:33:39.315717+0000","last_became_active":"2024-09-18T07:32:30.839186+0000","last_became_peered":"2024-09-18T07:32:30.839186+0000","last_unstale":"2024-09-18T07:33:39.315717+0000","last_undegraded":"2024-09-18T07:33:39.315717+0000","last_fullsized":"2024-09-18T07:33:39.315717+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:32:55.796771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:55.796771+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:57.092409+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"124'6264","reported_seq":10180,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:41.495829+0000","last_change":"2024-09-18T07:33:01.225875+0000","last_active":"2024-09-18T07:33:41.495829+0000","last_peered":"2024-09-18T07:33:41.495829+0000","last_clean":"2024-09-18T07:33:41.495829+0000","last_became_active":"2024-09-18T07:33:01.221136+0000","last_became_peered":"2024-09-18T07:33:01.221136+0000","last_unstale":"2024-09-18T07:33:41.495829+0000","last_undegraded":"2024-09-18T07:33:41.495829+0000","last_fullsized":"2024-09-18T07:33:41.495829+0000","mapping_epoch":107,"log_start":"87'600","ondisk_log_start":"87'600","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5664,"log_dups_size":0,"ondisk_log_size":5664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:38.955324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410517504,"num_objects":6264,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6264,"num_write_kb":400896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1904,"num_bytes_recovered":124780544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":173,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.320932+0000","last_change":"2024-09-18T07:33:27.040227+0000","last_active":"2024-09-18T07:33:39.320932+0000","last_peered":"2024-09-18T07:33:39.320932+0000","last_clean":"2024-09-18T07:33:39.320932+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:33:39.320932+0000","last_undegraded":"2024-09-18T07:33:39.320932+0000","last_fullsized":"2024-09-18T07:33:39.320932+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":144,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.316516+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:33:39.316516+0000","last_peered":"2024-09-18T07:33:39.316516+0000","last_clean":"2024-09-18T07:33:39.316516+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:33:39.316516+0000","last_undegraded":"2024-09-18T07:33:39.316516+0000","last_fullsized":"2024-09-18T07:33:39.316516+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":191,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:40.031644+0000","last_change":"2024-09-18T07:32:30.859165+0000","last_active":"2024-09-18T07:33:40.031644+0000","last_peered":"2024-09-18T07:33:40.031644+0000","last_clean":"2024-09-18T07:33:40.031644+0000","last_became_active":"2024-09-18T07:32:30.851004+0000","last_became_peered":"2024-09-18T07:32:30.851004+0000","last_unstale":"2024-09-18T07:33:40.031644+0000","last_undegraded":"2024-09-18T07:33:40.031644+0000","last_fullsized":"2024-09-18T07:33:40.031644+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:04.076666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":77,"num_read_kb":72,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":138,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315109+0000","last_change":"2024-09-18T07:33:27.394247+0000","last_active":"2024-09-18T07:33:39.315109+0000","last_peered":"2024-09-18T07:33:39.315109+0000","last_clean":"2024-09-18T07:33:39.315109+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:33:39.315109+0000","last_undegraded":"2024-09-18T07:33:39.315109+0000","last_fullsized":"2024-09-18T07:33:39.315109+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.394203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.394203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:31.541179+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":143,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.310639+0000","last_change":"2024-09-18T07:33:25.028842+0000","last_active":"2024-09-18T07:33:39.310639+0000","last_peered":"2024-09-18T07:33:39.310639+0000","last_clean":"2024-09-18T07:33:39.310639+0000","last_became_active":"2024-09-18T07:31:02.347390+0000","last_became_peered":"2024-09-18T07:31:02.347390+0000","last_unstale":"2024-09-18T07:33:39.310639+0000","last_undegraded":"2024-09-18T07:33:39.310639+0000","last_fullsized":"2024-09-18T07:33:39.310639+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:25.028609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:25.028609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:34.346431+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"124'6359","reported_seq":10322,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:41.592515+0000","last_change":"2024-09-18T07:32:59.213954+0000","last_active":"2024-09-18T07:33:41.592515+0000","last_peered":"2024-09-18T07:33:41.592515+0000","last_clean":"2024-09-18T07:33:41.592515+0000","last_became_active":"2024-09-18T07:32:59.204989+0000","last_became_peered":"2024-09-18T07:32:59.204989+0000","last_unstale":"2024-09-18T07:33:41.592515+0000","last_undegraded":"2024-09-18T07:33:41.592515+0000","last_fullsized":"2024-09-18T07:33:41.592515+0000","mapping_epoch":105,"log_start":"92'1790","ondisk_log_start":"92'1790","created":81,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4569,"log_dups_size":0,"ondisk_log_size":4569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416743424,"num_objects":6359,"num_object_clones":0,"num_object_copies":19077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6359,"num_write_kb":406976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1927,"num_bytes_recovered":126287872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"94'17","reported_seq":231,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315860+0000","last_change":"2024-09-18T07:33:33.849354+0000","last_active":"2024-09-18T07:33:39.315860+0000","last_peered":"2024-09-18T07:33:39.315860+0000","last_clean":"2024-09-18T07:33:39.315860+0000","last_became_active":"2024-09-18T07:32:30.842449+0000","last_became_peered":"2024-09-18T07:32:30.842449+0000","last_unstale":"2024-09-18T07:33:39.315860+0000","last_undegraded":"2024-09-18T07:33:39.315860+0000","last_fullsized":"2024-09-18T07:33:39.315860+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'17","last_scrub_stamp":"2024-09-18T07:33:33.849285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.849285+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:53.616735+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":146,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315499+0000","last_change":"2024-09-18T07:31:59.617046+0000","last_active":"2024-09-18T07:33:39.315499+0000","last_peered":"2024-09-18T07:33:39.315499+0000","last_clean":"2024-09-18T07:33:39.315499+0000","last_became_active":"2024-09-18T07:30:56.288149+0000","last_became_peered":"2024-09-18T07:30:56.288149+0000","last_unstale":"2024-09-18T07:33:39.315499+0000","last_undegraded":"2024-09-18T07:33:39.315499+0000","last_fullsized":"2024-09-18T07:33:39.315499+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:59.616988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:59.616988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":168,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.313787+0000","last_change":"2024-09-18T07:33:23.172950+0000","last_active":"2024-09-18T07:33:39.313787+0000","last_peered":"2024-09-18T07:33:39.313787+0000","last_clean":"2024-09-18T07:33:39.313787+0000","last_became_active":"2024-09-18T07:32:30.842828+0000","last_became_peered":"2024-09-18T07:32:30.842828+0000","last_unstale":"2024-09-18T07:33:39.313787+0000","last_undegraded":"2024-09-18T07:33:39.313787+0000","last_fullsized":"2024-09-18T07:33:39.313787+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:33:23.172848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:33:23.172848+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.696337+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":73,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.307622+0000","last_change":"2024-09-18T07:32:30.835729+0000","last_active":"2024-09-18T07:33:39.307622+0000","last_peered":"2024-09-18T07:33:39.307622+0000","last_clean":"2024-09-18T07:33:39.307622+0000","last_became_active":"2024-09-18T07:32:30.835105+0000","last_became_peered":"2024-09-18T07:32:30.835105+0000","last_unstale":"2024-09-18T07:33:39.307622+0000","last_undegraded":"2024-09-18T07:33:39.307622+0000","last_fullsized":"2024-09-18T07:33:39.307622+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:02.499041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.499041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.738806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314897+0000","last_change":"2024-09-18T07:33:22.358792+0000","last_active":"2024-09-18T07:33:39.314897+0000","last_peered":"2024-09-18T07:33:39.314897+0000","last_clean":"2024-09-18T07:33:39.314897+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:33:39.314897+0000","last_undegraded":"2024-09-18T07:33:39.314897+0000","last_fullsized":"2024-09-18T07:33:39.314897+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.358746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.358746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:26.289679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"124'6294","reported_seq":6390,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:41.397789+0000","last_change":"2024-09-18T07:32:08.063469+0000","last_active":"2024-09-18T07:33:41.397789+0000","last_peered":"2024-09-18T07:33:41.397789+0000","last_clean":"2024-09-18T07:33:41.397789+0000","last_became_active":"2024-09-18T07:32:08.063070+0000","last_became_peered":"2024-09-18T07:32:08.063070+0000","last_unstale":"2024-09-18T07:33:41.397789+0000","last_undegraded":"2024-09-18T07:33:41.397789+0000","last_fullsized":"2024-09-18T07:33:41.397789+0000","mapping_epoch":81,"log_start":"87'700","ondisk_log_start":"87'700","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:39.798722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412483584,"num_objects":6294,"num_object_clones":0,"num_object_copies":18882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6294,"num_write_kb":402816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":173,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.303615+0000","last_change":"2024-09-18T07:32:26.793474+0000","last_active":"2024-09-18T07:33:39.303615+0000","last_peered":"2024-09-18T07:33:39.303615+0000","last_clean":"2024-09-18T07:33:39.303615+0000","last_became_active":"2024-09-18T07:32:26.785711+0000","last_became_peered":"2024-09-18T07:32:26.785711+0000","last_unstale":"2024-09-18T07:33:39.303615+0000","last_undegraded":"2024-09-18T07:33:39.303615+0000","last_fullsized":"2024-09-18T07:33:39.303615+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":146,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.303024+0000","last_change":"2024-09-18T07:32:30.829352+0000","last_active":"2024-09-18T07:33:39.303024+0000","last_peered":"2024-09-18T07:33:39.303024+0000","last_clean":"2024-09-18T07:33:39.303024+0000","last_became_active":"2024-09-18T07:32:30.828736+0000","last_became_peered":"2024-09-18T07:32:30.828736+0000","last_unstale":"2024-09-18T07:33:39.303024+0000","last_undegraded":"2024-09-18T07:33:39.303024+0000","last_fullsized":"2024-09-18T07:33:39.303024+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:19.900370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":138,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314778+0000","last_change":"2024-09-18T07:33:36.582039+0000","last_active":"2024-09-18T07:33:39.314778+0000","last_peered":"2024-09-18T07:33:39.314778+0000","last_clean":"2024-09-18T07:33:39.314778+0000","last_became_active":"2024-09-18T07:31:00.336172+0000","last_became_peered":"2024-09-18T07:31:00.336172+0000","last_unstale":"2024-09-18T07:33:39.314778+0000","last_undegraded":"2024-09-18T07:33:39.314778+0000","last_fullsized":"2024-09-18T07:33:39.314778+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:36.581998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.581998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:04.288035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315610+0000","last_change":"2024-09-18T07:33:20.990542+0000","last_active":"2024-09-18T07:33:39.315610+0000","last_peered":"2024-09-18T07:33:39.315610+0000","last_clean":"2024-09-18T07:33:39.315610+0000","last_became_active":"2024-09-18T07:31:02.490953+0000","last_became_peered":"2024-09-18T07:31:02.490953+0000","last_unstale":"2024-09-18T07:33:39.315610+0000","last_undegraded":"2024-09-18T07:33:39.315610+0000","last_fullsized":"2024-09-18T07:33:39.315610+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:20.990401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:20.990401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.208665+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"124'6213","reported_seq":6310,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:41.500036+0000","last_change":"2024-09-18T07:32:08.060282+0000","last_active":"2024-09-18T07:33:41.500036+0000","last_peered":"2024-09-18T07:33:41.500036+0000","last_clean":"2024-09-18T07:33:41.500036+0000","last_became_active":"2024-09-18T07:32:08.059971+0000","last_became_peered":"2024-09-18T07:32:08.059971+0000","last_unstale":"2024-09-18T07:33:41.500036+0000","last_undegraded":"2024-09-18T07:33:41.500036+0000","last_fullsized":"2024-09-18T07:33:41.500036+0000","mapping_epoch":81,"log_start":"87'500","ondisk_log_start":"87'500","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5713,"log_dups_size":0,"ondisk_log_size":5713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407175168,"num_objects":6213,"num_object_clones":0,"num_object_copies":18639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6213,"num_write_kb":397632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":163,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.303928+0000","last_change":"2024-09-18T07:30:58.312636+0000","last_active":"2024-09-18T07:33:39.303928+0000","last_peered":"2024-09-18T07:33:39.303928+0000","last_clean":"2024-09-18T07:33:39.303928+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:33:39.303928+0000","last_undegraded":"2024-09-18T07:33:39.303928+0000","last_fullsized":"2024-09-18T07:33:39.303928+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":138,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.313622+0000","last_change":"2024-09-18T07:30:56.284303+0000","last_active":"2024-09-18T07:33:39.313622+0000","last_peered":"2024-09-18T07:33:39.313622+0000","last_clean":"2024-09-18T07:33:39.313622+0000","last_became_active":"2024-09-18T07:30:56.284020+0000","last_became_peered":"2024-09-18T07:30:56.284020+0000","last_unstale":"2024-09-18T07:33:39.313622+0000","last_undegraded":"2024-09-18T07:33:39.313622+0000","last_fullsized":"2024-09-18T07:33:39.313622+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.835065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":139,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.310714+0000","last_change":"2024-09-18T07:32:05.239202+0000","last_active":"2024-09-18T07:33:39.310714+0000","last_peered":"2024-09-18T07:33:39.310714+0000","last_clean":"2024-09-18T07:33:39.310714+0000","last_became_active":"2024-09-18T07:31:00.332836+0000","last_became_peered":"2024-09-18T07:31:00.332836+0000","last_unstale":"2024-09-18T07:33:39.310714+0000","last_undegraded":"2024-09-18T07:33:39.310714+0000","last_fullsized":"2024-09-18T07:33:39.310714+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:05.239160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.239160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:37.915616+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":135,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.306794+0000","last_change":"2024-09-18T07:33:21.860561+0000","last_active":"2024-09-18T07:33:39.306794+0000","last_peered":"2024-09-18T07:33:39.306794+0000","last_clean":"2024-09-18T07:33:39.306794+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:33:39.306794+0000","last_undegraded":"2024-09-18T07:33:39.306794+0000","last_fullsized":"2024-09-18T07:33:39.306794+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.860495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.860495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:51.431373+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"124'6307","reported_seq":10238,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:41.581843+0000","last_change":"2024-09-18T07:33:14.766074+0000","last_active":"2024-09-18T07:33:41.581843+0000","last_peered":"2024-09-18T07:33:41.581843+0000","last_clean":"2024-09-18T07:33:41.581843+0000","last_became_active":"2024-09-18T07:33:14.762251+0000","last_became_peered":"2024-09-18T07:33:14.762251+0000","last_unstale":"2024-09-18T07:33:41.581843+0000","last_undegraded":"2024-09-18T07:33:41.581843+0000","last_fullsized":"2024-09-18T07:33:41.581843+0000","mapping_epoch":115,"log_start":"92'1730","ondisk_log_start":"92'1730","created":81,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4577,"log_dups_size":0,"ondisk_log_size":4577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413335552,"num_objects":6307,"num_object_clones":0,"num_object_copies":18921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6307,"num_write_kb":403648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1912,"num_bytes_recovered":125304832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":158,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.304409+0000","last_change":"2024-09-18T07:32:05.000363+0000","last_active":"2024-09-18T07:33:39.304409+0000","last_peered":"2024-09-18T07:33:39.304409+0000","last_clean":"2024-09-18T07:33:39.304409+0000","last_became_active":"2024-09-18T07:30:58.316423+0000","last_became_peered":"2024-09-18T07:30:58.316423+0000","last_unstale":"2024-09-18T07:33:39.304409+0000","last_undegraded":"2024-09-18T07:33:39.304409+0000","last_fullsized":"2024-09-18T07:33:39.304409+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:32:05.000310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.000310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":146,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.316302+0000","last_change":"2024-09-18T07:32:06.688493+0000","last_active":"2024-09-18T07:33:39.316302+0000","last_peered":"2024-09-18T07:33:39.316302+0000","last_clean":"2024-09-18T07:33:39.316302+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:33:39.316302+0000","last_undegraded":"2024-09-18T07:33:39.316302+0000","last_fullsized":"2024-09-18T07:33:39.316302+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:06.688444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:06.688444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"120'10","reported_seq":203,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.307025+0000","last_change":"2024-09-18T07:33:33.001389+0000","last_active":"2024-09-18T07:33:39.307025+0000","last_peered":"2024-09-18T07:33:39.307025+0000","last_clean":"2024-09-18T07:33:39.307025+0000","last_became_active":"2024-09-18T07:33:27.007386+0000","last_became_peered":"2024-09-18T07:33:27.007386+0000","last_unstale":"2024-09-18T07:33:39.307025+0000","last_undegraded":"2024-09-18T07:33:39.307025+0000","last_fullsized":"2024-09-18T07:33:39.307025+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'10","last_scrub_stamp":"2024-09-18T07:33:33.000874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.000874+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:48.823137+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":127,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.303469+0000","last_change":"2024-09-18T07:31:02.347721+0000","last_active":"2024-09-18T07:33:39.303469+0000","last_peered":"2024-09-18T07:33:39.303469+0000","last_clean":"2024-09-18T07:33:39.303469+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:33:39.303469+0000","last_undegraded":"2024-09-18T07:33:39.303469+0000","last_fullsized":"2024-09-18T07:33:39.303469+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"124'6331","reported_seq":10463,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:41.443691+0000","last_change":"2024-09-18T07:33:04.288238+0000","last_active":"2024-09-18T07:33:41.443691+0000","last_peered":"2024-09-18T07:33:41.443691+0000","last_clean":"2024-09-18T07:33:41.443691+0000","last_became_active":"2024-09-18T07:33:04.282407+0000","last_became_peered":"2024-09-18T07:33:04.282407+0000","last_unstale":"2024-09-18T07:33:41.443691+0000","last_undegraded":"2024-09-18T07:33:41.443691+0000","last_fullsized":"2024-09-18T07:33:41.443691+0000","mapping_epoch":110,"log_start":"87'600","ondisk_log_start":"87'600","created":81,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5731,"log_dups_size":0,"ondisk_log_size":5731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414908416,"num_objects":6331,"num_object_clones":0,"num_object_copies":18993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6331,"num_write_kb":405184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2011,"num_bytes_recovered":131792896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":150,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314089+0000","last_change":"2024-09-18T07:30:58.314424+0000","last_active":"2024-09-18T07:33:39.314089+0000","last_peered":"2024-09-18T07:33:39.314089+0000","last_clean":"2024-09-18T07:33:39.314089+0000","last_became_active":"2024-09-18T07:30:58.314127+0000","last_became_peered":"2024-09-18T07:30:58.314127+0000","last_unstale":"2024-09-18T07:33:39.314089+0000","last_undegraded":"2024-09-18T07:33:39.314089+0000","last_fullsized":"2024-09-18T07:33:39.314089+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:37.131991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":146,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314453+0000","last_change":"2024-09-18T07:33:28.459796+0000","last_active":"2024-09-18T07:33:39.314453+0000","last_peered":"2024-09-18T07:33:39.314453+0000","last_clean":"2024-09-18T07:33:39.314453+0000","last_became_active":"2024-09-18T07:30:56.288868+0000","last_became_peered":"2024-09-18T07:30:56.288868+0000","last_unstale":"2024-09-18T07:33:39.314453+0000","last_undegraded":"2024-09-18T07:33:39.314453+0000","last_fullsized":"2024-09-18T07:33:39.314453+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:28.459747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:28.459747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:54.082899+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":131,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.308855+0000","last_change":"2024-09-18T07:31:00.336020+0000","last_active":"2024-09-18T07:33:39.308855+0000","last_peered":"2024-09-18T07:33:39.308855+0000","last_clean":"2024-09-18T07:33:39.308855+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:33:39.308855+0000","last_undegraded":"2024-09-18T07:33:39.308855+0000","last_fullsized":"2024-09-18T07:33:39.308855+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:12.700165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":133,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.316426+0000","last_change":"2024-09-18T07:32:30.854343+0000","last_active":"2024-09-18T07:33:39.316426+0000","last_peered":"2024-09-18T07:33:39.316426+0000","last_clean":"2024-09-18T07:33:39.316426+0000","last_became_active":"2024-09-18T07:32:30.854029+0000","last_became_peered":"2024-09-18T07:32:30.854029+0000","last_unstale":"2024-09-18T07:33:39.316426+0000","last_undegraded":"2024-09-18T07:33:39.316426+0000","last_fullsized":"2024-09-18T07:33:39.316426+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"124'6270","reported_seq":6366,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:41.586969+0000","last_change":"2024-09-18T07:32:08.057221+0000","last_active":"2024-09-18T07:33:41.586969+0000","last_peered":"2024-09-18T07:33:41.586969+0000","last_clean":"2024-09-18T07:33:41.586969+0000","last_became_active":"2024-09-18T07:32:08.056907+0000","last_became_peered":"2024-09-18T07:32:08.056907+0000","last_unstale":"2024-09-18T07:33:41.586969+0000","last_undegraded":"2024-09-18T07:33:41.586969+0000","last_fullsized":"2024-09-18T07:33:41.586969+0000","mapping_epoch":81,"log_start":"92'1702","ondisk_log_start":"92'1702","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4568,"log_dups_size":0,"ondisk_log_size":4568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410910720,"num_objects":6270,"num_object_clones":0,"num_object_copies":18810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6270,"num_write_kb":401280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":208,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.313369+0000","last_change":"2024-09-18T07:33:32.803961+0000","last_active":"2024-09-18T07:33:39.313369+0000","last_peered":"2024-09-18T07:33:39.313369+0000","last_clean":"2024-09-18T07:33:39.313369+0000","last_became_active":"2024-09-18T07:32:30.834955+0000","last_became_peered":"2024-09-18T07:32:30.834955+0000","last_unstale":"2024-09-18T07:33:39.313369+0000","last_undegraded":"2024-09-18T07:33:39.313369+0000","last_fullsized":"2024-09-18T07:33:39.313369+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:33:32.803884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.803884+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:34.097591+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":74,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315057+0000","last_change":"2024-09-18T07:32:30.840417+0000","last_active":"2024-09-18T07:33:39.315057+0000","last_peered":"2024-09-18T07:33:39.315057+0000","last_clean":"2024-09-18T07:33:39.315057+0000","last_became_active":"2024-09-18T07:32:30.840046+0000","last_became_peered":"2024-09-18T07:32:30.840046+0000","last_unstale":"2024-09-18T07:33:39.315057+0000","last_undegraded":"2024-09-18T07:33:39.315057+0000","last_fullsized":"2024-09-18T07:33:39.315057+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.481153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.481153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":146,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.305487+0000","last_change":"2024-09-18T07:33:34.966610+0000","last_active":"2024-09-18T07:33:39.305487+0000","last_peered":"2024-09-18T07:33:39.305487+0000","last_clean":"2024-09-18T07:33:39.305487+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:33:39.305487+0000","last_undegraded":"2024-09-18T07:33:39.305487+0000","last_fullsized":"2024-09-18T07:33:39.305487+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:34.966537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:34.966537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.041153+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":73,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.305822+0000","last_change":"2024-09-18T07:32:30.839764+0000","last_active":"2024-09-18T07:33:39.305822+0000","last_peered":"2024-09-18T07:33:39.305822+0000","last_clean":"2024-09-18T07:33:39.305822+0000","last_became_active":"2024-09-18T07:32:30.839399+0000","last_became_peered":"2024-09-18T07:32:30.839399+0000","last_unstale":"2024-09-18T07:33:39.305822+0000","last_undegraded":"2024-09-18T07:33:39.305822+0000","last_fullsized":"2024-09-18T07:33:39.305822+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:05.699246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.a","version":"124'6286","reported_seq":6382,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:41.509829+0000","last_change":"2024-09-18T07:32:08.065180+0000","last_active":"2024-09-18T07:33:41.509829+0000","last_peered":"2024-09-18T07:33:41.509829+0000","last_clean":"2024-09-18T07:33:41.509829+0000","last_became_active":"2024-09-18T07:32:08.064911+0000","last_became_peered":"2024-09-18T07:32:08.064911+0000","last_unstale":"2024-09-18T07:33:41.509829+0000","last_undegraded":"2024-09-18T07:33:41.509829+0000","last_fullsized":"2024-09-18T07:33:41.509829+0000","mapping_epoch":81,"log_start":"96'2100","ondisk_log_start":"96'2100","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4186,"log_dups_size":0,"ondisk_log_size":4186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411959296,"num_objects":6286,"num_object_clones":0,"num_object_copies":18858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6286,"num_write_kb":402304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":166,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.302888+0000","last_change":"2024-09-18T07:32:02.967912+0000","last_active":"2024-09-18T07:33:39.302888+0000","last_peered":"2024-09-18T07:33:39.302888+0000","last_clean":"2024-09-18T07:33:39.302888+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:33:39.302888+0000","last_undegraded":"2024-09-18T07:33:39.302888+0000","last_fullsized":"2024-09-18T07:33:39.302888+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:32:02.967835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.967835+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":74,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.303708+0000","last_change":"2024-09-18T07:32:30.846539+0000","last_active":"2024-09-18T07:33:39.303708+0000","last_peered":"2024-09-18T07:33:39.303708+0000","last_clean":"2024-09-18T07:33:39.303708+0000","last_became_active":"2024-09-18T07:32:30.846204+0000","last_became_peered":"2024-09-18T07:32:30.846204+0000","last_unstale":"2024-09-18T07:33:39.303708+0000","last_undegraded":"2024-09-18T07:33:39.303708+0000","last_fullsized":"2024-09-18T07:33:39.303708+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:17.888739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":139,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.305940+0000","last_change":"2024-09-18T07:32:04.417412+0000","last_active":"2024-09-18T07:33:39.305940+0000","last_peered":"2024-09-18T07:33:39.305940+0000","last_clean":"2024-09-18T07:33:39.305940+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:33:39.305940+0000","last_undegraded":"2024-09-18T07:33:39.305940+0000","last_fullsized":"2024-09-18T07:33:39.305940+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.417368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.417368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":133,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.316122+0000","last_change":"2024-09-18T07:32:30.849280+0000","last_active":"2024-09-18T07:33:39.316122+0000","last_peered":"2024-09-18T07:33:39.316122+0000","last_clean":"2024-09-18T07:33:39.316122+0000","last_became_active":"2024-09-18T07:32:30.847635+0000","last_became_peered":"2024-09-18T07:32:30.847635+0000","last_unstale":"2024-09-18T07:33:39.316122+0000","last_undegraded":"2024-09-18T07:33:39.316122+0000","last_fullsized":"2024-09-18T07:33:39.316122+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"124'6170","reported_seq":6266,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:41.484710+0000","last_change":"2024-09-18T07:32:08.062243+0000","last_active":"2024-09-18T07:33:41.484710+0000","last_peered":"2024-09-18T07:33:41.484710+0000","last_clean":"2024-09-18T07:33:41.484710+0000","last_became_active":"2024-09-18T07:32:08.061925+0000","last_became_peered":"2024-09-18T07:32:08.061925+0000","last_unstale":"2024-09-18T07:33:41.484710+0000","last_undegraded":"2024-09-18T07:33:41.484710+0000","last_fullsized":"2024-09-18T07:33:41.484710+0000","mapping_epoch":81,"log_start":"96'2000","ondisk_log_start":"96'2000","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4170,"log_dups_size":0,"ondisk_log_size":4170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.902340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404357120,"num_objects":6170,"num_object_clones":0,"num_object_copies":18510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6170,"num_write_kb":394880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":182,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314015+0000","last_change":"2024-09-18T07:32:30.858426+0000","last_active":"2024-09-18T07:33:39.314015+0000","last_peered":"2024-09-18T07:33:39.314015+0000","last_clean":"2024-09-18T07:33:39.314015+0000","last_became_active":"2024-09-18T07:32:30.832486+0000","last_became_peered":"2024-09-18T07:32:30.832486+0000","last_unstale":"2024-09-18T07:33:39.314015+0000","last_undegraded":"2024-09-18T07:33:39.314015+0000","last_fullsized":"2024-09-18T07:33:39.314015+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:01.244752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":90,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.304776+0000","last_change":"2024-09-18T07:32:30.865327+0000","last_active":"2024-09-18T07:33:39.304776+0000","last_peered":"2024-09-18T07:33:39.304776+0000","last_clean":"2024-09-18T07:33:39.304776+0000","last_became_active":"2024-09-18T07:32:30.844389+0000","last_became_peered":"2024-09-18T07:32:30.844389+0000","last_unstale":"2024-09-18T07:33:39.304776+0000","last_undegraded":"2024-09-18T07:33:39.304776+0000","last_fullsized":"2024-09-18T07:33:39.304776+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.313392+0000","last_change":"2024-09-18T07:32:57.792858+0000","last_active":"2024-09-18T07:33:39.313392+0000","last_peered":"2024-09-18T07:33:39.313392+0000","last_clean":"2024-09-18T07:33:39.313392+0000","last_became_active":"2024-09-18T07:32:30.831224+0000","last_became_peered":"2024-09-18T07:32:30.831224+0000","last_unstale":"2024-09-18T07:33:39.313392+0000","last_undegraded":"2024-09-18T07:33:39.313392+0000","last_fullsized":"2024-09-18T07:33:39.313392+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-09-18T07:32:57.792795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:57.792795+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:00.822109+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314228+0000","last_change":"2024-09-18T07:31:02.489914+0000","last_active":"2024-09-18T07:33:39.314228+0000","last_peered":"2024-09-18T07:33:39.314228+0000","last_clean":"2024-09-18T07:33:39.314228+0000","last_became_active":"2024-09-18T07:31:02.489619+0000","last_became_peered":"2024-09-18T07:31:02.489619+0000","last_unstale":"2024-09-18T07:33:39.314228+0000","last_undegraded":"2024-09-18T07:33:39.314228+0000","last_fullsized":"2024-09-18T07:33:39.314228+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:13.393902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":146,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315101+0000","last_change":"2024-09-18T07:32:04.369688+0000","last_active":"2024-09-18T07:33:39.315101+0000","last_peered":"2024-09-18T07:33:39.315101+0000","last_clean":"2024-09-18T07:33:39.315101+0000","last_became_active":"2024-09-18T07:30:56.282357+0000","last_became_peered":"2024-09-18T07:30:56.282357+0000","last_unstale":"2024-09-18T07:33:39.315101+0000","last_undegraded":"2024-09-18T07:33:39.315101+0000","last_fullsized":"2024-09-18T07:33:39.315101+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.369620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.369620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:14.844761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":158,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.321448+0000","last_change":"2024-09-18T07:32:30.872792+0000","last_active":"2024-09-18T07:33:39.321448+0000","last_peered":"2024-09-18T07:33:39.321448+0000","last_clean":"2024-09-18T07:33:39.321448+0000","last_became_active":"2024-09-18T07:32:30.856211+0000","last_became_peered":"2024-09-18T07:32:30.856211+0000","last_unstale":"2024-09-18T07:33:39.321448+0000","last_undegraded":"2024-09-18T07:33:39.321448+0000","last_fullsized":"2024-09-18T07:33:39.321448+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:19.033885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":138,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314733+0000","last_change":"2024-09-18T07:33:32.589362+0000","last_active":"2024-09-18T07:33:39.314733+0000","last_peered":"2024-09-18T07:33:39.314733+0000","last_clean":"2024-09-18T07:33:39.314733+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:33:39.314733+0000","last_undegraded":"2024-09-18T07:33:39.314733+0000","last_fullsized":"2024-09-18T07:33:39.314733+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.589289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.589289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:38.300848+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":81,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315615+0000","last_change":"2024-09-18T07:32:52.808559+0000","last_active":"2024-09-18T07:33:39.315615+0000","last_peered":"2024-09-18T07:33:39.315615+0000","last_clean":"2024-09-18T07:33:39.315615+0000","last_became_active":"2024-09-18T07:32:30.831805+0000","last_became_peered":"2024-09-18T07:32:30.831805+0000","last_unstale":"2024-09-18T07:33:39.315615+0000","last_undegraded":"2024-09-18T07:33:39.315615+0000","last_fullsized":"2024-09-18T07:33:39.315615+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:52.808493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:52.808493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:58.415698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":167,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.321019+0000","last_change":"2024-09-18T07:32:30.868702+0000","last_active":"2024-09-18T07:33:39.321019+0000","last_peered":"2024-09-18T07:33:39.321019+0000","last_clean":"2024-09-18T07:33:39.321019+0000","last_became_active":"2024-09-18T07:32:30.861402+0000","last_became_peered":"2024-09-18T07:32:30.861402+0000","last_unstale":"2024-09-18T07:33:39.321019+0000","last_undegraded":"2024-09-18T07:33:39.321019+0000","last_fullsized":"2024-09-18T07:33:39.321019+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":74,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.302809+0000","last_change":"2024-09-18T07:32:30.842334+0000","last_active":"2024-09-18T07:33:39.302809+0000","last_peered":"2024-09-18T07:33:39.302809+0000","last_clean":"2024-09-18T07:33:39.302809+0000","last_became_active":"2024-09-18T07:32:30.841928+0000","last_became_peered":"2024-09-18T07:32:30.841928+0000","last_unstale":"2024-09-18T07:33:39.302809+0000","last_undegraded":"2024-09-18T07:33:39.302809+0000","last_fullsized":"2024-09-18T07:33:39.302809+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":137,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.321358+0000","last_change":"2024-09-18T07:32:30.858456+0000","last_active":"2024-09-18T07:33:39.321358+0000","last_peered":"2024-09-18T07:33:39.321358+0000","last_clean":"2024-09-18T07:33:39.321358+0000","last_became_active":"2024-09-18T07:32:30.858167+0000","last_became_peered":"2024-09-18T07:32:30.858167+0000","last_unstale":"2024-09-18T07:33:39.321358+0000","last_undegraded":"2024-09-18T07:33:39.321358+0000","last_fullsized":"2024-09-18T07:33:39.321358+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.307482+0000","last_change":"2024-09-18T07:32:30.835415+0000","last_active":"2024-09-18T07:33:39.307482+0000","last_peered":"2024-09-18T07:33:39.307482+0000","last_clean":"2024-09-18T07:33:39.307482+0000","last_became_active":"2024-09-18T07:32:30.834708+0000","last_became_peered":"2024-09-18T07:32:30.834708+0000","last_unstale":"2024-09-18T07:33:39.307482+0000","last_undegraded":"2024-09-18T07:33:39.307482+0000","last_fullsized":"2024-09-18T07:33:39.307482+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:07.726137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":167,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315140+0000","last_change":"2024-09-18T07:32:30.851755+0000","last_active":"2024-09-18T07:33:39.315140+0000","last_peered":"2024-09-18T07:33:39.315140+0000","last_clean":"2024-09-18T07:33:39.315140+0000","last_became_active":"2024-09-18T07:32:30.837622+0000","last_became_peered":"2024-09-18T07:32:30.837622+0000","last_unstale":"2024-09-18T07:33:39.315140+0000","last_undegraded":"2024-09-18T07:33:39.315140+0000","last_fullsized":"2024-09-18T07:33:39.315140+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:10.412163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":145,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.316965+0000","last_change":"2024-09-18T07:32:30.854427+0000","last_active":"2024-09-18T07:33:39.316965+0000","last_peered":"2024-09-18T07:33:39.316965+0000","last_clean":"2024-09-18T07:33:39.316965+0000","last_became_active":"2024-09-18T07:32:30.853969+0000","last_became_peered":"2024-09-18T07:32:30.853969+0000","last_unstale":"2024-09-18T07:33:39.316965+0000","last_undegraded":"2024-09-18T07:33:39.316965+0000","last_fullsized":"2024-09-18T07:33:39.316965+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:15.041483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"94'9","reported_seq":183,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315802+0000","last_change":"2024-09-18T07:32:30.867244+0000","last_active":"2024-09-18T07:33:39.315802+0000","last_peered":"2024-09-18T07:33:39.315802+0000","last_clean":"2024-09-18T07:33:39.315802+0000","last_became_active":"2024-09-18T07:32:30.848580+0000","last_became_peered":"2024-09-18T07:32:30.848580+0000","last_unstale":"2024-09-18T07:33:39.315802+0000","last_undegraded":"2024-09-18T07:33:39.315802+0000","last_fullsized":"2024-09-18T07:33:39.315802+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.090692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":127,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.304598+0000","last_change":"2024-09-18T07:31:02.343206+0000","last_active":"2024-09-18T07:33:39.304598+0000","last_peered":"2024-09-18T07:33:39.304598+0000","last_clean":"2024-09-18T07:33:39.304598+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:33:39.304598+0000","last_undegraded":"2024-09-18T07:33:39.304598+0000","last_fullsized":"2024-09-18T07:33:39.304598+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":168,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.316123+0000","last_change":"2024-09-18T07:32:30.867689+0000","last_active":"2024-09-18T07:33:39.316123+0000","last_peered":"2024-09-18T07:33:39.316123+0000","last_clean":"2024-09-18T07:33:39.316123+0000","last_became_active":"2024-09-18T07:32:30.848136+0000","last_became_peered":"2024-09-18T07:32:30.848136+0000","last_unstale":"2024-09-18T07:33:39.316123+0000","last_undegraded":"2024-09-18T07:33:39.316123+0000","last_fullsized":"2024-09-18T07:33:39.316123+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":74,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.304100+0000","last_change":"2024-09-18T07:32:30.832648+0000","last_active":"2024-09-18T07:33:39.304100+0000","last_peered":"2024-09-18T07:33:39.304100+0000","last_clean":"2024-09-18T07:33:39.304100+0000","last_became_active":"2024-09-18T07:32:30.832312+0000","last_became_peered":"2024-09-18T07:32:30.832312+0000","last_unstale":"2024-09-18T07:33:39.304100+0000","last_undegraded":"2024-09-18T07:33:39.304100+0000","last_fullsized":"2024-09-18T07:33:39.304100+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.844687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":175,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314780+0000","last_change":"2024-09-18T07:31:00.332729+0000","last_active":"2024-09-18T07:33:39.314780+0000","last_peered":"2024-09-18T07:33:39.314780+0000","last_clean":"2024-09-18T07:33:39.314780+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:33:39.314780+0000","last_undegraded":"2024-09-18T07:33:39.314780+0000","last_fullsized":"2024-09-18T07:33:39.314780+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:11.427383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":141,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.317725+0000","last_change":"2024-09-18T07:33:32.135015+0000","last_active":"2024-09-18T07:33:39.317725+0000","last_peered":"2024-09-18T07:33:39.317725+0000","last_clean":"2024-09-18T07:33:39.317725+0000","last_became_active":"2024-09-18T07:32:30.854291+0000","last_became_peered":"2024-09-18T07:32:30.854291+0000","last_unstale":"2024-09-18T07:33:39.317725+0000","last_undegraded":"2024-09-18T07:33:39.317725+0000","last_fullsized":"2024-09-18T07:33:39.317725+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.134847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.134847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:34.913547+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":154,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.304013+0000","last_change":"2024-09-18T07:30:58.314730+0000","last_active":"2024-09-18T07:33:39.304013+0000","last_peered":"2024-09-18T07:33:39.304013+0000","last_clean":"2024-09-18T07:33:39.304013+0000","last_became_active":"2024-09-18T07:30:58.314428+0000","last_became_peered":"2024-09-18T07:30:58.314428+0000","last_unstale":"2024-09-18T07:33:39.304013+0000","last_undegraded":"2024-09-18T07:33:39.304013+0000","last_fullsized":"2024-09-18T07:33:39.304013+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":146,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.304632+0000","last_change":"2024-09-18T07:32:30.831845+0000","last_active":"2024-09-18T07:33:39.304632+0000","last_peered":"2024-09-18T07:33:39.304632+0000","last_clean":"2024-09-18T07:33:39.304632+0000","last_became_active":"2024-09-18T07:32:30.831546+0000","last_became_peered":"2024-09-18T07:32:30.831546+0000","last_unstale":"2024-09-18T07:33:39.304632+0000","last_undegraded":"2024-09-18T07:33:39.304632+0000","last_fullsized":"2024-09-18T07:33:39.304632+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:38.680767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":131,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.309188+0000","last_change":"2024-09-18T07:31:00.337826+0000","last_active":"2024-09-18T07:33:39.309188+0000","last_peered":"2024-09-18T07:33:39.309188+0000","last_clean":"2024-09-18T07:33:39.309188+0000","last_became_active":"2024-09-18T07:31:00.334158+0000","last_became_peered":"2024-09-18T07:31:00.334158+0000","last_unstale":"2024-09-18T07:33:39.309188+0000","last_undegraded":"2024-09-18T07:33:39.309188+0000","last_fullsized":"2024-09-18T07:33:39.309188+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.058588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314970+0000","last_change":"2024-09-18T07:31:02.351247+0000","last_active":"2024-09-18T07:33:39.314970+0000","last_peered":"2024-09-18T07:33:39.314970+0000","last_clean":"2024-09-18T07:33:39.314970+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:33:39.314970+0000","last_undegraded":"2024-09-18T07:33:39.314970+0000","last_fullsized":"2024-09-18T07:33:39.314970+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":30,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.313261+0000","last_change":"2024-09-18T07:33:27.033510+0000","last_active":"2024-09-18T07:33:39.313261+0000","last_peered":"2024-09-18T07:33:39.313261+0000","last_clean":"2024-09-18T07:33:39.313261+0000","last_became_active":"2024-09-18T07:33:27.003887+0000","last_became_peered":"2024-09-18T07:33:27.003887+0000","last_unstale":"2024-09-18T07:33:39.313261+0000","last_undegraded":"2024-09-18T07:33:39.313261+0000","last_fullsized":"2024-09-18T07:33:39.313261+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.187186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.187186+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":74,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315417+0000","last_change":"2024-09-18T07:32:30.847936+0000","last_active":"2024-09-18T07:33:39.315417+0000","last_peered":"2024-09-18T07:33:39.315417+0000","last_clean":"2024-09-18T07:33:39.315417+0000","last_became_active":"2024-09-18T07:32:30.847608+0000","last_became_peered":"2024-09-18T07:32:30.847608+0000","last_unstale":"2024-09-18T07:33:39.315417+0000","last_undegraded":"2024-09-18T07:33:39.315417+0000","last_fullsized":"2024-09-18T07:33:39.315417+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:13.935728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":130,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.316179+0000","last_change":"2024-09-18T07:31:00.333689+0000","last_active":"2024-09-18T07:33:39.316179+0000","last_peered":"2024-09-18T07:33:39.316179+0000","last_clean":"2024-09-18T07:33:39.316179+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:33:39.316179+0000","last_undegraded":"2024-09-18T07:33:39.316179+0000","last_fullsized":"2024-09-18T07:33:39.316179+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.313315+0000","last_change":"2024-09-18T07:32:04.503356+0000","last_active":"2024-09-18T07:33:39.313315+0000","last_peered":"2024-09-18T07:33:39.313315+0000","last_clean":"2024-09-18T07:33:39.313315+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:33:39.313315+0000","last_undegraded":"2024-09-18T07:33:39.313315+0000","last_fullsized":"2024-09-18T07:33:39.313315+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.503270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.503270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":172,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315172+0000","last_change":"2024-09-18T07:32:30.869693+0000","last_active":"2024-09-18T07:33:39.315172+0000","last_peered":"2024-09-18T07:33:39.315172+0000","last_clean":"2024-09-18T07:33:39.315172+0000","last_became_active":"2024-09-18T07:32:30.842218+0000","last_became_peered":"2024-09-18T07:32:30.842218+0000","last_unstale":"2024-09-18T07:33:39.315172+0000","last_undegraded":"2024-09-18T07:33:39.315172+0000","last_fullsized":"2024-09-18T07:33:39.315172+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":153,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.314922+0000","last_change":"2024-09-18T07:32:30.853892+0000","last_active":"2024-09-18T07:33:39.314922+0000","last_peered":"2024-09-18T07:33:39.314922+0000","last_clean":"2024-09-18T07:33:39.314922+0000","last_became_active":"2024-09-18T07:32:30.853486+0000","last_became_peered":"2024-09-18T07:32:30.853486+0000","last_unstale":"2024-09-18T07:33:39.314922+0000","last_undegraded":"2024-09-18T07:33:39.314922+0000","last_fullsized":"2024-09-18T07:33:39.314922+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:58.734027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:58.734027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":81,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.302258+0000","last_change":"2024-09-18T07:33:30.668600+0000","last_active":"2024-09-18T07:33:39.302258+0000","last_peered":"2024-09-18T07:33:39.302258+0000","last_clean":"2024-09-18T07:33:39.302258+0000","last_became_active":"2024-09-18T07:32:30.843194+0000","last_became_peered":"2024-09-18T07:32:30.843194+0000","last_unstale":"2024-09-18T07:33:39.302258+0000","last_undegraded":"2024-09-18T07:33:39.302258+0000","last_fullsized":"2024-09-18T07:33:39.302258+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.668428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.668428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:55.997265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.316257+0000","last_change":"2024-09-18T07:33:30.975087+0000","last_active":"2024-09-18T07:33:39.316257+0000","last_peered":"2024-09-18T07:33:39.316257+0000","last_clean":"2024-09-18T07:33:39.316257+0000","last_became_active":"2024-09-18T07:31:02.490994+0000","last_became_peered":"2024-09-18T07:31:02.490994+0000","last_unstale":"2024-09-18T07:33:39.316257+0000","last_undegraded":"2024-09-18T07:33:39.316257+0000","last_fullsized":"2024-09-18T07:33:39.316257+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.974886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.974886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.586219+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":161,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.316177+0000","last_change":"2024-09-18T07:33:29.988707+0000","last_active":"2024-09-18T07:33:39.316177+0000","last_peered":"2024-09-18T07:33:39.316177+0000","last_clean":"2024-09-18T07:33:39.316177+0000","last_became_active":"2024-09-18T07:32:26.791562+0000","last_became_peered":"2024-09-18T07:32:26.791562+0000","last_unstale":"2024-09-18T07:33:39.316177+0000","last_undegraded":"2024-09-18T07:33:39.316177+0000","last_fullsized":"2024-09-18T07:33:39.316177+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:33:29.988636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.988636+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:34.823616+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":138,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.319305+0000","last_change":"2024-09-18T07:30:56.283643+0000","last_active":"2024-09-18T07:33:39.319305+0000","last_peered":"2024-09-18T07:33:39.319305+0000","last_clean":"2024-09-18T07:33:39.319305+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:33:39.319305+0000","last_undegraded":"2024-09-18T07:33:39.319305+0000","last_fullsized":"2024-09-18T07:33:39.319305+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":145,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.313528+0000","last_change":"2024-09-18T07:32:31.593581+0000","last_active":"2024-09-18T07:33:39.313528+0000","last_peered":"2024-09-18T07:33:39.313528+0000","last_clean":"2024-09-18T07:33:39.313528+0000","last_became_active":"2024-09-18T07:32:30.841210+0000","last_became_peered":"2024-09-18T07:32:30.841210+0000","last_unstale":"2024-09-18T07:33:39.313528+0000","last_undegraded":"2024-09-18T07:33:39.313528+0000","last_fullsized":"2024-09-18T07:33:39.313528+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:31.593520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:31.593520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:50.013493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":81,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.308706+0000","last_change":"2024-09-18T07:33:36.017902+0000","last_active":"2024-09-18T07:33:39.308706+0000","last_peered":"2024-09-18T07:33:39.308706+0000","last_clean":"2024-09-18T07:33:39.308706+0000","last_became_active":"2024-09-18T07:32:30.850481+0000","last_became_peered":"2024-09-18T07:32:30.850481+0000","last_unstale":"2024-09-18T07:33:39.308706+0000","last_undegraded":"2024-09-18T07:33:39.308706+0000","last_fullsized":"2024-09-18T07:33:39.308706+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:36.017602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.017602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.466305+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":135,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.309562+0000","last_change":"2024-09-18T07:33:24.053054+0000","last_active":"2024-09-18T07:33:39.309562+0000","last_peered":"2024-09-18T07:33:39.309562+0000","last_clean":"2024-09-18T07:33:39.309562+0000","last_became_active":"2024-09-18T07:31:02.347537+0000","last_became_peered":"2024-09-18T07:31:02.347537+0000","last_unstale":"2024-09-18T07:33:39.309562+0000","last_undegraded":"2024-09-18T07:33:39.309562+0000","last_fullsized":"2024-09-18T07:33:39.309562+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:24.052908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:24.052908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:39.677071+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":152,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.316419+0000","last_change":"2024-09-18T07:33:33.072569+0000","last_active":"2024-09-18T07:33:39.316419+0000","last_peered":"2024-09-18T07:33:39.316419+0000","last_clean":"2024-09-18T07:33:39.316419+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:33:39.316419+0000","last_undegraded":"2024-09-18T07:33:39.316419+0000","last_fullsized":"2024-09-18T07:33:39.316419+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:33.072487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.072487+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:50.254990+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":161,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.315018+0000","last_change":"2024-09-18T07:33:27.742768+0000","last_active":"2024-09-18T07:33:39.315018+0000","last_peered":"2024-09-18T07:33:39.315018+0000","last_clean":"2024-09-18T07:33:39.315018+0000","last_became_active":"2024-09-18T07:30:58.313556+0000","last_became_peered":"2024-09-18T07:30:58.313556+0000","last_unstale":"2024-09-18T07:33:39.315018+0000","last_undegraded":"2024-09-18T07:33:39.315018+0000","last_fullsized":"2024-09-18T07:33:39.315018+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:33:27.742684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.742684+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:51.970382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":145,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-18T07:33:39.320708+0000","last_change":"2024-09-18T07:33:22.161956+0000","last_active":"2024-09-18T07:33:39.320708+0000","last_peered":"2024-09-18T07:33:39.320708+0000","last_clean":"2024-09-18T07:33:39.320708+0000","last_became_active":"2024-09-18T07:32:30.846635+0000","last_became_peered":"2024-09-18T07:32:30.846635+0000","last_unstale":"2024-09-18T07:33:39.320708+0000","last_undegraded":"2024-09-18T07:33:39.320708+0000","last_fullsized":"2024-09-18T07:33:39.320708+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.161894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.161894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:31.115412+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6557597696,"num_objects":100061,"num_object_clones":0,"num_object_copies":300183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100061,"num_write_kb":6403904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9693,"num_bytes_recovered":635240448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19924058112,"data_stored":19924058112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78674,"ondisk_log_size":78674,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":1584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":77,"num_read_kb":72,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":236223201317,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462955,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2050616,"kb_used_data":1979176,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91694536,"statfs":{"total":95995035648,"available":93895204864,"internally_reserved":0,"allocated":2026676224,"data_stored":2026305230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2058172,"kb_used_data":1982572,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91686980,"statfs":{"total":95995035648,"available":93887467520,"internally_reserved":0,"allocated":2030153728,"data_stored":2029778660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953527,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2456916,"kb_used_data":2393792,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91288236,"statfs":{"total":95995035648,"available":93479153664,"internally_reserved":0,"allocated":2451243008,"data_stored":2450847922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4092644,"kb_used_data":3989268,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":89652508,"statfs":{"total":95995035648,"available":91804168192,"internally_reserved":0,"allocated":4085010432,"data_stored":4084596840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":105839047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,3,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2883764,"kb_used_data":2785956,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":90861388,"statfs":{"total":95995035648,"available":93042061312,"internally_reserved":0,"allocated":2852818944,"data_stored":2852451760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":100137413},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2870084,"kb_used_data":2784756,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":90875068,"statfs":{"total":95995035648,"available":93056069632,"internally_reserved":0,"allocated":2851590144,"data_stored":2851144293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738445,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3290072,"kb_used_data":3196424,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":90455080,"statfs":{"total":95995035648,"available":92626001920,"internally_reserved":0,"allocated":3273138176,"data_stored":3272736778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3286171648,"data_stored":3286171648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2867200000,"data_stored":2867200000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2866610176,"data_stored":2866610176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4100521984,"data_stored":4100521984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2468544512,"data_stored":2468544512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2045575168,"data_stored":2045575168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2037448704,"data_stored":2037448704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":251985920,"data_stored":251985920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:33:49.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:33:49.175 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:49.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:49.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:49.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:49.319 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:49.343 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:49.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:49.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:49.386 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T07:33:49.391 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:49 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3714358173' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:49.392 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:49 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3714358173' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:49.431 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:49.432 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:33:49.432 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:33:49.448 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:49.478 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:33:49.478 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:33:49.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:49.534 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:49.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:49.539 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:33:49.540 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:33:49.581 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:33:49.581 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:33:49.614 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:33:49.614 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:33:49.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:49.645 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:33:49.645 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:33:49.679 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:33:49.680 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:33:49.682 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:49.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:49.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:49.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:49 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3714358173' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:49.712 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:33:49.712 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:33:49.732 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:33:49.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:49.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:49.824 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:49.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:49.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:49.957 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:49.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:49.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:50.060 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:50.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:50.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:50.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:50.199 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:50.226 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:50.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:50.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:50.272 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:33:50.272 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:33:50.272 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:33:50.272 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:33:50.272 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:33:50.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:50.385 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:50.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:50.408 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:50 smithi137 ceph-mon[29408]: pgmap v129: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 864 B/s rd, 38 MiB/s wr, 717 op/s 2024-09-18T07:33:50.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:50.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:50.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:50.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:50.631 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:50.655 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:50.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:50.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:50.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:50 smithi137 ceph-mon[39175]: pgmap v129: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 864 B/s rd, 38 MiB/s wr, 717 op/s 2024-09-18T07:33:50.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:50 smithi203 ceph-mon[32237]: pgmap v129: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 864 B/s rd, 38 MiB/s wr, 717 op/s 2024-09-18T07:33:50.757 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:50.793 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:50.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:50.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:50.889 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:50.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:50.919 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:50.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:50.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:51.021 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:51.046 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:51.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:51.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:51.136 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T07:33:51.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:51.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:51.192 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:51.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:51.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:51.328 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:51.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:51.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:51.410 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:33:51.430 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:51.440 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:33:51.462 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:51.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:51.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:51.564 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:51.608 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:51.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:51.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:51.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:51 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3380520870' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:33:51.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:51 smithi137 ceph-mon[29408]: from='client.24910 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:33:51.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:51 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3380520870' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:33:51.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:51 smithi137 ceph-mon[39175]: from='client.24910 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:33:51.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:51 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3380520870' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:33:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:51 smithi203 ceph-mon[32237]: from='client.24910 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:33:51.756 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:51.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:51.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:51.858 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:51.869 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:33:51.877 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:33:51.885 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:51.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:51.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:51.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:51.993 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:33:51.994 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:33:51.994 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:33:51.994 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:33:51.994 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:51.994 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:19147 7.5 7:aecdd252:::benchmark_data_smithi137_74180_object19146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T07:33:51.994 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:22.733920+0000", 2024-09-18T07:33:51.994 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 89.255727782999998, 2024-09-18T07:33:51.994 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.029998256000000001, 2024-09-18T07:33:51.994 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:51.994 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:51.994 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:51.994 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:51.995 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:51.995 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:51.995 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 19147 2024-09-18T07:33:51.995 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:51.995 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:51.995 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:51.995 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:51.995 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:22.733920+0000", 2024-09-18T07:33:51.995 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:51.995 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:51.996 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:51.996 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:51.996 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:22.733920+0000", 2024-09-18T07:33:51.996 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:51.996 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:51.996 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:51.996 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:51.996 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:22.733922+0000", 2024-09-18T07:33:51.997 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8479999999999999e-06 2024-09-18T07:33:51.997 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:51.997 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:51.997 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:51.997 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:22.733972+0000", 2024-09-18T07:33:51.997 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9845999999999997e-05 2024-09-18T07:33:51.997 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:51.997 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:51.997 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:51.997 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:22.733981+0000", 2024-09-18T07:33:51.997 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4120000000000007e-06 2024-09-18T07:33:51.998 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:51.998 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:51.998 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:51.998 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:22.733987+0000", 2024-09-18T07:33:51.998 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6690000000000003e-06 2024-09-18T07:33:51.998 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:51.998 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:51.998 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:51.998 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:22.734040+0000", 2024-09-18T07:33:51.998 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3338999999999999e-05 2024-09-18T07:33:51.998 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:51.998 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:51.999 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:51.999 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:22.734123+0000", 2024-09-18T07:33:51.999 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3226999999999994e-05 2024-09-18T07:33:51.999 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:51.999 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:51.999 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:33:51.999 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:22.734206+0000", 2024-09-18T07:33:51.999 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3084000000000006e-05 2024-09-18T07:33:51.999 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:51.999 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:51.999 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:51.999 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:22.735039+0000", 2024-09-18T07:33:51.999 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00083301499999999999 2024-09-18T07:33:52.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.000 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.000 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.000 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:22.763465+0000", 2024-09-18T07:33:52.000 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.028425458000000001 2024-09-18T07:33:52.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.000 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.000 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.000 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:22.763832+0000", 2024-09-18T07:33:52.000 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000367518 2024-09-18T07:33:52.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.000 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.000 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.001 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:22.763881+0000", 2024-09-18T07:33:52.001 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8532000000000002e-05 2024-09-18T07:33:52.001 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.001 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.001 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.001 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:22.763918+0000", 2024-09-18T07:33:52.001 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7308000000000003e-05 2024-09-18T07:33:52.001 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.001 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.001 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.001 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.001 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.001 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:20119 7.5 7:a056c01b:::benchmark_data_smithi137_74180_object20118:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T07:33:52.002 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.214505+0000", 2024-09-18T07:33:52.002 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 88.775142832, 2024-09-18T07:33:52.002 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.046754022999999999, 2024-09-18T07:33:52.002 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.002 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.002 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.002 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:52.002 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:52.002 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:52.002 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 20119 2024-09-18T07:33:52.002 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.002 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.003 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.003 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214505+0000", 2024-09-18T07:33:52.003 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.003 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.003 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214505+0000", 2024-09-18T07:33:52.003 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.003 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.003 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214507+0000", 2024-09-18T07:33:52.004 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0559999999999999e-06 2024-09-18T07:33:52.004 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.004 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.004 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.004 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214562+0000", 2024-09-18T07:33:52.004 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4873999999999997e-05 2024-09-18T07:33:52.004 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.004 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.004 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.004 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214571+0000", 2024-09-18T07:33:52.004 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9639999999999992e-06 2024-09-18T07:33:52.004 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.004 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.004 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.005 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214577+0000", 2024-09-18T07:33:52.005 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1279999999999996e-06 2024-09-18T07:33:52.005 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.005 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.005 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.005 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214630+0000", 2024-09-18T07:33:52.005 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3260999999999997e-05 2024-09-18T07:33:52.005 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.005 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.005 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.005 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214750+0000", 2024-09-18T07:33:52.005 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011973599999999999 2024-09-18T07:33:52.005 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.006 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.006 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:33:52.006 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214880+0000", 2024-09-18T07:33:52.006 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000129634 2024-09-18T07:33:52.006 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.006 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.006 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:52.006 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.215650+0000", 2024-09-18T07:33:52.006 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00077066999999999999 2024-09-18T07:33:52.006 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.006 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.006 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.006 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.216265+0000", 2024-09-18T07:33:52.006 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00061457799999999996 2024-09-18T07:33:52.007 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.007 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.007 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.007 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.261179+0000", 2024-09-18T07:33:52.007 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.044914557000000001 2024-09-18T07:33:52.007 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.007 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.007 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.007 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.261218+0000", 2024-09-18T07:33:52.007 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8603000000000003e-05 2024-09-18T07:33:52.007 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.007 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.007 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.008 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.261259+0000", 2024-09-18T07:33:52.008 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0961999999999998e-05 2024-09-18T07:33:52.008 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.008 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.008 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.008 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.008 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.008 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:21294 7.8 e89/81 7:15222838:::benchmark_data_smithi137_74180_object21293:head v 89'1394, mlcod=89'1392)", 2024-09-18T07:33:52.008 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.948825+0000", 2024-09-18T07:33:52.008 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 88.040822857999999, 2024-09-18T07:33:52.008 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.094945371000000001, 2024-09-18T07:33:52.008 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.009 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.009 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.009 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.009 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.010 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948825+0000", 2024-09-18T07:33:52.010 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.010 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.010 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.010 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948825+0000", 2024-09-18T07:33:52.010 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.010 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.010 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.011 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948826+0000", 2024-09-18T07:33:52.011 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3e-07 2024-09-18T07:33:52.011 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.011 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.011 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.011 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948939+0000", 2024-09-18T07:33:52.011 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000113757 2024-09-18T07:33:52.011 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.011 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.011 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.011 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948994+0000", 2024-09-18T07:33:52.011 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4622e-05 2024-09-18T07:33:52.011 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.012 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.012 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.012 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.949000+0000", 2024-09-18T07:33:52.012 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9599999999999997e-06 2024-09-18T07:33:52.012 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.012 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.012 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.012 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.949042+0000", 2024-09-18T07:33:52.012 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2558999999999997e-05 2024-09-18T07:33:52.012 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.012 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.012 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.012 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.949055+0000", 2024-09-18T07:33:52.013 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2969999999999999e-05 2024-09-18T07:33:52.013 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.013 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.013 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.013 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.043723+0000", 2024-09-18T07:33:52.013 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.094668034999999998 2024-09-18T07:33:52.013 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.013 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.013 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.013 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.043770+0000", 2024-09-18T07:33:52.013 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6838000000000001e-05 2024-09-18T07:33:52.013 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.013 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.014 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.014 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.014 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.014 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:21293 7.f e89/81 7:fa840974:::benchmark_data_smithi137_74180_object21292:head v 89'1337, mlcod=89'1334)", 2024-09-18T07:33:52.014 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.948830+0000", 2024-09-18T07:33:52.014 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 88.040817838999999, 2024-09-18T07:33:52.014 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.096476913999999997, 2024-09-18T07:33:52.014 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.014 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.014 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.014 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.014 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.014 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948830+0000", 2024-09-18T07:33:52.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948830+0000", 2024-09-18T07:33:52.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948831+0000", 2024-09-18T07:33:52.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.235e-06 2024-09-18T07:33:52.016 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.016 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.016 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.016 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948942+0000", 2024-09-18T07:33:52.016 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000110417 2024-09-18T07:33:52.016 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.016 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.016 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.016 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948954+0000", 2024-09-18T07:33:52.016 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1888999999999999e-05 2024-09-18T07:33:52.016 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.016 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.016 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.017 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948959+0000", 2024-09-18T07:33:52.017 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8259999999999998e-06 2024-09-18T07:33:52.017 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.017 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.017 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.017 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948998+0000", 2024-09-18T07:33:52.017 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8498999999999999e-05 2024-09-18T07:33:52.017 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.017 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.017 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.017 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.949009+0000", 2024-09-18T07:33:52.017 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1413999999999999e-05 2024-09-18T07:33:52.017 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.018 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.018 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.018 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.045250+0000", 2024-09-18T07:33:52.018 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.096240945999999994 2024-09-18T07:33:52.018 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.018 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.018 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.018 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.045307+0000", 2024-09-18T07:33:52.018 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6687999999999997e-05 2024-09-18T07:33:52.018 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.018 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.018 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.018 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.018 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.018 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21896 7.5 7:a849d1d6:::benchmark_data_smithi137_74180_object21895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:33:52.019 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.316136+0000", 2024-09-18T07:33:52.019 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 87.673511696999995, 2024-09-18T07:33:52.019 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.073939160000000004, 2024-09-18T07:33:52.019 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.019 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.019 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.019 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:52.019 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:52.019 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:52.019 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21896 2024-09-18T07:33:52.019 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.019 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.019 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.019 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.020 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316136+0000", 2024-09-18T07:33:52.020 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.020 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.020 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.020 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316136+0000", 2024-09-18T07:33:52.020 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.020 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.020 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.020 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316138+0000", 2024-09-18T07:33:52.020 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.426e-06 2024-09-18T07:33:52.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.021 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.021 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316190+0000", 2024-09-18T07:33:52.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2704000000000002e-05 2024-09-18T07:33:52.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.021 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.021 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316201+0000", 2024-09-18T07:33:52.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0298e-05 2024-09-18T07:33:52.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.021 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.021 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316205+0000", 2024-09-18T07:33:52.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8860000000000003e-06 2024-09-18T07:33:52.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.022 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.022 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316244+0000", 2024-09-18T07:33:52.022 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8535999999999998e-05 2024-09-18T07:33:52.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.022 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.022 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316328+0000", 2024-09-18T07:33:52.022 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4384999999999999e-05 2024-09-18T07:33:52.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.022 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:33:52.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316417+0000", 2024-09-18T07:33:52.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8794000000000004e-05 2024-09-18T07:33:52.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.023 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.023 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.388000+0000", 2024-09-18T07:33:52.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.071582792000000006 2024-09-18T07:33:52.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.023 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.023 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:52.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.388272+0000", 2024-09-18T07:33:52.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00027234600000000002 2024-09-18T07:33:52.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.023 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.389970+0000", 2024-09-18T07:33:52.024 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016974959999999999 2024-09-18T07:33:52.024 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.024 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.390033+0000", 2024-09-18T07:33:52.024 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3640999999999996e-05 2024-09-18T07:33:52.024 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.024 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.390075+0000", 2024-09-18T07:33:52.024 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1856e-05 2024-09-18T07:33:52.024 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.025 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.025 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.025 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.025 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.025 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop_reply(client.14865.0:21896 7.5 e89/81 ondisk, result = 0)", 2024-09-18T07:33:52.025 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.317455+0000", 2024-09-18T07:33:52.025 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 87.672193307000001, 2024-09-18T07:33:52.025 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.070552445000000005, 2024-09-18T07:33:52.025 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.025 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.025 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:33:52.025 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.026 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.026 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.026 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.317455+0000", 2024-09-18T07:33:52.026 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.026 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.026 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.026 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.026 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.317455+0000", 2024-09-18T07:33:52.026 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.026 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.026 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.026 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.026 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.317455+0000", 2024-09-18T07:33:52.026 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4399999999999998e-07 2024-09-18T07:33:52.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.317459+0000", 2024-09-18T07:33:52.027 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7749999999999999e-06 2024-09-18T07:33:52.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.386905+0000", 2024-09-18T07:33:52.027 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.069445328000000001 2024-09-18T07:33:52.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.028 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.386916+0000", 2024-09-18T07:33:52.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1331e-05 2024-09-18T07:33:52.028 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.028 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.028 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.028 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.387987+0000", 2024-09-18T07:33:52.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010713020000000001 2024-09-18T07:33:52.028 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.029 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.029 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.029 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.387995+0000", 2024-09-18T07:33:52.029 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3679999999999996e-06 2024-09-18T07:33:52.029 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.029 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.029 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.029 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.388007+0000", 2024-09-18T07:33:52.029 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2397e-05 2024-09-18T07:33:52.030 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.030 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.030 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.030 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.030 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.030 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21999 7.5 7:a6ca4f28:::benchmark_data_smithi137_74180_object21998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:33:52.030 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.424624+0000", 2024-09-18T07:33:52.030 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 87.565023815000004, 2024-09-18T07:33:52.030 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.112047436, 2024-09-18T07:33:52.030 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.030 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.030 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.030 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:52.030 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:52.031 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:52.031 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21999 2024-09-18T07:33:52.031 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.031 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.031 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.031 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.031 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424624+0000", 2024-09-18T07:33:52.031 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.031 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.031 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.031 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.031 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424624+0000", 2024-09-18T07:33:52.031 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.031 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.031 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.032 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.032 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424626+0000", 2024-09-18T07:33:52.032 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.075e-06 2024-09-18T07:33:52.032 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.032 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.032 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.032 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424682+0000", 2024-09-18T07:33:52.032 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5597999999999999e-05 2024-09-18T07:33:52.032 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.032 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.032 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.032 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424694+0000", 2024-09-18T07:33:52.032 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2656e-05 2024-09-18T07:33:52.032 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.033 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.033 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424702+0000", 2024-09-18T07:33:52.033 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3849999999999996e-06 2024-09-18T07:33:52.033 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.033 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.033 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424758+0000", 2024-09-18T07:33:52.033 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6020000000000002e-05 2024-09-18T07:33:52.033 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.033 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.033 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424884+0000", 2024-09-18T07:33:52.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012638800000000001 2024-09-18T07:33:52.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:33:52.034 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425020+0000", 2024-09-18T07:33:52.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013587300000000001 2024-09-18T07:33:52.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.034 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.426628+0000", 2024-09-18T07:33:52.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001607792 2024-09-18T07:33:52.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:52.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.426665+0000", 2024-09-18T07:33:52.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7052999999999999e-05 2024-09-18T07:33:52.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536540+0000", 2024-09-18T07:33:52.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10987469399999999 2024-09-18T07:33:52.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536614+0000", 2024-09-18T07:33:52.036 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4486999999999993e-05 2024-09-18T07:33:52.036 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.036 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.036 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.036 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536671+0000", 2024-09-18T07:33:52.036 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7414999999999998e-05 2024-09-18T07:33:52.036 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.036 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.036 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.036 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.036 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.036 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22002 7.5 7:a1dbf95a:::benchmark_data_smithi137_74180_object22001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:33:52.036 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.425607+0000", 2024-09-18T07:33:52.037 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 87.564041172000003, 2024-09-18T07:33:52.037 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.111294137, 2024-09-18T07:33:52.037 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.037 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.037 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.037 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:52.037 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:52.037 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:52.037 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22002 2024-09-18T07:33:52.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.037 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.038 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425607+0000", 2024-09-18T07:33:52.038 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.038 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.038 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.038 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425607+0000", 2024-09-18T07:33:52.038 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.038 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.038 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.038 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425608+0000", 2024-09-18T07:33:52.038 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.297e-06 2024-09-18T07:33:52.039 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.039 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.039 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.039 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425682+0000", 2024-09-18T07:33:52.039 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3891000000000001e-05 2024-09-18T07:33:52.039 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.039 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.039 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.039 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425693+0000", 2024-09-18T07:33:52.039 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0944000000000001e-05 2024-09-18T07:33:52.039 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.039 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.040 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.040 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425701+0000", 2024-09-18T07:33:52.040 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7489999999999998e-06 2024-09-18T07:33:52.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.040 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.040 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.040 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425809+0000", 2024-09-18T07:33:52.040 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108323 2024-09-18T07:33:52.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.040 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.040 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.040 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425883+0000", 2024-09-18T07:33:52.040 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4321999999999998e-05 2024-09-18T07:33:52.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.041 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:33:52.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.426014+0000", 2024-09-18T07:33:52.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013092400000000001 2024-09-18T07:33:52.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.041 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:52.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.426735+0000", 2024-09-18T07:33:52.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00072051000000000005 2024-09-18T07:33:52.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.427445+0000", 2024-09-18T07:33:52.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00070996799999999995 2024-09-18T07:33:52.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536803+0000", 2024-09-18T07:33:52.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10935832500000001 2024-09-18T07:33:52.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536856+0000", 2024-09-18T07:33:52.043 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3161000000000001e-05 2024-09-18T07:33:52.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.043 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.043 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536901+0000", 2024-09-18T07:33:52.043 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4722999999999999e-05 2024-09-18T07:33:52.043 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.043 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.043 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.043 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22010 7.5 7:a615b038:::benchmark_data_smithi137_74180_object22009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:33:52.043 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.428662+0000", 2024-09-18T07:33:52.044 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 87.560985537999997, 2024-09-18T07:33:52.044 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.11444004000000001, 2024-09-18T07:33:52.044 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.044 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.044 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.044 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:52.044 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:52.044 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:52.044 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22010 2024-09-18T07:33:52.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.044 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.044 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.045 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428662+0000", 2024-09-18T07:33:52.045 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.045 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.045 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.045 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428662+0000", 2024-09-18T07:33:52.045 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.045 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.045 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.045 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428664+0000", 2024-09-18T07:33:52.045 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.302e-06 2024-09-18T07:33:52.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.045 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428707+0000", 2024-09-18T07:33:52.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3692000000000001e-05 2024-09-18T07:33:52.046 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.046 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428718+0000", 2024-09-18T07:33:52.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1066999999999999e-05 2024-09-18T07:33:52.046 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.046 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428724+0000", 2024-09-18T07:33:52.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7590000000000003e-06 2024-09-18T07:33:52.047 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.047 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.047 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.047 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428771+0000", 2024-09-18T07:33:52.047 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6462999999999999e-05 2024-09-18T07:33:52.047 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.047 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.047 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.047 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428852+0000", 2024-09-18T07:33:52.047 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1533999999999996e-05 2024-09-18T07:33:52.047 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.047 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.047 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:33:52.048 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428937+0000", 2024-09-18T07:33:52.048 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5068999999999995e-05 2024-09-18T07:33:52.048 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.048 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.048 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:52.048 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.429584+0000", 2024-09-18T07:33:52.048 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00064701400000000001 2024-09-18T07:33:52.048 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.048 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.048 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.048 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.429931+0000", 2024-09-18T07:33:52.048 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00034729299999999998 2024-09-18T07:33:52.048 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.049 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.049 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.049 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.543034+0000", 2024-09-18T07:33:52.049 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.113102231 2024-09-18T07:33:52.049 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.049 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.049 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.049 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.543071+0000", 2024-09-18T07:33:52.049 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7333999999999999e-05 2024-09-18T07:33:52.049 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.049 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.049 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.049 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.543102+0000", 2024-09-18T07:33:52.050 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1282000000000003e-05 2024-09-18T07:33:52.050 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.050 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.050 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.050 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.050 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.050 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:22343 7.6 e89/81 7:6ec6138d:::benchmark_data_smithi137_74180_object22342:head v 89'1362, mlcod=89'1360)", 2024-09-18T07:33:52.050 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.661175+0000", 2024-09-18T07:33:52.050 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 87.328472679000001, 2024-09-18T07:33:52.050 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19171324300000001, 2024-09-18T07:33:52.050 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.050 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.050 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.050 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.051 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.051 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.051 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661175+0000", 2024-09-18T07:33:52.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.051 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.051 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.051 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661175+0000", 2024-09-18T07:33:52.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.051 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.051 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.051 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661177+0000", 2024-09-18T07:33:52.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.548e-06 2024-09-18T07:33:52.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.052 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661235+0000", 2024-09-18T07:33:52.052 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7877999999999997e-05 2024-09-18T07:33:52.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.052 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661244+0000", 2024-09-18T07:33:52.052 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2709999999999998e-06 2024-09-18T07:33:52.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.052 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661250+0000", 2024-09-18T07:33:52.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8619999999999999e-06 2024-09-18T07:33:52.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661288+0000", 2024-09-18T07:33:52.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8330000000000001e-05 2024-09-18T07:33:52.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661300+0000", 2024-09-18T07:33:52.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1483e-05 2024-09-18T07:33:52.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.054 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.054 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852855+0000", 2024-09-18T07:33:52.054 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19155582199999999 2024-09-18T07:33:52.056 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.056 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.056 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.056 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852888+0000", 2024-09-18T07:33:52.056 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3049000000000002e-05 2024-09-18T07:33:52.056 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.056 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.056 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.056 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.056 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.056 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:22347 7.f e89/81 7:fe6074eb:::benchmark_data_smithi137_74180_object22346:head v 89'1397, mlcod=89'1395)", 2024-09-18T07:33:52.057 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.662928+0000", 2024-09-18T07:33:52.057 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 87.326720186000003, 2024-09-18T07:33:52.057 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19092352700000001, 2024-09-18T07:33:52.057 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.057 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.057 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.057 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.057 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.057 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.057 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662928+0000", 2024-09-18T07:33:52.057 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.057 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.057 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.058 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.058 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662928+0000", 2024-09-18T07:33:52.058 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.058 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.058 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.058 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.058 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662929+0000", 2024-09-18T07:33:52.058 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2500000000000004e-07 2024-09-18T07:33:52.058 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.058 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.058 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.058 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662989+0000", 2024-09-18T07:33:52.058 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0466e-05 2024-09-18T07:33:52.059 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.059 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.059 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.851940+0000", 2024-09-18T07:33:52.059 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18895100100000001 2024-09-18T07:33:52.059 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.059 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.059 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.851968+0000", 2024-09-18T07:33:52.059 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7668e-05 2024-09-18T07:33:52.059 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.059 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.060 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852054+0000", 2024-09-18T07:33:52.060 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6370000000000001e-05 2024-09-18T07:33:52.060 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.060 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.060 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.060 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852082+0000", 2024-09-18T07:33:52.060 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7769000000000001e-05 2024-09-18T07:33:52.060 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.060 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.060 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.060 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.853810+0000", 2024-09-18T07:33:52.060 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001728168 2024-09-18T07:33:52.060 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.061 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.061 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.061 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.853851+0000", 2024-09-18T07:33:52.061 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1159999999999999e-05 2024-09-18T07:33:52.061 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.061 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.061 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.061 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.061 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.061 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22450 7.5 7:ad888e37:::benchmark_data_smithi137_74180_object22449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:33:52.061 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.977850+0000", 2024-09-18T07:33:52.061 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 87.011798283999994, 2024-09-18T07:33:52.061 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.077184682000000004, 2024-09-18T07:33:52.062 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.062 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.062 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.062 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:52.062 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:52.062 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:52.062 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22450 2024-09-18T07:33:52.062 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.062 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.062 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.062 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.062 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977850+0000", 2024-09-18T07:33:52.062 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.063 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.063 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.063 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.063 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977850+0000", 2024-09-18T07:33:52.063 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.063 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.063 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.063 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.063 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977852+0000", 2024-09-18T07:33:52.063 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.108e-06 2024-09-18T07:33:52.063 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.063 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.063 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.064 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977892+0000", 2024-09-18T07:33:52.064 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9928999999999997e-05 2024-09-18T07:33:52.064 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.064 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.064 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.064 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977904+0000", 2024-09-18T07:33:52.064 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2843999999999999e-05 2024-09-18T07:33:52.064 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.064 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.064 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.064 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977912+0000", 2024-09-18T07:33:52.064 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1169999999999998e-06 2024-09-18T07:33:52.064 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.065 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.065 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.065 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977968+0000", 2024-09-18T07:33:52.065 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6338000000000002e-05 2024-09-18T07:33:52.065 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.065 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.065 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.065 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.978050+0000", 2024-09-18T07:33:52.065 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1841000000000005e-05 2024-09-18T07:33:52.065 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.065 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.065 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:33:52.065 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.978130+0000", 2024-09-18T07:33:52.066 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9917000000000001e-05 2024-09-18T07:33:52.066 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.066 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.066 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:52.066 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.978838+0000", 2024-09-18T07:33:52.066 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000708594 2024-09-18T07:33:52.066 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.066 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.066 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.066 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.979190+0000", 2024-09-18T07:33:52.066 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00035225000000000002 2024-09-18T07:33:52.066 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.066 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.066 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.066 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.054919+0000", 2024-09-18T07:33:52.067 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.075728917000000007 2024-09-18T07:33:52.067 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.067 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.067 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.067 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.054990+0000", 2024-09-18T07:33:52.067 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0300000000000001e-05 2024-09-18T07:33:52.067 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.067 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.067 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.067 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.055034+0000", 2024-09-18T07:33:52.067 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4527000000000002e-05 2024-09-18T07:33:52.067 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.067 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.068 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.068 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.068 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.068 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22938 7.5 7:accec075:::benchmark_data_smithi137_74180_object22937:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:33:52.068 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.270501+0000", 2024-09-18T07:33:52.068 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 86.719146882000004, 2024-09-18T07:33:52.068 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.048186932000000002, 2024-09-18T07:33:52.069 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.069 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.069 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.069 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:52.069 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:52.070 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:52.070 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22938 2024-09-18T07:33:52.070 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.070 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.070 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.070 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.070 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270501+0000", 2024-09-18T07:33:52.070 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.071 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.071 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.071 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270501+0000", 2024-09-18T07:33:52.071 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.071 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.071 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.071 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270503+0000", 2024-09-18T07:33:52.071 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6670000000000001e-06 2024-09-18T07:33:52.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.071 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.071 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.072 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270533+0000", 2024-09-18T07:33:52.072 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0889000000000002e-05 2024-09-18T07:33:52.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.072 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.072 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.072 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270542+0000", 2024-09-18T07:33:52.072 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4910000000000005e-06 2024-09-18T07:33:52.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.072 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.072 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.072 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270548+0000", 2024-09-18T07:33:52.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4300000000000003e-06 2024-09-18T07:33:52.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.073 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.073 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270701+0000", 2024-09-18T07:33:52.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015209800000000001 2024-09-18T07:33:52.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.073 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.073 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270812+0000", 2024-09-18T07:33:52.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000111415 2024-09-18T07:33:52.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.074 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:33:52.074 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270942+0000", 2024-09-18T07:33:52.074 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000130209 2024-09-18T07:33:52.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.074 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.074 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:52.074 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.271936+0000", 2024-09-18T07:33:52.074 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000994256 2024-09-18T07:33:52.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.074 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.074 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.074 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.272727+0000", 2024-09-18T07:33:52.074 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000790674 2024-09-18T07:33:52.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.074 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.318626+0000", 2024-09-18T07:33:52.075 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.045899052000000003 2024-09-18T07:33:52.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.075 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.318659+0000", 2024-09-18T07:33:52.075 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3209e-05 2024-09-18T07:33:52.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.075 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.318688+0000", 2024-09-18T07:33:52.076 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8541999999999999e-05 2024-09-18T07:33:52.076 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.076 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.076 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.076 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.076 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.076 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22945 7.5 7:a76344dc:::benchmark_data_smithi137_74180_object22944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:33:52.076 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.272901+0000", 2024-09-18T07:33:52.076 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 86.716746857999993, 2024-09-18T07:33:52.076 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.046985232000000002, 2024-09-18T07:33:52.076 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.076 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.076 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.077 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:52.077 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:52.077 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:52.077 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22945 2024-09-18T07:33:52.077 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.077 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.077 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.077 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.077 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.272901+0000", 2024-09-18T07:33:52.077 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.077 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.077 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.078 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.078 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.272901+0000", 2024-09-18T07:33:52.078 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.078 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.078 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.078 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.078 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.272903+0000", 2024-09-18T07:33:52.078 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0609999999999998e-06 2024-09-18T07:33:52.078 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.078 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.078 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.078 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.272941+0000", 2024-09-18T07:33:52.078 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8339e-05 2024-09-18T07:33:52.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.079 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.272960+0000", 2024-09-18T07:33:52.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9096000000000001e-05 2024-09-18T07:33:52.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.079 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.272968+0000", 2024-09-18T07:33:52.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4209999999999998e-06 2024-09-18T07:33:52.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.273014+0000", 2024-09-18T07:33:52.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5840000000000002e-05 2024-09-18T07:33:52.080 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.080 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.080 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.273115+0000", 2024-09-18T07:33:52.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000100973 2024-09-18T07:33:52.080 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.080 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.080 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:33:52.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.273233+0000", 2024-09-18T07:33:52.081 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011877799999999999 2024-09-18T07:33:52.081 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.081 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.081 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:52.081 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.274254+0000", 2024-09-18T07:33:52.081 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001020267 2024-09-18T07:33:52.081 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.081 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.081 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.081 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.275840+0000", 2024-09-18T07:33:52.081 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0015865849999999999 2024-09-18T07:33:52.081 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.081 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.081 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.319797+0000", 2024-09-18T07:33:52.082 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.043956489000000001 2024-09-18T07:33:52.082 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.082 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.082 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.319846+0000", 2024-09-18T07:33:52.082 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9602999999999999e-05 2024-09-18T07:33:52.082 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.082 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.082 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.319886+0000", 2024-09-18T07:33:52.082 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9780000000000002e-05 2024-09-18T07:33:52.082 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.082 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.083 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.083 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.083 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22961 7.5 7:ae8e5e68:::benchmark_data_smithi137_74180_object22960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:33:52.083 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.278834+0000", 2024-09-18T07:33:52.083 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 86.710813907000002, 2024-09-18T07:33:52.083 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.044950704000000001, 2024-09-18T07:33:52.083 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.083 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.083 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.083 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:52.083 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:52.083 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:52.084 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22961 2024-09-18T07:33:52.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.084 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.084 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.084 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.084 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.278834+0000", 2024-09-18T07:33:52.084 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.084 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.084 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.084 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.278834+0000", 2024-09-18T07:33:52.084 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.085 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.085 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.085 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.278836+0000", 2024-09-18T07:33:52.085 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.73e-06 2024-09-18T07:33:52.085 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.085 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.085 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.085 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.278887+0000", 2024-09-18T07:33:52.085 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1088000000000003e-05 2024-09-18T07:33:52.085 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.085 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.085 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.085 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.278899+0000", 2024-09-18T07:33:52.085 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1780000000000001e-05 2024-09-18T07:33:52.085 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.086 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.086 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.086 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.278904+0000", 2024-09-18T07:33:52.086 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4269999999999999e-06 2024-09-18T07:33:52.086 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.086 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.086 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.086 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.278969+0000", 2024-09-18T07:33:52.086 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4970999999999998e-05 2024-09-18T07:33:52.086 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.087 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.087 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.087 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.279084+0000", 2024-09-18T07:33:52.087 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000115318 2024-09-18T07:33:52.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.087 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.087 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:33:52.087 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.279211+0000", 2024-09-18T07:33:52.087 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000126742 2024-09-18T07:33:52.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.087 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:52.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.280057+0000", 2024-09-18T07:33:52.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00084567300000000002 2024-09-18T07:33:52.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.280326+0000", 2024-09-18T07:33:52.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00026896899999999998 2024-09-18T07:33:52.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.323690+0000", 2024-09-18T07:33:52.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.043363959000000001 2024-09-18T07:33:52.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.323744+0000", 2024-09-18T07:33:52.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4058000000000003e-05 2024-09-18T07:33:52.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.323785+0000", 2024-09-18T07:33:52.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0988999999999997e-05 2024-09-18T07:33:52.089 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.090 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.090 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.090 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.090 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:23078 7.5 7:af53196f:::benchmark_data_smithi137_74180_object23077:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:33:52.090 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.454467+0000", 2024-09-18T07:33:52.090 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 86.535180933000007, 2024-09-18T07:33:52.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.088575311000000004, 2024-09-18T07:33:52.090 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.090 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.090 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.091 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:52.091 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:52.091 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:52.091 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 23078 2024-09-18T07:33:52.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.091 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.091 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454467+0000", 2024-09-18T07:33:52.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.091 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454467+0000", 2024-09-18T07:33:52.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454469+0000", 2024-09-18T07:33:52.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6089999999999999e-06 2024-09-18T07:33:52.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454512+0000", 2024-09-18T07:33:52.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2608999999999998e-05 2024-09-18T07:33:52.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.093 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.093 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.093 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454525+0000", 2024-09-18T07:33:52.093 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3352e-05 2024-09-18T07:33:52.093 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.093 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.093 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.093 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454534+0000", 2024-09-18T07:33:52.093 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0560000000000005e-06 2024-09-18T07:33:52.093 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.093 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.093 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.093 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454591+0000", 2024-09-18T07:33:52.094 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7633999999999997e-05 2024-09-18T07:33:52.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.094 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.094 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.094 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454715+0000", 2024-09-18T07:33:52.094 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012419300000000001 2024-09-18T07:33:52.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.094 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.094 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:33:52.094 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454817+0000", 2024-09-18T07:33:52.094 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000101452 2024-09-18T07:33:52.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.095 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:52.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.455733+0000", 2024-09-18T07:33:52.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00091630599999999998 2024-09-18T07:33:52.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.095 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.456978+0000", 2024-09-18T07:33:52.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001245248 2024-09-18T07:33:52.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.095 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.542970+0000", 2024-09-18T07:33:52.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.085991206000000001 2024-09-18T07:33:52.096 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.096 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.096 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.096 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543010+0000", 2024-09-18T07:33:52.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0936999999999998e-05 2024-09-18T07:33:52.096 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.096 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.096 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.096 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543042+0000", 2024-09-18T07:33:52.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1708999999999997e-05 2024-09-18T07:33:52.096 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.096 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.096 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.097 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.097 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:23089 7.5 7:abb1cc32:::benchmark_data_smithi137_74180_object23088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:33:52.097 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.457769+0000", 2024-09-18T07:33:52.097 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 86.531878954999996, 2024-09-18T07:33:52.097 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.085838029999999996, 2024-09-18T07:33:52.097 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.097 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.097 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.097 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:52.097 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:52.097 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:52.097 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 23089 2024-09-18T07:33:52.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.098 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.098 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.098 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.098 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457769+0000", 2024-09-18T07:33:52.098 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.098 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.098 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.098 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.098 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457769+0000", 2024-09-18T07:33:52.098 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.098 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.098 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.098 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.099 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457771+0000", 2024-09-18T07:33:52.099 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3199999999999998e-06 2024-09-18T07:33:52.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.099 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.099 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.099 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457831+0000", 2024-09-18T07:33:52.099 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9939999999999999e-05 2024-09-18T07:33:52.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.099 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.099 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.099 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457848+0000", 2024-09-18T07:33:52.099 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6682999999999999e-05 2024-09-18T07:33:52.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.099 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.099 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.100 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457856+0000", 2024-09-18T07:33:52.100 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0809999999999994e-06 2024-09-18T07:33:52.100 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.100 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.100 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.100 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457909+0000", 2024-09-18T07:33:52.100 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3143999999999998e-05 2024-09-18T07:33:52.100 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.100 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.100 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.100 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.458075+0000", 2024-09-18T07:33:52.100 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000166398 2024-09-18T07:33:52.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.101 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.101 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:33:52.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.458209+0000", 2024-09-18T07:33:52.101 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013364400000000001 2024-09-18T07:33:52.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.101 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.101 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:52.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.459558+0000", 2024-09-18T07:33:52.101 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0013487519999999999 2024-09-18T07:33:52.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.101 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.101 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.459987+0000", 2024-09-18T07:33:52.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00042892000000000001 2024-09-18T07:33:52.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.102 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543512+0000", 2024-09-18T07:33:52.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.083524980999999998 2024-09-18T07:33:52.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.102 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543569+0000", 2024-09-18T07:33:52.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7337999999999999e-05 2024-09-18T07:33:52.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.102 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543607+0000", 2024-09-18T07:33:52.103 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7829e-05 2024-09-18T07:33:52.103 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.103 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.103 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.103 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.103 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:23145 7.5 7:a67bcaca:::benchmark_data_smithi137_74180_object23144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:33:52.103 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.558406+0000", 2024-09-18T07:33:52.103 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 86.431242256999994, 2024-09-18T07:33:52.103 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.040502491000000002, 2024-09-18T07:33:52.103 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.103 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.103 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.103 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:52.103 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:52.104 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:52.104 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 23145 2024-09-18T07:33:52.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.104 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.104 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558406+0000", 2024-09-18T07:33:52.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.104 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558406+0000", 2024-09-18T07:33:52.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.105 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.105 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.105 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558408+0000", 2024-09-18T07:33:52.105 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1390000000000002e-06 2024-09-18T07:33:52.105 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.105 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.105 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.105 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558470+0000", 2024-09-18T07:33:52.105 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2756999999999996e-05 2024-09-18T07:33:52.105 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.105 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.105 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.105 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558493+0000", 2024-09-18T07:33:52.105 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2523000000000001e-05 2024-09-18T07:33:52.105 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.106 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.106 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.106 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558501+0000", 2024-09-18T07:33:52.106 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9289999999999998e-06 2024-09-18T07:33:52.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.106 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.106 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.106 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558565+0000", 2024-09-18T07:33:52.106 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4553000000000003e-05 2024-09-18T07:33:52.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.106 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.106 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.106 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558688+0000", 2024-09-18T07:33:52.106 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000122242 2024-09-18T07:33:52.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.107 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.107 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:33:52.107 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558812+0000", 2024-09-18T07:33:52.107 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012474599999999999 2024-09-18T07:33:52.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.107 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.107 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:52.107 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.559762+0000", 2024-09-18T07:33:52.107 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000949105 2024-09-18T07:33:52.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.107 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.107 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.107 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.560044+0000", 2024-09-18T07:33:52.108 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00028216800000000001 2024-09-18T07:33:52.108 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.108 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.108 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.108 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.598816+0000", 2024-09-18T07:33:52.108 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.038772136999999998 2024-09-18T07:33:52.108 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.108 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.108 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.108 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.598869+0000", 2024-09-18T07:33:52.108 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2858000000000001e-05 2024-09-18T07:33:52.108 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.109 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.109 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.109 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.598908+0000", 2024-09-18T07:33:52.109 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9334e-05 2024-09-18T07:33:52.109 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.109 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.109 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.109 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.109 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.109 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:26891 7.5 7:a92f246c:::benchmark_data_smithi137_74180_object26890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-18T07:33:52.109 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:27.401622+0000", 2024-09-18T07:33:52.109 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 84.588025725999998, 2024-09-18T07:33:52.109 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.080096663999999998, 2024-09-18T07:33:52.109 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.110 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.110 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.110 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:52.110 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:52.112 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:52.112 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 26891 2024-09-18T07:33:52.112 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.112 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.112 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.112 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.112 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401622+0000", 2024-09-18T07:33:52.112 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.112 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.113 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.113 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.113 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401622+0000", 2024-09-18T07:33:52.113 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.113 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.113 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.113 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.113 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401624+0000", 2024-09-18T07:33:52.113 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0880000000000002e-06 2024-09-18T07:33:52.113 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.113 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.113 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.113 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401694+0000", 2024-09-18T07:33:52.113 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9825000000000003e-05 2024-09-18T07:33:52.114 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.114 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.114 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.114 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401708+0000", 2024-09-18T07:33:52.114 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4249e-05 2024-09-18T07:33:52.114 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.114 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.114 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.114 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401716+0000", 2024-09-18T07:33:52.114 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0460000000000006e-06 2024-09-18T07:33:52.114 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.114 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.114 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.115 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401785+0000", 2024-09-18T07:33:52.115 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8583000000000004e-05 2024-09-18T07:33:52.115 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.115 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.115 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.115 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401892+0000", 2024-09-18T07:33:52.115 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000106885 2024-09-18T07:33:52.115 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.115 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.115 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:33:52.115 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402022+0000", 2024-09-18T07:33:52.115 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012980700000000001 2024-09-18T07:33:52.115 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.115 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.115 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:52.115 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.404167+0000", 2024-09-18T07:33:52.116 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021457550000000001 2024-09-18T07:33:52.116 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.116 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.116 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.116 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.404212+0000", 2024-09-18T07:33:52.116 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5052000000000002e-05 2024-09-18T07:33:52.116 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.116 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.116 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.116 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481614+0000", 2024-09-18T07:33:52.116 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.077401247000000006 2024-09-18T07:33:52.116 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.116 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.116 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.117 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481666+0000", 2024-09-18T07:33:52.117 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2198999999999999e-05 2024-09-18T07:33:52.117 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.117 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.117 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.117 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481719+0000", 2024-09-18T07:33:52.117 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2927999999999999e-05 2024-09-18T07:33:52.117 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.117 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.117 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.117 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.117 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.117 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:26893 7.5 7:a23e6cde:::benchmark_data_smithi137_74180_object26892:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-18T07:33:52.117 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:27.402662+0000", 2024-09-18T07:33:52.118 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 84.586985530000007, 2024-09-18T07:33:52.118 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.079194041000000007, 2024-09-18T07:33:52.118 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:33:52.118 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:33:52.118 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:33:52.118 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:33:52.118 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:33:52.118 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:33:52.118 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 26893 2024-09-18T07:33:52.118 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.118 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:33:52.118 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.118 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:33:52.118 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402662+0000", 2024-09-18T07:33:52.119 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.119 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.119 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.119 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:33:52.119 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402662+0000", 2024-09-18T07:33:52.119 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:33:52.119 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.119 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.119 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:33:52.119 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402664+0000", 2024-09-18T07:33:52.119 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.513e-06 2024-09-18T07:33:52.119 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.119 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.119 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:33:52.120 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402694+0000", 2024-09-18T07:33:52.120 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9984e-05 2024-09-18T07:33:52.120 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.120 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.120 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:33:52.120 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402706+0000", 2024-09-18T07:33:52.120 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2499e-05 2024-09-18T07:33:52.120 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.120 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.120 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:33:52.120 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402713+0000", 2024-09-18T07:33:52.120 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6429999999999998e-06 2024-09-18T07:33:52.120 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.120 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.120 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:33:52.121 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402959+0000", 2024-09-18T07:33:52.121 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024601700000000001 2024-09-18T07:33:52.121 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.121 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.121 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:33:52.121 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.403094+0000", 2024-09-18T07:33:52.121 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013474299999999999 2024-09-18T07:33:52.121 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.121 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.121 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:33:52.121 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.403289+0000", 2024-09-18T07:33:52.121 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019528500000000001 2024-09-18T07:33:52.121 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.121 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.121 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:33:52.122 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.404306+0000", 2024-09-18T07:33:52.122 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010166179999999999 2024-09-18T07:33:52.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.122 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.122 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.122 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.405112+0000", 2024-09-18T07:33:52.122 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00080637099999999995 2024-09-18T07:33:52.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.122 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.122 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:33:52.122 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481779+0000", 2024-09-18T07:33:52.122 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.076667400999999996 2024-09-18T07:33:52.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.122 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.122 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:33:52.122 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481825+0000", 2024-09-18T07:33:52.123 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5652000000000003e-05 2024-09-18T07:33:52.123 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:33:52.123 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:33:52.123 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:33:52.123 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481856+0000", 2024-09-18T07:33:52.123 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1315000000000001e-05 2024-09-18T07:33:52.123 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.123 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.123 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.123 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:33:52.123 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:33:52.123 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:33:52.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:52.124 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:52.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:52.225 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:52.250 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:52.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:52.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:52.351 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:52.376 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:52.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:52.478 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:52.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:52.524 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:52.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:52.625 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:52.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:52.653 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:52.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:52.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:52 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:33:52.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:52 smithi203 ceph-mon[32237]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:33:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:52 smithi203 ceph-mon[32237]: osdmap e128: 8 total, 8 up, 7 in 2024-09-18T07:33:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:52 smithi203 ceph-mon[32237]: pgmap v131: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 767 B/s rd, 34 MiB/s wr, 838 op/s 2024-09-18T07:33:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:52 smithi203 ceph-mon[32237]: 6.19 scrub starts 2024-09-18T07:33:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:52 smithi203 ceph-mon[32237]: 6.19 scrub ok 2024-09-18T07:33:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:52 smithi203 ceph-mon[32237]: from='client.24913 v1:172.21.15.137:0/1081126756' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:33:52.729 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:52 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:33:52.729 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:52 smithi137 ceph-mon[29408]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:33:52.729 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:52 smithi137 ceph-mon[29408]: osdmap e128: 8 total, 8 up, 7 in 2024-09-18T07:33:52.729 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:52 smithi137 ceph-mon[29408]: pgmap v131: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 767 B/s rd, 34 MiB/s wr, 838 op/s 2024-09-18T07:33:52.730 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:52 smithi137 ceph-mon[29408]: 6.19 scrub starts 2024-09-18T07:33:52.730 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:52 smithi137 ceph-mon[29408]: 6.19 scrub ok 2024-09-18T07:33:52.730 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:52 smithi137 ceph-mon[29408]: from='client.24913 v1:172.21.15.137:0/1081126756' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:33:52.730 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:52 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:33:52.730 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:52 smithi137 ceph-mon[39175]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:33:52.730 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:52 smithi137 ceph-mon[39175]: osdmap e128: 8 total, 8 up, 7 in 2024-09-18T07:33:52.730 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:52 smithi137 ceph-mon[39175]: pgmap v131: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 767 B/s rd, 34 MiB/s wr, 838 op/s 2024-09-18T07:33:52.730 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:52 smithi137 ceph-mon[39175]: 6.19 scrub starts 2024-09-18T07:33:52.730 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:52 smithi137 ceph-mon[39175]: 6.19 scrub ok 2024-09-18T07:33:52.730 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:52 smithi137 ceph-mon[39175]: from='client.24913 v1:172.21.15.137:0/1081126756' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:33:52.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:52.794 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:52.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:52.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:33:52.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:52.969 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:52.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:53.070 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:53.094 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:53.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:53.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:53.196 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:53.240 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:53.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:53.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:53.332 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-09-18T07:33:51.445694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6689200691,"num_objects":102260,"num_object_clones":0,"num_object_copies":306780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102260,"num_whiteouts":0,"num_read":907,"num_read_kb":766,"num_write":107316,"num_write_kb":6684106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9794,"num_bytes_recovered":636423665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79338,"ondisk_log_size":79338,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":20320848,"kb_used_data":19688656,"kb_used_omap":10,"kb_used_meta":631861,"kb_avail":635895216,"statfs":{"total":671965249536,"available":651156701184,"internally_reserved":0,"allocated":20161183744,"data_stored":20158291062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11162,"internal_metadata":647025766},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":131006488,"num_objects":2000,"num_object_clones":0,"num_object_copies":6000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2000,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":6706,"num_write_kb":278529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002322"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":152,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.379991+0000","last_change":"2024-09-18T07:33:31.183001+0000","last_active":"2024-09-18T07:33:44.379991+0000","last_peered":"2024-09-18T07:33:44.379991+0000","last_clean":"2024-09-18T07:33:44.379991+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:33:44.379991+0000","last_undegraded":"2024-09-18T07:33:44.379991+0000","last_fullsized":"2024-09-18T07:33:44.379991+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:31.182913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:31.182913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:44.795464+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":184,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.405516+0000","last_change":"2024-09-18T07:33:29.685825+0000","last_active":"2024-09-18T07:33:51.405516+0000","last_peered":"2024-09-18T07:33:51.405516+0000","last_clean":"2024-09-18T07:33:51.405516+0000","last_became_active":"2024-09-18T07:32:26.785727+0000","last_became_peered":"2024-09-18T07:32:26.785727+0000","last_unstale":"2024-09-18T07:33:51.405516+0000","last_undegraded":"2024-09-18T07:33:51.405516+0000","last_fullsized":"2024-09-18T07:33:51.405516+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:33:29.685767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.685767+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:33.086673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":152,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.365750+0000","last_change":"2024-09-18T07:32:31.030879+0000","last_active":"2024-09-18T07:33:44.365750+0000","last_peered":"2024-09-18T07:33:44.365750+0000","last_clean":"2024-09-18T07:33:44.365750+0000","last_became_active":"2024-09-18T07:32:30.831949+0000","last_became_peered":"2024-09-18T07:32:30.831949+0000","last_unstale":"2024-09-18T07:33:44.365750+0000","last_undegraded":"2024-09-18T07:33:44.365750+0000","last_fullsized":"2024-09-18T07:33:44.365750+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:31.030835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:31.030835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:55.282052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":143,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.405618+0000","last_change":"2024-09-18T07:32:03.953881+0000","last_active":"2024-09-18T07:33:51.405618+0000","last_peered":"2024-09-18T07:33:51.405618+0000","last_clean":"2024-09-18T07:33:51.405618+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:33:51.405618+0000","last_undegraded":"2024-09-18T07:33:51.405618+0000","last_fullsized":"2024-09-18T07:33:51.405618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.953827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.953827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":26,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.376131+0000","last_change":"2024-09-18T07:33:27.004713+0000","last_active":"2024-09-18T07:33:44.376131+0000","last_peered":"2024-09-18T07:33:44.376131+0000","last_clean":"2024-09-18T07:33:44.376131+0000","last_became_active":"2024-09-18T07:33:27.004385+0000","last_became_peered":"2024-09-18T07:33:27.004385+0000","last_unstale":"2024-09-18T07:33:44.376131+0000","last_undegraded":"2024-09-18T07:33:44.376131+0000","last_fullsized":"2024-09-18T07:33:44.376131+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:39.137803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":152,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.366644+0000","last_change":"2024-09-18T07:33:27.006545+0000","last_active":"2024-09-18T07:33:44.366644+0000","last_peered":"2024-09-18T07:33:44.366644+0000","last_clean":"2024-09-18T07:33:44.366644+0000","last_became_active":"2024-09-18T07:30:56.286040+0000","last_became_peered":"2024-09-18T07:30:56.286040+0000","last_unstale":"2024-09-18T07:33:44.366644+0000","last_undegraded":"2024-09-18T07:33:44.366644+0000","last_fullsized":"2024-09-18T07:33:44.366644+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.006488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.006488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.614689+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":173,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.363374+0000","last_change":"2024-09-18T07:33:22.032321+0000","last_active":"2024-09-18T07:33:44.363374+0000","last_peered":"2024-09-18T07:33:44.363374+0000","last_clean":"2024-09-18T07:33:44.363374+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:33:44.363374+0000","last_undegraded":"2024-09-18T07:33:44.363374+0000","last_fullsized":"2024-09-18T07:33:44.363374+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:22.032110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.032110+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.263345+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":139,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.405398+0000","last_change":"2024-09-18T07:31:00.331498+0000","last_active":"2024-09-18T07:33:51.405398+0000","last_peered":"2024-09-18T07:33:51.405398+0000","last_clean":"2024-09-18T07:33:51.405398+0000","last_became_active":"2024-09-18T07:31:00.331224+0000","last_became_peered":"2024-09-18T07:31:00.331224+0000","last_unstale":"2024-09-18T07:33:51.405398+0000","last_undegraded":"2024-09-18T07:33:51.405398+0000","last_fullsized":"2024-09-18T07:33:51.405398+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":154,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.417278+0000","last_change":"2024-09-18T07:33:21.338716+0000","last_active":"2024-09-18T07:33:51.417278+0000","last_peered":"2024-09-18T07:33:51.417278+0000","last_clean":"2024-09-18T07:33:51.417278+0000","last_became_active":"2024-09-18T07:30:56.285882+0000","last_became_peered":"2024-09-18T07:30:56.285882+0000","last_unstale":"2024-09-18T07:33:51.417278+0000","last_undegraded":"2024-09-18T07:33:51.417278+0000","last_fullsized":"2024-09-18T07:33:51.417278+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.338669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.338669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:26.027193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":167,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.405919+0000","last_change":"2024-09-18T07:32:02.002919+0000","last_active":"2024-09-18T07:33:51.405919+0000","last_peered":"2024-09-18T07:33:51.405919+0000","last_clean":"2024-09-18T07:33:51.405919+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:33:51.405919+0000","last_undegraded":"2024-09-18T07:33:51.405919+0000","last_fullsized":"2024-09-18T07:33:51.405919+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.002811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.002811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":145,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.416930+0000","last_change":"2024-09-18T07:32:30.842423+0000","last_active":"2024-09-18T07:33:51.416930+0000","last_peered":"2024-09-18T07:33:51.416930+0000","last_clean":"2024-09-18T07:33:51.416930+0000","last_became_active":"2024-09-18T07:32:30.841963+0000","last_became_peered":"2024-09-18T07:32:30.841963+0000","last_unstale":"2024-09-18T07:33:51.416930+0000","last_undegraded":"2024-09-18T07:33:51.416930+0000","last_fullsized":"2024-09-18T07:33:51.416930+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.385503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.373388+0000","last_change":"2024-09-18T07:32:30.840094+0000","last_active":"2024-09-18T07:33:44.373388+0000","last_peered":"2024-09-18T07:33:44.373388+0000","last_clean":"2024-09-18T07:33:44.373388+0000","last_became_active":"2024-09-18T07:32:30.839724+0000","last_became_peered":"2024-09-18T07:32:30.839724+0000","last_unstale":"2024-09-18T07:33:44.373388+0000","last_undegraded":"2024-09-18T07:33:44.373388+0000","last_fullsized":"2024-09-18T07:33:44.373388+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":146,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.416804+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:33:51.416804+0000","last_peered":"2024-09-18T07:33:51.416804+0000","last_clean":"2024-09-18T07:33:51.416804+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:33:51.416804+0000","last_undegraded":"2024-09-18T07:33:51.416804+0000","last_fullsized":"2024-09-18T07:33:51.416804+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:15.347368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":153,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.405761+0000","last_change":"2024-09-18T07:30:58.317041+0000","last_active":"2024-09-18T07:33:51.405761+0000","last_peered":"2024-09-18T07:33:51.405761+0000","last_clean":"2024-09-18T07:33:51.405761+0000","last_became_active":"2024-09-18T07:30:58.316479+0000","last_became_peered":"2024-09-18T07:30:58.316479+0000","last_unstale":"2024-09-18T07:33:51.405761+0000","last_undegraded":"2024-09-18T07:33:51.405761+0000","last_fullsized":"2024-09-18T07:33:51.405761+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:41.690380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":81,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.405513+0000","last_change":"2024-09-18T07:32:30.834493+0000","last_active":"2024-09-18T07:33:51.405513+0000","last_peered":"2024-09-18T07:33:51.405513+0000","last_clean":"2024-09-18T07:33:51.405513+0000","last_became_active":"2024-09-18T07:32:30.834192+0000","last_became_peered":"2024-09-18T07:32:30.834192+0000","last_unstale":"2024-09-18T07:33:51.405513+0000","last_undegraded":"2024-09-18T07:33:51.405513+0000","last_fullsized":"2024-09-18T07:33:51.405513+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.897464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":142,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.416371+0000","last_change":"2024-09-18T07:32:30.107805+0000","last_active":"2024-09-18T07:33:51.416371+0000","last_peered":"2024-09-18T07:33:51.416371+0000","last_clean":"2024-09-18T07:33:51.416371+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:33:51.416371+0000","last_undegraded":"2024-09-18T07:33:51.416371+0000","last_fullsized":"2024-09-18T07:33:51.416371+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:30.107745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:30.107745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":143,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.406429+0000","last_change":"2024-09-18T07:32:06.976159+0000","last_active":"2024-09-18T07:33:51.406429+0000","last_peered":"2024-09-18T07:33:51.406429+0000","last_clean":"2024-09-18T07:33:51.406429+0000","last_became_active":"2024-09-18T07:31:02.490014+0000","last_became_peered":"2024-09-18T07:31:02.490014+0000","last_unstale":"2024-09-18T07:33:51.406429+0000","last_undegraded":"2024-09-18T07:33:51.406429+0000","last_fullsized":"2024-09-18T07:33:51.406429+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:06.976113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:06.976113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.376746+0000","last_change":"2024-09-18T07:32:30.840267+0000","last_active":"2024-09-18T07:33:44.376746+0000","last_peered":"2024-09-18T07:33:44.376746+0000","last_clean":"2024-09-18T07:33:44.376746+0000","last_became_active":"2024-09-18T07:32:30.839952+0000","last_became_peered":"2024-09-18T07:32:30.839952+0000","last_unstale":"2024-09-18T07:33:44.376746+0000","last_undegraded":"2024-09-18T07:33:44.376746+0000","last_fullsized":"2024-09-18T07:33:44.376746+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":174,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.372275+0000","last_change":"2024-09-18T07:32:30.452884+0000","last_active":"2024-09-18T07:33:44.372275+0000","last_peered":"2024-09-18T07:33:44.372275+0000","last_clean":"2024-09-18T07:33:44.372275+0000","last_became_active":"2024-09-18T07:30:58.316266+0000","last_became_peered":"2024-09-18T07:30:58.316266+0000","last_unstale":"2024-09-18T07:33:44.372275+0000","last_undegraded":"2024-09-18T07:33:44.372275+0000","last_fullsized":"2024-09-18T07:33:44.372275+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:32:30.452816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:30.452816+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":145,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.367955+0000","last_change":"2024-09-18T07:33:30.955441+0000","last_active":"2024-09-18T07:33:44.367955+0000","last_peered":"2024-09-18T07:33:44.367955+0000","last_clean":"2024-09-18T07:33:44.367955+0000","last_became_active":"2024-09-18T07:31:00.337474+0000","last_became_peered":"2024-09-18T07:31:00.337474+0000","last_unstale":"2024-09-18T07:33:44.367955+0000","last_undegraded":"2024-09-18T07:33:44.367955+0000","last_fullsized":"2024-09-18T07:33:44.367955+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.955213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.955213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.470164+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.369014+0000","last_change":"2024-09-18T07:33:26.004195+0000","last_active":"2024-09-18T07:33:44.369014+0000","last_peered":"2024-09-18T07:33:44.369014+0000","last_clean":"2024-09-18T07:33:44.369014+0000","last_became_active":"2024-09-18T07:31:02.492348+0000","last_became_peered":"2024-09-18T07:31:02.492348+0000","last_unstale":"2024-09-18T07:33:44.369014+0000","last_undegraded":"2024-09-18T07:33:44.369014+0000","last_fullsized":"2024-09-18T07:33:44.369014+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:26.003425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:26.003425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:38.529586+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":147,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.406607+0000","last_change":"2024-09-18T07:30:56.289489+0000","last_active":"2024-09-18T07:33:51.406607+0000","last_peered":"2024-09-18T07:33:51.406607+0000","last_clean":"2024-09-18T07:33:51.406607+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:33:51.406607+0000","last_undegraded":"2024-09-18T07:33:51.406607+0000","last_fullsized":"2024-09-18T07:33:51.406607+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":163,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.363234+0000","last_change":"2024-09-18T07:30:58.319115+0000","last_active":"2024-09-18T07:33:44.363234+0000","last_peered":"2024-09-18T07:33:44.363234+0000","last_clean":"2024-09-18T07:33:44.363234+0000","last_became_active":"2024-09-18T07:30:58.316380+0000","last_became_peered":"2024-09-18T07:30:58.316380+0000","last_unstale":"2024-09-18T07:33:44.363234+0000","last_undegraded":"2024-09-18T07:33:44.363234+0000","last_fullsized":"2024-09-18T07:33:44.363234+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.013089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":139,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.405921+0000","last_change":"2024-09-18T07:31:00.334313+0000","last_active":"2024-09-18T07:33:51.405921+0000","last_peered":"2024-09-18T07:33:51.405921+0000","last_clean":"2024-09-18T07:33:51.405921+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:33:51.405921+0000","last_undegraded":"2024-09-18T07:33:51.405921+0000","last_fullsized":"2024-09-18T07:33:51.405921+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.867220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":105,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.416614+0000","last_change":"2024-09-18T07:33:33.565391+0000","last_active":"2024-09-18T07:33:51.416614+0000","last_peered":"2024-09-18T07:33:51.416614+0000","last_clean":"2024-09-18T07:33:51.416614+0000","last_became_active":"2024-09-18T07:32:30.840386+0000","last_became_peered":"2024-09-18T07:32:30.840386+0000","last_unstale":"2024-09-18T07:33:51.416614+0000","last_undegraded":"2024-09-18T07:33:51.416614+0000","last_fullsized":"2024-09-18T07:33:51.416614+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:33.565334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.565334+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:02.676072+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":162,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.365871+0000","last_change":"2024-09-18T07:33:32.976203+0000","last_active":"2024-09-18T07:33:44.365871+0000","last_peered":"2024-09-18T07:33:44.365871+0000","last_clean":"2024-09-18T07:33:44.365871+0000","last_became_active":"2024-09-18T07:30:56.283705+0000","last_became_peered":"2024-09-18T07:30:56.283705+0000","last_unstale":"2024-09-18T07:33:44.365871+0000","last_undegraded":"2024-09-18T07:33:44.365871+0000","last_fullsized":"2024-09-18T07:33:44.365871+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:32.976129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.976129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:54.706133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":182,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.377123+0000","last_change":"2024-09-18T07:33:36.118423+0000","last_active":"2024-09-18T07:33:44.377123+0000","last_peered":"2024-09-18T07:33:44.377123+0000","last_clean":"2024-09-18T07:33:44.377123+0000","last_became_active":"2024-09-18T07:32:30.860930+0000","last_became_peered":"2024-09-18T07:32:30.860930+0000","last_unstale":"2024-09-18T07:33:44.377123+0000","last_undegraded":"2024-09-18T07:33:44.377123+0000","last_fullsized":"2024-09-18T07:33:44.377123+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:33:36.118340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.118340+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:36.957408+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"94'9","reported_seq":196,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.415502+0000","last_change":"2024-09-18T07:32:30.853596+0000","last_active":"2024-09-18T07:33:51.415502+0000","last_peered":"2024-09-18T07:33:51.415502+0000","last_clean":"2024-09-18T07:33:51.415502+0000","last_became_active":"2024-09-18T07:32:30.839828+0000","last_became_peered":"2024-09-18T07:32:30.839828+0000","last_unstale":"2024-09-18T07:33:51.415502+0000","last_undegraded":"2024-09-18T07:33:51.415502+0000","last_fullsized":"2024-09-18T07:33:51.415502+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:58.766416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"127'6857","reported_seq":10915,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.414725+0000","last_change":"2024-09-18T07:33:32.018562+0000","last_active":"2024-09-18T07:33:51.414725+0000","last_peered":"2024-09-18T07:33:51.414725+0000","last_clean":"2024-09-18T07:33:51.414725+0000","last_became_active":"2024-09-18T07:32:54.163740+0000","last_became_peered":"2024-09-18T07:32:54.163740+0000","last_unstale":"2024-09-18T07:33:51.414725+0000","last_undegraded":"2024-09-18T07:33:51.414725+0000","last_fullsized":"2024-09-18T07:33:51.414725+0000","mapping_epoch":102,"log_start":"88'1300","ondisk_log_start":"88'1300","created":81,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'5666","last_scrub_stamp":"2024-09-18T07:33:32.018370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.018370+0000","objects_scrubbed":5641,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:57.391560+0000","scrub_duration":1289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400883712,"num_objects":6117,"num_object_clones":0,"num_object_copies":18351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6857,"num_write_kb":415168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":127074304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":166,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.366155+0000","last_change":"2024-09-18T07:30:58.312916+0000","last_active":"2024-09-18T07:33:44.366155+0000","last_peered":"2024-09-18T07:33:44.366155+0000","last_clean":"2024-09-18T07:33:44.366155+0000","last_became_active":"2024-09-18T07:30:58.312627+0000","last_became_peered":"2024-09-18T07:30:58.312627+0000","last_unstale":"2024-09-18T07:33:44.366155+0000","last_undegraded":"2024-09-18T07:33:44.366155+0000","last_fullsized":"2024-09-18T07:33:44.366155+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:15.056322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":159,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.370794+0000","last_change":"2024-09-18T07:32:30.838798+0000","last_active":"2024-09-18T07:33:44.370794+0000","last_peered":"2024-09-18T07:33:44.370794+0000","last_clean":"2024-09-18T07:33:44.370794+0000","last_became_active":"2024-09-18T07:32:30.838348+0000","last_became_peered":"2024-09-18T07:32:30.838348+0000","last_unstale":"2024-09-18T07:33:44.370794+0000","last_undegraded":"2024-09-18T07:33:44.370794+0000","last_fullsized":"2024-09-18T07:33:44.370794+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:56.565412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:56.565412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":191,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.406510+0000","last_change":"2024-09-18T07:33:33.706253+0000","last_active":"2024-09-18T07:33:51.406510+0000","last_peered":"2024-09-18T07:33:51.406510+0000","last_clean":"2024-09-18T07:33:51.406510+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:33:51.406510+0000","last_undegraded":"2024-09-18T07:33:51.406510+0000","last_fullsized":"2024-09-18T07:33:51.406510+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:33:33.706201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.706201+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.605129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":134,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.416363+0000","last_change":"2024-09-18T07:31:02.351412+0000","last_active":"2024-09-18T07:33:51.416363+0000","last_peered":"2024-09-18T07:33:51.416363+0000","last_clean":"2024-09-18T07:33:51.416363+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:33:51.416363+0000","last_undegraded":"2024-09-18T07:33:51.416363+0000","last_fullsized":"2024-09-18T07:33:51.416363+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.333040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"127'6488","reported_seq":6590,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:46.507884+0000","last_change":"2024-09-18T07:32:08.065129+0000","last_active":"2024-09-18T07:33:46.507884+0000","last_peered":"2024-09-18T07:33:46.507884+0000","last_clean":"2024-09-18T07:33:46.507884+0000","last_became_active":"2024-09-18T07:32:08.064876+0000","last_became_peered":"2024-09-18T07:32:08.064876+0000","last_unstale":"2024-09-18T07:33:46.507884+0000","last_undegraded":"2024-09-18T07:33:46.507884+0000","last_fullsized":"2024-09-18T07:33:46.507884+0000","mapping_epoch":81,"log_start":"96'2300","ondisk_log_start":"96'2300","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4188,"log_dups_size":0,"ondisk_log_size":4188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419561472,"num_objects":6402,"num_object_clones":0,"num_object_copies":19206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6488,"num_write_kb":412480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":205,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.370573+0000","last_change":"2024-09-18T07:32:30.876643+0000","last_active":"2024-09-18T07:33:44.370573+0000","last_peered":"2024-09-18T07:33:44.370573+0000","last_clean":"2024-09-18T07:33:44.370573+0000","last_became_active":"2024-09-18T07:32:30.852501+0000","last_became_peered":"2024-09-18T07:32:30.852501+0000","last_unstale":"2024-09-18T07:33:44.370573+0000","last_undegraded":"2024-09-18T07:33:44.370573+0000","last_fullsized":"2024-09-18T07:33:44.370573+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:32:00.660377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:00.660377+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":159,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.369635+0000","last_change":"2024-09-18T07:32:30.858438+0000","last_active":"2024-09-18T07:33:44.369635+0000","last_peered":"2024-09-18T07:33:44.369635+0000","last_clean":"2024-09-18T07:33:44.369635+0000","last_became_active":"2024-09-18T07:32:30.858176+0000","last_became_peered":"2024-09-18T07:32:30.858176+0000","last_unstale":"2024-09-18T07:33:44.369635+0000","last_undegraded":"2024-09-18T07:33:44.369635+0000","last_fullsized":"2024-09-18T07:33:44.369635+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.665591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.665591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":146,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.405673+0000","last_change":"2024-09-18T07:32:30.851490+0000","last_active":"2024-09-18T07:33:51.405673+0000","last_peered":"2024-09-18T07:33:51.405673+0000","last_clean":"2024-09-18T07:33:51.405673+0000","last_became_active":"2024-09-18T07:32:30.851161+0000","last_became_peered":"2024-09-18T07:32:30.851161+0000","last_unstale":"2024-09-18T07:33:51.405673+0000","last_undegraded":"2024-09-18T07:33:51.405673+0000","last_fullsized":"2024-09-18T07:33:51.405673+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:47.861786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":135,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.405832+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:33:51.405832+0000","last_peered":"2024-09-18T07:33:51.405832+0000","last_clean":"2024-09-18T07:33:51.405832+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:33:51.405832+0000","last_undegraded":"2024-09-18T07:33:51.405832+0000","last_fullsized":"2024-09-18T07:33:51.405832+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"127'6549","reported_seq":6651,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:46.417581+0000","last_change":"2024-09-18T07:32:08.066116+0000","last_active":"2024-09-18T07:33:46.417581+0000","last_peered":"2024-09-18T07:33:46.417581+0000","last_clean":"2024-09-18T07:33:46.417581+0000","last_became_active":"2024-09-18T07:32:08.065844+0000","last_became_peered":"2024-09-18T07:32:08.065844+0000","last_unstale":"2024-09-18T07:33:46.417581+0000","last_undegraded":"2024-09-18T07:33:46.417581+0000","last_fullsized":"2024-09-18T07:33:46.417581+0000","mapping_epoch":81,"log_start":"96'2300","ondisk_log_start":"96'2300","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4249,"log_dups_size":0,"ondisk_log_size":4249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423428096,"num_objects":6461,"num_object_clones":0,"num_object_copies":19383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6549,"num_write_kb":416320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":164,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.405767+0000","last_change":"2024-09-18T07:30:58.309938+0000","last_active":"2024-09-18T07:33:51.405767+0000","last_peered":"2024-09-18T07:33:51.405767+0000","last_clean":"2024-09-18T07:33:51.405767+0000","last_became_active":"2024-09-18T07:30:58.309649+0000","last_became_peered":"2024-09-18T07:30:58.309649+0000","last_unstale":"2024-09-18T07:33:51.405767+0000","last_undegraded":"2024-09-18T07:33:51.405767+0000","last_fullsized":"2024-09-18T07:33:51.405767+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":153,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.377365+0000","last_change":"2024-09-18T07:33:24.213432+0000","last_active":"2024-09-18T07:33:44.377365+0000","last_peered":"2024-09-18T07:33:44.377365+0000","last_clean":"2024-09-18T07:33:44.377365+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:33:44.377365+0000","last_undegraded":"2024-09-18T07:33:44.377365+0000","last_fullsized":"2024-09-18T07:33:44.377365+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:24.213356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:24.213356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:52.949578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":179,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.370377+0000","last_change":"2024-09-18T07:31:00.336059+0000","last_active":"2024-09-18T07:33:44.370377+0000","last_peered":"2024-09-18T07:33:44.370377+0000","last_clean":"2024-09-18T07:33:44.370377+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:33:44.370377+0000","last_undegraded":"2024-09-18T07:33:44.370377+0000","last_fullsized":"2024-09-18T07:33:44.370377+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:17.963381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":89,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.415160+0000","last_change":"2024-09-18T07:32:54.740291+0000","last_active":"2024-09-18T07:33:51.415160+0000","last_peered":"2024-09-18T07:33:51.415160+0000","last_clean":"2024-09-18T07:33:51.415160+0000","last_became_active":"2024-09-18T07:32:30.830346+0000","last_became_peered":"2024-09-18T07:32:30.830346+0000","last_unstale":"2024-09-18T07:33:51.415160+0000","last_undegraded":"2024-09-18T07:33:51.415160+0000","last_fullsized":"2024-09-18T07:33:51.415160+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:54.740249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:54.740249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.965276+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"128'6968","reported_seq":7073,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.405496+0000","last_change":"2024-09-18T07:32:08.062282+0000","last_active":"2024-09-18T07:33:51.405496+0000","last_peered":"2024-09-18T07:33:51.405496+0000","last_clean":"2024-09-18T07:33:51.405496+0000","last_became_active":"2024-09-18T07:32:08.061934+0000","last_became_peered":"2024-09-18T07:32:08.061934+0000","last_unstale":"2024-09-18T07:33:51.405496+0000","last_undegraded":"2024-09-18T07:33:51.405496+0000","last_fullsized":"2024-09-18T07:33:51.405496+0000","mapping_epoch":81,"log_start":"98'2600","ondisk_log_start":"98'2600","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4368,"log_dups_size":0,"ondisk_log_size":4368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409600000,"num_objects":6250,"num_object_clones":0,"num_object_copies":18750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6968,"num_write_kb":422976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":156,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.366620+0000","last_change":"2024-09-18T07:30:58.307470+0000","last_active":"2024-09-18T07:33:44.366620+0000","last_peered":"2024-09-18T07:33:44.366620+0000","last_clean":"2024-09-18T07:33:44.366620+0000","last_became_active":"2024-09-18T07:30:58.307035+0000","last_became_peered":"2024-09-18T07:30:58.307035+0000","last_unstale":"2024-09-18T07:33:44.366620+0000","last_undegraded":"2024-09-18T07:33:44.366620+0000","last_fullsized":"2024-09-18T07:33:44.366620+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:14.091632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":154,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.417394+0000","last_change":"2024-09-18T07:33:29.471394+0000","last_active":"2024-09-18T07:33:51.417394+0000","last_peered":"2024-09-18T07:33:51.417394+0000","last_clean":"2024-09-18T07:33:51.417394+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:33:51.417394+0000","last_undegraded":"2024-09-18T07:33:51.417394+0000","last_fullsized":"2024-09-18T07:33:51.417394+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:29.471330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.471330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:58.117832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":179,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.376376+0000","last_change":"2024-09-18T07:31:00.337241+0000","last_active":"2024-09-18T07:33:44.376376+0000","last_peered":"2024-09-18T07:33:44.376376+0000","last_clean":"2024-09-18T07:33:44.376376+0000","last_became_active":"2024-09-18T07:31:00.336905+0000","last_became_peered":"2024-09-18T07:31:00.336905+0000","last_unstale":"2024-09-18T07:33:44.376376+0000","last_undegraded":"2024-09-18T07:33:44.376376+0000","last_fullsized":"2024-09-18T07:33:44.376376+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":132,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.373527+0000","last_change":"2024-09-18T07:31:02.348538+0000","last_active":"2024-09-18T07:33:44.373527+0000","last_peered":"2024-09-18T07:33:44.373527+0000","last_clean":"2024-09-18T07:33:44.373527+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:33:44.373527+0000","last_undegraded":"2024-09-18T07:33:44.373527+0000","last_fullsized":"2024-09-18T07:33:44.373527+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.077245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"128'6833","reported_seq":6937,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.416108+0000","last_change":"2024-09-18T07:32:08.062584+0000","last_active":"2024-09-18T07:33:51.416108+0000","last_peered":"2024-09-18T07:33:51.416108+0000","last_clean":"2024-09-18T07:33:51.416108+0000","last_became_active":"2024-09-18T07:32:08.062129+0000","last_became_peered":"2024-09-18T07:32:08.062129+0000","last_unstale":"2024-09-18T07:33:51.416108+0000","last_undegraded":"2024-09-18T07:33:51.416108+0000","last_fullsized":"2024-09-18T07:33:51.416108+0000","mapping_epoch":81,"log_start":"92'1656","ondisk_log_start":"92'1656","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5177,"log_dups_size":0,"ondisk_log_size":5177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:50.732644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403374080,"num_objects":6155,"num_object_clones":0,"num_object_copies":18465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6833,"num_write_kb":415616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":201,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.376520+0000","last_change":"2024-09-18T07:32:30.859920+0000","last_active":"2024-09-18T07:33:44.376520+0000","last_peered":"2024-09-18T07:33:44.376520+0000","last_clean":"2024-09-18T07:33:44.376520+0000","last_became_active":"2024-09-18T07:32:30.842195+0000","last_became_peered":"2024-09-18T07:32:30.842195+0000","last_unstale":"2024-09-18T07:33:44.376520+0000","last_undegraded":"2024-09-18T07:33:44.376520+0000","last_fullsized":"2024-09-18T07:33:44.376520+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:18.104967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":145,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.367529+0000","last_change":"2024-09-18T07:30:56.284440+0000","last_active":"2024-09-18T07:33:44.367529+0000","last_peered":"2024-09-18T07:33:44.367529+0000","last_clean":"2024-09-18T07:33:44.367529+0000","last_became_active":"2024-09-18T07:30:56.284120+0000","last_became_peered":"2024-09-18T07:30:56.284120+0000","last_unstale":"2024-09-18T07:33:44.367529+0000","last_undegraded":"2024-09-18T07:33:44.367529+0000","last_fullsized":"2024-09-18T07:33:44.367529+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:06.088370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":146,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.414809+0000","last_change":"2024-09-18T07:33:21.834910+0000","last_active":"2024-09-18T07:33:51.414809+0000","last_peered":"2024-09-18T07:33:51.414809+0000","last_clean":"2024-09-18T07:33:51.414809+0000","last_became_active":"2024-09-18T07:31:00.334776+0000","last_became_peered":"2024-09-18T07:31:00.334776+0000","last_unstale":"2024-09-18T07:33:51.414809+0000","last_undegraded":"2024-09-18T07:33:51.414809+0000","last_fullsized":"2024-09-18T07:33:51.414809+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:24.513163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":133,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.365667+0000","last_change":"2024-09-18T07:31:02.490534+0000","last_active":"2024-09-18T07:33:44.365667+0000","last_peered":"2024-09-18T07:33:44.365667+0000","last_clean":"2024-09-18T07:33:44.365667+0000","last_became_active":"2024-09-18T07:31:02.490186+0000","last_became_peered":"2024-09-18T07:31:02.490186+0000","last_unstale":"2024-09-18T07:33:44.365667+0000","last_undegraded":"2024-09-18T07:33:44.365667+0000","last_fullsized":"2024-09-18T07:33:44.365667+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:19.074713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"127'6730","reported_seq":6896,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.414866+0000","last_change":"2024-09-18T07:33:31.181076+0000","last_active":"2024-09-18T07:33:51.414866+0000","last_peered":"2024-09-18T07:33:51.414866+0000","last_clean":"2024-09-18T07:33:51.414866+0000","last_became_active":"2024-09-18T07:32:08.064184+0000","last_became_peered":"2024-09-18T07:32:08.064184+0000","last_unstale":"2024-09-18T07:33:51.414866+0000","last_undegraded":"2024-09-18T07:33:51.414866+0000","last_fullsized":"2024-09-18T07:33:51.414866+0000","mapping_epoch":81,"log_start":"88'1100","ondisk_log_start":"88'1100","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'5435","last_scrub_stamp":"2024-09-18T07:33:31.180958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:33:31.180958+0000","objects_scrubbed":5388,"log_size":5630,"log_dups_size":0,"ondisk_log_size":5630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.901559+0000","scrub_duration":1433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392822784,"num_objects":5994,"num_object_clones":0,"num_object_copies":17982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6730,"num_write_kb":407168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":224,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.367435+0000","last_change":"2024-09-18T07:30:58.317477+0000","last_active":"2024-09-18T07:33:44.367435+0000","last_peered":"2024-09-18T07:33:44.367435+0000","last_clean":"2024-09-18T07:33:44.367435+0000","last_became_active":"2024-09-18T07:30:58.316812+0000","last_became_peered":"2024-09-18T07:30:58.316812+0000","last_unstale":"2024-09-18T07:33:44.367435+0000","last_undegraded":"2024-09-18T07:33:44.367435+0000","last_fullsized":"2024-09-18T07:33:44.367435+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:09.123487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":147,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.406376+0000","last_change":"2024-09-18T07:30:56.285353+0000","last_active":"2024-09-18T07:33:51.406376+0000","last_peered":"2024-09-18T07:33:51.406376+0000","last_clean":"2024-09-18T07:33:51.406376+0000","last_became_active":"2024-09-18T07:30:56.285032+0000","last_became_peered":"2024-09-18T07:30:56.285032+0000","last_unstale":"2024-09-18T07:33:51.406376+0000","last_undegraded":"2024-09-18T07:33:51.406376+0000","last_fullsized":"2024-09-18T07:33:51.406376+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":156,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.416757+0000","last_change":"2024-09-18T07:30:58.299308+0000","last_active":"2024-09-18T07:33:51.416757+0000","last_peered":"2024-09-18T07:33:51.416757+0000","last_clean":"2024-09-18T07:33:51.416757+0000","last_became_active":"2024-09-18T07:30:55.283292+0000","last_became_peered":"2024-09-18T07:30:55.283292+0000","last_unstale":"2024-09-18T07:33:51.416757+0000","last_undegraded":"2024-09-18T07:33:51.416757+0000","last_fullsized":"2024-09-18T07:33:51.416757+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:19.297747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":136,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.380717+0000","last_change":"2024-09-18T07:31:00.332738+0000","last_active":"2024-09-18T07:33:44.380717+0000","last_peered":"2024-09-18T07:33:44.380717+0000","last_clean":"2024-09-18T07:33:44.380717+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:33:44.380717+0000","last_undegraded":"2024-09-18T07:33:44.380717+0000","last_fullsized":"2024-09-18T07:33:44.380717+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.363676+0000","last_change":"2024-09-18T07:32:30.854643+0000","last_active":"2024-09-18T07:33:44.363676+0000","last_peered":"2024-09-18T07:33:44.363676+0000","last_clean":"2024-09-18T07:33:44.363676+0000","last_became_active":"2024-09-18T07:32:30.853087+0000","last_became_peered":"2024-09-18T07:32:30.853087+0000","last_unstale":"2024-09-18T07:33:44.363676+0000","last_undegraded":"2024-09-18T07:33:44.363676+0000","last_fullsized":"2024-09-18T07:33:44.363676+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"127'6611","reported_seq":6713,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:46.594771+0000","last_change":"2024-09-18T07:32:08.063002+0000","last_active":"2024-09-18T07:33:46.594771+0000","last_peered":"2024-09-18T07:33:46.594771+0000","last_clean":"2024-09-18T07:33:46.594771+0000","last_became_active":"2024-09-18T07:32:08.062715+0000","last_became_peered":"2024-09-18T07:32:08.062715+0000","last_unstale":"2024-09-18T07:33:46.594771+0000","last_undegraded":"2024-09-18T07:33:46.594771+0000","last_fullsized":"2024-09-18T07:33:46.594771+0000","mapping_epoch":81,"log_start":"95'2009","ondisk_log_start":"95'2009","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4602,"log_dups_size":0,"ondisk_log_size":4602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426967040,"num_objects":6515,"num_object_clones":0,"num_object_copies":19545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6611,"num_write_kb":420032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":168,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.366252+0000","last_change":"2024-09-18T07:30:58.310789+0000","last_active":"2024-09-18T07:33:44.366252+0000","last_peered":"2024-09-18T07:33:44.366252+0000","last_clean":"2024-09-18T07:33:44.366252+0000","last_became_active":"2024-09-18T07:30:58.310490+0000","last_became_peered":"2024-09-18T07:30:58.310490+0000","last_unstale":"2024-09-18T07:33:44.366252+0000","last_undegraded":"2024-09-18T07:33:44.366252+0000","last_fullsized":"2024-09-18T07:33:44.366252+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.090903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":153,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.365179+0000","last_change":"2024-09-18T07:33:23.038159+0000","last_active":"2024-09-18T07:33:44.365179+0000","last_peered":"2024-09-18T07:33:44.365179+0000","last_clean":"2024-09-18T07:33:44.365179+0000","last_became_active":"2024-09-18T07:30:56.284803+0000","last_became_peered":"2024-09-18T07:30:56.284803+0000","last_unstale":"2024-09-18T07:33:44.365179+0000","last_undegraded":"2024-09-18T07:33:44.365179+0000","last_fullsized":"2024-09-18T07:33:44.365179+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:23.037963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:23.037963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:45.045160+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":162,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.368384+0000","last_change":"2024-09-18T07:32:02.418590+0000","last_active":"2024-09-18T07:33:44.368384+0000","last_peered":"2024-09-18T07:33:44.368384+0000","last_clean":"2024-09-18T07:33:44.368384+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:33:44.368384+0000","last_undegraded":"2024-09-18T07:33:44.368384+0000","last_fullsized":"2024-09-18T07:33:44.368384+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:32:02.418507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.418507+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:39.933773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":144,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.373415+0000","last_change":"2024-09-18T07:32:30.837706+0000","last_active":"2024-09-18T07:33:44.373415+0000","last_peered":"2024-09-18T07:33:44.373415+0000","last_clean":"2024-09-18T07:33:44.373415+0000","last_became_active":"2024-09-18T07:32:30.837243+0000","last_became_peered":"2024-09-18T07:32:30.837243+0000","last_unstale":"2024-09-18T07:33:44.373415+0000","last_undegraded":"2024-09-18T07:33:44.373415+0000","last_fullsized":"2024-09-18T07:33:44.373415+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:09.666733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":105,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.415136+0000","last_change":"2024-09-18T07:32:55.796838+0000","last_active":"2024-09-18T07:33:51.415136+0000","last_peered":"2024-09-18T07:33:51.415136+0000","last_clean":"2024-09-18T07:33:51.415136+0000","last_became_active":"2024-09-18T07:32:30.839186+0000","last_became_peered":"2024-09-18T07:32:30.839186+0000","last_unstale":"2024-09-18T07:33:51.415136+0000","last_undegraded":"2024-09-18T07:33:51.415136+0000","last_fullsized":"2024-09-18T07:33:51.415136+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:32:55.796771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:55.796771+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:57.092409+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"127'6594","reported_seq":10515,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:46.489448+0000","last_change":"2024-09-18T07:33:01.225875+0000","last_active":"2024-09-18T07:33:46.489448+0000","last_peered":"2024-09-18T07:33:46.489448+0000","last_clean":"2024-09-18T07:33:46.489448+0000","last_became_active":"2024-09-18T07:33:01.221136+0000","last_became_peered":"2024-09-18T07:33:01.221136+0000","last_unstale":"2024-09-18T07:33:46.489448+0000","last_undegraded":"2024-09-18T07:33:46.489448+0000","last_fullsized":"2024-09-18T07:33:46.489448+0000","mapping_epoch":107,"log_start":"88'900","ondisk_log_start":"88'900","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5694,"log_dups_size":0,"ondisk_log_size":5694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:38.955324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426246144,"num_objects":6504,"num_object_clones":0,"num_object_copies":19512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6594,"num_write_kb":419136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1904,"num_bytes_recovered":124780544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":179,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.377207+0000","last_change":"2024-09-18T07:33:27.040227+0000","last_active":"2024-09-18T07:33:44.377207+0000","last_peered":"2024-09-18T07:33:44.377207+0000","last_clean":"2024-09-18T07:33:44.377207+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:33:44.377207+0000","last_undegraded":"2024-09-18T07:33:44.377207+0000","last_fullsized":"2024-09-18T07:33:44.377207+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":150,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.374287+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:33:44.374287+0000","last_peered":"2024-09-18T07:33:44.374287+0000","last_clean":"2024-09-18T07:33:44.374287+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:33:44.374287+0000","last_undegraded":"2024-09-18T07:33:44.374287+0000","last_fullsized":"2024-09-18T07:33:44.374287+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:43.132679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":202,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:45.042893+0000","last_change":"2024-09-18T07:32:30.859165+0000","last_active":"2024-09-18T07:33:45.042893+0000","last_peered":"2024-09-18T07:33:45.042893+0000","last_clean":"2024-09-18T07:33:45.042893+0000","last_became_active":"2024-09-18T07:32:30.851004+0000","last_became_peered":"2024-09-18T07:32:30.851004+0000","last_unstale":"2024-09-18T07:33:45.042893+0000","last_undegraded":"2024-09-18T07:33:45.042893+0000","last_fullsized":"2024-09-18T07:33:45.042893+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:04.076666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":77,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":146,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.417801+0000","last_change":"2024-09-18T07:33:27.394247+0000","last_active":"2024-09-18T07:33:51.417801+0000","last_peered":"2024-09-18T07:33:51.417801+0000","last_clean":"2024-09-18T07:33:51.417801+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:33:51.417801+0000","last_undegraded":"2024-09-18T07:33:51.417801+0000","last_fullsized":"2024-09-18T07:33:51.417801+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.394203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.394203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:31.541179+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":149,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.367088+0000","last_change":"2024-09-18T07:33:25.028842+0000","last_active":"2024-09-18T07:33:44.367088+0000","last_peered":"2024-09-18T07:33:44.367088+0000","last_clean":"2024-09-18T07:33:44.367088+0000","last_became_active":"2024-09-18T07:31:02.347390+0000","last_became_peered":"2024-09-18T07:31:02.347390+0000","last_unstale":"2024-09-18T07:33:44.367088+0000","last_undegraded":"2024-09-18T07:33:44.367088+0000","last_fullsized":"2024-09-18T07:33:44.367088+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:25.028609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:25.028609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:34.346431+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"127'6686","reported_seq":10655,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:46.583408+0000","last_change":"2024-09-18T07:32:59.213954+0000","last_active":"2024-09-18T07:33:46.583408+0000","last_peered":"2024-09-18T07:33:46.583408+0000","last_clean":"2024-09-18T07:33:46.583408+0000","last_became_active":"2024-09-18T07:32:59.204989+0000","last_became_peered":"2024-09-18T07:32:59.204989+0000","last_unstale":"2024-09-18T07:33:46.583408+0000","last_undegraded":"2024-09-18T07:33:46.583408+0000","last_fullsized":"2024-09-18T07:33:46.583408+0000","mapping_epoch":105,"log_start":"96'2090","ondisk_log_start":"96'2090","created":81,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4596,"log_dups_size":0,"ondisk_log_size":4596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432537600,"num_objects":6600,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6686,"num_write_kb":425152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1927,"num_bytes_recovered":126287872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"94'17","reported_seq":241,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.415193+0000","last_change":"2024-09-18T07:33:33.849354+0000","last_active":"2024-09-18T07:33:51.415193+0000","last_peered":"2024-09-18T07:33:51.415193+0000","last_clean":"2024-09-18T07:33:51.415193+0000","last_became_active":"2024-09-18T07:32:30.842449+0000","last_became_peered":"2024-09-18T07:32:30.842449+0000","last_unstale":"2024-09-18T07:33:51.415193+0000","last_undegraded":"2024-09-18T07:33:51.415193+0000","last_fullsized":"2024-09-18T07:33:51.415193+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'17","last_scrub_stamp":"2024-09-18T07:33:33.849285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.849285+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:53.616735+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":152,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.368163+0000","last_change":"2024-09-18T07:31:59.617046+0000","last_active":"2024-09-18T07:33:44.368163+0000","last_peered":"2024-09-18T07:33:44.368163+0000","last_clean":"2024-09-18T07:33:44.368163+0000","last_became_active":"2024-09-18T07:30:56.288149+0000","last_became_peered":"2024-09-18T07:30:56.288149+0000","last_unstale":"2024-09-18T07:33:44.368163+0000","last_undegraded":"2024-09-18T07:33:44.368163+0000","last_fullsized":"2024-09-18T07:33:44.368163+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:59.616988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:59.616988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":174,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.373737+0000","last_change":"2024-09-18T07:33:23.172950+0000","last_active":"2024-09-18T07:33:44.373737+0000","last_peered":"2024-09-18T07:33:44.373737+0000","last_clean":"2024-09-18T07:33:44.373737+0000","last_became_active":"2024-09-18T07:32:30.842828+0000","last_became_peered":"2024-09-18T07:32:30.842828+0000","last_unstale":"2024-09-18T07:33:44.373737+0000","last_undegraded":"2024-09-18T07:33:44.373737+0000","last_fullsized":"2024-09-18T07:33:44.373737+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:33:23.172848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:33:23.172848+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.696337+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":79,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.368532+0000","last_change":"2024-09-18T07:32:30.835729+0000","last_active":"2024-09-18T07:33:44.368532+0000","last_peered":"2024-09-18T07:33:44.368532+0000","last_clean":"2024-09-18T07:33:44.368532+0000","last_became_active":"2024-09-18T07:32:30.835105+0000","last_became_peered":"2024-09-18T07:32:30.835105+0000","last_unstale":"2024-09-18T07:33:44.368532+0000","last_undegraded":"2024-09-18T07:33:44.368532+0000","last_fullsized":"2024-09-18T07:33:44.368532+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:02.499041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.499041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.738806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":142,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.417672+0000","last_change":"2024-09-18T07:33:22.358792+0000","last_active":"2024-09-18T07:33:51.417672+0000","last_peered":"2024-09-18T07:33:51.417672+0000","last_clean":"2024-09-18T07:33:51.417672+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:33:51.417672+0000","last_undegraded":"2024-09-18T07:33:51.417672+0000","last_fullsized":"2024-09-18T07:33:51.417672+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.358746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.358746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:26.289679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"128'6936","reported_seq":7040,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.426918+0000","last_change":"2024-09-18T07:32:08.063469+0000","last_active":"2024-09-18T07:33:51.426918+0000","last_peered":"2024-09-18T07:33:51.426918+0000","last_clean":"2024-09-18T07:33:51.426918+0000","last_became_active":"2024-09-18T07:32:08.063070+0000","last_became_peered":"2024-09-18T07:32:08.063070+0000","last_unstale":"2024-09-18T07:33:51.426918+0000","last_undegraded":"2024-09-18T07:33:51.426918+0000","last_fullsized":"2024-09-18T07:33:51.426918+0000","mapping_epoch":81,"log_start":"88'1300","ondisk_log_start":"88'1300","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5636,"log_dups_size":0,"ondisk_log_size":5636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:39.798722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404422680,"num_objects":6172,"num_object_clones":0,"num_object_copies":18516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6172,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6936,"num_write_kb":419393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":181,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.406598+0000","last_change":"2024-09-18T07:32:26.793474+0000","last_active":"2024-09-18T07:33:51.406598+0000","last_peered":"2024-09-18T07:33:51.406598+0000","last_clean":"2024-09-18T07:33:51.406598+0000","last_became_active":"2024-09-18T07:32:26.785711+0000","last_became_peered":"2024-09-18T07:32:26.785711+0000","last_unstale":"2024-09-18T07:33:51.406598+0000","last_undegraded":"2024-09-18T07:33:51.406598+0000","last_fullsized":"2024-09-18T07:33:51.406598+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":154,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.406175+0000","last_change":"2024-09-18T07:32:30.829352+0000","last_active":"2024-09-18T07:33:51.406175+0000","last_peered":"2024-09-18T07:33:51.406175+0000","last_clean":"2024-09-18T07:33:51.406175+0000","last_became_active":"2024-09-18T07:32:30.828736+0000","last_became_peered":"2024-09-18T07:32:30.828736+0000","last_unstale":"2024-09-18T07:33:51.406175+0000","last_undegraded":"2024-09-18T07:33:51.406175+0000","last_fullsized":"2024-09-18T07:33:51.406175+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:19.900370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":146,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.417678+0000","last_change":"2024-09-18T07:33:36.582039+0000","last_active":"2024-09-18T07:33:51.417678+0000","last_peered":"2024-09-18T07:33:51.417678+0000","last_clean":"2024-09-18T07:33:51.417678+0000","last_became_active":"2024-09-18T07:31:00.336172+0000","last_became_peered":"2024-09-18T07:31:00.336172+0000","last_unstale":"2024-09-18T07:33:51.417678+0000","last_undegraded":"2024-09-18T07:33:51.417678+0000","last_fullsized":"2024-09-18T07:33:51.417678+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:36.581998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.581998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:04.288035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.363538+0000","last_change":"2024-09-18T07:33:20.990542+0000","last_active":"2024-09-18T07:33:44.363538+0000","last_peered":"2024-09-18T07:33:44.363538+0000","last_clean":"2024-09-18T07:33:44.363538+0000","last_became_active":"2024-09-18T07:31:02.490953+0000","last_became_peered":"2024-09-18T07:31:02.490953+0000","last_unstale":"2024-09-18T07:33:44.363538+0000","last_undegraded":"2024-09-18T07:33:44.363538+0000","last_fullsized":"2024-09-18T07:33:44.363538+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:20.990401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:20.990401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.208665+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"127'6558","reported_seq":6660,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:46.513271+0000","last_change":"2024-09-18T07:32:08.060282+0000","last_active":"2024-09-18T07:33:46.513271+0000","last_peered":"2024-09-18T07:33:46.513271+0000","last_clean":"2024-09-18T07:33:46.513271+0000","last_became_active":"2024-09-18T07:32:08.059971+0000","last_became_peered":"2024-09-18T07:32:08.059971+0000","last_unstale":"2024-09-18T07:33:46.513271+0000","last_undegraded":"2024-09-18T07:33:46.513271+0000","last_fullsized":"2024-09-18T07:33:46.513271+0000","mapping_epoch":81,"log_start":"88'800","ondisk_log_start":"88'800","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5758,"log_dups_size":0,"ondisk_log_size":5758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:45.960343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422182912,"num_objects":6442,"num_object_clones":0,"num_object_copies":19326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6558,"num_write_kb":416000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":171,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.406853+0000","last_change":"2024-09-18T07:30:58.312636+0000","last_active":"2024-09-18T07:33:51.406853+0000","last_peered":"2024-09-18T07:33:51.406853+0000","last_clean":"2024-09-18T07:33:51.406853+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:33:51.406853+0000","last_undegraded":"2024-09-18T07:33:51.406853+0000","last_fullsized":"2024-09-18T07:33:51.406853+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":146,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.415186+0000","last_change":"2024-09-18T07:30:56.284303+0000","last_active":"2024-09-18T07:33:51.415186+0000","last_peered":"2024-09-18T07:33:51.415186+0000","last_clean":"2024-09-18T07:33:51.415186+0000","last_became_active":"2024-09-18T07:30:56.284020+0000","last_became_peered":"2024-09-18T07:30:56.284020+0000","last_unstale":"2024-09-18T07:33:51.415186+0000","last_undegraded":"2024-09-18T07:33:51.415186+0000","last_fullsized":"2024-09-18T07:33:51.415186+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.835065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":145,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.367173+0000","last_change":"2024-09-18T07:32:05.239202+0000","last_active":"2024-09-18T07:33:44.367173+0000","last_peered":"2024-09-18T07:33:44.367173+0000","last_clean":"2024-09-18T07:33:44.367173+0000","last_became_active":"2024-09-18T07:31:00.332836+0000","last_became_peered":"2024-09-18T07:31:00.332836+0000","last_unstale":"2024-09-18T07:33:44.367173+0000","last_undegraded":"2024-09-18T07:33:44.367173+0000","last_fullsized":"2024-09-18T07:33:44.367173+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:05.239160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.239160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:37.915616+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":141,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.368168+0000","last_change":"2024-09-18T07:33:21.860561+0000","last_active":"2024-09-18T07:33:44.368168+0000","last_peered":"2024-09-18T07:33:44.368168+0000","last_clean":"2024-09-18T07:33:44.368168+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:33:44.368168+0000","last_undegraded":"2024-09-18T07:33:44.368168+0000","last_fullsized":"2024-09-18T07:33:44.368168+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.860495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.860495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:51.431373+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"127'6630","reported_seq":10567,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:46.576822+0000","last_change":"2024-09-18T07:33:14.766074+0000","last_active":"2024-09-18T07:33:46.576822+0000","last_peered":"2024-09-18T07:33:46.576822+0000","last_clean":"2024-09-18T07:33:46.576822+0000","last_became_active":"2024-09-18T07:33:14.762251+0000","last_became_peered":"2024-09-18T07:33:14.762251+0000","last_unstale":"2024-09-18T07:33:46.576822+0000","last_undegraded":"2024-09-18T07:33:46.576822+0000","last_fullsized":"2024-09-18T07:33:46.576822+0000","mapping_epoch":115,"log_start":"96'2030","ondisk_log_start":"96'2030","created":81,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4600,"log_dups_size":0,"ondisk_log_size":4600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426901504,"num_objects":6514,"num_object_clones":0,"num_object_copies":19542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6630,"num_write_kb":420608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1912,"num_bytes_recovered":125304832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":166,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.406017+0000","last_change":"2024-09-18T07:32:05.000363+0000","last_active":"2024-09-18T07:33:51.406017+0000","last_peered":"2024-09-18T07:33:51.406017+0000","last_clean":"2024-09-18T07:33:51.406017+0000","last_became_active":"2024-09-18T07:30:58.316423+0000","last_became_peered":"2024-09-18T07:30:58.316423+0000","last_unstale":"2024-09-18T07:33:51.406017+0000","last_undegraded":"2024-09-18T07:33:51.406017+0000","last_fullsized":"2024-09-18T07:33:51.406017+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:32:05.000310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.000310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":152,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.367156+0000","last_change":"2024-09-18T07:32:06.688493+0000","last_active":"2024-09-18T07:33:44.367156+0000","last_peered":"2024-09-18T07:33:44.367156+0000","last_clean":"2024-09-18T07:33:44.367156+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:33:44.367156+0000","last_undegraded":"2024-09-18T07:33:44.367156+0000","last_fullsized":"2024-09-18T07:33:44.367156+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:06.688444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:06.688444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"120'10","reported_seq":210,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.371289+0000","last_change":"2024-09-18T07:33:33.001389+0000","last_active":"2024-09-18T07:33:44.371289+0000","last_peered":"2024-09-18T07:33:44.371289+0000","last_clean":"2024-09-18T07:33:44.371289+0000","last_became_active":"2024-09-18T07:33:27.007386+0000","last_became_peered":"2024-09-18T07:33:27.007386+0000","last_unstale":"2024-09-18T07:33:44.371289+0000","last_undegraded":"2024-09-18T07:33:44.371289+0000","last_fullsized":"2024-09-18T07:33:44.371289+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'10","last_scrub_stamp":"2024-09-18T07:33:33.000874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.000874+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:48.823137+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":135,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.406340+0000","last_change":"2024-09-18T07:31:02.347721+0000","last_active":"2024-09-18T07:33:51.406340+0000","last_peered":"2024-09-18T07:33:51.406340+0000","last_clean":"2024-09-18T07:33:51.406340+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:33:51.406340+0000","last_undegraded":"2024-09-18T07:33:51.406340+0000","last_fullsized":"2024-09-18T07:33:51.406340+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"127'6648","reported_seq":10785,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:46.391748+0000","last_change":"2024-09-18T07:33:04.288238+0000","last_active":"2024-09-18T07:33:46.391748+0000","last_peered":"2024-09-18T07:33:46.391748+0000","last_clean":"2024-09-18T07:33:46.391748+0000","last_became_active":"2024-09-18T07:33:04.282407+0000","last_became_peered":"2024-09-18T07:33:04.282407+0000","last_unstale":"2024-09-18T07:33:46.391748+0000","last_undegraded":"2024-09-18T07:33:46.391748+0000","last_fullsized":"2024-09-18T07:33:46.391748+0000","mapping_epoch":110,"log_start":"88'900","ondisk_log_start":"88'900","created":81,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5748,"log_dups_size":0,"ondisk_log_size":5748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429522944,"num_objects":6554,"num_object_clones":0,"num_object_copies":19662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6648,"num_write_kb":422464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2011,"num_bytes_recovered":131792896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":156,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.376429+0000","last_change":"2024-09-18T07:30:58.314424+0000","last_active":"2024-09-18T07:33:44.376429+0000","last_peered":"2024-09-18T07:33:44.376429+0000","last_clean":"2024-09-18T07:33:44.376429+0000","last_became_active":"2024-09-18T07:30:58.314127+0000","last_became_peered":"2024-09-18T07:30:58.314127+0000","last_unstale":"2024-09-18T07:33:44.376429+0000","last_undegraded":"2024-09-18T07:33:44.376429+0000","last_fullsized":"2024-09-18T07:33:44.376429+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:37.131991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":154,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.416922+0000","last_change":"2024-09-18T07:33:28.459796+0000","last_active":"2024-09-18T07:33:51.416922+0000","last_peered":"2024-09-18T07:33:51.416922+0000","last_clean":"2024-09-18T07:33:51.416922+0000","last_became_active":"2024-09-18T07:30:56.288868+0000","last_became_peered":"2024-09-18T07:30:56.288868+0000","last_unstale":"2024-09-18T07:33:51.416922+0000","last_undegraded":"2024-09-18T07:33:51.416922+0000","last_fullsized":"2024-09-18T07:33:51.416922+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:28.459747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:28.459747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:54.082899+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":137,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.372923+0000","last_change":"2024-09-18T07:31:00.336020+0000","last_active":"2024-09-18T07:33:44.372923+0000","last_peered":"2024-09-18T07:33:44.372923+0000","last_clean":"2024-09-18T07:33:44.372923+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:33:44.372923+0000","last_undegraded":"2024-09-18T07:33:44.372923+0000","last_fullsized":"2024-09-18T07:33:44.372923+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:12.700165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.374207+0000","last_change":"2024-09-18T07:32:30.854343+0000","last_active":"2024-09-18T07:33:44.374207+0000","last_peered":"2024-09-18T07:33:44.374207+0000","last_clean":"2024-09-18T07:33:44.374207+0000","last_became_active":"2024-09-18T07:32:30.854029+0000","last_became_peered":"2024-09-18T07:32:30.854029+0000","last_unstale":"2024-09-18T07:33:44.374207+0000","last_undegraded":"2024-09-18T07:33:44.374207+0000","last_fullsized":"2024-09-18T07:33:44.374207+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"127'6652","reported_seq":6754,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:46.582907+0000","last_change":"2024-09-18T07:32:08.057221+0000","last_active":"2024-09-18T07:33:46.582907+0000","last_peered":"2024-09-18T07:33:46.582907+0000","last_clean":"2024-09-18T07:33:46.582907+0000","last_became_active":"2024-09-18T07:32:08.056907+0000","last_became_peered":"2024-09-18T07:32:08.056907+0000","last_unstale":"2024-09-18T07:33:46.582907+0000","last_undegraded":"2024-09-18T07:33:46.582907+0000","last_fullsized":"2024-09-18T07:33:46.582907+0000","mapping_epoch":81,"log_start":"96'2102","ondisk_log_start":"96'2102","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4550,"log_dups_size":0,"ondisk_log_size":4550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427294720,"num_objects":6520,"num_object_clones":0,"num_object_copies":19560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6652,"num_write_kb":421504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":216,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.414912+0000","last_change":"2024-09-18T07:33:32.803961+0000","last_active":"2024-09-18T07:33:51.414912+0000","last_peered":"2024-09-18T07:33:51.414912+0000","last_clean":"2024-09-18T07:33:51.414912+0000","last_became_active":"2024-09-18T07:32:30.834955+0000","last_became_peered":"2024-09-18T07:32:30.834955+0000","last_unstale":"2024-09-18T07:33:51.414912+0000","last_undegraded":"2024-09-18T07:33:51.414912+0000","last_fullsized":"2024-09-18T07:33:51.414912+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:33:32.803884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.803884+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:34.097591+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":82,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.417128+0000","last_change":"2024-09-18T07:32:30.840417+0000","last_active":"2024-09-18T07:33:51.417128+0000","last_peered":"2024-09-18T07:33:51.417128+0000","last_clean":"2024-09-18T07:33:51.417128+0000","last_became_active":"2024-09-18T07:32:30.840046+0000","last_became_peered":"2024-09-18T07:32:30.840046+0000","last_unstale":"2024-09-18T07:33:51.417128+0000","last_undegraded":"2024-09-18T07:33:51.417128+0000","last_fullsized":"2024-09-18T07:33:51.417128+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.481153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.481153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.709514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":152,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.368700+0000","last_change":"2024-09-18T07:33:34.966610+0000","last_active":"2024-09-18T07:33:44.368700+0000","last_peered":"2024-09-18T07:33:44.368700+0000","last_clean":"2024-09-18T07:33:44.368700+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:33:44.368700+0000","last_undegraded":"2024-09-18T07:33:44.368700+0000","last_fullsized":"2024-09-18T07:33:44.368700+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:34.966537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:34.966537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.041153+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":79,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.368264+0000","last_change":"2024-09-18T07:32:30.839764+0000","last_active":"2024-09-18T07:33:44.368264+0000","last_peered":"2024-09-18T07:33:44.368264+0000","last_clean":"2024-09-18T07:33:44.368264+0000","last_became_active":"2024-09-18T07:32:30.839399+0000","last_became_peered":"2024-09-18T07:32:30.839399+0000","last_unstale":"2024-09-18T07:33:44.368264+0000","last_undegraded":"2024-09-18T07:33:44.368264+0000","last_fullsized":"2024-09-18T07:33:44.368264+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:05.699246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.a","version":"127'6578","reported_seq":6680,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:46.483358+0000","last_change":"2024-09-18T07:32:08.065180+0000","last_active":"2024-09-18T07:33:46.483358+0000","last_peered":"2024-09-18T07:33:46.483358+0000","last_clean":"2024-09-18T07:33:46.483358+0000","last_became_active":"2024-09-18T07:32:08.064911+0000","last_became_peered":"2024-09-18T07:32:08.064911+0000","last_unstale":"2024-09-18T07:33:46.483358+0000","last_undegraded":"2024-09-18T07:33:46.483358+0000","last_fullsized":"2024-09-18T07:33:46.483358+0000","mapping_epoch":81,"log_start":"96'2400","ondisk_log_start":"96'2400","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4178,"log_dups_size":0,"ondisk_log_size":4178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425852928,"num_objects":6498,"num_object_clones":0,"num_object_copies":19494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6578,"num_write_kb":418432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":174,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.405959+0000","last_change":"2024-09-18T07:32:02.967912+0000","last_active":"2024-09-18T07:33:51.405959+0000","last_peered":"2024-09-18T07:33:51.405959+0000","last_clean":"2024-09-18T07:33:51.405959+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:33:51.405959+0000","last_undegraded":"2024-09-18T07:33:51.405959+0000","last_fullsized":"2024-09-18T07:33:51.405959+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:32:02.967835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.967835+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":82,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.406750+0000","last_change":"2024-09-18T07:32:30.846539+0000","last_active":"2024-09-18T07:33:51.406750+0000","last_peered":"2024-09-18T07:33:51.406750+0000","last_clean":"2024-09-18T07:33:51.406750+0000","last_became_active":"2024-09-18T07:32:30.846204+0000","last_became_peered":"2024-09-18T07:32:30.846204+0000","last_unstale":"2024-09-18T07:33:51.406750+0000","last_undegraded":"2024-09-18T07:33:51.406750+0000","last_fullsized":"2024-09-18T07:33:51.406750+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:17.888739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":145,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.368404+0000","last_change":"2024-09-18T07:32:04.417412+0000","last_active":"2024-09-18T07:33:44.368404+0000","last_peered":"2024-09-18T07:33:44.368404+0000","last_clean":"2024-09-18T07:33:44.368404+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:33:44.368404+0000","last_undegraded":"2024-09-18T07:33:44.368404+0000","last_fullsized":"2024-09-18T07:33:44.368404+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.417368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.417368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":139,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.366881+0000","last_change":"2024-09-18T07:32:30.849280+0000","last_active":"2024-09-18T07:33:44.366881+0000","last_peered":"2024-09-18T07:33:44.366881+0000","last_clean":"2024-09-18T07:33:44.366881+0000","last_became_active":"2024-09-18T07:32:30.847635+0000","last_became_peered":"2024-09-18T07:32:30.847635+0000","last_unstale":"2024-09-18T07:33:44.366881+0000","last_undegraded":"2024-09-18T07:33:44.366881+0000","last_fullsized":"2024-09-18T07:33:44.366881+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"127'6449","reported_seq":6552,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:46.400425+0000","last_change":"2024-09-18T07:32:08.062243+0000","last_active":"2024-09-18T07:33:46.400425+0000","last_peered":"2024-09-18T07:33:46.400425+0000","last_clean":"2024-09-18T07:33:46.400425+0000","last_became_active":"2024-09-18T07:32:08.061925+0000","last_became_peered":"2024-09-18T07:32:08.061925+0000","last_unstale":"2024-09-18T07:33:46.400425+0000","last_undegraded":"2024-09-18T07:33:46.400425+0000","last_fullsized":"2024-09-18T07:33:46.400425+0000","mapping_epoch":81,"log_start":"96'2200","ondisk_log_start":"96'2200","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4249,"log_dups_size":0,"ondisk_log_size":4249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.902340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417005568,"num_objects":6363,"num_object_clones":0,"num_object_copies":19089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6449,"num_write_kb":409984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":190,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.416139+0000","last_change":"2024-09-18T07:32:30.858426+0000","last_active":"2024-09-18T07:33:51.416139+0000","last_peered":"2024-09-18T07:33:51.416139+0000","last_clean":"2024-09-18T07:33:51.416139+0000","last_became_active":"2024-09-18T07:32:30.832486+0000","last_became_peered":"2024-09-18T07:32:30.832486+0000","last_unstale":"2024-09-18T07:33:51.416139+0000","last_undegraded":"2024-09-18T07:33:51.416139+0000","last_fullsized":"2024-09-18T07:33:51.416139+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:01.244752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":98,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.406235+0000","last_change":"2024-09-18T07:32:30.865327+0000","last_active":"2024-09-18T07:33:51.406235+0000","last_peered":"2024-09-18T07:33:51.406235+0000","last_clean":"2024-09-18T07:33:51.406235+0000","last_became_active":"2024-09-18T07:32:30.844389+0000","last_became_peered":"2024-09-18T07:32:30.844389+0000","last_unstale":"2024-09-18T07:33:51.406235+0000","last_undegraded":"2024-09-18T07:33:51.406235+0000","last_fullsized":"2024-09-18T07:33:51.406235+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":136,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.414538+0000","last_change":"2024-09-18T07:32:57.792858+0000","last_active":"2024-09-18T07:33:51.414538+0000","last_peered":"2024-09-18T07:33:51.414538+0000","last_clean":"2024-09-18T07:33:51.414538+0000","last_became_active":"2024-09-18T07:32:30.831224+0000","last_became_peered":"2024-09-18T07:32:30.831224+0000","last_unstale":"2024-09-18T07:33:51.414538+0000","last_undegraded":"2024-09-18T07:33:51.414538+0000","last_fullsized":"2024-09-18T07:33:51.414538+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-09-18T07:32:57.792795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:57.792795+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:00.822109+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":134,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.416551+0000","last_change":"2024-09-18T07:31:02.489914+0000","last_active":"2024-09-18T07:33:51.416551+0000","last_peered":"2024-09-18T07:33:51.416551+0000","last_clean":"2024-09-18T07:33:51.416551+0000","last_became_active":"2024-09-18T07:31:02.489619+0000","last_became_peered":"2024-09-18T07:31:02.489619+0000","last_unstale":"2024-09-18T07:33:51.416551+0000","last_undegraded":"2024-09-18T07:33:51.416551+0000","last_fullsized":"2024-09-18T07:33:51.416551+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:13.393902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":154,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.414488+0000","last_change":"2024-09-18T07:32:04.369688+0000","last_active":"2024-09-18T07:33:51.414488+0000","last_peered":"2024-09-18T07:33:51.414488+0000","last_clean":"2024-09-18T07:33:51.414488+0000","last_became_active":"2024-09-18T07:30:56.282357+0000","last_became_peered":"2024-09-18T07:30:56.282357+0000","last_unstale":"2024-09-18T07:33:51.414488+0000","last_undegraded":"2024-09-18T07:33:51.414488+0000","last_fullsized":"2024-09-18T07:33:51.414488+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.369620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.369620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:14.844761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":164,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.369489+0000","last_change":"2024-09-18T07:32:30.872792+0000","last_active":"2024-09-18T07:33:44.369489+0000","last_peered":"2024-09-18T07:33:44.369489+0000","last_clean":"2024-09-18T07:33:44.369489+0000","last_became_active":"2024-09-18T07:32:30.856211+0000","last_became_peered":"2024-09-18T07:32:30.856211+0000","last_unstale":"2024-09-18T07:33:44.369489+0000","last_undegraded":"2024-09-18T07:33:44.369489+0000","last_fullsized":"2024-09-18T07:33:44.369489+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:19.033885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":146,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.418839+0000","last_change":"2024-09-18T07:33:32.589362+0000","last_active":"2024-09-18T07:33:51.418839+0000","last_peered":"2024-09-18T07:33:51.418839+0000","last_clean":"2024-09-18T07:33:51.418839+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:33:51.418839+0000","last_undegraded":"2024-09-18T07:33:51.418839+0000","last_fullsized":"2024-09-18T07:33:51.418839+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.589289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.589289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:38.300848+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":89,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.415105+0000","last_change":"2024-09-18T07:32:52.808559+0000","last_active":"2024-09-18T07:33:51.415105+0000","last_peered":"2024-09-18T07:33:51.415105+0000","last_clean":"2024-09-18T07:33:51.415105+0000","last_became_active":"2024-09-18T07:32:30.831805+0000","last_became_peered":"2024-09-18T07:32:30.831805+0000","last_unstale":"2024-09-18T07:33:51.415105+0000","last_undegraded":"2024-09-18T07:33:51.415105+0000","last_fullsized":"2024-09-18T07:33:51.415105+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:52.808493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:52.808493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:58.415698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":173,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.377286+0000","last_change":"2024-09-18T07:32:30.868702+0000","last_active":"2024-09-18T07:33:44.377286+0000","last_peered":"2024-09-18T07:33:44.377286+0000","last_clean":"2024-09-18T07:33:44.377286+0000","last_became_active":"2024-09-18T07:32:30.861402+0000","last_became_peered":"2024-09-18T07:32:30.861402+0000","last_unstale":"2024-09-18T07:33:44.377286+0000","last_undegraded":"2024-09-18T07:33:44.377286+0000","last_fullsized":"2024-09-18T07:33:44.377286+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":82,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.405867+0000","last_change":"2024-09-18T07:32:30.842334+0000","last_active":"2024-09-18T07:33:51.405867+0000","last_peered":"2024-09-18T07:33:51.405867+0000","last_clean":"2024-09-18T07:33:51.405867+0000","last_became_active":"2024-09-18T07:32:30.841928+0000","last_became_peered":"2024-09-18T07:32:30.841928+0000","last_unstale":"2024-09-18T07:33:51.405867+0000","last_undegraded":"2024-09-18T07:33:51.405867+0000","last_fullsized":"2024-09-18T07:33:51.405867+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":143,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.369145+0000","last_change":"2024-09-18T07:32:30.858456+0000","last_active":"2024-09-18T07:33:44.369145+0000","last_peered":"2024-09-18T07:33:44.369145+0000","last_clean":"2024-09-18T07:33:44.369145+0000","last_became_active":"2024-09-18T07:32:30.858167+0000","last_became_peered":"2024-09-18T07:32:30.858167+0000","last_unstale":"2024-09-18T07:33:44.369145+0000","last_undegraded":"2024-09-18T07:33:44.369145+0000","last_fullsized":"2024-09-18T07:33:44.369145+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.368381+0000","last_change":"2024-09-18T07:32:30.835415+0000","last_active":"2024-09-18T07:33:44.368381+0000","last_peered":"2024-09-18T07:33:44.368381+0000","last_clean":"2024-09-18T07:33:44.368381+0000","last_became_active":"2024-09-18T07:32:30.834708+0000","last_became_peered":"2024-09-18T07:32:30.834708+0000","last_unstale":"2024-09-18T07:33:44.368381+0000","last_undegraded":"2024-09-18T07:33:44.368381+0000","last_fullsized":"2024-09-18T07:33:44.368381+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:07.726137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":175,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.417202+0000","last_change":"2024-09-18T07:32:30.851755+0000","last_active":"2024-09-18T07:33:51.417202+0000","last_peered":"2024-09-18T07:33:51.417202+0000","last_clean":"2024-09-18T07:33:51.417202+0000","last_became_active":"2024-09-18T07:32:30.837622+0000","last_became_peered":"2024-09-18T07:32:30.837622+0000","last_unstale":"2024-09-18T07:33:51.417202+0000","last_undegraded":"2024-09-18T07:33:51.417202+0000","last_fullsized":"2024-09-18T07:33:51.417202+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:10.412163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.374379+0000","last_change":"2024-09-18T07:32:30.854427+0000","last_active":"2024-09-18T07:33:44.374379+0000","last_peered":"2024-09-18T07:33:44.374379+0000","last_clean":"2024-09-18T07:33:44.374379+0000","last_became_active":"2024-09-18T07:32:30.853969+0000","last_became_peered":"2024-09-18T07:32:30.853969+0000","last_unstale":"2024-09-18T07:33:44.374379+0000","last_undegraded":"2024-09-18T07:33:44.374379+0000","last_fullsized":"2024-09-18T07:33:44.374379+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:15.041483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"94'9","reported_seq":190,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.363703+0000","last_change":"2024-09-18T07:32:30.867244+0000","last_active":"2024-09-18T07:33:44.363703+0000","last_peered":"2024-09-18T07:33:44.363703+0000","last_clean":"2024-09-18T07:33:44.363703+0000","last_became_active":"2024-09-18T07:32:30.848580+0000","last_became_peered":"2024-09-18T07:32:30.848580+0000","last_unstale":"2024-09-18T07:33:44.363703+0000","last_undegraded":"2024-09-18T07:33:44.363703+0000","last_fullsized":"2024-09-18T07:33:44.363703+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.090692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":135,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.405915+0000","last_change":"2024-09-18T07:31:02.343206+0000","last_active":"2024-09-18T07:33:51.405915+0000","last_peered":"2024-09-18T07:33:51.405915+0000","last_clean":"2024-09-18T07:33:51.405915+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:33:51.405915+0000","last_undegraded":"2024-09-18T07:33:51.405915+0000","last_fullsized":"2024-09-18T07:33:51.405915+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":174,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.367412+0000","last_change":"2024-09-18T07:32:30.867689+0000","last_active":"2024-09-18T07:33:44.367412+0000","last_peered":"2024-09-18T07:33:44.367412+0000","last_clean":"2024-09-18T07:33:44.367412+0000","last_became_active":"2024-09-18T07:32:30.848136+0000","last_became_peered":"2024-09-18T07:32:30.848136+0000","last_unstale":"2024-09-18T07:33:44.367412+0000","last_undegraded":"2024-09-18T07:33:44.367412+0000","last_fullsized":"2024-09-18T07:33:44.367412+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":82,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.406224+0000","last_change":"2024-09-18T07:32:30.832648+0000","last_active":"2024-09-18T07:33:51.406224+0000","last_peered":"2024-09-18T07:33:51.406224+0000","last_clean":"2024-09-18T07:33:51.406224+0000","last_became_active":"2024-09-18T07:32:30.832312+0000","last_became_peered":"2024-09-18T07:32:30.832312+0000","last_unstale":"2024-09-18T07:33:51.406224+0000","last_undegraded":"2024-09-18T07:33:51.406224+0000","last_fullsized":"2024-09-18T07:33:51.406224+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.844687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":185,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.416506+0000","last_change":"2024-09-18T07:31:00.332729+0000","last_active":"2024-09-18T07:33:51.416506+0000","last_peered":"2024-09-18T07:33:51.416506+0000","last_clean":"2024-09-18T07:33:51.416506+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:33:51.416506+0000","last_undegraded":"2024-09-18T07:33:51.416506+0000","last_fullsized":"2024-09-18T07:33:51.416506+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:11.427383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":147,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.376759+0000","last_change":"2024-09-18T07:33:32.135015+0000","last_active":"2024-09-18T07:33:44.376759+0000","last_peered":"2024-09-18T07:33:44.376759+0000","last_clean":"2024-09-18T07:33:44.376759+0000","last_became_active":"2024-09-18T07:32:30.854291+0000","last_became_peered":"2024-09-18T07:32:30.854291+0000","last_unstale":"2024-09-18T07:33:44.376759+0000","last_undegraded":"2024-09-18T07:33:44.376759+0000","last_fullsized":"2024-09-18T07:33:44.376759+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.134847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.134847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:34.913547+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":162,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.406131+0000","last_change":"2024-09-18T07:30:58.314730+0000","last_active":"2024-09-18T07:33:51.406131+0000","last_peered":"2024-09-18T07:33:51.406131+0000","last_clean":"2024-09-18T07:33:51.406131+0000","last_became_active":"2024-09-18T07:30:58.314428+0000","last_became_peered":"2024-09-18T07:30:58.314428+0000","last_unstale":"2024-09-18T07:33:51.406131+0000","last_undegraded":"2024-09-18T07:33:51.406131+0000","last_fullsized":"2024-09-18T07:33:51.406131+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":154,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.406451+0000","last_change":"2024-09-18T07:32:30.831845+0000","last_active":"2024-09-18T07:33:51.406451+0000","last_peered":"2024-09-18T07:33:51.406451+0000","last_clean":"2024-09-18T07:33:51.406451+0000","last_became_active":"2024-09-18T07:32:30.831546+0000","last_became_peered":"2024-09-18T07:32:30.831546+0000","last_unstale":"2024-09-18T07:33:51.406451+0000","last_undegraded":"2024-09-18T07:33:51.406451+0000","last_fullsized":"2024-09-18T07:33:51.406451+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:38.680767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":137,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.366471+0000","last_change":"2024-09-18T07:31:00.337826+0000","last_active":"2024-09-18T07:33:44.366471+0000","last_peered":"2024-09-18T07:33:44.366471+0000","last_clean":"2024-09-18T07:33:44.366471+0000","last_became_active":"2024-09-18T07:31:00.334158+0000","last_became_peered":"2024-09-18T07:31:00.334158+0000","last_unstale":"2024-09-18T07:33:44.366471+0000","last_undegraded":"2024-09-18T07:33:44.366471+0000","last_fullsized":"2024-09-18T07:33:44.366471+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.058588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":132,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.362874+0000","last_change":"2024-09-18T07:31:02.351247+0000","last_active":"2024-09-18T07:33:44.362874+0000","last_peered":"2024-09-18T07:33:44.362874+0000","last_clean":"2024-09-18T07:33:44.362874+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:33:44.362874+0000","last_undegraded":"2024-09-18T07:33:44.362874+0000","last_fullsized":"2024-09-18T07:33:44.362874+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":36,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.374126+0000","last_change":"2024-09-18T07:33:27.033510+0000","last_active":"2024-09-18T07:33:44.374126+0000","last_peered":"2024-09-18T07:33:44.374126+0000","last_clean":"2024-09-18T07:33:44.374126+0000","last_became_active":"2024-09-18T07:33:27.003887+0000","last_became_peered":"2024-09-18T07:33:27.003887+0000","last_unstale":"2024-09-18T07:33:44.374126+0000","last_undegraded":"2024-09-18T07:33:44.374126+0000","last_fullsized":"2024-09-18T07:33:44.374126+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.187186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.187186+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":80,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.363321+0000","last_change":"2024-09-18T07:32:30.847936+0000","last_active":"2024-09-18T07:33:44.363321+0000","last_peered":"2024-09-18T07:33:44.363321+0000","last_clean":"2024-09-18T07:33:44.363321+0000","last_became_active":"2024-09-18T07:32:30.847608+0000","last_became_peered":"2024-09-18T07:32:30.847608+0000","last_unstale":"2024-09-18T07:33:44.363321+0000","last_undegraded":"2024-09-18T07:33:44.363321+0000","last_fullsized":"2024-09-18T07:33:44.363321+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:13.935728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":136,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.366955+0000","last_change":"2024-09-18T07:31:00.333689+0000","last_active":"2024-09-18T07:33:44.366955+0000","last_peered":"2024-09-18T07:33:44.366955+0000","last_clean":"2024-09-18T07:33:44.366955+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:33:44.366955+0000","last_undegraded":"2024-09-18T07:33:44.366955+0000","last_fullsized":"2024-09-18T07:33:44.366955+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.373242+0000","last_change":"2024-09-18T07:32:04.503356+0000","last_active":"2024-09-18T07:33:44.373242+0000","last_peered":"2024-09-18T07:33:44.373242+0000","last_clean":"2024-09-18T07:33:44.373242+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:33:44.373242+0000","last_undegraded":"2024-09-18T07:33:44.373242+0000","last_fullsized":"2024-09-18T07:33:44.373242+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.503270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.503270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":178,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.363082+0000","last_change":"2024-09-18T07:32:30.869693+0000","last_active":"2024-09-18T07:33:44.363082+0000","last_peered":"2024-09-18T07:33:44.363082+0000","last_clean":"2024-09-18T07:33:44.363082+0000","last_became_active":"2024-09-18T07:32:30.842218+0000","last_became_peered":"2024-09-18T07:32:30.842218+0000","last_unstale":"2024-09-18T07:33:44.363082+0000","last_undegraded":"2024-09-18T07:33:44.363082+0000","last_fullsized":"2024-09-18T07:33:44.363082+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:43.146132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":161,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.418957+0000","last_change":"2024-09-18T07:32:30.853892+0000","last_active":"2024-09-18T07:33:51.418957+0000","last_peered":"2024-09-18T07:33:51.418957+0000","last_clean":"2024-09-18T07:33:51.418957+0000","last_became_active":"2024-09-18T07:32:30.853486+0000","last_became_peered":"2024-09-18T07:32:30.853486+0000","last_unstale":"2024-09-18T07:33:51.418957+0000","last_undegraded":"2024-09-18T07:33:51.418957+0000","last_fullsized":"2024-09-18T07:33:51.418957+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:58.734027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:58.734027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:42.752966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":89,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.405401+0000","last_change":"2024-09-18T07:33:30.668600+0000","last_active":"2024-09-18T07:33:51.405401+0000","last_peered":"2024-09-18T07:33:51.405401+0000","last_clean":"2024-09-18T07:33:51.405401+0000","last_became_active":"2024-09-18T07:32:30.843194+0000","last_became_peered":"2024-09-18T07:32:30.843194+0000","last_unstale":"2024-09-18T07:33:51.405401+0000","last_undegraded":"2024-09-18T07:33:51.405401+0000","last_fullsized":"2024-09-18T07:33:51.405401+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.668428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.668428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:55.997265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":140,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.367084+0000","last_change":"2024-09-18T07:33:30.975087+0000","last_active":"2024-09-18T07:33:44.367084+0000","last_peered":"2024-09-18T07:33:44.367084+0000","last_clean":"2024-09-18T07:33:44.367084+0000","last_became_active":"2024-09-18T07:31:02.490994+0000","last_became_peered":"2024-09-18T07:31:02.490994+0000","last_unstale":"2024-09-18T07:33:44.367084+0000","last_undegraded":"2024-09-18T07:33:44.367084+0000","last_fullsized":"2024-09-18T07:33:44.367084+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.974886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.974886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.586219+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":167,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.370485+0000","last_change":"2024-09-18T07:33:29.988707+0000","last_active":"2024-09-18T07:33:44.370485+0000","last_peered":"2024-09-18T07:33:44.370485+0000","last_clean":"2024-09-18T07:33:44.370485+0000","last_became_active":"2024-09-18T07:32:26.791562+0000","last_became_peered":"2024-09-18T07:32:26.791562+0000","last_unstale":"2024-09-18T07:33:44.370485+0000","last_undegraded":"2024-09-18T07:33:44.370485+0000","last_fullsized":"2024-09-18T07:33:44.370485+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:33:29.988636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.988636+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:34.823616+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":144,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.380174+0000","last_change":"2024-09-18T07:30:56.283643+0000","last_active":"2024-09-18T07:33:44.380174+0000","last_peered":"2024-09-18T07:33:44.380174+0000","last_clean":"2024-09-18T07:33:44.380174+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:33:44.380174+0000","last_undegraded":"2024-09-18T07:33:44.380174+0000","last_fullsized":"2024-09-18T07:33:44.380174+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:41.159206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":153,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.414700+0000","last_change":"2024-09-18T07:32:31.593581+0000","last_active":"2024-09-18T07:33:51.414700+0000","last_peered":"2024-09-18T07:33:51.414700+0000","last_clean":"2024-09-18T07:33:51.414700+0000","last_became_active":"2024-09-18T07:32:30.841210+0000","last_became_peered":"2024-09-18T07:32:30.841210+0000","last_unstale":"2024-09-18T07:33:51.414700+0000","last_undegraded":"2024-09-18T07:33:51.414700+0000","last_fullsized":"2024-09-18T07:33:51.414700+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:31.593520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:31.593520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:50.708269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":87,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.372719+0000","last_change":"2024-09-18T07:33:36.017902+0000","last_active":"2024-09-18T07:33:44.372719+0000","last_peered":"2024-09-18T07:33:44.372719+0000","last_clean":"2024-09-18T07:33:44.372719+0000","last_became_active":"2024-09-18T07:32:30.850481+0000","last_became_peered":"2024-09-18T07:32:30.850481+0000","last_unstale":"2024-09-18T07:33:44.372719+0000","last_undegraded":"2024-09-18T07:33:44.372719+0000","last_fullsized":"2024-09-18T07:33:44.372719+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:36.017602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.017602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.466305+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":141,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.367747+0000","last_change":"2024-09-18T07:33:24.053054+0000","last_active":"2024-09-18T07:33:44.367747+0000","last_peered":"2024-09-18T07:33:44.367747+0000","last_clean":"2024-09-18T07:33:44.367747+0000","last_became_active":"2024-09-18T07:31:02.347537+0000","last_became_peered":"2024-09-18T07:31:02.347537+0000","last_unstale":"2024-09-18T07:33:44.367747+0000","last_undegraded":"2024-09-18T07:33:44.367747+0000","last_fullsized":"2024-09-18T07:33:44.367747+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:24.052908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:24.052908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:39.677071+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":158,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.370712+0000","last_change":"2024-09-18T07:33:33.072569+0000","last_active":"2024-09-18T07:33:44.370712+0000","last_peered":"2024-09-18T07:33:44.370712+0000","last_clean":"2024-09-18T07:33:44.370712+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:33:44.370712+0000","last_undegraded":"2024-09-18T07:33:44.370712+0000","last_fullsized":"2024-09-18T07:33:44.370712+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:33.072487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.072487+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:50.254990+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":169,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-18T07:33:51.415527+0000","last_change":"2024-09-18T07:33:27.742768+0000","last_active":"2024-09-18T07:33:51.415527+0000","last_peered":"2024-09-18T07:33:51.415527+0000","last_clean":"2024-09-18T07:33:51.415527+0000","last_became_active":"2024-09-18T07:30:58.313556+0000","last_became_peered":"2024-09-18T07:30:58.313556+0000","last_unstale":"2024-09-18T07:33:51.415527+0000","last_undegraded":"2024-09-18T07:33:51.415527+0000","last_fullsized":"2024-09-18T07:33:51.415527+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:33:27.742684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.742684+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:51.970382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":151,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-18T07:33:44.376989+0000","last_change":"2024-09-18T07:33:22.161956+0000","last_active":"2024-09-18T07:33:44.376989+0000","last_peered":"2024-09-18T07:33:44.376989+0000","last_clean":"2024-09-18T07:33:44.376989+0000","last_became_active":"2024-09-18T07:32:30.846635+0000","last_became_peered":"2024-09-18T07:32:30.846635+0000","last_unstale":"2024-09-18T07:33:44.376989+0000","last_undegraded":"2024-09-18T07:33:44.376989+0000","last_fullsized":"2024-09-18T07:33:44.376989+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.161894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.161894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:31.115412+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6688604184,"num_objects":102061,"num_object_clones":0,"num_object_copies":306183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102061,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106767,"num_write_kb":6682433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9693,"num_bytes_recovered":635240448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20200566784,"data_stored":20200554568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78780,"ondisk_log_size":78780,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":1584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":77,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":236223201319,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462957,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2040224,"kb_used_data":1972944,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91704928,"statfs":{"total":95995035648,"available":93905846272,"internally_reserved":0,"allocated":2020294656,"data_stored":2019903691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724594,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2057888,"kb_used_data":1986448,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91687264,"statfs":{"total":95995035648,"available":93887758336,"internally_reserved":0,"allocated":2034122752,"data_stored":2033731785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953529,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2476404,"kb_used_data":2395236,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":91268748,"statfs":{"total":95995035648,"available":93459197952,"internally_reserved":0,"allocated":2452721664,"data_stored":2452310679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4295116,"kb_used_data":4161080,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":89450036,"statfs":{"total":95995035648,"available":91596836864,"internally_reserved":0,"allocated":4260945920,"data_stored":4260516429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":137099719},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3001180,"kb_used_data":2915784,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":90743972,"statfs":{"total":95995035648,"available":92921827328,"internally_reserved":0,"allocated":2985762816,"data_stored":2985379733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3010160,"kb_used_data":2920668,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":90734992,"statfs":{"total":95995035648,"available":92912631808,"internally_reserved":0,"allocated":2990764032,"data_stored":2990298210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3439876,"kb_used_data":3336496,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":90305276,"statfs":{"total":95995035648,"available":92472602624,"internally_reserved":0,"allocated":3416571904,"data_stored":3416150535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3400929280,"data_stored":3400925208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2971471872,"data_stored":2971467800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2969567232,"data_stored":2969567232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4247453696,"data_stored":4247453696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2436497408,"data_stored":2436497408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2017656832,"data_stored":2017656832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2003308544,"data_stored":2003304472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":153681920,"data_stored":153681920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:33:53.342 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-18T07:33:53.342 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:33:53.343 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-18T07:33:53.343 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T07:33:53.343 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T07:33:53.343 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-18T07:33:53.344 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:53.369 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:53.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:53.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:53.423 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:53.471 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:53.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:53.520 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:53.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:53.569 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:53 smithi137 ceph-mon[29408]: 5.3 scrub starts 2024-09-18T07:33:53.569 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:53 smithi137 ceph-mon[29408]: 5.3 scrub ok 2024-09-18T07:33:53.570 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:53 smithi137 ceph-mon[39175]: 5.3 scrub starts 2024-09-18T07:33:53.570 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:53 smithi137 ceph-mon[39175]: 5.3 scrub ok 2024-09-18T07:33:53.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:53.672 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:53.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:53.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:53.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:53 smithi203 ceph-mon[32237]: 5.3 scrub starts 2024-09-18T07:33:53.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:53 smithi203 ceph-mon[32237]: 5.3 scrub ok 2024-09-18T07:33:53.774 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:53.798 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:53.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:53.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:53.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:53.934 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:53.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:54.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:54.070 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:54.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:54.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:54.091 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T07:33:54.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:54.189 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:33:54.197 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:33:54.197 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:33:54.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:54.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:54.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:54.242 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:33:54.242 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:33:54.287 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:33:54.288 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:33:54.319 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:54.328 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:33:54.328 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:33:54.361 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:33:54.362 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:33:54.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:54.375 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:54.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:54.382 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:33:54.395 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:33:54.395 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:33:54.428 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:33:54.429 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:33:54.460 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:33:54.460 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:33:54.476 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:54.522 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:54.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:54.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:54.570 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:54 smithi203 ceph-mon[32237]: 3.1b scrub starts 2024-09-18T07:33:54.571 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:54 smithi203 ceph-mon[32237]: 3.1b scrub ok 2024-09-18T07:33:54.571 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:54 smithi203 ceph-mon[32237]: pgmap v132: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 30 MiB/s wr, 1.13k op/s 2024-09-18T07:33:54.571 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:54 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/407623218' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:54.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:54.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:54.665 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:54.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:54.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:54 smithi137 ceph-mon[29408]: 3.1b scrub starts 2024-09-18T07:33:54.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:54 smithi137 ceph-mon[29408]: 3.1b scrub ok 2024-09-18T07:33:54.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:54 smithi137 ceph-mon[29408]: pgmap v132: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 30 MiB/s wr, 1.13k op/s 2024-09-18T07:33:54.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:54 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/407623218' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:54.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:54 smithi137 ceph-mon[39175]: 3.1b scrub starts 2024-09-18T07:33:54.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:54 smithi137 ceph-mon[39175]: 3.1b scrub ok 2024-09-18T07:33:54.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:54 smithi137 ceph-mon[39175]: pgmap v132: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 30 MiB/s wr, 1.13k op/s 2024-09-18T07:33:54.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:54 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/407623218' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:33:54.693 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:33:54.693 INFO:teuthology.orchestra.run.smithi203.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-18T07:33:54.693 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:33:54.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:54.818 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:54.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:54.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:54.920 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:54.945 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:54.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:55.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:55.081 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:55.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:55.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:55.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:55.220 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:55.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:55.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:55.322 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:55.349 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:55.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:55.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:55.443 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:55.452 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:55.476 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:55.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:55.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:55.506 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:55.579 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:55.596 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:55 smithi137 ceph-mon[29408]: 3.1a scrub starts 2024-09-18T07:33:55.597 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:55 smithi137 ceph-mon[29408]: 3.1a scrub ok 2024-09-18T07:33:55.597 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:55 smithi137 ceph-mon[29408]: 4.14 scrub starts 2024-09-18T07:33:55.597 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:55 smithi137 ceph-mon[29408]: 4.14 scrub ok 2024-09-18T07:33:55.597 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:55 smithi137 ceph-mon[39175]: 3.1a scrub starts 2024-09-18T07:33:55.597 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:55 smithi137 ceph-mon[39175]: 3.1a scrub ok 2024-09-18T07:33:55.597 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:55 smithi137 ceph-mon[39175]: 4.14 scrub starts 2024-09-18T07:33:55.597 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:55 smithi137 ceph-mon[39175]: 4.14 scrub ok 2024-09-18T07:33:55.602 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:55.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:55.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:55.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:55.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:55 smithi203 ceph-mon[32237]: 3.1a scrub starts 2024-09-18T07:33:55.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:55 smithi203 ceph-mon[32237]: 3.1a scrub ok 2024-09-18T07:33:55.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:55 smithi203 ceph-mon[32237]: 4.14 scrub starts 2024-09-18T07:33:55.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:55 smithi203 ceph-mon[32237]: 4.14 scrub ok 2024-09-18T07:33:55.745 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:55.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:55.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:55.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:55.873 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:55.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:55.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:55.974 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:56.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:56.010 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:56.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:56.110 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:33:56.135 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:56.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:56.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:33:56.237 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:56.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:56.262 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:56.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:56.363 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:56.386 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:56.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:56.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:56.395 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:33:56.395 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:33:56.488 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:56.513 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:56.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:56.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:56.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:56.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:56.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:56.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:56.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:33:56.795 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:56.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:56.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:33:56.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:56.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:56 smithi137 ceph-mon[29408]: pgmap v133: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 27 MiB/s wr, 1.03k op/s 2024-09-18T07:33:56.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:56 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:33:56.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:56 smithi137 ceph-mon[39175]: pgmap v133: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 27 MiB/s wr, 1.03k op/s 2024-09-18T07:33:56.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:56 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:33:56.920 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:56.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:57.022 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:57.046 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:57.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:57.147 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:57.170 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:57.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:57.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:57.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:56 smithi203 ceph-mon[32237]: pgmap v133: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 27 MiB/s wr, 1.03k op/s 2024-09-18T07:33:57.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:56 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:33:57.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:33:56 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:33:57.272 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:57.298 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:57.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:57.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:57.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:33:57.399 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:57.407 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:33:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:33:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:33:57.422 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:57.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:57.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:57.474 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T07:33:57.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:57.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:57.617 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:57.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:57.718 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:57.742 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:57.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:57.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:57.826 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:33:57.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:33:57.868 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:57.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:57.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:33:57.970 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:57.994 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:57.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:57.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:58.059 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:58.081 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:57 smithi137 ceph-mon[29408]: 4.d scrub starts 2024-09-18T07:33:58.081 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:57 smithi137 ceph-mon[29408]: 4.d scrub ok 2024-09-18T07:33:58.081 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:57 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:58.081 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:57 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1169660409' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:58.081 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:57 smithi137 ceph-mon[29408]: from='client.24922 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:58.081 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:57 smithi137 ceph-mon[39175]: 4.d scrub starts 2024-09-18T07:33:58.081 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:57 smithi137 ceph-mon[39175]: 4.d scrub ok 2024-09-18T07:33:58.081 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:57 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:58.081 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:57 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1169660409' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:58.082 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:57 smithi137 ceph-mon[39175]: from='client.24922 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:58.096 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:58.137 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:58.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:58.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:58.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:57 smithi203 ceph-mon[32237]: 4.d scrub starts 2024-09-18T07:33:58.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:57 smithi203 ceph-mon[32237]: 4.d scrub ok 2024-09-18T07:33:58.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:57 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:33:58.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:57 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1169660409' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:58.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:57 smithi203 ceph-mon[32237]: from='client.24922 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:33:58.239 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:58.262 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:58.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:58.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:58.364 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:58.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:58.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:58.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:58.512 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:58.535 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:58.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:58.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:33:58.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:58.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:58.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:58.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:58.793 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:58.849 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:58.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:58.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:58.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:58.989 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:58.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:59.063 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[29408]: 5.8 scrub starts 2024-09-18T07:33:59.063 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[29408]: 5.8 scrub ok 2024-09-18T07:33:59.063 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[29408]: pgmap v134: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-18T07:33:59.064 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[29408]: 6.5 scrub starts 2024-09-18T07:33:59.064 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[29408]: 6.5 scrub ok 2024-09-18T07:33:59.064 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:33:59.064 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:33:59.064 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[29408]: from='client.24922 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:33:59.064 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[29408]: osdmap e129: 8 total, 8 up, 7 in 2024-09-18T07:33:59.064 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[29408]: 5.19 scrub starts 2024-09-18T07:33:59.064 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[29408]: 5.19 scrub ok 2024-09-18T07:33:59.065 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[39175]: 5.8 scrub starts 2024-09-18T07:33:59.065 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[39175]: 5.8 scrub ok 2024-09-18T07:33:59.065 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[39175]: pgmap v134: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-18T07:33:59.065 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[39175]: 6.5 scrub starts 2024-09-18T07:33:59.065 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[39175]: 6.5 scrub ok 2024-09-18T07:33:59.065 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:33:59.065 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:33:59.065 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[39175]: from='client.24922 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:33:59.065 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[39175]: osdmap e129: 8 total, 8 up, 7 in 2024-09-18T07:33:59.065 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[39175]: 5.19 scrub starts 2024-09-18T07:33:59.065 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:58 smithi137 ceph-mon[39175]: 5.19 scrub ok 2024-09-18T07:33:59.090 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:59.132 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:59.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:59.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:59.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:58 smithi203 ceph-mon[32237]: 5.8 scrub starts 2024-09-18T07:33:59.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:58 smithi203 ceph-mon[32237]: 5.8 scrub ok 2024-09-18T07:33:59.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:58 smithi203 ceph-mon[32237]: pgmap v134: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-18T07:33:59.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:58 smithi203 ceph-mon[32237]: 6.5 scrub starts 2024-09-18T07:33:59.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:58 smithi203 ceph-mon[32237]: 6.5 scrub ok 2024-09-18T07:33:59.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:58 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:33:59.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:58 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:33:59.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:58 smithi203 ceph-mon[32237]: from='client.24922 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:33:59.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:58 smithi203 ceph-mon[32237]: osdmap e129: 8 total, 8 up, 7 in 2024-09-18T07:33:59.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:58 smithi203 ceph-mon[32237]: 5.19 scrub starts 2024-09-18T07:33:59.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:58 smithi203 ceph-mon[32237]: 5.19 scrub ok 2024-09-18T07:33:59.234 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:33:59.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:59.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:59.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:33:59.337 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:33:59.372 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:33:59.394 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:59.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:59.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:33:59.408 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:33:59.408 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:33:59.454 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:33:59.454 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:33:59.490 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:33:59.490 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:33:59.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:59.525 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:33:59.525 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:33:59.538 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:59.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:59.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:59.557 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:33:59.558 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:33:59.590 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:33:59.591 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:33:59.630 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:33:59.630 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:33:59.639 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:33:59.674 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:33:59.675 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:33:59.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:59.686 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:59.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:33:59.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:33:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:59.837 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:33:59.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:33:59.938 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:33:59.962 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:33:59.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:33:59.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:00.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:00.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:00.102 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:00.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:00.124 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:59 smithi137 ceph-mon[29408]: 7.8 scrub starts 2024-09-18T07:34:00.124 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:59 smithi137 ceph-mon[29408]: 5.17 scrub starts 2024-09-18T07:34:00.124 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:59 smithi137 ceph-mon[29408]: 5.17 scrub ok 2024-09-18T07:34:00.124 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:59 smithi137 ceph-mon[29408]: 6.15 scrub starts 2024-09-18T07:34:00.125 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:59 smithi137 ceph-mon[29408]: 6.15 scrub ok 2024-09-18T07:34:00.125 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:33:59 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3722172279' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:00.125 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:59 smithi137 ceph-mon[39175]: 7.8 scrub starts 2024-09-18T07:34:00.125 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:59 smithi137 ceph-mon[39175]: 5.17 scrub starts 2024-09-18T07:34:00.125 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:59 smithi137 ceph-mon[39175]: 5.17 scrub ok 2024-09-18T07:34:00.125 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:59 smithi137 ceph-mon[39175]: 6.15 scrub starts 2024-09-18T07:34:00.125 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:59 smithi137 ceph-mon[39175]: 6.15 scrub ok 2024-09-18T07:34:00.125 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:33:59 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3722172279' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:00.203 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:00.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:59 smithi203 ceph-mon[32237]: 7.8 scrub starts 2024-09-18T07:34:00.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:59 smithi203 ceph-mon[32237]: 5.17 scrub starts 2024-09-18T07:34:00.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:59 smithi203 ceph-mon[32237]: 5.17 scrub ok 2024-09-18T07:34:00.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:59 smithi203 ceph-mon[32237]: 6.15 scrub starts 2024-09-18T07:34:00.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:59 smithi203 ceph-mon[32237]: 6.15 scrub ok 2024-09-18T07:34:00.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:33:59 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3722172279' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:00.228 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:00.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:00.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:00.330 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:00.351 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:00.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:00.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:00.373 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:34:00.373 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:34:00.374 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-18T07:34:00.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:34:00.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:00.517 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:00.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:00.619 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:00.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:00.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:00.771 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:00.792 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:00.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:00.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:34:00.894 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:00.919 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:00.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:00.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:01.021 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:01.075 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:01.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:01.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:01.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:00 smithi137 ceph-mon[29408]: pgmap v136: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-18T07:34:01.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:00 smithi137 ceph-mon[29408]: 5.1f scrub starts 2024-09-18T07:34:01.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:00 smithi137 ceph-mon[29408]: 5.1f scrub ok 2024-09-18T07:34:01.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:00 smithi137 ceph-mon[29408]: 5.18 scrub starts 2024-09-18T07:34:01.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:00 smithi137 ceph-mon[29408]: 5.18 scrub ok 2024-09-18T07:34:01.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:00 smithi137 ceph-mon[39175]: pgmap v136: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-18T07:34:01.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:00 smithi137 ceph-mon[39175]: 5.1f scrub starts 2024-09-18T07:34:01.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:00 smithi137 ceph-mon[39175]: 5.1f scrub ok 2024-09-18T07:34:01.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:00 smithi137 ceph-mon[39175]: 5.18 scrub starts 2024-09-18T07:34:01.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:00 smithi137 ceph-mon[39175]: 5.18 scrub ok 2024-09-18T07:34:01.176 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:01.201 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:01.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:01.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:01.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:00 smithi203 ceph-mon[32237]: pgmap v136: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-18T07:34:01.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:00 smithi203 ceph-mon[32237]: 5.1f scrub starts 2024-09-18T07:34:01.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:00 smithi203 ceph-mon[32237]: 5.1f scrub ok 2024-09-18T07:34:01.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:00 smithi203 ceph-mon[32237]: 5.18 scrub starts 2024-09-18T07:34:01.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:00 smithi203 ceph-mon[32237]: 5.18 scrub ok 2024-09-18T07:34:01.302 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:34:01.302 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:34:01.302 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:34:01.302 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:34:01.302 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:34:01.302 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:01.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:01.326 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:01.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:01.427 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:01.454 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:01.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:01.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:01.580 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:01.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:01.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:01.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:01.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:01.728 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:01.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:01.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:01.869 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:01.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:01.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:01.971 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:01.996 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:01.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:02.098 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:02.109 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:01 smithi203 ceph-mon[32237]: 4.1c scrub starts 2024-09-18T07:34:02.109 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:01 smithi203 ceph-mon[32237]: 4.1c scrub ok 2024-09-18T07:34:02.109 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:01 smithi203 ceph-mon[32237]: 7.8 scrub ok 2024-09-18T07:34:02.121 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:02.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:02.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:02.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:01 smithi137 ceph-mon[29408]: 4.1c scrub starts 2024-09-18T07:34:02.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:01 smithi137 ceph-mon[29408]: 4.1c scrub ok 2024-09-18T07:34:02.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:01 smithi137 ceph-mon[29408]: 7.8 scrub ok 2024-09-18T07:34:02.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:01 smithi137 ceph-mon[39175]: 4.1c scrub starts 2024-09-18T07:34:02.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:01 smithi137 ceph-mon[39175]: 4.1c scrub ok 2024-09-18T07:34:02.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:01 smithi137 ceph-mon[39175]: 7.8 scrub ok 2024-09-18T07:34:02.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:02.261 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:02.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:02.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:02.363 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:02.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:02.406 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:02.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:02.507 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:02.532 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:02.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:02.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:02.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:02.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:02.671 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:02.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:02.772 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:02.795 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:02.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:02.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:02.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T07:34:02.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:02.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:02.950 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:02.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:03.051 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:03.074 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:03.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:03.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:03.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:02 smithi137 ceph-mon[29408]: 3.8 scrub starts 2024-09-18T07:34:03.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:02 smithi137 ceph-mon[29408]: 3.8 scrub ok 2024-09-18T07:34:03.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:02 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:34:03.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:02 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:34:03.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:02 smithi137 ceph-mon[29408]: pgmap v137: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T07:34:03.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:02 smithi137 ceph-mon[29408]: osdmap e130: 8 total, 8 up, 7 in 2024-09-18T07:34:03.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:02 smithi137 ceph-mon[39175]: 3.8 scrub starts 2024-09-18T07:34:03.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:02 smithi137 ceph-mon[39175]: 3.8 scrub ok 2024-09-18T07:34:03.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:02 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:34:03.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:02 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:34:03.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:02 smithi137 ceph-mon[39175]: pgmap v137: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T07:34:03.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:02 smithi137 ceph-mon[39175]: osdmap e130: 8 total, 8 up, 7 in 2024-09-18T07:34:03.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:03.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:02 smithi203 ceph-mon[32237]: 3.8 scrub starts 2024-09-18T07:34:03.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:02 smithi203 ceph-mon[32237]: 3.8 scrub ok 2024-09-18T07:34:03.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:02 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:34:03.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:02 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:34:03.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:02 smithi203 ceph-mon[32237]: pgmap v137: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T07:34:03.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:02 smithi203 ceph-mon[32237]: osdmap e130: 8 total, 8 up, 7 in 2024-09-18T07:34:03.213 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:03.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:03.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:03.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:03.379 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:03.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:03.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:03.409 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:03.433 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:03.484 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:03.535 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:03.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:03.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:03.637 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:03.659 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:03.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:03.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:03.761 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:03.799 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:03.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:03.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:03.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:03.925 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:03.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:03.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:04.027 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:04.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:04.086 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:04.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:04.160 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:03 smithi137 ceph-mon[29408]: 6.9 scrub starts 2024-09-18T07:34:04.160 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:03 smithi137 ceph-mon[29408]: 6.9 scrub ok 2024-09-18T07:34:04.160 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:03 smithi137 ceph-mon[39175]: 6.9 scrub starts 2024-09-18T07:34:04.160 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:03 smithi137 ceph-mon[39175]: 6.9 scrub ok 2024-09-18T07:34:04.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:04.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:03 smithi203 ceph-mon[32237]: 6.9 scrub starts 2024-09-18T07:34:04.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:03 smithi203 ceph-mon[32237]: 6.9 scrub ok 2024-09-18T07:34:04.226 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:04.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:04.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:04.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:04.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:04.352 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:04.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:04.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:04.492 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:04.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:04.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:04.594 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:04.616 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:04.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:04.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:04.718 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:04.744 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:04.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:04.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:04.777 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:04.846 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:04.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:04.871 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:04.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:04.961 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:34:04.965 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:34:04.972 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:04.995 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:04.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:04.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:05.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:04 smithi137 ceph-mon[29408]: pgmap v139: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-18T07:34:05.085 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:04 smithi137 ceph-mon[29408]: 4.8 scrub starts 2024-09-18T07:34:05.086 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:04 smithi137 ceph-mon[29408]: 4.8 scrub ok 2024-09-18T07:34:05.086 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:04 smithi137 ceph-mon[29408]: 4.e scrub starts 2024-09-18T07:34:05.086 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:04 smithi137 ceph-mon[29408]: 4.e scrub ok 2024-09-18T07:34:05.086 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:04 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:34:05.086 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:04 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:34:05.086 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:04 smithi137 ceph-mon[39175]: pgmap v139: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-18T07:34:05.086 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:04 smithi137 ceph-mon[39175]: 4.8 scrub starts 2024-09-18T07:34:05.086 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:04 smithi137 ceph-mon[39175]: 4.8 scrub ok 2024-09-18T07:34:05.086 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:04 smithi137 ceph-mon[39175]: 4.e scrub starts 2024-09-18T07:34:05.086 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:04 smithi137 ceph-mon[39175]: 4.e scrub ok 2024-09-18T07:34:05.086 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:04 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:34:05.087 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:04 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:34:05.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:05.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:05.153 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:05.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:05.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:04 smithi203 ceph-mon[32237]: pgmap v139: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-18T07:34:05.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:04 smithi203 ceph-mon[32237]: 4.8 scrub starts 2024-09-18T07:34:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:04 smithi203 ceph-mon[32237]: 4.8 scrub ok 2024-09-18T07:34:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:04 smithi203 ceph-mon[32237]: 4.e scrub starts 2024-09-18T07:34:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:04 smithi203 ceph-mon[32237]: 4.e scrub ok 2024-09-18T07:34:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:04 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:34:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:04 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:34:05.253 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:05.292 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:05.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:05.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:05.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:05.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:34:05.437 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:05.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:05.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:05.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:05.594 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:05.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:05.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:05.643 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:34:05.643 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:34:05.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:34:05.643 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:34:05.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.643 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:22343 7.6 7:6ec6138d:::benchmark_data_smithi137_74180_object22342:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:34:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:24.660556+0000", 2024-09-18T07:34:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 100.96935858400001, 2024-09-18T07:34:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.19253478399999999, 2024-09-18T07:34:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 22343 2024-09-18T07:34:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.660556+0000", 2024-09-18T07:34:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.660556+0000", 2024-09-18T07:34:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.660557+0000", 2024-09-18T07:34:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.809e-06 2024-09-18T07:34:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.660582+0000", 2024-09-18T07:34:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4273999999999998e-05 2024-09-18T07:34:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.660595+0000", 2024-09-18T07:34:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3016999999999999e-05 2024-09-18T07:34:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.660602+0000", 2024-09-18T07:34:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2320000000000004e-06 2024-09-18T07:34:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.660696+0000", 2024-09-18T07:34:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3874000000000003e-05 2024-09-18T07:34:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.660819+0000", 2024-09-18T07:34:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000123533 2024-09-18T07:34:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:34:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.660955+0000", 2024-09-18T07:34:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013561500000000001 2024-09-18T07:34:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.663485+0000", 2024-09-18T07:34:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025302089999999998 2024-09-18T07:34:05.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.663589+0000", 2024-09-18T07:34:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103687 2024-09-18T07:34:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.853010+0000", 2024-09-18T07:34:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.18942072800000001 2024-09-18T07:34:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.853049+0000", 2024-09-18T07:34:05.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9881e-05 2024-09-18T07:34:05.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.853090+0000", 2024-09-18T07:34:05.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0924999999999999e-05 2024-09-18T07:34:05.650 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.650 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.650 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.650 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:22347 7.f 7:fe6074eb:::benchmark_data_smithi137_74180_object22346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:34:05.651 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:24.661712+0000", 2024-09-18T07:34:05.651 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 100.96820232499999, 2024-09-18T07:34:05.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.19245790099999999, 2024-09-18T07:34:05.651 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.651 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.651 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.651 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.651 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.651 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.651 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 22347 2024-09-18T07:34:05.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.651 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.661712+0000", 2024-09-18T07:34:05.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.661712+0000", 2024-09-18T07:34:05.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.661713+0000", 2024-09-18T07:34:05.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3909999999999999e-06 2024-09-18T07:34:05.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.661733+0000", 2024-09-18T07:34:05.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9765000000000001e-05 2024-09-18T07:34:05.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.661743+0000", 2024-09-18T07:34:05.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0308e-05 2024-09-18T07:34:05.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.661750+0000", 2024-09-18T07:34:05.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7349999999999999e-06 2024-09-18T07:34:05.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.661801+0000", 2024-09-18T07:34:05.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0489999999999999e-05 2024-09-18T07:34:05.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.661906+0000", 2024-09-18T07:34:05.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010575 2024-09-18T07:34:05.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:34:05.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.662161+0000", 2024-09-18T07:34:05.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00025498699999999998 2024-09-18T07:34:05.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.663773+0000", 2024-09-18T07:34:05.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016118320000000001 2024-09-18T07:34:05.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.663972+0000", 2024-09-18T07:34:05.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019845099999999999 2024-09-18T07:34:05.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.854073+0000", 2024-09-18T07:34:05.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.190101031 2024-09-18T07:34:05.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.854129+0000", 2024-09-18T07:34:05.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6706999999999998e-05 2024-09-18T07:34:05.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.854170+0000", 2024-09-18T07:34:05.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0454000000000003e-05 2024-09-18T07:34:05.657 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.657 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.657 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.657 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:22353 7.f 7:f2258ccd:::benchmark_data_smithi137_74180_object22352:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:34:05.657 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:24.664613+0000", 2024-09-18T07:34:05.657 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 100.965301248, 2024-09-18T07:34:05.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.19010458399999999, 2024-09-18T07:34:05.657 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.657 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.657 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.658 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.658 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.658 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.658 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 22353 2024-09-18T07:34:05.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.658 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.664613+0000", 2024-09-18T07:34:05.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.659 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.664613+0000", 2024-09-18T07:34:05.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.659 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.659 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.664614+0000", 2024-09-18T07:34:05.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2049999999999999e-06 2024-09-18T07:34:05.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.659 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.659 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.664630+0000", 2024-09-18T07:34:05.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6103000000000002e-05 2024-09-18T07:34:05.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.664639+0000", 2024-09-18T07:34:05.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9269999999999994e-06 2024-09-18T07:34:05.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.664646+0000", 2024-09-18T07:34:05.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5339999999999996e-06 2024-09-18T07:34:05.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.664686+0000", 2024-09-18T07:34:05.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0787000000000001e-05 2024-09-18T07:34:05.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.664816+0000", 2024-09-18T07:34:05.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000129658 2024-09-18T07:34:05.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:34:05.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.664895+0000", 2024-09-18T07:34:05.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8468999999999997e-05 2024-09-18T07:34:05.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.665646+0000", 2024-09-18T07:34:05.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00075133199999999995 2024-09-18T07:34:05.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.665888+0000", 2024-09-18T07:34:05.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000242203 2024-09-18T07:34:05.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.854628+0000", 2024-09-18T07:34:05.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.18874000699999999 2024-09-18T07:34:05.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.854680+0000", 2024-09-18T07:34:05.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1400999999999998e-05 2024-09-18T07:34:05.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.854718+0000", 2024-09-18T07:34:05.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7957999999999998e-05 2024-09-18T07:34:05.663 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.663 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.664 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.664 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:51269 7.6 7:68d3a188:::benchmark_data_smithi137_74180_object51268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:34:05.664 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.037001+0000", 2024-09-18T07:34:05.664 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 75.592913241999995, 2024-09-18T07:34:05.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17093683200000001, 2024-09-18T07:34:05.664 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.664 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.664 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.664 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.664 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.664 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.665 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 51269 2024-09-18T07:34:05.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.665 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.665 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.665 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.037001+0000", 2024-09-18T07:34:05.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.665 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.665 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.037001+0000", 2024-09-18T07:34:05.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.037003+0000", 2024-09-18T07:34:05.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0320000000000002e-06 2024-09-18T07:34:05.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.037029+0000", 2024-09-18T07:34:05.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5826999999999999e-05 2024-09-18T07:34:05.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.190084+0000", 2024-09-18T07:34:05.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15305543299999999 2024-09-18T07:34:05.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.190094+0000", 2024-09-18T07:34:05.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6330000000000008e-06 2024-09-18T07:34:05.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.192867+0000", 2024-09-18T07:34:05.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027728100000000001 2024-09-18T07:34:05.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.193133+0000", 2024-09-18T07:34:05.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00026640000000000002 2024-09-18T07:34:05.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-09-18T07:34:05.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.193284+0000", 2024-09-18T07:34:05.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015140499999999999 2024-09-18T07:34:05.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.198673+0000", 2024-09-18T07:34:05.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053881509999999999 2024-09-18T07:34:05.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.198953+0000", 2024-09-18T07:34:05.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027992000000000002 2024-09-18T07:34:05.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.206955+0000", 2024-09-18T07:34:05.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0080021290000000002 2024-09-18T07:34:05.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.207025+0000", 2024-09-18T07:34:05.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0316999999999998e-05 2024-09-18T07:34:05.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.207938+0000", 2024-09-18T07:34:05.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000912775 2024-09-18T07:34:05.670 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.670 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.670 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.670 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:72958 7.f 7:f6739288:::benchmark_data_smithi137_74180_object72957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-18T07:34:05.670 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:13.872544+0000", 2024-09-18T07:34:05.670 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 51.757370559999998, 2024-09-18T07:34:05.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.89321731699999996, 2024-09-18T07:34:05.671 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.671 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.671 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:34:05.671 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.671 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.671 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.671 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 72958 2024-09-18T07:34:05.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.671 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.872544+0000", 2024-09-18T07:34:05.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.872544+0000", 2024-09-18T07:34:05.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.872545+0000", 2024-09-18T07:34:05.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.161e-06 2024-09-18T07:34:05.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.872568+0000", 2024-09-18T07:34:05.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3315999999999998e-05 2024-09-18T07:34:05.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.872577+0000", 2024-09-18T07:34:05.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7260000000000008e-06 2024-09-18T07:34:05.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.872584+0000", 2024-09-18T07:34:05.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0539999999999997e-06 2024-09-18T07:34:05.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.872851+0000", 2024-09-18T07:34:05.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000266698 2024-09-18T07:34:05.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.873002+0000", 2024-09-18T07:34:05.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015104100000000001 2024-09-18T07:34:05.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:05.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.873203+0000", 2024-09-18T07:34:05.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000201583 2024-09-18T07:34:05.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.882308+0000", 2024-09-18T07:34:05.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0091045210000000008 2024-09-18T07:34:05.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.882781+0000", 2024-09-18T07:34:05.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00047297399999999997 2024-09-18T07:34:05.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.887888+0000", 2024-09-18T07:34:05.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051069469999999997 2024-09-18T07:34:05.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:05.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.887898+0000", 2024-09-18T07:34:05.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0256e-05 2024-09-18T07:34:05.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.765664+0000", 2024-09-18T07:34:05.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.87776570700000001 2024-09-18T07:34:05.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.765761+0000", 2024-09-18T07:34:05.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7332999999999998e-05 2024-09-18T07:34:05.677 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.677 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.677 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.677 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:72992 7.f 7:f57fcd6e:::benchmark_data_smithi137_74180_object72991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T07:34:05.677 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:13.905501+0000", 2024-09-18T07:34:05.677 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 51.724413337000001, 2024-09-18T07:34:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.86962962099999996, 2024-09-18T07:34:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 72992 2024-09-18T07:34:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.905501+0000", 2024-09-18T07:34:05.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.905501+0000", 2024-09-18T07:34:05.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.905501+0000", 2024-09-18T07:34:05.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2699999999999999e-07 2024-09-18T07:34:05.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.905520+0000", 2024-09-18T07:34:05.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8989000000000001e-05 2024-09-18T07:34:05.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.905526+0000", 2024-09-18T07:34:05.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2390000000000002e-06 2024-09-18T07:34:05.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.905531+0000", 2024-09-18T07:34:05.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8999999999999997e-06 2024-09-18T07:34:05.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.905906+0000", 2024-09-18T07:34:05.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00037503699999999999 2024-09-18T07:34:05.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:05.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.905915+0000", 2024-09-18T07:34:05.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2949999999999999e-06 2024-09-18T07:34:05.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.766351+0000", 2024-09-18T07:34:05.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.86043624600000002 2024-09-18T07:34:05.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.766476+0000", 2024-09-18T07:34:05.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000125294 2024-09-18T07:34:05.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:05.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.766643+0000", 2024-09-18T07:34:05.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000166393 2024-09-18T07:34:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.772455+0000", 2024-09-18T07:34:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058117409999999996 2024-09-18T07:34:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.772628+0000", 2024-09-18T07:34:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000173 2024-09-18T07:34:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.775058+0000", 2024-09-18T07:34:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024306919999999999 2024-09-18T07:34:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.775094+0000", 2024-09-18T07:34:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5808e-05 2024-09-18T07:34:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.775130+0000", 2024-09-18T07:34:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6359999999999997e-05 2024-09-18T07:34:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:73020 7.f 7:f94eaa62:::benchmark_data_smithi137_74180_object73019:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T07:34:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:13.928069+0000", 2024-09-18T07:34:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 51.701845116999998, 2024-09-18T07:34:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85381128100000003, 2024-09-18T07:34:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.685 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 73020 2024-09-18T07:34:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.928069+0000", 2024-09-18T07:34:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.928069+0000", 2024-09-18T07:34:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.928070+0000", 2024-09-18T07:34:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1769999999999999e-06 2024-09-18T07:34:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.928091+0000", 2024-09-18T07:34:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1199000000000002e-05 2024-09-18T07:34:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.928098+0000", 2024-09-18T07:34:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.584e-06 2024-09-18T07:34:05.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.928104+0000", 2024-09-18T07:34:05.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6869999999999999e-06 2024-09-18T07:34:05.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.928153+0000", 2024-09-18T07:34:05.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9508000000000001e-05 2024-09-18T07:34:05.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:05.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.928161+0000", 2024-09-18T07:34:05.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8739999999999995e-06 2024-09-18T07:34:05.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.769339+0000", 2024-09-18T07:34:05.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.84117801699999994 2024-09-18T07:34:05.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.769484+0000", 2024-09-18T07:34:05.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014481100000000001 2024-09-18T07:34:05.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:05.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.769651+0000", 2024-09-18T07:34:05.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016717699999999999 2024-09-18T07:34:05.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.777543+0000", 2024-09-18T07:34:05.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0078922839999999994 2024-09-18T07:34:05.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.777559+0000", 2024-09-18T07:34:05.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5608999999999999e-05 2024-09-18T07:34:05.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.781795+0000", 2024-09-18T07:34:05.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042364300000000002 2024-09-18T07:34:05.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.781833+0000", 2024-09-18T07:34:05.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7715000000000001e-05 2024-09-18T07:34:05.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.781880+0000", 2024-09-18T07:34:05.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7209000000000002e-05 2024-09-18T07:34:05.693 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.693 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.693 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.693 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:73026 7.f 7:f3ec277c:::benchmark_data_smithi137_74180_object73025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T07:34:05.693 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:13.931900+0000", 2024-09-18T07:34:05.693 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 51.698014010000001, 2024-09-18T07:34:05.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85790595300000005, 2024-09-18T07:34:05.693 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.694 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.694 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.694 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.694 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.694 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.694 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 73026 2024-09-18T07:34:05.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.694 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.931900+0000", 2024-09-18T07:34:05.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.931900+0000", 2024-09-18T07:34:05.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.931901+0000", 2024-09-18T07:34:05.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.99e-07 2024-09-18T07:34:05.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.931921+0000", 2024-09-18T07:34:05.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9731000000000001e-05 2024-09-18T07:34:05.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.931928+0000", 2024-09-18T07:34:05.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9639999999999997e-06 2024-09-18T07:34:05.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.931933+0000", 2024-09-18T07:34:05.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3739999999999997e-06 2024-09-18T07:34:05.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.932033+0000", 2024-09-18T07:34:05.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9928999999999998e-05 2024-09-18T07:34:05.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:05.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:13.932042+0000", 2024-09-18T07:34:05.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2459999999999992e-06 2024-09-18T07:34:05.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.772692+0000", 2024-09-18T07:34:05.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.84064957100000004 2024-09-18T07:34:05.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.772827+0000", 2024-09-18T07:34:05.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013494399999999999 2024-09-18T07:34:05.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:05.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.773002+0000", 2024-09-18T07:34:05.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017584499999999999 2024-09-18T07:34:05.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.781927+0000", 2024-09-18T07:34:05.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0089247130000000008 2024-09-18T07:34:05.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.789697+0000", 2024-09-18T07:34:05.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0077701749999999998 2024-09-18T07:34:05.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.789733+0000", 2024-09-18T07:34:05.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5491999999999998e-05 2024-09-18T07:34:05.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.789765+0000", 2024-09-18T07:34:05.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2546000000000003e-05 2024-09-18T07:34:05.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.789806+0000", 2024-09-18T07:34:05.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0723999999999998e-05 2024-09-18T07:34:05.700 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.700 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.700 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.700 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:73037 7.f 7:f10a9c6c:::benchmark_data_smithi137_74180_object73036:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T07:34:05.700 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:14.045464+0000", 2024-09-18T07:34:05.700 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 51.584449874999997, 2024-09-18T07:34:05.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.74972524500000004, 2024-09-18T07:34:05.701 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.701 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.701 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.701 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.701 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.701 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.701 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 73037 2024-09-18T07:34:05.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.701 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.045464+0000", 2024-09-18T07:34:05.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.045464+0000", 2024-09-18T07:34:05.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.045465+0000", 2024-09-18T07:34:05.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4e-07 2024-09-18T07:34:05.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.045537+0000", 2024-09-18T07:34:05.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1588000000000006e-05 2024-09-18T07:34:05.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.045545+0000", 2024-09-18T07:34:05.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9389999999999997e-06 2024-09-18T07:34:05.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.045551+0000", 2024-09-18T07:34:05.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0569999999999998e-06 2024-09-18T07:34:05.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.045695+0000", 2024-09-18T07:34:05.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014371699999999999 2024-09-18T07:34:05.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:05.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.045704+0000", 2024-09-18T07:34:05.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.859e-06 2024-09-18T07:34:05.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.775183+0000", 2024-09-18T07:34:05.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.72947877500000002 2024-09-18T07:34:05.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.775325+0000", 2024-09-18T07:34:05.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014223999999999999 2024-09-18T07:34:05.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:05.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.775494+0000", 2024-09-18T07:34:05.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016884199999999999 2024-09-18T07:34:05.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.781962+0000", 2024-09-18T07:34:05.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0064678260000000003 2024-09-18T07:34:05.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795004+0000", 2024-09-18T07:34:05.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013041634 2024-09-18T07:34:05.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795103+0000", 2024-09-18T07:34:05.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8889999999999997e-05 2024-09-18T07:34:05.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795149+0000", 2024-09-18T07:34:05.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5898e-05 2024-09-18T07:34:05.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795190+0000", 2024-09-18T07:34:05.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.104e-05 2024-09-18T07:34:05.707 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.707 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.708 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.708 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:73062 7.f 7:f47b448d:::benchmark_data_smithi137_74180_object73061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T07:34:05.708 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:14.065870+0000", 2024-09-18T07:34:05.708 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 51.564044316999997, 2024-09-18T07:34:05.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.72943285700000005, 2024-09-18T07:34:05.708 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.708 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.708 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.708 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.708 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.708 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.709 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 73062 2024-09-18T07:34:05.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.709 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.065870+0000", 2024-09-18T07:34:05.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.065870+0000", 2024-09-18T07:34:05.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.065872+0000", 2024-09-18T07:34:05.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7969999999999999e-06 2024-09-18T07:34:05.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.065905+0000", 2024-09-18T07:34:05.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3529999999999999e-05 2024-09-18T07:34:05.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.065914+0000", 2024-09-18T07:34:05.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9420000000000001e-06 2024-09-18T07:34:05.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.065921+0000", 2024-09-18T07:34:05.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8820000000000003e-06 2024-09-18T07:34:05.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.066005+0000", 2024-09-18T07:34:05.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4295999999999994e-05 2024-09-18T07:34:05.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:05.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.066013+0000", 2024-09-18T07:34:05.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0919999999999998e-06 2024-09-18T07:34:05.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.777575+0000", 2024-09-18T07:34:05.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71156208600000004 2024-09-18T07:34:05.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.777680+0000", 2024-09-18T07:34:05.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010429099999999999 2024-09-18T07:34:05.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:05.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.777845+0000", 2024-09-18T07:34:05.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016535399999999999 2024-09-18T07:34:05.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.782018+0000", 2024-09-18T07:34:05.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041728420000000004 2024-09-18T07:34:05.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795093+0000", 2024-09-18T07:34:05.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013074881 2024-09-18T07:34:05.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795219+0000", 2024-09-18T07:34:05.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126267 2024-09-18T07:34:05.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795266+0000", 2024-09-18T07:34:05.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6641000000000003e-05 2024-09-18T07:34:05.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795303+0000", 2024-09-18T07:34:05.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6956000000000003e-05 2024-09-18T07:34:05.715 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.715 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.715 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.715 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:73074 7.f 7:f659d2a9:::benchmark_data_smithi137_74180_object73073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T07:34:05.715 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:14.075840+0000", 2024-09-18T07:34:05.715 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 51.554074393999997, 2024-09-18T07:34:05.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71966354399999999, 2024-09-18T07:34:05.715 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.715 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.715 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.715 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.716 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.716 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.716 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 73074 2024-09-18T07:34:05.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.716 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.075840+0000", 2024-09-18T07:34:05.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.075840+0000", 2024-09-18T07:34:05.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.075842+0000", 2024-09-18T07:34:05.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0439999999999998e-06 2024-09-18T07:34:05.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.075863+0000", 2024-09-18T07:34:05.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1501e-05 2024-09-18T07:34:05.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.075875+0000", 2024-09-18T07:34:05.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2055000000000001e-05 2024-09-18T07:34:05.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.075882+0000", 2024-09-18T07:34:05.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.815e-06 2024-09-18T07:34:05.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.076629+0000", 2024-09-18T07:34:05.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00074673600000000001 2024-09-18T07:34:05.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:05.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.076638+0000", 2024-09-18T07:34:05.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7090000000000008e-06 2024-09-18T07:34:05.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.779892+0000", 2024-09-18T07:34:05.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.70325395599999996 2024-09-18T07:34:05.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.779990+0000", 2024-09-18T07:34:05.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8432999999999998e-05 2024-09-18T07:34:05.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:05.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.780083+0000", 2024-09-18T07:34:05.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3134000000000006e-05 2024-09-18T07:34:05.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.782104+0000", 2024-09-18T07:34:05.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020209300000000002 2024-09-18T07:34:05.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795313+0000", 2024-09-18T07:34:05.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01320901 2024-09-18T07:34:05.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795416+0000", 2024-09-18T07:34:05.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010315999999999999 2024-09-18T07:34:05.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795462+0000", 2024-09-18T07:34:05.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5438000000000001e-05 2024-09-18T07:34:05.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795503+0000", 2024-09-18T07:34:05.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1622999999999998e-05 2024-09-18T07:34:05.721 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.721 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.722 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.722 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:73076 7.f 7:fc3a316e:::benchmark_data_smithi137_74180_object73075:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T07:34:05.722 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:14.076129+0000", 2024-09-18T07:34:05.722 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 51.553785116999997, 2024-09-18T07:34:05.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71949481500000001, 2024-09-18T07:34:05.722 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.722 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.722 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.722 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.722 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.722 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.722 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 73076 2024-09-18T07:34:05.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.723 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.076129+0000", 2024-09-18T07:34:05.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.076129+0000", 2024-09-18T07:34:05.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.076130+0000", 2024-09-18T07:34:05.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3239999999999999e-06 2024-09-18T07:34:05.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.076153+0000", 2024-09-18T07:34:05.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2679000000000001e-05 2024-09-18T07:34:05.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.076163+0000", 2024-09-18T07:34:05.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0244999999999999e-05 2024-09-18T07:34:05.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.076170+0000", 2024-09-18T07:34:05.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.985e-06 2024-09-18T07:34:05.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.076658+0000", 2024-09-18T07:34:05.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00048777199999999998 2024-09-18T07:34:05.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:05.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.076664+0000", 2024-09-18T07:34:05.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8359999999999997e-06 2024-09-18T07:34:05.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.782148+0000", 2024-09-18T07:34:05.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.705484 2024-09-18T07:34:05.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.782259+0000", 2024-09-18T07:34:05.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111043 2024-09-18T07:34:05.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:05.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.782405+0000", 2024-09-18T07:34:05.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000146441 2024-09-18T07:34:05.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.785726+0000", 2024-09-18T07:34:05.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033202290000000001 2024-09-18T07:34:05.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795318+0000", 2024-09-18T07:34:05.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095925469999999999 2024-09-18T07:34:05.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.728 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.728 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795546+0000", 2024-09-18T07:34:05.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000227629 2024-09-18T07:34:05.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.728 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.728 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795589+0000", 2024-09-18T07:34:05.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3563000000000003e-05 2024-09-18T07:34:05.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.728 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.728 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795624+0000", 2024-09-18T07:34:05.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4521999999999999e-05 2024-09-18T07:34:05.729 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.729 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.729 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:05.730 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:73080 7.f 7:fbfd00fd:::benchmark_data_smithi137_74180_object73079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T07:34:05.730 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:14.077301+0000", 2024-09-18T07:34:05.731 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 51.552613145999999, 2024-09-18T07:34:05.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71843948899999999, 2024-09-18T07:34:05.731 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.731 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.731 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.731 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.731 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.731 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.731 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 73080 2024-09-18T07:34:05.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.731 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.077301+0000", 2024-09-18T07:34:05.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.077301+0000", 2024-09-18T07:34:05.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.077303+0000", 2024-09-18T07:34:05.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9640000000000002e-06 2024-09-18T07:34:05.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.077328+0000", 2024-09-18T07:34:05.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5109e-05 2024-09-18T07:34:05.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.077343+0000", 2024-09-18T07:34:05.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4423999999999999e-05 2024-09-18T07:34:05.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.077350+0000", 2024-09-18T07:34:05.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2250000000000003e-06 2024-09-18T07:34:05.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.078126+0000", 2024-09-18T07:34:05.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00077583600000000002 2024-09-18T07:34:05.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:05.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.078135+0000", 2024-09-18T07:34:05.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4360000000000008e-06 2024-09-18T07:34:05.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.784113+0000", 2024-09-18T07:34:05.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.70597842399999999 2024-09-18T07:34:05.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.784197+0000", 2024-09-18T07:34:05.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3652000000000005e-05 2024-09-18T07:34:05.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:05.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.784288+0000", 2024-09-18T07:34:05.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0923999999999999e-05 2024-09-18T07:34:05.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.785764+0000", 2024-09-18T07:34:05.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001476036 2024-09-18T07:34:05.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795322+0000", 2024-09-18T07:34:05.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095581799999999995 2024-09-18T07:34:05.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795664+0000", 2024-09-18T07:34:05.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034143900000000002 2024-09-18T07:34:05.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795707+0000", 2024-09-18T07:34:05.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3380000000000001e-05 2024-09-18T07:34:05.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795741+0000", 2024-09-18T07:34:05.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3460000000000002e-05 2024-09-18T07:34:05.737 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.737 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.738 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.738 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:73087 7.f 7:f306c6ff:::benchmark_data_smithi137_74180_object73086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T07:34:05.738 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:14.083058+0000", 2024-09-18T07:34:05.738 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 51.546855891, 2024-09-18T07:34:05.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71282635400000005, 2024-09-18T07:34:05.738 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.738 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.738 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.738 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.738 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.738 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.739 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 73087 2024-09-18T07:34:05.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.739 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.083058+0000", 2024-09-18T07:34:05.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.083058+0000", 2024-09-18T07:34:05.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.083060+0000", 2024-09-18T07:34:05.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6309999999999999e-06 2024-09-18T07:34:05.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.083081+0000", 2024-09-18T07:34:05.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0910000000000001e-05 2024-09-18T07:34:05.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.083094+0000", 2024-09-18T07:34:05.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2690999999999999e-05 2024-09-18T07:34:05.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.083101+0000", 2024-09-18T07:34:05.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4309999999999997e-06 2024-09-18T07:34:05.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.083175+0000", 2024-09-18T07:34:05.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4110999999999996e-05 2024-09-18T07:34:05.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:05.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.083184+0000", 2024-09-18T07:34:05.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7290000000000006e-06 2024-09-18T07:34:05.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.785790+0000", 2024-09-18T07:34:05.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.70260668199999998 2024-09-18T07:34:05.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.785862+0000", 2024-09-18T07:34:05.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1663999999999997e-05 2024-09-18T07:34:05.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:05.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.785949+0000", 2024-09-18T07:34:05.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7124000000000001e-05 2024-09-18T07:34:05.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.789686+0000", 2024-09-18T07:34:05.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003736634 2024-09-18T07:34:05.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795327+0000", 2024-09-18T07:34:05.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005640834 2024-09-18T07:34:05.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795791+0000", 2024-09-18T07:34:05.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00046446900000000001 2024-09-18T07:34:05.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795843+0000", 2024-09-18T07:34:05.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1694999999999999e-05 2024-09-18T07:34:05.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795885+0000", 2024-09-18T07:34:05.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1749000000000003e-05 2024-09-18T07:34:05.745 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.745 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.745 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.745 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:73118 7.f 7:faea0a3d:::benchmark_data_smithi137_74180_object73117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T07:34:05.745 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:14.111416+0000", 2024-09-18T07:34:05.745 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 51.518497750000002, 2024-09-18T07:34:05.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.68460115099999996, 2024-09-18T07:34:05.745 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.745 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.745 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.745 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.746 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.746 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.746 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 73118 2024-09-18T07:34:05.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.746 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.111416+0000", 2024-09-18T07:34:05.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.111416+0000", 2024-09-18T07:34:05.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.111418+0000", 2024-09-18T07:34:05.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.44e-06 2024-09-18T07:34:05.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.111466+0000", 2024-09-18T07:34:05.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8139000000000001e-05 2024-09-18T07:34:05.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.111480+0000", 2024-09-18T07:34:05.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4007e-05 2024-09-18T07:34:05.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.111488+0000", 2024-09-18T07:34:05.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1179999999999992e-06 2024-09-18T07:34:05.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.111594+0000", 2024-09-18T07:34:05.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000106026 2024-09-18T07:34:05.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:05.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.111605+0000", 2024-09-18T07:34:05.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0735e-05 2024-09-18T07:34:05.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.787500+0000", 2024-09-18T07:34:05.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.67589530499999995 2024-09-18T07:34:05.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.787576+0000", 2024-09-18T07:34:05.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5615e-05 2024-09-18T07:34:05.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:05.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.787686+0000", 2024-09-18T07:34:05.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110537 2024-09-18T07:34:05.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.789854+0000", 2024-09-18T07:34:05.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002167357 2024-09-18T07:34:05.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795375+0000", 2024-09-18T07:34:05.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055214390000000004 2024-09-18T07:34:05.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795929+0000", 2024-09-18T07:34:05.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00055433200000000005 2024-09-18T07:34:05.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795973+0000", 2024-09-18T07:34:05.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3439000000000002e-05 2024-09-18T07:34:05.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.796018+0000", 2024-09-18T07:34:05.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4662e-05 2024-09-18T07:34:05.752 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.752 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.752 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.752 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:73134 7.f 7:fd81af3b:::benchmark_data_smithi137_74180_object73133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T07:34:05.752 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:14.125621+0000", 2024-09-18T07:34:05.752 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 51.504293484000002, 2024-09-18T07:34:05.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.67051514999999995, 2024-09-18T07:34:05.752 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.753 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.753 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.753 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.753 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.753 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.753 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 73134 2024-09-18T07:34:05.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.753 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.125621+0000", 2024-09-18T07:34:05.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.125621+0000", 2024-09-18T07:34:05.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.125622+0000", 2024-09-18T07:34:05.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5239999999999999e-06 2024-09-18T07:34:05.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.125644+0000", 2024-09-18T07:34:05.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1466000000000001e-05 2024-09-18T07:34:05.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.125655+0000", 2024-09-18T07:34:05.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.147e-05 2024-09-18T07:34:05.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.125662+0000", 2024-09-18T07:34:05.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0260000000000002e-06 2024-09-18T07:34:05.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.125720+0000", 2024-09-18T07:34:05.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7457999999999998e-05 2024-09-18T07:34:05.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:05.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.125731+0000", 2024-09-18T07:34:05.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1029999999999999e-05 2024-09-18T07:34:05.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.789890+0000", 2024-09-18T07:34:05.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.664159363 2024-09-18T07:34:05.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.790015+0000", 2024-09-18T07:34:05.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000125215 2024-09-18T07:34:05.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:05.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.790159+0000", 2024-09-18T07:34:05.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014348900000000001 2024-09-18T07:34:05.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795050+0000", 2024-09-18T07:34:05.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048907810000000003 2024-09-18T07:34:05.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795380+0000", 2024-09-18T07:34:05.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00033025099999999999 2024-09-18T07:34:05.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.796059+0000", 2024-09-18T07:34:05.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00067938999999999999 2024-09-18T07:34:05.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.796102+0000", 2024-09-18T07:34:05.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3235000000000002e-05 2024-09-18T07:34:05.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.796136+0000", 2024-09-18T07:34:05.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3451999999999998e-05 2024-09-18T07:34:05.759 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.759 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.759 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.759 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:73170 7.f 7:f8b756cd:::benchmark_data_smithi137_74180_object73169:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T07:34:05.759 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:14.164335+0000", 2024-09-18T07:34:05.759 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 51.465578860000001, 2024-09-18T07:34:05.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.65129836699999999, 2024-09-18T07:34:05.760 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.760 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.760 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.760 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.760 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.760 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.760 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 73170 2024-09-18T07:34:05.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.760 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.164335+0000", 2024-09-18T07:34:05.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.164335+0000", 2024-09-18T07:34:05.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.164337+0000", 2024-09-18T07:34:05.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9609999999999999e-06 2024-09-18T07:34:05.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.164361+0000", 2024-09-18T07:34:05.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3716000000000001e-05 2024-09-18T07:34:05.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.164376+0000", 2024-09-18T07:34:05.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.465e-05 2024-09-18T07:34:05.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.164384+0000", 2024-09-18T07:34:05.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2139999999999996e-06 2024-09-18T07:34:05.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.164464+0000", 2024-09-18T07:34:05.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9717999999999998e-05 2024-09-18T07:34:05.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:05.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.164473+0000", 2024-09-18T07:34:05.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4250000000000004e-06 2024-09-18T07:34:05.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.792383+0000", 2024-09-18T07:34:05.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.62791041199999997 2024-09-18T07:34:05.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.792492+0000", 2024-09-18T07:34:05.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000108668 2024-09-18T07:34:05.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:05.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.792614+0000", 2024-09-18T07:34:05.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000122254 2024-09-18T07:34:05.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795365+0000", 2024-09-18T07:34:05.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00275034 2024-09-18T07:34:05.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.795384+0000", 2024-09-18T07:34:05.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9122000000000001e-05 2024-09-18T07:34:05.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.815530+0000", 2024-09-18T07:34:05.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020145929999999999 2024-09-18T07:34:05.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.815592+0000", 2024-09-18T07:34:05.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2405999999999998e-05 2024-09-18T07:34:05.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.766 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.815634+0000", 2024-09-18T07:34:05.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1551000000000002e-05 2024-09-18T07:34:05.766 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.766 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.766 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.766 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:73180 7.f 7:fbbffa71:::benchmark_data_smithi137_74180_object73179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T07:34:05.766 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:14.176114+0000", 2024-09-18T07:34:05.766 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 51.453799865000001, 2024-09-18T07:34:05.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.63962572699999998, 2024-09-18T07:34:05.766 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.766 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.766 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.766 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.766 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.767 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.767 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 73180 2024-09-18T07:34:05.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.767 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.176114+0000", 2024-09-18T07:34:05.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.176114+0000", 2024-09-18T07:34:05.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.176117+0000", 2024-09-18T07:34:05.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.21e-06 2024-09-18T07:34:05.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.176136+0000", 2024-09-18T07:34:05.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9758999999999998e-05 2024-09-18T07:34:05.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.176147+0000", 2024-09-18T07:34:05.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.039e-05 2024-09-18T07:34:05.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.176152+0000", 2024-09-18T07:34:05.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2630000000000003e-06 2024-09-18T07:34:05.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.176211+0000", 2024-09-18T07:34:05.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8588000000000002e-05 2024-09-18T07:34:05.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:05.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.176220+0000", 2024-09-18T07:34:05.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6670000000000008e-06 2024-09-18T07:34:05.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.796167+0000", 2024-09-18T07:34:05.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.61994652400000005 2024-09-18T07:34:05.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.796305+0000", 2024-09-18T07:34:05.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137878 2024-09-18T07:34:05.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:05.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.796449+0000", 2024-09-18T07:34:05.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014429500000000001 2024-09-18T07:34:05.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.804691+0000", 2024-09-18T07:34:05.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082423870000000003 2024-09-18T07:34:05.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.805579+0000", 2024-09-18T07:34:05.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00088768400000000002 2024-09-18T07:34:05.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.815667+0000", 2024-09-18T07:34:05.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010087626000000001 2024-09-18T07:34:05.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.815710+0000", 2024-09-18T07:34:05.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3642e-05 2024-09-18T07:34:05.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.815740+0000", 2024-09-18T07:34:05.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9813999999999999e-05 2024-09-18T07:34:05.773 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.773 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.773 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.773 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:73194 7.f 7:f734ed71:::benchmark_data_smithi137_74180_object73193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T07:34:05.773 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:14.197182+0000", 2024-09-18T07:34:05.773 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 51.432732586, 2024-09-18T07:34:05.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.61866978299999997, 2024-09-18T07:34:05.774 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.774 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.774 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.774 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.774 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.774 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.774 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 73194 2024-09-18T07:34:05.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.774 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.774 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.774 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.197182+0000", 2024-09-18T07:34:05.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.197182+0000", 2024-09-18T07:34:05.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.197184+0000", 2024-09-18T07:34:05.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1909999999999999e-06 2024-09-18T07:34:05.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.197204+0000", 2024-09-18T07:34:05.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0497e-05 2024-09-18T07:34:05.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.197217+0000", 2024-09-18T07:34:05.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2638e-05 2024-09-18T07:34:05.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.197225+0000", 2024-09-18T07:34:05.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8499999999999994e-06 2024-09-18T07:34:05.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.197282+0000", 2024-09-18T07:34:05.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7510999999999999e-05 2024-09-18T07:34:05.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:05.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.197289+0000", 2024-09-18T07:34:05.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5459999999999997e-06 2024-09-18T07:34:05.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.798805+0000", 2024-09-18T07:34:05.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.60151580400000004 2024-09-18T07:34:05.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.798899+0000", 2024-09-18T07:34:05.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4537000000000006e-05 2024-09-18T07:34:05.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:05.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.799031+0000", 2024-09-18T07:34:05.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013159099999999999 2024-09-18T07:34:05.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.804734+0000", 2024-09-18T07:34:05.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005703634 2024-09-18T07:34:05.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.805585+0000", 2024-09-18T07:34:05.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00085050600000000001 2024-09-18T07:34:05.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.815773+0000", 2024-09-18T07:34:05.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010188045999999999 2024-09-18T07:34:05.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.815819+0000", 2024-09-18T07:34:05.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5781000000000001e-05 2024-09-18T07:34:05.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.815851+0000", 2024-09-18T07:34:05.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2651000000000003e-05 2024-09-18T07:34:05.780 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.780 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.780 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.781 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:73206 7.f 7:fe658b00:::benchmark_data_smithi137_74180_object73205:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-18T07:34:05.781 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:14.229611+0000", 2024-09-18T07:34:05.781 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 51.400302969000002, 2024-09-18T07:34:05.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.58944259700000001, 2024-09-18T07:34:05.781 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:05.781 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:05.781 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:05.781 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:05.781 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:05.781 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:05.781 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 73206 2024-09-18T07:34:05.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.782 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:05.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:05.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.229611+0000", 2024-09-18T07:34:05.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:05.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.229611+0000", 2024-09-18T07:34:05.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:05.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:05.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.229613+0000", 2024-09-18T07:34:05.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7400000000000001e-06 2024-09-18T07:34:05.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:05.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.229633+0000", 2024-09-18T07:34:05.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0366000000000001e-05 2024-09-18T07:34:05.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:05.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.229644+0000", 2024-09-18T07:34:05.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0329999999999999e-05 2024-09-18T07:34:05.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:05.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.229649+0000", 2024-09-18T07:34:05.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7919999999999998e-06 2024-09-18T07:34:05.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.229703+0000", 2024-09-18T07:34:05.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3683000000000001e-05 2024-09-18T07:34:05.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:05.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.229710+0000", 2024-09-18T07:34:05.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6139999999999997e-06 2024-09-18T07:34:05.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:05.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.801290+0000", 2024-09-18T07:34:05.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.57158049 2024-09-18T07:34:05.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:05.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.801412+0000", 2024-09-18T07:34:05.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000121655 2024-09-18T07:34:05.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:05.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.801563+0000", 2024-09-18T07:34:05.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015071400000000001 2024-09-18T07:34:05.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.804799+0000", 2024-09-18T07:34:05.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00323624 2024-09-18T07:34:05.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:05.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.805782+0000", 2024-09-18T07:34:05.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000983004 2024-09-18T07:34:05.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:05.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.818947+0000", 2024-09-18T07:34:05.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013164732 2024-09-18T07:34:05.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:05.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.819011+0000", 2024-09-18T07:34:05.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4177999999999994e-05 2024-09-18T07:34:05.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:05.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:05.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:05.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:14.819054+0000", 2024-09-18T07:34:05.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3059000000000002e-05 2024-09-18T07:34:05.787 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.788 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:05.788 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:05.788 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:34:05.814 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:05.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:05.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:05.904 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:34:05.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:05.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:05.954 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:05.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:06.053 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:05 smithi137 ceph-mon[29408]: osdmap e131: 8 total, 8 up, 7 in 2024-09-18T07:34:06.053 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:05 smithi137 ceph-mon[29408]: from='client.24931 v1:172.21.15.137:0/3269983338' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:34:06.053 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:05 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/578005264' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:34:06.053 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:05 smithi137 ceph-mon[29408]: from='client.24934 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:34:06.054 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:05 smithi137 ceph-mon[39175]: osdmap e131: 8 total, 8 up, 7 in 2024-09-18T07:34:06.054 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:05 smithi137 ceph-mon[39175]: from='client.24931 v1:172.21.15.137:0/3269983338' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:34:06.054 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:05 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/578005264' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:34:06.054 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:05 smithi137 ceph-mon[39175]: from='client.24934 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:34:06.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:06.093 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:06.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:06.194 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:06.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:05 smithi203 ceph-mon[32237]: osdmap e131: 8 total, 8 up, 7 in 2024-09-18T07:34:06.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:05 smithi203 ceph-mon[32237]: from='client.24931 v1:172.21.15.137:0/3269983338' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:34:06.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:05 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/578005264' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:34:06.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:05 smithi203 ceph-mon[32237]: from='client.24934 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:34:06.220 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:06.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:06.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:06.322 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:06.355 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-09-18T07:34:04.872437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5794241075,"num_objects":88604,"num_object_clones":0,"num_object_copies":265812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88604,"num_whiteouts":0,"num_read":922,"num_read_kb":781,"num_write":120972,"num_write_kb":6684106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9794,"num_bytes_recovered":636423665,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79394,"ondisk_log_size":79394,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":17783388,"kb_used_data":17101232,"kb_used_omap":10,"kb_used_meta":681845,"kb_avail":638432676,"statfs":{"total":671965249536,"available":653755060224,"internally_reserved":0,"allocated":17511661568,"data_stored":17509110598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11162,"internal_metadata":698209382},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-309133312,"num_objects":-4717,"num_object_clones":0,"num_object_copies":-14151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4717,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4717,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.424396"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":156,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.832240+0000","last_change":"2024-09-18T07:33:31.183001+0000","last_active":"2024-09-18T07:33:57.832240+0000","last_peered":"2024-09-18T07:33:57.832240+0000","last_clean":"2024-09-18T07:33:57.832240+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:33:57.832240+0000","last_undegraded":"2024-09-18T07:33:57.832240+0000","last_fullsized":"2024-09-18T07:33:57.832240+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:31.182913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:31.182913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:44.795464+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":186,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.821607+0000","last_change":"2024-09-18T07:33:29.685825+0000","last_active":"2024-09-18T07:33:57.821607+0000","last_peered":"2024-09-18T07:33:57.821607+0000","last_clean":"2024-09-18T07:33:57.821607+0000","last_became_active":"2024-09-18T07:32:26.785727+0000","last_became_peered":"2024-09-18T07:32:26.785727+0000","last_unstale":"2024-09-18T07:33:57.821607+0000","last_undegraded":"2024-09-18T07:33:57.821607+0000","last_fullsized":"2024-09-18T07:33:57.821607+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:33:29.685767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.685767+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:33.086673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":164,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.894855+0000","last_change":"2024-09-18T07:33:57.894855+0000","last_active":"2024-09-18T07:33:57.894855+0000","last_peered":"2024-09-18T07:33:57.894855+0000","last_clean":"2024-09-18T07:33:57.894855+0000","last_became_active":"2024-09-18T07:32:30.831949+0000","last_became_peered":"2024-09-18T07:32:30.831949+0000","last_unstale":"2024-09-18T07:33:57.894855+0000","last_undegraded":"2024-09-18T07:33:57.894855+0000","last_fullsized":"2024-09-18T07:33:57.894855+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:57.894611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.894611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:07.665469+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":145,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.821604+0000","last_change":"2024-09-18T07:32:03.953881+0000","last_active":"2024-09-18T07:33:57.821604+0000","last_peered":"2024-09-18T07:33:57.821604+0000","last_clean":"2024-09-18T07:33:57.821604+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:33:57.821604+0000","last_undegraded":"2024-09-18T07:33:57.821604+0000","last_fullsized":"2024-09-18T07:33:57.821604+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.953827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.953827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":30,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.827757+0000","last_change":"2024-09-18T07:33:27.004713+0000","last_active":"2024-09-18T07:33:57.827757+0000","last_peered":"2024-09-18T07:33:57.827757+0000","last_clean":"2024-09-18T07:33:57.827757+0000","last_became_active":"2024-09-18T07:33:27.004385+0000","last_became_peered":"2024-09-18T07:33:27.004385+0000","last_unstale":"2024-09-18T07:33:57.827757+0000","last_undegraded":"2024-09-18T07:33:57.827757+0000","last_fullsized":"2024-09-18T07:33:57.827757+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:39.137803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":156,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.841367+0000","last_change":"2024-09-18T07:33:27.006545+0000","last_active":"2024-09-18T07:33:57.841367+0000","last_peered":"2024-09-18T07:33:57.841367+0000","last_clean":"2024-09-18T07:33:57.841367+0000","last_became_active":"2024-09-18T07:30:56.286040+0000","last_became_peered":"2024-09-18T07:30:56.286040+0000","last_unstale":"2024-09-18T07:33:57.841367+0000","last_undegraded":"2024-09-18T07:33:57.841367+0000","last_fullsized":"2024-09-18T07:33:57.841367+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.006488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.006488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.614689+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":177,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834545+0000","last_change":"2024-09-18T07:33:22.032321+0000","last_active":"2024-09-18T07:33:57.834545+0000","last_peered":"2024-09-18T07:33:57.834545+0000","last_clean":"2024-09-18T07:33:57.834545+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:33:57.834545+0000","last_undegraded":"2024-09-18T07:33:57.834545+0000","last_fullsized":"2024-09-18T07:33:57.834545+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:22.032110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.032110+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.263345+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":149,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:59.829707+0000","last_change":"2024-09-18T07:33:59.829707+0000","last_active":"2024-09-18T07:33:59.829707+0000","last_peered":"2024-09-18T07:33:59.829707+0000","last_clean":"2024-09-18T07:33:59.829707+0000","last_became_active":"2024-09-18T07:31:00.331224+0000","last_became_peered":"2024-09-18T07:31:00.331224+0000","last_unstale":"2024-09-18T07:33:59.829707+0000","last_undegraded":"2024-09-18T07:33:59.829707+0000","last_fullsized":"2024-09-18T07:33:59.829707+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:59.829644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:59.829644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:12.614654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":156,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834414+0000","last_change":"2024-09-18T07:33:21.338716+0000","last_active":"2024-09-18T07:33:57.834414+0000","last_peered":"2024-09-18T07:33:57.834414+0000","last_clean":"2024-09-18T07:33:57.834414+0000","last_became_active":"2024-09-18T07:30:56.285882+0000","last_became_peered":"2024-09-18T07:30:56.285882+0000","last_unstale":"2024-09-18T07:33:57.834414+0000","last_undegraded":"2024-09-18T07:33:57.834414+0000","last_fullsized":"2024-09-18T07:33:57.834414+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.338669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.338669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:26.027193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":169,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.821801+0000","last_change":"2024-09-18T07:32:02.002919+0000","last_active":"2024-09-18T07:33:57.821801+0000","last_peered":"2024-09-18T07:33:57.821801+0000","last_clean":"2024-09-18T07:33:57.821801+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:33:57.821801+0000","last_undegraded":"2024-09-18T07:33:57.821801+0000","last_fullsized":"2024-09-18T07:33:57.821801+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.002811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.002811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":147,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834596+0000","last_change":"2024-09-18T07:32:30.842423+0000","last_active":"2024-09-18T07:33:57.834596+0000","last_peered":"2024-09-18T07:33:57.834596+0000","last_clean":"2024-09-18T07:33:57.834596+0000","last_became_active":"2024-09-18T07:32:30.841963+0000","last_became_peered":"2024-09-18T07:32:30.841963+0000","last_unstale":"2024-09-18T07:33:57.834596+0000","last_undegraded":"2024-09-18T07:33:57.834596+0000","last_fullsized":"2024-09-18T07:33:57.834596+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.385503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.827839+0000","last_change":"2024-09-18T07:32:30.840094+0000","last_active":"2024-09-18T07:33:57.827839+0000","last_peered":"2024-09-18T07:33:57.827839+0000","last_clean":"2024-09-18T07:33:57.827839+0000","last_became_active":"2024-09-18T07:32:30.839724+0000","last_became_peered":"2024-09-18T07:32:30.839724+0000","last_unstale":"2024-09-18T07:33:57.827839+0000","last_undegraded":"2024-09-18T07:33:57.827839+0000","last_fullsized":"2024-09-18T07:33:57.827839+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":148,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834155+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:33:57.834155+0000","last_peered":"2024-09-18T07:33:57.834155+0000","last_clean":"2024-09-18T07:33:57.834155+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:33:57.834155+0000","last_undegraded":"2024-09-18T07:33:57.834155+0000","last_fullsized":"2024-09-18T07:33:57.834155+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:15.347368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":155,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.821843+0000","last_change":"2024-09-18T07:30:58.317041+0000","last_active":"2024-09-18T07:33:57.821843+0000","last_peered":"2024-09-18T07:33:57.821843+0000","last_clean":"2024-09-18T07:33:57.821843+0000","last_became_active":"2024-09-18T07:30:58.316479+0000","last_became_peered":"2024-09-18T07:30:58.316479+0000","last_unstale":"2024-09-18T07:33:57.821843+0000","last_undegraded":"2024-09-18T07:33:57.821843+0000","last_fullsized":"2024-09-18T07:33:57.821843+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:41.690380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":83,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.821595+0000","last_change":"2024-09-18T07:32:30.834493+0000","last_active":"2024-09-18T07:33:57.821595+0000","last_peered":"2024-09-18T07:33:57.821595+0000","last_clean":"2024-09-18T07:33:57.821595+0000","last_became_active":"2024-09-18T07:32:30.834192+0000","last_became_peered":"2024-09-18T07:32:30.834192+0000","last_unstale":"2024-09-18T07:33:57.821595+0000","last_undegraded":"2024-09-18T07:33:57.821595+0000","last_fullsized":"2024-09-18T07:33:57.821595+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.897464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":152,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.833791+0000","last_change":"2024-09-18T07:33:51.759330+0000","last_active":"2024-09-18T07:33:57.833791+0000","last_peered":"2024-09-18T07:33:57.833791+0000","last_clean":"2024-09-18T07:33:57.833791+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:33:57.833791+0000","last_undegraded":"2024-09-18T07:33:57.833791+0000","last_fullsized":"2024-09-18T07:33:57.833791+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:51.759259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:51.759259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:18.399187+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":145,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.822302+0000","last_change":"2024-09-18T07:32:06.976159+0000","last_active":"2024-09-18T07:33:57.822302+0000","last_peered":"2024-09-18T07:33:57.822302+0000","last_clean":"2024-09-18T07:33:57.822302+0000","last_became_active":"2024-09-18T07:31:02.490014+0000","last_became_peered":"2024-09-18T07:31:02.490014+0000","last_unstale":"2024-09-18T07:33:57.822302+0000","last_undegraded":"2024-09-18T07:33:57.822302+0000","last_fullsized":"2024-09-18T07:33:57.822302+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:06.976113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:06.976113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":163,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.829357+0000","last_change":"2024-09-18T07:33:53.330384+0000","last_active":"2024-09-18T07:33:57.829357+0000","last_peered":"2024-09-18T07:33:57.829357+0000","last_clean":"2024-09-18T07:33:57.829357+0000","last_became_active":"2024-09-18T07:32:30.839952+0000","last_became_peered":"2024-09-18T07:32:30.839952+0000","last_unstale":"2024-09-18T07:33:57.829357+0000","last_undegraded":"2024-09-18T07:33:57.829357+0000","last_fullsized":"2024-09-18T07:33:57.829357+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:53.330304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:53.330304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:58.692089+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":186,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:59.921943+0000","last_change":"2024-09-18T07:33:59.921943+0000","last_active":"2024-09-18T07:33:59.921943+0000","last_peered":"2024-09-18T07:33:59.921943+0000","last_clean":"2024-09-18T07:33:59.921943+0000","last_became_active":"2024-09-18T07:30:58.316266+0000","last_became_peered":"2024-09-18T07:30:58.316266+0000","last_unstale":"2024-09-18T07:33:59.921943+0000","last_undegraded":"2024-09-18T07:33:59.921943+0000","last_fullsized":"2024-09-18T07:33:59.921943+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:59.921856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:59.921856+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:28.827554+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":149,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.842239+0000","last_change":"2024-09-18T07:33:30.955441+0000","last_active":"2024-09-18T07:33:57.842239+0000","last_peered":"2024-09-18T07:33:57.842239+0000","last_clean":"2024-09-18T07:33:57.842239+0000","last_became_active":"2024-09-18T07:31:00.337474+0000","last_became_peered":"2024-09-18T07:31:00.337474+0000","last_unstale":"2024-09-18T07:33:57.842239+0000","last_undegraded":"2024-09-18T07:33:57.842239+0000","last_fullsized":"2024-09-18T07:33:57.842239+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.955213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.955213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.470164+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":144,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.840956+0000","last_change":"2024-09-18T07:33:26.004195+0000","last_active":"2024-09-18T07:33:57.840956+0000","last_peered":"2024-09-18T07:33:57.840956+0000","last_clean":"2024-09-18T07:33:57.840956+0000","last_became_active":"2024-09-18T07:31:02.492348+0000","last_became_peered":"2024-09-18T07:31:02.492348+0000","last_unstale":"2024-09-18T07:33:57.840956+0000","last_undegraded":"2024-09-18T07:33:57.840956+0000","last_fullsized":"2024-09-18T07:33:57.840956+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:26.003425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:26.003425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:38.529586+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":157,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.821904+0000","last_change":"2024-09-18T07:33:53.756188+0000","last_active":"2024-09-18T07:33:57.821904+0000","last_peered":"2024-09-18T07:33:57.821904+0000","last_clean":"2024-09-18T07:33:57.821904+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:33:57.821904+0000","last_undegraded":"2024-09-18T07:33:57.821904+0000","last_fullsized":"2024-09-18T07:33:57.821904+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:53.756119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:53.756119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:19.988286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":167,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.840505+0000","last_change":"2024-09-18T07:30:58.319115+0000","last_active":"2024-09-18T07:33:57.840505+0000","last_peered":"2024-09-18T07:33:57.840505+0000","last_clean":"2024-09-18T07:33:57.840505+0000","last_became_active":"2024-09-18T07:30:58.316380+0000","last_became_peered":"2024-09-18T07:30:58.316380+0000","last_unstale":"2024-09-18T07:33:57.840505+0000","last_undegraded":"2024-09-18T07:33:57.840505+0000","last_fullsized":"2024-09-18T07:33:57.840505+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.013089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":141,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.821939+0000","last_change":"2024-09-18T07:31:00.334313+0000","last_active":"2024-09-18T07:33:57.821939+0000","last_peered":"2024-09-18T07:33:57.821939+0000","last_clean":"2024-09-18T07:33:57.821939+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:33:57.821939+0000","last_undegraded":"2024-09-18T07:33:57.821939+0000","last_fullsized":"2024-09-18T07:33:57.821939+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.867220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":107,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834044+0000","last_change":"2024-09-18T07:33:33.565391+0000","last_active":"2024-09-18T07:33:57.834044+0000","last_peered":"2024-09-18T07:33:57.834044+0000","last_clean":"2024-09-18T07:33:57.834044+0000","last_became_active":"2024-09-18T07:32:30.840386+0000","last_became_peered":"2024-09-18T07:32:30.840386+0000","last_unstale":"2024-09-18T07:33:57.834044+0000","last_undegraded":"2024-09-18T07:33:57.834044+0000","last_fullsized":"2024-09-18T07:33:57.834044+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:33.565334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.565334+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:02.676072+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":166,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.835909+0000","last_change":"2024-09-18T07:33:32.976203+0000","last_active":"2024-09-18T07:33:57.835909+0000","last_peered":"2024-09-18T07:33:57.835909+0000","last_clean":"2024-09-18T07:33:57.835909+0000","last_became_active":"2024-09-18T07:30:56.283705+0000","last_became_peered":"2024-09-18T07:30:56.283705+0000","last_unstale":"2024-09-18T07:33:57.835909+0000","last_undegraded":"2024-09-18T07:33:57.835909+0000","last_fullsized":"2024-09-18T07:33:57.835909+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:32.976129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.976129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:54.706133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":186,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834870+0000","last_change":"2024-09-18T07:33:36.118423+0000","last_active":"2024-09-18T07:33:57.834870+0000","last_peered":"2024-09-18T07:33:57.834870+0000","last_clean":"2024-09-18T07:33:57.834870+0000","last_became_active":"2024-09-18T07:32:30.860930+0000","last_became_peered":"2024-09-18T07:32:30.860930+0000","last_unstale":"2024-09-18T07:33:57.834870+0000","last_undegraded":"2024-09-18T07:33:57.834870+0000","last_fullsized":"2024-09-18T07:33:57.834870+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:33:36.118340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.118340+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:36.957408+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"94'9","reported_seq":208,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:59.647736+0000","last_change":"2024-09-18T07:33:59.647674+0000","last_active":"2024-09-18T07:33:59.647736+0000","last_peered":"2024-09-18T07:33:59.647736+0000","last_clean":"2024-09-18T07:33:59.647736+0000","last_became_active":"2024-09-18T07:32:30.839828+0000","last_became_peered":"2024-09-18T07:32:30.839828+0000","last_unstale":"2024-09-18T07:33:59.647736+0000","last_undegraded":"2024-09-18T07:33:59.647736+0000","last_fullsized":"2024-09-18T07:33:59.647736+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-09-18T07:33:59.647617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:59.647617+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:03.386170+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"129'7492","reported_seq":11551,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:01.439314+0000","last_change":"2024-09-18T07:33:32.018562+0000","last_active":"2024-09-18T07:34:01.439314+0000","last_peered":"2024-09-18T07:34:01.439314+0000","last_clean":"2024-09-18T07:34:01.439314+0000","last_became_active":"2024-09-18T07:32:54.163740+0000","last_became_peered":"2024-09-18T07:32:54.163740+0000","last_unstale":"2024-09-18T07:34:01.439314+0000","last_undegraded":"2024-09-18T07:34:01.439314+0000","last_fullsized":"2024-09-18T07:34:01.439314+0000","mapping_epoch":102,"log_start":"92'1900","ondisk_log_start":"92'1900","created":81,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'5666","last_scrub_stamp":"2024-09-18T07:33:32.018370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.018370+0000","objects_scrubbed":5641,"log_size":5592,"log_dups_size":0,"ondisk_log_size":5592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:57.391560+0000","scrub_duration":1289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359268352,"num_objects":5482,"num_object_clones":0,"num_object_copies":16446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7492,"num_write_kb":415168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":127074304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":170,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.824520+0000","last_change":"2024-09-18T07:30:58.312916+0000","last_active":"2024-09-18T07:33:57.824520+0000","last_peered":"2024-09-18T07:33:57.824520+0000","last_clean":"2024-09-18T07:33:57.824520+0000","last_became_active":"2024-09-18T07:30:58.312627+0000","last_became_peered":"2024-09-18T07:30:58.312627+0000","last_unstale":"2024-09-18T07:33:57.824520+0000","last_undegraded":"2024-09-18T07:33:57.824520+0000","last_fullsized":"2024-09-18T07:33:57.824520+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:15.056322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":171,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:01.294683+0000","last_change":"2024-09-18T07:34:01.294683+0000","last_active":"2024-09-18T07:34:01.294683+0000","last_peered":"2024-09-18T07:34:01.294683+0000","last_clean":"2024-09-18T07:34:01.294683+0000","last_became_active":"2024-09-18T07:32:30.838348+0000","last_became_peered":"2024-09-18T07:32:30.838348+0000","last_unstale":"2024-09-18T07:34:01.294683+0000","last_undegraded":"2024-09-18T07:34:01.294683+0000","last_fullsized":"2024-09-18T07:34:01.294683+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:01.294615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:01.294615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:04.638687+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":195,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.822399+0000","last_change":"2024-09-18T07:33:33.706253+0000","last_active":"2024-09-18T07:33:57.822399+0000","last_peered":"2024-09-18T07:33:57.822399+0000","last_clean":"2024-09-18T07:33:57.822399+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:33:57.822399+0000","last_undegraded":"2024-09-18T07:33:57.822399+0000","last_fullsized":"2024-09-18T07:33:57.822399+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:33:33.706201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.706201+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.605129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":136,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834153+0000","last_change":"2024-09-18T07:31:02.351412+0000","last_active":"2024-09-18T07:33:57.834153+0000","last_peered":"2024-09-18T07:33:57.834153+0000","last_clean":"2024-09-18T07:33:57.834153+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:33:57.834153+0000","last_undegraded":"2024-09-18T07:33:57.834153+0000","last_fullsized":"2024-09-18T07:33:57.834153+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.333040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"129'7371","reported_seq":7476,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:01.553614+0000","last_change":"2024-09-18T07:32:08.065129+0000","last_active":"2024-09-18T07:34:01.553614+0000","last_peered":"2024-09-18T07:34:01.553614+0000","last_clean":"2024-09-18T07:34:01.553614+0000","last_became_active":"2024-09-18T07:32:08.064876+0000","last_became_peered":"2024-09-18T07:32:08.064876+0000","last_unstale":"2024-09-18T07:34:01.553614+0000","last_undegraded":"2024-09-18T07:34:01.553614+0000","last_fullsized":"2024-09-18T07:34:01.553614+0000","mapping_epoch":81,"log_start":"101'3200","ondisk_log_start":"101'3200","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4171,"log_dups_size":0,"ondisk_log_size":4171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361693184,"num_objects":5519,"num_object_clones":0,"num_object_copies":16557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7371,"num_write_kb":412480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":209,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.835190+0000","last_change":"2024-09-18T07:32:30.876643+0000","last_active":"2024-09-18T07:33:57.835190+0000","last_peered":"2024-09-18T07:33:57.835190+0000","last_clean":"2024-09-18T07:33:57.835190+0000","last_became_active":"2024-09-18T07:32:30.852501+0000","last_became_peered":"2024-09-18T07:32:30.852501+0000","last_unstale":"2024-09-18T07:33:57.835190+0000","last_undegraded":"2024-09-18T07:33:57.835190+0000","last_fullsized":"2024-09-18T07:33:57.835190+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:32:00.660377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:00.660377+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":163,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.841837+0000","last_change":"2024-09-18T07:32:30.858438+0000","last_active":"2024-09-18T07:33:57.841837+0000","last_peered":"2024-09-18T07:33:57.841837+0000","last_clean":"2024-09-18T07:33:57.841837+0000","last_became_active":"2024-09-18T07:32:30.858176+0000","last_became_peered":"2024-09-18T07:32:30.858176+0000","last_unstale":"2024-09-18T07:33:57.841837+0000","last_undegraded":"2024-09-18T07:33:57.841837+0000","last_fullsized":"2024-09-18T07:33:57.841837+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.665591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.665591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":148,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.821644+0000","last_change":"2024-09-18T07:32:30.851490+0000","last_active":"2024-09-18T07:33:57.821644+0000","last_peered":"2024-09-18T07:33:57.821644+0000","last_clean":"2024-09-18T07:33:57.821644+0000","last_became_active":"2024-09-18T07:32:30.851161+0000","last_became_peered":"2024-09-18T07:32:30.851161+0000","last_unstale":"2024-09-18T07:33:57.821644+0000","last_undegraded":"2024-09-18T07:33:57.821644+0000","last_fullsized":"2024-09-18T07:33:57.821644+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:47.861786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":137,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.821738+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:33:57.821738+0000","last_peered":"2024-09-18T07:33:57.821738+0000","last_clean":"2024-09-18T07:33:57.821738+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:33:57.821738+0000","last_undegraded":"2024-09-18T07:33:57.821738+0000","last_fullsized":"2024-09-18T07:33:57.821738+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"129'7506","reported_seq":7612,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:01.512001+0000","last_change":"2024-09-18T07:32:08.066116+0000","last_active":"2024-09-18T07:34:01.512001+0000","last_peered":"2024-09-18T07:34:01.512001+0000","last_clean":"2024-09-18T07:34:01.512001+0000","last_became_active":"2024-09-18T07:32:08.065844+0000","last_became_peered":"2024-09-18T07:32:08.065844+0000","last_unstale":"2024-09-18T07:34:01.512001+0000","last_undegraded":"2024-09-18T07:34:01.512001+0000","last_fullsized":"2024-09-18T07:34:01.512001+0000","mapping_epoch":81,"log_start":"101'3300","ondisk_log_start":"101'3300","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4206,"log_dups_size":0,"ondisk_log_size":4206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360710144,"num_objects":5504,"num_object_clones":0,"num_object_copies":16512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7506,"num_write_kb":416320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":166,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.821621+0000","last_change":"2024-09-18T07:30:58.309938+0000","last_active":"2024-09-18T07:33:57.821621+0000","last_peered":"2024-09-18T07:33:57.821621+0000","last_clean":"2024-09-18T07:33:57.821621+0000","last_became_active":"2024-09-18T07:30:58.309649+0000","last_became_peered":"2024-09-18T07:30:58.309649+0000","last_unstale":"2024-09-18T07:33:57.821621+0000","last_undegraded":"2024-09-18T07:33:57.821621+0000","last_fullsized":"2024-09-18T07:33:57.821621+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":157,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.835136+0000","last_change":"2024-09-18T07:33:24.213432+0000","last_active":"2024-09-18T07:33:57.835136+0000","last_peered":"2024-09-18T07:33:57.835136+0000","last_clean":"2024-09-18T07:33:57.835136+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:33:57.835136+0000","last_undegraded":"2024-09-18T07:33:57.835136+0000","last_fullsized":"2024-09-18T07:33:57.835136+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:24.213356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:24.213356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:52.949578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":186,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.835017+0000","last_change":"2024-09-18T07:31:00.336059+0000","last_active":"2024-09-18T07:33:57.835017+0000","last_peered":"2024-09-18T07:33:57.835017+0000","last_clean":"2024-09-18T07:33:57.835017+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:33:57.835017+0000","last_undegraded":"2024-09-18T07:33:57.835017+0000","last_fullsized":"2024-09-18T07:33:57.835017+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:17.963381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":91,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.826923+0000","last_change":"2024-09-18T07:32:54.740291+0000","last_active":"2024-09-18T07:33:57.826923+0000","last_peered":"2024-09-18T07:33:57.826923+0000","last_clean":"2024-09-18T07:33:57.826923+0000","last_became_active":"2024-09-18T07:32:30.830346+0000","last_became_peered":"2024-09-18T07:32:30.830346+0000","last_unstale":"2024-09-18T07:33:57.826923+0000","last_undegraded":"2024-09-18T07:33:57.826923+0000","last_fullsized":"2024-09-18T07:33:57.826923+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:54.740249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:54.740249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.965276+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"129'7601","reported_seq":7708,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:01.389894+0000","last_change":"2024-09-18T07:32:08.062282+0000","last_active":"2024-09-18T07:34:01.389894+0000","last_peered":"2024-09-18T07:34:01.389894+0000","last_clean":"2024-09-18T07:34:01.389894+0000","last_became_active":"2024-09-18T07:32:08.061934+0000","last_became_peered":"2024-09-18T07:32:08.061934+0000","last_unstale":"2024-09-18T07:34:01.389894+0000","last_undegraded":"2024-09-18T07:34:01.389894+0000","last_fullsized":"2024-09-18T07:34:01.389894+0000","mapping_epoch":81,"log_start":"100'3200","ondisk_log_start":"100'3200","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368115712,"num_objects":5617,"num_object_clones":0,"num_object_copies":16851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7601,"num_write_kb":422976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":160,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.838125+0000","last_change":"2024-09-18T07:30:58.307470+0000","last_active":"2024-09-18T07:33:57.838125+0000","last_peered":"2024-09-18T07:33:57.838125+0000","last_clean":"2024-09-18T07:33:57.838125+0000","last_became_active":"2024-09-18T07:30:58.307035+0000","last_became_peered":"2024-09-18T07:30:58.307035+0000","last_unstale":"2024-09-18T07:33:57.838125+0000","last_undegraded":"2024-09-18T07:33:57.838125+0000","last_fullsized":"2024-09-18T07:33:57.838125+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:14.091632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":156,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834483+0000","last_change":"2024-09-18T07:33:29.471394+0000","last_active":"2024-09-18T07:33:57.834483+0000","last_peered":"2024-09-18T07:33:57.834483+0000","last_clean":"2024-09-18T07:33:57.834483+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:33:57.834483+0000","last_undegraded":"2024-09-18T07:33:57.834483+0000","last_fullsized":"2024-09-18T07:33:57.834483+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:29.471330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.471330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:58.117832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":194,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.828768+0000","last_change":"2024-09-18T07:33:52.317149+0000","last_active":"2024-09-18T07:33:57.828768+0000","last_peered":"2024-09-18T07:33:57.828768+0000","last_clean":"2024-09-18T07:33:57.828768+0000","last_became_active":"2024-09-18T07:31:00.336905+0000","last_became_peered":"2024-09-18T07:31:00.336905+0000","last_unstale":"2024-09-18T07:33:57.828768+0000","last_undegraded":"2024-09-18T07:33:57.828768+0000","last_fullsized":"2024-09-18T07:33:57.828768+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:33:52.317075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:52.317075+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:12.567039+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":136,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.827964+0000","last_change":"2024-09-18T07:31:02.348538+0000","last_active":"2024-09-18T07:33:57.827964+0000","last_peered":"2024-09-18T07:33:57.827964+0000","last_clean":"2024-09-18T07:33:57.827964+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:33:57.827964+0000","last_undegraded":"2024-09-18T07:33:57.827964+0000","last_fullsized":"2024-09-18T07:33:57.827964+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.077245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"129'7468","reported_seq":7573,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:01.422366+0000","last_change":"2024-09-18T07:32:08.062584+0000","last_active":"2024-09-18T07:34:01.422366+0000","last_peered":"2024-09-18T07:34:01.422366+0000","last_clean":"2024-09-18T07:34:01.422366+0000","last_became_active":"2024-09-18T07:32:08.062129+0000","last_became_peered":"2024-09-18T07:32:08.062129+0000","last_unstale":"2024-09-18T07:34:01.422366+0000","last_undegraded":"2024-09-18T07:34:01.422366+0000","last_fullsized":"2024-09-18T07:34:01.422366+0000","mapping_epoch":81,"log_start":"96'2256","ondisk_log_start":"96'2256","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5212,"log_dups_size":0,"ondisk_log_size":5212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:50.732644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361758720,"num_objects":5520,"num_object_clones":0,"num_object_copies":16560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7468,"num_write_kb":415616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":205,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.828905+0000","last_change":"2024-09-18T07:32:30.859920+0000","last_active":"2024-09-18T07:33:57.828905+0000","last_peered":"2024-09-18T07:33:57.828905+0000","last_clean":"2024-09-18T07:33:57.828905+0000","last_became_active":"2024-09-18T07:32:30.842195+0000","last_became_peered":"2024-09-18T07:32:30.842195+0000","last_unstale":"2024-09-18T07:33:57.828905+0000","last_undegraded":"2024-09-18T07:33:57.828905+0000","last_fullsized":"2024-09-18T07:33:57.828905+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:18.104967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":149,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.837465+0000","last_change":"2024-09-18T07:30:56.284440+0000","last_active":"2024-09-18T07:33:57.837465+0000","last_peered":"2024-09-18T07:33:57.837465+0000","last_clean":"2024-09-18T07:33:57.837465+0000","last_became_active":"2024-09-18T07:30:56.284120+0000","last_became_peered":"2024-09-18T07:30:56.284120+0000","last_unstale":"2024-09-18T07:33:57.837465+0000","last_undegraded":"2024-09-18T07:33:57.837465+0000","last_fullsized":"2024-09-18T07:33:57.837465+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:06.088370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":148,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.826130+0000","last_change":"2024-09-18T07:33:21.834910+0000","last_active":"2024-09-18T07:33:57.826130+0000","last_peered":"2024-09-18T07:33:57.826130+0000","last_clean":"2024-09-18T07:33:57.826130+0000","last_became_active":"2024-09-18T07:31:00.334776+0000","last_became_peered":"2024-09-18T07:31:00.334776+0000","last_unstale":"2024-09-18T07:33:57.826130+0000","last_undegraded":"2024-09-18T07:33:57.826130+0000","last_fullsized":"2024-09-18T07:33:57.826130+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:24.513163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":137,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.836014+0000","last_change":"2024-09-18T07:31:02.490534+0000","last_active":"2024-09-18T07:33:57.836014+0000","last_peered":"2024-09-18T07:33:57.836014+0000","last_clean":"2024-09-18T07:33:57.836014+0000","last_became_active":"2024-09-18T07:31:02.490186+0000","last_became_peered":"2024-09-18T07:31:02.490186+0000","last_unstale":"2024-09-18T07:33:57.836014+0000","last_undegraded":"2024-09-18T07:33:57.836014+0000","last_fullsized":"2024-09-18T07:33:57.836014+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:19.074713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"129'7377","reported_seq":7545,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:01.396881+0000","last_change":"2024-09-18T07:33:31.181076+0000","last_active":"2024-09-18T07:34:01.396881+0000","last_peered":"2024-09-18T07:34:01.396881+0000","last_clean":"2024-09-18T07:34:01.396881+0000","last_became_active":"2024-09-18T07:32:08.064184+0000","last_became_peered":"2024-09-18T07:32:08.064184+0000","last_unstale":"2024-09-18T07:34:01.396881+0000","last_undegraded":"2024-09-18T07:34:01.396881+0000","last_fullsized":"2024-09-18T07:34:01.396881+0000","mapping_epoch":81,"log_start":"92'1800","ondisk_log_start":"92'1800","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'5435","last_scrub_stamp":"2024-09-18T07:33:31.180958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:33:31.180958+0000","objects_scrubbed":5388,"log_size":5577,"log_dups_size":0,"ondisk_log_size":5577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.901559+0000","scrub_duration":1433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350420992,"num_objects":5347,"num_object_clones":0,"num_object_copies":16041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7377,"num_write_kb":407168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":231,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.837415+0000","last_change":"2024-09-18T07:30:58.317477+0000","last_active":"2024-09-18T07:33:57.837415+0000","last_peered":"2024-09-18T07:33:57.837415+0000","last_clean":"2024-09-18T07:33:57.837415+0000","last_became_active":"2024-09-18T07:30:58.316812+0000","last_became_peered":"2024-09-18T07:30:58.316812+0000","last_unstale":"2024-09-18T07:33:57.837415+0000","last_undegraded":"2024-09-18T07:33:57.837415+0000","last_fullsized":"2024-09-18T07:33:57.837415+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:09.123487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":149,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.822112+0000","last_change":"2024-09-18T07:30:56.285353+0000","last_active":"2024-09-18T07:33:57.822112+0000","last_peered":"2024-09-18T07:33:57.822112+0000","last_clean":"2024-09-18T07:33:57.822112+0000","last_became_active":"2024-09-18T07:30:56.285032+0000","last_became_peered":"2024-09-18T07:30:56.285032+0000","last_unstale":"2024-09-18T07:33:57.822112+0000","last_undegraded":"2024-09-18T07:33:57.822112+0000","last_fullsized":"2024-09-18T07:33:57.822112+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":158,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834068+0000","last_change":"2024-09-18T07:30:58.299308+0000","last_active":"2024-09-18T07:33:57.834068+0000","last_peered":"2024-09-18T07:33:57.834068+0000","last_clean":"2024-09-18T07:33:57.834068+0000","last_became_active":"2024-09-18T07:30:55.283292+0000","last_became_peered":"2024-09-18T07:30:55.283292+0000","last_unstale":"2024-09-18T07:33:57.834068+0000","last_undegraded":"2024-09-18T07:33:57.834068+0000","last_fullsized":"2024-09-18T07:33:57.834068+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:19.297747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":140,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.830559+0000","last_change":"2024-09-18T07:31:00.332738+0000","last_active":"2024-09-18T07:33:57.830559+0000","last_peered":"2024-09-18T07:33:57.830559+0000","last_clean":"2024-09-18T07:33:57.830559+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:33:57.830559+0000","last_undegraded":"2024-09-18T07:33:57.830559+0000","last_fullsized":"2024-09-18T07:33:57.830559+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.840931+0000","last_change":"2024-09-18T07:32:30.854643+0000","last_active":"2024-09-18T07:33:57.840931+0000","last_peered":"2024-09-18T07:33:57.840931+0000","last_clean":"2024-09-18T07:33:57.840931+0000","last_became_active":"2024-09-18T07:32:30.853087+0000","last_became_peered":"2024-09-18T07:32:30.853087+0000","last_unstale":"2024-09-18T07:33:57.840931+0000","last_undegraded":"2024-09-18T07:33:57.840931+0000","last_fullsized":"2024-09-18T07:33:57.840931+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:45.131703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"129'7581","reported_seq":7687,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:01.663008+0000","last_change":"2024-09-18T07:32:08.063002+0000","last_active":"2024-09-18T07:34:01.663008+0000","last_peered":"2024-09-18T07:34:01.663008+0000","last_clean":"2024-09-18T07:34:01.663008+0000","last_became_active":"2024-09-18T07:32:08.062715+0000","last_became_peered":"2024-09-18T07:32:08.062715+0000","last_unstale":"2024-09-18T07:34:01.663008+0000","last_undegraded":"2024-09-18T07:34:01.663008+0000","last_fullsized":"2024-09-18T07:34:01.663008+0000","mapping_epoch":81,"log_start":"100'3009","ondisk_log_start":"100'3009","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4572,"log_dups_size":0,"ondisk_log_size":4572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363397120,"num_objects":5545,"num_object_clones":0,"num_object_copies":16635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7581,"num_write_kb":420032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":172,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.824624+0000","last_change":"2024-09-18T07:30:58.310789+0000","last_active":"2024-09-18T07:33:57.824624+0000","last_peered":"2024-09-18T07:33:57.824624+0000","last_clean":"2024-09-18T07:33:57.824624+0000","last_became_active":"2024-09-18T07:30:58.310490+0000","last_became_peered":"2024-09-18T07:30:58.310490+0000","last_unstale":"2024-09-18T07:33:57.824624+0000","last_undegraded":"2024-09-18T07:33:57.824624+0000","last_fullsized":"2024-09-18T07:33:57.824624+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.090903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":157,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.835595+0000","last_change":"2024-09-18T07:33:23.038159+0000","last_active":"2024-09-18T07:33:57.835595+0000","last_peered":"2024-09-18T07:33:57.835595+0000","last_clean":"2024-09-18T07:33:57.835595+0000","last_became_active":"2024-09-18T07:30:56.284803+0000","last_became_peered":"2024-09-18T07:30:56.284803+0000","last_unstale":"2024-09-18T07:33:57.835595+0000","last_undegraded":"2024-09-18T07:33:57.835595+0000","last_fullsized":"2024-09-18T07:33:57.835595+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:23.037963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:23.037963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:45.045160+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":166,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.836001+0000","last_change":"2024-09-18T07:32:02.418590+0000","last_active":"2024-09-18T07:33:57.836001+0000","last_peered":"2024-09-18T07:33:57.836001+0000","last_clean":"2024-09-18T07:33:57.836001+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:33:57.836001+0000","last_undegraded":"2024-09-18T07:33:57.836001+0000","last_fullsized":"2024-09-18T07:33:57.836001+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:32:02.418507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.418507+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:39.933773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":148,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.840406+0000","last_change":"2024-09-18T07:32:30.837706+0000","last_active":"2024-09-18T07:33:57.840406+0000","last_peered":"2024-09-18T07:33:57.840406+0000","last_clean":"2024-09-18T07:33:57.840406+0000","last_became_active":"2024-09-18T07:32:30.837243+0000","last_became_peered":"2024-09-18T07:32:30.837243+0000","last_unstale":"2024-09-18T07:33:57.840406+0000","last_undegraded":"2024-09-18T07:33:57.840406+0000","last_fullsized":"2024-09-18T07:33:57.840406+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:09.666733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":115,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.826563+0000","last_change":"2024-09-18T07:33:57.655849+0000","last_active":"2024-09-18T07:33:57.826563+0000","last_peered":"2024-09-18T07:33:57.826563+0000","last_clean":"2024-09-18T07:33:57.826563+0000","last_became_active":"2024-09-18T07:32:30.839186+0000","last_became_peered":"2024-09-18T07:32:30.839186+0000","last_unstale":"2024-09-18T07:33:57.826563+0000","last_undegraded":"2024-09-18T07:33:57.826563+0000","last_fullsized":"2024-09-18T07:33:57.826563+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:57.655797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.655797+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:04.938177+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"129'7528","reported_seq":11451,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:01.526562+0000","last_change":"2024-09-18T07:33:01.225875+0000","last_active":"2024-09-18T07:34:01.526562+0000","last_peered":"2024-09-18T07:34:01.526562+0000","last_clean":"2024-09-18T07:34:01.526562+0000","last_became_active":"2024-09-18T07:33:01.221136+0000","last_became_peered":"2024-09-18T07:33:01.221136+0000","last_unstale":"2024-09-18T07:34:01.526562+0000","last_undegraded":"2024-09-18T07:34:01.526562+0000","last_fullsized":"2024-09-18T07:34:01.526562+0000","mapping_epoch":107,"log_start":"92'1800","ondisk_log_start":"92'1800","created":81,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5728,"log_dups_size":0,"ondisk_log_size":5728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:38.955324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365035520,"num_objects":5570,"num_object_clones":0,"num_object_copies":16710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7528,"num_write_kb":419136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1904,"num_bytes_recovered":124780544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":183,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834950+0000","last_change":"2024-09-18T07:33:27.040227+0000","last_active":"2024-09-18T07:33:57.834950+0000","last_peered":"2024-09-18T07:33:57.834950+0000","last_clean":"2024-09-18T07:33:57.834950+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:33:57.834950+0000","last_undegraded":"2024-09-18T07:33:57.834950+0000","last_fullsized":"2024-09-18T07:33:57.834950+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:51.281723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":154,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.830361+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:33:57.830361+0000","last_peered":"2024-09-18T07:33:57.830361+0000","last_clean":"2024-09-18T07:33:57.830361+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:33:57.830361+0000","last_undegraded":"2024-09-18T07:33:57.830361+0000","last_fullsized":"2024-09-18T07:33:57.830361+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:43.132679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":221,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:00.081887+0000","last_change":"2024-09-18T07:32:30.859165+0000","last_active":"2024-09-18T07:34:00.081887+0000","last_peered":"2024-09-18T07:34:00.081887+0000","last_clean":"2024-09-18T07:34:00.081887+0000","last_became_active":"2024-09-18T07:32:30.851004+0000","last_became_peered":"2024-09-18T07:32:30.851004+0000","last_unstale":"2024-09-18T07:34:00.081887+0000","last_undegraded":"2024-09-18T07:34:00.081887+0000","last_fullsized":"2024-09-18T07:34:00.081887+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:04.076666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":97,"num_read_kb":92,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":148,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834769+0000","last_change":"2024-09-18T07:33:27.394247+0000","last_active":"2024-09-18T07:33:57.834769+0000","last_peered":"2024-09-18T07:33:57.834769+0000","last_clean":"2024-09-18T07:33:57.834769+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:33:57.834769+0000","last_undegraded":"2024-09-18T07:33:57.834769+0000","last_fullsized":"2024-09-18T07:33:57.834769+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.394203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.394203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:31.541179+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":153,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834628+0000","last_change":"2024-09-18T07:33:25.028842+0000","last_active":"2024-09-18T07:33:57.834628+0000","last_peered":"2024-09-18T07:33:57.834628+0000","last_clean":"2024-09-18T07:33:57.834628+0000","last_became_active":"2024-09-18T07:31:02.347390+0000","last_became_peered":"2024-09-18T07:31:02.347390+0000","last_unstale":"2024-09-18T07:33:57.834628+0000","last_undegraded":"2024-09-18T07:33:57.834628+0000","last_fullsized":"2024-09-18T07:33:57.834628+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:25.028609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:25.028609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:34.346431+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"129'7625","reported_seq":11598,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:01.634825+0000","last_change":"2024-09-18T07:32:59.213954+0000","last_active":"2024-09-18T07:34:01.634825+0000","last_peered":"2024-09-18T07:34:01.634825+0000","last_clean":"2024-09-18T07:34:01.634825+0000","last_became_active":"2024-09-18T07:32:59.204989+0000","last_became_peered":"2024-09-18T07:32:59.204989+0000","last_unstale":"2024-09-18T07:34:01.634825+0000","last_undegraded":"2024-09-18T07:34:01.634825+0000","last_fullsized":"2024-09-18T07:34:01.634825+0000","mapping_epoch":105,"log_start":"100'2990","ondisk_log_start":"100'2990","created":81,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4635,"log_dups_size":0,"ondisk_log_size":4635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370999296,"num_objects":5661,"num_object_clones":0,"num_object_copies":16983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7625,"num_write_kb":425152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1927,"num_bytes_recovered":126287872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"94'17","reported_seq":245,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.826643+0000","last_change":"2024-09-18T07:33:33.849354+0000","last_active":"2024-09-18T07:33:57.826643+0000","last_peered":"2024-09-18T07:33:57.826643+0000","last_clean":"2024-09-18T07:33:57.826643+0000","last_became_active":"2024-09-18T07:32:30.842449+0000","last_became_peered":"2024-09-18T07:32:30.842449+0000","last_unstale":"2024-09-18T07:33:57.826643+0000","last_undegraded":"2024-09-18T07:33:57.826643+0000","last_fullsized":"2024-09-18T07:33:57.826643+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'17","last_scrub_stamp":"2024-09-18T07:33:33.849285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.849285+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:53.616735+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":156,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.840375+0000","last_change":"2024-09-18T07:31:59.617046+0000","last_active":"2024-09-18T07:33:57.840375+0000","last_peered":"2024-09-18T07:33:57.840375+0000","last_clean":"2024-09-18T07:33:57.840375+0000","last_became_active":"2024-09-18T07:30:56.288149+0000","last_became_peered":"2024-09-18T07:30:56.288149+0000","last_unstale":"2024-09-18T07:33:57.840375+0000","last_undegraded":"2024-09-18T07:33:57.840375+0000","last_fullsized":"2024-09-18T07:33:57.840375+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:59.616988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:59.616988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":178,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.828229+0000","last_change":"2024-09-18T07:33:23.172950+0000","last_active":"2024-09-18T07:33:57.828229+0000","last_peered":"2024-09-18T07:33:57.828229+0000","last_clean":"2024-09-18T07:33:57.828229+0000","last_became_active":"2024-09-18T07:32:30.842828+0000","last_became_peered":"2024-09-18T07:32:30.842828+0000","last_unstale":"2024-09-18T07:33:57.828229+0000","last_undegraded":"2024-09-18T07:33:57.828229+0000","last_fullsized":"2024-09-18T07:33:57.828229+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:33:23.172848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:33:23.172848+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.696337+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":83,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.836876+0000","last_change":"2024-09-18T07:32:30.835729+0000","last_active":"2024-09-18T07:33:57.836876+0000","last_peered":"2024-09-18T07:33:57.836876+0000","last_clean":"2024-09-18T07:33:57.836876+0000","last_became_active":"2024-09-18T07:32:30.835105+0000","last_became_peered":"2024-09-18T07:32:30.835105+0000","last_unstale":"2024-09-18T07:33:57.836876+0000","last_undegraded":"2024-09-18T07:33:57.836876+0000","last_fullsized":"2024-09-18T07:33:57.836876+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:02.499041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.499041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.738806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":144,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834680+0000","last_change":"2024-09-18T07:33:22.358792+0000","last_active":"2024-09-18T07:33:57.834680+0000","last_peered":"2024-09-18T07:33:57.834680+0000","last_clean":"2024-09-18T07:33:57.834680+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:33:57.834680+0000","last_undegraded":"2024-09-18T07:33:57.834680+0000","last_fullsized":"2024-09-18T07:33:57.834680+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.358746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.358746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:26.289679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"129'7596","reported_seq":7703,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:01.400367+0000","last_change":"2024-09-18T07:32:08.063469+0000","last_active":"2024-09-18T07:34:01.400367+0000","last_peered":"2024-09-18T07:34:01.400367+0000","last_clean":"2024-09-18T07:34:01.400367+0000","last_became_active":"2024-09-18T07:32:08.063070+0000","last_became_peered":"2024-09-18T07:32:08.063070+0000","last_unstale":"2024-09-18T07:34:01.400367+0000","last_undegraded":"2024-09-18T07:34:01.400367+0000","last_fullsized":"2024-09-18T07:34:01.400367+0000","mapping_epoch":81,"log_start":"95'2000","ondisk_log_start":"95'2000","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:39.798722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361168920,"num_objects":5512,"num_object_clones":0,"num_object_copies":16536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5512,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7596,"num_write_kb":419393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":183,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.822428+0000","last_change":"2024-09-18T07:32:26.793474+0000","last_active":"2024-09-18T07:33:57.822428+0000","last_peered":"2024-09-18T07:33:57.822428+0000","last_clean":"2024-09-18T07:33:57.822428+0000","last_became_active":"2024-09-18T07:32:26.785711+0000","last_became_peered":"2024-09-18T07:32:26.785711+0000","last_unstale":"2024-09-18T07:33:57.822428+0000","last_undegraded":"2024-09-18T07:33:57.822428+0000","last_fullsized":"2024-09-18T07:33:57.822428+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":156,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.821862+0000","last_change":"2024-09-18T07:32:30.829352+0000","last_active":"2024-09-18T07:33:57.821862+0000","last_peered":"2024-09-18T07:33:57.821862+0000","last_clean":"2024-09-18T07:33:57.821862+0000","last_became_active":"2024-09-18T07:32:30.828736+0000","last_became_peered":"2024-09-18T07:32:30.828736+0000","last_unstale":"2024-09-18T07:33:57.821862+0000","last_undegraded":"2024-09-18T07:33:57.821862+0000","last_fullsized":"2024-09-18T07:33:57.821862+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:19.900370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":148,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834654+0000","last_change":"2024-09-18T07:33:36.582039+0000","last_active":"2024-09-18T07:33:57.834654+0000","last_peered":"2024-09-18T07:33:57.834654+0000","last_clean":"2024-09-18T07:33:57.834654+0000","last_became_active":"2024-09-18T07:31:00.336172+0000","last_became_peered":"2024-09-18T07:31:00.336172+0000","last_unstale":"2024-09-18T07:33:57.834654+0000","last_undegraded":"2024-09-18T07:33:57.834654+0000","last_fullsized":"2024-09-18T07:33:57.834654+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:36.581998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.581998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:04.288035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":144,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.839169+0000","last_change":"2024-09-18T07:33:20.990542+0000","last_active":"2024-09-18T07:33:57.839169+0000","last_peered":"2024-09-18T07:33:57.839169+0000","last_clean":"2024-09-18T07:33:57.839169+0000","last_became_active":"2024-09-18T07:31:02.490953+0000","last_became_peered":"2024-09-18T07:31:02.490953+0000","last_unstale":"2024-09-18T07:33:57.839169+0000","last_undegraded":"2024-09-18T07:33:57.839169+0000","last_fullsized":"2024-09-18T07:33:57.839169+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:20.990401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:20.990401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.208665+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"129'7552","reported_seq":7659,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:01.499046+0000","last_change":"2024-09-18T07:32:08.060282+0000","last_active":"2024-09-18T07:34:01.499046+0000","last_peered":"2024-09-18T07:34:01.499046+0000","last_clean":"2024-09-18T07:34:01.499046+0000","last_became_active":"2024-09-18T07:32:08.059971+0000","last_became_peered":"2024-09-18T07:32:08.059971+0000","last_unstale":"2024-09-18T07:34:01.499046+0000","last_undegraded":"2024-09-18T07:34:01.499046+0000","last_fullsized":"2024-09-18T07:34:01.499046+0000","mapping_epoch":81,"log_start":"92'1800","ondisk_log_start":"92'1800","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5752,"log_dups_size":0,"ondisk_log_size":5752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:45.960343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357040128,"num_objects":5448,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7552,"num_write_kb":416000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":181,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.822218+0000","last_change":"2024-09-18T07:33:56.797549+0000","last_active":"2024-09-18T07:33:57.822218+0000","last_peered":"2024-09-18T07:33:57.822218+0000","last_clean":"2024-09-18T07:33:57.822218+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:33:57.822218+0000","last_undegraded":"2024-09-18T07:33:57.822218+0000","last_fullsized":"2024-09-18T07:33:57.822218+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:33:56.797466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:56.797466+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:14.685704+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":148,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.826403+0000","last_change":"2024-09-18T07:30:56.284303+0000","last_active":"2024-09-18T07:33:57.826403+0000","last_peered":"2024-09-18T07:33:57.826403+0000","last_clean":"2024-09-18T07:33:57.826403+0000","last_became_active":"2024-09-18T07:30:56.284020+0000","last_became_peered":"2024-09-18T07:30:56.284020+0000","last_unstale":"2024-09-18T07:33:57.826403+0000","last_undegraded":"2024-09-18T07:33:57.826403+0000","last_fullsized":"2024-09-18T07:33:57.826403+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.835065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":149,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834728+0000","last_change":"2024-09-18T07:32:05.239202+0000","last_active":"2024-09-18T07:33:57.834728+0000","last_peered":"2024-09-18T07:33:57.834728+0000","last_clean":"2024-09-18T07:33:57.834728+0000","last_became_active":"2024-09-18T07:31:00.332836+0000","last_became_peered":"2024-09-18T07:31:00.332836+0000","last_unstale":"2024-09-18T07:33:57.834728+0000","last_undegraded":"2024-09-18T07:33:57.834728+0000","last_fullsized":"2024-09-18T07:33:57.834728+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:05.239160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.239160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:37.915616+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":145,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.836268+0000","last_change":"2024-09-18T07:33:21.860561+0000","last_active":"2024-09-18T07:33:57.836268+0000","last_peered":"2024-09-18T07:33:57.836268+0000","last_clean":"2024-09-18T07:33:57.836268+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:33:57.836268+0000","last_undegraded":"2024-09-18T07:33:57.836268+0000","last_fullsized":"2024-09-18T07:33:57.836268+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.860495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.860495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:51.431373+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"129'7570","reported_seq":11511,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:01.647622+0000","last_change":"2024-09-18T07:33:14.766074+0000","last_active":"2024-09-18T07:34:01.647622+0000","last_peered":"2024-09-18T07:34:01.647622+0000","last_clean":"2024-09-18T07:34:01.647622+0000","last_became_active":"2024-09-18T07:33:14.762251+0000","last_became_peered":"2024-09-18T07:33:14.762251+0000","last_unstale":"2024-09-18T07:34:01.647622+0000","last_undegraded":"2024-09-18T07:34:01.647622+0000","last_fullsized":"2024-09-18T07:34:01.647622+0000","mapping_epoch":115,"log_start":"100'2930","ondisk_log_start":"100'2930","created":81,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4640,"log_dups_size":0,"ondisk_log_size":4640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365297664,"num_objects":5574,"num_object_clones":0,"num_object_copies":16722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7570,"num_write_kb":420608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1912,"num_bytes_recovered":125304832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":168,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.821974+0000","last_change":"2024-09-18T07:32:05.000363+0000","last_active":"2024-09-18T07:33:57.821974+0000","last_peered":"2024-09-18T07:33:57.821974+0000","last_clean":"2024-09-18T07:33:57.821974+0000","last_became_active":"2024-09-18T07:30:58.316423+0000","last_became_peered":"2024-09-18T07:30:58.316423+0000","last_unstale":"2024-09-18T07:33:57.821974+0000","last_undegraded":"2024-09-18T07:33:57.821974+0000","last_fullsized":"2024-09-18T07:33:57.821974+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:32:05.000310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.000310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":156,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.839647+0000","last_change":"2024-09-18T07:32:06.688493+0000","last_active":"2024-09-18T07:33:57.839647+0000","last_peered":"2024-09-18T07:33:57.839647+0000","last_clean":"2024-09-18T07:33:57.839647+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:33:57.839647+0000","last_undegraded":"2024-09-18T07:33:57.839647+0000","last_fullsized":"2024-09-18T07:33:57.839647+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:06.688444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:06.688444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"120'10","reported_seq":217,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.836048+0000","last_change":"2024-09-18T07:33:33.001389+0000","last_active":"2024-09-18T07:33:57.836048+0000","last_peered":"2024-09-18T07:33:57.836048+0000","last_clean":"2024-09-18T07:33:57.836048+0000","last_became_active":"2024-09-18T07:33:27.007386+0000","last_became_peered":"2024-09-18T07:33:27.007386+0000","last_unstale":"2024-09-18T07:33:57.836048+0000","last_undegraded":"2024-09-18T07:33:57.836048+0000","last_fullsized":"2024-09-18T07:33:57.836048+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'10","last_scrub_stamp":"2024-09-18T07:33:33.000874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.000874+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:48.823137+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":137,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.822208+0000","last_change":"2024-09-18T07:31:02.347721+0000","last_active":"2024-09-18T07:33:57.822208+0000","last_peered":"2024-09-18T07:33:57.822208+0000","last_clean":"2024-09-18T07:33:57.822208+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:33:57.822208+0000","last_undegraded":"2024-09-18T07:33:57.822208+0000","last_fullsized":"2024-09-18T07:33:57.822208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"129'7637","reported_seq":11845,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:01.513005+0000","last_change":"2024-09-18T07:34:00.127301+0000","last_active":"2024-09-18T07:34:01.513005+0000","last_peered":"2024-09-18T07:34:01.513005+0000","last_clean":"2024-09-18T07:34:01.513005+0000","last_became_active":"2024-09-18T07:33:04.282407+0000","last_became_peered":"2024-09-18T07:33:04.282407+0000","last_unstale":"2024-09-18T07:34:01.513005+0000","last_undegraded":"2024-09-18T07:34:01.513005+0000","last_fullsized":"2024-09-18T07:34:01.513005+0000","mapping_epoch":110,"log_start":"92'1900","ondisk_log_start":"92'1900","created":81,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7533","last_scrub_stamp":"2024-09-18T07:34:00.127026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:00.127026+0000","objects_scrubbed":5726,"log_size":5737,"log_dups_size":0,"ondisk_log_size":5737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:29.810404+0000","scrub_duration":2209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364707840,"num_objects":5565,"num_object_clones":0,"num_object_copies":16695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7637,"num_write_kb":422464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2011,"num_bytes_recovered":131792896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":160,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.828817+0000","last_change":"2024-09-18T07:30:58.314424+0000","last_active":"2024-09-18T07:33:57.828817+0000","last_peered":"2024-09-18T07:33:57.828817+0000","last_clean":"2024-09-18T07:33:57.828817+0000","last_became_active":"2024-09-18T07:30:58.314127+0000","last_became_peered":"2024-09-18T07:30:58.314127+0000","last_unstale":"2024-09-18T07:33:57.828817+0000","last_undegraded":"2024-09-18T07:33:57.828817+0000","last_fullsized":"2024-09-18T07:33:57.828817+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:37.131991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":156,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834214+0000","last_change":"2024-09-18T07:33:28.459796+0000","last_active":"2024-09-18T07:33:57.834214+0000","last_peered":"2024-09-18T07:33:57.834214+0000","last_clean":"2024-09-18T07:33:57.834214+0000","last_became_active":"2024-09-18T07:30:56.288868+0000","last_became_peered":"2024-09-18T07:30:56.288868+0000","last_unstale":"2024-09-18T07:33:57.834214+0000","last_undegraded":"2024-09-18T07:33:57.834214+0000","last_fullsized":"2024-09-18T07:33:57.834214+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:28.459747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:28.459747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:54.082899+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":141,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.841428+0000","last_change":"2024-09-18T07:31:00.336020+0000","last_active":"2024-09-18T07:33:57.841428+0000","last_peered":"2024-09-18T07:33:57.841428+0000","last_clean":"2024-09-18T07:33:57.841428+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:33:57.841428+0000","last_undegraded":"2024-09-18T07:33:57.841428+0000","last_fullsized":"2024-09-18T07:33:57.841428+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:12.700165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.830271+0000","last_change":"2024-09-18T07:32:30.854343+0000","last_active":"2024-09-18T07:33:57.830271+0000","last_peered":"2024-09-18T07:33:57.830271+0000","last_clean":"2024-09-18T07:33:57.830271+0000","last_became_active":"2024-09-18T07:32:30.854029+0000","last_became_peered":"2024-09-18T07:32:30.854029+0000","last_unstale":"2024-09-18T07:33:57.830271+0000","last_undegraded":"2024-09-18T07:33:57.830271+0000","last_fullsized":"2024-09-18T07:33:57.830271+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"129'7636","reported_seq":7742,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:01.642804+0000","last_change":"2024-09-18T07:32:08.057221+0000","last_active":"2024-09-18T07:34:01.642804+0000","last_peered":"2024-09-18T07:34:01.642804+0000","last_clean":"2024-09-18T07:34:01.642804+0000","last_became_active":"2024-09-18T07:32:08.056907+0000","last_became_peered":"2024-09-18T07:32:08.056907+0000","last_unstale":"2024-09-18T07:34:01.642804+0000","last_undegraded":"2024-09-18T07:34:01.642804+0000","last_fullsized":"2024-09-18T07:34:01.642804+0000","mapping_epoch":81,"log_start":"100'3002","ondisk_log_start":"100'3002","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362807296,"num_objects":5536,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7636,"num_write_kb":421504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":218,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.826224+0000","last_change":"2024-09-18T07:33:32.803961+0000","last_active":"2024-09-18T07:33:57.826224+0000","last_peered":"2024-09-18T07:33:57.826224+0000","last_clean":"2024-09-18T07:33:57.826224+0000","last_became_active":"2024-09-18T07:32:30.834955+0000","last_became_peered":"2024-09-18T07:32:30.834955+0000","last_unstale":"2024-09-18T07:33:57.826224+0000","last_undegraded":"2024-09-18T07:33:57.826224+0000","last_fullsized":"2024-09-18T07:33:57.826224+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:33:32.803884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.803884+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:34.097591+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":84,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834345+0000","last_change":"2024-09-18T07:32:30.840417+0000","last_active":"2024-09-18T07:33:57.834345+0000","last_peered":"2024-09-18T07:33:57.834345+0000","last_clean":"2024-09-18T07:33:57.834345+0000","last_became_active":"2024-09-18T07:32:30.840046+0000","last_became_peered":"2024-09-18T07:32:30.840046+0000","last_unstale":"2024-09-18T07:33:57.834345+0000","last_undegraded":"2024-09-18T07:33:57.834345+0000","last_fullsized":"2024-09-18T07:33:57.834345+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.481153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.481153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.709514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":156,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.836538+0000","last_change":"2024-09-18T07:33:34.966610+0000","last_active":"2024-09-18T07:33:57.836538+0000","last_peered":"2024-09-18T07:33:57.836538+0000","last_clean":"2024-09-18T07:33:57.836538+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:33:57.836538+0000","last_undegraded":"2024-09-18T07:33:57.836538+0000","last_fullsized":"2024-09-18T07:33:57.836538+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:34.966537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:34.966537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.041153+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":83,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.836056+0000","last_change":"2024-09-18T07:32:30.839764+0000","last_active":"2024-09-18T07:33:57.836056+0000","last_peered":"2024-09-18T07:33:57.836056+0000","last_clean":"2024-09-18T07:33:57.836056+0000","last_became_active":"2024-09-18T07:32:30.839399+0000","last_became_peered":"2024-09-18T07:32:30.839399+0000","last_unstale":"2024-09-18T07:33:57.836056+0000","last_undegraded":"2024-09-18T07:33:57.836056+0000","last_fullsized":"2024-09-18T07:33:57.836056+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:05.699246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.a","version":"129'7508","reported_seq":7614,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:01.557022+0000","last_change":"2024-09-18T07:32:08.065180+0000","last_active":"2024-09-18T07:34:01.557022+0000","last_peered":"2024-09-18T07:34:01.557022+0000","last_clean":"2024-09-18T07:34:01.557022+0000","last_became_active":"2024-09-18T07:32:08.064911+0000","last_became_peered":"2024-09-18T07:32:08.064911+0000","last_unstale":"2024-09-18T07:34:01.557022+0000","last_undegraded":"2024-09-18T07:34:01.557022+0000","last_fullsized":"2024-09-18T07:34:01.557022+0000","mapping_epoch":81,"log_start":"101'3300","ondisk_log_start":"101'3300","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4208,"log_dups_size":0,"ondisk_log_size":4208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364904448,"num_objects":5568,"num_object_clones":0,"num_object_copies":16704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7508,"num_write_kb":418432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":176,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.821773+0000","last_change":"2024-09-18T07:32:02.967912+0000","last_active":"2024-09-18T07:33:57.821773+0000","last_peered":"2024-09-18T07:33:57.821773+0000","last_clean":"2024-09-18T07:33:57.821773+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:33:57.821773+0000","last_undegraded":"2024-09-18T07:33:57.821773+0000","last_fullsized":"2024-09-18T07:33:57.821773+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:32:02.967835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.967835+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":84,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.822508+0000","last_change":"2024-09-18T07:32:30.846539+0000","last_active":"2024-09-18T07:33:57.822508+0000","last_peered":"2024-09-18T07:33:57.822508+0000","last_clean":"2024-09-18T07:33:57.822508+0000","last_became_active":"2024-09-18T07:32:30.846204+0000","last_became_peered":"2024-09-18T07:32:30.846204+0000","last_unstale":"2024-09-18T07:33:57.822508+0000","last_undegraded":"2024-09-18T07:33:57.822508+0000","last_fullsized":"2024-09-18T07:33:57.822508+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:17.888739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":157,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.836291+0000","last_change":"2024-09-18T07:33:56.967701+0000","last_active":"2024-09-18T07:33:57.836291+0000","last_peered":"2024-09-18T07:33:57.836291+0000","last_clean":"2024-09-18T07:33:57.836291+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:33:57.836291+0000","last_undegraded":"2024-09-18T07:33:57.836291+0000","last_fullsized":"2024-09-18T07:33:57.836291+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:56.967627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:56.967627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:18.872883+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.839346+0000","last_change":"2024-09-18T07:32:30.849280+0000","last_active":"2024-09-18T07:33:57.839346+0000","last_peered":"2024-09-18T07:33:57.839346+0000","last_clean":"2024-09-18T07:33:57.839346+0000","last_became_active":"2024-09-18T07:32:30.847635+0000","last_became_peered":"2024-09-18T07:32:30.847635+0000","last_unstale":"2024-09-18T07:33:57.839346+0000","last_undegraded":"2024-09-18T07:33:57.839346+0000","last_fullsized":"2024-09-18T07:33:57.839346+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:48.144692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"129'7375","reported_seq":7482,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:34:01.484517+0000","last_change":"2024-09-18T07:32:08.062243+0000","last_active":"2024-09-18T07:34:01.484517+0000","last_peered":"2024-09-18T07:34:01.484517+0000","last_clean":"2024-09-18T07:34:01.484517+0000","last_became_active":"2024-09-18T07:32:08.061925+0000","last_became_peered":"2024-09-18T07:32:08.061925+0000","last_unstale":"2024-09-18T07:34:01.484517+0000","last_undegraded":"2024-09-18T07:34:01.484517+0000","last_fullsized":"2024-09-18T07:34:01.484517+0000","mapping_epoch":81,"log_start":"100'3200","ondisk_log_start":"100'3200","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.902340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356319232,"num_objects":5437,"num_object_clones":0,"num_object_copies":16311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7375,"num_write_kb":409984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":192,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.833763+0000","last_change":"2024-09-18T07:32:30.858426+0000","last_active":"2024-09-18T07:33:57.833763+0000","last_peered":"2024-09-18T07:33:57.833763+0000","last_clean":"2024-09-18T07:33:57.833763+0000","last_became_active":"2024-09-18T07:32:30.832486+0000","last_became_peered":"2024-09-18T07:32:30.832486+0000","last_unstale":"2024-09-18T07:33:57.833763+0000","last_undegraded":"2024-09-18T07:33:57.833763+0000","last_fullsized":"2024-09-18T07:33:57.833763+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:01.244752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":100,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.822463+0000","last_change":"2024-09-18T07:32:30.865327+0000","last_active":"2024-09-18T07:33:57.822463+0000","last_peered":"2024-09-18T07:33:57.822463+0000","last_clean":"2024-09-18T07:33:57.822463+0000","last_became_active":"2024-09-18T07:32:30.844389+0000","last_became_peered":"2024-09-18T07:32:30.844389+0000","last_unstale":"2024-09-18T07:33:57.822463+0000","last_undegraded":"2024-09-18T07:33:57.822463+0000","last_fullsized":"2024-09-18T07:33:57.822463+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":140,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.826113+0000","last_change":"2024-09-18T07:32:57.792858+0000","last_active":"2024-09-18T07:33:57.826113+0000","last_peered":"2024-09-18T07:33:57.826113+0000","last_clean":"2024-09-18T07:33:57.826113+0000","last_became_active":"2024-09-18T07:32:30.831224+0000","last_became_peered":"2024-09-18T07:32:30.831224+0000","last_unstale":"2024-09-18T07:33:57.826113+0000","last_undegraded":"2024-09-18T07:33:57.826113+0000","last_fullsized":"2024-09-18T07:33:57.826113+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-09-18T07:32:57.792795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:57.792795+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":136,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.833855+0000","last_change":"2024-09-18T07:31:02.489914+0000","last_active":"2024-09-18T07:33:57.833855+0000","last_peered":"2024-09-18T07:33:57.833855+0000","last_clean":"2024-09-18T07:33:57.833855+0000","last_became_active":"2024-09-18T07:31:02.489619+0000","last_became_peered":"2024-09-18T07:31:02.489619+0000","last_unstale":"2024-09-18T07:33:57.833855+0000","last_undegraded":"2024-09-18T07:33:57.833855+0000","last_fullsized":"2024-09-18T07:33:57.833855+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:13.393902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":156,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.826188+0000","last_change":"2024-09-18T07:32:04.369688+0000","last_active":"2024-09-18T07:33:57.826188+0000","last_peered":"2024-09-18T07:33:57.826188+0000","last_clean":"2024-09-18T07:33:57.826188+0000","last_became_active":"2024-09-18T07:30:56.282357+0000","last_became_peered":"2024-09-18T07:30:56.282357+0000","last_unstale":"2024-09-18T07:33:57.826188+0000","last_undegraded":"2024-09-18T07:33:57.826188+0000","last_fullsized":"2024-09-18T07:33:57.826188+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.369620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.369620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:14.844761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":168,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.841708+0000","last_change":"2024-09-18T07:32:30.872792+0000","last_active":"2024-09-18T07:33:57.841708+0000","last_peered":"2024-09-18T07:33:57.841708+0000","last_clean":"2024-09-18T07:33:57.841708+0000","last_became_active":"2024-09-18T07:32:30.856211+0000","last_became_peered":"2024-09-18T07:32:30.856211+0000","last_unstale":"2024-09-18T07:33:57.841708+0000","last_undegraded":"2024-09-18T07:33:57.841708+0000","last_fullsized":"2024-09-18T07:33:57.841708+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:19.033885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":148,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.836852+0000","last_change":"2024-09-18T07:33:32.589362+0000","last_active":"2024-09-18T07:33:57.836852+0000","last_peered":"2024-09-18T07:33:57.836852+0000","last_clean":"2024-09-18T07:33:57.836852+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:33:57.836852+0000","last_undegraded":"2024-09-18T07:33:57.836852+0000","last_fullsized":"2024-09-18T07:33:57.836852+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.589289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.589289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:38.300848+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":99,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:58.688688+0000","last_change":"2024-09-18T07:33:58.688688+0000","last_active":"2024-09-18T07:33:58.688688+0000","last_peered":"2024-09-18T07:33:58.688688+0000","last_clean":"2024-09-18T07:33:58.688688+0000","last_became_active":"2024-09-18T07:32:30.831805+0000","last_became_peered":"2024-09-18T07:32:30.831805+0000","last_unstale":"2024-09-18T07:33:58.688688+0000","last_undegraded":"2024-09-18T07:33:58.688688+0000","last_fullsized":"2024-09-18T07:33:58.688688+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:58.688631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:58.688631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:24.030416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":177,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.835047+0000","last_change":"2024-09-18T07:32:30.868702+0000","last_active":"2024-09-18T07:33:57.835047+0000","last_peered":"2024-09-18T07:33:57.835047+0000","last_clean":"2024-09-18T07:33:57.835047+0000","last_became_active":"2024-09-18T07:32:30.861402+0000","last_became_peered":"2024-09-18T07:32:30.861402+0000","last_unstale":"2024-09-18T07:33:57.835047+0000","last_undegraded":"2024-09-18T07:33:57.835047+0000","last_fullsized":"2024-09-18T07:33:57.835047+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":84,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.821727+0000","last_change":"2024-09-18T07:32:30.842334+0000","last_active":"2024-09-18T07:33:57.821727+0000","last_peered":"2024-09-18T07:33:57.821727+0000","last_clean":"2024-09-18T07:33:57.821727+0000","last_became_active":"2024-09-18T07:32:30.841928+0000","last_became_peered":"2024-09-18T07:32:30.841928+0000","last_unstale":"2024-09-18T07:33:57.821727+0000","last_undegraded":"2024-09-18T07:33:57.821727+0000","last_fullsized":"2024-09-18T07:33:57.821727+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":155,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:58.276319+0000","last_change":"2024-09-18T07:33:58.276319+0000","last_active":"2024-09-18T07:33:58.276319+0000","last_peered":"2024-09-18T07:33:58.276319+0000","last_clean":"2024-09-18T07:33:58.276319+0000","last_became_active":"2024-09-18T07:32:30.858167+0000","last_became_peered":"2024-09-18T07:32:30.858167+0000","last_unstale":"2024-09-18T07:33:58.276319+0000","last_undegraded":"2024-09-18T07:33:58.276319+0000","last_fullsized":"2024-09-18T07:33:58.276319+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:58.276259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:58.276259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:27.549455+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":144,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.836755+0000","last_change":"2024-09-18T07:32:30.835415+0000","last_active":"2024-09-18T07:33:57.836755+0000","last_peered":"2024-09-18T07:33:57.836755+0000","last_clean":"2024-09-18T07:33:57.836755+0000","last_became_active":"2024-09-18T07:32:30.834708+0000","last_became_peered":"2024-09-18T07:32:30.834708+0000","last_unstale":"2024-09-18T07:33:57.836755+0000","last_undegraded":"2024-09-18T07:33:57.836755+0000","last_fullsized":"2024-09-18T07:33:57.836755+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:07.726137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":177,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834357+0000","last_change":"2024-09-18T07:32:30.851755+0000","last_active":"2024-09-18T07:33:57.834357+0000","last_peered":"2024-09-18T07:33:57.834357+0000","last_clean":"2024-09-18T07:33:57.834357+0000","last_became_active":"2024-09-18T07:32:30.837622+0000","last_became_peered":"2024-09-18T07:32:30.837622+0000","last_unstale":"2024-09-18T07:33:57.834357+0000","last_undegraded":"2024-09-18T07:33:57.834357+0000","last_fullsized":"2024-09-18T07:33:57.834357+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:10.412163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":155,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.828583+0000","last_change":"2024-09-18T07:32:30.854427+0000","last_active":"2024-09-18T07:33:57.828583+0000","last_peered":"2024-09-18T07:33:57.828583+0000","last_clean":"2024-09-18T07:33:57.828583+0000","last_became_active":"2024-09-18T07:32:30.853969+0000","last_became_peered":"2024-09-18T07:32:30.853969+0000","last_unstale":"2024-09-18T07:33:57.828583+0000","last_undegraded":"2024-09-18T07:33:57.828583+0000","last_fullsized":"2024-09-18T07:33:57.828583+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:15.041483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"94'9","reported_seq":197,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.839361+0000","last_change":"2024-09-18T07:32:30.867244+0000","last_active":"2024-09-18T07:33:57.839361+0000","last_peered":"2024-09-18T07:33:57.839361+0000","last_clean":"2024-09-18T07:33:57.839361+0000","last_became_active":"2024-09-18T07:32:30.848580+0000","last_became_peered":"2024-09-18T07:32:30.848580+0000","last_unstale":"2024-09-18T07:33:57.839361+0000","last_undegraded":"2024-09-18T07:33:57.839361+0000","last_fullsized":"2024-09-18T07:33:57.839361+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.090692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":137,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.822320+0000","last_change":"2024-09-18T07:31:02.343206+0000","last_active":"2024-09-18T07:33:57.822320+0000","last_peered":"2024-09-18T07:33:57.822320+0000","last_clean":"2024-09-18T07:33:57.822320+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:33:57.822320+0000","last_undegraded":"2024-09-18T07:33:57.822320+0000","last_fullsized":"2024-09-18T07:33:57.822320+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":186,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.840583+0000","last_change":"2024-09-18T07:33:54.295407+0000","last_active":"2024-09-18T07:33:57.840583+0000","last_peered":"2024-09-18T07:33:57.840583+0000","last_clean":"2024-09-18T07:33:57.840583+0000","last_became_active":"2024-09-18T07:32:30.848136+0000","last_became_peered":"2024-09-18T07:32:30.848136+0000","last_unstale":"2024-09-18T07:33:57.840583+0000","last_undegraded":"2024-09-18T07:33:57.840583+0000","last_fullsized":"2024-09-18T07:33:57.840583+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:33:54.295330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:54.295330+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:20.154237+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":84,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.821968+0000","last_change":"2024-09-18T07:32:30.832648+0000","last_active":"2024-09-18T07:33:57.821968+0000","last_peered":"2024-09-18T07:33:57.821968+0000","last_clean":"2024-09-18T07:33:57.821968+0000","last_became_active":"2024-09-18T07:32:30.832312+0000","last_became_peered":"2024-09-18T07:32:30.832312+0000","last_unstale":"2024-09-18T07:33:57.821968+0000","last_undegraded":"2024-09-18T07:33:57.821968+0000","last_fullsized":"2024-09-18T07:33:57.821968+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.844687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":189,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834298+0000","last_change":"2024-09-18T07:31:00.332729+0000","last_active":"2024-09-18T07:33:57.834298+0000","last_peered":"2024-09-18T07:33:57.834298+0000","last_clean":"2024-09-18T07:33:57.834298+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:33:57.834298+0000","last_undegraded":"2024-09-18T07:33:57.834298+0000","last_fullsized":"2024-09-18T07:33:57.834298+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:11.427383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":151,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834469+0000","last_change":"2024-09-18T07:33:32.135015+0000","last_active":"2024-09-18T07:33:57.834469+0000","last_peered":"2024-09-18T07:33:57.834469+0000","last_clean":"2024-09-18T07:33:57.834469+0000","last_became_active":"2024-09-18T07:32:30.854291+0000","last_became_peered":"2024-09-18T07:32:30.854291+0000","last_unstale":"2024-09-18T07:33:57.834469+0000","last_undegraded":"2024-09-18T07:33:57.834469+0000","last_fullsized":"2024-09-18T07:33:57.834469+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.134847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.134847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:34.913547+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":164,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.821938+0000","last_change":"2024-09-18T07:30:58.314730+0000","last_active":"2024-09-18T07:33:57.821938+0000","last_peered":"2024-09-18T07:33:57.821938+0000","last_clean":"2024-09-18T07:33:57.821938+0000","last_became_active":"2024-09-18T07:30:58.314428+0000","last_became_peered":"2024-09-18T07:30:58.314428+0000","last_unstale":"2024-09-18T07:33:57.821938+0000","last_undegraded":"2024-09-18T07:33:57.821938+0000","last_fullsized":"2024-09-18T07:33:57.821938+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":156,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.822187+0000","last_change":"2024-09-18T07:32:30.831845+0000","last_active":"2024-09-18T07:33:57.822187+0000","last_peered":"2024-09-18T07:33:57.822187+0000","last_clean":"2024-09-18T07:33:57.822187+0000","last_became_active":"2024-09-18T07:32:30.831546+0000","last_became_peered":"2024-09-18T07:32:30.831546+0000","last_unstale":"2024-09-18T07:33:57.822187+0000","last_undegraded":"2024-09-18T07:33:57.822187+0000","last_fullsized":"2024-09-18T07:33:57.822187+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:38.680767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":141,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.837971+0000","last_change":"2024-09-18T07:31:00.337826+0000","last_active":"2024-09-18T07:33:57.837971+0000","last_peered":"2024-09-18T07:33:57.837971+0000","last_clean":"2024-09-18T07:33:57.837971+0000","last_became_active":"2024-09-18T07:31:00.334158+0000","last_became_peered":"2024-09-18T07:31:00.334158+0000","last_unstale":"2024-09-18T07:33:57.837971+0000","last_undegraded":"2024-09-18T07:33:57.837971+0000","last_fullsized":"2024-09-18T07:33:57.837971+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.058588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":136,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.833961+0000","last_change":"2024-09-18T07:31:02.351247+0000","last_active":"2024-09-18T07:33:57.833961+0000","last_peered":"2024-09-18T07:33:57.833961+0000","last_clean":"2024-09-18T07:33:57.833961+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:33:57.833961+0000","last_undegraded":"2024-09-18T07:33:57.833961+0000","last_fullsized":"2024-09-18T07:33:57.833961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.189968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":40,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.828328+0000","last_change":"2024-09-18T07:33:27.033510+0000","last_active":"2024-09-18T07:33:57.828328+0000","last_peered":"2024-09-18T07:33:57.828328+0000","last_clean":"2024-09-18T07:33:57.828328+0000","last_became_active":"2024-09-18T07:33:27.003887+0000","last_became_peered":"2024-09-18T07:33:27.003887+0000","last_unstale":"2024-09-18T07:33:57.828328+0000","last_undegraded":"2024-09-18T07:33:57.828328+0000","last_fullsized":"2024-09-18T07:33:57.828328+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.187186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.187186+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":84,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.840595+0000","last_change":"2024-09-18T07:32:30.847936+0000","last_active":"2024-09-18T07:33:57.840595+0000","last_peered":"2024-09-18T07:33:57.840595+0000","last_clean":"2024-09-18T07:33:57.840595+0000","last_became_active":"2024-09-18T07:32:30.847608+0000","last_became_peered":"2024-09-18T07:32:30.847608+0000","last_unstale":"2024-09-18T07:33:57.840595+0000","last_undegraded":"2024-09-18T07:33:57.840595+0000","last_fullsized":"2024-09-18T07:33:57.840595+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:13.935728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":140,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.839433+0000","last_change":"2024-09-18T07:31:00.333689+0000","last_active":"2024-09-18T07:33:57.839433+0000","last_peered":"2024-09-18T07:33:57.839433+0000","last_clean":"2024-09-18T07:33:57.839433+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:33:57.839433+0000","last_undegraded":"2024-09-18T07:33:57.839433+0000","last_fullsized":"2024-09-18T07:33:57.839433+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.167699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":144,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.827695+0000","last_change":"2024-09-18T07:32:04.503356+0000","last_active":"2024-09-18T07:33:57.827695+0000","last_peered":"2024-09-18T07:33:57.827695+0000","last_clean":"2024-09-18T07:33:57.827695+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:33:57.827695+0000","last_undegraded":"2024-09-18T07:33:57.827695+0000","last_fullsized":"2024-09-18T07:33:57.827695+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.503270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.503270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":182,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834262+0000","last_change":"2024-09-18T07:32:30.869693+0000","last_active":"2024-09-18T07:33:57.834262+0000","last_peered":"2024-09-18T07:33:57.834262+0000","last_clean":"2024-09-18T07:33:57.834262+0000","last_became_active":"2024-09-18T07:32:30.842218+0000","last_became_peered":"2024-09-18T07:32:30.842218+0000","last_unstale":"2024-09-18T07:33:57.834262+0000","last_undegraded":"2024-09-18T07:33:57.834262+0000","last_fullsized":"2024-09-18T07:33:57.834262+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:43.146132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":163,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.836962+0000","last_change":"2024-09-18T07:32:30.853892+0000","last_active":"2024-09-18T07:33:57.836962+0000","last_peered":"2024-09-18T07:33:57.836962+0000","last_clean":"2024-09-18T07:33:57.836962+0000","last_became_active":"2024-09-18T07:32:30.853486+0000","last_became_peered":"2024-09-18T07:32:30.853486+0000","last_unstale":"2024-09-18T07:33:57.836962+0000","last_undegraded":"2024-09-18T07:33:57.836962+0000","last_fullsized":"2024-09-18T07:33:57.836962+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:58.734027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:58.734027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:42.752966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":91,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.821458+0000","last_change":"2024-09-18T07:33:30.668600+0000","last_active":"2024-09-18T07:33:57.821458+0000","last_peered":"2024-09-18T07:33:57.821458+0000","last_clean":"2024-09-18T07:33:57.821458+0000","last_became_active":"2024-09-18T07:32:30.843194+0000","last_became_peered":"2024-09-18T07:32:30.843194+0000","last_unstale":"2024-09-18T07:33:57.821458+0000","last_undegraded":"2024-09-18T07:33:57.821458+0000","last_fullsized":"2024-09-18T07:33:57.821458+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.668428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.668428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:55.997265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":144,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.839568+0000","last_change":"2024-09-18T07:33:30.975087+0000","last_active":"2024-09-18T07:33:57.839568+0000","last_peered":"2024-09-18T07:33:57.839568+0000","last_clean":"2024-09-18T07:33:57.839568+0000","last_became_active":"2024-09-18T07:31:02.490994+0000","last_became_peered":"2024-09-18T07:31:02.490994+0000","last_unstale":"2024-09-18T07:33:57.839568+0000","last_undegraded":"2024-09-18T07:33:57.839568+0000","last_fullsized":"2024-09-18T07:33:57.839568+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.974886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.974886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.586219+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":171,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.835110+0000","last_change":"2024-09-18T07:33:29.988707+0000","last_active":"2024-09-18T07:33:57.835110+0000","last_peered":"2024-09-18T07:33:57.835110+0000","last_clean":"2024-09-18T07:33:57.835110+0000","last_became_active":"2024-09-18T07:32:26.791562+0000","last_became_peered":"2024-09-18T07:32:26.791562+0000","last_unstale":"2024-09-18T07:33:57.835110+0000","last_undegraded":"2024-09-18T07:33:57.835110+0000","last_fullsized":"2024-09-18T07:33:57.835110+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:33:29.988636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.988636+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:34.823616+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":148,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.832316+0000","last_change":"2024-09-18T07:30:56.283643+0000","last_active":"2024-09-18T07:33:57.832316+0000","last_peered":"2024-09-18T07:33:57.832316+0000","last_clean":"2024-09-18T07:33:57.832316+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:33:57.832316+0000","last_undegraded":"2024-09-18T07:33:57.832316+0000","last_fullsized":"2024-09-18T07:33:57.832316+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:41.159206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":155,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.826391+0000","last_change":"2024-09-18T07:32:31.593581+0000","last_active":"2024-09-18T07:33:57.826391+0000","last_peered":"2024-09-18T07:33:57.826391+0000","last_clean":"2024-09-18T07:33:57.826391+0000","last_became_active":"2024-09-18T07:32:30.841210+0000","last_became_peered":"2024-09-18T07:32:30.841210+0000","last_unstale":"2024-09-18T07:33:57.826391+0000","last_undegraded":"2024-09-18T07:33:57.826391+0000","last_fullsized":"2024-09-18T07:33:57.826391+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:31.593520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:31.593520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:50.708269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":91,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.841303+0000","last_change":"2024-09-18T07:33:36.017902+0000","last_active":"2024-09-18T07:33:57.841303+0000","last_peered":"2024-09-18T07:33:57.841303+0000","last_clean":"2024-09-18T07:33:57.841303+0000","last_became_active":"2024-09-18T07:32:30.850481+0000","last_became_peered":"2024-09-18T07:32:30.850481+0000","last_unstale":"2024-09-18T07:33:57.841303+0000","last_undegraded":"2024-09-18T07:33:57.841303+0000","last_fullsized":"2024-09-18T07:33:57.841303+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:36.017602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.017602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.466305+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":145,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.841931+0000","last_change":"2024-09-18T07:33:24.053054+0000","last_active":"2024-09-18T07:33:57.841931+0000","last_peered":"2024-09-18T07:33:57.841931+0000","last_clean":"2024-09-18T07:33:57.841931+0000","last_became_active":"2024-09-18T07:31:02.347537+0000","last_became_peered":"2024-09-18T07:31:02.347537+0000","last_unstale":"2024-09-18T07:33:57.841931+0000","last_undegraded":"2024-09-18T07:33:57.841931+0000","last_fullsized":"2024-09-18T07:33:57.841931+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:24.052908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:24.052908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:39.677071+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":162,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.835311+0000","last_change":"2024-09-18T07:33:33.072569+0000","last_active":"2024-09-18T07:33:57.835311+0000","last_peered":"2024-09-18T07:33:57.835311+0000","last_clean":"2024-09-18T07:33:57.835311+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:33:57.835311+0000","last_undegraded":"2024-09-18T07:33:57.835311+0000","last_fullsized":"2024-09-18T07:33:57.835311+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:33.072487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.072487+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:50.254990+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":171,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.826790+0000","last_change":"2024-09-18T07:33:27.742768+0000","last_active":"2024-09-18T07:33:57.826790+0000","last_peered":"2024-09-18T07:33:57.826790+0000","last_clean":"2024-09-18T07:33:57.826790+0000","last_became_active":"2024-09-18T07:30:58.313556+0000","last_became_peered":"2024-09-18T07:30:58.313556+0000","last_unstale":"2024-09-18T07:33:57.826790+0000","last_undegraded":"2024-09-18T07:33:57.826790+0000","last_fullsized":"2024-09-18T07:33:57.826790+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:33:27.742684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.742684+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:51.970382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":155,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-18T07:33:57.834753+0000","last_change":"2024-09-18T07:33:22.161956+0000","last_active":"2024-09-18T07:33:57.834753+0000","last_peered":"2024-09-18T07:33:57.834753+0000","last_clean":"2024-09-18T07:33:57.834753+0000","last_became_active":"2024-09-18T07:32:30.846635+0000","last_became_peered":"2024-09-18T07:32:30.846635+0000","last_unstale":"2024-09-18T07:33:57.834753+0000","last_undegraded":"2024-09-18T07:33:57.834753+0000","last_fullsized":"2024-09-18T07:33:57.834753+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.161894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.161894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:31.115412+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5793644568,"num_objects":88405,"num_object_clones":0,"num_object_copies":265215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88405,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120423,"num_write_kb":6682433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9693,"num_bytes_recovered":635240448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17442746368,"data_stored":17442734152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78836,"ondisk_log_size":78836,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":1584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":97,"num_read_kb":92,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":236223201321,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462959,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1833660,"kb_used_data":1770416,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91911492,"statfs":{"total":95995035648,"available":94117367808,"internally_reserved":0,"allocated":1812905984,"data_stored":1812563835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724596,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1855552,"kb_used_data":1788208,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91889600,"statfs":{"total":95995035648,"available":94094950400,"internally_reserved":0,"allocated":1831124992,"data_stored":1830782841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953531,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2238100,"kb_used_data":2156932,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":91507052,"statfs":{"total":95995035648,"available":93703221248,"internally_reserved":0,"allocated":2208698368,"data_stored":2208336199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3703464,"kb_used_data":3555672,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":90041688,"statfs":{"total":95995035648,"available":92202688512,"internally_reserved":0,"allocated":3641008128,"data_stored":3640627453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151321031},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411397,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2605112,"kb_used_data":2493416,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91140040,"statfs":{"total":95995035648,"available":93327400960,"internally_reserved":0,"allocated":2553257984,"data_stored":2552923717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2593740,"kb_used_data":2494524,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91151412,"statfs":{"total":95995035648,"available":93339045888,"internally_reserved":0,"allocated":2554392576,"data_stored":2553975570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2953760,"kb_used_data":2842064,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":90791392,"statfs":{"total":95995035648,"available":92970385408,"internally_reserved":0,"allocated":2910273536,"data_stored":2909900983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2892107776,"data_stored":2892103704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2532118528,"data_stored":2532114456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2533359616,"data_stored":2533359616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3625517056,"data_stored":3625517056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2191392768,"data_stored":2191392768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1812987904,"data_stored":1812987904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1795231744,"data_stored":1795227672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":60030976,"data_stored":60030976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:34:06.361 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-18T07:34:06.362 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 26 2024-09-18T07:34:06.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:06.382 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:06.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:06.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:06.524 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:06.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:06.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:06.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:06.675 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:06.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:06.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:06.777 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:06.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:06.826 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:06.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:06.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:06.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:06.963 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:06.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:07.064 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:07.089 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:07.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:07.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:07.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:06 smithi137 ceph-mon[29408]: pgmap v141: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 671 B/s rd, 0 B/s wr, 619 op/s 2024-09-18T07:34:07.102 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:06 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:07.102 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:06 smithi137 ceph-mon[29408]: from='client.24934 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:34:07.102 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:06 smithi137 ceph-mon[29408]: osdmap e132: 8 total, 8 up, 7 in 2024-09-18T07:34:07.102 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:06 smithi137 ceph-mon[39175]: pgmap v141: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 671 B/s rd, 0 B/s wr, 619 op/s 2024-09-18T07:34:07.102 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:06 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:07.102 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:06 smithi137 ceph-mon[39175]: from='client.24934 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:34:07.102 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:06 smithi137 ceph-mon[39175]: osdmap e132: 8 total, 8 up, 7 in 2024-09-18T07:34:07.191 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:07.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:34:06 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:34:07.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:06 smithi203 ceph-mon[32237]: pgmap v141: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 671 B/s rd, 0 B/s wr, 619 op/s 2024-09-18T07:34:07.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:06 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:07.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:06 smithi203 ceph-mon[32237]: from='client.24934 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:34:07.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:06 smithi203 ceph-mon[32237]: osdmap e132: 8 total, 8 up, 7 in 2024-09-18T07:34:07.214 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:07.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:07.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:07.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:07.360 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:07.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:07.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:07.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:34:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:34:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:34:07.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:07.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T07:34:07.511 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:07.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:07.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:07.522 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:07.613 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:07.638 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:07.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:07.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:07.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:07.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:07.781 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:07.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:07.882 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:07.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:07.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:08.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:08.082 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:08.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:08.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:08.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:07 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:08.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:07 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:08.184 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:08.208 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:08.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:08.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:07 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:08.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:08.350 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:08.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:08.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:08.452 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:08.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:08.479 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:08.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:08.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:08.605 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:08.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:08.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:08.632 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:08.650 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:08.706 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:08.714 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:34:08.715 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:34:08.730 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:08.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:08.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:08.751 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:34:08.751 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:34:08.789 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:34:08.790 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:34:08.824 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:34:08.825 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:34:08.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:08.857 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:34:08.857 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:34:08.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:08.882 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:08.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:08.890 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:34:08.890 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:34:08.920 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:34:08.920 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:34:08.955 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:34:08.955 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:34:08.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:09.052 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:09.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:09.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:09.154 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:09.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:08 smithi137 ceph-mon[29408]: pgmap v143: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-09-18T07:34:09.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:08 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2048749046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:09.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:08 smithi137 ceph-mon[39175]: pgmap v143: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-09-18T07:34:09.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:08 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2048749046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:09.178 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:09.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:09.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:09.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:08 smithi203 ceph-mon[32237]: pgmap v143: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-09-18T07:34:09.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:08 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2048749046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:09.280 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:09.304 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:09.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:09.406 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:09.430 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:09.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:09.532 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:09.579 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:09.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:09.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:09.681 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:09.702 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:09.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:09.804 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:09.828 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:09.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:09.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:09.930 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:09.960 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:09.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:09.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:10.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:34:10.061 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:10.084 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:10.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:10.186 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:10.210 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:10.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:10.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:10.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:10.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:10.517 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:10.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:10.618 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:10.642 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:10.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:10.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:10.744 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:10.769 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:10.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:10.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:10.870 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:10.893 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:10.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:10.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:10.901 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:34:10.901 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:34:10.994 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:10 smithi137 ceph-mon[29408]: pgmap v144: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 673 B/s rd, 0 B/s wr, 689 op/s 2024-09-18T07:34:10.995 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:10 smithi137 ceph-mon[39175]: pgmap v144: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 673 B/s rd, 0 B/s wr, 689 op/s 2024-09-18T07:34:10.995 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:11.034 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:11.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:11.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:11.135 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:11.158 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:11.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:11.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:11.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:10 smithi203 ceph-mon[32237]: pgmap v144: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 673 B/s rd, 0 B/s wr, 689 op/s 2024-09-18T07:34:11.260 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:11.298 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:11.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:11.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:11.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:11.450 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:11.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:11.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:11.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:11.596 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:11.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:11.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:11.698 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:11.724 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:11.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:11.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:11.825 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:11.850 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:11.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:11.925 INFO:teuthology.orchestra.run.smithi137.stderr:set pool 7 pg_num to 26 2024-09-18T07:34:11.952 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:11.976 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:11.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:11.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:12.078 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:12.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:12.117 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:12.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:12.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:11 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:34:12.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:11 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2726373626' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-18T07:34:12.218 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:12.242 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:12.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:12.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:12.344 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:12.378 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:12.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:12.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:12.381 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:11 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:34:12.382 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:11 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2726373626' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-18T07:34:12.382 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:11 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:34:12.382 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:11 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2726373626' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-18T07:34:12.480 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:12.503 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:12.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:12.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:12.647 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:12.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:12.748 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:12.773 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:12.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:12.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:12.875 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:12.908 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:12.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:12.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:13.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:13.051 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:13.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:13.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:13.146 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:12 smithi137 ceph-mon[39175]: pgmap v145: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 864 op/s 2024-09-18T07:34:13.146 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:12 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:34:13.146 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:12 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:34:13.146 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:12 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2726373626' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-18T07:34:13.146 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:12 smithi137 ceph-mon[39175]: osdmap e133: 8 total, 8 up, 7 in 2024-09-18T07:34:13.147 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:12 smithi137 ceph-mon[29408]: pgmap v145: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 864 op/s 2024-09-18T07:34:13.147 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:12 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:34:13.147 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:12 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:34:13.147 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:12 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2726373626' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-18T07:34:13.147 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:12 smithi137 ceph-mon[29408]: osdmap e133: 8 total, 8 up, 7 in 2024-09-18T07:34:13.152 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:13.176 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:13.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:13.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:13.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T07:34:13.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:12 smithi203 ceph-mon[32237]: pgmap v145: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 864 op/s 2024-09-18T07:34:13.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:12 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:34:13.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:12 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:34:13.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:12 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2726373626' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-18T07:34:13.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:12 smithi203 ceph-mon[32237]: osdmap e133: 8 total, 8 up, 7 in 2024-09-18T07:34:13.278 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:13.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:13.301 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:13.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:13.352 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:34:13.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:13.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:13.452 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:13.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:13.553 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:13.577 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:13.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:13.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:13.598 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:13.679 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:13.702 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:13.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:13.803 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:13.854 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:13.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:13.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:13.956 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:13.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:13.994 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:13.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:14.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:14.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:14.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:14.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:13 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-18T07:34:14.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:13 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-18T07:34:14.231 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:13 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-18T07:34:14.231 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:13 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-18T07:34:14.231 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:13 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-18T07:34:14.231 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:13 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-18T07:34:14.243 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:14.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:14.310 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:14.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:14.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:14.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:14.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:14.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:14.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:14.491 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:14.535 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:14.558 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:14.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:14.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:14.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:14.681 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:14.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:14.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:14.783 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:14.807 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:14.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:14.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:14.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:14.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:14.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:14.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:14.959 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:34:15.034 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:15.058 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:15.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:15.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:15.077 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:34:15.077 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:34:15.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:15.198 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:15.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:15.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:15.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:14 smithi203 ceph-mon[32237]: pgmap v147: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-18T07:34:15.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:14 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-18T07:34:15.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:14 smithi203 ceph-mon[32237]: osdmap e134: 8 total, 8 up, 7 in 2024-09-18T07:34:15.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:14 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2768868029' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:15.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:14 smithi203 ceph-mon[32237]: from='client.34922 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:15.278 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:14 smithi137 ceph-mon[29408]: pgmap v147: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-18T07:34:15.278 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:14 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-18T07:34:15.278 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:14 smithi137 ceph-mon[29408]: osdmap e134: 8 total, 8 up, 7 in 2024-09-18T07:34:15.278 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:14 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2768868029' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:15.279 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:14 smithi137 ceph-mon[29408]: from='client.34922 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:15.279 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:14 smithi137 ceph-mon[39175]: pgmap v147: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-18T07:34:15.279 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:14 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-18T07:34:15.279 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:14 smithi137 ceph-mon[39175]: osdmap e134: 8 total, 8 up, 7 in 2024-09-18T07:34:15.279 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:14 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2768868029' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:15.279 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:14 smithi137 ceph-mon[39175]: from='client.34922 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:15.299 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:15.324 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:15.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:15.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:15.475 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:15.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:15.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:15.577 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:15.603 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:15.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:15.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:15.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:15.741 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:15.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:15.843 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:15.868 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:15.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:15.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:15.889 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:34:15.889 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:34:15.924 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:34:15.925 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:34:15.959 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:34:15.960 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:34:15.970 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:15.992 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:34:15.993 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:34:15.993 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:15.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:15.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:16.015 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:34:16.015 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:34:16.032 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:34:16.032 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:34:16.048 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:34:16.049 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:34:16.065 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:34:16.065 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:34:16.074 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:15 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:16.074 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:15 smithi137 ceph-mon[29408]: from='client.34922 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:34:16.074 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:15 smithi137 ceph-mon[29408]: osdmap e135: 8 total, 8 up, 7 in 2024-09-18T07:34:16.075 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:15 smithi137 ceph-mon[29408]: 7.8 scrub starts 2024-09-18T07:34:16.075 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:15 smithi137 ceph-mon[29408]: 7.2 scrub starts 2024-09-18T07:34:16.075 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:15 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-18T07:34:16.075 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:15 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-18T07:34:16.075 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:15 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/243932264' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:16.075 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:15 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:16.075 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:15 smithi137 ceph-mon[39175]: from='client.34922 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:34:16.075 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:15 smithi137 ceph-mon[39175]: osdmap e135: 8 total, 8 up, 7 in 2024-09-18T07:34:16.075 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:15 smithi137 ceph-mon[39175]: 7.8 scrub starts 2024-09-18T07:34:16.075 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:15 smithi137 ceph-mon[39175]: 7.2 scrub starts 2024-09-18T07:34:16.076 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:15 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-18T07:34:16.076 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:15 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-18T07:34:16.076 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:15 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/243932264' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:16.095 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:16.116 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:16.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:16.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:16.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:15 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:16.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:15 smithi203 ceph-mon[32237]: from='client.34922 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:34:16.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:15 smithi203 ceph-mon[32237]: osdmap e135: 8 total, 8 up, 7 in 2024-09-18T07:34:16.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:15 smithi203 ceph-mon[32237]: 7.8 scrub starts 2024-09-18T07:34:16.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:15 smithi203 ceph-mon[32237]: 7.2 scrub starts 2024-09-18T07:34:16.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:15 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-18T07:34:16.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:15 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-18T07:34:16.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:15 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/243932264' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:16.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:16.241 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:16.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:16.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:16.343 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:16.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:16.364 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:16.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:16.466 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:16.476 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T07:34:16.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:16.488 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:16.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:16.589 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:16.611 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:16.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:16.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:16.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:16.734 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:16.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:16.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:16.836 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:16.858 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:16.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:16.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:16.960 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:16.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:16.986 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:16.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:17.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:17.100 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[39175]: 7.0 scrub starts 2024-09-18T07:34:17.100 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[39175]: pgmap v150: 158 pgs: 10 unknown, 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 887 op/s 2024-09-18T07:34:17.100 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[39175]: 7.6 scrub starts 2024-09-18T07:34:17.100 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[39175]: 7.7 scrub starts 2024-09-18T07:34:17.101 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[39175]: 7.1 scrub starts 2024-09-18T07:34:17.101 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-18T07:34:17.101 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[39175]: osdmap e136: 8 total, 8 up, 7 in 2024-09-18T07:34:17.101 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[39175]: 7.8 scrub starts 2024-09-18T07:34:17.101 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[39175]: 7.3 scrub starts 2024-09-18T07:34:17.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[29408]: 7.0 scrub starts 2024-09-18T07:34:17.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[29408]: pgmap v150: 158 pgs: 10 unknown, 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 887 op/s 2024-09-18T07:34:17.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[29408]: 7.6 scrub starts 2024-09-18T07:34:17.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[29408]: 7.7 scrub starts 2024-09-18T07:34:17.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[29408]: 7.1 scrub starts 2024-09-18T07:34:17.102 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-18T07:34:17.102 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[29408]: osdmap e136: 8 total, 8 up, 7 in 2024-09-18T07:34:17.102 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[29408]: 7.8 scrub starts 2024-09-18T07:34:17.102 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:16 smithi137 ceph-mon[29408]: 7.3 scrub starts 2024-09-18T07:34:17.108 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:17.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:17.190 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:17.210 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:17.231 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:17.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:17.332 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:17.354 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:17.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:17.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:17.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:34:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:34:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:34:17.455 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:17.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:34:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:34:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:16 smithi203 ceph-mon[32237]: 7.0 scrub starts 2024-09-18T07:34:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:16 smithi203 ceph-mon[32237]: pgmap v150: 158 pgs: 10 unknown, 148 active+clean; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 887 op/s 2024-09-18T07:34:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:16 smithi203 ceph-mon[32237]: 7.6 scrub starts 2024-09-18T07:34:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:16 smithi203 ceph-mon[32237]: 7.7 scrub starts 2024-09-18T07:34:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:16 smithi203 ceph-mon[32237]: 7.1 scrub starts 2024-09-18T07:34:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:16 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-18T07:34:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:16 smithi203 ceph-mon[32237]: osdmap e136: 8 total, 8 up, 7 in 2024-09-18T07:34:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:16 smithi203 ceph-mon[32237]: 7.8 scrub starts 2024-09-18T07:34:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:16 smithi203 ceph-mon[32237]: 7.3 scrub starts 2024-09-18T07:34:17.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:17.477 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:17.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:17.579 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:17.601 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:17.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:17.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:17.685 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:34:17.685 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:34:17.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:34:17.685 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:34:17.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.685 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:20747 7.b 7:d187519a:::benchmark_data_smithi137_74180_object20746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T07:34:17.688 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:23.526769+0000", 2024-09-18T07:34:17.688 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 114.154971909, 2024-09-18T07:34:17.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10768604800000001, 2024-09-18T07:34:17.688 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.688 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.688 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.688 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.688 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.689 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.689 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 20747 2024-09-18T07:34:17.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.689 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526769+0000", 2024-09-18T07:34:17.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526769+0000", 2024-09-18T07:34:17.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526771+0000", 2024-09-18T07:34:17.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2510000000000001e-06 2024-09-18T07:34:17.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526805+0000", 2024-09-18T07:34:17.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3915000000000003e-05 2024-09-18T07:34:17.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526821+0000", 2024-09-18T07:34:17.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6308e-05 2024-09-18T07:34:17.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526829+0000", 2024-09-18T07:34:17.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.481e-06 2024-09-18T07:34:17.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526888+0000", 2024-09-18T07:34:17.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9434000000000001e-05 2024-09-18T07:34:17.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.527009+0000", 2024-09-18T07:34:17.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012099699999999999 2024-09-18T07:34:17.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.527125+0000", 2024-09-18T07:34:17.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116107 2024-09-18T07:34:17.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.528610+0000", 2024-09-18T07:34:17.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014843829999999999 2024-09-18T07:34:17.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.529010+0000", 2024-09-18T07:34:17.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040065400000000002 2024-09-18T07:34:17.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.634285+0000", 2024-09-18T07:34:17.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10527474100000001 2024-09-18T07:34:17.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.634407+0000", 2024-09-18T07:34:17.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012228099999999999 2024-09-18T07:34:17.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.634455+0000", 2024-09-18T07:34:17.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7496000000000001e-05 2024-09-18T07:34:17.694 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.694 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.695 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.695 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:20767 7.b 7:d7fc5ec8:::benchmark_data_smithi137_74180_object20766:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T07:34:17.695 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:23.539452+0000", 2024-09-18T07:34:17.695 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 114.1422891, 2024-09-18T07:34:17.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.095136148000000004, 2024-09-18T07:34:17.695 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.695 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.695 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.695 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.695 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.695 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.695 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 20767 2024-09-18T07:34:17.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.696 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539452+0000", 2024-09-18T07:34:17.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539452+0000", 2024-09-18T07:34:17.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539454+0000", 2024-09-18T07:34:17.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.903e-06 2024-09-18T07:34:17.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539480+0000", 2024-09-18T07:34:17.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6180999999999999e-05 2024-09-18T07:34:17.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539493+0000", 2024-09-18T07:34:17.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.323e-05 2024-09-18T07:34:17.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539501+0000", 2024-09-18T07:34:17.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2139999999999996e-06 2024-09-18T07:34:17.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539554+0000", 2024-09-18T07:34:17.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2920000000000002e-05 2024-09-18T07:34:17.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539632+0000", 2024-09-18T07:34:17.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8258999999999997e-05 2024-09-18T07:34:17.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.539713+0000", 2024-09-18T07:34:17.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0840999999999994e-05 2024-09-18T07:34:17.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.540592+0000", 2024-09-18T07:34:17.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00087853499999999995 2024-09-18T07:34:17.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.540750+0000", 2024-09-18T07:34:17.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000158517 2024-09-18T07:34:17.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.634503+0000", 2024-09-18T07:34:17.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.093752775999999996 2024-09-18T07:34:17.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.634552+0000", 2024-09-18T07:34:17.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8874e-05 2024-09-18T07:34:17.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.634588+0000", 2024-09-18T07:34:17.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5898000000000001e-05 2024-09-18T07:34:17.701 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.701 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.701 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.701 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:21882 7.b 7:ddfde259:::benchmark_data_smithi137_74180_object21881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:34:17.701 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:24.308089+0000", 2024-09-18T07:34:17.701 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 113.373652072, 2024-09-18T07:34:17.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.077958042000000005, 2024-09-18T07:34:17.701 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.701 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.702 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.702 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.702 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:17.703 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.703 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.703 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 21882 2024-09-18T07:34:17.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.704 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308089+0000", 2024-09-18T07:34:17.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308089+0000", 2024-09-18T07:34:17.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308091+0000", 2024-09-18T07:34:17.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2469999999999998e-06 2024-09-18T07:34:17.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308115+0000", 2024-09-18T07:34:17.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3669e-05 2024-09-18T07:34:17.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308131+0000", 2024-09-18T07:34:17.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5891000000000001e-05 2024-09-18T07:34:17.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308139+0000", 2024-09-18T07:34:17.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1470000000000001e-06 2024-09-18T07:34:17.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308196+0000", 2024-09-18T07:34:17.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7092000000000002e-05 2024-09-18T07:34:17.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308300+0000", 2024-09-18T07:34:17.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104026 2024-09-18T07:34:17.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.308448+0000", 2024-09-18T07:34:17.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000148271 2024-09-18T07:34:17.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.309303+0000", 2024-09-18T07:34:17.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00085516499999999996 2024-09-18T07:34:17.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.309558+0000", 2024-09-18T07:34:17.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00025495900000000002 2024-09-18T07:34:17.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.385821+0000", 2024-09-18T07:34:17.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.076263132999999997 2024-09-18T07:34:17.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.385898+0000", 2024-09-18T07:34:17.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7080000000000001e-05 2024-09-18T07:34:17.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.386047+0000", 2024-09-18T07:34:17.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000148362 2024-09-18T07:34:17.709 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.709 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.709 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.709 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:21894 7.b 7:da78b007:::benchmark_data_smithi137_74180_object21893:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:34:17.709 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:24.315487+0000", 2024-09-18T07:34:17.709 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 113.366253503, 2024-09-18T07:34:17.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.071918722000000004, 2024-09-18T07:34:17.710 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.710 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.710 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.710 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.710 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.710 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.710 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 21894 2024-09-18T07:34:17.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.710 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315487+0000", 2024-09-18T07:34:17.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315487+0000", 2024-09-18T07:34:17.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315489+0000", 2024-09-18T07:34:17.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1179999999999999e-06 2024-09-18T07:34:17.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315509+0000", 2024-09-18T07:34:17.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9448000000000001e-05 2024-09-18T07:34:17.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315520+0000", 2024-09-18T07:34:17.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0771000000000001e-05 2024-09-18T07:34:17.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315527+0000", 2024-09-18T07:34:17.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4930000000000001e-06 2024-09-18T07:34:17.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315587+0000", 2024-09-18T07:34:17.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0297000000000002e-05 2024-09-18T07:34:17.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315696+0000", 2024-09-18T07:34:17.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010875899999999999 2024-09-18T07:34:17.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.315787+0000", 2024-09-18T07:34:17.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0686000000000006e-05 2024-09-18T07:34:17.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.316570+0000", 2024-09-18T07:34:17.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00078330100000000003 2024-09-18T07:34:17.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.316853+0000", 2024-09-18T07:34:17.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028248100000000001 2024-09-18T07:34:17.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.386216+0000", 2024-09-18T07:34:17.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.069363707999999996 2024-09-18T07:34:17.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.387354+0000", 2024-09-18T07:34:17.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001137742 2024-09-18T07:34:17.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.387406+0000", 2024-09-18T07:34:17.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1918e-05 2024-09-18T07:34:17.715 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.716 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.716 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.716 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:22340 7.b 7:dcab8849:::benchmark_data_smithi137_74180_object22339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:34:17.716 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:24.658442+0000", 2024-09-18T07:34:17.716 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 113.023298755, 2024-09-18T07:34:17.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.20618298299999999, 2024-09-18T07:34:17.716 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.716 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.716 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.716 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.716 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.717 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.717 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 22340 2024-09-18T07:34:17.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.717 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658442+0000", 2024-09-18T07:34:17.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658442+0000", 2024-09-18T07:34:17.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658445+0000", 2024-09-18T07:34:17.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4609999999999998e-06 2024-09-18T07:34:17.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658479+0000", 2024-09-18T07:34:17.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4393000000000001e-05 2024-09-18T07:34:17.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658497+0000", 2024-09-18T07:34:17.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7649999999999999e-05 2024-09-18T07:34:17.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658505+0000", 2024-09-18T07:34:17.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5350000000000004e-06 2024-09-18T07:34:17.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658567+0000", 2024-09-18T07:34:17.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1580000000000003e-05 2024-09-18T07:34:17.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658673+0000", 2024-09-18T07:34:17.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105907 2024-09-18T07:34:17.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658790+0000", 2024-09-18T07:34:17.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117494 2024-09-18T07:34:17.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.660870+0000", 2024-09-18T07:34:17.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020794849999999998 2024-09-18T07:34:17.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.661103+0000", 2024-09-18T07:34:17.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023344199999999999 2024-09-18T07:34:17.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.864502+0000", 2024-09-18T07:34:17.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.20339850100000001 2024-09-18T07:34:17.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.864571+0000", 2024-09-18T07:34:17.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9566000000000004e-05 2024-09-18T07:34:17.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.864625+0000", 2024-09-18T07:34:17.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3968999999999998e-05 2024-09-18T07:34:17.722 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.722 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.722 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.722 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:23002 7.b 7:d755eca4:::benchmark_data_smithi137_74180_object23001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:34:17.723 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:25.332555+0000", 2024-09-18T07:34:17.723 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 112.349185672, 2024-09-18T07:34:17.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.098253845000000006, 2024-09-18T07:34:17.723 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.723 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.723 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.723 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.723 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.723 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.723 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 23002 2024-09-18T07:34:17.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.723 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.332555+0000", 2024-09-18T07:34:17.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.332555+0000", 2024-09-18T07:34:17.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.332564+0000", 2024-09-18T07:34:17.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2699999999999993e-06 2024-09-18T07:34:17.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.332589+0000", 2024-09-18T07:34:17.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4139999999999999e-05 2024-09-18T07:34:17.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.332605+0000", 2024-09-18T07:34:17.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6874000000000001e-05 2024-09-18T07:34:17.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.332613+0000", 2024-09-18T07:34:17.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4390000000000003e-06 2024-09-18T07:34:17.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.332686+0000", 2024-09-18T07:34:17.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2898000000000006e-05 2024-09-18T07:34:17.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.332821+0000", 2024-09-18T07:34:17.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013513699999999999 2024-09-18T07:34:17.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.333031+0000", 2024-09-18T07:34:17.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020990000000000001 2024-09-18T07:34:17.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.334458+0000", 2024-09-18T07:34:17.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001427357 2024-09-18T07:34:17.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.334923+0000", 2024-09-18T07:34:17.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00046475999999999999 2024-09-18T07:34:17.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.728 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.728 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.430660+0000", 2024-09-18T07:34:17.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.095736694999999997 2024-09-18T07:34:17.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.728 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.728 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.430719+0000", 2024-09-18T07:34:17.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9698999999999999e-05 2024-09-18T07:34:17.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.728 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.728 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:25.430809+0000", 2024-09-18T07:34:17.729 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9676e-05 2024-09-18T07:34:17.729 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.729 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.729 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.729 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:51572 7.b 7:d1d30209:::benchmark_data_smithi137_74180_object51571:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:34:17.729 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.436664+0000", 2024-09-18T07:34:17.729 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 87.245076640999997, 2024-09-18T07:34:17.729 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.067515833999999997, 2024-09-18T07:34:17.729 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.729 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.730 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.730 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.730 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.730 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.730 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 51572 2024-09-18T07:34:17.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.730 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.436664+0000", 2024-09-18T07:34:17.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.436664+0000", 2024-09-18T07:34:17.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.436666+0000", 2024-09-18T07:34:17.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4869999999999999e-06 2024-09-18T07:34:17.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.436692+0000", 2024-09-18T07:34:17.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6624000000000001e-05 2024-09-18T07:34:17.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.436703+0000", 2024-09-18T07:34:17.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0740000000000001e-05 2024-09-18T07:34:17.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.436711+0000", 2024-09-18T07:34:17.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5340000000000002e-06 2024-09-18T07:34:17.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.438564+0000", 2024-09-18T07:34:17.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018533 2024-09-18T07:34:17.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.438627+0000", 2024-09-18T07:34:17.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2959000000000005e-05 2024-09-18T07:34:17.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.438697+0000", 2024-09-18T07:34:17.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0302999999999994e-05 2024-09-18T07:34:17.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.500697+0000", 2024-09-18T07:34:17.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.062000178000000003 2024-09-18T07:34:17.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.503801+0000", 2024-09-18T07:34:17.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031033689999999999 2024-09-18T07:34:17.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.504023+0000", 2024-09-18T07:34:17.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022267399999999999 2024-09-18T07:34:17.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.504128+0000", 2024-09-18T07:34:17.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104603 2024-09-18T07:34:17.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.504180+0000", 2024-09-18T07:34:17.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2062999999999999e-05 2024-09-18T07:34:17.736 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.736 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.736 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.736 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:51573 7.b 7:d73ce6f4:::benchmark_data_smithi137_74180_object51572:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:34:17.736 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.436722+0000", 2024-09-18T07:34:17.736 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 87.245018512000001, 2024-09-18T07:34:17.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.071089118000000007, 2024-09-18T07:34:17.736 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.736 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.737 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.737 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.737 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.737 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.737 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 51573 2024-09-18T07:34:17.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.737 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.436722+0000", 2024-09-18T07:34:17.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.436722+0000", 2024-09-18T07:34:17.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.436723+0000", 2024-09-18T07:34:17.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5300000000000004e-07 2024-09-18T07:34:17.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.436739+0000", 2024-09-18T07:34:17.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5654e-05 2024-09-18T07:34:17.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.436743+0000", 2024-09-18T07:34:17.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1400000000000002e-06 2024-09-18T07:34:17.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.436746+0000", 2024-09-18T07:34:17.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.304e-06 2024-09-18T07:34:17.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.439531+0000", 2024-09-18T07:34:17.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027853000000000001 2024-09-18T07:34:17.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.439595+0000", 2024-09-18T07:34:17.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3565999999999994e-05 2024-09-18T07:34:17.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.439668+0000", 2024-09-18T07:34:17.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3076000000000003e-05 2024-09-18T07:34:17.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.503846+0000", 2024-09-18T07:34:17.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.064177859000000004 2024-09-18T07:34:17.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.504219+0000", 2024-09-18T07:34:17.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000373605 2024-09-18T07:34:17.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.507550+0000", 2024-09-18T07:34:17.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033301749999999999 2024-09-18T07:34:17.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.507687+0000", 2024-09-18T07:34:17.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137076 2024-09-18T07:34:17.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.507811+0000", 2024-09-18T07:34:17.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012470999999999999 2024-09-18T07:34:17.743 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.743 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.743 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.743 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.14865.0:51572 7.b e100/81 ondisk, result = 0)", 2024-09-18T07:34:17.743 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.439757+0000", 2024-09-18T07:34:17.743 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 87.241983645000005, 2024-09-18T07:34:17.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.064429779000000006, 2024-09-18T07:34:17.743 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.743 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.743 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:34:17.743 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.439757+0000", 2024-09-18T07:34:17.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.439757+0000", 2024-09-18T07:34:17.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.439758+0000", 2024-09-18T07:34:17.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2999999999999999e-07 2024-09-18T07:34:17.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.439761+0000", 2024-09-18T07:34:17.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8490000000000002e-06 2024-09-18T07:34:17.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.499357+0000", 2024-09-18T07:34:17.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.059596461000000003 2024-09-18T07:34:17.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.499366+0000", 2024-09-18T07:34:17.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1500000000000005e-06 2024-09-18T07:34:17.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.503934+0000", 2024-09-18T07:34:17.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045678330000000003 2024-09-18T07:34:17.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.504018+0000", 2024-09-18T07:34:17.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3750000000000003e-05 2024-09-18T07:34:17.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.504187+0000", 2024-09-18T07:34:17.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000168906 2024-09-18T07:34:17.747 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.747 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.747 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.747 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:51957 7.b 7:dff3716a:::benchmark_data_smithi137_74180_object51956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:34:17.747 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.955279+0000", 2024-09-18T07:34:17.747 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 86.726461723, 2024-09-18T07:34:17.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.124984595, 2024-09-18T07:34:17.747 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.747 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.747 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.747 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.748 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.748 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.748 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 51957 2024-09-18T07:34:17.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.748 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955279+0000", 2024-09-18T07:34:17.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955279+0000", 2024-09-18T07:34:17.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955282+0000", 2024-09-18T07:34:17.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.728e-06 2024-09-18T07:34:17.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955308+0000", 2024-09-18T07:34:17.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6208e-05 2024-09-18T07:34:17.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955325+0000", 2024-09-18T07:34:17.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7071999999999998e-05 2024-09-18T07:34:17.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955335+0000", 2024-09-18T07:34:17.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5740000000000002e-06 2024-09-18T07:34:17.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955433+0000", 2024-09-18T07:34:17.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7992999999999995e-05 2024-09-18T07:34:17.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955564+0000", 2024-09-18T07:34:17.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013090399999999999 2024-09-18T07:34:17.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955692+0000", 2024-09-18T07:34:17.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012839399999999999 2024-09-18T07:34:17.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.957760+0000", 2024-09-18T07:34:17.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020681549999999999 2024-09-18T07:34:17.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.962122+0000", 2024-09-18T07:34:17.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043617259999999998 2024-09-18T07:34:17.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:51.080186+0000", 2024-09-18T07:34:17.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.118063826 2024-09-18T07:34:17.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:51.080232+0000", 2024-09-18T07:34:17.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6208999999999998e-05 2024-09-18T07:34:17.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:51.080264+0000", 2024-09-18T07:34:17.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1806e-05 2024-09-18T07:34:17.753 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.753 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.754 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.754 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:51965 7.b 7:dcde8c91:::benchmark_data_smithi137_74180_object51964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:34:17.754 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.959013+0000", 2024-09-18T07:34:17.754 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 86.722727797999994, 2024-09-18T07:34:17.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.121449817, 2024-09-18T07:34:17.754 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.754 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.754 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.754 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.754 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.754 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.754 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 51965 2024-09-18T07:34:17.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.755 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959013+0000", 2024-09-18T07:34:17.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959013+0000", 2024-09-18T07:34:17.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959016+0000", 2024-09-18T07:34:17.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8830000000000002e-06 2024-09-18T07:34:17.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959049+0000", 2024-09-18T07:34:17.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3175e-05 2024-09-18T07:34:17.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959067+0000", 2024-09-18T07:34:17.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7623000000000001e-05 2024-09-18T07:34:17.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959076+0000", 2024-09-18T07:34:17.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0149999999999992e-06 2024-09-18T07:34:17.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959138+0000", 2024-09-18T07:34:17.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2133999999999998e-05 2024-09-18T07:34:17.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959268+0000", 2024-09-18T07:34:17.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000130476 2024-09-18T07:34:17.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959418+0000", 2024-09-18T07:34:17.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014930499999999999 2024-09-18T07:34:17.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.962111+0000", 2024-09-18T07:34:17.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00269381 2024-09-18T07:34:17.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.962951+0000", 2024-09-18T07:34:17.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00083964200000000004 2024-09-18T07:34:17.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:51.080380+0000", 2024-09-18T07:34:17.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.117428406 2024-09-18T07:34:17.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:51.080428+0000", 2024-09-18T07:34:17.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.808e-05 2024-09-18T07:34:17.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:51.080463+0000", 2024-09-18T07:34:17.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5268000000000002e-05 2024-09-18T07:34:17.760 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.760 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.760 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.760 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:76515 7.b 7:d3d65bc7:::benchmark_data_smithi137_74180_object76514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:34:17.760 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:18.050800+0000", 2024-09-18T07:34:17.760 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.630940590999998, 2024-09-18T07:34:17.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.116946603, 2024-09-18T07:34:17.760 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.760 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.760 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.760 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.761 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.761 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.761 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 76515 2024-09-18T07:34:17.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.761 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.050800+0000", 2024-09-18T07:34:17.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.050800+0000", 2024-09-18T07:34:17.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.050803+0000", 2024-09-18T07:34:17.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4820000000000001e-06 2024-09-18T07:34:17.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.050827+0000", 2024-09-18T07:34:17.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.461e-05 2024-09-18T07:34:17.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.050843+0000", 2024-09-18T07:34:17.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5764999999999999e-05 2024-09-18T07:34:17.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.050851+0000", 2024-09-18T07:34:17.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9950000000000005e-06 2024-09-18T07:34:17.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.051019+0000", 2024-09-18T07:34:17.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000167472 2024-09-18T07:34:17.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.051212+0000", 2024-09-18T07:34:17.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000193298 2024-09-18T07:34:17.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.051358+0000", 2024-09-18T07:34:17.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014654499999999999 2024-09-18T07:34:17.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.053758+0000", 2024-09-18T07:34:17.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023991559999999999 2024-09-18T07:34:17.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.057140+0000", 2024-09-18T07:34:17.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033824520000000002 2024-09-18T07:34:17.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.167444+0000", 2024-09-18T07:34:17.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.11030403599999999 2024-09-18T07:34:17.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.167599+0000", 2024-09-18T07:34:17.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015494700000000001 2024-09-18T07:34:17.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.766 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.766 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.167747+0000", 2024-09-18T07:34:17.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014784499999999999 2024-09-18T07:34:17.766 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.766 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.766 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.766 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:76523 7.b 7:d3ed4aa3:::benchmark_data_smithi137_74180_object76522:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:34:17.766 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:18.053729+0000", 2024-09-18T07:34:17.766 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.628011682999997, 2024-09-18T07:34:17.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.114700012, 2024-09-18T07:34:17.767 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.767 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.767 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.767 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.767 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.767 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.767 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 76523 2024-09-18T07:34:17.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.767 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.053729+0000", 2024-09-18T07:34:17.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.053729+0000", 2024-09-18T07:34:17.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.053731+0000", 2024-09-18T07:34:17.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0329999999999998e-06 2024-09-18T07:34:17.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.053755+0000", 2024-09-18T07:34:17.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3838000000000001e-05 2024-09-18T07:34:17.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.053765+0000", 2024-09-18T07:34:17.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8090000000000004e-06 2024-09-18T07:34:17.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.053772+0000", 2024-09-18T07:34:17.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1409999999999999e-06 2024-09-18T07:34:17.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.053917+0000", 2024-09-18T07:34:17.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000145329 2024-09-18T07:34:17.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.054199+0000", 2024-09-18T07:34:17.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028138499999999999 2024-09-18T07:34:17.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.054595+0000", 2024-09-18T07:34:17.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000396332 2024-09-18T07:34:17.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.057578+0000", 2024-09-18T07:34:17.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029828210000000001 2024-09-18T07:34:17.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059437+0000", 2024-09-18T07:34:17.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018591790000000001 2024-09-18T07:34:17.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.167938+0000", 2024-09-18T07:34:17.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.108501287 2024-09-18T07:34:17.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.168288+0000", 2024-09-18T07:34:17.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034918900000000002 2024-09-18T07:34:17.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.168429+0000", 2024-09-18T07:34:17.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000141669 2024-09-18T07:34:17.773 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.773 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.773 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.774 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:76526 7.b 7:d8f2833a:::benchmark_data_smithi137_74180_object76525:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:34:17.774 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:18.059042+0000", 2024-09-18T07:34:17.774 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.622698458999999, 2024-09-18T07:34:17.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.110664889, 2024-09-18T07:34:17.774 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.774 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.774 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.774 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.774 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.774 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.774 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 76526 2024-09-18T07:34:17.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059042+0000", 2024-09-18T07:34:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059042+0000", 2024-09-18T07:34:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059044+0000", 2024-09-18T07:34:17.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7439999999999999e-06 2024-09-18T07:34:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059070+0000", 2024-09-18T07:34:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6207000000000002e-05 2024-09-18T07:34:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059080+0000", 2024-09-18T07:34:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9019999999999994e-06 2024-09-18T07:34:17.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059086+0000", 2024-09-18T07:34:17.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9580000000000004e-06 2024-09-18T07:34:17.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059565+0000", 2024-09-18T07:34:17.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00047865400000000002 2024-09-18T07:34:17.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059760+0000", 2024-09-18T07:34:17.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000194964 2024-09-18T07:34:17.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059889+0000", 2024-09-18T07:34:17.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012960700000000001 2024-09-18T07:34:17.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.063135+0000", 2024-09-18T07:34:17.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032460710000000001 2024-09-18T07:34:17.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.063597+0000", 2024-09-18T07:34:17.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000461831 2024-09-18T07:34:17.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.169438+0000", 2024-09-18T07:34:17.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.105840304 2024-09-18T07:34:17.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.169649+0000", 2024-09-18T07:34:17.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000211459 2024-09-18T07:34:17.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.169707+0000", 2024-09-18T07:34:17.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8187999999999999e-05 2024-09-18T07:34:17.781 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.781 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.781 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.782 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:76896 7.b 7:df325c0c:::benchmark_data_smithi137_74180_object76895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:34:17.782 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:18.439439+0000", 2024-09-18T07:34:17.782 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.242302090000003, 2024-09-18T07:34:17.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10739975, 2024-09-18T07:34:17.782 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.782 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.782 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.782 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.782 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.782 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.783 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 76896 2024-09-18T07:34:17.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.783 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439439+0000", 2024-09-18T07:34:17.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439439+0000", 2024-09-18T07:34:17.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439441+0000", 2024-09-18T07:34:17.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1730000000000002e-06 2024-09-18T07:34:17.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439463+0000", 2024-09-18T07:34:17.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2218999999999999e-05 2024-09-18T07:34:17.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439521+0000", 2024-09-18T07:34:17.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7603999999999999e-05 2024-09-18T07:34:17.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439530+0000", 2024-09-18T07:34:17.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4259999999999992e-06 2024-09-18T07:34:17.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439600+0000", 2024-09-18T07:34:17.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9635999999999995e-05 2024-09-18T07:34:17.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439726+0000", 2024-09-18T07:34:17.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126477 2024-09-18T07:34:17.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439870+0000", 2024-09-18T07:34:17.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014343599999999999 2024-09-18T07:34:17.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.443002+0000", 2024-09-18T07:34:17.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031327719999999998 2024-09-18T07:34:17.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.540144+0000", 2024-09-18T07:34:17.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.097141623999999996 2024-09-18T07:34:17.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.546736+0000", 2024-09-18T07:34:17.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065915189999999997 2024-09-18T07:34:17.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.546793+0000", 2024-09-18T07:34:17.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7005000000000001e-05 2024-09-18T07:34:17.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.546839+0000", 2024-09-18T07:34:17.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5859000000000003e-05 2024-09-18T07:34:17.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.788 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.788 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:77136 7.b 7:dc7d0839:::benchmark_data_smithi137_74180_object77135:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:34:17.788 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:18.773487+0000", 2024-09-18T07:34:17.788 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 58.908253936999998, 2024-09-18T07:34:17.789 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.092517149000000007, 2024-09-18T07:34:17.789 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.789 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.789 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.789 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.789 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.789 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.789 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 77136 2024-09-18T07:34:17.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.789 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.773487+0000", 2024-09-18T07:34:17.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.773487+0000", 2024-09-18T07:34:17.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.773489+0000", 2024-09-18T07:34:17.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3159999999999999e-06 2024-09-18T07:34:17.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.773511+0000", 2024-09-18T07:34:17.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1433999999999998e-05 2024-09-18T07:34:17.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.773526+0000", 2024-09-18T07:34:17.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5055000000000001e-05 2024-09-18T07:34:17.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.792 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.773533+0000", 2024-09-18T07:34:17.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0299999999999996e-06 2024-09-18T07:34:17.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.792 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.792 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.773688+0000", 2024-09-18T07:34:17.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015533300000000001 2024-09-18T07:34:17.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.792 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.792 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.773962+0000", 2024-09-18T07:34:17.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000273477 2024-09-18T07:34:17.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.774185+0000", 2024-09-18T07:34:17.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000223024 2024-09-18T07:34:17.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.777684+0000", 2024-09-18T07:34:17.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034994560000000002 2024-09-18T07:34:17.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.778445+0000", 2024-09-18T07:34:17.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00076058500000000004 2024-09-18T07:34:17.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.794 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.865885+0000", 2024-09-18T07:34:17.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.087440843000000004 2024-09-18T07:34:17.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.794 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.865942+0000", 2024-09-18T07:34:17.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6496000000000003e-05 2024-09-18T07:34:17.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.795 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.866004+0000", 2024-09-18T07:34:17.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2100000000000005e-05 2024-09-18T07:34:17.795 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.795 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.795 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.795 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:77149 7.b 7:d62578e8:::benchmark_data_smithi137_74180_object77148:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:34:17.795 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:18.784890+0000", 2024-09-18T07:34:17.795 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 58.896850903000001, 2024-09-18T07:34:17.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.081226715000000005, 2024-09-18T07:34:17.795 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.796 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.796 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.796 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.796 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.796 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.796 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 77149 2024-09-18T07:34:17.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.796 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.784890+0000", 2024-09-18T07:34:17.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.784890+0000", 2024-09-18T07:34:17.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.784892+0000", 2024-09-18T07:34:17.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.17e-06 2024-09-18T07:34:17.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.784917+0000", 2024-09-18T07:34:17.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5105000000000001e-05 2024-09-18T07:34:17.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.784933+0000", 2024-09-18T07:34:17.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5630000000000001e-05 2024-09-18T07:34:17.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.784941+0000", 2024-09-18T07:34:17.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1750000000000005e-06 2024-09-18T07:34:17.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.785224+0000", 2024-09-18T07:34:17.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028257800000000002 2024-09-18T07:34:17.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.785361+0000", 2024-09-18T07:34:17.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013782000000000001 2024-09-18T07:34:17.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.785557+0000", 2024-09-18T07:34:17.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019604299999999999 2024-09-18T07:34:17.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.788793+0000", 2024-09-18T07:34:17.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032353 2024-09-18T07:34:17.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.863056+0000", 2024-09-18T07:34:17.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.074263145000000003 2024-09-18T07:34:17.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.866045+0000", 2024-09-18T07:34:17.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029888470000000002 2024-09-18T07:34:17.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.866087+0000", 2024-09-18T07:34:17.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1770999999999997e-05 2024-09-18T07:34:17.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.802 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.866117+0000", 2024-09-18T07:34:17.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0131e-05 2024-09-18T07:34:17.802 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.802 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.802 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.802 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:77442 7.b 7:d29d5570:::benchmark_data_smithi137_74180_object77441:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:34:17.802 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:19.121837+0000", 2024-09-18T07:34:17.802 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 58.559903399, 2024-09-18T07:34:17.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.094733583999999996, 2024-09-18T07:34:17.802 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.802 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.803 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.803 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.803 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.803 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.803 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 77442 2024-09-18T07:34:17.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.803 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.121837+0000", 2024-09-18T07:34:17.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.121837+0000", 2024-09-18T07:34:17.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.121841+0000", 2024-09-18T07:34:17.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0680000000000001e-06 2024-09-18T07:34:17.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.121866+0000", 2024-09-18T07:34:17.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5100999999999999e-05 2024-09-18T07:34:17.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.121882+0000", 2024-09-18T07:34:17.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6589999999999999e-05 2024-09-18T07:34:17.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.121891+0000", 2024-09-18T07:34:17.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1479999999999995e-06 2024-09-18T07:34:17.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.122027+0000", 2024-09-18T07:34:17.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013562599999999999 2024-09-18T07:34:17.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.122147+0000", 2024-09-18T07:34:17.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000119579 2024-09-18T07:34:17.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.122505+0000", 2024-09-18T07:34:17.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000358687 2024-09-18T07:34:17.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.126185+0000", 2024-09-18T07:34:17.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036795899999999999 2024-09-18T07:34:17.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.126495+0000", 2024-09-18T07:34:17.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00031038399999999997 2024-09-18T07:34:17.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.216240+0000", 2024-09-18T07:34:17.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.089745170999999999 2024-09-18T07:34:17.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.216341+0000", 2024-09-18T07:34:17.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010101499999999999 2024-09-18T07:34:17.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.216571+0000", 2024-09-18T07:34:17.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022962500000000001 2024-09-18T07:34:17.809 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.809 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.809 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.809 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:77453 7.b 7:d0d02ba0:::benchmark_data_smithi137_74180_object77452:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:34:17.809 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:19.132316+0000", 2024-09-18T07:34:17.809 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 58.549424424000001, 2024-09-18T07:34:17.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.084388854999999999, 2024-09-18T07:34:17.809 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.809 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.809 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.810 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.810 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.810 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.810 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 77453 2024-09-18T07:34:17.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.810 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.132316+0000", 2024-09-18T07:34:17.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.132316+0000", 2024-09-18T07:34:17.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.132319+0000", 2024-09-18T07:34:17.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4820000000000001e-06 2024-09-18T07:34:17.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.132347+0000", 2024-09-18T07:34:17.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8490999999999999e-05 2024-09-18T07:34:17.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.132367+0000", 2024-09-18T07:34:17.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9516000000000001e-05 2024-09-18T07:34:17.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.132377+0000", 2024-09-18T07:34:17.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0015e-05 2024-09-18T07:34:17.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.132448+0000", 2024-09-18T07:34:17.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1161999999999994e-05 2024-09-18T07:34:17.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.132585+0000", 2024-09-18T07:34:17.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013738300000000001 2024-09-18T07:34:17.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.132743+0000", 2024-09-18T07:34:17.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015714599999999999 2024-09-18T07:34:17.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.135144+0000", 2024-09-18T07:34:17.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024009190000000001 2024-09-18T07:34:17.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.135158+0000", 2024-09-18T07:34:17.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4741e-05 2024-09-18T07:34:17.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.216617+0000", 2024-09-18T07:34:17.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.081458448000000003 2024-09-18T07:34:17.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.216668+0000", 2024-09-18T07:34:17.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1737999999999998e-05 2024-09-18T07:34:17.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.216705+0000", 2024-09-18T07:34:17.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6813999999999997e-05 2024-09-18T07:34:17.815 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.815 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.816 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.816 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:77467 7.b 7:d7f80a27:::benchmark_data_smithi137_74180_object77466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:34:17.816 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:19.143145+0000", 2024-09-18T07:34:17.816 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 58.538596267000003, 2024-09-18T07:34:17.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.099118181999999999, 2024-09-18T07:34:17.816 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:17.816 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:17.816 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:17.816 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:17.816 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:17.817 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:17.817 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 77467 2024-09-18T07:34:17.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.817 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:17.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:17.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.143145+0000", 2024-09-18T07:34:17.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:17.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.143145+0000", 2024-09-18T07:34:17.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:17.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:17.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.143147+0000", 2024-09-18T07:34:17.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.255e-06 2024-09-18T07:34:17.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:17.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.143170+0000", 2024-09-18T07:34:17.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3023e-05 2024-09-18T07:34:17.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:17.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.143186+0000", 2024-09-18T07:34:17.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5668e-05 2024-09-18T07:34:17.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:17.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.143193+0000", 2024-09-18T07:34:17.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7729999999999999e-06 2024-09-18T07:34:17.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:17.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.143945+0000", 2024-09-18T07:34:17.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000751562 2024-09-18T07:34:17.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:17.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.144078+0000", 2024-09-18T07:34:17.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000132667 2024-09-18T07:34:17.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:34:17.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.144219+0000", 2024-09-18T07:34:17.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000141268 2024-09-18T07:34:17.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:17.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.146626+0000", 2024-09-18T07:34:17.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002407251 2024-09-18T07:34:17.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.216809+0000", 2024-09-18T07:34:17.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.070182700000000001 2024-09-18T07:34:17.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:17.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.242113+0000", 2024-09-18T07:34:17.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025303897999999998 2024-09-18T07:34:17.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:17.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.242217+0000", 2024-09-18T07:34:17.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104016 2024-09-18T07:34:17.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:17.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:17.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:17.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:19.242263+0000", 2024-09-18T07:34:17.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6100999999999998e-05 2024-09-18T07:34:17.823 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.823 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.823 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.823 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:17.823 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:17.823 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:34:17.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:17.824 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:17.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:17.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:17.946 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:17.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:17.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:18.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:18.069 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:18.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:18.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:18.171 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:18.198 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:18.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:18.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:18.300 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:18.322 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:18.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:18.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:18.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:17 smithi137 ceph-mon[29408]: 7.0 scrub starts 2024-09-18T07:34:18.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:17 smithi137 ceph-mon[29408]: 7.9 scrub starts 2024-09-18T07:34:18.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:17 smithi137 ceph-mon[29408]: 7.7 scrub starts 2024-09-18T07:34:18.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:17 smithi137 ceph-mon[29408]: 7.1 scrub starts 2024-09-18T07:34:18.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:17 smithi137 ceph-mon[29408]: osdmap e137: 8 total, 8 up, 7 in 2024-09-18T07:34:18.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:17 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:18.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:17 smithi137 ceph-mon[29408]: 7.5 scrub starts 2024-09-18T07:34:18.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:17 smithi137 ceph-mon[29408]: 7.2 scrub starts 2024-09-18T07:34:18.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:17 smithi137 ceph-mon[29408]: osdmap e138: 8 total, 8 up, 7 in 2024-09-18T07:34:18.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:17 smithi137 ceph-mon[39175]: 7.0 scrub starts 2024-09-18T07:34:18.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.9 scrub starts 2024-09-18T07:34:18.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.7 scrub starts 2024-09-18T07:34:18.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.1 scrub starts 2024-09-18T07:34:18.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: osdmap e137: 8 total, 8 up, 7 in 2024-09-18T07:34:18.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:18.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.5 scrub starts 2024-09-18T07:34:18.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.2 scrub starts 2024-09-18T07:34:18.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: osdmap e138: 8 total, 8 up, 7 in 2024-09-18T07:34:18.423 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:18.450 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:18.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:18.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:18.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:17 smithi203 ceph-mon[32237]: 7.0 scrub starts 2024-09-18T07:34:18.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:17 smithi203 ceph-mon[32237]: 7.9 scrub starts 2024-09-18T07:34:18.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:17 smithi203 ceph-mon[32237]: 7.7 scrub starts 2024-09-18T07:34:18.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:17 smithi203 ceph-mon[32237]: 7.1 scrub starts 2024-09-18T07:34:18.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:17 smithi203 ceph-mon[32237]: osdmap e137: 8 total, 8 up, 7 in 2024-09-18T07:34:18.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:17 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:18.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:17 smithi203 ceph-mon[32237]: 7.5 scrub starts 2024-09-18T07:34:18.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:17 smithi203 ceph-mon[32237]: 7.2 scrub starts 2024-09-18T07:34:18.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:17 smithi203 ceph-mon[32237]: osdmap e138: 8 total, 8 up, 7 in 2024-09-18T07:34:18.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:18.590 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:18.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:18.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:18.656 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:34:18.656 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:34:18.656 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-09-18T07:34:18.656 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-09-18T07:34:18.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:34:18.693 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:18.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:18.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:18.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:18.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:18.885 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:18.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:18.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:18.987 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:18.998 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: 7.4 scrub starts 2024-09-18T07:34:18.998 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: pgmap v153: 158 pgs: 3 active+clean+scrubbing, 10 unknown, 145 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:34:18.998 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 135'8251 2024-09-18T07:34:18.998 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: 7.14 starting backfill to osd.2 from (0'0,0'0] MAX to 135'8473 2024-09-18T07:34:18.998 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: 7.11 starting backfill to osd.1 from (0'0,0'0] MAX to 135'8531 2024-09-18T07:34:18.998 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: 7.18 starting backfill to osd.0 from (0'0,0'0] MAX to 135'8535 2024-09-18T07:34:18.999 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 135'8468 2024-09-18T07:34:18.999 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: 7.17 starting backfill to osd.1 from (0'0,0'0] MAX to 135'8220 2024-09-18T07:34:18.999 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: 7.15 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8368 2024-09-18T07:34:18.999 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 135'8357 2024-09-18T07:34:18.999 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: 7.13 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8251 2024-09-18T07:34:18.999 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: 7.16 starting backfill to osd.2 from (0'0,0'0] MAX to 135'8490 2024-09-18T07:34:18.999 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.4 scrub starts 2024-09-18T07:34:18.999 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: pgmap v153: 158 pgs: 3 active+clean+scrubbing, 10 unknown, 145 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:34:18.999 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 135'8251 2024-09-18T07:34:18.999 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.14 starting backfill to osd.2 from (0'0,0'0] MAX to 135'8473 2024-09-18T07:34:18.999 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.11 starting backfill to osd.1 from (0'0,0'0] MAX to 135'8531 2024-09-18T07:34:19.000 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.18 starting backfill to osd.0 from (0'0,0'0] MAX to 135'8535 2024-09-18T07:34:19.013 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:19.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:19.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:19.115 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:19.140 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:19.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:19.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:19.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:34:19.242 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:19.266 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:19.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:19.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:19.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:19.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 135'8357 2024-09-18T07:34:19.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: 7.14 starting backfill to osd.3 from (0'0,0'0] MAX to 135'8473 2024-09-18T07:34:19.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: 7.10 starting backfill to osd.2 from (0'0,0'0] MAX to 135'8365 2024-09-18T07:34:19.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: 7.19 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8468 2024-09-18T07:34:19.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: 7.18 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8535 2024-09-18T07:34:19.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[29408]: 7.18 starting backfill to osd.6 from (0'0,0'0] MAX to 135'8535 2024-09-18T07:34:19.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 135'8468 2024-09-18T07:34:19.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.17 starting backfill to osd.1 from (0'0,0'0] MAX to 135'8220 2024-09-18T07:34:19.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.15 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8368 2024-09-18T07:34:19.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 135'8357 2024-09-18T07:34:19.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.13 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8251 2024-09-18T07:34:19.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.16 starting backfill to osd.2 from (0'0,0'0] MAX to 135'8490 2024-09-18T07:34:19.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 135'8357 2024-09-18T07:34:19.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.14 starting backfill to osd.3 from (0'0,0'0] MAX to 135'8473 2024-09-18T07:34:19.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.10 starting backfill to osd.2 from (0'0,0'0] MAX to 135'8365 2024-09-18T07:34:19.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.19 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8468 2024-09-18T07:34:19.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.18 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8535 2024-09-18T07:34:19.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:18 smithi137 ceph-mon[39175]: 7.18 starting backfill to osd.6 from (0'0,0'0] MAX to 135'8535 2024-09-18T07:34:19.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:19.431 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:19.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:19.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: 7.4 scrub starts 2024-09-18T07:34:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: pgmap v153: 158 pgs: 3 active+clean+scrubbing, 10 unknown, 145 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:34:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 135'8251 2024-09-18T07:34:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: 7.14 starting backfill to osd.2 from (0'0,0'0] MAX to 135'8473 2024-09-18T07:34:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: 7.11 starting backfill to osd.1 from (0'0,0'0] MAX to 135'8531 2024-09-18T07:34:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: 7.18 starting backfill to osd.0 from (0'0,0'0] MAX to 135'8535 2024-09-18T07:34:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 135'8468 2024-09-18T07:34:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: 7.17 starting backfill to osd.1 from (0'0,0'0] MAX to 135'8220 2024-09-18T07:34:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: 7.15 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8368 2024-09-18T07:34:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 135'8357 2024-09-18T07:34:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: 7.13 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8251 2024-09-18T07:34:19.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: 7.16 starting backfill to osd.2 from (0'0,0'0] MAX to 135'8490 2024-09-18T07:34:19.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 135'8357 2024-09-18T07:34:19.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: 7.14 starting backfill to osd.3 from (0'0,0'0] MAX to 135'8473 2024-09-18T07:34:19.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: 7.10 starting backfill to osd.2 from (0'0,0'0] MAX to 135'8365 2024-09-18T07:34:19.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: 7.19 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8468 2024-09-18T07:34:19.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: 7.18 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8535 2024-09-18T07:34:19.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:18 smithi203 ceph-mon[32237]: 7.18 starting backfill to osd.6 from (0'0,0'0] MAX to 135'8535 2024-09-18T07:34:19.480 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T07:34:19.533 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:19.557 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:19.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:19.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:19.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:19.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:19.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:19.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:19.809 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:19.836 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:19.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:19.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:19.937 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:19.963 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:19.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:19.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:20.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:20.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:20.149 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:20.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:20.156 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:20.173 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:20.196 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:20.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:20.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:20.300 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:20.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:20.315 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:20 smithi137 ceph-mon[39175]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8473 2024-09-18T07:34:20.315 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:20 smithi137 ceph-mon[39175]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 135'8468 2024-09-18T07:34:20.315 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:20 smithi137 ceph-mon[39175]: 7.16 starting backfill to osd.4 from (0'0,0'0] MAX to 135'8490 2024-09-18T07:34:20.315 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:20 smithi137 ceph-mon[39175]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 135'8220 2024-09-18T07:34:20.315 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:20 smithi137 ceph-mon[39175]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8220 2024-09-18T07:34:20.316 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:20 smithi137 ceph-mon[39175]: 7.6 scrub starts 2024-09-18T07:34:20.316 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:20 smithi137 ceph-mon[29408]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8473 2024-09-18T07:34:20.316 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:20 smithi137 ceph-mon[29408]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 135'8468 2024-09-18T07:34:20.316 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:20 smithi137 ceph-mon[29408]: 7.16 starting backfill to osd.4 from (0'0,0'0] MAX to 135'8490 2024-09-18T07:34:20.316 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:20 smithi137 ceph-mon[29408]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 135'8220 2024-09-18T07:34:20.316 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:20 smithi137 ceph-mon[29408]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8220 2024-09-18T07:34:20.316 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:20 smithi137 ceph-mon[29408]: 7.6 scrub starts 2024-09-18T07:34:20.401 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:20.429 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:20.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:20.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:20.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:20 smithi203 ceph-mon[32237]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8473 2024-09-18T07:34:20.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:20 smithi203 ceph-mon[32237]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 135'8468 2024-09-18T07:34:20.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:20 smithi203 ceph-mon[32237]: 7.16 starting backfill to osd.4 from (0'0,0'0] MAX to 135'8490 2024-09-18T07:34:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:20 smithi203 ceph-mon[32237]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 135'8220 2024-09-18T07:34:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:20 smithi203 ceph-mon[32237]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 135'8220 2024-09-18T07:34:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:20 smithi203 ceph-mon[32237]: 7.6 scrub starts 2024-09-18T07:34:20.531 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:20.556 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:20.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:20.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:20.658 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:20.684 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:20.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:20.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:20.786 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:20.811 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:20.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:20.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:20.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:20.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:20.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:20.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:21.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:21.111 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:21.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:21.213 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:21.239 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:21.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:21.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:21.273 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:21 smithi137 ceph-mon[29408]: pgmap v155: 158 pgs: 3 active+clean+scrubbing, 10 unknown, 145 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:34:21.274 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:21 smithi137 ceph-mon[39175]: pgmap v155: 158 pgs: 3 active+clean+scrubbing, 10 unknown, 145 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:34:21.341 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:34:21.344 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:21.344 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:34:21.362 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:21 smithi203 ceph-mon[32237]: pgmap v155: 158 pgs: 3 active+clean+scrubbing, 10 unknown, 145 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:34:21.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:21.387 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:21.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:21.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:21.528 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:21.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:21.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:21.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:21.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:21.668 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:21.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:21.769 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:21.796 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:21.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:21.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:21.898 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:21.925 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:21.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:21.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:22.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:34:22.024 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:34:22.026 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:22.052 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:22.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:22.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:22.154 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:22.179 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:22.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:22.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:22.243 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:22 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3212401796' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:34:22.243 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:22 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3212401796' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:34:22.281 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:22.307 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:22.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:22.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:22.409 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:22.439 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:22.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:22.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:22.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:22 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3212401796' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:34:22.494 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-09-18T07:34:19.460256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3780352563,"num_objects":57874,"num_object_clones":0,"num_object_copies":173622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57874,"num_whiteouts":0,"num_read":937,"num_read_kb":796,"num_write":105996,"num_write_kb":5221546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7828,"num_bytes_recovered":507383281,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93028,"ondisk_log_size":93028,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":460,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":14953584,"kb_used_data":14195096,"kb_used_omap":10,"kb_used_meta":758197,"kb_avail":641262480,"statfs":{"total":671965249536,"available":656652779520,"internally_reserved":0,"allocated":14535778304,"data_stored":14532990222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11162,"internal_metadata":776393830},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1321697280,"num_objects":-20168,"num_object_clones":0,"num_object_copies":-60504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-20168,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":-25538,"num_write_kb":-1462560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1966,"num_bytes_recovered":-129040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.515675"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":170,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.978490+0000","last_change":"2024-09-18T07:33:31.183001+0000","last_active":"2024-09-18T07:34:15.978490+0000","last_peered":"2024-09-18T07:34:15.978490+0000","last_clean":"2024-09-18T07:34:15.978490+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:34:15.978490+0000","last_undegraded":"2024-09-18T07:34:15.978490+0000","last_fullsized":"2024-09-18T07:34:15.978490+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:31.182913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:31.182913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:44.795464+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":200,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.964331+0000","last_change":"2024-09-18T07:33:29.685825+0000","last_active":"2024-09-18T07:34:15.964331+0000","last_peered":"2024-09-18T07:34:15.964331+0000","last_clean":"2024-09-18T07:34:15.964331+0000","last_became_active":"2024-09-18T07:32:26.785727+0000","last_became_peered":"2024-09-18T07:32:26.785727+0000","last_unstale":"2024-09-18T07:34:15.964331+0000","last_undegraded":"2024-09-18T07:34:15.964331+0000","last_fullsized":"2024-09-18T07:34:15.964331+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:33:29.685767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.685767+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:33.086673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":178,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.972457+0000","last_change":"2024-09-18T07:33:57.894855+0000","last_active":"2024-09-18T07:34:15.972457+0000","last_peered":"2024-09-18T07:34:15.972457+0000","last_clean":"2024-09-18T07:34:15.972457+0000","last_became_active":"2024-09-18T07:32:30.831949+0000","last_became_peered":"2024-09-18T07:32:30.831949+0000","last_unstale":"2024-09-18T07:34:15.972457+0000","last_undegraded":"2024-09-18T07:34:15.972457+0000","last_fullsized":"2024-09-18T07:34:15.972457+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:57.894611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.894611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:07.665469+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":159,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.964170+0000","last_change":"2024-09-18T07:32:03.953881+0000","last_active":"2024-09-18T07:34:15.964170+0000","last_peered":"2024-09-18T07:34:15.964170+0000","last_clean":"2024-09-18T07:34:15.964170+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:34:15.964170+0000","last_undegraded":"2024-09-18T07:34:15.964170+0000","last_fullsized":"2024-09-18T07:34:15.964170+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.953827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.953827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":44,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.976752+0000","last_change":"2024-09-18T07:33:27.004713+0000","last_active":"2024-09-18T07:34:15.976752+0000","last_peered":"2024-09-18T07:34:15.976752+0000","last_clean":"2024-09-18T07:34:15.976752+0000","last_became_active":"2024-09-18T07:33:27.004385+0000","last_became_peered":"2024-09-18T07:33:27.004385+0000","last_unstale":"2024-09-18T07:34:15.976752+0000","last_undegraded":"2024-09-18T07:34:15.976752+0000","last_fullsized":"2024-09-18T07:34:15.976752+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:39.137803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":170,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.968591+0000","last_change":"2024-09-18T07:33:27.006545+0000","last_active":"2024-09-18T07:34:15.968591+0000","last_peered":"2024-09-18T07:34:15.968591+0000","last_clean":"2024-09-18T07:34:15.968591+0000","last_became_active":"2024-09-18T07:30:56.286040+0000","last_became_peered":"2024-09-18T07:30:56.286040+0000","last_unstale":"2024-09-18T07:34:15.968591+0000","last_undegraded":"2024-09-18T07:34:15.968591+0000","last_fullsized":"2024-09-18T07:34:15.968591+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.006488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.006488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.614689+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":191,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.974593+0000","last_change":"2024-09-18T07:33:22.032321+0000","last_active":"2024-09-18T07:34:15.974593+0000","last_peered":"2024-09-18T07:34:15.974593+0000","last_clean":"2024-09-18T07:34:15.974593+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:34:15.974593+0000","last_undegraded":"2024-09-18T07:34:15.974593+0000","last_fullsized":"2024-09-18T07:34:15.974593+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:22.032110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.032110+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.263345+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":163,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.964133+0000","last_change":"2024-09-18T07:33:59.829707+0000","last_active":"2024-09-18T07:34:15.964133+0000","last_peered":"2024-09-18T07:34:15.964133+0000","last_clean":"2024-09-18T07:34:15.964133+0000","last_became_active":"2024-09-18T07:31:00.331224+0000","last_became_peered":"2024-09-18T07:31:00.331224+0000","last_unstale":"2024-09-18T07:34:15.964133+0000","last_undegraded":"2024-09-18T07:34:15.964133+0000","last_fullsized":"2024-09-18T07:34:15.964133+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:59.829644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:59.829644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:12.614654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"unknown","last_fresh":"2024-09-18T07:34:15.966589+0000","last_change":"2024-09-18T07:34:16.445627+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T07:34:15.966589+0000","last_undegraded":"2024-09-18T07:34:15.966589+0000","last_fullsized":"2024-09-18T07:34:15.966589+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":170,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966918+0000","last_change":"2024-09-18T07:33:21.338716+0000","last_active":"2024-09-18T07:34:15.966918+0000","last_peered":"2024-09-18T07:34:15.966918+0000","last_clean":"2024-09-18T07:34:15.966918+0000","last_became_active":"2024-09-18T07:30:56.285882+0000","last_became_peered":"2024-09-18T07:30:56.285882+0000","last_unstale":"2024-09-18T07:34:15.966918+0000","last_undegraded":"2024-09-18T07:34:15.966918+0000","last_fullsized":"2024-09-18T07:34:15.966918+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.338669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.338669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:26.027193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":183,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.964520+0000","last_change":"2024-09-18T07:32:02.002919+0000","last_active":"2024-09-18T07:34:15.964520+0000","last_peered":"2024-09-18T07:34:15.964520+0000","last_clean":"2024-09-18T07:34:15.964520+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:34:15.964520+0000","last_undegraded":"2024-09-18T07:34:15.964520+0000","last_fullsized":"2024-09-18T07:34:15.964520+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.002811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.002811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":161,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966480+0000","last_change":"2024-09-18T07:32:30.842423+0000","last_active":"2024-09-18T07:34:15.966480+0000","last_peered":"2024-09-18T07:34:15.966480+0000","last_clean":"2024-09-18T07:34:15.966480+0000","last_became_active":"2024-09-18T07:32:30.841963+0000","last_became_peered":"2024-09-18T07:32:30.841963+0000","last_unstale":"2024-09-18T07:34:15.966480+0000","last_undegraded":"2024-09-18T07:34:15.966480+0000","last_fullsized":"2024-09-18T07:34:15.966480+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.385503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":157,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.974730+0000","last_change":"2024-09-18T07:32:30.840094+0000","last_active":"2024-09-18T07:34:15.974730+0000","last_peered":"2024-09-18T07:34:15.974730+0000","last_clean":"2024-09-18T07:34:15.974730+0000","last_became_active":"2024-09-18T07:32:30.839724+0000","last_became_peered":"2024-09-18T07:32:30.839724+0000","last_unstale":"2024-09-18T07:34:15.974730+0000","last_undegraded":"2024-09-18T07:34:15.974730+0000","last_fullsized":"2024-09-18T07:34:15.974730+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"unknown","last_fresh":"2024-09-18T07:34:15.973413+0000","last_change":"2024-09-18T07:34:16.695747+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T07:34:15.973413+0000","last_undegraded":"2024-09-18T07:34:15.973413+0000","last_fullsized":"2024-09-18T07:34:15.973413+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7533","last_scrub_stamp":"2024-09-18T07:34:00.127026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:00.127026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":162,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966465+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:34:15.966465+0000","last_peered":"2024-09-18T07:34:15.966465+0000","last_clean":"2024-09-18T07:34:15.966465+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:34:15.966465+0000","last_undegraded":"2024-09-18T07:34:15.966465+0000","last_fullsized":"2024-09-18T07:34:15.966465+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":169,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.964901+0000","last_change":"2024-09-18T07:30:58.317041+0000","last_active":"2024-09-18T07:34:15.964901+0000","last_peered":"2024-09-18T07:34:15.964901+0000","last_clean":"2024-09-18T07:34:15.964901+0000","last_became_active":"2024-09-18T07:30:58.316479+0000","last_became_peered":"2024-09-18T07:30:58.316479+0000","last_unstale":"2024-09-18T07:34:15.964901+0000","last_undegraded":"2024-09-18T07:34:15.964901+0000","last_fullsized":"2024-09-18T07:34:15.964901+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:41.690380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":97,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965670+0000","last_change":"2024-09-18T07:32:30.834493+0000","last_active":"2024-09-18T07:34:15.965670+0000","last_peered":"2024-09-18T07:34:15.965670+0000","last_clean":"2024-09-18T07:34:15.965670+0000","last_became_active":"2024-09-18T07:32:30.834192+0000","last_became_peered":"2024-09-18T07:32:30.834192+0000","last_unstale":"2024-09-18T07:34:15.965670+0000","last_undegraded":"2024-09-18T07:34:15.965670+0000","last_fullsized":"2024-09-18T07:34:15.965670+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.897464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":166,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.964719+0000","last_change":"2024-09-18T07:33:51.759330+0000","last_active":"2024-09-18T07:34:15.964719+0000","last_peered":"2024-09-18T07:34:15.964719+0000","last_clean":"2024-09-18T07:34:15.964719+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:34:15.964719+0000","last_undegraded":"2024-09-18T07:34:15.964719+0000","last_fullsized":"2024-09-18T07:34:15.964719+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:51.759259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:51.759259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:18.399187+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":159,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965195+0000","last_change":"2024-09-18T07:32:06.976159+0000","last_active":"2024-09-18T07:34:15.965195+0000","last_peered":"2024-09-18T07:34:15.965195+0000","last_clean":"2024-09-18T07:34:15.965195+0000","last_became_active":"2024-09-18T07:31:02.490014+0000","last_became_peered":"2024-09-18T07:31:02.490014+0000","last_unstale":"2024-09-18T07:34:15.965195+0000","last_undegraded":"2024-09-18T07:34:15.965195+0000","last_fullsized":"2024-09-18T07:34:15.965195+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:06.976113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:06.976113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":177,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.977382+0000","last_change":"2024-09-18T07:33:53.330384+0000","last_active":"2024-09-18T07:34:15.977382+0000","last_peered":"2024-09-18T07:34:15.977382+0000","last_clean":"2024-09-18T07:34:15.977382+0000","last_became_active":"2024-09-18T07:32:30.839952+0000","last_became_peered":"2024-09-18T07:32:30.839952+0000","last_unstale":"2024-09-18T07:34:15.977382+0000","last_undegraded":"2024-09-18T07:34:15.977382+0000","last_fullsized":"2024-09-18T07:34:15.977382+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:53.330304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:53.330304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:58.692089+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":200,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.982273+0000","last_change":"2024-09-18T07:33:59.921943+0000","last_active":"2024-09-18T07:34:15.982273+0000","last_peered":"2024-09-18T07:34:15.982273+0000","last_clean":"2024-09-18T07:34:15.982273+0000","last_became_active":"2024-09-18T07:30:58.316266+0000","last_became_peered":"2024-09-18T07:30:58.316266+0000","last_unstale":"2024-09-18T07:34:15.982273+0000","last_undegraded":"2024-09-18T07:34:15.982273+0000","last_fullsized":"2024-09-18T07:34:15.982273+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:59.921856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:59.921856+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:28.827554+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":163,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.972306+0000","last_change":"2024-09-18T07:33:30.955441+0000","last_active":"2024-09-18T07:34:15.972306+0000","last_peered":"2024-09-18T07:34:15.972306+0000","last_clean":"2024-09-18T07:34:15.972306+0000","last_became_active":"2024-09-18T07:31:00.337474+0000","last_became_peered":"2024-09-18T07:31:00.337474+0000","last_unstale":"2024-09-18T07:34:15.972306+0000","last_undegraded":"2024-09-18T07:34:15.972306+0000","last_fullsized":"2024-09-18T07:34:15.972306+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.955213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.955213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.470164+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.974387+0000","last_change":"2024-09-18T07:33:26.004195+0000","last_active":"2024-09-18T07:34:15.974387+0000","last_peered":"2024-09-18T07:34:15.974387+0000","last_clean":"2024-09-18T07:34:15.974387+0000","last_became_active":"2024-09-18T07:31:02.492348+0000","last_became_peered":"2024-09-18T07:31:02.492348+0000","last_unstale":"2024-09-18T07:34:15.974387+0000","last_undegraded":"2024-09-18T07:34:15.974387+0000","last_fullsized":"2024-09-18T07:34:15.974387+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:26.003425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:26.003425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:38.529586+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":171,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.964667+0000","last_change":"2024-09-18T07:33:53.756188+0000","last_active":"2024-09-18T07:34:15.964667+0000","last_peered":"2024-09-18T07:34:15.964667+0000","last_clean":"2024-09-18T07:34:15.964667+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:34:15.964667+0000","last_undegraded":"2024-09-18T07:34:15.964667+0000","last_fullsized":"2024-09-18T07:34:15.964667+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:53.756119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:53.756119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:19.988286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":181,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965119+0000","last_change":"2024-09-18T07:30:58.319115+0000","last_active":"2024-09-18T07:34:15.965119+0000","last_peered":"2024-09-18T07:34:15.965119+0000","last_clean":"2024-09-18T07:34:15.965119+0000","last_became_active":"2024-09-18T07:30:58.316380+0000","last_became_peered":"2024-09-18T07:30:58.316380+0000","last_unstale":"2024-09-18T07:34:15.965119+0000","last_undegraded":"2024-09-18T07:34:15.965119+0000","last_fullsized":"2024-09-18T07:34:15.965119+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.013089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":155,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.964712+0000","last_change":"2024-09-18T07:31:00.334313+0000","last_active":"2024-09-18T07:34:15.964712+0000","last_peered":"2024-09-18T07:34:15.964712+0000","last_clean":"2024-09-18T07:34:15.964712+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:34:15.964712+0000","last_undegraded":"2024-09-18T07:34:15.964712+0000","last_fullsized":"2024-09-18T07:34:15.964712+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.867220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":121,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966272+0000","last_change":"2024-09-18T07:33:33.565391+0000","last_active":"2024-09-18T07:34:15.966272+0000","last_peered":"2024-09-18T07:34:15.966272+0000","last_clean":"2024-09-18T07:34:15.966272+0000","last_became_active":"2024-09-18T07:32:30.840386+0000","last_became_peered":"2024-09-18T07:32:30.840386+0000","last_unstale":"2024-09-18T07:34:15.966272+0000","last_undegraded":"2024-09-18T07:34:15.966272+0000","last_fullsized":"2024-09-18T07:34:15.966272+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:33.565334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.565334+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:02.676072+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":180,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.973697+0000","last_change":"2024-09-18T07:33:32.976203+0000","last_active":"2024-09-18T07:34:15.973697+0000","last_peered":"2024-09-18T07:34:15.973697+0000","last_clean":"2024-09-18T07:34:15.973697+0000","last_became_active":"2024-09-18T07:30:56.283705+0000","last_became_peered":"2024-09-18T07:30:56.283705+0000","last_unstale":"2024-09-18T07:34:15.973697+0000","last_undegraded":"2024-09-18T07:34:15.973697+0000","last_fullsized":"2024-09-18T07:34:15.973697+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:32.976129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.976129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:54.706133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":200,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.978770+0000","last_change":"2024-09-18T07:33:36.118423+0000","last_active":"2024-09-18T07:34:15.978770+0000","last_peered":"2024-09-18T07:34:15.978770+0000","last_clean":"2024-09-18T07:34:15.978770+0000","last_became_active":"2024-09-18T07:32:30.860930+0000","last_became_peered":"2024-09-18T07:32:30.860930+0000","last_unstale":"2024-09-18T07:34:15.978770+0000","last_undegraded":"2024-09-18T07:34:15.978770+0000","last_fullsized":"2024-09-18T07:34:15.978770+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:33:36.118340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.118340+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:36.957408+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"94'9","reported_seq":225,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.962427+0000","last_change":"2024-09-18T07:33:59.647674+0000","last_active":"2024-09-18T07:34:15.962427+0000","last_peered":"2024-09-18T07:34:15.962427+0000","last_clean":"2024-09-18T07:34:15.962427+0000","last_became_active":"2024-09-18T07:32:30.839828+0000","last_became_peered":"2024-09-18T07:32:30.839828+0000","last_unstale":"2024-09-18T07:34:15.962427+0000","last_undegraded":"2024-09-18T07:34:15.962427+0000","last_fullsized":"2024-09-18T07:34:15.962427+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-09-18T07:33:59.647617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:59.647617+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:03.386170+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"136'8396","reported_seq":12473,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:16.000940+0000","last_change":"2024-09-18T07:33:32.018562+0000","last_active":"2024-09-18T07:34:16.000940+0000","last_peered":"2024-09-18T07:34:16.000940+0000","last_clean":"2024-09-18T07:34:16.000940+0000","last_became_active":"2024-09-18T07:32:54.163740+0000","last_became_peered":"2024-09-18T07:32:54.163740+0000","last_unstale":"2024-09-18T07:34:16.000940+0000","last_undegraded":"2024-09-18T07:34:16.000940+0000","last_fullsized":"2024-09-18T07:34:16.000940+0000","mapping_epoch":102,"log_start":"100'3128","ondisk_log_start":"100'3128","created":81,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'5666","last_scrub_stamp":"2024-09-18T07:33:32.018370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.018370+0000","objects_scrubbed":5641,"log_size":5268,"log_dups_size":0,"ondisk_log_size":5268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:32.363594+0000","scrub_duration":1289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300023808,"num_objects":4578,"num_object_clones":0,"num_object_copies":13734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8396,"num_write_kb":415168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":127074304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":184,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.972934+0000","last_change":"2024-09-18T07:30:58.312916+0000","last_active":"2024-09-18T07:34:15.972934+0000","last_peered":"2024-09-18T07:34:15.972934+0000","last_clean":"2024-09-18T07:34:15.972934+0000","last_became_active":"2024-09-18T07:30:58.312627+0000","last_became_peered":"2024-09-18T07:30:58.312627+0000","last_unstale":"2024-09-18T07:34:15.972934+0000","last_undegraded":"2024-09-18T07:34:15.972934+0000","last_fullsized":"2024-09-18T07:34:15.972934+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":185,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.967359+0000","last_change":"2024-09-18T07:34:01.294683+0000","last_active":"2024-09-18T07:34:15.967359+0000","last_peered":"2024-09-18T07:34:15.967359+0000","last_clean":"2024-09-18T07:34:15.967359+0000","last_became_active":"2024-09-18T07:32:30.838348+0000","last_became_peered":"2024-09-18T07:32:30.838348+0000","last_unstale":"2024-09-18T07:34:15.967359+0000","last_undegraded":"2024-09-18T07:34:15.967359+0000","last_fullsized":"2024-09-18T07:34:15.967359+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:01.294615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:01.294615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:04.638687+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":212,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965330+0000","last_change":"2024-09-18T07:33:33.706253+0000","last_active":"2024-09-18T07:34:15.965330+0000","last_peered":"2024-09-18T07:34:15.965330+0000","last_clean":"2024-09-18T07:34:15.965330+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:34:15.965330+0000","last_undegraded":"2024-09-18T07:34:15.965330+0000","last_fullsized":"2024-09-18T07:34:15.965330+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:33:33.706201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.706201+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.605129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":150,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966699+0000","last_change":"2024-09-18T07:31:02.351412+0000","last_active":"2024-09-18T07:34:15.966699+0000","last_peered":"2024-09-18T07:34:15.966699+0000","last_clean":"2024-09-18T07:34:15.966699+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:34:15.966699+0000","last_undegraded":"2024-09-18T07:34:15.966699+0000","last_fullsized":"2024-09-18T07:34:15.966699+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.333040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"136'8258","reported_seq":8395,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:16.239523+0000","last_change":"2024-09-18T07:34:16.583660+0000","last_active":"2024-09-18T07:34:16.239523+0000","last_peered":"2024-09-18T07:34:16.239523+0000","last_clean":"2024-09-18T07:34:16.239523+0000","last_became_active":"2024-09-18T07:34:15.007659+0000","last_became_peered":"2024-09-18T07:34:15.007659+0000","last_unstale":"2024-09-18T07:34:16.239523+0000","last_undegraded":"2024-09-18T07:34:16.239523+0000","last_fullsized":"2024-09-18T07:34:16.239523+0000","mapping_epoch":81,"log_start":"113'4000","ondisk_log_start":"113'4000","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4258,"log_dups_size":0,"ondisk_log_size":4258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T07:34:46.239593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150405120,"num_objects":2295,"num_object_clones":0,"num_object_copies":6885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4150,"num_write_kb":206240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":223,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.967088+0000","last_change":"2024-09-18T07:32:30.876643+0000","last_active":"2024-09-18T07:34:15.967088+0000","last_peered":"2024-09-18T07:34:15.967088+0000","last_clean":"2024-09-18T07:34:15.967088+0000","last_became_active":"2024-09-18T07:32:30.852501+0000","last_became_peered":"2024-09-18T07:32:30.852501+0000","last_unstale":"2024-09-18T07:34:15.967088+0000","last_undegraded":"2024-09-18T07:34:15.967088+0000","last_fullsized":"2024-09-18T07:34:15.967088+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:32:00.660377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:00.660377+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":177,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.974841+0000","last_change":"2024-09-18T07:32:30.858438+0000","last_active":"2024-09-18T07:34:15.974841+0000","last_peered":"2024-09-18T07:34:15.974841+0000","last_clean":"2024-09-18T07:34:15.974841+0000","last_became_active":"2024-09-18T07:32:30.858176+0000","last_became_peered":"2024-09-18T07:32:30.858176+0000","last_unstale":"2024-09-18T07:34:15.974841+0000","last_undegraded":"2024-09-18T07:34:15.974841+0000","last_fullsized":"2024-09-18T07:34:15.974841+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.665591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.665591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":162,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965709+0000","last_change":"2024-09-18T07:32:30.851490+0000","last_active":"2024-09-18T07:34:15.965709+0000","last_peered":"2024-09-18T07:34:15.965709+0000","last_clean":"2024-09-18T07:34:15.965709+0000","last_became_active":"2024-09-18T07:32:30.851161+0000","last_became_peered":"2024-09-18T07:32:30.851161+0000","last_unstale":"2024-09-18T07:34:15.965709+0000","last_undegraded":"2024-09-18T07:34:15.965709+0000","last_fullsized":"2024-09-18T07:34:15.965709+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:47.861786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":151,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.964423+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:34:15.964423+0000","last_peered":"2024-09-18T07:34:15.964423+0000","last_clean":"2024-09-18T07:34:15.964423+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:34:15.964423+0000","last_undegraded":"2024-09-18T07:34:15.964423+0000","last_fullsized":"2024-09-18T07:34:15.964423+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:12.807504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"136'8364","reported_seq":8501,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.996303+0000","last_change":"2024-09-18T07:34:15.254095+0000","last_active":"2024-09-18T07:34:15.996303+0000","last_peered":"2024-09-18T07:34:15.996303+0000","last_clean":"2024-09-18T07:34:15.996303+0000","last_became_active":"2024-09-18T07:34:14.996533+0000","last_became_peered":"2024-09-18T07:34:14.996533+0000","last_unstale":"2024-09-18T07:34:15.996303+0000","last_undegraded":"2024-09-18T07:34:15.996303+0000","last_fullsized":"2024-09-18T07:34:15.996303+0000","mapping_epoch":81,"log_start":"113'4100","ondisk_log_start":"113'4100","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4264,"log_dups_size":0,"ondisk_log_size":4264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150863872,"num_objects":2302,"num_object_clones":0,"num_object_copies":6906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4203,"num_write_kb":208160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":180,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965797+0000","last_change":"2024-09-18T07:30:58.309938+0000","last_active":"2024-09-18T07:34:15.965797+0000","last_peered":"2024-09-18T07:34:15.965797+0000","last_clean":"2024-09-18T07:34:15.965797+0000","last_became_active":"2024-09-18T07:30:58.309649+0000","last_became_peered":"2024-09-18T07:30:58.309649+0000","last_unstale":"2024-09-18T07:34:15.965797+0000","last_undegraded":"2024-09-18T07:34:15.965797+0000","last_fullsized":"2024-09-18T07:34:15.965797+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":171,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.978872+0000","last_change":"2024-09-18T07:33:24.213432+0000","last_active":"2024-09-18T07:34:15.978872+0000","last_peered":"2024-09-18T07:34:15.978872+0000","last_clean":"2024-09-18T07:34:15.978872+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:34:15.978872+0000","last_undegraded":"2024-09-18T07:34:15.978872+0000","last_fullsized":"2024-09-18T07:34:15.978872+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:24.213356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:24.213356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:52.949578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":203,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966826+0000","last_change":"2024-09-18T07:31:00.336059+0000","last_active":"2024-09-18T07:34:15.966826+0000","last_peered":"2024-09-18T07:34:15.966826+0000","last_clean":"2024-09-18T07:34:15.966826+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:34:15.966826+0000","last_undegraded":"2024-09-18T07:34:15.966826+0000","last_fullsized":"2024-09-18T07:34:15.966826+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:17.963381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":105,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.961716+0000","last_change":"2024-09-18T07:32:54.740291+0000","last_active":"2024-09-18T07:34:15.961716+0000","last_peered":"2024-09-18T07:34:15.961716+0000","last_clean":"2024-09-18T07:34:15.961716+0000","last_became_active":"2024-09-18T07:32:30.830346+0000","last_became_peered":"2024-09-18T07:32:30.830346+0000","last_unstale":"2024-09-18T07:34:15.961716+0000","last_undegraded":"2024-09-18T07:34:15.961716+0000","last_fullsized":"2024-09-18T07:34:15.961716+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:54.740249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:54.740249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.965276+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"136'8524","reported_seq":8658,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.993480+0000","last_change":"2024-09-18T07:34:15.857522+0000","last_active":"2024-09-18T07:34:15.993480+0000","last_peered":"2024-09-18T07:34:15.993480+0000","last_clean":"2024-09-18T07:34:15.993480+0000","last_became_active":"2024-09-18T07:34:14.963613+0000","last_became_peered":"2024-09-18T07:34:14.963613+0000","last_unstale":"2024-09-18T07:34:15.993480+0000","last_undegraded":"2024-09-18T07:34:15.993480+0000","last_fullsized":"2024-09-18T07:34:15.993480+0000","mapping_epoch":81,"log_start":"113'4100","ondisk_log_start":"113'4100","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4424,"log_dups_size":0,"ondisk_log_size":4424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152141824,"num_objects":2321,"num_object_clones":0,"num_object_copies":6963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4287,"num_write_kb":211488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":174,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.972241+0000","last_change":"2024-09-18T07:30:58.307470+0000","last_active":"2024-09-18T07:34:15.972241+0000","last_peered":"2024-09-18T07:34:15.972241+0000","last_clean":"2024-09-18T07:34:15.972241+0000","last_became_active":"2024-09-18T07:30:58.307035+0000","last_became_peered":"2024-09-18T07:30:58.307035+0000","last_unstale":"2024-09-18T07:34:15.972241+0000","last_undegraded":"2024-09-18T07:34:15.972241+0000","last_fullsized":"2024-09-18T07:34:15.972241+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:15.814064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":170,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.967003+0000","last_change":"2024-09-18T07:33:29.471394+0000","last_active":"2024-09-18T07:34:15.967003+0000","last_peered":"2024-09-18T07:34:15.967003+0000","last_clean":"2024-09-18T07:34:15.967003+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:34:15.967003+0000","last_undegraded":"2024-09-18T07:34:15.967003+0000","last_fullsized":"2024-09-18T07:34:15.967003+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:29.471330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.471330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:58.117832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":211,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.976970+0000","last_change":"2024-09-18T07:33:52.317149+0000","last_active":"2024-09-18T07:34:15.976970+0000","last_peered":"2024-09-18T07:34:15.976970+0000","last_clean":"2024-09-18T07:34:15.976970+0000","last_became_active":"2024-09-18T07:31:00.336905+0000","last_became_peered":"2024-09-18T07:31:00.336905+0000","last_unstale":"2024-09-18T07:34:15.976970+0000","last_undegraded":"2024-09-18T07:34:15.976970+0000","last_fullsized":"2024-09-18T07:34:15.976970+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:33:52.317075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:52.317075+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:12.567039+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":150,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.974881+0000","last_change":"2024-09-18T07:31:02.348538+0000","last_active":"2024-09-18T07:34:15.974881+0000","last_peered":"2024-09-18T07:34:15.974881+0000","last_clean":"2024-09-18T07:34:15.974881+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:34:15.974881+0000","last_undegraded":"2024-09-18T07:34:15.974881+0000","last_fullsized":"2024-09-18T07:34:15.974881+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.077245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"136'8368","reported_seq":8504,"reported_epoch":136,"state":"active+clean+scrubbing","last_fresh":"2024-09-18T07:34:16.400460+0000","last_change":"2024-09-18T07:34:16.400237+0000","last_active":"2024-09-18T07:34:16.400460+0000","last_peered":"2024-09-18T07:34:16.400460+0000","last_clean":"2024-09-18T07:34:16.400460+0000","last_became_active":"2024-09-18T07:34:14.979581+0000","last_became_peered":"2024-09-18T07:34:14.979581+0000","last_unstale":"2024-09-18T07:34:16.400460+0000","last_undegraded":"2024-09-18T07:34:16.400460+0000","last_fullsized":"2024-09-18T07:34:16.400460+0000","mapping_epoch":81,"log_start":"100'3056","ondisk_log_start":"100'3056","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5312,"log_dups_size":0,"ondisk_log_size":5312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149979136,"num_objects":2288,"num_object_clones":0,"num_object_copies":6864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4205,"num_write_kb":207808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":219,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.977137+0000","last_change":"2024-09-18T07:32:30.859920+0000","last_active":"2024-09-18T07:34:15.977137+0000","last_peered":"2024-09-18T07:34:15.977137+0000","last_clean":"2024-09-18T07:34:15.977137+0000","last_became_active":"2024-09-18T07:32:30.842195+0000","last_became_peered":"2024-09-18T07:32:30.842195+0000","last_unstale":"2024-09-18T07:34:15.977137+0000","last_undegraded":"2024-09-18T07:34:15.977137+0000","last_fullsized":"2024-09-18T07:34:15.977137+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:18.104967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":163,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.973461+0000","last_change":"2024-09-18T07:30:56.284440+0000","last_active":"2024-09-18T07:34:15.973461+0000","last_peered":"2024-09-18T07:34:15.973461+0000","last_clean":"2024-09-18T07:34:15.973461+0000","last_became_active":"2024-09-18T07:30:56.284120+0000","last_became_peered":"2024-09-18T07:30:56.284120+0000","last_unstale":"2024-09-18T07:34:15.973461+0000","last_undegraded":"2024-09-18T07:34:15.973461+0000","last_fullsized":"2024-09-18T07:34:15.973461+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:12.896782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":162,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.961860+0000","last_change":"2024-09-18T07:33:21.834910+0000","last_active":"2024-09-18T07:34:15.961860+0000","last_peered":"2024-09-18T07:34:15.961860+0000","last_clean":"2024-09-18T07:34:15.961860+0000","last_became_active":"2024-09-18T07:31:00.334776+0000","last_became_peered":"2024-09-18T07:31:00.334776+0000","last_unstale":"2024-09-18T07:34:15.961860+0000","last_undegraded":"2024-09-18T07:34:15.961860+0000","last_fullsized":"2024-09-18T07:34:15.961860+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:24.513163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":151,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.972373+0000","last_change":"2024-09-18T07:31:02.490534+0000","last_active":"2024-09-18T07:34:15.972373+0000","last_peered":"2024-09-18T07:34:15.972373+0000","last_clean":"2024-09-18T07:34:15.972373+0000","last_became_active":"2024-09-18T07:31:02.490186+0000","last_became_peered":"2024-09-18T07:31:02.490186+0000","last_unstale":"2024-09-18T07:34:15.972373+0000","last_undegraded":"2024-09-18T07:34:15.972373+0000","last_fullsized":"2024-09-18T07:34:15.972373+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:19.074713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"136'8208","reported_seq":8403,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.962331+0000","last_change":"2024-09-18T07:34:15.814638+0000","last_active":"2024-09-18T07:34:15.962331+0000","last_peered":"2024-09-18T07:34:15.962331+0000","last_clean":"2024-09-18T07:34:15.962331+0000","last_became_active":"2024-09-18T07:34:14.964872+0000","last_became_peered":"2024-09-18T07:34:14.964872+0000","last_unstale":"2024-09-18T07:34:15.962331+0000","last_undegraded":"2024-09-18T07:34:15.962331+0000","last_fullsized":"2024-09-18T07:34:15.962331+0000","mapping_epoch":81,"log_start":"98'2600","ondisk_log_start":"98'2600","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'5435","last_scrub_stamp":"2024-09-18T07:33:31.180958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:33:31.180958+0000","objects_scrubbed":0,"log_size":5608,"log_dups_size":0,"ondisk_log_size":5608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146440192,"num_objects":2234,"num_object_clones":0,"num_object_copies":6702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":203584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":248,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.973365+0000","last_change":"2024-09-18T07:30:58.317477+0000","last_active":"2024-09-18T07:34:15.973365+0000","last_peered":"2024-09-18T07:34:15.973365+0000","last_clean":"2024-09-18T07:34:15.973365+0000","last_became_active":"2024-09-18T07:30:58.316812+0000","last_became_peered":"2024-09-18T07:30:58.316812+0000","last_unstale":"2024-09-18T07:34:15.973365+0000","last_undegraded":"2024-09-18T07:34:15.973365+0000","last_fullsized":"2024-09-18T07:34:15.973365+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":163,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965327+0000","last_change":"2024-09-18T07:30:56.285353+0000","last_active":"2024-09-18T07:34:15.965327+0000","last_peered":"2024-09-18T07:34:15.965327+0000","last_clean":"2024-09-18T07:34:15.965327+0000","last_became_active":"2024-09-18T07:30:56.285032+0000","last_became_peered":"2024-09-18T07:30:56.285032+0000","last_unstale":"2024-09-18T07:34:15.965327+0000","last_undegraded":"2024-09-18T07:34:15.965327+0000","last_fullsized":"2024-09-18T07:34:15.965327+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":172,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966186+0000","last_change":"2024-09-18T07:30:58.299308+0000","last_active":"2024-09-18T07:34:15.966186+0000","last_peered":"2024-09-18T07:34:15.966186+0000","last_clean":"2024-09-18T07:34:15.966186+0000","last_became_active":"2024-09-18T07:30:55.283292+0000","last_became_peered":"2024-09-18T07:30:55.283292+0000","last_unstale":"2024-09-18T07:34:15.966186+0000","last_undegraded":"2024-09-18T07:34:15.966186+0000","last_fullsized":"2024-09-18T07:34:15.966186+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:19.297747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":154,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.977210+0000","last_change":"2024-09-18T07:31:00.332738+0000","last_active":"2024-09-18T07:34:15.977210+0000","last_peered":"2024-09-18T07:34:15.977210+0000","last_clean":"2024-09-18T07:34:15.977210+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:34:15.977210+0000","last_undegraded":"2024-09-18T07:34:15.977210+0000","last_fullsized":"2024-09-18T07:34:15.977210+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:07.352960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":157,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.968336+0000","last_change":"2024-09-18T07:32:30.854643+0000","last_active":"2024-09-18T07:34:15.968336+0000","last_peered":"2024-09-18T07:34:15.968336+0000","last_clean":"2024-09-18T07:34:15.968336+0000","last_became_active":"2024-09-18T07:32:30.853087+0000","last_became_peered":"2024-09-18T07:32:30.853087+0000","last_unstale":"2024-09-18T07:34:15.968336+0000","last_undegraded":"2024-09-18T07:34:15.968336+0000","last_fullsized":"2024-09-18T07:34:15.968336+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:45.131703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"135'8466","reported_seq":8600,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.977011+0000","last_change":"2024-09-18T07:34:15.049838+0000","last_active":"2024-09-18T07:34:15.977011+0000","last_peered":"2024-09-18T07:34:15.977011+0000","last_clean":"2024-09-18T07:34:15.977011+0000","last_became_active":"2024-09-18T07:34:14.994644+0000","last_became_peered":"2024-09-18T07:34:14.994644+0000","last_unstale":"2024-09-18T07:34:15.977011+0000","last_undegraded":"2024-09-18T07:34:15.977011+0000","last_fullsized":"2024-09-18T07:34:15.977011+0000","mapping_epoch":81,"log_start":"109'3809","ondisk_log_start":"109'3809","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4657,"log_dups_size":0,"ondisk_log_size":4657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151126016,"num_objects":2306,"num_object_clones":0,"num_object_copies":6918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4257,"num_write_kb":210016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":186,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.973028+0000","last_change":"2024-09-18T07:30:58.310789+0000","last_active":"2024-09-18T07:34:15.973028+0000","last_peered":"2024-09-18T07:34:15.973028+0000","last_clean":"2024-09-18T07:34:15.973028+0000","last_became_active":"2024-09-18T07:30:58.310490+0000","last_became_peered":"2024-09-18T07:30:58.310490+0000","last_unstale":"2024-09-18T07:34:15.973028+0000","last_undegraded":"2024-09-18T07:34:15.973028+0000","last_fullsized":"2024-09-18T07:34:15.973028+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.090903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":171,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.973669+0000","last_change":"2024-09-18T07:33:23.038159+0000","last_active":"2024-09-18T07:34:15.973669+0000","last_peered":"2024-09-18T07:34:15.973669+0000","last_clean":"2024-09-18T07:34:15.973669+0000","last_became_active":"2024-09-18T07:30:56.284803+0000","last_became_peered":"2024-09-18T07:30:56.284803+0000","last_unstale":"2024-09-18T07:34:15.973669+0000","last_undegraded":"2024-09-18T07:34:15.973669+0000","last_fullsized":"2024-09-18T07:34:15.973669+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:23.037963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:23.037963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:45.045160+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":180,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.969864+0000","last_change":"2024-09-18T07:32:02.418590+0000","last_active":"2024-09-18T07:34:15.969864+0000","last_peered":"2024-09-18T07:34:15.969864+0000","last_clean":"2024-09-18T07:34:15.969864+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:34:15.969864+0000","last_undegraded":"2024-09-18T07:34:15.969864+0000","last_fullsized":"2024-09-18T07:34:15.969864+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:32:02.418507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.418507+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:39.933773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":162,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.978081+0000","last_change":"2024-09-18T07:32:30.837706+0000","last_active":"2024-09-18T07:34:15.978081+0000","last_peered":"2024-09-18T07:34:15.978081+0000","last_clean":"2024-09-18T07:34:15.978081+0000","last_became_active":"2024-09-18T07:32:30.837243+0000","last_became_peered":"2024-09-18T07:32:30.837243+0000","last_unstale":"2024-09-18T07:34:15.978081+0000","last_undegraded":"2024-09-18T07:34:15.978081+0000","last_fullsized":"2024-09-18T07:34:15.978081+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:09.936542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":129,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.963137+0000","last_change":"2024-09-18T07:33:57.655849+0000","last_active":"2024-09-18T07:34:15.963137+0000","last_peered":"2024-09-18T07:34:15.963137+0000","last_clean":"2024-09-18T07:34:15.963137+0000","last_became_active":"2024-09-18T07:32:30.839186+0000","last_became_peered":"2024-09-18T07:32:30.839186+0000","last_unstale":"2024-09-18T07:34:15.963137+0000","last_undegraded":"2024-09-18T07:34:15.963137+0000","last_fullsized":"2024-09-18T07:34:15.963137+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:57.655797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.655797+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:04.938177+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"135'8368","reported_seq":12318,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.977353+0000","last_change":"2024-09-18T07:34:15.005962+0000","last_active":"2024-09-18T07:34:15.977353+0000","last_peered":"2024-09-18T07:34:15.977353+0000","last_clean":"2024-09-18T07:34:15.977353+0000","last_became_active":"2024-09-18T07:34:14.998814+0000","last_became_peered":"2024-09-18T07:34:14.998814+0000","last_unstale":"2024-09-18T07:34:15.977353+0000","last_undegraded":"2024-09-18T07:34:15.977353+0000","last_fullsized":"2024-09-18T07:34:15.977353+0000","mapping_epoch":107,"log_start":"98'2600","ondisk_log_start":"98'2600","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5768,"log_dups_size":0,"ondisk_log_size":5768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153452544,"num_objects":2341,"num_object_clones":0,"num_object_copies":7023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4207,"num_write_kb":209568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":952,"num_bytes_recovered":62390272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":197,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.978682+0000","last_change":"2024-09-18T07:33:27.040227+0000","last_active":"2024-09-18T07:34:15.978682+0000","last_peered":"2024-09-18T07:34:15.978682+0000","last_clean":"2024-09-18T07:34:15.978682+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:34:15.978682+0000","last_undegraded":"2024-09-18T07:34:15.978682+0000","last_fullsized":"2024-09-18T07:34:15.978682+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:51.281723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":168,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.975899+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:34:15.975899+0000","last_peered":"2024-09-18T07:34:15.975899+0000","last_clean":"2024-09-18T07:34:15.975899+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:34:15.975899+0000","last_undegraded":"2024-09-18T07:34:15.975899+0000","last_fullsized":"2024-09-18T07:34:15.975899+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:43.132679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":250,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.973578+0000","last_change":"2024-09-18T07:32:30.859165+0000","last_active":"2024-09-18T07:34:15.973578+0000","last_peered":"2024-09-18T07:34:15.973578+0000","last_clean":"2024-09-18T07:34:15.973578+0000","last_became_active":"2024-09-18T07:32:30.851004+0000","last_became_peered":"2024-09-18T07:32:30.851004+0000","last_unstale":"2024-09-18T07:34:15.973578+0000","last_undegraded":"2024-09-18T07:34:15.973578+0000","last_fullsized":"2024-09-18T07:34:15.973578+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:07.886671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":107,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":162,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.967312+0000","last_change":"2024-09-18T07:33:27.394247+0000","last_active":"2024-09-18T07:34:15.967312+0000","last_peered":"2024-09-18T07:34:15.967312+0000","last_clean":"2024-09-18T07:34:15.967312+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:34:15.967312+0000","last_undegraded":"2024-09-18T07:34:15.967312+0000","last_fullsized":"2024-09-18T07:34:15.967312+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.394203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.394203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:31.541179+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":167,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.972840+0000","last_change":"2024-09-18T07:33:25.028842+0000","last_active":"2024-09-18T07:34:15.972840+0000","last_peered":"2024-09-18T07:34:15.972840+0000","last_clean":"2024-09-18T07:34:15.972840+0000","last_became_active":"2024-09-18T07:31:02.347390+0000","last_became_peered":"2024-09-18T07:31:02.347390+0000","last_unstale":"2024-09-18T07:34:15.972840+0000","last_undegraded":"2024-09-18T07:34:15.972840+0000","last_fullsized":"2024-09-18T07:34:15.972840+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:25.028609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:25.028609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:34.346431+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"136'8480","reported_seq":12480,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.978563+0000","last_change":"2024-09-18T07:34:15.477139+0000","last_active":"2024-09-18T07:34:15.978563+0000","last_peered":"2024-09-18T07:34:15.978563+0000","last_clean":"2024-09-18T07:34:15.978563+0000","last_became_active":"2024-09-18T07:34:14.989995+0000","last_became_peered":"2024-09-18T07:34:14.989995+0000","last_unstale":"2024-09-18T07:34:15.978563+0000","last_undegraded":"2024-09-18T07:34:15.978563+0000","last_fullsized":"2024-09-18T07:34:15.978563+0000","mapping_epoch":105,"log_start":"109'3790","ondisk_log_start":"109'3790","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155975680,"num_objects":2380,"num_object_clones":0,"num_object_copies":7140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4263,"num_write_kb":212576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":963,"num_bytes_recovered":63143936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"94'17","reported_seq":262,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.963278+0000","last_change":"2024-09-18T07:33:33.849354+0000","last_active":"2024-09-18T07:34:15.963278+0000","last_peered":"2024-09-18T07:34:15.963278+0000","last_clean":"2024-09-18T07:34:15.963278+0000","last_became_active":"2024-09-18T07:32:30.842449+0000","last_became_peered":"2024-09-18T07:32:30.842449+0000","last_unstale":"2024-09-18T07:34:15.963278+0000","last_undegraded":"2024-09-18T07:34:15.963278+0000","last_fullsized":"2024-09-18T07:34:15.963278+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'17","last_scrub_stamp":"2024-09-18T07:33:33.849285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.849285+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:53.616735+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":170,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.968920+0000","last_change":"2024-09-18T07:31:59.617046+0000","last_active":"2024-09-18T07:34:15.968920+0000","last_peered":"2024-09-18T07:34:15.968920+0000","last_clean":"2024-09-18T07:34:15.968920+0000","last_became_active":"2024-09-18T07:30:56.288149+0000","last_became_peered":"2024-09-18T07:30:56.288149+0000","last_unstale":"2024-09-18T07:34:15.968920+0000","last_undegraded":"2024-09-18T07:34:15.968920+0000","last_fullsized":"2024-09-18T07:34:15.968920+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:59.616988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:59.616988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:06.307919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":192,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.975141+0000","last_change":"2024-09-18T07:33:23.172950+0000","last_active":"2024-09-18T07:34:15.975141+0000","last_peered":"2024-09-18T07:34:15.975141+0000","last_clean":"2024-09-18T07:34:15.975141+0000","last_became_active":"2024-09-18T07:32:30.842828+0000","last_became_peered":"2024-09-18T07:32:30.842828+0000","last_unstale":"2024-09-18T07:34:15.975141+0000","last_undegraded":"2024-09-18T07:34:15.975141+0000","last_fullsized":"2024-09-18T07:34:15.975141+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:33:23.172848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:33:23.172848+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.696337+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":97,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.981971+0000","last_change":"2024-09-18T07:32:30.835729+0000","last_active":"2024-09-18T07:34:15.981971+0000","last_peered":"2024-09-18T07:34:15.981971+0000","last_clean":"2024-09-18T07:34:15.981971+0000","last_became_active":"2024-09-18T07:32:30.835105+0000","last_became_peered":"2024-09-18T07:32:30.835105+0000","last_unstale":"2024-09-18T07:34:15.981971+0000","last_undegraded":"2024-09-18T07:34:15.981971+0000","last_fullsized":"2024-09-18T07:34:15.981971+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:02.499041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.499041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.738806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.967164+0000","last_change":"2024-09-18T07:33:22.358792+0000","last_active":"2024-09-18T07:34:15.967164+0000","last_peered":"2024-09-18T07:34:15.967164+0000","last_clean":"2024-09-18T07:34:15.967164+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:34:15.967164+0000","last_undegraded":"2024-09-18T07:34:15.967164+0000","last_fullsized":"2024-09-18T07:34:15.967164+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.358746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.358746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:26.289679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"136'8501","reported_seq":8625,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.993450+0000","last_change":"2024-09-18T07:32:08.063469+0000","last_active":"2024-09-18T07:34:15.993450+0000","last_peered":"2024-09-18T07:34:15.993450+0000","last_clean":"2024-09-18T07:34:15.993450+0000","last_became_active":"2024-09-18T07:32:08.063070+0000","last_became_peered":"2024-09-18T07:32:08.063070+0000","last_unstale":"2024-09-18T07:34:15.993450+0000","last_undegraded":"2024-09-18T07:34:15.993450+0000","last_fullsized":"2024-09-18T07:34:15.993450+0000","mapping_epoch":81,"log_start":"101'3327","ondisk_log_start":"101'3327","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5174,"log_dups_size":0,"ondisk_log_size":5174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301858840,"num_objects":4607,"num_object_clones":0,"num_object_copies":13821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8501,"num_write_kb":419393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":205,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965382+0000","last_change":"2024-09-18T07:34:03.832922+0000","last_active":"2024-09-18T07:34:15.965382+0000","last_peered":"2024-09-18T07:34:15.965382+0000","last_clean":"2024-09-18T07:34:15.965382+0000","last_became_active":"2024-09-18T07:32:26.785711+0000","last_became_peered":"2024-09-18T07:32:26.785711+0000","last_unstale":"2024-09-18T07:34:15.965382+0000","last_undegraded":"2024-09-18T07:34:15.965382+0000","last_fullsized":"2024-09-18T07:34:15.965382+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:34:03.832858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:03.832858+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:25.331056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":170,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966126+0000","last_change":"2024-09-18T07:32:30.829352+0000","last_active":"2024-09-18T07:34:15.966126+0000","last_peered":"2024-09-18T07:34:15.966126+0000","last_clean":"2024-09-18T07:34:15.966126+0000","last_became_active":"2024-09-18T07:32:30.828736+0000","last_became_peered":"2024-09-18T07:32:30.828736+0000","last_unstale":"2024-09-18T07:34:15.966126+0000","last_undegraded":"2024-09-18T07:34:15.966126+0000","last_fullsized":"2024-09-18T07:34:15.966126+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:19.900370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":162,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.967138+0000","last_change":"2024-09-18T07:33:36.582039+0000","last_active":"2024-09-18T07:34:15.967138+0000","last_peered":"2024-09-18T07:34:15.967138+0000","last_clean":"2024-09-18T07:34:15.967138+0000","last_became_active":"2024-09-18T07:31:00.336172+0000","last_became_peered":"2024-09-18T07:31:00.336172+0000","last_unstale":"2024-09-18T07:34:15.967138+0000","last_undegraded":"2024-09-18T07:34:15.967138+0000","last_fullsized":"2024-09-18T07:34:15.967138+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:36.581998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.581998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:04.288035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.969330+0000","last_change":"2024-09-18T07:33:20.990542+0000","last_active":"2024-09-18T07:34:15.969330+0000","last_peered":"2024-09-18T07:34:15.969330+0000","last_clean":"2024-09-18T07:34:15.969330+0000","last_became_active":"2024-09-18T07:31:02.490953+0000","last_became_peered":"2024-09-18T07:31:02.490953+0000","last_unstale":"2024-09-18T07:34:15.969330+0000","last_undegraded":"2024-09-18T07:34:15.969330+0000","last_fullsized":"2024-09-18T07:34:15.969330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:20.990401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:20.990401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.208665+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"136'8449","reported_seq":8573,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.990340+0000","last_change":"2024-09-18T07:32:08.060282+0000","last_active":"2024-09-18T07:34:15.990340+0000","last_peered":"2024-09-18T07:34:15.990340+0000","last_clean":"2024-09-18T07:34:15.990340+0000","last_became_active":"2024-09-18T07:32:08.059971+0000","last_became_peered":"2024-09-18T07:32:08.059971+0000","last_unstale":"2024-09-18T07:34:15.990340+0000","last_undegraded":"2024-09-18T07:34:15.990340+0000","last_fullsized":"2024-09-18T07:34:15.990340+0000","mapping_epoch":81,"log_start":"100'3182","ondisk_log_start":"100'3182","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5267,"log_dups_size":0,"ondisk_log_size":5267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298254336,"num_objects":4551,"num_object_clones":0,"num_object_copies":13653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8449,"num_write_kb":416000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":195,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.964752+0000","last_change":"2024-09-18T07:33:56.797549+0000","last_active":"2024-09-18T07:34:15.964752+0000","last_peered":"2024-09-18T07:34:15.964752+0000","last_clean":"2024-09-18T07:34:15.964752+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:34:15.964752+0000","last_undegraded":"2024-09-18T07:34:15.964752+0000","last_fullsized":"2024-09-18T07:34:15.964752+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:33:56.797466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:56.797466+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:14.685704+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":162,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.962226+0000","last_change":"2024-09-18T07:30:56.284303+0000","last_active":"2024-09-18T07:34:15.962226+0000","last_peered":"2024-09-18T07:34:15.962226+0000","last_clean":"2024-09-18T07:34:15.962226+0000","last_became_active":"2024-09-18T07:30:56.284020+0000","last_became_peered":"2024-09-18T07:30:56.284020+0000","last_unstale":"2024-09-18T07:34:15.962226+0000","last_undegraded":"2024-09-18T07:34:15.962226+0000","last_fullsized":"2024-09-18T07:34:15.962226+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.835065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":163,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.973027+0000","last_change":"2024-09-18T07:32:05.239202+0000","last_active":"2024-09-18T07:34:15.973027+0000","last_peered":"2024-09-18T07:34:15.973027+0000","last_clean":"2024-09-18T07:34:15.973027+0000","last_became_active":"2024-09-18T07:31:00.332836+0000","last_became_peered":"2024-09-18T07:31:00.332836+0000","last_unstale":"2024-09-18T07:34:15.973027+0000","last_undegraded":"2024-09-18T07:34:15.973027+0000","last_fullsized":"2024-09-18T07:34:15.973027+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:05.239160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.239160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:37.915616+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":159,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.977405+0000","last_change":"2024-09-18T07:33:21.860561+0000","last_active":"2024-09-18T07:34:15.977405+0000","last_peered":"2024-09-18T07:34:15.977405+0000","last_clean":"2024-09-18T07:34:15.977405+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:34:15.977405+0000","last_undegraded":"2024-09-18T07:34:15.977405+0000","last_fullsized":"2024-09-18T07:34:15.977405+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.860495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.860495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:51.431373+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"136'8450","reported_seq":12408,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.978201+0000","last_change":"2024-09-18T07:33:14.766074+0000","last_active":"2024-09-18T07:34:15.978201+0000","last_peered":"2024-09-18T07:34:15.978201+0000","last_clean":"2024-09-18T07:34:15.978201+0000","last_became_active":"2024-09-18T07:33:14.762251+0000","last_became_peered":"2024-09-18T07:33:14.762251+0000","last_unstale":"2024-09-18T07:34:15.978201+0000","last_undegraded":"2024-09-18T07:34:15.978201+0000","last_fullsized":"2024-09-18T07:34:15.978201+0000","mapping_epoch":115,"log_start":"114'4340","ondisk_log_start":"114'4340","created":81,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4110,"log_dups_size":0,"ondisk_log_size":4110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307625984,"num_objects":4694,"num_object_clones":0,"num_object_copies":14082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8450,"num_write_kb":420608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1912,"num_bytes_recovered":125304832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":182,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.964894+0000","last_change":"2024-09-18T07:32:05.000363+0000","last_active":"2024-09-18T07:34:15.964894+0000","last_peered":"2024-09-18T07:34:15.964894+0000","last_clean":"2024-09-18T07:34:15.964894+0000","last_became_active":"2024-09-18T07:30:58.316423+0000","last_became_peered":"2024-09-18T07:30:58.316423+0000","last_unstale":"2024-09-18T07:34:15.964894+0000","last_undegraded":"2024-09-18T07:34:15.964894+0000","last_fullsized":"2024-09-18T07:34:15.964894+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:32:05.000310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.000310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":170,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966669+0000","last_change":"2024-09-18T07:32:06.688493+0000","last_active":"2024-09-18T07:34:15.966669+0000","last_peered":"2024-09-18T07:34:15.966669+0000","last_clean":"2024-09-18T07:34:15.966669+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:34:15.966669+0000","last_undegraded":"2024-09-18T07:34:15.966669+0000","last_fullsized":"2024-09-18T07:34:15.966669+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:06.688444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:06.688444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:12.324753+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"120'10","reported_seq":234,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.977560+0000","last_change":"2024-09-18T07:33:33.001389+0000","last_active":"2024-09-18T07:34:15.977560+0000","last_peered":"2024-09-18T07:34:15.977560+0000","last_clean":"2024-09-18T07:34:15.977560+0000","last_became_active":"2024-09-18T07:33:27.007386+0000","last_became_peered":"2024-09-18T07:33:27.007386+0000","last_unstale":"2024-09-18T07:34:15.977560+0000","last_undegraded":"2024-09-18T07:34:15.977560+0000","last_fullsized":"2024-09-18T07:34:15.977560+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'10","last_scrub_stamp":"2024-09-18T07:33:33.000874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.000874+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:48.823137+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":151,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965102+0000","last_change":"2024-09-18T07:31:02.347721+0000","last_active":"2024-09-18T07:34:15.965102+0000","last_peered":"2024-09-18T07:34:15.965102+0000","last_clean":"2024-09-18T07:34:15.965102+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:34:15.965102+0000","last_undegraded":"2024-09-18T07:34:15.965102+0000","last_fullsized":"2024-09-18T07:34:15.965102+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"136'8532","reported_seq":12768,"reported_epoch":136,"state":"active+clean+scrubbing","last_fresh":"2024-09-18T07:34:16.046197+0000","last_change":"2024-09-18T07:34:16.046084+0000","last_active":"2024-09-18T07:34:16.046197+0000","last_peered":"2024-09-18T07:34:16.046197+0000","last_clean":"2024-09-18T07:34:16.046197+0000","last_became_active":"2024-09-18T07:34:14.978102+0000","last_became_peered":"2024-09-18T07:34:14.978102+0000","last_unstale":"2024-09-18T07:34:16.046197+0000","last_undegraded":"2024-09-18T07:34:16.046197+0000","last_fullsized":"2024-09-18T07:34:16.046197+0000","mapping_epoch":110,"log_start":"98'2800","ondisk_log_start":"98'2800","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7533","last_scrub_stamp":"2024-09-18T07:34:00.127026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:00.127026+0000","objects_scrubbed":0,"log_size":5732,"log_dups_size":0,"ondisk_log_size":5732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"scrubbing for 0s","scrub_duration":2209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151650304,"num_objects":2314,"num_object_clones":0,"num_object_copies":6942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4287,"num_write_kb":211232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1005,"num_bytes_recovered":65896448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":174,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.977027+0000","last_change":"2024-09-18T07:30:58.314424+0000","last_active":"2024-09-18T07:34:15.977027+0000","last_peered":"2024-09-18T07:34:15.977027+0000","last_clean":"2024-09-18T07:34:15.977027+0000","last_became_active":"2024-09-18T07:30:58.314127+0000","last_became_peered":"2024-09-18T07:30:58.314127+0000","last_unstale":"2024-09-18T07:34:15.977027+0000","last_undegraded":"2024-09-18T07:34:15.977027+0000","last_fullsized":"2024-09-18T07:34:15.977027+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:37.131991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":170,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966598+0000","last_change":"2024-09-18T07:33:28.459796+0000","last_active":"2024-09-18T07:34:15.966598+0000","last_peered":"2024-09-18T07:34:15.966598+0000","last_clean":"2024-09-18T07:34:15.966598+0000","last_became_active":"2024-09-18T07:30:56.288868+0000","last_became_peered":"2024-09-18T07:30:56.288868+0000","last_unstale":"2024-09-18T07:34:15.966598+0000","last_undegraded":"2024-09-18T07:34:15.966598+0000","last_fullsized":"2024-09-18T07:34:15.966598+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:28.459747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:28.459747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:54.082899+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":155,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.970663+0000","last_change":"2024-09-18T07:31:00.336020+0000","last_active":"2024-09-18T07:34:15.970663+0000","last_peered":"2024-09-18T07:34:15.970663+0000","last_clean":"2024-09-18T07:34:15.970663+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:34:15.970663+0000","last_undegraded":"2024-09-18T07:34:15.970663+0000","last_fullsized":"2024-09-18T07:34:15.970663+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":165,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.974955+0000","last_change":"2024-09-18T07:34:02.301413+0000","last_active":"2024-09-18T07:34:15.974955+0000","last_peered":"2024-09-18T07:34:15.974955+0000","last_clean":"2024-09-18T07:34:15.974955+0000","last_became_active":"2024-09-18T07:32:30.854029+0000","last_became_peered":"2024-09-18T07:32:30.854029+0000","last_unstale":"2024-09-18T07:34:15.974955+0000","last_undegraded":"2024-09-18T07:34:15.974955+0000","last_fullsized":"2024-09-18T07:34:15.974955+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:02.301353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:34:02.301353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:16.198563+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"135'8456","reported_seq":8591,"reported_epoch":136,"state":"active+clean+scrubbing","last_fresh":"2024-09-18T07:34:16.467597+0000","last_change":"2024-09-18T07:34:16.467431+0000","last_active":"2024-09-18T07:34:16.467597+0000","last_peered":"2024-09-18T07:34:16.467597+0000","last_clean":"2024-09-18T07:34:16.467597+0000","last_became_active":"2024-09-18T07:34:15.002468+0000","last_became_peered":"2024-09-18T07:34:15.002468+0000","last_unstale":"2024-09-18T07:34:16.467597+0000","last_undegraded":"2024-09-18T07:34:16.467597+0000","last_fullsized":"2024-09-18T07:34:16.467597+0000","mapping_epoch":81,"log_start":"108'3802","ondisk_log_start":"108'3802","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4654,"log_dups_size":0,"ondisk_log_size":4654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153354240,"num_objects":2340,"num_object_clones":0,"num_object_copies":7020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4246,"num_write_kb":210752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":232,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.961964+0000","last_change":"2024-09-18T07:33:32.803961+0000","last_active":"2024-09-18T07:34:15.961964+0000","last_peered":"2024-09-18T07:34:15.961964+0000","last_clean":"2024-09-18T07:34:15.961964+0000","last_became_active":"2024-09-18T07:32:30.834955+0000","last_became_peered":"2024-09-18T07:32:30.834955+0000","last_unstale":"2024-09-18T07:34:15.961964+0000","last_undegraded":"2024-09-18T07:34:15.961964+0000","last_fullsized":"2024-09-18T07:34:15.961964+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:33:32.803884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.803884+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:34.097591+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":98,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966835+0000","last_change":"2024-09-18T07:32:30.840417+0000","last_active":"2024-09-18T07:34:15.966835+0000","last_peered":"2024-09-18T07:34:15.966835+0000","last_clean":"2024-09-18T07:34:15.966835+0000","last_became_active":"2024-09-18T07:32:30.840046+0000","last_became_peered":"2024-09-18T07:32:30.840046+0000","last_unstale":"2024-09-18T07:34:15.966835+0000","last_undegraded":"2024-09-18T07:34:15.966835+0000","last_fullsized":"2024-09-18T07:34:15.966835+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.481153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.481153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.709514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":170,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.972265+0000","last_change":"2024-09-18T07:33:34.966610+0000","last_active":"2024-09-18T07:34:15.972265+0000","last_peered":"2024-09-18T07:34:15.972265+0000","last_clean":"2024-09-18T07:34:15.972265+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:34:15.972265+0000","last_undegraded":"2024-09-18T07:34:15.972265+0000","last_fullsized":"2024-09-18T07:34:15.972265+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:34.966537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:34.966537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.041153+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":97,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.972660+0000","last_change":"2024-09-18T07:32:30.839764+0000","last_active":"2024-09-18T07:34:15.972660+0000","last_peered":"2024-09-18T07:34:15.972660+0000","last_clean":"2024-09-18T07:34:15.972660+0000","last_became_active":"2024-09-18T07:32:30.839399+0000","last_became_peered":"2024-09-18T07:32:30.839399+0000","last_unstale":"2024-09-18T07:34:15.972660+0000","last_undegraded":"2024-09-18T07:34:15.972660+0000","last_fullsized":"2024-09-18T07:34:15.972660+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:06.911907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.a","version":"136'8454","reported_seq":8577,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966353+0000","last_change":"2024-09-18T07:32:08.065180+0000","last_active":"2024-09-18T07:34:15.966353+0000","last_peered":"2024-09-18T07:34:15.966353+0000","last_clean":"2024-09-18T07:34:15.966353+0000","last_became_active":"2024-09-18T07:32:08.064911+0000","last_became_peered":"2024-09-18T07:32:08.064911+0000","last_unstale":"2024-09-18T07:34:15.966353+0000","last_undegraded":"2024-09-18T07:34:15.966353+0000","last_fullsized":"2024-09-18T07:34:15.966353+0000","mapping_epoch":81,"log_start":"116'4656","ondisk_log_start":"116'4656","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":3798,"log_dups_size":0,"ondisk_log_size":3798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302907392,"num_objects":4622,"num_object_clones":0,"num_object_copies":13866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8454,"num_write_kb":418432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":190,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965921+0000","last_change":"2024-09-18T07:32:02.967912+0000","last_active":"2024-09-18T07:34:15.965921+0000","last_peered":"2024-09-18T07:34:15.965921+0000","last_clean":"2024-09-18T07:34:15.965921+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:34:15.965921+0000","last_undegraded":"2024-09-18T07:34:15.965921+0000","last_fullsized":"2024-09-18T07:34:15.965921+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:32:02.967835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.967835+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":98,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965542+0000","last_change":"2024-09-18T07:32:30.846539+0000","last_active":"2024-09-18T07:34:15.965542+0000","last_peered":"2024-09-18T07:34:15.965542+0000","last_clean":"2024-09-18T07:34:15.965542+0000","last_became_active":"2024-09-18T07:32:30.846204+0000","last_became_peered":"2024-09-18T07:32:30.846204+0000","last_unstale":"2024-09-18T07:34:15.965542+0000","last_undegraded":"2024-09-18T07:34:15.965542+0000","last_fullsized":"2024-09-18T07:34:15.965542+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:17.888739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":171,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.972819+0000","last_change":"2024-09-18T07:33:56.967701+0000","last_active":"2024-09-18T07:34:15.972819+0000","last_peered":"2024-09-18T07:34:15.972819+0000","last_clean":"2024-09-18T07:34:15.972819+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:34:15.972819+0000","last_undegraded":"2024-09-18T07:34:15.972819+0000","last_fullsized":"2024-09-18T07:34:15.972819+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:56.967627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:56.967627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:18.872883+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":157,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966354+0000","last_change":"2024-09-18T07:32:30.849280+0000","last_active":"2024-09-18T07:34:15.966354+0000","last_peered":"2024-09-18T07:34:15.966354+0000","last_clean":"2024-09-18T07:34:15.966354+0000","last_became_active":"2024-09-18T07:32:30.847635+0000","last_became_peered":"2024-09-18T07:32:30.847635+0000","last_unstale":"2024-09-18T07:34:15.966354+0000","last_undegraded":"2024-09-18T07:34:15.966354+0000","last_fullsized":"2024-09-18T07:34:15.966354+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:48.144692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"136'8259","reported_seq":8383,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.995916+0000","last_change":"2024-09-18T07:32:08.062243+0000","last_active":"2024-09-18T07:34:15.995916+0000","last_peered":"2024-09-18T07:34:15.995916+0000","last_clean":"2024-09-18T07:34:15.995916+0000","last_became_active":"2024-09-18T07:32:08.061925+0000","last_became_peered":"2024-09-18T07:32:08.061925+0000","last_unstale":"2024-09-18T07:34:15.995916+0000","last_undegraded":"2024-09-18T07:34:15.995916+0000","last_fullsized":"2024-09-18T07:34:15.995916+0000","mapping_epoch":81,"log_start":"114'4281","ondisk_log_start":"114'4281","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":3978,"log_dups_size":0,"ondisk_log_size":3978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298385408,"num_objects":4553,"num_object_clones":0,"num_object_copies":13659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8259,"num_write_kb":409984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":214,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965553+0000","last_change":"2024-09-18T07:34:03.531341+0000","last_active":"2024-09-18T07:34:15.965553+0000","last_peered":"2024-09-18T07:34:15.965553+0000","last_clean":"2024-09-18T07:34:15.965553+0000","last_became_active":"2024-09-18T07:32:30.832486+0000","last_became_peered":"2024-09-18T07:32:30.832486+0000","last_unstale":"2024-09-18T07:34:15.965553+0000","last_undegraded":"2024-09-18T07:34:15.965553+0000","last_fullsized":"2024-09-18T07:34:15.965553+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:34:03.531262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:03.531262+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:10.299055+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":114,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965313+0000","last_change":"2024-09-18T07:32:30.865327+0000","last_active":"2024-09-18T07:34:15.965313+0000","last_peered":"2024-09-18T07:34:15.965313+0000","last_clean":"2024-09-18T07:34:15.965313+0000","last_became_active":"2024-09-18T07:32:30.844389+0000","last_became_peered":"2024-09-18T07:32:30.844389+0000","last_unstale":"2024-09-18T07:34:15.965313+0000","last_undegraded":"2024-09-18T07:34:15.965313+0000","last_fullsized":"2024-09-18T07:34:15.965313+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:09.798229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":157,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.962305+0000","last_change":"2024-09-18T07:32:57.792858+0000","last_active":"2024-09-18T07:34:15.962305+0000","last_peered":"2024-09-18T07:34:15.962305+0000","last_clean":"2024-09-18T07:34:15.962305+0000","last_became_active":"2024-09-18T07:32:30.831224+0000","last_became_peered":"2024-09-18T07:32:30.831224+0000","last_unstale":"2024-09-18T07:34:15.962305+0000","last_undegraded":"2024-09-18T07:34:15.962305+0000","last_fullsized":"2024-09-18T07:34:15.962305+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-09-18T07:32:57.792795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:57.792795+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:08.758097+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":150,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965959+0000","last_change":"2024-09-18T07:31:02.489914+0000","last_active":"2024-09-18T07:34:15.965959+0000","last_peered":"2024-09-18T07:34:15.965959+0000","last_clean":"2024-09-18T07:34:15.965959+0000","last_became_active":"2024-09-18T07:31:02.489619+0000","last_became_peered":"2024-09-18T07:31:02.489619+0000","last_unstale":"2024-09-18T07:34:15.965959+0000","last_undegraded":"2024-09-18T07:34:15.965959+0000","last_fullsized":"2024-09-18T07:34:15.965959+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"unknown","last_fresh":"2024-09-18T07:34:15.975124+0000","last_change":"2024-09-18T07:34:16.583682+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T07:34:15.975124+0000","last_undegraded":"2024-09-18T07:34:15.975124+0000","last_fullsized":"2024-09-18T07:34:15.975124+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":170,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.961402+0000","last_change":"2024-09-18T07:32:04.369688+0000","last_active":"2024-09-18T07:34:15.961402+0000","last_peered":"2024-09-18T07:34:15.961402+0000","last_clean":"2024-09-18T07:34:15.961402+0000","last_became_active":"2024-09-18T07:30:56.282357+0000","last_became_peered":"2024-09-18T07:30:56.282357+0000","last_unstale":"2024-09-18T07:34:15.961402+0000","last_undegraded":"2024-09-18T07:34:15.961402+0000","last_fullsized":"2024-09-18T07:34:15.961402+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.369620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.369620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":182,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.974697+0000","last_change":"2024-09-18T07:32:30.872792+0000","last_active":"2024-09-18T07:34:15.974697+0000","last_peered":"2024-09-18T07:34:15.974697+0000","last_clean":"2024-09-18T07:34:15.974697+0000","last_became_active":"2024-09-18T07:32:30.856211+0000","last_became_peered":"2024-09-18T07:32:30.856211+0000","last_unstale":"2024-09-18T07:34:15.974697+0000","last_undegraded":"2024-09-18T07:34:15.974697+0000","last_fullsized":"2024-09-18T07:34:15.974697+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:19.033885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":162,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966990+0000","last_change":"2024-09-18T07:33:32.589362+0000","last_active":"2024-09-18T07:34:15.966990+0000","last_peered":"2024-09-18T07:34:15.966990+0000","last_clean":"2024-09-18T07:34:15.966990+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:34:15.966990+0000","last_undegraded":"2024-09-18T07:34:15.966990+0000","last_fullsized":"2024-09-18T07:34:15.966990+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.589289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.589289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:38.300848+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":113,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.963043+0000","last_change":"2024-09-18T07:33:58.688688+0000","last_active":"2024-09-18T07:34:15.963043+0000","last_peered":"2024-09-18T07:34:15.963043+0000","last_clean":"2024-09-18T07:34:15.963043+0000","last_became_active":"2024-09-18T07:32:30.831805+0000","last_became_peered":"2024-09-18T07:32:30.831805+0000","last_unstale":"2024-09-18T07:34:15.963043+0000","last_undegraded":"2024-09-18T07:34:15.963043+0000","last_fullsized":"2024-09-18T07:34:15.963043+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:58.688631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:58.688631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:24.030416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.15","version":"135'8368","reported_seq":12312,"reported_epoch":136,"state":"unknown","last_fresh":"2024-09-18T07:34:15.972299+0000","last_change":"2024-09-18T07:34:16.520430+0000","last_active":"2024-09-18T07:34:15.972299+0000","last_peered":"2024-09-18T07:34:15.939809+0000","last_clean":"2024-09-18T07:34:15.939809+0000","last_became_active":"2024-09-18T07:34:15.027782+0000","last_became_peered":"2024-09-18T07:34:15.027782+0000","last_unstale":"2024-09-18T07:34:15.972299+0000","last_undegraded":"2024-09-18T07:34:15.972299+0000","last_fullsized":"2024-09-18T07:34:15.972299+0000","mapping_epoch":136,"log_start":"98'2600","ondisk_log_start":"98'2600","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5768,"log_dups_size":0,"ondisk_log_size":5768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153452544,"num_objects":2342,"num_object_clones":0,"num_object_copies":7026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4208,"num_write_kb":209568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":952,"num_bytes_recovered":62390272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":191,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.978776+0000","last_change":"2024-09-18T07:32:30.868702+0000","last_active":"2024-09-18T07:34:15.978776+0000","last_peered":"2024-09-18T07:34:15.978776+0000","last_clean":"2024-09-18T07:34:15.978776+0000","last_became_active":"2024-09-18T07:32:30.861402+0000","last_became_peered":"2024-09-18T07:32:30.861402+0000","last_unstale":"2024-09-18T07:34:15.978776+0000","last_undegraded":"2024-09-18T07:34:15.978776+0000","last_fullsized":"2024-09-18T07:34:15.978776+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:12.449151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":98,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965863+0000","last_change":"2024-09-18T07:32:30.842334+0000","last_active":"2024-09-18T07:34:15.965863+0000","last_peered":"2024-09-18T07:34:15.965863+0000","last_clean":"2024-09-18T07:34:15.965863+0000","last_became_active":"2024-09-18T07:32:30.841928+0000","last_became_peered":"2024-09-18T07:32:30.841928+0000","last_unstale":"2024-09-18T07:34:15.965863+0000","last_undegraded":"2024-09-18T07:34:15.965863+0000","last_fullsized":"2024-09-18T07:34:15.965863+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":169,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.974548+0000","last_change":"2024-09-18T07:33:58.276319+0000","last_active":"2024-09-18T07:34:15.974548+0000","last_peered":"2024-09-18T07:34:15.974548+0000","last_clean":"2024-09-18T07:34:15.974548+0000","last_became_active":"2024-09-18T07:32:30.858167+0000","last_became_peered":"2024-09-18T07:32:30.858167+0000","last_unstale":"2024-09-18T07:34:15.974548+0000","last_undegraded":"2024-09-18T07:34:15.974548+0000","last_fullsized":"2024-09-18T07:34:15.974548+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:58.276259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:58.276259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:27.549455+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.981812+0000","last_change":"2024-09-18T07:32:30.835415+0000","last_active":"2024-09-18T07:34:15.981812+0000","last_peered":"2024-09-18T07:34:15.981812+0000","last_clean":"2024-09-18T07:34:15.981812+0000","last_became_active":"2024-09-18T07:32:30.834708+0000","last_became_peered":"2024-09-18T07:32:30.834708+0000","last_unstale":"2024-09-18T07:34:15.981812+0000","last_undegraded":"2024-09-18T07:34:15.981812+0000","last_fullsized":"2024-09-18T07:34:15.981812+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:07.940458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"unknown","last_fresh":"2024-09-18T07:34:15.967070+0000","last_change":"2024-09-18T07:34:16.434105+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T07:34:15.967070+0000","last_undegraded":"2024-09-18T07:34:15.967070+0000","last_fullsized":"2024-09-18T07:34:15.967070+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":191,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966867+0000","last_change":"2024-09-18T07:32:30.851755+0000","last_active":"2024-09-18T07:34:15.966867+0000","last_peered":"2024-09-18T07:34:15.966867+0000","last_clean":"2024-09-18T07:34:15.966867+0000","last_became_active":"2024-09-18T07:32:30.837622+0000","last_became_peered":"2024-09-18T07:32:30.837622+0000","last_unstale":"2024-09-18T07:34:15.966867+0000","last_undegraded":"2024-09-18T07:34:15.966867+0000","last_fullsized":"2024-09-18T07:34:15.966867+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:12.441533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":169,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.976522+0000","last_change":"2024-09-18T07:32:30.854427+0000","last_active":"2024-09-18T07:34:15.976522+0000","last_peered":"2024-09-18T07:34:15.976522+0000","last_clean":"2024-09-18T07:34:15.976522+0000","last_became_active":"2024-09-18T07:32:30.853969+0000","last_became_peered":"2024-09-18T07:32:30.853969+0000","last_unstale":"2024-09-18T07:34:15.976522+0000","last_undegraded":"2024-09-18T07:34:15.976522+0000","last_fullsized":"2024-09-18T07:34:15.976522+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"94'9","reported_seq":214,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.969492+0000","last_change":"2024-09-18T07:32:30.867244+0000","last_active":"2024-09-18T07:34:15.969492+0000","last_peered":"2024-09-18T07:34:15.969492+0000","last_clean":"2024-09-18T07:34:15.969492+0000","last_became_active":"2024-09-18T07:32:30.848580+0000","last_became_peered":"2024-09-18T07:32:30.848580+0000","last_unstale":"2024-09-18T07:34:15.969492+0000","last_undegraded":"2024-09-18T07:34:15.969492+0000","last_fullsized":"2024-09-18T07:34:15.969492+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.090692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":151,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965050+0000","last_change":"2024-09-18T07:31:02.343206+0000","last_active":"2024-09-18T07:34:15.965050+0000","last_peered":"2024-09-18T07:34:15.965050+0000","last_clean":"2024-09-18T07:34:15.965050+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:34:15.965050+0000","last_undegraded":"2024-09-18T07:34:15.965050+0000","last_fullsized":"2024-09-18T07:34:15.965050+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"unknown","last_fresh":"2024-09-18T07:34:15.964125+0000","last_change":"2024-09-18T07:34:16.438027+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T07:34:15.964125+0000","last_undegraded":"2024-09-18T07:34:15.964125+0000","last_fullsized":"2024-09-18T07:34:15.964125+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'5435","last_scrub_stamp":"2024-09-18T07:33:31.180958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:33:31.180958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":200,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.969835+0000","last_change":"2024-09-18T07:33:54.295407+0000","last_active":"2024-09-18T07:34:15.969835+0000","last_peered":"2024-09-18T07:34:15.969835+0000","last_clean":"2024-09-18T07:34:15.969835+0000","last_became_active":"2024-09-18T07:32:30.848136+0000","last_became_peered":"2024-09-18T07:32:30.848136+0000","last_unstale":"2024-09-18T07:34:15.969835+0000","last_undegraded":"2024-09-18T07:34:15.969835+0000","last_fullsized":"2024-09-18T07:34:15.969835+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:33:54.295330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:54.295330+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:20.154237+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":98,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965123+0000","last_change":"2024-09-18T07:32:30.832648+0000","last_active":"2024-09-18T07:34:15.965123+0000","last_peered":"2024-09-18T07:34:15.965123+0000","last_clean":"2024-09-18T07:34:15.965123+0000","last_became_active":"2024-09-18T07:32:30.832312+0000","last_became_peered":"2024-09-18T07:32:30.832312+0000","last_unstale":"2024-09-18T07:34:15.965123+0000","last_undegraded":"2024-09-18T07:34:15.965123+0000","last_fullsized":"2024-09-18T07:34:15.965123+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:16.844687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":206,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966806+0000","last_change":"2024-09-18T07:31:00.332729+0000","last_active":"2024-09-18T07:34:15.966806+0000","last_peered":"2024-09-18T07:34:15.966806+0000","last_clean":"2024-09-18T07:34:15.966806+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:34:15.966806+0000","last_undegraded":"2024-09-18T07:34:15.966806+0000","last_fullsized":"2024-09-18T07:34:15.966806+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:14.413751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":165,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.978231+0000","last_change":"2024-09-18T07:33:32.135015+0000","last_active":"2024-09-18T07:34:15.978231+0000","last_peered":"2024-09-18T07:34:15.978231+0000","last_clean":"2024-09-18T07:34:15.978231+0000","last_became_active":"2024-09-18T07:32:30.854291+0000","last_became_peered":"2024-09-18T07:32:30.854291+0000","last_unstale":"2024-09-18T07:34:15.978231+0000","last_undegraded":"2024-09-18T07:34:15.978231+0000","last_fullsized":"2024-09-18T07:34:15.978231+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.134847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.134847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:34.913547+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"135'8365","reported_seq":8492,"reported_epoch":136,"state":"unknown","last_fresh":"2024-09-18T07:34:15.965530+0000","last_change":"2024-09-18T07:34:16.583587+0000","last_active":"2024-09-18T07:34:15.965530+0000","last_peered":"2024-09-18T07:34:15.862563+0000","last_clean":"2024-09-18T07:34:15.862563+0000","last_became_active":"2024-09-18T07:34:14.990958+0000","last_became_peered":"2024-09-18T07:34:14.990958+0000","last_unstale":"2024-09-18T07:34:15.965530+0000","last_undegraded":"2024-09-18T07:34:15.965530+0000","last_fullsized":"2024-09-18T07:34:15.965530+0000","mapping_epoch":136,"log_start":"100'3056","ondisk_log_start":"100'3056","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5309,"log_dups_size":0,"ondisk_log_size":5309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150175744,"num_objects":2292,"num_object_clones":0,"num_object_copies":6876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4203,"num_write_kb":207808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":178,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965012+0000","last_change":"2024-09-18T07:30:58.314730+0000","last_active":"2024-09-18T07:34:15.965012+0000","last_peered":"2024-09-18T07:34:15.965012+0000","last_clean":"2024-09-18T07:34:15.965012+0000","last_became_active":"2024-09-18T07:30:58.314428+0000","last_became_peered":"2024-09-18T07:30:58.314428+0000","last_unstale":"2024-09-18T07:34:15.965012+0000","last_undegraded":"2024-09-18T07:34:15.965012+0000","last_fullsized":"2024-09-18T07:34:15.965012+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":170,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965136+0000","last_change":"2024-09-18T07:32:30.831845+0000","last_active":"2024-09-18T07:34:15.965136+0000","last_peered":"2024-09-18T07:34:15.965136+0000","last_clean":"2024-09-18T07:34:15.965136+0000","last_became_active":"2024-09-18T07:32:30.831546+0000","last_became_peered":"2024-09-18T07:32:30.831546+0000","last_unstale":"2024-09-18T07:34:15.965136+0000","last_undegraded":"2024-09-18T07:34:15.965136+0000","last_fullsized":"2024-09-18T07:34:15.965136+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:38.680767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":155,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.972273+0000","last_change":"2024-09-18T07:31:00.337826+0000","last_active":"2024-09-18T07:34:15.972273+0000","last_peered":"2024-09-18T07:34:15.972273+0000","last_clean":"2024-09-18T07:34:15.972273+0000","last_became_active":"2024-09-18T07:31:00.334158+0000","last_became_peered":"2024-09-18T07:31:00.334158+0000","last_unstale":"2024-09-18T07:34:15.972273+0000","last_undegraded":"2024-09-18T07:34:15.972273+0000","last_fullsized":"2024-09-18T07:34:15.972273+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:20.058588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":150,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.969312+0000","last_change":"2024-09-18T07:31:02.351247+0000","last_active":"2024-09-18T07:34:15.969312+0000","last_peered":"2024-09-18T07:34:15.969312+0000","last_clean":"2024-09-18T07:34:15.969312+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:34:15.969312+0000","last_undegraded":"2024-09-18T07:34:15.969312+0000","last_fullsized":"2024-09-18T07:34:15.969312+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.189968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"135'8531","reported_seq":8658,"reported_epoch":136,"state":"unknown","last_fresh":"2024-09-18T07:34:15.961960+0000","last_change":"2024-09-18T07:34:16.445649+0000","last_active":"2024-09-18T07:34:15.961960+0000","last_peered":"2024-09-18T07:34:15.874672+0000","last_clean":"2024-09-18T07:34:15.874672+0000","last_became_active":"2024-09-18T07:34:14.982067+0000","last_became_peered":"2024-09-18T07:34:14.982067+0000","last_unstale":"2024-09-18T07:34:15.961960+0000","last_undegraded":"2024-09-18T07:34:15.961960+0000","last_fullsized":"2024-09-18T07:34:15.961960+0000","mapping_epoch":136,"log_start":"113'4100","ondisk_log_start":"113'4100","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4431,"log_dups_size":0,"ondisk_log_size":4431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151683072,"num_objects":2315,"num_object_clones":0,"num_object_copies":6945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4295,"num_write_kb":211488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":54,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.976234+0000","last_change":"2024-09-18T07:33:27.033510+0000","last_active":"2024-09-18T07:34:15.976234+0000","last_peered":"2024-09-18T07:34:15.976234+0000","last_clean":"2024-09-18T07:34:15.976234+0000","last_became_active":"2024-09-18T07:33:27.003887+0000","last_became_peered":"2024-09-18T07:33:27.003887+0000","last_unstale":"2024-09-18T07:34:15.976234+0000","last_undegraded":"2024-09-18T07:34:15.976234+0000","last_fullsized":"2024-09-18T07:34:15.976234+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.187186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.187186+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":98,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.965228+0000","last_change":"2024-09-18T07:32:30.847936+0000","last_active":"2024-09-18T07:34:15.965228+0000","last_peered":"2024-09-18T07:34:15.965228+0000","last_clean":"2024-09-18T07:34:15.965228+0000","last_became_active":"2024-09-18T07:32:30.847608+0000","last_became_peered":"2024-09-18T07:32:30.847608+0000","last_unstale":"2024-09-18T07:34:15.965228+0000","last_undegraded":"2024-09-18T07:34:15.965228+0000","last_fullsized":"2024-09-18T07:34:15.965228+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":154,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966445+0000","last_change":"2024-09-18T07:31:00.333689+0000","last_active":"2024-09-18T07:34:15.966445+0000","last_peered":"2024-09-18T07:34:15.966445+0000","last_clean":"2024-09-18T07:34:15.966445+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:34:15.966445+0000","last_undegraded":"2024-09-18T07:34:15.966445+0000","last_fullsized":"2024-09-18T07:34:15.966445+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.167699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.974577+0000","last_change":"2024-09-18T07:32:04.503356+0000","last_active":"2024-09-18T07:34:15.974577+0000","last_peered":"2024-09-18T07:34:15.974577+0000","last_clean":"2024-09-18T07:34:15.974577+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:34:15.974577+0000","last_undegraded":"2024-09-18T07:34:15.974577+0000","last_fullsized":"2024-09-18T07:34:15.974577+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.503270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.503270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"unknown","last_fresh":"2024-09-18T07:34:15.971868+0000","last_change":"2024-09-18T07:34:16.445666+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T07:34:15.971868+0000","last_undegraded":"2024-09-18T07:34:15.971868+0000","last_fullsized":"2024-09-18T07:34:15.971868+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":196,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.969498+0000","last_change":"2024-09-18T07:32:30.869693+0000","last_active":"2024-09-18T07:34:15.969498+0000","last_peered":"2024-09-18T07:34:15.969498+0000","last_clean":"2024-09-18T07:34:15.969498+0000","last_became_active":"2024-09-18T07:32:30.842218+0000","last_became_peered":"2024-09-18T07:32:30.842218+0000","last_unstale":"2024-09-18T07:34:15.969498+0000","last_undegraded":"2024-09-18T07:34:15.969498+0000","last_fullsized":"2024-09-18T07:34:15.969498+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:43.146132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":177,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.967538+0000","last_change":"2024-09-18T07:32:30.853892+0000","last_active":"2024-09-18T07:34:15.967538+0000","last_peered":"2024-09-18T07:34:15.967538+0000","last_clean":"2024-09-18T07:34:15.967538+0000","last_became_active":"2024-09-18T07:32:30.853486+0000","last_became_peered":"2024-09-18T07:32:30.853486+0000","last_unstale":"2024-09-18T07:34:15.967538+0000","last_undegraded":"2024-09-18T07:34:15.967538+0000","last_fullsized":"2024-09-18T07:34:15.967538+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:58.734027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:58.734027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:42.752966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":105,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.964046+0000","last_change":"2024-09-18T07:33:30.668600+0000","last_active":"2024-09-18T07:34:15.964046+0000","last_peered":"2024-09-18T07:34:15.964046+0000","last_clean":"2024-09-18T07:34:15.964046+0000","last_became_active":"2024-09-18T07:32:30.843194+0000","last_became_peered":"2024-09-18T07:32:30.843194+0000","last_unstale":"2024-09-18T07:34:15.964046+0000","last_undegraded":"2024-09-18T07:34:15.964046+0000","last_fullsized":"2024-09-18T07:34:15.964046+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.668428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.668428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:55.997265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966586+0000","last_change":"2024-09-18T07:33:30.975087+0000","last_active":"2024-09-18T07:34:15.966586+0000","last_peered":"2024-09-18T07:34:15.966586+0000","last_clean":"2024-09-18T07:34:15.966586+0000","last_became_active":"2024-09-18T07:31:02.490994+0000","last_became_peered":"2024-09-18T07:31:02.490994+0000","last_unstale":"2024-09-18T07:34:15.966586+0000","last_undegraded":"2024-09-18T07:34:15.966586+0000","last_fullsized":"2024-09-18T07:34:15.966586+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.974886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.974886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:40.586219+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"unknown","last_fresh":"2024-09-18T07:34:15.973430+0000","last_change":"2024-09-18T07:34:16.695806+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T07:34:15.973430+0000","last_undegraded":"2024-09-18T07:34:15.973430+0000","last_fullsized":"2024-09-18T07:34:15.973430+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":185,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.966938+0000","last_change":"2024-09-18T07:33:29.988707+0000","last_active":"2024-09-18T07:34:15.966938+0000","last_peered":"2024-09-18T07:34:15.966938+0000","last_clean":"2024-09-18T07:34:15.966938+0000","last_became_active":"2024-09-18T07:32:26.791562+0000","last_became_peered":"2024-09-18T07:32:26.791562+0000","last_unstale":"2024-09-18T07:34:15.966938+0000","last_undegraded":"2024-09-18T07:34:15.966938+0000","last_fullsized":"2024-09-18T07:34:15.966938+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:33:29.988636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.988636+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:34.823616+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":162,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.978507+0000","last_change":"2024-09-18T07:30:56.283643+0000","last_active":"2024-09-18T07:34:15.978507+0000","last_peered":"2024-09-18T07:34:15.978507+0000","last_clean":"2024-09-18T07:34:15.978507+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:34:15.978507+0000","last_undegraded":"2024-09-18T07:34:15.978507+0000","last_fullsized":"2024-09-18T07:34:15.978507+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:41.159206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":169,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.962348+0000","last_change":"2024-09-18T07:32:31.593581+0000","last_active":"2024-09-18T07:34:15.962348+0000","last_peered":"2024-09-18T07:34:15.962348+0000","last_clean":"2024-09-18T07:34:15.962348+0000","last_became_active":"2024-09-18T07:32:30.841210+0000","last_became_peered":"2024-09-18T07:32:30.841210+0000","last_unstale":"2024-09-18T07:34:15.962348+0000","last_undegraded":"2024-09-18T07:34:15.962348+0000","last_fullsized":"2024-09-18T07:34:15.962348+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:31.593520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:31.593520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:50.708269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":105,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.970518+0000","last_change":"2024-09-18T07:33:36.017902+0000","last_active":"2024-09-18T07:34:15.970518+0000","last_peered":"2024-09-18T07:34:15.970518+0000","last_clean":"2024-09-18T07:34:15.970518+0000","last_became_active":"2024-09-18T07:32:30.850481+0000","last_became_peered":"2024-09-18T07:32:30.850481+0000","last_unstale":"2024-09-18T07:34:15.970518+0000","last_undegraded":"2024-09-18T07:34:15.970518+0000","last_fullsized":"2024-09-18T07:34:15.970518+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:36.017602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.017602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.466305+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":159,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.972046+0000","last_change":"2024-09-18T07:33:24.053054+0000","last_active":"2024-09-18T07:34:15.972046+0000","last_peered":"2024-09-18T07:34:15.972046+0000","last_clean":"2024-09-18T07:34:15.972046+0000","last_became_active":"2024-09-18T07:31:02.347537+0000","last_became_peered":"2024-09-18T07:31:02.347537+0000","last_unstale":"2024-09-18T07:34:15.972046+0000","last_undegraded":"2024-09-18T07:34:15.972046+0000","last_fullsized":"2024-09-18T07:34:15.972046+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:24.052908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:24.052908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:39.677071+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":176,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.967274+0000","last_change":"2024-09-18T07:33:33.072569+0000","last_active":"2024-09-18T07:34:15.967274+0000","last_peered":"2024-09-18T07:34:15.967274+0000","last_clean":"2024-09-18T07:34:15.967274+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:34:15.967274+0000","last_undegraded":"2024-09-18T07:34:15.967274+0000","last_fullsized":"2024-09-18T07:34:15.967274+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:33.072487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.072487+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:50.254990+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":185,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.962472+0000","last_change":"2024-09-18T07:33:27.742768+0000","last_active":"2024-09-18T07:34:15.962472+0000","last_peered":"2024-09-18T07:34:15.962472+0000","last_clean":"2024-09-18T07:34:15.962472+0000","last_became_active":"2024-09-18T07:30:58.313556+0000","last_became_peered":"2024-09-18T07:30:58.313556+0000","last_unstale":"2024-09-18T07:34:15.962472+0000","last_undegraded":"2024-09-18T07:34:15.962472+0000","last_fullsized":"2024-09-18T07:34:15.962472+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:33:27.742684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.742684+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:51.970382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":169,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-18T07:34:15.978577+0000","last_change":"2024-09-18T07:33:22.161956+0000","last_active":"2024-09-18T07:34:15.978577+0000","last_peered":"2024-09-18T07:34:15.978577+0000","last_clean":"2024-09-18T07:34:15.978577+0000","last_became_active":"2024-09-18T07:32:30.846635+0000","last_became_peered":"2024-09-18T07:32:30.846635+0000","last_unstale":"2024-09-18T07:34:15.978577+0000","last_undegraded":"2024-09-18T07:34:15.978577+0000","last_fullsized":"2024-09-18T07:34:15.978577+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.161894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.161894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:31.115412+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":3779756056,"num_objects":57675,"num_object_clones":0,"num_object_copies":173025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57675,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105447,"num_write_kb":5219873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7727,"num_bytes_recovered":506200064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14516367360,"data_stored":14516355144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92470,"ondisk_log_size":92470,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":1584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":107,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":236223201324,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462962,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1534856,"kb_used_data":1467576,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92210296,"statfs":{"total":95995035648,"available":94423343104,"internally_reserved":0,"allocated":1502797824,"data_stored":1502424647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724599,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1565856,"kb_used_data":1476356,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":92179296,"statfs":{"total":95995035648,"available":94391599104,"internally_reserved":0,"allocated":1511788544,"data_stored":1511410386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953534,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1893020,"kb_used_data":1785484,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":91852132,"statfs":{"total":95995035648,"available":94056583168,"internally_reserved":0,"allocated":1828335616,"data_stored":1827942419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3116100,"kb_used_data":2954412,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":90629052,"statfs":{"total":95995035648,"available":92804149248,"internally_reserved":0,"allocated":3025317888,"data_stored":3024901206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":165542343},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2169236,"kb_used_data":2065852,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":91575916,"statfs":{"total":95995035648,"available":93773737984,"internally_reserved":0,"allocated":2115432448,"data_stored":2115062174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2182808,"kb_used_data":2075140,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":91562344,"statfs":{"total":95995035648,"available":93759840256,"internally_reserved":0,"allocated":2124943360,"data_stored":2124495326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2491708,"kb_used_data":2370276,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":91253444,"statfs":{"total":95995035648,"available":93443526656,"internally_reserved":0,"allocated":2427162624,"data_stored":2426754064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2425753600,"data_stored":2425749528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2117210112,"data_stored":2117206040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2114650112,"data_stored":2114650112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3024486400,"data_stored":3024486400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1823604736,"data_stored":1823604736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1510998016,"data_stored":1510998016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1496190976,"data_stored":1496186904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3473408,"data_stored":3473408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:34:22.501 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-09-18T07:34:22.502 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 16 2024-09-18T07:34:22.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:22.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:22.627 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:22.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:22.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:22.789 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:22.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:22.890 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:22.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:22.937 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:22.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:23.038 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:23.074 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:23.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:23.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:23.176 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:23.202 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:23.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:23.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:23 smithi203 ceph-mon[32237]: from='client.24958 v1:172.21.15.137:0/3968738237' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:34:23.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:23 smithi203 ceph-mon[32237]: pgmap v156: 158 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 9 unknown, 146 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 1639/170874 objects misplaced (0.959%) 2024-09-18T07:34:23.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:23 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:23.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:23 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3212401796' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:34:23.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:23 smithi203 ceph-mon[32237]: osdmap e139: 8 total, 8 up, 7 in 2024-09-18T07:34:23.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:23.309 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:34:23.309 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:34:23.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.309 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:139395 7.c e139/102 7:35d8f56f:::benchmark_data_smithi137_74180_object34979:head v 139'8667, mlcod=139'8665)", 2024-09-18T07:34:23.310 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:22.231339+0000", 2024-09-18T07:34:23.310 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0081676839999999997, 2024-09-18T07:34:23.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081954360000000004, 2024-09-18T07:34:23.310 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:23.310 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:23.310 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:34:23.310 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:23.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:23.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.231339+0000", 2024-09-18T07:34:23.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:23.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:23.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.231339+0000", 2024-09-18T07:34:23.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:23.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:23.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.231340+0000", 2024-09-18T07:34:23.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.308e-06 2024-09-18T07:34:23.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:23.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.231350+0000", 2024-09-18T07:34:23.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.027e-05 2024-09-18T07:34:23.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:23.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.231360+0000", 2024-09-18T07:34:23.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6350000000000001e-06 2024-09-18T07:34:23.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:23.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.231367+0000", 2024-09-18T07:34:23.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8759999999999999e-06 2024-09-18T07:34:23.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:23.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.231733+0000", 2024-09-18T07:34:23.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000365807 2024-09-18T07:34:23.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:23.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.231749+0000", 2024-09-18T07:34:23.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6481e-05 2024-09-18T07:34:23.313 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:23.313 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:23.313 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:23.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.314 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:139390 7.14 e139/137 7:2d6af277:::benchmark_data_smithi137_74180_object34974:head v 139'8615, mlcod=139'8613)", 2024-09-18T07:34:23.314 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:22.232955+0000", 2024-09-18T07:34:23.314 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0065516690000000004, 2024-09-18T07:34:23.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066359619999999996, 2024-09-18T07:34:23.314 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:23.314 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:23.314 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:34:23.314 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:23.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:23.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.232955+0000", 2024-09-18T07:34:23.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:23.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:23.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.232955+0000", 2024-09-18T07:34:23.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:23.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:23.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.232956+0000", 2024-09-18T07:34:23.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.299e-06 2024-09-18T07:34:23.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:23.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.232962+0000", 2024-09-18T07:34:23.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2609999999999999e-06 2024-09-18T07:34:23.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:23.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.232973+0000", 2024-09-18T07:34:23.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0631e-05 2024-09-18T07:34:23.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:23.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.232986+0000", 2024-09-18T07:34:23.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2855e-05 2024-09-18T07:34:23.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:23.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.233441+0000", 2024-09-18T07:34:23.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00045488899999999999 2024-09-18T07:34:23.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:23.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.233460+0000", 2024-09-18T07:34:23.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9568999999999998e-05 2024-09-18T07:34:23.317 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:23.317 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:23.317 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:23.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.318 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:139396 7.c e139/102 7:37330a81:::benchmark_data_smithi137_74180_object34980:head v 139'8668, mlcod=139'8665)", 2024-09-18T07:34:23.318 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:22.233113+0000", 2024-09-18T07:34:23.318 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0063937359999999997, 2024-09-18T07:34:23.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065297940000000002, 2024-09-18T07:34:23.318 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:23.318 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:23.318 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:34:23.318 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:23.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.318 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:23.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.233113+0000", 2024-09-18T07:34:23.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:23.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:23.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.233113+0000", 2024-09-18T07:34:23.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:23.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:23.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.233114+0000", 2024-09-18T07:34:23.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5600000000000004e-07 2024-09-18T07:34:23.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:23.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.233363+0000", 2024-09-18T07:34:23.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024890999999999997 2024-09-18T07:34:23.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:23.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.233373+0000", 2024-09-18T07:34:23.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0322e-05 2024-09-18T07:34:23.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:23.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.233380+0000", 2024-09-18T07:34:23.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1010000000000003e-06 2024-09-18T07:34:23.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:23.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.236523+0000", 2024-09-18T07:34:23.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031425569999999998 2024-09-18T07:34:23.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:23.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.236607+0000", 2024-09-18T07:34:23.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4160999999999996e-05 2024-09-18T07:34:23.321 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:23.321 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:23.321 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:23.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.322 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:139392 7.14 e139/137 7:2a2983fd:::benchmark_data_smithi137_74180_object34976:head v 139'8616, mlcod=139'8613)", 2024-09-18T07:34:23.322 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:22.236736+0000", 2024-09-18T07:34:23.322 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0027706499999999999, 2024-09-18T07:34:23.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029622509999999999, 2024-09-18T07:34:23.322 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:23.322 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:23.322 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:34:23.322 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:23.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:23.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.236736+0000", 2024-09-18T07:34:23.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:23.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:23.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.236736+0000", 2024-09-18T07:34:23.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:23.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:23.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.236738+0000", 2024-09-18T07:34:23.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6899999999999999e-06 2024-09-18T07:34:23.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:23.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.236745+0000", 2024-09-18T07:34:23.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8949999999999998e-06 2024-09-18T07:34:23.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:23.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.236757+0000", 2024-09-18T07:34:23.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.132e-05 2024-09-18T07:34:23.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:23.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.236765+0000", 2024-09-18T07:34:23.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9750000000000007e-06 2024-09-18T07:34:23.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:23.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.236840+0000", 2024-09-18T07:34:23.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5065e-05 2024-09-18T07:34:23.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:23.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.236862+0000", 2024-09-18T07:34:23.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1712000000000002e-05 2024-09-18T07:34:23.325 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:23.325 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:23.325 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:23.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.326 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.14865.0:139391 7.e e139/81)", 2024-09-18T07:34:23.326 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:22.238639+0000", 2024-09-18T07:34:23.326 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00086747099999999998, 2024-09-18T07:34:23.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001106868, 2024-09-18T07:34:23.326 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:23.326 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:23.326 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:34:23.326 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:23.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:23.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238639+0000", 2024-09-18T07:34:23.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:23.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:23.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238639+0000", 2024-09-18T07:34:23.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:23.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:23.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238640+0000", 2024-09-18T07:34:23.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3480000000000001e-06 2024-09-18T07:34:23.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:23.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238649+0000", 2024-09-18T07:34:23.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.303e-06 2024-09-18T07:34:23.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:23.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238660+0000", 2024-09-18T07:34:23.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0947e-05 2024-09-18T07:34:23.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:23.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238666+0000", 2024-09-18T07:34:23.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7900000000000002e-06 2024-09-18T07:34:23.328 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:23.329 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:23.329 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:23.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.329 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.14865.0:139391 7.e e139/81)", 2024-09-18T07:34:23.329 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:22.238665+0000", 2024-09-18T07:34:23.329 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00084182300000000001, 2024-09-18T07:34:23.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001119681, 2024-09-18T07:34:23.329 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:23.329 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:23.329 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:34:23.329 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:23.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:23.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238665+0000", 2024-09-18T07:34:23.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:23.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:23.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238665+0000", 2024-09-18T07:34:23.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:23.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:23.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238665+0000", 2024-09-18T07:34:23.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7000000000000004e-07 2024-09-18T07:34:23.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:23.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238669+0000", 2024-09-18T07:34:23.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4410000000000002e-06 2024-09-18T07:34:23.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:23.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238675+0000", 2024-09-18T07:34:23.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.251e-06 2024-09-18T07:34:23.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:23.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238680+0000", 2024-09-18T07:34:23.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1440000000000002e-06 2024-09-18T07:34:23.332 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:23.332 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:23.332 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:23.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.332 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:139397 7.2 e139/134 7:4090f0d0:::benchmark_data_smithi137_74180_object34981:head v 139'8505, mlcod=138'8503)", 2024-09-18T07:34:23.332 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:22.238690+0000", 2024-09-18T07:34:23.332 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00081611300000000004, 2024-09-18T07:34:23.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011342170000000001, 2024-09-18T07:34:23.332 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:23.332 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:23.333 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:34:23.333 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:23.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:23.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238690+0000", 2024-09-18T07:34:23.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:23.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:23.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238690+0000", 2024-09-18T07:34:23.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:23.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:23.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238691+0000", 2024-09-18T07:34:23.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4600000000000003e-07 2024-09-18T07:34:23.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:23.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238697+0000", 2024-09-18T07:34:23.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7640000000000002e-06 2024-09-18T07:34:23.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:23.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238703+0000", 2024-09-18T07:34:23.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3720000000000001e-06 2024-09-18T07:34:23.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:23.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238708+0000", 2024-09-18T07:34:23.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.729e-06 2024-09-18T07:34:23.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:23.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238758+0000", 2024-09-18T07:34:23.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9416000000000003e-05 2024-09-18T07:34:23.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:23.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.238776+0000", 2024-09-18T07:34:23.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8312999999999999e-05 2024-09-18T07:34:23.336 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:23.336 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:23.336 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:23.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.336 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:139391 7.e 7:7b98071c:::benchmark_data_smithi137_74180_object34975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-18T07:34:23.336 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:22.224702+0000", 2024-09-18T07:34:23.336 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.014804757, 2024-09-18T07:34:23.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015177235000000001, 2024-09-18T07:34:23.336 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:23.336 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:23.336 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:34:23.336 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:23.336 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:23.337 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:23.337 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 139391 2024-09-18T07:34:23.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.337 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:23.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:23.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.224702+0000", 2024-09-18T07:34:23.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:23.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:23.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.224702+0000", 2024-09-18T07:34:23.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:23.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:23.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.224703+0000", 2024-09-18T07:34:23.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1999999999999999e-07 2024-09-18T07:34:23.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:23.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.224706+0000", 2024-09-18T07:34:23.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1499999999999999e-06 2024-09-18T07:34:23.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:23.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.224712+0000", 2024-09-18T07:34:23.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0140000000000004e-06 2024-09-18T07:34:23.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:23.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.224717+0000", 2024-09-18T07:34:23.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.699e-06 2024-09-18T07:34:23.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:23.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.228705+0000", 2024-09-18T07:34:23.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039878960000000003 2024-09-18T07:34:23.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:23.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.228823+0000", 2024-09-18T07:34:23.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117843 2024-09-18T07:34:23.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:23.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:23.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:34:23.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:22.228936+0000", 2024-09-18T07:34:23.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011302999999999999 2024-09-18T07:34:23.340 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:23.340 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:23.340 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:23.340 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:23.341 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:34:23.341 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 8 2024-09-18T07:34:23.341 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:34:23.341 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:23 smithi137 ceph-mon[29408]: from='client.24958 v1:172.21.15.137:0/3968738237' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:34:23.341 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:23 smithi137 ceph-mon[29408]: pgmap v156: 158 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 9 unknown, 146 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 1639/170874 objects misplaced (0.959%) 2024-09-18T07:34:23.341 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:23 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:23.341 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:23 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3212401796' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:34:23.341 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:23 smithi137 ceph-mon[29408]: osdmap e139: 8 total, 8 up, 7 in 2024-09-18T07:34:23.342 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:23 smithi137 ceph-mon[39175]: from='client.24958 v1:172.21.15.137:0/3968738237' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:34:23.342 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:23 smithi137 ceph-mon[39175]: pgmap v156: 158 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 9 unknown, 146 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 1639/170874 objects misplaced (0.959%) 2024-09-18T07:34:23.342 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:23 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:23.342 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:23 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3212401796' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:34:23.342 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:23 smithi137 ceph-mon[39175]: osdmap e139: 8 total, 8 up, 7 in 2024-09-18T07:34:23.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:23.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:23.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:23.454 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:23.479 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:23.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:23.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:23.580 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:23.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:23.619 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:23.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:23.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:23.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:23.761 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:23.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:23.862 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:23.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:23.883 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:23.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:23.984 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:24.020 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:24.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:24.122 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:24.143 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:24.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:24.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:24.245 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:24.282 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:24.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:24.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:24.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:24.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:24.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:24.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:24 smithi137 ceph-mon[29408]: pgmap v158: 158 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 787 B/s rd, 220 MiB/s wr, 5.12k op/s; 44259/210951 objects misplaced (20.981%); 29 MiB/s, 468 objects/s recovering 2024-09-18T07:34:24.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:24 smithi137 ceph-mon[29408]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:34:24.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:24 smithi137 ceph-mon[39175]: pgmap v158: 158 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 787 B/s rd, 220 MiB/s wr, 5.12k op/s; 44259/210951 objects misplaced (20.981%); 29 MiB/s, 468 objects/s recovering 2024-09-18T07:34:24.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:24 smithi137 ceph-mon[39175]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:34:24.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:24 smithi203 ceph-mon[32237]: pgmap v158: 158 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 787 B/s rd, 220 MiB/s wr, 5.12k op/s; 44259/210951 objects misplaced (20.981%); 29 MiB/s, 468 objects/s recovering 2024-09-18T07:34:24.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:24 smithi203 ceph-mon[32237]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:34:24.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:24.547 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:24.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:24.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:24.649 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:24.671 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:24.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:24.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:24.773 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:24.801 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:24.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:24.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:24.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:24.943 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:24.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:25.044 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:25.071 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:25.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:25.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:25.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:25.179 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:34:25.199 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:25.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:25.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:25.301 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:25.326 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:25.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:25.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:25.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:25 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:34:25.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:25 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:34:25.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:25 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:34:25.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:25 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:34:25.428 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:25.453 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:25.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:25.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:25.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:25 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:34:25.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:25 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:34:25.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:25.578 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:25.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:25.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:25.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:25.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:25.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:25.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:25.836 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:25.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:25.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:25.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:25.966 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:25.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:26.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:26.093 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:26.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:26.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T07:34:26.195 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:26.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:26.282 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:26.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:26.294 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:26.380 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:26.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:26.393 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:26.423 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:26 smithi137 ceph-mon[39175]: osdmap e140: 8 total, 8 up, 7 in 2024-09-18T07:34:26.423 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:26 smithi137 ceph-mon[39175]: pgmap v160: 158 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 682 B/s rd, 190 MiB/s wr, 4.44k op/s; 44259/210951 objects misplaced (20.981%); 25 MiB/s, 406 objects/s recovering 2024-09-18T07:34:26.423 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:26.423 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:34:26.423 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:26 smithi137 ceph-mon[29408]: osdmap e140: 8 total, 8 up, 7 in 2024-09-18T07:34:26.423 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:26 smithi137 ceph-mon[29408]: pgmap v160: 158 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 682 B/s rd, 190 MiB/s wr, 4.44k op/s; 44259/210951 objects misplaced (20.981%); 25 MiB/s, 406 objects/s recovering 2024-09-18T07:34:26.424 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:26.424 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:34:26.452 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:26.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:26.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:26.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:26 smithi203 ceph-mon[32237]: osdmap e140: 8 total, 8 up, 7 in 2024-09-18T07:34:26.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:26 smithi203 ceph-mon[32237]: pgmap v160: 158 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 682 B/s rd, 190 MiB/s wr, 4.44k op/s; 44259/210951 objects misplaced (20.981%); 25 MiB/s, 406 objects/s recovering 2024-09-18T07:34:26.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:26.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:34:26.554 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:26.580 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:26.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:26.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:26.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:26.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:26.758 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:26.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:26.859 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:26.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:26.904 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:26.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:27.005 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:27.031 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:27.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:27.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:27.133 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:27.158 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:27.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:27.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:27.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:27.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:27.324 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:27.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:27.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:27 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:27.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:34:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:34:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:34:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:27 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:27.425 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:27.449 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:27.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:27.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:27.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:27 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:27.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:34:27 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:34:27.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:27.595 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:27.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:27.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:27.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:27.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:27.752 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:27.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:27.854 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:27.905 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:27.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:27.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:27.938 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:34:27.939 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:34:27.979 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:34:27.979 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:34:28.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:28.026 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:34:28.027 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:34:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:28.052 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:28.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:28.069 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:34:28.069 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:34:28.110 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:34:28.110 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:34:28.129 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:28.130 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:34:28.153 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:28.155 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:34:28.156 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:34:28.194 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:34:28.195 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:34:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:28.221 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:28.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:28.244 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:34:28.245 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:34:28.245 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:28 smithi137 ceph-mon[29408]: pgmap v161: 158 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 178 MiB/s wr, 4.81k op/s; 38602/195288 objects misplaced (19.767%); 35 MiB/s, 566 objects/s recovering 2024-09-18T07:34:28.245 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:28 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3509842010' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:28.245 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:28 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2430221990' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:28.245 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:28 smithi137 ceph-mon[29408]: from='client.24967 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:28.246 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:28 smithi137 ceph-mon[39175]: pgmap v161: 158 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 178 MiB/s wr, 4.81k op/s; 38602/195288 objects misplaced (19.767%); 35 MiB/s, 566 objects/s recovering 2024-09-18T07:34:28.246 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:28 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3509842010' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:28.246 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:28 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2430221990' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:28.246 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:28 smithi137 ceph-mon[39175]: from='client.24967 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:28.322 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:28.345 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:28.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:28.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:28.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:28.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:28 smithi203 ceph-mon[32237]: pgmap v161: 158 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 178 MiB/s wr, 4.81k op/s; 38602/195288 objects misplaced (19.767%); 35 MiB/s, 566 objects/s recovering 2024-09-18T07:34:28.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:28 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3509842010' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:28.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:28 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2430221990' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:28.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:28 smithi203 ceph-mon[32237]: from='client.24967 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:28.482 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:28.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:28.584 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:28.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:28.629 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:28.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:28.730 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:28.756 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:28.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:28.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:28.858 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:28.893 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:28.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:28.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:28.995 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:29.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:29.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:29.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:29.131 INFO:teuthology.orchestra.run.smithi137.stderr:set pool 7 pg_num to 16 2024-09-18T07:34:29.138 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:29.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:29.164 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:29.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:29.265 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:29.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:29.291 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:29.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:29.392 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:29.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:29 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:34:29.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:29 smithi137 ceph-mon[29408]: from='client.24967 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:34:29.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:29 smithi137 ceph-mon[29408]: osdmap e141: 8 total, 8 up, 7 in 2024-09-18T07:34:29.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:29 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3013926953' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"} : dispatch 2024-09-18T07:34:29.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:29 smithi137 ceph-mon[29408]: from='client.24979 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"} : dispatch 2024-09-18T07:34:29.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:29 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:34:29.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:29 smithi137 ceph-mon[39175]: from='client.24967 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:34:29.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:29 smithi137 ceph-mon[39175]: osdmap e141: 8 total, 8 up, 7 in 2024-09-18T07:34:29.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:29 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3013926953' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"} : dispatch 2024-09-18T07:34:29.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:29 smithi137 ceph-mon[39175]: from='client.24979 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"} : dispatch 2024-09-18T07:34:29.430 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:29.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:29.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:29.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:29 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:34:29.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:29 smithi203 ceph-mon[32237]: from='client.24967 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:34:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:29 smithi203 ceph-mon[32237]: osdmap e141: 8 total, 8 up, 7 in 2024-09-18T07:34:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:29 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3013926953' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"} : dispatch 2024-09-18T07:34:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:29 smithi203 ceph-mon[32237]: from='client.24979 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"} : dispatch 2024-09-18T07:34:29.531 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:29.556 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:29.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:29.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:29.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:29.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:29.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:29.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:29.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:29.828 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:29.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:29.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:29.930 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:29.975 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:29.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:29.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:30.077 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:30.119 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:30.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:30.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:30.244 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:30.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:30.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:30.258 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[29408]: from='client.24979 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"}]': finished 2024-09-18T07:34:30.258 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[29408]: osdmap e142: 8 total, 8 up, 7 in 2024-09-18T07:34:30.258 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[29408]: 7.3 scrub starts 2024-09-18T07:34:30.258 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[29408]: pgmap v164: 158 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 869 op/s; 38602/195288 objects misplaced (19.767%); 15 MiB/s, 247 objects/s recovering 2024-09-18T07:34:30.258 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"} : dispatch 2024-09-18T07:34:30.258 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"} : dispatch 2024-09-18T07:34:30.258 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[29408]: 7.0 scrub starts 2024-09-18T07:34:30.259 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[29408]: 7.3 scrub ok 2024-09-18T07:34:30.259 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[29408]: 7.5 scrub starts 2024-09-18T07:34:30.259 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[29408]: 7.0 scrub ok 2024-09-18T07:34:30.259 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[39175]: from='client.24979 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"}]': finished 2024-09-18T07:34:30.259 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[39175]: osdmap e142: 8 total, 8 up, 7 in 2024-09-18T07:34:30.259 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[39175]: 7.3 scrub starts 2024-09-18T07:34:30.259 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[39175]: pgmap v164: 158 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 869 op/s; 38602/195288 objects misplaced (19.767%); 15 MiB/s, 247 objects/s recovering 2024-09-18T07:34:30.259 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"} : dispatch 2024-09-18T07:34:30.259 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"} : dispatch 2024-09-18T07:34:30.259 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[39175]: 7.0 scrub starts 2024-09-18T07:34:30.260 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[39175]: 7.3 scrub ok 2024-09-18T07:34:30.260 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[39175]: 7.5 scrub starts 2024-09-18T07:34:30.260 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:30 smithi137 ceph-mon[39175]: 7.0 scrub ok 2024-09-18T07:34:30.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:30.369 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:30.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:30.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:30.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:30 smithi203 ceph-mon[32237]: from='client.24979 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"}]': finished 2024-09-18T07:34:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:30 smithi203 ceph-mon[32237]: osdmap e142: 8 total, 8 up, 7 in 2024-09-18T07:34:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:30 smithi203 ceph-mon[32237]: 7.3 scrub starts 2024-09-18T07:34:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:30 smithi203 ceph-mon[32237]: pgmap v164: 158 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 869 op/s; 38602/195288 objects misplaced (19.767%); 15 MiB/s, 247 objects/s recovering 2024-09-18T07:34:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:30 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"} : dispatch 2024-09-18T07:34:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:30 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"} : dispatch 2024-09-18T07:34:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:30 smithi203 ceph-mon[32237]: 7.0 scrub starts 2024-09-18T07:34:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:30 smithi203 ceph-mon[32237]: 7.3 scrub ok 2024-09-18T07:34:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:30 smithi203 ceph-mon[32237]: 7.5 scrub starts 2024-09-18T07:34:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:30 smithi203 ceph-mon[32237]: 7.0 scrub ok 2024-09-18T07:34:30.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:30.493 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:30.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:30.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:30.596 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:30.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:30.618 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:30.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:30.719 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:30.741 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:30.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:30.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:30.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:30.866 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:30.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:30.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:30.968 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:30.991 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:30.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:30.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:31.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:31.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:34:31.117 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:31.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:31.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:31.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:31.239 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:31.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:31.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:31.341 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:31.363 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:31.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:31.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:31.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[29408]: 7.9 scrub starts 2024-09-18T07:34:31.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[29408]: 7.7 scrub starts 2024-09-18T07:34:31.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[29408]: 7.1 scrub starts 2024-09-18T07:34:31.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[29408]: 7.7 scrub ok 2024-09-18T07:34:31.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"}]': finished 2024-09-18T07:34:31.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[29408]: osdmap e143: 8 total, 8 up, 7 in 2024-09-18T07:34:31.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[29408]: 7.1 scrub ok 2024-09-18T07:34:31.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[29408]: 7.2 scrub starts 2024-09-18T07:34:31.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[29408]: 7.5 scrub ok 2024-09-18T07:34:31.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[29408]: 7.2 scrub ok 2024-09-18T07:34:31.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[29408]: osdmap e144: 8 total, 8 up, 7 in 2024-09-18T07:34:31.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[39175]: 7.9 scrub starts 2024-09-18T07:34:31.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[39175]: 7.7 scrub starts 2024-09-18T07:34:31.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[39175]: 7.1 scrub starts 2024-09-18T07:34:31.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[39175]: 7.7 scrub ok 2024-09-18T07:34:31.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"}]': finished 2024-09-18T07:34:31.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[39175]: osdmap e143: 8 total, 8 up, 7 in 2024-09-18T07:34:31.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[39175]: 7.1 scrub ok 2024-09-18T07:34:31.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[39175]: 7.2 scrub starts 2024-09-18T07:34:31.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[39175]: 7.5 scrub ok 2024-09-18T07:34:31.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[39175]: 7.2 scrub ok 2024-09-18T07:34:31.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:31 smithi137 ceph-mon[39175]: osdmap e144: 8 total, 8 up, 7 in 2024-09-18T07:34:31.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:31 smithi203 ceph-mon[32237]: 7.9 scrub starts 2024-09-18T07:34:31.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:31 smithi203 ceph-mon[32237]: 7.7 scrub starts 2024-09-18T07:34:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:31 smithi203 ceph-mon[32237]: 7.1 scrub starts 2024-09-18T07:34:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:31 smithi203 ceph-mon[32237]: 7.7 scrub ok 2024-09-18T07:34:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:31 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"}]': finished 2024-09-18T07:34:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:31 smithi203 ceph-mon[32237]: osdmap e143: 8 total, 8 up, 7 in 2024-09-18T07:34:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:31 smithi203 ceph-mon[32237]: 7.1 scrub ok 2024-09-18T07:34:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:31 smithi203 ceph-mon[32237]: 7.2 scrub starts 2024-09-18T07:34:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:31 smithi203 ceph-mon[32237]: 7.5 scrub ok 2024-09-18T07:34:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:31 smithi203 ceph-mon[32237]: 7.2 scrub ok 2024-09-18T07:34:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:31 smithi203 ceph-mon[32237]: osdmap e144: 8 total, 8 up, 7 in 2024-09-18T07:34:31.465 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:31.486 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:31.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:31.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:31.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:31.609 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:31.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:31.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:31.696 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:34:31.696 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:34:31.711 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:31.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:31.734 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:31.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:31.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:31.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:34:31.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:31.858 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:31.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:31.959 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:31.980 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:31.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:31.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:32.081 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:32.103 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:32.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:32.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:32.204 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:32.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:32.226 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:32.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:32.327 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:32.353 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:32.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:32.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:32 smithi137 ceph-mon[29408]: 7.9 scrub ok 2024-09-18T07:34:32.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:32 smithi137 ceph-mon[29408]: 7.4 scrub starts 2024-09-18T07:34:32.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:32 smithi137 ceph-mon[29408]: 7.4 scrub ok 2024-09-18T07:34:32.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:32 smithi137 ceph-mon[29408]: 7.8 scrub starts 2024-09-18T07:34:32.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:32 smithi137 ceph-mon[29408]: 7.8 scrub ok 2024-09-18T07:34:32.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:32 smithi137 ceph-mon[29408]: pgmap v167: 158 pgs: 3 peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 148 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 30016/191700 objects misplaced (15.658%); 9.0 MiB/s, 144 objects/s recovering 2024-09-18T07:34:32.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:32 smithi137 ceph-mon[29408]: osdmap e145: 8 total, 8 up, 7 in 2024-09-18T07:34:32.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:32 smithi137 ceph-mon[39175]: 7.9 scrub ok 2024-09-18T07:34:32.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:32 smithi137 ceph-mon[39175]: 7.4 scrub starts 2024-09-18T07:34:32.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:32 smithi137 ceph-mon[39175]: 7.4 scrub ok 2024-09-18T07:34:32.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:32 smithi137 ceph-mon[39175]: 7.8 scrub starts 2024-09-18T07:34:32.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:32 smithi137 ceph-mon[39175]: 7.8 scrub ok 2024-09-18T07:34:32.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:32 smithi137 ceph-mon[39175]: pgmap v167: 158 pgs: 3 peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 148 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 30016/191700 objects misplaced (15.658%); 9.0 MiB/s, 144 objects/s recovering 2024-09-18T07:34:32.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:32 smithi137 ceph-mon[39175]: osdmap e145: 8 total, 8 up, 7 in 2024-09-18T07:34:32.455 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:32.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:32 smithi203 ceph-mon[32237]: 7.9 scrub ok 2024-09-18T07:34:32.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:32 smithi203 ceph-mon[32237]: 7.4 scrub starts 2024-09-18T07:34:32.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:32 smithi203 ceph-mon[32237]: 7.4 scrub ok 2024-09-18T07:34:32.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:32 smithi203 ceph-mon[32237]: 7.8 scrub starts 2024-09-18T07:34:32.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:32 smithi203 ceph-mon[32237]: 7.8 scrub ok 2024-09-18T07:34:32.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:32 smithi203 ceph-mon[32237]: pgmap v167: 158 pgs: 3 peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 148 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 30016/191700 objects misplaced (15.658%); 9.0 MiB/s, 144 objects/s recovering 2024-09-18T07:34:32.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:32 smithi203 ceph-mon[32237]: osdmap e145: 8 total, 8 up, 7 in 2024-09-18T07:34:32.475 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:32.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:32.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:32.577 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:32.599 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:32.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:32.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:32.701 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:32.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:32.729 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:32.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:32.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:32.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:32.870 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:32.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:32.971 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:33.006 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:33.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:33.108 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:33.155 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:33.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:33.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:33.257 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:33.283 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:33.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:33.385 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:33.409 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:33.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:33.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:33.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[29408]: 7.6 scrub starts 2024-09-18T07:34:33.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[29408]: 7.6 scrub ok 2024-09-18T07:34:33.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[29408]: 3.13 scrub starts 2024-09-18T07:34:33.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[29408]: 3.13 scrub ok 2024-09-18T07:34:33.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[29408]: 4.4 scrub starts 2024-09-18T07:34:33.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[29408]: 4.4 scrub ok 2024-09-18T07:34:33.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[29408]: 5.a scrub starts 2024-09-18T07:34:33.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[29408]: 5.a scrub ok 2024-09-18T07:34:33.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[29408]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:34:33.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[29408]: 7.12 scrub starts 2024-09-18T07:34:33.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[29408]: 7.10 scrub starts 2024-09-18T07:34:33.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[29408]: 7.10 scrub ok 2024-09-18T07:34:33.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[39175]: 7.6 scrub starts 2024-09-18T07:34:33.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[39175]: 7.6 scrub ok 2024-09-18T07:34:33.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[39175]: 3.13 scrub starts 2024-09-18T07:34:33.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[39175]: 3.13 scrub ok 2024-09-18T07:34:33.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[39175]: 4.4 scrub starts 2024-09-18T07:34:33.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[39175]: 4.4 scrub ok 2024-09-18T07:34:33.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[39175]: 5.a scrub starts 2024-09-18T07:34:33.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[39175]: 5.a scrub ok 2024-09-18T07:34:33.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[39175]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:34:33.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[39175]: 7.12 scrub starts 2024-09-18T07:34:33.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[39175]: 7.10 scrub starts 2024-09-18T07:34:33.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:33 smithi137 ceph-mon[39175]: 7.10 scrub ok 2024-09-18T07:34:33.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:33 smithi203 ceph-mon[32237]: 7.6 scrub starts 2024-09-18T07:34:33.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:33 smithi203 ceph-mon[32237]: 7.6 scrub ok 2024-09-18T07:34:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:33 smithi203 ceph-mon[32237]: 3.13 scrub starts 2024-09-18T07:34:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:33 smithi203 ceph-mon[32237]: 3.13 scrub ok 2024-09-18T07:34:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:33 smithi203 ceph-mon[32237]: 4.4 scrub starts 2024-09-18T07:34:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:33 smithi203 ceph-mon[32237]: 4.4 scrub ok 2024-09-18T07:34:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:33 smithi203 ceph-mon[32237]: 5.a scrub starts 2024-09-18T07:34:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:33 smithi203 ceph-mon[32237]: 5.a scrub ok 2024-09-18T07:34:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:33 smithi203 ceph-mon[32237]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:34:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:33 smithi203 ceph-mon[32237]: 7.12 scrub starts 2024-09-18T07:34:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:33 smithi203 ceph-mon[32237]: 7.10 scrub starts 2024-09-18T07:34:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:33 smithi203 ceph-mon[32237]: 7.10 scrub ok 2024-09-18T07:34:33.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:33.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:33.557 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:33.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:33.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:33.695 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:33.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:33.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:33.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:33.834 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:33.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:33.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:33.936 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:33.959 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:33.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:33.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:34.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:34.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:34.101 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:34.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:34.202 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:34.228 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:34.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:34.330 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:34.375 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:34.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:34.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:34.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:34 smithi203 ceph-mon[32237]: 7.17 scrub starts 2024-09-18T07:34:34.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:34 smithi203 ceph-mon[32237]: 7.19 scrub starts 2024-09-18T07:34:34.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:34 smithi203 ceph-mon[32237]: 7.11 scrub starts 2024-09-18T07:34:34.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:34 smithi203 ceph-mon[32237]: 7.12 scrub ok 2024-09-18T07:34:34.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:34 smithi203 ceph-mon[32237]: 7.15 scrub starts 2024-09-18T07:34:34.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:34 smithi203 ceph-mon[32237]: 7.17 scrub ok 2024-09-18T07:34:34.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:34 smithi203 ceph-mon[32237]: 7.13 scrub starts 2024-09-18T07:34:34.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:34 smithi203 ceph-mon[32237]: 7.11 scrub ok 2024-09-18T07:34:34.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:34 smithi203 ceph-mon[32237]: pgmap v169: 158 pgs: 1 active+clean+scrubbing, 8 peering, 149 active+clean; 3.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:34:34.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:34 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"} : dispatch 2024-09-18T07:34:34.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:34 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"} : dispatch 2024-09-18T07:34:34.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:34 smithi203 ceph-mon[32237]: 7.15 scrub ok 2024-09-18T07:34:34.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:34 smithi203 ceph-mon[32237]: 7.13 scrub ok 2024-09-18T07:34:34.478 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:34.499 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:34.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:34.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:34.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:34.622 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:34.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:34.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:34.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[29408]: 7.17 scrub starts 2024-09-18T07:34:34.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[29408]: 7.19 scrub starts 2024-09-18T07:34:34.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[29408]: 7.11 scrub starts 2024-09-18T07:34:34.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[29408]: 7.12 scrub ok 2024-09-18T07:34:34.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[29408]: 7.15 scrub starts 2024-09-18T07:34:34.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[29408]: 7.17 scrub ok 2024-09-18T07:34:34.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[29408]: 7.13 scrub starts 2024-09-18T07:34:34.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[29408]: 7.11 scrub ok 2024-09-18T07:34:34.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[29408]: pgmap v169: 158 pgs: 1 active+clean+scrubbing, 8 peering, 149 active+clean; 3.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:34:34.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"} : dispatch 2024-09-18T07:34:34.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"} : dispatch 2024-09-18T07:34:34.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[29408]: 7.15 scrub ok 2024-09-18T07:34:34.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[29408]: 7.13 scrub ok 2024-09-18T07:34:34.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[39175]: 7.17 scrub starts 2024-09-18T07:34:34.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[39175]: 7.19 scrub starts 2024-09-18T07:34:34.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[39175]: 7.11 scrub starts 2024-09-18T07:34:34.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[39175]: 7.12 scrub ok 2024-09-18T07:34:34.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[39175]: 7.15 scrub starts 2024-09-18T07:34:34.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[39175]: 7.17 scrub ok 2024-09-18T07:34:34.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[39175]: 7.13 scrub starts 2024-09-18T07:34:34.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[39175]: 7.11 scrub ok 2024-09-18T07:34:34.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[39175]: pgmap v169: 158 pgs: 1 active+clean+scrubbing, 8 peering, 149 active+clean; 3.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:34:34.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"} : dispatch 2024-09-18T07:34:34.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"} : dispatch 2024-09-18T07:34:34.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[39175]: 7.15 scrub ok 2024-09-18T07:34:34.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:34 smithi137 ceph-mon[39175]: 7.13 scrub ok 2024-09-18T07:34:34.724 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:34.747 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:34.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:34.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:34.848 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:34.870 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:34.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:34.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:34.972 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:34.993 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:34.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:34.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:35.095 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:35.116 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:35.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:35.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:35.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:35.243 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:35.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:35.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:35.297 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:34:35.297 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:34:35.297 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-18T07:34:35.298 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-18T07:34:35.345 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:35.367 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:35.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:35.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:35.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:35 smithi203 ceph-mon[32237]: 7.19 scrub ok 2024-09-18T07:34:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:35 smithi203 ceph-mon[32237]: 3.a scrub starts 2024-09-18T07:34:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:35 smithi203 ceph-mon[32237]: 3.a scrub ok 2024-09-18T07:34:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:35 smithi203 ceph-mon[32237]: 7.16 scrub starts 2024-09-18T07:34:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:35 smithi203 ceph-mon[32237]: 7.18 scrub starts 2024-09-18T07:34:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:35 smithi203 ceph-mon[32237]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull) 2024-09-18T07:34:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:35 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:34:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:35 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"}]': finished 2024-09-18T07:34:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:35 smithi203 ceph-mon[32237]: osdmap e146: 8 total, 8 up, 7 in 2024-09-18T07:34:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:35 smithi203 ceph-mon[32237]: 3.15 scrub starts 2024-09-18T07:34:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:35 smithi203 ceph-mon[32237]: 3.15 scrub ok 2024-09-18T07:34:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:35 smithi203 ceph-mon[32237]: 7.18 scrub ok 2024-09-18T07:34:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:35 smithi203 ceph-mon[32237]: osdmap e147: 8 total, 8 up, 7 in 2024-09-18T07:34:35.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:35.491 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:35.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:35.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:35.592 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:35.614 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:35.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:35.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:35.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[29408]: 7.19 scrub ok 2024-09-18T07:34:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[29408]: 3.a scrub starts 2024-09-18T07:34:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[29408]: 3.a scrub ok 2024-09-18T07:34:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[29408]: 7.16 scrub starts 2024-09-18T07:34:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[29408]: 7.18 scrub starts 2024-09-18T07:34:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[29408]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull) 2024-09-18T07:34:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:34:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"}]': finished 2024-09-18T07:34:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[29408]: osdmap e146: 8 total, 8 up, 7 in 2024-09-18T07:34:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[29408]: 3.15 scrub starts 2024-09-18T07:34:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[29408]: 3.15 scrub ok 2024-09-18T07:34:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[29408]: 7.18 scrub ok 2024-09-18T07:34:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[29408]: osdmap e147: 8 total, 8 up, 7 in 2024-09-18T07:34:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[39175]: 7.19 scrub ok 2024-09-18T07:34:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[39175]: 3.a scrub starts 2024-09-18T07:34:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[39175]: 3.a scrub ok 2024-09-18T07:34:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[39175]: 7.16 scrub starts 2024-09-18T07:34:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[39175]: 7.18 scrub starts 2024-09-18T07:34:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[39175]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull) 2024-09-18T07:34:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:34:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"}]': finished 2024-09-18T07:34:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[39175]: osdmap e146: 8 total, 8 up, 7 in 2024-09-18T07:34:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[39175]: 3.15 scrub starts 2024-09-18T07:34:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[39175]: 3.15 scrub ok 2024-09-18T07:34:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[39175]: 7.18 scrub ok 2024-09-18T07:34:35.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:35 smithi137 ceph-mon[39175]: osdmap e147: 8 total, 8 up, 7 in 2024-09-18T07:34:35.716 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:35.738 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:35.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:35.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:35.840 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:35.861 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:35.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:35.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:35.963 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:35.985 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:35.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:36.086 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:36.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:36.108 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:36.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:36.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:36.230 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:36.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:36.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:36.332 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:36.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:36.359 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:36.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:36.460 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:36.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:36 smithi203 ceph-mon[32237]: 7.16 scrub ok 2024-09-18T07:34:36.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:36 smithi203 ceph-mon[32237]: 7.14 scrub starts 2024-09-18T07:34:36.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:36 smithi203 ceph-mon[32237]: 7.14 scrub ok 2024-09-18T07:34:36.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:36 smithi203 ceph-mon[32237]: pgmap v172: 158 pgs: 1 active+clean+scrubbing, 8 peering, 149 active+clean; 3.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:34:36.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:36 smithi203 ceph-mon[32237]: osdmap e148: 8 total, 8 up, 7 in 2024-09-18T07:34:36.482 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:36.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:36.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:36.584 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:36.605 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:36.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:36.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:36 smithi137 ceph-mon[29408]: 7.16 scrub ok 2024-09-18T07:34:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:36 smithi137 ceph-mon[29408]: 7.14 scrub starts 2024-09-18T07:34:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:36 smithi137 ceph-mon[29408]: 7.14 scrub ok 2024-09-18T07:34:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:36 smithi137 ceph-mon[29408]: pgmap v172: 158 pgs: 1 active+clean+scrubbing, 8 peering, 149 active+clean; 3.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:34:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:36 smithi137 ceph-mon[29408]: osdmap e148: 8 total, 8 up, 7 in 2024-09-18T07:34:36.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:36 smithi137 ceph-mon[39175]: 7.16 scrub ok 2024-09-18T07:34:36.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:36 smithi137 ceph-mon[39175]: 7.14 scrub starts 2024-09-18T07:34:36.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:36 smithi137 ceph-mon[39175]: 7.14 scrub ok 2024-09-18T07:34:36.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:36 smithi137 ceph-mon[39175]: pgmap v172: 158 pgs: 1 active+clean+scrubbing, 8 peering, 149 active+clean; 3.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:34:36.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:36 smithi137 ceph-mon[39175]: osdmap e148: 8 total, 8 up, 7 in 2024-09-18T07:34:36.707 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:36.732 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:36.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:36.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:36.871 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:36.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:36.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:36.973 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:37.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:37.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:37.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:37.109 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:37.148 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:37.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:37.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:37.250 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:37.272 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:37.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:37.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:37.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:37.399 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:37.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:37.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:37.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:34:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:34:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:34:37.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:37 smithi137 ceph-mon[29408]: 3.12 scrub starts 2024-09-18T07:34:37.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:37 smithi137 ceph-mon[29408]: 3.12 scrub ok 2024-09-18T07:34:37.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:37 smithi137 ceph-mon[29408]: osdmap e149: 8 total, 8 up, 7 in 2024-09-18T07:34:37.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:37 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:37.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:37 smithi137 ceph-mon[39175]: 3.12 scrub starts 2024-09-18T07:34:37.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:37 smithi137 ceph-mon[39175]: 3.12 scrub ok 2024-09-18T07:34:37.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:37 smithi137 ceph-mon[39175]: osdmap e149: 8 total, 8 up, 7 in 2024-09-18T07:34:37.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:37 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:37.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:37 smithi203 ceph-mon[32237]: 3.12 scrub starts 2024-09-18T07:34:37.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:37 smithi203 ceph-mon[32237]: 3.12 scrub ok 2024-09-18T07:34:37.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:37 smithi203 ceph-mon[32237]: osdmap e149: 8 total, 8 up, 7 in 2024-09-18T07:34:37.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:37 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:37.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:34:37 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:34:37.500 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:37.542 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:37.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:37.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:37.644 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:37.662 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T07:34:37.666 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:37.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:37.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:37.729 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:37.758 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:37.768 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:37.773 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:37.790 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:37.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:37.892 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:37.916 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:37.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:37.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:38.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:38.079 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:38.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:38.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:38.181 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:38.206 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:38.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:38.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:38.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:38.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:38.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:38.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:38 smithi137 ceph-mon[39175]: 5.4 scrub starts 2024-09-18T07:34:38.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:38 smithi137 ceph-mon[39175]: 5.4 scrub ok 2024-09-18T07:34:38.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:38 smithi137 ceph-mon[39175]: 4.f scrub starts 2024-09-18T07:34:38.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:38 smithi137 ceph-mon[39175]: 4.f scrub ok 2024-09-18T07:34:38.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:38 smithi137 ceph-mon[39175]: pgmap v175: 157 pgs: 1 peering, 156 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:34:38.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:38 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"} : dispatch 2024-09-18T07:34:38.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:38 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"} : dispatch 2024-09-18T07:34:38.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:38 smithi137 ceph-mon[29408]: 5.4 scrub starts 2024-09-18T07:34:38.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:38 smithi137 ceph-mon[29408]: 5.4 scrub ok 2024-09-18T07:34:38.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:38 smithi137 ceph-mon[29408]: 4.f scrub starts 2024-09-18T07:34:38.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:38 smithi137 ceph-mon[29408]: 4.f scrub ok 2024-09-18T07:34:38.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:38 smithi137 ceph-mon[29408]: pgmap v175: 157 pgs: 1 peering, 156 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:34:38.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:38 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"} : dispatch 2024-09-18T07:34:38.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:38 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"} : dispatch 2024-09-18T07:34:38.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:38.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:38 smithi203 ceph-mon[32237]: 5.4 scrub starts 2024-09-18T07:34:38.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:38 smithi203 ceph-mon[32237]: 5.4 scrub ok 2024-09-18T07:34:38.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:38 smithi203 ceph-mon[32237]: 4.f scrub starts 2024-09-18T07:34:38.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:38 smithi203 ceph-mon[32237]: 4.f scrub ok 2024-09-18T07:34:38.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:38 smithi203 ceph-mon[32237]: pgmap v175: 157 pgs: 1 peering, 156 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:34:38.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:38 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"} : dispatch 2024-09-18T07:34:38.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:38 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"} : dispatch 2024-09-18T07:34:38.490 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:38.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:38.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:38.580 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:34:38.580 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:34:38.592 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:38.597 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:34:38.598 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:34:38.613 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:34:38.614 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:34:38.615 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:38.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:38.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:38.629 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:34:38.630 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:34:38.646 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:34:38.646 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:34:38.662 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:34:38.663 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:34:38.681 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:34:38.681 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:34:38.698 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:34:38.698 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:34:38.716 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:38.738 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:38.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:38.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:38.839 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:38.861 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:38.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:38.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:38.942 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:34:38.942 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:34:38.942 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 150, 2024-09-18T07:34:38.942 INFO:teuthology.orchestra.run.smithi137.stdout: "fsid": "f7ef099c-758e-11ef-bceb-c7b262605968", 2024-09-18T07:34:38.942 INFO:teuthology.orchestra.run.smithi137.stdout: "created": "2024-09-18T07:25:19.662367+0000", 2024-09-18T07:34:38.942 INFO:teuthology.orchestra.run.smithi137.stdout: "modified": "2024-09-18T07:34:38.179526+0000", 2024-09-18T07:34:38.942 INFO:teuthology.orchestra.run.smithi137.stdout: "last_up_change": "2024-09-18T07:30:44.708496+0000", 2024-09-18T07:34:38.942 INFO:teuthology.orchestra.run.smithi137.stdout: "last_in_change": "2024-09-18T07:32:29.796199+0000", 2024-09-18T07:34:38.943 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T07:34:38.943 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_num": 5799936, 2024-09-18T07:34:38.943 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_set": [ 2024-09-18T07:34:38.943 INFO:teuthology.orchestra.run.smithi137.stdout: "pglog_hardlimit", 2024-09-18T07:34:38.943 INFO:teuthology.orchestra.run.smithi137.stdout: "purged_snapdirs", 2024-09-18T07:34:38.943 INFO:teuthology.orchestra.run.smithi137.stdout: "recovery_deletes", 2024-09-18T07:34:38.943 INFO:teuthology.orchestra.run.smithi137.stdout: "sortbitwise" 2024-09-18T07:34:38.943 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:34:38.943 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_version": 18, 2024-09-18T07:34:38.943 INFO:teuthology.orchestra.run.smithi137.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T07:34:38.944 INFO:teuthology.orchestra.run.smithi137.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T07:34:38.944 INFO:teuthology.orchestra.run.smithi137.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T07:34:38.944 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_snapshot": "", 2024-09-18T07:34:38.944 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_max": 7, 2024-09-18T07:34:38.944 INFO:teuthology.orchestra.run.smithi137.stdout: "max_osd": 8, 2024-09-18T07:34:38.944 INFO:teuthology.orchestra.run.smithi137.stdout: "require_min_compat_client": "luminous", 2024-09-18T07:34:38.944 INFO:teuthology.orchestra.run.smithi137.stdout: "min_compat_client": "luminous", 2024-09-18T07:34:38.944 INFO:teuthology.orchestra.run.smithi137.stdout: "require_osd_release": "squid", 2024-09-18T07:34:38.944 INFO:teuthology.orchestra.run.smithi137.stdout: "allow_crimson": false, 2024-09-18T07:34:38.944 INFO:teuthology.orchestra.run.smithi137.stdout: "pools": [ 2024-09-18T07:34:38.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.944 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 1, 2024-09-18T07:34:38.944 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".mgr", 2024-09-18T07:34:38.945 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:28:20.899655+0000", 2024-09-18T07:34:38.945 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:34:38.945 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:34:38.945 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:34:38.945 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:34:38.945 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:34:38.945 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:34:38.945 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:34:38.945 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:34:38.945 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:34:38.945 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:34:38.945 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:34:38.946 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:34:38.946 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:34:38.946 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 1, 2024-09-18T07:34:38.946 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 1, 2024-09-18T07:34:38.946 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 1, 2024-09-18T07:34:38.946 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 1, 2024-09-18T07:34:38.946 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 1, 2024-09-18T07:34:38.946 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:34:38.946 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:34:38.946 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:34:38.947 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:34:38.947 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:34:38.947 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:34:38.947 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:34:38.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.947 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "21", 2024-09-18T07:34:38.947 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:34:38.947 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:34:38.947 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:34:38.947 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:34:38.947 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:34:38.947 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:34:38.947 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:34:38.948 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:34:38.948 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:34:38.948 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:34:38.948 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:34:38.948 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:34:38.948 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:34:38.948 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:34:38.948 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:34:38.949 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:34:38.949 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:34:38.949 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:34:38.949 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:34:38.949 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:34:38.949 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:34:38.949 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:34:38.949 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:34:38.949 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:34:38.949 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:34:38.949 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:34:38.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.949 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:34:38.950 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:34:38.950 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:34:38.950 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:34:38.950 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:34:38.950 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:34:38.950 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:34:38.950 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:34:38.950 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:34:38.950 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:34:38.950 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:34:38.951 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:34:38.951 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_max": 32, 2024-09-18T07:34:38.951 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_min": 1 2024-09-18T07:34:38.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.951 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:34:38.951 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": {} 2024-09-18T07:34:38.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.951 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:34:38.951 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:34:38.951 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 7.8899998664855957, 2024-09-18T07:34:38.951 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 7.8899998664855957, 2024-09-18T07:34:38.951 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.37999999523162842, 2024-09-18T07:34:38.951 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 3, 2024-09-18T07:34:38.952 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 3, 2024-09-18T07:34:38.952 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:34:38.952 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:34:38.952 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:34:38.952 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:38.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.952 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 2, 2024-09-18T07:34:38.952 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "datapool", 2024-09-18T07:34:38.952 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:53.715373+0000", 2024-09-18T07:34:38.952 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-18T07:34:38.952 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T07:34:38.953 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:34:38.953 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:34:38.953 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:34:38.953 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:34:38.953 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:34:38.953 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:34:38.953 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:34:38.953 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:34:38.953 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:34:38.953 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:34:38.953 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:34:38.953 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 3, 2024-09-18T07:34:38.954 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 3, 2024-09-18T07:34:38.954 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 3, 2024-09-18T07:34:38.954 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 3, 2024-09-18T07:34:38.954 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 3, 2024-09-18T07:34:38.954 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:34:38.954 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:34:38.954 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:34:38.954 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:34:38.954 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:34:38.954 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:34:38.954 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:34:38.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.954 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "63", 2024-09-18T07:34:38.954 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:34:38.955 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:34:38.955 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:34:38.955 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:34:38.955 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:34:38.955 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 2, 2024-09-18T07:34:38.955 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 63, 2024-09-18T07:34:38.955 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:34:38.955 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:34:38.955 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:34:38.955 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:34:38.955 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:34:38.955 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:34:38.955 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:34:38.955 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:34:38.955 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:34:38.956 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:34:38.956 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:34:38.956 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:34:38.956 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:34:38.956 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:34:38.956 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:34:38.956 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:34:38.956 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:34:38.956 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:34:38.956 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:34:38.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.956 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:34:38.956 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:34:38.956 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:34:38.957 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:34:38.957 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:34:38.957 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:34:38.957 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:34:38.957 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:34:38.957 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:34:38.957 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:34:38.957 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:34:38.957 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:34:38.957 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:34:38.957 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd": {} 2024-09-18T07:34:38.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.957 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:34:38.957 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:34:38.958 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.6700000762939453, 2024-09-18T07:34:38.958 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.6700000762939453, 2024-09-18T07:34:38.958 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.75, 2024-09-18T07:34:38.958 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2, 2024-09-18T07:34:38.958 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2, 2024-09-18T07:34:38.958 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:34:38.958 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:34:38.958 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:34:38.958 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:38.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.958 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 3, 2024-09-18T07:34:38.958 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".rgw.root", 2024-09-18T07:34:38.958 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:54.672127+0000", 2024-09-18T07:34:38.958 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:34:38.959 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:34:38.959 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:34:38.959 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:34:38.959 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:34:38.959 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:34:38.959 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:34:38.959 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:34:38.959 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:34:38.959 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:34:38.959 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:34:38.959 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:34:38.959 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:34:38.959 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:34:38.959 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:34:38.960 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:34:38.960 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:34:38.960 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:34:38.960 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:34:38.960 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:34:38.960 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:34:38.960 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:34:38.960 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:34:38.960 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:34:38.961 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:34:38.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.961 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "62", 2024-09-18T07:34:38.961 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:34:38.961 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:34:38.961 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:34:38.961 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:34:38.961 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:34:38.961 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:34:38.962 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:34:38.962 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:34:38.962 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:34:38.962 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:34:38.962 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:34:38.962 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:34:38.962 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:34:38.962 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:34:38.962 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:34:38.962 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:34:38.962 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:34:38.962 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:34:38.962 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:34:38.962 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:34:38.963 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:34:38.963 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:34:38.963 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:34:38.963 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:34:38.963 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:34:38.963 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:34:38.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.963 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:34:38.963 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:34:38.963 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:34:38.963 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:34:38.963 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:34:38.963 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:34:38.963 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:34:38.964 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:34:38.964 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:34:38.964 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:34:38.964 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:34:38.964 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:34:38.964 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:34:38.964 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:34:38.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.964 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:34:38.964 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:34:38.964 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.9900000095367432, 2024-09-18T07:34:38.964 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.9900000095367432, 2024-09-18T07:34:38.964 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.87999999523162842, 2024-09-18T07:34:38.964 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.75, 2024-09-18T07:34:38.965 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.75, 2024-09-18T07:34:38.965 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:34:38.965 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:34:38.965 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:34:38.965 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:38.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.965 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 4, 2024-09-18T07:34:38.965 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.log", 2024-09-18T07:34:38.965 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:56.351343+0000", 2024-09-18T07:34:38.965 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:34:38.965 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:34:38.965 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:34:38.966 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:34:38.966 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:34:38.966 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:34:38.966 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:34:38.966 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:34:38.966 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:34:38.966 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:34:38.966 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:34:38.966 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:34:38.966 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:34:38.966 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:34:38.966 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:34:38.966 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:34:38.967 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:34:38.967 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:34:38.967 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:34:38.967 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:34:38.967 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:34:38.967 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:34:38.967 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:34:38.967 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:34:38.967 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:34:38.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.967 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "64", 2024-09-18T07:34:38.967 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:34:38.967 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:34:38.967 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:34:38.968 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:34:38.968 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:34:38.968 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:34:38.968 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:34:38.968 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:34:38.968 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:34:38.968 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:34:38.968 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:34:38.968 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:34:38.968 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:34:38.968 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:34:38.969 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:34:38.969 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:34:38.969 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:34:38.969 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:34:38.969 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:34:38.969 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:34:38.969 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:34:38.969 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:34:38.969 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:34:38.969 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:34:38.969 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:34:38.969 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:34:38.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.970 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:34:38.970 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:34:38.970 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:34:38.970 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:34:38.970 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:34:38.970 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:34:38.970 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:34:38.970 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:34:38.970 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:34:38.970 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:34:38.970 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:34:38.970 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:34:38.970 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:34:38.970 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:34:38.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.971 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:34:38.971 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:34:38.971 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.2400000095367432, 2024-09-18T07:34:38.971 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.2400000095367432, 2024-09-18T07:34:38.971 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.87999999523162842, 2024-09-18T07:34:38.971 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-18T07:34:38.971 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-18T07:34:38.971 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:34:38.971 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:34:38.971 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:34:38.971 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:38.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.972 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 5, 2024-09-18T07:34:38.972 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.control", 2024-09-18T07:34:38.972 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:58.322401+0000", 2024-09-18T07:34:38.972 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:34:38.972 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:34:38.972 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:34:38.972 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:34:38.972 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:34:38.972 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:34:38.972 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:34:38.972 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:34:38.972 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:34:38.973 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:34:38.973 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:34:38.973 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:34:38.973 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:34:38.973 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:34:38.973 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:34:38.973 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:34:38.973 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:34:38.973 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:34:38.973 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:34:38.973 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:34:38.973 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:34:38.973 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:34:38.973 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:34:38.974 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:34:38.974 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:34:38.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.974 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "66", 2024-09-18T07:34:38.974 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:34:38.974 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:34:38.974 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:34:38.974 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:34:38.974 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:34:38.974 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:34:38.974 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:34:38.974 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:34:38.974 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:34:38.974 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:34:38.975 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:34:38.975 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:34:38.975 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:34:38.975 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:34:38.975 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:34:38.975 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:34:38.975 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:34:38.975 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:34:38.975 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:34:38.975 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:34:38.975 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:34:38.975 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:34:38.975 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:34:38.976 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:34:38.976 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:34:38.976 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:34:38.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.976 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:34:38.976 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:34:38.976 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:34:38.976 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:34:38.976 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:34:38.976 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:34:38.976 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:34:38.976 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:34:38.976 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:34:38.976 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:34:38.977 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:34:38.977 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:34:38.977 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:34:38.977 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:34:38.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.977 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:34:38.977 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:34:38.977 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.4900000095367432, 2024-09-18T07:34:38.977 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.4900000095367432, 2024-09-18T07:34:38.977 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.87999999523162842, 2024-09-18T07:34:38.977 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.309999942779541, 2024-09-18T07:34:38.977 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.309999942779541, 2024-09-18T07:34:38.977 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:34:38.977 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:34:38.978 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:34:38.978 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:38.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.978 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 6, 2024-09-18T07:34:38.978 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.meta", 2024-09-18T07:34:38.978 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:31:00.390499+0000", 2024-09-18T07:34:38.978 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:34:38.978 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:34:38.978 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:34:38.978 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:34:38.978 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:34:38.978 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:34:38.979 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:34:38.979 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:34:38.979 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:34:38.979 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:34:38.979 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:34:38.979 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:34:38.979 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:34:38.979 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:34:38.979 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:34:38.979 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:34:38.979 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:34:38.979 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:34:38.979 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:34:38.980 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:34:38.980 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:34:38.980 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:34:38.980 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:34:38.980 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:34:38.980 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:34:38.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.980 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "68", 2024-09-18T07:34:38.980 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:34:38.980 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:34:38.980 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:34:38.980 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:34:38.980 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:34:38.981 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:34:38.981 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:34:38.981 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:34:38.981 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:34:38.981 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:34:38.981 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:34:38.981 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:34:38.981 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:34:38.981 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:34:38.981 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:34:38.981 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:34:38.981 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:34:38.981 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:34:38.981 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:34:38.982 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:34:38.982 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:34:38.982 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:34:38.982 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:34:38.982 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:34:38.982 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:34:38.982 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:34:38.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.982 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:34:38.982 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:34:38.982 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:34:38.982 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:34:38.982 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:34:38.983 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:34:38.983 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:34:38.983 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:34:38.983 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:34:38.983 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:34:38.983 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:34:38.983 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:34:38.983 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_bias": 4 2024-09-18T07:34:38.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.983 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:34:38.983 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:34:38.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.983 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:34:38.983 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:34:38.984 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.4900000095367432, 2024-09-18T07:34:38.984 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.4900000095367432, 2024-09-18T07:34:38.984 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.87999999523162842, 2024-09-18T07:34:38.984 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.309999942779541, 2024-09-18T07:34:38.984 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.309999942779541, 2024-09-18T07:34:38.984 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:34:38.984 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:34:38.984 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:34:38.984 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:38.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.984 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-18T07:34:38.984 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "unique_pool_0", 2024-09-18T07:34:38.984 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:32:06.616722+0000", 2024-09-18T07:34:38.985 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:34:38.985 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:34:38.985 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:34:38.985 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:34:38.985 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:34:38.985 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:34:38.985 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:34:38.985 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:34:38.985 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:34:38.985 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:34:38.985 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:34:38.985 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:34:38.985 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:34:38.985 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 25, 2024-09-18T07:34:38.986 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 16, 2024-09-18T07:34:38.986 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 16, 2024-09-18T07:34:38.986 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 16, 2024-09-18T07:34:38.986 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 24, 2024-09-18T07:34:38.986 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:34:38.986 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "7.19", 2024-09-18T07:34:38.986 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 148, 2024-09-18T07:34:38.986 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 147, 2024-09-18T07:34:38.986 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 147, 2024-09-18T07:34:38.986 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "145'8895", 2024-09-18T07:34:38.986 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "145'8920" 2024-09-18T07:34:38.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.986 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "150", 2024-09-18T07:34:38.986 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:34:38.987 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "150", 2024-09-18T07:34:38.987 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "150", 2024-09-18T07:34:38.987 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:34:38.987 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:34:38.987 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:34:38.987 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:34:38.987 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:34:38.987 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:34:38.987 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:34:38.987 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:34:38.987 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:34:38.987 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:34:38.987 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:34:38.987 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:34:38.988 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:34:38.988 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:34:38.988 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:34:38.988 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:34:38.988 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:34:38.988 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:34:38.988 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:34:38.988 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:34:38.988 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:34:38.988 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:34:38.988 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:34:38.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.988 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:34:38.988 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:34:38.989 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:34:38.989 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:34:38.989 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:34:38.989 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:34:38.989 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:34:38.989 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:34:38.989 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:34:38.989 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:34:38.989 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:34:38.989 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:34:38.989 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:34:38.989 INFO:teuthology.orchestra.run.smithi137.stdout: "rados": {} 2024-09-18T07:34:38.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.989 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:34:38.990 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:34:38.990 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.9099999666213989, 2024-09-18T07:34:38.990 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.9099999666213989, 2024-09-18T07:34:38.990 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.87999999523162842, 2024-09-18T07:34:38.990 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.6799999475479126, 2024-09-18T07:34:38.990 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.6799999475479126, 2024-09-18T07:34:38.990 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:34:38.990 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:34:38.990 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:34:38.990 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:38.990 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:38.990 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:34:38.991 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-18T07:34:38.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.991 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:34:38.991 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "268cd580-c931-4992-bb93-d6f63c2bfe88", 2024-09-18T07:34:38.991 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:34:38.991 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:34:38.991 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:34:38.991 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:34:38.991 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:34:38.991 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:34:38.991 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 8, 2024-09-18T07:34:38.991 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 148, 2024-09-18T07:34:38.991 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:34:38.991 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:34:38.991 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:34:38.992 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:38.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.992 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:38.992 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6801", 2024-09-18T07:34:38.992 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:34:38.992 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:38.992 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:38.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.992 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:34:38.992 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:38.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.992 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:38.992 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6802", 2024-09-18T07:34:38.993 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:34:38.993 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:38.993 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:38.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.993 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:34:38.993 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:38.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.993 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:38.993 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6804", 2024-09-18T07:34:38.993 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:34:38.993 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:38.993 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:38.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.994 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:34:38.994 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:38.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.994 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:38.994 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6803", 2024-09-18T07:34:38.994 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:34:38.994 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:38.994 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:38.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.994 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6801/2811444477", 2024-09-18T07:34:38.994 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6802/2811444477", 2024-09-18T07:34:38.994 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6804/2811444477", 2024-09-18T07:34:38.994 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6803/2811444477", 2024-09-18T07:34:38.994 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:34:38.995 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:34:38.995 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:34:38.995 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:38.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.995 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:34:38.995 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575", 2024-09-18T07:34:38.995 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:34:38.995 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:34:38.995 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:34:38.995 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:34:38.995 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:34:38.995 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:34:38.995 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 13, 2024-09-18T07:34:38.996 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 143, 2024-09-18T07:34:38.996 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:34:38.996 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:34:38.996 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:34:38.996 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:38.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.996 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:38.996 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6805", 2024-09-18T07:34:38.996 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:34:38.996 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:38.996 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:38.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.996 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:34:38.996 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:38.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.997 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:38.997 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6806", 2024-09-18T07:34:38.997 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:34:38.997 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:38.997 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:38.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.997 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:34:38.997 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:38.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.997 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:38.997 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6808", 2024-09-18T07:34:38.998 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:34:38.998 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:38.998 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:38.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.998 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:34:38.998 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:38.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.998 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:38.998 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6807", 2024-09-18T07:34:38.998 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:34:38.998 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:38.998 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:38.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.999 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6805/874102487", 2024-09-18T07:34:38.999 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6806/874102487", 2024-09-18T07:34:38.999 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6808/874102487", 2024-09-18T07:34:38.999 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6807/874102487", 2024-09-18T07:34:38.999 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:34:38.999 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:34:38.999 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:34:38.999 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:38.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:38.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:38.999 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:34:39.000 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "480ad6c5-0e0b-4405-bbe2-381682d00bae", 2024-09-18T07:34:39.000 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:34:39.000 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:34:39.000 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:34:39.000 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:34:39.000 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:34:39.000 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:34:39.000 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 18, 2024-09-18T07:34:39.000 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 144, 2024-09-18T07:34:39.000 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:34:39.000 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:34:39.000 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:34:39.000 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.001 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.001 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6809", 2024-09-18T07:34:39.001 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:34:39.001 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.001 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.001 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:34:39.001 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.001 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.001 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6810", 2024-09-18T07:34:39.002 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:34:39.002 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.002 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.002 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:34:39.002 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.002 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.002 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6812", 2024-09-18T07:34:39.002 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:34:39.002 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.003 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.003 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:34:39.003 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.003 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.003 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6811", 2024-09-18T07:34:39.003 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:34:39.003 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.003 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.003 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6809/1754684686", 2024-09-18T07:34:39.003 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6810/1754684686", 2024-09-18T07:34:39.004 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6812/1754684686", 2024-09-18T07:34:39.004 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6811/1754684686", 2024-09-18T07:34:39.004 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:34:39.004 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:34:39.004 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:34:39.004 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.005 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:34:39.005 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14", 2024-09-18T07:34:39.005 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:34:39.006 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:34:39.006 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:34:39.006 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:34:39.006 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:34:39.006 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:34:39.006 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 26, 2024-09-18T07:34:39.006 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 144, 2024-09-18T07:34:39.007 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:34:39.007 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:34:39.007 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:34:39.007 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.007 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.008 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6813", 2024-09-18T07:34:39.008 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:34:39.008 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.008 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.008 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:34:39.009 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.009 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.009 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6814", 2024-09-18T07:34:39.009 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:34:39.009 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.010 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.010 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:34:39.010 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.010 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.011 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6816", 2024-09-18T07:34:39.011 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:34:39.011 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.011 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.011 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:34:39.012 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.012 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.012 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6815", 2024-09-18T07:34:39.012 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:34:39.013 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.013 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.013 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6813/116058392", 2024-09-18T07:34:39.013 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6814/116058392", 2024-09-18T07:34:39.013 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6816/116058392", 2024-09-18T07:34:39.014 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6815/116058392", 2024-09-18T07:34:39.014 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:34:39.014 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:34:39.014 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:34:39.014 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.015 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:34:39.015 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de", 2024-09-18T07:34:39.015 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:34:39.015 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:34:39.016 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:34:39.016 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:34:39.016 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:34:39.016 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:34:39.016 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 32, 2024-09-18T07:34:39.016 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 144, 2024-09-18T07:34:39.017 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:34:39.017 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:34:39.017 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:34:39.017 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.017 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.018 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6800", 2024-09-18T07:34:39.018 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:34:39.018 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.018 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.018 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:34:39.019 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.019 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.019 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6801", 2024-09-18T07:34:39.019 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:34:39.019 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.020 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.020 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:34:39.020 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.020 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.021 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.021 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6803", 2024-09-18T07:34:39.021 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:34:39.021 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.021 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.022 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:34:39.022 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.022 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.022 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6802", 2024-09-18T07:34:39.022 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:34:39.023 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.023 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.023 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6800/3654297954", 2024-09-18T07:34:39.023 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6801/3654297954", 2024-09-18T07:34:39.023 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6803/3654297954", 2024-09-18T07:34:39.024 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6802/3654297954", 2024-09-18T07:34:39.024 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:34:39.024 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:34:39.024 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:34:39.024 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.025 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:34:39.025 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c", 2024-09-18T07:34:39.025 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:34:39.025 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:34:39.025 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:34:39.026 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:34:39.026 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:34:39.026 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:34:39.026 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 39, 2024-09-18T07:34:39.026 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 144, 2024-09-18T07:34:39.026 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:34:39.027 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:34:39.027 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:34:39.027 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.027 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.027 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6804", 2024-09-18T07:34:39.028 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:34:39.028 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.028 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.028 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:34:39.029 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.029 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:39.031 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6805", 2024-09-18T07:34:39.031 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:34:39.031 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:34:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6807", 2024-09-18T07:34:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:34:39.032 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:34:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6806", 2024-09-18T07:34:39.033 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:34:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6804/837994222", 2024-09-18T07:34:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6805/837994222", 2024-09-18T07:34:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6807/837994222", 2024-09-18T07:34:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6806/837994222", 2024-09-18T07:34:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:34:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:34:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:34:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:34:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6", 2024-09-18T07:34:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:34:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:34:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:34:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:34:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:34:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:34:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 47, 2024-09-18T07:34:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 144, 2024-09-18T07:34:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:34:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:34:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:34:39.035 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6808", 2024-09-18T07:34:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:34:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:34:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.036 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6809", 2024-09-18T07:34:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:34:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:34:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6811", 2024-09-18T07:34:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:34:39.037 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:34:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6810", 2024-09-18T07:34:39.038 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:34:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6808/1494175466", 2024-09-18T07:34:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6809/1494175466", 2024-09-18T07:34:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6811/1494175466", 2024-09-18T07:34:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6810/1494175466", 2024-09-18T07:34:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:34:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:34:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:34:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:34:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a", 2024-09-18T07:34:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:34:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-18T07:34:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-18T07:34:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:34:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:34:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:34:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 55, 2024-09-18T07:34:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 81, 2024-09-18T07:34:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:34:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:34:39.040 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:34:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6812", 2024-09-18T07:34:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:34:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:34:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.041 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6813", 2024-09-18T07:34:39.042 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:34:39.042 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.042 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.042 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:34:39.042 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.042 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.042 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6815", 2024-09-18T07:34:39.042 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:34:39.043 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.043 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.043 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:34:39.043 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:34:39.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.043 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:34:39.043 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6814", 2024-09-18T07:34:39.043 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:34:39.043 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.043 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.043 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6812/2670809315", 2024-09-18T07:34:39.044 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6813/2670809315", 2024-09-18T07:34:39.044 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6815/2670809315", 2024-09-18T07:34:39.044 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6814/2670809315", 2024-09-18T07:34:39.044 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:34:39.044 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:34:39.044 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:34:39.044 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.044 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.044 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:34:39.044 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_xinfo": [ 2024-09-18T07:34:39.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:34:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:34:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:34:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:34:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:34:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:34:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:33:41.923788+0000", 2024-09-18T07:34:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:34:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:34:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:34:39.045 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:34:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:34:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:34:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:34:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:34:11.710796+0000", 2024-09-18T07:34:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:34:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:34:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:34:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:34:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:34:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:34:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:34:39.046 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:33:22.879864+0000", 2024-09-18T07:34:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:34:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:34:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:34:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:34:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:34:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:34:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:34:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:34:01.405459+0000", 2024-09-18T07:34:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:34:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:34:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:34:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:34:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:34:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:34:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:34:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:34:24.079197+0000", 2024-09-18T07:34:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:34:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:34:39.048 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:34:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:34:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:34:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:34:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:34:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:33:27.555272+0000", 2024-09-18T07:34:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:34:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:34:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:34:39.049 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:34:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:34:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:34:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:34:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:33:39.248426+0000", 2024-09-18T07:34:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:34:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:34:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:34:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:34:39.050 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:34:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:34:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-18T07:34:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:34:03.952450+0000", 2024-09-18T07:34:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:34:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:34:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap": [], 2024-09-18T07:34:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_items": [ 2024-09-18T07:34:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.6", 2024-09-18T07:34:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:34:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.051 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:34:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:34:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.e", 2024-09-18T07:34:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:34:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:34:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:34:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.10", 2024-09-18T07:34:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:34:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:34:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:34:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.053 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.12", 2024-09-18T07:34:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:34:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:34:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:34:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.18", 2024-09-18T07:34:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:34:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:34:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:34:39.054 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.055 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.055 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.b", 2024-09-18T07:34:39.055 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:34:39.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.055 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 0, 2024-09-18T07:34:39.055 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 3 2024-09-18T07:34:39.055 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.055 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.055 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.d", 2024-09-18T07:34:39.055 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:34:39.056 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.056 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 0, 2024-09-18T07:34:39.056 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 6 2024-09-18T07:34:39.056 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.056 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.056 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.056 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.1b", 2024-09-18T07:34:39.056 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:34:39.056 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:39.056 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:34:39.056 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:34:39.056 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.057 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:39.057 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.057 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:34:39.057 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_primaries": [], 2024-09-18T07:34:39.057 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_temp": [], 2024-09-18T07:34:39.057 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_temp": [], 2024-09-18T07:34:39.057 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklist": { 2024-09-18T07:34:39.057 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/2096039404": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:34:39.057 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/3636177939": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:34:39.057 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/334967443": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:34:39.057 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2101286111": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:34:39.057 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3387637412": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:34:39.057 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762989503": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:34:39.058 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3490817671": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:34:39.058 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1482147392": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:34:39.058 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1821482181": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:34:39.058 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/4028304023": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:34:39.058 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/621434521": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:34:39.058 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3160704935": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:34:39.058 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2357293499": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:34:39.058 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762352860": "2024-09-19T07:31:25.389124+0000" 2024-09-18T07:34:39.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.058 INFO:teuthology.orchestra.run.smithi137.stdout: "range_blocklist": {}, 2024-09-18T07:34:39.058 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profiles": { 2024-09-18T07:34:39.058 INFO:teuthology.orchestra.run.smithi137.stdout: "default": { 2024-09-18T07:34:39.058 INFO:teuthology.orchestra.run.smithi137.stdout: "crush-failure-domain": "osd", 2024-09-18T07:34:39.058 INFO:teuthology.orchestra.run.smithi137.stdout: "k": "2", 2024-09-18T07:34:39.059 INFO:teuthology.orchestra.run.smithi137.stdout: "m": "1", 2024-09-18T07:34:39.059 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin": "jerasure", 2024-09-18T07:34:39.059 INFO:teuthology.orchestra.run.smithi137.stdout: "technique": "reed_sol_van" 2024-09-18T07:34:39.059 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.059 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:39.059 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps_queue": [], 2024-09-18T07:34:39.059 INFO:teuthology.orchestra.run.smithi137.stdout: "new_removed_snaps": [], 2024-09-18T07:34:39.059 INFO:teuthology.orchestra.run.smithi137.stdout: "new_purged_snaps": [], 2024-09-18T07:34:39.059 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_node_flags": {}, 2024-09-18T07:34:39.059 INFO:teuthology.orchestra.run.smithi137.stdout: "device_class_flags": {}, 2024-09-18T07:34:39.059 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode": { 2024-09-18T07:34:39.059 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_enabled": false, 2024-09-18T07:34:39.059 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_bucket_count": 0, 2024-09-18T07:34:39.059 INFO:teuthology.orchestra.run.smithi137.stdout: "degraded_stretch_mode": 0, 2024-09-18T07:34:39.059 INFO:teuthology.orchestra.run.smithi137.stdout: "recovering_stretch_mode": 0, 2024-09-18T07:34:39.060 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_bucket": 0 2024-09-18T07:34:39.060 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:39.060 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:34:39.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:39.061 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:39.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:39.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:39.184 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:39.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:39.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:39.198 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:39 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"}]': finished 2024-09-18T07:34:39.287 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:39.308 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:39.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:39.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:39.410 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:39.430 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:39.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:39.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:39.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:39 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"}]': finished 2024-09-18T07:34:39.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:39 smithi203 ceph-mon[32237]: osdmap e150: 8 total, 8 up, 7 in 2024-09-18T07:34:39.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:39 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:34:39.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:39 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2799529736' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:39.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:39 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4069983967' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:34:39.532 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:39.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:39.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:39.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:39.562 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:39 smithi137 ceph-mon[29408]: osdmap e150: 8 total, 8 up, 7 in 2024-09-18T07:34:39.563 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:39 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:34:39.563 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:39 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2799529736' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:39.563 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:39 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4069983967' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:34:39.563 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:39 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"}]': finished 2024-09-18T07:34:39.563 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:39 smithi137 ceph-mon[39175]: osdmap e150: 8 total, 8 up, 7 in 2024-09-18T07:34:39.563 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:39 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:34:39.563 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:39 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2799529736' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:39.563 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:39 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4069983967' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:34:39.656 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:39.677 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:39.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:39.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:39.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:39.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:39.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:39.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:39.927 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:39.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:39.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:40.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:40.051 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:40.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:40.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:40.153 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:40.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:40.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:40.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:40.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:40.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:40.300 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:40.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:40.401 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:40.422 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:40.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:40.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:40.524 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:40.545 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:40.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:40.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:40.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:40 smithi203 ceph-mon[32237]: 4.7 scrub starts 2024-09-18T07:34:40.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:40 smithi203 ceph-mon[32237]: 4.7 scrub ok 2024-09-18T07:34:40.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:40 smithi203 ceph-mon[32237]: osdmap e151: 8 total, 8 up, 7 in 2024-09-18T07:34:40.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:40 smithi203 ceph-mon[32237]: 5.0 scrub starts 2024-09-18T07:34:40.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:40 smithi203 ceph-mon[32237]: 5.0 scrub ok 2024-09-18T07:34:40.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:40 smithi203 ceph-mon[32237]: 2.2 scrub starts 2024-09-18T07:34:40.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:40 smithi203 ceph-mon[32237]: 2.2 scrub ok 2024-09-18T07:34:40.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:40 smithi203 ceph-mon[32237]: pgmap v178: 157 pgs: 1 peering, 156 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:34:40.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:40 smithi203 ceph-mon[32237]: 4.3 scrub starts 2024-09-18T07:34:40.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:40 smithi203 ceph-mon[32237]: 4.3 scrub ok 2024-09-18T07:34:40.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:40.668 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:40.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:40.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[29408]: 4.7 scrub starts 2024-09-18T07:34:40.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[29408]: 4.7 scrub ok 2024-09-18T07:34:40.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[29408]: osdmap e151: 8 total, 8 up, 7 in 2024-09-18T07:34:40.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[29408]: 5.0 scrub starts 2024-09-18T07:34:40.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[29408]: 5.0 scrub ok 2024-09-18T07:34:40.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[29408]: 2.2 scrub starts 2024-09-18T07:34:40.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[29408]: 2.2 scrub ok 2024-09-18T07:34:40.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[29408]: pgmap v178: 157 pgs: 1 peering, 156 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:34:40.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[29408]: 4.3 scrub starts 2024-09-18T07:34:40.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[29408]: 4.3 scrub ok 2024-09-18T07:34:40.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[39175]: 4.7 scrub starts 2024-09-18T07:34:40.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[39175]: 4.7 scrub ok 2024-09-18T07:34:40.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[39175]: osdmap e151: 8 total, 8 up, 7 in 2024-09-18T07:34:40.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[39175]: 5.0 scrub starts 2024-09-18T07:34:40.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[39175]: 5.0 scrub ok 2024-09-18T07:34:40.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[39175]: 2.2 scrub starts 2024-09-18T07:34:40.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[39175]: 2.2 scrub ok 2024-09-18T07:34:40.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[39175]: pgmap v178: 157 pgs: 1 peering, 156 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:34:40.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[39175]: 4.3 scrub starts 2024-09-18T07:34:40.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:40 smithi137 ceph-mon[39175]: 4.3 scrub ok 2024-09-18T07:34:40.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:40.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:40.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:40.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:40.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:40.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:40.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:41.000 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:41.016 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:41.034 INFO:tasks.thrashosds.thrasher:j is {'epoch': 150, 'fsid': 'f7ef099c-758e-11ef-bceb-c7b262605968', 'created': '2024-09-18T07:25:19.662367+0000', 'modified': '2024-09-18T07:34:38.179526+0000', 'last_up_change': '2024-09-18T07:30:44.708496+0000', 'last_in_change': '2024-09-18T07:32:29.796199+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T07:28:20.899655+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-18T07:30:53.715373+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-18T07:30:54.672127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-18T07:30:56.351343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-18T07:30:58.322401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-18T07:31:00.390499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-18T07:32:06.616722+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 25, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 24, 'last_pg_merge_meta': {'source_pgid': '7.19', 'ready_epoch': 148, 'last_epoch_started': 147, 'last_epoch_clean': 147, 'source_version': "145'8895", 'target_version': "145'8920"}, 'last_change': '150', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '150', 'last_force_op_resend_preluminous': '150', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.6799999475479126, 'raw_score_stable': 1.6799999475479126, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '268cd580-c931-4992-bb93-d6f63c2bfe88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 148, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6801', 'nonce': 2811444477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6802', 'nonce': 2811444477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6804', 'nonce': 2811444477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6803', 'nonce': 2811444477}]}, 'public_addr': '172.21.15.137:6801/2811444477', 'cluster_addr': '172.21.15.137:6802/2811444477', 'heartbeat_back_addr': '172.21.15.137:6804/2811444477', 'heartbeat_front_addr': '172.21.15.137:6803/2811444477', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b06907bc-b499-46c0-8667-c5bdc0d85575', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 143, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6805', 'nonce': 874102487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6806', 'nonce': 874102487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6808', 'nonce': 874102487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6807', 'nonce': 874102487}]}, 'public_addr': '172.21.15.137:6805/874102487', 'cluster_addr': '172.21.15.137:6806/874102487', 'heartbeat_back_addr': '172.21.15.137:6808/874102487', 'heartbeat_front_addr': '172.21.15.137:6807/874102487', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '480ad6c5-0e0b-4405-bbe2-381682d00bae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 144, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6809', 'nonce': 1754684686}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6810', 'nonce': 1754684686}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6812', 'nonce': 1754684686}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6811', 'nonce': 1754684686}]}, 'public_addr': '172.21.15.137:6809/1754684686', 'cluster_addr': '172.21.15.137:6810/1754684686', 'heartbeat_back_addr': '172.21.15.137:6812/1754684686', 'heartbeat_front_addr': '172.21.15.137:6811/1754684686', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '178be3e0-f07b-4856-807b-2a1181e48a14', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 144, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6813', 'nonce': 116058392}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6814', 'nonce': 116058392}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6816', 'nonce': 116058392}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6815', 'nonce': 116058392}]}, 'public_addr': '172.21.15.137:6813/116058392', 'cluster_addr': '172.21.15.137:6814/116058392', 'heartbeat_back_addr': '172.21.15.137:6816/116058392', 'heartbeat_front_addr': '172.21.15.137:6815/116058392', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7547102c-951a-4077-a094-fb8a65bfe6de', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 144, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6800', 'nonce': 3654297954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6801', 'nonce': 3654297954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6803', 'nonce': 3654297954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6802', 'nonce': 3654297954}]}, 'public_addr': '172.21.15.203:6800/3654297954', 'cluster_addr': '172.21.15.203:6801/3654297954', 'heartbeat_back_addr': '172.21.15.203:6803/3654297954', 'heartbeat_front_addr': '172.21.15.203:6802/3654297954', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '49c17901-8102-48a4-9348-fa1bdb63032c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 144, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6804', 'nonce': 837994222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6805', 'nonce': 837994222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6807', 'nonce': 837994222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6806', 'nonce': 837994222}]}, 'public_addr': '172.21.15.203:6804/837994222', 'cluster_addr': '172.21.15.203:6805/837994222', 'heartbeat_back_addr': '172.21.15.203:6807/837994222', 'heartbeat_front_addr': '172.21.15.203:6806/837994222', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '75f0ead4-e981-4a15-b1c1-d83dce871ab6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 144, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6808', 'nonce': 1494175466}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6809', 'nonce': 1494175466}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6811', 'nonce': 1494175466}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6810', 'nonce': 1494175466}]}, 'public_addr': '172.21.15.203:6808/1494175466', 'cluster_addr': '172.21.15.203:6809/1494175466', 'heartbeat_back_addr': '172.21.15.203:6811/1494175466', 'heartbeat_front_addr': '172.21.15.203:6810/1494175466', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c0dbf299-6798-4b99-9a85-62b123971f2a', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 55, 'up_thru': 81, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6812', 'nonce': 2670809315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6813', 'nonce': 2670809315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6815', 'nonce': 2670809315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6814', 'nonce': 2670809315}]}, 'public_addr': '172.21.15.203:6812/2670809315', 'cluster_addr': '172.21.15.203:6813/2670809315', 'heartbeat_back_addr': '172.21.15.203:6815/2670809315', 'heartbeat_front_addr': '172.21.15.203:6814/2670809315', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:33:41.923788+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:34:11.710796+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:33:22.879864+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:34:01.405459+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:34:24.079197+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:33:27.555272+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:33:39.248426+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-18T07:34:03.952450+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.b', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '5.d', 'mappings': [{'from': 0, 'to': 6}]}, {'pgid': '6.1b', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.137:6800/2096039404': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:6800/3636177939': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/334967443': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/2101286111': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/3387637412': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1762989503': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3490817671': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:6800/1482147392': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1821482181': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/4028304023': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/621434521': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3160704935': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/2357293499': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/1762352860': '2024-09-19T07:31:25.389124+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T07:34:41.034 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:34:41.037 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:41.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:41.139 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:41.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:41.161 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:41.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:41.214 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:34:41.262 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:41.267 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[39175]: 3.10 scrub starts 2024-09-18T07:34:41.267 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[39175]: 3.10 scrub ok 2024-09-18T07:34:41.267 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[39175]: osdmap e152: 8 total, 8 up, 7 in 2024-09-18T07:34:41.267 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[39175]: 6.d scrub starts 2024-09-18T07:34:41.267 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[39175]: 6.d scrub ok 2024-09-18T07:34:41.267 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3827045326' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:34:41.267 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[39175]: from='client.24997 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:34:41.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:41.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:34:41.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[29408]: 3.10 scrub starts 2024-09-18T07:34:41.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[29408]: 3.10 scrub ok 2024-09-18T07:34:41.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[29408]: osdmap e152: 8 total, 8 up, 7 in 2024-09-18T07:34:41.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[29408]: 6.d scrub starts 2024-09-18T07:34:41.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[29408]: 6.d scrub ok 2024-09-18T07:34:41.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3827045326' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:34:41.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[29408]: from='client.24997 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:34:41.269 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:41.269 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:41 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:34:41.290 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:41.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:41.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:41.392 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:41.429 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:41.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:41.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:41 smithi203 ceph-mon[32237]: 3.10 scrub starts 2024-09-18T07:34:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:41 smithi203 ceph-mon[32237]: 3.10 scrub ok 2024-09-18T07:34:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:41 smithi203 ceph-mon[32237]: osdmap e152: 8 total, 8 up, 7 in 2024-09-18T07:34:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:41 smithi203 ceph-mon[32237]: 6.d scrub starts 2024-09-18T07:34:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:41 smithi203 ceph-mon[32237]: 6.d scrub ok 2024-09-18T07:34:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:41 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3827045326' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:34:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:41 smithi203 ceph-mon[32237]: from='client.24997 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:34:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:41 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:41 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:34:41.530 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:41.573 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:41.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:41.674 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:41.704 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:41.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:41.806 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:41.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:34:41.851 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:41.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:41.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:41.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:41.981 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:41.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:41.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:42.082 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:42.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:42.147 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:42.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:42.228 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:34:42.228 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:34:42.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:34:42.228 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:34:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135022 7.13 7:cab60f17:::benchmark_data_smithi137_74180_object30606:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:34:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964232+0000", 2024-09-18T07:34:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 25.255493724000001, 2024-09-18T07:34:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2320603539999999, 2024-09-18T07:34:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135022 2024-09-18T07:34:42.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964232+0000", 2024-09-18T07:34:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964232+0000", 2024-09-18T07:34:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964235+0000", 2024-09-18T07:34:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9340000000000002e-06 2024-09-18T07:34:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964242+0000", 2024-09-18T07:34:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.379e-06 2024-09-18T07:34:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964261+0000", 2024-09-18T07:34:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8851999999999998e-05 2024-09-18T07:34:42.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964270+0000", 2024-09-18T07:34:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5040000000000008e-06 2024-09-18T07:34:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964336+0000", 2024-09-18T07:34:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5092999999999995e-05 2024-09-18T07:34:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:34:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964344+0000", 2024-09-18T07:34:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0879999999999995e-06 2024-09-18T07:34:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971555+0000", 2024-09-18T07:34:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0072113109999999998 2024-09-18T07:34:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971563+0000", 2024-09-18T07:34:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4689999999999997e-06 2024-09-18T07:34:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.190908+0000", 2024-09-18T07:34:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2193444609999999 2024-09-18T07:34:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.191034+0000", 2024-09-18T07:34:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000125871 2024-09-18T07:34:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3,4,5", 2024-09-18T07:34:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.191148+0000", 2024-09-18T07:34:42.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114097 2024-09-18T07:34:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.193787+0000", 2024-09-18T07:34:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026389650000000001 2024-09-18T07:34:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.194990+0000", 2024-09-18T07:34:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001203523 2024-09-18T07:34:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.195790+0000", 2024-09-18T07:34:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00079991899999999996 2024-09-18T07:34:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.195803+0000", 2024-09-18T07:34:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2955999999999999e-05 2024-09-18T07:34:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.196126+0000", 2024-09-18T07:34:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032304700000000003 2024-09-18T07:34:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.196180+0000", 2024-09-18T07:34:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4178000000000002e-05 2024-09-18T07:34:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.196292+0000", 2024-09-18T07:34:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011170700000000001 2024-09-18T07:34:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151629 7.8 7:14ab7424:::benchmark_data_smithi137_74180_object47213:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-18T07:34:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.256327+0000", 2024-09-18T07:34:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.963398027, 2024-09-18T07:34:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95853319999999997, 2024-09-18T07:34:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:34:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.238 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151629 2024-09-18T07:34:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256327+0000", 2024-09-18T07:34:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256327+0000", 2024-09-18T07:34:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256330+0000", 2024-09-18T07:34:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9869999999999999e-06 2024-09-18T07:34:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256339+0000", 2024-09-18T07:34:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8950000000000004e-06 2024-09-18T07:34:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256359+0000", 2024-09-18T07:34:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9269000000000001e-05 2024-09-18T07:34:42.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256370+0000", 2024-09-18T07:34:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1401e-05 2024-09-18T07:34:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282101+0000", 2024-09-18T07:34:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025731431999999999 2024-09-18T07:34:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282121+0000", 2024-09-18T07:34:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9453000000000002e-05 2024-09-18T07:34:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.214785+0000", 2024-09-18T07:34:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93266407500000004 2024-09-18T07:34:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.214861+0000", 2024-09-18T07:34:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5687999999999998e-05 2024-09-18T07:34:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.242 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151715 7.8 7:13c29d4a:::benchmark_data_smithi137_74180_object47299:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-18T07:34:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.256389+0000", 2024-09-18T07:34:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.9633367509999999, 2024-09-18T07:34:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96412654600000003, 2024-09-18T07:34:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151715 2024-09-18T07:34:42.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256389+0000", 2024-09-18T07:34:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256389+0000", 2024-09-18T07:34:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256390+0000", 2024-09-18T07:34:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7400000000000002e-07 2024-09-18T07:34:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256393+0000", 2024-09-18T07:34:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6349999999999999e-06 2024-09-18T07:34:42.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256399+0000", 2024-09-18T07:34:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.161e-06 2024-09-18T07:34:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256405+0000", 2024-09-18T07:34:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4600000000000002e-06 2024-09-18T07:34:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282159+0000", 2024-09-18T07:34:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025754367 2024-09-18T07:34:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282167+0000", 2024-09-18T07:34:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7309999999999993e-06 2024-09-18T07:34:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.215245+0000", 2024-09-18T07:34:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93307778399999997 2024-09-18T07:34:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.215358+0000", 2024-09-18T07:34:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113099 2024-09-18T07:34:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.215473+0000", 2024-09-18T07:34:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114968 2024-09-18T07:34:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217068+0000", 2024-09-18T07:34:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001595143 2024-09-18T07:34:42.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218016+0000", 2024-09-18T07:34:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00094819399999999997 2024-09-18T07:34:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220422+0000", 2024-09-18T07:34:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024053669999999998 2024-09-18T07:34:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220472+0000", 2024-09-18T07:34:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9965e-05 2024-09-18T07:34:42.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220515+0000", 2024-09-18T07:34:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3798000000000003e-05 2024-09-18T07:34:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151745 7.8 7:11d4aec3:::benchmark_data_smithi137_74180_object47329:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-18T07:34:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.256412+0000", 2024-09-18T07:34:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.9633134880000001, 2024-09-18T07:34:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96419230700000003, 2024-09-18T07:34:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.250 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151745 2024-09-18T07:34:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256412+0000", 2024-09-18T07:34:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256412+0000", 2024-09-18T07:34:42.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256413+0000", 2024-09-18T07:34:42.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1900000000000002e-07 2024-09-18T07:34:42.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256415+0000", 2024-09-18T07:34:42.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4880000000000001e-06 2024-09-18T07:34:42.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256421+0000", 2024-09-18T07:34:42.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7019999999999998e-06 2024-09-18T07:34:42.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256426+0000", 2024-09-18T07:34:42.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1000000000000003e-06 2024-09-18T07:34:42.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282222+0000", 2024-09-18T07:34:42.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025796116000000001 2024-09-18T07:34:42.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282233+0000", 2024-09-18T07:34:42.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0539e-05 2024-09-18T07:34:42.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.215761+0000", 2024-09-18T07:34:42.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93352816100000002 2024-09-18T07:34:42.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:42.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.215857+0000", 2024-09-18T07:34:42.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6032999999999993e-05 2024-09-18T07:34:42.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:42.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.215960+0000", 2024-09-18T07:34:42.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103329 2024-09-18T07:34:42.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217112+0000", 2024-09-18T07:34:42.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011516709999999999 2024-09-18T07:34:42.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218052+0000", 2024-09-18T07:34:42.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00093980599999999995 2024-09-18T07:34:42.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:42.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220525+0000", 2024-09-18T07:34:42.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024736509999999999 2024-09-18T07:34:42.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:42.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220565+0000", 2024-09-18T07:34:42.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9847e-05 2024-09-18T07:34:42.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220604+0000", 2024-09-18T07:34:42.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9245000000000001e-05 2024-09-18T07:34:42.256 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.257 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.257 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.257 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151823 7.8 7:101e05c7:::benchmark_data_smithi137_74180_object47407:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-18T07:34:42.257 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.256432+0000", 2024-09-18T07:34:42.257 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.9632940080000001, 2024-09-18T07:34:42.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96425229099999998, 2024-09-18T07:34:42.257 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.257 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.257 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:42.257 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.257 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.257 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.258 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151823 2024-09-18T07:34:42.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.258 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256432+0000", 2024-09-18T07:34:42.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256432+0000", 2024-09-18T07:34:42.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256432+0000", 2024-09-18T07:34:42.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6199999999999998e-07 2024-09-18T07:34:42.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256435+0000", 2024-09-18T07:34:42.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4600000000000002e-06 2024-09-18T07:34:42.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256440+0000", 2024-09-18T07:34:42.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9830000000000004e-06 2024-09-18T07:34:42.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256444+0000", 2024-09-18T07:34:42.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6909999999999996e-06 2024-09-18T07:34:42.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282270+0000", 2024-09-18T07:34:42.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025825796000000002 2024-09-18T07:34:42.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282277+0000", 2024-09-18T07:34:42.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2989999999999999e-06 2024-09-18T07:34:42.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216194+0000", 2024-09-18T07:34:42.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93391638700000001 2024-09-18T07:34:42.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:42.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216288+0000", 2024-09-18T07:34:42.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3845999999999996e-05 2024-09-18T07:34:42.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:42.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216381+0000", 2024-09-18T07:34:42.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3960000000000002e-05 2024-09-18T07:34:42.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217934+0000", 2024-09-18T07:34:42.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001552257 2024-09-18T07:34:42.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218106+0000", 2024-09-18T07:34:42.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017274099999999999 2024-09-18T07:34:42.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:42.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220613+0000", 2024-09-18T07:34:42.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025060780000000001 2024-09-18T07:34:42.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:42.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220649+0000", 2024-09-18T07:34:42.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6380000000000001e-05 2024-09-18T07:34:42.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220684+0000", 2024-09-18T07:34:42.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4851000000000002e-05 2024-09-18T07:34:42.264 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.264 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.264 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.265 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151853 7.8 7:10bf2ca7:::benchmark_data_smithi137_74180_object47437:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-18T07:34:42.265 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.256449+0000", 2024-09-18T07:34:42.265 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.9632763500000001, 2024-09-18T07:34:42.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96431083500000003, 2024-09-18T07:34:42.265 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.265 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.265 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:42.265 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.265 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.265 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.265 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151853 2024-09-18T07:34:42.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.266 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256449+0000", 2024-09-18T07:34:42.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256449+0000", 2024-09-18T07:34:42.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256450+0000", 2024-09-18T07:34:42.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3799999999999998e-07 2024-09-18T07:34:42.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256452+0000", 2024-09-18T07:34:42.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.582e-06 2024-09-18T07:34:42.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256457+0000", 2024-09-18T07:34:42.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3029999999999999e-06 2024-09-18T07:34:42.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256463+0000", 2024-09-18T07:34:42.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0540000000000002e-06 2024-09-18T07:34:42.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282322+0000", 2024-09-18T07:34:42.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025859495999999999 2024-09-18T07:34:42.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282329+0000", 2024-09-18T07:34:42.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.391e-06 2024-09-18T07:34:42.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216585+0000", 2024-09-18T07:34:42.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93425595100000003 2024-09-18T07:34:42.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:42.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216677+0000", 2024-09-18T07:34:42.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1812999999999997e-05 2024-09-18T07:34:42.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:42.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216781+0000", 2024-09-18T07:34:42.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103607 2024-09-18T07:34:42.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217970+0000", 2024-09-18T07:34:42.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011890959999999999 2024-09-18T07:34:42.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218137+0000", 2024-09-18T07:34:42.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016691300000000001 2024-09-18T07:34:42.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:42.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220692+0000", 2024-09-18T07:34:42.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002555571 2024-09-18T07:34:42.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:42.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220729+0000", 2024-09-18T07:34:42.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6128999999999999e-05 2024-09-18T07:34:42.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.274 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:42.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220760+0000", 2024-09-18T07:34:42.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1491000000000001e-05 2024-09-18T07:34:42.275 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.276 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.276 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.276 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151969 7.8 7:10c7e652:::benchmark_data_smithi137_74180_object47553:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-18T07:34:42.276 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.256468+0000", 2024-09-18T07:34:42.276 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.9632578270000001, 2024-09-18T07:34:42.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96436882599999996, 2024-09-18T07:34:42.276 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.276 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.276 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:42.276 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.277 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.277 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.277 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151969 2024-09-18T07:34:42.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.277 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256468+0000", 2024-09-18T07:34:42.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256468+0000", 2024-09-18T07:34:42.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256468+0000", 2024-09-18T07:34:42.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9500000000000003e-07 2024-09-18T07:34:42.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256471+0000", 2024-09-18T07:34:42.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3410000000000001e-06 2024-09-18T07:34:42.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256476+0000", 2024-09-18T07:34:42.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3619999999999997e-06 2024-09-18T07:34:42.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256480+0000", 2024-09-18T07:34:42.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5879999999999999e-06 2024-09-18T07:34:42.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282362+0000", 2024-09-18T07:34:42.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025881686000000001 2024-09-18T07:34:42.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282370+0000", 2024-09-18T07:34:42.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0979999999999994e-06 2024-09-18T07:34:42.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217145+0000", 2024-09-18T07:34:42.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93477450600000001 2024-09-18T07:34:42.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:42.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217239+0000", 2024-09-18T07:34:42.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3744000000000002e-05 2024-09-18T07:34:42.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:42.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217328+0000", 2024-09-18T07:34:42.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9494999999999996e-05 2024-09-18T07:34:42.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218079+0000", 2024-09-18T07:34:42.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00075083099999999998 2024-09-18T07:34:42.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218172+0000", 2024-09-18T07:34:42.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2713999999999994e-05 2024-09-18T07:34:42.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:42.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220767+0000", 2024-09-18T07:34:42.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002595848 2024-09-18T07:34:42.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:42.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220804+0000", 2024-09-18T07:34:42.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6151e-05 2024-09-18T07:34:42.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220837+0000", 2024-09-18T07:34:42.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2966999999999998e-05 2024-09-18T07:34:42.283 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.283 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.283 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.283 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151980 7.8 7:1648d987:::benchmark_data_smithi137_74180_object47564:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-18T07:34:42.283 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.256485+0000", 2024-09-18T07:34:42.283 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.9632401430000002, 2024-09-18T07:34:42.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96460135300000005, 2024-09-18T07:34:42.284 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.284 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.284 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:42.284 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.284 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.284 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.284 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151980 2024-09-18T07:34:42.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.284 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256485+0000", 2024-09-18T07:34:42.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256485+0000", 2024-09-18T07:34:42.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256486+0000", 2024-09-18T07:34:42.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8299999999999997e-07 2024-09-18T07:34:42.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256488+0000", 2024-09-18T07:34:42.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1459999999999999e-06 2024-09-18T07:34:42.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256493+0000", 2024-09-18T07:34:42.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.147e-06 2024-09-18T07:34:42.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256498+0000", 2024-09-18T07:34:42.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6530000000000002e-06 2024-09-18T07:34:42.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282399+0000", 2024-09-18T07:34:42.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025901569999999999 2024-09-18T07:34:42.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282406+0000", 2024-09-18T07:34:42.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9380000000000003e-06 2024-09-18T07:34:42.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217538+0000", 2024-09-18T07:34:42.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93513140400000005 2024-09-18T07:34:42.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:42.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217632+0000", 2024-09-18T07:34:42.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4285999999999998e-05 2024-09-18T07:34:42.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:42.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217727+0000", 2024-09-18T07:34:42.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5168999999999996e-05 2024-09-18T07:34:42.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.219197+0000", 2024-09-18T07:34:42.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014697880000000001 2024-09-18T07:34:42.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.219258+0000", 2024-09-18T07:34:42.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1444000000000003e-05 2024-09-18T07:34:42.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:42.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221042+0000", 2024-09-18T07:34:42.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017834820000000001 2024-09-18T07:34:42.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:42.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221067+0000", 2024-09-18T07:34:42.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4714000000000001e-05 2024-09-18T07:34:42.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221087+0000", 2024-09-18T07:34:42.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0129e-05 2024-09-18T07:34:42.290 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.290 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.290 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.290 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151643 7.8 7:1d0c89a8:::benchmark_data_smithi137_74180_object47227:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e152)", 2024-09-18T07:34:42.290 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.282375+0000", 2024-09-18T07:34:42.290 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.937350358, 2024-09-18T07:34:42.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93875574799999995, 2024-09-18T07:34:42.291 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.291 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.291 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:42.291 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.291 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.291 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.291 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151643 2024-09-18T07:34:42.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.291 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282375+0000", 2024-09-18T07:34:42.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282375+0000", 2024-09-18T07:34:42.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282376+0000", 2024-09-18T07:34:42.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2500000000000004e-07 2024-09-18T07:34:42.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282380+0000", 2024-09-18T07:34:42.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4400000000000001e-06 2024-09-18T07:34:42.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282386+0000", 2024-09-18T07:34:42.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6540000000000002e-06 2024-09-18T07:34:42.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282392+0000", 2024-09-18T07:34:42.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7629999999999997e-06 2024-09-18T07:34:42.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282491+0000", 2024-09-18T07:34:42.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9512000000000005e-05 2024-09-18T07:34:42.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282500+0000", 2024-09-18T07:34:42.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1189999999999997e-06 2024-09-18T07:34:42.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218289+0000", 2024-09-18T07:34:42.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.935789659 2024-09-18T07:34:42.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:42.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218386+0000", 2024-09-18T07:34:42.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7065999999999995e-05 2024-09-18T07:34:42.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:42.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218494+0000", 2024-09-18T07:34:42.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000108119 2024-09-18T07:34:42.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.219300+0000", 2024-09-18T07:34:42.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00080511600000000003 2024-09-18T07:34:42.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.219441+0000", 2024-09-18T07:34:42.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014160900000000001 2024-09-18T07:34:42.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:42.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221094+0000", 2024-09-18T07:34:42.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016524160000000001 2024-09-18T07:34:42.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:42.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221114+0000", 2024-09-18T07:34:42.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0545000000000001e-05 2024-09-18T07:34:42.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221131+0000", 2024-09-18T07:34:42.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6804999999999999e-05 2024-09-18T07:34:42.297 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.297 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.297 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.298 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151668 7.8 7:194767be:::benchmark_data_smithi137_74180_object47252:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e152)", 2024-09-18T07:34:42.298 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.282400+0000", 2024-09-18T07:34:42.298 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.937325143, 2024-09-18T07:34:42.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93938683999999995, 2024-09-18T07:34:42.298 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.298 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.298 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:42.298 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.298 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.298 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.298 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151668 2024-09-18T07:34:42.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.299 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282400+0000", 2024-09-18T07:34:42.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282400+0000", 2024-09-18T07:34:42.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282401+0000", 2024-09-18T07:34:42.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7800000000000001e-07 2024-09-18T07:34:42.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282404+0000", 2024-09-18T07:34:42.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0960000000000001e-06 2024-09-18T07:34:42.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282410+0000", 2024-09-18T07:34:42.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5249999999999996e-06 2024-09-18T07:34:42.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282414+0000", 2024-09-18T07:34:42.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.707e-06 2024-09-18T07:34:42.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282657+0000", 2024-09-18T07:34:42.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000242585 2024-09-18T07:34:42.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282666+0000", 2024-09-18T07:34:42.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.755e-06 2024-09-18T07:34:42.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218720+0000", 2024-09-18T07:34:42.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93605417000000002 2024-09-18T07:34:42.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:42.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218826+0000", 2024-09-18T07:34:42.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010592 2024-09-18T07:34:42.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:42.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218926+0000", 2024-09-18T07:34:42.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010056 2024-09-18T07:34:42.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.219475+0000", 2024-09-18T07:34:42.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00054912799999999996 2024-09-18T07:34:42.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220375+0000", 2024-09-18T07:34:42.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00089989900000000001 2024-09-18T07:34:42.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:42.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221685+0000", 2024-09-18T07:34:42.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001309925 2024-09-18T07:34:42.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:42.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221751+0000", 2024-09-18T07:34:42.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5870000000000005e-05 2024-09-18T07:34:42.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221787+0000", 2024-09-18T07:34:42.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6121999999999997e-05 2024-09-18T07:34:42.304 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.304 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.305 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.305 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151733 7.8 7:181451fc:::benchmark_data_smithi137_74180_object47317:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e152)", 2024-09-18T07:34:42.305 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.282461+0000", 2024-09-18T07:34:42.305 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.9372641370000001, 2024-09-18T07:34:42.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.939403188, 2024-09-18T07:34:42.305 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.305 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.305 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:42.305 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.305 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.305 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.306 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151733 2024-09-18T07:34:42.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.306 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282461+0000", 2024-09-18T07:34:42.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282461+0000", 2024-09-18T07:34:42.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282462+0000", 2024-09-18T07:34:42.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7000000000000005e-07 2024-09-18T07:34:42.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282465+0000", 2024-09-18T07:34:42.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9699999999999999e-06 2024-09-18T07:34:42.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282473+0000", 2024-09-18T07:34:42.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2689999999999999e-06 2024-09-18T07:34:42.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282478+0000", 2024-09-18T07:34:42.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0679999999999996e-06 2024-09-18T07:34:42.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282750+0000", 2024-09-18T07:34:42.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027197099999999997 2024-09-18T07:34:42.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282758+0000", 2024-09-18T07:34:42.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0879999999999995e-06 2024-09-18T07:34:42.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.219504+0000", 2024-09-18T07:34:42.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93674544400000004 2024-09-18T07:34:42.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:42.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.219609+0000", 2024-09-18T07:34:42.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105011 2024-09-18T07:34:42.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:42.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.219713+0000", 2024-09-18T07:34:42.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103875 2024-09-18T07:34:42.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220964+0000", 2024-09-18T07:34:42.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001251528 2024-09-18T07:34:42.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221027+0000", 2024-09-18T07:34:42.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3009999999999995e-05 2024-09-18T07:34:42.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:42.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221805+0000", 2024-09-18T07:34:42.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00077785500000000004 2024-09-18T07:34:42.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:42.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221836+0000", 2024-09-18T07:34:42.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1211999999999999e-05 2024-09-18T07:34:42.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221865+0000", 2024-09-18T07:34:42.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8317e-05 2024-09-18T07:34:42.312 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.312 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.312 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.312 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151783 7.8 7:1e9a341c:::benchmark_data_smithi137_74180_object47367:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e152)", 2024-09-18T07:34:42.312 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.282518+0000", 2024-09-18T07:34:42.312 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.9372078849999999, 2024-09-18T07:34:42.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93944965199999997, 2024-09-18T07:34:42.312 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.312 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.312 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:42.312 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.312 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.313 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.313 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151783 2024-09-18T07:34:42.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.313 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282518+0000", 2024-09-18T07:34:42.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282518+0000", 2024-09-18T07:34:42.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282518+0000", 2024-09-18T07:34:42.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3e-07 2024-09-18T07:34:42.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282521+0000", 2024-09-18T07:34:42.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7889999999999999e-06 2024-09-18T07:34:42.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282526+0000", 2024-09-18T07:34:42.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.668e-06 2024-09-18T07:34:42.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282529+0000", 2024-09-18T07:34:42.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0299999999999998e-06 2024-09-18T07:34:42.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282835+0000", 2024-09-18T07:34:42.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000306244 2024-09-18T07:34:42.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282843+0000", 2024-09-18T07:34:42.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7279999999999995e-06 2024-09-18T07:34:42.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.219957+0000", 2024-09-18T07:34:42.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.937114259 2024-09-18T07:34:42.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:42.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220045+0000", 2024-09-18T07:34:42.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7683999999999996e-05 2024-09-18T07:34:42.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:42.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220143+0000", 2024-09-18T07:34:42.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8377999999999999e-05 2024-09-18T07:34:42.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221179+0000", 2024-09-18T07:34:42.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010355830000000001 2024-09-18T07:34:42.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221670+0000", 2024-09-18T07:34:42.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00049113200000000003 2024-09-18T07:34:42.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:42.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221892+0000", 2024-09-18T07:34:42.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022195099999999999 2024-09-18T07:34:42.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:42.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221925+0000", 2024-09-18T07:34:42.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.328e-05 2024-09-18T07:34:42.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221967+0000", 2024-09-18T07:34:42.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2196000000000001e-05 2024-09-18T07:34:42.320 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.320 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.320 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.321 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151840 7.8 7:18433c24:::benchmark_data_smithi137_74180_object47424:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e152)", 2024-09-18T07:34:42.321 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.282542+0000", 2024-09-18T07:34:42.321 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.937183052, 2024-09-18T07:34:42.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94386358999999997, 2024-09-18T07:34:42.321 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.321 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.321 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:42.321 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.321 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.321 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.321 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151840 2024-09-18T07:34:42.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.321 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282542+0000", 2024-09-18T07:34:42.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282542+0000", 2024-09-18T07:34:42.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282543+0000", 2024-09-18T07:34:42.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5400000000000002e-07 2024-09-18T07:34:42.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282544+0000", 2024-09-18T07:34:42.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.437e-06 2024-09-18T07:34:42.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282547+0000", 2024-09-18T07:34:42.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.553e-06 2024-09-18T07:34:42.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282549+0000", 2024-09-18T07:34:42.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1430000000000001e-06 2024-09-18T07:34:42.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282918+0000", 2024-09-18T07:34:42.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00036945100000000002 2024-09-18T07:34:42.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282926+0000", 2024-09-18T07:34:42.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8809999999999996e-06 2024-09-18T07:34:42.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221215+0000", 2024-09-18T07:34:42.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93828875 2024-09-18T07:34:42.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:42.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221315+0000", 2024-09-18T07:34:42.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9776000000000002e-05 2024-09-18T07:34:42.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:42.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221424+0000", 2024-09-18T07:34:42.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010908699999999999 2024-09-18T07:34:42.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.224065+0000", 2024-09-18T07:34:42.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026413000000000001 2024-09-18T07:34:42.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226072+0000", 2024-09-18T07:34:42.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020072229999999998 2024-09-18T07:34:42.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:42.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226296+0000", 2024-09-18T07:34:42.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022410999999999999 2024-09-18T07:34:42.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:42.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226363+0000", 2024-09-18T07:34:42.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6950000000000001e-05 2024-09-18T07:34:42.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226406+0000", 2024-09-18T07:34:42.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2675000000000001e-05 2024-09-18T07:34:42.327 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.327 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.328 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.328 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151853 7.8 7:10bf2ca7:::benchmark_data_smithi137_74180_object47437:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e152)", 2024-09-18T07:34:42.328 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.282551+0000", 2024-09-18T07:34:42.328 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.937174172, 2024-09-18T07:34:42.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93959601999999998, 2024-09-18T07:34:42.328 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.328 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.328 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:34:42.328 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.328 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.328 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.328 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151853 2024-09-18T07:34:42.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.329 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282551+0000", 2024-09-18T07:34:42.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282551+0000", 2024-09-18T07:34:42.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282552+0000", 2024-09-18T07:34:42.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7099999999999998e-07 2024-09-18T07:34:42.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282553+0000", 2024-09-18T07:34:42.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3e-06 2024-09-18T07:34:42.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282555+0000", 2024-09-18T07:34:42.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3929999999999998e-06 2024-09-18T07:34:42.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282558+0000", 2024-09-18T07:34:42.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4320000000000002e-06 2024-09-18T07:34:42.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282959+0000", 2024-09-18T07:34:42.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040164299999999998 2024-09-18T07:34:42.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282968+0000", 2024-09-18T07:34:42.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2339999999999994e-06 2024-09-18T07:34:42.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.222065+0000", 2024-09-18T07:34:42.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93909701400000001 2024-09-18T07:34:42.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.222147+0000", 2024-09-18T07:34:42.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2732999999999995e-05 2024-09-18T07:34:42.332 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.332 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.332 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.332 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151864 7.8 7:1a9263ed:::benchmark_data_smithi137_74180_object47448:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e152)", 2024-09-18T07:34:42.333 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.282560+0000", 2024-09-18T07:34:42.333 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.937165322, 2024-09-18T07:34:42.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94391931200000001, 2024-09-18T07:34:42.333 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.333 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.333 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:42.333 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.333 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.333 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.333 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151864 2024-09-18T07:34:42.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.333 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282560+0000", 2024-09-18T07:34:42.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282560+0000", 2024-09-18T07:34:42.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282560+0000", 2024-09-18T07:34:42.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5100000000000001e-07 2024-09-18T07:34:42.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282562+0000", 2024-09-18T07:34:42.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.285e-06 2024-09-18T07:34:42.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282564+0000", 2024-09-18T07:34:42.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.627e-06 2024-09-18T07:34:42.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282567+0000", 2024-09-18T07:34:42.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6400000000000001e-06 2024-09-18T07:34:42.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.283010+0000", 2024-09-18T07:34:42.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044346099999999997 2024-09-18T07:34:42.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.283019+0000", 2024-09-18T07:34:42.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2649999999999996e-06 2024-09-18T07:34:42.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.222174+0000", 2024-09-18T07:34:42.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.939155552 2024-09-18T07:34:42.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:42.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.222267+0000", 2024-09-18T07:34:42.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2880000000000005e-05 2024-09-18T07:34:42.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:42.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.222371+0000", 2024-09-18T07:34:42.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010398700000000001 2024-09-18T07:34:42.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226109+0000", 2024-09-18T07:34:42.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0037381110000000001 2024-09-18T07:34:42.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226142+0000", 2024-09-18T07:34:42.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2428999999999997e-05 2024-09-18T07:34:42.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:42.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226412+0000", 2024-09-18T07:34:42.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027067699999999998 2024-09-18T07:34:42.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:42.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226449+0000", 2024-09-18T07:34:42.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6545999999999999e-05 2024-09-18T07:34:42.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226480+0000", 2024-09-18T07:34:42.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0601000000000001e-05 2024-09-18T07:34:42.339 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.339 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.339 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.340 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151884 7.8 7:1a0e58f4:::benchmark_data_smithi137_74180_object47468:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e152)", 2024-09-18T07:34:42.340 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.282569+0000", 2024-09-18T07:34:42.340 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.9371561069999998, 2024-09-18T07:34:42.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94398025100000005, 2024-09-18T07:34:42.340 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.340 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.340 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:42.340 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.340 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.340 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.340 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151884 2024-09-18T07:34:42.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.341 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282569+0000", 2024-09-18T07:34:42.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282569+0000", 2024-09-18T07:34:42.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282570+0000", 2024-09-18T07:34:42.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5600000000000002e-07 2024-09-18T07:34:42.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282571+0000", 2024-09-18T07:34:42.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.31e-06 2024-09-18T07:34:42.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282574+0000", 2024-09-18T07:34:42.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5919999999999999e-06 2024-09-18T07:34:42.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282576+0000", 2024-09-18T07:34:42.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3609999999999999e-06 2024-09-18T07:34:42.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.283054+0000", 2024-09-18T07:34:42.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00047793 2024-09-18T07:34:42.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.283062+0000", 2024-09-18T07:34:42.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9100000000000005e-06 2024-09-18T07:34:42.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.222632+0000", 2024-09-18T07:34:42.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93957058400000004 2024-09-18T07:34:42.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:42.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.222749+0000", 2024-09-18T07:34:42.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116266 2024-09-18T07:34:42.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:42.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.222865+0000", 2024-09-18T07:34:42.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011679399999999999 2024-09-18T07:34:42.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226174+0000", 2024-09-18T07:34:42.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033085089999999998 2024-09-18T07:34:42.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226208+0000", 2024-09-18T07:34:42.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4322000000000001e-05 2024-09-18T07:34:42.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:42.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226484+0000", 2024-09-18T07:34:42.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027602399999999999 2024-09-18T07:34:42.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:42.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226519+0000", 2024-09-18T07:34:42.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4836000000000003e-05 2024-09-18T07:34:42.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226550+0000", 2024-09-18T07:34:42.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0556999999999999e-05 2024-09-18T07:34:42.347 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.347 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.347 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.347 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151916 7.8 7:1e1aab65:::benchmark_data_smithi137_74180_object47500:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e152)", 2024-09-18T07:34:42.347 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.282578+0000", 2024-09-18T07:34:42.347 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.937147137, 2024-09-18T07:34:42.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94526654399999999, 2024-09-18T07:34:42.347 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.347 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.347 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:42.347 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.348 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.348 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.348 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151916 2024-09-18T07:34:42.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.348 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282578+0000", 2024-09-18T07:34:42.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282578+0000", 2024-09-18T07:34:42.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282579+0000", 2024-09-18T07:34:42.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5400000000000002e-07 2024-09-18T07:34:42.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282580+0000", 2024-09-18T07:34:42.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.282e-06 2024-09-18T07:34:42.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282582+0000", 2024-09-18T07:34:42.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5110000000000002e-06 2024-09-18T07:34:42.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282585+0000", 2024-09-18T07:34:42.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8679999999999999e-06 2024-09-18T07:34:42.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.283096+0000", 2024-09-18T07:34:42.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00051036199999999999 2024-09-18T07:34:42.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.283104+0000", 2024-09-18T07:34:42.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9039999999999992e-06 2024-09-18T07:34:42.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.223105+0000", 2024-09-18T07:34:42.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.940000948 2024-09-18T07:34:42.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:42.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.223205+0000", 2024-09-18T07:34:42.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100352 2024-09-18T07:34:42.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:42.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.223305+0000", 2024-09-18T07:34:42.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100499 2024-09-18T07:34:42.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226241+0000", 2024-09-18T07:34:42.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002935493 2024-09-18T07:34:42.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:42.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226555+0000", 2024-09-18T07:34:42.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00031419899999999998 2024-09-18T07:34:42.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.227655+0000", 2024-09-18T07:34:42.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001099512 2024-09-18T07:34:42.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:42.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.227804+0000", 2024-09-18T07:34:42.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014929099999999999 2024-09-18T07:34:42.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.227845+0000", 2024-09-18T07:34:42.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1069000000000003e-05 2024-09-18T07:34:42.354 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.354 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.354 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.354 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151920 7.8 7:1e58d24b:::benchmark_data_smithi137_74180_object47504:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e152)", 2024-09-18T07:34:42.354 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.282588+0000", 2024-09-18T07:34:42.354 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.9371376979999999, 2024-09-18T07:34:42.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94541355999999999, 2024-09-18T07:34:42.355 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.355 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.355 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:42.355 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.355 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.355 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.355 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151920 2024-09-18T07:34:42.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.355 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282588+0000", 2024-09-18T07:34:42.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282588+0000", 2024-09-18T07:34:42.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282588+0000", 2024-09-18T07:34:42.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.48e-07 2024-09-18T07:34:42.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282589+0000", 2024-09-18T07:34:42.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2899999999999999e-06 2024-09-18T07:34:42.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282592+0000", 2024-09-18T07:34:42.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5509999999999998e-06 2024-09-18T07:34:42.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282594+0000", 2024-09-18T07:34:42.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3709999999999998e-06 2024-09-18T07:34:42.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.283138+0000", 2024-09-18T07:34:42.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00054404799999999999 2024-09-18T07:34:42.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.283146+0000", 2024-09-18T07:34:42.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8699999999999992e-06 2024-09-18T07:34:42.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.223507+0000", 2024-09-18T07:34:42.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94036082799999998 2024-09-18T07:34:42.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:42.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.223602+0000", 2024-09-18T07:34:42.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5105000000000005e-05 2024-09-18T07:34:42.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:34:42.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.223702+0000", 2024-09-18T07:34:42.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010012400000000001 2024-09-18T07:34:42.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226275+0000", 2024-09-18T07:34:42.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025725180000000002 2024-09-18T07:34:42.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:42.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.226560+0000", 2024-09-18T07:34:42.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028485400000000002 2024-09-18T07:34:42.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:42.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.227879+0000", 2024-09-18T07:34:42.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001319424 2024-09-18T07:34:42.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:42.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.227952+0000", 2024-09-18T07:34:42.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2844999999999998e-05 2024-09-18T07:34:42.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.228001+0000", 2024-09-18T07:34:42.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9484000000000003e-05 2024-09-18T07:34:42.361 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.361 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.361 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.362 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151969 7.8 7:10c7e652:::benchmark_data_smithi137_74180_object47553:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e152)", 2024-09-18T07:34:42.362 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.282597+0000", 2024-09-18T07:34:42.362 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.9371287609999999, 2024-09-18T07:34:42.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94135205300000002, 2024-09-18T07:34:42.362 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.362 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.362 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:34:42.362 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.362 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.362 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.362 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151969 2024-09-18T07:34:42.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.362 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282597+0000", 2024-09-18T07:34:42.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282597+0000", 2024-09-18T07:34:42.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282597+0000", 2024-09-18T07:34:42.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5899999999999998e-07 2024-09-18T07:34:42.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282598+0000", 2024-09-18T07:34:42.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3009999999999999e-06 2024-09-18T07:34:42.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282601+0000", 2024-09-18T07:34:42.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5909999999999998e-06 2024-09-18T07:34:42.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282603+0000", 2024-09-18T07:34:42.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5040000000000001e-06 2024-09-18T07:34:42.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.283181+0000", 2024-09-18T07:34:42.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00057724700000000002 2024-09-18T07:34:42.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.283189+0000", 2024-09-18T07:34:42.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0099999999999995e-06 2024-09-18T07:34:42.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.223891+0000", 2024-09-18T07:34:42.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94070240900000002 2024-09-18T07:34:42.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.223949+0000", 2024-09-18T07:34:42.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7732000000000002e-05 2024-09-18T07:34:42.366 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.366 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.366 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.366 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151980 7.8 7:1648d987:::benchmark_data_smithi137_74180_object47564:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e152)", 2024-09-18T07:34:42.367 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.282606+0000", 2024-09-18T07:34:42.367 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.9371194520000001, 2024-09-18T07:34:42.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94142903499999997, 2024-09-18T07:34:42.367 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:42.367 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:42.367 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:34:42.367 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:42.367 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:42.367 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:42.367 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151980 2024-09-18T07:34:42.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.367 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:42.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:42.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282606+0000", 2024-09-18T07:34:42.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:42.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282606+0000", 2024-09-18T07:34:42.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:42.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:42.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282606+0000", 2024-09-18T07:34:42.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5100000000000001e-07 2024-09-18T07:34:42.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:42.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282608+0000", 2024-09-18T07:34:42.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.331e-06 2024-09-18T07:34:42.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:42.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282610+0000", 2024-09-18T07:34:42.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.57e-06 2024-09-18T07:34:42.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:42.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282613+0000", 2024-09-18T07:34:42.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.328e-06 2024-09-18T07:34:42.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.283213+0000", 2024-09-18T07:34:42.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00060089599999999996 2024-09-18T07:34:42.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:42.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.283220+0000", 2024-09-18T07:34:42.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9800000000000001e-06 2024-09-18T07:34:42.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:42.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.223975+0000", 2024-09-18T07:34:42.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94075423599999997 2024-09-18T07:34:42.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:42.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:42.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:42.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.224035+0000", 2024-09-18T07:34:42.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0442999999999997e-05 2024-09-18T07:34:42.371 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.371 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.371 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.371 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:42.371 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:42.371 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:34:42.372 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:42.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:42.373 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:42.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:42.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:42 smithi137 ceph-mon[39175]: 7.d scrub starts 2024-09-18T07:34:42.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:42 smithi137 ceph-mon[39175]: 7.d scrub ok 2024-09-18T07:34:42.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:42 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:42 smithi137 ceph-mon[39175]: from='client.24997 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:34:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:42 smithi137 ceph-mon[39175]: osdmap e153: 8 total, 8 up, 7 in 2024-09-18T07:34:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:42 smithi137 ceph-mon[39175]: pgmap v181: 156 pgs: 1 peering, 155 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 188 op/s 2024-09-18T07:34:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:42 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"} : dispatch 2024-09-18T07:34:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:42 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"} : dispatch 2024-09-18T07:34:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:42 smithi137 ceph-mon[29408]: 7.d scrub starts 2024-09-18T07:34:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:42 smithi137 ceph-mon[29408]: 7.d scrub ok 2024-09-18T07:34:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:42 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:42 smithi137 ceph-mon[29408]: from='client.24997 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:34:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:42 smithi137 ceph-mon[29408]: osdmap e153: 8 total, 8 up, 7 in 2024-09-18T07:34:42.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:42 smithi137 ceph-mon[29408]: pgmap v181: 156 pgs: 1 peering, 155 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 188 op/s 2024-09-18T07:34:42.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:42 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"} : dispatch 2024-09-18T07:34:42.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:42 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"} : dispatch 2024-09-18T07:34:42.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:42 smithi203 ceph-mon[32237]: 7.d scrub starts 2024-09-18T07:34:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:42 smithi203 ceph-mon[32237]: 7.d scrub ok 2024-09-18T07:34:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:42 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:42 smithi203 ceph-mon[32237]: from='client.24997 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:34:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:42 smithi203 ceph-mon[32237]: osdmap e153: 8 total, 8 up, 7 in 2024-09-18T07:34:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:42 smithi203 ceph-mon[32237]: pgmap v181: 156 pgs: 1 peering, 155 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 188 op/s 2024-09-18T07:34:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:42 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"} : dispatch 2024-09-18T07:34:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:42 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"} : dispatch 2024-09-18T07:34:42.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:42.496 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:42.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:42.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:42.620 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:42.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:42.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:42.722 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:42.743 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:42.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:42.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:42.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:42.866 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:42.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:42.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:42.968 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:42.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:42.990 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:42.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:43.092 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:43.113 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:43.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:43.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:43.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:43.238 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:43.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:43.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:43.340 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:43.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:43.364 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:43.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:43 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"}]': finished 2024-09-18T07:34:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:43 smithi137 ceph-mon[29408]: osdmap e154: 8 total, 8 up, 7 in 2024-09-18T07:34:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:43 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:34:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:43 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:34:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:43 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:43 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:43 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"}]': finished 2024-09-18T07:34:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:43 smithi137 ceph-mon[39175]: osdmap e154: 8 total, 8 up, 7 in 2024-09-18T07:34:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:43 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:34:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:43 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:34:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:43 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:43 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:43.465 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:43.486 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:43.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:43.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:43.588 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:43.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:43.612 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:43.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:43.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:43 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"}]': finished 2024-09-18T07:34:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:43 smithi203 ceph-mon[32237]: osdmap e154: 8 total, 8 up, 7 in 2024-09-18T07:34:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:43 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:34:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:43 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:34:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:43 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:43 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:43.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:43.733 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:43.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:43.835 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:43.853 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:34:43.853 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:34:43.857 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:43.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:43.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:43.870 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:34:43.871 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:34:43.887 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:34:43.887 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:34:43.905 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:34:43.905 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:34:43.920 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:34:43.920 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:34:43.937 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:34:43.937 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:34:43.954 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:34:43.955 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:34:43.959 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:43.971 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:34:43.972 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:34:43.981 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:43.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:44.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:44.105 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:44.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:44.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:44.206 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:44.230 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:44.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:44.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:44.319 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:34:44.332 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:44.353 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:44.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:44.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:44.455 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:44.476 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:44.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:44.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:44.578 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:44.600 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:44.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:44.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:44 smithi137 ceph-mon[29408]: osdmap e155: 8 total, 8 up, 7 in 2024-09-18T07:34:44.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:44 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:34:44.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:44 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:34:44.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:44 smithi137 ceph-mon[29408]: pgmap v184: 156 pgs: 156 active+clean; 3.4 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 665 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:34:44.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:44 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3873838082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:44.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:44 smithi137 ceph-mon[39175]: osdmap e155: 8 total, 8 up, 7 in 2024-09-18T07:34:44.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:44 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:34:44.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:44 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:34:44.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:44 smithi137 ceph-mon[39175]: pgmap v184: 156 pgs: 156 active+clean; 3.4 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 665 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:34:44.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3873838082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:44.702 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:44.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:44 smithi203 ceph-mon[32237]: osdmap e155: 8 total, 8 up, 7 in 2024-09-18T07:34:44.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:44 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:34:44.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:44 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:34:44.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:44 smithi203 ceph-mon[32237]: pgmap v184: 156 pgs: 156 active+clean; 3.4 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 665 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:34:44.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:44 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3873838082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:44.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T07:34:44.724 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:44.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:44.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:44.826 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:44.849 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:44.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:44.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:44.907 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:44.952 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:44.978 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:44.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:44.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:45.080 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:45.113 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:45.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:45.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:45.215 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:45.239 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:45.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:45.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:45.341 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:45.343 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:45 smithi137 ceph-mon[29408]: osdmap e156: 8 total, 8 up, 7 in 2024-09-18T07:34:45.343 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:45 smithi137 ceph-mon[29408]: osdmap e157: 8 total, 8 up, 7 in 2024-09-18T07:34:45.344 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:45 smithi137 ceph-mon[39175]: osdmap e156: 8 total, 8 up, 7 in 2024-09-18T07:34:45.344 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:45 smithi137 ceph-mon[39175]: osdmap e157: 8 total, 8 up, 7 in 2024-09-18T07:34:45.363 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:45.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:45.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:45.465 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:45.490 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:45.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:45.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:45.592 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:45.617 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:45.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:45.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:45.664 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:45.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:45 smithi203 ceph-mon[32237]: osdmap e156: 8 total, 8 up, 7 in 2024-09-18T07:34:45.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:45 smithi203 ceph-mon[32237]: osdmap e157: 8 total, 8 up, 7 in 2024-09-18T07:34:45.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:45.777 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:45.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:45.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:45.879 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:45.903 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:45.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:45.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:46.005 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:46.043 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:46.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:46.144 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:46.167 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:46.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:46.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:46.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:46.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:46.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:46.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:46.379 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:34:46.382 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:34:46.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:46.442 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:46.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:46.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:46.477 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:46 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:34:46.477 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:46 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:34:46.478 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:46 smithi137 ceph-mon[29408]: pgmap v187: 155 pgs: 155 active+clean; 3.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 447 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:34:46.478 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:46 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"} : dispatch 2024-09-18T07:34:46.478 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:46 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"} : dispatch 2024-09-18T07:34:46.478 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:46 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:34:46.478 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:46 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:34:46.478 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:46 smithi137 ceph-mon[39175]: pgmap v187: 155 pgs: 155 active+clean; 3.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 447 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:34:46.478 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:46 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"} : dispatch 2024-09-18T07:34:46.478 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:46 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"} : dispatch 2024-09-18T07:34:46.544 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:46.565 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:46.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:46.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:46.666 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:46.688 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:46.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:46.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:46.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:46 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:34:46.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:46 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:34:46.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:46 smithi203 ceph-mon[32237]: pgmap v187: 155 pgs: 155 active+clean; 3.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 447 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:34:46.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:46 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"} : dispatch 2024-09-18T07:34:46.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:46 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"} : dispatch 2024-09-18T07:34:46.751 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:46.790 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:46.812 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:46.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:46.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:46.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:46.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:46.936 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:46.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:47.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:47.059 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:47.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:47.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:47.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:47.183 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:47.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:47.187 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-18T07:34:46.262905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3578796595,"num_objects":54799,"num_object_clones":0,"num_object_copies":164397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54799,"num_whiteouts":0,"num_read":962,"num_read_kb":821,"num_write":60412,"num_write_kb":2501258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3953,"num_bytes_recovered":253562353,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109476,"ondisk_log_size":109476,"up":465,"acting":465,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":475,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":11883588,"kb_used_data":11012704,"kb_used_omap":10,"kb_used_meta":870645,"kb_avail":644332476,"statfs":{"total":671965249536,"available":659796455424,"internally_reserved":0,"allocated":11277008896,"data_stored":11273629846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11162,"internal_metadata":891540582},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-110624768,"num_objects":-1688,"num_object_clones":0,"num_object_copies":-5064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-120,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.043778"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":204,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.237487+0000","last_change":"2024-09-18T07:33:31.183001+0000","last_active":"2024-09-18T07:34:41.237487+0000","last_peered":"2024-09-18T07:34:41.237487+0000","last_clean":"2024-09-18T07:34:41.237487+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:34:41.237487+0000","last_undegraded":"2024-09-18T07:34:41.237487+0000","last_fullsized":"2024-09-18T07:34:41.237487+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:31.182913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:31.182913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:44.795464+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":234,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210588+0000","last_change":"2024-09-18T07:33:29.685825+0000","last_active":"2024-09-18T07:34:41.210588+0000","last_peered":"2024-09-18T07:34:41.210588+0000","last_clean":"2024-09-18T07:34:41.210588+0000","last_became_active":"2024-09-18T07:32:26.785727+0000","last_became_peered":"2024-09-18T07:32:26.785727+0000","last_unstale":"2024-09-18T07:34:41.210588+0000","last_undegraded":"2024-09-18T07:34:41.210588+0000","last_fullsized":"2024-09-18T07:34:41.210588+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:33:29.685767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.685767+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":212,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.208771+0000","last_change":"2024-09-18T07:33:57.894855+0000","last_active":"2024-09-18T07:34:41.208771+0000","last_peered":"2024-09-18T07:34:41.208771+0000","last_clean":"2024-09-18T07:34:41.208771+0000","last_became_active":"2024-09-18T07:32:30.831949+0000","last_became_peered":"2024-09-18T07:32:30.831949+0000","last_unstale":"2024-09-18T07:34:41.208771+0000","last_undegraded":"2024-09-18T07:34:41.208771+0000","last_fullsized":"2024-09-18T07:34:41.208771+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:57.894611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.894611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:07.665469+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":193,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210493+0000","last_change":"2024-09-18T07:32:03.953881+0000","last_active":"2024-09-18T07:34:41.210493+0000","last_peered":"2024-09-18T07:34:41.210493+0000","last_clean":"2024-09-18T07:34:41.210493+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:34:41.210493+0000","last_undegraded":"2024-09-18T07:34:41.210493+0000","last_fullsized":"2024-09-18T07:34:41.210493+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.953827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.953827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":78,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.236777+0000","last_change":"2024-09-18T07:33:27.004713+0000","last_active":"2024-09-18T07:34:41.236777+0000","last_peered":"2024-09-18T07:34:41.236777+0000","last_clean":"2024-09-18T07:34:41.236777+0000","last_became_active":"2024-09-18T07:33:27.004385+0000","last_became_peered":"2024-09-18T07:33:27.004385+0000","last_unstale":"2024-09-18T07:34:41.236777+0000","last_undegraded":"2024-09-18T07:34:41.236777+0000","last_fullsized":"2024-09-18T07:34:41.236777+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":204,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.213777+0000","last_change":"2024-09-18T07:33:27.006545+0000","last_active":"2024-09-18T07:34:41.213777+0000","last_peered":"2024-09-18T07:34:41.213777+0000","last_clean":"2024-09-18T07:34:41.213777+0000","last_became_active":"2024-09-18T07:30:56.286040+0000","last_became_peered":"2024-09-18T07:30:56.286040+0000","last_unstale":"2024-09-18T07:34:41.213777+0000","last_undegraded":"2024-09-18T07:34:41.213777+0000","last_fullsized":"2024-09-18T07:34:41.213777+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.006488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.006488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.614689+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":225,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211917+0000","last_change":"2024-09-18T07:33:22.032321+0000","last_active":"2024-09-18T07:34:41.211917+0000","last_peered":"2024-09-18T07:34:41.211917+0000","last_clean":"2024-09-18T07:34:41.211917+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:34:41.211917+0000","last_undegraded":"2024-09-18T07:34:41.211917+0000","last_fullsized":"2024-09-18T07:34:41.211917+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:22.032110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.032110+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.263345+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":197,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210485+0000","last_change":"2024-09-18T07:33:59.829707+0000","last_active":"2024-09-18T07:34:41.210485+0000","last_peered":"2024-09-18T07:34:41.210485+0000","last_clean":"2024-09-18T07:34:41.210485+0000","last_became_active":"2024-09-18T07:31:00.331224+0000","last_became_peered":"2024-09-18T07:31:00.331224+0000","last_unstale":"2024-09-18T07:34:41.210485+0000","last_undegraded":"2024-09-18T07:34:41.210485+0000","last_fullsized":"2024-09-18T07:34:41.210485+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:59.829644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:59.829644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:12.614654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":204,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.222199+0000","last_change":"2024-09-18T07:33:21.338716+0000","last_active":"2024-09-18T07:34:41.222199+0000","last_peered":"2024-09-18T07:34:41.222199+0000","last_clean":"2024-09-18T07:34:41.222199+0000","last_became_active":"2024-09-18T07:30:56.285882+0000","last_became_peered":"2024-09-18T07:30:56.285882+0000","last_unstale":"2024-09-18T07:34:41.222199+0000","last_undegraded":"2024-09-18T07:34:41.222199+0000","last_fullsized":"2024-09-18T07:34:41.222199+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.338669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.338669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":217,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210781+0000","last_change":"2024-09-18T07:32:02.002919+0000","last_active":"2024-09-18T07:34:41.210781+0000","last_peered":"2024-09-18T07:34:41.210781+0000","last_clean":"2024-09-18T07:34:41.210781+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:34:41.210781+0000","last_undegraded":"2024-09-18T07:34:41.210781+0000","last_fullsized":"2024-09-18T07:34:41.210781+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.002811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.002811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":195,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.220299+0000","last_change":"2024-09-18T07:32:30.842423+0000","last_active":"2024-09-18T07:34:41.220299+0000","last_peered":"2024-09-18T07:34:41.220299+0000","last_clean":"2024-09-18T07:34:41.220299+0000","last_became_active":"2024-09-18T07:32:30.841963+0000","last_became_peered":"2024-09-18T07:32:30.841963+0000","last_unstale":"2024-09-18T07:34:41.220299+0000","last_undegraded":"2024-09-18T07:34:41.220299+0000","last_fullsized":"2024-09-18T07:34:41.220299+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":191,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.233998+0000","last_change":"2024-09-18T07:32:30.840094+0000","last_active":"2024-09-18T07:34:41.233998+0000","last_peered":"2024-09-18T07:34:41.233998+0000","last_clean":"2024-09-18T07:34:41.233998+0000","last_became_active":"2024-09-18T07:32:30.839724+0000","last_became_peered":"2024-09-18T07:32:30.839724+0000","last_unstale":"2024-09-18T07:34:41.233998+0000","last_undegraded":"2024-09-18T07:34:41.233998+0000","last_fullsized":"2024-09-18T07:34:41.233998+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":196,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.221570+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:34:41.221570+0000","last_peered":"2024-09-18T07:34:41.221570+0000","last_clean":"2024-09-18T07:34:41.221570+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:34:41.221570+0000","last_undegraded":"2024-09-18T07:34:41.221570+0000","last_fullsized":"2024-09-18T07:34:41.221570+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:17.419087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":203,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211034+0000","last_change":"2024-09-18T07:30:58.317041+0000","last_active":"2024-09-18T07:34:41.211034+0000","last_peered":"2024-09-18T07:34:41.211034+0000","last_clean":"2024-09-18T07:34:41.211034+0000","last_became_active":"2024-09-18T07:30:58.316479+0000","last_became_peered":"2024-09-18T07:30:58.316479+0000","last_unstale":"2024-09-18T07:34:41.211034+0000","last_undegraded":"2024-09-18T07:34:41.211034+0000","last_fullsized":"2024-09-18T07:34:41.211034+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:41.690380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":131,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210456+0000","last_change":"2024-09-18T07:32:30.834493+0000","last_active":"2024-09-18T07:34:41.210456+0000","last_peered":"2024-09-18T07:34:41.210456+0000","last_clean":"2024-09-18T07:34:41.210456+0000","last_became_active":"2024-09-18T07:32:30.834192+0000","last_became_peered":"2024-09-18T07:32:30.834192+0000","last_unstale":"2024-09-18T07:34:41.210456+0000","last_undegraded":"2024-09-18T07:34:41.210456+0000","last_fullsized":"2024-09-18T07:34:41.210456+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.897464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":200,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.219901+0000","last_change":"2024-09-18T07:33:51.759330+0000","last_active":"2024-09-18T07:34:41.219901+0000","last_peered":"2024-09-18T07:34:41.219901+0000","last_clean":"2024-09-18T07:34:41.219901+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:34:41.219901+0000","last_undegraded":"2024-09-18T07:34:41.219901+0000","last_fullsized":"2024-09-18T07:34:41.219901+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:51.759259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:51.759259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:18.399187+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":193,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211334+0000","last_change":"2024-09-18T07:32:06.976159+0000","last_active":"2024-09-18T07:34:41.211334+0000","last_peered":"2024-09-18T07:34:41.211334+0000","last_clean":"2024-09-18T07:34:41.211334+0000","last_became_active":"2024-09-18T07:31:02.490014+0000","last_became_peered":"2024-09-18T07:31:02.490014+0000","last_unstale":"2024-09-18T07:34:41.211334+0000","last_undegraded":"2024-09-18T07:34:41.211334+0000","last_fullsized":"2024-09-18T07:34:41.211334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:06.976113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:06.976113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":211,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.237387+0000","last_change":"2024-09-18T07:33:53.330384+0000","last_active":"2024-09-18T07:34:41.237387+0000","last_peered":"2024-09-18T07:34:41.237387+0000","last_clean":"2024-09-18T07:34:41.237387+0000","last_became_active":"2024-09-18T07:32:30.839952+0000","last_became_peered":"2024-09-18T07:32:30.839952+0000","last_unstale":"2024-09-18T07:34:41.237387+0000","last_undegraded":"2024-09-18T07:34:41.237387+0000","last_fullsized":"2024-09-18T07:34:41.237387+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:53.330304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:53.330304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:58.692089+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":234,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210855+0000","last_change":"2024-09-18T07:33:59.921943+0000","last_active":"2024-09-18T07:34:41.210855+0000","last_peered":"2024-09-18T07:34:41.210855+0000","last_clean":"2024-09-18T07:34:41.210855+0000","last_became_active":"2024-09-18T07:30:58.316266+0000","last_became_peered":"2024-09-18T07:30:58.316266+0000","last_unstale":"2024-09-18T07:34:41.210855+0000","last_undegraded":"2024-09-18T07:34:41.210855+0000","last_fullsized":"2024-09-18T07:34:41.210855+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:59.921856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:59.921856+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:28.827554+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":197,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.209932+0000","last_change":"2024-09-18T07:33:30.955441+0000","last_active":"2024-09-18T07:34:41.209932+0000","last_peered":"2024-09-18T07:34:41.209932+0000","last_clean":"2024-09-18T07:34:41.209932+0000","last_became_active":"2024-09-18T07:31:00.337474+0000","last_became_peered":"2024-09-18T07:31:00.337474+0000","last_unstale":"2024-09-18T07:34:41.209932+0000","last_undegraded":"2024-09-18T07:34:41.209932+0000","last_fullsized":"2024-09-18T07:34:41.209932+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.955213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.955213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.470164+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.213621+0000","last_change":"2024-09-18T07:33:26.004195+0000","last_active":"2024-09-18T07:34:41.213621+0000","last_peered":"2024-09-18T07:34:41.213621+0000","last_clean":"2024-09-18T07:34:41.213621+0000","last_became_active":"2024-09-18T07:31:02.492348+0000","last_became_peered":"2024-09-18T07:31:02.492348+0000","last_unstale":"2024-09-18T07:34:41.213621+0000","last_undegraded":"2024-09-18T07:34:41.213621+0000","last_fullsized":"2024-09-18T07:34:41.213621+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:26.003425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:26.003425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":205,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210871+0000","last_change":"2024-09-18T07:33:53.756188+0000","last_active":"2024-09-18T07:34:41.210871+0000","last_peered":"2024-09-18T07:34:41.210871+0000","last_clean":"2024-09-18T07:34:41.210871+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:34:41.210871+0000","last_undegraded":"2024-09-18T07:34:41.210871+0000","last_fullsized":"2024-09-18T07:34:41.210871+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:53.756119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:53.756119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:19.988286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":215,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.213333+0000","last_change":"2024-09-18T07:30:58.319115+0000","last_active":"2024-09-18T07:34:41.213333+0000","last_peered":"2024-09-18T07:34:41.213333+0000","last_clean":"2024-09-18T07:34:41.213333+0000","last_became_active":"2024-09-18T07:30:58.316380+0000","last_became_peered":"2024-09-18T07:30:58.316380+0000","last_unstale":"2024-09-18T07:34:41.213333+0000","last_undegraded":"2024-09-18T07:34:41.213333+0000","last_fullsized":"2024-09-18T07:34:41.213333+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":189,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211102+0000","last_change":"2024-09-18T07:31:00.334313+0000","last_active":"2024-09-18T07:34:41.211102+0000","last_peered":"2024-09-18T07:34:41.211102+0000","last_clean":"2024-09-18T07:34:41.211102+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:34:41.211102+0000","last_undegraded":"2024-09-18T07:34:41.211102+0000","last_fullsized":"2024-09-18T07:34:41.211102+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":155,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.220244+0000","last_change":"2024-09-18T07:33:33.565391+0000","last_active":"2024-09-18T07:34:41.220244+0000","last_peered":"2024-09-18T07:34:41.220244+0000","last_clean":"2024-09-18T07:34:41.220244+0000","last_became_active":"2024-09-18T07:32:30.840386+0000","last_became_peered":"2024-09-18T07:32:30.840386+0000","last_unstale":"2024-09-18T07:34:41.220244+0000","last_undegraded":"2024-09-18T07:34:41.220244+0000","last_fullsized":"2024-09-18T07:34:41.220244+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:33.565334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.565334+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:02.676072+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":214,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.208900+0000","last_change":"2024-09-18T07:33:32.976203+0000","last_active":"2024-09-18T07:34:41.208900+0000","last_peered":"2024-09-18T07:34:41.208900+0000","last_clean":"2024-09-18T07:34:41.208900+0000","last_became_active":"2024-09-18T07:30:56.283705+0000","last_became_peered":"2024-09-18T07:30:56.283705+0000","last_unstale":"2024-09-18T07:34:41.208900+0000","last_undegraded":"2024-09-18T07:34:41.208900+0000","last_fullsized":"2024-09-18T07:34:41.208900+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:32.976129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.976129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:54.706133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":234,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.237619+0000","last_change":"2024-09-18T07:33:36.118423+0000","last_active":"2024-09-18T07:34:41.237619+0000","last_peered":"2024-09-18T07:34:41.237619+0000","last_clean":"2024-09-18T07:34:41.237619+0000","last_became_active":"2024-09-18T07:32:30.860930+0000","last_became_peered":"2024-09-18T07:32:30.860930+0000","last_unstale":"2024-09-18T07:34:41.237619+0000","last_undegraded":"2024-09-18T07:34:41.237619+0000","last_fullsized":"2024-09-18T07:34:41.237619+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:33:36.118340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.118340+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"94'9","reported_seq":264,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.224184+0000","last_change":"2024-09-18T07:33:59.647674+0000","last_active":"2024-09-18T07:34:41.224184+0000","last_peered":"2024-09-18T07:34:41.224184+0000","last_clean":"2024-09-18T07:34:41.224184+0000","last_became_active":"2024-09-18T07:32:30.839828+0000","last_became_peered":"2024-09-18T07:32:30.839828+0000","last_unstale":"2024-09-18T07:34:41.224184+0000","last_undegraded":"2024-09-18T07:34:41.224184+0000","last_fullsized":"2024-09-18T07:34:41.224184+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-09-18T07:33:59.647617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:59.647617+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:03.386170+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"153'9497","reported_seq":13614,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.434230+0000","last_change":"2024-09-18T07:33:32.018562+0000","last_active":"2024-09-18T07:34:41.434230+0000","last_peered":"2024-09-18T07:34:41.434230+0000","last_clean":"2024-09-18T07:34:41.434230+0000","last_became_active":"2024-09-18T07:32:54.163740+0000","last_became_peered":"2024-09-18T07:32:54.163740+0000","last_unstale":"2024-09-18T07:34:41.434230+0000","last_undegraded":"2024-09-18T07:34:41.434230+0000","last_fullsized":"2024-09-18T07:34:41.434230+0000","mapping_epoch":102,"log_start":"113'4112","ondisk_log_start":"113'4112","created":81,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'5666","last_scrub_stamp":"2024-09-18T07:33:32.018370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.018370+0000","objects_scrubbed":5641,"log_size":5385,"log_dups_size":0,"ondisk_log_size":5385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:53.849853+0000","scrub_duration":1289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227868672,"num_objects":3477,"num_object_clones":0,"num_object_copies":10431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9497,"num_write_kb":415168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":127074304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":226,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.209786+0000","last_change":"2024-09-18T07:34:37.067259+0000","last_active":"2024-09-18T07:34:41.209786+0000","last_peered":"2024-09-18T07:34:41.209786+0000","last_clean":"2024-09-18T07:34:41.209786+0000","last_became_active":"2024-09-18T07:30:58.312627+0000","last_became_peered":"2024-09-18T07:30:58.312627+0000","last_unstale":"2024-09-18T07:34:41.209786+0000","last_undegraded":"2024-09-18T07:34:41.209786+0000","last_fullsized":"2024-09-18T07:34:41.209786+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:34:37.067162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:37.067162+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:52.474545+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":219,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.212935+0000","last_change":"2024-09-18T07:34:01.294683+0000","last_active":"2024-09-18T07:34:41.212935+0000","last_peered":"2024-09-18T07:34:41.212935+0000","last_clean":"2024-09-18T07:34:41.212935+0000","last_became_active":"2024-09-18T07:32:30.838348+0000","last_became_peered":"2024-09-18T07:32:30.838348+0000","last_unstale":"2024-09-18T07:34:41.212935+0000","last_undegraded":"2024-09-18T07:34:41.212935+0000","last_fullsized":"2024-09-18T07:34:41.212935+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:01.294615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:01.294615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:04.638687+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":251,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211495+0000","last_change":"2024-09-18T07:33:33.706253+0000","last_active":"2024-09-18T07:34:41.211495+0000","last_peered":"2024-09-18T07:34:41.211495+0000","last_clean":"2024-09-18T07:34:41.211495+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:34:41.211495+0000","last_undegraded":"2024-09-18T07:34:41.211495+0000","last_fullsized":"2024-09-18T07:34:41.211495+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:33:33.706201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.706201+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.221578+0000","last_change":"2024-09-18T07:34:40.386158+0000","last_active":"2024-09-18T07:34:41.221578+0000","last_peered":"2024-09-18T07:34:41.221578+0000","last_clean":"2024-09-18T07:34:41.221578+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:34:41.221578+0000","last_undegraded":"2024-09-18T07:34:41.221578+0000","last_fullsized":"2024-09-18T07:34:41.221578+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:40.386112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:34:40.386112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:03.303185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"153'8781","reported_seq":8986,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.551411+0000","last_change":"2024-09-18T07:34:29.885449+0000","last_active":"2024-09-18T07:34:41.551411+0000","last_peered":"2024-09-18T07:34:41.551411+0000","last_clean":"2024-09-18T07:34:41.551411+0000","last_became_active":"2024-09-18T07:34:15.007659+0000","last_became_peered":"2024-09-18T07:34:15.007659+0000","last_unstale":"2024-09-18T07:34:41.551411+0000","last_undegraded":"2024-09-18T07:34:41.551411+0000","last_fullsized":"2024-09-18T07:34:41.551411+0000","mapping_epoch":81,"log_start":"113'4000","ondisk_log_start":"113'4000","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'8619","last_scrub_stamp":"2024-09-18T07:34:29.885340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:29.885340+0000","objects_scrubbed":1910,"log_size":4781,"log_dups_size":0,"ondisk_log_size":4781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:41.703999+0000","scrub_duration":623,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113573888,"num_objects":1733,"num_object_clones":0,"num_object_copies":5199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":257,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.212701+0000","last_change":"2024-09-18T07:32:30.876643+0000","last_active":"2024-09-18T07:34:41.212701+0000","last_peered":"2024-09-18T07:34:41.212701+0000","last_clean":"2024-09-18T07:34:41.212701+0000","last_became_active":"2024-09-18T07:32:30.852501+0000","last_became_peered":"2024-09-18T07:32:30.852501+0000","last_unstale":"2024-09-18T07:34:41.212701+0000","last_undegraded":"2024-09-18T07:34:41.212701+0000","last_fullsized":"2024-09-18T07:34:41.212701+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:32:00.660377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:00.660377+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":211,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.213833+0000","last_change":"2024-09-18T07:32:30.858438+0000","last_active":"2024-09-18T07:34:41.213833+0000","last_peered":"2024-09-18T07:34:41.213833+0000","last_clean":"2024-09-18T07:34:41.213833+0000","last_became_active":"2024-09-18T07:32:30.858176+0000","last_became_peered":"2024-09-18T07:32:30.858176+0000","last_unstale":"2024-09-18T07:34:41.213833+0000","last_undegraded":"2024-09-18T07:34:41.213833+0000","last_fullsized":"2024-09-18T07:34:41.213833+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.665591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.665591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":196,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210487+0000","last_change":"2024-09-18T07:32:30.851490+0000","last_active":"2024-09-18T07:34:41.210487+0000","last_peered":"2024-09-18T07:34:41.210487+0000","last_clean":"2024-09-18T07:34:41.210487+0000","last_became_active":"2024-09-18T07:32:30.851161+0000","last_became_peered":"2024-09-18T07:32:30.851161+0000","last_unstale":"2024-09-18T07:34:41.210487+0000","last_undegraded":"2024-09-18T07:34:41.210487+0000","last_fullsized":"2024-09-18T07:34:41.210487+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:47.861786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":185,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210758+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:34:41.210758+0000","last_peered":"2024-09-18T07:34:41.210758+0000","last_clean":"2024-09-18T07:34:41.210758+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:34:41.210758+0000","last_undegraded":"2024-09-18T07:34:41.210758+0000","last_fullsized":"2024-09-18T07:34:41.210758+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:12.807504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"153'8922","reported_seq":9129,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.550295+0000","last_change":"2024-09-18T07:34:30.492135+0000","last_active":"2024-09-18T07:34:41.550295+0000","last_peered":"2024-09-18T07:34:41.550295+0000","last_clean":"2024-09-18T07:34:41.550295+0000","last_became_active":"2024-09-18T07:34:14.996533+0000","last_became_peered":"2024-09-18T07:34:14.996533+0000","last_unstale":"2024-09-18T07:34:41.550295+0000","last_undegraded":"2024-09-18T07:34:41.550295+0000","last_fullsized":"2024-09-18T07:34:41.550295+0000","mapping_epoch":81,"log_start":"113'4100","ondisk_log_start":"113'4100","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'8772","last_scrub_stamp":"2024-09-18T07:34:30.492072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:30.492072+0000","objects_scrubbed":1900,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:53.920306+0000","scrub_duration":228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114688000,"num_objects":1750,"num_object_clones":0,"num_object_copies":5250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":214,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210517+0000","last_change":"2024-09-18T07:30:58.309938+0000","last_active":"2024-09-18T07:34:41.210517+0000","last_peered":"2024-09-18T07:34:41.210517+0000","last_clean":"2024-09-18T07:34:41.210517+0000","last_became_active":"2024-09-18T07:30:58.309649+0000","last_became_peered":"2024-09-18T07:30:58.309649+0000","last_unstale":"2024-09-18T07:34:41.210517+0000","last_undegraded":"2024-09-18T07:34:41.210517+0000","last_fullsized":"2024-09-18T07:34:41.210517+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":205,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.237800+0000","last_change":"2024-09-18T07:33:24.213432+0000","last_active":"2024-09-18T07:34:41.237800+0000","last_peered":"2024-09-18T07:34:41.237800+0000","last_clean":"2024-09-18T07:34:41.237800+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:34:41.237800+0000","last_undegraded":"2024-09-18T07:34:41.237800+0000","last_fullsized":"2024-09-18T07:34:41.237800+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:24.213356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:24.213356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:52.949578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":250,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.212510+0000","last_change":"2024-09-18T07:34:39.253606+0000","last_active":"2024-09-18T07:34:41.212510+0000","last_peered":"2024-09-18T07:34:41.212510+0000","last_clean":"2024-09-18T07:34:41.212510+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:34:41.212510+0000","last_undegraded":"2024-09-18T07:34:41.212510+0000","last_fullsized":"2024-09-18T07:34:41.212510+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:34:39.253558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:39.253558+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:46.279033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":140,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.222878+0000","last_change":"2024-09-18T07:32:54.740291+0000","last_active":"2024-09-18T07:34:41.222878+0000","last_peered":"2024-09-18T07:34:41.222878+0000","last_clean":"2024-09-18T07:34:41.222878+0000","last_became_active":"2024-09-18T07:32:30.830346+0000","last_became_peered":"2024-09-18T07:32:30.830346+0000","last_unstale":"2024-09-18T07:34:41.222878+0000","last_undegraded":"2024-09-18T07:34:41.222878+0000","last_fullsized":"2024-09-18T07:34:41.222878+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:54.740249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:54.740249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"153'9086","reported_seq":9288,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.367506+0000","last_change":"2024-09-18T07:34:30.240915+0000","last_active":"2024-09-18T07:34:41.367506+0000","last_peered":"2024-09-18T07:34:41.367506+0000","last_clean":"2024-09-18T07:34:41.367506+0000","last_became_active":"2024-09-18T07:34:14.963613+0000","last_became_peered":"2024-09-18T07:34:14.963613+0000","last_unstale":"2024-09-18T07:34:41.367506+0000","last_undegraded":"2024-09-18T07:34:41.367506+0000","last_fullsized":"2024-09-18T07:34:41.367506+0000","mapping_epoch":81,"log_start":"113'4100","ondisk_log_start":"113'4100","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'8929","last_scrub_stamp":"2024-09-18T07:34:30.240820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:30.240820+0000","objects_scrubbed":1857,"log_size":4986,"log_dups_size":0,"ondisk_log_size":4986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:31.217551+0000","scrub_duration":268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111214592,"num_objects":1697,"num_object_clones":0,"num_object_copies":5091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":160,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":208,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210101+0000","last_change":"2024-09-18T07:30:58.307470+0000","last_active":"2024-09-18T07:34:41.210101+0000","last_peered":"2024-09-18T07:34:41.210101+0000","last_clean":"2024-09-18T07:34:41.210101+0000","last_became_active":"2024-09-18T07:30:58.307035+0000","last_became_peered":"2024-09-18T07:30:58.307035+0000","last_unstale":"2024-09-18T07:34:41.210101+0000","last_undegraded":"2024-09-18T07:34:41.210101+0000","last_fullsized":"2024-09-18T07:34:41.210101+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:15.814064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":204,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.222341+0000","last_change":"2024-09-18T07:33:29.471394+0000","last_active":"2024-09-18T07:34:41.222341+0000","last_peered":"2024-09-18T07:34:41.222341+0000","last_clean":"2024-09-18T07:34:41.222341+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:34:41.222341+0000","last_undegraded":"2024-09-18T07:34:41.222341+0000","last_fullsized":"2024-09-18T07:34:41.222341+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:29.471330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.471330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:58.117832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":250,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.237013+0000","last_change":"2024-09-18T07:33:52.317149+0000","last_active":"2024-09-18T07:34:41.237013+0000","last_peered":"2024-09-18T07:34:41.237013+0000","last_clean":"2024-09-18T07:34:41.237013+0000","last_became_active":"2024-09-18T07:31:00.336905+0000","last_became_peered":"2024-09-18T07:31:00.336905+0000","last_unstale":"2024-09-18T07:34:41.237013+0000","last_undegraded":"2024-09-18T07:34:41.237013+0000","last_fullsized":"2024-09-18T07:34:41.237013+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:33:52.317075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:52.317075+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:12.567039+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.234103+0000","last_change":"2024-09-18T07:31:02.348538+0000","last_active":"2024-09-18T07:34:41.234103+0000","last_peered":"2024-09-18T07:34:41.234103+0000","last_clean":"2024-09-18T07:34:41.234103+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:34:41.234103+0000","last_undegraded":"2024-09-18T07:34:41.234103+0000","last_fullsized":"2024-09-18T07:34:41.234103+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"153'8919","reported_seq":9122,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.429188+0000","last_change":"2024-09-18T07:34:29.900403+0000","last_active":"2024-09-18T07:34:41.429188+0000","last_peered":"2024-09-18T07:34:41.429188+0000","last_clean":"2024-09-18T07:34:41.429188+0000","last_became_active":"2024-09-18T07:34:14.979581+0000","last_became_peered":"2024-09-18T07:34:14.979581+0000","last_unstale":"2024-09-18T07:34:41.429188+0000","last_undegraded":"2024-09-18T07:34:41.429188+0000","last_fullsized":"2024-09-18T07:34:41.429188+0000","mapping_epoch":81,"log_start":"100'3056","ondisk_log_start":"100'3056","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'8745","last_scrub_stamp":"2024-09-18T07:34:29.900338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:29.900338+0000","objects_scrubbed":1866,"log_size":5863,"log_dups_size":0,"ondisk_log_size":5863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:41.111901+0000","scrub_duration":415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110428160,"num_objects":1685,"num_object_clones":0,"num_object_copies":5055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":181,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":261,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.237151+0000","last_change":"2024-09-18T07:34:39.610842+0000","last_active":"2024-09-18T07:34:41.237151+0000","last_peered":"2024-09-18T07:34:41.237151+0000","last_clean":"2024-09-18T07:34:41.237151+0000","last_became_active":"2024-09-18T07:32:30.842195+0000","last_became_peered":"2024-09-18T07:32:30.842195+0000","last_unstale":"2024-09-18T07:34:41.237151+0000","last_undegraded":"2024-09-18T07:34:41.237151+0000","last_fullsized":"2024-09-18T07:34:41.237151+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:34:39.610788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:39.610788+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:09.049285+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":197,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210454+0000","last_change":"2024-09-18T07:30:56.284440+0000","last_active":"2024-09-18T07:34:41.210454+0000","last_peered":"2024-09-18T07:34:41.210454+0000","last_clean":"2024-09-18T07:34:41.210454+0000","last_became_active":"2024-09-18T07:30:56.284120+0000","last_became_peered":"2024-09-18T07:30:56.284120+0000","last_unstale":"2024-09-18T07:34:41.210454+0000","last_undegraded":"2024-09-18T07:34:41.210454+0000","last_fullsized":"2024-09-18T07:34:41.210454+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:12.896782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":196,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.222641+0000","last_change":"2024-09-18T07:33:21.834910+0000","last_active":"2024-09-18T07:34:41.222641+0000","last_peered":"2024-09-18T07:34:41.222641+0000","last_clean":"2024-09-18T07:34:41.222641+0000","last_became_active":"2024-09-18T07:31:00.334776+0000","last_became_peered":"2024-09-18T07:31:00.334776+0000","last_unstale":"2024-09-18T07:34:41.222641+0000","last_undegraded":"2024-09-18T07:34:41.222641+0000","last_fullsized":"2024-09-18T07:34:41.222641+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":185,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.208682+0000","last_change":"2024-09-18T07:31:02.490534+0000","last_active":"2024-09-18T07:34:41.208682+0000","last_peered":"2024-09-18T07:34:41.208682+0000","last_clean":"2024-09-18T07:34:41.208682+0000","last_became_active":"2024-09-18T07:31:02.490186+0000","last_became_peered":"2024-09-18T07:31:02.490186+0000","last_unstale":"2024-09-18T07:34:41.208682+0000","last_undegraded":"2024-09-18T07:34:41.208682+0000","last_fullsized":"2024-09-18T07:34:41.208682+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"153'8723","reported_seq":8989,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.439229+0000","last_change":"2024-09-18T07:34:30.102295+0000","last_active":"2024-09-18T07:34:41.439229+0000","last_peered":"2024-09-18T07:34:41.439229+0000","last_clean":"2024-09-18T07:34:41.439229+0000","last_became_active":"2024-09-18T07:34:14.964872+0000","last_became_peered":"2024-09-18T07:34:14.964872+0000","last_unstale":"2024-09-18T07:34:41.439229+0000","last_undegraded":"2024-09-18T07:34:41.439229+0000","last_fullsized":"2024-09-18T07:34:41.439229+0000","mapping_epoch":81,"log_start":"98'2600","ondisk_log_start":"98'2600","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'8555","last_scrub_stamp":"2024-09-18T07:34:30.102158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:30.102158+0000","objects_scrubbed":1903,"log_size":6123,"log_dups_size":0,"ondisk_log_size":6123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:52.590520+0000","scrub_duration":460,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113442816,"num_objects":1731,"num_object_clones":0,"num_object_copies":5193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":172,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":295,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210361+0000","last_change":"2024-09-18T07:34:31.955399+0000","last_active":"2024-09-18T07:34:41.210361+0000","last_peered":"2024-09-18T07:34:41.210361+0000","last_clean":"2024-09-18T07:34:41.210361+0000","last_became_active":"2024-09-18T07:30:58.316812+0000","last_became_peered":"2024-09-18T07:30:58.316812+0000","last_unstale":"2024-09-18T07:34:41.210361+0000","last_undegraded":"2024-09-18T07:34:41.210361+0000","last_fullsized":"2024-09-18T07:34:41.210361+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'28","last_scrub_stamp":"2024-09-18T07:34:31.955336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.955336+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:43.708652+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":197,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211236+0000","last_change":"2024-09-18T07:30:56.285353+0000","last_active":"2024-09-18T07:34:41.211236+0000","last_peered":"2024-09-18T07:34:41.211236+0000","last_clean":"2024-09-18T07:34:41.211236+0000","last_became_active":"2024-09-18T07:30:56.285032+0000","last_became_peered":"2024-09-18T07:30:56.285032+0000","last_unstale":"2024-09-18T07:34:41.211236+0000","last_undegraded":"2024-09-18T07:34:41.211236+0000","last_fullsized":"2024-09-18T07:34:41.211236+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":214,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.220231+0000","last_change":"2024-09-18T07:34:39.420024+0000","last_active":"2024-09-18T07:34:41.220231+0000","last_peered":"2024-09-18T07:34:41.220231+0000","last_clean":"2024-09-18T07:34:41.220231+0000","last_became_active":"2024-09-18T07:30:55.283292+0000","last_became_peered":"2024-09-18T07:30:55.283292+0000","last_unstale":"2024-09-18T07:34:41.220231+0000","last_undegraded":"2024-09-18T07:34:41.220231+0000","last_fullsized":"2024-09-18T07:34:41.220231+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:34:39.419958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:34:39.419958+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:59.903039+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":188,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.238923+0000","last_change":"2024-09-18T07:31:00.332738+0000","last_active":"2024-09-18T07:34:41.238923+0000","last_peered":"2024-09-18T07:34:41.238923+0000","last_clean":"2024-09-18T07:34:41.238923+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:34:41.238923+0000","last_undegraded":"2024-09-18T07:34:41.238923+0000","last_fullsized":"2024-09-18T07:34:41.238923+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:07.352960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":191,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.213644+0000","last_change":"2024-09-18T07:32:30.854643+0000","last_active":"2024-09-18T07:34:41.213644+0000","last_peered":"2024-09-18T07:34:41.213644+0000","last_clean":"2024-09-18T07:34:41.213644+0000","last_became_active":"2024-09-18T07:32:30.853087+0000","last_became_peered":"2024-09-18T07:32:30.853087+0000","last_unstale":"2024-09-18T07:34:41.213644+0000","last_undegraded":"2024-09-18T07:34:41.213644+0000","last_fullsized":"2024-09-18T07:34:41.213644+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:45.131703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"153'8992","reported_seq":9196,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.739121+0000","last_change":"2024-09-18T07:34:30.780958+0000","last_active":"2024-09-18T07:34:41.739121+0000","last_peered":"2024-09-18T07:34:41.739121+0000","last_clean":"2024-09-18T07:34:41.739121+0000","last_became_active":"2024-09-18T07:34:14.994644+0000","last_became_peered":"2024-09-18T07:34:14.994644+0000","last_unstale":"2024-09-18T07:34:41.739121+0000","last_undegraded":"2024-09-18T07:34:41.739121+0000","last_fullsized":"2024-09-18T07:34:41.739121+0000","mapping_epoch":81,"log_start":"109'3809","ondisk_log_start":"109'3809","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'8823","last_scrub_stamp":"2024-09-18T07:34:30.780852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:30.780852+0000","objects_scrubbed":1972,"log_size":5183,"log_dups_size":0,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:42.152074+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118161408,"num_objects":1803,"num_object_clones":0,"num_object_copies":5409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":228,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.209875+0000","last_change":"2024-09-18T07:34:38.975221+0000","last_active":"2024-09-18T07:34:41.209875+0000","last_peered":"2024-09-18T07:34:41.209875+0000","last_clean":"2024-09-18T07:34:41.209875+0000","last_became_active":"2024-09-18T07:30:58.310490+0000","last_became_peered":"2024-09-18T07:30:58.310490+0000","last_unstale":"2024-09-18T07:34:41.209875+0000","last_undegraded":"2024-09-18T07:34:41.209875+0000","last_fullsized":"2024-09-18T07:34:41.209875+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-18T07:34:38.975156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:38.975156+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:05.736223+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":205,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.208741+0000","last_change":"2024-09-18T07:33:23.038159+0000","last_active":"2024-09-18T07:34:41.208741+0000","last_peered":"2024-09-18T07:34:41.208741+0000","last_clean":"2024-09-18T07:34:41.208741+0000","last_became_active":"2024-09-18T07:30:56.284803+0000","last_became_peered":"2024-09-18T07:30:56.284803+0000","last_unstale":"2024-09-18T07:34:41.208741+0000","last_undegraded":"2024-09-18T07:34:41.208741+0000","last_fullsized":"2024-09-18T07:34:41.208741+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:23.037963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:23.037963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:45.045160+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":214,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.208835+0000","last_change":"2024-09-18T07:32:02.418590+0000","last_active":"2024-09-18T07:34:41.208835+0000","last_peered":"2024-09-18T07:34:41.208835+0000","last_clean":"2024-09-18T07:34:41.208835+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:34:41.208835+0000","last_undegraded":"2024-09-18T07:34:41.208835+0000","last_fullsized":"2024-09-18T07:34:41.208835+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:32:02.418507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.418507+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":196,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211703+0000","last_change":"2024-09-18T07:32:30.837706+0000","last_active":"2024-09-18T07:34:41.211703+0000","last_peered":"2024-09-18T07:34:41.211703+0000","last_clean":"2024-09-18T07:34:41.211703+0000","last_became_active":"2024-09-18T07:32:30.837243+0000","last_became_peered":"2024-09-18T07:32:30.837243+0000","last_unstale":"2024-09-18T07:34:41.211703+0000","last_undegraded":"2024-09-18T07:34:41.211703+0000","last_fullsized":"2024-09-18T07:34:41.211703+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:09.936542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":164,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.223696+0000","last_change":"2024-09-18T07:33:57.655849+0000","last_active":"2024-09-18T07:34:41.223696+0000","last_peered":"2024-09-18T07:34:41.223696+0000","last_clean":"2024-09-18T07:34:41.223696+0000","last_became_active":"2024-09-18T07:32:30.839186+0000","last_became_peered":"2024-09-18T07:32:30.839186+0000","last_unstale":"2024-09-18T07:34:41.223696+0000","last_undegraded":"2024-09-18T07:34:41.223696+0000","last_fullsized":"2024-09-18T07:34:41.223696+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:57.655797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.655797+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:04.938177+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"153'8933","reported_seq":12953,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.547411+0000","last_change":"2024-09-18T07:34:30.341766+0000","last_active":"2024-09-18T07:34:41.547411+0000","last_peered":"2024-09-18T07:34:41.547411+0000","last_clean":"2024-09-18T07:34:41.547411+0000","last_became_active":"2024-09-18T07:34:14.998814+0000","last_became_peered":"2024-09-18T07:34:14.998814+0000","last_unstale":"2024-09-18T07:34:41.547411+0000","last_undegraded":"2024-09-18T07:34:41.547411+0000","last_fullsized":"2024-09-18T07:34:41.547411+0000","mapping_epoch":107,"log_start":"98'2600","ondisk_log_start":"98'2600","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'8779","last_scrub_stamp":"2024-09-18T07:34:30.341369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:30.341369+0000","objects_scrubbed":1918,"log_size":6333,"log_dups_size":0,"ondisk_log_size":6333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:44.625184+0000","scrub_duration":452,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115474432,"num_objects":1762,"num_object_clones":0,"num_object_copies":5286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":156,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":231,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.238176+0000","last_change":"2024-09-18T07:33:27.040227+0000","last_active":"2024-09-18T07:34:41.238176+0000","last_peered":"2024-09-18T07:34:41.238176+0000","last_clean":"2024-09-18T07:34:41.238176+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:34:41.238176+0000","last_undegraded":"2024-09-18T07:34:41.238176+0000","last_fullsized":"2024-09-18T07:34:41.238176+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:51.281723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":202,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.236889+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:34:41.236889+0000","last_peered":"2024-09-18T07:34:41.236889+0000","last_clean":"2024-09-18T07:34:41.236889+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:34:41.236889+0000","last_undegraded":"2024-09-18T07:34:41.236889+0000","last_fullsized":"2024-09-18T07:34:41.236889+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:43.132679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":310,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.208640+0000","last_change":"2024-09-18T07:32:30.859165+0000","last_active":"2024-09-18T07:34:41.208640+0000","last_peered":"2024-09-18T07:34:41.208640+0000","last_clean":"2024-09-18T07:34:41.208640+0000","last_became_active":"2024-09-18T07:32:30.851004+0000","last_became_peered":"2024-09-18T07:32:30.851004+0000","last_unstale":"2024-09-18T07:34:41.208640+0000","last_undegraded":"2024-09-18T07:34:41.208640+0000","last_fullsized":"2024-09-18T07:34:41.208640+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:07.886671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":137,"num_read_kb":132,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":196,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.222180+0000","last_change":"2024-09-18T07:33:27.394247+0000","last_active":"2024-09-18T07:34:41.222180+0000","last_peered":"2024-09-18T07:34:41.222180+0000","last_clean":"2024-09-18T07:34:41.222180+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:34:41.222180+0000","last_undegraded":"2024-09-18T07:34:41.222180+0000","last_fullsized":"2024-09-18T07:34:41.222180+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.394203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.394203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":201,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.209750+0000","last_change":"2024-09-18T07:33:25.028842+0000","last_active":"2024-09-18T07:34:41.209750+0000","last_peered":"2024-09-18T07:34:41.209750+0000","last_clean":"2024-09-18T07:34:41.209750+0000","last_became_active":"2024-09-18T07:31:02.347390+0000","last_became_peered":"2024-09-18T07:31:02.347390+0000","last_unstale":"2024-09-18T07:34:41.209750+0000","last_undegraded":"2024-09-18T07:34:41.209750+0000","last_fullsized":"2024-09-18T07:34:41.209750+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:25.028609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:25.028609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"153'9035","reported_seq":13107,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.744240+0000","last_change":"2024-09-18T07:34:31.931164+0000","last_active":"2024-09-18T07:34:41.744240+0000","last_peered":"2024-09-18T07:34:41.744240+0000","last_clean":"2024-09-18T07:34:41.744240+0000","last_became_active":"2024-09-18T07:34:14.989995+0000","last_became_peered":"2024-09-18T07:34:14.989995+0000","last_unstale":"2024-09-18T07:34:41.744240+0000","last_undegraded":"2024-09-18T07:34:41.744240+0000","last_fullsized":"2024-09-18T07:34:41.744240+0000","mapping_epoch":105,"log_start":"109'3790","ondisk_log_start":"109'3790","created":81,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'8868","last_scrub_stamp":"2024-09-18T07:34:31.930782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.930782+0000","objects_scrubbed":2001,"log_size":5245,"log_dups_size":0,"ondisk_log_size":5245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:57.008994+0000","scrub_duration":320,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120193024,"num_objects":1834,"num_object_clones":0,"num_object_copies":5502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":167,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"94'17","reported_seq":301,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.224129+0000","last_change":"2024-09-18T07:33:33.849354+0000","last_active":"2024-09-18T07:34:41.224129+0000","last_peered":"2024-09-18T07:34:41.224129+0000","last_clean":"2024-09-18T07:34:41.224129+0000","last_became_active":"2024-09-18T07:32:30.842449+0000","last_became_peered":"2024-09-18T07:32:30.842449+0000","last_unstale":"2024-09-18T07:34:41.224129+0000","last_undegraded":"2024-09-18T07:34:41.224129+0000","last_fullsized":"2024-09-18T07:34:41.224129+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'17","last_scrub_stamp":"2024-09-18T07:33:33.849285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.849285+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:53.616735+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":204,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.212967+0000","last_change":"2024-09-18T07:31:59.617046+0000","last_active":"2024-09-18T07:34:41.212967+0000","last_peered":"2024-09-18T07:34:41.212967+0000","last_clean":"2024-09-18T07:34:41.212967+0000","last_became_active":"2024-09-18T07:30:56.288149+0000","last_became_peered":"2024-09-18T07:30:56.288149+0000","last_unstale":"2024-09-18T07:34:41.212967+0000","last_undegraded":"2024-09-18T07:34:41.212967+0000","last_fullsized":"2024-09-18T07:34:41.212967+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:59.616988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:59.616988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:06.307919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":227,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.234201+0000","last_change":"2024-09-18T07:33:23.172950+0000","last_active":"2024-09-18T07:34:41.234201+0000","last_peered":"2024-09-18T07:34:41.234201+0000","last_clean":"2024-09-18T07:34:41.234201+0000","last_became_active":"2024-09-18T07:32:30.842828+0000","last_became_peered":"2024-09-18T07:32:30.842828+0000","last_unstale":"2024-09-18T07:34:41.234201+0000","last_undegraded":"2024-09-18T07:34:41.234201+0000","last_fullsized":"2024-09-18T07:34:41.234201+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:33:23.172848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:33:23.172848+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.696337+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":140,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210395+0000","last_change":"2024-09-18T07:34:36.964525+0000","last_active":"2024-09-18T07:34:41.210395+0000","last_peered":"2024-09-18T07:34:41.210395+0000","last_clean":"2024-09-18T07:34:41.210395+0000","last_became_active":"2024-09-18T07:32:30.835105+0000","last_became_peered":"2024-09-18T07:32:30.835105+0000","last_unstale":"2024-09-18T07:34:41.210395+0000","last_undegraded":"2024-09-18T07:34:41.210395+0000","last_fullsized":"2024-09-18T07:34:41.210395+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:36.964421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:36.964421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:39.426475+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.221974+0000","last_change":"2024-09-18T07:33:22.358792+0000","last_active":"2024-09-18T07:34:41.221974+0000","last_peered":"2024-09-18T07:34:41.221974+0000","last_clean":"2024-09-18T07:34:41.221974+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:34:41.221974+0000","last_undegraded":"2024-09-18T07:34:41.221974+0000","last_fullsized":"2024-09-18T07:34:41.221974+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.358746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.358746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"153'9606","reported_seq":9812,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.433825+0000","last_change":"2024-09-18T07:34:41.087606+0000","last_active":"2024-09-18T07:34:41.433825+0000","last_peered":"2024-09-18T07:34:41.433825+0000","last_clean":"2024-09-18T07:34:41.433825+0000","last_became_active":"2024-09-18T07:32:08.063070+0000","last_became_peered":"2024-09-18T07:32:08.063070+0000","last_unstale":"2024-09-18T07:34:41.433825+0000","last_undegraded":"2024-09-18T07:34:41.433825+0000","last_fullsized":"2024-09-18T07:34:41.433825+0000","mapping_epoch":81,"log_start":"113'4227","ondisk_log_start":"113'4227","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9587","last_scrub_stamp":"2024-09-18T07:34:41.087538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:41.087538+0000","objects_scrubbed":3521,"log_size":5379,"log_dups_size":0,"ondisk_log_size":5379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:59.053471+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229441560,"num_objects":3502,"num_object_clones":0,"num_object_copies":10506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3502,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9606,"num_write_kb":419393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":239,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211492+0000","last_change":"2024-09-18T07:34:03.832922+0000","last_active":"2024-09-18T07:34:41.211492+0000","last_peered":"2024-09-18T07:34:41.211492+0000","last_clean":"2024-09-18T07:34:41.211492+0000","last_became_active":"2024-09-18T07:32:26.785711+0000","last_became_peered":"2024-09-18T07:32:26.785711+0000","last_unstale":"2024-09-18T07:34:41.211492+0000","last_undegraded":"2024-09-18T07:34:41.211492+0000","last_fullsized":"2024-09-18T07:34:41.211492+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:34:03.832858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:03.832858+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:25.331056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":204,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210735+0000","last_change":"2024-09-18T07:32:30.829352+0000","last_active":"2024-09-18T07:34:41.210735+0000","last_peered":"2024-09-18T07:34:41.210735+0000","last_clean":"2024-09-18T07:34:41.210735+0000","last_became_active":"2024-09-18T07:32:30.828736+0000","last_became_peered":"2024-09-18T07:32:30.828736+0000","last_unstale":"2024-09-18T07:34:41.210735+0000","last_undegraded":"2024-09-18T07:34:41.210735+0000","last_fullsized":"2024-09-18T07:34:41.210735+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":196,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.221883+0000","last_change":"2024-09-18T07:33:36.582039+0000","last_active":"2024-09-18T07:34:41.221883+0000","last_peered":"2024-09-18T07:34:41.221883+0000","last_clean":"2024-09-18T07:34:41.221883+0000","last_became_active":"2024-09-18T07:31:00.336172+0000","last_became_peered":"2024-09-18T07:31:00.336172+0000","last_unstale":"2024-09-18T07:34:41.221883+0000","last_undegraded":"2024-09-18T07:34:41.221883+0000","last_fullsized":"2024-09-18T07:34:41.221883+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:36.581998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.581998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:04.288035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211968+0000","last_change":"2024-09-18T07:33:20.990542+0000","last_active":"2024-09-18T07:34:41.211968+0000","last_peered":"2024-09-18T07:34:41.211968+0000","last_clean":"2024-09-18T07:34:41.211968+0000","last_became_active":"2024-09-18T07:31:02.490953+0000","last_became_peered":"2024-09-18T07:31:02.490953+0000","last_unstale":"2024-09-18T07:34:41.211968+0000","last_undegraded":"2024-09-18T07:34:41.211968+0000","last_fullsized":"2024-09-18T07:34:41.211968+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:20.990401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:20.990401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.208665+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"153'9517","reported_seq":9682,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.550594+0000","last_change":"2024-09-18T07:32:08.060282+0000","last_active":"2024-09-18T07:34:41.550594+0000","last_peered":"2024-09-18T07:34:41.550594+0000","last_clean":"2024-09-18T07:34:41.550594+0000","last_became_active":"2024-09-18T07:32:08.059971+0000","last_became_peered":"2024-09-18T07:32:08.059971+0000","last_unstale":"2024-09-18T07:34:41.550594+0000","last_undegraded":"2024-09-18T07:34:41.550594+0000","last_fullsized":"2024-09-18T07:34:41.550594+0000","mapping_epoch":81,"log_start":"114'4311","ondisk_log_start":"114'4311","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5206,"log_dups_size":0,"ondisk_log_size":5206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228261888,"num_objects":3483,"num_object_clones":0,"num_object_copies":10449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9517,"num_write_kb":416000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":229,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211668+0000","last_change":"2024-09-18T07:33:56.797549+0000","last_active":"2024-09-18T07:34:41.211668+0000","last_peered":"2024-09-18T07:34:41.211668+0000","last_clean":"2024-09-18T07:34:41.211668+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:34:41.211668+0000","last_undegraded":"2024-09-18T07:34:41.211668+0000","last_fullsized":"2024-09-18T07:34:41.211668+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:33:56.797466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:56.797466+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:14.685704+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":204,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.222876+0000","last_change":"2024-09-18T07:34:33.614709+0000","last_active":"2024-09-18T07:34:41.222876+0000","last_peered":"2024-09-18T07:34:41.222876+0000","last_clean":"2024-09-18T07:34:41.222876+0000","last_became_active":"2024-09-18T07:30:56.284020+0000","last_became_peered":"2024-09-18T07:30:56.284020+0000","last_unstale":"2024-09-18T07:34:41.222876+0000","last_undegraded":"2024-09-18T07:34:41.222876+0000","last_fullsized":"2024-09-18T07:34:41.222876+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:33.614666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:33.614666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:35.196971+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":197,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.209841+0000","last_change":"2024-09-18T07:32:05.239202+0000","last_active":"2024-09-18T07:34:41.209841+0000","last_peered":"2024-09-18T07:34:41.209841+0000","last_clean":"2024-09-18T07:34:41.209841+0000","last_became_active":"2024-09-18T07:31:00.332836+0000","last_became_peered":"2024-09-18T07:31:00.332836+0000","last_unstale":"2024-09-18T07:34:41.209841+0000","last_undegraded":"2024-09-18T07:34:41.209841+0000","last_fullsized":"2024-09-18T07:34:41.209841+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:05.239160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.239160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":193,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211856+0000","last_change":"2024-09-18T07:33:21.860561+0000","last_active":"2024-09-18T07:34:41.211856+0000","last_peered":"2024-09-18T07:34:41.211856+0000","last_clean":"2024-09-18T07:34:41.211856+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:34:41.211856+0000","last_undegraded":"2024-09-18T07:34:41.211856+0000","last_fullsized":"2024-09-18T07:34:41.211856+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.860495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.860495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:51.431373+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"153'9593","reported_seq":13591,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.756153+0000","last_change":"2024-09-18T07:33:14.766074+0000","last_active":"2024-09-18T07:34:41.756153+0000","last_peered":"2024-09-18T07:34:41.756153+0000","last_clean":"2024-09-18T07:34:41.756153+0000","last_became_active":"2024-09-18T07:33:14.762251+0000","last_became_peered":"2024-09-18T07:33:14.762251+0000","last_unstale":"2024-09-18T07:34:41.756153+0000","last_undegraded":"2024-09-18T07:34:41.756153+0000","last_fullsized":"2024-09-18T07:34:41.756153+0000","mapping_epoch":115,"log_start":"118'5240","ondisk_log_start":"118'5240","created":81,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4353,"log_dups_size":0,"ondisk_log_size":4353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232718336,"num_objects":3551,"num_object_clones":0,"num_object_copies":10653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9593,"num_write_kb":420608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1912,"num_bytes_recovered":125304832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":216,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211107+0000","last_change":"2024-09-18T07:32:05.000363+0000","last_active":"2024-09-18T07:34:41.211107+0000","last_peered":"2024-09-18T07:34:41.211107+0000","last_clean":"2024-09-18T07:34:41.211107+0000","last_became_active":"2024-09-18T07:30:58.316423+0000","last_became_peered":"2024-09-18T07:30:58.316423+0000","last_unstale":"2024-09-18T07:34:41.211107+0000","last_undegraded":"2024-09-18T07:34:41.211107+0000","last_fullsized":"2024-09-18T07:34:41.211107+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:32:05.000310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.000310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":204,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.212299+0000","last_change":"2024-09-18T07:32:06.688493+0000","last_active":"2024-09-18T07:34:41.212299+0000","last_peered":"2024-09-18T07:34:41.212299+0000","last_clean":"2024-09-18T07:34:41.212299+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:34:41.212299+0000","last_undegraded":"2024-09-18T07:34:41.212299+0000","last_fullsized":"2024-09-18T07:34:41.212299+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:06.688444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:06.688444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:12.324753+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"120'10","reported_seq":273,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211210+0000","last_change":"2024-09-18T07:33:33.001389+0000","last_active":"2024-09-18T07:34:41.211210+0000","last_peered":"2024-09-18T07:34:41.211210+0000","last_clean":"2024-09-18T07:34:41.211210+0000","last_became_active":"2024-09-18T07:33:27.007386+0000","last_became_peered":"2024-09-18T07:33:27.007386+0000","last_unstale":"2024-09-18T07:34:41.211210+0000","last_undegraded":"2024-09-18T07:34:41.211210+0000","last_fullsized":"2024-09-18T07:34:41.211210+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'10","last_scrub_stamp":"2024-09-18T07:33:33.000874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.000874+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:48.823137+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":185,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211404+0000","last_change":"2024-09-18T07:31:02.347721+0000","last_active":"2024-09-18T07:34:41.211404+0000","last_peered":"2024-09-18T07:34:41.211404+0000","last_clean":"2024-09-18T07:34:41.211404+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:34:41.211404+0000","last_undegraded":"2024-09-18T07:34:41.211404+0000","last_fullsized":"2024-09-18T07:34:41.211404+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"153'9124","reported_seq":13436,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.551578+0000","last_change":"2024-09-18T07:34:41.215024+0000","last_active":"2024-09-18T07:34:41.551578+0000","last_peered":"2024-09-18T07:34:41.551578+0000","last_clean":"2024-09-18T07:34:41.551578+0000","last_became_active":"2024-09-18T07:34:41.214186+0000","last_became_peered":"2024-09-18T07:34:41.214186+0000","last_unstale":"2024-09-18T07:34:41.551578+0000","last_undegraded":"2024-09-18T07:34:41.551578+0000","last_fullsized":"2024-09-18T07:34:41.551578+0000","mapping_epoch":110,"log_start":"149'9067","ondisk_log_start":"149'9067","created":81,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'8921","last_scrub_stamp":"2024-09-18T07:34:31.186602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.186602+0000","objects_scrubbed":1940,"log_size":57,"log_dups_size":3573,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:36.785452+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229965824,"num_objects":3509,"num_object_clones":0,"num_object_copies":10527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":248,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":208,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.237061+0000","last_change":"2024-09-18T07:30:58.314424+0000","last_active":"2024-09-18T07:34:41.237061+0000","last_peered":"2024-09-18T07:34:41.237061+0000","last_clean":"2024-09-18T07:34:41.237061+0000","last_became_active":"2024-09-18T07:30:58.314127+0000","last_became_peered":"2024-09-18T07:30:58.314127+0000","last_unstale":"2024-09-18T07:34:41.237061+0000","last_undegraded":"2024-09-18T07:34:41.237061+0000","last_fullsized":"2024-09-18T07:34:41.237061+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":204,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.221682+0000","last_change":"2024-09-18T07:33:28.459796+0000","last_active":"2024-09-18T07:34:41.221682+0000","last_peered":"2024-09-18T07:34:41.221682+0000","last_clean":"2024-09-18T07:34:41.221682+0000","last_became_active":"2024-09-18T07:30:56.288868+0000","last_became_peered":"2024-09-18T07:30:56.288868+0000","last_unstale":"2024-09-18T07:34:41.221682+0000","last_undegraded":"2024-09-18T07:34:41.221682+0000","last_fullsized":"2024-09-18T07:34:41.221682+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:28.459747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:28.459747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:54.082899+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":197,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.209227+0000","last_change":"2024-09-18T07:34:31.959675+0000","last_active":"2024-09-18T07:34:41.209227+0000","last_peered":"2024-09-18T07:34:41.209227+0000","last_clean":"2024-09-18T07:34:41.209227+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:34:41.209227+0000","last_undegraded":"2024-09-18T07:34:41.209227+0000","last_fullsized":"2024-09-18T07:34:41.209227+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:31.959636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.959636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:50.298137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":199,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.236804+0000","last_change":"2024-09-18T07:34:02.301413+0000","last_active":"2024-09-18T07:34:41.236804+0000","last_peered":"2024-09-18T07:34:41.236804+0000","last_clean":"2024-09-18T07:34:41.236804+0000","last_became_active":"2024-09-18T07:32:30.854029+0000","last_became_peered":"2024-09-18T07:32:30.854029+0000","last_unstale":"2024-09-18T07:34:41.236804+0000","last_undegraded":"2024-09-18T07:34:41.236804+0000","last_fullsized":"2024-09-18T07:34:41.236804+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:02.301353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:34:02.301353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:16.198563+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"153'9121","reported_seq":9337,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.750203+0000","last_change":"2024-09-18T07:34:36.801770+0000","last_active":"2024-09-18T07:34:41.750203+0000","last_peered":"2024-09-18T07:34:41.750203+0000","last_clean":"2024-09-18T07:34:41.750203+0000","last_became_active":"2024-09-18T07:34:36.800541+0000","last_became_peered":"2024-09-18T07:34:36.800541+0000","last_unstale":"2024-09-18T07:34:41.750203+0000","last_undegraded":"2024-09-18T07:34:41.750203+0000","last_fullsized":"2024-09-18T07:34:41.750203+0000","mapping_epoch":81,"log_start":"145'8920","ondisk_log_start":"145'8920","created":81,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'8860","last_scrub_stamp":"2024-09-18T07:34:30.270402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:30.270402+0000","objects_scrubbed":1991,"log_size":201,"log_dups_size":3491,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:30.793855+0000","scrub_duration":670,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234356736,"num_objects":3576,"num_object_clones":0,"num_object_copies":10728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":303,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":266,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.222676+0000","last_change":"2024-09-18T07:33:32.803961+0000","last_active":"2024-09-18T07:34:41.222676+0000","last_peered":"2024-09-18T07:34:41.222676+0000","last_clean":"2024-09-18T07:34:41.222676+0000","last_became_active":"2024-09-18T07:32:30.834955+0000","last_became_peered":"2024-09-18T07:32:30.834955+0000","last_unstale":"2024-09-18T07:34:41.222676+0000","last_undegraded":"2024-09-18T07:34:41.222676+0000","last_fullsized":"2024-09-18T07:34:41.222676+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:33:32.803884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.803884+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":132,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.222017+0000","last_change":"2024-09-18T07:32:30.840417+0000","last_active":"2024-09-18T07:34:41.222017+0000","last_peered":"2024-09-18T07:34:41.222017+0000","last_clean":"2024-09-18T07:34:41.222017+0000","last_became_active":"2024-09-18T07:32:30.840046+0000","last_became_peered":"2024-09-18T07:32:30.840046+0000","last_unstale":"2024-09-18T07:34:41.222017+0000","last_undegraded":"2024-09-18T07:34:41.222017+0000","last_fullsized":"2024-09-18T07:34:41.222017+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.481153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.481153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.709514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":204,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.209805+0000","last_change":"2024-09-18T07:33:34.966610+0000","last_active":"2024-09-18T07:34:41.209805+0000","last_peered":"2024-09-18T07:34:41.209805+0000","last_clean":"2024-09-18T07:34:41.209805+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:34:41.209805+0000","last_undegraded":"2024-09-18T07:34:41.209805+0000","last_fullsized":"2024-09-18T07:34:41.209805+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:34.966537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:34.966537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":132,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.209099+0000","last_change":"2024-09-18T07:32:30.839764+0000","last_active":"2024-09-18T07:34:41.209099+0000","last_peered":"2024-09-18T07:34:41.209099+0000","last_clean":"2024-09-18T07:34:41.209099+0000","last_became_active":"2024-09-18T07:32:30.839399+0000","last_became_peered":"2024-09-18T07:32:30.839399+0000","last_unstale":"2024-09-18T07:34:41.209099+0000","last_undegraded":"2024-09-18T07:34:41.209099+0000","last_fullsized":"2024-09-18T07:34:41.209099+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:06.911907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.a","version":"153'9592","reported_seq":9754,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.601568+0000","last_change":"2024-09-18T07:32:08.065180+0000","last_active":"2024-09-18T07:34:41.601568+0000","last_peered":"2024-09-18T07:34:41.601568+0000","last_clean":"2024-09-18T07:34:41.601568+0000","last_became_active":"2024-09-18T07:32:08.064911+0000","last_became_peered":"2024-09-18T07:32:08.064911+0000","last_unstale":"2024-09-18T07:34:41.601568+0000","last_undegraded":"2024-09-18T07:34:41.601568+0000","last_fullsized":"2024-09-18T07:34:41.601568+0000","mapping_epoch":81,"log_start":"122'5756","ondisk_log_start":"122'5756","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":3836,"log_dups_size":0,"ondisk_log_size":3836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228327424,"num_objects":3484,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9592,"num_write_kb":418432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":224,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210617+0000","last_change":"2024-09-18T07:32:02.967912+0000","last_active":"2024-09-18T07:34:41.210617+0000","last_peered":"2024-09-18T07:34:41.210617+0000","last_clean":"2024-09-18T07:34:41.210617+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:34:41.210617+0000","last_undegraded":"2024-09-18T07:34:41.210617+0000","last_fullsized":"2024-09-18T07:34:41.210617+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:32:02.967835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.967835+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":132,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211587+0000","last_change":"2024-09-18T07:32:30.846539+0000","last_active":"2024-09-18T07:34:41.211587+0000","last_peered":"2024-09-18T07:34:41.211587+0000","last_clean":"2024-09-18T07:34:41.211587+0000","last_became_active":"2024-09-18T07:32:30.846204+0000","last_became_peered":"2024-09-18T07:32:30.846204+0000","last_unstale":"2024-09-18T07:34:41.211587+0000","last_undegraded":"2024-09-18T07:34:41.211587+0000","last_fullsized":"2024-09-18T07:34:41.211587+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":205,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.209199+0000","last_change":"2024-09-18T07:33:56.967701+0000","last_active":"2024-09-18T07:34:41.209199+0000","last_peered":"2024-09-18T07:34:41.209199+0000","last_clean":"2024-09-18T07:34:41.209199+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:34:41.209199+0000","last_undegraded":"2024-09-18T07:34:41.209199+0000","last_fullsized":"2024-09-18T07:34:41.209199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:56.967627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:56.967627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:18.872883+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":191,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.212051+0000","last_change":"2024-09-18T07:32:30.849280+0000","last_active":"2024-09-18T07:34:41.212051+0000","last_peered":"2024-09-18T07:34:41.212051+0000","last_clean":"2024-09-18T07:34:41.212051+0000","last_became_active":"2024-09-18T07:32:30.847635+0000","last_became_peered":"2024-09-18T07:32:30.847635+0000","last_unstale":"2024-09-18T07:34:41.212051+0000","last_undegraded":"2024-09-18T07:34:41.212051+0000","last_fullsized":"2024-09-18T07:34:41.212051+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:48.144692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"153'9348","reported_seq":9513,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.501964+0000","last_change":"2024-09-18T07:32:08.062243+0000","last_active":"2024-09-18T07:34:41.501964+0000","last_peered":"2024-09-18T07:34:41.501964+0000","last_clean":"2024-09-18T07:34:41.501964+0000","last_became_active":"2024-09-18T07:32:08.061925+0000","last_became_peered":"2024-09-18T07:32:08.061925+0000","last_unstale":"2024-09-18T07:34:41.501964+0000","last_undegraded":"2024-09-18T07:34:41.501964+0000","last_fullsized":"2024-09-18T07:34:41.501964+0000","mapping_epoch":81,"log_start":"121'5362","ondisk_log_start":"121'5362","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":3986,"log_dups_size":0,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227016704,"num_objects":3464,"num_object_clones":0,"num_object_copies":10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9348,"num_write_kb":409984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":248,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.219840+0000","last_change":"2024-09-18T07:34:03.531341+0000","last_active":"2024-09-18T07:34:41.219840+0000","last_peered":"2024-09-18T07:34:41.219840+0000","last_clean":"2024-09-18T07:34:41.219840+0000","last_became_active":"2024-09-18T07:32:30.832486+0000","last_became_peered":"2024-09-18T07:32:30.832486+0000","last_unstale":"2024-09-18T07:34:41.219840+0000","last_undegraded":"2024-09-18T07:34:41.219840+0000","last_fullsized":"2024-09-18T07:34:41.219840+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:34:03.531262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:03.531262+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:10.299055+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":148,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211613+0000","last_change":"2024-09-18T07:32:30.865327+0000","last_active":"2024-09-18T07:34:41.211613+0000","last_peered":"2024-09-18T07:34:41.211613+0000","last_clean":"2024-09-18T07:34:41.211613+0000","last_became_active":"2024-09-18T07:32:30.844389+0000","last_became_peered":"2024-09-18T07:32:30.844389+0000","last_unstale":"2024-09-18T07:34:41.211613+0000","last_undegraded":"2024-09-18T07:34:41.211613+0000","last_fullsized":"2024-09-18T07:34:41.211613+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:09.798229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":197,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.383778+0000","last_change":"2024-09-18T07:32:57.792858+0000","last_active":"2024-09-18T07:34:41.383778+0000","last_peered":"2024-09-18T07:34:41.383778+0000","last_clean":"2024-09-18T07:34:41.383778+0000","last_became_active":"2024-09-18T07:32:30.831224+0000","last_became_peered":"2024-09-18T07:32:30.831224+0000","last_unstale":"2024-09-18T07:34:41.383778+0000","last_undegraded":"2024-09-18T07:34:41.383778+0000","last_fullsized":"2024-09-18T07:34:41.383778+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-09-18T07:32:57.792795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:57.792795+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:08.758097+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.220032+0000","last_change":"2024-09-18T07:31:02.489914+0000","last_active":"2024-09-18T07:34:41.220032+0000","last_peered":"2024-09-18T07:34:41.220032+0000","last_clean":"2024-09-18T07:34:41.220032+0000","last_became_active":"2024-09-18T07:31:02.489619+0000","last_became_peered":"2024-09-18T07:31:02.489619+0000","last_unstale":"2024-09-18T07:34:41.220032+0000","last_undegraded":"2024-09-18T07:34:41.220032+0000","last_fullsized":"2024-09-18T07:34:41.220032+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"153'9017","reported_seq":12196,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.730847+0000","last_change":"2024-09-18T07:34:34.840072+0000","last_active":"2024-09-18T07:34:41.730847+0000","last_peered":"2024-09-18T07:34:41.730847+0000","last_clean":"2024-09-18T07:34:41.730847+0000","last_became_active":"2024-09-18T07:34:32.146429+0000","last_became_peered":"2024-09-18T07:34:32.146429+0000","last_unstale":"2024-09-18T07:34:41.730847+0000","last_undegraded":"2024-09-18T07:34:41.730847+0000","last_fullsized":"2024-09-18T07:34:41.730847+0000","mapping_epoch":144,"log_start":"109'3809","ondisk_log_start":"109'3809","created":134,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":5,"last_scrub":"146'8934","last_scrub_stamp":"2024-09-18T07:34:34.840000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:34.840000+0000","objects_scrubbed":1815,"log_size":5208,"log_dups_size":0,"ondisk_log_size":5208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:46.558016+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113508352,"num_objects":1732,"num_object_clones":0,"num_object_copies":5196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":212,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.222470+0000","last_change":"2024-09-18T07:34:39.639200+0000","last_active":"2024-09-18T07:34:41.222470+0000","last_peered":"2024-09-18T07:34:41.222470+0000","last_clean":"2024-09-18T07:34:41.222470+0000","last_became_active":"2024-09-18T07:30:56.282357+0000","last_became_peered":"2024-09-18T07:30:56.282357+0000","last_unstale":"2024-09-18T07:34:41.222470+0000","last_undegraded":"2024-09-18T07:34:41.222470+0000","last_fullsized":"2024-09-18T07:34:41.222470+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:39.639148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:39.639148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:58.385140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":216,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.214495+0000","last_change":"2024-09-18T07:32:30.872792+0000","last_active":"2024-09-18T07:34:41.214495+0000","last_peered":"2024-09-18T07:34:41.214495+0000","last_clean":"2024-09-18T07:34:41.214495+0000","last_became_active":"2024-09-18T07:32:30.856211+0000","last_became_peered":"2024-09-18T07:32:30.856211+0000","last_unstale":"2024-09-18T07:34:41.214495+0000","last_undegraded":"2024-09-18T07:34:41.214495+0000","last_fullsized":"2024-09-18T07:34:41.214495+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":196,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.222070+0000","last_change":"2024-09-18T07:33:32.589362+0000","last_active":"2024-09-18T07:34:41.222070+0000","last_peered":"2024-09-18T07:34:41.222070+0000","last_clean":"2024-09-18T07:34:41.222070+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:34:41.222070+0000","last_undegraded":"2024-09-18T07:34:41.222070+0000","last_fullsized":"2024-09-18T07:34:41.222070+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.589289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.589289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":147,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.223412+0000","last_change":"2024-09-18T07:33:58.688688+0000","last_active":"2024-09-18T07:34:41.223412+0000","last_peered":"2024-09-18T07:34:41.223412+0000","last_clean":"2024-09-18T07:34:41.223412+0000","last_became_active":"2024-09-18T07:32:30.831805+0000","last_became_peered":"2024-09-18T07:32:30.831805+0000","last_unstale":"2024-09-18T07:34:41.223412+0000","last_undegraded":"2024-09-18T07:34:41.223412+0000","last_fullsized":"2024-09-18T07:34:41.223412+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:58.688631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:58.688631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:24.030416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.15","version":"153'8893","reported_seq":12916,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.538355+0000","last_change":"2024-09-18T07:34:33.812290+0000","last_active":"2024-09-18T07:34:41.538355+0000","last_peered":"2024-09-18T07:34:41.538355+0000","last_clean":"2024-09-18T07:34:41.538355+0000","last_became_active":"2024-09-18T07:34:32.146360+0000","last_became_peered":"2024-09-18T07:34:32.146360+0000","last_unstale":"2024-09-18T07:34:41.538355+0000","last_undegraded":"2024-09-18T07:34:41.538355+0000","last_fullsized":"2024-09-18T07:34:41.538355+0000","mapping_epoch":144,"log_start":"98'2600","ondisk_log_start":"98'2600","created":134,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":5,"last_scrub":"145'8772","last_scrub_stamp":"2024-09-18T07:34:33.812158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:33.812158+0000","objects_scrubbed":1963,"log_size":6293,"log_dups_size":0,"ondisk_log_size":6293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:36.747012+0000","scrub_duration":805,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119996416,"num_objects":1831,"num_object_clones":0,"num_object_copies":5493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":225,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.237713+0000","last_change":"2024-09-18T07:32:30.868702+0000","last_active":"2024-09-18T07:34:41.237713+0000","last_peered":"2024-09-18T07:34:41.237713+0000","last_clean":"2024-09-18T07:34:41.237713+0000","last_became_active":"2024-09-18T07:32:30.861402+0000","last_became_peered":"2024-09-18T07:32:30.861402+0000","last_unstale":"2024-09-18T07:34:41.237713+0000","last_undegraded":"2024-09-18T07:34:41.237713+0000","last_fullsized":"2024-09-18T07:34:41.237713+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:12.449151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":132,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210574+0000","last_change":"2024-09-18T07:32:30.842334+0000","last_active":"2024-09-18T07:34:41.210574+0000","last_peered":"2024-09-18T07:34:41.210574+0000","last_clean":"2024-09-18T07:34:41.210574+0000","last_became_active":"2024-09-18T07:32:30.841928+0000","last_became_peered":"2024-09-18T07:32:30.841928+0000","last_unstale":"2024-09-18T07:34:41.210574+0000","last_undegraded":"2024-09-18T07:34:41.210574+0000","last_fullsized":"2024-09-18T07:34:41.210574+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":203,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.213681+0000","last_change":"2024-09-18T07:33:58.276319+0000","last_active":"2024-09-18T07:34:41.213681+0000","last_peered":"2024-09-18T07:34:41.213681+0000","last_clean":"2024-09-18T07:34:41.213681+0000","last_became_active":"2024-09-18T07:32:30.858167+0000","last_became_peered":"2024-09-18T07:32:30.858167+0000","last_unstale":"2024-09-18T07:34:41.213681+0000","last_undegraded":"2024-09-18T07:34:41.213681+0000","last_fullsized":"2024-09-18T07:34:41.213681+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:58.276259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:58.276259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:27.549455+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210250+0000","last_change":"2024-09-18T07:32:30.835415+0000","last_active":"2024-09-18T07:34:41.210250+0000","last_peered":"2024-09-18T07:34:41.210250+0000","last_clean":"2024-09-18T07:34:41.210250+0000","last_became_active":"2024-09-18T07:32:30.834708+0000","last_became_peered":"2024-09-18T07:32:30.834708+0000","last_unstale":"2024-09-18T07:34:41.210250+0000","last_undegraded":"2024-09-18T07:34:41.210250+0000","last_fullsized":"2024-09-18T07:34:41.210250+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:07.940458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"153'9056","reported_seq":15515,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.698472+0000","last_change":"2024-09-18T07:34:34.346573+0000","last_active":"2024-09-18T07:34:41.698472+0000","last_peered":"2024-09-18T07:34:41.698472+0000","last_clean":"2024-09-18T07:34:41.698472+0000","last_became_active":"2024-09-18T07:34:31.154550+0000","last_became_peered":"2024-09-18T07:34:31.154550+0000","last_unstale":"2024-09-18T07:34:41.698472+0000","last_undegraded":"2024-09-18T07:34:41.698472+0000","last_fullsized":"2024-09-18T07:34:41.698472+0000","mapping_epoch":143,"log_start":"109'3790","ondisk_log_start":"109'3790","created":134,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":5,"last_scrub":"146'8953","last_scrub_stamp":"2024-09-18T07:34:34.346288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:34.346288+0000","objects_scrubbed":1899,"log_size":5266,"log_dups_size":0,"ondisk_log_size":5266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:39.493302+0000","scrub_duration":654,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117637120,"num_objects":1795,"num_object_clones":0,"num_object_copies":5385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":225,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.222107+0000","last_change":"2024-09-18T07:32:30.851755+0000","last_active":"2024-09-18T07:34:41.222107+0000","last_peered":"2024-09-18T07:34:41.222107+0000","last_clean":"2024-09-18T07:34:41.222107+0000","last_became_active":"2024-09-18T07:32:30.837622+0000","last_became_peered":"2024-09-18T07:32:30.837622+0000","last_unstale":"2024-09-18T07:34:41.222107+0000","last_undegraded":"2024-09-18T07:34:41.222107+0000","last_fullsized":"2024-09-18T07:34:41.222107+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:12.441533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":211,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.236674+0000","last_change":"2024-09-18T07:34:35.707200+0000","last_active":"2024-09-18T07:34:41.236674+0000","last_peered":"2024-09-18T07:34:41.236674+0000","last_clean":"2024-09-18T07:34:41.236674+0000","last_became_active":"2024-09-18T07:32:30.853969+0000","last_became_peered":"2024-09-18T07:32:30.853969+0000","last_unstale":"2024-09-18T07:34:41.236674+0000","last_undegraded":"2024-09-18T07:34:41.236674+0000","last_fullsized":"2024-09-18T07:34:41.236674+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:35.707148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:35.707148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:43.973295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"94'9","reported_seq":253,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.212906+0000","last_change":"2024-09-18T07:32:30.867244+0000","last_active":"2024-09-18T07:34:41.212906+0000","last_peered":"2024-09-18T07:34:41.212906+0000","last_clean":"2024-09-18T07:34:41.212906+0000","last_became_active":"2024-09-18T07:32:30.848580+0000","last_became_peered":"2024-09-18T07:32:30.848580+0000","last_unstale":"2024-09-18T07:34:41.212906+0000","last_undegraded":"2024-09-18T07:34:41.212906+0000","last_fullsized":"2024-09-18T07:34:41.212906+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":185,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211461+0000","last_change":"2024-09-18T07:31:02.343206+0000","last_active":"2024-09-18T07:34:41.211461+0000","last_peered":"2024-09-18T07:34:41.211461+0000","last_clean":"2024-09-18T07:34:41.211461+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:34:41.211461+0000","last_undegraded":"2024-09-18T07:34:41.211461+0000","last_fullsized":"2024-09-18T07:34:41.211461+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":234,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.213323+0000","last_change":"2024-09-18T07:33:54.295407+0000","last_active":"2024-09-18T07:34:41.213323+0000","last_peered":"2024-09-18T07:34:41.213323+0000","last_clean":"2024-09-18T07:34:41.213323+0000","last_became_active":"2024-09-18T07:32:30.848136+0000","last_became_peered":"2024-09-18T07:32:30.848136+0000","last_unstale":"2024-09-18T07:34:41.213323+0000","last_undegraded":"2024-09-18T07:34:41.213323+0000","last_fullsized":"2024-09-18T07:34:41.213323+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:33:54.295330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:54.295330+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:20.154237+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":140,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211136+0000","last_change":"2024-09-18T07:34:31.934939+0000","last_active":"2024-09-18T07:34:41.211136+0000","last_peered":"2024-09-18T07:34:41.211136+0000","last_clean":"2024-09-18T07:34:41.211136+0000","last_became_active":"2024-09-18T07:32:30.832312+0000","last_became_peered":"2024-09-18T07:32:30.832312+0000","last_unstale":"2024-09-18T07:34:41.211136+0000","last_undegraded":"2024-09-18T07:34:41.211136+0000","last_fullsized":"2024-09-18T07:34:41.211136+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:31.934890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.934890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:45.059706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":245,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.222433+0000","last_change":"2024-09-18T07:31:00.332729+0000","last_active":"2024-09-18T07:34:41.222433+0000","last_peered":"2024-09-18T07:34:41.222433+0000","last_clean":"2024-09-18T07:34:41.222433+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:34:41.222433+0000","last_undegraded":"2024-09-18T07:34:41.222433+0000","last_fullsized":"2024-09-18T07:34:41.222433+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:14.413751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":199,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.236459+0000","last_change":"2024-09-18T07:33:32.135015+0000","last_active":"2024-09-18T07:34:41.236459+0000","last_peered":"2024-09-18T07:34:41.236459+0000","last_clean":"2024-09-18T07:34:41.236459+0000","last_became_active":"2024-09-18T07:32:30.854291+0000","last_became_peered":"2024-09-18T07:32:30.854291+0000","last_unstale":"2024-09-18T07:34:41.236459+0000","last_undegraded":"2024-09-18T07:34:41.236459+0000","last_fullsized":"2024-09-18T07:34:41.236459+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.134847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.134847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"153'8880","reported_seq":9086,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.438683+0000","last_change":"2024-09-18T07:34:32.908699+0000","last_active":"2024-09-18T07:34:41.438683+0000","last_peered":"2024-09-18T07:34:41.438683+0000","last_clean":"2024-09-18T07:34:41.438683+0000","last_became_active":"2024-09-18T07:34:32.239984+0000","last_became_peered":"2024-09-18T07:34:32.239984+0000","last_unstale":"2024-09-18T07:34:41.438683+0000","last_undegraded":"2024-09-18T07:34:41.438683+0000","last_fullsized":"2024-09-18T07:34:41.438683+0000","mapping_epoch":144,"log_start":"100'3056","ondisk_log_start":"100'3056","created":134,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":5,"last_scrub":"145'8753","last_scrub_stamp":"2024-09-18T07:34:32.908529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:32.908529+0000","objects_scrubbed":1965,"log_size":5824,"log_dups_size":0,"ondisk_log_size":5824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:51.391130+0000","scrub_duration":456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119865344,"num_objects":1829,"num_object_clones":0,"num_object_copies":5487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":136,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":212,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211037+0000","last_change":"2024-09-18T07:30:58.314730+0000","last_active":"2024-09-18T07:34:41.211037+0000","last_peered":"2024-09-18T07:34:41.211037+0000","last_clean":"2024-09-18T07:34:41.211037+0000","last_became_active":"2024-09-18T07:30:58.314428+0000","last_became_peered":"2024-09-18T07:30:58.314428+0000","last_unstale":"2024-09-18T07:34:41.211037+0000","last_undegraded":"2024-09-18T07:34:41.211037+0000","last_fullsized":"2024-09-18T07:34:41.211037+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":204,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211259+0000","last_change":"2024-09-18T07:32:30.831845+0000","last_active":"2024-09-18T07:34:41.211259+0000","last_peered":"2024-09-18T07:34:41.211259+0000","last_clean":"2024-09-18T07:34:41.211259+0000","last_became_active":"2024-09-18T07:32:30.831546+0000","last_became_peered":"2024-09-18T07:32:30.831546+0000","last_unstale":"2024-09-18T07:34:41.211259+0000","last_undegraded":"2024-09-18T07:34:41.211259+0000","last_fullsized":"2024-09-18T07:34:41.211259+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":189,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.209277+0000","last_change":"2024-09-18T07:31:00.337826+0000","last_active":"2024-09-18T07:34:41.209277+0000","last_peered":"2024-09-18T07:34:41.209277+0000","last_clean":"2024-09-18T07:34:41.209277+0000","last_became_active":"2024-09-18T07:31:00.334158+0000","last_became_peered":"2024-09-18T07:31:00.334158+0000","last_unstale":"2024-09-18T07:34:41.209277+0000","last_undegraded":"2024-09-18T07:34:41.209277+0000","last_fullsized":"2024-09-18T07:34:41.209277+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":184,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211642+0000","last_change":"2024-09-18T07:31:02.351247+0000","last_active":"2024-09-18T07:34:41.211642+0000","last_peered":"2024-09-18T07:34:41.211642+0000","last_clean":"2024-09-18T07:34:41.211642+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:34:41.211642+0000","last_undegraded":"2024-09-18T07:34:41.211642+0000","last_fullsized":"2024-09-18T07:34:41.211642+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:49.189968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"153'9079","reported_seq":13279,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.420531+0000","last_change":"2024-09-18T07:34:33.345888+0000","last_active":"2024-09-18T07:34:41.420531+0000","last_peered":"2024-09-18T07:34:41.420531+0000","last_clean":"2024-09-18T07:34:41.420531+0000","last_became_active":"2024-09-18T07:34:32.239051+0000","last_became_peered":"2024-09-18T07:34:32.239051+0000","last_unstale":"2024-09-18T07:34:41.420531+0000","last_undegraded":"2024-09-18T07:34:41.420531+0000","last_fullsized":"2024-09-18T07:34:41.420531+0000","mapping_epoch":144,"log_start":"113'4100","ondisk_log_start":"113'4100","created":134,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":5,"last_scrub":"145'8947","last_scrub_stamp":"2024-09-18T07:34:33.345727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:33.345727+0000","objects_scrubbed":1966,"log_size":4979,"log_dups_size":0,"ondisk_log_size":4979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:44.456554+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119865344,"num_objects":1829,"num_object_clones":0,"num_object_copies":5487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":137,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":88,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.236365+0000","last_change":"2024-09-18T07:33:27.033510+0000","last_active":"2024-09-18T07:34:41.236365+0000","last_peered":"2024-09-18T07:34:41.236365+0000","last_clean":"2024-09-18T07:34:41.236365+0000","last_became_active":"2024-09-18T07:33:27.003887+0000","last_became_peered":"2024-09-18T07:33:27.003887+0000","last_unstale":"2024-09-18T07:34:41.236365+0000","last_undegraded":"2024-09-18T07:34:41.236365+0000","last_fullsized":"2024-09-18T07:34:41.236365+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.187186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.187186+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":140,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.213378+0000","last_change":"2024-09-18T07:34:34.322733+0000","last_active":"2024-09-18T07:34:41.213378+0000","last_peered":"2024-09-18T07:34:41.213378+0000","last_clean":"2024-09-18T07:34:41.213378+0000","last_became_active":"2024-09-18T07:32:30.847608+0000","last_became_peered":"2024-09-18T07:32:30.847608+0000","last_unstale":"2024-09-18T07:34:41.213378+0000","last_undegraded":"2024-09-18T07:34:41.213378+0000","last_fullsized":"2024-09-18T07:34:41.213378+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:34.322659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:34.322659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:35.712663+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":188,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.212133+0000","last_change":"2024-09-18T07:31:00.333689+0000","last_active":"2024-09-18T07:34:41.212133+0000","last_peered":"2024-09-18T07:34:41.212133+0000","last_clean":"2024-09-18T07:34:41.212133+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:34:41.212133+0000","last_undegraded":"2024-09-18T07:34:41.212133+0000","last_fullsized":"2024-09-18T07:34:41.212133+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.167699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.233889+0000","last_change":"2024-09-18T07:32:04.503356+0000","last_active":"2024-09-18T07:34:41.233889+0000","last_peered":"2024-09-18T07:34:41.233889+0000","last_clean":"2024-09-18T07:34:41.233889+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:34:41.233889+0000","last_undegraded":"2024-09-18T07:34:41.233889+0000","last_fullsized":"2024-09-18T07:34:41.233889+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.503270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.503270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"153'8903","reported_seq":9107,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.523168+0000","last_change":"2024-09-18T07:34:32.972472+0000","last_active":"2024-09-18T07:34:41.523168+0000","last_peered":"2024-09-18T07:34:41.523168+0000","last_clean":"2024-09-18T07:34:41.523168+0000","last_became_active":"2024-09-18T07:34:32.239727+0000","last_became_peered":"2024-09-18T07:34:32.239727+0000","last_unstale":"2024-09-18T07:34:41.523168+0000","last_undegraded":"2024-09-18T07:34:41.523168+0000","last_fullsized":"2024-09-18T07:34:41.523168+0000","mapping_epoch":144,"log_start":"113'4100","ondisk_log_start":"113'4100","created":134,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":5,"last_scrub":"145'8786","last_scrub_stamp":"2024-09-18T07:34:32.972183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:32.972183+0000","objects_scrubbed":1883,"log_size":4803,"log_dups_size":0,"ondisk_log_size":4803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:52.658255+0000","scrub_duration":687,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115146752,"num_objects":1757,"num_object_clones":0,"num_object_copies":5271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":230,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.211746+0000","last_change":"2024-09-18T07:32:30.869693+0000","last_active":"2024-09-18T07:34:41.211746+0000","last_peered":"2024-09-18T07:34:41.211746+0000","last_clean":"2024-09-18T07:34:41.211746+0000","last_became_active":"2024-09-18T07:32:30.842218+0000","last_became_peered":"2024-09-18T07:32:30.842218+0000","last_unstale":"2024-09-18T07:34:41.211746+0000","last_undegraded":"2024-09-18T07:34:41.211746+0000","last_fullsized":"2024-09-18T07:34:41.211746+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:43.146132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":211,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.221998+0000","last_change":"2024-09-18T07:32:30.853892+0000","last_active":"2024-09-18T07:34:41.221998+0000","last_peered":"2024-09-18T07:34:41.221998+0000","last_clean":"2024-09-18T07:34:41.221998+0000","last_became_active":"2024-09-18T07:32:30.853486+0000","last_became_peered":"2024-09-18T07:32:30.853486+0000","last_unstale":"2024-09-18T07:34:41.221998+0000","last_undegraded":"2024-09-18T07:34:41.221998+0000","last_fullsized":"2024-09-18T07:34:41.221998+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:58.734027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:58.734027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:42.752966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":139,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.210391+0000","last_change":"2024-09-18T07:33:30.668600+0000","last_active":"2024-09-18T07:34:41.210391+0000","last_peered":"2024-09-18T07:34:41.210391+0000","last_clean":"2024-09-18T07:34:41.210391+0000","last_became_active":"2024-09-18T07:32:30.843194+0000","last_became_peered":"2024-09-18T07:32:30.843194+0000","last_unstale":"2024-09-18T07:34:41.210391+0000","last_undegraded":"2024-09-18T07:34:41.210391+0000","last_fullsized":"2024-09-18T07:34:41.210391+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.668428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.668428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:55.997265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.212211+0000","last_change":"2024-09-18T07:33:30.975087+0000","last_active":"2024-09-18T07:34:41.212211+0000","last_peered":"2024-09-18T07:34:41.212211+0000","last_clean":"2024-09-18T07:34:41.212211+0000","last_became_active":"2024-09-18T07:31:02.490994+0000","last_became_peered":"2024-09-18T07:31:02.490994+0000","last_unstale":"2024-09-18T07:34:41.212211+0000","last_undegraded":"2024-09-18T07:34:41.212211+0000","last_fullsized":"2024-09-18T07:34:41.212211+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.974886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.974886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"153'8806","reported_seq":9019,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.601287+0000","last_change":"2024-09-18T07:34:33.821081+0000","last_active":"2024-09-18T07:34:41.601287+0000","last_peered":"2024-09-18T07:34:41.601287+0000","last_clean":"2024-09-18T07:34:41.601287+0000","last_became_active":"2024-09-18T07:34:32.246591+0000","last_became_peered":"2024-09-18T07:34:32.246591+0000","last_unstale":"2024-09-18T07:34:41.601287+0000","last_undegraded":"2024-09-18T07:34:41.601287+0000","last_fullsized":"2024-09-18T07:34:41.601287+0000","mapping_epoch":144,"log_start":"113'4000","ondisk_log_start":"113'4000","created":134,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":5,"last_scrub":"145'8709","last_scrub_stamp":"2024-09-18T07:34:33.820793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:33.820793+0000","objects_scrubbed":1895,"log_size":4806,"log_dups_size":0,"ondisk_log_size":4806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:59.989740+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117047296,"num_objects":1786,"num_object_clones":0,"num_object_copies":5358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":219,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.212612+0000","last_change":"2024-09-18T07:33:29.988707+0000","last_active":"2024-09-18T07:34:41.212612+0000","last_peered":"2024-09-18T07:34:41.212612+0000","last_clean":"2024-09-18T07:34:41.212612+0000","last_became_active":"2024-09-18T07:32:26.791562+0000","last_became_peered":"2024-09-18T07:32:26.791562+0000","last_unstale":"2024-09-18T07:34:41.212612+0000","last_undegraded":"2024-09-18T07:34:41.212612+0000","last_fullsized":"2024-09-18T07:34:41.212612+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:33:29.988636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.988636+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":196,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.237661+0000","last_change":"2024-09-18T07:30:56.283643+0000","last_active":"2024-09-18T07:34:41.237661+0000","last_peered":"2024-09-18T07:34:41.237661+0000","last_clean":"2024-09-18T07:34:41.237661+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:34:41.237661+0000","last_undegraded":"2024-09-18T07:34:41.237661+0000","last_fullsized":"2024-09-18T07:34:41.237661+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":203,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.222800+0000","last_change":"2024-09-18T07:32:31.593581+0000","last_active":"2024-09-18T07:34:41.222800+0000","last_peered":"2024-09-18T07:34:41.222800+0000","last_clean":"2024-09-18T07:34:41.222800+0000","last_became_active":"2024-09-18T07:32:30.841210+0000","last_became_peered":"2024-09-18T07:32:30.841210+0000","last_unstale":"2024-09-18T07:34:41.222800+0000","last_undegraded":"2024-09-18T07:34:41.222800+0000","last_fullsized":"2024-09-18T07:34:41.222800+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:31.593520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:31.593520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:50.708269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":139,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.209101+0000","last_change":"2024-09-18T07:33:36.017902+0000","last_active":"2024-09-18T07:34:41.209101+0000","last_peered":"2024-09-18T07:34:41.209101+0000","last_clean":"2024-09-18T07:34:41.209101+0000","last_became_active":"2024-09-18T07:32:30.850481+0000","last_became_peered":"2024-09-18T07:32:30.850481+0000","last_unstale":"2024-09-18T07:34:41.209101+0000","last_undegraded":"2024-09-18T07:34:41.209101+0000","last_fullsized":"2024-09-18T07:34:41.209101+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:36.017602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.017602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:46.466305+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":193,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.209683+0000","last_change":"2024-09-18T07:33:24.053054+0000","last_active":"2024-09-18T07:34:41.209683+0000","last_peered":"2024-09-18T07:34:41.209683+0000","last_clean":"2024-09-18T07:34:41.209683+0000","last_became_active":"2024-09-18T07:31:02.347537+0000","last_became_peered":"2024-09-18T07:31:02.347537+0000","last_unstale":"2024-09-18T07:34:41.209683+0000","last_undegraded":"2024-09-18T07:34:41.209683+0000","last_fullsized":"2024-09-18T07:34:41.209683+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:24.052908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:24.052908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":210,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.212847+0000","last_change":"2024-09-18T07:33:33.072569+0000","last_active":"2024-09-18T07:34:41.212847+0000","last_peered":"2024-09-18T07:34:41.212847+0000","last_clean":"2024-09-18T07:34:41.212847+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:34:41.212847+0000","last_undegraded":"2024-09-18T07:34:41.212847+0000","last_fullsized":"2024-09-18T07:34:41.212847+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:33.072487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.072487+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:50.254990+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":219,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.223977+0000","last_change":"2024-09-18T07:33:27.742768+0000","last_active":"2024-09-18T07:34:41.223977+0000","last_peered":"2024-09-18T07:34:41.223977+0000","last_clean":"2024-09-18T07:34:41.223977+0000","last_became_active":"2024-09-18T07:30:58.313556+0000","last_became_peered":"2024-09-18T07:30:58.313556+0000","last_unstale":"2024-09-18T07:34:41.223977+0000","last_undegraded":"2024-09-18T07:34:41.223977+0000","last_fullsized":"2024-09-18T07:34:41.223977+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:33:27.742684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.742684+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:51.970382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":203,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-18T07:34:41.237435+0000","last_change":"2024-09-18T07:33:22.161956+0000","last_active":"2024-09-18T07:34:41.237435+0000","last_peered":"2024-09-18T07:34:41.237435+0000","last_clean":"2024-09-18T07:34:41.237435+0000","last_became_active":"2024-09-18T07:32:30.846635+0000","last_became_peered":"2024-09-18T07:32:30.846635+0000","last_unstale":"2024-09-18T07:34:41.237435+0000","last_undegraded":"2024-09-18T07:34:41.237435+0000","last_fullsized":"2024-09-18T07:34:41.237435+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.161894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.161894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":23,"stat_sum":{"num_bytes":3578200088,"num_objects":54600,"num_object_clones":0,"num_object_copies":163800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54600,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":59863,"num_write_kb":2499585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3852,"num_bytes_recovered":252379136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11176259584,"data_stored":11176247368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108918,"ondisk_log_size":108918,"up":69,"acting":69,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":1584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":137,"num_read_kb":132,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":236223201329,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462967,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1237464,"kb_used_data":1121604,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":92507688,"statfs":{"total":95995035648,"available":94727872512,"internally_reserved":0,"allocated":1148522496,"data_stored":1148064326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":118618566},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724604,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1244196,"kb_used_data":1132496,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92500956,"statfs":{"total":95995035648,"available":94720978944,"internally_reserved":0,"allocated":1159675904,"data_stored":1159212421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953539,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1528300,"kb_used_data":1402712,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92216852,"statfs":{"total":95995035648,"available":94430056448,"internally_reserved":0,"allocated":1436377088,"data_stored":1435898898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2451284,"kb_used_data":2297848,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":91293868,"statfs":{"total":95995035648,"available":93484920832,"internally_reserved":0,"allocated":2352996352,"data_stored":2352494345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":157022663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1767700,"kb_used_data":1642108,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":91977452,"statfs":{"total":95995035648,"available":94184910848,"internally_reserved":0,"allocated":1681518592,"data_stored":1681068304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1714664,"kb_used_data":1597392,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92030488,"statfs":{"total":95995035648,"available":94239219712,"internally_reserved":0,"allocated":1635729408,"data_stored":1635196381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1939980,"kb_used_data":1818544,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":91805172,"statfs":{"total":95995035648,"available":94008496128,"internally_reserved":0,"allocated":1862189056,"data_stored":1861695171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1847070720,"data_stored":1847066648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1621037056,"data_stored":1621032984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1660354560,"data_stored":1660354560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2323382272,"data_stored":2323382272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1424818176,"data_stored":1424818176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1156448256,"data_stored":1156448256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1139675136,"data_stored":1139671064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3473408,"data_stored":3473408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:34:47.193 INFO:tasks.thrashosds.thrasher:Setting 5.2 to [6, 5, 7, 2, 3, 0] 2024-09-18T07:34:47.193 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '5.2', '6', '5', '7', '2', '3', '0'] 2024-09-18T07:34:47.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 5.2 6 5 7 2 3 0 2024-09-18T07:34:47.274 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:34:47.285 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:47.309 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:47.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:47.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:47.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:34:47.410 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:47.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:34:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:34:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:34:47.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:47 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"}]': finished 2024-09-18T07:34:47.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:47 smithi137 ceph-mon[29408]: osdmap e158: 8 total, 8 up, 7 in 2024-09-18T07:34:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:47 smithi137 ceph-mon[29408]: from='client.25006 v1:172.21.15.137:0/3472657083' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:34:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:47 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1354012438' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:47 smithi137 ceph-mon[29408]: from='client.34979 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:47 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:47 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"}]': finished 2024-09-18T07:34:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:47 smithi137 ceph-mon[39175]: osdmap e158: 8 total, 8 up, 7 in 2024-09-18T07:34:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:47 smithi137 ceph-mon[39175]: from='client.25006 v1:172.21.15.137:0/3472657083' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:34:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:47 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1354012438' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:47 smithi137 ceph-mon[39175]: from='client.34979 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:47.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:47 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:47.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:47.451 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:47.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:47.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:34:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:34:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:47 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"}]': finished 2024-09-18T07:34:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:47 smithi203 ceph-mon[32237]: osdmap e158: 8 total, 8 up, 7 in 2024-09-18T07:34:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:47 smithi203 ceph-mon[32237]: from='client.25006 v1:172.21.15.137:0/3472657083' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:34:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:47 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1354012438' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:47 smithi203 ceph-mon[32237]: from='client.34979 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:34:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:47 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:47.552 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:47.575 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:47.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:47.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:47.677 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:47.700 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:47.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:47.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:47.802 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:47.827 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:47.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:47.928 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:47.951 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:47.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:47.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:48.053 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:48.076 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:48.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:48.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:48.178 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:48.202 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:48.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:48.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:48.208 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:34:48.208 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:34:48.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:48.328 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:48.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:48.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:48.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:48 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:48.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:48 smithi137 ceph-mon[29408]: from='client.34979 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:34:48.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:48 smithi137 ceph-mon[29408]: osdmap e159: 8 total, 8 up, 7 in 2024-09-18T07:34:48.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:48 smithi137 ceph-mon[29408]: pgmap v190: 155 pgs: 2 peering, 153 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 870 op/s 2024-09-18T07:34:48.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:48 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:48.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:48 smithi137 ceph-mon[39175]: from='client.34979 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:34:48.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:48 smithi137 ceph-mon[39175]: osdmap e159: 8 total, 8 up, 7 in 2024-09-18T07:34:48.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:48 smithi137 ceph-mon[39175]: pgmap v190: 155 pgs: 2 peering, 153 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 870 op/s 2024-09-18T07:34:48.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:48.452 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:48.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:48.553 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:48.574 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:48.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:48.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:48.676 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:48.700 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:48.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:48.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:48.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:48 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:48 smithi203 ceph-mon[32237]: from='client.34979 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:34:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:48 smithi203 ceph-mon[32237]: osdmap e159: 8 total, 8 up, 7 in 2024-09-18T07:34:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:48 smithi203 ceph-mon[32237]: pgmap v190: 155 pgs: 2 peering, 153 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 870 op/s 2024-09-18T07:34:48.802 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:48.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:48.827 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:48.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:48.882 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T07:34:48.928 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:48.951 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:48.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:49.053 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:49.077 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:49.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:49.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:49.178 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:49.202 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:49.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:49.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:49.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:49.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:49.338 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:49.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:49.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:49 smithi137 ceph-mon[29408]: osdmap e160: 8 total, 8 up, 7 in 2024-09-18T07:34:49.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:49 smithi137 ceph-mon[39175]: osdmap e160: 8 total, 8 up, 7 in 2024-09-18T07:34:49.439 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:49.463 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:49.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:49.520 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:49.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:49.613 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:49.643 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:49.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:49.644 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:49.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:49.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:49 smithi203 ceph-mon[32237]: osdmap e160: 8 total, 8 up, 7 in 2024-09-18T07:34:49.745 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:49.788 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:49.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:49.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:49.890 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:49.938 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:49.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:49.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:50.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:34:50.040 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:50.064 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:50.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:50.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:50.166 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:50.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:50.215 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:50.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:50.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:50.359 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:50.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:50.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:50.461 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:50.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:50.483 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:50.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:50.506 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:50 smithi137 ceph-mon[29408]: osdmap e161: 8 total, 8 up, 7 in 2024-09-18T07:34:50.507 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:50 smithi137 ceph-mon[29408]: pgmap v193: 154 pgs: 1 peering, 153 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 848 op/s 2024-09-18T07:34:50.507 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:50 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-18T07:34:50.507 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:50 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-18T07:34:50.507 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:50 smithi137 ceph-mon[39175]: osdmap e161: 8 total, 8 up, 7 in 2024-09-18T07:34:50.507 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:50 smithi137 ceph-mon[39175]: pgmap v193: 154 pgs: 1 peering, 153 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 848 op/s 2024-09-18T07:34:50.507 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:50 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-18T07:34:50.507 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:50 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-18T07:34:50.585 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:50.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:50.607 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:50.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:50.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:50.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:50 smithi203 ceph-mon[32237]: osdmap e161: 8 total, 8 up, 7 in 2024-09-18T07:34:50.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:50 smithi203 ceph-mon[32237]: pgmap v193: 154 pgs: 1 peering, 153 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 848 op/s 2024-09-18T07:34:50.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:50 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-18T07:34:50.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:50 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-18T07:34:50.729 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:50.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:50.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:50.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:50.853 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:50.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:50.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:50.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:50.978 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:50.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:50.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:51.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:51.101 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:51.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:51.204 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:51.226 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:51.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:51.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:51.315 INFO:teuthology.orchestra.run.smithi137.stderr:set 5.2 pg_upmap_items mapping to [6->5,7->2,3->0] 2024-09-18T07:34:51.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:51.349 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:51.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:51.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:51.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:51 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-18T07:34:51.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:51 smithi137 ceph-mon[39175]: osdmap e162: 8 total, 8 up, 7 in 2024-09-18T07:34:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:51 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:51 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:51 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:34:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:51 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:34:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:51 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:51 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/226961627' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "5.2", "id": [6, 5, 7, 2, 3, 0]} : dispatch 2024-09-18T07:34:51.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:51 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-18T07:34:51.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:51 smithi137 ceph-mon[29408]: osdmap e162: 8 total, 8 up, 7 in 2024-09-18T07:34:51.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:51 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:51.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:51 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:51.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:51 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:34:51.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:51 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:34:51.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:51 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:51.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:51 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/226961627' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "5.2", "id": [6, 5, 7, 2, 3, 0]} : dispatch 2024-09-18T07:34:51.450 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:51.474 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:51.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:51.575 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:34:51.576 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:34:51.576 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:34:51.576 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:34:51.576 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:34:51.576 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:51.599 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:51.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:51.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:51.611 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:51 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-18T07:34:51.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:51 smithi203 ceph-mon[32237]: osdmap e162: 8 total, 8 up, 7 in 2024-09-18T07:34:51.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:51 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:51.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:51 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:51.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:51 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:34:51.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:51 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:34:51.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:51 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:51.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:51 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/226961627' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "5.2", "id": [6, 5, 7, 2, 3, 0]} : dispatch 2024-09-18T07:34:51.701 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:51.723 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:51.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:51.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:51.825 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:51.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:51.847 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:51.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:51.948 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:51.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:51.971 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:51.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:52.072 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:52.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:52.094 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:52.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:52.122 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T07:34:52.196 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:52.217 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:52.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:52.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:52.318 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:52.341 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:52.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:52.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:52 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/226961627' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.2", "id": [6, 5, 7, 2, 3, 0]}]': finished 2024-09-18T07:34:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:52 smithi137 ceph-mon[29408]: osdmap e163: 8 total, 8 up, 7 in 2024-09-18T07:34:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:52 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1404090967' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:52 smithi137 ceph-mon[29408]: pgmap v196: 154 pgs: 1 peering, 153 active+clean; 3.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 72 op/s 2024-09-18T07:34:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:52 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/226961627' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.2", "id": [6, 5, 7, 2, 3, 0]}]': finished 2024-09-18T07:34:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:52 smithi137 ceph-mon[39175]: osdmap e163: 8 total, 8 up, 7 in 2024-09-18T07:34:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:52 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1404090967' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:52 smithi137 ceph-mon[39175]: pgmap v196: 154 pgs: 1 peering, 153 active+clean; 3.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 72 op/s 2024-09-18T07:34:52.442 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:52.464 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:52.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:52.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:52.490 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:34:52.490 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:34:52.490 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:34:52.490 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:34:52.490 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:34:52.490 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:34:52.490 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:34:52.490 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:34:52.490 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:34:52.491 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:34:52.491 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:34:52.491 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:34:52.491 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:34:52.491 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:34:52.491 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:34:52.491 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:34:52.566 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:52.587 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:52.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:52.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:52.689 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:52.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:52 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/226961627' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.2", "id": [6, 5, 7, 2, 3, 0]}]': finished 2024-09-18T07:34:52.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:52 smithi203 ceph-mon[32237]: osdmap e163: 8 total, 8 up, 7 in 2024-09-18T07:34:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:52 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1404090967' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:52 smithi203 ceph-mon[32237]: pgmap v196: 154 pgs: 1 peering, 153 active+clean; 3.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 72 op/s 2024-09-18T07:34:52.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:52.712 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:52.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:52.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:52.836 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:52.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:52.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:52.938 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:52.959 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:52.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:52.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:53.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:53.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:53.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:53.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:53.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:53.207 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:53.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:53.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:53.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:53.333 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:53.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:53.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:53.435 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:53.458 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:53.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:53.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:53.560 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:53.586 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:53.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:53.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:53.609 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:53 smithi137 ceph-mon[29408]: osdmap e164: 8 total, 8 up, 7 in 2024-09-18T07:34:53.609 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:53 smithi137 ceph-mon[39175]: osdmap e164: 8 total, 8 up, 7 in 2024-09-18T07:34:53.654 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:53.688 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:53.699 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:53.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:53 smithi203 ceph-mon[32237]: osdmap e164: 8 total, 8 up, 7 in 2024-09-18T07:34:53.713 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:53.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:53.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:53.815 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:53.841 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:53.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:53.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:53.944 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:53.971 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:53.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:54.072 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:54.097 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:54.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:54.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:54.199 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:54.224 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:54.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:54.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:54.326 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:54.351 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:54.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:54.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:54.475 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:54.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:54.556 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:54 smithi137 ceph-mon[29408]: osdmap e165: 8 total, 8 up, 7 in 2024-09-18T07:34:54.556 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:54 smithi137 ceph-mon[29408]: pgmap v199: 153 pgs: 1 clean+premerge+peered, 152 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 402 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:34:54.556 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:54 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-18T07:34:54.557 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:54 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-18T07:34:54.557 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:54 smithi137 ceph-mon[39175]: osdmap e165: 8 total, 8 up, 7 in 2024-09-18T07:34:54.557 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:54 smithi137 ceph-mon[39175]: pgmap v199: 153 pgs: 1 clean+premerge+peered, 152 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 402 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:34:54.557 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:54 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-18T07:34:54.557 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:54 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-18T07:34:54.577 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:54.598 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:54.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:54.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:54.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:54.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:54 smithi203 ceph-mon[32237]: osdmap e165: 8 total, 8 up, 7 in 2024-09-18T07:34:54.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:54 smithi203 ceph-mon[32237]: pgmap v199: 153 pgs: 1 clean+premerge+peered, 152 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 402 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:34:54.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:54 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-18T07:34:54.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:54 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"} : dispatch 2024-09-18T07:34:54.721 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:54.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:54.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:54.773 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:34:54.774 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:34:54.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:34:54.774 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:34:54.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.774 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:51270 7.3 7:c0637d4a:::benchmark_data_smithi137_74180_object51269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:34:54.774 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.036896+0000", 2024-09-18T07:34:54.774 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 124.724238887, 2024-09-18T07:34:54.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14513968099999999, 2024-09-18T07:34:54.774 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.774 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.774 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.775 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:54.775 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:54.775 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:54.775 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 51270 2024-09-18T07:34:54.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.775 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.036896+0000", 2024-09-18T07:34:54.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.036896+0000", 2024-09-18T07:34:54.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.036898+0000", 2024-09-18T07:34:54.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1380000000000001e-06 2024-09-18T07:34:54.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.036919+0000", 2024-09-18T07:34:54.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1509999999999999e-05 2024-09-18T07:34:54.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.036929+0000", 2024-09-18T07:34:54.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8139999999999995e-06 2024-09-18T07:34:54.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.036934+0000", 2024-09-18T07:34:54.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3750000000000002e-06 2024-09-18T07:34:54.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.036974+0000", 2024-09-18T07:34:54.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9292000000000003e-05 2024-09-18T07:34:54.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.037072+0000", 2024-09-18T07:34:54.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8060000000000006e-05 2024-09-18T07:34:54.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,4", 2024-09-18T07:34:54.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.037155+0000", 2024-09-18T07:34:54.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3220000000000006e-05 2024-09-18T07:34:54.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:54.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.038425+0000", 2024-09-18T07:34:54.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012695880000000001 2024-09-18T07:34:54.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.039360+0000", 2024-09-18T07:34:54.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00093562099999999998 2024-09-18T07:34:54.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.181937+0000", 2024-09-18T07:34:54.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14257709499999999 2024-09-18T07:34:54.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.182003+0000", 2024-09-18T07:34:54.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5606999999999997e-05 2024-09-18T07:34:54.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.182035+0000", 2024-09-18T07:34:54.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2360999999999997e-05 2024-09-18T07:34:54.780 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.780 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.781 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.781 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:51784 7.3 7:ccdb2d3f:::benchmark_data_smithi137_74180_object51783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:34:54.781 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.662878+0000", 2024-09-18T07:34:54.781 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 124.09825697700001, 2024-09-18T07:34:54.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.153506799, 2024-09-18T07:34:54.781 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.781 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.816 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.816 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:54.816 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:54.816 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:54.816 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 51784 2024-09-18T07:34:54.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.816 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.662878+0000", 2024-09-18T07:34:54.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.662878+0000", 2024-09-18T07:34:54.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.662879+0000", 2024-09-18T07:34:54.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.347e-06 2024-09-18T07:34:54.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.662900+0000", 2024-09-18T07:34:54.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1543e-05 2024-09-18T07:34:54.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.662909+0000", 2024-09-18T07:34:54.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.242e-06 2024-09-18T07:34:54.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.662915+0000", 2024-09-18T07:34:54.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3509999999999998e-06 2024-09-18T07:34:54.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.664142+0000", 2024-09-18T07:34:54.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001227398 2024-09-18T07:34:54.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.664242+0000", 2024-09-18T07:34:54.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9510000000000001e-05 2024-09-18T07:34:54.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,4", 2024-09-18T07:34:54.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.664376+0000", 2024-09-18T07:34:54.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000134506 2024-09-18T07:34:54.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667994+0000", 2024-09-18T07:34:54.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036176910000000001 2024-09-18T07:34:54.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:54.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.668014+0000", 2024-09-18T07:34:54.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0231e-05 2024-09-18T07:34:54.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.816297+0000", 2024-09-18T07:34:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14828264099999999 2024-09-18T07:34:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.816344+0000", 2024-09-18T07:34:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7445999999999999e-05 2024-09-18T07:34:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.816384+0000", 2024-09-18T07:34:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9892999999999999e-05 2024-09-18T07:34:54.822 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.822 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.822 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.822 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:51773 7.8 e100/94 7:1d9df492:::benchmark_data_smithi137_74180_object51772:head v 100'3323, mlcod=88'794)", 2024-09-18T07:34:54.822 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.666053+0000", 2024-09-18T07:34:54.823 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 124.09508187199999, 2024-09-18T07:34:54.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14866578999999999, 2024-09-18T07:34:54.823 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.823 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.823 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.823 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666053+0000", 2024-09-18T07:34:54.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666053+0000", 2024-09-18T07:34:54.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666053+0000", 2024-09-18T07:34:54.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2800000000000003e-07 2024-09-18T07:34:54.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666076+0000", 2024-09-18T07:34:54.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2617e-05 2024-09-18T07:34:54.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666081+0000", 2024-09-18T07:34:54.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0429999999999998e-06 2024-09-18T07:34:54.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666085+0000", 2024-09-18T07:34:54.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9269999999999998e-06 2024-09-18T07:34:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666560+0000", 2024-09-18T07:34:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00047523299999999998 2024-09-18T07:34:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666583+0000", 2024-09-18T07:34:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3298e-05 2024-09-18T07:34:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.814673+0000", 2024-09-18T07:34:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.148089526 2024-09-18T07:34:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.814718+0000", 2024-09-18T07:34:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5818000000000001e-05 2024-09-18T07:34:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:51786 7.2 7:432e2944:::benchmark_data_smithi137_74180_object51785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:34:54.827 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.666482+0000", 2024-09-18T07:34:54.828 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 124.094652801, 2024-09-18T07:34:54.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16421708800000001, 2024-09-18T07:34:54.828 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.828 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.828 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.828 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:54.828 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:54.828 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:54.828 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 51786 2024-09-18T07:34:54.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.828 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666482+0000", 2024-09-18T07:34:54.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666482+0000", 2024-09-18T07:34:54.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666484+0000", 2024-09-18T07:34:54.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.832e-06 2024-09-18T07:34:54.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666506+0000", 2024-09-18T07:34:54.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2058000000000001e-05 2024-09-18T07:34:54.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666512+0000", 2024-09-18T07:34:54.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8719999999999996e-06 2024-09-18T07:34:54.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666518+0000", 2024-09-18T07:34:54.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7150000000000003e-06 2024-09-18T07:34:54.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667073+0000", 2024-09-18T07:34:54.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00055530599999999999 2024-09-18T07:34:54.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667193+0000", 2024-09-18T07:34:54.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000119523 2024-09-18T07:34:54.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-18T07:34:54.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667316+0000", 2024-09-18T07:34:54.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012347700000000001 2024-09-18T07:34:54.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.823184+0000", 2024-09-18T07:34:54.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15586741600000001 2024-09-18T07:34:54.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.823221+0000", 2024-09-18T07:34:54.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7153000000000002e-05 2024-09-18T07:34:54.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:54.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.830604+0000", 2024-09-18T07:34:54.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0073824880000000004 2024-09-18T07:34:54.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.830656+0000", 2024-09-18T07:34:54.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1968000000000001e-05 2024-09-18T07:34:54.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.830699+0000", 2024-09-18T07:34:54.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3279999999999999e-05 2024-09-18T07:34:54.834 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.834 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.834 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.834 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:51787 7.2 7:41d0922e:::benchmark_data_smithi137_74180_object51786:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:34:54.834 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.666795+0000", 2024-09-18T07:34:54.834 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 124.094339727, 2024-09-18T07:34:54.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.164029129, 2024-09-18T07:34:54.834 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.834 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.834 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.835 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:54.835 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:54.835 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:54.835 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 51787 2024-09-18T07:34:54.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.835 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666795+0000", 2024-09-18T07:34:54.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666795+0000", 2024-09-18T07:34:54.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666796+0000", 2024-09-18T07:34:54.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4200000000000004e-07 2024-09-18T07:34:54.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666816+0000", 2024-09-18T07:34:54.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0661000000000001e-05 2024-09-18T07:34:54.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666824+0000", 2024-09-18T07:34:54.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5340000000000002e-06 2024-09-18T07:34:54.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.666830+0000", 2024-09-18T07:34:54.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7490000000000004e-06 2024-09-18T07:34:54.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.668911+0000", 2024-09-18T07:34:54.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020816089999999999 2024-09-18T07:34:54.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.669005+0000", 2024-09-18T07:34:54.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3225000000000003e-05 2024-09-18T07:34:54.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-18T07:34:54.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.815016+0000", 2024-09-18T07:34:54.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14601106899999999 2024-09-18T07:34:54.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.823251+0000", 2024-09-18T07:34:54.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082355559999999998 2024-09-18T07:34:54.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.823280+0000", 2024-09-18T07:34:54.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9128999999999998e-05 2024-09-18T07:34:54.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.830751+0000", 2024-09-18T07:34:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0074705090000000002 2024-09-18T07:34:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.830790+0000", 2024-09-18T07:34:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9607000000000002e-05 2024-09-18T07:34:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.830824+0000", 2024-09-18T07:34:54.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3538999999999999e-05 2024-09-18T07:34:54.841 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.841 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.841 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.841 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:51789 7.2 7:4d3b0bbf:::benchmark_data_smithi137_74180_object51788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:34:54.841 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.667080+0000", 2024-09-18T07:34:54.841 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 124.094054489, 2024-09-18T07:34:54.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16651132699999999, 2024-09-18T07:34:54.841 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.841 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.842 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.842 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:54.842 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:54.842 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:54.842 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 51789 2024-09-18T07:34:54.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.842 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667080+0000", 2024-09-18T07:34:54.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667080+0000", 2024-09-18T07:34:54.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667081+0000", 2024-09-18T07:34:54.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.294e-06 2024-09-18T07:34:54.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667105+0000", 2024-09-18T07:34:54.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3305000000000001e-05 2024-09-18T07:34:54.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667115+0000", 2024-09-18T07:34:54.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0271e-05 2024-09-18T07:34:54.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667122+0000", 2024-09-18T07:34:54.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0550000000000002e-06 2024-09-18T07:34:54.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.820133+0000", 2024-09-18T07:34:54.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15301097999999999 2024-09-18T07:34:54.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.820262+0000", 2024-09-18T07:34:54.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012933399999999999 2024-09-18T07:34:54.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-18T07:34:54.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.821021+0000", 2024-09-18T07:34:54.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00075895400000000005 2024-09-18T07:34:54.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.828725+0000", 2024-09-18T07:34:54.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0077036639999999998 2024-09-18T07:34:54.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:54.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.830830+0000", 2024-09-18T07:34:54.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0021048740000000001 2024-09-18T07:34:54.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.833499+0000", 2024-09-18T07:34:54.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026693670000000002 2024-09-18T07:34:54.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.833551+0000", 2024-09-18T07:34:54.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2249000000000001e-05 2024-09-18T07:34:54.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.833591+0000", 2024-09-18T07:34:54.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.998e-05 2024-09-18T07:34:54.847 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.847 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.847 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.848 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:51791 7.2 7:4b401dfb:::benchmark_data_smithi137_74180_object51790:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:34:54.848 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.667278+0000", 2024-09-18T07:34:54.848 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 124.09385690800001, 2024-09-18T07:34:54.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.177598219, 2024-09-18T07:34:54.848 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.848 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.848 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.848 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:54.848 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:54.848 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:54.849 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 51791 2024-09-18T07:34:54.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.849 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667278+0000", 2024-09-18T07:34:54.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667278+0000", 2024-09-18T07:34:54.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667279+0000", 2024-09-18T07:34:54.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8400000000000002e-07 2024-09-18T07:34:54.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667296+0000", 2024-09-18T07:34:54.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7812000000000002e-05 2024-09-18T07:34:54.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667304+0000", 2024-09-18T07:34:54.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4250000000000001e-06 2024-09-18T07:34:54.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667309+0000", 2024-09-18T07:34:54.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8239999999999999e-06 2024-09-18T07:34:54.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.824562+0000", 2024-09-18T07:34:54.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15725362600000001 2024-09-18T07:34:54.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.824685+0000", 2024-09-18T07:34:54.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012275400000000001 2024-09-18T07:34:54.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-18T07:34:54.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.826746+0000", 2024-09-18T07:34:54.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020614499999999998 2024-09-18T07:34:54.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.830740+0000", 2024-09-18T07:34:54.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039939370000000004 2024-09-18T07:34:54.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:54.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.830835+0000", 2024-09-18T07:34:54.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4371999999999997e-05 2024-09-18T07:34:54.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.844766+0000", 2024-09-18T07:34:54.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013931038 2024-09-18T07:34:54.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.844836+0000", 2024-09-18T07:34:54.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0043999999999995e-05 2024-09-18T07:34:54.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.844876+0000", 2024-09-18T07:34:54.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9953000000000002e-05 2024-09-18T07:34:54.854 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.854 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.854 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.854 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:51792 7.a 7:5ab6ed70:::benchmark_data_smithi137_74180_object51791:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:34:54.854 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.667318+0000", 2024-09-18T07:34:54.854 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 124.093816874, 2024-09-18T07:34:54.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17148329400000001, 2024-09-18T07:34:54.855 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.855 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.855 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.855 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:54.855 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:54.855 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:54.855 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 51792 2024-09-18T07:34:54.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.855 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667318+0000", 2024-09-18T07:34:54.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667318+0000", 2024-09-18T07:34:54.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667318+0000", 2024-09-18T07:34:54.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.39e-07 2024-09-18T07:34:54.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667333+0000", 2024-09-18T07:34:54.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4866e-05 2024-09-18T07:34:54.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667337+0000", 2024-09-18T07:34:54.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8759999999999998e-06 2024-09-18T07:34:54.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667340+0000", 2024-09-18T07:34:54.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.965e-06 2024-09-18T07:34:54.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.828756+0000", 2024-09-18T07:34:54.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.161416218 2024-09-18T07:34:54.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.828883+0000", 2024-09-18T07:34:54.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000127417 2024-09-18T07:34:54.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-18T07:34:54.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.829046+0000", 2024-09-18T07:34:54.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000162149 2024-09-18T07:34:54.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.833503+0000", 2024-09-18T07:34:54.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044575609999999996 2024-09-18T07:34:54.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:54.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.835701+0000", 2024-09-18T07:34:54.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0021974550000000001 2024-09-18T07:34:54.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.838708+0000", 2024-09-18T07:34:54.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003007825 2024-09-18T07:34:54.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.838761+0000", 2024-09-18T07:34:54.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2784000000000001e-05 2024-09-18T07:34:54.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.838801+0000", 2024-09-18T07:34:54.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9739e-05 2024-09-18T07:34:54.860 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.861 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.861 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.861 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:51788 7.1 e100/81 7:81b6d503:::benchmark_data_smithi137_74180_object51787:head v 100'3311, mlcod=100'3309)", 2024-09-18T07:34:54.861 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.667384+0000", 2024-09-18T07:34:54.861 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 124.093750087, 2024-09-18T07:34:54.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.146583449, 2024-09-18T07:34:54.861 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.861 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.861 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.861 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667384+0000", 2024-09-18T07:34:54.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667384+0000", 2024-09-18T07:34:54.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667385+0000", 2024-09-18T07:34:54.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3300000000000002e-07 2024-09-18T07:34:54.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667418+0000", 2024-09-18T07:34:54.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3414999999999998e-05 2024-09-18T07:34:54.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667423+0000", 2024-09-18T07:34:54.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9949999999999996e-06 2024-09-18T07:34:54.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667429+0000", 2024-09-18T07:34:54.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.135e-06 2024-09-18T07:34:54.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667473+0000", 2024-09-18T07:34:54.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4731000000000003e-05 2024-09-18T07:34:54.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.667492+0000", 2024-09-18T07:34:54.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8348000000000001e-05 2024-09-18T07:34:54.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.669037+0000", 2024-09-18T07:34:54.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001545278 2024-09-18T07:34:54.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.813968+0000", 2024-09-18T07:34:54.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14493091399999999 2024-09-18T07:34:54.865 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.865 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.865 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.866 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:51794 7.1 e100/81 7:8f789502:::benchmark_data_smithi137_74180_object51793:head v 100'3312, mlcod=100'3309)", 2024-09-18T07:34:54.866 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.669431+0000", 2024-09-18T07:34:54.866 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 124.09170359700001, 2024-09-18T07:34:54.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.149820431, 2024-09-18T07:34:54.866 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.866 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.866 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.866 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.669431+0000", 2024-09-18T07:34:54.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.669431+0000", 2024-09-18T07:34:54.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:54.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.669432+0000", 2024-09-18T07:34:54.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2640000000000001e-06 2024-09-18T07:34:54.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.669466+0000", 2024-09-18T07:34:54.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3589e-05 2024-09-18T07:34:54.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.815535+0000", 2024-09-18T07:34:54.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14606957700000001 2024-09-18T07:34:54.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.815545+0000", 2024-09-18T07:34:54.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0142e-05 2024-09-18T07:34:54.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.815598+0000", 2024-09-18T07:34:54.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2831000000000003e-05 2024-09-18T07:34:54.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.815618+0000", 2024-09-18T07:34:54.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9599999999999999e-05 2024-09-18T07:34:54.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.819210+0000", 2024-09-18T07:34:54.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035916949999999998 2024-09-18T07:34:54.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.819251+0000", 2024-09-18T07:34:54.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1733000000000002e-05 2024-09-18T07:34:54.871 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.871 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.871 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.871 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:51795 7.7 e100/81 7:ea3b9aab:::benchmark_data_smithi137_74180_object51794:head v 100'3187, mlcod=100'3185)", 2024-09-18T07:34:54.871 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.669903+0000", 2024-09-18T07:34:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 124.09123112, 2024-09-18T07:34:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.150332151, 2024-09-18T07:34:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.669903+0000", 2024-09-18T07:34:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.669903+0000", 2024-09-18T07:34:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.669904+0000", 2024-09-18T07:34:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2099999999999995e-07 2024-09-18T07:34:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.669957+0000", 2024-09-18T07:34:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2632e-05 2024-09-18T07:34:54.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.814888+0000", 2024-09-18T07:34:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14493097999999999 2024-09-18T07:34:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.814897+0000", 2024-09-18T07:34:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4490000000000005e-06 2024-09-18T07:34:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.816768+0000", 2024-09-18T07:34:54.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001870504 2024-09-18T07:34:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.816787+0000", 2024-09-18T07:34:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9589000000000002e-05 2024-09-18T07:34:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.820142+0000", 2024-09-18T07:34:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003354774 2024-09-18T07:34:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.820236+0000", 2024-09-18T07:34:54.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3301999999999995e-05 2024-09-18T07:34:54.876 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.876 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.876 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.876 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:74666 7.0 e116/81 7:058d7705:::benchmark_data_smithi137_74180_object74665:head v 116'4651, mlcod=116'4647)", 2024-09-18T07:34:54.876 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:16.166641+0000", 2024-09-18T07:34:54.876 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 98.594493216000004, 2024-09-18T07:34:54.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.150063793, 2024-09-18T07:34:54.876 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.877 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.877 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.877 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.166641+0000", 2024-09-18T07:34:54.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.166641+0000", 2024-09-18T07:34:54.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.166645+0000", 2024-09-18T07:34:54.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0060000000000003e-06 2024-09-18T07:34:54.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.166787+0000", 2024-09-18T07:34:54.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014217900000000001 2024-09-18T07:34:54.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.166793+0000", 2024-09-18T07:34:54.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5679999999999999e-06 2024-09-18T07:34:54.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.166799+0000", 2024-09-18T07:34:54.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0650000000000004e-06 2024-09-18T07:34:54.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.171548+0000", 2024-09-18T07:34:54.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047490709999999997 2024-09-18T07:34:54.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.171565+0000", 2024-09-18T07:34:54.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6365999999999999e-05 2024-09-18T07:34:54.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.316658+0000", 2024-09-18T07:34:54.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14509302800000001 2024-09-18T07:34:54.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.316705+0000", 2024-09-18T07:34:54.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7509999999999997e-05 2024-09-18T07:34:54.880 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.880 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.881 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.881 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:74664 7.8 e116/110 7:12a99db9:::benchmark_data_smithi137_74180_object74663:head v 116'4755, mlcod=116'4752)", 2024-09-18T07:34:54.881 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:16.168831+0000", 2024-09-18T07:34:54.881 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 98.592303623999996, 2024-09-18T07:34:54.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.145084772, 2024-09-18T07:34:54.881 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.881 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.881 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.881 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.168831+0000", 2024-09-18T07:34:54.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.168831+0000", 2024-09-18T07:34:54.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.168832+0000", 2024-09-18T07:34:54.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.528e-06 2024-09-18T07:34:54.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.168859+0000", 2024-09-18T07:34:54.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6183999999999999e-05 2024-09-18T07:34:54.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.168870+0000", 2024-09-18T07:34:54.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1201e-05 2024-09-18T07:34:54.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.168877+0000", 2024-09-18T07:34:54.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3490000000000003e-06 2024-09-18T07:34:54.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.173449+0000", 2024-09-18T07:34:54.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045717090000000002 2024-09-18T07:34:54.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.173467+0000", 2024-09-18T07:34:54.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8581999999999999e-05 2024-09-18T07:34:54.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.180219+0000", 2024-09-18T07:34:54.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0067517740000000003 2024-09-18T07:34:54.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:16.313916+0000", 2024-09-18T07:34:54.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.133696445 2024-09-18T07:34:54.885 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.885 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.885 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.886 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135011 7.14 7.3991ba34 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-18T07:34:54.886 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:15.993779+0000", 2024-09-18T07:34:54.886 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 38.767355064999997, 2024-09-18T07:34:54.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98326573299999998, 2024-09-18T07:34:54.886 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.886 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.886 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:34:54.886 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:54.886 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:54.886 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:54.886 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135011 2024-09-18T07:34:54.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.886 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993779+0000", 2024-09-18T07:34:54.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993779+0000", 2024-09-18T07:34:54.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993780+0000", 2024-09-18T07:34:54.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6899999999999996e-07 2024-09-18T07:34:54.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993784+0000", 2024-09-18T07:34:54.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6270000000000002e-06 2024-09-18T07:34:54.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993794+0000", 2024-09-18T07:34:54.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6889999999999999e-06 2024-09-18T07:34:54.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993799+0000", 2024-09-18T07:34:54.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1590000000000001e-06 2024-09-18T07:34:54.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993861+0000", 2024-09-18T07:34:54.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2421000000000004e-05 2024-09-18T07:34:54.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:54.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993869+0000", 2024-09-18T07:34:54.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1980000000000001e-06 2024-09-18T07:34:54.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.977033+0000", 2024-09-18T07:34:54.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98316372699999999 2024-09-18T07:34:54.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.977045+0000", 2024-09-18T07:34:54.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2043e-05 2024-09-18T07:34:54.890 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.890 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.890 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.891 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135018 7.10 7:0de93d52:::benchmark_data_smithi137_74180_object30602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-18T07:34:54.891 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:15.998577+0000", 2024-09-18T07:34:54.891 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 38.762557297999997, 2024-09-18T07:34:54.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1980523299999999, 2024-09-18T07:34:54.891 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.891 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.891 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.891 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:54.891 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:54.891 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:54.891 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135018 2024-09-18T07:34:54.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.892 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998577+0000", 2024-09-18T07:34:54.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998577+0000", 2024-09-18T07:34:54.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998579+0000", 2024-09-18T07:34:54.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.336e-06 2024-09-18T07:34:54.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998583+0000", 2024-09-18T07:34:54.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7190000000000001e-06 2024-09-18T07:34:54.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998593+0000", 2024-09-18T07:34:54.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4450000000000002e-06 2024-09-18T07:34:54.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998598+0000", 2024-09-18T07:34:54.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2329999999999997e-06 2024-09-18T07:34:54.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998649+0000", 2024-09-18T07:34:54.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0735999999999997e-05 2024-09-18T07:34:54.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:54.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998659+0000", 2024-09-18T07:34:54.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0156000000000001e-05 2024-09-18T07:34:54.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.970426+0000", 2024-09-18T07:34:54.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97176680299999996 2024-09-18T07:34:54.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:34:54.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.972639+0000", 2024-09-18T07:34:54.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022131070000000002 2024-09-18T07:34:54.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.183802+0000", 2024-09-18T07:34:54.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2111635359999999 2024-09-18T07:34:54.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.183932+0000", 2024-09-18T07:34:54.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000129686 2024-09-18T07:34:54.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2,5", 2024-09-18T07:34:54.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184136+0000", 2024-09-18T07:34:54.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000203885 2024-09-18T07:34:54.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:54.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185714+0000", 2024-09-18T07:34:54.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00157839 2024-09-18T07:34:54.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.188451+0000", 2024-09-18T07:34:54.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002736741 2024-09-18T07:34:54.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.188856+0000", 2024-09-18T07:34:54.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040506400000000001 2024-09-18T07:34:54.898 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:34:54.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.195963+0000", 2024-09-18T07:34:54.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0071068019999999997 2024-09-18T07:34:54.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.196200+0000", 2024-09-18T07:34:54.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023698600000000001 2024-09-18T07:34:54.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.196630+0000", 2024-09-18T07:34:54.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00042970500000000001 2024-09-18T07:34:54.900 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.900 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.900 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.900 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:142944 7.f e140/115 7:f392bc4c:::benchmark_data_smithi137_74180_object38528:head v 140'8969, mlcod=140'8967)", 2024-09-18T07:34:54.900 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:25.424727+0000", 2024-09-18T07:34:54.900 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 29.336407550000001, 2024-09-18T07:34:54.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.190748533, 2024-09-18T07:34:54.900 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.900 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.901 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.901 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.424727+0000", 2024-09-18T07:34:54.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.424727+0000", 2024-09-18T07:34:54.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.424728+0000", 2024-09-18T07:34:54.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.189e-06 2024-09-18T07:34:54.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.424735+0000", 2024-09-18T07:34:54.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.511e-06 2024-09-18T07:34:54.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.424744+0000", 2024-09-18T07:34:54.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.645e-06 2024-09-18T07:34:54.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.424751+0000", 2024-09-18T07:34:54.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5860000000000001e-06 2024-09-18T07:34:54.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.424903+0000", 2024-09-18T07:34:54.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000151986 2024-09-18T07:34:54.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.424923+0000", 2024-09-18T07:34:54.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0466999999999999e-05 2024-09-18T07:34:54.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.615425+0000", 2024-09-18T07:34:54.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.190502108 2024-09-18T07:34:54.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.615475+0000", 2024-09-18T07:34:54.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0040999999999997e-05 2024-09-18T07:34:54.904 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.905 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.905 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.905 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:142946 7.6 e140/134 7:6617a291:::benchmark_data_smithi137_74180_object38530:head v 140'8741, mlcod=140'8739)", 2024-09-18T07:34:54.905 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:25.425455+0000", 2024-09-18T07:34:54.905 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 29.335679842000001, 2024-09-18T07:34:54.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.190052784, 2024-09-18T07:34:54.905 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.905 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.905 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.905 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.425455+0000", 2024-09-18T07:34:54.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.425455+0000", 2024-09-18T07:34:54.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.425456+0000", 2024-09-18T07:34:54.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9000000000000005e-07 2024-09-18T07:34:54.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.425461+0000", 2024-09-18T07:34:54.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7649999999999999e-06 2024-09-18T07:34:54.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.425470+0000", 2024-09-18T07:34:54.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.551e-06 2024-09-18T07:34:54.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.425476+0000", 2024-09-18T07:34:54.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7790000000000001e-06 2024-09-18T07:34:54.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.425544+0000", 2024-09-18T07:34:54.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7842999999999994e-05 2024-09-18T07:34:54.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.425561+0000", 2024-09-18T07:34:54.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7635e-05 2024-09-18T07:34:54.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.615467+0000", 2024-09-18T07:34:54.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.189905772 2024-09-18T07:34:54.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.615507+0000", 2024-09-18T07:34:54.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0448999999999998e-05 2024-09-18T07:34:54.910 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.910 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.910 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.910 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:142955 7.2 7:477ed1d7:::benchmark_data_smithi137_74180_object38539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-18T07:34:54.910 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:25.427668+0000", 2024-09-18T07:34:54.910 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 29.333466464000001, 2024-09-18T07:34:54.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.19066476700000001, 2024-09-18T07:34:54.910 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.910 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.910 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.910 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:34:54.911 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:34:54.911 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:34:54.911 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 142955 2024-09-18T07:34:54.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.911 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.427668+0000", 2024-09-18T07:34:54.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.427668+0000", 2024-09-18T07:34:54.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.427670+0000", 2024-09-18T07:34:54.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.885e-06 2024-09-18T07:34:54.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.427675+0000", 2024-09-18T07:34:54.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.259e-06 2024-09-18T07:34:54.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.427686+0000", 2024-09-18T07:34:54.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0866999999999999e-05 2024-09-18T07:34:54.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.427694+0000", 2024-09-18T07:34:54.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.089e-06 2024-09-18T07:34:54.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.427744+0000", 2024-09-18T07:34:54.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0083000000000001e-05 2024-09-18T07:34:54.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.427860+0000", 2024-09-18T07:34:54.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115368 2024-09-18T07:34:54.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-18T07:34:54.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.613932+0000", 2024-09-18T07:34:54.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.186072821 2024-09-18T07:34:54.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:34:54.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.615878+0000", 2024-09-18T07:34:54.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019459760000000001 2024-09-18T07:34:54.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.618217+0000", 2024-09-18T07:34:54.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023386909999999999 2024-09-18T07:34:54.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:34:54.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.618248+0000", 2024-09-18T07:34:54.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0536e-05 2024-09-18T07:34:54.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.618296+0000", 2024-09-18T07:34:54.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8055e-05 2024-09-18T07:34:54.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.618333+0000", 2024-09-18T07:34:54.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7137e-05 2024-09-18T07:34:54.916 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.916 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.916 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.916 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:142945 7.f e140/115 7:fcb44080:::benchmark_data_smithi137_74180_object38529:head v 140'8970, mlcod=140'8967)", 2024-09-18T07:34:54.917 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:25.428764+0000", 2024-09-18T07:34:54.917 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 29.332370185999999, 2024-09-18T07:34:54.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.19625530499999999, 2024-09-18T07:34:54.917 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.917 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.917 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.917 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.428764+0000", 2024-09-18T07:34:54.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.428764+0000", 2024-09-18T07:34:54.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.428766+0000", 2024-09-18T07:34:54.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2240000000000001e-06 2024-09-18T07:34:54.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.428773+0000", 2024-09-18T07:34:54.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9619999999999996e-06 2024-09-18T07:34:54.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.615401+0000", 2024-09-18T07:34:54.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.186628245 2024-09-18T07:34:54.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.615413+0000", 2024-09-18T07:34:54.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1816e-05 2024-09-18T07:34:54.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.617187+0000", 2024-09-18T07:34:54.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017741370000000001 2024-09-18T07:34:54.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.617209+0000", 2024-09-18T07:34:54.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2232999999999999e-05 2024-09-18T07:34:54.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.624984+0000", 2024-09-18T07:34:54.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0077750850000000002 2024-09-18T07:34:54.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.625020+0000", 2024-09-18T07:34:54.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5602999999999998e-05 2024-09-18T07:34:54.921 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.921 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.921 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.921 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:142956 7.8 e140/134 7:17baae9e:::benchmark_data_smithi137_74180_object38540:head v 140'8782, mlcod=140'8780)", 2024-09-18T07:34:54.921 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:25.429669+0000", 2024-09-18T07:34:54.921 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 29.33146546, 2024-09-18T07:34:54.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.19019249999999999, 2024-09-18T07:34:54.921 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:34:54.922 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:34:54.922 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:34:54.922 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:34:54.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:34:54.922 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.429669+0000", 2024-09-18T07:34:54.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:34:54.922 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.429669+0000", 2024-09-18T07:34:54.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:34:54.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.923 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:34:54.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.429670+0000", 2024-09-18T07:34:54.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2559999999999999e-06 2024-09-18T07:34:54.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.923 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:34:54.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.429677+0000", 2024-09-18T07:34:54.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5910000000000001e-06 2024-09-18T07:34:54.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.923 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:34:54.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.615428+0000", 2024-09-18T07:34:54.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.18575066000000001 2024-09-18T07:34:54.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.924 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:34:54.924 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.615436+0000", 2024-09-18T07:34:54.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2930000000000001e-06 2024-09-18T07:34:54.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.924 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:34:54.924 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.615502+0000", 2024-09-18T07:34:54.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6005000000000002e-05 2024-09-18T07:34:54.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.924 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:34:54.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.618088+0000", 2024-09-18T07:34:54.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025863829999999998 2024-09-18T07:34:54.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:34:54.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.619820+0000", 2024-09-18T07:34:54.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001732176 2024-09-18T07:34:54.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:34:54.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:34:54.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:34:54.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.619862+0000", 2024-09-18T07:34:54.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1136000000000001e-05 2024-09-18T07:34:54.925 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.925 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.926 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.926 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:34:54.926 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:34:54.926 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:34:54.946 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:54.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:54.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:55.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:34:55.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:55.071 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:55.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:34:55.102 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:34:55.172 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:55.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:55.195 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:55.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:55.295 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:55.317 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:55.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:55.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:55.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:55 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-18T07:34:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:55 smithi137 ceph-mon[29408]: osdmap e166: 8 total, 8 up, 7 in 2024-09-18T07:34:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:55 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1869284639' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:34:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:55 smithi137 ceph-mon[29408]: from='client.34997 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:34:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:55 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:55 smithi137 ceph-mon[29408]: from='client.34997 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:34:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:55 smithi137 ceph-mon[29408]: osdmap e167: 8 total, 8 up, 7 in 2024-09-18T07:34:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:55 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-18T07:34:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:55 smithi137 ceph-mon[39175]: osdmap e166: 8 total, 8 up, 7 in 2024-09-18T07:34:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:55 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1869284639' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:34:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:55 smithi137 ceph-mon[39175]: from='client.34997 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:34:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:55 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:55 smithi137 ceph-mon[39175]: from='client.34997 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:34:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:55 smithi137 ceph-mon[39175]: osdmap e167: 8 total, 8 up, 7 in 2024-09-18T07:34:55.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:55.442 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:55.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:55.543 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:55.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:55.566 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:55.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:55.667 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:55.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:55.689 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:55.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:55.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:55 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-18T07:34:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:55 smithi203 ceph-mon[32237]: osdmap e166: 8 total, 8 up, 7 in 2024-09-18T07:34:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:55 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1869284639' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:34:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:55 smithi203 ceph-mon[32237]: from='client.34997 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:34:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:55 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:34:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:55 smithi203 ceph-mon[32237]: from='client.34997 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:34:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:55 smithi203 ceph-mon[32237]: osdmap e167: 8 total, 8 up, 7 in 2024-09-18T07:34:55.791 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:55.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:55.813 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:55.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:55.914 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:55.936 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:55.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:56.037 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:56.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:56.060 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:56.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:56.161 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:56.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:56.183 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:56.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:56.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:56.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:56.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:56.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:56.374 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T07:34:56.408 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:56.428 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:56.438 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:56.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:56.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:56.540 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:56.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:56.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:56.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:56.624 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:34:56.663 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:56.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[39175]: 5.1 scrub starts 2024-09-18T07:34:56.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[39175]: 5.1 scrub ok 2024-09-18T07:34:56.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[39175]: 7.b scrub starts 2024-09-18T07:34:56.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[39175]: 7.c scrub starts 2024-09-18T07:34:56.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[39175]: 7.b scrub ok 2024-09-18T07:34:56.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[39175]: pgmap v202: 153 pgs: 1 clean+premerge+peered, 152 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 277 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:34:56.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[39175]: 7.c scrub ok 2024-09-18T07:34:56.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[39175]: osdmap e168: 8 total, 8 up, 7 in 2024-09-18T07:34:56.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:56.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:34:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[29408]: 5.1 scrub starts 2024-09-18T07:34:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[29408]: 5.1 scrub ok 2024-09-18T07:34:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[29408]: 7.b scrub starts 2024-09-18T07:34:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[29408]: 7.c scrub starts 2024-09-18T07:34:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[29408]: 7.b scrub ok 2024-09-18T07:34:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[29408]: pgmap v202: 153 pgs: 1 clean+premerge+peered, 152 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 277 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:34:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[29408]: 7.c scrub ok 2024-09-18T07:34:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[29408]: osdmap e168: 8 total, 8 up, 7 in 2024-09-18T07:34:56.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:56.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:56 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:34:56.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:56.686 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:56.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:56.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:56 smithi203 ceph-mon[32237]: 5.1 scrub starts 2024-09-18T07:34:56.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:56 smithi203 ceph-mon[32237]: 5.1 scrub ok 2024-09-18T07:34:56.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:56 smithi203 ceph-mon[32237]: 7.b scrub starts 2024-09-18T07:34:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:56 smithi203 ceph-mon[32237]: 7.c scrub starts 2024-09-18T07:34:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:56 smithi203 ceph-mon[32237]: 7.b scrub ok 2024-09-18T07:34:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:56 smithi203 ceph-mon[32237]: pgmap v202: 153 pgs: 1 clean+premerge+peered, 152 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 277 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:34:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:56 smithi203 ceph-mon[32237]: 7.c scrub ok 2024-09-18T07:34:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:56 smithi203 ceph-mon[32237]: osdmap e168: 8 total, 8 up, 7 in 2024-09-18T07:34:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:56 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:34:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:56 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:34:56.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:56.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:56.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:56.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:56.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:56.951 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:56.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:56.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:57.038 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:34:57.038 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:34:57.052 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:57.077 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:57.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:57.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:57.083 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:34:57.083 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:34:57.119 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:34:57.119 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:34:57.153 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:34:57.153 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:34:57.179 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:57.183 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:34:57.183 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:34:57.215 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:57.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:57.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:57.220 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:34:57.220 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:34:57.253 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:34:57.253 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:34:57.253 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:34:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:34:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:34:57.284 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:34:57.285 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:34:57.317 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:57.342 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:57.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:57.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:57.345 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:34:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:34:57.445 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:57.483 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:57.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:57.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:57.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:57.620 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:57.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:57.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:57.631 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:57 smithi137 ceph-mon[29408]: 5.1a scrub starts 2024-09-18T07:34:57.631 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:57 smithi137 ceph-mon[29408]: 5.1a scrub ok 2024-09-18T07:34:57.631 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:57 smithi137 ceph-mon[29408]: 6.6 scrub starts 2024-09-18T07:34:57.631 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:57 smithi137 ceph-mon[29408]: 6.6 scrub ok 2024-09-18T07:34:57.632 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:57 smithi137 ceph-mon[29408]: osdmap e169: 8 total, 8 up, 7 in 2024-09-18T07:34:57.632 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:57 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1261736046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:57.632 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:57 smithi137 ceph-mon[39175]: 5.1a scrub starts 2024-09-18T07:34:57.632 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:57 smithi137 ceph-mon[39175]: 5.1a scrub ok 2024-09-18T07:34:57.632 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:57 smithi137 ceph-mon[39175]: 6.6 scrub starts 2024-09-18T07:34:57.632 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:57 smithi137 ceph-mon[39175]: 6.6 scrub ok 2024-09-18T07:34:57.632 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:57 smithi137 ceph-mon[39175]: osdmap e169: 8 total, 8 up, 7 in 2024-09-18T07:34:57.632 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:57 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1261736046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:57.692 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:57 smithi203 ceph-mon[32237]: 5.1a scrub starts 2024-09-18T07:34:57.693 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:57 smithi203 ceph-mon[32237]: 5.1a scrub ok 2024-09-18T07:34:57.693 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:57 smithi203 ceph-mon[32237]: 6.6 scrub starts 2024-09-18T07:34:57.693 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:57 smithi203 ceph-mon[32237]: 6.6 scrub ok 2024-09-18T07:34:57.693 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:57 smithi203 ceph-mon[32237]: osdmap e169: 8 total, 8 up, 7 in 2024-09-18T07:34:57.693 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:57 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1261736046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:34:57.718 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:34:57.719 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:34:57.722 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:34:57.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:57.757 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:57.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:34:57.858 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:57.886 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:57.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:57.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:57.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:34:57.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:58.043 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:58.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:58.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:58.197 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:58.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:58.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:58.299 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:58.326 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:58.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:58.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:58.371 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:58.429 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:58.454 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:58.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:58.529 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T07:34:58.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:58.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:58.579 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:58.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:58.609 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:34:58.610 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:34:58.610 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-18T07:34:58.610 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd in 7 2024-09-18T07:34:58.641 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:58 smithi137 ceph-mon[29408]: 4.13 scrub starts 2024-09-18T07:34:58.642 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:58 smithi137 ceph-mon[29408]: 4.13 scrub ok 2024-09-18T07:34:58.642 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:58 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:58.642 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:58 smithi137 ceph-mon[29408]: pgmap v205: 152 pgs: 1 peering, 151 active+clean; 3.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 567 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:34:58.642 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:58 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-18T07:34:58.642 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:58 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-18T07:34:58.642 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:58 smithi137 ceph-mon[29408]: 7.e scrub starts 2024-09-18T07:34:58.642 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:58 smithi137 ceph-mon[39175]: 4.13 scrub starts 2024-09-18T07:34:58.642 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:58 smithi137 ceph-mon[39175]: 4.13 scrub ok 2024-09-18T07:34:58.642 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:58 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:58.643 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:58 smithi137 ceph-mon[39175]: pgmap v205: 152 pgs: 1 peering, 151 active+clean; 3.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 567 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:34:58.643 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:58 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-18T07:34:58.643 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:58 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-18T07:34:58.643 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:58 smithi137 ceph-mon[39175]: 7.e scrub starts 2024-09-18T07:34:58.680 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:58.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:58.704 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:58.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:58.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:58 smithi203 ceph-mon[32237]: 4.13 scrub starts 2024-09-18T07:34:58.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:58 smithi203 ceph-mon[32237]: 4.13 scrub ok 2024-09-18T07:34:58.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:58 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:34:58.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:58 smithi203 ceph-mon[32237]: pgmap v205: 152 pgs: 1 peering, 151 active+clean; 3.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 567 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:34:58.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:58 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-18T07:34:58.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:58 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"} : dispatch 2024-09-18T07:34:58.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:58 smithi203 ceph-mon[32237]: 7.e scrub starts 2024-09-18T07:34:58.778 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:34:58.805 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:34:58.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:58.827 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:58.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:34:58.928 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:58.950 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:58.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:58.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:59.052 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:59.078 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:59.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:59.179 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:59.201 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:59.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:59.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:59.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:34:59.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:59.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:59.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:34:59.413 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:34:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:59 smithi137 ceph-mon[29408]: 6.4 scrub starts 2024-09-18T07:34:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:59 smithi137 ceph-mon[29408]: 6.4 scrub ok 2024-09-18T07:34:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:59 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-18T07:34:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:59 smithi137 ceph-mon[29408]: osdmap e170: 8 total, 8 up, 7 in 2024-09-18T07:34:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:59 smithi137 ceph-mon[29408]: 4.1f scrub starts 2024-09-18T07:34:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:59 smithi137 ceph-mon[29408]: 4.1f scrub ok 2024-09-18T07:34:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:34:59 smithi137 ceph-mon[29408]: 7.e scrub ok 2024-09-18T07:34:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:59 smithi137 ceph-mon[39175]: 6.4 scrub starts 2024-09-18T07:34:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:59 smithi137 ceph-mon[39175]: 6.4 scrub ok 2024-09-18T07:34:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:59 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-18T07:34:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:59 smithi137 ceph-mon[39175]: osdmap e170: 8 total, 8 up, 7 in 2024-09-18T07:34:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:59 smithi137 ceph-mon[39175]: 4.1f scrub starts 2024-09-18T07:34:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:59 smithi137 ceph-mon[39175]: 4.1f scrub ok 2024-09-18T07:34:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:34:59 smithi137 ceph-mon[39175]: 7.e scrub ok 2024-09-18T07:34:59.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:59.450 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:59.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:59.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:34:59.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:34:59.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:59.574 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:59.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:34:59.676 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:34:59.704 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:59.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:59.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:34:59.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:59 smithi203 ceph-mon[32237]: 6.4 scrub starts 2024-09-18T07:34:59.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:59 smithi203 ceph-mon[32237]: 6.4 scrub ok 2024-09-18T07:34:59.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:59 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-18T07:34:59.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:59 smithi203 ceph-mon[32237]: osdmap e170: 8 total, 8 up, 7 in 2024-09-18T07:34:59.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:59 smithi203 ceph-mon[32237]: 4.1f scrub starts 2024-09-18T07:34:59.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:59 smithi203 ceph-mon[32237]: 4.1f scrub ok 2024-09-18T07:34:59.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:34:59 smithi203 ceph-mon[32237]: 7.e scrub ok 2024-09-18T07:34:59.805 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:34:59.831 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:34:59.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:59.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:34:59.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:34:59.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:34:59.956 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:34:59.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:00.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:35:00.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:00.108 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:00.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:00.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:00.210 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:00.228 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:35:00.228 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:35:00.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:00.233 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:00.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:00.293 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:00.335 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:00.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:00.357 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:00.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:00.375 INFO:teuthology.orchestra.run.smithi137.stderr:marked in osd.7. 2024-09-18T07:35:00.376 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:35:00.376 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:00 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a[29404]: 2024-09-18T07:35:00.051+0000 7fe5973a1640 -1 mon.a@0(leader).osd e171 definitely_dead 0 2024-09-18T07:35:00.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:00.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:00.481 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:00.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:00.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:00.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:00.605 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:00.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:00.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:00 smithi137 ceph-mon[29408]: 3.11 scrub starts 2024-09-18T07:35:00.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:00 smithi137 ceph-mon[29408]: 3.11 scrub ok 2024-09-18T07:35:00.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:00 smithi137 ceph-mon[29408]: osdmap e171: 8 total, 8 up, 7 in 2024-09-18T07:35:00.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:00 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/42925643' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:00.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:00 smithi137 ceph-mon[29408]: pgmap v208: 152 pgs: 1 peering, 151 active+clean; 3.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 567 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:35:00.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:00 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/952600080' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-18T07:35:00.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:00 smithi137 ceph-mon[39175]: 3.11 scrub starts 2024-09-18T07:35:00.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:00 smithi137 ceph-mon[39175]: 3.11 scrub ok 2024-09-18T07:35:00.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:00 smithi137 ceph-mon[39175]: osdmap e171: 8 total, 8 up, 7 in 2024-09-18T07:35:00.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:00 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/42925643' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:00.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:00 smithi137 ceph-mon[39175]: pgmap v208: 152 pgs: 1 peering, 151 active+clean; 3.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 567 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:35:00.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:00 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/952600080' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-18T07:35:00.707 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:00.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:00 smithi203 ceph-mon[32237]: 3.11 scrub starts 2024-09-18T07:35:00.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:00 smithi203 ceph-mon[32237]: 3.11 scrub ok 2024-09-18T07:35:00.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:00 smithi203 ceph-mon[32237]: osdmap e171: 8 total, 8 up, 7 in 2024-09-18T07:35:00.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:00 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/42925643' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:00 smithi203 ceph-mon[32237]: pgmap v208: 152 pgs: 1 peering, 151 active+clean; 3.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 567 op/s; 0 B/s, 0 objects/s recovering 2024-09-18T07:35:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:00 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/952600080' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-18T07:35:00.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:00.729 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:00.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:00.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:00.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:00.853 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:00.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:00.929 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:35:00.929 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:35:00.944 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:35:00.944 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:35:00.954 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:00.960 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:35:00.960 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:35:00.976 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:35:00.976 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:35:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:00.977 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:00.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:00.992 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:35:00.993 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:35:01.009 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:35:01.009 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:35:01.025 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:35:01.026 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:35:01.042 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:35:01.042 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:35:01.078 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:01.100 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:01.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:01.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:01.149 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T07:35:01.202 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:01.224 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:01.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:01.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:01.326 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:01.348 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:01.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:01.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:01.412 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:35:01.451 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:01.475 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:01.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:01.577 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:01.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:01.599 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:01.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:01.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:01 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:35:01.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:01 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:35:01.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:01 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/42925643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:35:01.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:01 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/952600080' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-18T07:35:01.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:01 smithi137 ceph-mon[29408]: osdmap e172: 8 total, 8 up, 8 in 2024-09-18T07:35:01.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:01 smithi137 ceph-mon[29408]: 3.1d scrub starts 2024-09-18T07:35:01.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:01 smithi137 ceph-mon[29408]: 3.1d scrub ok 2024-09-18T07:35:01.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:01 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/792288787' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:01.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:01 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:35:01.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:01 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:35:01.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:01 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/42925643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:35:01.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:01 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/952600080' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-18T07:35:01.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:01 smithi137 ceph-mon[39175]: osdmap e172: 8 total, 8 up, 8 in 2024-09-18T07:35:01.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:01 smithi137 ceph-mon[39175]: 3.1d scrub starts 2024-09-18T07:35:01.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:01 smithi137 ceph-mon[39175]: 3.1d scrub ok 2024-09-18T07:35:01.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:01 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/792288787' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:01.692 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:01 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:35:01.693 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:01 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:35:01.693 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:01 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/42925643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:35:01.693 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:01 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/952600080' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-18T07:35:01.693 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:01 smithi203 ceph-mon[32237]: osdmap e172: 8 total, 8 up, 8 in 2024-09-18T07:35:01.693 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:01 smithi203 ceph-mon[32237]: 3.1d scrub starts 2024-09-18T07:35:01.693 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:01 smithi203 ceph-mon[32237]: 3.1d scrub ok 2024-09-18T07:35:01.693 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:01 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/792288787' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:01.701 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:01.723 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:01.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:01.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:01.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:01.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:01.847 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:01.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:01.903 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:35:01.903 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:35:01.903 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:35:01.903 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:35:01.903 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.903 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:20749 7.3 e88/81 7:c04d08df:::benchmark_data_smithi137_74180_object20748:head v 88'1239, mlcod=88'1237)", 2024-09-18T07:35:01.904 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.530035+0000", 2024-09-18T07:35:01.904 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 158.36916925899999, 2024-09-18T07:35:01.904 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13453020299999999, 2024-09-18T07:35:01.904 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:01.904 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:01.904 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:01.904 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:01.904 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.904 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:01.904 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530035+0000", 2024-09-18T07:35:01.904 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.905 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.905 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.905 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:01.905 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530035+0000", 2024-09-18T07:35:01.905 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.905 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.905 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.905 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:01.905 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530036+0000", 2024-09-18T07:35:01.905 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3200000000000004e-07 2024-09-18T07:35:01.905 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.905 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.905 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:01.906 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530106+0000", 2024-09-18T07:35:01.906 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0099999999999996e-05 2024-09-18T07:35:01.906 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.906 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.906 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:01.906 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530115+0000", 2024-09-18T07:35:01.906 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1419999999999999e-06 2024-09-18T07:35:01.906 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.906 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.906 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:01.906 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530122+0000", 2024-09-18T07:35:01.906 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3269999999999997e-06 2024-09-18T07:35:01.906 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.907 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.907 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:01.907 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530167+0000", 2024-09-18T07:35:01.907 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5247999999999998e-05 2024-09-18T07:35:01.907 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.907 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.907 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:01.907 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530185+0000", 2024-09-18T07:35:01.907 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8111e-05 2024-09-18T07:35:01.907 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.907 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.907 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:01.907 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530789+0000", 2024-09-18T07:35:01.908 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00060407400000000002 2024-09-18T07:35:01.908 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.908 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.908 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:01.908 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664566+0000", 2024-09-18T07:35:01.908 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13377636900000001 2024-09-18T07:35:01.908 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.908 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:01.908 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.908 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.908 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.908 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:20756 7.4 e88/81 7:206b3dce:::benchmark_data_smithi137_74180_object20755:head v 88'1303, mlcod=88'1301)", 2024-09-18T07:35:01.908 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.532983+0000", 2024-09-18T07:35:01.909 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 158.36622146900001, 2024-09-18T07:35:01.909 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13536040199999999, 2024-09-18T07:35:01.909 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:01.909 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:01.909 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:01.909 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:01.909 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.909 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:01.909 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.532983+0000", 2024-09-18T07:35:01.909 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.909 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.909 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.910 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:01.910 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.532983+0000", 2024-09-18T07:35:01.910 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.910 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.910 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.910 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:01.910 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.532984+0000", 2024-09-18T07:35:01.910 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7999999999999993e-07 2024-09-18T07:35:01.910 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.910 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.910 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:01.910 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533015+0000", 2024-09-18T07:35:01.910 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1195999999999997e-05 2024-09-18T07:35:01.911 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.911 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.911 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:01.911 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665392+0000", 2024-09-18T07:35:01.911 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13237665800000001 2024-09-18T07:35:01.911 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.911 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.911 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:01.911 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665400+0000", 2024-09-18T07:35:01.911 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7079999999999997e-06 2024-09-18T07:35:01.911 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.911 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.911 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:01.912 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665454+0000", 2024-09-18T07:35:01.912 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4116999999999997e-05 2024-09-18T07:35:01.912 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.912 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.912 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:01.912 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665474+0000", 2024-09-18T07:35:01.912 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0092999999999999e-05 2024-09-18T07:35:01.912 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.912 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.912 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:01.912 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.668302+0000", 2024-09-18T07:35:01.912 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0028280520000000002 2024-09-18T07:35:01.912 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.912 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.913 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:01.913 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.668343+0000", 2024-09-18T07:35:01.913 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1597999999999997e-05 2024-09-18T07:35:01.913 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.913 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:01.913 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.913 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.913 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.913 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:20750 7.3 e88/81 7:c7abdaa5:::benchmark_data_smithi137_74180_object20749:head v 88'1240, mlcod=88'1237)", 2024-09-18T07:35:01.913 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.533452+0000", 2024-09-18T07:35:01.913 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 158.365752284, 2024-09-18T07:35:01.913 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13358589900000001, 2024-09-18T07:35:01.913 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:01.914 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:01.914 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:01.914 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:01.914 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.914 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:01.914 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533452+0000", 2024-09-18T07:35:01.914 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.914 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.914 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.914 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:01.914 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533452+0000", 2024-09-18T07:35:01.914 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.914 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.914 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.915 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:01.915 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533453+0000", 2024-09-18T07:35:01.915 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.057e-06 2024-09-18T07:35:01.915 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.915 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.915 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:01.915 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533515+0000", 2024-09-18T07:35:01.915 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1557999999999996e-05 2024-09-18T07:35:01.915 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.915 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.915 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:01.915 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664524+0000", 2024-09-18T07:35:01.915 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13100931900000001 2024-09-18T07:35:01.916 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.916 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.916 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:01.916 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664540+0000", 2024-09-18T07:35:01.916 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5562000000000001e-05 2024-09-18T07:35:01.916 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.916 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.916 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:01.916 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664600+0000", 2024-09-18T07:35:01.916 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0124000000000002e-05 2024-09-18T07:35:01.916 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.916 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.916 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:01.916 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664620+0000", 2024-09-18T07:35:01.917 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9983000000000002e-05 2024-09-18T07:35:01.917 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.917 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.917 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:01.917 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.666988+0000", 2024-09-18T07:35:01.917 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023680559999999999 2024-09-18T07:35:01.917 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.917 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.917 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:01.917 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.667038+0000", 2024-09-18T07:35:01.917 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.024e-05 2024-09-18T07:35:01.917 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.917 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:01.918 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.918 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.918 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.918 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21270 7.1 7:801f54ca:::benchmark_data_smithi137_74180_object21269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:35:01.918 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.882531+0000", 2024-09-18T07:35:01.918 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 158.01667386099999, 2024-09-18T07:35:01.918 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16466234699999999, 2024-09-18T07:35:01.918 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:01.918 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:01.918 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:01.918 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:01.918 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:01.918 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:01.919 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21270 2024-09-18T07:35:01.919 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.919 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:01.919 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.919 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:01.919 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882531+0000", 2024-09-18T07:35:01.919 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.919 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.919 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.919 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:01.919 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882531+0000", 2024-09-18T07:35:01.919 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.919 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.919 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.920 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:01.920 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882533+0000", 2024-09-18T07:35:01.920 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8190000000000001e-06 2024-09-18T07:35:01.920 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.920 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.920 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:01.920 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882584+0000", 2024-09-18T07:35:01.920 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1919999999999998e-05 2024-09-18T07:35:01.920 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.920 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.920 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:01.920 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882593+0000", 2024-09-18T07:35:01.920 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8829999999999995e-06 2024-09-18T07:35:01.920 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.921 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.921 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:01.921 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882599+0000", 2024-09-18T07:35:01.921 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5849999999999999e-06 2024-09-18T07:35:01.921 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.921 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.921 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:01.921 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882634+0000", 2024-09-18T07:35:01.921 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5132999999999997e-05 2024-09-18T07:35:01.921 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.921 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.921 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:01.922 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882717+0000", 2024-09-18T07:35:01.922 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2819999999999996e-05 2024-09-18T07:35:01.922 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.922 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.922 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:35:01.922 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882824+0000", 2024-09-18T07:35:01.922 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010708400000000001 2024-09-18T07:35:01.922 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.922 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.922 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:01.922 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.883645+0000", 2024-09-18T07:35:01.922 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00082133399999999995 2024-09-18T07:35:01.923 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.923 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.923 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.923 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.940766+0000", 2024-09-18T07:35:01.923 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.057121166000000001 2024-09-18T07:35:01.923 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.923 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.923 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.923 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.046997+0000", 2024-09-18T07:35:01.923 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10623094900000001 2024-09-18T07:35:01.923 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.923 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.923 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:01.923 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047057+0000", 2024-09-18T07:35:01.924 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9580000000000002e-05 2024-09-18T07:35:01.924 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.924 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.924 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:01.924 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047193+0000", 2024-09-18T07:35:01.924 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000136074 2024-09-18T07:35:01.924 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.924 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:01.924 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.924 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.924 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.924 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21285 7.1 7:8dbe9f10:::benchmark_data_smithi137_74180_object21284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:35:01.925 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.944262+0000", 2024-09-18T07:35:01.925 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 157.95494281699999, 2024-09-18T07:35:01.925 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10313246500000001, 2024-09-18T07:35:01.925 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:01.925 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:01.925 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:01.925 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:01.925 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:01.925 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:01.925 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21285 2024-09-18T07:35:01.925 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.925 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:01.925 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.926 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:01.926 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944262+0000", 2024-09-18T07:35:01.926 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.926 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.926 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.926 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:01.926 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944262+0000", 2024-09-18T07:35:01.926 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.926 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.926 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.926 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:01.926 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944264+0000", 2024-09-18T07:35:01.926 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8679999999999999e-06 2024-09-18T07:35:01.926 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.927 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.927 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:01.927 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944324+0000", 2024-09-18T07:35:01.927 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0225e-05 2024-09-18T07:35:01.927 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.927 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.927 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:01.927 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944333+0000", 2024-09-18T07:35:01.927 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.064e-06 2024-09-18T07:35:01.927 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.927 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.927 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:01.927 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944338+0000", 2024-09-18T07:35:01.928 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9749999999999998e-06 2024-09-18T07:35:01.928 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.928 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.928 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:01.928 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944380+0000", 2024-09-18T07:35:01.928 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1773000000000001e-05 2024-09-18T07:35:01.928 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.928 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.928 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:01.928 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944476+0000", 2024-09-18T07:35:01.928 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6497999999999997e-05 2024-09-18T07:35:01.928 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.928 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.929 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:35:01.929 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944565+0000", 2024-09-18T07:35:01.929 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9288000000000003e-05 2024-09-18T07:35:01.929 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.929 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.929 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:01.929 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.945376+0000", 2024-09-18T07:35:01.929 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00081097899999999998 2024-09-18T07:35:01.929 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.929 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.929 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.929 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.946088+0000", 2024-09-18T07:35:01.929 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00071115600000000001 2024-09-18T07:35:01.930 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.930 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.930 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.930 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047252+0000", 2024-09-18T07:35:01.930 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.101164449 2024-09-18T07:35:01.930 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.930 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.930 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:01.930 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047340+0000", 2024-09-18T07:35:01.930 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8268000000000003e-05 2024-09-18T07:35:01.930 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.930 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.930 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:01.931 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047394+0000", 2024-09-18T07:35:01.931 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3922000000000003e-05 2024-09-18T07:35:01.931 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.931 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:01.931 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.931 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.931 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.931 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22331 7.1 7:8575e846:::benchmark_data_smithi137_74180_object22330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:35:01.931 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.655146+0000", 2024-09-18T07:35:01.931 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 157.244058696, 2024-09-18T07:35:01.931 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.29868304200000001, 2024-09-18T07:35:01.931 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:01.931 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:01.931 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:01.932 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:01.932 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:01.932 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:01.932 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22331 2024-09-18T07:35:01.932 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.932 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:01.932 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.932 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:01.933 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655146+0000", 2024-09-18T07:35:01.933 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.933 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.933 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.934 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:01.934 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655146+0000", 2024-09-18T07:35:01.934 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.934 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.934 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.934 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:01.934 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655148+0000", 2024-09-18T07:35:01.934 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.779e-06 2024-09-18T07:35:01.935 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.935 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.935 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:01.935 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655192+0000", 2024-09-18T07:35:01.935 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4023000000000002e-05 2024-09-18T07:35:01.935 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.935 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.935 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:01.935 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655201+0000", 2024-09-18T07:35:01.935 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3559999999999999e-06 2024-09-18T07:35:01.935 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.935 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.935 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:01.936 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655206+0000", 2024-09-18T07:35:01.936 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4539999999999998e-06 2024-09-18T07:35:01.936 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.936 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.936 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:01.936 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655250+0000", 2024-09-18T07:35:01.936 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3630000000000001e-05 2024-09-18T07:35:01.936 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.936 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.936 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:01.936 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655323+0000", 2024-09-18T07:35:01.936 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3382999999999999e-05 2024-09-18T07:35:01.936 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.937 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.937 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:35:01.937 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655411+0000", 2024-09-18T07:35:01.937 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7776999999999997e-05 2024-09-18T07:35:01.937 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.937 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.937 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:01.937 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.656141+0000", 2024-09-18T07:35:01.937 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00072986100000000005 2024-09-18T07:35:01.937 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.937 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.937 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.938 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.658652+0000", 2024-09-18T07:35:01.938 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025112569999999998 2024-09-18T07:35:01.938 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.938 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.938 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.938 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953705+0000", 2024-09-18T07:35:01.938 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.29505283900000001 2024-09-18T07:35:01.938 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.938 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.938 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:01.938 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953782+0000", 2024-09-18T07:35:01.938 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6793999999999997e-05 2024-09-18T07:35:01.938 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.939 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.939 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:01.939 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953829+0000", 2024-09-18T07:35:01.939 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6888999999999998e-05 2024-09-18T07:35:01.939 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.939 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:01.939 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.939 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.939 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.939 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22350 7.1 7:851589b5:::benchmark_data_smithi137_74180_object22349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:35:01.939 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.662419+0000", 2024-09-18T07:35:01.939 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 157.23678541500001, 2024-09-18T07:35:01.939 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.29155694799999998, 2024-09-18T07:35:01.939 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:01.940 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:01.940 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:01.940 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:01.940 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:01.940 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:01.940 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22350 2024-09-18T07:35:01.940 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.940 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:01.940 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.940 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:01.940 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662419+0000", 2024-09-18T07:35:01.940 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.940 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.941 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.941 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:01.941 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662419+0000", 2024-09-18T07:35:01.941 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.941 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.941 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.941 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:01.941 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662421+0000", 2024-09-18T07:35:01.941 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.39e-06 2024-09-18T07:35:01.941 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.941 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.941 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:01.941 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662473+0000", 2024-09-18T07:35:01.942 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2094e-05 2024-09-18T07:35:01.942 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.942 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.942 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:01.942 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662480+0000", 2024-09-18T07:35:01.942 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5449999999999998e-06 2024-09-18T07:35:01.942 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.942 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.942 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:01.942 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662485+0000", 2024-09-18T07:35:01.942 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9200000000000003e-06 2024-09-18T07:35:01.942 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.942 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.943 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:01.943 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662521+0000", 2024-09-18T07:35:01.943 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5559000000000003e-05 2024-09-18T07:35:01.943 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.943 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.943 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:01.943 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662589+0000", 2024-09-18T07:35:01.943 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8480999999999997e-05 2024-09-18T07:35:01.943 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.943 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.943 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:35:01.943 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662666+0000", 2024-09-18T07:35:01.943 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7224999999999993e-05 2024-09-18T07:35:01.944 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.944 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.944 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:01.944 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.663390+0000", 2024-09-18T07:35:01.944 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00072354199999999998 2024-09-18T07:35:01.944 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.944 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.944 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.944 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.663928+0000", 2024-09-18T07:35:01.944 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00053822699999999995 2024-09-18T07:35:01.944 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.944 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.945 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.945 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953879+0000", 2024-09-18T07:35:01.945 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.28995123699999997 2024-09-18T07:35:01.945 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.945 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.945 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:01.945 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953934+0000", 2024-09-18T07:35:01.945 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5047000000000003e-05 2024-09-18T07:35:01.945 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.945 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.945 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:01.945 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953976+0000", 2024-09-18T07:35:01.945 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1681000000000003e-05 2024-09-18T07:35:01.945 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.946 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:01.946 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.946 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.946 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.946 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22352 7.1 7:83b03301:::benchmark_data_smithi137_74180_object22351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:35:01.946 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.664495+0000", 2024-09-18T07:35:01.946 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 157.23470933900001, 2024-09-18T07:35:01.946 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.28962424599999997, 2024-09-18T07:35:01.946 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:01.946 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:01.946 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:01.946 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:01.946 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:01.946 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:01.947 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22352 2024-09-18T07:35:01.947 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.947 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:01.947 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.947 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:01.947 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664495+0000", 2024-09-18T07:35:01.947 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.947 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.947 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.947 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:01.947 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664495+0000", 2024-09-18T07:35:01.947 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.947 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.947 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.948 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:01.948 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664496+0000", 2024-09-18T07:35:01.948 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6000000000000003e-07 2024-09-18T07:35:01.948 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.948 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.948 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:01.948 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664568+0000", 2024-09-18T07:35:01.948 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2021999999999996e-05 2024-09-18T07:35:01.948 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.948 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.948 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:01.948 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664574+0000", 2024-09-18T07:35:01.949 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4620000000000001e-06 2024-09-18T07:35:01.949 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.949 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.949 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:01.949 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664579+0000", 2024-09-18T07:35:01.949 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0620000000000002e-06 2024-09-18T07:35:01.949 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.949 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.949 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:01.949 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664605+0000", 2024-09-18T07:35:01.949 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6126e-05 2024-09-18T07:35:01.949 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.949 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.950 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:01.950 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664676+0000", 2024-09-18T07:35:01.950 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1521999999999997e-05 2024-09-18T07:35:01.950 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.950 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.950 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:35:01.950 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664749+0000", 2024-09-18T07:35:01.950 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3232999999999995e-05 2024-09-18T07:35:01.950 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.950 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.950 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:01.950 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.665309+0000", 2024-09-18T07:35:01.950 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00055992900000000003 2024-09-18T07:35:01.951 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.951 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.951 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.951 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.665857+0000", 2024-09-18T07:35:01.951 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00054812600000000002 2024-09-18T07:35:01.951 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.951 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.951 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.951 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.954036+0000", 2024-09-18T07:35:01.951 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.288178926 2024-09-18T07:35:01.951 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.951 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.951 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:01.952 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.954085+0000", 2024-09-18T07:35:01.952 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8534e-05 2024-09-18T07:35:01.952 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.952 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.952 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:01.952 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.954119+0000", 2024-09-18T07:35:01.952 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4544e-05 2024-09-18T07:35:01.952 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.952 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:01.952 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.952 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.952 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.953 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:22994 7.9 e90/81 7:9ec32d99:::benchmark_data_smithi137_74180_object22993:head v 90'1458, mlcod=90'1455)", 2024-09-18T07:35:01.953 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.331557+0000", 2024-09-18T07:35:01.953 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 156.56764756699999, 2024-09-18T07:35:01.953 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.098100452000000005, 2024-09-18T07:35:01.953 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:01.953 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:01.953 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:01.953 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:01.953 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.953 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:01.953 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331557+0000", 2024-09-18T07:35:01.953 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.954 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.954 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.954 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:01.954 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331557+0000", 2024-09-18T07:35:01.954 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.954 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.954 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.954 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:01.954 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331558+0000", 2024-09-18T07:35:01.954 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4099999999999997e-07 2024-09-18T07:35:01.954 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.954 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.954 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:01.954 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331619+0000", 2024-09-18T07:35:01.955 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1216000000000005e-05 2024-09-18T07:35:01.955 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.955 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.955 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:01.955 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331626+0000", 2024-09-18T07:35:01.955 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3119999999999996e-06 2024-09-18T07:35:01.955 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.955 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.955 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:01.955 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331631+0000", 2024-09-18T07:35:01.955 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3370000000000001e-06 2024-09-18T07:35:01.955 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.955 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.955 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:01.956 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331670+0000", 2024-09-18T07:35:01.956 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9103999999999997e-05 2024-09-18T07:35:01.956 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.956 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.956 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:01.956 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331682+0000", 2024-09-18T07:35:01.956 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1926000000000001e-05 2024-09-18T07:35:01.956 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.956 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.956 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:01.956 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.429623+0000", 2024-09-18T07:35:01.956 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.097941316 2024-09-18T07:35:01.957 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.957 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.957 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:01.957 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.429657+0000", 2024-09-18T07:35:01.957 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4400000000000003e-05 2024-09-18T07:35:01.957 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.957 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:01.957 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.957 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.957 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.957 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:23001 7.1 7:8979af80:::benchmark_data_smithi137_74180_object23000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:35:01.957 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.332415+0000", 2024-09-18T07:35:01.957 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 156.56678997200001, 2024-09-18T07:35:01.958 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.100079928, 2024-09-18T07:35:01.958 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:01.958 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:01.958 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:01.958 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:01.958 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:01.958 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:01.958 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 23001 2024-09-18T07:35:01.958 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.958 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:01.958 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.958 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:01.958 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332415+0000", 2024-09-18T07:35:01.959 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.959 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.959 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.959 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:01.959 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332415+0000", 2024-09-18T07:35:01.959 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.959 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.959 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.959 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:01.959 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332416+0000", 2024-09-18T07:35:01.959 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8369999999999999e-06 2024-09-18T07:35:01.959 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.959 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.960 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:01.960 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332477+0000", 2024-09-18T07:35:01.960 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0333e-05 2024-09-18T07:35:01.960 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.960 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.960 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:01.960 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332487+0000", 2024-09-18T07:35:01.960 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0203e-05 2024-09-18T07:35:01.960 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.960 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.960 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:01.960 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332494+0000", 2024-09-18T07:35:01.960 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.515e-06 2024-09-18T07:35:01.961 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.961 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.961 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:01.961 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332560+0000", 2024-09-18T07:35:01.961 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5874e-05 2024-09-18T07:35:01.961 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.961 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.961 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:01.961 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.428551+0000", 2024-09-18T07:35:01.961 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.095990961 2024-09-18T07:35:01.961 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.962 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.962 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:35:01.962 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.428728+0000", 2024-09-18T07:35:01.962 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017673299999999999 2024-09-18T07:35:01.962 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.962 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.962 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:01.962 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.430995+0000", 2024-09-18T07:35:01.962 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022672819999999998 2024-09-18T07:35:01.962 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.962 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.962 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.962 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.432117+0000", 2024-09-18T07:35:01.963 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0011216939999999999 2024-09-18T07:35:01.963 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.963 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.963 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.963 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.432392+0000", 2024-09-18T07:35:01.963 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00027448799999999998 2024-09-18T07:35:01.963 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.963 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.963 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:01.963 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.432451+0000", 2024-09-18T07:35:01.963 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9233e-05 2024-09-18T07:35:01.963 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.963 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.964 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:01.964 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.432495+0000", 2024-09-18T07:35:01.964 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3775e-05 2024-09-18T07:35:01.964 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.964 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:01.964 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.964 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.964 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.964 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:44367 7.1 7:854d3320:::benchmark_data_smithi137_74180_object44366:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-18T07:35:01.964 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:43.591378+0000", 2024-09-18T07:35:01.964 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 138.307826775, 2024-09-18T07:35:01.964 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.113341394, 2024-09-18T07:35:01.964 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:01.965 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:01.965 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:01.965 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:01.965 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:01.965 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:01.965 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 44367 2024-09-18T07:35:01.965 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.965 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:01.965 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.965 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:01.965 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591378+0000", 2024-09-18T07:35:01.965 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.965 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.966 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.966 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:01.966 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591378+0000", 2024-09-18T07:35:01.966 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.966 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.966 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.966 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:01.966 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591379+0000", 2024-09-18T07:35:01.966 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7099999999999999e-06 2024-09-18T07:35:01.966 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.966 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.966 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:01.967 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591420+0000", 2024-09-18T07:35:01.967 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0559000000000002e-05 2024-09-18T07:35:01.967 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.967 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.967 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:01.967 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591432+0000", 2024-09-18T07:35:01.967 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.226e-05 2024-09-18T07:35:01.967 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.967 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.967 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:01.967 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591440+0000", 2024-09-18T07:35:01.967 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5830000000000001e-06 2024-09-18T07:35:01.967 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.968 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.968 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:01.968 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591497+0000", 2024-09-18T07:35:01.968 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7114999999999998e-05 2024-09-18T07:35:01.968 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.968 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.968 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:01.968 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591664+0000", 2024-09-18T07:35:01.968 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000167188 2024-09-18T07:35:01.968 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.968 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.968 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:35:01.969 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591884+0000", 2024-09-18T07:35:01.969 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000220223 2024-09-18T07:35:01.969 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.969 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.969 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:01.969 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.593492+0000", 2024-09-18T07:35:01.969 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016074139999999999 2024-09-18T07:35:01.969 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.969 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.969 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.969 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.593595+0000", 2024-09-18T07:35:01.969 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010319699999999999 2024-09-18T07:35:01.970 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.970 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.970 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.970 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.704655+0000", 2024-09-18T07:35:01.970 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.111060176 2024-09-18T07:35:01.970 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.970 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.970 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:01.970 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.704690+0000", 2024-09-18T07:35:01.970 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4916999999999998e-05 2024-09-18T07:35:01.970 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.970 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.970 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:01.971 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.704719+0000", 2024-09-18T07:35:01.971 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9051999999999999e-05 2024-09-18T07:35:01.971 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.971 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:01.971 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.971 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.971 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.971 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:51788 7.1 7:81b6d503:::benchmark_data_smithi137_74180_object51787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:35:01.971 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:35:01.971 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 131.23226059999999, 2024-09-18T07:35:01.971 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14889203200000001, 2024-09-18T07:35:01.971 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:01.971 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:01.972 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:01.972 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:01.972 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:01.972 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:01.972 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 51788 2024-09-18T07:35:01.972 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.972 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:01.972 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.972 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:01.972 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:35:01.972 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.972 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.972 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.973 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:01.973 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:35:01.973 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.973 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.973 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.973 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:01.973 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666966+0000", 2024-09-18T07:35:01.973 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1860000000000001e-05 2024-09-18T07:35:01.973 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.973 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.973 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:01.973 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666998+0000", 2024-09-18T07:35:01.974 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1856000000000001e-05 2024-09-18T07:35:01.974 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.974 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.974 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:01.974 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667012+0000", 2024-09-18T07:35:01.974 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4592e-05 2024-09-18T07:35:01.974 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.974 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.974 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:01.974 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667019+0000", 2024-09-18T07:35:01.974 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1940000000000001e-06 2024-09-18T07:35:01.974 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.974 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.975 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:01.975 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667083+0000", 2024-09-18T07:35:01.975 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3553999999999995e-05 2024-09-18T07:35:01.975 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.975 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.975 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:01.975 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667217+0000", 2024-09-18T07:35:01.975 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000133712 2024-09-18T07:35:01.975 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.975 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.975 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:35:01.975 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667345+0000", 2024-09-18T07:35:01.975 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012796199999999999 2024-09-18T07:35:01.976 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.976 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.976 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:01.976 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.668905+0000", 2024-09-18T07:35:01.976 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001560344 2024-09-18T07:35:01.976 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.976 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.976 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.976 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669080+0000", 2024-09-18T07:35:01.976 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000174964 2024-09-18T07:35:01.976 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.976 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.976 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.977 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815763+0000", 2024-09-18T07:35:01.977 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.146683217 2024-09-18T07:35:01.977 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.977 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.977 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:01.977 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815804+0000", 2024-09-18T07:35:01.977 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0535999999999999e-05 2024-09-18T07:35:01.977 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.977 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.977 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:01.977 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815836+0000", 2024-09-18T07:35:01.977 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2240999999999998e-05 2024-09-18T07:35:01.977 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.978 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:01.978 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.978 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.978 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.978 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:51794 7.1 7:8f789502:::benchmark_data_smithi137_74180_object51793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:35:01.978 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:35:01.978 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 131.23008748999999, 2024-09-18T07:35:01.978 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15046784899999999, 2024-09-18T07:35:01.978 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:01.978 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:01.978 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:01.978 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:01.978 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:01.979 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:01.979 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 51794 2024-09-18T07:35:01.979 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.979 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:01.979 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.979 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:01.979 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:35:01.979 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.979 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.979 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.979 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:01.979 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:35:01.979 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.980 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.980 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.980 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:01.980 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669118+0000", 2024-09-18T07:35:01.980 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4079999999999999e-06 2024-09-18T07:35:01.980 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.980 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.980 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:01.980 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669172+0000", 2024-09-18T07:35:01.980 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3808999999999999e-05 2024-09-18T07:35:01.980 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.980 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.980 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:01.981 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669184+0000", 2024-09-18T07:35:01.981 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1888e-05 2024-09-18T07:35:01.981 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.981 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.981 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:01.981 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669192+0000", 2024-09-18T07:35:01.981 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7549999999999994e-06 2024-09-18T07:35:01.981 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.981 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.981 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:01.981 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669229+0000", 2024-09-18T07:35:01.981 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7234000000000003e-05 2024-09-18T07:35:01.981 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.982 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.982 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:01.982 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669310+0000", 2024-09-18T07:35:01.982 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1248999999999994e-05 2024-09-18T07:35:01.982 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.982 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.982 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:35:01.982 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669391+0000", 2024-09-18T07:35:01.982 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0401000000000005e-05 2024-09-18T07:35:01.982 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.982 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.982 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:01.982 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.670397+0000", 2024-09-18T07:35:01.982 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010061619999999999 2024-09-18T07:35:01.983 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.983 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.983 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.983 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.670664+0000", 2024-09-18T07:35:01.983 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00026735700000000002 2024-09-18T07:35:01.983 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.983 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.983 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.983 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819479+0000", 2024-09-18T07:35:01.983 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14881438499999999 2024-09-18T07:35:01.983 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.983 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.983 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:01.984 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819537+0000", 2024-09-18T07:35:01.984 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8192999999999997e-05 2024-09-18T07:35:01.984 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.984 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.984 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:01.984 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819585+0000", 2024-09-18T07:35:01.984 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8007999999999998e-05 2024-09-18T07:35:01.984 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.984 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:01.984 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.984 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.984 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.984 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:74680 7.1 7:8999b183:::benchmark_data_smithi137_74180_object74679:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:35:01.985 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:35:01.985 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 105.72123944400001, 2024-09-18T07:35:01.985 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13647764200000001, 2024-09-18T07:35:01.985 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:01.985 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:01.985 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:01.985 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:01.985 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:01.985 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:01.985 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 74680 2024-09-18T07:35:01.985 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.985 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:01.985 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.985 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:01.986 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:35:01.986 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.986 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.986 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.986 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:01.986 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:35:01.986 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.986 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.986 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.986 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:01.986 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177967+0000", 2024-09-18T07:35:01.986 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.559e-06 2024-09-18T07:35:01.986 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.987 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.987 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:01.987 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178020+0000", 2024-09-18T07:35:01.987 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3810999999999997e-05 2024-09-18T07:35:01.987 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.987 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.987 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:01.987 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178030+0000", 2024-09-18T07:35:01.987 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5200000000000003e-06 2024-09-18T07:35:01.987 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.987 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.987 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:01.987 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178035+0000", 2024-09-18T07:35:01.988 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0429999999999998e-06 2024-09-18T07:35:01.988 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.988 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.988 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:01.988 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178074+0000", 2024-09-18T07:35:01.988 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9014999999999998e-05 2024-09-18T07:35:01.988 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.988 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.988 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:01.988 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178151+0000", 2024-09-18T07:35:01.988 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6526000000000006e-05 2024-09-18T07:35:01.988 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.989 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.989 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:35:01.989 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178233+0000", 2024-09-18T07:35:01.989 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2174000000000003e-05 2024-09-18T07:35:01.989 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.989 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.989 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:01.989 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.180222+0000", 2024-09-18T07:35:01.989 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0019888459999999998 2024-09-18T07:35:01.989 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.989 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.989 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.989 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.180453+0000", 2024-09-18T07:35:01.990 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023154 2024-09-18T07:35:01.990 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.990 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.990 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.990 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314370+0000", 2024-09-18T07:35:01.990 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.133916497 2024-09-18T07:35:01.990 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.990 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.990 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:01.990 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314411+0000", 2024-09-18T07:35:01.990 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1421000000000002e-05 2024-09-18T07:35:01.990 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.990 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.990 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:01.991 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314443+0000", 2024-09-18T07:35:01.991 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1690000000000003e-05 2024-09-18T07:35:01.991 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.991 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:01.991 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.991 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.991 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.991 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:76535 7.1 7:86123a56:::benchmark_data_smithi137_74180_object76534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:35:01.991 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.065654+0000", 2024-09-18T07:35:01.991 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 103.833550405, 2024-09-18T07:35:01.991 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.097319778999999995, 2024-09-18T07:35:01.991 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:01.991 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:01.991 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:01.992 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:01.992 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:01.992 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:01.992 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 76535 2024-09-18T07:35:01.992 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.992 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:01.992 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.992 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:01.992 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065654+0000", 2024-09-18T07:35:01.992 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.992 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.992 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.993 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:01.993 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065654+0000", 2024-09-18T07:35:01.993 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.993 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.993 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.993 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:01.993 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065656+0000", 2024-09-18T07:35:01.993 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8220000000000001e-06 2024-09-18T07:35:01.993 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.993 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.993 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:01.993 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065751+0000", 2024-09-18T07:35:01.994 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5267999999999997e-05 2024-09-18T07:35:01.994 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.994 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.994 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:01.994 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065761+0000", 2024-09-18T07:35:01.994 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2799999999999992e-06 2024-09-18T07:35:01.994 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.994 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.994 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:01.994 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065766+0000", 2024-09-18T07:35:01.994 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6699999999999999e-06 2024-09-18T07:35:01.994 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.994 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.995 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:01.995 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065808+0000", 2024-09-18T07:35:01.995 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2150999999999997e-05 2024-09-18T07:35:01.995 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.995 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.995 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:01.995 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065904+0000", 2024-09-18T07:35:01.995 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5330999999999999e-05 2024-09-18T07:35:01.995 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.995 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.995 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:35:01.995 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.066005+0000", 2024-09-18T07:35:01.995 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010100800000000001 2024-09-18T07:35:01.996 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.996 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.996 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:01.996 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.067931+0000", 2024-09-18T07:35:01.996 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0019264239999999999 2024-09-18T07:35:01.996 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.996 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.996 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.996 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.070368+0000", 2024-09-18T07:35:01.996 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0024367659999999999 2024-09-18T07:35:01.996 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.996 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.996 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:01.997 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.162870+0000", 2024-09-18T07:35:01.997 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.092502134999999999 2024-09-18T07:35:01.997 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.997 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.997 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:01.997 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.162928+0000", 2024-09-18T07:35:01.997 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8029999999999998e-05 2024-09-18T07:35:01.997 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.997 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.997 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:01.997 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.162974+0000", 2024-09-18T07:35:01.997 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5893999999999998e-05 2024-09-18T07:35:01.997 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.998 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:01.998 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:01.998 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.998 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.998 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:76538 7.1 7:826fb340:::benchmark_data_smithi137_74180_object76537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:35:01.998 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.068561+0000", 2024-09-18T07:35:01.998 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 103.830643083, 2024-09-18T07:35:01.998 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.097070977000000003, 2024-09-18T07:35:01.998 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:01.998 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:01.998 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:01.998 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:01.998 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:01.999 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:01.999 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 76538 2024-09-18T07:35:01.999 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.999 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:01.999 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.999 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:01.999 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068561+0000", 2024-09-18T07:35:01.999 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.999 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:01.999 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:01.999 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:01.999 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068561+0000", 2024-09-18T07:35:01.999 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:01.999 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.000 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.000 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:02.000 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068563+0000", 2024-09-18T07:35:02.000 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5629999999999999e-06 2024-09-18T07:35:02.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.000 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.000 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:02.000 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068611+0000", 2024-09-18T07:35:02.000 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8412999999999999e-05 2024-09-18T07:35:02.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.000 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.000 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:02.001 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068620+0000", 2024-09-18T07:35:02.001 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7530000000000007e-06 2024-09-18T07:35:02.001 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.001 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.001 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:02.001 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068626+0000", 2024-09-18T07:35:02.001 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.818e-06 2024-09-18T07:35:02.001 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.001 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.001 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:02.001 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068663+0000", 2024-09-18T07:35:02.001 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7086999999999999e-05 2024-09-18T07:35:02.001 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.001 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.002 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:02.002 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068734+0000", 2024-09-18T07:35:02.002 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0605000000000006e-05 2024-09-18T07:35:02.002 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.002 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.002 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:35:02.002 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068829+0000", 2024-09-18T07:35:02.002 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5533999999999997e-05 2024-09-18T07:35:02.002 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.002 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.002 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:02.002 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.070471+0000", 2024-09-18T07:35:02.002 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016418800000000001 2024-09-18T07:35:02.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.003 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:02.003 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.071217+0000", 2024-09-18T07:35:02.003 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00074625099999999997 2024-09-18T07:35:02.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.003 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:02.003 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.165516+0000", 2024-09-18T07:35:02.003 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.094298471999999994 2024-09-18T07:35:02.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.003 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:02.004 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.165590+0000", 2024-09-18T07:35:02.004 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4103999999999994e-05 2024-09-18T07:35:02.004 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.004 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.004 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:02.004 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.165632+0000", 2024-09-18T07:35:02.004 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2497000000000003e-05 2024-09-18T07:35:02.004 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:02.004 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:02.004 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:02.004 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.004 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.004 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:76881 7.1 7:8430cf73:::benchmark_data_smithi137_74180_object76880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:35:02.005 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.426831+0000", 2024-09-18T07:35:02.005 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 103.472373686, 2024-09-18T07:35:02.005 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.115187357, 2024-09-18T07:35:02.005 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:02.005 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:02.005 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:02.005 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:02.005 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:02.005 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:02.005 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 76881 2024-09-18T07:35:02.005 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.005 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:02.005 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.006 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:02.006 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426831+0000", 2024-09-18T07:35:02.006 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:02.006 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.006 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.006 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:02.006 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426831+0000", 2024-09-18T07:35:02.006 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:02.006 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.006 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.006 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:02.008 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:02.008 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426833+0000", 2024-09-18T07:35:02.008 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7710000000000001e-06 2024-09-18T07:35:02.008 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.008 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.008 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:02.008 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426869+0000", 2024-09-18T07:35:02.008 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5886999999999997e-05 2024-09-18T07:35:02.008 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.009 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.009 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:02.009 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426879+0000", 2024-09-18T07:35:02.009 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0667e-05 2024-09-18T07:35:02.009 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.009 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.009 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:02.009 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426886+0000", 2024-09-18T07:35:02.009 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3799999999999999e-06 2024-09-18T07:35:02.009 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.009 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.009 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:02.009 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426932+0000", 2024-09-18T07:35:02.009 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6143999999999997e-05 2024-09-18T07:35:02.010 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.010 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:02.010 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.427047+0000", 2024-09-18T07:35:02.010 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011568700000000001 2024-09-18T07:35:02.010 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.010 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:35:02.010 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.427168+0000", 2024-09-18T07:35:02.010 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000120118 2024-09-18T07:35:02.010 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.010 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:02.010 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.429449+0000", 2024-09-18T07:35:02.011 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002281394 2024-09-18T07:35:02.011 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.011 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.011 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:02.011 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.431262+0000", 2024-09-18T07:35:02.011 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0018125940000000001 2024-09-18T07:35:02.011 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.011 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.011 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:02.011 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541358+0000", 2024-09-18T07:35:02.011 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.110095996 2024-09-18T07:35:02.011 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.011 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.012 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:02.012 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541963+0000", 2024-09-18T07:35:02.012 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00060516200000000002 2024-09-18T07:35:02.012 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.012 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.012 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:02.012 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.542018+0000", 2024-09-18T07:35:02.012 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5556999999999997e-05 2024-09-18T07:35:02.012 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:02.012 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:02.012 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:02.012 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.012 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.013 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:76891 7.f e116/115 7:f77e09af:::benchmark_data_smithi137_74180_object76890:head v 116'4811, mlcod=116'4809)", 2024-09-18T07:35:02.013 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.436137+0000", 2024-09-18T07:35:02.013 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 103.46306756200001, 2024-09-18T07:35:02.013 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.105530608, 2024-09-18T07:35:02.013 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:02.013 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:02.013 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:02.013 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:02.013 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.013 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:02.013 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436137+0000", 2024-09-18T07:35:02.013 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:02.013 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.013 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.014 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:02.014 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436137+0000", 2024-09-18T07:35:02.014 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:02.014 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.014 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.014 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:02.014 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436139+0000", 2024-09-18T07:35:02.014 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3259999999999998e-06 2024-09-18T07:35:02.014 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.014 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.014 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:02.014 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436192+0000", 2024-09-18T07:35:02.014 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2339000000000002e-05 2024-09-18T07:35:02.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:02.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436209+0000", 2024-09-18T07:35:02.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6935999999999998e-05 2024-09-18T07:35:02.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:02.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436219+0000", 2024-09-18T07:35:02.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0593e-05 2024-09-18T07:35:02.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:02.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436314+0000", 2024-09-18T07:35:02.016 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5035000000000001e-05 2024-09-18T07:35:02.016 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.016 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.016 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:02.016 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436347+0000", 2024-09-18T07:35:02.016 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2558999999999998e-05 2024-09-18T07:35:02.016 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.016 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.016 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:02.016 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.437824+0000", 2024-09-18T07:35:02.016 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001477405 2024-09-18T07:35:02.016 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.016 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.017 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:02.017 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541668+0000", 2024-09-18T07:35:02.017 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10384341499999999 2024-09-18T07:35:02.017 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:02.017 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:02.017 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:02.017 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.017 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.017 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:76893 7.4 e116/81 7:2dcd0014:::benchmark_data_smithi137_74180_object76892:head v 116'4832, mlcod=116'4830)", 2024-09-18T07:35:02.017 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.438822+0000", 2024-09-18T07:35:02.017 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 103.460382585, 2024-09-18T07:35:02.017 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.106356858, 2024-09-18T07:35:02.017 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:02.018 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:02.018 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:02.018 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:02.018 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.018 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:02.018 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.438822+0000", 2024-09-18T07:35:02.018 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:02.018 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.018 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.018 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:02.018 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.438822+0000", 2024-09-18T07:35:02.018 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:02.018 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.019 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.019 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:02.019 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.438823+0000", 2024-09-18T07:35:02.019 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2979999999999999e-06 2024-09-18T07:35:02.019 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.019 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.019 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:02.019 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.438847+0000", 2024-09-18T07:35:02.019 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3370000000000002e-05 2024-09-18T07:35:02.019 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.019 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.019 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:02.019 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541862+0000", 2024-09-18T07:35:02.019 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10301571299999999 2024-09-18T07:35:02.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.020 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.020 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:02.020 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541872+0000", 2024-09-18T07:35:02.020 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0017000000000001e-05 2024-09-18T07:35:02.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.020 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.020 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:02.020 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541931+0000", 2024-09-18T07:35:02.020 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8990000000000003e-05 2024-09-18T07:35:02.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.020 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.020 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:02.021 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541958+0000", 2024-09-18T07:35:02.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6996e-05 2024-09-18T07:35:02.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.021 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:02.021 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.545137+0000", 2024-09-18T07:35:02.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003178859 2024-09-18T07:35:02.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.021 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:02.021 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.545179+0000", 2024-09-18T07:35:02.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1615000000000001e-05 2024-09-18T07:35:02.021 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:02.021 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:02.022 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:02.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.022 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:142957 7.1 7:830d8edf:::benchmark_data_smithi137_74180_object38541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-18T07:35:02.022 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:25.430263+0000", 2024-09-18T07:35:02.022 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 36.468941821000001, 2024-09-18T07:35:02.022 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18925961299999999, 2024-09-18T07:35:02.022 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:02.022 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:02.022 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:02.022 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:02.022 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:02.022 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:02.023 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 142957 2024-09-18T07:35:02.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.023 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:02.023 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.023 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:02.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430263+0000", 2024-09-18T07:35:02.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:02.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.023 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.023 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:02.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430263+0000", 2024-09-18T07:35:02.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:02.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.023 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:02.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430265+0000", 2024-09-18T07:35:02.024 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9800000000000001e-06 2024-09-18T07:35:02.024 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.024 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:02.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430270+0000", 2024-09-18T07:35:02.024 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6189999999999999e-06 2024-09-18T07:35:02.024 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.024 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:02.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430281+0000", 2024-09-18T07:35:02.024 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1146e-05 2024-09-18T07:35:02.024 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.025 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.025 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:02.025 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430288+0000", 2024-09-18T07:35:02.025 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9319999999999999e-06 2024-09-18T07:35:02.025 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.025 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.025 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:02.025 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430340+0000", 2024-09-18T07:35:02.025 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1904999999999999e-05 2024-09-18T07:35:02.025 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.025 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.025 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:02.025 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430468+0000", 2024-09-18T07:35:02.026 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000127448 2024-09-18T07:35:02.026 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.026 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.026 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:35:02.026 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430717+0000", 2024-09-18T07:35:02.026 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024888500000000001 2024-09-18T07:35:02.026 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.026 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.026 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:02.026 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.432189+0000", 2024-09-18T07:35:02.026 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014722470000000001 2024-09-18T07:35:02.026 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.026 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.026 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:02.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.432734+0000", 2024-09-18T07:35:02.027 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00054558799999999995 2024-09-18T07:35:02.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:02.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.619409+0000", 2024-09-18T07:35:02.027 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18667450399999999 2024-09-18T07:35:02.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:02.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.619471+0000", 2024-09-18T07:35:02.027 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1877000000000004e-05 2024-09-18T07:35:02.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:02.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:02.028 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:02.028 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.619522+0000", 2024-09-18T07:35:02.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1481999999999999e-05 2024-09-18T07:35:02.028 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:02.028 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:02.028 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:02.028 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:02.028 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:02.028 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:35:02.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:02.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:02.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:02.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:02.153 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:02.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:02.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:02.255 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:02.276 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:02.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:02.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:02.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:02.404 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:02.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:02.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:02.506 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:02.507 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[29408]: 4.1 scrub starts 2024-09-18T07:35:02.507 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[29408]: 4.1 scrub ok 2024-09-18T07:35:02.507 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[29408]: 5.13 scrub starts 2024-09-18T07:35:02.507 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[29408]: 5.13 scrub ok 2024-09-18T07:35:02.507 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[29408]: osdmap e173: 8 total, 8 up, 8 in 2024-09-18T07:35:02.508 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[29408]: pgmap v211: 151 pgs: 1 remapped+peering, 5 active+recovery_wait+degraded, 1 peering, 1 active+recovering, 143 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 121 op/s; 15/140892 objects degraded (0.011%); 2/140892 objects misplaced (0.001%); 0 B/s, 3 objects/s recovering 2024-09-18T07:35:02.508 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-18T07:35:02.508 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-18T07:35:02.508 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[29408]: 6.1b scrub starts 2024-09-18T07:35:02.508 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[29408]: 6.1b scrub ok 2024-09-18T07:35:02.508 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[39175]: 4.1 scrub starts 2024-09-18T07:35:02.508 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[39175]: 4.1 scrub ok 2024-09-18T07:35:02.508 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[39175]: 5.13 scrub starts 2024-09-18T07:35:02.508 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[39175]: 5.13 scrub ok 2024-09-18T07:35:02.508 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[39175]: osdmap e173: 8 total, 8 up, 8 in 2024-09-18T07:35:02.509 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[39175]: pgmap v211: 151 pgs: 1 remapped+peering, 5 active+recovery_wait+degraded, 1 peering, 1 active+recovering, 143 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 121 op/s; 15/140892 objects degraded (0.011%); 2/140892 objects misplaced (0.001%); 0 B/s, 3 objects/s recovering 2024-09-18T07:35:02.509 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-18T07:35:02.509 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-18T07:35:02.509 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[39175]: 6.1b scrub starts 2024-09-18T07:35:02.509 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:02 smithi137 ceph-mon[39175]: 6.1b scrub ok 2024-09-18T07:35:02.531 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:02.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:02.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:02.557 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-18T07:35:02.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:02.663 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:02.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:02.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:02.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:02 smithi203 ceph-mon[32237]: 4.1 scrub starts 2024-09-18T07:35:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:02 smithi203 ceph-mon[32237]: 4.1 scrub ok 2024-09-18T07:35:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:02 smithi203 ceph-mon[32237]: 5.13 scrub starts 2024-09-18T07:35:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:02 smithi203 ceph-mon[32237]: 5.13 scrub ok 2024-09-18T07:35:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:02 smithi203 ceph-mon[32237]: osdmap e173: 8 total, 8 up, 8 in 2024-09-18T07:35:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:02 smithi203 ceph-mon[32237]: pgmap v211: 151 pgs: 1 remapped+peering, 5 active+recovery_wait+degraded, 1 peering, 1 active+recovering, 143 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 121 op/s; 15/140892 objects degraded (0.011%); 2/140892 objects misplaced (0.001%); 0 B/s, 3 objects/s recovering 2024-09-18T07:35:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:02 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-18T07:35:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:02 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"} : dispatch 2024-09-18T07:35:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:02 smithi203 ceph-mon[32237]: 6.1b scrub starts 2024-09-18T07:35:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:02 smithi203 ceph-mon[32237]: 6.1b scrub ok 2024-09-18T07:35:02.765 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:02.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:02.789 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:02.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:02.890 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:02.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:02.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:02.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:02.996 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T07:35:03.018 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:03.064 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:03.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:03.166 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:03.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:03.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:03.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:03.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:03.296 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:35:03.317 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:03.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:03.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:03.419 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:03.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:03.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:03.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:03.545 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:03.569 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:03.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:03.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:03.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[29408]: 6.1d scrub starts 2024-09-18T07:35:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[29408]: 6.1d scrub ok 2024-09-18T07:35:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[29408]: 3.f scrub starts 2024-09-18T07:35:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[29408]: 3.f scrub ok 2024-09-18T07:35:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[29408]: 4.15 scrub starts 2024-09-18T07:35:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[29408]: 4.15 scrub ok 2024-09-18T07:35:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[29408]: Health check failed: Degraded data redundancy: 15/140892 objects degraded (0.011%), 5 pgs degraded (PG_DEGRADED) 2024-09-18T07:35:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-18T07:35:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[29408]: osdmap e174: 8 total, 8 up, 8 in 2024-09-18T07:35:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[29408]: 7.c starting backfill to osd.7 from (0'0,0'0] MAX to 170'10122 2024-09-18T07:35:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[29408]: 7.5 starting backfill to osd.7 from (0'0,0'0] MAX to 170'9380 2024-09-18T07:35:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[29408]: 7.8 starting backfill to osd.7 from (0'0,0'0] MAX to 170'9752 2024-09-18T07:35:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[29408]: 7.f starting backfill to osd.7 from (0'0,0'0] MAX to 170'10161 2024-09-18T07:35:03.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[29408]: 5.11 scrub starts 2024-09-18T07:35:03.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[29408]: 5.11 scrub ok 2024-09-18T07:35:03.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[29408]: 7.6 starting backfill to osd.7 from (0'0,0'0] MAX to 170'9558 2024-09-18T07:35:03.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[39175]: 6.1d scrub starts 2024-09-18T07:35:03.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[39175]: 6.1d scrub ok 2024-09-18T07:35:03.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[39175]: 3.f scrub starts 2024-09-18T07:35:03.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[39175]: 3.f scrub ok 2024-09-18T07:35:03.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[39175]: 4.15 scrub starts 2024-09-18T07:35:03.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[39175]: 4.15 scrub ok 2024-09-18T07:35:03.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[39175]: Health check failed: Degraded data redundancy: 15/140892 objects degraded (0.011%), 5 pgs degraded (PG_DEGRADED) 2024-09-18T07:35:03.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-18T07:35:03.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[39175]: osdmap e174: 8 total, 8 up, 8 in 2024-09-18T07:35:03.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[39175]: 7.c starting backfill to osd.7 from (0'0,0'0] MAX to 170'10122 2024-09-18T07:35:03.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[39175]: 7.5 starting backfill to osd.7 from (0'0,0'0] MAX to 170'9380 2024-09-18T07:35:03.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[39175]: 7.8 starting backfill to osd.7 from (0'0,0'0] MAX to 170'9752 2024-09-18T07:35:03.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[39175]: 7.f starting backfill to osd.7 from (0'0,0'0] MAX to 170'10161 2024-09-18T07:35:03.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[39175]: 5.11 scrub starts 2024-09-18T07:35:03.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[39175]: 5.11 scrub ok 2024-09-18T07:35:03.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:03 smithi137 ceph-mon[39175]: 7.6 starting backfill to osd.7 from (0'0,0'0] MAX to 170'9558 2024-09-18T07:35:03.672 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:03.695 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:03.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:03.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:03.705 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:03 smithi203 ceph-mon[32237]: 6.1d scrub starts 2024-09-18T07:35:03.706 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:03 smithi203 ceph-mon[32237]: 6.1d scrub ok 2024-09-18T07:35:03.706 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:03 smithi203 ceph-mon[32237]: 3.f scrub starts 2024-09-18T07:35:03.706 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:03 smithi203 ceph-mon[32237]: 3.f scrub ok 2024-09-18T07:35:03.706 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:03 smithi203 ceph-mon[32237]: 4.15 scrub starts 2024-09-18T07:35:03.706 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:03 smithi203 ceph-mon[32237]: 4.15 scrub ok 2024-09-18T07:35:03.706 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:03 smithi203 ceph-mon[32237]: Health check failed: Degraded data redundancy: 15/140892 objects degraded (0.011%), 5 pgs degraded (PG_DEGRADED) 2024-09-18T07:35:03.706 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:03 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-18T07:35:03.706 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:03 smithi203 ceph-mon[32237]: osdmap e174: 8 total, 8 up, 8 in 2024-09-18T07:35:03.706 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:03 smithi203 ceph-mon[32237]: 7.c starting backfill to osd.7 from (0'0,0'0] MAX to 170'10122 2024-09-18T07:35:03.706 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:03 smithi203 ceph-mon[32237]: 7.5 starting backfill to osd.7 from (0'0,0'0] MAX to 170'9380 2024-09-18T07:35:03.706 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:03 smithi203 ceph-mon[32237]: 7.8 starting backfill to osd.7 from (0'0,0'0] MAX to 170'9752 2024-09-18T07:35:03.706 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:03 smithi203 ceph-mon[32237]: 7.f starting backfill to osd.7 from (0'0,0'0] MAX to 170'10161 2024-09-18T07:35:03.706 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:03 smithi203 ceph-mon[32237]: 5.11 scrub starts 2024-09-18T07:35:03.707 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:03 smithi203 ceph-mon[32237]: 5.11 scrub ok 2024-09-18T07:35:03.707 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:03 smithi203 ceph-mon[32237]: 7.6 starting backfill to osd.7 from (0'0,0'0] MAX to 170'9558 2024-09-18T07:35:03.797 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:03.822 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:35:03.822 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:03.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:03.831 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:35:03.831 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:35:03.831 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-09-18T07:35:03.831 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:35:03.925 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:03.948 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:03.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:03.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:04.050 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:04.073 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:04.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:04.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:04.134 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:04.175 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:04.202 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:04.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:04.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:04.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:04.329 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:04.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:04.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:04.431 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:04.457 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:04.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:04.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:04.536 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T07:35:04.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:04.585 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:04.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:04.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:04.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:04 smithi137 ceph-mon[29408]: 3.0 scrub starts 2024-09-18T07:35:04.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:04 smithi137 ceph-mon[29408]: 3.0 scrub ok 2024-09-18T07:35:04.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:04 smithi137 ceph-mon[29408]: osdmap e175: 8 total, 8 up, 8 in 2024-09-18T07:35:04.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:04 smithi137 ceph-mon[29408]: pgmap v214: 151 pgs: 5 remapped+peering, 5 active+recovery_wait+degraded, 1 active+recovering, 140 active+clean; 2.8 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 15/140061 objects degraded (0.011%); 2/140061 objects misplaced (0.001%); 131 B/s, 13 objects/s recovering 2024-09-18T07:35:04.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:04 smithi137 ceph-mon[39175]: 3.0 scrub starts 2024-09-18T07:35:04.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:04 smithi137 ceph-mon[39175]: 3.0 scrub ok 2024-09-18T07:35:04.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:04 smithi137 ceph-mon[39175]: osdmap e175: 8 total, 8 up, 8 in 2024-09-18T07:35:04.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:04 smithi137 ceph-mon[39175]: pgmap v214: 151 pgs: 5 remapped+peering, 5 active+recovery_wait+degraded, 1 active+recovering, 140 active+clean; 2.8 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 15/140061 objects degraded (0.011%); 2/140061 objects misplaced (0.001%); 131 B/s, 13 objects/s recovering 2024-09-18T07:35:04.687 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:04.710 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:04 smithi203 ceph-mon[32237]: 3.0 scrub starts 2024-09-18T07:35:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:04 smithi203 ceph-mon[32237]: 3.0 scrub ok 2024-09-18T07:35:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:04 smithi203 ceph-mon[32237]: osdmap e175: 8 total, 8 up, 8 in 2024-09-18T07:35:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:04 smithi203 ceph-mon[32237]: pgmap v214: 151 pgs: 5 remapped+peering, 5 active+recovery_wait+degraded, 1 active+recovering, 140 active+clean; 2.8 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 15/140061 objects degraded (0.011%); 2/140061 objects misplaced (0.001%); 131 B/s, 13 objects/s recovering 2024-09-18T07:35:04.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:04.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:04.812 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:35:04.812 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:04.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:04.849 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:04.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:04.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:04.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:35:04.989 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:04.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:04.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:05.091 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:05.118 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:05.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:05.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:05.220 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:05.270 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:05.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:05.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:05.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:05.419 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:05.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:05.454 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:35:05.455 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:35:05.455 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:35:05.455 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:35:05.455 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:35:05.521 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:05.549 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:05.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:05.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:05.651 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:05.657 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:05 smithi137 ceph-mon[29408]: 3.9 scrub starts 2024-09-18T07:35:05.658 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:05 smithi137 ceph-mon[29408]: 3.9 scrub ok 2024-09-18T07:35:05.658 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:05 smithi137 ceph-mon[29408]: 5.1b scrub starts 2024-09-18T07:35:05.658 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:05 smithi137 ceph-mon[29408]: 5.1b scrub ok 2024-09-18T07:35:05.658 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:05 smithi137 ceph-mon[29408]: osdmap e176: 8 total, 8 up, 8 in 2024-09-18T07:35:05.658 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:05 smithi137 ceph-mon[29408]: osdmap e177: 8 total, 8 up, 8 in 2024-09-18T07:35:05.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:05 smithi137 ceph-mon[39175]: 3.9 scrub starts 2024-09-18T07:35:05.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:05 smithi137 ceph-mon[39175]: 3.9 scrub ok 2024-09-18T07:35:05.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:05 smithi137 ceph-mon[39175]: 5.1b scrub starts 2024-09-18T07:35:05.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:05 smithi137 ceph-mon[39175]: 5.1b scrub ok 2024-09-18T07:35:05.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:05 smithi137 ceph-mon[39175]: osdmap e176: 8 total, 8 up, 8 in 2024-09-18T07:35:05.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:05 smithi137 ceph-mon[39175]: osdmap e177: 8 total, 8 up, 8 in 2024-09-18T07:35:05.678 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:05.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:05.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:05.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:05 smithi203 ceph-mon[32237]: 3.9 scrub starts 2024-09-18T07:35:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:05 smithi203 ceph-mon[32237]: 3.9 scrub ok 2024-09-18T07:35:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:05 smithi203 ceph-mon[32237]: 5.1b scrub starts 2024-09-18T07:35:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:05 smithi203 ceph-mon[32237]: 5.1b scrub ok 2024-09-18T07:35:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:05 smithi203 ceph-mon[32237]: osdmap e176: 8 total, 8 up, 8 in 2024-09-18T07:35:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:05 smithi203 ceph-mon[32237]: osdmap e177: 8 total, 8 up, 8 in 2024-09-18T07:35:05.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:05.833 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:05.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:05.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:05.864 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:05.935 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:05.964 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:05.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:05.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:06.066 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:06.093 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:06.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:06.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:06.230 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T07:35:06.236 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:06.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:06.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:06.338 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:06.388 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:06.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:06.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:06.419 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:35:06.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:06 smithi203 ceph-mon[32237]: 6.e scrub starts 2024-09-18T07:35:06.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:06 smithi203 ceph-mon[32237]: 6.e scrub ok 2024-09-18T07:35:06.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:06 smithi203 ceph-mon[32237]: pgmap v217: 150 pgs: 5 remapped+peering, 5 active+recovery_wait+degraded, 1 active+recovering, 139 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 438 op/s; 15/135663 objects degraded (0.011%); 2/135663 objects misplaced (0.001%); 192 B/s, 10 objects/s recovering 2024-09-18T07:35:06.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:06 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-18T07:35:06.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:06 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-18T07:35:06.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:06 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1089018296' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:35:06.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:06 smithi203 ceph-mon[32237]: from='client.25060 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:35:06.491 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:06.520 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:06.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:06.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:06.549 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:35:06.549 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:35:06.576 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:35:06.580 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:35:06.580 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:35:06.613 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:35:06.613 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:06 smithi137 ceph-mon[29408]: 6.e scrub starts 2024-09-18T07:35:06.614 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:06 smithi137 ceph-mon[29408]: 6.e scrub ok 2024-09-18T07:35:06.614 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:06 smithi137 ceph-mon[29408]: pgmap v217: 150 pgs: 5 remapped+peering, 5 active+recovery_wait+degraded, 1 active+recovering, 139 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 438 op/s; 15/135663 objects degraded (0.011%); 2/135663 objects misplaced (0.001%); 192 B/s, 10 objects/s recovering 2024-09-18T07:35:06.614 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:06 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-18T07:35:06.614 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:06 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-18T07:35:06.614 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:06 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1089018296' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:35:06.614 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:06 smithi137 ceph-mon[29408]: from='client.25060 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:35:06.614 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:06 smithi137 ceph-mon[39175]: 6.e scrub starts 2024-09-18T07:35:06.614 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:06 smithi137 ceph-mon[39175]: 6.e scrub ok 2024-09-18T07:35:06.614 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:06 smithi137 ceph-mon[39175]: pgmap v217: 150 pgs: 5 remapped+peering, 5 active+recovery_wait+degraded, 1 active+recovering, 139 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 438 op/s; 15/135663 objects degraded (0.011%); 2/135663 objects misplaced (0.001%); 192 B/s, 10 objects/s recovering 2024-09-18T07:35:06.614 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:06 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-18T07:35:06.615 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:06 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"} : dispatch 2024-09-18T07:35:06.615 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:06 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1089018296' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:35:06.615 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:06 smithi137 ceph-mon[39175]: from='client.25060 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:35:06.615 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:35:06.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:06.644 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:35:06.644 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:35:06.656 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:06.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:06.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:06.692 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:35:06.692 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:35:06.701 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:35:06.701 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:35:06.727 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:35:06.737 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:35:06.749 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:35:06.749 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:35:06.793 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:06.817 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:06.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:06.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:06.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:06.944 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:06.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:06.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:07.046 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:07.073 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:07.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:07.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:07.112 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:35:07.112 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:35:07.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:07.203 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:07.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:07.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:07.305 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:07.330 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:07.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:07.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:07.352 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:35:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:35:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:35:07.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:35:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:35:07.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:07.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:07.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:07.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:07.557 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:35:07.557 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:35:07.558 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-18T07:35:07.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-18T07:35:07.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:07.586 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:07.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:07.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:07.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:07 smithi137 ceph-mon[29408]: 4.11 scrub starts 2024-09-18T07:35:07.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:07 smithi137 ceph-mon[29408]: 4.11 scrub ok 2024-09-18T07:35:07.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:07 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:07.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:07 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-18T07:35:07.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:07 smithi137 ceph-mon[29408]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:35:07.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:07 smithi137 ceph-mon[29408]: osdmap e178: 8 total, 8 up, 8 in 2024-09-18T07:35:07.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:07 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1313173921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:07.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:07 smithi137 ceph-mon[39175]: 4.11 scrub starts 2024-09-18T07:35:07.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:07 smithi137 ceph-mon[39175]: 4.11 scrub ok 2024-09-18T07:35:07.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:07 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:07.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:07 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-18T07:35:07.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:07 smithi137 ceph-mon[39175]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:35:07.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:07 smithi137 ceph-mon[39175]: osdmap e178: 8 total, 8 up, 8 in 2024-09-18T07:35:07.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:07 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1313173921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:07.688 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:07.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:07 smithi203 ceph-mon[32237]: 4.11 scrub starts 2024-09-18T07:35:07.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:07 smithi203 ceph-mon[32237]: 4.11 scrub ok 2024-09-18T07:35:07.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:07 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:07.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:07 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-18T07:35:07.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:07 smithi203 ceph-mon[32237]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:35:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:07 smithi203 ceph-mon[32237]: osdmap e178: 8 total, 8 up, 8 in 2024-09-18T07:35:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:07 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1313173921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:07.715 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:07.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:07.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:07.841 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:07.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:07.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:07.943 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:07.968 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:07.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:07.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:08.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:08.096 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:08.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:08.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:08.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:08.222 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:08.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:08.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:08.324 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:08.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:08.349 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:08.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:08.450 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:08.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:08.482 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:08.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:08.583 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:08.608 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:08.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:08.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:08.710 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:08.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:08 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:08.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:08 smithi203 ceph-mon[32237]: osdmap e179: 8 total, 8 up, 8 in 2024-09-18T07:35:08.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:08 smithi203 ceph-mon[32237]: pgmap v220: 150 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 peering, 1 active+remapped+backfill_toofull, 143 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 702 op/s; 9177/131649 objects misplaced (6.971%); 77 MiB/s, 1.23k objects/s recovering 2024-09-18T07:35:08.735 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:08.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:08.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:08.837 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:08.863 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:08.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:08.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:08.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:08 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:08.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:08 smithi137 ceph-mon[29408]: osdmap e179: 8 total, 8 up, 8 in 2024-09-18T07:35:08.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:08 smithi137 ceph-mon[29408]: pgmap v220: 150 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 peering, 1 active+remapped+backfill_toofull, 143 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 702 op/s; 9177/131649 objects misplaced (6.971%); 77 MiB/s, 1.23k objects/s recovering 2024-09-18T07:35:08.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:08 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:08.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:08 smithi137 ceph-mon[39175]: osdmap e179: 8 total, 8 up, 8 in 2024-09-18T07:35:08.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:08 smithi137 ceph-mon[39175]: pgmap v220: 150 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 peering, 1 active+remapped+backfill_toofull, 143 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 702 op/s; 9177/131649 objects misplaced (6.971%); 77 MiB/s, 1.23k objects/s recovering 2024-09-18T07:35:08.964 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:08.985 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:08.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:08.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:09.089 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:09.112 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:09.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:09.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:09.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:09.216 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:35:09.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:09.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:09.326 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T07:35:09.338 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:09.393 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:09.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:09.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:09.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:09.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:09.521 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:09.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:09.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:09.638 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:35:09.643 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:09.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:09.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:09.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:09 smithi137 ceph-mon[39175]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:35:09.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:09 smithi137 ceph-mon[39175]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15/135663 objects degraded (0.011%), 5 pgs degraded) 2024-09-18T07:35:09.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:09 smithi137 ceph-mon[39175]: osdmap e180: 8 total, 8 up, 8 in 2024-09-18T07:35:09.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:09 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:35:09.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:09 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:35:09.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:09 smithi137 ceph-mon[29408]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:35:09.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:09 smithi137 ceph-mon[29408]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15/135663 objects degraded (0.011%), 5 pgs degraded) 2024-09-18T07:35:09.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:09 smithi137 ceph-mon[29408]: osdmap e180: 8 total, 8 up, 8 in 2024-09-18T07:35:09.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:09 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:35:09.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:09 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:35:09.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:09 smithi203 ceph-mon[32237]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:35:09.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:09 smithi203 ceph-mon[32237]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15/135663 objects degraded (0.011%), 5 pgs degraded) 2024-09-18T07:35:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:09 smithi203 ceph-mon[32237]: osdmap e180: 8 total, 8 up, 8 in 2024-09-18T07:35:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:09 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:35:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:09 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:35:09.745 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:09.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:09.768 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:09.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:09.869 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:09.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:09.895 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:09.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:09.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:10.018 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:10.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:10.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:10.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:10.143 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:10.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:10.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:10.147 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:35:10.147 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:35:10.245 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:10.267 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:10.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:10.369 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:10.392 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:10.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:10.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:10.494 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:10.524 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:10.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:10.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:10.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:10.652 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:10.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:10.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:10.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:10 smithi203 ceph-mon[32237]: osdmap e181: 8 total, 8 up, 8 in 2024-09-18T07:35:10.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:10 smithi203 ceph-mon[32237]: pgmap v223: 149 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 1 peering, 1 active+remapped+backfill_toofull, 143 active+clean; 2.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 674 op/s; 9177/127332 objects misplaced (7.207%); 77 MiB/s, 1.23k objects/s recovering 2024-09-18T07:35:10.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:10 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-18T07:35:10.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:10 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-18T07:35:10.753 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:10.774 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:10.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:10.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:10.876 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:10.893 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:10 smithi137 ceph-mon[29408]: osdmap e181: 8 total, 8 up, 8 in 2024-09-18T07:35:10.894 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:10 smithi137 ceph-mon[29408]: pgmap v223: 149 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 1 peering, 1 active+remapped+backfill_toofull, 143 active+clean; 2.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 674 op/s; 9177/127332 objects misplaced (7.207%); 77 MiB/s, 1.23k objects/s recovering 2024-09-18T07:35:10.894 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:10 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-18T07:35:10.894 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:10 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-18T07:35:10.894 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:10 smithi137 ceph-mon[39175]: osdmap e181: 8 total, 8 up, 8 in 2024-09-18T07:35:10.894 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:10 smithi137 ceph-mon[39175]: pgmap v223: 149 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 1 peering, 1 active+remapped+backfill_toofull, 143 active+clean; 2.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 674 op/s; 9177/127332 objects misplaced (7.207%); 77 MiB/s, 1.23k objects/s recovering 2024-09-18T07:35:10.894 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:10 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-18T07:35:10.894 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:10 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"} : dispatch 2024-09-18T07:35:10.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:10.900 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:10.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:10.921 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T07:35:11.001 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:11.027 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:11.029 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:11.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:11.030 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:11.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:11.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:11.152 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:11.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:11.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:11.241 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:35:11.253 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:11.278 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:11.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:11.380 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:11.402 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:11.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:11.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:11.504 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:11.528 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:11.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:11.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:11.630 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:11.643 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:11 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-18T07:35:11.643 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:11 smithi203 ceph-mon[32237]: osdmap e182: 8 total, 8 up, 8 in 2024-09-18T07:35:11.643 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:11 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:35:11.643 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:11 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:35:11.655 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:11.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:11.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:11.757 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:11.782 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:11.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:11.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:11.859 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:35:11.860 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:35:11.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:11.910 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:11.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:11.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:11.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:11 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-18T07:35:11.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:11 smithi137 ceph-mon[39175]: osdmap e182: 8 total, 8 up, 8 in 2024-09-18T07:35:11.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:11 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:35:11.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:11 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:35:11.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:11 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-18T07:35:11.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:11 smithi137 ceph-mon[29408]: osdmap e182: 8 total, 8 up, 8 in 2024-09-18T07:35:11.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:11 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:35:11.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:11 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:35:12.011 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:12.035 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:12.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:12.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:12.136 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:12.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:12.163 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:12.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:12.264 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:12.287 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:12.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:12.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:12.389 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:12.411 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:12.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:12.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:12.513 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:12.537 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:12.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:12.565 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:12 smithi203 ceph-mon[32237]: osdmap e183: 8 total, 8 up, 8 in 2024-09-18T07:35:12.565 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:12 smithi203 ceph-mon[32237]: pgmap v226: 149 pgs: 2 peering, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 143 active+clean; 2.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 56 op/s; 6346/130980 objects misplaced (4.845%); 44 MiB/s, 697 objects/s recovering 2024-09-18T07:35:12.599 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T07:35:12.639 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:12.661 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:12.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:12.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:12.729 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:12 smithi137 ceph-mon[29408]: osdmap e183: 8 total, 8 up, 8 in 2024-09-18T07:35:12.729 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:12 smithi137 ceph-mon[29408]: pgmap v226: 149 pgs: 2 peering, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 143 active+clean; 2.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 56 op/s; 6346/130980 objects misplaced (4.845%); 44 MiB/s, 697 objects/s recovering 2024-09-18T07:35:12.729 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:12 smithi137 ceph-mon[39175]: osdmap e183: 8 total, 8 up, 8 in 2024-09-18T07:35:12.729 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:12 smithi137 ceph-mon[39175]: pgmap v226: 149 pgs: 2 peering, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 143 active+clean; 2.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 56 op/s; 6346/130980 objects misplaced (4.845%); 44 MiB/s, 697 objects/s recovering 2024-09-18T07:35:12.763 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:12.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:12.787 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:12.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:12.887 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:12.912 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:12.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:12.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:12.918 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:35:12.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:35:13.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:13.035 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:13.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:13.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:13.137 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:13.161 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:13.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:13.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:13.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:13.285 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:13.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:13.386 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:13.408 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:13.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:13.444 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:35:13.445 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:35:13.510 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:13.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:13.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:13.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:13.643 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:13.668 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:13.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:13.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:13.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:13 smithi203 ceph-mon[32237]: osdmap e184: 8 total, 8 up, 8 in 2024-09-18T07:35:13.770 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:13.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:13 smithi137 ceph-mon[29408]: osdmap e184: 8 total, 8 up, 8 in 2024-09-18T07:35:13.774 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:13 smithi137 ceph-mon[39175]: osdmap e184: 8 total, 8 up, 8 in 2024-09-18T07:35:13.791 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:13.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:13.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:13.892 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:13.914 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:13.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:14.015 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:14.037 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:14.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:14.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:14.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:14.160 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:14.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:14.185 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T07:35:14.262 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:14.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:14.285 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:14.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:14.369 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:14.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:14.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:14.409 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:14.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:14.433 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:35:14.479 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:35:14.497 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:35:14.497 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:35:14.497 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 186, 2024-09-18T07:35:14.497 INFO:teuthology.orchestra.run.smithi137.stdout: "fsid": "f7ef099c-758e-11ef-bceb-c7b262605968", 2024-09-18T07:35:14.497 INFO:teuthology.orchestra.run.smithi137.stdout: "created": "2024-09-18T07:25:19.662367+0000", 2024-09-18T07:35:14.497 INFO:teuthology.orchestra.run.smithi137.stdout: "modified": "2024-09-18T07:35:14.463373+0000", 2024-09-18T07:35:14.497 INFO:teuthology.orchestra.run.smithi137.stdout: "last_up_change": "2024-09-18T07:30:44.708496+0000", 2024-09-18T07:35:14.497 INFO:teuthology.orchestra.run.smithi137.stdout: "last_in_change": "2024-09-18T07:35:00.369558+0000", 2024-09-18T07:35:14.497 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T07:35:14.497 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_num": 5806080, 2024-09-18T07:35:14.498 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_set": [ 2024-09-18T07:35:14.498 INFO:teuthology.orchestra.run.smithi137.stdout: "nodeep-scrub", 2024-09-18T07:35:14.498 INFO:teuthology.orchestra.run.smithi137.stdout: "noscrub", 2024-09-18T07:35:14.498 INFO:teuthology.orchestra.run.smithi137.stdout: "pglog_hardlimit", 2024-09-18T07:35:14.498 INFO:teuthology.orchestra.run.smithi137.stdout: "purged_snapdirs", 2024-09-18T07:35:14.498 INFO:teuthology.orchestra.run.smithi137.stdout: "recovery_deletes", 2024-09-18T07:35:14.498 INFO:teuthology.orchestra.run.smithi137.stdout: "sortbitwise" 2024-09-18T07:35:14.498 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:35:14.498 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_version": 18, 2024-09-18T07:35:14.498 INFO:teuthology.orchestra.run.smithi137.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T07:35:14.498 INFO:teuthology.orchestra.run.smithi137.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T07:35:14.498 INFO:teuthology.orchestra.run.smithi137.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T07:35:14.498 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_snapshot": "", 2024-09-18T07:35:14.499 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_max": 7, 2024-09-18T07:35:14.499 INFO:teuthology.orchestra.run.smithi137.stdout: "max_osd": 8, 2024-09-18T07:35:14.499 INFO:teuthology.orchestra.run.smithi137.stdout: "require_min_compat_client": "luminous", 2024-09-18T07:35:14.499 INFO:teuthology.orchestra.run.smithi137.stdout: "min_compat_client": "luminous", 2024-09-18T07:35:14.499 INFO:teuthology.orchestra.run.smithi137.stdout: "require_osd_release": "squid", 2024-09-18T07:35:14.499 INFO:teuthology.orchestra.run.smithi137.stdout: "allow_crimson": false, 2024-09-18T07:35:14.499 INFO:teuthology.orchestra.run.smithi137.stdout: "pools": [ 2024-09-18T07:35:14.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.499 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 1, 2024-09-18T07:35:14.499 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".mgr", 2024-09-18T07:35:14.499 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:28:20.899655+0000", 2024-09-18T07:35:14.499 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:35:14.500 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:35:14.500 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:35:14.500 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:35:14.500 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:35:14.500 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:35:14.500 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:35:14.500 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:35:14.500 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:35:14.500 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:35:14.500 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:35:14.500 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:35:14.500 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:35:14.501 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 1, 2024-09-18T07:35:14.501 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 1, 2024-09-18T07:35:14.501 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 1, 2024-09-18T07:35:14.501 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 1, 2024-09-18T07:35:14.501 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 1, 2024-09-18T07:35:14.501 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:35:14.501 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:35:14.501 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:35:14.501 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:35:14.501 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:35:14.501 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:35:14.501 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:35:14.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.502 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "21", 2024-09-18T07:35:14.502 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:35:14.502 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:35:14.502 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:35:14.502 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:35:14.502 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:35:14.502 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:35:14.502 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:35:14.502 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:35:14.502 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:35:14.502 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:35:14.502 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:35:14.502 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:35:14.503 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:35:14.503 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:35:14.503 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:35:14.503 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:35:14.503 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:35:14.503 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:35:14.503 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:35:14.503 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:35:14.503 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:35:14.503 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:35:14.503 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:35:14.503 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:35:14.504 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:35:14.504 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:35:14.504 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.504 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:35:14.504 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:35:14.504 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:35:14.504 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:35:14.504 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:35:14.504 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:35:14.504 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:35:14.504 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:35:14.504 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:35:14.505 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:35:14.505 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:35:14.505 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:35:14.505 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_max": 32, 2024-09-18T07:35:14.505 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_min": 1 2024-09-18T07:35:14.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.505 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:35:14.505 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": {} 2024-09-18T07:35:14.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.505 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:35:14.505 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:35:14.505 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 7.8899998664855957, 2024-09-18T07:35:14.505 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 7.8899998664855957, 2024-09-18T07:35:14.506 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.37999999523162842, 2024-09-18T07:35:14.506 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 3, 2024-09-18T07:35:14.506 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 3, 2024-09-18T07:35:14.506 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:35:14.506 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:35:14.506 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:35:14.506 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.506 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 2, 2024-09-18T07:35:14.506 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "datapool", 2024-09-18T07:35:14.506 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:53.715373+0000", 2024-09-18T07:35:14.506 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-18T07:35:14.507 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T07:35:14.507 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:35:14.507 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:35:14.507 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:35:14.507 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:35:14.507 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:35:14.507 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:35:14.507 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:35:14.507 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:35:14.507 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:35:14.507 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:35:14.507 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:35:14.507 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 3, 2024-09-18T07:35:14.508 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 3, 2024-09-18T07:35:14.508 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 3, 2024-09-18T07:35:14.508 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 3, 2024-09-18T07:35:14.508 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 3, 2024-09-18T07:35:14.508 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:35:14.508 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:35:14.508 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:35:14.508 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:35:14.508 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:35:14.508 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:35:14.508 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:35:14.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.509 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "63", 2024-09-18T07:35:14.509 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:35:14.509 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:35:14.509 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:35:14.509 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:35:14.509 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:35:14.509 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 2, 2024-09-18T07:35:14.509 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 63, 2024-09-18T07:35:14.509 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:35:14.509 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:35:14.509 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:35:14.509 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:35:14.509 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:35:14.510 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:35:14.510 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:35:14.510 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:35:14.510 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:35:14.510 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:35:14.510 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:35:14.510 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:35:14.510 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:35:14.510 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:35:14.511 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:35:14.511 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:35:14.511 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:35:14.511 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:35:14.511 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:35:14.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.511 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:35:14.511 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:35:14.511 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:35:14.511 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:35:14.511 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:35:14.511 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:35:14.511 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:35:14.512 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:35:14.512 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:35:14.512 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:35:14.512 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:35:14.512 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:35:14.512 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:35:14.512 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd": {} 2024-09-18T07:35:14.512 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.512 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:35:14.512 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:35:14.512 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.6500000953674316, 2024-09-18T07:35:14.512 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.6500000953674316, 2024-09-18T07:35:14.512 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.87999999523162842, 2024-09-18T07:35:14.512 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-18T07:35:14.513 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-18T07:35:14.513 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:35:14.513 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:35:14.513 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:35:14.513 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.513 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 3, 2024-09-18T07:35:14.513 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".rgw.root", 2024-09-18T07:35:14.513 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:54.672127+0000", 2024-09-18T07:35:14.513 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:35:14.513 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:35:14.513 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:35:14.514 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:35:14.514 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:35:14.514 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:35:14.514 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:35:14.514 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:35:14.514 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:35:14.514 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:35:14.514 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:35:14.514 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:35:14.514 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:35:14.514 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:35:14.514 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:35:14.514 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:35:14.514 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:35:14.515 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:35:14.515 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:35:14.515 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:35:14.515 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:35:14.515 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:35:14.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:14.516 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:35:14.516 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:35:14.516 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:35:14.517 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.517 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "62", 2024-09-18T07:35:14.517 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:35:14.517 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:35:14.517 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:35:14.517 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:35:14.517 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:35:14.517 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:35:14.517 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:35:14.517 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:35:14.517 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:35:14.518 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:35:14.518 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:35:14.518 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:35:14.518 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:35:14.518 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:35:14.518 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:35:14.518 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:35:14.518 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:35:14.518 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:35:14.518 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:35:14.518 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:35:14.518 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:35:14.518 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:35:14.519 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:35:14.519 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:35:14.519 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:35:14.519 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:35:14.519 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.519 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:35:14.519 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:35:14.519 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:35:14.519 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:35:14.519 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:35:14.519 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:35:14.519 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:35:14.519 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:35:14.520 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:35:14.520 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:35:14.520 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:35:14.520 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:35:14.520 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:35:14.520 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:35:14.520 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.520 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:35:14.520 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:35:14.520 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.5, 2024-09-18T07:35:14.520 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.5, 2024-09-18T07:35:14.520 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:35:14.520 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.5, 2024-09-18T07:35:14.521 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.5, 2024-09-18T07:35:14.521 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:35:14.521 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:35:14.521 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:35:14.521 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.521 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.521 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.521 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 4, 2024-09-18T07:35:14.521 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.log", 2024-09-18T07:35:14.521 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:56.351343+0000", 2024-09-18T07:35:14.521 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:35:14.521 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:35:14.521 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:35:14.522 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:35:14.522 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:35:14.522 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:35:14.522 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:35:14.522 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:35:14.522 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:35:14.522 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:35:14.522 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:35:14.522 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:35:14.522 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:35:14.522 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:35:14.522 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:35:14.522 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:35:14.523 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:35:14.523 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:35:14.523 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:35:14.523 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:35:14.523 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:35:14.523 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:35:14.523 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:35:14.523 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:35:14.523 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:35:14.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.523 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "64", 2024-09-18T07:35:14.523 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:35:14.523 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:35:14.524 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:35:14.524 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:35:14.524 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:35:14.524 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:35:14.524 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:35:14.524 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:35:14.524 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:35:14.524 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:35:14.524 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:35:14.525 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:35:14.525 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:35:14.525 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:35:14.525 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:35:14.525 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:35:14.525 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:35:14.525 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:35:14.525 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:35:14.525 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:35:14.525 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:35:14.525 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:35:14.525 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:35:14.525 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:35:14.526 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:35:14.526 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:35:14.526 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.526 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:35:14.526 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:35:14.526 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:35:14.526 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:35:14.526 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:35:14.526 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:35:14.527 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:35:14.527 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:35:14.527 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:35:14.527 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:35:14.527 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:35:14.527 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:35:14.527 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:35:14.527 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:35:14.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.527 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:35:14.527 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:35:14.527 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.25, 2024-09-18T07:35:14.527 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.25, 2024-09-18T07:35:14.528 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:35:14.528 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.25, 2024-09-18T07:35:14.528 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.25, 2024-09-18T07:35:14.528 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:35:14.528 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:35:14.528 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:35:14.528 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.528 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.528 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 5, 2024-09-18T07:35:14.528 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.control", 2024-09-18T07:35:14.528 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:58.322401+0000", 2024-09-18T07:35:14.528 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:35:14.529 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:35:14.529 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:35:14.529 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:35:14.529 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:35:14.529 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:35:14.529 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:35:14.529 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:35:14.529 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:35:14.529 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:35:14.529 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:35:14.529 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:35:14.529 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:35:14.529 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:35:14.530 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:35:14.530 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:35:14.530 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:35:14.530 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:35:14.530 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:35:14.530 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:35:14.530 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:35:14.530 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:35:14.530 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:35:14.530 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:35:14.530 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:35:14.530 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.530 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "66", 2024-09-18T07:35:14.531 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:35:14.531 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:35:14.531 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:35:14.531 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:35:14.531 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:35:14.531 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:35:14.531 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:35:14.531 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:35:14.531 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:35:14.531 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:35:14.531 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:35:14.531 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:35:14.531 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:35:14.531 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:35:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:35:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:35:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:35:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:35:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:35:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:35:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:35:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:35:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:35:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:35:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:35:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:35:14.532 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:35:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:35:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:35:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:35:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:35:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:35:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:35:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:35:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:35:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:35:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:35:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:35:14.533 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:35:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:35:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:35:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:35:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.25, 2024-09-18T07:35:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.25, 2024-09-18T07:35:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:35:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.25, 2024-09-18T07:35:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.25, 2024-09-18T07:35:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:35:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:35:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:35:14.534 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 6, 2024-09-18T07:35:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.meta", 2024-09-18T07:35:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:31:00.390499+0000", 2024-09-18T07:35:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:35:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:35:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:35:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:35:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:35:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:35:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:35:14.535 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:35:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:35:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:35:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:35:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:35:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:35:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:35:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:35:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:35:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:35:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:35:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:35:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:35:14.536 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:35:14.537 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:35:14.537 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:35:14.537 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:35:14.537 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:35:14.537 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.537 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "68", 2024-09-18T07:35:14.537 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:35:14.537 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:35:14.537 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:35:14.537 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:35:14.537 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:35:14.537 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:35:14.537 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:35:14.538 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:35:14.538 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:35:14.538 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:35:14.538 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:35:14.538 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:35:14.538 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:35:14.538 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:35:14.538 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:35:14.538 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:35:14.538 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:35:14.538 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:35:14.538 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:35:14.538 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:35:14.538 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:35:14.539 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:35:14.539 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:35:14.539 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:35:14.539 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:35:14.539 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:35:14.539 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.539 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:35:14.539 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:35:14.539 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:35:14.539 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:35:14.539 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:35:14.539 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:35:14.539 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:35:14.540 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:35:14.540 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:35:14.540 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:35:14.540 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:35:14.540 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:35:14.540 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_bias": 4 2024-09-18T07:35:14.540 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.540 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:35:14.540 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:35:14.540 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.540 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:35:14.540 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:35:14.540 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.5, 2024-09-18T07:35:14.541 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.5, 2024-09-18T07:35:14.541 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:35:14.541 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.5, 2024-09-18T07:35:14.541 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.5, 2024-09-18T07:35:14.541 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:35:14.541 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:35:14.541 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:35:14.541 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.541 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.541 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.541 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-18T07:35:14.541 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "unique_pool_0", 2024-09-18T07:35:14.541 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:32:06.616722+0000", 2024-09-18T07:35:14.541 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:35:14.542 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:35:14.542 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:35:14.542 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:35:14.542 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:35:14.542 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:35:14.542 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:35:14.542 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:35:14.542 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:35:14.542 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:35:14.542 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:35:14.542 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:35:14.542 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:35:14.542 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 16, 2024-09-18T07:35:14.543 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 16, 2024-09-18T07:35:14.543 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 16, 2024-09-18T07:35:14.543 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 16, 2024-09-18T07:35:14.543 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 16, 2024-09-18T07:35:14.543 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:35:14.543 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "7.10", 2024-09-18T07:35:14.543 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 184, 2024-09-18T07:35:14.543 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 183, 2024-09-18T07:35:14.543 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 183, 2024-09-18T07:35:14.543 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "181'9339", 2024-09-18T07:35:14.543 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "181'9284" 2024-09-18T07:35:14.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.543 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "184", 2024-09-18T07:35:14.544 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:35:14.544 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "184", 2024-09-18T07:35:14.544 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "182", 2024-09-18T07:35:14.544 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:35:14.544 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:35:14.544 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:35:14.544 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:35:14.544 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:35:14.544 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:35:14.544 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:35:14.544 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:35:14.544 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:35:14.544 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:35:14.545 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:35:14.545 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:35:14.545 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:35:14.545 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:35:14.545 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:35:14.545 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:35:14.545 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:35:14.545 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:35:14.545 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:35:14.545 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:35:14.545 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:35:14.545 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:35:14.545 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:35:14.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.546 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:35:14.546 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:35:14.546 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:35:14.546 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:35:14.546 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:35:14.546 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:35:14.546 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:35:14.546 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:35:14.546 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:35:14.546 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:35:14.546 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:35:14.546 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:35:14.546 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:35:14.547 INFO:teuthology.orchestra.run.smithi137.stdout: "rados": {} 2024-09-18T07:35:14.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.547 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:35:14.547 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:35:14.547 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2, 2024-09-18T07:35:14.547 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2, 2024-09-18T07:35:14.547 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:35:14.547 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2, 2024-09-18T07:35:14.547 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2, 2024-09-18T07:35:14.547 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:35:14.547 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:35:14.547 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:35:14.547 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.548 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.548 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:35:14.548 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-18T07:35:14.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.548 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:35:14.548 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "268cd580-c931-4992-bb93-d6f63c2bfe88", 2024-09-18T07:35:14.548 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:35:14.548 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:35:14.548 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:35:14.548 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:35:14.548 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:35:14.548 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:35:14.548 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 8, 2024-09-18T07:35:14.549 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 181, 2024-09-18T07:35:14.549 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:35:14.549 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:35:14.549 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:35:14.549 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.549 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.549 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.549 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6801", 2024-09-18T07:35:14.549 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:35:14.549 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.549 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.549 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:35:14.549 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.550 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.550 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6802", 2024-09-18T07:35:14.550 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:35:14.550 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.550 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.550 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.550 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:35:14.550 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.550 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.550 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6804", 2024-09-18T07:35:14.550 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:35:14.551 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.551 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.551 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:35:14.551 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.551 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.551 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6803", 2024-09-18T07:35:14.551 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:35:14.551 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.551 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.551 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6801/2811444477", 2024-09-18T07:35:14.552 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6802/2811444477", 2024-09-18T07:35:14.552 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6804/2811444477", 2024-09-18T07:35:14.552 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6803/2811444477", 2024-09-18T07:35:14.552 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:35:14.552 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:35:14.552 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:35:14.552 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.552 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:35:14.552 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575", 2024-09-18T07:35:14.552 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:35:14.553 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:35:14.553 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:35:14.553 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:35:14.553 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:35:14.553 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:35:14.553 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 13, 2024-09-18T07:35:14.553 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 172, 2024-09-18T07:35:14.553 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:35:14.553 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:35:14.553 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:35:14.553 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.554 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.554 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6805", 2024-09-18T07:35:14.554 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:35:14.554 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.554 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.554 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:35:14.554 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.554 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.554 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.554 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6806", 2024-09-18T07:35:14.554 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:35:14.554 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.555 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.555 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.555 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:35:14.555 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.555 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.555 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.555 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6808", 2024-09-18T07:35:14.555 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:35:14.555 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.555 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.555 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.555 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:35:14.555 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.555 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.556 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.556 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6807", 2024-09-18T07:35:14.556 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:35:14.556 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.556 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.556 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.556 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6805/874102487", 2024-09-18T07:35:14.556 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6806/874102487", 2024-09-18T07:35:14.556 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6808/874102487", 2024-09-18T07:35:14.556 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6807/874102487", 2024-09-18T07:35:14.556 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:35:14.556 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:35:14.556 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:35:14.557 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.557 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.557 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.557 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:35:14.557 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "480ad6c5-0e0b-4405-bbe2-381682d00bae", 2024-09-18T07:35:14.557 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:35:14.557 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:35:14.557 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:35:14.557 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:35:14.557 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:35:14.557 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:35:14.557 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 18, 2024-09-18T07:35:14.557 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 181, 2024-09-18T07:35:14.558 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:35:14.558 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:35:14.558 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:35:14.558 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.558 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.558 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.558 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6809", 2024-09-18T07:35:14.558 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:35:14.558 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.558 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.558 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.558 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:35:14.558 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.559 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.559 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6810", 2024-09-18T07:35:14.559 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:35:14.559 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.559 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.559 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:35:14.559 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.559 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.559 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6812", 2024-09-18T07:35:14.559 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:35:14.560 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.560 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.560 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:35:14.560 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.560 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.560 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.560 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6811", 2024-09-18T07:35:14.560 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:35:14.560 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.560 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.560 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6809/1754684686", 2024-09-18T07:35:14.561 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6810/1754684686", 2024-09-18T07:35:14.561 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6812/1754684686", 2024-09-18T07:35:14.561 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6811/1754684686", 2024-09-18T07:35:14.561 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:35:14.561 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:35:14.561 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:35:14.561 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.561 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.561 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:35:14.561 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14", 2024-09-18T07:35:14.561 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:35:14.561 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:35:14.562 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:35:14.562 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:35:14.562 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:35:14.562 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:35:14.562 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 26, 2024-09-18T07:35:14.562 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 176, 2024-09-18T07:35:14.562 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:35:14.562 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:35:14.562 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:35:14.562 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.562 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.562 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.562 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6813", 2024-09-18T07:35:14.562 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:35:14.563 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.563 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.563 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.563 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:35:14.563 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.563 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.563 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.563 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6814", 2024-09-18T07:35:14.563 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:35:14.563 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.563 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.563 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.563 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:35:14.564 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.564 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.564 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.564 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6816", 2024-09-18T07:35:14.564 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:35:14.564 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.564 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.564 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.564 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:35:14.564 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.564 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.564 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.564 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6815", 2024-09-18T07:35:14.565 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:35:14.565 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.565 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.565 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6813/116058392", 2024-09-18T07:35:14.565 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6814/116058392", 2024-09-18T07:35:14.565 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6816/116058392", 2024-09-18T07:35:14.565 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6815/116058392", 2024-09-18T07:35:14.565 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:35:14.565 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:35:14.565 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:35:14.565 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:35:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de", 2024-09-18T07:35:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:35:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:35:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:35:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:35:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:35:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:35:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 32, 2024-09-18T07:35:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 180, 2024-09-18T07:35:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:35:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:35:14.566 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:35:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6800", 2024-09-18T07:35:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:35:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:35:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.567 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6801", 2024-09-18T07:35:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:35:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:35:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6803", 2024-09-18T07:35:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:35:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:35:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6802", 2024-09-18T07:35:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:35:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6800/3654297954", 2024-09-18T07:35:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6801/3654297954", 2024-09-18T07:35:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6803/3654297954", 2024-09-18T07:35:14.569 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6802/3654297954", 2024-09-18T07:35:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:35:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:35:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:35:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:35:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c", 2024-09-18T07:35:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:35:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:35:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:35:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:35:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:35:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:35:14.570 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 39, 2024-09-18T07:35:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 184, 2024-09-18T07:35:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:35:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:35:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:35:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6804", 2024-09-18T07:35:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:35:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:35:14.571 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.572 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.572 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6805", 2024-09-18T07:35:14.572 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:35:14.572 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.572 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.572 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:35:14.572 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.572 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.573 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6807", 2024-09-18T07:35:14.573 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:35:14.573 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.573 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.574 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:35:14.574 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.574 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.574 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.574 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6806", 2024-09-18T07:35:14.574 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:35:14.574 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.574 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.574 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6804/837994222", 2024-09-18T07:35:14.574 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6805/837994222", 2024-09-18T07:35:14.574 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6807/837994222", 2024-09-18T07:35:14.575 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6806/837994222", 2024-09-18T07:35:14.575 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:35:14.575 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:35:14.575 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:35:14.575 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.575 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.575 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:35:14.575 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6", 2024-09-18T07:35:14.575 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:35:14.575 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:35:14.575 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:35:14.575 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:35:14.576 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:35:14.576 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:35:14.576 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 47, 2024-09-18T07:35:14.576 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 185, 2024-09-18T07:35:14.576 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:35:14.576 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:35:14.576 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:35:14.576 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.576 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.576 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6808", 2024-09-18T07:35:14.576 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:35:14.576 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.577 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.577 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:35:14.577 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.577 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.577 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6809", 2024-09-18T07:35:14.577 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:35:14.577 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.577 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.577 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:35:14.577 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.578 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.578 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6811", 2024-09-18T07:35:14.578 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:35:14.578 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.578 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.578 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:35:14.578 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.578 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.578 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6810", 2024-09-18T07:35:14.578 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:35:14.578 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.579 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.579 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6808/1494175466", 2024-09-18T07:35:14.579 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6809/1494175466", 2024-09-18T07:35:14.579 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6811/1494175466", 2024-09-18T07:35:14.579 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6810/1494175466", 2024-09-18T07:35:14.579 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:35:14.579 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:35:14.579 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:35:14.579 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.579 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:35:14.580 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a", 2024-09-18T07:35:14.580 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:35:14.580 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:35:14.580 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:35:14.580 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:35:14.580 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:35:14.580 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:35:14.580 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 55, 2024-09-18T07:35:14.580 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 181, 2024-09-18T07:35:14.580 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:35:14.580 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:35:14.580 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:35:14.580 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.581 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.581 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6812", 2024-09-18T07:35:14.581 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:35:14.581 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.581 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.581 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:35:14.581 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.581 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.581 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6813", 2024-09-18T07:35:14.581 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:35:14.582 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.582 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.582 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:35:14.582 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.582 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.582 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6815", 2024-09-18T07:35:14.582 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:35:14.582 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.582 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.582 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:35:14.583 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:14.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.583 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:14.583 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6814", 2024-09-18T07:35:14.583 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:35:14.583 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.583 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.583 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6812/2670809315", 2024-09-18T07:35:14.583 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6813/2670809315", 2024-09-18T07:35:14.583 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6815/2670809315", 2024-09-18T07:35:14.583 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6814/2670809315", 2024-09-18T07:35:14.583 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:35:14.583 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:35:14.584 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:35:14.584 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.584 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.584 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:35:14.584 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_xinfo": [ 2024-09-18T07:35:14.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.584 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:35:14.584 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:35:14.584 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:35:14.584 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:35:14.584 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:35:14.584 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:35:14.584 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:34:43.271064+0000", 2024-09-18T07:35:14.585 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:35:14.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.585 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:35:14.585 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:35:14.585 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:35:14.585 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:35:14.585 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:35:14.585 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:35:14.585 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:34:11.710796+0000", 2024-09-18T07:35:14.585 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:35:14.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.585 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:35:14.586 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:35:14.586 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:35:14.586 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:35:14.586 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:35:14.586 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:35:14.586 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:34:45.346036+0000", 2024-09-18T07:35:14.586 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:35:14.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.586 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:35:14.586 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:35:14.586 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:35:14.586 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:35:14.587 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:35:14.587 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:35:14.587 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:34:01.405459+0000", 2024-09-18T07:35:14.587 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:35:14.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.587 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.587 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:35:14.587 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:35:14.587 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:35:14.587 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:35:14.587 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:35:14.587 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:35:14.587 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:34:24.079197+0000", 2024-09-18T07:35:14.587 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:35:14.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.588 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:35:14.588 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:35:14.588 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:35:14.588 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:35:14.588 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:35:14.588 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:35:14.588 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:34:42.334024+0000", 2024-09-18T07:35:14.588 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:35:14.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.588 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:35:14.588 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:35:14.589 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:35:14.589 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:35:14.589 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:35:14.589 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:35:14.589 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:35:09.257096+0000", 2024-09-18T07:35:14.589 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:35:14.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.589 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:35:14.589 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:35:14.589 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:35:14.589 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:35:14.589 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:35:14.590 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:35:14.590 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:34:03.952450+0000", 2024-09-18T07:35:14.590 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:35:14.590 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.590 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:35:14.590 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap": [], 2024-09-18T07:35:14.590 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_items": [ 2024-09-18T07:35:14.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.590 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.6", 2024-09-18T07:35:14.590 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:35:14.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.590 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:35:14.590 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:35:14.591 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.591 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.591 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.e", 2024-09-18T07:35:14.591 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:35:14.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.591 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:35:14.591 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:35:14.591 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.591 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.591 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.10", 2024-09-18T07:35:14.592 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:35:14.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.592 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:35:14.592 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:35:14.592 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.592 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.592 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.12", 2024-09-18T07:35:14.592 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:35:14.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.592 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:35:14.592 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:35:14.593 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.593 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.593 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.18", 2024-09-18T07:35:14.593 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:35:14.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.593 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:35:14.593 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:35:14.593 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.594 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.594 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.2", 2024-09-18T07:35:14.594 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:35:14.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.594 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 6, 2024-09-18T07:35:14.594 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:35:14.594 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.594 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.594 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.b", 2024-09-18T07:35:14.595 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:35:14.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.595 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 0, 2024-09-18T07:35:14.595 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 3 2024-09-18T07:35:14.595 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.595 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.595 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.1b", 2024-09-18T07:35:14.595 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:35:14.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.595 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:35:14.595 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:35:14.596 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.596 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.596 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.596 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:35:14.596 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_primaries": [], 2024-09-18T07:35:14.596 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_temp": [ 2024-09-18T07:35:14.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:14.596 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.f", 2024-09-18T07:35:14.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-18T07:35:14.596 INFO:teuthology.orchestra.run.smithi137.stdout: 0, 2024-09-18T07:35:14.596 INFO:teuthology.orchestra.run.smithi137.stdout: 3, 2024-09-18T07:35:14.596 INFO:teuthology.orchestra.run.smithi137.stdout: 4 2024-09-18T07:35:14.596 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:14.597 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.597 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:35:14.597 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_temp": [], 2024-09-18T07:35:14.597 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklist": { 2024-09-18T07:35:14.597 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/2096039404": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:35:14.597 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/3636177939": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:35:14.597 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/334967443": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:35:14.597 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2101286111": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:35:14.597 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3387637412": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:35:14.597 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762989503": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:35:14.597 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3490817671": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:35:14.597 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1482147392": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:35:14.597 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1821482181": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:35:14.597 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/4028304023": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:35:14.598 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/621434521": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:35:14.598 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3160704935": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:35:14.598 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2357293499": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:35:14.598 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762352860": "2024-09-19T07:31:25.389124+0000" 2024-09-18T07:35:14.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.598 INFO:teuthology.orchestra.run.smithi137.stdout: "range_blocklist": {}, 2024-09-18T07:35:14.598 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profiles": { 2024-09-18T07:35:14.598 INFO:teuthology.orchestra.run.smithi137.stdout: "default": { 2024-09-18T07:35:14.598 INFO:teuthology.orchestra.run.smithi137.stdout: "crush-failure-domain": "osd", 2024-09-18T07:35:14.598 INFO:teuthology.orchestra.run.smithi137.stdout: "k": "2", 2024-09-18T07:35:14.598 INFO:teuthology.orchestra.run.smithi137.stdout: "m": "1", 2024-09-18T07:35:14.598 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin": "jerasure", 2024-09-18T07:35:14.598 INFO:teuthology.orchestra.run.smithi137.stdout: "technique": "reed_sol_van" 2024-09-18T07:35:14.598 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:14.599 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps_queue": [], 2024-09-18T07:35:14.599 INFO:teuthology.orchestra.run.smithi137.stdout: "new_removed_snaps": [], 2024-09-18T07:35:14.599 INFO:teuthology.orchestra.run.smithi137.stdout: "new_purged_snaps": [], 2024-09-18T07:35:14.599 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_node_flags": {}, 2024-09-18T07:35:14.599 INFO:teuthology.orchestra.run.smithi137.stdout: "device_class_flags": {}, 2024-09-18T07:35:14.599 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode": { 2024-09-18T07:35:14.599 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_enabled": false, 2024-09-18T07:35:14.599 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_bucket_count": 0, 2024-09-18T07:35:14.599 INFO:teuthology.orchestra.run.smithi137.stdout: "degraded_stretch_mode": 0, 2024-09-18T07:35:14.599 INFO:teuthology.orchestra.run.smithi137.stdout: "recovering_stretch_mode": 0, 2024-09-18T07:35:14.599 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_bucket": 0 2024-09-18T07:35:14.599 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:14.600 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:35:14.606 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:14.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:14.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:14.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:14.738 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:14.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:14.813 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:14 smithi203 ceph-mon[32237]: osdmap e185: 8 total, 8 up, 8 in 2024-09-18T07:35:14.813 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:14 smithi203 ceph-mon[32237]: pgmap v229: 148 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 145 active+clean; 2.6 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 114 op/s; 3891/125967 objects misplaced (3.089%); 81 MiB/s, 1.30k objects/s recovering 2024-09-18T07:35:14.814 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:14 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1204825696' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:14.814 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:14 smithi203 ceph-mon[32237]: from='client.35039 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:14.840 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:14.875 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:14.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:14.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:14.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:14 smithi137 ceph-mon[29408]: osdmap e185: 8 total, 8 up, 8 in 2024-09-18T07:35:14.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:14 smithi137 ceph-mon[29408]: pgmap v229: 148 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 145 active+clean; 2.6 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 114 op/s; 3891/125967 objects misplaced (3.089%); 81 MiB/s, 1.30k objects/s recovering 2024-09-18T07:35:14.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:14 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1204825696' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:14.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:14 smithi137 ceph-mon[29408]: from='client.35039 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:14.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:14 smithi137 ceph-mon[39175]: osdmap e185: 8 total, 8 up, 8 in 2024-09-18T07:35:14.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:14 smithi137 ceph-mon[39175]: pgmap v229: 148 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 145 active+clean; 2.6 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 114 op/s; 3891/125967 objects misplaced (3.089%); 81 MiB/s, 1.30k objects/s recovering 2024-09-18T07:35:14.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:14 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1204825696' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:14.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:14 smithi137 ceph-mon[39175]: from='client.35039 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:14.957 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:35:14.957 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:35:14.977 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:15.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:15.009 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:15.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:15.110 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:15.139 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:15.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:15.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:15.172 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:35:15.172 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:35:15.211 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:35:15.211 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:35:15.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:15.250 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:35:15.250 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:35:15.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:15.276 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:15.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:15.288 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:35:15.288 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:35:15.320 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:35:15.320 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:35:15.353 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:35:15.353 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:35:15.377 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:15.386 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:35:15.386 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:35:15.404 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:15.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:15.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:15.418 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:35:15.418 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:35:15.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:15.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:15.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:15.647 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:15.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:15 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:15.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:15 smithi137 ceph-mon[29408]: from='client.35039 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:35:15.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:15 smithi137 ceph-mon[29408]: osdmap e186: 8 total, 8 up, 8 in 2024-09-18T07:35:15.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:15 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/909581309' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:35:15.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:15 smithi137 ceph-mon[29408]: osdmap e187: 8 total, 8 up, 8 in 2024-09-18T07:35:15.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:15 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1111435941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:15.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:15 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:15.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:15 smithi137 ceph-mon[39175]: from='client.35039 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:35:15.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:15 smithi137 ceph-mon[39175]: osdmap e186: 8 total, 8 up, 8 in 2024-09-18T07:35:15.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:15 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/909581309' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:35:15.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:15 smithi137 ceph-mon[39175]: osdmap e187: 8 total, 8 up, 8 in 2024-09-18T07:35:15.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:15 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1111435941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:15.675 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:15.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:15.687 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:15 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:15.687 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:15 smithi203 ceph-mon[32237]: from='client.35039 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:35:15.687 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:15 smithi203 ceph-mon[32237]: osdmap e186: 8 total, 8 up, 8 in 2024-09-18T07:35:15.687 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:15 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/909581309' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:35:15.687 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:15 smithi203 ceph-mon[32237]: osdmap e187: 8 total, 8 up, 8 in 2024-09-18T07:35:15.688 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:15 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1111435941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:15.731 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T07:35:15.777 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:15.826 INFO:tasks.thrashosds.thrasher:j is {'epoch': 186, 'fsid': 'f7ef099c-758e-11ef-bceb-c7b262605968', 'created': '2024-09-18T07:25:19.662367+0000', 'modified': '2024-09-18T07:35:14.463373+0000', 'last_up_change': '2024-09-18T07:30:44.708496+0000', 'last_in_change': '2024-09-18T07:35:00.369558+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T07:28:20.899655+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-18T07:30:53.715373+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-18T07:30:54.672127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-18T07:30:56.351343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-18T07:30:58.322401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-18T07:31:00.390499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-18T07:32:06.616722+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '7.10', 'ready_epoch': 184, 'last_epoch_started': 183, 'last_epoch_clean': 183, 'source_version': "181'9339", 'target_version': "181'9284"}, 'last_change': '184', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '184', 'last_force_op_resend_preluminous': '182', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '268cd580-c931-4992-bb93-d6f63c2bfe88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 181, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6801', 'nonce': 2811444477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6802', 'nonce': 2811444477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6804', 'nonce': 2811444477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6803', 'nonce': 2811444477}]}, 'public_addr': '172.21.15.137:6801/2811444477', 'cluster_addr': '172.21.15.137:6802/2811444477', 'heartbeat_back_addr': '172.21.15.137:6804/2811444477', 'heartbeat_front_addr': '172.21.15.137:6803/2811444477', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b06907bc-b499-46c0-8667-c5bdc0d85575', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6805', 'nonce': 874102487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6806', 'nonce': 874102487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6808', 'nonce': 874102487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6807', 'nonce': 874102487}]}, 'public_addr': '172.21.15.137:6805/874102487', 'cluster_addr': '172.21.15.137:6806/874102487', 'heartbeat_back_addr': '172.21.15.137:6808/874102487', 'heartbeat_front_addr': '172.21.15.137:6807/874102487', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '480ad6c5-0e0b-4405-bbe2-381682d00bae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 181, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6809', 'nonce': 1754684686}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6810', 'nonce': 1754684686}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6812', 'nonce': 1754684686}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6811', 'nonce': 1754684686}]}, 'public_addr': '172.21.15.137:6809/1754684686', 'cluster_addr': '172.21.15.137:6810/1754684686', 'heartbeat_back_addr': '172.21.15.137:6812/1754684686', 'heartbeat_front_addr': '172.21.15.137:6811/1754684686', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '178be3e0-f07b-4856-807b-2a1181e48a14', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 176, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6813', 'nonce': 116058392}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6814', 'nonce': 116058392}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6816', 'nonce': 116058392}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6815', 'nonce': 116058392}]}, 'public_addr': '172.21.15.137:6813/116058392', 'cluster_addr': '172.21.15.137:6814/116058392', 'heartbeat_back_addr': '172.21.15.137:6816/116058392', 'heartbeat_front_addr': '172.21.15.137:6815/116058392', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7547102c-951a-4077-a094-fb8a65bfe6de', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 180, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6800', 'nonce': 3654297954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6801', 'nonce': 3654297954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6803', 'nonce': 3654297954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6802', 'nonce': 3654297954}]}, 'public_addr': '172.21.15.203:6800/3654297954', 'cluster_addr': '172.21.15.203:6801/3654297954', 'heartbeat_back_addr': '172.21.15.203:6803/3654297954', 'heartbeat_front_addr': '172.21.15.203:6802/3654297954', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '49c17901-8102-48a4-9348-fa1bdb63032c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 184, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6804', 'nonce': 837994222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6805', 'nonce': 837994222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6807', 'nonce': 837994222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6806', 'nonce': 837994222}]}, 'public_addr': '172.21.15.203:6804/837994222', 'cluster_addr': '172.21.15.203:6805/837994222', 'heartbeat_back_addr': '172.21.15.203:6807/837994222', 'heartbeat_front_addr': '172.21.15.203:6806/837994222', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '75f0ead4-e981-4a15-b1c1-d83dce871ab6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 185, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6808', 'nonce': 1494175466}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6809', 'nonce': 1494175466}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6811', 'nonce': 1494175466}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6810', 'nonce': 1494175466}]}, 'public_addr': '172.21.15.203:6808/1494175466', 'cluster_addr': '172.21.15.203:6809/1494175466', 'heartbeat_back_addr': '172.21.15.203:6811/1494175466', 'heartbeat_front_addr': '172.21.15.203:6810/1494175466', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c0dbf299-6798-4b99-9a85-62b123971f2a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 55, 'up_thru': 181, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6812', 'nonce': 2670809315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6813', 'nonce': 2670809315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6815', 'nonce': 2670809315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6814', 'nonce': 2670809315}]}, 'public_addr': '172.21.15.203:6812/2670809315', 'cluster_addr': '172.21.15.203:6813/2670809315', 'heartbeat_back_addr': '172.21.15.203:6815/2670809315', 'heartbeat_front_addr': '172.21.15.203:6814/2670809315', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:34:43.271064+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:34:11.710796+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:34:45.346036+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:34:01.405459+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:34:24.079197+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:34:42.334024+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:35:09.257096+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:34:03.952450+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.2', 'mappings': [{'from': 6, 'to': 5}]}, {'pgid': '5.b', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '6.1b', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '7.f', 'osds': [0, 3, 4]}], 'primary_temp': [], 'blocklist': {'172.21.15.137:6800/2096039404': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:6800/3636177939': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/334967443': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/2101286111': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/3387637412': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1762989503': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3490817671': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:6800/1482147392': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1821482181': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/4028304023': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/621434521': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3160704935': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/2357293499': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/1762352860': '2024-09-19T07:31:25.389124+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T07:35:15.826 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:35:15.843 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:15.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:15.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:15.945 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:15.977 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:15.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:15.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:16.079 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:16.110 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:16.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:16.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:16.126 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:35:16.212 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:16.243 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:16.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:16.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:16.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:16.385 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:16.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:16.486 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:16.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:16.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:16.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:16.613 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:16 smithi203 ceph-mon[32237]: pgmap v232: 148 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 145 active+clean; 2.6 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 3891/125967 objects misplaced (3.089%); 38 MiB/s, 606 objects/s recovering 2024-09-18T07:35:16.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:16.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:16.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:16.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:16.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:16.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:16.815 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:16.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:16.887 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:35:16.888 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:35:16.888 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:35:16.888 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:35:16.888 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:35:16.915 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:16.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:16 smithi137 ceph-mon[29408]: pgmap v232: 148 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 145 active+clean; 2.6 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 3891/125967 objects misplaced (3.089%); 38 MiB/s, 606 objects/s recovering 2024-09-18T07:35:16.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:16 smithi137 ceph-mon[39175]: pgmap v232: 148 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 145 active+clean; 2.6 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 3891/125967 objects misplaced (3.089%); 38 MiB/s, 606 objects/s recovering 2024-09-18T07:35:16.946 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:16.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:16.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:17.049 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:17.081 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:17.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:17.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:17.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:17.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:35:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:35:17.219 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:17.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:17.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:17.256 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:35:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:35:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:35:17.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:17.344 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:17.348 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:17.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:17.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:17.450 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:17.479 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:17.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:17.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:17.513 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:17 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:35:17.513 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:17 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:35:17.513 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:17 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:35:17.513 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:17 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:35:17.581 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:17.610 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:17.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:17.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:17.676 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T07:35:17.712 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:17.751 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:17.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:17.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:17.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:17.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:17.893 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:17.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:17.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:17 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:35:17.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:17 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:35:17.995 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:18.027 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:35:18.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:18.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:18.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:18.137 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:18.164 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:18.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:18.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:18.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:18.295 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:35:18.304 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:35:18.304 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:18.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:18.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:18.406 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:18.435 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:18.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:18.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:18.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:18.544 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:35:18.546 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:18 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:18.547 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:18 smithi137 ceph-mon[29408]: pgmap v233: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1018 B/s rd, 0 B/s wr, 969 op/s; 2466/117270 objects misplaced (2.103%); 14 MiB/s, 222 objects/s recovering 2024-09-18T07:35:18.547 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:18 smithi137 ceph-mon[29408]: osdmap e188: 8 total, 8 up, 8 in 2024-09-18T07:35:18.547 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:18 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:18.547 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:18 smithi137 ceph-mon[39175]: pgmap v233: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1018 B/s rd, 0 B/s wr, 969 op/s; 2466/117270 objects misplaced (2.103%); 14 MiB/s, 222 objects/s recovering 2024-09-18T07:35:18.547 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:18 smithi137 ceph-mon[39175]: osdmap e188: 8 total, 8 up, 8 in 2024-09-18T07:35:18.564 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:18.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:18.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:18.599 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:35:18.599 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:35:18.599 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:35:18.599 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:35:18.599 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.599 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:20119 7.5 7:a056c01b:::benchmark_data_smithi137_74180_object20118:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T07:35:18.599 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.214505+0000", 2024-09-18T07:35:18.599 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 175.37874582800001, 2024-09-18T07:35:18.600 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.046754022999999999, 2024-09-18T07:35:18.600 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.600 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.600 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.600 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:18.600 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:18.600 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:18.600 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 20119 2024-09-18T07:35:18.600 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.600 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.600 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.600 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.600 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214505+0000", 2024-09-18T07:35:18.601 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.601 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.601 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.601 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.601 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214505+0000", 2024-09-18T07:35:18.601 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.601 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.601 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.601 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.602 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214507+0000", 2024-09-18T07:35:18.602 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0559999999999999e-06 2024-09-18T07:35:18.602 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.602 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.602 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.602 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214562+0000", 2024-09-18T07:35:18.602 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4873999999999997e-05 2024-09-18T07:35:18.602 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.602 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.602 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.603 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214571+0000", 2024-09-18T07:35:18.603 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9639999999999992e-06 2024-09-18T07:35:18.603 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.603 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.603 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.603 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214577+0000", 2024-09-18T07:35:18.603 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1279999999999996e-06 2024-09-18T07:35:18.603 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.603 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.603 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.603 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214630+0000", 2024-09-18T07:35:18.603 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3260999999999997e-05 2024-09-18T07:35:18.604 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.604 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.604 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.604 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214750+0000", 2024-09-18T07:35:18.604 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011973599999999999 2024-09-18T07:35:18.604 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.604 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.604 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:35:18.604 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.214880+0000", 2024-09-18T07:35:18.604 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000129634 2024-09-18T07:35:18.604 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.604 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.604 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:18.604 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.215650+0000", 2024-09-18T07:35:18.605 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00077066999999999999 2024-09-18T07:35:18.605 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.605 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.605 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.605 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.216265+0000", 2024-09-18T07:35:18.605 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00061457799999999996 2024-09-18T07:35:18.605 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.605 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.605 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.605 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.261179+0000", 2024-09-18T07:35:18.605 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.044914557000000001 2024-09-18T07:35:18.605 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.606 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.606 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.606 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.261218+0000", 2024-09-18T07:35:18.606 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8603000000000003e-05 2024-09-18T07:35:18.606 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.606 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.606 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.606 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.261259+0000", 2024-09-18T07:35:18.606 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0961999999999998e-05 2024-09-18T07:35:18.606 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.606 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.606 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.607 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.607 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.607 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:21294 7.8 e89/81 7:15222838:::benchmark_data_smithi137_74180_object21293:head v 89'1394, mlcod=89'1392)", 2024-09-18T07:35:18.607 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.948825+0000", 2024-09-18T07:35:18.607 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.64442585399999, 2024-09-18T07:35:18.607 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.094945371000000001, 2024-09-18T07:35:18.607 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.607 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.607 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.607 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.607 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.607 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.608 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948825+0000", 2024-09-18T07:35:18.608 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.608 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.608 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.608 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.608 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948825+0000", 2024-09-18T07:35:18.608 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.609 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.609 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.609 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.609 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948826+0000", 2024-09-18T07:35:18.609 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3e-07 2024-09-18T07:35:18.609 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.609 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.609 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.609 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948939+0000", 2024-09-18T07:35:18.609 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000113757 2024-09-18T07:35:18.609 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.609 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.609 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.610 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948994+0000", 2024-09-18T07:35:18.610 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4622e-05 2024-09-18T07:35:18.610 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.610 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.610 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.610 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.949000+0000", 2024-09-18T07:35:18.610 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9599999999999997e-06 2024-09-18T07:35:18.610 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.610 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.610 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.610 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.949042+0000", 2024-09-18T07:35:18.610 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2558999999999997e-05 2024-09-18T07:35:18.610 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.611 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.611 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.611 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.949055+0000", 2024-09-18T07:35:18.611 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2969999999999999e-05 2024-09-18T07:35:18.611 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.611 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.611 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.611 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.043723+0000", 2024-09-18T07:35:18.611 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.094668034999999998 2024-09-18T07:35:18.611 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.611 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.611 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.611 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.043770+0000", 2024-09-18T07:35:18.611 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6838000000000001e-05 2024-09-18T07:35:18.612 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.612 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.612 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.612 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.612 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.612 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:21293 7.f e89/81 7:fa840974:::benchmark_data_smithi137_74180_object21292:head v 89'1337, mlcod=89'1334)", 2024-09-18T07:35:18.612 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.948830+0000", 2024-09-18T07:35:18.613 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.64442083500001, 2024-09-18T07:35:18.613 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.096476913999999997, 2024-09-18T07:35:18.613 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.613 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.613 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.613 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.613 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.613 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.613 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948830+0000", 2024-09-18T07:35:18.613 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.613 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.613 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.613 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.614 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948830+0000", 2024-09-18T07:35:18.614 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.614 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.614 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.614 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.614 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948831+0000", 2024-09-18T07:35:18.614 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.235e-06 2024-09-18T07:35:18.614 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.614 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.614 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.614 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948942+0000", 2024-09-18T07:35:18.615 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000110417 2024-09-18T07:35:18.615 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.615 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.615 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.615 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948954+0000", 2024-09-18T07:35:18.615 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1888999999999999e-05 2024-09-18T07:35:18.615 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.615 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.616 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.616 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948959+0000", 2024-09-18T07:35:18.616 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8259999999999998e-06 2024-09-18T07:35:18.616 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.616 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.616 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.617 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948998+0000", 2024-09-18T07:35:18.617 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8498999999999999e-05 2024-09-18T07:35:18.617 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.617 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.617 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.617 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.949009+0000", 2024-09-18T07:35:18.618 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1413999999999999e-05 2024-09-18T07:35:18.618 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.618 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.618 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.618 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.045250+0000", 2024-09-18T07:35:18.618 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.096240945999999994 2024-09-18T07:35:18.619 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.619 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.619 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.619 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.045307+0000", 2024-09-18T07:35:18.619 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6687999999999997e-05 2024-09-18T07:35:18.619 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.620 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.620 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.620 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.620 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.620 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21896 7.5 7:a849d1d6:::benchmark_data_smithi137_74180_object21895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:35:18.620 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.316136+0000", 2024-09-18T07:35:18.621 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.27711469299999, 2024-09-18T07:35:18.621 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.073939160000000004, 2024-09-18T07:35:18.621 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.621 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.621 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.621 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:18.622 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:18.622 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:18.622 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21896 2024-09-18T07:35:18.622 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.622 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.623 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.623 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.623 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316136+0000", 2024-09-18T07:35:18.623 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.623 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.623 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.623 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.623 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316136+0000", 2024-09-18T07:35:18.623 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.624 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.624 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.624 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.624 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316138+0000", 2024-09-18T07:35:18.624 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.426e-06 2024-09-18T07:35:18.624 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.624 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.624 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.624 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316190+0000", 2024-09-18T07:35:18.625 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2704000000000002e-05 2024-09-18T07:35:18.625 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.625 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.625 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.625 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316201+0000", 2024-09-18T07:35:18.625 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0298e-05 2024-09-18T07:35:18.625 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.625 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.626 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.626 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316205+0000", 2024-09-18T07:35:18.626 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8860000000000003e-06 2024-09-18T07:35:18.626 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.626 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.626 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.626 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316244+0000", 2024-09-18T07:35:18.626 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8535999999999998e-05 2024-09-18T07:35:18.627 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.627 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.627 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.627 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316328+0000", 2024-09-18T07:35:18.627 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4384999999999999e-05 2024-09-18T07:35:18.627 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.627 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.627 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:35:18.628 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316417+0000", 2024-09-18T07:35:18.628 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8794000000000004e-05 2024-09-18T07:35:18.628 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.628 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.628 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.629 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.388000+0000", 2024-09-18T07:35:18.629 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.071582792000000006 2024-09-18T07:35:18.629 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.629 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.629 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:18.629 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.388272+0000", 2024-09-18T07:35:18.629 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00027234600000000002 2024-09-18T07:35:18.630 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.630 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.630 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.630 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.389970+0000", 2024-09-18T07:35:18.630 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016974959999999999 2024-09-18T07:35:18.630 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.630 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.630 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.631 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.390033+0000", 2024-09-18T07:35:18.631 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3640999999999996e-05 2024-09-18T07:35:18.631 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.631 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.631 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.631 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.390075+0000", 2024-09-18T07:35:18.631 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1856e-05 2024-09-18T07:35:18.631 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.631 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.632 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.632 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.632 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.632 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop_reply(client.14865.0:21896 7.5 e89/81 ondisk, result = 0)", 2024-09-18T07:35:18.632 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.317455+0000", 2024-09-18T07:35:18.632 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.27579630299999, 2024-09-18T07:35:18.632 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.070552445000000005, 2024-09-18T07:35:18.632 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.633 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.633 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:35:18.633 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.633 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.633 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.633 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.317455+0000", 2024-09-18T07:35:18.633 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.633 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.634 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.634 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.634 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.317455+0000", 2024-09-18T07:35:18.634 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.634 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.634 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.634 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.634 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.317455+0000", 2024-09-18T07:35:18.635 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4399999999999998e-07 2024-09-18T07:35:18.635 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.635 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.635 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.635 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.317459+0000", 2024-09-18T07:35:18.635 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7749999999999999e-06 2024-09-18T07:35:18.635 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.635 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.635 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.636 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.386905+0000", 2024-09-18T07:35:18.636 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.069445328000000001 2024-09-18T07:35:18.636 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.636 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.636 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.636 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.386916+0000", 2024-09-18T07:35:18.636 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1331e-05 2024-09-18T07:35:18.637 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.637 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.637 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.637 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.387987+0000", 2024-09-18T07:35:18.637 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010713020000000001 2024-09-18T07:35:18.637 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.637 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.637 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.637 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.387995+0000", 2024-09-18T07:35:18.638 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3679999999999996e-06 2024-09-18T07:35:18.638 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.638 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.638 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.638 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.388007+0000", 2024-09-18T07:35:18.638 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2397e-05 2024-09-18T07:35:18.638 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.638 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.639 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.639 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.639 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.639 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21999 7.5 7:a6ca4f28:::benchmark_data_smithi137_74180_object21998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:35:18.639 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.424624+0000", 2024-09-18T07:35:18.639 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.168626811, 2024-09-18T07:35:18.639 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.112047436, 2024-09-18T07:35:18.639 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.639 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.640 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.640 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:18.640 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:18.640 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:18.640 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21999 2024-09-18T07:35:18.640 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.640 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.640 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.641 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.641 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424624+0000", 2024-09-18T07:35:18.641 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.641 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.641 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.641 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.641 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424624+0000", 2024-09-18T07:35:18.641 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.641 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.642 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.643 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.644 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424626+0000", 2024-09-18T07:35:18.644 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.075e-06 2024-09-18T07:35:18.644 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.644 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.644 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.644 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424682+0000", 2024-09-18T07:35:18.645 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5597999999999999e-05 2024-09-18T07:35:18.645 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.645 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.645 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.645 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424694+0000", 2024-09-18T07:35:18.645 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2656e-05 2024-09-18T07:35:18.645 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.645 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.645 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.645 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424702+0000", 2024-09-18T07:35:18.645 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3849999999999996e-06 2024-09-18T07:35:18.646 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.646 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.646 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.646 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424758+0000", 2024-09-18T07:35:18.646 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6020000000000002e-05 2024-09-18T07:35:18.646 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.646 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.646 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.646 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424884+0000", 2024-09-18T07:35:18.646 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012638800000000001 2024-09-18T07:35:18.646 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.646 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.646 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:35:18.647 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425020+0000", 2024-09-18T07:35:18.647 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013587300000000001 2024-09-18T07:35:18.647 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.647 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.647 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.647 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.426628+0000", 2024-09-18T07:35:18.647 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001607792 2024-09-18T07:35:18.647 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.647 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.647 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:18.647 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.426665+0000", 2024-09-18T07:35:18.647 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7052999999999999e-05 2024-09-18T07:35:18.647 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.647 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.648 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.648 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536540+0000", 2024-09-18T07:35:18.648 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10987469399999999 2024-09-18T07:35:18.648 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.648 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.648 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.648 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536614+0000", 2024-09-18T07:35:18.648 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4486999999999993e-05 2024-09-18T07:35:18.648 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.648 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.648 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.648 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536671+0000", 2024-09-18T07:35:18.648 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7414999999999998e-05 2024-09-18T07:35:18.649 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.649 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.649 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.649 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.649 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.649 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22002 7.5 7:a1dbf95a:::benchmark_data_smithi137_74180_object22001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:35:18.649 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.425607+0000", 2024-09-18T07:35:18.649 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.16764416800001, 2024-09-18T07:35:18.649 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.111294137, 2024-09-18T07:35:18.649 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.649 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.649 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.649 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:18.650 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:18.650 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:18.650 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22002 2024-09-18T07:35:18.650 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.650 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.650 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.650 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.650 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425607+0000", 2024-09-18T07:35:18.650 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.650 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.650 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.650 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.650 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425607+0000", 2024-09-18T07:35:18.650 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.651 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.651 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.651 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.651 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425608+0000", 2024-09-18T07:35:18.651 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.297e-06 2024-09-18T07:35:18.651 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.651 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.651 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.651 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425682+0000", 2024-09-18T07:35:18.651 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3891000000000001e-05 2024-09-18T07:35:18.651 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.651 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.651 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.652 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425693+0000", 2024-09-18T07:35:18.652 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0944000000000001e-05 2024-09-18T07:35:18.652 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.652 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.652 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.652 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425701+0000", 2024-09-18T07:35:18.652 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7489999999999998e-06 2024-09-18T07:35:18.652 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.652 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.652 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.652 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425809+0000", 2024-09-18T07:35:18.652 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108323 2024-09-18T07:35:18.652 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.653 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.653 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.653 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425883+0000", 2024-09-18T07:35:18.653 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4321999999999998e-05 2024-09-18T07:35:18.653 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.653 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.653 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:35:18.653 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.426014+0000", 2024-09-18T07:35:18.653 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013092400000000001 2024-09-18T07:35:18.653 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.653 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.653 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:18.653 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.426735+0000", 2024-09-18T07:35:18.654 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00072051000000000005 2024-09-18T07:35:18.654 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.654 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.654 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.654 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.427445+0000", 2024-09-18T07:35:18.654 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00070996799999999995 2024-09-18T07:35:18.654 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.654 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.654 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.654 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536803+0000", 2024-09-18T07:35:18.654 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10935832500000001 2024-09-18T07:35:18.654 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.654 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.654 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.655 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536856+0000", 2024-09-18T07:35:18.655 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3161000000000001e-05 2024-09-18T07:35:18.655 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.655 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.655 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.655 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536901+0000", 2024-09-18T07:35:18.655 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4722999999999999e-05 2024-09-18T07:35:18.655 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.655 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.655 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.655 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.655 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.655 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22010 7.5 7:a615b038:::benchmark_data_smithi137_74180_object22009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:35:18.656 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.428662+0000", 2024-09-18T07:35:18.656 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.16458853399999, 2024-09-18T07:35:18.656 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.11444004000000001, 2024-09-18T07:35:18.656 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.656 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.656 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.656 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:18.656 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:18.656 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:18.656 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22010 2024-09-18T07:35:18.656 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.656 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.656 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.657 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.657 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428662+0000", 2024-09-18T07:35:18.657 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.657 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.657 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.657 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428662+0000", 2024-09-18T07:35:18.657 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.657 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.657 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.657 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428664+0000", 2024-09-18T07:35:18.657 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.302e-06 2024-09-18T07:35:18.658 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.658 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.658 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.658 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428707+0000", 2024-09-18T07:35:18.658 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3692000000000001e-05 2024-09-18T07:35:18.658 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.658 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.658 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.658 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428718+0000", 2024-09-18T07:35:18.658 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1066999999999999e-05 2024-09-18T07:35:18.658 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.658 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.658 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.659 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428724+0000", 2024-09-18T07:35:18.659 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7590000000000003e-06 2024-09-18T07:35:18.659 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.659 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.659 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.659 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428771+0000", 2024-09-18T07:35:18.659 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6462999999999999e-05 2024-09-18T07:35:18.659 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.659 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.659 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.659 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428852+0000", 2024-09-18T07:35:18.659 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1533999999999996e-05 2024-09-18T07:35:18.659 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.660 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.660 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:35:18.660 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428937+0000", 2024-09-18T07:35:18.660 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5068999999999995e-05 2024-09-18T07:35:18.660 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.660 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.660 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:18.661 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.429584+0000", 2024-09-18T07:35:18.661 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00064701400000000001 2024-09-18T07:35:18.661 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.661 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.661 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.661 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.429931+0000", 2024-09-18T07:35:18.661 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00034729299999999998 2024-09-18T07:35:18.661 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.661 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.661 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.661 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.543034+0000", 2024-09-18T07:35:18.662 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.113102231 2024-09-18T07:35:18.662 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.662 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.662 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.662 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.543071+0000", 2024-09-18T07:35:18.662 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7333999999999999e-05 2024-09-18T07:35:18.662 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.662 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.662 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.662 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.543102+0000", 2024-09-18T07:35:18.662 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1282000000000003e-05 2024-09-18T07:35:18.662 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.662 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.663 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.663 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.663 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.663 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:22343 7.6 e89/81 7:6ec6138d:::benchmark_data_smithi137_74180_object22342:head v 89'1362, mlcod=89'1360)", 2024-09-18T07:35:18.663 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.661175+0000", 2024-09-18T07:35:18.663 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 173.93207567499999, 2024-09-18T07:35:18.663 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19171324300000001, 2024-09-18T07:35:18.663 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.663 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.663 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.663 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.663 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.663 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.663 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661175+0000", 2024-09-18T07:35:18.664 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.664 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.664 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.664 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.664 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661175+0000", 2024-09-18T07:35:18.664 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.664 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.664 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.664 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.664 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661177+0000", 2024-09-18T07:35:18.664 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.548e-06 2024-09-18T07:35:18.664 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.664 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.665 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.665 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661235+0000", 2024-09-18T07:35:18.665 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7877999999999997e-05 2024-09-18T07:35:18.665 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.665 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.665 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.665 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661244+0000", 2024-09-18T07:35:18.665 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2709999999999998e-06 2024-09-18T07:35:18.665 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.665 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.665 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.665 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661250+0000", 2024-09-18T07:35:18.665 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8619999999999999e-06 2024-09-18T07:35:18.666 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.666 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.666 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.666 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661288+0000", 2024-09-18T07:35:18.666 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8330000000000001e-05 2024-09-18T07:35:18.666 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.666 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.666 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.666 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661300+0000", 2024-09-18T07:35:18.666 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1483e-05 2024-09-18T07:35:18.666 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.666 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.667 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.667 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852855+0000", 2024-09-18T07:35:18.667 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19155582199999999 2024-09-18T07:35:18.667 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.667 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.667 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.667 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852888+0000", 2024-09-18T07:35:18.667 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3049000000000002e-05 2024-09-18T07:35:18.667 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.667 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.667 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.667 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.667 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.668 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:22347 7.f e89/81 7:fe6074eb:::benchmark_data_smithi137_74180_object22346:head v 89'1397, mlcod=89'1395)", 2024-09-18T07:35:18.668 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.662928+0000", 2024-09-18T07:35:18.668 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 173.930323182, 2024-09-18T07:35:18.668 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19092352700000001, 2024-09-18T07:35:18.668 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.668 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.668 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.668 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.668 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.668 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.668 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662928+0000", 2024-09-18T07:35:18.668 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.668 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.669 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.669 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.669 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662928+0000", 2024-09-18T07:35:18.669 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.669 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.669 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.669 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.669 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662929+0000", 2024-09-18T07:35:18.669 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2500000000000004e-07 2024-09-18T07:35:18.669 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.669 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.669 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.669 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662989+0000", 2024-09-18T07:35:18.670 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0466e-05 2024-09-18T07:35:18.670 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.670 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.670 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.670 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.851940+0000", 2024-09-18T07:35:18.670 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18895100100000001 2024-09-18T07:35:18.670 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.670 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.670 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.670 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.851968+0000", 2024-09-18T07:35:18.670 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7668e-05 2024-09-18T07:35:18.670 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.670 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.671 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.671 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852054+0000", 2024-09-18T07:35:18.671 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6370000000000001e-05 2024-09-18T07:35:18.671 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.671 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.671 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.671 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852082+0000", 2024-09-18T07:35:18.671 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7769000000000001e-05 2024-09-18T07:35:18.671 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.671 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.671 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.671 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.853810+0000", 2024-09-18T07:35:18.671 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001728168 2024-09-18T07:35:18.671 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.672 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.672 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.672 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.853851+0000", 2024-09-18T07:35:18.672 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1159999999999999e-05 2024-09-18T07:35:18.672 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.672 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.672 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.672 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.672 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.672 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22450 7.5 7:ad888e37:::benchmark_data_smithi137_74180_object22449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:35:18.672 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.977850+0000", 2024-09-18T07:35:18.672 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 173.61540127999999, 2024-09-18T07:35:18.672 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.077184682000000004, 2024-09-18T07:35:18.673 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.673 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.673 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.673 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:18.673 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:18.673 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:18.673 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22450 2024-09-18T07:35:18.673 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.673 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.673 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.673 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.673 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977850+0000", 2024-09-18T07:35:18.673 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.674 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.674 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.674 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.674 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977850+0000", 2024-09-18T07:35:18.674 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.674 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.674 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.674 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.674 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977852+0000", 2024-09-18T07:35:18.674 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.108e-06 2024-09-18T07:35:18.674 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.674 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.674 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.675 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977892+0000", 2024-09-18T07:35:18.675 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9928999999999997e-05 2024-09-18T07:35:18.675 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.675 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.675 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.675 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977904+0000", 2024-09-18T07:35:18.675 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2843999999999999e-05 2024-09-18T07:35:18.675 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.675 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.675 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.675 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977912+0000", 2024-09-18T07:35:18.675 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1169999999999998e-06 2024-09-18T07:35:18.675 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.675 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.676 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.676 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977968+0000", 2024-09-18T07:35:18.676 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6338000000000002e-05 2024-09-18T07:35:18.676 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.676 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.676 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.676 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.978050+0000", 2024-09-18T07:35:18.676 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1841000000000005e-05 2024-09-18T07:35:18.676 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.676 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.676 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:35:18.676 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.978130+0000", 2024-09-18T07:35:18.676 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9917000000000001e-05 2024-09-18T07:35:18.677 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.677 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.677 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:18.677 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.978838+0000", 2024-09-18T07:35:18.677 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000708594 2024-09-18T07:35:18.677 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.677 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.677 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.677 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.979190+0000", 2024-09-18T07:35:18.677 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00035225000000000002 2024-09-18T07:35:18.677 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.677 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.677 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.678 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.054919+0000", 2024-09-18T07:35:18.678 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.075728917000000007 2024-09-18T07:35:18.678 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.678 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.678 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.678 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.054990+0000", 2024-09-18T07:35:18.678 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0300000000000001e-05 2024-09-18T07:35:18.678 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.678 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.678 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.678 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.055034+0000", 2024-09-18T07:35:18.678 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4527000000000002e-05 2024-09-18T07:35:18.678 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.678 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.678 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.679 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.679 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.679 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22938 7.5 7:accec075:::benchmark_data_smithi137_74180_object22937:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:35:18.679 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.270501+0000", 2024-09-18T07:35:18.679 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 173.322749878, 2024-09-18T07:35:18.679 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.048186932000000002, 2024-09-18T07:35:18.679 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.679 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.679 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.679 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:18.679 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:18.679 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:18.679 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22938 2024-09-18T07:35:18.680 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.680 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.680 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.680 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.680 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270501+0000", 2024-09-18T07:35:18.680 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.680 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.680 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.680 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.680 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270501+0000", 2024-09-18T07:35:18.680 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.680 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.680 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.680 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.681 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270503+0000", 2024-09-18T07:35:18.681 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6670000000000001e-06 2024-09-18T07:35:18.681 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.681 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.681 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.681 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270533+0000", 2024-09-18T07:35:18.681 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0889000000000002e-05 2024-09-18T07:35:18.681 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.681 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.681 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.682 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270542+0000", 2024-09-18T07:35:18.682 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4910000000000005e-06 2024-09-18T07:35:18.682 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.682 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.682 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.682 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270548+0000", 2024-09-18T07:35:18.682 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4300000000000003e-06 2024-09-18T07:35:18.682 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.683 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.683 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.683 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270701+0000", 2024-09-18T07:35:18.683 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015209800000000001 2024-09-18T07:35:18.683 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.683 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.683 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.683 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270812+0000", 2024-09-18T07:35:18.683 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000111415 2024-09-18T07:35:18.683 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.683 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.683 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:35:18.683 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.270942+0000", 2024-09-18T07:35:18.684 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000130209 2024-09-18T07:35:18.684 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.684 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.684 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:18.684 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.271936+0000", 2024-09-18T07:35:18.684 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000994256 2024-09-18T07:35:18.684 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.684 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.684 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.684 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.272727+0000", 2024-09-18T07:35:18.684 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000790674 2024-09-18T07:35:18.684 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.685 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.685 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.685 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.318626+0000", 2024-09-18T07:35:18.685 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.045899052000000003 2024-09-18T07:35:18.685 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.685 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.685 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.685 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.318659+0000", 2024-09-18T07:35:18.685 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3209e-05 2024-09-18T07:35:18.685 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.685 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.686 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.686 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.318688+0000", 2024-09-18T07:35:18.686 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8541999999999999e-05 2024-09-18T07:35:18.686 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.686 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.686 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.686 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.686 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.686 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22945 7.5 7:a76344dc:::benchmark_data_smithi137_74180_object22944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:35:18.686 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.272901+0000", 2024-09-18T07:35:18.687 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 173.320349854, 2024-09-18T07:35:18.687 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.046985232000000002, 2024-09-18T07:35:18.687 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.687 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.687 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.687 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:18.687 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:18.687 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:18.687 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22945 2024-09-18T07:35:18.687 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.687 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.687 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.688 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.688 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.272901+0000", 2024-09-18T07:35:18.688 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.688 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.688 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.688 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.688 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.272901+0000", 2024-09-18T07:35:18.688 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.688 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.688 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.688 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.688 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.272903+0000", 2024-09-18T07:35:18.689 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0609999999999998e-06 2024-09-18T07:35:18.689 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.689 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.689 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.689 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.272941+0000", 2024-09-18T07:35:18.689 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8339e-05 2024-09-18T07:35:18.689 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.689 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.689 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.689 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.272960+0000", 2024-09-18T07:35:18.689 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9096000000000001e-05 2024-09-18T07:35:18.690 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.690 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.690 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.690 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.272968+0000", 2024-09-18T07:35:18.690 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4209999999999998e-06 2024-09-18T07:35:18.690 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.690 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.690 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.690 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.273014+0000", 2024-09-18T07:35:18.690 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5840000000000002e-05 2024-09-18T07:35:18.690 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.691 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.691 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.691 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.273115+0000", 2024-09-18T07:35:18.691 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000100973 2024-09-18T07:35:18.691 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.691 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.691 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:35:18.691 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.273233+0000", 2024-09-18T07:35:18.691 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011877799999999999 2024-09-18T07:35:18.691 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.691 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.692 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:18.692 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.274254+0000", 2024-09-18T07:35:18.692 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001020267 2024-09-18T07:35:18.692 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.692 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.692 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.692 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.275840+0000", 2024-09-18T07:35:18.692 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0015865849999999999 2024-09-18T07:35:18.692 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.692 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.692 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.692 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.319797+0000", 2024-09-18T07:35:18.692 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.043956489000000001 2024-09-18T07:35:18.692 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.692 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.693 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.693 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.319846+0000", 2024-09-18T07:35:18.693 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9602999999999999e-05 2024-09-18T07:35:18.693 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.693 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.693 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.693 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.319886+0000", 2024-09-18T07:35:18.693 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9780000000000002e-05 2024-09-18T07:35:18.693 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.693 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.693 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.693 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.694 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.694 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22961 7.5 7:ae8e5e68:::benchmark_data_smithi137_74180_object22960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:35:18.694 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.278834+0000", 2024-09-18T07:35:18.694 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 173.31441690299999, 2024-09-18T07:35:18.694 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.044950704000000001, 2024-09-18T07:35:18.694 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.694 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.694 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.694 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:18.694 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:18.694 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:18.695 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22961 2024-09-18T07:35:18.695 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.695 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.695 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.695 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.695 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.278834+0000", 2024-09-18T07:35:18.695 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.695 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.695 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.695 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.696 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.278834+0000", 2024-09-18T07:35:18.696 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.696 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.696 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.696 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.696 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.278836+0000", 2024-09-18T07:35:18.696 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.73e-06 2024-09-18T07:35:18.696 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.696 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.696 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.696 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.278887+0000", 2024-09-18T07:35:18.697 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1088000000000003e-05 2024-09-18T07:35:18.697 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.697 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.697 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.697 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.278899+0000", 2024-09-18T07:35:18.697 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1780000000000001e-05 2024-09-18T07:35:18.697 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.697 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.697 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.697 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.278904+0000", 2024-09-18T07:35:18.697 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4269999999999999e-06 2024-09-18T07:35:18.697 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.697 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.697 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.698 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.278969+0000", 2024-09-18T07:35:18.698 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4970999999999998e-05 2024-09-18T07:35:18.698 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.698 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.698 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.698 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.279084+0000", 2024-09-18T07:35:18.698 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000115318 2024-09-18T07:35:18.698 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.698 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.698 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:35:18.698 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.279211+0000", 2024-09-18T07:35:18.698 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000126742 2024-09-18T07:35:18.699 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.699 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.699 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:18.699 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.280057+0000", 2024-09-18T07:35:18.699 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00084567300000000002 2024-09-18T07:35:18.699 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.699 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.699 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.699 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.280326+0000", 2024-09-18T07:35:18.699 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00026896899999999998 2024-09-18T07:35:18.699 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.699 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.699 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.699 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.323690+0000", 2024-09-18T07:35:18.700 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.043363959000000001 2024-09-18T07:35:18.700 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.700 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.700 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.700 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.323744+0000", 2024-09-18T07:35:18.700 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4058000000000003e-05 2024-09-18T07:35:18.700 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.700 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.700 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.700 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.323785+0000", 2024-09-18T07:35:18.700 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0988999999999997e-05 2024-09-18T07:35:18.700 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.700 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.701 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.701 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.701 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.701 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:23078 7.5 7:af53196f:::benchmark_data_smithi137_74180_object23077:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:35:18.701 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.454467+0000", 2024-09-18T07:35:18.701 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 173.138783929, 2024-09-18T07:35:18.701 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.088575311000000004, 2024-09-18T07:35:18.701 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.701 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.701 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.702 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:18.702 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:18.702 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:18.702 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 23078 2024-09-18T07:35:18.702 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.702 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.702 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.702 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.702 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454467+0000", 2024-09-18T07:35:18.702 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.702 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.703 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.703 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.703 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454467+0000", 2024-09-18T07:35:18.703 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.703 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.703 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.703 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.703 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454469+0000", 2024-09-18T07:35:18.703 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6089999999999999e-06 2024-09-18T07:35:18.703 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.703 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.704 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.704 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454512+0000", 2024-09-18T07:35:18.704 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2608999999999998e-05 2024-09-18T07:35:18.704 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.704 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.704 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.704 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454525+0000", 2024-09-18T07:35:18.704 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3352e-05 2024-09-18T07:35:18.704 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.704 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.704 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.704 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454534+0000", 2024-09-18T07:35:18.704 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0560000000000005e-06 2024-09-18T07:35:18.705 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.705 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.705 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.705 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454591+0000", 2024-09-18T07:35:18.705 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7633999999999997e-05 2024-09-18T07:35:18.705 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.705 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.705 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.705 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454715+0000", 2024-09-18T07:35:18.705 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012419300000000001 2024-09-18T07:35:18.705 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.705 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.706 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:35:18.706 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454817+0000", 2024-09-18T07:35:18.706 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000101452 2024-09-18T07:35:18.706 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.706 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.706 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:18.706 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.455733+0000", 2024-09-18T07:35:18.706 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00091630599999999998 2024-09-18T07:35:18.706 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.706 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.706 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.706 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.456978+0000", 2024-09-18T07:35:18.706 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001245248 2024-09-18T07:35:18.707 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.707 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.707 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.707 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.542970+0000", 2024-09-18T07:35:18.707 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.085991206000000001 2024-09-18T07:35:18.707 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.707 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.707 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.707 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543010+0000", 2024-09-18T07:35:18.707 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0936999999999998e-05 2024-09-18T07:35:18.707 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.707 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.708 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.708 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543042+0000", 2024-09-18T07:35:18.708 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1708999999999997e-05 2024-09-18T07:35:18.708 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.708 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.708 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.708 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.708 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.708 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:23089 7.5 7:abb1cc32:::benchmark_data_smithi137_74180_object23088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:35:18.709 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.457769+0000", 2024-09-18T07:35:18.709 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 173.135481951, 2024-09-18T07:35:18.709 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.085838029999999996, 2024-09-18T07:35:18.709 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.709 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.709 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.709 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:18.709 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:18.709 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:18.709 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 23089 2024-09-18T07:35:18.709 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.710 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.710 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.710 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.710 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457769+0000", 2024-09-18T07:35:18.710 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.710 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.710 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.710 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.710 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457769+0000", 2024-09-18T07:35:18.710 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.710 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.710 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.710 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.711 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457771+0000", 2024-09-18T07:35:18.711 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3199999999999998e-06 2024-09-18T07:35:18.711 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.711 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.711 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.711 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457831+0000", 2024-09-18T07:35:18.711 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9939999999999999e-05 2024-09-18T07:35:18.711 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.711 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.711 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.711 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457848+0000", 2024-09-18T07:35:18.711 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6682999999999999e-05 2024-09-18T07:35:18.711 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.712 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.712 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.712 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457856+0000", 2024-09-18T07:35:18.712 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0809999999999994e-06 2024-09-18T07:35:18.712 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.712 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.712 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.712 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457909+0000", 2024-09-18T07:35:18.713 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3143999999999998e-05 2024-09-18T07:35:18.713 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.713 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.713 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.713 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.458075+0000", 2024-09-18T07:35:18.713 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000166398 2024-09-18T07:35:18.713 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.713 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.713 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:35:18.713 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.458209+0000", 2024-09-18T07:35:18.713 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013364400000000001 2024-09-18T07:35:18.714 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.714 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.714 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:18.714 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.459558+0000", 2024-09-18T07:35:18.714 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0013487519999999999 2024-09-18T07:35:18.714 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.714 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.714 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.714 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.459987+0000", 2024-09-18T07:35:18.714 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00042892000000000001 2024-09-18T07:35:18.714 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.714 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.714 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.714 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543512+0000", 2024-09-18T07:35:18.715 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.083524980999999998 2024-09-18T07:35:18.715 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.715 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.715 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.715 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543569+0000", 2024-09-18T07:35:18.715 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7337999999999999e-05 2024-09-18T07:35:18.715 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.715 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.715 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.715 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543607+0000", 2024-09-18T07:35:18.715 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7829e-05 2024-09-18T07:35:18.715 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.715 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.715 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.716 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.716 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.716 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:23145 7.5 7:a67bcaca:::benchmark_data_smithi137_74180_object23144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:35:18.716 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.558406+0000", 2024-09-18T07:35:18.716 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 173.03484525299999, 2024-09-18T07:35:18.716 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.040502491000000002, 2024-09-18T07:35:18.716 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.716 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.716 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.716 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:18.716 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:18.716 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:18.716 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 23145 2024-09-18T07:35:18.716 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.716 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.717 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.717 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.717 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558406+0000", 2024-09-18T07:35:18.717 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.717 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.717 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.717 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.717 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558406+0000", 2024-09-18T07:35:18.717 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.717 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.717 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.717 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.717 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558408+0000", 2024-09-18T07:35:18.718 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1390000000000002e-06 2024-09-18T07:35:18.718 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.718 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.718 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.718 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558470+0000", 2024-09-18T07:35:18.718 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2756999999999996e-05 2024-09-18T07:35:18.718 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.718 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.718 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.718 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558493+0000", 2024-09-18T07:35:18.719 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2523000000000001e-05 2024-09-18T07:35:18.720 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.720 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.720 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.720 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558501+0000", 2024-09-18T07:35:18.720 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9289999999999998e-06 2024-09-18T07:35:18.720 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.720 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.720 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.720 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558565+0000", 2024-09-18T07:35:18.720 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4553000000000003e-05 2024-09-18T07:35:18.720 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.720 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.720 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.721 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558688+0000", 2024-09-18T07:35:18.721 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000122242 2024-09-18T07:35:18.721 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.721 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.721 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:35:18.721 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.558812+0000", 2024-09-18T07:35:18.721 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012474599999999999 2024-09-18T07:35:18.721 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.721 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.721 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:18.721 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.559762+0000", 2024-09-18T07:35:18.721 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000949105 2024-09-18T07:35:18.721 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.721 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.722 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.722 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.560044+0000", 2024-09-18T07:35:18.722 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00028216800000000001 2024-09-18T07:35:18.722 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.722 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.722 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.722 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.598816+0000", 2024-09-18T07:35:18.722 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.038772136999999998 2024-09-18T07:35:18.722 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.722 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.722 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.722 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.598869+0000", 2024-09-18T07:35:18.722 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2858000000000001e-05 2024-09-18T07:35:18.722 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.722 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.723 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.723 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.598908+0000", 2024-09-18T07:35:18.723 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9334e-05 2024-09-18T07:35:18.723 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.723 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.723 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.723 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.723 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.723 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:26891 7.5 7:a92f246c:::benchmark_data_smithi137_74180_object26890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-18T07:35:18.723 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:27.401622+0000", 2024-09-18T07:35:18.723 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 171.19162872199999, 2024-09-18T07:35:18.723 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.080096663999999998, 2024-09-18T07:35:18.723 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.723 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.723 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.724 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:18.724 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:18.724 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:18.724 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 26891 2024-09-18T07:35:18.724 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.724 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.724 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.724 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.724 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401622+0000", 2024-09-18T07:35:18.724 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.724 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.724 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.724 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.724 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401622+0000", 2024-09-18T07:35:18.725 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.725 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.725 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.725 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.725 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401624+0000", 2024-09-18T07:35:18.725 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0880000000000002e-06 2024-09-18T07:35:18.725 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.725 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.725 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.725 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401694+0000", 2024-09-18T07:35:18.725 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9825000000000003e-05 2024-09-18T07:35:18.725 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.725 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.725 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.726 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401708+0000", 2024-09-18T07:35:18.726 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4249e-05 2024-09-18T07:35:18.726 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.726 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.726 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.726 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401716+0000", 2024-09-18T07:35:18.726 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0460000000000006e-06 2024-09-18T07:35:18.726 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.726 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.726 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.726 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401785+0000", 2024-09-18T07:35:18.726 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8583000000000004e-05 2024-09-18T07:35:18.726 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.726 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.727 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.727 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401892+0000", 2024-09-18T07:35:18.727 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000106885 2024-09-18T07:35:18.727 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.727 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.727 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:35:18.727 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402022+0000", 2024-09-18T07:35:18.727 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012980700000000001 2024-09-18T07:35:18.727 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.727 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.727 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:18.727 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.404167+0000", 2024-09-18T07:35:18.727 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021457550000000001 2024-09-18T07:35:18.727 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.728 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.728 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.728 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.404212+0000", 2024-09-18T07:35:18.728 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5052000000000002e-05 2024-09-18T07:35:18.728 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.728 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.728 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.728 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481614+0000", 2024-09-18T07:35:18.728 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.077401247000000006 2024-09-18T07:35:18.728 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.728 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.728 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.728 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481666+0000", 2024-09-18T07:35:18.729 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2198999999999999e-05 2024-09-18T07:35:18.729 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.729 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.729 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.729 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481719+0000", 2024-09-18T07:35:18.729 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2927999999999999e-05 2024-09-18T07:35:18.729 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.729 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.729 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.729 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.729 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.730 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:26893 7.5 7:a23e6cde:::benchmark_data_smithi137_74180_object26892:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-18T07:35:18.730 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:27.402662+0000", 2024-09-18T07:35:18.730 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 171.190588526, 2024-09-18T07:35:18.730 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.079194041000000007, 2024-09-18T07:35:18.730 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.730 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.730 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:18.730 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:18.730 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:35:18.730 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:18.730 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 26893 2024-09-18T07:35:18.730 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.730 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.730 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.731 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.731 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402662+0000", 2024-09-18T07:35:18.731 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.731 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.731 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.731 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.731 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402662+0000", 2024-09-18T07:35:18.731 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.731 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.731 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.731 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.731 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402664+0000", 2024-09-18T07:35:18.731 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.513e-06 2024-09-18T07:35:18.731 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.732 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.732 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.732 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402694+0000", 2024-09-18T07:35:18.732 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9984e-05 2024-09-18T07:35:18.732 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.732 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.732 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.732 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402706+0000", 2024-09-18T07:35:18.732 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2499e-05 2024-09-18T07:35:18.732 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.732 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.732 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.732 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402713+0000", 2024-09-18T07:35:18.732 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6429999999999998e-06 2024-09-18T07:35:18.733 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.733 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.733 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.733 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402959+0000", 2024-09-18T07:35:18.733 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024601700000000001 2024-09-18T07:35:18.733 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.733 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.733 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.733 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.403094+0000", 2024-09-18T07:35:18.733 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013474299999999999 2024-09-18T07:35:18.733 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.733 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.733 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:35:18.733 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.403289+0000", 2024-09-18T07:35:18.734 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019528500000000001 2024-09-18T07:35:18.734 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.734 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.734 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:35:18.734 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.404306+0000", 2024-09-18T07:35:18.734 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010166179999999999 2024-09-18T07:35:18.734 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.734 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.734 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.734 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.405112+0000", 2024-09-18T07:35:18.734 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00080637099999999995 2024-09-18T07:35:18.734 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.734 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.734 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:18.735 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481779+0000", 2024-09-18T07:35:18.735 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.076667400999999996 2024-09-18T07:35:18.735 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:18.736 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.736 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:35:18.736 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481825+0000", 2024-09-18T07:35:18.736 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5652000000000003e-05 2024-09-18T07:35:18.736 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.737 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.737 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.737 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481856+0000", 2024-09-18T07:35:18.737 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1315000000000001e-05 2024-09-18T07:35:18.737 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.737 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.737 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.737 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.737 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.737 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.34466.0:233 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2024-09-18T07:35:18.737 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:01.270598+0000", 2024-09-18T07:35:18.737 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 17.322652365, 2024-09-18T07:35:18.738 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14644679199999999, 2024-09-18T07:35:18.738 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:35:18.738 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:35:18.738 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:35:18.738 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:35:18.738 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.34466", 2024-09-18T07:35:18.738 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.203:0/952514737", 2024-09-18T07:35:18.738 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 233 2024-09-18T07:35:18.738 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.738 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:35:18.738 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.738 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:35:18.738 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270598+0000", 2024-09-18T07:35:18.739 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.739 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.739 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.739 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:35:18.739 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270598+0000", 2024-09-18T07:35:18.739 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:35:18.739 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.739 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.739 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:35:18.739 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270602+0000", 2024-09-18T07:35:18.739 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4960000000000001e-06 2024-09-18T07:35:18.739 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.739 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.739 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:35:18.740 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270613+0000", 2024-09-18T07:35:18.740 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0890000000000001e-05 2024-09-18T07:35:18.740 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.740 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.740 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:35:18.740 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270634+0000", 2024-09-18T07:35:18.740 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1484e-05 2024-09-18T07:35:18.740 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.740 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.740 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:35:18.740 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270643+0000", 2024-09-18T07:35:18.740 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1249999999999999e-06 2024-09-18T07:35:18.740 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.740 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.741 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.741 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270712+0000", 2024-09-18T07:35:18.741 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8733999999999996e-05 2024-09-18T07:35:18.741 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.741 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.741 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:35:18.741 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270720+0000", 2024-09-18T07:35:18.741 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0209999999999999e-06 2024-09-18T07:35:18.741 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.741 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.741 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.741 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.402460+0000", 2024-09-18T07:35:18.741 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13173957 2024-09-18T07:35:18.742 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.742 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.742 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for missing object", 2024-09-18T07:35:18.742 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.402586+0000", 2024-09-18T07:35:18.742 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012606100000000001 2024-09-18T07:35:18.742 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.742 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.742 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:35:18.742 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.416785+0000", 2024-09-18T07:35:18.742 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.014199145 2024-09-18T07:35:18.742 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.742 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.742 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:35:18.742 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.416882+0000", 2024-09-18T07:35:18.743 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6990999999999993e-05 2024-09-18T07:35:18.743 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:35:18.743 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:35:18.743 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:35:18.743 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.417045+0000", 2024-09-18T07:35:18.743 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000163275 2024-09-18T07:35:18.743 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.743 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.743 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.743 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:35:18.743 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:35:18.743 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:35:18.744 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:18 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:18.744 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:18 smithi203 ceph-mon[32237]: pgmap v233: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1018 B/s rd, 0 B/s wr, 969 op/s; 2466/117270 objects misplaced (2.103%); 14 MiB/s, 222 objects/s recovering 2024-09-18T07:35:18.744 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:18 smithi203 ceph-mon[32237]: osdmap e188: 8 total, 8 up, 8 in 2024-09-18T07:35:18.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:18.781 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:18.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:18.882 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:18.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:18.924 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:18.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:19.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:19.071 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-18T07:35:17.492067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2549291571,"num_objects":39090,"num_object_clones":0,"num_object_copies":117270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2466,"num_objects_unfound":0,"num_objects_dirty":39090,"num_whiteouts":0,"num_read":846,"num_read_kb":705,"num_write":77929,"num_write_kb":2501258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15277,"num_bytes_recovered":991368439,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39943,"ondisk_log_size":39943,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":447,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9007740,"kb_used_data":7968104,"kb_used_omap":12,"kb_used_meta":1039283,"kb_avail":740953476,"statfs":{"total":767960285184,"available":758736359424,"internally_reserved":0,"allocated":8159338496,"data_stored":8154159417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1064226352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-189988864,"num_objects":-2899,"num_object_clones":0,"num_object_copies":-8697,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1425,"num_objects_unfound":0,"num_objects_dirty":-2899,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4864,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1118,"num_bytes_recovered":73269248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.033112"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":272,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.793236+0000","last_change":"2024-09-18T07:33:31.183001+0000","last_active":"2024-09-18T07:35:14.793236+0000","last_peered":"2024-09-18T07:35:14.793236+0000","last_clean":"2024-09-18T07:35:14.793236+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:35:14.793236+0000","last_undegraded":"2024-09-18T07:35:14.793236+0000","last_fullsized":"2024-09-18T07:35:14.793236+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:31.182913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:31.182913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":302,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.792390+0000","last_change":"2024-09-18T07:33:29.685825+0000","last_active":"2024-09-18T07:35:14.792390+0000","last_peered":"2024-09-18T07:35:14.792390+0000","last_clean":"2024-09-18T07:35:14.792390+0000","last_became_active":"2024-09-18T07:32:26.785727+0000","last_became_peered":"2024-09-18T07:32:26.785727+0000","last_unstale":"2024-09-18T07:35:14.792390+0000","last_undegraded":"2024-09-18T07:35:14.792390+0000","last_fullsized":"2024-09-18T07:35:14.792390+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:33:29.685767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.685767+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":287,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.798242+0000","last_change":"2024-09-18T07:35:01.397530+0000","last_active":"2024-09-18T07:35:14.798242+0000","last_peered":"2024-09-18T07:35:14.798242+0000","last_clean":"2024-09-18T07:35:14.798242+0000","last_became_active":"2024-09-18T07:35:01.397199+0000","last_became_peered":"2024-09-18T07:35:01.397199+0000","last_unstale":"2024-09-18T07:35:14.798242+0000","last_undegraded":"2024-09-18T07:35:14.798242+0000","last_fullsized":"2024-09-18T07:35:14.798242+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:57.894611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.894611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:09.974636+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":261,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.792568+0000","last_change":"2024-09-18T07:32:03.953881+0000","last_active":"2024-09-18T07:35:14.792568+0000","last_peered":"2024-09-18T07:35:14.792568+0000","last_clean":"2024-09-18T07:35:14.792568+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:35:14.792568+0000","last_undegraded":"2024-09-18T07:35:14.792568+0000","last_fullsized":"2024-09-18T07:35:14.792568+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.953827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.953827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":160,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.791647+0000","last_change":"2024-09-18T07:35:01.481557+0000","last_active":"2024-09-18T07:35:14.791647+0000","last_peered":"2024-09-18T07:35:14.791647+0000","last_clean":"2024-09-18T07:35:14.791647+0000","last_became_active":"2024-09-18T07:35:01.414396+0000","last_became_peered":"2024-09-18T07:35:01.414396+0000","last_unstale":"2024-09-18T07:35:14.791647+0000","last_undegraded":"2024-09-18T07:35:14.791647+0000","last_fullsized":"2024-09-18T07:35:14.791647+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:01.481514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:01.481514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:13.169861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.808718+0000","last_change":"2024-09-18T07:33:27.006545+0000","last_active":"2024-09-18T07:35:14.808718+0000","last_peered":"2024-09-18T07:35:14.808718+0000","last_clean":"2024-09-18T07:35:14.808718+0000","last_became_active":"2024-09-18T07:30:56.286040+0000","last_became_peered":"2024-09-18T07:30:56.286040+0000","last_unstale":"2024-09-18T07:35:14.808718+0000","last_undegraded":"2024-09-18T07:35:14.808718+0000","last_fullsized":"2024-09-18T07:35:14.808718+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.006488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.006488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":293,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.807926+0000","last_change":"2024-09-18T07:33:22.032321+0000","last_active":"2024-09-18T07:35:14.807926+0000","last_peered":"2024-09-18T07:35:14.807926+0000","last_clean":"2024-09-18T07:35:14.807926+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:35:14.807926+0000","last_undegraded":"2024-09-18T07:35:14.807926+0000","last_fullsized":"2024-09-18T07:35:14.807926+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:22.032110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.032110+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":265,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.792297+0000","last_change":"2024-09-18T07:33:59.829707+0000","last_active":"2024-09-18T07:35:14.792297+0000","last_peered":"2024-09-18T07:35:14.792297+0000","last_clean":"2024-09-18T07:35:14.792297+0000","last_became_active":"2024-09-18T07:31:00.331224+0000","last_became_peered":"2024-09-18T07:31:00.331224+0000","last_unstale":"2024-09-18T07:35:14.792297+0000","last_undegraded":"2024-09-18T07:35:14.792297+0000","last_fullsized":"2024-09-18T07:35:14.792297+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:59.829644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:59.829644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":279,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.800138+0000","last_change":"2024-09-18T07:35:00.392530+0000","last_active":"2024-09-18T07:35:14.800138+0000","last_peered":"2024-09-18T07:35:14.800138+0000","last_clean":"2024-09-18T07:35:14.800138+0000","last_became_active":"2024-09-18T07:30:56.285882+0000","last_became_peered":"2024-09-18T07:30:56.285882+0000","last_unstale":"2024-09-18T07:35:14.800138+0000","last_undegraded":"2024-09-18T07:35:14.800138+0000","last_fullsized":"2024-09-18T07:35:14.800138+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:00.392466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.392466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:06.365891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":285,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.792736+0000","last_change":"2024-09-18T07:32:02.002919+0000","last_active":"2024-09-18T07:35:14.792736+0000","last_peered":"2024-09-18T07:35:14.792736+0000","last_clean":"2024-09-18T07:35:14.792736+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:35:14.792736+0000","last_undegraded":"2024-09-18T07:35:14.792736+0000","last_fullsized":"2024-09-18T07:35:14.792736+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.002811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.002811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":277,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.800035+0000","last_change":"2024-09-18T07:35:04.267623+0000","last_active":"2024-09-18T07:35:14.800035+0000","last_peered":"2024-09-18T07:35:14.800035+0000","last_clean":"2024-09-18T07:35:14.800035+0000","last_became_active":"2024-09-18T07:35:01.413443+0000","last_became_peered":"2024-09-18T07:35:01.413443+0000","last_unstale":"2024-09-18T07:35:14.800035+0000","last_undegraded":"2024-09-18T07:35:14.800035+0000","last_fullsized":"2024-09-18T07:35:14.800035+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:04.267580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:04.267580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:19.052927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":265,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.791712+0000","last_change":"2024-09-18T07:35:01.410563+0000","last_active":"2024-09-18T07:35:14.791712+0000","last_peered":"2024-09-18T07:35:14.791712+0000","last_clean":"2024-09-18T07:35:14.791712+0000","last_became_active":"2024-09-18T07:35:01.410284+0000","last_became_peered":"2024-09-18T07:35:01.410284+0000","last_unstale":"2024-09-18T07:35:14.791712+0000","last_undegraded":"2024-09-18T07:35:14.791712+0000","last_fullsized":"2024-09-18T07:35:14.791712+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:10.525949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":263,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.799390+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:35:14.799390+0000","last_peered":"2024-09-18T07:35:14.799390+0000","last_clean":"2024-09-18T07:35:14.799390+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:35:14.799390+0000","last_undegraded":"2024-09-18T07:35:14.799390+0000","last_fullsized":"2024-09-18T07:35:14.799390+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:17.419087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":271,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.793078+0000","last_change":"2024-09-18T07:30:58.317041+0000","last_active":"2024-09-18T07:35:14.793078+0000","last_peered":"2024-09-18T07:35:14.793078+0000","last_clean":"2024-09-18T07:35:14.793078+0000","last_became_active":"2024-09-18T07:30:58.316479+0000","last_became_peered":"2024-09-18T07:30:58.316479+0000","last_unstale":"2024-09-18T07:35:14.793078+0000","last_undegraded":"2024-09-18T07:35:14.793078+0000","last_fullsized":"2024-09-18T07:35:14.793078+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:51.873972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":40,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.794669+0000","last_change":"2024-09-18T07:35:01.399995+0000","last_active":"2024-09-18T07:35:14.794669+0000","last_peered":"2024-09-18T07:35:14.794669+0000","last_clean":"2024-09-18T07:35:14.794669+0000","last_became_active":"2024-09-18T07:35:01.398605+0000","last_became_peered":"2024-09-18T07:35:01.398605+0000","last_unstale":"2024-09-18T07:35:14.794669+0000","last_undegraded":"2024-09-18T07:35:14.794669+0000","last_fullsized":"2024-09-18T07:35:14.794669+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:55.843864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:55.843864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:15.129566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":267,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.799017+0000","last_change":"2024-09-18T07:33:51.759330+0000","last_active":"2024-09-18T07:35:14.799017+0000","last_peered":"2024-09-18T07:35:14.799017+0000","last_clean":"2024-09-18T07:35:14.799017+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:35:14.799017+0000","last_undegraded":"2024-09-18T07:35:14.799017+0000","last_fullsized":"2024-09-18T07:35:14.799017+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:51.759259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:51.759259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:18.399187+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":262,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.793163+0000","last_change":"2024-09-18T07:32:06.976159+0000","last_active":"2024-09-18T07:35:14.793163+0000","last_peered":"2024-09-18T07:35:14.793163+0000","last_clean":"2024-09-18T07:35:14.793163+0000","last_became_active":"2024-09-18T07:31:02.490014+0000","last_became_peered":"2024-09-18T07:31:02.490014+0000","last_unstale":"2024-09-18T07:35:14.793163+0000","last_undegraded":"2024-09-18T07:35:14.793163+0000","last_fullsized":"2024-09-18T07:35:14.793163+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:06.976113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:06.976113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":285,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.792729+0000","last_change":"2024-09-18T07:35:01.415292+0000","last_active":"2024-09-18T07:35:14.792729+0000","last_peered":"2024-09-18T07:35:14.792729+0000","last_clean":"2024-09-18T07:35:14.792729+0000","last_became_active":"2024-09-18T07:35:01.414088+0000","last_became_peered":"2024-09-18T07:35:01.414088+0000","last_unstale":"2024-09-18T07:35:14.792729+0000","last_undegraded":"2024-09-18T07:35:14.792729+0000","last_fullsized":"2024-09-18T07:35:14.792729+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:53.330304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:53.330304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:15.078639+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":302,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.794395+0000","last_change":"2024-09-18T07:33:59.921943+0000","last_active":"2024-09-18T07:35:14.794395+0000","last_peered":"2024-09-18T07:35:14.794395+0000","last_clean":"2024-09-18T07:35:14.794395+0000","last_became_active":"2024-09-18T07:30:58.316266+0000","last_became_peered":"2024-09-18T07:30:58.316266+0000","last_unstale":"2024-09-18T07:35:14.794395+0000","last_undegraded":"2024-09-18T07:35:14.794395+0000","last_fullsized":"2024-09-18T07:35:14.794395+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:59.921856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:59.921856+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:28.827554+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":265,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.799591+0000","last_change":"2024-09-18T07:33:30.955441+0000","last_active":"2024-09-18T07:35:14.799591+0000","last_peered":"2024-09-18T07:35:14.799591+0000","last_clean":"2024-09-18T07:35:14.799591+0000","last_became_active":"2024-09-18T07:31:00.337474+0000","last_became_peered":"2024-09-18T07:31:00.337474+0000","last_unstale":"2024-09-18T07:35:14.799591+0000","last_undegraded":"2024-09-18T07:35:14.799591+0000","last_fullsized":"2024-09-18T07:35:14.799591+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.955213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.955213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:07.055140+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":261,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.809582+0000","last_change":"2024-09-18T07:33:26.004195+0000","last_active":"2024-09-18T07:35:14.809582+0000","last_peered":"2024-09-18T07:35:14.809582+0000","last_clean":"2024-09-18T07:35:14.809582+0000","last_became_active":"2024-09-18T07:31:02.492348+0000","last_became_peered":"2024-09-18T07:31:02.492348+0000","last_unstale":"2024-09-18T07:35:14.809582+0000","last_undegraded":"2024-09-18T07:35:14.809582+0000","last_fullsized":"2024-09-18T07:35:14.809582+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:26.003425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:26.003425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.792851+0000","last_change":"2024-09-18T07:33:53.756188+0000","last_active":"2024-09-18T07:35:14.792851+0000","last_peered":"2024-09-18T07:35:14.792851+0000","last_clean":"2024-09-18T07:35:14.792851+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:35:14.792851+0000","last_undegraded":"2024-09-18T07:35:14.792851+0000","last_fullsized":"2024-09-18T07:35:14.792851+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:53.756119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:53.756119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:19.988286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":283,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.808141+0000","last_change":"2024-09-18T07:30:58.319115+0000","last_active":"2024-09-18T07:35:14.808141+0000","last_peered":"2024-09-18T07:35:14.808141+0000","last_clean":"2024-09-18T07:35:14.808141+0000","last_became_active":"2024-09-18T07:30:58.316380+0000","last_became_peered":"2024-09-18T07:30:58.316380+0000","last_unstale":"2024-09-18T07:35:14.808141+0000","last_undegraded":"2024-09-18T07:35:14.808141+0000","last_fullsized":"2024-09-18T07:35:14.808141+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:42.212716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":257,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.792855+0000","last_change":"2024-09-18T07:31:00.334313+0000","last_active":"2024-09-18T07:35:14.792855+0000","last_peered":"2024-09-18T07:35:14.792855+0000","last_clean":"2024-09-18T07:35:14.792855+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:35:14.792855+0000","last_undegraded":"2024-09-18T07:35:14.792855+0000","last_fullsized":"2024-09-18T07:35:14.792855+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:45.918853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":44,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.797551+0000","last_change":"2024-09-18T07:35:01.435697+0000","last_active":"2024-09-18T07:35:14.797551+0000","last_peered":"2024-09-18T07:35:14.797551+0000","last_clean":"2024-09-18T07:35:14.797551+0000","last_became_active":"2024-09-18T07:35:01.413110+0000","last_became_peered":"2024-09-18T07:35:01.413110+0000","last_unstale":"2024-09-18T07:35:14.797551+0000","last_undegraded":"2024-09-18T07:35:14.797551+0000","last_fullsized":"2024-09-18T07:35:14.797551+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:33.565334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.565334+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":282,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.798374+0000","last_change":"2024-09-18T07:33:32.976203+0000","last_active":"2024-09-18T07:35:14.798374+0000","last_peered":"2024-09-18T07:35:14.798374+0000","last_clean":"2024-09-18T07:35:14.798374+0000","last_became_active":"2024-09-18T07:30:56.283705+0000","last_became_peered":"2024-09-18T07:30:56.283705+0000","last_unstale":"2024-09-18T07:35:14.798374+0000","last_undegraded":"2024-09-18T07:35:14.798374+0000","last_fullsized":"2024-09-18T07:35:14.798374+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:32.976129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.976129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:09.022890+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":321,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.794805+0000","last_change":"2024-09-18T07:35:01.438392+0000","last_active":"2024-09-18T07:35:14.794805+0000","last_peered":"2024-09-18T07:35:14.794805+0000","last_clean":"2024-09-18T07:35:14.794805+0000","last_became_active":"2024-09-18T07:35:01.410926+0000","last_became_peered":"2024-09-18T07:35:01.410926+0000","last_unstale":"2024-09-18T07:35:14.794805+0000","last_undegraded":"2024-09-18T07:35:14.794805+0000","last_fullsized":"2024-09-18T07:35:14.794805+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:33:36.118340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.118340+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"173'10","reported_seq":350,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.801894+0000","last_change":"2024-09-18T07:35:01.428003+0000","last_active":"2024-09-18T07:35:14.801894+0000","last_peered":"2024-09-18T07:35:14.801894+0000","last_clean":"2024-09-18T07:35:14.801894+0000","last_became_active":"2024-09-18T07:35:01.407926+0000","last_became_peered":"2024-09-18T07:35:01.407926+0000","last_unstale":"2024-09-18T07:35:14.801894+0000","last_undegraded":"2024-09-18T07:35:14.801894+0000","last_fullsized":"2024-09-18T07:35:14.801894+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-09-18T07:33:59.647617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:59.647617+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:28.224416+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"187'10556","reported_seq":20119,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:16.435387+0000","last_change":"2024-09-18T07:35:14.499552+0000","last_active":"2024-09-18T07:35:16.435387+0000","last_peered":"2024-09-18T07:35:16.435387+0000","last_clean":"2024-09-18T07:35:16.435387+0000","last_became_active":"2024-09-18T07:35:14.479753+0000","last_became_peered":"2024-09-18T07:35:14.479753+0000","last_unstale":"2024-09-18T07:35:16.435387+0000","last_undegraded":"2024-09-18T07:35:16.435387+0000","last_fullsized":"2024-09-18T07:35:16.435387+0000","mapping_epoch":185,"log_start":"116'4612","ondisk_log_start":"116'4612","created":81,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'9951","last_scrub_stamp":"2024-09-18T07:34:55.656779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:55.656779+0000","objects_scrubbed":3023,"log_size":5944,"log_dups_size":0,"ondisk_log_size":5944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:21.120338+0000","scrub_duration":304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158466048,"num_objects":2418,"num_object_clones":0,"num_object_copies":7254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10556,"num_write_kb":415168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4597,"num_bytes_recovered":301268992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":294,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.799081+0000","last_change":"2024-09-18T07:34:37.067259+0000","last_active":"2024-09-18T07:35:14.799081+0000","last_peered":"2024-09-18T07:35:14.799081+0000","last_clean":"2024-09-18T07:35:14.799081+0000","last_became_active":"2024-09-18T07:30:58.312627+0000","last_became_peered":"2024-09-18T07:30:58.312627+0000","last_unstale":"2024-09-18T07:35:14.799081+0000","last_undegraded":"2024-09-18T07:35:14.799081+0000","last_fullsized":"2024-09-18T07:35:14.799081+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:34:37.067162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:37.067162+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:52.474545+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":295,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.808660+0000","last_change":"2024-09-18T07:35:01.407698+0000","last_active":"2024-09-18T07:35:14.808660+0000","last_peered":"2024-09-18T07:35:14.808660+0000","last_clean":"2024-09-18T07:35:14.808660+0000","last_became_active":"2024-09-18T07:35:01.407339+0000","last_became_peered":"2024-09-18T07:35:01.407339+0000","last_unstale":"2024-09-18T07:35:14.808660+0000","last_undegraded":"2024-09-18T07:35:14.808660+0000","last_fullsized":"2024-09-18T07:35:14.808660+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:01.294615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:01.294615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":326,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.793294+0000","last_change":"2024-09-18T07:33:33.706253+0000","last_active":"2024-09-18T07:35:14.793294+0000","last_peered":"2024-09-18T07:35:14.793294+0000","last_clean":"2024-09-18T07:35:14.793294+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:35:14.793294+0000","last_undegraded":"2024-09-18T07:35:14.793294+0000","last_fullsized":"2024-09-18T07:35:14.793294+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:33:33.706201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.706201+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":259,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.798896+0000","last_change":"2024-09-18T07:34:40.386158+0000","last_active":"2024-09-18T07:35:14.798896+0000","last_peered":"2024-09-18T07:35:14.798896+0000","last_clean":"2024-09-18T07:35:14.798896+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:35:14.798896+0000","last_undegraded":"2024-09-18T07:35:14.798896+0000","last_fullsized":"2024-09-18T07:35:14.798896+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:40.386112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:34:40.386112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:03.303185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"187'9612","reported_seq":9864,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:16.628750+0000","last_change":"2024-09-18T07:35:01.404212+0000","last_active":"2024-09-18T07:35:16.628750+0000","last_peered":"2024-09-18T07:35:16.628750+0000","last_clean":"2024-09-18T07:35:16.628750+0000","last_became_active":"2024-09-18T07:35:01.402640+0000","last_became_peered":"2024-09-18T07:35:01.402640+0000","last_unstale":"2024-09-18T07:35:16.628750+0000","last_undegraded":"2024-09-18T07:35:16.628750+0000","last_fullsized":"2024-09-18T07:35:16.628750+0000","mapping_epoch":81,"log_start":"169'9115","ondisk_log_start":"169'9115","created":81,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'8619","last_scrub_stamp":"2024-09-18T07:34:29.885340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:29.885340+0000","objects_scrubbed":1910,"log_size":497,"log_dups_size":3868,"ondisk_log_size":497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:52.760695+0000","scrub_duration":623,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159055872,"num_objects":2427,"num_object_clones":0,"num_object_copies":7281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1378,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":351,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.808423+0000","last_change":"2024-09-18T07:35:01.463637+0000","last_active":"2024-09-18T07:35:14.808423+0000","last_peered":"2024-09-18T07:35:14.808423+0000","last_clean":"2024-09-18T07:35:14.808423+0000","last_became_active":"2024-09-18T07:35:01.418909+0000","last_became_peered":"2024-09-18T07:35:01.418909+0000","last_unstale":"2024-09-18T07:35:14.808423+0000","last_undegraded":"2024-09-18T07:35:14.808423+0000","last_fullsized":"2024-09-18T07:35:14.808423+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:32:00.660377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:00.660377+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":287,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.809960+0000","last_change":"2024-09-18T07:35:01.408650+0000","last_active":"2024-09-18T07:35:14.809960+0000","last_peered":"2024-09-18T07:35:14.809960+0000","last_clean":"2024-09-18T07:35:14.809960+0000","last_became_active":"2024-09-18T07:35:01.407422+0000","last_became_peered":"2024-09-18T07:35:01.407422+0000","last_unstale":"2024-09-18T07:35:14.809960+0000","last_undegraded":"2024-09-18T07:35:14.809960+0000","last_fullsized":"2024-09-18T07:35:14.809960+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.665591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.665591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":280,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.792720+0000","last_change":"2024-09-18T07:35:01.401507+0000","last_active":"2024-09-18T07:35:14.792720+0000","last_peered":"2024-09-18T07:35:14.792720+0000","last_clean":"2024-09-18T07:35:14.792720+0000","last_became_active":"2024-09-18T07:35:01.401024+0000","last_became_peered":"2024-09-18T07:35:01.401024+0000","last_unstale":"2024-09-18T07:35:14.792720+0000","last_undegraded":"2024-09-18T07:35:14.792720+0000","last_fullsized":"2024-09-18T07:35:14.792720+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:54.808785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:54.808785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:06.641181+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":254,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.792725+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:35:14.792725+0000","last_peered":"2024-09-18T07:35:14.792725+0000","last_clean":"2024-09-18T07:35:14.792725+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:35:14.792725+0000","last_undegraded":"2024-09-18T07:35:14.792725+0000","last_fullsized":"2024-09-18T07:35:14.792725+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:13.029688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"187'9737","reported_seq":10040,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:16.606813+0000","last_change":"2024-09-18T07:35:04.794511+0000","last_active":"2024-09-18T07:35:16.606813+0000","last_peered":"2024-09-18T07:35:16.606813+0000","last_clean":"2024-09-18T07:35:16.606813+0000","last_became_active":"2024-09-18T07:35:04.793376+0000","last_became_peered":"2024-09-18T07:35:04.793376+0000","last_unstale":"2024-09-18T07:35:16.606813+0000","last_undegraded":"2024-09-18T07:35:16.606813+0000","last_fullsized":"2024-09-18T07:35:16.606813+0000","mapping_epoch":81,"log_start":"173'9256","ondisk_log_start":"173'9256","created":81,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'8772","last_scrub_stamp":"2024-09-18T07:34:30.492072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:30.492072+0000","objects_scrubbed":1900,"log_size":481,"log_dups_size":3908,"ondisk_log_size":481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:59.397943+0000","scrub_duration":228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157351936,"num_objects":2401,"num_object_clones":0,"num_object_copies":7203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1382,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":291,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.792819+0000","last_change":"2024-09-18T07:35:00.882253+0000","last_active":"2024-09-18T07:35:14.792819+0000","last_peered":"2024-09-18T07:35:14.792819+0000","last_clean":"2024-09-18T07:35:14.792819+0000","last_became_active":"2024-09-18T07:30:58.309649+0000","last_became_peered":"2024-09-18T07:30:58.309649+0000","last_unstale":"2024-09-18T07:35:14.792819+0000","last_undegraded":"2024-09-18T07:35:14.792819+0000","last_fullsized":"2024-09-18T07:35:14.792819+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-18T07:35:00.882191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.882191+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:04.068834+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.795086+0000","last_change":"2024-09-18T07:33:24.213432+0000","last_active":"2024-09-18T07:35:14.795086+0000","last_peered":"2024-09-18T07:35:14.795086+0000","last_clean":"2024-09-18T07:35:14.795086+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:35:14.795086+0000","last_undegraded":"2024-09-18T07:35:14.795086+0000","last_fullsized":"2024-09-18T07:35:14.795086+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:24.213356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:24.213356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":326,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.808250+0000","last_change":"2024-09-18T07:34:39.253606+0000","last_active":"2024-09-18T07:35:14.808250+0000","last_peered":"2024-09-18T07:35:14.808250+0000","last_clean":"2024-09-18T07:35:14.808250+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:35:14.808250+0000","last_undegraded":"2024-09-18T07:35:14.808250+0000","last_fullsized":"2024-09-18T07:35:14.808250+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:34:39.253558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:39.253558+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:46.279033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":40,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.794759+0000","last_change":"2024-09-18T07:35:01.394273+0000","last_active":"2024-09-18T07:35:14.794759+0000","last_peered":"2024-09-18T07:35:14.794759+0000","last_clean":"2024-09-18T07:35:14.794759+0000","last_became_active":"2024-09-18T07:35:01.393846+0000","last_became_peered":"2024-09-18T07:35:01.393846+0000","last_unstale":"2024-09-18T07:35:14.794759+0000","last_undegraded":"2024-09-18T07:35:14.794759+0000","last_fullsized":"2024-09-18T07:35:14.794759+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:54.740249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:54.740249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"187'9767","reported_seq":10053,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:16.431435+0000","last_change":"2024-09-18T07:35:09.445846+0000","last_active":"2024-09-18T07:35:16.431435+0000","last_peered":"2024-09-18T07:35:16.431435+0000","last_clean":"2024-09-18T07:35:16.431435+0000","last_became_active":"2024-09-18T07:35:09.444538+0000","last_became_peered":"2024-09-18T07:35:09.444538+0000","last_unstale":"2024-09-18T07:35:16.431435+0000","last_undegraded":"2024-09-18T07:35:16.431435+0000","last_fullsized":"2024-09-18T07:35:16.431435+0000","mapping_epoch":81,"log_start":"177'9469","ondisk_log_start":"177'9469","created":81,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'8929","last_scrub_stamp":"2024-09-18T07:34:30.240820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:30.240820+0000","objects_scrubbed":1857,"log_size":298,"log_dups_size":3988,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:39.749113+0000","scrub_duration":268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161677312,"num_objects":2467,"num_object_clones":0,"num_object_copies":7401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1356,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":277,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.798476+0000","last_change":"2024-09-18T07:30:58.307470+0000","last_active":"2024-09-18T07:35:14.798476+0000","last_peered":"2024-09-18T07:35:14.798476+0000","last_clean":"2024-09-18T07:35:14.798476+0000","last_became_active":"2024-09-18T07:30:58.307035+0000","last_became_peered":"2024-09-18T07:30:58.307035+0000","last_unstale":"2024-09-18T07:35:14.798476+0000","last_undegraded":"2024-09-18T07:35:14.798476+0000","last_fullsized":"2024-09-18T07:35:14.798476+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:15.814064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":272,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.800225+0000","last_change":"2024-09-18T07:33:29.471394+0000","last_active":"2024-09-18T07:35:14.800225+0000","last_peered":"2024-09-18T07:35:14.800225+0000","last_clean":"2024-09-18T07:35:14.800225+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:35:14.800225+0000","last_undegraded":"2024-09-18T07:35:14.800225+0000","last_fullsized":"2024-09-18T07:35:14.800225+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:29.471330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.471330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":325,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.792096+0000","last_change":"2024-09-18T07:33:52.317149+0000","last_active":"2024-09-18T07:35:14.792096+0000","last_peered":"2024-09-18T07:35:14.792096+0000","last_clean":"2024-09-18T07:35:14.792096+0000","last_became_active":"2024-09-18T07:31:00.336905+0000","last_became_peered":"2024-09-18T07:31:00.336905+0000","last_unstale":"2024-09-18T07:35:14.792096+0000","last_undegraded":"2024-09-18T07:35:14.792096+0000","last_fullsized":"2024-09-18T07:35:14.792096+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:33:52.317075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:52.317075+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":252,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.791850+0000","last_change":"2024-09-18T07:31:02.348538+0000","last_active":"2024-09-18T07:35:14.791850+0000","last_peered":"2024-09-18T07:35:14.791850+0000","last_clean":"2024-09-18T07:35:14.791850+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:35:14.791850+0000","last_undegraded":"2024-09-18T07:35:14.791850+0000","last_fullsized":"2024-09-18T07:35:14.791850+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:42.474541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"187'9582","reported_seq":9825,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:16.424382+0000","last_change":"2024-09-18T07:35:13.465018+0000","last_active":"2024-09-18T07:35:16.424382+0000","last_peered":"2024-09-18T07:35:16.424382+0000","last_clean":"2024-09-18T07:35:16.424382+0000","last_became_active":"2024-09-18T07:35:13.464303+0000","last_became_peered":"2024-09-18T07:35:13.464303+0000","last_unstale":"2024-09-18T07:35:16.424382+0000","last_undegraded":"2024-09-18T07:35:16.424382+0000","last_fullsized":"2024-09-18T07:35:16.424382+0000","mapping_epoch":81,"log_start":"181'9339","ondisk_log_start":"181'9339","created":81,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'8745","last_scrub_stamp":"2024-09-18T07:34:29.900338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:29.900338+0000","objects_scrubbed":1866,"log_size":243,"log_dups_size":3983,"ondisk_log_size":243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:40.409627+0000","scrub_duration":415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160366592,"num_objects":2447,"num_object_clones":0,"num_object_copies":7341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1384,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":354,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.792235+0000","last_change":"2024-09-18T07:35:01.449045+0000","last_active":"2024-09-18T07:35:14.792235+0000","last_peered":"2024-09-18T07:35:14.792235+0000","last_clean":"2024-09-18T07:35:14.792235+0000","last_became_active":"2024-09-18T07:35:01.415275+0000","last_became_peered":"2024-09-18T07:35:01.415275+0000","last_unstale":"2024-09-18T07:35:14.792235+0000","last_undegraded":"2024-09-18T07:35:14.792235+0000","last_fullsized":"2024-09-18T07:35:14.792235+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:34:39.610788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:39.610788+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:42.977048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":266,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.799194+0000","last_change":"2024-09-18T07:30:56.284440+0000","last_active":"2024-09-18T07:35:14.799194+0000","last_peered":"2024-09-18T07:35:14.799194+0000","last_clean":"2024-09-18T07:35:14.799194+0000","last_became_active":"2024-09-18T07:30:56.284120+0000","last_became_peered":"2024-09-18T07:30:56.284120+0000","last_unstale":"2024-09-18T07:35:14.799194+0000","last_undegraded":"2024-09-18T07:35:14.799194+0000","last_fullsized":"2024-09-18T07:35:14.799194+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:13.961499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":264,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.798803+0000","last_change":"2024-09-18T07:33:21.834910+0000","last_active":"2024-09-18T07:35:14.798803+0000","last_peered":"2024-09-18T07:35:14.798803+0000","last_clean":"2024-09-18T07:35:14.798803+0000","last_became_active":"2024-09-18T07:31:00.334776+0000","last_became_peered":"2024-09-18T07:31:00.334776+0000","last_unstale":"2024-09-18T07:35:14.798803+0000","last_undegraded":"2024-09-18T07:35:14.798803+0000","last_fullsized":"2024-09-18T07:35:14.798803+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:47.509771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":254,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.798152+0000","last_change":"2024-09-18T07:31:02.490534+0000","last_active":"2024-09-18T07:35:14.798152+0000","last_peered":"2024-09-18T07:35:14.798152+0000","last_clean":"2024-09-18T07:35:14.798152+0000","last_became_active":"2024-09-18T07:31:02.490186+0000","last_became_peered":"2024-09-18T07:31:02.490186+0000","last_unstale":"2024-09-18T07:35:14.798152+0000","last_undegraded":"2024-09-18T07:35:14.798152+0000","last_fullsized":"2024-09-18T07:35:14.798152+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:42.011069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"187'9756","reported_seq":10094,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:16.445249+0000","last_change":"2024-09-18T07:34:44.792087+0000","last_active":"2024-09-18T07:35:16.445249+0000","last_peered":"2024-09-18T07:35:16.445249+0000","last_clean":"2024-09-18T07:35:16.445249+0000","last_became_active":"2024-09-18T07:34:44.791542+0000","last_became_peered":"2024-09-18T07:34:44.791542+0000","last_unstale":"2024-09-18T07:35:16.445249+0000","last_undegraded":"2024-09-18T07:35:16.445249+0000","last_fullsized":"2024-09-18T07:35:16.445249+0000","mapping_epoch":81,"log_start":"153'8782","ondisk_log_start":"153'8782","created":81,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'8555","last_scrub_stamp":"2024-09-18T07:34:30.102158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:30.102158+0000","objects_scrubbed":1903,"log_size":974,"log_dups_size":3609,"ondisk_log_size":974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:38.413684+0000","scrub_duration":460,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155254784,"num_objects":2369,"num_object_clones":0,"num_object_copies":7107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1342,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":371,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.799087+0000","last_change":"2024-09-18T07:34:31.955399+0000","last_active":"2024-09-18T07:35:14.799087+0000","last_peered":"2024-09-18T07:35:14.799087+0000","last_clean":"2024-09-18T07:35:14.799087+0000","last_became_active":"2024-09-18T07:30:58.316812+0000","last_became_peered":"2024-09-18T07:30:58.316812+0000","last_unstale":"2024-09-18T07:35:14.799087+0000","last_undegraded":"2024-09-18T07:35:14.799087+0000","last_fullsized":"2024-09-18T07:35:14.799087+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'28","last_scrub_stamp":"2024-09-18T07:34:31.955336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.955336+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:43.708652+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":266,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.793354+0000","last_change":"2024-09-18T07:30:56.285353+0000","last_active":"2024-09-18T07:35:14.793354+0000","last_peered":"2024-09-18T07:35:14.793354+0000","last_clean":"2024-09-18T07:35:14.793354+0000","last_became_active":"2024-09-18T07:30:56.285032+0000","last_became_peered":"2024-09-18T07:30:56.285032+0000","last_unstale":"2024-09-18T07:35:14.793354+0000","last_undegraded":"2024-09-18T07:35:14.793354+0000","last_fullsized":"2024-09-18T07:35:14.793354+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:06.856268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":282,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.799260+0000","last_change":"2024-09-18T07:34:39.420024+0000","last_active":"2024-09-18T07:35:14.799260+0000","last_peered":"2024-09-18T07:35:14.799260+0000","last_clean":"2024-09-18T07:35:14.799260+0000","last_became_active":"2024-09-18T07:30:55.283292+0000","last_became_peered":"2024-09-18T07:30:55.283292+0000","last_unstale":"2024-09-18T07:35:14.799260+0000","last_undegraded":"2024-09-18T07:35:14.799260+0000","last_fullsized":"2024-09-18T07:35:14.799260+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:34:39.419958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:34:39.419958+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:59.903039+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":256,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.793443+0000","last_change":"2024-09-18T07:31:00.332738+0000","last_active":"2024-09-18T07:35:14.793443+0000","last_peered":"2024-09-18T07:35:14.793443+0000","last_clean":"2024-09-18T07:35:14.793443+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:35:14.793443+0000","last_undegraded":"2024-09-18T07:35:14.793443+0000","last_fullsized":"2024-09-18T07:35:14.793443+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":274,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.808462+0000","last_change":"2024-09-18T07:35:01.419122+0000","last_active":"2024-09-18T07:35:14.808462+0000","last_peered":"2024-09-18T07:35:14.808462+0000","last_clean":"2024-09-18T07:35:14.808462+0000","last_became_active":"2024-09-18T07:35:01.418809+0000","last_became_peered":"2024-09-18T07:35:01.418809+0000","last_unstale":"2024-09-18T07:35:14.808462+0000","last_undegraded":"2024-09-18T07:35:14.808462+0000","last_fullsized":"2024-09-18T07:35:14.808462+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:56.289176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:34:56.289176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:12.265135+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"187'9889","reported_seq":10158,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:16.760220+0000","last_change":"2024-09-18T07:34:56.862493+0000","last_active":"2024-09-18T07:35:16.760220+0000","last_peered":"2024-09-18T07:35:16.760220+0000","last_clean":"2024-09-18T07:35:16.760220+0000","last_became_active":"2024-09-18T07:34:56.861791+0000","last_became_peered":"2024-09-18T07:34:56.861791+0000","last_unstale":"2024-09-18T07:35:16.760220+0000","last_undegraded":"2024-09-18T07:35:16.760220+0000","last_fullsized":"2024-09-18T07:35:16.760220+0000","mapping_epoch":81,"log_start":"165'9229","ondisk_log_start":"165'9229","created":134,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'8823","last_scrub_stamp":"2024-09-18T07:34:30.780852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:30.780852+0000","objects_scrubbed":1972,"log_size":660,"log_dups_size":3797,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:58.757449+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161218560,"num_objects":2460,"num_object_clones":0,"num_object_copies":7380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1327,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":297,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.799624+0000","last_change":"2024-09-18T07:34:38.975221+0000","last_active":"2024-09-18T07:35:14.799624+0000","last_peered":"2024-09-18T07:35:14.799624+0000","last_clean":"2024-09-18T07:35:14.799624+0000","last_became_active":"2024-09-18T07:30:58.310490+0000","last_became_peered":"2024-09-18T07:30:58.310490+0000","last_unstale":"2024-09-18T07:35:14.799624+0000","last_undegraded":"2024-09-18T07:35:14.799624+0000","last_fullsized":"2024-09-18T07:35:14.799624+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-18T07:34:38.975156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:38.975156+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:05.736223+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":282,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.798508+0000","last_change":"2024-09-18T07:35:03.095873+0000","last_active":"2024-09-18T07:35:14.798508+0000","last_peered":"2024-09-18T07:35:14.798508+0000","last_clean":"2024-09-18T07:35:14.798508+0000","last_became_active":"2024-09-18T07:30:56.284803+0000","last_became_peered":"2024-09-18T07:30:56.284803+0000","last_unstale":"2024-09-18T07:35:14.798508+0000","last_undegraded":"2024-09-18T07:35:14.798508+0000","last_fullsized":"2024-09-18T07:35:14.798508+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:03.095826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:03.095826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:03.655181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":283,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.793816+0000","last_change":"2024-09-18T07:32:02.418590+0000","last_active":"2024-09-18T07:35:14.793816+0000","last_peered":"2024-09-18T07:35:14.793816+0000","last_clean":"2024-09-18T07:35:14.793816+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:35:14.793816+0000","last_undegraded":"2024-09-18T07:35:14.793816+0000","last_fullsized":"2024-09-18T07:35:14.793816+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:32:02.418507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.418507+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:44.816446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":271,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.796025+0000","last_change":"2024-09-18T07:35:01.403942+0000","last_active":"2024-09-18T07:35:14.796025+0000","last_peered":"2024-09-18T07:35:14.796025+0000","last_clean":"2024-09-18T07:35:14.796025+0000","last_became_active":"2024-09-18T07:35:01.403680+0000","last_became_peered":"2024-09-18T07:35:01.403680+0000","last_unstale":"2024-09-18T07:35:14.796025+0000","last_undegraded":"2024-09-18T07:35:14.796025+0000","last_fullsized":"2024-09-18T07:35:14.796025+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:08.951679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":44,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.795689+0000","last_change":"2024-09-18T07:35:01.441600+0000","last_active":"2024-09-18T07:35:14.795689+0000","last_peered":"2024-09-18T07:35:14.795689+0000","last_clean":"2024-09-18T07:35:14.795689+0000","last_became_active":"2024-09-18T07:35:01.406712+0000","last_became_peered":"2024-09-18T07:35:01.406712+0000","last_unstale":"2024-09-18T07:35:14.795689+0000","last_undegraded":"2024-09-18T07:35:14.795689+0000","last_fullsized":"2024-09-18T07:35:14.795689+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:57.655797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.655797+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"187'9864","reported_seq":19725,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:16.410138+0000","last_change":"2024-09-18T07:35:10.454916+0000","last_active":"2024-09-18T07:35:16.410138+0000","last_peered":"2024-09-18T07:35:16.410138+0000","last_clean":"2024-09-18T07:35:16.410138+0000","last_became_active":"2024-09-18T07:35:10.450307+0000","last_became_peered":"2024-09-18T07:35:10.450307+0000","last_unstale":"2024-09-18T07:35:16.410138+0000","last_undegraded":"2024-09-18T07:35:16.410138+0000","last_fullsized":"2024-09-18T07:35:16.410138+0000","mapping_epoch":181,"log_start":"161'9099","ondisk_log_start":"161'9099","created":81,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'8779","last_scrub_stamp":"2024-09-18T07:34:30.341369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:30.341369+0000","objects_scrubbed":1918,"log_size":765,"log_dups_size":3452,"ondisk_log_size":765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:58.408765+0000","scrub_duration":452,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163905536,"num_objects":2501,"num_object_clones":0,"num_object_copies":7503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1380,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2870,"num_bytes_recovered":187957248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":299,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.794938+0000","last_change":"2024-09-18T07:33:27.040227+0000","last_active":"2024-09-18T07:35:14.794938+0000","last_peered":"2024-09-18T07:35:14.794938+0000","last_clean":"2024-09-18T07:35:14.794938+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:35:14.794938+0000","last_undegraded":"2024-09-18T07:35:14.794938+0000","last_fullsized":"2024-09-18T07:35:14.794938+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:15.523581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":270,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.792384+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:35:14.792384+0000","last_peered":"2024-09-18T07:35:14.792384+0000","last_clean":"2024-09-18T07:35:14.792384+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:35:14.792384+0000","last_undegraded":"2024-09-18T07:35:14.792384+0000","last_fullsized":"2024-09-18T07:35:14.792384+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:11.500230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":59,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.793960+0000","last_change":"2024-09-18T07:35:01.421692+0000","last_active":"2024-09-18T07:35:14.793960+0000","last_peered":"2024-09-18T07:35:14.793960+0000","last_clean":"2024-09-18T07:35:14.793960+0000","last_became_active":"2024-09-18T07:35:01.400524+0000","last_became_peered":"2024-09-18T07:35:01.400524+0000","last_unstale":"2024-09-18T07:35:14.793960+0000","last_undegraded":"2024-09-18T07:35:14.793960+0000","last_fullsized":"2024-09-18T07:35:14.793960+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":16,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":263,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.800792+0000","last_change":"2024-09-18T07:33:27.394247+0000","last_active":"2024-09-18T07:35:14.800792+0000","last_peered":"2024-09-18T07:35:14.800792+0000","last_clean":"2024-09-18T07:35:14.800792+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:35:14.800792+0000","last_undegraded":"2024-09-18T07:35:14.800792+0000","last_fullsized":"2024-09-18T07:35:14.800792+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.394203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.394203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:14.210979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":278,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.798740+0000","last_change":"2024-09-18T07:34:58.080008+0000","last_active":"2024-09-18T07:35:14.798740+0000","last_peered":"2024-09-18T07:35:14.798740+0000","last_clean":"2024-09-18T07:35:14.798740+0000","last_became_active":"2024-09-18T07:31:02.347390+0000","last_became_peered":"2024-09-18T07:31:02.347390+0000","last_unstale":"2024-09-18T07:35:14.798740+0000","last_undegraded":"2024-09-18T07:35:14.798740+0000","last_fullsized":"2024-09-18T07:35:14.798740+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:58.079941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:34:58.079941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:13.659972+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"187'10061","reported_seq":20097,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:16.747532+0000","last_change":"2024-09-18T07:35:10.455349+0000","last_active":"2024-09-18T07:35:16.747532+0000","last_peered":"2024-09-18T07:35:16.747532+0000","last_clean":"2024-09-18T07:35:16.747532+0000","last_became_active":"2024-09-18T07:35:10.451483+0000","last_became_peered":"2024-09-18T07:35:10.451483+0000","last_unstale":"2024-09-18T07:35:16.747532+0000","last_undegraded":"2024-09-18T07:35:16.747532+0000","last_fullsized":"2024-09-18T07:35:16.747532+0000","mapping_epoch":181,"log_start":"157'9161","ondisk_log_start":"157'9161","created":134,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'8868","last_scrub_stamp":"2024-09-18T07:34:31.930782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.930782+0000","objects_scrubbed":2001,"log_size":900,"log_dups_size":3711,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:38.949610+0000","scrub_duration":320,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165937152,"num_objects":2532,"num_object_clones":0,"num_object_copies":7596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1368,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2946,"num_bytes_recovered":193069056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"173'18","reported_seq":401,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.799388+0000","last_change":"2024-09-18T07:35:01.444206+0000","last_active":"2024-09-18T07:35:14.799388+0000","last_peered":"2024-09-18T07:35:14.799388+0000","last_clean":"2024-09-18T07:35:14.799388+0000","last_became_active":"2024-09-18T07:35:01.409012+0000","last_became_peered":"2024-09-18T07:35:01.409012+0000","last_unstale":"2024-09-18T07:35:14.799388+0000","last_undegraded":"2024-09-18T07:35:14.799388+0000","last_fullsized":"2024-09-18T07:35:14.799388+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'17","last_scrub_stamp":"2024-09-18T07:33:33.849285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.849285+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:14.193593+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.810609+0000","last_change":"2024-09-18T07:31:59.617046+0000","last_active":"2024-09-18T07:35:14.810609+0000","last_peered":"2024-09-18T07:35:14.810609+0000","last_clean":"2024-09-18T07:35:14.810609+0000","last_became_active":"2024-09-18T07:30:56.288149+0000","last_became_peered":"2024-09-18T07:30:56.288149+0000","last_unstale":"2024-09-18T07:35:14.810609+0000","last_undegraded":"2024-09-18T07:35:14.810609+0000","last_fullsized":"2024-09-18T07:35:14.810609+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:59.616988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:59.616988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:14.105579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":46,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.794009+0000","last_change":"2024-09-18T07:35:01.434847+0000","last_active":"2024-09-18T07:35:14.794009+0000","last_peered":"2024-09-18T07:35:14.794009+0000","last_clean":"2024-09-18T07:35:14.794009+0000","last_became_active":"2024-09-18T07:35:01.400273+0000","last_became_peered":"2024-09-18T07:35:01.400273+0000","last_unstale":"2024-09-18T07:35:14.794009+0000","last_undegraded":"2024-09-18T07:35:14.794009+0000","last_fullsized":"2024-09-18T07:35:14.794009+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:33:23.172848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:33:23.172848+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":40,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.797584+0000","last_change":"2024-09-18T07:35:01.413545+0000","last_active":"2024-09-18T07:35:14.797584+0000","last_peered":"2024-09-18T07:35:14.797584+0000","last_clean":"2024-09-18T07:35:14.797584+0000","last_became_active":"2024-09-18T07:35:01.412217+0000","last_became_peered":"2024-09-18T07:35:01.412217+0000","last_unstale":"2024-09-18T07:35:14.797584+0000","last_undegraded":"2024-09-18T07:35:14.797584+0000","last_fullsized":"2024-09-18T07:35:14.797584+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:36.964421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:36.964421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:48.586654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":259,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.800560+0000","last_change":"2024-09-18T07:33:22.358792+0000","last_active":"2024-09-18T07:35:14.800560+0000","last_peered":"2024-09-18T07:35:14.800560+0000","last_clean":"2024-09-18T07:35:14.800560+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:35:14.800560+0000","last_undegraded":"2024-09-18T07:35:14.800560+0000","last_fullsized":"2024-09-18T07:35:14.800560+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.358746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.358746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:11.158583+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"187'10716","reported_seq":11007,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:16.444567+0000","last_change":"2024-09-18T07:34:41.087606+0000","last_active":"2024-09-18T07:35:16.444567+0000","last_peered":"2024-09-18T07:35:16.444567+0000","last_clean":"2024-09-18T07:35:16.444567+0000","last_became_active":"2024-09-18T07:32:08.063070+0000","last_became_peered":"2024-09-18T07:32:08.063070+0000","last_unstale":"2024-09-18T07:35:16.444567+0000","last_undegraded":"2024-09-18T07:35:16.444567+0000","last_fullsized":"2024-09-18T07:35:16.444567+0000","mapping_epoch":81,"log_start":"116'4727","ondisk_log_start":"116'4727","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'9587","last_scrub_stamp":"2024-09-18T07:34:41.087538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:41.087538+0000","objects_scrubbed":3521,"log_size":5989,"log_dups_size":0,"ondisk_log_size":5989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:45.620523+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156696600,"num_objects":2392,"num_object_clones":0,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10716,"num_write_kb":419393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":307,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.793342+0000","last_change":"2024-09-18T07:34:03.832922+0000","last_active":"2024-09-18T07:35:14.793342+0000","last_peered":"2024-09-18T07:35:14.793342+0000","last_clean":"2024-09-18T07:35:14.793342+0000","last_became_active":"2024-09-18T07:32:26.785711+0000","last_became_peered":"2024-09-18T07:32:26.785711+0000","last_unstale":"2024-09-18T07:35:14.793342+0000","last_undegraded":"2024-09-18T07:35:14.793342+0000","last_fullsized":"2024-09-18T07:35:14.793342+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:34:03.832858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:03.832858+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:25.331056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":288,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.793165+0000","last_change":"2024-09-18T07:35:03.866215+0000","last_active":"2024-09-18T07:35:14.793165+0000","last_peered":"2024-09-18T07:35:14.793165+0000","last_clean":"2024-09-18T07:35:14.793165+0000","last_became_active":"2024-09-18T07:35:01.401063+0000","last_became_peered":"2024-09-18T07:35:01.401063+0000","last_unstale":"2024-09-18T07:35:14.793165+0000","last_undegraded":"2024-09-18T07:35:14.793165+0000","last_fullsized":"2024-09-18T07:35:14.793165+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:03.866148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:03.866148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:25.066815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":263,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.800462+0000","last_change":"2024-09-18T07:33:36.582039+0000","last_active":"2024-09-18T07:35:14.800462+0000","last_peered":"2024-09-18T07:35:14.800462+0000","last_clean":"2024-09-18T07:35:14.800462+0000","last_became_active":"2024-09-18T07:31:00.336172+0000","last_became_peered":"2024-09-18T07:31:00.336172+0000","last_unstale":"2024-09-18T07:35:14.800462+0000","last_undegraded":"2024-09-18T07:35:14.800462+0000","last_fullsized":"2024-09-18T07:35:14.800462+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:36.581998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.581998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":260,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.808989+0000","last_change":"2024-09-18T07:33:20.990542+0000","last_active":"2024-09-18T07:35:14.808989+0000","last_peered":"2024-09-18T07:35:14.808989+0000","last_clean":"2024-09-18T07:35:14.808989+0000","last_became_active":"2024-09-18T07:31:02.490953+0000","last_became_peered":"2024-09-18T07:31:02.490953+0000","last_unstale":"2024-09-18T07:35:14.808989+0000","last_undegraded":"2024-09-18T07:35:14.808989+0000","last_fullsized":"2024-09-18T07:35:14.808989+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:20.990401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:20.990401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"187'10609","reported_seq":10893,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:16.567465+0000","last_change":"2024-09-18T07:34:58.585934+0000","last_active":"2024-09-18T07:35:16.567465+0000","last_peered":"2024-09-18T07:35:16.567465+0000","last_clean":"2024-09-18T07:35:16.567465+0000","last_became_active":"2024-09-18T07:32:08.059971+0000","last_became_peered":"2024-09-18T07:32:08.059971+0000","last_unstale":"2024-09-18T07:35:16.567465+0000","last_undegraded":"2024-09-18T07:35:16.567465+0000","last_fullsized":"2024-09-18T07:35:16.567465+0000","mapping_epoch":81,"log_start":"116'4711","ondisk_log_start":"116'4711","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'10144","last_scrub_stamp":"2024-09-18T07:34:58.585803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:58.585803+0000","objects_scrubbed":2867,"log_size":5898,"log_dups_size":0,"ondisk_log_size":5898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:02.881562+0000","scrub_duration":673,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156696576,"num_objects":2391,"num_object_clones":0,"num_object_copies":7173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10609,"num_write_kb":416000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":297,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.793335+0000","last_change":"2024-09-18T07:33:56.797549+0000","last_active":"2024-09-18T07:35:14.793335+0000","last_peered":"2024-09-18T07:35:14.793335+0000","last_clean":"2024-09-18T07:35:14.793335+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:35:14.793335+0000","last_undegraded":"2024-09-18T07:35:14.793335+0000","last_fullsized":"2024-09-18T07:35:14.793335+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:33:56.797466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:56.797466+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":272,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.799182+0000","last_change":"2024-09-18T07:34:33.614709+0000","last_active":"2024-09-18T07:35:14.799182+0000","last_peered":"2024-09-18T07:35:14.799182+0000","last_clean":"2024-09-18T07:35:14.799182+0000","last_became_active":"2024-09-18T07:30:56.284020+0000","last_became_peered":"2024-09-18T07:30:56.284020+0000","last_unstale":"2024-09-18T07:35:14.799182+0000","last_undegraded":"2024-09-18T07:35:14.799182+0000","last_fullsized":"2024-09-18T07:35:14.799182+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:33.614666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:33.614666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:35.196971+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":265,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.798789+0000","last_change":"2024-09-18T07:32:05.239202+0000","last_active":"2024-09-18T07:35:14.798789+0000","last_peered":"2024-09-18T07:35:14.798789+0000","last_clean":"2024-09-18T07:35:14.798789+0000","last_became_active":"2024-09-18T07:31:00.332836+0000","last_became_peered":"2024-09-18T07:31:00.332836+0000","last_unstale":"2024-09-18T07:35:14.798789+0000","last_undegraded":"2024-09-18T07:35:14.798789+0000","last_fullsized":"2024-09-18T07:35:14.798789+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:05.239160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.239160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:45.089326+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":261,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.794899+0000","last_change":"2024-09-18T07:33:21.860561+0000","last_active":"2024-09-18T07:35:14.794899+0000","last_peered":"2024-09-18T07:35:14.794899+0000","last_clean":"2024-09-18T07:35:14.794899+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:35:14.794899+0000","last_undegraded":"2024-09-18T07:35:14.794899+0000","last_fullsized":"2024-09-18T07:35:14.794899+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.860495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.860495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"187'10678","reported_seq":14760,"reported_epoch":187,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-18T07:35:16.768700+0000","last_change":"2024-09-18T07:35:02.507514+0000","last_active":"2024-09-18T07:35:16.768700+0000","last_peered":"2024-09-18T07:35:16.768700+0000","last_clean":"2024-09-18T07:34:59.367879+0000","last_became_active":"2024-09-18T07:35:02.471693+0000","last_became_peered":"2024-09-18T07:35:02.471693+0000","last_unstale":"2024-09-18T07:35:16.768700+0000","last_undegraded":"2024-09-18T07:35:16.768700+0000","last_fullsized":"2024-09-18T07:35:16.768700+0000","mapping_epoch":173,"log_start":"121'5540","ondisk_log_start":"121'5540","created":81,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":5138,"log_dups_size":0,"ondisk_log_size":5138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161611776,"num_objects":2466,"num_object_clones":0,"num_object_copies":7398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2466,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10678,"num_write_kb":420608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1912,"num_bytes_recovered":125304832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,4],"avail_no_missing":["0","3","4"],"object_location_counts":[{"shards":"0,3,4","objects":2466}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":284,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.792938+0000","last_change":"2024-09-18T07:32:05.000363+0000","last_active":"2024-09-18T07:35:14.792938+0000","last_peered":"2024-09-18T07:35:14.792938+0000","last_clean":"2024-09-18T07:35:14.792938+0000","last_became_active":"2024-09-18T07:30:58.316423+0000","last_became_peered":"2024-09-18T07:30:58.316423+0000","last_unstale":"2024-09-18T07:35:14.792938+0000","last_undegraded":"2024-09-18T07:35:14.792938+0000","last_fullsized":"2024-09-18T07:35:14.792938+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:32:05.000310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.000310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":272,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.808890+0000","last_change":"2024-09-18T07:32:06.688493+0000","last_active":"2024-09-18T07:35:14.808890+0000","last_peered":"2024-09-18T07:35:14.808890+0000","last_clean":"2024-09-18T07:35:14.808890+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:35:14.808890+0000","last_undegraded":"2024-09-18T07:35:14.808890+0000","last_fullsized":"2024-09-18T07:35:14.808890+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:06.688444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:06.688444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"173'11","reported_seq":359,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.794663+0000","last_change":"2024-09-18T07:35:01.413702+0000","last_active":"2024-09-18T07:35:14.794663+0000","last_peered":"2024-09-18T07:35:14.794663+0000","last_clean":"2024-09-18T07:35:14.794663+0000","last_became_active":"2024-09-18T07:35:01.403765+0000","last_became_peered":"2024-09-18T07:35:01.403765+0000","last_unstale":"2024-09-18T07:35:14.794663+0000","last_undegraded":"2024-09-18T07:35:14.794663+0000","last_fullsized":"2024-09-18T07:35:14.794663+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'10","last_scrub_stamp":"2024-09-18T07:33:33.000874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.000874+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":261,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.793117+0000","last_change":"2024-09-18T07:35:04.907232+0000","last_active":"2024-09-18T07:35:14.793117+0000","last_peered":"2024-09-18T07:35:14.793117+0000","last_clean":"2024-09-18T07:35:14.793117+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:35:14.793117+0000","last_undegraded":"2024-09-18T07:35:14.793117+0000","last_fullsized":"2024-09-18T07:35:14.793117+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:04.907166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:04.907166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:25.077151+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"187'10232","reported_seq":20192,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:16.556041+0000","last_change":"2024-09-18T07:35:10.456841+0000","last_active":"2024-09-18T07:35:16.556041+0000","last_peered":"2024-09-18T07:35:16.556041+0000","last_clean":"2024-09-18T07:35:16.556041+0000","last_became_active":"2024-09-18T07:35:10.449030+0000","last_became_peered":"2024-09-18T07:35:10.449030+0000","last_unstale":"2024-09-18T07:35:16.556041+0000","last_undegraded":"2024-09-18T07:35:16.556041+0000","last_fullsized":"2024-09-18T07:35:16.556041+0000","mapping_epoch":181,"log_start":"149'9067","ondisk_log_start":"149'9067","created":81,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'8921","last_scrub_stamp":"2024-09-18T07:34:31.186602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.186602+0000","objects_scrubbed":1940,"log_size":1165,"log_dups_size":3573,"ondisk_log_size":1165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:40.094772+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157351936,"num_objects":2401,"num_object_clones":0,"num_object_copies":7203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1356,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2778,"num_bytes_recovered":181993472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":275,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.792146+0000","last_change":"2024-09-18T07:30:58.314424+0000","last_active":"2024-09-18T07:35:14.792146+0000","last_peered":"2024-09-18T07:35:14.792146+0000","last_clean":"2024-09-18T07:35:14.792146+0000","last_became_active":"2024-09-18T07:30:58.314127+0000","last_became_peered":"2024-09-18T07:30:58.314127+0000","last_unstale":"2024-09-18T07:35:14.792146+0000","last_undegraded":"2024-09-18T07:35:14.792146+0000","last_fullsized":"2024-09-18T07:35:14.792146+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:47.415855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":271,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.799543+0000","last_change":"2024-09-18T07:33:28.459796+0000","last_active":"2024-09-18T07:35:14.799543+0000","last_peered":"2024-09-18T07:35:14.799543+0000","last_clean":"2024-09-18T07:35:14.799543+0000","last_became_active":"2024-09-18T07:30:56.288868+0000","last_became_peered":"2024-09-18T07:30:56.288868+0000","last_unstale":"2024-09-18T07:35:14.799543+0000","last_undegraded":"2024-09-18T07:35:14.799543+0000","last_fullsized":"2024-09-18T07:35:14.799543+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:28.459747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:28.459747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":265,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.797549+0000","last_change":"2024-09-18T07:34:31.959675+0000","last_active":"2024-09-18T07:35:14.797549+0000","last_peered":"2024-09-18T07:35:14.797549+0000","last_clean":"2024-09-18T07:35:14.797549+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:35:14.797549+0000","last_undegraded":"2024-09-18T07:35:14.797549+0000","last_fullsized":"2024-09-18T07:35:14.797549+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:31.959636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.959636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:50.298137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.792306+0000","last_change":"2024-09-18T07:35:01.411215+0000","last_active":"2024-09-18T07:35:14.792306+0000","last_peered":"2024-09-18T07:35:14.792306+0000","last_clean":"2024-09-18T07:35:14.792306+0000","last_became_active":"2024-09-18T07:35:01.410890+0000","last_became_peered":"2024-09-18T07:35:01.410890+0000","last_unstale":"2024-09-18T07:35:14.792306+0000","last_undegraded":"2024-09-18T07:35:14.792306+0000","last_fullsized":"2024-09-18T07:35:14.792306+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:02.301353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:34:02.301353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:28.098361+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"187'10268","reported_seq":10567,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:16.752694+0000","last_change":"2024-09-18T07:34:36.801770+0000","last_active":"2024-09-18T07:35:16.752694+0000","last_peered":"2024-09-18T07:35:16.752694+0000","last_clean":"2024-09-18T07:35:16.752694+0000","last_became_active":"2024-09-18T07:34:36.800541+0000","last_became_peered":"2024-09-18T07:34:36.800541+0000","last_unstale":"2024-09-18T07:35:16.752694+0000","last_undegraded":"2024-09-18T07:35:16.752694+0000","last_fullsized":"2024-09-18T07:35:16.752694+0000","mapping_epoch":81,"log_start":"145'8920","ondisk_log_start":"145'8920","created":81,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'8860","last_scrub_stamp":"2024-09-18T07:34:30.270402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:30.270402+0000","objects_scrubbed":1991,"log_size":1348,"log_dups_size":3491,"ondisk_log_size":1348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:39.547972+0000","scrub_duration":670,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159186944,"num_objects":2429,"num_object_clones":0,"num_object_copies":7287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1450,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":360,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.798890+0000","last_change":"2024-09-18T07:35:01.443942+0000","last_active":"2024-09-18T07:35:14.798890+0000","last_peered":"2024-09-18T07:35:14.798890+0000","last_clean":"2024-09-18T07:35:14.798890+0000","last_became_active":"2024-09-18T07:35:01.409401+0000","last_became_peered":"2024-09-18T07:35:01.409401+0000","last_unstale":"2024-09-18T07:35:14.798890+0000","last_undegraded":"2024-09-18T07:35:14.798890+0000","last_fullsized":"2024-09-18T07:35:14.798890+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:33:32.803884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.803884+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":40,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.795968+0000","last_change":"2024-09-18T07:35:01.412993+0000","last_active":"2024-09-18T07:35:14.795968+0000","last_peered":"2024-09-18T07:35:14.795968+0000","last_clean":"2024-09-18T07:35:14.795968+0000","last_became_active":"2024-09-18T07:35:01.411821+0000","last_became_peered":"2024-09-18T07:35:01.411821+0000","last_unstale":"2024-09-18T07:35:14.795968+0000","last_undegraded":"2024-09-18T07:35:14.795968+0000","last_fullsized":"2024-09-18T07:35:14.795968+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.481153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.481153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":272,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.795176+0000","last_change":"2024-09-18T07:33:34.966610+0000","last_active":"2024-09-18T07:35:14.795176+0000","last_peered":"2024-09-18T07:35:14.795176+0000","last_clean":"2024-09-18T07:35:14.795176+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:35:14.795176+0000","last_undegraded":"2024-09-18T07:35:14.795176+0000","last_fullsized":"2024-09-18T07:35:14.795176+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:34.966537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:34.966537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:13.904823+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":40,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.794089+0000","last_change":"2024-09-18T07:35:01.408078+0000","last_active":"2024-09-18T07:35:14.794089+0000","last_peered":"2024-09-18T07:35:14.794089+0000","last_clean":"2024-09-18T07:35:14.794089+0000","last_became_active":"2024-09-18T07:35:01.406980+0000","last_became_peered":"2024-09-18T07:35:01.406980+0000","last_unstale":"2024-09-18T07:35:14.794089+0000","last_undegraded":"2024-09-18T07:35:14.794089+0000","last_fullsized":"2024-09-18T07:35:14.794089+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"187'10682","reported_seq":10930,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:16.627963+0000","last_change":"2024-09-18T07:32:08.065180+0000","last_active":"2024-09-18T07:35:16.627963+0000","last_peered":"2024-09-18T07:35:16.627963+0000","last_clean":"2024-09-18T07:35:16.627963+0000","last_became_active":"2024-09-18T07:32:08.064911+0000","last_became_peered":"2024-09-18T07:32:08.064911+0000","last_unstale":"2024-09-18T07:35:16.627963+0000","last_undegraded":"2024-09-18T07:35:16.627963+0000","last_fullsized":"2024-09-18T07:35:16.627963+0000","mapping_epoch":81,"log_start":"124'6156","ondisk_log_start":"124'6156","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:32:07.038817+0000","objects_scrubbed":0,"log_size":4526,"log_dups_size":0,"ondisk_log_size":4526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156893184,"num_objects":2394,"num_object_clones":0,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10682,"num_write_kb":418432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":292,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.792914+0000","last_change":"2024-09-18T07:32:02.967912+0000","last_active":"2024-09-18T07:35:14.792914+0000","last_peered":"2024-09-18T07:35:14.792914+0000","last_clean":"2024-09-18T07:35:14.792914+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:35:14.792914+0000","last_undegraded":"2024-09-18T07:35:14.792914+0000","last_fullsized":"2024-09-18T07:35:14.792914+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:32:02.967835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.967835+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":40,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.795587+0000","last_change":"2024-09-18T07:35:01.400050+0000","last_active":"2024-09-18T07:35:14.795587+0000","last_peered":"2024-09-18T07:35:14.795587+0000","last_clean":"2024-09-18T07:35:14.795587+0000","last_became_active":"2024-09-18T07:35:01.398499+0000","last_became_peered":"2024-09-18T07:35:01.398499+0000","last_unstale":"2024-09-18T07:35:14.795587+0000","last_undegraded":"2024-09-18T07:35:14.795587+0000","last_fullsized":"2024-09-18T07:35:14.795587+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.796621+0000","last_change":"2024-09-18T07:33:56.967701+0000","last_active":"2024-09-18T07:35:14.796621+0000","last_peered":"2024-09-18T07:35:14.796621+0000","last_clean":"2024-09-18T07:35:14.796621+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:35:14.796621+0000","last_undegraded":"2024-09-18T07:35:14.796621+0000","last_fullsized":"2024-09-18T07:35:14.796621+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:56.967627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:56.967627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:18.872883+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":266,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.808722+0000","last_change":"2024-09-18T07:35:01.419420+0000","last_active":"2024-09-18T07:35:14.808722+0000","last_peered":"2024-09-18T07:35:14.808722+0000","last_clean":"2024-09-18T07:35:14.808722+0000","last_became_active":"2024-09-18T07:35:01.418735+0000","last_became_peered":"2024-09-18T07:35:01.418735+0000","last_unstale":"2024-09-18T07:35:14.808722+0000","last_undegraded":"2024-09-18T07:35:14.808722+0000","last_fullsized":"2024-09-18T07:35:14.808722+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:10.188683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"187'10416","reported_seq":10703,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:16.497971+0000","last_change":"2024-09-18T07:34:55.402724+0000","last_active":"2024-09-18T07:35:16.497971+0000","last_peered":"2024-09-18T07:35:16.497971+0000","last_clean":"2024-09-18T07:35:16.497971+0000","last_became_active":"2024-09-18T07:32:08.061925+0000","last_became_peered":"2024-09-18T07:32:08.061925+0000","last_unstale":"2024-09-18T07:35:16.497971+0000","last_undegraded":"2024-09-18T07:35:16.497971+0000","last_fullsized":"2024-09-18T07:35:16.497971+0000","mapping_epoch":81,"log_start":"122'5862","ondisk_log_start":"122'5862","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'9792","last_scrub_stamp":"2024-09-18T07:34:55.402631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:32:07.038817+0000","last_clean_scrub_stamp":"2024-09-18T07:34:55.402631+0000","objects_scrubbed":3020,"log_size":4554,"log_dups_size":0,"ondisk_log_size":4554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:00.691137+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157024256,"num_objects":2396,"num_object_clones":0,"num_object_copies":7188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10416,"num_write_kb":409984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":338,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.798931+0000","last_change":"2024-09-18T07:35:01.458468+0000","last_active":"2024-09-18T07:35:14.798931+0000","last_peered":"2024-09-18T07:35:14.798931+0000","last_clean":"2024-09-18T07:35:14.798931+0000","last_became_active":"2024-09-18T07:35:01.416425+0000","last_became_peered":"2024-09-18T07:35:01.416425+0000","last_unstale":"2024-09-18T07:35:14.798931+0000","last_undegraded":"2024-09-18T07:35:14.798931+0000","last_fullsized":"2024-09-18T07:35:14.798931+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:34:03.531262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:03.531262+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:27.905479+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":54,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.795799+0000","last_change":"2024-09-18T07:35:02.205821+0000","last_active":"2024-09-18T07:35:14.795799+0000","last_peered":"2024-09-18T07:35:14.795799+0000","last_clean":"2024-09-18T07:35:14.795799+0000","last_became_active":"2024-09-18T07:35:01.406695+0000","last_became_peered":"2024-09-18T07:35:01.406695+0000","last_unstale":"2024-09-18T07:35:14.795799+0000","last_undegraded":"2024-09-18T07:35:14.795799+0000","last_fullsized":"2024-09-18T07:35:14.795799+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:35:02.205775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.205775+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:09.118035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"173'10","reported_seq":48,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.794401+0000","last_change":"2024-09-18T07:35:01.422700+0000","last_active":"2024-09-18T07:35:14.794401+0000","last_peered":"2024-09-18T07:35:14.794401+0000","last_clean":"2024-09-18T07:35:14.794401+0000","last_became_active":"2024-09-18T07:35:01.400330+0000","last_became_peered":"2024-09-18T07:35:01.400330+0000","last_unstale":"2024-09-18T07:35:14.794401+0000","last_undegraded":"2024-09-18T07:35:14.794401+0000","last_fullsized":"2024-09-18T07:35:14.794401+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-09-18T07:32:57.792795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:57.792795+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":251,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.798890+0000","last_change":"2024-09-18T07:31:02.489914+0000","last_active":"2024-09-18T07:35:14.798890+0000","last_peered":"2024-09-18T07:35:14.798890+0000","last_clean":"2024-09-18T07:35:14.798890+0000","last_became_active":"2024-09-18T07:31:02.489619+0000","last_became_peered":"2024-09-18T07:31:02.489619+0000","last_unstale":"2024-09-18T07:35:14.798890+0000","last_undegraded":"2024-09-18T07:35:14.798890+0000","last_fullsized":"2024-09-18T07:35:14.798890+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:47.456591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":279,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.798273+0000","last_change":"2024-09-18T07:34:39.639200+0000","last_active":"2024-09-18T07:35:14.798273+0000","last_peered":"2024-09-18T07:35:14.798273+0000","last_clean":"2024-09-18T07:35:14.798273+0000","last_became_active":"2024-09-18T07:30:56.282357+0000","last_became_peered":"2024-09-18T07:30:56.282357+0000","last_unstale":"2024-09-18T07:35:14.798273+0000","last_undegraded":"2024-09-18T07:35:14.798273+0000","last_fullsized":"2024-09-18T07:35:14.798273+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:39.639148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:39.639148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:58.385140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":299,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.809810+0000","last_change":"2024-09-18T07:35:01.453222+0000","last_active":"2024-09-18T07:35:14.809810+0000","last_peered":"2024-09-18T07:35:14.809810+0000","last_clean":"2024-09-18T07:35:14.809810+0000","last_became_active":"2024-09-18T07:35:01.407787+0000","last_became_peered":"2024-09-18T07:35:01.407787+0000","last_unstale":"2024-09-18T07:35:14.809810+0000","last_undegraded":"2024-09-18T07:35:14.809810+0000","last_fullsized":"2024-09-18T07:35:14.809810+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:08.125087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":263,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.801034+0000","last_change":"2024-09-18T07:33:32.589362+0000","last_active":"2024-09-18T07:35:14.801034+0000","last_peered":"2024-09-18T07:35:14.801034+0000","last_clean":"2024-09-18T07:35:14.801034+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:35:14.801034+0000","last_undegraded":"2024-09-18T07:35:14.801034+0000","last_fullsized":"2024-09-18T07:35:14.801034+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.589289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.589289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":40,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.795877+0000","last_change":"2024-09-18T07:35:01.404598+0000","last_active":"2024-09-18T07:35:14.795877+0000","last_peered":"2024-09-18T07:35:14.795877+0000","last_clean":"2024-09-18T07:35:14.795877+0000","last_became_active":"2024-09-18T07:35:01.403293+0000","last_became_peered":"2024-09-18T07:35:01.403293+0000","last_unstale":"2024-09-18T07:35:14.795877+0000","last_undegraded":"2024-09-18T07:35:14.795877+0000","last_fullsized":"2024-09-18T07:35:14.795877+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:58.688631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:58.688631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":309,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.795000+0000","last_change":"2024-09-18T07:35:01.448510+0000","last_active":"2024-09-18T07:35:14.795000+0000","last_peered":"2024-09-18T07:35:14.795000+0000","last_clean":"2024-09-18T07:35:14.795000+0000","last_became_active":"2024-09-18T07:35:01.414458+0000","last_became_peered":"2024-09-18T07:35:01.414458+0000","last_unstale":"2024-09-18T07:35:14.795000+0000","last_undegraded":"2024-09-18T07:35:14.795000+0000","last_fullsized":"2024-09-18T07:35:14.795000+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":40,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.794259+0000","last_change":"2024-09-18T07:35:01.398699+0000","last_active":"2024-09-18T07:35:14.794259+0000","last_peered":"2024-09-18T07:35:14.794259+0000","last_clean":"2024-09-18T07:35:14.794259+0000","last_became_active":"2024-09-18T07:35:01.398380+0000","last_became_peered":"2024-09-18T07:35:01.398380+0000","last_unstale":"2024-09-18T07:35:14.794259+0000","last_undegraded":"2024-09-18T07:35:14.794259+0000","last_fullsized":"2024-09-18T07:35:14.794259+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:58.901574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:58.901574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:00.777032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":278,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.809681+0000","last_change":"2024-09-18T07:35:01.420446+0000","last_active":"2024-09-18T07:35:14.809681+0000","last_peered":"2024-09-18T07:35:14.809681+0000","last_clean":"2024-09-18T07:35:14.809681+0000","last_became_active":"2024-09-18T07:35:01.420133+0000","last_became_peered":"2024-09-18T07:35:01.420133+0000","last_unstale":"2024-09-18T07:35:14.809681+0000","last_undegraded":"2024-09-18T07:35:14.809681+0000","last_fullsized":"2024-09-18T07:35:14.809681+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:58.276259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:58.276259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":267,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.793934+0000","last_change":"2024-09-18T07:35:01.396655+0000","last_active":"2024-09-18T07:35:14.793934+0000","last_peered":"2024-09-18T07:35:14.793934+0000","last_clean":"2024-09-18T07:35:14.793934+0000","last_became_active":"2024-09-18T07:35:01.396305+0000","last_became_peered":"2024-09-18T07:35:01.396305+0000","last_unstale":"2024-09-18T07:35:14.793934+0000","last_undegraded":"2024-09-18T07:35:14.793934+0000","last_fullsized":"2024-09-18T07:35:14.793934+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:09.906976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":317,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.800037+0000","last_change":"2024-09-18T07:35:02.342002+0000","last_active":"2024-09-18T07:35:14.800037+0000","last_peered":"2024-09-18T07:35:14.800037+0000","last_clean":"2024-09-18T07:35:14.800037+0000","last_became_active":"2024-09-18T07:35:01.416744+0000","last_became_peered":"2024-09-18T07:35:01.416744+0000","last_unstale":"2024-09-18T07:35:14.800037+0000","last_undegraded":"2024-09-18T07:35:14.800037+0000","last_fullsized":"2024-09-18T07:35:14.800037+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:35:02.341919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.341919+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:21.096808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":285,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.791535+0000","last_change":"2024-09-18T07:35:01.414704+0000","last_active":"2024-09-18T07:35:14.791535+0000","last_peered":"2024-09-18T07:35:14.791535+0000","last_clean":"2024-09-18T07:35:14.791535+0000","last_became_active":"2024-09-18T07:35:01.414024+0000","last_became_peered":"2024-09-18T07:35:01.414024+0000","last_unstale":"2024-09-18T07:35:14.791535+0000","last_undegraded":"2024-09-18T07:35:14.791535+0000","last_fullsized":"2024-09-18T07:35:14.791535+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:35.707148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:35.707148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:43.320630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"173'10","reported_seq":339,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.809104+0000","last_change":"2024-09-18T07:35:01.449006+0000","last_active":"2024-09-18T07:35:14.809104+0000","last_peered":"2024-09-18T07:35:14.809104+0000","last_clean":"2024-09-18T07:35:14.809104+0000","last_became_active":"2024-09-18T07:35:01.407568+0000","last_became_peered":"2024-09-18T07:35:01.407568+0000","last_unstale":"2024-09-18T07:35:14.809104+0000","last_undegraded":"2024-09-18T07:35:14.809104+0000","last_fullsized":"2024-09-18T07:35:14.809104+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:07.146036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":253,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.793824+0000","last_change":"2024-09-18T07:31:02.343206+0000","last_active":"2024-09-18T07:35:14.793824+0000","last_peered":"2024-09-18T07:35:14.793824+0000","last_clean":"2024-09-18T07:35:14.793824+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:35:14.793824+0000","last_undegraded":"2024-09-18T07:35:14.793824+0000","last_fullsized":"2024-09-18T07:35:14.793824+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:08.883011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":321,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.809204+0000","last_change":"2024-09-18T07:35:01.459777+0000","last_active":"2024-09-18T07:35:14.809204+0000","last_peered":"2024-09-18T07:35:14.809204+0000","last_clean":"2024-09-18T07:35:14.809204+0000","last_became_active":"2024-09-18T07:35:01.407738+0000","last_became_peered":"2024-09-18T07:35:01.407738+0000","last_unstale":"2024-09-18T07:35:14.809204+0000","last_undegraded":"2024-09-18T07:35:14.809204+0000","last_fullsized":"2024-09-18T07:35:14.809204+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:33:54.295330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:54.295330+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:17.230458+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":40,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.794865+0000","last_change":"2024-09-18T07:35:01.400514+0000","last_active":"2024-09-18T07:35:14.794865+0000","last_peered":"2024-09-18T07:35:14.794865+0000","last_clean":"2024-09-18T07:35:14.794865+0000","last_became_active":"2024-09-18T07:35:01.400197+0000","last_became_peered":"2024-09-18T07:35:01.400197+0000","last_unstale":"2024-09-18T07:35:14.794865+0000","last_undegraded":"2024-09-18T07:35:14.794865+0000","last_fullsized":"2024-09-18T07:35:14.794865+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:31.934890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.934890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:01.325843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":319,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.799825+0000","last_change":"2024-09-18T07:31:00.332729+0000","last_active":"2024-09-18T07:35:14.799825+0000","last_peered":"2024-09-18T07:35:14.799825+0000","last_clean":"2024-09-18T07:35:14.799825+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:35:14.799825+0000","last_undegraded":"2024-09-18T07:35:14.799825+0000","last_fullsized":"2024-09-18T07:35:14.799825+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:15.200098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.794119+0000","last_change":"2024-09-18T07:35:01.411256+0000","last_active":"2024-09-18T07:35:14.794119+0000","last_peered":"2024-09-18T07:35:14.794119+0000","last_clean":"2024-09-18T07:35:14.794119+0000","last_became_active":"2024-09-18T07:35:01.410561+0000","last_became_peered":"2024-09-18T07:35:01.410561+0000","last_unstale":"2024-09-18T07:35:14.794119+0000","last_undegraded":"2024-09-18T07:35:14.794119+0000","last_fullsized":"2024-09-18T07:35:14.794119+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.134847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.134847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":288,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.793117+0000","last_change":"2024-09-18T07:34:56.881850+0000","last_active":"2024-09-18T07:35:14.793117+0000","last_peered":"2024-09-18T07:35:14.793117+0000","last_clean":"2024-09-18T07:35:14.793117+0000","last_became_active":"2024-09-18T07:30:58.314428+0000","last_became_peered":"2024-09-18T07:30:58.314428+0000","last_unstale":"2024-09-18T07:35:14.793117+0000","last_undegraded":"2024-09-18T07:35:14.793117+0000","last_fullsized":"2024-09-18T07:35:14.793117+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:34:56.881769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:56.881769+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:01.109909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":279,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.793073+0000","last_change":"2024-09-18T07:35:01.401235+0000","last_active":"2024-09-18T07:35:14.793073+0000","last_peered":"2024-09-18T07:35:14.793073+0000","last_clean":"2024-09-18T07:35:14.793073+0000","last_became_active":"2024-09-18T07:35:01.400615+0000","last_became_peered":"2024-09-18T07:35:01.400615+0000","last_unstale":"2024-09-18T07:35:14.793073+0000","last_undegraded":"2024-09-18T07:35:14.793073+0000","last_fullsized":"2024-09-18T07:35:14.793073+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:12.005003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":257,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.798319+0000","last_change":"2024-09-18T07:31:00.337826+0000","last_active":"2024-09-18T07:35:14.798319+0000","last_peered":"2024-09-18T07:35:14.798319+0000","last_clean":"2024-09-18T07:35:14.798319+0000","last_became_active":"2024-09-18T07:31:00.334158+0000","last_became_peered":"2024-09-18T07:31:00.334158+0000","last_unstale":"2024-09-18T07:35:14.798319+0000","last_undegraded":"2024-09-18T07:35:14.798319+0000","last_fullsized":"2024-09-18T07:35:14.798319+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:44.061479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":252,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.807498+0000","last_change":"2024-09-18T07:31:02.351247+0000","last_active":"2024-09-18T07:35:14.807498+0000","last_peered":"2024-09-18T07:35:14.807498+0000","last_clean":"2024-09-18T07:35:14.807498+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:35:14.807498+0000","last_undegraded":"2024-09-18T07:35:14.807498+0000","last_fullsized":"2024-09-18T07:35:14.807498+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:12.115708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":155,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.791290+0000","last_change":"2024-09-18T07:33:27.033510+0000","last_active":"2024-09-18T07:35:14.791290+0000","last_peered":"2024-09-18T07:35:14.791290+0000","last_clean":"2024-09-18T07:35:14.791290+0000","last_became_active":"2024-09-18T07:33:27.003887+0000","last_became_peered":"2024-09-18T07:33:27.003887+0000","last_unstale":"2024-09-18T07:35:14.791290+0000","last_undegraded":"2024-09-18T07:35:14.791290+0000","last_fullsized":"2024-09-18T07:35:14.791290+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.187186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.187186+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":40,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.795791+0000","last_change":"2024-09-18T07:35:01.406178+0000","last_active":"2024-09-18T07:35:14.795791+0000","last_peered":"2024-09-18T07:35:14.795791+0000","last_clean":"2024-09-18T07:35:14.795791+0000","last_became_active":"2024-09-18T07:35:01.405287+0000","last_became_peered":"2024-09-18T07:35:01.405287+0000","last_unstale":"2024-09-18T07:35:14.795791+0000","last_undegraded":"2024-09-18T07:35:14.795791+0000","last_fullsized":"2024-09-18T07:35:14.795791+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:34.322659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:34.322659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:56.559071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":264,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.808811+0000","last_change":"2024-09-18T07:35:01.205647+0000","last_active":"2024-09-18T07:35:14.808811+0000","last_peered":"2024-09-18T07:35:14.808811+0000","last_clean":"2024-09-18T07:35:14.808811+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:35:14.808811+0000","last_undegraded":"2024-09-18T07:35:14.808811+0000","last_fullsized":"2024-09-18T07:35:14.808811+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:01.205596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:01.205596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:10.661983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":259,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.790995+0000","last_change":"2024-09-18T07:32:04.503356+0000","last_active":"2024-09-18T07:35:14.790995+0000","last_peered":"2024-09-18T07:35:14.790995+0000","last_clean":"2024-09-18T07:35:14.790995+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:35:14.790995+0000","last_undegraded":"2024-09-18T07:35:14.790995+0000","last_fullsized":"2024-09-18T07:35:14.790995+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:04.503270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:04.503270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":325,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.807692+0000","last_change":"2024-09-18T07:35:06.201799+0000","last_active":"2024-09-18T07:35:14.807692+0000","last_peered":"2024-09-18T07:35:14.807692+0000","last_clean":"2024-09-18T07:35:14.807692+0000","last_became_active":"2024-09-18T07:35:01.407576+0000","last_became_peered":"2024-09-18T07:35:01.407576+0000","last_unstale":"2024-09-18T07:35:14.807692+0000","last_undegraded":"2024-09-18T07:35:14.807692+0000","last_fullsized":"2024-09-18T07:35:14.807692+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:35:06.201725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:06.201725+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:24.293384+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":285,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.801202+0000","last_change":"2024-09-18T07:35:01.415691+0000","last_active":"2024-09-18T07:35:14.801202+0000","last_peered":"2024-09-18T07:35:14.801202+0000","last_clean":"2024-09-18T07:35:14.801202+0000","last_became_active":"2024-09-18T07:35:01.415261+0000","last_became_peered":"2024-09-18T07:35:01.415261+0000","last_unstale":"2024-09-18T07:35:14.801202+0000","last_undegraded":"2024-09-18T07:35:14.801202+0000","last_fullsized":"2024-09-18T07:35:14.801202+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:58.734027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:58.734027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:08.226648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":40,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.794165+0000","last_change":"2024-09-18T07:35:01.401087+0000","last_active":"2024-09-18T07:35:14.794165+0000","last_peered":"2024-09-18T07:35:14.794165+0000","last_clean":"2024-09-18T07:35:14.794165+0000","last_became_active":"2024-09-18T07:35:01.400714+0000","last_became_peered":"2024-09-18T07:35:01.400714+0000","last_unstale":"2024-09-18T07:35:14.794165+0000","last_undegraded":"2024-09-18T07:35:14.794165+0000","last_fullsized":"2024-09-18T07:35:14.794165+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.668428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.668428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":260,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.808881+0000","last_change":"2024-09-18T07:33:30.975087+0000","last_active":"2024-09-18T07:35:14.808881+0000","last_peered":"2024-09-18T07:35:14.808881+0000","last_clean":"2024-09-18T07:35:14.808881+0000","last_became_active":"2024-09-18T07:31:02.490994+0000","last_became_peered":"2024-09-18T07:31:02.490994+0000","last_unstale":"2024-09-18T07:35:14.808881+0000","last_undegraded":"2024-09-18T07:35:14.808881+0000","last_fullsized":"2024-09-18T07:35:14.808881+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.974886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.974886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":287,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.808342+0000","last_change":"2024-09-18T07:33:29.988707+0000","last_active":"2024-09-18T07:35:14.808342+0000","last_peered":"2024-09-18T07:35:14.808342+0000","last_clean":"2024-09-18T07:35:14.808342+0000","last_became_active":"2024-09-18T07:32:26.791562+0000","last_became_peered":"2024-09-18T07:32:26.791562+0000","last_unstale":"2024-09-18T07:35:14.808342+0000","last_undegraded":"2024-09-18T07:35:14.808342+0000","last_fullsized":"2024-09-18T07:35:14.808342+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:33:29.988636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:29.988636+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":263,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.793307+0000","last_change":"2024-09-18T07:30:56.283643+0000","last_active":"2024-09-18T07:35:14.793307+0000","last_peered":"2024-09-18T07:35:14.793307+0000","last_clean":"2024-09-18T07:35:14.793307+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:35:14.793307+0000","last_undegraded":"2024-09-18T07:35:14.793307+0000","last_fullsized":"2024-09-18T07:35:14.793307+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:52.458851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":285,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.798626+0000","last_change":"2024-09-18T07:35:02.394924+0000","last_active":"2024-09-18T07:35:14.798626+0000","last_peered":"2024-09-18T07:35:14.798626+0000","last_clean":"2024-09-18T07:35:14.798626+0000","last_became_active":"2024-09-18T07:35:01.407641+0000","last_became_peered":"2024-09-18T07:35:01.407641+0000","last_unstale":"2024-09-18T07:35:14.798626+0000","last_undegraded":"2024-09-18T07:35:14.798626+0000","last_fullsized":"2024-09-18T07:35:14.798626+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.394872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.394872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:29.732852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":40,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.794764+0000","last_change":"2024-09-18T07:35:01.402856+0000","last_active":"2024-09-18T07:35:14.794764+0000","last_peered":"2024-09-18T07:35:14.794764+0000","last_clean":"2024-09-18T07:35:14.794764+0000","last_became_active":"2024-09-18T07:35:01.400396+0000","last_became_peered":"2024-09-18T07:35:01.400396+0000","last_unstale":"2024-09-18T07:35:14.794764+0000","last_undegraded":"2024-09-18T07:35:14.794764+0000","last_fullsized":"2024-09-18T07:35:14.794764+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:36.017602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.017602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":269,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.799298+0000","last_change":"2024-09-18T07:35:02.071672+0000","last_active":"2024-09-18T07:35:14.799298+0000","last_peered":"2024-09-18T07:35:14.799298+0000","last_clean":"2024-09-18T07:35:14.799298+0000","last_became_active":"2024-09-18T07:31:02.347537+0000","last_became_peered":"2024-09-18T07:31:02.347537+0000","last_unstale":"2024-09-18T07:35:14.799298+0000","last_undegraded":"2024-09-18T07:35:14.799298+0000","last_fullsized":"2024-09-18T07:35:14.799298+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.071629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.071629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:17.310933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":278,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.808549+0000","last_change":"2024-09-18T07:33:33.072569+0000","last_active":"2024-09-18T07:35:14.808549+0000","last_peered":"2024-09-18T07:35:14.808549+0000","last_clean":"2024-09-18T07:35:14.808549+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:35:14.808549+0000","last_undegraded":"2024-09-18T07:35:14.808549+0000","last_fullsized":"2024-09-18T07:35:14.808549+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:33.072487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.072487+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":295,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.801943+0000","last_change":"2024-09-18T07:34:58.432839+0000","last_active":"2024-09-18T07:35:14.801943+0000","last_peered":"2024-09-18T07:35:14.801943+0000","last_clean":"2024-09-18T07:35:14.801943+0000","last_became_active":"2024-09-18T07:30:58.313556+0000","last_became_peered":"2024-09-18T07:30:58.313556+0000","last_unstale":"2024-09-18T07:35:14.801943+0000","last_undegraded":"2024-09-18T07:35:14.801943+0000","last_fullsized":"2024-09-18T07:35:14.801943+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:34:58.432781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:58.432781+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:13.006459+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":278,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-18T07:35:14.794628+0000","last_change":"2024-09-18T07:35:01.411222+0000","last_active":"2024-09-18T07:35:14.794628+0000","last_peered":"2024-09-18T07:35:14.794628+0000","last_clean":"2024-09-18T07:35:14.794628+0000","last_became_active":"2024-09-18T07:35:01.410469+0000","last_became_peered":"2024-09-18T07:35:01.410469+0000","last_unstale":"2024-09-18T07:35:14.794628+0000","last_undegraded":"2024-09-18T07:35:14.794628+0000","last_fullsized":"2024-09-18T07:35:14.794628+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.161894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.161894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2548695064,"num_objects":38891,"num_object_clones":0,"num_object_copies":116673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2466,"num_objects_unfound":0,"num_objects_dirty":38891,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":77380,"num_write_kb":2499585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15105,"num_bytes_recovered":989593600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7982559232,"data_stored":7982547016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39380,"ondisk_log_size":39380,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":2838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":16,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201336,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":759372,"kb_used_data":643508,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":92985780,"statfs":{"total":95995035648,"available":95217438720,"internally_reserved":0,"allocated":658952192,"data_stored":658306011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":118618570},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462974,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":886836,"kb_used_data":783460,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92858316,"statfs":{"total":95995035648,"available":95086915584,"internally_reserved":0,"allocated":802263040,"data_stored":801632182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724611,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744184,"kb_used_data":626912,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93000968,"statfs":{"total":95995035648,"available":95232991232,"internally_reserved":0,"allocated":641957888,"data_stored":641331014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953546,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1088008,"kb_used_data":970744,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92657144,"statfs":{"total":95995035648,"available":94880915456,"internally_reserved":0,"allocated":994041856,"data_stored":993390978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1773016,"kb_used_data":1601544,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":91972136,"statfs":{"total":95995035648,"available":94179467264,"internally_reserved":0,"allocated":1639981056,"data_stored":1639315086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":175503815},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1260904,"kb_used_data":1125528,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92484248,"statfs":{"total":95995035648,"available":94703869952,"internally_reserved":0,"allocated":1152540672,"data_stored":1151921779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1103144,"kb_used_data":959508,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92642008,"statfs":{"total":95995035648,"available":94865416192,"internally_reserved":0,"allocated":982536192,"data_stored":981858073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1392276,"kb_used_data":1256900,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92352876,"statfs":{"total":95995035648,"available":94569345024,"internally_reserved":0,"allocated":1287065600,"data_stored":1286404294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1271140352,"data_stored":1271136280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":955191296,"data_stored":955187224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1118830592,"data_stored":1118830592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1594490880,"data_stored":1594490880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":970588160,"data_stored":970588160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":633405440,"data_stored":633405440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":789385216,"data_stored":789381144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":649527296,"data_stored":649527296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:35:19.077 INFO:tasks.thrashosds.thrasher:Setting 6.4 to [7, 1, 6, 2, 3, 5] 2024-09-18T07:35:19.077 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.4', '7', '1', '6', '2', '3', '5'] 2024-09-18T07:35:19.078 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 6.4 7 1 6 2 3 5 2024-09-18T07:35:19.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:19.079 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:19.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:19.162 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:19.179 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:19.207 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:19.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:19.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:19.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:35:19.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:19.335 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:19.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:19.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:19.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:19.477 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:19.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:19.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:19.559 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:19 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:35:19.559 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:19 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:35:19.559 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:19 smithi137 ceph-mon[29408]: from='client.15162 v1:172.21.15.137:0/1044090931' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:35:19.559 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:19 smithi137 ceph-mon[29408]: osdmap e189: 8 total, 8 up, 8 in 2024-09-18T07:35:19.559 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:19 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:35:19.559 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:19 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:35:19.559 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:19 smithi137 ceph-mon[39175]: from='client.15162 v1:172.21.15.137:0/1044090931' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:35:19.559 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:19 smithi137 ceph-mon[39175]: osdmap e189: 8 total, 8 up, 8 in 2024-09-18T07:35:19.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:19.606 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:19.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:19.708 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:19.736 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:19.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:19.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:19.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:19 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:35:19.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:19 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:35:19.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:19 smithi203 ceph-mon[32237]: from='client.15162 v1:172.21.15.137:0/1044090931' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:35:19.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:19 smithi203 ceph-mon[32237]: osdmap e189: 8 total, 8 up, 8 in 2024-09-18T07:35:19.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T07:35:19.839 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:19.890 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:19.926 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:19.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:19.986 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:20.027 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:20.052 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:20.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:20.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:20.156 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:20.184 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:20.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:20.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:20.286 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:20.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:20.333 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:20.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:20.434 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:20.454 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:20.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:20.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:20.531 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:35:20.556 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:20.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:20.596 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:20.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:20.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:20 smithi137 ceph-mon[29408]: pgmap v236: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1021 B/s rd, 0 B/s wr, 971 op/s; 2466/117270 objects misplaced (2.103%); 14 MiB/s, 223 objects/s recovering 2024-09-18T07:35:20.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:20 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2300212229' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:35:20.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:20 smithi137 ceph-mon[39175]: pgmap v236: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1021 B/s rd, 0 B/s wr, 971 op/s; 2466/117270 objects misplaced (2.103%); 14 MiB/s, 223 objects/s recovering 2024-09-18T07:35:20.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:20 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2300212229' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:35:20.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:20.737 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:20.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:20.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:20.839 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:20.865 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:20.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:20.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:20.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:20 smithi203 ceph-mon[32237]: pgmap v236: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1021 B/s rd, 0 B/s wr, 971 op/s; 2466/117270 objects misplaced (2.103%); 14 MiB/s, 223 objects/s recovering 2024-09-18T07:35:20.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:20 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2300212229' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:35:20.967 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:21.003 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:21.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:21.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:21.105 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:21.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:21.135 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:21.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:21.236 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:21.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:21.280 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:21.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:21.381 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:21.408 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:21.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:21.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:21.510 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:21.533 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:21.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:21.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:21.573 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:21.623 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:35:21.623 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:35:21.636 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:21.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:21 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:21.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:21 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2300212229' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:35:21.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:21 smithi137 ceph-mon[39175]: osdmap e190: 8 total, 8 up, 8 in 2024-09-18T07:35:21.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:21 smithi137 ceph-mon[39175]: 6.10 scrub starts 2024-09-18T07:35:21.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:21 smithi137 ceph-mon[39175]: 6.10 scrub ok 2024-09-18T07:35:21.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:21 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:21.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:21 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2300212229' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:35:21.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:21 smithi137 ceph-mon[29408]: osdmap e190: 8 total, 8 up, 8 in 2024-09-18T07:35:21.672 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:21 smithi137 ceph-mon[29408]: 6.10 scrub starts 2024-09-18T07:35:21.672 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:21 smithi137 ceph-mon[29408]: 6.10 scrub ok 2024-09-18T07:35:21.674 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:35:21.674 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:35:21.696 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:21.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:21.712 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:35:21.712 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:35:21.750 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:35:21.750 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:35:21.781 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:35:21.781 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:35:21.799 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:21.814 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:35:21.814 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:35:21.826 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:21.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:21.845 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:35:21.846 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:35:21.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:21 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:21.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:21 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2300212229' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:35:21.861 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:21 smithi203 ceph-mon[32237]: osdmap e190: 8 total, 8 up, 8 in 2024-09-18T07:35:21.862 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:21 smithi203 ceph-mon[32237]: 6.10 scrub starts 2024-09-18T07:35:21.862 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:21 smithi203 ceph-mon[32237]: 6.10 scrub ok 2024-09-18T07:35:21.876 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:35:21.877 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:35:21.928 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:21.974 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:21.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:21.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:22.076 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:22.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:22.110 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:22.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:22.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:22.250 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:22.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:22.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:22.352 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:22.381 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:22.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:22.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:22.437 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:35:22.437 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:35:22.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:22.515 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:22.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:22.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:22.594 INFO:teuthology.orchestra.run.smithi137.stderr:set 6.4 pg_upmap_items mapping to [7->1,6->2,3->5] 2024-09-18T07:35:22.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:22.652 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:22.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:22.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:22.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:22 smithi203 ceph-mon[32237]: 7.a scrub starts 2024-09-18T07:35:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:22 smithi203 ceph-mon[32237]: 4.9 scrub starts 2024-09-18T07:35:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:22 smithi203 ceph-mon[32237]: 4.9 scrub ok 2024-09-18T07:35:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:22 smithi203 ceph-mon[32237]: 3.c scrub starts 2024-09-18T07:35:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:22 smithi203 ceph-mon[32237]: 3.c scrub ok 2024-09-18T07:35:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:22 smithi203 ceph-mon[32237]: 4.a scrub starts 2024-09-18T07:35:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:22 smithi203 ceph-mon[32237]: 4.a scrub ok 2024-09-18T07:35:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:22 smithi203 ceph-mon[32237]: pgmap v238: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.34k op/s; 2466/107763 objects misplaced (2.288%); 12 MiB/s, 186 objects/s recovering 2024-09-18T07:35:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:22 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1328216932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:22 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3496572453' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.4", "id": [7, 1, 6, 2, 3, 5]} : dispatch 2024-09-18T07:35:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:22 smithi203 ceph-mon[32237]: from='client.35066 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.4", "id": [7, 1, 6, 2, 3, 5]} : dispatch 2024-09-18T07:35:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:22 smithi203 ceph-mon[32237]: 7.a scrub ok 2024-09-18T07:35:22.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:22.793 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:22.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:22.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:22.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:22.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[29408]: 7.a scrub starts 2024-09-18T07:35:22.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[29408]: 4.9 scrub starts 2024-09-18T07:35:22.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[29408]: 4.9 scrub ok 2024-09-18T07:35:22.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[29408]: 3.c scrub starts 2024-09-18T07:35:22.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[29408]: 3.c scrub ok 2024-09-18T07:35:22.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[29408]: 4.a scrub starts 2024-09-18T07:35:22.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[29408]: 4.a scrub ok 2024-09-18T07:35:22.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[29408]: pgmap v238: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.34k op/s; 2466/107763 objects misplaced (2.288%); 12 MiB/s, 186 objects/s recovering 2024-09-18T07:35:22.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1328216932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:22.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3496572453' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.4", "id": [7, 1, 6, 2, 3, 5]} : dispatch 2024-09-18T07:35:22.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[29408]: from='client.35066 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.4", "id": [7, 1, 6, 2, 3, 5]} : dispatch 2024-09-18T07:35:22.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[29408]: 7.a scrub ok 2024-09-18T07:35:22.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[39175]: 7.a scrub starts 2024-09-18T07:35:22.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[39175]: 4.9 scrub starts 2024-09-18T07:35:22.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[39175]: 4.9 scrub ok 2024-09-18T07:35:22.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[39175]: 3.c scrub starts 2024-09-18T07:35:22.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[39175]: 3.c scrub ok 2024-09-18T07:35:22.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[39175]: 4.a scrub starts 2024-09-18T07:35:22.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[39175]: 4.a scrub ok 2024-09-18T07:35:22.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[39175]: pgmap v238: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.34k op/s; 2466/107763 objects misplaced (2.288%); 12 MiB/s, 186 objects/s recovering 2024-09-18T07:35:22.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1328216932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:22.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3496572453' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.4", "id": [7, 1, 6, 2, 3, 5]} : dispatch 2024-09-18T07:35:22.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[39175]: from='client.35066 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.4", "id": [7, 1, 6, 2, 3, 5]} : dispatch 2024-09-18T07:35:22.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:22 smithi137 ceph-mon[39175]: 7.a scrub ok 2024-09-18T07:35:22.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:22.930 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:22.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:23.031 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:23.067 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:23.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:23.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:23.168 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:23.201 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:23.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:23.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:23.303 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:23.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:23.338 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:23.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:23.440 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:23.475 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:23.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:23.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:23.576 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:23.613 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:23.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:23.714 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:23.755 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:23.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:23.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:23.857 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:23.881 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:23 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:35:23.881 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:23 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:35:23.882 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:23 smithi137 ceph-mon[29408]: from='client.35066 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.4", "id": [7, 1, 6, 2, 3, 5]}]': finished 2024-09-18T07:35:23.882 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:23 smithi137 ceph-mon[29408]: osdmap e191: 8 total, 8 up, 8 in 2024-09-18T07:35:23.882 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:23 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:35:23.882 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:23 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:35:23.882 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:23 smithi137 ceph-mon[39175]: from='client.35066 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.4", "id": [7, 1, 6, 2, 3, 5]}]': finished 2024-09-18T07:35:23.882 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:23 smithi137 ceph-mon[39175]: osdmap e191: 8 total, 8 up, 8 in 2024-09-18T07:35:23.900 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:23.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:23.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:23.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:23 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:35:23.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:23 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:35:23.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:23 smithi203 ceph-mon[32237]: from='client.35066 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.4", "id": [7, 1, 6, 2, 3, 5]}]': finished 2024-09-18T07:35:23.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:23 smithi203 ceph-mon[32237]: osdmap e191: 8 total, 8 up, 8 in 2024-09-18T07:35:24.002 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:24.032 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:24.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:24.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:24.134 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:24.170 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:24.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:24.272 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:24.299 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:24.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:24.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:24.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T07:35:24.401 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:24.430 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:24.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:24.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:24.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:24.578 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:24.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:24.680 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:24.709 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:24.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:24.810 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:24.837 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:24.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:24.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:24.863 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:24 smithi137 ceph-mon[29408]: pgmap v240: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 146 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 854 B/s rd, 0 B/s wr, 1.41k op/s; 1938/91956 objects misplaced (2.108%) 2024-09-18T07:35:24.863 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:24 smithi137 ceph-mon[29408]: 4.12 scrub starts 2024-09-18T07:35:24.863 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:24 smithi137 ceph-mon[29408]: 4.12 scrub ok 2024-09-18T07:35:24.864 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:24 smithi137 ceph-mon[39175]: pgmap v240: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 146 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 854 B/s rd, 0 B/s wr, 1.41k op/s; 1938/91956 objects misplaced (2.108%) 2024-09-18T07:35:24.864 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:24 smithi137 ceph-mon[39175]: 4.12 scrub starts 2024-09-18T07:35:24.864 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:24 smithi137 ceph-mon[39175]: 4.12 scrub ok 2024-09-18T07:35:24.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:24.946 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:24.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:24 smithi203 ceph-mon[32237]: pgmap v240: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 146 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 854 B/s rd, 0 B/s wr, 1.41k op/s; 1938/91956 objects misplaced (2.108%) 2024-09-18T07:35:24.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:24 smithi203 ceph-mon[32237]: 4.12 scrub starts 2024-09-18T07:35:24.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:24 smithi203 ceph-mon[32237]: 4.12 scrub ok 2024-09-18T07:35:24.969 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:24.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:24.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:25.070 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:25.099 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:25.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:25.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:25.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:25.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:25.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:25.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:25.339 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:25.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:25.370 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:25.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:25.379 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:35:25.471 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:25.497 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:25.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:25.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:25.585 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:35:25.585 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:35:25.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:25.624 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:25.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:25.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:25.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:25 smithi137 ceph-mon[29408]: 4.0 scrub starts 2024-09-18T07:35:25.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:25 smithi137 ceph-mon[29408]: 4.0 scrub ok 2024-09-18T07:35:25.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:25 smithi137 ceph-mon[29408]: 3.5 scrub starts 2024-09-18T07:35:25.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:25 smithi137 ceph-mon[29408]: 3.5 scrub ok 2024-09-18T07:35:25.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:25 smithi137 ceph-mon[39175]: 4.0 scrub starts 2024-09-18T07:35:25.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:25 smithi137 ceph-mon[39175]: 4.0 scrub ok 2024-09-18T07:35:25.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:25 smithi137 ceph-mon[39175]: 3.5 scrub starts 2024-09-18T07:35:25.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:25 smithi137 ceph-mon[39175]: 3.5 scrub ok 2024-09-18T07:35:25.726 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:25.754 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:25.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:25.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:25.856 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:25.883 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:35:25.895 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:25.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:25.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:25.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:25 smithi203 ceph-mon[32237]: 4.0 scrub starts 2024-09-18T07:35:25.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:25 smithi203 ceph-mon[32237]: 4.0 scrub ok 2024-09-18T07:35:25.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:25 smithi203 ceph-mon[32237]: 3.5 scrub starts 2024-09-18T07:35:25.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:25 smithi203 ceph-mon[32237]: 3.5 scrub ok 2024-09-18T07:35:25.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:26.043 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:26.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:26.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:26.145 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:26.202 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:26.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:26.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:26.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:26.344 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:26.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:26.445 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:26.475 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:26.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:26.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:26.577 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:26.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:26.599 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:26.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:26.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:26.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:26.737 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:26.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:26.838 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:26.863 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:26.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:26.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:26.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[29408]: 6.1e scrub starts 2024-09-18T07:35:26.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[29408]: 6.1e scrub ok 2024-09-18T07:35:26.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[29408]: 6.3 scrub starts 2024-09-18T07:35:26.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[29408]: 6.3 scrub ok 2024-09-18T07:35:26.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[29408]: pgmap v241: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 146 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 733 B/s rd, 0 B/s wr, 1.21k op/s; 1938/91956 objects misplaced (2.108%) 2024-09-18T07:35:26.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[29408]: 5.1e scrub starts 2024-09-18T07:35:26.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[29408]: 5.1e scrub ok 2024-09-18T07:35:26.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[29408]: 6.f scrub starts 2024-09-18T07:35:26.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[29408]: 6.f scrub ok 2024-09-18T07:35:26.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:35:26.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[39175]: 6.1e scrub starts 2024-09-18T07:35:26.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[39175]: 6.1e scrub ok 2024-09-18T07:35:26.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[39175]: 6.3 scrub starts 2024-09-18T07:35:26.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[39175]: 6.3 scrub ok 2024-09-18T07:35:26.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[39175]: pgmap v241: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 146 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 733 B/s rd, 0 B/s wr, 1.21k op/s; 1938/91956 objects misplaced (2.108%) 2024-09-18T07:35:26.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[39175]: 5.1e scrub starts 2024-09-18T07:35:26.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[39175]: 5.1e scrub ok 2024-09-18T07:35:26.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[39175]: 6.f scrub starts 2024-09-18T07:35:26.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[39175]: 6.f scrub ok 2024-09-18T07:35:26.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:35:26.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:26 smithi203 ceph-mon[32237]: 6.1e scrub starts 2024-09-18T07:35:26.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:26 smithi203 ceph-mon[32237]: 6.1e scrub ok 2024-09-18T07:35:26.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:26 smithi203 ceph-mon[32237]: 6.3 scrub starts 2024-09-18T07:35:26.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:26 smithi203 ceph-mon[32237]: 6.3 scrub ok 2024-09-18T07:35:26.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:26 smithi203 ceph-mon[32237]: pgmap v241: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 146 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 733 B/s rd, 0 B/s wr, 1.21k op/s; 1938/91956 objects misplaced (2.108%) 2024-09-18T07:35:26.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:26 smithi203 ceph-mon[32237]: 5.1e scrub starts 2024-09-18T07:35:26.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:26 smithi203 ceph-mon[32237]: 5.1e scrub ok 2024-09-18T07:35:26.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:26 smithi203 ceph-mon[32237]: 6.f scrub starts 2024-09-18T07:35:26.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:26 smithi203 ceph-mon[32237]: 6.f scrub ok 2024-09-18T07:35:26.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:35:26.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:27.004 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:27.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:27.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:27.106 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:27.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:27.137 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:27.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:27.238 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:27.278 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:27.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:27.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:27.379 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:27.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:27.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:27.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:27.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:35:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:35:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:35:27.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:35:27 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:35:27.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:27.552 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:27.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:27.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:27.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:27.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:27.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:27.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:27.791 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:27.816 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:27.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:27.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:27.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:27 smithi137 ceph-mon[29408]: 5.9 scrub starts 2024-09-18T07:35:27.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:27 smithi137 ceph-mon[29408]: 5.9 scrub ok 2024-09-18T07:35:27.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:27 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:35:27.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:27 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:35:27.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:27 smithi137 ceph-mon[39175]: 5.9 scrub starts 2024-09-18T07:35:27.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:27 smithi137 ceph-mon[39175]: 5.9 scrub ok 2024-09-18T07:35:27.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:27 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:35:27.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:27 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:35:27.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:27.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:27.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:27.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:27 smithi203 ceph-mon[32237]: 5.9 scrub starts 2024-09-18T07:35:27.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:27 smithi203 ceph-mon[32237]: 5.9 scrub ok 2024-09-18T07:35:27.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:27 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:35:27.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:27 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:35:28.060 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:28.095 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:28.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:28.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:28.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T07:35:28.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:28.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:28.259 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:28.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:28.288 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:28.295 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:28.361 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:28.387 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:28.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:28.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:28.488 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:28.513 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:28.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:28.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:28.615 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:28.641 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:28.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:28.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:28.726 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:28 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:28.726 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:28 smithi137 ceph-mon[29408]: 1.0 scrub starts 2024-09-18T07:35:28.726 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:28 smithi137 ceph-mon[29408]: 1.0 scrub ok 2024-09-18T07:35:28.726 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:28 smithi137 ceph-mon[29408]: pgmap v242: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.4 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2.08k op/s; 1375/67317 objects misplaced (2.043%) 2024-09-18T07:35:28.726 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:28 smithi137 ceph-mon[29408]: osdmap e192: 8 total, 8 up, 8 in 2024-09-18T07:35:28.727 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:28 smithi137 ceph-mon[29408]: 3.7 scrub starts 2024-09-18T07:35:28.727 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:28 smithi137 ceph-mon[29408]: 3.7 scrub ok 2024-09-18T07:35:28.727 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:28 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:28.727 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:28 smithi137 ceph-mon[39175]: 1.0 scrub starts 2024-09-18T07:35:28.727 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:28 smithi137 ceph-mon[39175]: 1.0 scrub ok 2024-09-18T07:35:28.727 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:28 smithi137 ceph-mon[39175]: pgmap v242: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.4 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2.08k op/s; 1375/67317 objects misplaced (2.043%) 2024-09-18T07:35:28.727 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:28 smithi137 ceph-mon[39175]: osdmap e192: 8 total, 8 up, 8 in 2024-09-18T07:35:28.727 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:28 smithi137 ceph-mon[39175]: 3.7 scrub starts 2024-09-18T07:35:28.727 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:28 smithi137 ceph-mon[39175]: 3.7 scrub ok 2024-09-18T07:35:28.743 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:28.771 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:28.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:28.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:28.873 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:28.930 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:28.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:28.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:28.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:28 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:28.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:28 smithi203 ceph-mon[32237]: 1.0 scrub starts 2024-09-18T07:35:28.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:28 smithi203 ceph-mon[32237]: 1.0 scrub ok 2024-09-18T07:35:28.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:28 smithi203 ceph-mon[32237]: pgmap v242: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.4 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2.08k op/s; 1375/67317 objects misplaced (2.043%) 2024-09-18T07:35:28.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:28 smithi203 ceph-mon[32237]: osdmap e192: 8 total, 8 up, 8 in 2024-09-18T07:35:28.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:28 smithi203 ceph-mon[32237]: 3.7 scrub starts 2024-09-18T07:35:28.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:28 smithi203 ceph-mon[32237]: 3.7 scrub ok 2024-09-18T07:35:29.032 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:29.059 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:29.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:29.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:29.161 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:29.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:29.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:29.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:29.292 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:29.319 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:29.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:29.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:29.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:29.454 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:29.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:29.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:29.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:29.587 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:29.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:29.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:29.689 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:29.716 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:29.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:29.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:29.805 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:35:29.806 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:35:29.818 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:29.830 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:35:29.830 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:35:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:29.860 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:29.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:29.866 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:35:29.866 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:35:29.866 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-18T07:35:29.867 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T07:35:29.867 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T07:35:29.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-18T07:35:29.868 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:35:29.869 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:35:29.869 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:29 smithi137 ceph-mon[29408]: 4.18 scrub starts 2024-09-18T07:35:29.870 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:29 smithi137 ceph-mon[29408]: 4.18 scrub ok 2024-09-18T07:35:29.870 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:29 smithi137 ceph-mon[29408]: 3.1f scrub starts 2024-09-18T07:35:29.870 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:29 smithi137 ceph-mon[29408]: 3.1f scrub ok 2024-09-18T07:35:29.870 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:29 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1115924803' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:29.870 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:29 smithi137 ceph-mon[29408]: from='client.25111 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:29.871 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:29 smithi137 ceph-mon[39175]: 4.18 scrub starts 2024-09-18T07:35:29.871 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:29 smithi137 ceph-mon[39175]: 4.18 scrub ok 2024-09-18T07:35:29.871 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:29 smithi137 ceph-mon[39175]: 3.1f scrub starts 2024-09-18T07:35:29.871 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:29 smithi137 ceph-mon[39175]: 3.1f scrub ok 2024-09-18T07:35:29.872 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:29 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1115924803' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:29.872 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:29 smithi137 ceph-mon[39175]: from='client.25111 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:29.902 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:35:29.902 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:35:29.937 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:35:29.937 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:35:29.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:29 smithi203 ceph-mon[32237]: 4.18 scrub starts 2024-09-18T07:35:29.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:29 smithi203 ceph-mon[32237]: 4.18 scrub ok 2024-09-18T07:35:29.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:29 smithi203 ceph-mon[32237]: 3.1f scrub starts 2024-09-18T07:35:29.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:29 smithi203 ceph-mon[32237]: 3.1f scrub ok 2024-09-18T07:35:29.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:29 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1115924803' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:29.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:29 smithi203 ceph-mon[32237]: from='client.25111 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:29.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:29.969 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:35:29.969 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:35:30.000 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:35:30.001 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:35:30.006 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:35:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:30.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:30.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:30.036 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:35:30.036 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:35:30.108 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:30.135 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:30.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:30.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:30.237 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:30.266 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:30.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:30.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:30.411 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:30.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:30.513 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:30.542 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:30.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:30.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:30.679 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:30.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:30.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:30.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:30.819 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:30.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:30.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:30.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[29408]: 6.12 scrub starts 2024-09-18T07:35:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[29408]: 6.12 scrub ok 2024-09-18T07:35:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[29408]: 5.1f scrub starts 2024-09-18T07:35:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[29408]: 5.1f scrub ok 2024-09-18T07:35:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[29408]: pgmap v244: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.4 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.69k op/s; 1375/67317 objects misplaced (2.043%) 2024-09-18T07:35:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:35:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[29408]: from='client.25111 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:35:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[29408]: osdmap e193: 8 total, 8 up, 8 in 2024-09-18T07:35:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/891535616' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[29408]: 5.8 scrub starts 2024-09-18T07:35:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[29408]: 5.8 scrub ok 2024-09-18T07:35:30.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[29408]: 4.10 scrub starts 2024-09-18T07:35:30.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[29408]: 4.10 scrub ok 2024-09-18T07:35:30.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[39175]: 6.12 scrub starts 2024-09-18T07:35:30.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[39175]: 6.12 scrub ok 2024-09-18T07:35:30.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[39175]: 5.1f scrub starts 2024-09-18T07:35:30.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[39175]: 5.1f scrub ok 2024-09-18T07:35:30.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[39175]: pgmap v244: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.4 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.69k op/s; 1375/67317 objects misplaced (2.043%) 2024-09-18T07:35:30.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:35:30.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[39175]: from='client.25111 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:35:30.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[39175]: osdmap e193: 8 total, 8 up, 8 in 2024-09-18T07:35:30.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/891535616' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:30.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[39175]: 5.8 scrub starts 2024-09-18T07:35:30.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[39175]: 5.8 scrub ok 2024-09-18T07:35:30.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[39175]: 4.10 scrub starts 2024-09-18T07:35:30.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:30 smithi137 ceph-mon[39175]: 4.10 scrub ok 2024-09-18T07:35:30.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:30.960 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:30 smithi203 ceph-mon[32237]: 6.12 scrub starts 2024-09-18T07:35:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:30 smithi203 ceph-mon[32237]: 6.12 scrub ok 2024-09-18T07:35:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:30 smithi203 ceph-mon[32237]: 5.1f scrub starts 2024-09-18T07:35:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:30 smithi203 ceph-mon[32237]: 5.1f scrub ok 2024-09-18T07:35:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:30 smithi203 ceph-mon[32237]: pgmap v244: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.4 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.69k op/s; 1375/67317 objects misplaced (2.043%) 2024-09-18T07:35:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:30 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:35:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:30 smithi203 ceph-mon[32237]: from='client.25111 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:35:30.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:30 smithi203 ceph-mon[32237]: osdmap e193: 8 total, 8 up, 8 in 2024-09-18T07:35:30.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:30 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/891535616' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:30.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:30 smithi203 ceph-mon[32237]: 5.8 scrub starts 2024-09-18T07:35:30.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:30 smithi203 ceph-mon[32237]: 5.8 scrub ok 2024-09-18T07:35:30.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:30 smithi203 ceph-mon[32237]: 4.10 scrub starts 2024-09-18T07:35:30.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:30 smithi203 ceph-mon[32237]: 4.10 scrub ok 2024-09-18T07:35:30.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:30.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:31.064 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:31.093 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:31.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:31.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:31.195 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:31.235 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:31.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:31.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:31.337 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:31.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:31.366 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:31.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:31.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:31.495 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:31.526 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:31.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:31.627 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:31.678 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:31.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:31.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:31.780 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:31.804 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:31 smithi137 ceph-mon[29408]: 5.e scrub starts 2024-09-18T07:35:31.804 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:31 smithi137 ceph-mon[29408]: 5.e scrub ok 2024-09-18T07:35:31.804 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:31 smithi137 ceph-mon[29408]: 6.1c scrub starts 2024-09-18T07:35:31.804 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:31 smithi137 ceph-mon[29408]: 6.1c scrub ok 2024-09-18T07:35:31.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:31 smithi137 ceph-mon[39175]: 5.e scrub starts 2024-09-18T07:35:31.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:31 smithi137 ceph-mon[39175]: 5.e scrub ok 2024-09-18T07:35:31.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:31 smithi137 ceph-mon[39175]: 6.1c scrub starts 2024-09-18T07:35:31.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:31 smithi137 ceph-mon[39175]: 6.1c scrub ok 2024-09-18T07:35:31.806 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:31.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:31.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:31.908 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:31.936 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:31.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:31.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:31.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:31 smithi203 ceph-mon[32237]: 5.e scrub starts 2024-09-18T07:35:31.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:31 smithi203 ceph-mon[32237]: 5.e scrub ok 2024-09-18T07:35:31.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:31 smithi203 ceph-mon[32237]: 6.1c scrub starts 2024-09-18T07:35:31.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:31 smithi203 ceph-mon[32237]: 6.1c scrub ok 2024-09-18T07:35:32.038 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:32.065 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:32.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:32.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:32.078 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:32.121 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:35:32.121 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:35:32.121 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:35:32.121 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:35:32.121 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.121 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134984 7.16 7:6e4eb25e:::benchmark_data_smithi137_74180_object30568:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:35:32.122 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964075+0000", 2024-09-18T07:35:32.122 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 75.148810069000007, 2024-09-18T07:35:32.122 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.253286275, 2024-09-18T07:35:32.122 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.122 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.122 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:35:32.122 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.122 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.122 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.122 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134984 2024-09-18T07:35:32.122 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.122 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.123 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964075+0000", 2024-09-18T07:35:32.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.123 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.123 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964075+0000", 2024-09-18T07:35:32.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.123 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.123 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964077+0000", 2024-09-18T07:35:32.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8870000000000001e-06 2024-09-18T07:35:32.124 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.124 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.124 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964083+0000", 2024-09-18T07:35:32.124 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4069999999999998e-06 2024-09-18T07:35:32.124 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.124 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.124 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964098+0000", 2024-09-18T07:35:32.124 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4119e-05 2024-09-18T07:35:32.124 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.124 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.125 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964108+0000", 2024-09-18T07:35:32.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0665999999999999e-05 2024-09-18T07:35:32.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.125 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.125 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964166+0000", 2024-09-18T07:35:32.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7819000000000003e-05 2024-09-18T07:35:32.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.125 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:35:32.125 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964174+0000", 2024-09-18T07:35:32.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5499999999999997e-06 2024-09-18T07:35:32.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.126 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.126 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.968251+0000", 2024-09-18T07:35:32.126 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004077448 2024-09-18T07:35:32.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.126 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.126 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.968261+0000", 2024-09-18T07:35:32.126 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0438000000000001e-05 2024-09-18T07:35:32.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.126 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.126 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.217288+0000", 2024-09-18T07:35:32.127 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.249026295 2024-09-18T07:35:32.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.127 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.127 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.217361+0000", 2024-09-18T07:35:32.127 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3646000000000006e-05 2024-09-18T07:35:32.127 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.127 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.127 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.127 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134991 7.19 7:9bc94497:::benchmark_data_smithi137_74180_object30575:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:35:32.127 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964132+0000", 2024-09-18T07:35:32.128 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 75.148752974000004, 2024-09-18T07:35:32.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2456100560000001, 2024-09-18T07:35:32.128 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.128 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.128 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:35:32.128 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.128 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.128 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.128 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134991 2024-09-18T07:35:32.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.128 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.128 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.129 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964132+0000", 2024-09-18T07:35:32.129 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.129 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.129 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.129 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964132+0000", 2024-09-18T07:35:32.129 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.129 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.129 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.129 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964133+0000", 2024-09-18T07:35:32.129 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4000000000000001e-07 2024-09-18T07:35:32.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.129 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.130 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.130 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964136+0000", 2024-09-18T07:35:32.130 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8009999999999999e-06 2024-09-18T07:35:32.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.130 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.130 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.130 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964141+0000", 2024-09-18T07:35:32.130 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.75e-06 2024-09-18T07:35:32.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.130 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.130 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.130 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964147+0000", 2024-09-18T07:35:32.130 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5439999999999998e-06 2024-09-18T07:35:32.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.130 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.131 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.131 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964319+0000", 2024-09-18T07:35:32.131 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017216000000000001 2024-09-18T07:35:32.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.131 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:35:32.131 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964324+0000", 2024-09-18T07:35:32.131 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.31e-06 2024-09-18T07:35:32.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.131 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.131 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.970873+0000", 2024-09-18T07:35:32.131 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065484250000000001 2024-09-18T07:35:32.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.970881+0000", 2024-09-18T07:35:32.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0600000000000008e-06 2024-09-18T07:35:32.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.201197+0000", 2024-09-18T07:35:32.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2303157170000001 2024-09-18T07:35:32.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-09-18T07:35:32.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.201321+0000", 2024-09-18T07:35:32.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012415000000000001 2024-09-18T07:35:32.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.133 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.133 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.209647+0000", 2024-09-18T07:35:32.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.008325931 2024-09-18T07:35:32.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.133 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.133 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.209742+0000", 2024-09-18T07:35:32.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5468000000000001e-05 2024-09-18T07:35:32.133 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.133 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.133 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.134 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135011 7.14 7:2c5d899c:::benchmark_data_smithi137_74180_object30595:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:35:32.134 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964161+0000", 2024-09-18T07:35:32.134 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 75.148723696999994, 2024-09-18T07:35:32.134 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2230384009999999, 2024-09-18T07:35:32.134 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.134 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.134 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:32.134 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.134 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.134 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.134 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135011 2024-09-18T07:35:32.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.134 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.134 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.134 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964161+0000", 2024-09-18T07:35:32.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.135 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964161+0000", 2024-09-18T07:35:32.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.135 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964162+0000", 2024-09-18T07:35:32.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2300000000000001e-07 2024-09-18T07:35:32.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.136 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.136 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964164+0000", 2024-09-18T07:35:32.136 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3690000000000001e-06 2024-09-18T07:35:32.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.136 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.136 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964169+0000", 2024-09-18T07:35:32.136 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.028e-06 2024-09-18T07:35:32.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.136 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.136 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964174+0000", 2024-09-18T07:35:32.136 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7809999999999996e-06 2024-09-18T07:35:32.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.137 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964271+0000", 2024-09-18T07:35:32.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6997000000000007e-05 2024-09-18T07:35:32.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:35:32.137 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964277+0000", 2024-09-18T07:35:32.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4369999999999998e-06 2024-09-18T07:35:32.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.969538+0000", 2024-09-18T07:35:32.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052614929999999999 2024-09-18T07:35:32.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.969546+0000", 2024-09-18T07:35:32.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9540000000000004e-06 2024-09-18T07:35:32.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.178869+0000", 2024-09-18T07:35:32.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2093228570000001 2024-09-18T07:35:32.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:32.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.179001+0000", 2024-09-18T07:35:32.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013228400000000001 2024-09-18T07:35:32.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2,3,4,5", 2024-09-18T07:35:32.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.180534+0000", 2024-09-18T07:35:32.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0015324399999999999 2024-09-18T07:35:32.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:32.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.183783+0000", 2024-09-18T07:35:32.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032497559999999999 2024-09-18T07:35:32.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.140 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184155+0000", 2024-09-18T07:35:32.140 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000371388 2024-09-18T07:35:32.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.140 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184621+0000", 2024-09-18T07:35:32.140 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00046647700000000003 2024-09-18T07:35:32.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.141 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185211+0000", 2024-09-18T07:35:32.141 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00058937699999999998 2024-09-18T07:35:32.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.141 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.141 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.141 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185835+0000", 2024-09-18T07:35:32.141 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000624269 2024-09-18T07:35:32.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.141 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.141 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.141 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186879+0000", 2024-09-18T07:35:32.141 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010440429999999999 2024-09-18T07:35:32.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186966+0000", 2024-09-18T07:35:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7031000000000001e-05 2024-09-18T07:35:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.187200+0000", 2024-09-18T07:35:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000233797 2024-09-18T07:35:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135015 7.17 7:e8fec392:::benchmark_data_smithi137_74180_object30599:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:35:32.143 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964189+0000", 2024-09-18T07:35:32.143 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 75.148695666999998, 2024-09-18T07:35:32.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.277466437, 2024-09-18T07:35:32.143 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.143 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.143 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:32.143 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.143 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.143 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.143 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135015 2024-09-18T07:35:32.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.143 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.143 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964189+0000", 2024-09-18T07:35:32.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964189+0000", 2024-09-18T07:35:32.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964190+0000", 2024-09-18T07:35:32.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7199999999999998e-07 2024-09-18T07:35:32.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.145 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964193+0000", 2024-09-18T07:35:32.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5670000000000002e-06 2024-09-18T07:35:32.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.145 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964199+0000", 2024-09-18T07:35:32.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8200000000000002e-06 2024-09-18T07:35:32.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964203+0000", 2024-09-18T07:35:32.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9189999999999998e-06 2024-09-18T07:35:32.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964356+0000", 2024-09-18T07:35:32.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015237200000000001 2024-09-18T07:35:32.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:35:32.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964361+0000", 2024-09-18T07:35:32.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1499999999999998e-06 2024-09-18T07:35:32.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.147 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971479+0000", 2024-09-18T07:35:32.147 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.007117511 2024-09-18T07:35:32.147 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.147 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.147 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.147 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971487+0000", 2024-09-18T07:35:32.147 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2800000000000003e-06 2024-09-18T07:35:32.147 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.147 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.147 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.147 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.224196+0000", 2024-09-18T07:35:32.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2527092280000001 2024-09-18T07:35:32.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.148 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:32.148 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.224324+0000", 2024-09-18T07:35:32.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000127582 2024-09-18T07:35:32.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.148 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2,3,5,6", 2024-09-18T07:35:32.148 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.224588+0000", 2024-09-18T07:35:32.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00026444499999999998 2024-09-18T07:35:32.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.148 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.149 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.230126+0000", 2024-09-18T07:35:32.149 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055381579999999996 2024-09-18T07:35:32.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.149 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:32.149 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.230140+0000", 2024-09-18T07:35:32.149 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4e-05 2024-09-18T07:35:32.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.150 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.150 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.230258+0000", 2024-09-18T07:35:32.150 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118002 2024-09-18T07:35:32.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.150 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.150 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.232997+0000", 2024-09-18T07:35:32.150 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027387829999999998 2024-09-18T07:35:32.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.233523+0000", 2024-09-18T07:35:32.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052651200000000003 2024-09-18T07:35:32.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.241477+0000", 2024-09-18T07:35:32.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079535379999999996 2024-09-18T07:35:32.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.152 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:32.152 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.241534+0000", 2024-09-18T07:35:32.152 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6906e-05 2024-09-18T07:35:32.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.152 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.152 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.241656+0000", 2024-09-18T07:35:32.152 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012199199999999999 2024-09-18T07:35:32.152 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.152 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.152 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.153 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135019 7.17 7:ec2155c5:::benchmark_data_smithi137_74180_object30603:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:35:32.153 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964215+0000", 2024-09-18T07:35:32.153 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 75.148670275000001, 2024-09-18T07:35:32.153 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2831853689999999, 2024-09-18T07:35:32.153 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.153 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.153 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:32.153 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.153 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.153 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.153 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135019 2024-09-18T07:35:32.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.154 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.154 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.154 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.154 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964215+0000", 2024-09-18T07:35:32.154 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.154 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.154 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.154 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.154 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964215+0000", 2024-09-18T07:35:32.154 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.154 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964216+0000", 2024-09-18T07:35:32.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3499999999999995e-07 2024-09-18T07:35:32.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964218+0000", 2024-09-18T07:35:32.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5790000000000002e-06 2024-09-18T07:35:32.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964224+0000", 2024-09-18T07:35:32.156 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3689999999999998e-06 2024-09-18T07:35:32.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.156 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964228+0000", 2024-09-18T07:35:32.156 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3560000000000003e-06 2024-09-18T07:35:32.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.156 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964383+0000", 2024-09-18T07:35:32.156 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000155217 2024-09-18T07:35:32.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.157 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-18T07:35:32.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964388+0000", 2024-09-18T07:35:32.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8550000000000001e-06 2024-09-18T07:35:32.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.157 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971509+0000", 2024-09-18T07:35:32.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0071211800000000004 2024-09-18T07:35:32.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.157 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971515+0000", 2024-09-18T07:35:32.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.224e-06 2024-09-18T07:35:32.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.158 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.158 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.158 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.226033+0000", 2024-09-18T07:35:32.158 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2545179440000001 2024-09-18T07:35:32.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.158 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.158 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:32.158 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.226140+0000", 2024-09-18T07:35:32.158 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010681200000000001 2024-09-18T07:35:32.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.158 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.158 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2,3,5,6", 2024-09-18T07:35:32.159 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.226373+0000", 2024-09-18T07:35:32.159 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023260699999999999 2024-09-18T07:35:32.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.159 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.159 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.230206+0000", 2024-09-18T07:35:32.159 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038327439999999999 2024-09-18T07:35:32.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.159 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.159 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.233570+0000", 2024-09-18T07:35:32.159 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033643129999999999 2024-09-18T07:35:32.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.233605+0000", 2024-09-18T07:35:32.160 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5290999999999998e-05 2024-09-18T07:35:32.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:32.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.241382+0000", 2024-09-18T07:35:32.160 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0077770239999999996 2024-09-18T07:35:32.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.241433+0000", 2024-09-18T07:35:32.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1044000000000001e-05 2024-09-18T07:35:32.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.247229+0000", 2024-09-18T07:35:32.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057953609999999997 2024-09-18T07:35:32.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:32.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.247282+0000", 2024-09-18T07:35:32.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3084999999999997e-05 2024-09-18T07:35:32.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.162 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.247400+0000", 2024-09-18T07:35:32.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011862900000000001 2024-09-18T07:35:32.162 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.162 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.162 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.162 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:159441 7.4 7:2597f405:::benchmark_data_smithi137_74180_object55025:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e167)", 2024-09-18T07:35:32.162 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:55.835150+0000", 2024-09-18T07:35:32.162 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 36.277735110000002, 2024-09-18T07:35:32.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0271538179999999, 2024-09-18T07:35:32.163 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.163 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.163 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:35:32.163 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.163 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.163 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.163 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 159441 2024-09-18T07:35:32.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.163 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.163 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.163 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835150+0000", 2024-09-18T07:35:32.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835150+0000", 2024-09-18T07:35:32.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835153+0000", 2024-09-18T07:35:32.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.553e-06 2024-09-18T07:35:32.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835160+0000", 2024-09-18T07:35:32.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4780000000000002e-06 2024-09-18T07:35:32.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835175+0000", 2024-09-18T07:35:32.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4912999999999999e-05 2024-09-18T07:35:32.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835186+0000", 2024-09-18T07:35:32.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0993e-05 2024-09-18T07:35:32.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.166 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858208+0000", 2024-09-18T07:35:32.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.023021823 2024-09-18T07:35:32.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.166 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.166 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858216+0000", 2024-09-18T07:35:32.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8709999999999997e-06 2024-09-18T07:35:32.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.166 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.166 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.862230+0000", 2024-09-18T07:35:32.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0040140200000001 2024-09-18T07:35:32.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.167 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.167 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.862304+0000", 2024-09-18T07:35:32.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4166999999999996e-05 2024-09-18T07:35:32.167 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.167 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.167 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.167 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:159490 7.4 7:2165327a:::benchmark_data_smithi137_74180_object55074:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e167)", 2024-09-18T07:35:32.167 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:55.835200+0000", 2024-09-18T07:35:32.167 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 36.277684991000001, 2024-09-18T07:35:32.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.043831822, 2024-09-18T07:35:32.167 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.167 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.168 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:32.168 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.168 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.168 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.168 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 159490 2024-09-18T07:35:32.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.168 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.168 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.168 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835200+0000", 2024-09-18T07:35:32.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.169 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835200+0000", 2024-09-18T07:35:32.169 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.169 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835201+0000", 2024-09-18T07:35:32.169 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8899999999999999e-07 2024-09-18T07:35:32.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.169 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835203+0000", 2024-09-18T07:35:32.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.633e-06 2024-09-18T07:35:32.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.170 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.170 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835207+0000", 2024-09-18T07:35:32.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8e-06 2024-09-18T07:35:32.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.170 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.170 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835211+0000", 2024-09-18T07:35:32.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3890000000000001e-06 2024-09-18T07:35:32.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858240+0000", 2024-09-18T07:35:32.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.023029287999999998 2024-09-18T07:35:32.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858244+0000", 2024-09-18T07:35:32.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3420000000000001e-06 2024-09-18T07:35:32.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.862671+0000", 2024-09-18T07:35:32.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.004426722 2024-09-18T07:35:32.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.172 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:32.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.862770+0000", 2024-09-18T07:35:32.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8945999999999998e-05 2024-09-18T07:35:32.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.172 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:35:32.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.862846+0000", 2024-09-18T07:35:32.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6496999999999996e-05 2024-09-18T07:35:32.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.172 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.173 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.864210+0000", 2024-09-18T07:35:32.173 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001363483 2024-09-18T07:35:32.173 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.173 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:32.173 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.864222+0000", 2024-09-18T07:35:32.173 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1751e-05 2024-09-18T07:35:32.173 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.173 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.173 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.878915+0000", 2024-09-18T07:35:32.173 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014693606999999999 2024-09-18T07:35:32.173 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:32.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.878972+0000", 2024-09-18T07:35:32.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6808000000000002e-05 2024-09-18T07:35:32.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.879032+0000", 2024-09-18T07:35:32.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9966999999999997e-05 2024-09-18T07:35:32.174 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.174 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.174 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.175 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:159519 7.4 7:27d593cd:::benchmark_data_smithi137_74180_object55103:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e167)", 2024-09-18T07:35:32.175 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:55.835214+0000", 2024-09-18T07:35:32.175 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 36.277670792000002, 2024-09-18T07:35:32.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.043949504, 2024-09-18T07:35:32.175 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.175 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.175 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:32.175 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.175 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.175 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.175 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 159519 2024-09-18T07:35:32.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.175 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.176 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835214+0000", 2024-09-18T07:35:32.176 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.176 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.176 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835214+0000", 2024-09-18T07:35:32.176 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.176 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.176 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835215+0000", 2024-09-18T07:35:32.176 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1800000000000002e-07 2024-09-18T07:35:32.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.177 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.177 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835216+0000", 2024-09-18T07:35:32.177 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4270000000000001e-06 2024-09-18T07:35:32.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.177 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.177 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835219+0000", 2024-09-18T07:35:32.177 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5979999999999999e-06 2024-09-18T07:35:32.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.177 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.177 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835222+0000", 2024-09-18T07:35:32.177 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.79e-06 2024-09-18T07:35:32.178 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.178 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.178 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858266+0000", 2024-09-18T07:35:32.178 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.023044212000000001 2024-09-18T07:35:32.178 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.178 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.178 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858269+0000", 2024-09-18T07:35:32.178 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7349999999999998e-06 2024-09-18T07:35:32.178 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.178 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.179 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.863052+0000", 2024-09-18T07:35:32.179 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0047829589999999 2024-09-18T07:35:32.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.179 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.179 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:32.179 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.863114+0000", 2024-09-18T07:35:32.179 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1635000000000002e-05 2024-09-18T07:35:32.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.179 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.179 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:35:32.179 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.863210+0000", 2024-09-18T07:35:32.179 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6051000000000006e-05 2024-09-18T07:35:32.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.179 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.180 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:32.180 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.864227+0000", 2024-09-18T07:35:32.180 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001016823 2024-09-18T07:35:32.180 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.180 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.180 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.180 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.864255+0000", 2024-09-18T07:35:32.180 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8351999999999999e-05 2024-09-18T07:35:32.180 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.180 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.180 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.180 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.879081+0000", 2024-09-18T07:35:32.181 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014826186 2024-09-18T07:35:32.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.181 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:32.181 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.879124+0000", 2024-09-18T07:35:32.181 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2596999999999999e-05 2024-09-18T07:35:32.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.181 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.181 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.879164+0000", 2024-09-18T07:35:32.181 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9820999999999997e-05 2024-09-18T07:35:32.181 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.182 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.182 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.182 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.182 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:159546 7.4 7:20b8eec8:::benchmark_data_smithi137_74180_object55130:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e167)", 2024-09-18T07:35:32.182 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:55.835224+0000", 2024-09-18T07:35:32.182 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 36.277660689000001, 2024-09-18T07:35:32.182 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0443191359999999, 2024-09-18T07:35:32.182 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.182 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.182 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:32.182 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.183 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.183 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.183 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 159546 2024-09-18T07:35:32.183 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.183 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.183 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.183 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.183 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835224+0000", 2024-09-18T07:35:32.183 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.183 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.183 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.183 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.183 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835224+0000", 2024-09-18T07:35:32.184 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.184 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.184 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.184 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835225+0000", 2024-09-18T07:35:32.184 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7700000000000001e-07 2024-09-18T07:35:32.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.184 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.184 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.184 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835226+0000", 2024-09-18T07:35:32.184 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3289999999999999e-06 2024-09-18T07:35:32.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.185 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.185 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.185 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835229+0000", 2024-09-18T07:35:32.185 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.694e-06 2024-09-18T07:35:32.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.185 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.185 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.185 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835231+0000", 2024-09-18T07:35:32.185 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7250000000000002e-06 2024-09-18T07:35:32.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.185 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.185 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.186 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858289+0000", 2024-09-18T07:35:32.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.023057667 2024-09-18T07:35:32.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.186 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.186 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858293+0000", 2024-09-18T07:35:32.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.659e-06 2024-09-18T07:35:32.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.186 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.186 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.863374+0000", 2024-09-18T07:35:32.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0050816549999999 2024-09-18T07:35:32.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.187 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.187 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:32.187 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.863440+0000", 2024-09-18T07:35:32.187 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5844000000000002e-05 2024-09-18T07:35:32.187 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.187 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.187 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:35:32.187 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.863496+0000", 2024-09-18T07:35:32.187 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5699999999999999e-05 2024-09-18T07:35:32.187 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.187 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.187 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.187 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.864312+0000", 2024-09-18T07:35:32.188 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00081550399999999999 2024-09-18T07:35:32.188 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.188 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.188 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:32.188 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.864323+0000", 2024-09-18T07:35:32.188 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.147e-05 2024-09-18T07:35:32.188 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.188 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.188 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.188 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.879443+0000", 2024-09-18T07:35:32.188 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015119654 2024-09-18T07:35:32.188 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.189 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:32.189 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.879495+0000", 2024-09-18T07:35:32.189 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2200000000000002e-05 2024-09-18T07:35:32.189 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.189 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.189 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.879544+0000", 2024-09-18T07:35:32.189 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8758000000000003e-05 2024-09-18T07:35:32.189 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.189 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.189 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.190 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.190 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.190 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:159599 7.4 7:2714517d:::benchmark_data_smithi137_74180_object55183:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e167)", 2024-09-18T07:35:32.190 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:55.835234+0000", 2024-09-18T07:35:32.190 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 36.277650940000001, 2024-09-18T07:35:32.190 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.044562599, 2024-09-18T07:35:32.190 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.190 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.190 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:32.190 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.190 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.190 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.190 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 159599 2024-09-18T07:35:32.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.191 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.191 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.191 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835234+0000", 2024-09-18T07:35:32.191 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.191 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.191 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835234+0000", 2024-09-18T07:35:32.191 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.191 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.192 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835234+0000", 2024-09-18T07:35:32.192 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3999999999999998e-07 2024-09-18T07:35:32.192 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.192 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.192 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.192 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835236+0000", 2024-09-18T07:35:32.192 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.223e-06 2024-09-18T07:35:32.192 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.192 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.192 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.192 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835239+0000", 2024-09-18T07:35:32.192 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5669999999999999e-06 2024-09-18T07:35:32.193 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.193 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.193 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.193 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835242+0000", 2024-09-18T07:35:32.193 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6079999999999998e-06 2024-09-18T07:35:32.193 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.193 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.193 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.193 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858313+0000", 2024-09-18T07:35:32.193 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.023070938999999999 2024-09-18T07:35:32.193 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.193 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.193 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.194 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858316+0000", 2024-09-18T07:35:32.194 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6349999999999999e-06 2024-09-18T07:35:32.194 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.194 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.194 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.194 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.863780+0000", 2024-09-18T07:35:32.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:32.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.005463953 2024-09-18T07:35:32.196 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.196 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.196 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:32.196 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.863888+0000", 2024-09-18T07:35:32.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107336 2024-09-18T07:35:32.196 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.196 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.196 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:35:32.196 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.864004+0000", 2024-09-18T07:35:32.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116069 2024-09-18T07:35:32.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.197 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:32.197 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.865120+0000", 2024-09-18T07:35:32.197 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011158190000000001 2024-09-18T07:35:32.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.197 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.197 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.865157+0000", 2024-09-18T07:35:32.197 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7020000000000001e-05 2024-09-18T07:35:32.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.197 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.198 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.879683+0000", 2024-09-18T07:35:32.198 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014526170999999999 2024-09-18T07:35:32.198 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.198 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.198 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:32.198 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.879732+0000", 2024-09-18T07:35:32.198 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9254e-05 2024-09-18T07:35:32.198 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.198 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.198 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.198 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.879797+0000", 2024-09-18T07:35:32.198 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4764999999999994e-05 2024-09-18T07:35:32.198 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.199 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.199 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.199 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.199 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.199 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:159604 7.4 7:26909147:::benchmark_data_smithi137_74180_object55188:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e167)", 2024-09-18T07:35:32.199 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:55.835245+0000", 2024-09-18T07:35:32.199 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 36.277640648999999, 2024-09-18T07:35:32.199 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.04470639, 2024-09-18T07:35:32.199 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.199 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.199 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:32.199 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.199 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.199 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.200 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 159604 2024-09-18T07:35:32.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.200 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.200 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.200 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835245+0000", 2024-09-18T07:35:32.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.200 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.200 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835245+0000", 2024-09-18T07:35:32.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.201 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.201 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835245+0000", 2024-09-18T07:35:32.201 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4999999999999999e-07 2024-09-18T07:35:32.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.201 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.201 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835246+0000", 2024-09-18T07:35:32.201 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2190000000000001e-06 2024-09-18T07:35:32.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.202 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.202 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835249+0000", 2024-09-18T07:35:32.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.348e-06 2024-09-18T07:35:32.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.202 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.202 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835253+0000", 2024-09-18T07:35:32.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6610000000000002e-06 2024-09-18T07:35:32.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.202 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.202 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858336+0000", 2024-09-18T07:35:32.203 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.023082910000000002 2024-09-18T07:35:32.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.203 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.203 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858340+0000", 2024-09-18T07:35:32.203 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6100000000000002e-06 2024-09-18T07:35:32.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.203 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.203 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.864357+0000", 2024-09-18T07:35:32.203 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0060173379999999 2024-09-18T07:35:32.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:32.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.864438+0000", 2024-09-18T07:35:32.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1063000000000006e-05 2024-09-18T07:35:32.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:35:32.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.864494+0000", 2024-09-18T07:35:32.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5991e-05 2024-09-18T07:35:32.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:32.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.865167+0000", 2024-09-18T07:35:32.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00067285299999999995 2024-09-18T07:35:32.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.205 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.205 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.205 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.866021+0000", 2024-09-18T07:35:32.205 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00085423599999999997 2024-09-18T07:35:32.205 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.205 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.205 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.205 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.879845+0000", 2024-09-18T07:35:32.205 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013824217999999999 2024-09-18T07:35:32.205 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.205 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.205 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:32.205 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.879899+0000", 2024-09-18T07:35:32.206 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4008000000000002e-05 2024-09-18T07:35:32.206 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.206 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.206 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.206 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.879951+0000", 2024-09-18T07:35:32.206 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1684999999999997e-05 2024-09-18T07:35:32.206 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.206 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.206 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.206 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.206 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.206 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:159625 7.4 7:2663779b:::benchmark_data_smithi137_74180_object55209:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e167)", 2024-09-18T07:35:32.206 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:55.835256+0000", 2024-09-18T07:35:32.206 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 36.277629382000001, 2024-09-18T07:35:32.207 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0448507709999999, 2024-09-18T07:35:32.207 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.207 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.207 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:32.207 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.207 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.207 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.207 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 159625 2024-09-18T07:35:32.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.207 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.207 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.208 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.208 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835256+0000", 2024-09-18T07:35:32.208 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.208 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.208 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.208 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.208 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835256+0000", 2024-09-18T07:35:32.208 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.208 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.208 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.208 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.208 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835256+0000", 2024-09-18T07:35:32.208 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5400000000000002e-07 2024-09-18T07:35:32.209 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.209 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.209 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.209 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835257+0000", 2024-09-18T07:35:32.209 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3149999999999999e-06 2024-09-18T07:35:32.209 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.209 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.209 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.209 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835260+0000", 2024-09-18T07:35:32.209 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8210000000000002e-06 2024-09-18T07:35:32.209 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.209 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.209 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.209 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835263+0000", 2024-09-18T07:35:32.210 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7209999999999999e-06 2024-09-18T07:35:32.210 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.210 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.210 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.210 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858359+0000", 2024-09-18T07:35:32.210 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.02309576 2024-09-18T07:35:32.210 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.210 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.210 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.210 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858362+0000", 2024-09-18T07:35:32.210 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6890000000000002e-06 2024-09-18T07:35:32.210 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.210 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.211 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.211 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.864727+0000", 2024-09-18T07:35:32.211 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0063649960000001 2024-09-18T07:35:32.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.211 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.211 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:32.211 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.864824+0000", 2024-09-18T07:35:32.211 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6815000000000001e-05 2024-09-18T07:35:32.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.211 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.211 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:35:32.211 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.864938+0000", 2024-09-18T07:35:32.211 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113755 2024-09-18T07:35:32.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.212 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.212 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:32.212 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.866966+0000", 2024-09-18T07:35:32.212 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020285339999999998 2024-09-18T07:35:32.212 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.212 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.212 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.212 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.867400+0000", 2024-09-18T07:35:32.212 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000433282 2024-09-18T07:35:32.212 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.212 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.212 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.212 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.880007+0000", 2024-09-18T07:35:32.213 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012607115 2024-09-18T07:35:32.213 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.213 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.213 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:32.213 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.880056+0000", 2024-09-18T07:35:32.213 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9453000000000002e-05 2024-09-18T07:35:32.213 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.213 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.213 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.213 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.880107+0000", 2024-09-18T07:35:32.213 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0260999999999999e-05 2024-09-18T07:35:32.213 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.213 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.214 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.214 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.214 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.214 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:159650 7.4 7:21ef5e2e:::benchmark_data_smithi137_74180_object55234:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e167)", 2024-09-18T07:35:32.214 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:55.835266+0000", 2024-09-18T07:35:32.214 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 36.277619623, 2024-09-18T07:35:32.214 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0451569730000001, 2024-09-18T07:35:32.214 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.214 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.214 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:32.214 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.214 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.214 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.214 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 159650 2024-09-18T07:35:32.215 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.215 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.215 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.215 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.215 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835266+0000", 2024-09-18T07:35:32.215 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.215 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.215 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.215 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.215 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835266+0000", 2024-09-18T07:35:32.215 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.215 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.215 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.215 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.216 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835266+0000", 2024-09-18T07:35:32.216 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.53e-07 2024-09-18T07:35:32.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.216 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.216 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.216 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835268+0000", 2024-09-18T07:35:32.216 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7460000000000001e-06 2024-09-18T07:35:32.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.216 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.216 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.216 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835271+0000", 2024-09-18T07:35:32.216 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0130000000000002e-06 2024-09-18T07:35:32.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.216 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.217 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.217 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835273+0000", 2024-09-18T07:35:32.217 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6070000000000002e-06 2024-09-18T07:35:32.217 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.217 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.217 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.217 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858382+0000", 2024-09-18T07:35:32.217 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.023108483999999999 2024-09-18T07:35:32.217 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.217 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.217 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.217 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858385+0000", 2024-09-18T07:35:32.217 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5760000000000002e-06 2024-09-18T07:35:32.218 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.218 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.218 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.218 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.865191+0000", 2024-09-18T07:35:32.218 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0068062170000001 2024-09-18T07:35:32.218 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.218 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.218 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:32.218 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.865283+0000", 2024-09-18T07:35:32.218 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1305999999999996e-05 2024-09-18T07:35:32.218 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.218 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.218 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:35:32.219 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.865379+0000", 2024-09-18T07:35:32.219 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5790000000000003e-05 2024-09-18T07:35:32.219 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.219 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.219 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:32.219 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.866972+0000", 2024-09-18T07:35:32.219 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0015932800000000001 2024-09-18T07:35:32.219 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.219 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.219 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.219 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.867433+0000", 2024-09-18T07:35:32.219 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000461225 2024-09-18T07:35:32.219 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.219 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.220 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.220 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.880333+0000", 2024-09-18T07:35:32.220 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01289973 2024-09-18T07:35:32.220 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.220 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.220 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:32.220 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.880378+0000", 2024-09-18T07:35:32.220 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4808999999999997e-05 2024-09-18T07:35:32.220 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.220 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.220 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.220 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.880423+0000", 2024-09-18T07:35:32.220 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4937e-05 2024-09-18T07:35:32.220 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.221 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.221 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.221 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:159687 7.4 7:278452ad:::benchmark_data_smithi137_74180_object55271:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e167)", 2024-09-18T07:35:32.221 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:55.835276+0000", 2024-09-18T07:35:32.221 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 36.277609423000001, 2024-09-18T07:35:32.221 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0452823739999999, 2024-09-18T07:35:32.221 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.221 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.221 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:32.221 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.221 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.221 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.222 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 159687 2024-09-18T07:35:32.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.222 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.222 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835276+0000", 2024-09-18T07:35:32.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.222 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835276+0000", 2024-09-18T07:35:32.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.223 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835276+0000", 2024-09-18T07:35:32.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4900000000000002e-07 2024-09-18T07:35:32.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.223 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835277+0000", 2024-09-18T07:35:32.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.283e-06 2024-09-18T07:35:32.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.223 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835281+0000", 2024-09-18T07:35:32.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8010000000000001e-06 2024-09-18T07:35:32.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.224 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.224 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.835284+0000", 2024-09-18T07:35:32.224 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6369999999999999e-06 2024-09-18T07:35:32.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.224 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.224 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858399+0000", 2024-09-18T07:35:32.224 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.023114935 2024-09-18T07:35:32.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.224 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.224 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:55.858402+0000", 2024-09-18T07:35:32.225 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6069999999999999e-06 2024-09-18T07:35:32.225 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.225 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.865554+0000", 2024-09-18T07:35:32.225 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0071515559999999 2024-09-18T07:35:32.225 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:32.225 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.865641+0000", 2024-09-18T07:35:32.225 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7126999999999994e-05 2024-09-18T07:35:32.225 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:35:32.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.865725+0000", 2024-09-18T07:35:32.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3842000000000001e-05 2024-09-18T07:35:32.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:32.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.866976+0000", 2024-09-18T07:35:32.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012516199999999999 2024-09-18T07:35:32.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.867465+0000", 2024-09-18T07:35:32.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000488376 2024-09-18T07:35:32.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.880469+0000", 2024-09-18T07:35:32.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013004402 2024-09-18T07:35:32.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:32.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.880513+0000", 2024-09-18T07:35:32.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4143000000000001e-05 2024-09-18T07:35:32.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:56.880558+0000", 2024-09-18T07:35:32.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4796000000000003e-05 2024-09-18T07:35:32.227 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.228 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.228 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.228 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162316 7.f 7:f95a8250:::benchmark_data_smithi137_74180_object57900:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:35:32.228 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.409395+0000", 2024-09-18T07:35:32.228 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 30.703490042999999, 2024-09-18T07:35:32.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0987874790000001, 2024-09-18T07:35:32.228 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.228 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.228 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:32.228 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.229 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.229 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.229 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162316 2024-09-18T07:35:32.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.229 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.229 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.229 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409395+0000", 2024-09-18T07:35:32.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.229 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.229 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409395+0000", 2024-09-18T07:35:32.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409397+0000", 2024-09-18T07:35:32.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3429999999999998e-06 2024-09-18T07:35:32.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409404+0000", 2024-09-18T07:35:32.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2600000000000002e-06 2024-09-18T07:35:32.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409418+0000", 2024-09-18T07:35:32.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3983e-05 2024-09-18T07:35:32.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409427+0000", 2024-09-18T07:35:32.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9269999999999994e-06 2024-09-18T07:35:32.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409490+0000", 2024-09-18T07:35:32.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2868999999999997e-05 2024-09-18T07:35:32.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409500+0000", 2024-09-18T07:35:32.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0621e-05 2024-09-18T07:35:32.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.497693+0000", 2024-09-18T07:35:32.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.088192405 2024-09-18T07:35:32.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:32.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.497816+0000", 2024-09-18T07:35:32.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012321500000000001 2024-09-18T07:35:32.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:35:32.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.497964+0000", 2024-09-18T07:35:32.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000148169 2024-09-18T07:35:32.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.500777+0000", 2024-09-18T07:35:32.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028126750000000002 2024-09-18T07:35:32.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.500835+0000", 2024-09-18T07:35:32.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8727000000000002e-05 2024-09-18T07:35:32.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.500878+0000", 2024-09-18T07:35:32.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2682000000000002e-05 2024-09-18T07:35:32.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:32.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507887+0000", 2024-09-18T07:35:32.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0070087789999999997 2024-09-18T07:35:32.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:32.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508067+0000", 2024-09-18T07:35:32.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017973300000000001 2024-09-18T07:35:32.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508183+0000", 2024-09-18T07:35:32.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116091 2024-09-18T07:35:32.235 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.235 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.235 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.236 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162322 7.f 7:f41da5a4:::benchmark_data_smithi137_74180_object57906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:35:32.236 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.420348+0000", 2024-09-18T07:35:32.236 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 30.692536749999999, 2024-09-18T07:35:32.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0968331140000001, 2024-09-18T07:35:32.236 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.236 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.236 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:32.236 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.236 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.236 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.236 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162322 2024-09-18T07:35:32.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.236 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420348+0000", 2024-09-18T07:35:32.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420348+0000", 2024-09-18T07:35:32.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420350+0000", 2024-09-18T07:35:32.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8670000000000001e-06 2024-09-18T07:35:32.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.238 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.238 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.238 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420356+0000", 2024-09-18T07:35:32.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0560000000000001e-06 2024-09-18T07:35:32.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.238 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.238 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.238 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420369+0000", 2024-09-18T07:35:32.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2507000000000001e-05 2024-09-18T07:35:32.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.238 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420376+0000", 2024-09-18T07:35:32.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.17e-06 2024-09-18T07:35:32.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420444+0000", 2024-09-18T07:35:32.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7801999999999999e-05 2024-09-18T07:35:32.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420454+0000", 2024-09-18T07:35:32.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0258000000000001e-05 2024-09-18T07:35:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.501094+0000", 2024-09-18T07:35:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0806402799999999 2024-09-18T07:35:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.501227+0000", 2024-09-18T07:35:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013271600000000001 2024-09-18T07:35:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:35:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.501467+0000", 2024-09-18T07:35:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024032599999999999 2024-09-18T07:35:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507729+0000", 2024-09-18T07:35:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0062618220000000002 2024-09-18T07:35:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507768+0000", 2024-09-18T07:35:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8668000000000003e-05 2024-09-18T07:35:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.516751+0000", 2024-09-18T07:35:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0089832390000000005 2024-09-18T07:35:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.516888+0000", 2024-09-18T07:35:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137372 2024-09-18T07:35:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517069+0000", 2024-09-18T07:35:32.243 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000180574 2024-09-18T07:35:32.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.243 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.243 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.243 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517182+0000", 2024-09-18T07:35:32.243 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112457 2024-09-18T07:35:32.243 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.243 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.243 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.243 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.243 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162323 7.f 7:fe31c7fa:::benchmark_data_smithi137_74180_object57907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:35:32.244 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.420396+0000", 2024-09-18T07:35:32.244 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 30.692489279, 2024-09-18T07:35:32.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.097250378, 2024-09-18T07:35:32.244 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.244 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.244 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:32.244 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.244 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.244 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.244 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162323 2024-09-18T07:35:32.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.244 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420396+0000", 2024-09-18T07:35:32.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420396+0000", 2024-09-18T07:35:32.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420396+0000", 2024-09-18T07:35:32.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9400000000000005e-07 2024-09-18T07:35:32.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420399+0000", 2024-09-18T07:35:32.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6879999999999999e-06 2024-09-18T07:35:32.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420404+0000", 2024-09-18T07:35:32.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2680000000000002e-06 2024-09-18T07:35:32.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420411+0000", 2024-09-18T07:35:32.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2519999999999996e-06 2024-09-18T07:35:32.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420540+0000", 2024-09-18T07:35:32.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012886899999999999 2024-09-18T07:35:32.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420547+0000", 2024-09-18T07:35:32.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9019999999999999e-06 2024-09-18T07:35:32.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.504020+0000", 2024-09-18T07:35:32.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0834720959999999 2024-09-18T07:35:32.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:32.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.504135+0000", 2024-09-18T07:35:32.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115697 2024-09-18T07:35:32.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:35:32.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.504251+0000", 2024-09-18T07:35:32.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116041 2024-09-18T07:35:32.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507805+0000", 2024-09-18T07:35:32.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035537749999999999 2024-09-18T07:35:32.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507870+0000", 2024-09-18T07:35:32.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4558000000000001e-05 2024-09-18T07:35:32.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:32.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.516761+0000", 2024-09-18T07:35:32.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0088909690000000003 2024-09-18T07:35:32.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517414+0000", 2024-09-18T07:35:32.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00065372100000000001 2024-09-18T07:35:32.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:32.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517543+0000", 2024-09-18T07:35:32.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012907199999999999 2024-09-18T07:35:32.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517646+0000", 2024-09-18T07:35:32.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102876 2024-09-18T07:35:32.251 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.251 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.252 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.252 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162327 7.f 7:fac5d155:::benchmark_data_smithi137_74180_object57911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:35:32.252 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.420507+0000", 2024-09-18T07:35:32.252 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 30.692378270999999, 2024-09-18T07:35:32.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.097389145, 2024-09-18T07:35:32.252 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.252 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.252 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:32.252 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.252 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.252 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162327 2024-09-18T07:35:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420507+0000", 2024-09-18T07:35:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420507+0000", 2024-09-18T07:35:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420508+0000", 2024-09-18T07:35:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7300000000000005e-07 2024-09-18T07:35:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420511+0000", 2024-09-18T07:35:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4089999999999999e-06 2024-09-18T07:35:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420518+0000", 2024-09-18T07:35:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1010000000000003e-06 2024-09-18T07:35:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420523+0000", 2024-09-18T07:35:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2089999999999996e-06 2024-09-18T07:35:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420570+0000", 2024-09-18T07:35:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6638000000000003e-05 2024-09-18T07:35:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420577+0000", 2024-09-18T07:35:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2679999999999997e-06 2024-09-18T07:35:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508213+0000", 2024-09-18T07:35:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0876354130000001 2024-09-18T07:35:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508312+0000", 2024-09-18T07:35:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9124999999999997e-05 2024-09-18T07:35:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:35:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508859+0000", 2024-09-18T07:35:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00054700599999999995 2024-09-18T07:35:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.516816+0000", 2024-09-18T07:35:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079569410000000004 2024-09-18T07:35:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517264+0000", 2024-09-18T07:35:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044864999999999998 2024-09-18T07:35:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517309+0000", 2024-09-18T07:35:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5033000000000001e-05 2024-09-18T07:35:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517755+0000", 2024-09-18T07:35:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044556899999999999 2024-09-18T07:35:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517803+0000", 2024-09-18T07:35:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7584999999999999e-05 2024-09-18T07:35:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517896+0000", 2024-09-18T07:35:32.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3424999999999994e-05 2024-09-18T07:35:32.261 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.261 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.261 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.261 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162336 7.f 7:f2e86194:::benchmark_data_smithi137_74180_object57920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:35:32.261 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.425606+0000", 2024-09-18T07:35:32.261 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 30.687279064999998, 2024-09-18T07:35:32.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0924705029999999, 2024-09-18T07:35:32.261 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.261 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.261 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:32.262 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.262 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.262 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.262 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162336 2024-09-18T07:35:32.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.262 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425606+0000", 2024-09-18T07:35:32.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425606+0000", 2024-09-18T07:35:32.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425608+0000", 2024-09-18T07:35:32.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2259999999999999e-06 2024-09-18T07:35:32.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425614+0000", 2024-09-18T07:35:32.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0549999999999996e-06 2024-09-18T07:35:32.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425626+0000", 2024-09-18T07:35:32.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1600000000000001e-05 2024-09-18T07:35:32.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425635+0000", 2024-09-18T07:35:32.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0440000000000001e-06 2024-09-18T07:35:32.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425680+0000", 2024-09-18T07:35:32.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5015999999999997e-05 2024-09-18T07:35:32.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425689+0000", 2024-09-18T07:35:32.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3419999999999997e-06 2024-09-18T07:35:32.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.513184+0000", 2024-09-18T07:35:32.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0874942320000001 2024-09-18T07:35:32.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:32.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.513298+0000", 2024-09-18T07:35:32.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114333 2024-09-18T07:35:32.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:35:32.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.513425+0000", 2024-09-18T07:35:32.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126711 2024-09-18T07:35:32.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:32.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517321+0000", 2024-09-18T07:35:32.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038961590000000002 2024-09-18T07:35:32.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517374+0000", 2024-09-18T07:35:32.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3003000000000001e-05 2024-09-18T07:35:32.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517718+0000", 2024-09-18T07:35:32.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034425100000000001 2024-09-18T07:35:32.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517935+0000", 2024-09-18T07:35:32.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021721899999999999 2024-09-18T07:35:32.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:32.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517986+0000", 2024-09-18T07:35:32.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0627999999999997e-05 2024-09-18T07:35:32.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.518077+0000", 2024-09-18T07:35:32.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0684000000000001e-05 2024-09-18T07:35:32.269 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.269 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.269 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.269 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162345 7.f 7:f017fa1c:::benchmark_data_smithi137_74180_object57929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:35:32.269 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.431398+0000", 2024-09-18T07:35:32.269 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 30.681487577999999, 2024-09-18T07:35:32.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0972635669999999, 2024-09-18T07:35:32.269 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:32.269 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:32.269 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:32.269 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:32.269 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:32.270 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:32.270 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162345 2024-09-18T07:35:32.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.270 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:32.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:32.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431398+0000", 2024-09-18T07:35:32.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:32.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431398+0000", 2024-09-18T07:35:32.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:32.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:32.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431399+0000", 2024-09-18T07:35:32.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.361e-06 2024-09-18T07:35:32.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:32.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431403+0000", 2024-09-18T07:35:32.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1520000000000002e-06 2024-09-18T07:35:32.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:32.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431414+0000", 2024-09-18T07:35:32.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1239e-05 2024-09-18T07:35:32.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:32.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431421+0000", 2024-09-18T07:35:32.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.99e-06 2024-09-18T07:35:32.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431537+0000", 2024-09-18T07:35:32.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115278 2024-09-18T07:35:32.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:32.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431547+0000", 2024-09-18T07:35:32.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0543999999999999e-05 2024-09-18T07:35:32.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:32.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.518110+0000", 2024-09-18T07:35:32.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.086562942 2024-09-18T07:35:32.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:32.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.518220+0000", 2024-09-18T07:35:32.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010951 2024-09-18T07:35:32.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:35:32.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.518334+0000", 2024-09-18T07:35:32.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114057 2024-09-18T07:35:32.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.527942+0000", 2024-09-18T07:35:32.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0096087029999999997 2024-09-18T07:35:32.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528087+0000", 2024-09-18T07:35:32.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000144871 2024-09-18T07:35:32.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:32.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528103+0000", 2024-09-18T07:35:32.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5531000000000001e-05 2024-09-18T07:35:32.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:32.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528148+0000", 2024-09-18T07:35:32.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5667000000000002e-05 2024-09-18T07:35:32.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:32.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528542+0000", 2024-09-18T07:35:32.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00039349200000000001 2024-09-18T07:35:32.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:32.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:32.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:32.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528661+0000", 2024-09-18T07:35:32.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011923 2024-09-18T07:35:32.276 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.276 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.276 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.276 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:32.276 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:32.276 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:35:32.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:32.277 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:32.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:32.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:32.405 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:32.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:32.507 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:32.533 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:32.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:32.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:32.635 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:32.664 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:32.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:32.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:32.764 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:35:32.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:32.788 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:32.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:32.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:32.890 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:32.916 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:32.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:32.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:32.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:32 smithi137 ceph-mon[29408]: 5.18 scrub starts 2024-09-18T07:35:32.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:32 smithi137 ceph-mon[29408]: 5.18 scrub ok 2024-09-18T07:35:32.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:32 smithi137 ceph-mon[29408]: 5.f scrub starts 2024-09-18T07:35:32.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:32 smithi137 ceph-mon[29408]: 5.f scrub ok 2024-09-18T07:35:32.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:32 smithi137 ceph-mon[29408]: pgmap v246: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.2 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 1375/58968 objects misplaced (2.332%) 2024-09-18T07:35:32.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:32 smithi137 ceph-mon[29408]: 4.1c scrub starts 2024-09-18T07:35:32.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:32 smithi137 ceph-mon[29408]: 4.1c scrub ok 2024-09-18T07:35:32.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:32 smithi137 ceph-mon[39175]: 5.18 scrub starts 2024-09-18T07:35:32.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:32 smithi137 ceph-mon[39175]: 5.18 scrub ok 2024-09-18T07:35:32.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:32 smithi137 ceph-mon[39175]: 5.f scrub starts 2024-09-18T07:35:32.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:32 smithi137 ceph-mon[39175]: 5.f scrub ok 2024-09-18T07:35:32.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:32 smithi137 ceph-mon[39175]: pgmap v246: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.2 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 1375/58968 objects misplaced (2.332%) 2024-09-18T07:35:32.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:32 smithi137 ceph-mon[39175]: 4.1c scrub starts 2024-09-18T07:35:32.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:32 smithi137 ceph-mon[39175]: 4.1c scrub ok 2024-09-18T07:35:32.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:32 smithi203 ceph-mon[32237]: 5.18 scrub starts 2024-09-18T07:35:32.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:32 smithi203 ceph-mon[32237]: 5.18 scrub ok 2024-09-18T07:35:32.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:32 smithi203 ceph-mon[32237]: 5.f scrub starts 2024-09-18T07:35:32.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:32 smithi203 ceph-mon[32237]: 5.f scrub ok 2024-09-18T07:35:32.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:32 smithi203 ceph-mon[32237]: pgmap v246: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 1.2 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 1375/58968 objects misplaced (2.332%) 2024-09-18T07:35:32.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:32 smithi203 ceph-mon[32237]: 4.1c scrub starts 2024-09-18T07:35:32.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:32 smithi203 ceph-mon[32237]: 4.1c scrub ok 2024-09-18T07:35:33.019 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:33.043 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:33.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:33.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:33.145 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:33.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:33.202 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:33.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:33.303 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:33.330 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:33.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:33.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:33.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:33.465 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:33.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:33.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:33.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:35:33.568 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:33.595 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:33.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:33.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:33.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:33.741 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:33.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:33.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:33.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:33.881 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:33.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:33.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:33.985 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:34.013 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:34.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:34.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:34.115 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:34.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T07:35:34.142 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:34.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:34.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:34.244 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:34.269 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:34.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:34.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:34.360 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:35:34.361 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-18T07:35:34.361 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:35:34.371 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:34.376 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:34.394 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:34.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:34.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:34.496 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:34.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:34.524 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:34.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:34.625 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:34.668 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:34.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:34.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:34.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:34.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:34.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:34.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:34.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:34.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:34 smithi137 ceph-mon[29408]: 4.d scrub starts 2024-09-18T07:35:34.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:34 smithi137 ceph-mon[29408]: 4.d scrub ok 2024-09-18T07:35:34.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:34 smithi137 ceph-mon[29408]: pgmap v247: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 926 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.96k op/s; 878/45018 objects misplaced (1.950%) 2024-09-18T07:35:34.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:34 smithi137 ceph-mon[39175]: 4.d scrub starts 2024-09-18T07:35:34.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:34 smithi137 ceph-mon[39175]: 4.d scrub ok 2024-09-18T07:35:34.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:34 smithi137 ceph-mon[39175]: pgmap v247: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 926 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.96k op/s; 878/45018 objects misplaced (1.950%) 2024-09-18T07:35:34.930 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:34.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:34.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:34.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:34 smithi203 ceph-mon[32237]: 4.d scrub starts 2024-09-18T07:35:34.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:34 smithi203 ceph-mon[32237]: 4.d scrub ok 2024-09-18T07:35:34.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:34 smithi203 ceph-mon[32237]: pgmap v247: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 926 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.96k op/s; 878/45018 objects misplaced (1.950%) 2024-09-18T07:35:35.032 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:35.058 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:35.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:35.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:35.187 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:35.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:35.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:35.288 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:35.317 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:35.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:35.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:35.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:35.447 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:35.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:35.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:35.550 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:35.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:35.578 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:35.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:35.679 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:35.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:35.702 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:35.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:35.803 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:35.832 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:35.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:35.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:35.933 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:35.960 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:35.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:35.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:36.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:36.094 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:36.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:36.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:36.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:35 smithi137 ceph-mon[29408]: 3.a scrub starts 2024-09-18T07:35:36.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:35 smithi137 ceph-mon[29408]: 3.a scrub ok 2024-09-18T07:35:36.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:35 smithi137 ceph-mon[39175]: 3.a scrub starts 2024-09-18T07:35:36.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:35 smithi137 ceph-mon[39175]: 3.a scrub ok 2024-09-18T07:35:36.196 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:36.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:35 smithi203 ceph-mon[32237]: 3.a scrub starts 2024-09-18T07:35:36.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:35 smithi203 ceph-mon[32237]: 3.a scrub ok 2024-09-18T07:35:36.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:36.224 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:36.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:36.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:36.362 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:36.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:36.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:36.464 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:36.508 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:36.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:36.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:36.610 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:36.642 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:36.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:36.744 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:36.772 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:36.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:36.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:36.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:36.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:36.913 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:36.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:37.014 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:37.042 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:37.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:37.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:37.090 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:36 smithi203 ceph-mon[32237]: 6.1f scrub starts 2024-09-18T07:35:37.090 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:36 smithi203 ceph-mon[32237]: 6.1f scrub ok 2024-09-18T07:35:37.090 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:36 smithi203 ceph-mon[32237]: 6.19 scrub starts 2024-09-18T07:35:37.090 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:36 smithi203 ceph-mon[32237]: 6.19 scrub ok 2024-09-18T07:35:37.090 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:36 smithi203 ceph-mon[32237]: pgmap v248: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 926 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 928 op/s; 878/45018 objects misplaced (1.950%) 2024-09-18T07:35:37.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:36 smithi137 ceph-mon[29408]: 6.1f scrub starts 2024-09-18T07:35:37.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:36 smithi137 ceph-mon[29408]: 6.1f scrub ok 2024-09-18T07:35:37.102 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:36 smithi137 ceph-mon[29408]: 6.19 scrub starts 2024-09-18T07:35:37.102 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:36 smithi137 ceph-mon[29408]: 6.19 scrub ok 2024-09-18T07:35:37.102 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:36 smithi137 ceph-mon[29408]: pgmap v248: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 926 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 928 op/s; 878/45018 objects misplaced (1.950%) 2024-09-18T07:35:37.102 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:36 smithi137 ceph-mon[39175]: 6.1f scrub starts 2024-09-18T07:35:37.102 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:36 smithi137 ceph-mon[39175]: 6.1f scrub ok 2024-09-18T07:35:37.102 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:36 smithi137 ceph-mon[39175]: 6.19 scrub starts 2024-09-18T07:35:37.102 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:36 smithi137 ceph-mon[39175]: 6.19 scrub ok 2024-09-18T07:35:37.102 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:36 smithi137 ceph-mon[39175]: pgmap v248: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 926 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 928 op/s; 878/45018 objects misplaced (1.950%) 2024-09-18T07:35:37.144 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:37.165 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:37.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:37.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:37.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:37.304 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:37.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:37.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:37.406 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:37.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:35:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:35:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:35:37.434 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:37.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:37.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:35:37 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:35:37.536 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:37.669 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:37.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:37.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:37.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:37.805 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:37.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:37.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:37.933 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:37.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:37.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:38.035 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:38.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:38.058 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:38.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:38.159 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:38.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:37 smithi137 ceph-mon[29408]: 6.13 scrub starts 2024-09-18T07:35:38.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:37 smithi137 ceph-mon[29408]: 6.13 scrub ok 2024-09-18T07:35:38.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:37 smithi137 ceph-mon[29408]: 4.8 scrub starts 2024-09-18T07:35:38.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:37 smithi137 ceph-mon[29408]: 4.8 scrub ok 2024-09-18T07:35:38.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:37 smithi137 ceph-mon[39175]: 6.13 scrub starts 2024-09-18T07:35:38.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:37 smithi137 ceph-mon[39175]: 6.13 scrub ok 2024-09-18T07:35:38.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:37 smithi137 ceph-mon[39175]: 4.8 scrub starts 2024-09-18T07:35:38.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:37 smithi137 ceph-mon[39175]: 4.8 scrub ok 2024-09-18T07:35:38.181 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:38.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:38.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:38.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:37 smithi203 ceph-mon[32237]: 6.13 scrub starts 2024-09-18T07:35:38.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:37 smithi203 ceph-mon[32237]: 6.13 scrub ok 2024-09-18T07:35:38.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:37 smithi203 ceph-mon[32237]: 4.8 scrub starts 2024-09-18T07:35:38.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:37 smithi203 ceph-mon[32237]: 4.8 scrub ok 2024-09-18T07:35:38.283 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:38.306 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:38.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:38.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:38.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:38.431 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:38.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:38.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:38.533 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:38.556 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:38.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:38.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:38.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:38.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:38.683 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:38.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:38.784 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:38.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:38.807 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:38.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:38.908 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:38.932 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:38.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:38.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:39.034 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:39.061 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:39.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:39.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:39.163 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:39.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:38 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:39.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:38 smithi137 ceph-mon[29408]: pgmap v249: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 505 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.44k op/s; 85/24792 objects misplaced (0.343%); 3.4 MiB/s, 53 objects/s recovering 2024-09-18T07:35:39.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:38 smithi137 ceph-mon[29408]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-18T07:35:39.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:38 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:35:39.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:38 smithi137 ceph-mon[29408]: osdmap e194: 8 total, 8 up, 8 in 2024-09-18T07:35:39.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:38 smithi137 ceph-mon[29408]: 3.1a scrub starts 2024-09-18T07:35:39.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:38 smithi137 ceph-mon[29408]: 3.1a scrub ok 2024-09-18T07:35:39.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:38 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:39.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:38 smithi137 ceph-mon[39175]: pgmap v249: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 505 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.44k op/s; 85/24792 objects misplaced (0.343%); 3.4 MiB/s, 53 objects/s recovering 2024-09-18T07:35:39.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:38 smithi137 ceph-mon[39175]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-18T07:35:39.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:38 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:35:39.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:38 smithi137 ceph-mon[39175]: osdmap e194: 8 total, 8 up, 8 in 2024-09-18T07:35:39.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:38 smithi137 ceph-mon[39175]: 3.1a scrub starts 2024-09-18T07:35:39.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:38 smithi137 ceph-mon[39175]: 3.1a scrub ok 2024-09-18T07:35:39.188 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:39.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:39.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:39.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:38 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:39.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:38 smithi203 ceph-mon[32237]: pgmap v249: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 505 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.44k op/s; 85/24792 objects misplaced (0.343%); 3.4 MiB/s, 53 objects/s recovering 2024-09-18T07:35:39.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:38 smithi203 ceph-mon[32237]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-18T07:35:39.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:38 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:35:39.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:38 smithi203 ceph-mon[32237]: osdmap e194: 8 total, 8 up, 8 in 2024-09-18T07:35:39.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:38 smithi203 ceph-mon[32237]: 3.1a scrub starts 2024-09-18T07:35:39.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:38 smithi203 ceph-mon[32237]: 3.1a scrub ok 2024-09-18T07:35:39.290 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:39.316 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:39.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:39.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:39.418 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:39.456 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:39.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:39.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:39.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:39.610 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:39.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:39.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:39.712 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:39.739 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:39.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:39.841 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:39.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:39.879 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:39.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:39.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:40.022 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:40.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:40.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:40.124 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:40.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:40.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:40.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:40.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:39 smithi137 ceph-mon[29408]: osdmap e195: 8 total, 8 up, 8 in 2024-09-18T07:35:40.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:39 smithi137 ceph-mon[29408]: 4.e scrub starts 2024-09-18T07:35:40.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:39 smithi137 ceph-mon[29408]: 4.e scrub ok 2024-09-18T07:35:40.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:39 smithi137 ceph-mon[29408]: 6.c scrub starts 2024-09-18T07:35:40.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:39 smithi137 ceph-mon[39175]: osdmap e195: 8 total, 8 up, 8 in 2024-09-18T07:35:40.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:39 smithi137 ceph-mon[39175]: 4.e scrub starts 2024-09-18T07:35:40.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:39 smithi137 ceph-mon[39175]: 4.e scrub ok 2024-09-18T07:35:40.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:39 smithi137 ceph-mon[39175]: 6.c scrub starts 2024-09-18T07:35:40.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:39 smithi203 ceph-mon[32237]: osdmap e195: 8 total, 8 up, 8 in 2024-09-18T07:35:40.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:39 smithi203 ceph-mon[32237]: 4.e scrub starts 2024-09-18T07:35:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:39 smithi203 ceph-mon[32237]: 4.e scrub ok 2024-09-18T07:35:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:39 smithi203 ceph-mon[32237]: 6.c scrub starts 2024-09-18T07:35:40.265 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:40.292 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:40.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:40.394 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:40.419 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:40.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:40.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:40.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:40.569 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:40.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:40.595 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:40.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:40.626 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:40.696 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:40.723 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:40.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:40.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:40.871 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:40.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:40.973 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:41.009 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:41.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:41.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:41.025 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:40 smithi137 ceph-mon[29408]: 6.c scrub ok 2024-09-18T07:35:41.025 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:40 smithi137 ceph-mon[29408]: pgmap v252: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 505 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.42k op/s; 85/24792 objects misplaced (0.343%); 4.1 MiB/s, 66 objects/s recovering 2024-09-18T07:35:41.025 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:40 smithi137 ceph-mon[39175]: 6.c scrub ok 2024-09-18T07:35:41.025 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:40 smithi137 ceph-mon[39175]: pgmap v252: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 505 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.42k op/s; 85/24792 objects misplaced (0.343%); 4.1 MiB/s, 66 objects/s recovering 2024-09-18T07:35:41.112 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:41.138 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:41.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:41.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:41.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:40 smithi203 ceph-mon[32237]: 6.c scrub ok 2024-09-18T07:35:41.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:40 smithi203 ceph-mon[32237]: pgmap v252: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 505 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.42k op/s; 85/24792 objects misplaced (0.343%); 4.1 MiB/s, 66 objects/s recovering 2024-09-18T07:35:41.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:41.274 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:41.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:41.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:41.376 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:41.402 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:41.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:41.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:41.413 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:35:41.414 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:35:41.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:41.414 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:205977 7.9 e195/148 7:96dc0611:::benchmark_data_smithi137_74180_object101561:head v 195'12492, mlcod=195'12490)", 2024-09-18T07:35:41.414 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:41.408018+0000", 2024-09-18T07:35:41.414 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.002956724, 2024-09-18T07:35:41.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029907100000000002, 2024-09-18T07:35:41.414 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:41.414 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:41.414 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:41.414 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:41.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:41.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:41.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:41.408018+0000", 2024-09-18T07:35:41.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:41.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:41.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:41.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:41.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:41.408018+0000", 2024-09-18T07:35:41.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:41.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:41.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:41.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:41.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:41.408019+0000", 2024-09-18T07:35:41.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.672e-06 2024-09-18T07:35:41.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:41.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:41.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:41.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:41.408026+0000", 2024-09-18T07:35:41.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3890000000000002e-06 2024-09-18T07:35:41.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:41.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:41.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:41.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:41.408039+0000", 2024-09-18T07:35:41.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3006999999999999e-05 2024-09-18T07:35:41.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:41.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:41.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:41.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:41.408046+0000", 2024-09-18T07:35:41.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6699999999999997e-06 2024-09-18T07:35:41.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:41.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:41.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:41.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:41.408451+0000", 2024-09-18T07:35:41.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040530399999999999 2024-09-18T07:35:41.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:41.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:41.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:41.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:41.408497+0000", 2024-09-18T07:35:41.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6218999999999999e-05 2024-09-18T07:35:41.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:41.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:41.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:41.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:41.410885+0000", 2024-09-18T07:35:41.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023879819999999999 2024-09-18T07:35:41.417 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:41.418 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:41.418 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:41.418 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:41.418 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:35:41.418 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 1 2024-09-18T07:35:41.418 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:35:41.504 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:41.529 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:41.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:41.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:41.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:41.664 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:35:41.665 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:35:41.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:41.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:41.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:41.699 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:35:41.699 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:35:41.738 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:35:41.738 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:35:41.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:41.772 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:35:41.773 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:35:41.807 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:35:41.807 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:35:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:41.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:41.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:41.838 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:35:41.839 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:35:41.862 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:35:41.878 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:35:41.879 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:35:41.912 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:35:41.912 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:35:41.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:41.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[29408]: 7.7 scrub starts 2024-09-18T07:35:41.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[29408]: 7.f scrub starts 2024-09-18T07:35:41.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[29408]: 7.7 scrub ok 2024-09-18T07:35:41.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[29408]: 7.f scrub ok 2024-09-18T07:35:41.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[29408]: 7.8 scrub starts 2024-09-18T07:35:41.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[29408]: 7.8 scrub ok 2024-09-18T07:35:41.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[29408]: 7.1 scrub starts 2024-09-18T07:35:41.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3050894694' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:35:41.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[29408]: from='client.35084 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:35:41.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[29408]: 7.1 scrub ok 2024-09-18T07:35:41.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:35:41.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:35:41.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3153116409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:41.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[39175]: 7.7 scrub starts 2024-09-18T07:35:41.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[39175]: 7.f scrub starts 2024-09-18T07:35:41.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[39175]: 7.7 scrub ok 2024-09-18T07:35:41.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[39175]: 7.f scrub ok 2024-09-18T07:35:41.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[39175]: 7.8 scrub starts 2024-09-18T07:35:41.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[39175]: 7.8 scrub ok 2024-09-18T07:35:41.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[39175]: 7.1 scrub starts 2024-09-18T07:35:41.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3050894694' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:35:41.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[39175]: from='client.35084 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:35:41.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[39175]: 7.1 scrub ok 2024-09-18T07:35:41.926 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:35:41.926 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:35:41.926 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:41 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3153116409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:41.950 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:41.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:42.052 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:42.079 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:42.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:42.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:42.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:41 smithi203 ceph-mon[32237]: 7.7 scrub starts 2024-09-18T07:35:42.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:41 smithi203 ceph-mon[32237]: 7.f scrub starts 2024-09-18T07:35:42.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:41 smithi203 ceph-mon[32237]: 7.7 scrub ok 2024-09-18T07:35:42.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:41 smithi203 ceph-mon[32237]: 7.f scrub ok 2024-09-18T07:35:42.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:41 smithi203 ceph-mon[32237]: 7.8 scrub starts 2024-09-18T07:35:42.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:41 smithi203 ceph-mon[32237]: 7.8 scrub ok 2024-09-18T07:35:42.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:41 smithi203 ceph-mon[32237]: 7.1 scrub starts 2024-09-18T07:35:42.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:41 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3050894694' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:35:42.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:41 smithi203 ceph-mon[32237]: from='client.35084 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:35:42.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:41 smithi203 ceph-mon[32237]: 7.1 scrub ok 2024-09-18T07:35:42.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:41 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:35:42.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:41 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:35:42.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:41 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3153116409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:42.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:42.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:42.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:42.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:42.359 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:42.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:42.460 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:42.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:42.497 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:42.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:42.598 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:42.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:42.639 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:42.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:42.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:42.776 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:42.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:42.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:42.878 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:42.905 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:42.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:42.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:43.006 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:43.040 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:43.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:43.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:43.142 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:42 smithi137 ceph-mon[29408]: 5.10 scrub starts 2024-09-18T07:35:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:42 smithi137 ceph-mon[29408]: 5.10 scrub ok 2024-09-18T07:35:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:42 smithi137 ceph-mon[29408]: pgmap v253: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 375 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 85/18564 objects misplaced (0.458%); 4.1 MiB/s, 66 objects/s recovering 2024-09-18T07:35:43.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:42 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:43.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:42 smithi137 ceph-mon[29408]: from='client.35084 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:35:43.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:42 smithi137 ceph-mon[29408]: osdmap e196: 8 total, 8 up, 8 in 2024-09-18T07:35:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:42 smithi137 ceph-mon[39175]: 5.10 scrub starts 2024-09-18T07:35:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:42 smithi137 ceph-mon[39175]: 5.10 scrub ok 2024-09-18T07:35:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:42 smithi137 ceph-mon[39175]: pgmap v253: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 375 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 85/18564 objects misplaced (0.458%); 4.1 MiB/s, 66 objects/s recovering 2024-09-18T07:35:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:42 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:42 smithi137 ceph-mon[39175]: from='client.35084 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:35:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:42 smithi137 ceph-mon[39175]: osdmap e196: 8 total, 8 up, 8 in 2024-09-18T07:35:43.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:43.179 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:43.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:43.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:42 smithi203 ceph-mon[32237]: 5.10 scrub starts 2024-09-18T07:35:43.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:42 smithi203 ceph-mon[32237]: 5.10 scrub ok 2024-09-18T07:35:43.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:42 smithi203 ceph-mon[32237]: pgmap v253: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 375 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 85/18564 objects misplaced (0.458%); 4.1 MiB/s, 66 objects/s recovering 2024-09-18T07:35:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:42 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:42 smithi203 ceph-mon[32237]: from='client.35084 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:35:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:42 smithi203 ceph-mon[32237]: osdmap e196: 8 total, 8 up, 8 in 2024-09-18T07:35:43.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:43.312 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T07:35:43.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:43.322 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:43.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:43.423 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:43.449 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:43.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:43.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:43.455 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Removed 104413 objects 2024-09-18T07:35:43.455 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Clean up completed and total clean up time :117.66 2024-09-18T07:35:43.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:43.572 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:43.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:43.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:43.674 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:43.695 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:43.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:43.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:43.797 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:43.819 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:43.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:43.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:43.921 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:43.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:43.943 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:43.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:44.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:44.068 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:44.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:44.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:44.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:44.193 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:44.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:44.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:44.285 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:44.295 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:44.321 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:44.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:44.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:44.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:44.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:44.448 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:44.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:44.550 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:44.576 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:44.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:44.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:44.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:44.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:44.701 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:44.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:44.796 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:35:44.796 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:35:44.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:44.828 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:44.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:44.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:44.862 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:35:44.930 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:44.956 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:44.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:44.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:45.058 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:45.084 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:45.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:45.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:45.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:44 smithi137 ceph-mon[29408]: pgmap v255: 148 pgs: 148 active+clean; 163 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 910 op/s; 863 KiB/s, 13 objects/s recovering 2024-09-18T07:35:45.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:44 smithi137 ceph-mon[29408]: osdmap e197: 8 total, 8 up, 8 in 2024-09-18T07:35:45.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:44 smithi137 ceph-mon[39175]: pgmap v255: 148 pgs: 148 active+clean; 163 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 910 op/s; 863 KiB/s, 13 objects/s recovering 2024-09-18T07:35:45.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:44 smithi137 ceph-mon[39175]: osdmap e197: 8 total, 8 up, 8 in 2024-09-18T07:35:45.185 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:45.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:45.209 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:45.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:45.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:44 smithi203 ceph-mon[32237]: pgmap v255: 148 pgs: 148 active+clean; 163 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 910 op/s; 863 KiB/s, 13 objects/s recovering 2024-09-18T07:35:45.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:44 smithi203 ceph-mon[32237]: osdmap e197: 8 total, 8 up, 8 in 2024-09-18T07:35:45.271 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:35:45.271 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:35:45.271 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-18T07:35:45.272 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-18T07:35:45.310 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:45.335 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:45.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:45.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:45.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:45.459 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:45.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:45.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:45.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:45.583 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:45.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:45.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:45.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:45.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:45.708 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:45.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:45.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:45.830 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:45.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:45.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:45.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:45.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T07:35:45.958 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:45.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:45.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:46.060 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:46.061 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:46.068 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:46.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:46.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:46.179 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:35:46.186 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:46.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:46.212 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:46.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:46.313 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:46.339 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:46.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:46.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:46.441 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:46.466 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:46.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:46.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:46.590 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:46.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:46.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:46.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:46.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:46.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:46.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:46.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:46.835 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:46.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:46.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:46.937 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:46.944 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:35:46.944 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:35:46.945 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 197, 2024-09-18T07:35:46.945 INFO:teuthology.orchestra.run.smithi137.stdout: "fsid": "f7ef099c-758e-11ef-bceb-c7b262605968", 2024-09-18T07:35:46.945 INFO:teuthology.orchestra.run.smithi137.stdout: "created": "2024-09-18T07:25:19.662367+0000", 2024-09-18T07:35:46.945 INFO:teuthology.orchestra.run.smithi137.stdout: "modified": "2024-09-18T07:35:44.790263+0000", 2024-09-18T07:35:46.945 INFO:teuthology.orchestra.run.smithi137.stdout: "last_up_change": "2024-09-18T07:30:44.708496+0000", 2024-09-18T07:35:46.945 INFO:teuthology.orchestra.run.smithi137.stdout: "last_in_change": "2024-09-18T07:35:00.369558+0000", 2024-09-18T07:35:46.945 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T07:35:46.945 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_num": 5801984, 2024-09-18T07:35:46.946 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_set": [ 2024-09-18T07:35:46.946 INFO:teuthology.orchestra.run.smithi137.stdout: "noscrub", 2024-09-18T07:35:46.946 INFO:teuthology.orchestra.run.smithi137.stdout: "pglog_hardlimit", 2024-09-18T07:35:46.946 INFO:teuthology.orchestra.run.smithi137.stdout: "purged_snapdirs", 2024-09-18T07:35:46.946 INFO:teuthology.orchestra.run.smithi137.stdout: "recovery_deletes", 2024-09-18T07:35:46.946 INFO:teuthology.orchestra.run.smithi137.stdout: "sortbitwise" 2024-09-18T07:35:46.946 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:35:46.946 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_version": 18, 2024-09-18T07:35:46.946 INFO:teuthology.orchestra.run.smithi137.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T07:35:46.946 INFO:teuthology.orchestra.run.smithi137.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T07:35:46.946 INFO:teuthology.orchestra.run.smithi137.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T07:35:46.946 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_snapshot": "", 2024-09-18T07:35:46.947 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_max": 7, 2024-09-18T07:35:46.947 INFO:teuthology.orchestra.run.smithi137.stdout: "max_osd": 8, 2024-09-18T07:35:46.947 INFO:teuthology.orchestra.run.smithi137.stdout: "require_min_compat_client": "luminous", 2024-09-18T07:35:46.947 INFO:teuthology.orchestra.run.smithi137.stdout: "min_compat_client": "luminous", 2024-09-18T07:35:46.947 INFO:teuthology.orchestra.run.smithi137.stdout: "require_osd_release": "squid", 2024-09-18T07:35:46.947 INFO:teuthology.orchestra.run.smithi137.stdout: "allow_crimson": false, 2024-09-18T07:35:46.947 INFO:teuthology.orchestra.run.smithi137.stdout: "pools": [ 2024-09-18T07:35:46.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:46.947 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 1, 2024-09-18T07:35:46.947 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".mgr", 2024-09-18T07:35:46.947 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:28:20.899655+0000", 2024-09-18T07:35:46.947 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:35:46.947 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:35:46.948 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:35:46.948 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:35:46.948 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:35:46.948 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:35:46.948 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:35:46.948 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:35:46.948 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:35:46.948 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:35:46.948 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:35:46.948 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:35:46.948 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:35:46.948 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 1, 2024-09-18T07:35:46.949 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 1, 2024-09-18T07:35:46.949 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 1, 2024-09-18T07:35:46.949 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 1, 2024-09-18T07:35:46.949 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 1, 2024-09-18T07:35:46.949 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:35:46.949 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:35:46.949 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:35:46.949 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:35:46.949 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:35:46.949 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:35:46.949 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:35:46.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.950 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "21", 2024-09-18T07:35:46.950 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:35:46.950 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:35:46.950 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:35:46.950 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:35:46.950 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:35:46.950 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:35:46.950 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:35:46.950 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:35:46.950 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:35:46.950 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:35:46.950 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:35:46.951 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:35:46.951 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:35:46.951 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:35:46.951 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:35:46.951 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:35:46.951 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:35:46.951 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:35:46.951 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:35:46.951 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:35:46.951 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:35:46.951 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:35:46.951 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:35:46.951 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:35:46.952 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:35:46.952 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:35:46.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.952 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:35:46.952 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:35:46.952 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:35:46.952 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:35:46.952 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:35:46.952 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:35:46.952 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:35:46.952 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:35:46.952 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:35:46.952 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:35:46.953 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:35:46.953 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:35:46.953 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_max": 32, 2024-09-18T07:35:46.953 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_min": 1 2024-09-18T07:35:46.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.953 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:35:46.953 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": {} 2024-09-18T07:35:46.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.954 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:35:46.954 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:35:46.954 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 7.8899998664855957, 2024-09-18T07:35:46.954 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 7.8899998664855957, 2024-09-18T07:35:46.954 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.37999999523162842, 2024-09-18T07:35:46.954 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 3, 2024-09-18T07:35:46.954 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 3, 2024-09-18T07:35:46.954 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:35:46.954 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:35:46.954 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:35:46.954 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:46.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:46.955 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 2, 2024-09-18T07:35:46.955 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "datapool", 2024-09-18T07:35:46.955 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:53.715373+0000", 2024-09-18T07:35:46.955 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-18T07:35:46.955 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T07:35:46.955 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:35:46.955 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:35:46.955 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:35:46.955 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:35:46.955 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:35:46.955 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:35:46.955 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:35:46.955 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:35:46.956 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:35:46.956 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:35:46.956 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:35:46.956 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 3, 2024-09-18T07:35:46.956 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 3, 2024-09-18T07:35:46.956 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 3, 2024-09-18T07:35:46.956 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 3, 2024-09-18T07:35:46.956 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 3, 2024-09-18T07:35:46.956 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:35:46.956 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:35:46.956 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:35:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:35:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:35:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:35:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:35:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "63", 2024-09-18T07:35:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:35:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:35:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:35:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:35:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:35:46.957 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 2, 2024-09-18T07:35:46.958 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 63, 2024-09-18T07:35:46.958 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:35:46.958 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:35:46.958 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:35:46.958 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:35:46.958 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:35:46.958 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:35:46.958 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:35:46.958 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:35:46.958 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:35:46.958 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:35:46.958 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:35:46.958 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:35:46.959 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:35:46.959 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:35:46.959 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:35:46.959 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:35:46.959 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:35:46.959 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:35:46.959 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:35:46.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.959 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:35:46.959 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:35:46.959 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:35:46.959 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:35:46.959 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:35:46.960 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:35:46.960 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:35:46.960 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:35:46.960 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:35:46.960 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:35:46.960 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:35:46.960 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:35:46.960 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:35:46.960 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd": {} 2024-09-18T07:35:46.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.960 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:35:46.960 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:35:46.960 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.6500000953674316, 2024-09-18T07:35:46.960 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.6500000953674316, 2024-09-18T07:35:46.961 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.87999999523162842, 2024-09-18T07:35:46.961 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-18T07:35:46.961 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-18T07:35:46.961 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:35:46.961 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:35:46.961 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:35:46.961 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:46.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:46.962 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 3, 2024-09-18T07:35:46.962 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".rgw.root", 2024-09-18T07:35:46.962 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:54.672127+0000", 2024-09-18T07:35:46.962 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:35:46.962 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:35:46.962 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:35:46.962 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:35:46.962 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:35:46.962 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:35:46.963 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:35:46.963 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:35:46.963 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:35:46.963 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:35:46.963 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:35:46.963 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:35:46.963 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:35:46.963 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:35:46.964 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:35:46.964 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:35:46.964 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:35:46.964 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:35:46.964 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:35:46.964 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:35:46.964 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:35:46.964 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:35:46.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:46.966 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:35:46.966 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:35:46.966 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:35:46.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.966 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "62", 2024-09-18T07:35:46.966 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:35:46.966 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:35:46.966 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:35:46.966 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:35:46.966 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:35:46.966 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:35:46.967 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:35:46.967 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:35:46.967 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:35:46.967 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:35:46.967 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:35:46.967 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:35:46.967 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:35:46.967 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:35:46.967 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:35:46.967 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:35:46.967 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:35:46.967 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:35:46.968 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:35:46.968 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:35:46.968 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:35:46.968 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:35:46.968 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:35:46.968 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:35:46.968 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:35:46.968 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:35:46.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.968 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:35:46.968 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:35:46.968 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:35:46.969 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:35:46.969 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:35:46.969 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:35:46.969 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:35:46.969 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:35:46.969 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:35:46.969 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:35:46.969 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:35:46.969 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:35:46.969 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:35:46.969 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:35:46.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.969 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:35:46.970 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:35:46.970 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.5, 2024-09-18T07:35:46.970 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.5, 2024-09-18T07:35:46.970 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:35:46.970 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.5, 2024-09-18T07:35:46.970 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.5, 2024-09-18T07:35:46.970 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:35:46.970 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:35:46.970 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:35:46.970 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:46.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:46.970 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 4, 2024-09-18T07:35:46.970 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.log", 2024-09-18T07:35:46.971 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:56.351343+0000", 2024-09-18T07:35:46.971 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:35:46.971 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:35:46.971 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:35:46.971 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:35:46.971 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:35:46.971 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:35:46.971 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:35:46.971 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:35:46.971 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:35:46.971 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:35:46.971 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:35:46.972 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:35:46.972 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:35:46.972 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:35:46.972 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:35:46.972 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:35:46.972 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:35:46.972 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:35:46.972 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:35:46.972 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:35:46.972 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:35:46.972 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:35:46.972 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:35:46.973 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:35:46.973 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:35:46.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.973 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "64", 2024-09-18T07:35:46.973 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:35:46.973 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:35:46.973 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:35:46.973 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:35:46.973 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:35:46.973 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:35:46.973 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:35:46.973 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:35:46.973 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:35:46.974 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:35:46.974 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:35:46.974 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:35:46.974 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:35:46.974 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:35:46.974 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:35:46.974 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:35:46.974 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:35:46.974 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:35:46.974 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:35:46.974 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:35:46.974 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:35:46.974 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:35:46.975 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:35:46.975 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:35:46.975 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:35:46.975 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:35:46.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.975 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:35:46.975 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:35:46.975 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:35:46.975 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:35:46.975 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:35:46.975 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:35:46.975 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:35:46.975 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:35:46.976 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:35:46.976 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:35:46.976 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:35:46.976 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:35:46.976 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:35:46.976 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:35:46.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.976 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:35:46.976 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:35:46.976 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.25, 2024-09-18T07:35:46.976 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.25, 2024-09-18T07:35:46.976 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:35:46.977 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.25, 2024-09-18T07:35:46.977 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.25, 2024-09-18T07:35:46.977 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:35:46.977 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:35:46.977 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:35:46.977 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:46.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:46.977 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 5, 2024-09-18T07:35:46.977 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.control", 2024-09-18T07:35:46.977 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:58.322401+0000", 2024-09-18T07:35:46.977 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:35:46.977 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:35:46.978 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:35:46.978 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:35:46.978 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:35:46.978 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:35:46.978 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:35:46.978 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:35:46.978 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:35:46.978 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:35:46.978 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:35:46.978 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:35:46.978 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:35:46.978 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:35:46.979 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:35:46.979 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:35:46.979 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:35:46.979 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:35:46.979 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:35:46.979 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:35:46.979 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:35:46.979 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:35:46.979 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:35:46.979 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:35:46.979 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:35:46.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.979 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "66", 2024-09-18T07:35:46.980 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:35:46.980 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:35:46.980 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:35:46.980 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:35:46.980 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:35:46.980 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:35:46.980 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:35:46.980 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:35:46.980 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:35:46.980 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:35:46.980 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:35:46.980 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:35:46.980 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:35:46.981 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:35:46.981 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:35:46.981 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:35:46.981 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:35:46.981 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:35:46.981 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:35:46.981 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:35:46.981 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:35:46.981 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:35:46.981 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:35:46.981 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:35:46.981 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:35:46.981 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:35:46.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.982 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:35:46.982 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:35:46.982 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:35:46.982 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:35:46.982 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:35:46.982 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:35:46.982 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:35:46.982 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:35:46.982 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:35:46.982 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:35:46.982 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:35:46.982 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:35:46.982 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:35:46.982 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:35:46.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.983 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:35:46.983 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:35:46.983 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.25, 2024-09-18T07:35:46.983 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.25, 2024-09-18T07:35:46.983 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:35:46.983 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.25, 2024-09-18T07:35:46.983 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.25, 2024-09-18T07:35:46.983 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:35:46.983 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:35:46.983 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:35:46.983 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:46.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:46.984 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 6, 2024-09-18T07:35:46.984 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.meta", 2024-09-18T07:35:46.984 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:31:00.390499+0000", 2024-09-18T07:35:46.984 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:35:46.984 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:35:46.984 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:35:46.984 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:35:46.984 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:35:46.984 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:35:46.984 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:35:46.984 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:35:46.984 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:35:46.984 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:35:46.985 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:35:46.985 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:35:46.985 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:35:46.985 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:35:46.985 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:35:46.985 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:35:46.985 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:35:46.985 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:35:46.985 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:35:46.985 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:35:46.985 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:35:46.985 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:35:46.985 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:35:46.985 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:35:46.986 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:35:46.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.986 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "68", 2024-09-18T07:35:46.986 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:35:46.986 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:35:46.986 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:35:46.986 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:35:46.986 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:35:46.986 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:35:46.986 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:35:46.986 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:35:46.986 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:35:46.986 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:35:46.987 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:35:46.987 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:35:46.987 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:35:46.987 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:35:46.987 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:35:46.987 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:35:46.987 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:35:46.987 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:35:46.987 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:35:46.987 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:35:46.987 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:35:46.987 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:35:46.987 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:35:46.988 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:35:46.988 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:35:46.988 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:35:46.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.988 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:35:46.988 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:35:46.988 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:35:46.988 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:35:46.988 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:35:46.988 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:35:46.988 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:35:46.988 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:35:46.988 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:35:46.989 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:35:46.989 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:35:46.989 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:35:46.989 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_bias": 4 2024-09-18T07:35:46.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.989 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:35:46.989 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:35:46.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.989 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:35:46.989 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:35:46.989 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.5, 2024-09-18T07:35:46.989 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.5, 2024-09-18T07:35:46.990 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:35:46.990 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.5, 2024-09-18T07:35:46.990 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.5, 2024-09-18T07:35:46.990 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:35:46.990 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:35:46.990 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:35:46.990 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:46.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:46.990 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-18T07:35:46.990 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "unique_pool_0", 2024-09-18T07:35:46.990 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:32:06.616722+0000", 2024-09-18T07:35:46.990 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:35:46.991 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:35:46.991 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:35:46.991 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:35:46.991 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:35:46.991 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:35:46.991 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:35:46.991 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:35:46.991 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:35:46.991 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:35:46.991 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:35:46.991 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:35:46.991 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:35:46.991 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 16, 2024-09-18T07:35:46.991 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 16, 2024-09-18T07:35:46.992 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 16, 2024-09-18T07:35:46.992 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 16, 2024-09-18T07:35:46.992 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 16, 2024-09-18T07:35:46.992 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:35:46.992 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "7.10", 2024-09-18T07:35:46.992 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 184, 2024-09-18T07:35:46.992 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 183, 2024-09-18T07:35:46.992 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 183, 2024-09-18T07:35:46.992 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "181'9339", 2024-09-18T07:35:46.992 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "181'9284" 2024-09-18T07:35:46.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.992 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "184", 2024-09-18T07:35:46.992 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:35:46.993 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "184", 2024-09-18T07:35:46.993 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "182", 2024-09-18T07:35:46.993 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:35:46.993 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:35:46.993 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:35:46.993 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:35:46.993 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:35:46.993 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:35:46.993 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:35:46.993 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:35:46.993 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:35:46.993 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:35:46.993 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:35:46.994 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:35:46.994 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:35:46.994 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:35:46.994 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:35:46.994 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:35:46.994 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:35:46.994 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:35:46.994 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:35:46.994 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:35:46.995 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:35:46.995 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:35:46.995 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:35:46.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.995 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:35:46.995 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:35:46.995 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:35:46.995 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:35:46.995 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:35:46.995 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:35:46.995 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:35:46.995 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:35:46.995 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:35:46.996 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:35:46.996 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:35:46.996 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:35:46.996 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:35:46.996 INFO:teuthology.orchestra.run.smithi137.stdout: "rados": {} 2024-09-18T07:35:46.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.996 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:35:46.996 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:35:46.996 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2, 2024-09-18T07:35:46.996 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2, 2024-09-18T07:35:46.996 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:35:46.996 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2, 2024-09-18T07:35:46.996 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2, 2024-09-18T07:35:46.997 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-18T07:35:46.997 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:35:46.997 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-18T07:35:46.997 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:46.997 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:46.997 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:35:46.997 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-18T07:35:46.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:46.997 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:35:46.997 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "268cd580-c931-4992-bb93-d6f63c2bfe88", 2024-09-18T07:35:46.997 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:35:46.997 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:35:46.998 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:35:46.998 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:35:46.998 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:35:46.998 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:35:46.998 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 8, 2024-09-18T07:35:46.998 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 194, 2024-09-18T07:35:46.998 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:35:46.998 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:35:46.998 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:35:46.998 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:46.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:46.998 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:46.998 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6801", 2024-09-18T07:35:46.998 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:35:46.999 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:46.999 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:46.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.999 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:35:46.999 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:46.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:46.999 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:46.999 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6802", 2024-09-18T07:35:46.999 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:35:46.999 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:46.999 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:46.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:46.999 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:35:47.000 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.000 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.000 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6804", 2024-09-18T07:35:47.000 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:35:47.000 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.000 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.000 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:35:47.000 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.000 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.001 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6803", 2024-09-18T07:35:47.001 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:35:47.001 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.001 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.001 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6801/2811444477", 2024-09-18T07:35:47.001 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6802/2811444477", 2024-09-18T07:35:47.001 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6804/2811444477", 2024-09-18T07:35:47.001 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6803/2811444477", 2024-09-18T07:35:47.001 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:35:47.001 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:35:47.001 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:35:47.001 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.002 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:35:47.002 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575", 2024-09-18T07:35:47.002 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:35:47.002 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:35:47.002 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:35:47.002 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:35:47.002 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:35:47.002 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:35:47.002 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 13, 2024-09-18T07:35:47.002 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 172, 2024-09-18T07:35:47.003 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:35:47.003 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:35:47.003 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:35:47.003 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.003 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.003 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6805", 2024-09-18T07:35:47.003 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:35:47.003 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.003 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.003 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:35:47.003 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.004 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.004 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6806", 2024-09-18T07:35:47.004 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:35:47.004 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.004 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.004 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:35:47.004 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.004 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.004 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6808", 2024-09-18T07:35:47.004 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:35:47.005 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.005 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.005 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:35:47.005 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.005 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.005 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6807", 2024-09-18T07:35:47.005 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:35:47.005 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.005 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.005 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6805/874102487", 2024-09-18T07:35:47.006 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6806/874102487", 2024-09-18T07:35:47.006 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6808/874102487", 2024-09-18T07:35:47.006 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6807/874102487", 2024-09-18T07:35:47.006 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:35:47.006 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:35:47.006 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:35:47.006 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.006 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:35:47.006 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "480ad6c5-0e0b-4405-bbe2-381682d00bae", 2024-09-18T07:35:47.006 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:35:47.006 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:35:47.007 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:35:47.007 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:35:47.007 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:35:47.007 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:35:47.007 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 18, 2024-09-18T07:35:47.007 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 181, 2024-09-18T07:35:47.007 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:35:47.007 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:35:47.007 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:35:47.007 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.007 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.007 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6809", 2024-09-18T07:35:47.008 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:35:47.008 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.008 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.008 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:35:47.008 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.008 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.008 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6810", 2024-09-18T07:35:47.008 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:35:47.008 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.008 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.009 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:35:47.009 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.009 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.009 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6812", 2024-09-18T07:35:47.009 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:35:47.009 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.009 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.009 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:35:47.009 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.010 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.010 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6811", 2024-09-18T07:35:47.010 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:35:47.010 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.010 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.010 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6809/1754684686", 2024-09-18T07:35:47.010 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6810/1754684686", 2024-09-18T07:35:47.010 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6812/1754684686", 2024-09-18T07:35:47.010 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6811/1754684686", 2024-09-18T07:35:47.010 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:35:47.010 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:35:47.010 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:35:47.011 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.011 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:35:47.011 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14", 2024-09-18T07:35:47.011 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:35:47.011 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:35:47.011 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:35:47.011 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:35:47.011 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:35:47.011 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:35:47.011 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 26, 2024-09-18T07:35:47.011 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 176, 2024-09-18T07:35:47.012 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:35:47.012 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:35:47.012 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:35:47.012 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.012 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.012 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6813", 2024-09-18T07:35:47.012 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:35:47.012 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.012 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.012 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:35:47.012 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.013 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.013 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6814", 2024-09-18T07:35:47.013 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:35:47.013 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.013 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.013 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:35:47.013 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.013 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.013 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6816", 2024-09-18T07:35:47.013 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:35:47.014 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.014 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.014 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:35:47.014 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.014 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.014 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6815", 2024-09-18T07:35:47.014 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:35:47.014 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.014 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.015 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6813/116058392", 2024-09-18T07:35:47.015 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6814/116058392", 2024-09-18T07:35:47.015 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6816/116058392", 2024-09-18T07:35:47.015 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6815/116058392", 2024-09-18T07:35:47.015 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:35:47.015 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:35:47.015 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:35:47.015 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.015 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:35:47.015 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de", 2024-09-18T07:35:47.015 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:35:47.015 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:35:47.016 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:35:47.016 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:35:47.016 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:35:47.016 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:35:47.016 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 32, 2024-09-18T07:35:47.016 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 180, 2024-09-18T07:35:47.016 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:35:47.016 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:35:47.016 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:35:47.016 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.016 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.016 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6800", 2024-09-18T07:35:47.017 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:35:47.017 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.017 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.017 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:35:47.017 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.017 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.017 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6801", 2024-09-18T07:35:47.017 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:35:47.017 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.017 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.018 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:35:47.018 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.018 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.018 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6803", 2024-09-18T07:35:47.018 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:35:47.018 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.018 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.018 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:35:47.018 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.018 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.018 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6802", 2024-09-18T07:35:47.019 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:35:47.019 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.019 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.019 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6800/3654297954", 2024-09-18T07:35:47.019 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6801/3654297954", 2024-09-18T07:35:47.019 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6803/3654297954", 2024-09-18T07:35:47.019 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6802/3654297954", 2024-09-18T07:35:47.019 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:35:47.019 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:35:47.019 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:35:47.019 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.020 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:35:47.020 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c", 2024-09-18T07:35:47.020 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:35:47.020 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:35:47.020 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:35:47.020 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:35:47.020 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:35:47.020 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:35:47.020 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 39, 2024-09-18T07:35:47.020 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 184, 2024-09-18T07:35:47.020 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:35:47.020 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:35:47.020 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:35:47.021 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.021 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.021 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6804", 2024-09-18T07:35:47.021 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:35:47.021 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.021 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.021 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:35:47.021 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.021 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.021 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6805", 2024-09-18T07:35:47.022 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:35:47.022 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.022 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.022 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:35:47.022 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.022 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.022 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:47.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:47.023 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6807", 2024-09-18T07:35:47.023 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:35:47.023 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.024 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.024 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:35:47.024 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.024 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.024 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6806", 2024-09-18T07:35:47.024 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:35:47.024 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.024 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.024 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6804/837994222", 2024-09-18T07:35:47.025 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6805/837994222", 2024-09-18T07:35:47.025 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6807/837994222", 2024-09-18T07:35:47.025 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6806/837994222", 2024-09-18T07:35:47.025 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:35:47.025 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:35:47.025 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:35:47.025 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.025 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:35:47.025 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6", 2024-09-18T07:35:47.025 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:35:47.025 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:35:47.026 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:35:47.026 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:35:47.026 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:35:47.026 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:35:47.026 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 47, 2024-09-18T07:35:47.026 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 185, 2024-09-18T07:35:47.026 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:35:47.026 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:35:47.026 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:35:47.026 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.026 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.026 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6808", 2024-09-18T07:35:47.026 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:35:47.027 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.027 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.027 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:35:47.027 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.027 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.027 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6809", 2024-09-18T07:35:47.027 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:35:47.027 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.027 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.027 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:35:47.028 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.028 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.028 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6811", 2024-09-18T07:35:47.028 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:35:47.028 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.028 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.028 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:35:47.028 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.028 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.029 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6810", 2024-09-18T07:35:47.029 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:35:47.029 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.029 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.029 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6808/1494175466", 2024-09-18T07:35:47.029 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6809/1494175466", 2024-09-18T07:35:47.029 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6811/1494175466", 2024-09-18T07:35:47.029 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6810/1494175466", 2024-09-18T07:35:47.029 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:35:47.029 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:35:47.029 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:35:47.030 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.030 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:35:47.030 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a", 2024-09-18T07:35:47.030 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:35:47.030 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:35:47.030 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:35:47.030 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:35:47.030 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:35:47.030 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:35:47.030 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 55, 2024-09-18T07:35:47.030 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 181, 2024-09-18T07:35:47.031 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:35:47.031 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:35:47.031 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:35:47.031 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.031 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.031 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6812", 2024-09-18T07:35:47.031 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:35:47.031 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.031 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.031 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:35:47.031 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.032 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.032 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6813", 2024-09-18T07:35:47.032 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:35:47.032 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.032 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.032 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:35:47.032 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.032 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.032 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6815", 2024-09-18T07:35:47.033 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:35:47.033 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.033 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.033 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:35:47.033 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:35:47.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.033 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:35:47.033 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6814", 2024-09-18T07:35:47.033 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:35:47.033 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.033 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.034 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6812/2670809315", 2024-09-18T07:35:47.034 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6813/2670809315", 2024-09-18T07:35:47.034 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6815/2670809315", 2024-09-18T07:35:47.034 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6814/2670809315", 2024-09-18T07:35:47.034 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:35:47.034 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:35:47.034 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:35:47.034 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.034 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.034 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:35:47.034 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_xinfo": [ 2024-09-18T07:35:47.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.034 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:35:47.035 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:35:47.035 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:35:47.035 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:35:47.035 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:35:47.035 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:35:47.035 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:34:43.271064+0000", 2024-09-18T07:35:47.035 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:35:47.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.035 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.035 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:35:47.035 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:35:47.035 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:35:47.036 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:35:47.036 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:35:47.036 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:35:47.036 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:35:18.240536+0000", 2024-09-18T07:35:47.036 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:35:47.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.036 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:35:47.036 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:35:47.036 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:35:47.036 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:35:47.036 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:35:47.036 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:35:47.037 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:34:45.346036+0000", 2024-09-18T07:35:47.037 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:35:47.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.037 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:35:47.037 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:35:47.037 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:35:47.037 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:35:47.037 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:35:47.037 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:35:47.037 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:35:16.834298+0000", 2024-09-18T07:35:47.037 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:35:47.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.038 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:35:47.038 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:35:47.038 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:35:47.038 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:35:47.038 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:35:47.038 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:35:47.038 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:35:26.641371+0000", 2024-09-18T07:35:47.038 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:35:47.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.038 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:35:47.039 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:35:47.039 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:35:47.039 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:35:47.039 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:35:47.039 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:35:47.039 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:34:42.334024+0000", 2024-09-18T07:35:47.039 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:35:47.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.039 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:35:47.039 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:35:47.039 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:35:47.040 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:35:47.040 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:35:47.040 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:35:47.040 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:35:09.257096+0000", 2024-09-18T07:35:47.040 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:35:47.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.040 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:35:47.040 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:35:47.040 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:35:47.040 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:35:47.040 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:35:47.040 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:35:47.041 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:35:22.029922+0000", 2024-09-18T07:35:47.041 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:35:47.041 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.041 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:35:47.041 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap": [], 2024-09-18T07:35:47.041 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_items": [ 2024-09-18T07:35:47.041 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.041 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.6", 2024-09-18T07:35:47.041 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:35:47.041 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.041 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:35:47.041 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:35:47.041 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.042 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.042 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.e", 2024-09-18T07:35:47.042 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:35:47.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.042 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:35:47.042 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:35:47.042 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.042 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.043 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.10", 2024-09-18T07:35:47.043 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:35:47.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.043 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:35:47.043 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:35:47.043 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.043 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.043 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.12", 2024-09-18T07:35:47.043 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:35:47.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.043 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:35:47.043 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:35:47.044 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.044 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.044 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.044 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.18", 2024-09-18T07:35:47.044 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:35:47.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.044 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:35:47.044 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:35:47.044 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.044 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.044 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.045 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.2", 2024-09-18T07:35:47.045 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:35:47.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.045 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 6, 2024-09-18T07:35:47.045 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:35:47.045 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.045 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.045 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.b", 2024-09-18T07:35:47.045 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:35:47.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.046 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 0, 2024-09-18T07:35:47.046 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 3 2024-09-18T07:35:47.046 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.046 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.046 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.4", 2024-09-18T07:35:47.046 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:35:47.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.046 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:35:47.046 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:35:47.046 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.046 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.047 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.1b", 2024-09-18T07:35:47.047 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:35:47.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:47.047 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:35:47.047 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:35:47.047 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.047 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:47.047 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.047 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:35:47.047 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_primaries": [], 2024-09-18T07:35:47.047 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_temp": [], 2024-09-18T07:35:47.048 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_temp": [], 2024-09-18T07:35:47.048 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklist": { 2024-09-18T07:35:47.048 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/2096039404": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:35:47.048 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/3636177939": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:35:47.048 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/334967443": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:35:47.048 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2101286111": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:35:47.048 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3387637412": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:35:47.048 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762989503": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:35:47.048 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3490817671": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:35:47.048 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1482147392": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:35:47.048 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1821482181": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:35:47.048 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/4028304023": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:35:47.048 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/621434521": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:35:47.049 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3160704935": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:35:47.049 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2357293499": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:35:47.049 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762352860": "2024-09-19T07:31:25.389124+0000" 2024-09-18T07:35:47.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.049 INFO:teuthology.orchestra.run.smithi137.stdout: "range_blocklist": {}, 2024-09-18T07:35:47.049 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profiles": { 2024-09-18T07:35:47.049 INFO:teuthology.orchestra.run.smithi137.stdout: "default": { 2024-09-18T07:35:47.049 INFO:teuthology.orchestra.run.smithi137.stdout: "crush-failure-domain": "osd", 2024-09-18T07:35:47.049 INFO:teuthology.orchestra.run.smithi137.stdout: "k": "2", 2024-09-18T07:35:47.049 INFO:teuthology.orchestra.run.smithi137.stdout: "m": "1", 2024-09-18T07:35:47.049 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin": "jerasure", 2024-09-18T07:35:47.049 INFO:teuthology.orchestra.run.smithi137.stdout: "technique": "reed_sol_van" 2024-09-18T07:35:47.049 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:47.050 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps_queue": [], 2024-09-18T07:35:47.050 INFO:teuthology.orchestra.run.smithi137.stdout: "new_removed_snaps": [], 2024-09-18T07:35:47.050 INFO:teuthology.orchestra.run.smithi137.stdout: "new_purged_snaps": [], 2024-09-18T07:35:47.050 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_node_flags": {}, 2024-09-18T07:35:47.050 INFO:teuthology.orchestra.run.smithi137.stdout: "device_class_flags": {}, 2024-09-18T07:35:47.050 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode": { 2024-09-18T07:35:47.050 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_enabled": false, 2024-09-18T07:35:47.050 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_bucket_count": 0, 2024-09-18T07:35:47.050 INFO:teuthology.orchestra.run.smithi137.stdout: "degraded_stretch_mode": 0, 2024-09-18T07:35:47.050 INFO:teuthology.orchestra.run.smithi137.stdout: "recovering_stretch_mode": 0, 2024-09-18T07:35:47.050 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_bucket": 0 2024-09-18T07:35:47.050 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:47.051 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:35:47.051 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:35:47.051 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:35:47.051 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:35:47.052 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:35:47.052 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:35:47.052 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:35:47.053 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:46 smithi137 ceph-mon[29408]: pgmap v257: 148 pgs: 148 active+clean; 163 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 766 B/s rd, 0 B/s wr, 819 op/s; 776 KiB/s, 12 objects/s recovering 2024-09-18T07:35:47.053 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:46 smithi137 ceph-mon[39175]: pgmap v257: 148 pgs: 148 active+clean; 163 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 766 B/s rd, 0 B/s wr, 819 op/s; 776 KiB/s, 12 objects/s recovering 2024-09-18T07:35:47.053 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:35:47.053 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:35:47.054 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:35:47.054 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:35:47.069 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:35:47.069 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:35:47.085 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:35:47.085 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:35:47.102 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:35:47.102 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:35:47.124 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:47.150 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:47.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:47.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:46 smithi203 ceph-mon[32237]: pgmap v257: 148 pgs: 148 active+clean; 163 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 766 B/s rd, 0 B/s wr, 819 op/s; 776 KiB/s, 12 objects/s recovering 2024-09-18T07:35:47.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:35:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:35:47.251 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:47.277 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:47.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:47.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:47.341 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:35:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:35:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:35:47.379 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:47.403 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:47.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:47.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:47.505 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:47.529 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:47.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:47.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:47.631 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:47.655 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:47.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:47.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:47.757 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:47.778 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:47.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:47.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:47.881 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:47.906 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:47.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:48.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:48.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:48.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:48.132 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:48.157 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:48.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:48.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:48.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:47 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1530572404' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:35:48.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:47 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/740395941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:48.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:47 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1530572404' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:35:48.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:47 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/740395941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:48.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:47 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1530572404' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:35:48.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:47 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/740395941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:48.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:48.281 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:48.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:48.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:48.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:48.404 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:48.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:48.506 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:48.532 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:48.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:48.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:48.612 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:48.614 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:48.620 INFO:tasks.thrashosds.thrasher:j is {'epoch': 197, 'fsid': 'f7ef099c-758e-11ef-bceb-c7b262605968', 'created': '2024-09-18T07:25:19.662367+0000', 'modified': '2024-09-18T07:35:44.790263+0000', 'last_up_change': '2024-09-18T07:30:44.708496+0000', 'last_in_change': '2024-09-18T07:35:00.369558+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T07:28:20.899655+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-18T07:30:53.715373+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-18T07:30:54.672127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-18T07:30:56.351343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-18T07:30:58.322401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-18T07:31:00.390499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-18T07:32:06.616722+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '7.10', 'ready_epoch': 184, 'last_epoch_started': 183, 'last_epoch_clean': 183, 'source_version': "181'9339", 'target_version': "181'9284"}, 'last_change': '184', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '184', 'last_force_op_resend_preluminous': '182', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '268cd580-c931-4992-bb93-d6f63c2bfe88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 194, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6801', 'nonce': 2811444477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6802', 'nonce': 2811444477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6804', 'nonce': 2811444477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6803', 'nonce': 2811444477}]}, 'public_addr': '172.21.15.137:6801/2811444477', 'cluster_addr': '172.21.15.137:6802/2811444477', 'heartbeat_back_addr': '172.21.15.137:6804/2811444477', 'heartbeat_front_addr': '172.21.15.137:6803/2811444477', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b06907bc-b499-46c0-8667-c5bdc0d85575', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6805', 'nonce': 874102487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6806', 'nonce': 874102487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6808', 'nonce': 874102487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6807', 'nonce': 874102487}]}, 'public_addr': '172.21.15.137:6805/874102487', 'cluster_addr': '172.21.15.137:6806/874102487', 'heartbeat_back_addr': '172.21.15.137:6808/874102487', 'heartbeat_front_addr': '172.21.15.137:6807/874102487', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '480ad6c5-0e0b-4405-bbe2-381682d00bae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 181, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6809', 'nonce': 1754684686}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6810', 'nonce': 1754684686}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6812', 'nonce': 1754684686}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6811', 'nonce': 1754684686}]}, 'public_addr': '172.21.15.137:6809/1754684686', 'cluster_addr': '172.21.15.137:6810/1754684686', 'heartbeat_back_addr': '172.21.15.137:6812/1754684686', 'heartbeat_front_addr': '172.21.15.137:6811/1754684686', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '178be3e0-f07b-4856-807b-2a1181e48a14', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 176, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6813', 'nonce': 116058392}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6814', 'nonce': 116058392}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6816', 'nonce': 116058392}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6815', 'nonce': 116058392}]}, 'public_addr': '172.21.15.137:6813/116058392', 'cluster_addr': '172.21.15.137:6814/116058392', 'heartbeat_back_addr': '172.21.15.137:6816/116058392', 'heartbeat_front_addr': '172.21.15.137:6815/116058392', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7547102c-951a-4077-a094-fb8a65bfe6de', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 180, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6800', 'nonce': 3654297954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6801', 'nonce': 3654297954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6803', 'nonce': 3654297954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6802', 'nonce': 3654297954}]}, 'public_addr': '172.21.15.203:6800/3654297954', 'cluster_addr': '172.21.15.203:6801/3654297954', 'heartbeat_back_addr': '172.21.15.203:6803/3654297954', 'heartbeat_front_addr': '172.21.15.203:6802/3654297954', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '49c17901-8102-48a4-9348-fa1bdb63032c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 184, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6804', 'nonce': 837994222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6805', 'nonce': 837994222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6807', 'nonce': 837994222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6806', 'nonce': 837994222}]}, 'public_addr': '172.21.15.203:6804/837994222', 'cluster_addr': '172.21.15.203:6805/837994222', 'heartbeat_back_addr': '172.21.15.203:6807/837994222', 'heartbeat_front_addr': '172.21.15.203:6806/837994222', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '75f0ead4-e981-4a15-b1c1-d83dce871ab6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 185, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6808', 'nonce': 1494175466}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6809', 'nonce': 1494175466}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6811', 'nonce': 1494175466}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6810', 'nonce': 1494175466}]}, 'public_addr': '172.21.15.203:6808/1494175466', 'cluster_addr': '172.21.15.203:6809/1494175466', 'heartbeat_back_addr': '172.21.15.203:6811/1494175466', 'heartbeat_front_addr': '172.21.15.203:6810/1494175466', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c0dbf299-6798-4b99-9a85-62b123971f2a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 55, 'up_thru': 181, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6812', 'nonce': 2670809315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6813', 'nonce': 2670809315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6815', 'nonce': 2670809315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6814', 'nonce': 2670809315}]}, 'public_addr': '172.21.15.203:6812/2670809315', 'cluster_addr': '172.21.15.203:6813/2670809315', 'heartbeat_back_addr': '172.21.15.203:6815/2670809315', 'heartbeat_front_addr': '172.21.15.203:6814/2670809315', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:34:43.271064+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:35:18.240536+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:34:45.346036+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:35:16.834298+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:35:26.641371+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:34:42.334024+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:35:09.257096+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:35:22.029922+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.2', 'mappings': [{'from': 6, 'to': 5}]}, {'pgid': '5.b', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '6.4', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '6.1b', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.137:6800/2096039404': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:6800/3636177939': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/334967443': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/2101286111': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/3387637412': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1762989503': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3490817671': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:6800/1482147392': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1821482181': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/4028304023': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/621434521': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3160704935': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/2357293499': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/1762352860': '2024-09-19T07:31:25.389124+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T07:35:48.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:35:48.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:48.685 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:48.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:48.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:48.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:48.809 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:48.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:48.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:48.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:48.932 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:48.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:49.033 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:49.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:49.060 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:49.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:49.127 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:48 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:49.127 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:48 smithi137 ceph-mon[29408]: pgmap v258: 148 pgs: 148 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 648 KiB/s, 10 objects/s recovering 2024-09-18T07:35:49.128 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:48 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:49.128 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:48 smithi137 ceph-mon[39175]: pgmap v258: 148 pgs: 148 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 648 KiB/s, 10 objects/s recovering 2024-09-18T07:35:49.161 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:49.187 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:49.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:49.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:49.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:48 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:49.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:48 smithi203 ceph-mon[32237]: pgmap v258: 148 pgs: 148 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 648 KiB/s, 10 objects/s recovering 2024-09-18T07:35:49.284 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:35:49.285 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:35:49.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:35:49.285 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:35:49.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.285 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:20747 7.b 7:d187519a:::benchmark_data_smithi137_74180_object20746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-18T07:35:49.285 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:23.526769+0000", 2024-09-18T07:35:49.285 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 205.75318408800001, 2024-09-18T07:35:49.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10768604800000001, 2024-09-18T07:35:49.285 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.285 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.285 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:49.285 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:49.286 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:49.286 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:49.286 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 20747 2024-09-18T07:35:49.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.286 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526769+0000", 2024-09-18T07:35:49.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526769+0000", 2024-09-18T07:35:49.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526771+0000", 2024-09-18T07:35:49.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2510000000000001e-06 2024-09-18T07:35:49.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526805+0000", 2024-09-18T07:35:49.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3915000000000003e-05 2024-09-18T07:35:49.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526821+0000", 2024-09-18T07:35:49.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6308e-05 2024-09-18T07:35:49.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526829+0000", 2024-09-18T07:35:49.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.481e-06 2024-09-18T07:35:49.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.526888+0000", 2024-09-18T07:35:49.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9434000000000001e-05 2024-09-18T07:35:49.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.527009+0000", 2024-09-18T07:35:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012099699999999999 2024-09-18T07:35:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:35:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.527125+0000", 2024-09-18T07:35:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116107 2024-09-18T07:35:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.528610+0000", 2024-09-18T07:35:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014843829999999999 2024-09-18T07:35:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.529010+0000", 2024-09-18T07:35:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040065400000000002 2024-09-18T07:35:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.634285+0000", 2024-09-18T07:35:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10527474100000001 2024-09-18T07:35:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.634407+0000", 2024-09-18T07:35:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012228099999999999 2024-09-18T07:35:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:23.634455+0000", 2024-09-18T07:35:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7496000000000001e-05 2024-09-18T07:35:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:22340 7.b 7:dcab8849:::benchmark_data_smithi137_74180_object22339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:35:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:24.658442+0000", 2024-09-18T07:35:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 204.62151093400001, 2024-09-18T07:35:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.20618298299999999, 2024-09-18T07:35:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 22340 2024-09-18T07:35:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658442+0000", 2024-09-18T07:35:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658442+0000", 2024-09-18T07:35:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658445+0000", 2024-09-18T07:35:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4609999999999998e-06 2024-09-18T07:35:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658479+0000", 2024-09-18T07:35:49.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4393000000000001e-05 2024-09-18T07:35:49.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658497+0000", 2024-09-18T07:35:49.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7649999999999999e-05 2024-09-18T07:35:49.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658505+0000", 2024-09-18T07:35:49.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5350000000000004e-06 2024-09-18T07:35:49.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658567+0000", 2024-09-18T07:35:49.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1580000000000003e-05 2024-09-18T07:35:49.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:49.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658673+0000", 2024-09-18T07:35:49.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105907 2024-09-18T07:35:49.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:35:49.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658790+0000", 2024-09-18T07:35:49.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117494 2024-09-18T07:35:49.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.660870+0000", 2024-09-18T07:35:49.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020794849999999998 2024-09-18T07:35:49.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:49.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.661103+0000", 2024-09-18T07:35:49.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023344199999999999 2024-09-18T07:35:49.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.864502+0000", 2024-09-18T07:35:49.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.20339850100000001 2024-09-18T07:35:49.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:49.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.864571+0000", 2024-09-18T07:35:49.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9566000000000004e-05 2024-09-18T07:35:49.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.864625+0000", 2024-09-18T07:35:49.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3968999999999998e-05 2024-09-18T07:35:49.297 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.297 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.297 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.298 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:51957 7.b 7:dff3716a:::benchmark_data_smithi137_74180_object51956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:35:49.298 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.955279+0000", 2024-09-18T07:35:49.298 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 178.324673902, 2024-09-18T07:35:49.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.124984595, 2024-09-18T07:35:49.298 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.298 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.298 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:49.298 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:49.298 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:49.298 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:49.298 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 51957 2024-09-18T07:35:49.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.299 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955279+0000", 2024-09-18T07:35:49.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955279+0000", 2024-09-18T07:35:49.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955282+0000", 2024-09-18T07:35:49.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.728e-06 2024-09-18T07:35:49.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955308+0000", 2024-09-18T07:35:49.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6208e-05 2024-09-18T07:35:49.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955325+0000", 2024-09-18T07:35:49.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7071999999999998e-05 2024-09-18T07:35:49.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955335+0000", 2024-09-18T07:35:49.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5740000000000002e-06 2024-09-18T07:35:49.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955433+0000", 2024-09-18T07:35:49.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7992999999999995e-05 2024-09-18T07:35:49.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:49.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955564+0000", 2024-09-18T07:35:49.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013090399999999999 2024-09-18T07:35:49.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:35:49.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.955692+0000", 2024-09-18T07:35:49.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012839399999999999 2024-09-18T07:35:49.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.957760+0000", 2024-09-18T07:35:49.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020681549999999999 2024-09-18T07:35:49.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:49.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.962122+0000", 2024-09-18T07:35:49.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043617259999999998 2024-09-18T07:35:49.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:51.080186+0000", 2024-09-18T07:35:49.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.118063826 2024-09-18T07:35:49.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:49.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:51.080232+0000", 2024-09-18T07:35:49.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6208999999999998e-05 2024-09-18T07:35:49.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:51.080264+0000", 2024-09-18T07:35:49.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1806e-05 2024-09-18T07:35:49.304 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.304 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.304 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.304 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:51965 7.b 7:dcde8c91:::benchmark_data_smithi137_74180_object51964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:35:49.304 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:50.959013+0000", 2024-09-18T07:35:49.304 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 178.32093997699999, 2024-09-18T07:35:49.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.121449817, 2024-09-18T07:35:49.305 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.305 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.305 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:49.305 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:49.305 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:49.305 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:49.305 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 51965 2024-09-18T07:35:49.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.305 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.305 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.305 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959013+0000", 2024-09-18T07:35:49.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959013+0000", 2024-09-18T07:35:49.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959016+0000", 2024-09-18T07:35:49.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8830000000000002e-06 2024-09-18T07:35:49.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959049+0000", 2024-09-18T07:35:49.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3175e-05 2024-09-18T07:35:49.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959067+0000", 2024-09-18T07:35:49.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7623000000000001e-05 2024-09-18T07:35:49.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959076+0000", 2024-09-18T07:35:49.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0149999999999992e-06 2024-09-18T07:35:49.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959138+0000", 2024-09-18T07:35:49.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2133999999999998e-05 2024-09-18T07:35:49.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:49.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959268+0000", 2024-09-18T07:35:49.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000130476 2024-09-18T07:35:49.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:35:49.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.959418+0000", 2024-09-18T07:35:49.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014930499999999999 2024-09-18T07:35:49.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.962111+0000", 2024-09-18T07:35:49.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00269381 2024-09-18T07:35:49.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:49.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:50.962951+0000", 2024-09-18T07:35:49.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00083964200000000004 2024-09-18T07:35:49.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:51.080380+0000", 2024-09-18T07:35:49.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.117428406 2024-09-18T07:35:49.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:49.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:51.080428+0000", 2024-09-18T07:35:49.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.808e-05 2024-09-18T07:35:49.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:51.080463+0000", 2024-09-18T07:35:49.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5268000000000002e-05 2024-09-18T07:35:49.310 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.310 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.310 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.311 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:76515 7.b 7:d3d65bc7:::benchmark_data_smithi137_74180_object76514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:35:49.311 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:18.050800+0000", 2024-09-18T07:35:49.311 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 151.22915277000001, 2024-09-18T07:35:49.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.116946603, 2024-09-18T07:35:49.311 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.311 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.311 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:49.311 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:49.311 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:49.311 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:49.311 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 76515 2024-09-18T07:35:49.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.311 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.050800+0000", 2024-09-18T07:35:49.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.050800+0000", 2024-09-18T07:35:49.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.312 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:49.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.050803+0000", 2024-09-18T07:35:49.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4820000000000001e-06 2024-09-18T07:35:49.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.050827+0000", 2024-09-18T07:35:49.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.461e-05 2024-09-18T07:35:49.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.050843+0000", 2024-09-18T07:35:49.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5764999999999999e-05 2024-09-18T07:35:49.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.050851+0000", 2024-09-18T07:35:49.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9950000000000005e-06 2024-09-18T07:35:49.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.051019+0000", 2024-09-18T07:35:49.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000167472 2024-09-18T07:35:49.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:49.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.051212+0000", 2024-09-18T07:35:49.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000193298 2024-09-18T07:35:49.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:35:49.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.051358+0000", 2024-09-18T07:35:49.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014654499999999999 2024-09-18T07:35:49.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.053758+0000", 2024-09-18T07:35:49.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023991559999999999 2024-09-18T07:35:49.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:49.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.057140+0000", 2024-09-18T07:35:49.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033824520000000002 2024-09-18T07:35:49.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.318 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.167444+0000", 2024-09-18T07:35:49.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.11030403599999999 2024-09-18T07:35:49.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.318 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:49.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.167599+0000", 2024-09-18T07:35:49.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015494700000000001 2024-09-18T07:35:49.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.167747+0000", 2024-09-18T07:35:49.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014784499999999999 2024-09-18T07:35:49.319 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.319 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.319 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.319 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:76523 7.b 7:d3ed4aa3:::benchmark_data_smithi137_74180_object76522:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:35:49.319 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:18.053729+0000", 2024-09-18T07:35:49.319 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 151.22622386200001, 2024-09-18T07:35:49.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.114700012, 2024-09-18T07:35:49.319 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.320 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.320 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:49.320 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:49.320 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:49.320 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:49.320 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 76523 2024-09-18T07:35:49.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.320 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.053729+0000", 2024-09-18T07:35:49.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.053729+0000", 2024-09-18T07:35:49.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.053731+0000", 2024-09-18T07:35:49.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0329999999999998e-06 2024-09-18T07:35:49.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.053755+0000", 2024-09-18T07:35:49.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3838000000000001e-05 2024-09-18T07:35:49.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.053765+0000", 2024-09-18T07:35:49.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8090000000000004e-06 2024-09-18T07:35:49.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.053772+0000", 2024-09-18T07:35:49.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1409999999999999e-06 2024-09-18T07:35:49.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.053917+0000", 2024-09-18T07:35:49.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000145329 2024-09-18T07:35:49.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:49.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.054199+0000", 2024-09-18T07:35:49.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028138499999999999 2024-09-18T07:35:49.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:35:49.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.054595+0000", 2024-09-18T07:35:49.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000396332 2024-09-18T07:35:49.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:49.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.057578+0000", 2024-09-18T07:35:49.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029828210000000001 2024-09-18T07:35:49.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059437+0000", 2024-09-18T07:35:49.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018591790000000001 2024-09-18T07:35:49.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.167938+0000", 2024-09-18T07:35:49.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.108501287 2024-09-18T07:35:49.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:49.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.168288+0000", 2024-09-18T07:35:49.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034918900000000002 2024-09-18T07:35:49.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.168429+0000", 2024-09-18T07:35:49.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000141669 2024-09-18T07:35:49.325 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.325 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.325 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.326 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:76526 7.b 7:d8f2833a:::benchmark_data_smithi137_74180_object76525:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:35:49.326 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:18.059042+0000", 2024-09-18T07:35:49.326 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 151.22091063799999, 2024-09-18T07:35:49.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.110664889, 2024-09-18T07:35:49.326 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.326 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.326 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:49.326 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:49.326 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:49.326 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:49.326 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 76526 2024-09-18T07:35:49.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.327 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059042+0000", 2024-09-18T07:35:49.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059042+0000", 2024-09-18T07:35:49.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059044+0000", 2024-09-18T07:35:49.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7439999999999999e-06 2024-09-18T07:35:49.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059070+0000", 2024-09-18T07:35:49.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6207000000000002e-05 2024-09-18T07:35:49.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059080+0000", 2024-09-18T07:35:49.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9019999999999994e-06 2024-09-18T07:35:49.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059086+0000", 2024-09-18T07:35:49.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9580000000000004e-06 2024-09-18T07:35:49.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059565+0000", 2024-09-18T07:35:49.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00047865400000000002 2024-09-18T07:35:49.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:49.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059760+0000", 2024-09-18T07:35:49.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000194964 2024-09-18T07:35:49.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:35:49.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.059889+0000", 2024-09-18T07:35:49.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012960700000000001 2024-09-18T07:35:49.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:49.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.063135+0000", 2024-09-18T07:35:49.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032460710000000001 2024-09-18T07:35:49.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.063597+0000", 2024-09-18T07:35:49.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000461831 2024-09-18T07:35:49.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.169438+0000", 2024-09-18T07:35:49.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.105840304 2024-09-18T07:35:49.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:49.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.169649+0000", 2024-09-18T07:35:49.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000211459 2024-09-18T07:35:49.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.169707+0000", 2024-09-18T07:35:49.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8187999999999999e-05 2024-09-18T07:35:49.332 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.332 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.332 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.332 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:76896 7.b 7:df325c0c:::benchmark_data_smithi137_74180_object76895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:35:49.332 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:33:18.439439+0000", 2024-09-18T07:35:49.332 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 150.84051426900001, 2024-09-18T07:35:49.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10739975, 2024-09-18T07:35:49.332 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.332 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.333 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:49.333 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:49.333 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:49.333 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:49.333 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 76896 2024-09-18T07:35:49.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.333 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439439+0000", 2024-09-18T07:35:49.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439439+0000", 2024-09-18T07:35:49.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439441+0000", 2024-09-18T07:35:49.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1730000000000002e-06 2024-09-18T07:35:49.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439463+0000", 2024-09-18T07:35:49.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2218999999999999e-05 2024-09-18T07:35:49.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439521+0000", 2024-09-18T07:35:49.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7603999999999999e-05 2024-09-18T07:35:49.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439530+0000", 2024-09-18T07:35:49.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4259999999999992e-06 2024-09-18T07:35:49.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439600+0000", 2024-09-18T07:35:49.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9635999999999995e-05 2024-09-18T07:35:49.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:49.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439726+0000", 2024-09-18T07:35:49.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126477 2024-09-18T07:35:49.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:35:49.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.439870+0000", 2024-09-18T07:35:49.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014343599999999999 2024-09-18T07:35:49.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:49.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.443002+0000", 2024-09-18T07:35:49.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031327719999999998 2024-09-18T07:35:49.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.540144+0000", 2024-09-18T07:35:49.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.097141623999999996 2024-09-18T07:35:49.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.546736+0000", 2024-09-18T07:35:49.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065915189999999997 2024-09-18T07:35:49.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:49.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.546793+0000", 2024-09-18T07:35:49.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7005000000000001e-05 2024-09-18T07:35:49.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:33:18.546839+0000", 2024-09-18T07:35:49.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5859000000000003e-05 2024-09-18T07:35:49.338 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.338 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.338 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.339 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134973 7.15 7:ace4f2fb:::benchmark_data_smithi137_74180_object30557:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e136)", 2024-09-18T07:35:49.339 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:15.967955+0000", 2024-09-18T07:35:49.339 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 93.311997765000001, 2024-09-18T07:35:49.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.149059142, 2024-09-18T07:35:49.339 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.339 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.339 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:35:49.339 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:49.339 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:49.339 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:49.339 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134973 2024-09-18T07:35:49.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.339 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967955+0000", 2024-09-18T07:35:49.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967955+0000", 2024-09-18T07:35:49.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967957+0000", 2024-09-18T07:35:49.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.154e-06 2024-09-18T07:35:49.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967964+0000", 2024-09-18T07:35:49.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1310000000000003e-06 2024-09-18T07:35:49.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967976+0000", 2024-09-18T07:35:49.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2870999999999999e-05 2024-09-18T07:35:49.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967991+0000", 2024-09-18T07:35:49.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4174e-05 2024-09-18T07:35:49.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.969717+0000", 2024-09-18T07:35:49.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001726694 2024-09-18T07:35:49.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:35:49.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.969724+0000", 2024-09-18T07:35:49.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4200000000000004e-06 2024-09-18T07:35:49.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972756+0000", 2024-09-18T07:35:49.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030324610000000002 2024-09-18T07:35:49.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:49.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972766+0000", 2024-09-18T07:35:49.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0047999999999999e-05 2024-09-18T07:35:49.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976318+0000", 2024-09-18T07:35:49.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.003551327 2024-09-18T07:35:49.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:49.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976329+0000", 2024-09-18T07:35:49.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1413000000000001e-05 2024-09-18T07:35:49.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.116866+0000", 2024-09-18T07:35:49.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.140536993 2024-09-18T07:35:49.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.117014+0000", 2024-09-18T07:35:49.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014845599999999999 2024-09-18T07:35:49.345 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.345 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.345 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.345 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134975 7.15 7:a9f95f32:::benchmark_data_smithi137_74180_object30559:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e136)", 2024-09-18T07:35:49.345 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:15.968007+0000", 2024-09-18T07:35:49.345 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 93.311946061, 2024-09-18T07:35:49.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1497606920000001, 2024-09-18T07:35:49.345 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.345 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.345 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:35:49.345 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:49.346 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:49.346 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:49.346 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134975 2024-09-18T07:35:49.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.346 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968007+0000", 2024-09-18T07:35:49.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968007+0000", 2024-09-18T07:35:49.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968008+0000", 2024-09-18T07:35:49.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9100000000000003e-07 2024-09-18T07:35:49.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968011+0000", 2024-09-18T07:35:49.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0589999999999998e-06 2024-09-18T07:35:49.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968017+0000", 2024-09-18T07:35:49.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5050000000000004e-06 2024-09-18T07:35:49.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968023+0000", 2024-09-18T07:35:49.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6400000000000002e-06 2024-09-18T07:35:49.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.970885+0000", 2024-09-18T07:35:49.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002862505 2024-09-18T07:35:49.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-18T07:35:49.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.970892+0000", 2024-09-18T07:35:49.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2890000000000003e-06 2024-09-18T07:35:49.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972803+0000", 2024-09-18T07:35:49.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019118270000000001 2024-09-18T07:35:49.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:49.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972812+0000", 2024-09-18T07:35:49.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.653e-06 2024-09-18T07:35:49.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976352+0000", 2024-09-18T07:35:49.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0035398280000001 2024-09-18T07:35:49.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:49.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976359+0000", 2024-09-18T07:35:49.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3520000000000001e-06 2024-09-18T07:35:49.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.117670+0000", 2024-09-18T07:35:49.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.141310979 2024-09-18T07:35:49.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.117768+0000", 2024-09-18T07:35:49.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7363999999999998e-05 2024-09-18T07:35:49.352 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.352 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.352 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.352 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134979 7.15 7:af63a923:::benchmark_data_smithi137_74180_object30563:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e136)", 2024-09-18T07:35:49.352 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:15.968030+0000", 2024-09-18T07:35:49.352 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 93.311922984000006, 2024-09-18T07:35:49.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1501027640000001, 2024-09-18T07:35:49.352 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.352 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.352 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:35:49.353 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:49.353 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:49.353 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:49.353 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134979 2024-09-18T07:35:49.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.353 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968030+0000", 2024-09-18T07:35:49.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968030+0000", 2024-09-18T07:35:49.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968031+0000", 2024-09-18T07:35:49.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5000000000000003e-07 2024-09-18T07:35:49.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968033+0000", 2024-09-18T07:35:49.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1969999999999999e-06 2024-09-18T07:35:49.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968038+0000", 2024-09-18T07:35:49.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0919999999999997e-06 2024-09-18T07:35:49.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968043+0000", 2024-09-18T07:35:49.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6619999999999996e-06 2024-09-18T07:35:49.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.970922+0000", 2024-09-18T07:35:49.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028795100000000001 2024-09-18T07:35:49.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-18T07:35:49.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.970927+0000", 2024-09-18T07:35:49.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0320000000000003e-06 2024-09-18T07:35:49.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972841+0000", 2024-09-18T07:35:49.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001913802 2024-09-18T07:35:49.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:49.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972849+0000", 2024-09-18T07:35:49.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9039999999999992e-06 2024-09-18T07:35:49.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976382+0000", 2024-09-18T07:35:49.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0035336779999999 2024-09-18T07:35:49.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:49.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976390+0000", 2024-09-18T07:35:49.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6399999999999997e-06 2024-09-18T07:35:49.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.118071+0000", 2024-09-18T07:35:49.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.141680993 2024-09-18T07:35:49.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.118133+0000", 2024-09-18T07:35:49.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1704000000000004e-05 2024-09-18T07:35:49.358 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.358 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.358 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.358 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134988 7.12 7:4ba1d377:::benchmark_data_smithi137_74180_object30572:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:35:49.358 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964095+0000", 2024-09-18T07:35:49.359 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 92.315858509999998, 2024-09-18T07:35:49.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2226311409999999, 2024-09-18T07:35:49.359 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.359 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.359 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:49.359 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:49.359 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:49.359 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:49.359 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134988 2024-09-18T07:35:49.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.359 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964095+0000", 2024-09-18T07:35:49.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964095+0000", 2024-09-18T07:35:49.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964096+0000", 2024-09-18T07:35:49.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.759e-06 2024-09-18T07:35:49.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964103+0000", 2024-09-18T07:35:49.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7950000000000002e-06 2024-09-18T07:35:49.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964118+0000", 2024-09-18T07:35:49.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5082999999999999e-05 2024-09-18T07:35:49.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964128+0000", 2024-09-18T07:35:49.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6900000000000004e-06 2024-09-18T07:35:49.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964200+0000", 2024-09-18T07:35:49.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2656000000000004e-05 2024-09-18T07:35:49.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:35:49.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964207+0000", 2024-09-18T07:35:49.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5320000000000003e-06 2024-09-18T07:35:49.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976866+0000", 2024-09-18T07:35:49.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012658763 2024-09-18T07:35:49.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:49.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976874+0000", 2024-09-18T07:35:49.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6459999999999999e-06 2024-09-18T07:35:49.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.174506+0000", 2024-09-18T07:35:49.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1976311799999999 2024-09-18T07:35:49.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:49.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.174658+0000", 2024-09-18T07:35:49.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000152516 2024-09-18T07:35:49.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,3,6", 2024-09-18T07:35:49.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.174933+0000", 2024-09-18T07:35:49.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027465900000000001 2024-09-18T07:35:49.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184279+0000", 2024-09-18T07:35:49.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093465449999999995 2024-09-18T07:35:49.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184325+0000", 2024-09-18T07:35:49.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5219999999999997e-05 2024-09-18T07:35:49.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184838+0000", 2024-09-18T07:35:49.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00051383799999999997 2024-09-18T07:35:49.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:49.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185572+0000", 2024-09-18T07:35:49.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00073320099999999995 2024-09-18T07:35:49.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186540+0000", 2024-09-18T07:35:49.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00096810500000000003 2024-09-18T07:35:49.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:49.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186603+0000", 2024-09-18T07:35:49.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3411000000000007e-05 2024-09-18T07:35:49.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186726+0000", 2024-09-18T07:35:49.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012254200000000001 2024-09-18T07:35:49.367 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.367 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.367 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.367 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134989 7.12 7:4aa85d7f:::benchmark_data_smithi137_74180_object30573:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:35:49.367 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964171+0000", 2024-09-18T07:35:49.367 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 92.315782232999993, 2024-09-18T07:35:49.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.222781653, 2024-09-18T07:35:49.367 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.367 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.368 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:49.368 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:49.368 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:49.368 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:49.368 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134989 2024-09-18T07:35:49.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.368 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964171+0000", 2024-09-18T07:35:49.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964171+0000", 2024-09-18T07:35:49.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964172+0000", 2024-09-18T07:35:49.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6899999999999996e-07 2024-09-18T07:35:49.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964176+0000", 2024-09-18T07:35:49.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1829999999999996e-06 2024-09-18T07:35:49.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964184+0000", 2024-09-18T07:35:49.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8290000000000008e-06 2024-09-18T07:35:49.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964190+0000", 2024-09-18T07:35:49.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8850000000000004e-06 2024-09-18T07:35:49.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964252+0000", 2024-09-18T07:35:49.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2842000000000006e-05 2024-09-18T07:35:49.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-18T07:35:49.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964257+0000", 2024-09-18T07:35:49.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4379999999999997e-06 2024-09-18T07:35:49.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.372 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-18T07:35:49.372 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-18T07:35:49.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976922+0000", 2024-09-18T07:35:49.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012665493 2024-09-18T07:35:49.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:49.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976933+0000", 2024-09-18T07:35:49.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0407999999999999e-05 2024-09-18T07:35:49.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.176181+0000", 2024-09-18T07:35:49.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1992481429999999 2024-09-18T07:35:49.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:49.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.176304+0000", 2024-09-18T07:35:49.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012344999999999999 2024-09-18T07:35:49.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,3,6", 2024-09-18T07:35:49.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.176507+0000", 2024-09-18T07:35:49.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020292699999999999 2024-09-18T07:35:49.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184373+0000", 2024-09-18T07:35:49.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0078657629999999996 2024-09-18T07:35:49.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184427+0000", 2024-09-18T07:35:49.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3863999999999998e-05 2024-09-18T07:35:49.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184878+0000", 2024-09-18T07:35:49.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00045142400000000002 2024-09-18T07:35:49.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:49.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185578+0000", 2024-09-18T07:35:49.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00069919800000000005 2024-09-18T07:35:49.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186770+0000", 2024-09-18T07:35:49.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011924430000000001 2024-09-18T07:35:49.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:49.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186830+0000", 2024-09-18T07:35:49.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0003000000000001e-05 2024-09-18T07:35:49.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186952+0000", 2024-09-18T07:35:49.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012239099999999999 2024-09-18T07:35:49.377 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.377 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.377 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.378 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135012 7.12 7:4b131569:::benchmark_data_smithi137_74180_object30596:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:35:49.378 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964206+0000", 2024-09-18T07:35:49.378 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 92.315747221999999, 2024-09-18T07:35:49.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2234539019999999, 2024-09-18T07:35:49.378 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.378 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.378 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:49.378 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:49.378 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:49.378 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:49.378 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135012 2024-09-18T07:35:49.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.378 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964206+0000", 2024-09-18T07:35:49.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964206+0000", 2024-09-18T07:35:49.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964207+0000", 2024-09-18T07:35:49.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6899999999999996e-07 2024-09-18T07:35:49.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964210+0000", 2024-09-18T07:35:49.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4419999999999998e-06 2024-09-18T07:35:49.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964217+0000", 2024-09-18T07:35:49.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4989999999999999e-06 2024-09-18T07:35:49.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964222+0000", 2024-09-18T07:35:49.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5670000000000002e-06 2024-09-18T07:35:49.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964287+0000", 2024-09-18T07:35:49.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5174999999999999e-05 2024-09-18T07:35:49.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-18T07:35:49.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964293+0000", 2024-09-18T07:35:49.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7209999999999999e-06 2024-09-18T07:35:49.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976971+0000", 2024-09-18T07:35:49.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012677541000000001 2024-09-18T07:35:49.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:35:49.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976991+0000", 2024-09-18T07:35:49.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0231999999999999e-05 2024-09-18T07:35:49.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.177610+0000", 2024-09-18T07:35:49.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.200619417 2024-09-18T07:35:49.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:49.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.177725+0000", 2024-09-18T07:35:49.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114815 2024-09-18T07:35:49.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,3,6", 2024-09-18T07:35:49.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.177908+0000", 2024-09-18T07:35:49.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000182683 2024-09-18T07:35:49.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184585+0000", 2024-09-18T07:35:49.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066772230000000004 2024-09-18T07:35:49.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184630+0000", 2024-09-18T07:35:49.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4642999999999999e-05 2024-09-18T07:35:49.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185558+0000", 2024-09-18T07:35:49.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00092809900000000005 2024-09-18T07:35:49.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.187003+0000", 2024-09-18T07:35:49.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001445483 2024-09-18T07:35:49.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:49.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.187470+0000", 2024-09-18T07:35:49.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00046721600000000001 2024-09-18T07:35:49.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:49.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.187517+0000", 2024-09-18T07:35:49.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6619000000000002e-05 2024-09-18T07:35:49.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.187660+0000", 2024-09-18T07:35:49.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014275799999999999 2024-09-18T07:35:49.386 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.386 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.386 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.386 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135121 7.15 7:af3f9edb:::benchmark_data_smithi137_74180_object30705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-18T07:35:49.386 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:18.264638+0000", 2024-09-18T07:35:49.386 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 91.015314787999998, 2024-09-18T07:35:49.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13365268899999999, 2024-09-18T07:35:49.387 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.387 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.387 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:49.387 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:49.387 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:49.387 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:49.387 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135121 2024-09-18T07:35:49.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.387 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.264638+0000", 2024-09-18T07:35:49.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.264638+0000", 2024-09-18T07:35:49.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.264640+0000", 2024-09-18T07:35:49.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4419999999999999e-06 2024-09-18T07:35:49.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.264645+0000", 2024-09-18T07:35:49.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.553e-06 2024-09-18T07:35:49.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.264655+0000", 2024-09-18T07:35:49.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0183999999999999e-05 2024-09-18T07:35:49.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.264662+0000", 2024-09-18T07:35:49.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0199999999999997e-06 2024-09-18T07:35:49.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.267451+0000", 2024-09-18T07:35:49.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002788329 2024-09-18T07:35:49.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:49.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.267632+0000", 2024-09-18T07:35:49.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000181702 2024-09-18T07:35:49.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,5", 2024-09-18T07:35:49.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.267927+0000", 2024-09-18T07:35:49.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029443500000000001 2024-09-18T07:35:49.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.269504+0000", 2024-09-18T07:35:49.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001577436 2024-09-18T07:35:49.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:49.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.273182+0000", 2024-09-18T07:35:49.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036773769999999999 2024-09-18T07:35:49.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.274472+0000", 2024-09-18T07:35:49.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012899400000000001 2024-09-18T07:35:49.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.394864+0000", 2024-09-18T07:35:49.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.120392206 2024-09-18T07:35:49.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:49.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.398187+0000", 2024-09-18T07:35:49.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033230540000000002 2024-09-18T07:35:49.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.398291+0000", 2024-09-18T07:35:49.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104011 2024-09-18T07:35:49.393 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.393 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.393 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.393 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135123 7.15 7:abc15c02:::benchmark_data_smithi137_74180_object30707:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-18T07:35:49.393 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:18.269573+0000", 2024-09-18T07:35:49.393 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 91.010380165000001, 2024-09-18T07:35:49.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13525563600000001, 2024-09-18T07:35:49.394 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.394 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.394 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:49.394 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:49.394 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:49.394 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:49.394 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135123 2024-09-18T07:35:49.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.394 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.269573+0000", 2024-09-18T07:35:49.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.269573+0000", 2024-09-18T07:35:49.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.269584+0000", 2024-09-18T07:35:49.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0654e-05 2024-09-18T07:35:49.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.269589+0000", 2024-09-18T07:35:49.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.75e-06 2024-09-18T07:35:49.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.269599+0000", 2024-09-18T07:35:49.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3239999999999992e-06 2024-09-18T07:35:49.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.269605+0000", 2024-09-18T07:35:49.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9399999999999999e-06 2024-09-18T07:35:49.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.270146+0000", 2024-09-18T07:35:49.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000541487 2024-09-18T07:35:49.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:49.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.270263+0000", 2024-09-18T07:35:49.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011716 2024-09-18T07:35:49.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,5", 2024-09-18T07:35:49.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.270491+0000", 2024-09-18T07:35:49.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000228075 2024-09-18T07:35:49.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.273935+0000", 2024-09-18T07:35:49.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034437059999999999 2024-09-18T07:35:49.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:49.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.274712+0000", 2024-09-18T07:35:49.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00077716600000000003 2024-09-18T07:35:49.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.275574+0000", 2024-09-18T07:35:49.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00086200100000000004 2024-09-18T07:35:49.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.404520+0000", 2024-09-18T07:35:49.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12894607399999999 2024-09-18T07:35:49.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:49.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.404712+0000", 2024-09-18T07:35:49.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019172699999999999 2024-09-18T07:35:49.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.404828+0000", 2024-09-18T07:35:49.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011657199999999999 2024-09-18T07:35:49.400 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.400 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.400 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.400 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14865.0:135142 7.d e138/81 7:bbff20ab:::benchmark_data_smithi137_74180_object30726:head v 138'8508, mlcod=138'8503)", 2024-09-18T07:35:49.400 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:18.281825+0000", 2024-09-18T07:35:49.400 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 90.998128304999994, 2024-09-18T07:35:49.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.107687581, 2024-09-18T07:35:49.400 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.401 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.401 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:49.401 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.281825+0000", 2024-09-18T07:35:49.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.281825+0000", 2024-09-18T07:35:49.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.281825+0000", 2024-09-18T07:35:49.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7899999999999997e-07 2024-09-18T07:35:49.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.281830+0000", 2024-09-18T07:35:49.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8810000000000004e-06 2024-09-18T07:35:49.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.281838+0000", 2024-09-18T07:35:49.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.267e-06 2024-09-18T07:35:49.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.281842+0000", 2024-09-18T07:35:49.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1869999999999999e-06 2024-09-18T07:35:49.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.281879+0000", 2024-09-18T07:35:49.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7580999999999998e-05 2024-09-18T07:35:49.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:49.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.281892+0000", 2024-09-18T07:35:49.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2352e-05 2024-09-18T07:35:49.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:49.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.389471+0000", 2024-09-18T07:35:49.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10757879400000001 2024-09-18T07:35:49.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.389512+0000", 2024-09-18T07:35:49.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1739999999999997e-05 2024-09-18T07:35:49.405 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.405 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.405 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.405 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:142960 7.12 7:4c4afa90:::benchmark_data_smithi137_74180_object38544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-18T07:35:49.405 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:25.433818+0000", 2024-09-18T07:35:49.405 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 83.846135340999993, 2024-09-18T07:35:49.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.185941245, 2024-09-18T07:35:49.406 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.406 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.406 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:35:49.406 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:35:49.406 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:35:49.406 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:35:49.406 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 142960 2024-09-18T07:35:49.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.406 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433818+0000", 2024-09-18T07:35:49.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433818+0000", 2024-09-18T07:35:49.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433820+0000", 2024-09-18T07:35:49.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3290000000000001e-06 2024-09-18T07:35:49.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433825+0000", 2024-09-18T07:35:49.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4589999999999997e-06 2024-09-18T07:35:49.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433837+0000", 2024-09-18T07:35:49.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1229999999999999e-05 2024-09-18T07:35:49.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433846+0000", 2024-09-18T07:35:49.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2790000000000004e-06 2024-09-18T07:35:49.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433902+0000", 2024-09-18T07:35:49.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5847999999999998e-05 2024-09-18T07:35:49.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:49.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.434031+0000", 2024-09-18T07:35:49.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012924099999999999 2024-09-18T07:35:49.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,3,6", 2024-09-18T07:35:49.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.434299+0000", 2024-09-18T07:35:49.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00026747400000000002 2024-09-18T07:35:49.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.436066+0000", 2024-09-18T07:35:49.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017675519999999999 2024-09-18T07:35:49.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:35:49.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.436687+0000", 2024-09-18T07:35:49.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00062042 2024-09-18T07:35:49.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.437458+0000", 2024-09-18T07:35:49.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00077107399999999996 2024-09-18T07:35:49.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.437498+0000", 2024-09-18T07:35:49.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0395999999999997e-05 2024-09-18T07:35:49.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:35:49.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.619551+0000", 2024-09-18T07:35:49.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.182052674 2024-09-18T07:35:49.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:35:49.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.619612+0000", 2024-09-18T07:35:49.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1678000000000001e-05 2024-09-18T07:35:49.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.619759+0000", 2024-09-18T07:35:49.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014659100000000001 2024-09-18T07:35:49.412 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.412 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.412 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.413 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "MOSDPGPush(7.11 140/137 [PushOp(7:8c792e11:::benchmark_data_smithi137_74180_object46017:head, version: 100'2970, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:8c792e11:::benchmark_data_smithi137_74180_object46017:head@100'2970, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-18T07:35:49.413 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:25.458022+0000", 2024-09-18T07:35:49.413 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 83.821931141999997, 2024-09-18T07:35:49.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14427894199999999, 2024-09-18T07:35:49.413 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.413 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.413 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:35:49.413 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.458022+0000", 2024-09-18T07:35:49.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.458022+0000", 2024-09-18T07:35:49.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.458024+0000", 2024-09-18T07:35:49.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.68e-06 2024-09-18T07:35:49.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.458090+0000", 2024-09-18T07:35:49.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6544999999999994e-05 2024-09-18T07:35:49.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.458116+0000", 2024-09-18T07:35:49.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6346000000000002e-05 2024-09-18T07:35:49.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.458141+0000", 2024-09-18T07:35:49.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4960000000000002e-05 2024-09-18T07:35:49.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.458283+0000", 2024-09-18T07:35:49.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000141428 2024-09-18T07:35:49.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:49.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.458303+0000", 2024-09-18T07:35:49.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0409000000000001e-05 2024-09-18T07:35:49.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.602301+0000", 2024-09-18T07:35:49.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14399757399999999 2024-09-18T07:35:49.417 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.417 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.417 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.417 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "pg_backfill(progress 7.11 e 140/140 lb 7:8c783410:::benchmark_data_smithi137_74180_object77421:head)", 2024-09-18T07:35:49.417 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:25.458180+0000", 2024-09-18T07:35:49.417 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 83.821772964999994, 2024-09-18T07:35:49.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.144217541, 2024-09-18T07:35:49.417 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:35:49.418 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:35:49.418 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:35:49.418 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:35:49.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:35:49.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.458180+0000", 2024-09-18T07:35:49.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:35:49.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.458180+0000", 2024-09-18T07:35:49.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:35:49.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:35:49.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.458181+0000", 2024-09-18T07:35:49.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1299999999999998e-07 2024-09-18T07:35:49.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:35:49.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.458191+0000", 2024-09-18T07:35:49.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.008e-05 2024-09-18T07:35:49.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:35:49.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.458201+0000", 2024-09-18T07:35:49.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0360000000000001e-05 2024-09-18T07:35:49.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:35:49.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.458210+0000", 2024-09-18T07:35:49.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4500000000000004e-06 2024-09-18T07:35:49.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.420 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:35:49.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.602350+0000", 2024-09-18T07:35:49.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14414002100000001 2024-09-18T07:35:49.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.420 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:35:49.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.602365+0000", 2024-09-18T07:35:49.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4807000000000001e-05 2024-09-18T07:35:49.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:35:49.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:35:49.421 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:35:49.421 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.602398+0000", 2024-09-18T07:35:49.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2910000000000002e-05 2024-09-18T07:35:49.421 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.421 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.421 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.421 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:35:49.421 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:35:49.421 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:35:49.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:49.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:49.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:49.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:49.546 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:49.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:49.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:49.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:49.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:49.670 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:49.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:49.771 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:49.798 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:49.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:49.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:49.892 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:35:49.899 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:49.927 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:49.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:49.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:50.029 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:50.038 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:35:50.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:50.052 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:50.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:50.072 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:49 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1991053879' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:50.072 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:49 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1991053879' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:50.154 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:50.176 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:50.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:49 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1991053879' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:35:50.276 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:50.302 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:50.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:50.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:50.404 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:50.430 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:50.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:50.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:50.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T07:35:50.531 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:50.557 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:50.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:50.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:50.659 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:50.683 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:50.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:50.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:50.749 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:50.785 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:50.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:50.809 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:50.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:50.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:50.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:50.932 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:50.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:51.034 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:51.060 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:51.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:51.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:51.130 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:50 smithi137 ceph-mon[29408]: pgmap v259: 148 pgs: 148 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 749 op/s; 648 KiB/s, 10 objects/s recovering 2024-09-18T07:35:51.130 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:50 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:51.130 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:50 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1991053879' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:35:51.130 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:50 smithi137 ceph-mon[29408]: osdmap e198: 8 total, 8 up, 8 in 2024-09-18T07:35:51.131 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:50 smithi137 ceph-mon[39175]: pgmap v259: 148 pgs: 148 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 749 op/s; 648 KiB/s, 10 objects/s recovering 2024-09-18T07:35:51.131 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:50 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:51.131 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:50 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1991053879' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:35:51.131 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:50 smithi137 ceph-mon[39175]: osdmap e198: 8 total, 8 up, 8 in 2024-09-18T07:35:51.162 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:51.187 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:51.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:51.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:51.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:50 smithi203 ceph-mon[32237]: pgmap v259: 148 pgs: 148 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 749 op/s; 648 KiB/s, 10 objects/s recovering 2024-09-18T07:35:51.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:50 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:50 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1991053879' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:35:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:50 smithi203 ceph-mon[32237]: osdmap e198: 8 total, 8 up, 8 in 2024-09-18T07:35:51.213 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:51.289 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:51.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:51.311 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:51.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:51.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:51.438 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:51.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:51.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:51.493 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:51.540 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:51.561 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:51.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:51.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:51.662 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:51.684 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:51.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:51.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:51.716 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:51.786 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:51.808 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:51.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:51.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:51.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:51.930 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:51.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:51.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:52.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:52.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:52.055 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:52.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:52.114 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:51 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:35:52.114 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:51 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:35:52.114 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:51 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:35:52.114 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:51 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:35:52.114 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:51 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:35:52.115 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:51 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:35:52.156 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:52.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:52.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:52.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:51 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:35:52.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:51 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:35:52.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:51 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:35:52.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:52.300 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:52.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:52.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:52.402 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:52.426 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:52.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:52.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:52.483 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:35:52.484 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:35:52.497 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:35:52.497 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:35:52.513 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:35:52.513 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:35:52.528 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:52.529 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:35:52.529 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:35:52.546 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:35:52.546 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:35:52.551 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:52.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:52.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:52.563 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:35:52.563 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:35:52.579 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:35:52.580 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:35:52.596 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:35:52.596 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:35:52.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:52.675 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:52.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:52.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:52.777 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:52.800 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:52.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:52.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:52.902 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:52.911 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'unique_pool_0' removed 2024-09-18T07:35:52.928 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:52.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:52.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:53.022 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:52 smithi137 ceph-mon[39175]: pgmap v261: 148 pgs: 148 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 326 op/s 2024-09-18T07:35:53.022 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:52 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:35:53.022 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:52 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3868071367' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:35:53.023 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:52 smithi137 ceph-mon[39175]: from='client.35114 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:35:53.023 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:52 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2646959675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:53.023 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:52 smithi137 ceph-mon[29408]: pgmap v261: 148 pgs: 148 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 326 op/s 2024-09-18T07:35:53.023 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:52 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:35:53.023 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:52 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3868071367' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:35:53.023 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:52 smithi137 ceph-mon[29408]: from='client.35114 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:35:53.023 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:52 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2646959675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:53.033 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:53.055 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:53.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:53.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:53.122 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:35:53.123 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:35:53.157 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:53.157 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:35:53.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:53.202 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:53.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:53.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:52 smithi203 ceph-mon[32237]: pgmap v261: 148 pgs: 148 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 326 op/s 2024-09-18T07:35:53.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:52 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:35:53.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:52 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3868071367' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:35:53.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:52 smithi203 ceph-mon[32237]: from='client.35114 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:35:53.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:52 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2646959675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:35:53.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:53.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:53.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:53.427 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:53.449 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:53.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:53.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:53.552 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:53.575 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:53.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:53.677 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:53.699 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:53.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:53.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:53.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:53.811 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-18T07:35:53.812 INFO:tasks.radosbench:Beginning radosbench... 2024-09-18T07:35:53.812 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-18T07:35:53.813 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-09-18T07:35:53.823 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:53.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:53.924 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:53.946 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:53.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:53.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:54.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:54.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:54.071 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:54.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:54.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:53 smithi137 ceph-mon[29408]: from='client.35114 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T07:35:54.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:53 smithi137 ceph-mon[29408]: osdmap e199: 8 total, 8 up, 8 in 2024-09-18T07:35:54.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:53 smithi137 ceph-mon[39175]: from='client.35114 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T07:35:54.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:53 smithi137 ceph-mon[39175]: osdmap e199: 8 total, 8 up, 8 in 2024-09-18T07:35:54.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:54.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:54.194 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:54.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:54.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:53 smithi203 ceph-mon[32237]: from='client.35114 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T07:35:54.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:53 smithi203 ceph-mon[32237]: osdmap e199: 8 total, 8 up, 8 in 2024-09-18T07:35:54.295 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:54.317 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:54.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:54.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:54.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:54.440 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:54.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:54.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:54.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:54.564 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:54.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:54.641 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:35:54.643 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:35:54.665 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:54.686 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:54.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:54.788 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:54.810 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:54.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:54.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:54.912 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:54.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:54.935 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:54.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:55.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:55.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:55.058 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:55.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:55.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:55.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:54 smithi137 ceph-mon[29408]: pgmap v263: 132 pgs: 132 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:35:55.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:54 smithi137 ceph-mon[39175]: pgmap v263: 132 pgs: 132 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:35:55.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:55.185 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:55.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:55.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:54 smithi203 ceph-mon[32237]: pgmap v263: 132 pgs: 132 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:35:55.287 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:55.289 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T07:35:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:55.311 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:55.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:55.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:55.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:55.435 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:55.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:55.536 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:55.559 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:55.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:55.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:55.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:55.683 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:55.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:55.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:55.808 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:55.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:55.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:55.816 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-18T07:35:53.499760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":880,"num_read_kb":739,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":1774839,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":563,"ondisk_log_size":563,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":447,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1325276,"kb_used_data":206716,"kb_used_omap":12,"kb_used_meta":1118387,"kb_avail":748635940,"statfs":{"total":767960285184,"available":766603202560,"internally_reserved":0,"allocated":211677184,"data_stored":206047817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1145228848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-170393624,"num_objects":-2601,"num_object_clones":0,"num_object_copies":-7803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2601,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":-113670,"num_write_kb":-2499585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-15716,"num_bytes_recovered":-1029636096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002165"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":302,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917499+0000","last_change":"2024-09-18T07:35:28.534496+0000","last_active":"2024-09-18T07:35:49.917499+0000","last_peered":"2024-09-18T07:35:49.917499+0000","last_clean":"2024-09-18T07:35:49.917499+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:35:49.917499+0000","last_undegraded":"2024-09-18T07:35:49.917499+0000","last_fullsized":"2024-09-18T07:35:49.917499+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:28.534391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:28.534391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:55.524474+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":333,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892196+0000","last_change":"2024-09-18T07:35:27.883344+0000","last_active":"2024-09-18T07:35:49.892196+0000","last_peered":"2024-09-18T07:35:49.892196+0000","last_clean":"2024-09-18T07:35:49.892196+0000","last_became_active":"2024-09-18T07:32:26.785727+0000","last_became_peered":"2024-09-18T07:32:26.785727+0000","last_unstale":"2024-09-18T07:35:49.892196+0000","last_undegraded":"2024-09-18T07:35:49.892196+0000","last_fullsized":"2024-09-18T07:35:49.892196+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:35:27.883281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:27.883281+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:40.767022+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":309,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892466+0000","last_change":"2024-09-18T07:35:01.397530+0000","last_active":"2024-09-18T07:35:49.892466+0000","last_peered":"2024-09-18T07:35:49.892466+0000","last_clean":"2024-09-18T07:35:49.892466+0000","last_became_active":"2024-09-18T07:35:01.397199+0000","last_became_peered":"2024-09-18T07:35:01.397199+0000","last_unstale":"2024-09-18T07:35:49.892466+0000","last_undegraded":"2024-09-18T07:35:49.892466+0000","last_fullsized":"2024-09-18T07:35:49.892466+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:57.894611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.894611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:09.974636+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":283,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892266+0000","last_change":"2024-09-18T07:32:03.953881+0000","last_active":"2024-09-18T07:35:49.892266+0000","last_peered":"2024-09-18T07:35:49.892266+0000","last_clean":"2024-09-18T07:35:49.892266+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:35:49.892266+0000","last_undegraded":"2024-09-18T07:35:49.892266+0000","last_fullsized":"2024-09-18T07:35:49.892266+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.953827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.953827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:16.008291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":182,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916801+0000","last_change":"2024-09-18T07:35:01.481557+0000","last_active":"2024-09-18T07:35:49.916801+0000","last_peered":"2024-09-18T07:35:49.916801+0000","last_clean":"2024-09-18T07:35:49.916801+0000","last_became_active":"2024-09-18T07:35:01.414396+0000","last_became_peered":"2024-09-18T07:35:01.414396+0000","last_unstale":"2024-09-18T07:35:49.916801+0000","last_undegraded":"2024-09-18T07:35:49.916801+0000","last_fullsized":"2024-09-18T07:35:49.916801+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:01.481514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:01.481514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:13.169861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":295,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.893694+0000","last_change":"2024-09-18T07:33:27.006545+0000","last_active":"2024-09-18T07:35:49.893694+0000","last_peered":"2024-09-18T07:35:49.893694+0000","last_clean":"2024-09-18T07:35:49.893694+0000","last_became_active":"2024-09-18T07:30:56.286040+0000","last_became_peered":"2024-09-18T07:30:56.286040+0000","last_unstale":"2024-09-18T07:35:49.893694+0000","last_undegraded":"2024-09-18T07:35:49.893694+0000","last_fullsized":"2024-09-18T07:35:49.893694+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.006488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.006488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":316,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892600+0000","last_change":"2024-09-18T07:33:22.032321+0000","last_active":"2024-09-18T07:35:49.892600+0000","last_peered":"2024-09-18T07:35:49.892600+0000","last_clean":"2024-09-18T07:35:49.892600+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:35:49.892600+0000","last_undegraded":"2024-09-18T07:35:49.892600+0000","last_fullsized":"2024-09-18T07:35:49.892600+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:22.032110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.032110+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:46.902576+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":296,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892112+0000","last_change":"2024-09-18T07:35:30.995338+0000","last_active":"2024-09-18T07:35:49.892112+0000","last_peered":"2024-09-18T07:35:49.892112+0000","last_clean":"2024-09-18T07:35:49.892112+0000","last_became_active":"2024-09-18T07:31:00.331224+0000","last_became_peered":"2024-09-18T07:31:00.331224+0000","last_unstale":"2024-09-18T07:35:49.892112+0000","last_undegraded":"2024-09-18T07:35:49.892112+0000","last_fullsized":"2024-09-18T07:35:49.892112+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:30.995282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:30.995282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:49.480100+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":301,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918145+0000","last_change":"2024-09-18T07:35:00.392530+0000","last_active":"2024-09-18T07:35:49.918145+0000","last_peered":"2024-09-18T07:35:49.918145+0000","last_clean":"2024-09-18T07:35:49.918145+0000","last_became_active":"2024-09-18T07:30:56.285882+0000","last_became_peered":"2024-09-18T07:30:56.285882+0000","last_unstale":"2024-09-18T07:35:49.918145+0000","last_undegraded":"2024-09-18T07:35:49.918145+0000","last_fullsized":"2024-09-18T07:35:49.918145+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:00.392466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.392466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:06.365891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":307,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892457+0000","last_change":"2024-09-18T07:32:02.002919+0000","last_active":"2024-09-18T07:35:49.892457+0000","last_peered":"2024-09-18T07:35:49.892457+0000","last_clean":"2024-09-18T07:35:49.892457+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:35:49.892457+0000","last_undegraded":"2024-09-18T07:35:49.892457+0000","last_fullsized":"2024-09-18T07:35:49.892457+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.002811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.002811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:14.980279+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":299,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917717+0000","last_change":"2024-09-18T07:35:04.267623+0000","last_active":"2024-09-18T07:35:49.917717+0000","last_peered":"2024-09-18T07:35:49.917717+0000","last_clean":"2024-09-18T07:35:49.917717+0000","last_became_active":"2024-09-18T07:35:01.413443+0000","last_became_peered":"2024-09-18T07:35:01.413443+0000","last_unstale":"2024-09-18T07:35:49.917717+0000","last_undegraded":"2024-09-18T07:35:49.917717+0000","last_fullsized":"2024-09-18T07:35:49.917717+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:04.267580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:04.267580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:19.052927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":287,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916856+0000","last_change":"2024-09-18T07:35:01.410563+0000","last_active":"2024-09-18T07:35:49.916856+0000","last_peered":"2024-09-18T07:35:49.916856+0000","last_clean":"2024-09-18T07:35:49.916856+0000","last_became_active":"2024-09-18T07:35:01.410284+0000","last_became_peered":"2024-09-18T07:35:01.410284+0000","last_unstale":"2024-09-18T07:35:49.916856+0000","last_undegraded":"2024-09-18T07:35:49.916856+0000","last_fullsized":"2024-09-18T07:35:49.916856+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:10.525949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":285,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917853+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:35:49.917853+0000","last_peered":"2024-09-18T07:35:49.917853+0000","last_clean":"2024-09-18T07:35:49.917853+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:35:49.917853+0000","last_undegraded":"2024-09-18T07:35:49.917853+0000","last_fullsized":"2024-09-18T07:35:49.917853+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:18.203226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":293,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892488+0000","last_change":"2024-09-18T07:30:58.317041+0000","last_active":"2024-09-18T07:35:49.892488+0000","last_peered":"2024-09-18T07:35:49.892488+0000","last_clean":"2024-09-18T07:35:49.892488+0000","last_became_active":"2024-09-18T07:30:58.316479+0000","last_became_peered":"2024-09-18T07:30:58.316479+0000","last_unstale":"2024-09-18T07:35:49.892488+0000","last_undegraded":"2024-09-18T07:35:49.892488+0000","last_fullsized":"2024-09-18T07:35:49.892488+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:51.873972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":62,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916962+0000","last_change":"2024-09-18T07:35:01.399995+0000","last_active":"2024-09-18T07:35:49.916962+0000","last_peered":"2024-09-18T07:35:49.916962+0000","last_clean":"2024-09-18T07:35:49.916962+0000","last_became_active":"2024-09-18T07:35:01.398605+0000","last_became_peered":"2024-09-18T07:35:01.398605+0000","last_unstale":"2024-09-18T07:35:49.916962+0000","last_undegraded":"2024-09-18T07:35:49.916962+0000","last_fullsized":"2024-09-18T07:35:49.916962+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:55.843864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:55.843864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:15.129566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":297,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916865+0000","last_change":"2024-09-18T07:35:35.248892+0000","last_active":"2024-09-18T07:35:49.916865+0000","last_peered":"2024-09-18T07:35:49.916865+0000","last_clean":"2024-09-18T07:35:49.916865+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:35:49.916865+0000","last_undegraded":"2024-09-18T07:35:49.916865+0000","last_fullsized":"2024-09-18T07:35:49.916865+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:35.248824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:35.248824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:41.491237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":292,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892868+0000","last_change":"2024-09-18T07:35:24.929178+0000","last_active":"2024-09-18T07:35:49.892868+0000","last_peered":"2024-09-18T07:35:49.892868+0000","last_clean":"2024-09-18T07:35:49.892868+0000","last_became_active":"2024-09-18T07:31:02.490014+0000","last_became_peered":"2024-09-18T07:31:02.490014+0000","last_unstale":"2024-09-18T07:35:49.892868+0000","last_undegraded":"2024-09-18T07:35:49.892868+0000","last_fullsized":"2024-09-18T07:35:49.892868+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:24.929124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:24.929124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:40.557387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":307,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917355+0000","last_change":"2024-09-18T07:35:01.415292+0000","last_active":"2024-09-18T07:35:49.917355+0000","last_peered":"2024-09-18T07:35:49.917355+0000","last_clean":"2024-09-18T07:35:49.917355+0000","last_became_active":"2024-09-18T07:35:01.414088+0000","last_became_peered":"2024-09-18T07:35:01.414088+0000","last_unstale":"2024-09-18T07:35:49.917355+0000","last_undegraded":"2024-09-18T07:35:49.917355+0000","last_fullsized":"2024-09-18T07:35:49.917355+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:53.330304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:53.330304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":332,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916413+0000","last_change":"2024-09-18T07:35:31.858126+0000","last_active":"2024-09-18T07:35:49.916413+0000","last_peered":"2024-09-18T07:35:49.916413+0000","last_clean":"2024-09-18T07:35:49.916413+0000","last_became_active":"2024-09-18T07:30:58.316266+0000","last_became_peered":"2024-09-18T07:30:58.316266+0000","last_unstale":"2024-09-18T07:35:49.916413+0000","last_undegraded":"2024-09-18T07:35:49.916413+0000","last_fullsized":"2024-09-18T07:35:49.916413+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:35:31.858040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:31.858040+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:55.124419+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":287,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.893455+0000","last_change":"2024-09-18T07:33:30.955441+0000","last_active":"2024-09-18T07:35:49.893455+0000","last_peered":"2024-09-18T07:35:49.893455+0000","last_clean":"2024-09-18T07:35:49.893455+0000","last_became_active":"2024-09-18T07:31:00.337474+0000","last_became_peered":"2024-09-18T07:31:00.337474+0000","last_unstale":"2024-09-18T07:35:49.893455+0000","last_undegraded":"2024-09-18T07:35:49.893455+0000","last_fullsized":"2024-09-18T07:35:49.893455+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:30.955213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:30.955213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:07.055140+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":291,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.894354+0000","last_change":"2024-09-18T07:35:34.992612+0000","last_active":"2024-09-18T07:35:49.894354+0000","last_peered":"2024-09-18T07:35:49.894354+0000","last_clean":"2024-09-18T07:35:49.894354+0000","last_became_active":"2024-09-18T07:31:02.492348+0000","last_became_peered":"2024-09-18T07:31:02.492348+0000","last_unstale":"2024-09-18T07:35:49.894354+0000","last_undegraded":"2024-09-18T07:35:49.894354+0000","last_fullsized":"2024-09-18T07:35:49.894354+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:34.992350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:34.992350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:58.140266+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":303,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892597+0000","last_change":"2024-09-18T07:35:37.941957+0000","last_active":"2024-09-18T07:35:49.892597+0000","last_peered":"2024-09-18T07:35:49.892597+0000","last_clean":"2024-09-18T07:35:49.892597+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:35:49.892597+0000","last_undegraded":"2024-09-18T07:35:49.892597+0000","last_fullsized":"2024-09-18T07:35:49.892597+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:37.941908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:37.941908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:49.652205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":305,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892230+0000","last_change":"2024-09-18T07:30:58.319115+0000","last_active":"2024-09-18T07:35:49.892230+0000","last_peered":"2024-09-18T07:35:49.892230+0000","last_clean":"2024-09-18T07:35:49.892230+0000","last_became_active":"2024-09-18T07:30:58.316380+0000","last_became_peered":"2024-09-18T07:30:58.316380+0000","last_unstale":"2024-09-18T07:35:49.892230+0000","last_undegraded":"2024-09-18T07:35:49.892230+0000","last_fullsized":"2024-09-18T07:35:49.892230+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:45.875022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":279,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892646+0000","last_change":"2024-09-18T07:31:00.334313+0000","last_active":"2024-09-18T07:35:49.892646+0000","last_peered":"2024-09-18T07:35:49.892646+0000","last_clean":"2024-09-18T07:35:49.892646+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:35:49.892646+0000","last_undegraded":"2024-09-18T07:35:49.892646+0000","last_fullsized":"2024-09-18T07:35:49.892646+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":74,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917987+0000","last_change":"2024-09-18T07:35:30.254797+0000","last_active":"2024-09-18T07:35:49.917987+0000","last_peered":"2024-09-18T07:35:49.917987+0000","last_clean":"2024-09-18T07:35:49.917987+0000","last_became_active":"2024-09-18T07:35:01.413110+0000","last_became_peered":"2024-09-18T07:35:01.413110+0000","last_unstale":"2024-09-18T07:35:49.917987+0000","last_undegraded":"2024-09-18T07:35:49.917987+0000","last_fullsized":"2024-09-18T07:35:49.917987+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:35:30.254725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:30.254725+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:43.829889+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":305,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892632+0000","last_change":"2024-09-18T07:33:32.976203+0000","last_active":"2024-09-18T07:35:49.892632+0000","last_peered":"2024-09-18T07:35:49.892632+0000","last_clean":"2024-09-18T07:35:49.892632+0000","last_became_active":"2024-09-18T07:30:56.283705+0000","last_became_peered":"2024-09-18T07:30:56.283705+0000","last_unstale":"2024-09-18T07:35:49.892632+0000","last_undegraded":"2024-09-18T07:35:49.892632+0000","last_fullsized":"2024-09-18T07:35:49.892632+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:32.976129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.976129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:09.022890+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":343,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917374+0000","last_change":"2024-09-18T07:35:01.438392+0000","last_active":"2024-09-18T07:35:49.917374+0000","last_peered":"2024-09-18T07:35:49.917374+0000","last_clean":"2024-09-18T07:35:49.917374+0000","last_became_active":"2024-09-18T07:35:01.410926+0000","last_became_peered":"2024-09-18T07:35:01.410926+0000","last_unstale":"2024-09-18T07:35:49.917374+0000","last_undegraded":"2024-09-18T07:35:49.917374+0000","last_fullsized":"2024-09-18T07:35:49.917374+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:33:36.118340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:36.118340+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"173'10","reported_seq":387,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918934+0000","last_change":"2024-09-18T07:35:29.400204+0000","last_active":"2024-09-18T07:35:49.918934+0000","last_peered":"2024-09-18T07:35:49.918934+0000","last_clean":"2024-09-18T07:35:49.918934+0000","last_became_active":"2024-09-18T07:35:01.407926+0000","last_became_peered":"2024-09-18T07:35:01.407926+0000","last_unstale":"2024-09-18T07:35:49.918934+0000","last_undegraded":"2024-09-18T07:35:49.918934+0000","last_fullsized":"2024-09-18T07:35:49.918934+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:35:29.400137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:29.400137+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:53.614676+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":317,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892190+0000","last_change":"2024-09-18T07:34:37.067259+0000","last_active":"2024-09-18T07:35:49.892190+0000","last_peered":"2024-09-18T07:35:49.892190+0000","last_clean":"2024-09-18T07:35:49.892190+0000","last_became_active":"2024-09-18T07:30:58.312627+0000","last_became_peered":"2024-09-18T07:30:58.312627+0000","last_unstale":"2024-09-18T07:35:49.892190+0000","last_undegraded":"2024-09-18T07:35:49.892190+0000","last_fullsized":"2024-09-18T07:35:49.892190+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:34:37.067162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:37.067162+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:52.474545+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":317,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892848+0000","last_change":"2024-09-18T07:35:01.407698+0000","last_active":"2024-09-18T07:35:49.892848+0000","last_peered":"2024-09-18T07:35:49.892848+0000","last_clean":"2024-09-18T07:35:49.892848+0000","last_became_active":"2024-09-18T07:35:01.407339+0000","last_became_peered":"2024-09-18T07:35:01.407339+0000","last_unstale":"2024-09-18T07:35:49.892848+0000","last_undegraded":"2024-09-18T07:35:49.892848+0000","last_fullsized":"2024-09-18T07:35:49.892848+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:01.294615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:01.294615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":364,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.893072+0000","last_change":"2024-09-18T07:35:29.961861+0000","last_active":"2024-09-18T07:35:49.893072+0000","last_peered":"2024-09-18T07:35:49.893072+0000","last_clean":"2024-09-18T07:35:49.893072+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:35:49.893072+0000","last_undegraded":"2024-09-18T07:35:49.893072+0000","last_fullsized":"2024-09-18T07:35:49.893072+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:35:29.961791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:29.961791+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:30.708743+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":282,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917678+0000","last_change":"2024-09-18T07:34:40.386158+0000","last_active":"2024-09-18T07:35:49.917678+0000","last_peered":"2024-09-18T07:35:49.917678+0000","last_clean":"2024-09-18T07:35:49.917678+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:35:49.917678+0000","last_undegraded":"2024-09-18T07:35:49.917678+0000","last_fullsized":"2024-09-18T07:35:49.917678+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:40.386112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:34:40.386112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:03.303185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":381,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892601+0000","last_change":"2024-09-18T07:35:23.983838+0000","last_active":"2024-09-18T07:35:49.892601+0000","last_peered":"2024-09-18T07:35:49.892601+0000","last_clean":"2024-09-18T07:35:49.892601+0000","last_became_active":"2024-09-18T07:35:01.418909+0000","last_became_peered":"2024-09-18T07:35:01.418909+0000","last_unstale":"2024-09-18T07:35:49.892601+0000","last_undegraded":"2024-09-18T07:35:49.892601+0000","last_fullsized":"2024-09-18T07:35:49.892601+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:35:23.983762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:23.983762+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:40.277352+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":317,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.894525+0000","last_change":"2024-09-18T07:35:27.922013+0000","last_active":"2024-09-18T07:35:49.894525+0000","last_peered":"2024-09-18T07:35:49.894525+0000","last_clean":"2024-09-18T07:35:49.894525+0000","last_became_active":"2024-09-18T07:35:01.407422+0000","last_became_peered":"2024-09-18T07:35:01.407422+0000","last_unstale":"2024-09-18T07:35:49.894525+0000","last_undegraded":"2024-09-18T07:35:49.894525+0000","last_fullsized":"2024-09-18T07:35:49.894525+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:27.921887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:27.921887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:45.772375+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":302,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892222+0000","last_change":"2024-09-18T07:35:01.401507+0000","last_active":"2024-09-18T07:35:49.892222+0000","last_peered":"2024-09-18T07:35:49.892222+0000","last_clean":"2024-09-18T07:35:49.892222+0000","last_became_active":"2024-09-18T07:35:01.401024+0000","last_became_peered":"2024-09-18T07:35:01.401024+0000","last_unstale":"2024-09-18T07:35:49.892222+0000","last_undegraded":"2024-09-18T07:35:49.892222+0000","last_fullsized":"2024-09-18T07:35:49.892222+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:54.808785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:54.808785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:06.641181+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":276,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892382+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:35:49.892382+0000","last_peered":"2024-09-18T07:35:49.892382+0000","last_clean":"2024-09-18T07:35:49.892382+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:35:49.892382+0000","last_undegraded":"2024-09-18T07:35:49.892382+0000","last_fullsized":"2024-09-18T07:35:49.892382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:13.029688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":313,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892222+0000","last_change":"2024-09-18T07:35:00.882253+0000","last_active":"2024-09-18T07:35:49.892222+0000","last_peered":"2024-09-18T07:35:49.892222+0000","last_clean":"2024-09-18T07:35:49.892222+0000","last_became_active":"2024-09-18T07:30:58.309649+0000","last_became_peered":"2024-09-18T07:30:58.309649+0000","last_unstale":"2024-09-18T07:35:49.892222+0000","last_undegraded":"2024-09-18T07:35:49.892222+0000","last_fullsized":"2024-09-18T07:35:49.892222+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-18T07:35:00.882191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.882191+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:04.068834+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":295,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917573+0000","last_change":"2024-09-18T07:33:24.213432+0000","last_active":"2024-09-18T07:35:49.917573+0000","last_peered":"2024-09-18T07:35:49.917573+0000","last_clean":"2024-09-18T07:35:49.917573+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:35:49.917573+0000","last_undegraded":"2024-09-18T07:35:49.917573+0000","last_fullsized":"2024-09-18T07:35:49.917573+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:24.213356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:24.213356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:46.472223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":355,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892428+0000","last_change":"2024-09-18T07:34:39.253606+0000","last_active":"2024-09-18T07:35:49.892428+0000","last_peered":"2024-09-18T07:35:49.892428+0000","last_clean":"2024-09-18T07:35:49.892428+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:35:49.892428+0000","last_undegraded":"2024-09-18T07:35:49.892428+0000","last_fullsized":"2024-09-18T07:35:49.892428+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:34:39.253558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:39.253558+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":70,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917199+0000","last_change":"2024-09-18T07:35:25.287397+0000","last_active":"2024-09-18T07:35:49.917199+0000","last_peered":"2024-09-18T07:35:49.917199+0000","last_clean":"2024-09-18T07:35:49.917199+0000","last_became_active":"2024-09-18T07:35:01.393846+0000","last_became_peered":"2024-09-18T07:35:01.393846+0000","last_unstale":"2024-09-18T07:35:49.917199+0000","last_undegraded":"2024-09-18T07:35:49.917199+0000","last_fullsized":"2024-09-18T07:35:49.917199+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:25.287352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:25.287352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:44.735842+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":299,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892831+0000","last_change":"2024-09-18T07:30:58.307470+0000","last_active":"2024-09-18T07:35:49.892831+0000","last_peered":"2024-09-18T07:35:49.892831+0000","last_clean":"2024-09-18T07:35:49.892831+0000","last_became_active":"2024-09-18T07:30:58.307035+0000","last_became_peered":"2024-09-18T07:30:58.307035+0000","last_unstale":"2024-09-18T07:35:49.892831+0000","last_undegraded":"2024-09-18T07:35:49.892831+0000","last_fullsized":"2024-09-18T07:35:49.892831+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:17.971586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":302,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918228+0000","last_change":"2024-09-18T07:35:24.210457+0000","last_active":"2024-09-18T07:35:49.918228+0000","last_peered":"2024-09-18T07:35:49.918228+0000","last_clean":"2024-09-18T07:35:49.918228+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:35:49.918228+0000","last_undegraded":"2024-09-18T07:35:49.918228+0000","last_fullsized":"2024-09-18T07:35:49.918228+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:24.210401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:24.210401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:42.826106+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":354,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917035+0000","last_change":"2024-09-18T07:33:52.317149+0000","last_active":"2024-09-18T07:35:49.917035+0000","last_peered":"2024-09-18T07:35:49.917035+0000","last_clean":"2024-09-18T07:35:49.917035+0000","last_became_active":"2024-09-18T07:31:00.336905+0000","last_became_peered":"2024-09-18T07:31:00.336905+0000","last_unstale":"2024-09-18T07:35:49.917035+0000","last_undegraded":"2024-09-18T07:35:49.917035+0000","last_fullsized":"2024-09-18T07:35:49.917035+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:33:52.317075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:52.317075+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":274,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917006+0000","last_change":"2024-09-18T07:31:02.348538+0000","last_active":"2024-09-18T07:35:49.917006+0000","last_peered":"2024-09-18T07:35:49.917006+0000","last_clean":"2024-09-18T07:35:49.917006+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:35:49.917006+0000","last_undegraded":"2024-09-18T07:35:49.917006+0000","last_fullsized":"2024-09-18T07:35:49.917006+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:44.477539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":376,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917124+0000","last_change":"2024-09-18T07:35:01.449045+0000","last_active":"2024-09-18T07:35:49.917124+0000","last_peered":"2024-09-18T07:35:49.917124+0000","last_clean":"2024-09-18T07:35:49.917124+0000","last_became_active":"2024-09-18T07:35:01.415275+0000","last_became_peered":"2024-09-18T07:35:01.415275+0000","last_unstale":"2024-09-18T07:35:49.917124+0000","last_undegraded":"2024-09-18T07:35:49.917124+0000","last_fullsized":"2024-09-18T07:35:49.917124+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:34:39.610788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:39.610788+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":288,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.893416+0000","last_change":"2024-09-18T07:30:56.284440+0000","last_active":"2024-09-18T07:35:49.893416+0000","last_peered":"2024-09-18T07:35:49.893416+0000","last_clean":"2024-09-18T07:35:49.893416+0000","last_became_active":"2024-09-18T07:30:56.284120+0000","last_became_peered":"2024-09-18T07:30:56.284120+0000","last_unstale":"2024-09-18T07:35:49.893416+0000","last_undegraded":"2024-09-18T07:35:49.893416+0000","last_fullsized":"2024-09-18T07:35:49.893416+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:13.961499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":286,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916905+0000","last_change":"2024-09-18T07:33:21.834910+0000","last_active":"2024-09-18T07:35:49.916905+0000","last_peered":"2024-09-18T07:35:49.916905+0000","last_clean":"2024-09-18T07:35:49.916905+0000","last_became_active":"2024-09-18T07:31:00.334776+0000","last_became_peered":"2024-09-18T07:31:00.334776+0000","last_unstale":"2024-09-18T07:35:49.916905+0000","last_undegraded":"2024-09-18T07:35:49.916905+0000","last_fullsized":"2024-09-18T07:35:49.916905+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:48.284140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":276,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892384+0000","last_change":"2024-09-18T07:31:02.490534+0000","last_active":"2024-09-18T07:35:49.892384+0000","last_peered":"2024-09-18T07:35:49.892384+0000","last_clean":"2024-09-18T07:35:49.892384+0000","last_became_active":"2024-09-18T07:31:02.490186+0000","last_became_peered":"2024-09-18T07:31:02.490186+0000","last_unstale":"2024-09-18T07:35:49.892384+0000","last_undegraded":"2024-09-18T07:35:49.892384+0000","last_fullsized":"2024-09-18T07:35:49.892384+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:43.026118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":400,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.893292+0000","last_change":"2024-09-18T07:34:31.955399+0000","last_active":"2024-09-18T07:35:49.893292+0000","last_peered":"2024-09-18T07:35:49.893292+0000","last_clean":"2024-09-18T07:35:49.893292+0000","last_became_active":"2024-09-18T07:30:58.316812+0000","last_became_peered":"2024-09-18T07:30:58.316812+0000","last_unstale":"2024-09-18T07:35:49.893292+0000","last_undegraded":"2024-09-18T07:35:49.893292+0000","last_fullsized":"2024-09-18T07:35:49.893292+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'28","last_scrub_stamp":"2024-09-18T07:34:31.955336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.955336+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:51.082565+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":288,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.893172+0000","last_change":"2024-09-18T07:30:56.285353+0000","last_active":"2024-09-18T07:35:49.893172+0000","last_peered":"2024-09-18T07:35:49.893172+0000","last_clean":"2024-09-18T07:35:49.893172+0000","last_became_active":"2024-09-18T07:30:56.285032+0000","last_became_peered":"2024-09-18T07:30:56.285032+0000","last_unstale":"2024-09-18T07:35:49.893172+0000","last_undegraded":"2024-09-18T07:35:49.893172+0000","last_fullsized":"2024-09-18T07:35:49.893172+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:06.856268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":304,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917511+0000","last_change":"2024-09-18T07:34:39.420024+0000","last_active":"2024-09-18T07:35:49.917511+0000","last_peered":"2024-09-18T07:35:49.917511+0000","last_clean":"2024-09-18T07:35:49.917511+0000","last_became_active":"2024-09-18T07:30:55.283292+0000","last_became_peered":"2024-09-18T07:30:55.283292+0000","last_unstale":"2024-09-18T07:35:49.917511+0000","last_undegraded":"2024-09-18T07:35:49.917511+0000","last_fullsized":"2024-09-18T07:35:49.917511+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:34:39.419958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:34:39.419958+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:59.903039+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":278,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916890+0000","last_change":"2024-09-18T07:31:00.332738+0000","last_active":"2024-09-18T07:35:49.916890+0000","last_peered":"2024-09-18T07:35:49.916890+0000","last_clean":"2024-09-18T07:35:49.916890+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:35:49.916890+0000","last_undegraded":"2024-09-18T07:35:49.916890+0000","last_fullsized":"2024-09-18T07:35:49.916890+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:19.517970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":297,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.893446+0000","last_change":"2024-09-18T07:35:01.419122+0000","last_active":"2024-09-18T07:35:49.893446+0000","last_peered":"2024-09-18T07:35:49.893446+0000","last_clean":"2024-09-18T07:35:49.893446+0000","last_became_active":"2024-09-18T07:35:01.418809+0000","last_became_peered":"2024-09-18T07:35:01.418809+0000","last_unstale":"2024-09-18T07:35:49.893446+0000","last_undegraded":"2024-09-18T07:35:49.893446+0000","last_fullsized":"2024-09-18T07:35:49.893446+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:56.289176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:34:56.289176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:12.265135+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":319,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892277+0000","last_change":"2024-09-18T07:34:38.975221+0000","last_active":"2024-09-18T07:35:49.892277+0000","last_peered":"2024-09-18T07:35:49.892277+0000","last_clean":"2024-09-18T07:35:49.892277+0000","last_became_active":"2024-09-18T07:30:58.310490+0000","last_became_peered":"2024-09-18T07:30:58.310490+0000","last_unstale":"2024-09-18T07:35:49.892277+0000","last_undegraded":"2024-09-18T07:35:49.892277+0000","last_fullsized":"2024-09-18T07:35:49.892277+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-18T07:34:38.975156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:38.975156+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:05.736223+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":304,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892122+0000","last_change":"2024-09-18T07:35:03.095873+0000","last_active":"2024-09-18T07:35:49.892122+0000","last_peered":"2024-09-18T07:35:49.892122+0000","last_clean":"2024-09-18T07:35:49.892122+0000","last_became_active":"2024-09-18T07:30:56.284803+0000","last_became_peered":"2024-09-18T07:30:56.284803+0000","last_unstale":"2024-09-18T07:35:49.892122+0000","last_undegraded":"2024-09-18T07:35:49.892122+0000","last_fullsized":"2024-09-18T07:35:49.892122+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:03.095826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:03.095826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:03.655181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":305,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916222+0000","last_change":"2024-09-18T07:32:02.418590+0000","last_active":"2024-09-18T07:35:49.916222+0000","last_peered":"2024-09-18T07:35:49.916222+0000","last_clean":"2024-09-18T07:35:49.916222+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:35:49.916222+0000","last_undegraded":"2024-09-18T07:35:49.916222+0000","last_fullsized":"2024-09-18T07:35:49.916222+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:32:02.418507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.418507+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:46.713520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":294,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916896+0000","last_change":"2024-09-18T07:35:01.403942+0000","last_active":"2024-09-18T07:35:49.916896+0000","last_peered":"2024-09-18T07:35:49.916896+0000","last_clean":"2024-09-18T07:35:49.916896+0000","last_became_active":"2024-09-18T07:35:01.403680+0000","last_became_peered":"2024-09-18T07:35:01.403680+0000","last_unstale":"2024-09-18T07:35:49.916896+0000","last_undegraded":"2024-09-18T07:35:49.916896+0000","last_fullsized":"2024-09-18T07:35:49.916896+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:08.951679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":66,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918900+0000","last_change":"2024-09-18T07:35:01.441600+0000","last_active":"2024-09-18T07:35:49.918900+0000","last_peered":"2024-09-18T07:35:49.918900+0000","last_clean":"2024-09-18T07:35:49.918900+0000","last_became_active":"2024-09-18T07:35:01.406712+0000","last_became_peered":"2024-09-18T07:35:01.406712+0000","last_unstale":"2024-09-18T07:35:49.918900+0000","last_undegraded":"2024-09-18T07:35:49.918900+0000","last_fullsized":"2024-09-18T07:35:49.918900+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:57.655797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.655797+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":321,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917470+0000","last_change":"2024-09-18T07:33:27.040227+0000","last_active":"2024-09-18T07:35:49.917470+0000","last_peered":"2024-09-18T07:35:49.917470+0000","last_clean":"2024-09-18T07:35:49.917470+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:35:49.917470+0000","last_undegraded":"2024-09-18T07:35:49.917470+0000","last_fullsized":"2024-09-18T07:35:49.917470+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:15.523581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":292,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917078+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:35:49.917078+0000","last_peered":"2024-09-18T07:35:49.917078+0000","last_clean":"2024-09-18T07:35:49.917078+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:35:49.917078+0000","last_undegraded":"2024-09-18T07:35:49.917078+0000","last_fullsized":"2024-09-18T07:35:49.917078+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:11.500230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":116,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918411+0000","last_change":"2024-09-18T07:35:01.421692+0000","last_active":"2024-09-18T07:35:49.918411+0000","last_peered":"2024-09-18T07:35:49.918411+0000","last_clean":"2024-09-18T07:35:49.918411+0000","last_became_active":"2024-09-18T07:35:01.400524+0000","last_became_peered":"2024-09-18T07:35:01.400524+0000","last_unstale":"2024-09-18T07:35:49.918411+0000","last_undegraded":"2024-09-18T07:35:49.918411+0000","last_fullsized":"2024-09-18T07:35:49.918411+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:19.260383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":56,"num_read_kb":51,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":286,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918495+0000","last_change":"2024-09-18T07:33:27.394247+0000","last_active":"2024-09-18T07:35:49.918495+0000","last_peered":"2024-09-18T07:35:49.918495+0000","last_clean":"2024-09-18T07:35:49.918495+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:35:49.918495+0000","last_undegraded":"2024-09-18T07:35:49.918495+0000","last_fullsized":"2024-09-18T07:35:49.918495+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.394203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.394203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:14.210979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":300,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892977+0000","last_change":"2024-09-18T07:34:58.080008+0000","last_active":"2024-09-18T07:35:49.892977+0000","last_peered":"2024-09-18T07:35:49.892977+0000","last_clean":"2024-09-18T07:35:49.892977+0000","last_became_active":"2024-09-18T07:31:02.347390+0000","last_became_peered":"2024-09-18T07:31:02.347390+0000","last_unstale":"2024-09-18T07:35:49.892977+0000","last_undegraded":"2024-09-18T07:35:49.892977+0000","last_fullsized":"2024-09-18T07:35:49.892977+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:58.079941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:34:58.079941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:13.659972+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"173'18","reported_seq":430,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918302+0000","last_change":"2024-09-18T07:35:01.444206+0000","last_active":"2024-09-18T07:35:49.918302+0000","last_peered":"2024-09-18T07:35:49.918302+0000","last_clean":"2024-09-18T07:35:49.918302+0000","last_became_active":"2024-09-18T07:35:01.409012+0000","last_became_peered":"2024-09-18T07:35:01.409012+0000","last_unstale":"2024-09-18T07:35:49.918302+0000","last_undegraded":"2024-09-18T07:35:49.918302+0000","last_fullsized":"2024-09-18T07:35:49.918302+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'17","last_scrub_stamp":"2024-09-18T07:33:33.849285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.849285+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:14.193593+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":295,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.894695+0000","last_change":"2024-09-18T07:31:59.617046+0000","last_active":"2024-09-18T07:35:49.894695+0000","last_peered":"2024-09-18T07:35:49.894695+0000","last_clean":"2024-09-18T07:35:49.894695+0000","last_became_active":"2024-09-18T07:30:56.288149+0000","last_became_peered":"2024-09-18T07:30:56.288149+0000","last_unstale":"2024-09-18T07:35:49.894695+0000","last_undegraded":"2024-09-18T07:35:49.894695+0000","last_fullsized":"2024-09-18T07:35:49.894695+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:59.616988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:59.616988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:14.105579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":76,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918405+0000","last_change":"2024-09-18T07:35:27.278161+0000","last_active":"2024-09-18T07:35:49.918405+0000","last_peered":"2024-09-18T07:35:49.918405+0000","last_clean":"2024-09-18T07:35:49.918405+0000","last_became_active":"2024-09-18T07:35:01.400273+0000","last_became_peered":"2024-09-18T07:35:01.400273+0000","last_unstale":"2024-09-18T07:35:49.918405+0000","last_undegraded":"2024-09-18T07:35:49.918405+0000","last_fullsized":"2024-09-18T07:35:49.918405+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:35:27.278095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:35:27.278095+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:51.801152+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":62,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918093+0000","last_change":"2024-09-18T07:35:01.413545+0000","last_active":"2024-09-18T07:35:49.918093+0000","last_peered":"2024-09-18T07:35:49.918093+0000","last_clean":"2024-09-18T07:35:49.918093+0000","last_became_active":"2024-09-18T07:35:01.412217+0000","last_became_peered":"2024-09-18T07:35:01.412217+0000","last_unstale":"2024-09-18T07:35:49.918093+0000","last_undegraded":"2024-09-18T07:35:49.918093+0000","last_fullsized":"2024-09-18T07:35:49.918093+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:36.964421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:36.964421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":282,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918270+0000","last_change":"2024-09-18T07:33:22.358792+0000","last_active":"2024-09-18T07:35:49.918270+0000","last_peered":"2024-09-18T07:35:49.918270+0000","last_clean":"2024-09-18T07:35:49.918270+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:35:49.918270+0000","last_undegraded":"2024-09-18T07:35:49.918270+0000","last_fullsized":"2024-09-18T07:35:49.918270+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.358746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.358746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:11.158583+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":338,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892993+0000","last_change":"2024-09-18T07:35:38.928479+0000","last_active":"2024-09-18T07:35:49.892993+0000","last_peered":"2024-09-18T07:35:49.892993+0000","last_clean":"2024-09-18T07:35:49.892993+0000","last_became_active":"2024-09-18T07:32:26.785711+0000","last_became_peered":"2024-09-18T07:32:26.785711+0000","last_unstale":"2024-09-18T07:35:49.892993+0000","last_undegraded":"2024-09-18T07:35:49.892993+0000","last_fullsized":"2024-09-18T07:35:49.892993+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:35:38.928407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:38.928407+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:58.999748+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":310,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892506+0000","last_change":"2024-09-18T07:35:03.866215+0000","last_active":"2024-09-18T07:35:49.892506+0000","last_peered":"2024-09-18T07:35:49.892506+0000","last_clean":"2024-09-18T07:35:49.892506+0000","last_became_active":"2024-09-18T07:35:01.401063+0000","last_became_peered":"2024-09-18T07:35:01.401063+0000","last_unstale":"2024-09-18T07:35:49.892506+0000","last_undegraded":"2024-09-18T07:35:49.892506+0000","last_fullsized":"2024-09-18T07:35:49.892506+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:03.866148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:03.866148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:25.066815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":294,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918175+0000","last_change":"2024-09-18T07:35:31.191601+0000","last_active":"2024-09-18T07:35:49.918175+0000","last_peered":"2024-09-18T07:35:49.918175+0000","last_clean":"2024-09-18T07:35:49.918175+0000","last_became_active":"2024-09-18T07:31:00.336172+0000","last_became_peered":"2024-09-18T07:31:00.336172+0000","last_unstale":"2024-09-18T07:35:49.918175+0000","last_undegraded":"2024-09-18T07:35:49.918175+0000","last_fullsized":"2024-09-18T07:35:49.918175+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:31.191528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:31.191528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:38.261888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":291,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892668+0000","last_change":"2024-09-18T07:35:38.955492+0000","last_active":"2024-09-18T07:35:49.892668+0000","last_peered":"2024-09-18T07:35:49.892668+0000","last_clean":"2024-09-18T07:35:49.892668+0000","last_became_active":"2024-09-18T07:31:02.490953+0000","last_became_peered":"2024-09-18T07:31:02.490953+0000","last_unstale":"2024-09-18T07:35:49.892668+0000","last_undegraded":"2024-09-18T07:35:49.892668+0000","last_fullsized":"2024-09-18T07:35:49.892668+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:38.955353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:38.955353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:43.588886+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":328,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892936+0000","last_change":"2024-09-18T07:35:33.011021+0000","last_active":"2024-09-18T07:35:49.892936+0000","last_peered":"2024-09-18T07:35:49.892936+0000","last_clean":"2024-09-18T07:35:49.892936+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:35:49.892936+0000","last_undegraded":"2024-09-18T07:35:49.892936+0000","last_fullsized":"2024-09-18T07:35:49.892936+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:35:33.010955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:33.010955+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:40.982191+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":302,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917297+0000","last_change":"2024-09-18T07:35:35.265329+0000","last_active":"2024-09-18T07:35:49.917297+0000","last_peered":"2024-09-18T07:35:49.917297+0000","last_clean":"2024-09-18T07:35:49.917297+0000","last_became_active":"2024-09-18T07:30:56.284020+0000","last_became_peered":"2024-09-18T07:30:56.284020+0000","last_unstale":"2024-09-18T07:35:49.917297+0000","last_undegraded":"2024-09-18T07:35:49.917297+0000","last_fullsized":"2024-09-18T07:35:49.917297+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:35.265259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:35.265259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:51.107776+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":288,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.893062+0000","last_change":"2024-09-18T07:32:05.239202+0000","last_active":"2024-09-18T07:35:49.893062+0000","last_peered":"2024-09-18T07:35:49.893062+0000","last_clean":"2024-09-18T07:35:49.893062+0000","last_became_active":"2024-09-18T07:31:00.332836+0000","last_became_peered":"2024-09-18T07:31:00.332836+0000","last_unstale":"2024-09-18T07:35:49.893062+0000","last_undegraded":"2024-09-18T07:35:49.893062+0000","last_fullsized":"2024-09-18T07:35:49.893062+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:05.239160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.239160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:47.040221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":292,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916434+0000","last_change":"2024-09-18T07:35:25.866286+0000","last_active":"2024-09-18T07:35:49.916434+0000","last_peered":"2024-09-18T07:35:49.916434+0000","last_clean":"2024-09-18T07:35:49.916434+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:35:49.916434+0000","last_undegraded":"2024-09-18T07:35:49.916434+0000","last_fullsized":"2024-09-18T07:35:49.916434+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:25.866225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:25.866225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:35.156540+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":307,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892688+0000","last_change":"2024-09-18T07:32:05.000363+0000","last_active":"2024-09-18T07:35:49.892688+0000","last_peered":"2024-09-18T07:35:49.892688+0000","last_clean":"2024-09-18T07:35:49.892688+0000","last_became_active":"2024-09-18T07:30:58.316423+0000","last_became_peered":"2024-09-18T07:30:58.316423+0000","last_unstale":"2024-09-18T07:35:49.892688+0000","last_undegraded":"2024-09-18T07:35:49.892688+0000","last_fullsized":"2024-09-18T07:35:49.892688+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:32:05.000310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.000310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:18.050711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":295,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.894393+0000","last_change":"2024-09-18T07:32:06.688493+0000","last_active":"2024-09-18T07:35:49.894393+0000","last_peered":"2024-09-18T07:35:49.894393+0000","last_clean":"2024-09-18T07:35:49.894393+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:35:49.894393+0000","last_undegraded":"2024-09-18T07:35:49.894393+0000","last_fullsized":"2024-09-18T07:35:49.894393+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:06.688444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:06.688444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:16.075093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"173'11","reported_seq":389,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916626+0000","last_change":"2024-09-18T07:35:01.413702+0000","last_active":"2024-09-18T07:35:49.916626+0000","last_peered":"2024-09-18T07:35:49.916626+0000","last_clean":"2024-09-18T07:35:49.916626+0000","last_became_active":"2024-09-18T07:35:01.403765+0000","last_became_peered":"2024-09-18T07:35:01.403765+0000","last_unstale":"2024-09-18T07:35:49.916626+0000","last_undegraded":"2024-09-18T07:35:49.916626+0000","last_fullsized":"2024-09-18T07:35:49.916626+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'10","last_scrub_stamp":"2024-09-18T07:33:33.000874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.000874+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:16.909843+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":284,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892843+0000","last_change":"2024-09-18T07:35:04.907232+0000","last_active":"2024-09-18T07:35:49.892843+0000","last_peered":"2024-09-18T07:35:49.892843+0000","last_clean":"2024-09-18T07:35:49.892843+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:35:49.892843+0000","last_undegraded":"2024-09-18T07:35:49.892843+0000","last_fullsized":"2024-09-18T07:35:49.892843+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:04.907166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:04.907166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:25.077151+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":298,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917065+0000","last_change":"2024-09-18T07:30:58.314424+0000","last_active":"2024-09-18T07:35:49.917065+0000","last_peered":"2024-09-18T07:35:49.917065+0000","last_clean":"2024-09-18T07:35:49.917065+0000","last_became_active":"2024-09-18T07:30:58.314127+0000","last_became_peered":"2024-09-18T07:30:58.314127+0000","last_unstale":"2024-09-18T07:35:49.917065+0000","last_undegraded":"2024-09-18T07:35:49.917065+0000","last_fullsized":"2024-09-18T07:35:49.917065+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:49.469792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":302,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918006+0000","last_change":"2024-09-18T07:35:21.222584+0000","last_active":"2024-09-18T07:35:49.918006+0000","last_peered":"2024-09-18T07:35:49.918006+0000","last_clean":"2024-09-18T07:35:49.918006+0000","last_became_active":"2024-09-18T07:30:56.288868+0000","last_became_peered":"2024-09-18T07:30:56.288868+0000","last_unstale":"2024-09-18T07:35:49.918006+0000","last_undegraded":"2024-09-18T07:35:49.918006+0000","last_fullsized":"2024-09-18T07:35:49.918006+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:21.222526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.222526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:30.513979+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":288,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916577+0000","last_change":"2024-09-18T07:34:31.959675+0000","last_active":"2024-09-18T07:35:49.916577+0000","last_peered":"2024-09-18T07:35:49.916577+0000","last_clean":"2024-09-18T07:35:49.916577+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:35:49.916577+0000","last_undegraded":"2024-09-18T07:35:49.916577+0000","last_fullsized":"2024-09-18T07:35:49.916577+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:31.959636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.959636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:50.298137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":296,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916985+0000","last_change":"2024-09-18T07:35:01.411215+0000","last_active":"2024-09-18T07:35:49.916985+0000","last_peered":"2024-09-18T07:35:49.916985+0000","last_clean":"2024-09-18T07:35:49.916985+0000","last_became_active":"2024-09-18T07:35:01.410890+0000","last_became_peered":"2024-09-18T07:35:01.410890+0000","last_unstale":"2024-09-18T07:35:49.916985+0000","last_undegraded":"2024-09-18T07:35:49.916985+0000","last_fullsized":"2024-09-18T07:35:49.916985+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:02.301353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:34:02.301353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":391,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917022+0000","last_change":"2024-09-18T07:35:21.344099+0000","last_active":"2024-09-18T07:35:49.917022+0000","last_peered":"2024-09-18T07:35:49.917022+0000","last_clean":"2024-09-18T07:35:49.917022+0000","last_became_active":"2024-09-18T07:35:01.409401+0000","last_became_peered":"2024-09-18T07:35:01.409401+0000","last_unstale":"2024-09-18T07:35:49.917022+0000","last_undegraded":"2024-09-18T07:35:49.917022+0000","last_fullsized":"2024-09-18T07:35:49.917022+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:35:21.344029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.344029+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:44.145569+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":62,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.919178+0000","last_change":"2024-09-18T07:35:01.412993+0000","last_active":"2024-09-18T07:35:49.919178+0000","last_peered":"2024-09-18T07:35:49.919178+0000","last_clean":"2024-09-18T07:35:49.919178+0000","last_became_active":"2024-09-18T07:35:01.411821+0000","last_became_peered":"2024-09-18T07:35:01.411821+0000","last_unstale":"2024-09-18T07:35:49.919178+0000","last_undegraded":"2024-09-18T07:35:49.919178+0000","last_fullsized":"2024-09-18T07:35:49.919178+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.481153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.481153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:20.307427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":295,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916558+0000","last_change":"2024-09-18T07:33:34.966610+0000","last_active":"2024-09-18T07:35:49.916558+0000","last_peered":"2024-09-18T07:35:49.916558+0000","last_clean":"2024-09-18T07:35:49.916558+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:35:49.916558+0000","last_undegraded":"2024-09-18T07:35:49.916558+0000","last_fullsized":"2024-09-18T07:35:49.916558+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:34.966537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:34.966537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:13.904823+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":62,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918469+0000","last_change":"2024-09-18T07:35:01.408078+0000","last_active":"2024-09-18T07:35:49.918469+0000","last_peered":"2024-09-18T07:35:49.918469+0000","last_clean":"2024-09-18T07:35:49.918469+0000","last_became_active":"2024-09-18T07:35:01.406980+0000","last_became_peered":"2024-09-18T07:35:01.406980+0000","last_unstale":"2024-09-18T07:35:49.918469+0000","last_undegraded":"2024-09-18T07:35:49.918469+0000","last_fullsized":"2024-09-18T07:35:49.918469+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:15.227647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":323,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892357+0000","last_change":"2024-09-18T07:35:21.085719+0000","last_active":"2024-09-18T07:35:49.892357+0000","last_peered":"2024-09-18T07:35:49.892357+0000","last_clean":"2024-09-18T07:35:49.892357+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:35:49.892357+0000","last_undegraded":"2024-09-18T07:35:49.892357+0000","last_fullsized":"2024-09-18T07:35:49.892357+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:35:21.085670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.085670+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:47.881459+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":62,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.919381+0000","last_change":"2024-09-18T07:35:01.400050+0000","last_active":"2024-09-18T07:35:49.919381+0000","last_peered":"2024-09-18T07:35:49.919381+0000","last_clean":"2024-09-18T07:35:49.919381+0000","last_became_active":"2024-09-18T07:35:01.398499+0000","last_became_peered":"2024-09-18T07:35:01.398499+0000","last_unstale":"2024-09-18T07:35:49.919381+0000","last_undegraded":"2024-09-18T07:35:49.919381+0000","last_fullsized":"2024-09-18T07:35:49.919381+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:18.240505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":304,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916218+0000","last_change":"2024-09-18T07:35:29.856761+0000","last_active":"2024-09-18T07:35:49.916218+0000","last_peered":"2024-09-18T07:35:49.916218+0000","last_clean":"2024-09-18T07:35:49.916218+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:35:49.916218+0000","last_undegraded":"2024-09-18T07:35:49.916218+0000","last_fullsized":"2024-09-18T07:35:49.916218+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:29.856305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:29.856305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:30.681292+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":289,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.894236+0000","last_change":"2024-09-18T07:35:01.419420+0000","last_active":"2024-09-18T07:35:49.894236+0000","last_peered":"2024-09-18T07:35:49.894236+0000","last_clean":"2024-09-18T07:35:49.894236+0000","last_became_active":"2024-09-18T07:35:01.418735+0000","last_became_peered":"2024-09-18T07:35:01.418735+0000","last_unstale":"2024-09-18T07:35:49.894236+0000","last_undegraded":"2024-09-18T07:35:49.894236+0000","last_fullsized":"2024-09-18T07:35:49.894236+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:10.188683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":369,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916910+0000","last_change":"2024-09-18T07:35:36.271956+0000","last_active":"2024-09-18T07:35:49.916910+0000","last_peered":"2024-09-18T07:35:49.916910+0000","last_clean":"2024-09-18T07:35:49.916910+0000","last_became_active":"2024-09-18T07:35:01.416425+0000","last_became_peered":"2024-09-18T07:35:01.416425+0000","last_unstale":"2024-09-18T07:35:49.916910+0000","last_undegraded":"2024-09-18T07:35:49.916910+0000","last_fullsized":"2024-09-18T07:35:49.916910+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:35:36.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:36.271863+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:39.581155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":76,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.919238+0000","last_change":"2024-09-18T07:35:02.205821+0000","last_active":"2024-09-18T07:35:49.919238+0000","last_peered":"2024-09-18T07:35:49.919238+0000","last_clean":"2024-09-18T07:35:49.919238+0000","last_became_active":"2024-09-18T07:35:01.406695+0000","last_became_peered":"2024-09-18T07:35:01.406695+0000","last_unstale":"2024-09-18T07:35:49.919238+0000","last_undegraded":"2024-09-18T07:35:49.919238+0000","last_fullsized":"2024-09-18T07:35:49.919238+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:35:02.205775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.205775+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:09.118035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"173'10","reported_seq":85,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918147+0000","last_change":"2024-09-18T07:35:26.319384+0000","last_active":"2024-09-18T07:35:49.918147+0000","last_peered":"2024-09-18T07:35:49.918147+0000","last_clean":"2024-09-18T07:35:49.918147+0000","last_became_active":"2024-09-18T07:35:01.400330+0000","last_became_peered":"2024-09-18T07:35:01.400330+0000","last_unstale":"2024-09-18T07:35:49.918147+0000","last_undegraded":"2024-09-18T07:35:49.918147+0000","last_fullsized":"2024-09-18T07:35:49.918147+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:35:26.319320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:26.319320+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:42.536463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":274,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917119+0000","last_change":"2024-09-18T07:31:02.489914+0000","last_active":"2024-09-18T07:35:49.917119+0000","last_peered":"2024-09-18T07:35:49.917119+0000","last_clean":"2024-09-18T07:35:49.917119+0000","last_became_active":"2024-09-18T07:31:02.489619+0000","last_became_peered":"2024-09-18T07:31:02.489619+0000","last_unstale":"2024-09-18T07:35:49.917119+0000","last_undegraded":"2024-09-18T07:35:49.917119+0000","last_fullsized":"2024-09-18T07:35:49.917119+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":302,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916632+0000","last_change":"2024-09-18T07:34:39.639200+0000","last_active":"2024-09-18T07:35:49.916632+0000","last_peered":"2024-09-18T07:35:49.916632+0000","last_clean":"2024-09-18T07:35:49.916632+0000","last_became_active":"2024-09-18T07:30:56.282357+0000","last_became_peered":"2024-09-18T07:30:56.282357+0000","last_unstale":"2024-09-18T07:35:49.916632+0000","last_undegraded":"2024-09-18T07:35:49.916632+0000","last_fullsized":"2024-09-18T07:35:49.916632+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:39.639148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:39.639148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:58.385140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":322,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.894497+0000","last_change":"2024-09-18T07:35:01.453222+0000","last_active":"2024-09-18T07:35:49.894497+0000","last_peered":"2024-09-18T07:35:49.894497+0000","last_clean":"2024-09-18T07:35:49.894497+0000","last_became_active":"2024-09-18T07:35:01.407787+0000","last_became_peered":"2024-09-18T07:35:01.407787+0000","last_unstale":"2024-09-18T07:35:49.894497+0000","last_undegraded":"2024-09-18T07:35:49.894497+0000","last_fullsized":"2024-09-18T07:35:49.894497+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:08.125087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":286,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917265+0000","last_change":"2024-09-18T07:33:32.589362+0000","last_active":"2024-09-18T07:35:49.917265+0000","last_peered":"2024-09-18T07:35:49.917265+0000","last_clean":"2024-09-18T07:35:49.917265+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:35:49.917265+0000","last_undegraded":"2024-09-18T07:35:49.917265+0000","last_fullsized":"2024-09-18T07:35:49.917265+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.589289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.589289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:16.243266+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":62,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.919028+0000","last_change":"2024-09-18T07:35:01.404598+0000","last_active":"2024-09-18T07:35:49.919028+0000","last_peered":"2024-09-18T07:35:49.919028+0000","last_clean":"2024-09-18T07:35:49.919028+0000","last_became_active":"2024-09-18T07:35:01.403293+0000","last_became_peered":"2024-09-18T07:35:01.403293+0000","last_unstale":"2024-09-18T07:35:49.919028+0000","last_undegraded":"2024-09-18T07:35:49.919028+0000","last_fullsized":"2024-09-18T07:35:49.919028+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:58.688631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:58.688631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":332,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917528+0000","last_change":"2024-09-18T07:35:01.448510+0000","last_active":"2024-09-18T07:35:49.917528+0000","last_peered":"2024-09-18T07:35:49.917528+0000","last_clean":"2024-09-18T07:35:49.917528+0000","last_became_active":"2024-09-18T07:35:01.414458+0000","last_became_peered":"2024-09-18T07:35:01.414458+0000","last_unstale":"2024-09-18T07:35:49.917528+0000","last_undegraded":"2024-09-18T07:35:49.917528+0000","last_fullsized":"2024-09-18T07:35:49.917528+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:17.528242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":62,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917504+0000","last_change":"2024-09-18T07:35:01.398699+0000","last_active":"2024-09-18T07:35:49.917504+0000","last_peered":"2024-09-18T07:35:49.917504+0000","last_clean":"2024-09-18T07:35:49.917504+0000","last_became_active":"2024-09-18T07:35:01.398380+0000","last_became_peered":"2024-09-18T07:35:01.398380+0000","last_unstale":"2024-09-18T07:35:49.917504+0000","last_undegraded":"2024-09-18T07:35:49.917504+0000","last_fullsized":"2024-09-18T07:35:49.917504+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:58.901574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:58.901574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:00.777032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":301,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.894446+0000","last_change":"2024-09-18T07:35:01.420446+0000","last_active":"2024-09-18T07:35:49.894446+0000","last_peered":"2024-09-18T07:35:49.894446+0000","last_clean":"2024-09-18T07:35:49.894446+0000","last_became_active":"2024-09-18T07:35:01.420133+0000","last_became_peered":"2024-09-18T07:35:01.420133+0000","last_unstale":"2024-09-18T07:35:49.894446+0000","last_undegraded":"2024-09-18T07:35:49.894446+0000","last_fullsized":"2024-09-18T07:35:49.894446+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:58.276259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:58.276259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":290,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916134+0000","last_change":"2024-09-18T07:35:01.396655+0000","last_active":"2024-09-18T07:35:49.916134+0000","last_peered":"2024-09-18T07:35:49.916134+0000","last_clean":"2024-09-18T07:35:49.916134+0000","last_became_active":"2024-09-18T07:35:01.396305+0000","last_became_peered":"2024-09-18T07:35:01.396305+0000","last_unstale":"2024-09-18T07:35:49.916134+0000","last_undegraded":"2024-09-18T07:35:49.916134+0000","last_fullsized":"2024-09-18T07:35:49.916134+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:09.906976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":340,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918092+0000","last_change":"2024-09-18T07:35:02.342002+0000","last_active":"2024-09-18T07:35:49.918092+0000","last_peered":"2024-09-18T07:35:49.918092+0000","last_clean":"2024-09-18T07:35:49.918092+0000","last_became_active":"2024-09-18T07:35:01.416744+0000","last_became_peered":"2024-09-18T07:35:01.416744+0000","last_unstale":"2024-09-18T07:35:49.918092+0000","last_undegraded":"2024-09-18T07:35:49.918092+0000","last_fullsized":"2024-09-18T07:35:49.918092+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:35:02.341919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.341919+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:21.096808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":308,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916618+0000","last_change":"2024-09-18T07:35:01.414704+0000","last_active":"2024-09-18T07:35:49.916618+0000","last_peered":"2024-09-18T07:35:49.916618+0000","last_clean":"2024-09-18T07:35:49.916618+0000","last_became_active":"2024-09-18T07:35:01.414024+0000","last_became_peered":"2024-09-18T07:35:01.414024+0000","last_unstale":"2024-09-18T07:35:49.916618+0000","last_undegraded":"2024-09-18T07:35:49.916618+0000","last_fullsized":"2024-09-18T07:35:49.916618+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:35.707148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:35.707148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"173'10","reported_seq":369,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892857+0000","last_change":"2024-09-18T07:35:01.449006+0000","last_active":"2024-09-18T07:35:49.892857+0000","last_peered":"2024-09-18T07:35:49.892857+0000","last_clean":"2024-09-18T07:35:49.892857+0000","last_became_active":"2024-09-18T07:35:01.407568+0000","last_became_peered":"2024-09-18T07:35:01.407568+0000","last_unstale":"2024-09-18T07:35:49.892857+0000","last_undegraded":"2024-09-18T07:35:49.892857+0000","last_fullsized":"2024-09-18T07:35:49.892857+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:07.146036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":276,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.893270+0000","last_change":"2024-09-18T07:31:02.343206+0000","last_active":"2024-09-18T07:35:49.893270+0000","last_peered":"2024-09-18T07:35:49.893270+0000","last_clean":"2024-09-18T07:35:49.893270+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:35:49.893270+0000","last_undegraded":"2024-09-18T07:35:49.893270+0000","last_fullsized":"2024-09-18T07:35:49.893270+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:08.883011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":344,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.893148+0000","last_change":"2024-09-18T07:35:01.459777+0000","last_active":"2024-09-18T07:35:49.893148+0000","last_peered":"2024-09-18T07:35:49.893148+0000","last_clean":"2024-09-18T07:35:49.893148+0000","last_became_active":"2024-09-18T07:35:01.407738+0000","last_became_peered":"2024-09-18T07:35:01.407738+0000","last_unstale":"2024-09-18T07:35:49.893148+0000","last_undegraded":"2024-09-18T07:35:49.893148+0000","last_fullsized":"2024-09-18T07:35:49.893148+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:33:54.295330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:54.295330+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":62,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917367+0000","last_change":"2024-09-18T07:35:01.400514+0000","last_active":"2024-09-18T07:35:49.917367+0000","last_peered":"2024-09-18T07:35:49.917367+0000","last_clean":"2024-09-18T07:35:49.917367+0000","last_became_active":"2024-09-18T07:35:01.400197+0000","last_became_peered":"2024-09-18T07:35:01.400197+0000","last_unstale":"2024-09-18T07:35:49.917367+0000","last_undegraded":"2024-09-18T07:35:49.917367+0000","last_fullsized":"2024-09-18T07:35:49.917367+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:31.934890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.934890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:01.325843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":349,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:51.396048+0000","last_change":"2024-09-18T07:31:00.332729+0000","last_active":"2024-09-18T07:35:51.396048+0000","last_peered":"2024-09-18T07:35:51.396048+0000","last_clean":"2024-09-18T07:35:51.396048+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:35:51.396048+0000","last_undegraded":"2024-09-18T07:35:51.396048+0000","last_fullsized":"2024-09-18T07:35:51.396048+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:15.200098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":295,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917086+0000","last_change":"2024-09-18T07:35:01.411256+0000","last_active":"2024-09-18T07:35:49.917086+0000","last_peered":"2024-09-18T07:35:49.917086+0000","last_clean":"2024-09-18T07:35:49.917086+0000","last_became_active":"2024-09-18T07:35:01.410561+0000","last_became_peered":"2024-09-18T07:35:01.410561+0000","last_unstale":"2024-09-18T07:35:49.917086+0000","last_undegraded":"2024-09-18T07:35:49.917086+0000","last_fullsized":"2024-09-18T07:35:49.917086+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.134847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.134847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":311,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892611+0000","last_change":"2024-09-18T07:34:56.881850+0000","last_active":"2024-09-18T07:35:49.892611+0000","last_peered":"2024-09-18T07:35:49.892611+0000","last_clean":"2024-09-18T07:35:49.892611+0000","last_became_active":"2024-09-18T07:30:58.314428+0000","last_became_peered":"2024-09-18T07:30:58.314428+0000","last_unstale":"2024-09-18T07:35:49.892611+0000","last_undegraded":"2024-09-18T07:35:49.892611+0000","last_fullsized":"2024-09-18T07:35:49.892611+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:34:56.881769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:56.881769+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:01.109909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":302,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892880+0000","last_change":"2024-09-18T07:35:01.401235+0000","last_active":"2024-09-18T07:35:49.892880+0000","last_peered":"2024-09-18T07:35:49.892880+0000","last_clean":"2024-09-18T07:35:49.892880+0000","last_became_active":"2024-09-18T07:35:01.400615+0000","last_became_peered":"2024-09-18T07:35:01.400615+0000","last_unstale":"2024-09-18T07:35:49.892880+0000","last_undegraded":"2024-09-18T07:35:49.892880+0000","last_fullsized":"2024-09-18T07:35:49.892880+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:12.005003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":280,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892770+0000","last_change":"2024-09-18T07:31:00.337826+0000","last_active":"2024-09-18T07:35:49.892770+0000","last_peered":"2024-09-18T07:35:49.892770+0000","last_clean":"2024-09-18T07:35:49.892770+0000","last_became_active":"2024-09-18T07:31:00.334158+0000","last_became_peered":"2024-09-18T07:31:00.334158+0000","last_unstale":"2024-09-18T07:35:49.892770+0000","last_undegraded":"2024-09-18T07:35:49.892770+0000","last_fullsized":"2024-09-18T07:35:49.892770+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:45.013191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":275,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.891776+0000","last_change":"2024-09-18T07:31:02.351247+0000","last_active":"2024-09-18T07:35:49.891776+0000","last_peered":"2024-09-18T07:35:49.891776+0000","last_clean":"2024-09-18T07:35:49.891776+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:35:49.891776+0000","last_undegraded":"2024-09-18T07:35:49.891776+0000","last_fullsized":"2024-09-18T07:35:49.891776+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:12.115708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":186,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916331+0000","last_change":"2024-09-18T07:35:23.572395+0000","last_active":"2024-09-18T07:35:49.916331+0000","last_peered":"2024-09-18T07:35:49.916331+0000","last_clean":"2024-09-18T07:35:49.916331+0000","last_became_active":"2024-09-18T07:33:27.003887+0000","last_became_peered":"2024-09-18T07:33:27.003887+0000","last_unstale":"2024-09-18T07:35:49.916331+0000","last_undegraded":"2024-09-18T07:35:49.916331+0000","last_fullsized":"2024-09-18T07:35:49.916331+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:35:23.572304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:23.572304+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:35.224701+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":62,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918976+0000","last_change":"2024-09-18T07:35:01.406178+0000","last_active":"2024-09-18T07:35:49.918976+0000","last_peered":"2024-09-18T07:35:49.918976+0000","last_clean":"2024-09-18T07:35:49.918976+0000","last_became_active":"2024-09-18T07:35:01.405287+0000","last_became_peered":"2024-09-18T07:35:01.405287+0000","last_unstale":"2024-09-18T07:35:49.918976+0000","last_undegraded":"2024-09-18T07:35:49.918976+0000","last_fullsized":"2024-09-18T07:35:49.918976+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:34.322659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:34.322659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:56.559071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":287,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.894325+0000","last_change":"2024-09-18T07:35:01.205647+0000","last_active":"2024-09-18T07:35:49.894325+0000","last_peered":"2024-09-18T07:35:49.894325+0000","last_clean":"2024-09-18T07:35:49.894325+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:35:49.894325+0000","last_undegraded":"2024-09-18T07:35:49.894325+0000","last_fullsized":"2024-09-18T07:35:49.894325+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:01.205596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:01.205596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:10.661983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":290,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.916720+0000","last_change":"2024-09-18T07:35:20.576655+0000","last_active":"2024-09-18T07:35:49.916720+0000","last_peered":"2024-09-18T07:35:49.916720+0000","last_clean":"2024-09-18T07:35:49.916720+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:35:49.916720+0000","last_undegraded":"2024-09-18T07:35:49.916720+0000","last_fullsized":"2024-09-18T07:35:49.916720+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:20.576587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:20.576587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:34.998253+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":348,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.891990+0000","last_change":"2024-09-18T07:35:06.201799+0000","last_active":"2024-09-18T07:35:49.891990+0000","last_peered":"2024-09-18T07:35:49.891990+0000","last_clean":"2024-09-18T07:35:49.891990+0000","last_became_active":"2024-09-18T07:35:01.407576+0000","last_became_peered":"2024-09-18T07:35:01.407576+0000","last_unstale":"2024-09-18T07:35:49.891990+0000","last_undegraded":"2024-09-18T07:35:49.891990+0000","last_fullsized":"2024-09-18T07:35:49.891990+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:35:06.201725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:06.201725+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:24.293384+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":308,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917443+0000","last_change":"2024-09-18T07:35:01.415691+0000","last_active":"2024-09-18T07:35:49.917443+0000","last_peered":"2024-09-18T07:35:49.917443+0000","last_clean":"2024-09-18T07:35:49.917443+0000","last_became_active":"2024-09-18T07:35:01.415261+0000","last_became_peered":"2024-09-18T07:35:01.415261+0000","last_unstale":"2024-09-18T07:35:49.917443+0000","last_undegraded":"2024-09-18T07:35:49.917443+0000","last_fullsized":"2024-09-18T07:35:49.917443+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:58.734027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:58.734027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:08.226648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":70,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918509+0000","last_change":"2024-09-18T07:35:41.311353+0000","last_active":"2024-09-18T07:35:49.918509+0000","last_peered":"2024-09-18T07:35:49.918509+0000","last_clean":"2024-09-18T07:35:49.918509+0000","last_became_active":"2024-09-18T07:35:01.400714+0000","last_became_peered":"2024-09-18T07:35:01.400714+0000","last_unstale":"2024-09-18T07:35:49.918509+0000","last_undegraded":"2024-09-18T07:35:49.918509+0000","last_fullsized":"2024-09-18T07:35:49.918509+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:41.311291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:41.311291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:58.412847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":291,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.894393+0000","last_change":"2024-09-18T07:35:36.026283+0000","last_active":"2024-09-18T07:35:49.894393+0000","last_peered":"2024-09-18T07:35:49.894393+0000","last_clean":"2024-09-18T07:35:49.894393+0000","last_became_active":"2024-09-18T07:31:02.490994+0000","last_became_peered":"2024-09-18T07:31:02.490994+0000","last_unstale":"2024-09-18T07:35:49.894393+0000","last_undegraded":"2024-09-18T07:35:49.894393+0000","last_fullsized":"2024-09-18T07:35:49.894393+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:36.026215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:36.026215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:40.267712+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":318,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892522+0000","last_change":"2024-09-18T07:35:29.942418+0000","last_active":"2024-09-18T07:35:49.892522+0000","last_peered":"2024-09-18T07:35:49.892522+0000","last_clean":"2024-09-18T07:35:49.892522+0000","last_became_active":"2024-09-18T07:32:26.791562+0000","last_became_peered":"2024-09-18T07:32:26.791562+0000","last_unstale":"2024-09-18T07:35:49.892522+0000","last_undegraded":"2024-09-18T07:35:49.892522+0000","last_fullsized":"2024-09-18T07:35:49.892522+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:35:29.937819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:29.937819+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:40.163643+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":286,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917504+0000","last_change":"2024-09-18T07:30:56.283643+0000","last_active":"2024-09-18T07:35:49.917504+0000","last_peered":"2024-09-18T07:35:49.917504+0000","last_clean":"2024-09-18T07:35:49.917504+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:35:49.917504+0000","last_undegraded":"2024-09-18T07:35:49.917504+0000","last_fullsized":"2024-09-18T07:35:49.917504+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:52.458851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":308,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917903+0000","last_change":"2024-09-18T07:35:02.394924+0000","last_active":"2024-09-18T07:35:49.917903+0000","last_peered":"2024-09-18T07:35:49.917903+0000","last_clean":"2024-09-18T07:35:49.917903+0000","last_became_active":"2024-09-18T07:35:01.407641+0000","last_became_peered":"2024-09-18T07:35:01.407641+0000","last_unstale":"2024-09-18T07:35:49.917903+0000","last_undegraded":"2024-09-18T07:35:49.917903+0000","last_fullsized":"2024-09-18T07:35:49.917903+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.394872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.394872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:29.732852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":70,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917075+0000","last_change":"2024-09-18T07:35:29.235047+0000","last_active":"2024-09-18T07:35:49.917075+0000","last_peered":"2024-09-18T07:35:49.917075+0000","last_clean":"2024-09-18T07:35:49.917075+0000","last_became_active":"2024-09-18T07:35:01.400396+0000","last_became_peered":"2024-09-18T07:35:01.400396+0000","last_unstale":"2024-09-18T07:35:49.917075+0000","last_undegraded":"2024-09-18T07:35:49.917075+0000","last_fullsized":"2024-09-18T07:35:49.917075+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:29.234993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:29.234993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:54.885730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":291,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.893255+0000","last_change":"2024-09-18T07:35:02.071672+0000","last_active":"2024-09-18T07:35:49.893255+0000","last_peered":"2024-09-18T07:35:49.893255+0000","last_clean":"2024-09-18T07:35:49.893255+0000","last_became_active":"2024-09-18T07:31:02.347537+0000","last_became_peered":"2024-09-18T07:31:02.347537+0000","last_unstale":"2024-09-18T07:35:49.893255+0000","last_undegraded":"2024-09-18T07:35:49.893255+0000","last_fullsized":"2024-09-18T07:35:49.893255+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.071629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.071629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:17.310933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":301,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.892770+0000","last_change":"2024-09-18T07:33:33.072569+0000","last_active":"2024-09-18T07:35:49.892770+0000","last_peered":"2024-09-18T07:35:49.892770+0000","last_clean":"2024-09-18T07:35:49.892770+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:35:49.892770+0000","last_undegraded":"2024-09-18T07:35:49.892770+0000","last_fullsized":"2024-09-18T07:35:49.892770+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:33.072487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.072487+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":317,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.918965+0000","last_change":"2024-09-18T07:34:58.432839+0000","last_active":"2024-09-18T07:35:49.918965+0000","last_peered":"2024-09-18T07:35:49.918965+0000","last_clean":"2024-09-18T07:35:49.918965+0000","last_became_active":"2024-09-18T07:30:58.313556+0000","last_became_peered":"2024-09-18T07:30:58.313556+0000","last_unstale":"2024-09-18T07:35:49.918965+0000","last_undegraded":"2024-09-18T07:35:49.918965+0000","last_fullsized":"2024-09-18T07:35:49.918965+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:34:58.432781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:58.432781+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:13.006459+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":308,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-18T07:35:49.917367+0000","last_change":"2024-09-18T07:35:25.576024+0000","last_active":"2024-09-18T07:35:49.917367+0000","last_peered":"2024-09-18T07:35:49.917367+0000","last_clean":"2024-09-18T07:35:49.917367+0000","last_became_active":"2024-09-18T07:35:01.410469+0000","last_became_peered":"2024-09-18T07:35:01.410469+0000","last_unstale":"2024-09-18T07:35:49.917367+0000","last_undegraded":"2024-09-18T07:35:49.917367+0000","last_fullsized":"2024-09-18T07:35:49.917367+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:25.575954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:25.575954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:44.318630+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":2838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":56,"num_read_kb":51,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201343,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105484,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93639668,"statfs":{"total":95995035648,"available":95887020032,"internally_reserved":0,"allocated":2154496,"data_stored":1450502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462981,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105464,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93639688,"statfs":{"total":95995035648,"available":95887040512,"internally_reserved":0,"allocated":2134016,"data_stored":1449417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724618,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":127096,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93618056,"statfs":{"total":95995035648,"available":95864889344,"internally_reserved":0,"allocated":1544192,"data_stored":859505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953553,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":145808,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93599344,"statfs":{"total":95995035648,"available":95845728256,"internally_reserved":0,"allocated":2224128,"data_stored":1515437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":292768,"kb_used_data":94988,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93452384,"statfs":{"total":95995035648,"available":95695241216,"internally_reserved":0,"allocated":97267712,"data_stored":96543929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":202504647},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":243824,"kb_used_data":100188,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93501328,"statfs":{"total":95995035648,"available":95745359872,"internally_reserved":0,"allocated":102592512,"data_stored":101915806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":149352,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93595800,"statfs":{"total":95995035648,"available":95842099200,"internally_reserved":0,"allocated":1593344,"data_stored":861484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":155480,"kb_used_data":2116,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93589672,"statfs":{"total":95995035648,"available":95835824128,"internally_reserved":0,"allocated":2166784,"data_stored":1451737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:35:55.823 INFO:tasks.thrashosds.thrasher:Setting 4.1b to [7, 6, 2, 4, 3, 5] 2024-09-18T07:35:55.823 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.1b', '7', '6', '2', '4', '3', '5'] 2024-09-18T07:35:55.823 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 4.1b 7 6 2 4 3 5 2024-09-18T07:35:55.825 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:55.865 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:55.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:55.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:55.933 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:55.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:56.034 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:56.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:56.056 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:56.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:56.105 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:55 smithi137 ceph-mon[29408]: from='client.15237 v1:172.21.15.137:0/2647393136' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:35:56.105 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:55 smithi137 ceph-mon[39175]: from='client.15237 v1:172.21.15.137:0/2647393136' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:35:56.105 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:56.158 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:56.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:56.181 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:56.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:56.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:55 smithi203 ceph-mon[32237]: from='client.15237 v1:172.21.15.137:0/2647393136' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:35:56.283 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:56.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:35:56.306 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:56.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:56.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:56.407 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:56.430 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:56.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:56.532 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:56.553 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:56.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:56.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:56.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:56.676 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:56.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:56.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:56.778 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:56.800 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:56.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:56.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:56.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:35:56.923 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:56.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:56.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:35:57.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:57.047 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:57.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:57.092 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:35:57.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:57.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:35:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:35:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:35:57.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:56 smithi137 ceph-mon[29408]: pgmap v264: 132 pgs: 132 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-18T07:35:57.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:56 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:35:57.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:56 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:35:57.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:56 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:35:57.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:56 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:35:57.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:56 smithi137 ceph-mon[39175]: pgmap v264: 132 pgs: 132 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-18T07:35:57.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:56 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:35:57.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:56 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:35:57.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:56 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:35:57.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:56 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:35:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:57.173 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:57.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:57.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:35:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:35:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:56 smithi203 ceph-mon[32237]: pgmap v264: 132 pgs: 132 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-18T07:35:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:56 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:35:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:56 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:35:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:56 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:35:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:56 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:35:57.274 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:57.293 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:35:57.293 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:35:57.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:57.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:57.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:57.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:57.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:57.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:57.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:57.523 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:57.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:57.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:57.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:57.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:57.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:57.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:57.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:57.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:57.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:57.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:57.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:57.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:57.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:57.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:58.015 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:58.042 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:58.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:58.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:58.093 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'unique_pool_1' created 2024-09-18T07:35:58.140 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:57 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/758733540' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:35:58.141 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:57 smithi137 ceph-mon[29408]: from='client.35132 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:35:58.141 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:57 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:58.141 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:57 smithi137 ceph-mon[29408]: from='client.35132 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:35:58.141 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:57 smithi137 ceph-mon[29408]: osdmap e200: 8 total, 8 up, 8 in 2024-09-18T07:35:58.141 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:57 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3924013374' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-18T07:35:58.141 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:57 smithi137 ceph-mon[29408]: 3.15 scrub starts 2024-09-18T07:35:58.141 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:57 smithi137 ceph-mon[29408]: 3.15 scrub ok 2024-09-18T07:35:58.141 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:57 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/758733540' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:35:58.141 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:57 smithi137 ceph-mon[39175]: from='client.35132 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:35:58.142 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:57 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:58.142 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:57 smithi137 ceph-mon[39175]: from='client.35132 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:35:58.142 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:57 smithi137 ceph-mon[39175]: osdmap e200: 8 total, 8 up, 8 in 2024-09-18T07:35:58.142 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:57 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3924013374' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-18T07:35:58.142 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:57 smithi137 ceph-mon[39175]: 3.15 scrub starts 2024-09-18T07:35:58.142 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:57 smithi137 ceph-mon[39175]: 3.15 scrub ok 2024-09-18T07:35:58.143 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:58.167 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:58.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:58.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:58.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:57 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/758733540' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:35:58.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:57 smithi203 ceph-mon[32237]: from='client.35132 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:35:58.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:57 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:35:58.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:57 smithi203 ceph-mon[32237]: from='client.35132 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:35:58.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:57 smithi203 ceph-mon[32237]: osdmap e200: 8 total, 8 up, 8 in 2024-09-18T07:35:58.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:57 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3924013374' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-18T07:35:58.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:57 smithi203 ceph-mon[32237]: 3.15 scrub starts 2024-09-18T07:35:58.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:57 smithi203 ceph-mon[32237]: 3.15 scrub ok 2024-09-18T07:35:58.270 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:58.294 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:58.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:58.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:58.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:58.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:58.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:58.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:58.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:35:58.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:58.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:58.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:35:58.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:58.668 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:58.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:58.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:58.770 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:35:58.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-18T07:35:58.796 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:58.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:58.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:35:58.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:58.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:58.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:58.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:59.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:59.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:59.045 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:59.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:59.146 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:59.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:58 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:59.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:58 smithi137 ceph-mon[29408]: 4.1e scrub starts 2024-09-18T07:35:59.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:58 smithi137 ceph-mon[29408]: 4.1e scrub ok 2024-09-18T07:35:59.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:58 smithi137 ceph-mon[29408]: pgmap v266: 132 pgs: 132 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:35:59.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:58 smithi137 ceph-mon[29408]: 3.18 scrub starts 2024-09-18T07:35:59.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:58 smithi137 ceph-mon[29408]: 3.18 scrub ok 2024-09-18T07:35:59.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:58 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3924013374' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-18T07:35:59.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:35:58 smithi137 ceph-mon[29408]: osdmap e201: 8 total, 8 up, 8 in 2024-09-18T07:35:59.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:58 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:59.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:58 smithi137 ceph-mon[39175]: 4.1e scrub starts 2024-09-18T07:35:59.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:58 smithi137 ceph-mon[39175]: 4.1e scrub ok 2024-09-18T07:35:59.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:58 smithi137 ceph-mon[39175]: pgmap v266: 132 pgs: 132 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:35:59.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:58 smithi137 ceph-mon[39175]: 3.18 scrub starts 2024-09-18T07:35:59.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:58 smithi137 ceph-mon[39175]: 3.18 scrub ok 2024-09-18T07:35:59.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:58 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3924013374' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-18T07:35:59.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:35:58 smithi137 ceph-mon[39175]: osdmap e201: 8 total, 8 up, 8 in 2024-09-18T07:35:59.172 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:59.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:59.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:35:59.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:58 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:35:59.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:58 smithi203 ceph-mon[32237]: 4.1e scrub starts 2024-09-18T07:35:59.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:58 smithi203 ceph-mon[32237]: 4.1e scrub ok 2024-09-18T07:35:59.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:58 smithi203 ceph-mon[32237]: pgmap v266: 132 pgs: 132 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:35:59.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:58 smithi203 ceph-mon[32237]: 3.18 scrub starts 2024-09-18T07:35:59.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:58 smithi203 ceph-mon[32237]: 3.18 scrub ok 2024-09-18T07:35:59.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:58 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3924013374' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-18T07:35:59.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:35:58 smithi203 ceph-mon[32237]: osdmap e201: 8 total, 8 up, 8 in 2024-09-18T07:35:59.238 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:59.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T07:35:59.273 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:35:59.330 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:59.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:59.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:35:59.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:59.458 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:59.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:59.560 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:35:59.585 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:59.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:59.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:35:59.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:35:59.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:35:59.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:59.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:35:59.727 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:59.737 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:35:59.815 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:35:59.840 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:59.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:59.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:35:59.942 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:35:59.966 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:35:59.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:35:59.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:00.068 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:00.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:00.095 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:00.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:00.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:36:00.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:00 smithi137 ceph-mon[29408]: osdmap e202: 8 total, 8 up, 8 in 2024-09-18T07:36:00.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:00 smithi137 ceph-mon[29408]: pgmap v269: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:36:00.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:00 smithi137 ceph-mon[29408]: osdmap e203: 8 total, 8 up, 8 in 2024-09-18T07:36:00.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:00 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4282648288' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.1b", "id": [7, 6, 2, 4, 3, 5]} : dispatch 2024-09-18T07:36:00.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:00 smithi137 ceph-mon[29408]: from='client.35144 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.1b", "id": [7, 6, 2, 4, 3, 5]} : dispatch 2024-09-18T07:36:00.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:00 smithi137 ceph-mon[39175]: osdmap e202: 8 total, 8 up, 8 in 2024-09-18T07:36:00.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:00 smithi137 ceph-mon[39175]: pgmap v269: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:36:00.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:00 smithi137 ceph-mon[39175]: osdmap e203: 8 total, 8 up, 8 in 2024-09-18T07:36:00.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:00 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4282648288' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.1b", "id": [7, 6, 2, 4, 3, 5]} : dispatch 2024-09-18T07:36:00.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:00 smithi137 ceph-mon[39175]: from='client.35144 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.1b", "id": [7, 6, 2, 4, 3, 5]} : dispatch 2024-09-18T07:36:00.196 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:00.222 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:00.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:00.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:00.324 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:00.348 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:00.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:00.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:00.402 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:36:00.403 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:36:00.424 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:36:00.425 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:36:00.440 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:36:00.441 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:36:00.450 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:00.456 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:36:00.456 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:36:00.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:00 smithi203 ceph-mon[32237]: osdmap e202: 8 total, 8 up, 8 in 2024-09-18T07:36:00.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:00 smithi203 ceph-mon[32237]: pgmap v269: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:36:00.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:00 smithi203 ceph-mon[32237]: osdmap e203: 8 total, 8 up, 8 in 2024-09-18T07:36:00.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:00 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4282648288' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.1b", "id": [7, 6, 2, 4, 3, 5]} : dispatch 2024-09-18T07:36:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:00 smithi203 ceph-mon[32237]: from='client.35144 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.1b", "id": [7, 6, 2, 4, 3, 5]} : dispatch 2024-09-18T07:36:00.473 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:36:00.473 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:36:00.474 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:00.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:00.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:00.489 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:36:00.490 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:36:00.506 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:36:00.506 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:36:00.524 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:36:00.524 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:36:00.575 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:00.599 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:00.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:00.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:00.619 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:36:00.619 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:36:00.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:36:00.620 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:36:00.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.620 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135022 7.13 7:cab60f17:::benchmark_data_smithi137_74180_object30606:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:36:00.620 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964232+0000", 2024-09-18T07:36:00.620 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 103.651662782, 2024-09-18T07:36:00.620 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2320603539999999, 2024-09-18T07:36:00.620 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.620 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.620 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.621 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.621 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.621 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.621 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135022 2024-09-18T07:36:00.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.621 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964232+0000", 2024-09-18T07:36:00.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964232+0000", 2024-09-18T07:36:00.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964235+0000", 2024-09-18T07:36:00.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9340000000000002e-06 2024-09-18T07:36:00.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964242+0000", 2024-09-18T07:36:00.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.379e-06 2024-09-18T07:36:00.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964261+0000", 2024-09-18T07:36:00.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8851999999999998e-05 2024-09-18T07:36:00.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964270+0000", 2024-09-18T07:36:00.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5040000000000008e-06 2024-09-18T07:36:00.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964336+0000", 2024-09-18T07:36:00.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5092999999999995e-05 2024-09-18T07:36:00.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:36:00.625 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964344+0000", 2024-09-18T07:36:00.625 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0879999999999995e-06 2024-09-18T07:36:00.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.625 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.625 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971555+0000", 2024-09-18T07:36:00.625 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0072113109999999998 2024-09-18T07:36:00.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.625 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.625 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971563+0000", 2024-09-18T07:36:00.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4689999999999997e-06 2024-09-18T07:36:00.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.626 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.190908+0000", 2024-09-18T07:36:00.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2193444609999999 2024-09-18T07:36:00.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.626 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.191034+0000", 2024-09-18T07:36:00.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000125871 2024-09-18T07:36:00.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3,4,5", 2024-09-18T07:36:00.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.191148+0000", 2024-09-18T07:36:00.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114097 2024-09-18T07:36:00.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.193787+0000", 2024-09-18T07:36:00.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026389650000000001 2024-09-18T07:36:00.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.194990+0000", 2024-09-18T07:36:00.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001203523 2024-09-18T07:36:00.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.195790+0000", 2024-09-18T07:36:00.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00079991899999999996 2024-09-18T07:36:00.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.195803+0000", 2024-09-18T07:36:00.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2955999999999999e-05 2024-09-18T07:36:00.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.196126+0000", 2024-09-18T07:36:00.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032304700000000003 2024-09-18T07:36:00.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.630 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.196180+0000", 2024-09-18T07:36:00.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4178000000000002e-05 2024-09-18T07:36:00.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.630 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.630 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.196292+0000", 2024-09-18T07:36:00.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011170700000000001 2024-09-18T07:36:00.630 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.630 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.630 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.631 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151745 7.8 7:11d4aec3:::benchmark_data_smithi137_74180_object47329:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-18T07:36:00.631 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.256412+0000", 2024-09-18T07:36:00.631 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 80.359482545999995, 2024-09-18T07:36:00.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96419230700000003, 2024-09-18T07:36:00.631 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.631 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.631 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.631 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.631 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.631 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.632 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151745 2024-09-18T07:36:00.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.632 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256412+0000", 2024-09-18T07:36:00.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256412+0000", 2024-09-18T07:36:00.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256413+0000", 2024-09-18T07:36:00.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1900000000000002e-07 2024-09-18T07:36:00.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256415+0000", 2024-09-18T07:36:00.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4880000000000001e-06 2024-09-18T07:36:00.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256421+0000", 2024-09-18T07:36:00.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7019999999999998e-06 2024-09-18T07:36:00.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256426+0000", 2024-09-18T07:36:00.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1000000000000003e-06 2024-09-18T07:36:00.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282222+0000", 2024-09-18T07:36:00.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025796116000000001 2024-09-18T07:36:00.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282233+0000", 2024-09-18T07:36:00.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0539e-05 2024-09-18T07:36:00.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.636 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.215761+0000", 2024-09-18T07:36:00.636 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93352816100000002 2024-09-18T07:36:00.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.636 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.636 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.215857+0000", 2024-09-18T07:36:00.636 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6032999999999993e-05 2024-09-18T07:36:00.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.636 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:36:00.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.215960+0000", 2024-09-18T07:36:00.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103329 2024-09-18T07:36:00.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217112+0000", 2024-09-18T07:36:00.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011516709999999999 2024-09-18T07:36:00.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218052+0000", 2024-09-18T07:36:00.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00093980599999999995 2024-09-18T07:36:00.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220525+0000", 2024-09-18T07:36:00.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024736509999999999 2024-09-18T07:36:00.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220565+0000", 2024-09-18T07:36:00.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9847e-05 2024-09-18T07:36:00.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220604+0000", 2024-09-18T07:36:00.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9245000000000001e-05 2024-09-18T07:36:00.640 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.640 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.641 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.641 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151823 7.8 7:101e05c7:::benchmark_data_smithi137_74180_object47407:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-18T07:36:00.641 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.256432+0000", 2024-09-18T07:36:00.641 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 80.359463066000004, 2024-09-18T07:36:00.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96425229099999998, 2024-09-18T07:36:00.641 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.641 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.641 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.641 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.642 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.642 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.642 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151823 2024-09-18T07:36:00.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.642 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.642 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.642 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256432+0000", 2024-09-18T07:36:00.642 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256432+0000", 2024-09-18T07:36:00.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256432+0000", 2024-09-18T07:36:00.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6199999999999998e-07 2024-09-18T07:36:00.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256435+0000", 2024-09-18T07:36:00.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4600000000000002e-06 2024-09-18T07:36:00.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256440+0000", 2024-09-18T07:36:00.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9830000000000004e-06 2024-09-18T07:36:00.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256444+0000", 2024-09-18T07:36:00.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6909999999999996e-06 2024-09-18T07:36:00.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282270+0000", 2024-09-18T07:36:00.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025825796000000002 2024-09-18T07:36:00.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.646 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282277+0000", 2024-09-18T07:36:00.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2989999999999999e-06 2024-09-18T07:36:00.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.646 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.646 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216194+0000", 2024-09-18T07:36:00.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93391638700000001 2024-09-18T07:36:00.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.646 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.646 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216288+0000", 2024-09-18T07:36:00.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3845999999999996e-05 2024-09-18T07:36:00.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:36:00.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216381+0000", 2024-09-18T07:36:00.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3960000000000002e-05 2024-09-18T07:36:00.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217934+0000", 2024-09-18T07:36:00.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001552257 2024-09-18T07:36:00.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218106+0000", 2024-09-18T07:36:00.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017274099999999999 2024-09-18T07:36:00.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220613+0000", 2024-09-18T07:36:00.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025060780000000001 2024-09-18T07:36:00.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220649+0000", 2024-09-18T07:36:00.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6380000000000001e-05 2024-09-18T07:36:00.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220684+0000", 2024-09-18T07:36:00.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4851000000000002e-05 2024-09-18T07:36:00.649 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.649 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.649 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.650 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151853 7.8 7:10bf2ca7:::benchmark_data_smithi137_74180_object47437:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-18T07:36:00.650 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.256449+0000", 2024-09-18T07:36:00.650 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 80.359445407999999, 2024-09-18T07:36:00.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96431083500000003, 2024-09-18T07:36:00.650 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.650 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.650 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.650 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.650 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.650 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.651 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151853 2024-09-18T07:36:00.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.651 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256449+0000", 2024-09-18T07:36:00.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256449+0000", 2024-09-18T07:36:00.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256450+0000", 2024-09-18T07:36:00.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3799999999999998e-07 2024-09-18T07:36:00.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256452+0000", 2024-09-18T07:36:00.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.582e-06 2024-09-18T07:36:00.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256457+0000", 2024-09-18T07:36:00.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3029999999999999e-06 2024-09-18T07:36:00.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256463+0000", 2024-09-18T07:36:00.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0540000000000002e-06 2024-09-18T07:36:00.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282322+0000", 2024-09-18T07:36:00.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025859495999999999 2024-09-18T07:36:00.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282329+0000", 2024-09-18T07:36:00.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.391e-06 2024-09-18T07:36:00.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216585+0000", 2024-09-18T07:36:00.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93425595100000003 2024-09-18T07:36:00.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216677+0000", 2024-09-18T07:36:00.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1812999999999997e-05 2024-09-18T07:36:00.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:36:00.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216781+0000", 2024-09-18T07:36:00.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103607 2024-09-18T07:36:00.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217970+0000", 2024-09-18T07:36:00.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011890959999999999 2024-09-18T07:36:00.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218137+0000", 2024-09-18T07:36:00.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016691300000000001 2024-09-18T07:36:00.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220692+0000", 2024-09-18T07:36:00.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002555571 2024-09-18T07:36:00.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220729+0000", 2024-09-18T07:36:00.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6128999999999999e-05 2024-09-18T07:36:00.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220760+0000", 2024-09-18T07:36:00.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1491000000000001e-05 2024-09-18T07:36:00.658 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.658 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.658 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.659 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151969 7.8 7:10c7e652:::benchmark_data_smithi137_74180_object47553:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-18T07:36:00.659 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.256468+0000", 2024-09-18T07:36:00.659 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 80.359426885000005, 2024-09-18T07:36:00.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96436882599999996, 2024-09-18T07:36:00.659 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.659 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.659 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.659 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.659 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.660 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.660 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151969 2024-09-18T07:36:00.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.660 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256468+0000", 2024-09-18T07:36:00.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256468+0000", 2024-09-18T07:36:00.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256468+0000", 2024-09-18T07:36:00.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9500000000000003e-07 2024-09-18T07:36:00.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256471+0000", 2024-09-18T07:36:00.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3410000000000001e-06 2024-09-18T07:36:00.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256476+0000", 2024-09-18T07:36:00.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3619999999999997e-06 2024-09-18T07:36:00.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256480+0000", 2024-09-18T07:36:00.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5879999999999999e-06 2024-09-18T07:36:00.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282362+0000", 2024-09-18T07:36:00.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025881686000000001 2024-09-18T07:36:00.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282370+0000", 2024-09-18T07:36:00.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0979999999999994e-06 2024-09-18T07:36:00.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217145+0000", 2024-09-18T07:36:00.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93477450600000001 2024-09-18T07:36:00.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217239+0000", 2024-09-18T07:36:00.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3744000000000002e-05 2024-09-18T07:36:00.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:36:00.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217328+0000", 2024-09-18T07:36:00.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9494999999999996e-05 2024-09-18T07:36:00.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.665 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.665 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218079+0000", 2024-09-18T07:36:00.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00075083099999999998 2024-09-18T07:36:00.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218172+0000", 2024-09-18T07:36:00.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2713999999999994e-05 2024-09-18T07:36:00.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220767+0000", 2024-09-18T07:36:00.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002595848 2024-09-18T07:36:00.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220804+0000", 2024-09-18T07:36:00.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6151e-05 2024-09-18T07:36:00.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220837+0000", 2024-09-18T07:36:00.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2966999999999998e-05 2024-09-18T07:36:00.667 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.667 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.667 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.668 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151980 7.8 7:1648d987:::benchmark_data_smithi137_74180_object47564:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-18T07:36:00.668 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.256485+0000", 2024-09-18T07:36:00.668 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 80.359409201000005, 2024-09-18T07:36:00.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96460135300000005, 2024-09-18T07:36:00.668 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.668 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.668 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.668 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.668 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.668 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.668 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151980 2024-09-18T07:36:00.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.669 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256485+0000", 2024-09-18T07:36:00.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256485+0000", 2024-09-18T07:36:00.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256486+0000", 2024-09-18T07:36:00.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8299999999999997e-07 2024-09-18T07:36:00.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256488+0000", 2024-09-18T07:36:00.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1459999999999999e-06 2024-09-18T07:36:00.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256493+0000", 2024-09-18T07:36:00.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.147e-06 2024-09-18T07:36:00.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256498+0000", 2024-09-18T07:36:00.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6530000000000002e-06 2024-09-18T07:36:00.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282399+0000", 2024-09-18T07:36:00.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025901569999999999 2024-09-18T07:36:00.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282406+0000", 2024-09-18T07:36:00.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9380000000000003e-06 2024-09-18T07:36:00.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217538+0000", 2024-09-18T07:36:00.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93513140400000005 2024-09-18T07:36:00.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217632+0000", 2024-09-18T07:36:00.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4285999999999998e-05 2024-09-18T07:36:00.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:36:00.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217727+0000", 2024-09-18T07:36:00.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5168999999999996e-05 2024-09-18T07:36:00.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.219197+0000", 2024-09-18T07:36:00.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014697880000000001 2024-09-18T07:36:00.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.219258+0000", 2024-09-18T07:36:00.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1444000000000003e-05 2024-09-18T07:36:00.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221042+0000", 2024-09-18T07:36:00.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017834820000000001 2024-09-18T07:36:00.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221067+0000", 2024-09-18T07:36:00.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4714000000000001e-05 2024-09-18T07:36:00.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221087+0000", 2024-09-18T07:36:00.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0129e-05 2024-09-18T07:36:00.676 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.676 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.676 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.676 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157484 7.5 7:a765acc6:::benchmark_data_smithi137_74180_object53068:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163)", 2024-09-18T07:36:00.677 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.351294+0000", 2024-09-18T07:36:00.677 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 68.264600635999997, 2024-09-18T07:36:00.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98376080499999996, 2024-09-18T07:36:00.677 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.677 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.677 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:36:00.677 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.677 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.677 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.677 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157484 2024-09-18T07:36:00.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.677 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351294+0000", 2024-09-18T07:36:00.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351294+0000", 2024-09-18T07:36:00.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351298+0000", 2024-09-18T07:36:00.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.929e-06 2024-09-18T07:36:00.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351307+0000", 2024-09-18T07:36:00.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0720000000000006e-06 2024-09-18T07:36:00.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351321+0000", 2024-09-18T07:36:00.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4113e-05 2024-09-18T07:36:00.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351332+0000", 2024-09-18T07:36:00.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1080999999999999e-05 2024-09-18T07:36:00.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.382508+0000", 2024-09-18T07:36:00.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031176124 2024-09-18T07:36:00.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.382519+0000", 2024-09-18T07:36:00.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1139e-05 2024-09-18T07:36:00.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.334948+0000", 2024-09-18T07:36:00.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95242890700000005 2024-09-18T07:36:00.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.335055+0000", 2024-09-18T07:36:00.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010644 2024-09-18T07:36:00.682 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.682 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.682 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.682 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157497 7.5 7:a0bb3991:::benchmark_data_smithi137_74180_object53081:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163)", 2024-09-18T07:36:00.682 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.351358+0000", 2024-09-18T07:36:00.682 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 68.264536531999994, 2024-09-18T07:36:00.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98430336900000004, 2024-09-18T07:36:00.682 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.683 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.683 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:36:00.683 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.683 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.683 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.683 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157497 2024-09-18T07:36:00.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.683 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351358+0000", 2024-09-18T07:36:00.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351358+0000", 2024-09-18T07:36:00.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351359+0000", 2024-09-18T07:36:00.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3699999999999999e-07 2024-09-18T07:36:00.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351362+0000", 2024-09-18T07:36:00.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.207e-06 2024-09-18T07:36:00.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351370+0000", 2024-09-18T07:36:00.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7030000000000006e-06 2024-09-18T07:36:00.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351375+0000", 2024-09-18T07:36:00.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.536e-06 2024-09-18T07:36:00.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.382542+0000", 2024-09-18T07:36:00.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031166337999999998 2024-09-18T07:36:00.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.382548+0000", 2024-09-18T07:36:00.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4910000000000002e-06 2024-09-18T07:36:00.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.335593+0000", 2024-09-18T07:36:00.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95304532900000005 2024-09-18T07:36:00.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.335661+0000", 2024-09-18T07:36:00.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7928000000000004e-05 2024-09-18T07:36:00.687 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.687 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.688 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.688 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157538 7.5 7:a5eca45d:::benchmark_data_smithi137_74180_object53122:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163)", 2024-09-18T07:36:00.688 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.351382+0000", 2024-09-18T07:36:00.688 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 68.264512746999998, 2024-09-18T07:36:00.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98539270599999995, 2024-09-18T07:36:00.688 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.688 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.688 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.688 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.689 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.689 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.689 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157538 2024-09-18T07:36:00.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.689 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351382+0000", 2024-09-18T07:36:00.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351382+0000", 2024-09-18T07:36:00.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351382+0000", 2024-09-18T07:36:00.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4900000000000001e-07 2024-09-18T07:36:00.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351385+0000", 2024-09-18T07:36:00.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3939999999999999e-06 2024-09-18T07:36:00.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351390+0000", 2024-09-18T07:36:00.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9919999999999998e-06 2024-09-18T07:36:00.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351393+0000", 2024-09-18T07:36:00.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7840000000000001e-06 2024-09-18T07:36:00.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.382569+0000", 2024-09-18T07:36:00.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031175986999999999 2024-09-18T07:36:00.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.382576+0000", 2024-09-18T07:36:00.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5220000000000004e-06 2024-09-18T07:36:00.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.335681+0000", 2024-09-18T07:36:00.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95310482900000004 2024-09-18T07:36:00.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.335770+0000", 2024-09-18T07:36:00.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9433999999999998e-05 2024-09-18T07:36:00.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:36:00.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.335841+0000", 2024-09-18T07:36:00.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0946999999999997e-05 2024-09-18T07:36:00.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336661+0000", 2024-09-18T07:36:00.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00082012999999999997 2024-09-18T07:36:00.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336720+0000", 2024-09-18T07:36:00.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8394999999999999e-05 2024-09-18T07:36:00.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336728+0000", 2024-09-18T07:36:00.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3960000000000006e-06 2024-09-18T07:36:00.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336753+0000", 2024-09-18T07:36:00.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4573999999999999e-05 2024-09-18T07:36:00.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336775+0000", 2024-09-18T07:36:00.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1872999999999999e-05 2024-09-18T07:36:00.696 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.696 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.696 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.696 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157554 7.5 7:a5757979:::benchmark_data_smithi137_74180_object53138:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163)", 2024-09-18T07:36:00.696 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.351397+0000", 2024-09-18T07:36:00.697 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 68.264497282999997, 2024-09-18T07:36:00.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98575066499999997, 2024-09-18T07:36:00.697 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.697 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.697 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.697 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.697 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.697 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.697 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157554 2024-09-18T07:36:00.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.697 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351397+0000", 2024-09-18T07:36:00.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351397+0000", 2024-09-18T07:36:00.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351398+0000", 2024-09-18T07:36:00.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8500000000000002e-07 2024-09-18T07:36:00.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351399+0000", 2024-09-18T07:36:00.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.764e-06 2024-09-18T07:36:00.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351402+0000", 2024-09-18T07:36:00.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7250000000000002e-06 2024-09-18T07:36:00.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351405+0000", 2024-09-18T07:36:00.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7439999999999999e-06 2024-09-18T07:36:00.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.382595+0000", 2024-09-18T07:36:00.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031190473999999999 2024-09-18T07:36:00.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.382601+0000", 2024-09-18T07:36:00.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4940000000000003e-06 2024-09-18T07:36:00.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336048+0000", 2024-09-18T07:36:00.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95344718399999995 2024-09-18T07:36:00.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336154+0000", 2024-09-18T07:36:00.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000106029 2024-09-18T07:36:00.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:36:00.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336265+0000", 2024-09-18T07:36:00.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011141000000000001 2024-09-18T07:36:00.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336782+0000", 2024-09-18T07:36:00.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00051622400000000002 2024-09-18T07:36:00.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.337003+0000", 2024-09-18T07:36:00.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000221373 2024-09-18T07:36:00.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.337059+0000", 2024-09-18T07:36:00.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6462999999999998e-05 2024-09-18T07:36:00.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.337109+0000", 2024-09-18T07:36:00.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9227999999999997e-05 2024-09-18T07:36:00.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.337148+0000", 2024-09-18T07:36:00.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9267999999999997e-05 2024-09-18T07:36:00.710 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.710 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.710 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.710 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157635 7.5 7:a8492353:::benchmark_data_smithi137_74180_object53219:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e164)", 2024-09-18T07:36:00.710 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.383256+0000", 2024-09-18T07:36:00.710 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 68.232638511000005, 2024-09-18T07:36:00.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96415525300000005, 2024-09-18T07:36:00.711 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.711 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.711 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.711 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.711 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.711 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.711 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157635 2024-09-18T07:36:00.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.711 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383256+0000", 2024-09-18T07:36:00.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383256+0000", 2024-09-18T07:36:00.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383256+0000", 2024-09-18T07:36:00.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5600000000000001e-07 2024-09-18T07:36:00.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383258+0000", 2024-09-18T07:36:00.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9750000000000001e-06 2024-09-18T07:36:00.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383262+0000", 2024-09-18T07:36:00.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.365e-06 2024-09-18T07:36:00.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383264+0000", 2024-09-18T07:36:00.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6359999999999998e-06 2024-09-18T07:36:00.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383431+0000", 2024-09-18T07:36:00.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016692899999999999 2024-09-18T07:36:00.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383439+0000", 2024-09-18T07:36:00.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0169999999999996e-06 2024-09-18T07:36:00.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.339552+0000", 2024-09-18T07:36:00.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95611299299999997 2024-09-18T07:36:00.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.339643+0000", 2024-09-18T07:36:00.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0787999999999999e-05 2024-09-18T07:36:00.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:36:00.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.339738+0000", 2024-09-18T07:36:00.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4581999999999997e-05 2024-09-18T07:36:00.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341908+0000", 2024-09-18T07:36:00.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0021700529999999999 2024-09-18T07:36:00.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.344559+0000", 2024-09-18T07:36:00.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026512200000000001 2024-09-18T07:36:00.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347336+0000", 2024-09-18T07:36:00.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002777135 2024-09-18T07:36:00.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347377+0000", 2024-09-18T07:36:00.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1074e-05 2024-09-18T07:36:00.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347411+0000", 2024-09-18T07:36:00.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.413e-05 2024-09-18T07:36:00.718 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.719 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.719 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.719 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157640 7.5 7:ace13225:::benchmark_data_smithi137_74180_object53224:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e164)", 2024-09-18T07:36:00.719 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.383278+0000", 2024-09-18T07:36:00.719 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 68.232616507000003, 2024-09-18T07:36:00.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96425578599999995, 2024-09-18T07:36:00.719 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.719 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.719 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.720 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.720 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.720 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.720 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157640 2024-09-18T07:36:00.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.720 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383278+0000", 2024-09-18T07:36:00.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383278+0000", 2024-09-18T07:36:00.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383278+0000", 2024-09-18T07:36:00.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3799999999999998e-07 2024-09-18T07:36:00.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383280+0000", 2024-09-18T07:36:00.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9420000000000002e-06 2024-09-18T07:36:00.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383284+0000", 2024-09-18T07:36:00.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2509999999999999e-06 2024-09-18T07:36:00.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383286+0000", 2024-09-18T07:36:00.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7599999999999998e-06 2024-09-18T07:36:00.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383477+0000", 2024-09-18T07:36:00.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000190261 2024-09-18T07:36:00.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383484+0000", 2024-09-18T07:36:00.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4619999999999999e-06 2024-09-18T07:36:00.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.339906+0000", 2024-09-18T07:36:00.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95642190199999999 2024-09-18T07:36:00.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.340001+0000", 2024-09-18T07:36:00.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4598999999999993e-05 2024-09-18T07:36:00.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:36:00.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.340095+0000", 2024-09-18T07:36:00.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4714000000000002e-05 2024-09-18T07:36:00.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341912+0000", 2024-09-18T07:36:00.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018169449999999999 2024-09-18T07:36:00.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.344597+0000", 2024-09-18T07:36:00.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026846940000000001 2024-09-18T07:36:00.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347445+0000", 2024-09-18T07:36:00.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028480329999999998 2024-09-18T07:36:00.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347485+0000", 2024-09-18T07:36:00.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0330000000000002e-05 2024-09-18T07:36:00.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347534+0000", 2024-09-18T07:36:00.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8554999999999998e-05 2024-09-18T07:36:00.727 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.727 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.727 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157677 7.5 7:ae2907ef:::benchmark_data_smithi137_74180_object53261:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e164)", 2024-09-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.383289+0000", 2024-09-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 68.232605289000006, 2024-09-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96435305199999999, 2024-09-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.729 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:00.730 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.730 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.731 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.731 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.731 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157677 2024-09-18T07:36:00.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.731 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383289+0000", 2024-09-18T07:36:00.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383289+0000", 2024-09-18T07:36:00.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383290+0000", 2024-09-18T07:36:00.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6100000000000002e-07 2024-09-18T07:36:00.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383294+0000", 2024-09-18T07:36:00.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.622e-06 2024-09-18T07:36:00.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383297+0000", 2024-09-18T07:36:00.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.554e-06 2024-09-18T07:36:00.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383299+0000", 2024-09-18T07:36:00.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5189999999999999e-06 2024-09-18T07:36:00.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383519+0000", 2024-09-18T07:36:00.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022010199999999999 2024-09-18T07:36:00.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383527+0000", 2024-09-18T07:36:00.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2860000000000002e-06 2024-09-18T07:36:00.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.340269+0000", 2024-09-18T07:36:00.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95674225800000001 2024-09-18T07:36:00.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.340359+0000", 2024-09-18T07:36:00.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0400000000000002e-05 2024-09-18T07:36:00.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:36:00.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.340468+0000", 2024-09-18T07:36:00.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000109095 2024-09-18T07:36:00.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341917+0000", 2024-09-18T07:36:00.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014483670000000001 2024-09-18T07:36:00.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.344642+0000", 2024-09-18T07:36:00.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002725744 2024-09-18T07:36:00.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347567+0000", 2024-09-18T07:36:00.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029242510000000001 2024-09-18T07:36:00.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347607+0000", 2024-09-18T07:36:00.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0159e-05 2024-09-18T07:36:00.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347642+0000", 2024-09-18T07:36:00.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5434e-05 2024-09-18T07:36:00.738 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.738 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.739 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.739 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157692 7.5 7:aab0905d:::benchmark_data_smithi137_74180_object53276:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e164)", 2024-09-18T07:36:00.739 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.383302+0000", 2024-09-18T07:36:00.739 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 68.232592715999999, 2024-09-18T07:36:00.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96446017699999997, 2024-09-18T07:36:00.739 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.739 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.739 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.739 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.739 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.740 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.740 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157692 2024-09-18T07:36:00.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.740 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383302+0000", 2024-09-18T07:36:00.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383302+0000", 2024-09-18T07:36:00.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383302+0000", 2024-09-18T07:36:00.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7399999999999999e-07 2024-09-18T07:36:00.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383303+0000", 2024-09-18T07:36:00.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.189e-06 2024-09-18T07:36:00.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383306+0000", 2024-09-18T07:36:00.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1760000000000002e-06 2024-09-18T07:36:00.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383311+0000", 2024-09-18T07:36:00.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.206e-06 2024-09-18T07:36:00.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383563+0000", 2024-09-18T07:36:00.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000252434 2024-09-18T07:36:00.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383570+0000", 2024-09-18T07:36:00.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8759999999999999e-06 2024-09-18T07:36:00.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.340670+0000", 2024-09-18T07:36:00.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95709987200000002 2024-09-18T07:36:00.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.340770+0000", 2024-09-18T07:36:00.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9798999999999998e-05 2024-09-18T07:36:00.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:36:00.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.340866+0000", 2024-09-18T07:36:00.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5851000000000001e-05 2024-09-18T07:36:00.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341921+0000", 2024-09-18T07:36:00.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010552319999999999 2024-09-18T07:36:00.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.345122+0000", 2024-09-18T07:36:00.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003201668 2024-09-18T07:36:00.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347679+0000", 2024-09-18T07:36:00.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002556961 2024-09-18T07:36:00.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347723+0000", 2024-09-18T07:36:00.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3884000000000002e-05 2024-09-18T07:36:00.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347762+0000", 2024-09-18T07:36:00.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8754999999999997e-05 2024-09-18T07:36:00.747 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.747 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.747 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.748 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157709 7.5 7:ab0da19b:::benchmark_data_smithi137_74180_object53293:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e164)", 2024-09-18T07:36:00.748 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.383316+0000", 2024-09-18T07:36:00.748 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 68.232579023, 2024-09-18T07:36:00.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96456343899999997, 2024-09-18T07:36:00.748 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.748 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.748 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.749 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.749 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.749 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.749 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157709 2024-09-18T07:36:00.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.749 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383316+0000", 2024-09-18T07:36:00.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383316+0000", 2024-09-18T07:36:00.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383316+0000", 2024-09-18T07:36:00.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5400000000000002e-07 2024-09-18T07:36:00.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383318+0000", 2024-09-18T07:36:00.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1409999999999999e-06 2024-09-18T07:36:00.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383333+0000", 2024-09-18T07:36:00.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4817999999999999e-05 2024-09-18T07:36:00.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383337+0000", 2024-09-18T07:36:00.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5220000000000001e-06 2024-09-18T07:36:00.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383615+0000", 2024-09-18T07:36:00.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027705399999999999 2024-09-18T07:36:00.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383623+0000", 2024-09-18T07:36:00.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2390000000000002e-06 2024-09-18T07:36:00.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341065+0000", 2024-09-18T07:36:00.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95744174100000001 2024-09-18T07:36:00.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341156+0000", 2024-09-18T07:36:00.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1150999999999995e-05 2024-09-18T07:36:00.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:36:00.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341272+0000", 2024-09-18T07:36:00.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116649 2024-09-18T07:36:00.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341925+0000", 2024-09-18T07:36:00.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00065243500000000004 2024-09-18T07:36:00.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.345159+0000", 2024-09-18T07:36:00.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032342820000000002 2024-09-18T07:36:00.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347794+0000", 2024-09-18T07:36:00.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026350280000000002 2024-09-18T07:36:00.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347844+0000", 2024-09-18T07:36:00.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9465000000000002e-05 2024-09-18T07:36:00.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347879+0000", 2024-09-18T07:36:00.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5460000000000003e-05 2024-09-18T07:36:00.757 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.757 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.757 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.757 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157721 7.5 7:a908ea8d:::benchmark_data_smithi137_74180_object53305:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e164)", 2024-09-18T07:36:00.757 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.383345+0000", 2024-09-18T07:36:00.757 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 68.232549961000004, 2024-09-18T07:36:00.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96466771200000001, 2024-09-18T07:36:00.757 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.757 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.758 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.758 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.758 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.758 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.758 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157721 2024-09-18T07:36:00.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.758 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383345+0000", 2024-09-18T07:36:00.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383345+0000", 2024-09-18T07:36:00.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383345+0000", 2024-09-18T07:36:00.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0200000000000002e-07 2024-09-18T07:36:00.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383348+0000", 2024-09-18T07:36:00.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.807e-06 2024-09-18T07:36:00.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383353+0000", 2024-09-18T07:36:00.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0719999999999999e-06 2024-09-18T07:36:00.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383357+0000", 2024-09-18T07:36:00.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3089999999999997e-06 2024-09-18T07:36:00.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383650+0000", 2024-09-18T07:36:00.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029264800000000001 2024-09-18T07:36:00.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383658+0000", 2024-09-18T07:36:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4239999999999993e-06 2024-09-18T07:36:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341469+0000", 2024-09-18T07:36:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95781031299999997 2024-09-18T07:36:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341561+0000", 2024-09-18T07:36:00.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2633999999999994e-05 2024-09-18T07:36:00.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:36:00.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341633+0000", 2024-09-18T07:36:00.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2050000000000003e-05 2024-09-18T07:36:00.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341947+0000", 2024-09-18T07:36:00.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00031341300000000002 2024-09-18T07:36:00.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.345190+0000", 2024-09-18T07:36:00.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032434059999999999 2024-09-18T07:36:00.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347913+0000", 2024-09-18T07:36:00.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027231769999999998 2024-09-18T07:36:00.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347971+0000", 2024-09-18T07:36:00.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7951000000000001e-05 2024-09-18T07:36:00.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.766 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.766 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.348012+0000", 2024-09-18T07:36:00.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0905999999999998e-05 2024-09-18T07:36:00.766 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.766 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.766 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.766 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162329 7.8 7:14bf872c:::benchmark_data_smithi137_74180_object57913:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:36:00.766 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.422282+0000", 2024-09-18T07:36:00.767 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.193612258000002, 2024-09-18T07:36:00.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.010768962, 2024-09-18T07:36:00.767 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.767 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.767 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.767 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.767 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.767 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.767 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162329 2024-09-18T07:36:00.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.768 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422282+0000", 2024-09-18T07:36:00.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422282+0000", 2024-09-18T07:36:00.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422284+0000", 2024-09-18T07:36:00.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4950000000000001e-06 2024-09-18T07:36:00.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422293+0000", 2024-09-18T07:36:00.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7029999999999995e-06 2024-09-18T07:36:00.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422302+0000", 2024-09-18T07:36:00.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5580000000000006e-06 2024-09-18T07:36:00.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422310+0000", 2024-09-18T07:36:00.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3010000000000007e-06 2024-09-18T07:36:00.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422376+0000", 2024-09-18T07:36:00.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6058999999999999e-05 2024-09-18T07:36:00.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422387+0000", 2024-09-18T07:36:00.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0560999999999999e-05 2024-09-18T07:36:00.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.429278+0000", 2024-09-18T07:36:00.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0068906500000001 2024-09-18T07:36:00.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.429407+0000", 2024-09-18T07:36:00.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012889600000000001 2024-09-18T07:36:00.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:36:00.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.429537+0000", 2024-09-18T07:36:00.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013015100000000001 2024-09-18T07:36:00.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431858+0000", 2024-09-18T07:36:00.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023215480000000001 2024-09-18T07:36:00.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431902+0000", 2024-09-18T07:36:00.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3331999999999998e-05 2024-09-18T07:36:00.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.774 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.774 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431988+0000", 2024-09-18T07:36:00.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6834000000000002e-05 2024-09-18T07:36:00.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.774 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.774 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432895+0000", 2024-09-18T07:36:00.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00090677800000000001 2024-09-18T07:36:00.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432943+0000", 2024-09-18T07:36:00.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8143000000000003e-05 2024-09-18T07:36:00.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433051+0000", 2024-09-18T07:36:00.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107953 2024-09-18T07:36:00.775 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.776 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.776 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.776 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162333 7.8 7:1673a0b1:::benchmark_data_smithi137_74180_object57917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:36:00.776 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.424085+0000", 2024-09-18T07:36:00.776 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.191809186999997, 2024-09-18T07:36:00.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0091520620000001, 2024-09-18T07:36:00.776 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.776 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.776 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.777 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.777 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.777 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.777 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162333 2024-09-18T07:36:00.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.777 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424085+0000", 2024-09-18T07:36:00.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424085+0000", 2024-09-18T07:36:00.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424087+0000", 2024-09-18T07:36:00.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4470000000000001e-06 2024-09-18T07:36:00.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424091+0000", 2024-09-18T07:36:00.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9319999999999998e-06 2024-09-18T07:36:00.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424098+0000", 2024-09-18T07:36:00.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.187e-06 2024-09-18T07:36:00.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424104+0000", 2024-09-18T07:36:00.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8730000000000004e-06 2024-09-18T07:36:00.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424145+0000", 2024-09-18T07:36:00.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1445e-05 2024-09-18T07:36:00.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424154+0000", 2024-09-18T07:36:00.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0729999999999994e-06 2024-09-18T07:36:00.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.430409+0000", 2024-09-18T07:36:00.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0062551159999999 2024-09-18T07:36:00.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.430534+0000", 2024-09-18T07:36:00.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012458400000000001 2024-09-18T07:36:00.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:36:00.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.430647+0000", 2024-09-18T07:36:00.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112537 2024-09-18T07:36:00.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431941+0000", 2024-09-18T07:36:00.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001294915 2024-09-18T07:36:00.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432032+0000", 2024-09-18T07:36:00.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0461e-05 2024-09-18T07:36:00.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433059+0000", 2024-09-18T07:36:00.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010266990000000001 2024-09-18T07:36:00.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433100+0000", 2024-09-18T07:36:00.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1770000000000002e-05 2024-09-18T07:36:00.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433146+0000", 2024-09-18T07:36:00.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5204000000000003e-05 2024-09-18T07:36:00.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433237+0000", 2024-09-18T07:36:00.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1818999999999996e-05 2024-09-18T07:36:00.784 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.784 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.784 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.785 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162348 7.8 7:1d66cfe5:::benchmark_data_smithi137_74180_object57932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:36:00.785 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.431806+0000", 2024-09-18T07:36:00.785 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.184088549999998, 2024-09-18T07:36:00.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0022084899999999, 2024-09-18T07:36:00.785 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.785 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.785 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.785 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.785 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.785 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.785 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162348 2024-09-18T07:36:00.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.786 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431806+0000", 2024-09-18T07:36:00.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431806+0000", 2024-09-18T07:36:00.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431808+0000", 2024-09-18T07:36:00.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6300000000000001e-06 2024-09-18T07:36:00.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431812+0000", 2024-09-18T07:36:00.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6140000000000002e-06 2024-09-18T07:36:00.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431823+0000", 2024-09-18T07:36:00.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0658e-05 2024-09-18T07:36:00.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431830+0000", 2024-09-18T07:36:00.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6529999999999997e-06 2024-09-18T07:36:00.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.789 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431897+0000", 2024-09-18T07:36:00.789 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7286000000000006e-05 2024-09-18T07:36:00.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.789 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431909+0000", 2024-09-18T07:36:00.789 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1739999999999999e-05 2024-09-18T07:36:00.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431207+0000", 2024-09-18T07:36:00.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99929881799999998 2024-09-18T07:36:00.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431322+0000", 2024-09-18T07:36:00.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011495800000000001 2024-09-18T07:36:00.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:36:00.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431437+0000", 2024-09-18T07:36:00.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114752 2024-09-18T07:36:00.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433064+0000", 2024-09-18T07:36:00.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016264280000000001 2024-09-18T07:36:00.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433276+0000", 2024-09-18T07:36:00.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021219300000000001 2024-09-18T07:36:00.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.792 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.792 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433318+0000", 2024-09-18T07:36:00.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1999999999999998e-05 2024-09-18T07:36:00.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.792 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.792 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433856+0000", 2024-09-18T07:36:00.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00053855899999999998 2024-09-18T07:36:00.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.792 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433907+0000", 2024-09-18T07:36:00.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0954e-05 2024-09-18T07:36:00.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.434015+0000", 2024-09-18T07:36:00.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107247 2024-09-18T07:36:00.793 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.793 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.793 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.794 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162355 7.8 7:13f232f0:::benchmark_data_smithi137_74180_object57939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:36:00.794 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.440402+0000", 2024-09-18T07:36:00.794 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.175492810000001, 2024-09-18T07:36:00.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99385697299999998, 2024-09-18T07:36:00.794 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:00.794 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:00.794 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:00.794 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:00.794 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:36:00.794 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:00.794 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162355 2024-09-18T07:36:00.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.795 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:00.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:00.795 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440402+0000", 2024-09-18T07:36:00.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:00.795 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440402+0000", 2024-09-18T07:36:00.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:00.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:00.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440403+0000", 2024-09-18T07:36:00.796 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4750000000000001e-06 2024-09-18T07:36:00.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:00.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440409+0000", 2024-09-18T07:36:00.796 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6690000000000003e-06 2024-09-18T07:36:00.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:00.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440420+0000", 2024-09-18T07:36:00.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0594000000000001e-05 2024-09-18T07:36:00.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:00.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440425+0000", 2024-09-18T07:36:00.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2460000000000003e-06 2024-09-18T07:36:00.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440471+0000", 2024-09-18T07:36:00.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6397000000000003e-05 2024-09-18T07:36:00.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:36:00.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440479+0000", 2024-09-18T07:36:00.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8220000000000007e-06 2024-09-18T07:36:00.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:00.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432099+0000", 2024-09-18T07:36:00.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99161995000000003 2024-09-18T07:36:00.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:00.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432216+0000", 2024-09-18T07:36:00.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117404 2024-09-18T07:36:00.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:36:00.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432332+0000", 2024-09-18T07:36:00.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115939 2024-09-18T07:36:00.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433814+0000", 2024-09-18T07:36:00.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00148196 2024-09-18T07:36:00.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:00.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433827+0000", 2024-09-18T07:36:00.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2583e-05 2024-09-18T07:36:00.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.434071+0000", 2024-09-18T07:36:00.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024380199999999999 2024-09-18T07:36:00.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:00.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.434123+0000", 2024-09-18T07:36:00.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2419000000000001e-05 2024-09-18T07:36:00.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:00.802 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.434167+0000", 2024-09-18T07:36:00.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3600000000000003e-05 2024-09-18T07:36:00.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:00.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:00.802 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:00.802 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.434259+0000", 2024-09-18T07:36:00.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2113000000000004e-05 2024-09-18T07:36:00.802 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.802 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.802 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.802 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:00.802 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:00.803 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:36:00.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:00.804 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:00.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:00.813 INFO:teuthology.orchestra.run.smithi137.stderr:set 4.1b pg_upmap_items mapping to [7->6,2->4,3->5] 2024-09-18T07:36:00.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:00.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:00.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:00.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:01.033 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:01.059 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:01.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:01.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:01.161 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:01.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:01.183 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:01.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:01.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:01.306 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:01.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:01.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:01.408 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:01.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:01 smithi137 ceph-mon[29408]: 5.17 scrub starts 2024-09-18T07:36:01.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:01 smithi137 ceph-mon[29408]: 5.17 scrub ok 2024-09-18T07:36:01.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:01 smithi137 ceph-mon[29408]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:36:01.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:01 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3208241251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:01.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:01 smithi137 ceph-mon[29408]: from='client.35144 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1b", "id": [7, 6, 2, 4, 3, 5]}]': finished 2024-09-18T07:36:01.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:01 smithi137 ceph-mon[29408]: osdmap e204: 8 total, 8 up, 8 in 2024-09-18T07:36:01.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:01 smithi137 ceph-mon[39175]: 5.17 scrub starts 2024-09-18T07:36:01.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:01 smithi137 ceph-mon[39175]: 5.17 scrub ok 2024-09-18T07:36:01.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:01 smithi137 ceph-mon[39175]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:36:01.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:01 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3208241251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:01.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:01 smithi137 ceph-mon[39175]: from='client.35144 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1b", "id": [7, 6, 2, 4, 3, 5]}]': finished 2024-09-18T07:36:01.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:01 smithi137 ceph-mon[39175]: osdmap e204: 8 total, 8 up, 8 in 2024-09-18T07:36:01.434 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:01.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:01.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:01.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:01 smithi203 ceph-mon[32237]: 5.17 scrub starts 2024-09-18T07:36:01.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:01 smithi203 ceph-mon[32237]: 5.17 scrub ok 2024-09-18T07:36:01.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:01 smithi203 ceph-mon[32237]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:36:01.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:01 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3208241251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:01.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:01 smithi203 ceph-mon[32237]: from='client.35144 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1b", "id": [7, 6, 2, 4, 3, 5]}]': finished 2024-09-18T07:36:01.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:01 smithi203 ceph-mon[32237]: osdmap e204: 8 total, 8 up, 8 in 2024-09-18T07:36:01.536 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:01.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:01.561 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:01.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:01.662 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:01.683 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:01.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:01.785 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:01.813 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:01.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:01.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:01.915 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:01.939 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:01.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:01.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:02.041 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:02.066 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:02.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:02.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:02.167 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:02.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:02.192 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:02.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:02.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:02 smithi137 ceph-mon[29408]: 3.10 scrub starts 2024-09-18T07:36:02.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:02 smithi137 ceph-mon[29408]: 3.10 scrub ok 2024-09-18T07:36:02.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:02 smithi137 ceph-mon[29408]: pgmap v272: 148 pgs: 1 peering, 10 unknown, 137 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:36:02.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:02 smithi137 ceph-mon[29408]: 2.2 scrub starts 2024-09-18T07:36:02.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:02 smithi137 ceph-mon[29408]: 2.2 scrub ok 2024-09-18T07:36:02.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:02 smithi137 ceph-mon[29408]: osdmap e205: 8 total, 8 up, 8 in 2024-09-18T07:36:02.290 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:02 smithi137 ceph-mon[39175]: 3.10 scrub starts 2024-09-18T07:36:02.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:02 smithi137 ceph-mon[39175]: 3.10 scrub ok 2024-09-18T07:36:02.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:02 smithi137 ceph-mon[39175]: pgmap v272: 148 pgs: 1 peering, 10 unknown, 137 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:36:02.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:02 smithi137 ceph-mon[39175]: 2.2 scrub starts 2024-09-18T07:36:02.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:02 smithi137 ceph-mon[39175]: 2.2 scrub ok 2024-09-18T07:36:02.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:02 smithi137 ceph-mon[39175]: osdmap e205: 8 total, 8 up, 8 in 2024-09-18T07:36:02.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T07:36:02.292 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:02.316 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:02.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:02.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:02.418 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:02.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:02.441 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:02.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:02.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:02 smithi203 ceph-mon[32237]: 3.10 scrub starts 2024-09-18T07:36:02.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:02 smithi203 ceph-mon[32237]: 3.10 scrub ok 2024-09-18T07:36:02.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:02 smithi203 ceph-mon[32237]: pgmap v272: 148 pgs: 1 peering, 10 unknown, 137 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:36:02.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:02 smithi203 ceph-mon[32237]: 2.2 scrub starts 2024-09-18T07:36:02.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:02 smithi203 ceph-mon[32237]: 2.2 scrub ok 2024-09-18T07:36:02.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:02 smithi203 ceph-mon[32237]: osdmap e205: 8 total, 8 up, 8 in 2024-09-18T07:36:02.542 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:02.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:02.568 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:02.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:02.669 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:02.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:02.694 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:02.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:02.795 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:02.820 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:02.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:02.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:02.922 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:02.948 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:02.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:02.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:03.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:03.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:03.075 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:03.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:03.096 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:03.132 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:03.175 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:03.201 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:03.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:03.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:03.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:03.324 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:03.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:03.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:03.416 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:03 smithi137 ceph-mon[29408]: 4.13 scrub starts 2024-09-18T07:36:03.416 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:03 smithi137 ceph-mon[29408]: 4.13 scrub ok 2024-09-18T07:36:03.416 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:03 smithi137 ceph-mon[29408]: 3.8 scrub starts 2024-09-18T07:36:03.416 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:03 smithi137 ceph-mon[29408]: 3.8 scrub ok 2024-09-18T07:36:03.416 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:03 smithi137 ceph-mon[39175]: 4.13 scrub starts 2024-09-18T07:36:03.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:03 smithi137 ceph-mon[39175]: 4.13 scrub ok 2024-09-18T07:36:03.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:03 smithi137 ceph-mon[39175]: 3.8 scrub starts 2024-09-18T07:36:03.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:03 smithi137 ceph-mon[39175]: 3.8 scrub ok 2024-09-18T07:36:03.426 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:03.452 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:03.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:03.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:03 smithi203 ceph-mon[32237]: 4.13 scrub starts 2024-09-18T07:36:03.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:03 smithi203 ceph-mon[32237]: 4.13 scrub ok 2024-09-18T07:36:03.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:03 smithi203 ceph-mon[32237]: 3.8 scrub starts 2024-09-18T07:36:03.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:03 smithi203 ceph-mon[32237]: 3.8 scrub ok 2024-09-18T07:36:03.554 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:03.577 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:03.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:03.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:03.599 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:36:03.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:03.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:03.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:03.808 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:03.834 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:03.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:03.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:03.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:03.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:03.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:03.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:04.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:04.083 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:04.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:04.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:04.117 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-18T07:36:04.185 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:04.207 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:04.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:04.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:04.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:04.330 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:04.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:04.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:04 smithi137 ceph-mon[29408]: 4.14 scrub starts 2024-09-18T07:36:04.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:04 smithi137 ceph-mon[29408]: 4.14 scrub ok 2024-09-18T07:36:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:04 smithi137 ceph-mon[29408]: pgmap v274: 148 pgs: 1 peering, 147 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:36:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:04 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3847894820' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:36:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:04 smithi137 ceph-mon[39175]: 4.14 scrub starts 2024-09-18T07:36:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:04 smithi137 ceph-mon[39175]: 4.14 scrub ok 2024-09-18T07:36:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:04 smithi137 ceph-mon[39175]: pgmap v274: 148 pgs: 1 peering, 147 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:36:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:04 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3847894820' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:36:04.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:04.453 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:04.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:04.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:04 smithi203 ceph-mon[32237]: 4.14 scrub starts 2024-09-18T07:36:04.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:04 smithi203 ceph-mon[32237]: 4.14 scrub ok 2024-09-18T07:36:04.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:04 smithi203 ceph-mon[32237]: pgmap v274: 148 pgs: 1 peering, 147 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:36:04.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:04 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3847894820' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:36:04.554 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:04.580 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:04.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:04.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:04.703 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:04.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:04.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:04.798 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:36:04.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:04.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:04.830 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:04.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:04.931 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:04.952 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:04.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:04.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:05.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:05.076 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:05.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:05.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:05.198 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:05.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:05.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:05.323 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:05.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:05.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:05 smithi137 ceph-mon[29408]: 3.0 scrub starts 2024-09-18T07:36:05.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:05 smithi137 ceph-mon[29408]: 3.0 scrub ok 2024-09-18T07:36:05.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:05 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3847894820' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:36:05.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:05 smithi137 ceph-mon[29408]: osdmap e206: 8 total, 8 up, 8 in 2024-09-18T07:36:05.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:05 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/838828141' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:05.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:05 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:36:05.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:05 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/838828141' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:36:05.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:05 smithi137 ceph-mon[29408]: osdmap e207: 8 total, 8 up, 8 in 2024-09-18T07:36:05.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:05 smithi137 ceph-mon[39175]: 3.0 scrub starts 2024-09-18T07:36:05.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:05 smithi137 ceph-mon[39175]: 3.0 scrub ok 2024-09-18T07:36:05.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:05 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3847894820' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:36:05.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:05 smithi137 ceph-mon[39175]: osdmap e206: 8 total, 8 up, 8 in 2024-09-18T07:36:05.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:05 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/838828141' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:05.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:05 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:36:05.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:05 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/838828141' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:36:05.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:05 smithi137 ceph-mon[39175]: osdmap e207: 8 total, 8 up, 8 in 2024-09-18T07:36:05.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:05.445 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:05.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:05.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:05.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:05 smithi203 ceph-mon[32237]: 3.0 scrub starts 2024-09-18T07:36:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:05 smithi203 ceph-mon[32237]: 3.0 scrub ok 2024-09-18T07:36:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:05 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3847894820' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:36:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:05 smithi203 ceph-mon[32237]: osdmap e206: 8 total, 8 up, 8 in 2024-09-18T07:36:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:05 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/838828141' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:05 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:36:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:05 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/838828141' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:36:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:05 smithi203 ceph-mon[32237]: osdmap e207: 8 total, 8 up, 8 in 2024-09-18T07:36:05.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:05.568 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:05.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:05.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:05.599 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:05.631 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:05.671 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:05.692 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:05.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:05.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:05.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:05.815 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:05.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:05.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:05.832 DEBUG:teuthology.orchestra.run.smithi137:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-18T07:36:05.859 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-18T07:36:05.859 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-18T07:36:05.916 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:hints = 1 2024-09-18T07:36:05.916 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-18T07:36:05.916 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Object prefix: benchmark_data_smithi137_101942 2024-09-18T07:36:05.916 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:36:05.916 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 0 0 0 0 0 0 - 0 2024-09-18T07:36:05.916 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:05.942 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:05.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:06.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:06.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:06.082 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:06.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:06.183 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:06.216 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:06.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:06.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:06.318 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:06.347 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:06.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:06.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:06.417 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:06 smithi137 ceph-mon[29408]: 5.0 scrub starts 2024-09-18T07:36:06.417 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:06 smithi137 ceph-mon[29408]: 5.0 scrub ok 2024-09-18T07:36:06.417 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:06 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:36:06.418 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:06 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:36:06.418 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:06 smithi137 ceph-mon[29408]: 3.11 scrub starts 2024-09-18T07:36:06.418 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:06 smithi137 ceph-mon[29408]: 3.11 scrub ok 2024-09-18T07:36:06.418 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:06 smithi137 ceph-mon[29408]: pgmap v277: 148 pgs: 1 peering, 147 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:36:06.418 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:06 smithi137 ceph-mon[39175]: 5.0 scrub starts 2024-09-18T07:36:06.418 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:06 smithi137 ceph-mon[39175]: 5.0 scrub ok 2024-09-18T07:36:06.418 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:06 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:36:06.418 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:06 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:36:06.418 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:06 smithi137 ceph-mon[39175]: 3.11 scrub starts 2024-09-18T07:36:06.418 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:06 smithi137 ceph-mon[39175]: 3.11 scrub ok 2024-09-18T07:36:06.418 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:06 smithi137 ceph-mon[39175]: pgmap v277: 148 pgs: 1 peering, 147 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:36:06.419 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:36:06.419 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:36:06.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.420 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.35171.0:1622 8.f e208/201 ondisk, result = 0)", 2024-09-18T07:36:06.420 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:06.414100+0000", 2024-09-18T07:36:06.420 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00023142700000000001, 2024-09-18T07:36:06.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023185799999999999, 2024-09-18T07:36:06.420 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:06.420 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:06.420 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:36:06.420 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:06.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.421 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:06.421 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414100+0000", 2024-09-18T07:36:06.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:06.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.421 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:06.421 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414100+0000", 2024-09-18T07:36:06.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:06.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.421 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:06.421 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414101+0000", 2024-09-18T07:36:06.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7899999999999997e-07 2024-09-18T07:36:06.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:06.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414104+0000", 2024-09-18T07:36:06.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.219e-06 2024-09-18T07:36:06.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:06.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414111+0000", 2024-09-18T07:36:06.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0090000000000001e-06 2024-09-18T07:36:06.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:06.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414116+0000", 2024-09-18T07:36:06.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0100000000000003e-06 2024-09-18T07:36:06.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:06.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414313+0000", 2024-09-18T07:36:06.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019741600000000001 2024-09-18T07:36:06.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:06.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414319+0000", 2024-09-18T07:36:06.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0100000000000001e-06 2024-09-18T07:36:06.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:36:06.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414332+0000", 2024-09-18T07:36:06.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2415000000000001e-05 2024-09-18T07:36:06.424 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:06.424 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:06.425 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:06.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.425 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:1633 8.f 8:fdfab462:::benchmark_data_smithi137_101942_object1632:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-09-18T07:36:06.425 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:06.414144+0000", 2024-09-18T07:36:06.425 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.000186987, 2024-09-18T07:36:06.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027903299999999998, 2024-09-18T07:36:06.425 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:06.426 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:06.426 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:36:06.426 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:06.426 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:36:06.426 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:36:06.426 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1633 2024-09-18T07:36:06.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.426 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:06.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:06.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414144+0000", 2024-09-18T07:36:06.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:06.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:06.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414144+0000", 2024-09-18T07:36:06.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:06.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:06.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414146+0000", 2024-09-18T07:36:06.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0140000000000001e-06 2024-09-18T07:36:06.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:06.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414171+0000", 2024-09-18T07:36:06.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4771999999999999e-05 2024-09-18T07:36:06.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:06.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414180+0000", 2024-09-18T07:36:06.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0799999999999995e-06 2024-09-18T07:36:06.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:06.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414188+0000", 2024-09-18T07:36:06.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9030000000000004e-06 2024-09-18T07:36:06.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:06.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414362+0000", 2024-09-18T07:36:06.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017387899999999999 2024-09-18T07:36:06.431 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:06.431 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:06.431 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:06.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.432 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:1634 8.f 8.184d10cf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-09-18T07:36:06.432 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:06.414199+0000", 2024-09-18T07:36:06.432 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00013246699999999999, 2024-09-18T07:36:06.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027904500000000001, 2024-09-18T07:36:06.432 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:06.432 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:06.432 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:36:06.433 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:06.433 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:36:06.433 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:36:06.433 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1634 2024-09-18T07:36:06.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.433 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:06.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:06.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414199+0000", 2024-09-18T07:36:06.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:06.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:06.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414199+0000", 2024-09-18T07:36:06.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:06.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:06.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414200+0000", 2024-09-18T07:36:06.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4600000000000004e-07 2024-09-18T07:36:06.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:06.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414221+0000", 2024-09-18T07:36:06.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1027999999999999e-05 2024-09-18T07:36:06.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:06.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414228+0000", 2024-09-18T07:36:06.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9689999999999997e-06 2024-09-18T07:36:06.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:06.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414233+0000", 2024-09-18T07:36:06.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.682e-06 2024-09-18T07:36:06.437 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:06.437 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:06.437 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:06.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.438 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35171.0:1630 8.c e208/201 8:394586b0:::benchmark_data_smithi137_101942_object1629:head v 208'97, mlcod=208'95)", 2024-09-18T07:36:06.438 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:06.414308+0000", 2024-09-18T07:36:06.438 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.3039000000000001e-05, 2024-09-18T07:36:06.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000218092, 2024-09-18T07:36:06.438 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:06.438 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:06.438 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:36:06.439 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:06.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:06.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414308+0000", 2024-09-18T07:36:06.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:06.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:06.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414308+0000", 2024-09-18T07:36:06.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:06.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:06.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414310+0000", 2024-09-18T07:36:06.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6169999999999999e-06 2024-09-18T07:36:06.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:06.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414334+0000", 2024-09-18T07:36:06.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3563000000000001e-05 2024-09-18T07:36:06.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:06.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414345+0000", 2024-09-18T07:36:06.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0964e-05 2024-09-18T07:36:06.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:06.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414352+0000", 2024-09-18T07:36:06.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0600000000000002e-06 2024-09-18T07:36:06.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:06.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414411+0000", 2024-09-18T07:36:06.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9657999999999997e-05 2024-09-18T07:36:06.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:06.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414430+0000", 2024-09-18T07:36:06.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8804000000000002e-05 2024-09-18T07:36:06.442 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:06.443 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:06.443 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:06.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.443 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:1618 8.f 8:fcd07ab3:::benchmark_data_smithi137_101942_object1617:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-09-18T07:36:06.443 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:06.409127+0000", 2024-09-18T07:36:06.443 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0052039640000000002, 2024-09-18T07:36:06.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054554360000000001, 2024-09-18T07:36:06.443 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:06.443 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:06.443 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:36:06.443 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:06.444 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:36:06.444 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:36:06.444 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1618 2024-09-18T07:36:06.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.444 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:06.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:06.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.409127+0000", 2024-09-18T07:36:06.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:06.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:06.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.409127+0000", 2024-09-18T07:36:06.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:06.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:06.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.409128+0000", 2024-09-18T07:36:06.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3799999999999996e-07 2024-09-18T07:36:06.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:06.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.409148+0000", 2024-09-18T07:36:06.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0241000000000002e-05 2024-09-18T07:36:06.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:06.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.409154+0000", 2024-09-18T07:36:06.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9540000000000001e-06 2024-09-18T07:36:06.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:06.451 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.409528+0000", 2024-09-18T07:36:06.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00037358699999999999 2024-09-18T07:36:06.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.451 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:06.451 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.410410+0000", 2024-09-18T07:36:06.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00088186299999999998 2024-09-18T07:36:06.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:06.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.410540+0000", 2024-09-18T07:36:06.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000129682 2024-09-18T07:36:06.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-18T07:36:06.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.410685+0000", 2024-09-18T07:36:06.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014542600000000001 2024-09-18T07:36:06.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:06.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.413547+0000", 2024-09-18T07:36:06.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028620820000000002 2024-09-18T07:36:06.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:06.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414286+0000", 2024-09-18T07:36:06.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00073924899999999998 2024-09-18T07:36:06.454 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:06.454 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:06.455 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:06.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.455 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:1622 8.f 8:fa03f268:::benchmark_data_smithi137_101942_object1621:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-09-18T07:36:06.455 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:06.409545+0000", 2024-09-18T07:36:06.455 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0047860250000000002, 2024-09-18T07:36:06.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005116415, 2024-09-18T07:36:06.456 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:06.456 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:06.456 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:36:06.456 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:06.456 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:36:06.456 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:36:06.457 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1622 2024-09-18T07:36:06.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.457 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:06.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:06.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.409545+0000", 2024-09-18T07:36:06.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:06.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.458 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:06.458 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.409545+0000", 2024-09-18T07:36:06.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:06.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.458 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:06.458 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.409547+0000", 2024-09-18T07:36:06.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1200000000000001e-06 2024-09-18T07:36:06.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.458 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:06.459 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.409632+0000", 2024-09-18T07:36:06.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4982999999999996e-05 2024-09-18T07:36:06.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:06.459 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.409640+0000", 2024-09-18T07:36:06.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0430000000000008e-06 2024-09-18T07:36:06.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:06.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.409646+0000", 2024-09-18T07:36:06.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2140000000000001e-06 2024-09-18T07:36:06.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:06.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.411320+0000", 2024-09-18T07:36:06.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00167384 2024-09-18T07:36:06.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:06.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.411873+0000", 2024-09-18T07:36:06.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00055312299999999998 2024-09-18T07:36:06.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-18T07:36:06.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.412170+0000", 2024-09-18T07:36:06.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029746900000000002 2024-09-18T07:36:06.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:06.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.413552+0000", 2024-09-18T07:36:06.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013813860000000001 2024-09-18T07:36:06.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:06.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414324+0000", 2024-09-18T07:36:06.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00077248200000000001 2024-09-18T07:36:06.462 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:06.462 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:06.462 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:06.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.462 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:1623 8.b 8:d34a5eb2:::benchmark_data_smithi137_101942_object1622:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-09-18T07:36:06.462 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:06.410185+0000", 2024-09-18T07:36:06.462 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0041466740000000004, 2024-09-18T07:36:06.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045533470000000001, 2024-09-18T07:36:06.463 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:06.463 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:06.463 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:36:06.463 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:06.463 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:36:06.463 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:36:06.463 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1623 2024-09-18T07:36:06.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.463 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:06.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:06.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.410185+0000", 2024-09-18T07:36:06.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:06.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:06.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.410185+0000", 2024-09-18T07:36:06.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:06.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:06.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.410186+0000", 2024-09-18T07:36:06.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.187e-06 2024-09-18T07:36:06.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:06.465 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.410516+0000", 2024-09-18T07:36:06.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032984000000000003 2024-09-18T07:36:06.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:06.465 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.410526+0000", 2024-09-18T07:36:06.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0036e-05 2024-09-18T07:36:06.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:06.465 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.410533+0000", 2024-09-18T07:36:06.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3200000000000002e-06 2024-09-18T07:36:06.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:06.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.410581+0000", 2024-09-18T07:36:06.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8186999999999998e-05 2024-09-18T07:36:06.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:06.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.410700+0000", 2024-09-18T07:36:06.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118168 2024-09-18T07:36:06.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:36:06.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.411159+0000", 2024-09-18T07:36:06.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00045999499999999999 2024-09-18T07:36:06.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:06.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.412243+0000", 2024-09-18T07:36:06.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001083313 2024-09-18T07:36:06.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:06.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:06.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:06.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:06.414217+0000", 2024-09-18T07:36:06.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019740249999999999 2024-09-18T07:36:06.467 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:06.467 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:06.467 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:06.468 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:06.468 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:36:06.468 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 7 2024-09-18T07:36:06.468 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:36:06.468 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:06.470 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:06 smithi203 ceph-mon[32237]: 5.0 scrub starts 2024-09-18T07:36:06.470 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:06 smithi203 ceph-mon[32237]: 5.0 scrub ok 2024-09-18T07:36:06.470 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:06 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:36:06.470 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:06 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:36:06.470 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:06 smithi203 ceph-mon[32237]: 3.11 scrub starts 2024-09-18T07:36:06.470 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:06 smithi203 ceph-mon[32237]: 3.11 scrub ok 2024-09-18T07:36:06.470 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:06 smithi203 ceph-mon[32237]: pgmap v277: 148 pgs: 1 peering, 147 active+clean; 583 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:36:06.509 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:06.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:06.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:06.611 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:06.643 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:06.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:06.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:06.745 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:06.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:06.790 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:06.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:06.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:06.916 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 1 15 3344 3329 208.012 208.062 0.00688136 0.0047972 2024-09-18T07:36:06.923 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:06.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:06.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:07.025 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:07.060 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:07.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:07.162 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:07.193 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:07.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:07.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:07.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:07.334 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:07.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:07.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:07.338 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:36:07.338 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:36:07.375 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:36:07.376 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:36:07.376 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:36:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:36:07.376 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[29408]: 5.3 scrub starts 2024-09-18T07:36:07.377 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[29408]: 5.3 scrub ok 2024-09-18T07:36:07.377 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[29408]: 4.1b scrub starts 2024-09-18T07:36:07.377 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[29408]: 4.1b scrub ok 2024-09-18T07:36:07.377 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[29408]: 4.7 scrub starts 2024-09-18T07:36:07.377 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[29408]: 4.7 scrub ok 2024-09-18T07:36:07.377 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[29408]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:36:07.377 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:36:07.377 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[29408]: osdmap e208: 8 total, 8 up, 8 in 2024-09-18T07:36:07.377 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[39175]: 5.3 scrub starts 2024-09-18T07:36:07.377 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[39175]: 5.3 scrub ok 2024-09-18T07:36:07.377 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[39175]: 4.1b scrub starts 2024-09-18T07:36:07.378 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[39175]: 4.1b scrub ok 2024-09-18T07:36:07.378 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[39175]: 4.7 scrub starts 2024-09-18T07:36:07.378 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[39175]: 4.7 scrub ok 2024-09-18T07:36:07.378 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[39175]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:36:07.378 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:36:07.378 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:07 smithi137 ceph-mon[39175]: osdmap e208: 8 total, 8 up, 8 in 2024-09-18T07:36:07.411 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:36:07.411 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:36:07.436 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:07.446 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:36:07.447 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:36:07.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:36:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:36:07.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:07 smithi203 ceph-mon[32237]: 5.3 scrub starts 2024-09-18T07:36:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:07 smithi203 ceph-mon[32237]: 5.3 scrub ok 2024-09-18T07:36:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:07 smithi203 ceph-mon[32237]: 4.1b scrub starts 2024-09-18T07:36:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:07 smithi203 ceph-mon[32237]: 4.1b scrub ok 2024-09-18T07:36:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:07 smithi203 ceph-mon[32237]: 4.7 scrub starts 2024-09-18T07:36:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:07 smithi203 ceph-mon[32237]: 4.7 scrub ok 2024-09-18T07:36:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:07 smithi203 ceph-mon[32237]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:36:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:07 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:36:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:07 smithi203 ceph-mon[32237]: osdmap e208: 8 total, 8 up, 8 in 2024-09-18T07:36:07.466 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:07.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:07.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:07.478 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:36:07.479 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:36:07.513 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:36:07.513 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:36:07.530 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:36:07.550 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:36:07.550 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:36:07.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:07.585 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:36:07.585 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:36:07.615 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:07.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:07.716 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:07.765 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:07.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:07.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:07.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T07:36:07.867 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:07.898 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:07.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:07.916 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 2 15 6598 6583 205.686 203.375 0.00372953 0.00484907 2024-09-18T07:36:08.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:08.039 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:08.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:08.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:08.110 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:36:08.110 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:36:08.110 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-18T07:36:08.110 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.000000 2024-09-18T07:36:08.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 0 2024-09-18T07:36:08.141 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:08.197 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:08.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:08.298 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:08.374 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:08.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:08.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:08.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[39175]: 3.1b scrub starts 2024-09-18T07:36:08.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[39175]: 3.1b scrub ok 2024-09-18T07:36:08.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[39175]: 3.3 scrub starts 2024-09-18T07:36:08.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[39175]: 3.3 scrub ok 2024-09-18T07:36:08.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:08.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2538372028' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:08.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[39175]: pgmap v279: 148 pgs: 148 active+clean; 144 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 898 B/s rd, 25 MiB/s wr, 403 op/s; 19 B/s, 0 objects/s recovering 2024-09-18T07:36:08.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[39175]: 6.d scrub starts 2024-09-18T07:36:08.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[39175]: 6.d scrub ok 2024-09-18T07:36:08.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[29408]: 3.1b scrub starts 2024-09-18T07:36:08.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[29408]: 3.1b scrub ok 2024-09-18T07:36:08.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[29408]: 3.3 scrub starts 2024-09-18T07:36:08.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[29408]: 3.3 scrub ok 2024-09-18T07:36:08.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:08.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2538372028' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:08.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[29408]: pgmap v279: 148 pgs: 148 active+clean; 144 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 898 B/s rd, 25 MiB/s wr, 403 op/s; 19 B/s, 0 objects/s recovering 2024-09-18T07:36:08.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[29408]: 6.d scrub starts 2024-09-18T07:36:08.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:08 smithi137 ceph-mon[29408]: 6.d scrub ok 2024-09-18T07:36:08.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:08 smithi203 ceph-mon[32237]: 3.1b scrub starts 2024-09-18T07:36:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:08 smithi203 ceph-mon[32237]: 3.1b scrub ok 2024-09-18T07:36:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:08 smithi203 ceph-mon[32237]: 3.3 scrub starts 2024-09-18T07:36:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:08 smithi203 ceph-mon[32237]: 3.3 scrub ok 2024-09-18T07:36:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:08 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:08 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2538372028' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:08 smithi203 ceph-mon[32237]: pgmap v279: 148 pgs: 148 active+clean; 144 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 898 B/s rd, 25 MiB/s wr, 403 op/s; 19 B/s, 0 objects/s recovering 2024-09-18T07:36:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:08 smithi203 ceph-mon[32237]: 6.d scrub starts 2024-09-18T07:36:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:08 smithi203 ceph-mon[32237]: 6.d scrub ok 2024-09-18T07:36:08.477 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:08.508 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:08.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:08.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:08.610 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:08.648 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:08.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:08.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:08.750 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:08.778 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:08.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:08.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:08.871 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:08.880 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:08.884 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:08.908 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:08.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:08.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:08.916 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 3 16 9553 9537 198.661 184.625 0.0100698 0.00501827 2024-09-18T07:36:08.940 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:09.010 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:09.046 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:09.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:09.147 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:09.179 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:09.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:09.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:09.281 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:09.311 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:09.312 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:09 smithi137 ceph-mon[39175]: 6.9 scrub starts 2024-09-18T07:36:09.312 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:09 smithi137 ceph-mon[39175]: 6.9 scrub ok 2024-09-18T07:36:09.312 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:09 smithi137 ceph-mon[39175]: 5.14 scrub starts 2024-09-18T07:36:09.312 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:09 smithi137 ceph-mon[39175]: 5.14 scrub ok 2024-09-18T07:36:09.312 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:09 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:36:09.312 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:09 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:36:09.312 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:09 smithi137 ceph-mon[39175]: 3.13 scrub starts 2024-09-18T07:36:09.312 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:09 smithi137 ceph-mon[39175]: 3.13 scrub ok 2024-09-18T07:36:09.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:09.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:09.316 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:09 smithi137 ceph-mon[29408]: 6.9 scrub starts 2024-09-18T07:36:09.316 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:09 smithi137 ceph-mon[29408]: 6.9 scrub ok 2024-09-18T07:36:09.316 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:09 smithi137 ceph-mon[29408]: 5.14 scrub starts 2024-09-18T07:36:09.316 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:09 smithi137 ceph-mon[29408]: 5.14 scrub ok 2024-09-18T07:36:09.316 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:09 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:36:09.317 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:09 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:36:09.317 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:09 smithi137 ceph-mon[29408]: 3.13 scrub starts 2024-09-18T07:36:09.317 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:09 smithi137 ceph-mon[29408]: 3.13 scrub ok 2024-09-18T07:36:09.414 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:09.444 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:09.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:09.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:09.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:09 smithi203 ceph-mon[32237]: 6.9 scrub starts 2024-09-18T07:36:09.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:09 smithi203 ceph-mon[32237]: 6.9 scrub ok 2024-09-18T07:36:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:09 smithi203 ceph-mon[32237]: 5.14 scrub starts 2024-09-18T07:36:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:09 smithi203 ceph-mon[32237]: 5.14 scrub ok 2024-09-18T07:36:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:09 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:36:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:09 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:36:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:09 smithi203 ceph-mon[32237]: 3.13 scrub starts 2024-09-18T07:36:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:09 smithi203 ceph-mon[32237]: 3.13 scrub ok 2024-09-18T07:36:09.546 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:09.575 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:09.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:09.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:09.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:09.714 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:09.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:09.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:09.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:09.847 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:09.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:09.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:09.917 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 4 15 12433 12418 193.968 180.062 0.002207 0.00514572 2024-09-18T07:36:09.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:09.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:09.993 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:09.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:10.094 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:10.131 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:10.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:10.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:10.178 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:36:10.179 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:36:10.179 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:10 smithi137 ceph-mon[29408]: 4.17 scrub starts 2024-09-18T07:36:10.179 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:10 smithi137 ceph-mon[29408]: 4.17 scrub ok 2024-09-18T07:36:10.179 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:10 smithi137 ceph-mon[29408]: osdmap e209: 8 total, 8 up, 8 in 2024-09-18T07:36:10.179 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:10 smithi137 ceph-mon[29408]: 3.f scrub starts 2024-09-18T07:36:10.179 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:10 smithi137 ceph-mon[29408]: 3.f scrub ok 2024-09-18T07:36:10.179 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:10 smithi137 ceph-mon[29408]: pgmap v281: 148 pgs: 148 active+clean; 144 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 950 B/s rd, 27 MiB/s wr, 427 op/s; 20 B/s, 0 objects/s recovering 2024-09-18T07:36:10.180 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:10 smithi137 ceph-mon[39175]: 4.17 scrub starts 2024-09-18T07:36:10.180 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:10 smithi137 ceph-mon[39175]: 4.17 scrub ok 2024-09-18T07:36:10.180 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:10 smithi137 ceph-mon[39175]: osdmap e209: 8 total, 8 up, 8 in 2024-09-18T07:36:10.180 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:10 smithi137 ceph-mon[39175]: 3.f scrub starts 2024-09-18T07:36:10.180 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:10 smithi137 ceph-mon[39175]: 3.f scrub ok 2024-09-18T07:36:10.180 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:10 smithi137 ceph-mon[39175]: pgmap v281: 148 pgs: 148 active+clean; 144 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 950 B/s rd, 27 MiB/s wr, 427 op/s; 20 B/s, 0 objects/s recovering 2024-09-18T07:36:10.234 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:10.272 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:10.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:10.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:10.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:10.413 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:10.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:10.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:10.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:10 smithi203 ceph-mon[32237]: 4.17 scrub starts 2024-09-18T07:36:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:10 smithi203 ceph-mon[32237]: 4.17 scrub ok 2024-09-18T07:36:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:10 smithi203 ceph-mon[32237]: osdmap e209: 8 total, 8 up, 8 in 2024-09-18T07:36:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:10 smithi203 ceph-mon[32237]: 3.f scrub starts 2024-09-18T07:36:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:10 smithi203 ceph-mon[32237]: 3.f scrub ok 2024-09-18T07:36:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:10 smithi203 ceph-mon[32237]: pgmap v281: 148 pgs: 148 active+clean; 144 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 950 B/s rd, 27 MiB/s wr, 427 op/s; 20 B/s, 0 objects/s recovering 2024-09-18T07:36:10.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:10.552 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:10.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:10.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:10.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:10.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:10.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:10.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:10.780 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:36:10.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:10.866 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:10.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:10.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:10.917 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 5 16 15164 15148 189.297 170.625 0.00580526 0.00527337 2024-09-18T07:36:10.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:11.008 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:11.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:11.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:11.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:11.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:11.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:11.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:11 smithi137 ceph-mon[29408]: 4.3 scrub starts 2024-09-18T07:36:11.265 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:11.295 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:11.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:11.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:11.397 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:11.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:11 smithi137 ceph-mon[29408]: 4.3 scrub ok 2024-09-18T07:36:11.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:11 smithi137 ceph-mon[29408]: 5.1d scrub starts 2024-09-18T07:36:11.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:11 smithi137 ceph-mon[29408]: 5.1d scrub ok 2024-09-18T07:36:11.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:11 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:11.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:11 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:36:11.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:11 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4102635566' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-18T07:36:11.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:11 smithi137 ceph-mon[29408]: from='client.25219 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-18T07:36:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:11 smithi137 ceph-mon[39175]: 4.3 scrub starts 2024-09-18T07:36:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:11 smithi137 ceph-mon[39175]: 4.3 scrub ok 2024-09-18T07:36:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:11 smithi137 ceph-mon[39175]: 5.1d scrub starts 2024-09-18T07:36:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:11 smithi137 ceph-mon[39175]: 5.1d scrub ok 2024-09-18T07:36:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:11 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:11 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:36:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:11 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4102635566' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-18T07:36:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:11 smithi137 ceph-mon[39175]: from='client.25219 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-18T07:36:11.426 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:11.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:11.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:11 smithi203 ceph-mon[32237]: 4.3 scrub starts 2024-09-18T07:36:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:11 smithi203 ceph-mon[32237]: 4.3 scrub ok 2024-09-18T07:36:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:11 smithi203 ceph-mon[32237]: 5.1d scrub starts 2024-09-18T07:36:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:11 smithi203 ceph-mon[32237]: 5.1d scrub ok 2024-09-18T07:36:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:11 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:11 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:36:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:11 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4102635566' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-18T07:36:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:11 smithi203 ceph-mon[32237]: from='client.25219 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-18T07:36:11.528 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:11.564 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:11.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:11.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:11.666 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:11.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:11.702 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:11.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:11.804 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:11.835 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:11.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:11.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:11.917 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 6 16 17632 17616 183.455 154.25 0.00489391 0.0054441 2024-09-18T07:36:11.936 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:11.965 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:11.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:11.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:12.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:12.100 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:12.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:12.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:12.112 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.1 primary-affinity to 0 (802) 2024-09-18T07:36:12.115 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:36:12.202 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:12.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:12.224 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:12.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:12.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:12.347 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:12.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:12.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[29408]: 6.18 scrub starts 2024-09-18T07:36:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[29408]: 6.18 scrub ok 2024-09-18T07:36:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/782815555' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:36:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[29408]: from='client.25225 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:36:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[29408]: pgmap v282: 148 pgs: 148 active+clean; 487 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 16 B/s, 0 objects/s recovering 2024-09-18T07:36:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[29408]: from='client.25219 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2024-09-18T07:36:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[29408]: from='client.25225 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:36:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[29408]: osdmap e210: 8 total, 8 up, 8 in 2024-09-18T07:36:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[39175]: 6.18 scrub starts 2024-09-18T07:36:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[39175]: 6.18 scrub ok 2024-09-18T07:36:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/782815555' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:36:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[39175]: from='client.25225 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:36:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[39175]: pgmap v282: 148 pgs: 148 active+clean; 487 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 16 B/s, 0 objects/s recovering 2024-09-18T07:36:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:12.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[39175]: from='client.25219 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2024-09-18T07:36:12.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[39175]: from='client.25225 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:36:12.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:12 smithi137 ceph-mon[39175]: osdmap e210: 8 total, 8 up, 8 in 2024-09-18T07:36:12.449 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:12.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:12 smithi203 ceph-mon[32237]: 6.18 scrub starts 2024-09-18T07:36:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:12 smithi203 ceph-mon[32237]: 6.18 scrub ok 2024-09-18T07:36:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:12 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/782815555' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:36:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:12 smithi203 ceph-mon[32237]: from='client.25225 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:36:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:12 smithi203 ceph-mon[32237]: pgmap v282: 148 pgs: 148 active+clean; 487 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 16 B/s, 0 objects/s recovering 2024-09-18T07:36:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:12 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:12 smithi203 ceph-mon[32237]: from='client.25219 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2024-09-18T07:36:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:12 smithi203 ceph-mon[32237]: from='client.25225 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:36:12.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:12 smithi203 ceph-mon[32237]: osdmap e210: 8 total, 8 up, 8 in 2024-09-18T07:36:12.471 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:12.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:12.573 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:12.596 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:12.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:12.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:12.698 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:12.722 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:12.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:12.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:12.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T07:36:12.824 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:12.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:12.850 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:12.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:12.918 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 7 16 18259 18243 162.847 39.1875 0.00169266 0.00545298 2024-09-18T07:36:12.951 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:12.975 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:12.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:12.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:13.077 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:13.099 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:13.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:13.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:13.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:13.241 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:13.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:13.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:13.343 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:13.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:13.379 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:13.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:13.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:13 smithi203 ceph-mon[32237]: 6.17 scrub starts 2024-09-18T07:36:13.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:13 smithi203 ceph-mon[32237]: 6.17 scrub ok 2024-09-18T07:36:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:13 smithi203 ceph-mon[32237]: osdmap e211: 8 total, 8 up, 8 in 2024-09-18T07:36:13.480 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:13.512 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:13.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:13.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:13.614 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:13.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:13.650 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:13.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:13.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:13 smithi137 ceph-mon[29408]: 6.17 scrub starts 2024-09-18T07:36:13.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:13 smithi137 ceph-mon[29408]: 6.17 scrub ok 2024-09-18T07:36:13.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:13 smithi137 ceph-mon[29408]: osdmap e211: 8 total, 8 up, 8 in 2024-09-18T07:36:13.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:13 smithi137 ceph-mon[39175]: 6.17 scrub starts 2024-09-18T07:36:13.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:13 smithi137 ceph-mon[39175]: 6.17 scrub ok 2024-09-18T07:36:13.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:13 smithi137 ceph-mon[39175]: osdmap e211: 8 total, 8 up, 8 in 2024-09-18T07:36:13.751 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:13.776 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:13.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:13.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:13.878 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:13.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:13.916 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:13.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:13.917 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 8 16 20296 20280 158.404 127.312 0.00642555 0.00630734 2024-09-18T07:36:14.018 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:14.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:14.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:14.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:14.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:14.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:14.204 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:14.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:14.305 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:14.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:14.342 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:14.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:14.407 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:14.443 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:14.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:14 smithi203 ceph-mon[32237]: pgmap v285: 148 pgs: 148 active+clean; 1.0 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-09-18T07:36:14.471 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:14.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:14.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:14.541 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:14 smithi137 ceph-mon[29408]: pgmap v285: 148 pgs: 148 active+clean; 1.0 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-09-18T07:36:14.542 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:14 smithi137 ceph-mon[39175]: pgmap v285: 148 pgs: 148 active+clean; 1.0 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-09-18T07:36:14.573 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:14.615 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:14.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:14.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:14.717 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:14.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:14.752 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:14.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:14.853 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:14.884 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:14.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:14.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:14.918 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 9 15 22829 22814 158.4 158.375 0.00680742 0.0063074 2024-09-18T07:36:14.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:15.007 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:15.034 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:15.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:15.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:15.046 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:36:15.046 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:36:15.136 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:15.168 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:15.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:15.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:15.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:15.311 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:15.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:15.413 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:15.444 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:15.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:15.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:15.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:15.583 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:15.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:15.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:15.686 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:15.719 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:15.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:15.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:15.822 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:15.852 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:15.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:15.918 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 10 16 25279 25263 157.865 153.062 0.00799037 0.00632906 2024-09-18T07:36:15.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:15.987 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:15.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:15.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:15.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:36:16.089 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:16.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:16.123 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:16.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:16.224 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:16.257 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:16.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:16.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:16.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:16.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:16.403 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:16.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:16.504 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:16.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:16.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:16.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:16.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:16.679 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:16.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:16.781 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:16.807 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:16.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:16.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:16.918 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 11 16 27323 27307 155.126 127.75 0.0096532 0.00644142 2024-09-18T07:36:16.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:16.946 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:16.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:17.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:17.068 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:17.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:17.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:17.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:16 smithi137 ceph-mon[29408]: pgmap v286: 148 pgs: 148 active+clean; 1.0 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 804 B/s rd, 142 MiB/s wr, 2.28k op/s 2024-09-18T07:36:17.101 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:16 smithi137 ceph-mon[39175]: pgmap v286: 148 pgs: 148 active+clean; 1.0 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 804 B/s rd, 142 MiB/s wr, 2.28k op/s 2024-09-18T07:36:17.126 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:16 smithi203 ceph-mon[32237]: pgmap v286: 148 pgs: 148 active+clean; 1.0 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 804 B/s rd, 142 MiB/s wr, 2.28k op/s 2024-09-18T07:36:17.171 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:17.209 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:17.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:17.310 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:17.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:17.335 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:17.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:17.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:36:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:36:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:36:17.436 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:17.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:17.459 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:17.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:17.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:36:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:36:17.560 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:17.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:17.584 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:17.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:17.686 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:17.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:17.725 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:17.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:17.826 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:17.865 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:17.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:17.918 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 12 16 28742 28726 149.59 88.6875 0.0102558 0.00667859 2024-09-18T07:36:17.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:18.004 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:18.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:18.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:18.107 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:18.137 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:18.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:18.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:18.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:18.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:18.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:18.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:18.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:18.402 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:18.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:18.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:18.504 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:18.521 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:36:18.521 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:36:18.537 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:18.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:18.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:18.554 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:36:18.555 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:36:18.589 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:36:18.589 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:36:18.622 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:36:18.623 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:36:18.638 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:18.654 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:36:18.654 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:36:18.679 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:36:18.679 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:36:18.681 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:18.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:18.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:18.699 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T07:36:18.717 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:36:18.717 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:36:18.775 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:36:18.776 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:36:18.783 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:18.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:18.815 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:18.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:18.916 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:18.918 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 13 16 30768 30752 147.821 126.625 0.00783253 0.00676042 2024-09-18T07:36:18.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:18 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:18.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:18 smithi137 ceph-mon[29408]: pgmap v287: 148 pgs: 148 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 767 B/s rd, 191 MiB/s wr, 3.06k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-09-18T07:36:18.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:18 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3124805641' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:18.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:18 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:18.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:18 smithi137 ceph-mon[39175]: pgmap v287: 148 pgs: 148 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 767 B/s rd, 191 MiB/s wr, 3.06k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-09-18T07:36:18.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:18 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3124805641' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:18.956 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:18.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:18.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:19.058 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:19.081 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:36:19.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:19.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:19.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:19.198 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:19.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:18 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:19.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:18 smithi203 ceph-mon[32237]: pgmap v287: 148 pgs: 148 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 767 B/s rd, 191 MiB/s wr, 3.06k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-09-18T07:36:19.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:18 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3124805641' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:19.229 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:19.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:19.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:19.332 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:19.364 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:19.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:19.422 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:36:19.422 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:36:19.422 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-18T07:36:19.422 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T07:36:19.422 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T07:36:19.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-18T07:36:19.465 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:19.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:19.488 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:19.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:19.589 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:19.618 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:19.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:19.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:19.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:19.748 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:19.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:19.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:19.850 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:19.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:19.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:19.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:19.918 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 14 16 32758 32742 146.146 124.375 0.00276698 0.00683646 2024-09-18T07:36:19.959 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:36:19.959 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:36:19.959 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.959 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:32812 8.2 8:4118d61f:::benchmark_data_smithi137_101942_object32811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-18T07:36:19.959 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:36:19.942766+0000", 2024-09-18T07:36:19.959 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0108252, 2024-09-18T07:36:19.959 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010847771000000001, 2024-09-18T07:36:19.959 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:19.959 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:19.960 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:36:19.960 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:19.960 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:36:19.960 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:36:19.960 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 32812 2024-09-18T07:36:19.960 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.960 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:19.960 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.960 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:19.960 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.942766+0000", 2024-09-18T07:36:19.960 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:19.960 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.961 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.961 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:19.961 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.942766+0000", 2024-09-18T07:36:19.961 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:19.961 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.961 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.961 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:19.961 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.942768+0000", 2024-09-18T07:36:19.961 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0229999999999999e-06 2024-09-18T07:36:19.961 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.961 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.961 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:19.961 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.942832+0000", 2024-09-18T07:36:19.962 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3547999999999995e-05 2024-09-18T07:36:19.962 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.962 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.962 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:19.962 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.942847+0000", 2024-09-18T07:36:19.962 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5656999999999999e-05 2024-09-18T07:36:19.962 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.962 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.962 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:19.962 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.942854+0000", 2024-09-18T07:36:19.962 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.277e-06 2024-09-18T07:36:19.962 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.962 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.963 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:19.963 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.942941+0000", 2024-09-18T07:36:19.963 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6567e-05 2024-09-18T07:36:19.963 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.963 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.963 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:19.963 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.943070+0000", 2024-09-18T07:36:19.963 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012895199999999999 2024-09-18T07:36:19.963 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.963 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.963 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:36:19.963 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.943215+0000", 2024-09-18T07:36:19.963 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014537600000000001 2024-09-18T07:36:19.964 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.964 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.964 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:19.964 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.946743+0000", 2024-09-18T07:36:19.964 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0035276639999999998 2024-09-18T07:36:19.964 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.964 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.964 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:19.964 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.948886+0000", 2024-09-18T07:36:19.965 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021434570000000001 2024-09-18T07:36:19.965 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:19.965 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:19.965 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:19.965 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.965 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.965 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:32820 8.2 8:48c571f2:::benchmark_data_smithi137_101942_object32819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-18T07:36:19.965 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:36:19.943698+0000", 2024-09-18T07:36:19.965 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0098933749999999994, 2024-09-18T07:36:19.965 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0099641650000000005, 2024-09-18T07:36:19.965 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:19.965 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:19.966 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:36:19.966 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:19.966 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:36:19.966 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:36:19.966 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 32820 2024-09-18T07:36:19.966 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.966 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:19.966 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.966 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:19.966 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.943698+0000", 2024-09-18T07:36:19.966 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:19.966 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.966 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.967 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:19.967 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.943698+0000", 2024-09-18T07:36:19.967 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:19.967 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.967 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.967 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:19.967 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.943706+0000", 2024-09-18T07:36:19.967 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2619999999999998e-06 2024-09-18T07:36:19.967 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.967 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.967 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:19.967 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.943756+0000", 2024-09-18T07:36:19.967 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9789000000000001e-05 2024-09-18T07:36:19.968 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.968 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.968 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:19.968 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.943765+0000", 2024-09-18T07:36:19.968 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5610000000000004e-06 2024-09-18T07:36:19.968 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.968 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.968 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:19.968 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.943773+0000", 2024-09-18T07:36:19.968 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2899999999999997e-06 2024-09-18T07:36:19.968 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.968 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.968 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:19.969 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.945337+0000", 2024-09-18T07:36:19.969 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0015646119999999999 2024-09-18T07:36:19.969 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.969 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.969 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:19.969 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.945440+0000", 2024-09-18T07:36:19.969 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010298699999999999 2024-09-18T07:36:19.969 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.969 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.969 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:36:19.969 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.945563+0000", 2024-09-18T07:36:19.969 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012314499999999999 2024-09-18T07:36:19.970 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.970 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.970 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:19.970 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.948891+0000", 2024-09-18T07:36:19.970 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0033279109999999998 2024-09-18T07:36:19.970 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.970 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.970 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:19.970 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.949772+0000", 2024-09-18T07:36:19.970 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000881086 2024-09-18T07:36:19.970 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:19.970 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:19.970 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:19.971 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.971 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.971 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:32829 8.2 8:4a8664d2:::benchmark_data_smithi137_101942_object32828:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-18T07:36:19.971 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:36:19.949708+0000", 2024-09-18T07:36:19.971 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0038831600000000001, 2024-09-18T07:36:19.971 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0039939090000000003, 2024-09-18T07:36:19.971 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:19.971 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:19.971 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:36:19.971 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:19.971 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:36:19.971 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:36:19.971 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 32829 2024-09-18T07:36:19.971 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.972 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:19.972 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.972 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:19.972 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.949708+0000", 2024-09-18T07:36:19.972 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:19.972 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.972 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.972 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:19.972 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.949708+0000", 2024-09-18T07:36:19.972 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:19.972 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.972 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.973 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:19.973 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.949711+0000", 2024-09-18T07:36:19.973 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6529999999999998e-06 2024-09-18T07:36:19.973 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.973 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.973 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:19.973 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.949823+0000", 2024-09-18T07:36:19.973 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000112416 2024-09-18T07:36:19.973 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.973 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.973 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:19.973 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.949838+0000", 2024-09-18T07:36:19.973 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5038000000000001e-05 2024-09-18T07:36:19.974 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.974 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.974 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:19.974 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.949850+0000", 2024-09-18T07:36:19.974 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1844e-05 2024-09-18T07:36:19.974 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.974 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.975 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:19.975 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.949917+0000", 2024-09-18T07:36:19.975 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7254000000000004e-05 2024-09-18T07:36:19.975 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.975 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.975 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:19.975 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.950063+0000", 2024-09-18T07:36:19.975 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014608000000000001 2024-09-18T07:36:19.975 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.975 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.975 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:36:19.976 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.950206+0000", 2024-09-18T07:36:19.976 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014244299999999999 2024-09-18T07:36:19.976 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.976 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.976 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:19.976 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.953036+0000", 2024-09-18T07:36:19.976 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002830192 2024-09-18T07:36:19.976 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:19.976 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:19.976 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:19.976 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.976 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.976 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:32835 8.2 8:4c292921:::benchmark_data_smithi137_101942_object32834:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-18T07:36:19.977 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:36:19.951290+0000", 2024-09-18T07:36:19.977 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0023009940000000002, 2024-09-18T07:36:19.977 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0024489870000000001, 2024-09-18T07:36:19.977 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:19.977 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:19.977 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:36:19.977 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:19.977 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:36:19.977 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:36:19.977 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 32835 2024-09-18T07:36:19.977 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.977 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:19.977 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.978 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:19.978 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.951290+0000", 2024-09-18T07:36:19.978 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:19.978 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.978 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.978 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:19.978 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.951290+0000", 2024-09-18T07:36:19.978 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:19.978 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.978 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.978 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:19.978 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.951292+0000", 2024-09-18T07:36:19.978 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5e-06 2024-09-18T07:36:19.979 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.979 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.979 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:19.979 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.951332+0000", 2024-09-18T07:36:19.979 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0077000000000003e-05 2024-09-18T07:36:19.979 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.979 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.979 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:19.979 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.951346+0000", 2024-09-18T07:36:19.979 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4756999999999999e-05 2024-09-18T07:36:19.979 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.979 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.979 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:19.980 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.951354+0000", 2024-09-18T07:36:19.980 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2710000000000003e-06 2024-09-18T07:36:19.980 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.980 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.980 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:19.980 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.951504+0000", 2024-09-18T07:36:19.980 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000150185 2024-09-18T07:36:19.980 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.980 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.980 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:19.980 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.951611+0000", 2024-09-18T07:36:19.980 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000106613 2024-09-18T07:36:19.980 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.981 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.981 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:36:19.981 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.951742+0000", 2024-09-18T07:36:19.981 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013101500000000001 2024-09-18T07:36:19.981 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.981 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.981 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:19.981 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.953225+0000", 2024-09-18T07:36:19.981 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014838480000000001 2024-09-18T07:36:19.981 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:19.981 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:19.981 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:19.981 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.982 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.982 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:32836 8.e e211/201 8:7450ced4:::benchmark_data_smithi137_101942_object32835:head v 211'2107, mlcod=211'2103)", 2024-09-18T07:36:19.982 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:36:19.952969+0000", 2024-09-18T07:36:19.982 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.00062223599999999997, 2024-09-18T07:36:19.982 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00080589500000000005, 2024-09-18T07:36:19.982 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:19.982 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:19.982 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:36:19.982 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:19.982 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.982 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:19.982 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.952969+0000", 2024-09-18T07:36:19.983 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:19.983 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.983 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.983 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:19.983 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.952969+0000", 2024-09-18T07:36:19.983 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:19.983 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.983 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.983 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:19.983 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.952970+0000", 2024-09-18T07:36:19.983 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3239999999999999e-06 2024-09-18T07:36:19.983 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.983 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.983 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:19.984 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.953005+0000", 2024-09-18T07:36:19.984 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4378999999999997e-05 2024-09-18T07:36:19.984 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.984 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.984 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:19.984 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.953015+0000", 2024-09-18T07:36:19.984 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0854e-05 2024-09-18T07:36:19.984 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.984 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.984 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:19.984 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.953025+0000", 2024-09-18T07:36:19.984 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7110000000000007e-06 2024-09-18T07:36:19.984 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.985 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.985 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:19.985 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.953070+0000", 2024-09-18T07:36:19.985 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4530999999999998e-05 2024-09-18T07:36:19.985 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:19.985 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:19.985 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:19.985 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:19.953090+0000", 2024-09-18T07:36:19.985 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0409000000000001e-05 2024-09-18T07:36:19.985 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:19.985 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:19.985 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:19.985 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:19.986 INFO:teuthology.orchestra.run.smithi203.stdout: ], 2024-09-18T07:36:19.986 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 5 2024-09-18T07:36:19.986 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:36:19.989 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:20.021 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:20.030 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:20.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:20.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:20.131 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:20.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:20.168 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:20.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:20.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:20.306 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:20.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:20.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:20.408 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:20.438 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:20.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:20.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:20.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:20.573 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:20.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:20.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:20.675 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:20.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:20.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:20.719 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:20.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:20.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:20.867 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:20.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:20.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:20.903 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T07:36:20.918 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 15 16 34729 34713 144.615 123.188 0.0171998 0.00691038 2024-09-18T07:36:20.968 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:20.974 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:20 smithi203 ceph-mon[32237]: pgmap v288: 148 pgs: 148 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 127 B/s rd, 149 MiB/s wr, 2.38k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-09-18T07:36:21.014 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:21.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:21.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:21.110 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:20 smithi137 ceph-mon[29408]: pgmap v288: 148 pgs: 148 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 127 B/s rd, 149 MiB/s wr, 2.38k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-09-18T07:36:21.110 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:20 smithi137 ceph-mon[39175]: pgmap v288: 148 pgs: 148 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 127 B/s rd, 149 MiB/s wr, 2.38k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-09-18T07:36:21.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:21.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:21.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:21.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:21.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:21.276 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:36:21.276 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-18T07:36:21.276 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:36:21.302 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:21.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:21.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:21.348 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:36:21.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:21.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:21.442 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:21.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:21.543 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:21.574 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:21.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:21.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:21.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:21.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:21.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:21.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:21.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:21.830 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:36:21.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:21.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:21.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:21.918 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 16 16 36629 36613 142.997 118.75 0.0116879 0.00698799 2024-09-18T07:36:21.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:21.959 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:36:21.959 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:36:21.959 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:36:21.959 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:36:21.959 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:36:21.981 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:21.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:21.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:22.027 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:36:22.064 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:21 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4342693' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:22.064 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:21 smithi137 ceph-mon[29408]: from='client.25237 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:22.065 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:21 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4342693' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:22.065 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:21 smithi137 ceph-mon[39175]: from='client.25237 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:22.083 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:22.113 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:22.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:22.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:22.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:21 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4342693' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:22.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:21 smithi203 ceph-mon[32237]: from='client.25237 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:22.215 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:22.245 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:22.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:22.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:22.388 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:22.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:22.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:22.490 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:22.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:22.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:22.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:22.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:22.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:22.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:22.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:22.730 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T07:36:22.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:22.803 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:22.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:22.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:22.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:22.918 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 17 16 38601 38585 141.835 123.25 0.00685208 0.00704516 2024-09-18T07:36:22.941 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:22.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:22.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:22.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:22 smithi203 ceph-mon[32237]: pgmap v289: 148 pgs: 148 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 653 B/s rd, 94 MiB/s wr, 1.50k op/s; 6.8 KiB/s, 0 objects/s recovering 2024-09-18T07:36:22.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:22 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:22.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:22 smithi203 ceph-mon[32237]: from='client.25237 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:36:22.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:22 smithi203 ceph-mon[32237]: osdmap e212: 8 total, 8 up, 8 in 2024-09-18T07:36:23.042 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:23.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:23.093 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:23.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:23.127 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:22 smithi137 ceph-mon[29408]: pgmap v289: 148 pgs: 148 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 653 B/s rd, 94 MiB/s wr, 1.50k op/s; 6.8 KiB/s, 0 objects/s recovering 2024-09-18T07:36:23.127 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:22 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:23.127 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:22 smithi137 ceph-mon[29408]: from='client.25237 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:36:23.127 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:22 smithi137 ceph-mon[29408]: osdmap e212: 8 total, 8 up, 8 in 2024-09-18T07:36:23.127 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:22 smithi137 ceph-mon[39175]: pgmap v289: 148 pgs: 148 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 653 B/s rd, 94 MiB/s wr, 1.50k op/s; 6.8 KiB/s, 0 objects/s recovering 2024-09-18T07:36:23.127 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:22 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:23.128 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:22 smithi137 ceph-mon[39175]: from='client.25237 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:36:23.128 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:22 smithi137 ceph-mon[39175]: osdmap e212: 8 total, 8 up, 8 in 2024-09-18T07:36:23.155 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:36:23.194 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:23.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:23.231 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:23.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:23.307 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:23.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:23.372 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:23.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:23.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:23.473 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:23.507 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:23.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:23.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:23.609 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:23.630 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:23.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:23.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:23.731 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:23.766 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:23.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:23.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:23.778 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:36:23.778 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:36:23.778 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:36:23.778 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:36:23.778 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.778 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:20749 7.3 e88/81 7:c04d08df:::benchmark_data_smithi137_74180_object20748:head v 88'1239, mlcod=88'1237)", 2024-09-18T07:36:23.778 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.530035+0000", 2024-09-18T07:36:23.778 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 240.24219233400001, 2024-09-18T07:36:23.778 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13453020299999999, 2024-09-18T07:36:23.778 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.779 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.779 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.779 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.779 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.779 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.779 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530035+0000", 2024-09-18T07:36:23.779 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.779 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.779 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.779 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.779 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530035+0000", 2024-09-18T07:36:23.779 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.780 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.780 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.780 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.780 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530036+0000", 2024-09-18T07:36:23.780 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3200000000000004e-07 2024-09-18T07:36:23.780 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.780 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.780 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.780 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530106+0000", 2024-09-18T07:36:23.780 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0099999999999996e-05 2024-09-18T07:36:23.780 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.780 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.781 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.781 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530115+0000", 2024-09-18T07:36:23.781 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1419999999999999e-06 2024-09-18T07:36:23.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.781 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.781 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.781 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530122+0000", 2024-09-18T07:36:23.781 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3269999999999997e-06 2024-09-18T07:36:23.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.781 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.781 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.781 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530167+0000", 2024-09-18T07:36:23.781 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5247999999999998e-05 2024-09-18T07:36:23.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.782 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.782 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.782 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530185+0000", 2024-09-18T07:36:23.782 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8111e-05 2024-09-18T07:36:23.782 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.782 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.782 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.782 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530789+0000", 2024-09-18T07:36:23.782 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00060407400000000002 2024-09-18T07:36:23.782 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.782 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.782 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.782 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664566+0000", 2024-09-18T07:36:23.783 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13377636900000001 2024-09-18T07:36:23.783 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.783 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.783 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.783 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.783 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.783 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:20756 7.4 e88/81 7:206b3dce:::benchmark_data_smithi137_74180_object20755:head v 88'1303, mlcod=88'1301)", 2024-09-18T07:36:23.783 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.532983+0000", 2024-09-18T07:36:23.783 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 240.239244544, 2024-09-18T07:36:23.783 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13536040199999999, 2024-09-18T07:36:23.783 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.783 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.783 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.784 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.784 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.784 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.784 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.532983+0000", 2024-09-18T07:36:23.784 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.784 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.784 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.784 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.532983+0000", 2024-09-18T07:36:23.784 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.784 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.784 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.785 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.532984+0000", 2024-09-18T07:36:23.785 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7999999999999993e-07 2024-09-18T07:36:23.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.785 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.785 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.785 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533015+0000", 2024-09-18T07:36:23.785 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1195999999999997e-05 2024-09-18T07:36:23.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.785 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.785 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.785 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665392+0000", 2024-09-18T07:36:23.785 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13237665800000001 2024-09-18T07:36:23.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.786 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.786 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.786 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665400+0000", 2024-09-18T07:36:23.786 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7079999999999997e-06 2024-09-18T07:36:23.786 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.786 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.786 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.786 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665454+0000", 2024-09-18T07:36:23.786 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4116999999999997e-05 2024-09-18T07:36:23.786 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.786 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.786 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.786 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665474+0000", 2024-09-18T07:36:23.787 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0092999999999999e-05 2024-09-18T07:36:23.787 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.787 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.787 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.787 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.668302+0000", 2024-09-18T07:36:23.787 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0028280520000000002 2024-09-18T07:36:23.787 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.787 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.787 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.787 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.668343+0000", 2024-09-18T07:36:23.787 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1597999999999997e-05 2024-09-18T07:36:23.787 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.787 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.788 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.788 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.788 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.788 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:20750 7.3 e88/81 7:c7abdaa5:::benchmark_data_smithi137_74180_object20749:head v 88'1240, mlcod=88'1237)", 2024-09-18T07:36:23.788 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.533452+0000", 2024-09-18T07:36:23.788 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 240.23877535899999, 2024-09-18T07:36:23.788 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13358589900000001, 2024-09-18T07:36:23.788 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.788 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.789 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.789 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.789 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.789 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.789 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533452+0000", 2024-09-18T07:36:23.789 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.789 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.789 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.789 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533452+0000", 2024-09-18T07:36:23.789 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.789 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.790 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.790 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533453+0000", 2024-09-18T07:36:23.790 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.057e-06 2024-09-18T07:36:23.790 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.790 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.790 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.790 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533515+0000", 2024-09-18T07:36:23.790 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1557999999999996e-05 2024-09-18T07:36:23.790 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.790 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.790 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.790 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664524+0000", 2024-09-18T07:36:23.790 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13100931900000001 2024-09-18T07:36:23.791 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.791 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.791 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.791 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664540+0000", 2024-09-18T07:36:23.791 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5562000000000001e-05 2024-09-18T07:36:23.791 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.791 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.791 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.791 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664600+0000", 2024-09-18T07:36:23.791 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0124000000000002e-05 2024-09-18T07:36:23.791 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.791 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.791 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.792 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664620+0000", 2024-09-18T07:36:23.792 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9983000000000002e-05 2024-09-18T07:36:23.792 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.792 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.792 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.792 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.666988+0000", 2024-09-18T07:36:23.792 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023680559999999999 2024-09-18T07:36:23.792 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.792 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.792 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.792 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.667038+0000", 2024-09-18T07:36:23.792 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.024e-05 2024-09-18T07:36:23.792 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.793 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.793 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.793 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.793 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.793 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21270 7.1 7:801f54ca:::benchmark_data_smithi137_74180_object21269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:36:23.793 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.882531+0000", 2024-09-18T07:36:23.793 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 239.88969693600001, 2024-09-18T07:36:23.793 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16466234699999999, 2024-09-18T07:36:23.793 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.793 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.793 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.793 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:23.793 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:23.794 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:23.794 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21270 2024-09-18T07:36:23.794 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.794 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.794 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.794 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.794 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882531+0000", 2024-09-18T07:36:23.794 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.794 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.794 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.794 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.794 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882531+0000", 2024-09-18T07:36:23.794 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.795 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.795 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.795 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.795 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882533+0000", 2024-09-18T07:36:23.795 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8190000000000001e-06 2024-09-18T07:36:23.795 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.795 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.795 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.795 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882584+0000", 2024-09-18T07:36:23.795 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1919999999999998e-05 2024-09-18T07:36:23.795 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.795 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.795 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.795 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882593+0000", 2024-09-18T07:36:23.796 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8829999999999995e-06 2024-09-18T07:36:23.796 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.796 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.796 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.796 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882599+0000", 2024-09-18T07:36:23.796 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5849999999999999e-06 2024-09-18T07:36:23.796 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.796 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.796 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.796 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882634+0000", 2024-09-18T07:36:23.796 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5132999999999997e-05 2024-09-18T07:36:23.796 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.796 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.797 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.797 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882717+0000", 2024-09-18T07:36:23.797 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2819999999999996e-05 2024-09-18T07:36:23.797 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.797 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.797 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:36:23.797 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882824+0000", 2024-09-18T07:36:23.797 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010708400000000001 2024-09-18T07:36:23.797 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.797 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.797 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:23.797 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.883645+0000", 2024-09-18T07:36:23.798 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00082133399999999995 2024-09-18T07:36:23.798 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.798 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.798 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.798 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.940766+0000", 2024-09-18T07:36:23.798 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.057121166000000001 2024-09-18T07:36:23.798 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.798 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.798 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.798 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.046997+0000", 2024-09-18T07:36:23.798 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10623094900000001 2024-09-18T07:36:23.798 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.798 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.798 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.799 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047057+0000", 2024-09-18T07:36:23.799 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9580000000000002e-05 2024-09-18T07:36:23.799 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.799 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.799 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.799 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047193+0000", 2024-09-18T07:36:23.799 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000136074 2024-09-18T07:36:23.799 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.799 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.799 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.799 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.799 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.799 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21285 7.1 7:8dbe9f10:::benchmark_data_smithi137_74180_object21284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:36:23.800 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.944262+0000", 2024-09-18T07:36:23.800 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 239.82796589200001, 2024-09-18T07:36:23.800 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10313246500000001, 2024-09-18T07:36:23.800 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.800 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.800 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.800 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:23.800 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:23.800 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:23.800 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21285 2024-09-18T07:36:23.800 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.800 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.800 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.801 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.801 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944262+0000", 2024-09-18T07:36:23.801 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.801 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.801 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.801 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.801 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944262+0000", 2024-09-18T07:36:23.801 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.801 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.801 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.801 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.801 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944264+0000", 2024-09-18T07:36:23.801 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8679999999999999e-06 2024-09-18T07:36:23.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.802 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.802 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.802 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944324+0000", 2024-09-18T07:36:23.802 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0225e-05 2024-09-18T07:36:23.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.802 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.802 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.802 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944333+0000", 2024-09-18T07:36:23.802 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.064e-06 2024-09-18T07:36:23.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.802 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.802 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.803 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944338+0000", 2024-09-18T07:36:23.803 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9749999999999998e-06 2024-09-18T07:36:23.803 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.803 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.803 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.803 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944380+0000", 2024-09-18T07:36:23.803 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1773000000000001e-05 2024-09-18T07:36:23.803 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.803 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.803 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.803 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944476+0000", 2024-09-18T07:36:23.803 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6497999999999997e-05 2024-09-18T07:36:23.803 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.803 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.804 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:36:23.804 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944565+0000", 2024-09-18T07:36:23.804 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9288000000000003e-05 2024-09-18T07:36:23.804 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.804 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.804 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:23.804 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.945376+0000", 2024-09-18T07:36:23.804 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00081097899999999998 2024-09-18T07:36:23.804 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.804 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.804 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.804 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.946088+0000", 2024-09-18T07:36:23.804 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00071115600000000001 2024-09-18T07:36:23.805 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.805 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.805 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.805 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047252+0000", 2024-09-18T07:36:23.805 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.101164449 2024-09-18T07:36:23.805 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.805 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.805 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.805 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047340+0000", 2024-09-18T07:36:23.805 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8268000000000003e-05 2024-09-18T07:36:23.805 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.805 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.805 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.806 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047394+0000", 2024-09-18T07:36:23.806 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3922000000000003e-05 2024-09-18T07:36:23.806 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.806 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.806 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.806 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.806 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.806 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22331 7.1 7:8575e846:::benchmark_data_smithi137_74180_object22330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:36:23.806 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.655146+0000", 2024-09-18T07:36:23.806 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 239.11708177099999, 2024-09-18T07:36:23.806 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.29868304200000001, 2024-09-18T07:36:23.806 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.806 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.806 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.807 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:23.807 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:23.807 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:23.807 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22331 2024-09-18T07:36:23.807 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.807 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.807 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.807 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.808 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655146+0000", 2024-09-18T07:36:23.808 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.808 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.808 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.808 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.808 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655146+0000", 2024-09-18T07:36:23.808 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.809 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.809 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.809 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655148+0000", 2024-09-18T07:36:23.809 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.779e-06 2024-09-18T07:36:23.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.809 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.809 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.809 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655192+0000", 2024-09-18T07:36:23.809 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4023000000000002e-05 2024-09-18T07:36:23.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.809 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.809 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.810 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655201+0000", 2024-09-18T07:36:23.810 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3559999999999999e-06 2024-09-18T07:36:23.810 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.810 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.810 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.810 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655206+0000", 2024-09-18T07:36:23.810 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4539999999999998e-06 2024-09-18T07:36:23.810 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.810 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.810 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.810 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655250+0000", 2024-09-18T07:36:23.810 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3630000000000001e-05 2024-09-18T07:36:23.810 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.811 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.811 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.811 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655323+0000", 2024-09-18T07:36:23.811 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3382999999999999e-05 2024-09-18T07:36:23.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.811 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.811 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:36:23.811 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655411+0000", 2024-09-18T07:36:23.811 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7776999999999997e-05 2024-09-18T07:36:23.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.811 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.811 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:23.811 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.656141+0000", 2024-09-18T07:36:23.811 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00072986100000000005 2024-09-18T07:36:23.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.812 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.812 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.812 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.658652+0000", 2024-09-18T07:36:23.812 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025112569999999998 2024-09-18T07:36:23.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.812 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.812 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.812 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953705+0000", 2024-09-18T07:36:23.812 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.29505283900000001 2024-09-18T07:36:23.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.812 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.813 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.813 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953782+0000", 2024-09-18T07:36:23.813 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6793999999999997e-05 2024-09-18T07:36:23.813 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.813 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.813 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.813 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953829+0000", 2024-09-18T07:36:23.813 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6888999999999998e-05 2024-09-18T07:36:23.813 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.813 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.813 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.813 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.813 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.814 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22350 7.1 7:851589b5:::benchmark_data_smithi137_74180_object22349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:36:23.814 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.662419+0000", 2024-09-18T07:36:23.814 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 239.10980849000001, 2024-09-18T07:36:23.814 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.29155694799999998, 2024-09-18T07:36:23.814 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.814 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.814 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.814 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:23.814 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:23.814 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:23.814 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22350 2024-09-18T07:36:23.814 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.814 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.815 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.815 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.815 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662419+0000", 2024-09-18T07:36:23.815 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.815 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.815 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.815 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.815 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662419+0000", 2024-09-18T07:36:23.815 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.815 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.815 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.815 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.815 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662421+0000", 2024-09-18T07:36:23.816 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.39e-06 2024-09-18T07:36:23.816 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.816 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.816 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.816 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662473+0000", 2024-09-18T07:36:23.816 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2094e-05 2024-09-18T07:36:23.816 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.816 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.816 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.816 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662480+0000", 2024-09-18T07:36:23.816 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5449999999999998e-06 2024-09-18T07:36:23.816 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.816 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.817 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.817 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662485+0000", 2024-09-18T07:36:23.817 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9200000000000003e-06 2024-09-18T07:36:23.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.817 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.817 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.817 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662521+0000", 2024-09-18T07:36:23.817 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5559000000000003e-05 2024-09-18T07:36:23.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.817 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.817 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.817 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662589+0000", 2024-09-18T07:36:23.817 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8480999999999997e-05 2024-09-18T07:36:23.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.818 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.818 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:36:23.818 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662666+0000", 2024-09-18T07:36:23.818 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7224999999999993e-05 2024-09-18T07:36:23.818 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.818 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.818 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:23.818 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.663390+0000", 2024-09-18T07:36:23.818 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00072354199999999998 2024-09-18T07:36:23.818 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.818 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.818 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.819 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.663928+0000", 2024-09-18T07:36:23.819 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00053822699999999995 2024-09-18T07:36:23.819 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.819 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.819 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.819 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953879+0000", 2024-09-18T07:36:23.819 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.28995123699999997 2024-09-18T07:36:23.819 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.819 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.819 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.819 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953934+0000", 2024-09-18T07:36:23.819 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5047000000000003e-05 2024-09-18T07:36:23.819 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.819 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.820 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.820 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953976+0000", 2024-09-18T07:36:23.820 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1681000000000003e-05 2024-09-18T07:36:23.820 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.820 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.820 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.820 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.820 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.820 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22352 7.1 7:83b03301:::benchmark_data_smithi137_74180_object22351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:36:23.820 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.664495+0000", 2024-09-18T07:36:23.820 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 239.107732414, 2024-09-18T07:36:23.820 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.28962424599999997, 2024-09-18T07:36:23.820 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.821 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.821 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.821 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:23.821 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:23.821 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:23.821 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22352 2024-09-18T07:36:23.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.821 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.821 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.821 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.821 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664495+0000", 2024-09-18T07:36:23.821 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.821 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.822 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.822 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664495+0000", 2024-09-18T07:36:23.822 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.822 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.822 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.822 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664496+0000", 2024-09-18T07:36:23.822 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6000000000000003e-07 2024-09-18T07:36:23.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.822 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.822 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.822 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664568+0000", 2024-09-18T07:36:23.822 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2021999999999996e-05 2024-09-18T07:36:23.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.823 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.823 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.823 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664574+0000", 2024-09-18T07:36:23.823 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4620000000000001e-06 2024-09-18T07:36:23.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.823 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.823 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.823 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664579+0000", 2024-09-18T07:36:23.823 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0620000000000002e-06 2024-09-18T07:36:23.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.823 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.823 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.823 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664605+0000", 2024-09-18T07:36:23.824 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6126e-05 2024-09-18T07:36:23.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.824 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.824 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.824 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664676+0000", 2024-09-18T07:36:23.824 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1521999999999997e-05 2024-09-18T07:36:23.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.824 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.824 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:36:23.824 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664749+0000", 2024-09-18T07:36:23.824 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3232999999999995e-05 2024-09-18T07:36:23.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.824 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.825 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:23.825 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.665309+0000", 2024-09-18T07:36:23.825 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00055992900000000003 2024-09-18T07:36:23.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.825 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.825 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.825 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.665857+0000", 2024-09-18T07:36:23.825 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00054812600000000002 2024-09-18T07:36:23.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.825 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.825 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.825 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.954036+0000", 2024-09-18T07:36:23.825 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.288178926 2024-09-18T07:36:23.826 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.826 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.826 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.826 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.954085+0000", 2024-09-18T07:36:23.826 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8534e-05 2024-09-18T07:36:23.826 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.826 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.826 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.826 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.954119+0000", 2024-09-18T07:36:23.826 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4544e-05 2024-09-18T07:36:23.826 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.826 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.826 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.827 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.827 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:22994 7.9 e90/81 7:9ec32d99:::benchmark_data_smithi137_74180_object22993:head v 90'1458, mlcod=90'1455)", 2024-09-18T07:36:23.827 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.331557+0000", 2024-09-18T07:36:23.827 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 238.44067064199999, 2024-09-18T07:36:23.827 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.098100452000000005, 2024-09-18T07:36:23.827 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.827 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.827 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.827 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.827 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.827 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.827 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331557+0000", 2024-09-18T07:36:23.828 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.828 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.828 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.828 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331557+0000", 2024-09-18T07:36:23.828 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.828 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.828 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.828 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331558+0000", 2024-09-18T07:36:23.828 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4099999999999997e-07 2024-09-18T07:36:23.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.828 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.829 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.829 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331619+0000", 2024-09-18T07:36:23.829 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1216000000000005e-05 2024-09-18T07:36:23.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.829 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.829 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.829 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331626+0000", 2024-09-18T07:36:23.829 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3119999999999996e-06 2024-09-18T07:36:23.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.829 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.829 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.829 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331631+0000", 2024-09-18T07:36:23.829 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3370000000000001e-06 2024-09-18T07:36:23.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.830 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.830 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.830 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331670+0000", 2024-09-18T07:36:23.830 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9103999999999997e-05 2024-09-18T07:36:23.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.830 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.830 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.830 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.331682+0000", 2024-09-18T07:36:23.830 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1926000000000001e-05 2024-09-18T07:36:23.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.830 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.830 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.830 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.429623+0000", 2024-09-18T07:36:23.831 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.097941316 2024-09-18T07:36:23.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.831 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.831 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.831 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.429657+0000", 2024-09-18T07:36:23.831 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4400000000000003e-05 2024-09-18T07:36:23.831 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.831 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.831 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.831 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.831 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:23001 7.1 7:8979af80:::benchmark_data_smithi137_74180_object23000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:36:23.831 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.332415+0000", 2024-09-18T07:36:23.832 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 238.439813047, 2024-09-18T07:36:23.832 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.100079928, 2024-09-18T07:36:23.832 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.832 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.832 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.832 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:23.832 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:23.832 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:23.832 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 23001 2024-09-18T07:36:23.832 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.832 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.832 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.832 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.833 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332415+0000", 2024-09-18T07:36:23.833 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.833 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.833 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.833 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332415+0000", 2024-09-18T07:36:23.833 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.833 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.833 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.833 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332416+0000", 2024-09-18T07:36:23.833 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8369999999999999e-06 2024-09-18T07:36:23.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332477+0000", 2024-09-18T07:36:23.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0333e-05 2024-09-18T07:36:23.834 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332487+0000", 2024-09-18T07:36:23.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0203e-05 2024-09-18T07:36:23.834 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332494+0000", 2024-09-18T07:36:23.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.515e-06 2024-09-18T07:36:23.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.835 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.835 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.332560+0000", 2024-09-18T07:36:23.835 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5874e-05 2024-09-18T07:36:23.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.835 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.835 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.428551+0000", 2024-09-18T07:36:23.835 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.095990961 2024-09-18T07:36:23.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.835 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:36:23.836 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.428728+0000", 2024-09-18T07:36:23.836 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017673299999999999 2024-09-18T07:36:23.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.836 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.836 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:23.836 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.430995+0000", 2024-09-18T07:36:23.836 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022672819999999998 2024-09-18T07:36:23.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.836 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.836 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.836 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.432117+0000", 2024-09-18T07:36:23.836 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0011216939999999999 2024-09-18T07:36:23.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.837 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.837 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.837 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.432392+0000", 2024-09-18T07:36:23.837 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00027448799999999998 2024-09-18T07:36:23.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.837 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.837 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.837 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.432451+0000", 2024-09-18T07:36:23.837 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9233e-05 2024-09-18T07:36:23.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.837 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.837 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.837 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.432495+0000", 2024-09-18T07:36:23.838 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3775e-05 2024-09-18T07:36:23.838 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.838 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.838 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.838 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.838 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:44367 7.1 7:854d3320:::benchmark_data_smithi137_74180_object44366:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-18T07:36:23.838 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:43.591378+0000", 2024-09-18T07:36:23.838 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 220.18084984999999, 2024-09-18T07:36:23.838 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.113341394, 2024-09-18T07:36:23.838 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.838 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.838 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.839 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:23.839 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:23.839 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:23.839 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 44367 2024-09-18T07:36:23.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.839 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.839 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.839 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.839 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591378+0000", 2024-09-18T07:36:23.839 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.839 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.839 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.840 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591378+0000", 2024-09-18T07:36:23.840 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.840 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.840 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.840 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.840 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591379+0000", 2024-09-18T07:36:23.840 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7099999999999999e-06 2024-09-18T07:36:23.840 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.840 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.840 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.840 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591420+0000", 2024-09-18T07:36:23.840 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0559000000000002e-05 2024-09-18T07:36:23.840 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.841 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.841 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.841 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591432+0000", 2024-09-18T07:36:23.841 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.226e-05 2024-09-18T07:36:23.841 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.841 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.841 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.841 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591440+0000", 2024-09-18T07:36:23.841 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5830000000000001e-06 2024-09-18T07:36:23.841 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.841 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.841 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.841 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591497+0000", 2024-09-18T07:36:23.841 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7114999999999998e-05 2024-09-18T07:36:23.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.842 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.842 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.842 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591664+0000", 2024-09-18T07:36:23.842 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000167188 2024-09-18T07:36:23.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.842 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.842 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:36:23.842 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591884+0000", 2024-09-18T07:36:23.842 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000220223 2024-09-18T07:36:23.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.842 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.842 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:23.842 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.593492+0000", 2024-09-18T07:36:23.843 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016074139999999999 2024-09-18T07:36:23.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.843 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.843 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.593595+0000", 2024-09-18T07:36:23.843 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010319699999999999 2024-09-18T07:36:23.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.843 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.843 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.704655+0000", 2024-09-18T07:36:23.843 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.111060176 2024-09-18T07:36:23.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.844 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.844 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.704690+0000", 2024-09-18T07:36:23.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4916999999999998e-05 2024-09-18T07:36:23.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.844 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.844 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.844 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.704719+0000", 2024-09-18T07:36:23.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9051999999999999e-05 2024-09-18T07:36:23.844 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.844 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.844 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.844 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.845 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:51788 7.1 7:81b6d503:::benchmark_data_smithi137_74180_object51787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:36:23.845 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:36:23.845 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 213.10528367500001, 2024-09-18T07:36:23.845 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14889203200000001, 2024-09-18T07:36:23.845 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.845 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.845 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.845 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:23.845 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:23.845 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:23.845 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 51788 2024-09-18T07:36:23.845 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.845 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.846 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.846 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.846 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:36:23.846 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.846 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.846 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.846 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.846 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:36:23.846 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.846 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.846 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.846 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.846 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666966+0000", 2024-09-18T07:36:23.847 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1860000000000001e-05 2024-09-18T07:36:23.847 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.847 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.847 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.847 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666998+0000", 2024-09-18T07:36:23.847 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1856000000000001e-05 2024-09-18T07:36:23.847 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.847 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.847 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.847 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667012+0000", 2024-09-18T07:36:23.847 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4592e-05 2024-09-18T07:36:23.847 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.847 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.848 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.848 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667019+0000", 2024-09-18T07:36:23.848 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1940000000000001e-06 2024-09-18T07:36:23.848 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.848 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.848 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.848 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667083+0000", 2024-09-18T07:36:23.848 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3553999999999995e-05 2024-09-18T07:36:23.848 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.848 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.848 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.848 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667217+0000", 2024-09-18T07:36:23.848 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000133712 2024-09-18T07:36:23.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.849 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:36:23.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667345+0000", 2024-09-18T07:36:23.849 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012796199999999999 2024-09-18T07:36:23.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.849 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:23.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.668905+0000", 2024-09-18T07:36:23.849 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001560344 2024-09-18T07:36:23.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.849 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.850 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669080+0000", 2024-09-18T07:36:23.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000174964 2024-09-18T07:36:23.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.850 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.850 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815763+0000", 2024-09-18T07:36:23.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.146683217 2024-09-18T07:36:23.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.850 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.850 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815804+0000", 2024-09-18T07:36:23.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0535999999999999e-05 2024-09-18T07:36:23.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.851 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.851 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815836+0000", 2024-09-18T07:36:23.851 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2240999999999998e-05 2024-09-18T07:36:23.851 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.851 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.851 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.851 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.851 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.852 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:51794 7.1 7:8f789502:::benchmark_data_smithi137_74180_object51793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:36:23.852 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:36:23.852 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 213.10311056500001, 2024-09-18T07:36:23.852 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15046784899999999, 2024-09-18T07:36:23.852 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.853 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.853 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.853 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:23.853 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:23.853 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:23.853 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 51794 2024-09-18T07:36:23.853 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.853 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.853 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.853 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.853 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:36:23.853 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.853 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.854 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.854 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.854 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:36:23.854 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.854 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.854 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.854 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.854 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669118+0000", 2024-09-18T07:36:23.854 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4079999999999999e-06 2024-09-18T07:36:23.854 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.854 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.854 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.854 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669172+0000", 2024-09-18T07:36:23.855 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3808999999999999e-05 2024-09-18T07:36:23.855 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.855 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.855 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.855 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669184+0000", 2024-09-18T07:36:23.855 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1888e-05 2024-09-18T07:36:23.855 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.855 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.855 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.855 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669192+0000", 2024-09-18T07:36:23.855 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7549999999999994e-06 2024-09-18T07:36:23.855 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.855 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.856 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.856 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669229+0000", 2024-09-18T07:36:23.856 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7234000000000003e-05 2024-09-18T07:36:23.856 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.856 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.856 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.856 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669310+0000", 2024-09-18T07:36:23.856 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1248999999999994e-05 2024-09-18T07:36:23.856 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.856 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.856 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:36:23.856 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669391+0000", 2024-09-18T07:36:23.856 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0401000000000005e-05 2024-09-18T07:36:23.857 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.857 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.857 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:23.857 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.670397+0000", 2024-09-18T07:36:23.857 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010061619999999999 2024-09-18T07:36:23.857 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.857 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.857 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.857 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.670664+0000", 2024-09-18T07:36:23.857 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00026735700000000002 2024-09-18T07:36:23.857 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.857 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.857 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.858 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819479+0000", 2024-09-18T07:36:23.858 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14881438499999999 2024-09-18T07:36:23.858 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.858 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.858 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.858 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819537+0000", 2024-09-18T07:36:23.858 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8192999999999997e-05 2024-09-18T07:36:23.858 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.858 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.858 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.858 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819585+0000", 2024-09-18T07:36:23.858 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8007999999999998e-05 2024-09-18T07:36:23.858 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.859 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.859 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.859 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.859 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.859 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:74680 7.1 7:8999b183:::benchmark_data_smithi137_74180_object74679:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:36:23.859 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:36:23.859 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 187.59426251900001, 2024-09-18T07:36:23.859 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13647764200000001, 2024-09-18T07:36:23.859 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.859 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.859 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.859 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:23.859 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:23.860 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:23.860 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 74680 2024-09-18T07:36:23.860 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.860 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.860 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.860 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.860 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:36:23.860 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.860 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.860 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.860 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.860 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:36:23.860 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.861 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.861 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.861 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.861 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177967+0000", 2024-09-18T07:36:23.861 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.559e-06 2024-09-18T07:36:23.861 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.861 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.861 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.861 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178020+0000", 2024-09-18T07:36:23.861 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3810999999999997e-05 2024-09-18T07:36:23.861 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.861 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.861 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.862 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178030+0000", 2024-09-18T07:36:23.862 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5200000000000003e-06 2024-09-18T07:36:23.862 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.862 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.862 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.862 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178035+0000", 2024-09-18T07:36:23.862 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0429999999999998e-06 2024-09-18T07:36:23.862 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.862 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.862 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.862 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178074+0000", 2024-09-18T07:36:23.862 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9014999999999998e-05 2024-09-18T07:36:23.862 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.863 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.863 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.863 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178151+0000", 2024-09-18T07:36:23.863 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6526000000000006e-05 2024-09-18T07:36:23.863 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.863 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.863 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:36:23.863 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178233+0000", 2024-09-18T07:36:23.863 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2174000000000003e-05 2024-09-18T07:36:23.863 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.863 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.863 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:23.863 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.180222+0000", 2024-09-18T07:36:23.863 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0019888459999999998 2024-09-18T07:36:23.864 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.864 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.864 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.864 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.180453+0000", 2024-09-18T07:36:23.864 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023154 2024-09-18T07:36:23.864 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.864 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.864 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.864 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314370+0000", 2024-09-18T07:36:23.864 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.133916497 2024-09-18T07:36:23.864 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.864 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.865 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.865 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314411+0000", 2024-09-18T07:36:23.865 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1421000000000002e-05 2024-09-18T07:36:23.865 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.865 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.865 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.865 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314443+0000", 2024-09-18T07:36:23.865 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1690000000000003e-05 2024-09-18T07:36:23.865 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.865 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.865 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.865 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.865 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.866 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:76535 7.1 7:86123a56:::benchmark_data_smithi137_74180_object76534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:36:23.866 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.065654+0000", 2024-09-18T07:36:23.866 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 185.70657348, 2024-09-18T07:36:23.866 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.097319778999999995, 2024-09-18T07:36:23.866 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.866 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.866 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.866 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:23.866 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:23.866 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:23.866 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 76535 2024-09-18T07:36:23.866 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.866 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.867 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.867 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.867 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065654+0000", 2024-09-18T07:36:23.867 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.867 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.867 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.867 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.867 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065654+0000", 2024-09-18T07:36:23.867 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.867 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.867 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.867 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.867 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065656+0000", 2024-09-18T07:36:23.868 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8220000000000001e-06 2024-09-18T07:36:23.868 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.868 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.868 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.868 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065751+0000", 2024-09-18T07:36:23.868 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5267999999999997e-05 2024-09-18T07:36:23.868 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.868 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.868 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.868 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065761+0000", 2024-09-18T07:36:23.868 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2799999999999992e-06 2024-09-18T07:36:23.868 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.868 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.869 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.869 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065766+0000", 2024-09-18T07:36:23.869 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6699999999999999e-06 2024-09-18T07:36:23.869 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.869 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.869 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.869 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065808+0000", 2024-09-18T07:36:23.869 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2150999999999997e-05 2024-09-18T07:36:23.869 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.869 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.869 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.869 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.065904+0000", 2024-09-18T07:36:23.869 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5330999999999999e-05 2024-09-18T07:36:23.870 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.870 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.870 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:36:23.870 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.066005+0000", 2024-09-18T07:36:23.870 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010100800000000001 2024-09-18T07:36:23.870 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.870 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.870 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:23.870 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.067931+0000", 2024-09-18T07:36:23.870 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0019264239999999999 2024-09-18T07:36:23.870 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.870 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.871 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.871 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.070368+0000", 2024-09-18T07:36:23.871 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0024367659999999999 2024-09-18T07:36:23.871 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.871 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.871 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.871 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.162870+0000", 2024-09-18T07:36:23.871 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.092502134999999999 2024-09-18T07:36:23.871 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.871 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.871 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.871 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.162928+0000", 2024-09-18T07:36:23.871 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8029999999999998e-05 2024-09-18T07:36:23.872 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.872 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.872 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.872 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.162974+0000", 2024-09-18T07:36:23.872 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5893999999999998e-05 2024-09-18T07:36:23.872 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.872 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.872 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.872 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.872 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.872 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:76538 7.1 7:826fb340:::benchmark_data_smithi137_74180_object76537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:36:23.872 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.068561+0000", 2024-09-18T07:36:23.872 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 185.703666158, 2024-09-18T07:36:23.873 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.097070977000000003, 2024-09-18T07:36:23.873 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.873 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.873 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.873 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:23.873 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:23.873 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:23.873 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 76538 2024-09-18T07:36:23.873 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.873 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.873 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.873 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.873 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068561+0000", 2024-09-18T07:36:23.874 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.874 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.874 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.874 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.874 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068561+0000", 2024-09-18T07:36:23.874 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.874 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.874 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.874 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.874 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068563+0000", 2024-09-18T07:36:23.874 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5629999999999999e-06 2024-09-18T07:36:23.874 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.874 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.875 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.875 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068611+0000", 2024-09-18T07:36:23.875 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8412999999999999e-05 2024-09-18T07:36:23.875 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.875 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.875 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.875 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068620+0000", 2024-09-18T07:36:23.875 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7530000000000007e-06 2024-09-18T07:36:23.875 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.875 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.875 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.875 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068626+0000", 2024-09-18T07:36:23.875 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.818e-06 2024-09-18T07:36:23.875 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.876 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.876 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.876 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068663+0000", 2024-09-18T07:36:23.876 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7086999999999999e-05 2024-09-18T07:36:23.876 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.876 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.876 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.876 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068734+0000", 2024-09-18T07:36:23.876 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0605000000000006e-05 2024-09-18T07:36:23.876 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.876 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.876 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:36:23.876 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.068829+0000", 2024-09-18T07:36:23.877 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5533999999999997e-05 2024-09-18T07:36:23.877 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.877 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.877 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:23.877 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.070471+0000", 2024-09-18T07:36:23.877 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016418800000000001 2024-09-18T07:36:23.877 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.877 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.877 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.877 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.071217+0000", 2024-09-18T07:36:23.877 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00074625099999999997 2024-09-18T07:36:23.877 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.877 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.878 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.878 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.165516+0000", 2024-09-18T07:36:23.878 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.094298471999999994 2024-09-18T07:36:23.878 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.878 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.878 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.878 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.165590+0000", 2024-09-18T07:36:23.878 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4103999999999994e-05 2024-09-18T07:36:23.878 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.878 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.878 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.878 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.165632+0000", 2024-09-18T07:36:23.878 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2497000000000003e-05 2024-09-18T07:36:23.878 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.879 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.879 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.879 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.879 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.879 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:76881 7.1 7:8430cf73:::benchmark_data_smithi137_74180_object76880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:36:23.879 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.426831+0000", 2024-09-18T07:36:23.879 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 185.34539676099999, 2024-09-18T07:36:23.879 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.115187357, 2024-09-18T07:36:23.879 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.879 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.879 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.879 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:23.879 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:23.880 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:23.880 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 76881 2024-09-18T07:36:23.880 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.880 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.880 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.880 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.880 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426831+0000", 2024-09-18T07:36:23.880 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.880 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.880 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.880 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.880 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426831+0000", 2024-09-18T07:36:23.880 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.881 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.881 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.881 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.881 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426833+0000", 2024-09-18T07:36:23.881 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7710000000000001e-06 2024-09-18T07:36:23.881 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.881 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.881 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.881 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426869+0000", 2024-09-18T07:36:23.881 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5886999999999997e-05 2024-09-18T07:36:23.881 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.881 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.881 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.882 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426879+0000", 2024-09-18T07:36:23.882 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0667e-05 2024-09-18T07:36:23.882 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.882 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.882 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.882 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426886+0000", 2024-09-18T07:36:23.882 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3799999999999999e-06 2024-09-18T07:36:23.882 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.882 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.882 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.882 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426932+0000", 2024-09-18T07:36:23.882 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6143999999999997e-05 2024-09-18T07:36:23.883 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.883 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.883 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.883 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.427047+0000", 2024-09-18T07:36:23.883 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011568700000000001 2024-09-18T07:36:23.883 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.883 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.883 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:36:23.883 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.427168+0000", 2024-09-18T07:36:23.883 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000120118 2024-09-18T07:36:23.883 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.883 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.883 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:23.884 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.429449+0000", 2024-09-18T07:36:23.884 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002281394 2024-09-18T07:36:23.884 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.884 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.884 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.884 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.431262+0000", 2024-09-18T07:36:23.884 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0018125940000000001 2024-09-18T07:36:23.884 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.884 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.884 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.884 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541358+0000", 2024-09-18T07:36:23.884 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.110095996 2024-09-18T07:36:23.884 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.885 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.885 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.885 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541963+0000", 2024-09-18T07:36:23.885 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00060516200000000002 2024-09-18T07:36:23.885 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.885 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.885 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.885 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.542018+0000", 2024-09-18T07:36:23.885 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5556999999999997e-05 2024-09-18T07:36:23.885 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.885 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.885 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.886 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.886 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.886 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:76891 7.f e116/115 7:f77e09af:::benchmark_data_smithi137_74180_object76890:head v 116'4811, mlcod=116'4809)", 2024-09-18T07:36:23.886 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.436137+0000", 2024-09-18T07:36:23.886 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 185.33609063700001, 2024-09-18T07:36:23.886 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.105530608, 2024-09-18T07:36:23.886 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.886 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.886 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.886 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.886 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.886 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.886 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436137+0000", 2024-09-18T07:36:23.887 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.887 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.887 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.887 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.887 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436137+0000", 2024-09-18T07:36:23.887 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.887 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.887 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.887 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.887 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436139+0000", 2024-09-18T07:36:23.887 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3259999999999998e-06 2024-09-18T07:36:23.887 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.887 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.888 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.888 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436192+0000", 2024-09-18T07:36:23.888 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2339000000000002e-05 2024-09-18T07:36:23.888 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.888 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.888 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.888 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436209+0000", 2024-09-18T07:36:23.888 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6935999999999998e-05 2024-09-18T07:36:23.888 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.888 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.888 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.888 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436219+0000", 2024-09-18T07:36:23.888 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0593e-05 2024-09-18T07:36:23.889 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.889 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.889 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.889 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436314+0000", 2024-09-18T07:36:23.889 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5035000000000001e-05 2024-09-18T07:36:23.889 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.889 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.889 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.889 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436347+0000", 2024-09-18T07:36:23.889 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2558999999999998e-05 2024-09-18T07:36:23.889 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.889 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.889 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.889 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.437824+0000", 2024-09-18T07:36:23.890 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001477405 2024-09-18T07:36:23.890 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.890 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.890 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.890 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541668+0000", 2024-09-18T07:36:23.890 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10384341499999999 2024-09-18T07:36:23.890 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.890 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.890 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.890 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.890 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.890 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:76893 7.4 e116/81 7:2dcd0014:::benchmark_data_smithi137_74180_object76892:head v 116'4832, mlcod=116'4830)", 2024-09-18T07:36:23.890 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.438822+0000", 2024-09-18T07:36:23.891 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 185.33340566000001, 2024-09-18T07:36:23.891 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.106356858, 2024-09-18T07:36:23.891 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.891 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.891 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.891 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.891 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.891 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.891 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.438822+0000", 2024-09-18T07:36:23.891 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.891 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.891 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.891 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.892 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.438822+0000", 2024-09-18T07:36:23.892 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.892 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.892 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.892 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.892 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.438823+0000", 2024-09-18T07:36:23.892 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2979999999999999e-06 2024-09-18T07:36:23.892 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.892 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.892 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.892 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.438847+0000", 2024-09-18T07:36:23.892 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3370000000000002e-05 2024-09-18T07:36:23.893 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.893 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.893 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.893 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541862+0000", 2024-09-18T07:36:23.893 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10301571299999999 2024-09-18T07:36:23.893 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.893 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.893 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.893 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541872+0000", 2024-09-18T07:36:23.893 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0017000000000001e-05 2024-09-18T07:36:23.893 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.893 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.893 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.893 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541931+0000", 2024-09-18T07:36:23.894 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8990000000000003e-05 2024-09-18T07:36:23.894 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.894 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.894 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.894 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541958+0000", 2024-09-18T07:36:23.894 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6996e-05 2024-09-18T07:36:23.894 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.894 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.894 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.894 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.545137+0000", 2024-09-18T07:36:23.894 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003178859 2024-09-18T07:36:23.894 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.894 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.895 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.895 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.545179+0000", 2024-09-18T07:36:23.895 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1615000000000001e-05 2024-09-18T07:36:23.895 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.895 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.895 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.895 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.895 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.895 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:142957 7.1 7:830d8edf:::benchmark_data_smithi137_74180_object38541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-18T07:36:23.895 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:25.430263+0000", 2024-09-18T07:36:23.895 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 118.34196489599999, 2024-09-18T07:36:23.895 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18925961299999999, 2024-09-18T07:36:23.895 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:23.896 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:23.896 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:23.896 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:23.896 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:23.896 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:23.896 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 142957 2024-09-18T07:36:23.896 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.896 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:23.896 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.896 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:23.896 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430263+0000", 2024-09-18T07:36:23.896 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.896 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.897 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.897 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:23.897 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430263+0000", 2024-09-18T07:36:23.897 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:23.897 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.897 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.897 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:23.897 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430265+0000", 2024-09-18T07:36:23.897 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9800000000000001e-06 2024-09-18T07:36:23.897 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.897 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.897 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:23.897 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430270+0000", 2024-09-18T07:36:23.897 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6189999999999999e-06 2024-09-18T07:36:23.898 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.898 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.898 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:23.898 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430281+0000", 2024-09-18T07:36:23.898 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1146e-05 2024-09-18T07:36:23.898 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.898 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.898 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:23.898 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430288+0000", 2024-09-18T07:36:23.898 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9319999999999999e-06 2024-09-18T07:36:23.898 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.898 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.898 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:23.898 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430340+0000", 2024-09-18T07:36:23.899 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1904999999999999e-05 2024-09-18T07:36:23.899 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.899 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.899 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:23.899 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430468+0000", 2024-09-18T07:36:23.899 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000127448 2024-09-18T07:36:23.899 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.899 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.899 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:36:23.899 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430717+0000", 2024-09-18T07:36:23.899 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024888500000000001 2024-09-18T07:36:23.899 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.899 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.900 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:23.900 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.432189+0000", 2024-09-18T07:36:23.900 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014722470000000001 2024-09-18T07:36:23.900 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.900 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.900 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.900 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.432734+0000", 2024-09-18T07:36:23.900 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00054558799999999995 2024-09-18T07:36:23.900 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.900 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.900 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:23.900 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.619409+0000", 2024-09-18T07:36:23.900 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18667450399999999 2024-09-18T07:36:23.901 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.901 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.901 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:23.901 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.619471+0000", 2024-09-18T07:36:23.901 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1877000000000004e-05 2024-09-18T07:36:23.901 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:23.901 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:23.901 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:23.901 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.619522+0000", 2024-09-18T07:36:23.901 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1481999999999999e-05 2024-09-18T07:36:23.901 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.901 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.901 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.901 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:23.902 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:23.902 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:36:23.902 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:23.918 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 18 16 40365 40349 140.08 110.25 0.00924382 0.00713378 2024-09-18T07:36:23.934 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:23.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:23.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:24.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:24.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:24.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:24.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:24.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:24.205 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:24.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:24.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:24.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:24.342 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:24.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:24.443 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:24.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:24.481 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:24.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:24.582 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:24.616 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:24.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:24.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:24.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:24.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:24.757 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:24.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:24.843 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:36:24.843 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:36:24.858 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:24.871 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:36:24.872 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:36:24.889 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:24.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:24.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:24.903 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:36:24.903 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:36:24.918 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 19 16 41869 41853 137.654 94 0.00616209 0.00725906 2024-09-18T07:36:24.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:24 smithi137 ceph-mon[29408]: pgmap v291: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 614 B/s rd, 120 MiB/s wr, 1.93k op/s; 6.4 KiB/s, 0 objects/s recovering 2024-09-18T07:36:24.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:24 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2809210358' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:24.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:24 smithi137 ceph-mon[29408]: osdmap e213: 8 total, 8 up, 8 in 2024-09-18T07:36:24.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:24 smithi137 ceph-mon[39175]: pgmap v291: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 614 B/s rd, 120 MiB/s wr, 1.93k op/s; 6.4 KiB/s, 0 objects/s recovering 2024-09-18T07:36:24.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:24 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2809210358' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:24.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:24 smithi137 ceph-mon[39175]: osdmap e213: 8 total, 8 up, 8 in 2024-09-18T07:36:24.939 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:36:24.940 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:36:24.972 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:36:24.972 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:36:24.991 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:25.004 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:36:25.004 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:36:25.019 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:25.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:25.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:25.041 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:36:25.041 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:36:25.075 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:36:25.075 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:36:25.121 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:25.127 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:24 smithi203 ceph-mon[32237]: pgmap v291: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 614 B/s rd, 120 MiB/s wr, 1.93k op/s; 6.4 KiB/s, 0 objects/s recovering 2024-09-18T07:36:25.127 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:24 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2809210358' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:25.128 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:24 smithi203 ceph-mon[32237]: osdmap e213: 8 total, 8 up, 8 in 2024-09-18T07:36:25.153 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:25.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:25.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:25.179 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:36:25.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:25.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:25.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:25.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:25.398 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:25.438 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:25.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:25.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:25.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:25.570 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T07:36:25.580 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:25.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:25.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:25.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:25.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:25.722 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:25.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:25.823 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:25.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:25.860 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:25.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:25.919 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:36:25.917517+0000 min lat: 0.00114205 max lat: 1.03286 avg lat: 0.00732551 2024-09-18T07:36:25.919 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:36:25.919 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 20 15 43677 43662 136.424 113.062 0.00666612 0.00732551 2024-09-18T07:36:25.962 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:25.978 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:36:26.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:26.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:26.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:26.049 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:26.113 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:26.145 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:26.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:26.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:26.247 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:26.283 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:26.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:26.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:26.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:26.420 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:26.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:26.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:26.522 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:26.560 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:26.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:26.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:26.593 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:36:26.593 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:36:26.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:26.698 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:26.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:26.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:26.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:26.833 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:26.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:26.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:26.919 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 21 16 45424 45408 135.124 109.125 0.00512116 0.00739476 2024-09-18T07:36:26.935 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:26.976 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:26.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:26.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:27.078 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:27.111 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:27.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:27.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:27.138 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:26 smithi203 ceph-mon[32237]: pgmap v293: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-18T07:36:27.138 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:27.139 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:36:27.154 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:36:27.154 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:36:27.154 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_1 2024-09-18T07:36:27.154 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_1 by 10 min 8 2024-09-18T07:36:27.154 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:36:27.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:36:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:36:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:36:27.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:26 smithi137 ceph-mon[29408]: pgmap v293: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-18T07:36:27.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:27.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:36:27.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:26 smithi137 ceph-mon[39175]: pgmap v293: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-18T07:36:27.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:27.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:36:27.213 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:27.254 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:27.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:27.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:27.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T07:36:27.356 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:27.395 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:27.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:27.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:27.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:36:27 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:36:27.497 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:27.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:27.547 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:27.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:27.588 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:27.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:27.681 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:27.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:27.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:27.710 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:36:27.784 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:27.816 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:27.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:27.873 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:36:27.918 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:27.919 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 22 16 47176 47160 133.959 109.5 0.0105818 0.00746013 2024-09-18T07:36:27.946 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:27.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:27.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:27.963 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:36:28.002 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:27 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1081118570' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:36:28.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:27 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1081118570' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:36:28.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:28.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:28.088 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:28.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:28.161 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:27 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1081118570' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:36:28.189 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:28.228 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:28.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:28.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:28.310 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:36:28.310 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:36:28.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:28.370 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:28.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:28.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:28.473 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:28.509 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:28.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:28.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:28.611 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:28.640 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:28.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:28.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:28.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:28.780 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:28.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:28.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:28.883 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:28.909 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:36:28.911 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:36:28.918 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:28.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:28.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:28.920 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 23 15 48803 48788 132.549 101.75 0.0206404 0.00753866 2024-09-18T07:36:29.000 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T07:36:29.020 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:29.083 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:29.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:29.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:29.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:28 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:29.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:28 smithi137 ceph-mon[29408]: pgmap v294: 148 pgs: 148 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 141 MiB/s wr, 2.26k op/s 2024-09-18T07:36:29.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:28 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:29.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:28 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1081118570' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:36:29.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:28 smithi137 ceph-mon[29408]: osdmap e214: 8 total, 8 up, 8 in 2024-09-18T07:36:29.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:28 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:29.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:28 smithi137 ceph-mon[39175]: pgmap v294: 148 pgs: 148 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 141 MiB/s wr, 2.26k op/s 2024-09-18T07:36:29.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:28 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:29.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:28 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1081118570' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:36:29.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:28 smithi137 ceph-mon[39175]: osdmap e214: 8 total, 8 up, 8 in 2024-09-18T07:36:29.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:29.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:28 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:29.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:28 smithi203 ceph-mon[32237]: pgmap v294: 148 pgs: 148 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 141 MiB/s wr, 2.26k op/s 2024-09-18T07:36:29.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:28 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:29.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:28 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1081118570' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:36:29.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:28 smithi203 ceph-mon[32237]: osdmap e214: 8 total, 8 up, 8 in 2024-09-18T07:36:29.222 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:29.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:29.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:29.324 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:29.358 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:29.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:29.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:29.389 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:36:29.460 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:29.494 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:29.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:29.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:29.631 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:29.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:29.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:29.733 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:29.767 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:29.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:29.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:29.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:29.889 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:29.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:29.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:29.921 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 24 16 50356 50340 131.068 97 0.00662531 0.00762448 2024-09-18T07:36:29.970 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:36:29.970 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:36:29.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:30.025 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:30.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:30.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:30.127 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:30.148 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:30.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:30.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:30.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:29 smithi137 ceph-mon[29408]: 3.d scrub starts 2024-09-18T07:36:30.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:29 smithi137 ceph-mon[29408]: 3.d scrub ok 2024-09-18T07:36:30.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:29 smithi137 ceph-mon[29408]: 5.7 scrub starts 2024-09-18T07:36:30.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:29 smithi137 ceph-mon[29408]: 5.7 scrub ok 2024-09-18T07:36:30.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:29 smithi137 ceph-mon[29408]: from='client.25252 v1:172.21.15.137:0/3699079955' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:36:30.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:29 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:36:30.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:29 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:36:30.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:29 smithi137 ceph-mon[29408]: osdmap e215: 8 total, 8 up, 8 in 2024-09-18T07:36:30.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:29 smithi137 ceph-mon[39175]: 3.d scrub starts 2024-09-18T07:36:30.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:29 smithi137 ceph-mon[39175]: 3.d scrub ok 2024-09-18T07:36:30.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:29 smithi137 ceph-mon[39175]: 5.7 scrub starts 2024-09-18T07:36:30.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:29 smithi137 ceph-mon[39175]: 5.7 scrub ok 2024-09-18T07:36:30.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:29 smithi137 ceph-mon[39175]: from='client.25252 v1:172.21.15.137:0/3699079955' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:36:30.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:29 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:36:30.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:29 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:36:30.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:29 smithi137 ceph-mon[39175]: osdmap e215: 8 total, 8 up, 8 in 2024-09-18T07:36:30.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:29 smithi203 ceph-mon[32237]: 3.d scrub starts 2024-09-18T07:36:30.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:29 smithi203 ceph-mon[32237]: 3.d scrub ok 2024-09-18T07:36:30.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:29 smithi203 ceph-mon[32237]: 5.7 scrub starts 2024-09-18T07:36:30.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:29 smithi203 ceph-mon[32237]: 5.7 scrub ok 2024-09-18T07:36:30.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:29 smithi203 ceph-mon[32237]: from='client.25252 v1:172.21.15.137:0/3699079955' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:36:30.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:29 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:36:30.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:29 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:36:30.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:29 smithi203 ceph-mon[32237]: osdmap e215: 8 total, 8 up, 8 in 2024-09-18T07:36:30.249 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:30.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:30.285 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:30.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:30.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:30.411 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:30.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:30.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:30.513 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:30.546 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:30.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:30.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:30.647 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T07:36:30.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:30.668 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:30.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:30.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:30.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:30.805 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:30.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:30.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:30.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:36:30.908 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:30.921 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 25 16 51779 51763 129.382 88.9375 0.00668164 0.007724 2024-09-18T07:36:30.940 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:30.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:31.015 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:36:31.042 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:31.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:31.083 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:31.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:31.142 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:30 smithi203 ceph-mon[32237]: 4.11 scrub starts 2024-09-18T07:36:31.142 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:30 smithi203 ceph-mon[32237]: 4.11 scrub ok 2024-09-18T07:36:31.142 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:30 smithi203 ceph-mon[32237]: pgmap v296: 148 pgs: 148 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 666 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-18T07:36:31.142 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:30 smithi203 ceph-mon[32237]: 5.1 scrub starts 2024-09-18T07:36:31.142 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:30 smithi203 ceph-mon[32237]: 5.1 scrub ok 2024-09-18T07:36:31.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:30 smithi137 ceph-mon[39175]: 4.11 scrub starts 2024-09-18T07:36:31.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:30 smithi137 ceph-mon[39175]: 4.11 scrub ok 2024-09-18T07:36:31.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:30 smithi137 ceph-mon[39175]: pgmap v296: 148 pgs: 148 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 666 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-18T07:36:31.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:30 smithi137 ceph-mon[39175]: 5.1 scrub starts 2024-09-18T07:36:31.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:30 smithi137 ceph-mon[39175]: 5.1 scrub ok 2024-09-18T07:36:31.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:30 smithi137 ceph-mon[29408]: 4.11 scrub starts 2024-09-18T07:36:31.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:30 smithi137 ceph-mon[29408]: 4.11 scrub ok 2024-09-18T07:36:31.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:30 smithi137 ceph-mon[29408]: pgmap v296: 148 pgs: 148 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 666 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-18T07:36:31.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:30 smithi137 ceph-mon[29408]: 5.1 scrub starts 2024-09-18T07:36:31.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:30 smithi137 ceph-mon[29408]: 5.1 scrub ok 2024-09-18T07:36:31.184 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:31.225 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:31.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:31.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:31.348 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:31.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:31.449 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:31.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:31.481 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:31.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:31.559 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:36:31.559 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:36:31.582 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:31.616 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:31.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:31.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:31.718 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:31.754 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:31.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:31.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:31.856 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:31.880 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:31.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:31.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:31.921 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 26 16 53161 53145 127.728 86.375 0.00929114 0.00782451 2024-09-18T07:36:31.982 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:32.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:32.017 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:32.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:32.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:32.139 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:32.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:32.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:32.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:31 smithi137 ceph-mon[29408]: 5.11 scrub starts 2024-09-18T07:36:32.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:31 smithi137 ceph-mon[29408]: 5.11 scrub ok 2024-09-18T07:36:32.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:31 smithi137 ceph-mon[39175]: 5.11 scrub starts 2024-09-18T07:36:32.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:31 smithi137 ceph-mon[39175]: 5.11 scrub ok 2024-09-18T07:36:32.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:31 smithi203 ceph-mon[32237]: 5.11 scrub starts 2024-09-18T07:36:32.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:31 smithi203 ceph-mon[32237]: 5.11 scrub ok 2024-09-18T07:36:32.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:32.261 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T07:36:32.284 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:32.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:32.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:32.390 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:32.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:32.431 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:32.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:32.533 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:32.578 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:32.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:32.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:32.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:32.705 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:36:32.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:32.729 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:32.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:32.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:32.871 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:32.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:32.917 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:32.922 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 27 16 54763 54747 126.701 100.125 0.0160083 0.00788611 2024-09-18T07:36:32.928 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-09-18T07:36:27.871500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2942638619,"num_objects":45091,"num_object_clones":0,"num_object_copies":135273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45091,"num_whiteouts":0,"num_read":911,"num_read_kb":770,"num_write":45441,"num_write_kb":2874761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":1840485,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45456,"ondisk_log_size":45456,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":446,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9884108,"kb_used_data":8536508,"kb_used_omap":12,"kb_used_meta":1347443,"kb_avail":740077108,"statfs":{"total":767960285184,"available":757838958592,"internally_reserved":0,"allocated":8741384192,"data_stored":8736991409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1379782192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,4,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":919011328,"num_objects":14023,"num_object_clones":0,"num_object_copies":42069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14023,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":14023,"num_write_kb":897472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.364085"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":332,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.820614+0000","last_change":"2024-09-18T07:35:28.534496+0000","last_active":"2024-09-18T07:36:24.820614+0000","last_peered":"2024-09-18T07:36:24.820614+0000","last_clean":"2024-09-18T07:36:24.820614+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:36:24.820614+0000","last_undegraded":"2024-09-18T07:36:24.820614+0000","last_fullsized":"2024-09-18T07:36:24.820614+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:28.534391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:28.534391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:55.524474+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":363,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.806321+0000","last_change":"2024-09-18T07:35:27.883344+0000","last_active":"2024-09-18T07:36:24.806321+0000","last_peered":"2024-09-18T07:36:24.806321+0000","last_clean":"2024-09-18T07:36:24.806321+0000","last_became_active":"2024-09-18T07:32:26.785727+0000","last_became_peered":"2024-09-18T07:32:26.785727+0000","last_unstale":"2024-09-18T07:36:24.806321+0000","last_undegraded":"2024-09-18T07:36:24.806321+0000","last_fullsized":"2024-09-18T07:36:24.806321+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:35:27.883281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:27.883281+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:40.767022+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":266,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.817759+0000","last_change":"2024-09-18T07:36:13.129229+0000","last_active":"2024-09-18T07:36:24.817759+0000","last_peered":"2024-09-18T07:36:24.817759+0000","last_clean":"2024-09-18T07:36:24.817759+0000","last_became_active":"2024-09-18T07:36:13.128805+0000","last_became_peered":"2024-09-18T07:36:13.128805+0000","last_unstale":"2024-09-18T07:36:24.817759+0000","last_undegraded":"2024-09-18T07:36:24.817759+0000","last_fullsized":"2024-09-18T07:36:24.817759+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:57.894611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.894611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":314,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.806628+0000","last_change":"2024-09-18T07:32:03.953881+0000","last_active":"2024-09-18T07:36:24.806628+0000","last_peered":"2024-09-18T07:36:24.806628+0000","last_clean":"2024-09-18T07:36:24.806628+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:36:24.806628+0000","last_undegraded":"2024-09-18T07:36:24.806628+0000","last_fullsized":"2024-09-18T07:36:24.806628+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.953827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.953827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:22.089067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":213,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.817125+0000","last_change":"2024-09-18T07:35:01.481557+0000","last_active":"2024-09-18T07:36:24.817125+0000","last_peered":"2024-09-18T07:36:24.817125+0000","last_clean":"2024-09-18T07:36:24.817125+0000","last_became_active":"2024-09-18T07:35:01.414396+0000","last_became_peered":"2024-09-18T07:35:01.414396+0000","last_unstale":"2024-09-18T07:36:24.817125+0000","last_undegraded":"2024-09-18T07:36:24.817125+0000","last_fullsized":"2024-09-18T07:36:24.817125+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:01.481514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:01.481514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":325,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.822358+0000","last_change":"2024-09-18T07:33:27.006545+0000","last_active":"2024-09-18T07:36:24.822358+0000","last_peered":"2024-09-18T07:36:24.822358+0000","last_clean":"2024-09-18T07:36:24.822358+0000","last_became_active":"2024-09-18T07:30:56.286040+0000","last_became_peered":"2024-09-18T07:30:56.286040+0000","last_unstale":"2024-09-18T07:36:24.822358+0000","last_undegraded":"2024-09-18T07:36:24.822358+0000","last_fullsized":"2024-09-18T07:36:24.822358+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.006488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.006488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:53.997087+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":346,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.824076+0000","last_change":"2024-09-18T07:33:22.032321+0000","last_active":"2024-09-18T07:36:24.824076+0000","last_peered":"2024-09-18T07:36:24.824076+0000","last_clean":"2024-09-18T07:36:24.824076+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:36:24.824076+0000","last_undegraded":"2024-09-18T07:36:24.824076+0000","last_fullsized":"2024-09-18T07:36:24.824076+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:22.032110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.032110+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:46.902576+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":326,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.806314+0000","last_change":"2024-09-18T07:35:30.995338+0000","last_active":"2024-09-18T07:36:24.806314+0000","last_peered":"2024-09-18T07:36:24.806314+0000","last_clean":"2024-09-18T07:36:24.806314+0000","last_became_active":"2024-09-18T07:31:00.331224+0000","last_became_peered":"2024-09-18T07:31:00.331224+0000","last_unstale":"2024-09-18T07:36:24.806314+0000","last_undegraded":"2024-09-18T07:36:24.806314+0000","last_fullsized":"2024-09-18T07:36:24.806314+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:30.995282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:30.995282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:49.480100+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":332,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.821759+0000","last_change":"2024-09-18T07:35:00.392530+0000","last_active":"2024-09-18T07:36:24.821759+0000","last_peered":"2024-09-18T07:36:24.821759+0000","last_clean":"2024-09-18T07:36:24.821759+0000","last_became_active":"2024-09-18T07:30:56.285882+0000","last_became_peered":"2024-09-18T07:30:56.285882+0000","last_unstale":"2024-09-18T07:36:24.821759+0000","last_undegraded":"2024-09-18T07:36:24.821759+0000","last_fullsized":"2024-09-18T07:36:24.821759+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:00.392466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.392466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":338,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.807163+0000","last_change":"2024-09-18T07:32:02.002919+0000","last_active":"2024-09-18T07:36:24.807163+0000","last_peered":"2024-09-18T07:36:24.807163+0000","last_clean":"2024-09-18T07:36:24.807163+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:36:24.807163+0000","last_undegraded":"2024-09-18T07:36:24.807163+0000","last_fullsized":"2024-09-18T07:36:24.807163+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.002811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.002811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:21.056265+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":330,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.821881+0000","last_change":"2024-09-18T07:35:04.267623+0000","last_active":"2024-09-18T07:36:24.821881+0000","last_peered":"2024-09-18T07:36:24.821881+0000","last_clean":"2024-09-18T07:36:24.821881+0000","last_became_active":"2024-09-18T07:35:01.413443+0000","last_became_peered":"2024-09-18T07:35:01.413443+0000","last_unstale":"2024-09-18T07:36:24.821881+0000","last_undegraded":"2024-09-18T07:36:24.821881+0000","last_fullsized":"2024-09-18T07:36:24.821881+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:04.267580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:04.267580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":326,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.816280+0000","last_change":"2024-09-18T07:36:10.551960+0000","last_active":"2024-09-18T07:36:24.816280+0000","last_peered":"2024-09-18T07:36:24.816280+0000","last_clean":"2024-09-18T07:36:24.816280+0000","last_became_active":"2024-09-18T07:35:01.410284+0000","last_became_peered":"2024-09-18T07:35:01.410284+0000","last_unstale":"2024-09-18T07:36:24.816280+0000","last_undegraded":"2024-09-18T07:36:24.816280+0000","last_fullsized":"2024-09-18T07:36:24.816280+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:10.551889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:10.551889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:19.838818+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":316,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.821596+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:36:24.821596+0000","last_peered":"2024-09-18T07:36:24.821596+0000","last_clean":"2024-09-18T07:36:24.821596+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:36:24.821596+0000","last_undegraded":"2024-09-18T07:36:24.821596+0000","last_fullsized":"2024-09-18T07:36:24.821596+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:22.517161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":345,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.806398+0000","last_change":"2024-09-18T07:36:05.903851+0000","last_active":"2024-09-18T07:36:24.806398+0000","last_peered":"2024-09-18T07:36:24.806398+0000","last_clean":"2024-09-18T07:36:24.806398+0000","last_became_active":"2024-09-18T07:36:02.526212+0000","last_became_peered":"2024-09-18T07:36:02.526212+0000","last_unstale":"2024-09-18T07:36:24.806398+0000","last_undegraded":"2024-09-18T07:36:24.806398+0000","last_fullsized":"2024-09-18T07:36:24.806398+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-18T07:36:05.903784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.903784+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:32.483823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":92,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.818762+0000","last_change":"2024-09-18T07:35:01.399995+0000","last_active":"2024-09-18T07:36:24.818762+0000","last_peered":"2024-09-18T07:36:24.818762+0000","last_clean":"2024-09-18T07:36:24.818762+0000","last_became_active":"2024-09-18T07:35:01.398605+0000","last_became_peered":"2024-09-18T07:35:01.398605+0000","last_unstale":"2024-09-18T07:36:24.818762+0000","last_undegraded":"2024-09-18T07:36:24.818762+0000","last_fullsized":"2024-09-18T07:36:24.818762+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:55.843864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:55.843864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:17.209503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":328,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.817853+0000","last_change":"2024-09-18T07:35:35.248892+0000","last_active":"2024-09-18T07:36:24.817853+0000","last_peered":"2024-09-18T07:36:24.817853+0000","last_clean":"2024-09-18T07:36:24.817853+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:36:24.817853+0000","last_undegraded":"2024-09-18T07:36:24.817853+0000","last_fullsized":"2024-09-18T07:36:24.817853+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:35.248824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:35.248824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:41.491237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":322,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.806801+0000","last_change":"2024-09-18T07:35:24.929178+0000","last_active":"2024-09-18T07:36:24.806801+0000","last_peered":"2024-09-18T07:36:24.806801+0000","last_clean":"2024-09-18T07:36:24.806801+0000","last_became_active":"2024-09-18T07:31:02.490014+0000","last_became_peered":"2024-09-18T07:31:02.490014+0000","last_unstale":"2024-09-18T07:36:24.806801+0000","last_undegraded":"2024-09-18T07:36:24.806801+0000","last_fullsized":"2024-09-18T07:36:24.806801+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:24.929124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:24.929124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:40.557387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":346,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819117+0000","last_change":"2024-09-18T07:36:06.523357+0000","last_active":"2024-09-18T07:36:24.819117+0000","last_peered":"2024-09-18T07:36:24.819117+0000","last_clean":"2024-09-18T07:36:24.819117+0000","last_became_active":"2024-09-18T07:35:01.414088+0000","last_became_peered":"2024-09-18T07:35:01.414088+0000","last_unstale":"2024-09-18T07:36:24.819117+0000","last_undegraded":"2024-09-18T07:36:24.819117+0000","last_fullsized":"2024-09-18T07:36:24.819117+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:06.523292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.523292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:20.736765+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":363,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.817218+0000","last_change":"2024-09-18T07:35:31.858126+0000","last_active":"2024-09-18T07:36:24.817218+0000","last_peered":"2024-09-18T07:36:24.817218+0000","last_clean":"2024-09-18T07:36:24.817218+0000","last_became_active":"2024-09-18T07:30:58.316266+0000","last_became_peered":"2024-09-18T07:30:58.316266+0000","last_unstale":"2024-09-18T07:36:24.817218+0000","last_undegraded":"2024-09-18T07:36:24.817218+0000","last_fullsized":"2024-09-18T07:36:24.817218+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:35:31.858040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:31.858040+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:55.124419+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":17,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.806571+0000","last_change":"2024-09-18T07:36:13.116882+0000","last_active":"2024-09-18T07:36:24.806571+0000","last_peered":"2024-09-18T07:36:24.806571+0000","last_clean":"2024-09-18T07:36:24.806571+0000","last_became_active":"2024-09-18T07:36:13.116416+0000","last_became_peered":"2024-09-18T07:36:13.116416+0000","last_unstale":"2024-09-18T07:36:24.806571+0000","last_undegraded":"2024-09-18T07:36:24.806571+0000","last_fullsized":"2024-09-18T07:36:24.806571+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:10.074367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:10.074367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:27.333633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":321,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.823199+0000","last_change":"2024-09-18T07:35:34.992612+0000","last_active":"2024-09-18T07:36:24.823199+0000","last_peered":"2024-09-18T07:36:24.823199+0000","last_clean":"2024-09-18T07:36:24.823199+0000","last_became_active":"2024-09-18T07:31:02.492348+0000","last_became_peered":"2024-09-18T07:31:02.492348+0000","last_unstale":"2024-09-18T07:36:24.823199+0000","last_undegraded":"2024-09-18T07:36:24.823199+0000","last_fullsized":"2024-09-18T07:36:24.823199+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:34.992350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:34.992350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:58.140266+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":334,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.807249+0000","last_change":"2024-09-18T07:35:37.941957+0000","last_active":"2024-09-18T07:36:24.807249+0000","last_peered":"2024-09-18T07:36:24.807249+0000","last_clean":"2024-09-18T07:36:24.807249+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:36:24.807249+0000","last_undegraded":"2024-09-18T07:36:24.807249+0000","last_fullsized":"2024-09-18T07:36:24.807249+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:37.941908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:37.941908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:49.652205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":336,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.818592+0000","last_change":"2024-09-18T07:30:58.319115+0000","last_active":"2024-09-18T07:36:24.818592+0000","last_peered":"2024-09-18T07:36:24.818592+0000","last_clean":"2024-09-18T07:36:24.818592+0000","last_became_active":"2024-09-18T07:30:58.316380+0000","last_became_peered":"2024-09-18T07:30:58.316380+0000","last_unstale":"2024-09-18T07:36:24.818592+0000","last_undegraded":"2024-09-18T07:36:24.818592+0000","last_fullsized":"2024-09-18T07:36:24.818592+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:45.875022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":310,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.806984+0000","last_change":"2024-09-18T07:31:00.334313+0000","last_active":"2024-09-18T07:36:24.806984+0000","last_peered":"2024-09-18T07:36:24.806984+0000","last_clean":"2024-09-18T07:36:24.806984+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:36:24.806984+0000","last_undegraded":"2024-09-18T07:36:24.806984+0000","last_fullsized":"2024-09-18T07:36:24.806984+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:50.025266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":104,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819329+0000","last_change":"2024-09-18T07:35:30.254797+0000","last_active":"2024-09-18T07:36:24.819329+0000","last_peered":"2024-09-18T07:36:24.819329+0000","last_clean":"2024-09-18T07:36:24.819329+0000","last_became_active":"2024-09-18T07:35:01.413110+0000","last_became_peered":"2024-09-18T07:35:01.413110+0000","last_unstale":"2024-09-18T07:36:24.819329+0000","last_undegraded":"2024-09-18T07:36:24.819329+0000","last_fullsized":"2024-09-18T07:36:24.819329+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:35:30.254725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:30.254725+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:43.829889+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":26,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.823901+0000","last_change":"2024-09-18T07:36:13.116657+0000","last_active":"2024-09-18T07:36:24.823901+0000","last_peered":"2024-09-18T07:36:24.823901+0000","last_clean":"2024-09-18T07:36:24.823901+0000","last_became_active":"2024-09-18T07:36:13.116295+0000","last_became_peered":"2024-09-18T07:36:13.116295+0000","last_unstale":"2024-09-18T07:36:24.823901+0000","last_undegraded":"2024-09-18T07:36:24.823901+0000","last_fullsized":"2024-09-18T07:36:24.823901+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:32.976129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.976129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:20.140300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":381,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.816984+0000","last_change":"2024-09-18T07:35:57.488019+0000","last_active":"2024-09-18T07:36:24.816984+0000","last_peered":"2024-09-18T07:36:24.816984+0000","last_clean":"2024-09-18T07:36:24.816984+0000","last_became_active":"2024-09-18T07:35:01.410926+0000","last_became_peered":"2024-09-18T07:35:01.410926+0000","last_unstale":"2024-09-18T07:36:24.816984+0000","last_undegraded":"2024-09-18T07:36:24.816984+0000","last_fullsized":"2024-09-18T07:36:24.816984+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:35:57.487960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.487960+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:14.891707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"173'10","reported_seq":424,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.818568+0000","last_change":"2024-09-18T07:35:29.400204+0000","last_active":"2024-09-18T07:36:24.818568+0000","last_peered":"2024-09-18T07:36:24.818568+0000","last_clean":"2024-09-18T07:36:24.818568+0000","last_became_active":"2024-09-18T07:35:01.407926+0000","last_became_peered":"2024-09-18T07:35:01.407926+0000","last_unstale":"2024-09-18T07:36:24.818568+0000","last_undegraded":"2024-09-18T07:36:24.818568+0000","last_fullsized":"2024-09-18T07:36:24.818568+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:35:29.400137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:29.400137+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:53.614676+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"213'2787","reported_seq":2835,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:26.459123+0000","last_change":"2024-09-18T07:36:13.140008+0000","last_active":"2024-09-18T07:36:26.459123+0000","last_peered":"2024-09-18T07:36:26.459123+0000","last_clean":"2024-09-18T07:36:26.459123+0000","last_became_active":"2024-09-18T07:36:13.118725+0000","last_became_peered":"2024-09-18T07:36:13.118725+0000","last_unstale":"2024-09-18T07:36:26.459123+0000","last_undegraded":"2024-09-18T07:36:26.459123+0000","last_fullsized":"2024-09-18T07:36:26.459123+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":2787,"log_dups_size":0,"ondisk_log_size":2787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:09.515132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182648832,"num_objects":2787,"num_object_clones":0,"num_object_copies":8361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2787,"num_write_kb":178368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":62,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.823457+0000","last_change":"2024-09-18T07:36:13.120127+0000","last_active":"2024-09-18T07:36:24.823457+0000","last_peered":"2024-09-18T07:36:24.823457+0000","last_clean":"2024-09-18T07:36:24.823457+0000","last_became_active":"2024-09-18T07:36:13.119098+0000","last_became_peered":"2024-09-18T07:36:13.119098+0000","last_unstale":"2024-09-18T07:36:24.823457+0000","last_undegraded":"2024-09-18T07:36:24.823457+0000","last_fullsized":"2024-09-18T07:36:24.823457+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:34:37.067162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:37.067162+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:23.103955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":355,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.823943+0000","last_change":"2024-09-18T07:36:01.988099+0000","last_active":"2024-09-18T07:36:24.823943+0000","last_peered":"2024-09-18T07:36:24.823943+0000","last_clean":"2024-09-18T07:36:24.823943+0000","last_became_active":"2024-09-18T07:35:01.407339+0000","last_became_peered":"2024-09-18T07:35:01.407339+0000","last_unstale":"2024-09-18T07:36:24.823943+0000","last_undegraded":"2024-09-18T07:36:24.823943+0000","last_fullsized":"2024-09-18T07:36:24.823943+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:01.988054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.988054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:13.291042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":401,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.806955+0000","last_change":"2024-09-18T07:35:29.961861+0000","last_active":"2024-09-18T07:36:24.806955+0000","last_peered":"2024-09-18T07:36:24.806955+0000","last_clean":"2024-09-18T07:36:24.806955+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:36:24.806955+0000","last_undegraded":"2024-09-18T07:36:24.806955+0000","last_fullsized":"2024-09-18T07:36:24.806955+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:35:29.961791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:29.961791+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:30.708743+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":320,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.821155+0000","last_change":"2024-09-18T07:36:07.588483+0000","last_active":"2024-09-18T07:36:24.821155+0000","last_peered":"2024-09-18T07:36:24.821155+0000","last_clean":"2024-09-18T07:36:24.821155+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:36:24.821155+0000","last_undegraded":"2024-09-18T07:36:24.821155+0000","last_fullsized":"2024-09-18T07:36:24.821155+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:07.588440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:07.588440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:07.900199+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"213'2721","reported_seq":2756,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:26.807137+0000","last_change":"2024-09-18T07:35:59.286428+0000","last_active":"2024-09-18T07:36:26.807137+0000","last_peered":"2024-09-18T07:36:26.807137+0000","last_clean":"2024-09-18T07:36:26.807137+0000","last_became_active":"2024-09-18T07:35:59.286120+0000","last_became_peered":"2024-09-18T07:35:59.286120+0000","last_unstale":"2024-09-18T07:36:26.807137+0000","last_undegraded":"2024-09-18T07:36:26.807137+0000","last_fullsized":"2024-09-18T07:36:26.807137+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":2721,"log_dups_size":0,"ondisk_log_size":2721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:23.474401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178323456,"num_objects":2721,"num_object_clones":0,"num_object_copies":8163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2721,"num_write_kb":174144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":411,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.823717+0000","last_change":"2024-09-18T07:35:23.983838+0000","last_active":"2024-09-18T07:36:24.823717+0000","last_peered":"2024-09-18T07:36:24.823717+0000","last_clean":"2024-09-18T07:36:24.823717+0000","last_became_active":"2024-09-18T07:35:01.418909+0000","last_became_peered":"2024-09-18T07:35:01.418909+0000","last_unstale":"2024-09-18T07:36:24.823717+0000","last_undegraded":"2024-09-18T07:36:24.823717+0000","last_fullsized":"2024-09-18T07:36:24.823717+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:35:23.983762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:23.983762+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:40.277352+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":347,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.823824+0000","last_change":"2024-09-18T07:35:27.922013+0000","last_active":"2024-09-18T07:36:24.823824+0000","last_peered":"2024-09-18T07:36:24.823824+0000","last_clean":"2024-09-18T07:36:24.823824+0000","last_became_active":"2024-09-18T07:35:01.407422+0000","last_became_peered":"2024-09-18T07:35:01.407422+0000","last_unstale":"2024-09-18T07:36:24.823824+0000","last_undegraded":"2024-09-18T07:36:24.823824+0000","last_fullsized":"2024-09-18T07:36:24.823824+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:27.921887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:27.921887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:45.772375+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":332,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.806321+0000","last_change":"2024-09-18T07:35:01.401507+0000","last_active":"2024-09-18T07:36:24.806321+0000","last_peered":"2024-09-18T07:36:24.806321+0000","last_clean":"2024-09-18T07:36:24.806321+0000","last_became_active":"2024-09-18T07:35:01.401024+0000","last_became_peered":"2024-09-18T07:35:01.401024+0000","last_unstale":"2024-09-18T07:36:24.806321+0000","last_undegraded":"2024-09-18T07:36:24.806321+0000","last_fullsized":"2024-09-18T07:36:24.806321+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:54.808785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:54.808785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":306,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.807067+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:36:24.807067+0000","last_peered":"2024-09-18T07:36:24.807067+0000","last_clean":"2024-09-18T07:36:24.807067+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:36:24.807067+0000","last_undegraded":"2024-09-18T07:36:24.807067+0000","last_fullsized":"2024-09-18T07:36:24.807067+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:15.921541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"213'2726","reported_seq":2761,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:26.457884+0000","last_change":"2024-09-18T07:35:59.287014+0000","last_active":"2024-09-18T07:36:26.457884+0000","last_peered":"2024-09-18T07:36:26.457884+0000","last_clean":"2024-09-18T07:36:26.457884+0000","last_became_active":"2024-09-18T07:35:59.286688+0000","last_became_peered":"2024-09-18T07:35:59.286688+0000","last_unstale":"2024-09-18T07:36:26.457884+0000","last_undegraded":"2024-09-18T07:36:26.457884+0000","last_fullsized":"2024-09-18T07:36:26.457884+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":2726,"log_dups_size":0,"ondisk_log_size":2726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:25.089038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178651136,"num_objects":2726,"num_object_clones":0,"num_object_copies":8178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2726,"num_write_kb":174464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":343,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.806330+0000","last_change":"2024-09-18T07:35:00.882253+0000","last_active":"2024-09-18T07:36:24.806330+0000","last_peered":"2024-09-18T07:36:24.806330+0000","last_clean":"2024-09-18T07:36:24.806330+0000","last_became_active":"2024-09-18T07:30:58.309649+0000","last_became_peered":"2024-09-18T07:30:58.309649+0000","last_unstale":"2024-09-18T07:36:24.806330+0000","last_undegraded":"2024-09-18T07:36:24.806330+0000","last_fullsized":"2024-09-18T07:36:24.806330+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-18T07:35:00.882191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.882191+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":325,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.817228+0000","last_change":"2024-09-18T07:33:24.213432+0000","last_active":"2024-09-18T07:36:24.817228+0000","last_peered":"2024-09-18T07:36:24.817228+0000","last_clean":"2024-09-18T07:36:24.817228+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:36:24.817228+0000","last_undegraded":"2024-09-18T07:36:24.817228+0000","last_fullsized":"2024-09-18T07:36:24.817228+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:24.213356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:24.213356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:46.472223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":400,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.823550+0000","last_change":"2024-09-18T07:36:05.016008+0000","last_active":"2024-09-18T07:36:24.823550+0000","last_peered":"2024-09-18T07:36:24.823550+0000","last_clean":"2024-09-18T07:36:24.823550+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:36:24.823550+0000","last_undegraded":"2024-09-18T07:36:24.823550+0000","last_fullsized":"2024-09-18T07:36:24.823550+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:05.015955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.015955+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:27.419964+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":100,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819117+0000","last_change":"2024-09-18T07:35:25.287397+0000","last_active":"2024-09-18T07:36:24.819117+0000","last_peered":"2024-09-18T07:36:24.819117+0000","last_clean":"2024-09-18T07:36:24.819117+0000","last_became_active":"2024-09-18T07:35:01.393846+0000","last_became_peered":"2024-09-18T07:35:01.393846+0000","last_unstale":"2024-09-18T07:36:24.819117+0000","last_undegraded":"2024-09-18T07:36:24.819117+0000","last_fullsized":"2024-09-18T07:36:24.819117+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:25.287352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:25.287352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:44.735842+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"213'2866","reported_seq":2901,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:26.838182+0000","last_change":"2024-09-18T07:35:59.102118+0000","last_active":"2024-09-18T07:36:26.838182+0000","last_peered":"2024-09-18T07:36:26.838182+0000","last_clean":"2024-09-18T07:36:26.838182+0000","last_became_active":"2024-09-18T07:35:59.101681+0000","last_became_peered":"2024-09-18T07:35:59.101681+0000","last_unstale":"2024-09-18T07:36:26.838182+0000","last_undegraded":"2024-09-18T07:36:26.838182+0000","last_fullsized":"2024-09-18T07:36:26.838182+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":2866,"log_dups_size":0,"ondisk_log_size":2866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:05.983334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187826176,"num_objects":2866,"num_object_clones":0,"num_object_copies":8598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2866,"num_write_kb":183424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":52,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819459+0000","last_change":"2024-09-18T07:36:13.119192+0000","last_active":"2024-09-18T07:36:24.819459+0000","last_peered":"2024-09-18T07:36:24.819459+0000","last_clean":"2024-09-18T07:36:24.819459+0000","last_became_active":"2024-09-18T07:36:13.118876+0000","last_became_peered":"2024-09-18T07:36:13.118876+0000","last_unstale":"2024-09-18T07:36:24.819459+0000","last_undegraded":"2024-09-18T07:36:24.819459+0000","last_fullsized":"2024-09-18T07:36:24.819459+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:13.485705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":332,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.821876+0000","last_change":"2024-09-18T07:35:24.210457+0000","last_active":"2024-09-18T07:36:24.821876+0000","last_peered":"2024-09-18T07:36:24.821876+0000","last_clean":"2024-09-18T07:36:24.821876+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:36:24.821876+0000","last_undegraded":"2024-09-18T07:36:24.821876+0000","last_fullsized":"2024-09-18T07:36:24.821876+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:24.210401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:24.210401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:42.826106+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":399,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.817372+0000","last_change":"2024-09-18T07:36:05.525816+0000","last_active":"2024-09-18T07:36:24.817372+0000","last_peered":"2024-09-18T07:36:24.817372+0000","last_clean":"2024-09-18T07:36:24.817372+0000","last_became_active":"2024-09-18T07:31:00.336905+0000","last_became_peered":"2024-09-18T07:31:00.336905+0000","last_unstale":"2024-09-18T07:36:24.817372+0000","last_undegraded":"2024-09-18T07:36:24.817372+0000","last_fullsized":"2024-09-18T07:36:24.817372+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:05.525762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.525762+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:17.875375+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":304,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.816433+0000","last_change":"2024-09-18T07:31:02.348538+0000","last_active":"2024-09-18T07:36:24.816433+0000","last_peered":"2024-09-18T07:36:24.816433+0000","last_clean":"2024-09-18T07:36:24.816433+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:36:24.816433+0000","last_undegraded":"2024-09-18T07:36:24.816433+0000","last_fullsized":"2024-09-18T07:36:24.816433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:44.477539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"213'2840","reported_seq":2875,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:26.678813+0000","last_change":"2024-09-18T07:35:59.102330+0000","last_active":"2024-09-18T07:36:26.678813+0000","last_peered":"2024-09-18T07:36:26.678813+0000","last_clean":"2024-09-18T07:36:26.678813+0000","last_became_active":"2024-09-18T07:35:59.101668+0000","last_became_peered":"2024-09-18T07:35:59.101668+0000","last_unstale":"2024-09-18T07:36:26.678813+0000","last_undegraded":"2024-09-18T07:36:26.678813+0000","last_fullsized":"2024-09-18T07:36:26.678813+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":2840,"log_dups_size":0,"ondisk_log_size":2840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:08.905772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186122240,"num_objects":2840,"num_object_clones":0,"num_object_copies":8520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2840,"num_write_kb":181760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":414,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.818886+0000","last_change":"2024-09-18T07:36:09.498239+0000","last_active":"2024-09-18T07:36:24.818886+0000","last_peered":"2024-09-18T07:36:24.818886+0000","last_clean":"2024-09-18T07:36:24.818886+0000","last_became_active":"2024-09-18T07:35:01.415275+0000","last_became_peered":"2024-09-18T07:35:01.415275+0000","last_unstale":"2024-09-18T07:36:24.818886+0000","last_undegraded":"2024-09-18T07:36:24.818886+0000","last_fullsized":"2024-09-18T07:36:24.818886+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:36:09.498142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.498142+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:35.878494+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.817313+0000","last_change":"2024-09-18T07:36:13.118833+0000","last_active":"2024-09-18T07:36:24.817313+0000","last_peered":"2024-09-18T07:36:24.817313+0000","last_clean":"2024-09-18T07:36:24.817313+0000","last_became_active":"2024-09-18T07:36:13.118553+0000","last_became_peered":"2024-09-18T07:36:13.118553+0000","last_unstale":"2024-09-18T07:36:24.817313+0000","last_undegraded":"2024-09-18T07:36:24.817313+0000","last_fullsized":"2024-09-18T07:36:24.817313+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:13.481723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":316,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.816827+0000","last_change":"2024-09-18T07:33:21.834910+0000","last_active":"2024-09-18T07:36:24.816827+0000","last_peered":"2024-09-18T07:36:24.816827+0000","last_clean":"2024-09-18T07:36:24.816827+0000","last_became_active":"2024-09-18T07:31:00.334776+0000","last_became_peered":"2024-09-18T07:31:00.334776+0000","last_unstale":"2024-09-18T07:36:24.816827+0000","last_undegraded":"2024-09-18T07:36:24.816827+0000","last_fullsized":"2024-09-18T07:36:24.816827+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:48.284140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":17,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.822731+0000","last_change":"2024-09-18T07:36:13.117081+0000","last_active":"2024-09-18T07:36:24.822731+0000","last_peered":"2024-09-18T07:36:24.822731+0000","last_clean":"2024-09-18T07:36:24.822731+0000","last_became_active":"2024-09-18T07:36:13.116718+0000","last_became_peered":"2024-09-18T07:36:13.116718+0000","last_unstale":"2024-09-18T07:36:24.822731+0000","last_undegraded":"2024-09-18T07:36:24.822731+0000","last_fullsized":"2024-09-18T07:36:24.822731+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:13.224693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"213'2887","reported_seq":2921,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:26.427623+0000","last_change":"2024-09-18T07:35:59.288477+0000","last_active":"2024-09-18T07:36:26.427623+0000","last_peered":"2024-09-18T07:36:26.427623+0000","last_clean":"2024-09-18T07:36:26.427623+0000","last_became_active":"2024-09-18T07:35:59.288058+0000","last_became_peered":"2024-09-18T07:35:59.288058+0000","last_unstale":"2024-09-18T07:36:26.427623+0000","last_undegraded":"2024-09-18T07:36:26.427623+0000","last_fullsized":"2024-09-18T07:36:26.427623+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":2887,"log_dups_size":0,"ondisk_log_size":2887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:58.502273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189202432,"num_objects":2887,"num_object_clones":0,"num_object_copies":8661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2887,"num_write_kb":184768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"211'29","reported_seq":90,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.817108+0000","last_change":"2024-09-18T07:36:13.115630+0000","last_active":"2024-09-18T07:36:24.817108+0000","last_peered":"2024-09-18T07:36:24.817108+0000","last_clean":"2024-09-18T07:36:24.817108+0000","last_became_active":"2024-09-18T07:36:13.115278+0000","last_became_peered":"2024-09-18T07:36:13.115278+0000","last_unstale":"2024-09-18T07:36:24.817108+0000","last_undegraded":"2024-09-18T07:36:24.817108+0000","last_fullsized":"2024-09-18T07:36:24.817108+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'28","last_scrub_stamp":"2024-09-18T07:34:31.955336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.955336+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:23.541188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":326,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.807029+0000","last_change":"2024-09-18T07:36:06.908769+0000","last_active":"2024-09-18T07:36:24.807029+0000","last_peered":"2024-09-18T07:36:24.807029+0000","last_clean":"2024-09-18T07:36:24.807029+0000","last_became_active":"2024-09-18T07:30:56.285032+0000","last_became_peered":"2024-09-18T07:30:56.285032+0000","last_unstale":"2024-09-18T07:36:24.807029+0000","last_undegraded":"2024-09-18T07:36:24.807029+0000","last_fullsized":"2024-09-18T07:36:24.807029+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:06.908713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.908713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:17.531168+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":342,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.820930+0000","last_change":"2024-09-18T07:36:01.529763+0000","last_active":"2024-09-18T07:36:24.820930+0000","last_peered":"2024-09-18T07:36:24.820930+0000","last_clean":"2024-09-18T07:36:24.820930+0000","last_became_active":"2024-09-18T07:30:55.283292+0000","last_became_peered":"2024-09-18T07:30:55.283292+0000","last_unstale":"2024-09-18T07:36:24.820930+0000","last_undegraded":"2024-09-18T07:36:24.820930+0000","last_fullsized":"2024-09-18T07:36:24.820930+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:36:01.529677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.529677+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:13.621751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":308,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.820951+0000","last_change":"2024-09-18T07:31:00.332738+0000","last_active":"2024-09-18T07:36:24.820951+0000","last_peered":"2024-09-18T07:36:24.820951+0000","last_clean":"2024-09-18T07:36:24.820951+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:36:24.820951+0000","last_undegraded":"2024-09-18T07:36:24.820951+0000","last_fullsized":"2024-09-18T07:36:24.820951+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:19.543881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":327,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.822125+0000","last_change":"2024-09-18T07:35:01.419122+0000","last_active":"2024-09-18T07:36:24.822125+0000","last_peered":"2024-09-18T07:36:24.822125+0000","last_clean":"2024-09-18T07:36:24.822125+0000","last_became_active":"2024-09-18T07:35:01.418809+0000","last_became_peered":"2024-09-18T07:35:01.418809+0000","last_unstale":"2024-09-18T07:36:24.822125+0000","last_undegraded":"2024-09-18T07:36:24.822125+0000","last_fullsized":"2024-09-18T07:36:24.822125+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:56.289176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:34:56.289176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"213'2749","reported_seq":2784,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:26.669093+0000","last_change":"2024-09-18T07:35:59.103129+0000","last_active":"2024-09-18T07:36:26.669093+0000","last_peered":"2024-09-18T07:36:26.669093+0000","last_clean":"2024-09-18T07:36:26.669093+0000","last_became_active":"2024-09-18T07:35:59.102853+0000","last_became_peered":"2024-09-18T07:35:59.102853+0000","last_unstale":"2024-09-18T07:36:26.669093+0000","last_undegraded":"2024-09-18T07:36:26.669093+0000","last_fullsized":"2024-09-18T07:36:26.669093+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":2749,"log_dups_size":0,"ondisk_log_size":2749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:59.929794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180158464,"num_objects":2749,"num_object_clones":0,"num_object_copies":8247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2749,"num_write_kb":175936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":64,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.822153+0000","last_change":"2024-09-18T07:36:13.118278+0000","last_active":"2024-09-18T07:36:24.822153+0000","last_peered":"2024-09-18T07:36:24.822153+0000","last_clean":"2024-09-18T07:36:24.822153+0000","last_became_active":"2024-09-18T07:36:13.117924+0000","last_became_peered":"2024-09-18T07:36:13.117924+0000","last_unstale":"2024-09-18T07:36:24.822153+0000","last_undegraded":"2024-09-18T07:36:24.822153+0000","last_fullsized":"2024-09-18T07:36:24.822153+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-18T07:36:06.083531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.083531+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:26.898216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.813445+0000","last_change":"2024-09-18T07:36:13.113998+0000","last_active":"2024-09-18T07:36:24.813445+0000","last_peered":"2024-09-18T07:36:24.813445+0000","last_clean":"2024-09-18T07:36:24.813445+0000","last_became_active":"2024-09-18T07:36:13.113712+0000","last_became_peered":"2024-09-18T07:36:13.113712+0000","last_unstale":"2024-09-18T07:36:24.813445+0000","last_undegraded":"2024-09-18T07:36:24.813445+0000","last_fullsized":"2024-09-18T07:36:24.813445+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:04.050757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:04.050757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:30.942620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":335,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819125+0000","last_change":"2024-09-18T07:32:02.418590+0000","last_active":"2024-09-18T07:36:24.819125+0000","last_peered":"2024-09-18T07:36:24.819125+0000","last_clean":"2024-09-18T07:36:24.819125+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:36:24.819125+0000","last_undegraded":"2024-09-18T07:36:24.819125+0000","last_fullsized":"2024-09-18T07:36:24.819125+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:32:02.418507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.418507+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:46.713520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":324,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.816048+0000","last_change":"2024-09-18T07:35:01.403942+0000","last_active":"2024-09-18T07:36:24.816048+0000","last_peered":"2024-09-18T07:36:24.816048+0000","last_clean":"2024-09-18T07:36:24.816048+0000","last_became_active":"2024-09-18T07:35:01.403680+0000","last_became_peered":"2024-09-18T07:35:01.403680+0000","last_unstale":"2024-09-18T07:36:24.816048+0000","last_undegraded":"2024-09-18T07:36:24.816048+0000","last_fullsized":"2024-09-18T07:36:24.816048+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:15.475161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":96,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819083+0000","last_change":"2024-09-18T07:35:01.441600+0000","last_active":"2024-09-18T07:36:24.819083+0000","last_peered":"2024-09-18T07:36:24.819083+0000","last_clean":"2024-09-18T07:36:24.819083+0000","last_became_active":"2024-09-18T07:35:01.406712+0000","last_became_peered":"2024-09-18T07:35:01.406712+0000","last_unstale":"2024-09-18T07:36:24.819083+0000","last_undegraded":"2024-09-18T07:36:24.819083+0000","last_fullsized":"2024-09-18T07:36:24.819083+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:57.655797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.655797+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:54.157555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"213'2788","reported_seq":2829,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:26.594106+0000","last_change":"2024-09-18T07:36:13.122322+0000","last_active":"2024-09-18T07:36:26.594106+0000","last_peered":"2024-09-18T07:36:26.594106+0000","last_clean":"2024-09-18T07:36:26.594106+0000","last_became_active":"2024-09-18T07:36:13.119156+0000","last_became_peered":"2024-09-18T07:36:13.119156+0000","last_unstale":"2024-09-18T07:36:26.594106+0000","last_undegraded":"2024-09-18T07:36:26.594106+0000","last_fullsized":"2024-09-18T07:36:26.594106+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":2788,"log_dups_size":0,"ondisk_log_size":2788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:25.854791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182714368,"num_objects":2788,"num_object_clones":0,"num_object_copies":8364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2788,"num_write_kb":178432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":351,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.817048+0000","last_change":"2024-09-18T07:33:27.040227+0000","last_active":"2024-09-18T07:36:24.817048+0000","last_peered":"2024-09-18T07:36:24.817048+0000","last_clean":"2024-09-18T07:36:24.817048+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:36:24.817048+0000","last_undegraded":"2024-09-18T07:36:24.817048+0000","last_fullsized":"2024-09-18T07:36:24.817048+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:15.592642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":322,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.816309+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:36:24.816309+0000","last_peered":"2024-09-18T07:36:24.816309+0000","last_clean":"2024-09-18T07:36:24.816309+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:36:24.816309+0000","last_undegraded":"2024-09-18T07:36:24.816309+0000","last_fullsized":"2024-09-18T07:36:24.816309+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:12.598176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":181,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819247+0000","last_change":"2024-09-18T07:35:01.421692+0000","last_active":"2024-09-18T07:36:24.819247+0000","last_peered":"2024-09-18T07:36:24.819247+0000","last_clean":"2024-09-18T07:36:24.819247+0000","last_became_active":"2024-09-18T07:35:01.400524+0000","last_became_peered":"2024-09-18T07:35:01.400524+0000","last_unstale":"2024-09-18T07:36:24.819247+0000","last_undegraded":"2024-09-18T07:36:24.819247+0000","last_fullsized":"2024-09-18T07:36:24.819247+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:21.233017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":91,"num_read_kb":86,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":316,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.822232+0000","last_change":"2024-09-18T07:33:27.394247+0000","last_active":"2024-09-18T07:36:24.822232+0000","last_peered":"2024-09-18T07:36:24.822232+0000","last_clean":"2024-09-18T07:36:24.822232+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:36:24.822232+0000","last_undegraded":"2024-09-18T07:36:24.822232+0000","last_fullsized":"2024-09-18T07:36:24.822232+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.394203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.394203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":17,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.821700+0000","last_change":"2024-09-18T07:36:13.118041+0000","last_active":"2024-09-18T07:36:24.821700+0000","last_peered":"2024-09-18T07:36:24.821700+0000","last_clean":"2024-09-18T07:36:24.821700+0000","last_became_active":"2024-09-18T07:36:13.117463+0000","last_became_peered":"2024-09-18T07:36:13.117463+0000","last_unstale":"2024-09-18T07:36:24.821700+0000","last_undegraded":"2024-09-18T07:36:24.821700+0000","last_fullsized":"2024-09-18T07:36:24.821700+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:58.079941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:34:58.079941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"213'2815","reported_seq":2850,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:26.470364+0000","last_change":"2024-09-18T07:35:59.107879+0000","last_active":"2024-09-18T07:36:26.470364+0000","last_peered":"2024-09-18T07:36:26.470364+0000","last_clean":"2024-09-18T07:36:26.470364+0000","last_became_active":"2024-09-18T07:35:59.107585+0000","last_became_peered":"2024-09-18T07:35:59.107585+0000","last_unstale":"2024-09-18T07:36:26.470364+0000","last_undegraded":"2024-09-18T07:36:26.470364+0000","last_fullsized":"2024-09-18T07:36:26.470364+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":2815,"log_dups_size":0,"ondisk_log_size":2815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:00.730670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184483840,"num_objects":2815,"num_object_clones":0,"num_object_copies":8445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2815,"num_write_kb":180160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"173'18","reported_seq":467,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.818259+0000","last_change":"2024-09-18T07:35:01.444206+0000","last_active":"2024-09-18T07:36:24.818259+0000","last_peered":"2024-09-18T07:36:24.818259+0000","last_clean":"2024-09-18T07:36:24.818259+0000","last_became_active":"2024-09-18T07:35:01.409012+0000","last_became_peered":"2024-09-18T07:35:01.409012+0000","last_unstale":"2024-09-18T07:36:24.818259+0000","last_undegraded":"2024-09-18T07:36:24.818259+0000","last_fullsized":"2024-09-18T07:36:24.818259+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'17","last_scrub_stamp":"2024-09-18T07:33:33.849285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.849285+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:14.242526+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":325,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.818399+0000","last_change":"2024-09-18T07:31:59.617046+0000","last_active":"2024-09-18T07:36:24.818399+0000","last_peered":"2024-09-18T07:36:24.818399+0000","last_clean":"2024-09-18T07:36:24.818399+0000","last_became_active":"2024-09-18T07:30:56.288149+0000","last_became_peered":"2024-09-18T07:30:56.288149+0000","last_unstale":"2024-09-18T07:36:24.818399+0000","last_undegraded":"2024-09-18T07:36:24.818399+0000","last_fullsized":"2024-09-18T07:36:24.818399+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:59.616988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:59.616988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:17.092605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":106,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819302+0000","last_change":"2024-09-18T07:35:27.278161+0000","last_active":"2024-09-18T07:36:24.819302+0000","last_peered":"2024-09-18T07:36:24.819302+0000","last_clean":"2024-09-18T07:36:24.819302+0000","last_became_active":"2024-09-18T07:35:01.400273+0000","last_became_peered":"2024-09-18T07:35:01.400273+0000","last_unstale":"2024-09-18T07:36:24.819302+0000","last_undegraded":"2024-09-18T07:36:24.819302+0000","last_fullsized":"2024-09-18T07:36:24.819302+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:35:27.278095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:35:27.278095+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:51.801152+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":92,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819438+0000","last_change":"2024-09-18T07:35:01.413545+0000","last_active":"2024-09-18T07:36:24.819438+0000","last_peered":"2024-09-18T07:36:24.819438+0000","last_clean":"2024-09-18T07:36:24.819438+0000","last_became_active":"2024-09-18T07:35:01.412217+0000","last_became_peered":"2024-09-18T07:35:01.412217+0000","last_unstale":"2024-09-18T07:36:24.819438+0000","last_undegraded":"2024-09-18T07:36:24.819438+0000","last_fullsized":"2024-09-18T07:36:24.819438+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:36.964421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:36.964421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:55.205380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":312,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.821985+0000","last_change":"2024-09-18T07:33:22.358792+0000","last_active":"2024-09-18T07:36:24.821985+0000","last_peered":"2024-09-18T07:36:24.821985+0000","last_clean":"2024-09-18T07:36:24.821985+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:36:24.821985+0000","last_undegraded":"2024-09-18T07:36:24.821985+0000","last_fullsized":"2024-09-18T07:36:24.821985+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.358746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.358746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"213'2784","reported_seq":2819,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:26.449079+0000","last_change":"2024-09-18T07:35:59.106988+0000","last_active":"2024-09-18T07:36:26.449079+0000","last_peered":"2024-09-18T07:36:26.449079+0000","last_clean":"2024-09-18T07:36:26.449079+0000","last_became_active":"2024-09-18T07:35:59.106659+0000","last_became_peered":"2024-09-18T07:35:59.106659+0000","last_unstale":"2024-09-18T07:36:26.449079+0000","last_undegraded":"2024-09-18T07:36:26.449079+0000","last_fullsized":"2024-09-18T07:36:26.449079+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":2784,"log_dups_size":0,"ondisk_log_size":2784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:06.637813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182452224,"num_objects":2784,"num_object_clones":0,"num_object_copies":8352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2784,"num_write_kb":178176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":368,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.807009+0000","last_change":"2024-09-18T07:35:38.928479+0000","last_active":"2024-09-18T07:36:24.807009+0000","last_peered":"2024-09-18T07:36:24.807009+0000","last_clean":"2024-09-18T07:36:24.807009+0000","last_became_active":"2024-09-18T07:32:26.785711+0000","last_became_peered":"2024-09-18T07:32:26.785711+0000","last_unstale":"2024-09-18T07:36:24.807009+0000","last_undegraded":"2024-09-18T07:36:24.807009+0000","last_fullsized":"2024-09-18T07:36:24.807009+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:35:38.928407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:38.928407+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:58.999748+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":340,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.806602+0000","last_change":"2024-09-18T07:35:03.866215+0000","last_active":"2024-09-18T07:36:24.806602+0000","last_peered":"2024-09-18T07:36:24.806602+0000","last_clean":"2024-09-18T07:36:24.806602+0000","last_became_active":"2024-09-18T07:35:01.401063+0000","last_became_peered":"2024-09-18T07:35:01.401063+0000","last_unstale":"2024-09-18T07:36:24.806602+0000","last_undegraded":"2024-09-18T07:36:24.806602+0000","last_fullsized":"2024-09-18T07:36:24.806602+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:03.866148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:03.866148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:25.066815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":324,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.821878+0000","last_change":"2024-09-18T07:35:31.191601+0000","last_active":"2024-09-18T07:36:24.821878+0000","last_peered":"2024-09-18T07:36:24.821878+0000","last_clean":"2024-09-18T07:36:24.821878+0000","last_became_active":"2024-09-18T07:31:00.336172+0000","last_became_peered":"2024-09-18T07:31:00.336172+0000","last_unstale":"2024-09-18T07:36:24.821878+0000","last_undegraded":"2024-09-18T07:36:24.821878+0000","last_fullsized":"2024-09-18T07:36:24.821878+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:31.191528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:31.191528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:38.261888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":321,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819283+0000","last_change":"2024-09-18T07:35:38.955492+0000","last_active":"2024-09-18T07:36:24.819283+0000","last_peered":"2024-09-18T07:36:24.819283+0000","last_clean":"2024-09-18T07:36:24.819283+0000","last_became_active":"2024-09-18T07:31:02.490953+0000","last_became_peered":"2024-09-18T07:31:02.490953+0000","last_unstale":"2024-09-18T07:36:24.819283+0000","last_undegraded":"2024-09-18T07:36:24.819283+0000","last_fullsized":"2024-09-18T07:36:24.819283+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:38.955353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:38.955353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:43.588886+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"213'2829","reported_seq":2870,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:26.846048+0000","last_change":"2024-09-18T07:36:13.120887+0000","last_active":"2024-09-18T07:36:26.846048+0000","last_peered":"2024-09-18T07:36:26.846048+0000","last_clean":"2024-09-18T07:36:26.846048+0000","last_became_active":"2024-09-18T07:36:13.116701+0000","last_became_peered":"2024-09-18T07:36:13.116701+0000","last_unstale":"2024-09-18T07:36:26.846048+0000","last_undegraded":"2024-09-18T07:36:26.846048+0000","last_fullsized":"2024-09-18T07:36:26.846048+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":2829,"log_dups_size":0,"ondisk_log_size":2829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:03.571387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185401344,"num_objects":2829,"num_object_clones":0,"num_object_copies":8487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2829,"num_write_kb":181056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":358,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.806874+0000","last_change":"2024-09-18T07:35:33.011021+0000","last_active":"2024-09-18T07:36:24.806874+0000","last_peered":"2024-09-18T07:36:24.806874+0000","last_clean":"2024-09-18T07:36:24.806874+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:36:24.806874+0000","last_undegraded":"2024-09-18T07:36:24.806874+0000","last_fullsized":"2024-09-18T07:36:24.806874+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:35:33.010955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:33.010955+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:40.982191+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":332,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.818552+0000","last_change":"2024-09-18T07:35:35.265329+0000","last_active":"2024-09-18T07:36:24.818552+0000","last_peered":"2024-09-18T07:36:24.818552+0000","last_clean":"2024-09-18T07:36:24.818552+0000","last_became_active":"2024-09-18T07:30:56.284020+0000","last_became_peered":"2024-09-18T07:30:56.284020+0000","last_unstale":"2024-09-18T07:36:24.818552+0000","last_undegraded":"2024-09-18T07:36:24.818552+0000","last_fullsized":"2024-09-18T07:36:24.818552+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:35.265259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:35.265259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:51.107776+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":17,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.806814+0000","last_change":"2024-09-18T07:36:13.116901+0000","last_active":"2024-09-18T07:36:24.806814+0000","last_peered":"2024-09-18T07:36:24.806814+0000","last_clean":"2024-09-18T07:36:24.806814+0000","last_became_active":"2024-09-18T07:36:13.116527+0000","last_became_peered":"2024-09-18T07:36:13.116527+0000","last_unstale":"2024-09-18T07:36:24.806814+0000","last_undegraded":"2024-09-18T07:36:24.806814+0000","last_fullsized":"2024-09-18T07:36:24.806814+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:05.239160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.239160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":322,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.814177+0000","last_change":"2024-09-18T07:35:25.866286+0000","last_active":"2024-09-18T07:36:24.814177+0000","last_peered":"2024-09-18T07:36:24.814177+0000","last_clean":"2024-09-18T07:36:24.814177+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:36:24.814177+0000","last_undegraded":"2024-09-18T07:36:24.814177+0000","last_fullsized":"2024-09-18T07:36:24.814177+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:25.866225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:25.866225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:35.156540+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"213'2829","reported_seq":2864,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:26.679210+0000","last_change":"2024-09-18T07:35:59.107531+0000","last_active":"2024-09-18T07:36:26.679210+0000","last_peered":"2024-09-18T07:36:26.679210+0000","last_clean":"2024-09-18T07:36:26.679210+0000","last_became_active":"2024-09-18T07:35:59.107232+0000","last_became_peered":"2024-09-18T07:35:59.107232+0000","last_unstale":"2024-09-18T07:36:26.679210+0000","last_undegraded":"2024-09-18T07:36:26.679210+0000","last_fullsized":"2024-09-18T07:36:26.679210+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":2829,"log_dups_size":0,"ondisk_log_size":2829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:00.505165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185401344,"num_objects":2829,"num_object_clones":0,"num_object_copies":8487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2829,"num_write_kb":181056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":337,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.807072+0000","last_change":"2024-09-18T07:32:05.000363+0000","last_active":"2024-09-18T07:36:24.807072+0000","last_peered":"2024-09-18T07:36:24.807072+0000","last_clean":"2024-09-18T07:36:24.807072+0000","last_became_active":"2024-09-18T07:30:58.316423+0000","last_became_peered":"2024-09-18T07:30:58.316423+0000","last_unstale":"2024-09-18T07:36:24.807072+0000","last_undegraded":"2024-09-18T07:36:24.807072+0000","last_fullsized":"2024-09-18T07:36:24.807072+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:32:05.000310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.000310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:24.107037+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":325,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.827254+0000","last_change":"2024-09-18T07:32:06.688493+0000","last_active":"2024-09-18T07:36:24.827254+0000","last_peered":"2024-09-18T07:36:24.827254+0000","last_clean":"2024-09-18T07:36:24.827254+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:36:24.827254+0000","last_undegraded":"2024-09-18T07:36:24.827254+0000","last_fullsized":"2024-09-18T07:36:24.827254+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:06.688444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:06.688444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:18.103625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"173'11","reported_seq":426,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.814808+0000","last_change":"2024-09-18T07:35:01.413702+0000","last_active":"2024-09-18T07:36:24.814808+0000","last_peered":"2024-09-18T07:36:24.814808+0000","last_clean":"2024-09-18T07:36:24.814808+0000","last_became_active":"2024-09-18T07:35:01.403765+0000","last_became_peered":"2024-09-18T07:35:01.403765+0000","last_unstale":"2024-09-18T07:36:24.814808+0000","last_undegraded":"2024-09-18T07:36:24.814808+0000","last_fullsized":"2024-09-18T07:36:24.814808+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'10","last_scrub_stamp":"2024-09-18T07:33:33.000874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.000874+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:22.520367+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":314,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.806713+0000","last_change":"2024-09-18T07:35:04.907232+0000","last_active":"2024-09-18T07:36:24.806713+0000","last_peered":"2024-09-18T07:36:24.806713+0000","last_clean":"2024-09-18T07:36:24.806713+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:36:24.806713+0000","last_undegraded":"2024-09-18T07:36:24.806713+0000","last_fullsized":"2024-09-18T07:36:24.806713+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:04.907166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:04.907166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:25.077151+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"213'2921","reported_seq":2963,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:26.637249+0000","last_change":"2024-09-18T07:36:13.119352+0000","last_active":"2024-09-18T07:36:26.637249+0000","last_peered":"2024-09-18T07:36:26.637249+0000","last_clean":"2024-09-18T07:36:26.637249+0000","last_became_active":"2024-09-18T07:36:13.115895+0000","last_became_peered":"2024-09-18T07:36:13.115895+0000","last_unstale":"2024-09-18T07:36:26.637249+0000","last_undegraded":"2024-09-18T07:36:26.637249+0000","last_fullsized":"2024-09-18T07:36:26.637249+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":2921,"log_dups_size":0,"ondisk_log_size":2921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:13.349496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191430656,"num_objects":2921,"num_object_clones":0,"num_object_copies":8763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2921,"num_write_kb":186944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":328,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819202+0000","last_change":"2024-09-18T07:30:58.314424+0000","last_active":"2024-09-18T07:36:24.819202+0000","last_peered":"2024-09-18T07:36:24.819202+0000","last_clean":"2024-09-18T07:36:24.819202+0000","last_became_active":"2024-09-18T07:30:58.314127+0000","last_became_peered":"2024-09-18T07:30:58.314127+0000","last_unstale":"2024-09-18T07:36:24.819202+0000","last_undegraded":"2024-09-18T07:36:24.819202+0000","last_fullsized":"2024-09-18T07:36:24.819202+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:49.469792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":332,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.821798+0000","last_change":"2024-09-18T07:35:21.222584+0000","last_active":"2024-09-18T07:36:24.821798+0000","last_peered":"2024-09-18T07:36:24.821798+0000","last_clean":"2024-09-18T07:36:24.821798+0000","last_became_active":"2024-09-18T07:30:56.288868+0000","last_became_peered":"2024-09-18T07:30:56.288868+0000","last_unstale":"2024-09-18T07:36:24.821798+0000","last_undegraded":"2024-09-18T07:36:24.821798+0000","last_fullsized":"2024-09-18T07:36:24.821798+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:21.222526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.222526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:30.513979+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":318,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.813664+0000","last_change":"2024-09-18T07:34:31.959675+0000","last_active":"2024-09-18T07:36:24.813664+0000","last_peered":"2024-09-18T07:36:24.813664+0000","last_clean":"2024-09-18T07:36:24.813664+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:36:24.813664+0000","last_undegraded":"2024-09-18T07:36:24.813664+0000","last_fullsized":"2024-09-18T07:36:24.813664+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:31.959636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.959636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:53.622193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":334,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.816218+0000","last_change":"2024-09-18T07:36:07.526609+0000","last_active":"2024-09-18T07:36:24.816218+0000","last_peered":"2024-09-18T07:36:24.816218+0000","last_clean":"2024-09-18T07:36:24.816218+0000","last_became_active":"2024-09-18T07:35:01.410890+0000","last_became_peered":"2024-09-18T07:35:01.410890+0000","last_unstale":"2024-09-18T07:36:24.816218+0000","last_undegraded":"2024-09-18T07:36:24.816218+0000","last_fullsized":"2024-09-18T07:36:24.816218+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:07.526536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:07.526536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:07.810175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"213'2837","reported_seq":2872,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:26.793366+0000","last_change":"2024-09-18T07:35:59.286576+0000","last_active":"2024-09-18T07:36:26.793366+0000","last_peered":"2024-09-18T07:36:26.793366+0000","last_clean":"2024-09-18T07:36:26.793366+0000","last_became_active":"2024-09-18T07:35:59.286262+0000","last_became_peered":"2024-09-18T07:35:59.286262+0000","last_unstale":"2024-09-18T07:36:26.793366+0000","last_undegraded":"2024-09-18T07:36:26.793366+0000","last_fullsized":"2024-09-18T07:36:26.793366+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":2837,"log_dups_size":0,"ondisk_log_size":2837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:26.758269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185925632,"num_objects":2837,"num_object_clones":0,"num_object_copies":8511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2837,"num_write_kb":181568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":421,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.818385+0000","last_change":"2024-09-18T07:35:21.344099+0000","last_active":"2024-09-18T07:36:24.818385+0000","last_peered":"2024-09-18T07:36:24.818385+0000","last_clean":"2024-09-18T07:36:24.818385+0000","last_became_active":"2024-09-18T07:35:01.409401+0000","last_became_peered":"2024-09-18T07:35:01.409401+0000","last_unstale":"2024-09-18T07:36:24.818385+0000","last_undegraded":"2024-09-18T07:36:24.818385+0000","last_fullsized":"2024-09-18T07:36:24.818385+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:35:21.344029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.344029+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:44.145569+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":92,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819573+0000","last_change":"2024-09-18T07:35:01.412993+0000","last_active":"2024-09-18T07:36:24.819573+0000","last_peered":"2024-09-18T07:36:24.819573+0000","last_clean":"2024-09-18T07:36:24.819573+0000","last_became_active":"2024-09-18T07:35:01.411821+0000","last_became_peered":"2024-09-18T07:35:01.411821+0000","last_unstale":"2024-09-18T07:36:24.819573+0000","last_undegraded":"2024-09-18T07:36:24.819573+0000","last_fullsized":"2024-09-18T07:36:24.819573+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.481153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.481153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":325,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.813897+0000","last_change":"2024-09-18T07:33:34.966610+0000","last_active":"2024-09-18T07:36:24.813897+0000","last_peered":"2024-09-18T07:36:24.813897+0000","last_clean":"2024-09-18T07:36:24.813897+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:36:24.813897+0000","last_undegraded":"2024-09-18T07:36:24.813897+0000","last_fullsized":"2024-09-18T07:36:24.813897+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:34.966537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:34.966537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:21.507482+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":92,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819394+0000","last_change":"2024-09-18T07:35:01.408078+0000","last_active":"2024-09-18T07:36:24.819394+0000","last_peered":"2024-09-18T07:36:24.819394+0000","last_clean":"2024-09-18T07:36:24.819394+0000","last_became_active":"2024-09-18T07:35:01.406980+0000","last_became_peered":"2024-09-18T07:35:01.406980+0000","last_unstale":"2024-09-18T07:36:24.819394+0000","last_undegraded":"2024-09-18T07:36:24.819394+0000","last_fullsized":"2024-09-18T07:36:24.819394+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:16.191826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"213'2697","reported_seq":2732,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:26.468403+0000","last_change":"2024-09-18T07:35:59.287124+0000","last_active":"2024-09-18T07:36:26.468403+0000","last_peered":"2024-09-18T07:36:26.468403+0000","last_clean":"2024-09-18T07:36:26.468403+0000","last_became_active":"2024-09-18T07:35:59.286789+0000","last_became_peered":"2024-09-18T07:35:59.286789+0000","last_unstale":"2024-09-18T07:36:26.468403+0000","last_undegraded":"2024-09-18T07:36:26.468403+0000","last_fullsized":"2024-09-18T07:36:26.468403+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":2697,"log_dups_size":0,"ondisk_log_size":2697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:27.076745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176750592,"num_objects":2697,"num_object_clones":0,"num_object_copies":8091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2697,"num_write_kb":172608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":353,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.806477+0000","last_change":"2024-09-18T07:35:21.085719+0000","last_active":"2024-09-18T07:36:24.806477+0000","last_peered":"2024-09-18T07:36:24.806477+0000","last_clean":"2024-09-18T07:36:24.806477+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:36:24.806477+0000","last_undegraded":"2024-09-18T07:36:24.806477+0000","last_fullsized":"2024-09-18T07:36:24.806477+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:35:21.085670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.085670+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:47.881459+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":92,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819331+0000","last_change":"2024-09-18T07:35:01.400050+0000","last_active":"2024-09-18T07:36:24.819331+0000","last_peered":"2024-09-18T07:36:24.819331+0000","last_clean":"2024-09-18T07:36:24.819331+0000","last_became_active":"2024-09-18T07:35:01.398499+0000","last_became_peered":"2024-09-18T07:35:01.398499+0000","last_unstale":"2024-09-18T07:36:24.819331+0000","last_undegraded":"2024-09-18T07:36:24.819331+0000","last_fullsized":"2024-09-18T07:36:24.819331+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:20.228088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":334,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.813352+0000","last_change":"2024-09-18T07:35:29.856761+0000","last_active":"2024-09-18T07:36:24.813352+0000","last_peered":"2024-09-18T07:36:24.813352+0000","last_clean":"2024-09-18T07:36:24.813352+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:36:24.813352+0000","last_undegraded":"2024-09-18T07:36:24.813352+0000","last_fullsized":"2024-09-18T07:36:24.813352+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:29.856305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:29.856305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:30.681292+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":319,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.823920+0000","last_change":"2024-09-18T07:35:01.419420+0000","last_active":"2024-09-18T07:36:24.823920+0000","last_peered":"2024-09-18T07:36:24.823920+0000","last_clean":"2024-09-18T07:36:24.823920+0000","last_became_active":"2024-09-18T07:35:01.418735+0000","last_became_peered":"2024-09-18T07:35:01.418735+0000","last_unstale":"2024-09-18T07:36:24.823920+0000","last_undegraded":"2024-09-18T07:36:24.823920+0000","last_fullsized":"2024-09-18T07:36:24.823920+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:13.052849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"213'2816","reported_seq":2851,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:26.450452+0000","last_change":"2024-09-18T07:35:59.107480+0000","last_active":"2024-09-18T07:36:26.450452+0000","last_peered":"2024-09-18T07:36:26.450452+0000","last_clean":"2024-09-18T07:36:26.450452+0000","last_became_active":"2024-09-18T07:35:59.106549+0000","last_became_peered":"2024-09-18T07:35:59.106549+0000","last_unstale":"2024-09-18T07:36:26.450452+0000","last_undegraded":"2024-09-18T07:36:26.450452+0000","last_fullsized":"2024-09-18T07:36:26.450452+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":2816,"log_dups_size":0,"ondisk_log_size":2816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:09.466780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":2816,"num_object_clones":0,"num_object_copies":8448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2816,"num_write_kb":180224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":399,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.820260+0000","last_change":"2024-09-18T07:35:36.271956+0000","last_active":"2024-09-18T07:36:24.820260+0000","last_peered":"2024-09-18T07:36:24.820260+0000","last_clean":"2024-09-18T07:36:24.820260+0000","last_became_active":"2024-09-18T07:35:01.416425+0000","last_became_peered":"2024-09-18T07:35:01.416425+0000","last_unstale":"2024-09-18T07:36:24.820260+0000","last_undegraded":"2024-09-18T07:36:24.820260+0000","last_fullsized":"2024-09-18T07:36:24.820260+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:35:36.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:36.271863+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:39.581155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":114,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819626+0000","last_change":"2024-09-18T07:36:09.261574+0000","last_active":"2024-09-18T07:36:24.819626+0000","last_peered":"2024-09-18T07:36:24.819626+0000","last_clean":"2024-09-18T07:36:24.819626+0000","last_became_active":"2024-09-18T07:35:01.406695+0000","last_became_peered":"2024-09-18T07:35:01.406695+0000","last_unstale":"2024-09-18T07:36:24.819626+0000","last_undegraded":"2024-09-18T07:36:24.819626+0000","last_fullsized":"2024-09-18T07:36:24.819626+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:36:09.261516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.261516+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:25.826538+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"173'10","reported_seq":122,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.818600+0000","last_change":"2024-09-18T07:35:26.319384+0000","last_active":"2024-09-18T07:36:24.818600+0000","last_peered":"2024-09-18T07:36:24.818600+0000","last_clean":"2024-09-18T07:36:24.818600+0000","last_became_active":"2024-09-18T07:35:01.400330+0000","last_became_peered":"2024-09-18T07:35:01.400330+0000","last_unstale":"2024-09-18T07:36:24.818600+0000","last_undegraded":"2024-09-18T07:36:24.818600+0000","last_fullsized":"2024-09-18T07:36:24.818600+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:35:26.319320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:26.319320+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:42.536463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":304,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819185+0000","last_change":"2024-09-18T07:31:02.489914+0000","last_active":"2024-09-18T07:36:24.819185+0000","last_peered":"2024-09-18T07:36:24.819185+0000","last_clean":"2024-09-18T07:36:24.819185+0000","last_became_active":"2024-09-18T07:31:02.489619+0000","last_became_peered":"2024-09-18T07:31:02.489619+0000","last_unstale":"2024-09-18T07:36:24.819185+0000","last_undegraded":"2024-09-18T07:36:24.819185+0000","last_fullsized":"2024-09-18T07:36:24.819185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:54.400933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":340,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.816687+0000","last_change":"2024-09-18T07:36:01.259486+0000","last_active":"2024-09-18T07:36:24.816687+0000","last_peered":"2024-09-18T07:36:24.816687+0000","last_clean":"2024-09-18T07:36:24.816687+0000","last_became_active":"2024-09-18T07:30:56.282357+0000","last_became_peered":"2024-09-18T07:30:56.282357+0000","last_unstale":"2024-09-18T07:36:24.816687+0000","last_undegraded":"2024-09-18T07:36:24.816687+0000","last_fullsized":"2024-09-18T07:36:24.816687+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:01.259296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.259296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:18.647109+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":360,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.823553+0000","last_change":"2024-09-18T07:36:09.061983+0000","last_active":"2024-09-18T07:36:24.823553+0000","last_peered":"2024-09-18T07:36:24.823553+0000","last_clean":"2024-09-18T07:36:24.823553+0000","last_became_active":"2024-09-18T07:35:01.407787+0000","last_became_peered":"2024-09-18T07:35:01.407787+0000","last_unstale":"2024-09-18T07:36:24.823553+0000","last_undegraded":"2024-09-18T07:36:24.823553+0000","last_fullsized":"2024-09-18T07:36:24.823553+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-18T07:36:09.061903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.061903+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:12.041619+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":316,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.821298+0000","last_change":"2024-09-18T07:33:32.589362+0000","last_active":"2024-09-18T07:36:24.821298+0000","last_peered":"2024-09-18T07:36:24.821298+0000","last_clean":"2024-09-18T07:36:24.821298+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:36:24.821298+0000","last_undegraded":"2024-09-18T07:36:24.821298+0000","last_fullsized":"2024-09-18T07:36:24.821298+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.589289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.589289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":92,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819351+0000","last_change":"2024-09-18T07:35:01.404598+0000","last_active":"2024-09-18T07:36:24.819351+0000","last_peered":"2024-09-18T07:36:24.819351+0000","last_clean":"2024-09-18T07:36:24.819351+0000","last_became_active":"2024-09-18T07:35:01.403293+0000","last_became_peered":"2024-09-18T07:35:01.403293+0000","last_unstale":"2024-09-18T07:36:24.819351+0000","last_undegraded":"2024-09-18T07:36:24.819351+0000","last_fullsized":"2024-09-18T07:36:24.819351+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:58.688631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:58.688631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:52.212655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":362,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.817139+0000","last_change":"2024-09-18T07:35:01.448510+0000","last_active":"2024-09-18T07:36:24.817139+0000","last_peered":"2024-09-18T07:36:24.817139+0000","last_clean":"2024-09-18T07:36:24.817139+0000","last_became_active":"2024-09-18T07:35:01.414458+0000","last_became_peered":"2024-09-18T07:35:01.414458+0000","last_unstale":"2024-09-18T07:36:24.817139+0000","last_undegraded":"2024-09-18T07:36:24.817139+0000","last_fullsized":"2024-09-18T07:36:24.817139+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:18.543358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":100,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.818440+0000","last_change":"2024-09-18T07:36:05.201804+0000","last_active":"2024-09-18T07:36:24.818440+0000","last_peered":"2024-09-18T07:36:24.818440+0000","last_clean":"2024-09-18T07:36:24.818440+0000","last_became_active":"2024-09-18T07:35:01.398380+0000","last_became_peered":"2024-09-18T07:35:01.398380+0000","last_unstale":"2024-09-18T07:36:24.818440+0000","last_undegraded":"2024-09-18T07:36:24.818440+0000","last_fullsized":"2024-09-18T07:36:24.818440+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:05.201723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.201723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:16.101113+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":339,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.823365+0000","last_change":"2024-09-18T07:35:59.997299+0000","last_active":"2024-09-18T07:36:24.823365+0000","last_peered":"2024-09-18T07:36:24.823365+0000","last_clean":"2024-09-18T07:36:24.823365+0000","last_became_active":"2024-09-18T07:35:01.420133+0000","last_became_peered":"2024-09-18T07:35:01.420133+0000","last_unstale":"2024-09-18T07:36:24.823365+0000","last_undegraded":"2024-09-18T07:36:24.823365+0000","last_fullsized":"2024-09-18T07:36:24.823365+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:59.997249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:59.997249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:14.368192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":320,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.816591+0000","last_change":"2024-09-18T07:35:01.396655+0000","last_active":"2024-09-18T07:36:24.816591+0000","last_peered":"2024-09-18T07:36:24.816591+0000","last_clean":"2024-09-18T07:36:24.816591+0000","last_became_active":"2024-09-18T07:35:01.396305+0000","last_became_peered":"2024-09-18T07:35:01.396305+0000","last_unstale":"2024-09-18T07:36:24.816591+0000","last_undegraded":"2024-09-18T07:36:24.816591+0000","last_fullsized":"2024-09-18T07:36:24.816591+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:20.545641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":370,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.821597+0000","last_change":"2024-09-18T07:35:02.342002+0000","last_active":"2024-09-18T07:36:24.821597+0000","last_peered":"2024-09-18T07:36:24.821597+0000","last_clean":"2024-09-18T07:36:24.821597+0000","last_became_active":"2024-09-18T07:35:01.416744+0000","last_became_peered":"2024-09-18T07:35:01.416744+0000","last_unstale":"2024-09-18T07:36:24.821597+0000","last_undegraded":"2024-09-18T07:36:24.821597+0000","last_fullsized":"2024-09-18T07:36:24.821597+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:35:02.341919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.341919+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":338,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.817027+0000","last_change":"2024-09-18T07:35:01.414704+0000","last_active":"2024-09-18T07:36:24.817027+0000","last_peered":"2024-09-18T07:36:24.817027+0000","last_clean":"2024-09-18T07:36:24.817027+0000","last_became_active":"2024-09-18T07:35:01.414024+0000","last_became_peered":"2024-09-18T07:35:01.414024+0000","last_unstale":"2024-09-18T07:36:24.817027+0000","last_undegraded":"2024-09-18T07:36:24.817027+0000","last_fullsized":"2024-09-18T07:36:24.817027+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:35.707148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:35.707148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:13.589551+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"173'10","reported_seq":414,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.822241+0000","last_change":"2024-09-18T07:36:08.041535+0000","last_active":"2024-09-18T07:36:24.822241+0000","last_peered":"2024-09-18T07:36:24.822241+0000","last_clean":"2024-09-18T07:36:24.822241+0000","last_became_active":"2024-09-18T07:35:01.407568+0000","last_became_peered":"2024-09-18T07:35:01.407568+0000","last_unstale":"2024-09-18T07:36:24.822241+0000","last_undegraded":"2024-09-18T07:36:24.822241+0000","last_fullsized":"2024-09-18T07:36:24.822241+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:36:08.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:08.041456+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:19.626850+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":314,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.806769+0000","last_change":"2024-09-18T07:36:11.910543+0000","last_active":"2024-09-18T07:36:24.806769+0000","last_peered":"2024-09-18T07:36:24.806769+0000","last_clean":"2024-09-18T07:36:24.806769+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:36:24.806769+0000","last_undegraded":"2024-09-18T07:36:24.806769+0000","last_fullsized":"2024-09-18T07:36:24.806769+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:11.910474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:11.910474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:22.591280+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":382,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.822472+0000","last_change":"2024-09-18T07:36:02.975929+0000","last_active":"2024-09-18T07:36:24.822472+0000","last_peered":"2024-09-18T07:36:24.822472+0000","last_clean":"2024-09-18T07:36:24.822472+0000","last_became_active":"2024-09-18T07:35:01.407738+0000","last_became_peered":"2024-09-18T07:35:01.407738+0000","last_unstale":"2024-09-18T07:36:24.822472+0000","last_undegraded":"2024-09-18T07:36:24.822472+0000","last_fullsized":"2024-09-18T07:36:24.822472+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:36:02.975863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:02.975863+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:30.185554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":100,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819277+0000","last_change":"2024-09-18T07:36:08.222898+0000","last_active":"2024-09-18T07:36:24.819277+0000","last_peered":"2024-09-18T07:36:24.819277+0000","last_clean":"2024-09-18T07:36:24.819277+0000","last_became_active":"2024-09-18T07:35:01.400197+0000","last_became_peered":"2024-09-18T07:35:01.400197+0000","last_unstale":"2024-09-18T07:36:24.819277+0000","last_undegraded":"2024-09-18T07:36:24.819277+0000","last_fullsized":"2024-09-18T07:36:24.819277+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:08.222838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:08.222838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:24.073390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":386,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.821336+0000","last_change":"2024-09-18T07:31:00.332729+0000","last_active":"2024-09-18T07:36:24.821336+0000","last_peered":"2024-09-18T07:36:24.821336+0000","last_clean":"2024-09-18T07:36:24.821336+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:36:24.821336+0000","last_undegraded":"2024-09-18T07:36:24.821336+0000","last_fullsized":"2024-09-18T07:36:24.821336+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:17.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":326,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.816655+0000","last_change":"2024-09-18T07:35:01.411256+0000","last_active":"2024-09-18T07:36:24.816655+0000","last_peered":"2024-09-18T07:36:24.816655+0000","last_clean":"2024-09-18T07:36:24.816655+0000","last_became_active":"2024-09-18T07:35:01.410561+0000","last_became_peered":"2024-09-18T07:35:01.410561+0000","last_unstale":"2024-09-18T07:36:24.816655+0000","last_undegraded":"2024-09-18T07:36:24.816655+0000","last_fullsized":"2024-09-18T07:36:24.816655+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.134847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.134847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:53.529639+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":349,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.806508+0000","last_change":"2024-09-18T07:36:01.987042+0000","last_active":"2024-09-18T07:36:24.806508+0000","last_peered":"2024-09-18T07:36:24.806508+0000","last_clean":"2024-09-18T07:36:24.806508+0000","last_became_active":"2024-09-18T07:30:58.314428+0000","last_became_peered":"2024-09-18T07:30:58.314428+0000","last_unstale":"2024-09-18T07:36:24.806508+0000","last_undegraded":"2024-09-18T07:36:24.806508+0000","last_fullsized":"2024-09-18T07:36:24.806508+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:36:01.986964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.986964+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:10.603454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":332,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.807253+0000","last_change":"2024-09-18T07:35:01.401235+0000","last_active":"2024-09-18T07:36:24.807253+0000","last_peered":"2024-09-18T07:36:24.807253+0000","last_clean":"2024-09-18T07:36:24.807253+0000","last_became_active":"2024-09-18T07:35:01.400615+0000","last_became_peered":"2024-09-18T07:35:01.400615+0000","last_unstale":"2024-09-18T07:36:24.807253+0000","last_undegraded":"2024-09-18T07:36:24.807253+0000","last_fullsized":"2024-09-18T07:36:24.807253+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:20.011870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":17,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819376+0000","last_change":"2024-09-18T07:36:13.118114+0000","last_active":"2024-09-18T07:36:24.819376+0000","last_peered":"2024-09-18T07:36:24.819376+0000","last_clean":"2024-09-18T07:36:24.819376+0000","last_became_active":"2024-09-18T07:36:13.117584+0000","last_became_peered":"2024-09-18T07:36:13.117584+0000","last_unstale":"2024-09-18T07:36:24.819376+0000","last_undegraded":"2024-09-18T07:36:24.819376+0000","last_fullsized":"2024-09-18T07:36:24.819376+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:23.545467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":305,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.817836+0000","last_change":"2024-09-18T07:31:02.351247+0000","last_active":"2024-09-18T07:36:24.817836+0000","last_peered":"2024-09-18T07:36:24.817836+0000","last_clean":"2024-09-18T07:36:24.817836+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:36:24.817836+0000","last_undegraded":"2024-09-18T07:36:24.817836+0000","last_fullsized":"2024-09-18T07:36:24.817836+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:16.072365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":216,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.816791+0000","last_change":"2024-09-18T07:35:23.572395+0000","last_active":"2024-09-18T07:36:24.816791+0000","last_peered":"2024-09-18T07:36:24.816791+0000","last_clean":"2024-09-18T07:36:24.816791+0000","last_became_active":"2024-09-18T07:33:27.003887+0000","last_became_peered":"2024-09-18T07:33:27.003887+0000","last_unstale":"2024-09-18T07:36:24.816791+0000","last_undegraded":"2024-09-18T07:36:24.816791+0000","last_fullsized":"2024-09-18T07:36:24.816791+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:35:23.572304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:23.572304+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:35.224701+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":100,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819259+0000","last_change":"2024-09-18T07:35:57.293607+0000","last_active":"2024-09-18T07:36:24.819259+0000","last_peered":"2024-09-18T07:36:24.819259+0000","last_clean":"2024-09-18T07:36:24.819259+0000","last_became_active":"2024-09-18T07:35:01.405287+0000","last_became_peered":"2024-09-18T07:35:01.405287+0000","last_unstale":"2024-09-18T07:36:24.819259+0000","last_undegraded":"2024-09-18T07:36:24.819259+0000","last_fullsized":"2024-09-18T07:36:24.819259+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:57.293555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.293555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:58.503316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":317,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.824062+0000","last_change":"2024-09-18T07:35:01.205647+0000","last_active":"2024-09-18T07:36:24.824062+0000","last_peered":"2024-09-18T07:36:24.824062+0000","last_clean":"2024-09-18T07:36:24.824062+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:36:24.824062+0000","last_undegraded":"2024-09-18T07:36:24.824062+0000","last_fullsized":"2024-09-18T07:36:24.824062+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:01.205596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:01.205596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:24.102316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":320,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.816128+0000","last_change":"2024-09-18T07:35:20.576655+0000","last_active":"2024-09-18T07:36:24.816128+0000","last_peered":"2024-09-18T07:36:24.816128+0000","last_clean":"2024-09-18T07:36:24.816128+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:36:24.816128+0000","last_undegraded":"2024-09-18T07:36:24.816128+0000","last_fullsized":"2024-09-18T07:36:24.816128+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:20.576587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:20.576587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:34.998253+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":378,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.823743+0000","last_change":"2024-09-18T07:35:06.201799+0000","last_active":"2024-09-18T07:36:24.823743+0000","last_peered":"2024-09-18T07:36:24.823743+0000","last_clean":"2024-09-18T07:36:24.823743+0000","last_became_active":"2024-09-18T07:35:01.407576+0000","last_became_peered":"2024-09-18T07:35:01.407576+0000","last_unstale":"2024-09-18T07:36:24.823743+0000","last_undegraded":"2024-09-18T07:36:24.823743+0000","last_fullsized":"2024-09-18T07:36:24.823743+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:35:06.201725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:06.201725+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":338,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.821500+0000","last_change":"2024-09-18T07:35:01.415691+0000","last_active":"2024-09-18T07:36:24.821500+0000","last_peered":"2024-09-18T07:36:24.821500+0000","last_clean":"2024-09-18T07:36:24.821500+0000","last_became_active":"2024-09-18T07:35:01.415261+0000","last_became_peered":"2024-09-18T07:35:01.415261+0000","last_unstale":"2024-09-18T07:36:24.821500+0000","last_undegraded":"2024-09-18T07:36:24.821500+0000","last_fullsized":"2024-09-18T07:36:24.821500+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:58.734027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:58.734027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":100,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.819481+0000","last_change":"2024-09-18T07:35:41.311353+0000","last_active":"2024-09-18T07:36:24.819481+0000","last_peered":"2024-09-18T07:36:24.819481+0000","last_clean":"2024-09-18T07:36:24.819481+0000","last_became_active":"2024-09-18T07:35:01.400714+0000","last_became_peered":"2024-09-18T07:35:01.400714+0000","last_unstale":"2024-09-18T07:36:24.819481+0000","last_undegraded":"2024-09-18T07:36:24.819481+0000","last_fullsized":"2024-09-18T07:36:24.819481+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:41.311291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:41.311291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:58.412847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":321,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.824171+0000","last_change":"2024-09-18T07:35:36.026283+0000","last_active":"2024-09-18T07:36:24.824171+0000","last_peered":"2024-09-18T07:36:24.824171+0000","last_clean":"2024-09-18T07:36:24.824171+0000","last_became_active":"2024-09-18T07:31:02.490994+0000","last_became_peered":"2024-09-18T07:31:02.490994+0000","last_unstale":"2024-09-18T07:36:24.824171+0000","last_undegraded":"2024-09-18T07:36:24.824171+0000","last_fullsized":"2024-09-18T07:36:24.824171+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:36.026215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:36.026215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:40.267712+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":348,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.823598+0000","last_change":"2024-09-18T07:35:29.942418+0000","last_active":"2024-09-18T07:36:24.823598+0000","last_peered":"2024-09-18T07:36:24.823598+0000","last_clean":"2024-09-18T07:36:24.823598+0000","last_became_active":"2024-09-18T07:32:26.791562+0000","last_became_peered":"2024-09-18T07:32:26.791562+0000","last_unstale":"2024-09-18T07:36:24.823598+0000","last_undegraded":"2024-09-18T07:36:24.823598+0000","last_fullsized":"2024-09-18T07:36:24.823598+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:35:29.937819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:29.937819+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:40.163643+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":316,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.820699+0000","last_change":"2024-09-18T07:30:56.283643+0000","last_active":"2024-09-18T07:36:24.820699+0000","last_peered":"2024-09-18T07:36:24.820699+0000","last_clean":"2024-09-18T07:36:24.820699+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:36:24.820699+0000","last_undegraded":"2024-09-18T07:36:24.820699+0000","last_fullsized":"2024-09-18T07:36:24.820699+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:52.540796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":338,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.822815+0000","last_change":"2024-09-18T07:35:02.394924+0000","last_active":"2024-09-18T07:36:24.822815+0000","last_peered":"2024-09-18T07:36:24.822815+0000","last_clean":"2024-09-18T07:36:24.822815+0000","last_became_active":"2024-09-18T07:35:01.407641+0000","last_became_peered":"2024-09-18T07:35:01.407641+0000","last_unstale":"2024-09-18T07:36:24.822815+0000","last_undegraded":"2024-09-18T07:36:24.822815+0000","last_fullsized":"2024-09-18T07:36:24.822815+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.394872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.394872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:29.732852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":100,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.818849+0000","last_change":"2024-09-18T07:35:29.235047+0000","last_active":"2024-09-18T07:36:24.818849+0000","last_peered":"2024-09-18T07:36:24.818849+0000","last_clean":"2024-09-18T07:36:24.818849+0000","last_became_active":"2024-09-18T07:35:01.400396+0000","last_became_peered":"2024-09-18T07:35:01.400396+0000","last_unstale":"2024-09-18T07:36:24.818849+0000","last_undegraded":"2024-09-18T07:36:24.818849+0000","last_fullsized":"2024-09-18T07:36:24.818849+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:29.234993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:29.234993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:54.885730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":17,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.821380+0000","last_change":"2024-09-18T07:36:13.118009+0000","last_active":"2024-09-18T07:36:24.821380+0000","last_peered":"2024-09-18T07:36:24.821380+0000","last_clean":"2024-09-18T07:36:24.821380+0000","last_became_active":"2024-09-18T07:36:13.117154+0000","last_became_peered":"2024-09-18T07:36:13.117154+0000","last_unstale":"2024-09-18T07:36:24.821380+0000","last_undegraded":"2024-09-18T07:36:24.821380+0000","last_fullsized":"2024-09-18T07:36:24.821380+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.071629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.071629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":339,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.823857+0000","last_change":"2024-09-18T07:35:57.951851+0000","last_active":"2024-09-18T07:36:24.823857+0000","last_peered":"2024-09-18T07:36:24.823857+0000","last_clean":"2024-09-18T07:36:24.823857+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:36:24.823857+0000","last_undegraded":"2024-09-18T07:36:24.823857+0000","last_fullsized":"2024-09-18T07:36:24.823857+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:35:57.951801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.951801+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:26.846166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":347,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.818615+0000","last_change":"2024-09-18T07:34:58.432839+0000","last_active":"2024-09-18T07:36:24.818615+0000","last_peered":"2024-09-18T07:36:24.818615+0000","last_clean":"2024-09-18T07:36:24.818615+0000","last_became_active":"2024-09-18T07:30:58.313556+0000","last_became_peered":"2024-09-18T07:30:58.313556+0000","last_unstale":"2024-09-18T07:36:24.818615+0000","last_undegraded":"2024-09-18T07:36:24.818615+0000","last_fullsized":"2024-09-18T07:36:24.818615+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:34:58.432781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:58.432781+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:15.227317+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":338,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-18T07:36:24.816888+0000","last_change":"2024-09-18T07:35:25.576024+0000","last_active":"2024-09-18T07:36:24.816888+0000","last_peered":"2024-09-18T07:36:24.816888+0000","last_clean":"2024-09-18T07:36:24.816888+0000","last_became_active":"2024-09-18T07:35:01.410469+0000","last_became_peered":"2024-09-18T07:35:01.410469+0000","last_unstale":"2024-09-18T07:36:24.816888+0000","last_undegraded":"2024-09-18T07:36:24.816888+0000","last_fullsized":"2024-09-18T07:36:24.816888+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:25.575954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:25.575954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:44.318630+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2942042112,"num_objects":44892,"num_object_clones":0,"num_object_copies":134676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44892,"num_write_kb":2873088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8813346816,"data_stored":8813346816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44892,"ondisk_log_size":44892,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":2948,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":91,"num_read_kb":86,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201350,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":998768,"kb_used_data":873184,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92746384,"statfs":{"total":95995035648,"available":94972297216,"internally_reserved":0,"allocated":894140416,"data_stored":893590995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128580042},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462988,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1019868,"kb_used_data":894284,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92725284,"statfs":{"total":95995035648,"available":94950690816,"internally_reserved":0,"allocated":915746816,"data_stored":915216790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128580038},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724625,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1530404,"kb_used_data":1377040,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":92214748,"statfs":{"total":95995035648,"available":94427901952,"internally_reserved":0,"allocated":1410088960,"data_stored":1409554860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953560,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1041080,"kb_used_data":875236,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92704072,"statfs":{"total":95995035648,"available":94928969728,"internally_reserved":0,"allocated":896241664,"data_stored":895687546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1346752,"kb_used_data":1086512,"kb_used_omap":1,"kb_used_meta":260222,"kb_avail":92398400,"statfs":{"total":95995035648,"available":94615961600,"internally_reserved":0,"allocated":1112588288,"data_stored":1112023064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":266467783},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411426,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1261528,"kb_used_data":1108164,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":92483624,"statfs":{"total":95995035648,"available":94703230976,"internally_reserved":0,"allocated":1134759936,"data_stored":1134237803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574953,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1427856,"kb_used_data":1243964,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92317296,"statfs":{"total":95995035648,"available":94532911104,"internally_reserved":0,"allocated":1273819136,"data_stored":1273241849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738478,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1257852,"kb_used_data":1078124,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92487300,"statfs":{"total":95995035648,"available":94706995200,"internally_reserved":0,"allocated":1103998976,"data_stored":1103438502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1110114304,"data_stored":1110114304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1291911168,"data_stored":1291911168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1103822848,"data_stored":1103822848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1111228416,"data_stored":1111228416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":908066816,"data_stored":908066816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1468530688,"data_stored":1468530688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":917569536,"data_stored":917569536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":902103040,"data_stored":902103040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:36:32.935 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-18T07:36:32.973 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:33.021 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:33.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:33.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:33.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:33.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:33.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:33.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:33.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:32 smithi137 ceph-mon[29408]: pgmap v298: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 114 MiB/s wr, 1.83k op/s 2024-09-18T07:36:33.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:32 smithi137 ceph-mon[29408]: 3.9 scrub starts 2024-09-18T07:36:33.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:32 smithi137 ceph-mon[29408]: 3.9 scrub ok 2024-09-18T07:36:33.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:32 smithi137 ceph-mon[39175]: pgmap v298: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 114 MiB/s wr, 1.83k op/s 2024-09-18T07:36:33.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:32 smithi137 ceph-mon[39175]: 3.9 scrub starts 2024-09-18T07:36:33.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:32 smithi137 ceph-mon[39175]: 3.9 scrub ok 2024-09-18T07:36:33.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:32 smithi203 ceph-mon[32237]: pgmap v298: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 114 MiB/s wr, 1.83k op/s 2024-09-18T07:36:33.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:32 smithi203 ceph-mon[32237]: 3.9 scrub starts 2024-09-18T07:36:33.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:32 smithi203 ceph-mon[32237]: 3.9 scrub ok 2024-09-18T07:36:33.268 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:33.298 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:33.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:33.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:33.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:33.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:33.429 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:33.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:33.494 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:36:33.495 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:36:33.518 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:36:33.519 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:36:33.529 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:33.554 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:36:33.554 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:36:33.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:33.577 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:33.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:33.586 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:36:33.586 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:36:33.609 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:33.620 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:36:33.621 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:36:33.650 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:36:33.650 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:36:33.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:33.685 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:36:33.686 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:36:33.719 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:36:33.719 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:36:33.725 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:33.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:33.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:33.827 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:33.863 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:33.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:33.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:33.922 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 28 16 56032 56016 125.009 79.3125 0.00816034 0.00799402 2024-09-18T07:36:33.965 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:34.000 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:34.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:34.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:34.102 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:34.142 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:34.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:34.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:34.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:33 smithi137 ceph-mon[29408]: 5.8 scrub starts 2024-09-18T07:36:34.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:33 smithi137 ceph-mon[29408]: 5.8 scrub ok 2024-09-18T07:36:34.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:33 smithi137 ceph-mon[29408]: 6.e scrub starts 2024-09-18T07:36:34.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:33 smithi137 ceph-mon[29408]: 6.e scrub ok 2024-09-18T07:36:34.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:33 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4148615730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:34.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:33 smithi137 ceph-mon[39175]: 5.8 scrub starts 2024-09-18T07:36:34.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:33 smithi137 ceph-mon[39175]: 5.8 scrub ok 2024-09-18T07:36:34.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:33 smithi137 ceph-mon[39175]: 6.e scrub starts 2024-09-18T07:36:34.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:33 smithi137 ceph-mon[39175]: 6.e scrub ok 2024-09-18T07:36:34.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:33 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4148615730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:34.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:33 smithi203 ceph-mon[32237]: 5.8 scrub starts 2024-09-18T07:36:34.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:33 smithi203 ceph-mon[32237]: 5.8 scrub ok 2024-09-18T07:36:34.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:33 smithi203 ceph-mon[32237]: 6.e scrub starts 2024-09-18T07:36:34.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:33 smithi203 ceph-mon[32237]: 6.e scrub ok 2024-09-18T07:36:34.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:33 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4148615730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:34.244 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:34.279 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:34.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:34.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:34.381 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:34.415 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:34.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:34.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:34.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:34.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:34.543 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:34.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:34.644 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:34.673 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:34.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:34.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:34.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:34.808 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:34.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:34.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:34.900 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:36:34.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:34.922 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 29 16 57365 57349 123.571 83.3125 0.0113747 0.00808724 2024-09-18T07:36:34.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:34.954 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:34.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:35.055 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:35.059 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:36:35.059 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:36:35.059 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:36:35.060 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:36:35.060 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:36:35.092 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:35.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:35.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:35.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:34 smithi137 ceph-mon[29408]: pgmap v299: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.08k op/s 2024-09-18T07:36:35.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:34 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/558436881' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:35.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:34 smithi137 ceph-mon[29408]: from='client.25264 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:35.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:34 smithi137 ceph-mon[39175]: pgmap v299: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.08k op/s 2024-09-18T07:36:35.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:34 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/558436881' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:35.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:34 smithi137 ceph-mon[39175]: from='client.25264 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:35.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:35.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:34 smithi203 ceph-mon[32237]: pgmap v299: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.08k op/s 2024-09-18T07:36:35.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:34 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/558436881' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:35.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:34 smithi203 ceph-mon[32237]: from='client.25264 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:35.225 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:35.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:35.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:35.366 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:35.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:35.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:35.488 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:35.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:35.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:35.590 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:35.624 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:35.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:35.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:35.726 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:35.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:35.754 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:35.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:35.855 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:35.891 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:35.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:35.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:35.922 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 30 16 58639 58623 122.106 79.625 0.00642869 0.00818418 2024-09-18T07:36:35.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:36.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:36.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:36.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:36.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:36.172 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:36.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:36.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:36.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:35 smithi203 ceph-mon[32237]: 5.16 scrub starts 2024-09-18T07:36:36.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:35 smithi203 ceph-mon[32237]: 5.16 scrub ok 2024-09-18T07:36:36.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:35 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:36:36.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:35 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:36:36.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:35 smithi203 ceph-mon[32237]: from='client.25264 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:36:36.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:35 smithi203 ceph-mon[32237]: osdmap e216: 8 total, 8 up, 8 in 2024-09-18T07:36:36.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:35 smithi203 ceph-mon[32237]: 5.e scrub starts 2024-09-18T07:36:36.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:35 smithi203 ceph-mon[32237]: 5.e scrub ok 2024-09-18T07:36:36.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:35 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:36:36.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:35 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:36:36.274 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:36.312 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:36.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:36.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:36.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:36.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[29408]: 5.16 scrub starts 2024-09-18T07:36:36.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[29408]: 5.16 scrub ok 2024-09-18T07:36:36.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:36:36.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:36:36.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[29408]: from='client.25264 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:36:36.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[29408]: osdmap e216: 8 total, 8 up, 8 in 2024-09-18T07:36:36.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[29408]: 5.e scrub starts 2024-09-18T07:36:36.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[29408]: 5.e scrub ok 2024-09-18T07:36:36.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:36:36.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:36:36.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[39175]: 5.16 scrub starts 2024-09-18T07:36:36.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[39175]: 5.16 scrub ok 2024-09-18T07:36:36.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:36:36.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:36:36.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[39175]: from='client.25264 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:36:36.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[39175]: osdmap e216: 8 total, 8 up, 8 in 2024-09-18T07:36:36.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[39175]: 5.e scrub starts 2024-09-18T07:36:36.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[39175]: 5.e scrub ok 2024-09-18T07:36:36.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:36:36.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:35 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:36:36.451 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:36.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:36.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:36.553 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:36.586 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:36.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:36.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:36.688 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:36.722 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:36.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:36.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:36.767 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T07:36:36.824 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:36.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:36.860 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:36.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:36.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:36:36.922 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 31 16 60262 60246 121.439 101.438 0.00790329 0.0082298 2024-09-18T07:36:36.961 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:37.005 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:37.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:37.106 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:37.149 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:37.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:37.149 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:36 smithi203 ceph-mon[32237]: pgmap v301: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 669 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-18T07:36:37.149 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:36 smithi203 ceph-mon[32237]: osdmap e217: 8 total, 8 up, 8 in 2024-09-18T07:36:37.166 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:36:37.250 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:37.278 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:37.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:37.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:37.324 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:37.379 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:37.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:37.413 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:37.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:37.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:36 smithi137 ceph-mon[29408]: pgmap v301: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 669 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-18T07:36:37.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:36 smithi137 ceph-mon[29408]: osdmap e217: 8 total, 8 up, 8 in 2024-09-18T07:36:37.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:36:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:36:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:36:37.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:36 smithi137 ceph-mon[39175]: pgmap v301: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 669 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-18T07:36:37.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:36 smithi137 ceph-mon[39175]: osdmap e217: 8 total, 8 up, 8 in 2024-09-18T07:36:37.458 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:36:37 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:36:37.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:37.553 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:37.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:37.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:37.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:37.694 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:37.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:36:37.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:37.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:37.737 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:36:37.737 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:36:37.737 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:36:37.737 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:36:37.737 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.737 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54188 7.c 7:3726bab8:::benchmark_data_smithi137_74180_object54187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T07:36:37.737 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.151582+0000", 2024-09-18T07:36:37.737 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 224.58045738800001, 2024-09-18T07:36:37.738 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0152226929999999, 2024-09-18T07:36:37.738 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.738 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.738 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:36:37.738 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.738 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.738 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.738 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54188 2024-09-18T07:36:37.738 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.738 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.738 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.738 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.739 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151582+0000", 2024-09-18T07:36:37.739 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.739 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.739 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.739 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.739 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151582+0000", 2024-09-18T07:36:37.739 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.739 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.739 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.739 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.739 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151583+0000", 2024-09-18T07:36:37.739 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6300000000000001e-06 2024-09-18T07:36:37.739 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.740 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.740 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.740 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151619+0000", 2024-09-18T07:36:37.740 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5178000000000001e-05 2024-09-18T07:36:37.740 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.740 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.740 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.740 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151627+0000", 2024-09-18T07:36:37.740 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2849999999999995e-06 2024-09-18T07:36:37.740 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.740 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.740 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.740 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151633+0000", 2024-09-18T07:36:37.741 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5729999999999996e-06 2024-09-18T07:36:37.741 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.741 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.741 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.741 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.152287+0000", 2024-09-18T07:36:37.741 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00065367599999999995 2024-09-18T07:36:37.741 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.741 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.741 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:37.741 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.152442+0000", 2024-09-18T07:36:37.741 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015491699999999999 2024-09-18T07:36:37.741 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.742 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.742 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:36:37.742 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.152627+0000", 2024-09-18T07:36:37.742 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018544300000000001 2024-09-18T07:36:37.742 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.742 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.742 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:37.742 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.156875+0000", 2024-09-18T07:36:37.742 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0042476329999999998 2024-09-18T07:36:37.742 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.742 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.742 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.742 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.161790+0000", 2024-09-18T07:36:37.743 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0049151050000000003 2024-09-18T07:36:37.743 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.743 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.743 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.743 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.161803+0000", 2024-09-18T07:36:37.743 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2646999999999999e-05 2024-09-18T07:36:37.743 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.743 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.743 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.743 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.166719+0000", 2024-09-18T07:36:37.743 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0049166270000001 2024-09-18T07:36:37.743 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.743 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.743 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.744 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.166804+0000", 2024-09-18T07:36:37.744 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4979000000000001e-05 2024-09-18T07:36:37.744 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.744 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.744 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.744 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.744 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.744 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54193 7.c 7:33299084:::benchmark_data_smithi137_74180_object54192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T07:36:37.744 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.154898+0000", 2024-09-18T07:36:37.744 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 224.57714142, 2024-09-18T07:36:37.744 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0124331120000001, 2024-09-18T07:36:37.744 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.744 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.745 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:36:37.745 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.745 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.745 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.745 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54193 2024-09-18T07:36:37.745 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.745 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.745 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.745 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.745 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154898+0000", 2024-09-18T07:36:37.746 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.746 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.746 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.746 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.746 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154898+0000", 2024-09-18T07:36:37.746 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.746 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.746 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.746 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.746 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154900+0000", 2024-09-18T07:36:37.746 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1100000000000001e-06 2024-09-18T07:36:37.746 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.746 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.747 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.747 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154938+0000", 2024-09-18T07:36:37.747 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8479999999999997e-05 2024-09-18T07:36:37.747 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.747 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.747 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.747 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154957+0000", 2024-09-18T07:36:37.747 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8437e-05 2024-09-18T07:36:37.747 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.747 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.747 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.747 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154965+0000", 2024-09-18T07:36:37.747 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1780000000000003e-06 2024-09-18T07:36:37.747 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.748 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.748 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.748 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.155029+0000", 2024-09-18T07:36:37.748 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4459000000000001e-05 2024-09-18T07:36:37.748 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.748 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.748 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:37.748 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.155164+0000", 2024-09-18T07:36:37.748 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013460699999999999 2024-09-18T07:36:37.748 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.748 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.748 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:36:37.748 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.155322+0000", 2024-09-18T07:36:37.748 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015807199999999999 2024-09-18T07:36:37.749 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.749 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.749 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:37.749 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.157099+0000", 2024-09-18T07:36:37.749 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0017770570000000001 2024-09-18T07:36:37.749 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.749 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.749 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.749 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.161956+0000", 2024-09-18T07:36:37.749 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.004857264 2024-09-18T07:36:37.749 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.749 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.750 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.750 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.161962+0000", 2024-09-18T07:36:37.750 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9479999999999996e-06 2024-09-18T07:36:37.750 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.750 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.750 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.750 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.167294+0000", 2024-09-18T07:36:37.750 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.005331441 2024-09-18T07:36:37.750 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.750 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.750 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.750 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.167331+0000", 2024-09-18T07:36:37.750 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7058999999999999e-05 2024-09-18T07:36:37.751 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.751 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.751 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.751 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.751 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.751 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54223 7.c 7:39274186:::benchmark_data_smithi137_74180_object54222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:36:37.751 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.192250+0000", 2024-09-18T07:36:37.751 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 224.539788856, 2024-09-18T07:36:37.751 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.98501845799999999, 2024-09-18T07:36:37.751 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.751 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.751 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:37.751 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.752 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.753 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.753 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54223 2024-09-18T07:36:37.753 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.753 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.753 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.753 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.753 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192250+0000", 2024-09-18T07:36:37.753 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.753 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.754 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.754 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.754 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192250+0000", 2024-09-18T07:36:37.754 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.754 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.754 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.754 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.754 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192251+0000", 2024-09-18T07:36:37.754 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.037e-06 2024-09-18T07:36:37.754 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.754 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.754 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.755 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192317+0000", 2024-09-18T07:36:37.755 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5246999999999994e-05 2024-09-18T07:36:37.755 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.755 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.755 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.755 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192327+0000", 2024-09-18T07:36:37.755 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0464e-05 2024-09-18T07:36:37.755 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.755 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.755 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.755 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192335+0000", 2024-09-18T07:36:37.755 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2239999999999995e-06 2024-09-18T07:36:37.756 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.756 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.756 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.756 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192402+0000", 2024-09-18T07:36:37.756 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6580000000000003e-05 2024-09-18T07:36:37.756 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.756 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.756 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.756 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192414+0000", 2024-09-18T07:36:37.756 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2475e-05 2024-09-18T07:36:37.756 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.756 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.756 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.756 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.167364+0000", 2024-09-18T07:36:37.757 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97494975800000006 2024-09-18T07:36:37.757 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.757 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.757 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:37.757 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.167483+0000", 2024-09-18T07:36:37.757 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000119238 2024-09-18T07:36:37.757 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.757 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.757 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:36:37.757 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.167628+0000", 2024-09-18T07:36:37.757 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000144709 2024-09-18T07:36:37.757 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.757 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.758 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:37.758 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.172321+0000", 2024-09-18T07:36:37.758 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0046927879999999998 2024-09-18T07:36:37.758 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.758 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.758 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.758 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173324+0000", 2024-09-18T07:36:37.758 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001003096 2024-09-18T07:36:37.758 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.758 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.758 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.758 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.177200+0000", 2024-09-18T07:36:37.758 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0038757449999999999 2024-09-18T07:36:37.758 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.759 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.759 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:37.759 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.177233+0000", 2024-09-18T07:36:37.759 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3358000000000001e-05 2024-09-18T07:36:37.759 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.759 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.759 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.759 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.177269+0000", 2024-09-18T07:36:37.759 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5738999999999998e-05 2024-09-18T07:36:37.759 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.759 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.759 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.759 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.760 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.760 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54246 7.c 7:39613093:::benchmark_data_smithi137_74180_object54245:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:36:37.760 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.212409+0000", 2024-09-18T07:36:37.760 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 224.51963016299999, 2024-09-18T07:36:37.760 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97093596400000004, 2024-09-18T07:36:37.760 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.760 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.760 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:37.760 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.760 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.760 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.760 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54246 2024-09-18T07:36:37.761 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.761 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.761 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.761 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.761 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212409+0000", 2024-09-18T07:36:37.761 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.761 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.761 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.761 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.761 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212409+0000", 2024-09-18T07:36:37.761 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.761 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.761 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.762 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.762 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212412+0000", 2024-09-18T07:36:37.762 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6350000000000002e-06 2024-09-18T07:36:37.762 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.762 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.762 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.762 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212483+0000", 2024-09-18T07:36:37.762 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1408999999999993e-05 2024-09-18T07:36:37.762 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.762 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.762 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.762 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212497+0000", 2024-09-18T07:36:37.762 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3576e-05 2024-09-18T07:36:37.762 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.763 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.763 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.763 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212505+0000", 2024-09-18T07:36:37.763 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1990000000000006e-06 2024-09-18T07:36:37.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.763 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.763 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.763 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212575+0000", 2024-09-18T07:36:37.763 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0662999999999998e-05 2024-09-18T07:36:37.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.763 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.763 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.763 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212588+0000", 2024-09-18T07:36:37.764 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2639e-05 2024-09-18T07:36:37.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.764 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.764 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.764 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.169092+0000", 2024-09-18T07:36:37.764 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.95650415300000002 2024-09-18T07:36:37.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.764 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.764 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:37.764 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.169206+0000", 2024-09-18T07:36:37.764 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000113328 2024-09-18T07:36:37.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.764 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.765 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:36:37.765 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.169341+0000", 2024-09-18T07:36:37.765 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013537600000000001 2024-09-18T07:36:37.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.765 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.765 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:37.765 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173339+0000", 2024-09-18T07:36:37.765 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0039977909999999997 2024-09-18T07:36:37.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.765 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.765 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.765 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.182250+0000", 2024-09-18T07:36:37.765 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.008910968 2024-09-18T07:36:37.766 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.766 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.766 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.766 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.183281+0000", 2024-09-18T07:36:37.766 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010308940000000001 2024-09-18T07:36:37.766 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.766 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.766 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:37.766 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.183314+0000", 2024-09-18T07:36:37.766 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3791000000000002e-05 2024-09-18T07:36:37.766 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.766 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.766 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.766 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.183345+0000", 2024-09-18T07:36:37.767 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0542e-05 2024-09-18T07:36:37.767 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.767 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.767 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.767 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.767 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54263 7.c 7:35b11aa8:::benchmark_data_smithi137_74180_object54262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:36:37.767 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.227767+0000", 2024-09-18T07:36:37.767 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 224.504272415, 2024-09-18T07:36:37.767 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.95717552100000003, 2024-09-18T07:36:37.767 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.767 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.768 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:37.768 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.768 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.768 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.768 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54263 2024-09-18T07:36:37.768 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.768 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.768 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.768 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.768 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227767+0000", 2024-09-18T07:36:37.768 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.768 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.768 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.769 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.769 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227767+0000", 2024-09-18T07:36:37.769 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.769 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.769 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.769 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.769 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227769+0000", 2024-09-18T07:36:37.769 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8640000000000001e-06 2024-09-18T07:36:37.769 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.769 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.769 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.769 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227814+0000", 2024-09-18T07:36:37.770 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5540000000000001e-05 2024-09-18T07:36:37.770 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.770 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.770 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.770 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227829+0000", 2024-09-18T07:36:37.770 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4586000000000001e-05 2024-09-18T07:36:37.770 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.770 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.770 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.770 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227837+0000", 2024-09-18T07:36:37.770 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3080000000000008e-06 2024-09-18T07:36:37.770 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.770 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.770 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.771 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227923+0000", 2024-09-18T07:36:37.771 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5531999999999994e-05 2024-09-18T07:36:37.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.771 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.771 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.771 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227934+0000", 2024-09-18T07:36:37.771 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1232e-05 2024-09-18T07:36:37.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.771 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.771 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.771 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.170335+0000", 2024-09-18T07:36:37.771 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.94240168099999999 2024-09-18T07:36:37.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.771 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.772 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:37.772 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.170404+0000", 2024-09-18T07:36:37.772 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9021000000000002e-05 2024-09-18T07:36:37.772 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.772 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.772 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:36:37.772 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.170496+0000", 2024-09-18T07:36:37.772 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1472999999999996e-05 2024-09-18T07:36:37.772 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.772 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.772 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:37.772 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173344+0000", 2024-09-18T07:36:37.772 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0028482970000000001 2024-09-18T07:36:37.773 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.773 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.773 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.773 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.182293+0000", 2024-09-18T07:36:37.773 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0089482840000000008 2024-09-18T07:36:37.773 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.773 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.773 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.773 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.184822+0000", 2024-09-18T07:36:37.773 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025292090000000001 2024-09-18T07:36:37.773 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.773 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.773 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:37.774 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.184880+0000", 2024-09-18T07:36:37.774 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8680000000000001e-05 2024-09-18T07:36:37.774 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.774 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.774 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.774 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.184942+0000", 2024-09-18T07:36:37.774 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1814000000000001e-05 2024-09-18T07:36:37.774 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.774 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.774 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.774 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.774 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.774 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54278 7.c 7:36e62dce:::benchmark_data_smithi137_74180_object54277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:36:37.775 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.241010+0000", 2024-09-18T07:36:37.775 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 224.49102866699999, 2024-09-18T07:36:37.775 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.94404960299999996, 2024-09-18T07:36:37.775 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.775 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.775 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:37.775 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.775 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.775 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.775 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54278 2024-09-18T07:36:37.775 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.775 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.775 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.776 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.776 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241010+0000", 2024-09-18T07:36:37.776 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.776 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.776 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.776 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.776 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241010+0000", 2024-09-18T07:36:37.776 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.776 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.776 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.776 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.776 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241014+0000", 2024-09-18T07:36:37.776 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.112e-06 2024-09-18T07:36:37.777 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.777 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.777 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.777 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241049+0000", 2024-09-18T07:36:37.777 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5605999999999998e-05 2024-09-18T07:36:37.777 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.777 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.777 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.777 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241067+0000", 2024-09-18T07:36:37.777 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8077e-05 2024-09-18T07:36:37.777 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.777 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.777 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.777 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241075+0000", 2024-09-18T07:36:37.778 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7510000000000008e-06 2024-09-18T07:36:37.778 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.778 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.778 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.778 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241153+0000", 2024-09-18T07:36:37.778 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7642e-05 2024-09-18T07:36:37.778 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.778 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.778 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.778 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241167+0000", 2024-09-18T07:36:37.778 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3987e-05 2024-09-18T07:36:37.778 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.778 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.779 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.779 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.171235+0000", 2024-09-18T07:36:37.779 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93006873700000003 2024-09-18T07:36:37.779 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.779 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.779 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:37.779 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.171328+0000", 2024-09-18T07:36:37.779 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2200999999999994e-05 2024-09-18T07:36:37.779 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.779 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.779 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:36:37.779 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.171447+0000", 2024-09-18T07:36:37.779 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000119874 2024-09-18T07:36:37.780 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.780 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.780 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:37.780 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.175350+0000", 2024-09-18T07:36:37.780 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0039030359999999999 2024-09-18T07:36:37.780 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.780 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.780 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.780 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.182332+0000", 2024-09-18T07:36:37.780 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069816849999999996 2024-09-18T07:36:37.780 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.780 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.781 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.781 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.184998+0000", 2024-09-18T07:36:37.781 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026659689999999998 2024-09-18T07:36:37.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.781 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.781 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:37.781 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185034+0000", 2024-09-18T07:36:37.781 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5694000000000001e-05 2024-09-18T07:36:37.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.781 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.781 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.781 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185060+0000", 2024-09-18T07:36:37.781 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6231999999999999e-05 2024-09-18T07:36:37.782 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.782 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.782 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.782 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.782 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.782 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54283 7.c 7:30afc400:::benchmark_data_smithi137_74180_object54282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:36:37.782 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.244474+0000", 2024-09-18T07:36:37.782 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 224.48756531199999, 2024-09-18T07:36:37.782 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.94069020800000003, 2024-09-18T07:36:37.782 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.782 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.782 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:37.782 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.783 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.783 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.783 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54283 2024-09-18T07:36:37.783 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.783 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.783 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.783 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.783 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244474+0000", 2024-09-18T07:36:37.783 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.783 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.783 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.783 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.783 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244474+0000", 2024-09-18T07:36:37.783 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.784 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.784 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.784 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244476+0000", 2024-09-18T07:36:37.784 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6390000000000001e-06 2024-09-18T07:36:37.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.784 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.784 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.784 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244521+0000", 2024-09-18T07:36:37.784 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4379999999999999e-05 2024-09-18T07:36:37.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.785 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.785 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.785 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244538+0000", 2024-09-18T07:36:37.785 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7354e-05 2024-09-18T07:36:37.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.785 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.785 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.785 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244548+0000", 2024-09-18T07:36:37.785 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5030000000000003e-06 2024-09-18T07:36:37.786 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.786 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.786 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.786 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244624+0000", 2024-09-18T07:36:37.786 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6244999999999999e-05 2024-09-18T07:36:37.786 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.786 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.786 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.786 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244636+0000", 2024-09-18T07:36:37.786 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2218000000000001e-05 2024-09-18T07:36:37.786 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.786 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.786 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.787 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.172361+0000", 2024-09-18T07:36:37.787 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92772458400000002 2024-09-18T07:36:37.787 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.787 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.787 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:37.787 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.172455+0000", 2024-09-18T07:36:37.787 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4475999999999995e-05 2024-09-18T07:36:37.787 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.787 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.787 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:36:37.787 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.172532+0000", 2024-09-18T07:36:37.787 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6969000000000001e-05 2024-09-18T07:36:37.787 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.788 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.788 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:37.788 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.177152+0000", 2024-09-18T07:36:37.788 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0046201699999999998 2024-09-18T07:36:37.788 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.788 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.788 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.788 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.182361+0000", 2024-09-18T07:36:37.788 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0052089659999999998 2024-09-18T07:36:37.788 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.788 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.788 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.789 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185097+0000", 2024-09-18T07:36:37.789 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027359110000000002 2024-09-18T07:36:37.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.789 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.789 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:37.789 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185130+0000", 2024-09-18T07:36:37.789 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3031000000000003e-05 2024-09-18T07:36:37.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.789 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.789 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.789 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185164+0000", 2024-09-18T07:36:37.789 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3762e-05 2024-09-18T07:36:37.789 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.790 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.790 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.790 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.790 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.790 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54286 7.c 7:3a3f8689:::benchmark_data_smithi137_74180_object54285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:36:37.790 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.245256+0000", 2024-09-18T07:36:37.790 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 224.48678305600001, 2024-09-18T07:36:37.790 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93997832100000001, 2024-09-18T07:36:37.790 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.790 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.790 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:37.790 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.790 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.791 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.791 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54286 2024-09-18T07:36:37.791 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.791 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.791 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.791 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.791 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245256+0000", 2024-09-18T07:36:37.791 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.791 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.791 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.791 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.791 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245256+0000", 2024-09-18T07:36:37.792 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.792 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.792 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.792 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.792 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245258+0000", 2024-09-18T07:36:37.792 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.578e-06 2024-09-18T07:36:37.792 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.792 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.792 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.792 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245346+0000", 2024-09-18T07:36:37.792 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8776000000000005e-05 2024-09-18T07:36:37.792 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.792 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.793 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.793 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245361+0000", 2024-09-18T07:36:37.793 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4687e-05 2024-09-18T07:36:37.793 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.793 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.793 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.793 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245368+0000", 2024-09-18T07:36:37.793 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2989999999999999e-06 2024-09-18T07:36:37.793 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.793 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.793 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.793 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245437+0000", 2024-09-18T07:36:37.793 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8869999999999996e-05 2024-09-18T07:36:37.793 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.794 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.794 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.794 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245447+0000", 2024-09-18T07:36:37.794 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0172e-05 2024-09-18T07:36:37.794 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.794 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.794 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.794 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173386+0000", 2024-09-18T07:36:37.794 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92793816900000003 2024-09-18T07:36:37.794 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.794 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.794 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:37.794 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173452+0000", 2024-09-18T07:36:37.794 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6050999999999995e-05 2024-09-18T07:36:37.795 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.795 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.795 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:36:37.795 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173533+0000", 2024-09-18T07:36:37.795 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1578999999999999e-05 2024-09-18T07:36:37.795 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.795 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.795 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:37.795 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.177162+0000", 2024-09-18T07:36:37.795 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0036291240000000001 2024-09-18T07:36:37.795 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.795 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.795 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.796 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.182390+0000", 2024-09-18T07:36:37.796 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0052279880000000003 2024-09-18T07:36:37.796 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.796 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.796 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.796 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185188+0000", 2024-09-18T07:36:37.796 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027977779999999999 2024-09-18T07:36:37.796 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.796 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.796 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:37.796 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185215+0000", 2024-09-18T07:36:37.796 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6764999999999999e-05 2024-09-18T07:36:37.796 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:37.798 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.798 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.798 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185234+0000", 2024-09-18T07:36:37.798 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9485000000000001e-05 2024-09-18T07:36:37.798 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.798 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.799 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.799 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.800 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.800 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54303 7.c 7:3c96a443:::benchmark_data_smithi137_74180_object54302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:36:37.800 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.265470+0000", 2024-09-18T07:36:37.800 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 224.46656943400001, 2024-09-18T07:36:37.800 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92892820399999998, 2024-09-18T07:36:37.800 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.800 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.800 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:37.800 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.800 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.800 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.800 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54303 2024-09-18T07:36:37.800 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.801 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.801 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.801 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.801 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265470+0000", 2024-09-18T07:36:37.801 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.801 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.801 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.801 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.801 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265470+0000", 2024-09-18T07:36:37.801 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.801 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.801 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.802 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.802 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265472+0000", 2024-09-18T07:36:37.802 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2400000000000002e-06 2024-09-18T07:36:37.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.802 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.802 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.802 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265528+0000", 2024-09-18T07:36:37.802 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6047000000000001e-05 2024-09-18T07:36:37.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.802 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.802 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.802 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265539+0000", 2024-09-18T07:36:37.802 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1119e-05 2024-09-18T07:36:37.803 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.803 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.803 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.803 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265547+0000", 2024-09-18T07:36:37.803 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7470000000000005e-06 2024-09-18T07:36:37.803 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.803 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.803 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.803 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265600+0000", 2024-09-18T07:36:37.803 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3062000000000001e-05 2024-09-18T07:36:37.803 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.803 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.804 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.804 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265608+0000", 2024-09-18T07:36:37.804 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8669999999999994e-06 2024-09-18T07:36:37.804 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.804 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.804 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.804 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.174396+0000", 2024-09-18T07:36:37.804 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90878840299999997 2024-09-18T07:36:37.804 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.804 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.804 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:37.804 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.174505+0000", 2024-09-18T07:36:37.804 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108837 2024-09-18T07:36:37.805 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.805 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.805 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:36:37.805 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.174589+0000", 2024-09-18T07:36:37.805 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3825000000000005e-05 2024-09-18T07:36:37.805 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.805 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.805 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:37.805 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.178295+0000", 2024-09-18T07:36:37.805 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0037061170000000001 2024-09-18T07:36:37.805 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.805 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.805 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.806 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.188101+0000", 2024-09-18T07:36:37.806 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0098061350000000005 2024-09-18T07:36:37.806 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.806 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.806 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.806 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194313+0000", 2024-09-18T07:36:37.806 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0062117589999999999 2024-09-18T07:36:37.806 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.806 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.806 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:37.807 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194353+0000", 2024-09-18T07:36:37.807 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9869000000000001e-05 2024-09-18T07:36:37.807 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.807 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.807 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.807 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194398+0000", 2024-09-18T07:36:37.807 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5176999999999998e-05 2024-09-18T07:36:37.807 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.807 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.807 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.807 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.807 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.807 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54304 7.c 7:36172b18:::benchmark_data_smithi137_74180_object54303:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:36:37.808 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.265725+0000", 2024-09-18T07:36:37.808 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 224.46631410699999, 2024-09-18T07:36:37.808 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92876516499999995, 2024-09-18T07:36:37.808 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.808 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.808 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:37.808 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.808 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.808 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.808 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54304 2024-09-18T07:36:37.808 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.808 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.809 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.809 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.809 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265725+0000", 2024-09-18T07:36:37.809 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.809 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.809 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.809 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265725+0000", 2024-09-18T07:36:37.809 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.809 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.809 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.809 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265726+0000", 2024-09-18T07:36:37.810 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0449999999999999e-06 2024-09-18T07:36:37.810 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.810 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.810 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.810 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265785+0000", 2024-09-18T07:36:37.810 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.94e-05 2024-09-18T07:36:37.810 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.810 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.810 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.810 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265795+0000", 2024-09-18T07:36:37.810 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2550000000000003e-06 2024-09-18T07:36:37.810 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.810 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.811 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.811 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265802+0000", 2024-09-18T07:36:37.811 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8190000000000003e-06 2024-09-18T07:36:37.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.811 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.811 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.811 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265846+0000", 2024-09-18T07:36:37.811 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4867999999999998e-05 2024-09-18T07:36:37.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.811 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.812 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.812 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265853+0000", 2024-09-18T07:36:37.812 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0469999999999996e-06 2024-09-18T07:36:37.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.812 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.812 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.812 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.175383+0000", 2024-09-18T07:36:37.812 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90952946099999998 2024-09-18T07:36:37.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.812 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.812 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:37.812 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.175462+0000", 2024-09-18T07:36:37.812 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9513999999999998e-05 2024-09-18T07:36:37.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.813 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.813 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:36:37.813 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.175546+0000", 2024-09-18T07:36:37.813 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3614000000000003e-05 2024-09-18T07:36:37.813 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.813 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.813 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:37.813 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.178303+0000", 2024-09-18T07:36:37.813 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027565210000000001 2024-09-18T07:36:37.813 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.813 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.813 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.813 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194058+0000", 2024-09-18T07:36:37.813 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.015755377000000001 2024-09-18T07:36:37.814 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.814 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.814 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.814 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194432+0000", 2024-09-18T07:36:37.814 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00037386800000000002 2024-09-18T07:36:37.814 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.814 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.814 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:37.814 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194464+0000", 2024-09-18T07:36:37.814 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2330999999999999e-05 2024-09-18T07:36:37.814 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.814 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.814 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.815 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194490+0000", 2024-09-18T07:36:37.815 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6044999999999999e-05 2024-09-18T07:36:37.815 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.815 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.815 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.815 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.815 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.815 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54310 7.c 7:30b8e4f0:::benchmark_data_smithi137_74180_object54309:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:36:37.815 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.271051+0000", 2024-09-18T07:36:37.815 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 224.46098786300001, 2024-09-18T07:36:37.815 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92352962000000005, 2024-09-18T07:36:37.815 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.815 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.816 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:37.816 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.816 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.816 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.816 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54310 2024-09-18T07:36:37.816 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.816 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.816 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.816 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.816 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271051+0000", 2024-09-18T07:36:37.816 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.816 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.816 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.817 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.817 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271051+0000", 2024-09-18T07:36:37.817 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.817 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.817 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.817 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271053+0000", 2024-09-18T07:36:37.817 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.013e-06 2024-09-18T07:36:37.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.817 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.817 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.817 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271110+0000", 2024-09-18T07:36:37.817 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6329000000000002e-05 2024-09-18T07:36:37.818 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.818 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.818 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.818 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271121+0000", 2024-09-18T07:36:37.818 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1761e-05 2024-09-18T07:36:37.818 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.818 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.818 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.818 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271127+0000", 2024-09-18T07:36:37.818 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0390000000000001e-06 2024-09-18T07:36:37.818 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.818 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.818 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.819 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271189+0000", 2024-09-18T07:36:37.819 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1492e-05 2024-09-18T07:36:37.819 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.819 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.819 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.819 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271200+0000", 2024-09-18T07:36:37.819 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1195999999999999e-05 2024-09-18T07:36:37.819 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.819 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.819 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.819 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.176292+0000", 2024-09-18T07:36:37.819 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90509152199999998 2024-09-18T07:36:37.819 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.820 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.820 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:37.820 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.176362+0000", 2024-09-18T07:36:37.820 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9901999999999996e-05 2024-09-18T07:36:37.820 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.820 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.820 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:36:37.820 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.176445+0000", 2024-09-18T07:36:37.820 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3622000000000007e-05 2024-09-18T07:36:37.820 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.820 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.820 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:37.820 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.180339+0000", 2024-09-18T07:36:37.820 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0038937350000000002 2024-09-18T07:36:37.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.821 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.821 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.821 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194104+0000", 2024-09-18T07:36:37.821 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.013765370000000001 2024-09-18T07:36:37.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.821 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.821 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.821 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194523+0000", 2024-09-18T07:36:37.822 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00041917199999999999 2024-09-18T07:36:37.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.822 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.822 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:37.822 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194551+0000", 2024-09-18T07:36:37.822 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7489999999999999e-05 2024-09-18T07:36:37.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.822 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.822 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.822 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194581+0000", 2024-09-18T07:36:37.822 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9977000000000001e-05 2024-09-18T07:36:37.822 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.822 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.822 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.823 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.823 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153406 7.7 7:e5535fc6:::benchmark_data_smithi137_74180_object48990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-18T07:36:37.823 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.169823+0000", 2024-09-18T07:36:37.823 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 115.562215643, 2024-09-18T07:36:37.823 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.083783358, 2024-09-18T07:36:37.823 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.823 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.823 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:36:37.823 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.823 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.823 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.823 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153406 2024-09-18T07:36:37.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.824 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.824 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.824 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.824 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169823+0000", 2024-09-18T07:36:37.824 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.824 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.824 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.824 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169823+0000", 2024-09-18T07:36:37.824 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.824 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.824 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.825 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169826+0000", 2024-09-18T07:36:37.825 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1229999999999998e-06 2024-09-18T07:36:37.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.825 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.825 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.825 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169833+0000", 2024-09-18T07:36:37.825 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6869999999999994e-06 2024-09-18T07:36:37.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.825 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.825 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.825 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169851+0000", 2024-09-18T07:36:37.825 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7978999999999998e-05 2024-09-18T07:36:37.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.826 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.826 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.826 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169867+0000", 2024-09-18T07:36:37.826 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5610000000000001e-05 2024-09-18T07:36:37.826 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.826 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.826 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.826 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169944+0000", 2024-09-18T07:36:37.826 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6998999999999999e-05 2024-09-18T07:36:37.826 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.826 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.826 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:37.826 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.170029+0000", 2024-09-18T07:36:37.826 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4765999999999994e-05 2024-09-18T07:36:37.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.827 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.827 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:36:37.827 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.170285+0000", 2024-09-18T07:36:37.827 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025587199999999998 2024-09-18T07:36:37.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.827 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.827 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:37.827 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.171936+0000", 2024-09-18T07:36:37.827 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001651189 2024-09-18T07:36:37.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.827 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.827 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.828 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.194310+0000", 2024-09-18T07:36:37.828 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.022374616999999999 2024-09-18T07:36:37.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.828 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.828 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.828 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238955+0000", 2024-09-18T07:36:37.828 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.044644716000000001 2024-09-18T07:36:37.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.828 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.828 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.828 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238967+0000", 2024-09-18T07:36:37.829 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1933000000000001e-05 2024-09-18T07:36:37.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.829 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.829 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.829 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.253578+0000", 2024-09-18T07:36:37.829 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0146105599999999 2024-09-18T07:36:37.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.829 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.829 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.829 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.253607+0000", 2024-09-18T07:36:37.829 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9306999999999999e-05 2024-09-18T07:36:37.829 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.829 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.829 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.830 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.830 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153419 7.17 7:e887289b:::benchmark_data_smithi137_74180_object49003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-18T07:36:37.830 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.193652+0000", 2024-09-18T07:36:37.830 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 115.538386819, 2024-09-18T07:36:37.830 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0605730499999999, 2024-09-18T07:36:37.830 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.830 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.830 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:36:37.830 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.830 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.830 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.830 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153419 2024-09-18T07:36:37.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.831 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.831 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.831 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.831 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193652+0000", 2024-09-18T07:36:37.831 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.831 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.831 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.831 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193652+0000", 2024-09-18T07:36:37.831 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.831 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.831 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.831 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193654+0000", 2024-09-18T07:36:37.832 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.683e-06 2024-09-18T07:36:37.832 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.832 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.832 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.832 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193658+0000", 2024-09-18T07:36:37.832 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6619999999999998e-06 2024-09-18T07:36:37.832 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.832 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.832 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.832 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193667+0000", 2024-09-18T07:36:37.832 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0389999999999993e-06 2024-09-18T07:36:37.832 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.833 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.833 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.833 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193671+0000", 2024-09-18T07:36:37.833 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6789999999999996e-06 2024-09-18T07:36:37.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.833 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.833 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.833 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193715+0000", 2024-09-18T07:36:37.833 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3186000000000003e-05 2024-09-18T07:36:37.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.833 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.833 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:37.833 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193799+0000", 2024-09-18T07:36:37.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4172999999999995e-05 2024-09-18T07:36:37.834 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:36:37.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.194037+0000", 2024-09-18T07:36:37.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023850299999999999 2024-09-18T07:36:37.834 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:37.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.195050+0000", 2024-09-18T07:36:37.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010123269999999999 2024-09-18T07:36:37.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.835 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.835 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.198824+0000", 2024-09-18T07:36:37.835 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0037742840000000001 2024-09-18T07:36:37.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.835 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.835 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238955+0000", 2024-09-18T07:36:37.835 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.040130714999999997 2024-09-18T07:36:37.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.835 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.835 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238967+0000", 2024-09-18T07:36:37.836 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2665e-05 2024-09-18T07:36:37.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.836 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.836 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.836 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254201+0000", 2024-09-18T07:36:37.836 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0152341279999999 2024-09-18T07:36:37.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.836 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.836 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.836 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254225+0000", 2024-09-18T07:36:37.836 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4006e-05 2024-09-18T07:36:37.836 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.836 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.837 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.837 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.837 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153453 7.7 7.146f22e7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:36:37.837 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.282355+0000", 2024-09-18T07:36:37.837 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 115.449683932, 2024-09-18T07:36:37.837 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97169607199999997, 2024-09-18T07:36:37.837 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.837 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.837 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:36:37.837 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.837 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.837 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.838 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153453 2024-09-18T07:36:37.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.838 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.838 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.838 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.838 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282355+0000", 2024-09-18T07:36:37.838 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.838 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.838 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.838 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282355+0000", 2024-09-18T07:36:37.838 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.838 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.838 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.839 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282357+0000", 2024-09-18T07:36:37.839 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2110000000000001e-06 2024-09-18T07:36:37.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.839 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.839 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.839 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282363+0000", 2024-09-18T07:36:37.839 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6740000000000002e-06 2024-09-18T07:36:37.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.839 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.839 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.839 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282377+0000", 2024-09-18T07:36:37.839 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4039000000000001e-05 2024-09-18T07:36:37.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.839 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.840 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.840 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282384+0000", 2024-09-18T07:36:37.840 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5769999999999999e-06 2024-09-18T07:36:37.840 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.840 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.840 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.840 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282438+0000", 2024-09-18T07:36:37.840 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4083999999999999e-05 2024-09-18T07:36:37.840 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.840 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.840 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.840 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282445+0000", 2024-09-18T07:36:37.840 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7030000000000006e-06 2024-09-18T07:36:37.840 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.840 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.841 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.841 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254044+0000", 2024-09-18T07:36:37.841 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97159860200000003 2024-09-18T07:36:37.841 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.841 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.841 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.841 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254051+0000", 2024-09-18T07:36:37.841 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.182e-06 2024-09-18T07:36:37.841 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.841 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.841 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.841 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.841 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.841 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153481 7.7 7.33bf19e7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:36:37.842 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.317589+0000", 2024-09-18T07:36:37.842 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 115.414449928, 2024-09-18T07:36:37.842 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93650330299999995, 2024-09-18T07:36:37.842 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.842 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.842 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:36:37.842 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.842 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.842 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.842 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153481 2024-09-18T07:36:37.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.842 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.842 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.842 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.842 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317589+0000", 2024-09-18T07:36:37.842 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.843 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.843 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317589+0000", 2024-09-18T07:36:37.843 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.843 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.843 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317590+0000", 2024-09-18T07:36:37.843 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6499999999999999e-07 2024-09-18T07:36:37.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.843 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.844 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317593+0000", 2024-09-18T07:36:37.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7870000000000002e-06 2024-09-18T07:36:37.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.844 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.844 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.844 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317604+0000", 2024-09-18T07:36:37.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0953e-05 2024-09-18T07:36:37.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.844 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.844 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.844 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317609+0000", 2024-09-18T07:36:37.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4099999999999999e-06 2024-09-18T07:36:37.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.844 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.845 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.845 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317654+0000", 2024-09-18T07:36:37.845 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5333000000000001e-05 2024-09-18T07:36:37.845 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.845 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.845 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.845 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317661+0000", 2024-09-18T07:36:37.845 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8310000000000003e-06 2024-09-18T07:36:37.845 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.845 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.845 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.845 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254083+0000", 2024-09-18T07:36:37.845 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93642186500000002 2024-09-18T07:36:37.846 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.846 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.846 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.846 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254093+0000", 2024-09-18T07:36:37.846 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4590000000000004e-06 2024-09-18T07:36:37.846 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.846 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.846 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.846 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.846 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.846 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153485 7.7 7.c6dac07 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:36:37.846 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.328201+0000", 2024-09-18T07:36:37.846 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 115.403837991, 2024-09-18T07:36:37.846 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.925907068, 2024-09-18T07:36:37.846 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.847 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.847 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:36:37.847 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.847 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.847 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.847 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153485 2024-09-18T07:36:37.847 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.847 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.847 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.847 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.847 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328201+0000", 2024-09-18T07:36:37.847 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.847 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.847 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.848 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.848 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328201+0000", 2024-09-18T07:36:37.848 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.848 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.848 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.848 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.848 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328203+0000", 2024-09-18T07:36:37.848 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3769999999999999e-06 2024-09-18T07:36:37.848 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.848 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328207+0000", 2024-09-18T07:36:37.849 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5120000000000002e-06 2024-09-18T07:36:37.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.849 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328217+0000", 2024-09-18T07:36:37.849 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0438000000000001e-05 2024-09-18T07:36:37.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.849 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328223+0000", 2024-09-18T07:36:37.849 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7130000000000002e-06 2024-09-18T07:36:37.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.850 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.850 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328270+0000", 2024-09-18T07:36:37.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7243000000000002e-05 2024-09-18T07:36:37.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.850 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.850 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328277+0000", 2024-09-18T07:36:37.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9229999999999996e-06 2024-09-18T07:36:37.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.851 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.851 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254104+0000", 2024-09-18T07:36:37.851 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92582690099999998 2024-09-18T07:36:37.851 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.851 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.851 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.851 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254108+0000", 2024-09-18T07:36:37.851 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9609999999999999e-06 2024-09-18T07:36:37.851 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.851 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.851 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.851 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.851 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.852 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153505 7.7 7.fb8b6b67 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:36:37.852 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.350178+0000", 2024-09-18T07:36:37.852 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 115.38186116200001, 2024-09-18T07:36:37.852 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90398860000000003, 2024-09-18T07:36:37.852 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.852 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.852 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:36:37.852 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.852 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.852 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.852 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153505 2024-09-18T07:36:37.852 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.852 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.852 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.852 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.853 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350178+0000", 2024-09-18T07:36:37.853 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.853 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.853 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.853 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.853 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350178+0000", 2024-09-18T07:36:37.853 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.853 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.853 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.853 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.853 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350179+0000", 2024-09-18T07:36:37.853 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.454e-06 2024-09-18T07:36:37.853 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.854 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.854 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.854 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350183+0000", 2024-09-18T07:36:37.854 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8480000000000003e-06 2024-09-18T07:36:37.854 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.854 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.854 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.854 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350192+0000", 2024-09-18T07:36:37.854 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0899999999999994e-06 2024-09-18T07:36:37.854 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.854 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.854 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.854 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350200+0000", 2024-09-18T07:36:37.854 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.328e-06 2024-09-18T07:36:37.855 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.855 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.855 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.855 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350252+0000", 2024-09-18T07:36:37.855 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2589000000000001e-05 2024-09-18T07:36:37.855 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.855 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.855 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.855 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350263+0000", 2024-09-18T07:36:37.855 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0831e-05 2024-09-18T07:36:37.855 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.855 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.855 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.856 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254124+0000", 2024-09-18T07:36:37.856 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90386114100000003 2024-09-18T07:36:37.856 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.856 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.856 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.856 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254167+0000", 2024-09-18T07:36:37.856 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2318999999999999e-05 2024-09-18T07:36:37.856 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.856 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.856 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.856 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.856 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.856 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153519 7.17 7.5ef16c17 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:36:37.857 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.363819+0000", 2024-09-18T07:36:37.857 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 115.368220409, 2024-09-18T07:36:37.857 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.89046919899999999, 2024-09-18T07:36:37.857 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.857 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.857 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:36:37.857 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.857 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.857 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.857 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153519 2024-09-18T07:36:37.857 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.857 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.857 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.858 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.858 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363819+0000", 2024-09-18T07:36:37.858 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.859 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.859 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.859 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.859 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363819+0000", 2024-09-18T07:36:37.859 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.859 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.859 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.859 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.859 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363820+0000", 2024-09-18T07:36:37.859 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.325e-06 2024-09-18T07:36:37.859 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.860 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.860 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.860 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363824+0000", 2024-09-18T07:36:37.860 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2459999999999997e-06 2024-09-18T07:36:37.860 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.860 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.860 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.860 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363834+0000", 2024-09-18T07:36:37.860 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6399999999999992e-06 2024-09-18T07:36:37.861 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.861 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.861 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.861 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363838+0000", 2024-09-18T07:36:37.861 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2509999999999996e-06 2024-09-18T07:36:37.861 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.861 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.861 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.861 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363900+0000", 2024-09-18T07:36:37.861 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1902000000000005e-05 2024-09-18T07:36:37.861 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.861 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.862 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.862 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363910+0000", 2024-09-18T07:36:37.862 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6879999999999994e-06 2024-09-18T07:36:37.862 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.862 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.862 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.862 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254254+0000", 2024-09-18T07:36:37.862 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.89034411800000002 2024-09-18T07:36:37.862 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.862 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.862 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.862 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254288+0000", 2024-09-18T07:36:37.862 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4029000000000002e-05 2024-09-18T07:36:37.863 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.863 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.863 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.863 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.863 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.863 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:162338 7.c 7:39946bc9:::benchmark_data_smithi137_74180_object57922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:36:37.863 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:01.426143+0000", 2024-09-18T07:36:37.863 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 96.305896598000004, 2024-09-18T07:36:37.863 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0750540070000001, 2024-09-18T07:36:37.863 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.863 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.863 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:37.863 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.864 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.864 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.864 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 162338 2024-09-18T07:36:37.864 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.864 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.864 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.864 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.864 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426143+0000", 2024-09-18T07:36:37.864 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.864 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.864 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.864 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.864 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426143+0000", 2024-09-18T07:36:37.865 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.865 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.865 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.865 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.865 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426145+0000", 2024-09-18T07:36:37.865 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4059999999999999e-06 2024-09-18T07:36:37.865 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.865 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.865 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.865 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426151+0000", 2024-09-18T07:36:37.865 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1260000000000003e-06 2024-09-18T07:36:37.865 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.866 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.866 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.866 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426174+0000", 2024-09-18T07:36:37.866 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3255e-05 2024-09-18T07:36:37.866 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.866 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.866 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.866 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426182+0000", 2024-09-18T07:36:37.866 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4189999999999996e-06 2024-09-18T07:36:37.866 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.866 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.866 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.866 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426256+0000", 2024-09-18T07:36:37.866 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4713000000000001e-05 2024-09-18T07:36:37.867 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.867 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.867 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.867 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426270+0000", 2024-09-18T07:36:37.867 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3405e-05 2024-09-18T07:36:37.867 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.867 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.867 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.867 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.491167+0000", 2024-09-18T07:36:37.867 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.064897582 2024-09-18T07:36:37.867 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.867 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.867 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:37.868 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.491280+0000", 2024-09-18T07:36:37.868 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011243099999999999 2024-09-18T07:36:37.868 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.868 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.868 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3,7", 2024-09-18T07:36:37.868 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.491357+0000", 2024-09-18T07:36:37.868 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7385999999999994e-05 2024-09-18T07:36:37.868 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.868 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.868 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.868 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496060+0000", 2024-09-18T07:36:37.868 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0047024600000000003 2024-09-18T07:36:37.868 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.869 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.869 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:37.869 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496072+0000", 2024-09-18T07:36:37.869 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2294e-05 2024-09-18T07:36:37.869 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.869 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.869 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.869 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496109+0000", 2024-09-18T07:36:37.869 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7218999999999997e-05 2024-09-18T07:36:37.869 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.869 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.869 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.869 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501071+0000", 2024-09-18T07:36:37.870 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0049614660000000003 2024-09-18T07:36:37.870 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.870 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.870 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:37.870 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501111+0000", 2024-09-18T07:36:37.870 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0776999999999999e-05 2024-09-18T07:36:37.870 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.870 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.870 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.870 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501197+0000", 2024-09-18T07:36:37.870 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5068000000000006e-05 2024-09-18T07:36:37.870 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.870 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.871 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.871 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.871 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.871 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:162350 7.c 7:3843094b:::benchmark_data_smithi137_74180_object57934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:36:37.871 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:01.433525+0000", 2024-09-18T07:36:37.871 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 96.298513787999994, 2024-09-18T07:36:37.871 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0678760330000001, 2024-09-18T07:36:37.871 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:37.871 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:37.871 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:37.871 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:37.871 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:37.871 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:37.872 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 162350 2024-09-18T07:36:37.872 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.872 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:37.872 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.872 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:37.872 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433525+0000", 2024-09-18T07:36:37.872 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.872 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.872 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.872 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:37.872 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433525+0000", 2024-09-18T07:36:37.872 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:37.872 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.872 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.872 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:37.873 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433527+0000", 2024-09-18T07:36:37.873 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.812e-06 2024-09-18T07:36:37.873 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.873 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.873 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:37.873 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433533+0000", 2024-09-18T07:36:37.873 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2920000000000001e-06 2024-09-18T07:36:37.873 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.873 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.873 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:37.873 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433544+0000", 2024-09-18T07:36:37.873 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0339999999999999e-05 2024-09-18T07:36:37.873 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.874 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.874 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:37.874 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433551+0000", 2024-09-18T07:36:37.874 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6490000000000008e-06 2024-09-18T07:36:37.874 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.874 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.874 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.874 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433620+0000", 2024-09-18T07:36:37.874 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8957999999999999e-05 2024-09-18T07:36:37.874 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.874 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.874 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:37.874 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433631+0000", 2024-09-18T07:36:37.874 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0397000000000001e-05 2024-09-18T07:36:37.874 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.875 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.875 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:37.875 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.493805+0000", 2024-09-18T07:36:37.875 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0601744559999999 2024-09-18T07:36:37.875 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.875 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.875 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:37.875 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.493907+0000", 2024-09-18T07:36:37.875 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000101377 2024-09-18T07:36:37.875 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.875 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.875 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3,7", 2024-09-18T07:36:37.875 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.493992+0000", 2024-09-18T07:36:37.875 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5672000000000002e-05 2024-09-18T07:36:37.876 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.876 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.876 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.876 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496146+0000", 2024-09-18T07:36:37.876 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021537470000000001 2024-09-18T07:36:37.876 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.876 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.876 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:37.876 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.498397+0000", 2024-09-18T07:36:37.876 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002250742 2024-09-18T07:36:37.876 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.876 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.876 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.877 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501029+0000", 2024-09-18T07:36:37.877 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026321780000000002 2024-09-18T07:36:37.877 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.877 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.877 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:37.877 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501274+0000", 2024-09-18T07:36:37.877 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000245472 2024-09-18T07:36:37.877 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.877 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.877 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:37.877 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501321+0000", 2024-09-18T07:36:37.877 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6405e-05 2024-09-18T07:36:37.877 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:37.877 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:37.878 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:37.878 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501401+0000", 2024-09-18T07:36:37.878 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0536000000000003e-05 2024-09-18T07:36:37.878 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.878 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.878 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.878 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:37.878 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:37.878 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:36:37.917 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:37.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:37.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:37.922 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 32 16 61650 61634 120.355 86.75 0.00755609 0.00830214 2024-09-18T07:36:37.935 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:36:37.936 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:36:37.936 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:36:37.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd out 6 2024-09-18T07:36:37.961 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:36:37.961 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:36:38.000 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:36:38.000 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:36:38.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:38.046 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:37 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1845312993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:38.046 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:36:38.047 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:36:38.047 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:37 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1845312993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:38.086 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:36:38.086 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:36:38.114 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:38.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:38.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:38.118 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:36:38.118 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:36:38.156 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:36:38.156 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:36:38.190 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:36:38.190 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:36:38.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:37 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1845312993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:38.216 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:38.216 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:38.220 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:36:38.220 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:36:38.243 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:38.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:38.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:38.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:38.384 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:38.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:38.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:38.487 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:38.522 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:38.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:38.583 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T07:36:38.623 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:38.660 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:38.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:38.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:38.762 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:38.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:38.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:38.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:38.903 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:38.922 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 33 16 63006 62990 119.276 84.75 0.00783578 0.00837909 2024-09-18T07:36:38.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:38.954 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:38.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:38.984 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:36:39.062 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:39.095 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:39.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:39.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:39.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:39.230 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:39.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:39.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:39.269 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:38 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:39.269 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:38 smithi203 ceph-mon[32237]: pgmap v303: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 121 MiB/s wr, 1.94k op/s 2024-09-18T07:36:39.269 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:38 smithi203 ceph-mon[32237]: 6.10 scrub starts 2024-09-18T07:36:39.269 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:38 smithi203 ceph-mon[32237]: 6.10 scrub ok 2024-09-18T07:36:39.333 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:39.362 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:39.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:39.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:39.386 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:38 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:39.386 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:38 smithi137 ceph-mon[29408]: pgmap v303: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 121 MiB/s wr, 1.94k op/s 2024-09-18T07:36:39.386 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:38 smithi137 ceph-mon[29408]: 6.10 scrub starts 2024-09-18T07:36:39.386 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:38 smithi137 ceph-mon[29408]: 6.10 scrub ok 2024-09-18T07:36:39.386 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:38 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:39.386 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:38 smithi137 ceph-mon[39175]: pgmap v303: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 121 MiB/s wr, 1.94k op/s 2024-09-18T07:36:39.387 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:38 smithi137 ceph-mon[39175]: 6.10 scrub starts 2024-09-18T07:36:39.387 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:38 smithi137 ceph-mon[39175]: 6.10 scrub ok 2024-09-18T07:36:39.465 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:39.502 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:39.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:39.554 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:36:39.554 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:36:39.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:39.646 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:39.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:39.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:39.701 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:39.748 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:39.778 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:39.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:39.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:39.879 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:39.907 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:39.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:39.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:39.922 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 34 16 64415 64399 118.358 88.0625 0.0172641 0.00844266 2024-09-18T07:36:40.009 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:40.038 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:40.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:40.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:40.140 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:40.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:39 smithi137 ceph-mon[29408]: 5.19 scrub starts 2024-09-18T07:36:40.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:39 smithi137 ceph-mon[29408]: 5.19 scrub ok 2024-09-18T07:36:40.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:39 smithi137 ceph-mon[29408]: 6.f scrub starts 2024-09-18T07:36:40.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:39 smithi137 ceph-mon[29408]: 4.12 scrub starts 2024-09-18T07:36:40.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:39 smithi137 ceph-mon[29408]: 6.f scrub ok 2024-09-18T07:36:40.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:39 smithi137 ceph-mon[29408]: 4.12 scrub ok 2024-09-18T07:36:40.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:39 smithi137 ceph-mon[39175]: 5.19 scrub starts 2024-09-18T07:36:40.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:39 smithi137 ceph-mon[39175]: 5.19 scrub ok 2024-09-18T07:36:40.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:39 smithi137 ceph-mon[39175]: 6.f scrub starts 2024-09-18T07:36:40.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:39 smithi137 ceph-mon[39175]: 4.12 scrub starts 2024-09-18T07:36:40.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:39 smithi137 ceph-mon[39175]: 6.f scrub ok 2024-09-18T07:36:40.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:39 smithi137 ceph-mon[39175]: 4.12 scrub ok 2024-09-18T07:36:40.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:40.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:40.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:40.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:39 smithi203 ceph-mon[32237]: 5.19 scrub starts 2024-09-18T07:36:40.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:39 smithi203 ceph-mon[32237]: 5.19 scrub ok 2024-09-18T07:36:40.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:39 smithi203 ceph-mon[32237]: 6.f scrub starts 2024-09-18T07:36:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:39 smithi203 ceph-mon[32237]: 4.12 scrub starts 2024-09-18T07:36:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:39 smithi203 ceph-mon[32237]: 6.f scrub ok 2024-09-18T07:36:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:39 smithi203 ceph-mon[32237]: 4.12 scrub ok 2024-09-18T07:36:40.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:40.290 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T07:36:40.323 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:40.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:40.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:40.425 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:40.461 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:40.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:40.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:40.562 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:40.599 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:40.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:40.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:40.682 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:36:40.701 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:40.743 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:40.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:40.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:40.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:40.866 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:40.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:40.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:40.923 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 35 16 65748 65732 117.357 83.3125 0.010984 0.00851624 2024-09-18T07:36:40.967 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:40.987 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:40 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a[29404]: 2024-09-18T07:36:40.726+0000 7fe5973a1640 -1 mon.a@0(leader).osd e217 definitely_dead 0 2024-09-18T07:36:41.007 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:36:41.007 INFO:teuthology.orchestra.run.smithi137.stderr:marked out osd.6. 2024-09-18T07:36:41.009 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:41.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:41.111 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:41.132 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:41.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:41.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:41.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:40 smithi203 ceph-mon[32237]: pgmap v304: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-18T07:36:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:40 smithi203 ceph-mon[32237]: 6.4 scrub starts 2024-09-18T07:36:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:40 smithi203 ceph-mon[32237]: 6.4 scrub ok 2024-09-18T07:36:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:40 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/319759633' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:36:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:40 smithi203 ceph-mon[32237]: from='client.25273 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:36:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:40 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/967050619' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-18T07:36:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:40 smithi203 ceph-mon[32237]: from='client.25279 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-18T07:36:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:40 smithi203 ceph-mon[32237]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-18T07:36:41.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:41.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:41.256 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:41.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:41.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:40 smithi137 ceph-mon[29408]: pgmap v304: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-18T07:36:41.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:40 smithi137 ceph-mon[29408]: 6.4 scrub starts 2024-09-18T07:36:41.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:40 smithi137 ceph-mon[29408]: 6.4 scrub ok 2024-09-18T07:36:41.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:40 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/319759633' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:36:41.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:40 smithi137 ceph-mon[29408]: from='client.25273 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:36:41.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:40 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/967050619' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-18T07:36:41.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:40 smithi137 ceph-mon[29408]: from='client.25279 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-18T07:36:41.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:40 smithi137 ceph-mon[29408]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-18T07:36:41.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:40 smithi137 ceph-mon[39175]: pgmap v304: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-18T07:36:41.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:40 smithi137 ceph-mon[39175]: 6.4 scrub starts 2024-09-18T07:36:41.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:40 smithi137 ceph-mon[39175]: 6.4 scrub ok 2024-09-18T07:36:41.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:40 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/319759633' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:36:41.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:40 smithi137 ceph-mon[39175]: from='client.25273 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:36:41.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:40 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/967050619' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-18T07:36:41.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:40 smithi137 ceph-mon[39175]: from='client.25279 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-18T07:36:41.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:40 smithi137 ceph-mon[39175]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-18T07:36:41.357 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:41.378 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:41.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:41.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:41.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:36:41.480 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:41.503 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:41.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:41.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:41.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:41.627 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:41.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:41.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:41.729 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:41.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:41.756 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:41.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:41.857 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:41.883 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:41.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:41.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:41.923 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 36 16 65899 65883 114.359 9.4375 0.00296888 0.00852826 2024-09-18T07:36:41.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:42.010 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:42.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:42.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:42.115 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:42.137 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:42.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:42.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:42.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:42.259 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:42.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:42.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:42.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:41 smithi203 ceph-mon[32237]: 4.10 scrub starts 2024-09-18T07:36:42.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:41 smithi203 ceph-mon[32237]: 4.10 scrub ok 2024-09-18T07:36:42.281 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:41 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:42.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:41 smithi203 ceph-mon[32237]: from='client.25273 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:36:42.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:41 smithi203 ceph-mon[32237]: from='client.25279 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-18T07:36:42.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:41 smithi203 ceph-mon[32237]: osdmap e218: 8 total, 8 up, 7 in 2024-09-18T07:36:42.282 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:41 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:36:42.362 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:42.382 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:36:42.383 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:36:42.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:42.384 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:42.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:42.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:41 smithi137 ceph-mon[39175]: 4.10 scrub starts 2024-09-18T07:36:42.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:41 smithi137 ceph-mon[39175]: 4.10 scrub ok 2024-09-18T07:36:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:41 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:41 smithi137 ceph-mon[39175]: from='client.25273 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:36:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:41 smithi137 ceph-mon[39175]: from='client.25279 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-18T07:36:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:41 smithi137 ceph-mon[39175]: osdmap e218: 8 total, 8 up, 7 in 2024-09-18T07:36:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:41 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:36:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:41 smithi137 ceph-mon[29408]: 4.10 scrub starts 2024-09-18T07:36:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:41 smithi137 ceph-mon[29408]: 4.10 scrub ok 2024-09-18T07:36:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:41 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:41 smithi137 ceph-mon[29408]: from='client.25273 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:36:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:41 smithi137 ceph-mon[29408]: from='client.25279 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-18T07:36:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:41 smithi137 ceph-mon[29408]: osdmap e218: 8 total, 8 up, 7 in 2024-09-18T07:36:42.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:41 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:36:42.484 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:42.506 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:42.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:42.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:42.608 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:42.630 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:42.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:42.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:42.648 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:42.732 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:42.754 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:42.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:42.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:42.855 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:42.877 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:42.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:42.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:42.923 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 37 16 65899 65883 111.269 0 - 0.00852826 2024-09-18T07:36:42.981 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:43.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:43.004 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:43.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:43.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:42 smithi203 ceph-mon[32237]: pgmap v306: 148 pgs: 2 unknown, 30 peering, 116 active+clean; 3.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-18T07:36:43.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:42 smithi203 ceph-mon[32237]: Health check failed: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-18T07:36:43.019 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:42 smithi203 ceph-mon[32237]: osdmap e219: 8 total, 8 up, 7 in 2024-09-18T07:36:43.069 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T07:36:43.105 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:43.131 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:43.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:42 smithi137 ceph-mon[29408]: pgmap v306: 148 pgs: 2 unknown, 30 peering, 116 active+clean; 3.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-18T07:36:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:42 smithi137 ceph-mon[29408]: Health check failed: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-18T07:36:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:42 smithi137 ceph-mon[29408]: osdmap e219: 8 total, 8 up, 7 in 2024-09-18T07:36:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:42 smithi137 ceph-mon[39175]: pgmap v306: 148 pgs: 2 unknown, 30 peering, 116 active+clean; 3.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-18T07:36:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:42 smithi137 ceph-mon[39175]: Health check failed: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-18T07:36:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:42 smithi137 ceph-mon[39175]: osdmap e219: 8 total, 8 up, 7 in 2024-09-18T07:36:43.233 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:43.264 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:43.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:43.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:43.366 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:43.396 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:43.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:43.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:43.438 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:36:43.451 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:36:43.451 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:36:43.486 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:36:43.487 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:36:43.498 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:43.522 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:36:43.523 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:36:43.524 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:43.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:43.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:43.557 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:36:43.557 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:36:43.586 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:36:43.587 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:36:43.621 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:36:43.621 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:36:43.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:43.653 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:36:43.653 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:36:43.663 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:43.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:43.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:43.687 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:36:43.687 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:36:43.765 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:43.800 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:43.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:43.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:43.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:43.923 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 38 16 66500 66484 109.328 18.7812 0.0313279 0.00913972 2024-09-18T07:36:43.937 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:43.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:43.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:43.991 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:36:43.991 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:36:43.991 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:36:43.991 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:36:43.991 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.992 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:21294 7.8 e89/81 7:15222838:::benchmark_data_smithi137_74180_object21293:head v 89'1394, mlcod=89'1392)", 2024-09-18T07:36:43.992 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.948825+0000", 2024-09-18T07:36:43.992 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 260.03826810499999, 2024-09-18T07:36:43.992 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.094945371000000001, 2024-09-18T07:36:43.992 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:43.992 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:43.992 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:43.992 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:43.992 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.992 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:43.992 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948825+0000", 2024-09-18T07:36:43.992 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:43.992 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:43.993 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.993 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:43.993 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948825+0000", 2024-09-18T07:36:43.993 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:43.993 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:43.993 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.993 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:43.993 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948826+0000", 2024-09-18T07:36:43.993 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3e-07 2024-09-18T07:36:43.993 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:43.993 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.993 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:43.993 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948939+0000", 2024-09-18T07:36:43.994 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000113757 2024-09-18T07:36:43.994 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:43.994 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.994 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:43.994 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948994+0000", 2024-09-18T07:36:43.994 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4622e-05 2024-09-18T07:36:43.994 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:43.994 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.994 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:43.994 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.949000+0000", 2024-09-18T07:36:43.994 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9599999999999997e-06 2024-09-18T07:36:43.994 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:43.995 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.995 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:43.995 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.949042+0000", 2024-09-18T07:36:43.995 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2558999999999997e-05 2024-09-18T07:36:43.995 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:43.995 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.995 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:43.995 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.949055+0000", 2024-09-18T07:36:43.995 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2969999999999999e-05 2024-09-18T07:36:43.995 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:43.995 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.995 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:43.995 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.043723+0000", 2024-09-18T07:36:43.996 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.094668034999999998 2024-09-18T07:36:43.996 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:43.996 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.996 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:43.996 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.043770+0000", 2024-09-18T07:36:43.996 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6838000000000001e-05 2024-09-18T07:36:43.996 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:43.996 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:43.996 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:43.996 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:43.996 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.996 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:21293 7.f e89/81 7:fa840974:::benchmark_data_smithi137_74180_object21292:head v 89'1337, mlcod=89'1334)", 2024-09-18T07:36:43.996 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.948830+0000", 2024-09-18T07:36:43.997 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 260.03826308599997, 2024-09-18T07:36:43.997 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.096476913999999997, 2024-09-18T07:36:43.997 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:43.997 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:43.997 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:43.997 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:43.997 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.997 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:43.997 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948830+0000", 2024-09-18T07:36:43.997 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:43.997 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:43.997 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.997 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:43.998 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948830+0000", 2024-09-18T07:36:43.998 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:43.998 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:43.998 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.998 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:43.998 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948831+0000", 2024-09-18T07:36:43.998 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.235e-06 2024-09-18T07:36:43.998 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:43.998 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.998 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:43.998 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948942+0000", 2024-09-18T07:36:43.998 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000110417 2024-09-18T07:36:43.998 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:43.999 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.999 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:43.999 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948954+0000", 2024-09-18T07:36:43.999 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1888999999999999e-05 2024-09-18T07:36:43.999 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:43.999 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.999 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:43.999 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948959+0000", 2024-09-18T07:36:43.999 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8259999999999998e-06 2024-09-18T07:36:43.999 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:43.999 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:43.999 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:43.999 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948998+0000", 2024-09-18T07:36:43.999 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8498999999999999e-05 2024-09-18T07:36:44.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.000 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.000 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.000 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.949009+0000", 2024-09-18T07:36:44.000 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1413999999999999e-05 2024-09-18T07:36:44.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.000 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.000 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:44.000 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.045250+0000", 2024-09-18T07:36:44.000 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.096240945999999994 2024-09-18T07:36:44.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.000 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.000 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.001 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.045307+0000", 2024-09-18T07:36:44.001 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6687999999999997e-05 2024-09-18T07:36:44.001 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.001 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.001 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.001 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.001 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.001 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21896 7.5 7:a849d1d6:::benchmark_data_smithi137_74180_object21895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:36:44.001 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.316136+0000", 2024-09-18T07:36:44.001 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 259.67095694400001, 2024-09-18T07:36:44.001 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.073939160000000004, 2024-09-18T07:36:44.001 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.001 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.001 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:44.002 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:44.002 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:44.002 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:44.002 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21896 2024-09-18T07:36:44.002 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.002 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.002 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.002 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.002 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316136+0000", 2024-09-18T07:36:44.002 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.002 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.002 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.002 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.002 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316136+0000", 2024-09-18T07:36:44.003 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.003 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.003 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316138+0000", 2024-09-18T07:36:44.003 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.426e-06 2024-09-18T07:36:44.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.003 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.003 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316190+0000", 2024-09-18T07:36:44.003 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2704000000000002e-05 2024-09-18T07:36:44.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.003 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.003 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316201+0000", 2024-09-18T07:36:44.004 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0298e-05 2024-09-18T07:36:44.004 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.004 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.004 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.004 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316205+0000", 2024-09-18T07:36:44.004 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8860000000000003e-06 2024-09-18T07:36:44.004 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.004 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.004 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.004 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316244+0000", 2024-09-18T07:36:44.004 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8535999999999998e-05 2024-09-18T07:36:44.004 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.004 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.004 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.005 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316328+0000", 2024-09-18T07:36:44.005 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4384999999999999e-05 2024-09-18T07:36:44.005 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.005 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.005 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:36:44.005 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316417+0000", 2024-09-18T07:36:44.005 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8794000000000004e-05 2024-09-18T07:36:44.005 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.005 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.005 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.005 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.388000+0000", 2024-09-18T07:36:44.005 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.071582792000000006 2024-09-18T07:36:44.005 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.005 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.006 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:44.006 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.388272+0000", 2024-09-18T07:36:44.006 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00027234600000000002 2024-09-18T07:36:44.006 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.007 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.007 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.007 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.389970+0000", 2024-09-18T07:36:44.007 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016974959999999999 2024-09-18T07:36:44.007 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.007 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.007 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:44.007 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.390033+0000", 2024-09-18T07:36:44.007 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3640999999999996e-05 2024-09-18T07:36:44.007 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.007 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.007 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.007 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.390075+0000", 2024-09-18T07:36:44.007 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1856e-05 2024-09-18T07:36:44.008 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.008 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.008 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.008 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.008 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.008 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop_reply(client.14865.0:21896 7.5 e89/81 ondisk, result = 0)", 2024-09-18T07:36:44.008 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.317455+0000", 2024-09-18T07:36:44.008 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 259.66963855400002, 2024-09-18T07:36:44.008 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.070552445000000005, 2024-09-18T07:36:44.008 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.008 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.008 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:36:44.009 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.009 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.009 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.009 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.317455+0000", 2024-09-18T07:36:44.009 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.009 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.009 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.009 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.009 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.317455+0000", 2024-09-18T07:36:44.009 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.009 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.009 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.009 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.009 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.317455+0000", 2024-09-18T07:36:44.010 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4399999999999998e-07 2024-09-18T07:36:44.010 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.010 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.010 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.317459+0000", 2024-09-18T07:36:44.010 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7749999999999999e-06 2024-09-18T07:36:44.010 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.010 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.010 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.386905+0000", 2024-09-18T07:36:44.010 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.069445328000000001 2024-09-18T07:36:44.010 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.010 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.011 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.386916+0000", 2024-09-18T07:36:44.011 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1331e-05 2024-09-18T07:36:44.011 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.011 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.011 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.011 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.387987+0000", 2024-09-18T07:36:44.011 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010713020000000001 2024-09-18T07:36:44.011 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.011 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.011 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.011 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.387995+0000", 2024-09-18T07:36:44.011 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3679999999999996e-06 2024-09-18T07:36:44.011 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.011 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.012 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.012 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.388007+0000", 2024-09-18T07:36:44.012 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2397e-05 2024-09-18T07:36:44.012 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.012 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.012 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.012 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.012 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.012 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21999 7.5 7:a6ca4f28:::benchmark_data_smithi137_74180_object21998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:36:44.012 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.424624+0000", 2024-09-18T07:36:44.012 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 259.56246906199999, 2024-09-18T07:36:44.012 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.112047436, 2024-09-18T07:36:44.012 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.013 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.013 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:44.013 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:44.013 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:44.013 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:44.013 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21999 2024-09-18T07:36:44.013 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.013 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.013 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.013 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.013 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424624+0000", 2024-09-18T07:36:44.013 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.013 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.014 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.014 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.014 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424624+0000", 2024-09-18T07:36:44.014 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.014 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.014 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.014 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.014 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424626+0000", 2024-09-18T07:36:44.014 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.075e-06 2024-09-18T07:36:44.014 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.014 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.014 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.014 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424682+0000", 2024-09-18T07:36:44.014 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5597999999999999e-05 2024-09-18T07:36:44.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424694+0000", 2024-09-18T07:36:44.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2656e-05 2024-09-18T07:36:44.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424702+0000", 2024-09-18T07:36:44.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3849999999999996e-06 2024-09-18T07:36:44.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.016 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424758+0000", 2024-09-18T07:36:44.016 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6020000000000002e-05 2024-09-18T07:36:44.016 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.016 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.016 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.016 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424884+0000", 2024-09-18T07:36:44.016 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012638800000000001 2024-09-18T07:36:44.016 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.016 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.016 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:36:44.016 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425020+0000", 2024-09-18T07:36:44.016 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013587300000000001 2024-09-18T07:36:44.016 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.017 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.017 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.017 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.426628+0000", 2024-09-18T07:36:44.017 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001607792 2024-09-18T07:36:44.017 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.017 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.017 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:44.017 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.426665+0000", 2024-09-18T07:36:44.017 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7052999999999999e-05 2024-09-18T07:36:44.017 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.017 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.017 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.017 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536540+0000", 2024-09-18T07:36:44.018 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10987469399999999 2024-09-18T07:36:44.018 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.018 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.018 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:44.018 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536614+0000", 2024-09-18T07:36:44.018 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4486999999999993e-05 2024-09-18T07:36:44.018 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.018 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.018 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.018 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536671+0000", 2024-09-18T07:36:44.018 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7414999999999998e-05 2024-09-18T07:36:44.018 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.018 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.019 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.019 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.019 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.019 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22002 7.5 7:a1dbf95a:::benchmark_data_smithi137_74180_object22001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:36:44.019 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.425607+0000", 2024-09-18T07:36:44.019 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 259.561486419, 2024-09-18T07:36:44.019 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.111294137, 2024-09-18T07:36:44.019 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.019 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.019 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:44.019 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:44.019 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:44.019 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:44.019 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22002 2024-09-18T07:36:44.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.020 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.020 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.020 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.020 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425607+0000", 2024-09-18T07:36:44.020 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.020 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.020 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.020 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425607+0000", 2024-09-18T07:36:44.020 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.021 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.021 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425608+0000", 2024-09-18T07:36:44.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.297e-06 2024-09-18T07:36:44.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.021 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.021 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425682+0000", 2024-09-18T07:36:44.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3891000000000001e-05 2024-09-18T07:36:44.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.021 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.021 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425693+0000", 2024-09-18T07:36:44.022 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0944000000000001e-05 2024-09-18T07:36:44.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.022 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.022 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425701+0000", 2024-09-18T07:36:44.022 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7489999999999998e-06 2024-09-18T07:36:44.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.022 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.022 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425809+0000", 2024-09-18T07:36:44.022 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108323 2024-09-18T07:36:44.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.022 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425883+0000", 2024-09-18T07:36:44.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4321999999999998e-05 2024-09-18T07:36:44.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.023 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.023 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:36:44.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.426014+0000", 2024-09-18T07:36:44.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013092400000000001 2024-09-18T07:36:44.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.023 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.023 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:44.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.426735+0000", 2024-09-18T07:36:44.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00072051000000000005 2024-09-18T07:36:44.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.024 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.427445+0000", 2024-09-18T07:36:44.024 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00070996799999999995 2024-09-18T07:36:44.024 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.024 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536803+0000", 2024-09-18T07:36:44.024 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10935832500000001 2024-09-18T07:36:44.024 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.024 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:44.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536856+0000", 2024-09-18T07:36:44.024 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3161000000000001e-05 2024-09-18T07:36:44.025 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.025 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.025 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.025 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536901+0000", 2024-09-18T07:36:44.025 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4722999999999999e-05 2024-09-18T07:36:44.025 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.025 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.025 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.025 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.025 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.025 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22010 7.5 7:a615b038:::benchmark_data_smithi137_74180_object22009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:36:44.025 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.428662+0000", 2024-09-18T07:36:44.025 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 259.55843078499998, 2024-09-18T07:36:44.026 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.11444004000000001, 2024-09-18T07:36:44.026 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.026 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.026 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:44.026 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:44.026 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:44.026 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:44.026 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22010 2024-09-18T07:36:44.026 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.026 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.026 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.026 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.026 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428662+0000", 2024-09-18T07:36:44.026 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428662+0000", 2024-09-18T07:36:44.027 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428664+0000", 2024-09-18T07:36:44.027 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.302e-06 2024-09-18T07:36:44.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428707+0000", 2024-09-18T07:36:44.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3692000000000001e-05 2024-09-18T07:36:44.028 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.028 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.028 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.028 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428718+0000", 2024-09-18T07:36:44.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1066999999999999e-05 2024-09-18T07:36:44.028 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.028 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.028 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.028 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428724+0000", 2024-09-18T07:36:44.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7590000000000003e-06 2024-09-18T07:36:44.028 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.028 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.028 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.029 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428771+0000", 2024-09-18T07:36:44.029 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6462999999999999e-05 2024-09-18T07:36:44.029 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.029 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.029 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.029 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428852+0000", 2024-09-18T07:36:44.029 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1533999999999996e-05 2024-09-18T07:36:44.029 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.029 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.029 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:36:44.029 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428937+0000", 2024-09-18T07:36:44.029 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5068999999999995e-05 2024-09-18T07:36:44.029 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.030 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.030 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:44.030 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.429584+0000", 2024-09-18T07:36:44.030 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00064701400000000001 2024-09-18T07:36:44.030 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.030 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.030 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.030 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.429931+0000", 2024-09-18T07:36:44.030 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00034729299999999998 2024-09-18T07:36:44.030 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.030 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.030 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.030 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.543034+0000", 2024-09-18T07:36:44.030 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.113102231 2024-09-18T07:36:44.031 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.031 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.031 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:44.031 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.543071+0000", 2024-09-18T07:36:44.031 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7333999999999999e-05 2024-09-18T07:36:44.031 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.031 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.031 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.031 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.543102+0000", 2024-09-18T07:36:44.031 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1282000000000003e-05 2024-09-18T07:36:44.031 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.031 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.031 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.031 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.032 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.032 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:22343 7.6 e89/81 7:6ec6138d:::benchmark_data_smithi137_74180_object22342:head v 89'1362, mlcod=89'1360)", 2024-09-18T07:36:44.032 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.661175+0000", 2024-09-18T07:36:44.032 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 259.32591792599999, 2024-09-18T07:36:44.032 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19171324300000001, 2024-09-18T07:36:44.032 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.032 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.032 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:44.032 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.032 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.032 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.032 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661175+0000", 2024-09-18T07:36:44.032 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.033 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.033 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.033 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661175+0000", 2024-09-18T07:36:44.033 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.033 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.033 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.033 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661177+0000", 2024-09-18T07:36:44.033 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.548e-06 2024-09-18T07:36:44.033 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.033 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.033 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661235+0000", 2024-09-18T07:36:44.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7877999999999997e-05 2024-09-18T07:36:44.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.034 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661244+0000", 2024-09-18T07:36:44.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2709999999999998e-06 2024-09-18T07:36:44.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.034 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661250+0000", 2024-09-18T07:36:44.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8619999999999999e-06 2024-09-18T07:36:44.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.034 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661288+0000", 2024-09-18T07:36:44.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8330000000000001e-05 2024-09-18T07:36:44.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661300+0000", 2024-09-18T07:36:44.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1483e-05 2024-09-18T07:36:44.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:44.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852855+0000", 2024-09-18T07:36:44.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19155582199999999 2024-09-18T07:36:44.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.036 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852888+0000", 2024-09-18T07:36:44.036 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3049000000000002e-05 2024-09-18T07:36:44.036 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.036 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.036 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.036 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.036 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.036 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:22347 7.f e89/81 7:fe6074eb:::benchmark_data_smithi137_74180_object22346:head v 89'1397, mlcod=89'1395)", 2024-09-18T07:36:44.036 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.662928+0000", 2024-09-18T07:36:44.036 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 259.32416543300002, 2024-09-18T07:36:44.036 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19092352700000001, 2024-09-18T07:36:44.036 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.036 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.037 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:44.037 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.037 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662928+0000", 2024-09-18T07:36:44.037 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.037 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662928+0000", 2024-09-18T07:36:44.037 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.038 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662929+0000", 2024-09-18T07:36:44.038 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2500000000000004e-07 2024-09-18T07:36:44.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.038 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.038 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.038 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662989+0000", 2024-09-18T07:36:44.038 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0466e-05 2024-09-18T07:36:44.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.038 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.038 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.038 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.851940+0000", 2024-09-18T07:36:44.038 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18895100100000001 2024-09-18T07:36:44.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.039 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.039 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.039 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.851968+0000", 2024-09-18T07:36:44.039 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7668e-05 2024-09-18T07:36:44.039 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.039 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.039 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.039 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852054+0000", 2024-09-18T07:36:44.039 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6370000000000001e-05 2024-09-18T07:36:44.039 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.039 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.039 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.039 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852082+0000", 2024-09-18T07:36:44.039 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7769000000000001e-05 2024-09-18T07:36:44.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.040 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.040 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:44.040 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.853810+0000", 2024-09-18T07:36:44.040 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001728168 2024-09-18T07:36:44.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.040 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.040 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.040 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.853851+0000", 2024-09-18T07:36:44.040 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1159999999999999e-05 2024-09-18T07:36:44.040 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.040 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.040 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.041 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22450 7.5 7:ad888e37:::benchmark_data_smithi137_74180_object22449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:36:44.041 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.977850+0000", 2024-09-18T07:36:44.041 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 259.00924353099998, 2024-09-18T07:36:44.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.077184682000000004, 2024-09-18T07:36:44.041 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.041 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.041 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:44.041 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:44.041 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:44.041 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:44.041 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22450 2024-09-18T07:36:44.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.042 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977850+0000", 2024-09-18T07:36:44.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977850+0000", 2024-09-18T07:36:44.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977852+0000", 2024-09-18T07:36:44.043 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.108e-06 2024-09-18T07:36:44.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.043 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.043 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977892+0000", 2024-09-18T07:36:44.043 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9928999999999997e-05 2024-09-18T07:36:44.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.043 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.043 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977904+0000", 2024-09-18T07:36:44.043 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2843999999999999e-05 2024-09-18T07:36:44.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.043 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.043 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977912+0000", 2024-09-18T07:36:44.044 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1169999999999998e-06 2024-09-18T07:36:44.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.044 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.044 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977968+0000", 2024-09-18T07:36:44.044 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6338000000000002e-05 2024-09-18T07:36:44.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.044 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.044 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.978050+0000", 2024-09-18T07:36:44.044 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1841000000000005e-05 2024-09-18T07:36:44.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.044 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:36:44.045 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.978130+0000", 2024-09-18T07:36:44.045 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9917000000000001e-05 2024-09-18T07:36:44.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.045 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.045 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:44.045 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.978838+0000", 2024-09-18T07:36:44.045 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000708594 2024-09-18T07:36:44.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.045 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.045 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.045 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.979190+0000", 2024-09-18T07:36:44.045 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00035225000000000002 2024-09-18T07:36:44.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.045 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.045 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.054919+0000", 2024-09-18T07:36:44.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.075728917000000007 2024-09-18T07:36:44.046 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.046 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:44.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.054990+0000", 2024-09-18T07:36:44.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0300000000000001e-05 2024-09-18T07:36:44.046 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.046 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.055034+0000", 2024-09-18T07:36:44.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4527000000000002e-05 2024-09-18T07:36:44.046 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.046 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.046 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.047 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.047 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.047 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:23078 7.5 7:af53196f:::benchmark_data_smithi137_74180_object23077:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:36:44.047 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.454467+0000", 2024-09-18T07:36:44.047 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 258.53262618000002, 2024-09-18T07:36:44.047 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.088575311000000004, 2024-09-18T07:36:44.047 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.047 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.047 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:44.047 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:44.047 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:44.047 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:44.047 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 23078 2024-09-18T07:36:44.047 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.048 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.048 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.048 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.048 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454467+0000", 2024-09-18T07:36:44.048 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.048 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:44.050 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.050 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.050 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454467+0000", 2024-09-18T07:36:44.050 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.050 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.050 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.050 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.051 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454469+0000", 2024-09-18T07:36:44.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6089999999999999e-06 2024-09-18T07:36:44.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.051 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.051 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.051 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454512+0000", 2024-09-18T07:36:44.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2608999999999998e-05 2024-09-18T07:36:44.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.051 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.051 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.051 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454525+0000", 2024-09-18T07:36:44.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3352e-05 2024-09-18T07:36:44.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.052 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454534+0000", 2024-09-18T07:36:44.052 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0560000000000005e-06 2024-09-18T07:36:44.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.052 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454591+0000", 2024-09-18T07:36:44.052 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7633999999999997e-05 2024-09-18T07:36:44.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.052 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454715+0000", 2024-09-18T07:36:44.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012419300000000001 2024-09-18T07:36:44.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:36:44.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454817+0000", 2024-09-18T07:36:44.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000101452 2024-09-18T07:36:44.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:44.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.455733+0000", 2024-09-18T07:36:44.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00091630599999999998 2024-09-18T07:36:44.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.054 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.456978+0000", 2024-09-18T07:36:44.054 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001245248 2024-09-18T07:36:44.054 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.054 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.054 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.054 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.542970+0000", 2024-09-18T07:36:44.054 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.085991206000000001 2024-09-18T07:36:44.054 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.054 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.054 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:44.054 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543010+0000", 2024-09-18T07:36:44.054 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0936999999999998e-05 2024-09-18T07:36:44.054 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.054 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.054 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.055 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543042+0000", 2024-09-18T07:36:44.055 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1708999999999997e-05 2024-09-18T07:36:44.055 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.055 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.055 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.055 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.055 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.055 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:23089 7.5 7:abb1cc32:::benchmark_data_smithi137_74180_object23088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:36:44.055 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.457769+0000", 2024-09-18T07:36:44.055 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 258.529324202, 2024-09-18T07:36:44.055 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.085838029999999996, 2024-09-18T07:36:44.055 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.055 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.055 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:44.055 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:44.056 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:44.056 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:44.056 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 23089 2024-09-18T07:36:44.056 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.056 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.056 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.056 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.056 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457769+0000", 2024-09-18T07:36:44.056 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.056 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.056 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.056 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.056 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457769+0000", 2024-09-18T07:36:44.057 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.057 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.057 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.057 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.057 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457771+0000", 2024-09-18T07:36:44.057 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3199999999999998e-06 2024-09-18T07:36:44.057 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.057 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.057 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.057 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457831+0000", 2024-09-18T07:36:44.057 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9939999999999999e-05 2024-09-18T07:36:44.057 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.057 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.057 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.058 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457848+0000", 2024-09-18T07:36:44.058 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6682999999999999e-05 2024-09-18T07:36:44.058 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.058 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.058 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.058 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457856+0000", 2024-09-18T07:36:44.058 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0809999999999994e-06 2024-09-18T07:36:44.058 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.058 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.058 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.058 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457909+0000", 2024-09-18T07:36:44.058 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3143999999999998e-05 2024-09-18T07:36:44.058 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.059 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.059 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.458075+0000", 2024-09-18T07:36:44.059 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000166398 2024-09-18T07:36:44.059 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.059 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:36:44.059 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.458209+0000", 2024-09-18T07:36:44.059 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013364400000000001 2024-09-18T07:36:44.059 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.059 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:44.059 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.459558+0000", 2024-09-18T07:36:44.059 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0013487519999999999 2024-09-18T07:36:44.060 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.060 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.060 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.060 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.459987+0000", 2024-09-18T07:36:44.060 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00042892000000000001 2024-09-18T07:36:44.060 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.060 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.060 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.060 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543512+0000", 2024-09-18T07:36:44.060 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.083524980999999998 2024-09-18T07:36:44.060 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.060 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.060 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:44.060 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543569+0000", 2024-09-18T07:36:44.060 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7337999999999999e-05 2024-09-18T07:36:44.061 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.061 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.061 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.061 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543607+0000", 2024-09-18T07:36:44.061 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7829e-05 2024-09-18T07:36:44.061 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.061 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.061 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.061 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.061 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.061 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:26891 7.5 7:a92f246c:::benchmark_data_smithi137_74180_object26890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-18T07:36:44.061 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:27.401622+0000", 2024-09-18T07:36:44.061 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 256.58547097299999, 2024-09-18T07:36:44.062 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.080096663999999998, 2024-09-18T07:36:44.062 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.062 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.062 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:44.062 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:44.062 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:44.062 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:44.062 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 26891 2024-09-18T07:36:44.062 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.062 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.062 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.062 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.062 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401622+0000", 2024-09-18T07:36:44.062 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.062 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.062 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.063 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.063 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401622+0000", 2024-09-18T07:36:44.063 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.063 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.063 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.063 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.063 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401624+0000", 2024-09-18T07:36:44.063 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0880000000000002e-06 2024-09-18T07:36:44.063 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.063 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.063 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.063 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401694+0000", 2024-09-18T07:36:44.063 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9825000000000003e-05 2024-09-18T07:36:44.064 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.064 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.064 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.064 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401708+0000", 2024-09-18T07:36:44.064 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4249e-05 2024-09-18T07:36:44.064 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.064 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.064 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.064 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401716+0000", 2024-09-18T07:36:44.064 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0460000000000006e-06 2024-09-18T07:36:44.064 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.064 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.064 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.064 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401785+0000", 2024-09-18T07:36:44.064 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8583000000000004e-05 2024-09-18T07:36:44.065 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.065 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.065 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.065 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401892+0000", 2024-09-18T07:36:44.065 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000106885 2024-09-18T07:36:44.065 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.065 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.065 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:36:44.065 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402022+0000", 2024-09-18T07:36:44.065 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012980700000000001 2024-09-18T07:36:44.065 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.065 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.065 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:44.066 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.404167+0000", 2024-09-18T07:36:44.066 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021457550000000001 2024-09-18T07:36:44.066 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.066 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.066 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.066 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.404212+0000", 2024-09-18T07:36:44.066 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5052000000000002e-05 2024-09-18T07:36:44.066 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.066 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.066 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.066 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481614+0000", 2024-09-18T07:36:44.066 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.077401247000000006 2024-09-18T07:36:44.066 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.067 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.067 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:44.067 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481666+0000", 2024-09-18T07:36:44.067 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2198999999999999e-05 2024-09-18T07:36:44.067 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.067 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.067 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.067 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481719+0000", 2024-09-18T07:36:44.067 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2927999999999999e-05 2024-09-18T07:36:44.067 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.067 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.067 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.067 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.067 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.068 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:26893 7.5 7:a23e6cde:::benchmark_data_smithi137_74180_object26892:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-18T07:36:44.068 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:27.402662+0000", 2024-09-18T07:36:44.068 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 256.58443077700002, 2024-09-18T07:36:44.068 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.079194041000000007, 2024-09-18T07:36:44.068 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.068 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.068 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:44.068 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:44.068 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:44.068 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:44.068 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 26893 2024-09-18T07:36:44.068 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.069 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.069 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.069 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.069 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402662+0000", 2024-09-18T07:36:44.069 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.069 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.069 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.069 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402662+0000", 2024-09-18T07:36:44.069 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.069 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.069 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.069 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402664+0000", 2024-09-18T07:36:44.070 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.513e-06 2024-09-18T07:36:44.070 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.070 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.070 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.070 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402694+0000", 2024-09-18T07:36:44.070 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9984e-05 2024-09-18T07:36:44.070 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.070 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.070 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.070 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402706+0000", 2024-09-18T07:36:44.070 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2499e-05 2024-09-18T07:36:44.070 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.070 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.071 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.071 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402713+0000", 2024-09-18T07:36:44.071 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6429999999999998e-06 2024-09-18T07:36:44.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.071 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.071 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.071 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402959+0000", 2024-09-18T07:36:44.071 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024601700000000001 2024-09-18T07:36:44.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.071 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.071 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.071 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.403094+0000", 2024-09-18T07:36:44.071 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013474299999999999 2024-09-18T07:36:44.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.072 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.072 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:36:44.072 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.403289+0000", 2024-09-18T07:36:44.072 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019528500000000001 2024-09-18T07:36:44.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.072 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.072 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:44.072 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.404306+0000", 2024-09-18T07:36:44.072 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010166179999999999 2024-09-18T07:36:44.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.072 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.072 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.072 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.405112+0000", 2024-09-18T07:36:44.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00080637099999999995 2024-09-18T07:36:44.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.073 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.073 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481779+0000", 2024-09-18T07:36:44.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.076667400999999996 2024-09-18T07:36:44.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.073 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:44.073 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481825+0000", 2024-09-18T07:36:44.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5652000000000003e-05 2024-09-18T07:36:44.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.073 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.074 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481856+0000", 2024-09-18T07:36:44.074 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1315000000000001e-05 2024-09-18T07:36:44.074 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.074 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.074 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.074 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.074 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.34466.0:233 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2024-09-18T07:36:44.074 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:01.270598+0000", 2024-09-18T07:36:44.074 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 102.71649461600001, 2024-09-18T07:36:44.074 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14644679199999999, 2024-09-18T07:36:44.074 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.074 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.074 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:36:44.075 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:44.075 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.34466", 2024-09-18T07:36:44.075 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.203:0/952514737", 2024-09-18T07:36:44.075 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 233 2024-09-18T07:36:44.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.075 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.075 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270598+0000", 2024-09-18T07:36:44.075 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.075 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270598+0000", 2024-09-18T07:36:44.076 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.076 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.076 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.076 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.076 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270602+0000", 2024-09-18T07:36:44.076 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4960000000000001e-06 2024-09-18T07:36:44.076 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.076 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.076 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.076 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270613+0000", 2024-09-18T07:36:44.076 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0890000000000001e-05 2024-09-18T07:36:44.076 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.076 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.076 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.077 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270634+0000", 2024-09-18T07:36:44.077 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1484e-05 2024-09-18T07:36:44.077 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.077 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.077 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.077 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270643+0000", 2024-09-18T07:36:44.077 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1249999999999999e-06 2024-09-18T07:36:44.077 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.077 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.077 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.077 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270712+0000", 2024-09-18T07:36:44.077 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8733999999999996e-05 2024-09-18T07:36:44.077 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.078 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.078 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:36:44.078 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270720+0000", 2024-09-18T07:36:44.078 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0209999999999999e-06 2024-09-18T07:36:44.078 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.078 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.078 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.078 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.402460+0000", 2024-09-18T07:36:44.078 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13173957 2024-09-18T07:36:44.078 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.078 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.078 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for missing object", 2024-09-18T07:36:44.078 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.402586+0000", 2024-09-18T07:36:44.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012606100000000001 2024-09-18T07:36:44.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.079 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.416785+0000", 2024-09-18T07:36:44.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.014199145 2024-09-18T07:36:44.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.079 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.416882+0000", 2024-09-18T07:36:44.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6990999999999993e-05 2024-09-18T07:36:44.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.417045+0000", 2024-09-18T07:36:44.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000163275 2024-09-18T07:36:44.080 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.080 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.080 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.080 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.080 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.080 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:176547 7.5 7:a7a591d7:::benchmark_data_smithi137_74180_object72131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-18T07:36:44.080 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:20.411674+0000", 2024-09-18T07:36:44.080 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 83.575419138000001, 2024-09-18T07:36:44.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.052701980000000002, 2024-09-18T07:36:44.080 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.080 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.081 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:44.081 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:44.081 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:44.081 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:44.081 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 176547 2024-09-18T07:36:44.081 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.081 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.081 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.081 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.081 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:20.411674+0000", 2024-09-18T07:36:44.081 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.081 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.081 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.082 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:20.411674+0000", 2024-09-18T07:36:44.082 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.082 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.082 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.082 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:20.411675+0000", 2024-09-18T07:36:44.082 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2669999999999999e-06 2024-09-18T07:36:44.082 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.082 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.082 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:20.411679+0000", 2024-09-18T07:36:44.082 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1760000000000003e-06 2024-09-18T07:36:44.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.083 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.083 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.083 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:20.411690+0000", 2024-09-18T07:36:44.083 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1044e-05 2024-09-18T07:36:44.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.083 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.083 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.083 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:20.411697+0000", 2024-09-18T07:36:44.083 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4779999999999996e-06 2024-09-18T07:36:44.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.083 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.083 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.083 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:20.411734+0000", 2024-09-18T07:36:44.084 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7382000000000002e-05 2024-09-18T07:36:44.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.084 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.084 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.084 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:20.411808+0000", 2024-09-18T07:36:44.084 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4098999999999996e-05 2024-09-18T07:36:44.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.084 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.084 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:36:44.084 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:20.411869+0000", 2024-09-18T07:36:44.084 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0594000000000003e-05 2024-09-18T07:36:44.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.084 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.084 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:44.085 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:20.412423+0000", 2024-09-18T07:36:44.085 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000553751 2024-09-18T07:36:44.085 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.085 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.085 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.085 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:20.413054+0000", 2024-09-18T07:36:44.085 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00063143400000000005 2024-09-18T07:36:44.085 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.085 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.085 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.085 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:20.464273+0000", 2024-09-18T07:36:44.085 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.051219059999999997 2024-09-18T07:36:44.085 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.086 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.086 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:44.086 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:20.464328+0000", 2024-09-18T07:36:44.086 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4936999999999999e-05 2024-09-18T07:36:44.086 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.086 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.086 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.086 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:20.464376+0000", 2024-09-18T07:36:44.086 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7757999999999999e-05 2024-09-18T07:36:44.086 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.086 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.086 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.086 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.087 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.087 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:198402 7.5 7:a05d75bf:::benchmark_data_smithi137_74180_object93986:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-18T07:36:44.087 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:34.766688+0000", 2024-09-18T07:36:44.087 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 69.220404819999999, 2024-09-18T07:36:44.087 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.051334891000000001, 2024-09-18T07:36:44.087 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.087 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.087 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:44.087 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:44.087 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:44.087 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:44.087 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 198402 2024-09-18T07:36:44.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.087 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:34.766688+0000", 2024-09-18T07:36:44.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:34.766688+0000", 2024-09-18T07:36:44.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:34.766690+0000", 2024-09-18T07:36:44.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5030000000000001e-06 2024-09-18T07:36:44.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:34.766694+0000", 2024-09-18T07:36:44.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.481e-06 2024-09-18T07:36:44.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:34.766705+0000", 2024-09-18T07:36:44.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0767e-05 2024-09-18T07:36:44.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:34.766712+0000", 2024-09-18T07:36:44.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2869999999999999e-06 2024-09-18T07:36:44.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.090 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.090 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.090 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:34.766773+0000", 2024-09-18T07:36:44.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0812e-05 2024-09-18T07:36:44.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.090 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.090 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.090 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:34.766897+0000", 2024-09-18T07:36:44.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012347899999999999 2024-09-18T07:36:44.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.090 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:36:44.091 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:34.767015+0000", 2024-09-18T07:36:44.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000117998 2024-09-18T07:36:44.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:44.091 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:34.768536+0000", 2024-09-18T07:36:44.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0015217290000000001 2024-09-18T07:36:44.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.091 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:34.770753+0000", 2024-09-18T07:36:44.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022163 2024-09-18T07:36:44.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:34.817927+0000", 2024-09-18T07:36:44.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.047174835999999998 2024-09-18T07:36:44.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:44.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:34.817984+0000", 2024-09-18T07:36:44.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6323000000000003e-05 2024-09-18T07:36:44.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.093 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:34.818023+0000", 2024-09-18T07:36:44.093 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9375999999999997e-05 2024-09-18T07:36:44.093 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.093 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.093 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.093 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.093 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.093 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:200515 7.5 7:a1f023bc:::benchmark_data_smithi137_74180_object96099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-18T07:36:44.093 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:36.399284+0000", 2024-09-18T07:36:44.093 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 67.587809527999994, 2024-09-18T07:36:44.093 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.065364955000000002, 2024-09-18T07:36:44.093 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.093 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.093 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:44.094 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:44.094 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:44.094 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:44.094 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 200515 2024-09-18T07:36:44.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.094 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.094 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.094 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.094 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399284+0000", 2024-09-18T07:36:44.094 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.094 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.094 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399284+0000", 2024-09-18T07:36:44.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.095 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399285+0000", 2024-09-18T07:36:44.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.347e-06 2024-09-18T07:36:44.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.095 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399289+0000", 2024-09-18T07:36:44.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0350000000000003e-06 2024-09-18T07:36:44.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.096 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.096 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399296+0000", 2024-09-18T07:36:44.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0639999999999996e-06 2024-09-18T07:36:44.096 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.096 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.096 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.096 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399302+0000", 2024-09-18T07:36:44.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3949999999999998e-06 2024-09-18T07:36:44.096 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.096 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.096 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.096 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399381+0000", 2024-09-18T07:36:44.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8978000000000002e-05 2024-09-18T07:36:44.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.097 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.097 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.097 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399493+0000", 2024-09-18T07:36:44.097 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000111155 2024-09-18T07:36:44.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.100 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.100 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:36:44.100 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399605+0000", 2024-09-18T07:36:44.100 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000112465 2024-09-18T07:36:44.100 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.100 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.100 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:44.100 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.400747+0000", 2024-09-18T07:36:44.100 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001141677 2024-09-18T07:36:44.100 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.100 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.100 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.401647+0000", 2024-09-18T07:36:44.101 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00090012499999999999 2024-09-18T07:36:44.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.101 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.101 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.464535+0000", 2024-09-18T07:36:44.101 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.062888682000000001 2024-09-18T07:36:44.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.101 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.101 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:44.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.464585+0000", 2024-09-18T07:36:44.101 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9109e-05 2024-09-18T07:36:44.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.102 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.464648+0000", 2024-09-18T07:36:44.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3923000000000004e-05 2024-09-18T07:36:44.102 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.102 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.102 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.102 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:201219 7.5 7:a97592c9:::benchmark_data_smithi137_74180_object96803:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-18T07:36:44.102 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:36.904266+0000", 2024-09-18T07:36:44.102 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 67.082827042999995, 2024-09-18T07:36:44.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.057898721, 2024-09-18T07:36:44.102 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.103 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.103 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:44.103 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:44.103 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:44.103 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:44.103 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 201219 2024-09-18T07:36:44.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.103 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.103 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.103 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.103 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.904266+0000", 2024-09-18T07:36:44.103 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.103 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.904266+0000", 2024-09-18T07:36:44.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.104 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.904267+0000", 2024-09-18T07:36:44.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.375e-06 2024-09-18T07:36:44.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.104 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.904271+0000", 2024-09-18T07:36:44.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9489999999999998e-06 2024-09-18T07:36:44.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.105 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.105 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.105 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.904285+0000", 2024-09-18T07:36:44.105 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3298e-05 2024-09-18T07:36:44.105 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.105 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.105 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.105 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.904291+0000", 2024-09-18T07:36:44.105 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6889999999999998e-06 2024-09-18T07:36:44.105 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.105 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.105 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.105 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.904343+0000", 2024-09-18T07:36:44.106 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1759999999999999e-05 2024-09-18T07:36:44.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.106 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.106 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.106 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.904433+0000", 2024-09-18T07:36:44.106 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9563999999999998e-05 2024-09-18T07:36:44.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.106 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.106 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:36:44.106 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.904502+0000", 2024-09-18T07:36:44.106 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9618999999999999e-05 2024-09-18T07:36:44.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.106 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.106 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:44.107 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.905830+0000", 2024-09-18T07:36:44.107 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0013282039999999999 2024-09-18T07:36:44.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.107 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.107 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.107 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.907970+0000", 2024-09-18T07:36:44.107 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021395619999999998 2024-09-18T07:36:44.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.107 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.107 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.107 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.962033+0000", 2024-09-18T07:36:44.107 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.054062823000000003 2024-09-18T07:36:44.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.107 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.108 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:44.108 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.962102+0000", 2024-09-18T07:36:44.108 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8961000000000006e-05 2024-09-18T07:36:44.108 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.108 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.108 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.108 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.962165+0000", 2024-09-18T07:36:44.108 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2916999999999994e-05 2024-09-18T07:36:44.108 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.108 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.108 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.108 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.108 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.108 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:203666 7.5 7:a8834af4:::benchmark_data_smithi137_74180_object99250:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-09-18T07:36:44.109 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:39.623328+0000", 2024-09-18T07:36:44.109 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 64.363765138999995, 2024-09-18T07:36:44.109 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.051224387000000003, 2024-09-18T07:36:44.109 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:36:44.109 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:36:44.109 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:44.109 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:36:44.109 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:36:44.109 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:36:44.109 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 203666 2024-09-18T07:36:44.109 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.109 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:36:44.109 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.110 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:36:44.110 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:39.623328+0000", 2024-09-18T07:36:44.110 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.110 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.110 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.110 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:36:44.110 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:39.623328+0000", 2024-09-18T07:36:44.110 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:36:44.110 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.110 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.110 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:36:44.110 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:39.623330+0000", 2024-09-18T07:36:44.110 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1440000000000001e-06 2024-09-18T07:36:44.110 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.111 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.111 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:36:44.111 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:39.623336+0000", 2024-09-18T07:36:44.111 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0050000000000001e-06 2024-09-18T07:36:44.111 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.111 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.111 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:36:44.111 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:39.623352+0000", 2024-09-18T07:36:44.111 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5726000000000002e-05 2024-09-18T07:36:44.111 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.111 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.111 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:36:44.111 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:39.623361+0000", 2024-09-18T07:36:44.111 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8470000000000002e-06 2024-09-18T07:36:44.112 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.112 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.112 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:36:44.112 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:39.623421+0000", 2024-09-18T07:36:44.112 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0677999999999997e-05 2024-09-18T07:36:44.112 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.112 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.112 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:36:44.112 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:39.623539+0000", 2024-09-18T07:36:44.112 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000118008 2024-09-18T07:36:44.112 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.112 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.112 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:36:44.113 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:39.623659+0000", 2024-09-18T07:36:44.113 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000119608 2024-09-18T07:36:44.113 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.113 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.113 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:36:44.113 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:39.625182+0000", 2024-09-18T07:36:44.113 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001523045 2024-09-18T07:36:44.113 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.113 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.113 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.113 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:39.633563+0000", 2024-09-18T07:36:44.113 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0083813359999999996 2024-09-18T07:36:44.113 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.113 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.114 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:44.114 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:39.674455+0000", 2024-09-18T07:36:44.114 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.040891851999999999 2024-09-18T07:36:44.114 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.114 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.114 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:36:44.114 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:39.674508+0000", 2024-09-18T07:36:44.114 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2651999999999997e-05 2024-09-18T07:36:44.114 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:36:44.114 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:36:44.114 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:36:44.114 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:39.674552+0000", 2024-09-18T07:36:44.114 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4486000000000001e-05 2024-09-18T07:36:44.114 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.115 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.115 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.115 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:36:44.115 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:36:44.115 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:36:44.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:44.116 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:44.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:44.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:44 smithi203 ceph-mon[32237]: osdmap e220: 8 total, 8 up, 7 in 2024-09-18T07:36:44.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:44 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:36:44.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:44 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:36:44.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:44 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1455577137' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:44.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:44.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:44.265 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:44.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:44.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:44.412 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:44.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:44.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:44.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:44 smithi137 ceph-mon[29408]: osdmap e220: 8 total, 8 up, 7 in 2024-09-18T07:36:44.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:44 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:36:44.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:44 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:36:44.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:44 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1455577137' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:44.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:44 smithi137 ceph-mon[39175]: osdmap e220: 8 total, 8 up, 7 in 2024-09-18T07:36:44.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:44 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:36:44.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:44 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:36:44.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1455577137' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:44.514 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:44.541 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:44.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:44.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:44.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:44.644 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:36:44.683 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:44.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:44.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:44.785 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:44.812 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:44.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:44.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:44.913 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:44.923 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 39 16 67264 67248 107.75 47.75 0.0169178 0.00926922 2024-09-18T07:36:44.938 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:44.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:44.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:45.040 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:45.093 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:45.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:45.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:45.106 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:45.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:45 smithi137 ceph-mon[29408]: pgmap v309: 148 pgs: 2 unknown, 46 peering, 100 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 341 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-18T07:36:45.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:45 smithi137 ceph-mon[29408]: osdmap e221: 8 total, 8 up, 7 in 2024-09-18T07:36:45.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:45 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:36:45.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:45 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:36:45.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:45 smithi137 ceph-mon[39175]: pgmap v309: 148 pgs: 2 unknown, 46 peering, 100 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 341 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-18T07:36:45.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:45 smithi137 ceph-mon[39175]: osdmap e221: 8 total, 8 up, 7 in 2024-09-18T07:36:45.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:45 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:36:45.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:45 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:36:45.195 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:45.236 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:45.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:45.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:45.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T07:36:45.338 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:45.363 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:45.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:45.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:45.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:45 smithi203 ceph-mon[32237]: pgmap v309: 148 pgs: 2 unknown, 46 peering, 100 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 341 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-18T07:36:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:45 smithi203 ceph-mon[32237]: osdmap e221: 8 total, 8 up, 7 in 2024-09-18T07:36:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:45 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:36:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:45 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:36:45.465 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:45.492 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:45.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:45.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:45.594 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:45.642 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:45.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:45.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:45.745 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:45.797 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:45.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:45.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:45.817 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:45.899 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:45.923 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:36:45.921990+0000 min lat: 0.00114205 max lat: 2.28882 avg lat: 0.00941503 2024-09-18T07:36:45.924 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:36:45.924 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 40 16 67952 67936 106.131 43 0.0384858 0.00941503 2024-09-18T07:36:45.924 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:45.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:45.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:46.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:46.066 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:46.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:46.168 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:46.193 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:46.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:46.278 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:46 smithi137 ceph-mon[29408]: osdmap e222: 8 total, 8 up, 7 in 2024-09-18T07:36:46.278 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:46 smithi137 ceph-mon[39175]: osdmap e222: 8 total, 8 up, 7 in 2024-09-18T07:36:46.295 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:46.324 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:46.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:46.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:46.426 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:46.443 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:36:46.443 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:36:46.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.443 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.35171.0:68332 8.e e222/201)", 2024-09-18T07:36:46.443 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:46.436286+0000", 2024-09-18T07:36:46.443 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00245547, 2024-09-18T07:36:46.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024802510000000002, 2024-09-18T07:36:46.444 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:46.444 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:46.444 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:36:46.444 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:46.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:46.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436286+0000", 2024-09-18T07:36:46.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:46.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436286+0000", 2024-09-18T07:36:46.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:46.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436289+0000", 2024-09-18T07:36:46.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0609999999999998e-06 2024-09-18T07:36:46.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:46.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436299+0000", 2024-09-18T07:36:46.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0305999999999999e-05 2024-09-18T07:36:46.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:46.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436310+0000", 2024-09-18T07:36:46.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0705e-05 2024-09-18T07:36:46.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:46.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436317+0000", 2024-09-18T07:36:46.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3869999999999997e-06 2024-09-18T07:36:46.446 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.446 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:46.446 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.447 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35171.0:68350 8.9 e222/219 8:9a638b96:::benchmark_data_smithi137_101942_object68349:head v 222'4308, mlcod=208'135)", 2024-09-18T07:36:46.447 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:46.436352+0000", 2024-09-18T07:36:46.447 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0023899669999999998, 2024-09-18T07:36:46.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024588560000000001, 2024-09-18T07:36:46.447 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:46.447 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:46.447 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:36:46.447 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:46.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:46.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436352+0000", 2024-09-18T07:36:46.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:46.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436352+0000", 2024-09-18T07:36:46.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:46.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436353+0000", 2024-09-18T07:36:46.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1799999999999999e-06 2024-09-18T07:36:46.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:46.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436446+0000", 2024-09-18T07:36:46.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2344999999999998e-05 2024-09-18T07:36:46.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:46.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436454+0000", 2024-09-18T07:36:46.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3119999999999994e-06 2024-09-18T07:36:46.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:46.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436459+0000", 2024-09-18T07:36:46.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4960000000000004e-06 2024-09-18T07:36:46.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:46.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436524+0000", 2024-09-18T07:36:46.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4908999999999998e-05 2024-09-18T07:36:46.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:46.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436548+0000", 2024-09-18T07:36:46.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4175000000000001e-05 2024-09-18T07:36:46.450 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.450 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:46.450 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.451 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.35171.0:68331 8.6 e222/219)", 2024-09-18T07:36:46.451 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:46.436647+0000", 2024-09-18T07:36:46.451 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0020953550000000001, 2024-09-18T07:36:46.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002205107, 2024-09-18T07:36:46.451 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:46.451 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:46.451 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:36:46.451 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:46.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:46.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436647+0000", 2024-09-18T07:36:46.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:46.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436647+0000", 2024-09-18T07:36:46.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:46.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436648+0000", 2024-09-18T07:36:46.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1230000000000001e-06 2024-09-18T07:36:46.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:46.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436654+0000", 2024-09-18T07:36:46.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.268e-06 2024-09-18T07:36:46.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:46.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436663+0000", 2024-09-18T07:36:46.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2259999999999994e-06 2024-09-18T07:36:46.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:46.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436669+0000", 2024-09-18T07:36:46.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9359999999999996e-06 2024-09-18T07:36:46.454 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.454 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:46.454 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.454 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35171.0:68351 8.9 e222/219)", 2024-09-18T07:36:46.455 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:46.438699+0000", 2024-09-18T07:36:46.455 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 4.2565999999999999e-05, 2024-09-18T07:36:46.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018496699999999999, 2024-09-18T07:36:46.455 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:46.455 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:46.455 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:36:46.455 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:46.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:46.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.438699+0000", 2024-09-18T07:36:46.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:46.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.438699+0000", 2024-09-18T07:36:46.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:46.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.438701+0000", 2024-09-18T07:36:46.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4869999999999999e-06 2024-09-18T07:36:46.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:46.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.438745+0000", 2024-09-18T07:36:46.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4169999999999999e-05 2024-09-18T07:36:46.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:46.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.438756+0000", 2024-09-18T07:36:46.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1296e-05 2024-09-18T07:36:46.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:46.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.438764+0000", 2024-09-18T07:36:46.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6040000000000004e-06 2024-09-18T07:36:46.458 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.458 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:46.458 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.458 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:68331 8.6 8:6b31050f:::benchmark_data_smithi137_101942_object68330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-18T07:36:46.459 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:46.411522+0000", 2024-09-18T07:36:46.459 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.027220323000000001, 2024-09-18T07:36:46.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027405288999999999, 2024-09-18T07:36:46.459 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:46.459 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:46.459 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:36:46.459 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:46.459 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:36:46.460 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:36:46.460 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 68331 2024-09-18T07:36:46.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.460 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:46.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:46.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.411522+0000", 2024-09-18T07:36:46.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:46.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:46.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.411522+0000", 2024-09-18T07:36:46.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:46.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.411523+0000", 2024-09-18T07:36:46.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6959999999999999e-06 2024-09-18T07:36:46.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:46.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.411547+0000", 2024-09-18T07:36:46.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3261e-05 2024-09-18T07:36:46.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:46.465 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.411560+0000", 2024-09-18T07:36:46.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3072e-05 2024-09-18T07:36:46.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:46.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.411567+0000", 2024-09-18T07:36:46.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4189999999999996e-06 2024-09-18T07:36:46.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:46.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.433496+0000", 2024-09-18T07:36:46.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021928670000000001 2024-09-18T07:36:46.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:46.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.433719+0000", 2024-09-18T07:36:46.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022374999999999999 2024-09-18T07:36:46.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:36:46.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.433942+0000", 2024-09-18T07:36:46.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022240500000000001 2024-09-18T07:36:46.468 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.468 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:46.468 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.469 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:68332 8.e 8:71723f4a:::benchmark_data_smithi137_101942_object68331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-18T07:36:46.469 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:46.411650+0000", 2024-09-18T07:36:46.469 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.027092215999999999, 2024-09-18T07:36:46.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027345452999999999, 2024-09-18T07:36:46.469 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:46.469 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:46.470 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:36:46.470 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:46.470 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:36:46.470 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:36:46.470 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 68332 2024-09-18T07:36:46.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.471 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:46.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:46.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.411650+0000", 2024-09-18T07:36:46.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:46.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.411650+0000", 2024-09-18T07:36:46.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:46.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.411651+0000", 2024-09-18T07:36:46.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1969999999999999e-06 2024-09-18T07:36:46.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:46.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.411673+0000", 2024-09-18T07:36:46.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2124e-05 2024-09-18T07:36:46.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:46.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.411689+0000", 2024-09-18T07:36:46.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5455999999999999e-05 2024-09-18T07:36:46.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:46.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.411696+0000", 2024-09-18T07:36:46.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1849999999999998e-06 2024-09-18T07:36:46.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:46.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.433098+0000", 2024-09-18T07:36:46.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021402053000000001 2024-09-18T07:36:46.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:46.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.433242+0000", 2024-09-18T07:36:46.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014396 2024-09-18T07:36:46.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,4", 2024-09-18T07:36:46.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.433397+0000", 2024-09-18T07:36:46.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015569999999999999 2024-09-18T07:36:46.475 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.475 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:46.476 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.476 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35171.0:68342 8.5 e222/219 8:ad99dae7:::benchmark_data_smithi137_101942_object68341:head v 222'4231, mlcod=208'139)", 2024-09-18T07:36:46.476 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:46.425407+0000", 2024-09-18T07:36:46.476 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.013334723999999999, 2024-09-18T07:36:46.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013642076, 2024-09-18T07:36:46.476 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:46.476 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:46.476 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:36:46.477 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:46.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:46.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.425407+0000", 2024-09-18T07:36:46.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:46.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.425407+0000", 2024-09-18T07:36:46.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:46.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.425408+0000", 2024-09-18T07:36:46.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0020000000000001e-06 2024-09-18T07:36:46.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:46.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.425436+0000", 2024-09-18T07:36:46.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8183000000000001e-05 2024-09-18T07:36:46.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:46.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.425481+0000", 2024-09-18T07:36:46.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4239000000000001e-05 2024-09-18T07:36:46.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:46.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.425488+0000", 2024-09-18T07:36:46.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8909999999999998e-06 2024-09-18T07:36:46.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:46.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.425687+0000", 2024-09-18T07:36:46.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019982999999999999 2024-09-18T07:36:46.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:46.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.426319+0000", 2024-09-18T07:36:46.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000631582 2024-09-18T07:36:46.480 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.480 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:46.480 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.480 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:68344 8.1 8:800c8f51:::benchmark_data_smithi137_101942_object68343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-18T07:36:46.480 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:46.426550+0000", 2024-09-18T07:36:46.480 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.012192128, 2024-09-18T07:36:46.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012546432999999999, 2024-09-18T07:36:46.480 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:46.480 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:46.480 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:36:46.480 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:36:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:36:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 68344 2024-09-18T07:36:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.426550+0000", 2024-09-18T07:36:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.426550+0000", 2024-09-18T07:36:46.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:46.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.426551+0000", 2024-09-18T07:36:46.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.268e-06 2024-09-18T07:36:46.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:46.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.426600+0000", 2024-09-18T07:36:46.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9277000000000003e-05 2024-09-18T07:36:46.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:46.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.426609+0000", 2024-09-18T07:36:46.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.49e-06 2024-09-18T07:36:46.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:46.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.426615+0000", 2024-09-18T07:36:46.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8019999999999997e-06 2024-09-18T07:36:46.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:46.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.426660+0000", 2024-09-18T07:36:46.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5062999999999999e-05 2024-09-18T07:36:46.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:46.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.426908+0000", 2024-09-18T07:36:46.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024812499999999997 2024-09-18T07:36:46.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-18T07:36:46.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.427177+0000", 2024-09-18T07:36:46.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000269645 2024-09-18T07:36:46.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:46.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.433381+0000", 2024-09-18T07:36:46.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0062038780000000003 2024-09-18T07:36:46.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:46.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.434184+0000", 2024-09-18T07:36:46.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00080218100000000001 2024-09-18T07:36:46.485 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.485 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:46.485 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.486 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35171.0:68343 8.5 e222/219)", 2024-09-18T07:36:46.486 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:46.429520+0000", 2024-09-18T07:36:46.486 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0092215920000000007, 2024-09-18T07:36:46.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0096360630000000003, 2024-09-18T07:36:46.486 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:46.486 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:46.486 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:36:46.486 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:46.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:46.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.429520+0000", 2024-09-18T07:36:46.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:46.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.429520+0000", 2024-09-18T07:36:46.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:46.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.429522+0000", 2024-09-18T07:36:46.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2860000000000001e-06 2024-09-18T07:36:46.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:46.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.429559+0000", 2024-09-18T07:36:46.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6980000000000002e-05 2024-09-18T07:36:46.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:46.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.429569+0000", 2024-09-18T07:36:46.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0507e-05 2024-09-18T07:36:46.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:46.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.429576+0000", 2024-09-18T07:36:46.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8859999999999998e-06 2024-09-18T07:36:46.489 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.489 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:46.490 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.490 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35171.0:68348 8.5 e222/219)", 2024-09-18T07:36:46.490 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:46.430073+0000", 2024-09-18T07:36:46.490 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0086688670000000002, 2024-09-18T07:36:46.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0091175389999999992, 2024-09-18T07:36:46.491 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:46.491 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:46.491 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:36:46.491 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:46.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:46.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.430073+0000", 2024-09-18T07:36:46.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:46.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.430073+0000", 2024-09-18T07:36:46.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:46.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.430074+0000", 2024-09-18T07:36:46.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6700000000000002e-07 2024-09-18T07:36:46.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:46.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.430129+0000", 2024-09-18T07:36:46.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4534999999999999e-05 2024-09-18T07:36:46.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:46.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.430138+0000", 2024-09-18T07:36:46.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3389999999999999e-06 2024-09-18T07:36:46.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.495 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.495 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:46.495 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.430144+0000", 2024-09-18T07:36:46.495 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5799999999999997e-06 2024-09-18T07:36:46.495 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.496 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:46.496 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.496 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:68321 8.6 8:673ae78d:::benchmark_data_smithi137_101942_object68320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-18T07:36:46.496 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:46.400742+0000", 2024-09-18T07:36:46.497 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.037999766999999997, 2024-09-18T07:36:46.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.038488755999999999, 2024-09-18T07:36:46.497 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:46.497 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:46.497 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:36:46.497 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:46.497 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:36:46.497 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:36:46.497 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 68321 2024-09-18T07:36:46.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.497 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:46.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:46.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.400742+0000", 2024-09-18T07:36:46.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:46.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.400742+0000", 2024-09-18T07:36:46.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:46.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.400745+0000", 2024-09-18T07:36:46.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8890000000000002e-06 2024-09-18T07:36:46.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:46.499 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.400778+0000", 2024-09-18T07:36:46.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2994000000000003e-05 2024-09-18T07:36:46.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:46.499 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.400794+0000", 2024-09-18T07:36:46.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5696e-05 2024-09-18T07:36:46.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:46.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.400803+0000", 2024-09-18T07:36:46.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6800000000000005e-06 2024-09-18T07:36:46.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:46.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.405918+0000", 2024-09-18T07:36:46.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051141190000000003 2024-09-18T07:36:46.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:46.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.406366+0000", 2024-09-18T07:36:46.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044812900000000002 2024-09-18T07:36:46.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:36:46.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.406520+0000", 2024-09-18T07:36:46.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000154365 2024-09-18T07:36:46.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:46.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.426869+0000", 2024-09-18T07:36:46.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020349253000000001 2024-09-18T07:36:46.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:46.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.433063+0000", 2024-09-18T07:36:46.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0061932840000000003 2024-09-18T07:36:46.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:36:46.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436605+0000", 2024-09-18T07:36:46.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035420170000000002 2024-09-18T07:36:46.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:36:46.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436659+0000", 2024-09-18T07:36:46.503 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3925000000000003e-05 2024-09-18T07:36:46.503 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.503 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:46.503 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.503 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.503 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:68346 8.e 8:7c03cbc4:::benchmark_data_smithi137_101942_object68345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-18T07:36:46.503 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:46.432864+0000", 2024-09-18T07:36:46.503 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0058777660000000004, 2024-09-18T07:36:46.503 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0064386900000000004, 2024-09-18T07:36:46.503 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:46.503 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:46.504 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:36:46.504 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:46.504 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:36:46.504 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:36:46.504 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 68346 2024-09-18T07:36:46.504 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.504 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:46.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.504 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:46.504 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.432864+0000", 2024-09-18T07:36:46.504 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.504 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.505 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:46.505 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.432864+0000", 2024-09-18T07:36:46.505 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.505 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.505 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:46.505 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.432867+0000", 2024-09-18T07:36:46.505 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5009999999999999e-06 2024-09-18T07:36:46.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:46.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.432928+0000", 2024-09-18T07:36:46.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1169999999999999e-05 2024-09-18T07:36:46.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:46.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.432937+0000", 2024-09-18T07:36:46.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4390000000000006e-06 2024-09-18T07:36:46.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:46.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.432944+0000", 2024-09-18T07:36:46.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4949999999999996e-06 2024-09-18T07:36:46.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.507 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:46.507 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436210+0000", 2024-09-18T07:36:46.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032665849999999998 2024-09-18T07:36:46.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.507 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:46.507 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436393+0000", 2024-09-18T07:36:46.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018296600000000001 2024-09-18T07:36:46.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.508 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,4", 2024-09-18T07:36:46.508 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.436569+0000", 2024-09-18T07:36:46.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000176002 2024-09-18T07:36:46.508 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.508 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:46.508 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.508 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:68324 8.e 8:73817f66:::benchmark_data_smithi137_101942_object68323:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-18T07:36:46.508 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:36:46.402143+0000", 2024-09-18T07:36:46.508 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.036598517999999997, 2024-09-18T07:36:46.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.037224328000000001, 2024-09-18T07:36:46.508 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:36:46.509 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:36:46.509 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:36:46.509 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:36:46.509 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:36:46.509 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:36:46.509 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 68324 2024-09-18T07:36:46.509 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.509 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:36:46.509 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.509 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:36:46.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.402143+0000", 2024-09-18T07:36:46.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.510 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:36:46.510 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.402143+0000", 2024-09-18T07:36:46.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:36:46.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.510 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:36:46.510 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.402145+0000", 2024-09-18T07:36:46.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.81e-06 2024-09-18T07:36:46.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:36:46.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.402169+0000", 2024-09-18T07:36:46.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4162e-05 2024-09-18T07:36:46.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:36:46.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.402178+0000", 2024-09-18T07:36:46.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9060000000000008e-06 2024-09-18T07:36:46.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:36:46.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.402186+0000", 2024-09-18T07:36:46.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3930000000000002e-06 2024-09-18T07:36:46.512 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.512 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.512 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:36:46.512 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.405573+0000", 2024-09-18T07:36:46.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033870749999999998 2024-09-18T07:36:46.512 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.512 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.512 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:36:46.512 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.405701+0000", 2024-09-18T07:36:46.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012817300000000001 2024-09-18T07:36:46.512 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.512 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.513 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,4", 2024-09-18T07:36:46.513 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.407053+0000", 2024-09-18T07:36:46.513 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013517080000000001 2024-09-18T07:36:46.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:36:46.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:36:46.513 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:36:46.513 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:36:46.411168+0000", 2024-09-18T07:36:46.513 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041156049999999996 2024-09-18T07:36:46.513 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.513 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:36:46.513 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.513 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:36:46.514 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:36:46.514 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 13 2024-09-18T07:36:46.514 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:36:46.514 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:46.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:46.515 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:46 smithi203 ceph-mon[32237]: osdmap e222: 8 total, 8 up, 7 in 2024-09-18T07:36:46.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:46.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:46.665 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:46.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:46.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:46.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:46.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:46.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:46.908 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:46.923 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 41 16 68748 68732 104.755 49.75 0.0106069 0.00953904 2024-09-18T07:36:46.936 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:46.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:46.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:46.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:36:47.038 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:47.038 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:36:47.038 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:36:47.038 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-18T07:36:47.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-18T07:36:47.058 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:36:47.098 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:47.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:47.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:47.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:47 smithi137 ceph-mon[29408]: pgmap v312: 148 pgs: 2 unknown, 46 peering, 100 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 51 MiB/s wr, 820 op/s 2024-09-18T07:36:47.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:47 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3469791465' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:47.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:47 smithi137 ceph-mon[29408]: from='client.25291 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:47.170 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:36:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:36:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:36:47.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:47 smithi137 ceph-mon[39175]: pgmap v312: 148 pgs: 2 unknown, 46 peering, 100 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 51 MiB/s wr, 820 op/s 2024-09-18T07:36:47.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:47 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3469791465' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:47.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:47 smithi137 ceph-mon[39175]: from='client.25291 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:47.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:47.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:47.274 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:47.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:47.375 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:47.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:47.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:47.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T07:36:47.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:47 smithi203 ceph-mon[32237]: pgmap v312: 148 pgs: 2 unknown, 46 peering, 100 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 51 MiB/s wr, 820 op/s 2024-09-18T07:36:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:47 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3469791465' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:47 smithi203 ceph-mon[32237]: from='client.25291 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:36:47.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:36:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:36:47.523 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:47.553 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:47.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:47.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:47.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:47.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:47.701 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:47.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:47.802 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:47.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:47.831 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:47.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:47.924 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 42 16 69443 69427 103.296 43.4375 0.0245808 0.00967285 2024-09-18T07:36:47.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:47.976 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:47.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:47.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:48.078 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:48.107 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:48.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:48.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:48.209 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:48.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:48.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:48.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:48.351 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:48.388 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:48.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:48.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:48.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:48 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:48.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:48 smithi137 ceph-mon[29408]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:36:48.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:48 smithi137 ceph-mon[29408]: osdmap e223: 8 total, 8 up, 7 in 2024-09-18T07:36:48.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:48 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:48.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:48 smithi137 ceph-mon[29408]: pgmap v314: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 36 MiB/s wr, 584 op/s; 20149/206148 objects misplaced (9.774%); 8.0 MiB/s, 142 objects/s recovering 2024-09-18T07:36:48.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:48 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:48.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:48 smithi137 ceph-mon[39175]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:36:48.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:48 smithi137 ceph-mon[39175]: osdmap e223: 8 total, 8 up, 7 in 2024-09-18T07:36:48.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:48 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:48.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:48 smithi137 ceph-mon[39175]: pgmap v314: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 36 MiB/s wr, 584 op/s; 20149/206148 objects misplaced (9.774%); 8.0 MiB/s, 142 objects/s recovering 2024-09-18T07:36:48.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:48 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:48 smithi203 ceph-mon[32237]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:36:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:48 smithi203 ceph-mon[32237]: osdmap e223: 8 total, 8 up, 7 in 2024-09-18T07:36:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:48 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:48 smithi203 ceph-mon[32237]: pgmap v314: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 36 MiB/s wr, 584 op/s; 20149/206148 objects misplaced (9.774%); 8.0 MiB/s, 142 objects/s recovering 2024-09-18T07:36:48.491 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:48.518 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:48.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:48.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:48.614 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:48.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:48.709 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:48.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:48.714 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:48.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:48.739 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:48.816 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:48.842 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:48.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:48.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:48.924 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 43 16 70208 70192 102.005 47.8125 0.00426829 0.00979561 2024-09-18T07:36:48.944 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:48.985 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:48.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:48.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:49.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:49.133 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:49.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:49.234 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:49.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:49.260 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:49.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:49.339 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:49 smithi137 ceph-mon[29408]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 10 pgs peering) 2024-09-18T07:36:49.339 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:49 smithi137 ceph-mon[39175]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 10 pgs peering) 2024-09-18T07:36:49.362 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:49.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:49.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:49.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:49.424 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:36:49.424 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:36:49.459 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:36:49.459 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:36:49.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:49 smithi203 ceph-mon[32237]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 10 pgs peering) 2024-09-18T07:36:49.490 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:36:49.490 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:36:49.519 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:49.523 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:36:49.523 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:36:49.545 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:49.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:49.555 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:36:49.555 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:36:49.587 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:36:49.587 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:36:49.618 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:36:49.618 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:36:49.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:49.656 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:36:49.656 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:36:49.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:49.694 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:49.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:49.795 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:49.847 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:49.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:49.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:49.928 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 44 15 70911 70896 100.678 44 0.011279 0.00992506 2024-09-18T07:36:49.949 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:49.978 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:49.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:49.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:50.080 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:50.107 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:50.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:50.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:50.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:50.238 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:50.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:50.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:50.256 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:36:50.256 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:36:50.257 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:50 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1193094209' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:50.257 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:50 smithi137 ceph-mon[29408]: pgmap v315: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 27 MiB/s wr, 438 op/s; 20149/206148 objects misplaced (9.774%); 6.0 MiB/s, 106 objects/s recovering 2024-09-18T07:36:50.257 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:50 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1193094209' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:50.257 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:50 smithi137 ceph-mon[39175]: pgmap v315: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 27 MiB/s wr, 438 op/s; 20149/206148 objects misplaced (9.774%); 6.0 MiB/s, 106 objects/s recovering 2024-09-18T07:36:50.340 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:50.379 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:50.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:50.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:50.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:50 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1193094209' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:36:50.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:50 smithi203 ceph-mon[32237]: pgmap v315: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 27 MiB/s wr, 438 op/s; 20149/206148 objects misplaced (9.774%); 6.0 MiB/s, 106 objects/s recovering 2024-09-18T07:36:50.481 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:50.510 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:50.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:50.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:50.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:36:50.612 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:50.638 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:50.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:50.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:50.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:50.765 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:50.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:50.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:50.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:50.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:50.925 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:50.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:50.928 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 45 16 71692 71676 99.5225 48.75 0.0162039 0.0100421 2024-09-18T07:36:50.970 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T07:36:51.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:51.115 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:51.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:51.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:51.216 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:51.242 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:51.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:51.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:51.244 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-18T07:36:51.244 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-18T07:36:51.244 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-18T07:36:51.244 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-18T07:36:51.244 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-18T07:36:51.244 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-18T07:36:51.244 INFO:teuthology.orchestra.run.smithi137.stderr:bench [] [] [] [] 2024-09-18T07:36:51.244 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-18T07:36:51.244 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-18T07:36:51.245 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-18T07:36:51.245 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator dump block 2024-09-18T07:36:51.245 INFO:teuthology.orchestra.run.smithi137.stderr:Error EINVAL: invalid command 2024-09-18T07:36:51.344 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:51.370 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:51.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:51.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:51.472 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:51.500 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:51.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:51.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:51.658 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:51.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:51.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:51.760 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:51.814 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:51.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:51.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:51.916 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:51.929 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 46 16 72411 72395 98.3355 44.9375 0.00605985 0.0101597 2024-09-18T07:36:51.951 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:51.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:51.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:52.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:52.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:52.096 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:52.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:52.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:52.234 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:52.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:52.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:52.336 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:52.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:52.400 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:52.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:52.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:52.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:52.566 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:52.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:52.629 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:52.630 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T07:36:52.630 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T07:36:52.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-18T07:36:52.631 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:52.667 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:52.692 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:52.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:52.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:52.794 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:52.816 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:52 smithi137 ceph-mon[29408]: pgmap v316: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 35 MiB/s wr, 568 op/s; 19493/211026 objects misplaced (9.237%); 10 MiB/s, 173 objects/s recovering 2024-09-18T07:36:52.816 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:52 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:36:52.816 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:52 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:52.816 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:52 smithi137 ceph-mon[39175]: pgmap v316: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 35 MiB/s wr, 568 op/s; 19493/211026 objects misplaced (9.237%); 10 MiB/s, 173 objects/s recovering 2024-09-18T07:36:52.817 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:52 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:36:52.817 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:52 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:52.821 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:52.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:52.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:52.924 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:52.930 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 47 16 73105 73089 97.1663 43.375 0.0179729 0.0102847 2024-09-18T07:36:52.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:52 smithi203 ceph-mon[32237]: pgmap v316: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 35 MiB/s wr, 568 op/s; 19493/211026 objects misplaced (9.237%); 10 MiB/s, 173 objects/s recovering 2024-09-18T07:36:52.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:52 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:36:52.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:52 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:53.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:53.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:53.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:53.133 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:53.184 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:53.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:53.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:53.286 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:53.316 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:53.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:53.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:53.418 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:53.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:53.515 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:53.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:53.616 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:53.642 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:53.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:53.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:53.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:53 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:53.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:53 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:53 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:53 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:36:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:53 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:36:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:53 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:53.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:53 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:53.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:53 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:53.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:53 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:53.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:53 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:36:53.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:53 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:36:53.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:53 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:53.744 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:53.769 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:53.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:53.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:53.795 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:36:53.871 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:53.894 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:53.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:53.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:53.929 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 48 16 73724 73708 95.9481 38.6875 0.0210726 0.0104144 2024-09-18T07:36:53.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:53 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:53.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:53 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:53.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:53 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:53.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:53 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:36:53.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:53 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:36:53.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:53 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:36:53.995 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:54.019 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:54.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:54.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:54.120 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:54.148 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:54.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:54.250 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:54.280 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:54.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:54.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:54.382 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:54.426 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:54.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:54.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:54.528 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:54.553 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:54.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:54.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:54.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:54.692 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:54.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:54.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:54.742 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:36:54.793 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:54.815 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:54.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:54.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:54.916 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:54.929 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 49 16 74382 74366 94.8288 41.125 0.0321603 0.0105358 2024-09-18T07:36:54.953 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:54.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:55.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:55.103 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:55.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:55.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:55.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:54 smithi137 ceph-mon[29408]: pgmap v317: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 744 op/s; 19297/217179 objects misplaced (8.885%); 11 MiB/s, 176 objects/s recovering 2024-09-18T07:36:55.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:54 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1920914904' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:36:55.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:54 smithi137 ceph-mon[39175]: pgmap v317: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 744 op/s; 19297/217179 objects misplaced (8.885%); 11 MiB/s, 176 objects/s recovering 2024-09-18T07:36:55.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:54 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1920914904' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:36:55.205 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:55.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:54 smithi203 ceph-mon[32237]: pgmap v317: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 744 op/s; 19297/217179 objects misplaced (8.885%); 11 MiB/s, 176 objects/s recovering 2024-09-18T07:36:55.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:54 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1920914904' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:36:55.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:55.255 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:55.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:55.356 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:55.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:55.384 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:55.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:55.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:55.526 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:55.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:55.546 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:36:55.546 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:36:55.627 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:55.672 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:55.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:55.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:55.774 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:55.801 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:55.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:55.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:55.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:55.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:55 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:55.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:55 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1920914904' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:36:55.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:55 smithi137 ceph-mon[39175]: osdmap e224: 8 total, 8 up, 7 in 2024-09-18T07:36:55.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:55 smithi137 ceph-mon[39175]: osdmap e225: 8 total, 8 up, 7 in 2024-09-18T07:36:55.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:55 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:55.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:55 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1920914904' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:36:55.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:55 smithi137 ceph-mon[29408]: osdmap e224: 8 total, 8 up, 7 in 2024-09-18T07:36:55.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:55 smithi137 ceph-mon[29408]: osdmap e225: 8 total, 8 up, 7 in 2024-09-18T07:36:55.928 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:55.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:55.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:55.929 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 50 16 75117 75101 93.8511 45.9375 0.00434688 0.0106442 2024-09-18T07:36:56.030 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:56.068 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:56.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:56.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:56.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:56.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:55 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:36:56.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:55 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1920914904' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:36:56.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:55 smithi203 ceph-mon[32237]: osdmap e224: 8 total, 8 up, 7 in 2024-09-18T07:36:56.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:55 smithi203 ceph-mon[32237]: osdmap e225: 8 total, 8 up, 7 in 2024-09-18T07:36:56.212 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:56.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:56.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:56.314 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:56.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:56.341 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:56.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:56.442 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:56.469 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:56.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:56.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:56.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:56.612 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:56.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:56.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:56.714 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:56.741 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:56.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:56.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:56.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:56.883 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:56.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:56.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:56.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T07:36:56.930 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 51 16 75915 75899 92.9888 49.875 0.0172412 0.010746 2024-09-18T07:36:56.985 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:57.010 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:57.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:57.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:57.104 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:56 smithi137 ceph-mon[29408]: pgmap v320: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 29 MiB/s wr, 460 op/s; 19297/217179 objects misplaced (8.885%); 6.7 MiB/s, 106 objects/s recovering 2024-09-18T07:36:57.104 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:56 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:36:57.104 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:56 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:36:57.104 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:56 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:36:57.104 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:56 smithi137 ceph-mon[39175]: pgmap v320: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 29 MiB/s wr, 460 op/s; 19297/217179 objects misplaced (8.885%); 6.7 MiB/s, 106 objects/s recovering 2024-09-18T07:36:57.104 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:56 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:36:57.104 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:56 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:36:57.105 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:56 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:36:57.111 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:57.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:57.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:57.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:57.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:56 smithi203 ceph-mon[32237]: pgmap v320: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 29 MiB/s wr, 460 op/s; 19297/217179 objects misplaced (8.885%); 6.7 MiB/s, 106 objects/s recovering 2024-09-18T07:36:57.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:56 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:36:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:56 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:36:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:56 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:36:57.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:36:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:36:57.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:57.306 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:57.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:57.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:57.421 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:36:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:36:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:36:57.454 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:57.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:57.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:57.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:57.587 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:57.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:57.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:57.689 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:36:57.715 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:57.737 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:57.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:57.769 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:57.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:36:57.775 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:36:57.871 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:57.930 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 52 16 76628 76612 92.0576 44.5625 0.0249281 0.0108558 2024-09-18T07:36:57.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:57.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:57.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:58.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:58.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:58.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:58.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:58.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:57 smithi137 ceph-mon[29408]: osdmap e226: 8 total, 8 up, 7 in 2024-09-18T07:36:58.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:57 smithi137 ceph-mon[39175]: osdmap e226: 8 total, 8 up, 7 in 2024-09-18T07:36:58.187 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:58.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:57 smithi203 ceph-mon[32237]: osdmap e226: 8 total, 8 up, 7 in 2024-09-18T07:36:58.216 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:58.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:58.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:36:58.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:58.364 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:58.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:58.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:58.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:58.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:58.514 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:58.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:58.615 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:36:58.642 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:58.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:58.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:36:58.744 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:58.790 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:58.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:58.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:58.892 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:58.919 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:58.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:58.930 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 53 16 77459 77443 91.3007 51.9375 0.00423715 0.0109459 2024-09-18T07:36:59.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:59.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:59.051 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:59.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:59.152 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:59.182 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:59.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:59.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:59.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:58 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:59.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:36:58 smithi203 ceph-mon[32237]: pgmap v322: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 927 op/s; 18456/227712 objects misplaced (8.105%); 11 MiB/s, 172 objects/s recovering 2024-09-18T07:36:59.283 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:58 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:59.283 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:36:58 smithi137 ceph-mon[29408]: pgmap v322: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 927 op/s; 18456/227712 objects misplaced (8.105%); 11 MiB/s, 172 objects/s recovering 2024-09-18T07:36:59.284 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:58 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:36:59.284 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:36:58 smithi137 ceph-mon[39175]: pgmap v322: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 58 MiB/s wr, 927 op/s; 18456/227712 objects misplaced (8.105%); 11 MiB/s, 172 objects/s recovering 2024-09-18T07:36:59.284 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:36:59.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:59.312 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:59.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:36:59.413 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:36:59.441 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:59.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:59.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:36:59.543 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:36:59.568 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:59.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:59.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:36:59.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:36:59.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:36:59.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:59.703 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:59.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:36:59.804 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:36:59.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:59.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:36:59.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:36:59.930 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 54 16 78190 78174 90.4561 45.6875 0.0236178 0.011049 2024-09-18T07:36:59.951 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:36:59.976 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:36:59.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:36:59.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:00.078 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:00.114 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:00.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:00.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:00.216 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:00.253 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:00.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:00.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:00.355 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:00.381 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:00.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:00.483 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:00.505 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:00.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:00.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:00.607 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:00.633 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:00.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:00.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:00.735 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:00.762 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:00.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:00.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:00.865 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:00.893 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:00.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:00.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:00.930 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 55 16 78958 78942 89.6842 48 0.0103525 0.0111433 2024-09-18T07:37:00.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:01.038 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:01.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:01.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:01.140 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:01.177 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:00 smithi137 ceph-mon[29408]: pgmap v323: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 37 MiB/s wr, 585 op/s; 18456/227712 objects misplaced (8.105%); 8.8 MiB/s, 140 objects/s recovering 2024-09-18T07:37:01.177 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:00 smithi137 ceph-mon[29408]: osdmap e227: 8 total, 8 up, 7 in 2024-09-18T07:37:01.177 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:00 smithi137 ceph-mon[39175]: pgmap v323: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 37 MiB/s wr, 585 op/s; 18456/227712 objects misplaced (8.105%); 8.8 MiB/s, 140 objects/s recovering 2024-09-18T07:37:01.178 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:00 smithi137 ceph-mon[39175]: osdmap e227: 8 total, 8 up, 7 in 2024-09-18T07:37:01.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:01.181 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:01.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:01.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:00 smithi203 ceph-mon[32237]: pgmap v323: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 37 MiB/s wr, 585 op/s; 18456/227712 objects misplaced (8.105%); 8.8 MiB/s, 140 objects/s recovering 2024-09-18T07:37:01.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:00 smithi203 ceph-mon[32237]: osdmap e227: 8 total, 8 up, 7 in 2024-09-18T07:37:01.282 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:01.290 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-18T07:37:01.290 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-18T07:37:01.290 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-18T07:37:01.290 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-18T07:37:01.290 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-18T07:37:01.290 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-18T07:37:01.290 INFO:teuthology.orchestra.run.smithi137.stderr:bench [] [] [] [] 2024-09-18T07:37:01.290 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-18T07:37:01.290 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-18T07:37:01.290 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-18T07:37:01.291 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator dump block 2024-09-18T07:37:01.291 INFO:teuthology.orchestra.run.smithi137.stderr:Error EINVAL: invalid command 2024-09-18T07:37:01.308 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:01.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:01.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:01.410 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:01.436 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:01.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:01.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:01.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:01.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:01.583 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:01.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:01.666 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:37:01.667 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:37:01.667 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:37:01.667 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:37:01.667 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:37:01.667 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:37:01.667 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:37:01.667 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:37:01.684 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:01.703 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:37:01.703 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:37:01.739 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:37:01.740 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:37:01.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:01.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:01.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:01.771 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:37:01.771 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:37:01.807 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:37:01.807 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:37:01.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:01.884 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:01.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:01.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:01.930 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 56 16 79664 79648 88.8707 44.125 0.00782332 0.0112463 2024-09-18T07:37:01.989 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:02.018 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:02.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:02.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:02.120 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:02.148 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:02.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:02.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:02.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:01 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3016927352' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:02.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:01 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3016927352' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:02.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:01 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3016927352' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:02.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:02.281 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T07:37:02.282 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T07:37:02.282 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-18T07:37:02.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:02.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:02.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:02.399 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:02.422 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:02.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:02.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:02.525 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:02.552 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:02.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:02.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:02.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:02.718 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:02.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:02.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:02.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:02.845 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:02.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:02.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:02.930 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 57 16 80414 80398 88.134 46.875 0.00631527 0.0113395 2024-09-18T07:37:02.948 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:02.968 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:02.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:02.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:03.038 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:02 smithi137 ceph-mon[29408]: pgmap v325: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.5 KiB/s rd, 38 MiB/s wr, 603 op/s; 18214/229161 objects misplaced (7.948%); 10 MiB/s, 163 objects/s recovering 2024-09-18T07:37:03.039 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:02 smithi137 ceph-mon[39175]: pgmap v325: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.5 KiB/s rd, 38 MiB/s wr, 603 op/s; 18214/229161 objects misplaced (7.948%); 10 MiB/s, 163 objects/s recovering 2024-09-18T07:37:03.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:03.083 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:37:03.083 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:37:03.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.083 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35171.0:80481 8.3 e227/219 8:c0beb681:::benchmark_data_smithi137_101942_object80480:head v 227'5062, mlcod=209'902)", 2024-09-18T07:37:03.083 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:03.069675+0000", 2024-09-18T07:37:03.083 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0094850479999999994, 2024-09-18T07:37:03.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095142490000000007, 2024-09-18T07:37:03.084 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:37:03.084 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:37:03.084 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:37:03.084 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:37:03.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:37:03.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.069675+0000", 2024-09-18T07:37:03.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:03.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:37:03.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.069675+0000", 2024-09-18T07:37:03.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:03.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:37:03.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.069678+0000", 2024-09-18T07:37:03.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1039999999999998e-06 2024-09-18T07:37:03.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:37:03.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.069750+0000", 2024-09-18T07:37:03.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2337999999999998e-05 2024-09-18T07:37:03.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:37:03.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.069770+0000", 2024-09-18T07:37:03.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0034000000000002e-05 2024-09-18T07:37:03.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:37:03.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.069781+0000", 2024-09-18T07:37:03.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0645999999999999e-05 2024-09-18T07:37:03.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:37:03.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.069848+0000", 2024-09-18T07:37:03.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7310000000000004e-05 2024-09-18T07:37:03.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:37:03.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.069875+0000", 2024-09-18T07:37:03.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6393999999999998e-05 2024-09-18T07:37:03.087 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:03.087 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:37:03.087 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:03.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.088 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35171.0:80482 8.f e227/201 8:f3de4751:::benchmark_data_smithi137_101942_object80481:head v 227'5052, mlcod=227'5050)", 2024-09-18T07:37:03.088 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:03.077144+0000", 2024-09-18T07:37:03.088 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0020157930000000001, 2024-09-18T07:37:03.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0021066520000000001, 2024-09-18T07:37:03.088 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:37:03.088 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:37:03.088 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:37:03.088 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:37:03.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:37:03.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.077144+0000", 2024-09-18T07:37:03.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:03.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:37:03.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.077144+0000", 2024-09-18T07:37:03.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:03.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:37:03.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.077145+0000", 2024-09-18T07:37:03.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5549999999999999e-06 2024-09-18T07:37:03.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:37:03.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.077170+0000", 2024-09-18T07:37:03.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4882e-05 2024-09-18T07:37:03.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:37:03.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.077180+0000", 2024-09-18T07:37:03.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0061000000000001e-05 2024-09-18T07:37:03.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:37:03.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.077188+0000", 2024-09-18T07:37:03.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.187e-06 2024-09-18T07:37:03.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:37:03.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.077264+0000", 2024-09-18T07:37:03.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6429999999999998e-05 2024-09-18T07:37:03.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:37:03.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.077286+0000", 2024-09-18T07:37:03.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1940999999999999e-05 2024-09-18T07:37:03.091 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:03.091 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:37:03.091 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:03.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.092 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35171.0:80480 8.a e227/210 8:502a0959:::benchmark_data_smithi137_101942_object80479:head v 227'4984, mlcod=227'4982)", 2024-09-18T07:37:03.092 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:03.078587+0000", 2024-09-18T07:37:03.092 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00057238999999999999, 2024-09-18T07:37:03.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00071552699999999998, 2024-09-18T07:37:03.092 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:37:03.092 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:37:03.092 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:37:03.092 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:37:03.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:37:03.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.078587+0000", 2024-09-18T07:37:03.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:03.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:37:03.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.078587+0000", 2024-09-18T07:37:03.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:03.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:37:03.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.078589+0000", 2024-09-18T07:37:03.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.452e-06 2024-09-18T07:37:03.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:37:03.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.078616+0000", 2024-09-18T07:37:03.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7566999999999999e-05 2024-09-18T07:37:03.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:37:03.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.078629+0000", 2024-09-18T07:37:03.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2836e-05 2024-09-18T07:37:03.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:37:03.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.078637+0000", 2024-09-18T07:37:03.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5050000000000001e-06 2024-09-18T07:37:03.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:37:03.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.079018+0000", 2024-09-18T07:37:03.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00038096700000000001 2024-09-18T07:37:03.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:03.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:03.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:37:03.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:03.079061+0000", 2024-09-18T07:37:03.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3671000000000003e-05 2024-09-18T07:37:03.095 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:03.095 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:37:03.095 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:03.096 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:03.096 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:37:03.096 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 3 2024-09-18T07:37:03.096 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:37:03.118 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:03.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:03.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:03.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:02 smithi203 ceph-mon[32237]: pgmap v325: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.5 KiB/s rd, 38 MiB/s wr, 603 op/s; 18214/229161 objects misplaced (7.948%); 10 MiB/s, 163 objects/s recovering 2024-09-18T07:37:03.220 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:03.251 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:03.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:03.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:03.353 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:03.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:03.399 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:03.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:03.500 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:03.525 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:03.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:03.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:03.627 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:03.655 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:03.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:03.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:03.747 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:03.757 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:03.780 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:03.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:03.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:03.881 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:03.908 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:03.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:03.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:03.930 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 58 16 81139 81123 87.3957 45.3125 0.0130616 0.0114345 2024-09-18T07:37:04.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:04.048 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:04.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:04.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:04.150 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:04.174 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:04.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:04.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:04.276 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:04.322 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:04.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:04.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:04.382 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T07:37:04.424 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:04.448 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:04.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:04.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:04.550 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:04.578 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:04.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:04.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:04.680 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:04.712 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:04.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:04.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:04.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:04.855 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:04.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:04.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:04.930 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 59 16 81872 81856 86.691 45.8125 0.0209999 0.0115293 2024-09-18T07:37:04.958 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:04.984 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:04.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:04.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:05.086 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:05.098 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:05.113 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:05.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:05.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:04 smithi203 ceph-mon[32237]: pgmap v326: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 915 op/s; 17579/239121 objects misplaced (7.352%); 13 MiB/s, 214 objects/s recovering 2024-09-18T07:37:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:04 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:37:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:04 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:37:05.214 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:05.237 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:04 smithi137 ceph-mon[29408]: pgmap v326: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 915 op/s; 17579/239121 objects misplaced (7.352%); 13 MiB/s, 214 objects/s recovering 2024-09-18T07:37:05.237 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:04 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:37:05.237 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:04 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:37:05.237 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:04 smithi137 ceph-mon[39175]: pgmap v326: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 915 op/s; 17579/239121 objects misplaced (7.352%); 13 MiB/s, 214 objects/s recovering 2024-09-18T07:37:05.237 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:04 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:37:05.237 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:04 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:37:05.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:05.241 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:05.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:05.342 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:05.366 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:05.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:05.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:05.468 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:05.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:05.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:05.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:05.527 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:37:05.611 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:05.636 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:05.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:05.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:05.738 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:05.763 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:05.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:05.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:05.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:05.925 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:05.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:05.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:05.931 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:37:05.929949+0000 min lat: 0.00114205 max lat: 2.28882 avg lat: 0.01163 2024-09-18T07:37:05.931 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:37:05.931 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 60 16 82527 82511 85.9271 40.9375 0.0331843 0.01163 2024-09-18T07:37:05.949 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:37:06.027 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:06.033 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-18T07:37:06.033 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-18T07:37:06.033 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-18T07:37:06.033 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-18T07:37:06.033 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-18T07:37:06.033 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-18T07:37:06.033 INFO:teuthology.orchestra.run.smithi137.stderr:bench [] [] [] [] 2024-09-18T07:37:06.033 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-18T07:37:06.034 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-18T07:37:06.034 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-18T07:37:06.034 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator dump block 2024-09-18T07:37:06.034 INFO:teuthology.orchestra.run.smithi137.stderr:Error EINVAL: invalid command 2024-09-18T07:37:06.055 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:06.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:06.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:06.157 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:06.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:05 smithi137 ceph-mon[29408]: osdmap e228: 8 total, 8 up, 7 in 2024-09-18T07:37:06.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:05 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3350630146' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:37:06.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:05 smithi137 ceph-mon[39175]: osdmap e228: 8 total, 8 up, 7 in 2024-09-18T07:37:06.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:05 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3350630146' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:37:06.182 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:06.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:06.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:06.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:05 smithi203 ceph-mon[32237]: osdmap e228: 8 total, 8 up, 7 in 2024-09-18T07:37:06.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:05 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3350630146' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:37:06.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:06.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:06.335 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:06.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:06.436 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:06.457 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:06.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:06.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:06.559 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:06.587 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:06.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:06.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:06.689 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:06.718 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:06.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:06.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:06.820 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:06.862 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:06.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:06.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:06.931 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 61 16 83272 83256 85.2818 46.5625 0.00384773 0.0117177 2024-09-18T07:37:06.964 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:06.985 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:06.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:06.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:07.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:07.124 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:07.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:07.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:07.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:37:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:37:07.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:06 smithi203 ceph-mon[32237]: pgmap v328: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 30 MiB/s wr, 475 op/s; 17579/239121 objects misplaced (7.352%); 6.8 MiB/s, 109 objects/s recovering 2024-09-18T07:37:07.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:06 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:37:07.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:06 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:37:07.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:06 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3350630146' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:37:07.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:06 smithi203 ceph-mon[32237]: osdmap e229: 8 total, 8 up, 7 in 2024-09-18T07:37:07.226 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:07.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:07.251 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:07.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:07.352 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:07.381 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:07.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:07.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:37:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:37:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:37:07.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:06 smithi137 ceph-mon[29408]: pgmap v328: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 30 MiB/s wr, 475 op/s; 17579/239121 objects misplaced (7.352%); 6.8 MiB/s, 109 objects/s recovering 2024-09-18T07:37:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:06 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:37:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:06 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:37:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:06 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3350630146' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:37:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:06 smithi137 ceph-mon[29408]: osdmap e229: 8 total, 8 up, 7 in 2024-09-18T07:37:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:06 smithi137 ceph-mon[39175]: pgmap v328: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 30 MiB/s wr, 475 op/s; 17579/239121 objects misplaced (7.352%); 6.8 MiB/s, 109 objects/s recovering 2024-09-18T07:37:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:06 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:37:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:06 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:37:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:06 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3350630146' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:37:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:06 smithi137 ceph-mon[39175]: osdmap e229: 8 total, 8 up, 7 in 2024-09-18T07:37:07.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:07.504 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:07.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:07.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:07.606 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:07.647 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:07.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:07.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:07.674 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:07.749 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:07.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:07.800 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:07.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:07.901 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:07.932 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 62 16 83997 83981 84.6367 45.3125 0.0222219 0.0118083 2024-09-18T07:37:07.938 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:07.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:08.040 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:08.079 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:08.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:08.180 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:08.207 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:08.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:08.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:08.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:08.335 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:08.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:08.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:08.417 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T07:37:08.417 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T07:37:08.418 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-18T07:37:08.436 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:08.520 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:08.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:08.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:08.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:08.661 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:08.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:08.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:08.764 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:08.792 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:08.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:08.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:08.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:08.932 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 63 16 84757 84741 84.0471 47.5 0.0210206 0.0118917 2024-09-18T07:37:08.944 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:08.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:08.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:09.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:09.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:09.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:09.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:09.160 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:08 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:09.160 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:08 smithi137 ceph-mon[29408]: pgmap v330: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.3 KiB/s rd, 60 MiB/s wr, 968 op/s; 16665/249879 objects misplaced (6.669%); 15 MiB/s, 234 objects/s recovering 2024-09-18T07:37:09.161 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:08 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:09.161 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:08 smithi137 ceph-mon[39175]: pgmap v330: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.3 KiB/s rd, 60 MiB/s wr, 968 op/s; 16665/249879 objects misplaced (6.669%); 15 MiB/s, 234 objects/s recovering 2024-09-18T07:37:09.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:09.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:08 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:09.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:08 smithi203 ceph-mon[32237]: pgmap v330: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.3 KiB/s rd, 60 MiB/s wr, 968 op/s; 16665/249879 objects misplaced (6.669%); 15 MiB/s, 234 objects/s recovering 2024-09-18T07:37:09.248 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:09.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:09.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:09.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:09.403 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:09.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:09.454 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:09.503 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:37:09.503 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:37:09.503 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:37:09.503 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:37:09.503 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:37:09.506 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:09.531 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:09.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:09.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:09.675 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:09.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:09.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:09.765 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:09.788 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:09.814 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:09.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:09.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:09.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:09.932 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 64 16 85520 85504 83.4784 47.6875 0.0176961 0.0119741 2024-09-18T07:37:09.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:09.956 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:09.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:10.057 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:10.088 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:10.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:10.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:10.226 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:10.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:10.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:10.273 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T07:37:10.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:10.357 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:10.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:10.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:10.458 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:10.485 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:10.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:10.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:10.587 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:10.616 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:10.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:10.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:10.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:10.757 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:10.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:10.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:10.858 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:10.889 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:10.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:10.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:10.936 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 65 16 86276 86260 82.9195 47.25 0.025989 0.0120542 2024-09-18T07:37:10.959 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:37:10.991 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:11.000 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:37:11.001 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:37:11.014 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:11.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:11.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:11.039 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:37:11.039 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:37:11.079 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:37:11.079 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:37:11.114 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:37:11.115 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:37:11.116 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:11.144 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:11.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:11.148 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:37:11.148 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:37:11.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:10 smithi137 ceph-mon[29408]: pgmap v331: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 863 op/s; 16665/249879 objects misplaced (6.669%); 12 MiB/s, 193 objects/s recovering 2024-09-18T07:37:11.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:10 smithi137 ceph-mon[39175]: pgmap v331: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 863 op/s; 16665/249879 objects misplaced (6.669%); 12 MiB/s, 193 objects/s recovering 2024-09-18T07:37:11.185 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:37:11.185 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:37:11.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:10 smithi203 ceph-mon[32237]: pgmap v331: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 863 op/s; 16665/249879 objects misplaced (6.669%); 12 MiB/s, 193 objects/s recovering 2024-09-18T07:37:11.216 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:37:11.216 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:37:11.246 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:11.250 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:37:11.250 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:37:11.271 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:11.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:11.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:11.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:11.397 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:11.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:11.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:11.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:11.534 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:11.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:11.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:11.608 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:11.637 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:11.665 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:11.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:11.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:11.794 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:11.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:11.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:11.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:11.916 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-18T07:37:11.916 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-18T07:37:11.916 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-18T07:37:11.916 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-18T07:37:11.917 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-18T07:37:11.917 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-18T07:37:11.917 INFO:teuthology.orchestra.run.smithi137.stderr:bench [] [] [] [] 2024-09-18T07:37:11.917 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-18T07:37:11.917 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-18T07:37:11.917 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-18T07:37:11.917 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator dump block 2024-09-18T07:37:11.917 INFO:teuthology.orchestra.run.smithi137.stderr:Error EINVAL: invalid command 2024-09-18T07:37:11.923 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:11.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:11.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:11.934 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 66 16 86925 86909 82.2778 40.5625 0.0359601 0.0121474 2024-09-18T07:37:12.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:12.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:12.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:12.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:12.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:11 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/512570923' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:12.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:11 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:37:12.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:11 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:37:12.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:11 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/512570923' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:12.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:11 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:37:12.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:11 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:37:12.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:12.201 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:12.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:12.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:12.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:11 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/512570923' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:12.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:11 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:37:12.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:11 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:37:12.303 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:12.330 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:12.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:12.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:12.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:12.459 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:12.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:12.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:12.561 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:12.589 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:12.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:12.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:12.674 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:12.691 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:12.716 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:12.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:12.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:12.818 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:12.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:12.858 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:12.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:12.934 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 67 16 87717 87701 81.7886 49.5 0.0286454 0.0122204 2024-09-18T07:37:12.959 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:12.984 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:12.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:12.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:13.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:13.120 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:13.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:13.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:13.151 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:12 smithi137 ceph-mon[29408]: pgmap v332: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 32 MiB/s wr, 509 op/s; 16419/251331 objects misplaced (6.533%); 9.1 MiB/s, 144 objects/s recovering 2024-09-18T07:37:13.151 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:12 smithi137 ceph-mon[39175]: pgmap v332: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 32 MiB/s wr, 509 op/s; 16419/251331 objects misplaced (6.533%); 9.1 MiB/s, 144 objects/s recovering 2024-09-18T07:37:13.207 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T07:37:13.207 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T07:37:13.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-18T07:37:13.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:12 smithi203 ceph-mon[32237]: pgmap v332: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 32 MiB/s wr, 509 op/s; 16419/251331 objects misplaced (6.533%); 9.1 MiB/s, 144 objects/s recovering 2024-09-18T07:37:13.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:13.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:13.289 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:13.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:13.390 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:13.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:13.419 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:13.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:13.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:13.566 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:13.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:13.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:13.668 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:13.696 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:13.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:13.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:13.760 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:37:13.761 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:37:13.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:13.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:13.859 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:13.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:13.934 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 68 16 88445 88429 81.255 45.5 0.00756254 0.0123013 2024-09-18T07:37:13.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:14.000 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:14.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:14.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:14.102 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:14.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:14.145 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:14.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:14.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:14.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:14.312 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:14.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:14.413 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:14.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:14.441 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:14.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:14.478 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:37:14.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:14.599 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:14.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:14.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:14.701 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:14.751 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:14.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:14.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:14.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T07:37:14.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:14.940 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 69 16 89183 89167 80.7411 46.125 0.00800388 0.0123761 2024-09-18T07:37:14.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:14.949 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:14.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:14.949 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:15.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:15.101 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:14 smithi137 ceph-mon[29408]: pgmap v333: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 848 op/s; 15724/260988 objects misplaced (6.025%); 13 MiB/s, 215 objects/s recovering 2024-09-18T07:37:15.101 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:14 smithi137 ceph-mon[39175]: pgmap v333: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 848 op/s; 15724/260988 objects misplaced (6.025%); 13 MiB/s, 215 objects/s recovering 2024-09-18T07:37:15.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:15.115 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:15.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:15.216 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:15.243 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:15.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:15.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:15.267 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:15.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:15.391 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:15.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:15.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:15.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:14 smithi203 ceph-mon[32237]: pgmap v333: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 848 op/s; 15724/260988 objects misplaced (6.025%); 13 MiB/s, 215 objects/s recovering 2024-09-18T07:37:15.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:15.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:15.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:15.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:15.668 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:15.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:15.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:15.770 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:15.795 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:15.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:15.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:15.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:15.939 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 70 16 89900 89884 80.2272 44.8125 0.0203729 0.0124574 2024-09-18T07:37:15.943 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:15.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:15.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:15.977 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:37:16.045 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:16.074 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:16.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:16.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:16.098 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:15 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2069683688' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:37:16.098 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:15 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2069683688' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:37:16.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:16.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:16.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:16.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:16.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:16.364 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:16.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:16.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:16.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:15 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2069683688' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:37:16.461 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:37:16.462 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:37:16.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:16.503 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:37:16.503 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:37:16.506 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:16.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:16.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:16.536 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:37:16.536 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:37:16.570 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:37:16.570 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:37:16.603 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:37:16.603 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:37:16.608 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:16.634 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-18T07:37:16.635 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-18T07:37:16.635 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-18T07:37:16.635 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-18T07:37:16.635 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-18T07:37:16.635 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-18T07:37:16.635 INFO:teuthology.orchestra.run.smithi137.stderr:bench [] [] [] [] 2024-09-18T07:37:16.635 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-18T07:37:16.635 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-18T07:37:16.635 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-18T07:37:16.635 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator dump block 2024-09-18T07:37:16.635 INFO:teuthology.orchestra.run.smithi137.stderr:Error EINVAL: invalid command 2024-09-18T07:37:16.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:16.637 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:16.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:16.638 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:37:16.638 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:37:16.671 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:37:16.671 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:37:16.704 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:37:16.705 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:37:16.737 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:16.771 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:16.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:16.873 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:16.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:16.920 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:16.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:16.939 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 71 16 90617 90601 79.7284 44.8125 0.0338943 0.0125366 2024-09-18T07:37:17.021 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:17.063 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:17.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:17.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:17.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:17.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:17.207 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:17.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:17.308 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:17.339 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:17.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:17.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:17.360 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:37:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:37:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:37:17.360 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:16 smithi137 ceph-mon[39175]: pgmap v334: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 46 MiB/s wr, 729 op/s; 15724/260988 objects misplaced (6.025%); 12 MiB/s, 185 objects/s recovering 2024-09-18T07:37:17.360 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:16 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:37:17.360 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:16 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2069683688' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:37:17.360 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:16 smithi137 ceph-mon[39175]: osdmap e230: 8 total, 8 up, 7 in 2024-09-18T07:37:17.360 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:16 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2532870100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:17.361 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:16 smithi137 ceph-mon[29408]: pgmap v334: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 46 MiB/s wr, 729 op/s; 15724/260988 objects misplaced (6.025%); 12 MiB/s, 185 objects/s recovering 2024-09-18T07:37:17.361 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:16 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:37:17.361 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:16 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2069683688' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:37:17.361 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:16 smithi137 ceph-mon[29408]: osdmap e230: 8 total, 8 up, 7 in 2024-09-18T07:37:17.361 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:16 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2532870100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:17.441 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:17.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:37:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:37:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:16 smithi203 ceph-mon[32237]: pgmap v334: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 46 MiB/s wr, 729 op/s; 15724/260988 objects misplaced (6.025%); 12 MiB/s, 185 objects/s recovering 2024-09-18T07:37:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:16 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:37:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:16 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2069683688' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:37:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:16 smithi203 ceph-mon[32237]: osdmap e230: 8 total, 8 up, 7 in 2024-09-18T07:37:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:16 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2532870100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:17.468 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:17.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:17.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:17.570 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:17.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:17.599 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:17.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:17.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:17.738 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:17.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:17.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:17.839 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:17.865 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:17.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:17.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:17.939 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 72 16 91438 91422 79.3338 51.3125 0.0306844 0.012598 2024-09-18T07:37:17.967 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:17.995 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:17.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:17.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:18.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:18.123 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:18.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:18.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:18.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:18.264 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T07:37:18.265 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T07:37:18.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-18T07:37:18.284 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:18.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:18.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:18.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:18.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:18.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:18.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:18.544 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:18.596 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:18.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:18.698 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:18.725 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:18.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:18.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:18.827 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:18.853 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:18.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:18.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:18.939 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 73 16 92218 92202 78.9149 48.75 0.0193976 0.012666 2024-09-18T07:37:18.955 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:18.958 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:18.979 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:18.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:18.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:19.082 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:19.108 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:19.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:19.206 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:18 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:19.206 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:18 smithi137 ceph-mon[29408]: pgmap v336: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 46 MiB/s wr, 736 op/s; 14777/271968 objects misplaced (5.433%); 12 MiB/s, 188 objects/s recovering 2024-09-18T07:37:19.206 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:18 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:19.206 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:18 smithi137 ceph-mon[39175]: pgmap v336: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 46 MiB/s wr, 736 op/s; 14777/271968 objects misplaced (5.433%); 12 MiB/s, 188 objects/s recovering 2024-09-18T07:37:19.210 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:19.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:19.237 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:19.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:19.338 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:19.370 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:19.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:19.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:19.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:18 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:18 smithi203 ceph-mon[32237]: pgmap v336: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 46 MiB/s wr, 736 op/s; 14777/271968 objects misplaced (5.433%); 12 MiB/s, 188 objects/s recovering 2024-09-18T07:37:19.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:19.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:19.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:19.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:19.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:19.649 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:19.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:19.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:19.651 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:19.691 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:37:19.691 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:37:19.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.691 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35171.0:92748 8.7 e230/219 8:ee3785b3:::benchmark_data_smithi137_101942_object92747:head v 230'5983, mlcod=208'115)", 2024-09-18T07:37:19.691 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:19.666780+0000", 2024-09-18T07:37:19.691 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.020528022999999999, 2024-09-18T07:37:19.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020558073, 2024-09-18T07:37:19.691 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:37:19.692 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:37:19.692 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:37:19.692 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:37:19.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:37:19.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.666780+0000", 2024-09-18T07:37:19.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:19.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:37:19.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.666780+0000", 2024-09-18T07:37:19.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:19.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:37:19.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.666780+0000", 2024-09-18T07:37:19.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3600000000000003e-07 2024-09-18T07:37:19.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:37:19.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.666800+0000", 2024-09-18T07:37:19.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9833000000000001e-05 2024-09-18T07:37:19.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:37:19.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.666806+0000", 2024-09-18T07:37:19.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.404e-06 2024-09-18T07:37:19.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:37:19.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.666811+0000", 2024-09-18T07:37:19.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.95e-06 2024-09-18T07:37:19.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:37:19.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.679631+0000", 2024-09-18T07:37:19.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012819551 2024-09-18T07:37:19.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:37:19.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.679717+0000", 2024-09-18T07:37:19.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5729999999999994e-05 2024-09-18T07:37:19.696 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:19.696 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:37:19.696 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:19.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.696 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:92765 8.a 8:525b5602:::benchmark_data_smithi137_101942_object92764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e230)", 2024-09-18T07:37:19.696 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:19.671283+0000", 2024-09-18T07:37:19.696 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.016024984999999999, 2024-09-18T07:37:19.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016102675, 2024-09-18T07:37:19.696 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:37:19.697 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:37:19.697 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:19.697 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:37:19.697 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:37:19.697 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:37:19.697 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 92765 2024-09-18T07:37:19.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.697 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:37:19.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:37:19.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.671283+0000", 2024-09-18T07:37:19.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:19.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:37:19.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.671283+0000", 2024-09-18T07:37:19.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:19.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:37:19.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.671285+0000", 2024-09-18T07:37:19.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.295e-06 2024-09-18T07:37:19.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:37:19.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.671319+0000", 2024-09-18T07:37:19.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3809000000000001e-05 2024-09-18T07:37:19.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:37:19.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.671335+0000", 2024-09-18T07:37:19.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6095000000000001e-05 2024-09-18T07:37:19.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:37:19.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.671343+0000", 2024-09-18T07:37:19.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.439e-06 2024-09-18T07:37:19.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:37:19.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.671432+0000", 2024-09-18T07:37:19.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9170000000000002e-05 2024-09-18T07:37:19.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:37:19.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.671562+0000", 2024-09-18T07:37:19.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012919900000000001 2024-09-18T07:37:19.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:37:19.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.671698+0000", 2024-09-18T07:37:19.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000136821 2024-09-18T07:37:19.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:19.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.675852+0000", 2024-09-18T07:37:19.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041531279999999999 2024-09-18T07:37:19.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:37:19.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.675875+0000", 2024-09-18T07:37:19.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3062999999999999e-05 2024-09-18T07:37:19.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:19.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.682652+0000", 2024-09-18T07:37:19.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0067771330000000003 2024-09-18T07:37:19.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:37:19.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.687334+0000", 2024-09-18T07:37:19.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046823029999999996 2024-09-18T07:37:19.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:37:19.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.687385+0000", 2024-09-18T07:37:19.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1220000000000001e-05 2024-09-18T07:37:19.704 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:19.704 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:37:19.704 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:19.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.704 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.35171.0:92765 8.a e230/210 ondisk, result = 0)", 2024-09-18T07:37:19.704 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:19.680510+0000", 2024-09-18T07:37:19.704 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0067978989999999996, 2024-09-18T07:37:19.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0069747969999999996, 2024-09-18T07:37:19.704 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:37:19.705 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:37:19.705 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:37:19.705 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:37:19.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:37:19.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.680510+0000", 2024-09-18T07:37:19.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:19.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:37:19.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.680510+0000", 2024-09-18T07:37:19.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:19.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:37:19.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.680511+0000", 2024-09-18T07:37:19.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3149999999999999e-06 2024-09-18T07:37:19.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:37:19.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.680516+0000", 2024-09-18T07:37:19.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.271e-06 2024-09-18T07:37:19.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:37:19.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.680526+0000", 2024-09-18T07:37:19.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3810000000000005e-06 2024-09-18T07:37:19.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:37:19.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.680532+0000", 2024-09-18T07:37:19.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3840000000000002e-06 2024-09-18T07:37:19.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:37:19.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.682610+0000", 2024-09-18T07:37:19.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002078426 2024-09-18T07:37:19.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:37:19.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.682630+0000", 2024-09-18T07:37:19.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.999e-05 2024-09-18T07:37:19.708 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:19.708 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:37:19.709 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:19.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.709 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "MOSDPGPush(8.7 230/219 [PushOp(8:e5638d85:::benchmark_data_smithi137_101942_object10876:head, version: 209'708, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:e5638d85:::benchmark_data_smithi137_101942_object10876:head@209'708, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-18T07:37:19.709 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:19.680749+0000", 2024-09-18T07:37:19.709 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.006558459, 2024-09-18T07:37:19.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068081299999999999, 2024-09-18T07:37:19.709 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:37:19.709 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:37:19.709 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:37:19.710 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:37:19.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:37:19.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.680749+0000", 2024-09-18T07:37:19.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:19.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:37:19.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.680749+0000", 2024-09-18T07:37:19.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:19.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:37:19.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.680750+0000", 2024-09-18T07:37:19.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3230000000000001e-06 2024-09-18T07:37:19.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:37:19.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.680780+0000", 2024-09-18T07:37:19.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9668000000000001e-05 2024-09-18T07:37:19.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:37:19.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.680803+0000", 2024-09-18T07:37:19.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3076000000000001e-05 2024-09-18T07:37:19.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:37:19.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.680825+0000", 2024-09-18T07:37:19.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1648e-05 2024-09-18T07:37:19.715 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:19.715 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:37:19.715 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:19.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.716 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35171.0:92768 8.e e230/201 8:773c538f:::benchmark_data_smithi137_101942_object92767:head v 230'5696, mlcod=230'5694)", 2024-09-18T07:37:19.716 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:19.685895+0000", 2024-09-18T07:37:19.716 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0014128649999999999, 2024-09-18T07:37:19.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017002969999999999, 2024-09-18T07:37:19.717 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:37:19.717 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:37:19.717 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:37:19.717 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:37:19.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:37:19.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.685895+0000", 2024-09-18T07:37:19.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:19.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:37:19.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.685895+0000", 2024-09-18T07:37:19.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:19.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:37:19.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.685896+0000", 2024-09-18T07:37:19.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5349999999999999e-06 2024-09-18T07:37:19.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:37:19.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.685921+0000", 2024-09-18T07:37:19.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5174e-05 2024-09-18T07:37:19.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:37:19.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.685933+0000", 2024-09-18T07:37:19.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1432e-05 2024-09-18T07:37:19.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:37:19.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.685940+0000", 2024-09-18T07:37:19.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9430000000000003e-06 2024-09-18T07:37:19.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:37:19.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.686016+0000", 2024-09-18T07:37:19.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6217999999999994e-05 2024-09-18T07:37:19.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:19.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:19.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:37:19.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:19.686035+0000", 2024-09-18T07:37:19.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8841000000000001e-05 2024-09-18T07:37:19.724 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:19.724 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:37:19.724 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:19.725 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:19.725 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:37:19.725 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 5 2024-09-18T07:37:19.725 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:37:19.753 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:19.795 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:19.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:19.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:19.897 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:19.929 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:19.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:19.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:19.939 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 74 16 92966 92950 78.4803 46.75 0.0368754 0.0127353 2024-09-18T07:37:19.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:37:20.031 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:20.058 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:20.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:20.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:20.161 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:20.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:20.207 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:20.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:20.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:20.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:20.358 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:20.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:20.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:20.500 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:20.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:20.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:20.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:20.639 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:20.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:20.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:20.741 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:20.786 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:20.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:20.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:20.887 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:20.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T07:37:20.932 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:20.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:20.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:20.940 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 75 16 93710 93694 78.0534 46.5 0.0135635 0.0128061 2024-09-18T07:37:21.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:37:21.034 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:21.098 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:20 smithi137 ceph-mon[29408]: pgmap v337: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 46 MiB/s wr, 737 op/s; 14777/271968 objects misplaced (5.433%); 12 MiB/s, 188 objects/s recovering 2024-09-18T07:37:21.098 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:20 smithi137 ceph-mon[29408]: osdmap e231: 8 total, 8 up, 7 in 2024-09-18T07:37:21.098 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:20 smithi137 ceph-mon[39175]: pgmap v337: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 46 MiB/s wr, 737 op/s; 14777/271968 objects misplaced (5.433%); 12 MiB/s, 188 objects/s recovering 2024-09-18T07:37:21.098 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:20 smithi137 ceph-mon[39175]: osdmap e231: 8 total, 8 up, 7 in 2024-09-18T07:37:21.155 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:21.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:21.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:21.182 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:21.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:20 smithi203 ceph-mon[32237]: pgmap v337: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 46 MiB/s wr, 737 op/s; 14777/271968 objects misplaced (5.433%); 12 MiB/s, 188 objects/s recovering 2024-09-18T07:37:21.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:20 smithi203 ceph-mon[32237]: osdmap e231: 8 total, 8 up, 7 in 2024-09-18T07:37:21.219 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-18T07:37:21.220 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-18T07:37:21.220 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-18T07:37:21.220 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-18T07:37:21.220 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-18T07:37:21.220 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-18T07:37:21.220 INFO:teuthology.orchestra.run.smithi137.stderr:bench [] [] [] [] 2024-09-18T07:37:21.220 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-18T07:37:21.220 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-18T07:37:21.220 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-18T07:37:21.220 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator dump block 2024-09-18T07:37:21.220 INFO:teuthology.orchestra.run.smithi137.stderr:Error EINVAL: invalid command 2024-09-18T07:37:21.258 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:21.284 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:21.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:21.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:21.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:21.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:21.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:21.545 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:21.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:21.574 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:21.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:21.675 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:21.703 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:21.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:21.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:21.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:21.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:21.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:21.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:21.940 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 76 16 94411 94395 77.603 43.8125 0.0132049 0.0128802 2024-09-18T07:37:21.950 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:21.978 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:21.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:21.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:22.080 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:22.119 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:22.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:22.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:22.216 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T07:37:22.217 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T07:37:22.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-18T07:37:22.222 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:22.253 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:22.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:22.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:22.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:22.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:22.405 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:22.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:22.506 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:22.526 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:22.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:22.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:22.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:22.665 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:22.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:22.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:22.767 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:22.796 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:22.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:22.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:22.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:22.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:22.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:22.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:22.941 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 77 16 95165 95149 77.207 47.125 0.0102049 0.0129466 2024-09-18T07:37:23.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:23.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:23.082 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:23.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:23.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:22 smithi137 ceph-mon[29408]: pgmap v339: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 32 MiB/s wr, 518 op/s; 14504/273399 objects misplaced (5.305%); 9.5 MiB/s, 152 objects/s recovering 2024-09-18T07:37:23.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:22 smithi137 ceph-mon[39175]: pgmap v339: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 32 MiB/s wr, 518 op/s; 14504/273399 objects misplaced (5.305%); 9.5 MiB/s, 152 objects/s recovering 2024-09-18T07:37:23.185 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:23.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:22 smithi203 ceph-mon[32237]: pgmap v339: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 32 MiB/s wr, 518 op/s; 14504/273399 objects misplaced (5.305%); 9.5 MiB/s, 152 objects/s recovering 2024-09-18T07:37:23.225 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:23.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:23.326 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:23.353 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:23.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:23.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:23.455 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:23.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:23.477 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:23.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:23.578 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:23.606 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:23.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:23.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:23.708 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:23.734 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:23.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:23.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:23.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:23.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:23.879 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:23.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:23.940 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 78 16 95844 95828 76.7612 42.4375 0.0533404 0.0130214 2024-09-18T07:37:23.980 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:24.019 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:24.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:24.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:24.121 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:24.152 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:24.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:24.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:24.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:23 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4088341095' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:37:24.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:23 smithi137 ceph-mon[29408]: from='client.35318 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:37:24.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:23 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4088341095' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:37:24.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:23 smithi137 ceph-mon[39175]: from='client.35318 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:37:24.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:23 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4088341095' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:37:24.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:23 smithi203 ceph-mon[32237]: from='client.35318 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:37:24.254 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:24.278 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:24.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:24.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:24.380 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:24.407 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:24.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:24.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:24.421 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:37:24.509 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:24.532 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:24.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:24.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:24.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:24.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:24.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:24.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:24.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:24.809 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:24.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:24.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:24.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:24.936 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:24.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:24.940 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 79 16 96584 96568 76.3751 46.25 0.080031 0.0130802 2024-09-18T07:37:25.038 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:25.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:25.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:25.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:25.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:24 smithi137 ceph-mon[39175]: pgmap v340: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 935 op/s; 13766/283413 objects misplaced (4.857%); 15 MiB/s, 244 objects/s recovering 2024-09-18T07:37:25.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:24 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:37:25.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:24 smithi137 ceph-mon[39175]: from='client.35318 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:37:25.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:24 smithi137 ceph-mon[39175]: osdmap e232: 8 total, 8 up, 7 in 2024-09-18T07:37:25.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:24 smithi137 ceph-mon[39175]: osdmap e233: 8 total, 8 up, 7 in 2024-09-18T07:37:25.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:24 smithi137 ceph-mon[29408]: pgmap v340: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 935 op/s; 13766/283413 objects misplaced (4.857%); 15 MiB/s, 244 objects/s recovering 2024-09-18T07:37:25.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:24 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:37:25.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:24 smithi137 ceph-mon[29408]: from='client.35318 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:37:25.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:24 smithi137 ceph-mon[29408]: osdmap e232: 8 total, 8 up, 7 in 2024-09-18T07:37:25.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:24 smithi137 ceph-mon[29408]: osdmap e233: 8 total, 8 up, 7 in 2024-09-18T07:37:25.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:25.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:24 smithi203 ceph-mon[32237]: pgmap v340: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 935 op/s; 13766/283413 objects misplaced (4.857%); 15 MiB/s, 244 objects/s recovering 2024-09-18T07:37:25.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:24 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:37:25.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:24 smithi203 ceph-mon[32237]: from='client.35318 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:37:25.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:24 smithi203 ceph-mon[32237]: osdmap e232: 8 total, 8 up, 7 in 2024-09-18T07:37:25.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:24 smithi203 ceph-mon[32237]: osdmap e233: 8 total, 8 up, 7 in 2024-09-18T07:37:25.213 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:25.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:25.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:25.315 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:25.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:25.341 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:25.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:25.442 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:25.470 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:25.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:25.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:25.572 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:25.601 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:25.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:25.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:25.703 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:25.728 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:25.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:25.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:25.734 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:25.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:25.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:25.890 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:25.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:25.941 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:37:25.939267+0000 min lat: 0.00114205 max lat: 2.28882 avg lat: 0.0131505 2024-09-18T07:37:25.941 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:37:25.941 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 80 16 97337 97321 76.0087 47.0625 0.0158884 0.0131505 2024-09-18T07:37:25.990 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:26.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:26.019 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:26.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:26.120 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:26.145 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:26.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:26.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:26.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:25 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:37:26.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:25 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:37:26.247 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:26.286 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:26.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:26.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:26.388 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:26.411 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:26.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:26.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:26.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:25 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:37:26.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:25 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:37:26.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:25 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:37:26.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:25 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:37:26.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:26.551 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:26.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:26.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:26.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:26.700 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:26.702 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:26.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:26.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:26.804 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:26.834 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:26.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:26.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:26.936 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:26.940 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 81 16 98169 98153 75.7123 52 0.0164496 0.0132016 2024-09-18T07:37:26.967 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:26.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:27.069 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:27.097 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:27.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:27.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:27.199 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:27.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:26 smithi203 ceph-mon[32237]: pgmap v343: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 40 MiB/s wr, 636 op/s; 13766/283413 objects misplaced (4.857%); 11 MiB/s, 168 objects/s recovering 2024-09-18T07:37:27.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]} : dispatch 2024-09-18T07:37:27.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5, 3, 7]} : dispatch 2024-09-18T07:37:27.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]} : dispatch 2024-09-18T07:37:27.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 0, 1, 7]} : dispatch 2024-09-18T07:37:27.200 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 2]} : dispatch 2024-09-18T07:37:27.201 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5, 3, 7]} : dispatch 2024-09-18T07:37:27.201 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 0, 1, 7]} : dispatch 2024-09-18T07:37:27.201 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 2]} : dispatch 2024-09-18T07:37:27.201 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:37:27.201 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:37:27.248 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:27.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:27.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:27.275 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:37:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:37:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:37:27.275 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[29408]: pgmap v343: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 40 MiB/s wr, 636 op/s; 13766/283413 objects misplaced (4.857%); 11 MiB/s, 168 objects/s recovering 2024-09-18T07:37:27.275 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]} : dispatch 2024-09-18T07:37:27.276 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5, 3, 7]} : dispatch 2024-09-18T07:37:27.276 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]} : dispatch 2024-09-18T07:37:27.276 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 0, 1, 7]} : dispatch 2024-09-18T07:37:27.276 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 2]} : dispatch 2024-09-18T07:37:27.276 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5, 3, 7]} : dispatch 2024-09-18T07:37:27.276 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 0, 1, 7]} : dispatch 2024-09-18T07:37:27.276 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 2]} : dispatch 2024-09-18T07:37:27.276 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:37:27.276 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:37:27.276 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[39175]: pgmap v343: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 40 MiB/s wr, 636 op/s; 13766/283413 objects misplaced (4.857%); 11 MiB/s, 168 objects/s recovering 2024-09-18T07:37:27.276 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]} : dispatch 2024-09-18T07:37:27.277 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5, 3, 7]} : dispatch 2024-09-18T07:37:27.277 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]} : dispatch 2024-09-18T07:37:27.277 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 0, 1, 7]} : dispatch 2024-09-18T07:37:27.277 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 2]} : dispatch 2024-09-18T07:37:27.277 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5, 3, 7]} : dispatch 2024-09-18T07:37:27.277 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 0, 1, 7]} : dispatch 2024-09-18T07:37:27.277 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 2]} : dispatch 2024-09-18T07:37:27.277 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:37:27.277 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:37:27.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:27.393 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:27.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:27.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:37:27 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:37:27.494 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:27.519 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:27.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:27.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:27.561 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:27.606 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:37:27.606 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:37:27.606 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:37:27.606 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:37:27.606 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:37:27.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:27.665 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:27.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:27.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:27.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:27.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:27.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:27.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:27.915 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:27.941 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:27.941 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 82 16 98945 98929 75.3803 48.5 0.00777925 0.0132605 2024-09-18T07:37:27.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:27.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:28.043 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:28.088 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:28.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:28.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:27 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-09-18T07:37:28.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:27 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5, 3, 7]}]': finished 2024-09-18T07:37:28.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:27 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 0, 1, 7]}]': finished 2024-09-18T07:37:28.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:27 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 2]}]': finished 2024-09-18T07:37:28.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:27 smithi137 ceph-mon[29408]: osdmap e234: 8 total, 8 up, 7 in 2024-09-18T07:37:28.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:27 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-09-18T07:37:28.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:27 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5, 3, 7]}]': finished 2024-09-18T07:37:28.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:27 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 0, 1, 7]}]': finished 2024-09-18T07:37:28.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:27 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 2]}]': finished 2024-09-18T07:37:28.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:27 smithi137 ceph-mon[39175]: osdmap e234: 8 total, 8 up, 7 in 2024-09-18T07:37:28.190 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:28.218 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:28.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:28.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:28.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:28.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:28.360 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:28.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:28.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:27 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-09-18T07:37:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:27 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5, 3, 7]}]': finished 2024-09-18T07:37:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:27 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [3, 0, 1, 7]}]': finished 2024-09-18T07:37:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:27 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [3, 2]}]': finished 2024-09-18T07:37:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:27 smithi203 ceph-mon[32237]: osdmap e234: 8 total, 8 up, 7 in 2024-09-18T07:37:28.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:28.506 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-18T07:37:28.507 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-18T07:37:28.507 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-18T07:37:28.507 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-18T07:37:28.507 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-18T07:37:28.507 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-18T07:37:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:bench [] [] [] [] 2024-09-18T07:37:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-18T07:37:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-18T07:37:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-18T07:37:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator dump block 2024-09-18T07:37:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:Error EINVAL: invalid command 2024-09-18T07:37:28.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:28.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:28.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:28.611 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:28.636 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:28.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:28.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:28.684 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:37:28.713 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:37:28.714 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:37:28.738 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:28.751 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:37:28.752 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:37:28.764 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:28.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:28.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:28.792 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:37:28.793 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:37:28.831 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:37:28.832 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:37:28.863 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:37:28.863 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:37:28.866 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:28.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:28.891 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:28.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:28.897 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:37:28.897 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:37:28.932 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:37:28.933 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:37:28.941 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 83 16 99646 99630 75 43.8125 0.0173194 0.0133247 2024-09-18T07:37:28.963 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:37:28.964 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:37:28.992 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:29.019 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:29.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:29.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:29.121 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:29.143 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:28 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:29.143 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:28 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:37:29.143 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:28 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:37:29.143 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:28 smithi137 ceph-mon[29408]: pgmap v345: 148 pgs: 1 peering, 5 active+recovering+undersized+remapped, 142 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.18k op/s; 12721/294687 objects misplaced (4.317%); 19 MiB/s, 297 objects/s recovering 2024-09-18T07:37:29.144 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:28 smithi137 ceph-mon[29408]: osdmap e235: 8 total, 8 up, 7 in 2024-09-18T07:37:29.144 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:28 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4160722731' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:29.144 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:28 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:29.144 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:28 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:37:29.144 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:28 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:37:29.144 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:28 smithi137 ceph-mon[39175]: pgmap v345: 148 pgs: 1 peering, 5 active+recovering+undersized+remapped, 142 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.18k op/s; 12721/294687 objects misplaced (4.317%); 19 MiB/s, 297 objects/s recovering 2024-09-18T07:37:29.144 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:28 smithi137 ceph-mon[39175]: osdmap e235: 8 total, 8 up, 7 in 2024-09-18T07:37:29.144 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:28 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4160722731' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:29.148 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:29.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:29.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:29.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:29.294 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:29.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:29.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:29.396 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:29.424 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:29.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:29.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:28 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:28 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:37:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:28 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:37:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:28 smithi203 ceph-mon[32237]: pgmap v345: 148 pgs: 1 peering, 5 active+recovering+undersized+remapped, 142 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.18k op/s; 12721/294687 objects misplaced (4.317%); 19 MiB/s, 297 objects/s recovering 2024-09-18T07:37:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:28 smithi203 ceph-mon[32237]: osdmap e235: 8 total, 8 up, 7 in 2024-09-18T07:37:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:28 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4160722731' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:29.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:29.568 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:29.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:29.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:29.670 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:29.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:29.716 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:29.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:29.817 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:29.844 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:29.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:29.941 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 84 16 100453 100437 74.7076 50.4375 0.00949439 0.0133802 2024-09-18T07:37:29.950 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:29.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:29.977 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:29.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:30.078 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:30.100 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:30.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:30.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:30.202 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:30.229 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:30.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:30.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:30.331 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:30.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:30.361 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:30.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:30.462 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:30.490 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:30.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:30.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:30.592 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:30.621 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:30.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:30.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:30.723 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:30.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:30.771 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:30.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:30.872 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:30.900 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:30.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:30.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:30.941 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 85 16 101264 101248 74.425 50.6875 0.019551 0.0134306 2024-09-18T07:37:31.002 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:31.032 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:31.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:31.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:31.134 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:31.174 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:31.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:31.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:31.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:31.304 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:31.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:31.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:31.406 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:31.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:30 smithi137 ceph-mon[29408]: pgmap v347: 148 pgs: 1 peering, 5 active+recovering+undersized+remapped, 142 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 910 B/s rd, 42 MiB/s wr, 668 op/s; 12721/294687 objects misplaced (4.317%); 12 MiB/s, 185 objects/s recovering 2024-09-18T07:37:31.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:30 smithi137 ceph-mon[39175]: pgmap v347: 148 pgs: 1 peering, 5 active+recovering+undersized+remapped, 142 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 910 B/s rd, 42 MiB/s wr, 668 op/s; 12721/294687 objects misplaced (4.317%); 12 MiB/s, 185 objects/s recovering 2024-09-18T07:37:31.427 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:31.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:31.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:31.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:30 smithi203 ceph-mon[32237]: pgmap v347: 148 pgs: 1 peering, 5 active+recovering+undersized+remapped, 142 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 910 B/s rd, 42 MiB/s wr, 668 op/s; 12721/294687 objects misplaced (4.317%); 12 MiB/s, 185 objects/s recovering 2024-09-18T07:37:31.529 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:31.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:31.556 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:31.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:31.657 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:31.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:31.689 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:31.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:31.791 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:31.815 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:31.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:31.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:31.917 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:31.941 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 86 16 101970 101954 74.0727 44.125 0.0453324 0.0134943 2024-09-18T07:37:31.945 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:31.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:31.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:32.047 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:32.074 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:32.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:32.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:32.176 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:32.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:32.200 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:32.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:32.301 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:32.330 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:32.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:32.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:32.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:32.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:32.467 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:32.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:32.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:32.611 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:32.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:32.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:32.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:32.755 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:32.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:32.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:32.857 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:32.878 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:32.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:32.941 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 87 16 102587 102571 73.6645 38.5625 0.00402159 0.0135521 2024-09-18T07:37:32.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:33.016 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:33.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:33.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:33.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:33.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:33.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:33.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:33.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:33.303 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:33.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:33.360 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:32 smithi137 ceph-mon[29408]: pgmap v348: 148 pgs: 1 peering, 5 active+recovering+undersized+remapped, 142 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.5 KiB/s rd, 40 MiB/s wr, 639 op/s; 12424/296121 objects misplaced (4.196%); 13 MiB/s, 202 objects/s recovering 2024-09-18T07:37:33.360 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:32 smithi137 ceph-mon[39175]: pgmap v348: 148 pgs: 1 peering, 5 active+recovering+undersized+remapped, 142 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.5 KiB/s rd, 40 MiB/s wr, 639 op/s; 12424/296121 objects misplaced (4.196%); 13 MiB/s, 202 objects/s recovering 2024-09-18T07:37:33.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:33.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:33.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:33.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:33.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:32 smithi203 ceph-mon[32237]: pgmap v348: 148 pgs: 1 peering, 5 active+recovering+undersized+remapped, 142 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.5 KiB/s rd, 40 MiB/s wr, 639 op/s; 12424/296121 objects misplaced (4.196%); 13 MiB/s, 202 objects/s recovering 2024-09-18T07:37:33.545 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:33.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:33.597 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:33.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:33.698 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:33.724 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:33.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:33.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:33.826 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:33.843 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T07:37:33.843 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T07:37:33.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:33.886 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:33.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:33.943 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 88 15 103404 103389 73.4077 51.125 0.00721485 0.0136158 2024-09-18T07:37:33.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:34.023 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:34.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:34.125 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:34.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:34.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:34.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:34.295 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:34.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:34.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:34.323 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T07:37:34.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:34.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:34.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:34.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:34.523 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:34.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:34.582 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:34.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:34.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:34.684 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:34.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:34.715 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:34.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:34.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:34.863 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:34.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:34.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:34.942 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 89 16 104128 104112 73.0907 45.1875 0.0212462 0.0136747 2024-09-18T07:37:34.965 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:34.999 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:35.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:35.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:35.101 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:35.142 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:35.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:35.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:35.197 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:35.244 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:35.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:35.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:35.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:35.337 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:34 smithi137 ceph-mon[29408]: pgmap v349: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 948 op/s; 11629/306165 objects misplaced (3.798%); 17 MiB/s, 269 objects/s recovering 2024-09-18T07:37:35.338 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:34 smithi137 ceph-mon[39175]: pgmap v349: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 948 op/s; 11629/306165 objects misplaced (3.798%); 17 MiB/s, 269 objects/s recovering 2024-09-18T07:37:35.402 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:35.426 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:35.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:35.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:35.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:34 smithi203 ceph-mon[32237]: pgmap v349: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 948 op/s; 11629/306165 objects misplaced (3.798%); 17 MiB/s, 269 objects/s recovering 2024-09-18T07:37:35.528 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:35.552 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:35.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:35.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:35.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:35.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:35.688 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:35.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:35.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:35.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:35.839 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:35.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:35.847 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:37:35.847 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:37:35.941 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:35.943 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Total time run: 90.01 2024-09-18T07:37:35.943 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Total writes made: 104807 2024-09-18T07:37:35.943 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Write size: 65536 2024-09-18T07:37:35.943 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Object size: 65536 2024-09-18T07:37:35.943 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Bandwidth (MB/sec): 72.7746 2024-09-18T07:37:35.943 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Stddev Bandwidth: 43.8181 2024-09-18T07:37:35.943 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Max bandwidth (MB/sec): 208.062 2024-09-18T07:37:35.943 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Min bandwidth (MB/sec): 0 2024-09-18T07:37:35.943 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Average IOPS: 1164 2024-09-18T07:37:35.943 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Stddev IOPS: 701.097 2024-09-18T07:37:35.943 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Max IOPS: 3329 2024-09-18T07:37:35.944 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Min IOPS: 0 2024-09-18T07:37:35.944 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Average Latency(s): 0.0137365 2024-09-18T07:37:35.944 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Stddev Latency(s): 0.0323478 2024-09-18T07:37:35.944 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Max latency(s): 2.28882 2024-09-18T07:37:35.944 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Min latency(s): 0.00114205 2024-09-18T07:37:35.947 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Cleaning up (deleting benchmark objects) 2024-09-18T07:37:35.966 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:35.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:35.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:36.000 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:37:36.068 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:36.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:36.108 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:36.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:36.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:36.235 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:36.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:36.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:36.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:36.378 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:36.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:36.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:36.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:35 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2897530970' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:37:36.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:35 smithi137 ceph-mon[29408]: from='client.25384 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:37:36.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:35 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2897530970' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:37:36.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:35 smithi137 ceph-mon[39175]: from='client.25384 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:37:36.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:35 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2897530970' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:37:36.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:35 smithi203 ceph-mon[32237]: from='client.25384 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:37:36.480 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:36.504 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:37:36.525 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:36.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:36.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:36.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:36.671 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:36.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:36.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:36.773 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:36.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:36.795 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:36.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:36.897 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:36.924 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:36.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:36.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:37.026 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:37.051 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:37.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:37.153 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:37.184 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:37.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:37.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:37.286 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:37.312 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:37.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:37.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:37.414 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:37.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:37:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:37:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:37:37.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:36 smithi137 ceph-mon[29408]: pgmap v350: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 476 B/s rd, 35 MiB/s wr, 553 op/s; 11629/306165 objects misplaced (3.798%); 8.7 MiB/s, 141 objects/s recovering 2024-09-18T07:37:37.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:36 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:37:37.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:36 smithi137 ceph-mon[29408]: from='client.25384 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:37:37.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:36 smithi137 ceph-mon[29408]: osdmap e236: 8 total, 8 up, 7 in 2024-09-18T07:37:37.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:36 smithi137 ceph-mon[39175]: pgmap v350: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 476 B/s rd, 35 MiB/s wr, 553 op/s; 11629/306165 objects misplaced (3.798%); 8.7 MiB/s, 141 objects/s recovering 2024-09-18T07:37:37.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:36 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:37:37.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:36 smithi137 ceph-mon[39175]: from='client.25384 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:37:37.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:36 smithi137 ceph-mon[39175]: osdmap e236: 8 total, 8 up, 7 in 2024-09-18T07:37:37.440 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:37.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:37.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:36 smithi203 ceph-mon[32237]: pgmap v350: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 476 B/s rd, 35 MiB/s wr, 553 op/s; 11629/306165 objects misplaced (3.798%); 8.7 MiB/s, 141 objects/s recovering 2024-09-18T07:37:37.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:36 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:37:37.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:36 smithi203 ceph-mon[32237]: from='client.25384 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:37:37.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:36 smithi203 ceph-mon[32237]: osdmap e236: 8 total, 8 up, 7 in 2024-09-18T07:37:37.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:37:37 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:37:37.542 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:37.569 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:37.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:37.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:37.671 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:37.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:37.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:37.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:37.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:37.846 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:37.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:37.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:37.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:37.987 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:37.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:37.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:38.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:38.130 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:38.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:38.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:38.232 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:38.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:38.272 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:38.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:38.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:38.399 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:38.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:38.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:38.501 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:38.526 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:38.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:38.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:38.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:38.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:38.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:38.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:38.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:38.809 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:38.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:38.844 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:37:38.845 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:37:38.845 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-09-18T07:37:38.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd in 6 2024-09-18T07:37:38.911 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:38.935 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:38.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:39.037 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:39.062 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:39.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:39.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:39.163 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:39.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:38 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:39.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:39 smithi137 ceph-mon[29408]: pgmap v352: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 44 MiB/s wr, 779 op/s; 10477/313008 objects misplaced (3.347%); 15 MiB/s, 236 objects/s recovering 2024-09-18T07:37:39.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:39 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:39.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:39 smithi137 ceph-mon[39175]: pgmap v352: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 44 MiB/s wr, 779 op/s; 10477/313008 objects misplaced (3.347%); 15 MiB/s, 236 objects/s recovering 2024-09-18T07:37:39.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:39.203 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:39.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:39.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:39.343 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:39.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:39.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:39.445 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:39.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:38 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:39.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:38 smithi203 ceph-mon[32237]: pgmap v352: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 44 MiB/s wr, 779 op/s; 10477/313008 objects misplaced (3.347%); 15 MiB/s, 236 objects/s recovering 2024-09-18T07:37:39.493 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:39.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:39.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:39.633 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:39.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:39.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:39.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:39.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T07:37:39.825 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:39.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:39.826 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:39.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:39.927 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:39.955 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:39.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:39.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:40.057 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:40.083 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:40.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:40.185 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:40.245 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:40.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:40.335 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:40 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:37:40.335 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:40 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:37:40.335 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:40 smithi137 ceph-mon[29408]: osdmap e237: 8 total, 8 up, 7 in 2024-09-18T07:37:40.335 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:40 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:37:40.336 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:40 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:37:40.336 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:40 smithi137 ceph-mon[39175]: osdmap e237: 8 total, 8 up, 7 in 2024-09-18T07:37:40.346 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:40.370 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:40.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:40.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:40.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:40 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:37:40.491 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:40 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:37:40.491 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:40 smithi203 ceph-mon[32237]: osdmap e237: 8 total, 8 up, 7 in 2024-09-18T07:37:40.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:37:40.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:40.493 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:40.536 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:37:40.536 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:37:40.575 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:37:40.576 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:37:40.582 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:40.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:40.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:40.616 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:37:40.616 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:37:40.664 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:37:40.665 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:37:40.684 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:40.700 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:37:40.700 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:37:40.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:40.711 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:40.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:40.736 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:37:40.737 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:37:40.767 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:37:40.767 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:37:40.801 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:37:40.801 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:37:40.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:40.836 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:40.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:40.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:40.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:40.984 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:40.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:40.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:41.086 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:41.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:41.124 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:41.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:41.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:41.270 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:41.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:41.300 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:41 smithi137 ceph-mon[29408]: pgmap v354: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 49 MiB/s wr, 871 op/s; 10477/313008 objects misplaced (3.347%); 15 MiB/s, 245 objects/s recovering 2024-09-18T07:37:41.300 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:41 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/922224423' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:41.301 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:41 smithi137 ceph-mon[39175]: pgmap v354: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 49 MiB/s wr, 871 op/s; 10477/313008 objects misplaced (3.347%); 15 MiB/s, 245 objects/s recovering 2024-09-18T07:37:41.301 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:41 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/922224423' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:41.371 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:41.397 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:41.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:41.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:41.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:41 smithi203 ceph-mon[32237]: pgmap v354: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 49 MiB/s wr, 871 op/s; 10477/313008 objects misplaced (3.347%); 15 MiB/s, 245 objects/s recovering 2024-09-18T07:37:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:41 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/922224423' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:41.499 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:41.525 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:41.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:41.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:41.627 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:41.663 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:41.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:41.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:41.744 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:41.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:41.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:41.826 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:41.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:41.927 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:41.956 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:41.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:42.030 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:41 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a[29404]: 2024-09-18T07:37:41.730+0000 7fe5973a1640 -1 mon.a@0(leader).osd e237 definitely_dead 0 2024-09-18T07:37:42.038 INFO:teuthology.orchestra.run.smithi137.stderr:marked in osd.6. 2024-09-18T07:37:42.057 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:42.067 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:42.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:42.092 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:42.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:42.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:42.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:42.216 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:42.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:42.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:42.326 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:42 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:37:42.326 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:42 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3955912886' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-18T07:37:42.326 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:42 smithi137 ceph-mon[29408]: from='client.35333 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-18T07:37:42.326 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:42 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:37:42.327 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:42 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3955912886' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-18T07:37:42.327 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:42 smithi137 ceph-mon[39175]: from='client.35333 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-18T07:37:42.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:42.340 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:42.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:42.426 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:37:42.427 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:37:42.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.427 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109567 8.7 8.b3afe247 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:37:42.427 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:37:42.427 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.320352362, 2024-09-18T07:37:42.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.320370404, 2024-09-18T07:37:42.427 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:37:42.427 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:37:42.427 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for peered", 2024-09-18T07:37:42.427 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:37:42.428 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:37:42.428 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:37:42.428 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109567 2024-09-18T07:37:42.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.428 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:37:42.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:37:42.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:37:42.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:42.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:37:42.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:37:42.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:42.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:37:42.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103384+0000", 2024-09-18T07:37:42.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.792e-06 2024-09-18T07:37:42.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:37:42.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103390+0000", 2024-09-18T07:37:42.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7139999999999998e-06 2024-09-18T07:37:42.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:37:42.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103401+0000", 2024-09-18T07:37:42.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1559000000000001e-05 2024-09-18T07:37:42.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:37:42.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103408+0000", 2024-09-18T07:37:42.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2110000000000001e-06 2024-09-18T07:37:42.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:37:42.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103464+0000", 2024-09-18T07:37:42.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6032999999999997e-05 2024-09-18T07:37:42.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:37:42.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103475+0000", 2024-09-18T07:37:42.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0380999999999999e-05 2024-09-18T07:37:42.431 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:42.432 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:37:42.432 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:42.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.432 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109537 8.7 8.6be120a7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:37:42.432 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:37:42.432 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.36732830100000002, 2024-09-18T07:37:42.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.36738442999999998, 2024-09-18T07:37:42.432 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:37:42.432 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:37:42.432 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for peered", 2024-09-18T07:37:42.432 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:37:42.433 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:37:42.433 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:37:42.433 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109537 2024-09-18T07:37:42.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.433 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:37:42.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:37:42.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:37:42.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:42.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:37:42.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:37:42.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:42.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:37:42.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056408+0000", 2024-09-18T07:37:42.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.562e-06 2024-09-18T07:37:42.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:37:42.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056415+0000", 2024-09-18T07:37:42.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9280000000000004e-06 2024-09-18T07:37:42.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:37:42.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056425+0000", 2024-09-18T07:37:42.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0220999999999999e-05 2024-09-18T07:37:42.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:37:42.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056434+0000", 2024-09-18T07:37:42.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7499999999999992e-06 2024-09-18T07:37:42.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:37:42.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.061190+0000", 2024-09-18T07:37:42.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047560620000000001 2024-09-18T07:37:42.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:37:42.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.061238+0000", 2024-09-18T07:37:42.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8730000000000003e-05 2024-09-18T07:37:42.436 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:42.436 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:37:42.436 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:42.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.436 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109540 8.7 8.e22a2e7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:37:42.437 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:37:42.437 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.35785631099999998, 2024-09-18T07:37:42.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.35794343000000001, 2024-09-18T07:37:42.437 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:37:42.437 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:37:42.437 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for peered", 2024-09-18T07:37:42.437 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:37:42.437 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:37:42.437 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:37:42.437 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109540 2024-09-18T07:37:42.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.437 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:37:42.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:37:42.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:37:42.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:42.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:37:42.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:37:42.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:42.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:37:42.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065880+0000", 2024-09-18T07:37:42.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8279999999999999e-06 2024-09-18T07:37:42.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:37:42.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065886+0000", 2024-09-18T07:37:42.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.117e-06 2024-09-18T07:37:42.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:37:42.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065899+0000", 2024-09-18T07:37:42.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2809999999999999e-05 2024-09-18T07:37:42.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:37:42.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065904+0000", 2024-09-18T07:37:42.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0350000000000001e-06 2024-09-18T07:37:42.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:37:42.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.066028+0000", 2024-09-18T07:37:42.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012424199999999999 2024-09-18T07:37:42.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:37:42.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.066039+0000", 2024-09-18T07:37:42.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0441999999999999e-05 2024-09-18T07:37:42.440 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:42.440 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:37:42.441 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:42.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.441 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109545 8.7 8.2f382d37 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:37:42.441 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:37:42.441 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.34945152499999999, 2024-09-18T07:37:42.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.34956964600000001, 2024-09-18T07:37:42.441 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:37:42.441 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:37:42.441 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for peered", 2024-09-18T07:37:42.441 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:37:42.441 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:37:42.441 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:37:42.442 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109545 2024-09-18T07:37:42.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.442 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:37:42.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:37:42.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:37:42.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:42.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:37:42.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:37:42.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:37:42.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:37:42.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074285+0000", 2024-09-18T07:37:42.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1169999999999998e-06 2024-09-18T07:37:42.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:37:42.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074292+0000", 2024-09-18T07:37:42.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.153e-06 2024-09-18T07:37:42.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:37:42.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074305+0000", 2024-09-18T07:37:42.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2363e-05 2024-09-18T07:37:42.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:37:42.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074312+0000", 2024-09-18T07:37:42.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3409999999999997e-06 2024-09-18T07:37:42.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:37:42.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074508+0000", 2024-09-18T07:37:42.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000196443 2024-09-18T07:37:42.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:37:42.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:37:42.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:37:42.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074519+0000", 2024-09-18T07:37:42.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0159999999999999e-05 2024-09-18T07:37:42.445 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:42.445 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:37:42.445 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:42.445 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:37:42.445 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:37:42.445 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 4 2024-09-18T07:37:42.445 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:37:42.446 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:42.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:42 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:37:42.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:42 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3955912886' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-18T07:37:42.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:42 smithi203 ceph-mon[32237]: from='client.35333 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-18T07:37:42.468 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:42.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:42.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:42.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:42.591 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:42.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:42.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:42.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:42.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:42.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:42.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:42.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:42.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:42.838 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:42.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:42.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:42.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:42.960 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:42.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:43.061 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:43.100 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:43.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:43.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:43.202 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:43.226 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:43.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:43.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:43.303 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:43 smithi137 ceph-mon[29408]: pgmap v355: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 23 MiB/s wr, 517 op/s; 10143/311484 objects misplaced (3.256%); 12 MiB/s, 185 objects/s recovering 2024-09-18T07:37:43.303 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:43 smithi137 ceph-mon[29408]: from='client.35333 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-18T07:37:43.303 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:43 smithi137 ceph-mon[29408]: osdmap e238: 8 total, 8 up, 8 in 2024-09-18T07:37:43.303 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:43 smithi137 ceph-mon[29408]: 5.15 scrub starts 2024-09-18T07:37:43.303 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:43 smithi137 ceph-mon[29408]: 5.15 scrub ok 2024-09-18T07:37:43.303 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:43 smithi137 ceph-mon[39175]: pgmap v355: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 23 MiB/s wr, 517 op/s; 10143/311484 objects misplaced (3.256%); 12 MiB/s, 185 objects/s recovering 2024-09-18T07:37:43.303 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:43 smithi137 ceph-mon[39175]: from='client.35333 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-18T07:37:43.304 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:43 smithi137 ceph-mon[39175]: osdmap e238: 8 total, 8 up, 8 in 2024-09-18T07:37:43.304 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:43 smithi137 ceph-mon[39175]: 5.15 scrub starts 2024-09-18T07:37:43.304 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:43 smithi137 ceph-mon[39175]: 5.15 scrub ok 2024-09-18T07:37:43.305 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T07:37:43.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:43.364 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:43.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:43.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:43.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:43 smithi203 ceph-mon[32237]: pgmap v355: 148 pgs: 5 active+recovering+undersized+remapped, 143 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 23 MiB/s wr, 517 op/s; 10143/311484 objects misplaced (3.256%); 12 MiB/s, 185 objects/s recovering 2024-09-18T07:37:43.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:43 smithi203 ceph-mon[32237]: from='client.35333 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-18T07:37:43.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:43 smithi203 ceph-mon[32237]: osdmap e238: 8 total, 8 up, 8 in 2024-09-18T07:37:43.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:43 smithi203 ceph-mon[32237]: 5.15 scrub starts 2024-09-18T07:37:43.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:43 smithi203 ceph-mon[32237]: 5.15 scrub ok 2024-09-18T07:37:43.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:43.527 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:43.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:43.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:43.629 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:43.667 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:43.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:43.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:43.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:43.800 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:43.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:43.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:43.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:37:43.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:43.949 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:43.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:43.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:44.029 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-09-18T07:37:44.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:44.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:44.095 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:44.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:44.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:44 smithi137 ceph-mon[29408]: 6.b scrub starts 2024-09-18T07:37:44.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:44 smithi137 ceph-mon[29408]: 6.b scrub ok 2024-09-18T07:37:44.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:44 smithi137 ceph-mon[29408]: osdmap e239: 8 total, 8 up, 8 in 2024-09-18T07:37:44.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:44 smithi137 ceph-mon[39175]: 6.b scrub starts 2024-09-18T07:37:44.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:44 smithi137 ceph-mon[39175]: 6.b scrub ok 2024-09-18T07:37:44.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:44 smithi137 ceph-mon[39175]: osdmap e239: 8 total, 8 up, 8 in 2024-09-18T07:37:44.196 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:44.228 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:44.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:44.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:44.330 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:44.365 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:44.378 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:44.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:44.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:44.427 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:44.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:44 smithi203 ceph-mon[32237]: 6.b scrub starts 2024-09-18T07:37:44.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:44 smithi203 ceph-mon[32237]: 6.b scrub ok 2024-09-18T07:37:44.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:44 smithi203 ceph-mon[32237]: osdmap e239: 8 total, 8 up, 8 in 2024-09-18T07:37:44.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:44.533 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:44.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:44.635 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:44.664 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:44.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:44.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:44.766 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:44.801 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:44.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:44.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:44.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:44.938 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:44.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:45.032 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:37:45.032 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:37:45.039 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:45.056 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:37:45.056 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:45 smithi137 ceph-mon[29408]: pgmap v358: 148 pgs: 1 remapped+peering, 2 peering, 4 active+recovering+undersized+remapped, 141 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 641 op/s; 6388/301470 objects misplaced (2.119%); 12 MiB/s, 197 objects/s recovering 2024-09-18T07:37:45.057 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:45 smithi137 ceph-mon[29408]: 4.1d scrub starts 2024-09-18T07:37:45.057 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:45 smithi137 ceph-mon[29408]: 4.1d scrub ok 2024-09-18T07:37:45.057 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:45 smithi137 ceph-mon[29408]: Health check failed: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-09-18T07:37:45.057 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:45 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3542360458' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:37:45.057 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:45 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3549854835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:45.057 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:45 smithi137 ceph-mon[39175]: pgmap v358: 148 pgs: 1 remapped+peering, 2 peering, 4 active+recovering+undersized+remapped, 141 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 641 op/s; 6388/301470 objects misplaced (2.119%); 12 MiB/s, 197 objects/s recovering 2024-09-18T07:37:45.057 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:45 smithi137 ceph-mon[39175]: 4.1d scrub starts 2024-09-18T07:37:45.057 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:45 smithi137 ceph-mon[39175]: 4.1d scrub ok 2024-09-18T07:37:45.057 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:45 smithi137 ceph-mon[39175]: Health check failed: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-09-18T07:37:45.058 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:45 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3542360458' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:37:45.058 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:45 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3549854835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:45.072 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:37:45.072 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:37:45.076 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:45.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:45.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:45.117 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:37:45.118 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:37:45.152 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:37:45.152 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:37:45.179 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:45.185 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:37:45.185 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:37:45.218 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:37:45.218 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:37:45.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:45.221 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:45.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:45.251 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:37:45.251 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:37:45.278 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:37:45.279 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:37:45.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:45.351 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:45.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:45.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:45.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:45 smithi203 ceph-mon[32237]: pgmap v358: 148 pgs: 1 remapped+peering, 2 peering, 4 active+recovering+undersized+remapped, 141 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 641 op/s; 6388/301470 objects misplaced (2.119%); 12 MiB/s, 197 objects/s recovering 2024-09-18T07:37:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:45 smithi203 ceph-mon[32237]: 4.1d scrub starts 2024-09-18T07:37:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:45 smithi203 ceph-mon[32237]: 4.1d scrub ok 2024-09-18T07:37:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:45 smithi203 ceph-mon[32237]: Health check failed: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-09-18T07:37:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:45 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3542360458' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:37:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:45 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3549854835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:45.491 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:45.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:45.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:45.593 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:45.633 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:45.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:45.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:45.735 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:45.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:45.777 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:45.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:45.811 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:37:45.811 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:37:45.878 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:45.908 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:45.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:45.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:46.010 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:46.047 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:46.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:46.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:46.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:46.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:46.188 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:46.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:46.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:46.332 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:46.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:46.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:46.335 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:46 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:37:46.335 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:46 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3542360458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:37:46.336 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:46 smithi137 ceph-mon[29408]: osdmap e240: 8 total, 8 up, 8 in 2024-09-18T07:37:46.336 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:46 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:37:46.336 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:46 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3542360458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:37:46.336 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:46 smithi137 ceph-mon[39175]: osdmap e240: 8 total, 8 up, 8 in 2024-09-18T07:37:46.435 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:46.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:46 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:37:46.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:46 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3542360458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:37:46.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:46 smithi203 ceph-mon[32237]: osdmap e240: 8 total, 8 up, 8 in 2024-09-18T07:37:46.477 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:46.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:46.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:46.579 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:46.621 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:46.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:46.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:46.723 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:46.766 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:46.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:46.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:46.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:46.898 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:46.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:46.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:47.000 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:47.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:47.033 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:47.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:47.071 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T07:37:47.134 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:47.182 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:47.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:47.284 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:47.326 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:47.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:47.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:47.366 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:37:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:37:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:37:47.367 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:47 smithi137 ceph-mon[39175]: pgmap v360: 148 pgs: 1 remapped+peering, 2 peering, 4 active+recovering+undersized+remapped, 141 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 641 op/s; 6388/301470 objects misplaced (2.119%); 12 MiB/s, 197 objects/s recovering 2024-09-18T07:37:47.367 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:47 smithi137 ceph-mon[29408]: pgmap v360: 148 pgs: 1 remapped+peering, 2 peering, 4 active+recovering+undersized+remapped, 141 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 641 op/s; 6388/301470 objects misplaced (2.119%); 12 MiB/s, 197 objects/s recovering 2024-09-18T07:37:47.427 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:47.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:47 smithi203 ceph-mon[32237]: pgmap v360: 148 pgs: 1 remapped+peering, 2 peering, 4 active+recovering+undersized+remapped, 141 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 641 op/s; 6388/301470 objects misplaced (2.119%); 12 MiB/s, 197 objects/s recovering 2024-09-18T07:37:47.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:37:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:37:47.462 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:47.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:47.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:47.564 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:47.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:47.601 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:47.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:47.702 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:47.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:47.744 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:47.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:47.820 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:47.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:47.881 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:47.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:47.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:47.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:48.020 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:48.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:48.122 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:48.155 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:48.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:48.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:48.257 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:48.289 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:48.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:48.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:48.360 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:48 smithi137 ceph-mon[29408]: 3.4 scrub starts 2024-09-18T07:37:48.360 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:48 smithi137 ceph-mon[29408]: 3.4 scrub ok 2024-09-18T07:37:48.361 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:48 smithi137 ceph-mon[39175]: 3.4 scrub starts 2024-09-18T07:37:48.361 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:48 smithi137 ceph-mon[39175]: 3.4 scrub ok 2024-09-18T07:37:48.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:37:48.391 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:48.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:48.419 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:48.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:48.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:48 smithi203 ceph-mon[32237]: 3.4 scrub starts 2024-09-18T07:37:48.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:48 smithi203 ceph-mon[32237]: 3.4 scrub ok 2024-09-18T07:37:48.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:48.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:48.567 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:48.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:48.668 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:48.711 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:48.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:48.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:48.813 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:48.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:48.848 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:48.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:48.949 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:48.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:48.984 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:48.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:49.030 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:37:49.030 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:37:49.030 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-18T07:37:49.031 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T07:37:49.031 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T07:37:49.031 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-18T07:37:49.085 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:49.134 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:49.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:49.235 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:49.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:49.258 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:49.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:49.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:49.366 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:49 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:49.366 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:49 smithi137 ceph-mon[39175]: pgmap v361: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 5481/289692 objects degraded (1.892%); 6923/289692 objects misplaced (2.390%); 14 MiB/s, 233 objects/s recovering 2024-09-18T07:37:49.366 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:49 smithi137 ceph-mon[39175]: 5.6 scrub starts 2024-09-18T07:37:49.367 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:49 smithi137 ceph-mon[39175]: 5.6 scrub ok 2024-09-18T07:37:49.367 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:37:49.367 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:37:49.367 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:49 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:49.367 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:49 smithi137 ceph-mon[29408]: pgmap v361: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 5481/289692 objects degraded (1.892%); 6923/289692 objects misplaced (2.390%); 14 MiB/s, 233 objects/s recovering 2024-09-18T07:37:49.367 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:49 smithi137 ceph-mon[29408]: 5.6 scrub starts 2024-09-18T07:37:49.367 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:49 smithi137 ceph-mon[29408]: 5.6 scrub ok 2024-09-18T07:37:49.386 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:49.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:49.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:49.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:49 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:49 smithi203 ceph-mon[32237]: pgmap v361: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 5481/289692 objects degraded (1.892%); 6923/289692 objects misplaced (2.390%); 14 MiB/s, 233 objects/s recovering 2024-09-18T07:37:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:49 smithi203 ceph-mon[32237]: 5.6 scrub starts 2024-09-18T07:37:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:49 smithi203 ceph-mon[32237]: 5.6 scrub ok 2024-09-18T07:37:49.488 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:49.526 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:49.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:49.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:49.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:49.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:49.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:49.743 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:37:49.768 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:49.798 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:49.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:49.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:49.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:49.942 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:49.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:50.043 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:50.086 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:50.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:50.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:50.215 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:50.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:50.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:50.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:50.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:50.358 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:50.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:50.372 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:50 smithi137 ceph-mon[29408]: 6.14 scrub starts 2024-09-18T07:37:50.372 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:50 smithi137 ceph-mon[29408]: 6.14 scrub ok 2024-09-18T07:37:50.372 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:50 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 5481/289692 objects degraded (1.892%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:37:50.372 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:50 smithi137 ceph-mon[29408]: osdmap e241: 8 total, 8 up, 8 in 2024-09-18T07:37:50.372 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:50 smithi137 ceph-mon[39175]: 6.14 scrub starts 2024-09-18T07:37:50.372 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:50 smithi137 ceph-mon[39175]: 6.14 scrub ok 2024-09-18T07:37:50.372 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:50 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 5481/289692 objects degraded (1.892%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:37:50.373 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:50 smithi137 ceph-mon[39175]: osdmap e241: 8 total, 8 up, 8 in 2024-09-18T07:37:50.426 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T07:37:50.456 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:50 smithi203 ceph-mon[32237]: 6.14 scrub starts 2024-09-18T07:37:50.456 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:50 smithi203 ceph-mon[32237]: 6.14 scrub ok 2024-09-18T07:37:50.456 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:50 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 5481/289692 objects degraded (1.892%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:37:50.457 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:50 smithi203 ceph-mon[32237]: osdmap e241: 8 total, 8 up, 8 in 2024-09-18T07:37:50.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:50.498 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:50.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:50.532 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:50.576 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:50.600 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:50.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:50.639 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:50.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:50.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:50.741 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:50.801 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:50.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:50.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:50.841 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:37:50.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:50.935 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:50.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:50.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:51.037 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:51.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:51.077 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:51.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:51.178 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:51.206 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:51.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:51.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:51.220 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:51 smithi137 ceph-mon[29408]: pgmap v362: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 681 B/s rd, 0 B/s wr, 644 op/s; 5481/289692 objects degraded (1.892%); 6923/289692 objects misplaced (2.390%); 5.1 MiB/s, 86 objects/s recovering 2024-09-18T07:37:51.220 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:51 smithi137 ceph-mon[29408]: 2.1 scrub starts 2024-09-18T07:37:51.220 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:51 smithi137 ceph-mon[29408]: 2.1 scrub ok 2024-09-18T07:37:51.220 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:51 smithi137 ceph-mon[39175]: pgmap v362: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 681 B/s rd, 0 B/s wr, 644 op/s; 5481/289692 objects degraded (1.892%); 6923/289692 objects misplaced (2.390%); 5.1 MiB/s, 86 objects/s recovering 2024-09-18T07:37:51.220 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:51 smithi137 ceph-mon[39175]: 2.1 scrub starts 2024-09-18T07:37:51.220 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:51 smithi137 ceph-mon[39175]: 2.1 scrub ok 2024-09-18T07:37:51.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:51.324 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:51 smithi203 ceph-mon[32237]: pgmap v362: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 681 B/s rd, 0 B/s wr, 644 op/s; 5481/289692 objects degraded (1.892%); 6923/289692 objects misplaced (2.390%); 5.1 MiB/s, 86 objects/s recovering 2024-09-18T07:37:51.324 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:51 smithi203 ceph-mon[32237]: 2.1 scrub starts 2024-09-18T07:37:51.324 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:51 smithi203 ceph-mon[32237]: 2.1 scrub ok 2024-09-18T07:37:51.344 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:51.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:51.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:51.400 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:37:51.400 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:37:51.437 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:37:51.437 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:37:51.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:51.473 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:37:51.474 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:37:51.478 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:37:51.478 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:37:51.493 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:51.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:51.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:51.515 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:37:51.515 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:37:51.546 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:37:51.547 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:37:51.584 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:37:51.584 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:37:51.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:51.622 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:37:51.623 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:37:51.626 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:37:51.627 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-18T07:37:51.627 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:37:51.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:51.654 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:51.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:51.657 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:37:51.657 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:37:51.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:51.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:51.794 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:51.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:51.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:51.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:51.937 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:51.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:52.038 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:52.072 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:52.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:52.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:52.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:52.210 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:52.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:52.268 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T07:37:52.306 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:52 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2828056865' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:52.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:52.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:52.348 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:52.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:52 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2828056865' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:52 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2828056865' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:52.449 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:52.489 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:52.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:52.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:52.591 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:52.626 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:52.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:52.695 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:37:52.727 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:52.751 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:52.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:52.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:52.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:52.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:52.888 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:52.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:52.989 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:53.026 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:53.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:53.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:53.155 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:53.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:53.164 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:37:53.256 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:53.279 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:53.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:53.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:53.381 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:53.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:53 smithi203 ceph-mon[32237]: pgmap v364: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 570 op/s; 5481/287799 objects degraded (1.904%); 6652/287799 objects misplaced (2.311%); 6.1 MiB/s, 102 objects/s recovering 2024-09-18T07:37:53.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:53 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4136789990' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:37:53.415 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:53.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:53.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:53 smithi137 ceph-mon[29408]: pgmap v364: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 570 op/s; 5481/287799 objects degraded (1.904%); 6652/287799 objects misplaced (2.311%); 6.1 MiB/s, 102 objects/s recovering 2024-09-18T07:37:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:53 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4136789990' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:37:53.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:53 smithi137 ceph-mon[39175]: pgmap v364: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 570 op/s; 5481/287799 objects degraded (1.904%); 6652/287799 objects misplaced (2.311%); 6.1 MiB/s, 102 objects/s recovering 2024-09-18T07:37:53.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:53 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4136789990' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:37:53.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:53.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:53.558 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:53.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:53.659 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:53.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:53.702 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:53.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:53.803 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:53.815 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:37:53.815 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:37:53.815 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:37:53.816 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:37:53.816 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:37:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:53.838 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:53.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:53.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:53.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:53.966 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:53.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:54.068 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:54.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:54.106 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:54.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:54.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:54 smithi137 ceph-mon[29408]: 5.b scrub starts 2024-09-18T07:37:54.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:54 smithi137 ceph-mon[29408]: 5.b scrub ok 2024-09-18T07:37:54.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:54 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:37:54.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:54 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4136789990' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:37:54.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:54 smithi137 ceph-mon[29408]: osdmap e242: 8 total, 8 up, 8 in 2024-09-18T07:37:54.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:54 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:37:54.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:54 smithi137 ceph-mon[39175]: 5.b scrub starts 2024-09-18T07:37:54.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:54 smithi137 ceph-mon[39175]: 5.b scrub ok 2024-09-18T07:37:54.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:54 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:37:54.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:54 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4136789990' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:37:54.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:54 smithi137 ceph-mon[39175]: osdmap e242: 8 total, 8 up, 8 in 2024-09-18T07:37:54.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:54 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:37:54.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:54.238 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:54.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:54.342 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:54.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:54.385 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:54.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:54.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:54 smithi203 ceph-mon[32237]: 5.b scrub starts 2024-09-18T07:37:54.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:54 smithi203 ceph-mon[32237]: 5.b scrub ok 2024-09-18T07:37:54.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:54 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:37:54.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:54 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4136789990' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:37:54.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:54 smithi203 ceph-mon[32237]: osdmap e242: 8 total, 8 up, 8 in 2024-09-18T07:37:54.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:54 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:37:54.486 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:54.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:54.526 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:54.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:54.627 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:54.664 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:54.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:54.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:54.745 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T07:37:54.766 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:54.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:54.850 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:54.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:54.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:54.986 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:54.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:55.071 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:37:55.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:55.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:55.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:55.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:55.212 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:37:55.278 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:55.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:55.321 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:55.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:55.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:55 smithi203 ceph-mon[32237]: pgmap v366: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 5481/274824 objects degraded (1.994%); 6325/274824 objects misplaced (2.301%); 8.7 MiB/s, 142 objects/s recovering 2024-09-18T07:37:55.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:55 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 5481/274824 objects degraded (1.994%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:37:55.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:55 smithi203 ceph-mon[32237]: osdmap e243: 8 total, 8 up, 8 in 2024-09-18T07:37:55.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:55 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:37:55.398 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:55 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:37:55.398 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:55 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:37:55.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:55 smithi137 ceph-mon[29408]: pgmap v366: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 5481/274824 objects degraded (1.994%); 6325/274824 objects misplaced (2.301%); 8.7 MiB/s, 142 objects/s recovering 2024-09-18T07:37:55.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:55 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 5481/274824 objects degraded (1.994%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:37:55.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:55 smithi137 ceph-mon[29408]: osdmap e243: 8 total, 8 up, 8 in 2024-09-18T07:37:55.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:55 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:37:55.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:55 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:37:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:55 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:37:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:55 smithi137 ceph-mon[39175]: pgmap v366: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 5481/274824 objects degraded (1.994%); 6325/274824 objects misplaced (2.301%); 8.7 MiB/s, 142 objects/s recovering 2024-09-18T07:37:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:55 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 5481/274824 objects degraded (1.994%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:37:55.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:55 smithi137 ceph-mon[39175]: osdmap e243: 8 total, 8 up, 8 in 2024-09-18T07:37:55.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:55 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:37:55.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:55 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:37:55.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:55 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:37:55.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:55.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:55.489 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:55.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:55.547 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:55.591 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:55.629 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:55.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:55.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:55.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:55.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:55.769 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:55.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:55.808 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:37:55.808 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:37:55.808 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:37:55.808 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:37:55.809 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.809 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:20749 7.3 e88/81 7:c04d08df:::benchmark_data_smithi137_74180_object20748:head v 88'1239, mlcod=88'1237)", 2024-09-18T07:37:55.809 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.530035+0000", 2024-09-18T07:37:55.809 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 332.273388689, 2024-09-18T07:37:55.809 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13453020299999999, 2024-09-18T07:37:55.809 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.809 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.809 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.809 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.809 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.809 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.809 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530035+0000", 2024-09-18T07:37:55.810 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.810 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.810 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.810 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.810 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530035+0000", 2024-09-18T07:37:55.810 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.810 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.810 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.810 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.810 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530036+0000", 2024-09-18T07:37:55.810 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3200000000000004e-07 2024-09-18T07:37:55.810 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.810 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.810 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.811 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530106+0000", 2024-09-18T07:37:55.811 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0099999999999996e-05 2024-09-18T07:37:55.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.811 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.811 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.811 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530115+0000", 2024-09-18T07:37:55.811 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1419999999999999e-06 2024-09-18T07:37:55.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.811 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.812 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.812 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530122+0000", 2024-09-18T07:37:55.812 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3269999999999997e-06 2024-09-18T07:37:55.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.812 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.812 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.812 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530167+0000", 2024-09-18T07:37:55.812 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5247999999999998e-05 2024-09-18T07:37:55.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.812 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.812 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.813 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530185+0000", 2024-09-18T07:37:55.813 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8111e-05 2024-09-18T07:37:55.813 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.813 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.813 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.813 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530789+0000", 2024-09-18T07:37:55.813 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00060407400000000002 2024-09-18T07:37:55.813 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.813 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.813 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.813 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664566+0000", 2024-09-18T07:37:55.813 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13377636900000001 2024-09-18T07:37:55.814 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.814 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.814 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.814 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.814 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.814 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:20756 7.4 e88/81 7:206b3dce:::benchmark_data_smithi137_74180_object20755:head v 88'1303, mlcod=88'1301)", 2024-09-18T07:37:55.814 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.532983+0000", 2024-09-18T07:37:55.814 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 332.27044089899999, 2024-09-18T07:37:55.814 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13536040199999999, 2024-09-18T07:37:55.814 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.814 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.814 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.814 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.815 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.815 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.815 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.532983+0000", 2024-09-18T07:37:55.815 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.815 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.815 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.815 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.815 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.532983+0000", 2024-09-18T07:37:55.815 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.815 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.815 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.815 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.816 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.532984+0000", 2024-09-18T07:37:55.816 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7999999999999993e-07 2024-09-18T07:37:55.816 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.816 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.816 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.816 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533015+0000", 2024-09-18T07:37:55.816 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1195999999999997e-05 2024-09-18T07:37:55.816 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.816 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.816 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.816 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665392+0000", 2024-09-18T07:37:55.816 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13237665800000001 2024-09-18T07:37:55.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.817 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.817 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.817 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665400+0000", 2024-09-18T07:37:55.817 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7079999999999997e-06 2024-09-18T07:37:55.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.817 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.817 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.817 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665454+0000", 2024-09-18T07:37:55.817 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4116999999999997e-05 2024-09-18T07:37:55.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.817 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.817 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.818 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665474+0000", 2024-09-18T07:37:55.818 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0092999999999999e-05 2024-09-18T07:37:55.818 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.818 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.818 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.818 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.668302+0000", 2024-09-18T07:37:55.818 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0028280520000000002 2024-09-18T07:37:55.818 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.818 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.818 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.818 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.668343+0000", 2024-09-18T07:37:55.818 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1597999999999997e-05 2024-09-18T07:37:55.818 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.818 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.819 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.819 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.819 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.819 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:20750 7.3 e88/81 7:c7abdaa5:::benchmark_data_smithi137_74180_object20749:head v 88'1240, mlcod=88'1237)", 2024-09-18T07:37:55.819 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.533452+0000", 2024-09-18T07:37:55.819 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 332.26997171400001, 2024-09-18T07:37:55.819 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13358589900000001, 2024-09-18T07:37:55.819 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.819 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.819 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.819 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.819 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.819 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.820 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533452+0000", 2024-09-18T07:37:55.820 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.820 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.820 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.820 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.820 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533452+0000", 2024-09-18T07:37:55.820 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.820 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.820 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.820 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.820 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533453+0000", 2024-09-18T07:37:55.820 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.057e-06 2024-09-18T07:37:55.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.821 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.821 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.821 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533515+0000", 2024-09-18T07:37:55.821 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1557999999999996e-05 2024-09-18T07:37:55.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.821 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.821 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.821 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664524+0000", 2024-09-18T07:37:55.822 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13100931900000001 2024-09-18T07:37:55.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.822 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.822 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.822 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664540+0000", 2024-09-18T07:37:55.822 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5562000000000001e-05 2024-09-18T07:37:55.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.822 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.822 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.822 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664600+0000", 2024-09-18T07:37:55.822 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0124000000000002e-05 2024-09-18T07:37:55.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.822 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.822 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.823 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664620+0000", 2024-09-18T07:37:55.823 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9983000000000002e-05 2024-09-18T07:37:55.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.823 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.823 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.823 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.666988+0000", 2024-09-18T07:37:55.823 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023680559999999999 2024-09-18T07:37:55.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.823 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.823 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.823 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.667038+0000", 2024-09-18T07:37:55.823 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.024e-05 2024-09-18T07:37:55.823 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.824 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.824 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.824 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.824 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21270 7.1 7:801f54ca:::benchmark_data_smithi137_74180_object21269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:37:55.824 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.882531+0000", 2024-09-18T07:37:55.824 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 331.92089329100003, 2024-09-18T07:37:55.824 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16466234699999999, 2024-09-18T07:37:55.824 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.824 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.824 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.825 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:37:55.825 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:37:55.825 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:37:55.825 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21270 2024-09-18T07:37:55.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.825 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.825 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.825 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.825 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882531+0000", 2024-09-18T07:37:55.825 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.825 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.825 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.825 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882531+0000", 2024-09-18T07:37:55.826 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.826 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.826 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.826 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.826 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882533+0000", 2024-09-18T07:37:55.826 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8190000000000001e-06 2024-09-18T07:37:55.826 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.826 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.826 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.826 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882584+0000", 2024-09-18T07:37:55.826 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1919999999999998e-05 2024-09-18T07:37:55.826 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.827 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.827 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.827 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882593+0000", 2024-09-18T07:37:55.827 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8829999999999995e-06 2024-09-18T07:37:55.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.827 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.827 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.827 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882599+0000", 2024-09-18T07:37:55.827 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5849999999999999e-06 2024-09-18T07:37:55.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.827 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.827 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.827 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882634+0000", 2024-09-18T07:37:55.828 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5132999999999997e-05 2024-09-18T07:37:55.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.828 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.828 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.828 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882717+0000", 2024-09-18T07:37:55.828 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2819999999999996e-05 2024-09-18T07:37:55.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.828 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.828 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:37:55.828 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882824+0000", 2024-09-18T07:37:55.828 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010708400000000001 2024-09-18T07:37:55.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.828 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.828 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:37:55.829 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.883645+0000", 2024-09-18T07:37:55.829 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00082133399999999995 2024-09-18T07:37:55.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.829 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.829 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.829 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.940766+0000", 2024-09-18T07:37:55.829 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.057121166000000001 2024-09-18T07:37:55.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.829 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.829 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.829 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.046997+0000", 2024-09-18T07:37:55.829 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10623094900000001 2024-09-18T07:37:55.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.830 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.830 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.830 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047057+0000", 2024-09-18T07:37:55.830 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9580000000000002e-05 2024-09-18T07:37:55.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.830 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.830 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.830 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047193+0000", 2024-09-18T07:37:55.830 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000136074 2024-09-18T07:37:55.830 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.830 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.830 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.831 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.831 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21285 7.1 7:8dbe9f10:::benchmark_data_smithi137_74180_object21284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:37:55.831 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.944262+0000", 2024-09-18T07:37:55.831 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 331.85916224699997, 2024-09-18T07:37:55.831 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10313246500000001, 2024-09-18T07:37:55.831 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.831 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.831 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.832 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:37:55.832 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:37:55.832 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:37:55.832 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21285 2024-09-18T07:37:55.832 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.832 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.832 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.832 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.832 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944262+0000", 2024-09-18T07:37:55.833 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.833 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.833 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.833 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944262+0000", 2024-09-18T07:37:55.833 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.833 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.833 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.833 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944264+0000", 2024-09-18T07:37:55.833 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8679999999999999e-06 2024-09-18T07:37:55.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944324+0000", 2024-09-18T07:37:55.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0225e-05 2024-09-18T07:37:55.834 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944333+0000", 2024-09-18T07:37:55.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.064e-06 2024-09-18T07:37:55.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.835 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.835 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944338+0000", 2024-09-18T07:37:55.835 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9749999999999998e-06 2024-09-18T07:37:55.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.835 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.835 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944380+0000", 2024-09-18T07:37:55.835 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1773000000000001e-05 2024-09-18T07:37:55.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.835 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.835 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944476+0000", 2024-09-18T07:37:55.836 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6497999999999997e-05 2024-09-18T07:37:55.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.836 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.836 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:37:55.836 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.944565+0000", 2024-09-18T07:37:55.836 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9288000000000003e-05 2024-09-18T07:37:55.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.836 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.836 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:37:55.836 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.945376+0000", 2024-09-18T07:37:55.836 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00081097899999999998 2024-09-18T07:37:55.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.836 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.836 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.837 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.946088+0000", 2024-09-18T07:37:55.837 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00071115600000000001 2024-09-18T07:37:55.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.837 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.837 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.837 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047252+0000", 2024-09-18T07:37:55.837 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.101164449 2024-09-18T07:37:55.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.837 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.837 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.837 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047340+0000", 2024-09-18T07:37:55.837 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8268000000000003e-05 2024-09-18T07:37:55.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.838 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.838 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.838 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047394+0000", 2024-09-18T07:37:55.838 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3922000000000003e-05 2024-09-18T07:37:55.838 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.838 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.838 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.838 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.838 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22331 7.1 7:8575e846:::benchmark_data_smithi137_74180_object22330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:37:55.838 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.655146+0000", 2024-09-18T07:37:55.839 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 331.14827812599998, 2024-09-18T07:37:55.839 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.29868304200000001, 2024-09-18T07:37:55.839 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.839 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.839 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.839 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:37:55.839 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:37:55.839 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:37:55.839 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22331 2024-09-18T07:37:55.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.839 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.839 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.839 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.840 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655146+0000", 2024-09-18T07:37:55.841 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.841 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.841 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.841 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.841 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655146+0000", 2024-09-18T07:37:55.841 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.841 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.841 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.841 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.841 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655148+0000", 2024-09-18T07:37:55.842 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.779e-06 2024-09-18T07:37:55.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.842 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.842 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.842 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655192+0000", 2024-09-18T07:37:55.842 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4023000000000002e-05 2024-09-18T07:37:55.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.842 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.842 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.842 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655201+0000", 2024-09-18T07:37:55.843 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3559999999999999e-06 2024-09-18T07:37:55.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.844 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.844 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655206+0000", 2024-09-18T07:37:55.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4539999999999998e-06 2024-09-18T07:37:55.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.844 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.844 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.844 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655250+0000", 2024-09-18T07:37:55.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3630000000000001e-05 2024-09-18T07:37:55.845 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.845 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.845 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.845 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655323+0000", 2024-09-18T07:37:55.845 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3382999999999999e-05 2024-09-18T07:37:55.845 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.845 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.845 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:37:55.845 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655411+0000", 2024-09-18T07:37:55.845 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7776999999999997e-05 2024-09-18T07:37:55.845 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.845 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.845 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:37:55.845 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.656141+0000", 2024-09-18T07:37:55.846 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00072986100000000005 2024-09-18T07:37:55.846 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.846 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.846 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.846 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.658652+0000", 2024-09-18T07:37:55.846 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025112569999999998 2024-09-18T07:37:55.846 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.846 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.846 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.846 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953705+0000", 2024-09-18T07:37:55.846 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.29505283900000001 2024-09-18T07:37:55.846 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.846 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.847 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.847 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953782+0000", 2024-09-18T07:37:55.847 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6793999999999997e-05 2024-09-18T07:37:55.847 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.847 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.847 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.847 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953829+0000", 2024-09-18T07:37:55.847 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6888999999999998e-05 2024-09-18T07:37:55.847 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.847 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.847 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.847 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.847 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.847 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22350 7.1 7:851589b5:::benchmark_data_smithi137_74180_object22349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:37:55.848 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.662419+0000", 2024-09-18T07:37:55.848 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 331.141004845, 2024-09-18T07:37:55.848 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.29155694799999998, 2024-09-18T07:37:55.848 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.848 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.848 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.848 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:37:55.848 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:37:55.848 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:37:55.848 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22350 2024-09-18T07:37:55.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.849 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.849 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662419+0000", 2024-09-18T07:37:55.849 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.849 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662419+0000", 2024-09-18T07:37:55.849 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.850 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.850 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662421+0000", 2024-09-18T07:37:55.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.39e-06 2024-09-18T07:37:55.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.850 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.850 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662473+0000", 2024-09-18T07:37:55.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2094e-05 2024-09-18T07:37:55.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.850 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.850 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662480+0000", 2024-09-18T07:37:55.851 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5449999999999998e-06 2024-09-18T07:37:55.851 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.851 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.851 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.851 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662485+0000", 2024-09-18T07:37:55.851 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9200000000000003e-06 2024-09-18T07:37:55.851 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.851 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.851 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.851 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662521+0000", 2024-09-18T07:37:55.851 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5559000000000003e-05 2024-09-18T07:37:55.851 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.851 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.851 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.852 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662589+0000", 2024-09-18T07:37:55.852 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8480999999999997e-05 2024-09-18T07:37:55.852 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.852 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.852 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:37:55.852 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662666+0000", 2024-09-18T07:37:55.852 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7224999999999993e-05 2024-09-18T07:37:55.852 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.852 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.852 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:37:55.852 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.663390+0000", 2024-09-18T07:37:55.852 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00072354199999999998 2024-09-18T07:37:55.852 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.853 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.853 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.853 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.663928+0000", 2024-09-18T07:37:55.853 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00053822699999999995 2024-09-18T07:37:55.853 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.853 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.853 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.853 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953879+0000", 2024-09-18T07:37:55.853 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.28995123699999997 2024-09-18T07:37:55.853 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.853 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.853 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.853 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953934+0000", 2024-09-18T07:37:55.854 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5047000000000003e-05 2024-09-18T07:37:55.854 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.854 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.854 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.854 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953976+0000", 2024-09-18T07:37:55.854 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1681000000000003e-05 2024-09-18T07:37:55.854 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.854 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.854 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.854 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.854 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.854 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22352 7.1 7:83b03301:::benchmark_data_smithi137_74180_object22351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:37:55.854 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.664495+0000", 2024-09-18T07:37:55.854 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 331.13892876900002, 2024-09-18T07:37:55.855 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.28962424599999997, 2024-09-18T07:37:55.855 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.855 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.855 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.855 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:37:55.855 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:37:55.855 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:37:55.855 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22352 2024-09-18T07:37:55.855 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.855 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.855 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.855 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.855 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664495+0000", 2024-09-18T07:37:55.856 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.856 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.856 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.856 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.856 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664495+0000", 2024-09-18T07:37:55.856 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.856 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.856 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.856 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.856 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664496+0000", 2024-09-18T07:37:55.856 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6000000000000003e-07 2024-09-18T07:37:55.856 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.856 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.856 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.857 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664568+0000", 2024-09-18T07:37:55.857 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2021999999999996e-05 2024-09-18T07:37:55.857 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.857 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.857 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.857 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664574+0000", 2024-09-18T07:37:55.857 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4620000000000001e-06 2024-09-18T07:37:55.857 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.857 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.857 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.857 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664579+0000", 2024-09-18T07:37:55.857 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0620000000000002e-06 2024-09-18T07:37:55.857 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.858 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.858 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.858 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664605+0000", 2024-09-18T07:37:55.858 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6126e-05 2024-09-18T07:37:55.858 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.858 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.858 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.858 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664676+0000", 2024-09-18T07:37:55.858 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1521999999999997e-05 2024-09-18T07:37:55.858 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.858 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.858 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:37:55.858 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664749+0000", 2024-09-18T07:37:55.858 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3232999999999995e-05 2024-09-18T07:37:55.859 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.859 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.859 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:37:55.859 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.665309+0000", 2024-09-18T07:37:55.859 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00055992900000000003 2024-09-18T07:37:55.859 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.859 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.859 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.859 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.665857+0000", 2024-09-18T07:37:55.859 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00054812600000000002 2024-09-18T07:37:55.859 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.859 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.859 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.860 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.954036+0000", 2024-09-18T07:37:55.860 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.288178926 2024-09-18T07:37:55.860 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.860 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.860 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.860 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.954085+0000", 2024-09-18T07:37:55.860 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8534e-05 2024-09-18T07:37:55.860 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.860 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.860 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.860 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.954119+0000", 2024-09-18T07:37:55.860 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4544e-05 2024-09-18T07:37:55.860 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.861 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.861 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.861 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.861 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.861 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:44367 7.1 7:854d3320:::benchmark_data_smithi137_74180_object44366:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-18T07:37:55.861 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:43.591378+0000", 2024-09-18T07:37:55.861 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 312.21204620499998, 2024-09-18T07:37:55.861 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.113341394, 2024-09-18T07:37:55.861 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.861 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.861 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.861 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:37:55.861 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:37:55.862 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:37:55.862 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 44367 2024-09-18T07:37:55.862 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.862 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.862 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.862 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.862 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591378+0000", 2024-09-18T07:37:55.862 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.862 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.862 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.862 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.862 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591378+0000", 2024-09-18T07:37:55.862 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.863 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.863 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.863 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.863 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591379+0000", 2024-09-18T07:37:55.863 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7099999999999999e-06 2024-09-18T07:37:55.863 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.863 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.863 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.863 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591420+0000", 2024-09-18T07:37:55.863 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0559000000000002e-05 2024-09-18T07:37:55.863 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.863 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.863 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.863 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591432+0000", 2024-09-18T07:37:55.864 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.226e-05 2024-09-18T07:37:55.864 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.864 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.864 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.864 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591440+0000", 2024-09-18T07:37:55.864 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5830000000000001e-06 2024-09-18T07:37:55.864 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.864 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.864 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.864 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591497+0000", 2024-09-18T07:37:55.864 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7114999999999998e-05 2024-09-18T07:37:55.864 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.864 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.865 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.865 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591664+0000", 2024-09-18T07:37:55.865 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000167188 2024-09-18T07:37:55.865 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.865 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.865 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:37:55.865 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591884+0000", 2024-09-18T07:37:55.865 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000220223 2024-09-18T07:37:55.865 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.865 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.865 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:37:55.865 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.593492+0000", 2024-09-18T07:37:55.865 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016074139999999999 2024-09-18T07:37:55.866 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.866 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.866 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.866 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.593595+0000", 2024-09-18T07:37:55.866 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010319699999999999 2024-09-18T07:37:55.866 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.866 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.866 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.866 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.704655+0000", 2024-09-18T07:37:55.866 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.111060176 2024-09-18T07:37:55.866 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.866 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.866 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.866 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.704690+0000", 2024-09-18T07:37:55.867 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4916999999999998e-05 2024-09-18T07:37:55.867 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.867 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.867 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.867 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.704719+0000", 2024-09-18T07:37:55.867 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9051999999999999e-05 2024-09-18T07:37:55.867 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.867 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.867 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.867 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.867 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.867 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:51788 7.1 7:81b6d503:::benchmark_data_smithi137_74180_object51787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:37:55.867 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:37:55.868 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 305.13648002999997, 2024-09-18T07:37:55.868 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14889203200000001, 2024-09-18T07:37:55.868 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.868 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.868 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.868 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:37:55.868 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:37:55.868 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:37:55.868 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 51788 2024-09-18T07:37:55.868 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.868 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.868 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.868 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.868 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:37:55.869 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.869 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.869 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.869 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.869 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:37:55.869 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.869 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.869 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.869 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.869 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666966+0000", 2024-09-18T07:37:55.869 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1860000000000001e-05 2024-09-18T07:37:55.869 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.869 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.870 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.870 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666998+0000", 2024-09-18T07:37:55.870 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1856000000000001e-05 2024-09-18T07:37:55.870 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.870 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.870 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.870 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667012+0000", 2024-09-18T07:37:55.870 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4592e-05 2024-09-18T07:37:55.870 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.870 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.870 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.870 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667019+0000", 2024-09-18T07:37:55.870 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1940000000000001e-06 2024-09-18T07:37:55.871 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.871 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.871 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.871 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667083+0000", 2024-09-18T07:37:55.871 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3553999999999995e-05 2024-09-18T07:37:55.871 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.871 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.871 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.871 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667217+0000", 2024-09-18T07:37:55.871 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000133712 2024-09-18T07:37:55.871 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.871 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.871 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:37:55.872 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667345+0000", 2024-09-18T07:37:55.872 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012796199999999999 2024-09-18T07:37:55.872 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.872 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.872 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:37:55.872 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.668905+0000", 2024-09-18T07:37:55.872 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001560344 2024-09-18T07:37:55.872 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.872 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.872 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.872 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669080+0000", 2024-09-18T07:37:55.872 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000174964 2024-09-18T07:37:55.872 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.873 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.873 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.873 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815763+0000", 2024-09-18T07:37:55.873 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.146683217 2024-09-18T07:37:55.873 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.873 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.873 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.873 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815804+0000", 2024-09-18T07:37:55.873 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0535999999999999e-05 2024-09-18T07:37:55.873 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.873 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.873 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.873 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815836+0000", 2024-09-18T07:37:55.874 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2240999999999998e-05 2024-09-18T07:37:55.874 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.874 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.874 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.874 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.874 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.874 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:51794 7.1 7:8f789502:::benchmark_data_smithi137_74180_object51793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:37:55.874 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:37:55.874 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 305.13430691999997, 2024-09-18T07:37:55.874 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15046784899999999, 2024-09-18T07:37:55.874 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.874 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.874 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.874 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:37:55.875 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:37:55.875 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:37:55.875 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 51794 2024-09-18T07:37:55.875 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.875 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.875 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.875 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.875 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:37:55.875 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.875 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.875 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.875 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.875 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:37:55.876 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.876 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.876 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.876 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.876 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669118+0000", 2024-09-18T07:37:55.876 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4079999999999999e-06 2024-09-18T07:37:55.876 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.876 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.876 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.876 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669172+0000", 2024-09-18T07:37:55.876 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3808999999999999e-05 2024-09-18T07:37:55.876 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.876 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.877 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.877 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669184+0000", 2024-09-18T07:37:55.877 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1888e-05 2024-09-18T07:37:55.877 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.877 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.877 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.877 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669192+0000", 2024-09-18T07:37:55.877 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7549999999999994e-06 2024-09-18T07:37:55.877 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.877 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.877 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.877 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669229+0000", 2024-09-18T07:37:55.877 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7234000000000003e-05 2024-09-18T07:37:55.877 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.878 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.878 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.878 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669310+0000", 2024-09-18T07:37:55.878 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1248999999999994e-05 2024-09-18T07:37:55.878 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.878 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.878 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:37:55.878 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669391+0000", 2024-09-18T07:37:55.878 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0401000000000005e-05 2024-09-18T07:37:55.878 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.878 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.878 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:37:55.878 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.670397+0000", 2024-09-18T07:37:55.879 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010061619999999999 2024-09-18T07:37:55.879 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.879 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.879 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.879 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.670664+0000", 2024-09-18T07:37:55.879 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00026735700000000002 2024-09-18T07:37:55.879 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.879 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.879 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.879 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819479+0000", 2024-09-18T07:37:55.879 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14881438499999999 2024-09-18T07:37:55.879 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.879 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.879 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.880 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819537+0000", 2024-09-18T07:37:55.880 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8192999999999997e-05 2024-09-18T07:37:55.880 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.880 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.880 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.880 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819585+0000", 2024-09-18T07:37:55.880 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8007999999999998e-05 2024-09-18T07:37:55.880 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.880 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.880 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.880 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.880 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.880 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:74680 7.1 7:8999b183:::benchmark_data_smithi137_74180_object74679:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:37:55.881 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:37:55.881 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 279.625458874, 2024-09-18T07:37:55.881 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13647764200000001, 2024-09-18T07:37:55.881 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.881 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.881 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.881 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:37:55.881 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:37:55.881 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:37:55.881 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 74680 2024-09-18T07:37:55.881 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.881 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.881 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.881 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.882 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:37:55.882 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.882 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.882 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.882 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.882 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:37:55.882 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.882 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.882 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.882 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.882 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177967+0000", 2024-09-18T07:37:55.882 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.559e-06 2024-09-18T07:37:55.882 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.882 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.883 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.883 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178020+0000", 2024-09-18T07:37:55.883 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3810999999999997e-05 2024-09-18T07:37:55.883 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.883 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.883 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.883 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178030+0000", 2024-09-18T07:37:55.883 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5200000000000003e-06 2024-09-18T07:37:55.883 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.883 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.883 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.883 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178035+0000", 2024-09-18T07:37:55.883 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0429999999999998e-06 2024-09-18T07:37:55.883 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.883 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.884 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.884 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178074+0000", 2024-09-18T07:37:55.884 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9014999999999998e-05 2024-09-18T07:37:55.884 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.884 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.884 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.884 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178151+0000", 2024-09-18T07:37:55.884 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6526000000000006e-05 2024-09-18T07:37:55.884 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.884 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.884 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:37:55.884 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178233+0000", 2024-09-18T07:37:55.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:55.886 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2174000000000003e-05 2024-09-18T07:37:55.886 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.886 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.886 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:37:55.886 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.180222+0000", 2024-09-18T07:37:55.886 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0019888459999999998 2024-09-18T07:37:55.887 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.887 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.887 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.887 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.180453+0000", 2024-09-18T07:37:55.887 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023154 2024-09-18T07:37:55.887 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.887 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.887 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.887 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314370+0000", 2024-09-18T07:37:55.887 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.133916497 2024-09-18T07:37:55.887 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.887 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.887 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.888 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314411+0000", 2024-09-18T07:37:55.888 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1421000000000002e-05 2024-09-18T07:37:55.888 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.888 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.888 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.888 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314443+0000", 2024-09-18T07:37:55.888 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1690000000000003e-05 2024-09-18T07:37:55.888 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.888 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.888 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.888 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.888 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.888 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:76881 7.1 7:8430cf73:::benchmark_data_smithi137_74180_object76880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:37:55.888 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.426831+0000", 2024-09-18T07:37:55.889 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 277.37659311599998, 2024-09-18T07:37:55.889 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.115187357, 2024-09-18T07:37:55.889 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.889 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.889 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.889 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:37:55.889 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:37:55.889 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:37:55.889 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 76881 2024-09-18T07:37:55.889 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.889 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.889 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.889 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.890 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426831+0000", 2024-09-18T07:37:55.890 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.890 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.890 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.890 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.890 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426831+0000", 2024-09-18T07:37:55.890 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.890 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.890 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.890 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.890 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426833+0000", 2024-09-18T07:37:55.890 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7710000000000001e-06 2024-09-18T07:37:55.890 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.890 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.891 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.891 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426869+0000", 2024-09-18T07:37:55.891 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5886999999999997e-05 2024-09-18T07:37:55.891 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.891 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.891 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.891 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426879+0000", 2024-09-18T07:37:55.891 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0667e-05 2024-09-18T07:37:55.891 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.891 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.891 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.891 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426886+0000", 2024-09-18T07:37:55.891 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3799999999999999e-06 2024-09-18T07:37:55.891 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.891 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.892 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.892 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426932+0000", 2024-09-18T07:37:55.892 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6143999999999997e-05 2024-09-18T07:37:55.892 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.892 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.892 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.892 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.427047+0000", 2024-09-18T07:37:55.892 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011568700000000001 2024-09-18T07:37:55.892 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.892 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.892 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:37:55.892 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.427168+0000", 2024-09-18T07:37:55.892 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000120118 2024-09-18T07:37:55.893 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.893 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.893 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:37:55.893 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.429449+0000", 2024-09-18T07:37:55.893 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002281394 2024-09-18T07:37:55.893 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.893 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.893 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.893 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.431262+0000", 2024-09-18T07:37:55.893 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0018125940000000001 2024-09-18T07:37:55.893 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.893 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.893 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.893 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541358+0000", 2024-09-18T07:37:55.894 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.110095996 2024-09-18T07:37:55.894 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.894 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.894 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.894 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541963+0000", 2024-09-18T07:37:55.894 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00060516200000000002 2024-09-18T07:37:55.894 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.894 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.894 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.894 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.542018+0000", 2024-09-18T07:37:55.894 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5556999999999997e-05 2024-09-18T07:37:55.894 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.894 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.894 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.895 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.895 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.895 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:76891 7.f e116/115 7:f77e09af:::benchmark_data_smithi137_74180_object76890:head v 116'4811, mlcod=116'4809)", 2024-09-18T07:37:55.895 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.436137+0000", 2024-09-18T07:37:55.895 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 277.367286992, 2024-09-18T07:37:55.895 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.105530608, 2024-09-18T07:37:55.895 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.895 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.895 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.895 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.895 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.895 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.895 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436137+0000", 2024-09-18T07:37:55.895 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.896 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.896 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.896 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.896 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436137+0000", 2024-09-18T07:37:55.896 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.896 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.896 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.896 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.896 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436139+0000", 2024-09-18T07:37:55.896 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3259999999999998e-06 2024-09-18T07:37:55.896 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.896 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.896 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.896 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436192+0000", 2024-09-18T07:37:55.896 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2339000000000002e-05 2024-09-18T07:37:55.897 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.897 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.897 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.897 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436209+0000", 2024-09-18T07:37:55.897 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6935999999999998e-05 2024-09-18T07:37:55.897 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.897 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.897 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.897 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436219+0000", 2024-09-18T07:37:55.897 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0593e-05 2024-09-18T07:37:55.897 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.897 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.898 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.898 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436314+0000", 2024-09-18T07:37:55.898 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5035000000000001e-05 2024-09-18T07:37:55.898 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.898 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.898 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.898 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.436347+0000", 2024-09-18T07:37:55.899 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2558999999999998e-05 2024-09-18T07:37:55.900 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.900 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.900 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.900 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.437824+0000", 2024-09-18T07:37:55.900 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001477405 2024-09-18T07:37:55.900 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.900 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.900 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.900 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541668+0000", 2024-09-18T07:37:55.900 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10384341499999999 2024-09-18T07:37:55.900 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.900 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.900 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.901 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.901 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.901 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:76893 7.4 e116/81 7:2dcd0014:::benchmark_data_smithi137_74180_object76892:head v 116'4832, mlcod=116'4830)", 2024-09-18T07:37:55.901 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.438822+0000", 2024-09-18T07:37:55.901 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 277.364602015, 2024-09-18T07:37:55.901 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.106356858, 2024-09-18T07:37:55.901 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.901 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.901 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.901 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.901 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.901 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.901 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.438822+0000", 2024-09-18T07:37:55.901 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.902 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.902 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.902 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.902 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.438822+0000", 2024-09-18T07:37:55.902 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.902 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.902 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.902 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.902 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.438823+0000", 2024-09-18T07:37:55.902 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2979999999999999e-06 2024-09-18T07:37:55.902 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.902 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.902 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.903 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.438847+0000", 2024-09-18T07:37:55.903 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3370000000000002e-05 2024-09-18T07:37:55.903 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.903 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.903 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.903 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541862+0000", 2024-09-18T07:37:55.903 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10301571299999999 2024-09-18T07:37:55.903 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.903 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.903 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.903 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541872+0000", 2024-09-18T07:37:55.903 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0017000000000001e-05 2024-09-18T07:37:55.903 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.904 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.904 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.904 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541931+0000", 2024-09-18T07:37:55.904 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8990000000000003e-05 2024-09-18T07:37:55.904 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.904 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.904 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.904 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541958+0000", 2024-09-18T07:37:55.904 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6996e-05 2024-09-18T07:37:55.904 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.904 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.904 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.904 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.545137+0000", 2024-09-18T07:37:55.905 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003178859 2024-09-18T07:37:55.905 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.905 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.905 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.905 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.545179+0000", 2024-09-18T07:37:55.905 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1615000000000001e-05 2024-09-18T07:37:55.905 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.905 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.905 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.905 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.905 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.905 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:142957 7.1 7:830d8edf:::benchmark_data_smithi137_74180_object38541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-18T07:37:55.905 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:25.430263+0000", 2024-09-18T07:37:55.905 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 210.373161251, 2024-09-18T07:37:55.906 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18925961299999999, 2024-09-18T07:37:55.906 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.906 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.906 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.906 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:37:55.906 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:37:55.906 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:37:55.906 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 142957 2024-09-18T07:37:55.906 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.906 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.906 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.906 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.906 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430263+0000", 2024-09-18T07:37:55.906 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.907 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.907 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.907 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.907 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430263+0000", 2024-09-18T07:37:55.907 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.907 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.907 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.907 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.907 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430265+0000", 2024-09-18T07:37:55.907 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9800000000000001e-06 2024-09-18T07:37:55.907 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.907 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.908 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.908 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430270+0000", 2024-09-18T07:37:55.908 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6189999999999999e-06 2024-09-18T07:37:55.908 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.908 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.908 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.908 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430281+0000", 2024-09-18T07:37:55.908 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1146e-05 2024-09-18T07:37:55.908 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.908 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.908 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.908 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430288+0000", 2024-09-18T07:37:55.908 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9319999999999999e-06 2024-09-18T07:37:55.908 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.909 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.909 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.909 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430340+0000", 2024-09-18T07:37:55.909 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1904999999999999e-05 2024-09-18T07:37:55.909 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.909 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.909 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.909 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430468+0000", 2024-09-18T07:37:55.909 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000127448 2024-09-18T07:37:55.909 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.909 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.909 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:37:55.909 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430717+0000", 2024-09-18T07:37:55.910 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024888500000000001 2024-09-18T07:37:55.910 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.910 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.910 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:37:55.910 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.432189+0000", 2024-09-18T07:37:55.910 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014722470000000001 2024-09-18T07:37:55.910 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.910 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.910 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.910 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.432734+0000", 2024-09-18T07:37:55.910 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00054558799999999995 2024-09-18T07:37:55.910 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.910 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.619409+0000", 2024-09-18T07:37:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18667450399999999 2024-09-18T07:37:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.619471+0000", 2024-09-18T07:37:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1877000000000004e-05 2024-09-18T07:37:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.619522+0000", 2024-09-18T07:37:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1481999999999999e-05 2024-09-18T07:37:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.912 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.912 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.912 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.912 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.912 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:57669 8.2 8:473596c9:::benchmark_data_smithi137_101942_object57668:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-09-18T07:37:55.912 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:36:35.111481+0000", 2024-09-18T07:37:55.912 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 80.691942667000006, 2024-09-18T07:37:55.912 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10199733800000001, 2024-09-18T07:37:55.912 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.912 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.912 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.912 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:37:55.912 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:37:55.913 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:37:55.913 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 57669 2024-09-18T07:37:55.913 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.913 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.913 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.913 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.913 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:35.111481+0000", 2024-09-18T07:37:55.913 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.913 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.913 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.913 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.913 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:35.111481+0000", 2024-09-18T07:37:55.913 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.913 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.914 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.914 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.914 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:35.111483+0000", 2024-09-18T07:37:55.914 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0329999999999998e-06 2024-09-18T07:37:55.914 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.914 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.914 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.914 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:35.111512+0000", 2024-09-18T07:37:55.914 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8929e-05 2024-09-18T07:37:55.914 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.914 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.914 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.914 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:35.111524+0000", 2024-09-18T07:37:55.914 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1878e-05 2024-09-18T07:37:55.914 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.914 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.915 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.915 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:35.111532+0000", 2024-09-18T07:37:55.915 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6190000000000002e-06 2024-09-18T07:37:55.915 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.915 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.915 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.915 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:35.111742+0000", 2024-09-18T07:37:55.915 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00021068899999999999 2024-09-18T07:37:55.915 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.915 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.915 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.915 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:35.111852+0000", 2024-09-18T07:37:55.915 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000109685 2024-09-18T07:37:55.915 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.916 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.916 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:37:55.916 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:35.111999+0000", 2024-09-18T07:37:55.916 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014664699999999999 2024-09-18T07:37:55.916 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.916 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.916 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:37:55.916 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:35.115415+0000", 2024-09-18T07:37:55.916 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0034165139999999998 2024-09-18T07:37:55.916 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.916 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.916 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.916 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:35.117660+0000", 2024-09-18T07:37:55.916 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022450019999999998 2024-09-18T07:37:55.917 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.917 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.917 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.917 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:35.213372+0000", 2024-09-18T07:37:55.917 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.095712113000000001 2024-09-18T07:37:55.917 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.917 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.917 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.917 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:35.213430+0000", 2024-09-18T07:37:55.917 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7670999999999997e-05 2024-09-18T07:37:55.917 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.917 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.917 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.918 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:36:35.213479+0000", 2024-09-18T07:37:55.918 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8557999999999998e-05 2024-09-18T07:37:55.918 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.918 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.918 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.918 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.918 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.918 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:83272 8.5 8:a0373d93:::benchmark_data_smithi137_101942_object83271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-18T07:37:55.918 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:06.919689+0000", 2024-09-18T07:37:55.918 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 48.883735444000003, 2024-09-18T07:37:55.918 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10420881, 2024-09-18T07:37:55.918 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.918 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.919 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.919 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:37:55.919 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:37:55.919 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:37:55.919 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 83272 2024-09-18T07:37:55.919 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.919 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.919 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.919 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.919 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:06.919689+0000", 2024-09-18T07:37:55.919 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.919 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.919 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.919 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.920 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:06.919689+0000", 2024-09-18T07:37:55.920 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.920 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.920 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.920 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.920 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:06.919690+0000", 2024-09-18T07:37:55.920 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.409e-06 2024-09-18T07:37:55.920 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.920 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.920 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.920 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:06.919742+0000", 2024-09-18T07:37:55.920 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2546999999999997e-05 2024-09-18T07:37:55.920 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.921 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.921 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.921 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:06.919753+0000", 2024-09-18T07:37:55.921 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.063e-05 2024-09-18T07:37:55.921 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.921 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.921 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.921 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:06.919759+0000", 2024-09-18T07:37:55.921 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5480000000000001e-06 2024-09-18T07:37:55.921 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.921 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.921 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.921 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:06.919799+0000", 2024-09-18T07:37:55.922 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0522000000000003e-05 2024-09-18T07:37:55.922 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.922 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.922 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.922 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:06.919904+0000", 2024-09-18T07:37:55.922 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000105136 2024-09-18T07:37:55.922 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.922 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.922 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,7", 2024-09-18T07:37:55.922 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:06.920030+0000", 2024-09-18T07:37:55.922 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012548999999999999 2024-09-18T07:37:55.922 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.922 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.923 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:37:55.923 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:06.922015+0000", 2024-09-18T07:37:55.923 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0019856420000000001 2024-09-18T07:37:55.923 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.923 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.923 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.923 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:06.922370+0000", 2024-09-18T07:37:55.923 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00035502 2024-09-18T07:37:55.923 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.923 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.923 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.923 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:07.022824+0000", 2024-09-18T07:37:55.923 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10045337 2024-09-18T07:37:55.923 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.924 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.924 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.924 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:07.022870+0000", 2024-09-18T07:37:55.924 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6385999999999999e-05 2024-09-18T07:37:55.924 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.924 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.924 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.924 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:07.023897+0000", 2024-09-18T07:37:55.924 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010271099999999999 2024-09-18T07:37:55.924 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.924 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.924 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.924 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.925 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.925 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:95519 8.5 8:af8c363a:::benchmark_data_smithi137_101942_object95518:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-09-18T07:37:55.925 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:23.377289+0000", 2024-09-18T07:37:55.925 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 32.426135246999998, 2024-09-18T07:37:55.925 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.12657783, 2024-09-18T07:37:55.925 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.925 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.925 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.925 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:37:55.925 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:37:55.925 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:37:55.925 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 95519 2024-09-18T07:37:55.925 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.925 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.926 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.926 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.926 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377289+0000", 2024-09-18T07:37:55.926 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.926 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.926 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.926 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.926 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377289+0000", 2024-09-18T07:37:55.926 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.926 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.926 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.926 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.926 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377300+0000", 2024-09-18T07:37:55.926 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0998e-05 2024-09-18T07:37:55.927 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.927 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.927 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.927 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377358+0000", 2024-09-18T07:37:55.927 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7935e-05 2024-09-18T07:37:55.927 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.927 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.927 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.927 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377368+0000", 2024-09-18T07:37:55.927 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0344e-05 2024-09-18T07:37:55.927 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.927 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.927 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.927 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377374+0000", 2024-09-18T07:37:55.928 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3990000000000001e-06 2024-09-18T07:37:55.928 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.928 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.928 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.928 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377452+0000", 2024-09-18T07:37:55.928 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7829000000000003e-05 2024-09-18T07:37:55.928 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.928 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.928 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.928 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377626+0000", 2024-09-18T07:37:55.928 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017389900000000001 2024-09-18T07:37:55.928 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.928 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.928 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,7", 2024-09-18T07:37:55.929 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377747+0000", 2024-09-18T07:37:55.929 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012108900000000001 2024-09-18T07:37:55.929 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.929 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.929 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:37:55.929 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.379961+0000", 2024-09-18T07:37:55.929 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022133560000000001 2024-09-18T07:37:55.929 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.929 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.929 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.929 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.380219+0000", 2024-09-18T07:37:55.929 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000258879 2024-09-18T07:37:55.930 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.930 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.930 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.930 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.502323+0000", 2024-09-18T07:37:55.930 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.12210367900000001 2024-09-18T07:37:55.930 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.930 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.930 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.930 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.502368+0000", 2024-09-18T07:37:55.930 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4557e-05 2024-09-18T07:37:55.930 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.930 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.930 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.930 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.503867+0000", 2024-09-18T07:37:55.931 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001498866 2024-09-18T07:37:55.931 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.931 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.931 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.931 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.931 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.931 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:119507 8.2 8:4faaf3aa:::benchmark_data_smithi137_101942_object14697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-18T07:37:55.931 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:53.113193+0000", 2024-09-18T07:37:55.931 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 2.6902306239999998, 2024-09-18T07:37:55.931 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17040597299999999, 2024-09-18T07:37:55.931 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:55.931 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:55.931 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:55.932 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:37:55.932 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:37:55.932 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:37:55.932 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 119507 2024-09-18T07:37:55.932 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.932 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:55.932 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.932 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:55.932 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113193+0000", 2024-09-18T07:37:55.932 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.932 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.932 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.932 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:55.932 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113193+0000", 2024-09-18T07:37:55.933 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:55.933 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.933 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.933 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:55.933 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113195+0000", 2024-09-18T07:37:55.933 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.956e-06 2024-09-18T07:37:55.933 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.933 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.933 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:55.933 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113201+0000", 2024-09-18T07:37:55.933 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3639999999999998e-06 2024-09-18T07:37:55.933 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.933 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.933 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:55.933 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113210+0000", 2024-09-18T07:37:55.934 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6649999999999998e-06 2024-09-18T07:37:55.934 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.934 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.934 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:55.934 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113216+0000", 2024-09-18T07:37:55.934 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0639999999999999e-06 2024-09-18T07:37:55.934 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.934 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.934 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:55.934 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113260+0000", 2024-09-18T07:37:55.934 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4013e-05 2024-09-18T07:37:55.934 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.934 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.934 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:55.935 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113348+0000", 2024-09-18T07:37:55.935 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7392000000000006e-05 2024-09-18T07:37:55.935 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.935 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.935 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:37:55.935 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113429+0000", 2024-09-18T07:37:55.935 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1111000000000003e-05 2024-09-18T07:37:55.935 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.935 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.935 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:37:55.935 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.115474+0000", 2024-09-18T07:37:55.935 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0020447769999999998 2024-09-18T07:37:55.935 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.935 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.936 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.936 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.116688+0000", 2024-09-18T07:37:55.936 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001214581 2024-09-18T07:37:55.936 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.936 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.936 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:55.936 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.283501+0000", 2024-09-18T07:37:55.936 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.166812558 2024-09-18T07:37:55.936 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.936 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.936 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:55.936 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.283557+0000", 2024-09-18T07:37:55.937 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5856999999999997e-05 2024-09-18T07:37:55.937 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:55.937 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:55.937 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:37:55.937 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.283599+0000", 2024-09-18T07:37:55.937 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2635000000000001e-05 2024-09-18T07:37:55.937 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.937 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.937 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.937 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:55.937 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:55.937 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:37:55.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:55.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:55.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:56.040 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:37:56.074 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:56.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:56.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:37:56.176 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:56.215 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:56.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:56.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:56.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:56.348 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:56.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:56.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:56.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:56.483 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:56.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:56.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:56.585 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:56.623 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:56.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:56.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:56.725 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:56.757 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:56.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:56.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:56.859 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:56.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:56.896 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:56.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:56.997 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:57.003 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T07:37:57.011 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:37:57.023 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:57.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:57.125 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:57.164 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:57.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:57.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:57 smithi203 ceph-mon[32237]: pgmap v368: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 826 op/s; 5481/274824 objects degraded (1.994%); 6325/274824 objects misplaced (2.301%); 6.2 MiB/s, 99 objects/s recovering 2024-09-18T07:37:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:57 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:37:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:57 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:37:57.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:57.311 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:57.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:57.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:57.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:57 smithi137 ceph-mon[39175]: pgmap v368: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 826 op/s; 5481/274824 objects degraded (1.994%); 6325/274824 objects misplaced (2.301%); 6.2 MiB/s, 99 objects/s recovering 2024-09-18T07:37:57.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:57 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:37:57.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:57 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:37:57.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:57 smithi137 ceph-mon[29408]: pgmap v368: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 826 op/s; 5481/274824 objects degraded (1.994%); 6325/274824 objects misplaced (2.301%); 6.2 MiB/s, 99 objects/s recovering 2024-09-18T07:37:57.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:57 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:37:57.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:57 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:37:57.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:37:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:37:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:37:57.434 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:37:57.447 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:57.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:57.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:57.549 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:57.568 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:37:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:37:57.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:57.587 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:57.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:57.688 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:37:57.711 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:57.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:57.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:37:57.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:57.852 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:57.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:57.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:57.954 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:37:57.996 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:57.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:57.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:37:58.030 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:37:58.030 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:37:58.030 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.030 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:123968 8.a e243/210 8:5258d51d:::benchmark_data_smithi137_101942_object19158:head v 243'7728, mlcod=243'7726)", 2024-09-18T07:37:58.030 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:58.022163+0000", 2024-09-18T07:37:58.030 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0038602739999999999, 2024-09-18T07:37:58.030 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0038895240000000001, 2024-09-18T07:37:58.030 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:58.031 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:58.031 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:37:58.031 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:58.031 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.031 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:58.031 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.022163+0000", 2024-09-18T07:37:58.031 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:58.031 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.031 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.031 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:58.031 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.022163+0000", 2024-09-18T07:37:58.031 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:58.031 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.032 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.032 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:58.032 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.022165+0000", 2024-09-18T07:37:58.032 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.457e-06 2024-09-18T07:37:58.032 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.032 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.032 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:58.032 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.022170+0000", 2024-09-18T07:37:58.032 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2140000000000004e-06 2024-09-18T07:37:58.032 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.032 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.032 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:58.033 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.022180+0000", 2024-09-18T07:37:58.033 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6600000000000007e-06 2024-09-18T07:37:58.033 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.033 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:58.033 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.022184+0000", 2024-09-18T07:37:58.033 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7870000000000001e-06 2024-09-18T07:37:58.033 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.033 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:58.033 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.022230+0000", 2024-09-18T07:37:58.033 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5847999999999999e-05 2024-09-18T07:37:58.033 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:58.034 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.022251+0000", 2024-09-18T07:37:58.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0808000000000001e-05 2024-09-18T07:37:58.034 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:58.034 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:58.034 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:58.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.034 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:123969 8.a e243/210 8:5fc6e7a6:::benchmark_data_smithi137_101942_object19159:head v 243'7729, mlcod=243'7726)", 2024-09-18T07:37:58.034 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:58.024698+0000", 2024-09-18T07:37:58.034 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0013257430000000001, 2024-09-18T07:37:58.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014096169999999999, 2024-09-18T07:37:58.035 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:58.035 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:58.035 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:37:58.035 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:58.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:58.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.024698+0000", 2024-09-18T07:37:58.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:58.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:58.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.024698+0000", 2024-09-18T07:37:58.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:58.036 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.036 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.036 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:58.036 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.024700+0000", 2024-09-18T07:37:58.036 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.697e-06 2024-09-18T07:37:58.036 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.036 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.036 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:58.036 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.024711+0000", 2024-09-18T07:37:58.036 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1122e-05 2024-09-18T07:37:58.036 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.036 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:58.037 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.024723+0000", 2024-09-18T07:37:58.037 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2238999999999999e-05 2024-09-18T07:37:58.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:58.037 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.024730+0000", 2024-09-18T07:37:58.037 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6649999999999997e-06 2024-09-18T07:37:58.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:58.038 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.024819+0000", 2024-09-18T07:37:58.038 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9767999999999998e-05 2024-09-18T07:37:58.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.038 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.038 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:58.038 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.024834+0000", 2024-09-18T07:37:58.038 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4431e-05 2024-09-18T07:37:58.038 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:58.038 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:58.038 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:58.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.038 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop_reply(client.35171.0:123966 8.3 e243/238 ondisk, result = 0)", 2024-09-18T07:37:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:58.025275+0000", 2024-09-18T07:37:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.00074898, 2024-09-18T07:37:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00087823899999999995, 2024-09-18T07:37:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:37:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.025275+0000", 2024-09-18T07:37:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.025275+0000", 2024-09-18T07:37:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.025276+0000", 2024-09-18T07:37:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1090000000000001e-06 2024-09-18T07:37:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.025281+0000", 2024-09-18T07:37:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9810000000000003e-06 2024-09-18T07:37:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.025292+0000", 2024-09-18T07:37:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0883999999999999e-05 2024-09-18T07:37:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.025297+0000", 2024-09-18T07:37:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8100000000000003e-06 2024-09-18T07:37:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.025336+0000", 2024-09-18T07:37:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8127999999999998e-05 2024-09-18T07:37:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.025342+0000", 2024-09-18T07:37:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.764e-06 2024-09-18T07:37:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:123973 8.d 8:bf3860cb:::benchmark_data_smithi137_101942_object19163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-18T07:37:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:58.025962+0000", 2024-09-18T07:37:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 6.1315999999999994e-05, 2024-09-18T07:37:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024024800000000001, 2024-09-18T07:37:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:37:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:37:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:37:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:37:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 123973 2024-09-18T07:37:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.025962+0000", 2024-09-18T07:37:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.025962+0000", 2024-09-18T07:37:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.025964+0000", 2024-09-18T07:37:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2189999999999998e-06 2024-09-18T07:37:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.025970+0000", 2024-09-18T07:37:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.733e-06 2024-09-18T07:37:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.025980+0000", 2024-09-18T07:37:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0037999999999999e-05 2024-09-18T07:37:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.025987+0000", 2024-09-18T07:37:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0010000000000004e-06 2024-09-18T07:37:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.026057+0000", 2024-09-18T07:37:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9915999999999999e-05 2024-09-18T07:37:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.026183+0000", 2024-09-18T07:37:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000126198 2024-09-18T07:37:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:123966 8.3 8:ca83b104:::benchmark_data_smithi137_101942_object19156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-09-18T07:37:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:58.016936+0000", 2024-09-18T07:37:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0090877030000000008, 2024-09-18T07:37:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0093251589999999995, 2024-09-18T07:37:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:37:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:37:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:37:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:37:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:37:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:37:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 123966 2024-09-18T07:37:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:37:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:37:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.016936+0000", 2024-09-18T07:37:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:37:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.016936+0000", 2024-09-18T07:37:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:37:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:37:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.016937+0000", 2024-09-18T07:37:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.539e-06 2024-09-18T07:37:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:37:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.016942+0000", 2024-09-18T07:37:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6959999999999996e-06 2024-09-18T07:37:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:37:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.016962+0000", 2024-09-18T07:37:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9439000000000001e-05 2024-09-18T07:37:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:37:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.016968+0000", 2024-09-18T07:37:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1380000000000004e-06 2024-09-18T07:37:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:37:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.018019+0000", 2024-09-18T07:37:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001051497 2024-09-18T07:37:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:37:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.018161+0000", 2024-09-18T07:37:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014147300000000001 2024-09-18T07:37:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:37:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.018299+0000", 2024-09-18T07:37:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013854899999999999 2024-09-18T07:37:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:37:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.021850+0000", 2024-09-18T07:37:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0035509560000000001 2024-09-18T07:37:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.025216+0000", 2024-09-18T07:37:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0033658170000000001 2024-09-18T07:37:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:37:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.025348+0000", 2024-09-18T07:37:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013175799999999999 2024-09-18T07:37:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:37:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:37:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:37:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:58.025402+0000", 2024-09-18T07:37:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4268000000000003e-05 2024-09-18T07:37:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:37:58.054 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:58.054 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:37:58.054 INFO:teuthology.orchestra.run.smithi203.stdout: ], 2024-09-18T07:37:58.054 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 5 2024-09-18T07:37:58.054 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:37:58.055 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:37:58.055 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:37:58.071 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:37:58.071 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:37:58.098 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:58.106 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:37:58.107 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:37:58.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:58.137 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:58.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:58.142 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:37:58.142 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:37:58.173 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:37:58.178 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:37:58.178 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:37:58.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:58 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1775313820' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:58.211 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:37:58.212 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:37:58.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:58.244 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:58 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1775313820' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:58.245 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:58 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1775313820' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:37:58.245 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:37:58.245 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:37:58.271 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:58.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:58.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:58.280 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:37:58.281 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:37:58.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:58.415 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:58.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:58.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:58.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:58.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:58.554 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:58.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:58.655 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:58.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:58.693 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:58.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:58.795 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:58.832 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:58.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:58.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:58.904 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:37:58.905 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:37:58.905 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-18T07:37:58.905 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T07:37:58.905 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T07:37:58.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-18T07:37:58.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:58.986 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:58.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:58.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:59.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:37:59.115 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:59.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:59.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:37:59.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:59.257 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:59.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:37:59.359 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:37:59.398 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:59 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:59.398 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:59 smithi203 ceph-mon[32237]: pgmap v369: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.26k op/s; 5481/260829 objects degraded (2.101%); 5673/260829 objects misplaced (2.175%); 10 MiB/s, 163 objects/s recovering 2024-09-18T07:37:59.398 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:59 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:37:59.398 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:37:59 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:37:59.399 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T07:37:59.403 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:59.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:59.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:37:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:59 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:59 smithi137 ceph-mon[29408]: pgmap v369: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.26k op/s; 5481/260829 objects degraded (2.101%); 5673/260829 objects misplaced (2.175%); 10 MiB/s, 163 objects/s recovering 2024-09-18T07:37:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:59 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:37:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:37:59 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:37:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:59 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:37:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:59 smithi137 ceph-mon[39175]: pgmap v369: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.26k op/s; 5481/260829 objects degraded (2.101%); 5673/260829 objects misplaced (2.175%); 10 MiB/s, 163 objects/s recovering 2024-09-18T07:37:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:59 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:37:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:37:59 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:37:59.506 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:59.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:59.560 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:59.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:59.661 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:37:59.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:59.708 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:37:59.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:37:59.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:37:59.837 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:59.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:59.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:37:59.860 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:37:59.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:37:59.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:37:59.972 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:37:59.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:00.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:00.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:00.096 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:00.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:00.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:00.244 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:00.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:00.346 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:00.373 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:00.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:00.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:00.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:00 smithi137 ceph-mon[29408]: osdmap e244: 8 total, 8 up, 8 in 2024-09-18T07:38:00.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:00 smithi137 ceph-mon[29408]: pgmap v371: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 802 B/s rd, 0 B/s wr, 731 op/s; 5481/260829 objects degraded (2.101%); 5673/260829 objects misplaced (2.175%); 6.4 MiB/s, 102 objects/s recovering 2024-09-18T07:38:00.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:00 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 5481/260829 objects degraded (2.101%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:38:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:00 smithi137 ceph-mon[29408]: osdmap e245: 8 total, 8 up, 8 in 2024-09-18T07:38:00.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:00 smithi137 ceph-mon[39175]: osdmap e244: 8 total, 8 up, 8 in 2024-09-18T07:38:00.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:00 smithi137 ceph-mon[39175]: pgmap v371: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 802 B/s rd, 0 B/s wr, 731 op/s; 5481/260829 objects degraded (2.101%); 5673/260829 objects misplaced (2.175%); 6.4 MiB/s, 102 objects/s recovering 2024-09-18T07:38:00.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:00 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 5481/260829 objects degraded (2.101%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:38:00.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:00 smithi137 ceph-mon[39175]: osdmap e245: 8 total, 8 up, 8 in 2024-09-18T07:38:00.420 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:00 smithi203 ceph-mon[32237]: osdmap e244: 8 total, 8 up, 8 in 2024-09-18T07:38:00.421 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:00 smithi203 ceph-mon[32237]: pgmap v371: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 802 B/s rd, 0 B/s wr, 731 op/s; 5481/260829 objects degraded (2.101%); 5673/260829 objects misplaced (2.175%); 6.4 MiB/s, 102 objects/s recovering 2024-09-18T07:38:00.421 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:00 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 5481/260829 objects degraded (2.101%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:38:00.421 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:00 smithi203 ceph-mon[32237]: osdmap e245: 8 total, 8 up, 8 in 2024-09-18T07:38:00.474 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:00.519 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:00.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:00.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:00.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:00.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:00.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:00.718 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:38:00.719 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:38:00.757 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:00.796 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:00.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:00.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:00.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:01.107 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:01.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:01.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:01.209 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:01.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:01.248 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:01.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:01.349 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:01.394 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:01.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:01.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:01.409 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T07:38:01.496 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:01.523 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:01.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:01.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:01.625 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:01.659 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:01.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:01.761 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:01.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:01.830 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:01.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:01.880 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:38:01.931 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:01.977 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:01.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:01.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:02.080 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:02.102 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:02.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:02.204 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:02.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:02.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:02.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:02.339 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:02.374 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:02.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:02.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:02.477 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:02.494 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:38:02.494 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:38:02.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:02.516 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:02.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:02.617 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:02.635 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:02.651 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:02.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:02.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:02.752 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:02.775 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:02 smithi137 ceph-mon[29408]: pgmap v373: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 783 op/s; 5481/259224 objects degraded (2.114%); 5363/259224 objects misplaced (2.069%); 9.0 MiB/s, 144 objects/s recovering 2024-09-18T07:38:02.775 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:02 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:38:02.776 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:02 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:38:02.776 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:02 smithi137 ceph-mon[39175]: pgmap v373: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 783 op/s; 5481/259224 objects degraded (2.114%); 5363/259224 objects misplaced (2.069%); 9.0 MiB/s, 144 objects/s recovering 2024-09-18T07:38:02.776 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:02 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:38:02.776 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:02 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:38:02.785 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:02.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:02.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:02.887 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:02.923 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:02.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:02.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:02.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:02 smithi203 ceph-mon[32237]: pgmap v373: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 783 op/s; 5481/259224 objects degraded (2.114%); 5363/259224 objects misplaced (2.069%); 9.0 MiB/s, 144 objects/s recovering 2024-09-18T07:38:02.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:02 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:38:02.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:02 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:38:03.024 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:03.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:03.058 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:03.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:03.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:03.201 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:03.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:03.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:03.208 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T07:38:03.239 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:38:03.239 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "" 2024-09-18T07:38:03.239 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:38:03.304 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:03.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:03.349 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:03.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:03.450 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:03.490 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:03.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:03.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:03.627 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:03.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:03.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:03.629 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:38:03.652 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:38:03.729 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:03.764 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:03.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:03.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:03.866 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:03.902 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:03.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:03.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:03.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:03 smithi137 ceph-mon[29408]: osdmap e246: 8 total, 8 up, 8 in 2024-09-18T07:38:03.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:03 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2905152308' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:38:03.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:03 smithi137 ceph-mon[39175]: osdmap e246: 8 total, 8 up, 8 in 2024-09-18T07:38:03.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:03 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2905152308' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:38:03.949 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:03 smithi203 ceph-mon[32237]: osdmap e246: 8 total, 8 up, 8 in 2024-09-18T07:38:03.949 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:03 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2905152308' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:38:04.004 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:04.039 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:04.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:04.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:04.141 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:04.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:04.181 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:04.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:04.282 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:04.288 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:38:04.289 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:38:04.313 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:04.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:04.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:04.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:38:04.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:04.460 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:04.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:04.562 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:04.602 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:04.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:04.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:04.635 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:04 smithi137 ceph-mon[29408]: pgmap v375: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 772 op/s; 5481/246936 objects degraded (2.220%); 4996/246936 objects misplaced (2.023%); 7.0 MiB/s, 112 objects/s recovering 2024-09-18T07:38:04.636 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:04 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:38:04.636 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:04 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2905152308' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:38:04.636 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:04 smithi137 ceph-mon[29408]: osdmap e247: 8 total, 8 up, 8 in 2024-09-18T07:38:04.706 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:04.743 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:04.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:04.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:04.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:04.883 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:04.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:04.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:04 smithi137 ceph-mon[39175]: pgmap v375: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 772 op/s; 5481/246936 objects degraded (2.220%); 4996/246936 objects misplaced (2.023%); 7.0 MiB/s, 112 objects/s recovering 2024-09-18T07:38:04.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:04 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:38:04.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:04 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2905152308' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:38:04.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:04 smithi137 ceph-mon[39175]: osdmap e247: 8 total, 8 up, 8 in 2024-09-18T07:38:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:04 smithi203 ceph-mon[32237]: pgmap v375: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 772 op/s; 5481/246936 objects degraded (2.220%); 4996/246936 objects misplaced (2.023%); 7.0 MiB/s, 112 objects/s recovering 2024-09-18T07:38:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:04 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:38:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:04 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2905152308' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:38:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:04 smithi203 ceph-mon[32237]: osdmap e247: 8 total, 8 up, 8 in 2024-09-18T07:38:04.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:05.021 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:05.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:05.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:05.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:05.160 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:05.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:05.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:05.261 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:05.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:05.290 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:05.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:05.391 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:05.392 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:05.428 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:05.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:05.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:05.530 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:05.565 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:05.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:05.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:05.667 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:05.698 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:05.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:05.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:05.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:05.835 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:05.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:05.850 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:05 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 5481/246936 objects degraded (2.220%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:38:05.850 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:05 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:38:05.850 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:05 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:38:05.851 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:05 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 5481/246936 objects degraded (2.220%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:38:05.851 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:05 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:38:05.851 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:05 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:38:05.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:05.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:05 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 5481/246936 objects degraded (2.220%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:38:05.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:05 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:38:05.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:05 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:38:05.983 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:05.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:06.040 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:38:06.040 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:38:06.074 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:38:06.075 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:38:06.085 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:06.100 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:38:06.101 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:38:06.133 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:38:06.133 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:38:06.134 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:06.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:06.143 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T07:38:06.166 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:38:06.166 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:38:06.201 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:38:06.201 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:38:06.236 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:06.238 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:38:06.238 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:38:06.270 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:38:06.271 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:38:06.275 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:06.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:06.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:06.405 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:06.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:06.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:06.507 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:06.557 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:06.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:06.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:06.602 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:38:06.661 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:06.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:06.695 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:06.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:06.796 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:06.837 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:06.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:06.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:06.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:06.968 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:06.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:06.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:07.058 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:06 smithi203 ceph-mon[32237]: pgmap v377: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 772 op/s; 5481/246936 objects degraded (2.220%); 4996/246936 objects misplaced (2.023%); 7.1 MiB/s, 112 objects/s recovering 2024-09-18T07:38:07.059 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:06 smithi203 ceph-mon[32237]: osdmap e248: 8 total, 8 up, 8 in 2024-09-18T07:38:07.059 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:06 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2141951531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:07.070 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:07.082 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:06 smithi137 ceph-mon[39175]: pgmap v377: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 772 op/s; 5481/246936 objects degraded (2.220%); 4996/246936 objects misplaced (2.023%); 7.1 MiB/s, 112 objects/s recovering 2024-09-18T07:38:07.082 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:06 smithi137 ceph-mon[39175]: osdmap e248: 8 total, 8 up, 8 in 2024-09-18T07:38:07.082 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:06 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2141951531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:07.083 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:06 smithi137 ceph-mon[29408]: pgmap v377: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 772 op/s; 5481/246936 objects degraded (2.220%); 4996/246936 objects misplaced (2.023%); 7.1 MiB/s, 112 objects/s recovering 2024-09-18T07:38:07.083 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:06 smithi137 ceph-mon[29408]: osdmap e248: 8 total, 8 up, 8 in 2024-09-18T07:38:07.083 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:06 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2141951531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:07.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:07.113 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:07.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:07.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:07.229 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:38:07.229 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:38:07.229 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:38:07.229 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:38:07.229 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:38:07.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:07.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:07.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:07.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:38:07.333 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:38:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:38:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:38:07.350 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:07.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:07.388 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:07.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:07.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:38:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:38:07.489 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:07.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:07.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:07.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:07.644 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:07.679 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:07.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:07.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:07.731 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:07.780 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:07.815 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:07.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:07.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:07.917 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:07.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:07.956 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:07.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:08.022 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T07:38:08.053 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:07 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:38:08.054 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:07 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:38:08.057 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:08.097 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:08.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:08.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:08.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:07 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:38:08.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:07 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:38:08.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:07 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:38:08.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:07 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:38:08.199 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:08.233 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:08.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:08.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:08.335 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:08.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:08.387 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:08.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:08.442 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:38:08.488 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:08.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:08.533 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:08.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:08.634 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:08.670 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:08.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:08.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:08.773 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:08.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:08.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:08.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:08.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:08.943 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:08.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:08.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:08.956 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:38:09.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:09.065 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:38:09.065 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:38:09.065 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:38:09.065 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:38:09.066 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.066 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54188 7.c 7:3726bab8:::benchmark_data_smithi137_74180_object54187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T07:38:09.066 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.151582+0000", 2024-09-18T07:38:09.066 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 315.90794435999999, 2024-09-18T07:38:09.066 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0152226929999999, 2024-09-18T07:38:09.066 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.067 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.067 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:38:09.067 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.067 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.067 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.067 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54188 2024-09-18T07:38:09.068 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.068 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.068 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.068 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.068 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151582+0000", 2024-09-18T07:38:09.068 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.069 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.069 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.069 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151582+0000", 2024-09-18T07:38:09.069 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.070 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.070 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.070 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151583+0000", 2024-09-18T07:38:09.070 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6300000000000001e-06 2024-09-18T07:38:09.070 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.071 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.071 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.071 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151619+0000", 2024-09-18T07:38:09.071 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5178000000000001e-05 2024-09-18T07:38:09.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.071 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.072 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.072 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151627+0000", 2024-09-18T07:38:09.072 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2849999999999995e-06 2024-09-18T07:38:09.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.072 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.072 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.073 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151633+0000", 2024-09-18T07:38:09.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5729999999999996e-06 2024-09-18T07:38:09.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.073 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.073 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.152287+0000", 2024-09-18T07:38:09.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00065367599999999995 2024-09-18T07:38:09.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.074 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:38:09.074 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.152442+0000", 2024-09-18T07:38:09.074 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015491699999999999 2024-09-18T07:38:09.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.074 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.074 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:38:09.074 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.152627+0000", 2024-09-18T07:38:09.074 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018544300000000001 2024-09-18T07:38:09.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.075 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:38:09.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.156875+0000", 2024-09-18T07:38:09.075 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0042476329999999998 2024-09-18T07:38:09.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.075 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.161790+0000", 2024-09-18T07:38:09.075 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0049151050000000003 2024-09-18T07:38:09.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.075 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.076 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.161803+0000", 2024-09-18T07:38:09.076 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2646999999999999e-05 2024-09-18T07:38:09.076 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.076 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.076 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.076 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.166719+0000", 2024-09-18T07:38:09.076 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0049166270000001 2024-09-18T07:38:09.077 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.077 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.077 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.077 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.166804+0000", 2024-09-18T07:38:09.077 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4979000000000001e-05 2024-09-18T07:38:09.077 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.077 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.077 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.077 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.077 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.077 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54193 7.c 7:33299084:::benchmark_data_smithi137_74180_object54192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T07:38:09.077 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.154898+0000", 2024-09-18T07:38:09.077 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 315.90462839200001, 2024-09-18T07:38:09.078 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0124331120000001, 2024-09-18T07:38:09.078 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.078 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.078 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:38:09.078 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.078 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.078 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.078 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54193 2024-09-18T07:38:09.078 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.078 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.078 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.078 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.078 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154898+0000", 2024-09-18T07:38:09.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.079 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154898+0000", 2024-09-18T07:38:09.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.079 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154900+0000", 2024-09-18T07:38:09.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1100000000000001e-06 2024-09-18T07:38:09.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154938+0000", 2024-09-18T07:38:09.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8479999999999997e-05 2024-09-18T07:38:09.080 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.080 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.080 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154957+0000", 2024-09-18T07:38:09.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8437e-05 2024-09-18T07:38:09.080 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.080 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.080 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154965+0000", 2024-09-18T07:38:09.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1780000000000003e-06 2024-09-18T07:38:09.080 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.080 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.081 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.081 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.155029+0000", 2024-09-18T07:38:09.081 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4459000000000001e-05 2024-09-18T07:38:09.081 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.081 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.081 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:38:09.081 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.155164+0000", 2024-09-18T07:38:09.081 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013460699999999999 2024-09-18T07:38:09.081 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.081 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.081 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:38:09.081 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.155322+0000", 2024-09-18T07:38:09.081 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015807199999999999 2024-09-18T07:38:09.081 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.082 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.082 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:38:09.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.157099+0000", 2024-09-18T07:38:09.082 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0017770570000000001 2024-09-18T07:38:09.082 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.082 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.082 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.161956+0000", 2024-09-18T07:38:09.082 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.004857264 2024-09-18T07:38:09.082 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.082 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.083 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.083 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.161962+0000", 2024-09-18T07:38:09.083 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9479999999999996e-06 2024-09-18T07:38:09.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.083 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.083 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.083 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.167294+0000", 2024-09-18T07:38:09.083 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.005331441 2024-09-18T07:38:09.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.083 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.083 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.083 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.167331+0000", 2024-09-18T07:38:09.083 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7058999999999999e-05 2024-09-18T07:38:09.084 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.084 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.084 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.084 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.084 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54223 7.c 7:39274186:::benchmark_data_smithi137_74180_object54222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:38:09.084 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.192250+0000", 2024-09-18T07:38:09.084 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 315.867275828, 2024-09-18T07:38:09.084 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.98501845799999999, 2024-09-18T07:38:09.084 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.084 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.084 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:38:09.086 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.086 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.086 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.086 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54223 2024-09-18T07:38:09.086 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.086 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.086 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.086 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.087 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192250+0000", 2024-09-18T07:38:09.087 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.087 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.087 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.087 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192250+0000", 2024-09-18T07:38:09.087 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.087 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.087 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.087 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192251+0000", 2024-09-18T07:38:09.087 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.037e-06 2024-09-18T07:38:09.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.087 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192317+0000", 2024-09-18T07:38:09.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5246999999999994e-05 2024-09-18T07:38:09.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192327+0000", 2024-09-18T07:38:09.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0464e-05 2024-09-18T07:38:09.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192335+0000", 2024-09-18T07:38:09.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2239999999999995e-06 2024-09-18T07:38:09.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192402+0000", 2024-09-18T07:38:09.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6580000000000003e-05 2024-09-18T07:38:09.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192414+0000", 2024-09-18T07:38:09.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2475e-05 2024-09-18T07:38:09.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.090 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.090 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.167364+0000", 2024-09-18T07:38:09.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97494975800000006 2024-09-18T07:38:09.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.090 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.090 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:38:09.090 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.167483+0000", 2024-09-18T07:38:09.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000119238 2024-09-18T07:38:09.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.090 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.090 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:38:09.090 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.167628+0000", 2024-09-18T07:38:09.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000144709 2024-09-18T07:38:09.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:38:09.091 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.172321+0000", 2024-09-18T07:38:09.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0046927879999999998 2024-09-18T07:38:09.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.091 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173324+0000", 2024-09-18T07:38:09.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001003096 2024-09-18T07:38:09.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.091 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.177200+0000", 2024-09-18T07:38:09.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0038757449999999999 2024-09-18T07:38:09.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:38:09.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.177233+0000", 2024-09-18T07:38:09.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3358000000000001e-05 2024-09-18T07:38:09.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.177269+0000", 2024-09-18T07:38:09.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5738999999999998e-05 2024-09-18T07:38:09.092 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.092 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.092 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.093 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.093 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.093 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54246 7.c 7:39613093:::benchmark_data_smithi137_74180_object54245:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:38:09.093 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.212409+0000", 2024-09-18T07:38:09.093 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 315.84711713500002, 2024-09-18T07:38:09.093 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97093596400000004, 2024-09-18T07:38:09.093 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.094 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.094 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:38:09.094 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.094 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.094 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.094 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54246 2024-09-18T07:38:09.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.095 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.095 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212409+0000", 2024-09-18T07:38:09.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.096 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.096 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.096 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212409+0000", 2024-09-18T07:38:09.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.096 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.096 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.097 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.097 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212412+0000", 2024-09-18T07:38:09.097 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6350000000000002e-06 2024-09-18T07:38:09.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.097 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.097 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.098 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212483+0000", 2024-09-18T07:38:09.098 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1408999999999993e-05 2024-09-18T07:38:09.098 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.098 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.098 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.098 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212497+0000", 2024-09-18T07:38:09.099 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3576e-05 2024-09-18T07:38:09.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.099 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.099 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.099 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212505+0000", 2024-09-18T07:38:09.100 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1990000000000006e-06 2024-09-18T07:38:09.100 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.100 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.100 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.100 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212575+0000", 2024-09-18T07:38:09.101 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0662999999999998e-05 2024-09-18T07:38:09.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.101 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.101 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212588+0000", 2024-09-18T07:38:09.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2639e-05 2024-09-18T07:38:09.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.102 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.169092+0000", 2024-09-18T07:38:09.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.95650415300000002 2024-09-18T07:38:09.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.103 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.103 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:38:09.103 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.169206+0000", 2024-09-18T07:38:09.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000113328 2024-09-18T07:38:09.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.104 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:38:09.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.169341+0000", 2024-09-18T07:38:09.105 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013537600000000001 2024-09-18T07:38:09.105 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.105 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.105 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:38:09.105 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173339+0000", 2024-09-18T07:38:09.105 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0039977909999999997 2024-09-18T07:38:09.105 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.105 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.105 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.105 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.182250+0000", 2024-09-18T07:38:09.105 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.008910968 2024-09-18T07:38:09.105 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.105 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.106 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.106 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.183281+0000", 2024-09-18T07:38:09.106 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010308940000000001 2024-09-18T07:38:09.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.106 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.106 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:38:09.106 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.183314+0000", 2024-09-18T07:38:09.106 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3791000000000002e-05 2024-09-18T07:38:09.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.106 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.106 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.106 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.183345+0000", 2024-09-18T07:38:09.106 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0542e-05 2024-09-18T07:38:09.106 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.107 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.107 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.107 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.107 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54263 7.c 7:35b11aa8:::benchmark_data_smithi137_74180_object54262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:38:09.107 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.227767+0000", 2024-09-18T07:38:09.107 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 315.83175938699998, 2024-09-18T07:38:09.107 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.95717552100000003, 2024-09-18T07:38:09.107 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.107 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.107 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:38:09.107 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.108 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.108 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.108 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54263 2024-09-18T07:38:09.108 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.108 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.108 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.108 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.108 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227767+0000", 2024-09-18T07:38:09.108 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.108 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.108 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.108 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.109 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227767+0000", 2024-09-18T07:38:09.109 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.109 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.109 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.109 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.109 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227769+0000", 2024-09-18T07:38:09.109 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8640000000000001e-06 2024-09-18T07:38:09.109 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.109 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.109 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.109 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227814+0000", 2024-09-18T07:38:09.109 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5540000000000001e-05 2024-09-18T07:38:09.109 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.109 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.110 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.110 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227829+0000", 2024-09-18T07:38:09.110 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4586000000000001e-05 2024-09-18T07:38:09.110 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.110 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.110 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.110 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227837+0000", 2024-09-18T07:38:09.110 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3080000000000008e-06 2024-09-18T07:38:09.110 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.110 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.110 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.110 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227923+0000", 2024-09-18T07:38:09.110 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5531999999999994e-05 2024-09-18T07:38:09.110 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.111 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.111 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.111 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227934+0000", 2024-09-18T07:38:09.111 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1232e-05 2024-09-18T07:38:09.111 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.111 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.111 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.111 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.170335+0000", 2024-09-18T07:38:09.111 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.94240168099999999 2024-09-18T07:38:09.111 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.111 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.111 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:38:09.111 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.170404+0000", 2024-09-18T07:38:09.111 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9021000000000002e-05 2024-09-18T07:38:09.112 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.112 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.112 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:38:09.112 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.170496+0000", 2024-09-18T07:38:09.112 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1472999999999996e-05 2024-09-18T07:38:09.112 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.112 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.112 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:38:09.112 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173344+0000", 2024-09-18T07:38:09.112 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0028482970000000001 2024-09-18T07:38:09.112 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.112 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.112 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.113 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.182293+0000", 2024-09-18T07:38:09.113 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0089482840000000008 2024-09-18T07:38:09.113 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.113 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.113 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.113 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.184822+0000", 2024-09-18T07:38:09.113 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025292090000000001 2024-09-18T07:38:09.113 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.113 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.113 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:38:09.113 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.184880+0000", 2024-09-18T07:38:09.113 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8680000000000001e-05 2024-09-18T07:38:09.114 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.114 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.114 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.114 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.184942+0000", 2024-09-18T07:38:09.114 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1814000000000001e-05 2024-09-18T07:38:09.114 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.114 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.114 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.114 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.114 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.114 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54278 7.c 7:36e62dce:::benchmark_data_smithi137_74180_object54277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:38:09.114 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.241010+0000", 2024-09-18T07:38:09.114 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 315.818515639, 2024-09-18T07:38:09.114 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.94404960299999996, 2024-09-18T07:38:09.114 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.115 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.115 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:38:09.115 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.115 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.115 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.115 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54278 2024-09-18T07:38:09.115 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.115 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.115 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.115 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.115 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241010+0000", 2024-09-18T07:38:09.115 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.115 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.116 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.116 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.116 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241010+0000", 2024-09-18T07:38:09.116 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.116 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.116 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.116 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.116 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241014+0000", 2024-09-18T07:38:09.116 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.112e-06 2024-09-18T07:38:09.116 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.116 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.116 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.116 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241049+0000", 2024-09-18T07:38:09.116 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5605999999999998e-05 2024-09-18T07:38:09.117 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.117 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.117 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.117 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241067+0000", 2024-09-18T07:38:09.117 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8077e-05 2024-09-18T07:38:09.117 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.117 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.117 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.117 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241075+0000", 2024-09-18T07:38:09.117 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7510000000000008e-06 2024-09-18T07:38:09.117 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.117 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.117 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.117 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241153+0000", 2024-09-18T07:38:09.118 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7642e-05 2024-09-18T07:38:09.118 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.118 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.118 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.118 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241167+0000", 2024-09-18T07:38:09.118 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3987e-05 2024-09-18T07:38:09.118 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.118 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.118 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.118 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.171235+0000", 2024-09-18T07:38:09.118 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93006873700000003 2024-09-18T07:38:09.118 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.118 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.119 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:38:09.119 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.171328+0000", 2024-09-18T07:38:09.119 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2200999999999994e-05 2024-09-18T07:38:09.119 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.119 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.119 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:38:09.119 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.171447+0000", 2024-09-18T07:38:09.119 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000119874 2024-09-18T07:38:09.119 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.119 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.119 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:38:09.119 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.175350+0000", 2024-09-18T07:38:09.119 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0039030359999999999 2024-09-18T07:38:09.119 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.119 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.120 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.120 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.182332+0000", 2024-09-18T07:38:09.120 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069816849999999996 2024-09-18T07:38:09.120 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.120 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.120 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.120 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.184998+0000", 2024-09-18T07:38:09.120 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026659689999999998 2024-09-18T07:38:09.120 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.120 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.120 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:38:09.120 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185034+0000", 2024-09-18T07:38:09.120 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5694000000000001e-05 2024-09-18T07:38:09.120 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.121 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.121 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.121 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185060+0000", 2024-09-18T07:38:09.121 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6231999999999999e-05 2024-09-18T07:38:09.121 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.121 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.121 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.121 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.121 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.121 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54283 7.c 7:30afc400:::benchmark_data_smithi137_74180_object54282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:38:09.121 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.244474+0000", 2024-09-18T07:38:09.121 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 315.81505228399999, 2024-09-18T07:38:09.121 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.94069020800000003, 2024-09-18T07:38:09.122 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.122 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.122 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:38:09.122 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.122 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.122 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.122 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54283 2024-09-18T07:38:09.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.122 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.122 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.122 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.122 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244474+0000", 2024-09-18T07:38:09.122 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.123 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.123 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.123 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244474+0000", 2024-09-18T07:38:09.123 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.123 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.123 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.123 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.123 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244476+0000", 2024-09-18T07:38:09.123 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6390000000000001e-06 2024-09-18T07:38:09.123 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.123 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.123 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.123 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244521+0000", 2024-09-18T07:38:09.124 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4379999999999999e-05 2024-09-18T07:38:09.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.124 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.124 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244538+0000", 2024-09-18T07:38:09.124 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7354e-05 2024-09-18T07:38:09.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.124 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.124 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244548+0000", 2024-09-18T07:38:09.124 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5030000000000003e-06 2024-09-18T07:38:09.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.124 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.125 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244624+0000", 2024-09-18T07:38:09.125 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6244999999999999e-05 2024-09-18T07:38:09.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.125 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.125 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.125 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244636+0000", 2024-09-18T07:38:09.125 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2218000000000001e-05 2024-09-18T07:38:09.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.125 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.125 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.125 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.172361+0000", 2024-09-18T07:38:09.125 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92772458400000002 2024-09-18T07:38:09.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.125 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.126 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:38:09.126 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.172455+0000", 2024-09-18T07:38:09.126 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4475999999999995e-05 2024-09-18T07:38:09.126 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.126 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.126 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:38:09.126 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.172532+0000", 2024-09-18T07:38:09.126 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6969000000000001e-05 2024-09-18T07:38:09.126 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.126 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.126 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:38:09.126 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.177152+0000", 2024-09-18T07:38:09.126 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0046201699999999998 2024-09-18T07:38:09.126 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.127 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.127 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.127 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.182361+0000", 2024-09-18T07:38:09.127 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0052089659999999998 2024-09-18T07:38:09.127 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.127 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.127 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.127 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185097+0000", 2024-09-18T07:38:09.127 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027359110000000002 2024-09-18T07:38:09.127 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.127 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.127 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:38:09.127 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185130+0000", 2024-09-18T07:38:09.127 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3031000000000003e-05 2024-09-18T07:38:09.127 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.128 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.128 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.128 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185164+0000", 2024-09-18T07:38:09.128 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3762e-05 2024-09-18T07:38:09.128 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.128 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.128 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.128 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.128 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.128 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54286 7.c 7:3a3f8689:::benchmark_data_smithi137_74180_object54285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:38:09.128 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.245256+0000", 2024-09-18T07:38:09.128 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 315.81427002800001, 2024-09-18T07:38:09.128 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93997832100000001, 2024-09-18T07:38:09.129 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.129 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.129 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:38:09.129 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.129 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.129 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.129 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54286 2024-09-18T07:38:09.129 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.129 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.129 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.129 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.129 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245256+0000", 2024-09-18T07:38:09.129 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.130 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.130 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245256+0000", 2024-09-18T07:38:09.130 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.130 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.130 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245258+0000", 2024-09-18T07:38:09.130 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.578e-06 2024-09-18T07:38:09.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.130 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245346+0000", 2024-09-18T07:38:09.131 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8776000000000005e-05 2024-09-18T07:38:09.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.131 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.131 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245361+0000", 2024-09-18T07:38:09.131 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4687e-05 2024-09-18T07:38:09.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.131 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.131 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245368+0000", 2024-09-18T07:38:09.131 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2989999999999999e-06 2024-09-18T07:38:09.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.131 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.132 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.132 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245437+0000", 2024-09-18T07:38:09.132 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8869999999999996e-05 2024-09-18T07:38:09.132 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.132 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.132 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.132 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245447+0000", 2024-09-18T07:38:09.132 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0172e-05 2024-09-18T07:38:09.132 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.132 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.132 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.132 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173386+0000", 2024-09-18T07:38:09.132 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92793816900000003 2024-09-18T07:38:09.133 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.133 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.133 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:38:09.133 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173452+0000", 2024-09-18T07:38:09.133 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6050999999999995e-05 2024-09-18T07:38:09.133 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.133 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.133 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:38:09.133 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173533+0000", 2024-09-18T07:38:09.133 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1578999999999999e-05 2024-09-18T07:38:09.133 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.133 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.133 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:38:09.133 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.177162+0000", 2024-09-18T07:38:09.134 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0036291240000000001 2024-09-18T07:38:09.134 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.134 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.134 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.134 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.182390+0000", 2024-09-18T07:38:09.134 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0052279880000000003 2024-09-18T07:38:09.134 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.134 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.134 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.134 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185188+0000", 2024-09-18T07:38:09.134 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027977779999999999 2024-09-18T07:38:09.134 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.134 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.134 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:38:09.135 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185215+0000", 2024-09-18T07:38:09.135 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6764999999999999e-05 2024-09-18T07:38:09.135 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:09.135 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:09.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:09.136 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:38:09.137 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.137 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.137 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185234+0000", 2024-09-18T07:38:09.137 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9485000000000001e-05 2024-09-18T07:38:09.137 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.137 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.137 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.138 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.138 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.138 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54303 7.c 7:3c96a443:::benchmark_data_smithi137_74180_object54302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:38:09.138 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.265470+0000", 2024-09-18T07:38:09.138 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 315.79405640599998, 2024-09-18T07:38:09.138 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92892820399999998, 2024-09-18T07:38:09.138 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.138 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.138 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:38:09.138 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.138 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.138 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.138 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54303 2024-09-18T07:38:09.138 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.139 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.139 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.139 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.139 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265470+0000", 2024-09-18T07:38:09.139 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.139 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.139 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.139 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.139 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265470+0000", 2024-09-18T07:38:09.139 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.139 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.139 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.139 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.140 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265472+0000", 2024-09-18T07:38:09.140 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2400000000000002e-06 2024-09-18T07:38:09.140 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.140 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.140 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.140 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265528+0000", 2024-09-18T07:38:09.140 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6047000000000001e-05 2024-09-18T07:38:09.140 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.140 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.140 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.140 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265539+0000", 2024-09-18T07:38:09.140 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1119e-05 2024-09-18T07:38:09.140 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.140 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.141 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.141 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265547+0000", 2024-09-18T07:38:09.141 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7470000000000005e-06 2024-09-18T07:38:09.141 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.141 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.141 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.141 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265600+0000", 2024-09-18T07:38:09.141 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3062000000000001e-05 2024-09-18T07:38:09.141 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.141 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.141 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.141 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265608+0000", 2024-09-18T07:38:09.141 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8669999999999994e-06 2024-09-18T07:38:09.142 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.142 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.142 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.142 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.174396+0000", 2024-09-18T07:38:09.142 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90878840299999997 2024-09-18T07:38:09.142 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.142 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.142 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:38:09.142 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.174505+0000", 2024-09-18T07:38:09.142 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108837 2024-09-18T07:38:09.142 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.142 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.142 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:38:09.142 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.174589+0000", 2024-09-18T07:38:09.143 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3825000000000005e-05 2024-09-18T07:38:09.143 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.143 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.143 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:38:09.143 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.178295+0000", 2024-09-18T07:38:09.143 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0037061170000000001 2024-09-18T07:38:09.143 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.143 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.143 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.143 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.188101+0000", 2024-09-18T07:38:09.143 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0098061350000000005 2024-09-18T07:38:09.143 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.143 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.143 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.144 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194313+0000", 2024-09-18T07:38:09.144 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0062117589999999999 2024-09-18T07:38:09.144 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.144 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.144 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:38:09.144 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194353+0000", 2024-09-18T07:38:09.144 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9869000000000001e-05 2024-09-18T07:38:09.144 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.144 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.144 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.144 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194398+0000", 2024-09-18T07:38:09.144 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5176999999999998e-05 2024-09-18T07:38:09.144 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.144 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.145 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.145 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.145 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.145 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54304 7.c 7:36172b18:::benchmark_data_smithi137_74180_object54303:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:38:09.145 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.265725+0000", 2024-09-18T07:38:09.145 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 315.79380107899999, 2024-09-18T07:38:09.145 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92876516499999995, 2024-09-18T07:38:09.145 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.145 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.145 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:38:09.145 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.145 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.145 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.146 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54304 2024-09-18T07:38:09.146 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.146 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.146 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.146 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.146 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265725+0000", 2024-09-18T07:38:09.146 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.146 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.146 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.146 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.146 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265725+0000", 2024-09-18T07:38:09.146 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.146 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.146 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.147 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.147 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265726+0000", 2024-09-18T07:38:09.147 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0449999999999999e-06 2024-09-18T07:38:09.147 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.147 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.147 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.147 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265785+0000", 2024-09-18T07:38:09.147 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.94e-05 2024-09-18T07:38:09.147 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.147 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.147 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.147 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265795+0000", 2024-09-18T07:38:09.147 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2550000000000003e-06 2024-09-18T07:38:09.147 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.148 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.148 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.148 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265802+0000", 2024-09-18T07:38:09.148 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8190000000000003e-06 2024-09-18T07:38:09.148 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.148 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.148 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.148 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265846+0000", 2024-09-18T07:38:09.148 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4867999999999998e-05 2024-09-18T07:38:09.148 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.148 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.148 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.148 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265853+0000", 2024-09-18T07:38:09.149 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0469999999999996e-06 2024-09-18T07:38:09.149 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.149 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.149 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.149 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.175383+0000", 2024-09-18T07:38:09.149 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90952946099999998 2024-09-18T07:38:09.149 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.149 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.149 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:38:09.149 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.175462+0000", 2024-09-18T07:38:09.149 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9513999999999998e-05 2024-09-18T07:38:09.149 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.149 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.149 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:38:09.150 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.175546+0000", 2024-09-18T07:38:09.150 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3614000000000003e-05 2024-09-18T07:38:09.150 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.150 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.150 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:38:09.150 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.178303+0000", 2024-09-18T07:38:09.150 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027565210000000001 2024-09-18T07:38:09.150 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.150 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.150 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.150 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194058+0000", 2024-09-18T07:38:09.150 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.015755377000000001 2024-09-18T07:38:09.150 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.150 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.151 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.151 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194432+0000", 2024-09-18T07:38:09.151 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00037386800000000002 2024-09-18T07:38:09.151 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.151 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.151 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:38:09.151 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194464+0000", 2024-09-18T07:38:09.151 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2330999999999999e-05 2024-09-18T07:38:09.151 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.151 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.151 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.151 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194490+0000", 2024-09-18T07:38:09.151 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6044999999999999e-05 2024-09-18T07:38:09.152 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.152 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.152 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.152 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.152 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.152 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54310 7.c 7:30b8e4f0:::benchmark_data_smithi137_74180_object54309:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:38:09.152 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.271051+0000", 2024-09-18T07:38:09.152 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 315.78847483499999, 2024-09-18T07:38:09.152 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92352962000000005, 2024-09-18T07:38:09.152 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.152 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.152 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:38:09.152 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.152 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.153 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.153 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54310 2024-09-18T07:38:09.153 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.153 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.153 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.153 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.153 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271051+0000", 2024-09-18T07:38:09.153 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.153 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.153 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.153 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.153 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271051+0000", 2024-09-18T07:38:09.153 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.153 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.154 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.154 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.154 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271053+0000", 2024-09-18T07:38:09.154 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.013e-06 2024-09-18T07:38:09.154 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.154 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.154 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.154 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271110+0000", 2024-09-18T07:38:09.154 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6329000000000002e-05 2024-09-18T07:38:09.154 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.154 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.154 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.154 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271121+0000", 2024-09-18T07:38:09.154 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1761e-05 2024-09-18T07:38:09.155 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.155 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.155 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.155 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271127+0000", 2024-09-18T07:38:09.155 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0390000000000001e-06 2024-09-18T07:38:09.155 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.155 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.155 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.155 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271189+0000", 2024-09-18T07:38:09.155 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1492e-05 2024-09-18T07:38:09.155 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.155 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.155 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.155 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271200+0000", 2024-09-18T07:38:09.156 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1195999999999999e-05 2024-09-18T07:38:09.156 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.156 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.156 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.156 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.176292+0000", 2024-09-18T07:38:09.156 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90509152199999998 2024-09-18T07:38:09.156 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.156 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.156 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:38:09.156 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.176362+0000", 2024-09-18T07:38:09.156 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9901999999999996e-05 2024-09-18T07:38:09.156 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.156 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.157 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:38:09.157 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.176445+0000", 2024-09-18T07:38:09.157 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3622000000000007e-05 2024-09-18T07:38:09.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.157 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.157 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:38:09.157 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.180339+0000", 2024-09-18T07:38:09.157 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0038937350000000002 2024-09-18T07:38:09.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.157 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.157 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.157 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194104+0000", 2024-09-18T07:38:09.157 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.013765370000000001 2024-09-18T07:38:09.158 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.158 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.158 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.158 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194523+0000", 2024-09-18T07:38:09.158 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00041917199999999999 2024-09-18T07:38:09.158 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.158 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.158 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:38:09.158 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194551+0000", 2024-09-18T07:38:09.158 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7489999999999999e-05 2024-09-18T07:38:09.158 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.158 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.158 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.158 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194581+0000", 2024-09-18T07:38:09.159 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9977000000000001e-05 2024-09-18T07:38:09.159 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.159 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.159 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.159 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.159 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.159 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153406 7.7 7:e5535fc6:::benchmark_data_smithi137_74180_object48990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-18T07:38:09.159 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.169823+0000", 2024-09-18T07:38:09.159 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 206.889702615, 2024-09-18T07:38:09.159 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.083783358, 2024-09-18T07:38:09.159 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.159 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.159 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:38:09.160 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.160 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.160 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.160 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153406 2024-09-18T07:38:09.160 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.160 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.160 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.160 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.160 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169823+0000", 2024-09-18T07:38:09.160 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.160 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.160 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.160 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.160 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169823+0000", 2024-09-18T07:38:09.161 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.161 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.161 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.161 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.161 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169826+0000", 2024-09-18T07:38:09.161 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1229999999999998e-06 2024-09-18T07:38:09.161 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.161 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.161 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.161 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169833+0000", 2024-09-18T07:38:09.161 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6869999999999994e-06 2024-09-18T07:38:09.161 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.161 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.162 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.162 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169851+0000", 2024-09-18T07:38:09.162 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7978999999999998e-05 2024-09-18T07:38:09.162 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.162 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.162 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.162 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169867+0000", 2024-09-18T07:38:09.162 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5610000000000001e-05 2024-09-18T07:38:09.162 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.162 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.162 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.162 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169944+0000", 2024-09-18T07:38:09.162 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6998999999999999e-05 2024-09-18T07:38:09.162 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.162 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.163 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:38:09.163 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.170029+0000", 2024-09-18T07:38:09.163 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4765999999999994e-05 2024-09-18T07:38:09.163 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.163 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.163 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:38:09.163 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.170285+0000", 2024-09-18T07:38:09.163 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025587199999999998 2024-09-18T07:38:09.163 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.163 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.163 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:38:09.163 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.171936+0000", 2024-09-18T07:38:09.163 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001651189 2024-09-18T07:38:09.163 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.164 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.164 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.164 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.194310+0000", 2024-09-18T07:38:09.164 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.022374616999999999 2024-09-18T07:38:09.164 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.164 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.164 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.164 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238955+0000", 2024-09-18T07:38:09.164 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.044644716000000001 2024-09-18T07:38:09.164 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.164 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.164 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.164 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238967+0000", 2024-09-18T07:38:09.165 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1933000000000001e-05 2024-09-18T07:38:09.165 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.165 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.165 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.165 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.253578+0000", 2024-09-18T07:38:09.165 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0146105599999999 2024-09-18T07:38:09.165 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.165 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.165 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.165 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.253607+0000", 2024-09-18T07:38:09.165 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9306999999999999e-05 2024-09-18T07:38:09.165 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.165 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.165 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.166 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.166 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.166 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153419 7.17 7:e887289b:::benchmark_data_smithi137_74180_object49003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-18T07:38:09.166 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.193652+0000", 2024-09-18T07:38:09.166 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 206.86587379100001, 2024-09-18T07:38:09.166 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0605730499999999, 2024-09-18T07:38:09.166 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.166 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.166 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:38:09.166 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.166 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.166 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.166 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153419 2024-09-18T07:38:09.166 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.167 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.167 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.167 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.167 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193652+0000", 2024-09-18T07:38:09.167 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.167 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.167 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.167 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.167 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193652+0000", 2024-09-18T07:38:09.167 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.167 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.167 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.167 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.168 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193654+0000", 2024-09-18T07:38:09.168 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.683e-06 2024-09-18T07:38:09.168 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.168 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.168 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.168 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193658+0000", 2024-09-18T07:38:09.168 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6619999999999998e-06 2024-09-18T07:38:09.168 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.168 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.168 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.168 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193667+0000", 2024-09-18T07:38:09.168 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0389999999999993e-06 2024-09-18T07:38:09.168 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.168 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.169 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.169 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193671+0000", 2024-09-18T07:38:09.169 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6789999999999996e-06 2024-09-18T07:38:09.169 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.169 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.169 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.169 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193715+0000", 2024-09-18T07:38:09.169 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3186000000000003e-05 2024-09-18T07:38:09.169 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.169 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.169 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:38:09.169 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193799+0000", 2024-09-18T07:38:09.169 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4172999999999995e-05 2024-09-18T07:38:09.170 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.170 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.170 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:38:09.170 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.194037+0000", 2024-09-18T07:38:09.170 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023850299999999999 2024-09-18T07:38:09.170 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.170 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.170 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:38:09.170 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.195050+0000", 2024-09-18T07:38:09.170 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010123269999999999 2024-09-18T07:38:09.170 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.170 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.170 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.170 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.198824+0000", 2024-09-18T07:38:09.171 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0037742840000000001 2024-09-18T07:38:09.171 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.171 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.171 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.171 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238955+0000", 2024-09-18T07:38:09.171 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.040130714999999997 2024-09-18T07:38:09.171 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.171 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.171 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.171 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238967+0000", 2024-09-18T07:38:09.171 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2665e-05 2024-09-18T07:38:09.171 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.171 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.171 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.172 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254201+0000", 2024-09-18T07:38:09.172 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0152341279999999 2024-09-18T07:38:09.172 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.172 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.172 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.172 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254225+0000", 2024-09-18T07:38:09.172 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4006e-05 2024-09-18T07:38:09.172 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.172 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.172 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.172 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.172 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.172 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153453 7.7 7.146f22e7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:38:09.173 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.282355+0000", 2024-09-18T07:38:09.173 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 206.777170904, 2024-09-18T07:38:09.173 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97169607199999997, 2024-09-18T07:38:09.173 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.173 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.173 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:38:09.173 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.173 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.173 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.173 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153453 2024-09-18T07:38:09.173 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.173 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.174 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.174 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.174 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282355+0000", 2024-09-18T07:38:09.174 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.174 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.174 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.174 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.174 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282355+0000", 2024-09-18T07:38:09.174 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.174 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.174 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.174 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.174 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282357+0000", 2024-09-18T07:38:09.174 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2110000000000001e-06 2024-09-18T07:38:09.175 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.175 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.175 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.175 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282363+0000", 2024-09-18T07:38:09.175 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6740000000000002e-06 2024-09-18T07:38:09.175 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.175 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.175 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.175 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282377+0000", 2024-09-18T07:38:09.175 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4039000000000001e-05 2024-09-18T07:38:09.175 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.175 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.175 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.176 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282384+0000", 2024-09-18T07:38:09.176 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5769999999999999e-06 2024-09-18T07:38:09.176 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.176 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.176 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.176 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282438+0000", 2024-09-18T07:38:09.176 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4083999999999999e-05 2024-09-18T07:38:09.176 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.176 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.176 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.176 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282445+0000", 2024-09-18T07:38:09.176 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7030000000000006e-06 2024-09-18T07:38:09.177 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.177 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.177 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.177 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254044+0000", 2024-09-18T07:38:09.177 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97159860200000003 2024-09-18T07:38:09.177 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.177 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.177 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.177 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254051+0000", 2024-09-18T07:38:09.177 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.182e-06 2024-09-18T07:38:09.177 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.177 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.178 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.178 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.178 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.178 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153481 7.7 7.33bf19e7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:38:09.178 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.317589+0000", 2024-09-18T07:38:09.178 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 206.74193690000001, 2024-09-18T07:38:09.178 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93650330299999995, 2024-09-18T07:38:09.178 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.178 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.178 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:38:09.178 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.178 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.178 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.178 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153481 2024-09-18T07:38:09.179 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.179 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.179 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.179 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.179 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317589+0000", 2024-09-18T07:38:09.179 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.179 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.179 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.179 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.179 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317589+0000", 2024-09-18T07:38:09.179 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.179 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.179 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.180 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.180 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317590+0000", 2024-09-18T07:38:09.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6499999999999999e-07 2024-09-18T07:38:09.180 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.180 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.180 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.180 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317593+0000", 2024-09-18T07:38:09.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7870000000000002e-06 2024-09-18T07:38:09.180 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.180 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.180 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.180 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317604+0000", 2024-09-18T07:38:09.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0953e-05 2024-09-18T07:38:09.181 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.181 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.181 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.181 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317609+0000", 2024-09-18T07:38:09.181 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4099999999999999e-06 2024-09-18T07:38:09.181 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.181 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.181 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.181 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317654+0000", 2024-09-18T07:38:09.181 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5333000000000001e-05 2024-09-18T07:38:09.181 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.182 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.182 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.182 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317661+0000", 2024-09-18T07:38:09.182 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8310000000000003e-06 2024-09-18T07:38:09.182 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.182 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.182 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.182 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254083+0000", 2024-09-18T07:38:09.182 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93642186500000002 2024-09-18T07:38:09.182 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.182 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.182 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.182 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254093+0000", 2024-09-18T07:38:09.182 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4590000000000004e-06 2024-09-18T07:38:09.183 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.183 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.183 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.183 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.183 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.183 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153485 7.7 7.c6dac07 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:38:09.183 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.328201+0000", 2024-09-18T07:38:09.183 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 206.73132496299999, 2024-09-18T07:38:09.183 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.925907068, 2024-09-18T07:38:09.183 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.183 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.183 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:38:09.183 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.184 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.184 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.184 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153485 2024-09-18T07:38:09.184 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.184 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.184 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.184 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.184 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328201+0000", 2024-09-18T07:38:09.184 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.184 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.184 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.184 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.184 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328201+0000", 2024-09-18T07:38:09.185 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.185 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.185 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.185 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328203+0000", 2024-09-18T07:38:09.185 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3769999999999999e-06 2024-09-18T07:38:09.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.185 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.185 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.185 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328207+0000", 2024-09-18T07:38:09.185 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5120000000000002e-06 2024-09-18T07:38:09.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.185 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.186 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.186 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328217+0000", 2024-09-18T07:38:09.186 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0438000000000001e-05 2024-09-18T07:38:09.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.186 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.186 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.186 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328223+0000", 2024-09-18T07:38:09.186 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7130000000000002e-06 2024-09-18T07:38:09.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.186 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.186 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.186 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328270+0000", 2024-09-18T07:38:09.186 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7243000000000002e-05 2024-09-18T07:38:09.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.187 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.187 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.187 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328277+0000", 2024-09-18T07:38:09.187 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9229999999999996e-06 2024-09-18T07:38:09.187 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.187 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.187 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.187 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254104+0000", 2024-09-18T07:38:09.187 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92582690099999998 2024-09-18T07:38:09.187 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.187 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.187 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.187 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254108+0000", 2024-09-18T07:38:09.188 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9609999999999999e-06 2024-09-18T07:38:09.188 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.188 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.188 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.188 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.188 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.188 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153505 7.7 7.fb8b6b67 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:38:09.188 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.350178+0000", 2024-09-18T07:38:09.188 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 206.70934813400001, 2024-09-18T07:38:09.188 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90398860000000003, 2024-09-18T07:38:09.188 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.188 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.188 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:38:09.188 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.189 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.189 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.189 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153505 2024-09-18T07:38:09.189 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.189 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.189 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.189 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.189 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350178+0000", 2024-09-18T07:38:09.189 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.189 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.189 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.189 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.189 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350178+0000", 2024-09-18T07:38:09.190 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.190 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.190 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.190 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.190 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350179+0000", 2024-09-18T07:38:09.190 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.454e-06 2024-09-18T07:38:09.190 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.191 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.191 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.191 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350183+0000", 2024-09-18T07:38:09.192 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8480000000000003e-06 2024-09-18T07:38:09.192 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.192 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.192 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.192 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350192+0000", 2024-09-18T07:38:09.192 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0899999999999994e-06 2024-09-18T07:38:09.192 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.192 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.192 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.192 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350200+0000", 2024-09-18T07:38:09.192 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.328e-06 2024-09-18T07:38:09.192 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.192 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.192 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.192 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350252+0000", 2024-09-18T07:38:09.193 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2589000000000001e-05 2024-09-18T07:38:09.193 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.193 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.193 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.193 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350263+0000", 2024-09-18T07:38:09.193 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0831e-05 2024-09-18T07:38:09.193 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.193 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.193 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.193 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254124+0000", 2024-09-18T07:38:09.193 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90386114100000003 2024-09-18T07:38:09.193 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.193 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.193 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.194 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254167+0000", 2024-09-18T07:38:09.194 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2318999999999999e-05 2024-09-18T07:38:09.194 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.194 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.194 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.194 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.194 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.194 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153519 7.17 7.5ef16c17 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:38:09.194 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.363819+0000", 2024-09-18T07:38:09.194 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 206.69570738100001, 2024-09-18T07:38:09.194 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.89046919899999999, 2024-09-18T07:38:09.194 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.194 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.194 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:38:09.195 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.195 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.195 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.195 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153519 2024-09-18T07:38:09.195 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.195 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.195 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.195 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.195 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363819+0000", 2024-09-18T07:38:09.195 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.195 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.195 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.195 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.195 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363819+0000", 2024-09-18T07:38:09.195 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.196 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.196 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.196 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.196 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363820+0000", 2024-09-18T07:38:09.196 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.325e-06 2024-09-18T07:38:09.196 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.196 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.196 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.196 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363824+0000", 2024-09-18T07:38:09.196 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2459999999999997e-06 2024-09-18T07:38:09.196 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.196 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.197 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.197 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363834+0000", 2024-09-18T07:38:09.197 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6399999999999992e-06 2024-09-18T07:38:09.197 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.197 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.197 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.197 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363838+0000", 2024-09-18T07:38:09.197 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2509999999999996e-06 2024-09-18T07:38:09.197 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.197 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.197 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.197 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363900+0000", 2024-09-18T07:38:09.197 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1902000000000005e-05 2024-09-18T07:38:09.197 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.198 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.198 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.198 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363910+0000", 2024-09-18T07:38:09.198 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6879999999999994e-06 2024-09-18T07:38:09.198 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.198 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.198 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.198 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254254+0000", 2024-09-18T07:38:09.198 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.89034411800000002 2024-09-18T07:38:09.198 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.198 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.198 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.198 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254288+0000", 2024-09-18T07:38:09.199 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4029000000000002e-05 2024-09-18T07:38:09.199 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.199 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.199 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.199 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.199 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.199 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:162338 7.c 7:39946bc9:::benchmark_data_smithi137_74180_object57922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:38:09.199 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:01.426143+0000", 2024-09-18T07:38:09.199 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 187.63338357000001, 2024-09-18T07:38:09.199 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0750540070000001, 2024-09-18T07:38:09.199 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.199 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.200 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:38:09.200 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.200 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.200 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.200 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 162338 2024-09-18T07:38:09.200 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.200 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.200 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.200 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.200 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426143+0000", 2024-09-18T07:38:09.200 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.200 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.200 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.201 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.201 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426143+0000", 2024-09-18T07:38:09.201 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.201 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.201 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.201 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426145+0000", 2024-09-18T07:38:09.201 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4059999999999999e-06 2024-09-18T07:38:09.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.201 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.201 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.201 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426151+0000", 2024-09-18T07:38:09.202 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1260000000000003e-06 2024-09-18T07:38:09.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.202 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.202 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.202 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426174+0000", 2024-09-18T07:38:09.202 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3255e-05 2024-09-18T07:38:09.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.202 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.202 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.202 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426182+0000", 2024-09-18T07:38:09.202 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4189999999999996e-06 2024-09-18T07:38:09.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.202 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.202 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.202 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426256+0000", 2024-09-18T07:38:09.203 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4713000000000001e-05 2024-09-18T07:38:09.203 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.203 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.203 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.203 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426270+0000", 2024-09-18T07:38:09.203 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3405e-05 2024-09-18T07:38:09.203 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.203 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.203 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.203 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.491167+0000", 2024-09-18T07:38:09.203 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.064897582 2024-09-18T07:38:09.203 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.203 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.203 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:38:09.203 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.491280+0000", 2024-09-18T07:38:09.204 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011243099999999999 2024-09-18T07:38:09.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.204 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.204 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3,7", 2024-09-18T07:38:09.204 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.491357+0000", 2024-09-18T07:38:09.204 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7385999999999994e-05 2024-09-18T07:38:09.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.204 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.204 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.204 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496060+0000", 2024-09-18T07:38:09.204 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0047024600000000003 2024-09-18T07:38:09.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.204 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.204 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:38:09.204 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496072+0000", 2024-09-18T07:38:09.205 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2294e-05 2024-09-18T07:38:09.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.205 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.205 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.205 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496109+0000", 2024-09-18T07:38:09.205 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7218999999999997e-05 2024-09-18T07:38:09.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.205 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.205 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.205 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501071+0000", 2024-09-18T07:38:09.205 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0049614660000000003 2024-09-18T07:38:09.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.205 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.205 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:38:09.206 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501111+0000", 2024-09-18T07:38:09.206 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0776999999999999e-05 2024-09-18T07:38:09.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.206 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.206 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.206 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501197+0000", 2024-09-18T07:38:09.206 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5068000000000006e-05 2024-09-18T07:38:09.206 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.206 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.206 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.206 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.206 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:162350 7.c 7:3843094b:::benchmark_data_smithi137_74180_object57934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:38:09.207 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:01.433525+0000", 2024-09-18T07:38:09.207 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 187.62600076000001, 2024-09-18T07:38:09.207 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0678760330000001, 2024-09-18T07:38:09.207 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:38:09.207 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:38:09.207 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:38:09.207 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:38:09.207 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:38:09.207 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:38:09.207 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 162350 2024-09-18T07:38:09.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.207 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:38:09.207 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.207 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:38:09.208 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433525+0000", 2024-09-18T07:38:09.208 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.208 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.208 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.208 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:38:09.208 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433525+0000", 2024-09-18T07:38:09.208 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:38:09.208 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.208 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.208 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:38:09.208 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433527+0000", 2024-09-18T07:38:09.208 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.812e-06 2024-09-18T07:38:09.208 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.209 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.209 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:38:09.209 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433533+0000", 2024-09-18T07:38:09.209 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2920000000000001e-06 2024-09-18T07:38:09.209 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.209 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.209 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:38:09.209 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433544+0000", 2024-09-18T07:38:09.210 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0339999999999999e-05 2024-09-18T07:38:09.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.210 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.210 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:38:09.210 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433551+0000", 2024-09-18T07:38:09.210 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6490000000000008e-06 2024-09-18T07:38:09.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.211 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433620+0000", 2024-09-18T07:38:09.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8957999999999999e-05 2024-09-18T07:38:09.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.211 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:38:09.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433631+0000", 2024-09-18T07:38:09.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0397000000000001e-05 2024-09-18T07:38:09.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.211 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:38:09.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.493805+0000", 2024-09-18T07:38:09.212 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0601744559999999 2024-09-18T07:38:09.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.212 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:38:09.212 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.493907+0000", 2024-09-18T07:38:09.212 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000101377 2024-09-18T07:38:09.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.212 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3,7", 2024-09-18T07:38:09.212 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.493992+0000", 2024-09-18T07:38:09.212 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5672000000000002e-05 2024-09-18T07:38:09.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.213 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496146+0000", 2024-09-18T07:38:09.213 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021537470000000001 2024-09-18T07:38:09.213 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.213 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.213 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:38:09.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.498397+0000", 2024-09-18T07:38:09.213 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002250742 2024-09-18T07:38:09.213 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.213 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.213 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501029+0000", 2024-09-18T07:38:09.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026321780000000002 2024-09-18T07:38:09.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.214 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:09.214 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501274+0000", 2024-09-18T07:38:09.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000245472 2024-09-18T07:38:09.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.214 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:38:09.214 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501321+0000", 2024-09-18T07:38:09.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6405e-05 2024-09-18T07:38:09.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:38:09.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:38:09.215 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:38:09.215 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501401+0000", 2024-09-18T07:38:09.215 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0536000000000003e-05 2024-09-18T07:38:09.215 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.215 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.215 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.215 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:38:09.215 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:38:09.215 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:38:09.215 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:08 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:09.216 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:08 smithi203 ceph-mon[32237]: pgmap v379: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.51k op/s; 5481/231963 objects degraded (2.363%); 4211/231963 objects misplaced (1.815%); 12 MiB/s, 191 objects/s recovering 2024-09-18T07:38:09.216 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:08 smithi203 ceph-mon[32237]: osdmap e249: 8 total, 8 up, 8 in 2024-09-18T07:38:09.216 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:08 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1620964786' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:38:09.216 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:08 smithi203 ceph-mon[32237]: from='client.35372 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:38:09.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:09.241 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:08 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:09.241 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:08 smithi137 ceph-mon[39175]: pgmap v379: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.51k op/s; 5481/231963 objects degraded (2.363%); 4211/231963 objects misplaced (1.815%); 12 MiB/s, 191 objects/s recovering 2024-09-18T07:38:09.241 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:08 smithi137 ceph-mon[39175]: osdmap e249: 8 total, 8 up, 8 in 2024-09-18T07:38:09.241 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:08 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1620964786' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:38:09.241 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:08 smithi137 ceph-mon[39175]: from='client.35372 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:38:09.241 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:08 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:09.241 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:08 smithi137 ceph-mon[29408]: pgmap v379: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.51k op/s; 5481/231963 objects degraded (2.363%); 4211/231963 objects misplaced (1.815%); 12 MiB/s, 191 objects/s recovering 2024-09-18T07:38:09.241 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:08 smithi137 ceph-mon[29408]: osdmap e249: 8 total, 8 up, 8 in 2024-09-18T07:38:09.241 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:08 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1620964786' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:38:09.242 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:08 smithi137 ceph-mon[29408]: from='client.35372 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:38:09.273 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:38:09.273 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:38:09.274 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_1 2024-09-18T07:38:09.274 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:38:09.302 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:09.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:09.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:09.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:09.444 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:09.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:09.545 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:09.578 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:09.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:09.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:09.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:09.721 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:09.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:09.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:09.824 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:09.860 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:09.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:09.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:09.962 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:09.995 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:09.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:10.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:10.149 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:10.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:10.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:10.204 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:10.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:09 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:38:10.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:09 smithi203 ceph-mon[32237]: from='client.35372 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:38:10.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:09 smithi203 ceph-mon[32237]: osdmap e250: 8 total, 8 up, 8 in 2024-09-18T07:38:10.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:09 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 5481/231963 objects degraded (2.363%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:38:10.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:09 smithi203 ceph-mon[32237]: osdmap e251: 8 total, 8 up, 8 in 2024-09-18T07:38:10.251 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:10.278 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T07:38:10.284 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:10.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:10.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:10.354 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:09 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:38:10.354 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:09 smithi137 ceph-mon[29408]: from='client.35372 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:38:10.354 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:09 smithi137 ceph-mon[29408]: osdmap e250: 8 total, 8 up, 8 in 2024-09-18T07:38:10.354 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:09 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 5481/231963 objects degraded (2.363%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:38:10.354 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:09 smithi137 ceph-mon[29408]: osdmap e251: 8 total, 8 up, 8 in 2024-09-18T07:38:10.354 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:09 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:38:10.354 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:09 smithi137 ceph-mon[39175]: from='client.35372 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:38:10.354 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:09 smithi137 ceph-mon[39175]: osdmap e250: 8 total, 8 up, 8 in 2024-09-18T07:38:10.355 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:09 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 5481/231963 objects degraded (2.363%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:38:10.355 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:09 smithi137 ceph-mon[39175]: osdmap e251: 8 total, 8 up, 8 in 2024-09-18T07:38:10.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:10.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:10.420 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:10.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:10.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:10.552 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:10.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:10.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:10.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:10.684 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:10.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:10.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:10.722 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:38:10.788 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:10.842 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:10.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:10.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:10.944 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:10.982 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:10.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:11.024 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:11.085 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:11.126 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:11.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:11.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:11.166 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:10 smithi137 ceph-mon[29408]: pgmap v382: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 865 B/s rd, 0 B/s wr, 844 op/s; 5481/231963 objects degraded (2.363%); 4211/231963 objects misplaced (1.815%); 8.3 MiB/s, 132 objects/s recovering 2024-09-18T07:38:11.167 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:10 smithi137 ceph-mon[39175]: pgmap v382: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 865 B/s rd, 0 B/s wr, 844 op/s; 5481/231963 objects degraded (2.363%); 4211/231963 objects misplaced (1.815%); 8.3 MiB/s, 132 objects/s recovering 2024-09-18T07:38:11.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:10 smithi203 ceph-mon[32237]: pgmap v382: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 865 B/s rd, 0 B/s wr, 844 op/s; 5481/231963 objects degraded (2.363%); 4211/231963 objects misplaced (1.815%); 8.3 MiB/s, 132 objects/s recovering 2024-09-18T07:38:11.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:11.276 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:11.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:11.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:11.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:11.410 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:11.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:11.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:11.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:11.551 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:11.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:11.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:38:11.649 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:38:11.653 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:11.684 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:11.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:11.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:11.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:11.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:11.809 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:11.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:11.911 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:11.951 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:11.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:11.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:12.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:12.092 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:12.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:12.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:12.135 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:38:12.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:11 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:38:12.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:11 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:38:12.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:11 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:38:12.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:11 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:38:12.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:11 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:38:12.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:11 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:38:12.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:12.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:11 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:38:12.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:11 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:38:12.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:11 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:38:12.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:12.246 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:12.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:12.347 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:12.375 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:12.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:12.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:12.444 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-09-18T07:38:11.548685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5013183027,"num_objects":76686,"num_object_clones":0,"num_object_copies":230058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5481,"num_objects_misplaced":3807,"num_objects_unfound":0,"num_objects_dirty":76686,"num_whiteouts":0,"num_read":1013,"num_read_kb":872,"num_write":133678,"num_write_kb":6709322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15328,"num_bytes_recovered":989143083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84396,"ondisk_log_size":84396,"up":444,"acting":439,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":449,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17716664,"kb_used_data":15667748,"kb_used_omap":12,"kb_used_meta":2048563,"kb_avail":732244552,"statfs":{"total":767960285184,"available":749818421248,"internally_reserved":0,"allocated":16043773952,"data_stored":16037943937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2097729072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,5,6,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-368705536,"num_objects":-5626,"num_object_clones":0,"num_object_copies":-16878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1189,"num_objects_unfound":0,"num_objects_dirty":-5626,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":5626,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1189,"num_bytes_recovered":77922304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.628419"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":403,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.924499+0000","last_change":"2024-09-18T07:35:28.534496+0000","last_active":"2024-09-18T07:38:05.924499+0000","last_peered":"2024-09-18T07:38:05.924499+0000","last_clean":"2024-09-18T07:38:05.924499+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:38:05.924499+0000","last_undegraded":"2024-09-18T07:38:05.924499+0000","last_fullsized":"2024-09-18T07:38:05.924499+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:28.534391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:28.534391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":473,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.915644+0000","last_change":"2024-09-18T07:37:43.103765+0000","last_active":"2024-09-18T07:38:09.915644+0000","last_peered":"2024-09-18T07:38:09.915644+0000","last_clean":"2024-09-18T07:38:09.915644+0000","last_became_active":"2024-09-18T07:37:43.061125+0000","last_became_peered":"2024-09-18T07:37:43.061125+0000","last_unstale":"2024-09-18T07:38:09.915644+0000","last_undegraded":"2024-09-18T07:38:09.915644+0000","last_fullsized":"2024-09-18T07:38:09.915644+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:35:27.883281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:27.883281+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":344,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.931167+0000","last_change":"2024-09-18T07:36:38.570405+0000","last_active":"2024-09-18T07:38:05.931167+0000","last_peered":"2024-09-18T07:38:05.931167+0000","last_clean":"2024-09-18T07:38:05.931167+0000","last_became_active":"2024-09-18T07:36:13.128805+0000","last_became_peered":"2024-09-18T07:36:13.128805+0000","last_unstale":"2024-09-18T07:38:05.931167+0000","last_undegraded":"2024-09-18T07:38:05.931167+0000","last_fullsized":"2024-09-18T07:38:05.931167+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:38.570339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:38.570339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:06.790521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":390,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.914959+0000","last_change":"2024-09-18T07:32:03.953881+0000","last_active":"2024-09-18T07:38:09.914959+0000","last_peered":"2024-09-18T07:38:09.914959+0000","last_clean":"2024-09-18T07:38:09.914959+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:38:09.914959+0000","last_undegraded":"2024-09-18T07:38:09.914959+0000","last_fullsized":"2024-09-18T07:38:09.914959+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.953827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.953827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":297,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.923437+0000","last_change":"2024-09-18T07:37:43.056617+0000","last_active":"2024-09-18T07:38:05.923437+0000","last_peered":"2024-09-18T07:38:05.923437+0000","last_clean":"2024-09-18T07:38:05.923437+0000","last_became_active":"2024-09-18T07:37:43.056380+0000","last_became_peered":"2024-09-18T07:37:43.056380+0000","last_unstale":"2024-09-18T07:38:05.923437+0000","last_undegraded":"2024-09-18T07:38:05.923437+0000","last_fullsized":"2024-09-18T07:38:05.923437+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:01.481514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:01.481514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":410,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.936970+0000","last_change":"2024-09-18T07:37:43.066814+0000","last_active":"2024-09-18T07:38:05.936970+0000","last_peered":"2024-09-18T07:38:05.936970+0000","last_clean":"2024-09-18T07:38:05.936970+0000","last_became_active":"2024-09-18T07:37:43.066508+0000","last_became_peered":"2024-09-18T07:37:43.066508+0000","last_unstale":"2024-09-18T07:38:05.936970+0000","last_undegraded":"2024-09-18T07:38:05.936970+0000","last_fullsized":"2024-09-18T07:38:05.936970+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.006488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.006488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":416,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.938370+0000","last_change":"2024-09-18T07:33:22.032321+0000","last_active":"2024-09-18T07:38:05.938370+0000","last_peered":"2024-09-18T07:38:05.938370+0000","last_clean":"2024-09-18T07:38:05.938370+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:38:05.938370+0000","last_undegraded":"2024-09-18T07:38:05.938370+0000","last_fullsized":"2024-09-18T07:38:05.938370+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:22.032110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.032110+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":416,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.915555+0000","last_change":"2024-09-18T07:37:43.052675+0000","last_active":"2024-09-18T07:38:09.915555+0000","last_peered":"2024-09-18T07:38:09.915555+0000","last_clean":"2024-09-18T07:38:09.915555+0000","last_became_active":"2024-09-18T07:37:43.052346+0000","last_became_peered":"2024-09-18T07:37:43.052346+0000","last_unstale":"2024-09-18T07:38:09.915555+0000","last_undegraded":"2024-09-18T07:38:09.915555+0000","last_fullsized":"2024-09-18T07:38:09.915555+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:30.995282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:30.995282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":416,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.930069+0000","last_change":"2024-09-18T07:37:43.080879+0000","last_active":"2024-09-18T07:38:05.930069+0000","last_peered":"2024-09-18T07:38:05.930069+0000","last_clean":"2024-09-18T07:38:05.930069+0000","last_became_active":"2024-09-18T07:37:43.080314+0000","last_became_peered":"2024-09-18T07:37:43.080314+0000","last_unstale":"2024-09-18T07:38:05.930069+0000","last_undegraded":"2024-09-18T07:38:05.930069+0000","last_fullsized":"2024-09-18T07:38:05.930069+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:00.392466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.392466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":414,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.915203+0000","last_change":"2024-09-18T07:32:02.002919+0000","last_active":"2024-09-18T07:38:09.915203+0000","last_peered":"2024-09-18T07:38:09.915203+0000","last_clean":"2024-09-18T07:38:09.915203+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:38:09.915203+0000","last_undegraded":"2024-09-18T07:38:09.915203+0000","last_fullsized":"2024-09-18T07:38:09.915203+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.002811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.002811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":400,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.931477+0000","last_change":"2024-09-18T07:35:04.267623+0000","last_active":"2024-09-18T07:38:05.931477+0000","last_peered":"2024-09-18T07:38:05.931477+0000","last_clean":"2024-09-18T07:38:05.931477+0000","last_became_active":"2024-09-18T07:35:01.413443+0000","last_became_peered":"2024-09-18T07:35:01.413443+0000","last_unstale":"2024-09-18T07:38:05.931477+0000","last_undegraded":"2024-09-18T07:38:05.931477+0000","last_fullsized":"2024-09-18T07:38:05.931477+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:04.267580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:04.267580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":396,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.923266+0000","last_change":"2024-09-18T07:36:10.551960+0000","last_active":"2024-09-18T07:38:05.923266+0000","last_peered":"2024-09-18T07:38:05.923266+0000","last_clean":"2024-09-18T07:38:05.923266+0000","last_became_active":"2024-09-18T07:35:01.410284+0000","last_became_peered":"2024-09-18T07:35:01.410284+0000","last_unstale":"2024-09-18T07:38:05.923266+0000","last_undegraded":"2024-09-18T07:38:05.923266+0000","last_fullsized":"2024-09-18T07:38:05.923266+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:10.551889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:10.551889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":386,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.929792+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:38:05.929792+0000","last_peered":"2024-09-18T07:38:05.929792+0000","last_clean":"2024-09-18T07:38:05.929792+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:38:05.929792+0000","last_undegraded":"2024-09-18T07:38:05.929792+0000","last_fullsized":"2024-09-18T07:38:05.929792+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":421,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.916200+0000","last_change":"2024-09-18T07:36:05.903851+0000","last_active":"2024-09-18T07:38:09.916200+0000","last_peered":"2024-09-18T07:38:09.916200+0000","last_clean":"2024-09-18T07:38:09.916200+0000","last_became_active":"2024-09-18T07:36:02.526212+0000","last_became_peered":"2024-09-18T07:36:02.526212+0000","last_unstale":"2024-09-18T07:38:09.916200+0000","last_undegraded":"2024-09-18T07:38:09.916200+0000","last_fullsized":"2024-09-18T07:38:09.916200+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-18T07:36:05.903784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.903784+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":169,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.922317+0000","last_change":"2024-09-18T07:35:01.399995+0000","last_active":"2024-09-18T07:38:09.922317+0000","last_peered":"2024-09-18T07:38:09.922317+0000","last_clean":"2024-09-18T07:38:09.922317+0000","last_became_active":"2024-09-18T07:35:01.398605+0000","last_became_peered":"2024-09-18T07:35:01.398605+0000","last_unstale":"2024-09-18T07:38:09.922317+0000","last_undegraded":"2024-09-18T07:38:09.922317+0000","last_fullsized":"2024-09-18T07:38:09.922317+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:55.843864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:55.843864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":398,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.931250+0000","last_change":"2024-09-18T07:35:35.248892+0000","last_active":"2024-09-18T07:38:05.931250+0000","last_peered":"2024-09-18T07:38:05.931250+0000","last_clean":"2024-09-18T07:38:05.931250+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:38:05.931250+0000","last_undegraded":"2024-09-18T07:38:05.931250+0000","last_fullsized":"2024-09-18T07:38:05.931250+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:35.248824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:35.248824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":413,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.915424+0000","last_change":"2024-09-18T07:37:43.067972+0000","last_active":"2024-09-18T07:38:09.915424+0000","last_peered":"2024-09-18T07:38:09.915424+0000","last_clean":"2024-09-18T07:38:09.915424+0000","last_became_active":"2024-09-18T07:37:43.067568+0000","last_became_peered":"2024-09-18T07:37:43.067568+0000","last_unstale":"2024-09-18T07:38:09.915424+0000","last_undegraded":"2024-09-18T07:38:09.915424+0000","last_fullsized":"2024-09-18T07:38:09.915424+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:24.929124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:24.929124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":416,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.923918+0000","last_change":"2024-09-18T07:36:06.523357+0000","last_active":"2024-09-18T07:38:05.923918+0000","last_peered":"2024-09-18T07:38:05.923918+0000","last_clean":"2024-09-18T07:38:05.923918+0000","last_became_active":"2024-09-18T07:35:01.414088+0000","last_became_peered":"2024-09-18T07:35:01.414088+0000","last_unstale":"2024-09-18T07:38:05.923918+0000","last_undegraded":"2024-09-18T07:38:05.923918+0000","last_fullsized":"2024-09-18T07:38:05.923918+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:06.523292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.523292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":463,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.930747+0000","last_change":"2024-09-18T07:37:28.072530+0000","last_active":"2024-09-18T07:38:05.930747+0000","last_peered":"2024-09-18T07:38:05.930747+0000","last_clean":"2024-09-18T07:38:05.930747+0000","last_became_active":"2024-09-18T07:37:28.002858+0000","last_became_peered":"2024-09-18T07:37:28.002858+0000","last_unstale":"2024-09-18T07:38:05.930747+0000","last_undegraded":"2024-09-18T07:38:05.930747+0000","last_fullsized":"2024-09-18T07:38:05.930747+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:35:31.858040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:31.858040+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:57.917053+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":93,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.917852+0000","last_change":"2024-09-18T07:36:13.116882+0000","last_active":"2024-09-18T07:38:09.917852+0000","last_peered":"2024-09-18T07:38:09.917852+0000","last_clean":"2024-09-18T07:38:09.917852+0000","last_became_active":"2024-09-18T07:36:13.116416+0000","last_became_peered":"2024-09-18T07:36:13.116416+0000","last_unstale":"2024-09-18T07:38:09.917852+0000","last_undegraded":"2024-09-18T07:38:09.917852+0000","last_fullsized":"2024-09-18T07:38:09.917852+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:10.074367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:10.074367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":406,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.944812+0000","last_change":"2024-09-18T07:37:43.067816+0000","last_active":"2024-09-18T07:38:05.944812+0000","last_peered":"2024-09-18T07:38:05.944812+0000","last_clean":"2024-09-18T07:38:05.944812+0000","last_became_active":"2024-09-18T07:37:43.067493+0000","last_became_peered":"2024-09-18T07:37:43.067493+0000","last_unstale":"2024-09-18T07:38:05.944812+0000","last_undegraded":"2024-09-18T07:38:05.944812+0000","last_fullsized":"2024-09-18T07:38:05.944812+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:34.992350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:34.992350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":410,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.915890+0000","last_change":"2024-09-18T07:35:37.941957+0000","last_active":"2024-09-18T07:38:09.915890+0000","last_peered":"2024-09-18T07:38:09.915890+0000","last_clean":"2024-09-18T07:38:09.915890+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:38:09.915890+0000","last_undegraded":"2024-09-18T07:38:09.915890+0000","last_fullsized":"2024-09-18T07:38:09.915890+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:37.941908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:37.941908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":58,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.928937+0000","last_change":"2024-09-18T07:37:43.756858+0000","last_active":"2024-09-18T07:38:05.928937+0000","last_peered":"2024-09-18T07:38:05.928937+0000","last_clean":"2024-09-18T07:38:05.928937+0000","last_became_active":"2024-09-18T07:37:27.981705+0000","last_became_peered":"2024-09-18T07:37:27.981705+0000","last_unstale":"2024-09-18T07:38:05.928937+0000","last_undegraded":"2024-09-18T07:38:05.928937+0000","last_fullsized":"2024-09-18T07:38:05.928937+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:37:43.756403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:37:43.756403+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:47.892762+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":386,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.915899+0000","last_change":"2024-09-18T07:31:00.334313+0000","last_active":"2024-09-18T07:38:09.915899+0000","last_peered":"2024-09-18T07:38:09.915899+0000","last_clean":"2024-09-18T07:38:09.915899+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:38:09.915899+0000","last_undegraded":"2024-09-18T07:38:09.915899+0000","last_fullsized":"2024-09-18T07:38:09.915899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":181,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.922571+0000","last_change":"2024-09-18T07:35:30.254797+0000","last_active":"2024-09-18T07:38:09.922571+0000","last_peered":"2024-09-18T07:38:09.922571+0000","last_clean":"2024-09-18T07:38:09.922571+0000","last_became_active":"2024-09-18T07:35:01.413110+0000","last_became_peered":"2024-09-18T07:35:01.413110+0000","last_unstale":"2024-09-18T07:38:09.922571+0000","last_undegraded":"2024-09-18T07:38:09.922571+0000","last_fullsized":"2024-09-18T07:38:09.922571+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:35:30.254725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:30.254725+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":96,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.938075+0000","last_change":"2024-09-18T07:36:13.116657+0000","last_active":"2024-09-18T07:38:05.938075+0000","last_peered":"2024-09-18T07:38:05.938075+0000","last_clean":"2024-09-18T07:38:05.938075+0000","last_became_active":"2024-09-18T07:36:13.116295+0000","last_became_peered":"2024-09-18T07:36:13.116295+0000","last_unstale":"2024-09-18T07:38:05.938075+0000","last_undegraded":"2024-09-18T07:38:05.938075+0000","last_fullsized":"2024-09-18T07:38:05.938075+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:32.976129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.976129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":452,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.931931+0000","last_change":"2024-09-18T07:35:57.488019+0000","last_active":"2024-09-18T07:38:05.931931+0000","last_peered":"2024-09-18T07:38:05.931931+0000","last_clean":"2024-09-18T07:38:05.931931+0000","last_became_active":"2024-09-18T07:35:01.410926+0000","last_became_peered":"2024-09-18T07:35:01.410926+0000","last_unstale":"2024-09-18T07:38:05.931931+0000","last_undegraded":"2024-09-18T07:38:05.931931+0000","last_fullsized":"2024-09-18T07:38:05.931931+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:35:57.487960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.487960+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"240'12","reported_seq":39,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.931719+0000","last_change":"2024-09-18T07:37:43.220303+0000","last_active":"2024-09-18T07:38:05.931719+0000","last_peered":"2024-09-18T07:38:05.931719+0000","last_clean":"2024-09-18T07:38:05.931719+0000","last_became_active":"2024-09-18T07:37:43.060715+0000","last_became_peered":"2024-09-18T07:37:43.060715+0000","last_unstale":"2024-09-18T07:38:05.931719+0000","last_undegraded":"2024-09-18T07:38:05.931719+0000","last_fullsized":"2024-09-18T07:38:05.931719+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:35:29.400137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:29.400137+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"248'8254","reported_seq":14861,"reported_epoch":248,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T07:38:06.646929+0000","last_change":"2024-09-18T07:37:43.240841+0000","last_active":"2024-09-18T07:38:06.646929+0000","last_peered":"2024-09-18T07:38:06.646929+0000","last_clean":"2024-09-18T07:36:40.992666+0000","last_became_active":"2024-09-18T07:37:43.231704+0000","last_became_peered":"2024-09-18T07:37:43.231704+0000","last_unstale":"2024-09-18T07:38:06.646929+0000","last_undegraded":"2024-09-18T07:37:43.163542+0000","last_fullsized":"2024-09-18T07:37:43.046004+0000","mapping_epoch":238,"log_start":"214'3114","ondisk_log_start":"214'3114","created":201,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5140,"log_dups_size":0,"ondisk_log_size":5140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315228160,"num_objects":4810,"num_object_clones":0,"num_object_copies":14430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2713,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8254,"num_write_kb":418048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3241,"num_bytes_recovered":212336640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,2,5","objects":2432},{"shards":"1,5","objects":2378}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":132,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.945047+0000","last_change":"2024-09-18T07:36:13.120127+0000","last_active":"2024-09-18T07:38:05.945047+0000","last_peered":"2024-09-18T07:38:05.945047+0000","last_clean":"2024-09-18T07:38:05.945047+0000","last_became_active":"2024-09-18T07:36:13.119098+0000","last_became_peered":"2024-09-18T07:36:13.119098+0000","last_unstale":"2024-09-18T07:38:05.945047+0000","last_undegraded":"2024-09-18T07:38:05.945047+0000","last_fullsized":"2024-09-18T07:38:05.945047+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:34:37.067162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:37.067162+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":426,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.939080+0000","last_change":"2024-09-18T07:36:01.988099+0000","last_active":"2024-09-18T07:38:05.939080+0000","last_peered":"2024-09-18T07:38:05.939080+0000","last_clean":"2024-09-18T07:38:05.939080+0000","last_became_active":"2024-09-18T07:35:01.407339+0000","last_became_peered":"2024-09-18T07:35:01.407339+0000","last_unstale":"2024-09-18T07:38:05.939080+0000","last_undegraded":"2024-09-18T07:38:05.939080+0000","last_fullsized":"2024-09-18T07:38:05.939080+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:01.988054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.988054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":507,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.915542+0000","last_change":"2024-09-18T07:36:35.210213+0000","last_active":"2024-09-18T07:38:09.915542+0000","last_peered":"2024-09-18T07:38:09.915542+0000","last_clean":"2024-09-18T07:38:09.915542+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:38:09.915542+0000","last_undegraded":"2024-09-18T07:38:09.915542+0000","last_fullsized":"2024-09-18T07:38:09.915542+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:35.209823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:35.209823+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":391,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.929856+0000","last_change":"2024-09-18T07:36:07.588483+0000","last_active":"2024-09-18T07:38:05.929856+0000","last_peered":"2024-09-18T07:38:05.929856+0000","last_clean":"2024-09-18T07:38:05.929856+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:38:05.929856+0000","last_undegraded":"2024-09-18T07:38:05.929856+0000","last_fullsized":"2024-09-18T07:38:05.929856+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:07.588440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:07.588440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"248'8097","reported_seq":8202,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:07.303813+0000","last_change":"2024-09-18T07:35:59.286428+0000","last_active":"2024-09-18T07:38:07.303813+0000","last_peered":"2024-09-18T07:38:07.303813+0000","last_clean":"2024-09-18T07:38:07.303813+0000","last_became_active":"2024-09-18T07:35:59.286120+0000","last_became_peered":"2024-09-18T07:35:59.286120+0000","last_unstale":"2024-09-18T07:38:07.303813+0000","last_undegraded":"2024-09-18T07:38:07.303813+0000","last_fullsized":"2024-09-18T07:38:07.303813+0000","mapping_epoch":201,"log_start":"213'2622","ondisk_log_start":"213'2622","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5475,"log_dups_size":0,"ondisk_log_size":5475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309788672,"num_objects":4727,"num_object_clones":0,"num_object_copies":14181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8097,"num_write_kb":410368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":482,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.938919+0000","last_change":"2024-09-18T07:35:23.983838+0000","last_active":"2024-09-18T07:38:05.938919+0000","last_peered":"2024-09-18T07:38:05.938919+0000","last_clean":"2024-09-18T07:38:05.938919+0000","last_became_active":"2024-09-18T07:35:01.418909+0000","last_became_peered":"2024-09-18T07:35:01.418909+0000","last_unstale":"2024-09-18T07:38:05.938919+0000","last_undegraded":"2024-09-18T07:38:05.938919+0000","last_fullsized":"2024-09-18T07:38:05.938919+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:35:23.983762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:23.983762+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":418,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.945376+0000","last_change":"2024-09-18T07:35:27.922013+0000","last_active":"2024-09-18T07:38:05.945376+0000","last_peered":"2024-09-18T07:38:05.945376+0000","last_clean":"2024-09-18T07:38:05.945376+0000","last_became_active":"2024-09-18T07:35:01.407422+0000","last_became_peered":"2024-09-18T07:35:01.407422+0000","last_unstale":"2024-09-18T07:38:05.945376+0000","last_undegraded":"2024-09-18T07:38:05.945376+0000","last_fullsized":"2024-09-18T07:38:05.945376+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:27.921887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:27.921887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":417,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.915038+0000","last_change":"2024-09-18T07:36:30.242479+0000","last_active":"2024-09-18T07:38:09.915038+0000","last_peered":"2024-09-18T07:38:09.915038+0000","last_clean":"2024-09-18T07:38:09.915038+0000","last_became_active":"2024-09-18T07:35:01.401024+0000","last_became_peered":"2024-09-18T07:35:01.401024+0000","last_unstale":"2024-09-18T07:38:09.915038+0000","last_undegraded":"2024-09-18T07:38:09.915038+0000","last_fullsized":"2024-09-18T07:38:09.915038+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:30.242420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:30.242420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":383,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.915128+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:38:09.915128+0000","last_peered":"2024-09-18T07:38:09.915128+0000","last_clean":"2024-09-18T07:38:09.915128+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:38:09.915128+0000","last_undegraded":"2024-09-18T07:38:09.915128+0000","last_fullsized":"2024-09-18T07:38:09.915128+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"248'8110","reported_seq":8216,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:06.644894+0000","last_change":"2024-09-18T07:35:59.287014+0000","last_active":"2024-09-18T07:38:06.644894+0000","last_peered":"2024-09-18T07:38:06.644894+0000","last_clean":"2024-09-18T07:38:06.644894+0000","last_became_active":"2024-09-18T07:35:59.286688+0000","last_became_peered":"2024-09-18T07:35:59.286688+0000","last_unstale":"2024-09-18T07:38:06.644894+0000","last_undegraded":"2024-09-18T07:38:06.644894+0000","last_fullsized":"2024-09-18T07:38:06.644894+0000","mapping_epoch":201,"log_start":"214'2943","ondisk_log_start":"214'2943","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5167,"log_dups_size":0,"ondisk_log_size":5167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315555864,"num_objects":4816,"num_object_clones":0,"num_object_copies":14448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4816,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8110,"num_write_kb":413569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":466,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.915126+0000","last_change":"2024-09-18T07:37:43.153068+0000","last_active":"2024-09-18T07:38:09.915126+0000","last_peered":"2024-09-18T07:38:09.915126+0000","last_clean":"2024-09-18T07:38:09.915126+0000","last_became_active":"2024-09-18T07:37:43.096236+0000","last_became_peered":"2024-09-18T07:37:43.096236+0000","last_unstale":"2024-09-18T07:38:09.915126+0000","last_undegraded":"2024-09-18T07:38:09.915126+0000","last_fullsized":"2024-09-18T07:38:09.915126+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-18T07:35:00.882191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.882191+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":396,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.932629+0000","last_change":"2024-09-18T07:33:24.213432+0000","last_active":"2024-09-18T07:38:05.932629+0000","last_peered":"2024-09-18T07:38:05.932629+0000","last_clean":"2024-09-18T07:38:05.932629+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:38:05.932629+0000","last_undegraded":"2024-09-18T07:38:05.932629+0000","last_fullsized":"2024-09-18T07:38:05.932629+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:24.213356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:24.213356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":491,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.938644+0000","last_change":"2024-09-18T07:36:05.016008+0000","last_active":"2024-09-18T07:38:05.938644+0000","last_peered":"2024-09-18T07:38:05.938644+0000","last_clean":"2024-09-18T07:38:05.938644+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:38:05.938644+0000","last_undegraded":"2024-09-18T07:38:05.938644+0000","last_fullsized":"2024-09-18T07:38:05.938644+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:05.015955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.015955+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":191,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.922651+0000","last_change":"2024-09-18T07:37:43.057439+0000","last_active":"2024-09-18T07:38:09.922651+0000","last_peered":"2024-09-18T07:38:09.922651+0000","last_clean":"2024-09-18T07:38:09.922651+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:38:09.922651+0000","last_undegraded":"2024-09-18T07:38:09.922651+0000","last_fullsized":"2024-09-18T07:38:09.922651+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:25.287352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:25.287352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"248'8258","reported_seq":8363,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:07.280567+0000","last_change":"2024-09-18T07:35:59.102118+0000","last_active":"2024-09-18T07:38:07.280567+0000","last_peered":"2024-09-18T07:38:07.280567+0000","last_clean":"2024-09-18T07:38:07.280567+0000","last_became_active":"2024-09-18T07:35:59.101681+0000","last_became_peered":"2024-09-18T07:35:59.101681+0000","last_unstale":"2024-09-18T07:38:07.280567+0000","last_undegraded":"2024-09-18T07:38:07.280567+0000","last_fullsized":"2024-09-18T07:38:07.280567+0000","mapping_epoch":201,"log_start":"213'2711","ondisk_log_start":"213'2711","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5547,"log_dups_size":0,"ondisk_log_size":5547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304218112,"num_objects":4642,"num_object_clones":0,"num_object_copies":13926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8258,"num_write_kb":412800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":122,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.929849+0000","last_change":"2024-09-18T07:36:13.119192+0000","last_active":"2024-09-18T07:38:05.929849+0000","last_peered":"2024-09-18T07:38:05.929849+0000","last_clean":"2024-09-18T07:38:05.929849+0000","last_became_active":"2024-09-18T07:36:13.118876+0000","last_became_peered":"2024-09-18T07:36:13.118876+0000","last_unstale":"2024-09-18T07:38:05.929849+0000","last_undegraded":"2024-09-18T07:38:05.929849+0000","last_fullsized":"2024-09-18T07:38:05.929849+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":403,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.930151+0000","last_change":"2024-09-18T07:35:24.210457+0000","last_active":"2024-09-18T07:38:05.930151+0000","last_peered":"2024-09-18T07:38:05.930151+0000","last_clean":"2024-09-18T07:38:05.930151+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:38:05.930151+0000","last_undegraded":"2024-09-18T07:38:05.930151+0000","last_fullsized":"2024-09-18T07:38:05.930151+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:24.210401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:24.210401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"240'10","reported_seq":512,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.923634+0000","last_change":"2024-09-18T07:37:43.170266+0000","last_active":"2024-09-18T07:38:05.923634+0000","last_peered":"2024-09-18T07:38:05.923634+0000","last_clean":"2024-09-18T07:38:05.923634+0000","last_became_active":"2024-09-18T07:37:43.167850+0000","last_became_peered":"2024-09-18T07:37:43.167850+0000","last_unstale":"2024-09-18T07:38:05.923634+0000","last_undegraded":"2024-09-18T07:38:05.923634+0000","last_fullsized":"2024-09-18T07:38:05.923634+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:05.525762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.525762+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":375,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.923643+0000","last_change":"2024-09-18T07:31:02.348538+0000","last_active":"2024-09-18T07:38:05.923643+0000","last_peered":"2024-09-18T07:38:05.923643+0000","last_clean":"2024-09-18T07:38:05.923643+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:38:05.923643+0000","last_undegraded":"2024-09-18T07:38:05.923643+0000","last_fullsized":"2024-09-18T07:38:05.923643+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"248'8322","reported_seq":8427,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:06.912824+0000","last_change":"2024-09-18T07:35:59.102330+0000","last_active":"2024-09-18T07:38:06.912824+0000","last_peered":"2024-09-18T07:38:06.912824+0000","last_clean":"2024-09-18T07:38:06.912824+0000","last_became_active":"2024-09-18T07:35:59.101668+0000","last_became_peered":"2024-09-18T07:35:59.101668+0000","last_unstale":"2024-09-18T07:38:06.912824+0000","last_undegraded":"2024-09-18T07:38:06.912824+0000","last_fullsized":"2024-09-18T07:38:06.912824+0000","mapping_epoch":201,"log_start":"215'3300","ondisk_log_start":"215'3300","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5022,"log_dups_size":0,"ondisk_log_size":5022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315097088,"num_objects":4808,"num_object_clones":0,"num_object_copies":14424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8322,"num_write_kb":420160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":485,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.923728+0000","last_change":"2024-09-18T07:36:09.498239+0000","last_active":"2024-09-18T07:38:05.923728+0000","last_peered":"2024-09-18T07:38:05.923728+0000","last_clean":"2024-09-18T07:38:05.923728+0000","last_became_active":"2024-09-18T07:35:01.415275+0000","last_became_peered":"2024-09-18T07:35:01.415275+0000","last_unstale":"2024-09-18T07:38:05.923728+0000","last_undegraded":"2024-09-18T07:38:05.923728+0000","last_fullsized":"2024-09-18T07:38:05.923728+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:36:09.498142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.498142+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":95,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.930829+0000","last_change":"2024-09-18T07:37:46.831436+0000","last_active":"2024-09-18T07:38:05.930829+0000","last_peered":"2024-09-18T07:38:05.930829+0000","last_clean":"2024-09-18T07:38:05.930829+0000","last_became_active":"2024-09-18T07:36:13.118553+0000","last_became_peered":"2024-09-18T07:36:13.118553+0000","last_unstale":"2024-09-18T07:38:05.930829+0000","last_undegraded":"2024-09-18T07:38:05.930829+0000","last_fullsized":"2024-09-18T07:38:05.930829+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:46.831373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:37:46.831373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:13.442615+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":30,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.931012+0000","last_change":"2024-09-18T07:37:43.070306+0000","last_active":"2024-09-18T07:38:05.931012+0000","last_peered":"2024-09-18T07:38:05.931012+0000","last_clean":"2024-09-18T07:38:05.931012+0000","last_became_active":"2024-09-18T07:37:43.070009+0000","last_became_peered":"2024-09-18T07:37:43.070009+0000","last_unstale":"2024-09-18T07:38:05.931012+0000","last_undegraded":"2024-09-18T07:38:05.931012+0000","last_fullsized":"2024-09-18T07:38:05.931012+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":30,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.931997+0000","last_change":"2024-09-18T07:37:43.052077+0000","last_active":"2024-09-18T07:38:05.931997+0000","last_peered":"2024-09-18T07:38:05.931997+0000","last_clean":"2024-09-18T07:38:05.931997+0000","last_became_active":"2024-09-18T07:37:43.051740+0000","last_became_peered":"2024-09-18T07:37:43.051740+0000","last_unstale":"2024-09-18T07:38:05.931997+0000","last_undegraded":"2024-09-18T07:38:05.931997+0000","last_fullsized":"2024-09-18T07:38:05.931997+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"248'8428","reported_seq":8533,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:06.646702+0000","last_change":"2024-09-18T07:35:59.288477+0000","last_active":"2024-09-18T07:38:06.646702+0000","last_peered":"2024-09-18T07:38:06.646702+0000","last_clean":"2024-09-18T07:38:06.646702+0000","last_became_active":"2024-09-18T07:35:59.288058+0000","last_became_peered":"2024-09-18T07:35:59.288058+0000","last_unstale":"2024-09-18T07:38:06.646702+0000","last_undegraded":"2024-09-18T07:38:06.646702+0000","last_fullsized":"2024-09-18T07:38:06.646702+0000","mapping_epoch":201,"log_start":"215'3335","ondisk_log_start":"215'3335","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5093,"log_dups_size":0,"ondisk_log_size":5093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321519616,"num_objects":4906,"num_object_clones":0,"num_object_copies":14718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8428,"num_write_kb":426688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"211'29","reported_seq":180,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.930696+0000","last_change":"2024-09-18T07:36:13.115630+0000","last_active":"2024-09-18T07:38:05.930696+0000","last_peered":"2024-09-18T07:38:05.930696+0000","last_clean":"2024-09-18T07:38:05.930696+0000","last_became_active":"2024-09-18T07:36:13.115278+0000","last_became_peered":"2024-09-18T07:36:13.115278+0000","last_unstale":"2024-09-18T07:38:05.930696+0000","last_undegraded":"2024-09-18T07:38:05.930696+0000","last_fullsized":"2024-09-18T07:38:05.930696+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'28","last_scrub_stamp":"2024-09-18T07:34:31.955336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.955336+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:54.943493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":417,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.916614+0000","last_change":"2024-09-18T07:37:43.061414+0000","last_active":"2024-09-18T07:38:09.916614+0000","last_peered":"2024-09-18T07:38:09.916614+0000","last_clean":"2024-09-18T07:38:09.916614+0000","last_became_active":"2024-09-18T07:37:43.060930+0000","last_became_peered":"2024-09-18T07:37:43.060930+0000","last_unstale":"2024-09-18T07:38:09.916614+0000","last_undegraded":"2024-09-18T07:38:09.916614+0000","last_fullsized":"2024-09-18T07:38:09.916614+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:06.908713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.908713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":435,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.930336+0000","last_change":"2024-09-18T07:37:43.155837+0000","last_active":"2024-09-18T07:38:05.930336+0000","last_peered":"2024-09-18T07:38:05.930336+0000","last_clean":"2024-09-18T07:38:05.930336+0000","last_became_active":"2024-09-18T07:37:43.076595+0000","last_became_peered":"2024-09-18T07:37:43.076595+0000","last_unstale":"2024-09-18T07:38:05.930336+0000","last_undegraded":"2024-09-18T07:38:05.930336+0000","last_fullsized":"2024-09-18T07:38:05.930336+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:36:01.529677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.529677+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":379,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.923605+0000","last_change":"2024-09-18T07:31:00.332738+0000","last_active":"2024-09-18T07:38:05.923605+0000","last_peered":"2024-09-18T07:38:05.923605+0000","last_clean":"2024-09-18T07:38:05.923605+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:38:05.923605+0000","last_undegraded":"2024-09-18T07:38:05.923605+0000","last_fullsized":"2024-09-18T07:38:05.923605+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":398,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.936755+0000","last_change":"2024-09-18T07:35:01.419122+0000","last_active":"2024-09-18T07:38:05.936755+0000","last_peered":"2024-09-18T07:38:05.936755+0000","last_clean":"2024-09-18T07:38:05.936755+0000","last_became_active":"2024-09-18T07:35:01.418809+0000","last_became_peered":"2024-09-18T07:35:01.418809+0000","last_unstale":"2024-09-18T07:38:05.936755+0000","last_undegraded":"2024-09-18T07:38:05.936755+0000","last_fullsized":"2024-09-18T07:38:05.936755+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:56.289176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:34:56.289176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"248'8271","reported_seq":8376,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:06.933028+0000","last_change":"2024-09-18T07:35:59.103129+0000","last_active":"2024-09-18T07:38:06.933028+0000","last_peered":"2024-09-18T07:38:06.933028+0000","last_clean":"2024-09-18T07:38:06.933028+0000","last_became_active":"2024-09-18T07:35:59.102853+0000","last_became_peered":"2024-09-18T07:35:59.102853+0000","last_unstale":"2024-09-18T07:38:06.933028+0000","last_undegraded":"2024-09-18T07:38:06.933028+0000","last_fullsized":"2024-09-18T07:38:06.933028+0000","mapping_epoch":201,"log_start":"215'3200","ondisk_log_start":"215'3200","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5071,"log_dups_size":0,"ondisk_log_size":5071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318046208,"num_objects":4853,"num_object_clones":0,"num_object_copies":14559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8271,"num_write_kb":419968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":134,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.930395+0000","last_change":"2024-09-18T07:36:13.118278+0000","last_active":"2024-09-18T07:38:05.930395+0000","last_peered":"2024-09-18T07:38:05.930395+0000","last_clean":"2024-09-18T07:38:05.930395+0000","last_became_active":"2024-09-18T07:36:13.117924+0000","last_became_peered":"2024-09-18T07:36:13.117924+0000","last_unstale":"2024-09-18T07:38:05.930395+0000","last_undegraded":"2024-09-18T07:38:05.930395+0000","last_fullsized":"2024-09-18T07:38:05.930395+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-18T07:36:06.083531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.083531+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":101,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.928740+0000","last_change":"2024-09-18T07:37:43.051318+0000","last_active":"2024-09-18T07:38:05.928740+0000","last_peered":"2024-09-18T07:38:05.928740+0000","last_clean":"2024-09-18T07:38:05.928740+0000","last_became_active":"2024-09-18T07:37:43.050971+0000","last_became_peered":"2024-09-18T07:37:43.050971+0000","last_unstale":"2024-09-18T07:38:05.928740+0000","last_undegraded":"2024-09-18T07:38:05.928740+0000","last_fullsized":"2024-09-18T07:38:05.928740+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:04.050757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:04.050757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:58.893486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":414,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.929468+0000","last_change":"2024-09-18T07:37:49.895264+0000","last_active":"2024-09-18T07:38:05.929468+0000","last_peered":"2024-09-18T07:38:05.929468+0000","last_clean":"2024-09-18T07:38:05.929468+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:38:05.929468+0000","last_undegraded":"2024-09-18T07:38:05.929468+0000","last_fullsized":"2024-09-18T07:38:05.929468+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:37:49.894707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:37:49.894707+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:04.276523+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":403,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.931535+0000","last_change":"2024-09-18T07:37:47.825723+0000","last_active":"2024-09-18T07:38:05.931535+0000","last_peered":"2024-09-18T07:38:05.931535+0000","last_clean":"2024-09-18T07:38:05.931535+0000","last_became_active":"2024-09-18T07:35:01.403680+0000","last_became_peered":"2024-09-18T07:35:01.403680+0000","last_unstale":"2024-09-18T07:38:05.931535+0000","last_undegraded":"2024-09-18T07:38:05.931535+0000","last_fullsized":"2024-09-18T07:38:05.931535+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:47.825656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:37:47.825656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:17.721036+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":195,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.922745+0000","last_change":"2024-09-18T07:37:43.118570+0000","last_active":"2024-09-18T07:38:09.922745+0000","last_peered":"2024-09-18T07:38:09.922745+0000","last_clean":"2024-09-18T07:38:09.922745+0000","last_became_active":"2024-09-18T07:37:43.060019+0000","last_became_peered":"2024-09-18T07:37:43.060019+0000","last_unstale":"2024-09-18T07:38:09.922745+0000","last_undegraded":"2024-09-18T07:38:09.922745+0000","last_fullsized":"2024-09-18T07:38:09.922745+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:57.655797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.655797+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"248'8251","reported_seq":8362,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:06.754185+0000","last_change":"2024-09-18T07:36:13.122322+0000","last_active":"2024-09-18T07:38:06.754185+0000","last_peered":"2024-09-18T07:38:06.754185+0000","last_clean":"2024-09-18T07:38:06.754185+0000","last_became_active":"2024-09-18T07:36:13.119156+0000","last_became_peered":"2024-09-18T07:36:13.119156+0000","last_unstale":"2024-09-18T07:38:06.754185+0000","last_undegraded":"2024-09-18T07:38:06.754185+0000","last_fullsized":"2024-09-18T07:38:06.754185+0000","mapping_epoch":210,"log_start":"211'1913","ondisk_log_start":"211'1913","created":201,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":6338,"log_dups_size":0,"ondisk_log_size":6338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:59.858201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312672256,"num_objects":4771,"num_object_clones":0,"num_object_copies":14313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8251,"num_write_kb":416704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":422,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.932021+0000","last_change":"2024-09-18T07:33:27.040227+0000","last_active":"2024-09-18T07:38:05.932021+0000","last_peered":"2024-09-18T07:38:05.932021+0000","last_clean":"2024-09-18T07:38:05.932021+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:38:05.932021+0000","last_undegraded":"2024-09-18T07:38:05.932021+0000","last_fullsized":"2024-09-18T07:38:05.932021+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":393,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.929882+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:38:05.929882+0000","last_peered":"2024-09-18T07:38:05.929882+0000","last_clean":"2024-09-18T07:38:05.929882+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:38:05.929882+0000","last_undegraded":"2024-09-18T07:38:05.929882+0000","last_fullsized":"2024-09-18T07:38:05.929882+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":363,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.922207+0000","last_change":"2024-09-18T07:35:01.421692+0000","last_active":"2024-09-18T07:38:09.922207+0000","last_peered":"2024-09-18T07:38:09.922207+0000","last_clean":"2024-09-18T07:38:09.922207+0000","last_became_active":"2024-09-18T07:35:01.400524+0000","last_became_peered":"2024-09-18T07:35:01.400524+0000","last_unstale":"2024-09-18T07:38:09.922207+0000","last_undegraded":"2024-09-18T07:38:09.922207+0000","last_fullsized":"2024-09-18T07:38:09.922207+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":196,"num_read_kb":191,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":395,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.930475+0000","last_change":"2024-09-18T07:36:28.551293+0000","last_active":"2024-09-18T07:38:05.930475+0000","last_peered":"2024-09-18T07:38:05.930475+0000","last_clean":"2024-09-18T07:38:05.930475+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:38:05.930475+0000","last_undegraded":"2024-09-18T07:38:05.930475+0000","last_fullsized":"2024-09-18T07:38:05.930475+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:28.551238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:28.551238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":95,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.929878+0000","last_change":"2024-09-18T07:36:39.601424+0000","last_active":"2024-09-18T07:38:05.929878+0000","last_peered":"2024-09-18T07:38:05.929878+0000","last_clean":"2024-09-18T07:38:05.929878+0000","last_became_active":"2024-09-18T07:36:13.117463+0000","last_became_peered":"2024-09-18T07:36:13.117463+0000","last_unstale":"2024-09-18T07:38:05.929878+0000","last_undegraded":"2024-09-18T07:38:05.929878+0000","last_fullsized":"2024-09-18T07:38:05.929878+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:39.601356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:39.601356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:08.405863+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"248'8338","reported_seq":14041,"reported_epoch":248,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T07:38:06.612628+0000","last_change":"2024-09-18T07:37:43.240254+0000","last_active":"2024-09-18T07:38:06.612628+0000","last_peered":"2024-09-18T07:38:06.612628+0000","last_clean":"2024-09-18T07:36:40.976872+0000","last_became_active":"2024-09-18T07:37:43.232620+0000","last_became_peered":"2024-09-18T07:37:43.232620+0000","last_unstale":"2024-09-18T07:38:06.612628+0000","last_undegraded":"2024-09-18T07:37:43.089912+0000","last_fullsized":"2024-09-18T07:37:43.046102+0000","mapping_epoch":238,"log_start":"214'3167","ondisk_log_start":"214'3167","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5171,"log_dups_size":0,"ondisk_log_size":5171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320208896,"num_objects":4886,"num_object_clones":0,"num_object_copies":14658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2768,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8338,"num_write_kb":423168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2796,"num_bytes_recovered":183173120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,3,5","objects":2452},{"shards":"0,5","objects":2434}],"blocked_by":[],"up_primary":6,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"240'20","reported_seq":53,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.933815+0000","last_change":"2024-09-18T07:37:43.157718+0000","last_active":"2024-09-18T07:38:05.933815+0000","last_peered":"2024-09-18T07:38:05.933815+0000","last_clean":"2024-09-18T07:38:05.933815+0000","last_became_active":"2024-09-18T07:37:43.087496+0000","last_became_peered":"2024-09-18T07:37:43.087496+0000","last_unstale":"2024-09-18T07:38:05.933815+0000","last_undegraded":"2024-09-18T07:38:05.933815+0000","last_fullsized":"2024-09-18T07:38:05.933815+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'17","last_scrub_stamp":"2024-09-18T07:33:33.849285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.849285+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":410,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.935496+0000","last_change":"2024-09-18T07:37:43.065584+0000","last_active":"2024-09-18T07:38:05.935496+0000","last_peered":"2024-09-18T07:38:05.935496+0000","last_clean":"2024-09-18T07:38:05.935496+0000","last_became_active":"2024-09-18T07:37:43.065279+0000","last_became_peered":"2024-09-18T07:37:43.065279+0000","last_unstale":"2024-09-18T07:38:05.935496+0000","last_undegraded":"2024-09-18T07:38:05.935496+0000","last_fullsized":"2024-09-18T07:38:05.935496+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:59.616988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:59.616988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":209,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.922255+0000","last_change":"2024-09-18T07:37:43.121827+0000","last_active":"2024-09-18T07:38:09.922255+0000","last_peered":"2024-09-18T07:38:09.922255+0000","last_clean":"2024-09-18T07:38:09.922255+0000","last_became_active":"2024-09-18T07:37:43.058402+0000","last_became_peered":"2024-09-18T07:37:43.058402+0000","last_unstale":"2024-09-18T07:38:09.922255+0000","last_undegraded":"2024-09-18T07:38:09.922255+0000","last_fullsized":"2024-09-18T07:38:09.922255+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:35:27.278095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:35:27.278095+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":169,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.922667+0000","last_change":"2024-09-18T07:35:01.413545+0000","last_active":"2024-09-18T07:38:09.922667+0000","last_peered":"2024-09-18T07:38:09.922667+0000","last_clean":"2024-09-18T07:38:09.922667+0000","last_became_active":"2024-09-18T07:35:01.412217+0000","last_became_peered":"2024-09-18T07:35:01.412217+0000","last_unstale":"2024-09-18T07:38:09.922667+0000","last_undegraded":"2024-09-18T07:38:09.922667+0000","last_fullsized":"2024-09-18T07:38:09.922667+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:36.964421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:36.964421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":383,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.930263+0000","last_change":"2024-09-18T07:33:22.358792+0000","last_active":"2024-09-18T07:38:05.930263+0000","last_peered":"2024-09-18T07:38:05.930263+0000","last_clean":"2024-09-18T07:38:05.930263+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:38:05.930263+0000","last_undegraded":"2024-09-18T07:38:05.930263+0000","last_fullsized":"2024-09-18T07:38:05.930263+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.358746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.358746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"251'8605","reported_seq":8717,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:11.486212+0000","last_change":"2024-09-18T07:35:59.106988+0000","last_active":"2024-09-18T07:38:11.486212+0000","last_peered":"2024-09-18T07:38:11.486212+0000","last_clean":"2024-09-18T07:38:11.486212+0000","last_became_active":"2024-09-18T07:35:59.106659+0000","last_became_peered":"2024-09-18T07:35:59.106659+0000","last_unstale":"2024-09-18T07:38:11.486212+0000","last_undegraded":"2024-09-18T07:38:11.486212+0000","last_fullsized":"2024-09-18T07:38:11.486212+0000","mapping_epoch":201,"log_start":"217'3800","ondisk_log_start":"217'3800","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":4805,"log_dups_size":0,"ondisk_log_size":4805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298647552,"num_objects":4557,"num_object_clones":0,"num_object_copies":13671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8605,"num_write_kb":421184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":502,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.915586+0000","last_change":"2024-09-18T07:37:43.229332+0000","last_active":"2024-09-18T07:38:09.915586+0000","last_peered":"2024-09-18T07:38:09.915586+0000","last_clean":"2024-09-18T07:38:09.915586+0000","last_became_active":"2024-09-18T07:37:43.063255+0000","last_became_peered":"2024-09-18T07:37:43.063255+0000","last_unstale":"2024-09-18T07:38:09.915586+0000","last_undegraded":"2024-09-18T07:38:09.915586+0000","last_fullsized":"2024-09-18T07:38:09.915586+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:35:38.928407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:38.928407+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":425,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.915456+0000","last_change":"2024-09-18T07:36:32.292127+0000","last_active":"2024-09-18T07:38:09.915456+0000","last_peered":"2024-09-18T07:38:09.915456+0000","last_clean":"2024-09-18T07:38:09.915456+0000","last_became_active":"2024-09-18T07:35:01.401063+0000","last_became_peered":"2024-09-18T07:35:01.401063+0000","last_unstale":"2024-09-18T07:38:09.915456+0000","last_undegraded":"2024-09-18T07:38:09.915456+0000","last_fullsized":"2024-09-18T07:38:09.915456+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:32.292055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:32.292055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":409,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.930187+0000","last_change":"2024-09-18T07:37:43.076010+0000","last_active":"2024-09-18T07:38:05.930187+0000","last_peered":"2024-09-18T07:38:05.930187+0000","last_clean":"2024-09-18T07:38:05.930187+0000","last_became_active":"2024-09-18T07:37:43.075535+0000","last_became_peered":"2024-09-18T07:37:43.075535+0000","last_unstale":"2024-09-18T07:38:05.930187+0000","last_undegraded":"2024-09-18T07:38:05.930187+0000","last_fullsized":"2024-09-18T07:38:05.930187+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:31.191528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:31.191528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":406,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.935783+0000","last_change":"2024-09-18T07:37:43.066497+0000","last_active":"2024-09-18T07:38:05.935783+0000","last_peered":"2024-09-18T07:38:05.935783+0000","last_clean":"2024-09-18T07:38:05.935783+0000","last_became_active":"2024-09-18T07:37:43.066153+0000","last_became_peered":"2024-09-18T07:37:43.066153+0000","last_unstale":"2024-09-18T07:38:05.935783+0000","last_undegraded":"2024-09-18T07:38:05.935783+0000","last_fullsized":"2024-09-18T07:38:05.935783+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:38.955353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:38.955353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"248'8262","reported_seq":8373,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:07.328571+0000","last_change":"2024-09-18T07:36:13.120887+0000","last_active":"2024-09-18T07:38:07.328571+0000","last_peered":"2024-09-18T07:38:07.328571+0000","last_clean":"2024-09-18T07:38:07.328571+0000","last_became_active":"2024-09-18T07:36:13.116701+0000","last_became_peered":"2024-09-18T07:36:13.116701+0000","last_unstale":"2024-09-18T07:38:07.328571+0000","last_undegraded":"2024-09-18T07:38:07.328571+0000","last_fullsized":"2024-09-18T07:38:07.328571+0000","mapping_epoch":210,"log_start":"213'2800","ondisk_log_start":"213'2800","created":201,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5462,"log_dups_size":0,"ondisk_log_size":5462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314966016,"num_objects":4806,"num_object_clones":0,"num_object_copies":14418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8262,"num_write_kb":418176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":435,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.919696+0000","last_change":"2024-09-18T07:35:33.011021+0000","last_active":"2024-09-18T07:38:09.919696+0000","last_peered":"2024-09-18T07:38:09.919696+0000","last_clean":"2024-09-18T07:38:09.919696+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:38:09.919696+0000","last_undegraded":"2024-09-18T07:38:09.919696+0000","last_fullsized":"2024-09-18T07:38:09.919696+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:35:33.010955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:33.010955+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":30,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.931103+0000","last_change":"2024-09-18T07:37:43.062115+0000","last_active":"2024-09-18T07:38:05.931103+0000","last_peered":"2024-09-18T07:38:05.931103+0000","last_clean":"2024-09-18T07:38:05.931103+0000","last_became_active":"2024-09-18T07:37:43.061808+0000","last_became_peered":"2024-09-18T07:37:43.061808+0000","last_unstale":"2024-09-18T07:38:05.931103+0000","last_undegraded":"2024-09-18T07:38:05.931103+0000","last_fullsized":"2024-09-18T07:38:05.931103+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:35.265259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:35.265259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":93,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.915763+0000","last_change":"2024-09-18T07:36:13.116901+0000","last_active":"2024-09-18T07:38:09.915763+0000","last_peered":"2024-09-18T07:38:09.915763+0000","last_clean":"2024-09-18T07:38:09.915763+0000","last_became_active":"2024-09-18T07:36:13.116527+0000","last_became_peered":"2024-09-18T07:36:13.116527+0000","last_unstale":"2024-09-18T07:38:09.915763+0000","last_undegraded":"2024-09-18T07:38:09.915763+0000","last_fullsized":"2024-09-18T07:38:09.915763+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:05.239160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.239160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":401,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.932563+0000","last_change":"2024-09-18T07:36:38.632189+0000","last_active":"2024-09-18T07:38:05.932563+0000","last_peered":"2024-09-18T07:38:05.932563+0000","last_clean":"2024-09-18T07:38:05.932563+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:38:05.932563+0000","last_undegraded":"2024-09-18T07:38:05.932563+0000","last_fullsized":"2024-09-18T07:38:05.932563+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:38.632040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:38.632040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:05.822462+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"248'8277","reported_seq":8382,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:06.930035+0000","last_change":"2024-09-18T07:35:59.107531+0000","last_active":"2024-09-18T07:38:06.930035+0000","last_peered":"2024-09-18T07:38:06.930035+0000","last_clean":"2024-09-18T07:38:06.930035+0000","last_became_active":"2024-09-18T07:35:59.107232+0000","last_became_peered":"2024-09-18T07:35:59.107232+0000","last_unstale":"2024-09-18T07:38:06.930035+0000","last_undegraded":"2024-09-18T07:38:06.930035+0000","last_fullsized":"2024-09-18T07:38:06.930035+0000","mapping_epoch":201,"log_start":"215'3200","ondisk_log_start":"215'3200","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5077,"log_dups_size":0,"ondisk_log_size":5077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313065472,"num_objects":4777,"num_object_clones":0,"num_object_copies":14331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8277,"num_write_kb":417728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":452,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.916045+0000","last_change":"2024-09-18T07:37:43.106862+0000","last_active":"2024-09-18T07:38:09.916045+0000","last_peered":"2024-09-18T07:38:09.916045+0000","last_clean":"2024-09-18T07:38:09.916045+0000","last_became_active":"2024-09-18T07:37:43.062724+0000","last_became_peered":"2024-09-18T07:37:43.062724+0000","last_unstale":"2024-09-18T07:38:09.916045+0000","last_undegraded":"2024-09-18T07:38:09.916045+0000","last_fullsized":"2024-09-18T07:38:09.916045+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:32:05.000310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.000310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":396,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.937275+0000","last_change":"2024-09-18T07:32:06.688493+0000","last_active":"2024-09-18T07:38:05.937275+0000","last_peered":"2024-09-18T07:38:05.937275+0000","last_clean":"2024-09-18T07:38:05.937275+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:38:05.937275+0000","last_undegraded":"2024-09-18T07:38:05.937275+0000","last_fullsized":"2024-09-18T07:38:05.937275+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:06.688444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:06.688444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"173'11","reported_seq":517,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.929127+0000","last_change":"2024-09-18T07:35:01.413702+0000","last_active":"2024-09-18T07:38:05.929127+0000","last_peered":"2024-09-18T07:38:05.929127+0000","last_clean":"2024-09-18T07:38:05.929127+0000","last_became_active":"2024-09-18T07:35:01.403765+0000","last_became_peered":"2024-09-18T07:35:01.403765+0000","last_unstale":"2024-09-18T07:38:05.929127+0000","last_undegraded":"2024-09-18T07:38:05.929127+0000","last_fullsized":"2024-09-18T07:38:05.929127+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'10","last_scrub_stamp":"2024-09-18T07:33:33.000874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.000874+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:53.964476+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":399,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.915347+0000","last_change":"2024-09-18T07:36:33.251164+0000","last_active":"2024-09-18T07:38:09.915347+0000","last_peered":"2024-09-18T07:38:09.915347+0000","last_clean":"2024-09-18T07:38:09.915347+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:38:09.915347+0000","last_undegraded":"2024-09-18T07:38:09.915347+0000","last_fullsized":"2024-09-18T07:38:09.915347+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:33.251093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:33.251093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"248'8543","reported_seq":12681,"reported_epoch":248,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T07:38:06.934974+0000","last_change":"2024-09-18T07:37:43.261591+0000","last_active":"2024-09-18T07:38:06.934974+0000","last_peered":"2024-09-18T07:38:06.934974+0000","last_clean":"2024-09-18T07:36:40.998393+0000","last_became_active":"2024-09-18T07:37:43.222069+0000","last_became_peered":"2024-09-18T07:37:43.222069+0000","last_unstale":"2024-09-18T07:38:06.934974+0000","last_undegraded":"2024-09-18T07:38:06.934974+0000","last_fullsized":"2024-09-18T07:37:43.044165+0000","mapping_epoch":238,"log_start":"215'3500","ondisk_log_start":"215'3500","created":201,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322633728,"num_objects":4923,"num_object_clones":0,"num_object_copies":14769,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1600,"num_objects_unfound":0,"num_objects_dirty":4923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8543,"num_write_kb":430912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2008,"num_bytes_recovered":131596288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,2,3","objects":1600},{"shards":"1,3","objects":3323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":416,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.923682+0000","last_change":"2024-09-18T07:37:28.049996+0000","last_active":"2024-09-18T07:38:05.923682+0000","last_peered":"2024-09-18T07:38:05.923682+0000","last_clean":"2024-09-18T07:38:05.923682+0000","last_became_active":"2024-09-18T07:37:27.994144+0000","last_became_peered":"2024-09-18T07:37:27.994144+0000","last_unstale":"2024-09-18T07:38:05.923682+0000","last_undegraded":"2024-09-18T07:38:05.923682+0000","last_fullsized":"2024-09-18T07:38:05.923682+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":417,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.929980+0000","last_change":"2024-09-18T07:37:43.076416+0000","last_active":"2024-09-18T07:38:05.929980+0000","last_peered":"2024-09-18T07:38:05.929980+0000","last_clean":"2024-09-18T07:38:05.929980+0000","last_became_active":"2024-09-18T07:37:43.075988+0000","last_became_peered":"2024-09-18T07:37:43.075988+0000","last_unstale":"2024-09-18T07:38:05.929980+0000","last_undegraded":"2024-09-18T07:38:05.929980+0000","last_fullsized":"2024-09-18T07:38:05.929980+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:21.222526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.222526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":389,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.929053+0000","last_change":"2024-09-18T07:34:31.959675+0000","last_active":"2024-09-18T07:38:05.929053+0000","last_peered":"2024-09-18T07:38:05.929053+0000","last_clean":"2024-09-18T07:38:05.929053+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:38:05.929053+0000","last_undegraded":"2024-09-18T07:38:05.929053+0000","last_fullsized":"2024-09-18T07:38:05.929053+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:31.959636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.959636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:56.892556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":405,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.928368+0000","last_change":"2024-09-18T07:36:07.526609+0000","last_active":"2024-09-18T07:38:05.928368+0000","last_peered":"2024-09-18T07:38:05.928368+0000","last_clean":"2024-09-18T07:38:05.928368+0000","last_became_active":"2024-09-18T07:35:01.410890+0000","last_became_peered":"2024-09-18T07:35:01.410890+0000","last_unstale":"2024-09-18T07:38:05.928368+0000","last_undegraded":"2024-09-18T07:38:05.928368+0000","last_fullsized":"2024-09-18T07:38:05.928368+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:07.526536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:07.526536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"248'8275","reported_seq":12577,"reported_epoch":248,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T07:38:07.320721+0000","last_change":"2024-09-18T07:37:43.243985+0000","last_active":"2024-09-18T07:38:07.320721+0000","last_peered":"2024-09-18T07:38:07.320721+0000","last_clean":"2024-09-18T07:36:40.991141+0000","last_became_active":"2024-09-18T07:37:43.212542+0000","last_became_peered":"2024-09-18T07:37:43.212542+0000","last_unstale":"2024-09-18T07:38:07.320721+0000","last_undegraded":"2024-09-18T07:38:07.320721+0000","last_fullsized":"2024-09-18T07:37:43.044251+0000","mapping_epoch":238,"log_start":"213'2800","ondisk_log_start":"213'2800","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5475,"log_dups_size":0,"ondisk_log_size":5475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312803328,"num_objects":4773,"num_object_clones":0,"num_object_copies":14319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1527,"num_objects_unfound":0,"num_objects_dirty":4773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8275,"num_write_kb":417536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2095,"num_bytes_recovered":137297920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,2,7","objects":1527},{"shards":"0,7","objects":3246}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":50,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.931153+0000","last_change":"2024-09-18T07:37:43.142793+0000","last_active":"2024-09-18T07:38:05.931153+0000","last_peered":"2024-09-18T07:38:05.931153+0000","last_clean":"2024-09-18T07:38:05.931153+0000","last_became_active":"2024-09-18T07:37:43.063319+0000","last_became_peered":"2024-09-18T07:37:43.063319+0000","last_unstale":"2024-09-18T07:38:05.931153+0000","last_undegraded":"2024-09-18T07:38:05.931153+0000","last_fullsized":"2024-09-18T07:38:05.931153+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:35:21.344029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.344029+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":191,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.922904+0000","last_change":"2024-09-18T07:37:43.073257+0000","last_active":"2024-09-18T07:38:09.922904+0000","last_peered":"2024-09-18T07:38:09.922904+0000","last_clean":"2024-09-18T07:38:09.922904+0000","last_became_active":"2024-09-18T07:37:43.072894+0000","last_became_peered":"2024-09-18T07:37:43.072894+0000","last_unstale":"2024-09-18T07:38:09.922904+0000","last_undegraded":"2024-09-18T07:38:09.922904+0000","last_fullsized":"2024-09-18T07:38:09.922904+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:28.171448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:28.171448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":404,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.928784+0000","last_change":"2024-09-18T07:37:52.961389+0000","last_active":"2024-09-18T07:38:05.928784+0000","last_peered":"2024-09-18T07:38:05.928784+0000","last_clean":"2024-09-18T07:38:05.928784+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:38:05.928784+0000","last_undegraded":"2024-09-18T07:38:05.928784+0000","last_fullsized":"2024-09-18T07:38:05.928784+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:52.961325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:37:52.961325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:19.975533+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":169,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.922336+0000","last_change":"2024-09-18T07:35:01.408078+0000","last_active":"2024-09-18T07:38:09.922336+0000","last_peered":"2024-09-18T07:38:09.922336+0000","last_clean":"2024-09-18T07:38:09.922336+0000","last_became_active":"2024-09-18T07:35:01.406980+0000","last_became_peered":"2024-09-18T07:35:01.406980+0000","last_unstale":"2024-09-18T07:38:09.922336+0000","last_undegraded":"2024-09-18T07:38:09.922336+0000","last_fullsized":"2024-09-18T07:38:09.922336+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"251'8514","reported_seq":18454,"reported_epoch":251,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T07:38:11.528411+0000","last_change":"2024-09-18T07:37:43.260864+0000","last_active":"2024-09-18T07:38:11.528411+0000","last_peered":"2024-09-18T07:38:11.528411+0000","last_clean":"2024-09-18T07:36:40.973984+0000","last_became_active":"2024-09-18T07:37:43.232636+0000","last_became_peered":"2024-09-18T07:37:43.232636+0000","last_unstale":"2024-09-18T07:38:11.528411+0000","last_undegraded":"2024-09-18T07:38:11.528411+0000","last_fullsized":"2024-09-18T07:37:43.048341+0000","mapping_epoch":238,"log_start":"217'3700","ondisk_log_start":"217'3700","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297402368,"num_objects":4538,"num_object_clones":0,"num_object_copies":13614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":680,"num_objects_unfound":0,"num_objects_dirty":4538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8514,"num_write_kb":417664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4910,"num_bytes_recovered":321781760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"0,4,7","objects":680},{"shards":"4,7","objects":3858}],"blocked_by":[],"up_primary":6,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":430,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.915263+0000","last_change":"2024-09-18T07:35:21.085719+0000","last_active":"2024-09-18T07:38:09.915263+0000","last_peered":"2024-09-18T07:38:09.915263+0000","last_clean":"2024-09-18T07:38:09.915263+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:38:09.915263+0000","last_undegraded":"2024-09-18T07:38:09.915263+0000","last_fullsized":"2024-09-18T07:38:09.915263+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:35:21.085670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.085670+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":169,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.927094+0000","last_change":"2024-09-18T07:35:01.400050+0000","last_active":"2024-09-18T07:38:09.927094+0000","last_peered":"2024-09-18T07:38:09.927094+0000","last_clean":"2024-09-18T07:38:09.927094+0000","last_became_active":"2024-09-18T07:35:01.398499+0000","last_became_peered":"2024-09-18T07:35:01.398499+0000","last_unstale":"2024-09-18T07:38:09.927094+0000","last_undegraded":"2024-09-18T07:38:09.927094+0000","last_fullsized":"2024-09-18T07:38:09.927094+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":413,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.928655+0000","last_change":"2024-09-18T07:36:32.564149+0000","last_active":"2024-09-18T07:38:05.928655+0000","last_peered":"2024-09-18T07:38:05.928655+0000","last_clean":"2024-09-18T07:38:05.928655+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:38:05.928655+0000","last_undegraded":"2024-09-18T07:38:05.928655+0000","last_fullsized":"2024-09-18T07:38:05.928655+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:32.564101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:32.564101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:02.841418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":398,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.937164+0000","last_change":"2024-09-18T07:37:43.016535+0000","last_active":"2024-09-18T07:38:05.937164+0000","last_peered":"2024-09-18T07:38:05.937164+0000","last_clean":"2024-09-18T07:38:05.937164+0000","last_became_active":"2024-09-18T07:35:01.418735+0000","last_became_peered":"2024-09-18T07:35:01.418735+0000","last_unstale":"2024-09-18T07:38:05.937164+0000","last_undegraded":"2024-09-18T07:38:05.937164+0000","last_fullsized":"2024-09-18T07:38:05.937164+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:43.016489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:37:43.016489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:12.741671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"248'8324","reported_seq":8430,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:06.630794+0000","last_change":"2024-09-18T07:35:59.107480+0000","last_active":"2024-09-18T07:38:06.630794+0000","last_peered":"2024-09-18T07:38:06.630794+0000","last_clean":"2024-09-18T07:38:06.630794+0000","last_became_active":"2024-09-18T07:35:59.106549+0000","last_became_peered":"2024-09-18T07:35:59.106549+0000","last_unstale":"2024-09-18T07:38:06.630794+0000","last_undegraded":"2024-09-18T07:38:06.630794+0000","last_fullsized":"2024-09-18T07:38:06.630794+0000","mapping_epoch":201,"log_start":"215'3200","ondisk_log_start":"215'3200","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320733184,"num_objects":4894,"num_object_clones":0,"num_object_copies":14682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8324,"num_write_kb":422976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":516,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.932790+0000","last_change":"2024-09-18T07:37:43.157488+0000","last_active":"2024-09-18T07:38:05.932790+0000","last_peered":"2024-09-18T07:38:05.932790+0000","last_clean":"2024-09-18T07:38:05.932790+0000","last_became_active":"2024-09-18T07:37:43.075552+0000","last_became_peered":"2024-09-18T07:37:43.075552+0000","last_unstale":"2024-09-18T07:38:05.932790+0000","last_undegraded":"2024-09-18T07:38:05.932790+0000","last_fullsized":"2024-09-18T07:38:05.932790+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:35:36.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:36.271863+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":191,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.926886+0000","last_change":"2024-09-18T07:36:09.261574+0000","last_active":"2024-09-18T07:38:09.926886+0000","last_peered":"2024-09-18T07:38:09.926886+0000","last_clean":"2024-09-18T07:38:09.926886+0000","last_became_active":"2024-09-18T07:35:01.406695+0000","last_became_peered":"2024-09-18T07:35:01.406695+0000","last_unstale":"2024-09-18T07:38:09.926886+0000","last_undegraded":"2024-09-18T07:38:09.926886+0000","last_fullsized":"2024-09-18T07:38:09.926886+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:36:09.261516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.261516+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"240'12","reported_seq":242,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.922994+0000","last_change":"2024-09-18T07:37:43.117484+0000","last_active":"2024-09-18T07:38:09.922994+0000","last_peered":"2024-09-18T07:38:09.922994+0000","last_clean":"2024-09-18T07:38:09.922994+0000","last_became_active":"2024-09-18T07:37:43.056714+0000","last_became_peered":"2024-09-18T07:37:43.056714+0000","last_unstale":"2024-09-18T07:38:09.922994+0000","last_undegraded":"2024-09-18T07:38:09.922994+0000","last_fullsized":"2024-09-18T07:38:09.922994+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:35:26.319320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:26.319320+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":389,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.929568+0000","last_change":"2024-09-18T07:37:43.078768+0000","last_active":"2024-09-18T07:38:05.929568+0000","last_peered":"2024-09-18T07:38:05.929568+0000","last_clean":"2024-09-18T07:38:05.929568+0000","last_became_active":"2024-09-18T07:37:43.078121+0000","last_became_peered":"2024-09-18T07:37:43.078121+0000","last_unstale":"2024-09-18T07:38:05.929568+0000","last_undegraded":"2024-09-18T07:38:05.929568+0000","last_fullsized":"2024-09-18T07:38:05.929568+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.930921+0000","last_change":"2024-09-18T07:37:43.088299+0000","last_active":"2024-09-18T07:38:05.930921+0000","last_peered":"2024-09-18T07:38:05.930921+0000","last_clean":"2024-09-18T07:38:05.930921+0000","last_became_active":"2024-09-18T07:37:43.087994+0000","last_became_peered":"2024-09-18T07:37:43.087994+0000","last_unstale":"2024-09-18T07:38:05.930921+0000","last_undegraded":"2024-09-18T07:38:05.930921+0000","last_fullsized":"2024-09-18T07:38:05.930921+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:01.259296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.259296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":430,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.945139+0000","last_change":"2024-09-18T07:36:09.061983+0000","last_active":"2024-09-18T07:38:05.945139+0000","last_peered":"2024-09-18T07:38:05.945139+0000","last_clean":"2024-09-18T07:38:05.945139+0000","last_became_active":"2024-09-18T07:35:01.407787+0000","last_became_peered":"2024-09-18T07:35:01.407787+0000","last_unstale":"2024-09-18T07:38:05.945139+0000","last_undegraded":"2024-09-18T07:38:05.945139+0000","last_fullsized":"2024-09-18T07:38:05.945139+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-18T07:36:09.061903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.061903+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":394,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.930696+0000","last_change":"2024-09-18T07:36:34.608531+0000","last_active":"2024-09-18T07:38:05.930696+0000","last_peered":"2024-09-18T07:38:05.930696+0000","last_clean":"2024-09-18T07:38:05.930696+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:38:05.930696+0000","last_undegraded":"2024-09-18T07:38:05.930696+0000","last_fullsized":"2024-09-18T07:38:05.930696+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:34.608486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:34.608486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":183,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.922858+0000","last_change":"2024-09-18T07:37:43.079723+0000","last_active":"2024-09-18T07:38:09.922858+0000","last_peered":"2024-09-18T07:38:09.922858+0000","last_clean":"2024-09-18T07:38:09.922858+0000","last_became_active":"2024-09-18T07:37:43.079313+0000","last_became_peered":"2024-09-18T07:37:43.079313+0000","last_unstale":"2024-09-18T07:38:09.922858+0000","last_undegraded":"2024-09-18T07:38:09.922858+0000","last_fullsized":"2024-09-18T07:38:09.922858+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:58.688631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:58.688631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":432,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.932101+0000","last_change":"2024-09-18T07:35:01.448510+0000","last_active":"2024-09-18T07:38:05.932101+0000","last_peered":"2024-09-18T07:38:05.932101+0000","last_clean":"2024-09-18T07:38:05.932101+0000","last_became_active":"2024-09-18T07:35:01.414458+0000","last_became_peered":"2024-09-18T07:35:01.414458+0000","last_unstale":"2024-09-18T07:38:05.932101+0000","last_undegraded":"2024-09-18T07:38:05.932101+0000","last_fullsized":"2024-09-18T07:38:05.932101+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":191,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.922854+0000","last_change":"2024-09-18T07:37:43.057776+0000","last_active":"2024-09-18T07:38:09.922854+0000","last_peered":"2024-09-18T07:38:09.922854+0000","last_clean":"2024-09-18T07:38:09.922854+0000","last_became_active":"2024-09-18T07:37:43.056675+0000","last_became_peered":"2024-09-18T07:37:43.056675+0000","last_unstale":"2024-09-18T07:38:09.922854+0000","last_undegraded":"2024-09-18T07:38:09.922854+0000","last_fullsized":"2024-09-18T07:38:09.922854+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:05.201723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.201723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":409,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.944954+0000","last_change":"2024-09-18T07:35:59.997299+0000","last_active":"2024-09-18T07:38:05.944954+0000","last_peered":"2024-09-18T07:38:05.944954+0000","last_clean":"2024-09-18T07:38:05.944954+0000","last_became_active":"2024-09-18T07:35:01.420133+0000","last_became_peered":"2024-09-18T07:35:01.420133+0000","last_unstale":"2024-09-18T07:38:05.944954+0000","last_undegraded":"2024-09-18T07:38:05.944954+0000","last_fullsized":"2024-09-18T07:38:05.944954+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:59.997249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:59.997249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":399,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.929591+0000","last_change":"2024-09-18T07:37:48.861965+0000","last_active":"2024-09-18T07:38:05.929591+0000","last_peered":"2024-09-18T07:38:05.929591+0000","last_clean":"2024-09-18T07:38:05.929591+0000","last_became_active":"2024-09-18T07:35:01.396305+0000","last_became_peered":"2024-09-18T07:35:01.396305+0000","last_unstale":"2024-09-18T07:38:05.929591+0000","last_undegraded":"2024-09-18T07:38:05.929591+0000","last_fullsized":"2024-09-18T07:38:05.929591+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:48.861784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:37:48.861784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:17.251260+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":440,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.930653+0000","last_change":"2024-09-18T07:35:02.342002+0000","last_active":"2024-09-18T07:38:05.930653+0000","last_peered":"2024-09-18T07:38:05.930653+0000","last_clean":"2024-09-18T07:38:05.930653+0000","last_became_active":"2024-09-18T07:35:01.416744+0000","last_became_peered":"2024-09-18T07:35:01.416744+0000","last_unstale":"2024-09-18T07:38:05.930653+0000","last_undegraded":"2024-09-18T07:38:05.930653+0000","last_fullsized":"2024-09-18T07:38:05.930653+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:35:02.341919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.341919+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":409,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.924029+0000","last_change":"2024-09-18T07:35:01.414704+0000","last_active":"2024-09-18T07:38:05.924029+0000","last_peered":"2024-09-18T07:38:05.924029+0000","last_clean":"2024-09-18T07:38:05.924029+0000","last_became_active":"2024-09-18T07:35:01.414024+0000","last_became_peered":"2024-09-18T07:35:01.414024+0000","last_unstale":"2024-09-18T07:38:05.924029+0000","last_undegraded":"2024-09-18T07:38:05.924029+0000","last_fullsized":"2024-09-18T07:38:05.924029+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:35.707148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:35.707148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"173'10","reported_seq":505,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.936364+0000","last_change":"2024-09-18T07:36:08.041535+0000","last_active":"2024-09-18T07:38:05.936364+0000","last_peered":"2024-09-18T07:38:05.936364+0000","last_clean":"2024-09-18T07:38:05.936364+0000","last_became_active":"2024-09-18T07:35:01.407568+0000","last_became_peered":"2024-09-18T07:35:01.407568+0000","last_unstale":"2024-09-18T07:38:05.936364+0000","last_undegraded":"2024-09-18T07:38:05.936364+0000","last_fullsized":"2024-09-18T07:38:05.936364+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:36:08.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:08.041456+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":390,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.915687+0000","last_change":"2024-09-18T07:36:11.910543+0000","last_active":"2024-09-18T07:38:09.915687+0000","last_peered":"2024-09-18T07:38:09.915687+0000","last_clean":"2024-09-18T07:38:09.915687+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:38:09.915687+0000","last_undegraded":"2024-09-18T07:38:09.915687+0000","last_fullsized":"2024-09-18T07:38:09.915687+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:11.910474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:11.910474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":453,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.936505+0000","last_change":"2024-09-18T07:36:02.975929+0000","last_active":"2024-09-18T07:38:05.936505+0000","last_peered":"2024-09-18T07:38:05.936505+0000","last_clean":"2024-09-18T07:38:05.936505+0000","last_became_active":"2024-09-18T07:35:01.407738+0000","last_became_peered":"2024-09-18T07:35:01.407738+0000","last_unstale":"2024-09-18T07:38:05.936505+0000","last_undegraded":"2024-09-18T07:38:05.936505+0000","last_fullsized":"2024-09-18T07:38:05.936505+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:36:02.975863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:02.975863+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":177,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.922787+0000","last_change":"2024-09-18T07:36:08.222898+0000","last_active":"2024-09-18T07:38:09.922787+0000","last_peered":"2024-09-18T07:38:09.922787+0000","last_clean":"2024-09-18T07:38:09.922787+0000","last_became_active":"2024-09-18T07:35:01.400197+0000","last_became_peered":"2024-09-18T07:35:01.400197+0000","last_unstale":"2024-09-18T07:38:09.922787+0000","last_undegraded":"2024-09-18T07:38:09.922787+0000","last_fullsized":"2024-09-18T07:38:09.922787+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:08.222838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:08.222838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":484,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.929964+0000","last_change":"2024-09-18T07:37:42.285094+0000","last_active":"2024-09-18T07:38:05.929964+0000","last_peered":"2024-09-18T07:38:05.929964+0000","last_clean":"2024-09-18T07:38:05.929964+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:38:05.929964+0000","last_undegraded":"2024-09-18T07:38:05.929964+0000","last_fullsized":"2024-09-18T07:38:05.929964+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:37:42.285042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:37:42.285042+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:58.228525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":396,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.930499+0000","last_change":"2024-09-18T07:35:01.411256+0000","last_active":"2024-09-18T07:38:05.930499+0000","last_peered":"2024-09-18T07:38:05.930499+0000","last_clean":"2024-09-18T07:38:05.930499+0000","last_became_active":"2024-09-18T07:35:01.410561+0000","last_became_peered":"2024-09-18T07:35:01.410561+0000","last_unstale":"2024-09-18T07:38:05.930499+0000","last_undegraded":"2024-09-18T07:38:05.930499+0000","last_fullsized":"2024-09-18T07:38:05.930499+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.134847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.134847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":464,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.916295+0000","last_change":"2024-09-18T07:37:43.105077+0000","last_active":"2024-09-18T07:38:09.916295+0000","last_peered":"2024-09-18T07:38:09.916295+0000","last_clean":"2024-09-18T07:38:09.916295+0000","last_became_active":"2024-09-18T07:37:43.054073+0000","last_became_peered":"2024-09-18T07:37:43.054073+0000","last_unstale":"2024-09-18T07:38:09.916295+0000","last_undegraded":"2024-09-18T07:38:09.916295+0000","last_fullsized":"2024-09-18T07:38:09.916295+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:36:01.986964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.986964+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":423,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.916137+0000","last_change":"2024-09-18T07:37:43.062429+0000","last_active":"2024-09-18T07:38:09.916137+0000","last_peered":"2024-09-18T07:38:09.916137+0000","last_clean":"2024-09-18T07:38:09.916137+0000","last_became_active":"2024-09-18T07:37:43.061577+0000","last_became_peered":"2024-09-18T07:37:43.061577+0000","last_unstale":"2024-09-18T07:38:09.916137+0000","last_undegraded":"2024-09-18T07:38:09.916137+0000","last_fullsized":"2024-09-18T07:38:09.916137+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":87,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.929748+0000","last_change":"2024-09-18T07:36:13.118114+0000","last_active":"2024-09-18T07:38:05.929748+0000","last_peered":"2024-09-18T07:38:05.929748+0000","last_clean":"2024-09-18T07:38:05.929748+0000","last_became_active":"2024-09-18T07:36:13.117584+0000","last_became_peered":"2024-09-18T07:36:13.117584+0000","last_unstale":"2024-09-18T07:38:05.929748+0000","last_undegraded":"2024-09-18T07:38:05.929748+0000","last_fullsized":"2024-09-18T07:38:05.929748+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":376,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.937651+0000","last_change":"2024-09-18T07:31:02.351247+0000","last_active":"2024-09-18T07:38:05.937651+0000","last_peered":"2024-09-18T07:38:05.937651+0000","last_clean":"2024-09-18T07:38:05.937651+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:38:05.937651+0000","last_undegraded":"2024-09-18T07:38:05.937651+0000","last_fullsized":"2024-09-18T07:38:05.937651+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":272,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.923841+0000","last_change":"2024-09-18T07:37:43.138346+0000","last_active":"2024-09-18T07:38:05.923841+0000","last_peered":"2024-09-18T07:38:05.923841+0000","last_clean":"2024-09-18T07:38:05.923841+0000","last_became_active":"2024-09-18T07:37:43.067598+0000","last_became_peered":"2024-09-18T07:37:43.067598+0000","last_unstale":"2024-09-18T07:38:05.923841+0000","last_undegraded":"2024-09-18T07:38:05.923841+0000","last_fullsized":"2024-09-18T07:38:05.923841+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:36:38.638226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:38.638226+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":177,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.922768+0000","last_change":"2024-09-18T07:35:57.293607+0000","last_active":"2024-09-18T07:38:09.922768+0000","last_peered":"2024-09-18T07:38:09.922768+0000","last_clean":"2024-09-18T07:38:09.922768+0000","last_became_active":"2024-09-18T07:35:01.405287+0000","last_became_peered":"2024-09-18T07:35:01.405287+0000","last_unstale":"2024-09-18T07:38:09.922768+0000","last_undegraded":"2024-09-18T07:38:09.922768+0000","last_fullsized":"2024-09-18T07:38:09.922768+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:57.293555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.293555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":388,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.937198+0000","last_change":"2024-09-18T07:35:01.205647+0000","last_active":"2024-09-18T07:38:05.937198+0000","last_peered":"2024-09-18T07:38:05.937198+0000","last_clean":"2024-09-18T07:38:05.937198+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:38:05.937198+0000","last_undegraded":"2024-09-18T07:38:05.937198+0000","last_fullsized":"2024-09-18T07:38:05.937198+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:01.205596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:01.205596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":399,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.923106+0000","last_change":"2024-09-18T07:36:37.593789+0000","last_active":"2024-09-18T07:38:05.923106+0000","last_peered":"2024-09-18T07:38:05.923106+0000","last_clean":"2024-09-18T07:38:05.923106+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:38:05.923106+0000","last_undegraded":"2024-09-18T07:38:05.923106+0000","last_fullsized":"2024-09-18T07:38:05.923106+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:37.593724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:37.593724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":495,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.937917+0000","last_change":"2024-09-18T07:37:43.138620+0000","last_active":"2024-09-18T07:38:05.937917+0000","last_peered":"2024-09-18T07:38:05.937917+0000","last_clean":"2024-09-18T07:38:05.937917+0000","last_became_active":"2024-09-18T07:37:43.066208+0000","last_became_peered":"2024-09-18T07:37:43.066208+0000","last_unstale":"2024-09-18T07:38:05.937917+0000","last_undegraded":"2024-09-18T07:38:05.937917+0000","last_fullsized":"2024-09-18T07:38:05.937917+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:36:29.051437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:29.051437+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":408,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.930861+0000","last_change":"2024-09-18T07:35:01.415691+0000","last_active":"2024-09-18T07:38:05.930861+0000","last_peered":"2024-09-18T07:38:05.930861+0000","last_clean":"2024-09-18T07:38:05.930861+0000","last_became_active":"2024-09-18T07:35:01.415261+0000","last_became_peered":"2024-09-18T07:35:01.415261+0000","last_unstale":"2024-09-18T07:38:05.930861+0000","last_undegraded":"2024-09-18T07:38:05.930861+0000","last_fullsized":"2024-09-18T07:38:05.930861+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:58.734027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:58.734027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":191,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.922416+0000","last_change":"2024-09-18T07:37:43.057780+0000","last_active":"2024-09-18T07:38:09.922416+0000","last_peered":"2024-09-18T07:38:09.922416+0000","last_clean":"2024-09-18T07:38:09.922416+0000","last_became_active":"2024-09-18T07:37:43.057483+0000","last_became_peered":"2024-09-18T07:37:43.057483+0000","last_unstale":"2024-09-18T07:38:09.922416+0000","last_undegraded":"2024-09-18T07:38:09.922416+0000","last_fullsized":"2024-09-18T07:38:09.922416+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:41.311291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:41.311291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":406,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.937281+0000","last_change":"2024-09-18T07:37:43.068364+0000","last_active":"2024-09-18T07:38:05.937281+0000","last_peered":"2024-09-18T07:38:05.937281+0000","last_clean":"2024-09-18T07:38:05.937281+0000","last_became_active":"2024-09-18T07:37:43.067066+0000","last_became_peered":"2024-09-18T07:37:43.067066+0000","last_unstale":"2024-09-18T07:38:05.937281+0000","last_undegraded":"2024-09-18T07:38:05.937281+0000","last_fullsized":"2024-09-18T07:38:05.937281+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:36.026215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:36.026215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":453,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.938780+0000","last_change":"2024-09-18T07:37:43.134956+0000","last_active":"2024-09-18T07:38:05.938780+0000","last_peered":"2024-09-18T07:38:05.938780+0000","last_clean":"2024-09-18T07:38:05.938780+0000","last_became_active":"2024-09-18T07:37:43.067503+0000","last_became_peered":"2024-09-18T07:37:43.067503+0000","last_unstale":"2024-09-18T07:38:05.938780+0000","last_undegraded":"2024-09-18T07:38:05.938780+0000","last_fullsized":"2024-09-18T07:38:05.938780+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:36:40.908629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:40.908629+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":386,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.924567+0000","last_change":"2024-09-18T07:30:56.283643+0000","last_active":"2024-09-18T07:38:05.924567+0000","last_peered":"2024-09-18T07:38:05.924567+0000","last_clean":"2024-09-18T07:38:05.924567+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:38:05.924567+0000","last_undegraded":"2024-09-18T07:38:05.924567+0000","last_fullsized":"2024-09-18T07:38:05.924567+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":30,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.931899+0000","last_change":"2024-09-18T07:37:43.061403+0000","last_active":"2024-09-18T07:38:05.931899+0000","last_peered":"2024-09-18T07:38:05.931899+0000","last_clean":"2024-09-18T07:38:05.931899+0000","last_became_active":"2024-09-18T07:37:43.061081+0000","last_became_peered":"2024-09-18T07:37:43.061081+0000","last_unstale":"2024-09-18T07:38:05.931899+0000","last_undegraded":"2024-09-18T07:38:05.931899+0000","last_fullsized":"2024-09-18T07:38:05.931899+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:30.345180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:30.345180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":177,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-18T07:38:09.922346+0000","last_change":"2024-09-18T07:35:29.235047+0000","last_active":"2024-09-18T07:38:09.922346+0000","last_peered":"2024-09-18T07:38:09.922346+0000","last_clean":"2024-09-18T07:38:09.922346+0000","last_became_active":"2024-09-18T07:35:01.400396+0000","last_became_peered":"2024-09-18T07:35:01.400396+0000","last_unstale":"2024-09-18T07:38:09.922346+0000","last_undegraded":"2024-09-18T07:38:09.922346+0000","last_fullsized":"2024-09-18T07:38:09.922346+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:29.234993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:29.234993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":87,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.929961+0000","last_change":"2024-09-18T07:36:13.118009+0000","last_active":"2024-09-18T07:38:05.929961+0000","last_peered":"2024-09-18T07:38:05.929961+0000","last_clean":"2024-09-18T07:38:05.929961+0000","last_became_active":"2024-09-18T07:36:13.117154+0000","last_became_peered":"2024-09-18T07:36:13.117154+0000","last_unstale":"2024-09-18T07:38:05.929961+0000","last_undegraded":"2024-09-18T07:38:05.929961+0000","last_fullsized":"2024-09-18T07:38:05.929961+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.071629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.071629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":409,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.938992+0000","last_change":"2024-09-18T07:35:57.951851+0000","last_active":"2024-09-18T07:38:05.938992+0000","last_peered":"2024-09-18T07:38:05.938992+0000","last_clean":"2024-09-18T07:38:05.938992+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:38:05.938992+0000","last_undegraded":"2024-09-18T07:38:05.938992+0000","last_fullsized":"2024-09-18T07:38:05.938992+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:35:57.951801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.951801+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":42,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.931768+0000","last_change":"2024-09-18T07:37:43.258255+0000","last_active":"2024-09-18T07:38:05.931768+0000","last_peered":"2024-09-18T07:38:05.931768+0000","last_clean":"2024-09-18T07:38:05.931768+0000","last_became_active":"2024-09-18T07:37:43.070186+0000","last_became_peered":"2024-09-18T07:37:43.070186+0000","last_unstale":"2024-09-18T07:38:05.931768+0000","last_undegraded":"2024-09-18T07:38:05.931768+0000","last_fullsized":"2024-09-18T07:38:05.931768+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:34:58.432781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:58.432781+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":409,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-18T07:38:05.931847+0000","last_change":"2024-09-18T07:35:25.576024+0000","last_active":"2024-09-18T07:38:05.931847+0000","last_peered":"2024-09-18T07:38:05.931847+0000","last_clean":"2024-09-18T07:38:05.931847+0000","last_became_active":"2024-09-18T07:35:01.410469+0000","last_became_peered":"2024-09-18T07:35:01.410469+0000","last_unstale":"2024-09-18T07:38:05.931847+0000","last_undegraded":"2024-09-18T07:38:05.931847+0000","last_fullsized":"2024-09-18T07:38:05.931847+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:25.575954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:25.575954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5012586520,"num_objects":76487,"num_object_clones":0,"num_object_copies":229461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5481,"num_objects_misplaced":3807,"num_objects_unfound":0,"num_objects_dirty":76487,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133129,"num_write_kb":6707649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15050,"num_bytes_recovered":986185728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15939743744,"data_stored":15939731528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83824,"ondisk_log_size":83824,"up":48,"acting":43,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":4554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":398,"ondisk_log_size":398,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":196,"num_read_kb":191,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201371,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1611776,"kb_used_data":1454256,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92133376,"statfs":{"total":95995035648,"available":94344577024,"internally_reserved":0,"allocated":1489158144,"data_stored":1488410664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":161282506},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463008,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1230020,"kb_used_data":1072496,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92515132,"statfs":{"total":95995035648,"available":94735495168,"internally_reserved":0,"allocated":1098235904,"data_stored":1097530881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":161282502},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724645,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2758032,"kb_used_data":2492092,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":90987120,"statfs":{"total":95995035648,"available":93170810880,"internally_reserved":0,"allocated":2551902208,"data_stored":2551185053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953581,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1678716,"kb_used_data":1458732,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92066436,"statfs":{"total":95995035648,"available":94276030464,"internally_reserved":0,"allocated":1493741568,"data_stored":1492997449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149808,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2544224,"kb_used_data":2192264,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91200928,"statfs":{"total":95995035648,"available":93389750272,"internally_reserved":0,"allocated":2244878336,"data_stored":2244150073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":360315335},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411446,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2962744,"kb_used_data":2638632,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":90782408,"statfs":{"total":95995035648,"available":92961185792,"internally_reserved":0,"allocated":2701959168,"data_stored":2701254962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574973,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2479916,"kb_used_data":2178008,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91265236,"statfs":{"total":95995035648,"available":93455601664,"internally_reserved":0,"allocated":2230280192,"data_stored":2229535880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738498,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2451236,"kb_used_data":2181268,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":91293916,"statfs":{"total":95995035648,"available":93484969984,"internally_reserved":0,"allocated":2233618432,"data_stored":2232878975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2221473792,"data_stored":2221473792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2218721280,"data_stored":2218721280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2676166656,"data_stored":2676162584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2223439872,"data_stored":2223439872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1481244672,"data_stored":1481244672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2540441600,"data_stored":2540437528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1092681728,"data_stored":1092681728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1485574144,"data_stored":1485570072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:38:12.451 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_1 pgp_num 16 2024-09-18T07:38:12.477 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:12.507 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:12.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:12.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:12.609 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:12.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:12.643 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:12.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:12.736 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:38:12.737 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:38:12.743 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:12.748 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:12.779 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:38:12.779 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:38:12.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:12.798 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:12.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:12.813 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:38:12.814 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:38:12.845 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:38:12.846 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:38:12.880 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:38:12.881 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:38:12.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:12.913 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:38:12.913 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:38:12.942 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:12.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:12.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:12.953 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:38:12.953 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:38:12.993 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:12 smithi137 ceph-mon[29408]: pgmap v384: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.00k op/s; 5481/230058 objects degraded (2.382%); 3807/230058 objects misplaced (1.655%); 13 MiB/s, 211 objects/s recovering 2024-09-18T07:38:12.993 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:12 smithi137 ceph-mon[29408]: from='client.15510 v1:172.21.15.137:0/2881386405' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:38:12.993 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:12 smithi137 ceph-mon[29408]: osdmap e252: 8 total, 8 up, 8 in 2024-09-18T07:38:12.993 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:12 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3746732271' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:12.993 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:12 smithi137 ceph-mon[39175]: pgmap v384: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.00k op/s; 5481/230058 objects degraded (2.382%); 3807/230058 objects misplaced (1.655%); 13 MiB/s, 211 objects/s recovering 2024-09-18T07:38:12.993 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:12 smithi137 ceph-mon[39175]: from='client.15510 v1:172.21.15.137:0/2881386405' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:38:12.994 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:12 smithi137 ceph-mon[39175]: osdmap e252: 8 total, 8 up, 8 in 2024-09-18T07:38:12.994 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:12 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3746732271' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:12.994 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:38:12.994 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:38:13.044 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:13.074 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:13.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:13.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:13.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:13.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:13.210 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:13.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:13.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:13.347 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:13.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:13.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:13.448 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:13.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:12 smithi203 ceph-mon[32237]: pgmap v384: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.00k op/s; 5481/230058 objects degraded (2.382%); 3807/230058 objects misplaced (1.655%); 13 MiB/s, 211 objects/s recovering 2024-09-18T07:38:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:12 smithi203 ceph-mon[32237]: from='client.15510 v1:172.21.15.137:0/2881386405' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:38:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:12 smithi203 ceph-mon[32237]: osdmap e252: 8 total, 8 up, 8 in 2024-09-18T07:38:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:12 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3746732271' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:13.477 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:13.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:13.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:13.579 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:13.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:13.619 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:13.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:13.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:13.757 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:13.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:13.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:13.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:13.890 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:13.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:13.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:13.992 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:14.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:14.033 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:14.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:14.135 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:14.164 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:14.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:14.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:14.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:14.302 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:14.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:14.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:14.425 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:14.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:14.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:14.528 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:14.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:14.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:14.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:14.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:14.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:14.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:14.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:14.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:14.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:14.840 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:14.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:14.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:14.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:14.963 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:14.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:15.064 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:15.105 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:15.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:15.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:15.111 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:38:15.111 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:38:15.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:15.229 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:15.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:15.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:15.331 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:15.368 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:15.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:15.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:15.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:14 smithi137 ceph-mon[29408]: pgmap v386: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 912 B/s rd, 0 B/s wr, 895 op/s; 5481/216900 objects degraded (2.527%); 3352/216900 objects misplaced (1.545%); 9.6 MiB/s, 153 objects/s recovering 2024-09-18T07:38:15.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:14 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 5481/216900 objects degraded (2.527%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:38:15.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:14 smithi137 ceph-mon[29408]: osdmap e253: 8 total, 8 up, 8 in 2024-09-18T07:38:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:14 smithi137 ceph-mon[39175]: pgmap v386: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 912 B/s rd, 0 B/s wr, 895 op/s; 5481/216900 objects degraded (2.527%); 3352/216900 objects misplaced (1.545%); 9.6 MiB/s, 153 objects/s recovering 2024-09-18T07:38:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:14 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 5481/216900 objects degraded (2.527%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:38:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:14 smithi137 ceph-mon[39175]: osdmap e253: 8 total, 8 up, 8 in 2024-09-18T07:38:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:14 smithi203 ceph-mon[32237]: pgmap v386: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 912 B/s rd, 0 B/s wr, 895 op/s; 5481/216900 objects degraded (2.527%); 3352/216900 objects misplaced (1.545%); 9.6 MiB/s, 153 objects/s recovering 2024-09-18T07:38:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:14 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 5481/216900 objects degraded (2.527%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:38:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:14 smithi203 ceph-mon[32237]: osdmap e253: 8 total, 8 up, 8 in 2024-09-18T07:38:15.470 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:15.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:15.496 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:15.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:15.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:15.633 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:15.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:15.734 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:15.767 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:15.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:15.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:15.869 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:15.900 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:15.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:15.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:16.002 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:16.036 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:16.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:16.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:16.138 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:16.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:16.173 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:16.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:16.274 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:16.311 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:16.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:16.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:16.413 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:16.454 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:16.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:16.556 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:16.598 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:16.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:16.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:16.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:16.733 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T07:38:16.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:16.736 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:16.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:16.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:16.884 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:16.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:16.921 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:16.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:17.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:17.059 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:17.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:17.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:17.100 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:16 smithi137 ceph-mon[29408]: pgmap v388: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 837 op/s; 5481/216900 objects degraded (2.527%); 3352/216900 objects misplaced (1.545%); 8.9 MiB/s, 143 objects/s recovering 2024-09-18T07:38:17.100 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:16 smithi137 ceph-mon[39175]: pgmap v388: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 837 op/s; 5481/216900 objects degraded (2.527%); 3352/216900 objects misplaced (1.545%); 8.9 MiB/s, 143 objects/s recovering 2024-09-18T07:38:17.161 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:17.162 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:38:17.201 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:17.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:17.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:17.254 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:16 smithi203 ceph-mon[32237]: pgmap v388: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 837 op/s; 5481/216900 objects degraded (2.527%); 3352/216900 objects misplaced (1.545%); 8.9 MiB/s, 143 objects/s recovering 2024-09-18T07:38:17.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:17.335 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:17.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:17.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:17.352 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:38:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:38:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:38:17.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:17.472 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:17.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:17.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:17.574 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:17.607 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:17.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:17.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:17.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:38:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:38:17.710 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:17.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:17.754 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:17.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:17.855 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:17.903 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:17.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:17.997 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:38:18.003 INFO:teuthology.orchestra.run.smithi137.stderr:set pool 8 pgp_num to 16 2024-09-18T07:38:18.004 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:18.023 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:38:18.023 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:38:18.042 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:18.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:18.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:18.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:18.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:18.183 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:18.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:18.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:17 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/6423941' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:38:18.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:17 smithi203 ceph-mon[32237]: from='client.35384 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:38:18.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:17 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4184922652' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-18T07:38:18.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:17 smithi203 ceph-mon[32237]: from='client.25456 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-18T07:38:18.284 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:18.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:18.324 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:18.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:18.395 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:17 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/6423941' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:38:18.395 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:17 smithi137 ceph-mon[29408]: from='client.35384 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:38:18.395 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:17 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4184922652' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-18T07:38:18.395 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:17 smithi137 ceph-mon[29408]: from='client.25456 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-18T07:38:18.395 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:17 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/6423941' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:38:18.395 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:17 smithi137 ceph-mon[39175]: from='client.35384 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:38:18.395 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:17 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4184922652' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-18T07:38:18.395 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:17 smithi137 ceph-mon[39175]: from='client.25456 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-18T07:38:18.425 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:18.458 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:18.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:18.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:18.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:18.593 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:18.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:18.694 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:18.727 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:18.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:18.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:18.769 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T07:38:18.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:18.893 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:18.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:18.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:18.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:38:18.998 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:19.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:19.030 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:19.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:19.131 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:19.142 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:18 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:19.142 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:18 smithi137 ceph-mon[29408]: pgmap v389: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.30k op/s; 5481/202269 objects degraded (2.710%); 2311/202269 objects misplaced (1.143%); 16 MiB/s, 248 objects/s recovering 2024-09-18T07:38:19.142 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:18 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:38:19.142 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:18 smithi137 ceph-mon[29408]: from='client.35384 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:38:19.142 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:18 smithi137 ceph-mon[29408]: from='client.25456 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-18T07:38:19.143 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:18 smithi137 ceph-mon[29408]: osdmap e254: 8 total, 8 up, 8 in 2024-09-18T07:38:19.143 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:18 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:19.143 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:18 smithi137 ceph-mon[39175]: pgmap v389: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.30k op/s; 5481/202269 objects degraded (2.710%); 2311/202269 objects misplaced (1.143%); 16 MiB/s, 248 objects/s recovering 2024-09-18T07:38:19.143 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:18 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:38:19.143 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:18 smithi137 ceph-mon[39175]: from='client.35384 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:38:19.143 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:18 smithi137 ceph-mon[39175]: from='client.25456 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-18T07:38:19.143 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:18 smithi137 ceph-mon[39175]: osdmap e254: 8 total, 8 up, 8 in 2024-09-18T07:38:19.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:19.176 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:19.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:19.237 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:38:19.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:19.326 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:19.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:19.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:19.376 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:18 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:19.376 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:18 smithi203 ceph-mon[32237]: pgmap v389: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.30k op/s; 5481/202269 objects degraded (2.710%); 2311/202269 objects misplaced (1.143%); 16 MiB/s, 248 objects/s recovering 2024-09-18T07:38:19.376 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:18 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:38:19.376 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:18 smithi203 ceph-mon[32237]: from='client.35384 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:38:19.376 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:18 smithi203 ceph-mon[32237]: from='client.25456 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-18T07:38:19.376 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:18 smithi203 ceph-mon[32237]: osdmap e254: 8 total, 8 up, 8 in 2024-09-18T07:38:19.429 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:19.463 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:19.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:19.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:19.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:19.611 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:19.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:19.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:19.676 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:19.714 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:19.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:19.761 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:19.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:19.862 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:19.863 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:38:19.863 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:38:19.903 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:19.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:19.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:20.005 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:20.039 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:20.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:20.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:20.142 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:20.164 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:19 smithi137 ceph-mon[29408]: 8.a scrub starts 2024-09-18T07:38:20.165 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:19 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 5481/202269 objects degraded (2.710%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:38:20.165 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:19 smithi137 ceph-mon[29408]: osdmap e255: 8 total, 8 up, 8 in 2024-09-18T07:38:20.165 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:19 smithi137 ceph-mon[39175]: 8.a scrub starts 2024-09-18T07:38:20.165 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:19 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 5481/202269 objects degraded (2.710%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:38:20.165 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:19 smithi137 ceph-mon[39175]: osdmap e255: 8 total, 8 up, 8 in 2024-09-18T07:38:20.171 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:20.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:20.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:20.273 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:20.296 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:20.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:20.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:20.298 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:38:20.299 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:38:20.317 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:38:20.317 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:38:20.334 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:38:20.334 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:38:20.352 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:38:20.352 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:38:20.369 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:38:20.370 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:38:20.388 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:38:20.389 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:38:20.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:20.408 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:38:20.408 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:38:20.419 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:20.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:20.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:20.430 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:38:20.431 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:38:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:19 smithi203 ceph-mon[32237]: 8.a scrub starts 2024-09-18T07:38:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:19 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 5481/202269 objects degraded (2.710%), 2 pgs degraded (PG_DEGRADED) 2024-09-18T07:38:20.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:19 smithi203 ceph-mon[32237]: osdmap e255: 8 total, 8 up, 8 in 2024-09-18T07:38:20.521 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:20.545 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:20.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:20.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T07:38:20.646 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:20.669 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:20.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:20.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:20.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:20.794 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:20.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:20.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:20.850 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:38:20.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:20.918 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:20.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:20.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:21.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:21.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:21.062 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:21.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:21.163 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:21.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:20 smithi137 ceph-mon[29408]: pgmap v391: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 675 B/s rd, 0 B/s wr, 804 op/s; 5481/202269 objects degraded (2.710%); 2311/202269 objects misplaced (1.143%); 10 MiB/s, 166 objects/s recovering 2024-09-18T07:38:21.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:20 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1214966142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:21.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:20 smithi137 ceph-mon[29408]: osdmap e256: 8 total, 8 up, 8 in 2024-09-18T07:38:21.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:20 smithi137 ceph-mon[39175]: pgmap v391: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 675 B/s rd, 0 B/s wr, 804 op/s; 5481/202269 objects degraded (2.710%); 2311/202269 objects misplaced (1.143%); 10 MiB/s, 166 objects/s recovering 2024-09-18T07:38:21.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:20 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1214966142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:20 smithi137 ceph-mon[39175]: osdmap e256: 8 total, 8 up, 8 in 2024-09-18T07:38:21.190 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:21.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:21.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:21.292 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:21.323 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:21.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:21.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:21.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:21.450 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:21.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:21.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:20 smithi203 ceph-mon[32237]: pgmap v391: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 675 B/s rd, 0 B/s wr, 804 op/s; 5481/202269 objects degraded (2.710%); 2311/202269 objects misplaced (1.143%); 10 MiB/s, 166 objects/s recovering 2024-09-18T07:38:21.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:20 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1214966142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:21.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:20 smithi203 ceph-mon[32237]: osdmap e256: 8 total, 8 up, 8 in 2024-09-18T07:38:21.551 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:21.584 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:21.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:21.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:21.688 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:21.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:21.711 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:21.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:21.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:21.847 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:21.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:21.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:21.949 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:21.984 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:21.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:21.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:22.085 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:22.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:22.120 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:22.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:22.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:22.261 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:22.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:22.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:22.363 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:22.405 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:22.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:22.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:22.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:22 smithi137 ceph-mon[29408]: 8.a scrub ok 2024-09-18T07:38:22.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:22 smithi137 ceph-mon[39175]: 8.a scrub ok 2024-09-18T07:38:22.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:22 smithi203 ceph-mon[32237]: 8.a scrub ok 2024-09-18T07:38:22.507 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:22.538 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:22.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:22.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:22.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:22.675 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:22.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:22.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:22.778 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:22.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:22.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:22.912 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:22.953 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:22.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:22.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:22.993 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:23.003 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:23.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:23.091 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:23.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:23.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:23.134 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:23 smithi137 ceph-mon[39175]: pgmap v394: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 861 op/s; 5481/201414 objects degraded (2.721%); 2311/201414 objects misplaced (1.147%); 11 MiB/s, 173 objects/s recovering 2024-09-18T07:38:23.134 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:23 smithi137 ceph-mon[29408]: pgmap v394: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 861 op/s; 5481/201414 objects degraded (2.721%); 2311/201414 objects misplaced (1.147%); 11 MiB/s, 173 objects/s recovering 2024-09-18T07:38:23.194 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:23.230 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:23.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:23.332 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:23.367 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:23.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:23.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:23.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:23 smithi203 ceph-mon[32237]: pgmap v394: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 861 op/s; 5481/201414 objects degraded (2.721%); 2311/201414 objects misplaced (1.147%); 11 MiB/s, 173 objects/s recovering 2024-09-18T07:38:23.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:23.506 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:23.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:23.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:23.608 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:23.645 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:23.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:23.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:23.747 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:23.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:23.779 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:23.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:23.880 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:23.914 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:23.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:24.015 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:24.059 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:24.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:24.161 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:24.185 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:24.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:24.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:24.287 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:24.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:24.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:24.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:24.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:24.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:24.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:24.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:24.519 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:38:24.519 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:38:24.520 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-18T07:38:24.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-18T07:38:24.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:24.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:24.614 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:24.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:24.715 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:24.749 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:24.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:24.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:24.851 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:24.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:24.893 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:24.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:24.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:38:24.998 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:25.023 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:25.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:25.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:25.125 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:25.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:25.156 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:25.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:25.257 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:25.295 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:25.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:25.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:25.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:25.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:25 smithi137 ceph-mon[29408]: pgmap v395: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 730 op/s; 2713/189138 objects degraded (1.434%); 3742/189138 objects misplaced (1.978%); 11 MiB/s, 222 objects/s recovering 2024-09-18T07:38:25.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:25 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 2713/189138 objects degraded (1.434%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:38:25.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:25 smithi137 ceph-mon[29408]: osdmap e257: 8 total, 8 up, 8 in 2024-09-18T07:38:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:25 smithi137 ceph-mon[39175]: pgmap v395: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 730 op/s; 2713/189138 objects degraded (1.434%); 3742/189138 objects misplaced (1.978%); 11 MiB/s, 222 objects/s recovering 2024-09-18T07:38:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:25 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 2713/189138 objects degraded (1.434%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:38:25.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:25 smithi137 ceph-mon[39175]: osdmap e257: 8 total, 8 up, 8 in 2024-09-18T07:38:25.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:25.442 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:25.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:25.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:25 smithi203 ceph-mon[32237]: pgmap v395: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 730 op/s; 2713/189138 objects degraded (1.434%); 3742/189138 objects misplaced (1.978%); 11 MiB/s, 222 objects/s recovering 2024-09-18T07:38:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:25 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 2713/189138 objects degraded (1.434%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:38:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:25 smithi203 ceph-mon[32237]: osdmap e257: 8 total, 8 up, 8 in 2024-09-18T07:38:25.542 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:25.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:25.579 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:25.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:25.681 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:25.715 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:25.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:25.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:25.817 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:25.848 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:25.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:25.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:25.950 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:25.987 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:25.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:26.089 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:26.126 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:26.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:26.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:26.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:26.264 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:26.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:26.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:26.367 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:26.390 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:26.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:26.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:26.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:26.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:26.528 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:26.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:26.630 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:26.664 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:26.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:26.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:26.766 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:26.798 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:26.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:26.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:26.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:26.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:26.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:26.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:27.024 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:27.060 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:27.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:27.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:27.081 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:38:27.081 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:38:27.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.081 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:151355 8.6 8:68c515eb:::benchmark_data_smithi137_101942_object46545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-09-18T07:38:27.081 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:38:26.167520+0000", 2024-09-18T07:38:27.081 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.019845832000000001, 2024-09-18T07:38:27.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01987628, 2024-09-18T07:38:27.081 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:38:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:38:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:38:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:38:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:38:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:38:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151355 2024-09-18T07:38:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:38:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:38:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.167520+0000", 2024-09-18T07:38:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:27.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:38:27.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.167520+0000", 2024-09-18T07:38:27.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:27.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:38:27.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.167522+0000", 2024-09-18T07:38:27.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.606e-06 2024-09-18T07:38:27.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:38:27.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.167527+0000", 2024-09-18T07:38:27.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2290000000000002e-06 2024-09-18T07:38:27.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:38:27.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.167537+0000", 2024-09-18T07:38:27.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0288e-05 2024-09-18T07:38:27.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:38:27.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.167545+0000", 2024-09-18T07:38:27.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6599999999999995e-06 2024-09-18T07:38:27.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:38:27.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.167606+0000", 2024-09-18T07:38:27.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0571e-05 2024-09-18T07:38:27.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:38:27.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.170751+0000", 2024-09-18T07:38:27.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031452210000000001 2024-09-18T07:38:27.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-09-18T07:38:27.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.170884+0000", 2024-09-18T07:38:27.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000133626 2024-09-18T07:38:27.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:27.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.175226+0000", 2024-09-18T07:38:27.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043411509999999997 2024-09-18T07:38:27.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:27.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.183898+0000", 2024-09-18T07:38:27.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0086721969999999995 2024-09-18T07:38:27.086 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:27.086 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:27.087 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:27.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.087 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:151363 8.c 8:340dde91:::benchmark_data_smithi137_101942_object46553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-09-18T07:38:27.087 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:38:26.174438+0000", 2024-09-18T07:38:27.087 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.01292829, 2024-09-18T07:38:27.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013045409000000001, 2024-09-18T07:38:27.087 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:38:27.087 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:38:27.087 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:38:27.087 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:38:27.087 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:38:27.087 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:38:27.088 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151363 2024-09-18T07:38:27.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.088 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:38:27.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:38:27.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.174438+0000", 2024-09-18T07:38:27.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:27.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:38:27.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.174438+0000", 2024-09-18T07:38:27.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:27.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:38:27.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.174440+0000", 2024-09-18T07:38:27.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2089999999999999e-06 2024-09-18T07:38:27.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:38:27.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.174446+0000", 2024-09-18T07:38:27.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5300000000000002e-06 2024-09-18T07:38:27.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:38:27.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.174458+0000", 2024-09-18T07:38:27.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2055000000000001e-05 2024-09-18T07:38:27.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:38:27.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.174466+0000", 2024-09-18T07:38:27.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8560000000000007e-06 2024-09-18T07:38:27.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:38:27.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.174520+0000", 2024-09-18T07:38:27.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3776999999999997e-05 2024-09-18T07:38:27.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:38:27.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.174643+0000", 2024-09-18T07:38:27.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000122994 2024-09-18T07:38:27.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:38:27.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.174761+0000", 2024-09-18T07:38:27.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011754799999999999 2024-09-18T07:38:27.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:27.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178196+0000", 2024-09-18T07:38:27.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034349979999999999 2024-09-18T07:38:27.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:38:27.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.183759+0000", 2024-09-18T07:38:27.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055634259999999998 2024-09-18T07:38:27.092 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:27.092 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:27.092 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:27.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.093 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:151364 8.e 8:7e55a620:::benchmark_data_smithi137_101942_object46554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-09-18T07:38:27.093 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:38:26.178115+0000", 2024-09-18T07:38:27.093 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0092512069999999991, 2024-09-18T07:38:27.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0094454610000000005, 2024-09-18T07:38:27.093 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:38:27.093 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:38:27.093 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:38:27.093 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:38:27.093 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:38:27.093 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:38:27.093 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151364 2024-09-18T07:38:27.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.094 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:38:27.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:38:27.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178115+0000", 2024-09-18T07:38:27.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:27.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:38:27.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178115+0000", 2024-09-18T07:38:27.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:27.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:38:27.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178116+0000", 2024-09-18T07:38:27.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6479999999999999e-06 2024-09-18T07:38:27.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:38:27.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178121+0000", 2024-09-18T07:38:27.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.729e-06 2024-09-18T07:38:27.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:38:27.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178128+0000", 2024-09-18T07:38:27.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8839999999999997e-06 2024-09-18T07:38:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:38:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178133+0000", 2024-09-18T07:38:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0189999999999997e-06 2024-09-18T07:38:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:38:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.183418+0000", 2024-09-18T07:38:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052851829999999997 2024-09-18T07:38:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:38:27.097 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.183545+0000", 2024-09-18T07:38:27.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012661500000000001 2024-09-18T07:38:27.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.097 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,4", 2024-09-18T07:38:27.097 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.183743+0000", 2024-09-18T07:38:27.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019810800000000001 2024-09-18T07:38:27.097 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:27.097 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:27.097 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:27.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.097 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:151365 8.c 8:3004bc2d:::benchmark_data_smithi137_101942_object46555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-09-18T07:38:27.097 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:38:26.178140+0000", 2024-09-18T07:38:27.098 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0092256060000000008, 2024-09-18T07:38:27.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0094860719999999999, 2024-09-18T07:38:27.098 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:38:27.098 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:38:27.098 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:38:27.098 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:38:27.098 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:38:27.098 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:38:27.098 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151365 2024-09-18T07:38:27.098 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.098 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:38:27.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:38:27.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178140+0000", 2024-09-18T07:38:27.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:27.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.099 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:38:27.099 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178140+0000", 2024-09-18T07:38:27.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:27.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.099 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:38:27.099 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178141+0000", 2024-09-18T07:38:27.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3000000000000001e-07 2024-09-18T07:38:27.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:38:27.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178143+0000", 2024-09-18T07:38:27.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.894e-06 2024-09-18T07:38:27.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:38:27.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178147+0000", 2024-09-18T07:38:27.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9910000000000004e-06 2024-09-18T07:38:27.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:38:27.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178151+0000", 2024-09-18T07:38:27.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.092e-06 2024-09-18T07:38:27.101 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.101 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:38:27.101 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178670+0000", 2024-09-18T07:38:27.101 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00051913399999999998 2024-09-18T07:38:27.101 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.101 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:38:27.101 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178779+0000", 2024-09-18T07:38:27.101 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000108937 2024-09-18T07:38:27.101 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.101 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:38:27.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178946+0000", 2024-09-18T07:38:27.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016674699999999999 2024-09-18T07:38:27.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:38:27.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.183887+0000", 2024-09-18T07:38:27.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0049412869999999999 2024-09-18T07:38:27.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:27.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.185324+0000", 2024-09-18T07:38:27.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014372460000000001 2024-09-18T07:38:27.102 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:27.103 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:27.103 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:27.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.103 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:151366 8.1 8:8a5f654c:::benchmark_data_smithi137_101942_object46556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-09-18T07:38:27.103 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:38:26.178156+0000", 2024-09-18T07:38:27.103 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0092103670000000006, 2024-09-18T07:38:27.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095425389999999992, 2024-09-18T07:38:27.103 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:38:27.103 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:38:27.103 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:38:27.103 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:38:27.103 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:38:27.103 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:38:27.104 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151366 2024-09-18T07:38:27.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.104 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:38:27.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:38:27.104 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178156+0000", 2024-09-18T07:38:27.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:27.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:38:27.104 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178156+0000", 2024-09-18T07:38:27.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:27.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.105 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:38:27.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178156+0000", 2024-09-18T07:38:27.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4900000000000001e-07 2024-09-18T07:38:27.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.105 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:38:27.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178158+0000", 2024-09-18T07:38:27.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8449999999999999e-06 2024-09-18T07:38:27.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.105 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:38:27.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178162+0000", 2024-09-18T07:38:27.106 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.844e-06 2024-09-18T07:38:27.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.106 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:38:27.106 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178166+0000", 2024-09-18T07:38:27.106 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9020000000000001e-06 2024-09-18T07:38:27.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.106 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:38:27.106 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178545+0000", 2024-09-18T07:38:27.106 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000379854 2024-09-18T07:38:27.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.106 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:38:27.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178659+0000", 2024-09-18T07:38:27.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113669 2024-09-18T07:38:27.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-18T07:38:27.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178775+0000", 2024-09-18T07:38:27.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011602199999999999 2024-09-18T07:38:27.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:38:27.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.185199+0000", 2024-09-18T07:38:27.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0064237840000000001 2024-09-18T07:38:27.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:38:27.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.185211+0000", 2024-09-18T07:38:27.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2306e-05 2024-09-18T07:38:27.108 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:27.108 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:27.108 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:27.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.109 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:151368 8.1 8:83483226:::benchmark_data_smithi137_101942_object46558:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-09-18T07:38:27.109 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:38:26.178170+0000", 2024-09-18T07:38:27.109 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0091959559999999999, 2024-09-18T07:38:27.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095951519999999992, 2024-09-18T07:38:27.109 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:38:27.109 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:38:27.109 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:38:27.109 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:38:27.109 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:38:27.109 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:38:27.109 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151368 2024-09-18T07:38:27.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.109 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:38:27.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:38:27.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178170+0000", 2024-09-18T07:38:27.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:27.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:38:27.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178170+0000", 2024-09-18T07:38:27.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:27.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:38:27.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178171+0000", 2024-09-18T07:38:27.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9800000000000004e-07 2024-09-18T07:38:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:38:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178172+0000", 2024-09-18T07:38:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9069999999999999e-06 2024-09-18T07:38:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:38:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178176+0000", 2024-09-18T07:38:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6729999999999998e-06 2024-09-18T07:38:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:38:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.178180+0000", 2024-09-18T07:38:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6820000000000001e-06 2024-09-18T07:38:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:38:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.181797+0000", 2024-09-18T07:38:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036168580000000001 2024-09-18T07:38:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:38:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.181914+0000", 2024-09-18T07:38:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117501 2024-09-18T07:38:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-18T07:38:27.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.182035+0000", 2024-09-18T07:38:27.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120883 2024-09-18T07:38:27.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:27.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:27.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:38:27.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:26.185711+0000", 2024-09-18T07:38:27.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036759420000000002 2024-09-18T07:38:27.113 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:27.113 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:27.113 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:27.113 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:27.113 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:38:27.113 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 6 2024-09-18T07:38:27.114 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:38:27.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:27.196 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:27.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:27.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:27.298 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:27.329 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:27.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:27.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:27.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:27 smithi137 ceph-mon[29408]: pgmap v397: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 730 op/s; 2713/189138 objects degraded (1.434%); 3742/189138 objects misplaced (1.978%); 11 MiB/s, 222 objects/s recovering 2024-09-18T07:38:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:27 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:38:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:27 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:38:27.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:38:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:38:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:38:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:27 smithi137 ceph-mon[39175]: pgmap v397: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 730 op/s; 2713/189138 objects degraded (1.434%); 3742/189138 objects misplaced (1.978%); 11 MiB/s, 222 objects/s recovering 2024-09-18T07:38:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:27 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:38:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:27 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:38:27.431 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:27.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:27 smithi203 ceph-mon[32237]: pgmap v397: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 730 op/s; 2713/189138 objects degraded (1.434%); 3742/189138 objects misplaced (1.978%); 11 MiB/s, 222 objects/s recovering 2024-09-18T07:38:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:27 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:38:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:27 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:38:27.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:38:27 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:38:27.485 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:27.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:27.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:27.587 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:27.629 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:27.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:27.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:27.734 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:27.775 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:27.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:27.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:27.877 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:27.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:27.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:27.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:28.017 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:28.040 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:28.050 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:28.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:28.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:28.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:28.190 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:28.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:28.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:28.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T07:38:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:28.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:28.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:28.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:28.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:28.488 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:28.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:28.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:28.661 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:28.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:28.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:28.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:28.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:28.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:28.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:28.934 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:28.963 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:28.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:28.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:29.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:29.101 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:29.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:29.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:29.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:29 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:29.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:29 smithi137 ceph-mon[29408]: pgmap v398: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.27k op/s; 2713/173094 objects degraded (1.567%); 2890/173094 objects misplaced (1.670%); 15 MiB/s, 285 objects/s recovering 2024-09-18T07:38:29.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:29 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:29.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:29 smithi137 ceph-mon[39175]: pgmap v398: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.27k op/s; 2713/173094 objects degraded (1.567%); 2890/173094 objects misplaced (1.670%); 15 MiB/s, 285 objects/s recovering 2024-09-18T07:38:29.202 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:29.230 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:29.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:29.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:29.245 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:29.255 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:29.316 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:38:29.317 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:38:29.317 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 257, 2024-09-18T07:38:29.317 INFO:teuthology.orchestra.run.smithi137.stdout: "fsid": "f7ef099c-758e-11ef-bceb-c7b262605968", 2024-09-18T07:38:29.317 INFO:teuthology.orchestra.run.smithi137.stdout: "created": "2024-09-18T07:25:19.662367+0000", 2024-09-18T07:38:29.317 INFO:teuthology.orchestra.run.smithi137.stdout: "modified": "2024-09-18T07:38:24.908434+0000", 2024-09-18T07:38:29.317 INFO:teuthology.orchestra.run.smithi137.stdout: "last_up_change": "2024-09-18T07:30:44.708496+0000", 2024-09-18T07:38:29.317 INFO:teuthology.orchestra.run.smithi137.stdout: "last_in_change": "2024-09-18T07:37:42.022730+0000", 2024-09-18T07:38:29.317 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T07:38:29.318 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_num": 5799936, 2024-09-18T07:38:29.318 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_set": [ 2024-09-18T07:38:29.318 INFO:teuthology.orchestra.run.smithi137.stdout: "pglog_hardlimit", 2024-09-18T07:38:29.318 INFO:teuthology.orchestra.run.smithi137.stdout: "purged_snapdirs", 2024-09-18T07:38:29.318 INFO:teuthology.orchestra.run.smithi137.stdout: "recovery_deletes", 2024-09-18T07:38:29.318 INFO:teuthology.orchestra.run.smithi137.stdout: "sortbitwise" 2024-09-18T07:38:29.318 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:38:29.318 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_version": 18, 2024-09-18T07:38:29.318 INFO:teuthology.orchestra.run.smithi137.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T07:38:29.318 INFO:teuthology.orchestra.run.smithi137.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T07:38:29.318 INFO:teuthology.orchestra.run.smithi137.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T07:38:29.318 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_snapshot": "", 2024-09-18T07:38:29.318 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_max": 8, 2024-09-18T07:38:29.319 INFO:teuthology.orchestra.run.smithi137.stdout: "max_osd": 8, 2024-09-18T07:38:29.319 INFO:teuthology.orchestra.run.smithi137.stdout: "require_min_compat_client": "luminous", 2024-09-18T07:38:29.319 INFO:teuthology.orchestra.run.smithi137.stdout: "min_compat_client": "luminous", 2024-09-18T07:38:29.319 INFO:teuthology.orchestra.run.smithi137.stdout: "require_osd_release": "squid", 2024-09-18T07:38:29.319 INFO:teuthology.orchestra.run.smithi137.stdout: "allow_crimson": false, 2024-09-18T07:38:29.319 INFO:teuthology.orchestra.run.smithi137.stdout: "pools": [ 2024-09-18T07:38:29.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.319 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 1, 2024-09-18T07:38:29.319 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".mgr", 2024-09-18T07:38:29.319 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:28:20.899655+0000", 2024-09-18T07:38:29.319 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:38:29.319 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:38:29.319 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:38:29.320 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:38:29.320 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:38:29.320 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:38:29.320 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:38:29.320 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:38:29.320 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:38:29.320 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:38:29.320 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:38:29.320 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:38:29.320 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:38:29.320 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 1, 2024-09-18T07:38:29.320 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 1, 2024-09-18T07:38:29.320 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 1, 2024-09-18T07:38:29.321 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 1, 2024-09-18T07:38:29.321 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 1, 2024-09-18T07:38:29.321 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:38:29.321 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:38:29.321 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:38:29.321 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:38:29.321 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:38:29.321 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:38:29.321 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:38:29.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.321 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "21", 2024-09-18T07:38:29.321 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:38:29.321 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:38:29.322 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:38:29.322 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:38:29.322 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:38:29.322 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:38:29.322 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:38:29.322 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:38:29.322 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:38:29.322 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:38:29.322 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:38:29.322 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:38:29.322 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:38:29.322 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:38:29.322 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:38:29.323 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:38:29.323 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:38:29.323 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:38:29.323 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:38:29.323 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:38:29.323 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:38:29.323 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:38:29.323 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:38:29.323 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:38:29.323 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:38:29.323 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:38:29.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.324 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:38:29.324 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:38:29.324 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:38:29.324 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:38:29.324 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:38:29.324 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:38:29.324 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:38:29.325 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:38:29.325 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:38:29.325 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:38:29.325 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:38:29.325 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:38:29.325 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_max": 32, 2024-09-18T07:38:29.325 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_min": 1 2024-09-18T07:38:29.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.325 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:38:29.325 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": {} 2024-09-18T07:38:29.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.325 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:38:29.326 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:38:29.326 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 6.9800000190734863, 2024-09-18T07:38:29.326 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 6.9800000190734863, 2024-09-18T07:38:29.326 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.43000000715255737, 2024-09-18T07:38:29.326 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 3, 2024-09-18T07:38:29.326 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 3, 2024-09-18T07:38:29.326 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.875, 2024-09-18T07:38:29.326 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:38:29.326 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-18T07:38:29.326 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.326 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 2, 2024-09-18T07:38:29.327 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "datapool", 2024-09-18T07:38:29.327 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:53.715373+0000", 2024-09-18T07:38:29.327 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-18T07:38:29.327 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T07:38:29.327 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:38:29.327 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:38:29.327 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:38:29.327 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:38:29.327 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:38:29.327 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:38:29.327 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:38:29.327 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:38:29.327 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:38:29.327 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:38:29.328 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:38:29.328 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 3, 2024-09-18T07:38:29.328 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 3, 2024-09-18T07:38:29.328 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 3, 2024-09-18T07:38:29.328 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 3, 2024-09-18T07:38:29.328 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 3, 2024-09-18T07:38:29.328 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:38:29.328 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:38:29.328 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:38:29.328 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:38:29.328 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:38:29.328 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:38:29.328 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:38:29.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.329 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "63", 2024-09-18T07:38:29.329 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:38:29.329 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:38:29.329 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:38:29.329 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:38:29.329 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:38:29.329 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 2, 2024-09-18T07:38:29.329 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 63, 2024-09-18T07:38:29.329 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:38:29.329 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:38:29.329 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:38:29.329 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:38:29.329 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:38:29.330 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:38:29.330 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:38:29.330 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:38:29.330 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:38:29.330 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:38:29.330 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:38:29.330 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:38:29.330 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:38:29.330 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:38:29.330 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:38:29.330 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:38:29.330 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:38:29.330 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:38:29.330 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:38:29.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.331 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:38:29.331 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:38:29.331 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:38:29.331 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:38:29.331 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:38:29.331 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:38:29.331 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:38:29.331 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:38:29.331 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:38:29.331 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:38:29.331 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:38:29.331 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:38:29.331 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:38:29.332 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd": {} 2024-09-18T07:38:29.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.332 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:38:29.332 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:38:29.332 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.3299999237060547, 2024-09-18T07:38:29.332 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.3299999237060547, 2024-09-18T07:38:29.332 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:38:29.332 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-18T07:38:29.332 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-18T07:38:29.332 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.875, 2024-09-18T07:38:29.332 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:38:29.332 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-18T07:38:29.332 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.333 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 3, 2024-09-18T07:38:29.333 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".rgw.root", 2024-09-18T07:38:29.333 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:54.672127+0000", 2024-09-18T07:38:29.333 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:38:29.333 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:38:29.333 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:38:29.333 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:38:29.333 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:38:29.333 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:38:29.333 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:38:29.333 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:38:29.333 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:38:29.334 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:38:29.334 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:38:29.334 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:38:29.334 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:38:29.334 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:38:29.334 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:38:29.334 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:38:29.334 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:38:29.334 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:38:29.334 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:38:29.334 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:38:29.334 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:38:29.334 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:38:29.334 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:38:29.335 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:29.336 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:38:29.336 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:38:29.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.336 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "62", 2024-09-18T07:38:29.336 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:38:29.336 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:38:29.337 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:38:29.337 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:38:29.337 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:38:29.337 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:38:29.337 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:38:29.337 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:38:29.337 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:38:29.337 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:38:29.337 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:38:29.337 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:38:29.337 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:38:29.337 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:38:29.337 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:38:29.337 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:38:29.338 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:38:29.338 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:38:29.338 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:38:29.338 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:38:29.338 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:38:29.338 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:38:29.338 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:38:29.338 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:38:29.338 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:38:29.338 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:38:29.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.338 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:38:29.338 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:38:29.339 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:38:29.339 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:38:29.339 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:38:29.339 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:38:29.339 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:38:29.339 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:38:29.339 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:38:29.339 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:38:29.339 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:38:29.339 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:38:29.339 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:38:29.339 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:38:29.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.339 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:38:29.340 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:38:29.340 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.5399999618530273, 2024-09-18T07:38:29.340 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.5399999618530273, 2024-09-18T07:38:29.340 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1.1399999856948853, 2024-09-18T07:38:29.340 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.75, 2024-09-18T07:38:29.340 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.75, 2024-09-18T07:38:29.340 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.875, 2024-09-18T07:38:29.340 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:38:29.340 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-18T07:38:29.340 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.341 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 4, 2024-09-18T07:38:29.341 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.log", 2024-09-18T07:38:29.341 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:56.351343+0000", 2024-09-18T07:38:29.341 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:38:29.341 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:38:29.341 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:38:29.341 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:38:29.341 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:38:29.341 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:38:29.341 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:38:29.341 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:38:29.341 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:38:29.341 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:38:29.342 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:38:29.342 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:38:29.342 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:38:29.342 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:38:29.342 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:38:29.342 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:38:29.342 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:38:29.342 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:38:29.342 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:38:29.342 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:38:29.342 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:38:29.342 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:38:29.342 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:38:29.342 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:38:29.343 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:38:29.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.343 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "64", 2024-09-18T07:38:29.343 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:38:29.343 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:38:29.343 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:38:29.343 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:38:29.343 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:38:29.343 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:38:29.343 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:38:29.343 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:38:29.343 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:38:29.343 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:38:29.343 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:38:29.344 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:38:29.344 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:38:29.344 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:38:29.344 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:38:29.344 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:38:29.344 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:38:29.344 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:38:29.344 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:38:29.344 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:38:29.344 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:38:29.344 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:38:29.344 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:38:29.345 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:38:29.345 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:38:29.345 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:38:29.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.345 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:38:29.345 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:38:29.345 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:38:29.345 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:38:29.345 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:38:29.346 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:38:29.346 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:38:29.346 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:38:29.346 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:38:29.346 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:38:29.346 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:38:29.346 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:38:29.346 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:38:29.346 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:38:29.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.346 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:38:29.346 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:38:29.346 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.9700000286102295, 2024-09-18T07:38:29.347 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.9700000286102295, 2024-09-18T07:38:29.347 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1.1399999856948853, 2024-09-18T07:38:29.347 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.25, 2024-09-18T07:38:29.347 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.25, 2024-09-18T07:38:29.347 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.875, 2024-09-18T07:38:29.347 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:38:29.347 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-18T07:38:29.347 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.347 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 5, 2024-09-18T07:38:29.347 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.control", 2024-09-18T07:38:29.347 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:58.322401+0000", 2024-09-18T07:38:29.347 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:38:29.348 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:38:29.348 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:38:29.348 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:38:29.348 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:38:29.348 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:38:29.348 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:38:29.348 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:38:29.348 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:38:29.348 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:38:29.348 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:38:29.348 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:38:29.348 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:38:29.348 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:38:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:38:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:38:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:38:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:38:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:38:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:38:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:38:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:38:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:38:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:38:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:38:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "66", 2024-09-18T07:38:29.349 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:38:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:38:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:38:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:38:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:38:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:38:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:38:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:38:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:38:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:38:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:38:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:38:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:38:29.350 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:38:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:38:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:38:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:38:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:38:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:38:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:38:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:38:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:38:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:38:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:38:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:38:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:38:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.351 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:38:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:38:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:38:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:38:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:38:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:38:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:38:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:38:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:38:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:38:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:38:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:38:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:38:29.352 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:38:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:38:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:38:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.5399999618530273, 2024-09-18T07:38:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.5399999618530273, 2024-09-18T07:38:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1.1399999856948853, 2024-09-18T07:38:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.75, 2024-09-18T07:38:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.75, 2024-09-18T07:38:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.875, 2024-09-18T07:38:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:38:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-18T07:38:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.354 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 6, 2024-09-18T07:38:29.354 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.meta", 2024-09-18T07:38:29.354 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:31:00.390499+0000", 2024-09-18T07:38:29.354 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:38:29.354 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:38:29.354 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:38:29.354 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:38:29.354 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:38:29.354 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:38:29.354 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:38:29.354 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:38:29.354 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:38:29.354 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:38:29.355 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:38:29.355 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:38:29.355 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:38:29.355 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:38:29.355 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:38:29.355 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:38:29.355 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:38:29.355 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:38:29.355 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:38:29.355 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:38:29.355 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:38:29.355 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:38:29.355 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:38:29.356 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:38:29.356 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:38:29.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.356 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "68", 2024-09-18T07:38:29.356 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:38:29.356 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:38:29.356 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:38:29.356 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:38:29.356 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:38:29.356 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:38:29.356 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:38:29.356 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:38:29.356 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:38:29.357 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:38:29.357 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:38:29.357 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:38:29.357 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:38:29.357 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:38:29.357 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:38:29.357 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:38:29.357 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:38:29.358 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:38:29.358 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:38:29.358 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:38:29.358 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:38:29.358 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:38:29.358 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:38:29.358 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:38:29.358 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:38:29.358 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:38:29.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.358 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:38:29.358 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:38:29.358 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:38:29.359 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:38:29.359 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:38:29.359 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:38:29.359 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:38:29.359 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:38:29.359 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:38:29.359 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:38:29.359 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:38:29.359 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:38:29.359 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_bias": 4 2024-09-18T07:38:29.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.359 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:38:29.360 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:38:29.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.360 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:38:29.360 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:38:29.360 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.3200000524520874, 2024-09-18T07:38:29.360 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.3200000524520874, 2024-09-18T07:38:29.360 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1.1399999856948853, 2024-09-18T07:38:29.360 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.5, 2024-09-18T07:38:29.360 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.5, 2024-09-18T07:38:29.360 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.875, 2024-09-18T07:38:29.360 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:38:29.360 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-18T07:38:29.360 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.361 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 8, 2024-09-18T07:38:29.361 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "unique_pool_1", 2024-09-18T07:38:29.361 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:35:57.201644+0000", 2024-09-18T07:38:29.361 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:38:29.361 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:38:29.361 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:38:29.361 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:38:29.361 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:38:29.361 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:38:29.361 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:38:29.361 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:38:29.362 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:38:29.362 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:38:29.362 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:38:29.362 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:38:29.362 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:38:29.362 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 16, 2024-09-18T07:38:29.362 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 16, 2024-09-18T07:38:29.362 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 16, 2024-09-18T07:38:29.362 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 16, 2024-09-18T07:38:29.362 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 16, 2024-09-18T07:38:29.362 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:38:29.362 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:38:29.362 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:38:29.362 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:38:29.363 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:38:29.363 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:38:29.363 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:38:29.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.363 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "254", 2024-09-18T07:38:29.363 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:38:29.363 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:38:29.363 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:38:29.363 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:38:29.363 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:38:29.363 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:38:29.363 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:38:29.363 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:38:29.364 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:38:29.364 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:38:29.364 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:38:29.364 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:38:29.364 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:38:29.364 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:38:29.364 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:38:29.364 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:38:29.364 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:38:29.364 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:38:29.364 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:38:29.364 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:38:29.364 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:38:29.365 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:38:29.365 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:38:29.365 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:38:29.365 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:38:29.365 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:38:29.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.365 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:38:29.365 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:38:29.365 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:38:29.365 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:38:29.365 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:38:29.365 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:38:29.365 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:38:29.365 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:38:29.366 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:38:29.366 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:38:29.366 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:38:29.366 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:38:29.366 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:38:29.366 INFO:teuthology.orchestra.run.smithi137.stdout: "rados": {} 2024-09-18T07:38:29.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.366 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:38:29.366 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:38:29.366 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.190000057220459, 2024-09-18T07:38:29.366 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.75, 2024-09-18T07:38:29.366 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1.1399999856948853, 2024-09-18T07:38:29.366 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.5, 2024-09-18T07:38:29.366 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2, 2024-09-18T07:38:29.367 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.875, 2024-09-18T07:38:29.367 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-18T07:38:29.367 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-18T07:38:29.367 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.367 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.367 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:38:29.367 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-18T07:38:29.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.367 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:38:29.367 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "268cd580-c931-4992-bb93-d6f63c2bfe88", 2024-09-18T07:38:29.367 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:38:29.367 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:38:29.367 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:38:29.367 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:38:29.368 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:38:29.368 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:38:29.368 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 8, 2024-09-18T07:38:29.368 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 238, 2024-09-18T07:38:29.368 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:38:29.368 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:38:29.368 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:38:29.368 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.368 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.368 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6801", 2024-09-18T07:38:29.368 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:38:29.368 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.369 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.369 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:38:29.369 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.369 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.369 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6802", 2024-09-18T07:38:29.369 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:38:29.369 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.369 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.369 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:38:29.369 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.370 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.370 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6804", 2024-09-18T07:38:29.370 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:38:29.370 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.370 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.370 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:38:29.370 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.370 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.370 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6803", 2024-09-18T07:38:29.370 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:38:29.370 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.371 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.371 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6801/2811444477", 2024-09-18T07:38:29.371 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6802/2811444477", 2024-09-18T07:38:29.371 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6804/2811444477", 2024-09-18T07:38:29.372 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:29 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:29.372 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:29 smithi203 ceph-mon[32237]: pgmap v398: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.27k op/s; 2713/173094 objects degraded (1.567%); 2890/173094 objects misplaced (1.670%); 15 MiB/s, 285 objects/s recovering 2024-09-18T07:38:29.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:29.373 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6803/2811444477", 2024-09-18T07:38:29.373 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:38:29.373 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:38:29.373 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:38:29.373 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.374 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:38:29.374 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575", 2024-09-18T07:38:29.374 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:38:29.374 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:38:29.374 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:38:29.374 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 0, 2024-09-18T07:38:29.374 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:38:29.374 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:38:29.374 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 13, 2024-09-18T07:38:29.374 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 201, 2024-09-18T07:38:29.374 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:38:29.374 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:38:29.374 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:38:29.375 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.375 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.375 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6805", 2024-09-18T07:38:29.375 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:38:29.375 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.375 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.375 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:38:29.375 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.375 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.375 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6806", 2024-09-18T07:38:29.375 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:38:29.376 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.376 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.376 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:38:29.376 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.376 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.376 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6808", 2024-09-18T07:38:29.376 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:38:29.376 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.376 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.376 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:38:29.377 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.377 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.377 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6807", 2024-09-18T07:38:29.377 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:38:29.377 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.377 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.377 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6805/874102487", 2024-09-18T07:38:29.377 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6806/874102487", 2024-09-18T07:38:29.377 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6808/874102487", 2024-09-18T07:38:29.377 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6807/874102487", 2024-09-18T07:38:29.377 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:38:29.378 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:38:29.378 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:38:29.378 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.378 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:38:29.378 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "480ad6c5-0e0b-4405-bbe2-381682d00bae", 2024-09-18T07:38:29.378 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:38:29.378 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:38:29.378 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:38:29.378 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:38:29.378 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:38:29.379 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:38:29.379 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 18, 2024-09-18T07:38:29.379 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 238, 2024-09-18T07:38:29.379 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:38:29.379 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:38:29.379 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:38:29.379 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.379 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.379 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6809", 2024-09-18T07:38:29.379 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:38:29.379 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.379 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.380 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:38:29.380 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.380 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.380 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6810", 2024-09-18T07:38:29.380 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:38:29.380 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.380 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.380 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:38:29.380 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.381 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.381 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6812", 2024-09-18T07:38:29.381 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:38:29.381 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.381 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.381 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:38:29.381 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.381 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.381 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6811", 2024-09-18T07:38:29.381 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:38:29.381 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.382 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.382 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6809/1754684686", 2024-09-18T07:38:29.382 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6810/1754684686", 2024-09-18T07:38:29.382 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6812/1754684686", 2024-09-18T07:38:29.382 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6811/1754684686", 2024-09-18T07:38:29.382 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:38:29.382 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:38:29.382 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:38:29.382 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.382 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:38:29.383 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14", 2024-09-18T07:38:29.383 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:38:29.383 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:38:29.383 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:38:29.383 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:38:29.383 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:38:29.383 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:38:29.383 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 26, 2024-09-18T07:38:29.383 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 238, 2024-09-18T07:38:29.383 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:38:29.383 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:38:29.383 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:38:29.383 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.384 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.384 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6813", 2024-09-18T07:38:29.384 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:38:29.384 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.384 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.384 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:38:29.384 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.384 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.384 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6814", 2024-09-18T07:38:29.384 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:38:29.385 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.385 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.385 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:38:29.385 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.385 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.385 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6816", 2024-09-18T07:38:29.385 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:38:29.385 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.385 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.385 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:38:29.385 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.386 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.386 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6815", 2024-09-18T07:38:29.386 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:38:29.386 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.386 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.386 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6813/116058392", 2024-09-18T07:38:29.386 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6814/116058392", 2024-09-18T07:38:29.386 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6816/116058392", 2024-09-18T07:38:29.386 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6815/116058392", 2024-09-18T07:38:29.386 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:38:29.386 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:38:29.386 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:38:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:38:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de", 2024-09-18T07:38:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:38:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:38:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:38:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:38:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:38:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:38:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 32, 2024-09-18T07:38:29.387 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 254, 2024-09-18T07:38:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:38:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:38:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:38:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6800", 2024-09-18T07:38:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:38:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:38:29.388 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6801", 2024-09-18T07:38:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:38:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:38:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6803", 2024-09-18T07:38:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:38:29.389 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:38:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6802", 2024-09-18T07:38:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:38:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6800/3654297954", 2024-09-18T07:38:29.390 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6801/3654297954", 2024-09-18T07:38:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6803/3654297954", 2024-09-18T07:38:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6802/3654297954", 2024-09-18T07:38:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:38:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:38:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:38:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:38:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c", 2024-09-18T07:38:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:38:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:38:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:38:29.391 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:38:29.392 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:38:29.392 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:38:29.392 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 39, 2024-09-18T07:38:29.392 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 238, 2024-09-18T07:38:29.392 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:38:29.392 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:38:29.392 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:38:29.392 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.392 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.392 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6804", 2024-09-18T07:38:29.392 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:38:29.392 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.393 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.393 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:38:29.393 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.393 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.393 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6805", 2024-09-18T07:38:29.393 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:38:29.393 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.393 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.393 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:38:29.393 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.394 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.394 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6807", 2024-09-18T07:38:29.394 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:38:29.394 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.394 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.394 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:38:29.394 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.394 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.394 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6806", 2024-09-18T07:38:29.394 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:38:29.394 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.395 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.395 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6804/837994222", 2024-09-18T07:38:29.395 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6805/837994222", 2024-09-18T07:38:29.395 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6807/837994222", 2024-09-18T07:38:29.395 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6806/837994222", 2024-09-18T07:38:29.395 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:38:29.395 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:38:29.395 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:38:29.395 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.395 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:38:29.395 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6", 2024-09-18T07:38:29.396 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:38:29.396 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:38:29.396 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:38:29.396 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:38:29.396 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:38:29.396 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:38:29.396 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 47, 2024-09-18T07:38:29.396 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 255, 2024-09-18T07:38:29.396 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:38:29.396 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:38:29.396 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:38:29.396 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.397 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.397 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6808", 2024-09-18T07:38:29.397 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:38:29.397 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.397 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.397 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:38:29.397 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.397 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.397 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6809", 2024-09-18T07:38:29.397 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:38:29.397 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.398 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.398 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:38:29.398 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.398 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.398 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6811", 2024-09-18T07:38:29.398 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:38:29.398 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.398 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.398 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:38:29.398 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.399 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.399 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6810", 2024-09-18T07:38:29.399 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:38:29.399 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.399 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.399 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6808/1494175466", 2024-09-18T07:38:29.399 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6809/1494175466", 2024-09-18T07:38:29.399 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6811/1494175466", 2024-09-18T07:38:29.399 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6810/1494175466", 2024-09-18T07:38:29.399 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:38:29.399 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:38:29.399 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:38:29.400 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.400 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:38:29.400 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a", 2024-09-18T07:38:29.400 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:38:29.400 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:38:29.400 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:38:29.400 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:38:29.400 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:38:29.400 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:38:29.400 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 55, 2024-09-18T07:38:29.400 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 238, 2024-09-18T07:38:29.400 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:38:29.401 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:38:29.401 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:38:29.401 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.401 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.401 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6812", 2024-09-18T07:38:29.401 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:38:29.401 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.401 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.401 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:38:29.401 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.402 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.402 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6813", 2024-09-18T07:38:29.402 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:38:29.402 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.402 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.402 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:38:29.402 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.402 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.402 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6815", 2024-09-18T07:38:29.402 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:38:29.402 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.402 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.403 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:38:29.403 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:38:29.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.403 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:38:29.403 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6814", 2024-09-18T07:38:29.403 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:38:29.403 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.403 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.403 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6812/2670809315", 2024-09-18T07:38:29.403 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6813/2670809315", 2024-09-18T07:38:29.403 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6815/2670809315", 2024-09-18T07:38:29.403 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6814/2670809315", 2024-09-18T07:38:29.404 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:38:29.404 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:38:29.404 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:38:29.404 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.404 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.404 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:38:29.404 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_xinfo": [ 2024-09-18T07:38:29.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.404 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:38:29.404 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:38:29.404 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:38:29.404 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:38:29.404 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:38:29.405 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:38:29.405 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:38:11.239539+0000", 2024-09-18T07:38:29.405 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:38:29.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.405 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:38:29.405 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:38:29.405 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:38:29.405 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:38:29.405 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:38:29.405 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:38:29.405 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:38:07.191573+0000", 2024-09-18T07:38:29.405 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:38:29.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.406 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:38:29.406 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:38:29.406 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:38:29.406 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:38:29.406 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:38:29.406 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:38:29.406 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:37:27.199171+0000", 2024-09-18T07:38:29.406 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:38:29.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.406 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:38:29.406 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:38:29.406 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:38:29.407 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:38:29.407 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:38:29.407 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:38:29.407 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:38:05.196711+0000", 2024-09-18T07:38:29.407 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:38:29.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.407 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:38:29.407 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:38:29.407 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:38:29.407 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:38:29.407 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:38:29.407 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:38:29.407 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:37:58.510695+0000", 2024-09-18T07:38:29.408 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:38:29.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.408 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:38:29.408 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:38:29.408 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:38:29.408 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:38:29.408 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:38:29.408 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:38:29.408 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:37:24.867976+0000", 2024-09-18T07:38:29.408 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:38:29.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.409 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:38:29.409 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:38:29.409 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:38:29.409 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:38:29.409 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:38:29.409 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:38:29.409 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:37:38.347437+0000", 2024-09-18T07:38:29.409 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:38:29.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.409 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:38:29.409 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:38:29.409 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:38:29.409 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:38:29.409 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:38:29.410 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:38:29.410 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:38:01.788758+0000", 2024-09-18T07:38:29.410 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:38:29.410 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.410 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:38:29.410 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap": [], 2024-09-18T07:38:29.410 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_items": [ 2024-09-18T07:38:29.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.410 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.6", 2024-09-18T07:38:29.410 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:38:29.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.410 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:38:29.410 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:38:29.410 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.411 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.411 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.b", 2024-09-18T07:38:29.411 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:38:29.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.411 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:38:29.411 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:38:29.411 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.411 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.411 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.e", 2024-09-18T07:38:29.411 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:38:29.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.412 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:38:29.412 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:38:29.412 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.412 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.412 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.10", 2024-09-18T07:38:29.412 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:38:29.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.412 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:38:29.412 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:38:29.412 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.413 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.413 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.12", 2024-09-18T07:38:29.413 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:38:29.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.413 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:38:29.413 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:38:29.413 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.413 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.413 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.18", 2024-09-18T07:38:29.413 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:38:29.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.414 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:38:29.414 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:38:29.414 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.414 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.414 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.1b", 2024-09-18T07:38:29.414 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:38:29.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.414 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:38:29.414 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:38:29.414 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.414 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.415 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.1c", 2024-09-18T07:38:29.415 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:38:29.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.415 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:38:29.415 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:38:29.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.415 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:38:29.415 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 7 2024-09-18T07:38:29.415 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.415 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.416 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.1d", 2024-09-18T07:38:29.416 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:38:29.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.416 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:38:29.416 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:38:29.416 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.416 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.416 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.b", 2024-09-18T07:38:29.416 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:38:29.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.417 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 0, 2024-09-18T07:38:29.417 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 3 2024-09-18T07:38:29.417 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.417 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.417 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.4", 2024-09-18T07:38:29.417 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:38:29.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.417 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:38:29.417 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:38:29.417 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.417 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.418 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.1b", 2024-09-18T07:38:29.418 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:38:29.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.418 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:38:29.418 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:38:29.418 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.418 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.418 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.418 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:38:29.418 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_primaries": [], 2024-09-18T07:38:29.418 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_temp": [ 2024-09-18T07:38:29.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.418 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "8.3", 2024-09-18T07:38:29.419 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-18T07:38:29.419 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-18T07:38:29.419 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-18T07:38:29.419 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.419 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "8.6", 2024-09-18T07:38:29.419 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-18T07:38:29.419 INFO:teuthology.orchestra.run.smithi137.stdout: 0, 2024-09-18T07:38:29.419 INFO:teuthology.orchestra.run.smithi137.stdout: 7 2024-09-18T07:38:29.419 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.419 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "8.7", 2024-09-18T07:38:29.419 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-18T07:38:29.420 INFO:teuthology.orchestra.run.smithi137.stdout: 3, 2024-09-18T07:38:29.420 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-18T07:38:29.420 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:29.420 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "8.9", 2024-09-18T07:38:29.420 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-18T07:38:29.420 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-18T07:38:29.420 INFO:teuthology.orchestra.run.smithi137.stdout: 0 2024-09-18T07:38:29.420 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:29.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.420 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:38:29.420 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_temp": [], 2024-09-18T07:38:29.421 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklist": { 2024-09-18T07:38:29.421 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/2096039404": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:38:29.421 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/3636177939": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:38:29.421 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/334967443": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:38:29.421 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2101286111": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:38:29.421 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3387637412": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:38:29.421 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762989503": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:38:29.421 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3490817671": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:38:29.421 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1482147392": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:38:29.421 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1821482181": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:38:29.421 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/4028304023": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:38:29.421 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/621434521": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:38:29.421 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3160704935": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:38:29.421 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2357293499": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:38:29.422 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762352860": "2024-09-19T07:31:25.389124+0000" 2024-09-18T07:38:29.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.422 INFO:teuthology.orchestra.run.smithi137.stdout: "range_blocklist": {}, 2024-09-18T07:38:29.422 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profiles": { 2024-09-18T07:38:29.422 INFO:teuthology.orchestra.run.smithi137.stdout: "default": { 2024-09-18T07:38:29.422 INFO:teuthology.orchestra.run.smithi137.stdout: "crush-failure-domain": "osd", 2024-09-18T07:38:29.422 INFO:teuthology.orchestra.run.smithi137.stdout: "k": "2", 2024-09-18T07:38:29.422 INFO:teuthology.orchestra.run.smithi137.stdout: "m": "1", 2024-09-18T07:38:29.422 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin": "jerasure", 2024-09-18T07:38:29.422 INFO:teuthology.orchestra.run.smithi137.stdout: "technique": "reed_sol_van" 2024-09-18T07:38:29.422 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:29.422 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps_queue": [], 2024-09-18T07:38:29.422 INFO:teuthology.orchestra.run.smithi137.stdout: "new_removed_snaps": [], 2024-09-18T07:38:29.423 INFO:teuthology.orchestra.run.smithi137.stdout: "new_purged_snaps": [], 2024-09-18T07:38:29.423 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_node_flags": {}, 2024-09-18T07:38:29.423 INFO:teuthology.orchestra.run.smithi137.stdout: "device_class_flags": {}, 2024-09-18T07:38:29.423 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode": { 2024-09-18T07:38:29.423 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_enabled": false, 2024-09-18T07:38:29.423 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_bucket_count": 0, 2024-09-18T07:38:29.423 INFO:teuthology.orchestra.run.smithi137.stdout: "degraded_stretch_mode": 0, 2024-09-18T07:38:29.423 INFO:teuthology.orchestra.run.smithi137.stdout: "recovering_stretch_mode": 0, 2024-09-18T07:38:29.423 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_bucket": 0 2024-09-18T07:38:29.423 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:29.423 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:38:29.424 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:29.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:29.525 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:29.573 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:29.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:29.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:29.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:29.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:29.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:29.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:29.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:29.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:29.849 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:29.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:29.950 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:29.986 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:29.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:30.051 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:38:30.086 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:30.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:30.133 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:30.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:30.234 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:30.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:30.274 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:30.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:30.306 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:30 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2917410346' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:38:30.306 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:30 smithi137 ceph-mon[29408]: from='client.25468 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:38:30.306 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:30 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4263191228' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:38:30.306 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:30 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 2713/173094 objects degraded (1.567%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:38:30.306 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:30 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2917410346' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:38:30.306 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:30 smithi137 ceph-mon[39175]: from='client.25468 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:38:30.306 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:30 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4263191228' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:38:30.307 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:30 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 2713/173094 objects degraded (1.567%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:38:30.375 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:30.411 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:30.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:30 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2917410346' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:38:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:30 smithi203 ceph-mon[32237]: from='client.25468 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:38:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:30 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4263191228' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:38:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:30 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 2713/173094 objects degraded (1.567%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:38:30.513 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:30.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:30.553 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:30.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:30.654 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:30.695 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:30.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:30.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:30.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:30.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:30.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:30.923 INFO:tasks.thrashosds.thrasher:j is {'epoch': 257, 'fsid': 'f7ef099c-758e-11ef-bceb-c7b262605968', 'created': '2024-09-18T07:25:19.662367+0000', 'modified': '2024-09-18T07:38:24.908434+0000', 'last_up_change': '2024-09-18T07:30:44.708496+0000', 'last_in_change': '2024-09-18T07:37:42.022730+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T07:28:20.899655+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.980000019073486, 'score_stable': 6.980000019073486, 'optimal_score': 0.4300000071525574, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-18T07:30:53.715373+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 1, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-18T07:30:54.672127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-18T07:30:56.351343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-18T07:30:58.322401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-18T07:31:00.390499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-18T07:35:57.201644+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '254', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.5, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '268cd580-c931-4992-bb93-d6f63c2bfe88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 238, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6801', 'nonce': 2811444477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6802', 'nonce': 2811444477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6804', 'nonce': 2811444477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6803', 'nonce': 2811444477}]}, 'public_addr': '172.21.15.137:6801/2811444477', 'cluster_addr': '172.21.15.137:6802/2811444477', 'heartbeat_back_addr': '172.21.15.137:6804/2811444477', 'heartbeat_front_addr': '172.21.15.137:6803/2811444477', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b06907bc-b499-46c0-8667-c5bdc0d85575', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 201, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6805', 'nonce': 874102487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6806', 'nonce': 874102487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6808', 'nonce': 874102487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6807', 'nonce': 874102487}]}, 'public_addr': '172.21.15.137:6805/874102487', 'cluster_addr': '172.21.15.137:6806/874102487', 'heartbeat_back_addr': '172.21.15.137:6808/874102487', 'heartbeat_front_addr': '172.21.15.137:6807/874102487', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '480ad6c5-0e0b-4405-bbe2-381682d00bae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 238, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6809', 'nonce': 1754684686}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6810', 'nonce': 1754684686}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6812', 'nonce': 1754684686}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6811', 'nonce': 1754684686}]}, 'public_addr': '172.21.15.137:6809/1754684686', 'cluster_addr': '172.21.15.137:6810/1754684686', 'heartbeat_back_addr': '172.21.15.137:6812/1754684686', 'heartbeat_front_addr': '172.21.15.137:6811/1754684686', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '178be3e0-f07b-4856-807b-2a1181e48a14', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 238, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6813', 'nonce': 116058392}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6814', 'nonce': 116058392}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6816', 'nonce': 116058392}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6815', 'nonce': 116058392}]}, 'public_addr': '172.21.15.137:6813/116058392', 'cluster_addr': '172.21.15.137:6814/116058392', 'heartbeat_back_addr': '172.21.15.137:6816/116058392', 'heartbeat_front_addr': '172.21.15.137:6815/116058392', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7547102c-951a-4077-a094-fb8a65bfe6de', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 254, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6800', 'nonce': 3654297954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6801', 'nonce': 3654297954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6803', 'nonce': 3654297954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6802', 'nonce': 3654297954}]}, 'public_addr': '172.21.15.203:6800/3654297954', 'cluster_addr': '172.21.15.203:6801/3654297954', 'heartbeat_back_addr': '172.21.15.203:6803/3654297954', 'heartbeat_front_addr': '172.21.15.203:6802/3654297954', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '49c17901-8102-48a4-9348-fa1bdb63032c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 238, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6804', 'nonce': 837994222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6805', 'nonce': 837994222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6807', 'nonce': 837994222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6806', 'nonce': 837994222}]}, 'public_addr': '172.21.15.203:6804/837994222', 'cluster_addr': '172.21.15.203:6805/837994222', 'heartbeat_back_addr': '172.21.15.203:6807/837994222', 'heartbeat_front_addr': '172.21.15.203:6806/837994222', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '75f0ead4-e981-4a15-b1c1-d83dce871ab6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 255, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6808', 'nonce': 1494175466}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6809', 'nonce': 1494175466}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6811', 'nonce': 1494175466}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6810', 'nonce': 1494175466}]}, 'public_addr': '172.21.15.203:6808/1494175466', 'cluster_addr': '172.21.15.203:6809/1494175466', 'heartbeat_back_addr': '172.21.15.203:6811/1494175466', 'heartbeat_front_addr': '172.21.15.203:6810/1494175466', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c0dbf299-6798-4b99-9a85-62b123971f2a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 55, 'up_thru': 238, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6812', 'nonce': 2670809315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6813', 'nonce': 2670809315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6815', 'nonce': 2670809315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6814', 'nonce': 2670809315}]}, 'public_addr': '172.21.15.203:6812/2670809315', 'cluster_addr': '172.21.15.203:6813/2670809315', 'heartbeat_back_addr': '172.21.15.203:6815/2670809315', 'heartbeat_front_addr': '172.21.15.203:6814/2670809315', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:38:11.239539+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:38:07.191573+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:37:27.199171+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:38:05.196711+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:37:58.510695+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:37:24.867976+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:37:38.347437+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:38:01.788758+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1b', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '4.1c', 'mappings': [{'from': 3, 'to': 0}, {'from': 1, 'to': 7}]}, {'pgid': '4.1d', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '5.b', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '6.4', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '6.1b', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '8.3', 'osds': [5, 1]}, {'pgid': '8.6', 'osds': [0, 7]}, {'pgid': '8.7', 'osds': [3, 1]}, {'pgid': '8.9', 'osds': [5, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.137:6800/2096039404': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:6800/3636177939': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/334967443': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/2101286111': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/3387637412': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1762989503': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3490817671': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:6800/1482147392': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1821482181': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/4028304023': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/621434521': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3160704935': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/2357293499': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/1762352860': '2024-09-19T07:31:25.389124+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T07:38:30.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:38:30.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:30.972 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:30.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:30.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:31.075 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:31.144 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:31 smithi137 ceph-mon[29408]: pgmap v399: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 2713/173094 objects degraded (1.567%); 2890/173094 objects misplaced (1.670%); 14 MiB/s, 253 objects/s recovering 2024-09-18T07:38:31.144 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:31 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:38:31.144 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:31 smithi137 ceph-mon[29408]: from='client.25468 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:38:31.144 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:31 smithi137 ceph-mon[29408]: osdmap e258: 8 total, 8 up, 8 in 2024-09-18T07:38:31.144 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:31 smithi137 ceph-mon[39175]: pgmap v399: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 2713/173094 objects degraded (1.567%); 2890/173094 objects misplaced (1.670%); 14 MiB/s, 253 objects/s recovering 2024-09-18T07:38:31.144 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:31 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:38:31.144 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:31 smithi137 ceph-mon[39175]: from='client.25468 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:38:31.145 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:31 smithi137 ceph-mon[39175]: osdmap e258: 8 total, 8 up, 8 in 2024-09-18T07:38:31.147 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:31.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:31.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:31.225 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:38:31.226 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:38:31.249 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:31.267 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:38:31.268 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:38:31.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:31.284 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:31.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:31.299 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:38:31.299 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:38:31.332 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:38:31.332 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:38:31.366 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:38:31.367 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:38:31.385 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:31.402 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:38:31.402 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:38:31.415 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:31.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:31.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:31.436 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:38:31.437 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:38:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:31 smithi203 ceph-mon[32237]: pgmap v399: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 2713/173094 objects degraded (1.567%); 2890/173094 objects misplaced (1.670%); 14 MiB/s, 253 objects/s recovering 2024-09-18T07:38:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:31 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:38:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:31 smithi203 ceph-mon[32237]: from='client.25468 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:38:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:31 smithi203 ceph-mon[32237]: osdmap e258: 8 total, 8 up, 8 in 2024-09-18T07:38:31.471 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:38:31.472 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:38:31.496 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:38:31.496 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:38:31.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:31.561 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:31.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:31.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:31.663 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:31.867 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:31.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:31.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:31.969 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:32.001 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:32.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:32.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:32.103 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:32.143 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:32.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:32.245 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:32.276 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:32.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:32.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:32.368 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:32 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1343386174' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:32.368 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:32 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1343386174' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:32.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:32.411 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:32.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:32.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:32.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:32 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1343386174' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:32.513 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:32.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:32.558 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:32.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:32.659 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:32.694 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:32.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:32.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:32.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:32.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:32.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:32.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:32.934 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:32.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:32.975 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:32.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:33.076 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:33.112 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:33.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:33.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:33.214 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:33.252 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:33.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:33.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:33.323 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:33 smithi137 ceph-mon[29408]: pgmap v401: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 708 op/s; 2713/172155 objects degraded (1.576%); 2890/172155 objects misplaced (1.679%); 6.7 MiB/s, 106 objects/s recovering 2024-09-18T07:38:33.323 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:33 smithi137 ceph-mon[39175]: pgmap v401: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 708 op/s; 2713/172155 objects degraded (1.576%); 2890/172155 objects misplaced (1.679%); 6.7 MiB/s, 106 objects/s recovering 2024-09-18T07:38:33.354 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:33.385 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:33.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:33.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:33 smithi203 ceph-mon[32237]: pgmap v401: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 708 op/s; 2713/172155 objects degraded (1.576%); 2890/172155 objects misplaced (1.679%); 6.7 MiB/s, 106 objects/s recovering 2024-09-18T07:38:33.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:33.518 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:33.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:33.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:33.620 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:33.654 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:33.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:33.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:33.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:33.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:33.793 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:33.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:33.894 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:33.934 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:33.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:33.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:34.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:34.078 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:34.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:34.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:34.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:34.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T07:38:34.188 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:34.238 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:34.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:34.340 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:34.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:34.388 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:34.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:34.489 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:34.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:34.525 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:34.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:34.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:34.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:34.664 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:34.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:34.765 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:34.793 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:34.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:34.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:34.839 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:34.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:38:34.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:34.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:34.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:34.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:35.049 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:35.081 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:38:35.084 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:38:35.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:35.086 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:35.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:35.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:35.233 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:35.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:35.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:35.335 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:35.371 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:35.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:35.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:35.375 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:38:35.390 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:35 smithi137 ceph-mon[29408]: pgmap v402: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 2713/158265 objects degraded (1.714%); 1922/158265 objects misplaced (1.214%); 13 MiB/s, 210 objects/s recovering 2024-09-18T07:38:35.391 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:35 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 2713/158265 objects degraded (1.714%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:38:35.391 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:35 smithi137 ceph-mon[29408]: osdmap e259: 8 total, 8 up, 8 in 2024-09-18T07:38:35.391 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:35 smithi137 ceph-mon[39175]: pgmap v402: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 2713/158265 objects degraded (1.714%); 1922/158265 objects misplaced (1.214%); 13 MiB/s, 210 objects/s recovering 2024-09-18T07:38:35.391 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:35 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 2713/158265 objects degraded (1.714%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:38:35.391 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:35 smithi137 ceph-mon[39175]: osdmap e259: 8 total, 8 up, 8 in 2024-09-18T07:38:35.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:35 smithi203 ceph-mon[32237]: pgmap v402: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 2713/158265 objects degraded (1.714%); 1922/158265 objects misplaced (1.214%); 13 MiB/s, 210 objects/s recovering 2024-09-18T07:38:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:35 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 2713/158265 objects degraded (1.714%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:38:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:35 smithi203 ceph-mon[32237]: osdmap e259: 8 total, 8 up, 8 in 2024-09-18T07:38:35.472 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:35.508 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:35.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:35.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:35.610 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:35.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:35.656 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:35.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:35.731 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:38:35.731 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:38:35.757 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:35.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:35.788 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:35.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:35.889 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:35.932 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:35.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:35.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:36.034 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:36.068 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:36.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:36.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:36.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:36.200 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-09-18T07:38:34.923105+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3444841011,"num_objects":52755,"num_object_clones":0,"num_object_copies":158265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2713,"num_objects_misplaced":1922,"num_objects_unfound":0,"num_objects_dirty":52755,"num_whiteouts":0,"num_read":1033,"num_read_kb":892,"num_write":157609,"num_write_kb":6709322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19978,"num_bytes_recovered":1273372715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85227,"ondisk_log_size":85227,"up":444,"acting":440,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":449,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13558232,"kb_used_data":11417864,"kb_used_omap":12,"kb_used_meta":2138803,"kb_avail":736402984,"statfs":{"total":767960285184,"available":754076655616,"internally_reserved":0,"allocated":11691892736,"data_stored":11686316689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2190134832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,4,8,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-674430976,"num_objects":-10291,"num_object_clones":0,"num_object_copies":-30873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1820,"num_objects_unfound":0,"num_objects_dirty":-10291,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10291,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1820,"num_bytes_recovered":119275520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.368964"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":423,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.056804+0000","last_change":"2024-09-18T07:35:28.534496+0000","last_active":"2024-09-18T07:38:30.056804+0000","last_peered":"2024-09-18T07:38:30.056804+0000","last_clean":"2024-09-18T07:38:30.056804+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:38:30.056804+0000","last_undegraded":"2024-09-18T07:38:30.056804+0000","last_fullsized":"2024-09-18T07:38:30.056804+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:28.534391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:28.534391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":488,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051110+0000","last_change":"2024-09-18T07:37:43.103765+0000","last_active":"2024-09-18T07:38:30.051110+0000","last_peered":"2024-09-18T07:38:30.051110+0000","last_clean":"2024-09-18T07:38:30.051110+0000","last_became_active":"2024-09-18T07:37:43.061125+0000","last_became_peered":"2024-09-18T07:37:43.061125+0000","last_unstale":"2024-09-18T07:38:30.051110+0000","last_undegraded":"2024-09-18T07:38:30.051110+0000","last_fullsized":"2024-09-18T07:38:30.051110+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:35:27.883281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:27.883281+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":364,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055624+0000","last_change":"2024-09-18T07:36:38.570405+0000","last_active":"2024-09-18T07:38:30.055624+0000","last_peered":"2024-09-18T07:38:30.055624+0000","last_clean":"2024-09-18T07:38:30.055624+0000","last_became_active":"2024-09-18T07:36:13.128805+0000","last_became_peered":"2024-09-18T07:36:13.128805+0000","last_unstale":"2024-09-18T07:38:30.055624+0000","last_undegraded":"2024-09-18T07:38:30.055624+0000","last_fullsized":"2024-09-18T07:38:30.055624+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:38.570339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:38.570339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":405,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051352+0000","last_change":"2024-09-18T07:32:03.953881+0000","last_active":"2024-09-18T07:38:30.051352+0000","last_peered":"2024-09-18T07:38:30.051352+0000","last_clean":"2024-09-18T07:38:30.051352+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:38:30.051352+0000","last_undegraded":"2024-09-18T07:38:30.051352+0000","last_fullsized":"2024-09-18T07:38:30.051352+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.953827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.953827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":318,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055694+0000","last_change":"2024-09-18T07:37:43.056617+0000","last_active":"2024-09-18T07:38:30.055694+0000","last_peered":"2024-09-18T07:38:30.055694+0000","last_clean":"2024-09-18T07:38:30.055694+0000","last_became_active":"2024-09-18T07:37:43.056380+0000","last_became_peered":"2024-09-18T07:37:43.056380+0000","last_unstale":"2024-09-18T07:38:30.055694+0000","last_undegraded":"2024-09-18T07:38:30.055694+0000","last_fullsized":"2024-09-18T07:38:30.055694+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:01.481514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:01.481514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":430,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.062540+0000","last_change":"2024-09-18T07:37:43.066814+0000","last_active":"2024-09-18T07:38:30.062540+0000","last_peered":"2024-09-18T07:38:30.062540+0000","last_clean":"2024-09-18T07:38:30.062540+0000","last_became_active":"2024-09-18T07:37:43.066508+0000","last_became_peered":"2024-09-18T07:37:43.066508+0000","last_unstale":"2024-09-18T07:38:30.062540+0000","last_undegraded":"2024-09-18T07:38:30.062540+0000","last_fullsized":"2024-09-18T07:38:30.062540+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.006488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.006488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":437,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.058593+0000","last_change":"2024-09-18T07:33:22.032321+0000","last_active":"2024-09-18T07:38:30.058593+0000","last_peered":"2024-09-18T07:38:30.058593+0000","last_clean":"2024-09-18T07:38:30.058593+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:38:30.058593+0000","last_undegraded":"2024-09-18T07:38:30.058593+0000","last_fullsized":"2024-09-18T07:38:30.058593+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:22.032110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.032110+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":431,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051010+0000","last_change":"2024-09-18T07:37:43.052675+0000","last_active":"2024-09-18T07:38:30.051010+0000","last_peered":"2024-09-18T07:38:30.051010+0000","last_clean":"2024-09-18T07:38:30.051010+0000","last_became_active":"2024-09-18T07:37:43.052346+0000","last_became_peered":"2024-09-18T07:37:43.052346+0000","last_unstale":"2024-09-18T07:38:30.051010+0000","last_undegraded":"2024-09-18T07:38:30.051010+0000","last_fullsized":"2024-09-18T07:38:30.051010+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:30.995282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:30.995282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":437,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055624+0000","last_change":"2024-09-18T07:37:43.080879+0000","last_active":"2024-09-18T07:38:30.055624+0000","last_peered":"2024-09-18T07:38:30.055624+0000","last_clean":"2024-09-18T07:38:30.055624+0000","last_became_active":"2024-09-18T07:37:43.080314+0000","last_became_peered":"2024-09-18T07:37:43.080314+0000","last_unstale":"2024-09-18T07:38:30.055624+0000","last_undegraded":"2024-09-18T07:38:30.055624+0000","last_fullsized":"2024-09-18T07:38:30.055624+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:00.392466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.392466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":429,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051648+0000","last_change":"2024-09-18T07:32:02.002919+0000","last_active":"2024-09-18T07:38:30.051648+0000","last_peered":"2024-09-18T07:38:30.051648+0000","last_clean":"2024-09-18T07:38:30.051648+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:38:30.051648+0000","last_undegraded":"2024-09-18T07:38:30.051648+0000","last_fullsized":"2024-09-18T07:38:30.051648+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.002811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.002811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":421,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.056594+0000","last_change":"2024-09-18T07:35:04.267623+0000","last_active":"2024-09-18T07:38:30.056594+0000","last_peered":"2024-09-18T07:38:30.056594+0000","last_clean":"2024-09-18T07:38:30.056594+0000","last_became_active":"2024-09-18T07:35:01.413443+0000","last_became_peered":"2024-09-18T07:35:01.413443+0000","last_unstale":"2024-09-18T07:38:30.056594+0000","last_undegraded":"2024-09-18T07:38:30.056594+0000","last_fullsized":"2024-09-18T07:38:30.056594+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:04.267580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:04.267580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":417,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055647+0000","last_change":"2024-09-18T07:36:10.551960+0000","last_active":"2024-09-18T07:38:30.055647+0000","last_peered":"2024-09-18T07:38:30.055647+0000","last_clean":"2024-09-18T07:38:30.055647+0000","last_became_active":"2024-09-18T07:35:01.410284+0000","last_became_peered":"2024-09-18T07:35:01.410284+0000","last_unstale":"2024-09-18T07:38:30.055647+0000","last_undegraded":"2024-09-18T07:38:30.055647+0000","last_fullsized":"2024-09-18T07:38:30.055647+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:10.551889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:10.551889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":407,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.054419+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:38:30.054419+0000","last_peered":"2024-09-18T07:38:30.054419+0000","last_clean":"2024-09-18T07:38:30.054419+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:38:30.054419+0000","last_undegraded":"2024-09-18T07:38:30.054419+0000","last_fullsized":"2024-09-18T07:38:30.054419+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":436,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051155+0000","last_change":"2024-09-18T07:36:05.903851+0000","last_active":"2024-09-18T07:38:30.051155+0000","last_peered":"2024-09-18T07:38:30.051155+0000","last_clean":"2024-09-18T07:38:30.051155+0000","last_became_active":"2024-09-18T07:36:02.526212+0000","last_became_peered":"2024-09-18T07:36:02.526212+0000","last_unstale":"2024-09-18T07:38:30.051155+0000","last_undegraded":"2024-09-18T07:38:30.051155+0000","last_fullsized":"2024-09-18T07:38:30.051155+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-18T07:36:05.903784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.903784+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":183,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.063227+0000","last_change":"2024-09-18T07:35:01.399995+0000","last_active":"2024-09-18T07:38:30.063227+0000","last_peered":"2024-09-18T07:38:30.063227+0000","last_clean":"2024-09-18T07:38:30.063227+0000","last_became_active":"2024-09-18T07:35:01.398605+0000","last_became_peered":"2024-09-18T07:35:01.398605+0000","last_unstale":"2024-09-18T07:38:30.063227+0000","last_undegraded":"2024-09-18T07:38:30.063227+0000","last_fullsized":"2024-09-18T07:38:30.063227+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:55.843864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:55.843864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":419,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055751+0000","last_change":"2024-09-18T07:35:35.248892+0000","last_active":"2024-09-18T07:38:30.055751+0000","last_peered":"2024-09-18T07:38:30.055751+0000","last_clean":"2024-09-18T07:38:30.055751+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:38:30.055751+0000","last_undegraded":"2024-09-18T07:38:30.055751+0000","last_fullsized":"2024-09-18T07:38:30.055751+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:35.248824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:35.248824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":427,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051584+0000","last_change":"2024-09-18T07:37:43.067972+0000","last_active":"2024-09-18T07:38:30.051584+0000","last_peered":"2024-09-18T07:38:30.051584+0000","last_clean":"2024-09-18T07:38:30.051584+0000","last_became_active":"2024-09-18T07:37:43.067568+0000","last_became_peered":"2024-09-18T07:37:43.067568+0000","last_unstale":"2024-09-18T07:38:30.051584+0000","last_undegraded":"2024-09-18T07:38:30.051584+0000","last_fullsized":"2024-09-18T07:38:30.051584+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:24.929124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:24.929124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":437,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.056603+0000","last_change":"2024-09-18T07:36:06.523357+0000","last_active":"2024-09-18T07:38:30.056603+0000","last_peered":"2024-09-18T07:38:30.056603+0000","last_clean":"2024-09-18T07:38:30.056603+0000","last_became_active":"2024-09-18T07:35:01.414088+0000","last_became_peered":"2024-09-18T07:35:01.414088+0000","last_unstale":"2024-09-18T07:38:30.056603+0000","last_undegraded":"2024-09-18T07:38:30.056603+0000","last_fullsized":"2024-09-18T07:38:30.056603+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:06.523292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.523292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":484,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.058966+0000","last_change":"2024-09-18T07:37:28.072530+0000","last_active":"2024-09-18T07:38:30.058966+0000","last_peered":"2024-09-18T07:38:30.058966+0000","last_clean":"2024-09-18T07:38:30.058966+0000","last_became_active":"2024-09-18T07:37:28.002858+0000","last_became_peered":"2024-09-18T07:37:28.002858+0000","last_unstale":"2024-09-18T07:38:30.058966+0000","last_undegraded":"2024-09-18T07:38:30.058966+0000","last_fullsized":"2024-09-18T07:38:30.058966+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:35:31.858040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:31.858040+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:57.917053+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":107,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051523+0000","last_change":"2024-09-18T07:36:13.116882+0000","last_active":"2024-09-18T07:38:30.051523+0000","last_peered":"2024-09-18T07:38:30.051523+0000","last_clean":"2024-09-18T07:38:30.051523+0000","last_became_active":"2024-09-18T07:36:13.116416+0000","last_became_peered":"2024-09-18T07:36:13.116416+0000","last_unstale":"2024-09-18T07:38:30.051523+0000","last_undegraded":"2024-09-18T07:38:30.051523+0000","last_fullsized":"2024-09-18T07:38:30.051523+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:10.074367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:10.074367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":426,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.070074+0000","last_change":"2024-09-18T07:37:43.067816+0000","last_active":"2024-09-18T07:38:30.070074+0000","last_peered":"2024-09-18T07:38:30.070074+0000","last_clean":"2024-09-18T07:38:30.070074+0000","last_became_active":"2024-09-18T07:37:43.067493+0000","last_became_peered":"2024-09-18T07:37:43.067493+0000","last_unstale":"2024-09-18T07:38:30.070074+0000","last_undegraded":"2024-09-18T07:38:30.070074+0000","last_fullsized":"2024-09-18T07:38:30.070074+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:34.992350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:34.992350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":425,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051758+0000","last_change":"2024-09-18T07:35:37.941957+0000","last_active":"2024-09-18T07:38:30.051758+0000","last_peered":"2024-09-18T07:38:30.051758+0000","last_clean":"2024-09-18T07:38:30.051758+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:38:30.051758+0000","last_undegraded":"2024-09-18T07:38:30.051758+0000","last_fullsized":"2024-09-18T07:38:30.051758+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:37.941908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:37.941908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":78,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.052143+0000","last_change":"2024-09-18T07:37:43.756858+0000","last_active":"2024-09-18T07:38:30.052143+0000","last_peered":"2024-09-18T07:38:30.052143+0000","last_clean":"2024-09-18T07:38:30.052143+0000","last_became_active":"2024-09-18T07:37:27.981705+0000","last_became_peered":"2024-09-18T07:37:27.981705+0000","last_unstale":"2024-09-18T07:38:30.052143+0000","last_undegraded":"2024-09-18T07:38:30.052143+0000","last_fullsized":"2024-09-18T07:38:30.052143+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:37:43.756403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:37:43.756403+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:47.892762+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":401,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.052549+0000","last_change":"2024-09-18T07:31:00.334313+0000","last_active":"2024-09-18T07:38:30.052549+0000","last_peered":"2024-09-18T07:38:30.052549+0000","last_clean":"2024-09-18T07:38:30.052549+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:38:30.052549+0000","last_undegraded":"2024-09-18T07:38:30.052549+0000","last_fullsized":"2024-09-18T07:38:30.052549+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:31.063052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":195,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.063859+0000","last_change":"2024-09-18T07:35:30.254797+0000","last_active":"2024-09-18T07:38:30.063859+0000","last_peered":"2024-09-18T07:38:30.063859+0000","last_clean":"2024-09-18T07:38:30.063859+0000","last_became_active":"2024-09-18T07:35:01.413110+0000","last_became_peered":"2024-09-18T07:35:01.413110+0000","last_unstale":"2024-09-18T07:38:30.063859+0000","last_undegraded":"2024-09-18T07:38:30.063859+0000","last_fullsized":"2024-09-18T07:38:30.063859+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:35:30.254725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:30.254725+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":116,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.058461+0000","last_change":"2024-09-18T07:36:13.116657+0000","last_active":"2024-09-18T07:38:30.058461+0000","last_peered":"2024-09-18T07:38:30.058461+0000","last_clean":"2024-09-18T07:38:30.058461+0000","last_became_active":"2024-09-18T07:36:13.116295+0000","last_became_peered":"2024-09-18T07:36:13.116295+0000","last_unstale":"2024-09-18T07:38:30.058461+0000","last_undegraded":"2024-09-18T07:38:30.058461+0000","last_fullsized":"2024-09-18T07:38:30.058461+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:32.976129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.976129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":472,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.060833+0000","last_change":"2024-09-18T07:35:57.488019+0000","last_active":"2024-09-18T07:38:30.060833+0000","last_peered":"2024-09-18T07:38:30.060833+0000","last_clean":"2024-09-18T07:38:30.060833+0000","last_became_active":"2024-09-18T07:35:01.410926+0000","last_became_peered":"2024-09-18T07:35:01.410926+0000","last_unstale":"2024-09-18T07:38:30.060833+0000","last_undegraded":"2024-09-18T07:38:30.060833+0000","last_fullsized":"2024-09-18T07:38:30.060833+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:35:57.487960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.487960+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"240'12","reported_seq":64,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.066433+0000","last_change":"2024-09-18T07:37:43.220303+0000","last_active":"2024-09-18T07:38:30.066433+0000","last_peered":"2024-09-18T07:38:30.066433+0000","last_clean":"2024-09-18T07:38:30.066433+0000","last_became_active":"2024-09-18T07:37:43.060715+0000","last_became_peered":"2024-09-18T07:37:43.060715+0000","last_unstale":"2024-09-18T07:38:30.066433+0000","last_undegraded":"2024-09-18T07:38:30.066433+0000","last_fullsized":"2024-09-18T07:38:30.066433+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:35:29.400137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:29.400137+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"258'9765","reported_seq":16394,"reported_epoch":258,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T07:38:31.695307+0000","last_change":"2024-09-18T07:37:43.240841+0000","last_active":"2024-09-18T07:38:31.695307+0000","last_peered":"2024-09-18T07:38:31.695307+0000","last_clean":"2024-09-18T07:36:40.992666+0000","last_became_active":"2024-09-18T07:37:43.231704+0000","last_became_peered":"2024-09-18T07:37:43.231704+0000","last_unstale":"2024-09-18T07:38:31.695307+0000","last_undegraded":"2024-09-18T07:37:43.163542+0000","last_fullsized":"2024-09-18T07:37:43.046004+0000","mapping_epoch":238,"log_start":"223'4614","ondisk_log_start":"223'4614","created":201,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5151,"log_dups_size":0,"ondisk_log_size":5151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216203264,"num_objects":3299,"num_object_clones":0,"num_object_copies":9897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2713,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9765,"num_write_kb":418048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3241,"num_bytes_recovered":212336640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,2,5","objects":2432},{"shards":"1,5","objects":867}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":153,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.070309+0000","last_change":"2024-09-18T07:36:13.120127+0000","last_active":"2024-09-18T07:38:30.070309+0000","last_peered":"2024-09-18T07:38:30.070309+0000","last_clean":"2024-09-18T07:38:30.070309+0000","last_became_active":"2024-09-18T07:36:13.119098+0000","last_became_peered":"2024-09-18T07:36:13.119098+0000","last_unstale":"2024-09-18T07:38:30.070309+0000","last_undegraded":"2024-09-18T07:38:30.070309+0000","last_fullsized":"2024-09-18T07:38:30.070309+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:34:37.067162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:37.067162+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":446,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.067655+0000","last_change":"2024-09-18T07:36:01.988099+0000","last_active":"2024-09-18T07:38:30.067655+0000","last_peered":"2024-09-18T07:38:30.067655+0000","last_clean":"2024-09-18T07:38:30.067655+0000","last_became_active":"2024-09-18T07:35:01.407339+0000","last_became_peered":"2024-09-18T07:35:01.407339+0000","last_unstale":"2024-09-18T07:38:30.067655+0000","last_undegraded":"2024-09-18T07:38:30.067655+0000","last_fullsized":"2024-09-18T07:38:30.067655+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:01.988054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.988054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":525,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051717+0000","last_change":"2024-09-18T07:36:35.210213+0000","last_active":"2024-09-18T07:38:30.051717+0000","last_peered":"2024-09-18T07:38:30.051717+0000","last_clean":"2024-09-18T07:38:30.051717+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:38:30.051717+0000","last_undegraded":"2024-09-18T07:38:30.051717+0000","last_fullsized":"2024-09-18T07:38:30.051717+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:35.209823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:35.209823+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":411,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055232+0000","last_change":"2024-09-18T07:36:07.588483+0000","last_active":"2024-09-18T07:38:30.055232+0000","last_peered":"2024-09-18T07:38:30.055232+0000","last_clean":"2024-09-18T07:38:30.055232+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:38:30.055232+0000","last_undegraded":"2024-09-18T07:38:30.055232+0000","last_fullsized":"2024-09-18T07:38:30.055232+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:07.588440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:07.588440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"258'9624","reported_seq":9751,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:32.376247+0000","last_change":"2024-09-18T07:35:59.286428+0000","last_active":"2024-09-18T07:38:32.376247+0000","last_peered":"2024-09-18T07:38:32.376247+0000","last_clean":"2024-09-18T07:38:32.376247+0000","last_became_active":"2024-09-18T07:35:59.286120+0000","last_became_peered":"2024-09-18T07:35:59.286120+0000","last_unstale":"2024-09-18T07:38:32.376247+0000","last_undegraded":"2024-09-18T07:38:32.376247+0000","last_fullsized":"2024-09-18T07:38:32.376247+0000","mapping_epoch":201,"log_start":"221'4122","ondisk_log_start":"221'4122","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5502,"log_dups_size":0,"ondisk_log_size":5502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":3200,"num_object_clones":0,"num_object_copies":9600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9624,"num_write_kb":410368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":502,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.067425+0000","last_change":"2024-09-18T07:35:23.983838+0000","last_active":"2024-09-18T07:38:30.067425+0000","last_peered":"2024-09-18T07:38:30.067425+0000","last_clean":"2024-09-18T07:38:30.067425+0000","last_became_active":"2024-09-18T07:35:01.418909+0000","last_became_peered":"2024-09-18T07:35:01.418909+0000","last_unstale":"2024-09-18T07:38:30.067425+0000","last_undegraded":"2024-09-18T07:38:30.067425+0000","last_fullsized":"2024-09-18T07:38:30.067425+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:35:23.983762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:23.983762+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":438,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.070560+0000","last_change":"2024-09-18T07:35:27.922013+0000","last_active":"2024-09-18T07:38:30.070560+0000","last_peered":"2024-09-18T07:38:30.070560+0000","last_clean":"2024-09-18T07:38:30.070560+0000","last_became_active":"2024-09-18T07:35:01.407422+0000","last_became_peered":"2024-09-18T07:35:01.407422+0000","last_unstale":"2024-09-18T07:38:30.070560+0000","last_undegraded":"2024-09-18T07:38:30.070560+0000","last_fullsized":"2024-09-18T07:38:30.070560+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:27.921887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:27.921887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":431,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051262+0000","last_change":"2024-09-18T07:36:30.242479+0000","last_active":"2024-09-18T07:38:30.051262+0000","last_peered":"2024-09-18T07:38:30.051262+0000","last_clean":"2024-09-18T07:38:30.051262+0000","last_became_active":"2024-09-18T07:35:01.401024+0000","last_became_peered":"2024-09-18T07:35:01.401024+0000","last_unstale":"2024-09-18T07:38:30.051262+0000","last_undegraded":"2024-09-18T07:38:30.051262+0000","last_fullsized":"2024-09-18T07:38:30.051262+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:30.242420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:30.242420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":397,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051478+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:38:30.051478+0000","last_peered":"2024-09-18T07:38:30.051478+0000","last_clean":"2024-09-18T07:38:30.051478+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:38:30.051478+0000","last_undegraded":"2024-09-18T07:38:30.051478+0000","last_fullsized":"2024-09-18T07:38:30.051478+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"258'9632","reported_seq":9760,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:31.702799+0000","last_change":"2024-09-18T07:35:59.287014+0000","last_active":"2024-09-18T07:38:31.702799+0000","last_peered":"2024-09-18T07:38:31.702799+0000","last_clean":"2024-09-18T07:38:31.702799+0000","last_became_active":"2024-09-18T07:35:59.286688+0000","last_became_peered":"2024-09-18T07:35:59.286688+0000","last_unstale":"2024-09-18T07:38:31.702799+0000","last_undegraded":"2024-09-18T07:38:31.702799+0000","last_fullsized":"2024-09-18T07:38:31.702799+0000","mapping_epoch":201,"log_start":"223'4543","ondisk_log_start":"223'4543","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5089,"log_dups_size":0,"ondisk_log_size":5089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215810072,"num_objects":3294,"num_object_clones":0,"num_object_copies":9882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3294,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9632,"num_write_kb":413569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":480,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051349+0000","last_change":"2024-09-18T07:37:43.153068+0000","last_active":"2024-09-18T07:38:30.051349+0000","last_peered":"2024-09-18T07:38:30.051349+0000","last_clean":"2024-09-18T07:38:30.051349+0000","last_became_active":"2024-09-18T07:37:43.096236+0000","last_became_peered":"2024-09-18T07:37:43.096236+0000","last_unstale":"2024-09-18T07:38:30.051349+0000","last_undegraded":"2024-09-18T07:38:30.051349+0000","last_fullsized":"2024-09-18T07:38:30.051349+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-18T07:35:00.882191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.882191+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":416,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.061440+0000","last_change":"2024-09-18T07:33:24.213432+0000","last_active":"2024-09-18T07:38:30.061440+0000","last_peered":"2024-09-18T07:38:30.061440+0000","last_clean":"2024-09-18T07:38:30.061440+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:38:30.061440+0000","last_undegraded":"2024-09-18T07:38:30.061440+0000","last_fullsized":"2024-09-18T07:38:30.061440+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:24.213356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:24.213356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":516,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.067287+0000","last_change":"2024-09-18T07:36:05.016008+0000","last_active":"2024-09-18T07:38:30.067287+0000","last_peered":"2024-09-18T07:38:30.067287+0000","last_clean":"2024-09-18T07:38:30.067287+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:38:30.067287+0000","last_undegraded":"2024-09-18T07:38:30.067287+0000","last_fullsized":"2024-09-18T07:38:30.067287+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:05.015955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.015955+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":205,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.063747+0000","last_change":"2024-09-18T07:37:43.057439+0000","last_active":"2024-09-18T07:38:30.063747+0000","last_peered":"2024-09-18T07:38:30.063747+0000","last_clean":"2024-09-18T07:38:30.063747+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:38:30.063747+0000","last_undegraded":"2024-09-18T07:38:30.063747+0000","last_fullsized":"2024-09-18T07:38:30.063747+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:25.287352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:25.287352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"258'9786","reported_seq":9913,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:32.374036+0000","last_change":"2024-09-18T07:35:59.102118+0000","last_active":"2024-09-18T07:38:32.374036+0000","last_peered":"2024-09-18T07:38:32.374036+0000","last_clean":"2024-09-18T07:38:32.374036+0000","last_became_active":"2024-09-18T07:35:59.101681+0000","last_became_peered":"2024-09-18T07:35:59.101681+0000","last_unstale":"2024-09-18T07:38:32.374036+0000","last_undegraded":"2024-09-18T07:38:32.374036+0000","last_fullsized":"2024-09-18T07:38:32.374036+0000","mapping_epoch":201,"log_start":"223'4311","ondisk_log_start":"223'4311","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5475,"log_dups_size":0,"ondisk_log_size":5475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204079104,"num_objects":3114,"num_object_clones":0,"num_object_copies":9342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9786,"num_write_kb":412800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":143,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.056150+0000","last_change":"2024-09-18T07:36:13.119192+0000","last_active":"2024-09-18T07:38:30.056150+0000","last_peered":"2024-09-18T07:38:30.056150+0000","last_clean":"2024-09-18T07:38:30.056150+0000","last_became_active":"2024-09-18T07:36:13.118876+0000","last_became_peered":"2024-09-18T07:36:13.118876+0000","last_unstale":"2024-09-18T07:38:30.056150+0000","last_undegraded":"2024-09-18T07:38:30.056150+0000","last_fullsized":"2024-09-18T07:38:30.056150+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":423,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055701+0000","last_change":"2024-09-18T07:35:24.210457+0000","last_active":"2024-09-18T07:38:30.055701+0000","last_peered":"2024-09-18T07:38:30.055701+0000","last_clean":"2024-09-18T07:38:30.055701+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:38:30.055701+0000","last_undegraded":"2024-09-18T07:38:30.055701+0000","last_fullsized":"2024-09-18T07:38:30.055701+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:24.210401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:24.210401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"240'10","reported_seq":537,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055927+0000","last_change":"2024-09-18T07:37:43.170266+0000","last_active":"2024-09-18T07:38:30.055927+0000","last_peered":"2024-09-18T07:38:30.055927+0000","last_clean":"2024-09-18T07:38:30.055927+0000","last_became_active":"2024-09-18T07:37:43.167850+0000","last_became_peered":"2024-09-18T07:37:43.167850+0000","last_unstale":"2024-09-18T07:38:30.055927+0000","last_undegraded":"2024-09-18T07:38:30.055927+0000","last_fullsized":"2024-09-18T07:38:30.055927+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:05.525762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.525762+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":395,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055798+0000","last_change":"2024-09-18T07:31:02.348538+0000","last_active":"2024-09-18T07:38:30.055798+0000","last_peered":"2024-09-18T07:38:30.055798+0000","last_clean":"2024-09-18T07:38:30.055798+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:38:30.055798+0000","last_undegraded":"2024-09-18T07:38:30.055798+0000","last_fullsized":"2024-09-18T07:38:30.055798+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"258'9864","reported_seq":9991,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:31.971489+0000","last_change":"2024-09-18T07:35:59.102330+0000","last_active":"2024-09-18T07:38:31.971489+0000","last_peered":"2024-09-18T07:38:31.971489+0000","last_clean":"2024-09-18T07:38:31.971489+0000","last_became_active":"2024-09-18T07:35:59.101668+0000","last_became_peered":"2024-09-18T07:35:59.101668+0000","last_unstale":"2024-09-18T07:38:31.971489+0000","last_undegraded":"2024-09-18T07:38:31.971489+0000","last_fullsized":"2024-09-18T07:38:31.971489+0000","mapping_epoch":201,"log_start":"226'4800","ondisk_log_start":"226'4800","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5064,"log_dups_size":0,"ondisk_log_size":5064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214040576,"num_objects":3266,"num_object_clones":0,"num_object_copies":9798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9864,"num_write_kb":420160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":505,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.056068+0000","last_change":"2024-09-18T07:36:09.498239+0000","last_active":"2024-09-18T07:38:30.056068+0000","last_peered":"2024-09-18T07:38:30.056068+0000","last_clean":"2024-09-18T07:38:30.056068+0000","last_became_active":"2024-09-18T07:35:01.415275+0000","last_became_peered":"2024-09-18T07:35:01.415275+0000","last_unstale":"2024-09-18T07:38:30.056068+0000","last_undegraded":"2024-09-18T07:38:30.056068+0000","last_fullsized":"2024-09-18T07:38:30.056068+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:36:09.498142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.498142+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":116,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.059017+0000","last_change":"2024-09-18T07:37:46.831436+0000","last_active":"2024-09-18T07:38:30.059017+0000","last_peered":"2024-09-18T07:38:30.059017+0000","last_clean":"2024-09-18T07:38:30.059017+0000","last_became_active":"2024-09-18T07:36:13.118553+0000","last_became_peered":"2024-09-18T07:36:13.118553+0000","last_unstale":"2024-09-18T07:38:30.059017+0000","last_undegraded":"2024-09-18T07:38:30.059017+0000","last_fullsized":"2024-09-18T07:38:30.059017+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:46.831373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:37:46.831373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:13.442615+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":50,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.065364+0000","last_change":"2024-09-18T07:37:43.070306+0000","last_active":"2024-09-18T07:38:30.065364+0000","last_peered":"2024-09-18T07:38:30.065364+0000","last_clean":"2024-09-18T07:38:30.065364+0000","last_became_active":"2024-09-18T07:37:43.070009+0000","last_became_peered":"2024-09-18T07:37:43.070009+0000","last_unstale":"2024-09-18T07:38:30.065364+0000","last_undegraded":"2024-09-18T07:38:30.065364+0000","last_fullsized":"2024-09-18T07:38:30.065364+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":50,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.066979+0000","last_change":"2024-09-18T07:37:43.052077+0000","last_active":"2024-09-18T07:38:30.066979+0000","last_peered":"2024-09-18T07:38:30.066979+0000","last_clean":"2024-09-18T07:38:30.066979+0000","last_became_active":"2024-09-18T07:37:43.051740+0000","last_became_peered":"2024-09-18T07:37:43.051740+0000","last_unstale":"2024-09-18T07:38:30.066979+0000","last_undegraded":"2024-09-18T07:38:30.066979+0000","last_fullsized":"2024-09-18T07:38:30.066979+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"258'10017","reported_seq":10144,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:31.675335+0000","last_change":"2024-09-18T07:35:59.288477+0000","last_active":"2024-09-18T07:38:31.675335+0000","last_peered":"2024-09-18T07:38:31.675335+0000","last_clean":"2024-09-18T07:38:31.675335+0000","last_became_active":"2024-09-18T07:35:59.288058+0000","last_became_peered":"2024-09-18T07:35:59.288058+0000","last_unstale":"2024-09-18T07:38:31.675335+0000","last_undegraded":"2024-09-18T07:38:31.675335+0000","last_fullsized":"2024-09-18T07:38:31.675335+0000","mapping_epoch":201,"log_start":"225'4835","ondisk_log_start":"225'4835","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5182,"log_dups_size":0,"ondisk_log_size":5182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217382912,"num_objects":3317,"num_object_clones":0,"num_object_copies":9951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10017,"num_write_kb":426688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"211'29","reported_seq":206,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.058887+0000","last_change":"2024-09-18T07:36:13.115630+0000","last_active":"2024-09-18T07:38:30.058887+0000","last_peered":"2024-09-18T07:38:30.058887+0000","last_clean":"2024-09-18T07:38:30.058887+0000","last_became_active":"2024-09-18T07:36:13.115278+0000","last_became_peered":"2024-09-18T07:36:13.115278+0000","last_unstale":"2024-09-18T07:38:30.058887+0000","last_undegraded":"2024-09-18T07:38:30.058887+0000","last_fullsized":"2024-09-18T07:38:30.058887+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'28","last_scrub_stamp":"2024-09-18T07:34:31.955336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.955336+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:54.943493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":431,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051623+0000","last_change":"2024-09-18T07:37:43.061414+0000","last_active":"2024-09-18T07:38:30.051623+0000","last_peered":"2024-09-18T07:38:30.051623+0000","last_clean":"2024-09-18T07:38:30.051623+0000","last_became_active":"2024-09-18T07:37:43.060930+0000","last_became_peered":"2024-09-18T07:37:43.060930+0000","last_unstale":"2024-09-18T07:38:30.051623+0000","last_undegraded":"2024-09-18T07:38:30.051623+0000","last_fullsized":"2024-09-18T07:38:30.051623+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:06.908713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.908713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":455,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.056254+0000","last_change":"2024-09-18T07:37:43.155837+0000","last_active":"2024-09-18T07:38:30.056254+0000","last_peered":"2024-09-18T07:38:30.056254+0000","last_clean":"2024-09-18T07:38:30.056254+0000","last_became_active":"2024-09-18T07:37:43.076595+0000","last_became_peered":"2024-09-18T07:37:43.076595+0000","last_unstale":"2024-09-18T07:38:30.056254+0000","last_undegraded":"2024-09-18T07:38:30.056254+0000","last_fullsized":"2024-09-18T07:38:30.056254+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:36:01.529677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.529677+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":399,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.056920+0000","last_change":"2024-09-18T07:31:00.332738+0000","last_active":"2024-09-18T07:38:30.056920+0000","last_peered":"2024-09-18T07:38:30.056920+0000","last_clean":"2024-09-18T07:38:30.056920+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:38:30.056920+0000","last_undegraded":"2024-09-18T07:38:30.056920+0000","last_fullsized":"2024-09-18T07:38:30.056920+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":418,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.062361+0000","last_change":"2024-09-18T07:35:01.419122+0000","last_active":"2024-09-18T07:38:30.062361+0000","last_peered":"2024-09-18T07:38:30.062361+0000","last_clean":"2024-09-18T07:38:30.062361+0000","last_became_active":"2024-09-18T07:35:01.418809+0000","last_became_peered":"2024-09-18T07:35:01.418809+0000","last_unstale":"2024-09-18T07:38:30.062361+0000","last_undegraded":"2024-09-18T07:38:30.062361+0000","last_fullsized":"2024-09-18T07:38:30.062361+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:56.289176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:34:56.289176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"258'9773","reported_seq":9900,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:31.932655+0000","last_change":"2024-09-18T07:35:59.103129+0000","last_active":"2024-09-18T07:38:31.932655+0000","last_peered":"2024-09-18T07:38:31.932655+0000","last_clean":"2024-09-18T07:38:31.932655+0000","last_became_active":"2024-09-18T07:35:59.102853+0000","last_became_peered":"2024-09-18T07:35:59.102853+0000","last_unstale":"2024-09-18T07:38:31.932655+0000","last_undegraded":"2024-09-18T07:38:31.932655+0000","last_fullsized":"2024-09-18T07:38:31.932655+0000","mapping_epoch":201,"log_start":"225'4700","ondisk_log_start":"225'4700","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5073,"log_dups_size":0,"ondisk_log_size":5073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219611136,"num_objects":3351,"num_object_clones":0,"num_object_copies":10053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9773,"num_write_kb":419968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":155,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055463+0000","last_change":"2024-09-18T07:36:13.118278+0000","last_active":"2024-09-18T07:38:30.055463+0000","last_peered":"2024-09-18T07:38:30.055463+0000","last_clean":"2024-09-18T07:38:30.055463+0000","last_became_active":"2024-09-18T07:36:13.117924+0000","last_became_peered":"2024-09-18T07:36:13.117924+0000","last_unstale":"2024-09-18T07:38:30.055463+0000","last_undegraded":"2024-09-18T07:38:30.055463+0000","last_fullsized":"2024-09-18T07:38:30.055463+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-18T07:36:06.083531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.083531+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":122,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.050528+0000","last_change":"2024-09-18T07:37:43.051318+0000","last_active":"2024-09-18T07:38:30.050528+0000","last_peered":"2024-09-18T07:38:30.050528+0000","last_clean":"2024-09-18T07:38:30.050528+0000","last_became_active":"2024-09-18T07:37:43.050971+0000","last_became_peered":"2024-09-18T07:37:43.050971+0000","last_unstale":"2024-09-18T07:38:30.050528+0000","last_undegraded":"2024-09-18T07:38:30.050528+0000","last_fullsized":"2024-09-18T07:38:30.050528+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:04.050757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:04.050757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:58.893486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":434,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051242+0000","last_change":"2024-09-18T07:37:49.895264+0000","last_active":"2024-09-18T07:38:30.051242+0000","last_peered":"2024-09-18T07:38:30.051242+0000","last_clean":"2024-09-18T07:38:30.051242+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:38:30.051242+0000","last_undegraded":"2024-09-18T07:38:30.051242+0000","last_fullsized":"2024-09-18T07:38:30.051242+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:37:49.894707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:37:49.894707+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:04.276523+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":423,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.050936+0000","last_change":"2024-09-18T07:37:47.825723+0000","last_active":"2024-09-18T07:38:30.050936+0000","last_peered":"2024-09-18T07:38:30.050936+0000","last_clean":"2024-09-18T07:38:30.050936+0000","last_became_active":"2024-09-18T07:35:01.403680+0000","last_became_peered":"2024-09-18T07:35:01.403680+0000","last_unstale":"2024-09-18T07:38:30.050936+0000","last_undegraded":"2024-09-18T07:38:30.050936+0000","last_fullsized":"2024-09-18T07:38:30.050936+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:47.825656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:37:47.825656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:17.721036+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":209,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.063984+0000","last_change":"2024-09-18T07:37:43.118570+0000","last_active":"2024-09-18T07:38:30.063984+0000","last_peered":"2024-09-18T07:38:30.063984+0000","last_clean":"2024-09-18T07:38:30.063984+0000","last_became_active":"2024-09-18T07:37:43.060019+0000","last_became_peered":"2024-09-18T07:37:43.060019+0000","last_unstale":"2024-09-18T07:38:30.063984+0000","last_undegraded":"2024-09-18T07:38:30.063984+0000","last_fullsized":"2024-09-18T07:38:30.063984+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:57.655797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.655797+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"258'9758","reported_seq":9940,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:31.740042+0000","last_change":"2024-09-18T07:38:20.242167+0000","last_active":"2024-09-18T07:38:31.740042+0000","last_peered":"2024-09-18T07:38:31.740042+0000","last_clean":"2024-09-18T07:38:31.740042+0000","last_became_active":"2024-09-18T07:36:13.119156+0000","last_became_peered":"2024-09-18T07:36:13.119156+0000","last_unstale":"2024-09-18T07:38:31.740042+0000","last_undegraded":"2024-09-18T07:38:31.740042+0000","last_fullsized":"2024-09-18T07:38:31.740042+0000","mapping_epoch":210,"log_start":"215'3413","ondisk_log_start":"215'3413","created":201,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'9080","last_scrub_stamp":"2024-09-18T07:38:20.241865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:38:20.241865+0000","objects_scrubbed":3965,"log_size":6345,"log_dups_size":0,"ondisk_log_size":6345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:21.802296+0000","scrub_duration":1210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213909504,"num_objects":3264,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9758,"num_write_kb":416704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":442,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.060910+0000","last_change":"2024-09-18T07:33:27.040227+0000","last_active":"2024-09-18T07:38:30.060910+0000","last_peered":"2024-09-18T07:38:30.060910+0000","last_clean":"2024-09-18T07:38:30.060910+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:38:30.060910+0000","last_undegraded":"2024-09-18T07:38:30.060910+0000","last_fullsized":"2024-09-18T07:38:30.060910+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":413,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.064712+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:38:30.064712+0000","last_peered":"2024-09-18T07:38:30.064712+0000","last_clean":"2024-09-18T07:38:30.064712+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:38:30.064712+0000","last_undegraded":"2024-09-18T07:38:30.064712+0000","last_fullsized":"2024-09-18T07:38:30.064712+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":397,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.063116+0000","last_change":"2024-09-18T07:35:01.421692+0000","last_active":"2024-09-18T07:38:30.063116+0000","last_peered":"2024-09-18T07:38:30.063116+0000","last_clean":"2024-09-18T07:38:30.063116+0000","last_became_active":"2024-09-18T07:35:01.400524+0000","last_became_peered":"2024-09-18T07:35:01.400524+0000","last_unstale":"2024-09-18T07:38:30.063116+0000","last_undegraded":"2024-09-18T07:38:30.063116+0000","last_fullsized":"2024-09-18T07:38:30.063116+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":216,"num_read_kb":211,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":415,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055558+0000","last_change":"2024-09-18T07:36:28.551293+0000","last_active":"2024-09-18T07:38:30.055558+0000","last_peered":"2024-09-18T07:38:30.055558+0000","last_clean":"2024-09-18T07:38:30.055558+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:38:30.055558+0000","last_undegraded":"2024-09-18T07:38:30.055558+0000","last_fullsized":"2024-09-18T07:38:30.055558+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:28.551238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:28.551238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":116,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.054460+0000","last_change":"2024-09-18T07:36:39.601424+0000","last_active":"2024-09-18T07:38:30.054460+0000","last_peered":"2024-09-18T07:38:30.054460+0000","last_clean":"2024-09-18T07:38:30.054460+0000","last_became_active":"2024-09-18T07:36:13.117463+0000","last_became_peered":"2024-09-18T07:36:13.117463+0000","last_unstale":"2024-09-18T07:38:30.054460+0000","last_undegraded":"2024-09-18T07:38:30.054460+0000","last_fullsized":"2024-09-18T07:38:30.054460+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:39.601356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:39.601356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"258'9896","reported_seq":17134,"reported_epoch":258,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T07:38:31.695406+0000","last_change":"2024-09-18T07:38:19.382796+0000","last_active":"2024-09-18T07:38:31.695406+0000","last_peered":"2024-09-18T07:38:31.695406+0000","last_clean":"2024-09-18T07:36:40.976872+0000","last_became_active":"2024-09-18T07:37:43.232620+0000","last_became_peered":"2024-09-18T07:37:43.232620+0000","last_unstale":"2024-09-18T07:38:31.695406+0000","last_undegraded":"2024-09-18T07:38:31.695406+0000","last_fullsized":"2024-09-18T07:37:43.046102+0000","mapping_epoch":238,"log_start":"225'4767","ondisk_log_start":"225'4767","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5129,"log_dups_size":0,"ondisk_log_size":5129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218103808,"num_objects":3328,"num_object_clones":0,"num_object_copies":9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1695,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9896,"num_write_kb":423168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3553,"num_bytes_recovered":232783872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,3,5","objects":1695},{"shards":"0,5","objects":1633}],"blocked_by":[],"up_primary":6,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"240'20","reported_seq":78,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.066152+0000","last_change":"2024-09-18T07:37:43.157718+0000","last_active":"2024-09-18T07:38:30.066152+0000","last_peered":"2024-09-18T07:38:30.066152+0000","last_clean":"2024-09-18T07:38:30.066152+0000","last_became_active":"2024-09-18T07:37:43.087496+0000","last_became_peered":"2024-09-18T07:37:43.087496+0000","last_unstale":"2024-09-18T07:38:30.066152+0000","last_undegraded":"2024-09-18T07:38:30.066152+0000","last_fullsized":"2024-09-18T07:38:30.066152+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'17","last_scrub_stamp":"2024-09-18T07:33:33.849285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.849285+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":430,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.061861+0000","last_change":"2024-09-18T07:37:43.065584+0000","last_active":"2024-09-18T07:38:30.061861+0000","last_peered":"2024-09-18T07:38:30.061861+0000","last_clean":"2024-09-18T07:38:30.061861+0000","last_became_active":"2024-09-18T07:37:43.065279+0000","last_became_peered":"2024-09-18T07:37:43.065279+0000","last_unstale":"2024-09-18T07:38:30.061861+0000","last_undegraded":"2024-09-18T07:38:30.061861+0000","last_fullsized":"2024-09-18T07:38:30.061861+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:59.616988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:59.616988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":223,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.063173+0000","last_change":"2024-09-18T07:37:43.121827+0000","last_active":"2024-09-18T07:38:30.063173+0000","last_peered":"2024-09-18T07:38:30.063173+0000","last_clean":"2024-09-18T07:38:30.063173+0000","last_became_active":"2024-09-18T07:37:43.058402+0000","last_became_peered":"2024-09-18T07:37:43.058402+0000","last_unstale":"2024-09-18T07:38:30.063173+0000","last_undegraded":"2024-09-18T07:38:30.063173+0000","last_fullsized":"2024-09-18T07:38:30.063173+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:35:27.278095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:35:27.278095+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":183,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.063971+0000","last_change":"2024-09-18T07:35:01.413545+0000","last_active":"2024-09-18T07:38:30.063971+0000","last_peered":"2024-09-18T07:38:30.063971+0000","last_clean":"2024-09-18T07:38:30.063971+0000","last_became_active":"2024-09-18T07:35:01.412217+0000","last_became_peered":"2024-09-18T07:35:01.412217+0000","last_unstale":"2024-09-18T07:38:30.063971+0000","last_undegraded":"2024-09-18T07:38:30.063971+0000","last_fullsized":"2024-09-18T07:38:30.063971+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:36.964421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:36.964421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":403,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055324+0000","last_change":"2024-09-18T07:33:22.358792+0000","last_active":"2024-09-18T07:38:30.055324+0000","last_peered":"2024-09-18T07:38:30.055324+0000","last_clean":"2024-09-18T07:38:30.055324+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:38:30.055324+0000","last_undegraded":"2024-09-18T07:38:30.055324+0000","last_fullsized":"2024-09-18T07:38:30.055324+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.358746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.358746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"258'9818","reported_seq":9945,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:31.512028+0000","last_change":"2024-09-18T07:35:59.106988+0000","last_active":"2024-09-18T07:38:31.512028+0000","last_peered":"2024-09-18T07:38:31.512028+0000","last_clean":"2024-09-18T07:38:31.512028+0000","last_became_active":"2024-09-18T07:35:59.106659+0000","last_became_peered":"2024-09-18T07:35:59.106659+0000","last_unstale":"2024-09-18T07:38:31.512028+0000","last_undegraded":"2024-09-18T07:38:31.512028+0000","last_fullsized":"2024-09-18T07:38:31.512028+0000","mapping_epoch":201,"log_start":"227'5000","ondisk_log_start":"227'5000","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":4818,"log_dups_size":0,"ondisk_log_size":4818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219152384,"num_objects":3344,"num_object_clones":0,"num_object_copies":10032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9818,"num_write_kb":421184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":516,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051766+0000","last_change":"2024-09-18T07:37:43.229332+0000","last_active":"2024-09-18T07:38:30.051766+0000","last_peered":"2024-09-18T07:38:30.051766+0000","last_clean":"2024-09-18T07:38:30.051766+0000","last_became_active":"2024-09-18T07:37:43.063255+0000","last_became_peered":"2024-09-18T07:37:43.063255+0000","last_unstale":"2024-09-18T07:38:30.051766+0000","last_undegraded":"2024-09-18T07:38:30.051766+0000","last_fullsized":"2024-09-18T07:38:30.051766+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:35:38.928407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:38.928407+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":439,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051760+0000","last_change":"2024-09-18T07:36:32.292127+0000","last_active":"2024-09-18T07:38:30.051760+0000","last_peered":"2024-09-18T07:38:30.051760+0000","last_clean":"2024-09-18T07:38:30.051760+0000","last_became_active":"2024-09-18T07:35:01.401063+0000","last_became_peered":"2024-09-18T07:35:01.401063+0000","last_unstale":"2024-09-18T07:38:30.051760+0000","last_undegraded":"2024-09-18T07:38:30.051760+0000","last_fullsized":"2024-09-18T07:38:30.051760+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:32.292055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:32.292055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":429,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055241+0000","last_change":"2024-09-18T07:37:43.076010+0000","last_active":"2024-09-18T07:38:30.055241+0000","last_peered":"2024-09-18T07:38:30.055241+0000","last_clean":"2024-09-18T07:38:30.055241+0000","last_became_active":"2024-09-18T07:37:43.075535+0000","last_became_peered":"2024-09-18T07:37:43.075535+0000","last_unstale":"2024-09-18T07:38:30.055241+0000","last_undegraded":"2024-09-18T07:38:30.055241+0000","last_fullsized":"2024-09-18T07:38:30.055241+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:31.191528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:31.191528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":426,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.056968+0000","last_change":"2024-09-18T07:37:43.066497+0000","last_active":"2024-09-18T07:38:30.056968+0000","last_peered":"2024-09-18T07:38:30.056968+0000","last_clean":"2024-09-18T07:38:30.056968+0000","last_became_active":"2024-09-18T07:37:43.066153+0000","last_became_peered":"2024-09-18T07:37:43.066153+0000","last_unstale":"2024-09-18T07:38:30.056968+0000","last_undegraded":"2024-09-18T07:38:30.056968+0000","last_fullsized":"2024-09-18T07:38:30.056968+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:38.955353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:38.955353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"258'9813","reported_seq":9946,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:32.341850+0000","last_change":"2024-09-18T07:36:13.120887+0000","last_active":"2024-09-18T07:38:32.341850+0000","last_peered":"2024-09-18T07:38:32.341850+0000","last_clean":"2024-09-18T07:38:32.341850+0000","last_became_active":"2024-09-18T07:36:13.116701+0000","last_became_peered":"2024-09-18T07:36:13.116701+0000","last_unstale":"2024-09-18T07:38:32.341850+0000","last_undegraded":"2024-09-18T07:38:32.341850+0000","last_fullsized":"2024-09-18T07:38:32.341850+0000","mapping_epoch":210,"log_start":"223'4300","ondisk_log_start":"223'4300","created":201,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5513,"log_dups_size":0,"ondisk_log_size":5513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213319680,"num_objects":3255,"num_object_clones":0,"num_object_copies":9765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9813,"num_write_kb":418176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":449,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051964+0000","last_change":"2024-09-18T07:35:33.011021+0000","last_active":"2024-09-18T07:38:30.051964+0000","last_peered":"2024-09-18T07:38:30.051964+0000","last_clean":"2024-09-18T07:38:30.051964+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:38:30.051964+0000","last_undegraded":"2024-09-18T07:38:30.051964+0000","last_fullsized":"2024-09-18T07:38:30.051964+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:35:33.010955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:33.010955+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":50,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.065490+0000","last_change":"2024-09-18T07:37:43.062115+0000","last_active":"2024-09-18T07:38:30.065490+0000","last_peered":"2024-09-18T07:38:30.065490+0000","last_clean":"2024-09-18T07:38:30.065490+0000","last_became_active":"2024-09-18T07:37:43.061808+0000","last_became_peered":"2024-09-18T07:37:43.061808+0000","last_unstale":"2024-09-18T07:38:30.065490+0000","last_undegraded":"2024-09-18T07:38:30.065490+0000","last_fullsized":"2024-09-18T07:38:30.065490+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:35.265259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:35.265259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":108,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051293+0000","last_change":"2024-09-18T07:36:13.116901+0000","last_active":"2024-09-18T07:38:30.051293+0000","last_peered":"2024-09-18T07:38:30.051293+0000","last_clean":"2024-09-18T07:38:30.051293+0000","last_became_active":"2024-09-18T07:36:13.116527+0000","last_became_peered":"2024-09-18T07:36:13.116527+0000","last_unstale":"2024-09-18T07:38:30.051293+0000","last_undegraded":"2024-09-18T07:38:30.051293+0000","last_fullsized":"2024-09-18T07:38:30.051293+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:05.239160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.239160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":421,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.052640+0000","last_change":"2024-09-18T07:36:38.632189+0000","last_active":"2024-09-18T07:38:30.052640+0000","last_peered":"2024-09-18T07:38:30.052640+0000","last_clean":"2024-09-18T07:38:30.052640+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:38:30.052640+0000","last_undegraded":"2024-09-18T07:38:30.052640+0000","last_fullsized":"2024-09-18T07:38:30.052640+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:38.632040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:38.632040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:05.822462+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"258'9807","reported_seq":9933,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:31.964973+0000","last_change":"2024-09-18T07:35:59.107531+0000","last_active":"2024-09-18T07:38:31.964973+0000","last_peered":"2024-09-18T07:38:31.964973+0000","last_clean":"2024-09-18T07:38:31.964973+0000","last_became_active":"2024-09-18T07:35:59.107232+0000","last_became_peered":"2024-09-18T07:35:59.107232+0000","last_unstale":"2024-09-18T07:38:31.964973+0000","last_undegraded":"2024-09-18T07:38:31.964973+0000","last_fullsized":"2024-09-18T07:38:31.964973+0000","mapping_epoch":201,"log_start":"226'4800","ondisk_log_start":"226'4800","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212795392,"num_objects":3247,"num_object_clones":0,"num_object_copies":9741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9807,"num_write_kb":417728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":466,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.052590+0000","last_change":"2024-09-18T07:37:43.106862+0000","last_active":"2024-09-18T07:38:30.052590+0000","last_peered":"2024-09-18T07:38:30.052590+0000","last_clean":"2024-09-18T07:38:30.052590+0000","last_became_active":"2024-09-18T07:37:43.062724+0000","last_became_peered":"2024-09-18T07:37:43.062724+0000","last_unstale":"2024-09-18T07:38:30.052590+0000","last_undegraded":"2024-09-18T07:38:30.052590+0000","last_fullsized":"2024-09-18T07:38:30.052590+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:32:05.000310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.000310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":416,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.057478+0000","last_change":"2024-09-18T07:32:06.688493+0000","last_active":"2024-09-18T07:38:30.057478+0000","last_peered":"2024-09-18T07:38:30.057478+0000","last_clean":"2024-09-18T07:38:30.057478+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:38:30.057478+0000","last_undegraded":"2024-09-18T07:38:30.057478+0000","last_fullsized":"2024-09-18T07:38:30.057478+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:06.688444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:06.688444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"173'11","reported_seq":542,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.052353+0000","last_change":"2024-09-18T07:35:01.413702+0000","last_active":"2024-09-18T07:38:30.052353+0000","last_peered":"2024-09-18T07:38:30.052353+0000","last_clean":"2024-09-18T07:38:30.052353+0000","last_became_active":"2024-09-18T07:35:01.403765+0000","last_became_peered":"2024-09-18T07:35:01.403765+0000","last_unstale":"2024-09-18T07:38:30.052353+0000","last_undegraded":"2024-09-18T07:38:30.052353+0000","last_fullsized":"2024-09-18T07:38:30.052353+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'10","last_scrub_stamp":"2024-09-18T07:33:33.000874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.000874+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:53.964476+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":413,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051495+0000","last_change":"2024-09-18T07:36:33.251164+0000","last_active":"2024-09-18T07:38:30.051495+0000","last_peered":"2024-09-18T07:38:30.051495+0000","last_clean":"2024-09-18T07:38:30.051495+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:38:30.051495+0000","last_undegraded":"2024-09-18T07:38:30.051495+0000","last_fullsized":"2024-09-18T07:38:30.051495+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:33.251093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:33.251093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"258'10100","reported_seq":17229,"reported_epoch":258,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T07:38:31.972277+0000","last_change":"2024-09-18T07:37:43.261591+0000","last_active":"2024-09-18T07:38:31.972277+0000","last_peered":"2024-09-18T07:38:31.972277+0000","last_clean":"2024-09-18T07:36:40.998393+0000","last_became_active":"2024-09-18T07:37:43.222069+0000","last_became_peered":"2024-09-18T07:37:43.222069+0000","last_unstale":"2024-09-18T07:38:31.972277+0000","last_undegraded":"2024-09-18T07:38:31.972277+0000","last_fullsized":"2024-09-18T07:37:43.044165+0000","mapping_epoch":238,"log_start":"226'5000","ondisk_log_start":"226'5000","created":201,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5100,"log_dups_size":0,"ondisk_log_size":5100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220594176,"num_objects":3366,"num_object_clones":0,"num_object_copies":10098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":115,"num_objects_unfound":0,"num_objects_dirty":3366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10100,"num_write_kb":430912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3493,"num_bytes_recovered":228917248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,2,3","objects":115},{"shards":"1,3","objects":3251}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":436,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055972+0000","last_change":"2024-09-18T07:37:28.049996+0000","last_active":"2024-09-18T07:38:30.055972+0000","last_peered":"2024-09-18T07:38:30.055972+0000","last_clean":"2024-09-18T07:38:30.055972+0000","last_became_active":"2024-09-18T07:37:27.994144+0000","last_became_peered":"2024-09-18T07:37:27.994144+0000","last_unstale":"2024-09-18T07:38:30.055972+0000","last_undegraded":"2024-09-18T07:38:30.055972+0000","last_fullsized":"2024-09-18T07:38:30.055972+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":437,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.054601+0000","last_change":"2024-09-18T07:37:43.076416+0000","last_active":"2024-09-18T07:38:30.054601+0000","last_peered":"2024-09-18T07:38:30.054601+0000","last_clean":"2024-09-18T07:38:30.054601+0000","last_became_active":"2024-09-18T07:37:43.075988+0000","last_became_peered":"2024-09-18T07:37:43.075988+0000","last_unstale":"2024-09-18T07:38:30.054601+0000","last_undegraded":"2024-09-18T07:38:30.054601+0000","last_fullsized":"2024-09-18T07:38:30.054601+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:21.222526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.222526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":409,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055668+0000","last_change":"2024-09-18T07:34:31.959675+0000","last_active":"2024-09-18T07:38:30.055668+0000","last_peered":"2024-09-18T07:38:30.055668+0000","last_clean":"2024-09-18T07:38:30.055668+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:38:30.055668+0000","last_undegraded":"2024-09-18T07:38:30.055668+0000","last_fullsized":"2024-09-18T07:38:30.055668+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:31.959636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.959636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:56.892556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":425,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.061309+0000","last_change":"2024-09-18T07:36:07.526609+0000","last_active":"2024-09-18T07:38:30.061309+0000","last_peered":"2024-09-18T07:38:30.061309+0000","last_clean":"2024-09-18T07:38:30.061309+0000","last_became_active":"2024-09-18T07:35:01.410890+0000","last_became_peered":"2024-09-18T07:35:01.410890+0000","last_unstale":"2024-09-18T07:38:30.061309+0000","last_undegraded":"2024-09-18T07:38:30.061309+0000","last_fullsized":"2024-09-18T07:38:30.061309+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:07.526536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:07.526536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"258'9844","reported_seq":17000,"reported_epoch":258,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T07:38:32.385353+0000","last_change":"2024-09-18T07:37:43.243985+0000","last_active":"2024-09-18T07:38:32.385353+0000","last_peered":"2024-09-18T07:38:32.385353+0000","last_clean":"2024-09-18T07:36:40.991141+0000","last_became_active":"2024-09-18T07:37:43.212542+0000","last_became_peered":"2024-09-18T07:37:43.212542+0000","last_unstale":"2024-09-18T07:38:32.385353+0000","last_undegraded":"2024-09-18T07:38:32.385353+0000","last_fullsized":"2024-09-18T07:37:43.044251+0000","mapping_epoch":238,"log_start":"222'4300","ondisk_log_start":"222'4300","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5544,"log_dups_size":0,"ondisk_log_size":5544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209977344,"num_objects":3204,"num_object_clones":0,"num_object_copies":9612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":112,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9844,"num_write_kb":417536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3510,"num_bytes_recovered":230031360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,2,7","objects":112},{"shards":"0,7","objects":3092}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":70,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.065611+0000","last_change":"2024-09-18T07:37:43.142793+0000","last_active":"2024-09-18T07:38:30.065611+0000","last_peered":"2024-09-18T07:38:30.065611+0000","last_clean":"2024-09-18T07:38:30.065611+0000","last_became_active":"2024-09-18T07:37:43.063319+0000","last_became_peered":"2024-09-18T07:37:43.063319+0000","last_unstale":"2024-09-18T07:38:30.065611+0000","last_undegraded":"2024-09-18T07:38:30.065611+0000","last_fullsized":"2024-09-18T07:38:30.065611+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:35:21.344029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.344029+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":205,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.064246+0000","last_change":"2024-09-18T07:37:43.073257+0000","last_active":"2024-09-18T07:38:30.064246+0000","last_peered":"2024-09-18T07:38:30.064246+0000","last_clean":"2024-09-18T07:38:30.064246+0000","last_became_active":"2024-09-18T07:37:43.072894+0000","last_became_peered":"2024-09-18T07:37:43.072894+0000","last_unstale":"2024-09-18T07:38:30.064246+0000","last_undegraded":"2024-09-18T07:38:30.064246+0000","last_fullsized":"2024-09-18T07:38:30.064246+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:28.171448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:28.171448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":424,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.050311+0000","last_change":"2024-09-18T07:37:52.961389+0000","last_active":"2024-09-18T07:38:30.050311+0000","last_peered":"2024-09-18T07:38:30.050311+0000","last_clean":"2024-09-18T07:38:30.050311+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:38:30.050311+0000","last_undegraded":"2024-09-18T07:38:30.050311+0000","last_fullsized":"2024-09-18T07:38:30.050311+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:52.961325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:37:52.961325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:19.975533+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":183,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.063271+0000","last_change":"2024-09-18T07:35:01.408078+0000","last_active":"2024-09-18T07:38:30.063271+0000","last_peered":"2024-09-18T07:38:30.063271+0000","last_clean":"2024-09-18T07:38:30.063271+0000","last_became_active":"2024-09-18T07:35:01.406980+0000","last_became_peered":"2024-09-18T07:35:01.406980+0000","last_unstale":"2024-09-18T07:38:30.063271+0000","last_undegraded":"2024-09-18T07:38:30.063271+0000","last_fullsized":"2024-09-18T07:38:30.063271+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"258'9707","reported_seq":21657,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:31.589815+0000","last_change":"2024-09-18T07:38:20.948583+0000","last_active":"2024-09-18T07:38:31.589815+0000","last_peered":"2024-09-18T07:38:31.589815+0000","last_clean":"2024-09-18T07:38:31.589815+0000","last_became_active":"2024-09-18T07:38:20.929620+0000","last_became_peered":"2024-09-18T07:38:20.929620+0000","last_unstale":"2024-09-18T07:38:31.589815+0000","last_undegraded":"2024-09-18T07:38:31.589815+0000","last_fullsized":"2024-09-18T07:38:31.589815+0000","mapping_epoch":255,"log_start":"222'4200","ondisk_log_start":"222'4200","created":201,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5507,"log_dups_size":0,"ondisk_log_size":5507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219217920,"num_objects":3345,"num_object_clones":0,"num_object_copies":10035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":417664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5903,"num_bytes_recovered":366346240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":444,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051485+0000","last_change":"2024-09-18T07:35:21.085719+0000","last_active":"2024-09-18T07:38:30.051485+0000","last_peered":"2024-09-18T07:38:30.051485+0000","last_clean":"2024-09-18T07:38:30.051485+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:38:30.051485+0000","last_undegraded":"2024-09-18T07:38:30.051485+0000","last_fullsized":"2024-09-18T07:38:30.051485+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:35:21.085670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.085670+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":183,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.068855+0000","last_change":"2024-09-18T07:35:01.400050+0000","last_active":"2024-09-18T07:38:30.068855+0000","last_peered":"2024-09-18T07:38:30.068855+0000","last_clean":"2024-09-18T07:38:30.068855+0000","last_became_active":"2024-09-18T07:35:01.398499+0000","last_became_peered":"2024-09-18T07:35:01.398499+0000","last_unstale":"2024-09-18T07:38:30.068855+0000","last_undegraded":"2024-09-18T07:38:30.068855+0000","last_fullsized":"2024-09-18T07:38:30.068855+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":433,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.050457+0000","last_change":"2024-09-18T07:36:32.564149+0000","last_active":"2024-09-18T07:38:30.050457+0000","last_peered":"2024-09-18T07:38:30.050457+0000","last_clean":"2024-09-18T07:38:30.050457+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:38:30.050457+0000","last_undegraded":"2024-09-18T07:38:30.050457+0000","last_fullsized":"2024-09-18T07:38:30.050457+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:32.564101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:32.564101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:02.841418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":418,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.057221+0000","last_change":"2024-09-18T07:37:43.016535+0000","last_active":"2024-09-18T07:38:30.057221+0000","last_peered":"2024-09-18T07:38:30.057221+0000","last_clean":"2024-09-18T07:38:30.057221+0000","last_became_active":"2024-09-18T07:35:01.418735+0000","last_became_peered":"2024-09-18T07:35:01.418735+0000","last_unstale":"2024-09-18T07:38:30.057221+0000","last_undegraded":"2024-09-18T07:38:30.057221+0000","last_fullsized":"2024-09-18T07:38:30.057221+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:43.016489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:37:43.016489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:12.741671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"258'9856","reported_seq":9983,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:31.707778+0000","last_change":"2024-09-18T07:35:59.107480+0000","last_active":"2024-09-18T07:38:31.707778+0000","last_peered":"2024-09-18T07:38:31.707778+0000","last_clean":"2024-09-18T07:38:31.707778+0000","last_became_active":"2024-09-18T07:35:59.106549+0000","last_became_peered":"2024-09-18T07:35:59.106549+0000","last_unstale":"2024-09-18T07:38:31.707778+0000","last_undegraded":"2024-09-18T07:38:31.707778+0000","last_fullsized":"2024-09-18T07:38:31.707778+0000","mapping_epoch":201,"log_start":"225'4700","ondisk_log_start":"225'4700","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5156,"log_dups_size":0,"ondisk_log_size":5156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220332032,"num_objects":3362,"num_object_clones":0,"num_object_copies":10086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9856,"num_write_kb":422976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":536,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.056910+0000","last_change":"2024-09-18T07:37:43.157488+0000","last_active":"2024-09-18T07:38:30.056910+0000","last_peered":"2024-09-18T07:38:30.056910+0000","last_clean":"2024-09-18T07:38:30.056910+0000","last_became_active":"2024-09-18T07:37:43.075552+0000","last_became_peered":"2024-09-18T07:37:43.075552+0000","last_unstale":"2024-09-18T07:38:30.056910+0000","last_undegraded":"2024-09-18T07:38:30.056910+0000","last_fullsized":"2024-09-18T07:38:30.056910+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:35:36.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:36.271863+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":205,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.067244+0000","last_change":"2024-09-18T07:36:09.261574+0000","last_active":"2024-09-18T07:38:30.067244+0000","last_peered":"2024-09-18T07:38:30.067244+0000","last_clean":"2024-09-18T07:38:30.067244+0000","last_became_active":"2024-09-18T07:35:01.406695+0000","last_became_peered":"2024-09-18T07:35:01.406695+0000","last_unstale":"2024-09-18T07:38:30.067244+0000","last_undegraded":"2024-09-18T07:38:30.067244+0000","last_fullsized":"2024-09-18T07:38:30.067244+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:36:09.261516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.261516+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"240'12","reported_seq":260,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.063587+0000","last_change":"2024-09-18T07:37:43.117484+0000","last_active":"2024-09-18T07:38:30.063587+0000","last_peered":"2024-09-18T07:38:30.063587+0000","last_clean":"2024-09-18T07:38:30.063587+0000","last_became_active":"2024-09-18T07:37:43.056714+0000","last_became_peered":"2024-09-18T07:37:43.056714+0000","last_unstale":"2024-09-18T07:38:30.063587+0000","last_undegraded":"2024-09-18T07:38:30.063587+0000","last_fullsized":"2024-09-18T07:38:30.063587+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:35:26.319320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:26.319320+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":409,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055897+0000","last_change":"2024-09-18T07:37:43.078768+0000","last_active":"2024-09-18T07:38:30.055897+0000","last_peered":"2024-09-18T07:38:30.055897+0000","last_clean":"2024-09-18T07:38:30.055897+0000","last_became_active":"2024-09-18T07:37:43.078121+0000","last_became_peered":"2024-09-18T07:37:43.078121+0000","last_unstale":"2024-09-18T07:38:30.055897+0000","last_undegraded":"2024-09-18T07:38:30.055897+0000","last_fullsized":"2024-09-18T07:38:30.055897+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":50,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.065033+0000","last_change":"2024-09-18T07:37:43.088299+0000","last_active":"2024-09-18T07:38:30.065033+0000","last_peered":"2024-09-18T07:38:30.065033+0000","last_clean":"2024-09-18T07:38:30.065033+0000","last_became_active":"2024-09-18T07:37:43.087994+0000","last_became_peered":"2024-09-18T07:37:43.087994+0000","last_unstale":"2024-09-18T07:38:30.065033+0000","last_undegraded":"2024-09-18T07:38:30.065033+0000","last_fullsized":"2024-09-18T07:38:30.065033+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:01.259296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.259296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":451,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.070351+0000","last_change":"2024-09-18T07:36:09.061983+0000","last_active":"2024-09-18T07:38:30.070351+0000","last_peered":"2024-09-18T07:38:30.070351+0000","last_clean":"2024-09-18T07:38:30.070351+0000","last_became_active":"2024-09-18T07:35:01.407787+0000","last_became_peered":"2024-09-18T07:35:01.407787+0000","last_unstale":"2024-09-18T07:38:30.070351+0000","last_undegraded":"2024-09-18T07:38:30.070351+0000","last_fullsized":"2024-09-18T07:38:30.070351+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-18T07:36:09.061903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.061903+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":415,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.054881+0000","last_change":"2024-09-18T07:36:34.608531+0000","last_active":"2024-09-18T07:38:30.054881+0000","last_peered":"2024-09-18T07:38:30.054881+0000","last_clean":"2024-09-18T07:38:30.054881+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:38:30.054881+0000","last_undegraded":"2024-09-18T07:38:30.054881+0000","last_fullsized":"2024-09-18T07:38:30.054881+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:34.608486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:34.608486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":197,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.064178+0000","last_change":"2024-09-18T07:37:43.079723+0000","last_active":"2024-09-18T07:38:30.064178+0000","last_peered":"2024-09-18T07:38:30.064178+0000","last_clean":"2024-09-18T07:38:30.064178+0000","last_became_active":"2024-09-18T07:37:43.079313+0000","last_became_peered":"2024-09-18T07:37:43.079313+0000","last_unstale":"2024-09-18T07:38:30.064178+0000","last_undegraded":"2024-09-18T07:38:30.064178+0000","last_fullsized":"2024-09-18T07:38:30.064178+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:58.688631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:58.688631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":453,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.061339+0000","last_change":"2024-09-18T07:35:01.448510+0000","last_active":"2024-09-18T07:38:30.061339+0000","last_peered":"2024-09-18T07:38:30.061339+0000","last_clean":"2024-09-18T07:38:30.061339+0000","last_became_active":"2024-09-18T07:35:01.414458+0000","last_became_peered":"2024-09-18T07:35:01.414458+0000","last_unstale":"2024-09-18T07:38:30.061339+0000","last_undegraded":"2024-09-18T07:38:30.061339+0000","last_fullsized":"2024-09-18T07:38:30.061339+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":205,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.063416+0000","last_change":"2024-09-18T07:37:43.057776+0000","last_active":"2024-09-18T07:38:30.063416+0000","last_peered":"2024-09-18T07:38:30.063416+0000","last_clean":"2024-09-18T07:38:30.063416+0000","last_became_active":"2024-09-18T07:37:43.056675+0000","last_became_peered":"2024-09-18T07:37:43.056675+0000","last_unstale":"2024-09-18T07:38:30.063416+0000","last_undegraded":"2024-09-18T07:38:30.063416+0000","last_fullsized":"2024-09-18T07:38:30.063416+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:05.201723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.201723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":430,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.070204+0000","last_change":"2024-09-18T07:35:59.997299+0000","last_active":"2024-09-18T07:38:30.070204+0000","last_peered":"2024-09-18T07:38:30.070204+0000","last_clean":"2024-09-18T07:38:30.070204+0000","last_became_active":"2024-09-18T07:35:01.420133+0000","last_became_peered":"2024-09-18T07:35:01.420133+0000","last_unstale":"2024-09-18T07:38:30.070204+0000","last_undegraded":"2024-09-18T07:38:30.070204+0000","last_fullsized":"2024-09-18T07:38:30.070204+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:59.997249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:59.997249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":419,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.057454+0000","last_change":"2024-09-18T07:37:48.861965+0000","last_active":"2024-09-18T07:38:30.057454+0000","last_peered":"2024-09-18T07:38:30.057454+0000","last_clean":"2024-09-18T07:38:30.057454+0000","last_became_active":"2024-09-18T07:35:01.396305+0000","last_became_peered":"2024-09-18T07:35:01.396305+0000","last_unstale":"2024-09-18T07:38:30.057454+0000","last_undegraded":"2024-09-18T07:38:30.057454+0000","last_fullsized":"2024-09-18T07:38:30.057454+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:48.861784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:37:48.861784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:17.251260+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":461,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055547+0000","last_change":"2024-09-18T07:35:02.342002+0000","last_active":"2024-09-18T07:38:30.055547+0000","last_peered":"2024-09-18T07:38:30.055547+0000","last_clean":"2024-09-18T07:38:30.055547+0000","last_became_active":"2024-09-18T07:35:01.416744+0000","last_became_peered":"2024-09-18T07:35:01.416744+0000","last_unstale":"2024-09-18T07:38:30.055547+0000","last_undegraded":"2024-09-18T07:38:30.055547+0000","last_fullsized":"2024-09-18T07:38:30.055547+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:35:02.341919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.341919+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":429,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.061028+0000","last_change":"2024-09-18T07:35:01.414704+0000","last_active":"2024-09-18T07:38:30.061028+0000","last_peered":"2024-09-18T07:38:30.061028+0000","last_clean":"2024-09-18T07:38:30.061028+0000","last_became_active":"2024-09-18T07:35:01.414024+0000","last_became_peered":"2024-09-18T07:35:01.414024+0000","last_unstale":"2024-09-18T07:38:30.061028+0000","last_undegraded":"2024-09-18T07:38:30.061028+0000","last_fullsized":"2024-09-18T07:38:30.061028+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:35.707148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:35.707148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"173'10","reported_seq":530,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.064171+0000","last_change":"2024-09-18T07:36:08.041535+0000","last_active":"2024-09-18T07:38:30.064171+0000","last_peered":"2024-09-18T07:38:30.064171+0000","last_clean":"2024-09-18T07:38:30.064171+0000","last_became_active":"2024-09-18T07:35:01.407568+0000","last_became_peered":"2024-09-18T07:35:01.407568+0000","last_unstale":"2024-09-18T07:38:30.064171+0000","last_undegraded":"2024-09-18T07:38:30.064171+0000","last_fullsized":"2024-09-18T07:38:30.064171+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:36:08.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:08.041456+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":405,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.052260+0000","last_change":"2024-09-18T07:36:11.910543+0000","last_active":"2024-09-18T07:38:30.052260+0000","last_peered":"2024-09-18T07:38:30.052260+0000","last_clean":"2024-09-18T07:38:30.052260+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:38:30.052260+0000","last_undegraded":"2024-09-18T07:38:30.052260+0000","last_fullsized":"2024-09-18T07:38:30.052260+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:11.910474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:11.910474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":473,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.064318+0000","last_change":"2024-09-18T07:36:02.975929+0000","last_active":"2024-09-18T07:38:30.064318+0000","last_peered":"2024-09-18T07:38:30.064318+0000","last_clean":"2024-09-18T07:38:30.064318+0000","last_became_active":"2024-09-18T07:35:01.407738+0000","last_became_peered":"2024-09-18T07:35:01.407738+0000","last_unstale":"2024-09-18T07:38:30.064318+0000","last_undegraded":"2024-09-18T07:38:30.064318+0000","last_fullsized":"2024-09-18T07:38:30.064318+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:36:02.975863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:02.975863+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":191,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.063899+0000","last_change":"2024-09-18T07:36:08.222898+0000","last_active":"2024-09-18T07:38:30.063899+0000","last_peered":"2024-09-18T07:38:30.063899+0000","last_clean":"2024-09-18T07:38:30.063899+0000","last_became_active":"2024-09-18T07:35:01.400197+0000","last_became_peered":"2024-09-18T07:35:01.400197+0000","last_unstale":"2024-09-18T07:38:30.063899+0000","last_undegraded":"2024-09-18T07:38:30.063899+0000","last_fullsized":"2024-09-18T07:38:30.063899+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:08.222838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:08.222838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":510,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055413+0000","last_change":"2024-09-18T07:37:42.285094+0000","last_active":"2024-09-18T07:38:30.055413+0000","last_peered":"2024-09-18T07:38:30.055413+0000","last_clean":"2024-09-18T07:38:30.055413+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:38:30.055413+0000","last_undegraded":"2024-09-18T07:38:30.055413+0000","last_fullsized":"2024-09-18T07:38:30.055413+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:37:42.285042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:37:42.285042+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:58.228525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":417,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.060394+0000","last_change":"2024-09-18T07:35:01.411256+0000","last_active":"2024-09-18T07:38:30.060394+0000","last_peered":"2024-09-18T07:38:30.060394+0000","last_clean":"2024-09-18T07:38:30.060394+0000","last_became_active":"2024-09-18T07:35:01.410561+0000","last_became_peered":"2024-09-18T07:35:01.410561+0000","last_unstale":"2024-09-18T07:38:30.060394+0000","last_undegraded":"2024-09-18T07:38:30.060394+0000","last_fullsized":"2024-09-18T07:38:30.060394+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.134847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.134847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":478,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.051207+0000","last_change":"2024-09-18T07:37:43.105077+0000","last_active":"2024-09-18T07:38:30.051207+0000","last_peered":"2024-09-18T07:38:30.051207+0000","last_clean":"2024-09-18T07:38:30.051207+0000","last_became_active":"2024-09-18T07:37:43.054073+0000","last_became_peered":"2024-09-18T07:37:43.054073+0000","last_unstale":"2024-09-18T07:38:30.051207+0000","last_undegraded":"2024-09-18T07:38:30.051207+0000","last_fullsized":"2024-09-18T07:38:30.051207+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:36:01.986964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.986964+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":437,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.052681+0000","last_change":"2024-09-18T07:37:43.062429+0000","last_active":"2024-09-18T07:38:30.052681+0000","last_peered":"2024-09-18T07:38:30.052681+0000","last_clean":"2024-09-18T07:38:30.052681+0000","last_became_active":"2024-09-18T07:37:43.061577+0000","last_became_peered":"2024-09-18T07:37:43.061577+0000","last_unstale":"2024-09-18T07:38:30.052681+0000","last_undegraded":"2024-09-18T07:38:30.052681+0000","last_fullsized":"2024-09-18T07:38:30.052681+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":107,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.056121+0000","last_change":"2024-09-18T07:36:13.118114+0000","last_active":"2024-09-18T07:38:30.056121+0000","last_peered":"2024-09-18T07:38:30.056121+0000","last_clean":"2024-09-18T07:38:30.056121+0000","last_became_active":"2024-09-18T07:36:13.117584+0000","last_became_peered":"2024-09-18T07:36:13.117584+0000","last_unstale":"2024-09-18T07:38:30.056121+0000","last_undegraded":"2024-09-18T07:38:30.056121+0000","last_fullsized":"2024-09-18T07:38:30.056121+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":396,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.057025+0000","last_change":"2024-09-18T07:31:02.351247+0000","last_active":"2024-09-18T07:38:30.057025+0000","last_peered":"2024-09-18T07:38:30.057025+0000","last_clean":"2024-09-18T07:38:30.057025+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:38:30.057025+0000","last_undegraded":"2024-09-18T07:38:30.057025+0000","last_fullsized":"2024-09-18T07:38:30.057025+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":292,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055809+0000","last_change":"2024-09-18T07:37:43.138346+0000","last_active":"2024-09-18T07:38:30.055809+0000","last_peered":"2024-09-18T07:38:30.055809+0000","last_clean":"2024-09-18T07:38:30.055809+0000","last_became_active":"2024-09-18T07:37:43.067598+0000","last_became_peered":"2024-09-18T07:37:43.067598+0000","last_unstale":"2024-09-18T07:38:30.055809+0000","last_undegraded":"2024-09-18T07:38:30.055809+0000","last_fullsized":"2024-09-18T07:38:30.055809+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:36:38.638226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:38.638226+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":191,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.064086+0000","last_change":"2024-09-18T07:35:57.293607+0000","last_active":"2024-09-18T07:38:30.064086+0000","last_peered":"2024-09-18T07:38:30.064086+0000","last_clean":"2024-09-18T07:38:30.064086+0000","last_became_active":"2024-09-18T07:35:01.405287+0000","last_became_peered":"2024-09-18T07:35:01.405287+0000","last_unstale":"2024-09-18T07:38:30.064086+0000","last_undegraded":"2024-09-18T07:38:30.064086+0000","last_fullsized":"2024-09-18T07:38:30.064086+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:57.293555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.293555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":408,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.057310+0000","last_change":"2024-09-18T07:35:01.205647+0000","last_active":"2024-09-18T07:38:30.057310+0000","last_peered":"2024-09-18T07:38:30.057310+0000","last_clean":"2024-09-18T07:38:30.057310+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:38:30.057310+0000","last_undegraded":"2024-09-18T07:38:30.057310+0000","last_fullsized":"2024-09-18T07:38:30.057310+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:01.205596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:01.205596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":419,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055435+0000","last_change":"2024-09-18T07:36:37.593789+0000","last_active":"2024-09-18T07:38:30.055435+0000","last_peered":"2024-09-18T07:38:30.055435+0000","last_clean":"2024-09-18T07:38:30.055435+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:38:30.055435+0000","last_undegraded":"2024-09-18T07:38:30.055435+0000","last_fullsized":"2024-09-18T07:38:30.055435+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:37.593724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:37.593724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":515,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.058323+0000","last_change":"2024-09-18T07:37:43.138620+0000","last_active":"2024-09-18T07:38:30.058323+0000","last_peered":"2024-09-18T07:38:30.058323+0000","last_clean":"2024-09-18T07:38:30.058323+0000","last_became_active":"2024-09-18T07:37:43.066208+0000","last_became_peered":"2024-09-18T07:37:43.066208+0000","last_unstale":"2024-09-18T07:38:30.058323+0000","last_undegraded":"2024-09-18T07:38:30.058323+0000","last_fullsized":"2024-09-18T07:38:30.058323+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:36:29.051437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:29.051437+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":429,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055075+0000","last_change":"2024-09-18T07:35:01.415691+0000","last_active":"2024-09-18T07:38:30.055075+0000","last_peered":"2024-09-18T07:38:30.055075+0000","last_clean":"2024-09-18T07:38:30.055075+0000","last_became_active":"2024-09-18T07:35:01.415261+0000","last_became_peered":"2024-09-18T07:35:01.415261+0000","last_unstale":"2024-09-18T07:38:30.055075+0000","last_undegraded":"2024-09-18T07:38:30.055075+0000","last_fullsized":"2024-09-18T07:38:30.055075+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:58.734027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:58.734027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":205,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.064073+0000","last_change":"2024-09-18T07:37:43.057780+0000","last_active":"2024-09-18T07:38:30.064073+0000","last_peered":"2024-09-18T07:38:30.064073+0000","last_clean":"2024-09-18T07:38:30.064073+0000","last_became_active":"2024-09-18T07:37:43.057483+0000","last_became_peered":"2024-09-18T07:37:43.057483+0000","last_unstale":"2024-09-18T07:38:30.064073+0000","last_undegraded":"2024-09-18T07:38:30.064073+0000","last_fullsized":"2024-09-18T07:38:30.064073+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:41.311291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:41.311291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":426,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.057393+0000","last_change":"2024-09-18T07:37:43.068364+0000","last_active":"2024-09-18T07:38:30.057393+0000","last_peered":"2024-09-18T07:38:30.057393+0000","last_clean":"2024-09-18T07:38:30.057393+0000","last_became_active":"2024-09-18T07:37:43.067066+0000","last_became_peered":"2024-09-18T07:37:43.067066+0000","last_unstale":"2024-09-18T07:38:30.057393+0000","last_undegraded":"2024-09-18T07:38:30.057393+0000","last_fullsized":"2024-09-18T07:38:30.057393+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:36.026215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:36.026215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":473,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.067339+0000","last_change":"2024-09-18T07:37:43.134956+0000","last_active":"2024-09-18T07:38:30.067339+0000","last_peered":"2024-09-18T07:38:30.067339+0000","last_clean":"2024-09-18T07:38:30.067339+0000","last_became_active":"2024-09-18T07:37:43.067503+0000","last_became_peered":"2024-09-18T07:37:43.067503+0000","last_unstale":"2024-09-18T07:38:30.067339+0000","last_undegraded":"2024-09-18T07:38:30.067339+0000","last_fullsized":"2024-09-18T07:38:30.067339+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:36:40.908629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:40.908629+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":407,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.056882+0000","last_change":"2024-09-18T07:30:56.283643+0000","last_active":"2024-09-18T07:38:30.056882+0000","last_peered":"2024-09-18T07:38:30.056882+0000","last_clean":"2024-09-18T07:38:30.056882+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:38:30.056882+0000","last_undegraded":"2024-09-18T07:38:30.056882+0000","last_fullsized":"2024-09-18T07:38:30.056882+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":50,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.066880+0000","last_change":"2024-09-18T07:37:43.061403+0000","last_active":"2024-09-18T07:38:30.066880+0000","last_peered":"2024-09-18T07:38:30.066880+0000","last_clean":"2024-09-18T07:38:30.066880+0000","last_became_active":"2024-09-18T07:37:43.061081+0000","last_became_peered":"2024-09-18T07:37:43.061081+0000","last_unstale":"2024-09-18T07:38:30.066880+0000","last_undegraded":"2024-09-18T07:38:30.066880+0000","last_fullsized":"2024-09-18T07:38:30.066880+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:30.345180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:30.345180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":191,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.063311+0000","last_change":"2024-09-18T07:35:29.235047+0000","last_active":"2024-09-18T07:38:30.063311+0000","last_peered":"2024-09-18T07:38:30.063311+0000","last_clean":"2024-09-18T07:38:30.063311+0000","last_became_active":"2024-09-18T07:35:01.400396+0000","last_became_peered":"2024-09-18T07:35:01.400396+0000","last_unstale":"2024-09-18T07:38:30.063311+0000","last_undegraded":"2024-09-18T07:38:30.063311+0000","last_fullsized":"2024-09-18T07:38:30.063311+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:29.234993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:29.234993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":107,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.055464+0000","last_change":"2024-09-18T07:36:13.118009+0000","last_active":"2024-09-18T07:38:30.055464+0000","last_peered":"2024-09-18T07:38:30.055464+0000","last_clean":"2024-09-18T07:38:30.055464+0000","last_became_active":"2024-09-18T07:36:13.117154+0000","last_became_peered":"2024-09-18T07:36:13.117154+0000","last_unstale":"2024-09-18T07:38:30.055464+0000","last_undegraded":"2024-09-18T07:38:30.055464+0000","last_fullsized":"2024-09-18T07:38:30.055464+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.071629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.071629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":430,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.067564+0000","last_change":"2024-09-18T07:35:57.951851+0000","last_active":"2024-09-18T07:38:30.067564+0000","last_peered":"2024-09-18T07:38:30.067564+0000","last_clean":"2024-09-18T07:38:30.067564+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:38:30.067564+0000","last_undegraded":"2024-09-18T07:38:30.067564+0000","last_fullsized":"2024-09-18T07:38:30.067564+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:35:57.951801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.951801+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":62,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.066479+0000","last_change":"2024-09-18T07:37:43.258255+0000","last_active":"2024-09-18T07:38:30.066479+0000","last_peered":"2024-09-18T07:38:30.066479+0000","last_clean":"2024-09-18T07:38:30.066479+0000","last_became_active":"2024-09-18T07:37:43.070186+0000","last_became_peered":"2024-09-18T07:37:43.070186+0000","last_unstale":"2024-09-18T07:38:30.066479+0000","last_undegraded":"2024-09-18T07:38:30.066479+0000","last_fullsized":"2024-09-18T07:38:30.066479+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:34:58.432781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:58.432781+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":429,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-09-18T07:38:30.060762+0000","last_change":"2024-09-18T07:35:25.576024+0000","last_active":"2024-09-18T07:38:30.060762+0000","last_peered":"2024-09-18T07:38:30.060762+0000","last_clean":"2024-09-18T07:38:30.060762+0000","last_became_active":"2024-09-18T07:35:01.410469+0000","last_became_peered":"2024-09-18T07:35:01.410469+0000","last_unstale":"2024-09-18T07:38:30.060762+0000","last_undegraded":"2024-09-18T07:38:30.060762+0000","last_fullsized":"2024-09-18T07:38:30.060762+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:25.575954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:25.575954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3444244504,"num_objects":52556,"num_object_clones":0,"num_object_copies":157668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2713,"num_objects_misplaced":1922,"num_objects_unfound":0,"num_objects_dirty":52556,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157060,"num_write_kb":6707649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19700,"num_bytes_recovered":1270415360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11584483328,"data_stored":11584471112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84655,"ondisk_log_size":84655,"up":48,"acting":44,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":4554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":398,"ondisk_log_size":398,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":216,"num_read_kb":211,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201375,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1265048,"kb_used_data":1062928,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92480104,"statfs":{"total":95995035648,"available":94699626496,"internally_reserved":0,"allocated":1088438272,"data_stored":1087734412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":206764490},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463013,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1039292,"kb_used_data":863412,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92705860,"statfs":{"total":95995035648,"available":94930800640,"internally_reserved":0,"allocated":884133888,"data_stored":883457481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":179763654},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724650,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1966024,"kb_used_data":1704064,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":91779128,"statfs":{"total":95995035648,"available":93981827072,"internally_reserved":0,"allocated":1744961536,"data_stored":1744272997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953585,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1282772,"kb_used_data":1075084,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":92462380,"statfs":{"total":95995035648,"available":94681477120,"internally_reserved":0,"allocated":1100886016,"data_stored":1100185517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1957788,"kb_used_data":1605836,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91787364,"statfs":{"total":95995035648,"available":93990260736,"internally_reserved":0,"allocated":1644376064,"data_stored":1643676417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":360315335},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411451,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2369600,"kb_used_data":2045304,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91375552,"statfs":{"total":95995035648,"available":93568565248,"internally_reserved":0,"allocated":2094391296,"data_stored":2093710738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574978,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1808612,"kb_used_data":1498204,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":91936540,"statfs":{"total":95995035648,"available":94143016960,"internally_reserved":0,"allocated":1534160896,"data_stored":1533445200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738503,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1869096,"kb_used_data":1563032,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":91876056,"statfs":{"total":95995035648,"available":94081081344,"internally_reserved":0,"allocated":1600544768,"data_stored":1599833927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1589051392,"data_stored":1589051392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1516634112,"data_stored":1516634112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2075660288,"data_stored":2075656216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1618542592,"data_stored":1618542592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1089732608,"data_stored":1089732608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1733627904,"data_stored":1733623832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":877264896,"data_stored":877264896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1083969536,"data_stored":1083965464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:38:36.278 INFO:tasks.thrashosds.thrasher:Setting 4.13 to [5, 3, 6, 7, 1, 2] 2024-09-18T07:38:36.278 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.13', '5', '3', '6', '7', '1', '2'] 2024-09-18T07:38:36.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 4.13 5 3 6 7 1 2 2024-09-18T07:38:36.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:36.281 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:36.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:36.353 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:36 smithi137 ceph-mon[29408]: from='client.15555 v1:172.21.15.137:0/3109672872' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:38:36.353 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:36 smithi137 ceph-mon[39175]: from='client.15555 v1:172.21.15.137:0/3109672872' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:38:36.382 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:36.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:36.413 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:36.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:36.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:36 smithi203 ceph-mon[32237]: from='client.15555 v1:172.21.15.137:0/3109672872' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:38:36.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:36.565 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:36.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:36.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:36.667 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:36.707 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:36.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:36.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:36.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:36.841 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:36.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:36.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:36.911 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T07:38:36.939 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:36.943 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:37.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:37.017 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:37.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:37.020 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:37.099 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:37.103 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:37 smithi137 ceph-mon[29408]: pgmap v404: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 618 op/s; 2713/158265 objects degraded (1.714%); 1922/158265 objects misplaced (1.214%); 7.6 MiB/s, 120 objects/s recovering 2024-09-18T07:38:37.103 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:37 smithi137 ceph-mon[39175]: pgmap v404: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 618 op/s; 2713/158265 objects degraded (1.714%); 1922/158265 objects misplaced (1.214%); 7.6 MiB/s, 120 objects/s recovering 2024-09-18T07:38:37.119 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:37.151 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:37.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:37.253 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:37.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:37.281 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:37.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:37.379 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:38:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:38:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:38:37.383 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:37.406 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:37.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:37.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:37.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:37 smithi203 ceph-mon[32237]: pgmap v404: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 144 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 618 op/s; 2713/158265 objects degraded (1.714%); 1922/158265 objects misplaced (1.214%); 7.6 MiB/s, 120 objects/s recovering 2024-09-18T07:38:37.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:38:37 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:38:37.509 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:37.530 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:37.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:37.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:37.632 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:37.655 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:37.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:37.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:37.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:37.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:37.780 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:37.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:37.845 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:38:37.845 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:38:37.865 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:38:37.865 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:38:37.880 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:38:37.880 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:38:37.881 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:37.896 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:38:37.896 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:38:37.903 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:37.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:37.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:37.912 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:38:37.912 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:38:37.929 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:38:37.929 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:38:37.945 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:38:37.945 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:38:37.961 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:38:37.962 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:38:38.004 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:38.028 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:38.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:38.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:38.130 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:38.159 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:38.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:38.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:38.261 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:38.298 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:38.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:38.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:38.400 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:38 smithi137 ceph-mon[29408]: osdmap e260: 8 total, 8 up, 8 in 2024-09-18T07:38:38.400 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:38 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3594840336' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:38.401 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:38 smithi137 ceph-mon[39175]: osdmap e260: 8 total, 8 up, 8 in 2024-09-18T07:38:38.401 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:38 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3594840336' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:38.441 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:38.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:38.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:38.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:38 smithi203 ceph-mon[32237]: osdmap e260: 8 total, 8 up, 8 in 2024-09-18T07:38:38.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:38 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3594840336' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:38.542 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:38.582 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:38.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:38.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:38.684 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:38.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:38.720 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:38.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:38.822 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:38.868 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:38.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:38.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:38.970 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:38.998 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:38.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:39.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:39.100 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:39.134 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:39.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:39.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:39.236 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:39.276 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:39.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:39.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:39.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:39.384 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:39 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:39.384 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:39 smithi137 ceph-mon[29408]: pgmap v406: 148 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 144 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.41k op/s; 2713/141423 objects degraded (1.918%); 1393/141423 objects misplaced (0.985%); 12 MiB/s, 277 objects/s recovering 2024-09-18T07:38:39.385 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:39 smithi137 ceph-mon[29408]: osdmap e261: 8 total, 8 up, 8 in 2024-09-18T07:38:39.385 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:39 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:39.385 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:39 smithi137 ceph-mon[39175]: pgmap v406: 148 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 144 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.41k op/s; 2713/141423 objects degraded (1.918%); 1393/141423 objects misplaced (0.985%); 12 MiB/s, 277 objects/s recovering 2024-09-18T07:38:39.385 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:39 smithi137 ceph-mon[39175]: osdmap e261: 8 total, 8 up, 8 in 2024-09-18T07:38:39.413 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:39.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:39.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:39 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:39.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:39 smithi203 ceph-mon[32237]: pgmap v406: 148 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 144 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.41k op/s; 2713/141423 objects degraded (1.918%); 1393/141423 objects misplaced (0.985%); 12 MiB/s, 277 objects/s recovering 2024-09-18T07:38:39.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:39 smithi203 ceph-mon[32237]: osdmap e261: 8 total, 8 up, 8 in 2024-09-18T07:38:39.515 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:39.550 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:39.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:39.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:39.651 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:39.685 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:39.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:39.778 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:39.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:39.838 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:39.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:39.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:39.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:39.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:39.979 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:39.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:40.080 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:40.118 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:38:40.121 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:40.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:40.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:40.125 INFO:teuthology.orchestra.run.smithi137.stderr:set 4.13 pg_upmap_items mapping to [5->3,6->7,1->2] 2024-09-18T07:38:40.224 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:40.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:40.260 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:40.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:40.264 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:40 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2947370524' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:38:40.264 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:40 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3178001201' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.13", "id": [5, 3, 6, 7, 1, 2]} : dispatch 2024-09-18T07:38:40.265 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:40 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 2713/141423 objects degraded (1.918%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:38:40.265 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:40 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2947370524' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:38:40.265 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:40 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3178001201' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.13", "id": [5, 3, 6, 7, 1, 2]} : dispatch 2024-09-18T07:38:40.265 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:40 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 2713/141423 objects degraded (1.918%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:38:40.362 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:40.404 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:40.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:40.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:40.432 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:38:40.432 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:38:40.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.432 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35171.0:165639 8.a e262/210 8:519f7779:::benchmark_data_smithi137_101942_object60829:head v 262'10294, mlcod=262'10292)", 2024-09-18T07:38:40.432 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:38:40.422353+0000", 2024-09-18T07:38:40.432 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0058092669999999999, 2024-09-18T07:38:40.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058279359999999997, 2024-09-18T07:38:40.433 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:38:40.433 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:38:40.433 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:38:40.433 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:38:40.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:38:40.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.422353+0000", 2024-09-18T07:38:40.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:40.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:38:40.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.422353+0000", 2024-09-18T07:38:40.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:40.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:38:40.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.422354+0000", 2024-09-18T07:38:40.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3480000000000001e-06 2024-09-18T07:38:40.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:38:40.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.422362+0000", 2024-09-18T07:38:40.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.464e-06 2024-09-18T07:38:40.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:38:40.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.422375+0000", 2024-09-18T07:38:40.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3576e-05 2024-09-18T07:38:40.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:38:40.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.422382+0000", 2024-09-18T07:38:40.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9330000000000004e-06 2024-09-18T07:38:40.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:38:40.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.422498+0000", 2024-09-18T07:38:40.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115981 2024-09-18T07:38:40.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:38:40.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.422540+0000", 2024-09-18T07:38:40.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1823999999999998e-05 2024-09-18T07:38:40.437 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:40.437 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:40.437 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:40.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.437 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35171.0:165647 8.a e262/210 8:5c06e866:::benchmark_data_smithi137_101942_object60837:head v 262'10295, mlcod=262'10292)", 2024-09-18T07:38:40.437 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:38:40.425146+0000", 2024-09-18T07:38:40.437 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0030162040000000002, 2024-09-18T07:38:40.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003068164, 2024-09-18T07:38:40.437 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:38:40.437 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:38:40.437 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:38:40.438 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:38:40.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:38:40.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.425146+0000", 2024-09-18T07:38:40.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:40.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:38:40.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.425146+0000", 2024-09-18T07:38:40.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:40.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:38:40.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.425148+0000", 2024-09-18T07:38:40.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.573e-06 2024-09-18T07:38:40.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:38:40.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.425154+0000", 2024-09-18T07:38:40.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0730000000000001e-06 2024-09-18T07:38:40.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:38:40.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.425164+0000", 2024-09-18T07:38:40.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0259e-05 2024-09-18T07:38:40.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:38:40.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.425170+0000", 2024-09-18T07:38:40.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2940000000000002e-06 2024-09-18T07:38:40.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:38:40.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.426417+0000", 2024-09-18T07:38:40.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001246299 2024-09-18T07:38:40.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:38:40.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.426459+0000", 2024-09-18T07:38:40.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1940999999999997e-05 2024-09-18T07:38:40.441 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:40.441 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:40.441 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:40.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.441 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35171.0:165643 8.b e262/201 8:d5e9179a:::benchmark_data_smithi137_101942_object60833:head v 262'10314, mlcod=262'10312)", 2024-09-18T07:38:40.441 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:38:40.425723+0000", 2024-09-18T07:38:40.442 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.002439348, 2024-09-18T07:38:40.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025190680000000002, 2024-09-18T07:38:40.442 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:38:40.442 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:38:40.442 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:38:40.442 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:38:40.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:38:40.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.425723+0000", 2024-09-18T07:38:40.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:40.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:38:40.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.425723+0000", 2024-09-18T07:38:40.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:40.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:38:40.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.425725+0000", 2024-09-18T07:38:40.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5859999999999999e-06 2024-09-18T07:38:40.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:38:40.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.425731+0000", 2024-09-18T07:38:40.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0100000000000001e-06 2024-09-18T07:38:40.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:38:40.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.425744+0000", 2024-09-18T07:38:40.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3519e-05 2024-09-18T07:38:40.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:38:40.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.425751+0000", 2024-09-18T07:38:40.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0520000000000004e-06 2024-09-18T07:38:40.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:38:40.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.425816+0000", 2024-09-18T07:38:40.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5010000000000003e-05 2024-09-18T07:38:40.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:38:40.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.425835+0000", 2024-09-18T07:38:40.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8556999999999999e-05 2024-09-18T07:38:40.446 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:40.446 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:40.446 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:40.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.446 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.35171.0:165653 8.7 e262/260 8:e1587cac:::benchmark_data_smithi137_101942_object60843:head v 262'10664, mlcod=262'10662)", 2024-09-18T07:38:40.446 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:38:40.426921+0000", 2024-09-18T07:38:40.446 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.001241219, 2024-09-18T07:38:40.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001348964, 2024-09-18T07:38:40.447 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:38:40.447 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:38:40.447 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:38:40.447 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:38:40.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:38:40.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.426921+0000", 2024-09-18T07:38:40.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:40.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:38:40.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.426921+0000", 2024-09-18T07:38:40.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:38:40.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:38:40.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.426922+0000", 2024-09-18T07:38:40.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2279999999999999e-06 2024-09-18T07:38:40.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:38:40.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.426929+0000", 2024-09-18T07:38:40.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3790000000000003e-06 2024-09-18T07:38:40.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:38:40.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.427003+0000", 2024-09-18T07:38:40.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3842000000000002e-05 2024-09-18T07:38:40.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:38:40.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.427040+0000", 2024-09-18T07:38:40.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7333999999999999e-05 2024-09-18T07:38:40.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.451 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:38:40.451 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.427230+0000", 2024-09-18T07:38:40.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018993999999999999 2024-09-18T07:38:40.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:38:40.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:38:40.451 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:38:40.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:38:40.427279+0000", 2024-09-18T07:38:40.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9106e-05 2024-09-18T07:38:40.452 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:40.452 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:38:40.452 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:40.452 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:38:40.453 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:38:40.453 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 4 2024-09-18T07:38:40.453 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:38:40.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:40 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2947370524' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:38:40.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:40 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3178001201' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.13", "id": [5, 3, 6, 7, 1, 2]} : dispatch 2024-09-18T07:38:40.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:40 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 2713/141423 objects degraded (1.918%), 1 pg degraded (PG_DEGRADED) 2024-09-18T07:38:40.507 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:40.540 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:40.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:40.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:40.679 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:40.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:40.781 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:40.816 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:40.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:40.917 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:40.961 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:40.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:40.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:41.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:41.098 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:41.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:41.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:41.245 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:41.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:41.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:41.347 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:41.388 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:41 smithi137 ceph-mon[39175]: pgmap v408: 148 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 144 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 936 op/s; 2713/141423 objects degraded (1.918%); 1393/141423 objects misplaced (0.985%); 5.5 MiB/s, 186 objects/s recovering 2024-09-18T07:38:41.388 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:41 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:38:41.388 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:41 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2947370524' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:38:41.388 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:41 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3178001201' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.13", "id": [5, 3, 6, 7, 1, 2]}]': finished 2024-09-18T07:38:41.388 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:41 smithi137 ceph-mon[39175]: osdmap e262: 8 total, 8 up, 8 in 2024-09-18T07:38:41.389 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:41 smithi137 ceph-mon[29408]: pgmap v408: 148 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 144 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 936 op/s; 2713/141423 objects degraded (1.918%); 1393/141423 objects misplaced (0.985%); 5.5 MiB/s, 186 objects/s recovering 2024-09-18T07:38:41.389 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:41 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:38:41.389 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:41 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2947370524' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:38:41.389 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:41 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3178001201' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.13", "id": [5, 3, 6, 7, 1, 2]}]': finished 2024-09-18T07:38:41.389 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:41 smithi137 ceph-mon[29408]: osdmap e262: 8 total, 8 up, 8 in 2024-09-18T07:38:41.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:41.391 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:41.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:41.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:41 smithi203 ceph-mon[32237]: pgmap v408: 148 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 144 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 936 op/s; 2713/141423 objects degraded (1.918%); 1393/141423 objects misplaced (0.985%); 5.5 MiB/s, 186 objects/s recovering 2024-09-18T07:38:41.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:41 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:38:41.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:41 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2947370524' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:38:41.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:41 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3178001201' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.13", "id": [5, 3, 6, 7, 1, 2]}]': finished 2024-09-18T07:38:41.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:41 smithi203 ceph-mon[32237]: osdmap e262: 8 total, 8 up, 8 in 2024-09-18T07:38:41.492 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:41.526 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:41.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:41.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:41.628 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:41.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:41.663 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:41.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:41.761 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:38:41.764 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:41.802 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:41.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:41.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:41.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:41.951 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:41.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:42.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:42.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:42.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:42.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:42.191 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:42.231 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:42.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:42.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:42.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T07:38:42.333 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:42.368 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:42.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:42.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:42.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:42 smithi137 ceph-mon[29408]: osdmap e263: 8 total, 8 up, 8 in 2024-09-18T07:38:42.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:42 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:38:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:42 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:38:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:42 smithi137 ceph-mon[39175]: osdmap e263: 8 total, 8 up, 8 in 2024-09-18T07:38:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:42 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:38:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:42 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:38:42.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:42 smithi203 ceph-mon[32237]: osdmap e263: 8 total, 8 up, 8 in 2024-09-18T07:38:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:42 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:38:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:42 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:38:42.470 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:42.506 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:42.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:42.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:42.608 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:42.651 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:42.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:42.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:42.753 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:42.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:42.786 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:42.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:42.887 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:42.912 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:42.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:42.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:43.017 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:43.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:43.040 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:43.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:43.142 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:43.176 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:43.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:43.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:43.278 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:43.310 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:43.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:43.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:43.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:43 smithi137 ceph-mon[29408]: pgmap v411: 148 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 144 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 357 op/s; 2713/140652 objects degraded (1.929%); 1393/140652 objects misplaced (0.990%); 1.6 MiB/s, 89 objects/s recovering 2024-09-18T07:38:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:43 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:38:43.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:43 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:38:43.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:43 smithi137 ceph-mon[29408]: osdmap e264: 8 total, 8 up, 8 in 2024-09-18T07:38:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:43 smithi137 ceph-mon[39175]: pgmap v411: 148 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 144 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 357 op/s; 2713/140652 objects degraded (1.929%); 1393/140652 objects misplaced (0.990%); 1.6 MiB/s, 89 objects/s recovering 2024-09-18T07:38:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:43 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:38:43.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:43 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:38:43.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:43 smithi137 ceph-mon[39175]: osdmap e264: 8 total, 8 up, 8 in 2024-09-18T07:38:43.448 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:43.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:43.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:43.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:43 smithi203 ceph-mon[32237]: pgmap v411: 148 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 144 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 357 op/s; 2713/140652 objects degraded (1.929%); 1393/140652 objects misplaced (0.990%); 1.6 MiB/s, 89 objects/s recovering 2024-09-18T07:38:43.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:43 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:38:43.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:43 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:38:43.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:43 smithi203 ceph-mon[32237]: osdmap e264: 8 total, 8 up, 8 in 2024-09-18T07:38:43.551 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:43.554 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:43.578 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:43.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:43.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:43.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:43.723 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:43.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:43.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:43.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:38:43.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:43.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:43.884 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:43.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:43.985 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:44.019 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:44.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:44.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:44.121 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:44.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:44.160 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:44.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:44.262 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:44.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:44.304 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:44.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:44.405 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:44.442 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:44.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:44.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:44.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:44 smithi203 ceph-mon[32237]: pgmap v413: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 937 B/s rd, 0 B/s wr, 864 op/s; 3095/127275 objects misplaced (2.432%); 8.4 MiB/s, 134 objects/s recovering 2024-09-18T07:38:44.544 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:44.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:44.578 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:44.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:44.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:44 smithi137 ceph-mon[29408]: pgmap v413: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 937 B/s rd, 0 B/s wr, 864 op/s; 3095/127275 objects misplaced (2.432%); 8.4 MiB/s, 134 objects/s recovering 2024-09-18T07:38:44.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:44 smithi137 ceph-mon[39175]: pgmap v413: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 937 B/s rd, 0 B/s wr, 864 op/s; 3095/127275 objects misplaced (2.432%); 8.4 MiB/s, 134 objects/s recovering 2024-09-18T07:38:44.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:44.714 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:44.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:44.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:44.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:44.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:44.837 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:44.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:44.938 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:44.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:44.978 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:44.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:45.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:45.102 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:45.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:45.204 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:45.246 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:45.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:45.348 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:45.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:45.387 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:45.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:45 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-18T07:38:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:45 smithi203 ceph-mon[32237]: osdmap e265: 8 total, 8 up, 8 in 2024-09-18T07:38:45.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:45.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:45 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-18T07:38:45.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:45 smithi137 ceph-mon[29408]: osdmap e265: 8 total, 8 up, 8 in 2024-09-18T07:38:45.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:45 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-18T07:38:45.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:45 smithi137 ceph-mon[39175]: osdmap e265: 8 total, 8 up, 8 in 2024-09-18T07:38:45.526 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:45.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:45.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:45.559 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:45.628 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:45.671 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:45.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:45.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:45.773 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:45.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:45.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:45.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:45.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:45.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:45.946 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:45.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:45.998 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:38:45.999 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:38:46.036 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:38:46.036 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:38:46.046 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:46.062 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:38:46.063 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:38:46.078 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:46.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:46.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:46.100 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:38:46.101 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:38:46.129 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:38:46.130 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:38:46.169 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:38:46.169 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:38:46.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:46.202 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:38:46.203 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:38:46.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:46.227 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:46.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:46.237 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:38:46.237 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:38:46.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:46.366 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:46.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:46.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:46.380 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:46 smithi137 ceph-mon[29408]: pgmap v415: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 937 B/s rd, 0 B/s wr, 864 op/s; 3095/127275 objects misplaced (2.432%); 8.4 MiB/s, 134 objects/s recovering 2024-09-18T07:38:46.380 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:46 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2925937744' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:46.381 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:46 smithi137 ceph-mon[39175]: pgmap v415: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 937 B/s rd, 0 B/s wr, 864 op/s; 3095/127275 objects misplaced (2.432%); 8.4 MiB/s, 134 objects/s recovering 2024-09-18T07:38:46.381 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:46 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2925937744' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:46.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:46 smithi203 ceph-mon[32237]: pgmap v415: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 937 B/s rd, 0 B/s wr, 864 op/s; 3095/127275 objects misplaced (2.432%); 8.4 MiB/s, 134 objects/s recovering 2024-09-18T07:38:46.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:46 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2925937744' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:46.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:46.501 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:46.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:46.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:46.545 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:38:46.545 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:38:46.603 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:46.639 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:46.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:46.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:46.741 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:46.772 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:46.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:46.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:46.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:46.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:46.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:46.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:47.017 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:47.056 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:47.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:47.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:47.158 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:47.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:47.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:47.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:47.293 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:47.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:47.332 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:47.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:47.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:47 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:38:47.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:47 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:38:47.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:38:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:38:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:38:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:47 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:38:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:47 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:38:47.433 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:47.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:47 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:38:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:47 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:38:47.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:38:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:38:47.464 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:47.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:47.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:47.503 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:47.566 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:47.601 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:47.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:47.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:47.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:47.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:47.737 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:47.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:47.838 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:47.862 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:47.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:47.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:47.964 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:47.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:47.992 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:47.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:48.093 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:48.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:48.126 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:48.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:48.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:48.260 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:48.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:48.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:48.362 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:48.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T07:38:48.393 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:48.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:48.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:48.495 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:48.504 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:48 smithi137 ceph-mon[29408]: osdmap e266: 8 total, 8 up, 8 in 2024-09-18T07:38:48.504 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:48 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:48.504 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:48 smithi137 ceph-mon[29408]: pgmap v417: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 1.62k op/s; 2386/111414 objects misplaced (2.142%); 15 MiB/s, 238 objects/s recovering 2024-09-18T07:38:48.505 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:48 smithi137 ceph-mon[39175]: osdmap e266: 8 total, 8 up, 8 in 2024-09-18T07:38:48.505 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:48 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:48.505 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:48 smithi137 ceph-mon[39175]: pgmap v417: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 1.62k op/s; 2386/111414 objects misplaced (2.142%); 15 MiB/s, 238 objects/s recovering 2024-09-18T07:38:48.530 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:48.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:48.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:48.539 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:38:48.539 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:38:48.539 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-18T07:38:48.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:38:48.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:48.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:48.705 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:48.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:48.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:48 smithi203 ceph-mon[32237]: osdmap e266: 8 total, 8 up, 8 in 2024-09-18T07:38:48.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:48 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:48 smithi203 ceph-mon[32237]: pgmap v417: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 1.62k op/s; 2386/111414 objects misplaced (2.142%); 15 MiB/s, 238 objects/s recovering 2024-09-18T07:38:48.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:48.835 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:48.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:48.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:48.937 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:48.965 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:48.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:48.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:49.043 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:49.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:49.074 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:49.127 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:49.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:49.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:49.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:49.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:49.274 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:49.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:49.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:38:49.375 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:49.414 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:49.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:49.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:49.516 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:49.550 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:49.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:49.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:49.652 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:49.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:49.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:49.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:49.791 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:49.826 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:49.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:49.842 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:38:49.842 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:38:49.928 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:49.965 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:49.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:49.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:50.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:50.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:50.107 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:50.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:50.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:50.243 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:50.244 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:38:50.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:50.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:50.346 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:50.380 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:50.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:50.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:50.482 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:50.508 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:50 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:38:50.509 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:50 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:38:50.509 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:50 smithi137 ceph-mon[29408]: osdmap e267: 8 total, 8 up, 8 in 2024-09-18T07:38:50.509 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:50 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1257796310' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:38:50.509 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:50 smithi137 ceph-mon[29408]: pgmap v419: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 878 op/s; 2386/111414 objects misplaced (2.142%); 7.4 MiB/s, 117 objects/s recovering 2024-09-18T07:38:50.509 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:50 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:38:50.509 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:50 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:38:50.509 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:50 smithi137 ceph-mon[39175]: osdmap e267: 8 total, 8 up, 8 in 2024-09-18T07:38:50.509 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:50 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1257796310' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:38:50.509 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:50 smithi137 ceph-mon[39175]: pgmap v419: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 878 op/s; 2386/111414 objects misplaced (2.142%); 7.4 MiB/s, 117 objects/s recovering 2024-09-18T07:38:50.521 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:50.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:50.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:50.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:50.659 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:50.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:50.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:50.661 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T07:38:50.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:50 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:38:50.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:50 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:38:50.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:50 smithi203 ceph-mon[32237]: osdmap e267: 8 total, 8 up, 8 in 2024-09-18T07:38:50.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:50 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1257796310' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:38:50.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:50 smithi203 ceph-mon[32237]: pgmap v419: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 878 op/s; 2386/111414 objects misplaced (2.142%); 7.4 MiB/s, 117 objects/s recovering 2024-09-18T07:38:50.761 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:50.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:50.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:50.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:50.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:50.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:50.949 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:50.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:51.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:51.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:51.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:51.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:51.187 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:51.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:51.224 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:51.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:51.270 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:38:51.273 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:38:51.325 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:51.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:51.358 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:51.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:51.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:51 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:38:51.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:51 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1257796310' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:38:51.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:51 smithi137 ceph-mon[29408]: osdmap e268: 8 total, 8 up, 8 in 2024-09-18T07:38:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:51 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:38:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:51 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1257796310' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:38:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:51 smithi137 ceph-mon[39175]: osdmap e268: 8 total, 8 up, 8 in 2024-09-18T07:38:51.460 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:51.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:51.492 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:51.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:51.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:51.630 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:51.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:51.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:51 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:38:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:51 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1257796310' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:38:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:51 smithi203 ceph-mon[32237]: osdmap e268: 8 total, 8 up, 8 in 2024-09-18T07:38:51.732 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:51.761 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:51.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:51.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:51.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:51.896 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:51.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:51.998 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:52.028 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:52.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:52.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:52.134 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:52.180 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:52.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:52.282 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:52.319 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:52.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:52 smithi137 ceph-mon[29408]: 3.19 scrub starts 2024-09-18T07:38:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:52 smithi137 ceph-mon[29408]: 3.19 scrub ok 2024-09-18T07:38:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:52 smithi137 ceph-mon[29408]: from='client.25507 v1:172.21.15.137:0/3195244161' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:38:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:52 smithi137 ceph-mon[29408]: pgmap v421: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 2.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 938 op/s; 2386/110409 objects misplaced (2.161%); 7.4 MiB/s, 118 objects/s recovering 2024-09-18T07:38:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:52 smithi137 ceph-mon[39175]: 3.19 scrub starts 2024-09-18T07:38:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:52 smithi137 ceph-mon[39175]: 3.19 scrub ok 2024-09-18T07:38:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:52 smithi137 ceph-mon[39175]: from='client.25507 v1:172.21.15.137:0/3195244161' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:38:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:52 smithi137 ceph-mon[39175]: pgmap v421: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 2.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 938 op/s; 2386/110409 objects misplaced (2.161%); 7.4 MiB/s, 118 objects/s recovering 2024-09-18T07:38:52.420 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:52.455 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:52.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:52.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:52.557 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:52.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:52.601 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:52.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:52.611 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-18T07:38:50.242229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2421365299,"num_objects":37138,"num_object_clones":0,"num_object_copies":111414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2386,"num_objects_unfound":0,"num_objects_dirty":37138,"num_whiteouts":0,"num_read":1048,"num_read_kb":907,"num_write":173226,"num_write_kb":6709322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22542,"num_bytes_recovered":1402347673,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85838,"ondisk_log_size":85838,"up":444,"acting":442,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":449,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10624312,"kb_used_data":8444968,"kb_used_omap":12,"kb_used_meta":2179059,"kb_avail":739336904,"statfs":{"total":767960285184,"available":757080989696,"internally_reserved":0,"allocated":8647647232,"data_stored":8643799521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2231356976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,6,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-346488832,"num_objects":-5287,"num_object_clones":0,"num_object_copies":-15861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-709,"num_objects_unfound":0,"num_objects_dirty":-5287,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5287,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":709,"num_bytes_recovered":46465024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.321434"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":439,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.221797+0000","last_change":"2024-09-18T07:35:28.534496+0000","last_active":"2024-09-18T07:38:47.221797+0000","last_peered":"2024-09-18T07:38:47.221797+0000","last_clean":"2024-09-18T07:38:47.221797+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:38:47.221797+0000","last_undegraded":"2024-09-18T07:38:47.221797+0000","last_fullsized":"2024-09-18T07:38:47.221797+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:28.534391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:28.534391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":502,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.924848+0000","last_change":"2024-09-18T07:37:43.103765+0000","last_active":"2024-09-18T07:38:44.924848+0000","last_peered":"2024-09-18T07:38:44.924848+0000","last_clean":"2024-09-18T07:38:44.924848+0000","last_became_active":"2024-09-18T07:37:43.061125+0000","last_became_peered":"2024-09-18T07:37:43.061125+0000","last_unstale":"2024-09-18T07:38:44.924848+0000","last_undegraded":"2024-09-18T07:38:44.924848+0000","last_fullsized":"2024-09-18T07:38:44.924848+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:35:27.883281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:27.883281+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":379,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929567+0000","last_change":"2024-09-18T07:36:38.570405+0000","last_active":"2024-09-18T07:38:44.929567+0000","last_peered":"2024-09-18T07:38:44.929567+0000","last_clean":"2024-09-18T07:38:44.929567+0000","last_became_active":"2024-09-18T07:36:13.128805+0000","last_became_peered":"2024-09-18T07:36:13.128805+0000","last_unstale":"2024-09-18T07:38:44.929567+0000","last_undegraded":"2024-09-18T07:38:44.929567+0000","last_fullsized":"2024-09-18T07:38:44.929567+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:38.570339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:38.570339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":419,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.925686+0000","last_change":"2024-09-18T07:32:03.953881+0000","last_active":"2024-09-18T07:38:44.925686+0000","last_peered":"2024-09-18T07:38:44.925686+0000","last_clean":"2024-09-18T07:38:44.925686+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:38:44.925686+0000","last_undegraded":"2024-09-18T07:38:44.925686+0000","last_fullsized":"2024-09-18T07:38:44.925686+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.953827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.953827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:37.927142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":334,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.220544+0000","last_change":"2024-09-18T07:37:43.056617+0000","last_active":"2024-09-18T07:38:47.220544+0000","last_peered":"2024-09-18T07:38:47.220544+0000","last_clean":"2024-09-18T07:38:47.220544+0000","last_became_active":"2024-09-18T07:37:43.056380+0000","last_became_peered":"2024-09-18T07:37:43.056380+0000","last_unstale":"2024-09-18T07:38:47.220544+0000","last_undegraded":"2024-09-18T07:38:47.220544+0000","last_fullsized":"2024-09-18T07:38:47.220544+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:01.481514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:01.481514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":444,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.937206+0000","last_change":"2024-09-18T07:37:43.066814+0000","last_active":"2024-09-18T07:38:44.937206+0000","last_peered":"2024-09-18T07:38:44.937206+0000","last_clean":"2024-09-18T07:38:44.937206+0000","last_became_active":"2024-09-18T07:37:43.066508+0000","last_became_peered":"2024-09-18T07:37:43.066508+0000","last_unstale":"2024-09-18T07:38:44.937206+0000","last_undegraded":"2024-09-18T07:38:44.937206+0000","last_fullsized":"2024-09-18T07:38:44.937206+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.006488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.006488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":451,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.932719+0000","last_change":"2024-09-18T07:33:22.032321+0000","last_active":"2024-09-18T07:38:44.932719+0000","last_peered":"2024-09-18T07:38:44.932719+0000","last_clean":"2024-09-18T07:38:44.932719+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:38:44.932719+0000","last_undegraded":"2024-09-18T07:38:44.932719+0000","last_fullsized":"2024-09-18T07:38:44.932719+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:22.032110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.032110+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":445,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.924764+0000","last_change":"2024-09-18T07:37:43.052675+0000","last_active":"2024-09-18T07:38:44.924764+0000","last_peered":"2024-09-18T07:38:44.924764+0000","last_clean":"2024-09-18T07:38:44.924764+0000","last_became_active":"2024-09-18T07:37:43.052346+0000","last_became_peered":"2024-09-18T07:37:43.052346+0000","last_unstale":"2024-09-18T07:38:44.924764+0000","last_undegraded":"2024-09-18T07:38:44.924764+0000","last_fullsized":"2024-09-18T07:38:44.924764+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:30.995282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:30.995282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":451,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.928703+0000","last_change":"2024-09-18T07:37:43.080879+0000","last_active":"2024-09-18T07:38:44.928703+0000","last_peered":"2024-09-18T07:38:44.928703+0000","last_clean":"2024-09-18T07:38:44.928703+0000","last_became_active":"2024-09-18T07:37:43.080314+0000","last_became_peered":"2024-09-18T07:37:43.080314+0000","last_unstale":"2024-09-18T07:38:44.928703+0000","last_undegraded":"2024-09-18T07:38:44.928703+0000","last_fullsized":"2024-09-18T07:38:44.928703+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:00.392466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.392466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":443,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.926593+0000","last_change":"2024-09-18T07:32:02.002919+0000","last_active":"2024-09-18T07:38:44.926593+0000","last_peered":"2024-09-18T07:38:44.926593+0000","last_clean":"2024-09-18T07:38:44.926593+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:38:44.926593+0000","last_undegraded":"2024-09-18T07:38:44.926593+0000","last_fullsized":"2024-09-18T07:38:44.926593+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.002811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.002811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:36.971075+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":435,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.942499+0000","last_change":"2024-09-18T07:35:04.267623+0000","last_active":"2024-09-18T07:38:44.942499+0000","last_peered":"2024-09-18T07:38:44.942499+0000","last_clean":"2024-09-18T07:38:44.942499+0000","last_became_active":"2024-09-18T07:35:01.413443+0000","last_became_peered":"2024-09-18T07:35:01.413443+0000","last_unstale":"2024-09-18T07:38:44.942499+0000","last_undegraded":"2024-09-18T07:38:44.942499+0000","last_fullsized":"2024-09-18T07:38:44.942499+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:04.267580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:04.267580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":433,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.219866+0000","last_change":"2024-09-18T07:36:10.551960+0000","last_active":"2024-09-18T07:38:47.219866+0000","last_peered":"2024-09-18T07:38:47.219866+0000","last_clean":"2024-09-18T07:38:47.219866+0000","last_became_active":"2024-09-18T07:35:01.410284+0000","last_became_peered":"2024-09-18T07:35:01.410284+0000","last_unstale":"2024-09-18T07:38:47.219866+0000","last_undegraded":"2024-09-18T07:38:47.219866+0000","last_fullsized":"2024-09-18T07:38:47.219866+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:10.551889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:10.551889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":421,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.927719+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:38:44.927719+0000","last_peered":"2024-09-18T07:38:44.927719+0000","last_clean":"2024-09-18T07:38:44.927719+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:38:44.927719+0000","last_undegraded":"2024-09-18T07:38:44.927719+0000","last_fullsized":"2024-09-18T07:38:44.927719+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":450,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.925757+0000","last_change":"2024-09-18T07:36:05.903851+0000","last_active":"2024-09-18T07:38:44.925757+0000","last_peered":"2024-09-18T07:38:44.925757+0000","last_clean":"2024-09-18T07:38:44.925757+0000","last_became_active":"2024-09-18T07:36:02.526212+0000","last_became_peered":"2024-09-18T07:36:02.526212+0000","last_unstale":"2024-09-18T07:38:44.925757+0000","last_undegraded":"2024-09-18T07:38:44.925757+0000","last_fullsized":"2024-09-18T07:38:44.925757+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-18T07:36:05.903784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.903784+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":197,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.931705+0000","last_change":"2024-09-18T07:35:01.399995+0000","last_active":"2024-09-18T07:38:44.931705+0000","last_peered":"2024-09-18T07:38:44.931705+0000","last_clean":"2024-09-18T07:38:44.931705+0000","last_became_active":"2024-09-18T07:35:01.398605+0000","last_became_peered":"2024-09-18T07:35:01.398605+0000","last_unstale":"2024-09-18T07:38:44.931705+0000","last_undegraded":"2024-09-18T07:38:44.931705+0000","last_fullsized":"2024-09-18T07:38:44.931705+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:55.843864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:55.843864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":433,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929645+0000","last_change":"2024-09-18T07:35:35.248892+0000","last_active":"2024-09-18T07:38:44.929645+0000","last_peered":"2024-09-18T07:38:44.929645+0000","last_clean":"2024-09-18T07:38:44.929645+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:38:44.929645+0000","last_undegraded":"2024-09-18T07:38:44.929645+0000","last_fullsized":"2024-09-18T07:38:44.929645+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:35.248824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:35.248824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":441,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.928398+0000","last_change":"2024-09-18T07:37:43.067972+0000","last_active":"2024-09-18T07:38:44.928398+0000","last_peered":"2024-09-18T07:38:44.928398+0000","last_clean":"2024-09-18T07:38:44.928398+0000","last_became_active":"2024-09-18T07:37:43.067568+0000","last_became_peered":"2024-09-18T07:37:43.067568+0000","last_unstale":"2024-09-18T07:38:44.928398+0000","last_undegraded":"2024-09-18T07:38:44.928398+0000","last_fullsized":"2024-09-18T07:38:44.928398+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:24.929124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:24.929124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":453,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.221197+0000","last_change":"2024-09-18T07:36:06.523357+0000","last_active":"2024-09-18T07:38:47.221197+0000","last_peered":"2024-09-18T07:38:47.221197+0000","last_clean":"2024-09-18T07:38:47.221197+0000","last_became_active":"2024-09-18T07:35:01.414088+0000","last_became_peered":"2024-09-18T07:35:01.414088+0000","last_unstale":"2024-09-18T07:38:47.221197+0000","last_undegraded":"2024-09-18T07:38:47.221197+0000","last_fullsized":"2024-09-18T07:38:47.221197+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:06.523292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.523292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":498,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.930905+0000","last_change":"2024-09-18T07:37:28.072530+0000","last_active":"2024-09-18T07:38:44.930905+0000","last_peered":"2024-09-18T07:38:44.930905+0000","last_clean":"2024-09-18T07:38:44.930905+0000","last_became_active":"2024-09-18T07:37:28.002858+0000","last_became_peered":"2024-09-18T07:37:28.002858+0000","last_unstale":"2024-09-18T07:38:44.930905+0000","last_undegraded":"2024-09-18T07:38:44.930905+0000","last_fullsized":"2024-09-18T07:38:44.930905+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:35:31.858040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:31.858040+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:57.917053+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":122,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.925957+0000","last_change":"2024-09-18T07:36:13.116882+0000","last_active":"2024-09-18T07:38:44.925957+0000","last_peered":"2024-09-18T07:38:44.925957+0000","last_clean":"2024-09-18T07:38:44.925957+0000","last_became_active":"2024-09-18T07:36:13.116416+0000","last_became_peered":"2024-09-18T07:36:13.116416+0000","last_unstale":"2024-09-18T07:38:44.925957+0000","last_undegraded":"2024-09-18T07:38:44.925957+0000","last_fullsized":"2024-09-18T07:38:44.925957+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:10.074367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:10.074367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":440,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.935590+0000","last_change":"2024-09-18T07:37:43.067816+0000","last_active":"2024-09-18T07:38:44.935590+0000","last_peered":"2024-09-18T07:38:44.935590+0000","last_clean":"2024-09-18T07:38:44.935590+0000","last_became_active":"2024-09-18T07:37:43.067493+0000","last_became_peered":"2024-09-18T07:37:43.067493+0000","last_unstale":"2024-09-18T07:38:44.935590+0000","last_undegraded":"2024-09-18T07:38:44.935590+0000","last_fullsized":"2024-09-18T07:38:44.935590+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:34.992350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:34.992350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":439,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.926709+0000","last_change":"2024-09-18T07:35:37.941957+0000","last_active":"2024-09-18T07:38:44.926709+0000","last_peered":"2024-09-18T07:38:44.926709+0000","last_clean":"2024-09-18T07:38:44.926709+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:38:44.926709+0000","last_undegraded":"2024-09-18T07:38:44.926709+0000","last_fullsized":"2024-09-18T07:38:44.926709+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:37.941908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:37.941908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":92,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929192+0000","last_change":"2024-09-18T07:37:43.756858+0000","last_active":"2024-09-18T07:38:44.929192+0000","last_peered":"2024-09-18T07:38:44.929192+0000","last_clean":"2024-09-18T07:38:44.929192+0000","last_became_active":"2024-09-18T07:37:27.981705+0000","last_became_peered":"2024-09-18T07:37:27.981705+0000","last_unstale":"2024-09-18T07:38:44.929192+0000","last_undegraded":"2024-09-18T07:38:44.929192+0000","last_fullsized":"2024-09-18T07:38:44.929192+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:37:43.756403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:37:43.756403+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:47.892762+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":415,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.926395+0000","last_change":"2024-09-18T07:31:00.334313+0000","last_active":"2024-09-18T07:38:44.926395+0000","last_peered":"2024-09-18T07:38:44.926395+0000","last_clean":"2024-09-18T07:38:44.926395+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:38:44.926395+0000","last_undegraded":"2024-09-18T07:38:44.926395+0000","last_fullsized":"2024-09-18T07:38:44.926395+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:31.063052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":209,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.931488+0000","last_change":"2024-09-18T07:35:30.254797+0000","last_active":"2024-09-18T07:38:44.931488+0000","last_peered":"2024-09-18T07:38:44.931488+0000","last_clean":"2024-09-18T07:38:44.931488+0000","last_became_active":"2024-09-18T07:35:01.413110+0000","last_became_peered":"2024-09-18T07:35:01.413110+0000","last_unstale":"2024-09-18T07:38:44.931488+0000","last_undegraded":"2024-09-18T07:38:44.931488+0000","last_fullsized":"2024-09-18T07:38:44.931488+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:35:30.254725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:30.254725+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":131,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.932660+0000","last_change":"2024-09-18T07:36:13.116657+0000","last_active":"2024-09-18T07:38:44.932660+0000","last_peered":"2024-09-18T07:38:44.932660+0000","last_clean":"2024-09-18T07:38:44.932660+0000","last_became_active":"2024-09-18T07:36:13.116295+0000","last_became_peered":"2024-09-18T07:36:13.116295+0000","last_unstale":"2024-09-18T07:38:44.932660+0000","last_undegraded":"2024-09-18T07:38:44.932660+0000","last_fullsized":"2024-09-18T07:38:44.932660+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:32.976129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.976129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":488,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.220938+0000","last_change":"2024-09-18T07:35:57.488019+0000","last_active":"2024-09-18T07:38:47.220938+0000","last_peered":"2024-09-18T07:38:47.220938+0000","last_clean":"2024-09-18T07:38:47.220938+0000","last_became_active":"2024-09-18T07:35:01.410926+0000","last_became_peered":"2024-09-18T07:35:01.410926+0000","last_unstale":"2024-09-18T07:38:47.220938+0000","last_undegraded":"2024-09-18T07:38:47.220938+0000","last_fullsized":"2024-09-18T07:38:47.220938+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:35:57.487960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.487960+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"240'12","reported_seq":81,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.931042+0000","last_change":"2024-09-18T07:37:43.220303+0000","last_active":"2024-09-18T07:38:44.931042+0000","last_peered":"2024-09-18T07:38:44.931042+0000","last_clean":"2024-09-18T07:38:44.931042+0000","last_became_active":"2024-09-18T07:37:43.060715+0000","last_became_peered":"2024-09-18T07:37:43.060715+0000","last_unstale":"2024-09-18T07:38:44.931042+0000","last_undegraded":"2024-09-18T07:38:44.931042+0000","last_fullsized":"2024-09-18T07:38:44.931042+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:35:29.400137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:29.400137+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"265'10741","reported_seq":18546,"reported_epoch":265,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T07:38:46.756161+0000","last_change":"2024-09-18T07:38:36.788944+0000","last_active":"2024-09-18T07:38:46.756161+0000","last_peered":"2024-09-18T07:38:46.756161+0000","last_clean":"2024-09-18T07:36:40.992666+0000","last_became_active":"2024-09-18T07:37:43.231704+0000","last_became_peered":"2024-09-18T07:37:43.231704+0000","last_unstale":"2024-09-18T07:38:46.756161+0000","last_undegraded":"2024-09-18T07:38:46.756161+0000","last_fullsized":"2024-09-18T07:37:43.046004+0000","mapping_epoch":238,"log_start":"229'5614","ondisk_log_start":"229'5614","created":201,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5127,"log_dups_size":0,"ondisk_log_size":5127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152240128,"num_objects":2323,"num_object_clones":0,"num_object_copies":6969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1851,"num_objects_unfound":0,"num_objects_dirty":2323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10741,"num_write_kb":418048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3822,"num_bytes_recovered":250413056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,2,5","objects":1851},{"shards":"1,5","objects":472}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":167,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.935796+0000","last_change":"2024-09-18T07:36:13.120127+0000","last_active":"2024-09-18T07:38:44.935796+0000","last_peered":"2024-09-18T07:38:44.935796+0000","last_clean":"2024-09-18T07:38:44.935796+0000","last_became_active":"2024-09-18T07:36:13.119098+0000","last_became_peered":"2024-09-18T07:36:13.119098+0000","last_unstale":"2024-09-18T07:38:44.935796+0000","last_undegraded":"2024-09-18T07:38:44.935796+0000","last_fullsized":"2024-09-18T07:38:44.935796+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:34:37.067162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:37.067162+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":460,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.933139+0000","last_change":"2024-09-18T07:36:01.988099+0000","last_active":"2024-09-18T07:38:44.933139+0000","last_peered":"2024-09-18T07:38:44.933139+0000","last_clean":"2024-09-18T07:38:44.933139+0000","last_became_active":"2024-09-18T07:35:01.407339+0000","last_became_peered":"2024-09-18T07:35:01.407339+0000","last_unstale":"2024-09-18T07:38:44.933139+0000","last_undegraded":"2024-09-18T07:38:44.933139+0000","last_fullsized":"2024-09-18T07:38:44.933139+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:01.988054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.988054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":542,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.928525+0000","last_change":"2024-09-18T07:36:35.210213+0000","last_active":"2024-09-18T07:38:44.928525+0000","last_peered":"2024-09-18T07:38:44.928525+0000","last_clean":"2024-09-18T07:38:44.928525+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:38:44.928525+0000","last_undegraded":"2024-09-18T07:38:44.928525+0000","last_fullsized":"2024-09-18T07:38:44.928525+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:35.209823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:35.209823+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":425,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.927500+0000","last_change":"2024-09-18T07:36:07.588483+0000","last_active":"2024-09-18T07:38:44.927500+0000","last_peered":"2024-09-18T07:38:44.927500+0000","last_clean":"2024-09-18T07:38:44.927500+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:38:44.927500+0000","last_undegraded":"2024-09-18T07:38:44.927500+0000","last_fullsized":"2024-09-18T07:38:44.927500+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:07.588440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:07.588440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"266'10628","reported_seq":10771,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.383928+0000","last_change":"2024-09-18T07:35:59.286428+0000","last_active":"2024-09-18T07:38:47.383928+0000","last_peered":"2024-09-18T07:38:47.383928+0000","last_clean":"2024-09-18T07:38:47.383928+0000","last_became_active":"2024-09-18T07:35:59.286120+0000","last_became_peered":"2024-09-18T07:35:59.286120+0000","last_unstale":"2024-09-18T07:38:47.383928+0000","last_undegraded":"2024-09-18T07:38:47.383928+0000","last_fullsized":"2024-09-18T07:38:47.383928+0000","mapping_epoch":201,"log_start":"229'5122","ondisk_log_start":"229'5122","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5506,"log_dups_size":0,"ondisk_log_size":5506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143917056,"num_objects":2196,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10628,"num_write_kb":410368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":516,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.932932+0000","last_change":"2024-09-18T07:35:23.983838+0000","last_active":"2024-09-18T07:38:44.932932+0000","last_peered":"2024-09-18T07:38:44.932932+0000","last_clean":"2024-09-18T07:38:44.932932+0000","last_became_active":"2024-09-18T07:35:01.418909+0000","last_became_peered":"2024-09-18T07:35:01.418909+0000","last_unstale":"2024-09-18T07:38:44.932932+0000","last_undegraded":"2024-09-18T07:38:44.932932+0000","last_fullsized":"2024-09-18T07:38:44.932932+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:35:23.983762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:23.983762+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":452,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.936123+0000","last_change":"2024-09-18T07:35:27.922013+0000","last_active":"2024-09-18T07:38:44.936123+0000","last_peered":"2024-09-18T07:38:44.936123+0000","last_clean":"2024-09-18T07:38:44.936123+0000","last_became_active":"2024-09-18T07:35:01.407422+0000","last_became_peered":"2024-09-18T07:35:01.407422+0000","last_unstale":"2024-09-18T07:38:44.936123+0000","last_undegraded":"2024-09-18T07:38:44.936123+0000","last_fullsized":"2024-09-18T07:38:44.936123+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:27.921887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:27.921887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":445,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.925047+0000","last_change":"2024-09-18T07:36:30.242479+0000","last_active":"2024-09-18T07:38:44.925047+0000","last_peered":"2024-09-18T07:38:44.925047+0000","last_clean":"2024-09-18T07:38:44.925047+0000","last_became_active":"2024-09-18T07:35:01.401024+0000","last_became_peered":"2024-09-18T07:35:01.401024+0000","last_unstale":"2024-09-18T07:38:44.925047+0000","last_undegraded":"2024-09-18T07:38:44.925047+0000","last_fullsized":"2024-09-18T07:38:44.925047+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:30.242420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:30.242420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":411,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.926509+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:38:44.926509+0000","last_peered":"2024-09-18T07:38:44.926509+0000","last_clean":"2024-09-18T07:38:44.926509+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:38:44.926509+0000","last_undegraded":"2024-09-18T07:38:44.926509+0000","last_fullsized":"2024-09-18T07:38:44.926509+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:33.040530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"265'10609","reported_seq":10749,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:46.758310+0000","last_change":"2024-09-18T07:35:59.287014+0000","last_active":"2024-09-18T07:38:46.758310+0000","last_peered":"2024-09-18T07:38:46.758310+0000","last_clean":"2024-09-18T07:38:46.758310+0000","last_became_active":"2024-09-18T07:35:59.286688+0000","last_became_peered":"2024-09-18T07:35:59.286688+0000","last_unstale":"2024-09-18T07:38:46.758310+0000","last_undegraded":"2024-09-18T07:38:46.758310+0000","last_fullsized":"2024-09-18T07:38:46.758310+0000","mapping_epoch":201,"log_start":"229'5443","ondisk_log_start":"229'5443","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5166,"log_dups_size":0,"ondisk_log_size":5166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151781400,"num_objects":2317,"num_object_clones":0,"num_object_copies":6951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2317,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10609,"num_write_kb":413569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":494,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.925134+0000","last_change":"2024-09-18T07:37:43.153068+0000","last_active":"2024-09-18T07:38:44.925134+0000","last_peered":"2024-09-18T07:38:44.925134+0000","last_clean":"2024-09-18T07:38:44.925134+0000","last_became_active":"2024-09-18T07:37:43.096236+0000","last_became_peered":"2024-09-18T07:37:43.096236+0000","last_unstale":"2024-09-18T07:38:44.925134+0000","last_undegraded":"2024-09-18T07:38:44.925134+0000","last_fullsized":"2024-09-18T07:38:44.925134+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-18T07:35:00.882191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.882191+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":432,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.221143+0000","last_change":"2024-09-18T07:33:24.213432+0000","last_active":"2024-09-18T07:38:47.221143+0000","last_peered":"2024-09-18T07:38:47.221143+0000","last_clean":"2024-09-18T07:38:47.221143+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:38:47.221143+0000","last_undegraded":"2024-09-18T07:38:47.221143+0000","last_fullsized":"2024-09-18T07:38:47.221143+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:33:24.213356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:24.213356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":533,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.932810+0000","last_change":"2024-09-18T07:36:05.016008+0000","last_active":"2024-09-18T07:38:44.932810+0000","last_peered":"2024-09-18T07:38:44.932810+0000","last_clean":"2024-09-18T07:38:44.932810+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:38:44.932810+0000","last_undegraded":"2024-09-18T07:38:44.932810+0000","last_fullsized":"2024-09-18T07:38:44.932810+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:05.015955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.015955+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":219,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.932099+0000","last_change":"2024-09-18T07:37:43.057439+0000","last_active":"2024-09-18T07:38:44.932099+0000","last_peered":"2024-09-18T07:38:44.932099+0000","last_clean":"2024-09-18T07:38:44.932099+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:38:44.932099+0000","last_undegraded":"2024-09-18T07:38:44.932099+0000","last_fullsized":"2024-09-18T07:38:44.932099+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:25.287352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:25.287352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"266'10709","reported_seq":10852,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.401965+0000","last_change":"2024-09-18T07:35:59.102118+0000","last_active":"2024-09-18T07:38:47.401965+0000","last_peered":"2024-09-18T07:38:47.401965+0000","last_clean":"2024-09-18T07:38:47.401965+0000","last_became_active":"2024-09-18T07:35:59.101681+0000","last_became_peered":"2024-09-18T07:35:59.101681+0000","last_unstale":"2024-09-18T07:38:47.401965+0000","last_undegraded":"2024-09-18T07:38:47.401965+0000","last_fullsized":"2024-09-18T07:38:47.401965+0000","mapping_epoch":201,"log_start":"229'5211","ondisk_log_start":"229'5211","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5498,"log_dups_size":0,"ondisk_log_size":5498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143589376,"num_objects":2191,"num_object_clones":0,"num_object_copies":6573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10709,"num_write_kb":412800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":157,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.927280+0000","last_change":"2024-09-18T07:36:13.119192+0000","last_active":"2024-09-18T07:38:44.927280+0000","last_peered":"2024-09-18T07:38:44.927280+0000","last_clean":"2024-09-18T07:38:44.927280+0000","last_became_active":"2024-09-18T07:36:13.118876+0000","last_became_peered":"2024-09-18T07:36:13.118876+0000","last_unstale":"2024-09-18T07:38:44.927280+0000","last_undegraded":"2024-09-18T07:38:44.927280+0000","last_fullsized":"2024-09-18T07:38:44.927280+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":437,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.928703+0000","last_change":"2024-09-18T07:35:24.210457+0000","last_active":"2024-09-18T07:38:44.928703+0000","last_peered":"2024-09-18T07:38:44.928703+0000","last_clean":"2024-09-18T07:38:44.928703+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:38:44.928703+0000","last_undegraded":"2024-09-18T07:38:44.928703+0000","last_fullsized":"2024-09-18T07:38:44.928703+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:24.210401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:24.210401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"240'10","reported_seq":556,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.220740+0000","last_change":"2024-09-18T07:37:43.170266+0000","last_active":"2024-09-18T07:38:47.220740+0000","last_peered":"2024-09-18T07:38:47.220740+0000","last_clean":"2024-09-18T07:38:47.220740+0000","last_became_active":"2024-09-18T07:37:43.167850+0000","last_became_peered":"2024-09-18T07:37:43.167850+0000","last_unstale":"2024-09-18T07:38:47.220740+0000","last_undegraded":"2024-09-18T07:38:47.220740+0000","last_fullsized":"2024-09-18T07:38:47.220740+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:05.525762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.525762+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":411,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.220001+0000","last_change":"2024-09-18T07:31:02.348538+0000","last_active":"2024-09-18T07:38:47.220001+0000","last_peered":"2024-09-18T07:38:47.220001+0000","last_clean":"2024-09-18T07:38:47.220001+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:38:47.220001+0000","last_undegraded":"2024-09-18T07:38:47.220001+0000","last_fullsized":"2024-09-18T07:38:47.220001+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"265'10866","reported_seq":11007,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:46.996586+0000","last_change":"2024-09-18T07:35:59.102330+0000","last_active":"2024-09-18T07:38:46.996586+0000","last_peered":"2024-09-18T07:38:46.996586+0000","last_clean":"2024-09-18T07:38:46.996586+0000","last_became_active":"2024-09-18T07:35:59.101668+0000","last_became_peered":"2024-09-18T07:35:59.101668+0000","last_unstale":"2024-09-18T07:38:46.996586+0000","last_undegraded":"2024-09-18T07:38:46.996586+0000","last_fullsized":"2024-09-18T07:38:46.996586+0000","mapping_epoch":201,"log_start":"230'5800","ondisk_log_start":"230'5800","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5066,"log_dups_size":0,"ondisk_log_size":5066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148373504,"num_objects":2264,"num_object_clones":0,"num_object_copies":6792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10866,"num_write_kb":420160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":521,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.220932+0000","last_change":"2024-09-18T07:36:09.498239+0000","last_active":"2024-09-18T07:38:47.220932+0000","last_peered":"2024-09-18T07:38:47.220932+0000","last_clean":"2024-09-18T07:38:47.220932+0000","last_became_active":"2024-09-18T07:35:01.415275+0000","last_became_peered":"2024-09-18T07:35:01.415275+0000","last_unstale":"2024-09-18T07:38:47.220932+0000","last_undegraded":"2024-09-18T07:38:47.220932+0000","last_fullsized":"2024-09-18T07:38:47.220932+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:36:09.498142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.498142+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":130,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.930962+0000","last_change":"2024-09-18T07:37:46.831436+0000","last_active":"2024-09-18T07:38:44.930962+0000","last_peered":"2024-09-18T07:38:44.930962+0000","last_clean":"2024-09-18T07:38:44.930962+0000","last_became_active":"2024-09-18T07:36:13.118553+0000","last_became_peered":"2024-09-18T07:36:13.118553+0000","last_unstale":"2024-09-18T07:38:44.930962+0000","last_undegraded":"2024-09-18T07:38:44.930962+0000","last_fullsized":"2024-09-18T07:38:44.930962+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:46.831373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:37:46.831373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:13.442615+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":64,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929529+0000","last_change":"2024-09-18T07:37:43.070306+0000","last_active":"2024-09-18T07:38:44.929529+0000","last_peered":"2024-09-18T07:38:44.929529+0000","last_clean":"2024-09-18T07:38:44.929529+0000","last_became_active":"2024-09-18T07:37:43.070009+0000","last_became_peered":"2024-09-18T07:37:43.070009+0000","last_unstale":"2024-09-18T07:38:44.929529+0000","last_undegraded":"2024-09-18T07:38:44.929529+0000","last_fullsized":"2024-09-18T07:38:44.929529+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":64,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929495+0000","last_change":"2024-09-18T07:37:43.052077+0000","last_active":"2024-09-18T07:38:44.929495+0000","last_peered":"2024-09-18T07:38:44.929495+0000","last_clean":"2024-09-18T07:38:44.929495+0000","last_became_active":"2024-09-18T07:37:43.051740+0000","last_became_peered":"2024-09-18T07:37:43.051740+0000","last_unstale":"2024-09-18T07:38:44.929495+0000","last_undegraded":"2024-09-18T07:38:44.929495+0000","last_fullsized":"2024-09-18T07:38:44.929495+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"265'11002","reported_seq":11143,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:46.727296+0000","last_change":"2024-09-18T07:35:59.288477+0000","last_active":"2024-09-18T07:38:46.727296+0000","last_peered":"2024-09-18T07:38:46.727296+0000","last_clean":"2024-09-18T07:38:46.727296+0000","last_became_active":"2024-09-18T07:35:59.288058+0000","last_became_peered":"2024-09-18T07:35:59.288058+0000","last_unstale":"2024-09-18T07:38:46.727296+0000","last_undegraded":"2024-09-18T07:38:46.727296+0000","last_fullsized":"2024-09-18T07:38:46.727296+0000","mapping_epoch":201,"log_start":"230'5835","ondisk_log_start":"230'5835","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5167,"log_dups_size":0,"ondisk_log_size":5167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152829952,"num_objects":2332,"num_object_clones":0,"num_object_copies":6996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11002,"num_write_kb":426688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"211'29","reported_seq":223,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.930952+0000","last_change":"2024-09-18T07:36:13.115630+0000","last_active":"2024-09-18T07:38:44.930952+0000","last_peered":"2024-09-18T07:38:44.930952+0000","last_clean":"2024-09-18T07:38:44.930952+0000","last_became_active":"2024-09-18T07:36:13.115278+0000","last_became_peered":"2024-09-18T07:36:13.115278+0000","last_unstale":"2024-09-18T07:38:44.930952+0000","last_undegraded":"2024-09-18T07:38:44.930952+0000","last_fullsized":"2024-09-18T07:38:44.930952+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'28","last_scrub_stamp":"2024-09-18T07:34:31.955336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.955336+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:54.943493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":445,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.926140+0000","last_change":"2024-09-18T07:37:43.061414+0000","last_active":"2024-09-18T07:38:44.926140+0000","last_peered":"2024-09-18T07:38:44.926140+0000","last_clean":"2024-09-18T07:38:44.926140+0000","last_became_active":"2024-09-18T07:37:43.060930+0000","last_became_peered":"2024-09-18T07:37:43.060930+0000","last_unstale":"2024-09-18T07:38:44.926140+0000","last_undegraded":"2024-09-18T07:38:44.926140+0000","last_fullsized":"2024-09-18T07:38:44.926140+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:06.908713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.908713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":469,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.927411+0000","last_change":"2024-09-18T07:37:43.155837+0000","last_active":"2024-09-18T07:38:44.927411+0000","last_peered":"2024-09-18T07:38:44.927411+0000","last_clean":"2024-09-18T07:38:44.927411+0000","last_became_active":"2024-09-18T07:37:43.076595+0000","last_became_peered":"2024-09-18T07:37:43.076595+0000","last_unstale":"2024-09-18T07:38:44.927411+0000","last_undegraded":"2024-09-18T07:38:44.927411+0000","last_fullsized":"2024-09-18T07:38:44.927411+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:36:01.529677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.529677+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":415,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.221524+0000","last_change":"2024-09-18T07:31:00.332738+0000","last_active":"2024-09-18T07:38:47.221524+0000","last_peered":"2024-09-18T07:38:47.221524+0000","last_clean":"2024-09-18T07:38:47.221524+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:38:47.221524+0000","last_undegraded":"2024-09-18T07:38:47.221524+0000","last_fullsized":"2024-09-18T07:38:47.221524+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":432,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.936948+0000","last_change":"2024-09-18T07:35:01.419122+0000","last_active":"2024-09-18T07:38:44.936948+0000","last_peered":"2024-09-18T07:38:44.936948+0000","last_clean":"2024-09-18T07:38:44.936948+0000","last_became_active":"2024-09-18T07:35:01.418809+0000","last_became_peered":"2024-09-18T07:35:01.418809+0000","last_unstale":"2024-09-18T07:38:44.936948+0000","last_undegraded":"2024-09-18T07:38:44.936948+0000","last_fullsized":"2024-09-18T07:38:44.936948+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:56.289176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:34:56.289176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"265'10757","reported_seq":10898,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:46.996287+0000","last_change":"2024-09-18T07:35:59.103129+0000","last_active":"2024-09-18T07:38:46.996287+0000","last_peered":"2024-09-18T07:38:46.996287+0000","last_clean":"2024-09-18T07:38:46.996287+0000","last_became_active":"2024-09-18T07:35:59.102853+0000","last_became_peered":"2024-09-18T07:35:59.102853+0000","last_unstale":"2024-09-18T07:38:46.996287+0000","last_undegraded":"2024-09-18T07:38:46.996287+0000","last_fullsized":"2024-09-18T07:38:46.996287+0000","mapping_epoch":201,"log_start":"230'5700","ondisk_log_start":"230'5700","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155123712,"num_objects":2367,"num_object_clones":0,"num_object_copies":7101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10757,"num_write_kb":419968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":169,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.928193+0000","last_change":"2024-09-18T07:36:13.118278+0000","last_active":"2024-09-18T07:38:44.928193+0000","last_peered":"2024-09-18T07:38:44.928193+0000","last_clean":"2024-09-18T07:38:44.928193+0000","last_became_active":"2024-09-18T07:36:13.117924+0000","last_became_peered":"2024-09-18T07:36:13.117924+0000","last_unstale":"2024-09-18T07:38:44.928193+0000","last_undegraded":"2024-09-18T07:38:44.928193+0000","last_fullsized":"2024-09-18T07:38:44.928193+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-18T07:36:06.083531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.083531+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":136,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929026+0000","last_change":"2024-09-18T07:37:43.051318+0000","last_active":"2024-09-18T07:38:44.929026+0000","last_peered":"2024-09-18T07:38:44.929026+0000","last_clean":"2024-09-18T07:38:44.929026+0000","last_became_active":"2024-09-18T07:37:43.050971+0000","last_became_peered":"2024-09-18T07:37:43.050971+0000","last_unstale":"2024-09-18T07:38:44.929026+0000","last_undegraded":"2024-09-18T07:38:44.929026+0000","last_fullsized":"2024-09-18T07:38:44.929026+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:04.050757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:04.050757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:58.893486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":448,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929067+0000","last_change":"2024-09-18T07:37:49.895264+0000","last_active":"2024-09-18T07:38:44.929067+0000","last_peered":"2024-09-18T07:38:44.929067+0000","last_clean":"2024-09-18T07:38:44.929067+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:38:44.929067+0000","last_undegraded":"2024-09-18T07:38:44.929067+0000","last_fullsized":"2024-09-18T07:38:44.929067+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:37:49.894707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:37:49.894707+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:04.276523+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":437,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929722+0000","last_change":"2024-09-18T07:37:47.825723+0000","last_active":"2024-09-18T07:38:44.929722+0000","last_peered":"2024-09-18T07:38:44.929722+0000","last_clean":"2024-09-18T07:38:44.929722+0000","last_became_active":"2024-09-18T07:35:01.403680+0000","last_became_peered":"2024-09-18T07:35:01.403680+0000","last_unstale":"2024-09-18T07:38:44.929722+0000","last_undegraded":"2024-09-18T07:38:44.929722+0000","last_fullsized":"2024-09-18T07:38:44.929722+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:47.825656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:37:47.825656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:17.721036+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":223,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.938004+0000","last_change":"2024-09-18T07:37:43.118570+0000","last_active":"2024-09-18T07:38:44.938004+0000","last_peered":"2024-09-18T07:38:44.938004+0000","last_clean":"2024-09-18T07:38:44.938004+0000","last_became_active":"2024-09-18T07:37:43.060019+0000","last_became_peered":"2024-09-18T07:37:43.060019+0000","last_unstale":"2024-09-18T07:38:44.938004+0000","last_undegraded":"2024-09-18T07:38:44.938004+0000","last_fullsized":"2024-09-18T07:38:44.938004+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:57.655797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.655797+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"265'10691","reported_seq":10887,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:46.765038+0000","last_change":"2024-09-18T07:38:20.242167+0000","last_active":"2024-09-18T07:38:46.765038+0000","last_peered":"2024-09-18T07:38:46.765038+0000","last_clean":"2024-09-18T07:38:46.765038+0000","last_became_active":"2024-09-18T07:36:13.119156+0000","last_became_peered":"2024-09-18T07:36:13.119156+0000","last_unstale":"2024-09-18T07:38:46.765038+0000","last_undegraded":"2024-09-18T07:38:46.765038+0000","last_fullsized":"2024-09-18T07:38:46.765038+0000","mapping_epoch":210,"log_start":"223'4519","ondisk_log_start":"223'4519","created":201,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'9080","last_scrub_stamp":"2024-09-18T07:38:20.241865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:38:20.241865+0000","objects_scrubbed":3965,"log_size":6172,"log_dups_size":0,"ondisk_log_size":6172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:21.802296+0000","scrub_duration":1210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152764416,"num_objects":2331,"num_object_clones":0,"num_object_copies":6993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10691,"num_write_kb":416704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":458,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.221044+0000","last_change":"2024-09-18T07:33:27.040227+0000","last_active":"2024-09-18T07:38:47.221044+0000","last_peered":"2024-09-18T07:38:47.221044+0000","last_clean":"2024-09-18T07:38:47.221044+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:38:47.221044+0000","last_undegraded":"2024-09-18T07:38:47.221044+0000","last_fullsized":"2024-09-18T07:38:47.221044+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":429,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.226563+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:38:47.226563+0000","last_peered":"2024-09-18T07:38:47.226563+0000","last_clean":"2024-09-18T07:38:47.226563+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:38:47.226563+0000","last_undegraded":"2024-09-18T07:38:47.226563+0000","last_fullsized":"2024-09-18T07:38:47.226563+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":426,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.932297+0000","last_change":"2024-09-18T07:35:01.421692+0000","last_active":"2024-09-18T07:38:44.932297+0000","last_peered":"2024-09-18T07:38:44.932297+0000","last_clean":"2024-09-18T07:38:44.932297+0000","last_became_active":"2024-09-18T07:35:01.400524+0000","last_became_peered":"2024-09-18T07:35:01.400524+0000","last_unstale":"2024-09-18T07:38:44.932297+0000","last_undegraded":"2024-09-18T07:38:44.932297+0000","last_fullsized":"2024-09-18T07:38:44.932297+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:39.380198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":231,"num_read_kb":226,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":429,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.928475+0000","last_change":"2024-09-18T07:36:28.551293+0000","last_active":"2024-09-18T07:38:44.928475+0000","last_peered":"2024-09-18T07:38:44.928475+0000","last_clean":"2024-09-18T07:38:44.928475+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:38:44.928475+0000","last_undegraded":"2024-09-18T07:38:44.928475+0000","last_fullsized":"2024-09-18T07:38:44.928475+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:28.551238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:28.551238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":130,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.927812+0000","last_change":"2024-09-18T07:36:39.601424+0000","last_active":"2024-09-18T07:38:44.927812+0000","last_peered":"2024-09-18T07:38:44.927812+0000","last_clean":"2024-09-18T07:38:44.927812+0000","last_became_active":"2024-09-18T07:36:13.117463+0000","last_became_peered":"2024-09-18T07:36:13.117463+0000","last_unstale":"2024-09-18T07:38:44.927812+0000","last_undegraded":"2024-09-18T07:38:44.927812+0000","last_fullsized":"2024-09-18T07:38:44.927812+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:39.601356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:39.601356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"265'10874","reported_seq":20447,"reported_epoch":265,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T07:38:46.776199+0000","last_change":"2024-09-18T07:38:19.382796+0000","last_active":"2024-09-18T07:38:46.776199+0000","last_peered":"2024-09-18T07:38:46.776199+0000","last_clean":"2024-09-18T07:36:40.976872+0000","last_became_active":"2024-09-18T07:37:43.232620+0000","last_became_peered":"2024-09-18T07:37:43.232620+0000","last_unstale":"2024-09-18T07:38:46.776199+0000","last_undegraded":"2024-09-18T07:38:46.776199+0000","last_fullsized":"2024-09-18T07:37:43.046102+0000","mapping_epoch":238,"log_start":"230'5767","ondisk_log_start":"230'5767","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5107,"log_dups_size":0,"ondisk_log_size":5107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154009600,"num_objects":2350,"num_object_clones":0,"num_object_copies":7050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":535,"num_objects_unfound":0,"num_objects_dirty":2350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10874,"num_write_kb":423168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4713,"num_bytes_recovered":308805632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,3,5","objects":535},{"shards":"0,5","objects":1815}],"blocked_by":[],"up_primary":6,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"240'20","reported_seq":95,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.930683+0000","last_change":"2024-09-18T07:37:43.157718+0000","last_active":"2024-09-18T07:38:44.930683+0000","last_peered":"2024-09-18T07:38:44.930683+0000","last_clean":"2024-09-18T07:38:44.930683+0000","last_became_active":"2024-09-18T07:37:43.087496+0000","last_became_peered":"2024-09-18T07:37:43.087496+0000","last_unstale":"2024-09-18T07:38:44.930683+0000","last_undegraded":"2024-09-18T07:38:44.930683+0000","last_fullsized":"2024-09-18T07:38:44.930683+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'17","last_scrub_stamp":"2024-09-18T07:33:33.849285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.849285+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":444,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.932569+0000","last_change":"2024-09-18T07:37:43.065584+0000","last_active":"2024-09-18T07:38:44.932569+0000","last_peered":"2024-09-18T07:38:44.932569+0000","last_clean":"2024-09-18T07:38:44.932569+0000","last_became_active":"2024-09-18T07:37:43.065279+0000","last_became_peered":"2024-09-18T07:37:43.065279+0000","last_unstale":"2024-09-18T07:38:44.932569+0000","last_undegraded":"2024-09-18T07:38:44.932569+0000","last_fullsized":"2024-09-18T07:38:44.932569+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:59.616988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:59.616988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:40.932007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":237,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.932346+0000","last_change":"2024-09-18T07:37:43.121827+0000","last_active":"2024-09-18T07:38:44.932346+0000","last_peered":"2024-09-18T07:38:44.932346+0000","last_clean":"2024-09-18T07:38:44.932346+0000","last_became_active":"2024-09-18T07:37:43.058402+0000","last_became_peered":"2024-09-18T07:37:43.058402+0000","last_unstale":"2024-09-18T07:38:44.932346+0000","last_undegraded":"2024-09-18T07:38:44.932346+0000","last_fullsized":"2024-09-18T07:38:44.932346+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:35:27.278095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:35:27.278095+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":197,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.931509+0000","last_change":"2024-09-18T07:35:01.413545+0000","last_active":"2024-09-18T07:38:44.931509+0000","last_peered":"2024-09-18T07:38:44.931509+0000","last_clean":"2024-09-18T07:38:44.931509+0000","last_became_active":"2024-09-18T07:35:01.412217+0000","last_became_peered":"2024-09-18T07:35:01.412217+0000","last_unstale":"2024-09-18T07:38:44.931509+0000","last_undegraded":"2024-09-18T07:38:44.931509+0000","last_fullsized":"2024-09-18T07:38:44.931509+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:36.964421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:36.964421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":417,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.927608+0000","last_change":"2024-09-18T07:33:22.358792+0000","last_active":"2024-09-18T07:38:44.927608+0000","last_peered":"2024-09-18T07:38:44.927608+0000","last_clean":"2024-09-18T07:38:44.927608+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:38:44.927608+0000","last_undegraded":"2024-09-18T07:38:44.927608+0000","last_fullsized":"2024-09-18T07:38:44.927608+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.358746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.358746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"265'10777","reported_seq":10917,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:46.529811+0000","last_change":"2024-09-18T07:35:59.106988+0000","last_active":"2024-09-18T07:38:46.529811+0000","last_peered":"2024-09-18T07:38:46.529811+0000","last_clean":"2024-09-18T07:38:46.529811+0000","last_became_active":"2024-09-18T07:35:59.106659+0000","last_became_peered":"2024-09-18T07:35:59.106659+0000","last_unstale":"2024-09-18T07:38:46.529811+0000","last_undegraded":"2024-09-18T07:38:46.529811+0000","last_fullsized":"2024-09-18T07:38:46.529811+0000","mapping_epoch":201,"log_start":"231'6000","ondisk_log_start":"231'6000","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156303360,"num_objects":2385,"num_object_clones":0,"num_object_copies":7155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10777,"num_write_kb":421184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":530,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.928707+0000","last_change":"2024-09-18T07:37:43.229332+0000","last_active":"2024-09-18T07:38:44.928707+0000","last_peered":"2024-09-18T07:38:44.928707+0000","last_clean":"2024-09-18T07:38:44.928707+0000","last_became_active":"2024-09-18T07:37:43.063255+0000","last_became_peered":"2024-09-18T07:37:43.063255+0000","last_unstale":"2024-09-18T07:38:44.928707+0000","last_undegraded":"2024-09-18T07:38:44.928707+0000","last_fullsized":"2024-09-18T07:38:44.928707+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:35:38.928407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:38.928407+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":453,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.925478+0000","last_change":"2024-09-18T07:36:32.292127+0000","last_active":"2024-09-18T07:38:44.925478+0000","last_peered":"2024-09-18T07:38:44.925478+0000","last_clean":"2024-09-18T07:38:44.925478+0000","last_became_active":"2024-09-18T07:35:01.401063+0000","last_became_peered":"2024-09-18T07:35:01.401063+0000","last_unstale":"2024-09-18T07:38:44.925478+0000","last_undegraded":"2024-09-18T07:38:44.925478+0000","last_fullsized":"2024-09-18T07:38:44.925478+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:32.292055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:32.292055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":443,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.927511+0000","last_change":"2024-09-18T07:37:43.076010+0000","last_active":"2024-09-18T07:38:44.927511+0000","last_peered":"2024-09-18T07:38:44.927511+0000","last_clean":"2024-09-18T07:38:44.927511+0000","last_became_active":"2024-09-18T07:37:43.075535+0000","last_became_peered":"2024-09-18T07:37:43.075535+0000","last_unstale":"2024-09-18T07:38:44.927511+0000","last_undegraded":"2024-09-18T07:38:44.927511+0000","last_fullsized":"2024-09-18T07:38:44.927511+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:31.191528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:31.191528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":440,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.935511+0000","last_change":"2024-09-18T07:37:43.066497+0000","last_active":"2024-09-18T07:38:44.935511+0000","last_peered":"2024-09-18T07:38:44.935511+0000","last_clean":"2024-09-18T07:38:44.935511+0000","last_became_active":"2024-09-18T07:37:43.066153+0000","last_became_peered":"2024-09-18T07:37:43.066153+0000","last_unstale":"2024-09-18T07:38:44.935511+0000","last_undegraded":"2024-09-18T07:38:44.935511+0000","last_fullsized":"2024-09-18T07:38:44.935511+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:38.955353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:38.955353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"266'10787","reported_seq":10936,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.378242+0000","last_change":"2024-09-18T07:36:13.120887+0000","last_active":"2024-09-18T07:38:47.378242+0000","last_peered":"2024-09-18T07:38:47.378242+0000","last_clean":"2024-09-18T07:38:47.378242+0000","last_became_active":"2024-09-18T07:36:13.116701+0000","last_became_peered":"2024-09-18T07:36:13.116701+0000","last_unstale":"2024-09-18T07:38:47.378242+0000","last_undegraded":"2024-09-18T07:38:47.378242+0000","last_fullsized":"2024-09-18T07:38:47.378242+0000","mapping_epoch":210,"log_start":"229'5300","ondisk_log_start":"229'5300","created":201,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5487,"log_dups_size":0,"ondisk_log_size":5487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149487616,"num_objects":2281,"num_object_clones":0,"num_object_copies":6843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10787,"num_write_kb":418176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":463,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.926298+0000","last_change":"2024-09-18T07:35:33.011021+0000","last_active":"2024-09-18T07:38:44.926298+0000","last_peered":"2024-09-18T07:38:44.926298+0000","last_clean":"2024-09-18T07:38:44.926298+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:38:44.926298+0000","last_undegraded":"2024-09-18T07:38:44.926298+0000","last_fullsized":"2024-09-18T07:38:44.926298+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:35:33.010955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:33.010955+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":64,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929688+0000","last_change":"2024-09-18T07:37:43.062115+0000","last_active":"2024-09-18T07:38:44.929688+0000","last_peered":"2024-09-18T07:38:44.929688+0000","last_clean":"2024-09-18T07:38:44.929688+0000","last_became_active":"2024-09-18T07:37:43.061808+0000","last_became_peered":"2024-09-18T07:37:43.061808+0000","last_unstale":"2024-09-18T07:38:44.929688+0000","last_undegraded":"2024-09-18T07:38:44.929688+0000","last_fullsized":"2024-09-18T07:38:44.929688+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:35.265259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:35.265259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":122,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.926252+0000","last_change":"2024-09-18T07:36:13.116901+0000","last_active":"2024-09-18T07:38:44.926252+0000","last_peered":"2024-09-18T07:38:44.926252+0000","last_clean":"2024-09-18T07:38:44.926252+0000","last_became_active":"2024-09-18T07:36:13.116527+0000","last_became_peered":"2024-09-18T07:36:13.116527+0000","last_unstale":"2024-09-18T07:38:44.926252+0000","last_undegraded":"2024-09-18T07:38:44.926252+0000","last_fullsized":"2024-09-18T07:38:44.926252+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:05.239160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.239160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:43.998255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":435,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929447+0000","last_change":"2024-09-18T07:36:38.632189+0000","last_active":"2024-09-18T07:38:44.929447+0000","last_peered":"2024-09-18T07:38:44.929447+0000","last_clean":"2024-09-18T07:38:44.929447+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:38:44.929447+0000","last_undegraded":"2024-09-18T07:38:44.929447+0000","last_fullsized":"2024-09-18T07:38:44.929447+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:38.632040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:38.632040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:05.822462+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"265'10744","reported_seq":10885,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:46.995255+0000","last_change":"2024-09-18T07:35:59.107531+0000","last_active":"2024-09-18T07:38:46.995255+0000","last_peered":"2024-09-18T07:38:46.995255+0000","last_clean":"2024-09-18T07:38:46.995255+0000","last_became_active":"2024-09-18T07:35:59.107232+0000","last_became_peered":"2024-09-18T07:35:59.107232+0000","last_unstale":"2024-09-18T07:38:46.995255+0000","last_undegraded":"2024-09-18T07:38:46.995255+0000","last_fullsized":"2024-09-18T07:38:46.995255+0000","mapping_epoch":201,"log_start":"230'5700","ondisk_log_start":"230'5700","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5044,"log_dups_size":0,"ondisk_log_size":5044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151388160,"num_objects":2310,"num_object_clones":0,"num_object_copies":6930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10744,"num_write_kb":417728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":480,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.926471+0000","last_change":"2024-09-18T07:37:43.106862+0000","last_active":"2024-09-18T07:38:44.926471+0000","last_peered":"2024-09-18T07:38:44.926471+0000","last_clean":"2024-09-18T07:38:44.926471+0000","last_became_active":"2024-09-18T07:37:43.062724+0000","last_became_peered":"2024-09-18T07:37:43.062724+0000","last_unstale":"2024-09-18T07:38:44.926471+0000","last_undegraded":"2024-09-18T07:38:44.926471+0000","last_fullsized":"2024-09-18T07:38:44.926471+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:32:05.000310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.000310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:42.013354+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":430,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.935413+0000","last_change":"2024-09-18T07:32:06.688493+0000","last_active":"2024-09-18T07:38:44.935413+0000","last_peered":"2024-09-18T07:38:44.935413+0000","last_clean":"2024-09-18T07:38:44.935413+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:38:44.935413+0000","last_undegraded":"2024-09-18T07:38:44.935413+0000","last_fullsized":"2024-09-18T07:38:44.935413+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:06.688444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:32:06.688444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:37.951495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"173'11","reported_seq":559,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929380+0000","last_change":"2024-09-18T07:35:01.413702+0000","last_active":"2024-09-18T07:38:44.929380+0000","last_peered":"2024-09-18T07:38:44.929380+0000","last_clean":"2024-09-18T07:38:44.929380+0000","last_became_active":"2024-09-18T07:35:01.403765+0000","last_became_peered":"2024-09-18T07:35:01.403765+0000","last_unstale":"2024-09-18T07:38:44.929380+0000","last_undegraded":"2024-09-18T07:38:44.929380+0000","last_fullsized":"2024-09-18T07:38:44.929380+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'10","last_scrub_stamp":"2024-09-18T07:33:33.000874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.000874+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:53.964476+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":427,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.928304+0000","last_change":"2024-09-18T07:36:33.251164+0000","last_active":"2024-09-18T07:38:44.928304+0000","last_peered":"2024-09-18T07:38:44.928304+0000","last_clean":"2024-09-18T07:38:44.928304+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:38:44.928304+0000","last_undegraded":"2024-09-18T07:38:44.928304+0000","last_fullsized":"2024-09-18T07:38:44.928304+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:33.251093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:33.251093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"265'11144","reported_seq":19136,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:46.995809+0000","last_change":"2024-09-18T07:38:38.138896+0000","last_active":"2024-09-18T07:38:46.995809+0000","last_peered":"2024-09-18T07:38:46.995809+0000","last_clean":"2024-09-18T07:38:46.995809+0000","last_became_active":"2024-09-18T07:38:38.108637+0000","last_became_peered":"2024-09-18T07:38:38.108637+0000","last_unstale":"2024-09-18T07:38:46.995809+0000","last_undegraded":"2024-09-18T07:38:46.995809+0000","last_fullsized":"2024-09-18T07:38:46.995809+0000","mapping_epoch":260,"log_start":"231'6100","ondisk_log_start":"231'6100","created":201,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5044,"log_dups_size":0,"ondisk_log_size":5044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152174592,"num_objects":2322,"num_object_clones":0,"num_object_copies":6966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11144,"num_write_kb":430912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3914,"num_bytes_recovered":236453888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":452,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.220836+0000","last_change":"2024-09-18T07:37:28.049996+0000","last_active":"2024-09-18T07:38:47.220836+0000","last_peered":"2024-09-18T07:38:47.220836+0000","last_clean":"2024-09-18T07:38:47.220836+0000","last_became_active":"2024-09-18T07:37:27.994144+0000","last_became_peered":"2024-09-18T07:37:27.994144+0000","last_unstale":"2024-09-18T07:38:47.220836+0000","last_undegraded":"2024-09-18T07:38:47.220836+0000","last_fullsized":"2024-09-18T07:38:47.220836+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":451,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929863+0000","last_change":"2024-09-18T07:37:43.076416+0000","last_active":"2024-09-18T07:38:44.929863+0000","last_peered":"2024-09-18T07:38:44.929863+0000","last_clean":"2024-09-18T07:38:44.929863+0000","last_became_active":"2024-09-18T07:37:43.075988+0000","last_became_peered":"2024-09-18T07:37:43.075988+0000","last_unstale":"2024-09-18T07:38:44.929863+0000","last_undegraded":"2024-09-18T07:38:44.929863+0000","last_fullsized":"2024-09-18T07:38:44.929863+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:21.222526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.222526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":423,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929224+0000","last_change":"2024-09-18T07:34:31.959675+0000","last_active":"2024-09-18T07:38:44.929224+0000","last_peered":"2024-09-18T07:38:44.929224+0000","last_clean":"2024-09-18T07:38:44.929224+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:38:44.929224+0000","last_undegraded":"2024-09-18T07:38:44.929224+0000","last_fullsized":"2024-09-18T07:38:44.929224+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:31.959636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:34:31.959636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:56.892556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":441,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.226488+0000","last_change":"2024-09-18T07:36:07.526609+0000","last_active":"2024-09-18T07:38:47.226488+0000","last_peered":"2024-09-18T07:38:47.226488+0000","last_clean":"2024-09-18T07:38:47.226488+0000","last_became_active":"2024-09-18T07:35:01.410890+0000","last_became_peered":"2024-09-18T07:35:01.410890+0000","last_unstale":"2024-09-18T07:38:47.226488+0000","last_undegraded":"2024-09-18T07:38:47.226488+0000","last_fullsized":"2024-09-18T07:38:47.226488+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:07.526536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:07.526536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"266'10793","reported_seq":18767,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.344699+0000","last_change":"2024-09-18T07:38:38.136828+0000","last_active":"2024-09-18T07:38:47.344699+0000","last_peered":"2024-09-18T07:38:47.344699+0000","last_clean":"2024-09-18T07:38:47.344699+0000","last_became_active":"2024-09-18T07:38:38.104241+0000","last_became_peered":"2024-09-18T07:38:38.104241+0000","last_unstale":"2024-09-18T07:38:47.344699+0000","last_undegraded":"2024-09-18T07:38:47.344699+0000","last_fullsized":"2024-09-18T07:38:47.344699+0000","mapping_epoch":260,"log_start":"229'5300","ondisk_log_start":"229'5300","created":201,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5493,"log_dups_size":0,"ondisk_log_size":5493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147783680,"num_objects":2255,"num_object_clones":0,"num_object_copies":6765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10793,"num_write_kb":417536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3907,"num_bytes_recovered":237371392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":84,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929823+0000","last_change":"2024-09-18T07:37:43.142793+0000","last_active":"2024-09-18T07:38:44.929823+0000","last_peered":"2024-09-18T07:38:44.929823+0000","last_clean":"2024-09-18T07:38:44.929823+0000","last_became_active":"2024-09-18T07:37:43.063319+0000","last_became_peered":"2024-09-18T07:37:43.063319+0000","last_unstale":"2024-09-18T07:38:44.929823+0000","last_undegraded":"2024-09-18T07:38:44.929823+0000","last_fullsized":"2024-09-18T07:38:44.929823+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:35:21.344029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.344029+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":219,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.935785+0000","last_change":"2024-09-18T07:37:43.073257+0000","last_active":"2024-09-18T07:38:44.935785+0000","last_peered":"2024-09-18T07:38:44.935785+0000","last_clean":"2024-09-18T07:38:44.935785+0000","last_became_active":"2024-09-18T07:37:43.072894+0000","last_became_peered":"2024-09-18T07:37:43.072894+0000","last_unstale":"2024-09-18T07:38:44.935785+0000","last_undegraded":"2024-09-18T07:38:44.935785+0000","last_fullsized":"2024-09-18T07:38:44.935785+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:28.171448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:28.171448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":438,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929493+0000","last_change":"2024-09-18T07:37:52.961389+0000","last_active":"2024-09-18T07:38:44.929493+0000","last_peered":"2024-09-18T07:38:44.929493+0000","last_clean":"2024-09-18T07:38:44.929493+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:38:44.929493+0000","last_undegraded":"2024-09-18T07:38:44.929493+0000","last_fullsized":"2024-09-18T07:38:44.929493+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:52.961325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:37:52.961325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:19.975533+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":197,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.932436+0000","last_change":"2024-09-18T07:35:01.408078+0000","last_active":"2024-09-18T07:38:44.932436+0000","last_peered":"2024-09-18T07:38:44.932436+0000","last_clean":"2024-09-18T07:38:44.932436+0000","last_became_active":"2024-09-18T07:35:01.406980+0000","last_became_peered":"2024-09-18T07:35:01.406980+0000","last_unstale":"2024-09-18T07:38:44.932436+0000","last_undegraded":"2024-09-18T07:38:44.932436+0000","last_fullsized":"2024-09-18T07:38:44.932436+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:37.378725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"265'10700","reported_seq":22664,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:46.608517+0000","last_change":"2024-09-18T07:38:20.948583+0000","last_active":"2024-09-18T07:38:46.608517+0000","last_peered":"2024-09-18T07:38:46.608517+0000","last_clean":"2024-09-18T07:38:46.608517+0000","last_became_active":"2024-09-18T07:38:20.929620+0000","last_became_peered":"2024-09-18T07:38:20.929620+0000","last_unstale":"2024-09-18T07:38:46.608517+0000","last_undegraded":"2024-09-18T07:38:46.608517+0000","last_fullsized":"2024-09-18T07:38:46.608517+0000","mapping_epoch":255,"log_start":"223'4300","ondisk_log_start":"223'4300","created":201,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":6400,"log_dups_size":0,"ondisk_log_size":6400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154140672,"num_objects":2352,"num_object_clones":0,"num_object_copies":7056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10700,"num_write_kb":417664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5903,"num_bytes_recovered":366346240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":458,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.925259+0000","last_change":"2024-09-18T07:35:21.085719+0000","last_active":"2024-09-18T07:38:44.925259+0000","last_peered":"2024-09-18T07:38:44.925259+0000","last_clean":"2024-09-18T07:38:44.925259+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:38:44.925259+0000","last_undegraded":"2024-09-18T07:38:44.925259+0000","last_fullsized":"2024-09-18T07:38:44.925259+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:35:21.085670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.085670+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":197,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.933056+0000","last_change":"2024-09-18T07:35:01.400050+0000","last_active":"2024-09-18T07:38:44.933056+0000","last_peered":"2024-09-18T07:38:44.933056+0000","last_clean":"2024-09-18T07:38:44.933056+0000","last_became_active":"2024-09-18T07:35:01.398499+0000","last_became_peered":"2024-09-18T07:35:01.398499+0000","last_unstale":"2024-09-18T07:38:44.933056+0000","last_undegraded":"2024-09-18T07:38:44.933056+0000","last_fullsized":"2024-09-18T07:38:44.933056+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:38.363366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":447,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929021+0000","last_change":"2024-09-18T07:36:32.564149+0000","last_active":"2024-09-18T07:38:44.929021+0000","last_peered":"2024-09-18T07:38:44.929021+0000","last_clean":"2024-09-18T07:38:44.929021+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:38:44.929021+0000","last_undegraded":"2024-09-18T07:38:44.929021+0000","last_fullsized":"2024-09-18T07:38:44.929021+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:32.564101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:32.564101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:02.841418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":432,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.935074+0000","last_change":"2024-09-18T07:37:43.016535+0000","last_active":"2024-09-18T07:38:44.935074+0000","last_peered":"2024-09-18T07:38:44.935074+0000","last_clean":"2024-09-18T07:38:44.935074+0000","last_became_active":"2024-09-18T07:35:01.418735+0000","last_became_peered":"2024-09-18T07:35:01.418735+0000","last_unstale":"2024-09-18T07:38:44.935074+0000","last_undegraded":"2024-09-18T07:38:44.935074+0000","last_fullsized":"2024-09-18T07:38:44.935074+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:43.016489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:37:43.016489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:12.741671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"265'10855","reported_seq":10996,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:46.752168+0000","last_change":"2024-09-18T07:35:59.107480+0000","last_active":"2024-09-18T07:38:46.752168+0000","last_peered":"2024-09-18T07:38:46.752168+0000","last_clean":"2024-09-18T07:38:46.752168+0000","last_became_active":"2024-09-18T07:35:59.106549+0000","last_became_peered":"2024-09-18T07:35:59.106549+0000","last_unstale":"2024-09-18T07:38:46.752168+0000","last_undegraded":"2024-09-18T07:38:46.752168+0000","last_fullsized":"2024-09-18T07:38:46.752168+0000","mapping_epoch":201,"log_start":"230'5700","ondisk_log_start":"230'5700","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:35:58.086571+0000","last_clean_scrub_stamp":"2024-09-18T07:35:58.086571+0000","objects_scrubbed":0,"log_size":5155,"log_dups_size":0,"ondisk_log_size":5155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154861568,"num_objects":2363,"num_object_clones":0,"num_object_copies":7089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10855,"num_write_kb":422976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":550,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.927929+0000","last_change":"2024-09-18T07:37:43.157488+0000","last_active":"2024-09-18T07:38:44.927929+0000","last_peered":"2024-09-18T07:38:44.927929+0000","last_clean":"2024-09-18T07:38:44.927929+0000","last_became_active":"2024-09-18T07:37:43.075552+0000","last_became_peered":"2024-09-18T07:37:43.075552+0000","last_unstale":"2024-09-18T07:38:44.927929+0000","last_undegraded":"2024-09-18T07:38:44.927929+0000","last_fullsized":"2024-09-18T07:38:44.927929+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:35:36.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:36.271863+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":219,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.932612+0000","last_change":"2024-09-18T07:36:09.261574+0000","last_active":"2024-09-18T07:38:44.932612+0000","last_peered":"2024-09-18T07:38:44.932612+0000","last_clean":"2024-09-18T07:38:44.932612+0000","last_became_active":"2024-09-18T07:35:01.406695+0000","last_became_peered":"2024-09-18T07:35:01.406695+0000","last_unstale":"2024-09-18T07:38:44.932612+0000","last_undegraded":"2024-09-18T07:38:44.932612+0000","last_fullsized":"2024-09-18T07:38:44.932612+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:36:09.261516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.261516+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"240'12","reported_seq":277,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.931532+0000","last_change":"2024-09-18T07:37:43.117484+0000","last_active":"2024-09-18T07:38:44.931532+0000","last_peered":"2024-09-18T07:38:44.931532+0000","last_clean":"2024-09-18T07:38:44.931532+0000","last_became_active":"2024-09-18T07:37:43.056714+0000","last_became_peered":"2024-09-18T07:37:43.056714+0000","last_unstale":"2024-09-18T07:38:44.931532+0000","last_undegraded":"2024-09-18T07:38:44.931532+0000","last_fullsized":"2024-09-18T07:38:44.931532+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:35:26.319320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:26.319320+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":423,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.927057+0000","last_change":"2024-09-18T07:37:43.078768+0000","last_active":"2024-09-18T07:38:44.927057+0000","last_peered":"2024-09-18T07:38:44.927057+0000","last_clean":"2024-09-18T07:38:44.927057+0000","last_became_active":"2024-09-18T07:37:43.078121+0000","last_became_peered":"2024-09-18T07:37:43.078121+0000","last_unstale":"2024-09-18T07:38:44.927057+0000","last_undegraded":"2024-09-18T07:38:44.927057+0000","last_fullsized":"2024-09-18T07:38:44.927057+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":64,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.928992+0000","last_change":"2024-09-18T07:37:43.088299+0000","last_active":"2024-09-18T07:38:44.928992+0000","last_peered":"2024-09-18T07:38:44.928992+0000","last_clean":"2024-09-18T07:38:44.928992+0000","last_became_active":"2024-09-18T07:37:43.087994+0000","last_became_peered":"2024-09-18T07:37:43.087994+0000","last_unstale":"2024-09-18T07:38:44.928992+0000","last_undegraded":"2024-09-18T07:38:44.928992+0000","last_fullsized":"2024-09-18T07:38:44.928992+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:01.259296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.259296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":465,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.935882+0000","last_change":"2024-09-18T07:36:09.061983+0000","last_active":"2024-09-18T07:38:44.935882+0000","last_peered":"2024-09-18T07:38:44.935882+0000","last_clean":"2024-09-18T07:38:44.935882+0000","last_became_active":"2024-09-18T07:35:01.407787+0000","last_became_peered":"2024-09-18T07:35:01.407787+0000","last_unstale":"2024-09-18T07:38:44.935882+0000","last_undegraded":"2024-09-18T07:38:44.935882+0000","last_fullsized":"2024-09-18T07:38:44.935882+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-18T07:36:09.061903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.061903+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":429,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.927674+0000","last_change":"2024-09-18T07:36:34.608531+0000","last_active":"2024-09-18T07:38:44.927674+0000","last_peered":"2024-09-18T07:38:44.927674+0000","last_clean":"2024-09-18T07:38:44.927674+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:38:44.927674+0000","last_undegraded":"2024-09-18T07:38:44.927674+0000","last_fullsized":"2024-09-18T07:38:44.927674+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:34.608486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:34.608486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":211,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.934130+0000","last_change":"2024-09-18T07:37:43.079723+0000","last_active":"2024-09-18T07:38:44.934130+0000","last_peered":"2024-09-18T07:38:44.934130+0000","last_clean":"2024-09-18T07:38:44.934130+0000","last_became_active":"2024-09-18T07:37:43.079313+0000","last_became_peered":"2024-09-18T07:37:43.079313+0000","last_unstale":"2024-09-18T07:38:44.934130+0000","last_undegraded":"2024-09-18T07:38:44.934130+0000","last_fullsized":"2024-09-18T07:38:44.934130+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:58.688631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:58.688631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:41.433549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":469,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.221072+0000","last_change":"2024-09-18T07:35:01.448510+0000","last_active":"2024-09-18T07:38:47.221072+0000","last_peered":"2024-09-18T07:38:47.221072+0000","last_clean":"2024-09-18T07:38:47.221072+0000","last_became_active":"2024-09-18T07:35:01.414458+0000","last_became_peered":"2024-09-18T07:35:01.414458+0000","last_unstale":"2024-09-18T07:38:47.221072+0000","last_undegraded":"2024-09-18T07:38:47.221072+0000","last_fullsized":"2024-09-18T07:38:47.221072+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":219,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.931386+0000","last_change":"2024-09-18T07:37:43.057776+0000","last_active":"2024-09-18T07:38:44.931386+0000","last_peered":"2024-09-18T07:38:44.931386+0000","last_clean":"2024-09-18T07:38:44.931386+0000","last_became_active":"2024-09-18T07:37:43.056675+0000","last_became_peered":"2024-09-18T07:37:43.056675+0000","last_unstale":"2024-09-18T07:38:44.931386+0000","last_undegraded":"2024-09-18T07:38:44.931386+0000","last_fullsized":"2024-09-18T07:38:44.931386+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:05.201723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.201723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":444,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.935717+0000","last_change":"2024-09-18T07:35:59.997299+0000","last_active":"2024-09-18T07:38:44.935717+0000","last_peered":"2024-09-18T07:38:44.935717+0000","last_clean":"2024-09-18T07:38:44.935717+0000","last_became_active":"2024-09-18T07:35:01.420133+0000","last_became_peered":"2024-09-18T07:35:01.420133+0000","last_unstale":"2024-09-18T07:38:44.935717+0000","last_undegraded":"2024-09-18T07:38:44.935717+0000","last_fullsized":"2024-09-18T07:38:44.935717+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:59.997249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:59.997249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":433,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929140+0000","last_change":"2024-09-18T07:37:48.861965+0000","last_active":"2024-09-18T07:38:44.929140+0000","last_peered":"2024-09-18T07:38:44.929140+0000","last_clean":"2024-09-18T07:38:44.929140+0000","last_became_active":"2024-09-18T07:35:01.396305+0000","last_became_peered":"2024-09-18T07:35:01.396305+0000","last_unstale":"2024-09-18T07:38:44.929140+0000","last_undegraded":"2024-09-18T07:38:44.929140+0000","last_fullsized":"2024-09-18T07:38:44.929140+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:48.861784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:37:48.861784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:17.251260+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":475,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.928070+0000","last_change":"2024-09-18T07:35:02.342002+0000","last_active":"2024-09-18T07:38:44.928070+0000","last_peered":"2024-09-18T07:38:44.928070+0000","last_clean":"2024-09-18T07:38:44.928070+0000","last_became_active":"2024-09-18T07:35:01.416744+0000","last_became_peered":"2024-09-18T07:35:01.416744+0000","last_unstale":"2024-09-18T07:38:44.928070+0000","last_undegraded":"2024-09-18T07:38:44.928070+0000","last_fullsized":"2024-09-18T07:38:44.928070+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:35:02.341919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.341919+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":445,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.220484+0000","last_change":"2024-09-18T07:35:01.414704+0000","last_active":"2024-09-18T07:38:47.220484+0000","last_peered":"2024-09-18T07:38:47.220484+0000","last_clean":"2024-09-18T07:38:47.220484+0000","last_became_active":"2024-09-18T07:35:01.414024+0000","last_became_peered":"2024-09-18T07:35:01.414024+0000","last_unstale":"2024-09-18T07:38:47.220484+0000","last_undegraded":"2024-09-18T07:38:47.220484+0000","last_fullsized":"2024-09-18T07:38:47.220484+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:34:35.707148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:34:35.707148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"173'10","reported_seq":547,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.935762+0000","last_change":"2024-09-18T07:36:08.041535+0000","last_active":"2024-09-18T07:38:44.935762+0000","last_peered":"2024-09-18T07:38:44.935762+0000","last_clean":"2024-09-18T07:38:44.935762+0000","last_became_active":"2024-09-18T07:35:01.407568+0000","last_became_peered":"2024-09-18T07:35:01.407568+0000","last_unstale":"2024-09-18T07:38:44.935762+0000","last_undegraded":"2024-09-18T07:38:44.935762+0000","last_fullsized":"2024-09-18T07:38:44.935762+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:36:08.041456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:08.041456+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":419,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.926092+0000","last_change":"2024-09-18T07:36:11.910543+0000","last_active":"2024-09-18T07:38:44.926092+0000","last_peered":"2024-09-18T07:38:44.926092+0000","last_clean":"2024-09-18T07:38:44.926092+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:38:44.926092+0000","last_undegraded":"2024-09-18T07:38:44.926092+0000","last_fullsized":"2024-09-18T07:38:44.926092+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:11.910474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:11.910474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":487,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.935936+0000","last_change":"2024-09-18T07:36:02.975929+0000","last_active":"2024-09-18T07:38:44.935936+0000","last_peered":"2024-09-18T07:38:44.935936+0000","last_clean":"2024-09-18T07:38:44.935936+0000","last_became_active":"2024-09-18T07:35:01.407738+0000","last_became_peered":"2024-09-18T07:35:01.407738+0000","last_unstale":"2024-09-18T07:38:44.935936+0000","last_undegraded":"2024-09-18T07:38:44.935936+0000","last_fullsized":"2024-09-18T07:38:44.935936+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:36:02.975863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:02.975863+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":205,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.932217+0000","last_change":"2024-09-18T07:36:08.222898+0000","last_active":"2024-09-18T07:38:44.932217+0000","last_peered":"2024-09-18T07:38:44.932217+0000","last_clean":"2024-09-18T07:38:44.932217+0000","last_became_active":"2024-09-18T07:35:01.400197+0000","last_became_peered":"2024-09-18T07:35:01.400197+0000","last_unstale":"2024-09-18T07:38:44.932217+0000","last_undegraded":"2024-09-18T07:38:44.932217+0000","last_fullsized":"2024-09-18T07:38:44.932217+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:08.222838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:08.222838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":527,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.927754+0000","last_change":"2024-09-18T07:37:42.285094+0000","last_active":"2024-09-18T07:38:44.927754+0000","last_peered":"2024-09-18T07:38:44.927754+0000","last_clean":"2024-09-18T07:38:44.927754+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:38:44.927754+0000","last_undegraded":"2024-09-18T07:38:44.927754+0000","last_fullsized":"2024-09-18T07:38:44.927754+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:37:42.285042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:37:42.285042+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:58.228525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":433,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.220682+0000","last_change":"2024-09-18T07:35:01.411256+0000","last_active":"2024-09-18T07:38:47.220682+0000","last_peered":"2024-09-18T07:38:47.220682+0000","last_clean":"2024-09-18T07:38:47.220682+0000","last_became_active":"2024-09-18T07:35:01.410561+0000","last_became_peered":"2024-09-18T07:35:01.410561+0000","last_unstale":"2024-09-18T07:38:47.220682+0000","last_undegraded":"2024-09-18T07:38:47.220682+0000","last_fullsized":"2024-09-18T07:38:47.220682+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:32.134847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:32.134847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":516,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.925856+0000","last_change":"2024-09-18T07:38:41.163393+0000","last_active":"2024-09-18T07:38:44.925856+0000","last_peered":"2024-09-18T07:38:44.925856+0000","last_clean":"2024-09-18T07:38:44.925856+0000","last_became_active":"2024-09-18T07:38:41.143108+0000","last_became_peered":"2024-09-18T07:38:41.143108+0000","last_unstale":"2024-09-18T07:38:44.925856+0000","last_undegraded":"2024-09-18T07:38:44.925856+0000","last_fullsized":"2024-09-18T07:38:44.925856+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:36:01.986964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.986964+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":451,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.926888+0000","last_change":"2024-09-18T07:37:43.062429+0000","last_active":"2024-09-18T07:38:44.926888+0000","last_peered":"2024-09-18T07:38:44.926888+0000","last_clean":"2024-09-18T07:38:44.926888+0000","last_became_active":"2024-09-18T07:37:43.061577+0000","last_became_peered":"2024-09-18T07:37:43.061577+0000","last_unstale":"2024-09-18T07:38:44.926888+0000","last_undegraded":"2024-09-18T07:38:44.926888+0000","last_fullsized":"2024-09-18T07:38:44.926888+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:32.047161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":122,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.927199+0000","last_change":"2024-09-18T07:36:13.118114+0000","last_active":"2024-09-18T07:38:44.927199+0000","last_peered":"2024-09-18T07:38:44.927199+0000","last_clean":"2024-09-18T07:38:44.927199+0000","last_became_active":"2024-09-18T07:36:13.117584+0000","last_became_peered":"2024-09-18T07:36:13.117584+0000","last_unstale":"2024-09-18T07:38:44.927199+0000","last_undegraded":"2024-09-18T07:38:44.927199+0000","last_fullsized":"2024-09-18T07:38:44.927199+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":410,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.932067+0000","last_change":"2024-09-18T07:31:02.351247+0000","last_active":"2024-09-18T07:38:44.932067+0000","last_peered":"2024-09-18T07:38:44.932067+0000","last_clean":"2024-09-18T07:38:44.932067+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:38:44.932067+0000","last_undegraded":"2024-09-18T07:38:44.932067+0000","last_fullsized":"2024-09-18T07:38:44.932067+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:36.944745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":308,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.219849+0000","last_change":"2024-09-18T07:37:43.138346+0000","last_active":"2024-09-18T07:38:47.219849+0000","last_peered":"2024-09-18T07:38:47.219849+0000","last_clean":"2024-09-18T07:38:47.219849+0000","last_became_active":"2024-09-18T07:37:43.067598+0000","last_became_peered":"2024-09-18T07:37:43.067598+0000","last_unstale":"2024-09-18T07:38:47.219849+0000","last_undegraded":"2024-09-18T07:38:47.219849+0000","last_fullsized":"2024-09-18T07:38:47.219849+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:36:38.638226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:38.638226+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":205,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.933982+0000","last_change":"2024-09-18T07:35:57.293607+0000","last_active":"2024-09-18T07:38:44.933982+0000","last_peered":"2024-09-18T07:38:44.933982+0000","last_clean":"2024-09-18T07:38:44.933982+0000","last_became_active":"2024-09-18T07:35:01.405287+0000","last_became_peered":"2024-09-18T07:35:01.405287+0000","last_unstale":"2024-09-18T07:38:44.933982+0000","last_undegraded":"2024-09-18T07:38:44.933982+0000","last_fullsized":"2024-09-18T07:38:44.933982+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:57.293555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.293555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":422,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.935230+0000","last_change":"2024-09-18T07:35:01.205647+0000","last_active":"2024-09-18T07:38:44.935230+0000","last_peered":"2024-09-18T07:38:44.935230+0000","last_clean":"2024-09-18T07:38:44.935230+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:38:44.935230+0000","last_undegraded":"2024-09-18T07:38:44.935230+0000","last_fullsized":"2024-09-18T07:38:44.935230+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:01.205596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:01.205596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":435,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.219770+0000","last_change":"2024-09-18T07:36:37.593789+0000","last_active":"2024-09-18T07:38:47.219770+0000","last_peered":"2024-09-18T07:38:47.219770+0000","last_clean":"2024-09-18T07:38:47.219770+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:38:47.219770+0000","last_undegraded":"2024-09-18T07:38:47.219770+0000","last_fullsized":"2024-09-18T07:38:47.219770+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:37.593724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:37.593724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":529,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.932242+0000","last_change":"2024-09-18T07:37:43.138620+0000","last_active":"2024-09-18T07:38:44.932242+0000","last_peered":"2024-09-18T07:38:44.932242+0000","last_clean":"2024-09-18T07:38:44.932242+0000","last_became_active":"2024-09-18T07:37:43.066208+0000","last_became_peered":"2024-09-18T07:37:43.066208+0000","last_unstale":"2024-09-18T07:38:44.932242+0000","last_undegraded":"2024-09-18T07:38:44.932242+0000","last_fullsized":"2024-09-18T07:38:44.932242+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:36:29.051437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:29.051437+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":443,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.927891+0000","last_change":"2024-09-18T07:35:01.415691+0000","last_active":"2024-09-18T07:38:44.927891+0000","last_peered":"2024-09-18T07:38:44.927891+0000","last_clean":"2024-09-18T07:38:44.927891+0000","last_became_active":"2024-09-18T07:35:01.415261+0000","last_became_peered":"2024-09-18T07:35:01.415261+0000","last_unstale":"2024-09-18T07:38:44.927891+0000","last_undegraded":"2024-09-18T07:38:44.927891+0000","last_fullsized":"2024-09-18T07:38:44.927891+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:58.734027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:58.734027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":219,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.932520+0000","last_change":"2024-09-18T07:37:43.057780+0000","last_active":"2024-09-18T07:38:44.932520+0000","last_peered":"2024-09-18T07:38:44.932520+0000","last_clean":"2024-09-18T07:38:44.932520+0000","last_became_active":"2024-09-18T07:37:43.057483+0000","last_became_peered":"2024-09-18T07:37:43.057483+0000","last_unstale":"2024-09-18T07:38:44.932520+0000","last_undegraded":"2024-09-18T07:38:44.932520+0000","last_fullsized":"2024-09-18T07:38:44.932520+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:41.311291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:41.311291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":440,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.935328+0000","last_change":"2024-09-18T07:37:43.068364+0000","last_active":"2024-09-18T07:38:44.935328+0000","last_peered":"2024-09-18T07:38:44.935328+0000","last_clean":"2024-09-18T07:38:44.935328+0000","last_became_active":"2024-09-18T07:37:43.067066+0000","last_became_peered":"2024-09-18T07:37:43.067066+0000","last_unstale":"2024-09-18T07:38:44.935328+0000","last_undegraded":"2024-09-18T07:38:44.935328+0000","last_fullsized":"2024-09-18T07:38:44.935328+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:36.026215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:36.026215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":487,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.932854+0000","last_change":"2024-09-18T07:37:43.134956+0000","last_active":"2024-09-18T07:38:44.932854+0000","last_peered":"2024-09-18T07:38:44.932854+0000","last_clean":"2024-09-18T07:38:44.932854+0000","last_became_active":"2024-09-18T07:37:43.067503+0000","last_became_peered":"2024-09-18T07:37:43.067503+0000","last_unstale":"2024-09-18T07:38:44.932854+0000","last_undegraded":"2024-09-18T07:38:44.932854+0000","last_fullsized":"2024-09-18T07:38:44.932854+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:36:40.908629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:40.908629+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":423,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.221867+0000","last_change":"2024-09-18T07:30:56.283643+0000","last_active":"2024-09-18T07:38:47.221867+0000","last_peered":"2024-09-18T07:38:47.221867+0000","last_clean":"2024-09-18T07:38:47.221867+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:38:47.221867+0000","last_undegraded":"2024-09-18T07:38:47.221867+0000","last_fullsized":"2024-09-18T07:38:47.221867+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":64,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.929409+0000","last_change":"2024-09-18T07:37:43.061403+0000","last_active":"2024-09-18T07:38:44.929409+0000","last_peered":"2024-09-18T07:38:44.929409+0000","last_clean":"2024-09-18T07:38:44.929409+0000","last_became_active":"2024-09-18T07:37:43.061081+0000","last_became_peered":"2024-09-18T07:37:43.061081+0000","last_unstale":"2024-09-18T07:38:44.929409+0000","last_undegraded":"2024-09-18T07:38:44.929409+0000","last_fullsized":"2024-09-18T07:38:44.929409+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:30.345180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:30.345180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":205,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.931776+0000","last_change":"2024-09-18T07:35:29.235047+0000","last_active":"2024-09-18T07:38:44.931776+0000","last_peered":"2024-09-18T07:38:44.931776+0000","last_clean":"2024-09-18T07:38:44.931776+0000","last_became_active":"2024-09-18T07:35:01.400396+0000","last_became_peered":"2024-09-18T07:35:01.400396+0000","last_unstale":"2024-09-18T07:38:44.931776+0000","last_undegraded":"2024-09-18T07:38:44.931776+0000","last_fullsized":"2024-09-18T07:38:44.931776+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:29.234993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:29.234993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":122,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.927833+0000","last_change":"2024-09-18T07:36:13.118009+0000","last_active":"2024-09-18T07:38:44.927833+0000","last_peered":"2024-09-18T07:38:44.927833+0000","last_clean":"2024-09-18T07:38:44.927833+0000","last_became_active":"2024-09-18T07:36:13.117154+0000","last_became_peered":"2024-09-18T07:36:13.117154+0000","last_unstale":"2024-09-18T07:38:44.927833+0000","last_undegraded":"2024-09-18T07:38:44.927833+0000","last_fullsized":"2024-09-18T07:38:44.927833+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.071629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.071629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":444,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.933063+0000","last_change":"2024-09-18T07:35:57.951851+0000","last_active":"2024-09-18T07:38:44.933063+0000","last_peered":"2024-09-18T07:38:44.933063+0000","last_clean":"2024-09-18T07:38:44.933063+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:38:44.933063+0000","last_undegraded":"2024-09-18T07:38:44.933063+0000","last_fullsized":"2024-09-18T07:38:44.933063+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:35:57.951801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.951801+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":76,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-09-18T07:38:44.931092+0000","last_change":"2024-09-18T07:37:43.258255+0000","last_active":"2024-09-18T07:38:44.931092+0000","last_peered":"2024-09-18T07:38:44.931092+0000","last_clean":"2024-09-18T07:38:44.931092+0000","last_became_active":"2024-09-18T07:37:43.070186+0000","last_became_peered":"2024-09-18T07:37:43.070186+0000","last_unstale":"2024-09-18T07:38:44.931092+0000","last_undegraded":"2024-09-18T07:38:44.931092+0000","last_fullsized":"2024-09-18T07:38:44.931092+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:34:58.432781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:58.432781+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":445,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-09-18T07:38:47.220973+0000","last_change":"2024-09-18T07:35:25.576024+0000","last_active":"2024-09-18T07:38:47.220973+0000","last_peered":"2024-09-18T07:38:47.220973+0000","last_clean":"2024-09-18T07:38:47.220973+0000","last_became_active":"2024-09-18T07:35:01.410469+0000","last_became_peered":"2024-09-18T07:35:01.410469+0000","last_unstale":"2024-09-18T07:38:47.220973+0000","last_undegraded":"2024-09-18T07:38:47.220973+0000","last_fullsized":"2024-09-18T07:38:47.220973+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:25.575954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:25.575954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2420768792,"num_objects":36939,"num_object_clones":0,"num_object_copies":110817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2386,"num_objects_unfound":0,"num_objects_dirty":36939,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172677,"num_write_kb":6707649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22259,"num_bytes_recovered":1399390208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8546824192,"data_stored":8546811976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85266,"ondisk_log_size":85266,"up":48,"acting":46,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":4664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":398,"ondisk_log_size":398,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":231,"num_read_kb":226,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201378,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":936420,"kb_used_data":746960,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92808732,"statfs":{"total":95995035648,"available":95036141568,"internally_reserved":0,"allocated":764887040,"data_stored":764395236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":193984970},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463016,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":848512,"kb_used_data":641004,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":92896640,"statfs":{"total":95995035648,"available":95126159360,"internally_reserved":0,"allocated":656388096,"data_stored":655931717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":212466118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724653,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1481036,"kb_used_data":1206780,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92264116,"statfs":{"total":95995035648,"available":94478454784,"internally_reserved":0,"allocated":1235742720,"data_stored":1235270223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953588,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":978972,"kb_used_data":767304,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92766180,"statfs":{"total":95995035648,"available":94992568320,"internally_reserved":0,"allocated":785719296,"data_stored":785234839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1557144,"kb_used_data":1201096,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92188008,"statfs":{"total":95995035648,"available":94400520192,"internally_reserved":0,"allocated":1229922304,"data_stored":1229438699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":364575175},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411454,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1973128,"kb_used_data":1644856,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":91772024,"statfs":{"total":95995035648,"available":93974552576,"internally_reserved":0,"allocated":1684332544,"data_stored":1683864042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574981,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1367080,"kb_used_data":1065172,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92378072,"statfs":{"total":95995035648,"available":94595145728,"internally_reserved":0,"allocated":1090736128,"data_stored":1090240460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738506,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1482020,"kb_used_data":1171796,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92263132,"statfs":{"total":95995035648,"available":94477447168,"internally_reserved":0,"allocated":1199919104,"data_stored":1199424305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1179516928,"data_stored":1179516928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1068892160,"data_stored":1068892160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1661341696,"data_stored":1661337624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1227292672,"data_stored":1227292672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":773390336,"data_stored":773390336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1225527296,"data_stored":1225523224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":648216576,"data_stored":648216576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":762646528,"data_stored":762642456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:38:52.617 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-18T07:38:52.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-18T07:38:52.629 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:52.658 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:52.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:52.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:52 smithi203 ceph-mon[32237]: 3.19 scrub starts 2024-09-18T07:38:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:52 smithi203 ceph-mon[32237]: 3.19 scrub ok 2024-09-18T07:38:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:52 smithi203 ceph-mon[32237]: from='client.25507 v1:172.21.15.137:0/3195244161' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:38:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:52 smithi203 ceph-mon[32237]: pgmap v421: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 2.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 938 op/s; 2386/110409 objects misplaced (2.161%); 7.4 MiB/s, 118 objects/s recovering 2024-09-18T07:38:52.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:52.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:52.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:52.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:52.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:52.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:52.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:52.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:53.035 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:53.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:53.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:53.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:53.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:53.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:53.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:53.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:53.315 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:38:53.316 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:38:53.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:53.330 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:53.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:53 smithi137 ceph-mon[29408]: 4.18 scrub starts 2024-09-18T07:38:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:53 smithi137 ceph-mon[29408]: 4.18 scrub ok 2024-09-18T07:38:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:53 smithi137 ceph-mon[29408]: 5.4 scrub starts 2024-09-18T07:38:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:53 smithi137 ceph-mon[29408]: 5.4 scrub ok 2024-09-18T07:38:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:53 smithi137 ceph-mon[39175]: 4.18 scrub starts 2024-09-18T07:38:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:53 smithi137 ceph-mon[39175]: 4.18 scrub ok 2024-09-18T07:38:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:53 smithi137 ceph-mon[39175]: 5.4 scrub starts 2024-09-18T07:38:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:53 smithi137 ceph-mon[39175]: 5.4 scrub ok 2024-09-18T07:38:53.431 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:53.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:53.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:53.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:53.557 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:53.596 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:53.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:53.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:53.698 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:53.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:53 smithi203 ceph-mon[32237]: 4.18 scrub starts 2024-09-18T07:38:53.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:53 smithi203 ceph-mon[32237]: 4.18 scrub ok 2024-09-18T07:38:53.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:53 smithi203 ceph-mon[32237]: 5.4 scrub starts 2024-09-18T07:38:53.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:53 smithi203 ceph-mon[32237]: 5.4 scrub ok 2024-09-18T07:38:53.729 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:53.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:53.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:53.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:38:53.831 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:53.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:53.866 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:53.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:53.967 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:54.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:54.008 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:54.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:54.108 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:54.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:54.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:54.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:54.272 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:54.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:54.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:54.374 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:54.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:54.405 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:54.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:54.506 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:54.527 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:54.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:54.628 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:54.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:54.651 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:54.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:54.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:54 smithi137 ceph-mon[29408]: pgmap v422: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 1.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 803 B/s rd, 0 B/s wr, 1.11k op/s; 1620/94068 objects misplaced (1.722%); 7.5 MiB/s, 120 objects/s recovering 2024-09-18T07:38:54.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:54 smithi137 ceph-mon[39175]: pgmap v422: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 1.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 803 B/s rd, 0 B/s wr, 1.11k op/s; 1620/94068 objects misplaced (1.722%); 7.5 MiB/s, 120 objects/s recovering 2024-09-18T07:38:54.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:54 smithi203 ceph-mon[32237]: pgmap v422: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 1.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 803 B/s rd, 0 B/s wr, 1.11k op/s; 1620/94068 objects misplaced (1.722%); 7.5 MiB/s, 120 objects/s recovering 2024-09-18T07:38:54.752 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:54.774 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:54.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:54.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:54.877 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:54.902 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:54.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:54.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:55.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:55.028 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:55.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:55.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:55.130 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:55.154 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:55.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:55.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:55.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:55.281 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:55.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:55.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:55.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:55.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:55.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:55.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:55.523 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:55.539 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:55 smithi137 ceph-mon[29408]: osdmap e269: 8 total, 8 up, 8 in 2024-09-18T07:38:55.539 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:55 smithi137 ceph-mon[29408]: 5.1a scrub starts 2024-09-18T07:38:55.539 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:55 smithi137 ceph-mon[29408]: 5.1a scrub ok 2024-09-18T07:38:55.539 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:55 smithi137 ceph-mon[39175]: osdmap e269: 8 total, 8 up, 8 in 2024-09-18T07:38:55.539 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:55 smithi137 ceph-mon[39175]: 5.1a scrub starts 2024-09-18T07:38:55.539 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:55 smithi137 ceph-mon[39175]: 5.1a scrub ok 2024-09-18T07:38:55.572 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:55.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:55.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:55.673 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:55.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:55.710 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:55.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:55 smithi203 ceph-mon[32237]: osdmap e269: 8 total, 8 up, 8 in 2024-09-18T07:38:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:55 smithi203 ceph-mon[32237]: 5.1a scrub starts 2024-09-18T07:38:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:55 smithi203 ceph-mon[32237]: 5.1a scrub ok 2024-09-18T07:38:55.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:55.835 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:55.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:55.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:55.937 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:55.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:55.979 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:55.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:56.081 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:56.115 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:56.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:56.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:56.217 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:56.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:56.252 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:56.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:56.355 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:56.386 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:56.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:56.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:56.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:56.536 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:56.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:56.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:56.544 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[29408]: 4.f scrub starts 2024-09-18T07:38:56.545 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[29408]: 4.f scrub ok 2024-09-18T07:38:56.545 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[29408]: 4.d scrub starts 2024-09-18T07:38:56.545 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[29408]: 4.d scrub ok 2024-09-18T07:38:56.545 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[29408]: 5.d scrub starts 2024-09-18T07:38:56.545 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[29408]: 5.d scrub ok 2024-09-18T07:38:56.545 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[29408]: osdmap e270: 8 total, 8 up, 8 in 2024-09-18T07:38:56.545 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[29408]: 5.9 scrub starts 2024-09-18T07:38:56.545 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[29408]: 5.9 scrub ok 2024-09-18T07:38:56.545 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:38:56.545 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[29408]: pgmap v425: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 1.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 964 op/s; 1620/94068 objects misplaced (1.722%); 8.0 MiB/s, 127 objects/s recovering 2024-09-18T07:38:56.546 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[39175]: 4.f scrub starts 2024-09-18T07:38:56.546 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[39175]: 4.f scrub ok 2024-09-18T07:38:56.546 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[39175]: 4.d scrub starts 2024-09-18T07:38:56.546 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[39175]: 4.d scrub ok 2024-09-18T07:38:56.546 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[39175]: 5.d scrub starts 2024-09-18T07:38:56.546 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[39175]: 5.d scrub ok 2024-09-18T07:38:56.546 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[39175]: osdmap e270: 8 total, 8 up, 8 in 2024-09-18T07:38:56.546 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[39175]: 5.9 scrub starts 2024-09-18T07:38:56.546 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[39175]: 5.9 scrub ok 2024-09-18T07:38:56.546 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:38:56.546 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:56 smithi137 ceph-mon[39175]: pgmap v425: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 1.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 964 op/s; 1620/94068 objects misplaced (1.722%); 8.0 MiB/s, 127 objects/s recovering 2024-09-18T07:38:56.575 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:56.625 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:38:56.625 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:38:56.638 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:56.665 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:38:56.666 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:38:56.667 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:56.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:56.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:56.703 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:38:56.703 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:38:56.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:56 smithi203 ceph-mon[32237]: 4.f scrub starts 2024-09-18T07:38:56.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:56 smithi203 ceph-mon[32237]: 4.f scrub ok 2024-09-18T07:38:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:56 smithi203 ceph-mon[32237]: 4.d scrub starts 2024-09-18T07:38:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:56 smithi203 ceph-mon[32237]: 4.d scrub ok 2024-09-18T07:38:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:56 smithi203 ceph-mon[32237]: 5.d scrub starts 2024-09-18T07:38:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:56 smithi203 ceph-mon[32237]: 5.d scrub ok 2024-09-18T07:38:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:56 smithi203 ceph-mon[32237]: osdmap e270: 8 total, 8 up, 8 in 2024-09-18T07:38:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:56 smithi203 ceph-mon[32237]: 5.9 scrub starts 2024-09-18T07:38:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:56 smithi203 ceph-mon[32237]: 5.9 scrub ok 2024-09-18T07:38:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:56 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:38:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:56 smithi203 ceph-mon[32237]: pgmap v425: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 1.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 964 op/s; 1620/94068 objects misplaced (1.722%); 8.0 MiB/s, 127 objects/s recovering 2024-09-18T07:38:56.738 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:38:56.739 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:38:56.769 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:56.770 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:38:56.770 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:38:56.798 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:56.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:56.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:56.803 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:38:56.804 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:38:56.834 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:38:56.834 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:38:56.865 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:38:56.866 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:38:56.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:56.951 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:56.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:57.053 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:57.090 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:57.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:57.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:57.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:57.229 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:57.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:57.330 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:57.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:57.369 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:57.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:57.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:38:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:38:57.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[29408]: 6.1e scrub starts 2024-09-18T07:38:57.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[29408]: 6.1e scrub ok 2024-09-18T07:38:57.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:38:57.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[29408]: 6.1c scrub starts 2024-09-18T07:38:57.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[29408]: 6.1c scrub ok 2024-09-18T07:38:57.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:38:57.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:38:57.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:38:57.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1917387405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:57.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[39175]: 6.1e scrub starts 2024-09-18T07:38:57.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[39175]: 6.1e scrub ok 2024-09-18T07:38:57.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:38:57.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[39175]: 6.1c scrub starts 2024-09-18T07:38:57.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[39175]: 6.1c scrub ok 2024-09-18T07:38:57.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:38:57.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:38:57.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:38:57.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:57 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1917387405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:57.471 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:57.509 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:57.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:57.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:57.611 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:57.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:57.654 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:57.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:57.710 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:38:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:38:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:57 smithi203 ceph-mon[32237]: 6.1e scrub starts 2024-09-18T07:38:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:57 smithi203 ceph-mon[32237]: 6.1e scrub ok 2024-09-18T07:38:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:57 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:38:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:57 smithi203 ceph-mon[32237]: 6.1c scrub starts 2024-09-18T07:38:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:57 smithi203 ceph-mon[32237]: 6.1c scrub ok 2024-09-18T07:38:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:57 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:38:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:57 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:38:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:57 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:38:57.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:57 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1917387405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:38:57.755 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:57.789 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:57.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:57.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:57.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:57.933 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:57.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:57.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:58.035 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:58.065 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:58.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:58.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:58.167 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:38:58.196 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:58.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:58.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:38:58.298 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:38:58.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:58.338 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:58.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:38:58.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T07:38:58.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:58.451 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:58 smithi137 ceph-mon[29408]: 5.13 scrub starts 2024-09-18T07:38:58.451 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:58 smithi137 ceph-mon[29408]: 5.13 scrub ok 2024-09-18T07:38:58.452 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:58 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:58.452 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:58 smithi137 ceph-mon[29408]: pgmap v426: 148 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 146 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.41k op/s; 1094/80361 objects misplaced (1.361%); 11 MiB/s, 218 objects/s recovering 2024-09-18T07:38:58.452 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:58 smithi137 ceph-mon[39175]: 5.13 scrub starts 2024-09-18T07:38:58.452 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:58 smithi137 ceph-mon[39175]: 5.13 scrub ok 2024-09-18T07:38:58.452 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:58 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:58.452 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:58 smithi137 ceph-mon[39175]: pgmap v426: 148 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 146 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.41k op/s; 1094/80361 objects misplaced (1.361%); 11 MiB/s, 218 objects/s recovering 2024-09-18T07:38:58.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:58.489 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:58.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:58.590 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:58.624 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:58.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:58.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:58.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:58 smithi203 ceph-mon[32237]: 5.13 scrub starts 2024-09-18T07:38:58.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:58 smithi203 ceph-mon[32237]: 5.13 scrub ok 2024-09-18T07:38:58.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:58 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:38:58.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:58 smithi203 ceph-mon[32237]: pgmap v426: 148 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 146 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.41k op/s; 1094/80361 objects misplaced (1.361%); 11 MiB/s, 218 objects/s recovering 2024-09-18T07:38:58.725 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:38:58.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:58.764 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:58.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:38:58.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:38:58.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:58.902 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:58.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:38:58.923 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:59.002 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:59.031 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:59.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:59.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:59.133 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:59.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:59.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:59.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:59.278 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:59.333 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:59.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:59.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:59.385 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:59 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-18T07:38:59.386 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:59 smithi137 ceph-mon[29408]: 6.3 scrub starts 2024-09-18T07:38:59.386 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:38:59 smithi137 ceph-mon[29408]: 6.3 scrub ok 2024-09-18T07:38:59.386 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:59 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-18T07:38:59.386 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:59 smithi137 ceph-mon[39175]: 6.3 scrub starts 2024-09-18T07:38:59.386 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:38:59 smithi137 ceph-mon[39175]: 6.3 scrub ok 2024-09-18T07:38:59.434 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:38:59.467 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:38:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:59.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:38:59.569 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:38:59.574 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:38:59.596 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:59.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:59.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:38:59.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:59 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-18T07:38:59.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:59 smithi203 ceph-mon[32237]: 6.3 scrub starts 2024-09-18T07:38:59.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:38:59 smithi203 ceph-mon[32237]: 6.3 scrub ok 2024-09-18T07:38:59.698 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:38:59.733 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:59.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:59.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:38:59.835 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:38:59.866 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:38:59.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:38:59.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:38:59.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:38:59.934 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:38:59.950 INFO:teuthology.orchestra.run.smithi137.stderr:set pool 1 pg_num to 11 2024-09-18T07:38:59.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:00.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:00.052 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:00.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:00.154 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:00.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:00.196 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:00.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:00.297 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:00.335 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:00.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:00.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:00.437 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:00.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:00.477 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:00.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:00.486 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:39:00.486 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:39:00.486 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:39:00.486 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:39:00.486 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:39:00.578 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:00.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:00.610 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:00.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:00.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[29408]: 4.9 scrub starts 2024-09-18T07:39:00.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[29408]: 4.9 scrub ok 2024-09-18T07:39:00.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3457358447' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-18T07:39:00.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[29408]: from='client.35447 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-18T07:39:00.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[29408]: pgmap v427: 148 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 146 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.25k op/s; 1094/80361 objects misplaced (1.361%); 10 MiB/s, 200 objects/s recovering 2024-09-18T07:39:00.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/917701519' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:00.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[29408]: from='client.25519 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:00.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:39:00.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[29408]: from='client.35447 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-18T07:39:00.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[29408]: from='client.25519 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:39:00.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[29408]: osdmap e271: 8 total, 8 up, 8 in 2024-09-18T07:39:00.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[39175]: 4.9 scrub starts 2024-09-18T07:39:00.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[39175]: 4.9 scrub ok 2024-09-18T07:39:00.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3457358447' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-18T07:39:00.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[39175]: from='client.35447 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-18T07:39:00.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[39175]: pgmap v427: 148 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 146 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.25k op/s; 1094/80361 objects misplaced (1.361%); 10 MiB/s, 200 objects/s recovering 2024-09-18T07:39:00.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/917701519' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:00.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[39175]: from='client.25519 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:00.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:39:00.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[39175]: from='client.35447 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-18T07:39:00.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[39175]: from='client.25519 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:39:00.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:00 smithi137 ceph-mon[39175]: osdmap e271: 8 total, 8 up, 8 in 2024-09-18T07:39:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:00 smithi203 ceph-mon[32237]: 4.9 scrub starts 2024-09-18T07:39:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:00 smithi203 ceph-mon[32237]: 4.9 scrub ok 2024-09-18T07:39:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:00 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3457358447' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-18T07:39:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:00 smithi203 ceph-mon[32237]: from='client.35447 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-18T07:39:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:00 smithi203 ceph-mon[32237]: pgmap v427: 148 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 146 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.25k op/s; 1094/80361 objects misplaced (1.361%); 10 MiB/s, 200 objects/s recovering 2024-09-18T07:39:00.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:00 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/917701519' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:00.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:00 smithi203 ceph-mon[32237]: from='client.25519 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:00.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:00 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:39:00.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:00 smithi203 ceph-mon[32237]: from='client.35447 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-18T07:39:00.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:00 smithi203 ceph-mon[32237]: from='client.25519 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:39:00.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:00 smithi203 ceph-mon[32237]: osdmap e271: 8 total, 8 up, 8 in 2024-09-18T07:39:00.712 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:00.750 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:00.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:00.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:00.852 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:00.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:00.885 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:00.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:00.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:01.015 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:01.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:01.118 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:01.157 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:01.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:01.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:01.259 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:01.301 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:01.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:01.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:01.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:01.436 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:01.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:01.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:01.537 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:01.574 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:01.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:01.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:01.653 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:01 smithi137 ceph-mon[29408]: 6.6 scrub starts 2024-09-18T07:39:01.654 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:01 smithi137 ceph-mon[29408]: 6.6 scrub ok 2024-09-18T07:39:01.654 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:01 smithi137 ceph-mon[29408]: 3.1a scrub starts 2024-09-18T07:39:01.654 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:01 smithi137 ceph-mon[29408]: 3.1a scrub ok 2024-09-18T07:39:01.654 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:01 smithi137 ceph-mon[29408]: 4.4 scrub starts 2024-09-18T07:39:01.654 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:01 smithi137 ceph-mon[29408]: 4.4 scrub ok 2024-09-18T07:39:01.654 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:01 smithi137 ceph-mon[29408]: 1.0 scrub starts 2024-09-18T07:39:01.654 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:01 smithi137 ceph-mon[29408]: 1.0 scrub ok 2024-09-18T07:39:01.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:01 smithi137 ceph-mon[39175]: 6.6 scrub starts 2024-09-18T07:39:01.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:01 smithi137 ceph-mon[39175]: 6.6 scrub ok 2024-09-18T07:39:01.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:01 smithi137 ceph-mon[39175]: 3.1a scrub starts 2024-09-18T07:39:01.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:01 smithi137 ceph-mon[39175]: 3.1a scrub ok 2024-09-18T07:39:01.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:01 smithi137 ceph-mon[39175]: 4.4 scrub starts 2024-09-18T07:39:01.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:01 smithi137 ceph-mon[39175]: 4.4 scrub ok 2024-09-18T07:39:01.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:01 smithi137 ceph-mon[39175]: 1.0 scrub starts 2024-09-18T07:39:01.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:01 smithi137 ceph-mon[39175]: 1.0 scrub ok 2024-09-18T07:39:01.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:01.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:01 smithi203 ceph-mon[32237]: 6.6 scrub starts 2024-09-18T07:39:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:01 smithi203 ceph-mon[32237]: 6.6 scrub ok 2024-09-18T07:39:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:01 smithi203 ceph-mon[32237]: 3.1a scrub starts 2024-09-18T07:39:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:01 smithi203 ceph-mon[32237]: 3.1a scrub ok 2024-09-18T07:39:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:01 smithi203 ceph-mon[32237]: 4.4 scrub starts 2024-09-18T07:39:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:01 smithi203 ceph-mon[32237]: 4.4 scrub ok 2024-09-18T07:39:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:01 smithi203 ceph-mon[32237]: 1.0 scrub starts 2024-09-18T07:39:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:01 smithi203 ceph-mon[32237]: 1.0 scrub ok 2024-09-18T07:39:01.734 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:01.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:01.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:01.836 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:01.874 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:01.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:01.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:01.976 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:02.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:02.013 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:02.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:02.114 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:02.163 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:02.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:02.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:02.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:02.304 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:02.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:02.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:02.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[29408]: 6.0 scrub starts 2024-09-18T07:39:02.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[29408]: 6.0 scrub ok 2024-09-18T07:39:02.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[29408]: 4.0 scrub starts 2024-09-18T07:39:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[29408]: 4.0 scrub ok 2024-09-18T07:39:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[29408]: 5.a scrub starts 2024-09-18T07:39:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[29408]: 5.a scrub ok 2024-09-18T07:39:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[29408]: pgmap v429: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 1.6 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 671 op/s; 1094/79356 objects misplaced (1.379%); 4.5 MiB/s, 115 objects/s recovering 2024-09-18T07:39:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-18T07:39:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-18T07:39:02.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[39175]: 6.0 scrub starts 2024-09-18T07:39:02.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[39175]: 6.0 scrub ok 2024-09-18T07:39:02.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[39175]: 4.0 scrub starts 2024-09-18T07:39:02.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[39175]: 4.0 scrub ok 2024-09-18T07:39:02.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[39175]: 5.a scrub starts 2024-09-18T07:39:02.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[39175]: 5.a scrub ok 2024-09-18T07:39:02.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[39175]: pgmap v429: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 1.6 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 671 op/s; 1094/79356 objects misplaced (1.379%); 4.5 MiB/s, 115 objects/s recovering 2024-09-18T07:39:02.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-18T07:39:02.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:02 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-18T07:39:02.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:02.447 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:02.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:02.548 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:02.584 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:02.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:02.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:02.686 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:02 smithi203 ceph-mon[32237]: 6.0 scrub starts 2024-09-18T07:39:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:02 smithi203 ceph-mon[32237]: 6.0 scrub ok 2024-09-18T07:39:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:02 smithi203 ceph-mon[32237]: 4.0 scrub starts 2024-09-18T07:39:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:02 smithi203 ceph-mon[32237]: 4.0 scrub ok 2024-09-18T07:39:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:02 smithi203 ceph-mon[32237]: 5.a scrub starts 2024-09-18T07:39:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:02 smithi203 ceph-mon[32237]: 5.a scrub ok 2024-09-18T07:39:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:02 smithi203 ceph-mon[32237]: pgmap v429: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 1.6 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 671 op/s; 1094/79356 objects misplaced (1.379%); 4.5 MiB/s, 115 objects/s recovering 2024-09-18T07:39:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:02 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-18T07:39:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:02 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-18T07:39:02.721 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:02.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:02.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:02.751 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T07:39:02.772 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:02.822 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:02.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:02.868 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:02.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:02.969 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:03.016 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:03.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:03.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:03.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:03.170 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:03.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:03.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:03.272 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:03.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:03.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:03.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:03.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:03.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:03.462 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:03.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:03.477 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:03.484 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:39:03.484 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:39:03.522 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:39:03.524 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:39:03.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:03 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-18T07:39:03.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:03 smithi137 ceph-mon[29408]: osdmap e272: 8 total, 8 up, 8 in 2024-09-18T07:39:03.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:03 smithi137 ceph-mon[29408]: 6.12 scrub starts 2024-09-18T07:39:03.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:03 smithi137 ceph-mon[29408]: 6.12 scrub ok 2024-09-18T07:39:03.554 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:03 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-18T07:39:03.554 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:03 smithi137 ceph-mon[39175]: osdmap e272: 8 total, 8 up, 8 in 2024-09-18T07:39:03.554 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:03 smithi137 ceph-mon[39175]: 6.12 scrub starts 2024-09-18T07:39:03.555 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:03 smithi137 ceph-mon[39175]: 6.12 scrub ok 2024-09-18T07:39:03.555 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:39:03.555 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:39:03.563 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:03.584 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:39:03.585 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:39:03.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:03.594 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:03.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:03.616 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:39:03.616 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:39:03.650 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:39:03.651 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:39:03.682 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:39:03.682 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:39:03.695 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:03.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:03 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-18T07:39:03.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:03 smithi203 ceph-mon[32237]: osdmap e272: 8 total, 8 up, 8 in 2024-09-18T07:39:03.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:03 smithi203 ceph-mon[32237]: 6.12 scrub starts 2024-09-18T07:39:03.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:03 smithi203 ceph-mon[32237]: 6.12 scrub ok 2024-09-18T07:39:03.719 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:39:03.719 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:39:03.730 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:03.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:03.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:03.832 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:03.860 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:03.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:03.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:03.962 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:04.000 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:04.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:04.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:04.102 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:04.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:04.137 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:04.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:04.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:39:04.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:04.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:04.306 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:04.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:04.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:04.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:04.450 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:04.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:04.527 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[29408]: 4.b scrub starts 2024-09-18T07:39:04.527 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[29408]: 4.b scrub ok 2024-09-18T07:39:04.527 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[29408]: 6.13 scrub starts 2024-09-18T07:39:04.527 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[29408]: 6.13 scrub ok 2024-09-18T07:39:04.527 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[29408]: osdmap e273: 8 total, 8 up, 8 in 2024-09-18T07:39:04.527 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4011140968' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:04.527 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[29408]: 1.0 scrub starts 2024-09-18T07:39:04.527 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[29408]: 1.0 scrub ok 2024-09-18T07:39:04.527 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[29408]: pgmap v432: 158 pgs: 1 active+recovering+undersized+remapped, 10 unknown, 147 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 917 op/s; 629/63861 objects misplaced (0.985%); 4.8 MiB/s, 77 objects/s recovering 2024-09-18T07:39:04.528 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-18T07:39:04.528 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-18T07:39:04.528 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[39175]: 4.b scrub starts 2024-09-18T07:39:04.528 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[39175]: 4.b scrub ok 2024-09-18T07:39:04.528 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[39175]: 6.13 scrub starts 2024-09-18T07:39:04.528 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[39175]: 6.13 scrub ok 2024-09-18T07:39:04.528 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[39175]: osdmap e273: 8 total, 8 up, 8 in 2024-09-18T07:39:04.528 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4011140968' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:04.528 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[39175]: 1.0 scrub starts 2024-09-18T07:39:04.529 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[39175]: 1.0 scrub ok 2024-09-18T07:39:04.529 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[39175]: pgmap v432: 158 pgs: 1 active+recovering+undersized+remapped, 10 unknown, 147 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 917 op/s; 629/63861 objects misplaced (0.985%); 4.8 MiB/s, 77 objects/s recovering 2024-09-18T07:39:04.529 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-18T07:39:04.529 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:04 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-18T07:39:04.551 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:04.582 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:04.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:04.683 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:04.701 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:39:04.701 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:39:04.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:04 smithi203 ceph-mon[32237]: 4.b scrub starts 2024-09-18T07:39:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:04 smithi203 ceph-mon[32237]: 4.b scrub ok 2024-09-18T07:39:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:04 smithi203 ceph-mon[32237]: 6.13 scrub starts 2024-09-18T07:39:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:04 smithi203 ceph-mon[32237]: 6.13 scrub ok 2024-09-18T07:39:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:04 smithi203 ceph-mon[32237]: osdmap e273: 8 total, 8 up, 8 in 2024-09-18T07:39:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:04 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4011140968' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:04 smithi203 ceph-mon[32237]: 1.0 scrub starts 2024-09-18T07:39:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:04 smithi203 ceph-mon[32237]: 1.0 scrub ok 2024-09-18T07:39:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:04 smithi203 ceph-mon[32237]: pgmap v432: 158 pgs: 1 active+recovering+undersized+remapped, 10 unknown, 147 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 917 op/s; 629/63861 objects misplaced (0.985%); 4.8 MiB/s, 77 objects/s recovering 2024-09-18T07:39:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:04 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-18T07:39:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:04 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-18T07:39:04.729 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:04.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:04.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:04.831 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:04.871 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:04.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:04.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:04.973 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:05.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:05.009 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:05.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:05.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:05.152 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:05.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:05.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:05.254 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:05.284 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:05.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:05.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:05.438 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:05.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:05.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:05.580 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:05.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:05.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:05.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:05 smithi137 ceph-mon[29408]: 4.1c scrub starts 2024-09-18T07:39:05.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:05 smithi137 ceph-mon[29408]: 4.1c scrub ok 2024-09-18T07:39:05.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:05 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-18T07:39:05.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:05 smithi137 ceph-mon[29408]: osdmap e274: 8 total, 8 up, 8 in 2024-09-18T07:39:05.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:05 smithi137 ceph-mon[29408]: osdmap e275: 8 total, 8 up, 8 in 2024-09-18T07:39:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:05 smithi137 ceph-mon[39175]: 4.1c scrub starts 2024-09-18T07:39:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:05 smithi137 ceph-mon[39175]: 4.1c scrub ok 2024-09-18T07:39:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:05 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-18T07:39:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:05 smithi137 ceph-mon[39175]: osdmap e274: 8 total, 8 up, 8 in 2024-09-18T07:39:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:05 smithi137 ceph-mon[39175]: osdmap e275: 8 total, 8 up, 8 in 2024-09-18T07:39:05.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:05.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:05 smithi203 ceph-mon[32237]: 4.1c scrub starts 2024-09-18T07:39:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:05 smithi203 ceph-mon[32237]: 4.1c scrub ok 2024-09-18T07:39:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:05 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-18T07:39:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:05 smithi203 ceph-mon[32237]: osdmap e274: 8 total, 8 up, 8 in 2024-09-18T07:39:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:05 smithi203 ceph-mon[32237]: osdmap e275: 8 total, 8 up, 8 in 2024-09-18T07:39:05.719 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:05.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:05.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:05.821 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:05.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:05.960 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:05.970 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:05.974 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T07:39:06.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:06.012 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:06.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:06.113 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:06.145 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:06.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:06.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:06.247 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:06.278 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:06.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:06.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:06.379 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:06.411 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:06.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:06.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:06.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:06.524 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[29408]: 1.5 scrub starts 2024-09-18T07:39:06.524 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[29408]: 1.5 scrub ok 2024-09-18T07:39:06.524 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[29408]: 1.3 scrub starts 2024-09-18T07:39:06.524 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[29408]: 1.3 scrub ok 2024-09-18T07:39:06.524 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[29408]: 1.9 scrub starts 2024-09-18T07:39:06.524 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[29408]: 1.9 scrub ok 2024-09-18T07:39:06.524 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[29408]: 1.1 scrub starts 2024-09-18T07:39:06.524 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[29408]: 1.1 scrub ok 2024-09-18T07:39:06.525 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[29408]: pgmap v435: 158 pgs: 1 active+recovering+undersized+remapped, 10 unknown, 147 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.29k op/s; 629/63861 objects misplaced (0.985%); 7.3 MiB/s, 116 objects/s recovering 2024-09-18T07:39:06.525 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[29408]: 1.4 scrub starts 2024-09-18T07:39:06.525 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[29408]: 1.4 scrub ok 2024-09-18T07:39:06.525 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[39175]: 1.5 scrub starts 2024-09-18T07:39:06.525 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[39175]: 1.5 scrub ok 2024-09-18T07:39:06.525 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[39175]: 1.3 scrub starts 2024-09-18T07:39:06.525 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[39175]: 1.3 scrub ok 2024-09-18T07:39:06.525 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[39175]: 1.9 scrub starts 2024-09-18T07:39:06.525 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[39175]: 1.9 scrub ok 2024-09-18T07:39:06.525 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[39175]: 1.1 scrub starts 2024-09-18T07:39:06.525 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[39175]: 1.1 scrub ok 2024-09-18T07:39:06.526 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[39175]: pgmap v435: 158 pgs: 1 active+recovering+undersized+remapped, 10 unknown, 147 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.29k op/s; 629/63861 objects misplaced (0.985%); 7.3 MiB/s, 116 objects/s recovering 2024-09-18T07:39:06.526 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[39175]: 1.4 scrub starts 2024-09-18T07:39:06.526 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:06 smithi137 ceph-mon[39175]: 1.4 scrub ok 2024-09-18T07:39:06.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:06.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:06.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:06.672 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:06.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:06.709 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:06.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:06.709 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:39:06.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-18T07:39:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:06 smithi203 ceph-mon[32237]: 1.5 scrub starts 2024-09-18T07:39:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:06 smithi203 ceph-mon[32237]: 1.5 scrub ok 2024-09-18T07:39:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:06 smithi203 ceph-mon[32237]: 1.3 scrub starts 2024-09-18T07:39:06.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:06 smithi203 ceph-mon[32237]: 1.3 scrub ok 2024-09-18T07:39:06.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:06 smithi203 ceph-mon[32237]: 1.9 scrub starts 2024-09-18T07:39:06.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:06 smithi203 ceph-mon[32237]: 1.9 scrub ok 2024-09-18T07:39:06.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:06 smithi203 ceph-mon[32237]: 1.1 scrub starts 2024-09-18T07:39:06.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:06 smithi203 ceph-mon[32237]: 1.1 scrub ok 2024-09-18T07:39:06.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:06 smithi203 ceph-mon[32237]: pgmap v435: 158 pgs: 1 active+recovering+undersized+remapped, 10 unknown, 147 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.29k op/s; 629/63861 objects misplaced (0.985%); 7.3 MiB/s, 116 objects/s recovering 2024-09-18T07:39:06.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:06 smithi203 ceph-mon[32237]: 1.4 scrub starts 2024-09-18T07:39:06.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:06 smithi203 ceph-mon[32237]: 1.4 scrub ok 2024-09-18T07:39:06.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:06.851 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:06.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:06.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:06.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:06.986 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:06.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:06.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:07.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:07.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:07.131 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:07.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:07.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:39:07.232 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:07.263 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:07.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:07.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:07.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:07.380 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:39:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:39:07.398 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:39:07.402 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:07.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:07.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:07.504 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:07.536 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:07.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:07.638 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:07.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[29408]: 3.17 scrub starts 2024-09-18T07:39:07.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[29408]: 3.17 scrub ok 2024-09-18T07:39:07.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[29408]: 1.7 scrub starts 2024-09-18T07:39:07.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[29408]: 1.7 scrub ok 2024-09-18T07:39:07.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[29408]: 1.2 scrub starts 2024-09-18T07:39:07.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[29408]: 1.2 scrub ok 2024-09-18T07:39:07.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[29408]: 1.a scrub starts 2024-09-18T07:39:07.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[29408]: 1.a scrub ok 2024-09-18T07:39:07.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/332228996' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:39:07.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[29408]: 3.7 scrub starts 2024-09-18T07:39:07.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[29408]: 3.7 scrub ok 2024-09-18T07:39:07.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[29408]: 1.6 scrub starts 2024-09-18T07:39:07.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[29408]: 1.6 scrub ok 2024-09-18T07:39:07.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[39175]: 3.17 scrub starts 2024-09-18T07:39:07.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[39175]: 3.17 scrub ok 2024-09-18T07:39:07.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[39175]: 1.7 scrub starts 2024-09-18T07:39:07.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[39175]: 1.7 scrub ok 2024-09-18T07:39:07.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[39175]: 1.2 scrub starts 2024-09-18T07:39:07.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[39175]: 1.2 scrub ok 2024-09-18T07:39:07.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[39175]: 1.a scrub starts 2024-09-18T07:39:07.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[39175]: 1.a scrub ok 2024-09-18T07:39:07.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/332228996' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:39:07.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[39175]: 3.7 scrub starts 2024-09-18T07:39:07.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[39175]: 3.7 scrub ok 2024-09-18T07:39:07.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[39175]: 1.6 scrub starts 2024-09-18T07:39:07.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:07 smithi137 ceph-mon[39175]: 1.6 scrub ok 2024-09-18T07:39:07.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:07.673 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:07.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:07.710 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:39:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:39:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:07 smithi203 ceph-mon[32237]: 3.17 scrub starts 2024-09-18T07:39:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:07 smithi203 ceph-mon[32237]: 3.17 scrub ok 2024-09-18T07:39:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:07 smithi203 ceph-mon[32237]: 1.7 scrub starts 2024-09-18T07:39:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:07 smithi203 ceph-mon[32237]: 1.7 scrub ok 2024-09-18T07:39:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:07 smithi203 ceph-mon[32237]: 1.2 scrub starts 2024-09-18T07:39:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:07 smithi203 ceph-mon[32237]: 1.2 scrub ok 2024-09-18T07:39:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:07 smithi203 ceph-mon[32237]: 1.a scrub starts 2024-09-18T07:39:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:07 smithi203 ceph-mon[32237]: 1.a scrub ok 2024-09-18T07:39:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:07 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/332228996' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:39:07.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:07 smithi203 ceph-mon[32237]: 3.7 scrub starts 2024-09-18T07:39:07.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:07 smithi203 ceph-mon[32237]: 3.7 scrub ok 2024-09-18T07:39:07.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:07 smithi203 ceph-mon[32237]: 1.6 scrub starts 2024-09-18T07:39:07.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:07 smithi203 ceph-mon[32237]: 1.6 scrub ok 2024-09-18T07:39:07.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:07.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:07.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:07.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:07.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:07.952 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:07.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:07.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:08.054 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:08.087 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:08.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:08.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:08.189 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:08.247 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:08.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:08.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:08.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:08.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:08.381 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:08.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:08.482 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:08.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:08.520 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:08.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:08.621 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:08.654 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:08.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:08.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:08.665 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:39:08.666 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:39:08.666 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[29408]: 4.e scrub starts 2024-09-18T07:39:08.666 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[29408]: 4.e scrub ok 2024-09-18T07:39:08.666 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:08.667 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[29408]: 3.0 scrub starts 2024-09-18T07:39:08.667 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[29408]: 3.0 scrub ok 2024-09-18T07:39:08.667 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:39:08.667 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/332228996' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:39:08.667 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[29408]: osdmap e276: 8 total, 8 up, 8 in 2024-09-18T07:39:08.667 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[29408]: 1.8 scrub starts 2024-09-18T07:39:08.667 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[29408]: pgmap v437: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 927 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.45k op/s; 70/45078 objects misplaced (0.155%); 7.7 MiB/s, 129 objects/s recovering 2024-09-18T07:39:08.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[39175]: 4.e scrub starts 2024-09-18T07:39:08.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[39175]: 4.e scrub ok 2024-09-18T07:39:08.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:08.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[39175]: 3.0 scrub starts 2024-09-18T07:39:08.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[39175]: 3.0 scrub ok 2024-09-18T07:39:08.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:39:08.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/332228996' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:39:08.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[39175]: osdmap e276: 8 total, 8 up, 8 in 2024-09-18T07:39:08.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[39175]: 1.8 scrub starts 2024-09-18T07:39:08.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:08 smithi137 ceph-mon[39175]: pgmap v437: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 927 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.45k op/s; 70/45078 objects misplaced (0.155%); 7.7 MiB/s, 129 objects/s recovering 2024-09-18T07:39:08.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:08 smithi203 ceph-mon[32237]: 4.e scrub starts 2024-09-18T07:39:08.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:08 smithi203 ceph-mon[32237]: 4.e scrub ok 2024-09-18T07:39:08.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:08 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:08.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:08 smithi203 ceph-mon[32237]: 3.0 scrub starts 2024-09-18T07:39:08.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:08 smithi203 ceph-mon[32237]: 3.0 scrub ok 2024-09-18T07:39:08.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:08 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:39:08.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:08 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/332228996' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:39:08.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:08 smithi203 ceph-mon[32237]: osdmap e276: 8 total, 8 up, 8 in 2024-09-18T07:39:08.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:08 smithi203 ceph-mon[32237]: 1.8 scrub starts 2024-09-18T07:39:08.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:08 smithi203 ceph-mon[32237]: pgmap v437: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 927 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.45k op/s; 70/45078 objects misplaced (0.155%); 7.7 MiB/s, 129 objects/s recovering 2024-09-18T07:39:08.757 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:08.793 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:08.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:08.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:08.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:08.944 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:08.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:08.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:09.046 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:09.081 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:09.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:09.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:09.183 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:09.218 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:09.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:09.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:09.320 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:09.357 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:09.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:09.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:09.458 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:09.504 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T07:39:09.513 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:09.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:09.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:09.607 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:09.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:09.654 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:09.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:09.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:09 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:39:09.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:09 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:39:09.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:09 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:39:09.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:09 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:39:09.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:09 smithi137 ceph-mon[29408]: osdmap e277: 8 total, 8 up, 8 in 2024-09-18T07:39:09.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:09 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:39:09.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:09 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:39:09.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:09 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:39:09.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:09 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:39:09.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:09 smithi137 ceph-mon[39175]: osdmap e277: 8 total, 8 up, 8 in 2024-09-18T07:39:09.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:09 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:39:09.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:09 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:39:09.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:09 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:39:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:09 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:39:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:09 smithi203 ceph-mon[32237]: osdmap e277: 8 total, 8 up, 8 in 2024-09-18T07:39:09.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:09.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:09.778 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:09.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:09.879 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:09.900 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:09.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:09.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:10.002 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:10.033 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:10.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:10.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:10.100 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:10.134 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:10.179 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:10.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:10.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:10.281 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:10.312 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:10.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:10.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:10.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:10.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:10.449 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:10.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:10.550 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:10.584 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:10.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:10.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:10.628 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:10 smithi137 ceph-mon[29408]: osdmap e278: 8 total, 8 up, 8 in 2024-09-18T07:39:10.628 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:10 smithi137 ceph-mon[29408]: pgmap v440: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 927 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s; 70/45078 objects misplaced (0.155%); 7.0 MiB/s, 118 objects/s recovering 2024-09-18T07:39:10.628 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:10 smithi137 ceph-mon[29408]: osdmap e279: 8 total, 8 up, 8 in 2024-09-18T07:39:10.629 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:10 smithi137 ceph-mon[39175]: osdmap e278: 8 total, 8 up, 8 in 2024-09-18T07:39:10.629 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:10 smithi137 ceph-mon[39175]: pgmap v440: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 927 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s; 70/45078 objects misplaced (0.155%); 7.0 MiB/s, 118 objects/s recovering 2024-09-18T07:39:10.629 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:10 smithi137 ceph-mon[39175]: osdmap e279: 8 total, 8 up, 8 in 2024-09-18T07:39:10.684 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:10 smithi203 ceph-mon[32237]: osdmap e278: 8 total, 8 up, 8 in 2024-09-18T07:39:10.685 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:10 smithi203 ceph-mon[32237]: pgmap v440: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 927 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.32k op/s; 70/45078 objects misplaced (0.155%); 7.0 MiB/s, 118 objects/s recovering 2024-09-18T07:39:10.685 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:10 smithi203 ceph-mon[32237]: osdmap e279: 8 total, 8 up, 8 in 2024-09-18T07:39:10.686 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:10.713 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:10.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:10.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:10.815 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:10.850 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:10.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:10.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:10.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:10.987 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:10.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:10.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:11.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:11.122 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:11.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:11.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:11.180 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:39:11.181 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:39:11.213 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:39:11.213 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:39:11.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:11.252 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:39:11.252 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:39:11.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:11.268 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:11.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:11.289 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:39:11.289 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:39:11.323 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:39:11.324 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:39:11.359 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:39:11.359 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:39:11.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:11.403 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:39:11.403 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:39:11.407 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:11.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:11.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:11.443 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:39:11.444 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:39:11.446 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:11.510 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:11.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:11.551 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:11.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:11.652 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:11.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:11 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:39:11.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:11 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:39:11.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:11 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/190167063' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:11.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:11 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:39:11.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:11 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:39:11.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:11 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:39:11.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:11 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/190167063' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:11.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:11 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:39:11.693 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:11.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:11.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:11.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:11 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:39:11.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:11 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:39:11.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:11 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/190167063' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:11.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:11 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:39:11.795 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:11.822 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:11.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:11.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:11.924 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:11.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:11.959 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:11.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:12.060 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:12.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:12.099 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:12.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:12.101 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:39:12.200 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:12.230 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:12.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:12.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:12.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:12.375 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:12.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:12.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:12.462 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.0 to 1 (10000) 2024-09-18T07:39:12.477 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:12.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:12.507 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:12.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:12.608 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:12.644 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:12.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:12.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:12.746 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:12.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:12.780 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:12.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:12.881 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:12.917 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:12.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:12.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:12.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:12 smithi137 ceph-mon[29408]: osdmap e280: 8 total, 8 up, 8 in 2024-09-18T07:39:12.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:12 smithi137 ceph-mon[29408]: pgmap v443: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 905 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 90 op/s; 70/44004 objects misplaced (0.159%) 2024-09-18T07:39:12.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:12 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2769595623' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:39:12.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:12 smithi137 ceph-mon[29408]: from='client.25543 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:39:12.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:12 smithi137 ceph-mon[39175]: osdmap e280: 8 total, 8 up, 8 in 2024-09-18T07:39:12.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:12 smithi137 ceph-mon[39175]: pgmap v443: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 905 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 90 op/s; 70/44004 objects misplaced (0.159%) 2024-09-18T07:39:12.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:12 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2769595623' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:39:12.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:12 smithi137 ceph-mon[39175]: from='client.25543 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:39:12.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:12 smithi203 ceph-mon[32237]: osdmap e280: 8 total, 8 up, 8 in 2024-09-18T07:39:12.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:12 smithi203 ceph-mon[32237]: pgmap v443: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 905 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 90 op/s; 70/44004 objects misplaced (0.159%) 2024-09-18T07:39:12.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:12 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2769595623' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:39:12.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:12 smithi203 ceph-mon[32237]: from='client.25543 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:39:13.019 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:13.053 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:13.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:13.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:13.155 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:13.186 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:13.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:13.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:13.288 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:13.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-18T07:39:13.310 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:13.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:13.319 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:13.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:13.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:13.456 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:13.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:13.557 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:13.604 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:13.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:13.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:13.609 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:13 smithi137 ceph-mon[29408]: from='client.25543 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:39:13.609 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:13 smithi137 ceph-mon[29408]: osdmap e281: 8 total, 8 up, 8 in 2024-09-18T07:39:13.610 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:13 smithi137 ceph-mon[39175]: from='client.25543 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:39:13.610 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:13 smithi137 ceph-mon[39175]: osdmap e281: 8 total, 8 up, 8 in 2024-09-18T07:39:13.706 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:13.742 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:13.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:13.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:13.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:13.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:13.877 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:13.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:13.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:13 smithi203 ceph-mon[32237]: from='client.25543 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:39:13.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:13 smithi203 ceph-mon[32237]: osdmap e281: 8 total, 8 up, 8 in 2024-09-18T07:39:13.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:14.020 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:14.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:14.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:14.122 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:14.156 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:14.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:14.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:14.158 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:39:14.158 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:39:14.158 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:39:14.158 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:39:14.158 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:39:14.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:14.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:14.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:14.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:14.400 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:14.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:14.437 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:14.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:14.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:14.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:14.579 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:14.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:14.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:39:14.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:14.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:14.750 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:14.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:14.766 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:14 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:39:14.766 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:14 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:39:14.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:14 smithi137 ceph-mon[29408]: osdmap e282: 8 total, 8 up, 8 in 2024-09-18T07:39:14.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:14 smithi137 ceph-mon[29408]: pgmap v446: 158 pgs: 158 active+clean; 564 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.45k op/s; 1.0 MiB/s, 87 objects/s recovering 2024-09-18T07:39:14.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:14 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:39:14.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:14 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:39:14.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:14 smithi137 ceph-mon[39175]: osdmap e282: 8 total, 8 up, 8 in 2024-09-18T07:39:14.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:14 smithi137 ceph-mon[39175]: pgmap v446: 158 pgs: 158 active+clean; 564 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.45k op/s; 1.0 MiB/s, 87 objects/s recovering 2024-09-18T07:39:14.851 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:14.895 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:14.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:14.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:14.937 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:39:14.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:14 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:39:14.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:14 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:39:14.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:14 smithi203 ceph-mon[32237]: osdmap e282: 8 total, 8 up, 8 in 2024-09-18T07:39:14.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:14 smithi203 ceph-mon[32237]: pgmap v446: 158 pgs: 158 active+clean; 564 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.45k op/s; 1.0 MiB/s, 87 objects/s recovering 2024-09-18T07:39:14.997 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:15.023 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:15.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:15.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:15.125 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:15.173 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:15.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:15.275 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:15.306 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:15.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:15.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:15.453 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:15.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:15.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:15.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:15.591 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:15.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:15.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:15.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T07:39:15.693 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:15.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:15.744 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:15.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:15.805 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:15.839 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:15 smithi137 ceph-mon[29408]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-18T07:39:15.839 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:15 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1198497361' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:15.839 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:15 smithi137 ceph-mon[29408]: from='client.25549 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:15.840 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:15 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:39:15.840 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:15 smithi137 ceph-mon[29408]: from='client.25549 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:39:15.840 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:15 smithi137 ceph-mon[29408]: osdmap e283: 8 total, 8 up, 8 in 2024-09-18T07:39:15.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:15 smithi137 ceph-mon[39175]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-18T07:39:15.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:15 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1198497361' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:15.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:15 smithi137 ceph-mon[39175]: from='client.25549 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:15.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:15 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:39:15.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:15 smithi137 ceph-mon[39175]: from='client.25549 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:39:15.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:15 smithi137 ceph-mon[39175]: osdmap e283: 8 total, 8 up, 8 in 2024-09-18T07:39:15.845 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:15.870 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:15.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:15.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:15.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:15 smithi203 ceph-mon[32237]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-18T07:39:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:15 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1198497361' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:15 smithi203 ceph-mon[32237]: from='client.25549 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:15 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:39:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:15 smithi203 ceph-mon[32237]: from='client.25549 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:39:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:15 smithi203 ceph-mon[32237]: osdmap e283: 8 total, 8 up, 8 in 2024-09-18T07:39:15.972 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:16.014 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:16.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:16.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:16.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:16.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:16.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:16.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:16.259 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:16.290 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:16.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:16.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:16.392 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:16.423 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:16.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:16.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:16.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:16.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:16.558 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:16.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:16.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:16.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:16.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:16.759 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:16 smithi137 ceph-mon[29408]: pgmap v448: 158 pgs: 158 active+clean; 564 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.39k op/s; 1.0 MiB/s, 84 objects/s recovering 2024-09-18T07:39:16.759 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:16 smithi137 ceph-mon[39175]: pgmap v448: 158 pgs: 158 active+clean; 564 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.39k op/s; 1.0 MiB/s, 84 objects/s recovering 2024-09-18T07:39:16.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:16.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:16.857 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:16.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:16.907 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:16.908 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:16.959 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:16.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:16 smithi203 ceph-mon[32237]: pgmap v448: 158 pgs: 158 active+clean; 564 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.39k op/s; 1.0 MiB/s, 84 objects/s recovering 2024-09-18T07:39:17.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:17.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:17.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:17.108 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:17.138 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:17.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:17.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:17.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:17.279 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:17.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:17.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:17.322 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:39:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:39:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:39:17.381 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:17.402 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:17.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:17.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:17.503 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:17.531 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:17.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:17.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:17.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:17.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:17.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:17.710 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:39:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:39:17.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:17.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:17.804 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:17.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:17.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:17.938 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:17.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:17.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:18.040 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:18.061 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:18.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:18.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:18.163 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:18.190 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:18.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:18.292 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:18.321 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:18.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:18.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:18.423 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:18.453 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:18.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:18.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:18.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:18.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:18.577 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:18.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:18.678 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:18.710 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:18.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:18.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:18.761 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:39:18.812 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:18.836 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:18.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:18.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:18.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:18 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:18.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:18 smithi137 ceph-mon[29408]: pgmap v449: 158 pgs: 158 active+clean; 183 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.92k op/s; 711 KiB/s, 58 objects/s recovering 2024-09-18T07:39:18.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:18 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:18.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:18 smithi137 ceph-mon[39175]: pgmap v449: 158 pgs: 158 active+clean; 183 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.92k op/s; 711 KiB/s, 58 objects/s recovering 2024-09-18T07:39:18.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:18.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:18 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:18.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:18 smithi203 ceph-mon[32237]: pgmap v449: 158 pgs: 158 active+clean; 183 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.92k op/s; 711 KiB/s, 58 objects/s recovering 2024-09-18T07:39:18.982 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:18.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:18.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:19.085 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:19.116 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:19.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:19.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:19.223 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:19.246 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:19.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:19.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:19.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:19.382 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:19.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:19.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:19.484 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:19.505 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:19.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:19.566 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Removed 104807 objects 2024-09-18T07:39:19.566 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Clean up completed and total clean up time :103.619 2024-09-18T07:39:19.607 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:19.628 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:19.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:19.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:19.733 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:19.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:19.755 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:19.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:19.857 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:19.878 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:19.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:19.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:19.980 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:19.995 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:39:19.995 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:39:19.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:39:19.995 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:39:19.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:19.996 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135011 7.14 7.3991ba34 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-18T07:39:19.996 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:15.993779+0000", 2024-09-18T07:39:19.996 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 303.52601751399999, 2024-09-18T07:39:19.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98326573299999998, 2024-09-18T07:39:19.996 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:19.996 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:19.996 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:39:19.996 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:19.996 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:39:19.996 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:19.996 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135011 2024-09-18T07:39:19.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:19.997 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:19.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:19.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:19.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993779+0000", 2024-09-18T07:39:19.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:19.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:19.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:19.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:19.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993779+0000", 2024-09-18T07:39:19.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:19.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:19.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:19.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:19.998 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993780+0000", 2024-09-18T07:39:19.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6899999999999996e-07 2024-09-18T07:39:19.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:19.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:19.998 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:19.998 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993784+0000", 2024-09-18T07:39:19.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6270000000000002e-06 2024-09-18T07:39:19.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:19.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:19.998 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:19.998 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993794+0000", 2024-09-18T07:39:19.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6889999999999999e-06 2024-09-18T07:39:19.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:19.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:19.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:19.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993799+0000", 2024-09-18T07:39:19.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1590000000000001e-06 2024-09-18T07:39:19.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:19.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:19.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:19.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993861+0000", 2024-09-18T07:39:19.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2421000000000004e-05 2024-09-18T07:39:19.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:19.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:19.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:19.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993869+0000", 2024-09-18T07:39:19.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1980000000000001e-06 2024-09-18T07:39:20.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.977033+0000", 2024-09-18T07:39:20.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98316372699999999 2024-09-18T07:39:20.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.977045+0000", 2024-09-18T07:39:20.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2043e-05 2024-09-18T07:39:20.000 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.000 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.000 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.001 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135018 7.10 7:0de93d52:::benchmark_data_smithi137_74180_object30602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-18T07:39:20.001 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:15.998577+0000", 2024-09-18T07:39:20.001 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 303.52121974699998, 2024-09-18T07:39:20.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1980523299999999, 2024-09-18T07:39:20.001 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.001 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.001 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:20.001 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.001 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:39:20.001 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:20.001 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135018 2024-09-18T07:39:20.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.002 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998577+0000", 2024-09-18T07:39:20.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998577+0000", 2024-09-18T07:39:20.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.003 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998579+0000", 2024-09-18T07:39:20.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.336e-06 2024-09-18T07:39:20.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.003 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.003 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998583+0000", 2024-09-18T07:39:20.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7190000000000001e-06 2024-09-18T07:39:20.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.003 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.003 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998593+0000", 2024-09-18T07:39:20.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4450000000000002e-06 2024-09-18T07:39:20.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.004 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998598+0000", 2024-09-18T07:39:20.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2329999999999997e-06 2024-09-18T07:39:20.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.004 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998649+0000", 2024-09-18T07:39:20.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0735999999999997e-05 2024-09-18T07:39:20.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.004 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998659+0000", 2024-09-18T07:39:20.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0156000000000001e-05 2024-09-18T07:39:20.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.005 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.005 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.970426+0000", 2024-09-18T07:39:20.005 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97176680299999996 2024-09-18T07:39:20.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.005 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.005 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.972639+0000", 2024-09-18T07:39:20.005 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022131070000000002 2024-09-18T07:39:20.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.005 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.005 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.183802+0000", 2024-09-18T07:39:20.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2111635359999999 2024-09-18T07:39:20.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:39:20.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.183932+0000", 2024-09-18T07:39:20.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000129686 2024-09-18T07:39:20.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2,5", 2024-09-18T07:39:20.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184136+0000", 2024-09-18T07:39:20.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000203885 2024-09-18T07:39:20.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:39:20.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185714+0000", 2024-09-18T07:39:20.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00157839 2024-09-18T07:39:20.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:20.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.188451+0000", 2024-09-18T07:39:20.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002736741 2024-09-18T07:39:20.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:20.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.188856+0000", 2024-09-18T07:39:20.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040506400000000001 2024-09-18T07:39:20.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.008 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:20.008 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.195963+0000", 2024-09-18T07:39:20.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0071068019999999997 2024-09-18T07:39:20.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.008 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:39:20.008 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.196200+0000", 2024-09-18T07:39:20.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023698600000000001 2024-09-18T07:39:20.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.008 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.008 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.196630+0000", 2024-09-18T07:39:20.009 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00042970500000000001 2024-09-18T07:39:20.009 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.009 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.009 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.009 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162368 7.2 7.85717ac2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:39:20.009 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.437764+0000", 2024-09-18T07:39:20.009 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 257.08203316499998, 2024-09-18T07:39:20.009 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97823369100000002, 2024-09-18T07:39:20.009 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.009 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.009 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:39:20.009 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.010 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:39:20.010 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:20.010 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162368 2024-09-18T07:39:20.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.010 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437764+0000", 2024-09-18T07:39:20.010 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437764+0000", 2024-09-18T07:39:20.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.011 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437766+0000", 2024-09-18T07:39:20.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8479999999999999e-06 2024-09-18T07:39:20.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.011 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437771+0000", 2024-09-18T07:39:20.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4580000000000001e-06 2024-09-18T07:39:20.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.011 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.012 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437781+0000", 2024-09-18T07:39:20.012 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0291e-05 2024-09-18T07:39:20.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.012 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.012 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437789+0000", 2024-09-18T07:39:20.012 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3599999999999998e-06 2024-09-18T07:39:20.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.012 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.012 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437856+0000", 2024-09-18T07:39:20.012 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7241000000000002e-05 2024-09-18T07:39:20.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.013 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.013 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437903+0000", 2024-09-18T07:39:20.013 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7348999999999997e-05 2024-09-18T07:39:20.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.013 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.013 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415935+0000", 2024-09-18T07:39:20.013 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97803116499999998 2024-09-18T07:39:20.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.013 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.013 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415997+0000", 2024-09-18T07:39:20.014 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2978999999999995e-05 2024-09-18T07:39:20.014 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.014 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.014 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.014 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162379 7.12 7.b17b62b2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:39:20.014 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.446092+0000", 2024-09-18T07:39:20.014 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 257.07370502399999, 2024-09-18T07:39:20.014 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96914699900000001, 2024-09-18T07:39:20.015 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.015 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.015 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:39:20.015 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.015 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:39:20.015 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:20.015 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162379 2024-09-18T07:39:20.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.015 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446092+0000", 2024-09-18T07:39:20.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446092+0000", 2024-09-18T07:39:20.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446095+0000", 2024-09-18T07:39:20.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7329999999999999e-06 2024-09-18T07:39:20.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.017 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446102+0000", 2024-09-18T07:39:20.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9110000000000004e-06 2024-09-18T07:39:20.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.017 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.017 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446118+0000", 2024-09-18T07:39:20.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6875999999999999e-05 2024-09-18T07:39:20.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.017 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.017 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446126+0000", 2024-09-18T07:39:20.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7030000000000006e-06 2024-09-18T07:39:20.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446193+0000", 2024-09-18T07:39:20.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6919000000000001e-05 2024-09-18T07:39:20.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446203+0000", 2024-09-18T07:39:20.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6439999999999995e-06 2024-09-18T07:39:20.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415214+0000", 2024-09-18T07:39:20.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96901095599999998 2024-09-18T07:39:20.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.019 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.019 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415239+0000", 2024-09-18T07:39:20.019 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5256999999999999e-05 2024-09-18T07:39:20.019 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.019 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.019 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.019 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162393 7.12 7.f00c1e52 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:39:20.019 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.460966+0000", 2024-09-18T07:39:20.019 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 257.05883131399997, 2024-09-18T07:39:20.019 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95474156300000002, 2024-09-18T07:39:20.019 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.020 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.020 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:39:20.020 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.020 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:39:20.020 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:20.020 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162393 2024-09-18T07:39:20.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.020 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.020 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.020 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.020 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.460966+0000", 2024-09-18T07:39:20.020 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.020 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.021 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.021 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.460966+0000", 2024-09-18T07:39:20.021 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.021 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.021 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.460968+0000", 2024-09-18T07:39:20.021 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9870000000000002e-06 2024-09-18T07:39:20.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.021 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.021 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.460973+0000", 2024-09-18T07:39:20.021 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5890000000000002e-06 2024-09-18T07:39:20.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.022 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.022 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.460997+0000", 2024-09-18T07:39:20.022 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3583000000000001e-05 2024-09-18T07:39:20.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.022 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.022 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.461004+0000", 2024-09-18T07:39:20.022 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0840000000000003e-06 2024-09-18T07:39:20.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.461064+0000", 2024-09-18T07:39:20.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9701000000000003e-05 2024-09-18T07:39:20.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.461074+0000", 2024-09-18T07:39:20.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9240000000000002e-06 2024-09-18T07:39:20.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415699+0000", 2024-09-18T07:39:20.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95462499899999997 2024-09-18T07:39:20.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415707+0000", 2024-09-18T07:39:20.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6959999999999994e-06 2024-09-18T07:39:20.024 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.024 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.024 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.024 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162407 7.2 7.5f39ee22 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:39:20.024 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.503104+0000", 2024-09-18T07:39:20.025 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 257.01669327799999, 2024-09-18T07:39:20.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91293045900000003, 2024-09-18T07:39:20.025 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.025 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.025 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:39:20.025 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.025 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:39:20.025 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:20.025 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162407 2024-09-18T07:39:20.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.025 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.026 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.026 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503104+0000", 2024-09-18T07:39:20.026 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.026 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.026 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.026 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503104+0000", 2024-09-18T07:39:20.026 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.026 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.026 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.027 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503105+0000", 2024-09-18T07:39:20.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3200000000000001e-06 2024-09-18T07:39:20.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.027 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.027 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503110+0000", 2024-09-18T07:39:20.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7709999999999997e-06 2024-09-18T07:39:20.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.027 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.027 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503120+0000", 2024-09-18T07:39:20.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0217000000000001e-05 2024-09-18T07:39:20.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.028 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.028 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503127+0000", 2024-09-18T07:39:20.028 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9430000000000003e-06 2024-09-18T07:39:20.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.028 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.028 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503227+0000", 2024-09-18T07:39:20.028 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100147 2024-09-18T07:39:20.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.028 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.028 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503235+0000", 2024-09-18T07:39:20.028 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3270000000000001e-06 2024-09-18T07:39:20.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.029 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.029 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416027+0000", 2024-09-18T07:39:20.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91279167400000005 2024-09-18T07:39:20.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.029 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.029 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416034+0000", 2024-09-18T07:39:20.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0600000000000002e-06 2024-09-18T07:39:20.029 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.029 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.029 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.030 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162429 7.12 7.8e24a052 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:39:20.030 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.531777+0000", 2024-09-18T07:39:20.030 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 256.98802018100002, 2024-09-18T07:39:20.030 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.883964953, 2024-09-18T07:39:20.030 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.030 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.030 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:39:20.030 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.030 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:39:20.030 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:20.030 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162429 2024-09-18T07:39:20.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.031 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531777+0000", 2024-09-18T07:39:20.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531777+0000", 2024-09-18T07:39:20.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531778+0000", 2024-09-18T07:39:20.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7120000000000001e-06 2024-09-18T07:39:20.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531783+0000", 2024-09-18T07:39:20.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.622e-06 2024-09-18T07:39:20.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531792+0000", 2024-09-18T07:39:20.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1889999999999996e-06 2024-09-18T07:39:20.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.033 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531798+0000", 2024-09-18T07:39:20.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9599999999999997e-06 2024-09-18T07:39:20.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.033 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.533462+0000", 2024-09-18T07:39:20.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001663351 2024-09-18T07:39:20.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.033 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.533471+0000", 2024-09-18T07:39:20.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5270000000000004e-06 2024-09-18T07:39:20.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.034 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.034 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415736+0000", 2024-09-18T07:39:20.034 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.88226443600000004 2024-09-18T07:39:20.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.034 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.034 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415742+0000", 2024-09-18T07:39:20.034 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.156e-06 2024-09-18T07:39:20.034 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.034 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.034 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.035 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.035 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162435 7.2 7.90a4f302 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:39:20.035 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.536705+0000", 2024-09-18T07:39:20.035 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 256.98309213300001, 2024-09-18T07:39:20.035 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.87935532000000005, 2024-09-18T07:39:20.035 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.035 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.035 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:39:20.035 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.035 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:39:20.035 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:20.035 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162435 2024-09-18T07:39:20.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.035 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536705+0000", 2024-09-18T07:39:20.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536705+0000", 2024-09-18T07:39:20.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536707+0000", 2024-09-18T07:39:20.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7940000000000001e-06 2024-09-18T07:39:20.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.037 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.037 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536712+0000", 2024-09-18T07:39:20.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9790000000000001e-06 2024-09-18T07:39:20.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.037 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.037 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536725+0000", 2024-09-18T07:39:20.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2986999999999999e-05 2024-09-18T07:39:20.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536732+0000", 2024-09-18T07:39:20.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6340000000000001e-06 2024-09-18T07:39:20.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536789+0000", 2024-09-18T07:39:20.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6675999999999997e-05 2024-09-18T07:39:20.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536798+0000", 2024-09-18T07:39:20.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5100000000000004e-06 2024-09-18T07:39:20.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416054+0000", 2024-09-18T07:39:20.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.87925583100000004 2024-09-18T07:39:20.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416060+0000", 2024-09-18T07:39:20.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9089999999999997e-06 2024-09-18T07:39:20.039 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.039 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.040 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.040 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162461 7.2 7.86b206e2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:39:20.040 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.562650+0000", 2024-09-18T07:39:20.040 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 256.95714699500002, 2024-09-18T07:39:20.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85345243000000004, 2024-09-18T07:39:20.040 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.040 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.040 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:39:20.040 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.040 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:39:20.041 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:20.041 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162461 2024-09-18T07:39:20.041 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.041 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.041 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.041 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.041 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562650+0000", 2024-09-18T07:39:20.041 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.041 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.041 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.041 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.041 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562650+0000", 2024-09-18T07:39:20.041 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.041 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.042 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.042 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562651+0000", 2024-09-18T07:39:20.042 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.246e-06 2024-09-18T07:39:20.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.042 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.042 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562656+0000", 2024-09-18T07:39:20.042 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3189999999999996e-06 2024-09-18T07:39:20.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.042 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.042 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562667+0000", 2024-09-18T07:39:20.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1623e-05 2024-09-18T07:39:20.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.043 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562675+0000", 2024-09-18T07:39:20.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5000000000000002e-06 2024-09-18T07:39:20.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.043 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562871+0000", 2024-09-18T07:39:20.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000196219 2024-09-18T07:39:20.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.044 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.044 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562939+0000", 2024-09-18T07:39:20.044 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8196999999999998e-05 2024-09-18T07:39:20.044 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.044 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.044 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416088+0000", 2024-09-18T07:39:20.044 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85314855700000003 2024-09-18T07:39:20.044 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.044 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.044 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416102+0000", 2024-09-18T07:39:20.044 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4769e-05 2024-09-18T07:39:20.044 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.045 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.045 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.045 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162466 7.12 7.e244e932 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:39:20.045 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.563197+0000", 2024-09-18T07:39:20.045 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 256.95659993999999, 2024-09-18T07:39:20.045 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85257313899999998, 2024-09-18T07:39:20.045 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.045 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.045 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:39:20.045 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.045 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:39:20.046 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:20.046 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162466 2024-09-18T07:39:20.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.046 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.046 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.046 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563197+0000", 2024-09-18T07:39:20.046 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.046 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.046 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563197+0000", 2024-09-18T07:39:20.046 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.047 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.047 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563198+0000", 2024-09-18T07:39:20.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3340000000000001e-06 2024-09-18T07:39:20.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.047 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.047 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563203+0000", 2024-09-18T07:39:20.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7110000000000003e-06 2024-09-18T07:39:20.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.047 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.047 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563217+0000", 2024-09-18T07:39:20.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4314999999999999e-05 2024-09-18T07:39:20.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.048 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.048 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563225+0000", 2024-09-18T07:39:20.048 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1420000000000004e-06 2024-09-18T07:39:20.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.048 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.048 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563294+0000", 2024-09-18T07:39:20.048 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9309999999999999e-05 2024-09-18T07:39:20.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.049 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563303+0000", 2024-09-18T07:39:20.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3659999999999998e-06 2024-09-18T07:39:20.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.049 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415764+0000", 2024-09-18T07:39:20.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85246111000000002 2024-09-18T07:39:20.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.049 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415770+0000", 2024-09-18T07:39:20.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8510000000000004e-06 2024-09-18T07:39:20.049 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.050 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.050 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.050 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162470 7.12 7.9c7c55b2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:39:20.050 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.566190+0000", 2024-09-18T07:39:20.050 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 256.95360695400001, 2024-09-18T07:39:20.050 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.84961990099999996, 2024-09-18T07:39:20.050 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.050 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.050 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:39:20.050 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.050 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:39:20.050 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:20.051 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162470 2024-09-18T07:39:20.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.051 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.051 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.051 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566190+0000", 2024-09-18T07:39:20.051 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.051 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.051 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566190+0000", 2024-09-18T07:39:20.051 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.051 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.052 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566192+0000", 2024-09-18T07:39:20.052 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.762e-06 2024-09-18T07:39:20.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.052 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.052 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566198+0000", 2024-09-18T07:39:20.052 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9499999999999998e-06 2024-09-18T07:39:20.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.052 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.052 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566211+0000", 2024-09-18T07:39:20.052 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3743999999999999e-05 2024-09-18T07:39:20.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566219+0000", 2024-09-18T07:39:20.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7810000000000005e-06 2024-09-18T07:39:20.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566282+0000", 2024-09-18T07:39:20.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2362999999999999e-05 2024-09-18T07:39:20.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566290+0000", 2024-09-18T07:39:20.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7749999999999998e-06 2024-09-18T07:39:20.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415803+0000", 2024-09-18T07:39:20.054 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.84951286599999998 2024-09-18T07:39:20.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415810+0000", 2024-09-18T07:39:20.055 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6599999999999998e-06 2024-09-18T07:39:20.055 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.055 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.055 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.055 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162495 7.12 7.468e4332 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:39:20.055 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.583195+0000", 2024-09-18T07:39:20.055 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 256.93660221699997, 2024-09-18T07:39:20.055 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.83264318599999998, 2024-09-18T07:39:20.055 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.055 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.055 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:39:20.056 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.056 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:39:20.056 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:20.056 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162495 2024-09-18T07:39:20.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.056 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.056 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.056 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.056 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583195+0000", 2024-09-18T07:39:20.056 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.056 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.056 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.056 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583195+0000", 2024-09-18T07:39:20.057 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.057 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.057 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.057 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583196+0000", 2024-09-18T07:39:20.057 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6819999999999999e-06 2024-09-18T07:39:20.057 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.057 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.057 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583201+0000", 2024-09-18T07:39:20.057 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0209999999999999e-06 2024-09-18T07:39:20.057 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.057 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.058 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583211+0000", 2024-09-18T07:39:20.058 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3610000000000007e-06 2024-09-18T07:39:20.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.058 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.058 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.058 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583218+0000", 2024-09-18T07:39:20.058 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8560000000000001e-06 2024-09-18T07:39:20.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.058 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.058 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.058 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583268+0000", 2024-09-18T07:39:20.058 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0294000000000003e-05 2024-09-18T07:39:20.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.058 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.059 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.059 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583277+0000", 2024-09-18T07:39:20.059 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2839999999999995e-06 2024-09-18T07:39:20.059 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.059 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.059 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415831+0000", 2024-09-18T07:39:20.059 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.83255423799999995 2024-09-18T07:39:20.059 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.059 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.059 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415838+0000", 2024-09-18T07:39:20.059 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4500000000000001e-06 2024-09-18T07:39:20.060 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.060 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.060 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.060 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162501 7.12 7.5f5c6f12 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:39:20.060 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.586226+0000", 2024-09-18T07:39:20.060 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 256.93357116200002, 2024-09-18T07:39:20.060 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.82964114600000005, 2024-09-18T07:39:20.060 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.060 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.060 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:39:20.060 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.060 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:39:20.061 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:20.061 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162501 2024-09-18T07:39:20.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.061 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.061 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.061 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586226+0000", 2024-09-18T07:39:20.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.061 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.061 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586226+0000", 2024-09-18T07:39:20.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.062 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.062 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586227+0000", 2024-09-18T07:39:20.062 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.598e-06 2024-09-18T07:39:20.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.062 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.062 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586233+0000", 2024-09-18T07:39:20.062 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5280000000000003e-06 2024-09-18T07:39:20.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.062 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.062 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586246+0000", 2024-09-18T07:39:20.062 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3123e-05 2024-09-18T07:39:20.063 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.063 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.063 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.063 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586254+0000", 2024-09-18T07:39:20.063 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3469999999999999e-06 2024-09-18T07:39:20.063 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.063 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.063 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.063 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586344+0000", 2024-09-18T07:39:20.063 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9617999999999995e-05 2024-09-18T07:39:20.063 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.063 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.063 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.064 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586354+0000", 2024-09-18T07:39:20.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8339999999999994e-06 2024-09-18T07:39:20.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.064 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.064 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415860+0000", 2024-09-18T07:39:20.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.82950663999999996 2024-09-18T07:39:20.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.064 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.064 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415867+0000", 2024-09-18T07:39:20.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4579999999999998e-06 2024-09-18T07:39:20.064 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.064 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.065 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.065 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162524 7.2 7.5cbbb662 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:39:20.065 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.609728+0000", 2024-09-18T07:39:20.065 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 256.91006925300002, 2024-09-18T07:39:20.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.80640156299999999, 2024-09-18T07:39:20.065 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.065 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.065 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:39:20.065 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.065 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:39:20.065 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:20.065 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162524 2024-09-18T07:39:20.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.066 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609728+0000", 2024-09-18T07:39:20.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609728+0000", 2024-09-18T07:39:20.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.067 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609730+0000", 2024-09-18T07:39:20.067 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.894e-06 2024-09-18T07:39:20.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.067 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609741+0000", 2024-09-18T07:39:20.067 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1466e-05 2024-09-18T07:39:20.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.067 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609755+0000", 2024-09-18T07:39:20.067 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3584000000000001e-05 2024-09-18T07:39:20.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609762+0000", 2024-09-18T07:39:20.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7950000000000008e-06 2024-09-18T07:39:20.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.068 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609832+0000", 2024-09-18T07:39:20.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0029000000000003e-05 2024-09-18T07:39:20.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.068 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609844+0000", 2024-09-18T07:39:20.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1154e-05 2024-09-18T07:39:20.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416123+0000", 2024-09-18T07:39:20.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.80627929200000004 2024-09-18T07:39:20.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416129+0000", 2024-09-18T07:39:20.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3489999999999997e-06 2024-09-18T07:39:20.069 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.069 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.069 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.070 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162529 7.12 7.8e054bd2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:39:20.070 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.612942+0000", 2024-09-18T07:39:20.070 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 256.906855243, 2024-09-18T07:39:20.070 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.80297076000000001, 2024-09-18T07:39:20.070 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.070 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.070 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:39:20.070 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.070 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:39:20.070 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:20.070 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162529 2024-09-18T07:39:20.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.070 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612942+0000", 2024-09-18T07:39:20.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612942+0000", 2024-09-18T07:39:20.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612943+0000", 2024-09-18T07:39:20.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.53e-06 2024-09-18T07:39:20.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612948+0000", 2024-09-18T07:39:20.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.006e-06 2024-09-18T07:39:20.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612959+0000", 2024-09-18T07:39:20.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.029e-05 2024-09-18T07:39:20.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612966+0000", 2024-09-18T07:39:20.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9459999999999998e-06 2024-09-18T07:39:20.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.073 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.613042+0000", 2024-09-18T07:39:20.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5358999999999995e-05 2024-09-18T07:39:20.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.073 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.613062+0000", 2024-09-18T07:39:20.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9783999999999999e-05 2024-09-18T07:39:20.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415904+0000", 2024-09-18T07:39:20.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.80284268700000005 2024-09-18T07:39:20.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415912+0000", 2024-09-18T07:39:20.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1580000000000005e-06 2024-09-18T07:39:20.074 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.074 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.074 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.074 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162535 7.12 7.1c115152 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:39:20.075 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.618377+0000", 2024-09-18T07:39:20.075 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 256.90142019500001, 2024-09-18T07:39:20.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.79781339699999998, 2024-09-18T07:39:20.075 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.075 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.075 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:39:20.075 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.075 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:39:20.075 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:20.075 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162535 2024-09-18T07:39:20.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.075 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618377+0000", 2024-09-18T07:39:20.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618377+0000", 2024-09-18T07:39:20.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618379+0000", 2024-09-18T07:39:20.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8509999999999999e-06 2024-09-18T07:39:20.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618384+0000", 2024-09-18T07:39:20.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9989999999999999e-06 2024-09-18T07:39:20.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618394+0000", 2024-09-18T07:39:20.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0045e-05 2024-09-18T07:39:20.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618403+0000", 2024-09-18T07:39:20.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8699999999999998e-06 2024-09-18T07:39:20.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618468+0000", 2024-09-18T07:39:20.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5273999999999999e-05 2024-09-18T07:39:20.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618483+0000", 2024-09-18T07:39:20.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5141e-05 2024-09-18T07:39:20.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416187+0000", 2024-09-18T07:39:20.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.79770383899999997 2024-09-18T07:39:20.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416190+0000", 2024-09-18T07:39:20.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3780000000000001e-06 2024-09-18T07:39:20.079 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.079 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.079 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.079 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109537 8.7 8:e50487d6:::benchmark_data_smithi137_101942_object4727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:39:20.079 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:39:20.079 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 97.463390778000004, 2024-09-18T07:39:20.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.185266763, 2024-09-18T07:39:20.079 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.080 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.080 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:20.080 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.080 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:39:20.080 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:39:20.080 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109537 2024-09-18T07:39:20.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.080 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:39:20.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:39:20.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056408+0000", 2024-09-18T07:39:20.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.562e-06 2024-09-18T07:39:20.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056415+0000", 2024-09-18T07:39:20.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9280000000000004e-06 2024-09-18T07:39:20.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056425+0000", 2024-09-18T07:39:20.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0220999999999999e-05 2024-09-18T07:39:20.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056434+0000", 2024-09-18T07:39:20.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7499999999999992e-06 2024-09-18T07:39:20.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.061190+0000", 2024-09-18T07:39:20.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047560620000000001 2024-09-18T07:39:20.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.061238+0000", 2024-09-18T07:39:20.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8730000000000003e-05 2024-09-18T07:39:20.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229082+0000", 2024-09-18T07:39:20.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.167843878 2024-09-18T07:39:20.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:39:20.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229243+0000", 2024-09-18T07:39:20.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000160426 2024-09-18T07:39:20.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:39:20.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229373+0000", 2024-09-18T07:39:20.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012994999999999999 2024-09-18T07:39:20.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:20.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236041+0000", 2024-09-18T07:39:20.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066680960000000001 2024-09-18T07:39:20.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:39:20.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236051+0000", 2024-09-18T07:39:20.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8309999999999996e-06 2024-09-18T07:39:20.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:20.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.240107+0000", 2024-09-18T07:39:20.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040564570000000003 2024-09-18T07:39:20.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:39:20.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.240152+0000", 2024-09-18T07:39:20.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4617000000000003e-05 2024-09-18T07:39:20.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.241673+0000", 2024-09-18T07:39:20.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001521255 2024-09-18T07:39:20.086 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.086 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.086 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.086 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109540 8.7 8:e7454470:::benchmark_data_smithi137_101942_object4730:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:39:20.086 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:39:20.086 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 97.453918787999996, 2024-09-18T07:39:20.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.186834677, 2024-09-18T07:39:20.087 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.087 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.087 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:20.087 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.087 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:39:20.087 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:39:20.087 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109540 2024-09-18T07:39:20.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.087 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:39:20.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:39:20.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065880+0000", 2024-09-18T07:39:20.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8279999999999999e-06 2024-09-18T07:39:20.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065886+0000", 2024-09-18T07:39:20.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.117e-06 2024-09-18T07:39:20.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065899+0000", 2024-09-18T07:39:20.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2809999999999999e-05 2024-09-18T07:39:20.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065904+0000", 2024-09-18T07:39:20.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0350000000000001e-06 2024-09-18T07:39:20.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.066028+0000", 2024-09-18T07:39:20.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012424199999999999 2024-09-18T07:39:20.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.066039+0000", 2024-09-18T07:39:20.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0441999999999999e-05 2024-09-18T07:39:20.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.233503+0000", 2024-09-18T07:39:20.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.167464622 2024-09-18T07:39:20.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:39:20.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.233624+0000", 2024-09-18T07:39:20.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012107 2024-09-18T07:39:20.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:39:20.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.233749+0000", 2024-09-18T07:39:20.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000124418 2024-09-18T07:39:20.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:20.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236078+0000", 2024-09-18T07:39:20.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002328746 2024-09-18T07:39:20.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:39:20.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.239968+0000", 2024-09-18T07:39:20.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038901220000000002 2024-09-18T07:39:20.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:20.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.250395+0000", 2024-09-18T07:39:20.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010427532999999999 2024-09-18T07:39:20.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:39:20.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.250537+0000", 2024-09-18T07:39:20.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014218100000000001 2024-09-18T07:39:20.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.252713+0000", 2024-09-18T07:39:20.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0021755110000000002 2024-09-18T07:39:20.093 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.093 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.093 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.093 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109545 8.7 8:ecb41cf4:::benchmark_data_smithi137_101942_object4735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:39:20.093 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:39:20.093 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 97.445514001999996, 2024-09-18T07:39:20.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1804624180000001, 2024-09-18T07:39:20.094 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.094 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.094 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:20.094 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.094 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:39:20.094 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:39:20.094 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109545 2024-09-18T07:39:20.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.094 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:39:20.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:39:20.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074285+0000", 2024-09-18T07:39:20.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1169999999999998e-06 2024-09-18T07:39:20.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074292+0000", 2024-09-18T07:39:20.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.153e-06 2024-09-18T07:39:20.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074305+0000", 2024-09-18T07:39:20.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2363e-05 2024-09-18T07:39:20.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074312+0000", 2024-09-18T07:39:20.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3409999999999997e-06 2024-09-18T07:39:20.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.097 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074508+0000", 2024-09-18T07:39:20.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000196443 2024-09-18T07:39:20.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.097 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.097 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074519+0000", 2024-09-18T07:39:20.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0159999999999999e-05 2024-09-18T07:39:20.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.097 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.097 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236329+0000", 2024-09-18T07:39:20.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1618104599999999 2024-09-18T07:39:20.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:39:20.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236416+0000", 2024-09-18T07:39:20.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6670999999999997e-05 2024-09-18T07:39:20.098 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:39:20.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236495+0000", 2024-09-18T07:39:20.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9256000000000001e-05 2024-09-18T07:39:20.098 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:39:20.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.241684+0000", 2024-09-18T07:39:20.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051890720000000003 2024-09-18T07:39:20.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.099 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:20.099 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.243812+0000", 2024-09-18T07:39:20.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002127912 2024-09-18T07:39:20.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.099 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:20.099 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.252772+0000", 2024-09-18T07:39:20.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.008960562 2024-09-18T07:39:20.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.099 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:39:20.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.252827+0000", 2024-09-18T07:39:20.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4500000000000003e-05 2024-09-18T07:39:20.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.254745+0000", 2024-09-18T07:39:20.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019184079999999999 2024-09-18T07:39:20.100 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.100 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.100 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.100 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109567 8.7 8:e247f5cd:::benchmark_data_smithi137_101942_object4757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:39:20.100 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:39:20.101 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 97.416414838999998, 2024-09-18T07:39:20.101 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.153246698, 2024-09-18T07:39:20.101 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:39:20.101 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:39:20.101 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:20.101 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:39:20.101 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:39:20.101 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:39:20.101 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109567 2024-09-18T07:39:20.101 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.101 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:39:20.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.101 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:39:20.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:39:20.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:39:20.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:39:20.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:39:20.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:39:20.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103384+0000", 2024-09-18T07:39:20.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.792e-06 2024-09-18T07:39:20.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:39:20.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103390+0000", 2024-09-18T07:39:20.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7139999999999998e-06 2024-09-18T07:39:20.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.103 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:39:20.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103401+0000", 2024-09-18T07:39:20.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1559000000000001e-05 2024-09-18T07:39:20.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.103 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:39:20.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103408+0000", 2024-09-18T07:39:20.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2110000000000001e-06 2024-09-18T07:39:20.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.104 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103464+0000", 2024-09-18T07:39:20.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6032999999999997e-05 2024-09-18T07:39:20.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:39:20.104 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103475+0000", 2024-09-18T07:39:20.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0380999999999999e-05 2024-09-18T07:39:20.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:39:20.104 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.238189+0000", 2024-09-18T07:39:20.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.134713723 2024-09-18T07:39:20.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.105 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:39:20.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.238319+0000", 2024-09-18T07:39:20.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013080700000000001 2024-09-18T07:39:20.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.105 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:39:20.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.238431+0000", 2024-09-18T07:39:20.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112126 2024-09-18T07:39:20.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.105 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:39:20.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.243830+0000", 2024-09-18T07:39:20.106 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053980390000000003 2024-09-18T07:39:20.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.106 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:20.106 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.243889+0000", 2024-09-18T07:39:20.106 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9024000000000003e-05 2024-09-18T07:39:20.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.106 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:20.106 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.254832+0000", 2024-09-18T07:39:20.106 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0109436 2024-09-18T07:39:20.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.106 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:39:20.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.255025+0000", 2024-09-18T07:39:20.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000192926 2024-09-18T07:39:20.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:39:20.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:39:20.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:39:20.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.256629+0000", 2024-09-18T07:39:20.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001603763 2024-09-18T07:39:20.107 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.107 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.107 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.107 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:39:20.107 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:39:20.107 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:39:20.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:20.109 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:20.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:20.210 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:20.231 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:20.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:20.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:20.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:20.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:20.356 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:20.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:20.457 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:20.478 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:20.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:20.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:20.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:20.602 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:20.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:20.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:20.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:20.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:20.728 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:20.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:20.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:20.852 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:20.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:20.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:20 smithi137 ceph-mon[29408]: pgmap v450: 158 pgs: 158 active+clean; 183 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 716 B/s rd, 0 B/s wr, 852 op/s 2024-09-18T07:39:20.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:20 smithi137 ceph-mon[39175]: pgmap v450: 158 pgs: 158 active+clean; 183 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 716 B/s rd, 0 B/s wr, 852 op/s 2024-09-18T07:39:20.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:20.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:20 smithi203 ceph-mon[32237]: pgmap v450: 158 pgs: 158 active+clean; 183 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 716 B/s rd, 0 B/s wr, 852 op/s 2024-09-18T07:39:20.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:20.974 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:20.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:21.075 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:21.096 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:21.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:21.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:21.102 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-18T07:39:21.102 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-18T07:39:21.198 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:21.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:21.221 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:21.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:21.316 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T07:39:21.322 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:21.382 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:21.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:21.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:21.484 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:21.507 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:21.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:21.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:21.567 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:39:21.609 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:21.630 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:21.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:21.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:21.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:21.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:21.754 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:21.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:21.855 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:21.865 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:39:21.865 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:39:21.870 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:39:21.870 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:39:21.877 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:21.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:21.886 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:39:21.886 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:39:21.901 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:39:21.902 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:39:21.918 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:39:21.918 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:39:21.934 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:39:21.935 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:39:21.951 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:39:21.951 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:39:21.969 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:39:21.969 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:39:21.979 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:22.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:22.002 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:22.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:22.078 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:39:22.078 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:39:22.103 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:22.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:22.125 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:22.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:22.226 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:22.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:22.247 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:22.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:22.282 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:22.306 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:22.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:22.369 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:22.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:22.471 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:22.495 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:22.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:22.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:22.596 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:22.622 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:22.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:22.663 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.1 to 1 (10000) 2024-09-18T07:39:22.722 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:22.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:22.745 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:22.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:22.846 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:22.945 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:22 smithi137 ceph-mon[39175]: pgmap v451: 158 pgs: 158 active+clean; 171 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 775 op/s 2024-09-18T07:39:22.945 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:22 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/489588883' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:22.945 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:22 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2611555617' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:39:22.945 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:22 smithi137 ceph-mon[39175]: from='client.35492 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:39:22.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:22 smithi137 ceph-mon[29408]: pgmap v451: 158 pgs: 158 active+clean; 171 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 775 op/s 2024-09-18T07:39:22.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:22 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/489588883' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:22.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:22 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2611555617' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:39:22.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:22 smithi137 ceph-mon[29408]: from='client.35492 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:39:22.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:22 smithi203 ceph-mon[32237]: pgmap v451: 158 pgs: 158 active+clean; 171 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 775 op/s 2024-09-18T07:39:22.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:22 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/489588883' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:22.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:22 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2611555617' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:39:22.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:22 smithi203 ceph-mon[32237]: from='client.35492 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:39:22.967 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:22.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:22.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:23.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:23.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:23.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:23.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:23.178 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T07:39:23.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:23.212 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:23.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:23.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:23.314 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:23.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:23.337 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:23.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:23.429 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:39:23.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:23.460 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:23.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:23.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:23.562 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:23.583 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:23.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:23.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:23.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:23.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:23.708 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:23.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:23.809 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:23.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:23.833 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:23.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:23.888 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:23 smithi203 ceph-mon[32237]: from='client.35492 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:39:23.889 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:23 smithi203 ceph-mon[32237]: osdmap e284: 8 total, 8 up, 8 in 2024-09-18T07:39:23.889 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:23 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:39:23.889 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:23 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:39:23.895 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:39:23.895 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:39:23.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:23 smithi137 ceph-mon[39175]: from='client.35492 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:39:23.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:23 smithi137 ceph-mon[39175]: osdmap e284: 8 total, 8 up, 8 in 2024-09-18T07:39:23.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:23 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:39:23.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:23 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:39:23.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:23 smithi137 ceph-mon[29408]: from='client.35492 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:39:23.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:23 smithi137 ceph-mon[29408]: osdmap e284: 8 total, 8 up, 8 in 2024-09-18T07:39:23.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:23 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:39:23.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:23 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:39:23.934 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:23.956 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:23.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:23.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:24.058 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:24.079 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:24.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:24.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:24.181 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:24.202 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:24.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:24.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:24.304 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:24.325 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:24.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:24.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-18T07:39:24.427 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:24.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:24.450 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:24.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:24.551 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:24.572 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:24.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:24.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:24.607 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:24.674 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:24.675 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'unique_pool_1' removed 2024-09-18T07:39:24.679 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:39:24.700 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:24.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:24.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:24.706 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T07:39:24.803 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:24.835 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:24.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:24.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:24 smithi137 ceph-mon[29408]: pgmap v453: 158 pgs: 158 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T07:39:24.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:24 smithi137 ceph-mon[29408]: osdmap e285: 8 total, 8 up, 8 in 2024-09-18T07:39:24.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:24 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/573224519' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:39:24.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:24 smithi137 ceph-mon[29408]: from='client.35498 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:39:24.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:24 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3651150137' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:39:24.930 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:24 smithi137 ceph-mon[39175]: pgmap v453: 158 pgs: 158 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T07:39:24.930 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:24 smithi137 ceph-mon[39175]: osdmap e285: 8 total, 8 up, 8 in 2024-09-18T07:39:24.930 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:24 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/573224519' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:39:24.930 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:24 smithi137 ceph-mon[39175]: from='client.35498 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:39:24.930 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:24 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3651150137' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:39:24.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:24.960 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:24.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:24.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:24.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:24 smithi203 ceph-mon[32237]: pgmap v453: 158 pgs: 158 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T07:39:24.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:24 smithi203 ceph-mon[32237]: osdmap e285: 8 total, 8 up, 8 in 2024-09-18T07:39:24.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:24 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/573224519' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:39:24.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:24 smithi203 ceph-mon[32237]: from='client.35498 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:39:24.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:24 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3651150137' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:39:25.018 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:39:25.030 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:39:25.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:25.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:25.083 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:25.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:25.184 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:25.206 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:25.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:25.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:25.307 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:25.329 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:25.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:25.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:25.431 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:25.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:25.453 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:25.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:25.495 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:39:25.495 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:39:25.554 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:25.578 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:25.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:25.629 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-18T07:39:25.630 INFO:tasks.radosbench:Beginning radosbench... 2024-09-18T07:39:25.631 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-18T07:39:25.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-09-18T07:39:25.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:25.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:25.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:25.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:25.800 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:25 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:39:25.801 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:25 smithi203 ceph-mon[32237]: from='client.35498 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:39:25.801 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:25 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3651150137' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T07:39:25.801 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:25 smithi203 ceph-mon[32237]: osdmap e286: 8 total, 8 up, 8 in 2024-09-18T07:39:25.801 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:25 smithi203 ceph-mon[32237]: osdmap e287: 8 total, 8 up, 8 in 2024-09-18T07:39:25.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:25.823 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:25.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:25.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:25.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:25 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:39:25.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:25 smithi137 ceph-mon[29408]: from='client.35498 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:39:25.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:25 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3651150137' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T07:39:25.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:25 smithi137 ceph-mon[29408]: osdmap e286: 8 total, 8 up, 8 in 2024-09-18T07:39:25.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:25 smithi137 ceph-mon[29408]: osdmap e287: 8 total, 8 up, 8 in 2024-09-18T07:39:25.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:25 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:39:25.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:25 smithi137 ceph-mon[39175]: from='client.35498 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:39:25.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:25 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3651150137' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T07:39:25.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:25 smithi137 ceph-mon[39175]: osdmap e286: 8 total, 8 up, 8 in 2024-09-18T07:39:25.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:25 smithi137 ceph-mon[39175]: osdmap e287: 8 total, 8 up, 8 in 2024-09-18T07:39:25.925 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:25.946 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:25.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:25.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:26.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:26.069 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:26.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:26.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:26.120 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T07:39:26.171 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:26.192 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:26.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:26.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:26.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:26.316 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:26.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:26.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:26.369 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:39:26.418 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:26.439 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:26.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:26.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:26.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:26.567 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:26.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:26.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:26.569 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:26.569 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:26.669 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:26.690 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:26.691 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[29408]: 3.d scrub starts 2024-09-18T07:39:26.691 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[29408]: 3.d scrub ok 2024-09-18T07:39:26.691 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[29408]: 4.15 scrub starts 2024-09-18T07:39:26.691 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[29408]: 4.15 scrub ok 2024-09-18T07:39:26.691 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[29408]: pgmap v457: 142 pgs: 142 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-18T07:39:26.691 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[29408]: 5.5 scrub starts 2024-09-18T07:39:26.691 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[29408]: 5.5 scrub ok 2024-09-18T07:39:26.691 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[29408]: 4.a scrub starts 2024-09-18T07:39:26.691 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[29408]: 4.a scrub ok 2024-09-18T07:39:26.692 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:39:26.692 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:39:26.692 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[39175]: 3.d scrub starts 2024-09-18T07:39:26.692 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[39175]: 3.d scrub ok 2024-09-18T07:39:26.692 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[39175]: 4.15 scrub starts 2024-09-18T07:39:26.692 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[39175]: 4.15 scrub ok 2024-09-18T07:39:26.692 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[39175]: pgmap v457: 142 pgs: 142 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-18T07:39:26.692 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[39175]: 5.5 scrub starts 2024-09-18T07:39:26.693 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[39175]: 5.5 scrub ok 2024-09-18T07:39:26.693 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[39175]: 4.a scrub starts 2024-09-18T07:39:26.693 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[39175]: 4.a scrub ok 2024-09-18T07:39:26.693 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:39:26.693 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:39:26.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:26.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:26.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:26 smithi203 ceph-mon[32237]: 3.d scrub starts 2024-09-18T07:39:26.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:26 smithi203 ceph-mon[32237]: 3.d scrub ok 2024-09-18T07:39:26.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:26 smithi203 ceph-mon[32237]: 4.15 scrub starts 2024-09-18T07:39:26.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:26 smithi203 ceph-mon[32237]: 4.15 scrub ok 2024-09-18T07:39:26.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:26 smithi203 ceph-mon[32237]: pgmap v457: 142 pgs: 142 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-18T07:39:26.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:26 smithi203 ceph-mon[32237]: 5.5 scrub starts 2024-09-18T07:39:26.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:26 smithi203 ceph-mon[32237]: 5.5 scrub ok 2024-09-18T07:39:26.781 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:26 smithi203 ceph-mon[32237]: 4.a scrub starts 2024-09-18T07:39:26.782 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:26 smithi203 ceph-mon[32237]: 4.a scrub ok 2024-09-18T07:39:26.782 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:39:26.782 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:39:26.797 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:26.818 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:26.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:26.882 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:39:26.882 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:39:26.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:26.942 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:26.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:26.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:27.044 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:27.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:27.066 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:27.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:27.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:27.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:27.192 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:27.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:27.287 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:39:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:39:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:39:27.288 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:39:27.288 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:39:27.293 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:27.303 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:39:27.303 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:39:27.316 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:27.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:27.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:27.319 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:39:27.319 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:39:27.335 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:39:27.336 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:39:27.353 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:39:27.353 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:39:27.368 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:39:27.369 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:39:27.385 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:39:27.385 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:39:27.401 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:39:27.401 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:39:27.418 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:27.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:27.440 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:27.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:27.538 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T07:39:27.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:27.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:27.561 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:27.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:27.662 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:27.672 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.2 to 1 (10000) 2024-09-18T07:39:27.682 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:39:27 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:39:27.686 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:27.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:27.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:27.788 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:27.789 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:39:27.810 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:27.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:27.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:27.912 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:27.933 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:27.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:27.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:27.937 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:27 smithi203 ceph-mon[32237]: 4.1b scrub starts 2024-09-18T07:39:27.937 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:27 smithi203 ceph-mon[32237]: 4.1b scrub ok 2024-09-18T07:39:27.937 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:27 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3617617803' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:39:27.937 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:27 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1227305519' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:28.036 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:28.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:28.059 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:28.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:28.081 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:39:28.102 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:27 smithi137 ceph-mon[39175]: 4.1b scrub starts 2024-09-18T07:39:28.102 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:27 smithi137 ceph-mon[39175]: 4.1b scrub ok 2024-09-18T07:39:28.102 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:27 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3617617803' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:39:28.103 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:27 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1227305519' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:28.103 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:27 smithi137 ceph-mon[29408]: 4.1b scrub starts 2024-09-18T07:39:28.103 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:27 smithi137 ceph-mon[29408]: 4.1b scrub ok 2024-09-18T07:39:28.103 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:27 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3617617803' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:39:28.103 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:27 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1227305519' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:28.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:28.180 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:28.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:28.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:28.282 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:28.302 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:28.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:28.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:28.404 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:28.426 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:28.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:28.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:28.528 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:28.549 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:28.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:28.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:28.607 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:39:28.607 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:39:28.607 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:39:28.607 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:39:28.608 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:39:28.651 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:28.675 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:28.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:28.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:28.691 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'unique_pool_2' created 2024-09-18T07:39:28.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:28.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:28.798 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:28.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:28.899 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:28.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:28 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:28.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:28 smithi137 ceph-mon[29408]: 4.8 scrub starts 2024-09-18T07:39:28.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:28 smithi137 ceph-mon[29408]: 4.8 scrub ok 2024-09-18T07:39:28.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:28 smithi137 ceph-mon[29408]: pgmap v458: 142 pgs: 142 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 829 B/s rd, 0 op/s 2024-09-18T07:39:28.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:28 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3617617803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:39:28.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:28 smithi137 ceph-mon[29408]: osdmap e288: 8 total, 8 up, 8 in 2024-09-18T07:39:28.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:28 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/447784515' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-18T07:39:28.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:28 smithi137 ceph-mon[29408]: from='client.35522 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-18T07:39:28.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:28 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:28.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:28 smithi137 ceph-mon[39175]: 4.8 scrub starts 2024-09-18T07:39:28.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:28 smithi137 ceph-mon[39175]: 4.8 scrub ok 2024-09-18T07:39:28.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:28 smithi137 ceph-mon[39175]: pgmap v458: 142 pgs: 142 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 829 B/s rd, 0 op/s 2024-09-18T07:39:28.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:28 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3617617803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:39:28.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:28 smithi137 ceph-mon[39175]: osdmap e288: 8 total, 8 up, 8 in 2024-09-18T07:39:28.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:28 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/447784515' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-18T07:39:28.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:28 smithi137 ceph-mon[39175]: from='client.35522 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-18T07:39:28.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:28 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:28.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:28 smithi203 ceph-mon[32237]: 4.8 scrub starts 2024-09-18T07:39:28.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:28 smithi203 ceph-mon[32237]: 4.8 scrub ok 2024-09-18T07:39:28.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:28 smithi203 ceph-mon[32237]: pgmap v458: 142 pgs: 142 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 829 B/s rd, 0 op/s 2024-09-18T07:39:28.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:28 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3617617803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:39:28.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:28 smithi203 ceph-mon[32237]: osdmap e288: 8 total, 8 up, 8 in 2024-09-18T07:39:28.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:28 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/447784515' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-18T07:39:28.921 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:28 smithi203 ceph-mon[32237]: from='client.35522 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-18T07:39:28.923 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:28.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:28.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:28.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-18T07:39:29.024 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:29.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:29.066 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:29.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:29.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:29.188 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:29.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:29.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:29.234 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T07:39:29.290 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:29.311 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:29.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:29.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:29.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:29.435 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:29.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:29.485 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:39:29.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:29.561 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:29.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:29.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:29.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:29.684 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:29.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:29.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:29.765 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:29 smithi203 ceph-mon[32237]: from='client.35522 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-18T07:39:29.765 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:29 smithi203 ceph-mon[32237]: osdmap e289: 8 total, 8 up, 8 in 2024-09-18T07:39:29.765 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:29 smithi203 ceph-mon[32237]: 3.6 scrub starts 2024-09-18T07:39:29.766 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:29 smithi203 ceph-mon[32237]: 3.6 scrub ok 2024-09-18T07:39:29.766 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:29 smithi203 ceph-mon[32237]: 3.8 scrub starts 2024-09-18T07:39:29.766 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:29 smithi203 ceph-mon[32237]: 3.8 scrub ok 2024-09-18T07:39:29.786 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:29.809 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:29.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:29.866 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-18T07:39:29.910 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:29.934 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:29.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:29.954 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:29.955 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:29.995 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:39:29.995 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:39:29.995 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:39:29.995 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:39:29.996 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:29.996 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22346 7.0 7:0d479b9b:::benchmark_data_smithi137_74180_object22345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:39:29.996 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.661502+0000", 2024-09-18T07:39:29.996 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 425.32925633799999, 2024-09-18T07:39:29.996 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19330275799999999, 2024-09-18T07:39:29.996 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:29.996 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:29.996 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:29.997 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:29.997 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:39:29.997 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:29.997 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22346 2024-09-18T07:39:29.997 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:29.997 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:29.997 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:29.997 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:29.998 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661502+0000", 2024-09-18T07:39:29.998 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:29.998 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:29.998 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:29.998 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:29.998 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661502+0000", 2024-09-18T07:39:29.998 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:29.998 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:29.999 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:29.999 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:29.999 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661503+0000", 2024-09-18T07:39:29.999 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7400000000000002e-07 2024-09-18T07:39:29.999 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:29.999 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:29.999 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:29.999 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661554+0000", 2024-09-18T07:39:30.000 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1014000000000003e-05 2024-09-18T07:39:30.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.000 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.000 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.000 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661561+0000", 2024-09-18T07:39:30.000 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3370000000000002e-06 2024-09-18T07:39:30.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.000 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.001 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.001 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661567+0000", 2024-09-18T07:39:30.001 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2580000000000003e-06 2024-09-18T07:39:30.001 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.001 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.001 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.001 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661598+0000", 2024-09-18T07:39:30.002 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1810999999999997e-05 2024-09-18T07:39:30.002 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.002 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.002 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.002 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852170+0000", 2024-09-18T07:39:30.002 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.190571617 2024-09-18T07:39:30.002 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.002 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:39:30.003 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852341+0000", 2024-09-18T07:39:30.003 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017095599999999999 2024-09-18T07:39:30.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.003 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:30.003 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.854512+0000", 2024-09-18T07:39:30.003 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021714299999999998 2024-09-18T07:39:30.004 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.004 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.004 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.004 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.854589+0000", 2024-09-18T07:39:30.004 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6168000000000007e-05 2024-09-18T07:39:30.004 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.004 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.005 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.005 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.854709+0000", 2024-09-18T07:39:30.005 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000120951 2024-09-18T07:39:30.005 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.005 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.005 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.005 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.854765+0000", 2024-09-18T07:39:30.005 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5761999999999999e-05 2024-09-18T07:39:30.005 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.006 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.006 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.006 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.854805+0000", 2024-09-18T07:39:30.006 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9579999999999997e-05 2024-09-18T07:39:30.006 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.006 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.006 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.006 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.007 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.007 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22349 7.0 7:0d8b0d01:::benchmark_data_smithi137_74180_object22348:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:39:30.007 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.662222+0000", 2024-09-18T07:39:30.007 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 425.32853623400001, 2024-09-18T07:39:30.007 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19509826899999999, 2024-09-18T07:39:30.007 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.008 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.008 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:30.008 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:30.008 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:39:30.008 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:30.008 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22349 2024-09-18T07:39:30.008 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.008 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.009 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.009 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.009 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662222+0000", 2024-09-18T07:39:30.009 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.009 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.009 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.009 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.009 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662222+0000", 2024-09-18T07:39:30.009 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.010 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.010 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.010 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662223+0000", 2024-09-18T07:39:30.010 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9399999999999993e-07 2024-09-18T07:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662284+0000", 2024-09-18T07:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0887000000000002e-05 2024-09-18T07:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.011 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852186+0000", 2024-09-18T07:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18990221400000001 2024-09-18T07:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852200+0000", 2024-09-18T07:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3703000000000001e-05 2024-09-18T07:39:30.012 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.013 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.013 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.013 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.853205+0000", 2024-09-18T07:39:30.013 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010050949999999999 2024-09-18T07:39:30.013 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.013 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.013 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.013 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.853308+0000", 2024-09-18T07:39:30.014 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010267700000000001 2024-09-18T07:39:30.014 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.014 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.014 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:39:30.014 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.853390+0000", 2024-09-18T07:39:30.014 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2711000000000001e-05 2024-09-18T07:39:30.014 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.014 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.014 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:30.014 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.854896+0000", 2024-09-18T07:39:30.014 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00150526 2024-09-18T07:39:30.014 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.855065+0000", 2024-09-18T07:39:30.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000169457 2024-09-18T07:39:30.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.857250+0000", 2024-09-18T07:39:30.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002184746 2024-09-18T07:39:30.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.857282+0000", 2024-09-18T07:39:30.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2005000000000003e-05 2024-09-18T07:39:30.016 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.016 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.016 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.016 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.857320+0000", 2024-09-18T07:39:30.016 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8519999999999997e-05 2024-09-18T07:39:30.016 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.016 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.016 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.016 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.016 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.016 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:74666 7.0 7:058d7705:::benchmark_data_smithi137_74180_object74665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:39:30.016 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:16.166293+0000", 2024-09-18T07:39:30.016 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 373.82446518400002, 2024-09-18T07:39:30.016 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15084240900000001, 2024-09-18T07:39:30.017 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.017 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.017 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:30.017 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:30.017 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:39:30.017 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:30.017 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 74666 2024-09-18T07:39:30.017 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.017 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.017 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.017 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.017 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166293+0000", 2024-09-18T07:39:30.017 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.018 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.018 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.018 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.018 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166293+0000", 2024-09-18T07:39:30.018 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.018 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.018 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.018 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.018 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166295+0000", 2024-09-18T07:39:30.019 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.53e-06 2024-09-18T07:39:30.019 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.019 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.019 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.019 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166353+0000", 2024-09-18T07:39:30.019 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8069000000000003e-05 2024-09-18T07:39:30.019 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.019 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.019 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.019 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166363+0000", 2024-09-18T07:39:30.019 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0195e-05 2024-09-18T07:39:30.019 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.020 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.020 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.020 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166370+0000", 2024-09-18T07:39:30.020 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4329999999999998e-06 2024-09-18T07:39:30.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.020 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.020 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.020 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166412+0000", 2024-09-18T07:39:30.020 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1375999999999998e-05 2024-09-18T07:39:30.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.020 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.020 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.020 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166498+0000", 2024-09-18T07:39:30.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6080000000000003e-05 2024-09-18T07:39:30.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.021 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:39:30.021 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.166594+0000", 2024-09-18T07:39:30.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6346000000000002e-05 2024-09-18T07:39:30.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.021 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:30.021 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.168253+0000", 2024-09-18T07:39:30.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001658423 2024-09-18T07:39:30.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.175204+0000", 2024-09-18T07:39:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069512180000000003 2024-09-18T07:39:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.317012+0000", 2024-09-18T07:39:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.141807866 2024-09-18T07:39:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.317080+0000", 2024-09-18T07:39:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7762000000000006e-05 2024-09-18T07:39:30.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.023 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.023 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.317136+0000", 2024-09-18T07:39:30.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6110999999999999e-05 2024-09-18T07:39:30.023 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.023 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.023 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.023 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.023 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:74672 7.0 7:0c3f6e43:::benchmark_data_smithi137_74180_object74671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:39:30.023 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:16.172698+0000", 2024-09-18T07:39:30.023 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 373.81806012099997, 2024-09-18T07:39:30.024 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.144571955, 2024-09-18T07:39:30.024 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.024 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.024 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:30.024 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:30.024 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:39:30.024 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:30.024 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 74672 2024-09-18T07:39:30.024 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.024 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.024 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.172698+0000", 2024-09-18T07:39:30.024 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.025 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.025 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.025 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.025 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.172698+0000", 2024-09-18T07:39:30.025 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.026 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.026 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.026 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.026 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.172700+0000", 2024-09-18T07:39:30.026 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8589999999999999e-06 2024-09-18T07:39:30.026 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.026 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.026 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.026 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.172755+0000", 2024-09-18T07:39:30.026 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5133000000000002e-05 2024-09-18T07:39:30.026 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.026 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.172766+0000", 2024-09-18T07:39:30.027 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0627e-05 2024-09-18T07:39:30.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.172773+0000", 2024-09-18T07:39:30.027 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6769999999999998e-06 2024-09-18T07:39:30.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.172852+0000", 2024-09-18T07:39:30.027 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9454000000000002e-05 2024-09-18T07:39:30.028 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.028 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.028 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.028 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.172954+0000", 2024-09-18T07:39:30.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000101702 2024-09-18T07:39:30.028 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.028 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.028 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:39:30.028 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.173051+0000", 2024-09-18T07:39:30.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7293000000000005e-05 2024-09-18T07:39:30.028 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.028 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.028 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:30.028 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.176801+0000", 2024-09-18T07:39:30.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003750087 2024-09-18T07:39:30.029 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.029 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.029 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.029 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.179269+0000", 2024-09-18T07:39:30.029 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0024674789999999999 2024-09-18T07:39:30.029 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.029 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.029 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.029 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.317180+0000", 2024-09-18T07:39:30.029 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.137911229 2024-09-18T07:39:30.029 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.029 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.029 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.030 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.317233+0000", 2024-09-18T07:39:30.030 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3597999999999997e-05 2024-09-18T07:39:30.030 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.030 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.030 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.030 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.317270+0000", 2024-09-18T07:39:30.030 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6816999999999997e-05 2024-09-18T07:39:30.030 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.030 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.030 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.031 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.031 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.031 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:74675 7.0 7:0421ef5f:::benchmark_data_smithi137_74180_object74674:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:39:30.031 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:16.173394+0000", 2024-09-18T07:39:30.031 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 373.81736427700002, 2024-09-18T07:39:30.031 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14466180100000001, 2024-09-18T07:39:30.031 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.031 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.031 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:30.031 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:30.031 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:39:30.031 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:30.031 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 74675 2024-09-18T07:39:30.032 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.032 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.032 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.032 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.032 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.173394+0000", 2024-09-18T07:39:30.032 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.032 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.032 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.032 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.032 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.173394+0000", 2024-09-18T07:39:30.032 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.034 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.173395+0000", 2024-09-18T07:39:30.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9600000000000008e-07 2024-09-18T07:39:30.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.173460+0000", 2024-09-18T07:39:30.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5114e-05 2024-09-18T07:39:30.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.173471+0000", 2024-09-18T07:39:30.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0506000000000001e-05 2024-09-18T07:39:30.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.173476+0000", 2024-09-18T07:39:30.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.254e-06 2024-09-18T07:39:30.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.036 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.036 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.036 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.175237+0000", 2024-09-18T07:39:30.036 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001761475 2024-09-18T07:39:30.036 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.036 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.036 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.036 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.175349+0000", 2024-09-18T07:39:30.036 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011191799999999999 2024-09-18T07:39:30.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:39:30.037 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.175484+0000", 2024-09-18T07:39:30.037 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000134864 2024-09-18T07:39:30.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:30.037 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.176926+0000", 2024-09-18T07:39:30.037 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014422390000000001 2024-09-18T07:39:30.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.038 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.038 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.179509+0000", 2024-09-18T07:39:30.038 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025827990000000002 2024-09-18T07:39:30.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.038 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.038 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.038 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.317964+0000", 2024-09-18T07:39:30.038 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13845500499999999 2024-09-18T07:39:30.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.038 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.038 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.038 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.318020+0000", 2024-09-18T07:39:30.039 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5599000000000001e-05 2024-09-18T07:39:30.039 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.039 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.039 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.039 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.318056+0000", 2024-09-18T07:39:30.039 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6032000000000003e-05 2024-09-18T07:39:30.039 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.039 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.039 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.039 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.039 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.039 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:109508 8.9 8:97b6bfc8:::benchmark_data_smithi137_101942_object4698:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-18T07:39:30.039 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:41.987094+0000", 2024-09-18T07:39:30.040 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 108.003664159, 2024-09-18T07:39:30.040 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2676243700000001, 2024-09-18T07:39:30.040 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.040 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.040 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:39:30.040 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:30.040 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:39:30.040 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:39:30.040 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 109508 2024-09-18T07:39:30.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.040 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.040 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.040 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.987094+0000", 2024-09-18T07:39:30.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.041 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.987094+0000", 2024-09-18T07:39:30.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.041 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.987096+0000", 2024-09-18T07:39:30.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7120000000000001e-06 2024-09-18T07:39:30.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.987101+0000", 2024-09-18T07:39:30.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9269999999999996e-06 2024-09-18T07:39:30.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.987111+0000", 2024-09-18T07:39:30.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.961e-06 2024-09-18T07:39:30.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.987117+0000", 2024-09-18T07:39:30.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6499999999999999e-06 2024-09-18T07:39:30.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.043 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.043 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.989469+0000", 2024-09-18T07:39:30.043 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023516660000000001 2024-09-18T07:39:30.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.043 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.043 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.989583+0000", 2024-09-18T07:39:30.043 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011409 2024-09-18T07:39:30.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.043 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:39:30.043 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.989980+0000", 2024-09-18T07:39:30.044 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00039646499999999998 2024-09-18T07:39:30.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.044 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:30.044 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.046941+0000", 2024-09-18T07:39:30.044 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.05696118 2024-09-18T07:39:30.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.044 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.044 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.046992+0000", 2024-09-18T07:39:30.044 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1535e-05 2024-09-18T07:39:30.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.045 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.045 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.045 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.052235+0000", 2024-09-18T07:39:30.045 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0052421760000000003 2024-09-18T07:39:30.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.045 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:39:30.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.052247+0000", 2024-09-18T07:39:30.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.203e-05 2024-09-18T07:39:30.046 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.046 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.047 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.239993+0000", 2024-09-18T07:39:30.047 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.187746747 2024-09-18T07:39:30.047 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.047 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.047 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for missing object", 2024-09-18T07:39:30.047 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.240136+0000", 2024-09-18T07:39:30.047 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014249299999999999 2024-09-18T07:39:30.048 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.048 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.048 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.048 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.254655+0000", 2024-09-18T07:39:30.048 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.01451916 2024-09-18T07:39:30.048 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.049 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.049 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.049 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.254719+0000", 2024-09-18T07:39:30.049 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3577999999999993e-05 2024-09-18T07:39:30.049 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.049 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.050 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.050 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.050 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.050 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:109510 8.9 8:9ab7fc49:::benchmark_data_smithi137_101942_object4700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-18T07:39:30.050 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:41.987415+0000", 2024-09-18T07:39:30.050 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 108.00334381899999, 2024-09-18T07:39:30.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.273351415, 2024-09-18T07:39:30.051 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.051 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.051 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:30.051 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:30.051 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:39:30.052 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:39:30.052 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 109510 2024-09-18T07:39:30.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.052 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.052 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.987415+0000", 2024-09-18T07:39:30.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.054 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.987415+0000", 2024-09-18T07:39:30.054 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.054 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.054 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.054 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.054 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.987415+0000", 2024-09-18T07:39:30.055 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8000000000000004e-07 2024-09-18T07:39:30.055 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.055 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.055 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.055 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.987419+0000", 2024-09-18T07:39:30.055 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0680000000000001e-06 2024-09-18T07:39:30.056 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.056 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.056 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.056 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.987426+0000", 2024-09-18T07:39:30.056 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9510000000000006e-06 2024-09-18T07:39:30.057 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.057 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.057 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.057 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.987431+0000", 2024-09-18T07:39:30.057 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2230000000000001e-06 2024-09-18T07:39:30.058 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.058 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.058 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.058 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.052700+0000", 2024-09-18T07:39:30.058 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.065269708999999995 2024-09-18T07:39:30.058 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.059 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:39:30.059 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.052721+0000", 2024-09-18T07:39:30.059 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0282e-05 2024-09-18T07:39:30.059 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.060 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.060 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.242065+0000", 2024-09-18T07:39:30.060 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.189344365 2024-09-18T07:39:30.060 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.060 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.060 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.061 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.242183+0000", 2024-09-18T07:39:30.061 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000118175 2024-09-18T07:39:30.061 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.061 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.061 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,6", 2024-09-18T07:39:30.061 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.242320+0000", 2024-09-18T07:39:30.062 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013660199999999999 2024-09-18T07:39:30.062 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.062 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.062 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:30.062 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.250436+0000", 2024-09-18T07:39:30.062 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0081163929999999995 2024-09-18T07:39:30.063 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.063 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.063 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.063 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.250547+0000", 2024-09-18T07:39:30.063 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000110845 2024-09-18T07:39:30.063 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.064 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.064 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.064 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.258878+0000", 2024-09-18T07:39:30.064 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0083307120000000005 2024-09-18T07:39:30.064 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.065 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.065 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.065 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.258956+0000", 2024-09-18T07:39:30.065 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7843000000000007e-05 2024-09-18T07:39:30.065 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.065 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.066 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.066 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.260766+0000", 2024-09-18T07:39:30.066 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001810367 2024-09-18T07:39:30.067 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.067 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.067 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.067 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.067 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.067 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:109516 8.9 8:954df695:::benchmark_data_smithi137_101942_object4706:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-18T07:39:30.068 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:41.992530+0000", 2024-09-18T07:39:30.068 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 107.998228016, 2024-09-18T07:39:30.068 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2765922089999999, 2024-09-18T07:39:30.068 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.068 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.069 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:30.069 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:30.069 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:39:30.069 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:39:30.069 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 109516 2024-09-18T07:39:30.070 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.070 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.070 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.070 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.070 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.992530+0000", 2024-09-18T07:39:30.070 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.071 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.071 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.071 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.992530+0000", 2024-09-18T07:39:30.071 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.072 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.072 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.072 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.992533+0000", 2024-09-18T07:39:30.072 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2560000000000001e-06 2024-09-18T07:39:30.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.073 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.073 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:41.992538+0000", 2024-09-18T07:39:30.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3589999999999999e-06 2024-09-18T07:39:30.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.074 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.074 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.042095+0000", 2024-09-18T07:39:30.074 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.049556827999999997 2024-09-18T07:39:30.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.074 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.042109+0000", 2024-09-18T07:39:30.075 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3866000000000001e-05 2024-09-18T07:39:30.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.075 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.076 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.052753+0000", 2024-09-18T07:39:30.076 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010644199999999999 2024-09-18T07:39:30.076 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.076 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.076 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:39:30.076 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.052760+0000", 2024-09-18T07:39:30.077 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0260000000000002e-06 2024-09-18T07:39:30.077 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.077 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.077 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.077 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.245097+0000", 2024-09-18T07:39:30.078 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1923366559999999 2024-09-18T07:39:30.078 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.078 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.078 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.078 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.245215+0000", 2024-09-18T07:39:30.078 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000118254 2024-09-18T07:39:30.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,6", 2024-09-18T07:39:30.079 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.245365+0000", 2024-09-18T07:39:30.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015008099999999999 2024-09-18T07:39:30.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.080 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.080 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:30.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.250509+0000", 2024-09-18T07:39:30.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0051445889999999998 2024-09-18T07:39:30.080 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.080 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.081 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.081 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.250589+0000", 2024-09-18T07:39:30.081 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9135999999999996e-05 2024-09-18T07:39:30.081 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.082 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.082 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.267167+0000", 2024-09-18T07:39:30.082 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.016578513 2024-09-18T07:39:30.082 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.082 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.083 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.083 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.267219+0000", 2024-09-18T07:39:30.083 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1801000000000001e-05 2024-09-18T07:39:30.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.083 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.083 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.083 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.269123+0000", 2024-09-18T07:39:30.084 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0019036439999999999 2024-09-18T07:39:30.084 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.084 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.084 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.084 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.085 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:109526 8.3 8:c2c082d9:::benchmark_data_smithi137_101942_object4716:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-18T07:39:30.085 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:42.042458+0000", 2024-09-18T07:39:30.085 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 107.948300813, 2024-09-18T07:39:30.085 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2164607039999999, 2024-09-18T07:39:30.085 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.086 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.086 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:39:30.086 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:30.086 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:39:30.086 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:39:30.086 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 109526 2024-09-18T07:39:30.086 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.087 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.087 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.087 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.087 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.042458+0000", 2024-09-18T07:39:30.087 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.087 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.087 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.042458+0000", 2024-09-18T07:39:30.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.042458+0000", 2024-09-18T07:39:30.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9400000000000004e-07 2024-09-18T07:39:30.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.042462+0000", 2024-09-18T07:39:30.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6969999999999999e-06 2024-09-18T07:39:30.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.042469+0000", 2024-09-18T07:39:30.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4699999999999999e-06 2024-09-18T07:39:30.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.042475+0000", 2024-09-18T07:39:30.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0610000000000001e-06 2024-09-18T07:39:30.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.045262+0000", 2024-09-18T07:39:30.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027876229999999999 2024-09-18T07:39:30.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.090 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.090 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.090 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.045402+0000", 2024-09-18T07:39:30.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000139868 2024-09-18T07:39:30.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.090 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.090 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,2", 2024-09-18T07:39:30.090 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.045555+0000", 2024-09-18T07:39:30.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015285 2024-09-18T07:39:30.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.090 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.090 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.090 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.056407+0000", 2024-09-18T07:39:30.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010851603 2024-09-18T07:39:30.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:39:30.091 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.056416+0000", 2024-09-18T07:39:30.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4939999999999992e-06 2024-09-18T07:39:30.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.091 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.240564+0000", 2024-09-18T07:39:30.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.184148247 2024-09-18T07:39:30.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for missing object", 2024-09-18T07:39:30.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.240754+0000", 2024-09-18T07:39:30.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019005500000000001 2024-09-18T07:39:30.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.258811+0000", 2024-09-18T07:39:30.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.018057113 2024-09-18T07:39:30.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.258918+0000", 2024-09-18T07:39:30.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000106829 2024-09-18T07:39:30.092 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.092 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.093 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.093 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.093 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.093 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:109531 8.9 8:9e3c6218:::benchmark_data_smithi137_101942_object4721:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-18T07:39:30.093 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:42.046126+0000", 2024-09-18T07:39:30.093 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 107.944632693, 2024-09-18T07:39:30.093 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2247508979999999, 2024-09-18T07:39:30.093 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.093 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.093 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:30.093 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:30.093 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:39:30.093 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:39:30.093 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 109531 2024-09-18T07:39:30.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.094 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.094 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.094 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.094 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.046126+0000", 2024-09-18T07:39:30.094 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.094 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.094 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.094 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.046126+0000", 2024-09-18T07:39:30.094 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.094 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.094 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.046127+0000", 2024-09-18T07:39:30.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9999999999999996e-07 2024-09-18T07:39:30.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.095 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.046130+0000", 2024-09-18T07:39:30.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5590000000000001e-06 2024-09-18T07:39:30.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.095 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.046138+0000", 2024-09-18T07:39:30.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9019999999999999e-06 2024-09-18T07:39:30.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.096 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.096 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.046145+0000", 2024-09-18T07:39:30.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.109e-06 2024-09-18T07:39:30.096 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.096 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.096 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.096 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.052787+0000", 2024-09-18T07:39:30.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0066419160000000003 2024-09-18T07:39:30.096 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.096 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.096 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:39:30.096 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.052795+0000", 2024-09-18T07:39:30.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.276e-06 2024-09-18T07:39:30.096 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.247931+0000", 2024-09-18T07:39:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.19513527 2024-09-18T07:39:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.248074+0000", 2024-09-18T07:39:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014358799999999999 2024-09-18T07:39:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,6", 2024-09-18T07:39:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.248214+0000", 2024-09-18T07:39:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013970700000000001 2024-09-18T07:39:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.254786+0000", 2024-09-18T07:39:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0065719510000000004 2024-09-18T07:39:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.254825+0000", 2024-09-18T07:39:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8677999999999998e-05 2024-09-18T07:39:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.269180+0000", 2024-09-18T07:39:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.014354986 2024-09-18T07:39:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.269233+0000", 2024-09-18T07:39:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3850000000000001e-05 2024-09-18T07:39:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.270877+0000", 2024-09-18T07:39:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016432059999999999 2024-09-18T07:39:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:109548 8.3 8:c5a7115f:::benchmark_data_smithi137_101942_object4738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:39:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:42.075213+0000", 2024-09-18T07:39:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 107.915545733, 2024-09-18T07:39:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.194679802, 2024-09-18T07:39:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:39:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:39:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 109548 2024-09-18T07:39:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.075213+0000", 2024-09-18T07:39:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.075213+0000", 2024-09-18T07:39:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.075215+0000", 2024-09-18T07:39:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5890000000000001e-06 2024-09-18T07:39:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.075228+0000", 2024-09-18T07:39:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3207000000000001e-05 2024-09-18T07:39:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.075249+0000", 2024-09-18T07:39:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0234e-05 2024-09-18T07:39:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.075258+0000", 2024-09-18T07:39:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1139999999999995e-06 2024-09-18T07:39:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.085093+0000", 2024-09-18T07:39:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0098353099999999999 2024-09-18T07:39:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:39:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.085107+0000", 2024-09-18T07:39:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3879999999999999e-05 2024-09-18T07:39:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.243117+0000", 2024-09-18T07:39:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1580104549999999 2024-09-18T07:39:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.243253+0000", 2024-09-18T07:39:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000135816 2024-09-18T07:39:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:39:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.243390+0000", 2024-09-18T07:39:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013656599999999999 2024-09-18T07:39:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.105 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.105 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.105 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.255284+0000", 2024-09-18T07:39:30.105 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011894267 2024-09-18T07:39:30.105 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.105 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.105 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.105 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.259216+0000", 2024-09-18T07:39:30.105 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0039318679999999998 2024-09-18T07:39:30.105 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.105 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.105 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:30.105 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.265487+0000", 2024-09-18T07:39:30.105 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0062706589999999996 2024-09-18T07:39:30.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.106 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.106 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.106 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.265563+0000", 2024-09-18T07:39:30.106 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6743999999999996e-05 2024-09-18T07:39:30.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.106 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.106 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.106 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.269892+0000", 2024-09-18T07:39:30.106 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.004329093 2024-09-18T07:39:30.106 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.106 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.106 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.107 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.107 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:109558 8.3 8:cea2c28a:::benchmark_data_smithi137_101942_object4748:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:39:30.107 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:42.094266+0000", 2024-09-18T07:39:30.107 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 107.896492321, 2024-09-18T07:39:30.107 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1774776090000001, 2024-09-18T07:39:30.107 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.107 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.107 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:30.107 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:30.107 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:39:30.107 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:39:30.107 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 109558 2024-09-18T07:39:30.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.107 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.108 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.108 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.108 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.094266+0000", 2024-09-18T07:39:30.108 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.108 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.108 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.108 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.108 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.094266+0000", 2024-09-18T07:39:30.108 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.108 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.108 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.108 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.108 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.094268+0000", 2024-09-18T07:39:30.109 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1679999999999998e-06 2024-09-18T07:39:30.109 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.109 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.109 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.109 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.094274+0000", 2024-09-18T07:39:30.109 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8930000000000002e-06 2024-09-18T07:39:30.109 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.109 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.109 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.109 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.094285+0000", 2024-09-18T07:39:30.109 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1072e-05 2024-09-18T07:39:30.109 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.109 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.109 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.110 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.094292+0000", 2024-09-18T07:39:30.110 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9630000000000001e-06 2024-09-18T07:39:30.110 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.110 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.110 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.110 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.094366+0000", 2024-09-18T07:39:30.110 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3523999999999996e-05 2024-09-18T07:39:30.110 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.110 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.110 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:39:30.110 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.094377+0000", 2024-09-18T07:39:30.110 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1766000000000001e-05 2024-09-18T07:39:30.110 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.110 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.111 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.111 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.246167+0000", 2024-09-18T07:39:30.111 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1517894470000001 2024-09-18T07:39:30.111 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.111 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.111 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.111 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.246282+0000", 2024-09-18T07:39:30.111 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000115418 2024-09-18T07:39:30.111 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.111 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.111 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:39:30.111 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.246399+0000", 2024-09-18T07:39:30.111 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011667699999999999 2024-09-18T07:39:30.111 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.112 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.112 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.112 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.255328+0000", 2024-09-18T07:39:30.112 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0089285560000000007 2024-09-18T07:39:30.112 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.112 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.112 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:30.112 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.269906+0000", 2024-09-18T07:39:30.112 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.014578371 2024-09-18T07:39:30.112 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.112 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.112 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.112 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.270059+0000", 2024-09-18T07:39:30.112 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015271100000000001 2024-09-18T07:39:30.113 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.113 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.113 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.113 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.270146+0000", 2024-09-18T07:39:30.113 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7523999999999998e-05 2024-09-18T07:39:30.113 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.113 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.113 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.113 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.271744+0000", 2024-09-18T07:39:30.113 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0015975189999999999 2024-09-18T07:39:30.113 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.113 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.113 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.113 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.114 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.114 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:109564 8.9 8:951ecca1:::benchmark_data_smithi137_101942_object4754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:39:30.114 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:42.099739+0000", 2024-09-18T07:39:30.114 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 107.89101902, 2024-09-18T07:39:30.114 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1728653840000001, 2024-09-18T07:39:30.114 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.114 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.114 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:30.114 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:30.114 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:39:30.114 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:39:30.114 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 109564 2024-09-18T07:39:30.114 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.114 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.115 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.115 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.115 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.099739+0000", 2024-09-18T07:39:30.115 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.115 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.115 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.115 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.115 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.099739+0000", 2024-09-18T07:39:30.115 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.115 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.115 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.115 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.115 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.099740+0000", 2024-09-18T07:39:30.115 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5000000000000002e-07 2024-09-18T07:39:30.116 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.116 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.116 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.116 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.099742+0000", 2024-09-18T07:39:30.116 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.311e-06 2024-09-18T07:39:30.116 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.116 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.116 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.116 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.099749+0000", 2024-09-18T07:39:30.116 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4439999999999996e-06 2024-09-18T07:39:30.116 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.116 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.116 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.117 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.099752+0000", 2024-09-18T07:39:30.117 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.416e-06 2024-09-18T07:39:30.117 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.117 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.117 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.117 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.100732+0000", 2024-09-18T07:39:30.117 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00098021099999999993 2024-09-18T07:39:30.117 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.117 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.117 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:39:30.117 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.100741+0000", 2024-09-18T07:39:30.117 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8789999999999992e-06 2024-09-18T07:39:30.117 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.117 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.118 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.118 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.254856+0000", 2024-09-18T07:39:30.118 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.154114511 2024-09-18T07:39:30.118 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.118 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.118 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.118 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.254929+0000", 2024-09-18T07:39:30.118 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3281000000000005e-05 2024-09-18T07:39:30.118 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.118 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.118 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,6", 2024-09-18T07:39:30.118 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.255019+0000", 2024-09-18T07:39:30.118 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9875000000000003e-05 2024-09-18T07:39:30.119 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.119 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.119 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:30.119 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.260784+0000", 2024-09-18T07:39:30.119 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.005764618 2024-09-18T07:39:30.119 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.119 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.119 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.119 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.264342+0000", 2024-09-18T07:39:30.119 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0035582090000000001 2024-09-18T07:39:30.119 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.119 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.119 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.119 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.270943+0000", 2024-09-18T07:39:30.120 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0066013640000000002 2024-09-18T07:39:30.120 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.120 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.120 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.120 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.270996+0000", 2024-09-18T07:39:30.120 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3087999999999997e-05 2024-09-18T07:39:30.120 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.120 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.120 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.120 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.272605+0000", 2024-09-18T07:39:30.120 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001608527 2024-09-18T07:39:30.120 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.120 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.120 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.121 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.121 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.121 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:109566 8.3 8:c82e1fbe:::benchmark_data_smithi137_101942_object4756:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:39:30.121 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:42.102690+0000", 2024-09-18T07:39:30.121 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 107.888068244, 2024-09-18T07:39:30.121 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.17458474, 2024-09-18T07:39:30.121 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.121 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.121 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:30.121 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:30.121 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:39:30.121 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:39:30.121 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 109566 2024-09-18T07:39:30.121 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.122 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.122 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.122 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.122 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.102690+0000", 2024-09-18T07:39:30.122 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.122 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.122 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.122 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.102690+0000", 2024-09-18T07:39:30.122 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.122 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.122 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.122 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.102691+0000", 2024-09-18T07:39:30.123 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0109999999999999e-06 2024-09-18T07:39:30.123 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.123 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.123 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.123 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.102695+0000", 2024-09-18T07:39:30.123 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.313e-06 2024-09-18T07:39:30.123 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.123 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.123 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.123 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.102703+0000", 2024-09-18T07:39:30.123 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0949999999999998e-06 2024-09-18T07:39:30.123 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.123 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.123 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.123 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.102707+0000", 2024-09-18T07:39:30.124 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2320000000000003e-06 2024-09-18T07:39:30.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.124 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.124 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.102748+0000", 2024-09-18T07:39:30.124 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1389e-05 2024-09-18T07:39:30.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.124 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:39:30.124 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.102755+0000", 2024-09-18T07:39:30.124 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.024e-06 2024-09-18T07:39:30.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.125 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.125 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.249600+0000", 2024-09-18T07:39:30.125 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.146844854 2024-09-18T07:39:30.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.125 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.125 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.125 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.249725+0000", 2024-09-18T07:39:30.125 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000124964 2024-09-18T07:39:30.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.125 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.125 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:39:30.125 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.249844+0000", 2024-09-18T07:39:30.125 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011934699999999999 2024-09-18T07:39:30.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.126 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.126 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.126 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.259168+0000", 2024-09-18T07:39:30.126 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0093238959999999999 2024-09-18T07:39:30.126 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.126 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.126 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:30.126 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.269912+0000", 2024-09-18T07:39:30.126 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010744004999999999 2024-09-18T07:39:30.126 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.126 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.126 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.126 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.275803+0000", 2024-09-18T07:39:30.126 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0058902700000000004 2024-09-18T07:39:30.127 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.127 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.127 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.127 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.275848+0000", 2024-09-18T07:39:30.127 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5784000000000001e-05 2024-09-18T07:39:30.127 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.127 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.127 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.127 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.277275+0000", 2024-09-18T07:39:30.127 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001426556 2024-09-18T07:39:30.127 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.127 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.127 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.127 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.128 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.128 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:109569 8.3 8:c7bc1d2f:::benchmark_data_smithi137_101942_object4759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:39:30.128 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:42.107598+0000", 2024-09-18T07:39:30.128 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 107.883160792, 2024-09-18T07:39:30.128 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.171422111, 2024-09-18T07:39:30.128 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.128 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.128 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:30.128 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:30.128 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:39:30.128 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:39:30.128 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 109569 2024-09-18T07:39:30.128 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.128 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.129 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.129 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.129 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.107598+0000", 2024-09-18T07:39:30.129 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.129 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.129 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.129 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.129 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.107598+0000", 2024-09-18T07:39:30.129 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.129 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.129 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.129 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.129 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.107599+0000", 2024-09-18T07:39:30.130 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1930000000000001e-06 2024-09-18T07:39:30.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.130 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.130 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.107603+0000", 2024-09-18T07:39:30.130 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9790000000000004e-06 2024-09-18T07:39:30.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.130 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.130 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.107613+0000", 2024-09-18T07:39:30.130 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7720000000000006e-06 2024-09-18T07:39:30.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.130 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.107618+0000", 2024-09-18T07:39:30.131 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4199999999999998e-06 2024-09-18T07:39:30.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.131 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.131 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.107657+0000", 2024-09-18T07:39:30.131 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9371e-05 2024-09-18T07:39:30.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.131 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.131 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:39:30.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:42.107663+0000", 2024-09-18T07:39:30.131 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7769999999999999e-06 2024-09-18T07:39:30.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.131 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.131 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.132 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.259308+0000", 2024-09-18T07:39:30.132 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1516448370000001 2024-09-18T07:39:30.132 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.132 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.132 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.132 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.259431+0000", 2024-09-18T07:39:30.132 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012285 2024-09-18T07:39:30.132 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.132 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.132 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:39:30.132 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.259554+0000", 2024-09-18T07:39:30.132 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000122946 2024-09-18T07:39:30.132 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.133 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.133 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.133 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.269963+0000", 2024-09-18T07:39:30.133 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010409349 2024-09-18T07:39:30.133 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.133 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.133 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:30.133 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.269976+0000", 2024-09-18T07:39:30.133 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2670999999999999e-05 2024-09-18T07:39:30.133 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.133 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.133 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.133 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.277345+0000", 2024-09-18T07:39:30.133 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0073694549999999996 2024-09-18T07:39:30.134 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.134 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.134 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.134 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.277403+0000", 2024-09-18T07:39:30.134 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7569999999999999e-05 2024-09-18T07:39:30.134 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.134 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.134 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.134 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:43.279020+0000", 2024-09-18T07:39:30.134 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016169209999999999 2024-09-18T07:39:30.134 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.134 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.134 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.134 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.135 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.135 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:119506 8.0 e241/201 8:0a9cf428:::benchmark_data_smithi137_101942_object14696:head v 241'7457, mlcod=241'7454)", 2024-09-18T07:39:30.135 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:53.113362+0000", 2024-09-18T07:39:30.135 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 96.877396125999994, 2024-09-18T07:39:30.135 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16945199899999999, 2024-09-18T07:39:30.135 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.135 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.135 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:30.135 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.135 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.135 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.135 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113362+0000", 2024-09-18T07:39:30.135 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.135 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.136 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.136 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.136 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113362+0000", 2024-09-18T07:39:30.136 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.136 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.136 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.136 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.136 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113364+0000", 2024-09-18T07:39:30.136 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.55e-06 2024-09-18T07:39:30.136 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.136 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.136 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.136 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113372+0000", 2024-09-18T07:39:30.136 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5690000000000005e-06 2024-09-18T07:39:30.137 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.137 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.137 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.137 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.196010+0000", 2024-09-18T07:39:30.137 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.082637639999999998 2024-09-18T07:39:30.137 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.137 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.137 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.137 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.196016+0000", 2024-09-18T07:39:30.137 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.348e-06 2024-09-18T07:39:30.137 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.137 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.137 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.137 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.196316+0000", 2024-09-18T07:39:30.138 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00029944200000000002 2024-09-18T07:39:30.138 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.138 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.138 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.138 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.196332+0000", 2024-09-18T07:39:30.138 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6231000000000001e-05 2024-09-18T07:39:30.138 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.138 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.138 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.138 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.282741+0000", 2024-09-18T07:39:30.138 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.086409383000000006 2024-09-18T07:39:30.138 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.138 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.138 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.139 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.282814+0000", 2024-09-18T07:39:30.139 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2836000000000005e-05 2024-09-18T07:39:30.139 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.139 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.139 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.139 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.139 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.139 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:152092 8.9 8:950104e1:::benchmark_data_smithi137_101942_object47282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-09-18T07:39:30.139 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:26.864007+0000", 2024-09-18T07:39:30.139 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 63.126751812000002, 2024-09-18T07:39:30.139 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14380670800000001, 2024-09-18T07:39:30.139 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.139 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.139 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:30.139 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:30.140 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:39:30.140 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:39:30.140 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 152092 2024-09-18T07:39:30.140 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.140 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.140 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.140 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.140 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.864007+0000", 2024-09-18T07:39:30.140 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.140 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.140 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.140 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.140 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.864007+0000", 2024-09-18T07:39:30.140 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.141 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.141 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.141 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.141 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.864008+0000", 2024-09-18T07:39:30.141 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.094e-06 2024-09-18T07:39:30.141 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.141 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.141 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.141 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.864011+0000", 2024-09-18T07:39:30.141 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8139999999999998e-06 2024-09-18T07:39:30.141 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.141 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.141 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.142 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.864020+0000", 2024-09-18T07:39:30.142 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3939999999999996e-06 2024-09-18T07:39:30.142 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.142 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.142 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.142 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.864059+0000", 2024-09-18T07:39:30.142 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9269e-05 2024-09-18T07:39:30.142 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.142 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.142 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.142 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.866670+0000", 2024-09-18T07:39:30.142 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002611234 2024-09-18T07:39:30.142 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.142 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.143 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.143 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.866826+0000", 2024-09-18T07:39:30.143 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015582200000000001 2024-09-18T07:39:30.143 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.143 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.143 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,6", 2024-09-18T07:39:30.143 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.867009+0000", 2024-09-18T07:39:30.143 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018317600000000001 2024-09-18T07:39:30.143 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.143 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.143 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:30.143 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.869535+0000", 2024-09-18T07:39:30.143 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025253989999999998 2024-09-18T07:39:30.144 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.144 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.144 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.144 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.877105+0000", 2024-09-18T07:39:30.144 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0075699210000000003 2024-09-18T07:39:30.144 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.144 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.144 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.144 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:27.006431+0000", 2024-09-18T07:39:30.144 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.12932639600000001 2024-09-18T07:39:30.144 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.144 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.144 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.144 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:27.006486+0000", 2024-09-18T07:39:30.145 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4496000000000001e-05 2024-09-18T07:39:30.145 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.145 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.145 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.145 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:27.007813+0000", 2024-09-18T07:39:30.145 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001327693 2024-09-18T07:39:30.145 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.145 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.145 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.145 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.145 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.145 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:170411 8.d 8:bde4aff4:::benchmark_data_smithi137_101942_object65601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2024-09-18T07:39:30.145 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:44.766297+0000", 2024-09-18T07:39:30.145 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 45.224461708, 2024-09-18T07:39:30.146 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.146087735, 2024-09-18T07:39:30.146 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.146 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.146 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:30.146 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:30.146 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:39:30.146 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:39:30.146 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 170411 2024-09-18T07:39:30.146 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.146 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.146 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.146 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.146 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.766297+0000", 2024-09-18T07:39:30.146 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.147 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.147 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.147 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.147 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.766297+0000", 2024-09-18T07:39:30.147 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.147 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.147 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.147 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.147 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.766299+0000", 2024-09-18T07:39:30.147 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.159e-06 2024-09-18T07:39:30.147 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.147 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.147 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.147 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.766305+0000", 2024-09-18T07:39:30.148 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8329999999999999e-06 2024-09-18T07:39:30.148 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.148 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.148 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.148 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.766316+0000", 2024-09-18T07:39:30.148 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1449e-05 2024-09-18T07:39:30.148 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.148 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.148 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.148 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.766323+0000", 2024-09-18T07:39:30.148 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1829999999999997e-06 2024-09-18T07:39:30.148 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.148 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.148 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.149 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.766405+0000", 2024-09-18T07:39:30.149 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1977999999999993e-05 2024-09-18T07:39:30.149 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.149 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.149 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.149 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.766509+0000", 2024-09-18T07:39:30.149 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000103744 2024-09-18T07:39:30.149 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.149 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.149 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:39:30.149 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.766608+0000", 2024-09-18T07:39:30.149 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8636999999999998e-05 2024-09-18T07:39:30.149 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.149 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.150 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:30.150 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.771338+0000", 2024-09-18T07:39:30.150 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0047307119999999998 2024-09-18T07:39:30.150 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.150 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.150 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.150 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.775356+0000", 2024-09-18T07:39:30.150 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0040178810000000001 2024-09-18T07:39:30.150 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.150 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.150 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.150 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.912271+0000", 2024-09-18T07:39:30.150 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13691498699999999 2024-09-18T07:39:30.150 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.151 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.151 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.151 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.912324+0000", 2024-09-18T07:39:30.151 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3007999999999998e-05 2024-09-18T07:39:30.151 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.151 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.151 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.151 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.912384+0000", 2024-09-18T07:39:30.151 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0164000000000002e-05 2024-09-18T07:39:30.151 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.151 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.151 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.151 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.151 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.152 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:202158 8.0 e281/201 8:04615c61:::benchmark_data_smithi137_101942_object97348:head v 281'12563, mlcod=281'12559)", 2024-09-18T07:39:30.152 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:39:13.182290+0000", 2024-09-18T07:39:30.152 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 16.808468763, 2024-09-18T07:39:30.152 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13823819400000001, 2024-09-18T07:39:30.152 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.152 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.152 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:30.152 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.152 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.152 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.152 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.182290+0000", 2024-09-18T07:39:30.152 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.152 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.152 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.153 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.153 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.182290+0000", 2024-09-18T07:39:30.153 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.153 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.153 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.153 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.153 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.182291+0000", 2024-09-18T07:39:30.153 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.116e-06 2024-09-18T07:39:30.153 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.153 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.153 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.153 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.182298+0000", 2024-09-18T07:39:30.153 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1230000000000002e-06 2024-09-18T07:39:30.153 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.154 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.154 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.154 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.182306+0000", 2024-09-18T07:39:30.154 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7479999999999993e-06 2024-09-18T07:39:30.154 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.154 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.154 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.154 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.182311+0000", 2024-09-18T07:39:30.154 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5090000000000001e-06 2024-09-18T07:39:30.154 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.154 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.154 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.154 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.182370+0000", 2024-09-18T07:39:30.154 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8587e-05 2024-09-18T07:39:30.154 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.155 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.155 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.155 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.182388+0000", 2024-09-18T07:39:30.155 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8033000000000002e-05 2024-09-18T07:39:30.155 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.155 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.155 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.155 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.320461+0000", 2024-09-18T07:39:30.155 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13807312799999999 2024-09-18T07:39:30.155 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.155 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.155 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.155 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.320528+0000", 2024-09-18T07:39:30.155 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6950000000000001e-05 2024-09-18T07:39:30.156 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.156 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.156 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.156 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.156 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.156 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:202164 8.8 8:1685d572:::benchmark_data_smithi137_101942_object97354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-18T07:39:30.156 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:39:13.182429+0000", 2024-09-18T07:39:30.156 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 16.808329175000001, 2024-09-18T07:39:30.156 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.138744385, 2024-09-18T07:39:30.156 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:30.156 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:30.156 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:30.156 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:30.156 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:39:30.157 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:39:30.157 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 202164 2024-09-18T07:39:30.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.157 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:30.157 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.157 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:30.157 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.182429+0000", 2024-09-18T07:39:30.157 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.157 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.157 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:30.157 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.182429+0000", 2024-09-18T07:39:30.157 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:30.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.158 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.158 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:30.158 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.182430+0000", 2024-09-18T07:39:30.158 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0470000000000001e-06 2024-09-18T07:39:30.158 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.158 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.158 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:30.158 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.182434+0000", 2024-09-18T07:39:30.158 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0509999999999998e-06 2024-09-18T07:39:30.158 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.158 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.158 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:30.158 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.182443+0000", 2024-09-18T07:39:30.158 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5890000000000003e-06 2024-09-18T07:39:30.158 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.159 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.159 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:30.159 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.182449+0000", 2024-09-18T07:39:30.159 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5729999999999996e-06 2024-09-18T07:39:30.159 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.159 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.159 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:30.159 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.188658+0000", 2024-09-18T07:39:30.159 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0062087920000000003 2024-09-18T07:39:30.159 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.159 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.159 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:30.159 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.188752+0000", 2024-09-18T07:39:30.159 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3356999999999994e-05 2024-09-18T07:39:30.160 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.160 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.160 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:39:30.160 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.188844+0000", 2024-09-18T07:39:30.160 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2161000000000001e-05 2024-09-18T07:39:30.160 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.160 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.160 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.160 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.320102+0000", 2024-09-18T07:39:30.160 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13125783199999999 2024-09-18T07:39:30.160 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.160 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.160 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:30.160 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.320564+0000", 2024-09-18T07:39:30.161 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000462254 2024-09-18T07:39:30.161 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.161 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.161 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:30.161 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.321062+0000", 2024-09-18T07:39:30.161 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00049852200000000005 2024-09-18T07:39:30.161 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.161 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.161 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:30.161 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.321139+0000", 2024-09-18T07:39:30.161 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6883000000000002e-05 2024-09-18T07:39:30.161 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:30.161 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:30.161 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:30.161 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.321174+0000", 2024-09-18T07:39:30.162 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4323999999999999e-05 2024-09-18T07:39:30.162 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.162 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.162 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.162 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:30.162 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:30.162 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:39:30.162 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:30.163 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:29 smithi137 ceph-mon[29408]: from='client.35522 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-18T07:39:30.163 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:29 smithi137 ceph-mon[29408]: osdmap e289: 8 total, 8 up, 8 in 2024-09-18T07:39:30.163 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:29 smithi137 ceph-mon[29408]: 3.6 scrub starts 2024-09-18T07:39:30.164 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:29 smithi137 ceph-mon[29408]: 3.6 scrub ok 2024-09-18T07:39:30.164 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:29 smithi137 ceph-mon[29408]: 3.8 scrub starts 2024-09-18T07:39:30.164 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:29 smithi137 ceph-mon[29408]: 3.8 scrub ok 2024-09-18T07:39:30.164 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:29 smithi137 ceph-mon[39175]: from='client.35522 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-18T07:39:30.164 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:29 smithi137 ceph-mon[39175]: osdmap e289: 8 total, 8 up, 8 in 2024-09-18T07:39:30.164 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:29 smithi137 ceph-mon[39175]: 3.6 scrub starts 2024-09-18T07:39:30.164 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:29 smithi137 ceph-mon[39175]: 3.6 scrub ok 2024-09-18T07:39:30.164 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:29 smithi137 ceph-mon[39175]: 3.8 scrub starts 2024-09-18T07:39:30.164 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:29 smithi137 ceph-mon[39175]: 3.8 scrub ok 2024-09-18T07:39:30.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:30.185 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:30.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:30.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:39:30.286 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:30.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:30.308 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:30.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:30.409 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:30.430 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:30.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:30.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:30.532 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:30.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:30.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:30.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:30.656 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:30.678 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:30.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:30.780 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:30.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:30.803 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:30.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:30.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:30.925 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:30.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:30 smithi203 ceph-mon[32237]: 3.9 scrub starts 2024-09-18T07:39:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:30 smithi203 ceph-mon[32237]: 3.9 scrub ok 2024-09-18T07:39:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:30 smithi203 ceph-mon[32237]: pgmap v461: 158 pgs: 16 unknown, 142 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-18T07:39:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:30 smithi203 ceph-mon[32237]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:39:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:30 smithi203 ceph-mon[32237]: osdmap e290: 8 total, 8 up, 8 in 2024-09-18T07:39:30.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:30 smithi203 ceph-mon[32237]: osdmap e291: 8 total, 8 up, 8 in 2024-09-18T07:39:31.027 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:31.038 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:30 smithi137 ceph-mon[29408]: 3.9 scrub starts 2024-09-18T07:39:31.038 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:30 smithi137 ceph-mon[29408]: 3.9 scrub ok 2024-09-18T07:39:31.038 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:30 smithi137 ceph-mon[29408]: pgmap v461: 158 pgs: 16 unknown, 142 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-18T07:39:31.038 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:30 smithi137 ceph-mon[29408]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:39:31.038 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:30 smithi137 ceph-mon[29408]: osdmap e290: 8 total, 8 up, 8 in 2024-09-18T07:39:31.038 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:30 smithi137 ceph-mon[29408]: osdmap e291: 8 total, 8 up, 8 in 2024-09-18T07:39:31.038 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:30 smithi137 ceph-mon[39175]: 3.9 scrub starts 2024-09-18T07:39:31.039 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:30 smithi137 ceph-mon[39175]: 3.9 scrub ok 2024-09-18T07:39:31.039 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:30 smithi137 ceph-mon[39175]: pgmap v461: 158 pgs: 16 unknown, 142 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-18T07:39:31.039 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:30 smithi137 ceph-mon[39175]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:39:31.039 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:30 smithi137 ceph-mon[39175]: osdmap e290: 8 total, 8 up, 8 in 2024-09-18T07:39:31.039 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:30 smithi137 ceph-mon[39175]: osdmap e291: 8 total, 8 up, 8 in 2024-09-18T07:39:31.046 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T07:39:31.048 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:31.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:31.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:31.074 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:31.113 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:31.151 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:31.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:31.173 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:31.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:31.275 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:31.297 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:31.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:31.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:31.299 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:39:31.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:31.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:31.428 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:31.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:31.529 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:31.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:31.551 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:31.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:31.653 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:31.674 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:31.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:31.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:31.702 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.3 to 1 (10000) 2024-09-18T07:39:31.702 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:39:31.775 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:39:31.776 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:39:31.776 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-09-18T07:39:31.776 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:39:31.776 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:31.794 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:39:31.794 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:39:31.798 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:39:31.798 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:39:31.799 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:31.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:31.814 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:39:31.815 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:39:31.830 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:39:31.831 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:39:31.847 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:39:31.848 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:39:31.864 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:39:31.864 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:39:31.880 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:39:31.881 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:39:31.897 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:39:31.897 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:39:31.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:31.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:31 smithi137 ceph-mon[39175]: 6.19 scrub starts 2024-09-18T07:39:31.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:31 smithi137 ceph-mon[39175]: 6.19 scrub ok 2024-09-18T07:39:31.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:31 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3313516913' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:31.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:31 smithi137 ceph-mon[39175]: from='client.35528 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:31.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:31 smithi137 ceph-mon[39175]: 5.17 scrub starts 2024-09-18T07:39:31.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:31 smithi137 ceph-mon[39175]: 5.17 scrub ok 2024-09-18T07:39:31.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:31 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2952278502' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:39:31.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:31 smithi137 ceph-mon[39175]: from='client.35534 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:39:31.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:31 smithi137 ceph-mon[29408]: 6.19 scrub starts 2024-09-18T07:39:31.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:31 smithi137 ceph-mon[29408]: 6.19 scrub ok 2024-09-18T07:39:31.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:31 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3313516913' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:31.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:31 smithi137 ceph-mon[29408]: from='client.35528 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:31.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:31 smithi137 ceph-mon[29408]: 5.17 scrub starts 2024-09-18T07:39:31.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:31 smithi137 ceph-mon[29408]: 5.17 scrub ok 2024-09-18T07:39:31.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:31 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2952278502' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:39:31.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:31 smithi137 ceph-mon[29408]: from='client.35534 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:39:31.923 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:31.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:31.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:31.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:31 smithi203 ceph-mon[32237]: 6.19 scrub starts 2024-09-18T07:39:31.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:31 smithi203 ceph-mon[32237]: 6.19 scrub ok 2024-09-18T07:39:31.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:31 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3313516913' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:31.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:31 smithi203 ceph-mon[32237]: from='client.35528 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:31.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:31 smithi203 ceph-mon[32237]: 5.17 scrub starts 2024-09-18T07:39:31.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:31 smithi203 ceph-mon[32237]: 5.17 scrub ok 2024-09-18T07:39:31.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:31 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2952278502' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:39:31.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:31 smithi203 ceph-mon[32237]: from='client.35534 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:39:32.024 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:32.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:32.046 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:32.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:32.148 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:32.170 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:32.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:32.271 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:32.292 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:32.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:32.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:32.394 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:32.410 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T07:39:32.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:32.417 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:32.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:32.518 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:32.540 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:32.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:32.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:32.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:32.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:32.664 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:32.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:32.665 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:39:32.697 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-18T07:39:32.698 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[29408]: 3.5 scrub starts 2024-09-18T07:39:32.698 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[29408]: 3.5 scrub ok 2024-09-18T07:39:32.698 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[29408]: pgmap v464: 158 pgs: 12 unknown, 146 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:39:32.698 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:39:32.698 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[29408]: from='client.35528 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:39:32.698 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[29408]: from='client.35534 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T07:39:32.698 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[29408]: osdmap e292: 8 total, 8 up, 8 in 2024-09-18T07:39:32.698 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2252493401' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:32.698 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[29408]: 6.16 scrub starts 2024-09-18T07:39:32.698 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[29408]: 6.16 scrub ok 2024-09-18T07:39:32.698 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1832760469' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:39:32.699 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[39175]: 3.5 scrub starts 2024-09-18T07:39:32.699 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[39175]: 3.5 scrub ok 2024-09-18T07:39:32.699 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[39175]: pgmap v464: 158 pgs: 12 unknown, 146 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:39:32.700 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:39:32.700 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[39175]: from='client.35528 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:39:32.700 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[39175]: from='client.35534 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T07:39:32.700 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[39175]: osdmap e292: 8 total, 8 up, 8 in 2024-09-18T07:39:32.700 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2252493401' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:32.700 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[39175]: 6.16 scrub starts 2024-09-18T07:39:32.700 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[39175]: 6.16 scrub ok 2024-09-18T07:39:32.700 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:32 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1832760469' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:39:32.766 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:32.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:32.788 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:32.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:32.889 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:32.912 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:32.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:32.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:32.943 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:32 smithi203 ceph-mon[32237]: 3.5 scrub starts 2024-09-18T07:39:32.943 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:32 smithi203 ceph-mon[32237]: 3.5 scrub ok 2024-09-18T07:39:32.943 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:32 smithi203 ceph-mon[32237]: pgmap v464: 158 pgs: 12 unknown, 146 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:39:32.943 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:32 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:39:32.943 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:32 smithi203 ceph-mon[32237]: from='client.35528 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:39:32.943 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:32 smithi203 ceph-mon[32237]: from='client.35534 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T07:39:32.943 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:32 smithi203 ceph-mon[32237]: osdmap e292: 8 total, 8 up, 8 in 2024-09-18T07:39:32.943 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:32 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2252493401' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:32.943 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:32 smithi203 ceph-mon[32237]: 6.16 scrub starts 2024-09-18T07:39:32.943 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:32 smithi203 ceph-mon[32237]: 6.16 scrub ok 2024-09-18T07:39:32.943 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:32 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1832760469' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:39:33.014 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:33.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:33.037 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:33.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:33.137 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:33.159 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:39:33.159 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:39:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:33.160 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:33.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:33.261 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:33.284 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:33.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:33.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:33.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:33.407 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:33.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:33.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:33.509 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:33.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:33.532 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:33.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:33.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:33.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:33.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:33.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:33.733 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T07:39:33.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:33.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:33.777 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:33.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:33.878 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:33.899 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:33.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:33.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:33.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:33 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1832760469' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:39:33.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:33 smithi137 ceph-mon[29408]: osdmap e293: 8 total, 8 up, 8 in 2024-09-18T07:39:33.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:33 smithi137 ceph-mon[29408]: 3.12 scrub starts 2024-09-18T07:39:33.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:33 smithi137 ceph-mon[29408]: 3.12 scrub ok 2024-09-18T07:39:33.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:33 smithi137 ceph-mon[29408]: 5.14 scrub starts 2024-09-18T07:39:33.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:33 smithi137 ceph-mon[29408]: 5.14 scrub ok 2024-09-18T07:39:33.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:33 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1832760469' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:39:33.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:33 smithi137 ceph-mon[39175]: osdmap e293: 8 total, 8 up, 8 in 2024-09-18T07:39:33.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:33 smithi137 ceph-mon[39175]: 3.12 scrub starts 2024-09-18T07:39:33.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:33 smithi137 ceph-mon[39175]: 3.12 scrub ok 2024-09-18T07:39:33.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:33 smithi137 ceph-mon[39175]: 5.14 scrub starts 2024-09-18T07:39:33.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:33 smithi137 ceph-mon[39175]: 5.14 scrub ok 2024-09-18T07:39:33.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:33 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1832760469' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:39:33.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:33 smithi203 ceph-mon[32237]: osdmap e293: 8 total, 8 up, 8 in 2024-09-18T07:39:33.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:33 smithi203 ceph-mon[32237]: 3.12 scrub starts 2024-09-18T07:39:33.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:33 smithi203 ceph-mon[32237]: 3.12 scrub ok 2024-09-18T07:39:33.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:33 smithi203 ceph-mon[32237]: 5.14 scrub starts 2024-09-18T07:39:33.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:33 smithi203 ceph-mon[32237]: 5.14 scrub ok 2024-09-18T07:39:33.982 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:39:34.001 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:34.023 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:34.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:34.124 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:34.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:34.147 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:34.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:34.248 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:34.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:34.270 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:34.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:34.371 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:34.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:34.394 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:34.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:34.478 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:39:34.478 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:39:34.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:34.516 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:34.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:34.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:34.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-18T07:39:34.619 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:34.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:34.651 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:34.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:34.752 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:34.773 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:34.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:34.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:34.839 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:34 smithi137 ceph-mon[29408]: pgmap v467: 158 pgs: 158 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:39:34.839 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:34 smithi137 ceph-mon[29408]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:39:34.840 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:34 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:39:34.840 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:34 smithi137 ceph-mon[29408]: 6.1b scrub starts 2024-09-18T07:39:34.840 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:34 smithi137 ceph-mon[29408]: 6.1b scrub ok 2024-09-18T07:39:34.840 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:34 smithi137 ceph-mon[29408]: 5.1f scrub starts 2024-09-18T07:39:34.840 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:34 smithi137 ceph-mon[29408]: 5.1f scrub ok 2024-09-18T07:39:34.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:34 smithi137 ceph-mon[39175]: pgmap v467: 158 pgs: 158 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:39:34.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:34 smithi137 ceph-mon[39175]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:39:34.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:34 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:39:34.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:34 smithi137 ceph-mon[39175]: 6.1b scrub starts 2024-09-18T07:39:34.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:34 smithi137 ceph-mon[39175]: 6.1b scrub ok 2024-09-18T07:39:34.841 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:34 smithi137 ceph-mon[39175]: 5.1f scrub starts 2024-09-18T07:39:34.841 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:34 smithi137 ceph-mon[39175]: 5.1f scrub ok 2024-09-18T07:39:34.875 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:34.897 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:34.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:34.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:34.919 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:34.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:34 smithi203 ceph-mon[32237]: pgmap v467: 158 pgs: 158 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:39:34.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:34 smithi203 ceph-mon[32237]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:39:34.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:34 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:39:34.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:34 smithi203 ceph-mon[32237]: 6.1b scrub starts 2024-09-18T07:39:34.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:34 smithi203 ceph-mon[32237]: 6.1b scrub ok 2024-09-18T07:39:34.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:34 smithi203 ceph-mon[32237]: 5.1f scrub starts 2024-09-18T07:39:34.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:34 smithi203 ceph-mon[32237]: 5.1f scrub ok 2024-09-18T07:39:34.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:35.001 DEBUG:teuthology.orchestra.run.smithi137:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-18T07:39:35.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:35.021 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:35.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:35.022 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-18T07:39:35.022 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-18T07:39:35.070 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:hints = 1 2024-09-18T07:39:35.070 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-18T07:39:35.070 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Object prefix: benchmark_data_smithi137_126072 2024-09-18T07:39:35.071 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:39:35.071 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 0 0 0 0 0 0 - 0 2024-09-18T07:39:35.122 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:35.161 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:35.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:35.199 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T07:39:35.262 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:35.320 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:35.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:35.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:35.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:35.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:35.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:35.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:35.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:39:35.560 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:35.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:35.632 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:35.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:35.640 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:39:35.734 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:35.767 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:35.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:35.869 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:35.901 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:35.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:35.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:35.950 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.4 to 1 (10000) 2024-09-18T07:39:36.004 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:36.035 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:36.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:36.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:36.071 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 1 15 3220 3205 200.216 200.312 0.00290511 0.004981 2024-09-18T07:39:36.096 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:35 smithi203 ceph-mon[32237]: 3.a scrub starts 2024-09-18T07:39:36.096 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:35 smithi203 ceph-mon[32237]: 3.a scrub ok 2024-09-18T07:39:36.096 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:35 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2008161309' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:39:36.096 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:35 smithi203 ceph-mon[32237]: from='client.35561 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:39:36.137 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:36.174 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:36.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:36.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:36.265 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:39:36.265 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:39:36.276 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:36.306 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:35 smithi137 ceph-mon[39175]: 3.a scrub starts 2024-09-18T07:39:36.307 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:35 smithi137 ceph-mon[39175]: 3.a scrub ok 2024-09-18T07:39:36.307 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:35 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2008161309' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:39:36.307 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:35 smithi137 ceph-mon[39175]: from='client.35561 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:39:36.307 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:35 smithi137 ceph-mon[29408]: 3.a scrub starts 2024-09-18T07:39:36.307 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:35 smithi137 ceph-mon[29408]: 3.a scrub ok 2024-09-18T07:39:36.307 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:35 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2008161309' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:39:36.307 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:35 smithi137 ceph-mon[29408]: from='client.35561 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:39:36.312 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:36.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:36.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:36.414 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:36.449 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:36.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:36.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:36.551 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:36.588 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:36.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:36.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:36.668 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:36.688 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-18T07:39:36.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:36.740 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:36.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:36.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:36.841 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:36.868 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:39:36.872 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:36.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:36.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:36.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:37.022 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T07:39:37.025 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:37.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:37.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:37.071 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 2 16 6578 6562 204.973 209.812 0.00425756 0.00486741 2024-09-18T07:39:37.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:37.166 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:37.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:37.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:37.168 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[29408]: 5.f scrub starts 2024-09-18T07:39:37.168 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[29408]: 5.f scrub ok 2024-09-18T07:39:37.168 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[29408]: 5.6 scrub starts 2024-09-18T07:39:37.168 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[29408]: 5.6 scrub ok 2024-09-18T07:39:37.168 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[29408]: pgmap v468: 158 pgs: 158 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2024-09-18T07:39:37.168 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[29408]: 5.1e scrub starts 2024-09-18T07:39:37.168 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[29408]: 5.1e scrub ok 2024-09-18T07:39:37.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[29408]: 3.18 scrub starts 2024-09-18T07:39:37.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[29408]: 3.18 scrub ok 2024-09-18T07:39:37.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[29408]: from='client.35561 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:39:37.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[29408]: osdmap e294: 8 total, 8 up, 8 in 2024-09-18T07:39:37.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:39:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:39:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:39:37.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[39175]: 5.f scrub starts 2024-09-18T07:39:37.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[39175]: 5.f scrub ok 2024-09-18T07:39:37.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[39175]: 5.6 scrub starts 2024-09-18T07:39:37.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[39175]: 5.6 scrub ok 2024-09-18T07:39:37.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[39175]: pgmap v468: 158 pgs: 158 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2024-09-18T07:39:37.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[39175]: 5.1e scrub starts 2024-09-18T07:39:37.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[39175]: 5.1e scrub ok 2024-09-18T07:39:37.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[39175]: 3.18 scrub starts 2024-09-18T07:39:37.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[39175]: 3.18 scrub ok 2024-09-18T07:39:37.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[39175]: from='client.35561 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:39:37.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:36 smithi137 ceph-mon[39175]: osdmap e294: 8 total, 8 up, 8 in 2024-09-18T07:39:37.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:36 smithi203 ceph-mon[32237]: 5.f scrub starts 2024-09-18T07:39:37.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:36 smithi203 ceph-mon[32237]: 5.f scrub ok 2024-09-18T07:39:37.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:36 smithi203 ceph-mon[32237]: 5.6 scrub starts 2024-09-18T07:39:37.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:36 smithi203 ceph-mon[32237]: 5.6 scrub ok 2024-09-18T07:39:37.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:36 smithi203 ceph-mon[32237]: pgmap v468: 158 pgs: 158 active+clean; 378 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2024-09-18T07:39:37.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:36 smithi203 ceph-mon[32237]: 5.1e scrub starts 2024-09-18T07:39:37.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:36 smithi203 ceph-mon[32237]: 5.1e scrub ok 2024-09-18T07:39:37.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:36 smithi203 ceph-mon[32237]: 3.18 scrub starts 2024-09-18T07:39:37.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:36 smithi203 ceph-mon[32237]: 3.18 scrub ok 2024-09-18T07:39:37.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:36 smithi203 ceph-mon[32237]: from='client.35561 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:39:37.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:36 smithi203 ceph-mon[32237]: osdmap e294: 8 total, 8 up, 8 in 2024-09-18T07:39:37.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:37.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:37.305 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:37.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:37.327 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:39:37.328 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:39:37.365 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:39:37.365 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:39:37.400 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:39:37.400 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:39:37.406 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:37.431 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:39:37.442 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:39:37.442 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:39:37.449 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:37.449 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:37.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:37.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:37.473 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:39:37.473 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:39:37.505 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:39:37.505 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:39:37.520 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:37.538 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:39:37.539 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:39:37.551 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:37.579 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:39:37.579 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:39:37.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:37.597 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:37.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:37.671 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:39:37 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:39:37.698 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:37.733 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:37.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:37.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:37.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:37.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:37.868 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:37.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:37.969 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:38.006 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:38.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:38.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:38.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:37 smithi137 ceph-mon[29408]: 3.1f scrub starts 2024-09-18T07:39:38.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:37 smithi137 ceph-mon[29408]: 3.1f scrub ok 2024-09-18T07:39:38.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:37 smithi137 ceph-mon[29408]: 3.10 scrub starts 2024-09-18T07:39:38.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:37 smithi137 ceph-mon[29408]: 3.10 scrub ok 2024-09-18T07:39:38.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:37 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1017137056' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:38.017 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:37 smithi137 ceph-mon[39175]: 3.1f scrub starts 2024-09-18T07:39:38.017 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:37 smithi137 ceph-mon[39175]: 3.1f scrub ok 2024-09-18T07:39:38.017 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:37 smithi137 ceph-mon[39175]: 3.10 scrub starts 2024-09-18T07:39:38.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:37 smithi137 ceph-mon[39175]: 3.10 scrub ok 2024-09-18T07:39:38.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:37 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1017137056' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:38.071 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 3 16 9431 9415 196.084 178.312 0.00131582 0.00506395 2024-09-18T07:39:38.080 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:39:38.080 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:39:38.080 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:39:38.080 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:39:38.080 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:39:38.108 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:38.142 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:38.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:38.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:38.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:37 smithi203 ceph-mon[32237]: 3.1f scrub starts 2024-09-18T07:39:38.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:37 smithi203 ceph-mon[32237]: 3.1f scrub ok 2024-09-18T07:39:38.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:37 smithi203 ceph-mon[32237]: 3.10 scrub starts 2024-09-18T07:39:38.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:37 smithi203 ceph-mon[32237]: 3.10 scrub ok 2024-09-18T07:39:38.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:37 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1017137056' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:38.245 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:38.296 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:38.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:38.398 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:38.439 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:38.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:38.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:38.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:38.582 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:38.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:38.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:38.684 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:38.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:38.715 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:38.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:38.793 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T07:39:38.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:38.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:38.853 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:38.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:38.954 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:38.972 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.5 to 1 (10000) 2024-09-18T07:39:38.992 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:38.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:38.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:39.071 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 4 16 12196 12180 190.264 172.812 0.00201236 0.00523489 2024-09-18T07:39:39.094 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:39.135 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:39.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:39.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:39.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:38 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:39.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:38 smithi203 ceph-mon[32237]: pgmap v470: 158 pgs: 158 active+clean; 382 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-18T07:39:39.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:38 smithi203 ceph-mon[32237]: 6.11 scrub starts 2024-09-18T07:39:39.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:38 smithi203 ceph-mon[32237]: 6.11 scrub ok 2024-09-18T07:39:39.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:38 smithi203 ceph-mon[32237]: 5.1c scrub starts 2024-09-18T07:39:39.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:38 smithi203 ceph-mon[32237]: 5.1c scrub ok 2024-09-18T07:39:39.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:38 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1740569655' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:39:39.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:38 smithi203 ceph-mon[32237]: from='client.25636 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:39:39.213 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:38 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4269272467' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:39:39.213 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:39:39.230 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:39.230 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[29408]: pgmap v470: 158 pgs: 158 active+clean; 382 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-18T07:39:39.230 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[29408]: 6.11 scrub starts 2024-09-18T07:39:39.230 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[29408]: 6.11 scrub ok 2024-09-18T07:39:39.230 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[29408]: 5.1c scrub starts 2024-09-18T07:39:39.230 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[29408]: 5.1c scrub ok 2024-09-18T07:39:39.230 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1740569655' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:39:39.230 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[29408]: from='client.25636 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:39:39.230 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4269272467' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:39:39.231 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:39.231 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[39175]: pgmap v470: 158 pgs: 158 active+clean; 382 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-18T07:39:39.231 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[39175]: 6.11 scrub starts 2024-09-18T07:39:39.231 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[39175]: 6.11 scrub ok 2024-09-18T07:39:39.231 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[39175]: 5.1c scrub starts 2024-09-18T07:39:39.231 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[39175]: 5.1c scrub ok 2024-09-18T07:39:39.231 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1740569655' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:39:39.231 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[39175]: from='client.25636 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:39:39.231 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:38 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4269272467' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:39:39.238 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:39.274 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:39.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:39.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:39.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:39.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:39.415 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:39.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:39.515 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:39.554 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:39.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:39.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:39.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:39.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:39.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:39.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:39.791 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:39.824 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:39:39.825 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:39:39.825 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:39:39.825 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:39:39.825 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.825 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:21294 7.8 e89/81 7:15222838:::benchmark_data_smithi137_74180_object21293:head v 89'1394, mlcod=89'1392)", 2024-09-18T07:39:39.825 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.948825+0000", 2024-09-18T07:39:39.825 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 435.86948611100001, 2024-09-18T07:39:39.825 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.094945371000000001, 2024-09-18T07:39:39.825 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.826 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.826 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.826 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.826 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.826 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.826 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948825+0000", 2024-09-18T07:39:39.826 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.826 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.826 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.826 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.826 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948825+0000", 2024-09-18T07:39:39.826 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.826 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.827 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.827 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.827 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948826+0000", 2024-09-18T07:39:39.827 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3e-07 2024-09-18T07:39:39.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.827 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.827 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.827 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948939+0000", 2024-09-18T07:39:39.827 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000113757 2024-09-18T07:39:39.827 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.827 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.827 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.828 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948994+0000", 2024-09-18T07:39:39.828 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4622e-05 2024-09-18T07:39:39.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.828 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.828 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.828 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.949000+0000", 2024-09-18T07:39:39.828 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9599999999999997e-06 2024-09-18T07:39:39.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.828 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.828 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.828 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.949042+0000", 2024-09-18T07:39:39.828 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2558999999999997e-05 2024-09-18T07:39:39.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.829 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.829 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.829 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.949055+0000", 2024-09-18T07:39:39.829 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2969999999999999e-05 2024-09-18T07:39:39.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.829 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.829 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.829 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.043723+0000", 2024-09-18T07:39:39.829 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.094668034999999998 2024-09-18T07:39:39.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.829 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.830 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.830 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.043770+0000", 2024-09-18T07:39:39.830 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6838000000000001e-05 2024-09-18T07:39:39.830 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.830 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.830 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.830 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.830 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:21293 7.f e89/81 7:fa840974:::benchmark_data_smithi137_74180_object21292:head v 89'1337, mlcod=89'1334)", 2024-09-18T07:39:39.830 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.948830+0000", 2024-09-18T07:39:39.830 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 435.869481092, 2024-09-18T07:39:39.830 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.096476913999999997, 2024-09-18T07:39:39.830 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.831 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.831 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.831 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.831 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.831 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.831 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948830+0000", 2024-09-18T07:39:39.831 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.831 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.831 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.831 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948830+0000", 2024-09-18T07:39:39.831 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.832 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.832 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.832 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948831+0000", 2024-09-18T07:39:39.832 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.235e-06 2024-09-18T07:39:39.832 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.832 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.832 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.832 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948942+0000", 2024-09-18T07:39:39.832 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000110417 2024-09-18T07:39:39.832 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.832 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.832 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.833 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948954+0000", 2024-09-18T07:39:39.833 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1888999999999999e-05 2024-09-18T07:39:39.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.833 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.833 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.833 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948959+0000", 2024-09-18T07:39:39.833 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8259999999999998e-06 2024-09-18T07:39:39.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.833 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.833 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.833 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.948998+0000", 2024-09-18T07:39:39.833 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8498999999999999e-05 2024-09-18T07:39:39.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.949009+0000", 2024-09-18T07:39:39.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1413999999999999e-05 2024-09-18T07:39:39.834 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.045250+0000", 2024-09-18T07:39:39.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.096240945999999994 2024-09-18T07:39:39.834 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.045307+0000", 2024-09-18T07:39:39.835 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6687999999999997e-05 2024-09-18T07:39:39.835 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.835 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.835 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.835 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21896 7.5 7:a849d1d6:::benchmark_data_smithi137_74180_object21895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:39:39.835 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.316136+0000", 2024-09-18T07:39:39.835 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 435.50217494999998, 2024-09-18T07:39:39.835 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.073939160000000004, 2024-09-18T07:39:39.835 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.835 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.835 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.836 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:39.836 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:39:39.836 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:39.836 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21896 2024-09-18T07:39:39.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.836 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.836 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.836 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.836 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316136+0000", 2024-09-18T07:39:39.836 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.836 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.836 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.837 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316136+0000", 2024-09-18T07:39:39.837 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.837 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.837 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.837 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316138+0000", 2024-09-18T07:39:39.837 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.426e-06 2024-09-18T07:39:39.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.837 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.837 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.837 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316190+0000", 2024-09-18T07:39:39.837 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2704000000000002e-05 2024-09-18T07:39:39.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.838 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.838 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.838 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316201+0000", 2024-09-18T07:39:39.838 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0298e-05 2024-09-18T07:39:39.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.838 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.838 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.838 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316205+0000", 2024-09-18T07:39:39.838 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8860000000000003e-06 2024-09-18T07:39:39.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.838 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.838 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.838 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316244+0000", 2024-09-18T07:39:39.839 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8535999999999998e-05 2024-09-18T07:39:39.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.839 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.839 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.839 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316328+0000", 2024-09-18T07:39:39.839 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4384999999999999e-05 2024-09-18T07:39:39.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.839 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.839 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:39:39.839 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.316417+0000", 2024-09-18T07:39:39.839 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8794000000000004e-05 2024-09-18T07:39:39.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.839 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.839 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.840 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.388000+0000", 2024-09-18T07:39:39.840 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.071582792000000006 2024-09-18T07:39:39.840 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.840 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.840 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:39.840 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.388272+0000", 2024-09-18T07:39:39.840 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00027234600000000002 2024-09-18T07:39:39.840 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.840 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.840 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.840 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.389970+0000", 2024-09-18T07:39:39.840 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016974959999999999 2024-09-18T07:39:39.841 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.841 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.841 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.841 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.390033+0000", 2024-09-18T07:39:39.841 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3640999999999996e-05 2024-09-18T07:39:39.841 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.841 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.841 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.841 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.390075+0000", 2024-09-18T07:39:39.841 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1856e-05 2024-09-18T07:39:39.841 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.842 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.842 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.842 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.842 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop_reply(client.14865.0:21896 7.5 e89/81 ondisk, result = 0)", 2024-09-18T07:39:39.842 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.317455+0000", 2024-09-18T07:39:39.842 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 435.50085655999999, 2024-09-18T07:39:39.842 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.070552445000000005, 2024-09-18T07:39:39.842 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.842 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.842 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:39:39.843 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.843 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.843 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.317455+0000", 2024-09-18T07:39:39.843 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.843 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.843 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.317455+0000", 2024-09-18T07:39:39.843 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.843 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.844 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.317455+0000", 2024-09-18T07:39:39.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4399999999999998e-07 2024-09-18T07:39:39.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.844 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.844 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.844 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.317459+0000", 2024-09-18T07:39:39.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7749999999999999e-06 2024-09-18T07:39:39.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.844 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.844 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.844 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.386905+0000", 2024-09-18T07:39:39.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.069445328000000001 2024-09-18T07:39:39.845 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.845 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.845 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.845 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.386916+0000", 2024-09-18T07:39:39.845 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1331e-05 2024-09-18T07:39:39.845 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.845 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.845 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.845 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.387987+0000", 2024-09-18T07:39:39.845 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010713020000000001 2024-09-18T07:39:39.845 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.845 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.846 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.846 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.387995+0000", 2024-09-18T07:39:39.846 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3679999999999996e-06 2024-09-18T07:39:39.846 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.846 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.846 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.846 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.388007+0000", 2024-09-18T07:39:39.846 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2397e-05 2024-09-18T07:39:39.846 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.846 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.846 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.846 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.847 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.847 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21999 7.5 7:a6ca4f28:::benchmark_data_smithi137_74180_object21998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:39:39.847 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.424624+0000", 2024-09-18T07:39:39.847 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 435.39368706800002, 2024-09-18T07:39:39.847 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.112047436, 2024-09-18T07:39:39.847 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.847 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.847 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.847 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:39.847 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:39:39.847 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:39.847 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21999 2024-09-18T07:39:39.847 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.848 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.848 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.848 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.848 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424624+0000", 2024-09-18T07:39:39.848 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.848 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.848 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.848 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.848 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424624+0000", 2024-09-18T07:39:39.848 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.848 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.848 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424626+0000", 2024-09-18T07:39:39.849 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.075e-06 2024-09-18T07:39:39.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.849 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424682+0000", 2024-09-18T07:39:39.849 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5597999999999999e-05 2024-09-18T07:39:39.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.849 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424694+0000", 2024-09-18T07:39:39.849 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2656e-05 2024-09-18T07:39:39.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.850 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.850 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424702+0000", 2024-09-18T07:39:39.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3849999999999996e-06 2024-09-18T07:39:39.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.850 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.850 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424758+0000", 2024-09-18T07:39:39.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6020000000000002e-05 2024-09-18T07:39:39.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.850 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.851 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.424884+0000", 2024-09-18T07:39:39.851 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012638800000000001 2024-09-18T07:39:39.851 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.851 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.851 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:39:39.851 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425020+0000", 2024-09-18T07:39:39.851 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013587300000000001 2024-09-18T07:39:39.851 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.851 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.851 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.851 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.426628+0000", 2024-09-18T07:39:39.851 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001607792 2024-09-18T07:39:39.851 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.852 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.852 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:39.852 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.426665+0000", 2024-09-18T07:39:39.852 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7052999999999999e-05 2024-09-18T07:39:39.852 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.852 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.852 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.852 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536540+0000", 2024-09-18T07:39:39.852 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10987469399999999 2024-09-18T07:39:39.852 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.852 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.852 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.852 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536614+0000", 2024-09-18T07:39:39.853 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4486999999999993e-05 2024-09-18T07:39:39.853 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.853 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.853 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.853 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536671+0000", 2024-09-18T07:39:39.853 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7414999999999998e-05 2024-09-18T07:39:39.853 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.853 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.853 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.853 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.853 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.853 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22002 7.5 7:a1dbf95a:::benchmark_data_smithi137_74180_object22001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:39:39.853 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.425607+0000", 2024-09-18T07:39:39.853 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 435.39270442499998, 2024-09-18T07:39:39.854 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.111294137, 2024-09-18T07:39:39.854 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.854 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.854 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.854 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:39.854 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:39:39.854 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:39.854 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22002 2024-09-18T07:39:39.854 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.854 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.854 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.854 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.854 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425607+0000", 2024-09-18T07:39:39.855 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.855 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.855 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.855 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.855 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425607+0000", 2024-09-18T07:39:39.855 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.855 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.855 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.855 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.856 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425608+0000", 2024-09-18T07:39:39.856 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.297e-06 2024-09-18T07:39:39.856 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.856 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.856 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.856 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425682+0000", 2024-09-18T07:39:39.856 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3891000000000001e-05 2024-09-18T07:39:39.856 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.856 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.856 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.856 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425693+0000", 2024-09-18T07:39:39.857 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0944000000000001e-05 2024-09-18T07:39:39.857 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.857 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.857 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.857 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425701+0000", 2024-09-18T07:39:39.857 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7489999999999998e-06 2024-09-18T07:39:39.857 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.857 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.857 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.857 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425809+0000", 2024-09-18T07:39:39.857 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108323 2024-09-18T07:39:39.857 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.857 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.858 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.858 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.425883+0000", 2024-09-18T07:39:39.858 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4321999999999998e-05 2024-09-18T07:39:39.858 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.858 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.858 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:39:39.858 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.426014+0000", 2024-09-18T07:39:39.858 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013092400000000001 2024-09-18T07:39:39.858 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.858 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.858 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:39.858 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.426735+0000", 2024-09-18T07:39:39.858 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00072051000000000005 2024-09-18T07:39:39.859 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.859 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.859 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.859 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.427445+0000", 2024-09-18T07:39:39.859 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00070996799999999995 2024-09-18T07:39:39.859 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.859 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.859 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.859 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536803+0000", 2024-09-18T07:39:39.859 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10935832500000001 2024-09-18T07:39:39.859 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.859 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.860 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.860 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536856+0000", 2024-09-18T07:39:39.860 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3161000000000001e-05 2024-09-18T07:39:39.860 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.860 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.860 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.860 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.536901+0000", 2024-09-18T07:39:39.860 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4722999999999999e-05 2024-09-18T07:39:39.860 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.860 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.860 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.860 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.861 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.861 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22010 7.5 7:a615b038:::benchmark_data_smithi137_74180_object22009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:39:39.861 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.428662+0000", 2024-09-18T07:39:39.861 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 435.38964879100001, 2024-09-18T07:39:39.861 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.11444004000000001, 2024-09-18T07:39:39.861 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.861 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.861 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.861 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:39.861 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:39:39.861 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:39.861 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22010 2024-09-18T07:39:39.861 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.861 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.862 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.862 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.862 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428662+0000", 2024-09-18T07:39:39.862 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.862 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.862 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.862 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.862 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428662+0000", 2024-09-18T07:39:39.862 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.862 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.862 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.862 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.862 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428664+0000", 2024-09-18T07:39:39.863 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.302e-06 2024-09-18T07:39:39.863 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.863 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.863 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.863 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428707+0000", 2024-09-18T07:39:39.863 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3692000000000001e-05 2024-09-18T07:39:39.863 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.863 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.863 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.863 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428718+0000", 2024-09-18T07:39:39.863 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1066999999999999e-05 2024-09-18T07:39:39.863 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.863 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.864 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.864 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428724+0000", 2024-09-18T07:39:39.864 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7590000000000003e-06 2024-09-18T07:39:39.864 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.864 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.864 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.864 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428771+0000", 2024-09-18T07:39:39.864 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6462999999999999e-05 2024-09-18T07:39:39.864 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.864 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.864 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.864 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428852+0000", 2024-09-18T07:39:39.864 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1533999999999996e-05 2024-09-18T07:39:39.864 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.865 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.865 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:39:39.865 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.428937+0000", 2024-09-18T07:39:39.865 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5068999999999995e-05 2024-09-18T07:39:39.865 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.865 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.865 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:39.865 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.429584+0000", 2024-09-18T07:39:39.865 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00064701400000000001 2024-09-18T07:39:39.865 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.865 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.865 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.865 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.429931+0000", 2024-09-18T07:39:39.866 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00034729299999999998 2024-09-18T07:39:39.866 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.866 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.866 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.866 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.543034+0000", 2024-09-18T07:39:39.866 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.113102231 2024-09-18T07:39:39.866 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.866 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.866 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.866 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.543071+0000", 2024-09-18T07:39:39.866 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7333999999999999e-05 2024-09-18T07:39:39.866 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.866 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.867 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.867 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.543102+0000", 2024-09-18T07:39:39.867 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1282000000000003e-05 2024-09-18T07:39:39.867 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.867 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.867 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.867 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.867 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.867 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:22343 7.6 e89/81 7:6ec6138d:::benchmark_data_smithi137_74180_object22342:head v 89'1362, mlcod=89'1360)", 2024-09-18T07:39:39.867 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.661175+0000", 2024-09-18T07:39:39.867 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 435.15713593200002, 2024-09-18T07:39:39.867 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19171324300000001, 2024-09-18T07:39:39.867 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.867 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.868 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.868 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.868 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.868 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.868 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661175+0000", 2024-09-18T07:39:39.868 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.868 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.868 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.868 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.868 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661175+0000", 2024-09-18T07:39:39.868 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.868 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.869 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.869 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.869 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661177+0000", 2024-09-18T07:39:39.869 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.548e-06 2024-09-18T07:39:39.869 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.869 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.869 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.869 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661235+0000", 2024-09-18T07:39:39.869 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7877999999999997e-05 2024-09-18T07:39:39.869 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.869 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.869 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.869 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661244+0000", 2024-09-18T07:39:39.870 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2709999999999998e-06 2024-09-18T07:39:39.870 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.870 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.870 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.870 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661250+0000", 2024-09-18T07:39:39.870 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8619999999999999e-06 2024-09-18T07:39:39.870 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.870 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.870 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:39.871 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:39.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:39.872 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661288+0000", 2024-09-18T07:39:39.872 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8330000000000001e-05 2024-09-18T07:39:39.872 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.873 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.873 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.873 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.661300+0000", 2024-09-18T07:39:39.873 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1483e-05 2024-09-18T07:39:39.873 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.873 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.873 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.873 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852855+0000", 2024-09-18T07:39:39.873 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19155582199999999 2024-09-18T07:39:39.873 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.873 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.873 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.873 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852888+0000", 2024-09-18T07:39:39.874 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3049000000000002e-05 2024-09-18T07:39:39.874 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.874 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.874 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.874 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.874 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.874 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:22347 7.f e89/81 7:fe6074eb:::benchmark_data_smithi137_74180_object22346:head v 89'1397, mlcod=89'1395)", 2024-09-18T07:39:39.874 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.662928+0000", 2024-09-18T07:39:39.874 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 435.15538343899999, 2024-09-18T07:39:39.874 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19092352700000001, 2024-09-18T07:39:39.874 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.874 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.874 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.874 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.875 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.875 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.875 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662928+0000", 2024-09-18T07:39:39.875 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.875 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.875 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.875 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.875 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662928+0000", 2024-09-18T07:39:39.875 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.875 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.875 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.875 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.875 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662929+0000", 2024-09-18T07:39:39.876 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2500000000000004e-07 2024-09-18T07:39:39.876 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.876 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.876 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.876 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662989+0000", 2024-09-18T07:39:39.876 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0466e-05 2024-09-18T07:39:39.876 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.876 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.876 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.876 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.851940+0000", 2024-09-18T07:39:39.876 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18895100100000001 2024-09-18T07:39:39.876 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.877 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.877 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.877 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.851968+0000", 2024-09-18T07:39:39.877 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7668e-05 2024-09-18T07:39:39.877 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.877 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.877 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.877 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852054+0000", 2024-09-18T07:39:39.877 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6370000000000001e-05 2024-09-18T07:39:39.877 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.877 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.877 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.877 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.852082+0000", 2024-09-18T07:39:39.878 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7769000000000001e-05 2024-09-18T07:39:39.878 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.878 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.878 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.878 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.853810+0000", 2024-09-18T07:39:39.878 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001728168 2024-09-18T07:39:39.878 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.878 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.878 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.878 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.853851+0000", 2024-09-18T07:39:39.878 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1159999999999999e-05 2024-09-18T07:39:39.878 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.878 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.879 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.879 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.879 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.879 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22450 7.5 7:ad888e37:::benchmark_data_smithi137_74180_object22449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:39:39.879 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.977850+0000", 2024-09-18T07:39:39.879 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 434.84046153700001, 2024-09-18T07:39:39.879 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.077184682000000004, 2024-09-18T07:39:39.879 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.879 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.879 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.879 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:39.879 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:39:39.879 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:39.880 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22450 2024-09-18T07:39:39.880 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.880 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.880 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.880 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.880 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977850+0000", 2024-09-18T07:39:39.880 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.880 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.880 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.880 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.880 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977850+0000", 2024-09-18T07:39:39.880 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.880 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.881 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.881 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.881 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977852+0000", 2024-09-18T07:39:39.881 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.108e-06 2024-09-18T07:39:39.881 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.881 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.881 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.881 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977892+0000", 2024-09-18T07:39:39.881 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9928999999999997e-05 2024-09-18T07:39:39.881 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.881 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.881 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.881 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977904+0000", 2024-09-18T07:39:39.881 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2843999999999999e-05 2024-09-18T07:39:39.882 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.882 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.882 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.882 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977912+0000", 2024-09-18T07:39:39.882 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1169999999999998e-06 2024-09-18T07:39:39.882 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.882 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.882 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.882 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.977968+0000", 2024-09-18T07:39:39.882 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6338000000000002e-05 2024-09-18T07:39:39.882 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.882 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.883 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.883 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.978050+0000", 2024-09-18T07:39:39.883 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1841000000000005e-05 2024-09-18T07:39:39.883 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.883 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.883 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:39:39.883 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.978130+0000", 2024-09-18T07:39:39.883 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9917000000000001e-05 2024-09-18T07:39:39.883 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.883 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.883 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:39.883 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.978838+0000", 2024-09-18T07:39:39.883 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000708594 2024-09-18T07:39:39.883 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.884 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.884 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.884 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.979190+0000", 2024-09-18T07:39:39.884 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00035225000000000002 2024-09-18T07:39:39.884 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.884 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.884 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.884 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.054919+0000", 2024-09-18T07:39:39.884 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.075728917000000007 2024-09-18T07:39:39.884 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.884 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.884 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.884 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.054990+0000", 2024-09-18T07:39:39.885 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0300000000000001e-05 2024-09-18T07:39:39.885 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.885 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.885 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.885 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.055034+0000", 2024-09-18T07:39:39.885 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4527000000000002e-05 2024-09-18T07:39:39.885 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.885 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.885 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.885 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.885 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.885 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:23078 7.5 7:af53196f:::benchmark_data_smithi137_74180_object23077:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:39:39.885 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.454467+0000", 2024-09-18T07:39:39.886 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 434.36384418599999, 2024-09-18T07:39:39.886 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.088575311000000004, 2024-09-18T07:39:39.886 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.886 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.886 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.886 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:39.886 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:39:39.886 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:39.886 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 23078 2024-09-18T07:39:39.886 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.886 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.886 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.886 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.886 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454467+0000", 2024-09-18T07:39:39.887 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.887 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.887 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.887 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.887 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454467+0000", 2024-09-18T07:39:39.887 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.887 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.887 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.887 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.887 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454469+0000", 2024-09-18T07:39:39.887 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6089999999999999e-06 2024-09-18T07:39:39.887 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.888 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.888 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.888 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454512+0000", 2024-09-18T07:39:39.888 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2608999999999998e-05 2024-09-18T07:39:39.888 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.888 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.888 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.888 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454525+0000", 2024-09-18T07:39:39.888 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3352e-05 2024-09-18T07:39:39.888 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.888 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.888 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.888 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454534+0000", 2024-09-18T07:39:39.889 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0560000000000005e-06 2024-09-18T07:39:39.889 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.889 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.889 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.889 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454591+0000", 2024-09-18T07:39:39.889 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7633999999999997e-05 2024-09-18T07:39:39.889 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.889 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.889 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.889 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454715+0000", 2024-09-18T07:39:39.889 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012419300000000001 2024-09-18T07:39:39.889 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.889 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.890 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:39:39.890 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.454817+0000", 2024-09-18T07:39:39.890 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000101452 2024-09-18T07:39:39.890 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.890 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.890 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:39.890 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.455733+0000", 2024-09-18T07:39:39.890 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00091630599999999998 2024-09-18T07:39:39.890 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.890 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.890 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.890 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.456978+0000", 2024-09-18T07:39:39.890 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001245248 2024-09-18T07:39:39.891 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.891 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.891 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.891 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.542970+0000", 2024-09-18T07:39:39.891 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.085991206000000001 2024-09-18T07:39:39.891 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.891 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.891 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.891 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543010+0000", 2024-09-18T07:39:39.891 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0936999999999998e-05 2024-09-18T07:39:39.891 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.891 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.891 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.892 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543042+0000", 2024-09-18T07:39:39.892 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1708999999999997e-05 2024-09-18T07:39:39.892 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.892 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.892 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.892 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.892 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.892 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:23089 7.5 7:abb1cc32:::benchmark_data_smithi137_74180_object23088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-18T07:39:39.892 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:25.457769+0000", 2024-09-18T07:39:39.892 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 434.36054220800003, 2024-09-18T07:39:39.892 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.085838029999999996, 2024-09-18T07:39:39.892 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.892 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.893 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.893 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:39.893 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:39:39.893 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:39.893 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 23089 2024-09-18T07:39:39.893 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.893 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.893 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.893 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.893 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457769+0000", 2024-09-18T07:39:39.893 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.893 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.893 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.894 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.894 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457769+0000", 2024-09-18T07:39:39.894 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.894 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.894 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.894 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.894 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457771+0000", 2024-09-18T07:39:39.894 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3199999999999998e-06 2024-09-18T07:39:39.894 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.894 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.894 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.894 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457831+0000", 2024-09-18T07:39:39.894 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9939999999999999e-05 2024-09-18T07:39:39.895 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.895 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.895 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.895 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457848+0000", 2024-09-18T07:39:39.895 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6682999999999999e-05 2024-09-18T07:39:39.895 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.895 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.895 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.895 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457856+0000", 2024-09-18T07:39:39.895 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0809999999999994e-06 2024-09-18T07:39:39.895 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.895 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.895 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.896 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.457909+0000", 2024-09-18T07:39:39.896 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3143999999999998e-05 2024-09-18T07:39:39.896 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.896 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.896 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.896 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.458075+0000", 2024-09-18T07:39:39.896 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000166398 2024-09-18T07:39:39.896 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.896 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.896 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:39:39.896 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.458209+0000", 2024-09-18T07:39:39.896 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013364400000000001 2024-09-18T07:39:39.896 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.897 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.897 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:39.897 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.459558+0000", 2024-09-18T07:39:39.897 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0013487519999999999 2024-09-18T07:39:39.897 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.897 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.897 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.897 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.459987+0000", 2024-09-18T07:39:39.897 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00042892000000000001 2024-09-18T07:39:39.897 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.897 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.897 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.897 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543512+0000", 2024-09-18T07:39:39.898 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.083524980999999998 2024-09-18T07:39:39.898 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.898 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.898 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.898 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543569+0000", 2024-09-18T07:39:39.898 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7337999999999999e-05 2024-09-18T07:39:39.898 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.898 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.898 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.898 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:25.543607+0000", 2024-09-18T07:39:39.898 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7829e-05 2024-09-18T07:39:39.898 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.898 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.898 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.899 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.899 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.899 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:26891 7.5 7:a92f246c:::benchmark_data_smithi137_74180_object26890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-18T07:39:39.899 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:27.401622+0000", 2024-09-18T07:39:39.899 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 432.41668897900001, 2024-09-18T07:39:39.899 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.080096663999999998, 2024-09-18T07:39:39.899 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.899 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.899 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.899 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:39.900 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:39:39.900 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:39.900 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 26891 2024-09-18T07:39:39.900 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.900 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.900 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.900 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.900 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401622+0000", 2024-09-18T07:39:39.900 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.900 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.900 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.900 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.900 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401622+0000", 2024-09-18T07:39:39.901 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.901 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.901 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.901 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.901 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401624+0000", 2024-09-18T07:39:39.901 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0880000000000002e-06 2024-09-18T07:39:39.901 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.901 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.901 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.901 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401694+0000", 2024-09-18T07:39:39.901 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9825000000000003e-05 2024-09-18T07:39:39.901 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.902 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.902 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.902 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401708+0000", 2024-09-18T07:39:39.903 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4249e-05 2024-09-18T07:39:39.903 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.903 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.903 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.903 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401716+0000", 2024-09-18T07:39:39.903 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0460000000000006e-06 2024-09-18T07:39:39.903 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.904 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.904 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.904 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401785+0000", 2024-09-18T07:39:39.904 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8583000000000004e-05 2024-09-18T07:39:39.904 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.904 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.904 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.904 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.401892+0000", 2024-09-18T07:39:39.904 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000106885 2024-09-18T07:39:39.904 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.904 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.904 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:39:39.904 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402022+0000", 2024-09-18T07:39:39.905 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012980700000000001 2024-09-18T07:39:39.905 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.905 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.905 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:39.905 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.404167+0000", 2024-09-18T07:39:39.905 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021457550000000001 2024-09-18T07:39:39.905 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.905 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.905 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.905 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.404212+0000", 2024-09-18T07:39:39.905 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5052000000000002e-05 2024-09-18T07:39:39.905 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.905 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.906 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.906 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481614+0000", 2024-09-18T07:39:39.906 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.077401247000000006 2024-09-18T07:39:39.906 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.906 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.906 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.906 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481666+0000", 2024-09-18T07:39:39.906 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2198999999999999e-05 2024-09-18T07:39:39.906 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.906 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.906 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.906 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481719+0000", 2024-09-18T07:39:39.906 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2927999999999999e-05 2024-09-18T07:39:39.907 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.907 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.907 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.907 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.907 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.907 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:26893 7.5 7:a23e6cde:::benchmark_data_smithi137_74180_object26892:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-18T07:39:39.907 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:27.402662+0000", 2024-09-18T07:39:39.907 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 432.41564878299999, 2024-09-18T07:39:39.907 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.079194041000000007, 2024-09-18T07:39:39.907 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.907 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.907 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.907 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:39.908 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:39:39.908 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:39.908 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 26893 2024-09-18T07:39:39.908 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.908 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.908 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.908 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.908 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402662+0000", 2024-09-18T07:39:39.908 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.908 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.908 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.908 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.908 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402662+0000", 2024-09-18T07:39:39.909 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.909 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.909 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.909 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.909 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402664+0000", 2024-09-18T07:39:39.909 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.513e-06 2024-09-18T07:39:39.909 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.909 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.909 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.909 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402694+0000", 2024-09-18T07:39:39.909 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9984e-05 2024-09-18T07:39:39.909 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.909 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.910 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.910 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402706+0000", 2024-09-18T07:39:39.910 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2499e-05 2024-09-18T07:39:39.910 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.910 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.910 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.910 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402713+0000", 2024-09-18T07:39:39.910 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6429999999999998e-06 2024-09-18T07:39:39.910 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.910 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.910 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.910 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.402959+0000", 2024-09-18T07:39:39.910 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024601700000000001 2024-09-18T07:39:39.910 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.911 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.911 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.911 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.403094+0000", 2024-09-18T07:39:39.911 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013474299999999999 2024-09-18T07:39:39.911 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.911 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.911 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:39:39.911 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.403289+0000", 2024-09-18T07:39:39.911 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019528500000000001 2024-09-18T07:39:39.911 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.911 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.911 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:39.911 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.404306+0000", 2024-09-18T07:39:39.912 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010166179999999999 2024-09-18T07:39:39.912 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.912 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.912 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.912 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.405112+0000", 2024-09-18T07:39:39.912 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00080637099999999995 2024-09-18T07:39:39.912 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.912 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.912 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.912 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481779+0000", 2024-09-18T07:39:39.912 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.076667400999999996 2024-09-18T07:39:39.912 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.912 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.913 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.913 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481825+0000", 2024-09-18T07:39:39.913 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5652000000000003e-05 2024-09-18T07:39:39.913 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.913 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.913 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.913 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:27.481856+0000", 2024-09-18T07:39:39.913 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1315000000000001e-05 2024-09-18T07:39:39.913 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.913 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.913 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.913 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.913 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.913 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.34466.0:233 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2024-09-18T07:39:39.914 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:01.270598+0000", 2024-09-18T07:39:39.914 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 278.54771262200001, 2024-09-18T07:39:39.914 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14644679199999999, 2024-09-18T07:39:39.914 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.914 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.914 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:39:39.914 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:39.914 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.34466", 2024-09-18T07:39:39.914 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.203:0/952514737", 2024-09-18T07:39:39.914 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 233 2024-09-18T07:39:39.914 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.914 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.914 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.915 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.915 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270598+0000", 2024-09-18T07:39:39.915 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.915 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.915 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.915 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.915 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270598+0000", 2024-09-18T07:39:39.915 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.915 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.915 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.915 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.915 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270602+0000", 2024-09-18T07:39:39.915 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4960000000000001e-06 2024-09-18T07:39:39.916 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.916 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.916 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.916 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270613+0000", 2024-09-18T07:39:39.916 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0890000000000001e-05 2024-09-18T07:39:39.916 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.916 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.916 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.916 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270634+0000", 2024-09-18T07:39:39.916 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1484e-05 2024-09-18T07:39:39.916 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.916 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.916 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.917 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270643+0000", 2024-09-18T07:39:39.917 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1249999999999999e-06 2024-09-18T07:39:39.917 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.917 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.917 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.917 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270712+0000", 2024-09-18T07:39:39.918 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8733999999999996e-05 2024-09-18T07:39:39.918 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.918 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.918 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:39:39.918 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270720+0000", 2024-09-18T07:39:39.918 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0209999999999999e-06 2024-09-18T07:39:39.919 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.919 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.919 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.919 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.402460+0000", 2024-09-18T07:39:39.919 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13173957 2024-09-18T07:39:39.919 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.919 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.919 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for missing object", 2024-09-18T07:39:39.919 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.402586+0000", 2024-09-18T07:39:39.919 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012606100000000001 2024-09-18T07:39:39.919 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.919 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.919 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.919 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.416785+0000", 2024-09-18T07:39:39.920 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.014199145 2024-09-18T07:39:39.920 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.920 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.920 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.920 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.416882+0000", 2024-09-18T07:39:39.920 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6990999999999993e-05 2024-09-18T07:39:39.920 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.920 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.920 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.920 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.417045+0000", 2024-09-18T07:39:39.920 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000163275 2024-09-18T07:39:39.920 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.921 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.921 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.921 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.921 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.921 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:200515 7.5 7:a1f023bc:::benchmark_data_smithi137_74180_object96099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-09-18T07:39:39.921 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:36.399284+0000", 2024-09-18T07:39:39.921 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 243.41902753400001, 2024-09-18T07:39:39.921 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.065364955000000002, 2024-09-18T07:39:39.921 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.921 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.921 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.921 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:39:39.921 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:39:39.922 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:39:39.922 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 200515 2024-09-18T07:39:39.922 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.922 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.922 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.922 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.922 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399284+0000", 2024-09-18T07:39:39.922 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.922 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.922 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.922 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.922 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399284+0000", 2024-09-18T07:39:39.922 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.923 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.923 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.923 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.923 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399285+0000", 2024-09-18T07:39:39.923 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.347e-06 2024-09-18T07:39:39.923 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.923 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.923 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.923 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399289+0000", 2024-09-18T07:39:39.923 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0350000000000003e-06 2024-09-18T07:39:39.923 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.923 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.923 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.923 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399296+0000", 2024-09-18T07:39:39.924 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0639999999999996e-06 2024-09-18T07:39:39.924 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.924 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.924 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.924 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399302+0000", 2024-09-18T07:39:39.924 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3949999999999998e-06 2024-09-18T07:39:39.924 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.924 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.924 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.924 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399381+0000", 2024-09-18T07:39:39.924 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8978000000000002e-05 2024-09-18T07:39:39.924 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.924 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.925 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.925 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399493+0000", 2024-09-18T07:39:39.925 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000111155 2024-09-18T07:39:39.925 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.925 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.925 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:39:39.925 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.399605+0000", 2024-09-18T07:39:39.925 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000112465 2024-09-18T07:39:39.925 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.925 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.925 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:39:39.925 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.400747+0000", 2024-09-18T07:39:39.926 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001141677 2024-09-18T07:39:39.926 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.926 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.926 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.926 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.401647+0000", 2024-09-18T07:39:39.926 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00090012499999999999 2024-09-18T07:39:39.926 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.926 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.926 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:39:39.926 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.464535+0000", 2024-09-18T07:39:39.926 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.062888682000000001 2024-09-18T07:39:39.926 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.926 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.926 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.927 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.464585+0000", 2024-09-18T07:39:39.927 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9109e-05 2024-09-18T07:39:39.927 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.927 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.927 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.927 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:36.464648+0000", 2024-09-18T07:39:39.927 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3923000000000004e-05 2024-09-18T07:39:39.927 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.927 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.927 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.927 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.927 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.927 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:170411 8.d e264/201 8:bde4aff4:::benchmark_data_smithi137_101942_object65601:head v 264'10478, mlcod=264'10476)", 2024-09-18T07:39:39.928 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:44.767677+0000", 2024-09-18T07:39:39.928 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 55.050634162999998, 2024-09-18T07:39:39.928 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.143244022, 2024-09-18T07:39:39.928 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.928 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.928 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.928 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.928 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.928 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.928 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.767677+0000", 2024-09-18T07:39:39.928 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.928 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.928 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.929 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.929 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.767677+0000", 2024-09-18T07:39:39.929 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.929 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.929 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.929 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.929 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.767679+0000", 2024-09-18T07:39:39.929 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.798e-06 2024-09-18T07:39:39.929 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.929 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.929 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.929 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.767688+0000", 2024-09-18T07:39:39.929 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0170000000000002e-06 2024-09-18T07:39:39.930 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.930 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.930 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.930 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.767700+0000", 2024-09-18T07:39:39.930 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1793e-05 2024-09-18T07:39:39.930 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.930 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.930 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.930 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.767708+0000", 2024-09-18T07:39:39.930 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1510000000000004e-06 2024-09-18T07:39:39.930 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.930 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.930 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.930 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.767764+0000", 2024-09-18T07:39:39.931 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6073999999999999e-05 2024-09-18T07:39:39.931 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.931 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.931 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.931 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.767784+0000", 2024-09-18T07:39:39.931 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0483e-05 2024-09-18T07:39:39.931 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.931 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.931 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.931 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.910878+0000", 2024-09-18T07:39:39.931 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14309329200000001 2024-09-18T07:39:39.931 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.931 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.932 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.932 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.910921+0000", 2024-09-18T07:39:39.932 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3414000000000001e-05 2024-09-18T07:39:39.932 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.932 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.932 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.932 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.932 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.932 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:170414 8.5 e264/255 8:a29be62f:::benchmark_data_smithi137_101942_object65604:head v 264'10576, mlcod=264'10574)", 2024-09-18T07:39:39.932 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:44.768915+0000", 2024-09-18T07:39:39.932 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 55.049396024000004, 2024-09-18T07:39:39.932 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15600961199999999, 2024-09-18T07:39:39.932 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.932 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.933 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.933 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.933 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.933 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.933 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768915+0000", 2024-09-18T07:39:39.933 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.933 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.933 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.933 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.933 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768915+0000", 2024-09-18T07:39:39.933 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.933 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.933 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.934 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.934 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768916+0000", 2024-09-18T07:39:39.934 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.24e-06 2024-09-18T07:39:39.934 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.934 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.934 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.934 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768923+0000", 2024-09-18T07:39:39.934 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7410000000000003e-06 2024-09-18T07:39:39.934 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.934 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.934 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.934 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768936+0000", 2024-09-18T07:39:39.934 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.345e-05 2024-09-18T07:39:39.935 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.935 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.935 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.935 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768944+0000", 2024-09-18T07:39:39.935 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0299999999999996e-06 2024-09-18T07:39:39.935 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.935 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.935 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.935 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.770341+0000", 2024-09-18T07:39:39.935 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001397061 2024-09-18T07:39:39.935 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.935 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.935 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.935 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.770358+0000", 2024-09-18T07:39:39.936 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7825999999999998e-05 2024-09-18T07:39:39.936 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.936 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.936 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.936 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.924876+0000", 2024-09-18T07:39:39.936 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.154517618 2024-09-18T07:39:39.936 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.936 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.936 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.936 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.924925+0000", 2024-09-18T07:39:39.936 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8646000000000001e-05 2024-09-18T07:39:39.936 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.937 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.937 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.937 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.937 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.937 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:170415 8.c e264/201 8:341230f3:::benchmark_data_smithi137_101942_object65605:head v 264'10461, mlcod=264'10459)", 2024-09-18T07:39:39.937 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:44.772649+0000", 2024-09-18T07:39:39.937 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 55.045661611, 2024-09-18T07:39:39.937 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.154977485, 2024-09-18T07:39:39.937 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.937 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.937 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.937 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.937 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.937 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.938 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.772649+0000", 2024-09-18T07:39:39.938 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.938 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.938 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.938 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.938 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.772649+0000", 2024-09-18T07:39:39.938 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.938 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.938 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.938 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.938 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.772652+0000", 2024-09-18T07:39:39.938 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3300000000000001e-06 2024-09-18T07:39:39.939 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.939 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.939 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.939 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.772662+0000", 2024-09-18T07:39:39.939 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0524e-05 2024-09-18T07:39:39.939 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.939 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.939 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.939 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.911734+0000", 2024-09-18T07:39:39.939 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13907166200000001 2024-09-18T07:39:39.939 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.939 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.939 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.940 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.911745+0000", 2024-09-18T07:39:39.940 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0925e-05 2024-09-18T07:39:39.940 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.940 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.940 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.940 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.911798+0000", 2024-09-18T07:39:39.940 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3384000000000003e-05 2024-09-18T07:39:39.940 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.940 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.940 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.940 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.911818+0000", 2024-09-18T07:39:39.940 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9242999999999999e-05 2024-09-18T07:39:39.941 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.941 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.941 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.941 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.927424+0000", 2024-09-18T07:39:39.941 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.015606201 2024-09-18T07:39:39.941 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.941 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.941 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.941 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.927627+0000", 2024-09-18T07:39:39.941 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00020321599999999999 2024-09-18T07:39:39.942 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.942 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.942 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.942 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.942 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.942 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:170425 8.5 e264/255 8:a030dc7e:::benchmark_data_smithi137_101942_object65615:head v 264'10577, mlcod=264'10574)", 2024-09-18T07:39:39.942 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:44.779176+0000", 2024-09-18T07:39:39.942 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 55.039135446000003, 2024-09-18T07:39:39.942 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.145788631, 2024-09-18T07:39:39.942 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:39:39.942 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:39:39.942 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:39:39.943 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:39:39.943 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.943 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:39:39.943 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.779176+0000", 2024-09-18T07:39:39.943 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.943 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.943 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.943 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:39:39.943 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.779176+0000", 2024-09-18T07:39:39.943 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:39:39.943 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.943 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.943 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:39:39.944 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.779178+0000", 2024-09-18T07:39:39.944 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6759999999999999e-06 2024-09-18T07:39:39.944 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.944 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.944 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:39:39.944 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.779188+0000", 2024-09-18T07:39:39.944 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9320000000000008e-06 2024-09-18T07:39:39.944 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.944 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.944 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:39:39.944 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.911967+0000", 2024-09-18T07:39:39.944 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13277883200000001 2024-09-18T07:39:39.944 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.945 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.945 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:39:39.945 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.911973+0000", 2024-09-18T07:39:39.945 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0209999999999996e-06 2024-09-18T07:39:39.945 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.945 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.945 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:39:39.945 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.913718+0000", 2024-09-18T07:39:39.945 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0017446479999999999 2024-09-18T07:39:39.945 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.945 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.945 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:39:39.945 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.913742+0000", 2024-09-18T07:39:39.945 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4189000000000002e-05 2024-09-18T07:39:39.946 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.946 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.946 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:39:39.946 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.924931+0000", 2024-09-18T07:39:39.946 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.01118863 2024-09-18T07:39:39.946 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:39:39.946 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:39:39.946 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:39:39.946 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.924964+0000", 2024-09-18T07:39:39.946 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3702999999999999e-05 2024-09-18T07:39:39.946 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.946 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.946 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.947 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:39:39.947 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:39:39.947 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:39:39.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-18T07:39:39.973 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:40.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:40.018 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:40.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:40.072 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 5 15 14936 14921 186.471 171.312 0.00783238 0.0053569 2024-09-18T07:39:40.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:40.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:40.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:40.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:40.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:39 smithi137 ceph-mon[29408]: 5.b scrub starts 2024-09-18T07:39:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[29408]: 5.b scrub ok 2024-09-18T07:39:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[29408]: 1.8 scrub starts 2024-09-18T07:39:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[29408]: 1.8 scrub ok 2024-09-18T07:39:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[29408]: 3.b scrub starts 2024-09-18T07:39:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[29408]: 3.b scrub ok 2024-09-18T07:39:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:39:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[29408]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:39:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4269272467' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:39:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[29408]: osdmap e295: 8 total, 8 up, 8 in 2024-09-18T07:39:40.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:39 smithi137 ceph-mon[39175]: 5.b scrub starts 2024-09-18T07:39:40.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[39175]: 5.b scrub ok 2024-09-18T07:39:40.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[39175]: 1.8 scrub starts 2024-09-18T07:39:40.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[39175]: 1.8 scrub ok 2024-09-18T07:39:40.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[39175]: 3.b scrub starts 2024-09-18T07:39:40.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[39175]: 3.b scrub ok 2024-09-18T07:39:40.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:39:40.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[39175]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:39:40.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4269272467' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:39:40.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:40 smithi137 ceph-mon[39175]: osdmap e295: 8 total, 8 up, 8 in 2024-09-18T07:39:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:39 smithi203 ceph-mon[32237]: 5.b scrub starts 2024-09-18T07:39:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:39 smithi203 ceph-mon[32237]: 5.b scrub ok 2024-09-18T07:39:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:39 smithi203 ceph-mon[32237]: 1.8 scrub starts 2024-09-18T07:39:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:39 smithi203 ceph-mon[32237]: 1.8 scrub ok 2024-09-18T07:39:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:39 smithi203 ceph-mon[32237]: 3.b scrub starts 2024-09-18T07:39:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:39 smithi203 ceph-mon[32237]: 3.b scrub ok 2024-09-18T07:39:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:39 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:39:40.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:39 smithi203 ceph-mon[32237]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:39:40.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:39 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4269272467' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:39:40.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:39 smithi203 ceph-mon[32237]: osdmap e295: 8 total, 8 up, 8 in 2024-09-18T07:39:40.260 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:40.281 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:40.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:40.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:40.346 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:39:40.383 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:40.419 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:40.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:40.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:40.521 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:40.558 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:40.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:40.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:40.660 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:40.699 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:40.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:40.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:40.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:40.840 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:40.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:40.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:40.942 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:40.976 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:40.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:40.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:41.072 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 6 16 17581 17565 182.915 165.25 0.00533944 0.00546215 2024-09-18T07:39:41.078 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:41.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:41.117 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:41.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:41.218 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:41.224 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T07:39:41.245 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:39:41.254 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:41.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:41.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:41.261 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:41 smithi137 ceph-mon[29408]: pgmap v472: 158 pgs: 158 active+clean; 382 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 740 B/s rd, 55 MiB/s wr, 883 op/s 2024-09-18T07:39:41.262 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:41 smithi137 ceph-mon[39175]: pgmap v472: 158 pgs: 158 active+clean; 382 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 740 B/s rd, 55 MiB/s wr, 883 op/s 2024-09-18T07:39:41.356 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:41.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:41.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:41.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:41.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:41 smithi203 ceph-mon[32237]: pgmap v472: 158 pgs: 158 active+clean; 382 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 740 B/s rd, 55 MiB/s wr, 883 op/s 2024-09-18T07:39:41.511 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:41.531 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:41.540 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:41.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:41.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:41.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:41.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:41.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:41.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:41.791 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:41.827 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:41.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:41.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:41.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:41.968 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:41.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:41.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:42.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:42.073 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 7 16 20084 20068 179.127 156.438 0.00464516 0.00557698 2024-09-18T07:39:42.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:42.115 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:42.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:42.192 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:42 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:39:42.193 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:42 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:39:42.216 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:42.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:42.249 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:42.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:42.295 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:42.351 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:42.359 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:42.382 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:42.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:42.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:42.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:42 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:39:42.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:42.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:42.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:42.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:42.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:42.672 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:42.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:42.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:42.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:42.810 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:42.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:42.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:42.913 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:42.944 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:42.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:42.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:43.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:43.061 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.7 to 1 (10000) 2024-09-18T07:39:43.072 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 8 16 22548 22532 175.985 154 0.0136821 0.00567368 2024-09-18T07:39:43.087 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:43.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:43.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:43.189 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:43.224 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:43.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:43.258 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:39:43.258 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:39:43.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:43 smithi137 ceph-mon[29408]: pgmap v473: 158 pgs: 158 active+clean; 595 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-18T07:39:43.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:43 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3048404363' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:39:43.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:43 smithi137 ceph-mon[39175]: pgmap v473: 158 pgs: 158 active+clean; 595 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-18T07:39:43.291 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:43 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3048404363' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:39:43.296 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:39:43.296 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:39:43.325 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:43.330 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:39:43.330 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:39:43.362 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:39:43.363 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:39:43.363 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:43.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:43.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:43.394 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:39:43.394 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:39:43.426 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:39:43.427 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:39:43.459 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:39:43.459 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:39:43.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:43 smithi203 ceph-mon[32237]: pgmap v473: 158 pgs: 158 active+clean; 595 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-18T07:39:43.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:43 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3048404363' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:39:43.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:43.477 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:39:43.481 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:39:43.482 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:39:43.496 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:43.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:43.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:43.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:43.634 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:43.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:43.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:43.737 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:43.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:43.782 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:43.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:43.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:43.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:43.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:43.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:44.023 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:44.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:44.062 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:44.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:44.073 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 9 16 24873 24857 172.576 145.312 0.00727001 0.00578788 2024-09-18T07:39:44.163 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:44.183 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:44 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3048404363' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T07:39:44.183 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:44 smithi137 ceph-mon[29408]: osdmap e296: 8 total, 8 up, 8 in 2024-09-18T07:39:44.183 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:44 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3354448620' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:44.183 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3048404363' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T07:39:44.183 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:44 smithi137 ceph-mon[39175]: osdmap e296: 8 total, 8 up, 8 in 2024-09-18T07:39:44.183 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3354448620' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:44.197 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:44.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:44.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:44.238 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-18T07:39:44.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:44.343 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:44.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:44.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:44.445 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:44.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:44 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3048404363' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T07:39:44.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:44 smithi203 ceph-mon[32237]: osdmap e296: 8 total, 8 up, 8 in 2024-09-18T07:39:44.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:44 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3354448620' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:44.482 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:44.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:44.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:44.489 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:39:44.490 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:39:44.584 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:44.618 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:44.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:44.720 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:44.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:44.752 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:44.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:44.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:44.890 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:44.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:44.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:44.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:45.028 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:45.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:45.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:45.073 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 10 16 27240 27224 170.111 147.938 0.00596491 0.00587226 2024-09-18T07:39:45.130 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:45.164 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:45.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:45.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:45.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:45.299 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:45.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:45.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:45.401 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:45 smithi137 ceph-mon[29408]: pgmap v475: 158 pgs: 158 active+clean; 1.2 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 160 MiB/s wr, 2.56k op/s 2024-09-18T07:39:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:45 smithi137 ceph-mon[29408]: osdmap e297: 8 total, 8 up, 8 in 2024-09-18T07:39:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:45 smithi137 ceph-mon[39175]: pgmap v475: 158 pgs: 158 active+clean; 1.2 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 160 MiB/s wr, 2.56k op/s 2024-09-18T07:39:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:45 smithi137 ceph-mon[39175]: osdmap e297: 8 total, 8 up, 8 in 2024-09-18T07:39:45.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:45.437 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:45.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:45.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:45 smithi203 ceph-mon[32237]: pgmap v475: 158 pgs: 158 active+clean; 1.2 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 160 MiB/s wr, 2.56k op/s 2024-09-18T07:39:45.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:45 smithi203 ceph-mon[32237]: osdmap e297: 8 total, 8 up, 8 in 2024-09-18T07:39:45.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:45.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:45.586 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:45.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:45.687 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:45.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:45.721 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:45.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:45.822 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:45.857 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:45.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:45.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:45.959 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:45.994 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:45.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:46.073 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 11 16 29641 29625 168.288 150.062 0.00832432 0.00593699 2024-09-18T07:39:46.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:46.129 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:46.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:46.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:46.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:46.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:46.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:46.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:46.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:46.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:46.396 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:46.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:46.497 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:46.535 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:46.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:46.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:46.637 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:46.670 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:46.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:46.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:46.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:46.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:46.799 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:46.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:46.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:46.934 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:46.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:46.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:47.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:47.063 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:47.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:47.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:47.073 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 12 16 31449 31433 163.68 113 0.00248713 0.00609623 2024-09-18T07:39:47.165 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:47.202 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:47.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:47.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:47.304 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:47.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:47.331 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:47.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:47.353 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:47 smithi203 ceph-mon[32237]: pgmap v477: 158 pgs: 158 active+clean; 1.2 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-18T07:39:47.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:47 smithi137 ceph-mon[29408]: pgmap v477: 158 pgs: 158 active+clean; 1.2 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-18T07:39:47.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:39:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:39:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:39:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:47 smithi137 ceph-mon[39175]: pgmap v477: 158 pgs: 158 active+clean; 1.2 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-18T07:39:47.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:47.468 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:47.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:47.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:47.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:39:47.569 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:47.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:47.593 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:47.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:47.694 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:47.710 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:39:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:39:47.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:47.728 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:47.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:47.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:47.863 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:47.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:47.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:47.965 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:47.996 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:47.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:47.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:48.073 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 13 15 33186 33171 159.445 108.625 0.00693052 0.00626697 2024-09-18T07:39:48.098 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:48.131 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:48.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:48.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:48.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:48.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:48.260 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:48.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:48.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:48.394 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:48.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:48.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:48.496 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:48.531 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:48.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:48.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:48.633 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:48.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:48.666 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:48.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:48.768 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:48.805 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:48.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:48.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:48.907 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:48.940 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:48.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:49.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:49.073 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 14 16 35144 35128 156.792 122.312 0.00614276 0.00637243 2024-09-18T07:39:49.079 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:49.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:49.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:49.181 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:49.213 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:49.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:49.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:49.315 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:49.348 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:49.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:49.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:49.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:49 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:49.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:49 smithi137 ceph-mon[29408]: pgmap v478: 158 pgs: 158 active+clean; 1.9 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 196 MiB/s wr, 3.14k op/s 2024-09-18T07:39:49.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:49 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:49.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:49 smithi137 ceph-mon[39175]: pgmap v478: 158 pgs: 158 active+clean; 1.9 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 196 MiB/s wr, 3.14k op/s 2024-09-18T07:39:49.450 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:49.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:49 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:49.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:49 smithi203 ceph-mon[32237]: pgmap v478: 158 pgs: 158 active+clean; 1.9 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 196 MiB/s wr, 3.14k op/s 2024-09-18T07:39:49.476 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:49.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:49.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:49.578 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:49.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:49.610 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:49.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:49.711 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:49.750 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:49.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:49.851 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:49.884 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:49.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:49.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:49.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:50.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:50.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:50.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:50.073 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 15 16 37034 37018 154.214 118.125 0.00197447 0.00647625 2024-09-18T07:39:50.114 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:50.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:50.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:50.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:50.258 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:50.294 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:50.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:50.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:50.396 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:50.433 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:50.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:50.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:50.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:50.569 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:50.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:50.671 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:50.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:50.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:50.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:50.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:50.845 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:50.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:50.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:50.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T07:39:50.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:51.009 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:51.017 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:51.021 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:51.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:51.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:51.074 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 16 16 38998 38982 152.24 122.75 0.00868247 0.00656269 2024-09-18T07:39:51.124 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:51.139 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:51 smithi137 ceph-mon[29408]: pgmap v479: 158 pgs: 158 active+clean; 1.9 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 169 MiB/s wr, 2.71k op/s 2024-09-18T07:39:51.139 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:51 smithi137 ceph-mon[39175]: pgmap v479: 158 pgs: 158 active+clean; 1.9 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 169 MiB/s wr, 2.71k op/s 2024-09-18T07:39:51.160 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:51.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:51.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:51.306 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:51.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:51.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:51.408 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:51.437 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:51.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:51.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:51.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:51 smithi203 ceph-mon[32237]: pgmap v479: 158 pgs: 158 active+clean; 1.9 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 169 MiB/s wr, 2.71k op/s 2024-09-18T07:39:51.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:51.572 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:51.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:51.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:51.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:51.714 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:51.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:51.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:51.816 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:51.844 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:51.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:51.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:51.946 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:51.986 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:51.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:51.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:51.987 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:39:51.987 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:39:52.027 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:39:52.027 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:39:52.065 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:39:52.065 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:39:52.074 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 17 16 40755 40739 149.742 109.812 0.00534847 0.00667125 2024-09-18T07:39:52.087 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:52.102 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:39:52.102 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:39:52.107 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:52.112 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:52.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:52.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:52.128 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:52.140 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:39:52.141 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:39:52.174 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:39:52.175 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:39:52.214 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:39:52.214 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:39:52.214 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:52.244 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:52.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:52.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:52.255 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:39:52.255 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:39:52.256 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:52 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1390395562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:52.256 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:52 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1390395562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:52.347 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:52.383 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:52.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:52.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:52 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1390395562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:52.485 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:52.513 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:52.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:52.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:52.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:52.650 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:52.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:52.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:52.752 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:52.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:52.788 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:52.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:52.889 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:52.925 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:52.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:52.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:53.027 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:53.057 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:53.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:53.074 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 18 16 42438 42422 147.267 105.188 0.0372978 0.00677817 2024-09-18T07:39:53.115 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.6 to 1 (10000) 2024-09-18T07:39:53.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:53.193 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:53.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:53.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:53.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:53.330 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:53.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:53.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:53 smithi137 ceph-mon[29408]: pgmap v480: 158 pgs: 158 active+clean; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-18T07:39:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:53 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2150753340' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:39:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:53 smithi137 ceph-mon[39175]: pgmap v480: 158 pgs: 158 active+clean; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-18T07:39:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:53 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2150753340' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:39:53.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:53.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:53 smithi203 ceph-mon[32237]: pgmap v480: 158 pgs: 158 active+clean; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-18T07:39:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:53 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2150753340' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:39:53.466 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:53.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:53.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:53.567 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:53.602 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:53.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:53.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:53.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:53.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:53.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:53.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:53.874 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:53.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:53.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:53.976 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:54.009 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:54.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:54.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:54.074 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 19 16 44325 44309 145.723 117.938 0.00895322 0.00685718 2024-09-18T07:39:54.112 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:54.149 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:54.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:54.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:54.214 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:54 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:39:54.214 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:54 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:39:54.215 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:54 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2150753340' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:39:54.215 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:54 smithi137 ceph-mon[29408]: osdmap e298: 8 total, 8 up, 8 in 2024-09-18T07:39:54.215 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:54 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:39:54.215 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:54 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:39:54.215 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:54 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2150753340' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:39:54.215 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:54 smithi137 ceph-mon[39175]: osdmap e298: 8 total, 8 up, 8 in 2024-09-18T07:39:54.251 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:54.289 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:54.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:54.391 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:54.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:54.427 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:54.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:54.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:54 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:39:54.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:54 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:39:54.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:54 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2150753340' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:39:54.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:54 smithi203 ceph-mon[32237]: osdmap e298: 8 total, 8 up, 8 in 2024-09-18T07:39:54.527 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:54.565 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:54.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:54.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:54.667 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:54.699 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:54.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:54.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:54.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:54.823 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-18T07:39:54.823 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:39:54.842 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:54.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:54.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:54.944 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:54.957 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:39:55.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:55.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:55.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:55.074 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:39:55.073147+0000 min lat: 0.00110395 max lat: 0.124316 avg lat: 0.0069325 2024-09-18T07:39:55.075 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:39:55.075 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 20 16 46151 46135 144.142 114.125 0.0247242 0.0069325 2024-09-18T07:39:55.108 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:55.147 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:55.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:55.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:55.249 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:55.283 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:55.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:55.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:55.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:55.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:55 smithi137 ceph-mon[39175]: pgmap v482: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 150 MiB/s wr, 2.41k op/s 2024-09-18T07:39:55.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:55 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/260112461' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:55 smithi137 ceph-mon[39175]: from='client.35600 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:55 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:39:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:55 smithi137 ceph-mon[39175]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:39:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:55 smithi137 ceph-mon[39175]: osdmap e299: 8 total, 8 up, 8 in 2024-09-18T07:39:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:55 smithi137 ceph-mon[29408]: pgmap v482: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 150 MiB/s wr, 2.41k op/s 2024-09-18T07:39:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:55 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/260112461' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:55 smithi137 ceph-mon[29408]: from='client.35600 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:55 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:39:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:55 smithi137 ceph-mon[29408]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:39:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:55 smithi137 ceph-mon[29408]: osdmap e299: 8 total, 8 up, 8 in 2024-09-18T07:39:55.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:55.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:55.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:55 smithi203 ceph-mon[32237]: pgmap v482: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 150 MiB/s wr, 2.41k op/s 2024-09-18T07:39:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:55 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/260112461' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:55 smithi203 ceph-mon[32237]: from='client.35600 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:39:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:55 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:39:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:55 smithi203 ceph-mon[32237]: from='client.35600 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:39:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:55 smithi203 ceph-mon[32237]: osdmap e299: 8 total, 8 up, 8 in 2024-09-18T07:39:55.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:55.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:55.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:55.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:55.663 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:55.698 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:55.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:55.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:55.800 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:55.835 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:55.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:55.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:55.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:39:55.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:55.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:55.979 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:55.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:56.075 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 21 16 47957 47941 142.653 112.875 0.0044243 0.00700419 2024-09-18T07:39:56.080 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:56.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:56.119 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:56.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:56.220 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:39:56.264 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:56.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:56.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:39:56.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:56.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:56.413 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:56.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:56.422 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:39:56.422 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:39:56.422 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:39:56.422 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:39:56.423 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:39:56.514 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:56.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:56.550 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:56.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:56.651 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:56.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:56.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:56.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:56.791 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:56.827 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:56.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:56.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:56.929 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:56.961 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:56.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:56.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:57.064 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:57.075 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 22 16 49687 49671 141.083 108.125 0.00925661 0.00708327 2024-09-18T07:39:57.099 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:57.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:57.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:57.201 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:57.218 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:57.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T07:39:57.230 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:57.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:57.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:57.259 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:57.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:39:57.368 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:39:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:39:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:39:57.369 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:57 smithi137 ceph-mon[29408]: pgmap v484: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-18T07:39:57.369 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:57 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:39:57.369 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:57 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:39:57.369 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:57 smithi137 ceph-mon[39175]: pgmap v484: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-18T07:39:57.369 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:57 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:39:57.370 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:57 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:39:57.396 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:57.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:57.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:39:57.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:39:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:39:57.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:57 smithi203 ceph-mon[32237]: pgmap v484: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-18T07:39:57.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:57 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:39:57.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:57 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:39:57.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:57.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:57.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:57.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:57.649 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:57.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:57.697 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:57.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:57.798 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:57.858 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:57.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:57.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:57.959 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:57.991 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:57.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:57.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:58.075 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 23 16 51052 51036 138.658 85.3125 0.00746631 0.00720608 2024-09-18T07:39:58.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:58.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:58.134 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:58.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:58.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:39:58.224 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:58 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:39:58.224 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:58 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:39:58.224 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:58 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:39:58.224 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:58 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:39:58.225 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:58 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:39:58.225 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:58 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:39:58.225 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:58 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:39:58.225 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:58 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:39:58.225 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:58 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:39:58.225 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:58 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:39:58.235 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:58.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:58.325 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:58.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:39:58.355 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:39:58.355 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:39:58.369 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:39:58.375 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:39:58.408 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:39:58.408 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:39:58.426 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:58.450 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:39:58.450 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:39:58.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:58.455 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:58.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:58.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:58 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:39:58.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:58 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:39:58.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:58 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:39:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:58 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:39:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:58 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:39:58.491 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:39:58.491 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:39:58.529 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:39:58.530 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:39:58.557 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:58.563 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:39:58.563 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:39:58.586 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:58.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:58.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:58.596 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:39:58.596 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:39:58.630 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:39:58.631 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:39:58.689 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:58.726 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:58.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:58.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:58.828 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:58.861 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:58.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:58.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:58.963 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:39:58.997 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:58.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:39:59.075 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:39:59.075 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 24 16 52337 52321 136.227 80.3125 0.00630751 0.00733407 2024-09-18T07:39:59.099 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:59.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:59.127 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:59.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:59.228 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:39:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:59.262 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:59.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:39:59.363 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:39:59.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:59.400 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:59.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:39:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:59 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:59 smithi137 ceph-mon[29408]: pgmap v485: 158 pgs: 158 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-09-18T07:39:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:59 smithi137 ceph-mon[29408]: osdmap e300: 8 total, 8 up, 8 in 2024-09-18T07:39:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:59 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3209400082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:59.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:39:59 smithi137 ceph-mon[29408]: from='client.15753 v1:172.21.15.137:0/3239206711' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:39:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:59 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:59 smithi137 ceph-mon[39175]: pgmap v485: 158 pgs: 158 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-09-18T07:39:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:59 smithi137 ceph-mon[39175]: osdmap e300: 8 total, 8 up, 8 in 2024-09-18T07:39:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:59 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3209400082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:39:59 smithi137 ceph-mon[39175]: from='client.15753 v1:172.21.15.137:0/3239206711' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:39:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:59 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:39:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:59 smithi203 ceph-mon[32237]: pgmap v485: 158 pgs: 158 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-09-18T07:39:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:59 smithi203 ceph-mon[32237]: osdmap e300: 8 total, 8 up, 8 in 2024-09-18T07:39:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:59 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3209400082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:39:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:39:59 smithi203 ceph-mon[32237]: from='client.15753 v1:172.21.15.137:0/3239206711' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:39:59.502 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:59.548 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:59.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:59.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:59.650 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:39:59.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:59.682 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:59.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:39:59.718 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":486,"stamp":"2024-09-18T07:39:58.166847+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3236633115,"num_objects":49577,"num_object_clones":0,"num_object_copies":148731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49577,"num_whiteouts":0,"num_read":1005,"num_read_kb":757,"num_write":49814,"num_write_kb":3160237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":5625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50740,"ondisk_log_size":50740,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":479,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12632688,"kb_used_data":10074396,"kb_used_omap":12,"kb_used_meta":2558067,"kb_avail":737328528,"statfs":{"total":767960285184,"available":755024412672,"internally_reserved":0,"allocated":10316181504,"data_stored":10311572745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2619461168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,8,6,8],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1038548992,"num_objects":15847,"num_object_clones":0,"num_object_copies":47541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15847,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":15847,"num_write_kb":1014208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.556528"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":512,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.965946+0000","last_change":"2024-09-18T07:39:36.700902+0000","last_active":"2024-09-18T07:39:54.965946+0000","last_peered":"2024-09-18T07:39:54.965946+0000","last_clean":"2024-09-18T07:39:54.965946+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:39:54.965946+0000","last_undegraded":"2024-09-18T07:39:54.965946+0000","last_fullsized":"2024-09-18T07:39:54.965946+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:36.700832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:36.700832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:57.523488+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":578,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.948901+0000","last_change":"2024-09-18T07:38:52.141085+0000","last_active":"2024-09-18T07:39:54.948901+0000","last_peered":"2024-09-18T07:39:54.948901+0000","last_clean":"2024-09-18T07:39:54.948901+0000","last_became_active":"2024-09-18T07:37:43.061125+0000","last_became_peered":"2024-09-18T07:37:43.061125+0000","last_unstale":"2024-09-18T07:39:54.948901+0000","last_undegraded":"2024-09-18T07:39:54.948901+0000","last_fullsized":"2024-09-18T07:39:54.948901+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:38:52.140959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:38:52.140959+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:17.219305+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":447,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.956302+0000","last_change":"2024-09-18T07:36:38.570405+0000","last_active":"2024-09-18T07:39:54.956302+0000","last_peered":"2024-09-18T07:39:54.956302+0000","last_clean":"2024-09-18T07:39:54.956302+0000","last_became_active":"2024-09-18T07:36:13.128805+0000","last_became_peered":"2024-09-18T07:36:13.128805+0000","last_unstale":"2024-09-18T07:39:54.956302+0000","last_undegraded":"2024-09-18T07:39:54.956302+0000","last_fullsized":"2024-09-18T07:39:54.956302+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:38.570339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:38.570339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":487,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.949384+0000","last_change":"2024-09-18T07:32:03.953881+0000","last_active":"2024-09-18T07:39:54.949384+0000","last_peered":"2024-09-18T07:39:54.949384+0000","last_clean":"2024-09-18T07:39:54.949384+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:39:54.949384+0000","last_undegraded":"2024-09-18T07:39:54.949384+0000","last_fullsized":"2024-09-18T07:39:54.949384+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.953827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.953827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:44.967981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":406,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.963734+0000","last_change":"2024-09-18T07:39:33.737527+0000","last_active":"2024-09-18T07:39:54.963734+0000","last_peered":"2024-09-18T07:39:54.963734+0000","last_clean":"2024-09-18T07:39:54.963734+0000","last_became_active":"2024-09-18T07:37:43.056380+0000","last_became_peered":"2024-09-18T07:37:43.056380+0000","last_unstale":"2024-09-18T07:39:54.963734+0000","last_undegraded":"2024-09-18T07:39:54.963734+0000","last_fullsized":"2024-09-18T07:39:54.963734+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:33.737478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:33.737478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:52.567818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":513,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.960643+0000","last_change":"2024-09-18T07:37:43.066814+0000","last_active":"2024-09-18T07:39:54.960643+0000","last_peered":"2024-09-18T07:39:54.960643+0000","last_clean":"2024-09-18T07:39:54.960643+0000","last_became_active":"2024-09-18T07:37:43.066508+0000","last_became_peered":"2024-09-18T07:37:43.066508+0000","last_unstale":"2024-09-18T07:39:54.960643+0000","last_undegraded":"2024-09-18T07:39:54.960643+0000","last_fullsized":"2024-09-18T07:39:54.960643+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.006488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.006488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:46.846955+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":519,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.963774+0000","last_change":"2024-09-18T07:33:22.032321+0000","last_active":"2024-09-18T07:39:54.963774+0000","last_peered":"2024-09-18T07:39:54.963774+0000","last_clean":"2024-09-18T07:39:54.963774+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:39:54.963774+0000","last_undegraded":"2024-09-18T07:39:54.963774+0000","last_fullsized":"2024-09-18T07:39:54.963774+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:22.032110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.032110+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:47.872878+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":513,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.948800+0000","last_change":"2024-09-18T07:37:43.052675+0000","last_active":"2024-09-18T07:39:54.948800+0000","last_peered":"2024-09-18T07:39:54.948800+0000","last_clean":"2024-09-18T07:39:54.948800+0000","last_became_active":"2024-09-18T07:37:43.052346+0000","last_became_peered":"2024-09-18T07:37:43.052346+0000","last_unstale":"2024-09-18T07:39:54.948800+0000","last_undegraded":"2024-09-18T07:39:54.948800+0000","last_fullsized":"2024-09-18T07:39:54.948800+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:30.995282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:30.995282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:11.031402+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":519,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.957402+0000","last_change":"2024-09-18T07:37:43.080879+0000","last_active":"2024-09-18T07:39:54.957402+0000","last_peered":"2024-09-18T07:39:54.957402+0000","last_clean":"2024-09-18T07:39:54.957402+0000","last_became_active":"2024-09-18T07:37:43.080314+0000","last_became_peered":"2024-09-18T07:37:43.080314+0000","last_unstale":"2024-09-18T07:39:54.957402+0000","last_undegraded":"2024-09-18T07:39:54.957402+0000","last_fullsized":"2024-09-18T07:39:54.957402+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:00.392466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.392466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:20.640427+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":511,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.949677+0000","last_change":"2024-09-18T07:32:02.002919+0000","last_active":"2024-09-18T07:39:54.949677+0000","last_peered":"2024-09-18T07:39:54.949677+0000","last_clean":"2024-09-18T07:39:54.949677+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:39:54.949677+0000","last_undegraded":"2024-09-18T07:39:54.949677+0000","last_fullsized":"2024-09-18T07:39:54.949677+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.002811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.002811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:43.931181+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":503,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.956628+0000","last_change":"2024-09-18T07:35:04.267623+0000","last_active":"2024-09-18T07:39:54.956628+0000","last_peered":"2024-09-18T07:39:54.956628+0000","last_clean":"2024-09-18T07:39:54.956628+0000","last_became_active":"2024-09-18T07:35:01.413443+0000","last_became_peered":"2024-09-18T07:35:01.413443+0000","last_unstale":"2024-09-18T07:39:54.956628+0000","last_undegraded":"2024-09-18T07:39:54.956628+0000","last_fullsized":"2024-09-18T07:39:54.956628+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:04.267580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:04.267580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:22.587151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":497,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.969127+0000","last_change":"2024-09-18T07:36:10.551960+0000","last_active":"2024-09-18T07:39:54.969127+0000","last_peered":"2024-09-18T07:39:54.969127+0000","last_clean":"2024-09-18T07:39:54.969127+0000","last_became_active":"2024-09-18T07:35:01.410284+0000","last_became_peered":"2024-09-18T07:35:01.410284+0000","last_unstale":"2024-09-18T07:39:54.969127+0000","last_undegraded":"2024-09-18T07:39:54.969127+0000","last_fullsized":"2024-09-18T07:39:54.969127+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:10.551889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:10.551889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":489,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.956872+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:39:54.956872+0000","last_peered":"2024-09-18T07:39:54.956872+0000","last_clean":"2024-09-18T07:39:54.956872+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:39:54.956872+0000","last_undegraded":"2024-09-18T07:39:54.956872+0000","last_fullsized":"2024-09-18T07:39:54.956872+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:10.625350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":526,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.949276+0000","last_change":"2024-09-18T07:39:26.028935+0000","last_active":"2024-09-18T07:39:54.949276+0000","last_peered":"2024-09-18T07:39:54.949276+0000","last_clean":"2024-09-18T07:39:54.949276+0000","last_became_active":"2024-09-18T07:36:02.526212+0000","last_became_peered":"2024-09-18T07:36:02.526212+0000","last_unstale":"2024-09-18T07:39:54.949276+0000","last_undegraded":"2024-09-18T07:39:54.949276+0000","last_fullsized":"2024-09-18T07:39:54.949276+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-18T07:39:26.028884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:26.028884+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:28.305457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":271,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.955141+0000","last_change":"2024-09-18T07:38:54.458848+0000","last_active":"2024-09-18T07:39:54.955141+0000","last_peered":"2024-09-18T07:39:54.955141+0000","last_clean":"2024-09-18T07:39:54.955141+0000","last_became_active":"2024-09-18T07:35:01.398605+0000","last_became_peered":"2024-09-18T07:35:01.398605+0000","last_unstale":"2024-09-18T07:39:54.955141+0000","last_undegraded":"2024-09-18T07:39:54.955141+0000","last_fullsized":"2024-09-18T07:39:54.955141+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:54.458787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:54.458787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.620791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":509,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.956348+0000","last_change":"2024-09-18T07:39:30.429935+0000","last_active":"2024-09-18T07:39:54.956348+0000","last_peered":"2024-09-18T07:39:54.956348+0000","last_clean":"2024-09-18T07:39:54.956348+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:39:54.956348+0000","last_undegraded":"2024-09-18T07:39:54.956348+0000","last_fullsized":"2024-09-18T07:39:54.956348+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:30.429890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:30.429890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:38.713228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":518,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.951573+0000","last_change":"2024-09-18T07:38:56.096769+0000","last_active":"2024-09-18T07:39:54.951573+0000","last_peered":"2024-09-18T07:39:54.951573+0000","last_clean":"2024-09-18T07:39:54.951573+0000","last_became_active":"2024-09-18T07:37:43.067568+0000","last_became_peered":"2024-09-18T07:37:43.067568+0000","last_unstale":"2024-09-18T07:39:54.951573+0000","last_undegraded":"2024-09-18T07:39:54.951573+0000","last_fullsized":"2024-09-18T07:39:54.951573+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:56.096591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:56.096591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:24.052212+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":517,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.968944+0000","last_change":"2024-09-18T07:36:06.523357+0000","last_active":"2024-09-18T07:39:54.968944+0000","last_peered":"2024-09-18T07:39:54.968944+0000","last_clean":"2024-09-18T07:39:54.968944+0000","last_became_active":"2024-09-18T07:35:01.414088+0000","last_became_peered":"2024-09-18T07:35:01.414088+0000","last_unstale":"2024-09-18T07:39:54.968944+0000","last_undegraded":"2024-09-18T07:39:54.968944+0000","last_fullsized":"2024-09-18T07:39:54.968944+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:06.523292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.523292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":574,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.964680+0000","last_change":"2024-09-18T07:39:04.314241+0000","last_active":"2024-09-18T07:39:54.964680+0000","last_peered":"2024-09-18T07:39:54.964680+0000","last_clean":"2024-09-18T07:39:54.964680+0000","last_became_active":"2024-09-18T07:37:28.002858+0000","last_became_peered":"2024-09-18T07:37:28.002858+0000","last_unstale":"2024-09-18T07:39:54.964680+0000","last_undegraded":"2024-09-18T07:39:54.964680+0000","last_fullsized":"2024-09-18T07:39:54.964680+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:39:04.313784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:04.313784+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:09.030441+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":190,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.950415+0000","last_change":"2024-09-18T07:36:13.116882+0000","last_active":"2024-09-18T07:39:54.950415+0000","last_peered":"2024-09-18T07:39:54.950415+0000","last_clean":"2024-09-18T07:39:54.950415+0000","last_became_active":"2024-09-18T07:36:13.116416+0000","last_became_peered":"2024-09-18T07:36:13.116416+0000","last_unstale":"2024-09-18T07:39:54.950415+0000","last_undegraded":"2024-09-18T07:39:54.950415+0000","last_fullsized":"2024-09-18T07:39:54.950415+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:10.074367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:10.074367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:21.035466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":509,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.960850+0000","last_change":"2024-09-18T07:37:43.067816+0000","last_active":"2024-09-18T07:39:54.960850+0000","last_peered":"2024-09-18T07:39:54.960850+0000","last_clean":"2024-09-18T07:39:54.960850+0000","last_became_active":"2024-09-18T07:37:43.067493+0000","last_became_peered":"2024-09-18T07:37:43.067493+0000","last_unstale":"2024-09-18T07:39:54.960850+0000","last_undegraded":"2024-09-18T07:39:54.960850+0000","last_fullsized":"2024-09-18T07:39:54.960850+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:34.992350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:34.992350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:11.064160+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":515,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.949742+0000","last_change":"2024-09-18T07:39:00.130395+0000","last_active":"2024-09-18T07:39:54.949742+0000","last_peered":"2024-09-18T07:39:54.949742+0000","last_clean":"2024-09-18T07:39:54.949742+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:39:54.949742+0000","last_undegraded":"2024-09-18T07:39:54.949742+0000","last_fullsized":"2024-09-18T07:39:54.949742+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:00.130344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.130344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:22.555080+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":160,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.964931+0000","last_change":"2024-09-18T07:37:43.756858+0000","last_active":"2024-09-18T07:39:54.964931+0000","last_peered":"2024-09-18T07:39:54.964931+0000","last_clean":"2024-09-18T07:39:54.964931+0000","last_became_active":"2024-09-18T07:37:27.981705+0000","last_became_peered":"2024-09-18T07:37:27.981705+0000","last_unstale":"2024-09-18T07:39:54.964931+0000","last_undegraded":"2024-09-18T07:39:54.964931+0000","last_fullsized":"2024-09-18T07:39:54.964931+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:37:43.756403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:37:43.756403+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:53.387865+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":491,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.950307+0000","last_change":"2024-09-18T07:39:37.930497+0000","last_active":"2024-09-18T07:39:54.950307+0000","last_peered":"2024-09-18T07:39:54.950307+0000","last_clean":"2024-09-18T07:39:54.950307+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:39:54.950307+0000","last_undegraded":"2024-09-18T07:39:54.950307+0000","last_fullsized":"2024-09-18T07:39:54.950307+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.930435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.930435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:55.595621+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":283,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.955320+0000","last_change":"2024-09-18T07:38:56.497595+0000","last_active":"2024-09-18T07:39:54.955320+0000","last_peered":"2024-09-18T07:39:54.955320+0000","last_clean":"2024-09-18T07:39:54.955320+0000","last_became_active":"2024-09-18T07:35:01.413110+0000","last_became_peered":"2024-09-18T07:35:01.413110+0000","last_unstale":"2024-09-18T07:39:54.955320+0000","last_undegraded":"2024-09-18T07:39:54.955320+0000","last_fullsized":"2024-09-18T07:39:54.955320+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:38:56.497544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:56.497544+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:05.310758+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":207,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.963633+0000","last_change":"2024-09-18T07:38:50.918758+0000","last_active":"2024-09-18T07:39:54.963633+0000","last_peered":"2024-09-18T07:39:54.963633+0000","last_clean":"2024-09-18T07:39:54.963633+0000","last_became_active":"2024-09-18T07:36:13.116295+0000","last_became_peered":"2024-09-18T07:36:13.116295+0000","last_unstale":"2024-09-18T07:39:54.963633+0000","last_undegraded":"2024-09-18T07:39:54.963633+0000","last_fullsized":"2024-09-18T07:39:54.963633+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:38:50.918686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:38:50.918686+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:19.981998+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":553,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.964442+0000","last_change":"2024-09-18T07:35:57.488019+0000","last_active":"2024-09-18T07:39:54.964442+0000","last_peered":"2024-09-18T07:39:54.964442+0000","last_clean":"2024-09-18T07:39:54.964442+0000","last_became_active":"2024-09-18T07:35:01.410926+0000","last_became_peered":"2024-09-18T07:35:01.410926+0000","last_unstale":"2024-09-18T07:39:54.964442+0000","last_undegraded":"2024-09-18T07:39:54.964442+0000","last_fullsized":"2024-09-18T07:39:54.964442+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:35:57.487960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.487960+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:52.681351+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"240'12","reported_seq":170,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.954030+0000","last_change":"2024-09-18T07:39:33.979364+0000","last_active":"2024-09-18T07:39:54.954030+0000","last_peered":"2024-09-18T07:39:54.954030+0000","last_clean":"2024-09-18T07:39:54.954030+0000","last_became_active":"2024-09-18T07:37:43.060715+0000","last_became_peered":"2024-09-18T07:37:43.060715+0000","last_unstale":"2024-09-18T07:39:54.954030+0000","last_undegraded":"2024-09-18T07:39:54.954030+0000","last_fullsized":"2024-09-18T07:39:54.954030+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:39:33.979291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:33.979291+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:39.389550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"299'3176","reported_seq":3207,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:57.079791+0000","last_change":"2024-09-18T07:39:29.700748+0000","last_active":"2024-09-18T07:39:57.079791+0000","last_peered":"2024-09-18T07:39:57.079791+0000","last_clean":"2024-09-18T07:39:57.079791+0000","last_became_active":"2024-09-18T07:39:29.700477+0000","last_became_peered":"2024-09-18T07:39:29.700477+0000","last_unstale":"2024-09-18T07:39:57.079791+0000","last_undegraded":"2024-09-18T07:39:57.079791+0000","last_fullsized":"2024-09-18T07:39:57.079791+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3176,"log_dups_size":0,"ondisk_log_size":3176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.879265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208142336,"num_objects":3176,"num_object_clones":0,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3176,"num_write_kb":203264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":68,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.956420+0000","last_change":"2024-09-18T07:39:06.546309+0000","last_active":"2024-09-18T07:39:54.956420+0000","last_peered":"2024-09-18T07:39:54.956420+0000","last_clean":"2024-09-18T07:39:54.956420+0000","last_became_active":"2024-09-18T07:39:04.962047+0000","last_became_peered":"2024-09-18T07:39:04.962047+0000","last_unstale":"2024-09-18T07:39:54.956420+0000","last_undegraded":"2024-09-18T07:39:54.956420+0000","last_fullsized":"2024-09-18T07:39:54.956420+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:06.546237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:06.546237+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:25.791546+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":243,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.961158+0000","last_change":"2024-09-18T07:38:54.967706+0000","last_active":"2024-09-18T07:39:54.961158+0000","last_peered":"2024-09-18T07:39:54.961158+0000","last_clean":"2024-09-18T07:39:54.961158+0000","last_became_active":"2024-09-18T07:36:13.119098+0000","last_became_peered":"2024-09-18T07:36:13.119098+0000","last_unstale":"2024-09-18T07:39:54.961158+0000","last_undegraded":"2024-09-18T07:39:54.961158+0000","last_fullsized":"2024-09-18T07:39:54.961158+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:38:54.967643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:38:54.967643+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:18.692985+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":537,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.960804+0000","last_change":"2024-09-18T07:39:29.049261+0000","last_active":"2024-09-18T07:39:54.960804+0000","last_peered":"2024-09-18T07:39:54.960804+0000","last_clean":"2024-09-18T07:39:54.960804+0000","last_became_active":"2024-09-18T07:35:01.407339+0000","last_became_peered":"2024-09-18T07:35:01.407339+0000","last_unstale":"2024-09-18T07:39:54.960804+0000","last_undegraded":"2024-09-18T07:39:54.960804+0000","last_fullsized":"2024-09-18T07:39:54.960804+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.049209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.049209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:31.230030+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":624,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.951787+0000","last_change":"2024-09-18T07:36:35.210213+0000","last_active":"2024-09-18T07:39:54.951787+0000","last_peered":"2024-09-18T07:39:54.951787+0000","last_clean":"2024-09-18T07:39:54.951787+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:39:54.951787+0000","last_undegraded":"2024-09-18T07:39:54.951787+0000","last_fullsized":"2024-09-18T07:39:54.951787+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:35.209823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:35.209823+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":493,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.956870+0000","last_change":"2024-09-18T07:36:07.588483+0000","last_active":"2024-09-18T07:39:54.956870+0000","last_peered":"2024-09-18T07:39:54.956870+0000","last_clean":"2024-09-18T07:39:54.956870+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:39:54.956870+0000","last_undegraded":"2024-09-18T07:39:54.956870+0000","last_fullsized":"2024-09-18T07:39:54.956870+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:07.588440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:07.588440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"299'3112","reported_seq":3143,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:56.503731+0000","last_change":"2024-09-18T07:39:29.704994+0000","last_active":"2024-09-18T07:39:56.503731+0000","last_peered":"2024-09-18T07:39:56.503731+0000","last_clean":"2024-09-18T07:39:56.503731+0000","last_became_active":"2024-09-18T07:39:29.704590+0000","last_became_peered":"2024-09-18T07:39:29.704590+0000","last_unstale":"2024-09-18T07:39:56.503731+0000","last_undegraded":"2024-09-18T07:39:56.503731+0000","last_fullsized":"2024-09-18T07:39:56.503731+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3112,"log_dups_size":0,"ondisk_log_size":3112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:33.304050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203948032,"num_objects":3112,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3112,"num_write_kb":199168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":68,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.950788+0000","last_change":"2024-09-18T07:39:05.085099+0000","last_active":"2024-09-18T07:39:54.950788+0000","last_peered":"2024-09-18T07:39:54.950788+0000","last_clean":"2024-09-18T07:39:54.950788+0000","last_became_active":"2024-09-18T07:39:04.955645+0000","last_became_peered":"2024-09-18T07:39:04.955645+0000","last_unstale":"2024-09-18T07:39:54.950788+0000","last_undegraded":"2024-09-18T07:39:54.950788+0000","last_fullsized":"2024-09-18T07:39:54.950788+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.084956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.084956+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:18.027466+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":593,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.960123+0000","last_change":"2024-09-18T07:39:00.919075+0000","last_active":"2024-09-18T07:39:54.960123+0000","last_peered":"2024-09-18T07:39:54.960123+0000","last_clean":"2024-09-18T07:39:54.960123+0000","last_became_active":"2024-09-18T07:35:01.418909+0000","last_became_peered":"2024-09-18T07:35:01.418909+0000","last_unstale":"2024-09-18T07:39:54.960123+0000","last_undegraded":"2024-09-18T07:39:54.960123+0000","last_fullsized":"2024-09-18T07:39:54.960123+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:39:00.918860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.918860+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:29.657171+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":529,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.968839+0000","last_change":"2024-09-18T07:39:06.987818+0000","last_active":"2024-09-18T07:39:54.968839+0000","last_peered":"2024-09-18T07:39:54.968839+0000","last_clean":"2024-09-18T07:39:54.968839+0000","last_became_active":"2024-09-18T07:35:01.407422+0000","last_became_peered":"2024-09-18T07:35:01.407422+0000","last_unstale":"2024-09-18T07:39:54.968839+0000","last_undegraded":"2024-09-18T07:39:54.968839+0000","last_fullsized":"2024-09-18T07:39:54.968839+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:06.987428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:06.987428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:25.403434+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":514,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.948795+0000","last_change":"2024-09-18T07:36:30.242479+0000","last_active":"2024-09-18T07:39:54.948795+0000","last_peered":"2024-09-18T07:39:54.948795+0000","last_clean":"2024-09-18T07:39:54.948795+0000","last_became_active":"2024-09-18T07:35:01.401024+0000","last_became_peered":"2024-09-18T07:35:01.401024+0000","last_unstale":"2024-09-18T07:39:54.948795+0000","last_undegraded":"2024-09-18T07:39:54.948795+0000","last_fullsized":"2024-09-18T07:39:54.948795+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:30.242420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:30.242420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":480,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.949584+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:39:54.949584+0000","last_peered":"2024-09-18T07:39:54.949584+0000","last_clean":"2024-09-18T07:39:54.949584+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:39:54.949584+0000","last_undegraded":"2024-09-18T07:39:54.949584+0000","last_fullsized":"2024-09-18T07:39:54.949584+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:41.905293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"299'3200","reported_seq":3231,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:57.094749+0000","last_change":"2024-09-18T07:39:29.703301+0000","last_active":"2024-09-18T07:39:57.094749+0000","last_peered":"2024-09-18T07:39:57.094749+0000","last_clean":"2024-09-18T07:39:57.094749+0000","last_became_active":"2024-09-18T07:39:29.702953+0000","last_became_peered":"2024-09-18T07:39:29.702953+0000","last_unstale":"2024-09-18T07:39:57.094749+0000","last_undegraded":"2024-09-18T07:39:57.094749+0000","last_fullsized":"2024-09-18T07:39:57.094749+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3200,"log_dups_size":0,"ondisk_log_size":3200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.620109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209715200,"num_objects":3200,"num_object_clones":0,"num_object_copies":9600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3200,"num_write_kb":204800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":73,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.962164+0000","last_change":"2024-09-18T07:39:05.959886+0000","last_active":"2024-09-18T07:39:54.962164+0000","last_peered":"2024-09-18T07:39:54.962164+0000","last_clean":"2024-09-18T07:39:54.962164+0000","last_became_active":"2024-09-18T07:39:04.945385+0000","last_became_peered":"2024-09-18T07:39:04.945385+0000","last_unstale":"2024-09-18T07:39:54.962164+0000","last_undegraded":"2024-09-18T07:39:54.962164+0000","last_fullsized":"2024-09-18T07:39:54.962164+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.959822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.959822+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:34.831011+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":563,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.948844+0000","last_change":"2024-09-18T07:37:43.153068+0000","last_active":"2024-09-18T07:39:54.948844+0000","last_peered":"2024-09-18T07:39:54.948844+0000","last_clean":"2024-09-18T07:39:54.948844+0000","last_became_active":"2024-09-18T07:37:43.096236+0000","last_became_peered":"2024-09-18T07:37:43.096236+0000","last_unstale":"2024-09-18T07:39:54.948844+0000","last_undegraded":"2024-09-18T07:39:54.948844+0000","last_fullsized":"2024-09-18T07:39:54.948844+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-18T07:35:00.882191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.882191+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":505,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.964718+0000","last_change":"2024-09-18T07:39:28.750779+0000","last_active":"2024-09-18T07:39:54.964718+0000","last_peered":"2024-09-18T07:39:54.964718+0000","last_clean":"2024-09-18T07:39:54.964718+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:39:54.964718+0000","last_undegraded":"2024-09-18T07:39:54.964718+0000","last_fullsized":"2024-09-18T07:39:54.964718+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:39:28.750720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.750720+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:52.195932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":616,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.962081+0000","last_change":"2024-09-18T07:36:05.016008+0000","last_active":"2024-09-18T07:39:54.962081+0000","last_peered":"2024-09-18T07:39:54.962081+0000","last_clean":"2024-09-18T07:39:54.962081+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:39:54.962081+0000","last_undegraded":"2024-09-18T07:39:54.962081+0000","last_fullsized":"2024-09-18T07:39:54.962081+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:05.015955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.015955+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:48.836489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":294,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.955075+0000","last_change":"2024-09-18T07:38:58.528134+0000","last_active":"2024-09-18T07:39:54.955075+0000","last_peered":"2024-09-18T07:39:54.955075+0000","last_clean":"2024-09-18T07:39:54.955075+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:39:54.955075+0000","last_undegraded":"2024-09-18T07:39:54.955075+0000","last_fullsized":"2024-09-18T07:39:54.955075+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:58.528059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:58.528059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:10.915828+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"299'3075","reported_seq":3106,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:56.802738+0000","last_change":"2024-09-18T07:39:29.702619+0000","last_active":"2024-09-18T07:39:56.802738+0000","last_peered":"2024-09-18T07:39:56.802738+0000","last_clean":"2024-09-18T07:39:56.802738+0000","last_became_active":"2024-09-18T07:39:29.702301+0000","last_became_peered":"2024-09-18T07:39:29.702301+0000","last_unstale":"2024-09-18T07:39:56.802738+0000","last_undegraded":"2024-09-18T07:39:56.802738+0000","last_fullsized":"2024-09-18T07:39:56.802738+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3075,"log_dups_size":0,"ondisk_log_size":3075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:33.630737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201523200,"num_objects":3075,"num_object_clones":0,"num_object_copies":9225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3075,"num_write_kb":196800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":236,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.953820+0000","last_change":"2024-09-18T07:39:06.342429+0000","last_active":"2024-09-18T07:39:54.953820+0000","last_peered":"2024-09-18T07:39:54.953820+0000","last_clean":"2024-09-18T07:39:54.953820+0000","last_became_active":"2024-09-18T07:39:04.941752+0000","last_became_peered":"2024-09-18T07:39:04.941752+0000","last_unstale":"2024-09-18T07:39:54.953820+0000","last_undegraded":"2024-09-18T07:39:54.953820+0000","last_fullsized":"2024-09-18T07:39:54.953820+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:06.342370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:06.342370+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:36.082977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":225,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.956799+0000","last_change":"2024-09-18T07:36:13.119192+0000","last_active":"2024-09-18T07:39:54.956799+0000","last_peered":"2024-09-18T07:39:54.956799+0000","last_clean":"2024-09-18T07:39:54.956799+0000","last_became_active":"2024-09-18T07:36:13.118876+0000","last_became_peered":"2024-09-18T07:36:13.118876+0000","last_unstale":"2024-09-18T07:39:54.956799+0000","last_undegraded":"2024-09-18T07:39:54.956799+0000","last_fullsized":"2024-09-18T07:39:54.956799+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.613045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":514,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.957489+0000","last_change":"2024-09-18T07:39:31.418964+0000","last_active":"2024-09-18T07:39:54.957489+0000","last_peered":"2024-09-18T07:39:54.957489+0000","last_clean":"2024-09-18T07:39:54.957489+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:39:54.957489+0000","last_undegraded":"2024-09-18T07:39:54.957489+0000","last_fullsized":"2024-09-18T07:39:54.957489+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:31.418910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:31.418910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:40.383317+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"240'10","reported_seq":635,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.964465+0000","last_change":"2024-09-18T07:37:43.170266+0000","last_active":"2024-09-18T07:39:54.964465+0000","last_peered":"2024-09-18T07:39:54.964465+0000","last_clean":"2024-09-18T07:39:54.964465+0000","last_became_active":"2024-09-18T07:37:43.167850+0000","last_became_peered":"2024-09-18T07:37:43.167850+0000","last_unstale":"2024-09-18T07:39:54.964465+0000","last_undegraded":"2024-09-18T07:39:54.964465+0000","last_fullsized":"2024-09-18T07:39:54.964465+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:05.525762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.525762+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:49.695027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":484,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.969274+0000","last_change":"2024-09-18T07:39:00.759483+0000","last_active":"2024-09-18T07:39:54.969274+0000","last_peered":"2024-09-18T07:39:54.969274+0000","last_clean":"2024-09-18T07:39:54.969274+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:39:54.969274+0000","last_undegraded":"2024-09-18T07:39:54.969274+0000","last_fullsized":"2024-09-18T07:39:54.969274+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:00.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:10.972661+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"299'2953","reported_seq":2984,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:56.506604+0000","last_change":"2024-09-18T07:39:29.703780+0000","last_active":"2024-09-18T07:39:56.506604+0000","last_peered":"2024-09-18T07:39:56.506604+0000","last_clean":"2024-09-18T07:39:56.506604+0000","last_became_active":"2024-09-18T07:39:29.703475+0000","last_became_peered":"2024-09-18T07:39:29.703475+0000","last_unstale":"2024-09-18T07:39:56.506604+0000","last_undegraded":"2024-09-18T07:39:56.506604+0000","last_fullsized":"2024-09-18T07:39:56.506604+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":2953,"log_dups_size":0,"ondisk_log_size":2953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:58.380502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193527808,"num_objects":2953,"num_object_clones":0,"num_object_copies":8859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2953,"num_write_kb":188992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":236,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.953594+0000","last_change":"2024-09-18T07:39:07.357549+0000","last_active":"2024-09-18T07:39:54.953594+0000","last_peered":"2024-09-18T07:39:54.953594+0000","last_clean":"2024-09-18T07:39:54.953594+0000","last_became_active":"2024-09-18T07:39:04.948571+0000","last_became_peered":"2024-09-18T07:39:04.948571+0000","last_unstale":"2024-09-18T07:39:54.953594+0000","last_undegraded":"2024-09-18T07:39:54.953594+0000","last_fullsized":"2024-09-18T07:39:54.953594+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:07.357480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:07.357480+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:12.531366+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":586,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.964699+0000","last_change":"2024-09-18T07:36:09.498239+0000","last_active":"2024-09-18T07:39:54.964699+0000","last_peered":"2024-09-18T07:39:54.964699+0000","last_clean":"2024-09-18T07:39:54.964699+0000","last_became_active":"2024-09-18T07:35:01.415275+0000","last_became_peered":"2024-09-18T07:35:01.415275+0000","last_unstale":"2024-09-18T07:39:54.964699+0000","last_undegraded":"2024-09-18T07:39:54.964699+0000","last_fullsized":"2024-09-18T07:39:54.964699+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:36:09.498142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.498142+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":198,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.964774+0000","last_change":"2024-09-18T07:37:46.831436+0000","last_active":"2024-09-18T07:39:54.964774+0000","last_peered":"2024-09-18T07:39:54.964774+0000","last_clean":"2024-09-18T07:39:54.964774+0000","last_became_active":"2024-09-18T07:36:13.118553+0000","last_became_peered":"2024-09-18T07:36:13.118553+0000","last_unstale":"2024-09-18T07:39:54.964774+0000","last_undegraded":"2024-09-18T07:39:54.964774+0000","last_fullsized":"2024-09-18T07:39:54.964774+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:46.831373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:37:46.831373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:16.388148+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":131,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.952701+0000","last_change":"2024-09-18T07:37:43.070306+0000","last_active":"2024-09-18T07:39:54.952701+0000","last_peered":"2024-09-18T07:39:54.952701+0000","last_clean":"2024-09-18T07:39:54.952701+0000","last_became_active":"2024-09-18T07:37:43.070009+0000","last_became_peered":"2024-09-18T07:37:43.070009+0000","last_unstale":"2024-09-18T07:39:54.952701+0000","last_undegraded":"2024-09-18T07:39:54.952701+0000","last_fullsized":"2024-09-18T07:39:54.952701+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.202683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":131,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.952657+0000","last_change":"2024-09-18T07:37:43.052077+0000","last_active":"2024-09-18T07:39:54.952657+0000","last_peered":"2024-09-18T07:39:54.952657+0000","last_clean":"2024-09-18T07:39:54.952657+0000","last_became_active":"2024-09-18T07:37:43.051740+0000","last_became_peered":"2024-09-18T07:37:43.051740+0000","last_unstale":"2024-09-18T07:39:54.952657+0000","last_undegraded":"2024-09-18T07:39:54.952657+0000","last_fullsized":"2024-09-18T07:39:54.952657+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:09.269847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"299'3085","reported_seq":3116,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:57.441624+0000","last_change":"2024-09-18T07:39:29.702264+0000","last_active":"2024-09-18T07:39:57.441624+0000","last_peered":"2024-09-18T07:39:57.441624+0000","last_clean":"2024-09-18T07:39:57.441624+0000","last_became_active":"2024-09-18T07:39:29.701239+0000","last_became_peered":"2024-09-18T07:39:29.701239+0000","last_unstale":"2024-09-18T07:39:57.441624+0000","last_undegraded":"2024-09-18T07:39:57.441624+0000","last_fullsized":"2024-09-18T07:39:57.441624+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3085,"log_dups_size":0,"ondisk_log_size":3085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:51.463801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202178560,"num_objects":3085,"num_object_clones":0,"num_object_copies":9255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3085,"num_write_kb":197440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":68,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.956164+0000","last_change":"2024-09-18T07:39:05.509522+0000","last_active":"2024-09-18T07:39:54.956164+0000","last_peered":"2024-09-18T07:39:54.956164+0000","last_clean":"2024-09-18T07:39:54.956164+0000","last_became_active":"2024-09-18T07:39:04.950564+0000","last_became_peered":"2024-09-18T07:39:04.950564+0000","last_unstale":"2024-09-18T07:39:54.956164+0000","last_undegraded":"2024-09-18T07:39:54.956164+0000","last_fullsized":"2024-09-18T07:39:54.956164+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.509453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.509453+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:28.588623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"211'29","reported_seq":313,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.964628+0000","last_change":"2024-09-18T07:39:00.207829+0000","last_active":"2024-09-18T07:39:54.964628+0000","last_peered":"2024-09-18T07:39:54.964628+0000","last_clean":"2024-09-18T07:39:54.964628+0000","last_became_active":"2024-09-18T07:36:13.115278+0000","last_became_peered":"2024-09-18T07:36:13.115278+0000","last_unstale":"2024-09-18T07:39:54.964628+0000","last_undegraded":"2024-09-18T07:39:54.964628+0000","last_fullsized":"2024-09-18T07:39:54.964628+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'29","last_scrub_stamp":"2024-09-18T07:39:00.207506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.207506+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:08.496876+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":514,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.949591+0000","last_change":"2024-09-18T07:37:43.061414+0000","last_active":"2024-09-18T07:39:54.949591+0000","last_peered":"2024-09-18T07:39:54.949591+0000","last_clean":"2024-09-18T07:39:54.949591+0000","last_became_active":"2024-09-18T07:37:43.060930+0000","last_became_peered":"2024-09-18T07:37:43.060930+0000","last_unstale":"2024-09-18T07:39:54.949591+0000","last_undegraded":"2024-09-18T07:39:54.949591+0000","last_fullsized":"2024-09-18T07:39:54.949591+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:06.908713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.908713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":538,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.957021+0000","last_change":"2024-09-18T07:37:43.155837+0000","last_active":"2024-09-18T07:39:54.957021+0000","last_peered":"2024-09-18T07:39:54.957021+0000","last_clean":"2024-09-18T07:39:54.957021+0000","last_became_active":"2024-09-18T07:37:43.076595+0000","last_became_peered":"2024-09-18T07:37:43.076595+0000","last_unstale":"2024-09-18T07:39:54.957021+0000","last_undegraded":"2024-09-18T07:39:54.957021+0000","last_fullsized":"2024-09-18T07:39:54.957021+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:36:01.529677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.529677+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":488,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.966695+0000","last_change":"2024-09-18T07:39:25.741714+0000","last_active":"2024-09-18T07:39:54.966695+0000","last_peered":"2024-09-18T07:39:54.966695+0000","last_clean":"2024-09-18T07:39:54.966695+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:39:54.966695+0000","last_undegraded":"2024-09-18T07:39:54.966695+0000","last_fullsized":"2024-09-18T07:39:54.966695+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:25.741643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.741643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:35.816396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":509,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.960377+0000","last_change":"2024-09-18T07:38:59.871379+0000","last_active":"2024-09-18T07:39:54.960377+0000","last_peered":"2024-09-18T07:39:54.960377+0000","last_clean":"2024-09-18T07:39:54.960377+0000","last_became_active":"2024-09-18T07:35:01.418809+0000","last_became_peered":"2024-09-18T07:35:01.418809+0000","last_unstale":"2024-09-18T07:39:54.960377+0000","last_undegraded":"2024-09-18T07:39:54.960377+0000","last_fullsized":"2024-09-18T07:39:54.960377+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:59.871308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:59.871308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:00.245683+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"299'3158","reported_seq":3189,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:56.798579+0000","last_change":"2024-09-18T07:39:29.702743+0000","last_active":"2024-09-18T07:39:56.798579+0000","last_peered":"2024-09-18T07:39:56.798579+0000","last_clean":"2024-09-18T07:39:56.798579+0000","last_became_active":"2024-09-18T07:39:29.702461+0000","last_became_peered":"2024-09-18T07:39:29.702461+0000","last_unstale":"2024-09-18T07:39:56.798579+0000","last_undegraded":"2024-09-18T07:39:56.798579+0000","last_fullsized":"2024-09-18T07:39:56.798579+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3158,"log_dups_size":0,"ondisk_log_size":3158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:31.412469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206962688,"num_objects":3158,"num_object_clones":0,"num_object_copies":9474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3158,"num_write_kb":202112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":336,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.954937+0000","last_change":"2024-09-18T07:39:06.465696+0000","last_active":"2024-09-18T07:39:54.954937+0000","last_peered":"2024-09-18T07:39:54.954937+0000","last_clean":"2024-09-18T07:39:54.954937+0000","last_became_active":"2024-09-18T07:39:04.944921+0000","last_became_peered":"2024-09-18T07:39:04.944921+0000","last_unstale":"2024-09-18T07:39:54.954937+0000","last_undegraded":"2024-09-18T07:39:54.954937+0000","last_fullsized":"2024-09-18T07:39:54.954937+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:06.465620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:06.465620+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:21.933997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":237,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.957728+0000","last_change":"2024-09-18T07:36:13.118278+0000","last_active":"2024-09-18T07:39:54.957728+0000","last_peered":"2024-09-18T07:39:54.957728+0000","last_clean":"2024-09-18T07:39:54.957728+0000","last_became_active":"2024-09-18T07:36:13.117924+0000","last_became_peered":"2024-09-18T07:36:13.117924+0000","last_unstale":"2024-09-18T07:39:54.957728+0000","last_undegraded":"2024-09-18T07:39:54.957728+0000","last_fullsized":"2024-09-18T07:39:54.957728+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-18T07:36:06.083531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.083531+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":212,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.963610+0000","last_change":"2024-09-18T07:39:07.370140+0000","last_active":"2024-09-18T07:39:54.963610+0000","last_peered":"2024-09-18T07:39:54.963610+0000","last_clean":"2024-09-18T07:39:54.963610+0000","last_became_active":"2024-09-18T07:37:43.050971+0000","last_became_peered":"2024-09-18T07:37:43.050971+0000","last_unstale":"2024-09-18T07:39:54.963610+0000","last_undegraded":"2024-09-18T07:39:54.963610+0000","last_fullsized":"2024-09-18T07:39:54.963610+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:07.370081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:07.370081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:28.043475+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":517,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.958701+0000","last_change":"2024-09-18T07:37:49.895264+0000","last_active":"2024-09-18T07:39:54.958701+0000","last_peered":"2024-09-18T07:39:54.958701+0000","last_clean":"2024-09-18T07:39:54.958701+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:39:54.958701+0000","last_undegraded":"2024-09-18T07:39:54.958701+0000","last_fullsized":"2024-09-18T07:39:54.958701+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:37:49.894707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:37:49.894707+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:14.409320+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":514,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.965455+0000","last_change":"2024-09-18T07:39:35.457812+0000","last_active":"2024-09-18T07:39:54.965455+0000","last_peered":"2024-09-18T07:39:54.965455+0000","last_clean":"2024-09-18T07:39:54.965455+0000","last_became_active":"2024-09-18T07:35:01.403680+0000","last_became_peered":"2024-09-18T07:35:01.403680+0000","last_unstale":"2024-09-18T07:39:54.965455+0000","last_undegraded":"2024-09-18T07:39:54.965455+0000","last_fullsized":"2024-09-18T07:39:54.965455+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.457734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.457734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:55.517500+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":290,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.955412+0000","last_change":"2024-09-18T07:37:43.118570+0000","last_active":"2024-09-18T07:39:54.955412+0000","last_peered":"2024-09-18T07:39:54.955412+0000","last_clean":"2024-09-18T07:39:54.955412+0000","last_became_active":"2024-09-18T07:37:43.060019+0000","last_became_peered":"2024-09-18T07:37:43.060019+0000","last_unstale":"2024-09-18T07:39:54.955412+0000","last_undegraded":"2024-09-18T07:39:54.955412+0000","last_fullsized":"2024-09-18T07:39:54.955412+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:57.655797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.655797+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:51.521311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"299'3166","reported_seq":3197,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:57.458526+0000","last_change":"2024-09-18T07:39:29.701833+0000","last_active":"2024-09-18T07:39:57.458526+0000","last_peered":"2024-09-18T07:39:57.458526+0000","last_clean":"2024-09-18T07:39:57.458526+0000","last_became_active":"2024-09-18T07:39:29.701554+0000","last_became_peered":"2024-09-18T07:39:29.701554+0000","last_unstale":"2024-09-18T07:39:57.458526+0000","last_undegraded":"2024-09-18T07:39:57.458526+0000","last_fullsized":"2024-09-18T07:39:57.458526+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3166,"log_dups_size":0,"ondisk_log_size":3166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:40.719037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207486976,"num_objects":3166,"num_object_clones":0,"num_object_copies":9498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3166,"num_write_kb":202624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":236,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.952542+0000","last_change":"2024-09-18T07:39:05.333676+0000","last_active":"2024-09-18T07:39:54.952542+0000","last_peered":"2024-09-18T07:39:54.952542+0000","last_clean":"2024-09-18T07:39:54.952542+0000","last_became_active":"2024-09-18T07:39:04.938080+0000","last_became_peered":"2024-09-18T07:39:04.938080+0000","last_unstale":"2024-09-18T07:39:54.952542+0000","last_undegraded":"2024-09-18T07:39:54.952542+0000","last_fullsized":"2024-09-18T07:39:54.952542+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.333608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.333608+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:28.401993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":523,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.964532+0000","last_change":"2024-09-18T07:33:27.040227+0000","last_active":"2024-09-18T07:39:54.964532+0000","last_peered":"2024-09-18T07:39:54.964532+0000","last_clean":"2024-09-18T07:39:54.964532+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:39:54.964532+0000","last_undegraded":"2024-09-18T07:39:54.964532+0000","last_fullsized":"2024-09-18T07:39:54.964532+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:17.634433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":494,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.967822+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:39:54.967822+0000","last_peered":"2024-09-18T07:39:54.967822+0000","last_clean":"2024-09-18T07:39:54.967822+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:39:54.967822+0000","last_undegraded":"2024-09-18T07:39:54.967822+0000","last_fullsized":"2024-09-18T07:39:54.967822+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:16.624959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":563,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.954118+0000","last_change":"2024-09-18T07:35:01.421692+0000","last_active":"2024-09-18T07:39:54.954118+0000","last_peered":"2024-09-18T07:39:54.954118+0000","last_clean":"2024-09-18T07:39:54.954118+0000","last_became_active":"2024-09-18T07:35:01.400524+0000","last_became_peered":"2024-09-18T07:35:01.400524+0000","last_unstale":"2024-09-18T07:39:54.954118+0000","last_undegraded":"2024-09-18T07:39:54.954118+0000","last_fullsized":"2024-09-18T07:39:54.954118+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:46.474589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":301,"num_read_kb":296,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":498,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.957529+0000","last_change":"2024-09-18T07:36:28.551293+0000","last_active":"2024-09-18T07:39:54.957529+0000","last_peered":"2024-09-18T07:39:54.957529+0000","last_clean":"2024-09-18T07:39:54.957529+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:39:54.957529+0000","last_undegraded":"2024-09-18T07:39:54.957529+0000","last_fullsized":"2024-09-18T07:39:54.957529+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:28.551238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:28.551238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":198,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.956904+0000","last_change":"2024-09-18T07:36:39.601424+0000","last_active":"2024-09-18T07:39:54.956904+0000","last_peered":"2024-09-18T07:39:54.956904+0000","last_clean":"2024-09-18T07:39:54.956904+0000","last_became_active":"2024-09-18T07:36:13.117463+0000","last_became_peered":"2024-09-18T07:36:13.117463+0000","last_unstale":"2024-09-18T07:39:54.956904+0000","last_undegraded":"2024-09-18T07:39:54.956904+0000","last_fullsized":"2024-09-18T07:39:54.956904+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:39.601356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:39.601356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"299'3081","reported_seq":3112,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:57.085243+0000","last_change":"2024-09-18T07:39:29.700116+0000","last_active":"2024-09-18T07:39:57.085243+0000","last_peered":"2024-09-18T07:39:57.085243+0000","last_clean":"2024-09-18T07:39:57.085243+0000","last_became_active":"2024-09-18T07:39:29.699733+0000","last_became_peered":"2024-09-18T07:39:29.699733+0000","last_unstale":"2024-09-18T07:39:57.085243+0000","last_undegraded":"2024-09-18T07:39:57.085243+0000","last_fullsized":"2024-09-18T07:39:57.085243+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3081,"log_dups_size":0,"ondisk_log_size":3081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:37.454373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201916416,"num_objects":3081,"num_object_clones":0,"num_object_copies":9243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3081,"num_write_kb":197184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'20","reported_seq":176,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.953532+0000","last_change":"2024-09-18T07:37:43.157718+0000","last_active":"2024-09-18T07:39:54.953532+0000","last_peered":"2024-09-18T07:39:54.953532+0000","last_clean":"2024-09-18T07:39:54.953532+0000","last_became_active":"2024-09-18T07:37:43.087496+0000","last_became_peered":"2024-09-18T07:37:43.087496+0000","last_unstale":"2024-09-18T07:39:54.953532+0000","last_undegraded":"2024-09-18T07:39:54.953532+0000","last_fullsized":"2024-09-18T07:39:54.953532+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'17","last_scrub_stamp":"2024-09-18T07:33:33.849285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.849285+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:15.202624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":513,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.960572+0000","last_change":"2024-09-18T07:37:43.065584+0000","last_active":"2024-09-18T07:39:54.960572+0000","last_peered":"2024-09-18T07:39:54.960572+0000","last_clean":"2024-09-18T07:39:54.960572+0000","last_became_active":"2024-09-18T07:37:43.065279+0000","last_became_peered":"2024-09-18T07:37:43.065279+0000","last_unstale":"2024-09-18T07:39:54.960572+0000","last_undegraded":"2024-09-18T07:39:54.960572+0000","last_fullsized":"2024-09-18T07:39:54.960572+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:59.616988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:59.616988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:45.852796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":330,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.954328+0000","last_change":"2024-09-18T07:39:03.518624+0000","last_active":"2024-09-18T07:39:54.954328+0000","last_peered":"2024-09-18T07:39:54.954328+0000","last_clean":"2024-09-18T07:39:54.954328+0000","last_became_active":"2024-09-18T07:39:03.350419+0000","last_became_peered":"2024-09-18T07:39:03.350419+0000","last_unstale":"2024-09-18T07:39:54.954328+0000","last_undegraded":"2024-09-18T07:39:54.954328+0000","last_fullsized":"2024-09-18T07:39:54.954328+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:03.518551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:03.518551+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:14.832014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":272,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.955381+0000","last_change":"2024-09-18T07:38:52.486107+0000","last_active":"2024-09-18T07:39:54.955381+0000","last_peered":"2024-09-18T07:39:54.955381+0000","last_clean":"2024-09-18T07:39:54.955381+0000","last_became_active":"2024-09-18T07:35:01.412217+0000","last_became_peered":"2024-09-18T07:35:01.412217+0000","last_unstale":"2024-09-18T07:39:54.955381+0000","last_undegraded":"2024-09-18T07:39:54.955381+0000","last_fullsized":"2024-09-18T07:39:54.955381+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:52.486047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:52.486047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.218642+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":486,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.957351+0000","last_change":"2024-09-18T07:33:22.358792+0000","last_active":"2024-09-18T07:39:54.957351+0000","last_peered":"2024-09-18T07:39:54.957351+0000","last_clean":"2024-09-18T07:39:54.957351+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:39:54.957351+0000","last_undegraded":"2024-09-18T07:39:54.957351+0000","last_fullsized":"2024-09-18T07:39:54.957351+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.358746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.358746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:18.619255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"299'3016","reported_seq":3047,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:56.493788+0000","last_change":"2024-09-18T07:39:29.703721+0000","last_active":"2024-09-18T07:39:56.493788+0000","last_peered":"2024-09-18T07:39:56.493788+0000","last_clean":"2024-09-18T07:39:56.493788+0000","last_became_active":"2024-09-18T07:39:29.703451+0000","last_became_peered":"2024-09-18T07:39:29.703451+0000","last_unstale":"2024-09-18T07:39:56.493788+0000","last_undegraded":"2024-09-18T07:39:56.493788+0000","last_fullsized":"2024-09-18T07:39:56.493788+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3016,"log_dups_size":0,"ondisk_log_size":3016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:52.017204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197656576,"num_objects":3016,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3016,"num_write_kb":193024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":606,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.951743+0000","last_change":"2024-09-18T07:39:07.038527+0000","last_active":"2024-09-18T07:39:54.951743+0000","last_peered":"2024-09-18T07:39:54.951743+0000","last_clean":"2024-09-18T07:39:54.951743+0000","last_became_active":"2024-09-18T07:37:43.063255+0000","last_became_peered":"2024-09-18T07:37:43.063255+0000","last_unstale":"2024-09-18T07:39:54.951743+0000","last_undegraded":"2024-09-18T07:39:54.951743+0000","last_fullsized":"2024-09-18T07:39:54.951743+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:39:07.038450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:07.038450+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:24.342260+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":530,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.949060+0000","last_change":"2024-09-18T07:39:29.078656+0000","last_active":"2024-09-18T07:39:54.949060+0000","last_peered":"2024-09-18T07:39:54.949060+0000","last_clean":"2024-09-18T07:39:54.949060+0000","last_became_active":"2024-09-18T07:35:01.401063+0000","last_became_peered":"2024-09-18T07:35:01.401063+0000","last_unstale":"2024-09-18T07:39:54.949060+0000","last_undegraded":"2024-09-18T07:39:54.949060+0000","last_fullsized":"2024-09-18T07:39:54.949060+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.078600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.078600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:32.657735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":519,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.957275+0000","last_change":"2024-09-18T07:39:35.406107+0000","last_active":"2024-09-18T07:39:54.957275+0000","last_peered":"2024-09-18T07:39:54.957275+0000","last_clean":"2024-09-18T07:39:54.957275+0000","last_became_active":"2024-09-18T07:37:43.075535+0000","last_became_peered":"2024-09-18T07:37:43.075535+0000","last_unstale":"2024-09-18T07:39:54.957275+0000","last_undegraded":"2024-09-18T07:39:54.957275+0000","last_fullsized":"2024-09-18T07:39:54.957275+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.406041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.406041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:44.739547+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":508,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.962250+0000","last_change":"2024-09-18T07:37:43.066497+0000","last_active":"2024-09-18T07:39:54.962250+0000","last_peered":"2024-09-18T07:39:54.962250+0000","last_clean":"2024-09-18T07:39:54.962250+0000","last_became_active":"2024-09-18T07:37:43.066153+0000","last_became_peered":"2024-09-18T07:37:43.066153+0000","last_unstale":"2024-09-18T07:39:54.962250+0000","last_undegraded":"2024-09-18T07:39:54.962250+0000","last_fullsized":"2024-09-18T07:39:54.962250+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:38.955353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:38.955353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:08.997218+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"299'3042","reported_seq":3073,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:56.629147+0000","last_change":"2024-09-18T07:39:29.700091+0000","last_active":"2024-09-18T07:39:56.629147+0000","last_peered":"2024-09-18T07:39:56.629147+0000","last_clean":"2024-09-18T07:39:56.629147+0000","last_became_active":"2024-09-18T07:39:29.699800+0000","last_became_peered":"2024-09-18T07:39:29.699800+0000","last_unstale":"2024-09-18T07:39:56.629147+0000","last_undegraded":"2024-09-18T07:39:56.629147+0000","last_fullsized":"2024-09-18T07:39:56.629147+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3042,"log_dups_size":0,"ondisk_log_size":3042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:42.451559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199360512,"num_objects":3042,"num_object_clones":0,"num_object_copies":9126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3042,"num_write_kb":194688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":78,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.956110+0000","last_change":"2024-09-18T07:39:38.447787+0000","last_active":"2024-09-18T07:39:54.956110+0000","last_peered":"2024-09-18T07:39:54.956110+0000","last_clean":"2024-09-18T07:39:54.956110+0000","last_became_active":"2024-09-18T07:39:04.950292+0000","last_became_peered":"2024-09-18T07:39:04.950292+0000","last_unstale":"2024-09-18T07:39:54.956110+0000","last_undegraded":"2024-09-18T07:39:54.956110+0000","last_fullsized":"2024-09-18T07:39:54.956110+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:38.447735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:38.447735+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:53.329499+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":539,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.950691+0000","last_change":"2024-09-18T07:38:55.096608+0000","last_active":"2024-09-18T07:39:54.950691+0000","last_peered":"2024-09-18T07:39:54.950691+0000","last_clean":"2024-09-18T07:39:54.950691+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:39:54.950691+0000","last_undegraded":"2024-09-18T07:39:54.950691+0000","last_fullsized":"2024-09-18T07:39:54.950691+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:38:55.096550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:38:55.096550+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:56.008928+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":139,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.952873+0000","last_change":"2024-09-18T07:39:35.015984+0000","last_active":"2024-09-18T07:39:54.952873+0000","last_peered":"2024-09-18T07:39:54.952873+0000","last_clean":"2024-09-18T07:39:54.952873+0000","last_became_active":"2024-09-18T07:37:43.061808+0000","last_became_peered":"2024-09-18T07:37:43.061808+0000","last_unstale":"2024-09-18T07:39:54.952873+0000","last_undegraded":"2024-09-18T07:39:54.952873+0000","last_fullsized":"2024-09-18T07:39:54.952873+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.015923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.015923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:42.634359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":190,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.950226+0000","last_change":"2024-09-18T07:36:13.116901+0000","last_active":"2024-09-18T07:39:54.950226+0000","last_peered":"2024-09-18T07:39:54.950226+0000","last_clean":"2024-09-18T07:39:54.950226+0000","last_became_active":"2024-09-18T07:36:13.116527+0000","last_became_peered":"2024-09-18T07:36:13.116527+0000","last_unstale":"2024-09-18T07:39:54.950226+0000","last_undegraded":"2024-09-18T07:39:54.950226+0000","last_fullsized":"2024-09-18T07:39:54.950226+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:05.239160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.239160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:50.048595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":503,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.965288+0000","last_change":"2024-09-18T07:36:38.632189+0000","last_active":"2024-09-18T07:39:54.965288+0000","last_peered":"2024-09-18T07:39:54.965288+0000","last_clean":"2024-09-18T07:39:54.965288+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:39:54.965288+0000","last_undegraded":"2024-09-18T07:39:54.965288+0000","last_fullsized":"2024-09-18T07:39:54.965288+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:38.632040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:38.632040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:09.340014+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"299'3110","reported_seq":3140,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:57.094407+0000","last_change":"2024-09-18T07:39:29.701529+0000","last_active":"2024-09-18T07:39:57.094407+0000","last_peered":"2024-09-18T07:39:57.094407+0000","last_clean":"2024-09-18T07:39:57.094407+0000","last_became_active":"2024-09-18T07:39:29.701147+0000","last_became_peered":"2024-09-18T07:39:29.701147+0000","last_unstale":"2024-09-18T07:39:57.094407+0000","last_undegraded":"2024-09-18T07:39:57.094407+0000","last_fullsized":"2024-09-18T07:39:57.094407+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3110,"log_dups_size":0,"ondisk_log_size":3110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:51.353620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203816960,"num_objects":3110,"num_object_clones":0,"num_object_copies":9330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3110,"num_write_kb":199040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":336,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.954236+0000","last_change":"2024-09-18T07:39:05.498235+0000","last_active":"2024-09-18T07:39:54.954236+0000","last_peered":"2024-09-18T07:39:54.954236+0000","last_clean":"2024-09-18T07:39:54.954236+0000","last_became_active":"2024-09-18T07:39:04.944753+0000","last_became_peered":"2024-09-18T07:39:04.944753+0000","last_unstale":"2024-09-18T07:39:54.954236+0000","last_undegraded":"2024-09-18T07:39:54.954236+0000","last_fullsized":"2024-09-18T07:39:54.954236+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.498159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.498159+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:05.777887+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":548,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.950381+0000","last_change":"2024-09-18T07:37:43.106862+0000","last_active":"2024-09-18T07:39:54.950381+0000","last_peered":"2024-09-18T07:39:54.950381+0000","last_clean":"2024-09-18T07:39:54.950381+0000","last_became_active":"2024-09-18T07:37:43.062724+0000","last_became_peered":"2024-09-18T07:37:43.062724+0000","last_unstale":"2024-09-18T07:39:54.950381+0000","last_undegraded":"2024-09-18T07:39:54.950381+0000","last_fullsized":"2024-09-18T07:39:54.950381+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:32:05.000310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.000310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:45.943194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":507,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.962682+0000","last_change":"2024-09-18T07:39:38.871492+0000","last_active":"2024-09-18T07:39:54.962682+0000","last_peered":"2024-09-18T07:39:54.962682+0000","last_clean":"2024-09-18T07:39:54.962682+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:39:54.962682+0000","last_undegraded":"2024-09-18T07:39:54.962682+0000","last_fullsized":"2024-09-18T07:39:54.962682+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:38.871426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:38.871426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:04.468858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"173'11","reported_seq":649,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.965069+0000","last_change":"2024-09-18T07:38:55.227421+0000","last_active":"2024-09-18T07:39:54.965069+0000","last_peered":"2024-09-18T07:39:54.965069+0000","last_clean":"2024-09-18T07:39:54.965069+0000","last_became_active":"2024-09-18T07:35:01.403765+0000","last_became_peered":"2024-09-18T07:35:01.403765+0000","last_unstale":"2024-09-18T07:39:54.965069+0000","last_undegraded":"2024-09-18T07:39:54.965069+0000","last_fullsized":"2024-09-18T07:39:54.965069+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'11","last_scrub_stamp":"2024-09-18T07:38:55.227366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:55.227366+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:17.922368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":495,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.951611+0000","last_change":"2024-09-18T07:36:33.251164+0000","last_active":"2024-09-18T07:39:54.951611+0000","last_peered":"2024-09-18T07:39:54.951611+0000","last_clean":"2024-09-18T07:39:54.951611+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:39:54.951611+0000","last_undegraded":"2024-09-18T07:39:54.951611+0000","last_fullsized":"2024-09-18T07:39:54.951611+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:33.251093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:33.251093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"299'3072","reported_seq":3102,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:56.817369+0000","last_change":"2024-09-18T07:39:29.704580+0000","last_active":"2024-09-18T07:39:56.817369+0000","last_peered":"2024-09-18T07:39:56.817369+0000","last_clean":"2024-09-18T07:39:56.817369+0000","last_became_active":"2024-09-18T07:39:29.704318+0000","last_became_peered":"2024-09-18T07:39:29.704318+0000","last_unstale":"2024-09-18T07:39:56.817369+0000","last_undegraded":"2024-09-18T07:39:56.817369+0000","last_fullsized":"2024-09-18T07:39:56.817369+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3072,"log_dups_size":0,"ondisk_log_size":3072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:40.830318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":3072,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3072,"num_write_kb":196608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":525,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.964655+0000","last_change":"2024-09-18T07:39:02.716084+0000","last_active":"2024-09-18T07:39:54.964655+0000","last_peered":"2024-09-18T07:39:54.964655+0000","last_clean":"2024-09-18T07:39:54.964655+0000","last_became_active":"2024-09-18T07:37:27.994144+0000","last_became_peered":"2024-09-18T07:37:27.994144+0000","last_unstale":"2024-09-18T07:39:54.964655+0000","last_undegraded":"2024-09-18T07:39:54.964655+0000","last_fullsized":"2024-09-18T07:39:54.964655+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:39:02.716009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:02.716009+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:04.590923+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":520,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.957012+0000","last_change":"2024-09-18T07:37:43.076416+0000","last_active":"2024-09-18T07:39:54.957012+0000","last_peered":"2024-09-18T07:39:54.957012+0000","last_clean":"2024-09-18T07:39:54.957012+0000","last_became_active":"2024-09-18T07:37:43.075988+0000","last_became_peered":"2024-09-18T07:37:43.075988+0000","last_unstale":"2024-09-18T07:39:54.957012+0000","last_undegraded":"2024-09-18T07:39:54.957012+0000","last_fullsized":"2024-09-18T07:39:54.957012+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:21.222526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.222526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:44.443625+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":500,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.963802+0000","last_change":"2024-09-18T07:39:01.216439+0000","last_active":"2024-09-18T07:39:54.963802+0000","last_peered":"2024-09-18T07:39:54.963802+0000","last_clean":"2024-09-18T07:39:54.963802+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:39:54.963802+0000","last_undegraded":"2024-09-18T07:39:54.963802+0000","last_fullsized":"2024-09-18T07:39:54.963802+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:01.216305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:01.216305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:23.547767+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":506,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.967629+0000","last_change":"2024-09-18T07:36:07.526609+0000","last_active":"2024-09-18T07:39:54.967629+0000","last_peered":"2024-09-18T07:39:54.967629+0000","last_clean":"2024-09-18T07:39:54.967629+0000","last_became_active":"2024-09-18T07:35:01.410890+0000","last_became_peered":"2024-09-18T07:35:01.410890+0000","last_unstale":"2024-09-18T07:39:54.967629+0000","last_undegraded":"2024-09-18T07:39:54.967629+0000","last_fullsized":"2024-09-18T07:39:54.967629+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:07.526536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:07.526536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:41.630367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"299'3020","reported_seq":3051,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:56.790052+0000","last_change":"2024-09-18T07:39:29.701848+0000","last_active":"2024-09-18T07:39:56.790052+0000","last_peered":"2024-09-18T07:39:56.790052+0000","last_clean":"2024-09-18T07:39:56.790052+0000","last_became_active":"2024-09-18T07:39:29.701593+0000","last_became_peered":"2024-09-18T07:39:29.701593+0000","last_unstale":"2024-09-18T07:39:56.790052+0000","last_undegraded":"2024-09-18T07:39:56.790052+0000","last_fullsized":"2024-09-18T07:39:56.790052+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3020,"log_dups_size":0,"ondisk_log_size":3020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:45.431316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197918720,"num_objects":3020,"num_object_clones":0,"num_object_copies":9060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":193280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":159,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.953002+0000","last_change":"2024-09-18T07:39:25.932376+0000","last_active":"2024-09-18T07:39:54.953002+0000","last_peered":"2024-09-18T07:39:54.953002+0000","last_clean":"2024-09-18T07:39:54.953002+0000","last_became_active":"2024-09-18T07:37:43.063319+0000","last_became_peered":"2024-09-18T07:37:43.063319+0000","last_unstale":"2024-09-18T07:39:54.953002+0000","last_undegraded":"2024-09-18T07:39:54.953002+0000","last_fullsized":"2024-09-18T07:39:54.953002+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:39:25.932315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.932315+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:32.693891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":294,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.955570+0000","last_change":"2024-09-18T07:39:25.301868+0000","last_active":"2024-09-18T07:39:54.955570+0000","last_peered":"2024-09-18T07:39:54.955570+0000","last_clean":"2024-09-18T07:39:54.955570+0000","last_became_active":"2024-09-18T07:37:43.072894+0000","last_became_peered":"2024-09-18T07:37:43.072894+0000","last_unstale":"2024-09-18T07:39:54.955570+0000","last_undegraded":"2024-09-18T07:39:54.955570+0000","last_fullsized":"2024-09-18T07:39:54.955570+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:25.301825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.301825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.461283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":514,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.964106+0000","last_change":"2024-09-18T07:39:38.400864+0000","last_active":"2024-09-18T07:39:54.964106+0000","last_peered":"2024-09-18T07:39:54.964106+0000","last_clean":"2024-09-18T07:39:54.964106+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:39:54.964106+0000","last_undegraded":"2024-09-18T07:39:54.964106+0000","last_fullsized":"2024-09-18T07:39:54.964106+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:38.400785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:38.400785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:55.378212+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":264,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.954422+0000","last_change":"2024-09-18T07:35:01.408078+0000","last_active":"2024-09-18T07:39:54.954422+0000","last_peered":"2024-09-18T07:39:54.954422+0000","last_clean":"2024-09-18T07:39:54.954422+0000","last_became_active":"2024-09-18T07:35:01.406980+0000","last_became_peered":"2024-09-18T07:35:01.406980+0000","last_unstale":"2024-09-18T07:39:54.954422+0000","last_undegraded":"2024-09-18T07:39:54.954422+0000","last_fullsized":"2024-09-18T07:39:54.954422+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:40.390575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"299'3095","reported_seq":3126,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:56.549985+0000","last_change":"2024-09-18T07:39:29.700522+0000","last_active":"2024-09-18T07:39:56.549985+0000","last_peered":"2024-09-18T07:39:56.549985+0000","last_clean":"2024-09-18T07:39:56.549985+0000","last_became_active":"2024-09-18T07:39:29.700244+0000","last_became_peered":"2024-09-18T07:39:29.700244+0000","last_unstale":"2024-09-18T07:39:56.549985+0000","last_undegraded":"2024-09-18T07:39:56.549985+0000","last_fullsized":"2024-09-18T07:39:56.549985+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:39.850144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202833920,"num_objects":3095,"num_object_clones":0,"num_object_copies":9285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3095,"num_write_kb":198080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":535,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.948955+0000","last_change":"2024-09-18T07:38:59.101584+0000","last_active":"2024-09-18T07:39:54.948955+0000","last_peered":"2024-09-18T07:39:54.948955+0000","last_clean":"2024-09-18T07:39:54.948955+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:39:54.948955+0000","last_undegraded":"2024-09-18T07:39:54.948955+0000","last_fullsized":"2024-09-18T07:39:54.948955+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:38:59.101512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:38:59.101512+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:07.212990+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":264,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.955703+0000","last_change":"2024-09-18T07:35:01.400050+0000","last_active":"2024-09-18T07:39:54.955703+0000","last_peered":"2024-09-18T07:39:54.955703+0000","last_clean":"2024-09-18T07:39:54.955703+0000","last_became_active":"2024-09-18T07:35:01.398499+0000","last_became_peered":"2024-09-18T07:35:01.398499+0000","last_unstale":"2024-09-18T07:39:54.955703+0000","last_undegraded":"2024-09-18T07:39:54.955703+0000","last_fullsized":"2024-09-18T07:39:54.955703+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:43.441418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":516,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.963512+0000","last_change":"2024-09-18T07:36:32.564149+0000","last_active":"2024-09-18T07:39:54.963512+0000","last_peered":"2024-09-18T07:39:54.963512+0000","last_clean":"2024-09-18T07:39:54.963512+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:39:54.963512+0000","last_undegraded":"2024-09-18T07:39:54.963512+0000","last_fullsized":"2024-09-18T07:39:54.963512+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:32.564101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:32.564101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:08.345410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":500,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.962419+0000","last_change":"2024-09-18T07:37:43.016535+0000","last_active":"2024-09-18T07:39:54.962419+0000","last_peered":"2024-09-18T07:39:54.962419+0000","last_clean":"2024-09-18T07:39:54.962419+0000","last_became_active":"2024-09-18T07:35:01.418735+0000","last_became_peered":"2024-09-18T07:35:01.418735+0000","last_unstale":"2024-09-18T07:39:54.962419+0000","last_undegraded":"2024-09-18T07:39:54.962419+0000","last_fullsized":"2024-09-18T07:39:54.962419+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:43.016489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:37:43.016489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"299'3017","reported_seq":3048,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:57.399571+0000","last_change":"2024-09-18T07:39:29.701895+0000","last_active":"2024-09-18T07:39:57.399571+0000","last_peered":"2024-09-18T07:39:57.399571+0000","last_clean":"2024-09-18T07:39:57.399571+0000","last_became_active":"2024-09-18T07:39:29.701617+0000","last_became_peered":"2024-09-18T07:39:29.701617+0000","last_unstale":"2024-09-18T07:39:57.399571+0000","last_undegraded":"2024-09-18T07:39:57.399571+0000","last_fullsized":"2024-09-18T07:39:57.399571+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3017,"log_dups_size":0,"ondisk_log_size":3017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:42.973327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197722112,"num_objects":3017,"num_object_clones":0,"num_object_copies":9051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":193088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":627,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.956285+0000","last_change":"2024-09-18T07:39:27.484003+0000","last_active":"2024-09-18T07:39:54.956285+0000","last_peered":"2024-09-18T07:39:54.956285+0000","last_clean":"2024-09-18T07:39:54.956285+0000","last_became_active":"2024-09-18T07:37:43.075552+0000","last_became_peered":"2024-09-18T07:37:43.075552+0000","last_unstale":"2024-09-18T07:39:54.956285+0000","last_undegraded":"2024-09-18T07:39:54.956285+0000","last_fullsized":"2024-09-18T07:39:54.956285+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:39:27.483930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:27.483930+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.149026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":286,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.955207+0000","last_change":"2024-09-18T07:36:09.261574+0000","last_active":"2024-09-18T07:39:54.955207+0000","last_peered":"2024-09-18T07:39:54.955207+0000","last_clean":"2024-09-18T07:39:54.955207+0000","last_became_active":"2024-09-18T07:35:01.406695+0000","last_became_peered":"2024-09-18T07:35:01.406695+0000","last_unstale":"2024-09-18T07:39:54.955207+0000","last_undegraded":"2024-09-18T07:39:54.955207+0000","last_fullsized":"2024-09-18T07:39:54.955207+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:36:09.261516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.261516+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:22.292832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"240'12","reported_seq":366,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.954511+0000","last_change":"2024-09-18T07:38:55.490250+0000","last_active":"2024-09-18T07:39:54.954511+0000","last_peered":"2024-09-18T07:39:54.954511+0000","last_clean":"2024-09-18T07:39:54.954511+0000","last_became_active":"2024-09-18T07:37:43.056714+0000","last_became_peered":"2024-09-18T07:37:43.056714+0000","last_unstale":"2024-09-18T07:39:54.954511+0000","last_undegraded":"2024-09-18T07:39:54.954511+0000","last_fullsized":"2024-09-18T07:39:54.954511+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:38:55.490178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:55.490178+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:07.849179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":492,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.956619+0000","last_change":"2024-09-18T07:37:43.078768+0000","last_active":"2024-09-18T07:39:54.956619+0000","last_peered":"2024-09-18T07:39:54.956619+0000","last_clean":"2024-09-18T07:39:54.956619+0000","last_became_active":"2024-09-18T07:37:43.078121+0000","last_became_peered":"2024-09-18T07:37:43.078121+0000","last_unstale":"2024-09-18T07:39:54.956619+0000","last_undegraded":"2024-09-18T07:39:54.956619+0000","last_fullsized":"2024-09-18T07:39:54.956619+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:12.586962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":138,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.952378+0000","last_change":"2024-09-18T07:39:37.072844+0000","last_active":"2024-09-18T07:39:54.952378+0000","last_peered":"2024-09-18T07:39:54.952378+0000","last_clean":"2024-09-18T07:39:54.952378+0000","last_became_active":"2024-09-18T07:37:43.087994+0000","last_became_peered":"2024-09-18T07:37:43.087994+0000","last_unstale":"2024-09-18T07:39:54.952378+0000","last_undegraded":"2024-09-18T07:39:54.952378+0000","last_fullsized":"2024-09-18T07:39:54.952378+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.072782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.072782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:52.412066+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":533,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.968677+0000","last_change":"2024-09-18T07:36:09.061983+0000","last_active":"2024-09-18T07:39:54.968677+0000","last_peered":"2024-09-18T07:39:54.968677+0000","last_clean":"2024-09-18T07:39:54.968677+0000","last_became_active":"2024-09-18T07:35:01.407787+0000","last_became_peered":"2024-09-18T07:35:01.407787+0000","last_unstale":"2024-09-18T07:39:54.968677+0000","last_undegraded":"2024-09-18T07:39:54.968677+0000","last_fullsized":"2024-09-18T07:39:54.968677+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-18T07:36:09.061903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.061903+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:16.050531+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":497,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.958881+0000","last_change":"2024-09-18T07:36:34.608531+0000","last_active":"2024-09-18T07:39:54.958881+0000","last_peered":"2024-09-18T07:39:54.958881+0000","last_clean":"2024-09-18T07:39:54.958881+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:39:54.958881+0000","last_undegraded":"2024-09-18T07:39:54.958881+0000","last_fullsized":"2024-09-18T07:39:54.958881+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:34.608486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:34.608486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":278,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.955514+0000","last_change":"2024-09-18T07:37:43.079723+0000","last_active":"2024-09-18T07:39:54.955514+0000","last_peered":"2024-09-18T07:39:54.955514+0000","last_clean":"2024-09-18T07:39:54.955514+0000","last_became_active":"2024-09-18T07:37:43.079313+0000","last_became_peered":"2024-09-18T07:37:43.079313+0000","last_unstale":"2024-09-18T07:39:54.955514+0000","last_undegraded":"2024-09-18T07:39:54.955514+0000","last_fullsized":"2024-09-18T07:39:54.955514+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:58.688631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:58.688631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:50.554490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":533,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.964624+0000","last_change":"2024-09-18T07:35:01.448510+0000","last_active":"2024-09-18T07:39:54.964624+0000","last_peered":"2024-09-18T07:39:54.964624+0000","last_clean":"2024-09-18T07:39:54.964624+0000","last_became_active":"2024-09-18T07:35:01.414458+0000","last_became_peered":"2024-09-18T07:35:01.414458+0000","last_unstale":"2024-09-18T07:39:54.964624+0000","last_undegraded":"2024-09-18T07:39:54.964624+0000","last_fullsized":"2024-09-18T07:39:54.964624+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:19.600284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":286,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.954454+0000","last_change":"2024-09-18T07:37:43.057776+0000","last_active":"2024-09-18T07:39:54.954454+0000","last_peered":"2024-09-18T07:39:54.954454+0000","last_clean":"2024-09-18T07:39:54.954454+0000","last_became_active":"2024-09-18T07:37:43.056675+0000","last_became_peered":"2024-09-18T07:37:43.056675+0000","last_unstale":"2024-09-18T07:39:54.954454+0000","last_undegraded":"2024-09-18T07:39:54.954454+0000","last_fullsized":"2024-09-18T07:39:54.954454+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:05.201723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.201723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:15.371131+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":520,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.961085+0000","last_change":"2024-09-18T07:39:30.966887+0000","last_active":"2024-09-18T07:39:54.961085+0000","last_peered":"2024-09-18T07:39:54.961085+0000","last_clean":"2024-09-18T07:39:54.961085+0000","last_became_active":"2024-09-18T07:35:01.420133+0000","last_became_peered":"2024-09-18T07:35:01.420133+0000","last_unstale":"2024-09-18T07:39:54.961085+0000","last_undegraded":"2024-09-18T07:39:54.961085+0000","last_fullsized":"2024-09-18T07:39:54.961085+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:30.966840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:30.966840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:33.098878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":501,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.964260+0000","last_change":"2024-09-18T07:37:48.861965+0000","last_active":"2024-09-18T07:39:54.964260+0000","last_peered":"2024-09-18T07:39:54.964260+0000","last_clean":"2024-09-18T07:39:54.964260+0000","last_became_active":"2024-09-18T07:35:01.396305+0000","last_became_peered":"2024-09-18T07:35:01.396305+0000","last_unstale":"2024-09-18T07:39:54.964260+0000","last_undegraded":"2024-09-18T07:39:54.964260+0000","last_fullsized":"2024-09-18T07:39:54.964260+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:48.861784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:37:48.861784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:23.433931+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":551,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.957253+0000","last_change":"2024-09-18T07:39:25.515172+0000","last_active":"2024-09-18T07:39:54.957253+0000","last_peered":"2024-09-18T07:39:54.957253+0000","last_clean":"2024-09-18T07:39:54.957253+0000","last_became_active":"2024-09-18T07:35:01.416744+0000","last_became_peered":"2024-09-18T07:35:01.416744+0000","last_unstale":"2024-09-18T07:39:54.957253+0000","last_undegraded":"2024-09-18T07:39:54.957253+0000","last_fullsized":"2024-09-18T07:39:54.957253+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:39:25.515085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.515085+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:40.279265+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":517,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.963944+0000","last_change":"2024-09-18T07:39:32.766405+0000","last_active":"2024-09-18T07:39:54.963944+0000","last_peered":"2024-09-18T07:39:54.963944+0000","last_clean":"2024-09-18T07:39:54.963944+0000","last_became_active":"2024-09-18T07:35:01.414024+0000","last_became_peered":"2024-09-18T07:35:01.414024+0000","last_unstale":"2024-09-18T07:39:54.963944+0000","last_undegraded":"2024-09-18T07:39:54.963944+0000","last_fullsized":"2024-09-18T07:39:54.963944+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:32.766366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:32.766366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:36.625283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"173'10","reported_seq":637,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.962457+0000","last_change":"2024-09-18T07:39:32.947933+0000","last_active":"2024-09-18T07:39:54.962457+0000","last_peered":"2024-09-18T07:39:54.962457+0000","last_clean":"2024-09-18T07:39:54.962457+0000","last_became_active":"2024-09-18T07:35:01.407568+0000","last_became_peered":"2024-09-18T07:35:01.407568+0000","last_unstale":"2024-09-18T07:39:54.962457+0000","last_undegraded":"2024-09-18T07:39:54.962457+0000","last_fullsized":"2024-09-18T07:39:54.962457+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:39:32.947887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:32.947887+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:58.809615+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":487,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.951034+0000","last_change":"2024-09-18T07:36:11.910543+0000","last_active":"2024-09-18T07:39:54.951034+0000","last_peered":"2024-09-18T07:39:54.951034+0000","last_clean":"2024-09-18T07:39:54.951034+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:39:54.951034+0000","last_undegraded":"2024-09-18T07:39:54.951034+0000","last_fullsized":"2024-09-18T07:39:54.951034+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:11.910474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:11.910474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.028676+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":555,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.962691+0000","last_change":"2024-09-18T07:36:02.975929+0000","last_active":"2024-09-18T07:39:54.962691+0000","last_peered":"2024-09-18T07:39:54.962691+0000","last_clean":"2024-09-18T07:39:54.962691+0000","last_became_active":"2024-09-18T07:35:01.407738+0000","last_became_peered":"2024-09-18T07:35:01.407738+0000","last_unstale":"2024-09-18T07:39:54.962691+0000","last_undegraded":"2024-09-18T07:39:54.962691+0000","last_fullsized":"2024-09-18T07:39:54.962691+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:36:02.975863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:02.975863+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:49.805199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":272,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.955156+0000","last_change":"2024-09-18T07:36:08.222898+0000","last_active":"2024-09-18T07:39:54.955156+0000","last_peered":"2024-09-18T07:39:54.955156+0000","last_clean":"2024-09-18T07:39:54.955156+0000","last_became_active":"2024-09-18T07:35:01.400197+0000","last_became_peered":"2024-09-18T07:35:01.400197+0000","last_unstale":"2024-09-18T07:39:54.955156+0000","last_undegraded":"2024-09-18T07:39:54.955156+0000","last_fullsized":"2024-09-18T07:39:54.955156+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:08.222838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:08.222838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:16.370989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":609,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.957507+0000","last_change":"2024-09-18T07:37:42.285094+0000","last_active":"2024-09-18T07:39:54.957507+0000","last_peered":"2024-09-18T07:39:54.957507+0000","last_clean":"2024-09-18T07:39:54.957507+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:39:54.957507+0000","last_undegraded":"2024-09-18T07:39:54.957507+0000","last_fullsized":"2024-09-18T07:39:54.957507+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:37:42.285042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:37:42.285042+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":505,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.964215+0000","last_change":"2024-09-18T07:39:31.784401+0000","last_active":"2024-09-18T07:39:54.964215+0000","last_peered":"2024-09-18T07:39:54.964215+0000","last_clean":"2024-09-18T07:39:54.964215+0000","last_became_active":"2024-09-18T07:35:01.410561+0000","last_became_peered":"2024-09-18T07:35:01.410561+0000","last_unstale":"2024-09-18T07:39:54.964215+0000","last_undegraded":"2024-09-18T07:39:54.964215+0000","last_fullsized":"2024-09-18T07:39:54.964215+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:31.784355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:31.784355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:37.247787+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":584,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.949378+0000","last_change":"2024-09-18T07:38:41.163393+0000","last_active":"2024-09-18T07:39:54.949378+0000","last_peered":"2024-09-18T07:39:54.949378+0000","last_clean":"2024-09-18T07:39:54.949378+0000","last_became_active":"2024-09-18T07:38:41.143108+0000","last_became_peered":"2024-09-18T07:38:41.143108+0000","last_unstale":"2024-09-18T07:39:54.949378+0000","last_undegraded":"2024-09-18T07:39:54.949378+0000","last_fullsized":"2024-09-18T07:39:54.949378+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:36:01.986964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.986964+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:14.037941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":519,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.950625+0000","last_change":"2024-09-18T07:37:43.062429+0000","last_active":"2024-09-18T07:39:54.950625+0000","last_peered":"2024-09-18T07:39:54.950625+0000","last_clean":"2024-09-18T07:39:54.950625+0000","last_became_active":"2024-09-18T07:37:43.061577+0000","last_became_peered":"2024-09-18T07:37:43.061577+0000","last_unstale":"2024-09-18T07:39:54.950625+0000","last_undegraded":"2024-09-18T07:39:54.950625+0000","last_fullsized":"2024-09-18T07:39:54.950625+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:39.923066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":190,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.956738+0000","last_change":"2024-09-18T07:36:13.118114+0000","last_active":"2024-09-18T07:39:54.956738+0000","last_peered":"2024-09-18T07:39:54.956738+0000","last_clean":"2024-09-18T07:39:54.956738+0000","last_became_active":"2024-09-18T07:36:13.117584+0000","last_became_peered":"2024-09-18T07:36:13.117584+0000","last_unstale":"2024-09-18T07:39:54.956738+0000","last_undegraded":"2024-09-18T07:39:54.956738+0000","last_fullsized":"2024-09-18T07:39:54.956738+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:15.687992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":486,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.960894+0000","last_change":"2024-09-18T07:39:37.892445+0000","last_active":"2024-09-18T07:39:54.960894+0000","last_peered":"2024-09-18T07:39:54.960894+0000","last_clean":"2024-09-18T07:39:54.960894+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:39:54.960894+0000","last_undegraded":"2024-09-18T07:39:54.960894+0000","last_fullsized":"2024-09-18T07:39:54.960894+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.892377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.892377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:56.957035+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":372,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.963605+0000","last_change":"2024-09-18T07:37:43.138346+0000","last_active":"2024-09-18T07:39:54.963605+0000","last_peered":"2024-09-18T07:39:54.963605+0000","last_clean":"2024-09-18T07:39:54.963605+0000","last_became_active":"2024-09-18T07:37:43.067598+0000","last_became_peered":"2024-09-18T07:37:43.067598+0000","last_unstale":"2024-09-18T07:39:54.963605+0000","last_undegraded":"2024-09-18T07:39:54.963605+0000","last_fullsized":"2024-09-18T07:39:54.963605+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:36:38.638226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:38.638226+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":272,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.955434+0000","last_change":"2024-09-18T07:35:57.293607+0000","last_active":"2024-09-18T07:39:54.955434+0000","last_peered":"2024-09-18T07:39:54.955434+0000","last_clean":"2024-09-18T07:39:54.955434+0000","last_became_active":"2024-09-18T07:35:01.405287+0000","last_became_peered":"2024-09-18T07:35:01.405287+0000","last_unstale":"2024-09-18T07:39:54.955434+0000","last_undegraded":"2024-09-18T07:39:54.955434+0000","last_fullsized":"2024-09-18T07:39:54.955434+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:57.293555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.293555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:12.492824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":498,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.962498+0000","last_change":"2024-09-18T07:38:56.920456+0000","last_active":"2024-09-18T07:39:54.962498+0000","last_peered":"2024-09-18T07:39:54.962498+0000","last_clean":"2024-09-18T07:39:54.962498+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:39:54.962498+0000","last_undegraded":"2024-09-18T07:39:54.962498+0000","last_fullsized":"2024-09-18T07:39:54.962498+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:56.920384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:56.920384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:11.540961+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":499,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.964279+0000","last_change":"2024-09-18T07:36:37.593789+0000","last_active":"2024-09-18T07:39:54.964279+0000","last_peered":"2024-09-18T07:39:54.964279+0000","last_clean":"2024-09-18T07:39:54.964279+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:39:54.964279+0000","last_undegraded":"2024-09-18T07:39:54.964279+0000","last_fullsized":"2024-09-18T07:39:54.964279+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:37.593724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:37.593724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":597,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.963485+0000","last_change":"2024-09-18T07:37:43.138620+0000","last_active":"2024-09-18T07:39:54.963485+0000","last_peered":"2024-09-18T07:39:54.963485+0000","last_clean":"2024-09-18T07:39:54.963485+0000","last_became_active":"2024-09-18T07:37:43.066208+0000","last_became_peered":"2024-09-18T07:37:43.066208+0000","last_unstale":"2024-09-18T07:39:54.963485+0000","last_undegraded":"2024-09-18T07:39:54.963485+0000","last_fullsized":"2024-09-18T07:39:54.963485+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:36:29.051437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:29.051437+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:53.872186+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":511,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.959239+0000","last_change":"2024-09-18T07:35:01.415691+0000","last_active":"2024-09-18T07:39:54.959239+0000","last_peered":"2024-09-18T07:39:54.959239+0000","last_clean":"2024-09-18T07:39:54.959239+0000","last_became_active":"2024-09-18T07:35:01.415261+0000","last_became_peered":"2024-09-18T07:35:01.415261+0000","last_unstale":"2024-09-18T07:39:54.959239+0000","last_undegraded":"2024-09-18T07:39:54.959239+0000","last_fullsized":"2024-09-18T07:39:54.959239+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:58.734027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:58.734027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:17.638037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":286,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.954507+0000","last_change":"2024-09-18T07:37:43.057780+0000","last_active":"2024-09-18T07:39:54.954507+0000","last_peered":"2024-09-18T07:39:54.954507+0000","last_clean":"2024-09-18T07:39:54.954507+0000","last_became_active":"2024-09-18T07:37:43.057483+0000","last_became_peered":"2024-09-18T07:37:43.057483+0000","last_unstale":"2024-09-18T07:39:54.954507+0000","last_undegraded":"2024-09-18T07:39:54.954507+0000","last_fullsized":"2024-09-18T07:39:54.954507+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:41.311291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:41.311291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.461425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":516,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.962573+0000","last_change":"2024-09-18T07:39:02.930467+0000","last_active":"2024-09-18T07:39:54.962573+0000","last_peered":"2024-09-18T07:39:54.962573+0000","last_clean":"2024-09-18T07:39:54.962573+0000","last_became_active":"2024-09-18T07:37:43.067066+0000","last_became_peered":"2024-09-18T07:37:43.067066+0000","last_unstale":"2024-09-18T07:39:54.962573+0000","last_undegraded":"2024-09-18T07:39:54.962573+0000","last_fullsized":"2024-09-18T07:39:54.962573+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:02.930392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:02.930392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.214721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":555,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.960016+0000","last_change":"2024-09-18T07:37:43.134956+0000","last_active":"2024-09-18T07:39:54.960016+0000","last_peered":"2024-09-18T07:39:54.960016+0000","last_clean":"2024-09-18T07:39:54.960016+0000","last_became_active":"2024-09-18T07:37:43.067503+0000","last_became_peered":"2024-09-18T07:37:43.067503+0000","last_unstale":"2024-09-18T07:39:54.960016+0000","last_undegraded":"2024-09-18T07:39:54.960016+0000","last_fullsized":"2024-09-18T07:39:54.960016+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:36:40.908629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:40.908629+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":495,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.966036+0000","last_change":"2024-09-18T07:39:05.648336+0000","last_active":"2024-09-18T07:39:54.966036+0000","last_peered":"2024-09-18T07:39:54.966036+0000","last_clean":"2024-09-18T07:39:54.966036+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:39:54.966036+0000","last_undegraded":"2024-09-18T07:39:54.966036+0000","last_fullsized":"2024-09-18T07:39:54.966036+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:05.648263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.648263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:07.539151+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":130,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.952540+0000","last_change":"2024-09-18T07:37:43.061403+0000","last_active":"2024-09-18T07:39:54.952540+0000","last_peered":"2024-09-18T07:39:54.952540+0000","last_clean":"2024-09-18T07:39:54.952540+0000","last_became_active":"2024-09-18T07:37:43.061081+0000","last_became_peered":"2024-09-18T07:37:43.061081+0000","last_unstale":"2024-09-18T07:39:54.952540+0000","last_undegraded":"2024-09-18T07:39:54.952540+0000","last_fullsized":"2024-09-18T07:39:54.952540+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:30.345180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:30.345180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:39.067422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":280,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.955182+0000","last_change":"2024-09-18T07:39:02.503766+0000","last_active":"2024-09-18T07:39:54.955182+0000","last_peered":"2024-09-18T07:39:54.955182+0000","last_clean":"2024-09-18T07:39:54.955182+0000","last_became_active":"2024-09-18T07:35:01.400396+0000","last_became_peered":"2024-09-18T07:35:01.400396+0000","last_unstale":"2024-09-18T07:39:54.955182+0000","last_undegraded":"2024-09-18T07:39:54.955182+0000","last_fullsized":"2024-09-18T07:39:54.955182+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:02.503698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:02.503698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:11.708815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":190,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.957035+0000","last_change":"2024-09-18T07:36:13.118009+0000","last_active":"2024-09-18T07:39:54.957035+0000","last_peered":"2024-09-18T07:39:54.957035+0000","last_clean":"2024-09-18T07:39:54.957035+0000","last_became_active":"2024-09-18T07:36:13.117154+0000","last_became_peered":"2024-09-18T07:36:13.117154+0000","last_unstale":"2024-09-18T07:39:54.957035+0000","last_undegraded":"2024-09-18T07:39:54.957035+0000","last_fullsized":"2024-09-18T07:39:54.957035+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.071629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.071629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:21.615923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":520,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.960379+0000","last_change":"2024-09-18T07:39:35.902385+0000","last_active":"2024-09-18T07:39:54.960379+0000","last_peered":"2024-09-18T07:39:54.960379+0000","last_clean":"2024-09-18T07:39:54.960379+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:39:54.960379+0000","last_undegraded":"2024-09-18T07:39:54.960379+0000","last_fullsized":"2024-09-18T07:39:54.960379+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:39:35.901306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.901306+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.222739+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":143,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.954073+0000","last_change":"2024-09-18T07:37:43.258255+0000","last_active":"2024-09-18T07:39:54.954073+0000","last_peered":"2024-09-18T07:39:54.954073+0000","last_clean":"2024-09-18T07:39:54.954073+0000","last_became_active":"2024-09-18T07:37:43.070186+0000","last_became_peered":"2024-09-18T07:37:43.070186+0000","last_unstale":"2024-09-18T07:39:54.954073+0000","last_undegraded":"2024-09-18T07:39:54.954073+0000","last_fullsized":"2024-09-18T07:39:54.954073+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:34:58.432781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:58.432781+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:22.023484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":518,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-09-18T07:39:54.964362+0000","last_change":"2024-09-18T07:39:35.694805+0000","last_active":"2024-09-18T07:39:54.964362+0000","last_peered":"2024-09-18T07:39:54.964362+0000","last_clean":"2024-09-18T07:39:54.964362+0000","last_became_active":"2024-09-18T07:35:01.410469+0000","last_became_peered":"2024-09-18T07:35:01.410469+0000","last_unstale":"2024-09-18T07:39:54.964362+0000","last_undegraded":"2024-09-18T07:39:54.964362+0000","last_fullsized":"2024-09-18T07:39:54.964362+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.694741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.694741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.064934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3236036608,"num_objects":49378,"num_object_clones":0,"num_object_copies":148134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49378,"num_write_kb":3160192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9678618624,"data_stored":9678618624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49378,"ondisk_log_size":49378,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":4664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":398,"ondisk_log_size":398,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":301,"num_read_kb":296,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201392,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1998084,"kb_used_data":1746160,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91747068,"statfs":{"total":95995035648,"available":93948997632,"internally_reserved":0,"allocated":1788067840,"data_stored":1787487002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":257948106},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463030,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":834084,"kb_used_data":578000,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":92911068,"statfs":{"total":95995035648,"available":95140933632,"internally_reserved":0,"allocated":591872000,"data_stored":591322515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":262207942},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724667,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2064184,"kb_used_data":1740068,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91680968,"statfs":{"total":95995035648,"available":93881311232,"internally_reserved":0,"allocated":1781829632,"data_stored":1781260997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953602,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":865472,"kb_used_data":581552,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":92879680,"statfs":{"total":95995035648,"available":95108792320,"internally_reserved":0,"allocated":595509248,"data_stored":594928133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149830,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2193860,"kb_used_data":1783668,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":91551292,"statfs":{"total":95995035648,"available":93748523008,"internally_reserved":0,"allocated":1826476032,"data_stored":1825892217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":420018631},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411468,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1303732,"kb_used_data":965728,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92441420,"statfs":{"total":95995035648,"available":94660014080,"internally_reserved":0,"allocated":988905472,"data_stored":988340832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574995,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2120080,"kb_used_data":1750140,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":91625072,"statfs":{"total":95995035648,"available":93824073728,"internally_reserved":0,"allocated":1792143360,"data_stored":1791551034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738520,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1253192,"kb_used_data":929080,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92491960,"statfs":{"total":95995035648,"available":94711767040,"internally_reserved":0,"allocated":951377920,"data_stored":950790015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":811794432,"data_stored":811794432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1820393472,"data_stored":1820393472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":607059968,"data_stored":607059968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1636696064,"data_stored":1636696064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":602931200,"data_stored":602931200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1809383424,"data_stored":1809383424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":602406912,"data_stored":602406912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1787953152,"data_stored":1787953152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:39:59.726 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:39:59.783 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:39:59.815 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:39:59.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:59.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:39:59.917 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:39:59.963 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:39:59.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:39:59.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:00.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:00.077 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 25 16 53794 53778 134.416 91.0625 0.0150577 0.00743287 2024-09-18T07:40:00.106 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:00.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:00.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:00.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:00.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:00.245 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:00.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:00.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:00.383 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:00.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:00.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:00 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:40:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:00 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:40:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:00 smithi137 ceph-mon[29408]: osdmap e301: 8 total, 8 up, 8 in 2024-09-18T07:40:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:00 smithi137 ceph-mon[29408]: pgmap v488: 158 pgs: 158 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-09-18T07:40:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:00 smithi137 ceph-mon[29408]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-18T07:40:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:00 smithi137 ceph-mon[29408]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-18T07:40:00.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:00 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:40:00.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:00 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:40:00.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:00 smithi137 ceph-mon[39175]: osdmap e301: 8 total, 8 up, 8 in 2024-09-18T07:40:00.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:00 smithi137 ceph-mon[39175]: pgmap v488: 158 pgs: 158 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-09-18T07:40:00.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:00 smithi137 ceph-mon[39175]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-18T07:40:00.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:00 smithi137 ceph-mon[39175]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-18T07:40:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:00 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:40:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:00 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:40:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:00 smithi203 ceph-mon[32237]: osdmap e301: 8 total, 8 up, 8 in 2024-09-18T07:40:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:00 smithi203 ceph-mon[32237]: pgmap v488: 158 pgs: 158 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-09-18T07:40:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:00 smithi203 ceph-mon[32237]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-18T07:40:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:00 smithi203 ceph-mon[32237]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-18T07:40:00.486 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:00.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:00.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:00.630 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:00.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:00.674 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:00.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:00.751 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:00.775 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:00.801 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:40:00.801 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:40:00.802 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:00.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:00.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:00.904 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:00.948 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:00.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:00.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:01.049 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:01.076 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 26 16 55270 55254 132.795 92.25 0.00341337 0.00752383 2024-09-18T07:40:01.078 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:01.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:01.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:01.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:01.221 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:01.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:01.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:01.323 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:01.356 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:01.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:01.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:01.458 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:01.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:01.494 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:01.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:01.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:01.631 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:01.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:01.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:01.733 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:01.776 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:01.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:01.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:01.878 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:01.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:01.913 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:01.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:02.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:02.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:02.055 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:02.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:02.076 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 27 16 56728 56712 131.251 91.125 0.0167586 0.00761334 2024-09-18T07:40:02.156 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:02.195 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:02.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:02.216 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:40:02.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:02.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:02.340 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:02.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:02.379 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:02.423 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T07:40:02.442 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:02.542 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:02.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:02.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:02.645 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:02.675 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:02.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:02.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:02.696 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:40:02.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:02.818 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:02.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:02.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:02.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:02 smithi137 ceph-mon[29408]: pgmap v489: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-18T07:40:02.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:02 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4262330717' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:40:02.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:02 smithi137 ceph-mon[39175]: pgmap v489: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-18T07:40:02.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:02 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4262330717' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:40:02.920 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:02.957 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:02.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:02.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:02.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:02 smithi203 ceph-mon[32237]: pgmap v489: 158 pgs: 158 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-18T07:40:02.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:02 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4262330717' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:40:03.059 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:03.078 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 28 15 58205 58190 129.856 92.375 0.00684952 0.00769535 2024-09-18T07:40:03.092 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:03.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:03.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:03.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:03.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:03.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:03.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:03.339 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:03.371 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:03.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:03.472 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:03.507 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:03.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:03.610 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:03.634 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:40:03.638 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:40:03.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:03.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:03.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:03.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:03.793 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:03.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:03.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:03.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:03.911 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:03.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:03 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:40:03.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:03 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4262330717' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:40:03.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:03 smithi137 ceph-mon[39175]: osdmap e302: 8 total, 8 up, 8 in 2024-09-18T07:40:03.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:03 smithi137 ceph-mon[39175]: 4.3 scrub starts 2024-09-18T07:40:03.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:03 smithi137 ceph-mon[39175]: 4.3 scrub ok 2024-09-18T07:40:03.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:03 smithi137 ceph-mon[39175]: 6.b scrub starts 2024-09-18T07:40:03.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:03 smithi137 ceph-mon[39175]: 6.b scrub ok 2024-09-18T07:40:03.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:03 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:40:03.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:03 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4262330717' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:40:03.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:03 smithi137 ceph-mon[29408]: osdmap e302: 8 total, 8 up, 8 in 2024-09-18T07:40:03.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:03 smithi137 ceph-mon[29408]: 4.3 scrub starts 2024-09-18T07:40:03.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:03 smithi137 ceph-mon[29408]: 4.3 scrub ok 2024-09-18T07:40:03.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:03 smithi137 ceph-mon[29408]: 6.b scrub starts 2024-09-18T07:40:03.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:03 smithi137 ceph-mon[29408]: 6.b scrub ok 2024-09-18T07:40:03.941 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:03.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:03.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:03.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:03 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:40:03.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:03 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4262330717' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:40:03.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:03 smithi203 ceph-mon[32237]: osdmap e302: 8 total, 8 up, 8 in 2024-09-18T07:40:03.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:03 smithi203 ceph-mon[32237]: 4.3 scrub starts 2024-09-18T07:40:03.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:03 smithi203 ceph-mon[32237]: 4.3 scrub ok 2024-09-18T07:40:03.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:03 smithi203 ceph-mon[32237]: 6.b scrub starts 2024-09-18T07:40:03.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:03 smithi203 ceph-mon[32237]: 6.b scrub ok 2024-09-18T07:40:04.043 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:04.077 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 29 16 59725 59709 128.652 94.9375 0.0106277 0.00776691 2024-09-18T07:40:04.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:04.080 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:04.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:04.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:04.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:04.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:04.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:04.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:04.360 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:04.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:04.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:04.426 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-09-18T07:40:02.688579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3702004251,"num_objects":56678,"num_object_clones":0,"num_object_copies":170034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56678,"num_whiteouts":0,"num_read":1010,"num_read_kb":762,"num_write":56915,"num_write_kb":3614701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":5625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57841,"ondisk_log_size":57841,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":479,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14016484,"kb_used_data":11412444,"kb_used_omap":12,"kb_used_meta":2603891,"kb_avail":735944732,"statfs":{"total":767960285184,"available":753607405568,"internally_reserved":0,"allocated":11686342656,"data_stored":11681652681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2666384944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1047461888,"num_objects":15983,"num_object_clones":0,"num_object_copies":47949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15983,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":15983,"num_write_kb":1022912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.076963"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":516,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174314+0000","last_change":"2024-09-18T07:39:36.700902+0000","last_active":"2024-09-18T07:39:59.174314+0000","last_peered":"2024-09-18T07:39:59.174314+0000","last_clean":"2024-09-18T07:39:59.174314+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:39:59.174314+0000","last_undegraded":"2024-09-18T07:39:59.174314+0000","last_fullsized":"2024-09-18T07:39:59.174314+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:36.700832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:36.700832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:57.523488+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":582,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.173863+0000","last_change":"2024-09-18T07:38:52.141085+0000","last_active":"2024-09-18T07:39:59.173863+0000","last_peered":"2024-09-18T07:39:59.173863+0000","last_clean":"2024-09-18T07:39:59.173863+0000","last_became_active":"2024-09-18T07:37:43.061125+0000","last_became_peered":"2024-09-18T07:37:43.061125+0000","last_unstale":"2024-09-18T07:39:59.173863+0000","last_undegraded":"2024-09-18T07:39:59.173863+0000","last_fullsized":"2024-09-18T07:39:59.173863+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:38:52.140959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:38:52.140959+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:17.219305+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":451,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178347+0000","last_change":"2024-09-18T07:36:38.570405+0000","last_active":"2024-09-18T07:39:59.178347+0000","last_peered":"2024-09-18T07:39:59.178347+0000","last_clean":"2024-09-18T07:39:59.178347+0000","last_became_active":"2024-09-18T07:36:13.128805+0000","last_became_peered":"2024-09-18T07:36:13.128805+0000","last_unstale":"2024-09-18T07:39:59.178347+0000","last_undegraded":"2024-09-18T07:39:59.178347+0000","last_fullsized":"2024-09-18T07:39:59.178347+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:38.570339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:38.570339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":491,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174146+0000","last_change":"2024-09-18T07:32:03.953881+0000","last_active":"2024-09-18T07:39:59.174146+0000","last_peered":"2024-09-18T07:39:59.174146+0000","last_clean":"2024-09-18T07:39:59.174146+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:39:59.174146+0000","last_undegraded":"2024-09-18T07:39:59.174146+0000","last_fullsized":"2024-09-18T07:39:59.174146+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.953827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.953827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:44.967981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":410,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.170302+0000","last_change":"2024-09-18T07:39:33.737527+0000","last_active":"2024-09-18T07:39:59.170302+0000","last_peered":"2024-09-18T07:39:59.170302+0000","last_clean":"2024-09-18T07:39:59.170302+0000","last_became_active":"2024-09-18T07:37:43.056380+0000","last_became_peered":"2024-09-18T07:37:43.056380+0000","last_unstale":"2024-09-18T07:39:59.170302+0000","last_undegraded":"2024-09-18T07:39:59.170302+0000","last_fullsized":"2024-09-18T07:39:59.170302+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:33.737478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:33.737478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:52.567818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":517,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179715+0000","last_change":"2024-09-18T07:37:43.066814+0000","last_active":"2024-09-18T07:39:59.179715+0000","last_peered":"2024-09-18T07:39:59.179715+0000","last_clean":"2024-09-18T07:39:59.179715+0000","last_became_active":"2024-09-18T07:37:43.066508+0000","last_became_peered":"2024-09-18T07:37:43.066508+0000","last_unstale":"2024-09-18T07:39:59.179715+0000","last_undegraded":"2024-09-18T07:39:59.179715+0000","last_fullsized":"2024-09-18T07:39:59.179715+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.006488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.006488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:46.846955+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":523,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179288+0000","last_change":"2024-09-18T07:33:22.032321+0000","last_active":"2024-09-18T07:39:59.179288+0000","last_peered":"2024-09-18T07:39:59.179288+0000","last_clean":"2024-09-18T07:39:59.179288+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:39:59.179288+0000","last_undegraded":"2024-09-18T07:39:59.179288+0000","last_fullsized":"2024-09-18T07:39:59.179288+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:22.032110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.032110+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:47.872878+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":517,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.173852+0000","last_change":"2024-09-18T07:37:43.052675+0000","last_active":"2024-09-18T07:39:59.173852+0000","last_peered":"2024-09-18T07:39:59.173852+0000","last_clean":"2024-09-18T07:39:59.173852+0000","last_became_active":"2024-09-18T07:37:43.052346+0000","last_became_peered":"2024-09-18T07:37:43.052346+0000","last_unstale":"2024-09-18T07:39:59.173852+0000","last_undegraded":"2024-09-18T07:39:59.173852+0000","last_fullsized":"2024-09-18T07:39:59.173852+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:30.995282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:30.995282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:11.031402+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":523,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179285+0000","last_change":"2024-09-18T07:37:43.080879+0000","last_active":"2024-09-18T07:39:59.179285+0000","last_peered":"2024-09-18T07:39:59.179285+0000","last_clean":"2024-09-18T07:39:59.179285+0000","last_became_active":"2024-09-18T07:37:43.080314+0000","last_became_peered":"2024-09-18T07:37:43.080314+0000","last_unstale":"2024-09-18T07:39:59.179285+0000","last_undegraded":"2024-09-18T07:39:59.179285+0000","last_fullsized":"2024-09-18T07:39:59.179285+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:00.392466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.392466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:20.640427+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":515,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174367+0000","last_change":"2024-09-18T07:32:02.002919+0000","last_active":"2024-09-18T07:39:59.174367+0000","last_peered":"2024-09-18T07:39:59.174367+0000","last_clean":"2024-09-18T07:39:59.174367+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:39:59.174367+0000","last_undegraded":"2024-09-18T07:39:59.174367+0000","last_fullsized":"2024-09-18T07:39:59.174367+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.002811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.002811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:43.931181+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":507,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178674+0000","last_change":"2024-09-18T07:35:04.267623+0000","last_active":"2024-09-18T07:39:59.178674+0000","last_peered":"2024-09-18T07:39:59.178674+0000","last_clean":"2024-09-18T07:39:59.178674+0000","last_became_active":"2024-09-18T07:35:01.413443+0000","last_became_peered":"2024-09-18T07:35:01.413443+0000","last_unstale":"2024-09-18T07:39:59.178674+0000","last_undegraded":"2024-09-18T07:39:59.178674+0000","last_fullsized":"2024-09-18T07:39:59.178674+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:04.267580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:04.267580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:22.587151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":501,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174634+0000","last_change":"2024-09-18T07:36:10.551960+0000","last_active":"2024-09-18T07:39:59.174634+0000","last_peered":"2024-09-18T07:39:59.174634+0000","last_clean":"2024-09-18T07:39:59.174634+0000","last_became_active":"2024-09-18T07:35:01.410284+0000","last_became_peered":"2024-09-18T07:35:01.410284+0000","last_unstale":"2024-09-18T07:39:59.174634+0000","last_undegraded":"2024-09-18T07:39:59.174634+0000","last_fullsized":"2024-09-18T07:39:59.174634+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:10.551889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:10.551889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:58.693520+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":493,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178806+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:39:59.178806+0000","last_peered":"2024-09-18T07:39:59.178806+0000","last_clean":"2024-09-18T07:39:59.178806+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:39:59.178806+0000","last_undegraded":"2024-09-18T07:39:59.178806+0000","last_fullsized":"2024-09-18T07:39:59.178806+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:10.625350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":530,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174282+0000","last_change":"2024-09-18T07:39:26.028935+0000","last_active":"2024-09-18T07:39:59.174282+0000","last_peered":"2024-09-18T07:39:59.174282+0000","last_clean":"2024-09-18T07:39:59.174282+0000","last_became_active":"2024-09-18T07:36:02.526212+0000","last_became_peered":"2024-09-18T07:36:02.526212+0000","last_unstale":"2024-09-18T07:39:59.174282+0000","last_undegraded":"2024-09-18T07:39:59.174282+0000","last_fullsized":"2024-09-18T07:39:59.174282+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-18T07:39:26.028884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:26.028884+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:28.305457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":276,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177448+0000","last_change":"2024-09-18T07:38:54.458848+0000","last_active":"2024-09-18T07:39:59.177448+0000","last_peered":"2024-09-18T07:39:59.177448+0000","last_clean":"2024-09-18T07:39:59.177448+0000","last_became_active":"2024-09-18T07:35:01.398605+0000","last_became_peered":"2024-09-18T07:35:01.398605+0000","last_unstale":"2024-09-18T07:39:59.177448+0000","last_undegraded":"2024-09-18T07:39:59.177448+0000","last_fullsized":"2024-09-18T07:39:59.177448+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:54.458787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:54.458787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.620791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":513,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178535+0000","last_change":"2024-09-18T07:39:30.429935+0000","last_active":"2024-09-18T07:39:59.178535+0000","last_peered":"2024-09-18T07:39:59.178535+0000","last_clean":"2024-09-18T07:39:59.178535+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:39:59.178535+0000","last_undegraded":"2024-09-18T07:39:59.178535+0000","last_fullsized":"2024-09-18T07:39:59.178535+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:30.429890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:30.429890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:38.713228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":522,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.175159+0000","last_change":"2024-09-18T07:38:56.096769+0000","last_active":"2024-09-18T07:39:59.175159+0000","last_peered":"2024-09-18T07:39:59.175159+0000","last_clean":"2024-09-18T07:39:59.175159+0000","last_became_active":"2024-09-18T07:37:43.067568+0000","last_became_peered":"2024-09-18T07:37:43.067568+0000","last_unstale":"2024-09-18T07:39:59.175159+0000","last_undegraded":"2024-09-18T07:39:59.175159+0000","last_fullsized":"2024-09-18T07:39:59.175159+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:56.096591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:56.096591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:24.052212+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":521,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.184494+0000","last_change":"2024-09-18T07:36:06.523357+0000","last_active":"2024-09-18T07:39:59.184494+0000","last_peered":"2024-09-18T07:39:59.184494+0000","last_clean":"2024-09-18T07:39:59.184494+0000","last_became_active":"2024-09-18T07:35:01.414088+0000","last_became_peered":"2024-09-18T07:35:01.414088+0000","last_unstale":"2024-09-18T07:39:59.184494+0000","last_undegraded":"2024-09-18T07:39:59.184494+0000","last_fullsized":"2024-09-18T07:39:59.184494+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:06.523292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.523292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":578,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177394+0000","last_change":"2024-09-18T07:39:04.314241+0000","last_active":"2024-09-18T07:39:59.177394+0000","last_peered":"2024-09-18T07:39:59.177394+0000","last_clean":"2024-09-18T07:39:59.177394+0000","last_became_active":"2024-09-18T07:37:28.002858+0000","last_became_peered":"2024-09-18T07:37:28.002858+0000","last_unstale":"2024-09-18T07:39:59.177394+0000","last_undegraded":"2024-09-18T07:39:59.177394+0000","last_fullsized":"2024-09-18T07:39:59.177394+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:39:04.313784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:04.313784+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:09.030441+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":194,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174551+0000","last_change":"2024-09-18T07:36:13.116882+0000","last_active":"2024-09-18T07:39:59.174551+0000","last_peered":"2024-09-18T07:39:59.174551+0000","last_clean":"2024-09-18T07:39:59.174551+0000","last_became_active":"2024-09-18T07:36:13.116416+0000","last_became_peered":"2024-09-18T07:36:13.116416+0000","last_unstale":"2024-09-18T07:39:59.174551+0000","last_undegraded":"2024-09-18T07:39:59.174551+0000","last_fullsized":"2024-09-18T07:39:59.174551+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:10.074367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:10.074367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:21.035466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":513,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179926+0000","last_change":"2024-09-18T07:37:43.067816+0000","last_active":"2024-09-18T07:39:59.179926+0000","last_peered":"2024-09-18T07:39:59.179926+0000","last_clean":"2024-09-18T07:39:59.179926+0000","last_became_active":"2024-09-18T07:37:43.067493+0000","last_became_peered":"2024-09-18T07:37:43.067493+0000","last_unstale":"2024-09-18T07:39:59.179926+0000","last_undegraded":"2024-09-18T07:39:59.179926+0000","last_fullsized":"2024-09-18T07:39:59.179926+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:34.992350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:34.992350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:11.064160+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":519,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174541+0000","last_change":"2024-09-18T07:39:00.130395+0000","last_active":"2024-09-18T07:39:59.174541+0000","last_peered":"2024-09-18T07:39:59.174541+0000","last_clean":"2024-09-18T07:39:59.174541+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:39:59.174541+0000","last_undegraded":"2024-09-18T07:39:59.174541+0000","last_fullsized":"2024-09-18T07:39:59.174541+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:00.130344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.130344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:22.555080+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":165,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177676+0000","last_change":"2024-09-18T07:37:43.756858+0000","last_active":"2024-09-18T07:39:59.177676+0000","last_peered":"2024-09-18T07:39:59.177676+0000","last_clean":"2024-09-18T07:39:59.177676+0000","last_became_active":"2024-09-18T07:37:27.981705+0000","last_became_peered":"2024-09-18T07:37:27.981705+0000","last_unstale":"2024-09-18T07:39:59.177676+0000","last_undegraded":"2024-09-18T07:39:59.177676+0000","last_fullsized":"2024-09-18T07:39:59.177676+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:37:43.756403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:37:43.756403+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:53.387865+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":495,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174398+0000","last_change":"2024-09-18T07:39:37.930497+0000","last_active":"2024-09-18T07:39:59.174398+0000","last_peered":"2024-09-18T07:39:59.174398+0000","last_clean":"2024-09-18T07:39:59.174398+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:39:59.174398+0000","last_undegraded":"2024-09-18T07:39:59.174398+0000","last_fullsized":"2024-09-18T07:39:59.174398+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.930435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.930435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:55.595621+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":288,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177872+0000","last_change":"2024-09-18T07:38:56.497595+0000","last_active":"2024-09-18T07:39:59.177872+0000","last_peered":"2024-09-18T07:39:59.177872+0000","last_clean":"2024-09-18T07:39:59.177872+0000","last_became_active":"2024-09-18T07:35:01.413110+0000","last_became_peered":"2024-09-18T07:35:01.413110+0000","last_unstale":"2024-09-18T07:39:59.177872+0000","last_undegraded":"2024-09-18T07:39:59.177872+0000","last_fullsized":"2024-09-18T07:39:59.177872+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:38:56.497544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:56.497544+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:05.310758+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":211,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179155+0000","last_change":"2024-09-18T07:38:50.918758+0000","last_active":"2024-09-18T07:39:59.179155+0000","last_peered":"2024-09-18T07:39:59.179155+0000","last_clean":"2024-09-18T07:39:59.179155+0000","last_became_active":"2024-09-18T07:36:13.116295+0000","last_became_peered":"2024-09-18T07:36:13.116295+0000","last_unstale":"2024-09-18T07:39:59.179155+0000","last_undegraded":"2024-09-18T07:39:59.179155+0000","last_fullsized":"2024-09-18T07:39:59.179155+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:38:50.918686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:38:50.918686+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:19.981998+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":557,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.171061+0000","last_change":"2024-09-18T07:35:57.488019+0000","last_active":"2024-09-18T07:39:59.171061+0000","last_peered":"2024-09-18T07:39:59.171061+0000","last_clean":"2024-09-18T07:39:59.171061+0000","last_became_active":"2024-09-18T07:35:01.410926+0000","last_became_peered":"2024-09-18T07:35:01.410926+0000","last_unstale":"2024-09-18T07:39:59.171061+0000","last_undegraded":"2024-09-18T07:39:59.171061+0000","last_fullsized":"2024-09-18T07:39:59.171061+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:35:57.487960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.487960+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:52.681351+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"240'12","reported_seq":175,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.194503+0000","last_change":"2024-09-18T07:39:33.979364+0000","last_active":"2024-09-18T07:39:59.194503+0000","last_peered":"2024-09-18T07:39:59.194503+0000","last_clean":"2024-09-18T07:39:59.194503+0000","last_became_active":"2024-09-18T07:37:43.060715+0000","last_became_peered":"2024-09-18T07:37:43.060715+0000","last_unstale":"2024-09-18T07:39:59.194503+0000","last_undegraded":"2024-09-18T07:39:59.194503+0000","last_fullsized":"2024-09-18T07:39:59.194503+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:39:33.979291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:33.979291+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:39.389550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"301'3625","reported_seq":3660,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:02.074124+0000","last_change":"2024-09-18T07:39:29.700748+0000","last_active":"2024-09-18T07:40:02.074124+0000","last_peered":"2024-09-18T07:40:02.074124+0000","last_clean":"2024-09-18T07:40:02.074124+0000","last_became_active":"2024-09-18T07:39:29.700477+0000","last_became_peered":"2024-09-18T07:39:29.700477+0000","last_unstale":"2024-09-18T07:40:02.074124+0000","last_undegraded":"2024-09-18T07:40:02.074124+0000","last_fullsized":"2024-09-18T07:40:02.074124+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3625,"log_dups_size":0,"ondisk_log_size":3625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.879265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237568000,"num_objects":3625,"num_object_clones":0,"num_object_copies":10875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3625,"num_write_kb":232000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":72,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178440+0000","last_change":"2024-09-18T07:39:06.546309+0000","last_active":"2024-09-18T07:39:59.178440+0000","last_peered":"2024-09-18T07:39:59.178440+0000","last_clean":"2024-09-18T07:39:59.178440+0000","last_became_active":"2024-09-18T07:39:04.962047+0000","last_became_peered":"2024-09-18T07:39:04.962047+0000","last_unstale":"2024-09-18T07:39:59.178440+0000","last_undegraded":"2024-09-18T07:39:59.178440+0000","last_fullsized":"2024-09-18T07:39:59.178440+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:06.546237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:06.546237+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:25.791546+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":247,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.180241+0000","last_change":"2024-09-18T07:38:54.967706+0000","last_active":"2024-09-18T07:39:59.180241+0000","last_peered":"2024-09-18T07:39:59.180241+0000","last_clean":"2024-09-18T07:39:59.180241+0000","last_became_active":"2024-09-18T07:36:13.119098+0000","last_became_peered":"2024-09-18T07:36:13.119098+0000","last_unstale":"2024-09-18T07:39:59.180241+0000","last_undegraded":"2024-09-18T07:39:59.180241+0000","last_fullsized":"2024-09-18T07:39:59.180241+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:38:54.967643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:38:54.967643+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:18.692985+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":541,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177450+0000","last_change":"2024-09-18T07:39:29.049261+0000","last_active":"2024-09-18T07:39:59.177450+0000","last_peered":"2024-09-18T07:39:59.177450+0000","last_clean":"2024-09-18T07:39:59.177450+0000","last_became_active":"2024-09-18T07:35:01.407339+0000","last_became_peered":"2024-09-18T07:35:01.407339+0000","last_unstale":"2024-09-18T07:39:59.177450+0000","last_undegraded":"2024-09-18T07:39:59.177450+0000","last_fullsized":"2024-09-18T07:39:59.177450+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.049209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.049209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:31.230030+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":630,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:01.464597+0000","last_change":"2024-09-18T07:36:35.210213+0000","last_active":"2024-09-18T07:40:01.464597+0000","last_peered":"2024-09-18T07:40:01.464597+0000","last_clean":"2024-09-18T07:40:01.464597+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:40:01.464597+0000","last_undegraded":"2024-09-18T07:40:01.464597+0000","last_fullsized":"2024-09-18T07:40:01.464597+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:35.209823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:35.209823+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":497,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178665+0000","last_change":"2024-09-18T07:36:07.588483+0000","last_active":"2024-09-18T07:39:59.178665+0000","last_peered":"2024-09-18T07:39:59.178665+0000","last_clean":"2024-09-18T07:39:59.178665+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:39:59.178665+0000","last_undegraded":"2024-09-18T07:39:59.178665+0000","last_fullsized":"2024-09-18T07:39:59.178665+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:07.588440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:07.588440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:58.358099+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"301'3580","reported_seq":3615,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:01.515040+0000","last_change":"2024-09-18T07:39:29.704994+0000","last_active":"2024-09-18T07:40:01.515040+0000","last_peered":"2024-09-18T07:40:01.515040+0000","last_clean":"2024-09-18T07:40:01.515040+0000","last_became_active":"2024-09-18T07:39:29.704590+0000","last_became_peered":"2024-09-18T07:39:29.704590+0000","last_unstale":"2024-09-18T07:40:01.515040+0000","last_undegraded":"2024-09-18T07:40:01.515040+0000","last_fullsized":"2024-09-18T07:40:01.515040+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3580,"log_dups_size":0,"ondisk_log_size":3580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:33.304050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234618880,"num_objects":3580,"num_object_clones":0,"num_object_copies":10740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3580,"num_write_kb":229120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":72,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174834+0000","last_change":"2024-09-18T07:39:05.085099+0000","last_active":"2024-09-18T07:39:59.174834+0000","last_peered":"2024-09-18T07:39:59.174834+0000","last_clean":"2024-09-18T07:39:59.174834+0000","last_became_active":"2024-09-18T07:39:04.955645+0000","last_became_peered":"2024-09-18T07:39:04.955645+0000","last_unstale":"2024-09-18T07:39:59.174834+0000","last_undegraded":"2024-09-18T07:39:59.174834+0000","last_fullsized":"2024-09-18T07:39:59.174834+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.084956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.084956+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:18.027466+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":597,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177123+0000","last_change":"2024-09-18T07:39:00.919075+0000","last_active":"2024-09-18T07:39:59.177123+0000","last_peered":"2024-09-18T07:39:59.177123+0000","last_clean":"2024-09-18T07:39:59.177123+0000","last_became_active":"2024-09-18T07:35:01.418909+0000","last_became_peered":"2024-09-18T07:35:01.418909+0000","last_unstale":"2024-09-18T07:39:59.177123+0000","last_undegraded":"2024-09-18T07:39:59.177123+0000","last_fullsized":"2024-09-18T07:39:59.177123+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:39:00.918860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.918860+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:29.657171+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":533,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.180598+0000","last_change":"2024-09-18T07:39:06.987818+0000","last_active":"2024-09-18T07:39:59.180598+0000","last_peered":"2024-09-18T07:39:59.180598+0000","last_clean":"2024-09-18T07:39:59.180598+0000","last_became_active":"2024-09-18T07:35:01.407422+0000","last_became_peered":"2024-09-18T07:35:01.407422+0000","last_unstale":"2024-09-18T07:39:59.180598+0000","last_undegraded":"2024-09-18T07:39:59.180598+0000","last_fullsized":"2024-09-18T07:39:59.180598+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:06.987428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:06.987428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:25.403434+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":518,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.173913+0000","last_change":"2024-09-18T07:36:30.242479+0000","last_active":"2024-09-18T07:39:59.173913+0000","last_peered":"2024-09-18T07:39:59.173913+0000","last_clean":"2024-09-18T07:39:59.173913+0000","last_became_active":"2024-09-18T07:35:01.401024+0000","last_became_peered":"2024-09-18T07:35:01.401024+0000","last_unstale":"2024-09-18T07:39:59.173913+0000","last_undegraded":"2024-09-18T07:39:59.173913+0000","last_fullsized":"2024-09-18T07:39:59.173913+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:30.242420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:30.242420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:57.178241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":484,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174559+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:39:59.174559+0000","last_peered":"2024-09-18T07:39:59.174559+0000","last_clean":"2024-09-18T07:39:59.174559+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:39:59.174559+0000","last_undegraded":"2024-09-18T07:39:59.174559+0000","last_fullsized":"2024-09-18T07:39:59.174559+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:41.905293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"301'3647","reported_seq":3682,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:02.085752+0000","last_change":"2024-09-18T07:39:29.703301+0000","last_active":"2024-09-18T07:40:02.085752+0000","last_peered":"2024-09-18T07:40:02.085752+0000","last_clean":"2024-09-18T07:40:02.085752+0000","last_became_active":"2024-09-18T07:39:29.702953+0000","last_became_peered":"2024-09-18T07:39:29.702953+0000","last_unstale":"2024-09-18T07:40:02.085752+0000","last_undegraded":"2024-09-18T07:40:02.085752+0000","last_fullsized":"2024-09-18T07:40:02.085752+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3647,"log_dups_size":0,"ondisk_log_size":3647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.620109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239009792,"num_objects":3647,"num_object_clones":0,"num_object_copies":10941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3647,"num_write_kb":233408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":77,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179445+0000","last_change":"2024-09-18T07:39:05.959886+0000","last_active":"2024-09-18T07:39:59.179445+0000","last_peered":"2024-09-18T07:39:59.179445+0000","last_clean":"2024-09-18T07:39:59.179445+0000","last_became_active":"2024-09-18T07:39:04.945385+0000","last_became_peered":"2024-09-18T07:39:04.945385+0000","last_unstale":"2024-09-18T07:39:59.179445+0000","last_undegraded":"2024-09-18T07:39:59.179445+0000","last_fullsized":"2024-09-18T07:39:59.179445+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.959822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.959822+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:34.831011+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":567,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.173925+0000","last_change":"2024-09-18T07:37:43.153068+0000","last_active":"2024-09-18T07:39:59.173925+0000","last_peered":"2024-09-18T07:39:59.173925+0000","last_clean":"2024-09-18T07:39:59.173925+0000","last_became_active":"2024-09-18T07:37:43.096236+0000","last_became_peered":"2024-09-18T07:37:43.096236+0000","last_unstale":"2024-09-18T07:39:59.173925+0000","last_undegraded":"2024-09-18T07:39:59.173925+0000","last_fullsized":"2024-09-18T07:39:59.173925+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-18T07:35:00.882191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.882191+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:55.159776+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":509,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.171333+0000","last_change":"2024-09-18T07:39:28.750779+0000","last_active":"2024-09-18T07:39:59.171333+0000","last_peered":"2024-09-18T07:39:59.171333+0000","last_clean":"2024-09-18T07:39:59.171333+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:39:59.171333+0000","last_undegraded":"2024-09-18T07:39:59.171333+0000","last_fullsized":"2024-09-18T07:39:59.171333+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:39:28.750720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.750720+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:52.195932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":621,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.176970+0000","last_change":"2024-09-18T07:36:05.016008+0000","last_active":"2024-09-18T07:39:59.176970+0000","last_peered":"2024-09-18T07:39:59.176970+0000","last_clean":"2024-09-18T07:39:59.176970+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:39:59.176970+0000","last_undegraded":"2024-09-18T07:39:59.176970+0000","last_fullsized":"2024-09-18T07:39:59.176970+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:05.015955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.015955+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:48.836489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":298,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177268+0000","last_change":"2024-09-18T07:38:58.528134+0000","last_active":"2024-09-18T07:39:59.177268+0000","last_peered":"2024-09-18T07:39:59.177268+0000","last_clean":"2024-09-18T07:39:59.177268+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:39:59.177268+0000","last_undegraded":"2024-09-18T07:39:59.177268+0000","last_fullsized":"2024-09-18T07:39:59.177268+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:58.528059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:58.528059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:10.915828+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"301'3509","reported_seq":3543,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:01.810030+0000","last_change":"2024-09-18T07:39:29.702619+0000","last_active":"2024-09-18T07:40:01.810030+0000","last_peered":"2024-09-18T07:40:01.810030+0000","last_clean":"2024-09-18T07:40:01.810030+0000","last_became_active":"2024-09-18T07:39:29.702301+0000","last_became_peered":"2024-09-18T07:39:29.702301+0000","last_unstale":"2024-09-18T07:40:01.810030+0000","last_undegraded":"2024-09-18T07:40:01.810030+0000","last_fullsized":"2024-09-18T07:40:01.810030+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3509,"log_dups_size":0,"ondisk_log_size":3509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:33.630737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229965824,"num_objects":3509,"num_object_clones":0,"num_object_copies":10527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3509,"num_write_kb":224576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":240,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.194268+0000","last_change":"2024-09-18T07:39:06.342429+0000","last_active":"2024-09-18T07:39:59.194268+0000","last_peered":"2024-09-18T07:39:59.194268+0000","last_clean":"2024-09-18T07:39:59.194268+0000","last_became_active":"2024-09-18T07:39:04.941752+0000","last_became_peered":"2024-09-18T07:39:04.941752+0000","last_unstale":"2024-09-18T07:39:59.194268+0000","last_undegraded":"2024-09-18T07:39:59.194268+0000","last_fullsized":"2024-09-18T07:39:59.194268+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:06.342370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:06.342370+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:36.082977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":229,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178903+0000","last_change":"2024-09-18T07:36:13.119192+0000","last_active":"2024-09-18T07:39:59.178903+0000","last_peered":"2024-09-18T07:39:59.178903+0000","last_clean":"2024-09-18T07:39:59.178903+0000","last_became_active":"2024-09-18T07:36:13.118876+0000","last_became_peered":"2024-09-18T07:36:13.118876+0000","last_unstale":"2024-09-18T07:39:59.178903+0000","last_undegraded":"2024-09-18T07:39:59.178903+0000","last_fullsized":"2024-09-18T07:39:59.178903+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.613045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":518,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179355+0000","last_change":"2024-09-18T07:39:31.418964+0000","last_active":"2024-09-18T07:39:59.179355+0000","last_peered":"2024-09-18T07:39:59.179355+0000","last_clean":"2024-09-18T07:39:59.179355+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:39:59.179355+0000","last_undegraded":"2024-09-18T07:39:59.179355+0000","last_fullsized":"2024-09-18T07:39:59.179355+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:31.418910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:31.418910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:40.383317+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"240'10","reported_seq":640,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.183920+0000","last_change":"2024-09-18T07:37:43.170266+0000","last_active":"2024-09-18T07:39:59.183920+0000","last_peered":"2024-09-18T07:39:59.183920+0000","last_clean":"2024-09-18T07:39:59.183920+0000","last_became_active":"2024-09-18T07:37:43.167850+0000","last_became_peered":"2024-09-18T07:37:43.167850+0000","last_unstale":"2024-09-18T07:39:59.183920+0000","last_undegraded":"2024-09-18T07:39:59.183920+0000","last_fullsized":"2024-09-18T07:39:59.183920+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:05.525762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.525762+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:49.695027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":488,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174768+0000","last_change":"2024-09-18T07:39:00.759483+0000","last_active":"2024-09-18T07:39:59.174768+0000","last_peered":"2024-09-18T07:39:59.174768+0000","last_clean":"2024-09-18T07:39:59.174768+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:39:59.174768+0000","last_undegraded":"2024-09-18T07:39:59.174768+0000","last_fullsized":"2024-09-18T07:39:59.174768+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:00.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:10.972661+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"301'3376","reported_seq":3411,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:01.476876+0000","last_change":"2024-09-18T07:39:29.703780+0000","last_active":"2024-09-18T07:40:01.476876+0000","last_peered":"2024-09-18T07:40:01.476876+0000","last_clean":"2024-09-18T07:40:01.476876+0000","last_became_active":"2024-09-18T07:39:29.703475+0000","last_became_peered":"2024-09-18T07:39:29.703475+0000","last_unstale":"2024-09-18T07:40:01.476876+0000","last_undegraded":"2024-09-18T07:40:01.476876+0000","last_fullsized":"2024-09-18T07:40:01.476876+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3376,"log_dups_size":0,"ondisk_log_size":3376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:58.380502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221249536,"num_objects":3376,"num_object_clones":0,"num_object_copies":10128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3376,"num_write_kb":216064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":240,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.194054+0000","last_change":"2024-09-18T07:39:07.357549+0000","last_active":"2024-09-18T07:39:59.194054+0000","last_peered":"2024-09-18T07:39:59.194054+0000","last_clean":"2024-09-18T07:39:59.194054+0000","last_became_active":"2024-09-18T07:39:04.948571+0000","last_became_peered":"2024-09-18T07:39:04.948571+0000","last_unstale":"2024-09-18T07:39:59.194054+0000","last_undegraded":"2024-09-18T07:39:59.194054+0000","last_fullsized":"2024-09-18T07:39:59.194054+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:07.357480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:07.357480+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:12.531366+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":590,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.184154+0000","last_change":"2024-09-18T07:36:09.498239+0000","last_active":"2024-09-18T07:39:59.184154+0000","last_peered":"2024-09-18T07:39:59.184154+0000","last_clean":"2024-09-18T07:39:59.184154+0000","last_became_active":"2024-09-18T07:35:01.415275+0000","last_became_peered":"2024-09-18T07:35:01.415275+0000","last_unstale":"2024-09-18T07:39:59.184154+0000","last_undegraded":"2024-09-18T07:39:59.184154+0000","last_fullsized":"2024-09-18T07:39:59.184154+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:36:09.498142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.498142+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":202,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177485+0000","last_change":"2024-09-18T07:37:46.831436+0000","last_active":"2024-09-18T07:39:59.177485+0000","last_peered":"2024-09-18T07:39:59.177485+0000","last_clean":"2024-09-18T07:39:59.177485+0000","last_became_active":"2024-09-18T07:36:13.118553+0000","last_became_peered":"2024-09-18T07:36:13.118553+0000","last_unstale":"2024-09-18T07:39:59.177485+0000","last_undegraded":"2024-09-18T07:39:59.177485+0000","last_fullsized":"2024-09-18T07:39:59.177485+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:46.831373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:37:46.831373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:16.388148+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":135,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.192938+0000","last_change":"2024-09-18T07:37:43.070306+0000","last_active":"2024-09-18T07:39:59.192938+0000","last_peered":"2024-09-18T07:39:59.192938+0000","last_clean":"2024-09-18T07:39:59.192938+0000","last_became_active":"2024-09-18T07:37:43.070009+0000","last_became_peered":"2024-09-18T07:37:43.070009+0000","last_unstale":"2024-09-18T07:39:59.192938+0000","last_undegraded":"2024-09-18T07:39:59.192938+0000","last_fullsized":"2024-09-18T07:39:59.192938+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.202683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":135,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.194339+0000","last_change":"2024-09-18T07:37:43.052077+0000","last_active":"2024-09-18T07:39:59.194339+0000","last_peered":"2024-09-18T07:39:59.194339+0000","last_clean":"2024-09-18T07:39:59.194339+0000","last_became_active":"2024-09-18T07:37:43.051740+0000","last_became_peered":"2024-09-18T07:37:43.051740+0000","last_unstale":"2024-09-18T07:39:59.194339+0000","last_undegraded":"2024-09-18T07:39:59.194339+0000","last_fullsized":"2024-09-18T07:39:59.194339+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:09.269847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"301'3548","reported_seq":3583,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:02.457297+0000","last_change":"2024-09-18T07:39:29.702264+0000","last_active":"2024-09-18T07:40:02.457297+0000","last_peered":"2024-09-18T07:40:02.457297+0000","last_clean":"2024-09-18T07:40:02.457297+0000","last_became_active":"2024-09-18T07:39:29.701239+0000","last_became_peered":"2024-09-18T07:39:29.701239+0000","last_unstale":"2024-09-18T07:40:02.457297+0000","last_undegraded":"2024-09-18T07:40:02.457297+0000","last_fullsized":"2024-09-18T07:40:02.457297+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:51.463801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232521728,"num_objects":3548,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3548,"num_write_kb":227072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":72,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178149+0000","last_change":"2024-09-18T07:39:05.509522+0000","last_active":"2024-09-18T07:39:59.178149+0000","last_peered":"2024-09-18T07:39:59.178149+0000","last_clean":"2024-09-18T07:39:59.178149+0000","last_became_active":"2024-09-18T07:39:04.950564+0000","last_became_peered":"2024-09-18T07:39:04.950564+0000","last_unstale":"2024-09-18T07:39:59.178149+0000","last_undegraded":"2024-09-18T07:39:59.178149+0000","last_fullsized":"2024-09-18T07:39:59.178149+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.509453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.509453+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:28.588623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"211'29","reported_seq":318,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177340+0000","last_change":"2024-09-18T07:39:00.207829+0000","last_active":"2024-09-18T07:39:59.177340+0000","last_peered":"2024-09-18T07:39:59.177340+0000","last_clean":"2024-09-18T07:39:59.177340+0000","last_became_active":"2024-09-18T07:36:13.115278+0000","last_became_peered":"2024-09-18T07:36:13.115278+0000","last_unstale":"2024-09-18T07:39:59.177340+0000","last_undegraded":"2024-09-18T07:39:59.177340+0000","last_fullsized":"2024-09-18T07:39:59.177340+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'29","last_scrub_stamp":"2024-09-18T07:39:00.207506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.207506+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:08.496876+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":518,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174482+0000","last_change":"2024-09-18T07:37:43.061414+0000","last_active":"2024-09-18T07:39:59.174482+0000","last_peered":"2024-09-18T07:39:59.174482+0000","last_clean":"2024-09-18T07:39:59.174482+0000","last_became_active":"2024-09-18T07:37:43.060930+0000","last_became_peered":"2024-09-18T07:37:43.060930+0000","last_unstale":"2024-09-18T07:39:59.174482+0000","last_undegraded":"2024-09-18T07:39:59.174482+0000","last_fullsized":"2024-09-18T07:39:59.174482+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:06.908713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.908713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:56.182156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":542,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179088+0000","last_change":"2024-09-18T07:37:43.155837+0000","last_active":"2024-09-18T07:39:59.179088+0000","last_peered":"2024-09-18T07:39:59.179088+0000","last_clean":"2024-09-18T07:39:59.179088+0000","last_became_active":"2024-09-18T07:37:43.076595+0000","last_became_peered":"2024-09-18T07:37:43.076595+0000","last_unstale":"2024-09-18T07:39:59.179088+0000","last_undegraded":"2024-09-18T07:39:59.179088+0000","last_fullsized":"2024-09-18T07:39:59.179088+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:36:01.529677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.529677+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:55.397407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":492,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174613+0000","last_change":"2024-09-18T07:39:25.741714+0000","last_active":"2024-09-18T07:39:59.174613+0000","last_peered":"2024-09-18T07:39:59.174613+0000","last_clean":"2024-09-18T07:39:59.174613+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:39:59.174613+0000","last_undegraded":"2024-09-18T07:39:59.174613+0000","last_fullsized":"2024-09-18T07:39:59.174613+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:25.741643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.741643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:35.816396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":513,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179478+0000","last_change":"2024-09-18T07:38:59.871379+0000","last_active":"2024-09-18T07:39:59.179478+0000","last_peered":"2024-09-18T07:39:59.179478+0000","last_clean":"2024-09-18T07:39:59.179478+0000","last_became_active":"2024-09-18T07:35:01.418809+0000","last_became_peered":"2024-09-18T07:35:01.418809+0000","last_unstale":"2024-09-18T07:39:59.179478+0000","last_undegraded":"2024-09-18T07:39:59.179478+0000","last_fullsized":"2024-09-18T07:39:59.179478+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:59.871308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:59.871308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:00.245683+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"301'3570","reported_seq":3605,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:01.805275+0000","last_change":"2024-09-18T07:39:29.702743+0000","last_active":"2024-09-18T07:40:01.805275+0000","last_peered":"2024-09-18T07:40:01.805275+0000","last_clean":"2024-09-18T07:40:01.805275+0000","last_became_active":"2024-09-18T07:39:29.702461+0000","last_became_peered":"2024-09-18T07:39:29.702461+0000","last_unstale":"2024-09-18T07:40:01.805275+0000","last_undegraded":"2024-09-18T07:40:01.805275+0000","last_fullsized":"2024-09-18T07:40:01.805275+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3570,"log_dups_size":0,"ondisk_log_size":3570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:31.412469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233963520,"num_objects":3570,"num_object_clones":0,"num_object_copies":10710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3570,"num_write_kb":228480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":340,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177026+0000","last_change":"2024-09-18T07:39:06.465696+0000","last_active":"2024-09-18T07:39:59.177026+0000","last_peered":"2024-09-18T07:39:59.177026+0000","last_clean":"2024-09-18T07:39:59.177026+0000","last_became_active":"2024-09-18T07:39:04.944921+0000","last_became_peered":"2024-09-18T07:39:04.944921+0000","last_unstale":"2024-09-18T07:39:59.177026+0000","last_undegraded":"2024-09-18T07:39:59.177026+0000","last_fullsized":"2024-09-18T07:39:59.177026+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:06.465620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:06.465620+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:21.933997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":241,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179627+0000","last_change":"2024-09-18T07:36:13.118278+0000","last_active":"2024-09-18T07:39:59.179627+0000","last_peered":"2024-09-18T07:39:59.179627+0000","last_clean":"2024-09-18T07:39:59.179627+0000","last_became_active":"2024-09-18T07:36:13.117924+0000","last_became_peered":"2024-09-18T07:36:13.117924+0000","last_unstale":"2024-09-18T07:39:59.179627+0000","last_undegraded":"2024-09-18T07:39:59.179627+0000","last_fullsized":"2024-09-18T07:39:59.179627+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-18T07:36:06.083531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.083531+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":216,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177486+0000","last_change":"2024-09-18T07:39:07.370140+0000","last_active":"2024-09-18T07:39:59.177486+0000","last_peered":"2024-09-18T07:39:59.177486+0000","last_clean":"2024-09-18T07:39:59.177486+0000","last_became_active":"2024-09-18T07:37:43.050971+0000","last_became_peered":"2024-09-18T07:37:43.050971+0000","last_unstale":"2024-09-18T07:39:59.177486+0000","last_undegraded":"2024-09-18T07:39:59.177486+0000","last_fullsized":"2024-09-18T07:39:59.177486+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:07.370081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:07.370081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:28.043475+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":521,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.176569+0000","last_change":"2024-09-18T07:37:49.895264+0000","last_active":"2024-09-18T07:39:59.176569+0000","last_peered":"2024-09-18T07:39:59.176569+0000","last_clean":"2024-09-18T07:39:59.176569+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:39:59.176569+0000","last_undegraded":"2024-09-18T07:39:59.176569+0000","last_fullsized":"2024-09-18T07:39:59.176569+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:37:49.894707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:37:49.894707+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:14.409320+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":518,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.176553+0000","last_change":"2024-09-18T07:39:35.457812+0000","last_active":"2024-09-18T07:39:59.176553+0000","last_peered":"2024-09-18T07:39:59.176553+0000","last_clean":"2024-09-18T07:39:59.176553+0000","last_became_active":"2024-09-18T07:35:01.403680+0000","last_became_peered":"2024-09-18T07:35:01.403680+0000","last_unstale":"2024-09-18T07:39:59.176553+0000","last_undegraded":"2024-09-18T07:39:59.176553+0000","last_fullsized":"2024-09-18T07:39:59.176553+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.457734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.457734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:55.517500+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":294,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178145+0000","last_change":"2024-09-18T07:37:43.118570+0000","last_active":"2024-09-18T07:39:59.178145+0000","last_peered":"2024-09-18T07:39:59.178145+0000","last_clean":"2024-09-18T07:39:59.178145+0000","last_became_active":"2024-09-18T07:37:43.060019+0000","last_became_peered":"2024-09-18T07:37:43.060019+0000","last_unstale":"2024-09-18T07:39:59.178145+0000","last_undegraded":"2024-09-18T07:39:59.178145+0000","last_fullsized":"2024-09-18T07:39:59.178145+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:57.655797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.655797+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:51.521311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"301'3597","reported_seq":3632,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:02.457128+0000","last_change":"2024-09-18T07:39:29.701833+0000","last_active":"2024-09-18T07:40:02.457128+0000","last_peered":"2024-09-18T07:40:02.457128+0000","last_clean":"2024-09-18T07:40:02.457128+0000","last_became_active":"2024-09-18T07:39:29.701554+0000","last_became_peered":"2024-09-18T07:39:29.701554+0000","last_unstale":"2024-09-18T07:40:02.457128+0000","last_undegraded":"2024-09-18T07:40:02.457128+0000","last_fullsized":"2024-09-18T07:40:02.457128+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3597,"log_dups_size":0,"ondisk_log_size":3597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:40.719037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235732992,"num_objects":3597,"num_object_clones":0,"num_object_copies":10791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":230208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":240,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.193526+0000","last_change":"2024-09-18T07:39:05.333676+0000","last_active":"2024-09-18T07:39:59.193526+0000","last_peered":"2024-09-18T07:39:59.193526+0000","last_clean":"2024-09-18T07:39:59.193526+0000","last_became_active":"2024-09-18T07:39:04.938080+0000","last_became_peered":"2024-09-18T07:39:04.938080+0000","last_unstale":"2024-09-18T07:39:59.193526+0000","last_undegraded":"2024-09-18T07:39:59.193526+0000","last_fullsized":"2024-09-18T07:39:59.193526+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.333608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.333608+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:28.401993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":527,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.171146+0000","last_change":"2024-09-18T07:33:27.040227+0000","last_active":"2024-09-18T07:39:59.171146+0000","last_peered":"2024-09-18T07:39:59.171146+0000","last_clean":"2024-09-18T07:39:59.171146+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:39:59.171146+0000","last_undegraded":"2024-09-18T07:39:59.171146+0000","last_fullsized":"2024-09-18T07:39:59.171146+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:17.634433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":498,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.171128+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:39:59.171128+0000","last_peered":"2024-09-18T07:39:59.171128+0000","last_clean":"2024-09-18T07:39:59.171128+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:39:59.171128+0000","last_undegraded":"2024-09-18T07:39:59.171128+0000","last_fullsized":"2024-09-18T07:39:59.171128+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:16.624959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":572,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.845327+0000","last_change":"2024-09-18T07:35:01.421692+0000","last_active":"2024-09-18T07:39:59.845327+0000","last_peered":"2024-09-18T07:39:59.845327+0000","last_clean":"2024-09-18T07:39:59.845327+0000","last_became_active":"2024-09-18T07:35:01.400524+0000","last_became_peered":"2024-09-18T07:35:01.400524+0000","last_unstale":"2024-09-18T07:39:59.845327+0000","last_undegraded":"2024-09-18T07:39:59.845327+0000","last_fullsized":"2024-09-18T07:39:59.845327+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:46.474589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":306,"num_read_kb":301,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":502,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179650+0000","last_change":"2024-09-18T07:36:28.551293+0000","last_active":"2024-09-18T07:39:59.179650+0000","last_peered":"2024-09-18T07:39:59.179650+0000","last_clean":"2024-09-18T07:39:59.179650+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:39:59.179650+0000","last_undegraded":"2024-09-18T07:39:59.179650+0000","last_fullsized":"2024-09-18T07:39:59.179650+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:28.551238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:28.551238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":202,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178914+0000","last_change":"2024-09-18T07:36:39.601424+0000","last_active":"2024-09-18T07:39:59.178914+0000","last_peered":"2024-09-18T07:39:59.178914+0000","last_clean":"2024-09-18T07:39:59.178914+0000","last_became_active":"2024-09-18T07:36:13.117463+0000","last_became_peered":"2024-09-18T07:36:13.117463+0000","last_unstale":"2024-09-18T07:39:59.178914+0000","last_undegraded":"2024-09-18T07:39:59.178914+0000","last_fullsized":"2024-09-18T07:39:59.178914+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:39.601356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:39.601356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"301'3517","reported_seq":3552,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:02.075159+0000","last_change":"2024-09-18T07:39:29.700116+0000","last_active":"2024-09-18T07:40:02.075159+0000","last_peered":"2024-09-18T07:40:02.075159+0000","last_clean":"2024-09-18T07:40:02.075159+0000","last_became_active":"2024-09-18T07:39:29.699733+0000","last_became_peered":"2024-09-18T07:39:29.699733+0000","last_unstale":"2024-09-18T07:40:02.075159+0000","last_undegraded":"2024-09-18T07:40:02.075159+0000","last_fullsized":"2024-09-18T07:40:02.075159+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:37.454373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230490112,"num_objects":3517,"num_object_clones":0,"num_object_copies":10551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3517,"num_write_kb":225088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'20","reported_seq":181,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.193984+0000","last_change":"2024-09-18T07:37:43.157718+0000","last_active":"2024-09-18T07:39:59.193984+0000","last_peered":"2024-09-18T07:39:59.193984+0000","last_clean":"2024-09-18T07:39:59.193984+0000","last_became_active":"2024-09-18T07:37:43.087496+0000","last_became_peered":"2024-09-18T07:37:43.087496+0000","last_unstale":"2024-09-18T07:39:59.193984+0000","last_undegraded":"2024-09-18T07:39:59.193984+0000","last_fullsized":"2024-09-18T07:39:59.193984+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'17","last_scrub_stamp":"2024-09-18T07:33:33.849285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.849285+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:15.202624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":517,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177215+0000","last_change":"2024-09-18T07:37:43.065584+0000","last_active":"2024-09-18T07:39:59.177215+0000","last_peered":"2024-09-18T07:39:59.177215+0000","last_clean":"2024-09-18T07:39:59.177215+0000","last_became_active":"2024-09-18T07:37:43.065279+0000","last_became_peered":"2024-09-18T07:37:43.065279+0000","last_unstale":"2024-09-18T07:39:59.177215+0000","last_undegraded":"2024-09-18T07:39:59.177215+0000","last_fullsized":"2024-09-18T07:39:59.177215+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:59.616988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:59.616988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:45.852796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":334,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177189+0000","last_change":"2024-09-18T07:39:03.518624+0000","last_active":"2024-09-18T07:39:59.177189+0000","last_peered":"2024-09-18T07:39:59.177189+0000","last_clean":"2024-09-18T07:39:59.177189+0000","last_became_active":"2024-09-18T07:39:03.350419+0000","last_became_peered":"2024-09-18T07:39:03.350419+0000","last_unstale":"2024-09-18T07:39:59.177189+0000","last_undegraded":"2024-09-18T07:39:59.177189+0000","last_fullsized":"2024-09-18T07:39:59.177189+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:03.518551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:03.518551+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:14.832014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":276,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177841+0000","last_change":"2024-09-18T07:38:52.486107+0000","last_active":"2024-09-18T07:39:59.177841+0000","last_peered":"2024-09-18T07:39:59.177841+0000","last_clean":"2024-09-18T07:39:59.177841+0000","last_became_active":"2024-09-18T07:35:01.412217+0000","last_became_peered":"2024-09-18T07:35:01.412217+0000","last_unstale":"2024-09-18T07:39:59.177841+0000","last_undegraded":"2024-09-18T07:39:59.177841+0000","last_fullsized":"2024-09-18T07:39:59.177841+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:52.486047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:52.486047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.218642+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":490,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179527+0000","last_change":"2024-09-18T07:33:22.358792+0000","last_active":"2024-09-18T07:39:59.179527+0000","last_peered":"2024-09-18T07:39:59.179527+0000","last_clean":"2024-09-18T07:39:59.179527+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:39:59.179527+0000","last_undegraded":"2024-09-18T07:39:59.179527+0000","last_fullsized":"2024-09-18T07:39:59.179527+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.358746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.358746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:18.619255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"301'3450","reported_seq":3484,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:01.514780+0000","last_change":"2024-09-18T07:39:29.703721+0000","last_active":"2024-09-18T07:40:01.514780+0000","last_peered":"2024-09-18T07:40:01.514780+0000","last_clean":"2024-09-18T07:40:01.514780+0000","last_became_active":"2024-09-18T07:39:29.703451+0000","last_became_peered":"2024-09-18T07:39:29.703451+0000","last_unstale":"2024-09-18T07:40:01.514780+0000","last_undegraded":"2024-09-18T07:40:01.514780+0000","last_fullsized":"2024-09-18T07:40:01.514780+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3450,"log_dups_size":0,"ondisk_log_size":3450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:52.017204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226099200,"num_objects":3450,"num_object_clones":0,"num_object_copies":10350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3450,"num_write_kb":220800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":611,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.175377+0000","last_change":"2024-09-18T07:39:07.038527+0000","last_active":"2024-09-18T07:39:59.175377+0000","last_peered":"2024-09-18T07:39:59.175377+0000","last_clean":"2024-09-18T07:39:59.175377+0000","last_became_active":"2024-09-18T07:37:43.063255+0000","last_became_peered":"2024-09-18T07:37:43.063255+0000","last_unstale":"2024-09-18T07:39:59.175377+0000","last_undegraded":"2024-09-18T07:39:59.175377+0000","last_fullsized":"2024-09-18T07:39:59.175377+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:39:07.038450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:07.038450+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:24.342260+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":534,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174185+0000","last_change":"2024-09-18T07:39:29.078656+0000","last_active":"2024-09-18T07:39:59.174185+0000","last_peered":"2024-09-18T07:39:59.174185+0000","last_clean":"2024-09-18T07:39:59.174185+0000","last_became_active":"2024-09-18T07:35:01.401063+0000","last_became_peered":"2024-09-18T07:35:01.401063+0000","last_unstale":"2024-09-18T07:39:59.174185+0000","last_undegraded":"2024-09-18T07:39:59.174185+0000","last_fullsized":"2024-09-18T07:39:59.174185+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.078600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.078600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:32.657735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":524,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179442+0000","last_change":"2024-09-18T07:39:35.406107+0000","last_active":"2024-09-18T07:39:59.179442+0000","last_peered":"2024-09-18T07:39:59.179442+0000","last_clean":"2024-09-18T07:39:59.179442+0000","last_became_active":"2024-09-18T07:37:43.075535+0000","last_became_peered":"2024-09-18T07:37:43.075535+0000","last_unstale":"2024-09-18T07:39:59.179442+0000","last_undegraded":"2024-09-18T07:39:59.179442+0000","last_fullsized":"2024-09-18T07:39:59.179442+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.406041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.406041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:44.739547+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":512,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179539+0000","last_change":"2024-09-18T07:37:43.066497+0000","last_active":"2024-09-18T07:39:59.179539+0000","last_peered":"2024-09-18T07:39:59.179539+0000","last_clean":"2024-09-18T07:39:59.179539+0000","last_became_active":"2024-09-18T07:37:43.066153+0000","last_became_peered":"2024-09-18T07:37:43.066153+0000","last_unstale":"2024-09-18T07:39:59.179539+0000","last_undegraded":"2024-09-18T07:39:59.179539+0000","last_fullsized":"2024-09-18T07:39:59.179539+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:38.955353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:38.955353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:08.997218+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"301'3499","reported_seq":3534,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:01.630570+0000","last_change":"2024-09-18T07:39:29.700091+0000","last_active":"2024-09-18T07:40:01.630570+0000","last_peered":"2024-09-18T07:40:01.630570+0000","last_clean":"2024-09-18T07:40:01.630570+0000","last_became_active":"2024-09-18T07:39:29.699800+0000","last_became_peered":"2024-09-18T07:39:29.699800+0000","last_unstale":"2024-09-18T07:40:01.630570+0000","last_undegraded":"2024-09-18T07:40:01.630570+0000","last_fullsized":"2024-09-18T07:40:01.630570+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3499,"log_dups_size":0,"ondisk_log_size":3499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:42.451559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229310464,"num_objects":3499,"num_object_clones":0,"num_object_copies":10497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3499,"num_write_kb":223936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":82,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178185+0000","last_change":"2024-09-18T07:39:38.447787+0000","last_active":"2024-09-18T07:39:59.178185+0000","last_peered":"2024-09-18T07:39:59.178185+0000","last_clean":"2024-09-18T07:39:59.178185+0000","last_became_active":"2024-09-18T07:39:04.950292+0000","last_became_peered":"2024-09-18T07:39:04.950292+0000","last_unstale":"2024-09-18T07:39:59.178185+0000","last_undegraded":"2024-09-18T07:39:59.178185+0000","last_fullsized":"2024-09-18T07:39:59.178185+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:38.447735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:38.447735+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:53.329499+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":544,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174870+0000","last_change":"2024-09-18T07:38:55.096608+0000","last_active":"2024-09-18T07:39:59.174870+0000","last_peered":"2024-09-18T07:39:59.174870+0000","last_clean":"2024-09-18T07:39:59.174870+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:39:59.174870+0000","last_undegraded":"2024-09-18T07:39:59.174870+0000","last_fullsized":"2024-09-18T07:39:59.174870+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:38:55.096550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:38:55.096550+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":143,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.193094+0000","last_change":"2024-09-18T07:39:35.015984+0000","last_active":"2024-09-18T07:39:59.193094+0000","last_peered":"2024-09-18T07:39:59.193094+0000","last_clean":"2024-09-18T07:39:59.193094+0000","last_became_active":"2024-09-18T07:37:43.061808+0000","last_became_peered":"2024-09-18T07:37:43.061808+0000","last_unstale":"2024-09-18T07:39:59.193094+0000","last_undegraded":"2024-09-18T07:39:59.193094+0000","last_fullsized":"2024-09-18T07:39:59.193094+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.015923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.015923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:42.634359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":194,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174301+0000","last_change":"2024-09-18T07:36:13.116901+0000","last_active":"2024-09-18T07:39:59.174301+0000","last_peered":"2024-09-18T07:39:59.174301+0000","last_clean":"2024-09-18T07:39:59.174301+0000","last_became_active":"2024-09-18T07:36:13.116527+0000","last_became_peered":"2024-09-18T07:36:13.116527+0000","last_unstale":"2024-09-18T07:39:59.174301+0000","last_undegraded":"2024-09-18T07:39:59.174301+0000","last_fullsized":"2024-09-18T07:39:59.174301+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:05.239160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.239160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:50.048595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":507,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177944+0000","last_change":"2024-09-18T07:36:38.632189+0000","last_active":"2024-09-18T07:39:59.177944+0000","last_peered":"2024-09-18T07:39:59.177944+0000","last_clean":"2024-09-18T07:39:59.177944+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:39:59.177944+0000","last_undegraded":"2024-09-18T07:39:59.177944+0000","last_fullsized":"2024-09-18T07:39:59.177944+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:38.632040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:38.632040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:09.340014+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"301'3582","reported_seq":3616,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:02.091454+0000","last_change":"2024-09-18T07:39:29.701529+0000","last_active":"2024-09-18T07:40:02.091454+0000","last_peered":"2024-09-18T07:40:02.091454+0000","last_clean":"2024-09-18T07:40:02.091454+0000","last_became_active":"2024-09-18T07:39:29.701147+0000","last_became_peered":"2024-09-18T07:39:29.701147+0000","last_unstale":"2024-09-18T07:40:02.091454+0000","last_undegraded":"2024-09-18T07:40:02.091454+0000","last_fullsized":"2024-09-18T07:40:02.091454+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3582,"log_dups_size":0,"ondisk_log_size":3582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:51.353620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234749952,"num_objects":3582,"num_object_clones":0,"num_object_copies":10746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3582,"num_write_kb":229248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":340,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177509+0000","last_change":"2024-09-18T07:39:05.498235+0000","last_active":"2024-09-18T07:39:59.177509+0000","last_peered":"2024-09-18T07:39:59.177509+0000","last_clean":"2024-09-18T07:39:59.177509+0000","last_became_active":"2024-09-18T07:39:04.944753+0000","last_became_peered":"2024-09-18T07:39:04.944753+0000","last_unstale":"2024-09-18T07:39:59.177509+0000","last_undegraded":"2024-09-18T07:39:59.177509+0000","last_fullsized":"2024-09-18T07:39:59.177509+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.498159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.498159+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:05.777887+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":553,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174536+0000","last_change":"2024-09-18T07:37:43.106862+0000","last_active":"2024-09-18T07:39:59.174536+0000","last_peered":"2024-09-18T07:39:59.174536+0000","last_clean":"2024-09-18T07:39:59.174536+0000","last_became_active":"2024-09-18T07:37:43.062724+0000","last_became_peered":"2024-09-18T07:37:43.062724+0000","last_unstale":"2024-09-18T07:39:59.174536+0000","last_undegraded":"2024-09-18T07:39:59.174536+0000","last_fullsized":"2024-09-18T07:39:59.174536+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:32:05.000310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.000310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:45.943194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":511,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179751+0000","last_change":"2024-09-18T07:39:38.871492+0000","last_active":"2024-09-18T07:39:59.179751+0000","last_peered":"2024-09-18T07:39:59.179751+0000","last_clean":"2024-09-18T07:39:59.179751+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:39:59.179751+0000","last_undegraded":"2024-09-18T07:39:59.179751+0000","last_fullsized":"2024-09-18T07:39:59.179751+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:38.871426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:38.871426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:04.468858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"173'11","reported_seq":655,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:01.464508+0000","last_change":"2024-09-18T07:38:55.227421+0000","last_active":"2024-09-18T07:40:01.464508+0000","last_peered":"2024-09-18T07:40:01.464508+0000","last_clean":"2024-09-18T07:40:01.464508+0000","last_became_active":"2024-09-18T07:35:01.403765+0000","last_became_peered":"2024-09-18T07:35:01.403765+0000","last_unstale":"2024-09-18T07:40:01.464508+0000","last_undegraded":"2024-09-18T07:40:01.464508+0000","last_fullsized":"2024-09-18T07:40:01.464508+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'11","last_scrub_stamp":"2024-09-18T07:38:55.227366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:55.227366+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:17.922368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":500,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.175022+0000","last_change":"2024-09-18T07:36:33.251164+0000","last_active":"2024-09-18T07:39:59.175022+0000","last_peered":"2024-09-18T07:39:59.175022+0000","last_clean":"2024-09-18T07:39:59.175022+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:39:59.175022+0000","last_undegraded":"2024-09-18T07:39:59.175022+0000","last_fullsized":"2024-09-18T07:39:59.175022+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:33.251093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:33.251093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:01.242543+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"301'3517","reported_seq":3552,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:01.781150+0000","last_change":"2024-09-18T07:39:29.704580+0000","last_active":"2024-09-18T07:40:01.781150+0000","last_peered":"2024-09-18T07:40:01.781150+0000","last_clean":"2024-09-18T07:40:01.781150+0000","last_became_active":"2024-09-18T07:39:29.704318+0000","last_became_peered":"2024-09-18T07:39:29.704318+0000","last_unstale":"2024-09-18T07:40:01.781150+0000","last_undegraded":"2024-09-18T07:40:01.781150+0000","last_fullsized":"2024-09-18T07:40:01.781150+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:40.830318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230490112,"num_objects":3517,"num_object_clones":0,"num_object_copies":10551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3517,"num_write_kb":225088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":529,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.184076+0000","last_change":"2024-09-18T07:39:02.716084+0000","last_active":"2024-09-18T07:39:59.184076+0000","last_peered":"2024-09-18T07:39:59.184076+0000","last_clean":"2024-09-18T07:39:59.184076+0000","last_became_active":"2024-09-18T07:37:27.994144+0000","last_became_peered":"2024-09-18T07:37:27.994144+0000","last_unstale":"2024-09-18T07:39:59.184076+0000","last_undegraded":"2024-09-18T07:39:59.184076+0000","last_fullsized":"2024-09-18T07:39:59.184076+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:39:02.716009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:02.716009+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:04.590923+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":524,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179006+0000","last_change":"2024-09-18T07:37:43.076416+0000","last_active":"2024-09-18T07:39:59.179006+0000","last_peered":"2024-09-18T07:39:59.179006+0000","last_clean":"2024-09-18T07:39:59.179006+0000","last_became_active":"2024-09-18T07:37:43.075988+0000","last_became_peered":"2024-09-18T07:37:43.075988+0000","last_unstale":"2024-09-18T07:39:59.179006+0000","last_undegraded":"2024-09-18T07:39:59.179006+0000","last_fullsized":"2024-09-18T07:39:59.179006+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:21.222526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.222526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:44.443625+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":504,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.176169+0000","last_change":"2024-09-18T07:39:01.216439+0000","last_active":"2024-09-18T07:39:59.176169+0000","last_peered":"2024-09-18T07:39:59.176169+0000","last_clean":"2024-09-18T07:39:59.176169+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:39:59.176169+0000","last_undegraded":"2024-09-18T07:39:59.176169+0000","last_fullsized":"2024-09-18T07:39:59.176169+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:01.216305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:01.216305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:23.547767+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":510,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.171030+0000","last_change":"2024-09-18T07:36:07.526609+0000","last_active":"2024-09-18T07:39:59.171030+0000","last_peered":"2024-09-18T07:39:59.171030+0000","last_clean":"2024-09-18T07:39:59.171030+0000","last_became_active":"2024-09-18T07:35:01.410890+0000","last_became_peered":"2024-09-18T07:35:01.410890+0000","last_unstale":"2024-09-18T07:39:59.171030+0000","last_undegraded":"2024-09-18T07:39:59.171030+0000","last_fullsized":"2024-09-18T07:39:59.171030+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:07.526536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:07.526536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:41.630367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"301'3423","reported_seq":3458,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:01.803298+0000","last_change":"2024-09-18T07:39:29.701848+0000","last_active":"2024-09-18T07:40:01.803298+0000","last_peered":"2024-09-18T07:40:01.803298+0000","last_clean":"2024-09-18T07:40:01.803298+0000","last_became_active":"2024-09-18T07:39:29.701593+0000","last_became_peered":"2024-09-18T07:39:29.701593+0000","last_unstale":"2024-09-18T07:40:01.803298+0000","last_undegraded":"2024-09-18T07:40:01.803298+0000","last_fullsized":"2024-09-18T07:40:01.803298+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3423,"log_dups_size":0,"ondisk_log_size":3423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:45.431316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224329728,"num_objects":3423,"num_object_clones":0,"num_object_copies":10269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3423,"num_write_kb":219072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":163,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.193220+0000","last_change":"2024-09-18T07:39:25.932376+0000","last_active":"2024-09-18T07:39:59.193220+0000","last_peered":"2024-09-18T07:39:59.193220+0000","last_clean":"2024-09-18T07:39:59.193220+0000","last_became_active":"2024-09-18T07:37:43.063319+0000","last_became_peered":"2024-09-18T07:37:43.063319+0000","last_unstale":"2024-09-18T07:39:59.193220+0000","last_undegraded":"2024-09-18T07:39:59.193220+0000","last_fullsized":"2024-09-18T07:39:59.193220+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:39:25.932315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.932315+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:32.693891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":298,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178512+0000","last_change":"2024-09-18T07:39:25.301868+0000","last_active":"2024-09-18T07:39:59.178512+0000","last_peered":"2024-09-18T07:39:59.178512+0000","last_clean":"2024-09-18T07:39:59.178512+0000","last_became_active":"2024-09-18T07:37:43.072894+0000","last_became_peered":"2024-09-18T07:37:43.072894+0000","last_unstale":"2024-09-18T07:39:59.178512+0000","last_undegraded":"2024-09-18T07:39:59.178512+0000","last_fullsized":"2024-09-18T07:39:59.178512+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:25.301825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.301825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.461283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":519,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.176882+0000","last_change":"2024-09-18T07:39:38.400864+0000","last_active":"2024-09-18T07:39:59.176882+0000","last_peered":"2024-09-18T07:39:59.176882+0000","last_clean":"2024-09-18T07:39:59.176882+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:39:59.176882+0000","last_undegraded":"2024-09-18T07:39:59.176882+0000","last_fullsized":"2024-09-18T07:39:59.176882+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:38.400785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:38.400785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:55.378212+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":268,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177193+0000","last_change":"2024-09-18T07:35:01.408078+0000","last_active":"2024-09-18T07:39:59.177193+0000","last_peered":"2024-09-18T07:39:59.177193+0000","last_clean":"2024-09-18T07:39:59.177193+0000","last_became_active":"2024-09-18T07:35:01.406980+0000","last_became_peered":"2024-09-18T07:35:01.406980+0000","last_unstale":"2024-09-18T07:39:59.177193+0000","last_undegraded":"2024-09-18T07:39:59.177193+0000","last_fullsized":"2024-09-18T07:39:59.177193+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:40.390575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"301'3558","reported_seq":3593,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:01.555019+0000","last_change":"2024-09-18T07:39:29.700522+0000","last_active":"2024-09-18T07:40:01.555019+0000","last_peered":"2024-09-18T07:40:01.555019+0000","last_clean":"2024-09-18T07:40:01.555019+0000","last_became_active":"2024-09-18T07:39:29.700244+0000","last_became_peered":"2024-09-18T07:39:29.700244+0000","last_unstale":"2024-09-18T07:40:01.555019+0000","last_undegraded":"2024-09-18T07:40:01.555019+0000","last_fullsized":"2024-09-18T07:40:01.555019+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:39.850144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233177088,"num_objects":3558,"num_object_clones":0,"num_object_copies":10674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3558,"num_write_kb":227712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":539,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174082+0000","last_change":"2024-09-18T07:38:59.101584+0000","last_active":"2024-09-18T07:39:59.174082+0000","last_peered":"2024-09-18T07:39:59.174082+0000","last_clean":"2024-09-18T07:39:59.174082+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:39:59.174082+0000","last_undegraded":"2024-09-18T07:39:59.174082+0000","last_fullsized":"2024-09-18T07:39:59.174082+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:38:59.101512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:38:59.101512+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:07.212990+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":268,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178811+0000","last_change":"2024-09-18T07:35:01.400050+0000","last_active":"2024-09-18T07:39:59.178811+0000","last_peered":"2024-09-18T07:39:59.178811+0000","last_clean":"2024-09-18T07:39:59.178811+0000","last_became_active":"2024-09-18T07:35:01.398499+0000","last_became_peered":"2024-09-18T07:35:01.398499+0000","last_unstale":"2024-09-18T07:39:59.178811+0000","last_undegraded":"2024-09-18T07:39:59.178811+0000","last_fullsized":"2024-09-18T07:39:59.178811+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:43.441418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":520,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177402+0000","last_change":"2024-09-18T07:36:32.564149+0000","last_active":"2024-09-18T07:39:59.177402+0000","last_peered":"2024-09-18T07:39:59.177402+0000","last_clean":"2024-09-18T07:39:59.177402+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:39:59.177402+0000","last_undegraded":"2024-09-18T07:39:59.177402+0000","last_fullsized":"2024-09-18T07:39:59.177402+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:32.564101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:32.564101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:08.345410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":504,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179502+0000","last_change":"2024-09-18T07:37:43.016535+0000","last_active":"2024-09-18T07:39:59.179502+0000","last_peered":"2024-09-18T07:39:59.179502+0000","last_clean":"2024-09-18T07:39:59.179502+0000","last_became_active":"2024-09-18T07:35:01.418735+0000","last_became_peered":"2024-09-18T07:35:01.418735+0000","last_unstale":"2024-09-18T07:39:59.179502+0000","last_undegraded":"2024-09-18T07:39:59.179502+0000","last_fullsized":"2024-09-18T07:39:59.179502+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:43.016489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:37:43.016489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"301'3481","reported_seq":3516,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:40:02.460634+0000","last_change":"2024-09-18T07:39:29.701895+0000","last_active":"2024-09-18T07:40:02.460634+0000","last_peered":"2024-09-18T07:40:02.460634+0000","last_clean":"2024-09-18T07:40:02.460634+0000","last_became_active":"2024-09-18T07:39:29.701617+0000","last_became_peered":"2024-09-18T07:39:29.701617+0000","last_unstale":"2024-09-18T07:40:02.460634+0000","last_undegraded":"2024-09-18T07:40:02.460634+0000","last_fullsized":"2024-09-18T07:40:02.460634+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3481,"log_dups_size":0,"ondisk_log_size":3481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:42.973327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228130816,"num_objects":3481,"num_object_clones":0,"num_object_copies":10443,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3481,"num_write_kb":222784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":631,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178352+0000","last_change":"2024-09-18T07:39:27.484003+0000","last_active":"2024-09-18T07:39:59.178352+0000","last_peered":"2024-09-18T07:39:59.178352+0000","last_clean":"2024-09-18T07:39:59.178352+0000","last_became_active":"2024-09-18T07:37:43.075552+0000","last_became_peered":"2024-09-18T07:37:43.075552+0000","last_unstale":"2024-09-18T07:39:59.178352+0000","last_undegraded":"2024-09-18T07:39:59.178352+0000","last_fullsized":"2024-09-18T07:39:59.178352+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:39:27.483930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:27.483930+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.149026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":290,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178437+0000","last_change":"2024-09-18T07:36:09.261574+0000","last_active":"2024-09-18T07:39:59.178437+0000","last_peered":"2024-09-18T07:39:59.178437+0000","last_clean":"2024-09-18T07:39:59.178437+0000","last_became_active":"2024-09-18T07:35:01.406695+0000","last_became_peered":"2024-09-18T07:35:01.406695+0000","last_unstale":"2024-09-18T07:39:59.178437+0000","last_undegraded":"2024-09-18T07:39:59.178437+0000","last_fullsized":"2024-09-18T07:39:59.178437+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:36:09.261516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.261516+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:22.292832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"240'12","reported_seq":371,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177982+0000","last_change":"2024-09-18T07:38:55.490250+0000","last_active":"2024-09-18T07:39:59.177982+0000","last_peered":"2024-09-18T07:39:59.177982+0000","last_clean":"2024-09-18T07:39:59.177982+0000","last_became_active":"2024-09-18T07:37:43.056714+0000","last_became_peered":"2024-09-18T07:37:43.056714+0000","last_unstale":"2024-09-18T07:39:59.177982+0000","last_undegraded":"2024-09-18T07:39:59.177982+0000","last_fullsized":"2024-09-18T07:39:59.177982+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:38:55.490178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:55.490178+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:07.849179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":496,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178578+0000","last_change":"2024-09-18T07:37:43.078768+0000","last_active":"2024-09-18T07:39:59.178578+0000","last_peered":"2024-09-18T07:39:59.178578+0000","last_clean":"2024-09-18T07:39:59.178578+0000","last_became_active":"2024-09-18T07:37:43.078121+0000","last_became_peered":"2024-09-18T07:37:43.078121+0000","last_unstale":"2024-09-18T07:39:59.178578+0000","last_undegraded":"2024-09-18T07:39:59.178578+0000","last_fullsized":"2024-09-18T07:39:59.178578+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:12.586962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":142,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.193631+0000","last_change":"2024-09-18T07:39:37.072844+0000","last_active":"2024-09-18T07:39:59.193631+0000","last_peered":"2024-09-18T07:39:59.193631+0000","last_clean":"2024-09-18T07:39:59.193631+0000","last_became_active":"2024-09-18T07:37:43.087994+0000","last_became_peered":"2024-09-18T07:37:43.087994+0000","last_unstale":"2024-09-18T07:39:59.193631+0000","last_undegraded":"2024-09-18T07:39:59.193631+0000","last_fullsized":"2024-09-18T07:39:59.193631+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.072782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.072782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:52.412066+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":537,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.180453+0000","last_change":"2024-09-18T07:36:09.061983+0000","last_active":"2024-09-18T07:39:59.180453+0000","last_peered":"2024-09-18T07:39:59.180453+0000","last_clean":"2024-09-18T07:39:59.180453+0000","last_became_active":"2024-09-18T07:35:01.407787+0000","last_became_peered":"2024-09-18T07:35:01.407787+0000","last_unstale":"2024-09-18T07:39:59.180453+0000","last_undegraded":"2024-09-18T07:39:59.180453+0000","last_fullsized":"2024-09-18T07:39:59.180453+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-18T07:36:09.061903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.061903+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:16.050531+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":501,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179052+0000","last_change":"2024-09-18T07:36:34.608531+0000","last_active":"2024-09-18T07:39:59.179052+0000","last_peered":"2024-09-18T07:39:59.179052+0000","last_clean":"2024-09-18T07:39:59.179052+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:39:59.179052+0000","last_undegraded":"2024-09-18T07:39:59.179052+0000","last_fullsized":"2024-09-18T07:39:59.179052+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:34.608486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:34.608486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":282,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178382+0000","last_change":"2024-09-18T07:37:43.079723+0000","last_active":"2024-09-18T07:39:59.178382+0000","last_peered":"2024-09-18T07:39:59.178382+0000","last_clean":"2024-09-18T07:39:59.178382+0000","last_became_active":"2024-09-18T07:37:43.079313+0000","last_became_peered":"2024-09-18T07:37:43.079313+0000","last_unstale":"2024-09-18T07:39:59.178382+0000","last_undegraded":"2024-09-18T07:39:59.178382+0000","last_fullsized":"2024-09-18T07:39:59.178382+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:58.688631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:58.688631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:50.554490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":537,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.171225+0000","last_change":"2024-09-18T07:35:01.448510+0000","last_active":"2024-09-18T07:39:59.171225+0000","last_peered":"2024-09-18T07:39:59.171225+0000","last_clean":"2024-09-18T07:39:59.171225+0000","last_became_active":"2024-09-18T07:35:01.414458+0000","last_became_peered":"2024-09-18T07:35:01.414458+0000","last_unstale":"2024-09-18T07:39:59.171225+0000","last_undegraded":"2024-09-18T07:39:59.171225+0000","last_fullsized":"2024-09-18T07:39:59.171225+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:19.600284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":290,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177867+0000","last_change":"2024-09-18T07:37:43.057776+0000","last_active":"2024-09-18T07:39:59.177867+0000","last_peered":"2024-09-18T07:39:59.177867+0000","last_clean":"2024-09-18T07:39:59.177867+0000","last_became_active":"2024-09-18T07:37:43.056675+0000","last_became_peered":"2024-09-18T07:37:43.056675+0000","last_unstale":"2024-09-18T07:39:59.177867+0000","last_undegraded":"2024-09-18T07:39:59.177867+0000","last_fullsized":"2024-09-18T07:39:59.177867+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:05.201723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.201723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:15.371131+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":524,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.180158+0000","last_change":"2024-09-18T07:39:30.966887+0000","last_active":"2024-09-18T07:39:59.180158+0000","last_peered":"2024-09-18T07:39:59.180158+0000","last_clean":"2024-09-18T07:39:59.180158+0000","last_became_active":"2024-09-18T07:35:01.420133+0000","last_became_peered":"2024-09-18T07:35:01.420133+0000","last_unstale":"2024-09-18T07:39:59.180158+0000","last_undegraded":"2024-09-18T07:39:59.180158+0000","last_fullsized":"2024-09-18T07:39:59.180158+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:30.966840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:30.966840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:33.098878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":505,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177000+0000","last_change":"2024-09-18T07:37:48.861965+0000","last_active":"2024-09-18T07:39:59.177000+0000","last_peered":"2024-09-18T07:39:59.177000+0000","last_clean":"2024-09-18T07:39:59.177000+0000","last_became_active":"2024-09-18T07:35:01.396305+0000","last_became_peered":"2024-09-18T07:35:01.396305+0000","last_unstale":"2024-09-18T07:39:59.177000+0000","last_undegraded":"2024-09-18T07:39:59.177000+0000","last_fullsized":"2024-09-18T07:39:59.177000+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:48.861784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:37:48.861784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:23.433931+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":555,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179250+0000","last_change":"2024-09-18T07:39:25.515172+0000","last_active":"2024-09-18T07:39:59.179250+0000","last_peered":"2024-09-18T07:39:59.179250+0000","last_clean":"2024-09-18T07:39:59.179250+0000","last_became_active":"2024-09-18T07:35:01.416744+0000","last_became_peered":"2024-09-18T07:35:01.416744+0000","last_unstale":"2024-09-18T07:39:59.179250+0000","last_undegraded":"2024-09-18T07:39:59.179250+0000","last_fullsized":"2024-09-18T07:39:59.179250+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:39:25.515085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.515085+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:40.279265+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":521,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.170589+0000","last_change":"2024-09-18T07:39:32.766405+0000","last_active":"2024-09-18T07:39:59.170589+0000","last_peered":"2024-09-18T07:39:59.170589+0000","last_clean":"2024-09-18T07:39:59.170589+0000","last_became_active":"2024-09-18T07:35:01.414024+0000","last_became_peered":"2024-09-18T07:35:01.414024+0000","last_unstale":"2024-09-18T07:39:59.170589+0000","last_undegraded":"2024-09-18T07:39:59.170589+0000","last_fullsized":"2024-09-18T07:39:59.170589+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:32.766366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:32.766366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:36.625283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"173'10","reported_seq":642,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179744+0000","last_change":"2024-09-18T07:39:32.947933+0000","last_active":"2024-09-18T07:39:59.179744+0000","last_peered":"2024-09-18T07:39:59.179744+0000","last_clean":"2024-09-18T07:39:59.179744+0000","last_became_active":"2024-09-18T07:35:01.407568+0000","last_became_peered":"2024-09-18T07:35:01.407568+0000","last_unstale":"2024-09-18T07:39:59.179744+0000","last_undegraded":"2024-09-18T07:39:59.179744+0000","last_fullsized":"2024-09-18T07:39:59.179744+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:39:32.947887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:32.947887+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:58.809615+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":491,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.175346+0000","last_change":"2024-09-18T07:36:11.910543+0000","last_active":"2024-09-18T07:39:59.175346+0000","last_peered":"2024-09-18T07:39:59.175346+0000","last_clean":"2024-09-18T07:39:59.175346+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:39:59.175346+0000","last_undegraded":"2024-09-18T07:39:59.175346+0000","last_fullsized":"2024-09-18T07:39:59.175346+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:11.910474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:11.910474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.028676+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":559,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179938+0000","last_change":"2024-09-18T07:36:02.975929+0000","last_active":"2024-09-18T07:39:59.179938+0000","last_peered":"2024-09-18T07:39:59.179938+0000","last_clean":"2024-09-18T07:39:59.179938+0000","last_became_active":"2024-09-18T07:35:01.407738+0000","last_became_peered":"2024-09-18T07:35:01.407738+0000","last_unstale":"2024-09-18T07:39:59.179938+0000","last_undegraded":"2024-09-18T07:39:59.179938+0000","last_fullsized":"2024-09-18T07:39:59.179938+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:36:02.975863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:02.975863+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:49.805199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":276,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177409+0000","last_change":"2024-09-18T07:36:08.222898+0000","last_active":"2024-09-18T07:39:59.177409+0000","last_peered":"2024-09-18T07:39:59.177409+0000","last_clean":"2024-09-18T07:39:59.177409+0000","last_became_active":"2024-09-18T07:35:01.400197+0000","last_became_peered":"2024-09-18T07:35:01.400197+0000","last_unstale":"2024-09-18T07:39:59.177409+0000","last_undegraded":"2024-09-18T07:39:59.177409+0000","last_fullsized":"2024-09-18T07:39:59.177409+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:08.222838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:08.222838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:16.370989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":614,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178924+0000","last_change":"2024-09-18T07:37:42.285094+0000","last_active":"2024-09-18T07:39:59.178924+0000","last_peered":"2024-09-18T07:39:59.178924+0000","last_clean":"2024-09-18T07:39:59.178924+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:39:59.178924+0000","last_undegraded":"2024-09-18T07:39:59.178924+0000","last_fullsized":"2024-09-18T07:39:59.178924+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:37:42.285042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:37:42.285042+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":509,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.170844+0000","last_change":"2024-09-18T07:39:31.784401+0000","last_active":"2024-09-18T07:39:59.170844+0000","last_peered":"2024-09-18T07:39:59.170844+0000","last_clean":"2024-09-18T07:39:59.170844+0000","last_became_active":"2024-09-18T07:35:01.410561+0000","last_became_peered":"2024-09-18T07:35:01.410561+0000","last_unstale":"2024-09-18T07:39:59.170844+0000","last_undegraded":"2024-09-18T07:39:59.170844+0000","last_fullsized":"2024-09-18T07:39:59.170844+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:31.784355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:31.784355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:37.247787+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":588,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174312+0000","last_change":"2024-09-18T07:38:41.163393+0000","last_active":"2024-09-18T07:39:59.174312+0000","last_peered":"2024-09-18T07:39:59.174312+0000","last_clean":"2024-09-18T07:39:59.174312+0000","last_became_active":"2024-09-18T07:38:41.143108+0000","last_became_peered":"2024-09-18T07:38:41.143108+0000","last_unstale":"2024-09-18T07:39:59.174312+0000","last_undegraded":"2024-09-18T07:39:59.174312+0000","last_fullsized":"2024-09-18T07:39:59.174312+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:36:01.986964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.986964+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:14.037941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":523,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174658+0000","last_change":"2024-09-18T07:37:43.062429+0000","last_active":"2024-09-18T07:39:59.174658+0000","last_peered":"2024-09-18T07:39:59.174658+0000","last_clean":"2024-09-18T07:39:59.174658+0000","last_became_active":"2024-09-18T07:37:43.061577+0000","last_became_peered":"2024-09-18T07:37:43.061577+0000","last_unstale":"2024-09-18T07:39:59.174658+0000","last_undegraded":"2024-09-18T07:39:59.174658+0000","last_fullsized":"2024-09-18T07:39:59.174658+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:39.923066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":194,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178758+0000","last_change":"2024-09-18T07:36:13.118114+0000","last_active":"2024-09-18T07:39:59.178758+0000","last_peered":"2024-09-18T07:39:59.178758+0000","last_clean":"2024-09-18T07:39:59.178758+0000","last_became_active":"2024-09-18T07:36:13.117584+0000","last_became_peered":"2024-09-18T07:36:13.117584+0000","last_unstale":"2024-09-18T07:39:59.178758+0000","last_undegraded":"2024-09-18T07:39:59.178758+0000","last_fullsized":"2024-09-18T07:39:59.178758+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:15.687992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":490,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178643+0000","last_change":"2024-09-18T07:39:37.892445+0000","last_active":"2024-09-18T07:39:59.178643+0000","last_peered":"2024-09-18T07:39:59.178643+0000","last_clean":"2024-09-18T07:39:59.178643+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:39:59.178643+0000","last_undegraded":"2024-09-18T07:39:59.178643+0000","last_fullsized":"2024-09-18T07:39:59.178643+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.892377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.892377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:56.957035+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":376,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.170349+0000","last_change":"2024-09-18T07:37:43.138346+0000","last_active":"2024-09-18T07:39:59.170349+0000","last_peered":"2024-09-18T07:39:59.170349+0000","last_clean":"2024-09-18T07:39:59.170349+0000","last_became_active":"2024-09-18T07:37:43.067598+0000","last_became_peered":"2024-09-18T07:37:43.067598+0000","last_unstale":"2024-09-18T07:39:59.170349+0000","last_undegraded":"2024-09-18T07:39:59.170349+0000","last_fullsized":"2024-09-18T07:39:59.170349+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:36:38.638226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:38.638226+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":276,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.178193+0000","last_change":"2024-09-18T07:35:57.293607+0000","last_active":"2024-09-18T07:39:59.178193+0000","last_peered":"2024-09-18T07:39:59.178193+0000","last_clean":"2024-09-18T07:39:59.178193+0000","last_became_active":"2024-09-18T07:35:01.405287+0000","last_became_peered":"2024-09-18T07:35:01.405287+0000","last_unstale":"2024-09-18T07:39:59.178193+0000","last_undegraded":"2024-09-18T07:39:59.178193+0000","last_fullsized":"2024-09-18T07:39:59.178193+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:57.293555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.293555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:12.492824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":502,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179585+0000","last_change":"2024-09-18T07:38:56.920456+0000","last_active":"2024-09-18T07:39:59.179585+0000","last_peered":"2024-09-18T07:39:59.179585+0000","last_clean":"2024-09-18T07:39:59.179585+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:39:59.179585+0000","last_undegraded":"2024-09-18T07:39:59.179585+0000","last_fullsized":"2024-09-18T07:39:59.179585+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:56.920384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:56.920384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:11.540961+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":503,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174497+0000","last_change":"2024-09-18T07:36:37.593789+0000","last_active":"2024-09-18T07:39:59.174497+0000","last_peered":"2024-09-18T07:39:59.174497+0000","last_clean":"2024-09-18T07:39:59.174497+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:39:59.174497+0000","last_undegraded":"2024-09-18T07:39:59.174497+0000","last_fullsized":"2024-09-18T07:39:59.174497+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:37.593724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:37.593724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":601,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179007+0000","last_change":"2024-09-18T07:37:43.138620+0000","last_active":"2024-09-18T07:39:59.179007+0000","last_peered":"2024-09-18T07:39:59.179007+0000","last_clean":"2024-09-18T07:39:59.179007+0000","last_became_active":"2024-09-18T07:37:43.066208+0000","last_became_peered":"2024-09-18T07:37:43.066208+0000","last_unstale":"2024-09-18T07:39:59.179007+0000","last_undegraded":"2024-09-18T07:39:59.179007+0000","last_fullsized":"2024-09-18T07:39:59.179007+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:36:29.051437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:29.051437+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:53.872186+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":515,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179239+0000","last_change":"2024-09-18T07:35:01.415691+0000","last_active":"2024-09-18T07:39:59.179239+0000","last_peered":"2024-09-18T07:39:59.179239+0000","last_clean":"2024-09-18T07:39:59.179239+0000","last_became_active":"2024-09-18T07:35:01.415261+0000","last_became_peered":"2024-09-18T07:35:01.415261+0000","last_unstale":"2024-09-18T07:39:59.179239+0000","last_undegraded":"2024-09-18T07:39:59.179239+0000","last_fullsized":"2024-09-18T07:39:59.179239+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:58.734027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:58.734027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:17.638037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":290,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177327+0000","last_change":"2024-09-18T07:37:43.057780+0000","last_active":"2024-09-18T07:39:59.177327+0000","last_peered":"2024-09-18T07:39:59.177327+0000","last_clean":"2024-09-18T07:39:59.177327+0000","last_became_active":"2024-09-18T07:37:43.057483+0000","last_became_peered":"2024-09-18T07:37:43.057483+0000","last_unstale":"2024-09-18T07:39:59.177327+0000","last_undegraded":"2024-09-18T07:39:59.177327+0000","last_fullsized":"2024-09-18T07:39:59.177327+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:41.311291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:41.311291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.461425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":520,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179672+0000","last_change":"2024-09-18T07:39:02.930467+0000","last_active":"2024-09-18T07:39:59.179672+0000","last_peered":"2024-09-18T07:39:59.179672+0000","last_clean":"2024-09-18T07:39:59.179672+0000","last_became_active":"2024-09-18T07:37:43.067066+0000","last_became_peered":"2024-09-18T07:37:43.067066+0000","last_unstale":"2024-09-18T07:39:59.179672+0000","last_undegraded":"2024-09-18T07:39:59.179672+0000","last_fullsized":"2024-09-18T07:39:59.179672+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:02.930392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:02.930392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.214721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":559,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177033+0000","last_change":"2024-09-18T07:37:43.134956+0000","last_active":"2024-09-18T07:39:59.177033+0000","last_peered":"2024-09-18T07:39:59.177033+0000","last_clean":"2024-09-18T07:39:59.177033+0000","last_became_active":"2024-09-18T07:37:43.067503+0000","last_became_peered":"2024-09-18T07:37:43.067503+0000","last_unstale":"2024-09-18T07:39:59.177033+0000","last_undegraded":"2024-09-18T07:39:59.177033+0000","last_fullsized":"2024-09-18T07:39:59.177033+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:36:40.908629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:40.908629+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:56.851964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":499,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.174404+0000","last_change":"2024-09-18T07:39:05.648336+0000","last_active":"2024-09-18T07:39:59.174404+0000","last_peered":"2024-09-18T07:39:59.174404+0000","last_clean":"2024-09-18T07:39:59.174404+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:39:59.174404+0000","last_undegraded":"2024-09-18T07:39:59.174404+0000","last_fullsized":"2024-09-18T07:39:59.174404+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:05.648263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.648263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:07.539151+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":134,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.194261+0000","last_change":"2024-09-18T07:37:43.061403+0000","last_active":"2024-09-18T07:39:59.194261+0000","last_peered":"2024-09-18T07:39:59.194261+0000","last_clean":"2024-09-18T07:39:59.194261+0000","last_became_active":"2024-09-18T07:37:43.061081+0000","last_became_peered":"2024-09-18T07:37:43.061081+0000","last_unstale":"2024-09-18T07:39:59.194261+0000","last_undegraded":"2024-09-18T07:39:59.194261+0000","last_fullsized":"2024-09-18T07:39:59.194261+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:30.345180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:30.345180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:39.067422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":284,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177479+0000","last_change":"2024-09-18T07:39:02.503766+0000","last_active":"2024-09-18T07:39:59.177479+0000","last_peered":"2024-09-18T07:39:59.177479+0000","last_clean":"2024-09-18T07:39:59.177479+0000","last_became_active":"2024-09-18T07:35:01.400396+0000","last_became_peered":"2024-09-18T07:35:01.400396+0000","last_unstale":"2024-09-18T07:39:59.177479+0000","last_undegraded":"2024-09-18T07:39:59.177479+0000","last_fullsized":"2024-09-18T07:39:59.177479+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:02.503698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:02.503698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:11.708815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":194,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.179022+0000","last_change":"2024-09-18T07:36:13.118009+0000","last_active":"2024-09-18T07:39:59.179022+0000","last_peered":"2024-09-18T07:39:59.179022+0000","last_clean":"2024-09-18T07:39:59.179022+0000","last_became_active":"2024-09-18T07:36:13.117154+0000","last_became_peered":"2024-09-18T07:36:13.117154+0000","last_unstale":"2024-09-18T07:39:59.179022+0000","last_undegraded":"2024-09-18T07:39:59.179022+0000","last_fullsized":"2024-09-18T07:39:59.179022+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.071629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.071629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:21.615923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":524,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.177364+0000","last_change":"2024-09-18T07:39:35.902385+0000","last_active":"2024-09-18T07:39:59.177364+0000","last_peered":"2024-09-18T07:39:59.177364+0000","last_clean":"2024-09-18T07:39:59.177364+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:39:59.177364+0000","last_undegraded":"2024-09-18T07:39:59.177364+0000","last_fullsized":"2024-09-18T07:39:59.177364+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:39:35.901306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.901306+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.222739+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":147,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.194553+0000","last_change":"2024-09-18T07:37:43.258255+0000","last_active":"2024-09-18T07:39:59.194553+0000","last_peered":"2024-09-18T07:39:59.194553+0000","last_clean":"2024-09-18T07:39:59.194553+0000","last_became_active":"2024-09-18T07:37:43.070186+0000","last_became_peered":"2024-09-18T07:37:43.070186+0000","last_unstale":"2024-09-18T07:39:59.194553+0000","last_undegraded":"2024-09-18T07:39:59.194553+0000","last_fullsized":"2024-09-18T07:39:59.194553+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:34:58.432781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:58.432781+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:22.023484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":522,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-18T07:39:59.170977+0000","last_change":"2024-09-18T07:39:35.694805+0000","last_active":"2024-09-18T07:39:59.170977+0000","last_peered":"2024-09-18T07:39:59.170977+0000","last_clean":"2024-09-18T07:39:59.170977+0000","last_became_active":"2024-09-18T07:35:01.410469+0000","last_became_peered":"2024-09-18T07:35:01.410469+0000","last_unstale":"2024-09-18T07:39:59.170977+0000","last_undegraded":"2024-09-18T07:39:59.170977+0000","last_fullsized":"2024-09-18T07:39:59.170977+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.694741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.694741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.064934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3701407744,"num_objects":56479,"num_object_clones":0,"num_object_copies":169437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56479,"num_write_kb":3614656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11078664192,"data_stored":11078664192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56479,"ondisk_log_size":56479,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":4664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":398,"ondisk_log_size":398,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":306,"num_read_kb":301,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201393,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2304152,"kb_used_data":2002248,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91441000,"statfs":{"total":95995035648,"available":93635584000,"internally_reserved":0,"allocated":2050301952,"data_stored":2049710962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":309131722},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463031,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":909368,"kb_used_data":665768,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":92835784,"statfs":{"total":95995035648,"available":95063842816,"internally_reserved":0,"allocated":681746432,"data_stored":681186795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":249428422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724668,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2317644,"kb_used_data":1993532,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91427508,"statfs":{"total":95995035648,"available":93621768192,"internally_reserved":0,"allocated":2041376768,"data_stored":2040797981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953603,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":958876,"kb_used_data":670856,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":92786276,"statfs":{"total":95995035648,"available":95013146624,"internally_reserved":0,"allocated":686956544,"data_stored":686365277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149831,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2411036,"kb_used_data":2005004,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":91334116,"statfs":{"total":95995035648,"available":93526134784,"internally_reserved":0,"allocated":2053124096,"data_stored":2052530129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":415758791},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411469,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1368840,"kb_used_data":1026680,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92376312,"statfs":{"total":95995035648,"available":94593343488,"internally_reserved":0,"allocated":1051320320,"data_stored":1050745528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574996,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2393828,"kb_used_data":2011412,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":91351324,"statfs":{"total":95995035648,"available":93543755776,"internally_reserved":0,"allocated":2059685888,"data_stored":2059083410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738521,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1352740,"kb_used_data":1036944,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92392412,"statfs":{"total":95995035648,"available":94609829888,"internally_reserved":0,"allocated":1061830656,"data_stored":1061232599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":928579584,"data_stored":928579584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2082799616,"data_stored":2082799616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":697696256,"data_stored":697696256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1866530816,"data_stored":1866530816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":691011584,"data_stored":691011584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2068774912,"data_stored":2068774912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":692256768,"data_stored":692256768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2051014656,"data_stored":2051014656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:40:04.435 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-18T07:40:04.435 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:40:04.435 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-18T07:40:04.435 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-18T07:40:04.436 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:40:04.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:04.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:04.504 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:04.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:04.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:04.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:04.647 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:04.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:04.748 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:04.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:04.792 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:04.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:04.893 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:04.927 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:04.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:04.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:04.956 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:04 smithi137 ceph-mon[29408]: 5.e scrub starts 2024-09-18T07:40:04.957 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:04 smithi137 ceph-mon[29408]: 5.e scrub ok 2024-09-18T07:40:04.957 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:04 smithi137 ceph-mon[29408]: from='client.35621 v1:172.21.15.137:0/851767751' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:40:04.957 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:04 smithi137 ceph-mon[29408]: pgmap v491: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-18T07:40:04.957 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:04 smithi137 ceph-mon[39175]: 5.e scrub starts 2024-09-18T07:40:04.957 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:04 smithi137 ceph-mon[39175]: 5.e scrub ok 2024-09-18T07:40:04.957 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:04 smithi137 ceph-mon[39175]: from='client.35621 v1:172.21.15.137:0/851767751' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:40:04.957 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:04 smithi137 ceph-mon[39175]: pgmap v491: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-18T07:40:04.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:04 smithi203 ceph-mon[32237]: 5.e scrub starts 2024-09-18T07:40:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:04 smithi203 ceph-mon[32237]: 5.e scrub ok 2024-09-18T07:40:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:04 smithi203 ceph-mon[32237]: from='client.35621 v1:172.21.15.137:0/851767751' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:40:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:04 smithi203 ceph-mon[32237]: pgmap v491: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-18T07:40:05.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:05.064 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:05.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:05.078 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 30 16 61224 61208 127.486 93.6875 0.00778393 0.00783946 2024-09-18T07:40:05.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:05.203 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:05.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:05.283 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:05.295 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:40:05.295 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:40:05.305 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:05.307 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:05.331 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:05.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:05.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:05.333 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:40:05.333 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:40:05.364 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:40:05.364 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:40:05.399 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:40:05.399 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:40:05.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:05.434 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:40:05.435 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:40:05.468 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:40:05.468 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:40:05.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:05.477 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:05.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:05.499 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:40:05.500 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:40:05.533 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:40:05.534 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:40:05.579 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:05.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:05.608 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:05.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:05.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:05.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:40:05.744 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:05.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:05.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:05.846 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:05.884 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:05.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:05.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:05 smithi137 ceph-mon[29408]: 4.7 scrub starts 2024-09-18T07:40:05.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:05 smithi137 ceph-mon[29408]: 4.7 scrub ok 2024-09-18T07:40:05.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:05 smithi137 ceph-mon[29408]: 4.d scrub starts 2024-09-18T07:40:05.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:05 smithi137 ceph-mon[29408]: 4.d scrub ok 2024-09-18T07:40:05.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:05 smithi137 ceph-mon[29408]: osdmap e303: 8 total, 8 up, 8 in 2024-09-18T07:40:05.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:05 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/762638646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:05.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:05 smithi137 ceph-mon[39175]: 4.7 scrub starts 2024-09-18T07:40:05.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:05 smithi137 ceph-mon[39175]: 4.7 scrub ok 2024-09-18T07:40:05.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:05 smithi137 ceph-mon[39175]: 4.d scrub starts 2024-09-18T07:40:05.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:05 smithi137 ceph-mon[39175]: 4.d scrub ok 2024-09-18T07:40:05.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:05 smithi137 ceph-mon[39175]: osdmap e303: 8 total, 8 up, 8 in 2024-09-18T07:40:05.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:05 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/762638646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:05.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:05 smithi203 ceph-mon[32237]: 4.7 scrub starts 2024-09-18T07:40:05.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:05 smithi203 ceph-mon[32237]: 4.7 scrub ok 2024-09-18T07:40:05.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:05 smithi203 ceph-mon[32237]: 4.d scrub starts 2024-09-18T07:40:05.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:05 smithi203 ceph-mon[32237]: 4.d scrub ok 2024-09-18T07:40:05.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:05 smithi203 ceph-mon[32237]: osdmap e303: 8 total, 8 up, 8 in 2024-09-18T07:40:05.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:05 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/762638646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:05.986 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:06.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:06.017 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:06.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:06.078 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 31 15 62566 62551 126.081 83.9375 0.00645729 0.00792638 2024-09-18T07:40:06.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:06.155 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:06.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:06.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:06.257 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:06.292 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:06.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:06.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:06.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:06.434 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:06.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:06.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:06.536 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:06.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:06.560 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:06.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:06.661 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:06.696 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:06.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:06.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:06.842 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:06.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:06.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:06.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:06.989 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:06.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:06.997 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:40:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:40:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:63284 9.2 e303/289 9:4c9b61be:::benchmark_data_smithi137_126072_object63283:head v 303'4030, mlcod=303'4028)", 2024-09-18T07:40:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:40:06.601801+0000", 2024-09-18T07:40:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.002438965, 2024-09-18T07:40:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024662809999999999, 2024-09-18T07:40:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:40:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.601801+0000", 2024-09-18T07:40:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.601801+0000", 2024-09-18T07:40:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.601802+0000", 2024-09-18T07:40:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2669999999999999e-06 2024-09-18T07:40:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.601826+0000", 2024-09-18T07:40:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3915000000000001e-05 2024-09-18T07:40:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.601839+0000", 2024-09-18T07:40:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3346000000000001e-05 2024-09-18T07:40:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.601847+0000", 2024-09-18T07:40:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4390000000000003e-06 2024-09-18T07:40:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.601943+0000", 2024-09-18T07:40:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6648e-05 2024-09-18T07:40:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:40:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.601963+0000", 2024-09-18T07:40:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9168999999999999e-05 2024-09-18T07:40:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:63287 9.a e303/289 9:5ee0e771:::benchmark_data_smithi137_126072_object63286:head v 303'4020, mlcod=303'4018)", 2024-09-18T07:40:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:40:06.603322+0000", 2024-09-18T07:40:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00091786199999999997, 2024-09-18T07:40:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0009973549999999999, 2024-09-18T07:40:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:40:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.603322+0000", 2024-09-18T07:40:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.603322+0000", 2024-09-18T07:40:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.603324+0000", 2024-09-18T07:40:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.782e-06 2024-09-18T07:40:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.603358+0000", 2024-09-18T07:40:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3865000000000002e-05 2024-09-18T07:40:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.603369+0000", 2024-09-18T07:40:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1430000000000001e-05 2024-09-18T07:40:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.603376+0000", 2024-09-18T07:40:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4850000000000003e-06 2024-09-18T07:40:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.603947+0000", 2024-09-18T07:40:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00057104499999999995 2024-09-18T07:40:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:40:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.603966+0000", 2024-09-18T07:40:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8929000000000001e-05 2024-09-18T07:40:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:63288 9.6 e303/289 9:6fbebfd1:::benchmark_data_smithi137_126072_object63287:head v 303'3942, mlcod=303'3940)", 2024-09-18T07:40:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:40:06.603391+0000", 2024-09-18T07:40:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00084852199999999999, 2024-09-18T07:40:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00097553899999999999, 2024-09-18T07:40:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:40:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.603391+0000", 2024-09-18T07:40:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.603391+0000", 2024-09-18T07:40:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.603392+0000", 2024-09-18T07:40:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4099999999999999e-07 2024-09-18T07:40:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.603425+0000", 2024-09-18T07:40:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3349999999999997e-05 2024-09-18T07:40:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.603432+0000", 2024-09-18T07:40:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8669999999999996e-06 2024-09-18T07:40:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.603437+0000", 2024-09-18T07:40:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4580000000000001e-06 2024-09-18T07:40:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.604095+0000", 2024-09-18T07:40:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00065801100000000001 2024-09-18T07:40:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:40:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:40:06.604115+0000", 2024-09-18T07:40:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9321e-05 2024-09-18T07:40:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:40:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 3 2024-09-18T07:40:07.012 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:40:07.078 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 32 16 64026 64010 124.99 91.1875 0.0100999 0.00799547 2024-09-18T07:40:07.090 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:07.131 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:07.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:07.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:07.232 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:07.251 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:06 smithi137 ceph-mon[29408]: 5.7 scrub starts 2024-09-18T07:40:07.251 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:06 smithi137 ceph-mon[29408]: 5.7 scrub ok 2024-09-18T07:40:07.251 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:06 smithi137 ceph-mon[29408]: pgmap v493: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 793 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-18T07:40:07.251 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:06 smithi137 ceph-mon[29408]: 6.10 scrub starts 2024-09-18T07:40:07.251 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:06 smithi137 ceph-mon[29408]: 6.10 scrub ok 2024-09-18T07:40:07.251 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:06 smithi137 ceph-mon[29408]: 6.6 scrub starts 2024-09-18T07:40:07.251 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:06 smithi137 ceph-mon[29408]: 6.6 scrub ok 2024-09-18T07:40:07.252 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:40:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:40:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:40:07.252 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:06 smithi137 ceph-mon[39175]: 5.7 scrub starts 2024-09-18T07:40:07.252 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:06 smithi137 ceph-mon[39175]: 5.7 scrub ok 2024-09-18T07:40:07.252 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:06 smithi137 ceph-mon[39175]: pgmap v493: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 793 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-18T07:40:07.252 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:06 smithi137 ceph-mon[39175]: 6.10 scrub starts 2024-09-18T07:40:07.252 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:06 smithi137 ceph-mon[39175]: 6.10 scrub ok 2024-09-18T07:40:07.252 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:06 smithi137 ceph-mon[39175]: 6.6 scrub starts 2024-09-18T07:40:07.252 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:06 smithi137 ceph-mon[39175]: 6.6 scrub ok 2024-09-18T07:40:07.269 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:07.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:07.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:07.372 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:07.376 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:06 smithi203 ceph-mon[32237]: 5.7 scrub starts 2024-09-18T07:40:07.376 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:06 smithi203 ceph-mon[32237]: 5.7 scrub ok 2024-09-18T07:40:07.376 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:06 smithi203 ceph-mon[32237]: pgmap v493: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 793 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-18T07:40:07.376 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:06 smithi203 ceph-mon[32237]: 6.10 scrub starts 2024-09-18T07:40:07.376 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:06 smithi203 ceph-mon[32237]: 6.10 scrub ok 2024-09-18T07:40:07.376 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:06 smithi203 ceph-mon[32237]: 6.6 scrub starts 2024-09-18T07:40:07.376 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:06 smithi203 ceph-mon[32237]: 6.6 scrub ok 2024-09-18T07:40:07.415 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:07.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:07.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:07.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:07.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:07.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:07.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:07.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:07.702 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:07.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:07.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:07.710 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:40:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:40:07.804 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:07.838 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:07.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:07.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:07.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:07.975 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:07.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:08.077 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:08.078 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 33 16 65543 65527 124.076 94.8125 0.00789083 0.00805438 2024-09-18T07:40:08.110 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:08.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:08.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:08.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:07 smithi137 ceph-mon[29408]: 5.16 scrub starts 2024-09-18T07:40:08.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:07 smithi137 ceph-mon[29408]: 5.16 scrub ok 2024-09-18T07:40:08.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:07 smithi137 ceph-mon[29408]: 4.12 scrub starts 2024-09-18T07:40:08.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:07 smithi137 ceph-mon[29408]: 4.12 scrub ok 2024-09-18T07:40:08.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:07 smithi137 ceph-mon[39175]: 5.16 scrub starts 2024-09-18T07:40:08.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:07 smithi137 ceph-mon[39175]: 5.16 scrub ok 2024-09-18T07:40:08.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:07 smithi137 ceph-mon[39175]: 4.12 scrub starts 2024-09-18T07:40:08.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:07 smithi137 ceph-mon[39175]: 4.12 scrub ok 2024-09-18T07:40:08.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:08.258 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:08.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:08.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:08.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T07:40:08.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:08.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:40:08.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:08.425 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:08.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:08.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:07 smithi203 ceph-mon[32237]: 5.16 scrub starts 2024-09-18T07:40:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:07 smithi203 ceph-mon[32237]: 5.16 scrub ok 2024-09-18T07:40:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:07 smithi203 ceph-mon[32237]: 4.12 scrub starts 2024-09-18T07:40:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:07 smithi203 ceph-mon[32237]: 4.12 scrub ok 2024-09-18T07:40:08.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:08.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:08.599 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:08.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:08.619 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:08.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:08.755 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:08.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:08.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:08.857 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:08.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:08.897 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:08.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:08.946 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:40:08.948 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:40:08.998 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:09.032 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:09.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:09.052 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:08 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:09.052 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:08 smithi137 ceph-mon[29408]: pgmap v494: 158 pgs: 158 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-09-18T07:40:09.052 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:08 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:09.052 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:08 smithi137 ceph-mon[39175]: pgmap v494: 158 pgs: 158 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-09-18T07:40:09.078 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 34 16 66774 66758 122.689 76.9375 0.0126952 0.00814588 2024-09-18T07:40:09.134 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:09.182 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:09.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:09.283 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:09.323 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:09.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:09.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:09.424 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:09.457 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:09.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:09.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:09.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:08 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:09.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:08 smithi203 ceph-mon[32237]: pgmap v494: 158 pgs: 158 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-09-18T07:40:09.559 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:09.592 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:09.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:09.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:09.694 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:09.732 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:09.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:09.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:09.783 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-09-18T07:40:07.625361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4185266715,"num_objects":64052,"num_object_clones":0,"num_object_copies":192156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64052,"num_whiteouts":0,"num_read":1015,"num_read_kb":767,"num_write":64289,"num_write_kb":4086637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":5625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65215,"ondisk_log_size":65215,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":479,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15407156,"kb_used_data":12729564,"kb_used_omap":12,"kb_used_meta":2677427,"kb_avail":734554060,"statfs":{"total":767960285184,"available":752183357440,"internally_reserved":0,"allocated":13035073536,"data_stored":13030301545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2741685808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":948633600,"num_objects":14475,"num_object_clones":0,"num_object_copies":43425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14475,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":14475,"num_write_kb":926400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.011483"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":520,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.964543+0000","last_change":"2024-09-18T07:39:36.700902+0000","last_active":"2024-09-18T07:40:04.964543+0000","last_peered":"2024-09-18T07:40:04.964543+0000","last_clean":"2024-09-18T07:40:04.964543+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:40:04.964543+0000","last_undegraded":"2024-09-18T07:40:04.964543+0000","last_fullsized":"2024-09-18T07:40:04.964543+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:36.700832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:36.700832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:57.523488+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":586,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.947459+0000","last_change":"2024-09-18T07:38:52.141085+0000","last_active":"2024-09-18T07:40:04.947459+0000","last_peered":"2024-09-18T07:40:04.947459+0000","last_clean":"2024-09-18T07:40:04.947459+0000","last_became_active":"2024-09-18T07:37:43.061125+0000","last_became_peered":"2024-09-18T07:37:43.061125+0000","last_unstale":"2024-09-18T07:40:04.947459+0000","last_undegraded":"2024-09-18T07:40:04.947459+0000","last_fullsized":"2024-09-18T07:40:04.947459+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:38:52.140959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:38:52.140959+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:17.219305+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":455,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963245+0000","last_change":"2024-09-18T07:36:38.570405+0000","last_active":"2024-09-18T07:40:04.963245+0000","last_peered":"2024-09-18T07:40:04.963245+0000","last_clean":"2024-09-18T07:40:04.963245+0000","last_became_active":"2024-09-18T07:36:13.128805+0000","last_became_peered":"2024-09-18T07:36:13.128805+0000","last_unstale":"2024-09-18T07:40:04.963245+0000","last_undegraded":"2024-09-18T07:40:04.963245+0000","last_fullsized":"2024-09-18T07:40:04.963245+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:38.570339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:38.570339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":495,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.948215+0000","last_change":"2024-09-18T07:32:03.953881+0000","last_active":"2024-09-18T07:40:04.948215+0000","last_peered":"2024-09-18T07:40:04.948215+0000","last_clean":"2024-09-18T07:40:04.948215+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:40:04.948215+0000","last_undegraded":"2024-09-18T07:40:04.948215+0000","last_fullsized":"2024-09-18T07:40:04.948215+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:03.953827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:32:03.953827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:44.967981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":414,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.961592+0000","last_change":"2024-09-18T07:39:33.737527+0000","last_active":"2024-09-18T07:40:04.961592+0000","last_peered":"2024-09-18T07:40:04.961592+0000","last_clean":"2024-09-18T07:40:04.961592+0000","last_became_active":"2024-09-18T07:37:43.056380+0000","last_became_peered":"2024-09-18T07:37:43.056380+0000","last_unstale":"2024-09-18T07:40:04.961592+0000","last_undegraded":"2024-09-18T07:40:04.961592+0000","last_fullsized":"2024-09-18T07:40:04.961592+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:33.737478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:33.737478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:52.567818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":521,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.957537+0000","last_change":"2024-09-18T07:37:43.066814+0000","last_active":"2024-09-18T07:40:04.957537+0000","last_peered":"2024-09-18T07:40:04.957537+0000","last_clean":"2024-09-18T07:40:04.957537+0000","last_became_active":"2024-09-18T07:37:43.066508+0000","last_became_peered":"2024-09-18T07:37:43.066508+0000","last_unstale":"2024-09-18T07:40:04.957537+0000","last_undegraded":"2024-09-18T07:40:04.957537+0000","last_fullsized":"2024-09-18T07:40:04.957537+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:27.006488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:33:27.006488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:46.846955+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":527,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.957633+0000","last_change":"2024-09-18T07:33:22.032321+0000","last_active":"2024-09-18T07:40:04.957633+0000","last_peered":"2024-09-18T07:40:04.957633+0000","last_clean":"2024-09-18T07:40:04.957633+0000","last_became_active":"2024-09-18T07:30:58.313428+0000","last_became_peered":"2024-09-18T07:30:58.313428+0000","last_unstale":"2024-09-18T07:40:04.957633+0000","last_undegraded":"2024-09-18T07:40:04.957633+0000","last_fullsized":"2024-09-18T07:40:04.957633+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:33:22.032110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.032110+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:47.872878+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":521,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.947366+0000","last_change":"2024-09-18T07:37:43.052675+0000","last_active":"2024-09-18T07:40:04.947366+0000","last_peered":"2024-09-18T07:40:04.947366+0000","last_clean":"2024-09-18T07:40:04.947366+0000","last_became_active":"2024-09-18T07:37:43.052346+0000","last_became_peered":"2024-09-18T07:37:43.052346+0000","last_unstale":"2024-09-18T07:40:04.947366+0000","last_undegraded":"2024-09-18T07:40:04.947366+0000","last_fullsized":"2024-09-18T07:40:04.947366+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:30.995282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:30.995282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:11.031402+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":527,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963869+0000","last_change":"2024-09-18T07:37:43.080879+0000","last_active":"2024-09-18T07:40:04.963869+0000","last_peered":"2024-09-18T07:40:04.963869+0000","last_clean":"2024-09-18T07:40:04.963869+0000","last_became_active":"2024-09-18T07:37:43.080314+0000","last_became_peered":"2024-09-18T07:37:43.080314+0000","last_unstale":"2024-09-18T07:40:04.963869+0000","last_undegraded":"2024-09-18T07:40:04.963869+0000","last_fullsized":"2024-09-18T07:40:04.963869+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:00.392466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.392466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:20.640427+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":519,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.948432+0000","last_change":"2024-09-18T07:32:02.002919+0000","last_active":"2024-09-18T07:40:04.948432+0000","last_peered":"2024-09-18T07:40:04.948432+0000","last_clean":"2024-09-18T07:40:04.948432+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:40:04.948432+0000","last_undegraded":"2024-09-18T07:40:04.948432+0000","last_fullsized":"2024-09-18T07:40:04.948432+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:32:02.002811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:02.002811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:43.931181+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":511,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.962829+0000","last_change":"2024-09-18T07:35:04.267623+0000","last_active":"2024-09-18T07:40:04.962829+0000","last_peered":"2024-09-18T07:40:04.962829+0000","last_clean":"2024-09-18T07:40:04.962829+0000","last_became_active":"2024-09-18T07:35:01.413443+0000","last_became_peered":"2024-09-18T07:35:01.413443+0000","last_unstale":"2024-09-18T07:40:04.962829+0000","last_undegraded":"2024-09-18T07:40:04.962829+0000","last_fullsized":"2024-09-18T07:40:04.962829+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:04.267580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:04.267580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:22.587151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":505,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.962544+0000","last_change":"2024-09-18T07:36:10.551960+0000","last_active":"2024-09-18T07:40:04.962544+0000","last_peered":"2024-09-18T07:40:04.962544+0000","last_clean":"2024-09-18T07:40:04.962544+0000","last_became_active":"2024-09-18T07:35:01.410284+0000","last_became_peered":"2024-09-18T07:35:01.410284+0000","last_unstale":"2024-09-18T07:40:04.962544+0000","last_undegraded":"2024-09-18T07:40:04.962544+0000","last_fullsized":"2024-09-18T07:40:04.962544+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:10.551889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:10.551889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:58.693520+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":497,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.962976+0000","last_change":"2024-09-18T07:30:56.282720+0000","last_active":"2024-09-18T07:40:04.962976+0000","last_peered":"2024-09-18T07:40:04.962976+0000","last_clean":"2024-09-18T07:40:04.962976+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:40:04.962976+0000","last_undegraded":"2024-09-18T07:40:04.962976+0000","last_fullsized":"2024-09-18T07:40:04.962976+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:10.625350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":534,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.947566+0000","last_change":"2024-09-18T07:39:26.028935+0000","last_active":"2024-09-18T07:40:04.947566+0000","last_peered":"2024-09-18T07:40:04.947566+0000","last_clean":"2024-09-18T07:40:04.947566+0000","last_became_active":"2024-09-18T07:36:02.526212+0000","last_became_peered":"2024-09-18T07:36:02.526212+0000","last_unstale":"2024-09-18T07:40:04.947566+0000","last_undegraded":"2024-09-18T07:40:04.947566+0000","last_fullsized":"2024-09-18T07:40:04.947566+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-18T07:39:26.028884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:26.028884+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:28.305457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":280,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949759+0000","last_change":"2024-09-18T07:38:54.458848+0000","last_active":"2024-09-18T07:40:04.949759+0000","last_peered":"2024-09-18T07:40:04.949759+0000","last_clean":"2024-09-18T07:40:04.949759+0000","last_became_active":"2024-09-18T07:35:01.398605+0000","last_became_peered":"2024-09-18T07:35:01.398605+0000","last_unstale":"2024-09-18T07:40:04.949759+0000","last_undegraded":"2024-09-18T07:40:04.949759+0000","last_fullsized":"2024-09-18T07:40:04.949759+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:54.458787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:54.458787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.620791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":517,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963343+0000","last_change":"2024-09-18T07:39:30.429935+0000","last_active":"2024-09-18T07:40:04.963343+0000","last_peered":"2024-09-18T07:40:04.963343+0000","last_clean":"2024-09-18T07:40:04.963343+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:40:04.963343+0000","last_undegraded":"2024-09-18T07:40:04.963343+0000","last_fullsized":"2024-09-18T07:40:04.963343+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:30.429890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:30.429890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:38.713228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":526,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.948075+0000","last_change":"2024-09-18T07:38:56.096769+0000","last_active":"2024-09-18T07:40:04.948075+0000","last_peered":"2024-09-18T07:40:04.948075+0000","last_clean":"2024-09-18T07:40:04.948075+0000","last_became_active":"2024-09-18T07:37:43.067568+0000","last_became_peered":"2024-09-18T07:37:43.067568+0000","last_unstale":"2024-09-18T07:40:04.948075+0000","last_undegraded":"2024-09-18T07:40:04.948075+0000","last_fullsized":"2024-09-18T07:40:04.948075+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:56.096591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:56.096591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:24.052212+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":525,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.962787+0000","last_change":"2024-09-18T07:36:06.523357+0000","last_active":"2024-09-18T07:40:04.962787+0000","last_peered":"2024-09-18T07:40:04.962787+0000","last_clean":"2024-09-18T07:40:04.962787+0000","last_became_active":"2024-09-18T07:35:01.414088+0000","last_became_peered":"2024-09-18T07:35:01.414088+0000","last_unstale":"2024-09-18T07:40:04.962787+0000","last_undegraded":"2024-09-18T07:40:04.962787+0000","last_fullsized":"2024-09-18T07:40:04.962787+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:06.523292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.523292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:00.748838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":582,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949708+0000","last_change":"2024-09-18T07:39:04.314241+0000","last_active":"2024-09-18T07:40:04.949708+0000","last_peered":"2024-09-18T07:40:04.949708+0000","last_clean":"2024-09-18T07:40:04.949708+0000","last_became_active":"2024-09-18T07:37:28.002858+0000","last_became_peered":"2024-09-18T07:37:28.002858+0000","last_unstale":"2024-09-18T07:40:04.949708+0000","last_undegraded":"2024-09-18T07:40:04.949708+0000","last_fullsized":"2024-09-18T07:40:04.949708+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:39:04.313784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:04.313784+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:09.030441+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":198,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.947788+0000","last_change":"2024-09-18T07:36:13.116882+0000","last_active":"2024-09-18T07:40:04.947788+0000","last_peered":"2024-09-18T07:40:04.947788+0000","last_clean":"2024-09-18T07:40:04.947788+0000","last_became_active":"2024-09-18T07:36:13.116416+0000","last_became_peered":"2024-09-18T07:36:13.116416+0000","last_unstale":"2024-09-18T07:40:04.947788+0000","last_undegraded":"2024-09-18T07:40:04.947788+0000","last_fullsized":"2024-09-18T07:40:04.947788+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:10.074367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:10.074367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:21.035466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":517,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.957874+0000","last_change":"2024-09-18T07:37:43.067816+0000","last_active":"2024-09-18T07:40:04.957874+0000","last_peered":"2024-09-18T07:40:04.957874+0000","last_clean":"2024-09-18T07:40:04.957874+0000","last_became_active":"2024-09-18T07:37:43.067493+0000","last_became_peered":"2024-09-18T07:37:43.067493+0000","last_unstale":"2024-09-18T07:40:04.957874+0000","last_undegraded":"2024-09-18T07:40:04.957874+0000","last_fullsized":"2024-09-18T07:40:04.957874+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:34.992350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:34.992350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:11.064160+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":523,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.948521+0000","last_change":"2024-09-18T07:39:00.130395+0000","last_active":"2024-09-18T07:40:04.948521+0000","last_peered":"2024-09-18T07:40:04.948521+0000","last_clean":"2024-09-18T07:40:04.948521+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:40:04.948521+0000","last_undegraded":"2024-09-18T07:40:04.948521+0000","last_fullsized":"2024-09-18T07:40:04.948521+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:00.130344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.130344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:22.555080+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":169,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949873+0000","last_change":"2024-09-18T07:37:43.756858+0000","last_active":"2024-09-18T07:40:04.949873+0000","last_peered":"2024-09-18T07:40:04.949873+0000","last_clean":"2024-09-18T07:40:04.949873+0000","last_became_active":"2024-09-18T07:37:27.981705+0000","last_became_peered":"2024-09-18T07:37:27.981705+0000","last_unstale":"2024-09-18T07:40:04.949873+0000","last_undegraded":"2024-09-18T07:40:04.949873+0000","last_fullsized":"2024-09-18T07:40:04.949873+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:37:43.756403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:37:43.756403+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:53.387865+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":499,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.947707+0000","last_change":"2024-09-18T07:39:37.930497+0000","last_active":"2024-09-18T07:40:04.947707+0000","last_peered":"2024-09-18T07:40:04.947707+0000","last_clean":"2024-09-18T07:40:04.947707+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:40:04.947707+0000","last_undegraded":"2024-09-18T07:40:04.947707+0000","last_fullsized":"2024-09-18T07:40:04.947707+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.930435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.930435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:55.595621+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":292,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.950196+0000","last_change":"2024-09-18T07:38:56.497595+0000","last_active":"2024-09-18T07:40:04.950196+0000","last_peered":"2024-09-18T07:40:04.950196+0000","last_clean":"2024-09-18T07:40:04.950196+0000","last_became_active":"2024-09-18T07:35:01.413110+0000","last_became_peered":"2024-09-18T07:35:01.413110+0000","last_unstale":"2024-09-18T07:40:04.950196+0000","last_undegraded":"2024-09-18T07:40:04.950196+0000","last_fullsized":"2024-09-18T07:40:04.950196+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:38:56.497544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:56.497544+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:05.310758+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":215,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.957487+0000","last_change":"2024-09-18T07:38:50.918758+0000","last_active":"2024-09-18T07:40:04.957487+0000","last_peered":"2024-09-18T07:40:04.957487+0000","last_clean":"2024-09-18T07:40:04.957487+0000","last_became_active":"2024-09-18T07:36:13.116295+0000","last_became_peered":"2024-09-18T07:36:13.116295+0000","last_unstale":"2024-09-18T07:40:04.957487+0000","last_undegraded":"2024-09-18T07:40:04.957487+0000","last_fullsized":"2024-09-18T07:40:04.957487+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:38:50.918686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:38:50.918686+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:19.981998+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":561,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.964284+0000","last_change":"2024-09-18T07:35:57.488019+0000","last_active":"2024-09-18T07:40:04.964284+0000","last_peered":"2024-09-18T07:40:04.964284+0000","last_clean":"2024-09-18T07:40:04.964284+0000","last_became_active":"2024-09-18T07:35:01.410926+0000","last_became_peered":"2024-09-18T07:35:01.410926+0000","last_unstale":"2024-09-18T07:40:04.964284+0000","last_undegraded":"2024-09-18T07:40:04.964284+0000","last_fullsized":"2024-09-18T07:40:04.964284+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:35:57.487960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.487960+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:52.681351+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"240'12","reported_seq":180,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.953373+0000","last_change":"2024-09-18T07:39:33.979364+0000","last_active":"2024-09-18T07:40:04.953373+0000","last_peered":"2024-09-18T07:40:04.953373+0000","last_clean":"2024-09-18T07:40:04.953373+0000","last_became_active":"2024-09-18T07:37:43.060715+0000","last_became_peered":"2024-09-18T07:37:43.060715+0000","last_unstale":"2024-09-18T07:40:04.953373+0000","last_undegraded":"2024-09-18T07:40:04.953373+0000","last_fullsized":"2024-09-18T07:40:04.953373+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:39:33.979291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:33.979291+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:39.389550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"303'4073","reported_seq":4112,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:07.077318+0000","last_change":"2024-09-18T07:39:29.700748+0000","last_active":"2024-09-18T07:40:07.077318+0000","last_peered":"2024-09-18T07:40:07.077318+0000","last_clean":"2024-09-18T07:40:07.077318+0000","last_became_active":"2024-09-18T07:39:29.700477+0000","last_became_peered":"2024-09-18T07:39:29.700477+0000","last_unstale":"2024-09-18T07:40:07.077318+0000","last_undegraded":"2024-09-18T07:40:07.077318+0000","last_fullsized":"2024-09-18T07:40:07.077318+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":4073,"log_dups_size":0,"ondisk_log_size":4073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.879265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266928128,"num_objects":4073,"num_object_clones":0,"num_object_copies":12219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4073,"num_write_kb":260672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":76,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.962327+0000","last_change":"2024-09-18T07:39:06.546309+0000","last_active":"2024-09-18T07:40:04.962327+0000","last_peered":"2024-09-18T07:40:04.962327+0000","last_clean":"2024-09-18T07:40:04.962327+0000","last_became_active":"2024-09-18T07:39:04.962047+0000","last_became_peered":"2024-09-18T07:39:04.962047+0000","last_unstale":"2024-09-18T07:40:04.962327+0000","last_undegraded":"2024-09-18T07:40:04.962327+0000","last_fullsized":"2024-09-18T07:40:04.962327+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:06.546237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:06.546237+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:25.791546+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":251,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963390+0000","last_change":"2024-09-18T07:38:54.967706+0000","last_active":"2024-09-18T07:40:04.963390+0000","last_peered":"2024-09-18T07:40:04.963390+0000","last_clean":"2024-09-18T07:40:04.963390+0000","last_became_active":"2024-09-18T07:36:13.119098+0000","last_became_peered":"2024-09-18T07:36:13.119098+0000","last_unstale":"2024-09-18T07:40:04.963390+0000","last_undegraded":"2024-09-18T07:40:04.963390+0000","last_fullsized":"2024-09-18T07:40:04.963390+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:38:54.967643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:38:54.967643+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:18.692985+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":545,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.957579+0000","last_change":"2024-09-18T07:39:29.049261+0000","last_active":"2024-09-18T07:40:04.957579+0000","last_peered":"2024-09-18T07:40:04.957579+0000","last_clean":"2024-09-18T07:40:04.957579+0000","last_became_active":"2024-09-18T07:35:01.407339+0000","last_became_peered":"2024-09-18T07:35:01.407339+0000","last_unstale":"2024-09-18T07:40:04.957579+0000","last_undegraded":"2024-09-18T07:40:04.957579+0000","last_fullsized":"2024-09-18T07:40:04.957579+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.049209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.049209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:31.230030+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":643,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.948214+0000","last_change":"2024-09-18T07:40:03.237115+0000","last_active":"2024-09-18T07:40:04.948214+0000","last_peered":"2024-09-18T07:40:04.948214+0000","last_clean":"2024-09-18T07:40:04.948214+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:40:04.948214+0000","last_undegraded":"2024-09-18T07:40:04.948214+0000","last_fullsized":"2024-09-18T07:40:04.948214+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:40:03.237041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:03.237041+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:25.843038+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":502,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963186+0000","last_change":"2024-09-18T07:36:07.588483+0000","last_active":"2024-09-18T07:40:04.963186+0000","last_peered":"2024-09-18T07:40:04.963186+0000","last_clean":"2024-09-18T07:40:04.963186+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:40:04.963186+0000","last_undegraded":"2024-09-18T07:40:04.963186+0000","last_fullsized":"2024-09-18T07:40:04.963186+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:07.588440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:07.588440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:58.358099+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"303'4009","reported_seq":4048,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:06.516749+0000","last_change":"2024-09-18T07:39:29.704994+0000","last_active":"2024-09-18T07:40:06.516749+0000","last_peered":"2024-09-18T07:40:06.516749+0000","last_clean":"2024-09-18T07:40:06.516749+0000","last_became_active":"2024-09-18T07:39:29.704590+0000","last_became_peered":"2024-09-18T07:39:29.704590+0000","last_unstale":"2024-09-18T07:40:06.516749+0000","last_undegraded":"2024-09-18T07:40:06.516749+0000","last_fullsized":"2024-09-18T07:40:06.516749+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":4009,"log_dups_size":0,"ondisk_log_size":4009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:33.304050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262733824,"num_objects":4009,"num_object_clones":0,"num_object_copies":12027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4009,"num_write_kb":256576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":76,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.948075+0000","last_change":"2024-09-18T07:39:05.085099+0000","last_active":"2024-09-18T07:40:04.948075+0000","last_peered":"2024-09-18T07:40:04.948075+0000","last_clean":"2024-09-18T07:40:04.948075+0000","last_became_active":"2024-09-18T07:39:04.955645+0000","last_became_peered":"2024-09-18T07:39:04.955645+0000","last_unstale":"2024-09-18T07:40:04.948075+0000","last_undegraded":"2024-09-18T07:40:04.948075+0000","last_fullsized":"2024-09-18T07:40:04.948075+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.084956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.084956+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:18.027466+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":601,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.957016+0000","last_change":"2024-09-18T07:39:00.919075+0000","last_active":"2024-09-18T07:40:04.957016+0000","last_peered":"2024-09-18T07:40:04.957016+0000","last_clean":"2024-09-18T07:40:04.957016+0000","last_became_active":"2024-09-18T07:35:01.418909+0000","last_became_peered":"2024-09-18T07:35:01.418909+0000","last_unstale":"2024-09-18T07:40:04.957016+0000","last_undegraded":"2024-09-18T07:40:04.957016+0000","last_fullsized":"2024-09-18T07:40:04.957016+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:39:00.918860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.918860+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:29.657171+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":537,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963696+0000","last_change":"2024-09-18T07:39:06.987818+0000","last_active":"2024-09-18T07:40:04.963696+0000","last_peered":"2024-09-18T07:40:04.963696+0000","last_clean":"2024-09-18T07:40:04.963696+0000","last_became_active":"2024-09-18T07:35:01.407422+0000","last_became_peered":"2024-09-18T07:35:01.407422+0000","last_unstale":"2024-09-18T07:40:04.963696+0000","last_undegraded":"2024-09-18T07:40:04.963696+0000","last_fullsized":"2024-09-18T07:40:04.963696+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:06.987428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:06.987428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:25.403434+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":522,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.947522+0000","last_change":"2024-09-18T07:36:30.242479+0000","last_active":"2024-09-18T07:40:04.947522+0000","last_peered":"2024-09-18T07:40:04.947522+0000","last_clean":"2024-09-18T07:40:04.947522+0000","last_became_active":"2024-09-18T07:35:01.401024+0000","last_became_peered":"2024-09-18T07:35:01.401024+0000","last_unstale":"2024-09-18T07:40:04.947522+0000","last_undegraded":"2024-09-18T07:40:04.947522+0000","last_fullsized":"2024-09-18T07:40:04.947522+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:30.242420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:30.242420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:57.178241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":488,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.948370+0000","last_change":"2024-09-18T07:31:02.348201+0000","last_active":"2024-09-18T07:40:04.948370+0000","last_peered":"2024-09-18T07:40:04.948370+0000","last_clean":"2024-09-18T07:40:04.948370+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:40:04.948370+0000","last_undegraded":"2024-09-18T07:40:04.948370+0000","last_fullsized":"2024-09-18T07:40:04.948370+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:41.905293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"303'4074","reported_seq":4113,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:07.084455+0000","last_change":"2024-09-18T07:39:29.703301+0000","last_active":"2024-09-18T07:40:07.084455+0000","last_peered":"2024-09-18T07:40:07.084455+0000","last_clean":"2024-09-18T07:40:07.084455+0000","last_became_active":"2024-09-18T07:39:29.702953+0000","last_became_peered":"2024-09-18T07:39:29.702953+0000","last_unstale":"2024-09-18T07:40:07.084455+0000","last_undegraded":"2024-09-18T07:40:07.084455+0000","last_fullsized":"2024-09-18T07:40:07.084455+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":4074,"log_dups_size":0,"ondisk_log_size":4074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.620109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266993664,"num_objects":4074,"num_object_clones":0,"num_object_copies":12222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4074,"num_write_kb":260736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":81,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.957822+0000","last_change":"2024-09-18T07:39:05.959886+0000","last_active":"2024-09-18T07:40:04.957822+0000","last_peered":"2024-09-18T07:40:04.957822+0000","last_clean":"2024-09-18T07:40:04.957822+0000","last_became_active":"2024-09-18T07:39:04.945385+0000","last_became_peered":"2024-09-18T07:39:04.945385+0000","last_unstale":"2024-09-18T07:40:04.957822+0000","last_undegraded":"2024-09-18T07:40:04.957822+0000","last_fullsized":"2024-09-18T07:40:04.957822+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.959822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.959822+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:34.831011+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":571,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.947589+0000","last_change":"2024-09-18T07:37:43.153068+0000","last_active":"2024-09-18T07:40:04.947589+0000","last_peered":"2024-09-18T07:40:04.947589+0000","last_clean":"2024-09-18T07:40:04.947589+0000","last_became_active":"2024-09-18T07:37:43.096236+0000","last_became_peered":"2024-09-18T07:37:43.096236+0000","last_unstale":"2024-09-18T07:40:04.947589+0000","last_undegraded":"2024-09-18T07:40:04.947589+0000","last_fullsized":"2024-09-18T07:40:04.947589+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-18T07:35:00.882191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.882191+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:55.159776+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":513,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.964557+0000","last_change":"2024-09-18T07:39:28.750779+0000","last_active":"2024-09-18T07:40:04.964557+0000","last_peered":"2024-09-18T07:40:04.964557+0000","last_clean":"2024-09-18T07:40:04.964557+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:40:04.964557+0000","last_undegraded":"2024-09-18T07:40:04.964557+0000","last_fullsized":"2024-09-18T07:40:04.964557+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:39:28.750720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.750720+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:52.195932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":626,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.956833+0000","last_change":"2024-09-18T07:36:05.016008+0000","last_active":"2024-09-18T07:40:04.956833+0000","last_peered":"2024-09-18T07:40:04.956833+0000","last_clean":"2024-09-18T07:40:04.956833+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:40:04.956833+0000","last_undegraded":"2024-09-18T07:40:04.956833+0000","last_fullsized":"2024-09-18T07:40:04.956833+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:05.015955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.015955+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:48.836489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":302,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949986+0000","last_change":"2024-09-18T07:38:58.528134+0000","last_active":"2024-09-18T07:40:04.949986+0000","last_peered":"2024-09-18T07:40:04.949986+0000","last_clean":"2024-09-18T07:40:04.949986+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:40:04.949986+0000","last_undegraded":"2024-09-18T07:40:04.949986+0000","last_fullsized":"2024-09-18T07:40:04.949986+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:58.528059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:58.528059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:10.915828+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"303'3984","reported_seq":4023,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:06.806798+0000","last_change":"2024-09-18T07:39:29.702619+0000","last_active":"2024-09-18T07:40:06.806798+0000","last_peered":"2024-09-18T07:40:06.806798+0000","last_clean":"2024-09-18T07:40:06.806798+0000","last_became_active":"2024-09-18T07:39:29.702301+0000","last_became_peered":"2024-09-18T07:39:29.702301+0000","last_unstale":"2024-09-18T07:40:06.806798+0000","last_undegraded":"2024-09-18T07:40:06.806798+0000","last_fullsized":"2024-09-18T07:40:06.806798+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:33.630737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261095424,"num_objects":3984,"num_object_clones":0,"num_object_copies":11952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3984,"num_write_kb":254976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":244,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.952968+0000","last_change":"2024-09-18T07:39:06.342429+0000","last_active":"2024-09-18T07:40:04.952968+0000","last_peered":"2024-09-18T07:40:04.952968+0000","last_clean":"2024-09-18T07:40:04.952968+0000","last_became_active":"2024-09-18T07:39:04.941752+0000","last_became_peered":"2024-09-18T07:39:04.941752+0000","last_unstale":"2024-09-18T07:40:04.952968+0000","last_undegraded":"2024-09-18T07:40:04.952968+0000","last_fullsized":"2024-09-18T07:40:04.952968+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:06.342370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:06.342370+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:36.082977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":233,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963471+0000","last_change":"2024-09-18T07:36:13.119192+0000","last_active":"2024-09-18T07:40:04.963471+0000","last_peered":"2024-09-18T07:40:04.963471+0000","last_clean":"2024-09-18T07:40:04.963471+0000","last_became_active":"2024-09-18T07:36:13.118876+0000","last_became_peered":"2024-09-18T07:36:13.118876+0000","last_unstale":"2024-09-18T07:40:04.963471+0000","last_undegraded":"2024-09-18T07:40:04.963471+0000","last_fullsized":"2024-09-18T07:40:04.963471+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.613045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":522,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963910+0000","last_change":"2024-09-18T07:39:31.418964+0000","last_active":"2024-09-18T07:40:04.963910+0000","last_peered":"2024-09-18T07:40:04.963910+0000","last_clean":"2024-09-18T07:40:04.963910+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:40:04.963910+0000","last_undegraded":"2024-09-18T07:40:04.963910+0000","last_fullsized":"2024-09-18T07:40:04.963910+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:31.418910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:31.418910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:40.383317+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"240'10","reported_seq":645,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.962286+0000","last_change":"2024-09-18T07:37:43.170266+0000","last_active":"2024-09-18T07:40:04.962286+0000","last_peered":"2024-09-18T07:40:04.962286+0000","last_clean":"2024-09-18T07:40:04.962286+0000","last_became_active":"2024-09-18T07:37:43.167850+0000","last_became_peered":"2024-09-18T07:37:43.167850+0000","last_unstale":"2024-09-18T07:40:04.962286+0000","last_undegraded":"2024-09-18T07:40:04.962286+0000","last_fullsized":"2024-09-18T07:40:04.962286+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:05.525762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.525762+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:49.695027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":492,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.962671+0000","last_change":"2024-09-18T07:39:00.759483+0000","last_active":"2024-09-18T07:40:04.962671+0000","last_peered":"2024-09-18T07:40:04.962671+0000","last_clean":"2024-09-18T07:40:04.962671+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:40:04.962671+0000","last_undegraded":"2024-09-18T07:40:04.962671+0000","last_fullsized":"2024-09-18T07:40:04.962671+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:00.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:10.972661+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"303'3882","reported_seq":3921,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:06.520925+0000","last_change":"2024-09-18T07:39:29.703780+0000","last_active":"2024-09-18T07:40:06.520925+0000","last_peered":"2024-09-18T07:40:06.520925+0000","last_clean":"2024-09-18T07:40:06.520925+0000","last_became_active":"2024-09-18T07:39:29.703475+0000","last_became_peered":"2024-09-18T07:39:29.703475+0000","last_unstale":"2024-09-18T07:40:06.520925+0000","last_undegraded":"2024-09-18T07:40:06.520925+0000","last_fullsized":"2024-09-18T07:40:06.520925+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3882,"log_dups_size":0,"ondisk_log_size":3882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:58.380502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254410752,"num_objects":3882,"num_object_clones":0,"num_object_copies":11646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3882,"num_write_kb":248448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":244,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.952965+0000","last_change":"2024-09-18T07:39:07.357549+0000","last_active":"2024-09-18T07:40:04.952965+0000","last_peered":"2024-09-18T07:40:04.952965+0000","last_clean":"2024-09-18T07:40:04.952965+0000","last_became_active":"2024-09-18T07:39:04.948571+0000","last_became_peered":"2024-09-18T07:39:04.948571+0000","last_unstale":"2024-09-18T07:40:04.952965+0000","last_undegraded":"2024-09-18T07:40:04.952965+0000","last_fullsized":"2024-09-18T07:40:04.952965+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:07.357480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:07.357480+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:12.531366+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":602,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.962531+0000","last_change":"2024-09-18T07:40:02.764120+0000","last_active":"2024-09-18T07:40:04.962531+0000","last_peered":"2024-09-18T07:40:04.962531+0000","last_clean":"2024-09-18T07:40:04.962531+0000","last_became_active":"2024-09-18T07:35:01.415275+0000","last_became_peered":"2024-09-18T07:35:01.415275+0000","last_unstale":"2024-09-18T07:40:04.962531+0000","last_undegraded":"2024-09-18T07:40:04.962531+0000","last_fullsized":"2024-09-18T07:40:04.962531+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:40:02.764017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:02.764017+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:05.158836+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":206,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949769+0000","last_change":"2024-09-18T07:37:46.831436+0000","last_active":"2024-09-18T07:40:04.949769+0000","last_peered":"2024-09-18T07:40:04.949769+0000","last_clean":"2024-09-18T07:40:04.949769+0000","last_became_active":"2024-09-18T07:36:13.118553+0000","last_became_peered":"2024-09-18T07:36:13.118553+0000","last_unstale":"2024-09-18T07:40:04.949769+0000","last_undegraded":"2024-09-18T07:40:04.949769+0000","last_fullsized":"2024-09-18T07:40:04.949769+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:46.831373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:37:46.831373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:16.388148+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":139,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.952765+0000","last_change":"2024-09-18T07:37:43.070306+0000","last_active":"2024-09-18T07:40:04.952765+0000","last_peered":"2024-09-18T07:40:04.952765+0000","last_clean":"2024-09-18T07:40:04.952765+0000","last_became_active":"2024-09-18T07:37:43.070009+0000","last_became_peered":"2024-09-18T07:37:43.070009+0000","last_unstale":"2024-09-18T07:40:04.952765+0000","last_undegraded":"2024-09-18T07:40:04.952765+0000","last_fullsized":"2024-09-18T07:40:04.952765+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:21.834833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:33:21.834833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.202683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":139,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.952871+0000","last_change":"2024-09-18T07:37:43.052077+0000","last_active":"2024-09-18T07:40:04.952871+0000","last_peered":"2024-09-18T07:40:04.952871+0000","last_clean":"2024-09-18T07:40:04.952871+0000","last_became_active":"2024-09-18T07:37:43.051740+0000","last_became_peered":"2024-09-18T07:37:43.051740+0000","last_unstale":"2024-09-18T07:40:04.952871+0000","last_undegraded":"2024-09-18T07:40:04.952871+0000","last_fullsized":"2024-09-18T07:40:04.952871+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:09.269847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"303'4036","reported_seq":4075,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:07.420238+0000","last_change":"2024-09-18T07:39:29.702264+0000","last_active":"2024-09-18T07:40:07.420238+0000","last_peered":"2024-09-18T07:40:07.420238+0000","last_clean":"2024-09-18T07:40:07.420238+0000","last_became_active":"2024-09-18T07:39:29.701239+0000","last_became_peered":"2024-09-18T07:39:29.701239+0000","last_unstale":"2024-09-18T07:40:07.420238+0000","last_undegraded":"2024-09-18T07:40:07.420238+0000","last_fullsized":"2024-09-18T07:40:07.420238+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":4036,"log_dups_size":0,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:51.463801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264503296,"num_objects":4036,"num_object_clones":0,"num_object_copies":12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":258304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":76,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963005+0000","last_change":"2024-09-18T07:39:05.509522+0000","last_active":"2024-09-18T07:40:04.963005+0000","last_peered":"2024-09-18T07:40:04.963005+0000","last_clean":"2024-09-18T07:40:04.963005+0000","last_became_active":"2024-09-18T07:39:04.950564+0000","last_became_peered":"2024-09-18T07:39:04.950564+0000","last_unstale":"2024-09-18T07:40:04.963005+0000","last_undegraded":"2024-09-18T07:40:04.963005+0000","last_fullsized":"2024-09-18T07:40:04.963005+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.509453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.509453+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:28.588623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"211'29","reported_seq":323,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949722+0000","last_change":"2024-09-18T07:39:00.207829+0000","last_active":"2024-09-18T07:40:04.949722+0000","last_peered":"2024-09-18T07:40:04.949722+0000","last_clean":"2024-09-18T07:40:04.949722+0000","last_became_active":"2024-09-18T07:36:13.115278+0000","last_became_peered":"2024-09-18T07:36:13.115278+0000","last_unstale":"2024-09-18T07:40:04.949722+0000","last_undegraded":"2024-09-18T07:40:04.949722+0000","last_fullsized":"2024-09-18T07:40:04.949722+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'29","last_scrub_stamp":"2024-09-18T07:39:00.207506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.207506+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:08.496876+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":522,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.948395+0000","last_change":"2024-09-18T07:37:43.061414+0000","last_active":"2024-09-18T07:40:04.948395+0000","last_peered":"2024-09-18T07:40:04.948395+0000","last_clean":"2024-09-18T07:40:04.948395+0000","last_became_active":"2024-09-18T07:37:43.060930+0000","last_became_peered":"2024-09-18T07:37:43.060930+0000","last_unstale":"2024-09-18T07:40:04.948395+0000","last_undegraded":"2024-09-18T07:40:04.948395+0000","last_fullsized":"2024-09-18T07:40:04.948395+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:06.908713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.908713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:56.182156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":546,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963630+0000","last_change":"2024-09-18T07:37:43.155837+0000","last_active":"2024-09-18T07:40:04.963630+0000","last_peered":"2024-09-18T07:40:04.963630+0000","last_clean":"2024-09-18T07:40:04.963630+0000","last_became_active":"2024-09-18T07:37:43.076595+0000","last_became_peered":"2024-09-18T07:37:43.076595+0000","last_unstale":"2024-09-18T07:40:04.963630+0000","last_undegraded":"2024-09-18T07:40:04.963630+0000","last_fullsized":"2024-09-18T07:40:04.963630+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:36:01.529677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.529677+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:55.397407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":496,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.964535+0000","last_change":"2024-09-18T07:39:25.741714+0000","last_active":"2024-09-18T07:40:04.964535+0000","last_peered":"2024-09-18T07:40:04.964535+0000","last_clean":"2024-09-18T07:40:04.964535+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:40:04.964535+0000","last_undegraded":"2024-09-18T07:40:04.964535+0000","last_fullsized":"2024-09-18T07:40:04.964535+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:25.741643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.741643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:35.816396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":525,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:05.858885+0000","last_change":"2024-09-18T07:40:05.858885+0000","last_active":"2024-09-18T07:40:05.858885+0000","last_peered":"2024-09-18T07:40:05.858885+0000","last_clean":"2024-09-18T07:40:05.858885+0000","last_became_active":"2024-09-18T07:35:01.418809+0000","last_became_peered":"2024-09-18T07:35:01.418809+0000","last_unstale":"2024-09-18T07:40:05.858885+0000","last_undegraded":"2024-09-18T07:40:05.858885+0000","last_fullsized":"2024-09-18T07:40:05.858885+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:05.858804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:05.858804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:32.928019+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"303'4036","reported_seq":4075,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:06.790060+0000","last_change":"2024-09-18T07:39:29.702743+0000","last_active":"2024-09-18T07:40:06.790060+0000","last_peered":"2024-09-18T07:40:06.790060+0000","last_clean":"2024-09-18T07:40:06.790060+0000","last_became_active":"2024-09-18T07:39:29.702461+0000","last_became_peered":"2024-09-18T07:39:29.702461+0000","last_unstale":"2024-09-18T07:40:06.790060+0000","last_undegraded":"2024-09-18T07:40:06.790060+0000","last_fullsized":"2024-09-18T07:40:06.790060+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":4036,"log_dups_size":0,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:31.412469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264503296,"num_objects":4036,"num_object_clones":0,"num_object_copies":12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":258304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":344,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949563+0000","last_change":"2024-09-18T07:39:06.465696+0000","last_active":"2024-09-18T07:40:04.949563+0000","last_peered":"2024-09-18T07:40:04.949563+0000","last_clean":"2024-09-18T07:40:04.949563+0000","last_became_active":"2024-09-18T07:39:04.944921+0000","last_became_peered":"2024-09-18T07:39:04.944921+0000","last_unstale":"2024-09-18T07:40:04.949563+0000","last_undegraded":"2024-09-18T07:40:04.949563+0000","last_fullsized":"2024-09-18T07:40:04.949563+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:06.465620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:06.465620+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:21.933997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":253,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963749+0000","last_change":"2024-09-18T07:40:04.227278+0000","last_active":"2024-09-18T07:40:04.963749+0000","last_peered":"2024-09-18T07:40:04.963749+0000","last_clean":"2024-09-18T07:40:04.963749+0000","last_became_active":"2024-09-18T07:36:13.117924+0000","last_became_peered":"2024-09-18T07:36:13.117924+0000","last_unstale":"2024-09-18T07:40:04.963749+0000","last_undegraded":"2024-09-18T07:40:04.963749+0000","last_fullsized":"2024-09-18T07:40:04.963749+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-18T07:40:04.226789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:04.226789+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:15.873556+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":220,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.950609+0000","last_change":"2024-09-18T07:39:07.370140+0000","last_active":"2024-09-18T07:40:04.950609+0000","last_peered":"2024-09-18T07:40:04.950609+0000","last_clean":"2024-09-18T07:40:04.950609+0000","last_became_active":"2024-09-18T07:37:43.050971+0000","last_became_peered":"2024-09-18T07:37:43.050971+0000","last_unstale":"2024-09-18T07:40:04.950609+0000","last_undegraded":"2024-09-18T07:40:04.950609+0000","last_fullsized":"2024-09-18T07:40:04.950609+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:07.370081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:07.370081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:28.043475+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":525,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949558+0000","last_change":"2024-09-18T07:37:49.895264+0000","last_active":"2024-09-18T07:40:04.949558+0000","last_peered":"2024-09-18T07:40:04.949558+0000","last_clean":"2024-09-18T07:40:04.949558+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:40:04.949558+0000","last_undegraded":"2024-09-18T07:40:04.949558+0000","last_fullsized":"2024-09-18T07:40:04.949558+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:37:49.894707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:37:49.894707+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:14.409320+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":522,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.950156+0000","last_change":"2024-09-18T07:39:35.457812+0000","last_active":"2024-09-18T07:40:04.950156+0000","last_peered":"2024-09-18T07:40:04.950156+0000","last_clean":"2024-09-18T07:40:04.950156+0000","last_became_active":"2024-09-18T07:35:01.403680+0000","last_became_peered":"2024-09-18T07:35:01.403680+0000","last_unstale":"2024-09-18T07:40:04.950156+0000","last_undegraded":"2024-09-18T07:40:04.950156+0000","last_fullsized":"2024-09-18T07:40:04.950156+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.457734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.457734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:55.517500+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":298,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.950199+0000","last_change":"2024-09-18T07:37:43.118570+0000","last_active":"2024-09-18T07:40:04.950199+0000","last_peered":"2024-09-18T07:40:04.950199+0000","last_clean":"2024-09-18T07:40:04.950199+0000","last_became_active":"2024-09-18T07:37:43.060019+0000","last_became_peered":"2024-09-18T07:37:43.060019+0000","last_unstale":"2024-09-18T07:40:04.950199+0000","last_undegraded":"2024-09-18T07:40:04.950199+0000","last_fullsized":"2024-09-18T07:40:04.950199+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:33:57.655797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:57.655797+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:51.521311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"303'4049","reported_seq":4087,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:07.462812+0000","last_change":"2024-09-18T07:39:29.701833+0000","last_active":"2024-09-18T07:40:07.462812+0000","last_peered":"2024-09-18T07:40:07.462812+0000","last_clean":"2024-09-18T07:40:07.462812+0000","last_became_active":"2024-09-18T07:39:29.701554+0000","last_became_peered":"2024-09-18T07:39:29.701554+0000","last_unstale":"2024-09-18T07:40:07.462812+0000","last_undegraded":"2024-09-18T07:40:07.462812+0000","last_fullsized":"2024-09-18T07:40:07.462812+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":4049,"log_dups_size":0,"ondisk_log_size":4049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:40.719037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265355264,"num_objects":4049,"num_object_clones":0,"num_object_copies":12147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4049,"num_write_kb":259136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":244,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.952772+0000","last_change":"2024-09-18T07:39:05.333676+0000","last_active":"2024-09-18T07:40:04.952772+0000","last_peered":"2024-09-18T07:40:04.952772+0000","last_clean":"2024-09-18T07:40:04.952772+0000","last_became_active":"2024-09-18T07:39:04.938080+0000","last_became_peered":"2024-09-18T07:39:04.938080+0000","last_unstale":"2024-09-18T07:40:04.952772+0000","last_undegraded":"2024-09-18T07:40:04.952772+0000","last_fullsized":"2024-09-18T07:40:04.952772+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.333608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.333608+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:28.401993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":531,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.964463+0000","last_change":"2024-09-18T07:33:27.040227+0000","last_active":"2024-09-18T07:40:04.964463+0000","last_peered":"2024-09-18T07:40:04.964463+0000","last_clean":"2024-09-18T07:40:04.964463+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:40:04.964463+0000","last_undegraded":"2024-09-18T07:40:04.964463+0000","last_fullsized":"2024-09-18T07:40:04.964463+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:17.634433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":502,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.964166+0000","last_change":"2024-09-18T07:30:56.285184+0000","last_active":"2024-09-18T07:40:04.964166+0000","last_peered":"2024-09-18T07:40:04.964166+0000","last_clean":"2024-09-18T07:40:04.964166+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:40:04.964166+0000","last_undegraded":"2024-09-18T07:40:04.964166+0000","last_fullsized":"2024-09-18T07:40:04.964166+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:16.624959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":581,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949587+0000","last_change":"2024-09-18T07:35:01.421692+0000","last_active":"2024-09-18T07:40:04.949587+0000","last_peered":"2024-09-18T07:40:04.949587+0000","last_clean":"2024-09-18T07:40:04.949587+0000","last_became_active":"2024-09-18T07:35:01.400524+0000","last_became_peered":"2024-09-18T07:35:01.400524+0000","last_unstale":"2024-09-18T07:40:04.949587+0000","last_undegraded":"2024-09-18T07:40:04.949587+0000","last_fullsized":"2024-09-18T07:40:04.949587+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:30:54.270008+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:46.474589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":311,"num_read_kb":306,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":514,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:05.263108+0000","last_change":"2024-09-18T07:40:05.263108+0000","last_active":"2024-09-18T07:40:05.263108+0000","last_peered":"2024-09-18T07:40:05.263108+0000","last_clean":"2024-09-18T07:40:05.263108+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:40:05.263108+0000","last_undegraded":"2024-09-18T07:40:05.263108+0000","last_fullsized":"2024-09-18T07:40:05.263108+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:05.263040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:05.263040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:33.762504+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":206,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963067+0000","last_change":"2024-09-18T07:36:39.601424+0000","last_active":"2024-09-18T07:40:04.963067+0000","last_peered":"2024-09-18T07:40:04.963067+0000","last_clean":"2024-09-18T07:40:04.963067+0000","last_became_active":"2024-09-18T07:36:13.117463+0000","last_became_peered":"2024-09-18T07:36:13.117463+0000","last_unstale":"2024-09-18T07:40:04.963067+0000","last_undegraded":"2024-09-18T07:40:04.963067+0000","last_fullsized":"2024-09-18T07:40:04.963067+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:39.601356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:39.601356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"303'3961","reported_seq":4000,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:07.091382+0000","last_change":"2024-09-18T07:39:29.700116+0000","last_active":"2024-09-18T07:40:07.091382+0000","last_peered":"2024-09-18T07:40:07.091382+0000","last_clean":"2024-09-18T07:40:07.091382+0000","last_became_active":"2024-09-18T07:39:29.699733+0000","last_became_peered":"2024-09-18T07:39:29.699733+0000","last_unstale":"2024-09-18T07:40:07.091382+0000","last_undegraded":"2024-09-18T07:40:07.091382+0000","last_fullsized":"2024-09-18T07:40:07.091382+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3961,"log_dups_size":0,"ondisk_log_size":3961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:37.454373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259588096,"num_objects":3961,"num_object_clones":0,"num_object_copies":11883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3961,"num_write_kb":253504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'20","reported_seq":186,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.953343+0000","last_change":"2024-09-18T07:37:43.157718+0000","last_active":"2024-09-18T07:40:04.953343+0000","last_peered":"2024-09-18T07:40:04.953343+0000","last_clean":"2024-09-18T07:40:04.953343+0000","last_became_active":"2024-09-18T07:37:43.087496+0000","last_became_peered":"2024-09-18T07:37:43.087496+0000","last_unstale":"2024-09-18T07:40:04.953343+0000","last_undegraded":"2024-09-18T07:40:04.953343+0000","last_fullsized":"2024-09-18T07:40:04.953343+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'17","last_scrub_stamp":"2024-09-18T07:33:33.849285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:33:33.849285+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:15.202624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":521,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963138+0000","last_change":"2024-09-18T07:37:43.065584+0000","last_active":"2024-09-18T07:40:04.963138+0000","last_peered":"2024-09-18T07:40:04.963138+0000","last_clean":"2024-09-18T07:40:04.963138+0000","last_became_active":"2024-09-18T07:37:43.065279+0000","last_became_peered":"2024-09-18T07:37:43.065279+0000","last_unstale":"2024-09-18T07:40:04.963138+0000","last_undegraded":"2024-09-18T07:40:04.963138+0000","last_fullsized":"2024-09-18T07:40:04.963138+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:59.616988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:59.616988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:45.852796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":338,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949537+0000","last_change":"2024-09-18T07:39:03.518624+0000","last_active":"2024-09-18T07:40:04.949537+0000","last_peered":"2024-09-18T07:40:04.949537+0000","last_clean":"2024-09-18T07:40:04.949537+0000","last_became_active":"2024-09-18T07:39:03.350419+0000","last_became_peered":"2024-09-18T07:39:03.350419+0000","last_unstale":"2024-09-18T07:40:04.949537+0000","last_undegraded":"2024-09-18T07:40:04.949537+0000","last_fullsized":"2024-09-18T07:40:04.949537+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:03.518551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:03.518551+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:14.832014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":280,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.950228+0000","last_change":"2024-09-18T07:38:52.486107+0000","last_active":"2024-09-18T07:40:04.950228+0000","last_peered":"2024-09-18T07:40:04.950228+0000","last_clean":"2024-09-18T07:40:04.950228+0000","last_became_active":"2024-09-18T07:35:01.412217+0000","last_became_peered":"2024-09-18T07:35:01.412217+0000","last_unstale":"2024-09-18T07:40:04.950228+0000","last_undegraded":"2024-09-18T07:40:04.950228+0000","last_fullsized":"2024-09-18T07:40:04.950228+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:52.486047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:52.486047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.218642+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":494,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963617+0000","last_change":"2024-09-18T07:33:22.358792+0000","last_active":"2024-09-18T07:40:04.963617+0000","last_peered":"2024-09-18T07:40:04.963617+0000","last_clean":"2024-09-18T07:40:04.963617+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:40:04.963617+0000","last_undegraded":"2024-09-18T07:40:04.963617+0000","last_fullsized":"2024-09-18T07:40:04.963617+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.358746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.358746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:18.619255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"303'3919","reported_seq":3958,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:06.512727+0000","last_change":"2024-09-18T07:39:29.703721+0000","last_active":"2024-09-18T07:40:06.512727+0000","last_peered":"2024-09-18T07:40:06.512727+0000","last_clean":"2024-09-18T07:40:06.512727+0000","last_became_active":"2024-09-18T07:39:29.703451+0000","last_became_peered":"2024-09-18T07:39:29.703451+0000","last_unstale":"2024-09-18T07:40:06.512727+0000","last_undegraded":"2024-09-18T07:40:06.512727+0000","last_fullsized":"2024-09-18T07:40:06.512727+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3919,"log_dups_size":0,"ondisk_log_size":3919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:52.017204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256835584,"num_objects":3919,"num_object_clones":0,"num_object_copies":11757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3919,"num_write_kb":250816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":615,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.948261+0000","last_change":"2024-09-18T07:39:07.038527+0000","last_active":"2024-09-18T07:40:04.948261+0000","last_peered":"2024-09-18T07:40:04.948261+0000","last_clean":"2024-09-18T07:40:04.948261+0000","last_became_active":"2024-09-18T07:37:43.063255+0000","last_became_peered":"2024-09-18T07:37:43.063255+0000","last_unstale":"2024-09-18T07:40:04.948261+0000","last_undegraded":"2024-09-18T07:40:04.948261+0000","last_fullsized":"2024-09-18T07:40:04.948261+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:39:07.038450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:07.038450+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:24.342260+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":538,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.947919+0000","last_change":"2024-09-18T07:39:29.078656+0000","last_active":"2024-09-18T07:40:04.947919+0000","last_peered":"2024-09-18T07:40:04.947919+0000","last_clean":"2024-09-18T07:40:04.947919+0000","last_became_active":"2024-09-18T07:35:01.401063+0000","last_became_peered":"2024-09-18T07:35:01.401063+0000","last_unstale":"2024-09-18T07:40:04.947919+0000","last_undegraded":"2024-09-18T07:40:04.947919+0000","last_fullsized":"2024-09-18T07:40:04.947919+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.078600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.078600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:32.657735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":528,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963503+0000","last_change":"2024-09-18T07:39:35.406107+0000","last_active":"2024-09-18T07:40:04.963503+0000","last_peered":"2024-09-18T07:40:04.963503+0000","last_clean":"2024-09-18T07:40:04.963503+0000","last_became_active":"2024-09-18T07:37:43.075535+0000","last_became_peered":"2024-09-18T07:37:43.075535+0000","last_unstale":"2024-09-18T07:40:04.963503+0000","last_undegraded":"2024-09-18T07:40:04.963503+0000","last_fullsized":"2024-09-18T07:40:04.963503+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.406041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.406041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:44.739547+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":517,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.957956+0000","last_change":"2024-09-18T07:37:43.066497+0000","last_active":"2024-09-18T07:40:04.957956+0000","last_peered":"2024-09-18T07:40:04.957956+0000","last_clean":"2024-09-18T07:40:04.957956+0000","last_became_active":"2024-09-18T07:37:43.066153+0000","last_became_peered":"2024-09-18T07:37:43.066153+0000","last_unstale":"2024-09-18T07:40:04.957956+0000","last_undegraded":"2024-09-18T07:40:04.957956+0000","last_fullsized":"2024-09-18T07:40:04.957956+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:38.955353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:38.955353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:08.997218+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"303'3973","reported_seq":4012,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:06.625119+0000","last_change":"2024-09-18T07:39:29.700091+0000","last_active":"2024-09-18T07:40:06.625119+0000","last_peered":"2024-09-18T07:40:06.625119+0000","last_clean":"2024-09-18T07:40:06.625119+0000","last_became_active":"2024-09-18T07:39:29.699800+0000","last_became_peered":"2024-09-18T07:39:29.699800+0000","last_unstale":"2024-09-18T07:40:06.625119+0000","last_undegraded":"2024-09-18T07:40:06.625119+0000","last_fullsized":"2024-09-18T07:40:06.625119+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3973,"log_dups_size":0,"ondisk_log_size":3973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:42.451559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260374528,"num_objects":3973,"num_object_clones":0,"num_object_copies":11919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3973,"num_write_kb":254272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":86,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.962292+0000","last_change":"2024-09-18T07:39:38.447787+0000","last_active":"2024-09-18T07:40:04.962292+0000","last_peered":"2024-09-18T07:40:04.962292+0000","last_clean":"2024-09-18T07:40:04.962292+0000","last_became_active":"2024-09-18T07:39:04.950292+0000","last_became_peered":"2024-09-18T07:39:04.950292+0000","last_unstale":"2024-09-18T07:40:04.962292+0000","last_undegraded":"2024-09-18T07:40:04.962292+0000","last_fullsized":"2024-09-18T07:40:04.962292+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:38.447735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:38.447735+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:53.329499+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":556,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.947983+0000","last_change":"2024-09-18T07:40:04.281816+0000","last_active":"2024-09-18T07:40:04.947983+0000","last_peered":"2024-09-18T07:40:04.947983+0000","last_clean":"2024-09-18T07:40:04.947983+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:40:04.947983+0000","last_undegraded":"2024-09-18T07:40:04.947983+0000","last_fullsized":"2024-09-18T07:40:04.947983+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:40:04.281758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:04.281758+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:29.914573+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":147,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.952877+0000","last_change":"2024-09-18T07:39:35.015984+0000","last_active":"2024-09-18T07:40:04.952877+0000","last_peered":"2024-09-18T07:40:04.952877+0000","last_clean":"2024-09-18T07:40:04.952877+0000","last_became_active":"2024-09-18T07:37:43.061808+0000","last_became_peered":"2024-09-18T07:37:43.061808+0000","last_unstale":"2024-09-18T07:40:04.952877+0000","last_undegraded":"2024-09-18T07:40:04.952877+0000","last_fullsized":"2024-09-18T07:40:04.952877+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.015923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.015923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:42.634359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":198,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.947552+0000","last_change":"2024-09-18T07:36:13.116901+0000","last_active":"2024-09-18T07:40:04.947552+0000","last_peered":"2024-09-18T07:40:04.947552+0000","last_clean":"2024-09-18T07:40:04.947552+0000","last_became_active":"2024-09-18T07:36:13.116527+0000","last_became_peered":"2024-09-18T07:36:13.116527+0000","last_unstale":"2024-09-18T07:40:04.947552+0000","last_undegraded":"2024-09-18T07:40:04.947552+0000","last_fullsized":"2024-09-18T07:40:04.947552+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:32:05.239160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.239160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:50.048595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":512,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949764+0000","last_change":"2024-09-18T07:36:38.632189+0000","last_active":"2024-09-18T07:40:04.949764+0000","last_peered":"2024-09-18T07:40:04.949764+0000","last_clean":"2024-09-18T07:40:04.949764+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:40:04.949764+0000","last_undegraded":"2024-09-18T07:40:04.949764+0000","last_fullsized":"2024-09-18T07:40:04.949764+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:38.632040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:38.632040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:09.340014+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"303'4037","reported_seq":4076,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:07.076226+0000","last_change":"2024-09-18T07:39:29.701529+0000","last_active":"2024-09-18T07:40:07.076226+0000","last_peered":"2024-09-18T07:40:07.076226+0000","last_clean":"2024-09-18T07:40:07.076226+0000","last_became_active":"2024-09-18T07:39:29.701147+0000","last_became_peered":"2024-09-18T07:39:29.701147+0000","last_unstale":"2024-09-18T07:40:07.076226+0000","last_undegraded":"2024-09-18T07:40:07.076226+0000","last_fullsized":"2024-09-18T07:40:07.076226+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":4037,"log_dups_size":0,"ondisk_log_size":4037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:51.353620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264568832,"num_objects":4037,"num_object_clones":0,"num_object_copies":12111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":258368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":344,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949544+0000","last_change":"2024-09-18T07:39:05.498235+0000","last_active":"2024-09-18T07:40:04.949544+0000","last_peered":"2024-09-18T07:40:04.949544+0000","last_clean":"2024-09-18T07:40:04.949544+0000","last_became_active":"2024-09-18T07:39:04.944753+0000","last_became_peered":"2024-09-18T07:39:04.944753+0000","last_unstale":"2024-09-18T07:40:04.949544+0000","last_undegraded":"2024-09-18T07:40:04.949544+0000","last_fullsized":"2024-09-18T07:40:04.949544+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.498159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.498159+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:05.777887+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":557,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.947734+0000","last_change":"2024-09-18T07:37:43.106862+0000","last_active":"2024-09-18T07:40:04.947734+0000","last_peered":"2024-09-18T07:40:04.947734+0000","last_clean":"2024-09-18T07:40:04.947734+0000","last_became_active":"2024-09-18T07:37:43.062724+0000","last_became_peered":"2024-09-18T07:37:43.062724+0000","last_unstale":"2024-09-18T07:40:04.947734+0000","last_undegraded":"2024-09-18T07:40:04.947734+0000","last_fullsized":"2024-09-18T07:40:04.947734+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:32:05.000310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:32:05.000310+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:45.943194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":515,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.971916+0000","last_change":"2024-09-18T07:39:38.871492+0000","last_active":"2024-09-18T07:40:04.971916+0000","last_peered":"2024-09-18T07:40:04.971916+0000","last_clean":"2024-09-18T07:40:04.971916+0000","last_became_active":"2024-09-18T07:30:56.287003+0000","last_became_peered":"2024-09-18T07:30:56.287003+0000","last_unstale":"2024-09-18T07:40:04.971916+0000","last_undegraded":"2024-09-18T07:40:04.971916+0000","last_fullsized":"2024-09-18T07:40:04.971916+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:38.871426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:38.871426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:04.468858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"173'11","reported_seq":660,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949931+0000","last_change":"2024-09-18T07:38:55.227421+0000","last_active":"2024-09-18T07:40:04.949931+0000","last_peered":"2024-09-18T07:40:04.949931+0000","last_clean":"2024-09-18T07:40:04.949931+0000","last_became_active":"2024-09-18T07:35:01.403765+0000","last_became_peered":"2024-09-18T07:35:01.403765+0000","last_unstale":"2024-09-18T07:40:04.949931+0000","last_undegraded":"2024-09-18T07:40:04.949931+0000","last_fullsized":"2024-09-18T07:40:04.949931+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'11","last_scrub_stamp":"2024-09-18T07:38:55.227366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:55.227366+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:17.922368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":504,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.947781+0000","last_change":"2024-09-18T07:36:33.251164+0000","last_active":"2024-09-18T07:40:04.947781+0000","last_peered":"2024-09-18T07:40:04.947781+0000","last_clean":"2024-09-18T07:40:04.947781+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:40:04.947781+0000","last_undegraded":"2024-09-18T07:40:04.947781+0000","last_fullsized":"2024-09-18T07:40:04.947781+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:33.251093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:33.251093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:01.242543+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"303'3965","reported_seq":4004,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:06.813459+0000","last_change":"2024-09-18T07:39:29.704580+0000","last_active":"2024-09-18T07:40:06.813459+0000","last_peered":"2024-09-18T07:40:06.813459+0000","last_clean":"2024-09-18T07:40:06.813459+0000","last_became_active":"2024-09-18T07:39:29.704318+0000","last_became_peered":"2024-09-18T07:39:29.704318+0000","last_unstale":"2024-09-18T07:40:06.813459+0000","last_undegraded":"2024-09-18T07:40:06.813459+0000","last_fullsized":"2024-09-18T07:40:06.813459+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3965,"log_dups_size":0,"ondisk_log_size":3965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:40.830318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259850240,"num_objects":3965,"num_object_clones":0,"num_object_copies":11895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3965,"num_write_kb":253760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":533,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.962438+0000","last_change":"2024-09-18T07:39:02.716084+0000","last_active":"2024-09-18T07:40:04.962438+0000","last_peered":"2024-09-18T07:40:04.962438+0000","last_clean":"2024-09-18T07:40:04.962438+0000","last_became_active":"2024-09-18T07:37:27.994144+0000","last_became_peered":"2024-09-18T07:37:27.994144+0000","last_unstale":"2024-09-18T07:40:04.962438+0000","last_undegraded":"2024-09-18T07:40:04.962438+0000","last_fullsized":"2024-09-18T07:40:04.962438+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:39:02.716009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:02.716009+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":528,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963158+0000","last_change":"2024-09-18T07:37:43.076416+0000","last_active":"2024-09-18T07:40:04.963158+0000","last_peered":"2024-09-18T07:40:04.963158+0000","last_clean":"2024-09-18T07:40:04.963158+0000","last_became_active":"2024-09-18T07:37:43.075988+0000","last_became_peered":"2024-09-18T07:37:43.075988+0000","last_unstale":"2024-09-18T07:40:04.963158+0000","last_undegraded":"2024-09-18T07:40:04.963158+0000","last_fullsized":"2024-09-18T07:40:04.963158+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:21.222526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:21.222526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:44.443625+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":508,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.950223+0000","last_change":"2024-09-18T07:39:01.216439+0000","last_active":"2024-09-18T07:40:04.950223+0000","last_peered":"2024-09-18T07:40:04.950223+0000","last_clean":"2024-09-18T07:40:04.950223+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:40:04.950223+0000","last_undegraded":"2024-09-18T07:40:04.950223+0000","last_fullsized":"2024-09-18T07:40:04.950223+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:01.216305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:01.216305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:23.547767+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":514,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.964076+0000","last_change":"2024-09-18T07:36:07.526609+0000","last_active":"2024-09-18T07:40:04.964076+0000","last_peered":"2024-09-18T07:40:04.964076+0000","last_clean":"2024-09-18T07:40:04.964076+0000","last_became_active":"2024-09-18T07:35:01.410890+0000","last_became_peered":"2024-09-18T07:35:01.410890+0000","last_unstale":"2024-09-18T07:40:04.964076+0000","last_undegraded":"2024-09-18T07:40:04.964076+0000","last_fullsized":"2024-09-18T07:40:04.964076+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:07.526536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:07.526536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:41.630367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"303'3884","reported_seq":3923,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:06.809266+0000","last_change":"2024-09-18T07:39:29.701848+0000","last_active":"2024-09-18T07:40:06.809266+0000","last_peered":"2024-09-18T07:40:06.809266+0000","last_clean":"2024-09-18T07:40:06.809266+0000","last_became_active":"2024-09-18T07:39:29.701593+0000","last_became_peered":"2024-09-18T07:39:29.701593+0000","last_unstale":"2024-09-18T07:40:06.809266+0000","last_undegraded":"2024-09-18T07:40:06.809266+0000","last_fullsized":"2024-09-18T07:40:06.809266+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3884,"log_dups_size":0,"ondisk_log_size":3884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:45.431316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254541824,"num_objects":3884,"num_object_clones":0,"num_object_copies":11652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3884,"num_write_kb":248576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":167,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.952938+0000","last_change":"2024-09-18T07:39:25.932376+0000","last_active":"2024-09-18T07:40:04.952938+0000","last_peered":"2024-09-18T07:40:04.952938+0000","last_clean":"2024-09-18T07:40:04.952938+0000","last_became_active":"2024-09-18T07:37:43.063319+0000","last_became_peered":"2024-09-18T07:37:43.063319+0000","last_unstale":"2024-09-18T07:40:04.952938+0000","last_undegraded":"2024-09-18T07:40:04.952938+0000","last_fullsized":"2024-09-18T07:40:04.952938+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:39:25.932315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.932315+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:32.693891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":302,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.950420+0000","last_change":"2024-09-18T07:39:25.301868+0000","last_active":"2024-09-18T07:40:04.950420+0000","last_peered":"2024-09-18T07:40:04.950420+0000","last_clean":"2024-09-18T07:40:04.950420+0000","last_became_active":"2024-09-18T07:37:43.072894+0000","last_became_peered":"2024-09-18T07:37:43.072894+0000","last_unstale":"2024-09-18T07:40:04.950420+0000","last_undegraded":"2024-09-18T07:40:04.950420+0000","last_fullsized":"2024-09-18T07:40:04.950420+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:25.301825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.301825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.461283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":523,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949514+0000","last_change":"2024-09-18T07:39:38.400864+0000","last_active":"2024-09-18T07:40:04.949514+0000","last_peered":"2024-09-18T07:40:04.949514+0000","last_clean":"2024-09-18T07:40:04.949514+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:40:04.949514+0000","last_undegraded":"2024-09-18T07:40:04.949514+0000","last_fullsized":"2024-09-18T07:40:04.949514+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:38.400785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:38.400785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:55.378212+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":272,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949634+0000","last_change":"2024-09-18T07:35:01.408078+0000","last_active":"2024-09-18T07:40:04.949634+0000","last_peered":"2024-09-18T07:40:04.949634+0000","last_clean":"2024-09-18T07:40:04.949634+0000","last_became_active":"2024-09-18T07:35:01.406980+0000","last_became_peered":"2024-09-18T07:35:01.406980+0000","last_unstale":"2024-09-18T07:40:04.949634+0000","last_undegraded":"2024-09-18T07:40:04.949634+0000","last_fullsized":"2024-09-18T07:40:04.949634+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:40.390575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"303'4057","reported_seq":4096,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:06.483018+0000","last_change":"2024-09-18T07:39:29.700522+0000","last_active":"2024-09-18T07:40:06.483018+0000","last_peered":"2024-09-18T07:40:06.483018+0000","last_clean":"2024-09-18T07:40:06.483018+0000","last_became_active":"2024-09-18T07:39:29.700244+0000","last_became_peered":"2024-09-18T07:39:29.700244+0000","last_unstale":"2024-09-18T07:40:06.483018+0000","last_undegraded":"2024-09-18T07:40:06.483018+0000","last_fullsized":"2024-09-18T07:40:06.483018+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":4057,"log_dups_size":0,"ondisk_log_size":4057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:39.850144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265879552,"num_objects":4057,"num_object_clones":0,"num_object_copies":12171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4057,"num_write_kb":259648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":543,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.947781+0000","last_change":"2024-09-18T07:38:59.101584+0000","last_active":"2024-09-18T07:40:04.947781+0000","last_peered":"2024-09-18T07:40:04.947781+0000","last_clean":"2024-09-18T07:40:04.947781+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:40:04.947781+0000","last_undegraded":"2024-09-18T07:40:04.947781+0000","last_fullsized":"2024-09-18T07:40:04.947781+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:38:59.101512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:38:59.101512+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:07.212990+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":272,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.950461+0000","last_change":"2024-09-18T07:35:01.400050+0000","last_active":"2024-09-18T07:40:04.950461+0000","last_peered":"2024-09-18T07:40:04.950461+0000","last_clean":"2024-09-18T07:40:04.950461+0000","last_became_active":"2024-09-18T07:35:01.398499+0000","last_became_peered":"2024-09-18T07:35:01.398499+0000","last_unstale":"2024-09-18T07:40:04.950461+0000","last_undegraded":"2024-09-18T07:40:04.950461+0000","last_fullsized":"2024-09-18T07:40:04.950461+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:43.441418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":524,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.950527+0000","last_change":"2024-09-18T07:36:32.564149+0000","last_active":"2024-09-18T07:40:04.950527+0000","last_peered":"2024-09-18T07:40:04.950527+0000","last_clean":"2024-09-18T07:40:04.950527+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:40:04.950527+0000","last_undegraded":"2024-09-18T07:40:04.950527+0000","last_fullsized":"2024-09-18T07:40:04.950527+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:32.564101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:32.564101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:08.345410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":517,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.971608+0000","last_change":"2024-09-18T07:40:02.932758+0000","last_active":"2024-09-18T07:40:04.971608+0000","last_peered":"2024-09-18T07:40:04.971608+0000","last_clean":"2024-09-18T07:40:04.971608+0000","last_became_active":"2024-09-18T07:35:01.418735+0000","last_became_peered":"2024-09-18T07:35:01.418735+0000","last_unstale":"2024-09-18T07:40:04.971608+0000","last_undegraded":"2024-09-18T07:40:04.971608+0000","last_fullsized":"2024-09-18T07:40:04.971608+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:02.932690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:02.932690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:27.869866+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"303'3914","reported_seq":3953,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:07.413494+0000","last_change":"2024-09-18T07:39:29.701895+0000","last_active":"2024-09-18T07:40:07.413494+0000","last_peered":"2024-09-18T07:40:07.413494+0000","last_clean":"2024-09-18T07:40:07.413494+0000","last_became_active":"2024-09-18T07:39:29.701617+0000","last_became_peered":"2024-09-18T07:39:29.701617+0000","last_unstale":"2024-09-18T07:40:07.413494+0000","last_undegraded":"2024-09-18T07:40:07.413494+0000","last_fullsized":"2024-09-18T07:40:07.413494+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:42.973327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256507904,"num_objects":3914,"num_object_clones":0,"num_object_copies":11742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3914,"num_write_kb":250496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":635,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.962444+0000","last_change":"2024-09-18T07:39:27.484003+0000","last_active":"2024-09-18T07:40:04.962444+0000","last_peered":"2024-09-18T07:40:04.962444+0000","last_clean":"2024-09-18T07:40:04.962444+0000","last_became_active":"2024-09-18T07:37:43.075552+0000","last_became_peered":"2024-09-18T07:37:43.075552+0000","last_unstale":"2024-09-18T07:40:04.962444+0000","last_undegraded":"2024-09-18T07:40:04.962444+0000","last_fullsized":"2024-09-18T07:40:04.962444+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:39:27.483930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:27.483930+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.149026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":294,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.950563+0000","last_change":"2024-09-18T07:36:09.261574+0000","last_active":"2024-09-18T07:40:04.950563+0000","last_peered":"2024-09-18T07:40:04.950563+0000","last_clean":"2024-09-18T07:40:04.950563+0000","last_became_active":"2024-09-18T07:35:01.406695+0000","last_became_peered":"2024-09-18T07:35:01.406695+0000","last_unstale":"2024-09-18T07:40:04.950563+0000","last_undegraded":"2024-09-18T07:40:04.950563+0000","last_fullsized":"2024-09-18T07:40:04.950563+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:36:09.261516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.261516+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:22.292832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"240'12","reported_seq":376,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949940+0000","last_change":"2024-09-18T07:38:55.490250+0000","last_active":"2024-09-18T07:40:04.949940+0000","last_peered":"2024-09-18T07:40:04.949940+0000","last_clean":"2024-09-18T07:40:04.949940+0000","last_became_active":"2024-09-18T07:37:43.056714+0000","last_became_peered":"2024-09-18T07:37:43.056714+0000","last_unstale":"2024-09-18T07:40:04.949940+0000","last_undegraded":"2024-09-18T07:40:04.949940+0000","last_fullsized":"2024-09-18T07:40:04.949940+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:38:55.490178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:55.490178+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:07.849179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":500,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.962457+0000","last_change":"2024-09-18T07:37:43.078768+0000","last_active":"2024-09-18T07:40:04.962457+0000","last_peered":"2024-09-18T07:40:04.962457+0000","last_clean":"2024-09-18T07:40:04.962457+0000","last_became_active":"2024-09-18T07:37:43.078121+0000","last_became_peered":"2024-09-18T07:37:43.078121+0000","last_unstale":"2024-09-18T07:40:04.962457+0000","last_undegraded":"2024-09-18T07:40:04.962457+0000","last_fullsized":"2024-09-18T07:40:04.962457+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:31:01.319795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:12.586962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":147,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.953172+0000","last_change":"2024-09-18T07:39:37.072844+0000","last_active":"2024-09-18T07:40:04.953172+0000","last_peered":"2024-09-18T07:40:04.953172+0000","last_clean":"2024-09-18T07:40:04.953172+0000","last_became_active":"2024-09-18T07:37:43.087994+0000","last_became_peered":"2024-09-18T07:37:43.087994+0000","last_unstale":"2024-09-18T07:40:04.953172+0000","last_undegraded":"2024-09-18T07:40:04.953172+0000","last_fullsized":"2024-09-18T07:40:04.953172+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.072782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.072782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:52.412066+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":541,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963571+0000","last_change":"2024-09-18T07:36:09.061983+0000","last_active":"2024-09-18T07:40:04.963571+0000","last_peered":"2024-09-18T07:40:04.963571+0000","last_clean":"2024-09-18T07:40:04.963571+0000","last_became_active":"2024-09-18T07:35:01.407787+0000","last_became_peered":"2024-09-18T07:35:01.407787+0000","last_unstale":"2024-09-18T07:40:04.963571+0000","last_undegraded":"2024-09-18T07:40:04.963571+0000","last_fullsized":"2024-09-18T07:40:04.963571+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-18T07:36:09.061903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.061903+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:16.050531+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":513,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:06.268555+0000","last_change":"2024-09-18T07:40:06.268555+0000","last_active":"2024-09-18T07:40:06.268555+0000","last_peered":"2024-09-18T07:40:06.268555+0000","last_clean":"2024-09-18T07:40:06.268555+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:40:06.268555+0000","last_undegraded":"2024-09-18T07:40:06.268555+0000","last_fullsized":"2024-09-18T07:40:06.268555+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:06.268501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:06.268501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:08.101614+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":286,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.950316+0000","last_change":"2024-09-18T07:37:43.079723+0000","last_active":"2024-09-18T07:40:04.950316+0000","last_peered":"2024-09-18T07:40:04.950316+0000","last_clean":"2024-09-18T07:40:04.950316+0000","last_became_active":"2024-09-18T07:37:43.079313+0000","last_became_peered":"2024-09-18T07:37:43.079313+0000","last_unstale":"2024-09-18T07:40:04.950316+0000","last_undegraded":"2024-09-18T07:40:04.950316+0000","last_fullsized":"2024-09-18T07:40:04.950316+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:58.688631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:58.688631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:50.554490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":541,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.964394+0000","last_change":"2024-09-18T07:35:01.448510+0000","last_active":"2024-09-18T07:40:04.964394+0000","last_peered":"2024-09-18T07:40:04.964394+0000","last_clean":"2024-09-18T07:40:04.964394+0000","last_became_active":"2024-09-18T07:35:01.414458+0000","last_became_peered":"2024-09-18T07:35:01.414458+0000","last_unstale":"2024-09-18T07:40:04.964394+0000","last_undegraded":"2024-09-18T07:40:04.964394+0000","last_fullsized":"2024-09-18T07:40:04.964394+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:19.600284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":294,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949881+0000","last_change":"2024-09-18T07:37:43.057776+0000","last_active":"2024-09-18T07:40:04.949881+0000","last_peered":"2024-09-18T07:40:04.949881+0000","last_clean":"2024-09-18T07:40:04.949881+0000","last_became_active":"2024-09-18T07:37:43.056675+0000","last_became_peered":"2024-09-18T07:37:43.056675+0000","last_unstale":"2024-09-18T07:40:04.949881+0000","last_undegraded":"2024-09-18T07:40:04.949881+0000","last_fullsized":"2024-09-18T07:40:04.949881+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:05.201723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.201723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:15.371131+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":528,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963317+0000","last_change":"2024-09-18T07:39:30.966887+0000","last_active":"2024-09-18T07:40:04.963317+0000","last_peered":"2024-09-18T07:40:04.963317+0000","last_clean":"2024-09-18T07:40:04.963317+0000","last_became_active":"2024-09-18T07:35:01.420133+0000","last_became_peered":"2024-09-18T07:35:01.420133+0000","last_unstale":"2024-09-18T07:40:04.963317+0000","last_undegraded":"2024-09-18T07:40:04.963317+0000","last_fullsized":"2024-09-18T07:40:04.963317+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:30.966840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:30.966840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:33.098878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":509,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949397+0000","last_change":"2024-09-18T07:37:48.861965+0000","last_active":"2024-09-18T07:40:04.949397+0000","last_peered":"2024-09-18T07:40:04.949397+0000","last_clean":"2024-09-18T07:40:04.949397+0000","last_became_active":"2024-09-18T07:35:01.396305+0000","last_became_peered":"2024-09-18T07:35:01.396305+0000","last_unstale":"2024-09-18T07:40:04.949397+0000","last_undegraded":"2024-09-18T07:40:04.949397+0000","last_fullsized":"2024-09-18T07:40:04.949397+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:48.861784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:37:48.861784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:23.433931+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":559,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963688+0000","last_change":"2024-09-18T07:39:25.515172+0000","last_active":"2024-09-18T07:40:04.963688+0000","last_peered":"2024-09-18T07:40:04.963688+0000","last_clean":"2024-09-18T07:40:04.963688+0000","last_became_active":"2024-09-18T07:35:01.416744+0000","last_became_peered":"2024-09-18T07:35:01.416744+0000","last_unstale":"2024-09-18T07:40:04.963688+0000","last_undegraded":"2024-09-18T07:40:04.963688+0000","last_fullsized":"2024-09-18T07:40:04.963688+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:39:25.515085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.515085+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:40.279265+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":525,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.961875+0000","last_change":"2024-09-18T07:39:32.766405+0000","last_active":"2024-09-18T07:40:04.961875+0000","last_peered":"2024-09-18T07:40:04.961875+0000","last_clean":"2024-09-18T07:40:04.961875+0000","last_became_active":"2024-09-18T07:35:01.414024+0000","last_became_peered":"2024-09-18T07:35:01.414024+0000","last_unstale":"2024-09-18T07:40:04.961875+0000","last_undegraded":"2024-09-18T07:40:04.961875+0000","last_fullsized":"2024-09-18T07:40:04.961875+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:32.766366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:32.766366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:36.625283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"173'10","reported_seq":647,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.958685+0000","last_change":"2024-09-18T07:39:32.947933+0000","last_active":"2024-09-18T07:40:04.958685+0000","last_peered":"2024-09-18T07:40:04.958685+0000","last_clean":"2024-09-18T07:40:04.958685+0000","last_became_active":"2024-09-18T07:35:01.407568+0000","last_became_peered":"2024-09-18T07:35:01.407568+0000","last_unstale":"2024-09-18T07:40:04.958685+0000","last_undegraded":"2024-09-18T07:40:04.958685+0000","last_fullsized":"2024-09-18T07:40:04.958685+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:39:32.947887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:32.947887+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:58.809615+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":495,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.947896+0000","last_change":"2024-09-18T07:36:11.910543+0000","last_active":"2024-09-18T07:40:04.947896+0000","last_peered":"2024-09-18T07:40:04.947896+0000","last_clean":"2024-09-18T07:40:04.947896+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:40:04.947896+0000","last_undegraded":"2024-09-18T07:40:04.947896+0000","last_fullsized":"2024-09-18T07:40:04.947896+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:11.910474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:11.910474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.028676+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":563,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.958940+0000","last_change":"2024-09-18T07:36:02.975929+0000","last_active":"2024-09-18T07:40:04.958940+0000","last_peered":"2024-09-18T07:40:04.958940+0000","last_clean":"2024-09-18T07:40:04.958940+0000","last_became_active":"2024-09-18T07:35:01.407738+0000","last_became_peered":"2024-09-18T07:35:01.407738+0000","last_unstale":"2024-09-18T07:40:04.958940+0000","last_undegraded":"2024-09-18T07:40:04.958940+0000","last_fullsized":"2024-09-18T07:40:04.958940+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:36:02.975863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:02.975863+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:49.805199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":280,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.950076+0000","last_change":"2024-09-18T07:36:08.222898+0000","last_active":"2024-09-18T07:40:04.950076+0000","last_peered":"2024-09-18T07:40:04.950076+0000","last_clean":"2024-09-18T07:40:04.950076+0000","last_became_active":"2024-09-18T07:35:01.400197+0000","last_became_peered":"2024-09-18T07:35:01.400197+0000","last_unstale":"2024-09-18T07:40:04.950076+0000","last_undegraded":"2024-09-18T07:40:04.950076+0000","last_fullsized":"2024-09-18T07:40:04.950076+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:08.222838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:08.222838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:16.370989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":619,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963493+0000","last_change":"2024-09-18T07:37:42.285094+0000","last_active":"2024-09-18T07:40:04.963493+0000","last_peered":"2024-09-18T07:40:04.963493+0000","last_clean":"2024-09-18T07:40:04.963493+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:40:04.963493+0000","last_undegraded":"2024-09-18T07:40:04.963493+0000","last_fullsized":"2024-09-18T07:40:04.963493+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:37:42.285042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:37:42.285042+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":513,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.964182+0000","last_change":"2024-09-18T07:39:31.784401+0000","last_active":"2024-09-18T07:40:04.964182+0000","last_peered":"2024-09-18T07:40:04.964182+0000","last_clean":"2024-09-18T07:40:04.964182+0000","last_became_active":"2024-09-18T07:35:01.410561+0000","last_became_peered":"2024-09-18T07:35:01.410561+0000","last_unstale":"2024-09-18T07:40:04.964182+0000","last_undegraded":"2024-09-18T07:40:04.964182+0000","last_fullsized":"2024-09-18T07:40:04.964182+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:31.784355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:31.784355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:37.247787+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":592,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.947682+0000","last_change":"2024-09-18T07:38:41.163393+0000","last_active":"2024-09-18T07:40:04.947682+0000","last_peered":"2024-09-18T07:40:04.947682+0000","last_clean":"2024-09-18T07:40:04.947682+0000","last_became_active":"2024-09-18T07:38:41.143108+0000","last_became_peered":"2024-09-18T07:38:41.143108+0000","last_unstale":"2024-09-18T07:40:04.947682+0000","last_undegraded":"2024-09-18T07:40:04.947682+0000","last_fullsized":"2024-09-18T07:40:04.947682+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:36:01.986964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.986964+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:14.037941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":527,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.947912+0000","last_change":"2024-09-18T07:37:43.062429+0000","last_active":"2024-09-18T07:40:04.947912+0000","last_peered":"2024-09-18T07:40:04.947912+0000","last_clean":"2024-09-18T07:40:04.947912+0000","last_became_active":"2024-09-18T07:37:43.061577+0000","last_became_peered":"2024-09-18T07:37:43.061577+0000","last_unstale":"2024-09-18T07:40:04.947912+0000","last_undegraded":"2024-09-18T07:40:04.947912+0000","last_fullsized":"2024-09-18T07:40:04.947912+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:30:55.271698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:39.923066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":198,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963378+0000","last_change":"2024-09-18T07:36:13.118114+0000","last_active":"2024-09-18T07:40:04.963378+0000","last_peered":"2024-09-18T07:40:04.963378+0000","last_clean":"2024-09-18T07:40:04.963378+0000","last_became_active":"2024-09-18T07:36:13.117584+0000","last_became_peered":"2024-09-18T07:36:13.117584+0000","last_unstale":"2024-09-18T07:40:04.963378+0000","last_undegraded":"2024-09-18T07:40:04.963378+0000","last_fullsized":"2024-09-18T07:40:04.963378+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:15.687992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":494,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.956937+0000","last_change":"2024-09-18T07:39:37.892445+0000","last_active":"2024-09-18T07:40:04.956937+0000","last_peered":"2024-09-18T07:40:04.956937+0000","last_clean":"2024-09-18T07:40:04.956937+0000","last_became_active":"2024-09-18T07:31:02.349478+0000","last_became_peered":"2024-09-18T07:31:02.349478+0000","last_unstale":"2024-09-18T07:40:04.956937+0000","last_undegraded":"2024-09-18T07:40:04.956937+0000","last_fullsized":"2024-09-18T07:40:04.956937+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.892377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.892377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:56.957035+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":388,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:06.762667+0000","last_change":"2024-09-18T07:40:06.762667+0000","last_active":"2024-09-18T07:40:06.762667+0000","last_peered":"2024-09-18T07:40:06.762667+0000","last_clean":"2024-09-18T07:40:06.762667+0000","last_became_active":"2024-09-18T07:37:43.067598+0000","last_became_peered":"2024-09-18T07:37:43.067598+0000","last_unstale":"2024-09-18T07:40:06.762667+0000","last_undegraded":"2024-09-18T07:40:06.762667+0000","last_fullsized":"2024-09-18T07:40:06.762667+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:06.762589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:06.762589+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:16.226171+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":280,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.950239+0000","last_change":"2024-09-18T07:35:57.293607+0000","last_active":"2024-09-18T07:40:04.950239+0000","last_peered":"2024-09-18T07:40:04.950239+0000","last_clean":"2024-09-18T07:40:04.950239+0000","last_became_active":"2024-09-18T07:35:01.405287+0000","last_became_peered":"2024-09-18T07:35:01.405287+0000","last_unstale":"2024-09-18T07:40:04.950239+0000","last_undegraded":"2024-09-18T07:40:04.950239+0000","last_fullsized":"2024-09-18T07:40:04.950239+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:57.293555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:57.293555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:12.492824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":506,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.971659+0000","last_change":"2024-09-18T07:38:56.920456+0000","last_active":"2024-09-18T07:40:04.971659+0000","last_peered":"2024-09-18T07:40:04.971659+0000","last_clean":"2024-09-18T07:40:04.971659+0000","last_became_active":"2024-09-18T07:31:00.333281+0000","last_became_peered":"2024-09-18T07:31:00.333281+0000","last_unstale":"2024-09-18T07:40:04.971659+0000","last_undegraded":"2024-09-18T07:40:04.971659+0000","last_fullsized":"2024-09-18T07:40:04.971659+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:56.920384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:56.920384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:11.540961+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":515,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:05.788436+0000","last_change":"2024-09-18T07:40:05.788436+0000","last_active":"2024-09-18T07:40:05.788436+0000","last_peered":"2024-09-18T07:40:05.788436+0000","last_clean":"2024-09-18T07:40:05.788436+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:40:05.788436+0000","last_undegraded":"2024-09-18T07:40:05.788436+0000","last_fullsized":"2024-09-18T07:40:05.788436+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:05.788371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:05.788371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:11.393036+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":605,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.957341+0000","last_change":"2024-09-18T07:37:43.138620+0000","last_active":"2024-09-18T07:40:04.957341+0000","last_peered":"2024-09-18T07:40:04.957341+0000","last_clean":"2024-09-18T07:40:04.957341+0000","last_became_active":"2024-09-18T07:37:43.066208+0000","last_became_peered":"2024-09-18T07:37:43.066208+0000","last_unstale":"2024-09-18T07:40:04.957341+0000","last_undegraded":"2024-09-18T07:40:04.957341+0000","last_fullsized":"2024-09-18T07:40:04.957341+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:36:29.051437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:29.051437+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:53.872186+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":519,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963125+0000","last_change":"2024-09-18T07:35:01.415691+0000","last_active":"2024-09-18T07:40:04.963125+0000","last_peered":"2024-09-18T07:40:04.963125+0000","last_clean":"2024-09-18T07:40:04.963125+0000","last_became_active":"2024-09-18T07:35:01.415261+0000","last_became_peered":"2024-09-18T07:35:01.415261+0000","last_unstale":"2024-09-18T07:40:04.963125+0000","last_undegraded":"2024-09-18T07:40:04.963125+0000","last_fullsized":"2024-09-18T07:40:04.963125+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:58.734027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:58.734027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:17.638037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":294,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949663+0000","last_change":"2024-09-18T07:37:43.057780+0000","last_active":"2024-09-18T07:40:04.949663+0000","last_peered":"2024-09-18T07:40:04.949663+0000","last_clean":"2024-09-18T07:40:04.949663+0000","last_became_active":"2024-09-18T07:37:43.057483+0000","last_became_peered":"2024-09-18T07:37:43.057483+0000","last_unstale":"2024-09-18T07:40:04.949663+0000","last_undegraded":"2024-09-18T07:40:04.949663+0000","last_fullsized":"2024-09-18T07:40:04.949663+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:41.311291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:35:41.311291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.461425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":524,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.971751+0000","last_change":"2024-09-18T07:39:02.930467+0000","last_active":"2024-09-18T07:40:04.971751+0000","last_peered":"2024-09-18T07:40:04.971751+0000","last_clean":"2024-09-18T07:40:04.971751+0000","last_became_active":"2024-09-18T07:37:43.067066+0000","last_became_peered":"2024-09-18T07:37:43.067066+0000","last_unstale":"2024-09-18T07:40:04.971751+0000","last_undegraded":"2024-09-18T07:40:04.971751+0000","last_fullsized":"2024-09-18T07:40:04.971751+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:02.930392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:02.930392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:13.214721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":563,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.956880+0000","last_change":"2024-09-18T07:37:43.134956+0000","last_active":"2024-09-18T07:40:04.956880+0000","last_peered":"2024-09-18T07:40:04.956880+0000","last_clean":"2024-09-18T07:40:04.956880+0000","last_became_active":"2024-09-18T07:37:43.067503+0000","last_became_peered":"2024-09-18T07:37:43.067503+0000","last_unstale":"2024-09-18T07:40:04.956880+0000","last_undegraded":"2024-09-18T07:40:04.956880+0000","last_fullsized":"2024-09-18T07:40:04.956880+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:36:40.908629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:40.908629+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:56.851964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":503,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.964620+0000","last_change":"2024-09-18T07:39:05.648336+0000","last_active":"2024-09-18T07:40:04.964620+0000","last_peered":"2024-09-18T07:40:04.964620+0000","last_clean":"2024-09-18T07:40:04.964620+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:40:04.964620+0000","last_undegraded":"2024-09-18T07:40:04.964620+0000","last_fullsized":"2024-09-18T07:40:04.964620+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:05.648263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.648263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:07.539151+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":139,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.952809+0000","last_change":"2024-09-18T07:37:43.061403+0000","last_active":"2024-09-18T07:40:04.952809+0000","last_peered":"2024-09-18T07:40:04.952809+0000","last_clean":"2024-09-18T07:40:04.952809+0000","last_became_active":"2024-09-18T07:37:43.061081+0000","last_became_peered":"2024-09-18T07:37:43.061081+0000","last_unstale":"2024-09-18T07:40:04.952809+0000","last_undegraded":"2024-09-18T07:40:04.952809+0000","last_fullsized":"2024-09-18T07:40:04.952809+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:30.345180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:30.345180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:39.067422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":288,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.949798+0000","last_change":"2024-09-18T07:39:02.503766+0000","last_active":"2024-09-18T07:40:04.949798+0000","last_peered":"2024-09-18T07:40:04.949798+0000","last_clean":"2024-09-18T07:40:04.949798+0000","last_became_active":"2024-09-18T07:35:01.400396+0000","last_became_peered":"2024-09-18T07:35:01.400396+0000","last_unstale":"2024-09-18T07:40:04.949798+0000","last_undegraded":"2024-09-18T07:40:04.949798+0000","last_fullsized":"2024-09-18T07:40:04.949798+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:02.503698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:02.503698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:11.708815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":198,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.963545+0000","last_change":"2024-09-18T07:36:13.118009+0000","last_active":"2024-09-18T07:40:04.963545+0000","last_peered":"2024-09-18T07:40:04.963545+0000","last_clean":"2024-09-18T07:40:04.963545+0000","last_became_active":"2024-09-18T07:36:13.117154+0000","last_became_peered":"2024-09-18T07:36:13.117154+0000","last_unstale":"2024-09-18T07:40:04.963545+0000","last_undegraded":"2024-09-18T07:40:04.963545+0000","last_fullsized":"2024-09-18T07:40:04.963545+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.071629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.071629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:21.615923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":528,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.957496+0000","last_change":"2024-09-18T07:39:35.902385+0000","last_active":"2024-09-18T07:40:04.957496+0000","last_peered":"2024-09-18T07:40:04.957496+0000","last_clean":"2024-09-18T07:40:04.957496+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:40:04.957496+0000","last_undegraded":"2024-09-18T07:40:04.957496+0000","last_fullsized":"2024-09-18T07:40:04.957496+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:39:35.901306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.901306+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.222739+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":151,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.953373+0000","last_change":"2024-09-18T07:37:43.258255+0000","last_active":"2024-09-18T07:40:04.953373+0000","last_peered":"2024-09-18T07:40:04.953373+0000","last_clean":"2024-09-18T07:40:04.953373+0000","last_became_active":"2024-09-18T07:37:43.070186+0000","last_became_peered":"2024-09-18T07:37:43.070186+0000","last_unstale":"2024-09-18T07:40:04.953373+0000","last_undegraded":"2024-09-18T07:40:04.953373+0000","last_fullsized":"2024-09-18T07:40:04.953373+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:34:58.432781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:58.432781+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:22.023484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":526,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-09-18T07:40:04.964270+0000","last_change":"2024-09-18T07:39:35.694805+0000","last_active":"2024-09-18T07:40:04.964270+0000","last_peered":"2024-09-18T07:40:04.964270+0000","last_clean":"2024-09-18T07:40:04.964270+0000","last_became_active":"2024-09-18T07:35:01.410469+0000","last_became_peered":"2024-09-18T07:35:01.410469+0000","last_unstale":"2024-09-18T07:40:04.964270+0000","last_undegraded":"2024-09-18T07:40:04.964270+0000","last_fullsized":"2024-09-18T07:40:04.964270+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.694741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.694741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:54.064934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4184670208,"num_objects":63853,"num_object_clones":0,"num_object_copies":191559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63853,"num_write_kb":4086592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12518490112,"data_stored":12518490112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63853,"ondisk_log_size":63853,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":4664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":398,"ondisk_log_size":398,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":311,"num_read_kb":306,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201394,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2520628,"kb_used_data":2222880,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":91224524,"statfs":{"total":95995035648,"available":93413912576,"internally_reserved":0,"allocated":2276229120,"data_stored":2275627878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":304871882},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463032,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1024656,"kb_used_data":754688,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92720496,"statfs":{"total":95995035648,"available":94945787904,"internally_reserved":0,"allocated":772800512,"data_stored":772230623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":276429254},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724669,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2577000,"kb_used_data":2252884,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91168152,"statfs":{"total":95995035648,"available":93356187648,"internally_reserved":0,"allocated":2306953216,"data_stored":2306364177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953604,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1057076,"kb_used_data":760736,"kb_used_omap":1,"kb_used_meta":296318,"kb_avail":92688076,"statfs":{"total":95995035648,"available":94912589824,"internally_reserved":0,"allocated":778993664,"data_stored":778392145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":303430085},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149832,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2645432,"kb_used_data":2226916,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":91099720,"statfs":{"total":95995035648,"available":93286113280,"internally_reserved":0,"allocated":2280361984,"data_stored":2279757765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":428538311},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411470,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1433572,"kb_used_data":1087248,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92311580,"statfs":{"total":95995035648,"available":94527057920,"internally_reserved":0,"allocated":1113341952,"data_stored":1112756908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574997,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2665020,"kb_used_data":2282604,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":91080132,"statfs":{"total":95995035648,"available":93266055168,"internally_reserved":0,"allocated":2337386496,"data_stored":2336773766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738522,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1483772,"kb_used_data":1141608,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92261380,"statfs":{"total":95995035648,"available":94475653120,"internally_reserved":0,"allocated":1169006592,"data_stored":1168398283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1047920640,"data_stored":1047920640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2356477952,"data_stored":2356477952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":786759680,"data_stored":786759680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2105671680,"data_stored":2105671680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":783941632,"data_stored":783941632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2333933568,"data_stored":2333933568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":780402688,"data_stored":780402688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2323382272,"data_stored":2323382272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:40:09.794 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-09-18T07:40:09.834 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:09.867 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:09.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:09.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:09.969 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:10.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:10.013 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:10.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:10.078 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 35 16 68198 68182 121.726 89 0.0193637 0.00821018 2024-09-18T07:40:10.092 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:10.100 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:10.114 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:10.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:10.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:10.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:10.266 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:09 smithi137 ceph-mon[29408]: from='client.15771 v1:172.21.15.137:0/3880171009' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:40:10.266 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:09 smithi137 ceph-mon[39175]: from='client.15771 v1:172.21.15.137:0/3880171009' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:40:10.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:10.315 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:10.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:10.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:10.417 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:10.453 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:10.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:10.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:10.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:09 smithi203 ceph-mon[32237]: from='client.15771 v1:172.21.15.137:0/3880171009' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:40:10.555 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:10.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:10.597 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:10.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:10.698 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:10.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:10.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:10.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:10.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:10.873 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:10.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:10.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:10.963 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:40:10.963 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:40:10.963 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:40:10.963 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:40:10.964 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:40:10.964 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:40:10.964 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:40:10.964 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:40:10.964 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:40:10.964 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:40:10.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:10.991 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:40:10.992 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:40:11.006 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:40:11.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:11.022 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:11.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:11.038 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:40:11.038 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:40:11.078 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 36 16 69512 69496 120.626 82.125 0.00432195 0.00828505 2024-09-18T07:40:11.079 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:40:11.079 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:40:11.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:11.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:11.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:11.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:11.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[29408]: 5.8 scrub starts 2024-09-18T07:40:11.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[29408]: 5.8 scrub ok 2024-09-18T07:40:11.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[29408]: pgmap v495: 158 pgs: 158 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-18T07:40:11.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[29408]: 6.c scrub starts 2024-09-18T07:40:11.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[29408]: 6.c scrub ok 2024-09-18T07:40:11.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[29408]: 6.1 scrub starts 2024-09-18T07:40:11.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[29408]: 6.1 scrub ok 2024-09-18T07:40:11.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[29408]: 4.9 scrub starts 2024-09-18T07:40:11.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[29408]: 4.9 scrub ok 2024-09-18T07:40:11.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3729939703' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:11.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[29408]: from='client.35636 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:11.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[29408]: 6.1c scrub starts 2024-09-18T07:40:11.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[29408]: 6.1c scrub ok 2024-09-18T07:40:11.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3342157435' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:11.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[39175]: 5.8 scrub starts 2024-09-18T07:40:11.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[39175]: 5.8 scrub ok 2024-09-18T07:40:11.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[39175]: pgmap v495: 158 pgs: 158 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-18T07:40:11.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[39175]: 6.c scrub starts 2024-09-18T07:40:11.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[39175]: 6.c scrub ok 2024-09-18T07:40:11.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[39175]: 6.1 scrub starts 2024-09-18T07:40:11.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[39175]: 6.1 scrub ok 2024-09-18T07:40:11.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[39175]: 4.9 scrub starts 2024-09-18T07:40:11.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[39175]: 4.9 scrub ok 2024-09-18T07:40:11.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3729939703' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:11.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[39175]: from='client.35636 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:11.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[39175]: 6.1c scrub starts 2024-09-18T07:40:11.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[39175]: 6.1c scrub ok 2024-09-18T07:40:11.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:11 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3342157435' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:11.268 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:11.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:11.303 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:11.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:11.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:11.440 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:11.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:11.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:11 smithi203 ceph-mon[32237]: 5.8 scrub starts 2024-09-18T07:40:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:11 smithi203 ceph-mon[32237]: 5.8 scrub ok 2024-09-18T07:40:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:11 smithi203 ceph-mon[32237]: pgmap v495: 158 pgs: 158 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-18T07:40:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:11 smithi203 ceph-mon[32237]: 6.c scrub starts 2024-09-18T07:40:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:11 smithi203 ceph-mon[32237]: 6.c scrub ok 2024-09-18T07:40:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:11 smithi203 ceph-mon[32237]: 6.1 scrub starts 2024-09-18T07:40:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:11 smithi203 ceph-mon[32237]: 6.1 scrub ok 2024-09-18T07:40:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:11 smithi203 ceph-mon[32237]: 4.9 scrub starts 2024-09-18T07:40:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:11 smithi203 ceph-mon[32237]: 4.9 scrub ok 2024-09-18T07:40:11.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:11 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3729939703' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:11.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:11 smithi203 ceph-mon[32237]: from='client.35636 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:11.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:11 smithi203 ceph-mon[32237]: 6.1c scrub starts 2024-09-18T07:40:11.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:11 smithi203 ceph-mon[32237]: 6.1c scrub ok 2024-09-18T07:40:11.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:11 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3342157435' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:11.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:11.579 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:11.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:11.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:11.681 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:11.720 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:11.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:11.822 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:11.858 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:11.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:11.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:11.960 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:11.998 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:11.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:11.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:12.078 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 37 16 70949 70933 119.793 89.8125 0.0136804 0.00834229 2024-09-18T07:40:12.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:12.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:12.141 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:12.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:12.223 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:40:12.223 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:40:12.243 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:12.278 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:12.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:12.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:12.380 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:12.407 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:12.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:12.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[29408]: 5.19 scrub starts 2024-09-18T07:40:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[29408]: 5.19 scrub ok 2024-09-18T07:40:12.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[29408]: 6.f scrub starts 2024-09-18T07:40:12.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[29408]: 6.f scrub ok 2024-09-18T07:40:12.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[29408]: 4.b scrub starts 2024-09-18T07:40:12.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[29408]: 4.b scrub ok 2024-09-18T07:40:12.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:40:12.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:40:12.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[29408]: from='client.35636 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:40:12.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[29408]: osdmap e304: 8 total, 8 up, 8 in 2024-09-18T07:40:12.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[29408]: 5.18 scrub starts 2024-09-18T07:40:12.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[29408]: 5.18 scrub ok 2024-09-18T07:40:12.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:40:12.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:40:12.423 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[39175]: 5.19 scrub starts 2024-09-18T07:40:12.423 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[39175]: 5.19 scrub ok 2024-09-18T07:40:12.423 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[39175]: 6.f scrub starts 2024-09-18T07:40:12.423 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[39175]: 6.f scrub ok 2024-09-18T07:40:12.424 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[39175]: 4.b scrub starts 2024-09-18T07:40:12.424 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[39175]: 4.b scrub ok 2024-09-18T07:40:12.424 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:40:12.424 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:40:12.424 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[39175]: from='client.35636 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:40:12.424 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[39175]: osdmap e304: 8 total, 8 up, 8 in 2024-09-18T07:40:12.425 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[39175]: 5.18 scrub starts 2024-09-18T07:40:12.425 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[39175]: 5.18 scrub ok 2024-09-18T07:40:12.425 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:40:12.426 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:40:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:12 smithi203 ceph-mon[32237]: 5.19 scrub starts 2024-09-18T07:40:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:12 smithi203 ceph-mon[32237]: 5.19 scrub ok 2024-09-18T07:40:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:12 smithi203 ceph-mon[32237]: 6.f scrub starts 2024-09-18T07:40:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:12 smithi203 ceph-mon[32237]: 6.f scrub ok 2024-09-18T07:40:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:12 smithi203 ceph-mon[32237]: 4.b scrub starts 2024-09-18T07:40:12.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:12 smithi203 ceph-mon[32237]: 4.b scrub ok 2024-09-18T07:40:12.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:12 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:40:12.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:12 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:40:12.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:12 smithi203 ceph-mon[32237]: from='client.35636 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:40:12.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:12 smithi203 ceph-mon[32237]: osdmap e304: 8 total, 8 up, 8 in 2024-09-18T07:40:12.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:12 smithi203 ceph-mon[32237]: 5.18 scrub starts 2024-09-18T07:40:12.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:12 smithi203 ceph-mon[32237]: 5.18 scrub ok 2024-09-18T07:40:12.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:12 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:40:12.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:40:12.510 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:12.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:12.537 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:12.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:12.638 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:12.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:12.674 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:12.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:12.775 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:12.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:12.798 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:12.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:12.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:12.936 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:12.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:12.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:13.038 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:13.069 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:13.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:13.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:13.078 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 38 16 72320 72304 118.896 85.6875 0.0125278 0.00840611 2024-09-18T07:40:13.171 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:13.207 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:13.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:13.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:13.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:13.345 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:13.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:13.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:13 smithi137 ceph-mon[39175]: 3.1c scrub starts 2024-09-18T07:40:13.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:13 smithi137 ceph-mon[39175]: 3.1c scrub ok 2024-09-18T07:40:13.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:13 smithi137 ceph-mon[39175]: pgmap v497: 158 pgs: 158 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-18T07:40:13.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:13 smithi137 ceph-mon[39175]: 6.1f scrub starts 2024-09-18T07:40:13.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:13 smithi137 ceph-mon[39175]: 6.1f scrub ok 2024-09-18T07:40:13.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:13 smithi137 ceph-mon[39175]: 3.17 scrub starts 2024-09-18T07:40:13.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:13 smithi137 ceph-mon[39175]: 3.17 scrub ok 2024-09-18T07:40:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:13 smithi137 ceph-mon[29408]: 3.1c scrub starts 2024-09-18T07:40:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:13 smithi137 ceph-mon[29408]: 3.1c scrub ok 2024-09-18T07:40:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:13 smithi137 ceph-mon[29408]: pgmap v497: 158 pgs: 158 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-18T07:40:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:13 smithi137 ceph-mon[29408]: 6.1f scrub starts 2024-09-18T07:40:13.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:13 smithi137 ceph-mon[29408]: 6.1f scrub ok 2024-09-18T07:40:13.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:13 smithi137 ceph-mon[29408]: 3.17 scrub starts 2024-09-18T07:40:13.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:13 smithi137 ceph-mon[29408]: 3.17 scrub ok 2024-09-18T07:40:13.446 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:13 smithi203 ceph-mon[32237]: 3.1c scrub starts 2024-09-18T07:40:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:13 smithi203 ceph-mon[32237]: 3.1c scrub ok 2024-09-18T07:40:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:13 smithi203 ceph-mon[32237]: pgmap v497: 158 pgs: 158 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-18T07:40:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:13 smithi203 ceph-mon[32237]: 6.1f scrub starts 2024-09-18T07:40:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:13 smithi203 ceph-mon[32237]: 6.1f scrub ok 2024-09-18T07:40:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:13 smithi203 ceph-mon[32237]: 3.17 scrub starts 2024-09-18T07:40:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:13 smithi203 ceph-mon[32237]: 3.17 scrub ok 2024-09-18T07:40:13.472 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:13.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:13.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:13.574 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:13.610 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:13.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:13.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:13.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:13.745 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:13.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:13.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:13.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:13.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:13.882 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:13.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:13.983 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:14.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:14.018 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:14.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:14.079 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 39 16 73664 73648 118.001 84 0.00274693 0.00846677 2024-09-18T07:40:14.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:40:14.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:14.176 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:14.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:14.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:14.278 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:14.357 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:14.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:14.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:14.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:14 smithi137 ceph-mon[29408]: 6.4 scrub starts 2024-09-18T07:40:14.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:14 smithi137 ceph-mon[29408]: 6.4 scrub ok 2024-09-18T07:40:14.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:14 smithi137 ceph-mon[29408]: 4.4 scrub starts 2024-09-18T07:40:14.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:14 smithi137 ceph-mon[29408]: 4.4 scrub ok 2024-09-18T07:40:14.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:14 smithi137 ceph-mon[29408]: 3.15 scrub starts 2024-09-18T07:40:14.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:14 smithi137 ceph-mon[29408]: 3.15 scrub ok 2024-09-18T07:40:14.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:14 smithi137 ceph-mon[39175]: 6.4 scrub starts 2024-09-18T07:40:14.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:14 smithi137 ceph-mon[39175]: 6.4 scrub ok 2024-09-18T07:40:14.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:14 smithi137 ceph-mon[39175]: 4.4 scrub starts 2024-09-18T07:40:14.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:14 smithi137 ceph-mon[39175]: 4.4 scrub ok 2024-09-18T07:40:14.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:14 smithi137 ceph-mon[39175]: 3.15 scrub starts 2024-09-18T07:40:14.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:14 smithi137 ceph-mon[39175]: 3.15 scrub ok 2024-09-18T07:40:14.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:14.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:14 smithi203 ceph-mon[32237]: 6.4 scrub starts 2024-09-18T07:40:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:14 smithi203 ceph-mon[32237]: 6.4 scrub ok 2024-09-18T07:40:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:14 smithi203 ceph-mon[32237]: 4.4 scrub starts 2024-09-18T07:40:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:14 smithi203 ceph-mon[32237]: 4.4 scrub ok 2024-09-18T07:40:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:14 smithi203 ceph-mon[32237]: 3.15 scrub starts 2024-09-18T07:40:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:14 smithi203 ceph-mon[32237]: 3.15 scrub ok 2024-09-18T07:40:14.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:14.493 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:14.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:14.594 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:14.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:14.635 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:14.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:14.736 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:14.774 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:14.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:14.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:14.795 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:40:14.795 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:40:14.795 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-18T07:40:14.795 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.759495 2024-09-18T07:40:14.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 0.7594952128607749 2024-09-18T07:40:14.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:14.917 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:14.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:14.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:15.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:15.059 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:15.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:15.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:15.079 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:40:15.077311+0000 min lat: 0.00110395 max lat: 0.124316 avg lat: 0.00854745 2024-09-18T07:40:15.079 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:40:15.079 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 40 16 74863 74847 116.924 74.9375 0.0106902 0.00854745 2024-09-18T07:40:15.161 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:15.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:15 smithi137 ceph-mon[29408]: pgmap v498: 158 pgs: 158 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-18T07:40:15.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:15 smithi137 ceph-mon[29408]: 5.2 scrub starts 2024-09-18T07:40:15.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:15 smithi137 ceph-mon[29408]: 5.2 scrub ok 2024-09-18T07:40:15.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:15 smithi137 ceph-mon[29408]: 5.10 scrub starts 2024-09-18T07:40:15.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:15 smithi137 ceph-mon[29408]: 5.10 scrub ok 2024-09-18T07:40:15.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:15 smithi137 ceph-mon[29408]: osdmap e305: 8 total, 8 up, 8 in 2024-09-18T07:40:15.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:15 smithi137 ceph-mon[39175]: pgmap v498: 158 pgs: 158 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-18T07:40:15.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:15 smithi137 ceph-mon[39175]: 5.2 scrub starts 2024-09-18T07:40:15.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:15 smithi137 ceph-mon[39175]: 5.2 scrub ok 2024-09-18T07:40:15.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:15 smithi137 ceph-mon[39175]: 5.10 scrub starts 2024-09-18T07:40:15.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:15 smithi137 ceph-mon[39175]: 5.10 scrub ok 2024-09-18T07:40:15.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:15 smithi137 ceph-mon[39175]: osdmap e305: 8 total, 8 up, 8 in 2024-09-18T07:40:15.204 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:15.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:15.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:15.306 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:15.343 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:15.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:15.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:15.445 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:15.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:15 smithi203 ceph-mon[32237]: pgmap v498: 158 pgs: 158 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-09-18T07:40:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:15 smithi203 ceph-mon[32237]: 5.2 scrub starts 2024-09-18T07:40:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:15 smithi203 ceph-mon[32237]: 5.2 scrub ok 2024-09-18T07:40:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:15 smithi203 ceph-mon[32237]: 5.10 scrub starts 2024-09-18T07:40:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:15 smithi203 ceph-mon[32237]: 5.10 scrub ok 2024-09-18T07:40:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:15 smithi203 ceph-mon[32237]: osdmap e305: 8 total, 8 up, 8 in 2024-09-18T07:40:15.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:15.487 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:15.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:15.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:15.616 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:15.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:15.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:15.753 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:15.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:15.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:15.855 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:15.892 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:15.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:15.994 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:16.031 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:16.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:16.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:16.079 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 41 16 76249 76233 116.185 86.625 0.00686031 0.00860229 2024-09-18T07:40:16.133 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:16.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T07:40:16.165 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:16.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:16.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:16.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:16.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:16.311 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:16.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:16.322 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:16 smithi137 ceph-mon[29408]: 4.1c scrub starts 2024-09-18T07:40:16.322 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:16 smithi137 ceph-mon[29408]: 4.1c scrub ok 2024-09-18T07:40:16.322 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:16 smithi137 ceph-mon[29408]: 6.17 scrub starts 2024-09-18T07:40:16.322 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:16 smithi137 ceph-mon[29408]: 6.17 scrub ok 2024-09-18T07:40:16.322 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:16 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:40:16.322 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:16 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:40:16.322 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:16 smithi137 ceph-mon[39175]: 4.1c scrub starts 2024-09-18T07:40:16.322 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:16 smithi137 ceph-mon[39175]: 4.1c scrub ok 2024-09-18T07:40:16.322 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:16 smithi137 ceph-mon[39175]: 6.17 scrub starts 2024-09-18T07:40:16.323 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:16 smithi137 ceph-mon[39175]: 6.17 scrub ok 2024-09-18T07:40:16.323 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:16 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:40:16.323 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:16 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:40:16.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:16.441 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:16.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:16.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:16 smithi203 ceph-mon[32237]: 4.1c scrub starts 2024-09-18T07:40:16.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:16 smithi203 ceph-mon[32237]: 4.1c scrub ok 2024-09-18T07:40:16.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:16 smithi203 ceph-mon[32237]: 6.17 scrub starts 2024-09-18T07:40:16.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:16 smithi203 ceph-mon[32237]: 6.17 scrub ok 2024-09-18T07:40:16.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:16 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:40:16.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:16 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:40:16.543 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:16.585 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:16.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:16.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:16.687 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:16.719 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:16.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:16.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:16.820 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:16.821 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:16.851 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:16.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:16.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:16.924 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:16.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:17.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:17.016 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:17.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:17.079 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 42 16 77535 77519 115.331 80.375 0.0142751 0.00866555 2024-09-18T07:40:17.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:17.155 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:17.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:17.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:17.257 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:17.284 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:17.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:17.312 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:40:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:40:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:40:17.312 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:17 smithi137 ceph-mon[39175]: pgmap v500: 158 pgs: 158 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 862 op/s 2024-09-18T07:40:17.312 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:17 smithi137 ceph-mon[39175]: osdmap e306: 8 total, 8 up, 8 in 2024-09-18T07:40:17.312 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:17 smithi137 ceph-mon[39175]: 4.13 scrub starts 2024-09-18T07:40:17.312 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:17 smithi137 ceph-mon[39175]: 4.13 scrub ok 2024-09-18T07:40:17.312 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:17 smithi137 ceph-mon[39175]: 3.11 scrub starts 2024-09-18T07:40:17.312 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:17 smithi137 ceph-mon[39175]: 3.11 scrub ok 2024-09-18T07:40:17.313 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:17 smithi137 ceph-mon[29408]: pgmap v500: 158 pgs: 158 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 862 op/s 2024-09-18T07:40:17.313 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:17 smithi137 ceph-mon[29408]: osdmap e306: 8 total, 8 up, 8 in 2024-09-18T07:40:17.313 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:17 smithi137 ceph-mon[29408]: 4.13 scrub starts 2024-09-18T07:40:17.313 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:17 smithi137 ceph-mon[29408]: 4.13 scrub ok 2024-09-18T07:40:17.313 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:17 smithi137 ceph-mon[29408]: 3.11 scrub starts 2024-09-18T07:40:17.313 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:17 smithi137 ceph-mon[29408]: 3.11 scrub ok 2024-09-18T07:40:17.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:17.387 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:17 smithi203 ceph-mon[32237]: pgmap v500: 158 pgs: 158 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 862 op/s 2024-09-18T07:40:17.387 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:17 smithi203 ceph-mon[32237]: osdmap e306: 8 total, 8 up, 8 in 2024-09-18T07:40:17.387 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:17 smithi203 ceph-mon[32237]: 4.13 scrub starts 2024-09-18T07:40:17.387 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:17 smithi203 ceph-mon[32237]: 4.13 scrub ok 2024-09-18T07:40:17.387 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:17 smithi203 ceph-mon[32237]: 3.11 scrub starts 2024-09-18T07:40:17.387 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:17 smithi203 ceph-mon[32237]: 3.11 scrub ok 2024-09-18T07:40:17.426 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:17.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:17.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:17.528 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:17.560 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:17.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:17.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:17.662 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:17.683 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:17.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:17.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:17.710 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:40:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:40:17.785 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:17.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:17.818 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:17.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:17.919 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:17.938 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:17.950 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:17.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:17.983 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:40:17.983 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:40:18.048 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.3 primary-affinity to 0.759495 (8497742) 2024-09-18T07:40:18.050 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:18.080 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 43 16 78627 78611 114.236 68.25 0.00789505 0.00874864 2024-09-18T07:40:18.081 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:18.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:18.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:18.182 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:18.207 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:18.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:18.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:18.229 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:18 smithi137 ceph-mon[29408]: 4.18 scrub starts 2024-09-18T07:40:18.230 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:18 smithi137 ceph-mon[29408]: 4.18 scrub ok 2024-09-18T07:40:18.230 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:18 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4177524907' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.7594952128607749} : dispatch 2024-09-18T07:40:18.230 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:18 smithi137 ceph-mon[39175]: 4.18 scrub starts 2024-09-18T07:40:18.230 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:18 smithi137 ceph-mon[39175]: 4.18 scrub ok 2024-09-18T07:40:18.230 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:18 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4177524907' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.7594952128607749} : dispatch 2024-09-18T07:40:18.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:18.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:18.332 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:18.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:18.433 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:18.456 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:18.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:18.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:18.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:18 smithi203 ceph-mon[32237]: 4.18 scrub starts 2024-09-18T07:40:18.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:18 smithi203 ceph-mon[32237]: 4.18 scrub ok 2024-09-18T07:40:18.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:18 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4177524907' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.7594952128607749} : dispatch 2024-09-18T07:40:18.558 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:18.583 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:18.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:18.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:18.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:18.709 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:18.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:18.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:40:18.810 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:18.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:18.836 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:18.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:18.937 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:18.963 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:18.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:18.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:19.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T07:40:19.065 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:19.079 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 44 16 78763 78747 111.833 8.5 0.003077 0.00874483 2024-09-18T07:40:19.095 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:19.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:19.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:19.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:19 smithi137 ceph-mon[29408]: 6.a scrub starts 2024-09-18T07:40:19.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:19 smithi137 ceph-mon[29408]: 6.a scrub ok 2024-09-18T07:40:19.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:19 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:19.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:19 smithi137 ceph-mon[29408]: pgmap v502: 158 pgs: 158 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-09-18T07:40:19.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:19 smithi137 ceph-mon[29408]: 4.5 scrub starts 2024-09-18T07:40:19.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:19 smithi137 ceph-mon[29408]: 4.5 scrub ok 2024-09-18T07:40:19.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:19 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4177524907' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.7594952128607749}]': finished 2024-09-18T07:40:19.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:19 smithi137 ceph-mon[29408]: osdmap e307: 8 total, 8 up, 8 in 2024-09-18T07:40:19.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:19 smithi137 ceph-mon[39175]: 6.a scrub starts 2024-09-18T07:40:19.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:19 smithi137 ceph-mon[39175]: 6.a scrub ok 2024-09-18T07:40:19.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:19 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:19.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:19 smithi137 ceph-mon[39175]: pgmap v502: 158 pgs: 158 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-09-18T07:40:19.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:19 smithi137 ceph-mon[39175]: 4.5 scrub starts 2024-09-18T07:40:19.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:19 smithi137 ceph-mon[39175]: 4.5 scrub ok 2024-09-18T07:40:19.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:19 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4177524907' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.7594952128607749}]': finished 2024-09-18T07:40:19.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:19 smithi137 ceph-mon[39175]: osdmap e307: 8 total, 8 up, 8 in 2024-09-18T07:40:19.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:19.226 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:19.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:19.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:19.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:19.363 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:19.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:19.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:19.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:19 smithi203 ceph-mon[32237]: 6.a scrub starts 2024-09-18T07:40:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:19 smithi203 ceph-mon[32237]: 6.a scrub ok 2024-09-18T07:40:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:19 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:19 smithi203 ceph-mon[32237]: pgmap v502: 158 pgs: 158 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-09-18T07:40:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:19 smithi203 ceph-mon[32237]: 4.5 scrub starts 2024-09-18T07:40:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:19 smithi203 ceph-mon[32237]: 4.5 scrub ok 2024-09-18T07:40:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:19 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4177524907' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.7594952128607749}]': finished 2024-09-18T07:40:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:19 smithi203 ceph-mon[32237]: osdmap e307: 8 total, 8 up, 8 in 2024-09-18T07:40:19.465 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:19.501 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:19.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:19.603 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:19.636 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:19.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:19.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:19.737 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:19.772 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:19.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:19.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:19.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:19.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:19.910 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:19.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:20.007 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:20.011 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:20.019 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:20.076 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:20 smithi137 ceph-mon[29408]: 4.2 scrub starts 2024-09-18T07:40:20.077 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:20.077 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:20 smithi137 ceph-mon[39175]: 4.2 scrub starts 2024-09-18T07:40:20.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:20.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:20.081 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 45 16 79975 79959 111.028 75.75 0.0236729 0.00899997 2024-09-18T07:40:20.180 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:20.210 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:20.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:20.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:20.312 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:20.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:20 smithi137 ceph-mon[29408]: 4.2 scrub ok 2024-09-18T07:40:20.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:20 smithi137 ceph-mon[29408]: 1.6 scrub starts 2024-09-18T07:40:20.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:20 smithi137 ceph-mon[29408]: 1.6 scrub ok 2024-09-18T07:40:20.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:20 smithi137 ceph-mon[29408]: osdmap e308: 8 total, 8 up, 8 in 2024-09-18T07:40:20.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:20 smithi137 ceph-mon[29408]: 1.5 scrub starts 2024-09-18T07:40:20.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:20 smithi137 ceph-mon[29408]: 1.5 scrub ok 2024-09-18T07:40:20.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:20 smithi137 ceph-mon[29408]: osdmap e309: 8 total, 8 up, 8 in 2024-09-18T07:40:20.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:20 smithi137 ceph-mon[39175]: 4.2 scrub ok 2024-09-18T07:40:20.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:20 smithi137 ceph-mon[39175]: 1.6 scrub starts 2024-09-18T07:40:20.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:20 smithi137 ceph-mon[39175]: 1.6 scrub ok 2024-09-18T07:40:20.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:20 smithi137 ceph-mon[39175]: osdmap e308: 8 total, 8 up, 8 in 2024-09-18T07:40:20.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:20 smithi137 ceph-mon[39175]: 1.5 scrub starts 2024-09-18T07:40:20.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:20 smithi137 ceph-mon[39175]: 1.5 scrub ok 2024-09-18T07:40:20.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:20 smithi137 ceph-mon[39175]: osdmap e309: 8 total, 8 up, 8 in 2024-09-18T07:40:20.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:20.347 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:20.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:20.448 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:20.463 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:20 smithi203 ceph-mon[32237]: 4.2 scrub starts 2024-09-18T07:40:20.463 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:20 smithi203 ceph-mon[32237]: 4.2 scrub ok 2024-09-18T07:40:20.463 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:20 smithi203 ceph-mon[32237]: 1.6 scrub starts 2024-09-18T07:40:20.463 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:20 smithi203 ceph-mon[32237]: 1.6 scrub ok 2024-09-18T07:40:20.463 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:20 smithi203 ceph-mon[32237]: osdmap e308: 8 total, 8 up, 8 in 2024-09-18T07:40:20.463 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:20 smithi203 ceph-mon[32237]: 1.5 scrub starts 2024-09-18T07:40:20.464 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:20 smithi203 ceph-mon[32237]: 1.5 scrub ok 2024-09-18T07:40:20.464 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:20 smithi203 ceph-mon[32237]: osdmap e309: 8 total, 8 up, 8 in 2024-09-18T07:40:20.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:20.488 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:20.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:20.590 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:20.627 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:20.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:20.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:20.772 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:20.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:20.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:20.874 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:20.911 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:20.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:20.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:21.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:21.052 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:21.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:21.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:21.081 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 46 16 81313 81297 110.433 83.625 0.00828973 0.00905054 2024-09-18T07:40:21.088 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:40:21.089 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:40:21.090 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:40:21.153 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:21.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:21 smithi137 ceph-mon[29408]: pgmap v505: 158 pgs: 158 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-18T07:40:21.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:21 smithi137 ceph-mon[29408]: 3.1 scrub starts 2024-09-18T07:40:21.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:21 smithi137 ceph-mon[29408]: 3.1 scrub ok 2024-09-18T07:40:21.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:21 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/340426445' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:40:21.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:21 smithi137 ceph-mon[29408]: from='client.35654 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:40:21.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:21 smithi137 ceph-mon[29408]: 3.13 scrub starts 2024-09-18T07:40:21.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:21 smithi137 ceph-mon[29408]: 3.13 scrub ok 2024-09-18T07:40:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:21 smithi137 ceph-mon[39175]: pgmap v505: 158 pgs: 158 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-18T07:40:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:21 smithi137 ceph-mon[39175]: 3.1 scrub starts 2024-09-18T07:40:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:21 smithi137 ceph-mon[39175]: 3.1 scrub ok 2024-09-18T07:40:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:21 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/340426445' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:40:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:21 smithi137 ceph-mon[39175]: from='client.35654 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:40:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:21 smithi137 ceph-mon[39175]: 3.13 scrub starts 2024-09-18T07:40:21.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:21 smithi137 ceph-mon[39175]: 3.13 scrub ok 2024-09-18T07:40:21.189 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:21.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:21.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:21.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:21.327 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:21.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:21.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:21.429 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:21.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:21 smithi203 ceph-mon[32237]: pgmap v505: 158 pgs: 158 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-18T07:40:21.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:21 smithi203 ceph-mon[32237]: 3.1 scrub starts 2024-09-18T07:40:21.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:21 smithi203 ceph-mon[32237]: 3.1 scrub ok 2024-09-18T07:40:21.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:21 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/340426445' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:40:21.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:21 smithi203 ceph-mon[32237]: from='client.35654 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:40:21.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:21 smithi203 ceph-mon[32237]: 3.13 scrub starts 2024-09-18T07:40:21.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:21 smithi203 ceph-mon[32237]: 3.13 scrub ok 2024-09-18T07:40:21.468 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:40:21.468 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:40:21.468 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:40:21.469 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:40:21.469 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:40:21.469 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:40:21.469 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:40:21.469 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:40:21.469 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:40:21.470 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:40:21.471 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:21.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:21.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:21.493 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:40:21.494 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:40:21.533 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:40:21.533 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:40:21.574 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:21.583 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:40:21.583 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:40:21.604 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:21.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:21.706 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:21.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:21.742 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:21.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:21.843 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:21.879 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:21.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:21.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:21.981 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:22.020 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:22.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:22.081 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 47 16 82640 82624 109.848 82.9375 0.0148082 0.00909874 2024-09-18T07:40:22.121 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:22.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:22.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:22.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:22.264 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:22.297 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:22.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:22.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:22.398 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:22.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:22 smithi137 ceph-mon[29408]: 4.19 scrub starts 2024-09-18T07:40:22.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:22 smithi137 ceph-mon[29408]: 4.19 scrub ok 2024-09-18T07:40:22.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:22 smithi137 ceph-mon[29408]: 4.17 scrub starts 2024-09-18T07:40:22.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:22 smithi137 ceph-mon[29408]: 4.17 scrub ok 2024-09-18T07:40:22.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:22 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:40:22.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:22 smithi137 ceph-mon[29408]: from='client.35654 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:40:22.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:22 smithi137 ceph-mon[29408]: osdmap e310: 8 total, 8 up, 8 in 2024-09-18T07:40:22.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:22 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/551401646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:22.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:22 smithi137 ceph-mon[39175]: 4.19 scrub starts 2024-09-18T07:40:22.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:22 smithi137 ceph-mon[39175]: 4.19 scrub ok 2024-09-18T07:40:22.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:22 smithi137 ceph-mon[39175]: 4.17 scrub starts 2024-09-18T07:40:22.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:22 smithi137 ceph-mon[39175]: 4.17 scrub ok 2024-09-18T07:40:22.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:22 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:40:22.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:22 smithi137 ceph-mon[39175]: from='client.35654 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:40:22.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:22 smithi137 ceph-mon[39175]: osdmap e310: 8 total, 8 up, 8 in 2024-09-18T07:40:22.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:22 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/551401646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:22.425 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:22.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:22.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:22.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:22 smithi203 ceph-mon[32237]: 4.19 scrub starts 2024-09-18T07:40:22.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:22 smithi203 ceph-mon[32237]: 4.19 scrub ok 2024-09-18T07:40:22.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:22 smithi203 ceph-mon[32237]: 4.17 scrub starts 2024-09-18T07:40:22.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:22 smithi203 ceph-mon[32237]: 4.17 scrub ok 2024-09-18T07:40:22.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:22 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:40:22.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:22 smithi203 ceph-mon[32237]: from='client.35654 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:40:22.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:22 smithi203 ceph-mon[32237]: osdmap e310: 8 total, 8 up, 8 in 2024-09-18T07:40:22.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:22 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/551401646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:22.528 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:22.560 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:22.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:22.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:22.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:22.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:22.698 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:22.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:22.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:22.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:22.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:22.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:22.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:22.969 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:22.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:22.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:23.071 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:23.081 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 48 16 83874 83858 109.166 77.125 0.00308293 0.00913968 2024-09-18T07:40:23.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:23.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:23.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:23.198 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:23.242 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:23.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:23.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:23.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T07:40:23.344 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:23.351 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:23 smithi137 ceph-mon[29408]: pgmap v508: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 19 MiB/s wr, 311 op/s 2024-09-18T07:40:23.352 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:23 smithi137 ceph-mon[39175]: pgmap v508: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 19 MiB/s wr, 311 op/s 2024-09-18T07:40:23.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:23.393 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:23.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:23.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:23 smithi203 ceph-mon[32237]: pgmap v508: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 19 MiB/s wr, 311 op/s 2024-09-18T07:40:23.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:23.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:23.539 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:23.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:23.640 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:23.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:23.675 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:23.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:23.777 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:23.813 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:23.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:23.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:23.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:23.948 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:23.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:23.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:24.050 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:24.081 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 49 16 85217 85201 108.651 83.9375 0.0231939 0.00919922 2024-09-18T07:40:24.083 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:24.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:24.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:24.185 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:24.218 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:24.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:24.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:24.253 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:24 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:40:24.253 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:24 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:40:24.253 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:24 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:40:24.254 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:24 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:40:24.291 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:24.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:24.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:24.367 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:24.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:24.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:24 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:40:24.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:24 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:40:24.469 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:24.506 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:24.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:24.607 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:24.636 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:24.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:24.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:24.668 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:40:24.668 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:40:24.668 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-18T07:40:24.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-18T07:40:24.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:24.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:24.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:24.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:24.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:24.933 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:24.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:24.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:24.941 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:40:24.941 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:40:24.941 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:40:24.941 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:40:24.941 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:40:25.035 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:25.075 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:25.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:25.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:25.081 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 50 15 86472 86457 108.048 78.5 0.016807 0.00925038 2024-09-18T07:40:25.118 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:25 smithi137 ceph-mon[29408]: pgmap v509: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 733 B/s rd, 57 MiB/s wr, 907 op/s 2024-09-18T07:40:25.118 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:25 smithi137 ceph-mon[29408]: osdmap e311: 8 total, 8 up, 8 in 2024-09-18T07:40:25.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:25.206 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:25.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:25.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:25.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:25.347 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:25.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:25.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:25 smithi137 ceph-mon[39175]: pgmap v509: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 733 B/s rd, 57 MiB/s wr, 907 op/s 2024-09-18T07:40:25.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:25 smithi137 ceph-mon[39175]: osdmap e311: 8 total, 8 up, 8 in 2024-09-18T07:40:25.449 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:25 smithi203 ceph-mon[32237]: pgmap v509: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 733 B/s rd, 57 MiB/s wr, 907 op/s 2024-09-18T07:40:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:25 smithi203 ceph-mon[32237]: osdmap e311: 8 total, 8 up, 8 in 2024-09-18T07:40:25.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:25.481 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:25.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:25.582 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:25.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:25.620 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:25.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:25.721 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:25.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:25.766 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:25.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:25.867 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:25.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:25.903 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:25.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:26.004 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:26.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:40:26.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:26.041 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:26.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:26.082 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 51 16 87762 87746 107.508 80.5625 0.0120206 0.00929674 2024-09-18T07:40:26.143 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:26.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:26.183 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:26.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:26.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:40:26.285 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:26.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:26.346 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:26.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:26.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:26.500 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:26.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:26.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:26.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:26.641 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:26.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:26.743 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:26.783 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:26.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:26.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:26.839 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T07:40:26.885 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:26.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:26.914 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:26.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:26.927 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:26.956 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:27.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:27.066 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:27.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:27.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:27.082 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 52 16 88924 88908 106.836 72.625 0.00968074 0.00935518 2024-09-18T07:40:27.168 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:27.205 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:27.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:27.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:27.260 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:27.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:27.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:27.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:27.380 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:27 smithi137 ceph-mon[29408]: pgmap v511: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 682 B/s rd, 53 MiB/s wr, 844 op/s 2024-09-18T07:40:27.380 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:27 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:40:27.380 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:27 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:40:27.380 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:40:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:40:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:40:27.380 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:27 smithi137 ceph-mon[39175]: pgmap v511: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 682 B/s rd, 53 MiB/s wr, 844 op/s 2024-09-18T07:40:27.380 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:27 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:40:27.381 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:27 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:40:27.398 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:27 smithi203 ceph-mon[32237]: pgmap v511: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 682 B/s rd, 53 MiB/s wr, 844 op/s 2024-09-18T07:40:27.398 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:27 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:40:27.399 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:27 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:40:27.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:27.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:27.497 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:27.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:27.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:27.633 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:27.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:27.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:27.710 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:40:27 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:40:27.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:27.778 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:27.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:27.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:27.856 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-18T07:40:27.856 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-18T07:40:27.856 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-18T07:40:27.856 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-18T07:40:27.857 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-18T07:40:27.857 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-18T07:40:27.857 INFO:teuthology.orchestra.run.smithi137.stderr:bench [] [] [] [] 2024-09-18T07:40:27.857 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-18T07:40:27.857 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-18T07:40:27.857 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-18T07:40:27.857 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator dump block 2024-09-18T07:40:27.857 INFO:teuthology.orchestra.run.smithi137.stderr:Error EINVAL: invalid command 2024-09-18T07:40:27.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:27.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:27.925 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:27.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:28.026 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:28.052 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:28.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:28.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:28.082 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 53 16 90200 90184 106.325 79.75 0.00842233 0.00940005 2024-09-18T07:40:28.154 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:28.190 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:28.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:28.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:28.329 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:28.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:28.430 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:28.468 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:28.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:28.570 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:28.609 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:28.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:28.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:28.711 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:28.714 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T07:40:28.715 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T07:40:28.715 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-18T07:40:28.747 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:28.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:28.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:28.849 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:28.902 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:28.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:28.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:29.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:29.082 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 54 16 91462 91446 105.817 78.875 0.0399422 0.00944481 2024-09-18T07:40:29.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:29.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:29.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:29 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:29.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:29 smithi137 ceph-mon[29408]: pgmap v512: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-18T07:40:29.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:29 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:29.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:29 smithi137 ceph-mon[39175]: pgmap v512: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-18T07:40:29.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:29.237 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:29.241 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:40:29.241 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:40:29.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:29.262 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:29.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:29.302 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:40:29.303 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:40:29.335 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:40:29.335 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:40:29.363 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:29.373 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:40:29.374 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:40:29.406 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:40:29.406 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:40:29.411 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:29.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:29.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:29.444 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:40:29.444 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:40:29.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:29 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:29 smithi203 ceph-mon[32237]: pgmap v512: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-18T07:40:29.477 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:40:29.477 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:40:29.508 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:40:29.508 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:40:29.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:29.566 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:29.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:29.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:29.668 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:29.698 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:29.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:29.800 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:29.838 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:29.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:29.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:29.959 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:29.990 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:40:29.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:29.992 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:29.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:30.082 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 55 16 92503 92487 105.076 65.0625 0.0110454 0.00951215 2024-09-18T07:40:30.093 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:30.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:30.127 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:30.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:30.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:30.267 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:40:30.268 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:40:30.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:30.269 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:30.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:30.370 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:30.409 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:30.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:30.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:40:30.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:40:30.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/276067684' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:30.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[29408]: osdmap e312: 8 total, 8 up, 8 in 2024-09-18T07:40:30.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2585952543' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:30.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[29408]: from='client.35669 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:30.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[29408]: pgmap v514: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 511 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-09-18T07:40:30.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:40:30.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[29408]: from='client.35669 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:40:30.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[29408]: osdmap e313: 8 total, 8 up, 8 in 2024-09-18T07:40:30.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:40:30.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:40:30.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/276067684' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:30.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[39175]: osdmap e312: 8 total, 8 up, 8 in 2024-09-18T07:40:30.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2585952543' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:30.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[39175]: from='client.35669 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:30.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[39175]: pgmap v514: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 511 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-09-18T07:40:30.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:40:30.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[39175]: from='client.35669 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:40:30.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:30 smithi137 ceph-mon[39175]: osdmap e313: 8 total, 8 up, 8 in 2024-09-18T07:40:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:30 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:40:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:30 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:40:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:30 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/276067684' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:30 smithi203 ceph-mon[32237]: osdmap e312: 8 total, 8 up, 8 in 2024-09-18T07:40:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:30 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2585952543' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:30 smithi203 ceph-mon[32237]: from='client.35669 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:30 smithi203 ceph-mon[32237]: pgmap v514: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 511 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-09-18T07:40:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:30 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:40:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:30 smithi203 ceph-mon[32237]: from='client.35669 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:40:30.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:30 smithi203 ceph-mon[32237]: osdmap e313: 8 total, 8 up, 8 in 2024-09-18T07:40:30.510 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:30.546 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:30.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:30.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:30.649 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:30.679 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:30.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:30.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:30.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:30.820 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:30.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:30.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:30.922 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:30.954 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:30.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:31.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:31.082 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 56 16 93730 93714 104.569 76.6875 0.0201156 0.0095581 2024-09-18T07:40:31.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:31.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:31.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:31.192 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:31.227 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:31.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:31.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:31.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:31.355 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:31.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:31.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:31.457 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:31.496 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:31.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:31.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:31.598 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:31.643 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:31.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:31.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:31.745 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:31.781 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:31.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:31.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:31.883 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:31.904 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:31.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:31.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:32.005 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:32.040 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:32.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:32.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:32.082 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 57 15 94868 94853 103.983 71.1875 0.017588 0.00961261 2024-09-18T07:40:32.142 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:32.175 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:32.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:32.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:32.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:32.319 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:32.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:32.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:32.473 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:32.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:32.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:32.575 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:32.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:32.614 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:32.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:32.715 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:32.750 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:32.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:32.851 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:32.881 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:32.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:32.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:32 smithi203 ceph-mon[32237]: pgmap v516: 158 pgs: 158 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-18T07:40:32.981 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:32 smithi137 ceph-mon[29408]: pgmap v516: 158 pgs: 158 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-18T07:40:32.981 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:32 smithi137 ceph-mon[39175]: pgmap v516: 158 pgs: 158 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-09-18T07:40:32.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:33.023 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:33.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:33.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:33.083 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 58 16 96208 96192 103.633 83.6875 0.0087504 0.00964441 2024-09-18T07:40:33.125 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:33.161 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:33.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:33.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:33.186 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-18T07:40:33.186 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-18T07:40:33.186 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-18T07:40:33.186 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-18T07:40:33.186 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-18T07:40:33.186 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-18T07:40:33.186 INFO:teuthology.orchestra.run.smithi137.stderr:bench [] [] [] [] 2024-09-18T07:40:33.186 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-18T07:40:33.187 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-18T07:40:33.187 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-18T07:40:33.187 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator dump block 2024-09-18T07:40:33.187 INFO:teuthology.orchestra.run.smithi137.stderr:Error EINVAL: invalid command 2024-09-18T07:40:33.263 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:33.298 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:33.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:33.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:33.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:33.435 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:33.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:33.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:33.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:33.571 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:33.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:33.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:33.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:33.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:33.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:33.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:33.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:33.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:33.851 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:33.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:33.865 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:33 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:40:33.865 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:33 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:40:33.866 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:33 smithi137 ceph-mon[39175]: osdmap e314: 8 total, 8 up, 8 in 2024-09-18T07:40:33.866 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:33 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:40:33.866 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:33 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:40:33.866 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:33 smithi137 ceph-mon[29408]: osdmap e314: 8 total, 8 up, 8 in 2024-09-18T07:40:33.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T07:40:33.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:33.988 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:33.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:33.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:34.083 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 59 16 97577 97561 103.327 85.5625 0.0100983 0.00967313 2024-09-18T07:40:34.089 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:34.119 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:34.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:34.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:34.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:33 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:40:34.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:33 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:40:34.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:33 smithi203 ceph-mon[32237]: osdmap e314: 8 total, 8 up, 8 in 2024-09-18T07:40:34.220 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:34.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:34.263 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:34.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:34.364 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:34.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:34.407 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:34.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:34.490 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:40:34.508 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:34.540 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:34.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:34.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T07:40:34.580 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T07:40:34.580 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-18T07:40:34.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:34.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:34.673 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:34.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:34.759 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:34.807 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:34.838 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:34.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:34.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:34.899 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:34 smithi137 ceph-mon[29408]: pgmap v518: 158 pgs: 158 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 1k op/s 2024-09-18T07:40:34.903 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:34 smithi137 ceph-mon[39175]: pgmap v518: 158 pgs: 158 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 1k op/s 2024-09-18T07:40:34.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:40:34.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:35.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:35.001 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:35.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:35.084 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:40:35.081518+0000 min lat: 0.00110395 max lat: 1.0234 avg lat: 0.00971863 2024-09-18T07:40:35.085 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:40:35.085 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 60 16 98766 98750 102.843 74.3125 0.0228893 0.00971863 2024-09-18T07:40:35.102 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:35.134 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:35.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:35.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:35.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:34 smithi203 ceph-mon[32237]: pgmap v518: 158 pgs: 158 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 1k op/s 2024-09-18T07:40:35.236 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:35.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:35.286 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:35.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:35.387 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:35.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:35.420 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:35.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:35.483 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:40:35.483 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:40:35.521 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:35.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:35.555 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:35.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:35.656 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:35.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:35.694 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:35.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:35.757 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:35.795 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:35.840 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:35.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:35.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:35.941 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:35.975 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:35.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:35.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:36.077 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:36.083 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 61 16 100026 100010 102.448 78.75 0.00795722 0.00975697 2024-09-18T07:40:36.115 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:36.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:36.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:36.216 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:36.217 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:35 smithi137 ceph-mon[29408]: osdmap e315: 8 total, 8 up, 8 in 2024-09-18T07:40:36.218 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:35 smithi137 ceph-mon[39175]: osdmap e315: 8 total, 8 up, 8 in 2024-09-18T07:40:36.263 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:36.265 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:36.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T07:40:36.293 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:36.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:36.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:36.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:36.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:36.442 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:36.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:36.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:35 smithi203 ceph-mon[32237]: osdmap e315: 8 total, 8 up, 8 in 2024-09-18T07:40:36.543 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:36.573 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:36.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:36.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:36.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:36.717 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:36.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:36.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:36.821 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:36.853 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:36.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:36.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:36.955 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:36.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:36.993 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:36.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:37.083 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 62 16 101263 101247 102.043 77.3125 0.0155609 0.00979521 2024-09-18T07:40:37.094 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:37.103 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:36 smithi137 ceph-mon[29408]: pgmap v520: 158 pgs: 158 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 1k op/s 2024-09-18T07:40:37.103 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:36 smithi137 ceph-mon[39175]: pgmap v520: 158 pgs: 158 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 1k op/s 2024-09-18T07:40:37.128 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:37.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:37.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:37.230 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:37.270 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:37.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:37.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:37.318 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-18T07:40:37.318 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-18T07:40:37.318 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-18T07:40:37.318 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-18T07:40:37.318 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-18T07:40:37.318 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-18T07:40:37.318 INFO:teuthology.orchestra.run.smithi137.stderr:bench [] [] [] [] 2024-09-18T07:40:37.319 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-18T07:40:37.319 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-18T07:40:37.319 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-18T07:40:37.319 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator dump block 2024-09-18T07:40:37.319 INFO:teuthology.orchestra.run.smithi137.stderr:Error EINVAL: invalid command 2024-09-18T07:40:37.372 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:37.409 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:36 smithi203 ceph-mon[32237]: pgmap v520: 158 pgs: 158 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 1k op/s 2024-09-18T07:40:37.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:40:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:40:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:40:37.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:37.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:37.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:37.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:37.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:37.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:37.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:37.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:37.685 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:37.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:37.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:37.711 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:40:37 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:40:37.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:37.818 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:37.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:37.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:37.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:37.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:37.953 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:37.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:37.995 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:40:38.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:38.083 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 63 16 102412 102396 101.563 71.8125 0.00833589 0.00984174 2024-09-18T07:40:38.087 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:38.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:38.189 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:38.224 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:38.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:38.326 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:38.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:38.361 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:38.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:38.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:37 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2658572730' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:40:38.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[39175]: from='client.25762 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:40:38.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:37 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2658572730' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:40:38.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[29408]: from='client.25762 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:40:38.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:37 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2658572730' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:40:38.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:37 smithi203 ceph-mon[32237]: from='client.25762 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:40:38.462 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:38.495 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:38.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:38.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:38.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:38.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:38.638 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:38.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:38.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:38.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:38.759 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:38.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:38.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:38.894 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:38.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:38.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:38.996 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:39.031 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:39.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:39.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:39.083 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 64 16 103619 103603 101.154 75.4375 0.00984159 0.00988146 2024-09-18T07:40:39.133 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:39.164 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:39.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:39.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:39.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:39.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:39.309 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:39.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:39.410 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:39.415 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:39.415 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[29408]: pgmap v521: 158 pgs: 158 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 100 MiB/s wr, 1.61k op/s 2024-09-18T07:40:39.416 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:40:39.416 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[29408]: from='client.25762 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:40:39.416 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[29408]: osdmap e316: 8 total, 8 up, 8 in 2024-09-18T07:40:39.416 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[29408]: 1.7 scrub starts 2024-09-18T07:40:39.416 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[29408]: 1.7 scrub ok 2024-09-18T07:40:39.417 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[29408]: 6.12 scrub starts 2024-09-18T07:40:39.417 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[29408]: 6.12 scrub ok 2024-09-18T07:40:39.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:39.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[39175]: pgmap v521: 158 pgs: 158 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 100 MiB/s wr, 1.61k op/s 2024-09-18T07:40:39.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:40:39.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[39175]: from='client.25762 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:40:39.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[39175]: osdmap e316: 8 total, 8 up, 8 in 2024-09-18T07:40:39.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[39175]: 1.7 scrub starts 2024-09-18T07:40:39.417 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[39175]: 1.7 scrub ok 2024-09-18T07:40:39.418 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[39175]: 6.12 scrub starts 2024-09-18T07:40:39.418 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:38 smithi137 ceph-mon[39175]: 6.12 scrub ok 2024-09-18T07:40:39.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:39.452 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:39.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:39.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:38 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:39.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:38 smithi203 ceph-mon[32237]: pgmap v521: 158 pgs: 158 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 100 MiB/s wr, 1.61k op/s 2024-09-18T07:40:39.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:38 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:40:39.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:38 smithi203 ceph-mon[32237]: from='client.25762 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:40:39.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:38 smithi203 ceph-mon[32237]: osdmap e316: 8 total, 8 up, 8 in 2024-09-18T07:40:39.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:38 smithi203 ceph-mon[32237]: 1.7 scrub starts 2024-09-18T07:40:39.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:38 smithi203 ceph-mon[32237]: 1.7 scrub ok 2024-09-18T07:40:39.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:38 smithi203 ceph-mon[32237]: 6.12 scrub starts 2024-09-18T07:40:39.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:38 smithi203 ceph-mon[32237]: 6.12 scrub ok 2024-09-18T07:40:39.553 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:39.589 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:39.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:39.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:39.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:39.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:39.727 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:39.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:39.828 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:39.859 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:39.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:39.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:40.001 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:40.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:40.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:40.053 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T07:40:40.054 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T07:40:40.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-18T07:40:40.083 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 65 16 104894 104878 100.824 79.6875 0.011729 0.00991367 2024-09-18T07:40:40.103 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:40.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:40.150 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:40.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:40.248 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:40 smithi137 ceph-mon[29408]: 5.1b scrub starts 2024-09-18T07:40:40.248 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:40 smithi137 ceph-mon[29408]: 5.1b scrub ok 2024-09-18T07:40:40.248 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:40 smithi137 ceph-mon[29408]: 3.0 scrub starts 2024-09-18T07:40:40.248 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:40 smithi137 ceph-mon[29408]: 3.0 scrub ok 2024-09-18T07:40:40.249 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:40 smithi137 ceph-mon[29408]: 9.1 scrub starts 2024-09-18T07:40:40.249 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:40 smithi137 ceph-mon[29408]: 5.4 scrub starts 2024-09-18T07:40:40.249 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:40 smithi137 ceph-mon[29408]: 5.4 scrub ok 2024-09-18T07:40:40.249 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:40 smithi137 ceph-mon[29408]: osdmap e317: 8 total, 8 up, 8 in 2024-09-18T07:40:40.249 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:39 smithi137 ceph-mon[39175]: 5.1b scrub starts 2024-09-18T07:40:40.249 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:39 smithi137 ceph-mon[39175]: 5.1b scrub ok 2024-09-18T07:40:40.249 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:39 smithi137 ceph-mon[39175]: 3.0 scrub starts 2024-09-18T07:40:40.249 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:39 smithi137 ceph-mon[39175]: 3.0 scrub ok 2024-09-18T07:40:40.250 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:39 smithi137 ceph-mon[39175]: 9.1 scrub starts 2024-09-18T07:40:40.250 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:40 smithi137 ceph-mon[39175]: 5.4 scrub starts 2024-09-18T07:40:40.250 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:40 smithi137 ceph-mon[39175]: 5.4 scrub ok 2024-09-18T07:40:40.250 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:40 smithi137 ceph-mon[39175]: osdmap e317: 8 total, 8 up, 8 in 2024-09-18T07:40:40.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:40.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:40.291 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:40.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:40.301 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:40:40.301 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:40:40.332 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:40:40.333 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:40:40.366 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:40:40.367 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:40:40.392 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:40.401 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:40:40.401 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:40:40.430 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:40:40.430 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:40:40.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:40.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:40.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:40.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:39 smithi203 ceph-mon[32237]: 5.1b scrub starts 2024-09-18T07:40:40.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:39 smithi203 ceph-mon[32237]: 5.1b scrub ok 2024-09-18T07:40:40.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:39 smithi203 ceph-mon[32237]: 3.0 scrub starts 2024-09-18T07:40:40.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:39 smithi203 ceph-mon[32237]: 3.0 scrub ok 2024-09-18T07:40:40.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:39 smithi203 ceph-mon[32237]: 9.1 scrub starts 2024-09-18T07:40:40.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:39 smithi203 ceph-mon[32237]: 5.4 scrub starts 2024-09-18T07:40:40.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:39 smithi203 ceph-mon[32237]: 5.4 scrub ok 2024-09-18T07:40:40.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:39 smithi203 ceph-mon[32237]: osdmap e317: 8 total, 8 up, 8 in 2024-09-18T07:40:40.465 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:40:40.466 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:40:40.499 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:40:40.499 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:40:40.535 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:40.537 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:40:40.537 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:40:40.569 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:40.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:40.670 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:40.705 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:40.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:40.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:40.807 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:40.849 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:40.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:40.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:40.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:40.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:40.985 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:40.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:41.084 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 66 16 106080 106064 100.42 74.125 0.00911679 0.0099538 2024-09-18T07:40:41.086 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:41.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:41.121 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:41.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:41.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:41.257 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:41.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:41.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:41.299 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[39175]: pgmap v523: 158 pgs: 158 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 738 B/s rd, 57 MiB/s wr, 912 op/s 2024-09-18T07:40:41.299 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[39175]: 3.12 scrub starts 2024-09-18T07:40:41.299 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[39175]: 3.12 scrub ok 2024-09-18T07:40:41.299 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[39175]: 5.11 scrub starts 2024-09-18T07:40:41.299 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[39175]: 5.11 scrub ok 2024-09-18T07:40:41.299 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[39175]: 3.14 scrub starts 2024-09-18T07:40:41.300 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[39175]: 3.14 scrub ok 2024-09-18T07:40:41.300 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2021100139' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:41.300 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[39175]: 6.8 scrub starts 2024-09-18T07:40:41.300 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[39175]: 6.8 scrub ok 2024-09-18T07:40:41.300 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[29408]: pgmap v523: 158 pgs: 158 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 738 B/s rd, 57 MiB/s wr, 912 op/s 2024-09-18T07:40:41.300 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[29408]: 3.12 scrub starts 2024-09-18T07:40:41.300 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[29408]: 3.12 scrub ok 2024-09-18T07:40:41.300 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[29408]: 5.11 scrub starts 2024-09-18T07:40:41.300 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[29408]: 5.11 scrub ok 2024-09-18T07:40:41.301 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[29408]: 3.14 scrub starts 2024-09-18T07:40:41.301 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[29408]: 3.14 scrub ok 2024-09-18T07:40:41.301 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2021100139' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:41.301 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[29408]: 6.8 scrub starts 2024-09-18T07:40:41.301 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:41 smithi137 ceph-mon[29408]: 6.8 scrub ok 2024-09-18T07:40:41.358 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:41.393 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:41.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:41 smithi203 ceph-mon[32237]: pgmap v523: 158 pgs: 158 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 738 B/s rd, 57 MiB/s wr, 912 op/s 2024-09-18T07:40:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:41 smithi203 ceph-mon[32237]: 3.12 scrub starts 2024-09-18T07:40:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:41 smithi203 ceph-mon[32237]: 3.12 scrub ok 2024-09-18T07:40:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:41 smithi203 ceph-mon[32237]: 5.11 scrub starts 2024-09-18T07:40:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:41 smithi203 ceph-mon[32237]: 5.11 scrub ok 2024-09-18T07:40:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:41 smithi203 ceph-mon[32237]: 3.14 scrub starts 2024-09-18T07:40:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:41 smithi203 ceph-mon[32237]: 3.14 scrub ok 2024-09-18T07:40:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:41 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2021100139' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:41 smithi203 ceph-mon[32237]: 6.8 scrub starts 2024-09-18T07:40:41.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:41 smithi203 ceph-mon[32237]: 6.8 scrub ok 2024-09-18T07:40:41.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:41.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:41.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:41.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:41.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:41.671 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:41.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:41.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:41.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:41.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:41.810 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:41.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:41.911 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:41.944 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:41.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:41.956 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:41.972 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:42.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:42.084 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 67 16 107317 107301 100.074 77.3125 0.0117067 0.00998748 2024-09-18T07:40:42.087 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:42.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:42.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:42.189 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:42.229 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:42.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:42.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:42.290 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[29408]: 5.15 scrub starts 2024-09-18T07:40:42.291 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[29408]: 5.15 scrub ok 2024-09-18T07:40:42.291 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:40:42.291 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:40:42.291 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[29408]: 6.16 scrub starts 2024-09-18T07:40:42.291 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[29408]: 6.16 scrub ok 2024-09-18T07:40:42.291 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[29408]: 9.1 scrub ok 2024-09-18T07:40:42.291 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[29408]: osdmap e318: 8 total, 8 up, 8 in 2024-09-18T07:40:42.291 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[29408]: 9.4 scrub starts 2024-09-18T07:40:42.291 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[29408]: 5.1a scrub starts 2024-09-18T07:40:42.291 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[29408]: 5.1a scrub ok 2024-09-18T07:40:42.291 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:40:42.292 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[39175]: 5.15 scrub starts 2024-09-18T07:40:42.292 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[39175]: 5.15 scrub ok 2024-09-18T07:40:42.292 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:40:42.292 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:40:42.292 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[39175]: 6.16 scrub starts 2024-09-18T07:40:42.292 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[39175]: 6.16 scrub ok 2024-09-18T07:40:42.292 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[39175]: 9.1 scrub ok 2024-09-18T07:40:42.292 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[39175]: osdmap e318: 8 total, 8 up, 8 in 2024-09-18T07:40:42.292 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[39175]: 9.4 scrub starts 2024-09-18T07:40:42.292 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[39175]: 5.1a scrub starts 2024-09-18T07:40:42.292 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[39175]: 5.1a scrub ok 2024-09-18T07:40:42.293 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:42 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:40:42.331 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:42.363 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:42.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:42.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:42 smithi203 ceph-mon[32237]: 5.15 scrub starts 2024-09-18T07:40:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:42 smithi203 ceph-mon[32237]: 5.15 scrub ok 2024-09-18T07:40:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:42 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:40:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:42 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:40:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:42 smithi203 ceph-mon[32237]: 6.16 scrub starts 2024-09-18T07:40:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:42 smithi203 ceph-mon[32237]: 6.16 scrub ok 2024-09-18T07:40:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:42 smithi203 ceph-mon[32237]: 9.1 scrub ok 2024-09-18T07:40:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:42 smithi203 ceph-mon[32237]: osdmap e318: 8 total, 8 up, 8 in 2024-09-18T07:40:42.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:42 smithi203 ceph-mon[32237]: 9.4 scrub starts 2024-09-18T07:40:42.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:42 smithi203 ceph-mon[32237]: 5.1a scrub starts 2024-09-18T07:40:42.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:42 smithi203 ceph-mon[32237]: 5.1a scrub ok 2024-09-18T07:40:42.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:42 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:40:42.464 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:42.503 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:42.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:42.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:42.604 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:42.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:42.638 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:42.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:42.710 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:40:42.710 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:40:42.710 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:40:42.710 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:40:42.710 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:40:42.738 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:42.774 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:42.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:42.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:42.876 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:42.910 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:42.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:42.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:43.012 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:43.047 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:43.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:43.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:43.084 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 68 15 108555 108540 99.7413 77.4375 0.0103208 0.0100219 2024-09-18T07:40:43.106 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:43 smithi137 ceph-mon[29408]: 1.a scrub starts 2024-09-18T07:40:43.106 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:43 smithi137 ceph-mon[29408]: 1.a scrub ok 2024-09-18T07:40:43.106 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:43 smithi137 ceph-mon[29408]: pgmap v526: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-18T07:40:43.106 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:43 smithi137 ceph-mon[29408]: 6.9 scrub starts 2024-09-18T07:40:43.106 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:43 smithi137 ceph-mon[29408]: 6.9 scrub ok 2024-09-18T07:40:43.106 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:43 smithi137 ceph-mon[29408]: 1.0 scrub starts 2024-09-18T07:40:43.106 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:43 smithi137 ceph-mon[29408]: 1.0 scrub ok 2024-09-18T07:40:43.106 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:43 smithi137 ceph-mon[39175]: 1.a scrub starts 2024-09-18T07:40:43.106 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:43 smithi137 ceph-mon[39175]: 1.a scrub ok 2024-09-18T07:40:43.107 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:43 smithi137 ceph-mon[39175]: pgmap v526: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-18T07:40:43.107 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:43 smithi137 ceph-mon[39175]: 6.9 scrub starts 2024-09-18T07:40:43.107 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:43 smithi137 ceph-mon[39175]: 6.9 scrub ok 2024-09-18T07:40:43.107 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:43 smithi137 ceph-mon[39175]: 1.0 scrub starts 2024-09-18T07:40:43.107 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:43 smithi137 ceph-mon[39175]: 1.0 scrub ok 2024-09-18T07:40:43.149 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:43.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:43.181 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:43.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:43.282 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:43.305 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-18T07:40:43.305 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-18T07:40:43.306 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-18T07:40:43.306 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-18T07:40:43.306 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-18T07:40:43.306 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-18T07:40:43.306 INFO:teuthology.orchestra.run.smithi137.stderr:bench [] [] [] [] 2024-09-18T07:40:43.306 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-18T07:40:43.306 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-18T07:40:43.306 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-18T07:40:43.306 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator dump block 2024-09-18T07:40:43.306 INFO:teuthology.orchestra.run.smithi137.stderr:Error EINVAL: invalid command 2024-09-18T07:40:43.318 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:43.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:43.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:43.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:43.454 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:43.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:43.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:43.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:43 smithi203 ceph-mon[32237]: 1.a scrub starts 2024-09-18T07:40:43.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:43 smithi203 ceph-mon[32237]: 1.a scrub ok 2024-09-18T07:40:43.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:43 smithi203 ceph-mon[32237]: pgmap v526: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-18T07:40:43.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:43 smithi203 ceph-mon[32237]: 6.9 scrub starts 2024-09-18T07:40:43.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:43 smithi203 ceph-mon[32237]: 6.9 scrub ok 2024-09-18T07:40:43.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:43 smithi203 ceph-mon[32237]: 1.0 scrub starts 2024-09-18T07:40:43.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:43 smithi203 ceph-mon[32237]: 1.0 scrub ok 2024-09-18T07:40:43.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:43.589 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:43.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:43.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:43.691 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:43.727 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:43.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:43.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:43.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:40:43.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:43.861 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:43.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:43.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:43.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T07:40:43.942 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:40:43.962 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:44.055 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:44.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:44.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:44.087 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 69 16 109768 109752 99.3893 75.75 0.0185739 0.0100555 2024-09-18T07:40:44.157 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:44.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:44.214 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:44.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:44.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:44.359 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:44.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:44.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:44.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:44 smithi137 ceph-mon[29408]: 5.1f scrub starts 2024-09-18T07:40:44.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:44 smithi137 ceph-mon[29408]: 5.1f scrub ok 2024-09-18T07:40:44.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:44 smithi137 ceph-mon[29408]: 9.4 scrub ok 2024-09-18T07:40:44.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:44 smithi137 ceph-mon[29408]: 3.e scrub starts 2024-09-18T07:40:44.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:44 smithi137 ceph-mon[29408]: 3.e scrub ok 2024-09-18T07:40:44.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:44 smithi137 ceph-mon[39175]: 5.1f scrub starts 2024-09-18T07:40:44.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:44 smithi137 ceph-mon[39175]: 5.1f scrub ok 2024-09-18T07:40:44.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:44 smithi137 ceph-mon[39175]: 9.4 scrub ok 2024-09-18T07:40:44.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:44 smithi137 ceph-mon[39175]: 3.e scrub starts 2024-09-18T07:40:44.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:44 smithi137 ceph-mon[39175]: 3.e scrub ok 2024-09-18T07:40:44.460 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:44.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:44 smithi203 ceph-mon[32237]: 5.1f scrub starts 2024-09-18T07:40:44.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:44 smithi203 ceph-mon[32237]: 5.1f scrub ok 2024-09-18T07:40:44.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:44 smithi203 ceph-mon[32237]: 9.4 scrub ok 2024-09-18T07:40:44.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:44 smithi203 ceph-mon[32237]: 3.e scrub starts 2024-09-18T07:40:44.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:44 smithi203 ceph-mon[32237]: 3.e scrub ok 2024-09-18T07:40:44.500 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:44.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:44.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:44.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:44.640 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:44.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:44.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:44.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:44.774 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:44.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:44.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:44.876 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:44.908 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:44.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:44.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:45.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:45.048 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:45.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:45.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:45.087 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 70 15 110949 110934 99.0248 73.875 0.0277252 0.0100936 2024-09-18T07:40:45.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:45.192 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:45.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:45.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:45.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:45.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:45.336 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:45.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[29408]: 1.1 scrub starts 2024-09-18T07:40:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[29408]: 1.1 scrub ok 2024-09-18T07:40:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[29408]: pgmap v527: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-18T07:40:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[29408]: 9.b scrub starts 2024-09-18T07:40:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[29408]: 6.2 scrub starts 2024-09-18T07:40:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[29408]: 6.2 scrub ok 2024-09-18T07:40:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[29408]: 1.2 scrub starts 2024-09-18T07:40:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[29408]: 1.2 scrub ok 2024-09-18T07:40:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[29408]: osdmap e319: 8 total, 8 up, 8 in 2024-09-18T07:40:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[39175]: 1.1 scrub starts 2024-09-18T07:40:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[39175]: 1.1 scrub ok 2024-09-18T07:40:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[39175]: pgmap v527: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-18T07:40:45.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[39175]: 9.b scrub starts 2024-09-18T07:40:45.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[39175]: 6.2 scrub starts 2024-09-18T07:40:45.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[39175]: 6.2 scrub ok 2024-09-18T07:40:45.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[39175]: 1.2 scrub starts 2024-09-18T07:40:45.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[39175]: 1.2 scrub ok 2024-09-18T07:40:45.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:45 smithi137 ceph-mon[39175]: osdmap e319: 8 total, 8 up, 8 in 2024-09-18T07:40:45.437 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:45 smithi203 ceph-mon[32237]: 1.1 scrub starts 2024-09-18T07:40:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:45 smithi203 ceph-mon[32237]: 1.1 scrub ok 2024-09-18T07:40:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:45 smithi203 ceph-mon[32237]: pgmap v527: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-18T07:40:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:45 smithi203 ceph-mon[32237]: 9.b scrub starts 2024-09-18T07:40:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:45 smithi203 ceph-mon[32237]: 6.2 scrub starts 2024-09-18T07:40:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:45 smithi203 ceph-mon[32237]: 6.2 scrub ok 2024-09-18T07:40:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:45 smithi203 ceph-mon[32237]: 1.2 scrub starts 2024-09-18T07:40:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:45 smithi203 ceph-mon[32237]: 1.2 scrub ok 2024-09-18T07:40:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:45 smithi203 ceph-mon[32237]: osdmap e319: 8 total, 8 up, 8 in 2024-09-18T07:40:45.469 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:45.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:45.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:45.508 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T07:40:45.508 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T07:40:45.509 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-18T07:40:45.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:45.642 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:45.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:45.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:45.666 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:45.736 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:45.744 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:45.748 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:45.820 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:45.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:45.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:45.922 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:45.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:45.964 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:45.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:46.065 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:46.087 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 71 16 111995 111979 98.5498 65.3125 0.025373 0.010141 2024-09-18T07:40:46.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:46.101 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:46.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:46.202 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:46.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:46.241 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:46.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:46.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:46.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:46.383 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:46.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:46.403 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:46 smithi137 ceph-mon[39175]: 9.d scrub starts 2024-09-18T07:40:46.403 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:46 smithi137 ceph-mon[29408]: 9.d scrub starts 2024-09-18T07:40:46.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:46 smithi203 ceph-mon[32237]: 9.d scrub starts 2024-09-18T07:40:46.484 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:46.523 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:46.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:46.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:46.572 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:40:46.572 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:40:46.572 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:40:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:40:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135011 7.14 7.3991ba34 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-18T07:40:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:15.993779+0000", 2024-09-18T07:40:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 390.573612432, 2024-09-18T07:40:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98326573299999998, 2024-09-18T07:40:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:40:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:40:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:46.573 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135011 2024-09-18T07:40:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993779+0000", 2024-09-18T07:40:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993779+0000", 2024-09-18T07:40:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.575 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.575 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993780+0000", 2024-09-18T07:40:46.575 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6899999999999996e-07 2024-09-18T07:40:46.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.575 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.575 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.575 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993784+0000", 2024-09-18T07:40:46.575 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6270000000000002e-06 2024-09-18T07:40:46.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.575 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.575 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.576 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993794+0000", 2024-09-18T07:40:46.576 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6889999999999999e-06 2024-09-18T07:40:46.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.576 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.576 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993799+0000", 2024-09-18T07:40:46.576 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1590000000000001e-06 2024-09-18T07:40:46.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.576 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.576 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993861+0000", 2024-09-18T07:40:46.576 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2421000000000004e-05 2024-09-18T07:40:46.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.577 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.577 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993869+0000", 2024-09-18T07:40:46.577 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1980000000000001e-06 2024-09-18T07:40:46.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.577 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.577 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.977033+0000", 2024-09-18T07:40:46.577 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98316372699999999 2024-09-18T07:40:46.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.578 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.578 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.977045+0000", 2024-09-18T07:40:46.578 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2043e-05 2024-09-18T07:40:46.578 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.578 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.578 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.578 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135018 7.10 7:0de93d52:::benchmark_data_smithi137_74180_object30602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-18T07:40:46.578 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:15.998577+0000", 2024-09-18T07:40:46.578 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 390.56881466499999, 2024-09-18T07:40:46.578 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1980523299999999, 2024-09-18T07:40:46.578 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.579 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.579 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:46.579 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.579 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:40:46.579 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:46.579 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135018 2024-09-18T07:40:46.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.579 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.579 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.579 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998577+0000", 2024-09-18T07:40:46.579 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.580 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998577+0000", 2024-09-18T07:40:46.580 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.580 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998579+0000", 2024-09-18T07:40:46.580 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.336e-06 2024-09-18T07:40:46.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998583+0000", 2024-09-18T07:40:46.581 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7190000000000001e-06 2024-09-18T07:40:46.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.581 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998593+0000", 2024-09-18T07:40:46.581 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4450000000000002e-06 2024-09-18T07:40:46.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.581 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998598+0000", 2024-09-18T07:40:46.581 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2329999999999997e-06 2024-09-18T07:40:46.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.582 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.582 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998649+0000", 2024-09-18T07:40:46.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0735999999999997e-05 2024-09-18T07:40:46.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.582 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.582 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998659+0000", 2024-09-18T07:40:46.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0156000000000001e-05 2024-09-18T07:40:46.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.582 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.970426+0000", 2024-09-18T07:40:46.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97176680299999996 2024-09-18T07:40:46.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.583 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.972639+0000", 2024-09-18T07:40:46.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022131070000000002 2024-09-18T07:40:46.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.583 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.183802+0000", 2024-09-18T07:40:46.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2111635359999999 2024-09-18T07:40:46.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.584 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:40:46.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.183932+0000", 2024-09-18T07:40:46.584 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000129686 2024-09-18T07:40:46.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.584 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2,5", 2024-09-18T07:40:46.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184136+0000", 2024-09-18T07:40:46.584 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000203885 2024-09-18T07:40:46.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.584 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:40:46.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185714+0000", 2024-09-18T07:40:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00157839 2024-09-18T07:40:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.188451+0000", 2024-09-18T07:40:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002736741 2024-09-18T07:40:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.188856+0000", 2024-09-18T07:40:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040506400000000001 2024-09-18T07:40:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.195963+0000", 2024-09-18T07:40:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0071068019999999997 2024-09-18T07:40:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:40:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.196200+0000", 2024-09-18T07:40:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023698600000000001 2024-09-18T07:40:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.196630+0000", 2024-09-18T07:40:46.586 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00042970500000000001 2024-09-18T07:40:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162368 7.2 7.85717ac2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:40:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.437764+0000", 2024-09-18T07:40:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 344.129628083, 2024-09-18T07:40:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97823369100000002, 2024-09-18T07:40:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:40:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.587 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:40:46.588 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:46.588 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162368 2024-09-18T07:40:46.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437764+0000", 2024-09-18T07:40:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437764+0000", 2024-09-18T07:40:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.590 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.590 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437766+0000", 2024-09-18T07:40:46.590 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8479999999999999e-06 2024-09-18T07:40:46.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.590 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.590 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437771+0000", 2024-09-18T07:40:46.590 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4580000000000001e-06 2024-09-18T07:40:46.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.590 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.590 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437781+0000", 2024-09-18T07:40:46.590 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0291e-05 2024-09-18T07:40:46.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.591 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.591 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437789+0000", 2024-09-18T07:40:46.591 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3599999999999998e-06 2024-09-18T07:40:46.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.591 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.591 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437856+0000", 2024-09-18T07:40:46.591 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7241000000000002e-05 2024-09-18T07:40:46.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.591 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.591 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437903+0000", 2024-09-18T07:40:46.592 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7348999999999997e-05 2024-09-18T07:40:46.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.592 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.592 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415935+0000", 2024-09-18T07:40:46.592 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97803116499999998 2024-09-18T07:40:46.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.592 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.592 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415997+0000", 2024-09-18T07:40:46.592 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2978999999999995e-05 2024-09-18T07:40:46.592 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.592 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.593 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.593 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162379 7.12 7.b17b62b2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:40:46.593 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.446092+0000", 2024-09-18T07:40:46.593 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 344.12129994200001, 2024-09-18T07:40:46.593 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96914699900000001, 2024-09-18T07:40:46.593 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.593 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.593 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:40:46.593 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.593 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:40:46.594 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:46.594 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162379 2024-09-18T07:40:46.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.594 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.594 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.594 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446092+0000", 2024-09-18T07:40:46.594 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.594 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.594 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446092+0000", 2024-09-18T07:40:46.594 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446095+0000", 2024-09-18T07:40:46.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7329999999999999e-06 2024-09-18T07:40:46.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446102+0000", 2024-09-18T07:40:46.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9110000000000004e-06 2024-09-18T07:40:46.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446118+0000", 2024-09-18T07:40:46.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6875999999999999e-05 2024-09-18T07:40:46.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.596 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446126+0000", 2024-09-18T07:40:46.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7030000000000006e-06 2024-09-18T07:40:46.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.596 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446193+0000", 2024-09-18T07:40:46.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6919000000000001e-05 2024-09-18T07:40:46.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.597 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446203+0000", 2024-09-18T07:40:46.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6439999999999995e-06 2024-09-18T07:40:46.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.597 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415214+0000", 2024-09-18T07:40:46.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96901095599999998 2024-09-18T07:40:46.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.597 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415239+0000", 2024-09-18T07:40:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5256999999999999e-05 2024-09-18T07:40:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162393 7.12 7.f00c1e52 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:40:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.460966+0000", 2024-09-18T07:40:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 344.10642623199999, 2024-09-18T07:40:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95474156300000002, 2024-09-18T07:40:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.598 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:40:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:40:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162393 2024-09-18T07:40:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.460966+0000", 2024-09-18T07:40:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.460966+0000", 2024-09-18T07:40:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.460968+0000", 2024-09-18T07:40:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9870000000000002e-06 2024-09-18T07:40:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.460973+0000", 2024-09-18T07:40:46.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5890000000000002e-06 2024-09-18T07:40:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.460997+0000", 2024-09-18T07:40:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3583000000000001e-05 2024-09-18T07:40:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.461004+0000", 2024-09-18T07:40:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0840000000000003e-06 2024-09-18T07:40:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.461064+0000", 2024-09-18T07:40:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9701000000000003e-05 2024-09-18T07:40:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.461074+0000", 2024-09-18T07:40:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9240000000000002e-06 2024-09-18T07:40:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415699+0000", 2024-09-18T07:40:46.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95462499899999997 2024-09-18T07:40:46.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415707+0000", 2024-09-18T07:40:46.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6959999999999994e-06 2024-09-18T07:40:46.603 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.603 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.603 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.603 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162407 7.2 7.5f39ee22 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:40:46.603 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.503104+0000", 2024-09-18T07:40:46.603 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 344.06428819600001, 2024-09-18T07:40:46.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91293045900000003, 2024-09-18T07:40:46.604 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.604 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.604 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:40:46.604 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.604 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:40:46.604 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:46.604 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162407 2024-09-18T07:40:46.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.604 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.604 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503104+0000", 2024-09-18T07:40:46.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.605 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503104+0000", 2024-09-18T07:40:46.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.605 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503105+0000", 2024-09-18T07:40:46.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3200000000000001e-06 2024-09-18T07:40:46.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.606 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.606 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503110+0000", 2024-09-18T07:40:46.606 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7709999999999997e-06 2024-09-18T07:40:46.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.606 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.606 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.606 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503120+0000", 2024-09-18T07:40:46.606 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0217000000000001e-05 2024-09-18T07:40:46.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.606 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.606 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.606 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503127+0000", 2024-09-18T07:40:46.606 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9430000000000003e-06 2024-09-18T07:40:46.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.607 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.607 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503227+0000", 2024-09-18T07:40:46.607 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100147 2024-09-18T07:40:46.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.607 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.607 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503235+0000", 2024-09-18T07:40:46.607 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3270000000000001e-06 2024-09-18T07:40:46.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.607 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.608 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416027+0000", 2024-09-18T07:40:46.608 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91279167400000005 2024-09-18T07:40:46.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.608 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.608 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416034+0000", 2024-09-18T07:40:46.608 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0600000000000002e-06 2024-09-18T07:40:46.608 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.608 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.608 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.608 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162429 7.12 7.8e24a052 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:40:46.609 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.531777+0000", 2024-09-18T07:40:46.609 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 344.03561509899998, 2024-09-18T07:40:46.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.883964953, 2024-09-18T07:40:46.609 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.609 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.609 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:40:46.609 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.609 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:40:46.609 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:46.609 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162429 2024-09-18T07:40:46.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.609 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531777+0000", 2024-09-18T07:40:46.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531777+0000", 2024-09-18T07:40:46.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531778+0000", 2024-09-18T07:40:46.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7120000000000001e-06 2024-09-18T07:40:46.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.611 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.611 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531783+0000", 2024-09-18T07:40:46.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.622e-06 2024-09-18T07:40:46.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.611 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.611 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531792+0000", 2024-09-18T07:40:46.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1889999999999996e-06 2024-09-18T07:40:46.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.611 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.612 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531798+0000", 2024-09-18T07:40:46.612 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9599999999999997e-06 2024-09-18T07:40:46.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.612 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.612 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.533462+0000", 2024-09-18T07:40:46.612 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001663351 2024-09-18T07:40:46.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.612 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.612 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.533471+0000", 2024-09-18T07:40:46.612 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5270000000000004e-06 2024-09-18T07:40:46.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415736+0000", 2024-09-18T07:40:46.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.88226443600000004 2024-09-18T07:40:46.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415742+0000", 2024-09-18T07:40:46.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.156e-06 2024-09-18T07:40:46.613 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.613 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.613 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.614 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162435 7.2 7.90a4f302 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:40:46.614 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.536705+0000", 2024-09-18T07:40:46.614 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 344.03068705099997, 2024-09-18T07:40:46.614 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.87935532000000005, 2024-09-18T07:40:46.614 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.614 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.614 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:40:46.614 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.614 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:40:46.614 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:46.614 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162435 2024-09-18T07:40:46.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.615 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536705+0000", 2024-09-18T07:40:46.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536705+0000", 2024-09-18T07:40:46.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.616 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536707+0000", 2024-09-18T07:40:46.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7940000000000001e-06 2024-09-18T07:40:46.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.616 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.616 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536712+0000", 2024-09-18T07:40:46.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9790000000000001e-06 2024-09-18T07:40:46.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.616 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.616 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536725+0000", 2024-09-18T07:40:46.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2986999999999999e-05 2024-09-18T07:40:46.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.617 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.617 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536732+0000", 2024-09-18T07:40:46.617 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6340000000000001e-06 2024-09-18T07:40:46.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.617 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.617 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536789+0000", 2024-09-18T07:40:46.617 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6675999999999997e-05 2024-09-18T07:40:46.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.617 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.617 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536798+0000", 2024-09-18T07:40:46.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5100000000000004e-06 2024-09-18T07:40:46.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416054+0000", 2024-09-18T07:40:46.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.87925583100000004 2024-09-18T07:40:46.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416060+0000", 2024-09-18T07:40:46.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9089999999999997e-06 2024-09-18T07:40:46.618 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.618 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.618 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.620 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162461 7.2 7.86b206e2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:40:46.620 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.562650+0000", 2024-09-18T07:40:46.620 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 344.00474191299998, 2024-09-18T07:40:46.620 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85345243000000004, 2024-09-18T07:40:46.620 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.620 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.620 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:40:46.620 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.620 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:40:46.620 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:46.620 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162461 2024-09-18T07:40:46.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.621 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562650+0000", 2024-09-18T07:40:46.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562650+0000", 2024-09-18T07:40:46.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562651+0000", 2024-09-18T07:40:46.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.246e-06 2024-09-18T07:40:46.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562656+0000", 2024-09-18T07:40:46.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3189999999999996e-06 2024-09-18T07:40:46.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562667+0000", 2024-09-18T07:40:46.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1623e-05 2024-09-18T07:40:46.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562675+0000", 2024-09-18T07:40:46.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5000000000000002e-06 2024-09-18T07:40:46.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562871+0000", 2024-09-18T07:40:46.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000196219 2024-09-18T07:40:46.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562939+0000", 2024-09-18T07:40:46.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8196999999999998e-05 2024-09-18T07:40:46.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416088+0000", 2024-09-18T07:40:46.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85314855700000003 2024-09-18T07:40:46.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416102+0000", 2024-09-18T07:40:46.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4769e-05 2024-09-18T07:40:46.625 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.625 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.625 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.625 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162466 7.12 7.e244e932 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:40:46.625 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.563197+0000", 2024-09-18T07:40:46.625 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 344.00419485800001, 2024-09-18T07:40:46.625 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85257313899999998, 2024-09-18T07:40:46.625 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.625 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.625 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:40:46.626 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.626 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:40:46.626 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:46.626 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162466 2024-09-18T07:40:46.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.626 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.626 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563197+0000", 2024-09-18T07:40:46.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563197+0000", 2024-09-18T07:40:46.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563198+0000", 2024-09-18T07:40:46.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3340000000000001e-06 2024-09-18T07:40:46.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563203+0000", 2024-09-18T07:40:46.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7110000000000003e-06 2024-09-18T07:40:46.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563217+0000", 2024-09-18T07:40:46.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4314999999999999e-05 2024-09-18T07:40:46.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563225+0000", 2024-09-18T07:40:46.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1420000000000004e-06 2024-09-18T07:40:46.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563294+0000", 2024-09-18T07:40:46.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9309999999999999e-05 2024-09-18T07:40:46.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563303+0000", 2024-09-18T07:40:46.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3659999999999998e-06 2024-09-18T07:40:46.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415764+0000", 2024-09-18T07:40:46.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85246111000000002 2024-09-18T07:40:46.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.630 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.630 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415770+0000", 2024-09-18T07:40:46.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8510000000000004e-06 2024-09-18T07:40:46.630 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.630 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.630 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.630 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162470 7.12 7.9c7c55b2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:40:46.630 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.566190+0000", 2024-09-18T07:40:46.630 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 344.00120187200002, 2024-09-18T07:40:46.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.84961990099999996, 2024-09-18T07:40:46.630 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.631 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.631 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:40:46.631 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.631 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:40:46.631 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:46.631 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162470 2024-09-18T07:40:46.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.631 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566190+0000", 2024-09-18T07:40:46.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566190+0000", 2024-09-18T07:40:46.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566192+0000", 2024-09-18T07:40:46.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.762e-06 2024-09-18T07:40:46.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566198+0000", 2024-09-18T07:40:46.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9499999999999998e-06 2024-09-18T07:40:46.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566211+0000", 2024-09-18T07:40:46.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3743999999999999e-05 2024-09-18T07:40:46.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566219+0000", 2024-09-18T07:40:46.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7810000000000005e-06 2024-09-18T07:40:46.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566282+0000", 2024-09-18T07:40:46.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2362999999999999e-05 2024-09-18T07:40:46.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566290+0000", 2024-09-18T07:40:46.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7749999999999998e-06 2024-09-18T07:40:46.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415803+0000", 2024-09-18T07:40:46.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.84951286599999998 2024-09-18T07:40:46.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415810+0000", 2024-09-18T07:40:46.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6599999999999998e-06 2024-09-18T07:40:46.635 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.635 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.635 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.636 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162495 7.12 7.468e4332 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:40:46.636 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.583195+0000", 2024-09-18T07:40:46.636 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 343.98419713499999, 2024-09-18T07:40:46.636 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.83264318599999998, 2024-09-18T07:40:46.636 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.636 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.636 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:40:46.636 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.636 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:40:46.636 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:46.636 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162495 2024-09-18T07:40:46.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.637 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583195+0000", 2024-09-18T07:40:46.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583195+0000", 2024-09-18T07:40:46.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583196+0000", 2024-09-18T07:40:46.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6819999999999999e-06 2024-09-18T07:40:46.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583201+0000", 2024-09-18T07:40:46.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0209999999999999e-06 2024-09-18T07:40:46.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583211+0000", 2024-09-18T07:40:46.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3610000000000007e-06 2024-09-18T07:40:46.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583218+0000", 2024-09-18T07:40:46.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8560000000000001e-06 2024-09-18T07:40:46.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583268+0000", 2024-09-18T07:40:46.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0294000000000003e-05 2024-09-18T07:40:46.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583277+0000", 2024-09-18T07:40:46.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2839999999999995e-06 2024-09-18T07:40:46.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415831+0000", 2024-09-18T07:40:46.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.83255423799999995 2024-09-18T07:40:46.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415838+0000", 2024-09-18T07:40:46.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4500000000000001e-06 2024-09-18T07:40:46.640 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.641 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.641 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.641 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162501 7.12 7.5f5c6f12 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:40:46.641 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.586226+0000", 2024-09-18T07:40:46.641 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 343.98116607999998, 2024-09-18T07:40:46.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.82964114600000005, 2024-09-18T07:40:46.641 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.641 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.641 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:40:46.641 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.641 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:40:46.641 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:46.642 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162501 2024-09-18T07:40:46.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.642 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.642 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.642 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586226+0000", 2024-09-18T07:40:46.642 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.642 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.642 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586226+0000", 2024-09-18T07:40:46.642 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586227+0000", 2024-09-18T07:40:46.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.598e-06 2024-09-18T07:40:46.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586233+0000", 2024-09-18T07:40:46.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5280000000000003e-06 2024-09-18T07:40:46.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586246+0000", 2024-09-18T07:40:46.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3123e-05 2024-09-18T07:40:46.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586254+0000", 2024-09-18T07:40:46.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3469999999999999e-06 2024-09-18T07:40:46.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586344+0000", 2024-09-18T07:40:46.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9617999999999995e-05 2024-09-18T07:40:46.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586354+0000", 2024-09-18T07:40:46.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8339999999999994e-06 2024-09-18T07:40:46.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415860+0000", 2024-09-18T07:40:46.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.82950663999999996 2024-09-18T07:40:46.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415867+0000", 2024-09-18T07:40:46.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4579999999999998e-06 2024-09-18T07:40:46.645 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.646 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.646 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.646 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162524 7.2 7.5cbbb662 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:40:46.646 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.609728+0000", 2024-09-18T07:40:46.646 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 343.95766417099998, 2024-09-18T07:40:46.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.80640156299999999, 2024-09-18T07:40:46.646 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.646 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.646 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:40:46.646 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.646 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:40:46.647 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:46.647 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162524 2024-09-18T07:40:46.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.647 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609728+0000", 2024-09-18T07:40:46.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609728+0000", 2024-09-18T07:40:46.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609730+0000", 2024-09-18T07:40:46.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.894e-06 2024-09-18T07:40:46.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609741+0000", 2024-09-18T07:40:46.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1466e-05 2024-09-18T07:40:46.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609755+0000", 2024-09-18T07:40:46.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3584000000000001e-05 2024-09-18T07:40:46.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609762+0000", 2024-09-18T07:40:46.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7950000000000008e-06 2024-09-18T07:40:46.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609832+0000", 2024-09-18T07:40:46.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0029000000000003e-05 2024-09-18T07:40:46.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609844+0000", 2024-09-18T07:40:46.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1154e-05 2024-09-18T07:40:46.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416123+0000", 2024-09-18T07:40:46.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.80627929200000004 2024-09-18T07:40:46.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416129+0000", 2024-09-18T07:40:46.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3489999999999997e-06 2024-09-18T07:40:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162529 7.12 7.8e054bd2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:40:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.612942+0000", 2024-09-18T07:40:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 343.95445016100001, 2024-09-18T07:40:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.80297076000000001, 2024-09-18T07:40:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:40:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:40:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162529 2024-09-18T07:40:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612942+0000", 2024-09-18T07:40:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612942+0000", 2024-09-18T07:40:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612943+0000", 2024-09-18T07:40:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.53e-06 2024-09-18T07:40:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612948+0000", 2024-09-18T07:40:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.006e-06 2024-09-18T07:40:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612959+0000", 2024-09-18T07:40:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.029e-05 2024-09-18T07:40:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612966+0000", 2024-09-18T07:40:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9459999999999998e-06 2024-09-18T07:40:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.613042+0000", 2024-09-18T07:40:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5358999999999995e-05 2024-09-18T07:40:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.613062+0000", 2024-09-18T07:40:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9783999999999999e-05 2024-09-18T07:40:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415904+0000", 2024-09-18T07:40:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.80284268700000005 2024-09-18T07:40:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415912+0000", 2024-09-18T07:40:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1580000000000005e-06 2024-09-18T07:40:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162535 7.12 7.1c115152 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:40:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.618377+0000", 2024-09-18T07:40:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 343.94901511299997, 2024-09-18T07:40:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.79781339699999998, 2024-09-18T07:40:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:40:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:40:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162535 2024-09-18T07:40:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618377+0000", 2024-09-18T07:40:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618377+0000", 2024-09-18T07:40:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618379+0000", 2024-09-18T07:40:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8509999999999999e-06 2024-09-18T07:40:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618384+0000", 2024-09-18T07:40:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9989999999999999e-06 2024-09-18T07:40:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618394+0000", 2024-09-18T07:40:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0045e-05 2024-09-18T07:40:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618403+0000", 2024-09-18T07:40:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8699999999999998e-06 2024-09-18T07:40:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618468+0000", 2024-09-18T07:40:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5273999999999999e-05 2024-09-18T07:40:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618483+0000", 2024-09-18T07:40:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5141e-05 2024-09-18T07:40:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416187+0000", 2024-09-18T07:40:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.79770383899999997 2024-09-18T07:40:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416190+0000", 2024-09-18T07:40:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3780000000000001e-06 2024-09-18T07:40:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109537 8.7 8:e50487d6:::benchmark_data_smithi137_101942_object4727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:40:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:40:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 184.51098569600001, 2024-09-18T07:40:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.185266763, 2024-09-18T07:40:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:40:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:40:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109537 2024-09-18T07:40:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:40:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:40:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056408+0000", 2024-09-18T07:40:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.562e-06 2024-09-18T07:40:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056415+0000", 2024-09-18T07:40:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9280000000000004e-06 2024-09-18T07:40:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056425+0000", 2024-09-18T07:40:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0220999999999999e-05 2024-09-18T07:40:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056434+0000", 2024-09-18T07:40:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7499999999999992e-06 2024-09-18T07:40:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.061190+0000", 2024-09-18T07:40:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047560620000000001 2024-09-18T07:40:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.061238+0000", 2024-09-18T07:40:46.665 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8730000000000003e-05 2024-09-18T07:40:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229082+0000", 2024-09-18T07:40:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.167843878 2024-09-18T07:40:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:40:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229243+0000", 2024-09-18T07:40:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000160426 2024-09-18T07:40:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:40:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229373+0000", 2024-09-18T07:40:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012994999999999999 2024-09-18T07:40:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236041+0000", 2024-09-18T07:40:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066680960000000001 2024-09-18T07:40:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:40:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236051+0000", 2024-09-18T07:40:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8309999999999996e-06 2024-09-18T07:40:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.240107+0000", 2024-09-18T07:40:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040564570000000003 2024-09-18T07:40:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:40:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.240152+0000", 2024-09-18T07:40:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4617000000000003e-05 2024-09-18T07:40:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.241673+0000", 2024-09-18T07:40:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001521255 2024-09-18T07:40:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109540 8.7 8:e7454470:::benchmark_data_smithi137_101942_object4730:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:40:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:40:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 184.501513706, 2024-09-18T07:40:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.186834677, 2024-09-18T07:40:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:40:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:40:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109540 2024-09-18T07:40:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:40:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:40:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065880+0000", 2024-09-18T07:40:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8279999999999999e-06 2024-09-18T07:40:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065886+0000", 2024-09-18T07:40:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.117e-06 2024-09-18T07:40:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065899+0000", 2024-09-18T07:40:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2809999999999999e-05 2024-09-18T07:40:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065904+0000", 2024-09-18T07:40:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0350000000000001e-06 2024-09-18T07:40:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.066028+0000", 2024-09-18T07:40:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012424199999999999 2024-09-18T07:40:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.066039+0000", 2024-09-18T07:40:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0441999999999999e-05 2024-09-18T07:40:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.233503+0000", 2024-09-18T07:40:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.167464622 2024-09-18T07:40:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:40:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.233624+0000", 2024-09-18T07:40:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012107 2024-09-18T07:40:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:40:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.233749+0000", 2024-09-18T07:40:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000124418 2024-09-18T07:40:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236078+0000", 2024-09-18T07:40:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002328746 2024-09-18T07:40:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:40:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.239968+0000", 2024-09-18T07:40:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038901220000000002 2024-09-18T07:40:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.250395+0000", 2024-09-18T07:40:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010427532999999999 2024-09-18T07:40:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:40:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.250537+0000", 2024-09-18T07:40:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014218100000000001 2024-09-18T07:40:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.252713+0000", 2024-09-18T07:40:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0021755110000000002 2024-09-18T07:40:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109545 8.7 8:ecb41cf4:::benchmark_data_smithi137_101942_object4735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:40:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:40:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 184.49310892, 2024-09-18T07:40:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1804624180000001, 2024-09-18T07:40:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:40:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:40:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109545 2024-09-18T07:40:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:40:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:40:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074285+0000", 2024-09-18T07:40:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1169999999999998e-06 2024-09-18T07:40:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074292+0000", 2024-09-18T07:40:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.153e-06 2024-09-18T07:40:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074305+0000", 2024-09-18T07:40:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2363e-05 2024-09-18T07:40:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074312+0000", 2024-09-18T07:40:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3409999999999997e-06 2024-09-18T07:40:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074508+0000", 2024-09-18T07:40:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000196443 2024-09-18T07:40:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074519+0000", 2024-09-18T07:40:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0159999999999999e-05 2024-09-18T07:40:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236329+0000", 2024-09-18T07:40:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1618104599999999 2024-09-18T07:40:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:40:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236416+0000", 2024-09-18T07:40:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6670999999999997e-05 2024-09-18T07:40:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:40:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236495+0000", 2024-09-18T07:40:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9256000000000001e-05 2024-09-18T07:40:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:40:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.241684+0000", 2024-09-18T07:40:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051890720000000003 2024-09-18T07:40:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.243812+0000", 2024-09-18T07:40:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002127912 2024-09-18T07:40:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.252772+0000", 2024-09-18T07:40:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.008960562 2024-09-18T07:40:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:40:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.252827+0000", 2024-09-18T07:40:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4500000000000003e-05 2024-09-18T07:40:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.254745+0000", 2024-09-18T07:40:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019184079999999999 2024-09-18T07:40:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109567 8.7 8:e247f5cd:::benchmark_data_smithi137_101942_object4757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:40:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:40:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 184.46400975700001, 2024-09-18T07:40:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.153246698, 2024-09-18T07:40:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:40:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:40:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:40:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:40:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:40:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109567 2024-09-18T07:40:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:40:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:40:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:40:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:40:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:40:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:40:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:40:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103384+0000", 2024-09-18T07:40:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.792e-06 2024-09-18T07:40:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:40:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103390+0000", 2024-09-18T07:40:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7139999999999998e-06 2024-09-18T07:40:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:40:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103401+0000", 2024-09-18T07:40:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1559000000000001e-05 2024-09-18T07:40:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:40:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103408+0000", 2024-09-18T07:40:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2110000000000001e-06 2024-09-18T07:40:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103464+0000", 2024-09-18T07:40:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6032999999999997e-05 2024-09-18T07:40:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:40:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103475+0000", 2024-09-18T07:40:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0380999999999999e-05 2024-09-18T07:40:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:40:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.238189+0000", 2024-09-18T07:40:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.134713723 2024-09-18T07:40:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:40:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.238319+0000", 2024-09-18T07:40:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013080700000000001 2024-09-18T07:40:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:40:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.238431+0000", 2024-09-18T07:40:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112126 2024-09-18T07:40:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:40:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.243830+0000", 2024-09-18T07:40:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053980390000000003 2024-09-18T07:40:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.243889+0000", 2024-09-18T07:40:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9024000000000003e-05 2024-09-18T07:40:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.254832+0000", 2024-09-18T07:40:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0109436 2024-09-18T07:40:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:40:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.255025+0000", 2024-09-18T07:40:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000192926 2024-09-18T07:40:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:40:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:40:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:40:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.256629+0000", 2024-09-18T07:40:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001603763 2024-09-18T07:40:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:40:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:40:46.692 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:40:46.705 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:46.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:46.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:46.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:46.845 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:46.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:46.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:46.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:46.979 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:46.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:46.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:47.081 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:47.087 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 72 16 113096 113080 98.1368 68.8125 0.0155977 0.0101838 2024-09-18T07:40:47.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:47.123 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:47.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:47.224 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:47.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:47.260 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:47.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:47.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:47.401 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:47.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:47.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:47.409 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:40:47.409 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:40:47.409 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:40:47.410 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:40:47.410 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:40:47.410 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:40:47.410 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:40:47.410 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:40:47.410 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:40:47.410 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:40:47.410 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:40:47.411 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:40:47.411 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:40:47.411 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:40:47.411 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:40:47.411 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:40:47.411 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:40:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:40:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:40:47.412 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:47 smithi137 ceph-mon[29408]: 3.c scrub starts 2024-09-18T07:40:47.412 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:47 smithi137 ceph-mon[29408]: 3.c scrub ok 2024-09-18T07:40:47.412 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:47 smithi137 ceph-mon[29408]: pgmap v529: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-18T07:40:47.412 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:47 smithi137 ceph-mon[29408]: 9.0 scrub starts 2024-09-18T07:40:47.412 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:47 smithi137 ceph-mon[29408]: 4.1a scrub starts 2024-09-18T07:40:47.412 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:47 smithi137 ceph-mon[29408]: 4.1a scrub ok 2024-09-18T07:40:47.412 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:47 smithi137 ceph-mon[29408]: 9.b scrub ok 2024-09-18T07:40:47.412 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:47 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1560589470' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:47.412 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:47 smithi137 ceph-mon[39175]: 3.c scrub starts 2024-09-18T07:40:47.412 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:47 smithi137 ceph-mon[39175]: 3.c scrub ok 2024-09-18T07:40:47.413 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:47 smithi137 ceph-mon[39175]: pgmap v529: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-18T07:40:47.413 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:47 smithi137 ceph-mon[39175]: 9.0 scrub starts 2024-09-18T07:40:47.413 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:47 smithi137 ceph-mon[39175]: 4.1a scrub starts 2024-09-18T07:40:47.413 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:47 smithi137 ceph-mon[39175]: 4.1a scrub ok 2024-09-18T07:40:47.413 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:47 smithi137 ceph-mon[39175]: 9.b scrub ok 2024-09-18T07:40:47.413 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:47 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1560589470' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:47.419 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:47 smithi203 ceph-mon[32237]: 3.c scrub starts 2024-09-18T07:40:47.419 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:47 smithi203 ceph-mon[32237]: 3.c scrub ok 2024-09-18T07:40:47.419 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:47 smithi203 ceph-mon[32237]: pgmap v529: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-18T07:40:47.420 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:47 smithi203 ceph-mon[32237]: 9.0 scrub starts 2024-09-18T07:40:47.420 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:47 smithi203 ceph-mon[32237]: 4.1a scrub starts 2024-09-18T07:40:47.420 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:47 smithi203 ceph-mon[32237]: 4.1a scrub ok 2024-09-18T07:40:47.420 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:47 smithi203 ceph-mon[32237]: 9.b scrub ok 2024-09-18T07:40:47.420 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:47 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1560589470' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:47.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:47.536 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:47.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:47.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:47.638 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:47.677 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:47.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:47.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:47.711 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:40:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:40:47.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:47.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:47.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:47.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:47.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:47.953 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:47.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:48.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:48.088 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 73 16 114337 114321 97.8548 77.5625 0.0139582 0.0102146 2024-09-18T07:40:48.088 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:48.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:48.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:48.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:48.221 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:48.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:48.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:48.301 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:48 smithi137 ceph-mon[29408]: 3.a scrub starts 2024-09-18T07:40:48.301 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:48 smithi137 ceph-mon[29408]: 3.a scrub ok 2024-09-18T07:40:48.302 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:48 smithi137 ceph-mon[29408]: 6.1a scrub starts 2024-09-18T07:40:48.302 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:48 smithi137 ceph-mon[29408]: 6.1a scrub ok 2024-09-18T07:40:48.302 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:48 smithi137 ceph-mon[29408]: 9.d scrub ok 2024-09-18T07:40:48.302 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:48 smithi137 ceph-mon[29408]: 2.0 scrub starts 2024-09-18T07:40:48.302 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:48 smithi137 ceph-mon[29408]: 2.0 scrub ok 2024-09-18T07:40:48.302 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:48 smithi137 ceph-mon[39175]: 3.a scrub starts 2024-09-18T07:40:48.302 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:48 smithi137 ceph-mon[39175]: 3.a scrub ok 2024-09-18T07:40:48.302 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:48 smithi137 ceph-mon[39175]: 6.1a scrub starts 2024-09-18T07:40:48.302 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:48 smithi137 ceph-mon[39175]: 6.1a scrub ok 2024-09-18T07:40:48.302 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:48 smithi137 ceph-mon[39175]: 9.d scrub ok 2024-09-18T07:40:48.303 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:48 smithi137 ceph-mon[39175]: 2.0 scrub starts 2024-09-18T07:40:48.303 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:48 smithi137 ceph-mon[39175]: 2.0 scrub ok 2024-09-18T07:40:48.322 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:48.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:48.362 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:48.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:48 smithi203 ceph-mon[32237]: 3.a scrub starts 2024-09-18T07:40:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:48 smithi203 ceph-mon[32237]: 3.a scrub ok 2024-09-18T07:40:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:48 smithi203 ceph-mon[32237]: 6.1a scrub starts 2024-09-18T07:40:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:48 smithi203 ceph-mon[32237]: 6.1a scrub ok 2024-09-18T07:40:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:48 smithi203 ceph-mon[32237]: 9.d scrub ok 2024-09-18T07:40:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:48 smithi203 ceph-mon[32237]: 2.0 scrub starts 2024-09-18T07:40:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:48 smithi203 ceph-mon[32237]: 2.0 scrub ok 2024-09-18T07:40:48.463 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:48.498 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:48.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:48.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:48.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:48.644 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:48.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:48.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:48.746 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:48.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:48.784 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:48.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:48.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:48.920 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:48.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:48.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:49.013 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T07:40:49.022 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:49.087 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 74 16 115553 115537 97.5595 76 0.0139445 0.0102444 2024-09-18T07:40:49.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:49.102 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:49.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:49.105 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:40:49.202 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:49.245 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:49.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:49.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:49.347 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:49.378 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:49.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:49.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:49.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:49 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:49.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:49 smithi137 ceph-mon[29408]: pgmap v530: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-09-18T07:40:49.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:49 smithi137 ceph-mon[29408]: 3.b scrub starts 2024-09-18T07:40:49.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:49 smithi137 ceph-mon[29408]: 3.b scrub ok 2024-09-18T07:40:49.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:49 smithi137 ceph-mon[29408]: 9.0 scrub ok 2024-09-18T07:40:49.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:49 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1798840114' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:49.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:49 smithi137 ceph-mon[29408]: from='client.35711 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:49.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:49 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:49.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:49 smithi137 ceph-mon[39175]: pgmap v530: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-09-18T07:40:49.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:49 smithi137 ceph-mon[39175]: 3.b scrub starts 2024-09-18T07:40:49.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:49 smithi137 ceph-mon[39175]: 3.b scrub ok 2024-09-18T07:40:49.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:49 smithi137 ceph-mon[39175]: 9.0 scrub ok 2024-09-18T07:40:49.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:49 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1798840114' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:49.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:49 smithi137 ceph-mon[39175]: from='client.35711 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:49.458 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:40:49.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:49 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:49.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:49 smithi203 ceph-mon[32237]: pgmap v530: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-09-18T07:40:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:49 smithi203 ceph-mon[32237]: 3.b scrub starts 2024-09-18T07:40:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:49 smithi203 ceph-mon[32237]: 3.b scrub ok 2024-09-18T07:40:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:49 smithi203 ceph-mon[32237]: 9.0 scrub ok 2024-09-18T07:40:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:49 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1798840114' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:49 smithi203 ceph-mon[32237]: from='client.35711 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:40:49.480 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:49.507 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:49.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:49.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:49.609 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:49.650 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:49.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:49.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:49.752 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:49.795 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:49.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:49.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:49.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:49.932 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:49.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:49.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:50.034 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:50.066 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:50.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:50.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:50.088 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 75 16 116590 116574 97.1228 64.8125 0.00655302 0.0102868 2024-09-18T07:40:50.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:50.201 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:50.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:50.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:50.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:50 smithi203 ceph-mon[32237]: 9.a scrub starts 2024-09-18T07:40:50.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:50 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:40:50.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:50 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:40:50.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:50 smithi203 ceph-mon[32237]: from='client.35711 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:40:50.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:50 smithi203 ceph-mon[32237]: osdmap e320: 8 total, 8 up, 8 in 2024-09-18T07:40:50.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:50 smithi203 ceph-mon[32237]: 9.f scrub starts 2024-09-18T07:40:50.204 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:50 smithi203 ceph-mon[32237]: osdmap e321: 8 total, 8 up, 8 in 2024-09-18T07:40:50.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:50.333 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:50.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:50.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:50.379 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:40:50.380 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:40:50.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:50 smithi137 ceph-mon[29408]: 9.a scrub starts 2024-09-18T07:40:50.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:50 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:40:50.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:50 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:40:50.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:50 smithi137 ceph-mon[29408]: from='client.35711 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:40:50.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:50 smithi137 ceph-mon[29408]: osdmap e320: 8 total, 8 up, 8 in 2024-09-18T07:40:50.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:50 smithi137 ceph-mon[29408]: 9.f scrub starts 2024-09-18T07:40:50.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:50 smithi137 ceph-mon[29408]: osdmap e321: 8 total, 8 up, 8 in 2024-09-18T07:40:50.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:50 smithi137 ceph-mon[39175]: 9.a scrub starts 2024-09-18T07:40:50.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:50 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:40:50.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:50 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:40:50.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:50 smithi137 ceph-mon[39175]: from='client.35711 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:40:50.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:50 smithi137 ceph-mon[39175]: osdmap e320: 8 total, 8 up, 8 in 2024-09-18T07:40:50.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:50 smithi137 ceph-mon[39175]: 9.f scrub starts 2024-09-18T07:40:50.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:50 smithi137 ceph-mon[39175]: osdmap e321: 8 total, 8 up, 8 in 2024-09-18T07:40:50.435 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:50.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:50.470 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:50.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:50.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:50.603 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:50.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:50.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:50.705 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:50.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:50.742 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:50.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:50.746 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:50.843 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:50.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:50.891 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:50.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:50.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:51.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:51.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:51.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:51.088 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 76 16 117609 117593 96.6829 63.6875 0.0061228 0.0103383 2024-09-18T07:40:51.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:51.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:51.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:51.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:51.259 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:51.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:51.297 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:51.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:51.378 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:51 smithi203 ceph-mon[32237]: pgmap v532: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-18T07:40:51.378 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:51 smithi203 ceph-mon[32237]: 4.c scrub starts 2024-09-18T07:40:51.378 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:51 smithi203 ceph-mon[32237]: 4.c scrub ok 2024-09-18T07:40:51.398 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:51.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:51 smithi137 ceph-mon[29408]: pgmap v532: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-18T07:40:51.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:51 smithi137 ceph-mon[29408]: 4.c scrub starts 2024-09-18T07:40:51.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:51 smithi137 ceph-mon[29408]: 4.c scrub ok 2024-09-18T07:40:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:51 smithi137 ceph-mon[39175]: pgmap v532: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-18T07:40:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:51 smithi137 ceph-mon[39175]: 4.c scrub starts 2024-09-18T07:40:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:51 smithi137 ceph-mon[39175]: 4.c scrub ok 2024-09-18T07:40:51.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:51.431 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:51.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:51.532 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:51.572 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:51.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:51.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:51.674 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:51.713 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:51.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:51.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:51.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:40:51.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:51.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:51.874 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:51.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:51.950 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-18T07:40:51.950 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-18T07:40:51.950 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-18T07:40:51.950 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-18T07:40:51.950 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-18T07:40:51.950 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-18T07:40:51.950 INFO:teuthology.orchestra.run.smithi137.stderr:bench [] [] [] [] 2024-09-18T07:40:51.950 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-18T07:40:51.950 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-18T07:40:51.950 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-18T07:40:51.951 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator dump block 2024-09-18T07:40:51.951 INFO:teuthology.orchestra.run.smithi137.stderr:Error EINVAL: invalid command 2024-09-18T07:40:51.975 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:51.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:51.997 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:51.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:52.088 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 77 15 118581 118566 96.2168 60.8125 0.0404102 0.0103885 2024-09-18T07:40:52.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:52.123 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:52 smithi137 ceph-mon[29408]: 6.19 scrub starts 2024-09-18T07:40:52.134 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:52.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:52.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:52.236 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:52.268 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:52.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:52.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:52.370 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:52.403 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:52.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:52.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:52 smithi137 ceph-mon[29408]: 6.19 scrub ok 2024-09-18T07:40:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:52 smithi137 ceph-mon[29408]: 9.a scrub ok 2024-09-18T07:40:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:52 smithi137 ceph-mon[29408]: 5.3 scrub starts 2024-09-18T07:40:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:52 smithi137 ceph-mon[29408]: 5.3 scrub ok 2024-09-18T07:40:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:52 smithi137 ceph-mon[29408]: 6.15 scrub starts 2024-09-18T07:40:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:52 smithi137 ceph-mon[29408]: 6.15 scrub ok 2024-09-18T07:40:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:52 smithi137 ceph-mon[39175]: 6.19 scrub starts 2024-09-18T07:40:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:52 smithi137 ceph-mon[39175]: 6.19 scrub ok 2024-09-18T07:40:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:52 smithi137 ceph-mon[39175]: 9.a scrub ok 2024-09-18T07:40:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:52 smithi137 ceph-mon[39175]: 5.3 scrub starts 2024-09-18T07:40:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:52 smithi137 ceph-mon[39175]: 5.3 scrub ok 2024-09-18T07:40:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:52 smithi137 ceph-mon[39175]: 6.15 scrub starts 2024-09-18T07:40:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:52 smithi137 ceph-mon[39175]: 6.15 scrub ok 2024-09-18T07:40:52.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:52 smithi203 ceph-mon[32237]: 6.19 scrub starts 2024-09-18T07:40:52.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:52 smithi203 ceph-mon[32237]: 6.19 scrub ok 2024-09-18T07:40:52.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:52 smithi203 ceph-mon[32237]: 9.a scrub ok 2024-09-18T07:40:52.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:52 smithi203 ceph-mon[32237]: 5.3 scrub starts 2024-09-18T07:40:52.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:52 smithi203 ceph-mon[32237]: 5.3 scrub ok 2024-09-18T07:40:52.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:52 smithi203 ceph-mon[32237]: 6.15 scrub starts 2024-09-18T07:40:52.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:52 smithi203 ceph-mon[32237]: 6.15 scrub ok 2024-09-18T07:40:52.506 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:52.537 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:52.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:52.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:52.639 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:52.673 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:52.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:52.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:52.775 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:52.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:52.802 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:52.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:52.853 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T07:40:52.854 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T07:40:52.854 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-18T07:40:52.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:52.911 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:52.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:52.965 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:52.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:53.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:53.089 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 78 16 119529 119513 95.741 59.1875 0.0605858 0.0104355 2024-09-18T07:40:53.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:53.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:53.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:53.191 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:53.223 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:53.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:53.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:53.358 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:40:53.364 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:53.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:53.380 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:53 smithi137 ceph-mon[29408]: pgmap v534: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-09-18T07:40:53.380 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:53 smithi137 ceph-mon[29408]: 9.5 scrub starts 2024-09-18T07:40:53.380 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:53 smithi137 ceph-mon[29408]: 5.c scrub starts 2024-09-18T07:40:53.380 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:53 smithi137 ceph-mon[29408]: 5.c scrub ok 2024-09-18T07:40:53.380 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:53 smithi137 ceph-mon[39175]: pgmap v534: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-09-18T07:40:53.380 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:53 smithi137 ceph-mon[39175]: 9.5 scrub starts 2024-09-18T07:40:53.380 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:53 smithi137 ceph-mon[39175]: 5.c scrub starts 2024-09-18T07:40:53.380 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:53 smithi137 ceph-mon[39175]: 5.c scrub ok 2024-09-18T07:40:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:53 smithi203 ceph-mon[32237]: pgmap v534: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-09-18T07:40:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:53 smithi203 ceph-mon[32237]: 9.5 scrub starts 2024-09-18T07:40:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:53 smithi203 ceph-mon[32237]: 5.c scrub starts 2024-09-18T07:40:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:53 smithi203 ceph-mon[32237]: 5.c scrub ok 2024-09-18T07:40:53.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:53.529 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T07:40:53.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:53.531 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:53.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:53.568 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:40:53.568 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:40:53.569 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:53.618 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:40:53.619 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:40:53.632 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:53.652 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:40:53.652 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:40:53.693 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:40:53.693 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:40:53.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:53.695 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:53.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:53.724 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:40:53.725 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:40:53.754 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:40:53.755 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:40:53.777 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:40:53.777 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:40:53.795 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:40:53.796 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:40:53.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:53.852 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:53.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:53.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:53.900 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:40:53.954 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:53.994 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:53.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:53.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:54.089 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 79 16 120550 120534 95.3369 63.8125 0.0129115 0.0104849 2024-09-18T07:40:54.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:54.134 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:54.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:54.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:54.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:54.269 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:54.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:54.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:54.344 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:54 smithi203 ceph-mon[32237]: 4.15 scrub starts 2024-09-18T07:40:54.344 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:54 smithi203 ceph-mon[32237]: 4.15 scrub ok 2024-09-18T07:40:54.345 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:54 smithi203 ceph-mon[32237]: 9.f scrub ok 2024-09-18T07:40:54.345 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:54 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1279857202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:54.370 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:54.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:54.417 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:54.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:54.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:54 smithi137 ceph-mon[29408]: 4.15 scrub starts 2024-09-18T07:40:54.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:54 smithi137 ceph-mon[29408]: 4.15 scrub ok 2024-09-18T07:40:54.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:54 smithi137 ceph-mon[29408]: 9.f scrub ok 2024-09-18T07:40:54.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:54 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1279857202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:54.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:54 smithi137 ceph-mon[39175]: 4.15 scrub starts 2024-09-18T07:40:54.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:54 smithi137 ceph-mon[39175]: 4.15 scrub ok 2024-09-18T07:40:54.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:54 smithi137 ceph-mon[39175]: 9.f scrub ok 2024-09-18T07:40:54.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:54 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1279857202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:40:54.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:54.524 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:40:54.524 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:40:54.524 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:40:54.524 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:40:54.524 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:40:54.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:54.561 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:54.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:54.662 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:54.670 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-18T07:40:54.670 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-18T07:40:54.670 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-18T07:40:54.670 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-18T07:40:54.670 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-18T07:40:54.671 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-18T07:40:54.671 INFO:teuthology.orchestra.run.smithi137.stderr:bench [] [] [] [] 2024-09-18T07:40:54.671 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-18T07:40:54.671 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-18T07:40:54.671 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-18T07:40:54.671 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator dump block 2024-09-18T07:40:54.672 INFO:teuthology.orchestra.run.smithi137.stderr:Error EINVAL: invalid command 2024-09-18T07:40:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:54.697 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:54.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:54.798 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:54.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:54.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:54.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:54.934 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:54.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:54.972 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:54.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:55.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:55.090 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:40:55.088475+0000 min lat: 0.00110395 max lat: 1.0234 avg lat: 0.0105163 2024-09-18T07:40:55.090 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:40:55.090 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 80 16 121707 121691 95.048 72.3125 0.0157478 0.0105163 2024-09-18T07:40:55.110 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:55.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:55.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:55.213 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:55.240 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T07:40:55.257 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:55.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:55.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:55.359 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:55.380 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:55.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:55.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:55.404 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:55.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:55 smithi137 ceph-mon[29408]: 3.5 scrub starts 2024-09-18T07:40:55.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:55 smithi137 ceph-mon[29408]: 3.5 scrub ok 2024-09-18T07:40:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:55 smithi137 ceph-mon[29408]: pgmap v535: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.42k op/s 2024-09-18T07:40:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:55 smithi137 ceph-mon[29408]: 9.5 scrub ok 2024-09-18T07:40:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:55 smithi137 ceph-mon[29408]: 6.5 scrub starts 2024-09-18T07:40:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:55 smithi137 ceph-mon[29408]: 6.5 scrub ok 2024-09-18T07:40:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:55 smithi137 ceph-mon[39175]: 3.5 scrub starts 2024-09-18T07:40:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:55 smithi137 ceph-mon[39175]: 3.5 scrub ok 2024-09-18T07:40:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:55 smithi137 ceph-mon[39175]: pgmap v535: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.42k op/s 2024-09-18T07:40:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:55 smithi137 ceph-mon[39175]: 9.5 scrub ok 2024-09-18T07:40:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:55 smithi137 ceph-mon[39175]: 6.5 scrub starts 2024-09-18T07:40:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:55 smithi137 ceph-mon[39175]: 6.5 scrub ok 2024-09-18T07:40:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:55 smithi203 ceph-mon[32237]: 3.5 scrub starts 2024-09-18T07:40:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:55 smithi203 ceph-mon[32237]: 3.5 scrub ok 2024-09-18T07:40:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:55 smithi203 ceph-mon[32237]: pgmap v535: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.42k op/s 2024-09-18T07:40:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:55 smithi203 ceph-mon[32237]: 9.5 scrub ok 2024-09-18T07:40:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:55 smithi203 ceph-mon[32237]: 6.5 scrub starts 2024-09-18T07:40:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:55 smithi203 ceph-mon[32237]: 6.5 scrub ok 2024-09-18T07:40:55.483 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:55.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:55.530 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:55.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:55.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:55.664 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:55.665 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:40:55.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:55.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:55.766 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:55.806 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:55.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:55.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:55.935 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:55.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:56.037 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:56.072 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:56.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:56.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:56.090 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 81 16 122692 122676 94.6345 61.5625 0.0141591 0.0105605 2024-09-18T07:40:56.174 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:56.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:56.203 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:56.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:56.304 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:56.340 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:56.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:56.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:56.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:56 smithi137 ceph-mon[29408]: 9.6 scrub starts 2024-09-18T07:40:56.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:56 smithi137 ceph-mon[29408]: 9.3 scrub starts 2024-09-18T07:40:56.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:56 smithi137 ceph-mon[39175]: 9.6 scrub starts 2024-09-18T07:40:56.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:56 smithi137 ceph-mon[39175]: 9.3 scrub starts 2024-09-18T07:40:56.442 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:56.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:56 smithi203 ceph-mon[32237]: 9.6 scrub starts 2024-09-18T07:40:56.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:56 smithi203 ceph-mon[32237]: 9.3 scrub starts 2024-09-18T07:40:56.475 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:56.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:56.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:56.557 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T07:40:56.557 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T07:40:56.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-18T07:40:56.578 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:56.639 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:56.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:56.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:56.741 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:56.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:56.789 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:56.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:56.891 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:56.925 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:56.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:56.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:57.027 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:57.057 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:57.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:57.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:57.090 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 82 16 123698 123682 94.2472 62.875 0.0118548 0.010606 2024-09-18T07:40:57.159 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:57.205 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:57.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:57.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:57.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:40:57.251 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:57.307 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:57 smithi203 ceph-mon[32237]: pgmap v536: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 692 op/s 2024-09-18T07:40:57.307 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:57 smithi203 ceph-mon[32237]: 4.1 scrub starts 2024-09-18T07:40:57.307 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:57 smithi203 ceph-mon[32237]: 4.1 scrub ok 2024-09-18T07:40:57.307 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:57 smithi203 ceph-mon[32237]: 3.d scrub starts 2024-09-18T07:40:57.307 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:57 smithi203 ceph-mon[32237]: 3.d scrub ok 2024-09-18T07:40:57.307 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:57 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:40:57.307 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:57 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:40:57.307 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:57.346 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:57.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:57.389 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:57 smithi137 ceph-mon[29408]: pgmap v536: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 692 op/s 2024-09-18T07:40:57.389 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:57 smithi137 ceph-mon[29408]: 4.1 scrub starts 2024-09-18T07:40:57.390 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:57 smithi137 ceph-mon[29408]: 4.1 scrub ok 2024-09-18T07:40:57.390 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:57 smithi137 ceph-mon[29408]: 3.d scrub starts 2024-09-18T07:40:57.390 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:57 smithi137 ceph-mon[29408]: 3.d scrub ok 2024-09-18T07:40:57.390 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:57 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:40:57.390 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:57 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:40:57.390 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:40:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:40:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:40:57.390 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:57 smithi137 ceph-mon[39175]: pgmap v536: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 692 op/s 2024-09-18T07:40:57.391 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:57 smithi137 ceph-mon[39175]: 4.1 scrub starts 2024-09-18T07:40:57.391 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:57 smithi137 ceph-mon[39175]: 4.1 scrub ok 2024-09-18T07:40:57.391 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:57 smithi137 ceph-mon[39175]: 3.d scrub starts 2024-09-18T07:40:57.391 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:57 smithi137 ceph-mon[39175]: 3.d scrub ok 2024-09-18T07:40:57.391 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:57 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:40:57.391 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:57 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:40:57.449 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:57.472 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:40:57.473 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:40:57.473 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:40:57.473 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:40:57.473 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.473 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:20749 7.3 e88/81 7:c04d08df:::benchmark_data_smithi137_74180_object20748:head v 88'1239, mlcod=88'1237)", 2024-09-18T07:40:57.473 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.530035+0000", 2024-09-18T07:40:57.473 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 513.93800973600003, 2024-09-18T07:40:57.473 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13453020299999999, 2024-09-18T07:40:57.473 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.473 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.473 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.473 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.473 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.474 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.474 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530035+0000", 2024-09-18T07:40:57.474 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.474 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.474 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.474 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.474 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530035+0000", 2024-09-18T07:40:57.474 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.474 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.474 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.474 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.474 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530036+0000", 2024-09-18T07:40:57.474 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3200000000000004e-07 2024-09-18T07:40:57.475 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.475 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.475 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.475 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530106+0000", 2024-09-18T07:40:57.475 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0099999999999996e-05 2024-09-18T07:40:57.475 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.475 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.475 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.475 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530115+0000", 2024-09-18T07:40:57.475 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1419999999999999e-06 2024-09-18T07:40:57.475 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.475 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.476 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.476 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530122+0000", 2024-09-18T07:40:57.476 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3269999999999997e-06 2024-09-18T07:40:57.476 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.476 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.476 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.476 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530167+0000", 2024-09-18T07:40:57.476 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5247999999999998e-05 2024-09-18T07:40:57.476 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.476 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.476 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.476 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530185+0000", 2024-09-18T07:40:57.476 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8111e-05 2024-09-18T07:40:57.476 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.477 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.477 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.477 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.530789+0000", 2024-09-18T07:40:57.477 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00060407400000000002 2024-09-18T07:40:57.477 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.477 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.477 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.477 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664566+0000", 2024-09-18T07:40:57.477 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13377636900000001 2024-09-18T07:40:57.477 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.477 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.477 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.477 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.477 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.478 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:20756 7.4 e88/81 7:206b3dce:::benchmark_data_smithi137_74180_object20755:head v 88'1303, mlcod=88'1301)", 2024-09-18T07:40:57.478 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.532983+0000", 2024-09-18T07:40:57.478 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 513.93506194600002, 2024-09-18T07:40:57.478 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13536040199999999, 2024-09-18T07:40:57.478 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.478 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.478 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.478 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.478 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.478 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.478 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.532983+0000", 2024-09-18T07:40:57.478 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.478 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.478 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.478 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.479 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.532983+0000", 2024-09-18T07:40:57.479 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.479 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.479 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.479 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.479 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.532984+0000", 2024-09-18T07:40:57.479 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7999999999999993e-07 2024-09-18T07:40:57.479 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.479 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.480 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.480 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533015+0000", 2024-09-18T07:40:57.480 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1195999999999997e-05 2024-09-18T07:40:57.480 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.480 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.480 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.480 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665392+0000", 2024-09-18T07:40:57.480 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13237665800000001 2024-09-18T07:40:57.480 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.480 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.480 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.480 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665400+0000", 2024-09-18T07:40:57.480 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7079999999999997e-06 2024-09-18T07:40:57.480 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665454+0000", 2024-09-18T07:40:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4116999999999997e-05 2024-09-18T07:40:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.665474+0000", 2024-09-18T07:40:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0092999999999999e-05 2024-09-18T07:40:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.481 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.668302+0000", 2024-09-18T07:40:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0028280520000000002 2024-09-18T07:40:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.668343+0000", 2024-09-18T07:40:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1597999999999997e-05 2024-09-18T07:40:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.14865.0:20750 7.3 e88/81 7:c7abdaa5:::benchmark_data_smithi137_74180_object20749:head v 88'1240, mlcod=88'1237)", 2024-09-18T07:40:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.533452+0000", 2024-09-18T07:40:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 513.93459276099998, 2024-09-18T07:40:57.482 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13358589900000001, 2024-09-18T07:40:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533452+0000", 2024-09-18T07:40:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533452+0000", 2024-09-18T07:40:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.483 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.484 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533453+0000", 2024-09-18T07:40:57.484 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.057e-06 2024-09-18T07:40:57.484 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.484 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.484 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.484 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.533515+0000", 2024-09-18T07:40:57.484 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1557999999999996e-05 2024-09-18T07:40:57.484 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.484 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.484 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.484 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664524+0000", 2024-09-18T07:40:57.484 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13100931900000001 2024-09-18T07:40:57.484 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.484 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.485 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.485 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664540+0000", 2024-09-18T07:40:57.485 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5562000000000001e-05 2024-09-18T07:40:57.485 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.485 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.485 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.485 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664600+0000", 2024-09-18T07:40:57.485 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0124000000000002e-05 2024-09-18T07:40:57.485 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.485 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.485 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.485 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.664620+0000", 2024-09-18T07:40:57.485 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9983000000000002e-05 2024-09-18T07:40:57.486 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.486 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.486 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.486 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.666988+0000", 2024-09-18T07:40:57.486 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023680559999999999 2024-09-18T07:40:57.486 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.486 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.486 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.486 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.667038+0000", 2024-09-18T07:40:57.486 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.024e-05 2024-09-18T07:40:57.486 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.486 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.486 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.486 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.486 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.486 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:21270 7.1 7:801f54ca:::benchmark_data_smithi137_74180_object21269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:40:57.487 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:23.882531+0000", 2024-09-18T07:40:57.487 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 513.58551433800005, 2024-09-18T07:40:57.487 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16466234699999999, 2024-09-18T07:40:57.487 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.487 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.487 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.487 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:57.487 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:40:57.488 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:57.488 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 21270 2024-09-18T07:40:57.488 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.488 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.488 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.488 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.488 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882531+0000", 2024-09-18T07:40:57.488 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.488 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.488 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.488 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.488 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882531+0000", 2024-09-18T07:40:57.488 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.488 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.489 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.489 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.489 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882533+0000", 2024-09-18T07:40:57.489 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8190000000000001e-06 2024-09-18T07:40:57.489 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.489 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.489 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.489 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882584+0000", 2024-09-18T07:40:57.489 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1919999999999998e-05 2024-09-18T07:40:57.489 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.489 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.489 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.489 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882593+0000", 2024-09-18T07:40:57.489 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8829999999999995e-06 2024-09-18T07:40:57.490 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.490 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.490 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.490 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882599+0000", 2024-09-18T07:40:57.490 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5849999999999999e-06 2024-09-18T07:40:57.490 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.490 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.490 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.490 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882634+0000", 2024-09-18T07:40:57.490 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5132999999999997e-05 2024-09-18T07:40:57.490 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.490 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.490 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.490 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882717+0000", 2024-09-18T07:40:57.491 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2819999999999996e-05 2024-09-18T07:40:57.491 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.491 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.491 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:40:57.491 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.882824+0000", 2024-09-18T07:40:57.491 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010708400000000001 2024-09-18T07:40:57.491 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.491 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.491 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:40:57.491 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.883645+0000", 2024-09-18T07:40:57.491 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00082133399999999995 2024-09-18T07:40:57.491 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.491 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.491 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.491 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:23.940766+0000", 2024-09-18T07:40:57.492 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.057121166000000001 2024-09-18T07:40:57.492 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.492 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.492 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.492 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.046997+0000", 2024-09-18T07:40:57.492 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.10623094900000001 2024-09-18T07:40:57.492 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.492 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.492 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.492 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047057+0000", 2024-09-18T07:40:57.492 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9580000000000002e-05 2024-09-18T07:40:57.492 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.492 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.492 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.493 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.047193+0000", 2024-09-18T07:40:57.493 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000136074 2024-09-18T07:40:57.493 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.493 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.493 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.493 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.493 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.493 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22331 7.1 7:8575e846:::benchmark_data_smithi137_74180_object22330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:40:57.493 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.655146+0000", 2024-09-18T07:40:57.493 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 512.81289917300001, 2024-09-18T07:40:57.493 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.29868304200000001, 2024-09-18T07:40:57.493 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.493 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.494 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.494 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:57.494 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:40:57.494 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:57.494 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22331 2024-09-18T07:40:57.494 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.494 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.494 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.494 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.494 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655146+0000", 2024-09-18T07:40:57.494 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.494 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.494 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.494 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.494 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655146+0000", 2024-09-18T07:40:57.495 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.495 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.495 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.495 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.495 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655148+0000", 2024-09-18T07:40:57.495 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.779e-06 2024-09-18T07:40:57.495 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.495 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.495 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.495 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655192+0000", 2024-09-18T07:40:57.495 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4023000000000002e-05 2024-09-18T07:40:57.495 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.495 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.495 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.495 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655201+0000", 2024-09-18T07:40:57.496 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3559999999999999e-06 2024-09-18T07:40:57.496 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.496 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.496 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.496 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655206+0000", 2024-09-18T07:40:57.496 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4539999999999998e-06 2024-09-18T07:40:57.496 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.496 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.496 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.496 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655250+0000", 2024-09-18T07:40:57.496 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3630000000000001e-05 2024-09-18T07:40:57.496 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.496 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.496 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.497 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655323+0000", 2024-09-18T07:40:57.497 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3382999999999999e-05 2024-09-18T07:40:57.497 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.497 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.497 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:40:57.497 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.655411+0000", 2024-09-18T07:40:57.497 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7776999999999997e-05 2024-09-18T07:40:57.497 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.497 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.497 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:40:57.497 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.656141+0000", 2024-09-18T07:40:57.497 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00072986100000000005 2024-09-18T07:40:57.497 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.497 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.498 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.498 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.658652+0000", 2024-09-18T07:40:57.498 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025112569999999998 2024-09-18T07:40:57.498 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.498 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.498 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.498 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953705+0000", 2024-09-18T07:40:57.498 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.29505283900000001 2024-09-18T07:40:57.498 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.498 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.499 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.499 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953782+0000", 2024-09-18T07:40:57.499 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6793999999999997e-05 2024-09-18T07:40:57.499 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.499 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.499 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.499 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953829+0000", 2024-09-18T07:40:57.499 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6888999999999998e-05 2024-09-18T07:40:57.499 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.499 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.499 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.499 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.499 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.499 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22350 7.1 7:851589b5:::benchmark_data_smithi137_74180_object22349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:40:57.499 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.662419+0000", 2024-09-18T07:40:57.499 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 512.80562589199997, 2024-09-18T07:40:57.500 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.29155694799999998, 2024-09-18T07:40:57.500 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.500 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.500 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.500 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:57.500 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:40:57.500 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:57.500 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22350 2024-09-18T07:40:57.500 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.500 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.500 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.500 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.500 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662419+0000", 2024-09-18T07:40:57.500 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.501 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.501 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.501 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.501 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662419+0000", 2024-09-18T07:40:57.501 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.501 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.501 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.501 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.501 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662421+0000", 2024-09-18T07:40:57.501 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.39e-06 2024-09-18T07:40:57.501 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.501 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.501 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.502 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662473+0000", 2024-09-18T07:40:57.502 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2094e-05 2024-09-18T07:40:57.502 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.502 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.502 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.502 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662480+0000", 2024-09-18T07:40:57.502 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5449999999999998e-06 2024-09-18T07:40:57.502 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.502 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.502 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.502 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662485+0000", 2024-09-18T07:40:57.502 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9200000000000003e-06 2024-09-18T07:40:57.502 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.503 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.503 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.503 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662521+0000", 2024-09-18T07:40:57.503 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5559000000000003e-05 2024-09-18T07:40:57.503 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.503 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.503 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.503 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662589+0000", 2024-09-18T07:40:57.503 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8480999999999997e-05 2024-09-18T07:40:57.503 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.503 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.503 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:40:57.503 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.662666+0000", 2024-09-18T07:40:57.504 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7224999999999993e-05 2024-09-18T07:40:57.504 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.504 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.504 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:40:57.504 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.663390+0000", 2024-09-18T07:40:57.504 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00072354199999999998 2024-09-18T07:40:57.504 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.504 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.504 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.504 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.663928+0000", 2024-09-18T07:40:57.504 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00053822699999999995 2024-09-18T07:40:57.504 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.504 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.505 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.505 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953879+0000", 2024-09-18T07:40:57.505 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.28995123699999997 2024-09-18T07:40:57.505 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.505 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.505 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.505 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953934+0000", 2024-09-18T07:40:57.505 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5047000000000003e-05 2024-09-18T07:40:57.505 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.505 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.505 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.505 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.953976+0000", 2024-09-18T07:40:57.505 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1681000000000003e-05 2024-09-18T07:40:57.505 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.506 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.506 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.506 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.506 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.506 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:22352 7.1 7:83b03301:::benchmark_data_smithi137_74180_object22351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:40:57.506 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:24.664495+0000", 2024-09-18T07:40:57.506 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 512.80354981599999, 2024-09-18T07:40:57.506 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.28962424599999997, 2024-09-18T07:40:57.506 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.506 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.506 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.506 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:57.506 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:40:57.507 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:57.507 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 22352 2024-09-18T07:40:57.507 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.507 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.507 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.507 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.507 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664495+0000", 2024-09-18T07:40:57.507 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.507 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.507 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.507 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.507 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664495+0000", 2024-09-18T07:40:57.507 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.507 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.508 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.508 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.508 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664496+0000", 2024-09-18T07:40:57.508 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6000000000000003e-07 2024-09-18T07:40:57.508 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.508 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.508 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.508 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664568+0000", 2024-09-18T07:40:57.508 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2021999999999996e-05 2024-09-18T07:40:57.508 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.508 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.508 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.509 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664574+0000", 2024-09-18T07:40:57.509 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4620000000000001e-06 2024-09-18T07:40:57.509 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.509 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.509 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.509 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664579+0000", 2024-09-18T07:40:57.509 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0620000000000002e-06 2024-09-18T07:40:57.509 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.509 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.509 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.509 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664605+0000", 2024-09-18T07:40:57.509 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6126e-05 2024-09-18T07:40:57.509 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.510 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.510 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.510 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664676+0000", 2024-09-18T07:40:57.510 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1521999999999997e-05 2024-09-18T07:40:57.510 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.510 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.510 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:40:57.510 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.664749+0000", 2024-09-18T07:40:57.510 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3232999999999995e-05 2024-09-18T07:40:57.510 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.510 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.510 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:40:57.510 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.665309+0000", 2024-09-18T07:40:57.510 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00055992900000000003 2024-09-18T07:40:57.511 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.511 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.511 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.511 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.665857+0000", 2024-09-18T07:40:57.511 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00054812600000000002 2024-09-18T07:40:57.511 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.511 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.511 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.511 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.954036+0000", 2024-09-18T07:40:57.511 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.288178926 2024-09-18T07:40:57.511 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.511 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.511 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.511 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.954085+0000", 2024-09-18T07:40:57.512 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8534e-05 2024-09-18T07:40:57.512 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.512 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.512 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.512 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:24.954119+0000", 2024-09-18T07:40:57.512 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4544e-05 2024-09-18T07:40:57.512 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.512 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.512 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.512 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.512 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.512 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:44367 7.1 7:854d3320:::benchmark_data_smithi137_74180_object44366:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-18T07:40:57.512 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:43.591378+0000", 2024-09-18T07:40:57.513 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 493.876667252, 2024-09-18T07:40:57.513 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.113341394, 2024-09-18T07:40:57.513 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.513 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.513 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.513 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:57.513 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:40:57.513 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:57.513 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 44367 2024-09-18T07:40:57.513 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.513 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.513 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.513 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.513 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591378+0000", 2024-09-18T07:40:57.513 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.514 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.514 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.514 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.514 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591378+0000", 2024-09-18T07:40:57.514 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.514 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.514 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.514 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.514 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591379+0000", 2024-09-18T07:40:57.514 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7099999999999999e-06 2024-09-18T07:40:57.514 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.514 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:57.516 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.516 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591420+0000", 2024-09-18T07:40:57.516 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0559000000000002e-05 2024-09-18T07:40:57.516 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.516 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.516 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.516 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591432+0000", 2024-09-18T07:40:57.516 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.226e-05 2024-09-18T07:40:57.516 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.517 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.517 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.517 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591440+0000", 2024-09-18T07:40:57.517 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5830000000000001e-06 2024-09-18T07:40:57.517 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.517 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.517 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.517 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591497+0000", 2024-09-18T07:40:57.517 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7114999999999998e-05 2024-09-18T07:40:57.517 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.517 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.517 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.517 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591664+0000", 2024-09-18T07:40:57.517 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000167188 2024-09-18T07:40:57.518 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.518 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.518 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:40:57.518 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.591884+0000", 2024-09-18T07:40:57.518 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000220223 2024-09-18T07:40:57.518 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.518 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.518 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:40:57.518 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.593492+0000", 2024-09-18T07:40:57.518 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016074139999999999 2024-09-18T07:40:57.518 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.518 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.518 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.519 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.593595+0000", 2024-09-18T07:40:57.519 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010319699999999999 2024-09-18T07:40:57.519 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.519 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.519 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.519 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.704655+0000", 2024-09-18T07:40:57.519 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.111060176 2024-09-18T07:40:57.519 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.519 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.519 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.519 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.704690+0000", 2024-09-18T07:40:57.519 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4916999999999998e-05 2024-09-18T07:40:57.519 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.519 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.520 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.520 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:43.704719+0000", 2024-09-18T07:40:57.520 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9051999999999999e-05 2024-09-18T07:40:57.520 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.520 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.520 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.520 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.520 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.520 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:51788 7.1 7:81b6d503:::benchmark_data_smithi137_74180_object51787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:40:57.520 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:40:57.520 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 486.801101077, 2024-09-18T07:40:57.520 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14889203200000001, 2024-09-18T07:40:57.520 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.521 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.521 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.521 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:57.521 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:40:57.521 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:57.521 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 51788 2024-09-18T07:40:57.521 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.521 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.521 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.521 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.521 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:40:57.521 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.521 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.521 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.522 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.522 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:40:57.522 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.522 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.522 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.522 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.522 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666966+0000", 2024-09-18T07:40:57.522 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1860000000000001e-05 2024-09-18T07:40:57.522 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.522 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.522 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.522 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666998+0000", 2024-09-18T07:40:57.522 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1856000000000001e-05 2024-09-18T07:40:57.523 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.523 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.523 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.523 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667012+0000", 2024-09-18T07:40:57.523 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4592e-05 2024-09-18T07:40:57.523 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.523 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.523 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.523 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667019+0000", 2024-09-18T07:40:57.523 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1940000000000001e-06 2024-09-18T07:40:57.523 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.523 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.523 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.524 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667083+0000", 2024-09-18T07:40:57.524 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3553999999999995e-05 2024-09-18T07:40:57.524 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.524 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.524 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.524 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667217+0000", 2024-09-18T07:40:57.524 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000133712 2024-09-18T07:40:57.524 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.524 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.524 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:40:57.524 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667345+0000", 2024-09-18T07:40:57.524 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012796199999999999 2024-09-18T07:40:57.524 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.524 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.525 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:40:57.525 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.668905+0000", 2024-09-18T07:40:57.525 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001560344 2024-09-18T07:40:57.525 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.525 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.525 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.525 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669080+0000", 2024-09-18T07:40:57.525 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000174964 2024-09-18T07:40:57.525 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.525 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.525 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.525 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815763+0000", 2024-09-18T07:40:57.525 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.146683217 2024-09-18T07:40:57.525 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.526 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.526 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.526 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815804+0000", 2024-09-18T07:40:57.526 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0535999999999999e-05 2024-09-18T07:40:57.526 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.526 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.526 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.526 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815836+0000", 2024-09-18T07:40:57.526 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2240999999999998e-05 2024-09-18T07:40:57.526 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.526 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.526 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.526 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.526 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.526 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:51794 7.1 7:8f789502:::benchmark_data_smithi137_74180_object51793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:40:57.527 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:40:57.527 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 486.798927967, 2024-09-18T07:40:57.527 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15046784899999999, 2024-09-18T07:40:57.527 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.527 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.527 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.527 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:57.527 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:40:57.527 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:57.527 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 51794 2024-09-18T07:40:57.527 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.527 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.527 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.527 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.527 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:40:57.527 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.528 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.528 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.528 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.528 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:40:57.528 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.528 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.528 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.528 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.528 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669118+0000", 2024-09-18T07:40:57.528 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4079999999999999e-06 2024-09-18T07:40:57.528 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.528 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.528 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.528 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669172+0000", 2024-09-18T07:40:57.529 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3808999999999999e-05 2024-09-18T07:40:57.529 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.529 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.529 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.529 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669184+0000", 2024-09-18T07:40:57.529 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1888e-05 2024-09-18T07:40:57.529 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.529 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.529 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.529 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669192+0000", 2024-09-18T07:40:57.529 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7549999999999994e-06 2024-09-18T07:40:57.529 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.529 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.530 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.530 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669229+0000", 2024-09-18T07:40:57.530 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7234000000000003e-05 2024-09-18T07:40:57.530 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.530 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.530 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.530 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669310+0000", 2024-09-18T07:40:57.530 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1248999999999994e-05 2024-09-18T07:40:57.530 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.530 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.530 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:40:57.530 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669391+0000", 2024-09-18T07:40:57.530 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0401000000000005e-05 2024-09-18T07:40:57.530 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.531 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.531 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:40:57.531 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.670397+0000", 2024-09-18T07:40:57.531 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010061619999999999 2024-09-18T07:40:57.531 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.531 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.531 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.531 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.670664+0000", 2024-09-18T07:40:57.531 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00026735700000000002 2024-09-18T07:40:57.531 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.531 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.531 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.531 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819479+0000", 2024-09-18T07:40:57.532 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14881438499999999 2024-09-18T07:40:57.532 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.532 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.532 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.532 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819537+0000", 2024-09-18T07:40:57.532 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8192999999999997e-05 2024-09-18T07:40:57.532 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.532 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.532 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.532 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819585+0000", 2024-09-18T07:40:57.532 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8007999999999998e-05 2024-09-18T07:40:57.532 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.532 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.532 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.533 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.533 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.533 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:74680 7.1 7:8999b183:::benchmark_data_smithi137_74180_object74679:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:40:57.533 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:40:57.533 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 461.29007992099997, 2024-09-18T07:40:57.533 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13647764200000001, 2024-09-18T07:40:57.533 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.533 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.533 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.533 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:57.533 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:40:57.533 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:57.533 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 74680 2024-09-18T07:40:57.534 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.534 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.534 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.534 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.534 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:40:57.534 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.534 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.534 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.534 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.534 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:40:57.534 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.534 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.534 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.535 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.535 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177967+0000", 2024-09-18T07:40:57.535 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.559e-06 2024-09-18T07:40:57.535 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.535 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.535 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.535 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178020+0000", 2024-09-18T07:40:57.535 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3810999999999997e-05 2024-09-18T07:40:57.535 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.535 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.535 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.535 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178030+0000", 2024-09-18T07:40:57.535 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5200000000000003e-06 2024-09-18T07:40:57.535 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.536 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.536 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.536 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178035+0000", 2024-09-18T07:40:57.536 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0429999999999998e-06 2024-09-18T07:40:57.536 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.536 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.536 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.536 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178074+0000", 2024-09-18T07:40:57.536 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9014999999999998e-05 2024-09-18T07:40:57.536 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.536 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.536 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.536 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178151+0000", 2024-09-18T07:40:57.536 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6526000000000006e-05 2024-09-18T07:40:57.537 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.537 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.537 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:40:57.537 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178233+0000", 2024-09-18T07:40:57.537 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2174000000000003e-05 2024-09-18T07:40:57.537 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.537 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.537 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:40:57.537 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.180222+0000", 2024-09-18T07:40:57.537 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0019888459999999998 2024-09-18T07:40:57.537 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.537 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.537 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.537 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.180453+0000", 2024-09-18T07:40:57.538 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023154 2024-09-18T07:40:57.538 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.538 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.538 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.538 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314370+0000", 2024-09-18T07:40:57.538 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.133916497 2024-09-18T07:40:57.538 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.538 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.538 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.538 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314411+0000", 2024-09-18T07:40:57.538 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1421000000000002e-05 2024-09-18T07:40:57.538 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.538 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.538 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.538 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314443+0000", 2024-09-18T07:40:57.539 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1690000000000003e-05 2024-09-18T07:40:57.539 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.539 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.539 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.539 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.539 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.539 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:76881 7.1 7:8430cf73:::benchmark_data_smithi137_74180_object76880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:40:57.539 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:18.426831+0000", 2024-09-18T07:40:57.539 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 459.04121416300001, 2024-09-18T07:40:57.539 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.115187357, 2024-09-18T07:40:57.539 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.539 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.539 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.539 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:57.540 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:40:57.540 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:57.540 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 76881 2024-09-18T07:40:57.540 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.540 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.540 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.540 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.540 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426831+0000", 2024-09-18T07:40:57.540 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.540 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.540 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.540 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.540 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426831+0000", 2024-09-18T07:40:57.540 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.541 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.541 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.541 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.541 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426833+0000", 2024-09-18T07:40:57.541 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7710000000000001e-06 2024-09-18T07:40:57.541 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.541 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.541 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.541 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426869+0000", 2024-09-18T07:40:57.541 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5886999999999997e-05 2024-09-18T07:40:57.541 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.541 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.541 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.541 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426879+0000", 2024-09-18T07:40:57.542 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0667e-05 2024-09-18T07:40:57.542 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.542 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.542 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.542 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426886+0000", 2024-09-18T07:40:57.542 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3799999999999999e-06 2024-09-18T07:40:57.542 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.542 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.542 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.542 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.426932+0000", 2024-09-18T07:40:57.542 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6143999999999997e-05 2024-09-18T07:40:57.542 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.542 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.542 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.543 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.427047+0000", 2024-09-18T07:40:57.543 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011568700000000001 2024-09-18T07:40:57.543 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.543 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.543 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:40:57.543 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.427168+0000", 2024-09-18T07:40:57.543 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000120118 2024-09-18T07:40:57.543 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.543 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.543 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:40:57.543 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.429449+0000", 2024-09-18T07:40:57.543 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002281394 2024-09-18T07:40:57.543 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.543 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.544 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.544 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.431262+0000", 2024-09-18T07:40:57.544 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0018125940000000001 2024-09-18T07:40:57.544 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.544 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.544 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.544 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541358+0000", 2024-09-18T07:40:57.544 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.110095996 2024-09-18T07:40:57.544 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.544 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.544 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.544 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.541963+0000", 2024-09-18T07:40:57.544 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00060516200000000002 2024-09-18T07:40:57.544 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.545 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.545 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.545 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:18.542018+0000", 2024-09-18T07:40:57.545 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5556999999999997e-05 2024-09-18T07:40:57.545 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.545 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.545 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.545 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.545 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.545 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:142957 7.1 7:830d8edf:::benchmark_data_smithi137_74180_object38541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-18T07:40:57.545 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:25.430263+0000", 2024-09-18T07:40:57.545 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 392.03778229800002, 2024-09-18T07:40:57.545 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18925961299999999, 2024-09-18T07:40:57.545 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.545 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.546 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.546 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:57.546 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:40:57.546 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:40:57.546 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 142957 2024-09-18T07:40:57.546 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.546 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.546 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.546 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.546 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430263+0000", 2024-09-18T07:40:57.546 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.546 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.546 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.546 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.547 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430263+0000", 2024-09-18T07:40:57.547 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.547 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.547 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.547 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.547 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430265+0000", 2024-09-18T07:40:57.547 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9800000000000001e-06 2024-09-18T07:40:57.547 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.547 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.547 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.547 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430270+0000", 2024-09-18T07:40:57.547 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6189999999999999e-06 2024-09-18T07:40:57.547 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.548 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.548 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.548 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430281+0000", 2024-09-18T07:40:57.548 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1146e-05 2024-09-18T07:40:57.548 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.548 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.548 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.548 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430288+0000", 2024-09-18T07:40:57.548 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9319999999999999e-06 2024-09-18T07:40:57.548 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.548 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.548 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.548 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430340+0000", 2024-09-18T07:40:57.548 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1904999999999999e-05 2024-09-18T07:40:57.549 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.549 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.549 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.549 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430468+0000", 2024-09-18T07:40:57.549 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000127448 2024-09-18T07:40:57.549 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.549 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.549 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:40:57.549 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430717+0000", 2024-09-18T07:40:57.549 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024888500000000001 2024-09-18T07:40:57.549 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.549 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.549 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:40:57.549 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.432189+0000", 2024-09-18T07:40:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014722470000000001 2024-09-18T07:40:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.432734+0000", 2024-09-18T07:40:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00054558799999999995 2024-09-18T07:40:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.619409+0000", 2024-09-18T07:40:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18667450399999999 2024-09-18T07:40:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.550 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.551 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.619471+0000", 2024-09-18T07:40:57.551 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1877000000000004e-05 2024-09-18T07:40:57.551 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.551 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.551 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.551 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.619522+0000", 2024-09-18T07:40:57.551 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1481999999999999e-05 2024-09-18T07:40:57.551 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.551 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.551 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.551 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.551 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.551 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:95519 8.5 8:af8c363a:::benchmark_data_smithi137_101942_object95518:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-09-18T07:40:57.551 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:23.377289+0000", 2024-09-18T07:40:57.552 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 214.09075629399999, 2024-09-18T07:40:57.552 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.12657783, 2024-09-18T07:40:57.552 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.552 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.552 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.552 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:57.552 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:40:57.552 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:40:57.552 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 95519 2024-09-18T07:40:57.552 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.552 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.552 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.552 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.553 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377289+0000", 2024-09-18T07:40:57.553 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.553 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.553 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.553 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.553 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377289+0000", 2024-09-18T07:40:57.553 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.553 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.553 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.553 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.553 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377300+0000", 2024-09-18T07:40:57.553 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0998e-05 2024-09-18T07:40:57.553 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377358+0000", 2024-09-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7935e-05 2024-09-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377368+0000", 2024-09-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0344e-05 2024-09-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377374+0000", 2024-09-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3990000000000001e-06 2024-09-18T07:40:57.555 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.555 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.555 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.555 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377452+0000", 2024-09-18T07:40:57.555 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7829000000000003e-05 2024-09-18T07:40:57.555 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.555 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.555 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.555 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377626+0000", 2024-09-18T07:40:57.555 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017389900000000001 2024-09-18T07:40:57.555 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.555 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.555 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,7", 2024-09-18T07:40:57.555 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.377747+0000", 2024-09-18T07:40:57.555 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012108900000000001 2024-09-18T07:40:57.556 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.556 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.556 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:40:57.556 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.379961+0000", 2024-09-18T07:40:57.556 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022133560000000001 2024-09-18T07:40:57.556 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.556 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.556 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.556 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.380219+0000", 2024-09-18T07:40:57.556 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000258879 2024-09-18T07:40:57.556 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.556 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.556 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.556 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.502323+0000", 2024-09-18T07:40:57.557 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:57.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:57.558 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.12210367900000001 2024-09-18T07:40:57.558 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.558 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.558 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.558 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.502368+0000", 2024-09-18T07:40:57.558 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4557e-05 2024-09-18T07:40:57.558 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.558 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.558 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.558 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:23.503867+0000", 2024-09-18T07:40:57.558 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001498866 2024-09-18T07:40:57.558 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.559 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.559 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.559 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.559 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.559 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:119507 8.2 8:4faaf3aa:::benchmark_data_smithi137_101942_object14697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-18T07:40:57.559 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:53.113193+0000", 2024-09-18T07:40:57.559 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 184.35485167100001, 2024-09-18T07:40:57.559 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17040597299999999, 2024-09-18T07:40:57.559 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.559 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.559 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.559 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:57.559 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:40:57.560 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:40:57.560 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 119507 2024-09-18T07:40:57.560 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.560 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.560 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.560 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.560 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113193+0000", 2024-09-18T07:40:57.560 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.560 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.560 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.560 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.560 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113193+0000", 2024-09-18T07:40:57.560 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.561 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.561 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.561 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.561 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113195+0000", 2024-09-18T07:40:57.561 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.956e-06 2024-09-18T07:40:57.561 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.561 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.561 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.561 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113201+0000", 2024-09-18T07:40:57.561 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3639999999999998e-06 2024-09-18T07:40:57.561 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.561 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.561 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.562 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113210+0000", 2024-09-18T07:40:57.562 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6649999999999998e-06 2024-09-18T07:40:57.562 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.562 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.562 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.562 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113216+0000", 2024-09-18T07:40:57.562 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0639999999999999e-06 2024-09-18T07:40:57.562 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.562 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.562 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.562 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113260+0000", 2024-09-18T07:40:57.562 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4013e-05 2024-09-18T07:40:57.562 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.563 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.563 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.563 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113348+0000", 2024-09-18T07:40:57.563 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7392000000000006e-05 2024-09-18T07:40:57.563 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.563 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.563 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:40:57.563 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113429+0000", 2024-09-18T07:40:57.563 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1111000000000003e-05 2024-09-18T07:40:57.563 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.563 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.563 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:40:57.563 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.115474+0000", 2024-09-18T07:40:57.564 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0020447769999999998 2024-09-18T07:40:57.564 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.564 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.564 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.564 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.116688+0000", 2024-09-18T07:40:57.564 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001214581 2024-09-18T07:40:57.564 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.564 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.564 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.564 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.283501+0000", 2024-09-18T07:40:57.564 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.166812558 2024-09-18T07:40:57.564 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.564 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.564 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.565 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.283557+0000", 2024-09-18T07:40:57.565 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5856999999999997e-05 2024-09-18T07:40:57.565 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.565 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.565 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.565 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.283599+0000", 2024-09-18T07:40:57.565 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2635000000000001e-05 2024-09-18T07:40:57.565 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.565 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.565 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.565 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.565 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.565 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:152088 8.2 8:4be317d2:::benchmark_data_smithi137_101942_object47278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-09-18T07:40:57.565 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:26.863366+0000", 2024-09-18T07:40:57.566 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 150.604678689, 2024-09-18T07:40:57.566 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.122412571, 2024-09-18T07:40:57.566 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.566 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.566 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.566 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:57.566 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:40:57.566 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:40:57.566 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 152088 2024-09-18T07:40:57.566 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.566 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.566 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.566 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.566 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.863366+0000", 2024-09-18T07:40:57.567 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.567 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.567 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.567 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.567 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.863366+0000", 2024-09-18T07:40:57.567 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.567 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.567 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.567 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.567 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.863369+0000", 2024-09-18T07:40:57.567 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.819e-06 2024-09-18T07:40:57.567 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.567 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.567 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.568 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.863377+0000", 2024-09-18T07:40:57.568 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5140000000000004e-06 2024-09-18T07:40:57.568 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.568 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.568 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.568 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.863392+0000", 2024-09-18T07:40:57.568 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5586999999999998e-05 2024-09-18T07:40:57.568 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.568 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.568 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.568 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.863400+0000", 2024-09-18T07:40:57.568 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6690000000000006e-06 2024-09-18T07:40:57.568 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.568 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.569 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.569 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.863461+0000", 2024-09-18T07:40:57.569 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1420000000000005e-05 2024-09-18T07:40:57.569 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.569 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.569 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.569 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.863594+0000", 2024-09-18T07:40:57.569 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013225699999999999 2024-09-18T07:40:57.569 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.569 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.569 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:40:57.569 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.863724+0000", 2024-09-18T07:40:57.569 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013000899999999999 2024-09-18T07:40:57.569 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.570 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.570 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:40:57.570 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.866428+0000", 2024-09-18T07:40:57.570 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027039529999999998 2024-09-18T07:40:57.570 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.570 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.570 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.570 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.872055+0000", 2024-09-18T07:40:57.570 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0056270089999999997 2024-09-18T07:40:57.570 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.570 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.570 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.570 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.985708+0000", 2024-09-18T07:40:57.571 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.11365310000000001 2024-09-18T07:40:57.571 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.571 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.571 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.571 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.985747+0000", 2024-09-18T07:40:57.571 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8998999999999997e-05 2024-09-18T07:40:57.571 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.571 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.571 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.571 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:26.985779+0000", 2024-09-18T07:40:57.571 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2234999999999999e-05 2024-09-18T07:40:57.571 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.571 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.571 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.572 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.572 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.572 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:170259 8.b e264/201 8:dbf0040f:::benchmark_data_smithi137_101942_object65449:head v 264'10603, mlcod=264'10600)", 2024-09-18T07:40:57.572 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:44.540417+0000", 2024-09-18T07:40:57.572 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 132.92762842600001, 2024-09-18T07:40:57.572 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14423132799999999, 2024-09-18T07:40:57.572 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.572 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.572 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.572 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.572 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.572 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.572 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.540417+0000", 2024-09-18T07:40:57.573 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.573 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.573 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.573 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.573 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.540417+0000", 2024-09-18T07:40:57.573 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.573 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.573 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.573 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.573 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.540418+0000", 2024-09-18T07:40:57.573 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2219999999999999e-06 2024-09-18T07:40:57.573 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.573 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.574 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.574 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.540424+0000", 2024-09-18T07:40:57.574 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.28e-06 2024-09-18T07:40:57.574 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.574 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.574 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.574 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.674941+0000", 2024-09-18T07:40:57.574 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13451724300000001 2024-09-18T07:40:57.574 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.574 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.574 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.574 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.674964+0000", 2024-09-18T07:40:57.574 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2813e-05 2024-09-18T07:40:57.575 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.575 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.575 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.575 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.675047+0000", 2024-09-18T07:40:57.575 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2547999999999996e-05 2024-09-18T07:40:57.575 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.575 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.575 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.575 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.675067+0000", 2024-09-18T07:40:57.575 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0767e-05 2024-09-18T07:40:57.575 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.575 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.575 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.575 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.684593+0000", 2024-09-18T07:40:57.576 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0095254080000000008 2024-09-18T07:40:57.576 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.576 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.576 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.576 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.684648+0000", 2024-09-18T07:40:57.576 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5047000000000003e-05 2024-09-18T07:40:57.576 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.576 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.576 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.576 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.576 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.576 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:170269 8.e e264/201 8:70bf8783:::benchmark_data_smithi137_101942_object65459:head v 264'10510, mlcod=264'10508)", 2024-09-18T07:40:57.576 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:44.547119+0000", 2024-09-18T07:40:57.577 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 132.920925762, 2024-09-18T07:40:57.577 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13849322, 2024-09-18T07:40:57.577 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.577 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.577 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.577 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.577 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.577 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.577 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.547119+0000", 2024-09-18T07:40:57.577 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.577 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.577 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.577 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.577 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.547119+0000", 2024-09-18T07:40:57.578 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.578 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.578 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.578 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.578 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.547122+0000", 2024-09-18T07:40:57.578 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.277e-06 2024-09-18T07:40:57.578 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.578 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.578 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.578 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.547131+0000", 2024-09-18T07:40:57.578 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6649999999999998e-06 2024-09-18T07:40:57.578 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.578 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.578 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.579 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.675047+0000", 2024-09-18T07:40:57.579 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.12791548799999999 2024-09-18T07:40:57.579 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.579 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.579 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.579 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.675056+0000", 2024-09-18T07:40:57.579 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9630000000000004e-06 2024-09-18T07:40:57.579 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.579 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.579 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.580 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.675222+0000", 2024-09-18T07:40:57.580 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000166324 2024-09-18T07:40:57.580 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.580 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.580 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.580 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.675241+0000", 2024-09-18T07:40:57.580 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9480999999999999e-05 2024-09-18T07:40:57.580 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.580 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.580 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.580 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.685550+0000", 2024-09-18T07:40:57.580 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010308126000000001 2024-09-18T07:40:57.580 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.580 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.581 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.581 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.685612+0000", 2024-09-18T07:40:57.581 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2896000000000002e-05 2024-09-18T07:40:57.581 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.581 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.581 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.581 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.581 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.581 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:202181 8.2 8:43b6ef0b:::benchmark_data_smithi137_101942_object97371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-18T07:40:57.581 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:39:13.195522+0000", 2024-09-18T07:40:57.581 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 104.27252292599999, 2024-09-18T07:40:57.581 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13193157799999999, 2024-09-18T07:40:57.581 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.582 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.582 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.582 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:57.582 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:40:57.582 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:40:57.582 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 202181 2024-09-18T07:40:57.582 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.582 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.582 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.582 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.582 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195522+0000", 2024-09-18T07:40:57.582 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.582 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.582 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.583 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.583 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195522+0000", 2024-09-18T07:40:57.583 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.583 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.583 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.583 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.583 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195524+0000", 2024-09-18T07:40:57.583 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.86e-06 2024-09-18T07:40:57.583 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.583 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.583 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.583 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195530+0000", 2024-09-18T07:40:57.583 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9390000000000002e-06 2024-09-18T07:40:57.583 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.584 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.584 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.584 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195542+0000", 2024-09-18T07:40:57.584 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2197e-05 2024-09-18T07:40:57.584 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.584 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.584 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.584 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195549+0000", 2024-09-18T07:40:57.584 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2640000000000002e-06 2024-09-18T07:40:57.584 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.584 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.584 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.584 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195595+0000", 2024-09-18T07:40:57.584 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5355000000000002e-05 2024-09-18T07:40:57.585 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.585 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.585 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.585 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195679+0000", 2024-09-18T07:40:57.585 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3979000000000004e-05 2024-09-18T07:40:57.585 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.585 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.585 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:40:57.585 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195750+0000", 2024-09-18T07:40:57.585 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1229000000000005e-05 2024-09-18T07:40:57.585 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.585 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.585 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:40:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.197685+0000", 2024-09-18T07:40:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0019350019999999999 2024-09-18T07:40:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.202152+0000", 2024-09-18T07:40:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0044671550000000004 2024-09-18T07:40:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.327295+0000", 2024-09-18T07:40:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.12514278300000001 2024-09-18T07:40:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.327408+0000", 2024-09-18T07:40:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000113173 2024-09-18T07:40:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.327454+0000", 2024-09-18T07:40:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5642000000000001e-05 2024-09-18T07:40:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:202183 8.2 8:47b706de:::benchmark_data_smithi137_101942_object97373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-18T07:40:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:39:13.198689+0000", 2024-09-18T07:40:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 104.269356235, 2024-09-18T07:40:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.12892711600000001, 2024-09-18T07:40:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:40:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:40:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:40:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 202183 2024-09-18T07:40:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.198689+0000", 2024-09-18T07:40:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.198689+0000", 2024-09-18T07:40:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.198691+0000", 2024-09-18T07:40:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6759999999999999e-06 2024-09-18T07:40:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.198703+0000", 2024-09-18T07:40:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1185999999999999e-05 2024-09-18T07:40:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.198726+0000", 2024-09-18T07:40:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3047000000000001e-05 2024-09-18T07:40:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.198739+0000", 2024-09-18T07:40:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3036e-05 2024-09-18T07:40:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:57.591 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.198822+0000", 2024-09-18T07:40:57.591 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3108000000000005e-05 2024-09-18T07:40:57.591 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.591 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.591 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:57.591 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.198970+0000", 2024-09-18T07:40:57.591 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014830999999999999 2024-09-18T07:40:57.591 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.591 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.591 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:40:57.591 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.199039+0000", 2024-09-18T07:40:57.591 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8854000000000002e-05 2024-09-18T07:40:57.591 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.591 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.591 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:40:57.592 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.200125+0000", 2024-09-18T07:40:57.592 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010859719999999999 2024-09-18T07:40:57.592 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.592 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.592 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.592 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.202262+0000", 2024-09-18T07:40:57.592 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021368530000000002 2024-09-18T07:40:57.592 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.592 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.592 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:57.592 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.327495+0000", 2024-09-18T07:40:57.592 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.12523358200000001 2024-09-18T07:40:57.592 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.592 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.593 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:40:57.593 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.327570+0000", 2024-09-18T07:40:57.593 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4803999999999997e-05 2024-09-18T07:40:57.593 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:57.593 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:57.593 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:40:57.593 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.327616+0000", 2024-09-18T07:40:57.593 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5688e-05 2024-09-18T07:40:57.593 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.593 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.593 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.593 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:57.593 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:57.593 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:40:57.594 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:40:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:40:57.657 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:57.695 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:57.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:57.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:57.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:57.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:57.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:57.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:57.890 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:40:57.927 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-18T07:40:57.927 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-18T07:40:57.927 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-18T07:40:57.927 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-18T07:40:57.928 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-18T07:40:57.928 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-18T07:40:57.928 INFO:teuthology.orchestra.run.smithi137.stderr:bench [] [] [] [] 2024-09-18T07:40:57.928 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-18T07:40:57.928 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-18T07:40:57.928 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-18T07:40:57.928 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator dump block 2024-09-18T07:40:57.928 INFO:teuthology.orchestra.run.smithi137.stderr:Error EINVAL: invalid command 2024-09-18T07:40:57.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:57.991 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:57.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:57.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:58.090 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 83 16 124702 124686 93.8677 62.75 0.0162251 0.0106489 2024-09-18T07:40:58.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:40:58.129 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:58.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:58.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:40:58.152 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:40:58.231 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:58.272 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:58.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:58.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:58.374 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:58.412 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:58.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:58.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:58.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:58 smithi137 ceph-mon[29408]: 9.6 scrub ok 2024-09-18T07:40:58.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:58 smithi137 ceph-mon[29408]: 3.3 scrub starts 2024-09-18T07:40:58.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:58 smithi137 ceph-mon[29408]: 3.3 scrub ok 2024-09-18T07:40:58.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:58 smithi137 ceph-mon[29408]: 3.1e scrub starts 2024-09-18T07:40:58.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:58 smithi137 ceph-mon[29408]: 3.1e scrub ok 2024-09-18T07:40:58.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:58 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1586553236' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:40:58.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:58 smithi137 ceph-mon[39175]: 9.6 scrub ok 2024-09-18T07:40:58.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:58 smithi137 ceph-mon[39175]: 3.3 scrub starts 2024-09-18T07:40:58.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:58 smithi137 ceph-mon[39175]: 3.3 scrub ok 2024-09-18T07:40:58.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:58 smithi137 ceph-mon[39175]: 3.1e scrub starts 2024-09-18T07:40:58.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:58 smithi137 ceph-mon[39175]: 3.1e scrub ok 2024-09-18T07:40:58.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:58 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1586553236' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:40:58.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:58 smithi203 ceph-mon[32237]: 9.6 scrub ok 2024-09-18T07:40:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:58 smithi203 ceph-mon[32237]: 3.3 scrub starts 2024-09-18T07:40:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:58 smithi203 ceph-mon[32237]: 3.3 scrub ok 2024-09-18T07:40:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:58 smithi203 ceph-mon[32237]: 3.1e scrub starts 2024-09-18T07:40:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:58 smithi203 ceph-mon[32237]: 3.1e scrub ok 2024-09-18T07:40:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:58 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1586553236' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:40:58.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:40:58.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:58.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:58.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:40:58.594 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T07:40:58.672 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:40:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:58.723 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:58.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:40:58.824 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:58.856 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:58.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:58.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:58.958 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:58.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:58.982 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:58.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:59.039 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:40:59.083 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:59.090 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 84 16 125809 125793 93.5738 69.1875 0.017018 0.0106823 2024-09-18T07:40:59.133 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:59.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:59.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:59.235 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:40:59.269 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:59.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:59.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:40:59.371 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:40:59.401 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:59.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:40:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[29408]: pgmap v537: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-18T07:40:59.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[29408]: 4.1e scrub starts 2024-09-18T07:40:59.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[29408]: 4.1e scrub ok 2024-09-18T07:40:59.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[29408]: 9.3 scrub ok 2024-09-18T07:40:59.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:40:59.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1586553236' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:40:59.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[29408]: osdmap e322: 8 total, 8 up, 8 in 2024-09-18T07:40:59.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:40:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[39175]: pgmap v537: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-18T07:40:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[39175]: 4.1e scrub starts 2024-09-18T07:40:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[39175]: 4.1e scrub ok 2024-09-18T07:40:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[39175]: 9.3 scrub ok 2024-09-18T07:40:59.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:40:59.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1586553236' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:40:59.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[39175]: osdmap e322: 8 total, 8 up, 8 in 2024-09-18T07:40:59.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:40:59 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:40:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:59 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:40:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:59 smithi203 ceph-mon[32237]: pgmap v537: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-18T07:40:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:59 smithi203 ceph-mon[32237]: 4.1e scrub starts 2024-09-18T07:40:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:59 smithi203 ceph-mon[32237]: 4.1e scrub ok 2024-09-18T07:40:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:59 smithi203 ceph-mon[32237]: 9.3 scrub ok 2024-09-18T07:40:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:59 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:40:59.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:59 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1586553236' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:40:59.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:59 smithi203 ceph-mon[32237]: osdmap e322: 8 total, 8 up, 8 in 2024-09-18T07:40:59.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:40:59 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:40:59.503 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:40:59.551 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:59.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:59.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:40:59.652 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:40:59.652 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:40:59.652 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.652 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.15699.0:126462 9.5 e322/289 9:a610df6a:::benchmark_data_smithi137_126072_object126461:head v 322'7801, mlcod=322'7796)", 2024-09-18T07:40:59.652 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:40:59.633974+0000", 2024-09-18T07:40:59.652 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.014050178999999999, 2024-09-18T07:40:59.653 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.014078937999999999, 2024-09-18T07:40:59.653 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:59.653 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:59.653 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:40:59.653 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:59.653 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.653 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:59.653 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.633974+0000", 2024-09-18T07:40:59.653 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.653 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.653 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.653 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:59.653 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.633974+0000", 2024-09-18T07:40:59.653 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.653 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.654 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.654 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:59.654 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.633975+0000", 2024-09-18T07:40:59.654 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.088e-06 2024-09-18T07:40:59.654 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.654 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.654 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:59.654 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.634084+0000", 2024-09-18T07:40:59.654 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108899 2024-09-18T07:40:59.654 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.654 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.654 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:59.654 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.634093+0000", 2024-09-18T07:40:59.654 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3139999999999993e-06 2024-09-18T07:40:59.654 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.655 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.655 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:59.655 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.634099+0000", 2024-09-18T07:40:59.655 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8830000000000003e-06 2024-09-18T07:40:59.655 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.655 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.655 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:59.655 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.638826+0000", 2024-09-18T07:40:59.655 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0047271500000000003 2024-09-18T07:40:59.655 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.655 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.655 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:59.655 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.638845+0000", 2024-09-18T07:40:59.656 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8760999999999999e-05 2024-09-18T07:40:59.656 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.656 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:59.656 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.656 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.656 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.656 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:126468 9.7 9:e9ac00a6:::benchmark_data_smithi137_126072_object126467:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-18T07:40:59.656 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:40:59.634783+0000", 2024-09-18T07:40:59.656 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.013240632, 2024-09-18T07:40:59.656 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.013331150999999999, 2024-09-18T07:40:59.656 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:59.656 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:59.656 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:40:59.657 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:59.657 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:40:59.657 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:40:59.657 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 126468 2024-09-18T07:40:59.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.657 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:59.657 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.657 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:59.657 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.634783+0000", 2024-09-18T07:40:59.657 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.657 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.657 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:59.658 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.634783+0000", 2024-09-18T07:40:59.658 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.658 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.658 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.658 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:59.658 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.634785+0000", 2024-09-18T07:40:59.658 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8929999999999999e-06 2024-09-18T07:40:59.658 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.658 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.658 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:59.658 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.634840+0000", 2024-09-18T07:40:59.658 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5155999999999998e-05 2024-09-18T07:40:59.658 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.658 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.659 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:59.659 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.634849+0000", 2024-09-18T07:40:59.659 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6710000000000005e-06 2024-09-18T07:40:59.659 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.659 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.659 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:59.659 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.634855+0000", 2024-09-18T07:40:59.659 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4910000000000002e-06 2024-09-18T07:40:59.659 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.659 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.659 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:59.659 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.635048+0000", 2024-09-18T07:40:59.659 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019275800000000001 2024-09-18T07:40:59.659 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.659 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.660 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:59.660 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.635274+0000", 2024-09-18T07:40:59.660 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000225319 2024-09-18T07:40:59.660 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.660 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.660 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:40:59.660 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.635404+0000", 2024-09-18T07:40:59.660 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000130266 2024-09-18T07:40:59.660 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.660 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.660 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:40:59.660 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.644261+0000", 2024-09-18T07:40:59.660 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0088573660000000002 2024-09-18T07:40:59.661 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.661 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.661 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:59.661 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.644359+0000", 2024-09-18T07:40:59.661 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7886000000000004e-05 2024-09-18T07:40:59.661 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.661 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.661 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:59.661 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.648017+0000", 2024-09-18T07:40:59.661 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0036575459999999998 2024-09-18T07:40:59.661 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.661 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:59.661 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.661 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.662 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.662 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:126471 9.7 9:ef02bf93:::benchmark_data_smithi137_126072_object126470:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-18T07:40:59.662 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:40:59.636681+0000", 2024-09-18T07:40:59.662 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.011342701, 2024-09-18T07:40:59.662 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011517329999999999, 2024-09-18T07:40:59.662 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:59.662 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:59.662 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:40:59.662 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:59.662 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:40:59.662 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:40:59.662 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 126471 2024-09-18T07:40:59.662 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.662 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:59.663 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.663 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:59.663 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.636681+0000", 2024-09-18T07:40:59.663 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.663 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.663 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.663 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:59.663 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.636681+0000", 2024-09-18T07:40:59.663 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.663 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.663 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.663 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:59.663 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.636683+0000", 2024-09-18T07:40:59.663 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.384e-06 2024-09-18T07:40:59.664 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.664 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.664 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:59.664 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.636887+0000", 2024-09-18T07:40:59.664 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00020407800000000001 2024-09-18T07:40:59.664 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.664 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.664 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:59.664 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.636897+0000", 2024-09-18T07:40:59.664 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0287e-05 2024-09-18T07:40:59.664 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.664 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.664 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:59.664 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.636905+0000", 2024-09-18T07:40:59.665 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0129999999999993e-06 2024-09-18T07:40:59.665 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.665 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.665 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:59.665 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.640383+0000", 2024-09-18T07:40:59.665 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0034781479999999999 2024-09-18T07:40:59.665 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.665 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.665 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:59.665 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.640782+0000", 2024-09-18T07:40:59.665 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00039857299999999999 2024-09-18T07:40:59.665 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.665 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.666 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:40:59.666 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.640913+0000", 2024-09-18T07:40:59.666 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:40:59.667 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000131166 2024-09-18T07:40:59.667 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.667 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.667 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:40:59.667 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.644475+0000", 2024-09-18T07:40:59.667 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0035626859999999998 2024-09-18T07:40:59.667 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.667 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.668 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:40:59.668 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.644517+0000", 2024-09-18T07:40:59.668 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1071e-05 2024-09-18T07:40:59.668 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.668 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:59.668 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.668 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.668 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.668 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.15699.0:126467 9.5 e322/289 9:a269f02c:::benchmark_data_smithi137_126072_object126466:head v 322'7802, mlcod=322'7796)", 2024-09-18T07:40:59.668 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:40:59.637924+0000", 2024-09-18T07:40:59.668 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.010099924, 2024-09-18T07:40:59.668 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010347231, 2024-09-18T07:40:59.668 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:59.669 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:59.669 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:40:59.669 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:59.669 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.669 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:59.669 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.637924+0000", 2024-09-18T07:40:59.669 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.669 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.669 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.669 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:59.669 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.637924+0000", 2024-09-18T07:40:59.669 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.669 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.670 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.670 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:59.670 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.637925+0000", 2024-09-18T07:40:59.670 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.093e-06 2024-09-18T07:40:59.670 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.670 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.670 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:59.670 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.638044+0000", 2024-09-18T07:40:59.670 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000118725 2024-09-18T07:40:59.670 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.670 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.670 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:59.670 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.638053+0000", 2024-09-18T07:40:59.670 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1290000000000002e-06 2024-09-18T07:40:59.671 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.671 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.671 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:59.671 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.638059+0000", 2024-09-18T07:40:59.671 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5030000000000002e-06 2024-09-18T07:40:59.671 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.671 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.671 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:59.671 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.642968+0000", 2024-09-18T07:40:59.671 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0049081929999999999 2024-09-18T07:40:59.671 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.671 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.671 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:59.671 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.642986+0000", 2024-09-18T07:40:59.671 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8601e-05 2024-09-18T07:40:59.672 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.672 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:59.672 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.672 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.672 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.672 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.15699.0:126470 9.5 e322/289 9:ad05b536:::benchmark_data_smithi137_126072_object126469:head v 322'7803, mlcod=322'7796)", 2024-09-18T07:40:59.672 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:40:59.640431+0000", 2024-09-18T07:40:59.672 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0075932309999999998, 2024-09-18T07:40:59.672 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0078890550000000007, 2024-09-18T07:40:59.672 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:59.672 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:59.672 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:40:59.672 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:59.673 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.673 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:59.673 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.640431+0000", 2024-09-18T07:40:59.673 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.673 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.673 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.673 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:59.673 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.640431+0000", 2024-09-18T07:40:59.673 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.673 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.673 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.673 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:59.673 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.640432+0000", 2024-09-18T07:40:59.673 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.015e-06 2024-09-18T07:40:59.674 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.674 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.674 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:59.674 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.640544+0000", 2024-09-18T07:40:59.674 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000112433 2024-09-18T07:40:59.674 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.674 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.674 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:59.674 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.640556+0000", 2024-09-18T07:40:59.674 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1569000000000001e-05 2024-09-18T07:40:59.674 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.674 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.674 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:59.674 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.640561+0000", 2024-09-18T07:40:59.675 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5779999999999998e-06 2024-09-18T07:40:59.675 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.675 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.675 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:59.675 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.645781+0000", 2024-09-18T07:40:59.675 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0052202419999999999 2024-09-18T07:40:59.675 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.675 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.675 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:59.675 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.645803+0000", 2024-09-18T07:40:59.675 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1233000000000002e-05 2024-09-18T07:40:59.675 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.675 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:59.675 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.675 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.676 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.676 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:126475 9.f 9:ffcd0da6:::benchmark_data_smithi137_126072_object126474:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-18T07:40:59.676 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:40:59.640631+0000", 2024-09-18T07:40:59.676 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0073929679999999998, 2024-09-18T07:40:59.676 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0077446399999999997, 2024-09-18T07:40:59.676 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:59.676 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:59.676 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:40:59.676 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:59.676 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:40:59.676 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:40:59.676 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 126475 2024-09-18T07:40:59.676 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.676 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:59.676 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.677 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:59.677 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.640631+0000", 2024-09-18T07:40:59.677 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.677 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.677 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.677 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:59.677 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.640631+0000", 2024-09-18T07:40:59.677 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.677 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.677 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.677 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:59.677 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.640633+0000", 2024-09-18T07:40:59.677 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6220000000000001e-06 2024-09-18T07:40:59.678 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.678 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.678 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:59.678 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.640692+0000", 2024-09-18T07:40:59.678 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9021000000000003e-05 2024-09-18T07:40:59.678 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.678 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.678 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:59.678 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.640700+0000", 2024-09-18T07:40:59.678 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0860000000000002e-06 2024-09-18T07:40:59.678 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.678 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.678 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:59.679 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.640706+0000", 2024-09-18T07:40:59.679 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0129999999999999e-06 2024-09-18T07:40:59.679 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.679 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.679 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:59.679 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.645047+0000", 2024-09-18T07:40:59.679 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.004341536 2024-09-18T07:40:59.679 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.679 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.679 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:59.679 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.645160+0000", 2024-09-18T07:40:59.679 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000113194 2024-09-18T07:40:59.679 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.679 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.680 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:40:59.680 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.645296+0000", 2024-09-18T07:40:59.680 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013563200000000001 2024-09-18T07:40:59.680 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.680 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:59.680 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.680 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.680 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.680 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:126477 9.7 9:e3c82774:::benchmark_data_smithi137_126072_object126476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-18T07:40:59.680 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:40:59.641122+0000", 2024-09-18T07:40:59.680 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0069015389999999999, 2024-09-18T07:40:59.680 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.007319903, 2024-09-18T07:40:59.680 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:59.680 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:59.680 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:40:59.680 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:59.681 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:40:59.681 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:40:59.681 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 126477 2024-09-18T07:40:59.681 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.681 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:59.681 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.681 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:59.681 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.641122+0000", 2024-09-18T07:40:59.681 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.681 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.681 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.681 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:59.681 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.641122+0000", 2024-09-18T07:40:59.681 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.681 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.682 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.682 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:59.682 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.641123+0000", 2024-09-18T07:40:59.682 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.144e-06 2024-09-18T07:40:59.682 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.682 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.682 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:59.682 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.641181+0000", 2024-09-18T07:40:59.682 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7383000000000003e-05 2024-09-18T07:40:59.682 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.682 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.682 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:59.682 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.641191+0000", 2024-09-18T07:40:59.682 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0560000000000001e-05 2024-09-18T07:40:59.682 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.683 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.683 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:59.683 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.641208+0000", 2024-09-18T07:40:59.683 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6549999999999999e-05 2024-09-18T07:40:59.683 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.683 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.683 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:59.683 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.644630+0000", 2024-09-18T07:40:59.683 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0034223399999999998 2024-09-18T07:40:59.683 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.683 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.683 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:59.683 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.644766+0000", 2024-09-18T07:40:59.683 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013576699999999999 2024-09-18T07:40:59.684 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.684 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.684 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:40:59.684 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.644914+0000", 2024-09-18T07:40:59.684 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014812000000000001 2024-09-18T07:40:59.684 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.684 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:59.684 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.684 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.684 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.684 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:126478 9.7 9.6e4275c7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-18T07:40:59.684 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:40:59.643119+0000", 2024-09-18T07:40:59.684 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0049045360000000001, 2024-09-18T07:40:59.685 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0053830120000000004, 2024-09-18T07:40:59.685 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:59.685 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:59.685 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "queued for pg", 2024-09-18T07:40:59.685 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:40:59.685 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:40:59.685 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:40:59.685 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 126478 2024-09-18T07:40:59.685 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.685 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:59.685 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.685 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:59.685 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.643119+0000", 2024-09-18T07:40:59.685 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.686 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.686 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.686 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:59.686 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.643119+0000", 2024-09-18T07:40:59.686 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.686 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.686 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.686 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:59.686 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.643121+0000", 2024-09-18T07:40:59.686 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6670000000000001e-06 2024-09-18T07:40:59.686 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.686 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.686 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:59.686 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.643190+0000", 2024-09-18T07:40:59.687 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8930000000000006e-05 2024-09-18T07:40:59.687 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.687 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.687 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:59.687 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.643202+0000", 2024-09-18T07:40:59.687 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2099999999999999e-05 2024-09-18T07:40:59.687 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.687 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.687 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:59.687 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.643210+0000", 2024-09-18T07:40:59.687 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0450000000000001e-06 2024-09-18T07:40:59.687 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.688 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:59.688 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.688 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.688 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.688 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.15699.0:126472 9.5 e322/289 9:ae647ed5:::benchmark_data_smithi137_126072_object126471:head v 322'7804, mlcod=322'7797)", 2024-09-18T07:40:59.688 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:40:59.643268+0000", 2024-09-18T07:40:59.688 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0047562350000000002, 2024-09-18T07:40:59.688 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0052816510000000001, 2024-09-18T07:40:59.688 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:59.688 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:59.688 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:40:59.688 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:59.688 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.689 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:59.689 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.643268+0000", 2024-09-18T07:40:59.689 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.689 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.689 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.689 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:59.689 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.643268+0000", 2024-09-18T07:40:59.689 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.689 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.689 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.689 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:59.689 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.643269+0000", 2024-09-18T07:40:59.689 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.11e-06 2024-09-18T07:40:59.689 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.689 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.690 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:59.690 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.643387+0000", 2024-09-18T07:40:59.690 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000118091 2024-09-18T07:40:59.690 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.690 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.690 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:59.690 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.643398+0000", 2024-09-18T07:40:59.690 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1463e-05 2024-09-18T07:40:59.690 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.690 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.690 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:59.690 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.643404+0000", 2024-09-18T07:40:59.690 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7509999999999997e-06 2024-09-18T07:40:59.690 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.690 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.691 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:59.691 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.648280+0000", 2024-09-18T07:40:59.691 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0048757410000000003 2024-09-18T07:40:59.691 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.691 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.691 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:59.691 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.648300+0000", 2024-09-18T07:40:59.691 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9921000000000001e-05 2024-09-18T07:40:59.691 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.691 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:59.691 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.691 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.691 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.691 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop_reply(client.15699.0:126468 9.7 e322/289 ondisk, result = 0)", 2024-09-18T07:40:59.691 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:40:59.646241+0000", 2024-09-18T07:40:59.692 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0017829020000000001, 2024-09-18T07:40:59.692 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002352519, 2024-09-18T07:40:59.692 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:59.692 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:59.692 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:40:59.692 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:59.692 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.692 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:59.692 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.646241+0000", 2024-09-18T07:40:59.692 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.692 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.692 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.692 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:59.692 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.646241+0000", 2024-09-18T07:40:59.693 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.693 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.693 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.693 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:59.693 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.646242+0000", 2024-09-18T07:40:59.693 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5149999999999999e-06 2024-09-18T07:40:59.693 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.693 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.693 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:59.693 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.646247+0000", 2024-09-18T07:40:59.693 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5469999999999998e-06 2024-09-18T07:40:59.693 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.693 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.694 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:59.694 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.646257+0000", 2024-09-18T07:40:59.694 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7620000000000007e-06 2024-09-18T07:40:59.694 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.694 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.694 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:59.694 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.646263+0000", 2024-09-18T07:40:59.694 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6279999999999999e-06 2024-09-18T07:40:59.694 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.694 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.694 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:59.694 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.648002+0000", 2024-09-18T07:40:59.694 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001738912 2024-09-18T07:40:59.694 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.695 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.695 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:59.695 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.648011+0000", 2024-09-18T07:40:59.695 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2649999999999996e-06 2024-09-18T07:40:59.695 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.695 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:59.695 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.695 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.695 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.695 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop_reply(client.15699.0:126471 9.7 e322/289)", 2024-09-18T07:40:59.695 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:40:59.646275+0000", 2024-09-18T07:40:59.695 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.001748589, 2024-09-18T07:40:59.695 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023630529999999999, 2024-09-18T07:40:59.696 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:59.696 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:59.696 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "queued for pg", 2024-09-18T07:40:59.696 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:59.696 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.696 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:59.696 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.646275+0000", 2024-09-18T07:40:59.696 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.696 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.696 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.696 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:59.696 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.646275+0000", 2024-09-18T07:40:59.696 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.697 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.697 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.697 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:59.697 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.646276+0000", 2024-09-18T07:40:59.697 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.63e-07 2024-09-18T07:40:59.697 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.697 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.697 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:59.697 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.646278+0000", 2024-09-18T07:40:59.697 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0480000000000001e-06 2024-09-18T07:40:59.697 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.697 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.697 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:59.698 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.646282+0000", 2024-09-18T07:40:59.698 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3120000000000004e-06 2024-09-18T07:40:59.698 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.698 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.698 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:59.698 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.646286+0000", 2024-09-18T07:40:59.698 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7230000000000002e-06 2024-09-18T07:40:59.698 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.698 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:59.698 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.698 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.698 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.698 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop_reply(client.15699.0:126475 9.f e322/289)", 2024-09-18T07:40:59.698 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:40:59.648548+0000", 2024-09-18T07:40:59.699 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 4294967295.9994755, 2024-09-18T07:40:59.699 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000127945, 2024-09-18T07:40:59.699 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:59.699 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:59.699 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "queued for pg", 2024-09-18T07:40:59.699 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:59.699 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.699 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:59.699 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.648548+0000", 2024-09-18T07:40:59.699 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.699 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.699 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.699 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:59.699 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.648548+0000", 2024-09-18T07:40:59.700 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.700 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.700 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.700 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:59.700 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.648550+0000", 2024-09-18T07:40:59.700 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5400000000000001e-06 2024-09-18T07:40:59.700 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.700 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.700 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:59.700 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.648558+0000", 2024-09-18T07:40:59.700 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5169999999999999e-06 2024-09-18T07:40:59.700 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.700 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.701 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:59.701 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.648572+0000", 2024-09-18T07:40:59.701 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3158e-05 2024-09-18T07:40:59.701 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.701 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.701 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:59.701 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.648579+0000", 2024-09-18T07:40:59.701 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2099999999999996e-06 2024-09-18T07:40:59.701 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.701 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:59.701 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.701 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.701 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.701 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop_reply(client.15699.0:126477 9.7 e322/289)", 2024-09-18T07:40:59.701 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:40:59.648590+0000", 2024-09-18T07:40:59.702 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 4294967295.9994335, 2024-09-18T07:40:59.702 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000120708, 2024-09-18T07:40:59.702 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:59.702 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:59.702 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "queued for pg", 2024-09-18T07:40:59.702 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:59.702 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.702 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:59.702 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.648590+0000", 2024-09-18T07:40:59.702 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.702 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.702 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.702 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:59.702 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.648590+0000", 2024-09-18T07:40:59.702 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.703 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.703 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.703 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:59.703 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.648591+0000", 2024-09-18T07:40:59.703 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1399999999999997e-07 2024-09-18T07:40:59.703 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.703 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.703 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:59.703 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.648593+0000", 2024-09-18T07:40:59.703 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.695e-06 2024-09-18T07:40:59.703 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.703 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.703 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:59.703 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.648598+0000", 2024-09-18T07:40:59.703 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9899999999999997e-06 2024-09-18T07:40:59.704 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.704 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.704 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:59.704 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.648603+0000", 2024-09-18T07:40:59.704 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.437e-06 2024-09-18T07:40:59.704 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.704 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:59.704 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.704 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.704 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.704 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.15699.0:126461 9.d e322/289 9:b244e3c5:::benchmark_data_smithi137_126072_object126460:head v 322'7996, mlcod=322'7993)", 2024-09-18T07:40:59.704 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:40:59.632840+0000", 2024-09-18T07:40:59.704 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.015184318, 2024-09-18T07:40:59.704 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.015915182, 2024-09-18T07:40:59.705 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:59.705 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:59.705 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:40:59.705 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:59.705 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.705 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:59.705 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.632840+0000", 2024-09-18T07:40:59.705 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.705 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.705 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.705 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:59.705 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.632840+0000", 2024-09-18T07:40:59.705 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.705 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.705 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.706 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:59.706 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.632840+0000", 2024-09-18T07:40:59.706 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4399999999999999e-07 2024-09-18T07:40:59.706 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.706 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.706 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:59.706 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.632867+0000", 2024-09-18T07:40:59.706 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6641000000000001e-05 2024-09-18T07:40:59.706 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.706 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.706 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:59.706 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.632874+0000", 2024-09-18T07:40:59.706 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0010000000000004e-06 2024-09-18T07:40:59.707 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.707 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.707 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:59.707 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.632879+0000", 2024-09-18T07:40:59.707 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3129999999999998e-06 2024-09-18T07:40:59.707 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.707 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.707 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:59.707 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.635368+0000", 2024-09-18T07:40:59.707 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002489186 2024-09-18T07:40:59.707 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.707 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.707 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:59.707 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.635387+0000", 2024-09-18T07:40:59.707 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8872000000000002e-05 2024-09-18T07:40:59.708 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.708 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:59.708 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.708 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.708 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.708 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.15699.0:126453 9.5 e322/289 9:a2126164:::benchmark_data_smithi137_126072_object126452:head v 322'7800, mlcod=322'7796)", 2024-09-18T07:40:59.708 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:40:59.632899+0000", 2024-09-18T07:40:59.708 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.015124797000000001, 2024-09-18T07:40:59.708 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.015902191, 2024-09-18T07:40:59.708 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:40:59.708 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:40:59.708 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:40:59.708 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:40:59.709 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.709 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:40:59.709 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.632899+0000", 2024-09-18T07:40:59.709 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.709 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.709 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.709 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:40:59.709 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.632899+0000", 2024-09-18T07:40:59.709 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:40:59.709 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.709 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.709 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:40:59.710 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.632900+0000", 2024-09-18T07:40:59.710 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3200000000000004e-07 2024-09-18T07:40:59.710 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.710 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.710 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:40:59.710 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.632935+0000", 2024-09-18T07:40:59.711 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4751e-05 2024-09-18T07:40:59.711 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.711 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.711 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:40:59.711 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.632941+0000", 2024-09-18T07:40:59.711 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9830000000000001e-06 2024-09-18T07:40:59.711 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.711 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.711 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:40:59.711 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.632955+0000", 2024-09-18T07:40:59.711 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4542000000000001e-05 2024-09-18T07:40:59.711 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.711 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.711 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:40:59.711 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.635928+0000", 2024-09-18T07:40:59.712 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0029730609999999999 2024-09-18T07:40:59.712 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:40:59.712 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:40:59.712 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:40:59.712 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:40:59.635953+0000", 2024-09-18T07:40:59.712 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5332000000000001e-05 2024-09-18T07:40:59.712 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.712 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:40:59.712 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.712 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:40:59.712 INFO:teuthology.orchestra.run.smithi203.stdout: ], 2024-09-18T07:40:59.712 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 15 2024-09-18T07:40:59.713 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:40:59.752 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:40:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:59.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:40:59.854 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:40:59.886 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:40:59.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:40:59.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:40:59.987 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:00.016 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:00.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:00.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:00.091 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 85 15 126990 126975 93.3421 73.875 0.0124712 0.0107089 2024-09-18T07:41:00.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:00.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:00.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:00.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:00.258 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:00.291 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:00.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:00.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:00.393 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:00.450 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:00.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:00.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:00.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:00.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:00.588 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:00.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:00.654 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T07:41:00.689 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:00.732 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:00.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:00.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:00.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:00.875 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:00.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:00.977 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:01.021 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:01.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:01.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:01.048 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:41:01.090 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 86 16 127957 127941 92.9587 60.375 0.00854496 0.0107531 2024-09-18T07:41:01.123 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:01.125 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:00 smithi137 ceph-mon[29408]: pgmap v539: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-18T07:41:01.125 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:00 smithi137 ceph-mon[29408]: osdmap e323: 8 total, 8 up, 8 in 2024-09-18T07:41:01.125 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:00 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:41:01.125 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:00 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:41:01.125 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:00 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:41:01.125 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:00 smithi137 ceph-mon[39175]: pgmap v539: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-18T07:41:01.126 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:00 smithi137 ceph-mon[39175]: osdmap e323: 8 total, 8 up, 8 in 2024-09-18T07:41:01.126 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:00 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:41:01.126 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:00 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:41:01.126 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:00 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:41:01.157 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:01.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:01.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:01.259 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:01.293 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:01.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:01.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:01.336 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:00 smithi203 ceph-mon[32237]: pgmap v539: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-18T07:41:01.336 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:00 smithi203 ceph-mon[32237]: osdmap e323: 8 total, 8 up, 8 in 2024-09-18T07:41:01.336 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:00 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:41:01.337 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:00 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:41:01.337 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:00 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:41:01.395 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:01.428 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:01.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:01.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:01.530 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:01.569 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:01.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:01.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:01.630 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:41:01.631 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:41:01.631 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:41:01.631 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:41:01.631 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:41:01.671 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:01.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:01.709 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:01.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:01.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:01.855 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:01.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:01.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:01.905 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:41:01.905 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:41:01.946 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:41:01.946 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:41:01.957 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:01.985 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:41:01.985 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:41:01.997 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:01.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:01.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:02.017 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:41:02.017 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:41:02.048 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:41:02.048 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:41:02.082 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:41:02.082 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:41:02.090 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 87 16 129172 129156 92.763 75.9375 0.0086507 0.0107758 2024-09-18T07:41:02.099 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:02.104 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:41:02.104 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:41:02.127 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:02.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:02.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:02.145 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:41:02.145 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:01 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2592511710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:02.146 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:01 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2592511710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:02.146 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:41:02.230 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:02.276 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:02.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:02.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:02.378 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:02.412 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:02.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:02.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:01 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2592511710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:02.476 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T07:41:02.476 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T07:41:02.476 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-18T07:41:02.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:02.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:02.548 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:02.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:02.649 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:02.684 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:02.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:02.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:02.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:02.825 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:02.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:02.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:02.963 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:02.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:02.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:03.065 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:03.090 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 88 16 130340 130324 92.5384 73 0.0134599 0.0108018 2024-09-18T07:41:03.101 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:03.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:03.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:03.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:02 smithi137 ceph-mon[29408]: pgmap v541: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 793 op/s 2024-09-18T07:41:03.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:02 smithi137 ceph-mon[39175]: pgmap v541: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 793 op/s 2024-09-18T07:41:03.202 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:03.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:03.231 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:03.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:03.332 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:03.362 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:02 smithi203 ceph-mon[32237]: pgmap v541: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 793 op/s 2024-09-18T07:41:03.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:03.366 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:03.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:03.467 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:03.498 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:03.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:03.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:03.600 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:03.636 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:03.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:03.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:03.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:41:03.738 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:03.758 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:03.775 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:03.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:03.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:03.878 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:03.929 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:03.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:03.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:04.031 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:04.061 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:04.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:04.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:04.093 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 89 16 131421 131405 92.2553 67.5625 0.0116234 0.0108352 2024-09-18T07:41:04.164 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:04.204 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:04.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:04.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:04.306 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:04.336 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:04.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:04.437 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:04.461 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:04.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:04.562 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:04.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:04.595 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:04.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:04.696 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:04.733 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:04.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:04.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:04.835 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:04.855 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:04.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:04.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:04.957 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:04.996 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:04.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:04.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:05.093 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Total time run: 90.008 2024-09-18T07:41:05.093 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Total writes made: 132497 2024-09-18T07:41:05.093 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Write size: 65536 2024-09-18T07:41:05.094 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Object size: 65536 2024-09-18T07:41:05.094 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Bandwidth (MB/sec): 92.0037 2024-09-18T07:41:05.094 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Stddev Bandwidth: 33.9069 2024-09-18T07:41:05.094 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Max bandwidth (MB/sec): 209.812 2024-09-18T07:41:05.094 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Min bandwidth (MB/sec): 8.5 2024-09-18T07:41:05.094 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Average IOPS: 1472 2024-09-18T07:41:05.094 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Stddev IOPS: 542.51 2024-09-18T07:41:05.094 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Max IOPS: 3357 2024-09-18T07:41:05.094 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Min IOPS: 136 2024-09-18T07:41:05.094 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Average Latency(s): 0.0108653 2024-09-18T07:41:05.094 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Stddev Latency(s): 0.0134545 2024-09-18T07:41:05.094 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Max latency(s): 1.0234 2024-09-18T07:41:05.094 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Min latency(s): 0.00110395 2024-09-18T07:41:05.096 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Cleaning up (deleting benchmark objects) 2024-09-18T07:41:05.098 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:05.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:05.137 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:05.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:05.222 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-18T07:41:05.222 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-18T07:41:05.222 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-18T07:41:05.222 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-18T07:41:05.222 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-18T07:41:05.222 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-18T07:41:05.222 INFO:teuthology.orchestra.run.smithi137.stderr:bench [] [] [] [] 2024-09-18T07:41:05.222 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs debug_inject_read_zeros 2024-09-18T07:41:05.222 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs files list 2024-09-18T07:41:05.222 INFO:teuthology.orchestra.run.smithi137.stderr:bluefs stats 2024-09-18T07:41:05.222 INFO:teuthology.orchestra.run.smithi137.stderr:bluestore allocator dump block 2024-09-18T07:41:05.222 INFO:teuthology.orchestra.run.smithi137.stderr:Error EINVAL: invalid command 2024-09-18T07:41:05.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:05.263 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:05.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:05.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:05.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:05.396 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:05.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:05.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:05.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:04 smithi137 ceph-mon[29408]: pgmap v542: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-18T07:41:05.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:04 smithi137 ceph-mon[39175]: pgmap v542: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-18T07:41:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:04 smithi203 ceph-mon[32237]: pgmap v542: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-18T07:41:05.497 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:05.531 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:05.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:05.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:05.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:05.660 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:05.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:05.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:41:05.762 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:05.799 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:05.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:05.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:05.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:05.935 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:05.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:05.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:06.026 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T07:41:06.027 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-18T07:41:06.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:06.080 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:06.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:06.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:06.122 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:06.138 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:06.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:06.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:06.205 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:06.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:06.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:06.328 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:06.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:06.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:06.430 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:06.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:06.469 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:06.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:06.500 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T07:41:06.571 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:06.625 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:06.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:06.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:06.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:06.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:06.771 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:06.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:06.873 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:06.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:06.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:06.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:06.926 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:41:07.017 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:07.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:07.054 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:07.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:07.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:07.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:07.193 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:07.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:07.294 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:07.326 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:07.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:07.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:07.389 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:07 smithi203 ceph-mon[32237]: pgmap v543: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 683 op/s 2024-09-18T07:41:07.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:07 smithi137 ceph-mon[29408]: pgmap v543: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 683 op/s 2024-09-18T07:41:07.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:41:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:41:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:41:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:07 smithi137 ceph-mon[39175]: pgmap v543: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 683 op/s 2024-09-18T07:41:07.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:07.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:07.466 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:07.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:07.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:07.600 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:07.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:07.660 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:41:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:41:07.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:07.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:07.743 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:07.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:07.819 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:41:07.819 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:41:07.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:07.881 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:07.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:07.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:07.983 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:08.016 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:41:08.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:08.020 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:08.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:08.083 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:41:08.083 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:41:08.122 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:08.124 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:41:08.124 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:41:08.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:08.153 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:08.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:08.154 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:41:08.154 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:41:08.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:08 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4202872506' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:08.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:08 smithi137 ceph-mon[29408]: from='client.25828 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:08.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:08 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4202872506' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:08.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:08 smithi137 ceph-mon[39175]: from='client.25828 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:08.186 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:41:08.186 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:41:08.219 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:41:08.220 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:41:08.254 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:08.258 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:41:08.258 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:41:08.290 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:08.291 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:41:08.291 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:41:08.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:08.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:08.330 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:41:08.330 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:41:08.393 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:08.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:08 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4202872506' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:08.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:08 smithi203 ceph-mon[32237]: from='client.25828 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:08.430 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:08.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:08.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:08.530 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T07:41:08.532 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:08.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:08.577 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:08.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:08.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:08.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:08.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:08.809 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:08.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:08.860 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:08.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:08.961 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:08.965 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:41:09.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:09.011 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:09.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:09.113 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:09.153 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:09.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:09.254 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:09.287 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:09 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:09.287 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:09 smithi203 ceph-mon[32237]: pgmap v544: 158 pgs: 158 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 46 KiB/s rd, 59 MiB/s wr, 1.24k op/s 2024-09-18T07:41:09.287 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:09 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:41:09.287 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:09 smithi203 ceph-mon[32237]: from='client.25828 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:41:09.287 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:09 smithi203 ceph-mon[32237]: osdmap e324: 8 total, 8 up, 8 in 2024-09-18T07:41:09.287 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:09 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/95244568' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:09.298 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:09.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:09.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:09.400 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:09.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:09 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:09.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:09 smithi137 ceph-mon[39175]: pgmap v544: 158 pgs: 158 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 46 KiB/s rd, 59 MiB/s wr, 1.24k op/s 2024-09-18T07:41:09.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:09 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:41:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:09 smithi137 ceph-mon[39175]: from='client.25828 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:41:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:09 smithi137 ceph-mon[39175]: osdmap e324: 8 total, 8 up, 8 in 2024-09-18T07:41:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:09 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/95244568' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:09 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:09 smithi137 ceph-mon[29408]: pgmap v544: 158 pgs: 158 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 46 KiB/s rd, 59 MiB/s wr, 1.24k op/s 2024-09-18T07:41:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:09 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:41:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:09 smithi137 ceph-mon[29408]: from='client.25828 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:41:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:09 smithi137 ceph-mon[29408]: osdmap e324: 8 total, 8 up, 8 in 2024-09-18T07:41:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:09 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/95244568' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:09.438 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:09.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:09.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:09.540 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:09.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:09.573 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:09.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:09.610 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:41:09.611 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:41:09.611 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-09-18T07:41:09.611 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:41:09.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:09.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:09.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:09.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:09.809 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:09.849 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:09.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:09.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:09.951 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:09.985 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:09.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:09.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:10.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:10.123 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:10.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:10.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:10.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:10.263 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:10.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:10.311 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T07:41:10.363 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:10.417 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:10.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:10.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:10.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:10.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:10.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:10.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:10.651 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:10.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:10.694 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:10.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:10.755 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:41:10.796 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:10.849 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:10.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:10.927 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:41:10.950 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:10.984 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:10.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:10.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:11.027 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:41:11.027 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:41:11.028 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-18T07:41:11.028 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-18T07:41:11.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:41:11.085 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:11.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:11.134 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:11.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:11.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:10 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:41:11.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:10 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:41:11.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:10 smithi203 ceph-mon[32237]: pgmap v546: 158 pgs: 158 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 45 KiB/s rd, 58 MiB/s wr, 1.22k op/s 2024-09-18T07:41:11.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:10 smithi203 ceph-mon[32237]: osdmap e325: 8 total, 8 up, 8 in 2024-09-18T07:41:11.235 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:11.277 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:11.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:11.370 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:11.379 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:11.394 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:41:11.395 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:41:11.395 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:41:11.395 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:41:11.395 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:41:11.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:10 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:41:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:10 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:41:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:10 smithi137 ceph-mon[39175]: pgmap v546: 158 pgs: 158 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 45 KiB/s rd, 58 MiB/s wr, 1.22k op/s 2024-09-18T07:41:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:10 smithi137 ceph-mon[39175]: osdmap e325: 8 total, 8 up, 8 in 2024-09-18T07:41:11.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:10 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:41:11.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:10 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:41:11.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:10 smithi137 ceph-mon[29408]: pgmap v546: 158 pgs: 158 active+clean; 8.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 45 KiB/s rd, 58 MiB/s wr, 1.22k op/s 2024-09-18T07:41:11.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:10 smithi137 ceph-mon[29408]: osdmap e325: 8 total, 8 up, 8 in 2024-09-18T07:41:11.439 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:11.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:11.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:11.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:11.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:11.587 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:11.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:11.607 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:41:11.689 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:11.717 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:11.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:11.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:11.819 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:11.852 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:11.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:11.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:11.954 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:11.981 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:11.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:11.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:12.016 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:12.087 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:12.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:12.108 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:12.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:12.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:12.243 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:12.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:12.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:12.344 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:12.378 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:12.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:12.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:12.480 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:12.515 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:12.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:12.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:12.614 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T07:41:12.616 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:12.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:12.679 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:12.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:12.706 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:41:12.780 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:12.812 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:12.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:12.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:12.914 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:12.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:12 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:41:12.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:41:12.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:12 smithi137 ceph-mon[29408]: pgmap v548: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 55 KiB/s rd, 27 MiB/s wr, 999 op/s 2024-09-18T07:41:12.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:12 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1310332930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:41:12.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:12 smithi137 ceph-mon[29408]: from='client.35759 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:41:12.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:12 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:41:12.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:41:12.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:12 smithi137 ceph-mon[39175]: pgmap v548: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 55 KiB/s rd, 27 MiB/s wr, 999 op/s 2024-09-18T07:41:12.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:12 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1310332930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:41:12.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:12 smithi137 ceph-mon[39175]: from='client.35759 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:41:12.969 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:12 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:41:12.969 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:41:12.969 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:12 smithi203 ceph-mon[32237]: pgmap v548: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 55 KiB/s rd, 27 MiB/s wr, 999 op/s 2024-09-18T07:41:12.969 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:12 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1310332930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:41:12.970 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:12 smithi203 ceph-mon[32237]: from='client.35759 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:41:12.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:12.972 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:12.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:12.989 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:41:13.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:13.108 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:13.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:13.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:13.205 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:41:13.209 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:41:13.210 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:13.253 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:13.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:13.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:13.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:13.390 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:13.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:13.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:13.492 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:13.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:13.532 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:13.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:13.633 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:13.678 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:13.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:13.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:13.744 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:13 smithi137 ceph-mon[29408]: from='client.35759 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:41:13.744 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:13 smithi137 ceph-mon[29408]: osdmap e326: 8 total, 8 up, 8 in 2024-09-18T07:41:13.744 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:13 smithi137 ceph-mon[29408]: from='client.35765 v1:172.21.15.137:0/3305737794' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:41:13.745 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:13 smithi137 ceph-mon[39175]: from='client.35759 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:41:13.745 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:13 smithi137 ceph-mon[39175]: osdmap e326: 8 total, 8 up, 8 in 2024-09-18T07:41:13.745 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:13 smithi137 ceph-mon[39175]: from='client.35765 v1:172.21.15.137:0/3305737794' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:41:13.753 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:13 smithi203 ceph-mon[32237]: from='client.35759 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:41:13.753 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:13 smithi203 ceph-mon[32237]: osdmap e326: 8 total, 8 up, 8 in 2024-09-18T07:41:13.753 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:13 smithi203 ceph-mon[32237]: from='client.35765 v1:172.21.15.137:0/3305737794' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:41:13.779 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:41:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:41:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:41:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:41:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54188 7.c 7:3726bab8:::benchmark_data_smithi137_74180_object54187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T07:41:13.780 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.151582+0000", 2024-09-18T07:41:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 500.62291691199999, 2024-09-18T07:41:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0152226929999999, 2024-09-18T07:41:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:41:13.781 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54188 2024-09-18T07:41:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.782 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151582+0000", 2024-09-18T07:41:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.783 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151582+0000", 2024-09-18T07:41:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.784 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151583+0000", 2024-09-18T07:41:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6300000000000001e-06 2024-09-18T07:41:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.785 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151619+0000", 2024-09-18T07:41:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5178000000000001e-05 2024-09-18T07:41:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.786 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151627+0000", 2024-09-18T07:41:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2849999999999995e-06 2024-09-18T07:41:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.151633+0000", 2024-09-18T07:41:13.787 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5729999999999996e-06 2024-09-18T07:41:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.152287+0000", 2024-09-18T07:41:13.788 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00065367599999999995 2024-09-18T07:41:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.152442+0000", 2024-09-18T07:41:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015491699999999999 2024-09-18T07:41:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:41:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.152627+0000", 2024-09-18T07:41:13.790 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018544300000000001 2024-09-18T07:41:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:41:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.156875+0000", 2024-09-18T07:41:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0042476329999999998 2024-09-18T07:41:13.791 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.161790+0000", 2024-09-18T07:41:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0049151050000000003 2024-09-18T07:41:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.792 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.161803+0000", 2024-09-18T07:41:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2646999999999999e-05 2024-09-18T07:41:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.793 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.166719+0000", 2024-09-18T07:41:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0049166270000001 2024-09-18T07:41:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.794 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.166804+0000", 2024-09-18T07:41:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4979000000000001e-05 2024-09-18T07:41:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.795 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54193 7.c 7:33299084:::benchmark_data_smithi137_74180_object54192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-18T07:41:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.154898+0000", 2024-09-18T07:41:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 500.61960094400001, 2024-09-18T07:41:13.796 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0124331120000001, 2024-09-18T07:41:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:41:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.797 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54193 2024-09-18T07:41:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.798 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154898+0000", 2024-09-18T07:41:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154898+0000", 2024-09-18T07:41:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154900+0000", 2024-09-18T07:41:13.799 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1100000000000001e-06 2024-09-18T07:41:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154938+0000", 2024-09-18T07:41:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8479999999999997e-05 2024-09-18T07:41:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154957+0000", 2024-09-18T07:41:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8437e-05 2024-09-18T07:41:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.800 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.154965+0000", 2024-09-18T07:41:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1780000000000003e-06 2024-09-18T07:41:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.155029+0000", 2024-09-18T07:41:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4459000000000001e-05 2024-09-18T07:41:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.155164+0000", 2024-09-18T07:41:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013460699999999999 2024-09-18T07:41:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.801 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:41:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.155322+0000", 2024-09-18T07:41:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015807199999999999 2024-09-18T07:41:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:41:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.157099+0000", 2024-09-18T07:41:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0017770570000000001 2024-09-18T07:41:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.161956+0000", 2024-09-18T07:41:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.004857264 2024-09-18T07:41:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.802 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.161962+0000", 2024-09-18T07:41:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9479999999999996e-06 2024-09-18T07:41:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.167294+0000", 2024-09-18T07:41:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.005331441 2024-09-18T07:41:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.167331+0000", 2024-09-18T07:41:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7058999999999999e-05 2024-09-18T07:41:13.803 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54223 7.c 7:39274186:::benchmark_data_smithi137_74180_object54222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:41:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.192250+0000", 2024-09-18T07:41:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 500.58224838000001, 2024-09-18T07:41:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.98501845799999999, 2024-09-18T07:41:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.804 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:13.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54223 2024-09-18T07:41:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192250+0000", 2024-09-18T07:41:13.806 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192250+0000", 2024-09-18T07:41:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192251+0000", 2024-09-18T07:41:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.037e-06 2024-09-18T07:41:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.807 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192317+0000", 2024-09-18T07:41:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5246999999999994e-05 2024-09-18T07:41:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192327+0000", 2024-09-18T07:41:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0464e-05 2024-09-18T07:41:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192335+0000", 2024-09-18T07:41:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2239999999999995e-06 2024-09-18T07:41:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.808 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192402+0000", 2024-09-18T07:41:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6580000000000003e-05 2024-09-18T07:41:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.192414+0000", 2024-09-18T07:41:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2475e-05 2024-09-18T07:41:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.167364+0000", 2024-09-18T07:41:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97494975800000006 2024-09-18T07:41:13.809 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.167483+0000", 2024-09-18T07:41:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000119238 2024-09-18T07:41:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:41:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.167628+0000", 2024-09-18T07:41:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000144709 2024-09-18T07:41:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:41:13.810 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.172321+0000", 2024-09-18T07:41:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0046927879999999998 2024-09-18T07:41:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173324+0000", 2024-09-18T07:41:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001003096 2024-09-18T07:41:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.177200+0000", 2024-09-18T07:41:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0038757449999999999 2024-09-18T07:41:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.811 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:41:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.177233+0000", 2024-09-18T07:41:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3358000000000001e-05 2024-09-18T07:41:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.177269+0000", 2024-09-18T07:41:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5738999999999998e-05 2024-09-18T07:41:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.812 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54246 7.c 7:39613093:::benchmark_data_smithi137_74180_object54245:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:41:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.212409+0000", 2024-09-18T07:41:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 500.56208968700003, 2024-09-18T07:41:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97093596400000004, 2024-09-18T07:41:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54246 2024-09-18T07:41:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.813 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212409+0000", 2024-09-18T07:41:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212409+0000", 2024-09-18T07:41:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212412+0000", 2024-09-18T07:41:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6350000000000002e-06 2024-09-18T07:41:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.814 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212483+0000", 2024-09-18T07:41:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1408999999999993e-05 2024-09-18T07:41:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212497+0000", 2024-09-18T07:41:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3576e-05 2024-09-18T07:41:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212505+0000", 2024-09-18T07:41:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1990000000000006e-06 2024-09-18T07:41:13.815 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212575+0000", 2024-09-18T07:41:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0662999999999998e-05 2024-09-18T07:41:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.212588+0000", 2024-09-18T07:41:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2639e-05 2024-09-18T07:41:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.816 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.169092+0000", 2024-09-18T07:41:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.95650415300000002 2024-09-18T07:41:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.169206+0000", 2024-09-18T07:41:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000113328 2024-09-18T07:41:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:41:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.169341+0000", 2024-09-18T07:41:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013537600000000001 2024-09-18T07:41:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.817 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:41:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173339+0000", 2024-09-18T07:41:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0039977909999999997 2024-09-18T07:41:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.182250+0000", 2024-09-18T07:41:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.008910968 2024-09-18T07:41:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.183281+0000", 2024-09-18T07:41:13.818 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010308940000000001 2024-09-18T07:41:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:41:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.183314+0000", 2024-09-18T07:41:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3791000000000002e-05 2024-09-18T07:41:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.183345+0000", 2024-09-18T07:41:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0542e-05 2024-09-18T07:41:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.819 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54263 7.c 7:35b11aa8:::benchmark_data_smithi137_74180_object54262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:41:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.227767+0000", 2024-09-18T07:41:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 500.54673193899998, 2024-09-18T07:41:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.95717552100000003, 2024-09-18T07:41:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.820 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54263 2024-09-18T07:41:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227767+0000", 2024-09-18T07:41:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227767+0000", 2024-09-18T07:41:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227769+0000", 2024-09-18T07:41:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8640000000000001e-06 2024-09-18T07:41:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227814+0000", 2024-09-18T07:41:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5540000000000001e-05 2024-09-18T07:41:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227829+0000", 2024-09-18T07:41:13.822 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4586000000000001e-05 2024-09-18T07:41:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227837+0000", 2024-09-18T07:41:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3080000000000008e-06 2024-09-18T07:41:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227923+0000", 2024-09-18T07:41:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5531999999999994e-05 2024-09-18T07:41:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.823 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.227934+0000", 2024-09-18T07:41:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1232e-05 2024-09-18T07:41:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.170335+0000", 2024-09-18T07:41:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.94240168099999999 2024-09-18T07:41:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.170404+0000", 2024-09-18T07:41:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9021000000000002e-05 2024-09-18T07:41:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.824 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:41:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.170496+0000", 2024-09-18T07:41:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1472999999999996e-05 2024-09-18T07:41:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:41:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173344+0000", 2024-09-18T07:41:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0028482970000000001 2024-09-18T07:41:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.182293+0000", 2024-09-18T07:41:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0089482840000000008 2024-09-18T07:41:13.825 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.184822+0000", 2024-09-18T07:41:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025292090000000001 2024-09-18T07:41:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:41:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.184880+0000", 2024-09-18T07:41:13.826 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8680000000000001e-05 2024-09-18T07:41:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.829 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.184942+0000", 2024-09-18T07:41:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1814000000000001e-05 2024-09-18T07:41:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.830 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54278 7.c 7:36e62dce:::benchmark_data_smithi137_74180_object54277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:41:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.241010+0000", 2024-09-18T07:41:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 500.533488191, 2024-09-18T07:41:13.831 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.94404960299999996, 2024-09-18T07:41:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.832 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54278 2024-09-18T07:41:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241010+0000", 2024-09-18T07:41:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.833 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241010+0000", 2024-09-18T07:41:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241014+0000", 2024-09-18T07:41:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.112e-06 2024-09-18T07:41:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241049+0000", 2024-09-18T07:41:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5605999999999998e-05 2024-09-18T07:41:13.834 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241067+0000", 2024-09-18T07:41:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8077e-05 2024-09-18T07:41:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241075+0000", 2024-09-18T07:41:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7510000000000008e-06 2024-09-18T07:41:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241153+0000", 2024-09-18T07:41:13.835 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7642e-05 2024-09-18T07:41:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.241167+0000", 2024-09-18T07:41:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3987e-05 2024-09-18T07:41:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.171235+0000", 2024-09-18T07:41:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93006873700000003 2024-09-18T07:41:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.836 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.171328+0000", 2024-09-18T07:41:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2200999999999994e-05 2024-09-18T07:41:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:41:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.171447+0000", 2024-09-18T07:41:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000119874 2024-09-18T07:41:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:41:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.175350+0000", 2024-09-18T07:41:13.837 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0039030359999999999 2024-09-18T07:41:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.182332+0000", 2024-09-18T07:41:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069816849999999996 2024-09-18T07:41:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.184998+0000", 2024-09-18T07:41:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026659689999999998 2024-09-18T07:41:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.838 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:41:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185034+0000", 2024-09-18T07:41:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5694000000000001e-05 2024-09-18T07:41:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185060+0000", 2024-09-18T07:41:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6231999999999999e-05 2024-09-18T07:41:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.839 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54283 7.c 7:30afc400:::benchmark_data_smithi137_74180_object54282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:41:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.244474+0000", 2024-09-18T07:41:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 500.530024836, 2024-09-18T07:41:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.94069020800000003, 2024-09-18T07:41:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54283 2024-09-18T07:41:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244474+0000", 2024-09-18T07:41:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.841 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244474+0000", 2024-09-18T07:41:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244476+0000", 2024-09-18T07:41:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6390000000000001e-06 2024-09-18T07:41:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244521+0000", 2024-09-18T07:41:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4379999999999999e-05 2024-09-18T07:41:13.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.843 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.843 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244538+0000", 2024-09-18T07:41:13.843 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7354e-05 2024-09-18T07:41:13.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.843 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.843 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244548+0000", 2024-09-18T07:41:13.843 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5030000000000003e-06 2024-09-18T07:41:13.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.843 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.843 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244624+0000", 2024-09-18T07:41:13.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6244999999999999e-05 2024-09-18T07:41:13.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.844 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.844 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.844 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.244636+0000", 2024-09-18T07:41:13.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2218000000000001e-05 2024-09-18T07:41:13.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.844 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.844 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.844 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.172361+0000", 2024-09-18T07:41:13.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92772458400000002 2024-09-18T07:41:13.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.845 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.845 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:13.845 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.172455+0000", 2024-09-18T07:41:13.845 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4475999999999995e-05 2024-09-18T07:41:13.845 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.845 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.845 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:41:13.845 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.172532+0000", 2024-09-18T07:41:13.845 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6969000000000001e-05 2024-09-18T07:41:13.845 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.845 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.845 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:41:13.846 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.177152+0000", 2024-09-18T07:41:13.846 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0046201699999999998 2024-09-18T07:41:13.846 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.846 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.846 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.846 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.182361+0000", 2024-09-18T07:41:13.846 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0052089659999999998 2024-09-18T07:41:13.846 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.846 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.846 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.846 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185097+0000", 2024-09-18T07:41:13.846 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027359110000000002 2024-09-18T07:41:13.846 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.847 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.847 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:41:13.847 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185130+0000", 2024-09-18T07:41:13.847 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3031000000000003e-05 2024-09-18T07:41:13.847 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.847 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.847 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.847 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185164+0000", 2024-09-18T07:41:13.847 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3762e-05 2024-09-18T07:41:13.847 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.847 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.847 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.847 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.847 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.848 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54286 7.c 7:3a3f8689:::benchmark_data_smithi137_74180_object54285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:41:13.848 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.245256+0000", 2024-09-18T07:41:13.848 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 500.52924258000002, 2024-09-18T07:41:13.848 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93997832100000001, 2024-09-18T07:41:13.848 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.848 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.848 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:13.848 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.848 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.848 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.848 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54286 2024-09-18T07:41:13.848 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.848 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.849 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245256+0000", 2024-09-18T07:41:13.849 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.849 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245256+0000", 2024-09-18T07:41:13.849 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.849 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245258+0000", 2024-09-18T07:41:13.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.578e-06 2024-09-18T07:41:13.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.850 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.850 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245346+0000", 2024-09-18T07:41:13.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8776000000000005e-05 2024-09-18T07:41:13.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.850 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.850 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245361+0000", 2024-09-18T07:41:13.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4687e-05 2024-09-18T07:41:13.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.851 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.851 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.851 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245368+0000", 2024-09-18T07:41:13.851 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2989999999999999e-06 2024-09-18T07:41:13.851 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.851 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.851 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.851 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245437+0000", 2024-09-18T07:41:13.851 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8869999999999996e-05 2024-09-18T07:41:13.851 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.851 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.852 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.852 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.245447+0000", 2024-09-18T07:41:13.852 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0172e-05 2024-09-18T07:41:13.852 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.852 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.852 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.852 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173386+0000", 2024-09-18T07:41:13.852 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92793816900000003 2024-09-18T07:41:13.852 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.852 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.852 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:13.852 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173452+0000", 2024-09-18T07:41:13.852 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6050999999999995e-05 2024-09-18T07:41:13.852 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.853 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.853 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:41:13.853 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.173533+0000", 2024-09-18T07:41:13.853 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1578999999999999e-05 2024-09-18T07:41:13.853 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.853 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.853 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:41:13.853 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.177162+0000", 2024-09-18T07:41:13.853 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0036291240000000001 2024-09-18T07:41:13.853 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.853 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.853 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.853 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.182390+0000", 2024-09-18T07:41:13.854 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0052279880000000003 2024-09-18T07:41:13.854 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.854 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.854 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.854 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185188+0000", 2024-09-18T07:41:13.854 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027977779999999999 2024-09-18T07:41:13.855 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.855 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.855 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:41:13.855 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185215+0000", 2024-09-18T07:41:13.855 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6764999999999999e-05 2024-09-18T07:41:13.855 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.856 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.856 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.856 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.185234+0000", 2024-09-18T07:41:13.856 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9485000000000001e-05 2024-09-18T07:41:13.856 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.856 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.856 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.856 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.856 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.856 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54303 7.c 7:3c96a443:::benchmark_data_smithi137_74180_object54302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:41:13.857 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.265470+0000", 2024-09-18T07:41:13.857 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 500.50902895799999, 2024-09-18T07:41:13.857 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92892820399999998, 2024-09-18T07:41:13.857 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.857 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.857 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:13.857 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.857 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.857 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.857 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54303 2024-09-18T07:41:13.857 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.857 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.857 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.858 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.858 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265470+0000", 2024-09-18T07:41:13.858 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.858 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.858 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.858 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.858 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265470+0000", 2024-09-18T07:41:13.858 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.858 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.858 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.858 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.858 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265472+0000", 2024-09-18T07:41:13.858 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2400000000000002e-06 2024-09-18T07:41:13.859 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.859 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.859 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.859 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265528+0000", 2024-09-18T07:41:13.859 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6047000000000001e-05 2024-09-18T07:41:13.859 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.859 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.859 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.859 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265539+0000", 2024-09-18T07:41:13.859 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1119e-05 2024-09-18T07:41:13.859 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.859 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.859 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.859 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265547+0000", 2024-09-18T07:41:13.860 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7470000000000005e-06 2024-09-18T07:41:13.860 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.860 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.860 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.860 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265600+0000", 2024-09-18T07:41:13.860 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3062000000000001e-05 2024-09-18T07:41:13.860 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.860 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.860 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.860 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265608+0000", 2024-09-18T07:41:13.860 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8669999999999994e-06 2024-09-18T07:41:13.860 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.861 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.861 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.861 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.174396+0000", 2024-09-18T07:41:13.861 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90878840299999997 2024-09-18T07:41:13.861 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.861 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.862 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:13.862 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.174505+0000", 2024-09-18T07:41:13.862 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108837 2024-09-18T07:41:13.862 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.862 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.862 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:41:13.863 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.174589+0000", 2024-09-18T07:41:13.863 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3825000000000005e-05 2024-09-18T07:41:13.863 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.863 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.863 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:41:13.863 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.178295+0000", 2024-09-18T07:41:13.863 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0037061170000000001 2024-09-18T07:41:13.863 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.864 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.864 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.864 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.188101+0000", 2024-09-18T07:41:13.864 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0098061350000000005 2024-09-18T07:41:13.864 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.864 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.864 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.864 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194313+0000", 2024-09-18T07:41:13.864 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0062117589999999999 2024-09-18T07:41:13.864 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.864 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.864 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:41:13.865 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194353+0000", 2024-09-18T07:41:13.865 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9869000000000001e-05 2024-09-18T07:41:13.865 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.865 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.865 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.865 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194398+0000", 2024-09-18T07:41:13.865 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5176999999999998e-05 2024-09-18T07:41:13.865 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.865 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.865 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.865 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.865 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.865 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54304 7.c 7:36172b18:::benchmark_data_smithi137_74180_object54303:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:41:13.866 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.265725+0000", 2024-09-18T07:41:13.866 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 500.508773631, 2024-09-18T07:41:13.866 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92876516499999995, 2024-09-18T07:41:13.866 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.866 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.866 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:13.866 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.866 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.866 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.866 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54304 2024-09-18T07:41:13.866 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.867 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.867 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.867 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.867 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265725+0000", 2024-09-18T07:41:13.867 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.867 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.867 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.867 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.867 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265725+0000", 2024-09-18T07:41:13.868 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.868 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.868 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.868 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.868 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265726+0000", 2024-09-18T07:41:13.868 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0449999999999999e-06 2024-09-18T07:41:13.868 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.868 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.869 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.869 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265785+0000", 2024-09-18T07:41:13.869 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.94e-05 2024-09-18T07:41:13.869 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.869 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.869 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.869 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265795+0000", 2024-09-18T07:41:13.869 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2550000000000003e-06 2024-09-18T07:41:13.869 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.869 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.869 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.869 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265802+0000", 2024-09-18T07:41:13.869 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8190000000000003e-06 2024-09-18T07:41:13.870 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.870 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.870 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.870 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265846+0000", 2024-09-18T07:41:13.870 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4867999999999998e-05 2024-09-18T07:41:13.870 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.870 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.870 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.870 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.265853+0000", 2024-09-18T07:41:13.870 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0469999999999996e-06 2024-09-18T07:41:13.870 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.870 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.870 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.871 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.175383+0000", 2024-09-18T07:41:13.871 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90952946099999998 2024-09-18T07:41:13.871 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.871 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.871 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:13.871 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.175462+0000", 2024-09-18T07:41:13.871 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9513999999999998e-05 2024-09-18T07:41:13.871 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.871 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.871 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:41:13.871 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.175546+0000", 2024-09-18T07:41:13.871 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3614000000000003e-05 2024-09-18T07:41:13.871 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.871 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.872 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:41:13.872 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.178303+0000", 2024-09-18T07:41:13.872 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027565210000000001 2024-09-18T07:41:13.872 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.872 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.872 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.872 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194058+0000", 2024-09-18T07:41:13.872 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.015755377000000001 2024-09-18T07:41:13.872 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.872 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.872 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.874 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194432+0000", 2024-09-18T07:41:13.874 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00037386800000000002 2024-09-18T07:41:13.874 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.874 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.874 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:41:13.875 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194464+0000", 2024-09-18T07:41:13.875 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2330999999999999e-05 2024-09-18T07:41:13.875 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.875 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.875 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.875 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194490+0000", 2024-09-18T07:41:13.875 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6044999999999999e-05 2024-09-18T07:41:13.875 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.875 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.875 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.875 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.876 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.876 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:54310 7.c 7:30b8e4f0:::benchmark_data_smithi137_74180_object54309:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-18T07:41:13.876 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:53.271051+0000", 2024-09-18T07:41:13.876 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 500.50344738699999, 2024-09-18T07:41:13.876 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92352962000000005, 2024-09-18T07:41:13.876 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.876 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.876 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:13.876 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.876 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.876 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.876 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 54310 2024-09-18T07:41:13.877 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.877 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.877 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.877 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.877 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271051+0000", 2024-09-18T07:41:13.877 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.877 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.877 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.877 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.878 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271051+0000", 2024-09-18T07:41:13.878 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.878 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.878 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.878 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.878 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271053+0000", 2024-09-18T07:41:13.878 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.013e-06 2024-09-18T07:41:13.878 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.878 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.878 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.878 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271110+0000", 2024-09-18T07:41:13.878 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6329000000000002e-05 2024-09-18T07:41:13.878 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.879 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.879 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.879 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271121+0000", 2024-09-18T07:41:13.879 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1761e-05 2024-09-18T07:41:13.879 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.879 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.879 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.879 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271127+0000", 2024-09-18T07:41:13.879 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0390000000000001e-06 2024-09-18T07:41:13.879 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.879 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.879 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.880 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271189+0000", 2024-09-18T07:41:13.880 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1492e-05 2024-09-18T07:41:13.880 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.880 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.880 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.880 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:53.271200+0000", 2024-09-18T07:41:13.880 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1195999999999999e-05 2024-09-18T07:41:13.880 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.880 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.880 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.880 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.176292+0000", 2024-09-18T07:41:13.880 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90509152199999998 2024-09-18T07:41:13.880 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.881 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.881 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:13.881 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.176362+0000", 2024-09-18T07:41:13.881 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9901999999999996e-05 2024-09-18T07:41:13.881 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.881 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.881 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:41:13.881 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.176445+0000", 2024-09-18T07:41:13.881 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3622000000000007e-05 2024-09-18T07:41:13.881 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.881 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.881 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:41:13.882 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.180339+0000", 2024-09-18T07:41:13.882 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0038937350000000002 2024-09-18T07:41:13.882 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.882 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.882 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.882 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194104+0000", 2024-09-18T07:41:13.882 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.013765370000000001 2024-09-18T07:41:13.882 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.882 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.882 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.882 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194523+0000", 2024-09-18T07:41:13.882 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00041917199999999999 2024-09-18T07:41:13.882 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.882 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.883 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:41:13.883 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194551+0000", 2024-09-18T07:41:13.883 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7489999999999999e-05 2024-09-18T07:41:13.883 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.883 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.883 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.883 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:54.194581+0000", 2024-09-18T07:41:13.883 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9977000000000001e-05 2024-09-18T07:41:13.883 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.883 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.883 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.883 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.884 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.884 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153406 7.7 7:e5535fc6:::benchmark_data_smithi137_74180_object48990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-18T07:41:13.884 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.169823+0000", 2024-09-18T07:41:13.884 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 391.60467516699998, 2024-09-18T07:41:13.884 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.083783358, 2024-09-18T07:41:13.884 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.884 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.884 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:41:13.884 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.884 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.884 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.884 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153406 2024-09-18T07:41:13.885 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.885 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.885 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.885 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.885 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169823+0000", 2024-09-18T07:41:13.885 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.885 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.885 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.885 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.885 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169823+0000", 2024-09-18T07:41:13.885 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.885 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.885 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.886 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.886 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169826+0000", 2024-09-18T07:41:13.886 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1229999999999998e-06 2024-09-18T07:41:13.886 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.886 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.886 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.886 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169833+0000", 2024-09-18T07:41:13.886 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6869999999999994e-06 2024-09-18T07:41:13.886 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.886 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.886 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.886 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169851+0000", 2024-09-18T07:41:13.886 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7978999999999998e-05 2024-09-18T07:41:13.887 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.887 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.887 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.887 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169867+0000", 2024-09-18T07:41:13.887 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5610000000000001e-05 2024-09-18T07:41:13.887 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.887 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.887 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.887 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169944+0000", 2024-09-18T07:41:13.887 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6998999999999999e-05 2024-09-18T07:41:13.887 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.887 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.888 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:13.888 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.170029+0000", 2024-09-18T07:41:13.888 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4765999999999994e-05 2024-09-18T07:41:13.888 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.888 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.888 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:41:13.888 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.170285+0000", 2024-09-18T07:41:13.888 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025587199999999998 2024-09-18T07:41:13.888 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.888 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.888 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:41:13.888 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.171936+0000", 2024-09-18T07:41:13.888 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001651189 2024-09-18T07:41:13.889 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.889 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.889 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.889 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.194310+0000", 2024-09-18T07:41:13.889 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.022374616999999999 2024-09-18T07:41:13.889 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.889 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.889 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.889 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238955+0000", 2024-09-18T07:41:13.889 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.044644716000000001 2024-09-18T07:41:13.889 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.889 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.889 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.890 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238967+0000", 2024-09-18T07:41:13.890 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1933000000000001e-05 2024-09-18T07:41:13.890 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.890 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.890 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.890 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.253578+0000", 2024-09-18T07:41:13.890 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0146105599999999 2024-09-18T07:41:13.890 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.890 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.891 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.891 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.253607+0000", 2024-09-18T07:41:13.891 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9306999999999999e-05 2024-09-18T07:41:13.891 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.891 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.891 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.891 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.891 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.891 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153419 7.17 7:e887289b:::benchmark_data_smithi137_74180_object49003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-18T07:41:13.891 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.193652+0000", 2024-09-18T07:41:13.891 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 391.58084634300002, 2024-09-18T07:41:13.891 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0605730499999999, 2024-09-18T07:41:13.891 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.892 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.892 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:41:13.892 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.892 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.892 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.892 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153419 2024-09-18T07:41:13.892 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.892 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.892 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.892 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.892 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193652+0000", 2024-09-18T07:41:13.892 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.893 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.893 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.893 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.893 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193652+0000", 2024-09-18T07:41:13.893 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.893 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.893 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.893 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.893 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193654+0000", 2024-09-18T07:41:13.893 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.683e-06 2024-09-18T07:41:13.893 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.893 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.893 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.894 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193658+0000", 2024-09-18T07:41:13.894 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6619999999999998e-06 2024-09-18T07:41:13.894 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.894 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.894 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.894 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193667+0000", 2024-09-18T07:41:13.894 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0389999999999993e-06 2024-09-18T07:41:13.894 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.894 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.894 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.894 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193671+0000", 2024-09-18T07:41:13.894 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6789999999999996e-06 2024-09-18T07:41:13.894 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.895 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.895 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.895 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193715+0000", 2024-09-18T07:41:13.895 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3186000000000003e-05 2024-09-18T07:41:13.895 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.895 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.895 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:13.895 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193799+0000", 2024-09-18T07:41:13.895 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4172999999999995e-05 2024-09-18T07:41:13.895 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.896 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.896 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:41:13.896 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.194037+0000", 2024-09-18T07:41:13.896 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023850299999999999 2024-09-18T07:41:13.896 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.896 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.896 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:41:13.896 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.195050+0000", 2024-09-18T07:41:13.896 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010123269999999999 2024-09-18T07:41:13.896 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.897 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.897 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.897 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.198824+0000", 2024-09-18T07:41:13.897 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0037742840000000001 2024-09-18T07:41:13.897 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.897 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.897 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.897 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238955+0000", 2024-09-18T07:41:13.897 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.040130714999999997 2024-09-18T07:41:13.897 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.897 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.898 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.898 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238967+0000", 2024-09-18T07:41:13.898 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2665e-05 2024-09-18T07:41:13.898 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.898 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.898 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.898 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254201+0000", 2024-09-18T07:41:13.898 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0152341279999999 2024-09-18T07:41:13.898 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.898 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.898 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.898 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254225+0000", 2024-09-18T07:41:13.898 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4006e-05 2024-09-18T07:41:13.899 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.899 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.899 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.899 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.899 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.899 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153453 7.7 7.146f22e7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:41:13.899 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.282355+0000", 2024-09-18T07:41:13.899 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 391.49214345600001, 2024-09-18T07:41:13.900 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97169607199999997, 2024-09-18T07:41:13.900 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.900 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.900 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:41:13.900 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.900 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.900 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.900 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153453 2024-09-18T07:41:13.900 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.900 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.900 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.901 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.901 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282355+0000", 2024-09-18T07:41:13.901 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.901 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.901 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.901 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.901 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282355+0000", 2024-09-18T07:41:13.901 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.901 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.901 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.902 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.902 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282357+0000", 2024-09-18T07:41:13.902 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2110000000000001e-06 2024-09-18T07:41:13.902 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.902 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.902 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.902 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282363+0000", 2024-09-18T07:41:13.902 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6740000000000002e-06 2024-09-18T07:41:13.902 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.902 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.902 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.902 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282377+0000", 2024-09-18T07:41:13.902 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4039000000000001e-05 2024-09-18T07:41:13.903 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.903 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.903 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.903 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282384+0000", 2024-09-18T07:41:13.903 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5769999999999999e-06 2024-09-18T07:41:13.903 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.903 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.903 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.903 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282438+0000", 2024-09-18T07:41:13.903 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4083999999999999e-05 2024-09-18T07:41:13.903 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.903 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.903 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.903 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282445+0000", 2024-09-18T07:41:13.904 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7030000000000006e-06 2024-09-18T07:41:13.904 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.904 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.904 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.904 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254044+0000", 2024-09-18T07:41:13.904 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97159860200000003 2024-09-18T07:41:13.904 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.904 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.904 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.904 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254051+0000", 2024-09-18T07:41:13.904 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.182e-06 2024-09-18T07:41:13.905 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.905 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.905 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.905 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.905 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.905 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153481 7.7 7.33bf19e7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:41:13.905 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.317589+0000", 2024-09-18T07:41:13.905 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 391.45690945199999, 2024-09-18T07:41:13.905 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93650330299999995, 2024-09-18T07:41:13.905 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.905 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.905 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:41:13.906 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.906 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.906 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.906 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153481 2024-09-18T07:41:13.906 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.906 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.906 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.906 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.906 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317589+0000", 2024-09-18T07:41:13.906 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.906 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.906 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.906 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.907 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317589+0000", 2024-09-18T07:41:13.907 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.907 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.907 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.907 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.907 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317590+0000", 2024-09-18T07:41:13.907 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6499999999999999e-07 2024-09-18T07:41:13.907 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.907 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.907 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.907 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317593+0000", 2024-09-18T07:41:13.907 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7870000000000002e-06 2024-09-18T07:41:13.907 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.908 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.908 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.908 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317604+0000", 2024-09-18T07:41:13.908 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0953e-05 2024-09-18T07:41:13.908 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.908 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.908 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.908 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317609+0000", 2024-09-18T07:41:13.908 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4099999999999999e-06 2024-09-18T07:41:13.909 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.909 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.909 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.909 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317654+0000", 2024-09-18T07:41:13.909 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5333000000000001e-05 2024-09-18T07:41:13.909 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.909 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.909 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.909 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317661+0000", 2024-09-18T07:41:13.909 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8310000000000003e-06 2024-09-18T07:41:13.909 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.909 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.909 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.910 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254083+0000", 2024-09-18T07:41:13.910 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93642186500000002 2024-09-18T07:41:13.910 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.910 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.910 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.910 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254093+0000", 2024-09-18T07:41:13.911 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4590000000000004e-06 2024-09-18T07:41:13.911 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.911 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.911 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.911 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.911 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.912 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153485 7.7 7.c6dac07 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:41:13.912 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.328201+0000", 2024-09-18T07:41:13.912 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 391.44629751500003, 2024-09-18T07:41:13.912 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.925907068, 2024-09-18T07:41:13.912 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.912 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.913 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:41:13.913 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.913 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.913 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.913 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153485 2024-09-18T07:41:13.913 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.913 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.913 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.913 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.913 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328201+0000", 2024-09-18T07:41:13.913 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.913 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.914 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.914 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.914 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328201+0000", 2024-09-18T07:41:13.914 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.914 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.914 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.914 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.914 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328203+0000", 2024-09-18T07:41:13.914 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3769999999999999e-06 2024-09-18T07:41:13.914 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.914 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.914 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.914 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328207+0000", 2024-09-18T07:41:13.915 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5120000000000002e-06 2024-09-18T07:41:13.915 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.915 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.915 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.915 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328217+0000", 2024-09-18T07:41:13.915 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0438000000000001e-05 2024-09-18T07:41:13.915 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.915 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.915 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.915 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328223+0000", 2024-09-18T07:41:13.915 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7130000000000002e-06 2024-09-18T07:41:13.915 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.915 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.916 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.916 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328270+0000", 2024-09-18T07:41:13.916 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7243000000000002e-05 2024-09-18T07:41:13.916 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.916 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.916 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.916 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328277+0000", 2024-09-18T07:41:13.916 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9229999999999996e-06 2024-09-18T07:41:13.916 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.916 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.916 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.916 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254104+0000", 2024-09-18T07:41:13.917 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92582690099999998 2024-09-18T07:41:13.917 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.917 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.917 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.917 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254108+0000", 2024-09-18T07:41:13.917 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9609999999999999e-06 2024-09-18T07:41:13.917 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.917 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.917 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.917 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.917 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.917 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153505 7.7 7.fb8b6b67 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:41:13.918 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.350178+0000", 2024-09-18T07:41:13.918 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 391.42432068599999, 2024-09-18T07:41:13.918 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90398860000000003, 2024-09-18T07:41:13.918 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.918 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.918 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:41:13.918 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.918 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.918 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.918 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153505 2024-09-18T07:41:13.918 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.918 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.918 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.918 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.919 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350178+0000", 2024-09-18T07:41:13.919 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.919 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.919 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.919 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.919 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350178+0000", 2024-09-18T07:41:13.919 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.919 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.919 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.919 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.919 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350179+0000", 2024-09-18T07:41:13.919 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.454e-06 2024-09-18T07:41:13.919 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.920 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.920 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.920 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350183+0000", 2024-09-18T07:41:13.920 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8480000000000003e-06 2024-09-18T07:41:13.920 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.920 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.920 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.920 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350192+0000", 2024-09-18T07:41:13.920 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0899999999999994e-06 2024-09-18T07:41:13.920 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.920 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.920 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.921 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350200+0000", 2024-09-18T07:41:13.921 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.328e-06 2024-09-18T07:41:13.921 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.921 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.921 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.921 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350252+0000", 2024-09-18T07:41:13.921 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2589000000000001e-05 2024-09-18T07:41:13.921 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.921 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.921 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.921 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350263+0000", 2024-09-18T07:41:13.921 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0831e-05 2024-09-18T07:41:13.922 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.922 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.922 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.922 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254124+0000", 2024-09-18T07:41:13.922 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90386114100000003 2024-09-18T07:41:13.922 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.922 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.922 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.922 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254167+0000", 2024-09-18T07:41:13.922 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2318999999999999e-05 2024-09-18T07:41:13.922 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.922 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.922 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.922 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.923 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.923 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153519 7.17 7.5ef16c17 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:41:13.923 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.363819+0000", 2024-09-18T07:41:13.923 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 391.41067993299998, 2024-09-18T07:41:13.923 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.89046919899999999, 2024-09-18T07:41:13.923 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.923 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.923 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:41:13.923 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.923 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.923 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.923 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153519 2024-09-18T07:41:13.923 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.924 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.924 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.924 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.924 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363819+0000", 2024-09-18T07:41:13.924 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.924 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.924 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.924 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.924 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363819+0000", 2024-09-18T07:41:13.924 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.924 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.924 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.925 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.925 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363820+0000", 2024-09-18T07:41:13.925 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.325e-06 2024-09-18T07:41:13.925 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.925 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.925 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.925 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363824+0000", 2024-09-18T07:41:13.925 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2459999999999997e-06 2024-09-18T07:41:13.925 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.925 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.925 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.925 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363834+0000", 2024-09-18T07:41:13.925 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6399999999999992e-06 2024-09-18T07:41:13.925 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.926 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.926 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.926 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363838+0000", 2024-09-18T07:41:13.926 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2509999999999996e-06 2024-09-18T07:41:13.926 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.926 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.926 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.926 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363900+0000", 2024-09-18T07:41:13.926 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1902000000000005e-05 2024-09-18T07:41:13.926 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.926 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.926 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.926 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363910+0000", 2024-09-18T07:41:13.927 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6879999999999994e-06 2024-09-18T07:41:13.927 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.927 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.927 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.927 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254254+0000", 2024-09-18T07:41:13.927 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.89034411800000002 2024-09-18T07:41:13.927 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.927 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.927 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.927 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254288+0000", 2024-09-18T07:41:13.927 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4029000000000002e-05 2024-09-18T07:41:13.927 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.927 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.928 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.928 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.928 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.928 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:162338 7.c 7:39946bc9:::benchmark_data_smithi137_74180_object57922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:41:13.928 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:01.426143+0000", 2024-09-18T07:41:13.928 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 372.34835612199998, 2024-09-18T07:41:13.928 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0750540070000001, 2024-09-18T07:41:13.928 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.928 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.928 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:13.928 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.928 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.929 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.929 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 162338 2024-09-18T07:41:13.929 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.929 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.929 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.929 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.929 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426143+0000", 2024-09-18T07:41:13.929 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.929 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.929 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.929 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.929 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426143+0000", 2024-09-18T07:41:13.929 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.930 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.930 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.930 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.930 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426145+0000", 2024-09-18T07:41:13.930 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4059999999999999e-06 2024-09-18T07:41:13.930 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.930 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.930 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.930 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426151+0000", 2024-09-18T07:41:13.930 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1260000000000003e-06 2024-09-18T07:41:13.930 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.931 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.931 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.931 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426174+0000", 2024-09-18T07:41:13.931 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3255e-05 2024-09-18T07:41:13.931 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.931 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.931 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.931 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426182+0000", 2024-09-18T07:41:13.931 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4189999999999996e-06 2024-09-18T07:41:13.931 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.932 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.932 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.932 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426256+0000", 2024-09-18T07:41:13.932 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4713000000000001e-05 2024-09-18T07:41:13.932 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.932 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.932 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.932 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426270+0000", 2024-09-18T07:41:13.932 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3405e-05 2024-09-18T07:41:13.932 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.933 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.933 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.933 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.491167+0000", 2024-09-18T07:41:13.933 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.064897582 2024-09-18T07:41:13.933 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.933 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.933 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:13.933 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.491280+0000", 2024-09-18T07:41:13.933 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011243099999999999 2024-09-18T07:41:13.933 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.933 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.933 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3,7", 2024-09-18T07:41:13.934 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.491357+0000", 2024-09-18T07:41:13.934 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7385999999999994e-05 2024-09-18T07:41:13.934 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.934 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.934 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.934 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496060+0000", 2024-09-18T07:41:13.935 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0047024600000000003 2024-09-18T07:41:13.935 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.935 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.935 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:41:13.935 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496072+0000", 2024-09-18T07:41:13.935 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2294e-05 2024-09-18T07:41:13.936 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.936 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.936 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.936 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496109+0000", 2024-09-18T07:41:13.936 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7218999999999997e-05 2024-09-18T07:41:13.936 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.936 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.936 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.936 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501071+0000", 2024-09-18T07:41:13.936 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0049614660000000003 2024-09-18T07:41:13.936 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.936 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.936 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:41:13.937 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501111+0000", 2024-09-18T07:41:13.937 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0776999999999999e-05 2024-09-18T07:41:13.937 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.937 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.937 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.937 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501197+0000", 2024-09-18T07:41:13.937 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5068000000000006e-05 2024-09-18T07:41:13.938 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.938 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.938 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.938 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.938 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.938 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:162350 7.c 7:3843094b:::benchmark_data_smithi137_74180_object57934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:41:13.938 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:01.433525+0000", 2024-09-18T07:41:13.938 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 372.34097331200002, 2024-09-18T07:41:13.938 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0678760330000001, 2024-09-18T07:41:13.939 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:13.939 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:13.939 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:13.939 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:13.939 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:41:13.939 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:13.939 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 162350 2024-09-18T07:41:13.939 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.939 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:13.939 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.939 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:13.939 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433525+0000", 2024-09-18T07:41:13.939 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.940 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.940 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.940 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:13.940 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433525+0000", 2024-09-18T07:41:13.940 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:13.940 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.940 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.940 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:13.940 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433527+0000", 2024-09-18T07:41:13.940 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.812e-06 2024-09-18T07:41:13.940 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.940 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.940 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:13.941 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433533+0000", 2024-09-18T07:41:13.941 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2920000000000001e-06 2024-09-18T07:41:13.941 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.941 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.941 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:13.941 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433544+0000", 2024-09-18T07:41:13.941 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0339999999999999e-05 2024-09-18T07:41:13.941 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.941 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.941 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:13.941 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433551+0000", 2024-09-18T07:41:13.942 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6490000000000008e-06 2024-09-18T07:41:13.942 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.942 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.942 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.942 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433620+0000", 2024-09-18T07:41:13.942 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8957999999999999e-05 2024-09-18T07:41:13.942 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.942 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.942 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:41:13.942 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433631+0000", 2024-09-18T07:41:13.942 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0397000000000001e-05 2024-09-18T07:41:13.942 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.942 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.943 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:13.943 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.493805+0000", 2024-09-18T07:41:13.943 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0601744559999999 2024-09-18T07:41:13.943 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.943 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.943 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:13.943 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.493907+0000", 2024-09-18T07:41:13.943 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000101377 2024-09-18T07:41:13.943 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.943 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.943 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3,7", 2024-09-18T07:41:13.943 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.493992+0000", 2024-09-18T07:41:13.943 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5672000000000002e-05 2024-09-18T07:41:13.944 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.944 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.944 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.944 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496146+0000", 2024-09-18T07:41:13.944 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021537470000000001 2024-09-18T07:41:13.944 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.944 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.944 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:41:13.944 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.498397+0000", 2024-09-18T07:41:13.944 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002250742 2024-09-18T07:41:13.944 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.944 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.944 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.945 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501029+0000", 2024-09-18T07:41:13.945 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026321780000000002 2024-09-18T07:41:13.945 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.945 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.945 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:13.945 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501274+0000", 2024-09-18T07:41:13.945 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000245472 2024-09-18T07:41:13.945 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.945 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.945 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:41:13.945 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501321+0000", 2024-09-18T07:41:13.945 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6405e-05 2024-09-18T07:41:13.945 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:13.946 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:13.946 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:41:13.946 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501401+0000", 2024-09-18T07:41:13.946 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0536000000000003e-05 2024-09-18T07:41:13.946 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.946 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.946 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.946 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:13.946 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:13.946 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:41:13.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:13.948 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:13.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:14.049 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:14.080 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:14.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:14.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:14.182 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:14.219 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:14.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:14.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:14.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:14.359 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:14.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:14.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:14.493 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:14.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:14.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:14.595 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:14.631 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:14.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:14.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:14.670 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:14.733 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:14.758 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T07:41:14.788 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:14.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:14.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:14.803 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:14 smithi137 ceph-mon[29408]: 6.1b scrub starts 2024-09-18T07:41:14.803 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:14 smithi137 ceph-mon[29408]: 6.1b scrub ok 2024-09-18T07:41:14.803 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:14 smithi137 ceph-mon[29408]: pgmap v550: 158 pgs: 158 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-18T07:41:14.803 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:14 smithi137 ceph-mon[39175]: 6.1b scrub starts 2024-09-18T07:41:14.803 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:14 smithi137 ceph-mon[39175]: 6.1b scrub ok 2024-09-18T07:41:14.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:14 smithi137 ceph-mon[39175]: pgmap v550: 158 pgs: 158 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-18T07:41:14.890 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:14.940 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:14.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:14.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:14.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:14 smithi203 ceph-mon[32237]: 6.1b scrub starts 2024-09-18T07:41:14.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:14 smithi203 ceph-mon[32237]: 6.1b scrub ok 2024-09-18T07:41:14.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:14 smithi203 ceph-mon[32237]: pgmap v550: 158 pgs: 158 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-18T07:41:15.042 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:15.083 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:15.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:15.184 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:15.219 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:41:15.229 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:15.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:15.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:15.331 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:15.372 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:15.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:15.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:15.474 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:15.509 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:15.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:15.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:15.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:15.656 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:15.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:15.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:15.757 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:15.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:15 smithi137 ceph-mon[29408]: 5.1e scrub starts 2024-09-18T07:41:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:15 smithi137 ceph-mon[29408]: 5.1e scrub ok 2024-09-18T07:41:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:15 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:41:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:15 smithi137 ceph-mon[29408]: osdmap e327: 8 total, 8 up, 8 in 2024-09-18T07:41:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:15 smithi137 ceph-mon[39175]: 5.1e scrub starts 2024-09-18T07:41:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:15 smithi137 ceph-mon[39175]: 5.1e scrub ok 2024-09-18T07:41:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:15 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:41:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:15 smithi137 ceph-mon[39175]: osdmap e327: 8 total, 8 up, 8 in 2024-09-18T07:41:15.792 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:15.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:15.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:15.838 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:41:15.839 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:41:15.840 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:41:15.841 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:41:15.841 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.841 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:146080 9.3 9:c44690b1:::benchmark_data_smithi137_126072_object13580:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-18T07:41:15.841 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:15.829888+0000", 2024-09-18T07:41:15.841 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0074365569999999999, 2024-09-18T07:41:15.841 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00746446, 2024-09-18T07:41:15.841 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:15.841 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:15.841 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:41:15.841 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:15.841 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:41:15.841 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:41:15.842 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 146080 2024-09-18T07:41:15.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.842 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:15.842 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.842 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:15.842 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.829888+0000", 2024-09-18T07:41:15.842 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:15.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.842 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.842 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:15.842 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.829888+0000", 2024-09-18T07:41:15.842 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:15.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.843 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:15.843 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.829889+0000", 2024-09-18T07:41:15.843 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1319999999999999e-06 2024-09-18T07:41:15.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.843 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:15.843 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.829892+0000", 2024-09-18T07:41:15.843 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.579e-06 2024-09-18T07:41:15.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.843 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.843 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:15.843 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.829900+0000", 2024-09-18T07:41:15.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3540000000000002e-06 2024-09-18T07:41:15.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.844 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.844 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:15.844 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.829905+0000", 2024-09-18T07:41:15.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.169e-06 2024-09-18T07:41:15.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.844 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.844 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:15.844 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.830434+0000", 2024-09-18T07:41:15.844 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00052891799999999997 2024-09-18T07:41:15.844 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.844 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.845 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:15.845 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.830529+0000", 2024-09-18T07:41:15.845 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5333000000000004e-05 2024-09-18T07:41:15.845 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.845 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.845 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:41:15.845 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.830627+0000", 2024-09-18T07:41:15.845 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8040000000000003e-05 2024-09-18T07:41:15.845 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.845 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.845 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:41:15.846 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.833544+0000", 2024-09-18T07:41:15.846 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0029167030000000001 2024-09-18T07:41:15.846 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.846 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.846 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:15.846 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.834299+0000", 2024-09-18T07:41:15.846 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00075535399999999996 2024-09-18T07:41:15.846 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:15.846 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:15.846 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:15.846 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.847 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.847 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:146085 9.3 9:c859957f:::benchmark_data_smithi137_126072_object13585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-18T07:41:15.847 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:15.834197+0000", 2024-09-18T07:41:15.847 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.003126906, 2024-09-18T07:41:15.847 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0032671470000000002, 2024-09-18T07:41:15.847 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:15.847 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:15.847 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:41:15.847 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:41:15.847 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:41:15.847 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:41:15.847 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 146085 2024-09-18T07:41:15.848 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.848 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:15.848 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.848 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:15.848 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.834197+0000", 2024-09-18T07:41:15.848 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:15.848 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.848 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.848 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:15.848 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.834197+0000", 2024-09-18T07:41:15.848 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:15.848 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.848 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:15.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.834199+0000", 2024-09-18T07:41:15.849 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6279999999999999e-06 2024-09-18T07:41:15.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.849 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:15.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.834204+0000", 2024-09-18T07:41:15.849 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1209999999999998e-06 2024-09-18T07:41:15.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.849 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.849 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:15.849 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.834217+0000", 2024-09-18T07:41:15.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3475000000000001e-05 2024-09-18T07:41:15.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.850 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:15.850 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.834225+0000", 2024-09-18T07:41:15.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3690000000000001e-06 2024-09-18T07:41:15.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.850 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:15.850 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.834342+0000", 2024-09-18T07:41:15.850 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000116832 2024-09-18T07:41:15.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.850 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.850 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:15.851 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.834451+0000", 2024-09-18T07:41:15.851 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010891900000000001 2024-09-18T07:41:15.851 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.851 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.851 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:41:15.851 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.834557+0000", 2024-09-18T07:41:15.851 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000106611 2024-09-18T07:41:15.851 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.851 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.851 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:41:15.851 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.836730+0000", 2024-09-18T07:41:15.851 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021730360000000002 2024-09-18T07:41:15.851 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:15.852 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:15.852 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:15.852 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.852 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.852 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.15699.0:146089 9.a e327/289 9:59944503:::benchmark_data_smithi137_126072_object13589:head v 327'9190, mlcod=327'9188)", 2024-09-18T07:41:15.852 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:15.837430+0000", 2024-09-18T07:41:15.852 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 4294967295.9998946, 2024-09-18T07:41:15.852 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012553600000000001, 2024-09-18T07:41:15.852 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:41:15.852 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:41:15.852 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:41:15.852 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:41:15.853 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.853 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:41:15.853 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.837430+0000", 2024-09-18T07:41:15.853 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:15.853 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.853 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.853 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:41:15.853 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.837430+0000", 2024-09-18T07:41:15.853 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:41:15.853 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.853 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.854 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:41:15.854 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.837431+0000", 2024-09-18T07:41:15.854 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3179999999999999e-06 2024-09-18T07:41:15.854 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.854 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.854 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:41:15.854 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.837438+0000", 2024-09-18T07:41:15.854 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9310000000000002e-06 2024-09-18T07:41:15.854 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.854 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.854 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:41:15.854 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.837451+0000", 2024-09-18T07:41:15.854 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3728e-05 2024-09-18T07:41:15.854 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.855 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.855 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:41:15.855 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.837458+0000", 2024-09-18T07:41:15.855 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.861e-06 2024-09-18T07:41:15.855 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.855 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.855 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:41:15.855 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.837503+0000", 2024-09-18T07:41:15.855 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4505999999999997e-05 2024-09-18T07:41:15.855 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:41:15.855 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:41:15.855 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:41:15.855 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:15.837516+0000", 2024-09-18T07:41:15.856 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3016e-05 2024-09-18T07:41:15.856 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:15.856 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:41:15.856 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:15.856 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:41:15.856 INFO:teuthology.orchestra.run.smithi203.stdout: ], 2024-09-18T07:41:15.856 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 3 2024-09-18T07:41:15.856 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:41:15.876 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:41:15.876 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:41:15.895 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:15.906 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:41:15.906 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:41:15.930 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:15.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:15.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:15.937 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:41:15.938 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:41:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:15 smithi203 ceph-mon[32237]: 5.1e scrub starts 2024-09-18T07:41:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:15 smithi203 ceph-mon[32237]: 5.1e scrub ok 2024-09-18T07:41:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:15 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:41:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:15 smithi203 ceph-mon[32237]: osdmap e327: 8 total, 8 up, 8 in 2024-09-18T07:41:15.973 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:41:15.973 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:41:16.009 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:41:16.010 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:41:16.031 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:16.046 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:41:16.046 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:41:16.070 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:16.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:16.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:16.077 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:41:16.078 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:41:16.172 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:16.212 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:16.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:16.314 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:16.356 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:16.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:16.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:16.359 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":549,"stamp":"2024-09-18T07:41:12.605713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8110742067,"num_objects":123951,"num_object_clones":0,"num_object_copies":371853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123951,"num_whiteouts":0,"num_read":1509,"num_read_kb":1261,"num_write":141966,"num_write_kb":8479854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":5625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80592,"ondisk_log_size":80592,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":477,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":27230776,"kb_used_data":24142056,"kb_used_omap":12,"kb_used_meta":3088371,"kb_avail":722730440,"statfs":{"total":767960285184,"available":740075970560,"internally_reserved":0,"allocated":24721465344,"data_stored":24715781745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":3162492464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,4,4,2,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-345767912,"num_objects":-5275,"num_object_clones":0,"num_object_copies":-15825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5275,"num_whiteouts":0,"num_read":440,"num_read_kb":440,"num_write":12503,"num_write_kb":222081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.955154"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":564,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.978227+0000","last_change":"2024-09-18T07:39:36.700902+0000","last_active":"2024-09-18T07:41:09.978227+0000","last_peered":"2024-09-18T07:41:09.978227+0000","last_clean":"2024-09-18T07:41:09.978227+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:41:09.978227+0000","last_undegraded":"2024-09-18T07:41:09.978227+0000","last_fullsized":"2024-09-18T07:41:09.978227+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:36.700832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:36.700832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"323'15","reported_seq":648,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.993260+0000","last_change":"2024-09-18T07:40:17.270061+0000","last_active":"2024-09-18T07:41:09.993260+0000","last_peered":"2024-09-18T07:41:09.993260+0000","last_clean":"2024-09-18T07:41:09.993260+0000","last_became_active":"2024-09-18T07:37:43.061125+0000","last_became_peered":"2024-09-18T07:37:43.061125+0000","last_unstale":"2024-09-18T07:41:09.993260+0000","last_undegraded":"2024-09-18T07:41:09.993260+0000","last_fullsized":"2024-09-18T07:41:09.993260+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:17.269977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:17.269977+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:29.059025+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":508,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.988508+0000","last_change":"2024-09-18T07:40:10.282613+0000","last_active":"2024-09-18T07:41:09.988508+0000","last_peered":"2024-09-18T07:41:09.988508+0000","last_clean":"2024-09-18T07:41:09.988508+0000","last_became_active":"2024-09-18T07:36:13.128805+0000","last_became_peered":"2024-09-18T07:36:13.128805+0000","last_unstale":"2024-09-18T07:41:09.988508+0000","last_undegraded":"2024-09-18T07:41:09.988508+0000","last_fullsized":"2024-09-18T07:41:09.988508+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:10.282540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:10.282540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:33.586865+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":548,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.993502+0000","last_change":"2024-09-18T07:40:47.287019+0000","last_active":"2024-09-18T07:41:09.993502+0000","last_peered":"2024-09-18T07:41:09.993502+0000","last_clean":"2024-09-18T07:41:09.993502+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:41:09.993502+0000","last_undegraded":"2024-09-18T07:41:09.993502+0000","last_fullsized":"2024-09-18T07:41:09.993502+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:47.286927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.286927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:02.409705+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":459,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.977483+0000","last_change":"2024-09-18T07:39:33.737527+0000","last_active":"2024-09-18T07:41:09.977483+0000","last_peered":"2024-09-18T07:41:09.977483+0000","last_clean":"2024-09-18T07:41:09.977483+0000","last_became_active":"2024-09-18T07:37:43.056380+0000","last_became_peered":"2024-09-18T07:37:43.056380+0000","last_unstale":"2024-09-18T07:41:09.977483+0000","last_undegraded":"2024-09-18T07:41:09.977483+0000","last_fullsized":"2024-09-18T07:41:09.977483+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:33.737478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:33.737478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":574,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.985214+0000","last_change":"2024-09-18T07:40:57.319270+0000","last_active":"2024-09-18T07:41:09.985214+0000","last_peered":"2024-09-18T07:41:09.985214+0000","last_clean":"2024-09-18T07:41:09.985214+0000","last_became_active":"2024-09-18T07:37:43.066508+0000","last_became_peered":"2024-09-18T07:37:43.066508+0000","last_unstale":"2024-09-18T07:41:09.985214+0000","last_undegraded":"2024-09-18T07:41:09.985214+0000","last_fullsized":"2024-09-18T07:41:09.985214+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:57.319204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:57.319204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:17.957297+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"323'27","reported_seq":118,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.977727+0000","last_change":"2024-09-18T07:40:20.436791+0000","last_active":"2024-09-18T07:41:09.977727+0000","last_peered":"2024-09-18T07:41:09.977727+0000","last_clean":"2024-09-18T07:41:09.977727+0000","last_became_active":"2024-09-18T07:40:19.064755+0000","last_became_peered":"2024-09-18T07:40:19.064755+0000","last_unstale":"2024-09-18T07:41:09.977727+0000","last_undegraded":"2024-09-18T07:41:09.977727+0000","last_fullsized":"2024-09-18T07:41:09.977727+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:40:20.436702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:20.436702+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:30.219060+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":574,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.993166+0000","last_change":"2024-09-18T07:40:11.212222+0000","last_active":"2024-09-18T07:41:09.993166+0000","last_peered":"2024-09-18T07:41:09.993166+0000","last_clean":"2024-09-18T07:41:09.993166+0000","last_became_active":"2024-09-18T07:37:43.052346+0000","last_became_peered":"2024-09-18T07:37:43.052346+0000","last_unstale":"2024-09-18T07:41:09.993166+0000","last_undegraded":"2024-09-18T07:41:09.993166+0000","last_fullsized":"2024-09-18T07:41:09.993166+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:11.212160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:11.212160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:40.950906+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":572,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.992351+0000","last_change":"2024-09-18T07:37:43.080879+0000","last_active":"2024-09-18T07:41:09.992351+0000","last_peered":"2024-09-18T07:41:09.992351+0000","last_clean":"2024-09-18T07:41:09.992351+0000","last_became_active":"2024-09-18T07:37:43.080314+0000","last_became_peered":"2024-09-18T07:37:43.080314+0000","last_unstale":"2024-09-18T07:41:09.992351+0000","last_undegraded":"2024-09-18T07:41:09.992351+0000","last_fullsized":"2024-09-18T07:41:09.992351+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:00.392466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.392466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:32.349615+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"323'15","reported_seq":581,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.993765+0000","last_change":"2024-09-18T07:40:46.250855+0000","last_active":"2024-09-18T07:41:09.993765+0000","last_peered":"2024-09-18T07:41:09.993765+0000","last_clean":"2024-09-18T07:41:09.993765+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:41:09.993765+0000","last_undegraded":"2024-09-18T07:41:09.993765+0000","last_fullsized":"2024-09-18T07:41:09.993765+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:46.250776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:46.250776+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:46.642247+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":564,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.992431+0000","last_change":"2024-09-18T07:40:38.360192+0000","last_active":"2024-09-18T07:41:09.992431+0000","last_peered":"2024-09-18T07:41:09.992431+0000","last_clean":"2024-09-18T07:41:09.992431+0000","last_became_active":"2024-09-18T07:35:01.413443+0000","last_became_peered":"2024-09-18T07:35:01.413443+0000","last_unstale":"2024-09-18T07:41:09.992431+0000","last_undegraded":"2024-09-18T07:41:09.992431+0000","last_fullsized":"2024-09-18T07:41:09.992431+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:38.360127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.360127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:52.488281+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":550,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.977550+0000","last_change":"2024-09-18T07:36:10.551960+0000","last_active":"2024-09-18T07:41:09.977550+0000","last_peered":"2024-09-18T07:41:09.977550+0000","last_clean":"2024-09-18T07:41:09.977550+0000","last_became_active":"2024-09-18T07:35:01.410284+0000","last_became_peered":"2024-09-18T07:35:01.410284+0000","last_unstale":"2024-09-18T07:41:09.977550+0000","last_undegraded":"2024-09-18T07:41:09.977550+0000","last_fullsized":"2024-09-18T07:41:09.977550+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:10.551889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:10.551889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:59.867668+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":550,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.988756+0000","last_change":"2024-09-18T07:40:11.300378+0000","last_active":"2024-09-18T07:41:09.988756+0000","last_peered":"2024-09-18T07:41:09.988756+0000","last_clean":"2024-09-18T07:41:09.988756+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:41:09.988756+0000","last_undegraded":"2024-09-18T07:41:09.988756+0000","last_fullsized":"2024-09-18T07:41:09.988756+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:11.300303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:11.300303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:23.791956+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"323'7","reported_seq":582,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.994097+0000","last_change":"2024-09-18T07:39:26.028935+0000","last_active":"2024-09-18T07:41:09.994097+0000","last_peered":"2024-09-18T07:41:09.994097+0000","last_clean":"2024-09-18T07:41:09.994097+0000","last_became_active":"2024-09-18T07:36:02.526212+0000","last_became_peered":"2024-09-18T07:36:02.526212+0000","last_unstale":"2024-09-18T07:41:09.994097+0000","last_undegraded":"2024-09-18T07:41:09.994097+0000","last_fullsized":"2024-09-18T07:41:09.994097+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-18T07:39:26.028884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:26.028884+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:34.147061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":332,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.987677+0000","last_change":"2024-09-18T07:40:41.457394+0000","last_active":"2024-09-18T07:41:09.987677+0000","last_peered":"2024-09-18T07:41:09.987677+0000","last_clean":"2024-09-18T07:41:09.987677+0000","last_became_active":"2024-09-18T07:35:01.398605+0000","last_became_peered":"2024-09-18T07:35:01.398605+0000","last_unstale":"2024-09-18T07:41:09.987677+0000","last_undegraded":"2024-09-18T07:41:09.987677+0000","last_fullsized":"2024-09-18T07:41:09.987677+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:41.457338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:41.457338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:03.232388+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":570,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.988585+0000","last_change":"2024-09-18T07:40:50.367758+0000","last_active":"2024-09-18T07:41:09.988585+0000","last_peered":"2024-09-18T07:41:09.988585+0000","last_clean":"2024-09-18T07:41:09.988585+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:41:09.988585+0000","last_undegraded":"2024-09-18T07:41:09.988585+0000","last_fullsized":"2024-09-18T07:41:09.988585+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:50.367694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.367694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:57.617198+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":571,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.995295+0000","last_change":"2024-09-18T07:38:56.096769+0000","last_active":"2024-09-18T07:41:09.995295+0000","last_peered":"2024-09-18T07:41:09.995295+0000","last_clean":"2024-09-18T07:41:09.995295+0000","last_became_active":"2024-09-18T07:37:43.067568+0000","last_became_peered":"2024-09-18T07:37:43.067568+0000","last_unstale":"2024-09-18T07:41:09.995295+0000","last_undegraded":"2024-09-18T07:41:09.995295+0000","last_fullsized":"2024-09-18T07:41:09.995295+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:56.096591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:56.096591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:29.294651+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":570,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.981384+0000","last_change":"2024-09-18T07:36:06.523357+0000","last_active":"2024-09-18T07:41:09.981384+0000","last_peered":"2024-09-18T07:41:09.981384+0000","last_clean":"2024-09-18T07:41:09.981384+0000","last_became_active":"2024-09-18T07:35:01.414088+0000","last_became_peered":"2024-09-18T07:35:01.414088+0000","last_unstale":"2024-09-18T07:41:09.981384+0000","last_undegraded":"2024-09-18T07:41:09.981384+0000","last_fullsized":"2024-09-18T07:41:09.981384+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:06.523292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:06.523292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:01.872546+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"323'27","reported_seq":653,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.981885+0000","last_change":"2024-09-18T07:40:14.323875+0000","last_active":"2024-09-18T07:41:09.981885+0000","last_peered":"2024-09-18T07:41:09.981885+0000","last_clean":"2024-09-18T07:41:09.981885+0000","last_became_active":"2024-09-18T07:37:28.002858+0000","last_became_peered":"2024-09-18T07:37:28.002858+0000","last_unstale":"2024-09-18T07:41:09.981885+0000","last_undegraded":"2024-09-18T07:41:09.981885+0000","last_fullsized":"2024-09-18T07:41:09.981885+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:40:14.323821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:14.323821+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:31.036792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":242,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.994644+0000","last_change":"2024-09-18T07:36:13.116882+0000","last_active":"2024-09-18T07:41:09.994644+0000","last_peered":"2024-09-18T07:41:09.994644+0000","last_clean":"2024-09-18T07:41:09.994644+0000","last_became_active":"2024-09-18T07:36:13.116416+0000","last_became_peered":"2024-09-18T07:36:13.116416+0000","last_unstale":"2024-09-18T07:41:09.994644+0000","last_undegraded":"2024-09-18T07:41:09.994644+0000","last_fullsized":"2024-09-18T07:41:09.994644+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:10.074367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:10.074367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:26.238982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":425,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.980072+0000","last_change":"2024-09-18T07:40:19.070769+0000","last_active":"2024-09-18T07:41:09.980072+0000","last_peered":"2024-09-18T07:41:09.980072+0000","last_clean":"2024-09-18T07:41:09.980072+0000","last_became_active":"2024-09-18T07:40:19.070463+0000","last_became_peered":"2024-09-18T07:40:19.070463+0000","last_unstale":"2024-09-18T07:41:09.980072+0000","last_undegraded":"2024-09-18T07:41:09.980072+0000","last_fullsized":"2024-09-18T07:41:09.980072+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:11.831891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:11.831891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:37.466083+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":568,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.993902+0000","last_change":"2024-09-18T07:39:00.130395+0000","last_active":"2024-09-18T07:41:09.993902+0000","last_peered":"2024-09-18T07:41:09.993902+0000","last_clean":"2024-09-18T07:41:09.993902+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:41:09.993902+0000","last_undegraded":"2024-09-18T07:41:09.993902+0000","last_fullsized":"2024-09-18T07:41:09.993902+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:00.130344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.130344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:28.269345+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"323'18","reported_seq":222,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.982000+0000","last_change":"2024-09-18T07:37:43.756858+0000","last_active":"2024-09-18T07:41:09.982000+0000","last_peered":"2024-09-18T07:41:09.982000+0000","last_clean":"2024-09-18T07:41:09.982000+0000","last_became_active":"2024-09-18T07:37:27.981705+0000","last_became_peered":"2024-09-18T07:37:27.981705+0000","last_unstale":"2024-09-18T07:41:09.982000+0000","last_undegraded":"2024-09-18T07:41:09.982000+0000","last_fullsized":"2024-09-18T07:41:09.982000+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:37:43.756403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:37:43.756403+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:58.748085+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":544,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.995331+0000","last_change":"2024-09-18T07:39:37.930497+0000","last_active":"2024-09-18T07:41:09.995331+0000","last_peered":"2024-09-18T07:41:09.995331+0000","last_clean":"2024-09-18T07:41:09.995331+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:41:09.995331+0000","last_undegraded":"2024-09-18T07:41:09.995331+0000","last_fullsized":"2024-09-18T07:41:09.995331+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.930435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.930435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.059930+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":344,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.988268+0000","last_change":"2024-09-18T07:40:10.644587+0000","last_active":"2024-09-18T07:41:09.988268+0000","last_peered":"2024-09-18T07:41:09.988268+0000","last_clean":"2024-09-18T07:41:09.988268+0000","last_became_active":"2024-09-18T07:35:01.413110+0000","last_became_peered":"2024-09-18T07:35:01.413110+0000","last_unstale":"2024-09-18T07:41:09.988268+0000","last_undegraded":"2024-09-18T07:41:09.988268+0000","last_fullsized":"2024-09-18T07:41:09.988268+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:40:10.644461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:10.644461+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:10.999910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":260,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.987931+0000","last_change":"2024-09-18T07:38:50.918758+0000","last_active":"2024-09-18T07:41:09.987931+0000","last_peered":"2024-09-18T07:41:09.987931+0000","last_clean":"2024-09-18T07:41:09.987931+0000","last_became_active":"2024-09-18T07:36:13.116295+0000","last_became_peered":"2024-09-18T07:36:13.116295+0000","last_unstale":"2024-09-18T07:41:09.987931+0000","last_undegraded":"2024-09-18T07:41:09.987931+0000","last_fullsized":"2024-09-18T07:41:09.987931+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:38:50.918686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:38:50.918686+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:33.015322+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"323'20","reported_seq":629,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.978355+0000","last_change":"2024-09-18T07:40:57.823251+0000","last_active":"2024-09-18T07:41:09.978355+0000","last_peered":"2024-09-18T07:41:09.978355+0000","last_clean":"2024-09-18T07:41:09.978355+0000","last_became_active":"2024-09-18T07:35:01.410926+0000","last_became_peered":"2024-09-18T07:35:01.410926+0000","last_unstale":"2024-09-18T07:41:09.978355+0000","last_undegraded":"2024-09-18T07:41:09.978355+0000","last_fullsized":"2024-09-18T07:41:09.978355+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:40:57.823151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:57.823151+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:19.183260+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"240'12","reported_seq":245,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.980182+0000","last_change":"2024-09-18T07:40:43.112526+0000","last_active":"2024-09-18T07:41:09.980182+0000","last_peered":"2024-09-18T07:41:09.980182+0000","last_clean":"2024-09-18T07:41:09.980182+0000","last_became_active":"2024-09-18T07:37:43.060715+0000","last_became_peered":"2024-09-18T07:37:43.060715+0000","last_unstale":"2024-09-18T07:41:09.980182+0000","last_undegraded":"2024-09-18T07:41:09.980182+0000","last_fullsized":"2024-09-18T07:41:09.980182+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:40:43.112459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:43.112459+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:10.247279+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"325'8835","reported_seq":8918,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:12.216898+0000","last_change":"2024-09-18T07:39:29.700748+0000","last_active":"2024-09-18T07:41:12.216898+0000","last_peered":"2024-09-18T07:41:12.216898+0000","last_clean":"2024-09-18T07:41:12.216898+0000","last_became_active":"2024-09-18T07:39:29.700477+0000","last_became_peered":"2024-09-18T07:39:29.700477+0000","last_unstale":"2024-09-18T07:41:12.216898+0000","last_undegraded":"2024-09-18T07:41:12.216898+0000","last_fullsized":"2024-09-18T07:41:12.216898+0000","mapping_epoch":289,"log_start":"303'4300","ondisk_log_start":"303'4300","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505479168,"num_objects":7713,"num_object_clones":0,"num_object_copies":23139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8835,"num_write_kb":529536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":128,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.989370+0000","last_change":"2024-09-18T07:40:41.373625+0000","last_active":"2024-09-18T07:41:09.989370+0000","last_peered":"2024-09-18T07:41:09.989370+0000","last_clean":"2024-09-18T07:41:09.989370+0000","last_became_active":"2024-09-18T07:39:04.962047+0000","last_became_peered":"2024-09-18T07:39:04.962047+0000","last_unstale":"2024-09-18T07:41:09.989370+0000","last_undegraded":"2024-09-18T07:41:09.989370+0000","last_fullsized":"2024-09-18T07:41:09.989370+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:41.373545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:41.373545+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:53.635822+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"323'27","reported_seq":314,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.987133+0000","last_change":"2024-09-18T07:38:54.967706+0000","last_active":"2024-09-18T07:41:09.987133+0000","last_peered":"2024-09-18T07:41:09.987133+0000","last_clean":"2024-09-18T07:41:09.987133+0000","last_became_active":"2024-09-18T07:36:13.119098+0000","last_became_peered":"2024-09-18T07:36:13.119098+0000","last_unstale":"2024-09-18T07:41:09.987133+0000","last_undegraded":"2024-09-18T07:41:09.987133+0000","last_fullsized":"2024-09-18T07:41:09.987133+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:38:54.967643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:38:54.967643+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:26.983363+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":589,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.984727+0000","last_change":"2024-09-18T07:39:29.049261+0000","last_active":"2024-09-18T07:41:09.984727+0000","last_peered":"2024-09-18T07:41:09.984727+0000","last_clean":"2024-09-18T07:41:09.984727+0000","last_became_active":"2024-09-18T07:35:01.407339+0000","last_became_peered":"2024-09-18T07:35:01.407339+0000","last_unstale":"2024-09-18T07:41:09.984727+0000","last_undegraded":"2024-09-18T07:41:09.984727+0000","last_fullsized":"2024-09-18T07:41:09.984727+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.049209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.049209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":700,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.995878+0000","last_change":"2024-09-18T07:40:03.237115+0000","last_active":"2024-09-18T07:41:09.995878+0000","last_peered":"2024-09-18T07:41:09.995878+0000","last_clean":"2024-09-18T07:41:09.995878+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:41:09.995878+0000","last_undegraded":"2024-09-18T07:41:09.995878+0000","last_fullsized":"2024-09-18T07:41:09.995878+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:40:03.237041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:03.237041+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:25.843038+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":546,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.990700+0000","last_change":"2024-09-18T07:36:07.588483+0000","last_active":"2024-09-18T07:41:09.990700+0000","last_peered":"2024-09-18T07:41:09.990700+0000","last_clean":"2024-09-18T07:41:09.990700+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:41:09.990700+0000","last_undegraded":"2024-09-18T07:41:09.990700+0000","last_fullsized":"2024-09-18T07:41:09.990700+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:07.588440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:07.588440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:06.305516+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"325'8886","reported_seq":9050,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:11.537208+0000","last_change":"2024-09-18T07:40:47.444123+0000","last_active":"2024-09-18T07:41:11.537208+0000","last_peered":"2024-09-18T07:41:11.537208+0000","last_clean":"2024-09-18T07:41:11.537208+0000","last_became_active":"2024-09-18T07:39:29.704590+0000","last_became_peered":"2024-09-18T07:39:29.704590+0000","last_unstale":"2024-09-18T07:41:11.537208+0000","last_undegraded":"2024-09-18T07:41:11.537208+0000","last_fullsized":"2024-09-18T07:41:11.537208+0000","mapping_epoch":289,"log_start":"304'4400","ondisk_log_start":"304'4400","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'7163","last_scrub_stamp":"2024-09-18T07:40:47.443974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.443974+0000","objects_scrubbed":7103,"log_size":4486,"log_dups_size":0,"ondisk_log_size":4486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:52.035188+0000","scrub_duration":1939,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511115288,"num_objects":7800,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7800,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8886,"num_write_kb":533889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":129,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.995165+0000","last_change":"2024-09-18T07:40:19.271780+0000","last_active":"2024-09-18T07:41:09.995165+0000","last_peered":"2024-09-18T07:41:09.995165+0000","last_clean":"2024-09-18T07:41:09.995165+0000","last_became_active":"2024-09-18T07:39:04.955645+0000","last_became_peered":"2024-09-18T07:39:04.955645+0000","last_unstale":"2024-09-18T07:41:09.995165+0000","last_undegraded":"2024-09-18T07:41:09.995165+0000","last_fullsized":"2024-09-18T07:41:09.995165+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:19.271704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:19.271704+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:43.872634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"323'30","reported_seq":664,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.984476+0000","last_change":"2024-09-18T07:39:00.919075+0000","last_active":"2024-09-18T07:41:09.984476+0000","last_peered":"2024-09-18T07:41:09.984476+0000","last_clean":"2024-09-18T07:41:09.984476+0000","last_became_active":"2024-09-18T07:35:01.418909+0000","last_became_peered":"2024-09-18T07:35:01.418909+0000","last_unstale":"2024-09-18T07:41:09.984476+0000","last_undegraded":"2024-09-18T07:41:09.984476+0000","last_fullsized":"2024-09-18T07:41:09.984476+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:39:00.918860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.918860+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:34.958741+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":295,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.990504+0000","last_change":"2024-09-18T07:40:19.069243+0000","last_active":"2024-09-18T07:41:09.990504+0000","last_peered":"2024-09-18T07:41:09.990504+0000","last_clean":"2024-09-18T07:41:09.990504+0000","last_became_active":"2024-09-18T07:40:19.068296+0000","last_became_peered":"2024-09-18T07:40:19.068296+0000","last_unstale":"2024-09-18T07:41:09.990504+0000","last_undegraded":"2024-09-18T07:41:09.990504+0000","last_fullsized":"2024-09-18T07:41:09.990504+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:06.987428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:06.987428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:30.575116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":566,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.993278+0000","last_change":"2024-09-18T07:36:30.242479+0000","last_active":"2024-09-18T07:41:09.993278+0000","last_peered":"2024-09-18T07:41:09.993278+0000","last_clean":"2024-09-18T07:41:09.993278+0000","last_became_active":"2024-09-18T07:35:01.401024+0000","last_became_peered":"2024-09-18T07:35:01.401024+0000","last_unstale":"2024-09-18T07:41:09.993278+0000","last_undegraded":"2024-09-18T07:41:09.993278+0000","last_fullsized":"2024-09-18T07:41:09.993278+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:30.242420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:30.242420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:04.089377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":540,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.993682+0000","last_change":"2024-09-18T07:40:44.219626+0000","last_active":"2024-09-18T07:41:09.993682+0000","last_peered":"2024-09-18T07:41:09.993682+0000","last_clean":"2024-09-18T07:41:09.993682+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:41:09.993682+0000","last_undegraded":"2024-09-18T07:41:09.993682+0000","last_fullsized":"2024-09-18T07:41:09.993682+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:44.219547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:44.219547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:05.414040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"325'9049","reported_seq":9132,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:12.181258+0000","last_change":"2024-09-18T07:39:29.703301+0000","last_active":"2024-09-18T07:41:12.181258+0000","last_peered":"2024-09-18T07:41:12.181258+0000","last_clean":"2024-09-18T07:41:12.181258+0000","last_became_active":"2024-09-18T07:39:29.702953+0000","last_became_peered":"2024-09-18T07:39:29.702953+0000","last_unstale":"2024-09-18T07:41:12.181258+0000","last_undegraded":"2024-09-18T07:41:12.181258+0000","last_fullsized":"2024-09-18T07:41:12.181258+0000","mapping_epoch":289,"log_start":"304'4500","ondisk_log_start":"304'4500","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":4549,"log_dups_size":0,"ondisk_log_size":4549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":519241728,"num_objects":7923,"num_object_clones":0,"num_object_copies":23769,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9049,"num_write_kb":543104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":126,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.985190+0000","last_change":"2024-09-18T07:39:05.959886+0000","last_active":"2024-09-18T07:41:09.985190+0000","last_peered":"2024-09-18T07:41:09.985190+0000","last_clean":"2024-09-18T07:41:09.985190+0000","last_became_active":"2024-09-18T07:39:04.945385+0000","last_became_peered":"2024-09-18T07:39:04.945385+0000","last_unstale":"2024-09-18T07:41:09.985190+0000","last_undegraded":"2024-09-18T07:41:09.985190+0000","last_fullsized":"2024-09-18T07:41:09.985190+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.959822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.959822+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"323'28","reported_seq":644,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.993370+0000","last_change":"2024-09-18T07:40:56.183888+0000","last_active":"2024-09-18T07:41:09.993370+0000","last_peered":"2024-09-18T07:41:09.993370+0000","last_clean":"2024-09-18T07:41:09.993370+0000","last_became_active":"2024-09-18T07:37:43.096236+0000","last_became_peered":"2024-09-18T07:37:43.096236+0000","last_unstale":"2024-09-18T07:41:09.993370+0000","last_undegraded":"2024-09-18T07:41:09.993370+0000","last_fullsized":"2024-09-18T07:41:09.993370+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-18T07:40:56.183802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.183802+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:23.795324+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":557,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.978633+0000","last_change":"2024-09-18T07:39:28.750779+0000","last_active":"2024-09-18T07:41:09.978633+0000","last_peered":"2024-09-18T07:41:09.978633+0000","last_clean":"2024-09-18T07:41:09.978633+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:41:09.978633+0000","last_undegraded":"2024-09-18T07:41:09.978633+0000","last_fullsized":"2024-09-18T07:41:09.978633+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:39:28.750720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.750720+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":684,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.984424+0000","last_change":"2024-09-18T07:36:05.016008+0000","last_active":"2024-09-18T07:41:09.984424+0000","last_peered":"2024-09-18T07:41:09.984424+0000","last_clean":"2024-09-18T07:41:09.984424+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:41:09.984424+0000","last_undegraded":"2024-09-18T07:41:09.984424+0000","last_fullsized":"2024-09-18T07:41:09.984424+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:36:05.015955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:36:05.015955+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:03.319473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":346,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.987728+0000","last_change":"2024-09-18T07:38:58.528134+0000","last_active":"2024-09-18T07:41:09.987728+0000","last_peered":"2024-09-18T07:41:09.987728+0000","last_clean":"2024-09-18T07:41:09.987728+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:41:09.987728+0000","last_undegraded":"2024-09-18T07:41:09.987728+0000","last_fullsized":"2024-09-18T07:41:09.987728+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:58.528059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:58.528059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:34.523987+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"325'8837","reported_seq":9004,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:11.953572+0000","last_change":"2024-09-18T07:40:52.501916+0000","last_active":"2024-09-18T07:41:11.953572+0000","last_peered":"2024-09-18T07:41:11.953572+0000","last_clean":"2024-09-18T07:41:11.953572+0000","last_became_active":"2024-09-18T07:39:29.702301+0000","last_became_peered":"2024-09-18T07:39:29.702301+0000","last_unstale":"2024-09-18T07:41:11.953572+0000","last_undegraded":"2024-09-18T07:41:11.953572+0000","last_fullsized":"2024-09-18T07:41:11.953572+0000","mapping_epoch":289,"log_start":"303'4100","ondisk_log_start":"303'4100","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7465","last_scrub_stamp":"2024-09-18T07:40:52.501716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:52.501716+0000","objects_scrubbed":7371,"log_size":4737,"log_dups_size":0,"ondisk_log_size":4737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:54.799219+0000","scrub_duration":3127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":510590976,"num_objects":7791,"num_object_clones":0,"num_object_copies":23373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8837,"num_write_kb":532096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":296,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.979951+0000","last_change":"2024-09-18T07:40:38.066508+0000","last_active":"2024-09-18T07:41:09.979951+0000","last_peered":"2024-09-18T07:41:09.979951+0000","last_clean":"2024-09-18T07:41:09.979951+0000","last_became_active":"2024-09-18T07:39:04.941752+0000","last_became_peered":"2024-09-18T07:39:04.941752+0000","last_unstale":"2024-09-18T07:41:09.979951+0000","last_undegraded":"2024-09-18T07:41:09.979951+0000","last_fullsized":"2024-09-18T07:41:09.979951+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:38.066443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.066443+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:00.580966+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"323'20","reported_seq":301,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.989720+0000","last_change":"2024-09-18T07:40:18.320050+0000","last_active":"2024-09-18T07:41:09.989720+0000","last_peered":"2024-09-18T07:41:09.989720+0000","last_clean":"2024-09-18T07:41:09.989720+0000","last_became_active":"2024-09-18T07:36:13.118876+0000","last_became_peered":"2024-09-18T07:36:13.118876+0000","last_unstale":"2024-09-18T07:41:09.989720+0000","last_undegraded":"2024-09-18T07:41:09.989720+0000","last_fullsized":"2024-09-18T07:41:09.989720+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:40:18.319988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:18.319988+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:38.668118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":574,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.992424+0000","last_change":"2024-09-18T07:40:53.350083+0000","last_active":"2024-09-18T07:41:09.992424+0000","last_peered":"2024-09-18T07:41:09.992424+0000","last_clean":"2024-09-18T07:41:09.992424+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:41:09.992424+0000","last_undegraded":"2024-09-18T07:41:09.992424+0000","last_fullsized":"2024-09-18T07:41:09.992424+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:53.350019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:53.350019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:05.453667+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"240'10","reported_seq":710,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.981567+0000","last_change":"2024-09-18T07:40:50.839710+0000","last_active":"2024-09-18T07:41:09.981567+0000","last_peered":"2024-09-18T07:41:09.981567+0000","last_clean":"2024-09-18T07:41:09.981567+0000","last_became_active":"2024-09-18T07:37:43.167850+0000","last_became_peered":"2024-09-18T07:37:43.167850+0000","last_unstale":"2024-09-18T07:41:09.981567+0000","last_undegraded":"2024-09-18T07:41:09.981567+0000","last_fullsized":"2024-09-18T07:41:09.981567+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'10","last_scrub_stamp":"2024-09-18T07:40:50.839504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.839504+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:03.826154+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":537,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.977734+0000","last_change":"2024-09-18T07:39:00.759483+0000","last_active":"2024-09-18T07:41:09.977734+0000","last_peered":"2024-09-18T07:41:09.977734+0000","last_clean":"2024-09-18T07:41:09.977734+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:41:09.977734+0000","last_undegraded":"2024-09-18T07:41:09.977734+0000","last_fullsized":"2024-09-18T07:41:09.977734+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:00.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:28.638359+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"325'8712","reported_seq":8795,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:11.545717+0000","last_change":"2024-09-18T07:39:29.703780+0000","last_active":"2024-09-18T07:41:11.545717+0000","last_peered":"2024-09-18T07:41:11.545717+0000","last_clean":"2024-09-18T07:41:11.545717+0000","last_became_active":"2024-09-18T07:39:29.703475+0000","last_became_peered":"2024-09-18T07:39:29.703475+0000","last_unstale":"2024-09-18T07:41:11.545717+0000","last_undegraded":"2024-09-18T07:41:11.545717+0000","last_fullsized":"2024-09-18T07:41:11.545717+0000","mapping_epoch":289,"log_start":"304'4300","ondisk_log_start":"304'4300","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:58.617483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506724352,"num_objects":7732,"num_object_clones":0,"num_object_copies":23196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8712,"num_write_kb":526208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":296,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.979698+0000","last_change":"2024-09-18T07:40:18.954153+0000","last_active":"2024-09-18T07:41:09.979698+0000","last_peered":"2024-09-18T07:41:09.979698+0000","last_clean":"2024-09-18T07:41:09.979698+0000","last_became_active":"2024-09-18T07:39:04.948571+0000","last_became_peered":"2024-09-18T07:39:04.948571+0000","last_unstale":"2024-09-18T07:41:09.979698+0000","last_undegraded":"2024-09-18T07:41:09.979698+0000","last_fullsized":"2024-09-18T07:41:09.979698+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:18.954089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:18.954089+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:36.574798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"323'29","reported_seq":661,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.981232+0000","last_change":"2024-09-18T07:40:02.764120+0000","last_active":"2024-09-18T07:41:09.981232+0000","last_peered":"2024-09-18T07:41:09.981232+0000","last_clean":"2024-09-18T07:41:09.981232+0000","last_became_active":"2024-09-18T07:35:01.415275+0000","last_became_peered":"2024-09-18T07:35:01.415275+0000","last_unstale":"2024-09-18T07:41:09.981232+0000","last_undegraded":"2024-09-18T07:41:09.981232+0000","last_fullsized":"2024-09-18T07:41:09.981232+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:40:02.764017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:02.764017+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":251,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.982012+0000","last_change":"2024-09-18T07:37:46.831436+0000","last_active":"2024-09-18T07:41:09.982012+0000","last_peered":"2024-09-18T07:41:09.982012+0000","last_clean":"2024-09-18T07:41:09.982012+0000","last_became_active":"2024-09-18T07:36:13.118553+0000","last_became_peered":"2024-09-18T07:36:13.118553+0000","last_unstale":"2024-09-18T07:41:09.982012+0000","last_undegraded":"2024-09-18T07:41:09.982012+0000","last_fullsized":"2024-09-18T07:41:09.982012+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:46.831373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:37:46.831373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:28.464967+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":191,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.980338+0000","last_change":"2024-09-18T07:40:14.044975+0000","last_active":"2024-09-18T07:41:09.980338+0000","last_peered":"2024-09-18T07:41:09.980338+0000","last_clean":"2024-09-18T07:41:09.980338+0000","last_became_active":"2024-09-18T07:37:43.070009+0000","last_became_peered":"2024-09-18T07:37:43.070009+0000","last_unstale":"2024-09-18T07:41:09.980338+0000","last_undegraded":"2024-09-18T07:41:09.980338+0000","last_fullsized":"2024-09-18T07:41:09.980338+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:14.044907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:14.044907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:22.276650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":191,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.983495+0000","last_change":"2024-09-18T07:40:10.073596+0000","last_active":"2024-09-18T07:41:09.983495+0000","last_peered":"2024-09-18T07:41:09.983495+0000","last_clean":"2024-09-18T07:41:09.983495+0000","last_became_active":"2024-09-18T07:37:43.051740+0000","last_became_peered":"2024-09-18T07:37:43.051740+0000","last_unstale":"2024-09-18T07:41:09.983495+0000","last_undegraded":"2024-09-18T07:41:09.983495+0000","last_fullsized":"2024-09-18T07:41:09.983495+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:10.073499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:10.073499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:32.816184+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"325'8828","reported_seq":8911,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:12.528235+0000","last_change":"2024-09-18T07:39:29.702264+0000","last_active":"2024-09-18T07:41:12.528235+0000","last_peered":"2024-09-18T07:41:12.528235+0000","last_clean":"2024-09-18T07:41:12.528235+0000","last_became_active":"2024-09-18T07:39:29.701239+0000","last_became_peered":"2024-09-18T07:39:29.701239+0000","last_unstale":"2024-09-18T07:41:12.528235+0000","last_undegraded":"2024-09-18T07:41:12.528235+0000","last_fullsized":"2024-09-18T07:41:12.528235+0000","mapping_epoch":289,"log_start":"301'3400","ondisk_log_start":"301'3400","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":5428,"log_dups_size":0,"ondisk_log_size":5428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:05.902015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500957184,"num_objects":7644,"num_object_clones":0,"num_object_copies":22932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8828,"num_write_kb":527104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":129,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.988411+0000","last_change":"2024-09-18T07:40:43.346516+0000","last_active":"2024-09-18T07:41:09.988411+0000","last_peered":"2024-09-18T07:41:09.988411+0000","last_clean":"2024-09-18T07:41:09.988411+0000","last_became_active":"2024-09-18T07:39:04.950564+0000","last_became_peered":"2024-09-18T07:39:04.950564+0000","last_unstale":"2024-09-18T07:41:09.988411+0000","last_undegraded":"2024-09-18T07:41:09.988411+0000","last_fullsized":"2024-09-18T07:41:09.988411+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:43.346447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:43.346447+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:49.159746+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"323'41","reported_seq":407,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.981831+0000","last_change":"2024-09-18T07:40:12.315524+0000","last_active":"2024-09-18T07:41:09.981831+0000","last_peered":"2024-09-18T07:41:09.981831+0000","last_clean":"2024-09-18T07:41:09.981831+0000","last_became_active":"2024-09-18T07:36:13.115278+0000","last_became_peered":"2024-09-18T07:36:13.115278+0000","last_unstale":"2024-09-18T07:41:09.981831+0000","last_undegraded":"2024-09-18T07:41:09.981831+0000","last_fullsized":"2024-09-18T07:41:09.981831+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'29","last_scrub_stamp":"2024-09-18T07:40:12.315424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:12.315424+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:31.219772+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":574,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.994379+0000","last_change":"2024-09-18T07:40:57.156765+0000","last_active":"2024-09-18T07:41:09.994379+0000","last_peered":"2024-09-18T07:41:09.994379+0000","last_clean":"2024-09-18T07:41:09.994379+0000","last_became_active":"2024-09-18T07:37:43.060930+0000","last_became_peered":"2024-09-18T07:37:43.060930+0000","last_unstale":"2024-09-18T07:41:09.994379+0000","last_undegraded":"2024-09-18T07:41:09.994379+0000","last_fullsized":"2024-09-18T07:41:09.994379+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:57.156706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:57.156706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:58.567971+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":590,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.989885+0000","last_change":"2024-09-18T07:37:43.155837+0000","last_active":"2024-09-18T07:41:09.989885+0000","last_peered":"2024-09-18T07:41:09.989885+0000","last_clean":"2024-09-18T07:41:09.989885+0000","last_became_active":"2024-09-18T07:37:43.076595+0000","last_became_peered":"2024-09-18T07:37:43.076595+0000","last_unstale":"2024-09-18T07:41:09.989885+0000","last_undegraded":"2024-09-18T07:41:09.989885+0000","last_fullsized":"2024-09-18T07:41:09.989885+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:36:01.529677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:36:01.529677+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:59.294998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":540,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.978353+0000","last_change":"2024-09-18T07:39:25.741714+0000","last_active":"2024-09-18T07:41:09.978353+0000","last_peered":"2024-09-18T07:41:09.978353+0000","last_clean":"2024-09-18T07:41:09.978353+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:41:09.978353+0000","last_undegraded":"2024-09-18T07:41:09.978353+0000","last_fullsized":"2024-09-18T07:41:09.978353+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:25.741643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.741643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:37.686029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":569,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.985016+0000","last_change":"2024-09-18T07:40:05.858885+0000","last_active":"2024-09-18T07:41:09.985016+0000","last_peered":"2024-09-18T07:41:09.985016+0000","last_clean":"2024-09-18T07:41:09.985016+0000","last_became_active":"2024-09-18T07:35:01.418809+0000","last_became_peered":"2024-09-18T07:35:01.418809+0000","last_unstale":"2024-09-18T07:41:09.985016+0000","last_undegraded":"2024-09-18T07:41:09.985016+0000","last_fullsized":"2024-09-18T07:41:09.985016+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:05.858804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:05.858804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:32.928019+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"325'8846","reported_seq":9010,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:11.953579+0000","last_change":"2024-09-18T07:40:50.786329+0000","last_active":"2024-09-18T07:41:11.953579+0000","last_peered":"2024-09-18T07:41:11.953579+0000","last_clean":"2024-09-18T07:41:11.953579+0000","last_became_active":"2024-09-18T07:39:29.702461+0000","last_became_peered":"2024-09-18T07:39:29.702461+0000","last_unstale":"2024-09-18T07:41:11.953579+0000","last_undegraded":"2024-09-18T07:41:11.953579+0000","last_fullsized":"2024-09-18T07:41:11.953579+0000","mapping_epoch":289,"log_start":"303'4100","ondisk_log_start":"303'4100","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7357","last_scrub_stamp":"2024-09-18T07:40:50.786248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.786248+0000","objects_scrubbed":7280,"log_size":4746,"log_dups_size":0,"ondisk_log_size":4746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:01.633118+0000","scrub_duration":2406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506331136,"num_objects":7726,"num_object_clones":0,"num_object_copies":23178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8846,"num_write_kb":530304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":397,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.987534+0000","last_change":"2024-09-18T07:40:44.476464+0000","last_active":"2024-09-18T07:41:09.987534+0000","last_peered":"2024-09-18T07:41:09.987534+0000","last_clean":"2024-09-18T07:41:09.987534+0000","last_became_active":"2024-09-18T07:39:04.944921+0000","last_became_peered":"2024-09-18T07:39:04.944921+0000","last_unstale":"2024-09-18T07:41:09.987534+0000","last_undegraded":"2024-09-18T07:41:09.987534+0000","last_fullsized":"2024-09-18T07:41:09.987534+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:44.476278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:44.476278+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:00.492060+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"323'17","reported_seq":304,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.989747+0000","last_change":"2024-09-18T07:40:04.227278+0000","last_active":"2024-09-18T07:41:09.989747+0000","last_peered":"2024-09-18T07:41:09.989747+0000","last_clean":"2024-09-18T07:41:09.989747+0000","last_became_active":"2024-09-18T07:36:13.117924+0000","last_became_peered":"2024-09-18T07:36:13.117924+0000","last_unstale":"2024-09-18T07:41:09.989747+0000","last_undegraded":"2024-09-18T07:41:09.989747+0000","last_fullsized":"2024-09-18T07:41:09.989747+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-18T07:40:04.226789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:04.226789+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:15.873556+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":273,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.977956+0000","last_change":"2024-09-18T07:40:38.553788+0000","last_active":"2024-09-18T07:41:09.977956+0000","last_peered":"2024-09-18T07:41:09.977956+0000","last_clean":"2024-09-18T07:41:09.977956+0000","last_became_active":"2024-09-18T07:37:43.050971+0000","last_became_peered":"2024-09-18T07:37:43.050971+0000","last_unstale":"2024-09-18T07:41:09.977956+0000","last_undegraded":"2024-09-18T07:41:09.977956+0000","last_fullsized":"2024-09-18T07:41:09.977956+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:38.553480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.553480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:40.213737+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":569,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.977827+0000","last_change":"2024-09-18T07:37:49.895264+0000","last_active":"2024-09-18T07:41:09.977827+0000","last_peered":"2024-09-18T07:41:09.977827+0000","last_clean":"2024-09-18T07:41:09.977827+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:41:09.977827+0000","last_undegraded":"2024-09-18T07:41:09.977827+0000","last_fullsized":"2024-09-18T07:41:09.977827+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:37:49.894707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:37:49.894707+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:25.419604+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":566,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.982237+0000","last_change":"2024-09-18T07:39:35.457812+0000","last_active":"2024-09-18T07:41:09.982237+0000","last_peered":"2024-09-18T07:41:09.982237+0000","last_clean":"2024-09-18T07:41:09.982237+0000","last_became_active":"2024-09-18T07:35:01.403680+0000","last_became_peered":"2024-09-18T07:35:01.403680+0000","last_unstale":"2024-09-18T07:41:09.982237+0000","last_undegraded":"2024-09-18T07:41:09.982237+0000","last_fullsized":"2024-09-18T07:41:09.982237+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.457734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.457734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:03.777297+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":350,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.991077+0000","last_change":"2024-09-18T07:40:54.574254+0000","last_active":"2024-09-18T07:41:09.991077+0000","last_peered":"2024-09-18T07:41:09.991077+0000","last_clean":"2024-09-18T07:41:09.991077+0000","last_became_active":"2024-09-18T07:37:43.060019+0000","last_became_peered":"2024-09-18T07:37:43.060019+0000","last_unstale":"2024-09-18T07:41:09.991077+0000","last_undegraded":"2024-09-18T07:41:09.991077+0000","last_fullsized":"2024-09-18T07:41:09.991077+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:40:54.574077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:54.574077+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:12.005144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"325'8903","reported_seq":9065,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:12.538941+0000","last_change":"2024-09-18T07:40:46.264109+0000","last_active":"2024-09-18T07:41:12.538941+0000","last_peered":"2024-09-18T07:41:12.538941+0000","last_clean":"2024-09-18T07:41:12.538941+0000","last_became_active":"2024-09-18T07:39:29.701554+0000","last_became_peered":"2024-09-18T07:39:29.701554+0000","last_unstale":"2024-09-18T07:41:12.538941+0000","last_undegraded":"2024-09-18T07:41:12.538941+0000","last_fullsized":"2024-09-18T07:41:12.538941+0000","mapping_epoch":289,"log_start":"301'3500","ondisk_log_start":"301'3500","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'7054","last_scrub_stamp":"2024-09-18T07:40:46.263923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:46.263923+0000","objects_scrubbed":6963,"log_size":5403,"log_dups_size":0,"ondisk_log_size":5403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:59.824520+0000","scrub_duration":2499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505872384,"num_objects":7719,"num_object_clones":0,"num_object_copies":23157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8903,"num_write_kb":531904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":289,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.979240+0000","last_change":"2024-09-18T07:39:05.333676+0000","last_active":"2024-09-18T07:41:09.979240+0000","last_peered":"2024-09-18T07:41:09.979240+0000","last_clean":"2024-09-18T07:41:09.979240+0000","last_became_active":"2024-09-18T07:39:04.938080+0000","last_became_peered":"2024-09-18T07:39:04.938080+0000","last_unstale":"2024-09-18T07:41:09.979240+0000","last_undegraded":"2024-09-18T07:41:09.979240+0000","last_fullsized":"2024-09-18T07:41:09.979240+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.333608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.333608+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:29.016443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"323'24","reported_seq":593,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.978444+0000","last_change":"2024-09-18T07:33:27.040227+0000","last_active":"2024-09-18T07:41:09.978444+0000","last_peered":"2024-09-18T07:41:09.978444+0000","last_clean":"2024-09-18T07:41:09.978444+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:41:09.978444+0000","last_undegraded":"2024-09-18T07:41:09.978444+0000","last_fullsized":"2024-09-18T07:41:09.978444+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:24.682676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":554,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.978094+0000","last_change":"2024-09-18T07:40:19.694501+0000","last_active":"2024-09-18T07:41:09.978094+0000","last_peered":"2024-09-18T07:41:09.978094+0000","last_clean":"2024-09-18T07:41:09.978094+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:41:09.978094+0000","last_undegraded":"2024-09-18T07:41:09.978094+0000","last_fullsized":"2024-09-18T07:41:09.978094+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:40:19.694421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:19.694421+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:21.914705+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":699,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:10.042301+0000","last_change":"2024-09-18T07:40:47.554230+0000","last_active":"2024-09-18T07:41:10.042301+0000","last_peered":"2024-09-18T07:41:10.042301+0000","last_clean":"2024-09-18T07:41:10.042301+0000","last_became_active":"2024-09-18T07:35:01.400524+0000","last_became_peered":"2024-09-18T07:35:01.400524+0000","last_unstale":"2024-09-18T07:41:10.042301+0000","last_undegraded":"2024-09-18T07:41:10.042301+0000","last_fullsized":"2024-09-18T07:41:10.042301+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T07:40:47.554161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.554161+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.477483+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":376,"num_read_kb":371,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":558,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.989829+0000","last_change":"2024-09-18T07:40:05.263108+0000","last_active":"2024-09-18T07:41:09.989829+0000","last_peered":"2024-09-18T07:41:09.989829+0000","last_clean":"2024-09-18T07:41:09.989829+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:41:09.989829+0000","last_undegraded":"2024-09-18T07:41:09.989829+0000","last_fullsized":"2024-09-18T07:41:09.989829+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:05.263040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:05.263040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:33.762504+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":259,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.988831+0000","last_change":"2024-09-18T07:40:12.269264+0000","last_active":"2024-09-18T07:41:09.988831+0000","last_peered":"2024-09-18T07:41:09.988831+0000","last_clean":"2024-09-18T07:41:09.988831+0000","last_became_active":"2024-09-18T07:36:13.117463+0000","last_became_peered":"2024-09-18T07:36:13.117463+0000","last_unstale":"2024-09-18T07:41:09.988831+0000","last_undegraded":"2024-09-18T07:41:09.988831+0000","last_fullsized":"2024-09-18T07:41:09.988831+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:12.269201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:12.269201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:41.445307+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"325'8661","reported_seq":8744,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:12.208851+0000","last_change":"2024-09-18T07:39:29.700116+0000","last_active":"2024-09-18T07:41:12.208851+0000","last_peered":"2024-09-18T07:41:12.208851+0000","last_clean":"2024-09-18T07:41:12.208851+0000","last_became_active":"2024-09-18T07:39:29.699733+0000","last_became_peered":"2024-09-18T07:39:29.699733+0000","last_unstale":"2024-09-18T07:41:12.208851+0000","last_undegraded":"2024-09-18T07:41:12.208851+0000","last_fullsized":"2024-09-18T07:41:12.208851+0000","mapping_epoch":289,"log_start":"303'4100","ondisk_log_start":"303'4100","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495910912,"num_objects":7567,"num_object_clones":0,"num_object_copies":22701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8661,"num_write_kb":519296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"323'32","reported_seq":269,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.980823+0000","last_change":"2024-09-18T07:40:17.950392+0000","last_active":"2024-09-18T07:41:09.980823+0000","last_peered":"2024-09-18T07:41:09.980823+0000","last_clean":"2024-09-18T07:41:09.980823+0000","last_became_active":"2024-09-18T07:37:43.087496+0000","last_became_peered":"2024-09-18T07:37:43.087496+0000","last_unstale":"2024-09-18T07:41:09.980823+0000","last_undegraded":"2024-09-18T07:41:09.980823+0000","last_fullsized":"2024-09-18T07:41:09.980823+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-18T07:40:17.950298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:17.950298+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:36.252264+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":429,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.989804+0000","last_change":"2024-09-18T07:40:19.067973+0000","last_active":"2024-09-18T07:41:09.989804+0000","last_peered":"2024-09-18T07:41:09.989804+0000","last_clean":"2024-09-18T07:41:09.989804+0000","last_became_active":"2024-09-18T07:40:19.067624+0000","last_became_peered":"2024-09-18T07:40:19.067624+0000","last_unstale":"2024-09-18T07:41:09.989804+0000","last_undegraded":"2024-09-18T07:41:09.989804+0000","last_fullsized":"2024-09-18T07:41:09.989804+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:59.616988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:59.616988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:26.342415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":391,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.987024+0000","last_change":"2024-09-18T07:40:42.452815+0000","last_active":"2024-09-18T07:41:09.987024+0000","last_peered":"2024-09-18T07:41:09.987024+0000","last_clean":"2024-09-18T07:41:09.987024+0000","last_became_active":"2024-09-18T07:39:03.350419+0000","last_became_peered":"2024-09-18T07:39:03.350419+0000","last_unstale":"2024-09-18T07:41:09.987024+0000","last_undegraded":"2024-09-18T07:41:09.987024+0000","last_fullsized":"2024-09-18T07:41:09.987024+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:42.452744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:42.452744+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:56.076651+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":332,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.988355+0000","last_change":"2024-09-18T07:40:39.483334+0000","last_active":"2024-09-18T07:41:09.988355+0000","last_peered":"2024-09-18T07:41:09.988355+0000","last_clean":"2024-09-18T07:41:09.988355+0000","last_became_active":"2024-09-18T07:35:01.412217+0000","last_became_peered":"2024-09-18T07:35:01.412217+0000","last_unstale":"2024-09-18T07:41:09.988355+0000","last_undegraded":"2024-09-18T07:41:09.988355+0000","last_fullsized":"2024-09-18T07:41:09.988355+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:39.483269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:39.483269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:43.918520+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":538,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.989616+0000","last_change":"2024-09-18T07:33:22.358792+0000","last_active":"2024-09-18T07:41:09.989616+0000","last_peered":"2024-09-18T07:41:09.989616+0000","last_clean":"2024-09-18T07:41:09.989616+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:41:09.989616+0000","last_undegraded":"2024-09-18T07:41:09.989616+0000","last_fullsized":"2024-09-18T07:41:09.989616+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.358746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.358746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:28.287255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"325'8771","reported_seq":8941,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:11.565353+0000","last_change":"2024-09-18T07:40:58.098898+0000","last_active":"2024-09-18T07:41:11.565353+0000","last_peered":"2024-09-18T07:41:11.565353+0000","last_clean":"2024-09-18T07:41:11.565353+0000","last_became_active":"2024-09-18T07:39:29.703451+0000","last_became_peered":"2024-09-18T07:39:29.703451+0000","last_unstale":"2024-09-18T07:41:11.565353+0000","last_undegraded":"2024-09-18T07:41:11.565353+0000","last_fullsized":"2024-09-18T07:41:11.565353+0000","mapping_epoch":289,"log_start":"303'4300","ondisk_log_start":"303'4300","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7779","last_scrub_stamp":"2024-09-18T07:40:58.098697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:58.098697+0000","objects_scrubbed":7710,"log_size":4471,"log_dups_size":0,"ondisk_log_size":4471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:21.707853+0000","scrub_duration":2544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506658816,"num_objects":7731,"num_object_clones":0,"num_object_copies":23193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8771,"num_write_kb":528064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"323'19","reported_seq":671,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.995405+0000","last_change":"2024-09-18T07:39:07.038527+0000","last_active":"2024-09-18T07:41:09.995405+0000","last_peered":"2024-09-18T07:41:09.995405+0000","last_clean":"2024-09-18T07:41:09.995405+0000","last_became_active":"2024-09-18T07:37:43.063255+0000","last_became_peered":"2024-09-18T07:37:43.063255+0000","last_unstale":"2024-09-18T07:41:09.995405+0000","last_undegraded":"2024-09-18T07:41:09.995405+0000","last_fullsized":"2024-09-18T07:41:09.995405+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:39:07.038450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:07.038450+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:31.263147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":582,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.993715+0000","last_change":"2024-09-18T07:39:29.078656+0000","last_active":"2024-09-18T07:41:09.993715+0000","last_peered":"2024-09-18T07:41:09.993715+0000","last_clean":"2024-09-18T07:41:09.993715+0000","last_became_active":"2024-09-18T07:35:01.401063+0000","last_became_peered":"2024-09-18T07:35:01.401063+0000","last_unstale":"2024-09-18T07:41:09.993715+0000","last_undegraded":"2024-09-18T07:41:09.993715+0000","last_fullsized":"2024-09-18T07:41:09.993715+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.078600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.078600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:35.101441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":572,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.989357+0000","last_change":"2024-09-18T07:39:35.406107+0000","last_active":"2024-09-18T07:41:09.989357+0000","last_peered":"2024-09-18T07:41:09.989357+0000","last_clean":"2024-09-18T07:41:09.989357+0000","last_became_active":"2024-09-18T07:37:43.075535+0000","last_became_peered":"2024-09-18T07:37:43.075535+0000","last_unstale":"2024-09-18T07:41:09.989357+0000","last_undegraded":"2024-09-18T07:41:09.989357+0000","last_fullsized":"2024-09-18T07:41:09.989357+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.406041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.406041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.343859+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":424,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.979335+0000","last_change":"2024-09-18T07:40:19.069328+0000","last_active":"2024-09-18T07:41:09.979335+0000","last_peered":"2024-09-18T07:41:09.979335+0000","last_clean":"2024-09-18T07:41:09.979335+0000","last_became_active":"2024-09-18T07:40:19.068998+0000","last_became_peered":"2024-09-18T07:40:19.068998+0000","last_unstale":"2024-09-18T07:41:09.979335+0000","last_undegraded":"2024-09-18T07:41:09.979335+0000","last_fullsized":"2024-09-18T07:41:09.979335+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:09.801523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:09.801523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:18.838558+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"325'8823","reported_seq":8986,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:11.687537+0000","last_change":"2024-09-18T07:40:48.247625+0000","last_active":"2024-09-18T07:41:11.687537+0000","last_peered":"2024-09-18T07:41:11.687537+0000","last_clean":"2024-09-18T07:41:11.687537+0000","last_became_active":"2024-09-18T07:39:29.699800+0000","last_became_peered":"2024-09-18T07:39:29.699800+0000","last_unstale":"2024-09-18T07:41:11.687537+0000","last_undegraded":"2024-09-18T07:41:11.687537+0000","last_fullsized":"2024-09-18T07:41:11.687537+0000","mapping_epoch":289,"log_start":"298'2800","ondisk_log_start":"298'2800","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'7166","last_scrub_stamp":"2024-09-18T07:40:48.247448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:48.247448+0000","objects_scrubbed":7089,"log_size":6023,"log_dups_size":0,"ondisk_log_size":6023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:59.164081+0000","scrub_duration":2089,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508755968,"num_objects":7763,"num_object_clones":0,"num_object_copies":23289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8823,"num_write_kb":530752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":130,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.989103+0000","last_change":"2024-09-18T07:39:38.447787+0000","last_active":"2024-09-18T07:41:09.989103+0000","last_peered":"2024-09-18T07:41:09.989103+0000","last_clean":"2024-09-18T07:41:09.989103+0000","last_became_active":"2024-09-18T07:39:04.950292+0000","last_became_peered":"2024-09-18T07:39:04.950292+0000","last_unstale":"2024-09-18T07:41:09.989103+0000","last_undegraded":"2024-09-18T07:41:09.989103+0000","last_fullsized":"2024-09-18T07:41:09.989103+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:38.447735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:38.447735+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"323'31","reported_seq":621,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.995006+0000","last_change":"2024-09-18T07:40:04.281816+0000","last_active":"2024-09-18T07:41:09.995006+0000","last_peered":"2024-09-18T07:41:09.995006+0000","last_clean":"2024-09-18T07:41:09.995006+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:41:09.995006+0000","last_undegraded":"2024-09-18T07:41:09.995006+0000","last_fullsized":"2024-09-18T07:41:09.995006+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:40:04.281758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:04.281758+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:29.914573+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":199,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.981010+0000","last_change":"2024-09-18T07:40:47.156023+0000","last_active":"2024-09-18T07:41:09.981010+0000","last_peered":"2024-09-18T07:41:09.981010+0000","last_clean":"2024-09-18T07:41:09.981010+0000","last_became_active":"2024-09-18T07:37:43.061808+0000","last_became_peered":"2024-09-18T07:37:43.061808+0000","last_unstale":"2024-09-18T07:41:09.981010+0000","last_undegraded":"2024-09-18T07:41:09.981010+0000","last_fullsized":"2024-09-18T07:41:09.981010+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:47.155953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.155953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:52.522517+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":251,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.993364+0000","last_change":"2024-09-18T07:40:52.167232+0000","last_active":"2024-09-18T07:41:09.993364+0000","last_peered":"2024-09-18T07:41:09.993364+0000","last_clean":"2024-09-18T07:41:09.993364+0000","last_became_active":"2024-09-18T07:36:13.116527+0000","last_became_peered":"2024-09-18T07:36:13.116527+0000","last_unstale":"2024-09-18T07:41:09.993364+0000","last_undegraded":"2024-09-18T07:41:09.993364+0000","last_fullsized":"2024-09-18T07:41:09.993364+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:52.167048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:52.167048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:59.961543+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":564,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.982362+0000","last_change":"2024-09-18T07:40:10.351667+0000","last_active":"2024-09-18T07:41:09.982362+0000","last_peered":"2024-09-18T07:41:09.982362+0000","last_clean":"2024-09-18T07:41:09.982362+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:41:09.982362+0000","last_undegraded":"2024-09-18T07:41:09.982362+0000","last_fullsized":"2024-09-18T07:41:09.982362+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:10.351594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:10.351594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:33.113808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"325'8777","reported_seq":8940,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:11.662352+0000","last_change":"2024-09-18T07:40:41.001448+0000","last_active":"2024-09-18T07:41:11.662352+0000","last_peered":"2024-09-18T07:41:11.662352+0000","last_clean":"2024-09-18T07:41:11.662352+0000","last_became_active":"2024-09-18T07:40:19.066715+0000","last_became_peered":"2024-09-18T07:40:19.066715+0000","last_unstale":"2024-09-18T07:41:11.662352+0000","last_undegraded":"2024-09-18T07:41:11.662352+0000","last_fullsized":"2024-09-18T07:41:11.662352+0000","mapping_epoch":307,"log_start":"298'2700","ondisk_log_start":"298'2700","created":289,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'6605","last_scrub_stamp":"2024-09-18T07:40:41.001367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:41.001367+0000","objects_scrubbed":6534,"log_size":6077,"log_dups_size":0,"ondisk_log_size":6077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:43.490590+0000","scrub_duration":1931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":507969536,"num_objects":7751,"num_object_clones":0,"num_object_copies":23253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":528896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":388,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.989215+0000","last_change":"2024-09-18T07:39:05.498235+0000","last_active":"2024-09-18T07:41:09.989215+0000","last_peered":"2024-09-18T07:41:09.989215+0000","last_clean":"2024-09-18T07:41:09.989215+0000","last_became_active":"2024-09-18T07:39:04.944753+0000","last_became_peered":"2024-09-18T07:39:04.944753+0000","last_unstale":"2024-09-18T07:41:09.989215+0000","last_undegraded":"2024-09-18T07:41:09.989215+0000","last_fullsized":"2024-09-18T07:41:09.989215+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.498159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.498159+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:26.532863+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"323'20","reported_seq":624,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.995427+0000","last_change":"2024-09-18T07:40:50.210059+0000","last_active":"2024-09-18T07:41:09.995427+0000","last_peered":"2024-09-18T07:41:09.995427+0000","last_clean":"2024-09-18T07:41:09.995427+0000","last_became_active":"2024-09-18T07:37:43.062724+0000","last_became_peered":"2024-09-18T07:37:43.062724+0000","last_unstale":"2024-09-18T07:41:09.995427+0000","last_undegraded":"2024-09-18T07:41:09.995427+0000","last_fullsized":"2024-09-18T07:41:09.995427+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:40:50.209824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.209824+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:03.858021+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":56,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.981314+0000","last_change":"2024-09-18T07:40:47.799512+0000","last_active":"2024-09-18T07:41:09.981314+0000","last_peered":"2024-09-18T07:41:09.981314+0000","last_clean":"2024-09-18T07:41:09.981314+0000","last_became_active":"2024-09-18T07:40:19.065652+0000","last_became_peered":"2024-09-18T07:40:19.065652+0000","last_unstale":"2024-09-18T07:41:09.981314+0000","last_undegraded":"2024-09-18T07:41:09.981314+0000","last_fullsized":"2024-09-18T07:41:09.981314+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:47.799342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.799342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.410981+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"173'11","reported_seq":717,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.982023+0000","last_change":"2024-09-18T07:38:55.227421+0000","last_active":"2024-09-18T07:41:09.982023+0000","last_peered":"2024-09-18T07:41:09.982023+0000","last_clean":"2024-09-18T07:41:09.982023+0000","last_became_active":"2024-09-18T07:35:01.403765+0000","last_became_peered":"2024-09-18T07:35:01.403765+0000","last_unstale":"2024-09-18T07:41:09.982023+0000","last_undegraded":"2024-09-18T07:41:09.982023+0000","last_fullsized":"2024-09-18T07:41:09.982023+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'11","last_scrub_stamp":"2024-09-18T07:38:55.227366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:55.227366+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:31.467614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":548,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.994878+0000","last_change":"2024-09-18T07:36:33.251164+0000","last_active":"2024-09-18T07:41:09.994878+0000","last_peered":"2024-09-18T07:41:09.994878+0000","last_clean":"2024-09-18T07:41:09.994878+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:41:09.994878+0000","last_undegraded":"2024-09-18T07:41:09.994878+0000","last_fullsized":"2024-09-18T07:41:09.994878+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:33.251093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:33.251093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:05.051072+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"325'8807","reported_seq":8977,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:11.964088+0000","last_change":"2024-09-18T07:40:56.968567+0000","last_active":"2024-09-18T07:41:11.964088+0000","last_peered":"2024-09-18T07:41:11.964088+0000","last_clean":"2024-09-18T07:41:11.964088+0000","last_became_active":"2024-09-18T07:39:29.704318+0000","last_became_peered":"2024-09-18T07:39:29.704318+0000","last_unstale":"2024-09-18T07:41:11.964088+0000","last_undegraded":"2024-09-18T07:41:11.964088+0000","last_fullsized":"2024-09-18T07:41:11.964088+0000","mapping_epoch":289,"log_start":"303'4100","ondisk_log_start":"303'4100","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7695","last_scrub_stamp":"2024-09-18T07:40:56.968381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.968381+0000","objects_scrubbed":7617,"log_size":4707,"log_dups_size":0,"ondisk_log_size":4707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.940759+0000","scrub_duration":2651,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505085952,"num_objects":7707,"num_object_clones":0,"num_object_copies":23121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8807,"num_write_kb":528448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"323'15","reported_seq":594,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.981182+0000","last_change":"2024-09-18T07:40:10.816522+0000","last_active":"2024-09-18T07:41:09.981182+0000","last_peered":"2024-09-18T07:41:09.981182+0000","last_clean":"2024-09-18T07:41:09.981182+0000","last_became_active":"2024-09-18T07:37:27.994144+0000","last_became_peered":"2024-09-18T07:37:27.994144+0000","last_unstale":"2024-09-18T07:41:09.981182+0000","last_undegraded":"2024-09-18T07:41:09.981182+0000","last_fullsized":"2024-09-18T07:41:09.981182+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:10.816290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:10.816290+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:19.604228+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":580,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.988905+0000","last_change":"2024-09-18T07:40:45.343597+0000","last_active":"2024-09-18T07:41:09.988905+0000","last_peered":"2024-09-18T07:41:09.988905+0000","last_clean":"2024-09-18T07:41:09.988905+0000","last_became_active":"2024-09-18T07:37:43.075988+0000","last_became_peered":"2024-09-18T07:37:43.075988+0000","last_unstale":"2024-09-18T07:41:09.988905+0000","last_undegraded":"2024-09-18T07:41:09.988905+0000","last_fullsized":"2024-09-18T07:41:09.988905+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:45.343533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:45.343533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:58.374007+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":552,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.980617+0000","last_change":"2024-09-18T07:39:01.216439+0000","last_active":"2024-09-18T07:41:09.980617+0000","last_peered":"2024-09-18T07:41:09.980617+0000","last_clean":"2024-09-18T07:41:09.980617+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:41:09.980617+0000","last_undegraded":"2024-09-18T07:41:09.980617+0000","last_fullsized":"2024-09-18T07:41:09.980617+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:01.216305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:01.216305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:33.453222+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":566,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.977812+0000","last_change":"2024-09-18T07:40:41.786465+0000","last_active":"2024-09-18T07:41:09.977812+0000","last_peered":"2024-09-18T07:41:09.977812+0000","last_clean":"2024-09-18T07:41:09.977812+0000","last_became_active":"2024-09-18T07:35:01.410890+0000","last_became_peered":"2024-09-18T07:35:01.410890+0000","last_unstale":"2024-09-18T07:41:09.977812+0000","last_undegraded":"2024-09-18T07:41:09.977812+0000","last_fullsized":"2024-09-18T07:41:09.977812+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:41.786315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:41.786315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:50.659737+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"325'8910","reported_seq":8993,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:11.945195+0000","last_change":"2024-09-18T07:39:29.701848+0000","last_active":"2024-09-18T07:41:11.945195+0000","last_peered":"2024-09-18T07:41:11.945195+0000","last_clean":"2024-09-18T07:41:11.945195+0000","last_became_active":"2024-09-18T07:39:29.701593+0000","last_became_peered":"2024-09-18T07:39:29.701593+0000","last_unstale":"2024-09-18T07:41:11.945195+0000","last_undegraded":"2024-09-18T07:41:11.945195+0000","last_fullsized":"2024-09-18T07:41:11.945195+0000","mapping_epoch":289,"log_start":"303'4200","ondisk_log_start":"303'4200","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.678492+0000","objects_scrubbed":0,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":509739008,"num_objects":7778,"num_object_clones":0,"num_object_copies":23334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8910,"num_write_kb":534016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"323'35","reported_seq":235,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.981155+0000","last_change":"2024-09-18T07:39:25.932376+0000","last_active":"2024-09-18T07:41:09.981155+0000","last_peered":"2024-09-18T07:41:09.981155+0000","last_clean":"2024-09-18T07:41:09.981155+0000","last_became_active":"2024-09-18T07:37:43.063319+0000","last_became_peered":"2024-09-18T07:37:43.063319+0000","last_unstale":"2024-09-18T07:41:09.981155+0000","last_undegraded":"2024-09-18T07:41:09.981155+0000","last_fullsized":"2024-09-18T07:41:09.981155+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:39:25.932315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.932315+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:35.039633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":354,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.991347+0000","last_change":"2024-09-18T07:40:56.546321+0000","last_active":"2024-09-18T07:41:09.991347+0000","last_peered":"2024-09-18T07:41:09.991347+0000","last_clean":"2024-09-18T07:41:09.991347+0000","last_became_active":"2024-09-18T07:37:43.072894+0000","last_became_peered":"2024-09-18T07:37:43.072894+0000","last_unstale":"2024-09-18T07:41:09.991347+0000","last_undegraded":"2024-09-18T07:41:09.991347+0000","last_fullsized":"2024-09-18T07:41:09.991347+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:56.546251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.546251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:09.934892+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":567,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.980912+0000","last_change":"2024-09-18T07:39:38.400864+0000","last_active":"2024-09-18T07:41:09.980912+0000","last_peered":"2024-09-18T07:41:09.980912+0000","last_clean":"2024-09-18T07:41:09.980912+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:41:09.980912+0000","last_undegraded":"2024-09-18T07:41:09.980912+0000","last_fullsized":"2024-09-18T07:41:09.980912+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:38.400785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:38.400785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:02.729842+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":324,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.987100+0000","last_change":"2024-09-18T07:40:40.453421+0000","last_active":"2024-09-18T07:41:09.987100+0000","last_peered":"2024-09-18T07:41:09.987100+0000","last_clean":"2024-09-18T07:41:09.987100+0000","last_became_active":"2024-09-18T07:35:01.406980+0000","last_became_peered":"2024-09-18T07:35:01.406980+0000","last_unstale":"2024-09-18T07:41:09.987100+0000","last_undegraded":"2024-09-18T07:41:09.987100+0000","last_fullsized":"2024-09-18T07:41:09.987100+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:40.453345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.453345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:50.849963+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"325'8810","reported_seq":8967,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:11.601483+0000","last_change":"2024-09-18T07:40:43.229079+0000","last_active":"2024-09-18T07:41:11.601483+0000","last_peered":"2024-09-18T07:41:11.601483+0000","last_clean":"2024-09-18T07:41:11.601483+0000","last_became_active":"2024-09-18T07:39:29.700244+0000","last_became_peered":"2024-09-18T07:39:29.700244+0000","last_unstale":"2024-09-18T07:41:11.601483+0000","last_undegraded":"2024-09-18T07:41:11.601483+0000","last_fullsized":"2024-09-18T07:41:11.601483+0000","mapping_epoch":289,"log_start":"303'4100","ondisk_log_start":"303'4100","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'6831","last_scrub_stamp":"2024-09-18T07:40:43.228883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:43.228883+0000","objects_scrubbed":6759,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:07.962708+0000","scrub_duration":2043,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508559360,"num_objects":7760,"num_object_clones":0,"num_object_copies":23280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8810,"num_write_kb":530240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"323'18","reported_seq":604,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.993512+0000","last_change":"2024-09-18T07:40:10.263303+0000","last_active":"2024-09-18T07:41:09.993512+0000","last_peered":"2024-09-18T07:41:09.993512+0000","last_clean":"2024-09-18T07:41:09.993512+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:41:09.993512+0000","last_undegraded":"2024-09-18T07:41:09.993512+0000","last_fullsized":"2024-09-18T07:41:09.993512+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-18T07:40:10.263206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:10.263206+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:30.221303+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":324,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.995092+0000","last_change":"2024-09-18T07:40:43.498451+0000","last_active":"2024-09-18T07:41:09.995092+0000","last_peered":"2024-09-18T07:41:09.995092+0000","last_clean":"2024-09-18T07:41:09.995092+0000","last_became_active":"2024-09-18T07:35:01.398499+0000","last_became_peered":"2024-09-18T07:35:01.398499+0000","last_unstale":"2024-09-18T07:41:09.995092+0000","last_undegraded":"2024-09-18T07:41:09.995092+0000","last_fullsized":"2024-09-18T07:41:09.995092+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:43.498377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:43.498377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:46.724354+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":576,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.977942+0000","last_change":"2024-09-18T07:40:09.371668+0000","last_active":"2024-09-18T07:41:09.977942+0000","last_peered":"2024-09-18T07:41:09.977942+0000","last_clean":"2024-09-18T07:41:09.977942+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:41:09.977942+0000","last_undegraded":"2024-09-18T07:41:09.977942+0000","last_fullsized":"2024-09-18T07:41:09.977942+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:09.371577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:09.371577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:35.948767+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":561,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.986094+0000","last_change":"2024-09-18T07:40:02.932758+0000","last_active":"2024-09-18T07:41:09.986094+0000","last_peered":"2024-09-18T07:41:09.986094+0000","last_clean":"2024-09-18T07:41:09.986094+0000","last_became_active":"2024-09-18T07:35:01.418735+0000","last_became_peered":"2024-09-18T07:35:01.418735+0000","last_unstale":"2024-09-18T07:41:09.986094+0000","last_undegraded":"2024-09-18T07:41:09.986094+0000","last_fullsized":"2024-09-18T07:41:09.986094+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:02.932690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:02.932690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:27.869866+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"325'8789","reported_seq":8956,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:12.542367+0000","last_change":"2024-09-18T07:40:54.295785+0000","last_active":"2024-09-18T07:41:12.542367+0000","last_peered":"2024-09-18T07:41:12.542367+0000","last_clean":"2024-09-18T07:41:12.542367+0000","last_became_active":"2024-09-18T07:39:29.701617+0000","last_became_peered":"2024-09-18T07:39:29.701617+0000","last_unstale":"2024-09-18T07:41:12.542367+0000","last_undegraded":"2024-09-18T07:41:12.542367+0000","last_fullsized":"2024-09-18T07:41:12.542367+0000","mapping_epoch":289,"log_start":"301'3400","ondisk_log_start":"301'3400","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7478","last_scrub_stamp":"2024-09-18T07:40:54.295602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:54.295602+0000","objects_scrubbed":7411,"log_size":5389,"log_dups_size":0,"ondisk_log_size":5389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:14.431832+0000","scrub_duration":2507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":501153792,"num_objects":7647,"num_object_clones":0,"num_object_copies":22941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8789,"num_write_kb":525952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"323'27","reported_seq":697,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.990124+0000","last_change":"2024-09-18T07:39:27.484003+0000","last_active":"2024-09-18T07:41:09.990124+0000","last_peered":"2024-09-18T07:41:09.990124+0000","last_clean":"2024-09-18T07:41:09.990124+0000","last_became_active":"2024-09-18T07:37:43.075552+0000","last_became_peered":"2024-09-18T07:37:43.075552+0000","last_unstale":"2024-09-18T07:41:09.990124+0000","last_undegraded":"2024-09-18T07:41:09.990124+0000","last_fullsized":"2024-09-18T07:41:09.990124+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:39:27.483930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:27.483930+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":338,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.990626+0000","last_change":"2024-09-18T07:36:09.261574+0000","last_active":"2024-09-18T07:41:09.990626+0000","last_peered":"2024-09-18T07:41:09.990626+0000","last_clean":"2024-09-18T07:41:09.990626+0000","last_became_active":"2024-09-18T07:35:01.406695+0000","last_became_peered":"2024-09-18T07:35:01.406695+0000","last_unstale":"2024-09-18T07:41:09.990626+0000","last_undegraded":"2024-09-18T07:41:09.990626+0000","last_fullsized":"2024-09-18T07:41:09.990626+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:36:09.261516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:36:09.261516+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:24.586284+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"240'12","reported_seq":433,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.989826+0000","last_change":"2024-09-18T07:38:55.490250+0000","last_active":"2024-09-18T07:41:09.989826+0000","last_peered":"2024-09-18T07:41:09.989826+0000","last_clean":"2024-09-18T07:41:09.989826+0000","last_became_active":"2024-09-18T07:37:43.056714+0000","last_became_peered":"2024-09-18T07:37:43.056714+0000","last_unstale":"2024-09-18T07:41:09.989826+0000","last_undegraded":"2024-09-18T07:41:09.989826+0000","last_fullsized":"2024-09-18T07:41:09.989826+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:38:55.490178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:55.490178+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:27.494193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":552,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.989505+0000","last_change":"2024-09-18T07:40:17.338311+0000","last_active":"2024-09-18T07:41:09.989505+0000","last_peered":"2024-09-18T07:41:09.989505+0000","last_clean":"2024-09-18T07:41:09.989505+0000","last_became_active":"2024-09-18T07:37:43.078121+0000","last_became_peered":"2024-09-18T07:37:43.078121+0000","last_unstale":"2024-09-18T07:41:09.989505+0000","last_undegraded":"2024-09-18T07:41:09.989505+0000","last_fullsized":"2024-09-18T07:41:09.989505+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:17.338244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:17.338244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:40.095453+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":191,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.979733+0000","last_change":"2024-09-18T07:39:37.072844+0000","last_active":"2024-09-18T07:41:09.979733+0000","last_peered":"2024-09-18T07:41:09.979733+0000","last_clean":"2024-09-18T07:41:09.979733+0000","last_became_active":"2024-09-18T07:37:43.087994+0000","last_became_peered":"2024-09-18T07:37:43.087994+0000","last_unstale":"2024-09-18T07:41:09.979733+0000","last_undegraded":"2024-09-18T07:41:09.979733+0000","last_fullsized":"2024-09-18T07:41:09.979733+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.072782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.072782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:58.222614+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"323'12","reported_seq":603,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.987269+0000","last_change":"2024-09-18T07:40:20.931967+0000","last_active":"2024-09-18T07:41:09.987269+0000","last_peered":"2024-09-18T07:41:09.987269+0000","last_clean":"2024-09-18T07:41:09.987269+0000","last_became_active":"2024-09-18T07:35:01.407787+0000","last_became_peered":"2024-09-18T07:35:01.407787+0000","last_unstale":"2024-09-18T07:41:09.987269+0000","last_undegraded":"2024-09-18T07:41:09.987269+0000","last_fullsized":"2024-09-18T07:41:09.987269+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-18T07:40:20.931889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:20.931889+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:43.496379+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":558,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.989682+0000","last_change":"2024-09-18T07:40:06.268555+0000","last_active":"2024-09-18T07:41:09.989682+0000","last_peered":"2024-09-18T07:41:09.989682+0000","last_clean":"2024-09-18T07:41:09.989682+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:41:09.989682+0000","last_undegraded":"2024-09-18T07:41:09.989682+0000","last_fullsized":"2024-09-18T07:41:09.989682+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:06.268501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:06.268501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":338,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.991227+0000","last_change":"2024-09-18T07:40:51.554184+0000","last_active":"2024-09-18T07:41:09.991227+0000","last_peered":"2024-09-18T07:41:09.991227+0000","last_clean":"2024-09-18T07:41:09.991227+0000","last_became_active":"2024-09-18T07:37:43.079313+0000","last_became_peered":"2024-09-18T07:37:43.079313+0000","last_unstale":"2024-09-18T07:41:09.991227+0000","last_undegraded":"2024-09-18T07:41:09.991227+0000","last_fullsized":"2024-09-18T07:41:09.991227+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:51.554073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:51.554073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:00.652147+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"323'15","reported_seq":595,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.978541+0000","last_change":"2024-09-18T07:35:01.448510+0000","last_active":"2024-09-18T07:41:09.978541+0000","last_peered":"2024-09-18T07:41:09.978541+0000","last_clean":"2024-09-18T07:41:09.978541+0000","last_became_active":"2024-09-18T07:35:01.414458+0000","last_became_peered":"2024-09-18T07:35:01.414458+0000","last_unstale":"2024-09-18T07:41:09.978541+0000","last_undegraded":"2024-09-18T07:41:09.978541+0000","last_fullsized":"2024-09-18T07:41:09.978541+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:25.649724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":346,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.989687+0000","last_change":"2024-09-18T07:40:16.673481+0000","last_active":"2024-09-18T07:41:09.989687+0000","last_peered":"2024-09-18T07:41:09.989687+0000","last_clean":"2024-09-18T07:41:09.989687+0000","last_became_active":"2024-09-18T07:37:43.056675+0000","last_became_peered":"2024-09-18T07:37:43.056675+0000","last_unstale":"2024-09-18T07:41:09.989687+0000","last_undegraded":"2024-09-18T07:41:09.989687+0000","last_fullsized":"2024-09-18T07:41:09.989687+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:16.673420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:16.673420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:33.901439+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":573,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.987043+0000","last_change":"2024-09-18T07:39:30.966887+0000","last_active":"2024-09-18T07:41:09.987043+0000","last_peered":"2024-09-18T07:41:09.987043+0000","last_clean":"2024-09-18T07:41:09.987043+0000","last_became_active":"2024-09-18T07:35:01.420133+0000","last_became_peered":"2024-09-18T07:35:01.420133+0000","last_unstale":"2024-09-18T07:41:09.987043+0000","last_undegraded":"2024-09-18T07:41:09.987043+0000","last_fullsized":"2024-09-18T07:41:09.987043+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:30.966840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:30.966840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":554,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.981104+0000","last_change":"2024-09-18T07:37:48.861965+0000","last_active":"2024-09-18T07:41:09.981104+0000","last_peered":"2024-09-18T07:41:09.981104+0000","last_clean":"2024-09-18T07:41:09.981104+0000","last_became_active":"2024-09-18T07:35:01.396305+0000","last_became_peered":"2024-09-18T07:35:01.396305+0000","last_unstale":"2024-09-18T07:41:09.981104+0000","last_undegraded":"2024-09-18T07:41:09.981104+0000","last_fullsized":"2024-09-18T07:41:09.981104+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:48.861784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:37:48.861784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:30.426468+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"323'15","reported_seq":621,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.992265+0000","last_change":"2024-09-18T07:40:52.366957+0000","last_active":"2024-09-18T07:41:09.992265+0000","last_peered":"2024-09-18T07:41:09.992265+0000","last_clean":"2024-09-18T07:41:09.992265+0000","last_became_active":"2024-09-18T07:35:01.416744+0000","last_became_peered":"2024-09-18T07:35:01.416744+0000","last_unstale":"2024-09-18T07:41:09.992265+0000","last_undegraded":"2024-09-18T07:41:09.992265+0000","last_fullsized":"2024-09-18T07:41:09.992265+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:52.366672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:52.366672+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:10.105909+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":578,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.978273+0000","last_change":"2024-09-18T07:40:39.723346+0000","last_active":"2024-09-18T07:41:09.978273+0000","last_peered":"2024-09-18T07:41:09.978273+0000","last_clean":"2024-09-18T07:41:09.978273+0000","last_became_active":"2024-09-18T07:35:01.414024+0000","last_became_peered":"2024-09-18T07:35:01.414024+0000","last_unstale":"2024-09-18T07:41:09.978273+0000","last_undegraded":"2024-09-18T07:41:09.978273+0000","last_fullsized":"2024-09-18T07:41:09.978273+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:39.723284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:39.723284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:47.455866+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"173'10","reported_seq":705,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.987699+0000","last_change":"2024-09-18T07:39:32.947933+0000","last_active":"2024-09-18T07:41:09.987699+0000","last_peered":"2024-09-18T07:41:09.987699+0000","last_clean":"2024-09-18T07:41:09.987699+0000","last_became_active":"2024-09-18T07:35:01.407568+0000","last_became_peered":"2024-09-18T07:35:01.407568+0000","last_unstale":"2024-09-18T07:41:09.987699+0000","last_undegraded":"2024-09-18T07:41:09.987699+0000","last_fullsized":"2024-09-18T07:41:09.987699+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:39:32.947887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:32.947887+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":548,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.995569+0000","last_change":"2024-09-18T07:40:15.264060+0000","last_active":"2024-09-18T07:41:09.995569+0000","last_peered":"2024-09-18T07:41:09.995569+0000","last_clean":"2024-09-18T07:41:09.995569+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:41:09.995569+0000","last_undegraded":"2024-09-18T07:41:09.995569+0000","last_fullsized":"2024-09-18T07:41:09.995569+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:15.263990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:15.263990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:37.761145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"323'20","reported_seq":333,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.987896+0000","last_change":"2024-09-18T07:40:19.070502+0000","last_active":"2024-09-18T07:41:09.987896+0000","last_peered":"2024-09-18T07:41:09.987896+0000","last_clean":"2024-09-18T07:41:09.987896+0000","last_became_active":"2024-09-18T07:40:19.069759+0000","last_became_peered":"2024-09-18T07:40:19.069759+0000","last_unstale":"2024-09-18T07:41:09.987896+0000","last_undegraded":"2024-09-18T07:41:09.987896+0000","last_fullsized":"2024-09-18T07:41:09.987896+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:36:02.975863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:02.975863+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:25.563322+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":332,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.987813+0000","last_change":"2024-09-18T07:40:20.702012+0000","last_active":"2024-09-18T07:41:09.987813+0000","last_peered":"2024-09-18T07:41:09.987813+0000","last_clean":"2024-09-18T07:41:09.987813+0000","last_became_active":"2024-09-18T07:35:01.400197+0000","last_became_peered":"2024-09-18T07:35:01.400197+0000","last_unstale":"2024-09-18T07:41:09.987813+0000","last_undegraded":"2024-09-18T07:41:09.987813+0000","last_fullsized":"2024-09-18T07:41:09.987813+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:20.701927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:20.701927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:38.393698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":685,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.992038+0000","last_change":"2024-09-18T07:40:40.352088+0000","last_active":"2024-09-18T07:41:09.992038+0000","last_peered":"2024-09-18T07:41:09.992038+0000","last_clean":"2024-09-18T07:41:09.992038+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:41:09.992038+0000","last_undegraded":"2024-09-18T07:41:09.992038+0000","last_fullsized":"2024-09-18T07:41:09.992038+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:40:40.352040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.352040+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:06.126960+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":566,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.977747+0000","last_change":"2024-09-18T07:40:40.761823+0000","last_active":"2024-09-18T07:41:09.977747+0000","last_peered":"2024-09-18T07:41:09.977747+0000","last_clean":"2024-09-18T07:41:09.977747+0000","last_became_active":"2024-09-18T07:35:01.410561+0000","last_became_peered":"2024-09-18T07:35:01.410561+0000","last_unstale":"2024-09-18T07:41:09.977747+0000","last_undegraded":"2024-09-18T07:41:09.977747+0000","last_fullsized":"2024-09-18T07:41:09.977747+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:40.761753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.761753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:51.341252+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"323'19","reported_seq":657,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.994185+0000","last_change":"2024-09-18T07:40:16.313850+0000","last_active":"2024-09-18T07:41:09.994185+0000","last_peered":"2024-09-18T07:41:09.994185+0000","last_clean":"2024-09-18T07:41:09.994185+0000","last_became_active":"2024-09-18T07:38:41.143108+0000","last_became_peered":"2024-09-18T07:38:41.143108+0000","last_unstale":"2024-09-18T07:41:09.994185+0000","last_undegraded":"2024-09-18T07:41:09.994185+0000","last_fullsized":"2024-09-18T07:41:09.994185+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:40:16.313749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:16.313749+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:44.518197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":580,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.995680+0000","last_change":"2024-09-18T07:40:40.149372+0000","last_active":"2024-09-18T07:41:09.995680+0000","last_peered":"2024-09-18T07:41:09.995680+0000","last_clean":"2024-09-18T07:41:09.995680+0000","last_became_active":"2024-09-18T07:37:43.061577+0000","last_became_peered":"2024-09-18T07:37:43.061577+0000","last_unstale":"2024-09-18T07:41:09.995680+0000","last_undegraded":"2024-09-18T07:41:09.995680+0000","last_fullsized":"2024-09-18T07:41:09.995680+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:40.149313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.149313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:59.476037+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":243,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.989637+0000","last_change":"2024-09-18T07:36:13.118114+0000","last_active":"2024-09-18T07:41:09.989637+0000","last_peered":"2024-09-18T07:41:09.989637+0000","last_clean":"2024-09-18T07:41:09.989637+0000","last_became_active":"2024-09-18T07:36:13.117584+0000","last_became_peered":"2024-09-18T07:36:13.117584+0000","last_unstale":"2024-09-18T07:41:09.989637+0000","last_undegraded":"2024-09-18T07:41:09.989637+0000","last_fullsized":"2024-09-18T07:41:09.989637+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:30:59.305166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:22.335589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":47,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.977559+0000","last_change":"2024-09-18T07:40:19.066294+0000","last_active":"2024-09-18T07:41:09.977559+0000","last_peered":"2024-09-18T07:41:09.977559+0000","last_clean":"2024-09-18T07:41:09.977559+0000","last_became_active":"2024-09-18T07:40:19.065468+0000","last_became_peered":"2024-09-18T07:40:19.065468+0000","last_unstale":"2024-09-18T07:41:09.977559+0000","last_undegraded":"2024-09-18T07:41:09.977559+0000","last_fullsized":"2024-09-18T07:41:09.977559+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.892377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.892377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:03.885516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"323'15","reported_seq":442,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.978058+0000","last_change":"2024-09-18T07:40:06.762667+0000","last_active":"2024-09-18T07:41:09.978058+0000","last_peered":"2024-09-18T07:41:09.978058+0000","last_clean":"2024-09-18T07:41:09.978058+0000","last_became_active":"2024-09-18T07:37:43.067598+0000","last_became_peered":"2024-09-18T07:37:43.067598+0000","last_unstale":"2024-09-18T07:41:09.978058+0000","last_undegraded":"2024-09-18T07:41:09.978058+0000","last_fullsized":"2024-09-18T07:41:09.978058+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:06.762589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:06.762589+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:16.226171+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":332,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.991158+0000","last_change":"2024-09-18T07:40:13.729483+0000","last_active":"2024-09-18T07:41:09.991158+0000","last_peered":"2024-09-18T07:41:09.991158+0000","last_clean":"2024-09-18T07:41:09.991158+0000","last_became_active":"2024-09-18T07:35:01.405287+0000","last_became_peered":"2024-09-18T07:35:01.405287+0000","last_unstale":"2024-09-18T07:41:09.991158+0000","last_undegraded":"2024-09-18T07:41:09.991158+0000","last_fullsized":"2024-09-18T07:41:09.991158+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:13.729415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:13.729415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:30.652065+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":47,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.977584+0000","last_change":"2024-09-18T07:40:19.068620+0000","last_active":"2024-09-18T07:41:09.977584+0000","last_peered":"2024-09-18T07:41:09.977584+0000","last_clean":"2024-09-18T07:41:09.977584+0000","last_became_active":"2024-09-18T07:40:19.068237+0000","last_became_peered":"2024-09-18T07:40:19.068237+0000","last_unstale":"2024-09-18T07:41:09.977584+0000","last_undegraded":"2024-09-18T07:41:09.977584+0000","last_fullsized":"2024-09-18T07:41:09.977584+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:56.920384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:56.920384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:27.660508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":560,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.977373+0000","last_change":"2024-09-18T07:40:05.788436+0000","last_active":"2024-09-18T07:41:09.977373+0000","last_peered":"2024-09-18T07:41:09.977373+0000","last_clean":"2024-09-18T07:41:09.977373+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:41:09.977373+0000","last_undegraded":"2024-09-18T07:41:09.977373+0000","last_fullsized":"2024-09-18T07:41:09.977373+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:05.788371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:05.788371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:11.393036+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"323'19","reported_seq":662,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.987874+0000","last_change":"2024-09-18T07:37:43.138620+0000","last_active":"2024-09-18T07:41:09.987874+0000","last_peered":"2024-09-18T07:41:09.987874+0000","last_clean":"2024-09-18T07:41:09.987874+0000","last_became_active":"2024-09-18T07:37:43.066208+0000","last_became_peered":"2024-09-18T07:37:43.066208+0000","last_unstale":"2024-09-18T07:41:09.987874+0000","last_undegraded":"2024-09-18T07:41:09.987874+0000","last_fullsized":"2024-09-18T07:41:09.987874+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:36:29.051437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:29.051437+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.329451+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":564,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.989914+0000","last_change":"2024-09-18T07:35:01.415691+0000","last_active":"2024-09-18T07:41:09.989914+0000","last_peered":"2024-09-18T07:41:09.989914+0000","last_clean":"2024-09-18T07:41:09.989914+0000","last_became_active":"2024-09-18T07:35:01.415261+0000","last_became_peered":"2024-09-18T07:35:01.415261+0000","last_unstale":"2024-09-18T07:41:09.989914+0000","last_undegraded":"2024-09-18T07:41:09.989914+0000","last_fullsized":"2024-09-18T07:41:09.989914+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:58.734027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:58.734027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:23.358852+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":346,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.987186+0000","last_change":"2024-09-18T07:40:14.681431+0000","last_active":"2024-09-18T07:41:09.987186+0000","last_peered":"2024-09-18T07:41:09.987186+0000","last_clean":"2024-09-18T07:41:09.987186+0000","last_became_active":"2024-09-18T07:37:43.057483+0000","last_became_peered":"2024-09-18T07:37:43.057483+0000","last_unstale":"2024-09-18T07:41:09.987186+0000","last_undegraded":"2024-09-18T07:41:09.987186+0000","last_fullsized":"2024-09-18T07:41:09.987186+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:14.681292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:14.681292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:39.636405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":569,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.986233+0000","last_change":"2024-09-18T07:39:02.930467+0000","last_active":"2024-09-18T07:41:09.986233+0000","last_peered":"2024-09-18T07:41:09.986233+0000","last_clean":"2024-09-18T07:41:09.986233+0000","last_became_active":"2024-09-18T07:37:43.067066+0000","last_became_peered":"2024-09-18T07:37:43.067066+0000","last_unstale":"2024-09-18T07:41:09.986233+0000","last_undegraded":"2024-09-18T07:41:09.986233+0000","last_fullsized":"2024-09-18T07:41:09.986233+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:02.930392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:02.930392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:26.000097+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"323'8","reported_seq":614,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.984445+0000","last_change":"2024-09-18T07:37:43.134956+0000","last_active":"2024-09-18T07:41:09.984445+0000","last_peered":"2024-09-18T07:41:09.984445+0000","last_clean":"2024-09-18T07:41:09.984445+0000","last_became_active":"2024-09-18T07:37:43.067503+0000","last_became_peered":"2024-09-18T07:37:43.067503+0000","last_unstale":"2024-09-18T07:41:09.984445+0000","last_undegraded":"2024-09-18T07:41:09.984445+0000","last_fullsized":"2024-09-18T07:41:09.984445+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:36:40.908629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:40.908629+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:09.334510+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":556,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.978276+0000","last_change":"2024-09-18T07:40:11.843693+0000","last_active":"2024-09-18T07:41:09.978276+0000","last_peered":"2024-09-18T07:41:09.978276+0000","last_clean":"2024-09-18T07:41:09.978276+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:41:09.978276+0000","last_undegraded":"2024-09-18T07:41:09.978276+0000","last_fullsized":"2024-09-18T07:41:09.978276+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:11.843625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:11.843625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:41.454218+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":191,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.983414+0000","last_change":"2024-09-18T07:40:40.106989+0000","last_active":"2024-09-18T07:41:09.983414+0000","last_peered":"2024-09-18T07:41:09.983414+0000","last_clean":"2024-09-18T07:41:09.983414+0000","last_became_active":"2024-09-18T07:37:43.061081+0000","last_became_peered":"2024-09-18T07:37:43.061081+0000","last_unstale":"2024-09-18T07:41:09.983414+0000","last_undegraded":"2024-09-18T07:41:09.983414+0000","last_fullsized":"2024-09-18T07:41:09.983414+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:40.106921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.106921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:57.884280+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":340,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.987723+0000","last_change":"2024-09-18T07:40:38.492338+0000","last_active":"2024-09-18T07:41:09.987723+0000","last_peered":"2024-09-18T07:41:09.987723+0000","last_clean":"2024-09-18T07:41:09.987723+0000","last_became_active":"2024-09-18T07:35:01.400396+0000","last_became_peered":"2024-09-18T07:35:01.400396+0000","last_unstale":"2024-09-18T07:41:09.987723+0000","last_undegraded":"2024-09-18T07:41:09.987723+0000","last_fullsized":"2024-09-18T07:41:09.987723+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:38.492269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.492269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:58.522141+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":242,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.992091+0000","last_change":"2024-09-18T07:36:13.118009+0000","last_active":"2024-09-18T07:41:09.992091+0000","last_peered":"2024-09-18T07:41:09.992091+0000","last_clean":"2024-09-18T07:41:09.992091+0000","last_became_active":"2024-09-18T07:36:13.117154+0000","last_became_peered":"2024-09-18T07:36:13.117154+0000","last_unstale":"2024-09-18T07:41:09.992091+0000","last_undegraded":"2024-09-18T07:41:09.992091+0000","last_fullsized":"2024-09-18T07:41:09.992091+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.071629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.071629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:35.347289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":573,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.984641+0000","last_change":"2024-09-18T07:39:35.902385+0000","last_active":"2024-09-18T07:41:09.984641+0000","last_peered":"2024-09-18T07:41:09.984641+0000","last_clean":"2024-09-18T07:41:09.984641+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:41:09.984641+0000","last_undegraded":"2024-09-18T07:41:09.984641+0000","last_fullsized":"2024-09-18T07:41:09.984641+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:39:35.901306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.901306+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"323'19","reported_seq":207,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.980229+0000","last_change":"2024-09-18T07:37:43.258255+0000","last_active":"2024-09-18T07:41:09.980229+0000","last_peered":"2024-09-18T07:41:09.980229+0000","last_clean":"2024-09-18T07:41:09.980229+0000","last_became_active":"2024-09-18T07:37:43.070186+0000","last_became_peered":"2024-09-18T07:37:43.070186+0000","last_unstale":"2024-09-18T07:41:09.980229+0000","last_undegraded":"2024-09-18T07:41:09.980229+0000","last_fullsized":"2024-09-18T07:41:09.980229+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:34:58.432781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:34:58.432781+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:23.022602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":571,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-09-18T07:41:09.977904+0000","last_change":"2024-09-18T07:39:35.694805+0000","last_active":"2024-09-18T07:41:09.977904+0000","last_peered":"2024-09-18T07:41:09.977904+0000","last_clean":"2024-09-18T07:41:09.977904+0000","last_became_active":"2024-09-18T07:35:01.410469+0000","last_became_peered":"2024-09-18T07:35:01.410469+0000","last_unstale":"2024-09-18T07:41:09.977904+0000","last_undegraded":"2024-09-18T07:41:09.977904+0000","last_fullsized":"2024-09-18T07:41:09.977904+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.694741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.694741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":8110145560,"num_objects":123752,"num_object_clones":0,"num_object_copies":371256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123752,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141244,"num_write_kb":8479809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24347488256,"data_stored":24347476040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78944,"ondisk_log_size":78944,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1118,"num_read_kb":875,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":4664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":684,"ondisk_log_size":684,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":376,"num_read_kb":371,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201407,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4884864,"kb_used_data":4524652,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":88860288,"statfs":{"total":95995035648,"available":90992934912,"internally_reserved":0,"allocated":4633243648,"data_stored":4632525910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":368835018},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463045,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1815836,"kb_used_data":1500040,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":91929316,"statfs":{"total":95995035648,"available":94135619584,"internally_reserved":0,"allocated":1536040960,"data_stored":1535358647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":323353030},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724682,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4862256,"kb_used_data":4479840,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":88882896,"statfs":{"total":95995035648,"available":91016085504,"internally_reserved":0,"allocated":4587356160,"data_stored":4586650625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953617,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1797500,"kb_used_data":1509480,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91947652,"statfs":{"total":95995035648,"available":94154395648,"internally_reserved":0,"allocated":1545707520,"data_stored":1544993577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149845,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4557440,"kb_used_data":4048556,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":89187712,"statfs":{"total":95995035648,"available":91328217088,"internally_reserved":0,"allocated":4145721344,"data_stored":4145004701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":520944071},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411483,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1990568,"kb_used_data":1616472,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91754584,"statfs":{"total":95995035648,"available":93956694016,"internally_reserved":0,"allocated":1655267328,"data_stored":1654569860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575010,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4972680,"kb_used_data":4481912,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":88772472,"statfs":{"total":95995035648,"available":90903011328,"internally_reserved":0,"allocated":4589477888,"data_stored":4588748662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738535,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2349632,"kb_used_data":1981104,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":91395520,"statfs":{"total":95995035648,"available":93589012480,"internally_reserved":0,"allocated":2028650496,"data_stored":2027929763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377354693},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2001993728,"data_stored":2001993728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4566355968,"data_stored":4566351896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1520893952,"data_stored":1520893952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4054908928,"data_stored":4054908928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1536557056,"data_stored":1536557056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4552986624,"data_stored":4552982552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1526595584,"data_stored":1526595584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4587196416,"data_stored":4587192344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:41:16.366 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-09-18T07:41:16.458 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:16.494 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:16.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:16.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T07:41:16.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:16.629 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:16.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:16.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:16.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:16 smithi203 ceph-mon[32237]: pgmap v552: 158 pgs: 158 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-18T07:41:16.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:16 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3882260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:16.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:16 smithi203 ceph-mon[32237]: 5.17 scrub starts 2024-09-18T07:41:16.732 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:16.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:16.771 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:16.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:16.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:16.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:16.909 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:16.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:16.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:16 smithi137 ceph-mon[29408]: pgmap v552: 158 pgs: 158 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-18T07:41:16.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:16 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3882260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:16.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:16 smithi137 ceph-mon[29408]: 5.17 scrub starts 2024-09-18T07:41:16.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:16 smithi137 ceph-mon[39175]: pgmap v552: 158 pgs: 158 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-18T07:41:16.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:16 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3882260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:16.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:16 smithi137 ceph-mon[39175]: 5.17 scrub starts 2024-09-18T07:41:16.968 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:41:17.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:17.044 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:17.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:17.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:17.146 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:17.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:17.184 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:17.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:17.247 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:41:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:41:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:41:17.286 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:17.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:17.317 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:17.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:17.418 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:17.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:17.460 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:17.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:17.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:41:17.561 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:17.603 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:17.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:17.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:17.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:41:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:41:17.705 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:17.746 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:17.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:17.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:17.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:17.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:17.893 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:17.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:17.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:17 smithi137 ceph-mon[29408]: 3.1f scrub starts 2024-09-18T07:41:17.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:17 smithi137 ceph-mon[29408]: 3.1f scrub ok 2024-09-18T07:41:17.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:17 smithi137 ceph-mon[29408]: 5.17 scrub ok 2024-09-18T07:41:17.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:17 smithi137 ceph-mon[39175]: 3.1f scrub starts 2024-09-18T07:41:17.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:17 smithi137 ceph-mon[39175]: 3.1f scrub ok 2024-09-18T07:41:17.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:17 smithi137 ceph-mon[39175]: 5.17 scrub ok 2024-09-18T07:41:17.936 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:17 smithi203 ceph-mon[32237]: 3.1f scrub starts 2024-09-18T07:41:17.936 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:17 smithi203 ceph-mon[32237]: 3.1f scrub ok 2024-09-18T07:41:17.936 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:17 smithi203 ceph-mon[32237]: 5.17 scrub ok 2024-09-18T07:41:17.948 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:17.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:17.995 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:41:17.995 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:41:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:18.032 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:18.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:18.133 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:18.171 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:18.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:18.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:18.273 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:18.300 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:18.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:18.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:18.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:18.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:18.441 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:18.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:18.545 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:18.580 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:18.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:18.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:18.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:18.709 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:18.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:18.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:18.811 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:18.843 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:18.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:18.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:18.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:18 smithi137 ceph-mon[29408]: 4.3 scrub starts 2024-09-18T07:41:18.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:18 smithi137 ceph-mon[29408]: 4.3 scrub ok 2024-09-18T07:41:18.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:18 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:18.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:18 smithi137 ceph-mon[29408]: pgmap v553: 158 pgs: 158 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-09-18T07:41:18.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:18 smithi137 ceph-mon[39175]: 4.3 scrub starts 2024-09-18T07:41:18.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:18 smithi137 ceph-mon[39175]: 4.3 scrub ok 2024-09-18T07:41:18.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:18 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:18.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:18 smithi137 ceph-mon[39175]: pgmap v553: 158 pgs: 158 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-09-18T07:41:18.946 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:18.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:18 smithi203 ceph-mon[32237]: 4.3 scrub starts 2024-09-18T07:41:18.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:18 smithi203 ceph-mon[32237]: 4.3 scrub ok 2024-09-18T07:41:18.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:18 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:18.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:18 smithi203 ceph-mon[32237]: pgmap v553: 158 pgs: 158 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-09-18T07:41:18.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:18.984 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:18.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:19.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:41:19.084 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:19.116 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:19.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:19.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:19.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:19.256 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:19.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:19.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:19.362 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:19.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:19.412 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:19.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:19.465 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T07:41:19.480 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:19.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:19.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:19.558 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:19.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:19.659 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:19.661 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:41:19.691 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:19.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:19.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:19 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:41:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:19 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:41:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:19 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2712094283' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:19 smithi203 ceph-mon[32237]: from='client.25858 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:19.794 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:19.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:19.847 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:19.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:19.887 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:41:19.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:19 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:41:19.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:19 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:41:19.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:19 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2712094283' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:19.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:19 smithi137 ceph-mon[29408]: from='client.25858 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:19.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:19 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:41:19.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:19 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:41:19.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:19 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2712094283' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:19.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:19 smithi137 ceph-mon[39175]: from='client.25858 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:19.948 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:19.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:19.987 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:19.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:20.088 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:20.123 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:20.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:20.225 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:20.259 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:20.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:20.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:20.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:20.398 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:20.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:20.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:20.499 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:41:20.499 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:41:20.500 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:20.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:20.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:20.634 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:20.671 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:20.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:20.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:20.737 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[29408]: 6.10 scrub starts 2024-09-18T07:41:20.737 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[29408]: 6.10 scrub ok 2024-09-18T07:41:20.737 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[29408]: 1.4 scrub starts 2024-09-18T07:41:20.737 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[29408]: 1.4 scrub ok 2024-09-18T07:41:20.737 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:41:20.737 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:41:20.738 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[29408]: from='client.25858 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:41:20.738 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[29408]: pgmap v554: 158 pgs: 158 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.42k op/s 2024-09-18T07:41:20.738 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[29408]: osdmap e328: 8 total, 8 up, 8 in 2024-09-18T07:41:20.738 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[29408]: osdmap e329: 8 total, 8 up, 8 in 2024-09-18T07:41:20.738 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[29408]: 6.c scrub starts 2024-09-18T07:41:20.738 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[29408]: 6.c scrub ok 2024-09-18T07:41:20.738 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4252852027' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:20.738 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[29408]: 9.8 scrub starts 2024-09-18T07:41:20.742 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[39175]: 6.10 scrub starts 2024-09-18T07:41:20.742 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[39175]: 6.10 scrub ok 2024-09-18T07:41:20.742 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[39175]: 1.4 scrub starts 2024-09-18T07:41:20.743 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[39175]: 1.4 scrub ok 2024-09-18T07:41:20.743 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:41:20.743 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:41:20.743 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[39175]: from='client.25858 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:41:20.743 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[39175]: pgmap v554: 158 pgs: 158 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.42k op/s 2024-09-18T07:41:20.743 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[39175]: osdmap e328: 8 total, 8 up, 8 in 2024-09-18T07:41:20.743 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[39175]: osdmap e329: 8 total, 8 up, 8 in 2024-09-18T07:41:20.743 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[39175]: 6.c scrub starts 2024-09-18T07:41:20.743 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[39175]: 6.c scrub ok 2024-09-18T07:41:20.743 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4252852027' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:20.743 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:20 smithi137 ceph-mon[39175]: 9.8 scrub starts 2024-09-18T07:41:20.773 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:20.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:20.820 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:20.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:20.920 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:41:20.920 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:41:20.920 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:41:20.920 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:41:20.921 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:41:20.921 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:41:20.921 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:41:20.921 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:41:20.921 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:41:20.921 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:41:20.921 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:41:20.922 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:41:20.922 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:41:20.922 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:41:20.922 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:41:20.922 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:41:20.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:20.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:20 smithi203 ceph-mon[32237]: 6.10 scrub starts 2024-09-18T07:41:20.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:20 smithi203 ceph-mon[32237]: 6.10 scrub ok 2024-09-18T07:41:20.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:20 smithi203 ceph-mon[32237]: 1.4 scrub starts 2024-09-18T07:41:20.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:20 smithi203 ceph-mon[32237]: 1.4 scrub ok 2024-09-18T07:41:20.950 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:20 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:41:20.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:20 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:41:20.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:20 smithi203 ceph-mon[32237]: from='client.25858 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:41:20.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:20 smithi203 ceph-mon[32237]: pgmap v554: 158 pgs: 158 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.42k op/s 2024-09-18T07:41:20.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:20 smithi203 ceph-mon[32237]: osdmap e328: 8 total, 8 up, 8 in 2024-09-18T07:41:20.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:20 smithi203 ceph-mon[32237]: osdmap e329: 8 total, 8 up, 8 in 2024-09-18T07:41:20.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:20 smithi203 ceph-mon[32237]: 6.c scrub starts 2024-09-18T07:41:20.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:20 smithi203 ceph-mon[32237]: 6.c scrub ok 2024-09-18T07:41:20.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:20 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4252852027' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:20.951 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:20 smithi203 ceph-mon[32237]: 9.8 scrub starts 2024-09-18T07:41:20.954 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:20.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:20.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:21.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:21.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:21.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:21.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:21.192 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:21.226 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:21.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:21.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:21.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:21.360 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:21.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:21.367 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:41:21.367 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:41:21.367 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-18T07:41:21.367 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T07:41:21.367 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T07:41:21.368 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-18T07:41:21.372 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T07:41:21.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:21.501 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:21.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:21.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:21.604 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:21.664 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:21.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:21.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:21.766 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:21.811 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:21.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:21.813 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:21.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:21.827 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:41:21.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:21.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:21 smithi137 ceph-mon[29408]: 9.7 scrub starts 2024-09-18T07:41:21.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:21 smithi137 ceph-mon[39175]: 9.7 scrub starts 2024-09-18T07:41:21.960 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:21.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:21.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:21 smithi203 ceph-mon[32237]: 9.7 scrub starts 2024-09-18T07:41:22.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:22.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:22.109 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:22.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:22.210 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:22.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:22.251 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:22.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:22.352 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:22.381 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:22.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:22.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:22.440 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:41:22.441 INFO:teuthology.orchestra.run.smithi203.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-18T07:41:22.441 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:41:22.482 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:22.514 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:22.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:22.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:22.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:22.649 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:22.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:22.751 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:22.785 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:41:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:22.789 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:22.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:22.890 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:22.914 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:22.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:22.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:22.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:22 smithi203 ceph-mon[32237]: 4.12 scrub starts 2024-09-18T07:41:22.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:22 smithi203 ceph-mon[32237]: 4.12 scrub ok 2024-09-18T07:41:22.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:22 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:41:22.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:22 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:41:22.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:22 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:41:22.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:22 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:41:22.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:22 smithi203 ceph-mon[32237]: pgmap v557: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-18T07:41:22.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:22 smithi203 ceph-mon[32237]: osdmap e330: 8 total, 8 up, 8 in 2024-09-18T07:41:22.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:22 smithi203 ceph-mon[32237]: 9.8 scrub ok 2024-09-18T07:41:22.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:22 smithi203 ceph-mon[32237]: 3.18 scrub starts 2024-09-18T07:41:23.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:23.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:23.057 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:23.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:23.067 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:41:23.067 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:41:23.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.067 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.15699.0:154195 9.5 9:aef12e55:::benchmark_data_smithi137_126072_object21695:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:41:23.067 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:22.392285+0000", 2024-09-18T07:41:23.067 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.025679674, 2024-09-18T07:41:23.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025714403, 2024-09-18T07:41:23.068 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:23.068 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:23.068 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:41:23.068 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:23.068 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.15699", 2024-09-18T07:41:23.068 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:41:23.068 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 154195 2024-09-18T07:41:23.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.068 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:23.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.068 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:23.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.392285+0000", 2024-09-18T07:41:23.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:23.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.392285+0000", 2024-09-18T07:41:23.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:23.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.392288+0000", 2024-09-18T07:41:23.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2129999999999998e-06 2024-09-18T07:41:23.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.070 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:23.070 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.392294+0000", 2024-09-18T07:41:23.070 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5250000000000002e-06 2024-09-18T07:41:23.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.070 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:23.070 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.392306+0000", 2024-09-18T07:41:23.070 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1649e-05 2024-09-18T07:41:23.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.070 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:23.070 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.392313+0000", 2024-09-18T07:41:23.070 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5020000000000003e-06 2024-09-18T07:41:23.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:23.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.392369+0000", 2024-09-18T07:41:23.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5628999999999999e-05 2024-09-18T07:41:23.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:23.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.392473+0000", 2024-09-18T07:41:23.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103791 2024-09-18T07:41:23.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-18T07:41:23.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.392543+0000", 2024-09-18T07:41:23.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0582999999999998e-05 2024-09-18T07:41:23.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:23.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.398232+0000", 2024-09-18T07:41:23.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005689131 2024-09-18T07:41:23.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:23.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408211+0000", 2024-09-18T07:41:23.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0099786730000000004 2024-09-18T07:41:23.073 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.073 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:23.073 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.073 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.15699.0:154196 9.5 9:a9c3a200:::benchmark_data_smithi137_126072_object21696:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:41:23.073 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:22.392909+0000", 2024-09-18T07:41:23.073 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.025056352, 2024-09-18T07:41:23.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025140458000000001, 2024-09-18T07:41:23.073 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:23.073 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:23.074 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:41:23.074 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:23.074 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.15699", 2024-09-18T07:41:23.074 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:41:23.074 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 154196 2024-09-18T07:41:23.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.074 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:23.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:23.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.392909+0000", 2024-09-18T07:41:23.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:23.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.392909+0000", 2024-09-18T07:41:23.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:23.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.392910+0000", 2024-09-18T07:41:23.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3400000000000001e-06 2024-09-18T07:41:23.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:23.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.392916+0000", 2024-09-18T07:41:23.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5289999999999999e-06 2024-09-18T07:41:23.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:23.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.392926+0000", 2024-09-18T07:41:23.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0581000000000001e-05 2024-09-18T07:41:23.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:23.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.392934+0000", 2024-09-18T07:41:23.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6499999999999996e-06 2024-09-18T07:41:23.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:23.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.394870+0000", 2024-09-18T07:41:23.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001935738 2024-09-18T07:41:23.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:23.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.394977+0000", 2024-09-18T07:41:23.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107178 2024-09-18T07:41:23.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-18T07:41:23.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.395102+0000", 2024-09-18T07:41:23.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012512100000000001 2024-09-18T07:41:23.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:23.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.398238+0000", 2024-09-18T07:41:23.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031364660000000001 2024-09-18T07:41:23.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:23.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408247+0000", 2024-09-18T07:41:23.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010008354000000001 2024-09-18T07:41:23.078 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.078 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:23.078 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.079 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.15699.0:154201 9.b 9:dc5fbe53:::benchmark_data_smithi137_126072_object21701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:41:23.079 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:22.397701+0000", 2024-09-18T07:41:23.079 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.020264349000000001, 2024-09-18T07:41:23.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020388406000000001, 2024-09-18T07:41:23.079 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:23.079 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:23.079 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:23.079 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:23.079 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.15699", 2024-09-18T07:41:23.079 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:41:23.079 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 154201 2024-09-18T07:41:23.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.079 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:23.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:23.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.397701+0000", 2024-09-18T07:41:23.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:23.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.397701+0000", 2024-09-18T07:41:23.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:23.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.397703+0000", 2024-09-18T07:41:23.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8029999999999999e-06 2024-09-18T07:41:23.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:23.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.397708+0000", 2024-09-18T07:41:23.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4569999999999996e-06 2024-09-18T07:41:23.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:23.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.397719+0000", 2024-09-18T07:41:23.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1358999999999999e-05 2024-09-18T07:41:23.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:23.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.397727+0000", 2024-09-18T07:41:23.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7710000000000006e-06 2024-09-18T07:41:23.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:23.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.397780+0000", 2024-09-18T07:41:23.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3276000000000003e-05 2024-09-18T07:41:23.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:23.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.397899+0000", 2024-09-18T07:41:23.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118657 2024-09-18T07:41:23.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:41:23.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.398019+0000", 2024-09-18T07:41:23.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120462 2024-09-18T07:41:23.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:23.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.403418+0000", 2024-09-18T07:41:23.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053986759999999998 2024-09-18T07:41:23.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:23.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.407214+0000", 2024-09-18T07:41:23.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0037958599999999999 2024-09-18T07:41:23.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:23.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408165+0000", 2024-09-18T07:41:23.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00095135100000000002 2024-09-18T07:41:23.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:23.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408227+0000", 2024-09-18T07:41:23.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1124000000000006e-05 2024-09-18T07:41:23.086 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.086 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:23.086 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.086 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.15699.0:154203 9.5 9:aced6ffa:::benchmark_data_smithi137_126072_object21703:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:41:23.086 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:22.398327+0000", 2024-09-18T07:41:23.086 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.019638557000000001, 2024-09-18T07:41:23.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019807394999999998, 2024-09-18T07:41:23.086 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:23.087 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:23.087 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:41:23.087 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:23.087 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.15699", 2024-09-18T07:41:23.087 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:41:23.087 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 154203 2024-09-18T07:41:23.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.087 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:23.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:23.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.398327+0000", 2024-09-18T07:41:23.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:23.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.398327+0000", 2024-09-18T07:41:23.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:23.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.398328+0000", 2024-09-18T07:41:23.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.299e-06 2024-09-18T07:41:23.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:23.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.398333+0000", 2024-09-18T07:41:23.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.848e-06 2024-09-18T07:41:23.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:23.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.398342+0000", 2024-09-18T07:41:23.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7890000000000006e-06 2024-09-18T07:41:23.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:23.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.398349+0000", 2024-09-18T07:41:23.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8029999999999999e-06 2024-09-18T07:41:23.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:23.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.398408+0000", 2024-09-18T07:41:23.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9169999999999998e-05 2024-09-18T07:41:23.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:23.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.398522+0000", 2024-09-18T07:41:23.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113799 2024-09-18T07:41:23.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-18T07:41:23.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.398631+0000", 2024-09-18T07:41:23.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000108866 2024-09-18T07:41:23.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:23.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.401163+0000", 2024-09-18T07:41:23.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025315989999999998 2024-09-18T07:41:23.091 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.091 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:23.091 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.091 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.15699.0:154204 9.b 9:dae841ab:::benchmark_data_smithi137_126072_object21704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:41:23.091 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:22.399070+0000", 2024-09-18T07:41:23.091 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.018895327999999999, 2024-09-18T07:41:23.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019099392999999999, 2024-09-18T07:41:23.092 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:23.092 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:23.092 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:41:23.092 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:23.092 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.15699", 2024-09-18T07:41:23.092 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:41:23.092 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 154204 2024-09-18T07:41:23.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.092 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:23.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:23.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.399070+0000", 2024-09-18T07:41:23.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:23.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.399070+0000", 2024-09-18T07:41:23.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:23.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.399071+0000", 2024-09-18T07:41:23.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.308e-06 2024-09-18T07:41:23.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:23.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.399076+0000", 2024-09-18T07:41:23.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5909999999999998e-06 2024-09-18T07:41:23.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:23.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.399086+0000", 2024-09-18T07:41:23.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0086e-05 2024-09-18T07:41:23.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:23.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.399093+0000", 2024-09-18T07:41:23.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8709999999999999e-06 2024-09-18T07:41:23.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:23.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.400634+0000", 2024-09-18T07:41:23.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001541167 2024-09-18T07:41:23.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:23.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.400708+0000", 2024-09-18T07:41:23.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4010000000000005e-05 2024-09-18T07:41:23.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:41:23.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.400813+0000", 2024-09-18T07:41:23.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105376 2024-09-18T07:41:23.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:23.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.403546+0000", 2024-09-18T07:41:23.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027326049999999999 2024-09-18T07:41:23.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:23.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.407439+0000", 2024-09-18T07:41:23.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038930940000000002 2024-09-18T07:41:23.096 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.096 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:23.096 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.097 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.15699.0:154208 9.5 9:a7dea059:::benchmark_data_smithi137_126072_object21708:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:41:23.097 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:22.405610+0000", 2024-09-18T07:41:23.097 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.012354626, 2024-09-18T07:41:23.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0125959, 2024-09-18T07:41:23.097 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:23.097 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:23.097 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:41:23.097 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:23.097 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.15699", 2024-09-18T07:41:23.097 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:41:23.097 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 154208 2024-09-18T07:41:23.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.097 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:23.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:23.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.405610+0000", 2024-09-18T07:41:23.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.098 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:23.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.405610+0000", 2024-09-18T07:41:23.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.098 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:23.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.405613+0000", 2024-09-18T07:41:23.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.182e-06 2024-09-18T07:41:23.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.099 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:23.099 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.405619+0000", 2024-09-18T07:41:23.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4989999999999999e-06 2024-09-18T07:41:23.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.099 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:23.099 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.405632+0000", 2024-09-18T07:41:23.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2747e-05 2024-09-18T07:41:23.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:23.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.405640+0000", 2024-09-18T07:41:23.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3729999999999993e-06 2024-09-18T07:41:23.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:23.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.405693+0000", 2024-09-18T07:41:23.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2358999999999998e-05 2024-09-18T07:41:23.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:23.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.405775+0000", 2024-09-18T07:41:23.101 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2847000000000001e-05 2024-09-18T07:41:23.101 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.101 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-18T07:41:23.101 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.405847+0000", 2024-09-18T07:41:23.101 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1731999999999997e-05 2024-09-18T07:41:23.101 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.101 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:23.101 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.101 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.101 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.15699.0:154201 9.b e330/289 ondisk, result = 0)", 2024-09-18T07:41:23.102 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:22.408035+0000", 2024-09-18T07:41:23.102 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0099303659999999995, 2024-09-18T07:41:23.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010204464, 2024-09-18T07:41:23.102 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:23.102 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:23.102 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:41:23.102 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:23.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:23.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408035+0000", 2024-09-18T07:41:23.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.103 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:23.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408035+0000", 2024-09-18T07:41:23.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.103 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:23.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408047+0000", 2024-09-18T07:41:23.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1917e-05 2024-09-18T07:41:23.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.103 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:23.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408052+0000", 2024-09-18T07:41:23.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3000000000000001e-06 2024-09-18T07:41:23.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:23.104 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408062+0000", 2024-09-18T07:41:23.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9879999999999999e-06 2024-09-18T07:41:23.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:23.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408068+0000", 2024-09-18T07:41:23.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4409999999999998e-06 2024-09-18T07:41:23.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.105 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:23.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408151+0000", 2024-09-18T07:41:23.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2222e-05 2024-09-18T07:41:23.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.105 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:23.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408159+0000", 2024-09-18T07:41:23.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6449999999999994e-06 2024-09-18T07:41:23.106 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.106 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:23.106 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.106 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.15699.0:154204 9.b e330/289)", 2024-09-18T07:41:23.106 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:22.408086+0000", 2024-09-18T07:41:23.106 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0098785739999999993, 2024-09-18T07:41:23.106 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010177331, 2024-09-18T07:41:23.106 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:23.106 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:23.106 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:41:23.106 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:23.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:23.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408086+0000", 2024-09-18T07:41:23.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:23.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408086+0000", 2024-09-18T07:41:23.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:23.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408087+0000", 2024-09-18T07:41:23.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7899999999999998e-07 2024-09-18T07:41:23.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:23.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408090+0000", 2024-09-18T07:41:23.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9629999999999998e-06 2024-09-18T07:41:23.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:23.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408096+0000", 2024-09-18T07:41:23.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4750000000000001e-06 2024-09-18T07:41:23.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:23.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408100+0000", 2024-09-18T07:41:23.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.233e-06 2024-09-18T07:41:23.109 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.109 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:23.109 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.109 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.15699.0:154196 9.5 e330/289 ondisk, result = 0)", 2024-09-18T07:41:23.109 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:22.408176+0000", 2024-09-18T07:41:23.109 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0097889440000000008, 2024-09-18T07:41:23.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010108160999999999, 2024-09-18T07:41:23.109 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:23.109 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:23.109 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:41:23.109 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:23.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:23.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408176+0000", 2024-09-18T07:41:23.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:23.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408176+0000", 2024-09-18T07:41:23.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:23.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408177+0000", 2024-09-18T07:41:23.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5699999999999998e-07 2024-09-18T07:41:23.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:23.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408179+0000", 2024-09-18T07:41:23.111 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6309999999999999e-06 2024-09-18T07:41:23.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:23.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408184+0000", 2024-09-18T07:41:23.111 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1240000000000004e-06 2024-09-18T07:41:23.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:23.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408188+0000", 2024-09-18T07:41:23.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8340000000000001e-06 2024-09-18T07:41:23.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.112 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:23.112 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408237+0000", 2024-09-18T07:41:23.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8424000000000003e-05 2024-09-18T07:41:23.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.112 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:23.112 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408242+0000", 2024-09-18T07:41:23.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4199999999999998e-06 2024-09-18T07:41:23.112 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.112 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:23.112 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.113 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.15699.0:154203 9.5 e330/289)", 2024-09-18T07:41:23.113 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:22.408194+0000", 2024-09-18T07:41:23.113 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0097708729999999994, 2024-09-18T07:41:23.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010113571999999999, 2024-09-18T07:41:23.113 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:23.113 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:23.113 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:41:23.113 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:23.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:23.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408194+0000", 2024-09-18T07:41:23.114 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:23.114 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408194+0000", 2024-09-18T07:41:23.114 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:23.114 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408195+0000", 2024-09-18T07:41:23.114 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.34e-07 2024-09-18T07:41:23.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:23.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408197+0000", 2024-09-18T07:41:23.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.137e-06 2024-09-18T07:41:23.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.115 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:23.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408201+0000", 2024-09-18T07:41:23.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3850000000000004e-06 2024-09-18T07:41:23.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.115 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:23.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.408206+0000", 2024-09-18T07:41:23.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4689999999999999e-06 2024-09-18T07:41:23.115 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.115 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:23.116 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.116 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.15699.0:154179 9.5 9:af6660c4:::benchmark_data_smithi137_126072_object21679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:41:23.116 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:22.378739+0000", 2024-09-18T07:41:23.116 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.039225721999999998, 2024-09-18T07:41:23.116 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.039593160000000002, 2024-09-18T07:41:23.116 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:23.116 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:23.116 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:41:23.116 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:23.116 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.15699", 2024-09-18T07:41:23.116 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:41:23.117 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 154179 2024-09-18T07:41:23.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.117 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:23.117 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.117 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:23.117 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.378739+0000", 2024-09-18T07:41:23.117 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.117 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.117 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:23.117 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.378739+0000", 2024-09-18T07:41:23.117 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:23.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.118 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:23.118 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.378742+0000", 2024-09-18T07:41:23.118 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3889999999999999e-06 2024-09-18T07:41:23.118 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.118 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:23.118 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.378749+0000", 2024-09-18T07:41:23.118 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3000000000000004e-06 2024-09-18T07:41:23.118 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.118 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:23.118 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.378766+0000", 2024-09-18T07:41:23.118 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7300999999999999e-05 2024-09-18T07:41:23.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.119 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.119 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:23.119 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.378775+0000", 2024-09-18T07:41:23.119 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0729999999999994e-06 2024-09-18T07:41:23.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.119 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.119 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:23.119 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.378838+0000", 2024-09-18T07:41:23.119 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3026000000000003e-05 2024-09-18T07:41:23.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.119 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.119 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:23.120 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.378976+0000", 2024-09-18T07:41:23.120 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137105 2024-09-18T07:41:23.120 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.120 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.120 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-18T07:41:23.120 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.379139+0000", 2024-09-18T07:41:23.120 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000163473 2024-09-18T07:41:23.120 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.120 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.120 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:23.120 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.382924+0000", 2024-09-18T07:41:23.120 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003784595 2024-09-18T07:41:23.120 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:23.120 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:23.121 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:23.121 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:22.387320+0000", 2024-09-18T07:41:23.121 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043964909999999998 2024-09-18T07:41:23.121 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.121 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:23.121 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.121 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:23.121 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:41:23.121 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 11 2024-09-18T07:41:23.121 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:41:23.122 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[29408]: 4.12 scrub starts 2024-09-18T07:41:23.122 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[29408]: 4.12 scrub ok 2024-09-18T07:41:23.122 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:41:23.122 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:41:23.122 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:41:23.122 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:41:23.122 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[29408]: pgmap v557: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-18T07:41:23.122 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[29408]: osdmap e330: 8 total, 8 up, 8 in 2024-09-18T07:41:23.123 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[29408]: 9.8 scrub ok 2024-09-18T07:41:23.123 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[29408]: 3.18 scrub starts 2024-09-18T07:41:23.123 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[39175]: 4.12 scrub starts 2024-09-18T07:41:23.123 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[39175]: 4.12 scrub ok 2024-09-18T07:41:23.123 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:41:23.123 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:41:23.123 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:41:23.123 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:41:23.123 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[39175]: pgmap v557: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-18T07:41:23.123 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[39175]: osdmap e330: 8 total, 8 up, 8 in 2024-09-18T07:41:23.123 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[39175]: 9.8 scrub ok 2024-09-18T07:41:23.124 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:22 smithi137 ceph-mon[39175]: 3.18 scrub starts 2024-09-18T07:41:23.159 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:23.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:23.205 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:23.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:23.306 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:23.348 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:23.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:23.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:23.450 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:23.481 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:23.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:23.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:23.583 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:23.625 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:23.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:23.727 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:23.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:41:23.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:23.760 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:23.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:23.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:23.903 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:23.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:23.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:23 smithi137 ceph-mon[29408]: 4.b scrub starts 2024-09-18T07:41:23.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:23 smithi137 ceph-mon[29408]: 4.b scrub ok 2024-09-18T07:41:23.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:23 smithi137 ceph-mon[29408]: 3.18 scrub ok 2024-09-18T07:41:23.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:23 smithi137 ceph-mon[29408]: 4.1f scrub starts 2024-09-18T07:41:23.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:23 smithi137 ceph-mon[29408]: 4.1f scrub ok 2024-09-18T07:41:23.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:23 smithi137 ceph-mon[29408]: 9.c scrub starts 2024-09-18T07:41:23.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:23 smithi137 ceph-mon[39175]: 4.b scrub starts 2024-09-18T07:41:23.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:23 smithi137 ceph-mon[39175]: 4.b scrub ok 2024-09-18T07:41:23.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:23 smithi137 ceph-mon[39175]: 3.18 scrub ok 2024-09-18T07:41:23.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:23 smithi137 ceph-mon[39175]: 4.1f scrub starts 2024-09-18T07:41:23.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:23 smithi137 ceph-mon[39175]: 4.1f scrub ok 2024-09-18T07:41:23.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:23 smithi137 ceph-mon[39175]: 9.c scrub starts 2024-09-18T07:41:23.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:23 smithi203 ceph-mon[32237]: 4.b scrub starts 2024-09-18T07:41:23.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:23 smithi203 ceph-mon[32237]: 4.b scrub ok 2024-09-18T07:41:23.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:23 smithi203 ceph-mon[32237]: 3.18 scrub ok 2024-09-18T07:41:23.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:23 smithi203 ceph-mon[32237]: 4.1f scrub starts 2024-09-18T07:41:23.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:23 smithi203 ceph-mon[32237]: 4.1f scrub ok 2024-09-18T07:41:23.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:23 smithi203 ceph-mon[32237]: 9.c scrub starts 2024-09-18T07:41:24.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:24.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T07:41:24.059 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:24.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:24.153 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:24.161 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:24.189 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:24.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:24.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:24.194 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:24.290 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:24.312 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:24.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:24.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:24.436 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:24.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:24.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:24.538 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:24.561 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:24.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:24.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:24.663 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:24.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:24.697 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:24.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:24.798 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:24.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:24.838 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:24.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:24.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:24.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:24 smithi203 ceph-mon[32237]: 9.7 scrub ok 2024-09-18T07:41:24.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:24 smithi203 ceph-mon[32237]: pgmap v559: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T07:41:24.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:24 smithi203 ceph-mon[32237]: 5.2 scrub starts 2024-09-18T07:41:24.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:24 smithi203 ceph-mon[32237]: 5.2 scrub ok 2024-09-18T07:41:24.974 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:24.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:24.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:25.076 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:25.115 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:25.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:25.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:25.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:24 smithi137 ceph-mon[29408]: 9.7 scrub ok 2024-09-18T07:41:25.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:24 smithi137 ceph-mon[29408]: pgmap v559: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T07:41:25.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:24 smithi137 ceph-mon[29408]: 5.2 scrub starts 2024-09-18T07:41:25.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:24 smithi137 ceph-mon[29408]: 5.2 scrub ok 2024-09-18T07:41:25.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:24 smithi137 ceph-mon[39175]: 9.7 scrub ok 2024-09-18T07:41:25.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:24 smithi137 ceph-mon[39175]: pgmap v559: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T07:41:25.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:24 smithi137 ceph-mon[39175]: 5.2 scrub starts 2024-09-18T07:41:25.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:24 smithi137 ceph-mon[39175]: 5.2 scrub ok 2024-09-18T07:41:25.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:25.252 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:25.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:25.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:25.354 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:25.386 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:25.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:25.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:25.488 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:25.510 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:25.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:25.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:25.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:25.640 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:25.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:25.741 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:25.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:25.772 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:25.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:25.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:25.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:25.916 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:25.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:26.017 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:26.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:26.053 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:26.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:26.154 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:26.182 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:26.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:26.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:26.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:26.322 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:26.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:26.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:25 smithi137 ceph-mon[29408]: 5.12 scrub starts 2024-09-18T07:41:26.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:25 smithi137 ceph-mon[29408]: 5.12 scrub ok 2024-09-18T07:41:26.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:25 smithi137 ceph-mon[29408]: osdmap e331: 8 total, 8 up, 8 in 2024-09-18T07:41:26.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:25 smithi137 ceph-mon[39175]: 5.12 scrub starts 2024-09-18T07:41:26.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:25 smithi137 ceph-mon[39175]: 5.12 scrub ok 2024-09-18T07:41:26.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:25 smithi137 ceph-mon[39175]: osdmap e331: 8 total, 8 up, 8 in 2024-09-18T07:41:26.423 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:26.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:26.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:26.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:26.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:25 smithi203 ceph-mon[32237]: 5.12 scrub starts 2024-09-18T07:41:26.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:25 smithi203 ceph-mon[32237]: 5.12 scrub ok 2024-09-18T07:41:26.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:25 smithi203 ceph-mon[32237]: osdmap e331: 8 total, 8 up, 8 in 2024-09-18T07:41:26.560 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:26.594 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:26.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:26.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:26.696 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:26.730 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:26.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:26.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:26.862 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:26.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:26.964 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:26.992 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:26.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:26.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:27.094 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:27.131 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:27.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:27.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:27.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:27.265 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:27.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:27.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:27.387 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:27.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:27.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:27.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:41:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:41:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:41:27.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:26 smithi137 ceph-mon[29408]: 9.2 scrub starts 2024-09-18T07:41:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:26 smithi137 ceph-mon[29408]: 9.c scrub ok 2024-09-18T07:41:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:26 smithi137 ceph-mon[29408]: pgmap v561: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T07:41:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:41:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:41:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:26 smithi137 ceph-mon[39175]: 9.2 scrub starts 2024-09-18T07:41:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:26 smithi137 ceph-mon[39175]: 9.c scrub ok 2024-09-18T07:41:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:26 smithi137 ceph-mon[39175]: pgmap v561: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T07:41:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:41:27.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:41:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:26 smithi203 ceph-mon[32237]: 9.2 scrub starts 2024-09-18T07:41:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:26 smithi203 ceph-mon[32237]: 9.c scrub ok 2024-09-18T07:41:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:26 smithi203 ceph-mon[32237]: pgmap v561: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-18T07:41:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:41:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:41:27.489 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:27.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:27.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:27.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:27.612 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:27.638 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:27.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:27.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:27.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:27.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:27.774 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:27.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:27.875 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:27.912 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:27.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:27.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:27.960 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:41:27 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:41:28.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:28.058 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:28.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:28.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:28.124 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:41:28.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:28.178 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:41:28.178 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-18T07:41:28.194 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:28.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:28.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:28.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:28.334 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:28.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:28.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:28.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:28 smithi137 ceph-mon[39175]: 5.1d scrub starts 2024-09-18T07:41:28.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:28 smithi137 ceph-mon[39175]: 5.1d scrub ok 2024-09-18T07:41:28.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:28 smithi137 ceph-mon[39175]: 2.1 scrub starts 2024-09-18T07:41:28.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:28 smithi137 ceph-mon[39175]: 2.1 scrub ok 2024-09-18T07:41:28.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:28 smithi137 ceph-mon[39175]: 4.6 scrub starts 2024-09-18T07:41:28.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:28 smithi137 ceph-mon[39175]: 4.6 scrub ok 2024-09-18T07:41:28.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:28 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2914719156' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:41:28.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:28 smithi137 ceph-mon[39175]: from='client.35786 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:41:28.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:28 smithi137 ceph-mon[29408]: 5.1d scrub starts 2024-09-18T07:41:28.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:28 smithi137 ceph-mon[29408]: 5.1d scrub ok 2024-09-18T07:41:28.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:28 smithi137 ceph-mon[29408]: 2.1 scrub starts 2024-09-18T07:41:28.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:28 smithi137 ceph-mon[29408]: 2.1 scrub ok 2024-09-18T07:41:28.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:28 smithi137 ceph-mon[29408]: 4.6 scrub starts 2024-09-18T07:41:28.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:28 smithi137 ceph-mon[29408]: 4.6 scrub ok 2024-09-18T07:41:28.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:28 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2914719156' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:41:28.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:28 smithi137 ceph-mon[29408]: from='client.35786 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:41:28.436 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:28 smithi203 ceph-mon[32237]: 5.1d scrub starts 2024-09-18T07:41:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:28 smithi203 ceph-mon[32237]: 5.1d scrub ok 2024-09-18T07:41:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:28 smithi203 ceph-mon[32237]: 2.1 scrub starts 2024-09-18T07:41:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:28 smithi203 ceph-mon[32237]: 2.1 scrub ok 2024-09-18T07:41:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:28 smithi203 ceph-mon[32237]: 4.6 scrub starts 2024-09-18T07:41:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:28 smithi203 ceph-mon[32237]: 4.6 scrub ok 2024-09-18T07:41:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:28 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2914719156' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:41:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:28 smithi203 ceph-mon[32237]: from='client.35786 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:41:28.466 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:28.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:28.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:28.606 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:28.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:28.709 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:28.735 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:28.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:28.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:28.837 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:28.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:28.876 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:28.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:28.977 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:29.009 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:29.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:29.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:29.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:29.142 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:29.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:29.243 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:29.276 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:29.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:29.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:29.282 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[29408]: 3.16 scrub starts 2024-09-18T07:41:29.282 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[29408]: 3.16 scrub ok 2024-09-18T07:41:29.282 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:29.282 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[29408]: 6.13 scrub starts 2024-09-18T07:41:29.282 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[29408]: 6.13 scrub ok 2024-09-18T07:41:29.282 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[29408]: pgmap v562: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-18T07:41:29.282 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[29408]: 3.f scrub starts 2024-09-18T07:41:29.283 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[29408]: 3.f scrub ok 2024-09-18T07:41:29.283 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[29408]: 5.e scrub starts 2024-09-18T07:41:29.283 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[29408]: 5.e scrub ok 2024-09-18T07:41:29.283 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:41:29.283 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[29408]: from='client.35786 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:41:29.283 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[29408]: osdmap e332: 8 total, 8 up, 8 in 2024-09-18T07:41:29.283 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[39175]: 3.16 scrub starts 2024-09-18T07:41:29.283 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[39175]: 3.16 scrub ok 2024-09-18T07:41:29.283 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:29.283 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[39175]: 6.13 scrub starts 2024-09-18T07:41:29.284 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[39175]: 6.13 scrub ok 2024-09-18T07:41:29.284 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[39175]: pgmap v562: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-18T07:41:29.284 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[39175]: 3.f scrub starts 2024-09-18T07:41:29.284 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[39175]: 3.f scrub ok 2024-09-18T07:41:29.284 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[39175]: 5.e scrub starts 2024-09-18T07:41:29.284 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[39175]: 5.e scrub ok 2024-09-18T07:41:29.284 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:41:29.284 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[39175]: from='client.35786 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:41:29.284 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:29 smithi137 ceph-mon[39175]: osdmap e332: 8 total, 8 up, 8 in 2024-09-18T07:41:29.362 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:29 smithi203 ceph-mon[32237]: 3.16 scrub starts 2024-09-18T07:41:29.362 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:29 smithi203 ceph-mon[32237]: 3.16 scrub ok 2024-09-18T07:41:29.362 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:29 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:29.362 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:29 smithi203 ceph-mon[32237]: 6.13 scrub starts 2024-09-18T07:41:29.362 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:29 smithi203 ceph-mon[32237]: 6.13 scrub ok 2024-09-18T07:41:29.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:29 smithi203 ceph-mon[32237]: pgmap v562: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-18T07:41:29.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:29 smithi203 ceph-mon[32237]: 3.f scrub starts 2024-09-18T07:41:29.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:29 smithi203 ceph-mon[32237]: 3.f scrub ok 2024-09-18T07:41:29.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:29 smithi203 ceph-mon[32237]: 5.e scrub starts 2024-09-18T07:41:29.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:29 smithi203 ceph-mon[32237]: 5.e scrub ok 2024-09-18T07:41:29.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:29 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:41:29.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:29 smithi203 ceph-mon[32237]: from='client.35786 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:41:29.363 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:29 smithi203 ceph-mon[32237]: osdmap e332: 8 total, 8 up, 8 in 2024-09-18T07:41:29.377 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:29.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:29.412 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:29.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:29.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:29.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:29.556 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:29.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:29.657 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:29.694 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:29.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:29.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:29.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:29.836 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:29.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:29.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:29.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:29.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:29.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:29.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:30.079 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:41:30.079 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:41:30.079 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:41:30.079 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:41:30.079 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:41:30.079 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:41:30.079 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:41:30.079 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:41:30.080 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:41:30.080 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:41:30.080 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:41:30.080 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:41:30.080 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:41:30.080 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:41:30.080 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:41:30.080 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:41:30.081 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:30 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4154537557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:30.081 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:30 smithi137 ceph-mon[29408]: osdmap e333: 8 total, 8 up, 8 in 2024-09-18T07:41:30.081 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:30 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4154537557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:30.081 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:30 smithi137 ceph-mon[39175]: osdmap e333: 8 total, 8 up, 8 in 2024-09-18T07:41:30.081 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:30.119 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:30.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:30.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:30.221 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:30.254 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:30.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:30.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:30.356 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:30.394 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:30.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:30.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:30 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4154537557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:30 smithi203 ceph-mon[32237]: osdmap e333: 8 total, 8 up, 8 in 2024-09-18T07:41:30.496 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:30.531 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:30.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:30.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:30.633 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:30.668 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:30.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:30.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:30.738 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:30.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:30.821 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:30.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:30.826 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:30.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:30.927 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:30.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:30.963 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:30.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:31.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:31.104 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:31.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:31.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:31.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:31.245 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:31.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:31.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:31.287 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:31 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:41:31.287 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:31 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:41:31.287 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:31 smithi137 ceph-mon[29408]: pgmap v564: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 641 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-18T07:41:31.288 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:31 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:41:31.288 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:31 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:41:31.288 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:31 smithi137 ceph-mon[39175]: pgmap v564: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 641 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-18T07:41:31.347 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:31.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:31.392 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:31.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:31 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:41:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:31 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:41:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:31 smithi203 ceph-mon[32237]: pgmap v564: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 641 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-18T07:41:31.493 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:31.501 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:41:31.501 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:41:31.526 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:31.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:31.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:31.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:31.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:31.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:31.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:31.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:31.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:31.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:31.879 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:41:31.908 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:31.937 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:31.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:32.039 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:32.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:32.069 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:32.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:32.170 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:32.200 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:32.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:32.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:32.302 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:32.343 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:32.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:32.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:32.445 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:32.483 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:32.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:32.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:32.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:32.622 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:32.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:32.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:32.724 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:32.759 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:32.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:32.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:32.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:41:32.862 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:32.896 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:32.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:32.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:32.998 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:33.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:33.034 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:33.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:33.046 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.0 to 1 (10000) 2024-09-18T07:41:33.135 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:33.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:33 smithi137 ceph-mon[29408]: pgmap v566: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T07:41:33.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:33 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:41:33.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:33 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:41:33.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:33 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2655616864' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:41:33.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:33 smithi137 ceph-mon[29408]: from='client.25876 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:41:33.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:33 smithi137 ceph-mon[39175]: pgmap v566: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T07:41:33.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:33 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:41:33.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:33 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:41:33.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:33 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2655616864' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:41:33.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:33 smithi137 ceph-mon[39175]: from='client.25876 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:41:33.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:33.171 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:33.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:33.272 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:33.309 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:33.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:33.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:33.410 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:33.446 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:33.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:33.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:33.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:33 smithi203 ceph-mon[32237]: pgmap v566: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T07:41:33.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:33 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:41:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:33 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:41:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:33 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2655616864' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:41:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:33 smithi203 ceph-mon[32237]: from='client.25876 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:41:33.548 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:33.582 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:33.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:33.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:33.684 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:33.718 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:33.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:33.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:33.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:33.859 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:33.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:33.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:33.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:33.982 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:33.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:33.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:34.084 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:34.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:34.126 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:34.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:34.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:34 smithi137 ceph-mon[39175]: from='client.25876 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:41:34.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:34 smithi137 ceph-mon[39175]: osdmap e334: 8 total, 8 up, 8 in 2024-09-18T07:41:34.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:34 smithi137 ceph-mon[29408]: from='client.25876 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:41:34.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:34 smithi137 ceph-mon[29408]: osdmap e334: 8 total, 8 up, 8 in 2024-09-18T07:41:34.227 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:34.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:34.266 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:34.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:34.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:34.403 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:34.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:34.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:34.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:34 smithi203 ceph-mon[32237]: from='client.25876 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:41:34.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:34 smithi203 ceph-mon[32237]: osdmap e334: 8 total, 8 up, 8 in 2024-09-18T07:41:34.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:34.543 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:34.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:34.646 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:34.682 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:34.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:34.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:34.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:34.822 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:34.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:34.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:34.924 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:34.957 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:34.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:34.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:35.059 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:35.083 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:35.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:35.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:35.095 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:35 smithi137 ceph-mon[29408]: pgmap v568: 158 pgs: 158 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T07:41:35.096 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:35 smithi137 ceph-mon[39175]: pgmap v568: 158 pgs: 158 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T07:41:35.186 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:35.207 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:35.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:35.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:35.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:35.330 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:35.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:35.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:35.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:35 smithi137 ceph-mon[39175]: osdmap e335: 8 total, 8 up, 8 in 2024-09-18T07:41:35.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:35 smithi137 ceph-mon[29408]: osdmap e335: 8 total, 8 up, 8 in 2024-09-18T07:41:35.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:35.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:35 smithi203 ceph-mon[32237]: pgmap v568: 158 pgs: 158 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T07:41:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:35 smithi203 ceph-mon[32237]: osdmap e335: 8 total, 8 up, 8 in 2024-09-18T07:41:35.468 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:35.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:35.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:35.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:35.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:35.606 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:35.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:35.707 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:35.743 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:35.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:35.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:35.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:35.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:35.878 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:35.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:35.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T07:41:35.979 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:36.012 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:36.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:36.114 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:36.148 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:36.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:36.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:36.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-18T07:41:36.283 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:36.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:36.349 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:36.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:36.450 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:36.478 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:36.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:36.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:36.580 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:36.634 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:36.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:36.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:36.779 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:36.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:36.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:36.881 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:36.910 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:36.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:36.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:36.978 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:36.978 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:37.012 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:37.045 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:41:37.045 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:41:37.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:37.059 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:37.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:37.073 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:37.079 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:41:37.079 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:41:37.117 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:41:37.117 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:41:37.152 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:41:37.152 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:41:37.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:37.186 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:41:37.186 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:41:37.189 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:37.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:37.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:37.217 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:41:37.217 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:41:37.217 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:41:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:41:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:41:37.218 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:37 smithi137 ceph-mon[29408]: pgmap v570: 158 pgs: 158 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T07:41:37.218 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:37 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/798695676' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:37.218 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:37 smithi137 ceph-mon[39175]: pgmap v570: 158 pgs: 158 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T07:41:37.218 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:37 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/798695676' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:37.247 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:41:37.248 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:41:37.279 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:41:37.280 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:41:37.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:37.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:37.331 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:37.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:37.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:37.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:37 smithi203 ceph-mon[32237]: pgmap v570: 158 pgs: 158 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-18T07:41:37.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:37 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/798695676' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:37.464 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:37.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:37.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:37.566 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:37.598 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:37.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:37.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:37.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:37.735 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:37.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:37.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:37.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:37.877 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:37.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:37.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:37.960 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:41:37 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:41:37.979 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:38.008 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:38.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:38.108 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:38.141 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:38.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:38.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:38.243 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:38.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:38.264 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:38.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:38.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:38.385 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:38.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:38.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:38.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:38.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:38.523 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:38.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:38.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:38.646 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:38.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:38.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:38.769 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:38.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:38.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:38.871 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:38.891 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:38.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:38.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:38.993 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:39.025 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:39.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:39.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:39.069 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:41:39.128 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:39.162 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:39.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:39.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:39.264 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:39.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:39.300 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:39.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:39.401 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:39.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:39 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:39.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:39 smithi137 ceph-mon[39175]: pgmap v571: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-18T07:41:39.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:39 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3285610413' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:39.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:39 smithi137 ceph-mon[39175]: from='client.25888 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:39 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:39 smithi137 ceph-mon[29408]: pgmap v571: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-18T07:41:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:39 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3285610413' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:39 smithi137 ceph-mon[29408]: from='client.25888 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:39.438 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:39 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:39.438 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:39 smithi203 ceph-mon[32237]: pgmap v571: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-09-18T07:41:39.438 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:39 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3285610413' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:39.439 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:39 smithi203 ceph-mon[32237]: from='client.25888 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:39.445 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:39.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:39.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:39.599 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:39.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:39.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:39.701 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:39.760 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:39.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:39.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:39.794 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:41:39.794 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:41:39.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:41:39.795 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:41:39.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.795 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134984 7.16 7:6e4eb25e:::benchmark_data_smithi137_74180_object30568:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:41:39.795 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964075+0000", 2024-09-18T07:41:39.795 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 442.75427924299998, 2024-09-18T07:41:39.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.253286275, 2024-09-18T07:41:39.795 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.796 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.796 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:41:39.796 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.796 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.796 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.796 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134984 2024-09-18T07:41:39.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.796 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964075+0000", 2024-09-18T07:41:39.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964075+0000", 2024-09-18T07:41:39.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964077+0000", 2024-09-18T07:41:39.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8870000000000001e-06 2024-09-18T07:41:39.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964083+0000", 2024-09-18T07:41:39.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4069999999999998e-06 2024-09-18T07:41:39.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964098+0000", 2024-09-18T07:41:39.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4119e-05 2024-09-18T07:41:39.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964108+0000", 2024-09-18T07:41:39.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0665999999999999e-05 2024-09-18T07:41:39.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964166+0000", 2024-09-18T07:41:39.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7819000000000003e-05 2024-09-18T07:41:39.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:41:39.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964174+0000", 2024-09-18T07:41:39.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5499999999999997e-06 2024-09-18T07:41:39.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.968251+0000", 2024-09-18T07:41:39.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004077448 2024-09-18T07:41:39.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.968261+0000", 2024-09-18T07:41:39.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0438000000000001e-05 2024-09-18T07:41:39.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.217288+0000", 2024-09-18T07:41:39.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.249026295 2024-09-18T07:41:39.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.217361+0000", 2024-09-18T07:41:39.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3646000000000006e-05 2024-09-18T07:41:39.801 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.801 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.802 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.802 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134991 7.19 7:9bc94497:::benchmark_data_smithi137_74180_object30575:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:41:39.802 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964132+0000", 2024-09-18T07:41:39.802 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 442.754222148, 2024-09-18T07:41:39.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2456100560000001, 2024-09-18T07:41:39.802 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.802 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.802 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:41:39.802 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.802 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.803 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.803 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134991 2024-09-18T07:41:39.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.803 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964132+0000", 2024-09-18T07:41:39.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964132+0000", 2024-09-18T07:41:39.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964133+0000", 2024-09-18T07:41:39.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4000000000000001e-07 2024-09-18T07:41:39.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964136+0000", 2024-09-18T07:41:39.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8009999999999999e-06 2024-09-18T07:41:39.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964141+0000", 2024-09-18T07:41:39.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.75e-06 2024-09-18T07:41:39.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964147+0000", 2024-09-18T07:41:39.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5439999999999998e-06 2024-09-18T07:41:39.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964319+0000", 2024-09-18T07:41:39.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017216000000000001 2024-09-18T07:41:39.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:41:39.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964324+0000", 2024-09-18T07:41:39.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.31e-06 2024-09-18T07:41:39.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.970873+0000", 2024-09-18T07:41:39.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065484250000000001 2024-09-18T07:41:39.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.970881+0000", 2024-09-18T07:41:39.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0600000000000008e-06 2024-09-18T07:41:39.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.201197+0000", 2024-09-18T07:41:39.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2303157170000001 2024-09-18T07:41:39.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-09-18T07:41:39.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.201321+0000", 2024-09-18T07:41:39.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012415000000000001 2024-09-18T07:41:39.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.209647+0000", 2024-09-18T07:41:39.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.008325931 2024-09-18T07:41:39.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.209742+0000", 2024-09-18T07:41:39.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5468000000000001e-05 2024-09-18T07:41:39.811 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.811 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.811 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.811 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135011 7.14 7:2c5d899c:::benchmark_data_smithi137_74180_object30595:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:41:39.811 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964161+0000", 2024-09-18T07:41:39.811 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 442.75419287099999, 2024-09-18T07:41:39.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2230384009999999, 2024-09-18T07:41:39.811 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.811 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.811 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.811 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.811 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.812 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.812 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135011 2024-09-18T07:41:39.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.812 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964161+0000", 2024-09-18T07:41:39.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964161+0000", 2024-09-18T07:41:39.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964162+0000", 2024-09-18T07:41:39.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2300000000000001e-07 2024-09-18T07:41:39.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964164+0000", 2024-09-18T07:41:39.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3690000000000001e-06 2024-09-18T07:41:39.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964169+0000", 2024-09-18T07:41:39.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.028e-06 2024-09-18T07:41:39.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964174+0000", 2024-09-18T07:41:39.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7809999999999996e-06 2024-09-18T07:41:39.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964271+0000", 2024-09-18T07:41:39.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6997000000000007e-05 2024-09-18T07:41:39.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:41:39.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964277+0000", 2024-09-18T07:41:39.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4369999999999998e-06 2024-09-18T07:41:39.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.969538+0000", 2024-09-18T07:41:39.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052614929999999999 2024-09-18T07:41:39.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.969546+0000", 2024-09-18T07:41:39.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9540000000000004e-06 2024-09-18T07:41:39.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.178869+0000", 2024-09-18T07:41:39.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2093228570000001 2024-09-18T07:41:39.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.179001+0000", 2024-09-18T07:41:39.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013228400000000001 2024-09-18T07:41:39.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2,3,4,5", 2024-09-18T07:41:39.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.180534+0000", 2024-09-18T07:41:39.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0015324399999999999 2024-09-18T07:41:39.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.183783+0000", 2024-09-18T07:41:39.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032497559999999999 2024-09-18T07:41:39.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184155+0000", 2024-09-18T07:41:39.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000371388 2024-09-18T07:41:39.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184621+0000", 2024-09-18T07:41:39.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00046647700000000003 2024-09-18T07:41:39.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185211+0000", 2024-09-18T07:41:39.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00058937699999999998 2024-09-18T07:41:39.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185835+0000", 2024-09-18T07:41:39.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000624269 2024-09-18T07:41:39.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186879+0000", 2024-09-18T07:41:39.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010440429999999999 2024-09-18T07:41:39.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186966+0000", 2024-09-18T07:41:39.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7031000000000001e-05 2024-09-18T07:41:39.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.187200+0000", 2024-09-18T07:41:39.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000233797 2024-09-18T07:41:39.821 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.821 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.821 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.821 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135015 7.17 7:e8fec392:::benchmark_data_smithi137_74180_object30599:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:41:39.821 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964189+0000", 2024-09-18T07:41:39.821 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 442.75416484099998, 2024-09-18T07:41:39.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.277466437, 2024-09-18T07:41:39.821 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.821 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.821 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.821 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.822 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.822 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.822 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135015 2024-09-18T07:41:39.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.822 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964189+0000", 2024-09-18T07:41:39.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964189+0000", 2024-09-18T07:41:39.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964190+0000", 2024-09-18T07:41:39.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7199999999999998e-07 2024-09-18T07:41:39.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964193+0000", 2024-09-18T07:41:39.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5670000000000002e-06 2024-09-18T07:41:39.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964199+0000", 2024-09-18T07:41:39.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8200000000000002e-06 2024-09-18T07:41:39.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964203+0000", 2024-09-18T07:41:39.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9189999999999998e-06 2024-09-18T07:41:39.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964356+0000", 2024-09-18T07:41:39.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015237200000000001 2024-09-18T07:41:39.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:41:39.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964361+0000", 2024-09-18T07:41:39.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1499999999999998e-06 2024-09-18T07:41:39.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971479+0000", 2024-09-18T07:41:39.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.007117511 2024-09-18T07:41:39.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971487+0000", 2024-09-18T07:41:39.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2800000000000003e-06 2024-09-18T07:41:39.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.224196+0000", 2024-09-18T07:41:39.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2527092280000001 2024-09-18T07:41:39.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.224324+0000", 2024-09-18T07:41:39.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000127582 2024-09-18T07:41:39.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2,3,5,6", 2024-09-18T07:41:39.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.224588+0000", 2024-09-18T07:41:39.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00026444499999999998 2024-09-18T07:41:39.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.827 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.230126+0000", 2024-09-18T07:41:39.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055381579999999996 2024-09-18T07:41:39.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.827 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.230140+0000", 2024-09-18T07:41:39.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4e-05 2024-09-18T07:41:39.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.230258+0000", 2024-09-18T07:41:39.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118002 2024-09-18T07:41:39.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.232997+0000", 2024-09-18T07:41:39.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027387829999999998 2024-09-18T07:41:39.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.233523+0000", 2024-09-18T07:41:39.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052651200000000003 2024-09-18T07:41:39.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.241477+0000", 2024-09-18T07:41:39.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079535379999999996 2024-09-18T07:41:39.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.241534+0000", 2024-09-18T07:41:39.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6906e-05 2024-09-18T07:41:39.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.241656+0000", 2024-09-18T07:41:39.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012199199999999999 2024-09-18T07:41:39.830 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.830 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.830 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.830 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135019 7.17 7:ec2155c5:::benchmark_data_smithi137_74180_object30603:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:41:39.830 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964215+0000", 2024-09-18T07:41:39.831 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 442.75413944899998, 2024-09-18T07:41:39.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2831853689999999, 2024-09-18T07:41:39.831 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.831 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.831 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.831 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.831 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.831 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.831 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135019 2024-09-18T07:41:39.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.831 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964215+0000", 2024-09-18T07:41:39.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964215+0000", 2024-09-18T07:41:39.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964216+0000", 2024-09-18T07:41:39.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3499999999999995e-07 2024-09-18T07:41:39.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964218+0000", 2024-09-18T07:41:39.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5790000000000002e-06 2024-09-18T07:41:39.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964224+0000", 2024-09-18T07:41:39.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3689999999999998e-06 2024-09-18T07:41:39.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964228+0000", 2024-09-18T07:41:39.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3560000000000003e-06 2024-09-18T07:41:39.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964383+0000", 2024-09-18T07:41:39.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000155217 2024-09-18T07:41:39.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-18T07:41:39.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964388+0000", 2024-09-18T07:41:39.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8550000000000001e-06 2024-09-18T07:41:39.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971509+0000", 2024-09-18T07:41:39.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0071211800000000004 2024-09-18T07:41:39.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971515+0000", 2024-09-18T07:41:39.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.224e-06 2024-09-18T07:41:39.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.226033+0000", 2024-09-18T07:41:39.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2545179440000001 2024-09-18T07:41:39.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.226140+0000", 2024-09-18T07:41:39.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010681200000000001 2024-09-18T07:41:39.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2,3,5,6", 2024-09-18T07:41:39.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.226373+0000", 2024-09-18T07:41:39.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023260699999999999 2024-09-18T07:41:39.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.230206+0000", 2024-09-18T07:41:39.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038327439999999999 2024-09-18T07:41:39.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.233570+0000", 2024-09-18T07:41:39.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033643129999999999 2024-09-18T07:41:39.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.233605+0000", 2024-09-18T07:41:39.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5290999999999998e-05 2024-09-18T07:41:39.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.241382+0000", 2024-09-18T07:41:39.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0077770239999999996 2024-09-18T07:41:39.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.241433+0000", 2024-09-18T07:41:39.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1044000000000001e-05 2024-09-18T07:41:39.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.247229+0000", 2024-09-18T07:41:39.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057953609999999997 2024-09-18T07:41:39.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.247282+0000", 2024-09-18T07:41:39.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3084999999999997e-05 2024-09-18T07:41:39.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.247400+0000", 2024-09-18T07:41:39.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011862900000000001 2024-09-18T07:41:39.839 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.839 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.839 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.839 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162316 7.f 7:f95a8250:::benchmark_data_smithi137_74180_object57900:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:41:39.839 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.409395+0000", 2024-09-18T07:41:39.839 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 398.30895921699999, 2024-09-18T07:41:39.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0987874790000001, 2024-09-18T07:41:39.840 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.840 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.840 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.840 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.840 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.840 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.840 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162316 2024-09-18T07:41:39.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.840 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409395+0000", 2024-09-18T07:41:39.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409395+0000", 2024-09-18T07:41:39.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409397+0000", 2024-09-18T07:41:39.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3429999999999998e-06 2024-09-18T07:41:39.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409404+0000", 2024-09-18T07:41:39.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2600000000000002e-06 2024-09-18T07:41:39.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409418+0000", 2024-09-18T07:41:39.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3983e-05 2024-09-18T07:41:39.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409427+0000", 2024-09-18T07:41:39.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9269999999999994e-06 2024-09-18T07:41:39.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409490+0000", 2024-09-18T07:41:39.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2868999999999997e-05 2024-09-18T07:41:39.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409500+0000", 2024-09-18T07:41:39.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0621e-05 2024-09-18T07:41:39.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.497693+0000", 2024-09-18T07:41:39.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.088192405 2024-09-18T07:41:39.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.497816+0000", 2024-09-18T07:41:39.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012321500000000001 2024-09-18T07:41:39.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:41:39.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.497964+0000", 2024-09-18T07:41:39.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000148169 2024-09-18T07:41:39.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.500777+0000", 2024-09-18T07:41:39.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028126750000000002 2024-09-18T07:41:39.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.500835+0000", 2024-09-18T07:41:39.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8727000000000002e-05 2024-09-18T07:41:39.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.500878+0000", 2024-09-18T07:41:39.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2682000000000002e-05 2024-09-18T07:41:39.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507887+0000", 2024-09-18T07:41:39.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0070087789999999997 2024-09-18T07:41:39.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508067+0000", 2024-09-18T07:41:39.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017973300000000001 2024-09-18T07:41:39.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508183+0000", 2024-09-18T07:41:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116091 2024-09-18T07:41:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162322 7.f 7:f41da5a4:::benchmark_data_smithi137_74180_object57906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:41:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.420348+0000", 2024-09-18T07:41:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 398.29800592399999, 2024-09-18T07:41:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0968331140000001, 2024-09-18T07:41:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162322 2024-09-18T07:41:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420348+0000", 2024-09-18T07:41:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420348+0000", 2024-09-18T07:41:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420350+0000", 2024-09-18T07:41:39.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8670000000000001e-06 2024-09-18T07:41:39.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420356+0000", 2024-09-18T07:41:39.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0560000000000001e-06 2024-09-18T07:41:39.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420369+0000", 2024-09-18T07:41:39.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2507000000000001e-05 2024-09-18T07:41:39.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420376+0000", 2024-09-18T07:41:39.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.17e-06 2024-09-18T07:41:39.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420444+0000", 2024-09-18T07:41:39.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7801999999999999e-05 2024-09-18T07:41:39.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420454+0000", 2024-09-18T07:41:39.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0258000000000001e-05 2024-09-18T07:41:39.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.501094+0000", 2024-09-18T07:41:39.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0806402799999999 2024-09-18T07:41:39.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.501227+0000", 2024-09-18T07:41:39.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013271600000000001 2024-09-18T07:41:39.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:41:39.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.501467+0000", 2024-09-18T07:41:39.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024032599999999999 2024-09-18T07:41:39.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507729+0000", 2024-09-18T07:41:39.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0062618220000000002 2024-09-18T07:41:39.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507768+0000", 2024-09-18T07:41:39.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8668000000000003e-05 2024-09-18T07:41:39.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.516751+0000", 2024-09-18T07:41:39.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0089832390000000005 2024-09-18T07:41:39.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.516888+0000", 2024-09-18T07:41:39.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137372 2024-09-18T07:41:39.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517069+0000", 2024-09-18T07:41:39.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000180574 2024-09-18T07:41:39.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517182+0000", 2024-09-18T07:41:39.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112457 2024-09-18T07:41:39.854 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.854 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.854 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.854 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162323 7.f 7:fe31c7fa:::benchmark_data_smithi137_74180_object57907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:41:39.854 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.420396+0000", 2024-09-18T07:41:39.854 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 398.29795845299998, 2024-09-18T07:41:39.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.097250378, 2024-09-18T07:41:39.854 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.854 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.854 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.855 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.855 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.855 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.855 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162323 2024-09-18T07:41:39.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.855 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420396+0000", 2024-09-18T07:41:39.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420396+0000", 2024-09-18T07:41:39.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420396+0000", 2024-09-18T07:41:39.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9400000000000005e-07 2024-09-18T07:41:39.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420399+0000", 2024-09-18T07:41:39.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6879999999999999e-06 2024-09-18T07:41:39.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420404+0000", 2024-09-18T07:41:39.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2680000000000002e-06 2024-09-18T07:41:39.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420411+0000", 2024-09-18T07:41:39.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2519999999999996e-06 2024-09-18T07:41:39.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420540+0000", 2024-09-18T07:41:39.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012886899999999999 2024-09-18T07:41:39.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420547+0000", 2024-09-18T07:41:39.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9019999999999999e-06 2024-09-18T07:41:39.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.504020+0000", 2024-09-18T07:41:39.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0834720959999999 2024-09-18T07:41:39.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.504135+0000", 2024-09-18T07:41:39.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115697 2024-09-18T07:41:39.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:41:39.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.504251+0000", 2024-09-18T07:41:39.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116041 2024-09-18T07:41:39.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507805+0000", 2024-09-18T07:41:39.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035537749999999999 2024-09-18T07:41:39.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507870+0000", 2024-09-18T07:41:39.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4558000000000001e-05 2024-09-18T07:41:39.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.516761+0000", 2024-09-18T07:41:39.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0088909690000000003 2024-09-18T07:41:39.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517414+0000", 2024-09-18T07:41:39.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00065372100000000001 2024-09-18T07:41:39.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517543+0000", 2024-09-18T07:41:39.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012907199999999999 2024-09-18T07:41:39.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517646+0000", 2024-09-18T07:41:39.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102876 2024-09-18T07:41:39.861 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.861 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.861 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.862 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162327 7.f 7:fac5d155:::benchmark_data_smithi137_74180_object57911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:41:39.862 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.420507+0000", 2024-09-18T07:41:39.862 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 398.297847445, 2024-09-18T07:41:39.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.097389145, 2024-09-18T07:41:39.862 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.862 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.862 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.862 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.862 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.862 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.862 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162327 2024-09-18T07:41:39.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420507+0000", 2024-09-18T07:41:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420507+0000", 2024-09-18T07:41:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420508+0000", 2024-09-18T07:41:39.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7300000000000005e-07 2024-09-18T07:41:39.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420511+0000", 2024-09-18T07:41:39.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4089999999999999e-06 2024-09-18T07:41:39.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420518+0000", 2024-09-18T07:41:39.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1010000000000003e-06 2024-09-18T07:41:39.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420523+0000", 2024-09-18T07:41:39.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2089999999999996e-06 2024-09-18T07:41:39.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420570+0000", 2024-09-18T07:41:39.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6638000000000003e-05 2024-09-18T07:41:39.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420577+0000", 2024-09-18T07:41:39.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2679999999999997e-06 2024-09-18T07:41:39.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508213+0000", 2024-09-18T07:41:39.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0876354130000001 2024-09-18T07:41:39.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508312+0000", 2024-09-18T07:41:39.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9124999999999997e-05 2024-09-18T07:41:39.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:41:39.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508859+0000", 2024-09-18T07:41:39.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00054700599999999995 2024-09-18T07:41:39.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.516816+0000", 2024-09-18T07:41:39.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079569410000000004 2024-09-18T07:41:39.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517264+0000", 2024-09-18T07:41:39.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044864999999999998 2024-09-18T07:41:39.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517309+0000", 2024-09-18T07:41:39.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5033000000000001e-05 2024-09-18T07:41:39.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517755+0000", 2024-09-18T07:41:39.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044556899999999999 2024-09-18T07:41:39.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517803+0000", 2024-09-18T07:41:39.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7584999999999999e-05 2024-09-18T07:41:39.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517896+0000", 2024-09-18T07:41:39.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3424999999999994e-05 2024-09-18T07:41:39.869 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.869 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.869 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.870 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162336 7.f 7:f2e86194:::benchmark_data_smithi137_74180_object57920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:41:39.870 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.425606+0000", 2024-09-18T07:41:39.870 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 398.29274823899999, 2024-09-18T07:41:39.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0924705029999999, 2024-09-18T07:41:39.870 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.870 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.870 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.870 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.870 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.870 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.870 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162336 2024-09-18T07:41:39.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.871 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425606+0000", 2024-09-18T07:41:39.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425606+0000", 2024-09-18T07:41:39.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425608+0000", 2024-09-18T07:41:39.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2259999999999999e-06 2024-09-18T07:41:39.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425614+0000", 2024-09-18T07:41:39.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0549999999999996e-06 2024-09-18T07:41:39.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425626+0000", 2024-09-18T07:41:39.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1600000000000001e-05 2024-09-18T07:41:39.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425635+0000", 2024-09-18T07:41:39.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0440000000000001e-06 2024-09-18T07:41:39.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425680+0000", 2024-09-18T07:41:39.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5015999999999997e-05 2024-09-18T07:41:39.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425689+0000", 2024-09-18T07:41:39.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3419999999999997e-06 2024-09-18T07:41:39.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.513184+0000", 2024-09-18T07:41:39.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0874942320000001 2024-09-18T07:41:39.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.513298+0000", 2024-09-18T07:41:39.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114333 2024-09-18T07:41:39.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:41:39.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.513425+0000", 2024-09-18T07:41:39.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126711 2024-09-18T07:41:39.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517321+0000", 2024-09-18T07:41:39.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038961590000000002 2024-09-18T07:41:39.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517374+0000", 2024-09-18T07:41:39.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3003000000000001e-05 2024-09-18T07:41:39.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517718+0000", 2024-09-18T07:41:39.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034425100000000001 2024-09-18T07:41:39.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517935+0000", 2024-09-18T07:41:39.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021721899999999999 2024-09-18T07:41:39.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517986+0000", 2024-09-18T07:41:39.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0627999999999997e-05 2024-09-18T07:41:39.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.518077+0000", 2024-09-18T07:41:39.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0684000000000001e-05 2024-09-18T07:41:39.877 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.877 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.877 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.877 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162345 7.f 7:f017fa1c:::benchmark_data_smithi137_74180_object57929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:41:39.878 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.431398+0000", 2024-09-18T07:41:39.878 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 398.28695675199998, 2024-09-18T07:41:39.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0972635669999999, 2024-09-18T07:41:39.878 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.878 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.878 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.878 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.878 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.878 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.878 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162345 2024-09-18T07:41:39.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.878 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431398+0000", 2024-09-18T07:41:39.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431398+0000", 2024-09-18T07:41:39.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431399+0000", 2024-09-18T07:41:39.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.361e-06 2024-09-18T07:41:39.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431403+0000", 2024-09-18T07:41:39.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1520000000000002e-06 2024-09-18T07:41:39.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431414+0000", 2024-09-18T07:41:39.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1239e-05 2024-09-18T07:41:39.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431421+0000", 2024-09-18T07:41:39.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.99e-06 2024-09-18T07:41:39.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431537+0000", 2024-09-18T07:41:39.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115278 2024-09-18T07:41:39.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431547+0000", 2024-09-18T07:41:39.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0543999999999999e-05 2024-09-18T07:41:39.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.518110+0000", 2024-09-18T07:41:39.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.086562942 2024-09-18T07:41:39.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.518220+0000", 2024-09-18T07:41:39.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010951 2024-09-18T07:41:39.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:41:39.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.518334+0000", 2024-09-18T07:41:39.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114057 2024-09-18T07:41:39.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.527942+0000", 2024-09-18T07:41:39.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0096087029999999997 2024-09-18T07:41:39.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528087+0000", 2024-09-18T07:41:39.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000144871 2024-09-18T07:41:39.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528103+0000", 2024-09-18T07:41:39.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5531000000000001e-05 2024-09-18T07:41:39.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528148+0000", 2024-09-18T07:41:39.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5667000000000002e-05 2024-09-18T07:41:39.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528542+0000", 2024-09-18T07:41:39.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00039349200000000001 2024-09-18T07:41:39.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528661+0000", 2024-09-18T07:41:39.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011923 2024-09-18T07:41:39.884 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.885 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.885 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.885 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202539 7.f 7:fdd02dfa:::benchmark_data_smithi137_74180_object98123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:41:39.885 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.829809+0000", 2024-09-18T07:41:39.885 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 361.88854508200001, 2024-09-18T07:41:39.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0954651049999999, 2024-09-18T07:41:39.885 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.885 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.885 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.885 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.885 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.885 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.886 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202539 2024-09-18T07:41:39.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.886 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829809+0000", 2024-09-18T07:41:39.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829809+0000", 2024-09-18T07:41:39.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829811+0000", 2024-09-18T07:41:39.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9360000000000002e-06 2024-09-18T07:41:39.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829816+0000", 2024-09-18T07:41:39.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6979999999999997e-06 2024-09-18T07:41:39.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829824+0000", 2024-09-18T07:41:39.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6799999999999993e-06 2024-09-18T07:41:39.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829829+0000", 2024-09-18T07:41:39.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9159999999999998e-06 2024-09-18T07:41:39.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829890+0000", 2024-09-18T07:41:39.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0643000000000002e-05 2024-09-18T07:41:39.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829898+0000", 2024-09-18T07:41:39.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.576e-06 2024-09-18T07:41:39.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.917838+0000", 2024-09-18T07:41:39.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.087940012 2024-09-18T07:41:39.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.917970+0000", 2024-09-18T07:41:39.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013214499999999999 2024-09-18T07:41:39.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:41:39.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.918095+0000", 2024-09-18T07:41:39.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012521 2024-09-18T07:41:39.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.921001+0000", 2024-09-18T07:41:39.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029057789999999998 2024-09-18T07:41:39.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.921054+0000", 2024-09-18T07:41:39.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3038999999999998e-05 2024-09-18T07:41:39.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925195+0000", 2024-09-18T07:41:39.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004140632 2024-09-18T07:41:39.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925229+0000", 2024-09-18T07:41:39.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4783000000000002e-05 2024-09-18T07:41:39.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925274+0000", 2024-09-18T07:41:39.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5055999999999997e-05 2024-09-18T07:41:39.892 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.892 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.892 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.892 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202541 7.f 7:f489b710:::benchmark_data_smithi137_74180_object98125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:41:39.892 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.829844+0000", 2024-09-18T07:41:39.892 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 361.88850990100002, 2024-09-18T07:41:39.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0983412960000001, 2024-09-18T07:41:39.892 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.893 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.893 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.893 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.893 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.893 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.893 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202541 2024-09-18T07:41:39.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.893 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829844+0000", 2024-09-18T07:41:39.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829844+0000", 2024-09-18T07:41:39.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829845+0000", 2024-09-18T07:41:39.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0100000000000004e-07 2024-09-18T07:41:39.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829848+0000", 2024-09-18T07:41:39.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0170000000000001e-06 2024-09-18T07:41:39.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829854+0000", 2024-09-18T07:41:39.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9529999999999996e-06 2024-09-18T07:41:39.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829859+0000", 2024-09-18T07:41:39.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1499999999999998e-06 2024-09-18T07:41:39.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.830023+0000", 2024-09-18T07:41:39.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000163473 2024-09-18T07:41:39.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.830030+0000", 2024-09-18T07:41:39.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.396e-06 2024-09-18T07:41:39.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.921101+0000", 2024-09-18T07:41:39.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0910708730000001 2024-09-18T07:41:39.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.921187+0000", 2024-09-18T07:41:39.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6408000000000003e-05 2024-09-18T07:41:39.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:41:39.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.921265+0000", 2024-09-18T07:41:39.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8019e-05 2024-09-18T07:41:39.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925399+0000", 2024-09-18T07:41:39.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004133205 2024-09-18T07:41:39.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925470+0000", 2024-09-18T07:41:39.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1420000000000004e-05 2024-09-18T07:41:39.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928114+0000", 2024-09-18T07:41:39.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026441580000000002 2024-09-18T07:41:39.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928155+0000", 2024-09-18T07:41:39.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0620000000000001e-05 2024-09-18T07:41:39.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928186+0000", 2024-09-18T07:41:39.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0902999999999999e-05 2024-09-18T07:41:39.899 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.899 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.899 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.899 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202543 7.f 7:f02d8d75:::benchmark_data_smithi137_74180_object98127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:41:39.900 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.829915+0000", 2024-09-18T07:41:39.900 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 361.88843895299999, 2024-09-18T07:41:39.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.098406115, 2024-09-18T07:41:39.900 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.900 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.900 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.900 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.900 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.900 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.900 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202543 2024-09-18T07:41:39.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.900 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829915+0000", 2024-09-18T07:41:39.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829915+0000", 2024-09-18T07:41:39.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829916+0000", 2024-09-18T07:41:39.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3900000000000003e-07 2024-09-18T07:41:39.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829921+0000", 2024-09-18T07:41:39.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2250000000000002e-06 2024-09-18T07:41:39.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829927+0000", 2024-09-18T07:41:39.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3600000000000001e-06 2024-09-18T07:41:39.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829933+0000", 2024-09-18T07:41:39.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0240000000000003e-06 2024-09-18T07:41:39.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.830051+0000", 2024-09-18T07:41:39.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118131 2024-09-18T07:41:39.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.830057+0000", 2024-09-18T07:41:39.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4929999999999998e-06 2024-09-18T07:41:39.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.923373+0000", 2024-09-18T07:41:39.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.093316306 2024-09-18T07:41:39.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.923465+0000", 2024-09-18T07:41:39.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2260000000000001e-05 2024-09-18T07:41:39.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:41:39.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.923543+0000", 2024-09-18T07:41:39.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7601999999999993e-05 2024-09-18T07:41:39.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925460+0000", 2024-09-18T07:41:39.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001916846 2024-09-18T07:41:39.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925474+0000", 2024-09-18T07:41:39.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4708999999999999e-05 2024-09-18T07:41:39.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928254+0000", 2024-09-18T07:41:39.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002779662 2024-09-18T07:41:39.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928293+0000", 2024-09-18T07:41:39.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9419000000000003e-05 2024-09-18T07:41:39.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928321+0000", 2024-09-18T07:41:39.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8138999999999999e-05 2024-09-18T07:41:39.906 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.906 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.907 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.907 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202553 7.f 7:f853e395:::benchmark_data_smithi137_74180_object98137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:41:39.907 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.836313+0000", 2024-09-18T07:41:39.907 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 361.882041249, 2024-09-18T07:41:39.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.099117246, 2024-09-18T07:41:39.907 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.907 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.907 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.907 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.907 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.907 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.907 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202553 2024-09-18T07:41:39.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.908 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836313+0000", 2024-09-18T07:41:39.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836313+0000", 2024-09-18T07:41:39.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836314+0000", 2024-09-18T07:41:39.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3519999999999999e-06 2024-09-18T07:41:39.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836319+0000", 2024-09-18T07:41:39.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6469999999999997e-06 2024-09-18T07:41:39.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836329+0000", 2024-09-18T07:41:39.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0356e-05 2024-09-18T07:41:39.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836337+0000", 2024-09-18T07:41:39.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4290000000000004e-06 2024-09-18T07:41:39.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836372+0000", 2024-09-18T07:41:39.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5577000000000002e-05 2024-09-18T07:41:39.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836377+0000", 2024-09-18T07:41:39.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7199999999999997e-06 2024-09-18T07:41:39.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925531+0000", 2024-09-18T07:41:39.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0891534789999999 2024-09-18T07:41:39.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925628+0000", 2024-09-18T07:41:39.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7720999999999995e-05 2024-09-18T07:41:39.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:41:39.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925719+0000", 2024-09-18T07:41:39.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0804999999999995e-05 2024-09-18T07:41:39.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928378+0000", 2024-09-18T07:41:39.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026585670000000001 2024-09-18T07:41:39.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935204+0000", 2024-09-18T07:41:39.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068259189999999997 2024-09-18T07:41:39.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.913 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:39.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935218+0000", 2024-09-18T07:41:39.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4562000000000001e-05 2024-09-18T07:41:39.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935381+0000", 2024-09-18T07:41:39.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000162473 2024-09-18T07:41:39.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935430+0000", 2024-09-18T07:41:39.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9638999999999997e-05 2024-09-18T07:41:39.915 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.915 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.915 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.916 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202576 7.f 7:f8260e72:::benchmark_data_smithi137_74180_object98160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:41:39.916 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.852688+0000", 2024-09-18T07:41:39.916 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 361.86566591600001, 2024-09-18T07:41:39.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.082910424, 2024-09-18T07:41:39.916 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.916 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.916 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.916 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.916 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.916 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.916 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202576 2024-09-18T07:41:39.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.916 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852688+0000", 2024-09-18T07:41:39.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852688+0000", 2024-09-18T07:41:39.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852690+0000", 2024-09-18T07:41:39.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5039999999999999e-06 2024-09-18T07:41:39.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852694+0000", 2024-09-18T07:41:39.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4889999999999997e-06 2024-09-18T07:41:39.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852704+0000", 2024-09-18T07:41:39.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1689999999999998e-06 2024-09-18T07:41:39.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852709+0000", 2024-09-18T07:41:39.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6189999999999999e-06 2024-09-18T07:41:39.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852757+0000", 2024-09-18T07:41:39.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7676000000000002e-05 2024-09-18T07:41:39.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852765+0000", 2024-09-18T07:41:39.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9400000000000002e-06 2024-09-18T07:41:39.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928432+0000", 2024-09-18T07:41:39.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.075667572 2024-09-18T07:41:39.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928561+0000", 2024-09-18T07:41:39.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012874199999999999 2024-09-18T07:41:39.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:41:39.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928674+0000", 2024-09-18T07:41:39.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011271400000000001 2024-09-18T07:41:39.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935474+0000", 2024-09-18T07:41:39.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068001270000000004 2024-09-18T07:41:39.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935484+0000", 2024-09-18T07:41:39.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0515e-05 2024-09-18T07:41:39.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.922 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935518+0000", 2024-09-18T07:41:39.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3895e-05 2024-09-18T07:41:39.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.922 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935565+0000", 2024-09-18T07:41:39.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6241e-05 2024-09-18T07:41:39.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935599+0000", 2024-09-18T07:41:39.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4221000000000003e-05 2024-09-18T07:41:39.923 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.923 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.923 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.923 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202651 7.f 7:f9b36136:::benchmark_data_smithi137_74180_object98235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:41:39.923 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.895907+0000", 2024-09-18T07:41:39.923 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 361.822447146, 2024-09-18T07:41:39.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.071674998, 2024-09-18T07:41:39.923 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.923 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.924 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.924 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.924 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.924 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.924 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202651 2024-09-18T07:41:39.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.924 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.924 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.924 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895907+0000", 2024-09-18T07:41:39.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895907+0000", 2024-09-18T07:41:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895909+0000", 2024-09-18T07:41:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8989999999999999e-06 2024-09-18T07:41:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895915+0000", 2024-09-18T07:41:39.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6350000000000003e-06 2024-09-18T07:41:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895929+0000", 2024-09-18T07:41:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4174e-05 2024-09-18T07:41:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895936+0000", 2024-09-18T07:41:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1450000000000002e-06 2024-09-18T07:41:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895992+0000", 2024-09-18T07:41:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6255999999999998e-05 2024-09-18T07:41:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.896001+0000", 2024-09-18T07:41:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4919999999999993e-06 2024-09-18T07:41:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.947291+0000", 2024-09-18T07:41:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.051289881 2024-09-18T07:41:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.947431+0000", 2024-09-18T07:41:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014024799999999999 2024-09-18T07:41:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:41:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.947553+0000", 2024-09-18T07:41:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000122032 2024-09-18T07:41:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.957558+0000", 2024-09-18T07:41:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010004948 2024-09-18T07:41:39.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.957904+0000", 2024-09-18T07:41:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034637699999999998 2024-09-18T07:41:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967474+0000", 2024-09-18T07:41:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095700709999999994 2024-09-18T07:41:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.929 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967537+0000", 2024-09-18T07:41:39.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2881999999999999e-05 2024-09-18T07:41:39.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967582+0000", 2024-09-18T07:41:39.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4957999999999999e-05 2024-09-18T07:41:39.930 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.930 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.930 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.930 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202659 7.f 7:f246d9b4:::benchmark_data_smithi137_74180_object98243:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:41:39.930 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.901704+0000", 2024-09-18T07:41:39.931 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 361.81665031799997, 2024-09-18T07:41:39.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0662714099999999, 2024-09-18T07:41:39.931 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.931 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.931 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.931 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.931 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:41:39.931 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:41:39.931 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202659 2024-09-18T07:41:39.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.931 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.931 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.932 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901704+0000", 2024-09-18T07:41:39.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.932 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.932 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901704+0000", 2024-09-18T07:41:39.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.932 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.932 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901705+0000", 2024-09-18T07:41:39.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.296e-06 2024-09-18T07:41:39.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901709+0000", 2024-09-18T07:41:39.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9280000000000003e-06 2024-09-18T07:41:39.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901719+0000", 2024-09-18T07:41:39.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3049999999999998e-06 2024-09-18T07:41:39.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901724+0000", 2024-09-18T07:41:39.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2070000000000003e-06 2024-09-18T07:41:39.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901762+0000", 2024-09-18T07:41:39.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8578000000000002e-05 2024-09-18T07:41:39.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901784+0000", 2024-09-18T07:41:39.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1883e-05 2024-09-18T07:41:39.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.950958+0000", 2024-09-18T07:41:39.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.049174042 2024-09-18T07:41:39.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.935 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.951088+0000", 2024-09-18T07:41:39.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000129729 2024-09-18T07:41:39.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.935 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:41:39.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.951211+0000", 2024-09-18T07:41:39.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012286099999999999 2024-09-18T07:41:39.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.936 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.936 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.936 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.957893+0000", 2024-09-18T07:41:39.936 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066821709999999998 2024-09-18T07:41:39.936 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.936 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.936 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.936 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.958138+0000", 2024-09-18T07:41:39.936 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024456100000000002 2024-09-18T07:41:39.936 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.936 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.936 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.936 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967712+0000", 2024-09-18T07:41:39.936 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095747760000000001 2024-09-18T07:41:39.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.937 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967934+0000", 2024-09-18T07:41:39.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000221225 2024-09-18T07:41:39.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.937 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967975+0000", 2024-09-18T07:41:39.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1848000000000003e-05 2024-09-18T07:41:39.937 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.937 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.937 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.938 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109534 8.6 8:67e394d3:::benchmark_data_smithi137_101942_object4724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-18T07:41:39.938 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.048516+0000", 2024-09-18T07:41:39.938 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 237.66983796299999, 2024-09-18T07:41:39.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.182592823, 2024-09-18T07:41:39.938 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.938 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.938 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.938 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.938 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:41:39.938 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:41:39.938 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109534 2024-09-18T07:41:39.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.939 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048516+0000", 2024-09-18T07:41:39.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048516+0000", 2024-09-18T07:41:39.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048517+0000", 2024-09-18T07:41:39.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2e-07 2024-09-18T07:41:39.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048519+0000", 2024-09-18T07:41:39.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.232e-06 2024-09-18T07:41:39.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048523+0000", 2024-09-18T07:41:39.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4479999999999996e-06 2024-09-18T07:41:39.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048528+0000", 2024-09-18T07:41:39.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2169999999999996e-06 2024-09-18T07:41:39.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065112+0000", 2024-09-18T07:41:39.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016583912999999999 2024-09-18T07:41:39.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065121+0000", 2024-09-18T07:41:39.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8640000000000008e-06 2024-09-18T07:41:39.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.219070+0000", 2024-09-18T07:41:39.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1539486139999999 2024-09-18T07:41:39.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.219228+0000", 2024-09-18T07:41:39.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000157751 2024-09-18T07:41:39.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-09-18T07:41:39.943 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.219376+0000", 2024-09-18T07:41:39.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014781899999999999 2024-09-18T07:41:39.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.943 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.943 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226242+0000", 2024-09-18T07:41:39.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068661970000000001 2024-09-18T07:41:39.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.943 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.943 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226253+0000", 2024-09-18T07:41:39.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0929e-05 2024-09-18T07:41:39.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229198+0000", 2024-09-18T07:41:39.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029453790000000001 2024-09-18T07:41:39.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229278+0000", 2024-09-18T07:41:39.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9431000000000006e-05 2024-09-18T07:41:39.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.231109+0000", 2024-09-18T07:41:39.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018316090000000001 2024-09-18T07:41:39.945 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.945 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.945 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.945 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109539 8.6 8:66837289:::benchmark_data_smithi137_101942_object4729:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:41:39.945 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.065140+0000", 2024-09-18T07:41:39.945 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 237.653214494, 2024-09-18T07:41:39.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1676551340000001, 2024-09-18T07:41:39.945 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:39.945 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:39.945 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:41:39.946 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:41:39.946 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:41:39.946 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:41:39.946 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109539 2024-09-18T07:41:39.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.946 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:39.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:39.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065140+0000", 2024-09-18T07:41:39.946 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:39.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065140+0000", 2024-09-18T07:41:39.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:39.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:39.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065142+0000", 2024-09-18T07:41:39.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.548e-06 2024-09-18T07:41:39.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:39.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065149+0000", 2024-09-18T07:41:39.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7989999999999996e-06 2024-09-18T07:41:39.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:39.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065162+0000", 2024-09-18T07:41:39.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3027000000000001e-05 2024-09-18T07:41:39.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:39.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065171+0000", 2024-09-18T07:41:39.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1730000000000001e-06 2024-09-18T07:41:39.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065215+0000", 2024-09-18T07:41:39.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3679e-05 2024-09-18T07:41:39.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:41:39.949 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065222+0000", 2024-09-18T07:41:39.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9879999999999998e-06 2024-09-18T07:41:39.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:39.949 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.223751+0000", 2024-09-18T07:41:39.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1585293800000001 2024-09-18T07:41:39.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:39.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.223906+0000", 2024-09-18T07:41:39.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015412600000000001 2024-09-18T07:41:39.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.950 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-09-18T07:41:39.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.224068+0000", 2024-09-18T07:41:39.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016266499999999999 2024-09-18T07:41:39.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.950 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:41:39.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226399+0000", 2024-09-18T07:41:39.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002330557 2024-09-18T07:41:39.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226549+0000", 2024-09-18T07:41:39.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000150693 2024-09-18T07:41:39.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:41:39.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.231185+0000", 2024-09-18T07:41:39.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004635775 2024-09-18T07:41:39.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:41:39.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.231241+0000", 2024-09-18T07:41:39.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6001999999999997e-05 2024-09-18T07:41:39.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:39.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:39.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:41:39.952 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.232795+0000", 2024-09-18T07:41:39.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001553722 2024-09-18T07:41:39.952 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.952 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.952 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.952 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:39.952 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:39.952 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:41:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:39.954 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:39.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:40.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:40.091 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:40.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:40.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:40.192 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:40.228 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:40.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:40.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:40.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:40.364 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:40.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:40.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:40 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:41:40.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:40 smithi137 ceph-mon[29408]: from='client.25888 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:41:40.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:40 smithi137 ceph-mon[29408]: osdmap e336: 8 total, 8 up, 8 in 2024-09-18T07:41:40.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:40 smithi137 ceph-mon[29408]: osdmap e337: 8 total, 8 up, 8 in 2024-09-18T07:41:40.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:40 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:41:40.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:40 smithi137 ceph-mon[39175]: from='client.25888 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:41:40.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:40 smithi137 ceph-mon[39175]: osdmap e336: 8 total, 8 up, 8 in 2024-09-18T07:41:40.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:40 smithi137 ceph-mon[39175]: osdmap e337: 8 total, 8 up, 8 in 2024-09-18T07:41:40.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:40 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:41:40.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:40 smithi203 ceph-mon[32237]: from='client.25888 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:41:40.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:40 smithi203 ceph-mon[32237]: osdmap e336: 8 total, 8 up, 8 in 2024-09-18T07:41:40.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:40 smithi203 ceph-mon[32237]: osdmap e337: 8 total, 8 up, 8 in 2024-09-18T07:41:40.466 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:40.503 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:40.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:40.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:40.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:40.645 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:40.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:40.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:40.747 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:40.780 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:40.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:40.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:40.882 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:40.917 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:40.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:40.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:41.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:41.053 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:41.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:41.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:41.104 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.1 to 1 (10000) 2024-09-18T07:41:41.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:41.188 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:41.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:41.290 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:41.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:41.327 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:41.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:41.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:41 smithi137 ceph-mon[39175]: pgmap v573: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 772 B/s rd, 0 B/s wr, 875 op/s 2024-09-18T07:41:41.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:41 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/50553982' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:41:41.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:41 smithi137 ceph-mon[39175]: from='client.35813 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:41:41.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:41 smithi137 ceph-mon[29408]: pgmap v573: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 772 B/s rd, 0 B/s wr, 875 op/s 2024-09-18T07:41:41.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:41 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/50553982' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:41:41.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:41 smithi137 ceph-mon[29408]: from='client.35813 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:41:41.429 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:41 smithi203 ceph-mon[32237]: pgmap v573: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 772 B/s rd, 0 B/s wr, 875 op/s 2024-09-18T07:41:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:41 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/50553982' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:41:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:41 smithi203 ceph-mon[32237]: from='client.35813 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:41:41.465 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:41.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:41.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:41.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:41.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:41.610 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:41.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:41.711 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:41.748 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:41.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:41.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:41.850 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:41.887 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:41.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:41.989 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:42.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:42.028 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:42.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:42.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:42.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:42.168 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:42.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:42.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:42.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:42.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:42.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:42.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:42 smithi137 ceph-mon[29408]: from='client.35813 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:41:42.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:42 smithi137 ceph-mon[29408]: osdmap e338: 8 total, 8 up, 8 in 2024-09-18T07:41:42.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:42 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:41:42.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:42 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:41:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:42 smithi137 ceph-mon[39175]: from='client.35813 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:41:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:42 smithi137 ceph-mon[39175]: osdmap e338: 8 total, 8 up, 8 in 2024-09-18T07:41:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:42 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:41:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:42 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:41:42.456 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:42.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:42.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:42.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:42 smithi203 ceph-mon[32237]: from='client.35813 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:41:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:42 smithi203 ceph-mon[32237]: osdmap e338: 8 total, 8 up, 8 in 2024-09-18T07:41:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:42 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:41:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:42 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:41:42.558 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:42.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:42.594 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:42.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:42.696 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:42.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:42.733 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:42.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:42.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-18T07:41:42.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:42.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:42.885 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:42.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:42.986 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:43.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:43.021 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:43.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:43.122 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:43.170 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:43.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:43.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:43.273 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:43.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:43.312 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:43.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:43.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:43 smithi137 ceph-mon[29408]: pgmap v576: 158 pgs: 158 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-18T07:41:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:43 smithi137 ceph-mon[39175]: pgmap v576: 158 pgs: 158 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-18T07:41:43.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:43.452 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:43.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:43.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:43 smithi203 ceph-mon[32237]: pgmap v576: 158 pgs: 158 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-09-18T07:41:43.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T07:41:43.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:41:43.553 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:43.576 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:43.586 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:43.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:43.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:43.688 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:43.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:43.736 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:43.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:43.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:43.901 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:43.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:43.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:44.006 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:44.043 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:44.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:44.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:44.145 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:44.183 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:44.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:44.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:44.285 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:44.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:41:44.339 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:44.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:44.413 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:44.414 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:44.441 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:44.468 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:44.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:44.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:44.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:44.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:44.612 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:44.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:44.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:44.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:44.786 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:44.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:44.887 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:44.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:44.920 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:44.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:44.998 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.2 to 1 (10000) 2024-09-18T07:41:45.021 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:45.055 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:45.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:45.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:45.157 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:45.191 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:45.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:45.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:45.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:45.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:45.334 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:45.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:45.355 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:45 smithi137 ceph-mon[39175]: pgmap v577: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 989 op/s 2024-09-18T07:41:45.355 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:45 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3069375288' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:41:45.355 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:45 smithi137 ceph-mon[39175]: from='client.25894 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:41:45.355 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:45 smithi137 ceph-mon[39175]: from='client.25894 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:41:45.355 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:45 smithi137 ceph-mon[39175]: osdmap e339: 8 total, 8 up, 8 in 2024-09-18T07:41:45.355 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:45 smithi137 ceph-mon[29408]: pgmap v577: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 989 op/s 2024-09-18T07:41:45.355 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:45 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3069375288' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:41:45.355 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:45 smithi137 ceph-mon[29408]: from='client.25894 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:41:45.355 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:45 smithi137 ceph-mon[29408]: from='client.25894 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:41:45.355 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:45 smithi137 ceph-mon[29408]: osdmap e339: 8 total, 8 up, 8 in 2024-09-18T07:41:45.435 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:45.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:45 smithi203 ceph-mon[32237]: pgmap v577: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 989 op/s 2024-09-18T07:41:45.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:45 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3069375288' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:41:45.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:45 smithi203 ceph-mon[32237]: from='client.25894 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:41:45.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:45 smithi203 ceph-mon[32237]: from='client.25894 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:41:45.467 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:45 smithi203 ceph-mon[32237]: osdmap e339: 8 total, 8 up, 8 in 2024-09-18T07:41:45.469 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:45.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:45.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:45.520 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:41:45.520 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:41:45.520 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.520 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:181807 9.4 e339/289 9:2e226180:::benchmark_data_smithi137_126072_object49307:head v 339'11417, mlcod=339'11415)", 2024-09-18T07:41:45.521 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:45.507039+0000", 2024-09-18T07:41:45.521 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0081824189999999998, 2024-09-18T07:41:45.521 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082083390000000003, 2024-09-18T07:41:45.521 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:45.521 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:45.521 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:41:45.521 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:45.521 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.521 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:45.521 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.507039+0000", 2024-09-18T07:41:45.521 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:45.521 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.522 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.522 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:45.522 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.507039+0000", 2024-09-18T07:41:45.522 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:45.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.522 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.522 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:45.522 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.507040+0000", 2024-09-18T07:41:45.522 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2600000000000001e-07 2024-09-18T07:41:45.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.522 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.522 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:45.523 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.507048+0000", 2024-09-18T07:41:45.523 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6320000000000008e-06 2024-09-18T07:41:45.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.523 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.523 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:45.523 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.507055+0000", 2024-09-18T07:41:45.523 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6209999999999998e-06 2024-09-18T07:41:45.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.524 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.524 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:45.524 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.507060+0000", 2024-09-18T07:41:45.525 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3680000000000001e-06 2024-09-18T07:41:45.525 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.525 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.525 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:45.525 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.508722+0000", 2024-09-18T07:41:45.525 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016616720000000001 2024-09-18T07:41:45.525 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.525 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.525 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:45.525 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.508739+0000", 2024-09-18T07:41:45.525 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7380999999999998e-05 2024-09-18T07:41:45.525 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:45.525 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:45.526 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:45.526 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.526 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.526 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:181812 9.4 e339/289 9:275bb4f0:::benchmark_data_smithi137_126072_object49312:head v 339'11418, mlcod=339'11415)", 2024-09-18T07:41:45.526 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:45.509288+0000", 2024-09-18T07:41:45.526 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0059341660000000003, 2024-09-18T07:41:45.526 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0060121460000000003, 2024-09-18T07:41:45.526 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:45.526 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:45.526 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:41:45.527 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:45.527 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.527 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:45.527 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.509288+0000", 2024-09-18T07:41:45.527 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:45.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.527 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.527 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:45.527 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.509288+0000", 2024-09-18T07:41:45.527 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:45.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.528 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:45.528 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.509289+0000", 2024-09-18T07:41:45.528 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0920000000000001e-06 2024-09-18T07:41:45.528 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.528 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:45.528 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.509297+0000", 2024-09-18T07:41:45.528 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5380000000000002e-06 2024-09-18T07:41:45.528 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.528 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:45.529 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.509305+0000", 2024-09-18T07:41:45.529 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1580000000000005e-06 2024-09-18T07:41:45.529 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.529 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.529 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:45.529 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.509311+0000", 2024-09-18T07:41:45.529 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7749999999999998e-06 2024-09-18T07:41:45.529 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.529 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.529 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:45.529 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.511230+0000", 2024-09-18T07:41:45.529 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019190850000000001 2024-09-18T07:41:45.529 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.530 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.530 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:45.530 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.511248+0000", 2024-09-18T07:41:45.530 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7215e-05 2024-09-18T07:41:45.530 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:45.530 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:45.530 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:45.530 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.530 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.530 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:181816 9.6 e339/289 9:6b45d1ad:::benchmark_data_smithi137_126072_object49316:head v 339'11333, mlcod=339'11331)", 2024-09-18T07:41:45.530 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:45.511274+0000", 2024-09-18T07:41:45.530 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00394772, 2024-09-18T07:41:45.530 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040740860000000002, 2024-09-18T07:41:45.531 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:45.531 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:45.531 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:41:45.531 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:45.531 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.531 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:45.531 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.511274+0000", 2024-09-18T07:41:45.531 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:45.531 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.531 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.531 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:45.531 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.511274+0000", 2024-09-18T07:41:45.532 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:45.532 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.532 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.532 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:45.532 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.511275+0000", 2024-09-18T07:41:45.532 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1939999999999999e-06 2024-09-18T07:41:45.532 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.532 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.532 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:45.532 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.511282+0000", 2024-09-18T07:41:45.532 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4559999999999997e-06 2024-09-18T07:41:45.532 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.532 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.533 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:45.533 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.511304+0000", 2024-09-18T07:41:45.533 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2585000000000001e-05 2024-09-18T07:41:45.533 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.533 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.533 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:45.533 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.511311+0000", 2024-09-18T07:41:45.533 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7610000000000002e-06 2024-09-18T07:41:45.533 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.533 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.533 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:45.534 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.511386+0000", 2024-09-18T07:41:45.534 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4802000000000007e-05 2024-09-18T07:41:45.534 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.534 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.534 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:45.534 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.511403+0000", 2024-09-18T07:41:45.534 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6932e-05 2024-09-18T07:41:45.534 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:45.534 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:45.534 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:45.534 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.534 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.534 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:181814 9.4 e339/289 9:24998bac:::benchmark_data_smithi137_126072_object49314:head v 339'11419, mlcod=339'11415)", 2024-09-18T07:41:45.535 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:45.511809+0000", 2024-09-18T07:41:45.535 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0034129379999999999, 2024-09-18T07:41:45.535 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035837239999999999, 2024-09-18T07:41:45.535 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:45.535 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:45.535 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:41:45.535 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:45.535 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.535 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:45.535 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.511809+0000", 2024-09-18T07:41:45.535 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:45.535 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.535 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.536 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:45.536 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.511809+0000", 2024-09-18T07:41:45.536 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:45.536 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.536 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.536 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:45.536 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.511810+0000", 2024-09-18T07:41:45.536 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.252e-06 2024-09-18T07:41:45.536 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.536 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.536 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:45.536 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.511818+0000", 2024-09-18T07:41:45.536 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2169999999999994e-06 2024-09-18T07:41:45.537 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.537 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.537 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:45.537 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.511828+0000", 2024-09-18T07:41:45.537 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0056e-05 2024-09-18T07:41:45.540 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.540 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.540 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:45.540 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.511835+0000", 2024-09-18T07:41:45.540 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7680000000000003e-06 2024-09-18T07:41:45.541 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.541 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.541 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:45.541 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.514907+0000", 2024-09-18T07:41:45.541 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030714190000000001 2024-09-18T07:41:45.541 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.541 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.541 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:45.542 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.514927+0000", 2024-09-18T07:41:45.542 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0701000000000001e-05 2024-09-18T07:41:45.542 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:45.542 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:45.542 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:45.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.542 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.542 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:181819 9.f e339/289 9:f22db417:::benchmark_data_smithi137_126072_object49319:head v 339'11398, mlcod=339'11396)", 2024-09-18T07:41:45.542 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:45.512647+0000", 2024-09-18T07:41:45.542 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0025746160000000001, 2024-09-18T07:41:45.542 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002794183, 2024-09-18T07:41:45.542 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:45.543 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:45.543 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:41:45.543 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:45.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:45.543 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.512647+0000", 2024-09-18T07:41:45.543 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:45.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:45.543 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.512647+0000", 2024-09-18T07:41:45.543 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:45.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.544 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:45.544 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.512648+0000", 2024-09-18T07:41:45.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1599999999999999e-06 2024-09-18T07:41:45.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.544 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:45.544 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.512669+0000", 2024-09-18T07:41:45.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0996e-05 2024-09-18T07:41:45.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.544 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:45.545 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.512680+0000", 2024-09-18T07:41:45.545 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0942e-05 2024-09-18T07:41:45.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.545 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.545 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:45.545 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.512686+0000", 2024-09-18T07:41:45.545 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.84e-06 2024-09-18T07:41:45.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.545 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.545 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:41:45.545 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.512746+0000", 2024-09-18T07:41:45.545 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.94e-05 2024-09-18T07:41:45.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.546 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:41:45.546 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.512762+0000", 2024-09-18T07:41:45.546 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6838e-05 2024-09-18T07:41:45.546 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:45.546 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:45.546 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:45.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.546 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:181817 9.4 e339/289)", 2024-09-18T07:41:45.546 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:45.513684+0000", 2024-09-18T07:41:45.546 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.001537496, 2024-09-18T07:41:45.546 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001799412, 2024-09-18T07:41:45.547 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:45.547 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:45.547 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:41:45.547 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:45.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:45.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.513684+0000", 2024-09-18T07:41:45.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:45.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:45.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.513684+0000", 2024-09-18T07:41:45.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:45.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.548 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:45.548 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.513686+0000", 2024-09-18T07:41:45.548 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.243e-06 2024-09-18T07:41:45.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.548 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:45.548 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.513693+0000", 2024-09-18T07:41:45.548 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7419999999999997e-06 2024-09-18T07:41:45.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.548 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:45.548 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.513704+0000", 2024-09-18T07:41:45.549 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0349e-05 2024-09-18T07:41:45.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.549 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.549 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:45.549 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.513711+0000", 2024-09-18T07:41:45.549 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8689999999999998e-06 2024-09-18T07:41:45.549 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:45.549 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:45.549 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:45.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.549 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.549 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:181822 9.4 e339/289)", 2024-09-18T07:41:45.550 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:45.515420+0000", 2024-09-18T07:41:45.550 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 4294967295.9998016, 2024-09-18T07:41:45.550 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010021700000000001, 2024-09-18T07:41:45.550 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:41:45.550 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:41:45.550 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:41:45.550 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:41:45.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.550 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:41:45.550 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.515420+0000", 2024-09-18T07:41:45.550 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:45.550 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.550 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:41:45.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.515420+0000", 2024-09-18T07:41:45.551 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:41:45.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:41:45.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.515421+0000", 2024-09-18T07:41:45.551 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.229e-06 2024-09-18T07:41:45.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:41:45.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.515431+0000", 2024-09-18T07:41:45.551 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5259999999999999e-06 2024-09-18T07:41:45.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.552 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:41:45.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.515440+0000", 2024-09-18T07:41:45.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0960000000000007e-06 2024-09-18T07:41:45.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:41:45.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:41:45.552 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:41:45.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:45.515446+0000", 2024-09-18T07:41:45.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1759999999999998e-06 2024-09-18T07:41:45.552 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:45.552 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:41:45.552 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:45.553 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:41:45.553 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:41:45.553 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 7 2024-09-18T07:41:45.553 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:41:45.571 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:45.607 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:45.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:45.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:45.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:45.748 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:45.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:45.850 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:45.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:45.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:45.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:45.989 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:46.019 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:46.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:46.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:46.035 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-18T07:41:46.121 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:46.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:46.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:46.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:46.271 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:46.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:46.302 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:46.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:46.404 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:46.435 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:46.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:46.536 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:46.570 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:46.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:46.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:46.672 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:46.710 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:46.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:46.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:46.811 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:46.846 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:46.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:46.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:46.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:46.986 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:46.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:47.088 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:47.126 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:47.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:47.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:47.140 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:41:47.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:47.267 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:47.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:47.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:47.402 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:47.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:47.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:47.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:41:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:41:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:41:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:47 smithi137 ceph-mon[39175]: pgmap v579: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 989 op/s 2024-09-18T07:41:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:47 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1758498378' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:41:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:47 smithi137 ceph-mon[39175]: from='client.25900 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:41:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:47 smithi137 ceph-mon[29408]: pgmap v579: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 989 op/s 2024-09-18T07:41:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:47 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1758498378' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:41:47.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:47 smithi137 ceph-mon[29408]: from='client.25900 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:41:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:47 smithi203 ceph-mon[32237]: pgmap v579: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 989 op/s 2024-09-18T07:41:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:47 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1758498378' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:41:47.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:47 smithi203 ceph-mon[32237]: from='client.25900 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:41:47.504 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:47.529 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T07:41:47.533 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:47.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:47.634 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:47.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:47.668 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:47.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:47.769 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:47.803 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:47.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:47.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:47.904 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:47.936 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:47.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:47.961 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:41:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:41:48.039 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:48.074 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:48.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:48.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:48.176 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:48.216 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:48.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:48.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:48.312 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:48.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:48.372 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:48.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:48.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:48.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:48 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:41:48.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:48 smithi137 ceph-mon[29408]: from='client.25900 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:41:48.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:48 smithi137 ceph-mon[29408]: osdmap e340: 8 total, 8 up, 8 in 2024-09-18T07:41:48.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:48 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:41:48.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:48 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:41:48.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:48 smithi137 ceph-mon[29408]: 3.15 scrub starts 2024-09-18T07:41:48.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:48 smithi137 ceph-mon[29408]: 3.15 scrub ok 2024-09-18T07:41:48.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:48 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:41:48.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:48 smithi137 ceph-mon[39175]: from='client.25900 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:41:48.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:48 smithi137 ceph-mon[39175]: osdmap e340: 8 total, 8 up, 8 in 2024-09-18T07:41:48.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:48 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:41:48.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:48 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:41:48.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:48 smithi137 ceph-mon[39175]: 3.15 scrub starts 2024-09-18T07:41:48.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:48 smithi137 ceph-mon[39175]: 3.15 scrub ok 2024-09-18T07:41:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:48 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:41:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:48 smithi203 ceph-mon[32237]: from='client.25900 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:41:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:48 smithi203 ceph-mon[32237]: osdmap e340: 8 total, 8 up, 8 in 2024-09-18T07:41:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:48 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:41:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:48 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:41:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:48 smithi203 ceph-mon[32237]: 3.15 scrub starts 2024-09-18T07:41:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:48 smithi203 ceph-mon[32237]: 3.15 scrub ok 2024-09-18T07:41:48.474 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:48.496 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:48.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:48.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:48.598 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:48.638 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:48.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:48.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:48.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:48.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:48.779 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:48.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:48.881 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:48.916 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:48.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:48.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:48.970 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:49.014 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:41:49.015 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:41:49.018 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:49.027 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:49.056 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:41:49.056 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:41:49.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:49.083 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:49.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:49.089 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:41:49.089 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:41:49.121 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:41:49.121 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:41:49.159 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:41:49.159 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:41:49.184 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:49.191 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:41:49.191 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:41:49.218 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:49.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:49.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:49.220 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:41:49.220 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:41:49.254 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:41:49.255 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:41:49.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:49.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:49.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[39175]: pgmap v581: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.90k op/s 2024-09-18T07:41:49.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[39175]: 4.9 scrub starts 2024-09-18T07:41:49.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[39175]: 4.9 scrub ok 2024-09-18T07:41:49.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[39175]: 4.1c scrub starts 2024-09-18T07:41:49.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[39175]: 4.1c scrub ok 2024-09-18T07:41:49.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[39175]: osdmap e341: 8 total, 8 up, 8 in 2024-09-18T07:41:49.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[39175]: 4.17 scrub starts 2024-09-18T07:41:49.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[39175]: 4.17 scrub ok 2024-09-18T07:41:49.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2306560835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:49.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:49.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[29408]: pgmap v581: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.90k op/s 2024-09-18T07:41:49.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[29408]: 4.9 scrub starts 2024-09-18T07:41:49.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[29408]: 4.9 scrub ok 2024-09-18T07:41:49.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[29408]: 4.1c scrub starts 2024-09-18T07:41:49.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[29408]: 4.1c scrub ok 2024-09-18T07:41:49.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[29408]: osdmap e341: 8 total, 8 up, 8 in 2024-09-18T07:41:49.348 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[29408]: 4.17 scrub starts 2024-09-18T07:41:49.348 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[29408]: 4.17 scrub ok 2024-09-18T07:41:49.348 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:49 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2306560835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:49.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:49.370 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:49.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:49 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:49 smithi203 ceph-mon[32237]: pgmap v581: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.90k op/s 2024-09-18T07:41:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:49 smithi203 ceph-mon[32237]: 4.9 scrub starts 2024-09-18T07:41:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:49 smithi203 ceph-mon[32237]: 4.9 scrub ok 2024-09-18T07:41:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:49 smithi203 ceph-mon[32237]: 4.1c scrub starts 2024-09-18T07:41:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:49 smithi203 ceph-mon[32237]: 4.1c scrub ok 2024-09-18T07:41:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:49 smithi203 ceph-mon[32237]: osdmap e341: 8 total, 8 up, 8 in 2024-09-18T07:41:49.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:49 smithi203 ceph-mon[32237]: 4.17 scrub starts 2024-09-18T07:41:49.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:49 smithi203 ceph-mon[32237]: 4.17 scrub ok 2024-09-18T07:41:49.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:49 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2306560835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:49.471 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:49.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:49.505 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:49.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:49.606 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:49.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:49.644 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:49.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:49.745 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:49.780 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:49.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:49.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:49.881 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:49.911 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:49.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:49.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:50.013 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:50.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:50.042 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:50.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:50.143 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:50.201 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:50.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:50.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:50.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:50.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:41:50.339 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:50.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:50.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:50.372 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:50 smithi137 ceph-mon[29408]: 6.17 scrub starts 2024-09-18T07:41:50.372 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:50 smithi137 ceph-mon[29408]: 6.17 scrub ok 2024-09-18T07:41:50.372 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:50 smithi137 ceph-mon[39175]: 6.17 scrub starts 2024-09-18T07:41:50.372 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:50 smithi137 ceph-mon[39175]: 6.17 scrub ok 2024-09-18T07:41:50.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:50.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:50 smithi203 ceph-mon[32237]: 6.17 scrub starts 2024-09-18T07:41:50.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:50 smithi203 ceph-mon[32237]: 6.17 scrub ok 2024-09-18T07:41:50.485 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:50.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:50.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:50.587 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:50.619 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:50.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:50.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:50.722 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:50.755 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:50.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:50.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:50.857 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:50.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:50.896 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:50.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:50.905 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:41:50.906 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:41:50.997 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:51.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:51.031 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:51.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:51.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:51.170 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:51.170 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:51.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:51.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:51.172 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.3 to 1 (10000) 2024-09-18T07:41:51.273 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:51.310 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:51.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:51.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:51.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:51.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:51.448 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:51.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:51.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:51 smithi203 ceph-mon[32237]: pgmap v583: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-18T07:41:51.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:51 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2169933917' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:41:51.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:51 smithi203 ceph-mon[32237]: from='client.25912 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:41:51.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:51 smithi203 ceph-mon[32237]: from='client.25912 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T07:41:51.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:51 smithi203 ceph-mon[32237]: osdmap e342: 8 total, 8 up, 8 in 2024-09-18T07:41:51.533 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:51 smithi137 ceph-mon[29408]: pgmap v583: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-18T07:41:51.533 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:51 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2169933917' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:41:51.533 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:51 smithi137 ceph-mon[29408]: from='client.25912 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:41:51.534 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:51 smithi137 ceph-mon[29408]: from='client.25912 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T07:41:51.534 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:51 smithi137 ceph-mon[29408]: osdmap e342: 8 total, 8 up, 8 in 2024-09-18T07:41:51.534 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:51 smithi137 ceph-mon[39175]: pgmap v583: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s 2024-09-18T07:41:51.534 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:51 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2169933917' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:41:51.534 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:51 smithi137 ceph-mon[39175]: from='client.25912 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:41:51.534 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:51 smithi137 ceph-mon[39175]: from='client.25912 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T07:41:51.534 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:51 smithi137 ceph-mon[39175]: osdmap e342: 8 total, 8 up, 8 in 2024-09-18T07:41:51.549 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:51.589 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:51.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:51.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:51.691 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:51.729 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:51.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:51.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:51.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:51.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:51.867 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:51.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:51.968 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:51.998 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:51.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:51.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:52.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-18T07:41:52.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:52.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:52.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:52.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:52.242 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:52.270 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:52.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:52.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:52.372 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:52.408 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:52.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:52.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:52 smithi137 ceph-mon[29408]: 5.18 scrub starts 2024-09-18T07:41:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:52 smithi137 ceph-mon[29408]: 5.18 scrub ok 2024-09-18T07:41:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:52 smithi137 ceph-mon[29408]: 4.8 scrub starts 2024-09-18T07:41:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:52 smithi137 ceph-mon[29408]: 4.8 scrub ok 2024-09-18T07:41:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:52 smithi137 ceph-mon[39175]: 5.18 scrub starts 2024-09-18T07:41:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:52 smithi137 ceph-mon[39175]: 5.18 scrub ok 2024-09-18T07:41:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:52 smithi137 ceph-mon[39175]: 4.8 scrub starts 2024-09-18T07:41:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:52 smithi137 ceph-mon[39175]: 4.8 scrub ok 2024-09-18T07:41:52.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:52 smithi203 ceph-mon[32237]: 5.18 scrub starts 2024-09-18T07:41:52.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:52 smithi203 ceph-mon[32237]: 5.18 scrub ok 2024-09-18T07:41:52.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:52 smithi203 ceph-mon[32237]: 4.8 scrub starts 2024-09-18T07:41:52.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:52 smithi203 ceph-mon[32237]: 4.8 scrub ok 2024-09-18T07:41:52.510 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:52.547 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:52.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:52.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:52.649 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:52.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:41:52.683 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:52.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:52.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:52.785 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:52.821 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:52.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:52.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:52.923 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:52.953 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:52.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:52.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:53.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:53.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:53.092 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:53.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:53.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:53.229 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:53.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:53.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:53.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:53.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:53.353 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:53.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:53 smithi137 ceph-mon[29408]: pgmap v585: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-18T07:41:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:53 smithi137 ceph-mon[29408]: 5.16 scrub starts 2024-09-18T07:41:53.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:53 smithi137 ceph-mon[29408]: 5.16 scrub ok 2024-09-18T07:41:53.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:53 smithi137 ceph-mon[29408]: 3.11 scrub starts 2024-09-18T07:41:53.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:53 smithi137 ceph-mon[29408]: 3.11 scrub ok 2024-09-18T07:41:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:53 smithi137 ceph-mon[39175]: pgmap v585: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-18T07:41:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:53 smithi137 ceph-mon[39175]: 5.16 scrub starts 2024-09-18T07:41:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:53 smithi137 ceph-mon[39175]: 5.16 scrub ok 2024-09-18T07:41:53.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:53 smithi137 ceph-mon[39175]: 3.11 scrub starts 2024-09-18T07:41:53.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:53 smithi137 ceph-mon[39175]: 3.11 scrub ok 2024-09-18T07:41:53.454 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:53.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:53 smithi203 ceph-mon[32237]: pgmap v585: 158 pgs: 158 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-18T07:41:53.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:53 smithi203 ceph-mon[32237]: 5.16 scrub starts 2024-09-18T07:41:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:53 smithi203 ceph-mon[32237]: 5.16 scrub ok 2024-09-18T07:41:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:53 smithi203 ceph-mon[32237]: 3.11 scrub starts 2024-09-18T07:41:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:53 smithi203 ceph-mon[32237]: 3.11 scrub ok 2024-09-18T07:41:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:53.487 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:53.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:53.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:53.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:53.624 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:53.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:53.725 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:53.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:53.759 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:53.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:53.860 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:53.893 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:53.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:53.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:53.995 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:54.028 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:54.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:54.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:54.130 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:54.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:54.170 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:54.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:54.271 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:54.293 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:54.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:54.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:54.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:54.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:54.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:54.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:54.518 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:54.550 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:54.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:54.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:54.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:54.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:54 smithi137 ceph-mon[29408]: 1.5 scrub starts 2024-09-18T07:41:54.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:54 smithi137 ceph-mon[29408]: 1.5 scrub ok 2024-09-18T07:41:54.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:54 smithi137 ceph-mon[39175]: 1.5 scrub starts 2024-09-18T07:41:54.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:54 smithi137 ceph-mon[39175]: 1.5 scrub ok 2024-09-18T07:41:54.685 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:54.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:54.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:54.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:54 smithi203 ceph-mon[32237]: 1.5 scrub starts 2024-09-18T07:41:54.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:54 smithi203 ceph-mon[32237]: 1.5 scrub ok 2024-09-18T07:41:54.787 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:54.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:54.813 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:54.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:54.914 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:54.949 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:54.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:54.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:55.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:55.082 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:55.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:55.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:55.184 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:55.223 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:55.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:55.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:55.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:55.365 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:55.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:55.467 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:55.498 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:55.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:55.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:55.600 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:55.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:55.637 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:55.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:55.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:55 smithi137 ceph-mon[29408]: pgmap v586: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 783 B/s rd, 0 B/s wr, 1.39k op/s 2024-09-18T07:41:55.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:55 smithi137 ceph-mon[29408]: 4.13 scrub starts 2024-09-18T07:41:55.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:55 smithi137 ceph-mon[29408]: 4.13 scrub ok 2024-09-18T07:41:55.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:55 smithi137 ceph-mon[29408]: 3.a scrub starts 2024-09-18T07:41:55.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:55 smithi137 ceph-mon[29408]: 3.a scrub ok 2024-09-18T07:41:55.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:55 smithi137 ceph-mon[29408]: 4.7 scrub starts 2024-09-18T07:41:55.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:55 smithi137 ceph-mon[29408]: 4.7 scrub ok 2024-09-18T07:41:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:55 smithi137 ceph-mon[39175]: pgmap v586: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 783 B/s rd, 0 B/s wr, 1.39k op/s 2024-09-18T07:41:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:55 smithi137 ceph-mon[39175]: 4.13 scrub starts 2024-09-18T07:41:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:55 smithi137 ceph-mon[39175]: 4.13 scrub ok 2024-09-18T07:41:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:55 smithi137 ceph-mon[39175]: 3.a scrub starts 2024-09-18T07:41:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:55 smithi137 ceph-mon[39175]: 3.a scrub ok 2024-09-18T07:41:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:55 smithi137 ceph-mon[39175]: 4.7 scrub starts 2024-09-18T07:41:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:55 smithi137 ceph-mon[39175]: 4.7 scrub ok 2024-09-18T07:41:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:55 smithi203 ceph-mon[32237]: pgmap v586: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 783 B/s rd, 0 B/s wr, 1.39k op/s 2024-09-18T07:41:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:55 smithi203 ceph-mon[32237]: 4.13 scrub starts 2024-09-18T07:41:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:55 smithi203 ceph-mon[32237]: 4.13 scrub ok 2024-09-18T07:41:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:55 smithi203 ceph-mon[32237]: 3.a scrub starts 2024-09-18T07:41:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:55 smithi203 ceph-mon[32237]: 3.a scrub ok 2024-09-18T07:41:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:55 smithi203 ceph-mon[32237]: 4.7 scrub starts 2024-09-18T07:41:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:55 smithi203 ceph-mon[32237]: 4.7 scrub ok 2024-09-18T07:41:55.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:55.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:55.792 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:55.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:55.893 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:55.924 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:55.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:56.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:56.053 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:56.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:56.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:56.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:56.192 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:56.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:56.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:56.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T07:41:56.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:56.301 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:56.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:56.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:56.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:56.454 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:56.489 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:56.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:56.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:56.568 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:56 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:41:56.568 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:56 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:41:56.568 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:56 smithi137 ceph-mon[29408]: 4.4 scrub starts 2024-09-18T07:41:56.568 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:56 smithi137 ceph-mon[29408]: 4.4 scrub ok 2024-09-18T07:41:56.568 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:56 smithi137 ceph-mon[29408]: osdmap e343: 8 total, 8 up, 8 in 2024-09-18T07:41:56.568 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:56 smithi137 ceph-mon[29408]: pgmap v588: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 680 B/s rd, 0 B/s wr, 874 op/s 2024-09-18T07:41:56.569 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:56 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:41:56.569 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:56 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:41:56.569 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:56 smithi137 ceph-mon[39175]: 4.4 scrub starts 2024-09-18T07:41:56.569 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:56 smithi137 ceph-mon[39175]: 4.4 scrub ok 2024-09-18T07:41:56.569 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:56 smithi137 ceph-mon[39175]: osdmap e343: 8 total, 8 up, 8 in 2024-09-18T07:41:56.570 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:56 smithi137 ceph-mon[39175]: pgmap v588: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 680 B/s rd, 0 B/s wr, 874 op/s 2024-09-18T07:41:56.591 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:56.650 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:56.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:56.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:56 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:41:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:56 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:41:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:56 smithi203 ceph-mon[32237]: 4.4 scrub starts 2024-09-18T07:41:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:56 smithi203 ceph-mon[32237]: 4.4 scrub ok 2024-09-18T07:41:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:56 smithi203 ceph-mon[32237]: osdmap e343: 8 total, 8 up, 8 in 2024-09-18T07:41:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:56 smithi203 ceph-mon[32237]: pgmap v588: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 680 B/s rd, 0 B/s wr, 874 op/s 2024-09-18T07:41:56.753 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:56.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:56.792 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:56.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:56.855 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:41:56.893 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:56.930 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:56.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:56.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:57.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:57.073 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:57.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:57.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:57.175 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:57.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:57.207 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:57.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:57.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:41:57.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:57.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:57.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:41:57.381 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:41:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:41:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:41:57.381 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:57 smithi137 ceph-mon[29408]: 4.1a scrub starts 2024-09-18T07:41:57.381 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:57 smithi137 ceph-mon[29408]: 4.1a scrub ok 2024-09-18T07:41:57.381 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:57 smithi137 ceph-mon[29408]: 3.12 scrub starts 2024-09-18T07:41:57.381 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:57 smithi137 ceph-mon[29408]: 3.12 scrub ok 2024-09-18T07:41:57.381 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:57 smithi137 ceph-mon[29408]: 6.f scrub starts 2024-09-18T07:41:57.381 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:57 smithi137 ceph-mon[29408]: 6.f scrub ok 2024-09-18T07:41:57.381 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:57 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:41:57.382 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:57 smithi137 ceph-mon[39175]: 4.1a scrub starts 2024-09-18T07:41:57.382 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:57 smithi137 ceph-mon[39175]: 4.1a scrub ok 2024-09-18T07:41:57.382 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:57 smithi137 ceph-mon[39175]: 3.12 scrub starts 2024-09-18T07:41:57.382 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:57 smithi137 ceph-mon[39175]: 3.12 scrub ok 2024-09-18T07:41:57.382 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:57 smithi137 ceph-mon[39175]: 6.f scrub starts 2024-09-18T07:41:57.382 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:57 smithi137 ceph-mon[39175]: 6.f scrub ok 2024-09-18T07:41:57.382 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:57 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:41:57.382 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:41:57.383 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:41:57.426 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:41:57.426 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:41:57.452 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:57.469 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:41:57.469 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:41:57.495 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:57 smithi203 ceph-mon[32237]: 4.1a scrub starts 2024-09-18T07:41:57.495 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:57 smithi203 ceph-mon[32237]: 4.1a scrub ok 2024-09-18T07:41:57.495 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:57 smithi203 ceph-mon[32237]: 3.12 scrub starts 2024-09-18T07:41:57.495 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:57 smithi203 ceph-mon[32237]: 3.12 scrub ok 2024-09-18T07:41:57.496 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:57 smithi203 ceph-mon[32237]: 6.f scrub starts 2024-09-18T07:41:57.496 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:57 smithi203 ceph-mon[32237]: 6.f scrub ok 2024-09-18T07:41:57.496 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:57 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:41:57.503 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:57.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:57.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:57.515 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:41:57.515 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:41:57.555 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:41:57.555 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:41:57.590 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:41:57.590 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:41:57.605 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:57.625 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:41:57.625 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:41:57.632 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:57.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:57.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:57.661 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:41:57.662 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:41:57.734 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:57.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:57.771 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:57.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:57.872 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:57.909 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:57.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:57.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:57.960 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:41:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:41:58.011 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:58.049 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:58.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:41:58.151 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:58.174 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:58.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:58.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:58.258 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:41:58.276 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:58.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:58.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:58.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:58.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:58.454 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:58.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:58.523 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:58 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/562365325' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:58.523 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:58 smithi137 ceph-mon[29408]: 3.1c scrub starts 2024-09-18T07:41:58.523 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:58 smithi137 ceph-mon[29408]: 3.1c scrub ok 2024-09-18T07:41:58.523 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:58 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:58.523 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:58 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3432615315' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:58.523 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:58 smithi137 ceph-mon[29408]: pgmap v589: 158 pgs: 158 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-09-18T07:41:58.523 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:58 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/562365325' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:58.524 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:58 smithi137 ceph-mon[39175]: 3.1c scrub starts 2024-09-18T07:41:58.524 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:58 smithi137 ceph-mon[39175]: 3.1c scrub ok 2024-09-18T07:41:58.524 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:58 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:58.524 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:58 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3432615315' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:58.524 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:58 smithi137 ceph-mon[39175]: pgmap v589: 158 pgs: 158 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-09-18T07:41:58.555 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:58.584 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:58.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:58.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:58.686 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:58.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:58 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/562365325' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:41:58.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:58 smithi203 ceph-mon[32237]: 3.1c scrub starts 2024-09-18T07:41:58.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:58 smithi203 ceph-mon[32237]: 3.1c scrub ok 2024-09-18T07:41:58.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:58 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:41:58.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:58 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3432615315' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:41:58.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:58 smithi203 ceph-mon[32237]: pgmap v589: 158 pgs: 158 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-09-18T07:41:58.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:58.717 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:58.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:58.818 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:41:58.853 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:58.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:58.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:41:58.955 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:41:58.980 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:58.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:58.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:41:59.082 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:59.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:59.121 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:59.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:59.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:59.255 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:59.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:59.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:59.270 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.4 to 1 (10000) 2024-09-18T07:41:59.358 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:41:59.391 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:59.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:59.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:41:59.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:41:59.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:59.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:59.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:41:59.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:59 smithi203 ceph-mon[32237]: 5.8 scrub starts 2024-09-18T07:41:59.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:59 smithi203 ceph-mon[32237]: 5.8 scrub ok 2024-09-18T07:41:59.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:59 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:41:59.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:59 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:41:59.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:59 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3432615315' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:41:59.613 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:59 smithi203 ceph-mon[32237]: osdmap e344: 8 total, 8 up, 8 in 2024-09-18T07:41:59.613 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:59 smithi203 ceph-mon[32237]: 5.19 scrub starts 2024-09-18T07:41:59.613 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:59 smithi203 ceph-mon[32237]: 5.19 scrub ok 2024-09-18T07:41:59.613 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:59 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1743031556' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:41:59.613 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:41:59 smithi203 ceph-mon[32237]: from='client.35849 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:41:59.633 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:41:59.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:59.665 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:59.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:41:59.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[29408]: 5.8 scrub starts 2024-09-18T07:41:59.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[29408]: 5.8 scrub ok 2024-09-18T07:41:59.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:41:59.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:41:59.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3432615315' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:41:59.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[29408]: osdmap e344: 8 total, 8 up, 8 in 2024-09-18T07:41:59.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[29408]: 5.19 scrub starts 2024-09-18T07:41:59.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[29408]: 5.19 scrub ok 2024-09-18T07:41:59.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1743031556' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:41:59.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[29408]: from='client.35849 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:41:59.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[39175]: 5.8 scrub starts 2024-09-18T07:41:59.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[39175]: 5.8 scrub ok 2024-09-18T07:41:59.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:41:59.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:41:59.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3432615315' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:41:59.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[39175]: osdmap e344: 8 total, 8 up, 8 in 2024-09-18T07:41:59.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[39175]: 5.19 scrub starts 2024-09-18T07:41:59.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[39175]: 5.19 scrub ok 2024-09-18T07:41:59.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1743031556' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:41:59.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:41:59 smithi137 ceph-mon[39175]: from='client.35849 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:41:59.766 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:41:59.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:59.799 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:41:59.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:41:59.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:41:59.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:41:59.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:41:59.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:00.040 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:00.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:00.062 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:00.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:00.163 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:00.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:00.204 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:00.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:00.305 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:00.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:00.345 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:00.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:00 smithi137 ceph-mon[29408]: 6.9 scrub starts 2024-09-18T07:42:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:00 smithi137 ceph-mon[29408]: 6.9 scrub ok 2024-09-18T07:42:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:00 smithi137 ceph-mon[29408]: 4.1d scrub starts 2024-09-18T07:42:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:00 smithi137 ceph-mon[29408]: 4.1d scrub ok 2024-09-18T07:42:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:00 smithi137 ceph-mon[29408]: from='client.35849 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:42:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:00 smithi137 ceph-mon[29408]: osdmap e345: 8 total, 8 up, 8 in 2024-09-18T07:42:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:00 smithi137 ceph-mon[29408]: 9.2 scrub starts 2024-09-18T07:42:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:00 smithi137 ceph-mon[29408]: pgmap v592: 158 pgs: 158 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-18T07:42:00.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:00 smithi137 ceph-mon[39175]: 6.9 scrub starts 2024-09-18T07:42:00.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:00 smithi137 ceph-mon[39175]: 6.9 scrub ok 2024-09-18T07:42:00.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:00 smithi137 ceph-mon[39175]: 4.1d scrub starts 2024-09-18T07:42:00.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:00 smithi137 ceph-mon[39175]: 4.1d scrub ok 2024-09-18T07:42:00.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:00 smithi137 ceph-mon[39175]: from='client.35849 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:42:00.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:00 smithi137 ceph-mon[39175]: osdmap e345: 8 total, 8 up, 8 in 2024-09-18T07:42:00.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:00 smithi137 ceph-mon[39175]: 9.2 scrub starts 2024-09-18T07:42:00.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:00 smithi137 ceph-mon[39175]: pgmap v592: 158 pgs: 158 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-18T07:42:00.446 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:00.477 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:00.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:00.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:00.579 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:00.591 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-18T07:42:00.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:00.610 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:00.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:00 smithi203 ceph-mon[32237]: 6.9 scrub starts 2024-09-18T07:42:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:00 smithi203 ceph-mon[32237]: 6.9 scrub ok 2024-09-18T07:42:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:00 smithi203 ceph-mon[32237]: 4.1d scrub starts 2024-09-18T07:42:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:00 smithi203 ceph-mon[32237]: 4.1d scrub ok 2024-09-18T07:42:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:00 smithi203 ceph-mon[32237]: from='client.35849 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:42:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:00 smithi203 ceph-mon[32237]: osdmap e345: 8 total, 8 up, 8 in 2024-09-18T07:42:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:00 smithi203 ceph-mon[32237]: 9.2 scrub starts 2024-09-18T07:42:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:00 smithi203 ceph-mon[32237]: pgmap v592: 158 pgs: 158 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.05k op/s 2024-09-18T07:42:00.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:00.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:00.752 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:00.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:00.853 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:00.886 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:00.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:00.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:00.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:01.043 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:01.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:01.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:01.145 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:01.186 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:01.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:01.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:01.287 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:01.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:01.331 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:01.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:01.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:01.455 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:01.476 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:01.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:01.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:01.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:01.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:01 smithi137 ceph-mon[29408]: 6.18 scrub starts 2024-09-18T07:42:01.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:01 smithi137 ceph-mon[29408]: 6.18 scrub ok 2024-09-18T07:42:01.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:01 smithi137 ceph-mon[29408]: 2.2 scrub starts 2024-09-18T07:42:01.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:01 smithi137 ceph-mon[29408]: 2.2 scrub ok 2024-09-18T07:42:01.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:01 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:42:01.611 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:01 smithi137 ceph-mon[39175]: 6.18 scrub starts 2024-09-18T07:42:01.611 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:01 smithi137 ceph-mon[39175]: 6.18 scrub ok 2024-09-18T07:42:01.611 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:01 smithi137 ceph-mon[39175]: 2.2 scrub starts 2024-09-18T07:42:01.611 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:01 smithi137 ceph-mon[39175]: 2.2 scrub ok 2024-09-18T07:42:01.611 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:01 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:42:01.612 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:01.645 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:01.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:01.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:01 smithi203 ceph-mon[32237]: 6.18 scrub starts 2024-09-18T07:42:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:01 smithi203 ceph-mon[32237]: 6.18 scrub ok 2024-09-18T07:42:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:01 smithi203 ceph-mon[32237]: 2.2 scrub starts 2024-09-18T07:42:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:01 smithi203 ceph-mon[32237]: 2.2 scrub ok 2024-09-18T07:42:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:01 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:42:01.748 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:01.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:01.782 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:01.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:01.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:42:01.884 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:01.912 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:01.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:01.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:02.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:02.052 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:02.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:02.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:02.154 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:02.184 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:02.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:02.286 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:02.317 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:02.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:02.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:02.390 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:42:02.390 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:42:02.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:42:02.390 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:42:02.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.390 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:22340 7.b 7:dcab8849:::benchmark_data_smithi137_74180_object22339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-18T07:42:02.390 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:32:24.658442+0000", 2024-09-18T07:42:02.391 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 577.72474882899996, 2024-09-18T07:42:02.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.20618298299999999, 2024-09-18T07:42:02.391 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.391 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.391 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:02.391 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:02.391 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:02.391 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:02.391 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 22340 2024-09-18T07:42:02.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.391 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658442+0000", 2024-09-18T07:42:02.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658442+0000", 2024-09-18T07:42:02.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658445+0000", 2024-09-18T07:42:02.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4609999999999998e-06 2024-09-18T07:42:02.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658479+0000", 2024-09-18T07:42:02.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4393000000000001e-05 2024-09-18T07:42:02.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658497+0000", 2024-09-18T07:42:02.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7649999999999999e-05 2024-09-18T07:42:02.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658505+0000", 2024-09-18T07:42:02.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5350000000000004e-06 2024-09-18T07:42:02.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658567+0000", 2024-09-18T07:42:02.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1580000000000003e-05 2024-09-18T07:42:02.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:02.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658673+0000", 2024-09-18T07:42:02.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105907 2024-09-18T07:42:02.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:42:02.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.658790+0000", 2024-09-18T07:42:02.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117494 2024-09-18T07:42:02.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.660870+0000", 2024-09-18T07:42:02.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020794849999999998 2024-09-18T07:42:02.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:02.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.661103+0000", 2024-09-18T07:42:02.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023344199999999999 2024-09-18T07:42:02.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.864502+0000", 2024-09-18T07:42:02.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.20339850100000001 2024-09-18T07:42:02.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:02.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.864571+0000", 2024-09-18T07:42:02.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9566000000000004e-05 2024-09-18T07:42:02.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:32:24.864625+0000", 2024-09-18T07:42:02.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3968999999999998e-05 2024-09-18T07:42:02.397 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.397 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.397 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.397 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134973 7.15 7:ace4f2fb:::benchmark_data_smithi137_74180_object30557:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e136)", 2024-09-18T07:42:02.397 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:15.967955+0000", 2024-09-18T07:42:02.397 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 466.41523566000001, 2024-09-18T07:42:02.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.149059142, 2024-09-18T07:42:02.397 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.398 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.398 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:02.398 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:02.398 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:02.398 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:02.398 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134973 2024-09-18T07:42:02.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.398 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967955+0000", 2024-09-18T07:42:02.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967955+0000", 2024-09-18T07:42:02.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967957+0000", 2024-09-18T07:42:02.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.154e-06 2024-09-18T07:42:02.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967964+0000", 2024-09-18T07:42:02.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1310000000000003e-06 2024-09-18T07:42:02.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967976+0000", 2024-09-18T07:42:02.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2870999999999999e-05 2024-09-18T07:42:02.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967991+0000", 2024-09-18T07:42:02.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4174e-05 2024-09-18T07:42:02.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.969717+0000", 2024-09-18T07:42:02.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001726694 2024-09-18T07:42:02.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:42:02.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.969724+0000", 2024-09-18T07:42:02.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4200000000000004e-06 2024-09-18T07:42:02.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972756+0000", 2024-09-18T07:42:02.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030324610000000002 2024-09-18T07:42:02.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:02.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972766+0000", 2024-09-18T07:42:02.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0047999999999999e-05 2024-09-18T07:42:02.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976318+0000", 2024-09-18T07:42:02.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.003551327 2024-09-18T07:42:02.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:02.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976329+0000", 2024-09-18T07:42:02.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1413000000000001e-05 2024-09-18T07:42:02.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.116866+0000", 2024-09-18T07:42:02.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.140536993 2024-09-18T07:42:02.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.117014+0000", 2024-09-18T07:42:02.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014845599999999999 2024-09-18T07:42:02.403 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.403 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.403 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.404 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134975 7.15 7:a9f95f32:::benchmark_data_smithi137_74180_object30559:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e136)", 2024-09-18T07:42:02.404 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:15.968007+0000", 2024-09-18T07:42:02.404 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 466.41518395600002, 2024-09-18T07:42:02.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1497606920000001, 2024-09-18T07:42:02.404 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.404 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.404 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:02.404 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:02.404 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:02.404 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:02.404 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134975 2024-09-18T07:42:02.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.405 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968007+0000", 2024-09-18T07:42:02.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968007+0000", 2024-09-18T07:42:02.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968008+0000", 2024-09-18T07:42:02.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9100000000000003e-07 2024-09-18T07:42:02.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968011+0000", 2024-09-18T07:42:02.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0589999999999998e-06 2024-09-18T07:42:02.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968017+0000", 2024-09-18T07:42:02.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5050000000000004e-06 2024-09-18T07:42:02.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968023+0000", 2024-09-18T07:42:02.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6400000000000002e-06 2024-09-18T07:42:02.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.970885+0000", 2024-09-18T07:42:02.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002862505 2024-09-18T07:42:02.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-18T07:42:02.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.970892+0000", 2024-09-18T07:42:02.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2890000000000003e-06 2024-09-18T07:42:02.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972803+0000", 2024-09-18T07:42:02.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019118270000000001 2024-09-18T07:42:02.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:02.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972812+0000", 2024-09-18T07:42:02.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.653e-06 2024-09-18T07:42:02.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976352+0000", 2024-09-18T07:42:02.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0035398280000001 2024-09-18T07:42:02.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:02.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976359+0000", 2024-09-18T07:42:02.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3520000000000001e-06 2024-09-18T07:42:02.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.117670+0000", 2024-09-18T07:42:02.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.141310979 2024-09-18T07:42:02.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.117768+0000", 2024-09-18T07:42:02.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7363999999999998e-05 2024-09-18T07:42:02.410 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.410 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.410 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.411 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134979 7.15 7:af63a923:::benchmark_data_smithi137_74180_object30563:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e136)", 2024-09-18T07:42:02.411 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:15.968030+0000", 2024-09-18T07:42:02.411 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 466.41516087899998, 2024-09-18T07:42:02.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1501027640000001, 2024-09-18T07:42:02.411 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.411 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.411 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:02.411 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:02.411 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:02.411 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:02.411 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134979 2024-09-18T07:42:02.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.412 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968030+0000", 2024-09-18T07:42:02.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968030+0000", 2024-09-18T07:42:02.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968031+0000", 2024-09-18T07:42:02.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5000000000000003e-07 2024-09-18T07:42:02.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968033+0000", 2024-09-18T07:42:02.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1969999999999999e-06 2024-09-18T07:42:02.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968038+0000", 2024-09-18T07:42:02.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0919999999999997e-06 2024-09-18T07:42:02.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968043+0000", 2024-09-18T07:42:02.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6619999999999996e-06 2024-09-18T07:42:02.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.970922+0000", 2024-09-18T07:42:02.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028795100000000001 2024-09-18T07:42:02.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-18T07:42:02.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.970927+0000", 2024-09-18T07:42:02.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0320000000000003e-06 2024-09-18T07:42:02.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972841+0000", 2024-09-18T07:42:02.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001913802 2024-09-18T07:42:02.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:02.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972849+0000", 2024-09-18T07:42:02.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9039999999999992e-06 2024-09-18T07:42:02.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976382+0000", 2024-09-18T07:42:02.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0035336779999999 2024-09-18T07:42:02.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:02.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976390+0000", 2024-09-18T07:42:02.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6399999999999997e-06 2024-09-18T07:42:02.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.118071+0000", 2024-09-18T07:42:02.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.141680993 2024-09-18T07:42:02.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.118133+0000", 2024-09-18T07:42:02.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1704000000000004e-05 2024-09-18T07:42:02.417 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.417 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.417 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.417 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134988 7.12 7:4ba1d377:::benchmark_data_smithi137_74180_object30572:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:42:02.417 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964095+0000", 2024-09-18T07:42:02.417 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 465.419096405, 2024-09-18T07:42:02.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2226311409999999, 2024-09-18T07:42:02.418 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.418 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.418 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:02.418 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:02.418 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:02.418 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:02.418 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134988 2024-09-18T07:42:02.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.418 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964095+0000", 2024-09-18T07:42:02.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964095+0000", 2024-09-18T07:42:02.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964096+0000", 2024-09-18T07:42:02.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.759e-06 2024-09-18T07:42:02.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.419 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:02.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.421 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.421 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964103+0000", 2024-09-18T07:42:02.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7950000000000002e-06 2024-09-18T07:42:02.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.421 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.421 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964118+0000", 2024-09-18T07:42:02.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5082999999999999e-05 2024-09-18T07:42:02.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.421 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.421 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964128+0000", 2024-09-18T07:42:02.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6900000000000004e-06 2024-09-18T07:42:02.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964200+0000", 2024-09-18T07:42:02.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2656000000000004e-05 2024-09-18T07:42:02.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:42:02.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964207+0000", 2024-09-18T07:42:02.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5320000000000003e-06 2024-09-18T07:42:02.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976866+0000", 2024-09-18T07:42:02.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012658763 2024-09-18T07:42:02.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:02.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976874+0000", 2024-09-18T07:42:02.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6459999999999999e-06 2024-09-18T07:42:02.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.174506+0000", 2024-09-18T07:42:02.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1976311799999999 2024-09-18T07:42:02.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:02.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.174658+0000", 2024-09-18T07:42:02.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000152516 2024-09-18T07:42:02.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,3,6", 2024-09-18T07:42:02.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.174933+0000", 2024-09-18T07:42:02.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027465900000000001 2024-09-18T07:42:02.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184279+0000", 2024-09-18T07:42:02.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093465449999999995 2024-09-18T07:42:02.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184325+0000", 2024-09-18T07:42:02.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5219999999999997e-05 2024-09-18T07:42:02.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184838+0000", 2024-09-18T07:42:02.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00051383799999999997 2024-09-18T07:42:02.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:02.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185572+0000", 2024-09-18T07:42:02.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00073320099999999995 2024-09-18T07:42:02.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186540+0000", 2024-09-18T07:42:02.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00096810500000000003 2024-09-18T07:42:02.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:02.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186603+0000", 2024-09-18T07:42:02.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3411000000000007e-05 2024-09-18T07:42:02.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186726+0000", 2024-09-18T07:42:02.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012254200000000001 2024-09-18T07:42:02.427 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.427 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.427 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.427 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134989 7.12 7:4aa85d7f:::benchmark_data_smithi137_74180_object30573:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:42:02.427 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964171+0000", 2024-09-18T07:42:02.427 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 465.419020128, 2024-09-18T07:42:02.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.222781653, 2024-09-18T07:42:02.428 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.428 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.428 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:02.428 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:02.428 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:02.428 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:02.428 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134989 2024-09-18T07:42:02.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.428 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964171+0000", 2024-09-18T07:42:02.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964171+0000", 2024-09-18T07:42:02.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964172+0000", 2024-09-18T07:42:02.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6899999999999996e-07 2024-09-18T07:42:02.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964176+0000", 2024-09-18T07:42:02.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1829999999999996e-06 2024-09-18T07:42:02.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964184+0000", 2024-09-18T07:42:02.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8290000000000008e-06 2024-09-18T07:42:02.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964190+0000", 2024-09-18T07:42:02.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8850000000000004e-06 2024-09-18T07:42:02.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964252+0000", 2024-09-18T07:42:02.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2842000000000006e-05 2024-09-18T07:42:02.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-18T07:42:02.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964257+0000", 2024-09-18T07:42:02.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4379999999999997e-06 2024-09-18T07:42:02.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976922+0000", 2024-09-18T07:42:02.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012665493 2024-09-18T07:42:02.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:02.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976933+0000", 2024-09-18T07:42:02.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0407999999999999e-05 2024-09-18T07:42:02.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.176181+0000", 2024-09-18T07:42:02.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1992481429999999 2024-09-18T07:42:02.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:02.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.176304+0000", 2024-09-18T07:42:02.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012344999999999999 2024-09-18T07:42:02.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,3,6", 2024-09-18T07:42:02.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.176507+0000", 2024-09-18T07:42:02.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020292699999999999 2024-09-18T07:42:02.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184373+0000", 2024-09-18T07:42:02.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0078657629999999996 2024-09-18T07:42:02.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184427+0000", 2024-09-18T07:42:02.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3863999999999998e-05 2024-09-18T07:42:02.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184878+0000", 2024-09-18T07:42:02.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00045142400000000002 2024-09-18T07:42:02.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:02.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185578+0000", 2024-09-18T07:42:02.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00069919800000000005 2024-09-18T07:42:02.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186770+0000", 2024-09-18T07:42:02.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011924430000000001 2024-09-18T07:42:02.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:02.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186830+0000", 2024-09-18T07:42:02.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0003000000000001e-05 2024-09-18T07:42:02.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186952+0000", 2024-09-18T07:42:02.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012239099999999999 2024-09-18T07:42:02.435 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.435 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.436 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.436 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135012 7.12 7:4b131569:::benchmark_data_smithi137_74180_object30596:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:42:02.436 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964206+0000", 2024-09-18T07:42:02.436 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 465.41898511699998, 2024-09-18T07:42:02.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2234539019999999, 2024-09-18T07:42:02.436 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.436 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.436 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:02.436 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:02.437 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:02.437 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:02.437 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135012 2024-09-18T07:42:02.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.437 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964206+0000", 2024-09-18T07:42:02.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964206+0000", 2024-09-18T07:42:02.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964207+0000", 2024-09-18T07:42:02.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6899999999999996e-07 2024-09-18T07:42:02.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964210+0000", 2024-09-18T07:42:02.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4419999999999998e-06 2024-09-18T07:42:02.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964217+0000", 2024-09-18T07:42:02.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4989999999999999e-06 2024-09-18T07:42:02.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964222+0000", 2024-09-18T07:42:02.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5670000000000002e-06 2024-09-18T07:42:02.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964287+0000", 2024-09-18T07:42:02.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5174999999999999e-05 2024-09-18T07:42:02.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-18T07:42:02.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964293+0000", 2024-09-18T07:42:02.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7209999999999999e-06 2024-09-18T07:42:02.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976971+0000", 2024-09-18T07:42:02.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012677541000000001 2024-09-18T07:42:02.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:02.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976991+0000", 2024-09-18T07:42:02.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0231999999999999e-05 2024-09-18T07:42:02.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.177610+0000", 2024-09-18T07:42:02.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.200619417 2024-09-18T07:42:02.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:02.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.177725+0000", 2024-09-18T07:42:02.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114815 2024-09-18T07:42:02.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,3,6", 2024-09-18T07:42:02.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.177908+0000", 2024-09-18T07:42:02.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000182683 2024-09-18T07:42:02.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184585+0000", 2024-09-18T07:42:02.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066772230000000004 2024-09-18T07:42:02.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184630+0000", 2024-09-18T07:42:02.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4642999999999999e-05 2024-09-18T07:42:02.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185558+0000", 2024-09-18T07:42:02.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00092809900000000005 2024-09-18T07:42:02.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.187003+0000", 2024-09-18T07:42:02.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001445483 2024-09-18T07:42:02.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:02.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.187470+0000", 2024-09-18T07:42:02.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00046721600000000001 2024-09-18T07:42:02.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:02.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.187517+0000", 2024-09-18T07:42:02.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6619000000000002e-05 2024-09-18T07:42:02.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.187660+0000", 2024-09-18T07:42:02.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014275799999999999 2024-09-18T07:42:02.445 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.445 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.445 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.445 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:142960 7.12 7:4c4afa90:::benchmark_data_smithi137_74180_object38544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-18T07:42:02.445 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:25.433818+0000", 2024-09-18T07:42:02.445 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 456.94937323599999, 2024-09-18T07:42:02.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.185941245, 2024-09-18T07:42:02.446 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.446 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.446 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:02.446 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:02.446 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:02.446 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:02.446 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 142960 2024-09-18T07:42:02.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.446 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433818+0000", 2024-09-18T07:42:02.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433818+0000", 2024-09-18T07:42:02.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433820+0000", 2024-09-18T07:42:02.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3290000000000001e-06 2024-09-18T07:42:02.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433825+0000", 2024-09-18T07:42:02.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4589999999999997e-06 2024-09-18T07:42:02.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433837+0000", 2024-09-18T07:42:02.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1229999999999999e-05 2024-09-18T07:42:02.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433846+0000", 2024-09-18T07:42:02.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2790000000000004e-06 2024-09-18T07:42:02.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433902+0000", 2024-09-18T07:42:02.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5847999999999998e-05 2024-09-18T07:42:02.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:02.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.434031+0000", 2024-09-18T07:42:02.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012924099999999999 2024-09-18T07:42:02.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,3,6", 2024-09-18T07:42:02.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.434299+0000", 2024-09-18T07:42:02.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00026747400000000002 2024-09-18T07:42:02.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.451 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.436066+0000", 2024-09-18T07:42:02.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017675519999999999 2024-09-18T07:42:02.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.451 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:02.451 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.436687+0000", 2024-09-18T07:42:02.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00062042 2024-09-18T07:42:02.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.451 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.451 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.437458+0000", 2024-09-18T07:42:02.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00077107399999999996 2024-09-18T07:42:02.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.437498+0000", 2024-09-18T07:42:02.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0395999999999997e-05 2024-09-18T07:42:02.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:02.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.619551+0000", 2024-09-18T07:42:02.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.182052674 2024-09-18T07:42:02.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:02.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.619612+0000", 2024-09-18T07:42:02.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1678000000000001e-05 2024-09-18T07:42:02.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.619759+0000", 2024-09-18T07:42:02.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014659100000000001 2024-09-18T07:42:02.453 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.453 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.453 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.453 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:156447 9.0 e330/289 9:089f7411:::benchmark_data_smithi137_126072_object23947:head v 330'9825, mlcod=330'9823)", 2024-09-18T07:42:02.453 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:24.395752+0000", 2024-09-18T07:42:02.454 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 37.987438644000001, 2024-09-18T07:42:02.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.158938625, 2024-09-18T07:42:02.454 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.454 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.454 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:02.454 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.395752+0000", 2024-09-18T07:42:02.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.395752+0000", 2024-09-18T07:42:02.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.395753+0000", 2024-09-18T07:42:02.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.226e-06 2024-09-18T07:42:02.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.395760+0000", 2024-09-18T07:42:02.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8569999999999997e-06 2024-09-18T07:42:02.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.395769+0000", 2024-09-18T07:42:02.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3420000000000008e-06 2024-09-18T07:42:02.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.395774+0000", 2024-09-18T07:42:02.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7259999999999999e-06 2024-09-18T07:42:02.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.395894+0000", 2024-09-18T07:42:02.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000119629 2024-09-18T07:42:02.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:02.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.395911+0000", 2024-09-18T07:42:02.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7328000000000001e-05 2024-09-18T07:42:02.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:02.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.401097+0000", 2024-09-18T07:42:02.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051856589999999996 2024-09-18T07:42:02.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.554691+0000", 2024-09-18T07:42:02.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.153593858 2024-09-18T07:42:02.457 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.458 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.458 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.458 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:156455 9.0 e330/289 9:03da8d0f:::benchmark_data_smithi137_126072_object23955:head v 330'9826, mlcod=330'9823)", 2024-09-18T07:42:02.458 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:24.401729+0000", 2024-09-18T07:42:02.458 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 37.981461828999997, 2024-09-18T07:42:02.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15955804000000001, 2024-09-18T07:42:02.458 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.458 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.458 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:02.458 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.458 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.459 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.401729+0000", 2024-09-18T07:42:02.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.459 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.401729+0000", 2024-09-18T07:42:02.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.459 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.401730+0000", 2024-09-18T07:42:02.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1939999999999999e-06 2024-09-18T07:42:02.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.401737+0000", 2024-09-18T07:42:02.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8059999999999997e-06 2024-09-18T07:42:02.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.554669+0000", 2024-09-18T07:42:02.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15293205600000001 2024-09-18T07:42:02.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.554679+0000", 2024-09-18T07:42:02.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9809999999999998e-06 2024-09-18T07:42:02.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.554722+0000", 2024-09-18T07:42:02.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2656e-05 2024-09-18T07:42:02.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:02.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.554743+0000", 2024-09-18T07:42:02.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1447e-05 2024-09-18T07:42:02.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:02.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.561254+0000", 2024-09-18T07:42:02.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065112629999999998 2024-09-18T07:42:02.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.561287+0000", 2024-09-18T07:42:02.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2636999999999999e-05 2024-09-18T07:42:02.462 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.462 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.462 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.462 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:156460 9.4 e330/289 9:2f71b420:::benchmark_data_smithi137_126072_object23960:head v 330'9781, mlcod=330'9779)", 2024-09-18T07:42:02.463 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:24.401875+0000", 2024-09-18T07:42:02.463 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 37.981316071000002, 2024-09-18T07:42:02.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15930688000000001, 2024-09-18T07:42:02.463 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.463 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.463 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:02.463 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.401875+0000", 2024-09-18T07:42:02.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.401875+0000", 2024-09-18T07:42:02.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.401876+0000", 2024-09-18T07:42:02.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.201e-06 2024-09-18T07:42:02.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.401883+0000", 2024-09-18T07:42:02.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5910000000000001e-06 2024-09-18T07:42:02.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.465 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.554863+0000", 2024-09-18T07:42:02.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15298060799999999 2024-09-18T07:42:02.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.465 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.554871+0000", 2024-09-18T07:42:02.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0819999999999999e-06 2024-09-18T07:42:02.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.465 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.555028+0000", 2024-09-18T07:42:02.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015654800000000001 2024-09-18T07:42:02.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:02.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.555049+0000", 2024-09-18T07:42:02.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0727e-05 2024-09-18T07:42:02.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:02.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.561140+0000", 2024-09-18T07:42:02.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0060913440000000003 2024-09-18T07:42:02.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:24.561182+0000", 2024-09-18T07:42:02.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1779000000000001e-05 2024-09-18T07:42:02.467 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.467 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.467 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.467 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172965 9.e e335/289 9:7fd9b0fe:::benchmark_data_smithi137_126072_object40465:head v 335'10641, mlcod=335'10639)", 2024-09-18T07:42:02.467 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.554501+0000", 2024-09-18T07:42:02.467 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 23.828690090999999, 2024-09-18T07:42:02.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.178309934, 2024-09-18T07:42:02.467 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.467 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.467 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:02.468 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554501+0000", 2024-09-18T07:42:02.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554501+0000", 2024-09-18T07:42:02.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554502+0000", 2024-09-18T07:42:02.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.443e-06 2024-09-18T07:42:02.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554510+0000", 2024-09-18T07:42:02.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0660000000000004e-06 2024-09-18T07:42:02.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554518+0000", 2024-09-18T07:42:02.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0339999999999997e-06 2024-09-18T07:42:02.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554523+0000", 2024-09-18T07:42:02.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8740000000000003e-06 2024-09-18T07:42:02.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.470 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554567+0000", 2024-09-18T07:42:02.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3294000000000003e-05 2024-09-18T07:42:02.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.470 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:02.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554586+0000", 2024-09-18T07:42:02.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9419000000000002e-05 2024-09-18T07:42:02.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:02.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732782+0000", 2024-09-18T07:42:02.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.178196153 2024-09-18T07:42:02.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732811+0000", 2024-09-18T07:42:02.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8651000000000001e-05 2024-09-18T07:42:02.471 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.471 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.471 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.472 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172968 9.0 e335/289 9:02c69111:::benchmark_data_smithi137_126072_object40468:head v 335'10807, mlcod=335'10804)", 2024-09-18T07:42:02.472 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.554702+0000", 2024-09-18T07:42:02.472 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 23.828488744000001, 2024-09-18T07:42:02.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.179882141, 2024-09-18T07:42:02.472 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.472 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.472 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:02.472 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554702+0000", 2024-09-18T07:42:02.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554702+0000", 2024-09-18T07:42:02.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554704+0000", 2024-09-18T07:42:02.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.359e-06 2024-09-18T07:42:02.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554711+0000", 2024-09-18T07:42:02.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.199e-06 2024-09-18T07:42:02.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554720+0000", 2024-09-18T07:42:02.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5629999999999998e-06 2024-09-18T07:42:02.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554727+0000", 2024-09-18T07:42:02.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5889999999999999e-06 2024-09-18T07:42:02.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554767+0000", 2024-09-18T07:42:02.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9712000000000002e-05 2024-09-18T07:42:02.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:02.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554782+0000", 2024-09-18T07:42:02.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5639000000000001e-05 2024-09-18T07:42:02.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:02.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734508+0000", 2024-09-18T07:42:02.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17972560900000001 2024-09-18T07:42:02.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734584+0000", 2024-09-18T07:42:02.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6470999999999993e-05 2024-09-18T07:42:02.476 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.476 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.476 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.476 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172969 9.6 e335/289 9:60d9b40c:::benchmark_data_smithi137_126072_object40469:head v 335'10756, mlcod=335'10754)", 2024-09-18T07:42:02.476 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.554970+0000", 2024-09-18T07:42:02.476 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 23.828221407000001, 2024-09-18T07:42:02.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17787228899999999, 2024-09-18T07:42:02.476 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.477 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.477 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:02.477 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554970+0000", 2024-09-18T07:42:02.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554970+0000", 2024-09-18T07:42:02.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554971+0000", 2024-09-18T07:42:02.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0070000000000001e-06 2024-09-18T07:42:02.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554976+0000", 2024-09-18T07:42:02.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6330000000000001e-06 2024-09-18T07:42:02.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554993+0000", 2024-09-18T07:42:02.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6949e-05 2024-09-18T07:42:02.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.555000+0000", 2024-09-18T07:42:02.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5579999999999997e-06 2024-09-18T07:42:02.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.555176+0000", 2024-09-18T07:42:02.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017662999999999999 2024-09-18T07:42:02.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:02.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.555193+0000", 2024-09-18T07:42:02.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7212000000000001e-05 2024-09-18T07:42:02.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:02.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732814+0000", 2024-09-18T07:42:02.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.177620315 2024-09-18T07:42:02.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732842+0000", 2024-09-18T07:42:02.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7985000000000001e-05 2024-09-18T07:42:02.480 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.481 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.481 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.481 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172970 9.3 e335/289 9:c7e8a587:::benchmark_data_smithi137_126072_object40470:head v 335'10785, mlcod=335'10782)", 2024-09-18T07:42:02.481 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.557577+0000", 2024-09-18T07:42:02.481 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 23.825613486999998, 2024-09-18T07:42:02.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.173544268, 2024-09-18T07:42:02.481 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.481 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.481 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:02.481 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557577+0000", 2024-09-18T07:42:02.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557577+0000", 2024-09-18T07:42:02.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557579+0000", 2024-09-18T07:42:02.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5099999999999999e-06 2024-09-18T07:42:02.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557589+0000", 2024-09-18T07:42:02.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8079999999999999e-06 2024-09-18T07:42:02.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557598+0000", 2024-09-18T07:42:02.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2490000000000007e-06 2024-09-18T07:42:02.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557604+0000", 2024-09-18T07:42:02.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2079999999999997e-06 2024-09-18T07:42:02.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726578+0000", 2024-09-18T07:42:02.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.168973331 2024-09-18T07:42:02.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:02.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726596+0000", 2024-09-18T07:42:02.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8159e-05 2024-09-18T07:42:02.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:02.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731084+0000", 2024-09-18T07:42:02.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044883980000000002 2024-09-18T07:42:02.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731122+0000", 2024-09-18T07:42:02.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7605000000000003e-05 2024-09-18T07:42:02.485 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.485 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.485 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.486 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172972 9.e e335/289 9:796b9798:::benchmark_data_smithi137_126072_object40472:head v 335'10642, mlcod=335'10639)", 2024-09-18T07:42:02.486 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.557623+0000", 2024-09-18T07:42:02.486 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 23.825567628000002, 2024-09-18T07:42:02.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.175241702, 2024-09-18T07:42:02.486 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.486 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.486 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:02.486 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557623+0000", 2024-09-18T07:42:02.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557623+0000", 2024-09-18T07:42:02.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557624+0000", 2024-09-18T07:42:02.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4200000000000001e-07 2024-09-18T07:42:02.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557628+0000", 2024-09-18T07:42:02.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8139999999999998e-06 2024-09-18T07:42:02.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557632+0000", 2024-09-18T07:42:02.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6909999999999996e-06 2024-09-18T07:42:02.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557636+0000", 2024-09-18T07:42:02.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6629999999999999e-06 2024-09-18T07:42:02.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726230+0000", 2024-09-18T07:42:02.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.168594563 2024-09-18T07:42:02.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:02.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726286+0000", 2024-09-18T07:42:02.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5702000000000003e-05 2024-09-18T07:42:02.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:02.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732845+0000", 2024-09-18T07:42:02.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065588560000000001 2024-09-18T07:42:02.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732865+0000", 2024-09-18T07:42:02.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9970999999999999e-05 2024-09-18T07:42:02.490 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.490 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.490 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.490 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172971 9.4 e335/289 9:2cb28e72:::benchmark_data_smithi137_126072_object40471:head v 335'10859, mlcod=335'10857)", 2024-09-18T07:42:02.490 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.557654+0000", 2024-09-18T07:42:02.490 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 23.825536602, 2024-09-18T07:42:02.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17145278799999999, 2024-09-18T07:42:02.490 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.490 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.490 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:02.490 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557654+0000", 2024-09-18T07:42:02.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557654+0000", 2024-09-18T07:42:02.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557655+0000", 2024-09-18T07:42:02.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1099999999999995e-07 2024-09-18T07:42:02.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557661+0000", 2024-09-18T07:42:02.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2400000000000004e-06 2024-09-18T07:42:02.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557667+0000", 2024-09-18T07:42:02.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9139999999999996e-06 2024-09-18T07:42:02.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557672+0000", 2024-09-18T07:42:02.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6290000000000001e-06 2024-09-18T07:42:02.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.494 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[29408]: 9.2 scrub ok 2024-09-18T07:42:02.494 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[29408]: 6.16 scrub starts 2024-09-18T07:42:02.494 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[29408]: 6.16 scrub ok 2024-09-18T07:42:02.494 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.494 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.494 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.494 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.494 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.494 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.494 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.494 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.495 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[29408]: Adjusting osd_memory_target on smithi203 to 3248M 2024-09-18T07:42:02.495 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:42:02.495 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[29408]: pgmap v593: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-09-18T07:42:02.495 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[39175]: 9.2 scrub ok 2024-09-18T07:42:02.495 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[39175]: 6.16 scrub starts 2024-09-18T07:42:02.495 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[39175]: 6.16 scrub ok 2024-09-18T07:42:02.495 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.495 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.495 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.495 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.496 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.496 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.496 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.496 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.496 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[39175]: Adjusting osd_memory_target on smithi203 to 3248M 2024-09-18T07:42:02.496 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:42:02.496 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:02 smithi137 ceph-mon[39175]: pgmap v593: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-09-18T07:42:02.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726169+0000", 2024-09-18T07:42:02.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.168497014 2024-09-18T07:42:02.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:02.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726237+0000", 2024-09-18T07:42:02.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8677999999999995e-05 2024-09-18T07:42:02.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:02.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.729066+0000", 2024-09-18T07:42:02.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028290170000000001 2024-09-18T07:42:02.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.729107+0000", 2024-09-18T07:42:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0685000000000001e-05 2024-09-18T07:42:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172976 9.f e335/289 9:f0577982:::benchmark_data_smithi137_126072_object40476:head v 335'10844, mlcod=335'10842)", 2024-09-18T07:42:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.560675+0000", 2024-09-18T07:42:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 23.822515497000001, 2024-09-18T07:42:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17558478, 2024-09-18T07:42:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.499 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:02.499 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.499 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560675+0000", 2024-09-18T07:42:02.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.499 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560675+0000", 2024-09-18T07:42:02.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560677+0000", 2024-09-18T07:42:02.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.711e-06 2024-09-18T07:42:02.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560685+0000", 2024-09-18T07:42:02.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1880000000000002e-06 2024-09-18T07:42:02.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726219+0000", 2024-09-18T07:42:02.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16553400300000001 2024-09-18T07:42:02.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726233+0000", 2024-09-18T07:42:02.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3893000000000001e-05 2024-09-18T07:42:02.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726381+0000", 2024-09-18T07:42:02.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014760199999999999 2024-09-18T07:42:02.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:02.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726402+0000", 2024-09-18T07:42:02.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1664000000000001e-05 2024-09-18T07:42:02.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:02.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.736225+0000", 2024-09-18T07:42:02.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0098227820000000004 2024-09-18T07:42:02.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.736260+0000", 2024-09-18T07:42:02.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4937000000000001e-05 2024-09-18T07:42:02.502 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.502 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.502 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172980 9.3 e335/289 9:ce0a7506:::benchmark_data_smithi137_126072_object40480:head v 335'10786, mlcod=335'10783)", 2024-09-18T07:42:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.560718+0000", 2024-09-18T07:42:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 23.822473025000001, 2024-09-18T07:42:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17056611599999999, 2024-09-18T07:42:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560718+0000", 2024-09-18T07:42:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.504 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.504 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560718+0000", 2024-09-18T07:42:02.504 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.504 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.504 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.504 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560719+0000", 2024-09-18T07:42:02.504 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8400000000000003e-07 2024-09-18T07:42:02.504 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.504 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.504 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560726+0000", 2024-09-18T07:42:02.505 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4000000000000003e-06 2024-09-18T07:42:02.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.505 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.505 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.505 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726135+0000", 2024-09-18T07:42:02.505 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16540897700000001 2024-09-18T07:42:02.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.505 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.505 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.505 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726166+0000", 2024-09-18T07:42:02.505 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.099e-05 2024-09-18T07:42:02.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.505 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.728991+0000", 2024-09-18T07:42:02.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028248840000000002 2024-09-18T07:42:02.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:02.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.729009+0000", 2024-09-18T07:42:02.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7804000000000001e-05 2024-09-18T07:42:02.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:02.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731249+0000", 2024-09-18T07:42:02.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022400050000000002 2024-09-18T07:42:02.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.507 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.507 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731284+0000", 2024-09-18T07:42:02.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5271999999999997e-05 2024-09-18T07:42:02.507 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.507 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.507 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.507 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172984 9.0 e335/289 9:066846aa:::benchmark_data_smithi137_126072_object40484:head v 335'10808, mlcod=335'10805)", 2024-09-18T07:42:02.507 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.567459+0000", 2024-09-18T07:42:02.507 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 23.815731930999998, 2024-09-18T07:42:02.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16718034100000001, 2024-09-18T07:42:02.507 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:02.508 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:02.508 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:02.508 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:02.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.508 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:02.508 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567459+0000", 2024-09-18T07:42:02.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.509 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:02.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567459+0000", 2024-09-18T07:42:02.509 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:02.509 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.509 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.509 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:02.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567461+0000", 2024-09-18T07:42:02.509 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1550000000000001e-06 2024-09-18T07:42:02.509 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.509 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.509 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:02.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567471+0000", 2024-09-18T07:42:02.509 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0348e-05 2024-09-18T07:42:02.509 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.510 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:02.510 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726150+0000", 2024-09-18T07:42:02.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15867869100000001 2024-09-18T07:42:02.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.510 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:02.510 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726175+0000", 2024-09-18T07:42:02.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5316e-05 2024-09-18T07:42:02.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.510 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:02.510 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.728757+0000", 2024-09-18T07:42:02.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002581262 2024-09-18T07:42:02.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:02.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.728778+0000", 2024-09-18T07:42:02.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1739999999999999e-05 2024-09-18T07:42:02.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:02.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734588+0000", 2024-09-18T07:42:02.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005809519 2024-09-18T07:42:02.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:02.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:02.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:02.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734639+0000", 2024-09-18T07:42:02.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1310000000000002e-05 2024-09-18T07:42:02.512 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.512 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.512 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.512 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:02.512 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:02.512 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:42:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:02.513 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:02.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:02.614 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:02.652 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:02.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:02.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:02 smithi203 ceph-mon[32237]: 9.2 scrub ok 2024-09-18T07:42:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:02 smithi203 ceph-mon[32237]: 6.16 scrub starts 2024-09-18T07:42:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:02 smithi203 ceph-mon[32237]: 6.16 scrub ok 2024-09-18T07:42:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:02 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:02 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:02 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:02 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:02 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:02 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:02 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:02 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:02.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:02 smithi203 ceph-mon[32237]: Adjusting osd_memory_target on smithi203 to 3248M 2024-09-18T07:42:02.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:02 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:42:02.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:02 smithi203 ceph-mon[32237]: pgmap v593: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-09-18T07:42:02.754 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:02.789 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:02.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:02.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:02.919 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:02.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:02.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:03.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:03.055 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:03.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:03.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:03.157 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:03.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:03.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:03.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:03.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:03.318 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:03.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:03.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:03.393 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:42:03.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:03 smithi137 ceph-mon[29408]: 3.0 scrub starts 2024-09-18T07:42:03.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:03 smithi137 ceph-mon[29408]: 3.0 scrub ok 2024-09-18T07:42:03.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:03 smithi137 ceph-mon[29408]: 5.7 scrub starts 2024-09-18T07:42:03.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:03 smithi137 ceph-mon[29408]: 5.7 scrub ok 2024-09-18T07:42:03.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:03 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3799678575' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:42:03.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:03 smithi137 ceph-mon[29408]: from='client.35855 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:42:03.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:03 smithi137 ceph-mon[39175]: 3.0 scrub starts 2024-09-18T07:42:03.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:03 smithi137 ceph-mon[39175]: 3.0 scrub ok 2024-09-18T07:42:03.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:03 smithi137 ceph-mon[39175]: 5.7 scrub starts 2024-09-18T07:42:03.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:03 smithi137 ceph-mon[39175]: 5.7 scrub ok 2024-09-18T07:42:03.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:03 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3799678575' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:42:03.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:03 smithi137 ceph-mon[39175]: from='client.35855 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:42:03.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:03.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:03.467 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:03.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:03.568 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:03.598 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:03.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:03.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:03.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:03.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:03 smithi203 ceph-mon[32237]: 3.0 scrub starts 2024-09-18T07:42:03.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:03 smithi203 ceph-mon[32237]: 3.0 scrub ok 2024-09-18T07:42:03.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:03 smithi203 ceph-mon[32237]: 5.7 scrub starts 2024-09-18T07:42:03.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:03 smithi203 ceph-mon[32237]: 5.7 scrub ok 2024-09-18T07:42:03.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:03 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3799678575' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:42:03.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:03 smithi203 ceph-mon[32237]: from='client.35855 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:42:03.751 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:03.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:03.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:03.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:03.890 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:03.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:03.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:03.992 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:04.030 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:04.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:04.131 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:04.166 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:04.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:04.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:04.268 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:04.304 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:04.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:04.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:04.406 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:04.444 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:04.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:04.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:04.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T07:42:04.546 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:04.582 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:04.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:04.633 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.5 to 1 (10000) 2024-09-18T07:42:04.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:04.738 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:04.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:04.839 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:04.871 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:04.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:04.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:04.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[29408]: 3.3 scrub starts 2024-09-18T07:42:04.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[29408]: 3.3 scrub ok 2024-09-18T07:42:04.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[29408]: 4.2 scrub starts 2024-09-18T07:42:04.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[29408]: 4.2 scrub ok 2024-09-18T07:42:04.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[29408]: from='client.35855 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:42:04.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[29408]: osdmap e346: 8 total, 8 up, 8 in 2024-09-18T07:42:04.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[29408]: pgmap v595: 158 pgs: 158 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-18T07:42:04.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[29408]: 9.e scrub starts 2024-09-18T07:42:04.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[29408]: 5.b scrub starts 2024-09-18T07:42:04.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[29408]: 5.b scrub ok 2024-09-18T07:42:04.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[29408]: 5.0 scrub starts 2024-09-18T07:42:04.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[29408]: 5.0 scrub ok 2024-09-18T07:42:04.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[39175]: 3.3 scrub starts 2024-09-18T07:42:04.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[39175]: 3.3 scrub ok 2024-09-18T07:42:04.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[39175]: 4.2 scrub starts 2024-09-18T07:42:04.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[39175]: 4.2 scrub ok 2024-09-18T07:42:04.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[39175]: from='client.35855 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:42:04.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[39175]: osdmap e346: 8 total, 8 up, 8 in 2024-09-18T07:42:04.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[39175]: pgmap v595: 158 pgs: 158 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-18T07:42:04.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[39175]: 9.e scrub starts 2024-09-18T07:42:04.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[39175]: 5.b scrub starts 2024-09-18T07:42:04.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[39175]: 5.b scrub ok 2024-09-18T07:42:04.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[39175]: 5.0 scrub starts 2024-09-18T07:42:04.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:04 smithi137 ceph-mon[39175]: 5.0 scrub ok 2024-09-18T07:42:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:04 smithi203 ceph-mon[32237]: 3.3 scrub starts 2024-09-18T07:42:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:04 smithi203 ceph-mon[32237]: 3.3 scrub ok 2024-09-18T07:42:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:04 smithi203 ceph-mon[32237]: 4.2 scrub starts 2024-09-18T07:42:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:04 smithi203 ceph-mon[32237]: 4.2 scrub ok 2024-09-18T07:42:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:04 smithi203 ceph-mon[32237]: from='client.35855 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:42:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:04 smithi203 ceph-mon[32237]: osdmap e346: 8 total, 8 up, 8 in 2024-09-18T07:42:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:04 smithi203 ceph-mon[32237]: pgmap v595: 158 pgs: 158 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-18T07:42:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:04 smithi203 ceph-mon[32237]: 9.e scrub starts 2024-09-18T07:42:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:04 smithi203 ceph-mon[32237]: 5.b scrub starts 2024-09-18T07:42:04.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:04 smithi203 ceph-mon[32237]: 5.b scrub ok 2024-09-18T07:42:04.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:04 smithi203 ceph-mon[32237]: 5.0 scrub starts 2024-09-18T07:42:04.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:04 smithi203 ceph-mon[32237]: 5.0 scrub ok 2024-09-18T07:42:04.973 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:05.008 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:05.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:05.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:05.110 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:05.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:05.151 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:05.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:05.252 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:05.287 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:05.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:05.388 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:05.440 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:05.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:05.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-18T07:42:05.468 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:05.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:05.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:05.570 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:05.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:05.671 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:05.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:05.716 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:05.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:05.817 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:05.845 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:05.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:05.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:05.947 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:05.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: 3.1b scrub starts 2024-09-18T07:42:05.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: 3.1b scrub ok 2024-09-18T07:42:05.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: 3.13 scrub starts 2024-09-18T07:42:05.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: 3.13 scrub ok 2024-09-18T07:42:05.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: 9.e scrub ok 2024-09-18T07:42:05.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: Adjusting osd_memory_target on smithi137 to 3248M 2024-09-18T07:42:05.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:42:05.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:42:05.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:42:05.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:05 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:42:05.973 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: 3.1b scrub starts 2024-09-18T07:42:05.973 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: 3.1b scrub ok 2024-09-18T07:42:05.973 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: 3.13 scrub starts 2024-09-18T07:42:05.973 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: 3.13 scrub ok 2024-09-18T07:42:05.973 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: 9.e scrub ok 2024-09-18T07:42:05.973 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.973 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.973 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: Adjusting osd_memory_target on smithi137 to 3248M 2024-09-18T07:42:05.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:42:05.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:42:05.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:42:05.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:42:05.974 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: 3.1b scrub starts 2024-09-18T07:42:05.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: 3.1b scrub ok 2024-09-18T07:42:05.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: 3.13 scrub starts 2024-09-18T07:42:05.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: 3.13 scrub ok 2024-09-18T07:42:05.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: 9.e scrub ok 2024-09-18T07:42:05.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-18T07:42:05.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: Adjusting osd_memory_target on smithi137 to 3248M 2024-09-18T07:42:05.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:42:05.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:42:05.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:42:05.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:05 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:42:05.977 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:05.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:05.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:06.079 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:06.106 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:06.107 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:06.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:06.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:06.134 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:42:06.135 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:42:06.152 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:42:06.152 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:42:06.168 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:42:06.168 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:42:06.184 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:42:06.184 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:42:06.203 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:42:06.203 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:42:06.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:06.235 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:42:06.236 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:42:06.241 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:06.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:06.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:06.268 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:42:06.269 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:42:06.316 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:42:06.316 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:42:06.344 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:06.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:06.374 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:06.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:06.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:06.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:06.511 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:06.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:06.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:06.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:06.646 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:06.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:06.749 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:06.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:06.785 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:06.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:06.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:06.889 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:42:06.889 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:42:06.919 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:06.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:06.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:07.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:07.079 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:07.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:07.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:07.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:07.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:07.234 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:07.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:07.335 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:07.360 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:07.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:07.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:07.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:42:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:42:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:42:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:06 smithi137 ceph-mon[29408]: pgmap v596: 158 pgs: 158 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 689 B/s rd, 0 B/s wr, 824 op/s 2024-09-18T07:42:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:06 smithi137 ceph-mon[29408]: 5.10 scrub starts 2024-09-18T07:42:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:06 smithi137 ceph-mon[29408]: 5.10 scrub ok 2024-09-18T07:42:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:06 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/997624185' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:06 smithi137 ceph-mon[29408]: 6.11 scrub starts 2024-09-18T07:42:07.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:06 smithi137 ceph-mon[29408]: 6.11 scrub ok 2024-09-18T07:42:07.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:06 smithi137 ceph-mon[29408]: 5.6 scrub starts 2024-09-18T07:42:07.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:06 smithi137 ceph-mon[29408]: 5.6 scrub ok 2024-09-18T07:42:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:06 smithi137 ceph-mon[39175]: pgmap v596: 158 pgs: 158 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 689 B/s rd, 0 B/s wr, 824 op/s 2024-09-18T07:42:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:06 smithi137 ceph-mon[39175]: 5.10 scrub starts 2024-09-18T07:42:07.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:06 smithi137 ceph-mon[39175]: 5.10 scrub ok 2024-09-18T07:42:07.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:06 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/997624185' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:07.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:06 smithi137 ceph-mon[39175]: 6.11 scrub starts 2024-09-18T07:42:07.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:06 smithi137 ceph-mon[39175]: 6.11 scrub ok 2024-09-18T07:42:07.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:06 smithi137 ceph-mon[39175]: 5.6 scrub starts 2024-09-18T07:42:07.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:06 smithi137 ceph-mon[39175]: 5.6 scrub ok 2024-09-18T07:42:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:06 smithi203 ceph-mon[32237]: pgmap v596: 158 pgs: 158 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 689 B/s rd, 0 B/s wr, 824 op/s 2024-09-18T07:42:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:06 smithi203 ceph-mon[32237]: 5.10 scrub starts 2024-09-18T07:42:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:06 smithi203 ceph-mon[32237]: 5.10 scrub ok 2024-09-18T07:42:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:06 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/997624185' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:06 smithi203 ceph-mon[32237]: 6.11 scrub starts 2024-09-18T07:42:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:06 smithi203 ceph-mon[32237]: 6.11 scrub ok 2024-09-18T07:42:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:06 smithi203 ceph-mon[32237]: 5.6 scrub starts 2024-09-18T07:42:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:06 smithi203 ceph-mon[32237]: 5.6 scrub ok 2024-09-18T07:42:07.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:07.497 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:07.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:07.598 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:07.619 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:07.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:07.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:07.721 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:07.742 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:07.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:07.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:07.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:07.873 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:07.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:07.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:07.961 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:42:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:42:07.975 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:08.009 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:08.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:08.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:08.111 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:08.147 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:08.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:08.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:08.249 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:08.283 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:08.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:08.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:08.385 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:08.419 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:08.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:08 smithi137 ceph-mon[29408]: 6.d scrub starts 2024-09-18T07:42:08.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:08 smithi137 ceph-mon[29408]: 6.d scrub ok 2024-09-18T07:42:08.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:08 smithi137 ceph-mon[29408]: 5.4 scrub starts 2024-09-18T07:42:08.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:08 smithi137 ceph-mon[29408]: 5.4 scrub ok 2024-09-18T07:42:08.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:08 smithi137 ceph-mon[39175]: 6.d scrub starts 2024-09-18T07:42:08.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:08 smithi137 ceph-mon[39175]: 6.d scrub ok 2024-09-18T07:42:08.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:08 smithi137 ceph-mon[39175]: 5.4 scrub starts 2024-09-18T07:42:08.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:08 smithi137 ceph-mon[39175]: 5.4 scrub ok 2024-09-18T07:42:08.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:08.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:08 smithi203 ceph-mon[32237]: 6.d scrub starts 2024-09-18T07:42:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:08 smithi203 ceph-mon[32237]: 6.d scrub ok 2024-09-18T07:42:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:08 smithi203 ceph-mon[32237]: 5.4 scrub starts 2024-09-18T07:42:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:08 smithi203 ceph-mon[32237]: 5.4 scrub ok 2024-09-18T07:42:08.523 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:08.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:08.564 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:08.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:08.665 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:08.697 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:08.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:08.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:08.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:08.820 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:08.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:08.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:08.922 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:08.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:08.958 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:08.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:09.059 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:09.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:09.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:09.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:09.198 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:09.232 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:09.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:09.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:09.334 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:09.366 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:09.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:09.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:09 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:09 smithi137 ceph-mon[29408]: 6.a scrub starts 2024-09-18T07:42:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:09 smithi137 ceph-mon[29408]: 6.a scrub ok 2024-09-18T07:42:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:09 smithi137 ceph-mon[29408]: pgmap v597: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-18T07:42:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:09 smithi137 ceph-mon[29408]: 3.e scrub starts 2024-09-18T07:42:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:09 smithi137 ceph-mon[29408]: 3.e scrub ok 2024-09-18T07:42:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:09 smithi137 ceph-mon[29408]: 3.10 scrub starts 2024-09-18T07:42:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:09 smithi137 ceph-mon[29408]: 3.10 scrub ok 2024-09-18T07:42:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:09 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:09 smithi137 ceph-mon[39175]: 6.a scrub starts 2024-09-18T07:42:09.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:09 smithi137 ceph-mon[39175]: 6.a scrub ok 2024-09-18T07:42:09.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:09 smithi137 ceph-mon[39175]: pgmap v597: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-18T07:42:09.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:09 smithi137 ceph-mon[39175]: 3.e scrub starts 2024-09-18T07:42:09.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:09 smithi137 ceph-mon[39175]: 3.e scrub ok 2024-09-18T07:42:09.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:09 smithi137 ceph-mon[39175]: 3.10 scrub starts 2024-09-18T07:42:09.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:09 smithi137 ceph-mon[39175]: 3.10 scrub ok 2024-09-18T07:42:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:09 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:09 smithi203 ceph-mon[32237]: 6.a scrub starts 2024-09-18T07:42:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:09 smithi203 ceph-mon[32237]: 6.a scrub ok 2024-09-18T07:42:09.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:09 smithi203 ceph-mon[32237]: pgmap v597: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-18T07:42:09.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:09 smithi203 ceph-mon[32237]: 3.e scrub starts 2024-09-18T07:42:09.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:09 smithi203 ceph-mon[32237]: 3.e scrub ok 2024-09-18T07:42:09.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:09 smithi203 ceph-mon[32237]: 3.10 scrub starts 2024-09-18T07:42:09.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:09 smithi203 ceph-mon[32237]: 3.10 scrub ok 2024-09-18T07:42:09.468 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:09.493 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:09.495 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:09.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:09.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:09.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T07:42:09.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:09.660 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:09.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:09.762 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:09.801 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:09.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:09.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:09.924 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:09.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:09.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:10.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:10.061 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:10.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:10.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:10.163 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:10.197 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:10.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:10.229 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:10 smithi137 ceph-mon[29408]: 5.1 scrub starts 2024-09-18T07:42:10.229 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:10 smithi137 ceph-mon[29408]: 5.1 scrub ok 2024-09-18T07:42:10.230 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:10 smithi137 ceph-mon[29408]: 9.9 scrub starts 2024-09-18T07:42:10.230 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:10 smithi137 ceph-mon[39175]: 5.1 scrub starts 2024-09-18T07:42:10.230 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:10 smithi137 ceph-mon[39175]: 5.1 scrub ok 2024-09-18T07:42:10.230 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:10 smithi137 ceph-mon[39175]: 9.9 scrub starts 2024-09-18T07:42:10.299 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:10.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:10.331 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:10.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:10.431 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:10.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:10 smithi203 ceph-mon[32237]: 5.1 scrub starts 2024-09-18T07:42:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:10 smithi203 ceph-mon[32237]: 5.1 scrub ok 2024-09-18T07:42:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:10 smithi203 ceph-mon[32237]: 9.9 scrub starts 2024-09-18T07:42:10.469 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:10.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:10.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:10.543 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:10.574 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:10.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:10.641 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:10.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:10.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:10.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:10.784 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:10.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:10.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:10.912 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:10.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:10.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:10.961 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:42:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=cleanup t=2024-09-18T07:42:10.498039868Z level=info msg="Completed cleanup jobs" duration=38.325245ms 2024-09-18T07:42:10.961 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:42:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=grafana.update.checker t=2024-09-18T07:42:10.604668241Z level=info msg="Update check succeeded" duration=67.596679ms 2024-09-18T07:42:10.961 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:42:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=plugins.update.checker t=2024-09-18T07:42:10.606596131Z level=info msg="Update check succeeded" duration=65.949488ms 2024-09-18T07:42:11.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:11.043 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:42:11.053 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:11.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:11.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:11.155 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:11.167 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:11 smithi137 ceph-mon[29408]: pgmap v598: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-18T07:42:11.167 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:11 smithi137 ceph-mon[29408]: 6.8 scrub starts 2024-09-18T07:42:11.167 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:11 smithi137 ceph-mon[29408]: 6.8 scrub ok 2024-09-18T07:42:11.168 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:11 smithi137 ceph-mon[29408]: 9.9 scrub ok 2024-09-18T07:42:11.168 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:11 smithi137 ceph-mon[29408]: 4.11 scrub starts 2024-09-18T07:42:11.168 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:11 smithi137 ceph-mon[29408]: 4.11 scrub ok 2024-09-18T07:42:11.168 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:11 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/749996379' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:42:11.168 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:11 smithi137 ceph-mon[39175]: pgmap v598: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-18T07:42:11.168 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:11 smithi137 ceph-mon[39175]: 6.8 scrub starts 2024-09-18T07:42:11.168 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:11 smithi137 ceph-mon[39175]: 6.8 scrub ok 2024-09-18T07:42:11.168 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:11 smithi137 ceph-mon[39175]: 9.9 scrub ok 2024-09-18T07:42:11.168 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:11 smithi137 ceph-mon[39175]: 4.11 scrub starts 2024-09-18T07:42:11.168 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:11 smithi137 ceph-mon[39175]: 4.11 scrub ok 2024-09-18T07:42:11.168 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:11 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/749996379' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:42:11.189 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:11.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:11.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:11.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:11.325 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:11.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:11.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:11.427 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:11.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:11 smithi203 ceph-mon[32237]: pgmap v598: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-18T07:42:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:11 smithi203 ceph-mon[32237]: 6.8 scrub starts 2024-09-18T07:42:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:11 smithi203 ceph-mon[32237]: 6.8 scrub ok 2024-09-18T07:42:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:11 smithi203 ceph-mon[32237]: 9.9 scrub ok 2024-09-18T07:42:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:11 smithi203 ceph-mon[32237]: 4.11 scrub starts 2024-09-18T07:42:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:11 smithi203 ceph-mon[32237]: 4.11 scrub ok 2024-09-18T07:42:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:11 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/749996379' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:42:11.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:11.469 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:11.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:11.473 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:11.571 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:11.603 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:11.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:11.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:11.705 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:11.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:11.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:11.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:11.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:11.877 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:11.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:11.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:11.979 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:12.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:12.011 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:12.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:12.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:42:12.056 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.7 to 1 (10000) 2024-09-18T07:42:12.112 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:12.144 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:12.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:12.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:12.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:12.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:12.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:12.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:12.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:12 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:42:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:12 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/749996379' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:42:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:12 smithi137 ceph-mon[29408]: osdmap e347: 8 total, 8 up, 8 in 2024-09-18T07:42:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:12 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1260437527' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:42:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:12 smithi137 ceph-mon[29408]: from='client.25945 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:42:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:42:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:12 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:42:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:12 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/749996379' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:42:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:12 smithi137 ceph-mon[39175]: osdmap e347: 8 total, 8 up, 8 in 2024-09-18T07:42:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:12 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1260437527' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:42:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:12 smithi137 ceph-mon[39175]: from='client.25945 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:42:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:42:12.453 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:12.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:12.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:12.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:12 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:42:12.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:12 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/749996379' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:42:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:12 smithi203 ceph-mon[32237]: osdmap e347: 8 total, 8 up, 8 in 2024-09-18T07:42:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:12 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1260437527' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:42:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:12 smithi203 ceph-mon[32237]: from='client.25945 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:42:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:42:12.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:12.596 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:12.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:12.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:12.698 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:12.734 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:12.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:12.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:12.859 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:12.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:12.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:12.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:12.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:12.996 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:12.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:13.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:13.130 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:13.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:13.232 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:13.270 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:13.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:13.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:13.353 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:13 smithi137 ceph-mon[29408]: pgmap v600: 158 pgs: 158 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 951 op/s 2024-09-18T07:42:13.353 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:13 smithi137 ceph-mon[29408]: from='client.25945 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T07:42:13.354 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:13 smithi137 ceph-mon[29408]: osdmap e348: 8 total, 8 up, 8 in 2024-09-18T07:42:13.354 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:13 smithi137 ceph-mon[39175]: pgmap v600: 158 pgs: 158 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 951 op/s 2024-09-18T07:42:13.354 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:13 smithi137 ceph-mon[39175]: from='client.25945 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T07:42:13.354 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:13 smithi137 ceph-mon[39175]: osdmap e348: 8 total, 8 up, 8 in 2024-09-18T07:42:13.372 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:13.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:13.428 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:13.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:13.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:13 smithi203 ceph-mon[32237]: pgmap v600: 158 pgs: 158 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 951 op/s 2024-09-18T07:42:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:13 smithi203 ceph-mon[32237]: from='client.25945 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T07:42:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:13 smithi203 ceph-mon[32237]: osdmap e348: 8 total, 8 up, 8 in 2024-09-18T07:42:13.528 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:13.567 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:13.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:13.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:13.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:13.704 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:13.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:13.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:13.807 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:13.839 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:13.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:13.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:13.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:13.959 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-18T07:42:13.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:13.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:13.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:14.035 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:42:14.079 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:14.107 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:14.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:14.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:14.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:14.235 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:14.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:14.282 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:42:14.282 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:42:14.336 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:14.389 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:14.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:14.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:14.490 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:14.523 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:14.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:14.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:14.625 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:14.652 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:14.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:14.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:14.754 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:14.786 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:14.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:14.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:14.888 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:14.928 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:14.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:15.030 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:15.063 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:15.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:15.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:15.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:15.200 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:15.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:15.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:15.302 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:15.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:15.341 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:15.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:15.418 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:15 smithi137 ceph-mon[39175]: pgmap v602: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-18T07:42:15.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:15 smithi137 ceph-mon[29408]: pgmap v602: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-18T07:42:15.442 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:15 smithi203 ceph-mon[32237]: pgmap v602: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-18T07:42:15.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T07:42:15.478 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:15.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:15.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:15.521 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:15.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:15.588 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:15.611 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:15.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:15.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:15.713 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:15.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:15.747 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:15.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:15.848 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:15.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:15.902 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:15.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:16.003 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:16.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:16.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:16.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:16.138 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:16.176 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:16.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:16.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:16.278 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:16.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:16.313 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:16.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:16.414 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:16.446 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:16.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:16.549 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:16.575 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:16.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:16.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:16.677 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:16.707 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:16.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:16.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:16.731 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:42:16.731 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:42:16.731 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:42:16.731 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:42:16.731 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:42:16.731 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:42:16.731 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:42:16.731 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:42:16.732 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:42:16.732 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:42:16.732 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:42:16.732 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:42:16.732 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:42:16.732 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:42:16.732 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:42:16.732 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:42:16.745 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:16.809 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:16.841 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:16.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:16.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:16.943 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:16.983 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:16.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:16.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:16.993 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:17.072 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.6 to 1 (10000) 2024-09-18T07:42:17.085 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:17.115 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:17.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:17.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:17.217 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:17.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:17.253 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:17.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:17.354 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:17.389 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:17.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:17.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:42:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:42:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:42:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:17 smithi137 ceph-mon[39175]: pgmap v603: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 750 op/s 2024-09-18T07:42:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:17 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2855728003' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:17 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1305410650' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:42:17.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:17 smithi137 ceph-mon[29408]: pgmap v603: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 750 op/s 2024-09-18T07:42:17.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:17 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2855728003' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:17.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:17 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1305410650' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:42:17.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:17 smithi203 ceph-mon[32237]: pgmap v603: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 750 op/s 2024-09-18T07:42:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:17 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2855728003' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:17 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1305410650' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:42:17.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:17.528 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:17.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:17.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:17.632 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:17.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:17.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:17.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:17.769 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:17.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:17.805 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:17.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:17.906 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:17.944 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:17.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:17.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:17.961 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:42:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:42:18.046 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:18.083 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:18.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:18.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:18.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:18 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1305410650' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:42:18.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:18 smithi137 ceph-mon[29408]: osdmap e349: 8 total, 8 up, 8 in 2024-09-18T07:42:18.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:18 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1305410650' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:42:18.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:18 smithi137 ceph-mon[39175]: osdmap e349: 8 total, 8 up, 8 in 2024-09-18T07:42:18.184 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:18.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:18.215 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:18.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:18.317 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:18.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:18.354 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:18.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:18.455 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:18.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:18 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1305410650' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:42:18.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:18 smithi203 ceph-mon[32237]: osdmap e349: 8 total, 8 up, 8 in 2024-09-18T07:42:18.487 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:18.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:18.589 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:18.622 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:18.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:18.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:18.724 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:18.752 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-18T07:42:18.753 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:42:18.772 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:18.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:18.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:18.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:18.921 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:18.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:18.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:19.003 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:42:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:42:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:42:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:42:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.004 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135022 7.13 7:cab60f17:::benchmark_data_smithi137_74180_object30606:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:42:19.005 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964232+0000", 2024-09-18T07:42:19.005 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 482.034003111, 2024-09-18T07:42:19.005 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2320603539999999, 2024-09-18T07:42:19.005 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.005 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135022 2024-09-18T07:42:19.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964232+0000", 2024-09-18T07:42:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964232+0000", 2024-09-18T07:42:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964235+0000", 2024-09-18T07:42:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9340000000000002e-06 2024-09-18T07:42:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964242+0000", 2024-09-18T07:42:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.379e-06 2024-09-18T07:42:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964261+0000", 2024-09-18T07:42:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8851999999999998e-05 2024-09-18T07:42:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964270+0000", 2024-09-18T07:42:19.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5040000000000008e-06 2024-09-18T07:42:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964336+0000", 2024-09-18T07:42:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5092999999999995e-05 2024-09-18T07:42:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:42:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964344+0000", 2024-09-18T07:42:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0879999999999995e-06 2024-09-18T07:42:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971555+0000", 2024-09-18T07:42:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0072113109999999998 2024-09-18T07:42:19.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971563+0000", 2024-09-18T07:42:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4689999999999997e-06 2024-09-18T07:42:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.014 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.190908+0000", 2024-09-18T07:42:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2193444609999999 2024-09-18T07:42:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.191034+0000", 2024-09-18T07:42:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000125871 2024-09-18T07:42:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3,4,5", 2024-09-18T07:42:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.191148+0000", 2024-09-18T07:42:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114097 2024-09-18T07:42:19.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.193787+0000", 2024-09-18T07:42:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026389650000000001 2024-09-18T07:42:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.194990+0000", 2024-09-18T07:42:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001203523 2024-09-18T07:42:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.195790+0000", 2024-09-18T07:42:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00079991899999999996 2024-09-18T07:42:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.195803+0000", 2024-09-18T07:42:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2955999999999999e-05 2024-09-18T07:42:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.196126+0000", 2024-09-18T07:42:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032304700000000003 2024-09-18T07:42:19.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.196180+0000", 2024-09-18T07:42:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4178000000000002e-05 2024-09-18T07:42:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.196292+0000", 2024-09-18T07:42:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011170700000000001 2024-09-18T07:42:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.018 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151745 7.8 7:11d4aec3:::benchmark_data_smithi137_74180_object47329:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-18T07:42:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.256412+0000", 2024-09-18T07:42:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 458.74182287500003, 2024-09-18T07:42:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96419230700000003, 2024-09-18T07:42:19.019 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151745 2024-09-18T07:42:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256412+0000", 2024-09-18T07:42:19.020 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256412+0000", 2024-09-18T07:42:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.021 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256413+0000", 2024-09-18T07:42:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1900000000000002e-07 2024-09-18T07:42:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256415+0000", 2024-09-18T07:42:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4880000000000001e-06 2024-09-18T07:42:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.022 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256421+0000", 2024-09-18T07:42:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7019999999999998e-06 2024-09-18T07:42:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256426+0000", 2024-09-18T07:42:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1000000000000003e-06 2024-09-18T07:42:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282222+0000", 2024-09-18T07:42:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025796116000000001 2024-09-18T07:42:19.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282233+0000", 2024-09-18T07:42:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0539e-05 2024-09-18T07:42:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.215761+0000", 2024-09-18T07:42:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93352816100000002 2024-09-18T07:42:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.215857+0000", 2024-09-18T07:42:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6032999999999993e-05 2024-09-18T07:42:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:42:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.215960+0000", 2024-09-18T07:42:19.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103329 2024-09-18T07:42:19.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217112+0000", 2024-09-18T07:42:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011516709999999999 2024-09-18T07:42:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218052+0000", 2024-09-18T07:42:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00093980599999999995 2024-09-18T07:42:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220525+0000", 2024-09-18T07:42:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024736509999999999 2024-09-18T07:42:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220565+0000", 2024-09-18T07:42:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9847e-05 2024-09-18T07:42:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220604+0000", 2024-09-18T07:42:19.028 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9245000000000001e-05 2024-09-18T07:42:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151823 7.8 7:101e05c7:::benchmark_data_smithi137_74180_object47407:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-18T07:42:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.256432+0000", 2024-09-18T07:42:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 458.74180339499998, 2024-09-18T07:42:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96425229099999998, 2024-09-18T07:42:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.029 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151823 2024-09-18T07:42:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256432+0000", 2024-09-18T07:42:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256432+0000", 2024-09-18T07:42:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256432+0000", 2024-09-18T07:42:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6199999999999998e-07 2024-09-18T07:42:19.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256435+0000", 2024-09-18T07:42:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4600000000000002e-06 2024-09-18T07:42:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256440+0000", 2024-09-18T07:42:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9830000000000004e-06 2024-09-18T07:42:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256444+0000", 2024-09-18T07:42:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6909999999999996e-06 2024-09-18T07:42:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282270+0000", 2024-09-18T07:42:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025825796000000002 2024-09-18T07:42:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282277+0000", 2024-09-18T07:42:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2989999999999999e-06 2024-09-18T07:42:19.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216194+0000", 2024-09-18T07:42:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93391638700000001 2024-09-18T07:42:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216288+0000", 2024-09-18T07:42:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3845999999999996e-05 2024-09-18T07:42:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:42:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216381+0000", 2024-09-18T07:42:19.034 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3960000000000002e-05 2024-09-18T07:42:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217934+0000", 2024-09-18T07:42:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001552257 2024-09-18T07:42:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218106+0000", 2024-09-18T07:42:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017274099999999999 2024-09-18T07:42:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.035 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220613+0000", 2024-09-18T07:42:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025060780000000001 2024-09-18T07:42:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220649+0000", 2024-09-18T07:42:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6380000000000001e-05 2024-09-18T07:42:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220684+0000", 2024-09-18T07:42:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4851000000000002e-05 2024-09-18T07:42:19.036 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.037 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.037 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.037 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151853 7.8 7:10bf2ca7:::benchmark_data_smithi137_74180_object47437:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-18T07:42:19.037 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.256449+0000", 2024-09-18T07:42:19.037 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 458.74178573699999, 2024-09-18T07:42:19.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96431083500000003, 2024-09-18T07:42:19.037 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.037 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.037 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.037 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.037 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.037 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.038 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151853 2024-09-18T07:42:19.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.038 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256449+0000", 2024-09-18T07:42:19.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256449+0000", 2024-09-18T07:42:19.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256450+0000", 2024-09-18T07:42:19.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3799999999999998e-07 2024-09-18T07:42:19.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256452+0000", 2024-09-18T07:42:19.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.582e-06 2024-09-18T07:42:19.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256457+0000", 2024-09-18T07:42:19.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3029999999999999e-06 2024-09-18T07:42:19.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.040 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256463+0000", 2024-09-18T07:42:19.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0540000000000002e-06 2024-09-18T07:42:19.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.040 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282322+0000", 2024-09-18T07:42:19.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025859495999999999 2024-09-18T07:42:19.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.041 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.041 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282329+0000", 2024-09-18T07:42:19.041 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.391e-06 2024-09-18T07:42:19.041 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.041 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.041 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.041 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216585+0000", 2024-09-18T07:42:19.041 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93425595100000003 2024-09-18T07:42:19.041 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.041 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.041 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.041 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216677+0000", 2024-09-18T07:42:19.042 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1812999999999997e-05 2024-09-18T07:42:19.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.042 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:42:19.042 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.216781+0000", 2024-09-18T07:42:19.042 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103607 2024-09-18T07:42:19.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.042 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.042 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217970+0000", 2024-09-18T07:42:19.042 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011890959999999999 2024-09-18T07:42:19.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.043 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218137+0000", 2024-09-18T07:42:19.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016691300000000001 2024-09-18T07:42:19.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.043 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220692+0000", 2024-09-18T07:42:19.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002555571 2024-09-18T07:42:19.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220729+0000", 2024-09-18T07:42:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6128999999999999e-05 2024-09-18T07:42:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220760+0000", 2024-09-18T07:42:19.044 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1491000000000001e-05 2024-09-18T07:42:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151969 7.8 7:10c7e652:::benchmark_data_smithi137_74180_object47553:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-18T07:42:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.256468+0000", 2024-09-18T07:42:19.045 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 458.74176721399999, 2024-09-18T07:42:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96436882599999996, 2024-09-18T07:42:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.046 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151969 2024-09-18T07:42:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256468+0000", 2024-09-18T07:42:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.047 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.048 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256468+0000", 2024-09-18T07:42:19.048 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.048 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256468+0000", 2024-09-18T07:42:19.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9500000000000003e-07 2024-09-18T07:42:19.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.049 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256471+0000", 2024-09-18T07:42:19.050 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3410000000000001e-06 2024-09-18T07:42:19.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.050 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.050 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256476+0000", 2024-09-18T07:42:19.050 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3619999999999997e-06 2024-09-18T07:42:19.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.051 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.051 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256480+0000", 2024-09-18T07:42:19.051 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5879999999999999e-06 2024-09-18T07:42:19.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.051 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.051 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282362+0000", 2024-09-18T07:42:19.052 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025881686000000001 2024-09-18T07:42:19.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.052 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.052 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282370+0000", 2024-09-18T07:42:19.052 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0979999999999994e-06 2024-09-18T07:42:19.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217145+0000", 2024-09-18T07:42:19.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93477450600000001 2024-09-18T07:42:19.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217239+0000", 2024-09-18T07:42:19.054 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3744000000000002e-05 2024-09-18T07:42:19.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:42:19.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217328+0000", 2024-09-18T07:42:19.054 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9494999999999996e-05 2024-09-18T07:42:19.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.055 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.055 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218079+0000", 2024-09-18T07:42:19.055 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00075083099999999998 2024-09-18T07:42:19.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.055 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.055 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.218172+0000", 2024-09-18T07:42:19.056 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2713999999999994e-05 2024-09-18T07:42:19.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.056 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.056 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.056 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220767+0000", 2024-09-18T07:42:19.056 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002595848 2024-09-18T07:42:19.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.057 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.057 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220804+0000", 2024-09-18T07:42:19.057 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6151e-05 2024-09-18T07:42:19.057 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.057 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.057 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.220837+0000", 2024-09-18T07:42:19.058 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2966999999999998e-05 2024-09-18T07:42:19.058 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.058 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.058 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.058 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.058 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:151980 7.8 7:1648d987:::benchmark_data_smithi137_74180_object47564:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e151)", 2024-09-18T07:42:19.058 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:40.256485+0000", 2024-09-18T07:42:19.059 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 458.74174952999999, 2024-09-18T07:42:19.059 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96460135300000005, 2024-09-18T07:42:19.059 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.059 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.059 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.059 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.059 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.060 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.060 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 151980 2024-09-18T07:42:19.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.060 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.060 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.060 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256485+0000", 2024-09-18T07:42:19.060 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.060 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.060 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256485+0000", 2024-09-18T07:42:19.060 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.061 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.061 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256486+0000", 2024-09-18T07:42:19.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8299999999999997e-07 2024-09-18T07:42:19.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.061 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.061 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256488+0000", 2024-09-18T07:42:19.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1459999999999999e-06 2024-09-18T07:42:19.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.061 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.061 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256493+0000", 2024-09-18T07:42:19.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.147e-06 2024-09-18T07:42:19.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.062 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.062 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.256498+0000", 2024-09-18T07:42:19.062 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6530000000000002e-06 2024-09-18T07:42:19.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.062 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.062 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282399+0000", 2024-09-18T07:42:19.062 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025901569999999999 2024-09-18T07:42:19.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.062 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.062 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:40.282406+0000", 2024-09-18T07:42:19.063 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9380000000000003e-06 2024-09-18T07:42:19.063 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.063 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.063 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.063 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217538+0000", 2024-09-18T07:42:19.063 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93513140400000005 2024-09-18T07:42:19.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.064 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.064 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217632+0000", 2024-09-18T07:42:19.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4285999999999998e-05 2024-09-18T07:42:19.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.064 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:42:19.064 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.217727+0000", 2024-09-18T07:42:19.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5168999999999996e-05 2024-09-18T07:42:19.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.064 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.219197+0000", 2024-09-18T07:42:19.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014697880000000001 2024-09-18T07:42:19.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.219258+0000", 2024-09-18T07:42:19.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1444000000000003e-05 2024-09-18T07:42:19.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221042+0000", 2024-09-18T07:42:19.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017834820000000001 2024-09-18T07:42:19.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221067+0000", 2024-09-18T07:42:19.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4714000000000001e-05 2024-09-18T07:42:19.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:41.221087+0000", 2024-09-18T07:42:19.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0129e-05 2024-09-18T07:42:19.067 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.067 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.067 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.067 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157484 7.5 7:a765acc6:::benchmark_data_smithi137_74180_object53068:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163)", 2024-09-18T07:42:19.067 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.351294+0000", 2024-09-18T07:42:19.067 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 446.646940965, 2024-09-18T07:42:19.067 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98376080499999996, 2024-09-18T07:42:19.067 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.067 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.067 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:19.067 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.068 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.068 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.068 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157484 2024-09-18T07:42:19.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.068 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.068 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351294+0000", 2024-09-18T07:42:19.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.068 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351294+0000", 2024-09-18T07:42:19.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351298+0000", 2024-09-18T07:42:19.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.929e-06 2024-09-18T07:42:19.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351307+0000", 2024-09-18T07:42:19.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0720000000000006e-06 2024-09-18T07:42:19.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.070 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351321+0000", 2024-09-18T07:42:19.070 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4113e-05 2024-09-18T07:42:19.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.070 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351332+0000", 2024-09-18T07:42:19.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1080999999999999e-05 2024-09-18T07:42:19.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.382508+0000", 2024-09-18T07:42:19.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031176124 2024-09-18T07:42:19.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.382519+0000", 2024-09-18T07:42:19.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1139e-05 2024-09-18T07:42:19.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.334948+0000", 2024-09-18T07:42:19.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95242890700000005 2024-09-18T07:42:19.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.335055+0000", 2024-09-18T07:42:19.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010644 2024-09-18T07:42:19.072 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.072 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.072 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.073 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157497 7.5 7:a0bb3991:::benchmark_data_smithi137_74180_object53081:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163)", 2024-09-18T07:42:19.073 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.351358+0000", 2024-09-18T07:42:19.073 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 446.64687686100001, 2024-09-18T07:42:19.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98430336900000004, 2024-09-18T07:42:19.073 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.073 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.073 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:19.073 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.073 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.073 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.073 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157497 2024-09-18T07:42:19.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.074 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351358+0000", 2024-09-18T07:42:19.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351358+0000", 2024-09-18T07:42:19.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351359+0000", 2024-09-18T07:42:19.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3699999999999999e-07 2024-09-18T07:42:19.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351362+0000", 2024-09-18T07:42:19.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.207e-06 2024-09-18T07:42:19.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351370+0000", 2024-09-18T07:42:19.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7030000000000006e-06 2024-09-18T07:42:19.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351375+0000", 2024-09-18T07:42:19.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.536e-06 2024-09-18T07:42:19.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.382542+0000", 2024-09-18T07:42:19.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031166337999999998 2024-09-18T07:42:19.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.382548+0000", 2024-09-18T07:42:19.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4910000000000002e-06 2024-09-18T07:42:19.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.335593+0000", 2024-09-18T07:42:19.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95304532900000005 2024-09-18T07:42:19.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.335661+0000", 2024-09-18T07:42:19.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7928000000000004e-05 2024-09-18T07:42:19.077 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.077 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.077 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.078 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157538 7.5 7:a5eca45d:::benchmark_data_smithi137_74180_object53122:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163)", 2024-09-18T07:42:19.078 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.351382+0000", 2024-09-18T07:42:19.078 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 446.64685307600001, 2024-09-18T07:42:19.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98539270599999995, 2024-09-18T07:42:19.078 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.078 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.078 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.078 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.078 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.078 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.078 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157538 2024-09-18T07:42:19.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.079 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351382+0000", 2024-09-18T07:42:19.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351382+0000", 2024-09-18T07:42:19.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351382+0000", 2024-09-18T07:42:19.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4900000000000001e-07 2024-09-18T07:42:19.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351385+0000", 2024-09-18T07:42:19.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3939999999999999e-06 2024-09-18T07:42:19.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351390+0000", 2024-09-18T07:42:19.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9919999999999998e-06 2024-09-18T07:42:19.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351393+0000", 2024-09-18T07:42:19.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7840000000000001e-06 2024-09-18T07:42:19.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.382569+0000", 2024-09-18T07:42:19.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031175986999999999 2024-09-18T07:42:19.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.382576+0000", 2024-09-18T07:42:19.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5220000000000004e-06 2024-09-18T07:42:19.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.335681+0000", 2024-09-18T07:42:19.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95310482900000004 2024-09-18T07:42:19.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.335770+0000", 2024-09-18T07:42:19.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9433999999999998e-05 2024-09-18T07:42:19.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:42:19.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.335841+0000", 2024-09-18T07:42:19.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0946999999999997e-05 2024-09-18T07:42:19.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336661+0000", 2024-09-18T07:42:19.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00082012999999999997 2024-09-18T07:42:19.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336720+0000", 2024-09-18T07:42:19.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8394999999999999e-05 2024-09-18T07:42:19.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336728+0000", 2024-09-18T07:42:19.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3960000000000006e-06 2024-09-18T07:42:19.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336753+0000", 2024-09-18T07:42:19.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4573999999999999e-05 2024-09-18T07:42:19.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336775+0000", 2024-09-18T07:42:19.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1872999999999999e-05 2024-09-18T07:42:19.088 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.088 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.088 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.089 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157554 7.5 7:a5757979:::benchmark_data_smithi137_74180_object53138:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163)", 2024-09-18T07:42:19.089 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.351397+0000", 2024-09-18T07:42:19.089 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 446.64683761200001, 2024-09-18T07:42:19.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98575066499999997, 2024-09-18T07:42:19.089 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.090 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.090 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.090 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.090 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.090 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.090 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157554 2024-09-18T07:42:19.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.090 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351397+0000", 2024-09-18T07:42:19.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351397+0000", 2024-09-18T07:42:19.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351398+0000", 2024-09-18T07:42:19.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8500000000000002e-07 2024-09-18T07:42:19.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351399+0000", 2024-09-18T07:42:19.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.764e-06 2024-09-18T07:42:19.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351402+0000", 2024-09-18T07:42:19.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7250000000000002e-06 2024-09-18T07:42:19.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.351405+0000", 2024-09-18T07:42:19.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7439999999999999e-06 2024-09-18T07:42:19.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.382595+0000", 2024-09-18T07:42:19.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031190473999999999 2024-09-18T07:42:19.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.382601+0000", 2024-09-18T07:42:19.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4940000000000003e-06 2024-09-18T07:42:19.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336048+0000", 2024-09-18T07:42:19.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95344718399999995 2024-09-18T07:42:19.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336154+0000", 2024-09-18T07:42:19.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000106029 2024-09-18T07:42:19.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:42:19.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336265+0000", 2024-09-18T07:42:19.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011141000000000001 2024-09-18T07:42:19.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.336782+0000", 2024-09-18T07:42:19.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00051622400000000002 2024-09-18T07:42:19.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.337003+0000", 2024-09-18T07:42:19.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000221373 2024-09-18T07:42:19.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.337059+0000", 2024-09-18T07:42:19.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6462999999999998e-05 2024-09-18T07:42:19.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.097 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.097 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.337109+0000", 2024-09-18T07:42:19.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9227999999999997e-05 2024-09-18T07:42:19.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.097 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.097 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.337148+0000", 2024-09-18T07:42:19.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9267999999999997e-05 2024-09-18T07:42:19.097 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.097 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.097 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.098 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157635 7.5 7:a8492353:::benchmark_data_smithi137_74180_object53219:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e164)", 2024-09-18T07:42:19.098 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.383256+0000", 2024-09-18T07:42:19.098 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 446.61497883999999, 2024-09-18T07:42:19.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96415525300000005, 2024-09-18T07:42:19.098 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.098 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.098 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.098 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.098 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.098 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.098 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157635 2024-09-18T07:42:19.098 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.099 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.099 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.099 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383256+0000", 2024-09-18T07:42:19.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.099 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.099 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383256+0000", 2024-09-18T07:42:19.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.099 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.099 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383256+0000", 2024-09-18T07:42:19.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5600000000000001e-07 2024-09-18T07:42:19.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383258+0000", 2024-09-18T07:42:19.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9750000000000001e-06 2024-09-18T07:42:19.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383262+0000", 2024-09-18T07:42:19.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.365e-06 2024-09-18T07:42:19.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.101 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.101 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383264+0000", 2024-09-18T07:42:19.101 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6359999999999998e-06 2024-09-18T07:42:19.101 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.101 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.101 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383431+0000", 2024-09-18T07:42:19.101 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016692899999999999 2024-09-18T07:42:19.101 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.101 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.101 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383439+0000", 2024-09-18T07:42:19.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0169999999999996e-06 2024-09-18T07:42:19.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.339552+0000", 2024-09-18T07:42:19.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95611299299999997 2024-09-18T07:42:19.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.339643+0000", 2024-09-18T07:42:19.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0787999999999999e-05 2024-09-18T07:42:19.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:42:19.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.339738+0000", 2024-09-18T07:42:19.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4581999999999997e-05 2024-09-18T07:42:19.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.103 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341908+0000", 2024-09-18T07:42:19.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0021700529999999999 2024-09-18T07:42:19.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.103 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.344559+0000", 2024-09-18T07:42:19.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026512200000000001 2024-09-18T07:42:19.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.104 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347336+0000", 2024-09-18T07:42:19.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002777135 2024-09-18T07:42:19.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.104 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347377+0000", 2024-09-18T07:42:19.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1074e-05 2024-09-18T07:42:19.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.105 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347411+0000", 2024-09-18T07:42:19.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.413e-05 2024-09-18T07:42:19.105 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.105 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.105 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.105 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157640 7.5 7:ace13225:::benchmark_data_smithi137_74180_object53224:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e164)", 2024-09-18T07:42:19.105 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.383278+0000", 2024-09-18T07:42:19.105 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 446.61495683599998, 2024-09-18T07:42:19.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96425578599999995, 2024-09-18T07:42:19.106 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.106 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.106 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.106 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.106 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.106 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.106 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157640 2024-09-18T07:42:19.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.106 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383278+0000", 2024-09-18T07:42:19.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383278+0000", 2024-09-18T07:42:19.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383278+0000", 2024-09-18T07:42:19.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3799999999999998e-07 2024-09-18T07:42:19.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383280+0000", 2024-09-18T07:42:19.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9420000000000002e-06 2024-09-18T07:42:19.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383284+0000", 2024-09-18T07:42:19.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2509999999999999e-06 2024-09-18T07:42:19.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383286+0000", 2024-09-18T07:42:19.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7599999999999998e-06 2024-09-18T07:42:19.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383477+0000", 2024-09-18T07:42:19.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000190261 2024-09-18T07:42:19.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383484+0000", 2024-09-18T07:42:19.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4619999999999999e-06 2024-09-18T07:42:19.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.339906+0000", 2024-09-18T07:42:19.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95642190199999999 2024-09-18T07:42:19.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.340001+0000", 2024-09-18T07:42:19.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4598999999999993e-05 2024-09-18T07:42:19.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:42:19.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.340095+0000", 2024-09-18T07:42:19.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4714000000000002e-05 2024-09-18T07:42:19.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341912+0000", 2024-09-18T07:42:19.111 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018169449999999999 2024-09-18T07:42:19.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.344597+0000", 2024-09-18T07:42:19.111 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026846940000000001 2024-09-18T07:42:19.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347445+0000", 2024-09-18T07:42:19.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028480329999999998 2024-09-18T07:42:19.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.112 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.112 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347485+0000", 2024-09-18T07:42:19.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0330000000000002e-05 2024-09-18T07:42:19.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.112 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.112 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347534+0000", 2024-09-18T07:42:19.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8554999999999998e-05 2024-09-18T07:42:19.112 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.112 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.112 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.113 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157677 7.5 7:ae2907ef:::benchmark_data_smithi137_74180_object53261:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e164)", 2024-09-18T07:42:19.113 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.383289+0000", 2024-09-18T07:42:19.113 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 446.61494561799998, 2024-09-18T07:42:19.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96435305199999999, 2024-09-18T07:42:19.113 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.113 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.113 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.113 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.113 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.113 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.113 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157677 2024-09-18T07:42:19.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.114 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.114 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383289+0000", 2024-09-18T07:42:19.114 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.114 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383289+0000", 2024-09-18T07:42:19.114 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383290+0000", 2024-09-18T07:42:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6100000000000002e-07 2024-09-18T07:42:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383294+0000", 2024-09-18T07:42:19.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.622e-06 2024-09-18T07:42:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383297+0000", 2024-09-18T07:42:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.554e-06 2024-09-18T07:42:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383299+0000", 2024-09-18T07:42:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5189999999999999e-06 2024-09-18T07:42:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.116 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383519+0000", 2024-09-18T07:42:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022010199999999999 2024-09-18T07:42:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383527+0000", 2024-09-18T07:42:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2860000000000002e-06 2024-09-18T07:42:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.340269+0000", 2024-09-18T07:42:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95674225800000001 2024-09-18T07:42:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.117 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.340359+0000", 2024-09-18T07:42:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0400000000000002e-05 2024-09-18T07:42:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:42:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.340468+0000", 2024-09-18T07:42:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000109095 2024-09-18T07:42:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341917+0000", 2024-09-18T07:42:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014483670000000001 2024-09-18T07:42:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.344642+0000", 2024-09-18T07:42:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002725744 2024-09-18T07:42:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347567+0000", 2024-09-18T07:42:19.119 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029242510000000001 2024-09-18T07:42:19.120 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.120 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.120 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.120 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347607+0000", 2024-09-18T07:42:19.120 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0159e-05 2024-09-18T07:42:19.120 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.120 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.120 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.120 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347642+0000", 2024-09-18T07:42:19.120 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5434e-05 2024-09-18T07:42:19.120 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.120 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.120 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.121 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.121 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.121 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157692 7.5 7:aab0905d:::benchmark_data_smithi137_74180_object53276:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e164)", 2024-09-18T07:42:19.121 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.383302+0000", 2024-09-18T07:42:19.121 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 446.61493304499999, 2024-09-18T07:42:19.121 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96446017699999997, 2024-09-18T07:42:19.121 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.121 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.121 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.121 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.121 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.122 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.122 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157692 2024-09-18T07:42:19.122 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.122 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.122 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.122 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.122 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383302+0000", 2024-09-18T07:42:19.122 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.122 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.123 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383302+0000", 2024-09-18T07:42:19.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.123 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.123 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383302+0000", 2024-09-18T07:42:19.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7399999999999999e-07 2024-09-18T07:42:19.123 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.123 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383303+0000", 2024-09-18T07:42:19.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.189e-06 2024-09-18T07:42:19.124 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.124 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.124 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383306+0000", 2024-09-18T07:42:19.124 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1760000000000002e-06 2024-09-18T07:42:19.124 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.124 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.124 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383311+0000", 2024-09-18T07:42:19.124 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.206e-06 2024-09-18T07:42:19.124 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.124 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.125 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383563+0000", 2024-09-18T07:42:19.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000252434 2024-09-18T07:42:19.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.125 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.125 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383570+0000", 2024-09-18T07:42:19.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8759999999999999e-06 2024-09-18T07:42:19.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.125 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.125 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.340670+0000", 2024-09-18T07:42:19.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95709987200000002 2024-09-18T07:42:19.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.126 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.126 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.340770+0000", 2024-09-18T07:42:19.126 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9798999999999998e-05 2024-09-18T07:42:19.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.126 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:42:19.126 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.340866+0000", 2024-09-18T07:42:19.126 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5851000000000001e-05 2024-09-18T07:42:19.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.126 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.126 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341921+0000", 2024-09-18T07:42:19.126 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010552319999999999 2024-09-18T07:42:19.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.127 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.127 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.345122+0000", 2024-09-18T07:42:19.127 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003201668 2024-09-18T07:42:19.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.127 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.127 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347679+0000", 2024-09-18T07:42:19.127 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002556961 2024-09-18T07:42:19.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.127 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.127 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347723+0000", 2024-09-18T07:42:19.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3884000000000002e-05 2024-09-18T07:42:19.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.128 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.128 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347762+0000", 2024-09-18T07:42:19.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8754999999999997e-05 2024-09-18T07:42:19.128 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.128 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.128 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.128 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157709 7.5 7:ab0da19b:::benchmark_data_smithi137_74180_object53293:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e164)", 2024-09-18T07:42:19.129 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.383316+0000", 2024-09-18T07:42:19.129 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 446.61491935200002, 2024-09-18T07:42:19.129 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96456343899999997, 2024-09-18T07:42:19.129 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.129 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.129 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.129 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.129 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.129 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.129 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157709 2024-09-18T07:42:19.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.129 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.129 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.130 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.130 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383316+0000", 2024-09-18T07:42:19.130 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.130 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.130 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.130 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383316+0000", 2024-09-18T07:42:19.130 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.130 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.130 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.130 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383316+0000", 2024-09-18T07:42:19.130 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5400000000000002e-07 2024-09-18T07:42:19.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.131 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.131 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383318+0000", 2024-09-18T07:42:19.131 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1409999999999999e-06 2024-09-18T07:42:19.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.131 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383333+0000", 2024-09-18T07:42:19.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4817999999999999e-05 2024-09-18T07:42:19.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383337+0000", 2024-09-18T07:42:19.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5220000000000001e-06 2024-09-18T07:42:19.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383615+0000", 2024-09-18T07:42:19.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027705399999999999 2024-09-18T07:42:19.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.133 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.133 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383623+0000", 2024-09-18T07:42:19.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2390000000000002e-06 2024-09-18T07:42:19.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.133 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.133 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341065+0000", 2024-09-18T07:42:19.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95744174100000001 2024-09-18T07:42:19.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.134 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.134 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341156+0000", 2024-09-18T07:42:19.134 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1150999999999995e-05 2024-09-18T07:42:19.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.134 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.134 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:42:19.134 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341272+0000", 2024-09-18T07:42:19.134 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116649 2024-09-18T07:42:19.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.134 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.134 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.134 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341925+0000", 2024-09-18T07:42:19.134 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00065243500000000004 2024-09-18T07:42:19.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.135 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.345159+0000", 2024-09-18T07:42:19.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032342820000000002 2024-09-18T07:42:19.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.135 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347794+0000", 2024-09-18T07:42:19.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026350280000000002 2024-09-18T07:42:19.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.135 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347844+0000", 2024-09-18T07:42:19.136 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9465000000000002e-05 2024-09-18T07:42:19.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.136 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.136 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347879+0000", 2024-09-18T07:42:19.136 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5460000000000003e-05 2024-09-18T07:42:19.136 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.136 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.136 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.136 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:157721 7.5 7:a908ea8d:::benchmark_data_smithi137_74180_object53305:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e164)", 2024-09-18T07:42:19.137 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:52.383345+0000", 2024-09-18T07:42:19.137 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 446.61489029000001, 2024-09-18T07:42:19.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96466771200000001, 2024-09-18T07:42:19.137 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.137 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.137 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.137 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.137 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.137 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.137 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 157721 2024-09-18T07:42:19.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.137 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383345+0000", 2024-09-18T07:42:19.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383345+0000", 2024-09-18T07:42:19.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383345+0000", 2024-09-18T07:42:19.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0200000000000002e-07 2024-09-18T07:42:19.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383348+0000", 2024-09-18T07:42:19.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.807e-06 2024-09-18T07:42:19.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383353+0000", 2024-09-18T07:42:19.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0719999999999999e-06 2024-09-18T07:42:19.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383357+0000", 2024-09-18T07:42:19.140 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3089999999999997e-06 2024-09-18T07:42:19.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.140 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383650+0000", 2024-09-18T07:42:19.140 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029264800000000001 2024-09-18T07:42:19.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.141 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.141 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:52.383658+0000", 2024-09-18T07:42:19.141 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4239999999999993e-06 2024-09-18T07:42:19.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.141 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.141 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.141 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341469+0000", 2024-09-18T07:42:19.141 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95781031299999997 2024-09-18T07:42:19.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.142 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341561+0000", 2024-09-18T07:42:19.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2633999999999994e-05 2024-09-18T07:42:19.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:42:19.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341633+0000", 2024-09-18T07:42:19.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2050000000000003e-05 2024-09-18T07:42:19.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.143 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.341947+0000", 2024-09-18T07:42:19.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00031341300000000002 2024-09-18T07:42:19.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.345190+0000", 2024-09-18T07:42:19.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032434059999999999 2024-09-18T07:42:19.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347913+0000", 2024-09-18T07:42:19.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027231769999999998 2024-09-18T07:42:19.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.145 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.347971+0000", 2024-09-18T07:42:19.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7951000000000001e-05 2024-09-18T07:42:19.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:53.348012+0000", 2024-09-18T07:42:19.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0905999999999998e-05 2024-09-18T07:42:19.146 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.146 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.146 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.147 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162329 7.8 7:14bf872c:::benchmark_data_smithi137_74180_object57913:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:42:19.148 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.422282+0000", 2024-09-18T07:42:19.148 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 437.57595258700002, 2024-09-18T07:42:19.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.010768962, 2024-09-18T07:42:19.148 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.148 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.148 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.148 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.148 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.148 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.148 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162329 2024-09-18T07:42:19.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.148 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422282+0000", 2024-09-18T07:42:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422282+0000", 2024-09-18T07:42:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.149 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.150 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422284+0000", 2024-09-18T07:42:19.150 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4950000000000001e-06 2024-09-18T07:42:19.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.150 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.150 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422293+0000", 2024-09-18T07:42:19.150 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7029999999999995e-06 2024-09-18T07:42:19.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422302+0000", 2024-09-18T07:42:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5580000000000006e-06 2024-09-18T07:42:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422310+0000", 2024-09-18T07:42:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3010000000000007e-06 2024-09-18T07:42:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422376+0000", 2024-09-18T07:42:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6058999999999999e-05 2024-09-18T07:42:19.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422387+0000", 2024-09-18T07:42:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0560999999999999e-05 2024-09-18T07:42:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.429278+0000", 2024-09-18T07:42:19.152 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0068906500000001 2024-09-18T07:42:19.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.153 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.153 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.153 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.429407+0000", 2024-09-18T07:42:19.153 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012889600000000001 2024-09-18T07:42:19.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:42:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.429537+0000", 2024-09-18T07:42:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013015100000000001 2024-09-18T07:42:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.154 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431858+0000", 2024-09-18T07:42:19.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023215480000000001 2024-09-18T07:42:19.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431902+0000", 2024-09-18T07:42:19.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3331999999999998e-05 2024-09-18T07:42:19.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431988+0000", 2024-09-18T07:42:19.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6834000000000002e-05 2024-09-18T07:42:19.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.156 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432895+0000", 2024-09-18T07:42:19.156 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00090677800000000001 2024-09-18T07:42:19.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.156 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432943+0000", 2024-09-18T07:42:19.156 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8143000000000003e-05 2024-09-18T07:42:19.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433051+0000", 2024-09-18T07:42:19.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107953 2024-09-18T07:42:19.157 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.157 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.157 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.158 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162333 7.8 7:1673a0b1:::benchmark_data_smithi137_74180_object57917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:42:19.158 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.424085+0000", 2024-09-18T07:42:19.158 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 437.57414951599998, 2024-09-18T07:42:19.158 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0091520620000001, 2024-09-18T07:42:19.158 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.158 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.158 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.158 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.158 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.159 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.159 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162333 2024-09-18T07:42:19.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.159 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.159 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.159 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424085+0000", 2024-09-18T07:42:19.159 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424085+0000", 2024-09-18T07:42:19.160 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424087+0000", 2024-09-18T07:42:19.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4470000000000001e-06 2024-09-18T07:42:19.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424091+0000", 2024-09-18T07:42:19.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9319999999999998e-06 2024-09-18T07:42:19.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.162 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424098+0000", 2024-09-18T07:42:19.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.187e-06 2024-09-18T07:42:19.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.162 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424104+0000", 2024-09-18T07:42:19.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8730000000000004e-06 2024-09-18T07:42:19.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.163 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.163 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424145+0000", 2024-09-18T07:42:19.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1445e-05 2024-09-18T07:42:19.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.163 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.163 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424154+0000", 2024-09-18T07:42:19.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0729999999999994e-06 2024-09-18T07:42:19.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.430409+0000", 2024-09-18T07:42:19.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0062551159999999 2024-09-18T07:42:19.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.430534+0000", 2024-09-18T07:42:19.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012458400000000001 2024-09-18T07:42:19.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:42:19.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.430647+0000", 2024-09-18T07:42:19.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112537 2024-09-18T07:42:19.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431941+0000", 2024-09-18T07:42:19.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001294915 2024-09-18T07:42:19.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432032+0000", 2024-09-18T07:42:19.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0461e-05 2024-09-18T07:42:19.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.166 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.166 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433059+0000", 2024-09-18T07:42:19.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010266990000000001 2024-09-18T07:42:19.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.166 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.166 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433100+0000", 2024-09-18T07:42:19.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1770000000000002e-05 2024-09-18T07:42:19.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.167 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.167 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433146+0000", 2024-09-18T07:42:19.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5204000000000003e-05 2024-09-18T07:42:19.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.167 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.167 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433237+0000", 2024-09-18T07:42:19.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1818999999999996e-05 2024-09-18T07:42:19.167 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.167 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.167 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.168 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162348 7.8 7:1d66cfe5:::benchmark_data_smithi137_74180_object57932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:42:19.168 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.431806+0000", 2024-09-18T07:42:19.168 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 437.566428879, 2024-09-18T07:42:19.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0022084899999999, 2024-09-18T07:42:19.168 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.168 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.168 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.168 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.168 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.168 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.168 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162348 2024-09-18T07:42:19.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.169 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.169 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431806+0000", 2024-09-18T07:42:19.169 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.170 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.170 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431806+0000", 2024-09-18T07:42:19.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.170 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.170 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431808+0000", 2024-09-18T07:42:19.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6300000000000001e-06 2024-09-18T07:42:19.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431812+0000", 2024-09-18T07:42:19.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6140000000000002e-06 2024-09-18T07:42:19.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431823+0000", 2024-09-18T07:42:19.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0658e-05 2024-09-18T07:42:19.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431830+0000", 2024-09-18T07:42:19.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6529999999999997e-06 2024-09-18T07:42:19.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.172 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431897+0000", 2024-09-18T07:42:19.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7286000000000006e-05 2024-09-18T07:42:19.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.173 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.173 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431909+0000", 2024-09-18T07:42:19.173 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1739999999999999e-05 2024-09-18T07:42:19.173 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431207+0000", 2024-09-18T07:42:19.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99929881799999998 2024-09-18T07:42:19.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431322+0000", 2024-09-18T07:42:19.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011495800000000001 2024-09-18T07:42:19.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:42:19.175 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431437+0000", 2024-09-18T07:42:19.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114752 2024-09-18T07:42:19.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.176 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433064+0000", 2024-09-18T07:42:19.176 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016264280000000001 2024-09-18T07:42:19.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.176 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.176 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433276+0000", 2024-09-18T07:42:19.176 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021219300000000001 2024-09-18T07:42:19.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.176 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.176 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433318+0000", 2024-09-18T07:42:19.177 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1999999999999998e-05 2024-09-18T07:42:19.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.177 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.177 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433856+0000", 2024-09-18T07:42:19.177 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00053855899999999998 2024-09-18T07:42:19.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.177 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.177 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433907+0000", 2024-09-18T07:42:19.177 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0954e-05 2024-09-18T07:42:19.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.177 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.178 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.434015+0000", 2024-09-18T07:42:19.178 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107247 2024-09-18T07:42:19.178 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.178 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.178 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.178 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.178 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162355 7.8 7:13f232f0:::benchmark_data_smithi137_74180_object57939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:42:19.178 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.440402+0000", 2024-09-18T07:42:19.178 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 437.55783313900002, 2024-09-18T07:42:19.178 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99385697299999998, 2024-09-18T07:42:19.178 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:19.178 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:19.179 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:19.179 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:19.179 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:19.179 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:19.179 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162355 2024-09-18T07:42:19.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.179 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:19.179 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.179 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:19.179 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440402+0000", 2024-09-18T07:42:19.179 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.179 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.180 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:19.180 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440402+0000", 2024-09-18T07:42:19.180 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:19.180 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.180 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.180 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:19.180 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440403+0000", 2024-09-18T07:42:19.180 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4750000000000001e-06 2024-09-18T07:42:19.180 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.181 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:19.181 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440409+0000", 2024-09-18T07:42:19.181 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6690000000000003e-06 2024-09-18T07:42:19.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.182 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:19.182 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440420+0000", 2024-09-18T07:42:19.182 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0594000000000001e-05 2024-09-18T07:42:19.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.182 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.182 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:19.182 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440425+0000", 2024-09-18T07:42:19.182 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2460000000000003e-06 2024-09-18T07:42:19.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.183 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.183 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.183 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440471+0000", 2024-09-18T07:42:19.183 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6397000000000003e-05 2024-09-18T07:42:19.183 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.183 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.183 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:19.183 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440479+0000", 2024-09-18T07:42:19.184 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8220000000000007e-06 2024-09-18T07:42:19.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.184 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.184 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:19.184 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432099+0000", 2024-09-18T07:42:19.184 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99161995000000003 2024-09-18T07:42:19.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.184 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.184 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:19.184 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432216+0000", 2024-09-18T07:42:19.184 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117404 2024-09-18T07:42:19.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.184 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.184 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:42:19.185 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432332+0000", 2024-09-18T07:42:19.185 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115939 2024-09-18T07:42:19.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.185 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.185 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.186 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433814+0000", 2024-09-18T07:42:19.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00148196 2024-09-18T07:42:19.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.186 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:19.186 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433827+0000", 2024-09-18T07:42:19.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2583e-05 2024-09-18T07:42:19.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.186 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.186 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.434071+0000", 2024-09-18T07:42:19.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024380199999999999 2024-09-18T07:42:19.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.187 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.187 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:19.187 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.434123+0000", 2024-09-18T07:42:19.187 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2419000000000001e-05 2024-09-18T07:42:19.187 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.187 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.187 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:19.187 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.434167+0000", 2024-09-18T07:42:19.187 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3600000000000003e-05 2024-09-18T07:42:19.187 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:19.187 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:19.187 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:19.187 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.434259+0000", 2024-09-18T07:42:19.187 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2113000000000004e-05 2024-09-18T07:42:19.188 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.188 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.188 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.188 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:19.188 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:19.188 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:42:19.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:19.190 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:19.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:19.290 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:19.328 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:19.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:19.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:19.337 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:19 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:19.337 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:19 smithi137 ceph-mon[29408]: pgmap v605: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.88k op/s 2024-09-18T07:42:19.338 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:19 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:19.338 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:19 smithi137 ceph-mon[39175]: pgmap v605: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.88k op/s 2024-09-18T07:42:19.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:19.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:19 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:19 smithi203 ceph-mon[32237]: pgmap v605: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.88k op/s 2024-09-18T07:42:19.464 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:19.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:19.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:19.566 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:19.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:19.602 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:19.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:19.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:42:19.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:19.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:19.743 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:19.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:19.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:19.881 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:19.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:19.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:19.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:20.060 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:20.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:20.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:20.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:20.206 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:20.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:20.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:20.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:20.347 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:20.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:20.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:20.449 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:20.483 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:20.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:20.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:20.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:20.619 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:20.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:20.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:20.721 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:20.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T07:42:20.747 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:20.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:20.768 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:20.769 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:20.849 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:20.871 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:20.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:20.973 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:21.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:21.004 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:21.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:21.105 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:21.127 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:42:21.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:21.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:21.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:21.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:21.264 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:21 smithi137 ceph-mon[29408]: pgmap v606: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-18T07:42:21.264 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:21 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/240109816' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:42:21.264 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:21 smithi137 ceph-mon[29408]: from='client.25960 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:42:21.264 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:21 smithi137 ceph-mon[39175]: pgmap v606: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-18T07:42:21.264 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:21 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/240109816' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:42:21.264 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:21 smithi137 ceph-mon[39175]: from='client.25960 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:42:21.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:21.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:21.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:21.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:21.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:21.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:21.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:21 smithi203 ceph-mon[32237]: pgmap v606: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-18T07:42:21.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:21 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/240109816' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:42:21.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:21 smithi203 ceph-mon[32237]: from='client.25960 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:42:21.545 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:21.579 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:21.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:21.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:21.681 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:21.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:21.717 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:21.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:21.819 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:21.848 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:21.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:21.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:21.949 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:21.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:21.986 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:21.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:22.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:22.122 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:22.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:22.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:22.224 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:22.233 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:22 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:42:22.233 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:22 smithi137 ceph-mon[29408]: from='client.25960 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:42:22.233 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:22 smithi137 ceph-mon[29408]: osdmap e350: 8 total, 8 up, 8 in 2024-09-18T07:42:22.233 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:22 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:42:22.234 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:22 smithi137 ceph-mon[39175]: from='client.25960 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:42:22.234 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:22 smithi137 ceph-mon[39175]: osdmap e350: 8 total, 8 up, 8 in 2024-09-18T07:42:22.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:22.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:22.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:22.372 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:22.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:22.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:22.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:22 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:42:22.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:22 smithi203 ceph-mon[32237]: from='client.25960 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:42:22.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:22 smithi203 ceph-mon[32237]: osdmap e350: 8 total, 8 up, 8 in 2024-09-18T07:42:22.511 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:22.540 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:22.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:22.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:22.668 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:42:22.669 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:42:22.678 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:22.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:22.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:22.698 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:42:22.698 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:42:22.732 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:42:22.732 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:42:22.767 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:42:22.767 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:42:22.780 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:22.805 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:42:22.805 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:42:22.810 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:22.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:22.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:22.841 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:42:22.842 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:42:22.880 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:42:22.880 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:42:22.912 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:22.913 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:42:22.913 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:42:22.937 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:22.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:22.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:23.039 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:23.073 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:23.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:23.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:23.175 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:23.207 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:23.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:23.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:23.233 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:42:23.236 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:42:23.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:23.345 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:23.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:23.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:23.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:23 smithi137 ceph-mon[29408]: pgmap v608: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-09-18T07:42:23.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:23 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3011527754' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:23.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:23 smithi137 ceph-mon[39175]: pgmap v608: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-09-18T07:42:23.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:23 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3011527754' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:23.447 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:23.463 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:23 smithi203 ceph-mon[32237]: pgmap v608: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-09-18T07:42:23.463 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:23 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3011527754' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:23.486 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:23.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:23.587 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:23.623 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:23.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:23.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:23.724 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:23.759 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:23.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:23.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:23.860 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:23.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:23.897 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:23.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:23.999 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:24.040 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:24.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:24.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:24.142 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:24.177 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:24.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:24.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:24.279 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:24.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:42:24.313 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:24.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:24.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:24.416 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:24.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:24 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:42:24.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:24 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:42:24.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:24 smithi137 ceph-mon[29408]: osdmap e351: 8 total, 8 up, 8 in 2024-09-18T07:42:24.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:24 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:42:24.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:24 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:42:24.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:24 smithi137 ceph-mon[39175]: osdmap e351: 8 total, 8 up, 8 in 2024-09-18T07:42:24.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:24.450 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:24.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:24.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:24 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:42:24.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:24 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:42:24.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:24 smithi203 ceph-mon[32237]: osdmap e351: 8 total, 8 up, 8 in 2024-09-18T07:42:24.551 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:24.595 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:24.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:24.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:24.697 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:24.733 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:24.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:24.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:24.868 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:24.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:24.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:24.970 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:25.006 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:25.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:25.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:25.055 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:25.103 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":609,"stamp":"2024-09-18T07:42:23.145260+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2426673715,"num_objects":37219,"num_object_clones":0,"num_object_copies":111657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37219,"num_whiteouts":0,"num_read":1579,"num_read_kb":1331,"num_write":228698,"num_write_kb":8479854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":5625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81024,"ondisk_log_size":81024,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10688596,"kb_used_data":7316420,"kb_used_omap":12,"kb_used_meta":3371635,"kb_avail":739272620,"statfs":{"total":767960285184,"available":757015162880,"internally_reserved":0,"allocated":7492014080,"data_stored":7488917283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":3452554800},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,13,11,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-852099072,"num_objects":-13002,"num_object_clones":0,"num_object_copies":-39006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-13002,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":13002,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.466059"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":623,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121901+0000","last_change":"2024-09-18T07:41:15.846355+0000","last_active":"2024-09-18T07:42:21.121901+0000","last_peered":"2024-09-18T07:42:21.121901+0000","last_clean":"2024-09-18T07:42:21.121901+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:42:21.121901+0000","last_undegraded":"2024-09-18T07:42:21.121901+0000","last_fullsized":"2024-09-18T07:42:21.121901+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:15.846257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:15.846257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:39.140539+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"323'15","reported_seq":698,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.127038+0000","last_change":"2024-09-18T07:40:17.270061+0000","last_active":"2024-09-18T07:42:21.127038+0000","last_peered":"2024-09-18T07:42:21.127038+0000","last_clean":"2024-09-18T07:42:21.127038+0000","last_became_active":"2024-09-18T07:37:43.061125+0000","last_became_peered":"2024-09-18T07:37:43.061125+0000","last_unstale":"2024-09-18T07:42:21.127038+0000","last_undegraded":"2024-09-18T07:42:21.127038+0000","last_fullsized":"2024-09-18T07:42:21.127038+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:17.269977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:17.269977+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:43.934455+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":566,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.116894+0000","last_change":"2024-09-18T07:41:58.378943+0000","last_active":"2024-09-18T07:42:21.116894+0000","last_peered":"2024-09-18T07:42:21.116894+0000","last_clean":"2024-09-18T07:42:21.116894+0000","last_became_active":"2024-09-18T07:36:13.128805+0000","last_became_peered":"2024-09-18T07:36:13.128805+0000","last_unstale":"2024-09-18T07:42:21.116894+0000","last_undegraded":"2024-09-18T07:42:21.116894+0000","last_fullsized":"2024-09-18T07:42:21.116894+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:58.378571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:58.378571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:23.350846+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":598,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.127509+0000","last_change":"2024-09-18T07:40:47.287019+0000","last_active":"2024-09-18T07:42:21.127509+0000","last_peered":"2024-09-18T07:42:21.127509+0000","last_clean":"2024-09-18T07:42:21.127509+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:42:21.127509+0000","last_undegraded":"2024-09-18T07:42:21.127509+0000","last_fullsized":"2024-09-18T07:42:21.127509+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:47.286927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.286927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":517,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.124114+0000","last_change":"2024-09-18T07:41:12.872510+0000","last_active":"2024-09-18T07:42:21.124114+0000","last_peered":"2024-09-18T07:42:21.124114+0000","last_clean":"2024-09-18T07:42:21.124114+0000","last_became_active":"2024-09-18T07:37:43.056380+0000","last_became_peered":"2024-09-18T07:37:43.056380+0000","last_unstale":"2024-09-18T07:42:21.124114+0000","last_undegraded":"2024-09-18T07:42:21.124114+0000","last_fullsized":"2024-09-18T07:42:21.124114+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:12.872387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:12.872387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:41.658070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":624,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.117546+0000","last_change":"2024-09-18T07:40:57.319270+0000","last_active":"2024-09-18T07:42:21.117546+0000","last_peered":"2024-09-18T07:42:21.117546+0000","last_clean":"2024-09-18T07:42:21.117546+0000","last_became_active":"2024-09-18T07:37:43.066508+0000","last_became_peered":"2024-09-18T07:37:43.066508+0000","last_unstale":"2024-09-18T07:42:21.117546+0000","last_undegraded":"2024-09-18T07:42:21.117546+0000","last_fullsized":"2024-09-18T07:42:21.117546+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:57.319204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:57.319204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"323'27","reported_seq":168,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.137626+0000","last_change":"2024-09-18T07:40:20.436791+0000","last_active":"2024-09-18T07:42:21.137626+0000","last_peered":"2024-09-18T07:42:21.137626+0000","last_clean":"2024-09-18T07:42:21.137626+0000","last_became_active":"2024-09-18T07:40:19.064755+0000","last_became_peered":"2024-09-18T07:40:19.064755+0000","last_unstale":"2024-09-18T07:42:21.137626+0000","last_undegraded":"2024-09-18T07:42:21.137626+0000","last_fullsized":"2024-09-18T07:42:21.137626+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:40:20.436702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:20.436702+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:46.054779+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":632,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.127013+0000","last_change":"2024-09-18T07:41:51.001121+0000","last_active":"2024-09-18T07:42:21.127013+0000","last_peered":"2024-09-18T07:42:21.127013+0000","last_clean":"2024-09-18T07:42:21.127013+0000","last_became_active":"2024-09-18T07:37:43.052346+0000","last_became_peered":"2024-09-18T07:37:43.052346+0000","last_unstale":"2024-09-18T07:42:21.127013+0000","last_undegraded":"2024-09-18T07:42:21.127013+0000","last_fullsized":"2024-09-18T07:42:21.127013+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:51.001046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:51.001046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:55.981852+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":622,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.120085+0000","last_change":"2024-09-18T07:37:43.080879+0000","last_active":"2024-09-18T07:42:21.120085+0000","last_peered":"2024-09-18T07:42:21.120085+0000","last_clean":"2024-09-18T07:42:21.120085+0000","last_became_active":"2024-09-18T07:37:43.080314+0000","last_became_peered":"2024-09-18T07:37:43.080314+0000","last_unstale":"2024-09-18T07:42:21.120085+0000","last_undegraded":"2024-09-18T07:42:21.120085+0000","last_fullsized":"2024-09-18T07:42:21.120085+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:00.392466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.392466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:39.270967+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"323'15","reported_seq":639,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.127616+0000","last_change":"2024-09-18T07:41:56.021202+0000","last_active":"2024-09-18T07:42:21.127616+0000","last_peered":"2024-09-18T07:42:21.127616+0000","last_clean":"2024-09-18T07:42:21.127616+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:42:21.127616+0000","last_undegraded":"2024-09-18T07:42:21.127616+0000","last_fullsized":"2024-09-18T07:42:21.127616+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:56.021118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:56.021118+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:16.099361+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":614,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.119294+0000","last_change":"2024-09-18T07:40:38.360192+0000","last_active":"2024-09-18T07:42:21.119294+0000","last_peered":"2024-09-18T07:42:21.119294+0000","last_clean":"2024-09-18T07:42:21.119294+0000","last_became_active":"2024-09-18T07:35:01.413443+0000","last_became_peered":"2024-09-18T07:35:01.413443+0000","last_unstale":"2024-09-18T07:42:21.119294+0000","last_undegraded":"2024-09-18T07:42:21.119294+0000","last_fullsized":"2024-09-18T07:42:21.119294+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:38.360127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.360127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":608,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.123726+0000","last_change":"2024-09-18T07:42:00.028623+0000","last_active":"2024-09-18T07:42:21.123726+0000","last_peered":"2024-09-18T07:42:21.123726+0000","last_clean":"2024-09-18T07:42:21.123726+0000","last_became_active":"2024-09-18T07:35:01.410284+0000","last_became_peered":"2024-09-18T07:35:01.410284+0000","last_unstale":"2024-09-18T07:42:21.123726+0000","last_undegraded":"2024-09-18T07:42:21.123726+0000","last_fullsized":"2024-09-18T07:42:21.123726+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:00.028559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:00.028559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:26.482816+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":608,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.117266+0000","last_change":"2024-09-18T07:41:57.358913+0000","last_active":"2024-09-18T07:42:21.117266+0000","last_peered":"2024-09-18T07:42:21.117266+0000","last_clean":"2024-09-18T07:42:21.117266+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:42:21.117266+0000","last_undegraded":"2024-09-18T07:42:21.117266+0000","last_fullsized":"2024-09-18T07:42:21.117266+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:57.358853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:57.358853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:12.134323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"323'7","reported_seq":632,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.127615+0000","last_change":"2024-09-18T07:39:26.028935+0000","last_active":"2024-09-18T07:42:21.127615+0000","last_peered":"2024-09-18T07:42:21.127615+0000","last_clean":"2024-09-18T07:42:21.127615+0000","last_became_active":"2024-09-18T07:36:02.526212+0000","last_became_peered":"2024-09-18T07:36:02.526212+0000","last_unstale":"2024-09-18T07:42:21.127615+0000","last_undegraded":"2024-09-18T07:42:21.127615+0000","last_fullsized":"2024-09-18T07:42:21.127615+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-18T07:39:26.028884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:26.028884+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:39.929629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":382,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121994+0000","last_change":"2024-09-18T07:40:41.457394+0000","last_active":"2024-09-18T07:42:21.121994+0000","last_peered":"2024-09-18T07:42:21.121994+0000","last_clean":"2024-09-18T07:42:21.121994+0000","last_became_active":"2024-09-18T07:35:01.398605+0000","last_became_peered":"2024-09-18T07:35:01.398605+0000","last_unstale":"2024-09-18T07:42:21.121994+0000","last_undegraded":"2024-09-18T07:42:21.121994+0000","last_fullsized":"2024-09-18T07:42:21.121994+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:41.457338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:41.457338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":620,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.116979+0000","last_change":"2024-09-18T07:40:50.367758+0000","last_active":"2024-09-18T07:42:21.116979+0000","last_peered":"2024-09-18T07:42:21.116979+0000","last_clean":"2024-09-18T07:42:21.116979+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:42:21.116979+0000","last_undegraded":"2024-09-18T07:42:21.116979+0000","last_fullsized":"2024-09-18T07:42:21.116979+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:50.367694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.367694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":621,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.128480+0000","last_change":"2024-09-18T07:38:56.096769+0000","last_active":"2024-09-18T07:42:21.128480+0000","last_peered":"2024-09-18T07:42:21.128480+0000","last_clean":"2024-09-18T07:42:21.128480+0000","last_became_active":"2024-09-18T07:37:43.067568+0000","last_became_peered":"2024-09-18T07:37:43.067568+0000","last_unstale":"2024-09-18T07:42:21.128480+0000","last_undegraded":"2024-09-18T07:42:21.128480+0000","last_fullsized":"2024-09-18T07:42:21.128480+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:56.096591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:56.096591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:30.878059+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":628,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.125336+0000","last_change":"2024-09-18T07:42:04.117872+0000","last_active":"2024-09-18T07:42:21.125336+0000","last_peered":"2024-09-18T07:42:21.125336+0000","last_clean":"2024-09-18T07:42:21.125336+0000","last_became_active":"2024-09-18T07:35:01.414088+0000","last_became_peered":"2024-09-18T07:35:01.414088+0000","last_unstale":"2024-09-18T07:42:21.125336+0000","last_undegraded":"2024-09-18T07:42:21.125336+0000","last_fullsized":"2024-09-18T07:42:21.125336+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:04.117653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.117653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:22.791068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"323'27","reported_seq":711,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.140535+0000","last_change":"2024-09-18T07:41:48.128836+0000","last_active":"2024-09-18T07:42:21.140535+0000","last_peered":"2024-09-18T07:42:21.140535+0000","last_clean":"2024-09-18T07:42:21.140535+0000","last_became_active":"2024-09-18T07:37:28.002858+0000","last_became_peered":"2024-09-18T07:37:28.002858+0000","last_unstale":"2024-09-18T07:42:21.140535+0000","last_undegraded":"2024-09-18T07:42:21.140535+0000","last_fullsized":"2024-09-18T07:42:21.140535+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:41:48.128756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.128756+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:01.870808+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":301,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.128305+0000","last_change":"2024-09-18T07:41:26.922547+0000","last_active":"2024-09-18T07:42:21.128305+0000","last_peered":"2024-09-18T07:42:21.128305+0000","last_clean":"2024-09-18T07:42:21.128305+0000","last_became_active":"2024-09-18T07:36:13.116416+0000","last_became_peered":"2024-09-18T07:36:13.116416+0000","last_unstale":"2024-09-18T07:42:21.128305+0000","last_undegraded":"2024-09-18T07:42:21.128305+0000","last_fullsized":"2024-09-18T07:42:21.128305+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:26.922506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:26.922506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:55.427286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":475,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.155428+0000","last_change":"2024-09-18T07:40:19.070769+0000","last_active":"2024-09-18T07:42:21.155428+0000","last_peered":"2024-09-18T07:42:21.155428+0000","last_clean":"2024-09-18T07:42:21.155428+0000","last_became_active":"2024-09-18T07:40:19.070463+0000","last_became_peered":"2024-09-18T07:40:19.070463+0000","last_unstale":"2024-09-18T07:42:21.155428+0000","last_undegraded":"2024-09-18T07:42:21.155428+0000","last_fullsized":"2024-09-18T07:42:21.155428+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:11.831891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:11.831891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:43.316131+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":618,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.127710+0000","last_change":"2024-09-18T07:39:00.130395+0000","last_active":"2024-09-18T07:42:21.127710+0000","last_peered":"2024-09-18T07:42:21.127710+0000","last_clean":"2024-09-18T07:42:21.127710+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:42:21.127710+0000","last_undegraded":"2024-09-18T07:42:21.127710+0000","last_fullsized":"2024-09-18T07:42:21.127710+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:00.130344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.130344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:28.926409+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"323'18","reported_seq":280,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.140262+0000","last_change":"2024-09-18T07:41:59.149841+0000","last_active":"2024-09-18T07:42:21.140262+0000","last_peered":"2024-09-18T07:42:21.140262+0000","last_clean":"2024-09-18T07:42:21.140262+0000","last_became_active":"2024-09-18T07:37:27.981705+0000","last_became_peered":"2024-09-18T07:37:27.981705+0000","last_unstale":"2024-09-18T07:42:21.140262+0000","last_undegraded":"2024-09-18T07:42:21.140262+0000","last_fullsized":"2024-09-18T07:42:21.140262+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:41:59.149629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:59.149629+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:25.055296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":594,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.128055+0000","last_change":"2024-09-18T07:39:37.930497+0000","last_active":"2024-09-18T07:42:21.128055+0000","last_peered":"2024-09-18T07:42:21.128055+0000","last_clean":"2024-09-18T07:42:21.128055+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:42:21.128055+0000","last_undegraded":"2024-09-18T07:42:21.128055+0000","last_fullsized":"2024-09-18T07:42:21.128055+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.930435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.930435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:13.036846+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":394,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121789+0000","last_change":"2024-09-18T07:40:10.644587+0000","last_active":"2024-09-18T07:42:21.121789+0000","last_peered":"2024-09-18T07:42:21.121789+0000","last_clean":"2024-09-18T07:42:21.121789+0000","last_became_active":"2024-09-18T07:35:01.413110+0000","last_became_peered":"2024-09-18T07:35:01.413110+0000","last_unstale":"2024-09-18T07:42:21.121789+0000","last_undegraded":"2024-09-18T07:42:21.121789+0000","last_fullsized":"2024-09-18T07:42:21.121789+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:40:10.644461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:10.644461+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:11.785210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":310,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.119605+0000","last_change":"2024-09-18T07:38:50.918758+0000","last_active":"2024-09-18T07:42:21.119605+0000","last_peered":"2024-09-18T07:42:21.119605+0000","last_clean":"2024-09-18T07:42:21.119605+0000","last_became_active":"2024-09-18T07:36:13.116295+0000","last_became_peered":"2024-09-18T07:36:13.116295+0000","last_unstale":"2024-09-18T07:42:21.119605+0000","last_undegraded":"2024-09-18T07:42:21.119605+0000","last_fullsized":"2024-09-18T07:42:21.119605+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:38:50.918686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:38:50.918686+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:35.433557+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"323'20","reported_seq":679,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121940+0000","last_change":"2024-09-18T07:40:57.823251+0000","last_active":"2024-09-18T07:42:21.121940+0000","last_peered":"2024-09-18T07:42:21.121940+0000","last_clean":"2024-09-18T07:42:21.121940+0000","last_became_active":"2024-09-18T07:35:01.410926+0000","last_became_peered":"2024-09-18T07:35:01.410926+0000","last_unstale":"2024-09-18T07:42:21.121940+0000","last_undegraded":"2024-09-18T07:42:21.121940+0000","last_fullsized":"2024-09-18T07:42:21.121940+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:40:57.823151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:57.823151+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"240'12","reported_seq":310,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.155494+0000","last_change":"2024-09-18T07:40:43.112526+0000","last_active":"2024-09-18T07:42:21.155494+0000","last_peered":"2024-09-18T07:42:21.155494+0000","last_clean":"2024-09-18T07:42:21.155494+0000","last_became_active":"2024-09-18T07:37:43.060715+0000","last_became_peered":"2024-09-18T07:37:43.060715+0000","last_unstale":"2024-09-18T07:42:21.155494+0000","last_undegraded":"2024-09-18T07:42:21.155494+0000","last_fullsized":"2024-09-18T07:42:21.155494+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:40:43.112459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:43.112459+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:21.098107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"350'14310","reported_seq":14553,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:22.359843+0000","last_change":"2024-09-18T07:42:00.674258+0000","last_active":"2024-09-18T07:42:22.359843+0000","last_peered":"2024-09-18T07:42:22.359843+0000","last_clean":"2024-09-18T07:42:22.359843+0000","last_became_active":"2024-09-18T07:39:29.700477+0000","last_became_peered":"2024-09-18T07:39:29.700477+0000","last_unstale":"2024-09-18T07:42:22.359843+0000","last_undegraded":"2024-09-18T07:42:22.359843+0000","last_fullsized":"2024-09-18T07:42:22.359843+0000","mapping_epoch":289,"log_start":"330'9700","ondisk_log_start":"330'9700","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'12643","last_scrub_stamp":"2024-09-18T07:42:00.674143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:42:00.674143+0000","objects_scrubbed":3950,"log_size":4610,"log_dups_size":0,"ondisk_log_size":4610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:17.584126+0000","scrub_duration":1266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146669568,"num_objects":2238,"num_object_clones":0,"num_object_copies":6714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14310,"num_write_kb":529536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":179,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.119056+0000","last_change":"2024-09-18T07:40:41.373625+0000","last_active":"2024-09-18T07:42:21.119056+0000","last_peered":"2024-09-18T07:42:21.119056+0000","last_clean":"2024-09-18T07:42:21.119056+0000","last_became_active":"2024-09-18T07:39:04.962047+0000","last_became_peered":"2024-09-18T07:39:04.962047+0000","last_unstale":"2024-09-18T07:42:21.119056+0000","last_undegraded":"2024-09-18T07:42:21.119056+0000","last_fullsized":"2024-09-18T07:42:21.119056+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:41.373545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:41.373545+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"323'27","reported_seq":364,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.120708+0000","last_change":"2024-09-18T07:38:54.967706+0000","last_active":"2024-09-18T07:42:21.120708+0000","last_peered":"2024-09-18T07:42:21.120708+0000","last_clean":"2024-09-18T07:42:21.120708+0000","last_became_active":"2024-09-18T07:36:13.119098+0000","last_became_peered":"2024-09-18T07:36:13.119098+0000","last_unstale":"2024-09-18T07:42:21.120708+0000","last_undegraded":"2024-09-18T07:42:21.120708+0000","last_fullsized":"2024-09-18T07:42:21.120708+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:38:54.967643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:38:54.967643+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:29.535790+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":640,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.118216+0000","last_change":"2024-09-18T07:39:29.049261+0000","last_active":"2024-09-18T07:42:21.118216+0000","last_peered":"2024-09-18T07:42:21.118216+0000","last_clean":"2024-09-18T07:42:21.118216+0000","last_became_active":"2024-09-18T07:35:01.407339+0000","last_became_peered":"2024-09-18T07:35:01.407339+0000","last_unstale":"2024-09-18T07:42:21.118216+0000","last_undegraded":"2024-09-18T07:42:21.118216+0000","last_fullsized":"2024-09-18T07:42:21.118216+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.049209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.049209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:20.464343+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":773,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.128613+0000","last_change":"2024-09-18T07:41:27.965286+0000","last_active":"2024-09-18T07:42:21.128613+0000","last_peered":"2024-09-18T07:42:21.128613+0000","last_clean":"2024-09-18T07:42:21.128613+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:42:21.128613+0000","last_undegraded":"2024-09-18T07:42:21.128613+0000","last_fullsized":"2024-09-18T07:42:21.128613+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:41:27.965205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.965205+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:47.540373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":605,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.119724+0000","last_change":"2024-09-18T07:42:06.507229+0000","last_active":"2024-09-18T07:42:21.119724+0000","last_peered":"2024-09-18T07:42:21.119724+0000","last_clean":"2024-09-18T07:42:21.119724+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:42:21.119724+0000","last_undegraded":"2024-09-18T07:42:21.119724+0000","last_fullsized":"2024-09-18T07:42:21.119724+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.507166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.507166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:10.596707+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"350'14369","reported_seq":14582,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.637850+0000","last_change":"2024-09-18T07:40:47.444123+0000","last_active":"2024-09-18T07:42:21.637850+0000","last_peered":"2024-09-18T07:42:21.637850+0000","last_clean":"2024-09-18T07:42:21.637850+0000","last_became_active":"2024-09-18T07:39:29.704590+0000","last_became_peered":"2024-09-18T07:39:29.704590+0000","last_unstale":"2024-09-18T07:42:21.637850+0000","last_undegraded":"2024-09-18T07:42:21.637850+0000","last_fullsized":"2024-09-18T07:42:21.637850+0000","mapping_epoch":289,"log_start":"330'9900","ondisk_log_start":"330'9900","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'7163","last_scrub_stamp":"2024-09-18T07:40:47.443974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.443974+0000","objects_scrubbed":7103,"log_size":4469,"log_dups_size":0,"ondisk_log_size":4469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:13.769013+0000","scrub_duration":1939,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151781400,"num_objects":2317,"num_object_clones":0,"num_object_copies":6951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2317,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14369,"num_write_kb":533889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":187,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.128753+0000","last_change":"2024-09-18T07:41:53.004272+0000","last_active":"2024-09-18T07:42:21.128753+0000","last_peered":"2024-09-18T07:42:21.128753+0000","last_clean":"2024-09-18T07:42:21.128753+0000","last_became_active":"2024-09-18T07:39:04.955645+0000","last_became_peered":"2024-09-18T07:39:04.955645+0000","last_unstale":"2024-09-18T07:42:21.128753+0000","last_undegraded":"2024-09-18T07:42:21.128753+0000","last_fullsized":"2024-09-18T07:42:21.128753+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:41:53.004186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:41:53.004186+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:16.639727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"323'30","reported_seq":714,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.117893+0000","last_change":"2024-09-18T07:39:00.919075+0000","last_active":"2024-09-18T07:42:21.117893+0000","last_peered":"2024-09-18T07:42:21.117893+0000","last_clean":"2024-09-18T07:42:21.117893+0000","last_became_active":"2024-09-18T07:35:01.418909+0000","last_became_peered":"2024-09-18T07:35:01.418909+0000","last_unstale":"2024-09-18T07:42:21.117893+0000","last_undegraded":"2024-09-18T07:42:21.117893+0000","last_fullsized":"2024-09-18T07:42:21.117893+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:39:00.918860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.918860+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:39.501904+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":345,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121619+0000","last_change":"2024-09-18T07:40:19.069243+0000","last_active":"2024-09-18T07:42:21.121619+0000","last_peered":"2024-09-18T07:42:21.121619+0000","last_clean":"2024-09-18T07:42:21.121619+0000","last_became_active":"2024-09-18T07:40:19.068296+0000","last_became_peered":"2024-09-18T07:40:19.068296+0000","last_unstale":"2024-09-18T07:42:21.121619+0000","last_undegraded":"2024-09-18T07:42:21.121619+0000","last_fullsized":"2024-09-18T07:42:21.121619+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:06.987428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:06.987428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:34.748523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":625,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.127188+0000","last_change":"2024-09-18T07:42:09.068025+0000","last_active":"2024-09-18T07:42:21.127188+0000","last_peered":"2024-09-18T07:42:21.127188+0000","last_clean":"2024-09-18T07:42:21.127188+0000","last_became_active":"2024-09-18T07:35:01.401024+0000","last_became_peered":"2024-09-18T07:35:01.401024+0000","last_unstale":"2024-09-18T07:42:21.127188+0000","last_undegraded":"2024-09-18T07:42:21.127188+0000","last_fullsized":"2024-09-18T07:42:21.127188+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:09.067941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:09.067941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:25.306181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":591,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.127598+0000","last_change":"2024-09-18T07:40:44.219626+0000","last_active":"2024-09-18T07:42:21.127598+0000","last_peered":"2024-09-18T07:42:21.127598+0000","last_clean":"2024-09-18T07:42:21.127598+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:42:21.127598+0000","last_undegraded":"2024-09-18T07:42:21.127598+0000","last_fullsized":"2024-09-18T07:42:21.127598+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:44.219547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:44.219547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"350'14577","reported_seq":14789,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:22.377616+0000","last_change":"2024-09-18T07:41:25.657036+0000","last_active":"2024-09-18T07:42:22.377616+0000","last_peered":"2024-09-18T07:42:22.377616+0000","last_clean":"2024-09-18T07:42:22.377616+0000","last_became_active":"2024-09-18T07:39:29.702953+0000","last_became_peered":"2024-09-18T07:39:29.702953+0000","last_unstale":"2024-09-18T07:42:22.377616+0000","last_undegraded":"2024-09-18T07:42:22.377616+0000","last_fullsized":"2024-09-18T07:42:22.377616+0000","mapping_epoch":289,"log_start":"330'10000","ondisk_log_start":"330'10000","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'10075","last_scrub_stamp":"2024-09-18T07:41:25.656756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:41:25.656756+0000","objects_scrubbed":6968,"log_size":4577,"log_dups_size":0,"ondisk_log_size":4577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:27.089508+0000","scrub_duration":2236,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156958720,"num_objects":2395,"num_object_clones":0,"num_object_copies":7185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14577,"num_write_kb":543104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":184,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.119898+0000","last_change":"2024-09-18T07:41:19.479946+0000","last_active":"2024-09-18T07:42:21.119898+0000","last_peered":"2024-09-18T07:42:21.119898+0000","last_clean":"2024-09-18T07:42:21.119898+0000","last_became_active":"2024-09-18T07:39:04.945385+0000","last_became_peered":"2024-09-18T07:39:04.945385+0000","last_unstale":"2024-09-18T07:42:21.119898+0000","last_undegraded":"2024-09-18T07:42:21.119898+0000","last_fullsized":"2024-09-18T07:42:21.119898+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:41:19.479442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:41:19.479442+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:41.906548+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"323'28","reported_seq":695,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.127213+0000","last_change":"2024-09-18T07:40:56.183888+0000","last_active":"2024-09-18T07:42:21.127213+0000","last_peered":"2024-09-18T07:42:21.127213+0000","last_clean":"2024-09-18T07:42:21.127213+0000","last_became_active":"2024-09-18T07:37:43.096236+0000","last_became_peered":"2024-09-18T07:37:43.096236+0000","last_unstale":"2024-09-18T07:42:21.127213+0000","last_undegraded":"2024-09-18T07:42:21.127213+0000","last_fullsized":"2024-09-18T07:42:21.127213+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-18T07:40:56.183802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.183802+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:23.795324+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":608,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.122214+0000","last_change":"2024-09-18T07:39:28.750779+0000","last_active":"2024-09-18T07:42:21.122214+0000","last_peered":"2024-09-18T07:42:21.122214+0000","last_clean":"2024-09-18T07:42:21.122214+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:42:21.122214+0000","last_undegraded":"2024-09-18T07:42:21.122214+0000","last_fullsized":"2024-09-18T07:42:21.122214+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:39:28.750720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.750720+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:13.105049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":756,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.117773+0000","last_change":"2024-09-18T07:42:04.428459+0000","last_active":"2024-09-18T07:42:21.117773+0000","last_peered":"2024-09-18T07:42:21.117773+0000","last_clean":"2024-09-18T07:42:21.117773+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:42:21.117773+0000","last_undegraded":"2024-09-18T07:42:21.117773+0000","last_fullsized":"2024-09-18T07:42:21.117773+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:42:04.428395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.428395+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:17.768730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":397,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121689+0000","last_change":"2024-09-18T07:38:58.528134+0000","last_active":"2024-09-18T07:42:21.121689+0000","last_peered":"2024-09-18T07:42:21.121689+0000","last_clean":"2024-09-18T07:42:21.121689+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:42:21.121689+0000","last_undegraded":"2024-09-18T07:42:21.121689+0000","last_fullsized":"2024-09-18T07:42:21.121689+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:58.528059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:58.528059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:39.895035+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"350'14317","reported_seq":14534,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:22.064135+0000","last_change":"2024-09-18T07:40:52.501916+0000","last_active":"2024-09-18T07:42:22.064135+0000","last_peered":"2024-09-18T07:42:22.064135+0000","last_clean":"2024-09-18T07:42:22.064135+0000","last_became_active":"2024-09-18T07:39:29.702301+0000","last_became_peered":"2024-09-18T07:39:29.702301+0000","last_unstale":"2024-09-18T07:42:22.064135+0000","last_undegraded":"2024-09-18T07:42:22.064135+0000","last_fullsized":"2024-09-18T07:42:22.064135+0000","mapping_epoch":289,"log_start":"329'9600","ondisk_log_start":"329'9600","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7465","last_scrub_stamp":"2024-09-18T07:40:52.501716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:52.501716+0000","objects_scrubbed":7371,"log_size":4717,"log_dups_size":0,"ondisk_log_size":4717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151453696,"num_objects":2311,"num_object_clones":0,"num_object_copies":6933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14317,"num_write_kb":532096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":347,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.155304+0000","last_change":"2024-09-18T07:40:38.066508+0000","last_active":"2024-09-18T07:42:21.155304+0000","last_peered":"2024-09-18T07:42:21.155304+0000","last_clean":"2024-09-18T07:42:21.155304+0000","last_became_active":"2024-09-18T07:39:04.941752+0000","last_became_peered":"2024-09-18T07:39:04.941752+0000","last_unstale":"2024-09-18T07:42:21.155304+0000","last_undegraded":"2024-09-18T07:42:21.155304+0000","last_fullsized":"2024-09-18T07:42:21.155304+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:38.066443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.066443+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:19.103720+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"323'20","reported_seq":359,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.119513+0000","last_change":"2024-09-18T07:42:03.427924+0000","last_active":"2024-09-18T07:42:21.119513+0000","last_peered":"2024-09-18T07:42:21.119513+0000","last_clean":"2024-09-18T07:42:21.119513+0000","last_became_active":"2024-09-18T07:36:13.118876+0000","last_became_peered":"2024-09-18T07:36:13.118876+0000","last_unstale":"2024-09-18T07:42:21.119513+0000","last_undegraded":"2024-09-18T07:42:21.119513+0000","last_fullsized":"2024-09-18T07:42:21.119513+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:42:03.427843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:03.427843+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:24.815361+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":625,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.120140+0000","last_change":"2024-09-18T07:40:53.350083+0000","last_active":"2024-09-18T07:42:21.120140+0000","last_peered":"2024-09-18T07:42:21.120140+0000","last_clean":"2024-09-18T07:42:21.120140+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:42:21.120140+0000","last_undegraded":"2024-09-18T07:42:21.120140+0000","last_fullsized":"2024-09-18T07:42:21.120140+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:53.350019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:53.350019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"240'10","reported_seq":775,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.124367+0000","last_change":"2024-09-18T07:40:50.839710+0000","last_active":"2024-09-18T07:42:21.124367+0000","last_peered":"2024-09-18T07:42:21.124367+0000","last_clean":"2024-09-18T07:42:21.124367+0000","last_became_active":"2024-09-18T07:37:43.167850+0000","last_became_peered":"2024-09-18T07:37:43.167850+0000","last_unstale":"2024-09-18T07:42:21.124367+0000","last_undegraded":"2024-09-18T07:42:21.124367+0000","last_fullsized":"2024-09-18T07:42:21.124367+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'10","last_scrub_stamp":"2024-09-18T07:40:50.839504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.839504+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":587,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.123866+0000","last_change":"2024-09-18T07:39:00.759483+0000","last_active":"2024-09-18T07:42:21.123866+0000","last_peered":"2024-09-18T07:42:21.123866+0000","last_clean":"2024-09-18T07:42:21.123866+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:42:21.123866+0000","last_undegraded":"2024-09-18T07:42:21.123866+0000","last_fullsized":"2024-09-18T07:42:21.123866+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:00.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:35.026613+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"350'14041","reported_seq":14221,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.637396+0000","last_change":"2024-09-18T07:42:04.807783+0000","last_active":"2024-09-18T07:42:21.637396+0000","last_peered":"2024-09-18T07:42:21.637396+0000","last_clean":"2024-09-18T07:42:21.637396+0000","last_became_active":"2024-09-18T07:39:29.703475+0000","last_became_peered":"2024-09-18T07:39:29.703475+0000","last_unstale":"2024-09-18T07:42:21.637396+0000","last_undegraded":"2024-09-18T07:42:21.637396+0000","last_fullsized":"2024-09-18T07:42:21.637396+0000","mapping_epoch":289,"log_start":"330'9600","ondisk_log_start":"330'9600","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'12776","last_scrub_stamp":"2024-09-18T07:42:04.807635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.807635+0000","objects_scrubbed":3710,"log_size":4441,"log_dups_size":0,"ondisk_log_size":4441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:27.077916+0000","scrub_duration":1049,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157483008,"num_objects":2403,"num_object_clones":0,"num_object_copies":7209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14041,"num_write_kb":526208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":347,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.155534+0000","last_change":"2024-09-18T07:40:18.954153+0000","last_active":"2024-09-18T07:42:21.155534+0000","last_peered":"2024-09-18T07:42:21.155534+0000","last_clean":"2024-09-18T07:42:21.155534+0000","last_became_active":"2024-09-18T07:39:04.948571+0000","last_became_peered":"2024-09-18T07:39:04.948571+0000","last_unstale":"2024-09-18T07:42:21.155534+0000","last_undegraded":"2024-09-18T07:42:21.155534+0000","last_fullsized":"2024-09-18T07:42:21.155534+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:18.954089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:18.954089+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:40.336036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"323'29","reported_seq":720,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.124994+0000","last_change":"2024-09-18T07:41:16.895792+0000","last_active":"2024-09-18T07:42:21.124994+0000","last_peered":"2024-09-18T07:42:21.124994+0000","last_clean":"2024-09-18T07:42:21.124994+0000","last_became_active":"2024-09-18T07:35:01.415275+0000","last_became_peered":"2024-09-18T07:35:01.415275+0000","last_unstale":"2024-09-18T07:42:21.124994+0000","last_undegraded":"2024-09-18T07:42:21.124994+0000","last_fullsized":"2024-09-18T07:42:21.124994+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'29","last_scrub_stamp":"2024-09-18T07:41:16.895595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:16.895595+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:34.282784+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":301,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.140616+0000","last_change":"2024-09-18T07:37:46.831436+0000","last_active":"2024-09-18T07:42:21.140616+0000","last_peered":"2024-09-18T07:42:21.140616+0000","last_clean":"2024-09-18T07:42:21.140616+0000","last_became_active":"2024-09-18T07:36:13.118553+0000","last_became_peered":"2024-09-18T07:36:13.118553+0000","last_unstale":"2024-09-18T07:42:21.140616+0000","last_undegraded":"2024-09-18T07:42:21.140616+0000","last_fullsized":"2024-09-18T07:42:21.140616+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:46.831373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:37:46.831373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:29.975230+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":250,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.153942+0000","last_change":"2024-09-18T07:41:24.300282+0000","last_active":"2024-09-18T07:42:21.153942+0000","last_peered":"2024-09-18T07:42:21.153942+0000","last_clean":"2024-09-18T07:42:21.153942+0000","last_became_active":"2024-09-18T07:37:43.070009+0000","last_became_peered":"2024-09-18T07:37:43.070009+0000","last_unstale":"2024-09-18T07:42:21.153942+0000","last_undegraded":"2024-09-18T07:42:21.153942+0000","last_fullsized":"2024-09-18T07:42:21.153942+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:24.300235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:24.300235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:52.556718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":242,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.153895+0000","last_change":"2024-09-18T07:40:10.073596+0000","last_active":"2024-09-18T07:42:21.153895+0000","last_peered":"2024-09-18T07:42:21.153895+0000","last_clean":"2024-09-18T07:42:21.153895+0000","last_became_active":"2024-09-18T07:37:43.051740+0000","last_became_peered":"2024-09-18T07:37:43.051740+0000","last_unstale":"2024-09-18T07:42:21.153895+0000","last_undegraded":"2024-09-18T07:42:21.153895+0000","last_fullsized":"2024-09-18T07:42:21.153895+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:10.073499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:10.073499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:33.283517+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"350'14236","reported_seq":14407,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:22.593816+0000","last_change":"2024-09-18T07:42:10.328791+0000","last_active":"2024-09-18T07:42:22.593816+0000","last_peered":"2024-09-18T07:42:22.593816+0000","last_clean":"2024-09-18T07:42:22.593816+0000","last_became_active":"2024-09-18T07:39:29.701239+0000","last_became_peered":"2024-09-18T07:39:29.701239+0000","last_unstale":"2024-09-18T07:42:22.593816+0000","last_undegraded":"2024-09-18T07:42:22.593816+0000","last_fullsized":"2024-09-18T07:42:22.593816+0000","mapping_epoch":289,"log_start":"325'8700","ondisk_log_start":"325'8700","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'13234","last_scrub_stamp":"2024-09-18T07:42:10.328441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:42:10.328441+0000","objects_scrubbed":3282,"log_size":5536,"log_dups_size":0,"ondisk_log_size":5536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:26.764761+0000","scrub_duration":1183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146538496,"num_objects":2236,"num_object_clones":0,"num_object_copies":6708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14236,"num_write_kb":527104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":179,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.116741+0000","last_change":"2024-09-18T07:40:43.346516+0000","last_active":"2024-09-18T07:42:21.116741+0000","last_peered":"2024-09-18T07:42:21.116741+0000","last_clean":"2024-09-18T07:42:21.116741+0000","last_became_active":"2024-09-18T07:39:04.950564+0000","last_became_peered":"2024-09-18T07:39:04.950564+0000","last_unstale":"2024-09-18T07:42:21.116741+0000","last_undegraded":"2024-09-18T07:42:21.116741+0000","last_fullsized":"2024-09-18T07:42:21.116741+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:43.346447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:43.346447+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"323'41","reported_seq":479,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.140677+0000","last_change":"2024-09-18T07:41:55.160514+0000","last_active":"2024-09-18T07:42:21.140677+0000","last_peered":"2024-09-18T07:42:21.140677+0000","last_clean":"2024-09-18T07:42:21.140677+0000","last_became_active":"2024-09-18T07:36:13.115278+0000","last_became_peered":"2024-09-18T07:36:13.115278+0000","last_unstale":"2024-09-18T07:42:21.140677+0000","last_undegraded":"2024-09-18T07:42:21.140677+0000","last_fullsized":"2024-09-18T07:42:21.140677+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'41","last_scrub_stamp":"2024-09-18T07:41:55.160421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:55.160421+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:01.495646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":633,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.127915+0000","last_change":"2024-09-18T07:42:03.028626+0000","last_active":"2024-09-18T07:42:21.127915+0000","last_peered":"2024-09-18T07:42:21.127915+0000","last_clean":"2024-09-18T07:42:21.127915+0000","last_became_active":"2024-09-18T07:37:43.060930+0000","last_became_peered":"2024-09-18T07:37:43.060930+0000","last_unstale":"2024-09-18T07:42:21.127915+0000","last_undegraded":"2024-09-18T07:42:21.127915+0000","last_fullsized":"2024-09-18T07:42:21.127915+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:03.028558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:03.028558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:12.110802+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":649,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.119586+0000","last_change":"2024-09-18T07:42:00.366508+0000","last_active":"2024-09-18T07:42:21.119586+0000","last_peered":"2024-09-18T07:42:21.119586+0000","last_clean":"2024-09-18T07:42:21.119586+0000","last_became_active":"2024-09-18T07:37:43.076595+0000","last_became_peered":"2024-09-18T07:37:43.076595+0000","last_unstale":"2024-09-18T07:42:21.119586+0000","last_undegraded":"2024-09-18T07:42:21.119586+0000","last_fullsized":"2024-09-18T07:42:21.119586+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:42:00.366441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:42:00.366441+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:21.576301+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":591,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.125992+0000","last_change":"2024-09-18T07:39:25.741714+0000","last_active":"2024-09-18T07:42:21.125992+0000","last_peered":"2024-09-18T07:42:21.125992+0000","last_clean":"2024-09-18T07:42:21.125992+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:42:21.125992+0000","last_undegraded":"2024-09-18T07:42:21.125992+0000","last_fullsized":"2024-09-18T07:42:21.125992+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:25.741643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.741643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:38.000377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":620,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.117319+0000","last_change":"2024-09-18T07:40:05.858885+0000","last_active":"2024-09-18T07:42:21.117319+0000","last_peered":"2024-09-18T07:42:21.117319+0000","last_clean":"2024-09-18T07:42:21.117319+0000","last_became_active":"2024-09-18T07:35:01.418809+0000","last_became_peered":"2024-09-18T07:35:01.418809+0000","last_unstale":"2024-09-18T07:42:21.117319+0000","last_undegraded":"2024-09-18T07:42:21.117319+0000","last_fullsized":"2024-09-18T07:42:21.117319+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:05.858804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:05.858804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:44.323523+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"350'14310","reported_seq":14525,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:22.066096+0000","last_change":"2024-09-18T07:40:50.786329+0000","last_active":"2024-09-18T07:42:22.066096+0000","last_peered":"2024-09-18T07:42:22.066096+0000","last_clean":"2024-09-18T07:42:22.066096+0000","last_became_active":"2024-09-18T07:39:29.702461+0000","last_became_peered":"2024-09-18T07:39:29.702461+0000","last_unstale":"2024-09-18T07:42:22.066096+0000","last_undegraded":"2024-09-18T07:42:22.066096+0000","last_fullsized":"2024-09-18T07:42:22.066096+0000","mapping_epoch":289,"log_start":"329'9600","ondisk_log_start":"329'9600","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7357","last_scrub_stamp":"2024-09-18T07:40:50.786248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.786248+0000","objects_scrubbed":7280,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148242432,"num_objects":2262,"num_object_clones":0,"num_object_copies":6786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14310,"num_write_kb":530304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":447,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121883+0000","last_change":"2024-09-18T07:40:44.476464+0000","last_active":"2024-09-18T07:42:21.121883+0000","last_peered":"2024-09-18T07:42:21.121883+0000","last_clean":"2024-09-18T07:42:21.121883+0000","last_became_active":"2024-09-18T07:39:04.944921+0000","last_became_peered":"2024-09-18T07:39:04.944921+0000","last_unstale":"2024-09-18T07:42:21.121883+0000","last_undegraded":"2024-09-18T07:42:21.121883+0000","last_fullsized":"2024-09-18T07:42:21.121883+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:44.476278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:44.476278+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:20.701318+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"323'17","reported_seq":362,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.118902+0000","last_change":"2024-09-18T07:41:54.340669+0000","last_active":"2024-09-18T07:42:21.118902+0000","last_peered":"2024-09-18T07:42:21.118902+0000","last_clean":"2024-09-18T07:42:21.118902+0000","last_became_active":"2024-09-18T07:36:13.117924+0000","last_became_peered":"2024-09-18T07:36:13.117924+0000","last_unstale":"2024-09-18T07:42:21.118902+0000","last_undegraded":"2024-09-18T07:42:21.118902+0000","last_fullsized":"2024-09-18T07:42:21.118902+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'17","last_scrub_stamp":"2024-09-18T07:41:54.340587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.340587+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:05.584840+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":332,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.137532+0000","last_change":"2024-09-18T07:42:02.127909+0000","last_active":"2024-09-18T07:42:21.137532+0000","last_peered":"2024-09-18T07:42:21.137532+0000","last_clean":"2024-09-18T07:42:21.137532+0000","last_became_active":"2024-09-18T07:37:43.050971+0000","last_became_peered":"2024-09-18T07:37:43.050971+0000","last_unstale":"2024-09-18T07:42:21.137532+0000","last_undegraded":"2024-09-18T07:42:21.137532+0000","last_fullsized":"2024-09-18T07:42:21.137532+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:02.127517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:02.127517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:03.074068+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":628,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.137713+0000","last_change":"2024-09-18T07:41:26.957957+0000","last_active":"2024-09-18T07:42:21.137713+0000","last_peered":"2024-09-18T07:42:21.137713+0000","last_clean":"2024-09-18T07:42:21.137713+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:42:21.137713+0000","last_undegraded":"2024-09-18T07:42:21.137713+0000","last_fullsized":"2024-09-18T07:42:21.137713+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:41:26.957881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:41:26.957881+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:55.274602+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":625,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.137561+0000","last_change":"2024-09-18T07:42:06.212915+0000","last_active":"2024-09-18T07:42:21.137561+0000","last_peered":"2024-09-18T07:42:21.137561+0000","last_clean":"2024-09-18T07:42:21.137561+0000","last_became_active":"2024-09-18T07:35:01.403680+0000","last_became_peered":"2024-09-18T07:35:01.403680+0000","last_unstale":"2024-09-18T07:42:21.137561+0000","last_undegraded":"2024-09-18T07:42:21.137561+0000","last_fullsized":"2024-09-18T07:42:21.137561+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.212683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.212683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:33.302456+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":401,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.122038+0000","last_change":"2024-09-18T07:40:54.574254+0000","last_active":"2024-09-18T07:42:21.122038+0000","last_peered":"2024-09-18T07:42:21.122038+0000","last_clean":"2024-09-18T07:42:21.122038+0000","last_became_active":"2024-09-18T07:37:43.060019+0000","last_became_peered":"2024-09-18T07:37:43.060019+0000","last_unstale":"2024-09-18T07:42:21.122038+0000","last_undegraded":"2024-09-18T07:42:21.122038+0000","last_fullsized":"2024-09-18T07:42:21.122038+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:40:54.574077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:54.574077+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"350'14322","reported_seq":14535,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:22.590787+0000","last_change":"2024-09-18T07:40:46.264109+0000","last_active":"2024-09-18T07:42:22.590787+0000","last_peered":"2024-09-18T07:42:22.590787+0000","last_clean":"2024-09-18T07:42:22.590787+0000","last_became_active":"2024-09-18T07:39:29.701554+0000","last_became_peered":"2024-09-18T07:39:29.701554+0000","last_unstale":"2024-09-18T07:42:22.590787+0000","last_undegraded":"2024-09-18T07:42:22.590787+0000","last_fullsized":"2024-09-18T07:42:22.590787+0000","mapping_epoch":289,"log_start":"325'8800","ondisk_log_start":"325'8800","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'7054","last_scrub_stamp":"2024-09-18T07:40:46.263923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:46.263923+0000","objects_scrubbed":6963,"log_size":5522,"log_dups_size":0,"ondisk_log_size":5522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:14.068650+0000","scrub_duration":2499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150732800,"num_objects":2300,"num_object_clones":0,"num_object_copies":6900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14322,"num_write_kb":531904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":339,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.154160+0000","last_change":"2024-09-18T07:39:05.333676+0000","last_active":"2024-09-18T07:42:21.154160+0000","last_peered":"2024-09-18T07:42:21.154160+0000","last_clean":"2024-09-18T07:42:21.154160+0000","last_became_active":"2024-09-18T07:39:04.938080+0000","last_became_peered":"2024-09-18T07:39:04.938080+0000","last_unstale":"2024-09-18T07:42:21.154160+0000","last_undegraded":"2024-09-18T07:42:21.154160+0000","last_fullsized":"2024-09-18T07:42:21.154160+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.333608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.333608+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:32.330218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"323'24","reported_seq":652,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.122040+0000","last_change":"2024-09-18T07:41:27.030793+0000","last_active":"2024-09-18T07:42:21.122040+0000","last_peered":"2024-09-18T07:42:21.122040+0000","last_clean":"2024-09-18T07:42:21.122040+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:42:21.122040+0000","last_undegraded":"2024-09-18T07:42:21.122040+0000","last_fullsized":"2024-09-18T07:42:21.122040+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'24","last_scrub_stamp":"2024-09-18T07:41:27.030705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.030705+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:29.631547+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":605,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121657+0000","last_change":"2024-09-18T07:40:19.694501+0000","last_active":"2024-09-18T07:42:21.121657+0000","last_peered":"2024-09-18T07:42:21.121657+0000","last_clean":"2024-09-18T07:42:21.121657+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:42:21.121657+0000","last_undegraded":"2024-09-18T07:42:21.121657+0000","last_fullsized":"2024-09-18T07:42:21.121657+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:40:19.694421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:19.694421+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:36.061992+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":819,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121077+0000","last_change":"2024-09-18T07:40:47.554230+0000","last_active":"2024-09-18T07:42:21.121077+0000","last_peered":"2024-09-18T07:42:21.121077+0000","last_clean":"2024-09-18T07:42:21.121077+0000","last_became_active":"2024-09-18T07:35:01.400524+0000","last_became_peered":"2024-09-18T07:35:01.400524+0000","last_unstale":"2024-09-18T07:42:21.121077+0000","last_undegraded":"2024-09-18T07:42:21.121077+0000","last_fullsized":"2024-09-18T07:42:21.121077+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T07:40:47.554161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.554161+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":446,"num_read_kb":441,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":617,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.118994+0000","last_change":"2024-09-18T07:42:02.399342+0000","last_active":"2024-09-18T07:42:21.118994+0000","last_peered":"2024-09-18T07:42:21.118994+0000","last_clean":"2024-09-18T07:42:21.118994+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:42:21.118994+0000","last_undegraded":"2024-09-18T07:42:21.118994+0000","last_fullsized":"2024-09-18T07:42:21.118994+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:02.399269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:02.399269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:08.509887+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":309,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.117357+0000","last_change":"2024-09-18T07:40:12.269264+0000","last_active":"2024-09-18T07:42:21.117357+0000","last_peered":"2024-09-18T07:42:21.117357+0000","last_clean":"2024-09-18T07:42:21.117357+0000","last_became_active":"2024-09-18T07:36:13.117463+0000","last_became_peered":"2024-09-18T07:36:13.117463+0000","last_unstale":"2024-09-18T07:42:21.117357+0000","last_undegraded":"2024-09-18T07:42:21.117357+0000","last_fullsized":"2024-09-18T07:42:21.117357+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:12.269201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:12.269201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:20.549505+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"350'13997","reported_seq":14208,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:22.390636+0000","last_change":"2024-09-18T07:41:22.252954+0000","last_active":"2024-09-18T07:42:22.390636+0000","last_peered":"2024-09-18T07:42:22.390636+0000","last_clean":"2024-09-18T07:42:22.390636+0000","last_became_active":"2024-09-18T07:39:29.699733+0000","last_became_peered":"2024-09-18T07:39:29.699733+0000","last_unstale":"2024-09-18T07:42:22.390636+0000","last_undegraded":"2024-09-18T07:42:22.390636+0000","last_fullsized":"2024-09-18T07:42:22.390636+0000","mapping_epoch":289,"log_start":"329'9400","ondisk_log_start":"329'9400","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'9472","last_scrub_stamp":"2024-09-18T07:41:22.252822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:41:22.252822+0000","objects_scrubbed":6806,"log_size":4597,"log_dups_size":0,"ondisk_log_size":4597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:24.249010+0000","scrub_duration":1819,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146210816,"num_objects":2231,"num_object_clones":0,"num_object_copies":6693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13997,"num_write_kb":519296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"323'32","reported_seq":334,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.155077+0000","last_change":"2024-09-18T07:40:17.950392+0000","last_active":"2024-09-18T07:42:21.155077+0000","last_peered":"2024-09-18T07:42:21.155077+0000","last_clean":"2024-09-18T07:42:21.155077+0000","last_became_active":"2024-09-18T07:37:43.087496+0000","last_became_peered":"2024-09-18T07:37:43.087496+0000","last_unstale":"2024-09-18T07:42:21.155077+0000","last_undegraded":"2024-09-18T07:42:21.155077+0000","last_fullsized":"2024-09-18T07:42:21.155077+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-18T07:40:17.950298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:17.950298+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:39.343760+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":479,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.119574+0000","last_change":"2024-09-18T07:40:19.067973+0000","last_active":"2024-09-18T07:42:21.119574+0000","last_peered":"2024-09-18T07:42:21.119574+0000","last_clean":"2024-09-18T07:42:21.119574+0000","last_became_active":"2024-09-18T07:40:19.067624+0000","last_became_peered":"2024-09-18T07:40:19.067624+0000","last_unstale":"2024-09-18T07:42:21.119574+0000","last_undegraded":"2024-09-18T07:42:21.119574+0000","last_fullsized":"2024-09-18T07:42:21.119574+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:59.616988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:59.616988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:34.408847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":441,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121075+0000","last_change":"2024-09-18T07:40:42.452815+0000","last_active":"2024-09-18T07:42:21.121075+0000","last_peered":"2024-09-18T07:42:21.121075+0000","last_clean":"2024-09-18T07:42:21.121075+0000","last_became_active":"2024-09-18T07:39:03.350419+0000","last_became_peered":"2024-09-18T07:39:03.350419+0000","last_unstale":"2024-09-18T07:42:21.121075+0000","last_undegraded":"2024-09-18T07:42:21.121075+0000","last_fullsized":"2024-09-18T07:42:21.121075+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:42.452744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:42.452744+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:16.814737+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":391,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121796+0000","last_change":"2024-09-18T07:42:06.726510+0000","last_active":"2024-09-18T07:42:21.121796+0000","last_peered":"2024-09-18T07:42:21.121796+0000","last_clean":"2024-09-18T07:42:21.121796+0000","last_became_active":"2024-09-18T07:35:01.412217+0000","last_became_peered":"2024-09-18T07:35:01.412217+0000","last_unstale":"2024-09-18T07:42:21.121796+0000","last_undegraded":"2024-09-18T07:42:21.121796+0000","last_fullsized":"2024-09-18T07:42:21.121796+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.726445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.726445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:15.201416+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":589,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.118825+0000","last_change":"2024-09-18T07:33:22.358792+0000","last_active":"2024-09-18T07:42:21.118825+0000","last_peered":"2024-09-18T07:42:21.118825+0000","last_clean":"2024-09-18T07:42:21.118825+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:42:21.118825+0000","last_undegraded":"2024-09-18T07:42:21.118825+0000","last_fullsized":"2024-09-18T07:42:21.118825+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.358746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.358746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:35.384551+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"350'14136","reported_seq":14355,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.628107+0000","last_change":"2024-09-18T07:40:58.098898+0000","last_active":"2024-09-18T07:42:21.628107+0000","last_peered":"2024-09-18T07:42:21.628107+0000","last_clean":"2024-09-18T07:42:21.628107+0000","last_became_active":"2024-09-18T07:39:29.703451+0000","last_became_peered":"2024-09-18T07:39:29.703451+0000","last_unstale":"2024-09-18T07:42:21.628107+0000","last_undegraded":"2024-09-18T07:42:21.628107+0000","last_fullsized":"2024-09-18T07:42:21.628107+0000","mapping_epoch":289,"log_start":"330'9700","ondisk_log_start":"330'9700","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7779","last_scrub_stamp":"2024-09-18T07:40:58.098697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:58.098697+0000","objects_scrubbed":7710,"log_size":4436,"log_dups_size":0,"ondisk_log_size":4436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:21.707853+0000","scrub_duration":2544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155058176,"num_objects":2366,"num_object_clones":0,"num_object_copies":7098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14136,"num_write_kb":528064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"323'19","reported_seq":722,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.128687+0000","last_change":"2024-09-18T07:39:07.038527+0000","last_active":"2024-09-18T07:42:21.128687+0000","last_peered":"2024-09-18T07:42:21.128687+0000","last_clean":"2024-09-18T07:42:21.128687+0000","last_became_active":"2024-09-18T07:37:43.063255+0000","last_became_peered":"2024-09-18T07:37:43.063255+0000","last_unstale":"2024-09-18T07:42:21.128687+0000","last_undegraded":"2024-09-18T07:42:21.128687+0000","last_fullsized":"2024-09-18T07:42:21.128687+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:39:07.038450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:07.038450+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:34.882660+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":633,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.127410+0000","last_change":"2024-09-18T07:39:29.078656+0000","last_active":"2024-09-18T07:42:21.127410+0000","last_peered":"2024-09-18T07:42:21.127410+0000","last_clean":"2024-09-18T07:42:21.127410+0000","last_became_active":"2024-09-18T07:35:01.401063+0000","last_became_peered":"2024-09-18T07:35:01.401063+0000","last_unstale":"2024-09-18T07:42:21.127410+0000","last_undegraded":"2024-09-18T07:42:21.127410+0000","last_fullsized":"2024-09-18T07:42:21.127410+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.078600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.078600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:40.967780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":623,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.118655+0000","last_change":"2024-09-18T07:39:35.406107+0000","last_active":"2024-09-18T07:42:21.118655+0000","last_peered":"2024-09-18T07:42:21.118655+0000","last_clean":"2024-09-18T07:42:21.118655+0000","last_became_active":"2024-09-18T07:37:43.075535+0000","last_became_peered":"2024-09-18T07:37:43.075535+0000","last_unstale":"2024-09-18T07:42:21.118655+0000","last_undegraded":"2024-09-18T07:42:21.118655+0000","last_fullsized":"2024-09-18T07:42:21.118655+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.406041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.406041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:15.529066+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":482,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.154653+0000","last_change":"2024-09-18T07:41:20.220326+0000","last_active":"2024-09-18T07:42:21.154653+0000","last_peered":"2024-09-18T07:42:21.154653+0000","last_clean":"2024-09-18T07:42:21.154653+0000","last_became_active":"2024-09-18T07:40:19.068998+0000","last_became_peered":"2024-09-18T07:40:19.068998+0000","last_unstale":"2024-09-18T07:42:21.154653+0000","last_undegraded":"2024-09-18T07:42:21.154653+0000","last_fullsized":"2024-09-18T07:42:21.154653+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:20.220262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:20.220262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"350'14220","reported_seq":14433,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.731516+0000","last_change":"2024-09-18T07:40:48.247625+0000","last_active":"2024-09-18T07:42:21.731516+0000","last_peered":"2024-09-18T07:42:21.731516+0000","last_clean":"2024-09-18T07:42:21.731516+0000","last_became_active":"2024-09-18T07:39:29.699800+0000","last_became_peered":"2024-09-18T07:39:29.699800+0000","last_unstale":"2024-09-18T07:42:21.731516+0000","last_undegraded":"2024-09-18T07:42:21.731516+0000","last_fullsized":"2024-09-18T07:42:21.731516+0000","mapping_epoch":289,"log_start":"323'8200","ondisk_log_start":"323'8200","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'7166","last_scrub_stamp":"2024-09-18T07:40:48.247448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:48.247448+0000","objects_scrubbed":7089,"log_size":6020,"log_dups_size":0,"ondisk_log_size":6020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:16.070539+0000","scrub_duration":2089,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155058176,"num_objects":2366,"num_object_clones":0,"num_object_copies":7098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14220,"num_write_kb":530752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":181,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.116784+0000","last_change":"2024-09-18T07:39:38.447787+0000","last_active":"2024-09-18T07:42:21.116784+0000","last_peered":"2024-09-18T07:42:21.116784+0000","last_clean":"2024-09-18T07:42:21.116784+0000","last_became_active":"2024-09-18T07:39:04.950292+0000","last_became_peered":"2024-09-18T07:39:04.950292+0000","last_unstale":"2024-09-18T07:42:21.116784+0000","last_undegraded":"2024-09-18T07:42:21.116784+0000","last_fullsized":"2024-09-18T07:42:21.116784+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:38.447735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:38.447735+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:46.260853+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"323'31","reported_seq":672,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.128613+0000","last_change":"2024-09-18T07:40:04.281816+0000","last_active":"2024-09-18T07:42:21.128613+0000","last_peered":"2024-09-18T07:42:21.128613+0000","last_clean":"2024-09-18T07:42:21.128613+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:42:21.128613+0000","last_undegraded":"2024-09-18T07:42:21.128613+0000","last_fullsized":"2024-09-18T07:42:21.128613+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:40:04.281758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:04.281758+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:46.955052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":258,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.154164+0000","last_change":"2024-09-18T07:41:54.294370+0000","last_active":"2024-09-18T07:42:21.154164+0000","last_peered":"2024-09-18T07:42:21.154164+0000","last_clean":"2024-09-18T07:42:21.154164+0000","last_became_active":"2024-09-18T07:37:43.061808+0000","last_became_peered":"2024-09-18T07:37:43.061808+0000","last_unstale":"2024-09-18T07:42:21.154164+0000","last_undegraded":"2024-09-18T07:42:21.154164+0000","last_fullsized":"2024-09-18T07:42:21.154164+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:54.294307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.294307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:57.341592+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":301,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.128052+0000","last_change":"2024-09-18T07:40:52.167232+0000","last_active":"2024-09-18T07:42:21.128052+0000","last_peered":"2024-09-18T07:42:21.128052+0000","last_clean":"2024-09-18T07:42:21.128052+0000","last_became_active":"2024-09-18T07:36:13.116527+0000","last_became_peered":"2024-09-18T07:36:13.116527+0000","last_unstale":"2024-09-18T07:42:21.128052+0000","last_undegraded":"2024-09-18T07:42:21.128052+0000","last_fullsized":"2024-09-18T07:42:21.128052+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:52.167048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:52.167048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:17.172256+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":623,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.140101+0000","last_change":"2024-09-18T07:41:56.172872+0000","last_active":"2024-09-18T07:42:21.140101+0000","last_peered":"2024-09-18T07:42:21.140101+0000","last_clean":"2024-09-18T07:42:21.140101+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:42:21.140101+0000","last_undegraded":"2024-09-18T07:42:21.140101+0000","last_fullsized":"2024-09-18T07:42:21.140101+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:56.172797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:56.172797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:05.871802+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"350'14160","reported_seq":14371,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.696921+0000","last_change":"2024-09-18T07:40:41.001448+0000","last_active":"2024-09-18T07:42:21.696921+0000","last_peered":"2024-09-18T07:42:21.696921+0000","last_clean":"2024-09-18T07:42:21.696921+0000","last_became_active":"2024-09-18T07:40:19.066715+0000","last_became_peered":"2024-09-18T07:40:19.066715+0000","last_unstale":"2024-09-18T07:42:21.696921+0000","last_undegraded":"2024-09-18T07:42:21.696921+0000","last_fullsized":"2024-09-18T07:42:21.696921+0000","mapping_epoch":307,"log_start":"323'8100","ondisk_log_start":"323'8100","created":289,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'6605","last_scrub_stamp":"2024-09-18T07:40:41.001367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:41.001367+0000","objects_scrubbed":6534,"log_size":6060,"log_dups_size":0,"ondisk_log_size":6060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:46.302322+0000","scrub_duration":1931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155189248,"num_objects":2368,"num_object_clones":0,"num_object_copies":7104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14160,"num_write_kb":528896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":439,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121103+0000","last_change":"2024-09-18T07:39:05.498235+0000","last_active":"2024-09-18T07:42:21.121103+0000","last_peered":"2024-09-18T07:42:21.121103+0000","last_clean":"2024-09-18T07:42:21.121103+0000","last_became_active":"2024-09-18T07:39:04.944753+0000","last_became_peered":"2024-09-18T07:39:04.944753+0000","last_unstale":"2024-09-18T07:42:21.121103+0000","last_undegraded":"2024-09-18T07:42:21.121103+0000","last_fullsized":"2024-09-18T07:42:21.121103+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.498159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.498159+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:30.837196+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"323'20","reported_seq":675,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.128161+0000","last_change":"2024-09-18T07:40:50.210059+0000","last_active":"2024-09-18T07:42:21.128161+0000","last_peered":"2024-09-18T07:42:21.128161+0000","last_clean":"2024-09-18T07:42:21.128161+0000","last_became_active":"2024-09-18T07:37:43.062724+0000","last_became_peered":"2024-09-18T07:37:43.062724+0000","last_unstale":"2024-09-18T07:42:21.128161+0000","last_undegraded":"2024-09-18T07:42:21.128161+0000","last_fullsized":"2024-09-18T07:42:21.128161+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:40:50.209824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.209824+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":106,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.125181+0000","last_change":"2024-09-18T07:40:47.799512+0000","last_active":"2024-09-18T07:42:21.125181+0000","last_peered":"2024-09-18T07:42:21.125181+0000","last_clean":"2024-09-18T07:42:21.125181+0000","last_became_active":"2024-09-18T07:40:19.065652+0000","last_became_peered":"2024-09-18T07:40:19.065652+0000","last_unstale":"2024-09-18T07:42:21.125181+0000","last_undegraded":"2024-09-18T07:42:21.125181+0000","last_fullsized":"2024-09-18T07:42:21.125181+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:47.799342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.799342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"173'11","reported_seq":782,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.140380+0000","last_change":"2024-09-18T07:38:55.227421+0000","last_active":"2024-09-18T07:42:21.140380+0000","last_peered":"2024-09-18T07:42:21.140380+0000","last_clean":"2024-09-18T07:42:21.140380+0000","last_became_active":"2024-09-18T07:35:01.403765+0000","last_became_peered":"2024-09-18T07:35:01.403765+0000","last_unstale":"2024-09-18T07:42:21.140380+0000","last_undegraded":"2024-09-18T07:42:21.140380+0000","last_fullsized":"2024-09-18T07:42:21.140380+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'11","last_scrub_stamp":"2024-09-18T07:38:55.227366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:55.227366+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:33.902149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":599,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.128390+0000","last_change":"2024-09-18T07:36:33.251164+0000","last_active":"2024-09-18T07:42:21.128390+0000","last_peered":"2024-09-18T07:42:21.128390+0000","last_clean":"2024-09-18T07:42:21.128390+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:42:21.128390+0000","last_undegraded":"2024-09-18T07:42:21.128390+0000","last_fullsized":"2024-09-18T07:42:21.128390+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:33.251093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:33.251093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:12.044091+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"350'14205","reported_seq":14424,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:22.070184+0000","last_change":"2024-09-18T07:40:56.968567+0000","last_active":"2024-09-18T07:42:22.070184+0000","last_peered":"2024-09-18T07:42:22.070184+0000","last_clean":"2024-09-18T07:42:22.070184+0000","last_became_active":"2024-09-18T07:39:29.704318+0000","last_became_peered":"2024-09-18T07:39:29.704318+0000","last_unstale":"2024-09-18T07:42:22.070184+0000","last_undegraded":"2024-09-18T07:42:22.070184+0000","last_fullsized":"2024-09-18T07:42:22.070184+0000","mapping_epoch":289,"log_start":"329'9500","ondisk_log_start":"329'9500","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7695","last_scrub_stamp":"2024-09-18T07:40:56.968381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.968381+0000","objects_scrubbed":7617,"log_size":4705,"log_dups_size":0,"ondisk_log_size":4705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2651,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151322624,"num_objects":2309,"num_object_clones":0,"num_object_copies":6927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14205,"num_write_kb":528448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"323'15","reported_seq":653,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.124887+0000","last_change":"2024-09-18T07:41:21.946159+0000","last_active":"2024-09-18T07:42:21.124887+0000","last_peered":"2024-09-18T07:42:21.124887+0000","last_clean":"2024-09-18T07:42:21.124887+0000","last_became_active":"2024-09-18T07:37:27.994144+0000","last_became_peered":"2024-09-18T07:37:27.994144+0000","last_unstale":"2024-09-18T07:42:21.124887+0000","last_undegraded":"2024-09-18T07:42:21.124887+0000","last_fullsized":"2024-09-18T07:42:21.124887+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:21.946066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:21.946066+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:30.620162+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":631,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.117442+0000","last_change":"2024-09-18T07:40:45.343597+0000","last_active":"2024-09-18T07:42:21.117442+0000","last_peered":"2024-09-18T07:42:21.117442+0000","last_clean":"2024-09-18T07:42:21.117442+0000","last_became_active":"2024-09-18T07:37:43.075988+0000","last_became_peered":"2024-09-18T07:37:43.075988+0000","last_unstale":"2024-09-18T07:42:21.117442+0000","last_undegraded":"2024-09-18T07:42:21.117442+0000","last_fullsized":"2024-09-18T07:42:21.117442+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:45.343533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:45.343533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":603,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.137908+0000","last_change":"2024-09-18T07:39:01.216439+0000","last_active":"2024-09-18T07:42:21.137908+0000","last_peered":"2024-09-18T07:42:21.137908+0000","last_clean":"2024-09-18T07:42:21.137908+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:42:21.137908+0000","last_undegraded":"2024-09-18T07:42:21.137908+0000","last_fullsized":"2024-09-18T07:42:21.137908+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:01.216305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:01.216305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:34.950724+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":625,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121574+0000","last_change":"2024-09-18T07:41:59.006637+0000","last_active":"2024-09-18T07:42:21.121574+0000","last_peered":"2024-09-18T07:42:21.121574+0000","last_clean":"2024-09-18T07:42:21.121574+0000","last_became_active":"2024-09-18T07:35:01.410890+0000","last_became_peered":"2024-09-18T07:35:01.410890+0000","last_unstale":"2024-09-18T07:42:21.121574+0000","last_undegraded":"2024-09-18T07:42:21.121574+0000","last_fullsized":"2024-09-18T07:42:21.121574+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:59.006386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:59.006386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:18.266924+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"350'14333","reported_seq":14545,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:22.024931+0000","last_change":"2024-09-18T07:41:23.184909+0000","last_active":"2024-09-18T07:42:22.024931+0000","last_peered":"2024-09-18T07:42:22.024931+0000","last_clean":"2024-09-18T07:42:22.024931+0000","last_became_active":"2024-09-18T07:39:29.701593+0000","last_became_peered":"2024-09-18T07:39:29.701593+0000","last_unstale":"2024-09-18T07:42:22.024931+0000","last_undegraded":"2024-09-18T07:42:22.024931+0000","last_fullsized":"2024-09-18T07:42:22.024931+0000","mapping_epoch":289,"log_start":"329'9600","ondisk_log_start":"329'9600","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'9778","last_scrub_stamp":"2024-09-18T07:41:23.184760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:41:23.184760+0000","objects_scrubbed":7006,"log_size":4733,"log_dups_size":0,"ondisk_log_size":4733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:31.684623+0000","scrub_duration":2841,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154337280,"num_objects":2355,"num_object_clones":0,"num_object_copies":7065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14333,"num_write_kb":534016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"323'35","reported_seq":286,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.154298+0000","last_change":"2024-09-18T07:39:25.932376+0000","last_active":"2024-09-18T07:42:21.154298+0000","last_peered":"2024-09-18T07:42:21.154298+0000","last_clean":"2024-09-18T07:42:21.154298+0000","last_became_active":"2024-09-18T07:37:43.063319+0000","last_became_peered":"2024-09-18T07:37:43.063319+0000","last_unstale":"2024-09-18T07:42:21.154298+0000","last_undegraded":"2024-09-18T07:42:21.154298+0000","last_fullsized":"2024-09-18T07:42:21.154298+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:39:25.932315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.932315+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:36.321873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":405,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.122350+0000","last_change":"2024-09-18T07:40:56.546321+0000","last_active":"2024-09-18T07:42:21.122350+0000","last_peered":"2024-09-18T07:42:21.122350+0000","last_clean":"2024-09-18T07:42:21.122350+0000","last_became_active":"2024-09-18T07:37:43.072894+0000","last_became_peered":"2024-09-18T07:37:43.072894+0000","last_unstale":"2024-09-18T07:42:21.122350+0000","last_undegraded":"2024-09-18T07:42:21.122350+0000","last_fullsized":"2024-09-18T07:42:21.122350+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:56.546251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.546251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":626,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.137910+0000","last_change":"2024-09-18T07:42:04.179746+0000","last_active":"2024-09-18T07:42:21.137910+0000","last_peered":"2024-09-18T07:42:21.137910+0000","last_clean":"2024-09-18T07:42:21.137910+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:42:21.137910+0000","last_undegraded":"2024-09-18T07:42:21.137910+0000","last_fullsized":"2024-09-18T07:42:21.137910+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:04.179670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.179670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:04.388489+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":383,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121144+0000","last_change":"2024-09-18T07:42:09.794539+0000","last_active":"2024-09-18T07:42:21.121144+0000","last_peered":"2024-09-18T07:42:21.121144+0000","last_clean":"2024-09-18T07:42:21.121144+0000","last_became_active":"2024-09-18T07:35:01.406980+0000","last_became_peered":"2024-09-18T07:35:01.406980+0000","last_unstale":"2024-09-18T07:42:21.121144+0000","last_undegraded":"2024-09-18T07:42:21.121144+0000","last_fullsized":"2024-09-18T07:42:21.121144+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:09.794459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:09.794459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:32.016537+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"350'14273","reported_seq":14482,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.645558+0000","last_change":"2024-09-18T07:40:43.229079+0000","last_active":"2024-09-18T07:42:21.645558+0000","last_peered":"2024-09-18T07:42:21.645558+0000","last_clean":"2024-09-18T07:42:21.645558+0000","last_became_active":"2024-09-18T07:39:29.700244+0000","last_became_peered":"2024-09-18T07:39:29.700244+0000","last_unstale":"2024-09-18T07:42:21.645558+0000","last_undegraded":"2024-09-18T07:42:21.645558+0000","last_fullsized":"2024-09-18T07:42:21.645558+0000","mapping_epoch":289,"log_start":"329'9500","ondisk_log_start":"329'9500","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'6831","last_scrub_stamp":"2024-09-18T07:40:43.228883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:43.228883+0000","objects_scrubbed":6759,"log_size":4773,"log_dups_size":0,"ondisk_log_size":4773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2043,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150536192,"num_objects":2297,"num_object_clones":0,"num_object_copies":6891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14273,"num_write_kb":530240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"323'18","reported_seq":663,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.127299+0000","last_change":"2024-09-18T07:41:47.950163+0000","last_active":"2024-09-18T07:42:21.127299+0000","last_peered":"2024-09-18T07:42:21.127299+0000","last_clean":"2024-09-18T07:42:21.127299+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:42:21.127299+0000","last_undegraded":"2024-09-18T07:42:21.127299+0000","last_fullsized":"2024-09-18T07:42:21.127299+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:41:47.950068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:47.950068+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:01.623082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":383,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121917+0000","last_change":"2024-09-18T07:42:07.729518+0000","last_active":"2024-09-18T07:42:21.121917+0000","last_peered":"2024-09-18T07:42:21.121917+0000","last_clean":"2024-09-18T07:42:21.121917+0000","last_became_active":"2024-09-18T07:35:01.398499+0000","last_became_peered":"2024-09-18T07:35:01.398499+0000","last_unstale":"2024-09-18T07:42:21.121917+0000","last_undegraded":"2024-09-18T07:42:21.121917+0000","last_fullsized":"2024-09-18T07:42:21.121917+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:07.729438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:07.729438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:33.911294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":635,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.137453+0000","last_change":"2024-09-18T07:41:58.202324+0000","last_active":"2024-09-18T07:42:21.137453+0000","last_peered":"2024-09-18T07:42:21.137453+0000","last_clean":"2024-09-18T07:42:21.137453+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:42:21.137453+0000","last_undegraded":"2024-09-18T07:42:21.137453+0000","last_fullsized":"2024-09-18T07:42:21.137453+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:58.202229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:58.202229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:08.344962+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":612,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.118558+0000","last_change":"2024-09-18T07:40:02.932758+0000","last_active":"2024-09-18T07:42:21.118558+0000","last_peered":"2024-09-18T07:42:21.118558+0000","last_clean":"2024-09-18T07:42:21.118558+0000","last_became_active":"2024-09-18T07:35:01.418735+0000","last_became_peered":"2024-09-18T07:35:01.418735+0000","last_unstale":"2024-09-18T07:42:21.118558+0000","last_undegraded":"2024-09-18T07:42:21.118558+0000","last_fullsized":"2024-09-18T07:42:21.118558+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:02.932690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:02.932690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:42.406919+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"350'14170","reported_seq":14387,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:22.587247+0000","last_change":"2024-09-18T07:40:54.295785+0000","last_active":"2024-09-18T07:42:22.587247+0000","last_peered":"2024-09-18T07:42:22.587247+0000","last_clean":"2024-09-18T07:42:22.587247+0000","last_became_active":"2024-09-18T07:39:29.701617+0000","last_became_peered":"2024-09-18T07:39:29.701617+0000","last_unstale":"2024-09-18T07:42:22.587247+0000","last_undegraded":"2024-09-18T07:42:22.587247+0000","last_fullsized":"2024-09-18T07:42:22.587247+0000","mapping_epoch":289,"log_start":"325'8700","ondisk_log_start":"325'8700","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7478","last_scrub_stamp":"2024-09-18T07:40:54.295602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:54.295602+0000","objects_scrubbed":7411,"log_size":5470,"log_dups_size":0,"ondisk_log_size":5470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148504576,"num_objects":2266,"num_object_clones":0,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14170,"num_write_kb":525952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"323'27","reported_seq":756,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.116939+0000","last_change":"2024-09-18T07:41:51.332873+0000","last_active":"2024-09-18T07:42:21.116939+0000","last_peered":"2024-09-18T07:42:21.116939+0000","last_clean":"2024-09-18T07:42:21.116939+0000","last_became_active":"2024-09-18T07:37:43.075552+0000","last_became_peered":"2024-09-18T07:37:43.075552+0000","last_unstale":"2024-09-18T07:42:21.116939+0000","last_undegraded":"2024-09-18T07:42:21.116939+0000","last_fullsized":"2024-09-18T07:42:21.116939+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:41:51.332791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:51.332791+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:13.912522+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":397,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121724+0000","last_change":"2024-09-18T07:41:27.819345+0000","last_active":"2024-09-18T07:42:21.121724+0000","last_peered":"2024-09-18T07:42:21.121724+0000","last_clean":"2024-09-18T07:42:21.121724+0000","last_became_active":"2024-09-18T07:35:01.406695+0000","last_became_peered":"2024-09-18T07:35:01.406695+0000","last_unstale":"2024-09-18T07:42:21.121724+0000","last_undegraded":"2024-09-18T07:42:21.121724+0000","last_fullsized":"2024-09-18T07:42:21.121724+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:41:27.819269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.819269+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:32.041258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"240'12","reported_seq":498,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121573+0000","last_change":"2024-09-18T07:38:55.490250+0000","last_active":"2024-09-18T07:42:21.121573+0000","last_peered":"2024-09-18T07:42:21.121573+0000","last_clean":"2024-09-18T07:42:21.121573+0000","last_became_active":"2024-09-18T07:37:43.056714+0000","last_became_peered":"2024-09-18T07:37:43.056714+0000","last_unstale":"2024-09-18T07:42:21.121573+0000","last_undegraded":"2024-09-18T07:42:21.121573+0000","last_fullsized":"2024-09-18T07:42:21.121573+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:38:55.490178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:55.490178+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:33.717958+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":611,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.119223+0000","last_change":"2024-09-18T07:42:07.547399+0000","last_active":"2024-09-18T07:42:21.119223+0000","last_peered":"2024-09-18T07:42:21.119223+0000","last_clean":"2024-09-18T07:42:21.119223+0000","last_became_active":"2024-09-18T07:37:43.078121+0000","last_became_peered":"2024-09-18T07:37:43.078121+0000","last_unstale":"2024-09-18T07:42:21.119223+0000","last_undegraded":"2024-09-18T07:42:21.119223+0000","last_fullsized":"2024-09-18T07:42:21.119223+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:07.547352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:07.547352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:10.534419+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":249,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.153786+0000","last_change":"2024-09-18T07:42:08.181612+0000","last_active":"2024-09-18T07:42:21.153786+0000","last_peered":"2024-09-18T07:42:21.153786+0000","last_clean":"2024-09-18T07:42:21.153786+0000","last_became_active":"2024-09-18T07:37:43.087994+0000","last_became_peered":"2024-09-18T07:37:43.087994+0000","last_unstale":"2024-09-18T07:42:21.153786+0000","last_undegraded":"2024-09-18T07:42:21.153786+0000","last_fullsized":"2024-09-18T07:42:21.153786+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:08.181420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:08.181420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:21.009353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"323'12","reported_seq":661,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.120884+0000","last_change":"2024-09-18T07:41:48.359809+0000","last_active":"2024-09-18T07:42:21.120884+0000","last_peered":"2024-09-18T07:42:21.120884+0000","last_clean":"2024-09-18T07:42:21.120884+0000","last_became_active":"2024-09-18T07:35:01.407787+0000","last_became_peered":"2024-09-18T07:35:01.407787+0000","last_unstale":"2024-09-18T07:42:21.120884+0000","last_undegraded":"2024-09-18T07:42:21.120884+0000","last_fullsized":"2024-09-18T07:42:21.120884+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'12","last_scrub_stamp":"2024-09-18T07:41:48.359711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.359711+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:03.524359+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":616,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.118700+0000","last_change":"2024-09-18T07:41:52.319134+0000","last_active":"2024-09-18T07:42:21.118700+0000","last_peered":"2024-09-18T07:42:21.118700+0000","last_clean":"2024-09-18T07:42:21.118700+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:42:21.118700+0000","last_undegraded":"2024-09-18T07:42:21.118700+0000","last_fullsized":"2024-09-18T07:42:21.118700+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:52.319073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:52.319073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:54.508152+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":389,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.122214+0000","last_change":"2024-09-18T07:40:51.554184+0000","last_active":"2024-09-18T07:42:21.122214+0000","last_peered":"2024-09-18T07:42:21.122214+0000","last_clean":"2024-09-18T07:42:21.122214+0000","last_became_active":"2024-09-18T07:37:43.079313+0000","last_became_peered":"2024-09-18T07:37:43.079313+0000","last_unstale":"2024-09-18T07:42:21.122214+0000","last_undegraded":"2024-09-18T07:42:21.122214+0000","last_fullsized":"2024-09-18T07:42:21.122214+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:51.554073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:51.554073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"323'15","reported_seq":645,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.122128+0000","last_change":"2024-09-18T07:35:01.448510+0000","last_active":"2024-09-18T07:42:21.122128+0000","last_peered":"2024-09-18T07:42:21.122128+0000","last_clean":"2024-09-18T07:42:21.122128+0000","last_became_active":"2024-09-18T07:35:01.414458+0000","last_became_peered":"2024-09-18T07:35:01.414458+0000","last_unstale":"2024-09-18T07:42:21.122128+0000","last_undegraded":"2024-09-18T07:42:21.122128+0000","last_fullsized":"2024-09-18T07:42:21.122128+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:30.006114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":405,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121421+0000","last_change":"2024-09-18T07:41:52.891853+0000","last_active":"2024-09-18T07:42:21.121421+0000","last_peered":"2024-09-18T07:42:21.121421+0000","last_clean":"2024-09-18T07:42:21.121421+0000","last_became_active":"2024-09-18T07:37:43.056675+0000","last_became_peered":"2024-09-18T07:37:43.056675+0000","last_unstale":"2024-09-18T07:42:21.121421+0000","last_undegraded":"2024-09-18T07:42:21.121421+0000","last_fullsized":"2024-09-18T07:42:21.121421+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:52.891811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:52.891811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:55.106484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":631,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.120634+0000","last_change":"2024-09-18T07:41:16.472282+0000","last_active":"2024-09-18T07:42:21.120634+0000","last_peered":"2024-09-18T07:42:21.120634+0000","last_clean":"2024-09-18T07:42:21.120634+0000","last_became_active":"2024-09-18T07:35:01.420133+0000","last_became_peered":"2024-09-18T07:35:01.420133+0000","last_unstale":"2024-09-18T07:42:21.120634+0000","last_undegraded":"2024-09-18T07:42:21.120634+0000","last_fullsized":"2024-09-18T07:42:21.120634+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:16.472210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:16.472210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:38.782593+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":604,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.138078+0000","last_change":"2024-09-18T07:37:48.861965+0000","last_active":"2024-09-18T07:42:21.138078+0000","last_peered":"2024-09-18T07:42:21.138078+0000","last_clean":"2024-09-18T07:42:21.138078+0000","last_became_active":"2024-09-18T07:35:01.396305+0000","last_became_peered":"2024-09-18T07:35:01.396305+0000","last_unstale":"2024-09-18T07:42:21.138078+0000","last_undegraded":"2024-09-18T07:42:21.138078+0000","last_fullsized":"2024-09-18T07:42:21.138078+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:48.861784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:37:48.861784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:31.915133+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"323'15","reported_seq":671,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.119988+0000","last_change":"2024-09-18T07:40:52.366957+0000","last_active":"2024-09-18T07:42:21.119988+0000","last_peered":"2024-09-18T07:42:21.119988+0000","last_clean":"2024-09-18T07:42:21.119988+0000","last_became_active":"2024-09-18T07:35:01.416744+0000","last_became_peered":"2024-09-18T07:35:01.416744+0000","last_unstale":"2024-09-18T07:42:21.119988+0000","last_undegraded":"2024-09-18T07:42:21.119988+0000","last_fullsized":"2024-09-18T07:42:21.119988+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:52.366672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:52.366672+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":636,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.123478+0000","last_change":"2024-09-18T07:41:56.060035+0000","last_active":"2024-09-18T07:42:21.123478+0000","last_peered":"2024-09-18T07:42:21.123478+0000","last_clean":"2024-09-18T07:42:21.123478+0000","last_became_active":"2024-09-18T07:35:01.414024+0000","last_became_peered":"2024-09-18T07:35:01.414024+0000","last_unstale":"2024-09-18T07:42:21.123478+0000","last_undegraded":"2024-09-18T07:42:21.123478+0000","last_fullsized":"2024-09-18T07:42:21.123478+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:56.059922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:56.059922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:07.678767+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"173'10","reported_seq":769,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.120112+0000","last_change":"2024-09-18T07:39:32.947933+0000","last_active":"2024-09-18T07:42:21.120112+0000","last_peered":"2024-09-18T07:42:21.120112+0000","last_clean":"2024-09-18T07:42:21.120112+0000","last_became_active":"2024-09-18T07:35:01.407568+0000","last_became_peered":"2024-09-18T07:35:01.407568+0000","last_unstale":"2024-09-18T07:42:21.120112+0000","last_undegraded":"2024-09-18T07:42:21.120112+0000","last_fullsized":"2024-09-18T07:42:21.120112+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:39:32.947887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:32.947887+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:40.468020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":606,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.128963+0000","last_change":"2024-09-18T07:41:48.955522+0000","last_active":"2024-09-18T07:42:21.128963+0000","last_peered":"2024-09-18T07:42:21.128963+0000","last_clean":"2024-09-18T07:42:21.128963+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:42:21.128963+0000","last_undegraded":"2024-09-18T07:42:21.128963+0000","last_fullsized":"2024-09-18T07:42:21.128963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:48.955463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.955463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:00.936125+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"323'20","reported_seq":383,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121310+0000","last_change":"2024-09-18T07:40:19.070502+0000","last_active":"2024-09-18T07:42:21.121310+0000","last_peered":"2024-09-18T07:42:21.121310+0000","last_clean":"2024-09-18T07:42:21.121310+0000","last_became_active":"2024-09-18T07:40:19.069759+0000","last_became_peered":"2024-09-18T07:40:19.069759+0000","last_unstale":"2024-09-18T07:42:21.121310+0000","last_undegraded":"2024-09-18T07:42:21.121310+0000","last_fullsized":"2024-09-18T07:42:21.121310+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:36:02.975863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:02.975863+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:28.833047+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":391,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121906+0000","last_change":"2024-09-18T07:42:04.758461+0000","last_active":"2024-09-18T07:42:21.121906+0000","last_peered":"2024-09-18T07:42:21.121906+0000","last_clean":"2024-09-18T07:42:21.121906+0000","last_became_active":"2024-09-18T07:35:01.400197+0000","last_became_peered":"2024-09-18T07:35:01.400197+0000","last_unstale":"2024-09-18T07:42:21.121906+0000","last_undegraded":"2024-09-18T07:42:21.121906+0000","last_fullsized":"2024-09-18T07:42:21.121906+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:04.758396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.758396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:27.831062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":749,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.119872+0000","last_change":"2024-09-18T07:40:40.352088+0000","last_active":"2024-09-18T07:42:21.119872+0000","last_peered":"2024-09-18T07:42:21.119872+0000","last_clean":"2024-09-18T07:42:21.119872+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:42:21.119872+0000","last_undegraded":"2024-09-18T07:42:21.119872+0000","last_fullsized":"2024-09-18T07:42:21.119872+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:40:40.352040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.352040+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":624,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121697+0000","last_change":"2024-09-18T07:42:01.071103+0000","last_active":"2024-09-18T07:42:21.121697+0000","last_peered":"2024-09-18T07:42:21.121697+0000","last_clean":"2024-09-18T07:42:21.121697+0000","last_became_active":"2024-09-18T07:35:01.410561+0000","last_became_peered":"2024-09-18T07:35:01.410561+0000","last_unstale":"2024-09-18T07:42:21.121697+0000","last_undegraded":"2024-09-18T07:42:21.121697+0000","last_fullsized":"2024-09-18T07:42:21.121697+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:01.071028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:01.071028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:12.908296+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"323'19","reported_seq":715,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.127702+0000","last_change":"2024-09-18T07:41:54.022517+0000","last_active":"2024-09-18T07:42:21.127702+0000","last_peered":"2024-09-18T07:42:21.127702+0000","last_clean":"2024-09-18T07:42:21.127702+0000","last_became_active":"2024-09-18T07:38:41.143108+0000","last_became_peered":"2024-09-18T07:38:41.143108+0000","last_unstale":"2024-09-18T07:42:21.127702+0000","last_undegraded":"2024-09-18T07:42:21.127702+0000","last_fullsized":"2024-09-18T07:42:21.127702+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:41:54.022421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.022421+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:19.855029+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":630,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.128314+0000","last_change":"2024-09-18T07:40:40.149372+0000","last_active":"2024-09-18T07:42:21.128314+0000","last_peered":"2024-09-18T07:42:21.128314+0000","last_clean":"2024-09-18T07:42:21.128314+0000","last_became_active":"2024-09-18T07:37:43.061577+0000","last_became_peered":"2024-09-18T07:37:43.061577+0000","last_unstale":"2024-09-18T07:42:21.128314+0000","last_undegraded":"2024-09-18T07:42:21.128314+0000","last_fullsized":"2024-09-18T07:42:21.128314+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:40.149313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.149313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:16.130628+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":301,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.119363+0000","last_change":"2024-09-18T07:41:24.566420+0000","last_active":"2024-09-18T07:42:21.119363+0000","last_peered":"2024-09-18T07:42:21.119363+0000","last_clean":"2024-09-18T07:42:21.119363+0000","last_became_active":"2024-09-18T07:36:13.117584+0000","last_became_peered":"2024-09-18T07:36:13.117584+0000","last_unstale":"2024-09-18T07:42:21.119363+0000","last_undegraded":"2024-09-18T07:42:21.119363+0000","last_fullsized":"2024-09-18T07:42:21.119363+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:24.566345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:24.566345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:52.792357+0000","scrub_duration":160,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":105,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121487+0000","last_change":"2024-09-18T07:42:06.204026+0000","last_active":"2024-09-18T07:42:21.121487+0000","last_peered":"2024-09-18T07:42:21.121487+0000","last_clean":"2024-09-18T07:42:21.121487+0000","last_became_active":"2024-09-18T07:40:19.065468+0000","last_became_peered":"2024-09-18T07:40:19.065468+0000","last_unstale":"2024-09-18T07:42:21.121487+0000","last_undegraded":"2024-09-18T07:42:21.121487+0000","last_fullsized":"2024-09-18T07:42:21.121487+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.203932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.203932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:27.863077+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"323'15","reported_seq":500,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121611+0000","last_change":"2024-09-18T07:41:20.975210+0000","last_active":"2024-09-18T07:42:21.121611+0000","last_peered":"2024-09-18T07:42:21.121611+0000","last_clean":"2024-09-18T07:42:21.121611+0000","last_became_active":"2024-09-18T07:37:43.067598+0000","last_became_peered":"2024-09-18T07:37:43.067598+0000","last_unstale":"2024-09-18T07:42:21.121611+0000","last_undegraded":"2024-09-18T07:42:21.121611+0000","last_fullsized":"2024-09-18T07:42:21.121611+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:20.975135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:20.975135+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:26.367636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":391,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.122135+0000","last_change":"2024-09-18T07:41:47.881147+0000","last_active":"2024-09-18T07:42:21.122135+0000","last_peered":"2024-09-18T07:42:21.122135+0000","last_clean":"2024-09-18T07:42:21.122135+0000","last_became_active":"2024-09-18T07:35:01.405287+0000","last_became_peered":"2024-09-18T07:35:01.405287+0000","last_unstale":"2024-09-18T07:42:21.122135+0000","last_undegraded":"2024-09-18T07:42:21.122135+0000","last_fullsized":"2024-09-18T07:42:21.122135+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:47.880911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:47.880911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:00.989233+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":97,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.124198+0000","last_change":"2024-09-18T07:40:19.068620+0000","last_active":"2024-09-18T07:42:21.124198+0000","last_peered":"2024-09-18T07:42:21.124198+0000","last_clean":"2024-09-18T07:42:21.124198+0000","last_became_active":"2024-09-18T07:40:19.068237+0000","last_became_peered":"2024-09-18T07:40:19.068237+0000","last_unstale":"2024-09-18T07:42:21.124198+0000","last_undegraded":"2024-09-18T07:42:21.124198+0000","last_fullsized":"2024-09-18T07:42:21.124198+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:56.920384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:56.920384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:34.021901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":618,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.123594+0000","last_change":"2024-09-18T07:41:18.963607+0000","last_active":"2024-09-18T07:42:21.123594+0000","last_peered":"2024-09-18T07:42:21.123594+0000","last_clean":"2024-09-18T07:42:21.123594+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:42:21.123594+0000","last_undegraded":"2024-09-18T07:42:21.123594+0000","last_fullsized":"2024-09-18T07:42:21.123594+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:18.963490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:18.963490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:42.985064+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"323'19","reported_seq":720,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.119474+0000","last_change":"2024-09-18T07:42:10.458954+0000","last_active":"2024-09-18T07:42:21.119474+0000","last_peered":"2024-09-18T07:42:21.119474+0000","last_clean":"2024-09-18T07:42:21.119474+0000","last_became_active":"2024-09-18T07:37:43.066208+0000","last_became_peered":"2024-09-18T07:37:43.066208+0000","last_unstale":"2024-09-18T07:42:21.119474+0000","last_undegraded":"2024-09-18T07:42:21.119474+0000","last_fullsized":"2024-09-18T07:42:21.119474+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:42:10.458865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:10.458865+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:33.857115+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":622,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.118911+0000","last_change":"2024-09-18T07:41:27.341104+0000","last_active":"2024-09-18T07:42:21.118911+0000","last_peered":"2024-09-18T07:42:21.118911+0000","last_clean":"2024-09-18T07:42:21.118911+0000","last_became_active":"2024-09-18T07:35:01.415261+0000","last_became_peered":"2024-09-18T07:35:01.415261+0000","last_unstale":"2024-09-18T07:42:21.118911+0000","last_undegraded":"2024-09-18T07:42:21.118911+0000","last_fullsized":"2024-09-18T07:42:21.118911+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:27.341063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.341063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:38.597550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":405,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121220+0000","last_change":"2024-09-18T07:42:05.775215+0000","last_active":"2024-09-18T07:42:21.121220+0000","last_peered":"2024-09-18T07:42:21.121220+0000","last_clean":"2024-09-18T07:42:21.121220+0000","last_became_active":"2024-09-18T07:37:43.057483+0000","last_became_peered":"2024-09-18T07:37:43.057483+0000","last_unstale":"2024-09-18T07:42:21.121220+0000","last_undegraded":"2024-09-18T07:42:21.121220+0000","last_fullsized":"2024-09-18T07:42:21.121220+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:05.775170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:05.775170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:20.076140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":627,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.118652+0000","last_change":"2024-09-18T07:41:27.610895+0000","last_active":"2024-09-18T07:42:21.118652+0000","last_peered":"2024-09-18T07:42:21.118652+0000","last_clean":"2024-09-18T07:42:21.118652+0000","last_became_active":"2024-09-18T07:37:43.067066+0000","last_became_peered":"2024-09-18T07:37:43.067066+0000","last_unstale":"2024-09-18T07:42:21.118652+0000","last_undegraded":"2024-09-18T07:42:21.118652+0000","last_fullsized":"2024-09-18T07:42:21.118652+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:27.610707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.610707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:28.092460+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"323'8","reported_seq":664,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.117815+0000","last_change":"2024-09-18T07:37:43.134956+0000","last_active":"2024-09-18T07:42:21.117815+0000","last_peered":"2024-09-18T07:42:21.117815+0000","last_clean":"2024-09-18T07:42:21.117815+0000","last_became_active":"2024-09-18T07:37:43.067503+0000","last_became_peered":"2024-09-18T07:37:43.067503+0000","last_unstale":"2024-09-18T07:42:21.117815+0000","last_undegraded":"2024-09-18T07:42:21.117815+0000","last_fullsized":"2024-09-18T07:42:21.117815+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:36:40.908629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:40.908629+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:13.423257+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":606,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121985+0000","last_change":"2024-09-18T07:40:11.843693+0000","last_active":"2024-09-18T07:42:21.121985+0000","last_peered":"2024-09-18T07:42:21.121985+0000","last_clean":"2024-09-18T07:42:21.121985+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:42:21.121985+0000","last_undegraded":"2024-09-18T07:42:21.121985+0000","last_fullsized":"2024-09-18T07:42:21.121985+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:11.843625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:11.843625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:42.965878+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":241,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.153859+0000","last_change":"2024-09-18T07:40:40.106989+0000","last_active":"2024-09-18T07:42:21.153859+0000","last_peered":"2024-09-18T07:42:21.153859+0000","last_clean":"2024-09-18T07:42:21.153859+0000","last_became_active":"2024-09-18T07:37:43.061081+0000","last_became_peered":"2024-09-18T07:37:43.061081+0000","last_unstale":"2024-09-18T07:42:21.153859+0000","last_undegraded":"2024-09-18T07:42:21.153859+0000","last_fullsized":"2024-09-18T07:42:21.153859+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:40.106921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.106921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:11.100563+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":391,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.122078+0000","last_change":"2024-09-18T07:40:38.492338+0000","last_active":"2024-09-18T07:42:21.122078+0000","last_peered":"2024-09-18T07:42:21.122078+0000","last_clean":"2024-09-18T07:42:21.122078+0000","last_became_active":"2024-09-18T07:35:01.400396+0000","last_became_peered":"2024-09-18T07:35:01.400396+0000","last_unstale":"2024-09-18T07:42:21.122078+0000","last_undegraded":"2024-09-18T07:42:21.122078+0000","last_fullsized":"2024-09-18T07:42:21.122078+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:38.492269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.492269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:17.788359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":293,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.119853+0000","last_change":"2024-09-18T07:36:13.118009+0000","last_active":"2024-09-18T07:42:21.119853+0000","last_peered":"2024-09-18T07:42:21.119853+0000","last_clean":"2024-09-18T07:42:21.119853+0000","last_became_active":"2024-09-18T07:36:13.117154+0000","last_became_peered":"2024-09-18T07:36:13.117154+0000","last_unstale":"2024-09-18T07:42:21.119853+0000","last_undegraded":"2024-09-18T07:42:21.119853+0000","last_fullsized":"2024-09-18T07:42:21.119853+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.071629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.071629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:43.255444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":631,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.118108+0000","last_change":"2024-09-18T07:41:22.478552+0000","last_active":"2024-09-18T07:42:21.118108+0000","last_peered":"2024-09-18T07:42:21.118108+0000","last_clean":"2024-09-18T07:42:21.118108+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:42:21.118108+0000","last_undegraded":"2024-09-18T07:42:21.118108+0000","last_fullsized":"2024-09-18T07:42:21.118108+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:41:22.478474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:22.478474+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:37.260325+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"323'19","reported_seq":266,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.155519+0000","last_change":"2024-09-18T07:41:23.281610+0000","last_active":"2024-09-18T07:42:21.155519+0000","last_peered":"2024-09-18T07:42:21.155519+0000","last_clean":"2024-09-18T07:42:21.155519+0000","last_became_active":"2024-09-18T07:37:43.070186+0000","last_became_peered":"2024-09-18T07:37:43.070186+0000","last_unstale":"2024-09-18T07:42:21.155519+0000","last_undegraded":"2024-09-18T07:42:21.155519+0000","last_fullsized":"2024-09-18T07:42:21.155519+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:41:23.281547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:23.281547+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:41.164144+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":629,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-18T07:42:21.121852+0000","last_change":"2024-09-18T07:41:13.842121+0000","last_active":"2024-09-18T07:42:21.121852+0000","last_peered":"2024-09-18T07:42:21.121852+0000","last_clean":"2024-09-18T07:42:21.121852+0000","last_became_active":"2024-09-18T07:35:01.410469+0000","last_became_peered":"2024-09-18T07:35:01.410469+0000","last_unstale":"2024-09-18T07:42:21.121852+0000","last_undegraded":"2024-09-18T07:42:21.121852+0000","last_fullsized":"2024-09-18T07:42:21.121852+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:13.842068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:13.842068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2426077208,"num_objects":37020,"num_object_clones":0,"num_object_copies":111060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37020,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":227976,"num_write_kb":8479809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7297052672,"data_stored":7297040456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79376,"ondisk_log_size":79376,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1118,"num_read_kb":875,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":4664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":684,"ondisk_log_size":684,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":446,"num_read_kb":441,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201421,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1811504,"kb_used_data":1401120,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":91933648,"statfs":{"total":95995035648,"available":94140055552,"internally_reserved":0,"allocated":1434746880,"data_stored":1434355696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":420018634},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463059,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":793676,"kb_used_data":465404,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92951476,"statfs":{"total":95995035648,"available":95182311424,"internally_reserved":0,"allocated":476573696,"data_stored":476217937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":336132550},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724696,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1841780,"kb_used_data":1355232,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91903372,"statfs":{"total":95995035648,"available":94109052928,"internally_reserved":0,"allocated":1387757568,"data_stored":1387373461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953631,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":786684,"kb_used_data":466728,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92958468,"statfs":{"total":95995035648,"available":95189471232,"internally_reserved":0,"allocated":477929472,"data_stored":477536957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1749952,"kb_used_data":1217580,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":91995200,"statfs":{"total":95995035648,"available":94203084800,"internally_reserved":0,"allocated":1246801920,"data_stored":1246406705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":545126855},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411497,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":839916,"kb_used_data":447704,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92905236,"statfs":{"total":95995035648,"available":95134961664,"internally_reserved":0,"allocated":458448896,"data_stored":458072856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575024,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1889020,"kb_used_data":1374636,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":91856132,"statfs":{"total":95995035648,"available":94060679168,"internally_reserved":0,"allocated":1407627264,"data_stored":1407224592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738549,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":976064,"kb_used_data":588016,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":92769088,"statfs":{"total":95995035648,"available":94995546112,"internally_reserved":0,"allocated":602128384,"data_stored":601729079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":590675968,"data_stored":590675968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1363152896,"data_stored":1363148824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":456196096,"data_stored":456196096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1200029696,"data_stored":1200029696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":469237760,"data_stored":469237760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1357254656,"data_stored":1357250584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":465895424,"data_stored":465895424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1394610176,"data_stored":1394606104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:42:25.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:42:25.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:25.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:25.155 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:25.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:25.256 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:25.289 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:25.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:25.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:25.391 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:25.398 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:25 smithi137 ceph-mon[29408]: from='client.35900 v1:172.21.15.137:0/1835474972' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:42:25.398 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:25 smithi137 ceph-mon[29408]: pgmap v610: 158 pgs: 158 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 774 B/s rd, 0 B/s wr, 1.60k op/s 2024-09-18T07:42:25.398 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:25 smithi137 ceph-mon[39175]: from='client.35900 v1:172.21.15.137:0/1835474972' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:42:25.398 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:25 smithi137 ceph-mon[39175]: pgmap v610: 158 pgs: 158 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 774 B/s rd, 0 B/s wr, 1.60k op/s 2024-09-18T07:42:25.442 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:25.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:25.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:25.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:25 smithi203 ceph-mon[32237]: from='client.35900 v1:172.21.15.137:0/1835474972' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:42:25.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:25 smithi203 ceph-mon[32237]: pgmap v610: 158 pgs: 158 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 774 B/s rd, 0 B/s wr, 1.60k op/s 2024-09-18T07:42:25.544 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:25.578 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:25.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:25.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:25.680 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:25.689 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:25.706 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:25.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:25.752 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:42:25.752 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:42:25.808 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:25.840 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:25.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:25.942 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:25.974 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:25.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:25.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:26.076 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:26.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:26.119 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:26.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:26.220 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:26.257 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:26.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:26.358 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:26.380 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:42:26.393 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:26.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:26.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:26.495 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:26.524 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:26.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:26.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:26.626 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:26.655 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:26.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:26.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:26.757 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:26.779 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:26.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:26.817 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:26.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:26.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:26.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:26.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:26.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:27.060 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:27.097 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:27.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:27.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:27.177 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:42:27.199 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:27.228 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:27.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:27.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:27.330 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:27.362 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:27.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:27.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:27.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:27 smithi137 ceph-mon[29408]: pgmap v611: 158 pgs: 158 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 815 op/s 2024-09-18T07:42:27.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:27 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:42:27.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:27 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/383289003' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:42:27.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:27 smithi137 ceph-mon[29408]: from='client.25972 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:42:27.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:42:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:42:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:42:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:27 smithi137 ceph-mon[39175]: pgmap v611: 158 pgs: 158 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 815 op/s 2024-09-18T07:42:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:27 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:42:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:27 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/383289003' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:42:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:27 smithi137 ceph-mon[39175]: from='client.25972 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:42:27.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:27 smithi203 ceph-mon[32237]: pgmap v611: 158 pgs: 158 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 815 op/s 2024-09-18T07:42:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:27 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:42:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:27 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/383289003' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:42:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:27 smithi203 ceph-mon[32237]: from='client.25972 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:42:27.464 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:27.499 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:27.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:27.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:27.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:27.635 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:27.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:27.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:27.737 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:27.774 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:27.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:27.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:27.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:27.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:27.897 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:27.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:27.961 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:42:27 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:42:27.998 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:28.028 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:28.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:28.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:28.130 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:28.162 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:28.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:28.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:28.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:28.298 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:28.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:28.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:28.375 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:28 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:42:28.375 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:28 smithi137 ceph-mon[29408]: from='client.25972 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:42:28.375 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:28 smithi137 ceph-mon[29408]: osdmap e352: 8 total, 8 up, 8 in 2024-09-18T07:42:28.375 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:28 smithi137 ceph-mon[29408]: 4.1e scrub starts 2024-09-18T07:42:28.375 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:28 smithi137 ceph-mon[29408]: 4.1e scrub ok 2024-09-18T07:42:28.375 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:28 smithi137 ceph-mon[29408]: 9.c scrub starts 2024-09-18T07:42:28.375 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:28 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:42:28.376 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:28 smithi137 ceph-mon[39175]: from='client.25972 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:42:28.376 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:28 smithi137 ceph-mon[39175]: osdmap e352: 8 total, 8 up, 8 in 2024-09-18T07:42:28.376 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:28 smithi137 ceph-mon[39175]: 4.1e scrub starts 2024-09-18T07:42:28.376 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:28 smithi137 ceph-mon[39175]: 4.1e scrub ok 2024-09-18T07:42:28.376 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:28 smithi137 ceph-mon[39175]: 9.c scrub starts 2024-09-18T07:42:28.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:28.427 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:28.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:28.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:28.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:28 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:42:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:28 smithi203 ceph-mon[32237]: from='client.25972 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:42:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:28 smithi203 ceph-mon[32237]: osdmap e352: 8 total, 8 up, 8 in 2024-09-18T07:42:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:28 smithi203 ceph-mon[32237]: 4.1e scrub starts 2024-09-18T07:42:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:28 smithi203 ceph-mon[32237]: 4.1e scrub ok 2024-09-18T07:42:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:28 smithi203 ceph-mon[32237]: 9.c scrub starts 2024-09-18T07:42:28.528 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:28.566 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:28.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:28.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:28.668 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:28.704 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:28.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:28.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:28.806 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:28.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:28.828 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:28.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:28.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:28.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:28.963 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:28.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:29.064 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:29.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:29.101 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:29.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:29.202 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:29.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:29.224 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:29.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:29.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:29.346 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:29.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:29.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:29.448 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:29.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:29 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:29.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:29 smithi203 ceph-mon[32237]: pgmap v613: 158 pgs: 158 active+clean; 1.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.99k op/s 2024-09-18T07:42:29.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:29 smithi203 ceph-mon[32237]: 9.c scrub ok 2024-09-18T07:42:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:29 smithi203 ceph-mon[32237]: 5.1a scrub starts 2024-09-18T07:42:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:29 smithi203 ceph-mon[32237]: 5.1a scrub ok 2024-09-18T07:42:29.469 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:29.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:29.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:29.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:29.593 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:29.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:29.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:29.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:29 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:29 smithi137 ceph-mon[29408]: pgmap v613: 158 pgs: 158 active+clean; 1.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.99k op/s 2024-09-18T07:42:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:29 smithi137 ceph-mon[29408]: 9.c scrub ok 2024-09-18T07:42:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:29 smithi137 ceph-mon[29408]: 5.1a scrub starts 2024-09-18T07:42:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:29 smithi137 ceph-mon[29408]: 5.1a scrub ok 2024-09-18T07:42:29.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:29 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:29 smithi137 ceph-mon[39175]: pgmap v613: 158 pgs: 158 active+clean; 1.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.99k op/s 2024-09-18T07:42:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:29 smithi137 ceph-mon[39175]: 9.c scrub ok 2024-09-18T07:42:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:29 smithi137 ceph-mon[39175]: 5.1a scrub starts 2024-09-18T07:42:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:29 smithi137 ceph-mon[39175]: 5.1a scrub ok 2024-09-18T07:42:29.695 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:29.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:29.717 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:29.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:29.818 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:29.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:29.854 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:29.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:29.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:29.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:29.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:29.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:30.079 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:30.110 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:30.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:30.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:30.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:30.235 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:30.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:30.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:30.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:30.371 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:30.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:30 smithi203 ceph-mon[32237]: 5.1b scrub starts 2024-09-18T07:42:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:30 smithi203 ceph-mon[32237]: 5.1b scrub ok 2024-09-18T07:42:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:30 smithi203 ceph-mon[32237]: 4.14 scrub starts 2024-09-18T07:42:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:30 smithi203 ceph-mon[32237]: 4.14 scrub ok 2024-09-18T07:42:30.472 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:30.493 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:30.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:30.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:30.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:30.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:30.626 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:30.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:30.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:30 smithi137 ceph-mon[29408]: 5.1b scrub starts 2024-09-18T07:42:30.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:30 smithi137 ceph-mon[29408]: 5.1b scrub ok 2024-09-18T07:42:30.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:30 smithi137 ceph-mon[29408]: 4.14 scrub starts 2024-09-18T07:42:30.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:30 smithi137 ceph-mon[29408]: 4.14 scrub ok 2024-09-18T07:42:30.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:30 smithi137 ceph-mon[39175]: 5.1b scrub starts 2024-09-18T07:42:30.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:30 smithi137 ceph-mon[39175]: 5.1b scrub ok 2024-09-18T07:42:30.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:30 smithi137 ceph-mon[39175]: 4.14 scrub starts 2024-09-18T07:42:30.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:30 smithi137 ceph-mon[39175]: 4.14 scrub ok 2024-09-18T07:42:30.727 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:30.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:30.749 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:30.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:30.850 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:30.891 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:30.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:30.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:30.993 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:31.030 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:31.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:31.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:31.132 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:31.156 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:31.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:31.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:31.302 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:31.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:31.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:31.437 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:31.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:31.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:31 smithi203 ceph-mon[32237]: 1.a scrub starts 2024-09-18T07:42:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:31 smithi203 ceph-mon[32237]: 1.a scrub ok 2024-09-18T07:42:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:31 smithi203 ceph-mon[32237]: pgmap v614: 158 pgs: 158 active+clean; 1.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.43k op/s 2024-09-18T07:42:31.468 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:31 smithi137 ceph-mon[29408]: 1.a scrub starts 2024-09-18T07:42:31.468 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:31 smithi137 ceph-mon[29408]: 1.a scrub ok 2024-09-18T07:42:31.468 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:31 smithi137 ceph-mon[29408]: pgmap v614: 158 pgs: 158 active+clean; 1.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.43k op/s 2024-09-18T07:42:31.468 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:31 smithi137 ceph-mon[39175]: 1.a scrub starts 2024-09-18T07:42:31.468 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:31 smithi137 ceph-mon[39175]: 1.a scrub ok 2024-09-18T07:42:31.469 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:31 smithi137 ceph-mon[39175]: pgmap v614: 158 pgs: 158 active+clean; 1.9 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.43k op/s 2024-09-18T07:42:31.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T07:42:31.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:31.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:31.600 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:31.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:31.701 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:31.742 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:31.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:31.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:31.843 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:31.874 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:31.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:31.975 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:32.006 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:32.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:32.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:32.108 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:32.139 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:32.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:32.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:32.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:32.277 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:32.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:32.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:32.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:32.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:32.415 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:32.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:32.483 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:32 smithi137 ceph-mon[29408]: 3.1a scrub starts 2024-09-18T07:42:32.484 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:32 smithi137 ceph-mon[29408]: 3.1a scrub ok 2024-09-18T07:42:32.484 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:32 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:42:32.484 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:32 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:42:32.484 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:32 smithi137 ceph-mon[39175]: 3.1a scrub starts 2024-09-18T07:42:32.484 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:32 smithi137 ceph-mon[39175]: 3.1a scrub ok 2024-09-18T07:42:32.484 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:32 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:42:32.484 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:32 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:42:32.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:32.545 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:32.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:32.647 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:32.677 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:32.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:32.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:32.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:32 smithi203 ceph-mon[32237]: 3.1a scrub starts 2024-09-18T07:42:32.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:32 smithi203 ceph-mon[32237]: 3.1a scrub ok 2024-09-18T07:42:32.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:32 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:42:32.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:32 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:42:32.779 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:32.813 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:32.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:32.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:32.915 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:32.938 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:32.945 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:32.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:32.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:33.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:33.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:33.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:33.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:33.187 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:33.212 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:33.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:33.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:33.238 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:33.313 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:33.328 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:42:33.330 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:42:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:33.345 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:33.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:33.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:33 smithi137 ceph-mon[29408]: pgmap v615: 158 pgs: 158 active+clean; 1.8 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s 2024-09-18T07:42:33.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:33 smithi137 ceph-mon[29408]: 3.4 scrub starts 2024-09-18T07:42:33.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:33 smithi137 ceph-mon[29408]: 3.4 scrub ok 2024-09-18T07:42:33.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:33 smithi137 ceph-mon[29408]: osdmap e353: 8 total, 8 up, 8 in 2024-09-18T07:42:33.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:33 smithi137 ceph-mon[29408]: 1.9 scrub starts 2024-09-18T07:42:33.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:33 smithi137 ceph-mon[29408]: 1.9 scrub ok 2024-09-18T07:42:33.347 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:33 smithi137 ceph-mon[39175]: pgmap v615: 158 pgs: 158 active+clean; 1.8 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s 2024-09-18T07:42:33.348 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:33 smithi137 ceph-mon[39175]: 3.4 scrub starts 2024-09-18T07:42:33.348 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:33 smithi137 ceph-mon[39175]: 3.4 scrub ok 2024-09-18T07:42:33.348 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:33 smithi137 ceph-mon[39175]: osdmap e353: 8 total, 8 up, 8 in 2024-09-18T07:42:33.348 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:33 smithi137 ceph-mon[39175]: 1.9 scrub starts 2024-09-18T07:42:33.348 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:33 smithi137 ceph-mon[39175]: 1.9 scrub ok 2024-09-18T07:42:33.445 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:33.480 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:33.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:33.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:33.582 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:33.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:33.632 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:33.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:33.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:33 smithi203 ceph-mon[32237]: pgmap v615: 158 pgs: 158 active+clean; 1.8 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s 2024-09-18T07:42:33.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:33 smithi203 ceph-mon[32237]: 3.4 scrub starts 2024-09-18T07:42:33.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:33 smithi203 ceph-mon[32237]: 3.4 scrub ok 2024-09-18T07:42:33.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:33 smithi203 ceph-mon[32237]: osdmap e353: 8 total, 8 up, 8 in 2024-09-18T07:42:33.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:33 smithi203 ceph-mon[32237]: 1.9 scrub starts 2024-09-18T07:42:33.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:33 smithi203 ceph-mon[32237]: 1.9 scrub ok 2024-09-18T07:42:33.733 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:33.764 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:33.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:33.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:33.866 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:33.897 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:33.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:33.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:33.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:34.038 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:34.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:34.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:34.108 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:42:34.108 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:42:34.140 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:34.146 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:42:34.147 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:42:34.172 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:42:34.172 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:42:34.174 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:34.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:34.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:34.211 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:42:34.211 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:42:34.257 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:42:34.257 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:42:34.276 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:34.296 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:42:34.297 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:42:34.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:34.318 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:34.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:34.329 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:42:34.330 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:42:34.352 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:42:34.352 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:42:34.352 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:42:34.352 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:42:34.352 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:42:34.370 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:42:34.370 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:42:34.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:42:34.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:34.448 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:34.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:34.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:34.451 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:34 smithi137 ceph-mon[29408]: 9.f scrub starts 2024-09-18T07:42:34.451 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:34 smithi137 ceph-mon[29408]: 9.f scrub ok 2024-09-18T07:42:34.451 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:34 smithi137 ceph-mon[29408]: from='client.25978 v1:172.21.15.137:0/2231634491' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:42:34.451 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:34 smithi137 ceph-mon[29408]: 4.f scrub starts 2024-09-18T07:42:34.451 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:34 smithi137 ceph-mon[29408]: 4.f scrub ok 2024-09-18T07:42:34.451 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:34 smithi137 ceph-mon[29408]: pgmap v617: 158 pgs: 158 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-18T07:42:34.451 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:34 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/728214740' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:34.451 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:34 smithi137 ceph-mon[39175]: 9.f scrub starts 2024-09-18T07:42:34.451 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:34 smithi137 ceph-mon[39175]: 9.f scrub ok 2024-09-18T07:42:34.452 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:34 smithi137 ceph-mon[39175]: from='client.25978 v1:172.21.15.137:0/2231634491' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:42:34.452 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:34 smithi137 ceph-mon[39175]: 4.f scrub starts 2024-09-18T07:42:34.452 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:34 smithi137 ceph-mon[39175]: 4.f scrub ok 2024-09-18T07:42:34.452 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:34 smithi137 ceph-mon[39175]: pgmap v617: 158 pgs: 158 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-18T07:42:34.452 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:34 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/728214740' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:34.550 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:34.589 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:34.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:34.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:34.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:34.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:34 smithi203 ceph-mon[32237]: 9.f scrub starts 2024-09-18T07:42:34.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:34 smithi203 ceph-mon[32237]: 9.f scrub ok 2024-09-18T07:42:34.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:34 smithi203 ceph-mon[32237]: from='client.25978 v1:172.21.15.137:0/2231634491' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:42:34.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:34 smithi203 ceph-mon[32237]: 4.f scrub starts 2024-09-18T07:42:34.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:34 smithi203 ceph-mon[32237]: 4.f scrub ok 2024-09-18T07:42:34.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:34 smithi203 ceph-mon[32237]: pgmap v617: 158 pgs: 158 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-18T07:42:34.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:34 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/728214740' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:34.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:34.741 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:34.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:34.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:34.897 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:34.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:34.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:35.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:35.043 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:35.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:35.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:35.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:35.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:35.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:35.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:35.301 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:35.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:35.403 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:35.424 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:35.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:35.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:35.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:35.566 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:35.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:35.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:35.667 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:35.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:35 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:42:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:35 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:42:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:35 smithi137 ceph-mon[29408]: 4.16 scrub starts 2024-09-18T07:42:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:35 smithi137 ceph-mon[29408]: 4.16 scrub ok 2024-09-18T07:42:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:35 smithi137 ceph-mon[29408]: 6.13 scrub starts 2024-09-18T07:42:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:35 smithi137 ceph-mon[29408]: 6.13 scrub ok 2024-09-18T07:42:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:35 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:42:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:35 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:42:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:35 smithi137 ceph-mon[39175]: 4.16 scrub starts 2024-09-18T07:42:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:35 smithi137 ceph-mon[39175]: 4.16 scrub ok 2024-09-18T07:42:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:35 smithi137 ceph-mon[39175]: 6.13 scrub starts 2024-09-18T07:42:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:35 smithi137 ceph-mon[39175]: 6.13 scrub ok 2024-09-18T07:42:35.701 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:35.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:35.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:35.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:35 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:42:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:35 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:42:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:35 smithi203 ceph-mon[32237]: 4.16 scrub starts 2024-09-18T07:42:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:35 smithi203 ceph-mon[32237]: 4.16 scrub ok 2024-09-18T07:42:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:35 smithi203 ceph-mon[32237]: 6.13 scrub starts 2024-09-18T07:42:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:35 smithi203 ceph-mon[32237]: 6.13 scrub ok 2024-09-18T07:42:35.803 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:35.838 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:35.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:35.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:35.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:35.974 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:35.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:35.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:36.076 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:36.109 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:36.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:36.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:36.211 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:36.250 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:36.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:36.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:36.352 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:36.373 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:36.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:36.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:36.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:36.499 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:36.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:36.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:36.600 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:36.631 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:36.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:36.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:36.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[29408]: 6.14 scrub starts 2024-09-18T07:42:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[29408]: 6.14 scrub ok 2024-09-18T07:42:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[29408]: osdmap e354: 8 total, 8 up, 8 in 2024-09-18T07:42:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:42:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:42:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[29408]: 3.2 scrub starts 2024-09-18T07:42:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[29408]: 3.2 scrub ok 2024-09-18T07:42:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[29408]: pgmap v619: 158 pgs: 158 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 673 op/s 2024-09-18T07:42:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[29408]: 5.9 scrub starts 2024-09-18T07:42:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[29408]: 5.9 scrub ok 2024-09-18T07:42:36.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[39175]: 6.14 scrub starts 2024-09-18T07:42:36.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[39175]: 6.14 scrub ok 2024-09-18T07:42:36.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[39175]: osdmap e354: 8 total, 8 up, 8 in 2024-09-18T07:42:36.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:42:36.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:42:36.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[39175]: 3.2 scrub starts 2024-09-18T07:42:36.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[39175]: 3.2 scrub ok 2024-09-18T07:42:36.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[39175]: pgmap v619: 158 pgs: 158 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 673 op/s 2024-09-18T07:42:36.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[39175]: 5.9 scrub starts 2024-09-18T07:42:36.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:36 smithi137 ceph-mon[39175]: 5.9 scrub ok 2024-09-18T07:42:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:36 smithi203 ceph-mon[32237]: 6.14 scrub starts 2024-09-18T07:42:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:36 smithi203 ceph-mon[32237]: 6.14 scrub ok 2024-09-18T07:42:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:36 smithi203 ceph-mon[32237]: osdmap e354: 8 total, 8 up, 8 in 2024-09-18T07:42:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:36 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:42:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:36 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:42:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:36 smithi203 ceph-mon[32237]: 3.2 scrub starts 2024-09-18T07:42:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:36 smithi203 ceph-mon[32237]: 3.2 scrub ok 2024-09-18T07:42:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:36 smithi203 ceph-mon[32237]: pgmap v619: 158 pgs: 158 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 673 op/s 2024-09-18T07:42:36.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:36 smithi203 ceph-mon[32237]: 5.9 scrub starts 2024-09-18T07:42:36.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:36 smithi203 ceph-mon[32237]: 5.9 scrub ok 2024-09-18T07:42:36.733 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:36.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:36.775 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:36.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:36.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:36.906 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:36.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:36.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:37.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:37.043 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:37.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:37.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:37.103 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T07:42:37.145 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:37.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:37.193 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:37.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:37.248 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:42:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:42:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:42:37.295 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:37.335 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:37.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:37.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:37.437 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:37.474 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:37.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:37.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:37.504 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:37 smithi137 ceph-mon[29408]: osdmap e355: 8 total, 8 up, 8 in 2024-09-18T07:42:37.505 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:37 smithi137 ceph-mon[29408]: 6.7 scrub starts 2024-09-18T07:42:37.505 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:37 smithi137 ceph-mon[29408]: 6.7 scrub ok 2024-09-18T07:42:37.505 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:37 smithi137 ceph-mon[39175]: osdmap e355: 8 total, 8 up, 8 in 2024-09-18T07:42:37.505 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:37 smithi137 ceph-mon[39175]: 6.7 scrub starts 2024-09-18T07:42:37.505 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:37 smithi137 ceph-mon[39175]: 6.7 scrub ok 2024-09-18T07:42:37.536 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:37 smithi203 ceph-mon[32237]: osdmap e355: 8 total, 8 up, 8 in 2024-09-18T07:42:37.536 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:37 smithi203 ceph-mon[32237]: 6.7 scrub starts 2024-09-18T07:42:37.537 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:37 smithi203 ceph-mon[32237]: 6.7 scrub ok 2024-09-18T07:42:37.577 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:37.611 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:37.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:37.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:37.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:38.020 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":616,"stamp":"2024-09-18T07:42:32.215818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1782520371,"num_objects":27390,"num_object_clones":0,"num_object_copies":82170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27390,"num_whiteouts":0,"num_read":1589,"num_read_kb":1341,"num_write":238527,"num_write_kb":8479854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":5625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81253,"ondisk_log_size":81253,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8701392,"kb_used_data":5269780,"kb_used_omap":12,"kb_used_meta":3431283,"kb_avail":741259824,"statfs":{"total":767960285184,"available":759050059776,"internally_reserved":0,"allocated":5396254720,"data_stored":5393419408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":3513634352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-644153344,"num_objects":-9829,"num_object_clones":0,"num_object_copies":-29487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9829,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9829,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.534127"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":627,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.172361+0000","last_change":"2024-09-18T07:41:15.846355+0000","last_active":"2024-09-18T07:42:27.172361+0000","last_peered":"2024-09-18T07:42:27.172361+0000","last_clean":"2024-09-18T07:42:27.172361+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:42:27.172361+0000","last_undegraded":"2024-09-18T07:42:27.172361+0000","last_fullsized":"2024-09-18T07:42:27.172361+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:15.846257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:15.846257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:39.140539+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"323'15","reported_seq":702,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.173983+0000","last_change":"2024-09-18T07:40:17.270061+0000","last_active":"2024-09-18T07:42:27.173983+0000","last_peered":"2024-09-18T07:42:27.173983+0000","last_clean":"2024-09-18T07:42:27.173983+0000","last_became_active":"2024-09-18T07:37:43.061125+0000","last_became_peered":"2024-09-18T07:37:43.061125+0000","last_unstale":"2024-09-18T07:42:27.173983+0000","last_undegraded":"2024-09-18T07:42:27.173983+0000","last_fullsized":"2024-09-18T07:42:27.173983+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:17.269977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:17.269977+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:43.934455+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":570,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.176706+0000","last_change":"2024-09-18T07:41:58.378943+0000","last_active":"2024-09-18T07:42:27.176706+0000","last_peered":"2024-09-18T07:42:27.176706+0000","last_clean":"2024-09-18T07:42:27.176706+0000","last_became_active":"2024-09-18T07:36:13.128805+0000","last_became_peered":"2024-09-18T07:36:13.128805+0000","last_unstale":"2024-09-18T07:42:27.176706+0000","last_undegraded":"2024-09-18T07:42:27.176706+0000","last_fullsized":"2024-09-18T07:42:27.176706+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:58.378571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:58.378571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:23.350846+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":602,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.173729+0000","last_change":"2024-09-18T07:40:47.287019+0000","last_active":"2024-09-18T07:42:27.173729+0000","last_peered":"2024-09-18T07:42:27.173729+0000","last_clean":"2024-09-18T07:42:27.173729+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:42:27.173729+0000","last_undegraded":"2024-09-18T07:42:27.173729+0000","last_fullsized":"2024-09-18T07:42:27.173729+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:47.286927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.286927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":521,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.170044+0000","last_change":"2024-09-18T07:41:12.872510+0000","last_active":"2024-09-18T07:42:27.170044+0000","last_peered":"2024-09-18T07:42:27.170044+0000","last_clean":"2024-09-18T07:42:27.170044+0000","last_became_active":"2024-09-18T07:37:43.056380+0000","last_became_peered":"2024-09-18T07:37:43.056380+0000","last_unstale":"2024-09-18T07:42:27.170044+0000","last_undegraded":"2024-09-18T07:42:27.170044+0000","last_fullsized":"2024-09-18T07:42:27.170044+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:12.872387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:12.872387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:41.658070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":628,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.180057+0000","last_change":"2024-09-18T07:40:57.319270+0000","last_active":"2024-09-18T07:42:27.180057+0000","last_peered":"2024-09-18T07:42:27.180057+0000","last_clean":"2024-09-18T07:42:27.180057+0000","last_became_active":"2024-09-18T07:37:43.066508+0000","last_became_peered":"2024-09-18T07:37:43.066508+0000","last_unstale":"2024-09-18T07:42:27.180057+0000","last_undegraded":"2024-09-18T07:42:27.180057+0000","last_fullsized":"2024-09-18T07:42:27.180057+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:57.319204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:57.319204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:22.438323+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"323'27","reported_seq":172,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.173285+0000","last_change":"2024-09-18T07:40:20.436791+0000","last_active":"2024-09-18T07:42:27.173285+0000","last_peered":"2024-09-18T07:42:27.173285+0000","last_clean":"2024-09-18T07:42:27.173285+0000","last_became_active":"2024-09-18T07:40:19.064755+0000","last_became_peered":"2024-09-18T07:40:19.064755+0000","last_unstale":"2024-09-18T07:42:27.173285+0000","last_undegraded":"2024-09-18T07:42:27.173285+0000","last_fullsized":"2024-09-18T07:42:27.173285+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:40:20.436702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:20.436702+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:46.054779+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":636,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.173887+0000","last_change":"2024-09-18T07:41:51.001121+0000","last_active":"2024-09-18T07:42:27.173887+0000","last_peered":"2024-09-18T07:42:27.173887+0000","last_clean":"2024-09-18T07:42:27.173887+0000","last_became_active":"2024-09-18T07:37:43.052346+0000","last_became_peered":"2024-09-18T07:37:43.052346+0000","last_unstale":"2024-09-18T07:42:27.173887+0000","last_undegraded":"2024-09-18T07:42:27.173887+0000","last_fullsized":"2024-09-18T07:42:27.173887+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:51.001046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:51.001046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:55.981852+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":626,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.178205+0000","last_change":"2024-09-18T07:37:43.080879+0000","last_active":"2024-09-18T07:42:27.178205+0000","last_peered":"2024-09-18T07:42:27.178205+0000","last_clean":"2024-09-18T07:42:27.178205+0000","last_became_active":"2024-09-18T07:37:43.080314+0000","last_became_peered":"2024-09-18T07:37:43.080314+0000","last_unstale":"2024-09-18T07:42:27.178205+0000","last_undegraded":"2024-09-18T07:42:27.178205+0000","last_fullsized":"2024-09-18T07:42:27.178205+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:00.392466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:35:00.392466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:39.270967+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"323'15","reported_seq":643,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174042+0000","last_change":"2024-09-18T07:41:56.021202+0000","last_active":"2024-09-18T07:42:27.174042+0000","last_peered":"2024-09-18T07:42:27.174042+0000","last_clean":"2024-09-18T07:42:27.174042+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:42:27.174042+0000","last_undegraded":"2024-09-18T07:42:27.174042+0000","last_fullsized":"2024-09-18T07:42:27.174042+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:56.021118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:56.021118+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:16.099361+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":626,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:28.519199+0000","last_change":"2024-09-18T07:42:28.519199+0000","last_active":"2024-09-18T07:42:28.519199+0000","last_peered":"2024-09-18T07:42:28.519199+0000","last_clean":"2024-09-18T07:42:28.519199+0000","last_became_active":"2024-09-18T07:35:01.413443+0000","last_became_peered":"2024-09-18T07:35:01.413443+0000","last_unstale":"2024-09-18T07:42:28.519199+0000","last_undegraded":"2024-09-18T07:42:28.519199+0000","last_fullsized":"2024-09-18T07:42:28.519199+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:28.518968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:28.518968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:46.526654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":612,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174328+0000","last_change":"2024-09-18T07:42:00.028623+0000","last_active":"2024-09-18T07:42:27.174328+0000","last_peered":"2024-09-18T07:42:27.174328+0000","last_clean":"2024-09-18T07:42:27.174328+0000","last_became_active":"2024-09-18T07:35:01.410284+0000","last_became_peered":"2024-09-18T07:35:01.410284+0000","last_unstale":"2024-09-18T07:42:27.174328+0000","last_undegraded":"2024-09-18T07:42:27.174328+0000","last_fullsized":"2024-09-18T07:42:27.174328+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:00.028559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:00.028559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:26.482816+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":612,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.177349+0000","last_change":"2024-09-18T07:41:57.358913+0000","last_active":"2024-09-18T07:42:27.177349+0000","last_peered":"2024-09-18T07:42:27.177349+0000","last_clean":"2024-09-18T07:42:27.177349+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:42:27.177349+0000","last_undegraded":"2024-09-18T07:42:27.177349+0000","last_fullsized":"2024-09-18T07:42:27.177349+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:57.358853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:57.358853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:12.134323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"323'7","reported_seq":636,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.173808+0000","last_change":"2024-09-18T07:39:26.028935+0000","last_active":"2024-09-18T07:42:27.173808+0000","last_peered":"2024-09-18T07:42:27.173808+0000","last_clean":"2024-09-18T07:42:27.173808+0000","last_became_active":"2024-09-18T07:36:02.526212+0000","last_became_peered":"2024-09-18T07:36:02.526212+0000","last_unstale":"2024-09-18T07:42:27.173808+0000","last_undegraded":"2024-09-18T07:42:27.173808+0000","last_fullsized":"2024-09-18T07:42:27.173808+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-18T07:39:26.028884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:26.028884+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:39.929629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":395,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:28.855611+0000","last_change":"2024-09-18T07:42:28.855544+0000","last_active":"2024-09-18T07:42:28.855611+0000","last_peered":"2024-09-18T07:42:28.855611+0000","last_clean":"2024-09-18T07:42:28.855611+0000","last_became_active":"2024-09-18T07:35:01.398605+0000","last_became_peered":"2024-09-18T07:35:01.398605+0000","last_unstale":"2024-09-18T07:42:28.855611+0000","last_undegraded":"2024-09-18T07:42:28.855611+0000","last_fullsized":"2024-09-18T07:42:28.855611+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:28.855481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:28.855481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:37.908975+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":624,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.176783+0000","last_change":"2024-09-18T07:40:50.367758+0000","last_active":"2024-09-18T07:42:27.176783+0000","last_peered":"2024-09-18T07:42:27.176783+0000","last_clean":"2024-09-18T07:42:27.176783+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:42:27.176783+0000","last_undegraded":"2024-09-18T07:42:27.176783+0000","last_fullsized":"2024-09-18T07:42:27.176783+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:50.367694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.367694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":625,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.175028+0000","last_change":"2024-09-18T07:38:56.096769+0000","last_active":"2024-09-18T07:42:27.175028+0000","last_peered":"2024-09-18T07:42:27.175028+0000","last_clean":"2024-09-18T07:42:27.175028+0000","last_became_active":"2024-09-18T07:37:43.067568+0000","last_became_peered":"2024-09-18T07:37:43.067568+0000","last_unstale":"2024-09-18T07:42:27.175028+0000","last_undegraded":"2024-09-18T07:42:27.175028+0000","last_fullsized":"2024-09-18T07:42:27.175028+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:56.096591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:56.096591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:30.878059+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":632,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.170765+0000","last_change":"2024-09-18T07:42:04.117872+0000","last_active":"2024-09-18T07:42:27.170765+0000","last_peered":"2024-09-18T07:42:27.170765+0000","last_clean":"2024-09-18T07:42:27.170765+0000","last_became_active":"2024-09-18T07:35:01.414088+0000","last_became_peered":"2024-09-18T07:35:01.414088+0000","last_unstale":"2024-09-18T07:42:27.170765+0000","last_undegraded":"2024-09-18T07:42:27.170765+0000","last_fullsized":"2024-09-18T07:42:27.170765+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:04.117653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.117653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:22.791068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"323'27","reported_seq":715,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.175253+0000","last_change":"2024-09-18T07:41:48.128836+0000","last_active":"2024-09-18T07:42:27.175253+0000","last_peered":"2024-09-18T07:42:27.175253+0000","last_clean":"2024-09-18T07:42:27.175253+0000","last_became_active":"2024-09-18T07:37:28.002858+0000","last_became_peered":"2024-09-18T07:37:28.002858+0000","last_unstale":"2024-09-18T07:42:27.175253+0000","last_undegraded":"2024-09-18T07:42:27.175253+0000","last_fullsized":"2024-09-18T07:42:27.175253+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:41:48.128756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.128756+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:01.870808+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":305,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174608+0000","last_change":"2024-09-18T07:41:26.922547+0000","last_active":"2024-09-18T07:42:27.174608+0000","last_peered":"2024-09-18T07:42:27.174608+0000","last_clean":"2024-09-18T07:42:27.174608+0000","last_became_active":"2024-09-18T07:36:13.116416+0000","last_became_peered":"2024-09-18T07:36:13.116416+0000","last_unstale":"2024-09-18T07:42:27.174608+0000","last_undegraded":"2024-09-18T07:42:27.174608+0000","last_fullsized":"2024-09-18T07:42:27.174608+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:26.922506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:26.922506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:55.427286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":480,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.184745+0000","last_change":"2024-09-18T07:40:19.070769+0000","last_active":"2024-09-18T07:42:27.184745+0000","last_peered":"2024-09-18T07:42:27.184745+0000","last_clean":"2024-09-18T07:42:27.184745+0000","last_became_active":"2024-09-18T07:40:19.070463+0000","last_became_peered":"2024-09-18T07:40:19.070463+0000","last_unstale":"2024-09-18T07:42:27.184745+0000","last_undegraded":"2024-09-18T07:42:27.184745+0000","last_fullsized":"2024-09-18T07:42:27.184745+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:11.831891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:11.831891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:43.316131+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":630,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:31.324040+0000","last_change":"2024-09-18T07:42:31.324040+0000","last_active":"2024-09-18T07:42:31.324040+0000","last_peered":"2024-09-18T07:42:31.324040+0000","last_clean":"2024-09-18T07:42:31.324040+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:42:31.324040+0000","last_undegraded":"2024-09-18T07:42:31.324040+0000","last_fullsized":"2024-09-18T07:42:31.324040+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:31.323963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:31.323963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:32.270399+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"323'18","reported_seq":285,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174056+0000","last_change":"2024-09-18T07:41:59.149841+0000","last_active":"2024-09-18T07:42:27.174056+0000","last_peered":"2024-09-18T07:42:27.174056+0000","last_clean":"2024-09-18T07:42:27.174056+0000","last_became_active":"2024-09-18T07:37:27.981705+0000","last_became_peered":"2024-09-18T07:37:27.981705+0000","last_unstale":"2024-09-18T07:42:27.174056+0000","last_undegraded":"2024-09-18T07:42:27.174056+0000","last_fullsized":"2024-09-18T07:42:27.174056+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:41:59.149629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:59.149629+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:25.055296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":598,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174460+0000","last_change":"2024-09-18T07:39:37.930497+0000","last_active":"2024-09-18T07:42:27.174460+0000","last_peered":"2024-09-18T07:42:27.174460+0000","last_clean":"2024-09-18T07:42:27.174460+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:42:27.174460+0000","last_undegraded":"2024-09-18T07:42:27.174460+0000","last_fullsized":"2024-09-18T07:42:27.174460+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.930435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.930435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:13.036846+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":399,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.187864+0000","last_change":"2024-09-18T07:40:10.644587+0000","last_active":"2024-09-18T07:42:27.187864+0000","last_peered":"2024-09-18T07:42:27.187864+0000","last_clean":"2024-09-18T07:42:27.187864+0000","last_became_active":"2024-09-18T07:35:01.413110+0000","last_became_peered":"2024-09-18T07:35:01.413110+0000","last_unstale":"2024-09-18T07:42:27.187864+0000","last_undegraded":"2024-09-18T07:42:27.187864+0000","last_fullsized":"2024-09-18T07:42:27.187864+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:40:10.644461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:10.644461+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:11.785210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":314,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.187856+0000","last_change":"2024-09-18T07:38:50.918758+0000","last_active":"2024-09-18T07:42:27.187856+0000","last_peered":"2024-09-18T07:42:27.187856+0000","last_clean":"2024-09-18T07:42:27.187856+0000","last_became_active":"2024-09-18T07:36:13.116295+0000","last_became_peered":"2024-09-18T07:36:13.116295+0000","last_unstale":"2024-09-18T07:42:27.187856+0000","last_undegraded":"2024-09-18T07:42:27.187856+0000","last_fullsized":"2024-09-18T07:42:27.187856+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:38:50.918686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:38:50.918686+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:35.433557+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"323'20","reported_seq":691,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.188820+0000","last_change":"2024-09-18T07:42:27.188820+0000","last_active":"2024-09-18T07:42:27.188820+0000","last_peered":"2024-09-18T07:42:27.188820+0000","last_clean":"2024-09-18T07:42:27.188820+0000","last_became_active":"2024-09-18T07:35:01.410926+0000","last_became_peered":"2024-09-18T07:35:01.410926+0000","last_unstale":"2024-09-18T07:42:27.188820+0000","last_undegraded":"2024-09-18T07:42:27.188820+0000","last_fullsized":"2024-09-18T07:42:27.188820+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:42:27.188530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:27.188530+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:44.776667+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"240'12","reported_seq":316,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.184881+0000","last_change":"2024-09-18T07:40:43.112526+0000","last_active":"2024-09-18T07:42:27.184881+0000","last_peered":"2024-09-18T07:42:27.184881+0000","last_clean":"2024-09-18T07:42:27.184881+0000","last_became_active":"2024-09-18T07:37:43.060715+0000","last_became_peered":"2024-09-18T07:37:43.060715+0000","last_unstale":"2024-09-18T07:42:27.184881+0000","last_undegraded":"2024-09-18T07:42:27.184881+0000","last_fullsized":"2024-09-18T07:42:27.184881+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:40:43.112459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:43.112459+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:21.098107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"352'14711","reported_seq":14956,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.394076+0000","last_change":"2024-09-18T07:42:00.674258+0000","last_active":"2024-09-18T07:42:27.394076+0000","last_peered":"2024-09-18T07:42:27.394076+0000","last_clean":"2024-09-18T07:42:27.394076+0000","last_became_active":"2024-09-18T07:39:29.700477+0000","last_became_peered":"2024-09-18T07:39:29.700477+0000","last_unstale":"2024-09-18T07:42:27.394076+0000","last_undegraded":"2024-09-18T07:42:27.394076+0000","last_fullsized":"2024-09-18T07:42:27.394076+0000","mapping_epoch":289,"log_start":"332'10100","ondisk_log_start":"332'10100","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'12643","last_scrub_stamp":"2024-09-18T07:42:00.674143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:42:00.674143+0000","objects_scrubbed":3950,"log_size":4611,"log_dups_size":0,"ondisk_log_size":4611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:17.584126+0000","scrub_duration":1266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120389632,"num_objects":1837,"num_object_clones":0,"num_object_copies":5511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14711,"num_write_kb":529536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":191,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:29.515462+0000","last_change":"2024-09-18T07:42:29.515462+0000","last_active":"2024-09-18T07:42:29.515462+0000","last_peered":"2024-09-18T07:42:29.515462+0000","last_clean":"2024-09-18T07:42:29.515462+0000","last_became_active":"2024-09-18T07:39:04.962047+0000","last_became_peered":"2024-09-18T07:39:04.962047+0000","last_unstale":"2024-09-18T07:42:29.515462+0000","last_undegraded":"2024-09-18T07:42:29.515462+0000","last_fullsized":"2024-09-18T07:42:29.515462+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:42:29.515416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:42:29.515416+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:48.241376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"323'27","reported_seq":368,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.180285+0000","last_change":"2024-09-18T07:38:54.967706+0000","last_active":"2024-09-18T07:42:27.180285+0000","last_peered":"2024-09-18T07:42:27.180285+0000","last_clean":"2024-09-18T07:42:27.180285+0000","last_became_active":"2024-09-18T07:36:13.119098+0000","last_became_peered":"2024-09-18T07:36:13.119098+0000","last_unstale":"2024-09-18T07:42:27.180285+0000","last_undegraded":"2024-09-18T07:42:27.180285+0000","last_fullsized":"2024-09-18T07:42:27.180285+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:38:54.967643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:38:54.967643+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:29.535790+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":644,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.184483+0000","last_change":"2024-09-18T07:39:29.049261+0000","last_active":"2024-09-18T07:42:27.184483+0000","last_peered":"2024-09-18T07:42:27.184483+0000","last_clean":"2024-09-18T07:42:27.184483+0000","last_became_active":"2024-09-18T07:35:01.407339+0000","last_became_peered":"2024-09-18T07:35:01.407339+0000","last_unstale":"2024-09-18T07:42:27.184483+0000","last_undegraded":"2024-09-18T07:42:27.184483+0000","last_fullsized":"2024-09-18T07:42:27.184483+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.049209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.049209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:20.464343+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":779,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.175164+0000","last_change":"2024-09-18T07:41:27.965286+0000","last_active":"2024-09-18T07:42:27.175164+0000","last_peered":"2024-09-18T07:42:27.175164+0000","last_clean":"2024-09-18T07:42:27.175164+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:42:27.175164+0000","last_undegraded":"2024-09-18T07:42:27.175164+0000","last_fullsized":"2024-09-18T07:42:27.175164+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:41:27.965205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.965205+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:47.540373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":609,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.177119+0000","last_change":"2024-09-18T07:42:06.507229+0000","last_active":"2024-09-18T07:42:27.177119+0000","last_peered":"2024-09-18T07:42:27.177119+0000","last_clean":"2024-09-18T07:42:27.177119+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:42:27.177119+0000","last_undegraded":"2024-09-18T07:42:27.177119+0000","last_fullsized":"2024-09-18T07:42:27.177119+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.507166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.507166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:10.596707+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"352'15104","reported_seq":15322,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:31.643348+0000","last_change":"2024-09-18T07:40:47.444123+0000","last_active":"2024-09-18T07:42:31.643348+0000","last_peered":"2024-09-18T07:42:31.643348+0000","last_clean":"2024-09-18T07:42:31.643348+0000","last_became_active":"2024-09-18T07:39:29.704590+0000","last_became_peered":"2024-09-18T07:39:29.704590+0000","last_unstale":"2024-09-18T07:42:31.643348+0000","last_undegraded":"2024-09-18T07:42:31.643348+0000","last_fullsized":"2024-09-18T07:42:31.643348+0000","mapping_epoch":289,"log_start":"334'10600","ondisk_log_start":"334'10600","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'7163","last_scrub_stamp":"2024-09-18T07:40:47.443974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.443974+0000","objects_scrubbed":7103,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:13.769013+0000","scrub_duration":1939,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103612440,"num_objects":1582,"num_object_clones":0,"num_object_copies":4746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1582,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15104,"num_write_kb":533889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":191,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174986+0000","last_change":"2024-09-18T07:41:53.004272+0000","last_active":"2024-09-18T07:42:27.174986+0000","last_peered":"2024-09-18T07:42:27.174986+0000","last_clean":"2024-09-18T07:42:27.174986+0000","last_became_active":"2024-09-18T07:39:04.955645+0000","last_became_peered":"2024-09-18T07:39:04.955645+0000","last_unstale":"2024-09-18T07:42:27.174986+0000","last_undegraded":"2024-09-18T07:42:27.174986+0000","last_fullsized":"2024-09-18T07:42:27.174986+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:41:53.004186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:41:53.004186+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:16.639727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"323'30","reported_seq":718,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.179055+0000","last_change":"2024-09-18T07:39:00.919075+0000","last_active":"2024-09-18T07:42:27.179055+0000","last_peered":"2024-09-18T07:42:27.179055+0000","last_clean":"2024-09-18T07:42:27.179055+0000","last_became_active":"2024-09-18T07:35:01.418909+0000","last_became_peered":"2024-09-18T07:35:01.418909+0000","last_unstale":"2024-09-18T07:42:27.179055+0000","last_undegraded":"2024-09-18T07:42:27.179055+0000","last_fullsized":"2024-09-18T07:42:27.179055+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:39:00.918860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.918860+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:39.501904+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":350,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.183311+0000","last_change":"2024-09-18T07:40:19.069243+0000","last_active":"2024-09-18T07:42:27.183311+0000","last_peered":"2024-09-18T07:42:27.183311+0000","last_clean":"2024-09-18T07:42:27.183311+0000","last_became_active":"2024-09-18T07:40:19.068296+0000","last_became_peered":"2024-09-18T07:40:19.068296+0000","last_unstale":"2024-09-18T07:42:27.183311+0000","last_undegraded":"2024-09-18T07:42:27.183311+0000","last_fullsized":"2024-09-18T07:42:27.183311+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:06.987428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:06.987428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:34.748523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":629,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174319+0000","last_change":"2024-09-18T07:42:09.068025+0000","last_active":"2024-09-18T07:42:27.174319+0000","last_peered":"2024-09-18T07:42:27.174319+0000","last_clean":"2024-09-18T07:42:27.174319+0000","last_became_active":"2024-09-18T07:35:01.401024+0000","last_became_peered":"2024-09-18T07:35:01.401024+0000","last_unstale":"2024-09-18T07:42:27.174319+0000","last_undegraded":"2024-09-18T07:42:27.174319+0000","last_fullsized":"2024-09-18T07:42:27.174319+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:09.067941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:09.067941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:25.306181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":595,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.173931+0000","last_change":"2024-09-18T07:40:44.219626+0000","last_active":"2024-09-18T07:42:27.173931+0000","last_peered":"2024-09-18T07:42:27.173931+0000","last_clean":"2024-09-18T07:42:27.173931+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:42:27.173931+0000","last_undegraded":"2024-09-18T07:42:27.173931+0000","last_fullsized":"2024-09-18T07:42:27.173931+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:44.219547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:44.219547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"352'15004","reported_seq":15220,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.368428+0000","last_change":"2024-09-18T07:41:25.657036+0000","last_active":"2024-09-18T07:42:27.368428+0000","last_peered":"2024-09-18T07:42:27.368428+0000","last_clean":"2024-09-18T07:42:27.368428+0000","last_became_active":"2024-09-18T07:39:29.702953+0000","last_became_peered":"2024-09-18T07:39:29.702953+0000","last_unstale":"2024-09-18T07:42:27.368428+0000","last_undegraded":"2024-09-18T07:42:27.368428+0000","last_fullsized":"2024-09-18T07:42:27.368428+0000","mapping_epoch":289,"log_start":"332'10400","ondisk_log_start":"332'10400","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'10075","last_scrub_stamp":"2024-09-18T07:41:25.656756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:41:25.656756+0000","objects_scrubbed":6968,"log_size":4604,"log_dups_size":0,"ondisk_log_size":4604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2236,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128974848,"num_objects":1968,"num_object_clones":0,"num_object_copies":5904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15004,"num_write_kb":543104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":188,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.180377+0000","last_change":"2024-09-18T07:41:19.479946+0000","last_active":"2024-09-18T07:42:27.180377+0000","last_peered":"2024-09-18T07:42:27.180377+0000","last_clean":"2024-09-18T07:42:27.180377+0000","last_became_active":"2024-09-18T07:39:04.945385+0000","last_became_peered":"2024-09-18T07:39:04.945385+0000","last_unstale":"2024-09-18T07:42:27.180377+0000","last_undegraded":"2024-09-18T07:42:27.180377+0000","last_fullsized":"2024-09-18T07:42:27.180377+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:41:19.479442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:41:19.479442+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:41.906548+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"323'28","reported_seq":699,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174809+0000","last_change":"2024-09-18T07:40:56.183888+0000","last_active":"2024-09-18T07:42:27.174809+0000","last_peered":"2024-09-18T07:42:27.174809+0000","last_clean":"2024-09-18T07:42:27.174809+0000","last_became_active":"2024-09-18T07:37:43.096236+0000","last_became_peered":"2024-09-18T07:37:43.096236+0000","last_unstale":"2024-09-18T07:42:27.174809+0000","last_undegraded":"2024-09-18T07:42:27.174809+0000","last_fullsized":"2024-09-18T07:42:27.174809+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-18T07:40:56.183802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.183802+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":612,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.172076+0000","last_change":"2024-09-18T07:39:28.750779+0000","last_active":"2024-09-18T07:42:27.172076+0000","last_peered":"2024-09-18T07:42:27.172076+0000","last_clean":"2024-09-18T07:42:27.172076+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:42:27.172076+0000","last_undegraded":"2024-09-18T07:42:27.172076+0000","last_fullsized":"2024-09-18T07:42:27.172076+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:39:28.750720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.750720+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:13.105049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":761,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.178865+0000","last_change":"2024-09-18T07:42:04.428459+0000","last_active":"2024-09-18T07:42:27.178865+0000","last_peered":"2024-09-18T07:42:27.178865+0000","last_clean":"2024-09-18T07:42:27.178865+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:42:27.178865+0000","last_undegraded":"2024-09-18T07:42:27.178865+0000","last_fullsized":"2024-09-18T07:42:27.178865+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:42:04.428395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.428395+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:17.768730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":401,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.185598+0000","last_change":"2024-09-18T07:38:58.528134+0000","last_active":"2024-09-18T07:42:27.185598+0000","last_peered":"2024-09-18T07:42:27.185598+0000","last_clean":"2024-09-18T07:42:27.185598+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:42:27.185598+0000","last_undegraded":"2024-09-18T07:42:27.185598+0000","last_fullsized":"2024-09-18T07:42:27.185598+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:58.528059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:58.528059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:39.895035+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"352'15067","reported_seq":15288,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:32.061499+0000","last_change":"2024-09-18T07:40:52.501916+0000","last_active":"2024-09-18T07:42:32.061499+0000","last_peered":"2024-09-18T07:42:32.061499+0000","last_clean":"2024-09-18T07:42:32.061499+0000","last_became_active":"2024-09-18T07:39:29.702301+0000","last_became_peered":"2024-09-18T07:39:29.702301+0000","last_unstale":"2024-09-18T07:42:32.061499+0000","last_undegraded":"2024-09-18T07:42:32.061499+0000","last_fullsized":"2024-09-18T07:42:32.061499+0000","mapping_epoch":289,"log_start":"333'10300","ondisk_log_start":"333'10300","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7465","last_scrub_stamp":"2024-09-18T07:40:52.501716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:52.501716+0000","objects_scrubbed":7371,"log_size":4767,"log_dups_size":0,"ondisk_log_size":4767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102301696,"num_objects":1561,"num_object_clones":0,"num_object_copies":4683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15067,"num_write_kb":532096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":351,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.184593+0000","last_change":"2024-09-18T07:40:38.066508+0000","last_active":"2024-09-18T07:42:27.184593+0000","last_peered":"2024-09-18T07:42:27.184593+0000","last_clean":"2024-09-18T07:42:27.184593+0000","last_became_active":"2024-09-18T07:39:04.941752+0000","last_became_peered":"2024-09-18T07:39:04.941752+0000","last_unstale":"2024-09-18T07:42:27.184593+0000","last_undegraded":"2024-09-18T07:42:27.184593+0000","last_fullsized":"2024-09-18T07:42:27.184593+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:38.066443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.066443+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:19.103720+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"323'20","reported_seq":364,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.177800+0000","last_change":"2024-09-18T07:42:03.427924+0000","last_active":"2024-09-18T07:42:27.177800+0000","last_peered":"2024-09-18T07:42:27.177800+0000","last_clean":"2024-09-18T07:42:27.177800+0000","last_became_active":"2024-09-18T07:36:13.118876+0000","last_became_peered":"2024-09-18T07:36:13.118876+0000","last_unstale":"2024-09-18T07:42:27.177800+0000","last_undegraded":"2024-09-18T07:42:27.177800+0000","last_fullsized":"2024-09-18T07:42:27.177800+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:42:03.427843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:03.427843+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:24.815361+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":629,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.178260+0000","last_change":"2024-09-18T07:40:53.350083+0000","last_active":"2024-09-18T07:42:27.178260+0000","last_peered":"2024-09-18T07:42:27.178260+0000","last_clean":"2024-09-18T07:42:27.178260+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:42:27.178260+0000","last_undegraded":"2024-09-18T07:42:27.178260+0000","last_fullsized":"2024-09-18T07:42:27.178260+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:53.350019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:53.350019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"240'10","reported_seq":780,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.170255+0000","last_change":"2024-09-18T07:40:50.839710+0000","last_active":"2024-09-18T07:42:27.170255+0000","last_peered":"2024-09-18T07:42:27.170255+0000","last_clean":"2024-09-18T07:42:27.170255+0000","last_became_active":"2024-09-18T07:37:43.167850+0000","last_became_peered":"2024-09-18T07:37:43.167850+0000","last_unstale":"2024-09-18T07:42:27.170255+0000","last_undegraded":"2024-09-18T07:42:27.170255+0000","last_fullsized":"2024-09-18T07:42:27.170255+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'10","last_scrub_stamp":"2024-09-18T07:40:50.839504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.839504+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:23.163039+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":591,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174455+0000","last_change":"2024-09-18T07:39:00.759483+0000","last_active":"2024-09-18T07:42:27.174455+0000","last_peered":"2024-09-18T07:42:27.174455+0000","last_clean":"2024-09-18T07:42:27.174455+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:42:27.174455+0000","last_undegraded":"2024-09-18T07:42:27.174455+0000","last_fullsized":"2024-09-18T07:42:27.174455+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:00.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:35.026613+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"352'14780","reported_seq":14964,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:31.643280+0000","last_change":"2024-09-18T07:42:04.807783+0000","last_active":"2024-09-18T07:42:31.643280+0000","last_peered":"2024-09-18T07:42:31.643280+0000","last_clean":"2024-09-18T07:42:31.643280+0000","last_became_active":"2024-09-18T07:39:29.703475+0000","last_became_peered":"2024-09-18T07:39:29.703475+0000","last_unstale":"2024-09-18T07:42:31.643280+0000","last_undegraded":"2024-09-18T07:42:31.643280+0000","last_fullsized":"2024-09-18T07:42:31.643280+0000","mapping_epoch":289,"log_start":"333'10300","ondisk_log_start":"333'10300","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'12776","last_scrub_stamp":"2024-09-18T07:42:04.807635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.807635+0000","objects_scrubbed":3710,"log_size":4480,"log_dups_size":0,"ondisk_log_size":4480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:27.077916+0000","scrub_duration":1049,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":1664,"num_object_clones":0,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14780,"num_write_kb":526208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":351,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.184305+0000","last_change":"2024-09-18T07:40:18.954153+0000","last_active":"2024-09-18T07:42:27.184305+0000","last_peered":"2024-09-18T07:42:27.184305+0000","last_clean":"2024-09-18T07:42:27.184305+0000","last_became_active":"2024-09-18T07:39:04.948571+0000","last_became_peered":"2024-09-18T07:39:04.948571+0000","last_unstale":"2024-09-18T07:42:27.184305+0000","last_undegraded":"2024-09-18T07:42:27.184305+0000","last_fullsized":"2024-09-18T07:42:27.184305+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:18.954089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:18.954089+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:40.336036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"323'29","reported_seq":724,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.170501+0000","last_change":"2024-09-18T07:41:16.895792+0000","last_active":"2024-09-18T07:42:27.170501+0000","last_peered":"2024-09-18T07:42:27.170501+0000","last_clean":"2024-09-18T07:42:27.170501+0000","last_became_active":"2024-09-18T07:35:01.415275+0000","last_became_peered":"2024-09-18T07:35:01.415275+0000","last_unstale":"2024-09-18T07:42:27.170501+0000","last_undegraded":"2024-09-18T07:42:27.170501+0000","last_fullsized":"2024-09-18T07:42:27.170501+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'29","last_scrub_stamp":"2024-09-18T07:41:16.895595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:16.895595+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:34.282784+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":314,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:31.793926+0000","last_change":"2024-09-18T07:42:31.793848+0000","last_active":"2024-09-18T07:42:31.793926+0000","last_peered":"2024-09-18T07:42:31.793926+0000","last_clean":"2024-09-18T07:42:31.793926+0000","last_became_active":"2024-09-18T07:36:13.118553+0000","last_became_peered":"2024-09-18T07:36:13.118553+0000","last_unstale":"2024-09-18T07:42:31.793926+0000","last_undegraded":"2024-09-18T07:42:31.793926+0000","last_fullsized":"2024-09-18T07:42:31.793926+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:31.793760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:31.793760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:58.491919+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":254,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.183231+0000","last_change":"2024-09-18T07:41:24.300282+0000","last_active":"2024-09-18T07:42:27.183231+0000","last_peered":"2024-09-18T07:42:27.183231+0000","last_clean":"2024-09-18T07:42:27.183231+0000","last_became_active":"2024-09-18T07:37:43.070009+0000","last_became_peered":"2024-09-18T07:37:43.070009+0000","last_unstale":"2024-09-18T07:42:27.183231+0000","last_undegraded":"2024-09-18T07:42:27.183231+0000","last_fullsized":"2024-09-18T07:42:27.183231+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:24.300235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:24.300235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:52.556718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":246,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.185428+0000","last_change":"2024-09-18T07:40:10.073596+0000","last_active":"2024-09-18T07:42:27.185428+0000","last_peered":"2024-09-18T07:42:27.185428+0000","last_clean":"2024-09-18T07:42:27.185428+0000","last_became_active":"2024-09-18T07:37:43.051740+0000","last_became_peered":"2024-09-18T07:37:43.051740+0000","last_unstale":"2024-09-18T07:42:27.185428+0000","last_undegraded":"2024-09-18T07:42:27.185428+0000","last_fullsized":"2024-09-18T07:42:27.185428+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:10.073499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:10.073499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:33.283517+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"352'14650","reported_seq":14827,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.614991+0000","last_change":"2024-09-18T07:42:10.328791+0000","last_active":"2024-09-18T07:42:27.614991+0000","last_peered":"2024-09-18T07:42:27.614991+0000","last_clean":"2024-09-18T07:42:27.614991+0000","last_became_active":"2024-09-18T07:39:29.701239+0000","last_became_peered":"2024-09-18T07:39:29.701239+0000","last_unstale":"2024-09-18T07:42:27.614991+0000","last_undegraded":"2024-09-18T07:42:27.614991+0000","last_fullsized":"2024-09-18T07:42:27.614991+0000","mapping_epoch":289,"log_start":"327'9100","ondisk_log_start":"327'9100","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'13234","last_scrub_stamp":"2024-09-18T07:42:10.328441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:42:10.328441+0000","objects_scrubbed":3282,"log_size":5550,"log_dups_size":0,"ondisk_log_size":5550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:26.764761+0000","scrub_duration":1183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119406592,"num_objects":1822,"num_object_clones":0,"num_object_copies":5466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14650,"num_write_kb":527104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":183,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.176549+0000","last_change":"2024-09-18T07:40:43.346516+0000","last_active":"2024-09-18T07:42:27.176549+0000","last_peered":"2024-09-18T07:42:27.176549+0000","last_clean":"2024-09-18T07:42:27.176549+0000","last_became_active":"2024-09-18T07:39:04.950564+0000","last_became_peered":"2024-09-18T07:39:04.950564+0000","last_unstale":"2024-09-18T07:42:27.176549+0000","last_undegraded":"2024-09-18T07:42:27.176549+0000","last_fullsized":"2024-09-18T07:42:27.176549+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:43.346447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:43.346447+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:23.534594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"323'41","reported_seq":486,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:31.497827+0000","last_change":"2024-09-18T07:41:55.160514+0000","last_active":"2024-09-18T07:42:31.497827+0000","last_peered":"2024-09-18T07:42:31.497827+0000","last_clean":"2024-09-18T07:42:31.497827+0000","last_became_active":"2024-09-18T07:36:13.115278+0000","last_became_peered":"2024-09-18T07:36:13.115278+0000","last_unstale":"2024-09-18T07:42:31.497827+0000","last_undegraded":"2024-09-18T07:42:31.497827+0000","last_fullsized":"2024-09-18T07:42:31.497827+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'41","last_scrub_stamp":"2024-09-18T07:41:55.160421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:55.160421+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:01.495646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":637,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174166+0000","last_change":"2024-09-18T07:42:03.028626+0000","last_active":"2024-09-18T07:42:27.174166+0000","last_peered":"2024-09-18T07:42:27.174166+0000","last_clean":"2024-09-18T07:42:27.174166+0000","last_became_active":"2024-09-18T07:37:43.060930+0000","last_became_peered":"2024-09-18T07:37:43.060930+0000","last_unstale":"2024-09-18T07:42:27.174166+0000","last_undegraded":"2024-09-18T07:42:27.174166+0000","last_fullsized":"2024-09-18T07:42:27.174166+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:03.028558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:03.028558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:12.110802+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":653,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.177928+0000","last_change":"2024-09-18T07:42:00.366508+0000","last_active":"2024-09-18T07:42:27.177928+0000","last_peered":"2024-09-18T07:42:27.177928+0000","last_clean":"2024-09-18T07:42:27.177928+0000","last_became_active":"2024-09-18T07:37:43.076595+0000","last_became_peered":"2024-09-18T07:37:43.076595+0000","last_unstale":"2024-09-18T07:42:27.177928+0000","last_undegraded":"2024-09-18T07:42:27.177928+0000","last_fullsized":"2024-09-18T07:42:27.177928+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:42:00.366441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:42:00.366441+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:21.576301+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":595,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.171454+0000","last_change":"2024-09-18T07:39:25.741714+0000","last_active":"2024-09-18T07:42:27.171454+0000","last_peered":"2024-09-18T07:42:27.171454+0000","last_clean":"2024-09-18T07:42:27.171454+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:42:27.171454+0000","last_undegraded":"2024-09-18T07:42:27.171454+0000","last_fullsized":"2024-09-18T07:42:27.171454+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:25.741643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.741643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:38.000377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":624,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.179695+0000","last_change":"2024-09-18T07:40:05.858885+0000","last_active":"2024-09-18T07:42:27.179695+0000","last_peered":"2024-09-18T07:42:27.179695+0000","last_clean":"2024-09-18T07:42:27.179695+0000","last_became_active":"2024-09-18T07:35:01.418809+0000","last_became_peered":"2024-09-18T07:35:01.418809+0000","last_unstale":"2024-09-18T07:42:27.179695+0000","last_undegraded":"2024-09-18T07:42:27.179695+0000","last_fullsized":"2024-09-18T07:42:27.179695+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:05.858804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:05.858804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:44.323523+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"352'15037","reported_seq":15256,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:32.033080+0000","last_change":"2024-09-18T07:40:50.786329+0000","last_active":"2024-09-18T07:42:32.033080+0000","last_peered":"2024-09-18T07:42:32.033080+0000","last_clean":"2024-09-18T07:42:32.033080+0000","last_became_active":"2024-09-18T07:39:29.702461+0000","last_became_peered":"2024-09-18T07:39:29.702461+0000","last_unstale":"2024-09-18T07:42:32.033080+0000","last_undegraded":"2024-09-18T07:42:32.033080+0000","last_fullsized":"2024-09-18T07:42:32.033080+0000","mapping_epoch":289,"log_start":"333'10300","ondisk_log_start":"333'10300","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7357","last_scrub_stamp":"2024-09-18T07:40:50.786248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.786248+0000","objects_scrubbed":7280,"log_size":4737,"log_dups_size":0,"ondisk_log_size":4737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100597760,"num_objects":1535,"num_object_clones":0,"num_object_copies":4605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15037,"num_write_kb":530304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":451,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.181965+0000","last_change":"2024-09-18T07:40:44.476464+0000","last_active":"2024-09-18T07:42:27.181965+0000","last_peered":"2024-09-18T07:42:27.181965+0000","last_clean":"2024-09-18T07:42:27.181965+0000","last_became_active":"2024-09-18T07:39:04.944921+0000","last_became_peered":"2024-09-18T07:39:04.944921+0000","last_unstale":"2024-09-18T07:42:27.181965+0000","last_undegraded":"2024-09-18T07:42:27.181965+0000","last_fullsized":"2024-09-18T07:42:27.181965+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:44.476278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:44.476278+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:20.701318+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"323'17","reported_seq":366,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.178764+0000","last_change":"2024-09-18T07:41:54.340669+0000","last_active":"2024-09-18T07:42:27.178764+0000","last_peered":"2024-09-18T07:42:27.178764+0000","last_clean":"2024-09-18T07:42:27.178764+0000","last_became_active":"2024-09-18T07:36:13.117924+0000","last_became_peered":"2024-09-18T07:36:13.117924+0000","last_unstale":"2024-09-18T07:42:27.178764+0000","last_undegraded":"2024-09-18T07:42:27.178764+0000","last_fullsized":"2024-09-18T07:42:27.178764+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'17","last_scrub_stamp":"2024-09-18T07:41:54.340587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.340587+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:05.584840+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":336,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.175142+0000","last_change":"2024-09-18T07:42:02.127909+0000","last_active":"2024-09-18T07:42:27.175142+0000","last_peered":"2024-09-18T07:42:27.175142+0000","last_clean":"2024-09-18T07:42:27.175142+0000","last_became_active":"2024-09-18T07:37:43.050971+0000","last_became_peered":"2024-09-18T07:37:43.050971+0000","last_unstale":"2024-09-18T07:42:27.175142+0000","last_undegraded":"2024-09-18T07:42:27.175142+0000","last_fullsized":"2024-09-18T07:42:27.175142+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:02.127517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:02.127517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:03.074068+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":632,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.173373+0000","last_change":"2024-09-18T07:41:26.957957+0000","last_active":"2024-09-18T07:42:27.173373+0000","last_peered":"2024-09-18T07:42:27.173373+0000","last_clean":"2024-09-18T07:42:27.173373+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:42:27.173373+0000","last_undegraded":"2024-09-18T07:42:27.173373+0000","last_fullsized":"2024-09-18T07:42:27.173373+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:41:26.957881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:41:26.957881+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:55.274602+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":629,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174043+0000","last_change":"2024-09-18T07:42:06.212915+0000","last_active":"2024-09-18T07:42:27.174043+0000","last_peered":"2024-09-18T07:42:27.174043+0000","last_clean":"2024-09-18T07:42:27.174043+0000","last_became_active":"2024-09-18T07:35:01.403680+0000","last_became_peered":"2024-09-18T07:35:01.403680+0000","last_unstale":"2024-09-18T07:42:27.174043+0000","last_undegraded":"2024-09-18T07:42:27.174043+0000","last_fullsized":"2024-09-18T07:42:27.174043+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.212683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.212683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:33.302456+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":405,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.184558+0000","last_change":"2024-09-18T07:40:54.574254+0000","last_active":"2024-09-18T07:42:27.184558+0000","last_peered":"2024-09-18T07:42:27.184558+0000","last_clean":"2024-09-18T07:42:27.184558+0000","last_became_active":"2024-09-18T07:37:43.060019+0000","last_became_peered":"2024-09-18T07:37:43.060019+0000","last_unstale":"2024-09-18T07:42:27.184558+0000","last_undegraded":"2024-09-18T07:42:27.184558+0000","last_fullsized":"2024-09-18T07:42:27.184558+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:40:54.574077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:54.574077+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"352'14726","reported_seq":14943,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.611908+0000","last_change":"2024-09-18T07:40:46.264109+0000","last_active":"2024-09-18T07:42:27.611908+0000","last_peered":"2024-09-18T07:42:27.611908+0000","last_clean":"2024-09-18T07:42:27.611908+0000","last_became_active":"2024-09-18T07:39:29.701554+0000","last_became_peered":"2024-09-18T07:39:29.701554+0000","last_unstale":"2024-09-18T07:42:27.611908+0000","last_undegraded":"2024-09-18T07:42:27.611908+0000","last_fullsized":"2024-09-18T07:42:27.611908+0000","mapping_epoch":289,"log_start":"327'9200","ondisk_log_start":"327'9200","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'7054","last_scrub_stamp":"2024-09-18T07:40:46.263923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:46.263923+0000","objects_scrubbed":6963,"log_size":5526,"log_dups_size":0,"ondisk_log_size":5526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:14.068650+0000","scrub_duration":2499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124256256,"num_objects":1896,"num_object_clones":0,"num_object_copies":5688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14726,"num_write_kb":531904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":343,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.183677+0000","last_change":"2024-09-18T07:39:05.333676+0000","last_active":"2024-09-18T07:42:27.183677+0000","last_peered":"2024-09-18T07:42:27.183677+0000","last_clean":"2024-09-18T07:42:27.183677+0000","last_became_active":"2024-09-18T07:39:04.938080+0000","last_became_peered":"2024-09-18T07:39:04.938080+0000","last_unstale":"2024-09-18T07:42:27.183677+0000","last_undegraded":"2024-09-18T07:42:27.183677+0000","last_fullsized":"2024-09-18T07:42:27.183677+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.333608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.333608+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:32.330218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"323'24","reported_seq":656,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.171904+0000","last_change":"2024-09-18T07:41:27.030793+0000","last_active":"2024-09-18T07:42:27.171904+0000","last_peered":"2024-09-18T07:42:27.171904+0000","last_clean":"2024-09-18T07:42:27.171904+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:42:27.171904+0000","last_undegraded":"2024-09-18T07:42:27.171904+0000","last_fullsized":"2024-09-18T07:42:27.171904+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'24","last_scrub_stamp":"2024-09-18T07:41:27.030705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.030705+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:29.631547+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":609,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.171565+0000","last_change":"2024-09-18T07:40:19.694501+0000","last_active":"2024-09-18T07:42:27.171565+0000","last_peered":"2024-09-18T07:42:27.171565+0000","last_clean":"2024-09-18T07:42:27.171565+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:42:27.171565+0000","last_undegraded":"2024-09-18T07:42:27.171565+0000","last_fullsized":"2024-09-18T07:42:27.171565+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:40:19.694421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:19.694421+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:36.061992+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":833,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:30.201007+0000","last_change":"2024-09-18T07:40:47.554230+0000","last_active":"2024-09-18T07:42:30.201007+0000","last_peered":"2024-09-18T07:42:30.201007+0000","last_clean":"2024-09-18T07:42:30.201007+0000","last_became_active":"2024-09-18T07:35:01.400524+0000","last_became_peered":"2024-09-18T07:35:01.400524+0000","last_unstale":"2024-09-18T07:42:30.201007+0000","last_undegraded":"2024-09-18T07:42:30.201007+0000","last_fullsized":"2024-09-18T07:42:30.201007+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T07:40:47.554161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.554161+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":456,"num_read_kb":451,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":621,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.178791+0000","last_change":"2024-09-18T07:42:02.399342+0000","last_active":"2024-09-18T07:42:27.178791+0000","last_peered":"2024-09-18T07:42:27.178791+0000","last_clean":"2024-09-18T07:42:27.178791+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:42:27.178791+0000","last_undegraded":"2024-09-18T07:42:27.178791+0000","last_fullsized":"2024-09-18T07:42:27.178791+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:02.399269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:02.399269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:08.509887+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":314,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.177475+0000","last_change":"2024-09-18T07:40:12.269264+0000","last_active":"2024-09-18T07:42:27.177475+0000","last_peered":"2024-09-18T07:42:27.177475+0000","last_clean":"2024-09-18T07:42:27.177475+0000","last_became_active":"2024-09-18T07:36:13.117463+0000","last_became_peered":"2024-09-18T07:36:13.117463+0000","last_unstale":"2024-09-18T07:42:27.177475+0000","last_undegraded":"2024-09-18T07:42:27.177475+0000","last_fullsized":"2024-09-18T07:42:27.177475+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:12.269201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:12.269201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:20.549505+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"352'14389","reported_seq":14604,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.376461+0000","last_change":"2024-09-18T07:41:22.252954+0000","last_active":"2024-09-18T07:42:27.376461+0000","last_peered":"2024-09-18T07:42:27.376461+0000","last_clean":"2024-09-18T07:42:27.376461+0000","last_became_active":"2024-09-18T07:39:29.699733+0000","last_became_peered":"2024-09-18T07:39:29.699733+0000","last_unstale":"2024-09-18T07:42:27.376461+0000","last_undegraded":"2024-09-18T07:42:27.376461+0000","last_fullsized":"2024-09-18T07:42:27.376461+0000","mapping_epoch":289,"log_start":"331'9800","ondisk_log_start":"331'9800","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'9472","last_scrub_stamp":"2024-09-18T07:41:22.252822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:41:22.252822+0000","objects_scrubbed":6806,"log_size":4589,"log_dups_size":0,"ondisk_log_size":4589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:26.433396+0000","scrub_duration":1819,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120520704,"num_objects":1839,"num_object_clones":0,"num_object_copies":5517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14389,"num_write_kb":519296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"323'32","reported_seq":340,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.184519+0000","last_change":"2024-09-18T07:40:17.950392+0000","last_active":"2024-09-18T07:42:27.184519+0000","last_peered":"2024-09-18T07:42:27.184519+0000","last_clean":"2024-09-18T07:42:27.184519+0000","last_became_active":"2024-09-18T07:37:43.087496+0000","last_became_peered":"2024-09-18T07:37:43.087496+0000","last_unstale":"2024-09-18T07:42:27.184519+0000","last_undegraded":"2024-09-18T07:42:27.184519+0000","last_fullsized":"2024-09-18T07:42:27.184519+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-18T07:40:17.950298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:17.950298+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:39.343760+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":484,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.177844+0000","last_change":"2024-09-18T07:40:19.067973+0000","last_active":"2024-09-18T07:42:27.177844+0000","last_peered":"2024-09-18T07:42:27.177844+0000","last_clean":"2024-09-18T07:42:27.177844+0000","last_became_active":"2024-09-18T07:40:19.067624+0000","last_became_peered":"2024-09-18T07:40:19.067624+0000","last_unstale":"2024-09-18T07:42:27.177844+0000","last_undegraded":"2024-09-18T07:42:27.177844+0000","last_fullsized":"2024-09-18T07:42:27.177844+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:31:59.616988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:31:59.616988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:34.408847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":445,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.181723+0000","last_change":"2024-09-18T07:40:42.452815+0000","last_active":"2024-09-18T07:42:27.181723+0000","last_peered":"2024-09-18T07:42:27.181723+0000","last_clean":"2024-09-18T07:42:27.181723+0000","last_became_active":"2024-09-18T07:39:03.350419+0000","last_became_peered":"2024-09-18T07:39:03.350419+0000","last_unstale":"2024-09-18T07:42:27.181723+0000","last_undegraded":"2024-09-18T07:42:27.181723+0000","last_fullsized":"2024-09-18T07:42:27.181723+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:42.452744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:42.452744+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:16.814737+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":395,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.187908+0000","last_change":"2024-09-18T07:42:06.726510+0000","last_active":"2024-09-18T07:42:27.187908+0000","last_peered":"2024-09-18T07:42:27.187908+0000","last_clean":"2024-09-18T07:42:27.187908+0000","last_became_active":"2024-09-18T07:35:01.412217+0000","last_became_peered":"2024-09-18T07:35:01.412217+0000","last_unstale":"2024-09-18T07:42:27.187908+0000","last_undegraded":"2024-09-18T07:42:27.187908+0000","last_fullsized":"2024-09-18T07:42:27.187908+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.726445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.726445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:15.201416+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":593,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.178703+0000","last_change":"2024-09-18T07:33:22.358792+0000","last_active":"2024-09-18T07:42:27.178703+0000","last_peered":"2024-09-18T07:42:27.178703+0000","last_clean":"2024-09-18T07:42:27.178703+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:42:27.178703+0000","last_undegraded":"2024-09-18T07:42:27.178703+0000","last_fullsized":"2024-09-18T07:42:27.178703+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:33:22.358746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:33:22.358746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:35.384551+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"352'14901","reported_seq":15124,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:31.664657+0000","last_change":"2024-09-18T07:40:58.098898+0000","last_active":"2024-09-18T07:42:31.664657+0000","last_peered":"2024-09-18T07:42:31.664657+0000","last_clean":"2024-09-18T07:42:31.664657+0000","last_became_active":"2024-09-18T07:39:29.703451+0000","last_became_peered":"2024-09-18T07:39:29.703451+0000","last_unstale":"2024-09-18T07:42:31.664657+0000","last_undegraded":"2024-09-18T07:42:31.664657+0000","last_fullsized":"2024-09-18T07:42:31.664657+0000","mapping_epoch":289,"log_start":"334'10400","ondisk_log_start":"334'10400","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7779","last_scrub_stamp":"2024-09-18T07:40:58.098697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:58.098697+0000","objects_scrubbed":7710,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104923136,"num_objects":1601,"num_object_clones":0,"num_object_copies":4803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14901,"num_write_kb":528064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"323'19","reported_seq":726,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.175214+0000","last_change":"2024-09-18T07:39:07.038527+0000","last_active":"2024-09-18T07:42:27.175214+0000","last_peered":"2024-09-18T07:42:27.175214+0000","last_clean":"2024-09-18T07:42:27.175214+0000","last_became_active":"2024-09-18T07:37:43.063255+0000","last_became_peered":"2024-09-18T07:37:43.063255+0000","last_unstale":"2024-09-18T07:42:27.175214+0000","last_undegraded":"2024-09-18T07:42:27.175214+0000","last_fullsized":"2024-09-18T07:42:27.175214+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:39:07.038450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:07.038450+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:34.882660+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":637,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174659+0000","last_change":"2024-09-18T07:39:29.078656+0000","last_active":"2024-09-18T07:42:27.174659+0000","last_peered":"2024-09-18T07:42:27.174659+0000","last_clean":"2024-09-18T07:42:27.174659+0000","last_became_active":"2024-09-18T07:35:01.401063+0000","last_became_peered":"2024-09-18T07:35:01.401063+0000","last_unstale":"2024-09-18T07:42:27.174659+0000","last_undegraded":"2024-09-18T07:42:27.174659+0000","last_fullsized":"2024-09-18T07:42:27.174659+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.078600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.078600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:40.967780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":627,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.178539+0000","last_change":"2024-09-18T07:39:35.406107+0000","last_active":"2024-09-18T07:42:27.178539+0000","last_peered":"2024-09-18T07:42:27.178539+0000","last_clean":"2024-09-18T07:42:27.178539+0000","last_became_active":"2024-09-18T07:37:43.075535+0000","last_became_peered":"2024-09-18T07:37:43.075535+0000","last_unstale":"2024-09-18T07:42:27.178539+0000","last_undegraded":"2024-09-18T07:42:27.178539+0000","last_fullsized":"2024-09-18T07:42:27.178539+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.406041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.406041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:15.529066+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":486,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.184069+0000","last_change":"2024-09-18T07:41:20.220326+0000","last_active":"2024-09-18T07:42:27.184069+0000","last_peered":"2024-09-18T07:42:27.184069+0000","last_clean":"2024-09-18T07:42:27.184069+0000","last_became_active":"2024-09-18T07:40:19.068998+0000","last_became_peered":"2024-09-18T07:40:19.068998+0000","last_unstale":"2024-09-18T07:42:27.184069+0000","last_undegraded":"2024-09-18T07:42:27.184069+0000","last_fullsized":"2024-09-18T07:42:27.184069+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:20.220262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:20.220262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:23.019047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"352'14965","reported_seq":15182,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:31.694039+0000","last_change":"2024-09-18T07:40:48.247625+0000","last_active":"2024-09-18T07:42:31.694039+0000","last_peered":"2024-09-18T07:42:31.694039+0000","last_clean":"2024-09-18T07:42:31.694039+0000","last_became_active":"2024-09-18T07:39:29.699800+0000","last_became_peered":"2024-09-18T07:39:29.699800+0000","last_unstale":"2024-09-18T07:42:31.694039+0000","last_undegraded":"2024-09-18T07:42:31.694039+0000","last_fullsized":"2024-09-18T07:42:31.694039+0000","mapping_epoch":289,"log_start":"326'8900","ondisk_log_start":"326'8900","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'7166","last_scrub_stamp":"2024-09-18T07:40:48.247448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:48.247448+0000","objects_scrubbed":7089,"log_size":6065,"log_dups_size":0,"ondisk_log_size":6065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:16.070539+0000","scrub_duration":2089,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106233856,"num_objects":1621,"num_object_clones":0,"num_object_copies":4863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14965,"num_write_kb":530752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":185,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.176331+0000","last_change":"2024-09-18T07:39:38.447787+0000","last_active":"2024-09-18T07:42:27.176331+0000","last_peered":"2024-09-18T07:42:27.176331+0000","last_clean":"2024-09-18T07:42:27.176331+0000","last_became_active":"2024-09-18T07:39:04.950292+0000","last_became_peered":"2024-09-18T07:39:04.950292+0000","last_unstale":"2024-09-18T07:42:27.176331+0000","last_undegraded":"2024-09-18T07:42:27.176331+0000","last_fullsized":"2024-09-18T07:42:27.176331+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:38.447735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:38.447735+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:46.260853+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"323'31","reported_seq":676,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174884+0000","last_change":"2024-09-18T07:40:04.281816+0000","last_active":"2024-09-18T07:42:27.174884+0000","last_peered":"2024-09-18T07:42:27.174884+0000","last_clean":"2024-09-18T07:42:27.174884+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:42:27.174884+0000","last_undegraded":"2024-09-18T07:42:27.174884+0000","last_fullsized":"2024-09-18T07:42:27.174884+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:40:04.281758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:04.281758+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:46.955052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":262,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.183363+0000","last_change":"2024-09-18T07:41:54.294370+0000","last_active":"2024-09-18T07:42:27.183363+0000","last_peered":"2024-09-18T07:42:27.183363+0000","last_clean":"2024-09-18T07:42:27.183363+0000","last_became_active":"2024-09-18T07:37:43.061808+0000","last_became_peered":"2024-09-18T07:37:43.061808+0000","last_unstale":"2024-09-18T07:42:27.183363+0000","last_undegraded":"2024-09-18T07:42:27.183363+0000","last_fullsized":"2024-09-18T07:42:27.183363+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:54.294307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.294307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:57.341592+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":305,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.173650+0000","last_change":"2024-09-18T07:40:52.167232+0000","last_active":"2024-09-18T07:42:27.173650+0000","last_peered":"2024-09-18T07:42:27.173650+0000","last_clean":"2024-09-18T07:42:27.173650+0000","last_became_active":"2024-09-18T07:36:13.116527+0000","last_became_peered":"2024-09-18T07:36:13.116527+0000","last_unstale":"2024-09-18T07:42:27.173650+0000","last_undegraded":"2024-09-18T07:42:27.173650+0000","last_fullsized":"2024-09-18T07:42:27.173650+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:52.167048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:52.167048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:17.172256+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":627,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174328+0000","last_change":"2024-09-18T07:41:56.172872+0000","last_active":"2024-09-18T07:42:27.174328+0000","last_peered":"2024-09-18T07:42:27.174328+0000","last_clean":"2024-09-18T07:42:27.174328+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:42:27.174328+0000","last_undegraded":"2024-09-18T07:42:27.174328+0000","last_fullsized":"2024-09-18T07:42:27.174328+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:56.172797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:56.172797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:05.871802+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"352'14929","reported_seq":15147,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:31.745566+0000","last_change":"2024-09-18T07:40:41.001448+0000","last_active":"2024-09-18T07:42:31.745566+0000","last_peered":"2024-09-18T07:42:31.745566+0000","last_clean":"2024-09-18T07:42:31.745566+0000","last_became_active":"2024-09-18T07:40:19.066715+0000","last_became_peered":"2024-09-18T07:40:19.066715+0000","last_unstale":"2024-09-18T07:42:31.745566+0000","last_undegraded":"2024-09-18T07:42:31.745566+0000","last_fullsized":"2024-09-18T07:42:31.745566+0000","mapping_epoch":307,"log_start":"326'8900","ondisk_log_start":"326'8900","created":289,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'6605","last_scrub_stamp":"2024-09-18T07:40:41.001367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:41.001367+0000","objects_scrubbed":6534,"log_size":6029,"log_dups_size":0,"ondisk_log_size":6029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:46.302322+0000","scrub_duration":1931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104792064,"num_objects":1599,"num_object_clones":0,"num_object_copies":4797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14929,"num_write_kb":528896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":443,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.186807+0000","last_change":"2024-09-18T07:39:05.498235+0000","last_active":"2024-09-18T07:42:27.186807+0000","last_peered":"2024-09-18T07:42:27.186807+0000","last_clean":"2024-09-18T07:42:27.186807+0000","last_became_active":"2024-09-18T07:39:04.944753+0000","last_became_peered":"2024-09-18T07:39:04.944753+0000","last_unstale":"2024-09-18T07:42:27.186807+0000","last_undegraded":"2024-09-18T07:42:27.186807+0000","last_fullsized":"2024-09-18T07:42:27.186807+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:05.498159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:05.498159+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:30.837196+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"323'20","reported_seq":679,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174546+0000","last_change":"2024-09-18T07:40:50.210059+0000","last_active":"2024-09-18T07:42:27.174546+0000","last_peered":"2024-09-18T07:42:27.174546+0000","last_clean":"2024-09-18T07:42:27.174546+0000","last_became_active":"2024-09-18T07:37:43.062724+0000","last_became_peered":"2024-09-18T07:37:43.062724+0000","last_unstale":"2024-09-18T07:42:27.174546+0000","last_undegraded":"2024-09-18T07:42:27.174546+0000","last_fullsized":"2024-09-18T07:42:27.174546+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:40:50.209824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.209824+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":110,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.170645+0000","last_change":"2024-09-18T07:40:47.799512+0000","last_active":"2024-09-18T07:42:27.170645+0000","last_peered":"2024-09-18T07:42:27.170645+0000","last_clean":"2024-09-18T07:42:27.170645+0000","last_became_active":"2024-09-18T07:40:19.065652+0000","last_became_peered":"2024-09-18T07:40:19.065652+0000","last_unstale":"2024-09-18T07:42:27.170645+0000","last_undegraded":"2024-09-18T07:42:27.170645+0000","last_fullsized":"2024-09-18T07:42:27.170645+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:47.799342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.799342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:24.143277+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"173'11","reported_seq":788,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174140+0000","last_change":"2024-09-18T07:38:55.227421+0000","last_active":"2024-09-18T07:42:27.174140+0000","last_peered":"2024-09-18T07:42:27.174140+0000","last_clean":"2024-09-18T07:42:27.174140+0000","last_became_active":"2024-09-18T07:35:01.403765+0000","last_became_peered":"2024-09-18T07:35:01.403765+0000","last_unstale":"2024-09-18T07:42:27.174140+0000","last_undegraded":"2024-09-18T07:42:27.174140+0000","last_fullsized":"2024-09-18T07:42:27.174140+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'11","last_scrub_stamp":"2024-09-18T07:38:55.227366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:55.227366+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:33.902149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":603,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174931+0000","last_change":"2024-09-18T07:36:33.251164+0000","last_active":"2024-09-18T07:42:27.174931+0000","last_peered":"2024-09-18T07:42:27.174931+0000","last_clean":"2024-09-18T07:42:27.174931+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:42:27.174931+0000","last_undegraded":"2024-09-18T07:42:27.174931+0000","last_fullsized":"2024-09-18T07:42:27.174931+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:33.251093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:33.251093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:12.044091+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"352'14948","reported_seq":15172,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:32.081729+0000","last_change":"2024-09-18T07:40:56.968567+0000","last_active":"2024-09-18T07:42:32.081729+0000","last_peered":"2024-09-18T07:42:32.081729+0000","last_clean":"2024-09-18T07:42:32.081729+0000","last_became_active":"2024-09-18T07:39:29.704318+0000","last_became_peered":"2024-09-18T07:39:29.704318+0000","last_unstale":"2024-09-18T07:42:32.081729+0000","last_undegraded":"2024-09-18T07:42:32.081729+0000","last_fullsized":"2024-09-18T07:42:32.081729+0000","mapping_epoch":289,"log_start":"333'10200","ondisk_log_start":"333'10200","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7695","last_scrub_stamp":"2024-09-18T07:40:56.968381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.968381+0000","objects_scrubbed":7617,"log_size":4748,"log_dups_size":0,"ondisk_log_size":4748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2651,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102629376,"num_objects":1566,"num_object_clones":0,"num_object_copies":4698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14948,"num_write_kb":528448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"323'15","reported_seq":657,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.170417+0000","last_change":"2024-09-18T07:41:21.946159+0000","last_active":"2024-09-18T07:42:27.170417+0000","last_peered":"2024-09-18T07:42:27.170417+0000","last_clean":"2024-09-18T07:42:27.170417+0000","last_became_active":"2024-09-18T07:37:27.994144+0000","last_became_peered":"2024-09-18T07:37:27.994144+0000","last_unstale":"2024-09-18T07:42:27.170417+0000","last_undegraded":"2024-09-18T07:42:27.170417+0000","last_fullsized":"2024-09-18T07:42:27.170417+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:21.946066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:21.946066+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:30.620162+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":635,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.177514+0000","last_change":"2024-09-18T07:40:45.343597+0000","last_active":"2024-09-18T07:42:27.177514+0000","last_peered":"2024-09-18T07:42:27.177514+0000","last_clean":"2024-09-18T07:42:27.177514+0000","last_became_active":"2024-09-18T07:37:43.075988+0000","last_became_peered":"2024-09-18T07:37:43.075988+0000","last_unstale":"2024-09-18T07:42:27.177514+0000","last_undegraded":"2024-09-18T07:42:27.177514+0000","last_fullsized":"2024-09-18T07:42:27.177514+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:45.343533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:45.343533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":607,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.173650+0000","last_change":"2024-09-18T07:39:01.216439+0000","last_active":"2024-09-18T07:42:27.173650+0000","last_peered":"2024-09-18T07:42:27.173650+0000","last_clean":"2024-09-18T07:42:27.173650+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:42:27.173650+0000","last_undegraded":"2024-09-18T07:42:27.173650+0000","last_fullsized":"2024-09-18T07:42:27.173650+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:01.216305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:01.216305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:34.950724+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":629,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.171538+0000","last_change":"2024-09-18T07:41:59.006637+0000","last_active":"2024-09-18T07:42:27.171538+0000","last_peered":"2024-09-18T07:42:27.171538+0000","last_clean":"2024-09-18T07:42:27.171538+0000","last_became_active":"2024-09-18T07:35:01.410890+0000","last_became_peered":"2024-09-18T07:35:01.410890+0000","last_unstale":"2024-09-18T07:42:27.171538+0000","last_undegraded":"2024-09-18T07:42:27.171538+0000","last_fullsized":"2024-09-18T07:42:27.171538+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:59.006386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:59.006386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:18.266924+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"352'15036","reported_seq":15251,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:32.086192+0000","last_change":"2024-09-18T07:41:23.184909+0000","last_active":"2024-09-18T07:42:32.086192+0000","last_peered":"2024-09-18T07:42:32.086192+0000","last_clean":"2024-09-18T07:42:32.086192+0000","last_became_active":"2024-09-18T07:39:29.701593+0000","last_became_peered":"2024-09-18T07:39:29.701593+0000","last_unstale":"2024-09-18T07:42:32.086192+0000","last_undegraded":"2024-09-18T07:42:32.086192+0000","last_fullsized":"2024-09-18T07:42:32.086192+0000","mapping_epoch":289,"log_start":"333'10300","ondisk_log_start":"333'10300","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'9778","last_scrub_stamp":"2024-09-18T07:41:23.184760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:41:23.184760+0000","objects_scrubbed":7006,"log_size":4736,"log_dups_size":0,"ondisk_log_size":4736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2841,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108265472,"num_objects":1652,"num_object_clones":0,"num_object_copies":4956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15036,"num_write_kb":534016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"323'35","reported_seq":290,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.183506+0000","last_change":"2024-09-18T07:39:25.932376+0000","last_active":"2024-09-18T07:42:27.183506+0000","last_peered":"2024-09-18T07:42:27.183506+0000","last_clean":"2024-09-18T07:42:27.183506+0000","last_became_active":"2024-09-18T07:37:43.063319+0000","last_became_peered":"2024-09-18T07:37:43.063319+0000","last_unstale":"2024-09-18T07:42:27.183506+0000","last_undegraded":"2024-09-18T07:42:27.183506+0000","last_fullsized":"2024-09-18T07:42:27.183506+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:39:25.932315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.932315+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:36.321873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":409,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.184936+0000","last_change":"2024-09-18T07:40:56.546321+0000","last_active":"2024-09-18T07:42:27.184936+0000","last_peered":"2024-09-18T07:42:27.184936+0000","last_clean":"2024-09-18T07:42:27.184936+0000","last_became_active":"2024-09-18T07:37:43.072894+0000","last_became_peered":"2024-09-18T07:37:43.072894+0000","last_unstale":"2024-09-18T07:42:27.184936+0000","last_undegraded":"2024-09-18T07:42:27.184936+0000","last_fullsized":"2024-09-18T07:42:27.184936+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:56.546251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.546251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":630,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.173816+0000","last_change":"2024-09-18T07:42:04.179746+0000","last_active":"2024-09-18T07:42:27.173816+0000","last_peered":"2024-09-18T07:42:27.173816+0000","last_clean":"2024-09-18T07:42:27.173816+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:42:27.173816+0000","last_undegraded":"2024-09-18T07:42:27.173816+0000","last_fullsized":"2024-09-18T07:42:27.173816+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:04.179670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.179670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:04.388489+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":387,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.181732+0000","last_change":"2024-09-18T07:42:09.794539+0000","last_active":"2024-09-18T07:42:27.181732+0000","last_peered":"2024-09-18T07:42:27.181732+0000","last_clean":"2024-09-18T07:42:27.181732+0000","last_became_active":"2024-09-18T07:35:01.406980+0000","last_became_peered":"2024-09-18T07:35:01.406980+0000","last_unstale":"2024-09-18T07:42:27.181732+0000","last_undegraded":"2024-09-18T07:42:27.181732+0000","last_fullsized":"2024-09-18T07:42:27.181732+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:09.794459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:09.794459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:32.016537+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"352'14995","reported_seq":15206,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:31.644665+0000","last_change":"2024-09-18T07:40:43.229079+0000","last_active":"2024-09-18T07:42:31.644665+0000","last_peered":"2024-09-18T07:42:31.644665+0000","last_clean":"2024-09-18T07:42:31.644665+0000","last_became_active":"2024-09-18T07:39:29.700244+0000","last_became_peered":"2024-09-18T07:39:29.700244+0000","last_unstale":"2024-09-18T07:42:31.644665+0000","last_undegraded":"2024-09-18T07:42:31.644665+0000","last_fullsized":"2024-09-18T07:42:31.644665+0000","mapping_epoch":289,"log_start":"333'10300","ondisk_log_start":"333'10300","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'6831","last_scrub_stamp":"2024-09-18T07:40:43.228883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:43.228883+0000","objects_scrubbed":6759,"log_size":4695,"log_dups_size":0,"ondisk_log_size":4695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2043,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103219200,"num_objects":1575,"num_object_clones":0,"num_object_copies":4725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14995,"num_write_kb":530240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"323'18","reported_seq":667,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174468+0000","last_change":"2024-09-18T07:41:47.950163+0000","last_active":"2024-09-18T07:42:27.174468+0000","last_peered":"2024-09-18T07:42:27.174468+0000","last_clean":"2024-09-18T07:42:27.174468+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:42:27.174468+0000","last_undegraded":"2024-09-18T07:42:27.174468+0000","last_fullsized":"2024-09-18T07:42:27.174468+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:41:47.950068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:47.950068+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:01.623082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":387,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.183871+0000","last_change":"2024-09-18T07:42:07.729518+0000","last_active":"2024-09-18T07:42:27.183871+0000","last_peered":"2024-09-18T07:42:27.183871+0000","last_clean":"2024-09-18T07:42:27.183871+0000","last_became_active":"2024-09-18T07:35:01.398499+0000","last_became_peered":"2024-09-18T07:35:01.398499+0000","last_unstale":"2024-09-18T07:42:27.183871+0000","last_undegraded":"2024-09-18T07:42:27.183871+0000","last_fullsized":"2024-09-18T07:42:27.183871+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:07.729438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:07.729438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:33.911294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":639,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.175062+0000","last_change":"2024-09-18T07:41:58.202324+0000","last_active":"2024-09-18T07:42:27.175062+0000","last_peered":"2024-09-18T07:42:27.175062+0000","last_clean":"2024-09-18T07:42:27.175062+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:42:27.175062+0000","last_undegraded":"2024-09-18T07:42:27.175062+0000","last_fullsized":"2024-09-18T07:42:27.175062+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:58.202229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:58.202229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:08.344962+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":616,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.179868+0000","last_change":"2024-09-18T07:40:02.932758+0000","last_active":"2024-09-18T07:42:27.179868+0000","last_peered":"2024-09-18T07:42:27.179868+0000","last_clean":"2024-09-18T07:42:27.179868+0000","last_became_active":"2024-09-18T07:35:01.418735+0000","last_became_peered":"2024-09-18T07:35:01.418735+0000","last_unstale":"2024-09-18T07:42:27.179868+0000","last_undegraded":"2024-09-18T07:42:27.179868+0000","last_fullsized":"2024-09-18T07:42:27.179868+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:02.932690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:02.932690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:42.406919+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"352'14563","reported_seq":14784,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.565655+0000","last_change":"2024-09-18T07:40:54.295785+0000","last_active":"2024-09-18T07:42:27.565655+0000","last_peered":"2024-09-18T07:42:27.565655+0000","last_clean":"2024-09-18T07:42:27.565655+0000","last_became_active":"2024-09-18T07:39:29.701617+0000","last_became_peered":"2024-09-18T07:39:29.701617+0000","last_unstale":"2024-09-18T07:42:27.565655+0000","last_undegraded":"2024-09-18T07:42:27.565655+0000","last_fullsized":"2024-09-18T07:42:27.565655+0000","mapping_epoch":289,"log_start":"327'9100","ondisk_log_start":"327'9100","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7478","last_scrub_stamp":"2024-09-18T07:40:54.295602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:54.295602+0000","objects_scrubbed":7411,"log_size":5463,"log_dups_size":0,"ondisk_log_size":5463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:26.152256+0000","scrub_duration":2507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122748928,"num_objects":1873,"num_object_clones":0,"num_object_copies":5619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14563,"num_write_kb":525952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"323'27","reported_seq":760,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.176463+0000","last_change":"2024-09-18T07:41:51.332873+0000","last_active":"2024-09-18T07:42:27.176463+0000","last_peered":"2024-09-18T07:42:27.176463+0000","last_clean":"2024-09-18T07:42:27.176463+0000","last_became_active":"2024-09-18T07:37:43.075552+0000","last_became_peered":"2024-09-18T07:37:43.075552+0000","last_unstale":"2024-09-18T07:42:27.176463+0000","last_undegraded":"2024-09-18T07:42:27.176463+0000","last_fullsized":"2024-09-18T07:42:27.176463+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:41:51.332791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:51.332791+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:13.912522+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":401,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.183406+0000","last_change":"2024-09-18T07:41:27.819345+0000","last_active":"2024-09-18T07:42:27.183406+0000","last_peered":"2024-09-18T07:42:27.183406+0000","last_clean":"2024-09-18T07:42:27.183406+0000","last_became_active":"2024-09-18T07:35:01.406695+0000","last_became_peered":"2024-09-18T07:35:01.406695+0000","last_unstale":"2024-09-18T07:42:27.183406+0000","last_undegraded":"2024-09-18T07:42:27.183406+0000","last_fullsized":"2024-09-18T07:42:27.183406+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:41:27.819269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.819269+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:32.041258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"240'12","reported_seq":504,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.187332+0000","last_change":"2024-09-18T07:38:55.490250+0000","last_active":"2024-09-18T07:42:27.187332+0000","last_peered":"2024-09-18T07:42:27.187332+0000","last_clean":"2024-09-18T07:42:27.187332+0000","last_became_active":"2024-09-18T07:37:43.056714+0000","last_became_peered":"2024-09-18T07:37:43.056714+0000","last_unstale":"2024-09-18T07:42:27.187332+0000","last_undegraded":"2024-09-18T07:42:27.187332+0000","last_fullsized":"2024-09-18T07:42:27.187332+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:38:55.490178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:55.490178+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:33.717958+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":615,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.177557+0000","last_change":"2024-09-18T07:42:07.547399+0000","last_active":"2024-09-18T07:42:27.177557+0000","last_peered":"2024-09-18T07:42:27.177557+0000","last_clean":"2024-09-18T07:42:27.177557+0000","last_became_active":"2024-09-18T07:37:43.078121+0000","last_became_peered":"2024-09-18T07:37:43.078121+0000","last_unstale":"2024-09-18T07:42:27.177557+0000","last_undegraded":"2024-09-18T07:42:27.177557+0000","last_fullsized":"2024-09-18T07:42:27.177557+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:07.547352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:07.547352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:10.534419+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":254,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.183691+0000","last_change":"2024-09-18T07:42:08.181612+0000","last_active":"2024-09-18T07:42:27.183691+0000","last_peered":"2024-09-18T07:42:27.183691+0000","last_clean":"2024-09-18T07:42:27.183691+0000","last_became_active":"2024-09-18T07:37:43.087994+0000","last_became_peered":"2024-09-18T07:37:43.087994+0000","last_unstale":"2024-09-18T07:42:27.183691+0000","last_undegraded":"2024-09-18T07:42:27.183691+0000","last_fullsized":"2024-09-18T07:42:27.183691+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:08.181420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:08.181420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:21.009353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"323'12","reported_seq":665,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.180452+0000","last_change":"2024-09-18T07:41:48.359809+0000","last_active":"2024-09-18T07:42:27.180452+0000","last_peered":"2024-09-18T07:42:27.180452+0000","last_clean":"2024-09-18T07:42:27.180452+0000","last_became_active":"2024-09-18T07:35:01.407787+0000","last_became_peered":"2024-09-18T07:35:01.407787+0000","last_unstale":"2024-09-18T07:42:27.180452+0000","last_undegraded":"2024-09-18T07:42:27.180452+0000","last_fullsized":"2024-09-18T07:42:27.180452+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'12","last_scrub_stamp":"2024-09-18T07:41:48.359711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.359711+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:03.524359+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":620,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.179968+0000","last_change":"2024-09-18T07:41:52.319134+0000","last_active":"2024-09-18T07:42:27.179968+0000","last_peered":"2024-09-18T07:42:27.179968+0000","last_clean":"2024-09-18T07:42:27.179968+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:42:27.179968+0000","last_undegraded":"2024-09-18T07:42:27.179968+0000","last_fullsized":"2024-09-18T07:42:27.179968+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:52.319073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:52.319073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:54.508152+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":393,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.184805+0000","last_change":"2024-09-18T07:40:51.554184+0000","last_active":"2024-09-18T07:42:27.184805+0000","last_peered":"2024-09-18T07:42:27.184805+0000","last_clean":"2024-09-18T07:42:27.184805+0000","last_became_active":"2024-09-18T07:37:43.079313+0000","last_became_peered":"2024-09-18T07:37:43.079313+0000","last_unstale":"2024-09-18T07:42:27.184805+0000","last_undegraded":"2024-09-18T07:42:27.184805+0000","last_fullsized":"2024-09-18T07:42:27.184805+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:51.554073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:51.554073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:23.781937+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"323'15","reported_seq":649,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.171990+0000","last_change":"2024-09-18T07:35:01.448510+0000","last_active":"2024-09-18T07:42:27.171990+0000","last_peered":"2024-09-18T07:42:27.171990+0000","last_clean":"2024-09-18T07:42:27.171990+0000","last_became_active":"2024-09-18T07:35:01.414458+0000","last_became_peered":"2024-09-18T07:35:01.414458+0000","last_unstale":"2024-09-18T07:42:27.171990+0000","last_undegraded":"2024-09-18T07:42:27.171990+0000","last_fullsized":"2024-09-18T07:42:27.171990+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:30:57.286504+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:30.006114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":409,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.187224+0000","last_change":"2024-09-18T07:41:52.891853+0000","last_active":"2024-09-18T07:42:27.187224+0000","last_peered":"2024-09-18T07:42:27.187224+0000","last_clean":"2024-09-18T07:42:27.187224+0000","last_became_active":"2024-09-18T07:37:43.056675+0000","last_became_peered":"2024-09-18T07:37:43.056675+0000","last_unstale":"2024-09-18T07:42:27.187224+0000","last_undegraded":"2024-09-18T07:42:27.187224+0000","last_fullsized":"2024-09-18T07:42:27.187224+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:52.891811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:52.891811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:55.106484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":635,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.180602+0000","last_change":"2024-09-18T07:41:16.472282+0000","last_active":"2024-09-18T07:42:27.180602+0000","last_peered":"2024-09-18T07:42:27.180602+0000","last_clean":"2024-09-18T07:42:27.180602+0000","last_became_active":"2024-09-18T07:35:01.420133+0000","last_became_peered":"2024-09-18T07:35:01.420133+0000","last_unstale":"2024-09-18T07:42:27.180602+0000","last_undegraded":"2024-09-18T07:42:27.180602+0000","last_fullsized":"2024-09-18T07:42:27.180602+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:16.472210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:16.472210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:38.782593+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":608,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174822+0000","last_change":"2024-09-18T07:37:48.861965+0000","last_active":"2024-09-18T07:42:27.174822+0000","last_peered":"2024-09-18T07:42:27.174822+0000","last_clean":"2024-09-18T07:42:27.174822+0000","last_became_active":"2024-09-18T07:35:01.396305+0000","last_became_peered":"2024-09-18T07:35:01.396305+0000","last_unstale":"2024-09-18T07:42:27.174822+0000","last_undegraded":"2024-09-18T07:42:27.174822+0000","last_fullsized":"2024-09-18T07:42:27.174822+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:37:48.861784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:37:48.861784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:31.915133+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"323'15","reported_seq":675,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.177832+0000","last_change":"2024-09-18T07:40:52.366957+0000","last_active":"2024-09-18T07:42:27.177832+0000","last_peered":"2024-09-18T07:42:27.177832+0000","last_clean":"2024-09-18T07:42:27.177832+0000","last_became_active":"2024-09-18T07:35:01.416744+0000","last_became_peered":"2024-09-18T07:35:01.416744+0000","last_unstale":"2024-09-18T07:42:27.177832+0000","last_undegraded":"2024-09-18T07:42:27.177832+0000","last_fullsized":"2024-09-18T07:42:27.177832+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:52.366672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:52.366672+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":640,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.169682+0000","last_change":"2024-09-18T07:41:56.060035+0000","last_active":"2024-09-18T07:42:27.169682+0000","last_peered":"2024-09-18T07:42:27.169682+0000","last_clean":"2024-09-18T07:42:27.169682+0000","last_became_active":"2024-09-18T07:35:01.414024+0000","last_became_peered":"2024-09-18T07:35:01.414024+0000","last_unstale":"2024-09-18T07:42:27.169682+0000","last_undegraded":"2024-09-18T07:42:27.169682+0000","last_fullsized":"2024-09-18T07:42:27.169682+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:56.059922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:56.059922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:07.678767+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"173'10","reported_seq":774,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.181431+0000","last_change":"2024-09-18T07:39:32.947933+0000","last_active":"2024-09-18T07:42:27.181431+0000","last_peered":"2024-09-18T07:42:27.181431+0000","last_clean":"2024-09-18T07:42:27.181431+0000","last_became_active":"2024-09-18T07:35:01.407568+0000","last_became_peered":"2024-09-18T07:35:01.407568+0000","last_unstale":"2024-09-18T07:42:27.181431+0000","last_undegraded":"2024-09-18T07:42:27.181431+0000","last_fullsized":"2024-09-18T07:42:27.181431+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:39:32.947887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:32.947887+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:40.468020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":610,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174830+0000","last_change":"2024-09-18T07:41:48.955522+0000","last_active":"2024-09-18T07:42:27.174830+0000","last_peered":"2024-09-18T07:42:27.174830+0000","last_clean":"2024-09-18T07:42:27.174830+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:42:27.174830+0000","last_undegraded":"2024-09-18T07:42:27.174830+0000","last_fullsized":"2024-09-18T07:42:27.174830+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:48.955463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.955463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:00.936125+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"323'20","reported_seq":395,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:29.867736+0000","last_change":"2024-09-18T07:42:29.867736+0000","last_active":"2024-09-18T07:42:29.867736+0000","last_peered":"2024-09-18T07:42:29.867736+0000","last_clean":"2024-09-18T07:42:29.867736+0000","last_became_active":"2024-09-18T07:40:19.069759+0000","last_became_peered":"2024-09-18T07:40:19.069759+0000","last_unstale":"2024-09-18T07:42:29.867736+0000","last_undegraded":"2024-09-18T07:42:29.867736+0000","last_fullsized":"2024-09-18T07:42:29.867736+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:42:29.867645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:29.867645+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:40.470633+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":395,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.185762+0000","last_change":"2024-09-18T07:42:04.758461+0000","last_active":"2024-09-18T07:42:27.185762+0000","last_peered":"2024-09-18T07:42:27.185762+0000","last_clean":"2024-09-18T07:42:27.185762+0000","last_became_active":"2024-09-18T07:35:01.400197+0000","last_became_peered":"2024-09-18T07:35:01.400197+0000","last_unstale":"2024-09-18T07:42:27.185762+0000","last_undegraded":"2024-09-18T07:42:27.185762+0000","last_fullsized":"2024-09-18T07:42:27.185762+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:04.758396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.758396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:27.831062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":755,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.177325+0000","last_change":"2024-09-18T07:40:40.352088+0000","last_active":"2024-09-18T07:42:27.177325+0000","last_peered":"2024-09-18T07:42:27.177325+0000","last_clean":"2024-09-18T07:42:27.177325+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:42:27.177325+0000","last_undegraded":"2024-09-18T07:42:27.177325+0000","last_fullsized":"2024-09-18T07:42:27.177325+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:40:40.352040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.352040+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":628,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.171615+0000","last_change":"2024-09-18T07:42:01.071103+0000","last_active":"2024-09-18T07:42:27.171615+0000","last_peered":"2024-09-18T07:42:27.171615+0000","last_clean":"2024-09-18T07:42:27.171615+0000","last_became_active":"2024-09-18T07:35:01.410561+0000","last_became_peered":"2024-09-18T07:35:01.410561+0000","last_unstale":"2024-09-18T07:42:27.171615+0000","last_undegraded":"2024-09-18T07:42:27.171615+0000","last_fullsized":"2024-09-18T07:42:27.171615+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:01.071028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:01.071028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:12.908296+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"323'19","reported_seq":719,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.173904+0000","last_change":"2024-09-18T07:41:54.022517+0000","last_active":"2024-09-18T07:42:27.173904+0000","last_peered":"2024-09-18T07:42:27.173904+0000","last_clean":"2024-09-18T07:42:27.173904+0000","last_became_active":"2024-09-18T07:38:41.143108+0000","last_became_peered":"2024-09-18T07:38:41.143108+0000","last_unstale":"2024-09-18T07:42:27.173904+0000","last_undegraded":"2024-09-18T07:42:27.173904+0000","last_fullsized":"2024-09-18T07:42:27.173904+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:41:54.022421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.022421+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:19.855029+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":634,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174838+0000","last_change":"2024-09-18T07:40:40.149372+0000","last_active":"2024-09-18T07:42:27.174838+0000","last_peered":"2024-09-18T07:42:27.174838+0000","last_clean":"2024-09-18T07:42:27.174838+0000","last_became_active":"2024-09-18T07:37:43.061577+0000","last_became_peered":"2024-09-18T07:37:43.061577+0000","last_unstale":"2024-09-18T07:42:27.174838+0000","last_undegraded":"2024-09-18T07:42:27.174838+0000","last_fullsized":"2024-09-18T07:42:27.174838+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:40.149313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.149313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:16.130628+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":305,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.177680+0000","last_change":"2024-09-18T07:41:24.566420+0000","last_active":"2024-09-18T07:42:27.177680+0000","last_peered":"2024-09-18T07:42:27.177680+0000","last_clean":"2024-09-18T07:42:27.177680+0000","last_became_active":"2024-09-18T07:36:13.117584+0000","last_became_peered":"2024-09-18T07:36:13.117584+0000","last_unstale":"2024-09-18T07:42:27.177680+0000","last_undegraded":"2024-09-18T07:42:27.177680+0000","last_fullsized":"2024-09-18T07:42:27.177680+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:24.566345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:24.566345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:52.792357+0000","scrub_duration":160,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":109,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.171440+0000","last_change":"2024-09-18T07:42:06.204026+0000","last_active":"2024-09-18T07:42:27.171440+0000","last_peered":"2024-09-18T07:42:27.171440+0000","last_clean":"2024-09-18T07:42:27.171440+0000","last_became_active":"2024-09-18T07:40:19.065468+0000","last_became_peered":"2024-09-18T07:40:19.065468+0000","last_unstale":"2024-09-18T07:42:27.171440+0000","last_undegraded":"2024-09-18T07:42:27.171440+0000","last_fullsized":"2024-09-18T07:42:27.171440+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.203932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.203932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:27.863077+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"323'15","reported_seq":504,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.169455+0000","last_change":"2024-09-18T07:41:20.975210+0000","last_active":"2024-09-18T07:42:27.169455+0000","last_peered":"2024-09-18T07:42:27.169455+0000","last_clean":"2024-09-18T07:42:27.169455+0000","last_became_active":"2024-09-18T07:37:43.067598+0000","last_became_peered":"2024-09-18T07:37:43.067598+0000","last_unstale":"2024-09-18T07:42:27.169455+0000","last_undegraded":"2024-09-18T07:42:27.169455+0000","last_fullsized":"2024-09-18T07:42:27.169455+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:20.975135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:20.975135+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":395,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.184647+0000","last_change":"2024-09-18T07:41:47.881147+0000","last_active":"2024-09-18T07:42:27.184647+0000","last_peered":"2024-09-18T07:42:27.184647+0000","last_clean":"2024-09-18T07:42:27.184647+0000","last_became_active":"2024-09-18T07:35:01.405287+0000","last_became_peered":"2024-09-18T07:35:01.405287+0000","last_unstale":"2024-09-18T07:42:27.184647+0000","last_undegraded":"2024-09-18T07:42:27.184647+0000","last_fullsized":"2024-09-18T07:42:27.184647+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:47.880911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:47.880911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:00.989233+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":101,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.170138+0000","last_change":"2024-09-18T07:40:19.068620+0000","last_active":"2024-09-18T07:42:27.170138+0000","last_peered":"2024-09-18T07:42:27.170138+0000","last_clean":"2024-09-18T07:42:27.170138+0000","last_became_active":"2024-09-18T07:40:19.068237+0000","last_became_peered":"2024-09-18T07:40:19.068237+0000","last_unstale":"2024-09-18T07:42:27.170138+0000","last_undegraded":"2024-09-18T07:42:27.170138+0000","last_fullsized":"2024-09-18T07:42:27.170138+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:56.920384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:38:56.920384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:34.021901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":622,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.174190+0000","last_change":"2024-09-18T07:41:18.963607+0000","last_active":"2024-09-18T07:42:27.174190+0000","last_peered":"2024-09-18T07:42:27.174190+0000","last_clean":"2024-09-18T07:42:27.174190+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:42:27.174190+0000","last_undegraded":"2024-09-18T07:42:27.174190+0000","last_fullsized":"2024-09-18T07:42:27.174190+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:18.963490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:18.963490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:42.985064+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"323'19","reported_seq":724,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.187698+0000","last_change":"2024-09-18T07:42:10.458954+0000","last_active":"2024-09-18T07:42:27.187698+0000","last_peered":"2024-09-18T07:42:27.187698+0000","last_clean":"2024-09-18T07:42:27.187698+0000","last_became_active":"2024-09-18T07:37:43.066208+0000","last_became_peered":"2024-09-18T07:37:43.066208+0000","last_unstale":"2024-09-18T07:42:27.187698+0000","last_undegraded":"2024-09-18T07:42:27.187698+0000","last_fullsized":"2024-09-18T07:42:27.187698+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:42:10.458865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:10.458865+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:33.857115+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":626,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.180177+0000","last_change":"2024-09-18T07:41:27.341104+0000","last_active":"2024-09-18T07:42:27.180177+0000","last_peered":"2024-09-18T07:42:27.180177+0000","last_clean":"2024-09-18T07:42:27.180177+0000","last_became_active":"2024-09-18T07:35:01.415261+0000","last_became_peered":"2024-09-18T07:35:01.415261+0000","last_unstale":"2024-09-18T07:42:27.180177+0000","last_undegraded":"2024-09-18T07:42:27.180177+0000","last_fullsized":"2024-09-18T07:42:27.180177+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:27.341063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.341063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:38.597550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":409,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.181815+0000","last_change":"2024-09-18T07:42:05.775215+0000","last_active":"2024-09-18T07:42:27.181815+0000","last_peered":"2024-09-18T07:42:27.181815+0000","last_clean":"2024-09-18T07:42:27.181815+0000","last_became_active":"2024-09-18T07:37:43.057483+0000","last_became_peered":"2024-09-18T07:37:43.057483+0000","last_unstale":"2024-09-18T07:42:27.181815+0000","last_undegraded":"2024-09-18T07:42:27.181815+0000","last_fullsized":"2024-09-18T07:42:27.181815+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:05.775170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:05.775170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:20.076140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":631,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.179999+0000","last_change":"2024-09-18T07:41:27.610895+0000","last_active":"2024-09-18T07:42:27.179999+0000","last_peered":"2024-09-18T07:42:27.179999+0000","last_clean":"2024-09-18T07:42:27.179999+0000","last_became_active":"2024-09-18T07:37:43.067066+0000","last_became_peered":"2024-09-18T07:37:43.067066+0000","last_unstale":"2024-09-18T07:42:27.179999+0000","last_undegraded":"2024-09-18T07:42:27.179999+0000","last_fullsized":"2024-09-18T07:42:27.179999+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:27.610707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.610707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:28.092460+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"323'8","reported_seq":668,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.178959+0000","last_change":"2024-09-18T07:37:43.134956+0000","last_active":"2024-09-18T07:42:27.178959+0000","last_peered":"2024-09-18T07:42:27.178959+0000","last_clean":"2024-09-18T07:42:27.178959+0000","last_became_active":"2024-09-18T07:37:43.067503+0000","last_became_peered":"2024-09-18T07:37:43.067503+0000","last_unstale":"2024-09-18T07:42:27.178959+0000","last_undegraded":"2024-09-18T07:42:27.178959+0000","last_fullsized":"2024-09-18T07:42:27.178959+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:36:40.908629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:40.908629+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:13.423257+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":610,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.172441+0000","last_change":"2024-09-18T07:40:11.843693+0000","last_active":"2024-09-18T07:42:27.172441+0000","last_peered":"2024-09-18T07:42:27.172441+0000","last_clean":"2024-09-18T07:42:27.172441+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:42:27.172441+0000","last_undegraded":"2024-09-18T07:42:27.172441+0000","last_fullsized":"2024-09-18T07:42:27.172441+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:11.843625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:11.843625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:42.965878+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":246,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.185327+0000","last_change":"2024-09-18T07:40:40.106989+0000","last_active":"2024-09-18T07:42:27.185327+0000","last_peered":"2024-09-18T07:42:27.185327+0000","last_clean":"2024-09-18T07:42:27.185327+0000","last_became_active":"2024-09-18T07:37:43.061081+0000","last_became_peered":"2024-09-18T07:37:43.061081+0000","last_unstale":"2024-09-18T07:42:27.185327+0000","last_undegraded":"2024-09-18T07:42:27.185327+0000","last_fullsized":"2024-09-18T07:42:27.185327+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:40.106921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.106921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:11.100563+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":395,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.182215+0000","last_change":"2024-09-18T07:40:38.492338+0000","last_active":"2024-09-18T07:42:27.182215+0000","last_peered":"2024-09-18T07:42:27.182215+0000","last_clean":"2024-09-18T07:42:27.182215+0000","last_became_active":"2024-09-18T07:35:01.400396+0000","last_became_peered":"2024-09-18T07:35:01.400396+0000","last_unstale":"2024-09-18T07:42:27.182215+0000","last_undegraded":"2024-09-18T07:42:27.182215+0000","last_fullsized":"2024-09-18T07:42:27.182215+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:38.492269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.492269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:17.788359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":297,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.177376+0000","last_change":"2024-09-18T07:36:13.118009+0000","last_active":"2024-09-18T07:42:27.177376+0000","last_peered":"2024-09-18T07:42:27.177376+0000","last_clean":"2024-09-18T07:42:27.177376+0000","last_became_active":"2024-09-18T07:36:13.117154+0000","last_became_peered":"2024-09-18T07:36:13.117154+0000","last_unstale":"2024-09-18T07:42:27.177376+0000","last_undegraded":"2024-09-18T07:42:27.177376+0000","last_fullsized":"2024-09-18T07:42:27.177376+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.071629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.071629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:43.255444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":635,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.184386+0000","last_change":"2024-09-18T07:41:22.478552+0000","last_active":"2024-09-18T07:42:27.184386+0000","last_peered":"2024-09-18T07:42:27.184386+0000","last_clean":"2024-09-18T07:42:27.184386+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:42:27.184386+0000","last_undegraded":"2024-09-18T07:42:27.184386+0000","last_fullsized":"2024-09-18T07:42:27.184386+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:41:22.478474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:22.478474+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:37.260325+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"323'19","reported_seq":270,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.184933+0000","last_change":"2024-09-18T07:41:23.281610+0000","last_active":"2024-09-18T07:42:27.184933+0000","last_peered":"2024-09-18T07:42:27.184933+0000","last_clean":"2024-09-18T07:42:27.184933+0000","last_became_active":"2024-09-18T07:37:43.070186+0000","last_became_peered":"2024-09-18T07:37:43.070186+0000","last_unstale":"2024-09-18T07:42:27.184933+0000","last_undegraded":"2024-09-18T07:42:27.184933+0000","last_fullsized":"2024-09-18T07:42:27.184933+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:41:23.281547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:23.281547+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:41.164144+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":633,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-18T07:42:27.171752+0000","last_change":"2024-09-18T07:41:13.842121+0000","last_active":"2024-09-18T07:42:27.171752+0000","last_peered":"2024-09-18T07:42:27.171752+0000","last_clean":"2024-09-18T07:42:27.171752+0000","last_became_active":"2024-09-18T07:35:01.410469+0000","last_became_peered":"2024-09-18T07:35:01.410469+0000","last_unstale":"2024-09-18T07:42:27.171752+0000","last_undegraded":"2024-09-18T07:42:27.171752+0000","last_fullsized":"2024-09-18T07:42:27.171752+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:13.842068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:13.842068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1781923864,"num_objects":27191,"num_object_clones":0,"num_object_copies":81573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27191,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237805,"num_write_kb":8479809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5225590784,"data_stored":5225578568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79605,"ondisk_log_size":79605,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1118,"num_read_kb":875,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":4664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":684,"ondisk_log_size":684,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":456,"num_read_kb":451,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201423,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1384796,"kb_used_data":966284,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92360356,"statfs":{"total":95995035648,"available":94577004544,"internally_reserved":0,"allocated":989474816,"data_stored":989118783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":428538314},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463061,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":645048,"kb_used_data":320936,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93100104,"statfs":{"total":95995035648,"available":95334506496,"internally_reserved":0,"allocated":328638464,"data_stored":328317856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":331872710},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724698,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1404496,"kb_used_data":926272,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":92340656,"statfs":{"total":95995035648,"available":94556831744,"internally_reserved":0,"allocated":948502528,"data_stored":948158698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953633,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653848,"kb_used_data":321416,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":93091304,"statfs":{"total":95995035648,"available":95325495296,"internally_reserved":0,"allocated":329129984,"data_stored":328777746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":340392389},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1548488,"kb_used_data":1003576,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":92196664,"statfs":{"total":95995035648,"available":94409383936,"internally_reserved":0,"allocated":1027661824,"data_stored":1027261533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":557906375},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411499,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":724872,"kb_used_data":306360,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93020280,"statfs":{"total":95995035648,"available":95252766720,"internally_reserved":0,"allocated":313712640,"data_stored":313376877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575026,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1462952,"kb_used_data":940312,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":92282200,"statfs":{"total":95995035648,"available":94496972800,"internally_reserved":0,"allocated":962879488,"data_stored":962511967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738550,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":876892,"kb_used_data":484624,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92868260,"statfs":{"total":95995035648,"available":95097098240,"internally_reserved":0,"allocated":496254976,"data_stored":495895948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":485883904,"data_stored":485883904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":927207424,"data_stored":927203352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":311623680,"data_stored":311623680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":988413952,"data_stored":988413952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":322371584,"data_stored":322371584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":920129536,"data_stored":920125464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":317718528,"data_stored":317718528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":952242176,"data_stored":952238104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:42:38.027 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-18T07:42:38.027 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:42:38.028 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-18T07:42:38.028 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-18T07:42:38.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:42:38.028 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:42:37 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:42:38.067 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:38.106 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:38.130 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:38.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:38.231 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:38.265 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:38.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:38.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:38.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:38.425 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:38.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:38.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:38.484 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:38 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:38.484 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:38 smithi137 ceph-mon[29408]: pgmap v621: 158 pgs: 158 active+clean; 1.2 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.67k op/s 2024-09-18T07:42:38.484 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:38 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:38.484 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:38 smithi137 ceph-mon[39175]: pgmap v621: 158 pgs: 158 active+clean; 1.2 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.67k op/s 2024-09-18T07:42:38.527 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:38.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:38.565 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:38.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:38.666 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:38.708 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:38.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:38.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:38.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:38 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:38 smithi203 ceph-mon[32237]: pgmap v621: 158 pgs: 158 active+clean; 1.2 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.67k op/s 2024-09-18T07:42:38.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:38.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:38.844 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:38.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:38.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:38.980 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:38.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:38.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:39.082 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:39.117 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:39.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:39.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:39.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:39.257 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:39.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:39.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:39.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:39.372 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:42:39.372 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:42:39.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:42:39.373 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:42:39.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.373 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135011 7.14 7.3991ba34 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-18T07:42:39.373 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:15.993779+0000", 2024-09-18T07:42:39.373 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 503.37144480299997, 2024-09-18T07:42:39.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98326573299999998, 2024-09-18T07:42:39.373 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.373 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.373 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:39.373 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.373 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:39.374 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:39.374 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135011 2024-09-18T07:42:39.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.374 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993779+0000", 2024-09-18T07:42:39.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993779+0000", 2024-09-18T07:42:39.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993780+0000", 2024-09-18T07:42:39.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6899999999999996e-07 2024-09-18T07:42:39.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993784+0000", 2024-09-18T07:42:39.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6270000000000002e-06 2024-09-18T07:42:39.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993794+0000", 2024-09-18T07:42:39.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6889999999999999e-06 2024-09-18T07:42:39.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993799+0000", 2024-09-18T07:42:39.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1590000000000001e-06 2024-09-18T07:42:39.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993861+0000", 2024-09-18T07:42:39.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2421000000000004e-05 2024-09-18T07:42:39.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.993869+0000", 2024-09-18T07:42:39.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1980000000000001e-06 2024-09-18T07:42:39.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.977033+0000", 2024-09-18T07:42:39.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98316372699999999 2024-09-18T07:42:39.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.977045+0000", 2024-09-18T07:42:39.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2043e-05 2024-09-18T07:42:39.378 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.378 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.378 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.378 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135018 7.10 7:0de93d52:::benchmark_data_smithi137_74180_object30602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-18T07:42:39.378 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:15.998577+0000", 2024-09-18T07:42:39.378 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 503.36664703600002, 2024-09-18T07:42:39.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1980523299999999, 2024-09-18T07:42:39.378 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.378 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.379 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:39.379 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.379 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:39.379 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:39.379 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135018 2024-09-18T07:42:39.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.379 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998577+0000", 2024-09-18T07:42:39.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998577+0000", 2024-09-18T07:42:39.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998579+0000", 2024-09-18T07:42:39.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.336e-06 2024-09-18T07:42:39.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998583+0000", 2024-09-18T07:42:39.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7190000000000001e-06 2024-09-18T07:42:39.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998593+0000", 2024-09-18T07:42:39.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4450000000000002e-06 2024-09-18T07:42:39.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998598+0000", 2024-09-18T07:42:39.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2329999999999997e-06 2024-09-18T07:42:39.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998649+0000", 2024-09-18T07:42:39.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0735999999999997e-05 2024-09-18T07:42:39.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.998659+0000", 2024-09-18T07:42:39.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0156000000000001e-05 2024-09-18T07:42:39.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.970426+0000", 2024-09-18T07:42:39.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97176680299999996 2024-09-18T07:42:39.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.972639+0000", 2024-09-18T07:42:39.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022131070000000002 2024-09-18T07:42:39.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.183802+0000", 2024-09-18T07:42:39.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2111635359999999 2024-09-18T07:42:39.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:39.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.183932+0000", 2024-09-18T07:42:39.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000129686 2024-09-18T07:42:39.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2,5", 2024-09-18T07:42:39.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184136+0000", 2024-09-18T07:42:39.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000203885 2024-09-18T07:42:39.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:39.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185714+0000", 2024-09-18T07:42:39.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00157839 2024-09-18T07:42:39.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:39.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.188451+0000", 2024-09-18T07:42:39.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002736741 2024-09-18T07:42:39.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:39.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.188856+0000", 2024-09-18T07:42:39.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040506400000000001 2024-09-18T07:42:39.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:39.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.195963+0000", 2024-09-18T07:42:39.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0071068019999999997 2024-09-18T07:42:39.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:39.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.196200+0000", 2024-09-18T07:42:39.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023698600000000001 2024-09-18T07:42:39.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.196630+0000", 2024-09-18T07:42:39.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00042970500000000001 2024-09-18T07:42:39.386 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.386 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.386 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.387 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162368 7.2 7.85717ac2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:42:39.387 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.437764+0000", 2024-09-18T07:42:39.387 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 456.92746045400003, 2024-09-18T07:42:39.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97823369100000002, 2024-09-18T07:42:39.387 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.387 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.387 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:39.387 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.387 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:39.387 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:39.388 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162368 2024-09-18T07:42:39.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.388 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437764+0000", 2024-09-18T07:42:39.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437764+0000", 2024-09-18T07:42:39.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437766+0000", 2024-09-18T07:42:39.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8479999999999999e-06 2024-09-18T07:42:39.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437771+0000", 2024-09-18T07:42:39.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4580000000000001e-06 2024-09-18T07:42:39.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437781+0000", 2024-09-18T07:42:39.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0291e-05 2024-09-18T07:42:39.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437789+0000", 2024-09-18T07:42:39.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3599999999999998e-06 2024-09-18T07:42:39.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437856+0000", 2024-09-18T07:42:39.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7241000000000002e-05 2024-09-18T07:42:39.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.437903+0000", 2024-09-18T07:42:39.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7348999999999997e-05 2024-09-18T07:42:39.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415935+0000", 2024-09-18T07:42:39.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97803116499999998 2024-09-18T07:42:39.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415997+0000", 2024-09-18T07:42:39.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2978999999999995e-05 2024-09-18T07:42:39.392 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.392 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.392 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.392 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162379 7.12 7.b17b62b2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:42:39.392 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.446092+0000", 2024-09-18T07:42:39.392 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 456.91913231299998, 2024-09-18T07:42:39.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96914699900000001, 2024-09-18T07:42:39.392 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.392 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.393 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:39.393 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.393 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:39.393 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:39.393 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162379 2024-09-18T07:42:39.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.393 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446092+0000", 2024-09-18T07:42:39.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446092+0000", 2024-09-18T07:42:39.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446095+0000", 2024-09-18T07:42:39.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7329999999999999e-06 2024-09-18T07:42:39.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446102+0000", 2024-09-18T07:42:39.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9110000000000004e-06 2024-09-18T07:42:39.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446118+0000", 2024-09-18T07:42:39.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6875999999999999e-05 2024-09-18T07:42:39.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446126+0000", 2024-09-18T07:42:39.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7030000000000006e-06 2024-09-18T07:42:39.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446193+0000", 2024-09-18T07:42:39.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6919000000000001e-05 2024-09-18T07:42:39.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.446203+0000", 2024-09-18T07:42:39.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6439999999999995e-06 2024-09-18T07:42:39.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415214+0000", 2024-09-18T07:42:39.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96901095599999998 2024-09-18T07:42:39.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415239+0000", 2024-09-18T07:42:39.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5256999999999999e-05 2024-09-18T07:42:39.397 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.397 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.397 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.397 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162393 7.12 7.f00c1e52 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:42:39.398 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.460966+0000", 2024-09-18T07:42:39.398 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 456.90425860300002, 2024-09-18T07:42:39.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95474156300000002, 2024-09-18T07:42:39.398 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.398 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.398 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:39.398 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.398 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:39.398 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:39.398 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162393 2024-09-18T07:42:39.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.398 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.460966+0000", 2024-09-18T07:42:39.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.460966+0000", 2024-09-18T07:42:39.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.460968+0000", 2024-09-18T07:42:39.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9870000000000002e-06 2024-09-18T07:42:39.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.460973+0000", 2024-09-18T07:42:39.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5890000000000002e-06 2024-09-18T07:42:39.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.460997+0000", 2024-09-18T07:42:39.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3583000000000001e-05 2024-09-18T07:42:39.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.461004+0000", 2024-09-18T07:42:39.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0840000000000003e-06 2024-09-18T07:42:39.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.461064+0000", 2024-09-18T07:42:39.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9701000000000003e-05 2024-09-18T07:42:39.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.461074+0000", 2024-09-18T07:42:39.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9240000000000002e-06 2024-09-18T07:42:39.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415699+0000", 2024-09-18T07:42:39.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95462499899999997 2024-09-18T07:42:39.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415707+0000", 2024-09-18T07:42:39.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6959999999999994e-06 2024-09-18T07:42:39.402 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.402 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.403 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.403 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162407 7.2 7.5f39ee22 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:42:39.403 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.503104+0000", 2024-09-18T07:42:39.403 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 456.86212056699998, 2024-09-18T07:42:39.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91293045900000003, 2024-09-18T07:42:39.403 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.404 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.404 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:39.404 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.404 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:39.404 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:39.404 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162407 2024-09-18T07:42:39.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.404 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503104+0000", 2024-09-18T07:42:39.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503104+0000", 2024-09-18T07:42:39.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503105+0000", 2024-09-18T07:42:39.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3200000000000001e-06 2024-09-18T07:42:39.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503110+0000", 2024-09-18T07:42:39.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7709999999999997e-06 2024-09-18T07:42:39.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503120+0000", 2024-09-18T07:42:39.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0217000000000001e-05 2024-09-18T07:42:39.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503127+0000", 2024-09-18T07:42:39.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9430000000000003e-06 2024-09-18T07:42:39.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503227+0000", 2024-09-18T07:42:39.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100147 2024-09-18T07:42:39.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.503235+0000", 2024-09-18T07:42:39.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3270000000000001e-06 2024-09-18T07:42:39.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416027+0000", 2024-09-18T07:42:39.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91279167400000005 2024-09-18T07:42:39.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416034+0000", 2024-09-18T07:42:39.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0600000000000002e-06 2024-09-18T07:42:39.408 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.408 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.408 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.408 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162429 7.12 7.8e24a052 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:42:39.409 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.531777+0000", 2024-09-18T07:42:39.409 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 456.83344747000001, 2024-09-18T07:42:39.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.883964953, 2024-09-18T07:42:39.409 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.409 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.409 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:39.409 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.409 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:39.409 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:39.409 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162429 2024-09-18T07:42:39.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.409 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531777+0000", 2024-09-18T07:42:39.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531777+0000", 2024-09-18T07:42:39.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531778+0000", 2024-09-18T07:42:39.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7120000000000001e-06 2024-09-18T07:42:39.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531783+0000", 2024-09-18T07:42:39.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.622e-06 2024-09-18T07:42:39.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531792+0000", 2024-09-18T07:42:39.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1889999999999996e-06 2024-09-18T07:42:39.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.531798+0000", 2024-09-18T07:42:39.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9599999999999997e-06 2024-09-18T07:42:39.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.533462+0000", 2024-09-18T07:42:39.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001663351 2024-09-18T07:42:39.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.533471+0000", 2024-09-18T07:42:39.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5270000000000004e-06 2024-09-18T07:42:39.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415736+0000", 2024-09-18T07:42:39.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.88226443600000004 2024-09-18T07:42:39.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415742+0000", 2024-09-18T07:42:39.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.156e-06 2024-09-18T07:42:39.413 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.413 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.413 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.413 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162435 7.2 7.90a4f302 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:42:39.413 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.536705+0000", 2024-09-18T07:42:39.414 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 456.828519422, 2024-09-18T07:42:39.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.87935532000000005, 2024-09-18T07:42:39.414 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.414 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.414 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:39.414 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.414 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:39.414 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:39.414 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162435 2024-09-18T07:42:39.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.414 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536705+0000", 2024-09-18T07:42:39.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536705+0000", 2024-09-18T07:42:39.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536707+0000", 2024-09-18T07:42:39.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7940000000000001e-06 2024-09-18T07:42:39.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536712+0000", 2024-09-18T07:42:39.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9790000000000001e-06 2024-09-18T07:42:39.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536725+0000", 2024-09-18T07:42:39.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2986999999999999e-05 2024-09-18T07:42:39.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536732+0000", 2024-09-18T07:42:39.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6340000000000001e-06 2024-09-18T07:42:39.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536789+0000", 2024-09-18T07:42:39.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6675999999999997e-05 2024-09-18T07:42:39.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.536798+0000", 2024-09-18T07:42:39.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5100000000000004e-06 2024-09-18T07:42:39.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416054+0000", 2024-09-18T07:42:39.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.87925583100000004 2024-09-18T07:42:39.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416060+0000", 2024-09-18T07:42:39.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9089999999999997e-06 2024-09-18T07:42:39.418 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.418 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.418 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.419 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:39.419 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162461 7.2 7.86b206e2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:42:39.419 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.562650+0000", 2024-09-18T07:42:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 456.802574284, 2024-09-18T07:42:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85345243000000004, 2024-09-18T07:42:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162461 2024-09-18T07:42:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.420 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.421 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562650+0000", 2024-09-18T07:42:39.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.421 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.421 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562650+0000", 2024-09-18T07:42:39.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.421 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.421 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562651+0000", 2024-09-18T07:42:39.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.246e-06 2024-09-18T07:42:39.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562656+0000", 2024-09-18T07:42:39.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3189999999999996e-06 2024-09-18T07:42:39.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562667+0000", 2024-09-18T07:42:39.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1623e-05 2024-09-18T07:42:39.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562675+0000", 2024-09-18T07:42:39.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5000000000000002e-06 2024-09-18T07:42:39.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562871+0000", 2024-09-18T07:42:39.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000196219 2024-09-18T07:42:39.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.562939+0000", 2024-09-18T07:42:39.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8196999999999998e-05 2024-09-18T07:42:39.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416088+0000", 2024-09-18T07:42:39.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85314855700000003 2024-09-18T07:42:39.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416102+0000", 2024-09-18T07:42:39.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4769e-05 2024-09-18T07:42:39.424 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.424 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.424 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.424 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162466 7.12 7.e244e932 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:42:39.425 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.563197+0000", 2024-09-18T07:42:39.425 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 456.80202722899998, 2024-09-18T07:42:39.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85257313899999998, 2024-09-18T07:42:39.425 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.425 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.425 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:39.425 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.425 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:39.425 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:39.425 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162466 2024-09-18T07:42:39.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.425 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563197+0000", 2024-09-18T07:42:39.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563197+0000", 2024-09-18T07:42:39.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563198+0000", 2024-09-18T07:42:39.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3340000000000001e-06 2024-09-18T07:42:39.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563203+0000", 2024-09-18T07:42:39.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7110000000000003e-06 2024-09-18T07:42:39.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563217+0000", 2024-09-18T07:42:39.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4314999999999999e-05 2024-09-18T07:42:39.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563225+0000", 2024-09-18T07:42:39.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1420000000000004e-06 2024-09-18T07:42:39.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563294+0000", 2024-09-18T07:42:39.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9309999999999999e-05 2024-09-18T07:42:39.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.563303+0000", 2024-09-18T07:42:39.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3659999999999998e-06 2024-09-18T07:42:39.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415764+0000", 2024-09-18T07:42:39.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.85246111000000002 2024-09-18T07:42:39.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415770+0000", 2024-09-18T07:42:39.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8510000000000004e-06 2024-09-18T07:42:39.429 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.429 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.430 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.430 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162470 7.12 7.9c7c55b2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:42:39.430 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.566190+0000", 2024-09-18T07:42:39.430 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 456.79903424299999, 2024-09-18T07:42:39.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.84961990099999996, 2024-09-18T07:42:39.430 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.430 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.430 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:39.430 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.430 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:39.430 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:39.431 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162470 2024-09-18T07:42:39.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.431 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566190+0000", 2024-09-18T07:42:39.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566190+0000", 2024-09-18T07:42:39.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566192+0000", 2024-09-18T07:42:39.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.762e-06 2024-09-18T07:42:39.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566198+0000", 2024-09-18T07:42:39.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9499999999999998e-06 2024-09-18T07:42:39.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566211+0000", 2024-09-18T07:42:39.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3743999999999999e-05 2024-09-18T07:42:39.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566219+0000", 2024-09-18T07:42:39.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7810000000000005e-06 2024-09-18T07:42:39.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566282+0000", 2024-09-18T07:42:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2362999999999999e-05 2024-09-18T07:42:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.566290+0000", 2024-09-18T07:42:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7749999999999998e-06 2024-09-18T07:42:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415803+0000", 2024-09-18T07:42:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.84951286599999998 2024-09-18T07:42:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415810+0000", 2024-09-18T07:42:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6599999999999998e-06 2024-09-18T07:42:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162495 7.12 7.468e4332 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:42:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.583195+0000", 2024-09-18T07:42:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 456.78202950600001, 2024-09-18T07:42:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.83264318599999998, 2024-09-18T07:42:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.435 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162495 2024-09-18T07:42:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583195+0000", 2024-09-18T07:42:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583195+0000", 2024-09-18T07:42:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583196+0000", 2024-09-18T07:42:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6819999999999999e-06 2024-09-18T07:42:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583201+0000", 2024-09-18T07:42:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0209999999999999e-06 2024-09-18T07:42:39.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583211+0000", 2024-09-18T07:42:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3610000000000007e-06 2024-09-18T07:42:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583218+0000", 2024-09-18T07:42:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8560000000000001e-06 2024-09-18T07:42:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583268+0000", 2024-09-18T07:42:39.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0294000000000003e-05 2024-09-18T07:42:39.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.583277+0000", 2024-09-18T07:42:39.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2839999999999995e-06 2024-09-18T07:42:39.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415831+0000", 2024-09-18T07:42:39.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.83255423799999995 2024-09-18T07:42:39.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415838+0000", 2024-09-18T07:42:39.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4500000000000001e-06 2024-09-18T07:42:39.440 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.440 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.440 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.440 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162501 7.12 7.5f5c6f12 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:42:39.440 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.586226+0000", 2024-09-18T07:42:39.440 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 456.77899845100001, 2024-09-18T07:42:39.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.82964114600000005, 2024-09-18T07:42:39.440 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.441 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.441 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:39.441 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.441 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:39.441 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:39.441 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162501 2024-09-18T07:42:39.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.441 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586226+0000", 2024-09-18T07:42:39.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586226+0000", 2024-09-18T07:42:39.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586227+0000", 2024-09-18T07:42:39.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.598e-06 2024-09-18T07:42:39.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586233+0000", 2024-09-18T07:42:39.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5280000000000003e-06 2024-09-18T07:42:39.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586246+0000", 2024-09-18T07:42:39.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3123e-05 2024-09-18T07:42:39.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586254+0000", 2024-09-18T07:42:39.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3469999999999999e-06 2024-09-18T07:42:39.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586344+0000", 2024-09-18T07:42:39.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9617999999999995e-05 2024-09-18T07:42:39.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.586354+0000", 2024-09-18T07:42:39.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8339999999999994e-06 2024-09-18T07:42:39.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415860+0000", 2024-09-18T07:42:39.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.82950663999999996 2024-09-18T07:42:39.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415867+0000", 2024-09-18T07:42:39.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4579999999999998e-06 2024-09-18T07:42:39.445 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.445 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.445 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.445 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162524 7.2 7.5cbbb662 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:42:39.445 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.609728+0000", 2024-09-18T07:42:39.445 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 456.755496542, 2024-09-18T07:42:39.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.80640156299999999, 2024-09-18T07:42:39.446 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.446 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.446 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:39.446 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.446 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:39.446 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:39.446 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162524 2024-09-18T07:42:39.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.446 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609728+0000", 2024-09-18T07:42:39.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609728+0000", 2024-09-18T07:42:39.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609730+0000", 2024-09-18T07:42:39.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.894e-06 2024-09-18T07:42:39.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609741+0000", 2024-09-18T07:42:39.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1466e-05 2024-09-18T07:42:39.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609755+0000", 2024-09-18T07:42:39.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3584000000000001e-05 2024-09-18T07:42:39.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609762+0000", 2024-09-18T07:42:39.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7950000000000008e-06 2024-09-18T07:42:39.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609832+0000", 2024-09-18T07:42:39.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0029000000000003e-05 2024-09-18T07:42:39.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.609844+0000", 2024-09-18T07:42:39.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1154e-05 2024-09-18T07:42:39.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416123+0000", 2024-09-18T07:42:39.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.80627929200000004 2024-09-18T07:42:39.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416129+0000", 2024-09-18T07:42:39.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3489999999999997e-06 2024-09-18T07:42:39.450 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.450 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.450 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.450 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162529 7.12 7.8e054bd2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:42:39.451 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.612942+0000", 2024-09-18T07:42:39.451 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 456.75228253199998, 2024-09-18T07:42:39.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.80297076000000001, 2024-09-18T07:42:39.451 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.451 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.451 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:39.451 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.451 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:39.451 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:39.451 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162529 2024-09-18T07:42:39.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.451 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.451 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612942+0000", 2024-09-18T07:42:39.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612942+0000", 2024-09-18T07:42:39.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612943+0000", 2024-09-18T07:42:39.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.53e-06 2024-09-18T07:42:39.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612948+0000", 2024-09-18T07:42:39.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.006e-06 2024-09-18T07:42:39.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612959+0000", 2024-09-18T07:42:39.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.029e-05 2024-09-18T07:42:39.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.612966+0000", 2024-09-18T07:42:39.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9459999999999998e-06 2024-09-18T07:42:39.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.613042+0000", 2024-09-18T07:42:39.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5358999999999995e-05 2024-09-18T07:42:39.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.613062+0000", 2024-09-18T07:42:39.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9783999999999999e-05 2024-09-18T07:42:39.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415904+0000", 2024-09-18T07:42:39.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.80284268700000005 2024-09-18T07:42:39.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.415912+0000", 2024-09-18T07:42:39.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1580000000000005e-06 2024-09-18T07:42:39.455 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.455 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.455 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.455 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162535 7.12 7.1c115152 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-09-18T07:42:39.455 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:02.618377+0000", 2024-09-18T07:42:39.455 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 456.746847484, 2024-09-18T07:42:39.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.79781339699999998, 2024-09-18T07:42:39.456 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.456 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.456 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:42:39.456 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.456 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:42:39.456 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:39.456 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162535 2024-09-18T07:42:39.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.456 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618377+0000", 2024-09-18T07:42:39.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618377+0000", 2024-09-18T07:42:39.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618379+0000", 2024-09-18T07:42:39.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8509999999999999e-06 2024-09-18T07:42:39.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.458 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.458 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618384+0000", 2024-09-18T07:42:39.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9989999999999999e-06 2024-09-18T07:42:39.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.458 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.458 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618394+0000", 2024-09-18T07:42:39.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0045e-05 2024-09-18T07:42:39.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.458 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.458 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618403+0000", 2024-09-18T07:42:39.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8699999999999998e-06 2024-09-18T07:42:39.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.459 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618468+0000", 2024-09-18T07:42:39.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5273999999999999e-05 2024-09-18T07:42:39.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.459 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.618483+0000", 2024-09-18T07:42:39.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5141e-05 2024-09-18T07:42:39.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416187+0000", 2024-09-18T07:42:39.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.79770383899999997 2024-09-18T07:42:39.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:03.416190+0000", 2024-09-18T07:42:39.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3780000000000001e-06 2024-09-18T07:42:39.460 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.460 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.460 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.460 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109537 8.7 8:e50487d6:::benchmark_data_smithi137_101942_object4727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:42:39.460 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:42:39.461 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 297.308818067, 2024-09-18T07:42:39.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.185266763, 2024-09-18T07:42:39.461 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.461 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.461 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:39.461 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.461 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:42:39.461 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:42:39.461 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109537 2024-09-18T07:42:39.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.461 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:42:39.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:42:39.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056408+0000", 2024-09-18T07:42:39.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.562e-06 2024-09-18T07:42:39.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056415+0000", 2024-09-18T07:42:39.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9280000000000004e-06 2024-09-18T07:42:39.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056425+0000", 2024-09-18T07:42:39.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0220999999999999e-05 2024-09-18T07:42:39.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056434+0000", 2024-09-18T07:42:39.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7499999999999992e-06 2024-09-18T07:42:39.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.061190+0000", 2024-09-18T07:42:39.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047560620000000001 2024-09-18T07:42:39.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.061238+0000", 2024-09-18T07:42:39.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8730000000000003e-05 2024-09-18T07:42:39.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:39.465 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:39.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:39.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229082+0000", 2024-09-18T07:42:39.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.167843878 2024-09-18T07:42:39.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:39.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229243+0000", 2024-09-18T07:42:39.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000160426 2024-09-18T07:42:39.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:42:39.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229373+0000", 2024-09-18T07:42:39.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012994999999999999 2024-09-18T07:42:39.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:39.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236041+0000", 2024-09-18T07:42:39.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066680960000000001 2024-09-18T07:42:39.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:39.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236051+0000", 2024-09-18T07:42:39.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8309999999999996e-06 2024-09-18T07:42:39.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:39.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.240107+0000", 2024-09-18T07:42:39.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040564570000000003 2024-09-18T07:42:39.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:39.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.240152+0000", 2024-09-18T07:42:39.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4617000000000003e-05 2024-09-18T07:42:39.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.241673+0000", 2024-09-18T07:42:39.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001521255 2024-09-18T07:42:39.469 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.469 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.469 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.469 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109540 8.7 8:e7454470:::benchmark_data_smithi137_101942_object4730:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:42:39.469 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:42:39.469 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 297.299346077, 2024-09-18T07:42:39.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.186834677, 2024-09-18T07:42:39.469 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.469 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.469 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:39.469 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.470 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:42:39.470 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:42:39.470 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109540 2024-09-18T07:42:39.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.470 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.470 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:42:39.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.470 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:42:39.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065880+0000", 2024-09-18T07:42:39.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8279999999999999e-06 2024-09-18T07:42:39.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065886+0000", 2024-09-18T07:42:39.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.117e-06 2024-09-18T07:42:39.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065899+0000", 2024-09-18T07:42:39.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2809999999999999e-05 2024-09-18T07:42:39.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065904+0000", 2024-09-18T07:42:39.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0350000000000001e-06 2024-09-18T07:42:39.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.066028+0000", 2024-09-18T07:42:39.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012424199999999999 2024-09-18T07:42:39.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.066039+0000", 2024-09-18T07:42:39.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0441999999999999e-05 2024-09-18T07:42:39.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.233503+0000", 2024-09-18T07:42:39.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.167464622 2024-09-18T07:42:39.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:39.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.233624+0000", 2024-09-18T07:42:39.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012107 2024-09-18T07:42:39.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:42:39.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.233749+0000", 2024-09-18T07:42:39.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000124418 2024-09-18T07:42:39.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:39.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236078+0000", 2024-09-18T07:42:39.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002328746 2024-09-18T07:42:39.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:39.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.239968+0000", 2024-09-18T07:42:39.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038901220000000002 2024-09-18T07:42:39.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:39.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.250395+0000", 2024-09-18T07:42:39.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010427532999999999 2024-09-18T07:42:39.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:39.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.250537+0000", 2024-09-18T07:42:39.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014218100000000001 2024-09-18T07:42:39.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.252713+0000", 2024-09-18T07:42:39.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0021755110000000002 2024-09-18T07:42:39.476 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.476 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.476 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.476 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109545 8.7 8:ecb41cf4:::benchmark_data_smithi137_101942_object4735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:42:39.476 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:42:39.476 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 297.29094129100002, 2024-09-18T07:42:39.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1804624180000001, 2024-09-18T07:42:39.477 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.477 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.477 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:39.477 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.477 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:42:39.477 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:42:39.477 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109545 2024-09-18T07:42:39.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.477 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:42:39.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:42:39.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074285+0000", 2024-09-18T07:42:39.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1169999999999998e-06 2024-09-18T07:42:39.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074292+0000", 2024-09-18T07:42:39.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.153e-06 2024-09-18T07:42:39.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074305+0000", 2024-09-18T07:42:39.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2363e-05 2024-09-18T07:42:39.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074312+0000", 2024-09-18T07:42:39.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3409999999999997e-06 2024-09-18T07:42:39.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074508+0000", 2024-09-18T07:42:39.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000196443 2024-09-18T07:42:39.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074519+0000", 2024-09-18T07:42:39.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0159999999999999e-05 2024-09-18T07:42:39.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236329+0000", 2024-09-18T07:42:39.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1618104599999999 2024-09-18T07:42:39.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:39.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236416+0000", 2024-09-18T07:42:39.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6670999999999997e-05 2024-09-18T07:42:39.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:42:39.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236495+0000", 2024-09-18T07:42:39.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9256000000000001e-05 2024-09-18T07:42:39.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:39.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.241684+0000", 2024-09-18T07:42:39.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051890720000000003 2024-09-18T07:42:39.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:39.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.243812+0000", 2024-09-18T07:42:39.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002127912 2024-09-18T07:42:39.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:39.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.252772+0000", 2024-09-18T07:42:39.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.008960562 2024-09-18T07:42:39.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:39.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.252827+0000", 2024-09-18T07:42:39.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4500000000000003e-05 2024-09-18T07:42:39.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.254745+0000", 2024-09-18T07:42:39.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019184079999999999 2024-09-18T07:42:39.483 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.483 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.483 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.484 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109567 8.7 8:e247f5cd:::benchmark_data_smithi137_101942_object4757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:42:39.484 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:42:39.484 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 297.26184212800001, 2024-09-18T07:42:39.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.153246698, 2024-09-18T07:42:39.484 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:42:39.484 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:42:39.484 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:39.484 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:42:39.484 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:42:39.484 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:42:39.484 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109567 2024-09-18T07:42:39.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.485 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:42:39.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:42:39.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:42:39.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:42:39.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:42:39.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:42:39.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:42:39.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103384+0000", 2024-09-18T07:42:39.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.792e-06 2024-09-18T07:42:39.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:42:39.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103390+0000", 2024-09-18T07:42:39.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7139999999999998e-06 2024-09-18T07:42:39.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:42:39.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103401+0000", 2024-09-18T07:42:39.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1559000000000001e-05 2024-09-18T07:42:39.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:42:39.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103408+0000", 2024-09-18T07:42:39.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2110000000000001e-06 2024-09-18T07:42:39.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103464+0000", 2024-09-18T07:42:39.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6032999999999997e-05 2024-09-18T07:42:39.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:42:39.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103475+0000", 2024-09-18T07:42:39.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0380999999999999e-05 2024-09-18T07:42:39.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:42:39.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.238189+0000", 2024-09-18T07:42:39.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.134713723 2024-09-18T07:42:39.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:42:39.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.238319+0000", 2024-09-18T07:42:39.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013080700000000001 2024-09-18T07:42:39.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:42:39.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.238431+0000", 2024-09-18T07:42:39.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112126 2024-09-18T07:42:39.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:42:39.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.243830+0000", 2024-09-18T07:42:39.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053980390000000003 2024-09-18T07:42:39.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:39.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.243889+0000", 2024-09-18T07:42:39.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9024000000000003e-05 2024-09-18T07:42:39.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.490 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:39.490 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.254832+0000", 2024-09-18T07:42:39.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0109436 2024-09-18T07:42:39.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.490 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:42:39.490 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.255025+0000", 2024-09-18T07:42:39.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000192926 2024-09-18T07:42:39.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:42:39.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:42:39.490 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:42:39.490 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.256629+0000", 2024-09-18T07:42:39.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001603763 2024-09-18T07:42:39.491 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.491 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.491 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.491 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:42:39.491 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:42:39.491 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:42:39.491 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:42:39.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:42:39.566 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:39.588 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:39 smithi137 ceph-mon[29408]: 1.3 scrub starts 2024-09-18T07:42:39.588 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:39 smithi137 ceph-mon[29408]: 1.3 scrub ok 2024-09-18T07:42:39.588 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:39 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2892797803' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:42:39.588 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:39 smithi137 ceph-mon[39175]: 1.3 scrub starts 2024-09-18T07:42:39.589 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:39 smithi137 ceph-mon[39175]: 1.3 scrub ok 2024-09-18T07:42:39.589 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:39 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2892797803' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:42:39.601 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:39.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:39.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:39.703 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:39.738 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:39.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:39.840 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:39.881 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:39.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:39.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:39.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:39 smithi203 ceph-mon[32237]: 1.3 scrub starts 2024-09-18T07:42:39.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:39 smithi203 ceph-mon[32237]: 1.3 scrub ok 2024-09-18T07:42:39.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:39 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2892797803' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:42:39.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:40.018 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:40.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:40.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:40.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:40.155 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:40.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:40.257 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:40.290 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:40.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:40.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:40.392 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:40.425 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:40.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:40.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:40.527 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:40.568 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:40.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:40.670 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:40.708 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:40.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:40.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:40.810 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:40.845 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:40.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:40.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:40 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:42:40.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:40 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:42:40.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:40 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2892797803' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:42:40.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:40 smithi137 ceph-mon[29408]: osdmap e356: 8 total, 8 up, 8 in 2024-09-18T07:42:40.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:40 smithi137 ceph-mon[29408]: pgmap v623: 158 pgs: 158 active+clean; 1.2 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 983 op/s 2024-09-18T07:42:40.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:40 smithi137 ceph-mon[29408]: 3.7 scrub starts 2024-09-18T07:42:40.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:40 smithi137 ceph-mon[29408]: 3.7 scrub ok 2024-09-18T07:42:40.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:40 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:42:40.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:40 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:42:40.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:40 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2892797803' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:42:40.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:40 smithi137 ceph-mon[39175]: osdmap e356: 8 total, 8 up, 8 in 2024-09-18T07:42:40.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:40 smithi137 ceph-mon[39175]: pgmap v623: 158 pgs: 158 active+clean; 1.2 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 983 op/s 2024-09-18T07:42:40.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:40 smithi137 ceph-mon[39175]: 3.7 scrub starts 2024-09-18T07:42:40.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:40 smithi137 ceph-mon[39175]: 3.7 scrub ok 2024-09-18T07:42:40.946 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:40.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:40 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:42:40.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:40 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:42:40.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:40 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2892797803' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:42:40.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:40 smithi203 ceph-mon[32237]: osdmap e356: 8 total, 8 up, 8 in 2024-09-18T07:42:40.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:40 smithi203 ceph-mon[32237]: pgmap v623: 158 pgs: 158 active+clean; 1.2 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 983 op/s 2024-09-18T07:42:40.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:40 smithi203 ceph-mon[32237]: 3.7 scrub starts 2024-09-18T07:42:40.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:40 smithi203 ceph-mon[32237]: 3.7 scrub ok 2024-09-18T07:42:40.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:40.981 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:40.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:41.082 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:41.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:41.114 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:41.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:41.215 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:41.244 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:41.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:41.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:41.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:41.383 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:41.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:41.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:41.485 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:41.518 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:41.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:41.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:41.566 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T07:42:41.620 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:41.662 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:41.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:41.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:41.764 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:41.799 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:41.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:41.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:41.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:41.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:41 smithi137 ceph-mon[29408]: 6.1e scrub starts 2024-09-18T07:42:41.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:41 smithi137 ceph-mon[29408]: 6.1e scrub ok 2024-09-18T07:42:41.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:41 smithi137 ceph-mon[39175]: 6.1e scrub starts 2024-09-18T07:42:41.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:41 smithi137 ceph-mon[39175]: 6.1e scrub ok 2024-09-18T07:42:41.950 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:41.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:41.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:41 smithi203 ceph-mon[32237]: 6.1e scrub starts 2024-09-18T07:42:41.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:41 smithi203 ceph-mon[32237]: 6.1e scrub ok 2024-09-18T07:42:42.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:42.086 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:42.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:42.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:42.188 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:42.229 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:42.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:42.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:42.236 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:42:42.332 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:42.372 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:42.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:42.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:42.474 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:42.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:42.516 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:42.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:42.617 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:42.653 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:42.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:42.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[29408]: 5.d scrub starts 2024-09-18T07:42:42.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[29408]: 5.d scrub ok 2024-09-18T07:42:42.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[29408]: 5.13 scrub starts 2024-09-18T07:42:42.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[29408]: 5.13 scrub ok 2024-09-18T07:42:42.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[29408]: 3.1d scrub starts 2024-09-18T07:42:42.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[29408]: 3.1d scrub ok 2024-09-18T07:42:42.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[29408]: pgmap v624: 158 pgs: 158 active+clean; 1.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-18T07:42:42.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:42:42.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:42:42.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[39175]: 5.d scrub starts 2024-09-18T07:42:42.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[39175]: 5.d scrub ok 2024-09-18T07:42:42.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[39175]: 5.13 scrub starts 2024-09-18T07:42:42.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[39175]: 5.13 scrub ok 2024-09-18T07:42:42.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[39175]: 3.1d scrub starts 2024-09-18T07:42:42.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[39175]: 3.1d scrub ok 2024-09-18T07:42:42.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[39175]: pgmap v624: 158 pgs: 158 active+clean; 1.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-18T07:42:42.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:42:42.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:42 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:42:42.690 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:42 smithi203 ceph-mon[32237]: 5.d scrub starts 2024-09-18T07:42:42.690 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:42 smithi203 ceph-mon[32237]: 5.d scrub ok 2024-09-18T07:42:42.690 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:42 smithi203 ceph-mon[32237]: 5.13 scrub starts 2024-09-18T07:42:42.690 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:42 smithi203 ceph-mon[32237]: 5.13 scrub ok 2024-09-18T07:42:42.690 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:42 smithi203 ceph-mon[32237]: 3.1d scrub starts 2024-09-18T07:42:42.690 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:42 smithi203 ceph-mon[32237]: 3.1d scrub ok 2024-09-18T07:42:42.691 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:42 smithi203 ceph-mon[32237]: pgmap v624: 158 pgs: 158 active+clean; 1.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-18T07:42:42.691 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:42 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:42:42.691 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:42 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:42:42.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:42.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:42.794 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:42.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:42.858 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:42:42.859 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:42:42.859 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:42.859 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:253287 9.4 9:2c7ca95a:::benchmark_data_smithi137_126072_object120787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-18T07:42:42.859 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:42.853161+0000", 2024-09-18T07:42:42.859 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0016662109999999999, 2024-09-18T07:42:42.859 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0016951780000000001, 2024-09-18T07:42:42.859 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:42.859 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:42.859 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:42:42.859 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:42.859 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:42:42.859 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:42:42.859 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 253287 2024-09-18T07:42:42.860 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:42.860 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:42.860 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:42.860 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:42.860 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:42.853161+0000", 2024-09-18T07:42:42.860 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:42.860 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:42.860 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:42.860 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:42.860 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:42.853161+0000", 2024-09-18T07:42:42.860 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:42.860 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:42.860 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:42.861 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:42.861 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:42.853163+0000", 2024-09-18T07:42:42.861 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0470000000000001e-06 2024-09-18T07:42:42.861 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:42.861 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:42.861 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:42.861 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:42.853172+0000", 2024-09-18T07:42:42.861 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1439999999999992e-06 2024-09-18T07:42:42.861 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:42.861 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:42.861 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:42.861 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:42.853189+0000", 2024-09-18T07:42:42.861 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6884e-05 2024-09-18T07:42:42.861 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:42.862 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:42.862 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:42.862 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:42.853198+0000", 2024-09-18T07:42:42.862 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8049999999999996e-06 2024-09-18T07:42:42.862 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:42.862 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:42.862 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:42.862 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:42.853403+0000", 2024-09-18T07:42:42.862 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00020541900000000001 2024-09-18T07:42:42.862 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:42.862 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:42.862 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:42.862 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:42.853629+0000", 2024-09-18T07:42:42.862 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00022552800000000001 2024-09-18T07:42:42.863 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:42.863 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:42.863 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:42:42.863 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:42.853745+0000", 2024-09-18T07:42:42.863 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000116767 2024-09-18T07:42:42.863 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:42.863 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:42.863 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:42.863 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:42.863 INFO:teuthology.orchestra.run.smithi203.stdout: ], 2024-09-18T07:42:42.863 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 1 2024-09-18T07:42:42.863 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:42:42.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:42.899 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:42.930 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:42.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:42.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:43.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:43.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:43.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:43.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:43.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:43.207 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:43.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:43.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:43.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:43.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:43.338 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:43.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:43.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:43.505 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:43.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:43.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:43.606 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:43.642 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:43.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:43.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:43.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:43 smithi137 ceph-mon[39175]: 6.0 scrub starts 2024-09-18T07:42:43.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:43 smithi137 ceph-mon[39175]: 6.0 scrub ok 2024-09-18T07:42:43.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:43 smithi137 ceph-mon[39175]: 3.19 scrub starts 2024-09-18T07:42:43.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:43 smithi137 ceph-mon[39175]: 3.19 scrub ok 2024-09-18T07:42:43.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:43 smithi137 ceph-mon[29408]: 6.0 scrub starts 2024-09-18T07:42:43.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:43 smithi137 ceph-mon[29408]: 6.0 scrub ok 2024-09-18T07:42:43.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:43 smithi137 ceph-mon[29408]: 3.19 scrub starts 2024-09-18T07:42:43.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:43 smithi137 ceph-mon[29408]: 3.19 scrub ok 2024-09-18T07:42:43.745 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:43.782 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:43.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:43.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:43.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:43.921 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:43.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:43.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:43 smithi203 ceph-mon[32237]: 6.0 scrub starts 2024-09-18T07:42:43.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:43 smithi203 ceph-mon[32237]: 6.0 scrub ok 2024-09-18T07:42:43.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:43 smithi203 ceph-mon[32237]: 3.19 scrub starts 2024-09-18T07:42:43.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:43 smithi203 ceph-mon[32237]: 3.19 scrub ok 2024-09-18T07:42:44.022 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:44.031 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:42:44.033 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:42:44.043 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:44.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:44.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:44.049 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:42:44.050 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:42:44.067 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:42:44.068 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:42:44.097 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:42:44.097 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:42:44.129 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:42:44.129 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:42:44.145 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:44.165 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:42:44.166 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:42:44.177 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:44.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:44.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:44.193 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:42:44.194 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:42:44.221 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:42:44.221 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:42:44.249 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:42:44.249 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:42:44.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:44.301 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:44.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:44.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:44.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:44.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:44.432 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:44.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:44.533 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:44.565 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:44.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:44.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:44.667 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:44.693 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:44.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:44.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:44.795 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:44.835 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:44.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:44.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:44.873 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:42:44.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[29408]: 5.a scrub starts 2024-09-18T07:42:44.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[29408]: 5.a scrub ok 2024-09-18T07:42:44.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[29408]: 5.5 scrub starts 2024-09-18T07:42:44.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[29408]: 5.5 scrub ok 2024-09-18T07:42:44.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[29408]: 4.e scrub starts 2024-09-18T07:42:44.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[29408]: 4.e scrub ok 2024-09-18T07:42:44.911 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[29408]: pgmap v625: 158 pgs: 158 active+clean; 792 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-09-18T07:42:44.911 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[29408]: from='client.35927 v1:172.21.15.137:0/3068980924' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:42:44.911 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2280610824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:44.911 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[29408]: 4.a scrub starts 2024-09-18T07:42:44.911 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[29408]: 4.a scrub ok 2024-09-18T07:42:44.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[39175]: 5.a scrub starts 2024-09-18T07:42:44.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[39175]: 5.a scrub ok 2024-09-18T07:42:44.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[39175]: 5.5 scrub starts 2024-09-18T07:42:44.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[39175]: 5.5 scrub ok 2024-09-18T07:42:44.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[39175]: 4.e scrub starts 2024-09-18T07:42:44.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[39175]: 4.e scrub ok 2024-09-18T07:42:44.912 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[39175]: pgmap v625: 158 pgs: 158 active+clean; 792 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-09-18T07:42:44.912 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[39175]: from='client.35927 v1:172.21.15.137:0/3068980924' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:42:44.912 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2280610824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:44.912 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[39175]: 4.a scrub starts 2024-09-18T07:42:44.912 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:44 smithi137 ceph-mon[39175]: 4.a scrub ok 2024-09-18T07:42:44.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:44.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:44 smithi203 ceph-mon[32237]: 5.a scrub starts 2024-09-18T07:42:44.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:44 smithi203 ceph-mon[32237]: 5.a scrub ok 2024-09-18T07:42:44.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:44 smithi203 ceph-mon[32237]: 5.5 scrub starts 2024-09-18T07:42:44.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:44 smithi203 ceph-mon[32237]: 5.5 scrub ok 2024-09-18T07:42:44.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:44 smithi203 ceph-mon[32237]: 4.e scrub starts 2024-09-18T07:42:44.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:44 smithi203 ceph-mon[32237]: 4.e scrub ok 2024-09-18T07:42:44.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:44 smithi203 ceph-mon[32237]: pgmap v625: 158 pgs: 158 active+clean; 792 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-09-18T07:42:44.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:44 smithi203 ceph-mon[32237]: from='client.35927 v1:172.21.15.137:0/3068980924' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:42:44.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:44 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2280610824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:44.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:44 smithi203 ceph-mon[32237]: 4.a scrub starts 2024-09-18T07:42:44.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:44 smithi203 ceph-mon[32237]: 4.a scrub ok 2024-09-18T07:42:44.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:44.982 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:44.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:45.000 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":625,"stamp":"2024-09-18T07:42:43.688028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":830413363,"num_objects":12862,"num_object_clones":0,"num_object_copies":38586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12862,"num_whiteouts":0,"num_read":1599,"num_read_kb":1351,"num_write":253055,"num_write_kb":8479854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":5625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81181,"ondisk_log_size":81181,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6009268,"kb_used_data":2588776,"kb_used_omap":12,"kb_used_meta":3420211,"kb_avail":743951948,"statfs":{"total":767960285184,"available":761806794752,"internally_reserved":0,"allocated":2650906624,"data_stored":2647953833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":3502296624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,10,11,10,6,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-814612480,"num_objects":-12430,"num_object_clones":0,"num_object_copies":-37290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12430,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12430,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002215"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":635,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.480498+0000","last_change":"2024-09-18T07:41:15.846355+0000","last_active":"2024-09-18T07:42:39.480498+0000","last_peered":"2024-09-18T07:42:39.480498+0000","last_clean":"2024-09-18T07:42:39.480498+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:42:39.480498+0000","last_undegraded":"2024-09-18T07:42:39.480498+0000","last_fullsized":"2024-09-18T07:42:39.480498+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:15.846257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:15.846257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"323'15","reported_seq":710,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.488422+0000","last_change":"2024-09-18T07:40:17.270061+0000","last_active":"2024-09-18T07:42:39.488422+0000","last_peered":"2024-09-18T07:42:39.488422+0000","last_clean":"2024-09-18T07:42:39.488422+0000","last_became_active":"2024-09-18T07:37:43.061125+0000","last_became_peered":"2024-09-18T07:37:43.061125+0000","last_unstale":"2024-09-18T07:42:39.488422+0000","last_undegraded":"2024-09-18T07:42:39.488422+0000","last_fullsized":"2024-09-18T07:42:39.488422+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:17.269977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:17.269977+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:43.934455+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":578,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478830+0000","last_change":"2024-09-18T07:41:58.378943+0000","last_active":"2024-09-18T07:42:39.478830+0000","last_peered":"2024-09-18T07:42:39.478830+0000","last_clean":"2024-09-18T07:42:39.478830+0000","last_became_active":"2024-09-18T07:36:13.128805+0000","last_became_peered":"2024-09-18T07:36:13.128805+0000","last_unstale":"2024-09-18T07:42:39.478830+0000","last_undegraded":"2024-09-18T07:42:39.478830+0000","last_fullsized":"2024-09-18T07:42:39.478830+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:58.378571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:58.378571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:23.350846+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":610,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.488762+0000","last_change":"2024-09-18T07:40:47.287019+0000","last_active":"2024-09-18T07:42:39.488762+0000","last_peered":"2024-09-18T07:42:39.488762+0000","last_clean":"2024-09-18T07:42:39.488762+0000","last_became_active":"2024-09-18T07:31:02.342213+0000","last_became_peered":"2024-09-18T07:31:02.342213+0000","last_unstale":"2024-09-18T07:42:39.488762+0000","last_undegraded":"2024-09-18T07:42:39.488762+0000","last_fullsized":"2024-09-18T07:42:39.488762+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:47.286927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.286927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":529,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477647+0000","last_change":"2024-09-18T07:41:12.872510+0000","last_active":"2024-09-18T07:42:39.477647+0000","last_peered":"2024-09-18T07:42:39.477647+0000","last_clean":"2024-09-18T07:42:39.477647+0000","last_became_active":"2024-09-18T07:37:43.056380+0000","last_became_peered":"2024-09-18T07:37:43.056380+0000","last_unstale":"2024-09-18T07:42:39.477647+0000","last_undegraded":"2024-09-18T07:42:39.477647+0000","last_fullsized":"2024-09-18T07:42:39.477647+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:12.872387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:12.872387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:41.658070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":636,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.487432+0000","last_change":"2024-09-18T07:40:57.319270+0000","last_active":"2024-09-18T07:42:39.487432+0000","last_peered":"2024-09-18T07:42:39.487432+0000","last_clean":"2024-09-18T07:42:39.487432+0000","last_became_active":"2024-09-18T07:37:43.066508+0000","last_became_peered":"2024-09-18T07:37:43.066508+0000","last_unstale":"2024-09-18T07:42:39.487432+0000","last_undegraded":"2024-09-18T07:42:39.487432+0000","last_fullsized":"2024-09-18T07:42:39.487432+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:57.319204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:57.319204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:22.438323+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"323'27","reported_seq":180,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.476470+0000","last_change":"2024-09-18T07:40:20.436791+0000","last_active":"2024-09-18T07:42:39.476470+0000","last_peered":"2024-09-18T07:42:39.476470+0000","last_clean":"2024-09-18T07:42:39.476470+0000","last_became_active":"2024-09-18T07:40:19.064755+0000","last_became_peered":"2024-09-18T07:40:19.064755+0000","last_unstale":"2024-09-18T07:42:39.476470+0000","last_undegraded":"2024-09-18T07:42:39.476470+0000","last_fullsized":"2024-09-18T07:42:39.476470+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-18T07:40:20.436702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:20.436702+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:46.054779+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":644,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.488411+0000","last_change":"2024-09-18T07:41:51.001121+0000","last_active":"2024-09-18T07:42:39.488411+0000","last_peered":"2024-09-18T07:42:39.488411+0000","last_clean":"2024-09-18T07:42:39.488411+0000","last_became_active":"2024-09-18T07:37:43.052346+0000","last_became_peered":"2024-09-18T07:37:43.052346+0000","last_unstale":"2024-09-18T07:42:39.488411+0000","last_undegraded":"2024-09-18T07:42:39.488411+0000","last_fullsized":"2024-09-18T07:42:39.488411+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:51.001046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:51.001046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:55.981852+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":642,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:41.502287+0000","last_change":"2024-09-18T07:42:41.502287+0000","last_active":"2024-09-18T07:42:41.502287+0000","last_peered":"2024-09-18T07:42:41.502287+0000","last_clean":"2024-09-18T07:42:41.502287+0000","last_became_active":"2024-09-18T07:37:43.080314+0000","last_became_peered":"2024-09-18T07:37:43.080314+0000","last_unstale":"2024-09-18T07:42:41.502287+0000","last_undegraded":"2024-09-18T07:42:41.502287+0000","last_fullsized":"2024-09-18T07:42:41.502287+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:41.502137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:41.502137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:50.989696+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"323'15","reported_seq":651,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.489050+0000","last_change":"2024-09-18T07:41:56.021202+0000","last_active":"2024-09-18T07:42:39.489050+0000","last_peered":"2024-09-18T07:42:39.489050+0000","last_clean":"2024-09-18T07:42:39.489050+0000","last_became_active":"2024-09-18T07:30:58.316556+0000","last_became_peered":"2024-09-18T07:30:58.316556+0000","last_unstale":"2024-09-18T07:42:39.489050+0000","last_undegraded":"2024-09-18T07:42:39.489050+0000","last_fullsized":"2024-09-18T07:42:39.489050+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:56.021118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:56.021118+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:16.099361+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":634,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478457+0000","last_change":"2024-09-18T07:42:28.519199+0000","last_active":"2024-09-18T07:42:39.478457+0000","last_peered":"2024-09-18T07:42:39.478457+0000","last_clean":"2024-09-18T07:42:39.478457+0000","last_became_active":"2024-09-18T07:35:01.413443+0000","last_became_peered":"2024-09-18T07:35:01.413443+0000","last_unstale":"2024-09-18T07:42:39.478457+0000","last_undegraded":"2024-09-18T07:42:39.478457+0000","last_fullsized":"2024-09-18T07:42:39.478457+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:28.518968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:28.518968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:46.526654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":620,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478462+0000","last_change":"2024-09-18T07:42:00.028623+0000","last_active":"2024-09-18T07:42:39.478462+0000","last_peered":"2024-09-18T07:42:39.478462+0000","last_clean":"2024-09-18T07:42:39.478462+0000","last_became_active":"2024-09-18T07:35:01.410284+0000","last_became_peered":"2024-09-18T07:35:01.410284+0000","last_unstale":"2024-09-18T07:42:39.478462+0000","last_undegraded":"2024-09-18T07:42:39.478462+0000","last_fullsized":"2024-09-18T07:42:39.478462+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:00.028559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:00.028559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:26.482816+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":620,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478457+0000","last_change":"2024-09-18T07:41:57.358913+0000","last_active":"2024-09-18T07:42:39.478457+0000","last_peered":"2024-09-18T07:42:39.478457+0000","last_clean":"2024-09-18T07:42:39.478457+0000","last_became_active":"2024-09-18T07:30:56.282445+0000","last_became_peered":"2024-09-18T07:30:56.282445+0000","last_unstale":"2024-09-18T07:42:39.478457+0000","last_undegraded":"2024-09-18T07:42:39.478457+0000","last_fullsized":"2024-09-18T07:42:39.478457+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:57.358853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:57.358853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:12.134323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"323'7","reported_seq":644,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.490136+0000","last_change":"2024-09-18T07:39:26.028935+0000","last_active":"2024-09-18T07:42:39.490136+0000","last_peered":"2024-09-18T07:42:39.490136+0000","last_clean":"2024-09-18T07:42:39.490136+0000","last_became_active":"2024-09-18T07:36:02.526212+0000","last_became_peered":"2024-09-18T07:36:02.526212+0000","last_unstale":"2024-09-18T07:42:39.490136+0000","last_undegraded":"2024-09-18T07:42:39.490136+0000","last_fullsized":"2024-09-18T07:42:39.490136+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-18T07:39:26.028884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:26.028884+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:39.929629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":403,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.471988+0000","last_change":"2024-09-18T07:42:28.855544+0000","last_active":"2024-09-18T07:42:39.471988+0000","last_peered":"2024-09-18T07:42:39.471988+0000","last_clean":"2024-09-18T07:42:39.471988+0000","last_became_active":"2024-09-18T07:35:01.398605+0000","last_became_peered":"2024-09-18T07:35:01.398605+0000","last_unstale":"2024-09-18T07:42:39.471988+0000","last_undegraded":"2024-09-18T07:42:39.471988+0000","last_fullsized":"2024-09-18T07:42:39.471988+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:28.855481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:28.855481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:37.908975+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":632,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478910+0000","last_change":"2024-09-18T07:40:50.367758+0000","last_active":"2024-09-18T07:42:39.478910+0000","last_peered":"2024-09-18T07:42:39.478910+0000","last_clean":"2024-09-18T07:42:39.478910+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:42:39.478910+0000","last_undegraded":"2024-09-18T07:42:39.478910+0000","last_fullsized":"2024-09-18T07:42:39.478910+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:50.367694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.367694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":641,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:40.432996+0000","last_change":"2024-09-18T07:42:40.432996+0000","last_active":"2024-09-18T07:42:40.432996+0000","last_peered":"2024-09-18T07:42:40.432996+0000","last_clean":"2024-09-18T07:42:40.432996+0000","last_became_active":"2024-09-18T07:37:43.067568+0000","last_became_peered":"2024-09-18T07:37:43.067568+0000","last_unstale":"2024-09-18T07:42:40.432996+0000","last_undegraded":"2024-09-18T07:42:40.432996+0000","last_fullsized":"2024-09-18T07:42:40.432996+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:40.432906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:40.432906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:01.732070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":640,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.479787+0000","last_change":"2024-09-18T07:42:04.117872+0000","last_active":"2024-09-18T07:42:39.479787+0000","last_peered":"2024-09-18T07:42:39.479787+0000","last_clean":"2024-09-18T07:42:39.479787+0000","last_became_active":"2024-09-18T07:35:01.414088+0000","last_became_peered":"2024-09-18T07:35:01.414088+0000","last_unstale":"2024-09-18T07:42:39.479787+0000","last_undegraded":"2024-09-18T07:42:39.479787+0000","last_fullsized":"2024-09-18T07:42:39.479787+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:04.117653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.117653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:22.791068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"323'27","reported_seq":723,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477042+0000","last_change":"2024-09-18T07:41:48.128836+0000","last_active":"2024-09-18T07:42:39.477042+0000","last_peered":"2024-09-18T07:42:39.477042+0000","last_clean":"2024-09-18T07:42:39.477042+0000","last_became_active":"2024-09-18T07:37:28.002858+0000","last_became_peered":"2024-09-18T07:37:28.002858+0000","last_unstale":"2024-09-18T07:42:39.477042+0000","last_undegraded":"2024-09-18T07:42:39.477042+0000","last_fullsized":"2024-09-18T07:42:39.477042+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:41:48.128756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.128756+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:01.870808+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":313,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.490576+0000","last_change":"2024-09-18T07:41:26.922547+0000","last_active":"2024-09-18T07:42:39.490576+0000","last_peered":"2024-09-18T07:42:39.490576+0000","last_clean":"2024-09-18T07:42:39.490576+0000","last_became_active":"2024-09-18T07:36:13.116416+0000","last_became_peered":"2024-09-18T07:36:13.116416+0000","last_unstale":"2024-09-18T07:42:39.490576+0000","last_undegraded":"2024-09-18T07:42:39.490576+0000","last_fullsized":"2024-09-18T07:42:39.490576+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:26.922506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:26.922506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:55.427286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":488,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477922+0000","last_change":"2024-09-18T07:40:19.070769+0000","last_active":"2024-09-18T07:42:39.477922+0000","last_peered":"2024-09-18T07:42:39.477922+0000","last_clean":"2024-09-18T07:42:39.477922+0000","last_became_active":"2024-09-18T07:40:19.070463+0000","last_became_peered":"2024-09-18T07:40:19.070463+0000","last_unstale":"2024-09-18T07:42:39.477922+0000","last_undegraded":"2024-09-18T07:42:39.477922+0000","last_fullsized":"2024-09-18T07:42:39.477922+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:11.831891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:11.831891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:43.316131+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":638,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.489186+0000","last_change":"2024-09-18T07:42:31.324040+0000","last_active":"2024-09-18T07:42:39.489186+0000","last_peered":"2024-09-18T07:42:39.489186+0000","last_clean":"2024-09-18T07:42:39.489186+0000","last_became_active":"2024-09-18T07:30:56.289206+0000","last_became_peered":"2024-09-18T07:30:56.289206+0000","last_unstale":"2024-09-18T07:42:39.489186+0000","last_undegraded":"2024-09-18T07:42:39.489186+0000","last_fullsized":"2024-09-18T07:42:39.489186+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:31.323963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:31.323963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:32.270399+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"323'18","reported_seq":293,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477232+0000","last_change":"2024-09-18T07:41:59.149841+0000","last_active":"2024-09-18T07:42:39.477232+0000","last_peered":"2024-09-18T07:42:39.477232+0000","last_clean":"2024-09-18T07:42:39.477232+0000","last_became_active":"2024-09-18T07:37:27.981705+0000","last_became_peered":"2024-09-18T07:37:27.981705+0000","last_unstale":"2024-09-18T07:42:39.477232+0000","last_undegraded":"2024-09-18T07:42:39.477232+0000","last_fullsized":"2024-09-18T07:42:39.477232+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:41:59.149629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:59.149629+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:25.055296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":606,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.490198+0000","last_change":"2024-09-18T07:39:37.930497+0000","last_active":"2024-09-18T07:42:39.490198+0000","last_peered":"2024-09-18T07:42:39.490198+0000","last_clean":"2024-09-18T07:42:39.490198+0000","last_became_active":"2024-09-18T07:31:00.333352+0000","last_became_peered":"2024-09-18T07:31:00.333352+0000","last_unstale":"2024-09-18T07:42:39.490198+0000","last_undegraded":"2024-09-18T07:42:39.490198+0000","last_fullsized":"2024-09-18T07:42:39.490198+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.930435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.930435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:13.036846+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":407,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.472495+0000","last_change":"2024-09-18T07:40:10.644587+0000","last_active":"2024-09-18T07:42:39.472495+0000","last_peered":"2024-09-18T07:42:39.472495+0000","last_clean":"2024-09-18T07:42:39.472495+0000","last_became_active":"2024-09-18T07:35:01.413110+0000","last_became_peered":"2024-09-18T07:35:01.413110+0000","last_unstale":"2024-09-18T07:42:39.472495+0000","last_undegraded":"2024-09-18T07:42:39.472495+0000","last_fullsized":"2024-09-18T07:42:39.472495+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:40:10.644461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:10.644461+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:11.785210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":330,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:42.333035+0000","last_change":"2024-09-18T07:42:42.333035+0000","last_active":"2024-09-18T07:42:42.333035+0000","last_peered":"2024-09-18T07:42:42.333035+0000","last_clean":"2024-09-18T07:42:42.333035+0000","last_became_active":"2024-09-18T07:36:13.116295+0000","last_became_peered":"2024-09-18T07:36:13.116295+0000","last_unstale":"2024-09-18T07:42:42.333035+0000","last_undegraded":"2024-09-18T07:42:42.333035+0000","last_fullsized":"2024-09-18T07:42:42.333035+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:42:42.332950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:42.332950+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:47.146448+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"323'20","reported_seq":699,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.480418+0000","last_change":"2024-09-18T07:42:27.188820+0000","last_active":"2024-09-18T07:42:39.480418+0000","last_peered":"2024-09-18T07:42:39.480418+0000","last_clean":"2024-09-18T07:42:39.480418+0000","last_became_active":"2024-09-18T07:35:01.410926+0000","last_became_peered":"2024-09-18T07:35:01.410926+0000","last_unstale":"2024-09-18T07:42:39.480418+0000","last_undegraded":"2024-09-18T07:42:39.480418+0000","last_fullsized":"2024-09-18T07:42:39.480418+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:42:27.188530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:27.188530+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:44.776667+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"240'12","reported_seq":326,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478046+0000","last_change":"2024-09-18T07:40:43.112526+0000","last_active":"2024-09-18T07:42:39.478046+0000","last_peered":"2024-09-18T07:42:39.478046+0000","last_clean":"2024-09-18T07:42:39.478046+0000","last_became_active":"2024-09-18T07:37:43.060715+0000","last_became_peered":"2024-09-18T07:37:43.060715+0000","last_unstale":"2024-09-18T07:42:39.478046+0000","last_undegraded":"2024-09-18T07:42:39.478046+0000","last_fullsized":"2024-09-18T07:42:39.478046+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:40:43.112459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:43.112459+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:21.098107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"356'15791","reported_seq":16046,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:42.349891+0000","last_change":"2024-09-18T07:42:00.674258+0000","last_active":"2024-09-18T07:42:42.349891+0000","last_peered":"2024-09-18T07:42:42.349891+0000","last_clean":"2024-09-18T07:42:42.349891+0000","last_became_active":"2024-09-18T07:39:29.700477+0000","last_became_peered":"2024-09-18T07:39:29.700477+0000","last_unstale":"2024-09-18T07:42:42.349891+0000","last_undegraded":"2024-09-18T07:42:42.349891+0000","last_fullsized":"2024-09-18T07:42:42.349891+0000","mapping_epoch":289,"log_start":"338'11200","ondisk_log_start":"338'11200","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'12643","last_scrub_stamp":"2024-09-18T07:42:00.674143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:42:00.674143+0000","objects_scrubbed":3950,"log_size":4591,"log_dups_size":0,"ondisk_log_size":4591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:17.584126+0000","scrub_duration":1266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49610752,"num_objects":757,"num_object_clones":0,"num_object_copies":2271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15791,"num_write_kb":529536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":199,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477565+0000","last_change":"2024-09-18T07:42:29.515462+0000","last_active":"2024-09-18T07:42:39.477565+0000","last_peered":"2024-09-18T07:42:39.477565+0000","last_clean":"2024-09-18T07:42:39.477565+0000","last_became_active":"2024-09-18T07:39:04.962047+0000","last_became_peered":"2024-09-18T07:39:04.962047+0000","last_unstale":"2024-09-18T07:42:39.477565+0000","last_undegraded":"2024-09-18T07:42:39.477565+0000","last_fullsized":"2024-09-18T07:42:39.477565+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:42:29.515416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:42:29.515416+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:48.241376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"323'27","reported_seq":384,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.487075+0000","last_change":"2024-09-18T07:42:33.391873+0000","last_active":"2024-09-18T07:42:39.487075+0000","last_peered":"2024-09-18T07:42:39.487075+0000","last_clean":"2024-09-18T07:42:39.487075+0000","last_became_active":"2024-09-18T07:36:13.119098+0000","last_became_peered":"2024-09-18T07:36:13.119098+0000","last_unstale":"2024-09-18T07:42:39.487075+0000","last_undegraded":"2024-09-18T07:42:39.487075+0000","last_fullsized":"2024-09-18T07:42:39.487075+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:42:33.391715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:33.391715+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:39.082277+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":652,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.485591+0000","last_change":"2024-09-18T07:39:29.049261+0000","last_active":"2024-09-18T07:42:39.485591+0000","last_peered":"2024-09-18T07:42:39.485591+0000","last_clean":"2024-09-18T07:42:39.485591+0000","last_became_active":"2024-09-18T07:35:01.407339+0000","last_became_peered":"2024-09-18T07:35:01.407339+0000","last_unstale":"2024-09-18T07:42:39.485591+0000","last_undegraded":"2024-09-18T07:42:39.485591+0000","last_fullsized":"2024-09-18T07:42:39.485591+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.049209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.049209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:20.464343+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":789,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.492989+0000","last_change":"2024-09-18T07:41:27.965286+0000","last_active":"2024-09-18T07:42:39.492989+0000","last_peered":"2024-09-18T07:42:39.492989+0000","last_clean":"2024-09-18T07:42:39.492989+0000","last_became_active":"2024-09-18T07:31:00.332620+0000","last_became_peered":"2024-09-18T07:31:00.332620+0000","last_unstale":"2024-09-18T07:42:39.492989+0000","last_undegraded":"2024-09-18T07:42:39.492989+0000","last_fullsized":"2024-09-18T07:42:39.492989+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:41:27.965205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.965205+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:47.540373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":617,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477357+0000","last_change":"2024-09-18T07:42:06.507229+0000","last_active":"2024-09-18T07:42:39.477357+0000","last_peered":"2024-09-18T07:42:39.477357+0000","last_clean":"2024-09-18T07:42:39.477357+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:42:39.477357+0000","last_undegraded":"2024-09-18T07:42:39.477357+0000","last_fullsized":"2024-09-18T07:42:39.477357+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.507166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.507166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:10.596707+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"356'15890","reported_seq":16116,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:41.665681+0000","last_change":"2024-09-18T07:40:47.444123+0000","last_active":"2024-09-18T07:42:41.665681+0000","last_peered":"2024-09-18T07:42:41.665681+0000","last_clean":"2024-09-18T07:42:41.665681+0000","last_became_active":"2024-09-18T07:39:29.704590+0000","last_became_peered":"2024-09-18T07:39:29.704590+0000","last_unstale":"2024-09-18T07:42:41.665681+0000","last_undegraded":"2024-09-18T07:42:41.665681+0000","last_fullsized":"2024-09-18T07:42:41.665681+0000","mapping_epoch":289,"log_start":"338'11400","ondisk_log_start":"338'11400","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'7163","last_scrub_stamp":"2024-09-18T07:40:47.443974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.443974+0000","objects_scrubbed":7103,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:13.769013+0000","scrub_duration":1939,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52101144,"num_objects":796,"num_object_clones":0,"num_object_copies":2388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":796,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15890,"num_write_kb":533889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":199,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.490997+0000","last_change":"2024-09-18T07:41:53.004272+0000","last_active":"2024-09-18T07:42:39.490997+0000","last_peered":"2024-09-18T07:42:39.490997+0000","last_clean":"2024-09-18T07:42:39.490997+0000","last_became_active":"2024-09-18T07:39:04.955645+0000","last_became_peered":"2024-09-18T07:39:04.955645+0000","last_unstale":"2024-09-18T07:42:39.490997+0000","last_undegraded":"2024-09-18T07:42:39.490997+0000","last_fullsized":"2024-09-18T07:42:39.490997+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:41:53.004186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:41:53.004186+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:16.639727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"323'30","reported_seq":726,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.485117+0000","last_change":"2024-09-18T07:39:00.919075+0000","last_active":"2024-09-18T07:42:39.485117+0000","last_peered":"2024-09-18T07:42:39.485117+0000","last_clean":"2024-09-18T07:42:39.485117+0000","last_became_active":"2024-09-18T07:35:01.418909+0000","last_became_peered":"2024-09-18T07:35:01.418909+0000","last_unstale":"2024-09-18T07:42:39.485117+0000","last_undegraded":"2024-09-18T07:42:39.485117+0000","last_fullsized":"2024-09-18T07:42:39.485117+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:39:00.918860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.918860+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:39.501904+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":366,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.814649+0000","last_change":"2024-09-18T07:42:39.814649+0000","last_active":"2024-09-18T07:42:39.814649+0000","last_peered":"2024-09-18T07:42:39.814649+0000","last_clean":"2024-09-18T07:42:39.814649+0000","last_became_active":"2024-09-18T07:40:19.068296+0000","last_became_peered":"2024-09-18T07:40:19.068296+0000","last_unstale":"2024-09-18T07:42:39.814649+0000","last_undegraded":"2024-09-18T07:42:39.814649+0000","last_fullsized":"2024-09-18T07:42:39.814649+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:39.814582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:39.814582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:59.295469+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":637,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.489278+0000","last_change":"2024-09-18T07:42:09.068025+0000","last_active":"2024-09-18T07:42:39.489278+0000","last_peered":"2024-09-18T07:42:39.489278+0000","last_clean":"2024-09-18T07:42:39.489278+0000","last_became_active":"2024-09-18T07:35:01.401024+0000","last_became_peered":"2024-09-18T07:35:01.401024+0000","last_unstale":"2024-09-18T07:42:39.489278+0000","last_undegraded":"2024-09-18T07:42:39.489278+0000","last_fullsized":"2024-09-18T07:42:39.489278+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:09.067941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:09.067941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:25.306181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":603,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.488940+0000","last_change":"2024-09-18T07:40:44.219626+0000","last_active":"2024-09-18T07:42:39.488940+0000","last_peered":"2024-09-18T07:42:39.488940+0000","last_clean":"2024-09-18T07:42:39.488940+0000","last_became_active":"2024-09-18T07:31:02.347661+0000","last_became_peered":"2024-09-18T07:31:02.347661+0000","last_unstale":"2024-09-18T07:42:39.488940+0000","last_undegraded":"2024-09-18T07:42:39.488940+0000","last_fullsized":"2024-09-18T07:42:39.488940+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:44.219547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:44.219547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"356'16156","reported_seq":16407,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:42.405626+0000","last_change":"2024-09-18T07:42:28.107772+0000","last_active":"2024-09-18T07:42:42.405626+0000","last_peered":"2024-09-18T07:42:42.405626+0000","last_clean":"2024-09-18T07:42:42.405626+0000","last_became_active":"2024-09-18T07:39:29.702953+0000","last_became_peered":"2024-09-18T07:39:29.702953+0000","last_unstale":"2024-09-18T07:42:42.405626+0000","last_undegraded":"2024-09-18T07:42:42.405626+0000","last_fullsized":"2024-09-18T07:42:42.405626+0000","mapping_epoch":289,"log_start":"338'11600","ondisk_log_start":"338'11600","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'15056","last_scrub_stamp":"2024-09-18T07:42:28.107610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:42:28.107610+0000","objects_scrubbed":1949,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:56.671400+0000","scrub_duration":688,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53477376,"num_objects":816,"num_object_clones":0,"num_object_copies":2448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16156,"num_write_kb":543104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":196,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.486349+0000","last_change":"2024-09-18T07:41:19.479946+0000","last_active":"2024-09-18T07:42:39.486349+0000","last_peered":"2024-09-18T07:42:39.486349+0000","last_clean":"2024-09-18T07:42:39.486349+0000","last_became_active":"2024-09-18T07:39:04.945385+0000","last_became_peered":"2024-09-18T07:39:04.945385+0000","last_unstale":"2024-09-18T07:42:39.486349+0000","last_undegraded":"2024-09-18T07:42:39.486349+0000","last_fullsized":"2024-09-18T07:42:39.486349+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:41:19.479442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:41:19.479442+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:41.906548+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"323'28","reported_seq":707,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.489359+0000","last_change":"2024-09-18T07:40:56.183888+0000","last_active":"2024-09-18T07:42:39.489359+0000","last_peered":"2024-09-18T07:42:39.489359+0000","last_clean":"2024-09-18T07:42:39.489359+0000","last_became_active":"2024-09-18T07:37:43.096236+0000","last_became_peered":"2024-09-18T07:37:43.096236+0000","last_unstale":"2024-09-18T07:42:39.489359+0000","last_undegraded":"2024-09-18T07:42:39.489359+0000","last_fullsized":"2024-09-18T07:42:39.489359+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-18T07:40:56.183802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.183802+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":620,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.480747+0000","last_change":"2024-09-18T07:39:28.750779+0000","last_active":"2024-09-18T07:42:39.480747+0000","last_peered":"2024-09-18T07:42:39.480747+0000","last_clean":"2024-09-18T07:42:39.480747+0000","last_became_active":"2024-09-18T07:30:56.285678+0000","last_became_peered":"2024-09-18T07:30:56.285678+0000","last_unstale":"2024-09-18T07:42:39.480747+0000","last_undegraded":"2024-09-18T07:42:39.480747+0000","last_fullsized":"2024-09-18T07:42:39.480747+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:39:28.750720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.750720+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:13.105049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":772,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.484820+0000","last_change":"2024-09-18T07:42:04.428459+0000","last_active":"2024-09-18T07:42:39.484820+0000","last_peered":"2024-09-18T07:42:39.484820+0000","last_clean":"2024-09-18T07:42:39.484820+0000","last_became_active":"2024-09-18T07:31:00.335730+0000","last_became_peered":"2024-09-18T07:31:00.335730+0000","last_unstale":"2024-09-18T07:42:39.484820+0000","last_undegraded":"2024-09-18T07:42:39.484820+0000","last_fullsized":"2024-09-18T07:42:39.484820+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:42:04.428395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.428395+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:17.768730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":409,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.472254+0000","last_change":"2024-09-18T07:38:58.528134+0000","last_active":"2024-09-18T07:42:39.472254+0000","last_peered":"2024-09-18T07:42:39.472254+0000","last_clean":"2024-09-18T07:42:39.472254+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:42:39.472254+0000","last_undegraded":"2024-09-18T07:42:39.472254+0000","last_fullsized":"2024-09-18T07:42:39.472254+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:38:58.528059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:38:58.528059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:39.895035+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"356'15835","reported_seq":16089,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:42.103244+0000","last_change":"2024-09-18T07:42:33.175207+0000","last_active":"2024-09-18T07:42:42.103244+0000","last_peered":"2024-09-18T07:42:42.103244+0000","last_clean":"2024-09-18T07:42:42.103244+0000","last_became_active":"2024-09-18T07:39:29.702301+0000","last_became_peered":"2024-09-18T07:39:29.702301+0000","last_unstale":"2024-09-18T07:42:42.103244+0000","last_undegraded":"2024-09-18T07:42:42.103244+0000","last_fullsized":"2024-09-18T07:42:42.103244+0000","mapping_epoch":289,"log_start":"338'11100","ondisk_log_start":"338'11100","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'15145","last_scrub_stamp":"2024-09-18T07:42:33.175105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:42:33.175105+0000","objects_scrubbed":1493,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:43.983027+0000","scrub_duration":651,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51970048,"num_objects":793,"num_object_clones":0,"num_object_copies":2379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15835,"num_write_kb":532096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":359,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477800+0000","last_change":"2024-09-18T07:40:38.066508+0000","last_active":"2024-09-18T07:42:39.477800+0000","last_peered":"2024-09-18T07:42:39.477800+0000","last_clean":"2024-09-18T07:42:39.477800+0000","last_became_active":"2024-09-18T07:39:04.941752+0000","last_became_peered":"2024-09-18T07:39:04.941752+0000","last_unstale":"2024-09-18T07:42:39.477800+0000","last_undegraded":"2024-09-18T07:42:39.477800+0000","last_fullsized":"2024-09-18T07:42:39.477800+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:38.066443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.066443+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:19.103720+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"323'20","reported_seq":372,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477926+0000","last_change":"2024-09-18T07:42:03.427924+0000","last_active":"2024-09-18T07:42:39.477926+0000","last_peered":"2024-09-18T07:42:39.477926+0000","last_clean":"2024-09-18T07:42:39.477926+0000","last_became_active":"2024-09-18T07:36:13.118876+0000","last_became_peered":"2024-09-18T07:36:13.118876+0000","last_unstale":"2024-09-18T07:42:39.477926+0000","last_undegraded":"2024-09-18T07:42:39.477926+0000","last_fullsized":"2024-09-18T07:42:39.477926+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:42:03.427843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:03.427843+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:24.815361+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":637,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477990+0000","last_change":"2024-09-18T07:40:53.350083+0000","last_active":"2024-09-18T07:42:39.477990+0000","last_peered":"2024-09-18T07:42:39.477990+0000","last_clean":"2024-09-18T07:42:39.477990+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:42:39.477990+0000","last_undegraded":"2024-09-18T07:42:39.477990+0000","last_fullsized":"2024-09-18T07:42:39.477990+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:53.350019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:53.350019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"240'10","reported_seq":791,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.479334+0000","last_change":"2024-09-18T07:40:50.839710+0000","last_active":"2024-09-18T07:42:39.479334+0000","last_peered":"2024-09-18T07:42:39.479334+0000","last_clean":"2024-09-18T07:42:39.479334+0000","last_became_active":"2024-09-18T07:37:43.167850+0000","last_became_peered":"2024-09-18T07:37:43.167850+0000","last_unstale":"2024-09-18T07:42:39.479334+0000","last_undegraded":"2024-09-18T07:42:39.479334+0000","last_fullsized":"2024-09-18T07:42:39.479334+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'10","last_scrub_stamp":"2024-09-18T07:40:50.839504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.839504+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:23.163039+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":607,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:42.223288+0000","last_change":"2024-09-18T07:42:42.223288+0000","last_active":"2024-09-18T07:42:42.223288+0000","last_peered":"2024-09-18T07:42:42.223288+0000","last_clean":"2024-09-18T07:42:42.223288+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:42:42.223288+0000","last_undegraded":"2024-09-18T07:42:42.223288+0000","last_fullsized":"2024-09-18T07:42:42.223288+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:42.223235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:42.223235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:04.908111+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"356'15599","reported_seq":15791,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:41.674131+0000","last_change":"2024-09-18T07:42:04.807783+0000","last_active":"2024-09-18T07:42:41.674131+0000","last_peered":"2024-09-18T07:42:41.674131+0000","last_clean":"2024-09-18T07:42:41.674131+0000","last_became_active":"2024-09-18T07:39:29.703475+0000","last_became_peered":"2024-09-18T07:39:29.703475+0000","last_unstale":"2024-09-18T07:42:41.674131+0000","last_undegraded":"2024-09-18T07:42:41.674131+0000","last_fullsized":"2024-09-18T07:42:41.674131+0000","mapping_epoch":289,"log_start":"338'11100","ondisk_log_start":"338'11100","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'12776","last_scrub_stamp":"2024-09-18T07:42:04.807635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.807635+0000","objects_scrubbed":3710,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:27.077916+0000","scrub_duration":1049,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55377920,"num_objects":845,"num_object_clones":0,"num_object_copies":2535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15599,"num_write_kb":526208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":359,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477567+0000","last_change":"2024-09-18T07:40:18.954153+0000","last_active":"2024-09-18T07:42:39.477567+0000","last_peered":"2024-09-18T07:42:39.477567+0000","last_clean":"2024-09-18T07:42:39.477567+0000","last_became_active":"2024-09-18T07:39:04.948571+0000","last_became_peered":"2024-09-18T07:39:04.948571+0000","last_unstale":"2024-09-18T07:42:39.477567+0000","last_undegraded":"2024-09-18T07:42:39.477567+0000","last_fullsized":"2024-09-18T07:42:39.477567+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:18.954089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:18.954089+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:40.336036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"323'29","reported_seq":732,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.479559+0000","last_change":"2024-09-18T07:41:16.895792+0000","last_active":"2024-09-18T07:42:39.479559+0000","last_peered":"2024-09-18T07:42:39.479559+0000","last_clean":"2024-09-18T07:42:39.479559+0000","last_became_active":"2024-09-18T07:35:01.415275+0000","last_became_peered":"2024-09-18T07:35:01.415275+0000","last_unstale":"2024-09-18T07:42:39.479559+0000","last_undegraded":"2024-09-18T07:42:39.479559+0000","last_fullsized":"2024-09-18T07:42:39.479559+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'29","last_scrub_stamp":"2024-09-18T07:41:16.895595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:16.895595+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":322,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477127+0000","last_change":"2024-09-18T07:42:31.793848+0000","last_active":"2024-09-18T07:42:39.477127+0000","last_peered":"2024-09-18T07:42:39.477127+0000","last_clean":"2024-09-18T07:42:39.477127+0000","last_became_active":"2024-09-18T07:36:13.118553+0000","last_became_peered":"2024-09-18T07:36:13.118553+0000","last_unstale":"2024-09-18T07:42:39.477127+0000","last_undegraded":"2024-09-18T07:42:39.477127+0000","last_fullsized":"2024-09-18T07:42:39.477127+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:31.793760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:31.793760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:58.491919+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":262,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.476469+0000","last_change":"2024-09-18T07:41:24.300282+0000","last_active":"2024-09-18T07:42:39.476469+0000","last_peered":"2024-09-18T07:42:39.476469+0000","last_clean":"2024-09-18T07:42:39.476469+0000","last_became_active":"2024-09-18T07:37:43.070009+0000","last_became_peered":"2024-09-18T07:37:43.070009+0000","last_unstale":"2024-09-18T07:42:39.476469+0000","last_undegraded":"2024-09-18T07:42:39.476469+0000","last_fullsized":"2024-09-18T07:42:39.476469+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:24.300235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:24.300235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:52.556718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":254,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478699+0000","last_change":"2024-09-18T07:40:10.073596+0000","last_active":"2024-09-18T07:42:39.478699+0000","last_peered":"2024-09-18T07:42:39.478699+0000","last_clean":"2024-09-18T07:42:39.478699+0000","last_became_active":"2024-09-18T07:37:43.051740+0000","last_became_peered":"2024-09-18T07:37:43.051740+0000","last_unstale":"2024-09-18T07:42:39.478699+0000","last_undegraded":"2024-09-18T07:42:39.478699+0000","last_fullsized":"2024-09-18T07:42:39.478699+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:10.073499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:10.073499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"356'15736","reported_seq":15922,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:42.615314+0000","last_change":"2024-09-18T07:42:10.328791+0000","last_active":"2024-09-18T07:42:42.615314+0000","last_peered":"2024-09-18T07:42:42.615314+0000","last_clean":"2024-09-18T07:42:42.615314+0000","last_became_active":"2024-09-18T07:39:29.701239+0000","last_became_peered":"2024-09-18T07:39:29.701239+0000","last_unstale":"2024-09-18T07:42:42.615314+0000","last_undegraded":"2024-09-18T07:42:42.615314+0000","last_fullsized":"2024-09-18T07:42:42.615314+0000","mapping_epoch":289,"log_start":"333'10200","ondisk_log_start":"333'10200","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'13234","last_scrub_stamp":"2024-09-18T07:42:10.328441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:42:10.328441+0000","objects_scrubbed":3282,"log_size":5536,"log_dups_size":0,"ondisk_log_size":5536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:26.764761+0000","scrub_duration":1183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48234496,"num_objects":736,"num_object_clones":0,"num_object_copies":2208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15736,"num_write_kb":527104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":191,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478683+0000","last_change":"2024-09-18T07:40:43.346516+0000","last_active":"2024-09-18T07:42:39.478683+0000","last_peered":"2024-09-18T07:42:39.478683+0000","last_clean":"2024-09-18T07:42:39.478683+0000","last_became_active":"2024-09-18T07:39:04.950564+0000","last_became_peered":"2024-09-18T07:39:04.950564+0000","last_unstale":"2024-09-18T07:42:39.478683+0000","last_undegraded":"2024-09-18T07:42:39.478683+0000","last_fullsized":"2024-09-18T07:42:39.478683+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:43.346447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:43.346447+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:23.534594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"323'41","reported_seq":496,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.476991+0000","last_change":"2024-09-18T07:41:55.160514+0000","last_active":"2024-09-18T07:42:39.476991+0000","last_peered":"2024-09-18T07:42:39.476991+0000","last_clean":"2024-09-18T07:42:39.476991+0000","last_became_active":"2024-09-18T07:36:13.115278+0000","last_became_peered":"2024-09-18T07:36:13.115278+0000","last_unstale":"2024-09-18T07:42:39.476991+0000","last_undegraded":"2024-09-18T07:42:39.476991+0000","last_fullsized":"2024-09-18T07:42:39.476991+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'41","last_scrub_stamp":"2024-09-18T07:41:55.160421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:55.160421+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:01.495646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":645,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.490434+0000","last_change":"2024-09-18T07:42:03.028626+0000","last_active":"2024-09-18T07:42:39.490434+0000","last_peered":"2024-09-18T07:42:39.490434+0000","last_clean":"2024-09-18T07:42:39.490434+0000","last_became_active":"2024-09-18T07:37:43.060930+0000","last_became_peered":"2024-09-18T07:37:43.060930+0000","last_unstale":"2024-09-18T07:42:39.490434+0000","last_undegraded":"2024-09-18T07:42:39.490434+0000","last_fullsized":"2024-09-18T07:42:39.490434+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:03.028558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:03.028558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:12.110802+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":661,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478092+0000","last_change":"2024-09-18T07:42:00.366508+0000","last_active":"2024-09-18T07:42:39.478092+0000","last_peered":"2024-09-18T07:42:39.478092+0000","last_clean":"2024-09-18T07:42:39.478092+0000","last_became_active":"2024-09-18T07:37:43.076595+0000","last_became_peered":"2024-09-18T07:37:43.076595+0000","last_unstale":"2024-09-18T07:42:39.478092+0000","last_undegraded":"2024-09-18T07:42:39.478092+0000","last_fullsized":"2024-09-18T07:42:39.478092+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:42:00.366441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:42:00.366441+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:21.576301+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":603,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.480207+0000","last_change":"2024-09-18T07:39:25.741714+0000","last_active":"2024-09-18T07:42:39.480207+0000","last_peered":"2024-09-18T07:42:39.480207+0000","last_clean":"2024-09-18T07:42:39.480207+0000","last_became_active":"2024-09-18T07:31:00.332428+0000","last_became_peered":"2024-09-18T07:31:00.332428+0000","last_unstale":"2024-09-18T07:42:39.480207+0000","last_undegraded":"2024-09-18T07:42:39.480207+0000","last_fullsized":"2024-09-18T07:42:39.480207+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:25.741643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.741643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":632,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.487218+0000","last_change":"2024-09-18T07:40:05.858885+0000","last_active":"2024-09-18T07:42:39.487218+0000","last_peered":"2024-09-18T07:42:39.487218+0000","last_clean":"2024-09-18T07:42:39.487218+0000","last_became_active":"2024-09-18T07:35:01.418809+0000","last_became_peered":"2024-09-18T07:35:01.418809+0000","last_unstale":"2024-09-18T07:42:39.487218+0000","last_undegraded":"2024-09-18T07:42:39.487218+0000","last_fullsized":"2024-09-18T07:42:39.487218+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:05.858804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:05.858804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:44.323523+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"356'15794","reported_seq":16021,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:42.070101+0000","last_change":"2024-09-18T07:40:50.786329+0000","last_active":"2024-09-18T07:42:42.070101+0000","last_peered":"2024-09-18T07:42:42.070101+0000","last_clean":"2024-09-18T07:42:42.070101+0000","last_became_active":"2024-09-18T07:39:29.702461+0000","last_became_peered":"2024-09-18T07:39:29.702461+0000","last_unstale":"2024-09-18T07:42:42.070101+0000","last_undegraded":"2024-09-18T07:42:42.070101+0000","last_fullsized":"2024-09-18T07:42:42.070101+0000","mapping_epoch":289,"log_start":"338'11100","ondisk_log_start":"338'11100","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7357","last_scrub_stamp":"2024-09-18T07:40:50.786248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.786248+0000","objects_scrubbed":7280,"log_size":4694,"log_dups_size":0,"ondisk_log_size":4694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50987008,"num_objects":778,"num_object_clones":0,"num_object_copies":2334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15794,"num_write_kb":530304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":459,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.471859+0000","last_change":"2024-09-18T07:40:44.476464+0000","last_active":"2024-09-18T07:42:39.471859+0000","last_peered":"2024-09-18T07:42:39.471859+0000","last_clean":"2024-09-18T07:42:39.471859+0000","last_became_active":"2024-09-18T07:39:04.944921+0000","last_became_peered":"2024-09-18T07:39:04.944921+0000","last_unstale":"2024-09-18T07:42:39.471859+0000","last_undegraded":"2024-09-18T07:42:39.471859+0000","last_fullsized":"2024-09-18T07:42:39.471859+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:44.476278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:44.476278+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:20.701318+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"323'17","reported_seq":374,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.479115+0000","last_change":"2024-09-18T07:41:54.340669+0000","last_active":"2024-09-18T07:42:39.479115+0000","last_peered":"2024-09-18T07:42:39.479115+0000","last_clean":"2024-09-18T07:42:39.479115+0000","last_became_active":"2024-09-18T07:36:13.117924+0000","last_became_peered":"2024-09-18T07:36:13.117924+0000","last_unstale":"2024-09-18T07:42:39.479115+0000","last_undegraded":"2024-09-18T07:42:39.479115+0000","last_fullsized":"2024-09-18T07:42:39.479115+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'17","last_scrub_stamp":"2024-09-18T07:41:54.340587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.340587+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:05.584840+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":344,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.482875+0000","last_change":"2024-09-18T07:42:02.127909+0000","last_active":"2024-09-18T07:42:39.482875+0000","last_peered":"2024-09-18T07:42:39.482875+0000","last_clean":"2024-09-18T07:42:39.482875+0000","last_became_active":"2024-09-18T07:37:43.050971+0000","last_became_peered":"2024-09-18T07:37:43.050971+0000","last_unstale":"2024-09-18T07:42:39.482875+0000","last_undegraded":"2024-09-18T07:42:39.482875+0000","last_fullsized":"2024-09-18T07:42:39.482875+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:02.127517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:02.127517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:03.074068+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":640,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.476573+0000","last_change":"2024-09-18T07:41:26.957957+0000","last_active":"2024-09-18T07:42:39.476573+0000","last_peered":"2024-09-18T07:42:39.476573+0000","last_clean":"2024-09-18T07:42:39.476573+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:42:39.476573+0000","last_undegraded":"2024-09-18T07:42:39.476573+0000","last_fullsized":"2024-09-18T07:42:39.476573+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:41:26.957881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:41:26.957881+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:55.274602+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":637,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.480483+0000","last_change":"2024-09-18T07:42:06.212915+0000","last_active":"2024-09-18T07:42:39.480483+0000","last_peered":"2024-09-18T07:42:39.480483+0000","last_clean":"2024-09-18T07:42:39.480483+0000","last_became_active":"2024-09-18T07:35:01.403680+0000","last_became_peered":"2024-09-18T07:35:01.403680+0000","last_unstale":"2024-09-18T07:42:39.480483+0000","last_undegraded":"2024-09-18T07:42:39.480483+0000","last_fullsized":"2024-09-18T07:42:39.480483+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.212683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.212683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:33.302456+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":413,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.472318+0000","last_change":"2024-09-18T07:40:54.574254+0000","last_active":"2024-09-18T07:42:39.472318+0000","last_peered":"2024-09-18T07:42:39.472318+0000","last_clean":"2024-09-18T07:42:39.472318+0000","last_became_active":"2024-09-18T07:37:43.060019+0000","last_became_peered":"2024-09-18T07:37:43.060019+0000","last_unstale":"2024-09-18T07:42:39.472318+0000","last_undegraded":"2024-09-18T07:42:39.472318+0000","last_fullsized":"2024-09-18T07:42:39.472318+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:40:54.574077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:54.574077+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"356'15871","reported_seq":16096,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:42.582495+0000","last_change":"2024-09-18T07:40:46.264109+0000","last_active":"2024-09-18T07:42:42.582495+0000","last_peered":"2024-09-18T07:42:42.582495+0000","last_clean":"2024-09-18T07:42:42.582495+0000","last_became_active":"2024-09-18T07:39:29.701554+0000","last_became_peered":"2024-09-18T07:39:29.701554+0000","last_unstale":"2024-09-18T07:42:42.582495+0000","last_undegraded":"2024-09-18T07:42:42.582495+0000","last_fullsized":"2024-09-18T07:42:42.582495+0000","mapping_epoch":289,"log_start":"333'10400","ondisk_log_start":"333'10400","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'7054","last_scrub_stamp":"2024-09-18T07:40:46.263923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:46.263923+0000","objects_scrubbed":6963,"log_size":5471,"log_dups_size":0,"ondisk_log_size":5471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:14.068650+0000","scrub_duration":2499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49217536,"num_objects":751,"num_object_clones":0,"num_object_copies":2253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15871,"num_write_kb":531904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":359,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.476427+0000","last_change":"2024-09-18T07:42:39.191671+0000","last_active":"2024-09-18T07:42:39.476427+0000","last_peered":"2024-09-18T07:42:39.476427+0000","last_clean":"2024-09-18T07:42:39.476427+0000","last_became_active":"2024-09-18T07:39:04.938080+0000","last_became_peered":"2024-09-18T07:39:04.938080+0000","last_unstale":"2024-09-18T07:42:39.476427+0000","last_undegraded":"2024-09-18T07:42:39.476427+0000","last_fullsized":"2024-09-18T07:42:39.476427+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:42:39.191605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:42:39.191605+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:59.024220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"323'24","reported_seq":664,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.480509+0000","last_change":"2024-09-18T07:41:27.030793+0000","last_active":"2024-09-18T07:42:39.480509+0000","last_peered":"2024-09-18T07:42:39.480509+0000","last_clean":"2024-09-18T07:42:39.480509+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:42:39.480509+0000","last_undegraded":"2024-09-18T07:42:39.480509+0000","last_fullsized":"2024-09-18T07:42:39.480509+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'24","last_scrub_stamp":"2024-09-18T07:41:27.030705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.030705+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":617,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.479140+0000","last_change":"2024-09-18T07:40:19.694501+0000","last_active":"2024-09-18T07:42:39.479140+0000","last_peered":"2024-09-18T07:42:39.479140+0000","last_clean":"2024-09-18T07:42:39.479140+0000","last_became_active":"2024-09-18T07:30:56.284892+0000","last_became_peered":"2024-09-18T07:30:56.284892+0000","last_unstale":"2024-09-18T07:42:39.479140+0000","last_undegraded":"2024-09-18T07:42:39.479140+0000","last_fullsized":"2024-09-18T07:42:39.479140+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:40:19.694421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:19.694421+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":851,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:40.216293+0000","last_change":"2024-09-18T07:40:47.554230+0000","last_active":"2024-09-18T07:42:40.216293+0000","last_peered":"2024-09-18T07:42:40.216293+0000","last_clean":"2024-09-18T07:42:40.216293+0000","last_became_active":"2024-09-18T07:35:01.400524+0000","last_became_peered":"2024-09-18T07:35:01.400524+0000","last_unstale":"2024-09-18T07:42:40.216293+0000","last_undegraded":"2024-09-18T07:42:40.216293+0000","last_fullsized":"2024-09-18T07:42:40.216293+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T07:40:47.554161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.554161+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":466,"num_read_kb":461,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":629,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.479114+0000","last_change":"2024-09-18T07:42:02.399342+0000","last_active":"2024-09-18T07:42:39.479114+0000","last_peered":"2024-09-18T07:42:39.479114+0000","last_clean":"2024-09-18T07:42:39.479114+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:42:39.479114+0000","last_undegraded":"2024-09-18T07:42:39.479114+0000","last_fullsized":"2024-09-18T07:42:39.479114+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:02.399269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:02.399269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:08.509887+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":322,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478488+0000","last_change":"2024-09-18T07:40:12.269264+0000","last_active":"2024-09-18T07:42:39.478488+0000","last_peered":"2024-09-18T07:42:39.478488+0000","last_clean":"2024-09-18T07:42:39.478488+0000","last_became_active":"2024-09-18T07:36:13.117463+0000","last_became_peered":"2024-09-18T07:36:13.117463+0000","last_unstale":"2024-09-18T07:42:39.478488+0000","last_undegraded":"2024-09-18T07:42:39.478488+0000","last_fullsized":"2024-09-18T07:42:39.478488+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:12.269201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:12.269201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:20.549505+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"356'15507","reported_seq":15729,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:42.413798+0000","last_change":"2024-09-18T07:41:22.252954+0000","last_active":"2024-09-18T07:42:42.413798+0000","last_peered":"2024-09-18T07:42:42.413798+0000","last_clean":"2024-09-18T07:42:42.413798+0000","last_became_active":"2024-09-18T07:39:29.699733+0000","last_became_peered":"2024-09-18T07:39:29.699733+0000","last_unstale":"2024-09-18T07:42:42.413798+0000","last_undegraded":"2024-09-18T07:42:42.413798+0000","last_fullsized":"2024-09-18T07:42:42.413798+0000","mapping_epoch":289,"log_start":"338'10900","ondisk_log_start":"338'10900","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'9472","last_scrub_stamp":"2024-09-18T07:41:22.252822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:41:22.252822+0000","objects_scrubbed":6806,"log_size":4607,"log_dups_size":0,"ondisk_log_size":4607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:26.433396+0000","scrub_duration":1819,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47251456,"num_objects":721,"num_object_clones":0,"num_object_copies":2163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15507,"num_write_kb":519296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"323'32","reported_seq":350,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477506+0000","last_change":"2024-09-18T07:40:17.950392+0000","last_active":"2024-09-18T07:42:39.477506+0000","last_peered":"2024-09-18T07:42:39.477506+0000","last_clean":"2024-09-18T07:42:39.477506+0000","last_became_active":"2024-09-18T07:37:43.087496+0000","last_became_peered":"2024-09-18T07:37:43.087496+0000","last_unstale":"2024-09-18T07:42:39.477506+0000","last_undegraded":"2024-09-18T07:42:39.477506+0000","last_fullsized":"2024-09-18T07:42:39.477506+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-18T07:40:17.950298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:17.950298+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":500,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478017+0000","last_change":"2024-09-18T07:42:35.555848+0000","last_active":"2024-09-18T07:42:39.478017+0000","last_peered":"2024-09-18T07:42:39.478017+0000","last_clean":"2024-09-18T07:42:39.478017+0000","last_became_active":"2024-09-18T07:40:19.067624+0000","last_became_peered":"2024-09-18T07:40:19.067624+0000","last_unstale":"2024-09-18T07:42:39.478017+0000","last_undegraded":"2024-09-18T07:42:39.478017+0000","last_fullsized":"2024-09-18T07:42:39.478017+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:35.555785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:35.555785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:50.301467+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":453,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.471839+0000","last_change":"2024-09-18T07:40:42.452815+0000","last_active":"2024-09-18T07:42:39.471839+0000","last_peered":"2024-09-18T07:42:39.471839+0000","last_clean":"2024-09-18T07:42:39.471839+0000","last_became_active":"2024-09-18T07:39:03.350419+0000","last_became_peered":"2024-09-18T07:39:03.350419+0000","last_unstale":"2024-09-18T07:42:39.471839+0000","last_undegraded":"2024-09-18T07:42:39.471839+0000","last_fullsized":"2024-09-18T07:42:39.471839+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:42.452744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:42.452744+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:16.814737+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":403,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.472617+0000","last_change":"2024-09-18T07:42:06.726510+0000","last_active":"2024-09-18T07:42:39.472617+0000","last_peered":"2024-09-18T07:42:39.472617+0000","last_clean":"2024-09-18T07:42:39.472617+0000","last_became_active":"2024-09-18T07:35:01.412217+0000","last_became_peered":"2024-09-18T07:35:01.412217+0000","last_unstale":"2024-09-18T07:42:39.472617+0000","last_undegraded":"2024-09-18T07:42:39.472617+0000","last_fullsized":"2024-09-18T07:42:39.472617+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.726445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.726445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:15.201416+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":609,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478901+0000","last_change":"2024-09-18T07:42:36.592267+0000","last_active":"2024-09-18T07:42:39.478901+0000","last_peered":"2024-09-18T07:42:39.478901+0000","last_clean":"2024-09-18T07:42:39.478901+0000","last_became_active":"2024-09-18T07:31:02.347938+0000","last_became_peered":"2024-09-18T07:31:02.347938+0000","last_unstale":"2024-09-18T07:42:39.478901+0000","last_undegraded":"2024-09-18T07:42:39.478901+0000","last_fullsized":"2024-09-18T07:42:39.478901+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:36.592215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:36.592215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:54.869914+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"356'15682","reported_seq":15914,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:41.651161+0000","last_change":"2024-09-18T07:40:58.098898+0000","last_active":"2024-09-18T07:42:41.651161+0000","last_peered":"2024-09-18T07:42:41.651161+0000","last_clean":"2024-09-18T07:42:41.651161+0000","last_became_active":"2024-09-18T07:39:29.703451+0000","last_became_peered":"2024-09-18T07:39:29.703451+0000","last_unstale":"2024-09-18T07:42:41.651161+0000","last_undegraded":"2024-09-18T07:42:41.651161+0000","last_fullsized":"2024-09-18T07:42:41.651161+0000","mapping_epoch":289,"log_start":"338'11200","ondisk_log_start":"338'11200","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7779","last_scrub_stamp":"2024-09-18T07:40:58.098697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:58.098697+0000","objects_scrubbed":7710,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53739520,"num_objects":820,"num_object_clones":0,"num_object_copies":2460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15682,"num_write_kb":528064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"323'19","reported_seq":734,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.493034+0000","last_change":"2024-09-18T07:39:07.038527+0000","last_active":"2024-09-18T07:42:39.493034+0000","last_peered":"2024-09-18T07:42:39.493034+0000","last_clean":"2024-09-18T07:42:39.493034+0000","last_became_active":"2024-09-18T07:37:43.063255+0000","last_became_peered":"2024-09-18T07:37:43.063255+0000","last_unstale":"2024-09-18T07:42:39.493034+0000","last_undegraded":"2024-09-18T07:42:39.493034+0000","last_fullsized":"2024-09-18T07:42:39.493034+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-18T07:39:07.038450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:07.038450+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":645,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.489733+0000","last_change":"2024-09-18T07:39:29.078656+0000","last_active":"2024-09-18T07:42:39.489733+0000","last_peered":"2024-09-18T07:42:39.489733+0000","last_clean":"2024-09-18T07:42:39.489733+0000","last_became_active":"2024-09-18T07:35:01.401063+0000","last_became_peered":"2024-09-18T07:35:01.401063+0000","last_unstale":"2024-09-18T07:42:39.489733+0000","last_undegraded":"2024-09-18T07:42:39.489733+0000","last_fullsized":"2024-09-18T07:42:39.489733+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.078600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.078600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:40.967780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":635,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477643+0000","last_change":"2024-09-18T07:39:35.406107+0000","last_active":"2024-09-18T07:42:39.477643+0000","last_peered":"2024-09-18T07:42:39.477643+0000","last_clean":"2024-09-18T07:42:39.477643+0000","last_became_active":"2024-09-18T07:37:43.075535+0000","last_became_peered":"2024-09-18T07:37:43.075535+0000","last_unstale":"2024-09-18T07:42:39.477643+0000","last_undegraded":"2024-09-18T07:42:39.477643+0000","last_fullsized":"2024-09-18T07:42:39.477643+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.406041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.406041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:15.529066+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":495,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477107+0000","last_change":"2024-09-18T07:41:20.220326+0000","last_active":"2024-09-18T07:42:39.477107+0000","last_peered":"2024-09-18T07:42:39.477107+0000","last_clean":"2024-09-18T07:42:39.477107+0000","last_became_active":"2024-09-18T07:40:19.068998+0000","last_became_peered":"2024-09-18T07:40:19.068998+0000","last_unstale":"2024-09-18T07:42:39.477107+0000","last_undegraded":"2024-09-18T07:42:39.477107+0000","last_fullsized":"2024-09-18T07:42:39.477107+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:20.220262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:20.220262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:23.019047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"356'15777","reported_seq":16002,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:41.753735+0000","last_change":"2024-09-18T07:40:48.247625+0000","last_active":"2024-09-18T07:42:41.753735+0000","last_peered":"2024-09-18T07:42:41.753735+0000","last_clean":"2024-09-18T07:42:41.753735+0000","last_became_active":"2024-09-18T07:39:29.699800+0000","last_became_peered":"2024-09-18T07:39:29.699800+0000","last_unstale":"2024-09-18T07:42:41.753735+0000","last_undegraded":"2024-09-18T07:42:41.753735+0000","last_fullsized":"2024-09-18T07:42:41.753735+0000","mapping_epoch":289,"log_start":"330'9700","ondisk_log_start":"330'9700","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'7166","last_scrub_stamp":"2024-09-18T07:40:48.247448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:48.247448+0000","objects_scrubbed":7089,"log_size":6077,"log_dups_size":0,"ondisk_log_size":6077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:16.070539+0000","scrub_duration":2089,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53018624,"num_objects":809,"num_object_clones":0,"num_object_copies":2427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15777,"num_write_kb":530752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":193,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477652+0000","last_change":"2024-09-18T07:39:38.447787+0000","last_active":"2024-09-18T07:42:39.477652+0000","last_peered":"2024-09-18T07:42:39.477652+0000","last_clean":"2024-09-18T07:42:39.477652+0000","last_became_active":"2024-09-18T07:39:04.950292+0000","last_became_peered":"2024-09-18T07:39:04.950292+0000","last_unstale":"2024-09-18T07:42:39.477652+0000","last_undegraded":"2024-09-18T07:42:39.477652+0000","last_fullsized":"2024-09-18T07:42:39.477652+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:38.447735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:38.447735+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:46.260853+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"323'31","reported_seq":684,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.490843+0000","last_change":"2024-09-18T07:40:04.281816+0000","last_active":"2024-09-18T07:42:39.490843+0000","last_peered":"2024-09-18T07:42:39.490843+0000","last_clean":"2024-09-18T07:42:39.490843+0000","last_became_active":"2024-09-18T07:30:58.312272+0000","last_became_peered":"2024-09-18T07:30:58.312272+0000","last_unstale":"2024-09-18T07:42:39.490843+0000","last_undegraded":"2024-09-18T07:42:39.490843+0000","last_fullsized":"2024-09-18T07:42:39.490843+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:40:04.281758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:04.281758+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:46.955052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":270,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.476603+0000","last_change":"2024-09-18T07:41:54.294370+0000","last_active":"2024-09-18T07:42:39.476603+0000","last_peered":"2024-09-18T07:42:39.476603+0000","last_clean":"2024-09-18T07:42:39.476603+0000","last_became_active":"2024-09-18T07:37:43.061808+0000","last_became_peered":"2024-09-18T07:37:43.061808+0000","last_unstale":"2024-09-18T07:42:39.476603+0000","last_undegraded":"2024-09-18T07:42:39.476603+0000","last_fullsized":"2024-09-18T07:42:39.476603+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:54.294307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.294307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:57.341592+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":313,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.490126+0000","last_change":"2024-09-18T07:40:52.167232+0000","last_active":"2024-09-18T07:42:39.490126+0000","last_peered":"2024-09-18T07:42:39.490126+0000","last_clean":"2024-09-18T07:42:39.490126+0000","last_became_active":"2024-09-18T07:36:13.116527+0000","last_became_peered":"2024-09-18T07:36:13.116527+0000","last_unstale":"2024-09-18T07:42:39.490126+0000","last_undegraded":"2024-09-18T07:42:39.490126+0000","last_fullsized":"2024-09-18T07:42:39.490126+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:52.167048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:52.167048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:17.172256+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":635,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.479480+0000","last_change":"2024-09-18T07:41:56.172872+0000","last_active":"2024-09-18T07:42:39.479480+0000","last_peered":"2024-09-18T07:42:39.479480+0000","last_clean":"2024-09-18T07:42:39.479480+0000","last_became_active":"2024-09-18T07:31:02.347345+0000","last_became_peered":"2024-09-18T07:31:02.347345+0000","last_unstale":"2024-09-18T07:42:39.479480+0000","last_undegraded":"2024-09-18T07:42:39.479480+0000","last_fullsized":"2024-09-18T07:42:39.479480+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:56.172797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:56.172797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:05.871802+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"356'15676","reported_seq":15902,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:41.676293+0000","last_change":"2024-09-18T07:40:41.001448+0000","last_active":"2024-09-18T07:42:41.676293+0000","last_peered":"2024-09-18T07:42:41.676293+0000","last_clean":"2024-09-18T07:42:41.676293+0000","last_became_active":"2024-09-18T07:40:19.066715+0000","last_became_peered":"2024-09-18T07:40:19.066715+0000","last_unstale":"2024-09-18T07:42:41.676293+0000","last_undegraded":"2024-09-18T07:42:41.676293+0000","last_fullsized":"2024-09-18T07:42:41.676293+0000","mapping_epoch":307,"log_start":"330'9600","ondisk_log_start":"330'9600","created":289,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'6605","last_scrub_stamp":"2024-09-18T07:40:41.001367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:41.001367+0000","objects_scrubbed":6534,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:46.302322+0000","scrub_duration":1931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55836672,"num_objects":852,"num_object_clones":0,"num_object_copies":2556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15676,"num_write_kb":528896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":459,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.472260+0000","last_change":"2024-09-18T07:42:32.807852+0000","last_active":"2024-09-18T07:42:39.472260+0000","last_peered":"2024-09-18T07:42:39.472260+0000","last_clean":"2024-09-18T07:42:39.472260+0000","last_became_active":"2024-09-18T07:39:04.944753+0000","last_became_peered":"2024-09-18T07:39:04.944753+0000","last_unstale":"2024-09-18T07:42:39.472260+0000","last_undegraded":"2024-09-18T07:42:39.472260+0000","last_fullsized":"2024-09-18T07:42:39.472260+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:42:32.807800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:42:32.807800+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:48.653502+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"323'20","reported_seq":687,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.490271+0000","last_change":"2024-09-18T07:40:50.210059+0000","last_active":"2024-09-18T07:42:39.490271+0000","last_peered":"2024-09-18T07:42:39.490271+0000","last_clean":"2024-09-18T07:42:39.490271+0000","last_became_active":"2024-09-18T07:37:43.062724+0000","last_became_peered":"2024-09-18T07:37:43.062724+0000","last_unstale":"2024-09-18T07:42:39.490271+0000","last_undegraded":"2024-09-18T07:42:39.490271+0000","last_fullsized":"2024-09-18T07:42:39.490271+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:40:50.209824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.209824+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":119,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.479708+0000","last_change":"2024-09-18T07:40:47.799512+0000","last_active":"2024-09-18T07:42:39.479708+0000","last_peered":"2024-09-18T07:42:39.479708+0000","last_clean":"2024-09-18T07:42:39.479708+0000","last_became_active":"2024-09-18T07:40:19.065652+0000","last_became_peered":"2024-09-18T07:40:19.065652+0000","last_unstale":"2024-09-18T07:42:39.479708+0000","last_undegraded":"2024-09-18T07:42:39.479708+0000","last_fullsized":"2024-09-18T07:42:39.479708+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:47.799342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.799342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:24.143277+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"173'11","reported_seq":806,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:40.909491+0000","last_change":"2024-09-18T07:42:40.909417+0000","last_active":"2024-09-18T07:42:40.909491+0000","last_peered":"2024-09-18T07:42:40.909491+0000","last_clean":"2024-09-18T07:42:40.909491+0000","last_became_active":"2024-09-18T07:35:01.403765+0000","last_became_peered":"2024-09-18T07:35:01.403765+0000","last_unstale":"2024-09-18T07:42:40.909491+0000","last_undegraded":"2024-09-18T07:42:40.909491+0000","last_fullsized":"2024-09-18T07:42:40.909491+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'11","last_scrub_stamp":"2024-09-18T07:42:40.909351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:40.909351+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:44.244643+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":611,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.492734+0000","last_change":"2024-09-18T07:36:33.251164+0000","last_active":"2024-09-18T07:42:39.492734+0000","last_peered":"2024-09-18T07:42:39.492734+0000","last_clean":"2024-09-18T07:42:39.492734+0000","last_became_active":"2024-09-18T07:31:02.347095+0000","last_became_peered":"2024-09-18T07:31:02.347095+0000","last_unstale":"2024-09-18T07:42:39.492734+0000","last_undegraded":"2024-09-18T07:42:39.492734+0000","last_fullsized":"2024-09-18T07:42:39.492734+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:33.251093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:33.251093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:12.044091+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"356'15725","reported_seq":15957,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:42.059594+0000","last_change":"2024-09-18T07:40:56.968567+0000","last_active":"2024-09-18T07:42:42.059594+0000","last_peered":"2024-09-18T07:42:42.059594+0000","last_clean":"2024-09-18T07:42:42.059594+0000","last_became_active":"2024-09-18T07:39:29.704318+0000","last_became_peered":"2024-09-18T07:39:29.704318+0000","last_unstale":"2024-09-18T07:42:42.059594+0000","last_undegraded":"2024-09-18T07:42:42.059594+0000","last_fullsized":"2024-09-18T07:42:42.059594+0000","mapping_epoch":289,"log_start":"338'11000","ondisk_log_start":"338'11000","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7695","last_scrub_stamp":"2024-09-18T07:40:56.968381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.968381+0000","objects_scrubbed":7617,"log_size":4725,"log_dups_size":0,"ondisk_log_size":4725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2651,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51707904,"num_objects":789,"num_object_clones":0,"num_object_copies":2367,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15725,"num_write_kb":528448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"323'15","reported_seq":665,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.479475+0000","last_change":"2024-09-18T07:41:21.946159+0000","last_active":"2024-09-18T07:42:39.479475+0000","last_peered":"2024-09-18T07:42:39.479475+0000","last_clean":"2024-09-18T07:42:39.479475+0000","last_became_active":"2024-09-18T07:37:27.994144+0000","last_became_peered":"2024-09-18T07:37:27.994144+0000","last_unstale":"2024-09-18T07:42:39.479475+0000","last_undegraded":"2024-09-18T07:42:39.479475+0000","last_fullsized":"2024-09-18T07:42:39.479475+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:21.946066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:21.946066+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":643,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478541+0000","last_change":"2024-09-18T07:40:45.343597+0000","last_active":"2024-09-18T07:42:39.478541+0000","last_peered":"2024-09-18T07:42:39.478541+0000","last_clean":"2024-09-18T07:42:39.478541+0000","last_became_active":"2024-09-18T07:37:43.075988+0000","last_became_peered":"2024-09-18T07:37:43.075988+0000","last_unstale":"2024-09-18T07:42:39.478541+0000","last_undegraded":"2024-09-18T07:42:39.478541+0000","last_fullsized":"2024-09-18T07:42:39.478541+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:45.343533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:45.343533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":615,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.481632+0000","last_change":"2024-09-18T07:39:01.216439+0000","last_active":"2024-09-18T07:42:39.481632+0000","last_peered":"2024-09-18T07:42:39.481632+0000","last_clean":"2024-09-18T07:42:39.481632+0000","last_became_active":"2024-09-18T07:31:00.335685+0000","last_became_peered":"2024-09-18T07:31:00.335685+0000","last_unstale":"2024-09-18T07:42:39.481632+0000","last_undegraded":"2024-09-18T07:42:39.481632+0000","last_fullsized":"2024-09-18T07:42:39.481632+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:01.216305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:01.216305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":637,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.479065+0000","last_change":"2024-09-18T07:41:59.006637+0000","last_active":"2024-09-18T07:42:39.479065+0000","last_peered":"2024-09-18T07:42:39.479065+0000","last_clean":"2024-09-18T07:42:39.479065+0000","last_became_active":"2024-09-18T07:35:01.410890+0000","last_became_peered":"2024-09-18T07:35:01.410890+0000","last_unstale":"2024-09-18T07:42:39.479065+0000","last_undegraded":"2024-09-18T07:42:39.479065+0000","last_fullsized":"2024-09-18T07:42:39.479065+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:59.006386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:59.006386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:18.266924+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"356'15838","reported_seq":16062,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:42.088885+0000","last_change":"2024-09-18T07:41:23.184909+0000","last_active":"2024-09-18T07:42:42.088885+0000","last_peered":"2024-09-18T07:42:42.088885+0000","last_clean":"2024-09-18T07:42:42.088885+0000","last_became_active":"2024-09-18T07:39:29.701593+0000","last_became_peered":"2024-09-18T07:39:29.701593+0000","last_unstale":"2024-09-18T07:42:42.088885+0000","last_undegraded":"2024-09-18T07:42:42.088885+0000","last_fullsized":"2024-09-18T07:42:42.088885+0000","mapping_epoch":289,"log_start":"338'11100","ondisk_log_start":"338'11100","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'9778","last_scrub_stamp":"2024-09-18T07:41:23.184760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:41:23.184760+0000","objects_scrubbed":7006,"log_size":4738,"log_dups_size":0,"ondisk_log_size":4738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2841,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55705600,"num_objects":850,"num_object_clones":0,"num_object_copies":2550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15838,"num_write_kb":534016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"323'35","reported_seq":298,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.476736+0000","last_change":"2024-09-18T07:39:25.932376+0000","last_active":"2024-09-18T07:42:39.476736+0000","last_peered":"2024-09-18T07:42:39.476736+0000","last_clean":"2024-09-18T07:42:39.476736+0000","last_became_active":"2024-09-18T07:37:43.063319+0000","last_became_peered":"2024-09-18T07:37:43.063319+0000","last_unstale":"2024-09-18T07:42:39.476736+0000","last_undegraded":"2024-09-18T07:42:39.476736+0000","last_fullsized":"2024-09-18T07:42:39.476736+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-18T07:39:25.932315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:25.932315+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":417,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.472596+0000","last_change":"2024-09-18T07:40:56.546321+0000","last_active":"2024-09-18T07:42:39.472596+0000","last_peered":"2024-09-18T07:42:39.472596+0000","last_clean":"2024-09-18T07:42:39.472596+0000","last_became_active":"2024-09-18T07:37:43.072894+0000","last_became_peered":"2024-09-18T07:37:43.072894+0000","last_unstale":"2024-09-18T07:42:39.472596+0000","last_undegraded":"2024-09-18T07:42:39.472596+0000","last_fullsized":"2024-09-18T07:42:39.472596+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:56.546251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.546251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":638,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.476463+0000","last_change":"2024-09-18T07:42:04.179746+0000","last_active":"2024-09-18T07:42:39.476463+0000","last_peered":"2024-09-18T07:42:39.476463+0000","last_clean":"2024-09-18T07:42:39.476463+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:42:39.476463+0000","last_undegraded":"2024-09-18T07:42:39.476463+0000","last_fullsized":"2024-09-18T07:42:39.476463+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:04.179670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.179670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:04.388489+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":395,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.471973+0000","last_change":"2024-09-18T07:42:09.794539+0000","last_active":"2024-09-18T07:42:39.471973+0000","last_peered":"2024-09-18T07:42:39.471973+0000","last_clean":"2024-09-18T07:42:39.471973+0000","last_became_active":"2024-09-18T07:35:01.406980+0000","last_became_peered":"2024-09-18T07:35:01.406980+0000","last_unstale":"2024-09-18T07:42:39.471973+0000","last_undegraded":"2024-09-18T07:42:39.471973+0000","last_fullsized":"2024-09-18T07:42:39.471973+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:09.794459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:09.794459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:32.016537+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"356'15776","reported_seq":15997,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:41.660168+0000","last_change":"2024-09-18T07:40:43.229079+0000","last_active":"2024-09-18T07:42:41.660168+0000","last_peered":"2024-09-18T07:42:41.660168+0000","last_clean":"2024-09-18T07:42:41.660168+0000","last_became_active":"2024-09-18T07:39:29.700244+0000","last_became_peered":"2024-09-18T07:39:29.700244+0000","last_unstale":"2024-09-18T07:42:41.660168+0000","last_undegraded":"2024-09-18T07:42:41.660168+0000","last_fullsized":"2024-09-18T07:42:41.660168+0000","mapping_epoch":289,"log_start":"337'11000","ondisk_log_start":"337'11000","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'6831","last_scrub_stamp":"2024-09-18T07:40:43.228883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:43.228883+0000","objects_scrubbed":6759,"log_size":4776,"log_dups_size":0,"ondisk_log_size":4776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2043,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52035584,"num_objects":794,"num_object_clones":0,"num_object_copies":2382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15776,"num_write_kb":530240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"323'18","reported_seq":675,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.489494+0000","last_change":"2024-09-18T07:41:47.950163+0000","last_active":"2024-09-18T07:42:39.489494+0000","last_peered":"2024-09-18T07:42:39.489494+0000","last_clean":"2024-09-18T07:42:39.489494+0000","last_became_active":"2024-09-18T07:30:58.319740+0000","last_became_peered":"2024-09-18T07:30:58.319740+0000","last_unstale":"2024-09-18T07:42:39.489494+0000","last_undegraded":"2024-09-18T07:42:39.489494+0000","last_fullsized":"2024-09-18T07:42:39.489494+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:41:47.950068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:47.950068+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:01.623082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":395,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.472757+0000","last_change":"2024-09-18T07:42:07.729518+0000","last_active":"2024-09-18T07:42:39.472757+0000","last_peered":"2024-09-18T07:42:39.472757+0000","last_clean":"2024-09-18T07:42:39.472757+0000","last_became_active":"2024-09-18T07:35:01.398499+0000","last_became_peered":"2024-09-18T07:35:01.398499+0000","last_unstale":"2024-09-18T07:42:39.472757+0000","last_undegraded":"2024-09-18T07:42:39.472757+0000","last_fullsized":"2024-09-18T07:42:39.472757+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:07.729438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:07.729438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:33.911294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":647,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.482776+0000","last_change":"2024-09-18T07:41:58.202324+0000","last_active":"2024-09-18T07:42:39.482776+0000","last_peered":"2024-09-18T07:42:39.482776+0000","last_clean":"2024-09-18T07:42:39.482776+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:42:39.482776+0000","last_undegraded":"2024-09-18T07:42:39.482776+0000","last_fullsized":"2024-09-18T07:42:39.482776+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:58.202229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:58.202229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:08.344962+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":624,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.485029+0000","last_change":"2024-09-18T07:40:02.932758+0000","last_active":"2024-09-18T07:42:39.485029+0000","last_peered":"2024-09-18T07:42:39.485029+0000","last_clean":"2024-09-18T07:42:39.485029+0000","last_became_active":"2024-09-18T07:35:01.418735+0000","last_became_peered":"2024-09-18T07:35:01.418735+0000","last_unstale":"2024-09-18T07:42:39.485029+0000","last_undegraded":"2024-09-18T07:42:39.485029+0000","last_fullsized":"2024-09-18T07:42:39.485029+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:02.932690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:02.932690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:42.406919+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"356'15680","reported_seq":15909,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:42.600816+0000","last_change":"2024-09-18T07:40:54.295785+0000","last_active":"2024-09-18T07:42:42.600816+0000","last_peered":"2024-09-18T07:42:42.600816+0000","last_clean":"2024-09-18T07:42:42.600816+0000","last_became_active":"2024-09-18T07:39:29.701617+0000","last_became_peered":"2024-09-18T07:39:29.701617+0000","last_unstale":"2024-09-18T07:42:42.600816+0000","last_undegraded":"2024-09-18T07:42:42.600816+0000","last_fullsized":"2024-09-18T07:42:42.600816+0000","mapping_epoch":289,"log_start":"333'10200","ondisk_log_start":"333'10200","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"321'7478","last_scrub_stamp":"2024-09-18T07:40:54.295602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:39:28.678492+0000","last_clean_scrub_stamp":"2024-09-18T07:40:54.295602+0000","objects_scrubbed":7411,"log_size":5480,"log_dups_size":0,"ondisk_log_size":5480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:26.152256+0000","scrub_duration":2507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49545216,"num_objects":756,"num_object_clones":0,"num_object_copies":2268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15680,"num_write_kb":525952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"323'27","reported_seq":768,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477801+0000","last_change":"2024-09-18T07:41:51.332873+0000","last_active":"2024-09-18T07:42:39.477801+0000","last_peered":"2024-09-18T07:42:39.477801+0000","last_clean":"2024-09-18T07:42:39.477801+0000","last_became_active":"2024-09-18T07:37:43.075552+0000","last_became_peered":"2024-09-18T07:37:43.075552+0000","last_unstale":"2024-09-18T07:42:39.477801+0000","last_undegraded":"2024-09-18T07:42:39.477801+0000","last_fullsized":"2024-09-18T07:42:39.477801+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:41:51.332791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:51.332791+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:13.912522+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":409,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.472608+0000","last_change":"2024-09-18T07:41:27.819345+0000","last_active":"2024-09-18T07:42:39.472608+0000","last_peered":"2024-09-18T07:42:39.472608+0000","last_clean":"2024-09-18T07:42:39.472608+0000","last_became_active":"2024-09-18T07:35:01.406695+0000","last_became_peered":"2024-09-18T07:35:01.406695+0000","last_unstale":"2024-09-18T07:42:39.472608+0000","last_undegraded":"2024-09-18T07:42:39.472608+0000","last_fullsized":"2024-09-18T07:42:39.472608+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:41:27.819269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.819269+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"240'12","reported_seq":522,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.472741+0000","last_change":"2024-09-18T07:42:35.793651+0000","last_active":"2024-09-18T07:42:39.472741+0000","last_peered":"2024-09-18T07:42:39.472741+0000","last_clean":"2024-09-18T07:42:39.472741+0000","last_became_active":"2024-09-18T07:37:43.056714+0000","last_became_peered":"2024-09-18T07:37:43.056714+0000","last_unstale":"2024-09-18T07:42:39.472741+0000","last_undegraded":"2024-09-18T07:42:39.472741+0000","last_fullsized":"2024-09-18T07:42:39.472741+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:42:35.793579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:35.793579+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:49.202372+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":623,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477715+0000","last_change":"2024-09-18T07:42:07.547399+0000","last_active":"2024-09-18T07:42:39.477715+0000","last_peered":"2024-09-18T07:42:39.477715+0000","last_clean":"2024-09-18T07:42:39.477715+0000","last_became_active":"2024-09-18T07:37:43.078121+0000","last_became_peered":"2024-09-18T07:37:43.078121+0000","last_unstale":"2024-09-18T07:42:39.477715+0000","last_undegraded":"2024-09-18T07:42:39.477715+0000","last_fullsized":"2024-09-18T07:42:39.477715+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:07.547352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:07.547352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:10.534419+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":262,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478762+0000","last_change":"2024-09-18T07:42:08.181612+0000","last_active":"2024-09-18T07:42:39.478762+0000","last_peered":"2024-09-18T07:42:39.478762+0000","last_clean":"2024-09-18T07:42:39.478762+0000","last_became_active":"2024-09-18T07:37:43.087994+0000","last_became_peered":"2024-09-18T07:37:43.087994+0000","last_unstale":"2024-09-18T07:42:39.478762+0000","last_undegraded":"2024-09-18T07:42:39.478762+0000","last_fullsized":"2024-09-18T07:42:39.478762+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:08.181420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:08.181420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:21.009353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"323'12","reported_seq":673,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.487270+0000","last_change":"2024-09-18T07:41:48.359809+0000","last_active":"2024-09-18T07:42:39.487270+0000","last_peered":"2024-09-18T07:42:39.487270+0000","last_clean":"2024-09-18T07:42:39.487270+0000","last_became_active":"2024-09-18T07:35:01.407787+0000","last_became_peered":"2024-09-18T07:35:01.407787+0000","last_unstale":"2024-09-18T07:42:39.487270+0000","last_undegraded":"2024-09-18T07:42:39.487270+0000","last_fullsized":"2024-09-18T07:42:39.487270+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'12","last_scrub_stamp":"2024-09-18T07:41:48.359711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.359711+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:03.524359+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":628,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478276+0000","last_change":"2024-09-18T07:41:52.319134+0000","last_active":"2024-09-18T07:42:39.478276+0000","last_peered":"2024-09-18T07:42:39.478276+0000","last_clean":"2024-09-18T07:42:39.478276+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:42:39.478276+0000","last_undegraded":"2024-09-18T07:42:39.478276+0000","last_fullsized":"2024-09-18T07:42:39.478276+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:52.319073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:52.319073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:54.508152+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":401,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.472432+0000","last_change":"2024-09-18T07:40:51.554184+0000","last_active":"2024-09-18T07:42:39.472432+0000","last_peered":"2024-09-18T07:42:39.472432+0000","last_clean":"2024-09-18T07:42:39.472432+0000","last_became_active":"2024-09-18T07:37:43.079313+0000","last_became_peered":"2024-09-18T07:37:43.079313+0000","last_unstale":"2024-09-18T07:42:39.472432+0000","last_undegraded":"2024-09-18T07:42:39.472432+0000","last_fullsized":"2024-09-18T07:42:39.472432+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:51.554073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:51.554073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:23.781937+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"323'15","reported_seq":665,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.480623+0000","last_change":"2024-09-18T07:42:34.313755+0000","last_active":"2024-09-18T07:42:39.480623+0000","last_peered":"2024-09-18T07:42:39.480623+0000","last_clean":"2024-09-18T07:42:39.480623+0000","last_became_active":"2024-09-18T07:35:01.414458+0000","last_became_peered":"2024-09-18T07:35:01.414458+0000","last_unstale":"2024-09-18T07:42:39.480623+0000","last_undegraded":"2024-09-18T07:42:39.480623+0000","last_fullsized":"2024-09-18T07:42:39.480623+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:42:34.313665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:34.313665+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:49.255841+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":417,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.472672+0000","last_change":"2024-09-18T07:41:52.891853+0000","last_active":"2024-09-18T07:42:39.472672+0000","last_peered":"2024-09-18T07:42:39.472672+0000","last_clean":"2024-09-18T07:42:39.472672+0000","last_became_active":"2024-09-18T07:37:43.056675+0000","last_became_peered":"2024-09-18T07:37:43.056675+0000","last_unstale":"2024-09-18T07:42:39.472672+0000","last_undegraded":"2024-09-18T07:42:39.472672+0000","last_fullsized":"2024-09-18T07:42:39.472672+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:52.891811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:52.891811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:55.106484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":643,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.487067+0000","last_change":"2024-09-18T07:41:16.472282+0000","last_active":"2024-09-18T07:42:39.487067+0000","last_peered":"2024-09-18T07:42:39.487067+0000","last_clean":"2024-09-18T07:42:39.487067+0000","last_became_active":"2024-09-18T07:35:01.420133+0000","last_became_peered":"2024-09-18T07:35:01.420133+0000","last_unstale":"2024-09-18T07:42:39.487067+0000","last_undegraded":"2024-09-18T07:42:39.487067+0000","last_fullsized":"2024-09-18T07:42:39.487067+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:16.472210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:16.472210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":624,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.476672+0000","last_change":"2024-09-18T07:42:34.876497+0000","last_active":"2024-09-18T07:42:39.476672+0000","last_peered":"2024-09-18T07:42:39.476672+0000","last_clean":"2024-09-18T07:42:39.476672+0000","last_became_active":"2024-09-18T07:35:01.396305+0000","last_became_peered":"2024-09-18T07:35:01.396305+0000","last_unstale":"2024-09-18T07:42:39.476672+0000","last_undegraded":"2024-09-18T07:42:39.476672+0000","last_fullsized":"2024-09-18T07:42:39.476672+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:34.876429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:34.876429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:01.710289+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"323'15","reported_seq":683,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477824+0000","last_change":"2024-09-18T07:40:52.366957+0000","last_active":"2024-09-18T07:42:39.477824+0000","last_peered":"2024-09-18T07:42:39.477824+0000","last_clean":"2024-09-18T07:42:39.477824+0000","last_became_active":"2024-09-18T07:35:01.416744+0000","last_became_peered":"2024-09-18T07:35:01.416744+0000","last_unstale":"2024-09-18T07:42:39.477824+0000","last_undegraded":"2024-09-18T07:42:39.477824+0000","last_fullsized":"2024-09-18T07:42:39.477824+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:52.366672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:52.366672+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":649,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477892+0000","last_change":"2024-09-18T07:41:56.060035+0000","last_active":"2024-09-18T07:42:39.477892+0000","last_peered":"2024-09-18T07:42:39.477892+0000","last_clean":"2024-09-18T07:42:39.477892+0000","last_became_active":"2024-09-18T07:35:01.414024+0000","last_became_peered":"2024-09-18T07:35:01.414024+0000","last_unstale":"2024-09-18T07:42:39.477892+0000","last_undegraded":"2024-09-18T07:42:39.477892+0000","last_fullsized":"2024-09-18T07:42:39.477892+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:56.059922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:56.059922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:07.678767+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"173'10","reported_seq":786,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:41.501557+0000","last_change":"2024-09-18T07:39:32.947933+0000","last_active":"2024-09-18T07:42:41.501557+0000","last_peered":"2024-09-18T07:42:41.501557+0000","last_clean":"2024-09-18T07:42:41.501557+0000","last_became_active":"2024-09-18T07:35:01.407568+0000","last_became_peered":"2024-09-18T07:35:01.407568+0000","last_unstale":"2024-09-18T07:42:41.501557+0000","last_undegraded":"2024-09-18T07:42:41.501557+0000","last_fullsized":"2024-09-18T07:42:41.501557+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:39:32.947887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:32.947887+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":618,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.490806+0000","last_change":"2024-09-18T07:41:48.955522+0000","last_active":"2024-09-18T07:42:39.490806+0000","last_peered":"2024-09-18T07:42:39.490806+0000","last_clean":"2024-09-18T07:42:39.490806+0000","last_became_active":"2024-09-18T07:31:02.342897+0000","last_became_peered":"2024-09-18T07:31:02.342897+0000","last_unstale":"2024-09-18T07:42:39.490806+0000","last_undegraded":"2024-09-18T07:42:39.490806+0000","last_fullsized":"2024-09-18T07:42:39.490806+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:48.955463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.955463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:00.936125+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"323'20","reported_seq":404,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.472335+0000","last_change":"2024-09-18T07:42:29.867736+0000","last_active":"2024-09-18T07:42:39.472335+0000","last_peered":"2024-09-18T07:42:39.472335+0000","last_clean":"2024-09-18T07:42:39.472335+0000","last_became_active":"2024-09-18T07:40:19.069759+0000","last_became_peered":"2024-09-18T07:40:19.069759+0000","last_unstale":"2024-09-18T07:42:39.472335+0000","last_undegraded":"2024-09-18T07:42:39.472335+0000","last_fullsized":"2024-09-18T07:42:39.472335+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:42:29.867645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:29.867645+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:40.470633+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":403,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.472373+0000","last_change":"2024-09-18T07:42:04.758461+0000","last_active":"2024-09-18T07:42:39.472373+0000","last_peered":"2024-09-18T07:42:39.472373+0000","last_clean":"2024-09-18T07:42:39.472373+0000","last_became_active":"2024-09-18T07:35:01.400197+0000","last_became_peered":"2024-09-18T07:35:01.400197+0000","last_unstale":"2024-09-18T07:42:39.472373+0000","last_undegraded":"2024-09-18T07:42:39.472373+0000","last_fullsized":"2024-09-18T07:42:39.472373+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:04.758396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.758396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:27.831062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":765,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477588+0000","last_change":"2024-09-18T07:40:40.352088+0000","last_active":"2024-09-18T07:42:39.477588+0000","last_peered":"2024-09-18T07:42:39.477588+0000","last_clean":"2024-09-18T07:42:39.477588+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:42:39.477588+0000","last_undegraded":"2024-09-18T07:42:39.477588+0000","last_fullsized":"2024-09-18T07:42:39.477588+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:40:40.352040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.352040+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":636,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.480241+0000","last_change":"2024-09-18T07:42:01.071103+0000","last_active":"2024-09-18T07:42:39.480241+0000","last_peered":"2024-09-18T07:42:39.480241+0000","last_clean":"2024-09-18T07:42:39.480241+0000","last_became_active":"2024-09-18T07:35:01.410561+0000","last_became_peered":"2024-09-18T07:35:01.410561+0000","last_unstale":"2024-09-18T07:42:39.480241+0000","last_undegraded":"2024-09-18T07:42:39.480241+0000","last_fullsized":"2024-09-18T07:42:39.480241+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:01.071028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:01.071028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:12.908296+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"323'19","reported_seq":728,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.490261+0000","last_change":"2024-09-18T07:41:54.022517+0000","last_active":"2024-09-18T07:42:39.490261+0000","last_peered":"2024-09-18T07:42:39.490261+0000","last_clean":"2024-09-18T07:42:39.490261+0000","last_became_active":"2024-09-18T07:38:41.143108+0000","last_became_peered":"2024-09-18T07:38:41.143108+0000","last_unstale":"2024-09-18T07:42:39.490261+0000","last_undegraded":"2024-09-18T07:42:39.490261+0000","last_fullsized":"2024-09-18T07:42:39.490261+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:41:54.022421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.022421+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:19.855029+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":643,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.490533+0000","last_change":"2024-09-18T07:40:40.149372+0000","last_active":"2024-09-18T07:42:39.490533+0000","last_peered":"2024-09-18T07:42:39.490533+0000","last_clean":"2024-09-18T07:42:39.490533+0000","last_became_active":"2024-09-18T07:37:43.061577+0000","last_became_peered":"2024-09-18T07:37:43.061577+0000","last_unstale":"2024-09-18T07:42:39.490533+0000","last_undegraded":"2024-09-18T07:42:39.490533+0000","last_fullsized":"2024-09-18T07:42:39.490533+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:40.149313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.149313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:16.130628+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":313,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477849+0000","last_change":"2024-09-18T07:41:24.566420+0000","last_active":"2024-09-18T07:42:39.477849+0000","last_peered":"2024-09-18T07:42:39.477849+0000","last_clean":"2024-09-18T07:42:39.477849+0000","last_became_active":"2024-09-18T07:36:13.117584+0000","last_became_peered":"2024-09-18T07:36:13.117584+0000","last_unstale":"2024-09-18T07:42:39.477849+0000","last_undegraded":"2024-09-18T07:42:39.477849+0000","last_fullsized":"2024-09-18T07:42:39.477849+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:24.566345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:24.566345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:52.792357+0000","scrub_duration":160,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":118,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.479088+0000","last_change":"2024-09-18T07:42:06.204026+0000","last_active":"2024-09-18T07:42:39.479088+0000","last_peered":"2024-09-18T07:42:39.479088+0000","last_clean":"2024-09-18T07:42:39.479088+0000","last_became_active":"2024-09-18T07:40:19.065468+0000","last_became_peered":"2024-09-18T07:40:19.065468+0000","last_unstale":"2024-09-18T07:42:39.479088+0000","last_undegraded":"2024-09-18T07:42:39.479088+0000","last_fullsized":"2024-09-18T07:42:39.479088+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.203932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.203932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:27.863077+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"323'15","reported_seq":512,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477648+0000","last_change":"2024-09-18T07:41:20.975210+0000","last_active":"2024-09-18T07:42:39.477648+0000","last_peered":"2024-09-18T07:42:39.477648+0000","last_clean":"2024-09-18T07:42:39.477648+0000","last_became_active":"2024-09-18T07:37:43.067598+0000","last_became_peered":"2024-09-18T07:37:43.067598+0000","last_unstale":"2024-09-18T07:42:39.477648+0000","last_undegraded":"2024-09-18T07:42:39.477648+0000","last_fullsized":"2024-09-18T07:42:39.477648+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:20.975135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:20.975135+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":403,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.472509+0000","last_change":"2024-09-18T07:41:47.881147+0000","last_active":"2024-09-18T07:42:39.472509+0000","last_peered":"2024-09-18T07:42:39.472509+0000","last_clean":"2024-09-18T07:42:39.472509+0000","last_became_active":"2024-09-18T07:35:01.405287+0000","last_became_peered":"2024-09-18T07:35:01.405287+0000","last_unstale":"2024-09-18T07:42:39.472509+0000","last_undegraded":"2024-09-18T07:42:39.472509+0000","last_fullsized":"2024-09-18T07:42:39.472509+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:47.880911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:47.880911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:00.989233+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":118,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:41.220830+0000","last_change":"2024-09-18T07:42:41.220234+0000","last_active":"2024-09-18T07:42:41.220830+0000","last_peered":"2024-09-18T07:42:41.220830+0000","last_clean":"2024-09-18T07:42:41.220830+0000","last_became_active":"2024-09-18T07:40:19.068237+0000","last_became_peered":"2024-09-18T07:40:19.068237+0000","last_unstale":"2024-09-18T07:42:41.220830+0000","last_undegraded":"2024-09-18T07:42:41.220830+0000","last_fullsized":"2024-09-18T07:42:41.220830+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:41.219755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:41.219755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:45.866220+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":631,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.479280+0000","last_change":"2024-09-18T07:41:18.963607+0000","last_active":"2024-09-18T07:42:39.479280+0000","last_peered":"2024-09-18T07:42:39.479280+0000","last_clean":"2024-09-18T07:42:39.479280+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:42:39.479280+0000","last_undegraded":"2024-09-18T07:42:39.479280+0000","last_fullsized":"2024-09-18T07:42:39.479280+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:18.963490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:18.963490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:42.985064+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"323'19","reported_seq":733,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.486846+0000","last_change":"2024-09-18T07:42:10.458954+0000","last_active":"2024-09-18T07:42:39.486846+0000","last_peered":"2024-09-18T07:42:39.486846+0000","last_clean":"2024-09-18T07:42:39.486846+0000","last_became_active":"2024-09-18T07:37:43.066208+0000","last_became_peered":"2024-09-18T07:37:43.066208+0000","last_unstale":"2024-09-18T07:42:39.486846+0000","last_undegraded":"2024-09-18T07:42:39.486846+0000","last_fullsized":"2024-09-18T07:42:39.486846+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:42:10.458865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:10.458865+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:33.857115+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":634,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478593+0000","last_change":"2024-09-18T07:41:27.341104+0000","last_active":"2024-09-18T07:42:39.478593+0000","last_peered":"2024-09-18T07:42:39.478593+0000","last_clean":"2024-09-18T07:42:39.478593+0000","last_became_active":"2024-09-18T07:35:01.415261+0000","last_became_peered":"2024-09-18T07:35:01.415261+0000","last_unstale":"2024-09-18T07:42:39.478593+0000","last_undegraded":"2024-09-18T07:42:39.478593+0000","last_fullsized":"2024-09-18T07:42:39.478593+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:27.341063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.341063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":417,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.471992+0000","last_change":"2024-09-18T07:42:05.775215+0000","last_active":"2024-09-18T07:42:39.471992+0000","last_peered":"2024-09-18T07:42:39.471992+0000","last_clean":"2024-09-18T07:42:39.471992+0000","last_became_active":"2024-09-18T07:37:43.057483+0000","last_became_peered":"2024-09-18T07:37:43.057483+0000","last_unstale":"2024-09-18T07:42:39.471992+0000","last_undegraded":"2024-09-18T07:42:39.471992+0000","last_fullsized":"2024-09-18T07:42:39.471992+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:05.775170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:05.775170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:20.076140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":647,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.485164+0000","last_change":"2024-09-18T07:42:34.361617+0000","last_active":"2024-09-18T07:42:39.485164+0000","last_peered":"2024-09-18T07:42:39.485164+0000","last_clean":"2024-09-18T07:42:39.485164+0000","last_became_active":"2024-09-18T07:37:43.067066+0000","last_became_peered":"2024-09-18T07:37:43.067066+0000","last_unstale":"2024-09-18T07:42:39.485164+0000","last_undegraded":"2024-09-18T07:42:39.485164+0000","last_fullsized":"2024-09-18T07:42:39.485164+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:34.361382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:34.361382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:02.969586+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"323'8","reported_seq":677,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.485061+0000","last_change":"2024-09-18T07:37:43.134956+0000","last_active":"2024-09-18T07:42:39.485061+0000","last_peered":"2024-09-18T07:42:39.485061+0000","last_clean":"2024-09-18T07:42:39.485061+0000","last_became_active":"2024-09-18T07:37:43.067503+0000","last_became_peered":"2024-09-18T07:37:43.067503+0000","last_unstale":"2024-09-18T07:42:39.485061+0000","last_undegraded":"2024-09-18T07:42:39.485061+0000","last_fullsized":"2024-09-18T07:42:39.485061+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:36:40.908629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:40.908629+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:13.423257+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":618,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.480551+0000","last_change":"2024-09-18T07:40:11.843693+0000","last_active":"2024-09-18T07:42:39.480551+0000","last_peered":"2024-09-18T07:42:39.480551+0000","last_clean":"2024-09-18T07:42:39.480551+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:42:39.480551+0000","last_undegraded":"2024-09-18T07:42:39.480551+0000","last_fullsized":"2024-09-18T07:42:39.480551+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:11.843625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:11.843625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:42.965878+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":254,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478605+0000","last_change":"2024-09-18T07:40:40.106989+0000","last_active":"2024-09-18T07:42:39.478605+0000","last_peered":"2024-09-18T07:42:39.478605+0000","last_clean":"2024-09-18T07:42:39.478605+0000","last_became_active":"2024-09-18T07:37:43.061081+0000","last_became_peered":"2024-09-18T07:37:43.061081+0000","last_unstale":"2024-09-18T07:42:39.478605+0000","last_undegraded":"2024-09-18T07:42:39.478605+0000","last_fullsized":"2024-09-18T07:42:39.478605+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:40.106921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.106921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:11.100563+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":403,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.472101+0000","last_change":"2024-09-18T07:40:38.492338+0000","last_active":"2024-09-18T07:42:39.472101+0000","last_peered":"2024-09-18T07:42:39.472101+0000","last_clean":"2024-09-18T07:42:39.472101+0000","last_became_active":"2024-09-18T07:35:01.400396+0000","last_became_peered":"2024-09-18T07:35:01.400396+0000","last_unstale":"2024-09-18T07:42:39.472101+0000","last_undegraded":"2024-09-18T07:42:39.472101+0000","last_fullsized":"2024-09-18T07:42:39.472101+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:38.492269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.492269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:17.788359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":305,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.477637+0000","last_change":"2024-09-18T07:36:13.118009+0000","last_active":"2024-09-18T07:42:39.477637+0000","last_peered":"2024-09-18T07:42:39.477637+0000","last_clean":"2024-09-18T07:42:39.477637+0000","last_became_active":"2024-09-18T07:36:13.117154+0000","last_became_peered":"2024-09-18T07:36:13.117154+0000","last_unstale":"2024-09-18T07:42:39.477637+0000","last_undegraded":"2024-09-18T07:42:39.477637+0000","last_fullsized":"2024-09-18T07:42:39.477637+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.071629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.071629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:43.255444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":643,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.485512+0000","last_change":"2024-09-18T07:41:22.478552+0000","last_active":"2024-09-18T07:42:39.485512+0000","last_peered":"2024-09-18T07:42:39.485512+0000","last_clean":"2024-09-18T07:42:39.485512+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:42:39.485512+0000","last_undegraded":"2024-09-18T07:42:39.485512+0000","last_fullsized":"2024-09-18T07:42:39.485512+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:41:22.478474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:22.478474+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"323'19","reported_seq":278,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.478088+0000","last_change":"2024-09-18T07:41:23.281610+0000","last_active":"2024-09-18T07:42:39.478088+0000","last_peered":"2024-09-18T07:42:39.478088+0000","last_clean":"2024-09-18T07:42:39.478088+0000","last_became_active":"2024-09-18T07:37:43.070186+0000","last_became_peered":"2024-09-18T07:37:43.070186+0000","last_unstale":"2024-09-18T07:42:39.478088+0000","last_undegraded":"2024-09-18T07:42:39.478088+0000","last_fullsized":"2024-09-18T07:42:39.478088+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:41:23.281547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:23.281547+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:41.164144+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":641,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-18T07:42:39.480332+0000","last_change":"2024-09-18T07:41:13.842121+0000","last_active":"2024-09-18T07:42:39.480332+0000","last_peered":"2024-09-18T07:42:39.480332+0000","last_clean":"2024-09-18T07:42:39.480332+0000","last_became_active":"2024-09-18T07:35:01.410469+0000","last_became_peered":"2024-09-18T07:35:01.410469+0000","last_unstale":"2024-09-18T07:42:39.480332+0000","last_undegraded":"2024-09-18T07:42:39.480332+0000","last_fullsized":"2024-09-18T07:42:39.480332+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:13.842068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:13.842068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":829816856,"num_objects":12663,"num_object_clones":0,"num_object_copies":37989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12663,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252333,"num_write_kb":8479809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2503225344,"data_stored":2503213128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79533,"ondisk_log_size":79533,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1118,"num_read_kb":875,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":4664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":684,"ondisk_log_size":684,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":466,"num_read_kb":461,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201425,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":930828,"kb_used_data":520636,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92814324,"statfs":{"total":95995035648,"available":95041867776,"internally_reserved":0,"allocated":533131264,"data_stored":532754877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":420018634},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463063,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":500648,"kb_used_data":166808,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93244504,"statfs":{"total":95995035648,"available":95482372096,"internally_reserved":0,"allocated":170811392,"data_stored":170470430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":341834182},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724700,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":970176,"kb_used_data":479344,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92774976,"statfs":{"total":95995035648,"available":95001575424,"internally_reserved":0,"allocated":490848256,"data_stored":490484072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953635,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":489484,"kb_used_data":165368,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93255668,"statfs":{"total":95995035648,"available":95493804032,"internally_reserved":0,"allocated":169336832,"data_stored":168964240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":943372,"kb_used_data":402684,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92801780,"statfs":{"total":95995035648,"available":95029022720,"internally_reserved":0,"allocated":412348416,"data_stored":411973124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":553646535},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411501,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":579132,"kb_used_data":168936,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93166020,"statfs":{"total":95995035648,"available":95402004480,"internally_reserved":0,"allocated":172990464,"data_stored":172634347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575028,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1002140,"kb_used_data":487816,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":92743012,"statfs":{"total":95995035648,"available":94968844288,"internally_reserved":0,"allocated":499523584,"data_stored":499135709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738553,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":593488,"kb_used_data":197184,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93151664,"statfs":{"total":95995035648,"available":95387303936,"internally_reserved":0,"allocated":201916416,"data_stored":201537034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":193069056,"data_stored":193069056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":466423808,"data_stored":466419736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":156565504,"data_stored":156565504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":405274624,"data_stored":405274624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":164691968,"data_stored":164691968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":464261120,"data_stored":464257048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":161873920,"data_stored":161873920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":491065344,"data_stored":491061272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:42:45.006 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-09-18T07:42:45.083 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:45.104 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:45.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:45.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:45.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:45.227 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:45.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:45.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:45.329 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:45.368 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:45.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:45.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:45.505 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:45.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:45.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:45.607 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:45.656 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:45.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:45.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:45.691 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:45.758 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:45.786 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:45.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:45.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:45.825 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:45 smithi137 ceph-mon[29408]: 3.1 scrub starts 2024-09-18T07:42:45.826 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:45 smithi137 ceph-mon[29408]: 3.1 scrub ok 2024-09-18T07:42:45.826 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:45 smithi137 ceph-mon[39175]: 3.1 scrub starts 2024-09-18T07:42:45.826 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:45 smithi137 ceph-mon[39175]: 3.1 scrub ok 2024-09-18T07:42:45.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:45.918 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:45.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:45.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:45 smithi203 ceph-mon[32237]: 3.1 scrub starts 2024-09-18T07:42:45.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:45 smithi203 ceph-mon[32237]: 3.1 scrub ok 2024-09-18T07:42:46.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:46.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:46.053 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:46.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:46.154 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:46.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:46.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:46.292 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:46.322 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:46.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:46.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:46.424 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:46.456 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:46.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:46.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:46.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:46.599 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:46.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:46.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:46.701 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:46.732 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:46.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:46.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:46.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:46.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:46.867 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:46.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:46.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:46 smithi137 ceph-mon[29408]: pgmap v626: 158 pgs: 158 active+clean; 792 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-18T07:42:46.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:46 smithi137 ceph-mon[29408]: 6.3 scrub starts 2024-09-18T07:42:46.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:46 smithi137 ceph-mon[29408]: 6.3 scrub ok 2024-09-18T07:42:46.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:46 smithi137 ceph-mon[39175]: pgmap v626: 158 pgs: 158 active+clean; 792 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-18T07:42:46.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:46 smithi137 ceph-mon[39175]: 6.3 scrub starts 2024-09-18T07:42:46.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:46 smithi137 ceph-mon[39175]: 6.3 scrub ok 2024-09-18T07:42:46.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:46 smithi203 ceph-mon[32237]: pgmap v626: 158 pgs: 158 active+clean; 792 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-18T07:42:46.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:46 smithi203 ceph-mon[32237]: 6.3 scrub starts 2024-09-18T07:42:46.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:46 smithi203 ceph-mon[32237]: 6.3 scrub ok 2024-09-18T07:42:46.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:46.999 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:47.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:47.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:47.101 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:47.132 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:47.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:47.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:47.160 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T07:42:47.234 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:47.286 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:47.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:47.388 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:47.414 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:47.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:47.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:47.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:42:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:42:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:42:47.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:47.544 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:47.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:47.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:47.547 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:42:47.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:42:47.603 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:42:47.646 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:47.700 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:47.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:47.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:47.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:42:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:42:47.793 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:47 smithi203 ceph-mon[32237]: 3.17 scrub starts 2024-09-18T07:42:47.793 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:47 smithi203 ceph-mon[32237]: 3.17 scrub ok 2024-09-18T07:42:47.793 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:47 smithi203 ceph-mon[32237]: 4.1b scrub starts 2024-09-18T07:42:47.793 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:47 smithi203 ceph-mon[32237]: 4.1b scrub ok 2024-09-18T07:42:47.793 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:47 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2104754688' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:42:47.793 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:47 smithi203 ceph-mon[32237]: 6.1 scrub starts 2024-09-18T07:42:47.793 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:47 smithi203 ceph-mon[32237]: 6.1 scrub ok 2024-09-18T07:42:47.803 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:47.803 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:47 smithi137 ceph-mon[29408]: 3.17 scrub starts 2024-09-18T07:42:47.803 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:47 smithi137 ceph-mon[29408]: 3.17 scrub ok 2024-09-18T07:42:47.804 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:47 smithi137 ceph-mon[29408]: 4.1b scrub starts 2024-09-18T07:42:47.804 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:47 smithi137 ceph-mon[29408]: 4.1b scrub ok 2024-09-18T07:42:47.804 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:47 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2104754688' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:42:47.804 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:47 smithi137 ceph-mon[29408]: 6.1 scrub starts 2024-09-18T07:42:47.804 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:47 smithi137 ceph-mon[29408]: 6.1 scrub ok 2024-09-18T07:42:47.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:47 smithi137 ceph-mon[39175]: 3.17 scrub starts 2024-09-18T07:42:47.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:47 smithi137 ceph-mon[39175]: 3.17 scrub ok 2024-09-18T07:42:47.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:47 smithi137 ceph-mon[39175]: 4.1b scrub starts 2024-09-18T07:42:47.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:47 smithi137 ceph-mon[39175]: 4.1b scrub ok 2024-09-18T07:42:47.805 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:47 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2104754688' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:42:47.805 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:47 smithi137 ceph-mon[39175]: 6.1 scrub starts 2024-09-18T07:42:47.805 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:47 smithi137 ceph-mon[39175]: 6.1 scrub ok 2024-09-18T07:42:47.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:47.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:47.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:47.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:47.980 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:47.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:47.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:48.082 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:42:48.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:48.120 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:48.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:42:48.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:48.241 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:42:48.241 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:42:48.251 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:48.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:48.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:48.352 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:48.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:48.389 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:48.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:48.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:48.492 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:48.549 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:48.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:48.613 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:48 smithi137 ceph-mon[29408]: 4.19 scrub starts 2024-09-18T07:42:48.613 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:48 smithi137 ceph-mon[29408]: 4.19 scrub ok 2024-09-18T07:42:48.614 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:48 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:42:48.614 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:48 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:48.614 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:48 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2104754688' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:42:48.614 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:48 smithi137 ceph-mon[29408]: osdmap e357: 8 total, 8 up, 8 in 2024-09-18T07:42:48.614 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:48 smithi137 ceph-mon[29408]: pgmap v628: 158 pgs: 158 active+clean; 450 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-18T07:42:48.614 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:48 smithi137 ceph-mon[39175]: 4.19 scrub starts 2024-09-18T07:42:48.614 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:48 smithi137 ceph-mon[39175]: 4.19 scrub ok 2024-09-18T07:42:48.614 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:48 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:42:48.614 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:48 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:48.615 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:48 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2104754688' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:42:48.615 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:48 smithi137 ceph-mon[39175]: osdmap e357: 8 total, 8 up, 8 in 2024-09-18T07:42:48.615 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:48 smithi137 ceph-mon[39175]: pgmap v628: 158 pgs: 158 active+clean; 450 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-18T07:42:48.651 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:48.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:48.689 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:48.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:48.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:48.797 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:48 smithi203 ceph-mon[32237]: 4.19 scrub starts 2024-09-18T07:42:48.797 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:48 smithi203 ceph-mon[32237]: 4.19 scrub ok 2024-09-18T07:42:48.797 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:48 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:42:48.797 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:48 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:48.797 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:48 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2104754688' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:42:48.797 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:48 smithi203 ceph-mon[32237]: osdmap e357: 8 total, 8 up, 8 in 2024-09-18T07:42:48.797 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:48 smithi203 ceph-mon[32237]: pgmap v628: 158 pgs: 158 active+clean; 450 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-18T07:42:48.830 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:48.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:48.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:48.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:48.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:48.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:48.986 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T07:42:49.063 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:49.101 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:49.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:49.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:49.161 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:42:49.161 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:42:49.193 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:42:49.193 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:42:49.203 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:49.223 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:42:49.224 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:42:49.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:49.245 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:49.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:49.249 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:42:49.249 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:42:49.288 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:42:49.288 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:42:49.330 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:42:49.330 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:42:49.347 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:49.359 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:42:49.359 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:42:49.392 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:42:49.392 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:42:49.394 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:49.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:49.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:49.433 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:42:49.496 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:49.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:49.538 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:49.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:49.639 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:49.672 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:49.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:49.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:49.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:49.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:49.808 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:49.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:49.882 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:49 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2692655606' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:49.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:49.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:49 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2692655606' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:49.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:49 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2692655606' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:49.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:49.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:49.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:50.008 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:42:50.008 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:42:50.008 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-18T07:42:50.008 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-18T07:42:50.008 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.4 2024-09-18T07:42:50.041 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:50.057 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:42:50.057 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:42:50.057 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:42:50.057 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:42:50.057 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.057 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:51788 7.1 7:81b6d503:::benchmark_data_smithi137_74180_object51787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:42:50.057 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:42:50.057 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 599.38311437899995, 2024-09-18T07:42:50.057 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14889203200000001, 2024-09-18T07:42:50.057 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.057 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.057 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.058 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:50.058 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:42:50.058 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:50.058 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 51788 2024-09-18T07:42:50.058 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.058 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.058 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.058 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.058 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:42:50.058 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.058 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.058 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.058 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.059 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666944+0000", 2024-09-18T07:42:50.059 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.059 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.059 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.059 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666966+0000", 2024-09-18T07:42:50.059 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1860000000000001e-05 2024-09-18T07:42:50.059 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.059 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.059 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.666998+0000", 2024-09-18T07:42:50.059 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1856000000000001e-05 2024-09-18T07:42:50.059 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.060 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.060 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667012+0000", 2024-09-18T07:42:50.060 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4592e-05 2024-09-18T07:42:50.060 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.060 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.060 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.060 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667019+0000", 2024-09-18T07:42:50.060 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1940000000000001e-06 2024-09-18T07:42:50.060 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.060 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.060 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.060 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667083+0000", 2024-09-18T07:42:50.060 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3553999999999995e-05 2024-09-18T07:42:50.061 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.061 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.061 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.061 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667217+0000", 2024-09-18T07:42:50.061 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000133712 2024-09-18T07:42:50.061 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.061 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.061 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:42:50.061 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.667345+0000", 2024-09-18T07:42:50.061 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012796199999999999 2024-09-18T07:42:50.061 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.061 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.061 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:50.062 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.668905+0000", 2024-09-18T07:42:50.062 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001560344 2024-09-18T07:42:50.062 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.062 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.062 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.062 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669080+0000", 2024-09-18T07:42:50.062 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000174964 2024-09-18T07:42:50.062 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.062 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.062 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.062 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815763+0000", 2024-09-18T07:42:50.062 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.146683217 2024-09-18T07:42:50.062 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.063 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.063 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.063 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815804+0000", 2024-09-18T07:42:50.063 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0535999999999999e-05 2024-09-18T07:42:50.063 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.063 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.063 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.063 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.815836+0000", 2024-09-18T07:42:50.063 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2240999999999998e-05 2024-09-18T07:42:50.063 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.063 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.063 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.064 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.064 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.064 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:51794 7.1 7:8f789502:::benchmark_data_smithi137_74180_object51793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-18T07:42:50.064 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:42:50.064 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 599.380941269, 2024-09-18T07:42:50.064 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15046784899999999, 2024-09-18T07:42:50.064 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.064 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.064 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.064 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:50.064 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:42:50.064 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:50.064 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 51794 2024-09-18T07:42:50.064 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.065 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.065 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.065 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.065 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:42:50.065 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.065 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.065 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.065 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.065 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669117+0000", 2024-09-18T07:42:50.065 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.065 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.066 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.066 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.066 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669118+0000", 2024-09-18T07:42:50.066 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4079999999999999e-06 2024-09-18T07:42:50.066 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.066 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.066 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.066 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669172+0000", 2024-09-18T07:42:50.066 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3808999999999999e-05 2024-09-18T07:42:50.066 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.066 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.066 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.066 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669184+0000", 2024-09-18T07:42:50.066 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1888e-05 2024-09-18T07:42:50.067 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.067 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.067 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.067 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669192+0000", 2024-09-18T07:42:50.067 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7549999999999994e-06 2024-09-18T07:42:50.067 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.067 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.067 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.067 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669229+0000", 2024-09-18T07:42:50.067 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7234000000000003e-05 2024-09-18T07:42:50.067 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.067 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.067 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.067 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669310+0000", 2024-09-18T07:42:50.067 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1248999999999994e-05 2024-09-18T07:42:50.068 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.068 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.068 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:42:50.068 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.669391+0000", 2024-09-18T07:42:50.068 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0401000000000005e-05 2024-09-18T07:42:50.068 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.068 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.068 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:50.068 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.670397+0000", 2024-09-18T07:42:50.068 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010061619999999999 2024-09-18T07:42:50.068 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.068 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.068 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.068 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.670664+0000", 2024-09-18T07:42:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00026735700000000002 2024-09-18T07:42:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819479+0000", 2024-09-18T07:42:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14881438499999999 2024-09-18T07:42:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819537+0000", 2024-09-18T07:42:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8192999999999997e-05 2024-09-18T07:42:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.069 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:32:50.819585+0000", 2024-09-18T07:42:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8007999999999998e-05 2024-09-18T07:42:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:74680 7.1 7:8999b183:::benchmark_data_smithi137_74180_object74679:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-18T07:42:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:42:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 573.87209322299998, 2024-09-18T07:42:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13647764200000001, 2024-09-18T07:42:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.070 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:42:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 74680 2024-09-18T07:42:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:42:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177965+0000", 2024-09-18T07:42:50.071 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.177967+0000", 2024-09-18T07:42:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.559e-06 2024-09-18T07:42:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178020+0000", 2024-09-18T07:42:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3810999999999997e-05 2024-09-18T07:42:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.072 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178030+0000", 2024-09-18T07:42:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5200000000000003e-06 2024-09-18T07:42:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178035+0000", 2024-09-18T07:42:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0429999999999998e-06 2024-09-18T07:42:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178074+0000", 2024-09-18T07:42:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9014999999999998e-05 2024-09-18T07:42:50.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178151+0000", 2024-09-18T07:42:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6526000000000006e-05 2024-09-18T07:42:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:42:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.178233+0000", 2024-09-18T07:42:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2174000000000003e-05 2024-09-18T07:42:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.180222+0000", 2024-09-18T07:42:50.074 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0019888459999999998 2024-09-18T07:42:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.180453+0000", 2024-09-18T07:42:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023154 2024-09-18T07:42:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314370+0000", 2024-09-18T07:42:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.133916497 2024-09-18T07:42:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314411+0000", 2024-09-18T07:42:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1421000000000002e-05 2024-09-18T07:42:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:33:16.314443+0000", 2024-09-18T07:42:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1690000000000003e-05 2024-09-18T07:42:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:142957 7.1 7:830d8edf:::benchmark_data_smithi137_74180_object38541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-18T07:42:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:25.430263+0000", 2024-09-18T07:42:50.076 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 504.61979559999997, 2024-09-18T07:42:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18925961299999999, 2024-09-18T07:42:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:42:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 142957 2024-09-18T07:42:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430263+0000", 2024-09-18T07:42:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430263+0000", 2024-09-18T07:42:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430265+0000", 2024-09-18T07:42:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9800000000000001e-06 2024-09-18T07:42:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430270+0000", 2024-09-18T07:42:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6189999999999999e-06 2024-09-18T07:42:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430281+0000", 2024-09-18T07:42:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1146e-05 2024-09-18T07:42:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430288+0000", 2024-09-18T07:42:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9319999999999999e-06 2024-09-18T07:42:50.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.080 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.080 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430340+0000", 2024-09-18T07:42:50.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1904999999999999e-05 2024-09-18T07:42:50.080 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.080 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.080 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430468+0000", 2024-09-18T07:42:50.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000127448 2024-09-18T07:42:50.080 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.080 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.080 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 3,6", 2024-09-18T07:42:50.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.430717+0000", 2024-09-18T07:42:50.081 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024888500000000001 2024-09-18T07:42:50.081 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.081 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.081 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:50.081 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.432189+0000", 2024-09-18T07:42:50.081 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014722470000000001 2024-09-18T07:42:50.081 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.081 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.081 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.081 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.432734+0000", 2024-09-18T07:42:50.081 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00054558799999999995 2024-09-18T07:42:50.081 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.082 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.082 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.619409+0000", 2024-09-18T07:42:50.082 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18667450399999999 2024-09-18T07:42:50.082 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.082 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.082 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.619471+0000", 2024-09-18T07:42:50.082 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1877000000000004e-05 2024-09-18T07:42:50.082 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.082 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.082 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:25.619522+0000", 2024-09-18T07:42:50.082 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1481999999999999e-05 2024-09-18T07:42:50.083 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.083 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.083 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.083 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.083 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:119507 8.2 8:4faaf3aa:::benchmark_data_smithi137_101942_object14697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-18T07:42:50.083 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:37:53.113193+0000", 2024-09-18T07:42:50.083 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 296.93686497300001, 2024-09-18T07:42:50.083 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17040597299999999, 2024-09-18T07:42:50.083 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.083 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.083 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.083 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:50.084 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:42:50.084 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:42:50.084 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 119507 2024-09-18T07:42:50.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.084 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.084 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.084 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.084 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113193+0000", 2024-09-18T07:42:50.084 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.084 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.084 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.084 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113193+0000", 2024-09-18T07:42:50.085 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.085 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.085 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.085 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.086 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113195+0000", 2024-09-18T07:42:50.086 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.956e-06 2024-09-18T07:42:50.086 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.086 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.086 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.086 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113201+0000", 2024-09-18T07:42:50.086 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3639999999999998e-06 2024-09-18T07:42:50.086 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.086 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.086 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.086 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113210+0000", 2024-09-18T07:42:50.086 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6649999999999998e-06 2024-09-18T07:42:50.086 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.087 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.087 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.087 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113216+0000", 2024-09-18T07:42:50.087 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0639999999999999e-06 2024-09-18T07:42:50.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.087 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.087 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.087 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113260+0000", 2024-09-18T07:42:50.087 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4013e-05 2024-09-18T07:42:50.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.087 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.087 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113348+0000", 2024-09-18T07:42:50.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7392000000000006e-05 2024-09-18T07:42:50.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:42:50.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.113429+0000", 2024-09-18T07:42:50.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1111000000000003e-05 2024-09-18T07:42:50.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:50.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.115474+0000", 2024-09-18T07:42:50.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0020447769999999998 2024-09-18T07:42:50.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.116688+0000", 2024-09-18T07:42:50.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001214581 2024-09-18T07:42:50.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.283501+0000", 2024-09-18T07:42:50.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.166812558 2024-09-18T07:42:50.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.090 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.283557+0000", 2024-09-18T07:42:50.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5856999999999997e-05 2024-09-18T07:42:50.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.090 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.090 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.090 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:37:53.283599+0000", 2024-09-18T07:42:50.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2635000000000001e-05 2024-09-18T07:42:50.090 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.090 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.090 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.090 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.090 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:170259 8.b e264/201 8:dbf0040f:::benchmark_data_smithi137_101942_object65449:head v 264'10603, mlcod=264'10600)", 2024-09-18T07:42:50.090 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:44.540417+0000", 2024-09-18T07:42:50.091 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 245.50964172799999, 2024-09-18T07:42:50.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14423132799999999, 2024-09-18T07:42:50.091 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.091 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.091 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.091 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.091 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.540417+0000", 2024-09-18T07:42:50.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.540417+0000", 2024-09-18T07:42:50.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.540418+0000", 2024-09-18T07:42:50.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2219999999999999e-06 2024-09-18T07:42:50.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.540424+0000", 2024-09-18T07:42:50.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.28e-06 2024-09-18T07:42:50.093 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.093 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.093 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.093 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.674941+0000", 2024-09-18T07:42:50.093 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13451724300000001 2024-09-18T07:42:50.093 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.093 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.093 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.093 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.674964+0000", 2024-09-18T07:42:50.093 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2813e-05 2024-09-18T07:42:50.093 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.093 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.093 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.094 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.675047+0000", 2024-09-18T07:42:50.094 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2547999999999996e-05 2024-09-18T07:42:50.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.094 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.094 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.094 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.675067+0000", 2024-09-18T07:42:50.094 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0767e-05 2024-09-18T07:42:50.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.094 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.094 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.094 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.684593+0000", 2024-09-18T07:42:50.094 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0095254080000000008 2024-09-18T07:42:50.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.095 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.684648+0000", 2024-09-18T07:42:50.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5047000000000003e-05 2024-09-18T07:42:50.095 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.095 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.095 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.095 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:170269 8.e e264/201 8:70bf8783:::benchmark_data_smithi137_101942_object65459:head v 264'10510, mlcod=264'10508)", 2024-09-18T07:42:50.095 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:44.547119+0000", 2024-09-18T07:42:50.095 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 245.502939064, 2024-09-18T07:42:50.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13849322, 2024-09-18T07:42:50.096 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.096 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.096 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.096 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.096 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.096 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.096 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.547119+0000", 2024-09-18T07:42:50.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.096 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.096 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.096 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.096 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.547119+0000", 2024-09-18T07:42:50.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.097 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.097 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.097 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.547122+0000", 2024-09-18T07:42:50.097 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.277e-06 2024-09-18T07:42:50.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.097 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.097 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.097 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.547131+0000", 2024-09-18T07:42:50.097 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6649999999999998e-06 2024-09-18T07:42:50.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.097 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.097 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.098 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.675047+0000", 2024-09-18T07:42:50.098 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.12791548799999999 2024-09-18T07:42:50.098 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.098 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.098 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.098 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.675056+0000", 2024-09-18T07:42:50.098 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9630000000000004e-06 2024-09-18T07:42:50.098 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.098 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.098 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.098 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.675222+0000", 2024-09-18T07:42:50.098 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000166324 2024-09-18T07:42:50.098 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.099 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.099 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.099 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.675241+0000", 2024-09-18T07:42:50.099 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9480999999999999e-05 2024-09-18T07:42:50.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.099 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.099 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.099 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.685550+0000", 2024-09-18T07:42:50.099 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010308126000000001 2024-09-18T07:42:50.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.099 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.099 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.099 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.685612+0000", 2024-09-18T07:42:50.099 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2896000000000002e-05 2024-09-18T07:42:50.099 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.100 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.100 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.100 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.100 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.100 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:202181 8.2 8:43b6ef0b:::benchmark_data_smithi137_101942_object97371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-18T07:42:50.100 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:39:13.195522+0000", 2024-09-18T07:42:50.100 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 216.854536228, 2024-09-18T07:42:50.100 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13193157799999999, 2024-09-18T07:42:50.100 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.101 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.101 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.101 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:50.101 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:42:50.101 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:42:50.101 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 202181 2024-09-18T07:42:50.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.101 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.101 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.101 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195522+0000", 2024-09-18T07:42:50.101 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.102 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195522+0000", 2024-09-18T07:42:50.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.102 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195524+0000", 2024-09-18T07:42:50.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.86e-06 2024-09-18T07:42:50.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.102 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195530+0000", 2024-09-18T07:42:50.103 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9390000000000002e-06 2024-09-18T07:42:50.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.103 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.103 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.103 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195542+0000", 2024-09-18T07:42:50.103 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2197e-05 2024-09-18T07:42:50.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.103 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.103 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.103 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195549+0000", 2024-09-18T07:42:50.103 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2640000000000002e-06 2024-09-18T07:42:50.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.103 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195595+0000", 2024-09-18T07:42:50.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5355000000000002e-05 2024-09-18T07:42:50.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.104 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195679+0000", 2024-09-18T07:42:50.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3979000000000004e-05 2024-09-18T07:42:50.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.104 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:42:50.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.195750+0000", 2024-09-18T07:42:50.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1229000000000005e-05 2024-09-18T07:42:50.105 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.105 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.105 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:50.105 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.197685+0000", 2024-09-18T07:42:50.105 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0019350019999999999 2024-09-18T07:42:50.105 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.105 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.105 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.105 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.202152+0000", 2024-09-18T07:42:50.105 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0044671550000000004 2024-09-18T07:42:50.105 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.105 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.105 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.105 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.327295+0000", 2024-09-18T07:42:50.106 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.12514278300000001 2024-09-18T07:42:50.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.106 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.106 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.106 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.327408+0000", 2024-09-18T07:42:50.106 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000113173 2024-09-18T07:42:50.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.106 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.106 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.106 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.327454+0000", 2024-09-18T07:42:50.106 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5642000000000001e-05 2024-09-18T07:42:50.106 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.106 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.106 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.107 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.107 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.35171.0:202183 8.2 8:47b706de:::benchmark_data_smithi137_101942_object97373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-18T07:42:50.107 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:39:13.198689+0000", 2024-09-18T07:42:50.107 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 216.85136953700001, 2024-09-18T07:42:50.107 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.12892711600000001, 2024-09-18T07:42:50.107 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.107 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.107 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.107 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:50.107 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.35171", 2024-09-18T07:42:50.107 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:42:50.107 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 202183 2024-09-18T07:42:50.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.108 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.108 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.108 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.108 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.198689+0000", 2024-09-18T07:42:50.108 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.108 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.108 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.108 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.108 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.198689+0000", 2024-09-18T07:42:50.108 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.108 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.108 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.108 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.109 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.198691+0000", 2024-09-18T07:42:50.109 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6759999999999999e-06 2024-09-18T07:42:50.109 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.109 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.109 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.109 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.198703+0000", 2024-09-18T07:42:50.109 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1185999999999999e-05 2024-09-18T07:42:50.109 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.109 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.109 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.109 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.198726+0000", 2024-09-18T07:42:50.109 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3047000000000001e-05 2024-09-18T07:42:50.109 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.109 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.110 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.110 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.198739+0000", 2024-09-18T07:42:50.110 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3036e-05 2024-09-18T07:42:50.110 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.110 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.110 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.110 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.198822+0000", 2024-09-18T07:42:50.110 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3108000000000005e-05 2024-09-18T07:42:50.110 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.110 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.110 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.110 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.198970+0000", 2024-09-18T07:42:50.110 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014830999999999999 2024-09-18T07:42:50.110 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.111 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.111 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:42:50.111 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.199039+0000", 2024-09-18T07:42:50.111 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8854000000000002e-05 2024-09-18T07:42:50.111 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.111 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.111 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:50.111 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.200125+0000", 2024-09-18T07:42:50.111 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010859719999999999 2024-09-18T07:42:50.111 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.111 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.111 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.111 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.202262+0000", 2024-09-18T07:42:50.111 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021368530000000002 2024-09-18T07:42:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.327495+0000", 2024-09-18T07:42:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.12523358200000001 2024-09-18T07:42:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.327570+0000", 2024-09-18T07:42:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4803999999999997e-05 2024-09-18T07:42:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.113 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:39:13.327616+0000", 2024-09-18T07:42:50.113 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5688e-05 2024-09-18T07:42:50.113 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.113 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.113 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.113 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.113 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.113 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:156376 9.4 9:2e51f89f:::benchmark_data_smithi137_126072_object23876:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:42:50.113 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:24.235238+0000", 2024-09-18T07:42:50.113 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 85.814820345000001, 2024-09-18T07:42:50.113 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13710950199999999, 2024-09-18T07:42:50.113 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.113 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.114 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.114 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:50.114 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:42:50.114 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:42:50.114 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 156376 2024-09-18T07:42:50.114 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.114 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.114 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.114 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.114 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.235238+0000", 2024-09-18T07:42:50.114 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.114 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.114 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.114 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.115 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.235238+0000", 2024-09-18T07:42:50.115 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.115 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.115 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.115 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.115 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.235240+0000", 2024-09-18T07:42:50.115 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3470000000000001e-06 2024-09-18T07:42:50.115 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.115 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.115 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.115 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.235246+0000", 2024-09-18T07:42:50.115 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5589999999999996e-06 2024-09-18T07:42:50.115 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.116 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.116 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.116 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.235256+0000", 2024-09-18T07:42:50.116 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0069e-05 2024-09-18T07:42:50.116 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.116 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.116 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.116 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.235263+0000", 2024-09-18T07:42:50.116 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4070000000000004e-06 2024-09-18T07:42:50.116 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.116 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.116 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.116 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.235329+0000", 2024-09-18T07:42:50.116 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5382000000000005e-05 2024-09-18T07:42:50.117 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.117 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.117 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.117 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.235455+0000", 2024-09-18T07:42:50.117 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012602199999999999 2024-09-18T07:42:50.117 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.117 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.117 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:42:50.117 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.235574+0000", 2024-09-18T07:42:50.117 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000119387 2024-09-18T07:42:50.117 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.117 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.117 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:50.117 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.238028+0000", 2024-09-18T07:42:50.118 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0024540640000000002 2024-09-18T07:42:50.118 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.118 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.118 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.118 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.241653+0000", 2024-09-18T07:42:50.118 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0036248040000000001 2024-09-18T07:42:50.118 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.118 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.118 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.118 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.372232+0000", 2024-09-18T07:42:50.118 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13057876099999999 2024-09-18T07:42:50.118 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.118 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.118 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.119 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.372303+0000", 2024-09-18T07:42:50.119 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0932000000000005e-05 2024-09-18T07:42:50.119 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.119 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.119 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.119 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.372347+0000", 2024-09-18T07:42:50.119 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4768000000000002e-05 2024-09-18T07:42:50.119 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.119 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.119 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.119 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.119 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.119 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:156460 9.4 9:2f71b420:::benchmark_data_smithi137_126072_object23960:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:42:50.119 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:24.401401+0000", 2024-09-18T07:42:50.120 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 85.648657722999999, 2024-09-18T07:42:50.120 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16091408800000001, 2024-09-18T07:42:50.120 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.120 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.120 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.120 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:50.120 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:42:50.120 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:42:50.120 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 156460 2024-09-18T07:42:50.120 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.120 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.120 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.120 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.121 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401401+0000", 2024-09-18T07:42:50.121 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.121 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.121 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.121 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.121 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401401+0000", 2024-09-18T07:42:50.121 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.121 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.121 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.121 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.121 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401404+0000", 2024-09-18T07:42:50.121 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2590000000000001e-06 2024-09-18T07:42:50.121 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.122 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.122 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.122 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401412+0000", 2024-09-18T07:42:50.122 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5390000000000007e-06 2024-09-18T07:42:50.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.122 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.122 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.122 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401433+0000", 2024-09-18T07:42:50.122 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0387e-05 2024-09-18T07:42:50.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.122 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.122 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.122 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401443+0000", 2024-09-18T07:42:50.122 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9410000000000002e-06 2024-09-18T07:42:50.123 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.123 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.123 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.123 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401512+0000", 2024-09-18T07:42:50.123 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9638e-05 2024-09-18T07:42:50.123 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.123 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.123 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.123 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401624+0000", 2024-09-18T07:42:50.123 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011163 2024-09-18T07:42:50.123 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.123 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.123 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:42:50.123 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401759+0000", 2024-09-18T07:42:50.124 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000135136 2024-09-18T07:42:50.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.124 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:50.124 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.407165+0000", 2024-09-18T07:42:50.124 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0054057970000000004 2024-09-18T07:42:50.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.124 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.124 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.410863+0000", 2024-09-18T07:42:50.124 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0036984370000000002 2024-09-18T07:42:50.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.124 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.125 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.562212+0000", 2024-09-18T07:42:50.125 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15134899900000001 2024-09-18T07:42:50.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.125 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.125 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.125 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.562268+0000", 2024-09-18T07:42:50.125 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5887999999999998e-05 2024-09-18T07:42:50.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.125 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.125 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.125 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.562315+0000", 2024-09-18T07:42:50.125 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6437000000000003e-05 2024-09-18T07:42:50.125 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.125 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.126 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.126 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.126 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.126 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:156464 9.4 9:2a498700:::benchmark_data_smithi137_126072_object23964:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:42:50.126 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:24.403611+0000", 2024-09-18T07:42:50.126 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 85.646447765999994, 2024-09-18T07:42:50.126 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15881830999999999, 2024-09-18T07:42:50.126 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.126 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.126 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.126 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:50.126 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:42:50.126 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:42:50.126 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 156464 2024-09-18T07:42:50.127 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.127 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.127 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.127 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.127 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403611+0000", 2024-09-18T07:42:50.127 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.127 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.127 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.127 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.127 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403611+0000", 2024-09-18T07:42:50.127 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.127 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.127 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.127 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.128 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403612+0000", 2024-09-18T07:42:50.128 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.835e-06 2024-09-18T07:42:50.128 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.128 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.128 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.128 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403618+0000", 2024-09-18T07:42:50.128 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9869999999999996e-06 2024-09-18T07:42:50.128 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.128 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.128 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.128 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403635+0000", 2024-09-18T07:42:50.128 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6427e-05 2024-09-18T07:42:50.128 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.129 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.129 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.129 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403643+0000", 2024-09-18T07:42:50.129 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1550000000000007e-06 2024-09-18T07:42:50.129 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.129 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.129 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.130 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404404+0000", 2024-09-18T07:42:50.130 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00076056699999999997 2024-09-18T07:42:50.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.130 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.130 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404518+0000", 2024-09-18T07:42:50.130 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000114864 2024-09-18T07:42:50.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.130 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:42:50.130 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404621+0000", 2024-09-18T07:42:50.130 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000103057 2024-09-18T07:42:50.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.131 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:50.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.407172+0000", 2024-09-18T07:42:50.131 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025507139999999999 2024-09-18T07:42:50.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.131 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.131 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.410929+0000", 2024-09-18T07:42:50.131 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0037564719999999998 2024-09-18T07:42:50.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.131 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.131 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.562350+0000", 2024-09-18T07:42:50.131 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15142180299999999 2024-09-18T07:42:50.132 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.132 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.132 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.132 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.562394+0000", 2024-09-18T07:42:50.132 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3278000000000001e-05 2024-09-18T07:42:50.132 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.132 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.132 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.132 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.562429+0000", 2024-09-18T07:42:50.132 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5151000000000003e-05 2024-09-18T07:42:50.132 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.132 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.132 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.133 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.133 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.133 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:172971 9.4 9:2cb28e72:::benchmark_data_smithi137_126072_object40471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-18T07:42:50.133 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:38.556100+0000", 2024-09-18T07:42:50.133 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 71.493958448000001, 2024-09-18T07:42:50.133 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17327221400000001, 2024-09-18T07:42:50.133 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.133 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.133 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.133 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:50.133 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:42:50.133 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:42:50.133 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 172971 2024-09-18T07:42:50.133 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.134 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.134 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.134 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.134 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556100+0000", 2024-09-18T07:42:50.134 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.134 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.134 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.134 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.134 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556100+0000", 2024-09-18T07:42:50.134 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.134 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.134 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.134 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.135 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556102+0000", 2024-09-18T07:42:50.135 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9479999999999998e-06 2024-09-18T07:42:50.135 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.135 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.135 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.135 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556107+0000", 2024-09-18T07:42:50.135 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4249999999999997e-06 2024-09-18T07:42:50.135 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.135 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.135 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.135 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556119+0000", 2024-09-18T07:42:50.135 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1534e-05 2024-09-18T07:42:50.135 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.135 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.136 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.136 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556126+0000", 2024-09-18T07:42:50.136 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3490000000000003e-06 2024-09-18T07:42:50.136 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.136 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.136 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.136 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556184+0000", 2024-09-18T07:42:50.136 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7920000000000001e-05 2024-09-18T07:42:50.136 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.136 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.136 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.136 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556293+0000", 2024-09-18T07:42:50.137 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108464 2024-09-18T07:42:50.137 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.137 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.137 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:42:50.137 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556394+0000", 2024-09-18T07:42:50.137 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000101142 2024-09-18T07:42:50.137 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.137 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.137 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:50.137 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.559040+0000", 2024-09-18T07:42:50.137 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026468030000000001 2024-09-18T07:42:50.137 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.137 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.137 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.138 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.559659+0000", 2024-09-18T07:42:50.138 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000618568 2024-09-18T07:42:50.138 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.138 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.138 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.138 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.729303+0000", 2024-09-18T07:42:50.138 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16964425999999999 2024-09-18T07:42:50.138 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.138 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.138 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.138 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.729342+0000", 2024-09-18T07:42:50.138 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8228000000000001e-05 2024-09-18T07:42:50.138 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.138 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.139 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.139 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.729372+0000", 2024-09-18T07:42:50.139 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0573e-05 2024-09-18T07:42:50.139 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.139 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.139 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.139 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.139 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.139 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.15699.0:243135 9.e e353/289 9:7ed57789:::benchmark_data_smithi137_126072_object110635:head v 353'15036, mlcod=353'15034)", 2024-09-18T07:42:50.139 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:34.718644+0000", 2024-09-18T07:42:50.139 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 15.331414797000001, 2024-09-18T07:42:50.139 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.054129071000000001, 2024-09-18T07:42:50.139 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.140 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.140 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.140 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.140 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.140 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.140 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.718644+0000", 2024-09-18T07:42:50.140 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.140 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.140 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.140 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.140 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.718644+0000", 2024-09-18T07:42:50.140 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.140 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.140 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.141 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.141 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.718645+0000", 2024-09-18T07:42:50.141 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.906e-06 2024-09-18T07:42:50.141 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.141 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.141 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.141 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.718655+0000", 2024-09-18T07:42:50.141 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.645e-06 2024-09-18T07:42:50.141 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.141 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.141 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.141 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.718668+0000", 2024-09-18T07:42:50.141 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3344e-05 2024-09-18T07:42:50.142 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.142 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.142 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.142 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.718677+0000", 2024-09-18T07:42:50.142 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6880000000000005e-06 2024-09-18T07:42:50.142 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.142 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.142 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.142 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.718738+0000", 2024-09-18T07:42:50.142 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1267999999999997e-05 2024-09-18T07:42:50.142 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.142 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.142 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.142 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.718758+0000", 2024-09-18T07:42:50.143 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0109e-05 2024-09-18T07:42:50.143 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.143 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.143 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.143 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.720957+0000", 2024-09-18T07:42:50.143 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021980459999999999 2024-09-18T07:42:50.143 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.143 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.143 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.143 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.772773+0000", 2024-09-18T07:42:50.143 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.051816065000000001 2024-09-18T07:42:50.143 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:50.145 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:50.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:50.145 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.149 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.149 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.149 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.149 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:243138 9.4 9:25410a69:::benchmark_data_smithi137_126072_object110638:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-18T07:42:50.149 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:34.720957+0000", 2024-09-18T07:42:50.149 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 15.329101044, 2024-09-18T07:42:50.149 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.060920572999999999, 2024-09-18T07:42:50.149 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.149 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.149 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.149 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:50.149 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:42:50.149 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:42:50.150 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 243138 2024-09-18T07:42:50.150 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.150 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.150 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.150 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.150 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.720957+0000", 2024-09-18T07:42:50.150 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.150 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.150 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.150 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.150 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.720957+0000", 2024-09-18T07:42:50.150 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.150 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.720959+0000", 2024-09-18T07:42:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0480000000000001e-06 2024-09-18T07:42:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.720965+0000", 2024-09-18T07:42:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1079999999999998e-06 2024-09-18T07:42:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.720977+0000", 2024-09-18T07:42:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1644000000000001e-05 2024-09-18T07:42:50.152 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.152 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.152 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.152 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.720985+0000", 2024-09-18T07:42:50.152 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8229999999999995e-06 2024-09-18T07:42:50.152 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.152 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.152 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.152 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.772837+0000", 2024-09-18T07:42:50.152 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.051852250000000003 2024-09-18T07:42:50.152 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.152 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.152 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.153 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.772957+0000", 2024-09-18T07:42:50.153 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000120034 2024-09-18T07:42:50.153 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.153 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.153 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:42:50.153 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.773066+0000", 2024-09-18T07:42:50.153 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108717 2024-09-18T07:42:50.153 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.153 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.153 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.153 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.779542+0000", 2024-09-18T07:42:50.153 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0064756429999999997 2024-09-18T07:42:50.153 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.153 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.154 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.154 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.779607+0000", 2024-09-18T07:42:50.154 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5172999999999994e-05 2024-09-18T07:42:50.154 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.154 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.154 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:50.154 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.781815+0000", 2024-09-18T07:42:50.154 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022079650000000001 2024-09-18T07:42:50.154 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.154 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.154 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.154 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.781849+0000", 2024-09-18T07:42:50.154 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4585999999999997e-05 2024-09-18T07:42:50.155 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.155 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.155 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.155 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:34.781878+0000", 2024-09-18T07:42:50.155 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8581999999999998e-05 2024-09-18T07:42:50.155 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.155 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.155 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.155 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.155 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.155 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:244340 9.4 9:2439509c:::benchmark_data_smithi137_126072_object111840:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-18T07:42:50.155 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:35.855506+0000", 2024-09-18T07:42:50.155 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 14.194552766999999, 2024-09-18T07:42:50.155 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.042110382000000002, 2024-09-18T07:42:50.156 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.156 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.156 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.156 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:50.156 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:42:50.156 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:42:50.156 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 244340 2024-09-18T07:42:50.156 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.156 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.156 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.156 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.156 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.855506+0000", 2024-09-18T07:42:50.156 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.157 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.157 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.157 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.855506+0000", 2024-09-18T07:42:50.157 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.157 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.157 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.157 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.855507+0000", 2024-09-18T07:42:50.157 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.888e-06 2024-09-18T07:42:50.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.157 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.157 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.157 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.855512+0000", 2024-09-18T07:42:50.158 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.335e-06 2024-09-18T07:42:50.158 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.158 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.158 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.158 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.855523+0000", 2024-09-18T07:42:50.158 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0852999999999999e-05 2024-09-18T07:42:50.158 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.158 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.158 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.158 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.855529+0000", 2024-09-18T07:42:50.158 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8290000000000002e-06 2024-09-18T07:42:50.158 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.158 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.158 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.159 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.856240+0000", 2024-09-18T07:42:50.159 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000710646 2024-09-18T07:42:50.159 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.159 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.159 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.159 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.856357+0000", 2024-09-18T07:42:50.159 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000116928 2024-09-18T07:42:50.159 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.159 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.159 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:42:50.159 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.856457+0000", 2024-09-18T07:42:50.159 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9828999999999996e-05 2024-09-18T07:42:50.159 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.159 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.160 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:50.160 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.861336+0000", 2024-09-18T07:42:50.160 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.004879563 2024-09-18T07:42:50.160 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.160 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.160 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.160 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.862552+0000", 2024-09-18T07:42:50.160 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001215364 2024-09-18T07:42:50.160 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.160 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.160 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.160 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.897489+0000", 2024-09-18T07:42:50.160 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.034937140999999998 2024-09-18T07:42:50.160 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.161 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.161 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.161 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.897568+0000", 2024-09-18T07:42:50.161 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8772999999999999e-05 2024-09-18T07:42:50.161 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.161 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.161 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.161 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.897616+0000", 2024-09-18T07:42:50.161 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8232999999999997e-05 2024-09-18T07:42:50.161 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.161 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.161 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.161 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.161 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.162 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:244345 9.4 9:29411c51:::benchmark_data_smithi137_126072_object111845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-18T07:42:50.162 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:35.858763+0000", 2024-09-18T07:42:50.162 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 14.191294887, 2024-09-18T07:42:50.162 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.042433922999999998, 2024-09-18T07:42:50.162 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.162 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.162 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.162 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:50.162 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:42:50.162 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:42:50.162 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 244345 2024-09-18T07:42:50.162 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.163 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:42:50 smithi203 systemd[1]: Stopping Ceph osd.4 for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:42:50.163 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.163 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.163 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.163 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.858763+0000", 2024-09-18T07:42:50.163 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.164 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.164 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.164 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.164 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.858763+0000", 2024-09-18T07:42:50.164 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.164 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.164 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.164 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.164 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.858765+0000", 2024-09-18T07:42:50.164 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9999999999999999e-06 2024-09-18T07:42:50.164 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.164 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.164 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.165 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.858771+0000", 2024-09-18T07:42:50.165 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7289999999999997e-06 2024-09-18T07:42:50.165 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.165 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.165 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.165 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.858783+0000", 2024-09-18T07:42:50.165 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1839e-05 2024-09-18T07:42:50.165 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.165 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.165 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.165 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.858797+0000", 2024-09-18T07:42:50.165 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3723000000000001e-05 2024-09-18T07:42:50.165 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.165 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.166 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.166 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.858841+0000", 2024-09-18T07:42:50.166 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4026999999999997e-05 2024-09-18T07:42:50.166 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.166 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.166 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.166 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.858911+0000", 2024-09-18T07:42:50.166 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9811e-05 2024-09-18T07:42:50.166 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.166 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.166 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:42:50.166 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.859015+0000", 2024-09-18T07:42:50.166 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000103923 2024-09-18T07:42:50.166 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.167 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.167 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:50.167 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.861572+0000", 2024-09-18T07:42:50.167 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025573560000000002 2024-09-18T07:42:50.167 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.167 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.167 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.167 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.862597+0000", 2024-09-18T07:42:50.167 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010250159999999999 2024-09-18T07:42:50.167 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.167 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.167 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.167 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.901047+0000", 2024-09-18T07:42:50.167 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.038449762999999998 2024-09-18T07:42:50.168 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.168 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.168 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.168 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.901141+0000", 2024-09-18T07:42:50.168 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3953e-05 2024-09-18T07:42:50.168 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.168 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.168 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.168 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:35.901197+0000", 2024-09-18T07:42:50.168 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6783000000000002e-05 2024-09-18T07:42:50.168 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.168 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.168 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.169 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.169 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.169 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:244727 9.4 9:21550001:::benchmark_data_smithi137_126072_object112227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-18T07:42:50.169 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:36.154841+0000", 2024-09-18T07:42:50.169 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 13.895216926, 2024-09-18T07:42:50.169 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.041348071, 2024-09-18T07:42:50.169 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.169 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.169 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.169 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:50.169 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:42:50.169 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:42:50.169 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 244727 2024-09-18T07:42:50.170 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.170 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.170 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.170 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.170 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.154841+0000", 2024-09-18T07:42:50.170 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.170 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.170 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.170 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.170 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.154841+0000", 2024-09-18T07:42:50.170 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.170 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.171 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.171 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.171 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.154843+0000", 2024-09-18T07:42:50.171 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7150000000000001e-06 2024-09-18T07:42:50.171 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.171 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.171 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.171 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.154849+0000", 2024-09-18T07:42:50.171 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4430000000000002e-06 2024-09-18T07:42:50.171 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.171 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.171 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.171 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.154957+0000", 2024-09-18T07:42:50.171 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108214 2024-09-18T07:42:50.171 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.172 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.172 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.172 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.154966+0000", 2024-09-18T07:42:50.172 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7909999999999993e-06 2024-09-18T07:42:50.172 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.172 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.172 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.172 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.155030+0000", 2024-09-18T07:42:50.172 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4072999999999995e-05 2024-09-18T07:42:50.172 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.172 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.172 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.172 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.155146+0000", 2024-09-18T07:42:50.173 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000116332 2024-09-18T07:42:50.173 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.173 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.173 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:42:50.173 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.155252+0000", 2024-09-18T07:42:50.173 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010618100000000001 2024-09-18T07:42:50.173 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.173 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.173 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.173 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.158399+0000", 2024-09-18T07:42:50.173 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0031466279999999998 2024-09-18T07:42:50.173 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.173 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.173 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:50.174 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.158412+0000", 2024-09-18T07:42:50.174 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.358e-05 2024-09-18T07:42:50.174 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.174 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.174 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.174 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.196082+0000", 2024-09-18T07:42:50.174 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.037669459000000002 2024-09-18T07:42:50.174 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.174 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.174 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.174 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.196141+0000", 2024-09-18T07:42:50.174 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9066e-05 2024-09-18T07:42:50.174 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.174 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.175 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.175 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.196189+0000", 2024-09-18T07:42:50.175 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8588999999999998e-05 2024-09-18T07:42:50.175 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.175 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.175 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.175 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.175 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.175 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:244734 9.4 9:2c171cf8:::benchmark_data_smithi137_126072_object112234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-18T07:42:50.175 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:36.162477+0000", 2024-09-18T07:42:50.175 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 13.887581766, 2024-09-18T07:42:50.175 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.033827392999999997, 2024-09-18T07:42:50.175 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.175 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.175 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.176 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:50.176 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:42:50.176 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:42:50.176 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 244734 2024-09-18T07:42:50.176 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.176 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.176 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.176 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.176 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.162477+0000", 2024-09-18T07:42:50.176 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.176 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.176 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.176 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.176 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.162477+0000", 2024-09-18T07:42:50.177 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.177 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.177 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.177 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.177 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.162479+0000", 2024-09-18T07:42:50.177 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.294e-06 2024-09-18T07:42:50.177 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.177 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.177 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.177 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.162494+0000", 2024-09-18T07:42:50.177 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5034999999999999e-05 2024-09-18T07:42:50.177 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.177 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.177 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.178 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.162513+0000", 2024-09-18T07:42:50.178 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8638999999999999e-05 2024-09-18T07:42:50.178 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.178 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.178 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.178 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.162521+0000", 2024-09-18T07:42:50.178 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6000000000000007e-06 2024-09-18T07:42:50.178 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.178 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.178 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.178 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.162684+0000", 2024-09-18T07:42:50.178 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016311900000000001 2024-09-18T07:42:50.178 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.179 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.179 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.179 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.162917+0000", 2024-09-18T07:42:50.179 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000232873 2024-09-18T07:42:50.179 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.179 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.179 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:42:50.179 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.163026+0000", 2024-09-18T07:42:50.179 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000109169 2024-09-18T07:42:50.179 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.179 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.179 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.179 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.165570+0000", 2024-09-18T07:42:50.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025432940000000002 2024-09-18T07:42:50.180 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.180 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.180 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:50.180 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.165645+0000", 2024-09-18T07:42:50.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4957999999999996e-05 2024-09-18T07:42:50.180 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.180 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.180 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.180 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.196225+0000", 2024-09-18T07:42:50.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.030580007999999999 2024-09-18T07:42:50.180 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.180 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.181 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.181 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.196262+0000", 2024-09-18T07:42:50.181 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7450000000000002e-05 2024-09-18T07:42:50.181 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.181 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.181 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.181 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.196304+0000", 2024-09-18T07:42:50.181 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1953999999999998e-05 2024-09-18T07:42:50.181 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.181 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.181 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.181 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.181 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.182 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:248578 9.4 9:2824d134:::benchmark_data_smithi137_126072_object116078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-09-18T07:42:50.182 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:39.196787+0000", 2024-09-18T07:42:50.182 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 10.853271596999999, 2024-09-18T07:42:50.182 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.033134078999999997, 2024-09-18T07:42:50.182 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:50.182 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:50.182 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:50.182 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:50.182 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:42:50.182 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:42:50.182 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 248578 2024-09-18T07:42:50.182 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.182 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:50.182 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.183 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:50.183 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:39.196787+0000", 2024-09-18T07:42:50.183 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.183 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.183 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.183 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:50.183 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:39.196787+0000", 2024-09-18T07:42:50.183 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:50.183 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.183 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.183 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:50.183 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:39.196789+0000", 2024-09-18T07:42:50.183 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2340000000000001e-06 2024-09-18T07:42:50.183 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.184 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.184 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:50.184 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:39.196795+0000", 2024-09-18T07:42:50.184 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2929999999999997e-06 2024-09-18T07:42:50.184 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.184 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.184 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:50.184 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:39.196811+0000", 2024-09-18T07:42:50.184 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5358000000000001e-05 2024-09-18T07:42:50.184 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.184 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.184 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:50.184 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:39.196819+0000", 2024-09-18T07:42:50.185 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.85e-06 2024-09-18T07:42:50.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.185 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.185 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:50.185 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:39.196882+0000", 2024-09-18T07:42:50.185 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2090999999999999e-05 2024-09-18T07:42:50.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.185 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.185 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:50.185 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:39.196992+0000", 2024-09-18T07:42:50.185 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011031699999999999 2024-09-18T07:42:50.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.185 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.185 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:42:50.186 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:39.197094+0000", 2024-09-18T07:42:50.186 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000102 2024-09-18T07:42:50.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.186 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.186 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:50.186 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:39.202153+0000", 2024-09-18T07:42:50.186 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0050586509999999999 2024-09-18T07:42:50.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.186 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.186 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.186 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:39.219620+0000", 2024-09-18T07:42:50.186 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.017467916999999999 2024-09-18T07:42:50.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.187 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.187 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:50.187 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:39.229850+0000", 2024-09-18T07:42:50.187 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010229613 2024-09-18T07:42:50.187 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.187 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.187 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:50.187 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:39.229885+0000", 2024-09-18T07:42:50.187 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5088e-05 2024-09-18T07:42:50.187 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:50.187 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:50.187 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:50.187 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:39.229921+0000", 2024-09-18T07:42:50.187 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5667000000000003e-05 2024-09-18T07:42:50.187 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.188 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.188 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.188 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:50.188 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:50.188 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:42:50.245 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:50.285 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:50.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:50.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:50.387 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:50.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:42:50.418 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:50.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:50.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:50.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:50.559 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:50.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:50.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:50.661 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:50.702 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:50.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:50.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:50.803 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:50.841 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:50.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:50.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:50.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:50.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:50 smithi137 ceph-mon[39175]: pgmap v629: 158 pgs: 158 active+clean; 450 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-18T07:42:50.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:50 smithi137 ceph-mon[29408]: pgmap v629: 158 pgs: 158 active+clean; 450 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-18T07:42:50.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:50.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:50 smithi203 ceph-mon[32237]: pgmap v629: 158 pgs: 158 active+clean; 450 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-18T07:42:50.993 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:50.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:50.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:51.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:51.136 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:51.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:51.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:51.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:51.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:51.277 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:51.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:51.378 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:51.405 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:51.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:51.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:51.507 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:51.527 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:51.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:51.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:51.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:51.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:51.668 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:51.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:51.714 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:42:51.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:51.809 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:51.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:51.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:51.911 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:51.944 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:51.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:52.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:52.081 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:52.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:52.153 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:51 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2365048140' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:42:52.153 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:51 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2365048140' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:42:52.173 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:51 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2365048140' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:42:52.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:52.220 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:52.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:52.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:52.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:42:52.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:52.358 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:52.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:52.460 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:52.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:52.503 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:52.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:52.604 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:52.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:52.643 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:52.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:52.715 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:52.745 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:52.784 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:52.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:52.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:52.854 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:52 smithi137 ceph-mon[29408]: pgmap v630: 158 pgs: 158 active+clean; 361 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-18T07:42:52.855 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:52 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2365048140' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:42:52.855 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:52 smithi137 ceph-mon[29408]: osdmap e358: 8 total, 8 up, 8 in 2024-09-18T07:42:52.855 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:52 smithi137 ceph-mon[39175]: pgmap v630: 158 pgs: 158 active+clean; 361 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-18T07:42:52.855 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:52 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2365048140' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:42:52.855 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:52 smithi137 ceph-mon[39175]: osdmap e358: 8 total, 8 up, 8 in 2024-09-18T07:42:52.877 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T07:42:52.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:52.964 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:52.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:52.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:53.066 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:53.107 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:53.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:53.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:53.128 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:52 smithi203 ceph-mon[32237]: pgmap v630: 158 pgs: 158 active+clean; 361 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-18T07:42:53.128 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:52 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2365048140' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:42:53.128 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:52 smithi203 ceph-mon[32237]: osdmap e358: 8 total, 8 up, 8 in 2024-09-18T07:42:53.147 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Removed 132497 objects 2024-09-18T07:42:53.147 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Clean up completed and total clean up time :108.05 2024-09-18T07:42:53.209 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:53.231 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:53.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:53.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:53.235 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:42:53.332 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:42:53.332 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:42:53.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:53.349 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:42:53.349 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:42:53.356 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:53.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:53.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:53.368 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:42:53.368 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:42:53.384 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:42:53.384 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:42:53.400 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:42:53.401 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:42:53.417 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:42:53.417 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:42:53.433 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:42:53.434 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:42:53.450 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:42:53.450 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:42:53.458 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:53.461 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:42:53 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[36417]: 2024-09-18T07:42:53.125+0000 7fae2dea1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T07:42:53.461 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:42:53 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[36417]: 2024-09-18T07:42:53.125+0000 7fae2dea1640 -1 osd.4 358 *** Got signal Terminated *** 2024-09-18T07:42:53.461 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:42:53 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[36417]: 2024-09-18T07:42:53.125+0000 7fae2dea1640 -1 osd.4 358 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T07:42:53.479 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:53.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:53.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:53.581 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:53.603 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:53.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:53.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:53.707 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:42:53.707 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:42:53.726 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:53.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:53.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:53.828 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:53.849 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:53.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:53.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:53.951 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:53.974 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:53.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:54.075 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:54.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:54.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:54.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:54.124 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:53 smithi203 ceph-mon[32237]: osd.4 marked itself down and dead 2024-09-18T07:42:54.124 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:53 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/524682643' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:54.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:53 smithi137 ceph-mon[29408]: osd.4 marked itself down and dead 2024-09-18T07:42:54.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:53 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/524682643' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:54.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:53 smithi137 ceph-mon[39175]: osd.4 marked itself down and dead 2024-09-18T07:42:54.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:53 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/524682643' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:54.199 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:54.221 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:54.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:54.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:54.323 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:54.345 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:54.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:54.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:54.448 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:54.470 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:54.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:54.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:54.505 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T07:42:54.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:54.593 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:54.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:54.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:54.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:42:54.694 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:54.716 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:54.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:54.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:54.746 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:42:54 smithi203 podman[109636]: 2024-09-18 07:42:54.470439928 +0000 UTC m=+1.497465673 container died 98d5ea05ab263c15a07f83c858e2b74f9a3962aad36c65ad7dfdf5fa6b519509 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux ) 2024-09-18T07:42:54.818 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:54.840 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:54.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:54.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:54.905 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:54.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:54 smithi137 ceph-mon[39175]: pgmap v632: 158 pgs: 158 active+clean; 81 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-09-18T07:42:54.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:54 smithi137 ceph-mon[39175]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-18T07:42:54.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:54 smithi137 ceph-mon[39175]: osdmap e359: 8 total, 7 up, 8 in 2024-09-18T07:42:54.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:54 smithi137 ceph-mon[29408]: pgmap v632: 158 pgs: 158 active+clean; 81 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-09-18T07:42:54.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:54 smithi137 ceph-mon[29408]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-18T07:42:54.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:54 smithi137 ceph-mon[29408]: osdmap e359: 8 total, 7 up, 8 in 2024-09-18T07:42:54.921 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:42:54.942 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:54.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:54.965 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:54.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:55.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:54 smithi203 ceph-mon[32237]: pgmap v632: 158 pgs: 158 active+clean; 81 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-09-18T07:42:55.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:54 smithi203 ceph-mon[32237]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-18T07:42:55.045 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:54 smithi203 ceph-mon[32237]: osdmap e359: 8 total, 7 up, 8 in 2024-09-18T07:42:55.045 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:42:54 smithi203 podman[109636]: 2024-09-18 07:42:54.86251215 +0000 UTC m=+1.889537884 container cleanup 98d5ea05ab263c15a07f83c858e2b74f9a3962aad36c65ad7dfdf5fa6b519509 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-18T07:42:55.045 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:42:54 smithi203 bash[109636]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4 2024-09-18T07:42:55.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:55.087 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:55.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:55.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:55.189 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:55.211 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:55.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:55.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:55.313 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:55.335 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:55.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:55.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:55.437 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:42:55.458 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:55.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:55.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:42:55.560 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:55.583 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:55.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:55.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:55.684 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:55.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:55.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:55.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:55.767 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:42:55.767 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:42:55.775 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:42:55 smithi203 podman[109811]: 2024-09-18 07:42:55.517895685 +0000 UTC m=+1.116277286 container remove 98d5ea05ab263c15a07f83c858e2b74f9a3962aad36c65ad7dfdf5fa6b519509 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T07:42:55.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:55 smithi203 ceph-mon[32237]: osdmap e360: 8 total, 7 up, 8 in 2024-09-18T07:42:55.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:55 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:42:55.775 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:55 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2426242690' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:42:55.776 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:55 smithi203 ceph-mon[32237]: from='client.35963 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:42:55.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:55.830 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:55.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:55.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:55.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:55.953 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:55.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:55.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:56.019 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:42:56.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:55 smithi137 ceph-mon[39175]: osdmap e360: 8 total, 7 up, 8 in 2024-09-18T07:42:56.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:55 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:42:56.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:55 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2426242690' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:42:56.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:55 smithi137 ceph-mon[39175]: from='client.35963 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:42:56.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:55 smithi137 ceph-mon[29408]: osdmap e360: 8 total, 7 up, 8 in 2024-09-18T07:42:56.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:55 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:42:56.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:55 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2426242690' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:42:56.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:55 smithi137 ceph-mon[29408]: from='client.35963 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:42:56.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:56.076 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:56.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:56.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:56.178 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:56.202 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:56.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:56.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:56.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:42:56.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:56.325 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:56.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:56.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:56.361 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-18T07:42:56.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-18T07:42:56.427 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:56.450 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:56.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:56.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:56.552 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:56.574 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:56.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:56.596 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T07:42:56.676 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:56.701 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:56.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:56.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:56.757 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:56.803 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:56.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:56.825 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:56.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:56.926 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:56.948 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:56.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:56.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:57.049 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:57.086 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:57.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:57.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:57.187 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:57 smithi203 ceph-mon[32237]: pgmap v635: 158 pgs: 26 stale+active+clean, 132 active+clean; 81 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 985 op/s 2024-09-18T07:42:57.188 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:57 smithi203 ceph-mon[32237]: from='client.35963 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:42:57.188 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:57 smithi203 ceph-mon[32237]: osdmap e361: 8 total, 7 up, 8 in 2024-09-18T07:42:57.188 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:57 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:42:57.188 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:57 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:42:57.188 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:42:56 smithi203 podman[110163]: 2024-09-18 07:42:56.829376196 +0000 UTC m=+0.022097294 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:42:57.188 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:42:56 smithi203 podman[110163]: 2024-09-18 07:42:56.949166841 +0000 UTC m=+0.141887935 container create 3fb91f6ed6bb884da04c731fe7d836c409912a416d5751388823c8c97adc4d59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True) 2024-09-18T07:42:57.188 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:42:57 smithi203 podman[110163]: 2024-09-18 07:42:57.185532263 +0000 UTC m=+0.378253365 container init 3fb91f6ed6bb884da04c731fe7d836c409912a416d5751388823c8c97adc4d59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-18T07:42:57.188 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:57.209 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:57.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:57.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:57.240 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:57.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:57.332 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:57.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:57.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:57.351 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:42:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:42:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:42:57.352 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:57 smithi137 ceph-mon[29408]: pgmap v635: 158 pgs: 26 stale+active+clean, 132 active+clean; 81 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 985 op/s 2024-09-18T07:42:57.352 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:57 smithi137 ceph-mon[29408]: from='client.35963 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:42:57.352 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:57 smithi137 ceph-mon[29408]: osdmap e361: 8 total, 7 up, 8 in 2024-09-18T07:42:57.352 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:57 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:42:57.352 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:57 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:42:57.352 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:57 smithi137 ceph-mon[39175]: pgmap v635: 158 pgs: 26 stale+active+clean, 132 active+clean; 81 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 985 op/s 2024-09-18T07:42:57.352 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:57 smithi137 ceph-mon[39175]: from='client.35963 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:42:57.352 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:57 smithi137 ceph-mon[39175]: osdmap e361: 8 total, 7 up, 8 in 2024-09-18T07:42:57.352 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:57 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:42:57.352 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:57 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:42:57.371 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:42:57.434 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:57.456 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:57.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:57.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:57.461 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:42:57 smithi203 podman[110163]: 2024-09-18 07:42:57.189204775 +0000 UTC m=+0.381925876 container start 3fb91f6ed6bb884da04c731fe7d836c409912a416d5751388823c8c97adc4d59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-deactivate, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-18T07:42:57.461 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:42:57 smithi203 podman[110163]: 2024-09-18 07:42:57.285632881 +0000 UTC m=+0.478353975 container attach 3fb91f6ed6bb884da04c731fe7d836c409912a416d5751388823c8c97adc4d59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-18T07:42:57.461 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:42:57 smithi203 podman[110163]: 2024-09-18 07:42:57.398944374 +0000 UTC m=+0.591665470 container died 3fb91f6ed6bb884da04c731fe7d836c409912a416d5751388823c8c97adc4d59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-09-18T07:42:57.559 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:57.581 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:57.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:57.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:57.678 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:42:57.678 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:42:57.683 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:57.692 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:42:57.693 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:42:57.704 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:57.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:57.709 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:42:57.709 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:42:57.724 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:42:57.725 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:42:57.725 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-18T07:42:57.741 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:42:57.741 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:42:57.757 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:42:57.757 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:42:57.774 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:42:57.774 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:42:57.799 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:42:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:42:57.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:57.828 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:57.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:57.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:57.930 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:57.953 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:57.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:57.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:58.028 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'unique_pool_2' removed 2024-09-18T07:42:58.038 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:42:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:42:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:42:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:42:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153406 7.7 7:e5535fc6:::benchmark_data_smithi137_74180_object48990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-18T07:42:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.169823+0000", 2024-09-18T07:42:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 495.86002273700001, 2024-09-18T07:42:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.083783358, 2024-09-18T07:42:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:58.039 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:42:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:42:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153406 2024-09-18T07:42:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169823+0000", 2024-09-18T07:42:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.040 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169823+0000", 2024-09-18T07:42:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169826+0000", 2024-09-18T07:42:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1229999999999998e-06 2024-09-18T07:42:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169833+0000", 2024-09-18T07:42:58.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6869999999999994e-06 2024-09-18T07:42:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169851+0000", 2024-09-18T07:42:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7978999999999998e-05 2024-09-18T07:42:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169867+0000", 2024-09-18T07:42:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5610000000000001e-05 2024-09-18T07:42:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169944+0000", 2024-09-18T07:42:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6998999999999999e-05 2024-09-18T07:42:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.170029+0000", 2024-09-18T07:42:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4765999999999994e-05 2024-09-18T07:42:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:42:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.170285+0000", 2024-09-18T07:42:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025587199999999998 2024-09-18T07:42:58.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.171936+0000", 2024-09-18T07:42:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001651189 2024-09-18T07:42:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.194310+0000", 2024-09-18T07:42:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.022374616999999999 2024-09-18T07:42:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238955+0000", 2024-09-18T07:42:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.044644716000000001 2024-09-18T07:42:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:42:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238967+0000", 2024-09-18T07:42:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1933000000000001e-05 2024-09-18T07:42:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.045 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.253578+0000", 2024-09-18T07:42:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0146105599999999 2024-09-18T07:42:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.253607+0000", 2024-09-18T07:42:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9306999999999999e-05 2024-09-18T07:42:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.046 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153419 7.17 7:e887289b:::benchmark_data_smithi137_74180_object49003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-18T07:42:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.193652+0000", 2024-09-18T07:42:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 495.83619391299999, 2024-09-18T07:42:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0605730499999999, 2024-09-18T07:42:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:58.047 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:42:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:42:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153419 2024-09-18T07:42:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193652+0000", 2024-09-18T07:42:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.048 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193652+0000", 2024-09-18T07:42:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193654+0000", 2024-09-18T07:42:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.683e-06 2024-09-18T07:42:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193658+0000", 2024-09-18T07:42:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6619999999999998e-06 2024-09-18T07:42:58.049 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193667+0000", 2024-09-18T07:42:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0389999999999993e-06 2024-09-18T07:42:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193671+0000", 2024-09-18T07:42:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6789999999999996e-06 2024-09-18T07:42:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.050 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193715+0000", 2024-09-18T07:42:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3186000000000003e-05 2024-09-18T07:42:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193799+0000", 2024-09-18T07:42:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4172999999999995e-05 2024-09-18T07:42:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:42:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.194037+0000", 2024-09-18T07:42:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023850299999999999 2024-09-18T07:42:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.051 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.195050+0000", 2024-09-18T07:42:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010123269999999999 2024-09-18T07:42:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.198824+0000", 2024-09-18T07:42:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0037742840000000001 2024-09-18T07:42:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238955+0000", 2024-09-18T07:42:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.040130714999999997 2024-09-18T07:42:58.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:42:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238967+0000", 2024-09-18T07:42:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2665e-05 2024-09-18T07:42:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254201+0000", 2024-09-18T07:42:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0152341279999999 2024-09-18T07:42:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254225+0000", 2024-09-18T07:42:58.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4006e-05 2024-09-18T07:42:58.054 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.054 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:58.054 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.054 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.054 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.054 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153453 7.7 7.146f22e7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:42:58.054 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.282355+0000", 2024-09-18T07:42:58.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:58.055 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 495.74749102599998, 2024-09-18T07:42:58.055 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97169607199999997, 2024-09-18T07:42:58.055 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:58.056 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:58.056 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:42:58.056 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:58.056 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:42:58.056 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:58.056 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153453 2024-09-18T07:42:58.056 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.056 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:58.056 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.056 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:58.056 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282355+0000", 2024-09-18T07:42:58.056 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.057 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.057 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.057 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:58.057 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282355+0000", 2024-09-18T07:42:58.057 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.057 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.057 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.057 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:58.057 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282357+0000", 2024-09-18T07:42:58.057 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2110000000000001e-06 2024-09-18T07:42:58.057 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.057 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.057 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:58.058 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282363+0000", 2024-09-18T07:42:58.058 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6740000000000002e-06 2024-09-18T07:42:58.058 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.058 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.058 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:58.058 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282377+0000", 2024-09-18T07:42:58.058 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4039000000000001e-05 2024-09-18T07:42:58.058 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.058 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.058 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:58.058 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282384+0000", 2024-09-18T07:42:58.058 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5769999999999999e-06 2024-09-18T07:42:58.058 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.058 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.059 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.059 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282438+0000", 2024-09-18T07:42:58.059 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4083999999999999e-05 2024-09-18T07:42:58.059 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.059 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:42:58.059 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282445+0000", 2024-09-18T07:42:58.059 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7030000000000006e-06 2024-09-18T07:42:58.059 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.059 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.059 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254044+0000", 2024-09-18T07:42:58.059 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97159860200000003 2024-09-18T07:42:58.059 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.060 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.060 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:58.060 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254051+0000", 2024-09-18T07:42:58.060 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.182e-06 2024-09-18T07:42:58.060 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.060 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:58.060 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.060 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.060 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.060 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153481 7.7 7.33bf19e7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:42:58.060 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.317589+0000", 2024-09-18T07:42:58.060 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 495.71225702200002, 2024-09-18T07:42:58.060 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93650330299999995, 2024-09-18T07:42:58.061 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:58.061 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:58.061 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:42:58.061 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:58.061 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:42:58.061 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:58.061 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153481 2024-09-18T07:42:58.061 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.061 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:58.061 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.061 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:58.061 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317589+0000", 2024-09-18T07:42:58.061 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.062 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.062 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.062 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:58.062 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317589+0000", 2024-09-18T07:42:58.062 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.062 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.062 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.062 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:58.062 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317590+0000", 2024-09-18T07:42:58.062 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6499999999999999e-07 2024-09-18T07:42:58.062 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.062 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.063 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:58.063 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317593+0000", 2024-09-18T07:42:58.063 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7870000000000002e-06 2024-09-18T07:42:58.063 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.063 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.063 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:58.063 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317604+0000", 2024-09-18T07:42:58.063 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0953e-05 2024-09-18T07:42:58.063 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.063 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.063 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:58.063 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317609+0000", 2024-09-18T07:42:58.063 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4099999999999999e-06 2024-09-18T07:42:58.064 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.064 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.064 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.064 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317654+0000", 2024-09-18T07:42:58.064 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5333000000000001e-05 2024-09-18T07:42:58.064 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.064 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.064 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:42:58.064 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317661+0000", 2024-09-18T07:42:58.064 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8310000000000003e-06 2024-09-18T07:42:58.064 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.064 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.064 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.065 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254083+0000", 2024-09-18T07:42:58.065 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93642186500000002 2024-09-18T07:42:58.065 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.065 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.065 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:58.065 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254093+0000", 2024-09-18T07:42:58.065 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4590000000000004e-06 2024-09-18T07:42:58.065 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.065 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:58.065 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.065 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.065 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.065 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153485 7.7 7.c6dac07 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:42:58.066 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.328201+0000", 2024-09-18T07:42:58.066 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 495.701645085, 2024-09-18T07:42:58.066 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.925907068, 2024-09-18T07:42:58.066 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:58.066 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:58.066 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:42:58.066 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:58.066 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:42:58.066 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:58.066 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153485 2024-09-18T07:42:58.066 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.066 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:58.066 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.066 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:58.067 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328201+0000", 2024-09-18T07:42:58.067 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.067 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.067 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.067 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:58.067 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328201+0000", 2024-09-18T07:42:58.067 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.067 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.067 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.067 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:58.067 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328203+0000", 2024-09-18T07:42:58.067 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3769999999999999e-06 2024-09-18T07:42:58.067 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.067 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.068 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:58.068 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328207+0000", 2024-09-18T07:42:58.068 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5120000000000002e-06 2024-09-18T07:42:58.068 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.068 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.068 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:58.068 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328217+0000", 2024-09-18T07:42:58.069 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0438000000000001e-05 2024-09-18T07:42:58.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.069 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.069 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:58.069 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328223+0000", 2024-09-18T07:42:58.069 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7130000000000002e-06 2024-09-18T07:42:58.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.069 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.069 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.069 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328270+0000", 2024-09-18T07:42:58.069 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7243000000000002e-05 2024-09-18T07:42:58.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.069 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.070 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:42:58.070 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328277+0000", 2024-09-18T07:42:58.070 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9229999999999996e-06 2024-09-18T07:42:58.070 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.070 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.070 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.070 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254104+0000", 2024-09-18T07:42:58.070 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92582690099999998 2024-09-18T07:42:58.070 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.070 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.070 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:58.070 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254108+0000", 2024-09-18T07:42:58.070 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9609999999999999e-06 2024-09-18T07:42:58.070 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.071 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:58.071 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.071 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.071 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153505 7.7 7.fb8b6b67 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:42:58.071 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.350178+0000", 2024-09-18T07:42:58.071 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 495.67966825600001, 2024-09-18T07:42:58.071 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90398860000000003, 2024-09-18T07:42:58.071 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:58.071 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:58.071 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:42:58.071 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:58.071 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:42:58.072 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:58.072 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153505 2024-09-18T07:42:58.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.072 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:58.072 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.072 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:58.072 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350178+0000", 2024-09-18T07:42:58.072 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.072 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.072 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:58.072 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350178+0000", 2024-09-18T07:42:58.072 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.073 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:58.073 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350179+0000", 2024-09-18T07:42:58.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.454e-06 2024-09-18T07:42:58.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.073 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:58.073 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350183+0000", 2024-09-18T07:42:58.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8480000000000003e-06 2024-09-18T07:42:58.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.073 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:58.074 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350192+0000", 2024-09-18T07:42:58.074 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0899999999999994e-06 2024-09-18T07:42:58.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.074 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.074 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:58.074 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350200+0000", 2024-09-18T07:42:58.074 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.328e-06 2024-09-18T07:42:58.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.074 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.074 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.074 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350252+0000", 2024-09-18T07:42:58.074 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2589000000000001e-05 2024-09-18T07:42:58.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.074 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:42:58.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350263+0000", 2024-09-18T07:42:58.075 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0831e-05 2024-09-18T07:42:58.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.075 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254124+0000", 2024-09-18T07:42:58.075 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90386114100000003 2024-09-18T07:42:58.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.075 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:58.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254167+0000", 2024-09-18T07:42:58.075 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2318999999999999e-05 2024-09-18T07:42:58.075 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.076 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:58.076 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.076 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.076 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.076 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153519 7.17 7.5ef16c17 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:42:58.076 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.363819+0000", 2024-09-18T07:42:58.076 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 495.66602750300001, 2024-09-18T07:42:58.076 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.89046919899999999, 2024-09-18T07:42:58.076 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:58.076 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:58.076 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:42:58.076 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:58.076 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:42:58.077 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:58.077 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153519 2024-09-18T07:42:58.077 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.077 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:58.077 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.077 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:58.077 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363819+0000", 2024-09-18T07:42:58.077 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.077 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.077 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.077 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:58.077 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363819+0000", 2024-09-18T07:42:58.077 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.078 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.078 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.078 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:58.078 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363820+0000", 2024-09-18T07:42:58.078 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.325e-06 2024-09-18T07:42:58.078 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.078 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.078 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:58.078 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363824+0000", 2024-09-18T07:42:58.078 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2459999999999997e-06 2024-09-18T07:42:58.078 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.078 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.078 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:58.078 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363834+0000", 2024-09-18T07:42:58.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6399999999999992e-06 2024-09-18T07:42:58.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:58.079 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363838+0000", 2024-09-18T07:42:58.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2509999999999996e-06 2024-09-18T07:42:58.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.079 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363900+0000", 2024-09-18T07:42:58.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1902000000000005e-05 2024-09-18T07:42:58.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.080 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:42:58.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363910+0000", 2024-09-18T07:42:58.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6879999999999994e-06 2024-09-18T07:42:58.080 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.080 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.080 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254254+0000", 2024-09-18T07:42:58.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.89034411800000002 2024-09-18T07:42:58.080 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.080 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.080 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:58.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254288+0000", 2024-09-18T07:42:58.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4029000000000002e-05 2024-09-18T07:42:58.080 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.081 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:58.081 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.081 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.081 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.081 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:162338 7.c 7:39946bc9:::benchmark_data_smithi137_74180_object57922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:42:58.081 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:01.426143+0000", 2024-09-18T07:42:58.081 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 476.60370369200001, 2024-09-18T07:42:58.081 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0750540070000001, 2024-09-18T07:42:58.081 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:58.081 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:58.081 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:58.081 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:58.081 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:42:58.081 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:58.082 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 162338 2024-09-18T07:42:58.082 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.082 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:58.082 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.082 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:58.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426143+0000", 2024-09-18T07:42:58.082 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.082 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.082 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.082 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:58.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426143+0000", 2024-09-18T07:42:58.082 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.082 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.083 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.083 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:58.083 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426145+0000", 2024-09-18T07:42:58.083 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4059999999999999e-06 2024-09-18T07:42:58.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.083 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.083 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:58.083 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426151+0000", 2024-09-18T07:42:58.083 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1260000000000003e-06 2024-09-18T07:42:58.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.083 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.083 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:58.083 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426174+0000", 2024-09-18T07:42:58.083 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3255e-05 2024-09-18T07:42:58.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.084 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.084 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:58.084 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426182+0000", 2024-09-18T07:42:58.084 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4189999999999996e-06 2024-09-18T07:42:58.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.084 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.084 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.084 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426256+0000", 2024-09-18T07:42:58.084 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4713000000000001e-05 2024-09-18T07:42:58.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.084 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.084 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:42:58.084 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426270+0000", 2024-09-18T07:42:58.085 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3405e-05 2024-09-18T07:42:58.085 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.085 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.085 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.085 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.491167+0000", 2024-09-18T07:42:58.085 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.064897582 2024-09-18T07:42:58.085 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.085 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.085 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:58.085 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.491280+0000", 2024-09-18T07:42:58.085 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011243099999999999 2024-09-18T07:42:58.085 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.085 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.086 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3,7", 2024-09-18T07:42:58.086 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.491357+0000", 2024-09-18T07:42:58.086 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7385999999999994e-05 2024-09-18T07:42:58.086 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.086 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.086 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:58.086 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496060+0000", 2024-09-18T07:42:58.086 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0047024600000000003 2024-09-18T07:42:58.086 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.086 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.086 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:58.086 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496072+0000", 2024-09-18T07:42:58.086 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2294e-05 2024-09-18T07:42:58.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.087 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.087 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:58.087 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496109+0000", 2024-09-18T07:42:58.087 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7218999999999997e-05 2024-09-18T07:42:58.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.087 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.087 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:58.087 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501071+0000", 2024-09-18T07:42:58.087 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0049614660000000003 2024-09-18T07:42:58.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.087 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.087 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:58.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501111+0000", 2024-09-18T07:42:58.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0776999999999999e-05 2024-09-18T07:42:58.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:58.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501197+0000", 2024-09-18T07:42:58.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5068000000000006e-05 2024-09-18T07:42:58.088 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.088 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:58.088 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.088 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:162350 7.c 7:3843094b:::benchmark_data_smithi137_74180_object57934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:42:58.089 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:01.433525+0000", 2024-09-18T07:42:58.089 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 476.59632088199999, 2024-09-18T07:42:58.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0678760330000001, 2024-09-18T07:42:58.089 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:42:58.089 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:42:58.089 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:42:58.089 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:42:58.089 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:42:58.089 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:42:58.089 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 162350 2024-09-18T07:42:58.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.089 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:42:58.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:42:58.090 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433525+0000", 2024-09-18T07:42:58.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.090 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.090 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:42:58.090 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433525+0000", 2024-09-18T07:42:58.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:42:58.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.090 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.090 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:42:58.090 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433527+0000", 2024-09-18T07:42:58.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.812e-06 2024-09-18T07:42:58.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:42:58.091 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433533+0000", 2024-09-18T07:42:58.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2920000000000001e-06 2024-09-18T07:42:58.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:42:58.091 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433544+0000", 2024-09-18T07:42:58.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0339999999999999e-05 2024-09-18T07:42:58.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:42:58.091 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433551+0000", 2024-09-18T07:42:58.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6490000000000008e-06 2024-09-18T07:42:58.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433620+0000", 2024-09-18T07:42:58.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8957999999999999e-05 2024-09-18T07:42:58.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:42:58.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433631+0000", 2024-09-18T07:42:58.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0397000000000001e-05 2024-09-18T07:42:58.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:42:58.093 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.493805+0000", 2024-09-18T07:42:58.093 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0601744559999999 2024-09-18T07:42:58.093 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.093 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.093 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:42:58.093 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.493907+0000", 2024-09-18T07:42:58.093 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000101377 2024-09-18T07:42:58.093 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.093 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.093 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3,7", 2024-09-18T07:42:58.093 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.493992+0000", 2024-09-18T07:42:58.093 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5672000000000002e-05 2024-09-18T07:42:58.093 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.093 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.094 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:58.094 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496146+0000", 2024-09-18T07:42:58.094 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021537470000000001 2024-09-18T07:42:58.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.094 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.094 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:42:58.094 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.498397+0000", 2024-09-18T07:42:58.094 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002250742 2024-09-18T07:42:58.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.094 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.094 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:58.094 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501029+0000", 2024-09-18T07:42:58.094 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026321780000000002 2024-09-18T07:42:58.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.095 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:42:58.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501274+0000", 2024-09-18T07:42:58.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000245472 2024-09-18T07:42:58.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.095 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:42:58.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501321+0000", 2024-09-18T07:42:58.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6405e-05 2024-09-18T07:42:58.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:42:58.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:42:58.095 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:42:58.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501401+0000", 2024-09-18T07:42:58.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0536000000000003e-05 2024-09-18T07:42:58.096 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.096 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:58.096 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.096 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:42:58.096 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:42:58.096 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:42:58.096 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:58 smithi203 ceph-mon[32237]: 6.1f scrub starts 2024-09-18T07:42:58.096 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:58 smithi203 ceph-mon[32237]: 6.1f scrub ok 2024-09-18T07:42:58.096 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:58 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3690749623' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:42:58.097 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:58 smithi203 ceph-mon[32237]: from='client.35969 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:42:58.097 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:58 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3153572866' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:58.117 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:58.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:58.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:58.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:58.241 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:58.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:58.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:58.288 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:58 smithi137 ceph-mon[29408]: 6.1f scrub starts 2024-09-18T07:42:58.288 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:58 smithi137 ceph-mon[29408]: 6.1f scrub ok 2024-09-18T07:42:58.289 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:58 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3690749623' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:42:58.289 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:58 smithi137 ceph-mon[29408]: from='client.35969 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:42:58.289 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:58 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3153572866' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:58.289 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:58 smithi137 ceph-mon[39175]: 6.1f scrub starts 2024-09-18T07:42:58.290 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:58 smithi137 ceph-mon[39175]: 6.1f scrub ok 2024-09-18T07:42:58.290 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:58 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3690749623' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:42:58.290 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:58 smithi137 ceph-mon[39175]: from='client.35969 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:42:58.290 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:58 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3153572866' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:42:58.343 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:58.366 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:58.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:58.468 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:58.490 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:58.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:58.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:58.554 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:42:58.554 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:42:58.592 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:58.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:58.614 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:58.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:58.715 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:58.725 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:42:58.736 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:58.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:58.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:58.838 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:58.859 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:58.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:58.961 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:58.968 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T07:42:58.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:58.983 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:58.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:59.084 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:42:59.105 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:59.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:59.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:42:59.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:42:59.228 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:59.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:59.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:42:59.237 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:59 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:59.237 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:59 smithi137 ceph-mon[29408]: pgmap v637: 158 pgs: 42 active+undersized, 22 active+undersized+degraded, 94 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 25 KiB/s wr, 968 op/s; 83/597 objects degraded (13.903%); 18 KiB/s, 0 objects/s recovering 2024-09-18T07:42:59.237 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:59 smithi137 ceph-mon[29408]: Health check failed: Degraded data redundancy: 83/597 objects degraded (13.903%), 22 pgs degraded (PG_DEGRADED) 2024-09-18T07:42:59.238 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:59 smithi137 ceph-mon[29408]: from='client.35969 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T07:42:59.238 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:59 smithi137 ceph-mon[29408]: osdmap e362: 8 total, 7 up, 8 in 2024-09-18T07:42:59.238 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:59 smithi137 ceph-mon[29408]: 5.14 scrub starts 2024-09-18T07:42:59.238 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:42:59 smithi137 ceph-mon[29408]: 5.14 scrub ok 2024-09-18T07:42:59.238 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:59 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:59.238 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:59 smithi137 ceph-mon[39175]: pgmap v637: 158 pgs: 42 active+undersized, 22 active+undersized+degraded, 94 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 25 KiB/s wr, 968 op/s; 83/597 objects degraded (13.903%); 18 KiB/s, 0 objects/s recovering 2024-09-18T07:42:59.238 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:59 smithi137 ceph-mon[39175]: Health check failed: Degraded data redundancy: 83/597 objects degraded (13.903%), 22 pgs degraded (PG_DEGRADED) 2024-09-18T07:42:59.238 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:59 smithi137 ceph-mon[39175]: from='client.35969 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T07:42:59.238 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:59 smithi137 ceph-mon[39175]: osdmap e362: 8 total, 7 up, 8 in 2024-09-18T07:42:59.238 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:59 smithi137 ceph-mon[39175]: 5.14 scrub starts 2024-09-18T07:42:59.238 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:42:59 smithi137 ceph-mon[39175]: 5.14 scrub ok 2024-09-18T07:42:59.271 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-18T07:42:59.272 INFO:tasks.radosbench:Beginning radosbench... 2024-09-18T07:42:59.272 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-18T07:42:59.272 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-09-18T07:42:59.306 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:59.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:59.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:59.373 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:59.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:59.435 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:59 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:42:59.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:59 smithi203 ceph-mon[32237]: pgmap v637: 158 pgs: 42 active+undersized, 22 active+undersized+degraded, 94 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 25 KiB/s wr, 968 op/s; 83/597 objects degraded (13.903%); 18 KiB/s, 0 objects/s recovering 2024-09-18T07:42:59.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:59 smithi203 ceph-mon[32237]: Health check failed: Degraded data redundancy: 83/597 objects degraded (13.903%), 22 pgs degraded (PG_DEGRADED) 2024-09-18T07:42:59.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:59 smithi203 ceph-mon[32237]: from='client.35969 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T07:42:59.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:59 smithi203 ceph-mon[32237]: osdmap e362: 8 total, 7 up, 8 in 2024-09-18T07:42:59.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:59 smithi203 ceph-mon[32237]: 5.14 scrub starts 2024-09-18T07:42:59.436 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:42:59 smithi203 ceph-mon[32237]: 5.14 scrub ok 2024-09-18T07:42:59.467 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.4.service' 2024-09-18T07:42:59.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:59.496 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:59.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:59.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:42:59.541 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T07:42:59.541 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-18T07:42:59.541 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.4 2024-09-18T07:42:59.541 INFO:tasks.thrashosds.thrasher:remote for osd 4 is ubuntu@smithi203.front.sepia.ceph.com 2024-09-18T07:42:59.542 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -n osd.4 --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op list-pgs 2024-09-18T07:42:59.569 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:42:59.598 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:42:59.619 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:59.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:59.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:42:59.721 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:42:59.743 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:59.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:59.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:42:59.814 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/osd.4/config 2024-09-18T07:42:59.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:42:59.867 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:42:59.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:59.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:42:59.869 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:42:59.969 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:42:59.990 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:42:59.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:42:59.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:00.047 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:43:00.048 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:43:00.048 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-09-18T07:43:00.048 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:43:00.093 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:00.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:00.116 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:00.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:00.146 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:00 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:00.147 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:00 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:00.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:00 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:00.218 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:00.239 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:00.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:00.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:00.341 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:00.363 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:00.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:00.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:00.465 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:00.488 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:00.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:00.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:00.556 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:43:00.590 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:00.620 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:00.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:00.722 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:00.744 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:00.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:00.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:00.755 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T07:43:00.809 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:00.846 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:00.867 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:00.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:00.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:00.969 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:00.991 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:00.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:00.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:01.041 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:43:01.041 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'unique_pool_3' created 2024-09-18T07:43:01.093 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:01.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:01.116 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:01.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:01.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:01.239 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:01.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:01.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:01.341 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:01.342 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:01 smithi203 ceph-mon[32237]: pgmap v639: 142 pgs: 39 active+undersized, 22 active+undersized+degraded, 81 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 op/s; 83/597 objects degraded (13.903%); 18 KiB/s, 0 objects/s recovering 2024-09-18T07:43:01.342 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:01 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:43:01.342 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:01 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:43:01.342 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:01 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2645426852' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:01.342 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:01 smithi203 ceph-mon[32237]: from='client.35981 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:01.342 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:01 smithi203 ceph-mon[32237]: osdmap e363: 8 total, 7 up, 8 in 2024-09-18T07:43:01.342 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:01 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2147548339' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-18T07:43:01.342 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:01 smithi203 ceph-mon[32237]: from='client.35987 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-18T07:43:01.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:01.365 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:01.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:01.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:01 smithi137 ceph-mon[39175]: pgmap v639: 142 pgs: 39 active+undersized, 22 active+undersized+degraded, 81 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 op/s; 83/597 objects degraded (13.903%); 18 KiB/s, 0 objects/s recovering 2024-09-18T07:43:01.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:01 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:43:01.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:01 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:43:01.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:01 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2645426852' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:01.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:01 smithi137 ceph-mon[39175]: from='client.35981 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:01.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:01 smithi137 ceph-mon[39175]: osdmap e363: 8 total, 7 up, 8 in 2024-09-18T07:43:01.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:01 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2147548339' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-18T07:43:01.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:01 smithi137 ceph-mon[39175]: from='client.35987 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-18T07:43:01.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:01 smithi137 ceph-mon[29408]: pgmap v639: 142 pgs: 39 active+undersized, 22 active+undersized+degraded, 81 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 op/s; 83/597 objects degraded (13.903%); 18 KiB/s, 0 objects/s recovering 2024-09-18T07:43:01.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:01 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:43:01.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:01 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:43:01.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:01 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2645426852' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:01.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:01 smithi137 ceph-mon[29408]: from='client.35981 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:01.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:01 smithi137 ceph-mon[29408]: osdmap e363: 8 total, 7 up, 8 in 2024-09-18T07:43:01.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:01 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2147548339' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-18T07:43:01.423 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:01 smithi137 ceph-mon[29408]: from='client.35987 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-18T07:43:01.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:01.483 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:43:01.487 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:01.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:01.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:01.589 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:01.611 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:01.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:01.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:01.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:01.737 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:01.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:01.838 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:01.859 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:01.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:01.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:01.944 INFO:teuthology.orchestra.run.smithi203.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-18T07:43:01.961 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:01.982 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:01.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:01.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:02.084 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:02.105 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:02.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:02.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:02.155 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:02 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:43:02.155 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:02 smithi203 ceph-mon[32237]: from='client.35981 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:43:02.155 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:02 smithi203 ceph-mon[32237]: from='client.35987 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-18T07:43:02.155 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:02 smithi203 ceph-mon[32237]: osdmap e364: 8 total, 7 up, 8 in 2024-09-18T07:43:02.155 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:02 smithi203 ceph-mon[32237]: 2.1 scrub starts 2024-09-18T07:43:02.155 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:02 smithi203 ceph-mon[32237]: 2.1 scrub ok 2024-09-18T07:43:02.155 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:02 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:43:02.155 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:02 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:43:02.155 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:02 smithi203 ceph-mon[32237]: 6.b scrub starts 2024-09-18T07:43:02.156 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:02 smithi203 ceph-mon[32237]: 6.b scrub ok 2024-09-18T07:43:02.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:02.232 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:02.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:02.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:02.334 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:02.356 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:02.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:02.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:43:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[29408]: from='client.35981 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:43:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[29408]: from='client.35987 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-18T07:43:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[29408]: osdmap e364: 8 total, 7 up, 8 in 2024-09-18T07:43:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[29408]: 2.1 scrub starts 2024-09-18T07:43:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[29408]: 2.1 scrub ok 2024-09-18T07:43:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:43:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:43:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[29408]: 6.b scrub starts 2024-09-18T07:43:02.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[29408]: 6.b scrub ok 2024-09-18T07:43:02.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:43:02.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[39175]: from='client.35981 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:43:02.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[39175]: from='client.35987 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-18T07:43:02.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[39175]: osdmap e364: 8 total, 7 up, 8 in 2024-09-18T07:43:02.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[39175]: 2.1 scrub starts 2024-09-18T07:43:02.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[39175]: 2.1 scrub ok 2024-09-18T07:43:02.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:43:02.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:43:02.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[39175]: 6.b scrub starts 2024-09-18T07:43:02.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:02 smithi137 ceph-mon[39175]: 6.b scrub ok 2024-09-18T07:43:02.442 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:43:02.442 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:43:02.442 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:43:02.442 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:43:02.442 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:43:02.458 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:02.479 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:02.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:02.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:02.581 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:02.602 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:02.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:02.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:02.729 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:02.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:02.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:02.831 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:02.855 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:02.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:02.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:02.957 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:02.979 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:02.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:03.081 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:03.103 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:03.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:03.161 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:03 smithi203 ceph-mon[32237]: 6.19 scrub starts 2024-09-18T07:43:03.161 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:03 smithi203 ceph-mon[32237]: 6.19 scrub ok 2024-09-18T07:43:03.161 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:03 smithi203 ceph-mon[32237]: pgmap v642: 158 pgs: 2 creating+peering, 14 unknown, 39 active+undersized, 22 active+undersized+degraded, 81 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 1 op/s; 83/597 objects degraded (13.903%); 18 KiB/s, 0 objects/s recovering 2024-09-18T07:43:03.161 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:03 smithi203 ceph-mon[32237]: osdmap e365: 8 total, 7 up, 8 in 2024-09-18T07:43:03.161 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:03 smithi203 ceph-mon[32237]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:43:03.161 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:03 smithi203 ceph-mon[32237]: osdmap e366: 8 total, 7 up, 8 in 2024-09-18T07:43:03.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:03 smithi137 ceph-mon[29408]: 6.19 scrub starts 2024-09-18T07:43:03.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:03 smithi137 ceph-mon[29408]: 6.19 scrub ok 2024-09-18T07:43:03.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:03 smithi137 ceph-mon[29408]: pgmap v642: 158 pgs: 2 creating+peering, 14 unknown, 39 active+undersized, 22 active+undersized+degraded, 81 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 1 op/s; 83/597 objects degraded (13.903%); 18 KiB/s, 0 objects/s recovering 2024-09-18T07:43:03.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:03 smithi137 ceph-mon[29408]: osdmap e365: 8 total, 7 up, 8 in 2024-09-18T07:43:03.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:03 smithi137 ceph-mon[29408]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:43:03.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:03 smithi137 ceph-mon[29408]: osdmap e366: 8 total, 7 up, 8 in 2024-09-18T07:43:03.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:03 smithi137 ceph-mon[39175]: 6.19 scrub starts 2024-09-18T07:43:03.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:03 smithi137 ceph-mon[39175]: 6.19 scrub ok 2024-09-18T07:43:03.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:03 smithi137 ceph-mon[39175]: pgmap v642: 158 pgs: 2 creating+peering, 14 unknown, 39 active+undersized, 22 active+undersized+degraded, 81 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 1 op/s; 83/597 objects degraded (13.903%); 18 KiB/s, 0 objects/s recovering 2024-09-18T07:43:03.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:03 smithi137 ceph-mon[39175]: osdmap e365: 8 total, 7 up, 8 in 2024-09-18T07:43:03.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:03 smithi137 ceph-mon[39175]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:43:03.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:03 smithi137 ceph-mon[39175]: osdmap e366: 8 total, 7 up, 8 in 2024-09-18T07:43:03.189 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T07:43:03.204 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:03.253 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:03.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:03.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:03.286 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:43:03.286 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:43:03.292 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:43:03.292 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:43:03.308 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:43:03.308 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:43:03.324 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:43:03.324 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:43:03.324 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-18T07:43:03.341 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:43:03.341 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:43:03.355 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:03.410 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:43:03.411 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:43:03.415 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:03.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:03.429 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:43:03.429 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:43:03.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:03.538 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:03.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:03.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:03.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:03.662 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:03.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:03.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:03.675 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:43:03.763 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:03.785 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:03.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:03.886 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:03.907 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:03.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:03.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:04.009 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:04.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:04.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:04.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:04.063 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:04 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3327926661' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:04.131 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:04.155 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:04.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:04.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:04.209 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:43:04.210 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:43:04.244 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:43:04.245 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:43:04.257 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:04.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:04.282 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:04.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:04.383 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:04.384 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:04 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3327926661' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:04.385 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:04 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3327926661' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:04.408 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:04.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:04.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:04.510 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:04.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:04.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:04.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:04.634 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:04.655 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:04.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:04.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:04.757 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:04.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:04.781 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:04.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:04.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-18T07:43:04.882 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:04.903 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:04.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:04.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:04.929 INFO:teuthology.orchestra.run.smithi203.stdout:9.e 2024-09-18T07:43:04.929 INFO:teuthology.orchestra.run.smithi203.stdout:9.c 2024-09-18T07:43:04.929 INFO:teuthology.orchestra.run.smithi203.stdout:9.4 2024-09-18T07:43:04.929 INFO:teuthology.orchestra.run.smithi203.stdout:6.f 2024-09-18T07:43:04.929 INFO:teuthology.orchestra.run.smithi203.stdout:6.e 2024-09-18T07:43:04.929 INFO:teuthology.orchestra.run.smithi203.stdout:4.5 2024-09-18T07:43:04.929 INFO:teuthology.orchestra.run.smithi203.stdout:4.2 2024-09-18T07:43:04.929 INFO:teuthology.orchestra.run.smithi203.stdout:4.1 2024-09-18T07:43:04.929 INFO:teuthology.orchestra.run.smithi203.stdout:3.15 2024-09-18T07:43:04.929 INFO:teuthology.orchestra.run.smithi203.stdout:4.13 2024-09-18T07:43:04.930 INFO:teuthology.orchestra.run.smithi203.stdout:4.18 2024-09-18T07:43:04.930 INFO:teuthology.orchestra.run.smithi203.stdout:1.2 2024-09-18T07:43:04.930 INFO:teuthology.orchestra.run.smithi203.stdout:3.f 2024-09-18T07:43:04.930 INFO:teuthology.orchestra.run.smithi203.stdout:3.e 2024-09-18T07:43:04.930 INFO:teuthology.orchestra.run.smithi203.stdout:3.9 2024-09-18T07:43:04.930 INFO:teuthology.orchestra.run.smithi203.stdout:1.3 2024-09-18T07:43:04.930 INFO:teuthology.orchestra.run.smithi203.stdout:3.6 2024-09-18T07:43:04.930 INFO:teuthology.orchestra.run.smithi203.stdout:3.3 2024-09-18T07:43:04.930 INFO:teuthology.orchestra.run.smithi203.stdout:3.11 2024-09-18T07:43:04.930 INFO:teuthology.orchestra.run.smithi203.stdout:3.1 2024-09-18T07:43:04.930 INFO:teuthology.orchestra.run.smithi203.stdout:5.a 2024-09-18T07:43:04.930 INFO:teuthology.orchestra.run.smithi203.stdout:4.9 2024-09-18T07:43:04.930 INFO:teuthology.orchestra.run.smithi203.stdout:3.19 2024-09-18T07:43:04.931 INFO:teuthology.orchestra.run.smithi203.stdout:3.1d 2024-09-18T07:43:04.931 INFO:teuthology.orchestra.run.smithi203.stdout:3.13 2024-09-18T07:43:04.931 INFO:teuthology.orchestra.run.smithi203.stdout:5.18 2024-09-18T07:43:04.931 INFO:teuthology.orchestra.run.smithi203.stdout:4.1b 2024-09-18T07:43:04.931 INFO:teuthology.orchestra.run.smithi203.stdout:3.a 2024-09-18T07:43:04.931 INFO:teuthology.orchestra.run.smithi203.stdout:3.14 2024-09-18T07:43:04.931 INFO:teuthology.orchestra.run.smithi203.stdout:3.1b 2024-09-18T07:43:04.931 INFO:teuthology.orchestra.run.smithi203.stdout:3.1a 2024-09-18T07:43:04.931 INFO:teuthology.orchestra.run.smithi203.stdout:3.b 2024-09-18T07:43:04.931 INFO:teuthology.orchestra.run.smithi203.stdout:6.14 2024-09-18T07:43:04.931 INFO:teuthology.orchestra.run.smithi203.stdout:6.6 2024-09-18T07:43:04.931 INFO:teuthology.orchestra.run.smithi203.stdout:5.0 2024-09-18T07:43:04.931 INFO:teuthology.orchestra.run.smithi203.stdout:1.5 2024-09-18T07:43:04.932 INFO:teuthology.orchestra.run.smithi203.stdout:4.c 2024-09-18T07:43:04.932 INFO:teuthology.orchestra.run.smithi203.stdout:4.b 2024-09-18T07:43:04.932 INFO:teuthology.orchestra.run.smithi203.stdout:5.1 2024-09-18T07:43:04.932 INFO:teuthology.orchestra.run.smithi203.stdout:4.1d 2024-09-18T07:43:04.932 INFO:teuthology.orchestra.run.smithi203.stdout:4.d 2024-09-18T07:43:04.932 INFO:teuthology.orchestra.run.smithi203.stdout:4.e 2024-09-18T07:43:04.932 INFO:teuthology.orchestra.run.smithi203.stdout:4.f 2024-09-18T07:43:04.932 INFO:teuthology.orchestra.run.smithi203.stdout:5.10 2024-09-18T07:43:04.932 INFO:teuthology.orchestra.run.smithi203.stdout:5.11 2024-09-18T07:43:04.932 INFO:teuthology.orchestra.run.smithi203.stdout:5.1a 2024-09-18T07:43:04.932 INFO:teuthology.orchestra.run.smithi203.stdout:6.1d 2024-09-18T07:43:04.932 INFO:teuthology.orchestra.run.smithi203.stdout:5.1c 2024-09-18T07:43:04.933 INFO:teuthology.orchestra.run.smithi203.stdout:6.12 2024-09-18T07:43:04.933 INFO:teuthology.orchestra.run.smithi203.stdout:5.1d 2024-09-18T07:43:04.933 INFO:teuthology.orchestra.run.smithi203.stdout:5.1f 2024-09-18T07:43:04.933 INFO:teuthology.orchestra.run.smithi203.stdout:5.5 2024-09-18T07:43:04.933 INFO:teuthology.orchestra.run.smithi203.stdout:3.1c 2024-09-18T07:43:04.933 INFO:teuthology.orchestra.run.smithi203.stdout:5.9 2024-09-18T07:43:04.933 INFO:teuthology.orchestra.run.smithi203.stdout:5.c 2024-09-18T07:43:04.933 INFO:teuthology.orchestra.run.smithi203.stdout:6.17 2024-09-18T07:43:04.933 INFO:teuthology.orchestra.run.smithi203.stdout:5.e 2024-09-18T07:43:04.933 INFO:teuthology.orchestra.run.smithi203.stdout:5.f 2024-09-18T07:43:04.933 INFO:teuthology.orchestra.run.smithi203.stdout:6.1e 2024-09-18T07:43:04.933 INFO:teuthology.orchestra.run.smithi203.stdout:6.15 2024-09-18T07:43:04.933 INFO:teuthology.orchestra.run.smithi203.stdout:6.1a 2024-09-18T07:43:04.934 INFO:teuthology.orchestra.run.smithi203.stdout:6.2 2024-09-18T07:43:04.934 INFO:teuthology.orchestra.run.smithi203.stdout:4.1a 2024-09-18T07:43:04.934 INFO:teuthology.orchestra.run.smithi203.stdout:6.7 2024-09-18T07:43:04.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T07:43:05.005 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:05.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:05.028 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:05.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:05.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:05.150 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:05.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:05.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:05 smithi203 ceph-mon[32237]: pgmap v645: 158 pgs: 11 creating+peering, 42 active+undersized, 22 active+undersized+degraded, 83 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 83/597 objects degraded (13.903%) 2024-09-18T07:43:05.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:05.274 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:05.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:05.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:05.333 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:05 smithi137 ceph-mon[29408]: pgmap v645: 158 pgs: 11 creating+peering, 42 active+undersized, 22 active+undersized+degraded, 83 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 83/597 objects degraded (13.903%) 2024-09-18T07:43:05.333 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:05 smithi137 ceph-mon[39175]: pgmap v645: 158 pgs: 11 creating+peering, 42 active+undersized, 22 active+undersized+degraded, 83 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 83/597 objects degraded (13.903%) 2024-09-18T07:43:05.356 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:05.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:05.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:05.399 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:05.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:05.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:05.523 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:05.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:05.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:05.624 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:05.646 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:05.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:05.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:05.748 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:05.769 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:05.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:05.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:05.871 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -n osd.4 --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op export --pgid 1.2 --file /var/log/ceph/exp.1.2.4 2024-09-18T07:43:05.871 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:05.893 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:05.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:05.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:05.900 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:05.913 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:43:05.913 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:43:05.996 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:06.017 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:06.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:06.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:06.066 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/osd.4/config 2024-09-18T07:43:06.119 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:06.141 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:06.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:06.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:06.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:06 smithi137 ceph-mon[29408]: 3.c scrub starts 2024-09-18T07:43:06.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:06 smithi137 ceph-mon[29408]: 3.c scrub ok 2024-09-18T07:43:06.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:06 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:43:06.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:06 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:43:06.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:06 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:43:06.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:06 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:43:06.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:06 smithi137 ceph-mon[39175]: 3.c scrub starts 2024-09-18T07:43:06.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:06 smithi137 ceph-mon[39175]: 3.c scrub ok 2024-09-18T07:43:06.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:06 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:43:06.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:06 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:43:06.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:06 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:43:06.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:06 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:43:06.189 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:06 smithi203 ceph-mon[32237]: 3.c scrub starts 2024-09-18T07:43:06.189 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:06 smithi203 ceph-mon[32237]: 3.c scrub ok 2024-09-18T07:43:06.189 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:06 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:43:06.189 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:06 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:43:06.189 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:06 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:43:06.189 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:06 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:43:06.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:06.245 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:43:06.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:06.267 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:06.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:06.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:06.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:06.390 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:06.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:06.490 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:06.512 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:06.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:06.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:06.614 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:06.635 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:06.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:06.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:06.737 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:06.758 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:06.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:06.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:06.860 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:06.885 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:06.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:06.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:06.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T07:43:06.987 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:07.010 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:07.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:07.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:07.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:07.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:07.138 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:07.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:07.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:07 smithi137 ceph-mon[29408]: pgmap v646: 158 pgs: 11 creating+peering, 42 active+undersized, 22 active+undersized+degraded, 83 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 904 B/s rd, 0 op/s; 83/597 objects degraded (13.903%) 2024-09-18T07:43:07.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:07 smithi137 ceph-mon[39175]: pgmap v646: 158 pgs: 11 creating+peering, 42 active+undersized, 22 active+undersized+degraded, 83 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 904 B/s rd, 0 op/s; 83/597 objects degraded (13.903%) 2024-09-18T07:43:07.170 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:43:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:43:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:43:07.231 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:07.239 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:07.250 INFO:teuthology.orchestra.run.smithi203.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-18T07:43:07.261 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:07.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:07.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:07.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:43:07.362 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:07.371 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:07 smithi203 ceph-mon[32237]: pgmap v646: 158 pgs: 11 creating+peering, 42 active+undersized, 22 active+undersized+degraded, 83 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 904 B/s rd, 0 op/s; 83/597 objects degraded (13.903%) 2024-09-18T07:43:07.403 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:07.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:07.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:07.505 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:07.530 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:07.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:07.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:07.627 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:43:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:43:07.632 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:07.654 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:07.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:07.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:07.696 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:07.697 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:43:07.698 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:43:07.718 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:43:07.718 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:43:07.736 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:43:07.736 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:43:07.752 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:43:07.752 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:43:07.752 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-18T07:43:07.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:07.769 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:43:07.769 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:43:07.778 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:07.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:07.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:07.785 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:43:07.785 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:43:07.802 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:43:07.802 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:43:07.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:07.902 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:07.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:07.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:08.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:08.025 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:08.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:08.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:08.083 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-18T07:43:08.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:08.149 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:08.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:08.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:08.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:08 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1273976675' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:43:08.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:08 smithi203 ceph-mon[32237]: from='client.35999 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:43:08.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:08 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2300724115' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:08.251 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:08.272 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:08.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:08.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:08.374 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:08.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:08 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1273976675' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:43:08.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:08 smithi137 ceph-mon[29408]: from='client.35999 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:43:08.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:08 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2300724115' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:08.448 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:08 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1273976675' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:43:08.448 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:08 smithi137 ceph-mon[39175]: from='client.35999 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:43:08.449 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:08 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2300724115' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:08.470 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:08.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:08.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:08.572 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:08.593 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:08.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:08.695 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:08.717 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:08.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:08.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:08.819 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:08.841 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:08.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:08.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:08.943 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:08.964 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:08.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:08.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:09.066 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:09.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:09.089 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:09.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:09.107 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:09 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:09.107 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:09 smithi137 ceph-mon[39175]: pgmap v647: 158 pgs: 47 active+undersized, 22 active+undersized+degraded, 89 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 83/597 objects degraded (13.903%) 2024-09-18T07:43:09.107 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:09 smithi137 ceph-mon[39175]: from='client.35999 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:43:09.107 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:09 smithi137 ceph-mon[39175]: osdmap e367: 8 total, 7 up, 8 in 2024-09-18T07:43:09.108 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:09 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:09.108 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:09 smithi137 ceph-mon[29408]: pgmap v647: 158 pgs: 47 active+undersized, 22 active+undersized+degraded, 89 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 83/597 objects degraded (13.903%) 2024-09-18T07:43:09.108 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:09 smithi137 ceph-mon[29408]: from='client.35999 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:43:09.108 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:09 smithi137 ceph-mon[29408]: osdmap e367: 8 total, 7 up, 8 in 2024-09-18T07:43:09.114 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:43:09.114 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:43:09.114 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:43:09.114 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:43:09.114 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:43:09.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:09.212 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:09.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:09.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:09.313 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:09.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:09.336 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:09.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:09.337 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:09 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:09.337 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:09 smithi203 ceph-mon[32237]: pgmap v647: 158 pgs: 47 active+undersized, 22 active+undersized+degraded, 89 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 83/597 objects degraded (13.903%) 2024-09-18T07:43:09.337 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:09 smithi203 ceph-mon[32237]: from='client.35999 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:43:09.337 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:09 smithi203 ceph-mon[32237]: osdmap e367: 8 total, 7 up, 8 in 2024-09-18T07:43:09.337 INFO:teuthology.orchestra.run.smithi203.stderr:Exporting 1.2 info 1.2( v 73'79 (0'0,73'79] local-lis/les=274/275 n=1 ec=272/19 lis/c=274/274 les/c/f=275/275/0 sis=274) 2024-09-18T07:43:09.337 INFO:teuthology.orchestra.run.smithi203.stderr:Export successful 2024-09-18T07:43:09.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:09.459 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:09.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:09.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:09.561 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:09.583 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:09.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:09.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:09.685 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:09.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:09.707 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:09.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:09.747 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:43:09.747 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:43:09.775 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:09.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:09.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:09.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:09.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:09.869 DEBUG:teuthology.orchestra.run.smithi137:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-18T07:43:09.875 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-18T07:43:09.875 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-18T07:43:09.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:09.954 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:09.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:09.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:10.052 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:hints = 1 2024-09-18T07:43:10.053 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-18T07:43:10.053 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Object prefix: benchmark_data_smithi137_151160 2024-09-18T07:43:10.053 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:43:10.053 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 0 0 0 0 0 0 - 0 2024-09-18T07:43:10.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:10.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:10.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:10.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:10.192 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:10.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:10.226 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:10.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:10.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T07:43:10.327 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:10.360 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:10.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:10.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:10.434 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -n osd.4 --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --force --op remove --pgid 1.2 2024-09-18T07:43:10.462 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:10.510 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:10.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:10.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:10.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:10.673 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:10.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:10.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:10.744 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/osd.4/config 2024-09-18T07:43:10.775 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:10.813 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:10.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:10.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:10.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:10.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:10.958 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:10.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:10.989 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:11.053 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 1 15 3747 3732 233.235 233.25 0.00546196 0.0042733 2024-09-18T07:43:11.059 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:11.096 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:11.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:11.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:11.154 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:11 smithi137 ceph-mon[39175]: pgmap v649: 158 pgs: 47 active+undersized, 22 active+undersized+degraded, 89 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 668 B/s rd, 0 op/s; 83/597 objects degraded (13.903%) 2024-09-18T07:43:11.154 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:11 smithi137 ceph-mon[39175]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:43:11.154 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:11 smithi137 ceph-mon[39175]: 4.1f scrub starts 2024-09-18T07:43:11.154 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:11 smithi137 ceph-mon[39175]: 4.1f scrub ok 2024-09-18T07:43:11.154 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:11 smithi137 ceph-mon[29408]: pgmap v649: 158 pgs: 47 active+undersized, 22 active+undersized+degraded, 89 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 668 B/s rd, 0 op/s; 83/597 objects degraded (13.903%) 2024-09-18T07:43:11.154 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:11 smithi137 ceph-mon[29408]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:43:11.155 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:11 smithi137 ceph-mon[29408]: 4.1f scrub starts 2024-09-18T07:43:11.155 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:11 smithi137 ceph-mon[29408]: 4.1f scrub ok 2024-09-18T07:43:11.198 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:11.206 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:11 smithi203 ceph-mon[32237]: pgmap v649: 158 pgs: 47 active+undersized, 22 active+undersized+degraded, 89 active+clean; 635 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 668 B/s rd, 0 op/s; 83/597 objects degraded (13.903%) 2024-09-18T07:43:11.206 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:11 smithi203 ceph-mon[32237]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:43:11.206 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:11 smithi203 ceph-mon[32237]: 4.1f scrub starts 2024-09-18T07:43:11.206 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:11 smithi203 ceph-mon[32237]: 4.1f scrub ok 2024-09-18T07:43:11.232 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:11.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:11.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:11.334 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:11.366 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:11.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:11.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:11.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:11.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:11.507 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:11.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:11.607 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:11.623 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:43:11.623 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:43:11.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:43:11.623 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:43:11.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.623 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134984 7.16 7:6e4eb25e:::benchmark_data_smithi137_74180_object30568:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:43:11.623 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964075+0000", 2024-09-18T07:43:11.623 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 534.65401395200001, 2024-09-18T07:43:11.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.253286275, 2024-09-18T07:43:11.623 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.623 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.624 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:43:11.624 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.624 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.624 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.624 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134984 2024-09-18T07:43:11.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.624 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964075+0000", 2024-09-18T07:43:11.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.625 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964075+0000", 2024-09-18T07:43:11.625 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.625 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.625 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964077+0000", 2024-09-18T07:43:11.625 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8870000000000001e-06 2024-09-18T07:43:11.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.626 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964083+0000", 2024-09-18T07:43:11.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4069999999999998e-06 2024-09-18T07:43:11.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.626 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964098+0000", 2024-09-18T07:43:11.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4119e-05 2024-09-18T07:43:11.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.626 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964108+0000", 2024-09-18T07:43:11.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0665999999999999e-05 2024-09-18T07:43:11.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964166+0000", 2024-09-18T07:43:11.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7819000000000003e-05 2024-09-18T07:43:11.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:43:11.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964174+0000", 2024-09-18T07:43:11.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5499999999999997e-06 2024-09-18T07:43:11.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.968251+0000", 2024-09-18T07:43:11.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004077448 2024-09-18T07:43:11.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.968261+0000", 2024-09-18T07:43:11.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0438000000000001e-05 2024-09-18T07:43:11.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.217288+0000", 2024-09-18T07:43:11.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.249026295 2024-09-18T07:43:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.217361+0000", 2024-09-18T07:43:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3646000000000006e-05 2024-09-18T07:43:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134991 7.19 7:9bc94497:::benchmark_data_smithi137_74180_object30575:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:43:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964132+0000", 2024-09-18T07:43:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 534.65395685700003, 2024-09-18T07:43:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2456100560000001, 2024-09-18T07:43:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:43:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134991 2024-09-18T07:43:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964132+0000", 2024-09-18T07:43:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964132+0000", 2024-09-18T07:43:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964133+0000", 2024-09-18T07:43:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4000000000000001e-07 2024-09-18T07:43:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964136+0000", 2024-09-18T07:43:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8009999999999999e-06 2024-09-18T07:43:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964141+0000", 2024-09-18T07:43:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.75e-06 2024-09-18T07:43:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964147+0000", 2024-09-18T07:43:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5439999999999998e-06 2024-09-18T07:43:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964319+0000", 2024-09-18T07:43:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017216000000000001 2024-09-18T07:43:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:43:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964324+0000", 2024-09-18T07:43:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.31e-06 2024-09-18T07:43:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.970873+0000", 2024-09-18T07:43:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065484250000000001 2024-09-18T07:43:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.970881+0000", 2024-09-18T07:43:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0600000000000008e-06 2024-09-18T07:43:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.201197+0000", 2024-09-18T07:43:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2303157170000001 2024-09-18T07:43:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-09-18T07:43:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.201321+0000", 2024-09-18T07:43:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012415000000000001 2024-09-18T07:43:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.209647+0000", 2024-09-18T07:43:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.008325931 2024-09-18T07:43:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.209742+0000", 2024-09-18T07:43:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5468000000000001e-05 2024-09-18T07:43:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135011 7.14 7:2c5d899c:::benchmark_data_smithi137_74180_object30595:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:43:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964161+0000", 2024-09-18T07:43:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 534.65392757999996, 2024-09-18T07:43:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2230384009999999, 2024-09-18T07:43:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135011 2024-09-18T07:43:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964161+0000", 2024-09-18T07:43:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964161+0000", 2024-09-18T07:43:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964162+0000", 2024-09-18T07:43:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2300000000000001e-07 2024-09-18T07:43:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964164+0000", 2024-09-18T07:43:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3690000000000001e-06 2024-09-18T07:43:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964169+0000", 2024-09-18T07:43:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.028e-06 2024-09-18T07:43:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964174+0000", 2024-09-18T07:43:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7809999999999996e-06 2024-09-18T07:43:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964271+0000", 2024-09-18T07:43:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6997000000000007e-05 2024-09-18T07:43:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:43:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964277+0000", 2024-09-18T07:43:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4369999999999998e-06 2024-09-18T07:43:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.969538+0000", 2024-09-18T07:43:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052614929999999999 2024-09-18T07:43:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.969546+0000", 2024-09-18T07:43:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9540000000000004e-06 2024-09-18T07:43:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.178869+0000", 2024-09-18T07:43:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2093228570000001 2024-09-18T07:43:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.179001+0000", 2024-09-18T07:43:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013228400000000001 2024-09-18T07:43:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2,3,4,5", 2024-09-18T07:43:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.180534+0000", 2024-09-18T07:43:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0015324399999999999 2024-09-18T07:43:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.183783+0000", 2024-09-18T07:43:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032497559999999999 2024-09-18T07:43:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184155+0000", 2024-09-18T07:43:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000371388 2024-09-18T07:43:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184621+0000", 2024-09-18T07:43:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00046647700000000003 2024-09-18T07:43:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185211+0000", 2024-09-18T07:43:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00058937699999999998 2024-09-18T07:43:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185835+0000", 2024-09-18T07:43:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000624269 2024-09-18T07:43:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186879+0000", 2024-09-18T07:43:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010440429999999999 2024-09-18T07:43:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186966+0000", 2024-09-18T07:43:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7031000000000001e-05 2024-09-18T07:43:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.187200+0000", 2024-09-18T07:43:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000233797 2024-09-18T07:43:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135015 7.17 7:e8fec392:::benchmark_data_smithi137_74180_object30599:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:43:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964189+0000", 2024-09-18T07:43:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 534.65389955000001, 2024-09-18T07:43:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.277466437, 2024-09-18T07:43:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135015 2024-09-18T07:43:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964189+0000", 2024-09-18T07:43:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964189+0000", 2024-09-18T07:43:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964190+0000", 2024-09-18T07:43:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7199999999999998e-07 2024-09-18T07:43:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964193+0000", 2024-09-18T07:43:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5670000000000002e-06 2024-09-18T07:43:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964199+0000", 2024-09-18T07:43:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8200000000000002e-06 2024-09-18T07:43:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964203+0000", 2024-09-18T07:43:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9189999999999998e-06 2024-09-18T07:43:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964356+0000", 2024-09-18T07:43:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015237200000000001 2024-09-18T07:43:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:43:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964361+0000", 2024-09-18T07:43:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1499999999999998e-06 2024-09-18T07:43:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971479+0000", 2024-09-18T07:43:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.007117511 2024-09-18T07:43:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971487+0000", 2024-09-18T07:43:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2800000000000003e-06 2024-09-18T07:43:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.224196+0000", 2024-09-18T07:43:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2527092280000001 2024-09-18T07:43:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.224324+0000", 2024-09-18T07:43:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000127582 2024-09-18T07:43:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2,3,5,6", 2024-09-18T07:43:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.224588+0000", 2024-09-18T07:43:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00026444499999999998 2024-09-18T07:43:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.230126+0000", 2024-09-18T07:43:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055381579999999996 2024-09-18T07:43:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.230140+0000", 2024-09-18T07:43:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4e-05 2024-09-18T07:43:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.230258+0000", 2024-09-18T07:43:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118002 2024-09-18T07:43:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.232997+0000", 2024-09-18T07:43:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027387829999999998 2024-09-18T07:43:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.233523+0000", 2024-09-18T07:43:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052651200000000003 2024-09-18T07:43:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.241477+0000", 2024-09-18T07:43:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079535379999999996 2024-09-18T07:43:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.241534+0000", 2024-09-18T07:43:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6906e-05 2024-09-18T07:43:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.241656+0000", 2024-09-18T07:43:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012199199999999999 2024-09-18T07:43:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135019 7.17 7:ec2155c5:::benchmark_data_smithi137_74180_object30603:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:43:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964215+0000", 2024-09-18T07:43:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 534.65387415800001, 2024-09-18T07:43:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2831853689999999, 2024-09-18T07:43:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135019 2024-09-18T07:43:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964215+0000", 2024-09-18T07:43:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964215+0000", 2024-09-18T07:43:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964216+0000", 2024-09-18T07:43:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3499999999999995e-07 2024-09-18T07:43:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964218+0000", 2024-09-18T07:43:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5790000000000002e-06 2024-09-18T07:43:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964224+0000", 2024-09-18T07:43:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3689999999999998e-06 2024-09-18T07:43:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964228+0000", 2024-09-18T07:43:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3560000000000003e-06 2024-09-18T07:43:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964383+0000", 2024-09-18T07:43:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000155217 2024-09-18T07:43:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-18T07:43:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964388+0000", 2024-09-18T07:43:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8550000000000001e-06 2024-09-18T07:43:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971509+0000", 2024-09-18T07:43:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0071211800000000004 2024-09-18T07:43:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.971515+0000", 2024-09-18T07:43:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.224e-06 2024-09-18T07:43:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.226033+0000", 2024-09-18T07:43:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2545179440000001 2024-09-18T07:43:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.226140+0000", 2024-09-18T07:43:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010681200000000001 2024-09-18T07:43:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2,3,5,6", 2024-09-18T07:43:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.226373+0000", 2024-09-18T07:43:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023260699999999999 2024-09-18T07:43:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.230206+0000", 2024-09-18T07:43:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038327439999999999 2024-09-18T07:43:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.233570+0000", 2024-09-18T07:43:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033643129999999999 2024-09-18T07:43:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.233605+0000", 2024-09-18T07:43:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5290999999999998e-05 2024-09-18T07:43:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.241382+0000", 2024-09-18T07:43:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0077770239999999996 2024-09-18T07:43:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.241433+0000", 2024-09-18T07:43:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1044000000000001e-05 2024-09-18T07:43:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.247229+0000", 2024-09-18T07:43:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057953609999999997 2024-09-18T07:43:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.247282+0000", 2024-09-18T07:43:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3084999999999997e-05 2024-09-18T07:43:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.247400+0000", 2024-09-18T07:43:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011862900000000001 2024-09-18T07:43:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162316 7.f 7:f95a8250:::benchmark_data_smithi137_74180_object57900:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:43:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.409395+0000", 2024-09-18T07:43:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 490.20869392600002, 2024-09-18T07:43:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0987874790000001, 2024-09-18T07:43:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162316 2024-09-18T07:43:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409395+0000", 2024-09-18T07:43:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409395+0000", 2024-09-18T07:43:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409397+0000", 2024-09-18T07:43:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3429999999999998e-06 2024-09-18T07:43:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409404+0000", 2024-09-18T07:43:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2600000000000002e-06 2024-09-18T07:43:11.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:11.668 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:11.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409418+0000", 2024-09-18T07:43:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3983e-05 2024-09-18T07:43:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409427+0000", 2024-09-18T07:43:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9269999999999994e-06 2024-09-18T07:43:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409490+0000", 2024-09-18T07:43:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2868999999999997e-05 2024-09-18T07:43:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409500+0000", 2024-09-18T07:43:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0621e-05 2024-09-18T07:43:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.497693+0000", 2024-09-18T07:43:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.088192405 2024-09-18T07:43:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.497816+0000", 2024-09-18T07:43:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012321500000000001 2024-09-18T07:43:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:43:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.497964+0000", 2024-09-18T07:43:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000148169 2024-09-18T07:43:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.500777+0000", 2024-09-18T07:43:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028126750000000002 2024-09-18T07:43:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.500835+0000", 2024-09-18T07:43:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8727000000000002e-05 2024-09-18T07:43:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.500878+0000", 2024-09-18T07:43:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2682000000000002e-05 2024-09-18T07:43:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507887+0000", 2024-09-18T07:43:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0070087789999999997 2024-09-18T07:43:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508067+0000", 2024-09-18T07:43:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017973300000000001 2024-09-18T07:43:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508183+0000", 2024-09-18T07:43:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116091 2024-09-18T07:43:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162322 7.f 7:f41da5a4:::benchmark_data_smithi137_74180_object57906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:43:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.420348+0000", 2024-09-18T07:43:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 490.19774063300002, 2024-09-18T07:43:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0968331140000001, 2024-09-18T07:43:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162322 2024-09-18T07:43:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420348+0000", 2024-09-18T07:43:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420348+0000", 2024-09-18T07:43:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420350+0000", 2024-09-18T07:43:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8670000000000001e-06 2024-09-18T07:43:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420356+0000", 2024-09-18T07:43:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0560000000000001e-06 2024-09-18T07:43:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420369+0000", 2024-09-18T07:43:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2507000000000001e-05 2024-09-18T07:43:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420376+0000", 2024-09-18T07:43:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.17e-06 2024-09-18T07:43:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420444+0000", 2024-09-18T07:43:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7801999999999999e-05 2024-09-18T07:43:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420454+0000", 2024-09-18T07:43:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0258000000000001e-05 2024-09-18T07:43:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.501094+0000", 2024-09-18T07:43:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0806402799999999 2024-09-18T07:43:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.501227+0000", 2024-09-18T07:43:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013271600000000001 2024-09-18T07:43:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:43:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.501467+0000", 2024-09-18T07:43:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024032599999999999 2024-09-18T07:43:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507729+0000", 2024-09-18T07:43:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0062618220000000002 2024-09-18T07:43:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507768+0000", 2024-09-18T07:43:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8668000000000003e-05 2024-09-18T07:43:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.516751+0000", 2024-09-18T07:43:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0089832390000000005 2024-09-18T07:43:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.516888+0000", 2024-09-18T07:43:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137372 2024-09-18T07:43:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517069+0000", 2024-09-18T07:43:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000180574 2024-09-18T07:43:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517182+0000", 2024-09-18T07:43:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112457 2024-09-18T07:43:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162323 7.f 7:fe31c7fa:::benchmark_data_smithi137_74180_object57907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:43:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.420396+0000", 2024-09-18T07:43:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 490.19769316200001, 2024-09-18T07:43:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.097250378, 2024-09-18T07:43:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162323 2024-09-18T07:43:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420396+0000", 2024-09-18T07:43:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420396+0000", 2024-09-18T07:43:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420396+0000", 2024-09-18T07:43:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9400000000000005e-07 2024-09-18T07:43:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420399+0000", 2024-09-18T07:43:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6879999999999999e-06 2024-09-18T07:43:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420404+0000", 2024-09-18T07:43:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2680000000000002e-06 2024-09-18T07:43:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420411+0000", 2024-09-18T07:43:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2519999999999996e-06 2024-09-18T07:43:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420540+0000", 2024-09-18T07:43:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012886899999999999 2024-09-18T07:43:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420547+0000", 2024-09-18T07:43:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9019999999999999e-06 2024-09-18T07:43:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.504020+0000", 2024-09-18T07:43:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0834720959999999 2024-09-18T07:43:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.504135+0000", 2024-09-18T07:43:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115697 2024-09-18T07:43:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:43:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.504251+0000", 2024-09-18T07:43:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116041 2024-09-18T07:43:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507805+0000", 2024-09-18T07:43:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035537749999999999 2024-09-18T07:43:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507870+0000", 2024-09-18T07:43:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4558000000000001e-05 2024-09-18T07:43:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.516761+0000", 2024-09-18T07:43:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0088909690000000003 2024-09-18T07:43:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517414+0000", 2024-09-18T07:43:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00065372100000000001 2024-09-18T07:43:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517543+0000", 2024-09-18T07:43:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012907199999999999 2024-09-18T07:43:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517646+0000", 2024-09-18T07:43:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102876 2024-09-18T07:43:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162327 7.f 7:fac5d155:::benchmark_data_smithi137_74180_object57911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:43:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.420507+0000", 2024-09-18T07:43:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 490.19758215399997, 2024-09-18T07:43:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.097389145, 2024-09-18T07:43:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162327 2024-09-18T07:43:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420507+0000", 2024-09-18T07:43:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420507+0000", 2024-09-18T07:43:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420508+0000", 2024-09-18T07:43:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7300000000000005e-07 2024-09-18T07:43:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420511+0000", 2024-09-18T07:43:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4089999999999999e-06 2024-09-18T07:43:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420518+0000", 2024-09-18T07:43:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1010000000000003e-06 2024-09-18T07:43:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420523+0000", 2024-09-18T07:43:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2089999999999996e-06 2024-09-18T07:43:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420570+0000", 2024-09-18T07:43:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6638000000000003e-05 2024-09-18T07:43:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420577+0000", 2024-09-18T07:43:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2679999999999997e-06 2024-09-18T07:43:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508213+0000", 2024-09-18T07:43:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0876354130000001 2024-09-18T07:43:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508312+0000", 2024-09-18T07:43:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9124999999999997e-05 2024-09-18T07:43:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:43:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508859+0000", 2024-09-18T07:43:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00054700599999999995 2024-09-18T07:43:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.516816+0000", 2024-09-18T07:43:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079569410000000004 2024-09-18T07:43:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517264+0000", 2024-09-18T07:43:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044864999999999998 2024-09-18T07:43:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517309+0000", 2024-09-18T07:43:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5033000000000001e-05 2024-09-18T07:43:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517755+0000", 2024-09-18T07:43:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044556899999999999 2024-09-18T07:43:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517803+0000", 2024-09-18T07:43:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7584999999999999e-05 2024-09-18T07:43:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517896+0000", 2024-09-18T07:43:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3424999999999994e-05 2024-09-18T07:43:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162336 7.f 7:f2e86194:::benchmark_data_smithi137_74180_object57920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:43:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.425606+0000", 2024-09-18T07:43:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 490.19248294800002, 2024-09-18T07:43:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0924705029999999, 2024-09-18T07:43:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162336 2024-09-18T07:43:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425606+0000", 2024-09-18T07:43:11.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425606+0000", 2024-09-18T07:43:11.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425608+0000", 2024-09-18T07:43:11.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2259999999999999e-06 2024-09-18T07:43:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425614+0000", 2024-09-18T07:43:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0549999999999996e-06 2024-09-18T07:43:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425626+0000", 2024-09-18T07:43:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1600000000000001e-05 2024-09-18T07:43:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425635+0000", 2024-09-18T07:43:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0440000000000001e-06 2024-09-18T07:43:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425680+0000", 2024-09-18T07:43:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5015999999999997e-05 2024-09-18T07:43:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425689+0000", 2024-09-18T07:43:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3419999999999997e-06 2024-09-18T07:43:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.513184+0000", 2024-09-18T07:43:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0874942320000001 2024-09-18T07:43:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.513298+0000", 2024-09-18T07:43:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114333 2024-09-18T07:43:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:43:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.513425+0000", 2024-09-18T07:43:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126711 2024-09-18T07:43:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517321+0000", 2024-09-18T07:43:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038961590000000002 2024-09-18T07:43:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517374+0000", 2024-09-18T07:43:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3003000000000001e-05 2024-09-18T07:43:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517718+0000", 2024-09-18T07:43:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034425100000000001 2024-09-18T07:43:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517935+0000", 2024-09-18T07:43:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021721899999999999 2024-09-18T07:43:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517986+0000", 2024-09-18T07:43:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0627999999999997e-05 2024-09-18T07:43:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.518077+0000", 2024-09-18T07:43:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0684000000000001e-05 2024-09-18T07:43:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162345 7.f 7:f017fa1c:::benchmark_data_smithi137_74180_object57929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:43:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.431398+0000", 2024-09-18T07:43:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 490.18669146100001, 2024-09-18T07:43:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0972635669999999, 2024-09-18T07:43:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162345 2024-09-18T07:43:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431398+0000", 2024-09-18T07:43:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431398+0000", 2024-09-18T07:43:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431399+0000", 2024-09-18T07:43:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.361e-06 2024-09-18T07:43:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431403+0000", 2024-09-18T07:43:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1520000000000002e-06 2024-09-18T07:43:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431414+0000", 2024-09-18T07:43:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1239e-05 2024-09-18T07:43:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431421+0000", 2024-09-18T07:43:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.99e-06 2024-09-18T07:43:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431537+0000", 2024-09-18T07:43:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115278 2024-09-18T07:43:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431547+0000", 2024-09-18T07:43:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0543999999999999e-05 2024-09-18T07:43:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.518110+0000", 2024-09-18T07:43:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.086562942 2024-09-18T07:43:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.518220+0000", 2024-09-18T07:43:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010951 2024-09-18T07:43:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:43:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.518334+0000", 2024-09-18T07:43:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114057 2024-09-18T07:43:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.527942+0000", 2024-09-18T07:43:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0096087029999999997 2024-09-18T07:43:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528087+0000", 2024-09-18T07:43:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000144871 2024-09-18T07:43:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528103+0000", 2024-09-18T07:43:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5531000000000001e-05 2024-09-18T07:43:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528148+0000", 2024-09-18T07:43:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5667000000000002e-05 2024-09-18T07:43:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528542+0000", 2024-09-18T07:43:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00039349200000000001 2024-09-18T07:43:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528661+0000", 2024-09-18T07:43:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011923 2024-09-18T07:43:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202539 7.f 7:fdd02dfa:::benchmark_data_smithi137_74180_object98123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:43:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.829809+0000", 2024-09-18T07:43:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 453.78827979099998, 2024-09-18T07:43:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0954651049999999, 2024-09-18T07:43:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202539 2024-09-18T07:43:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829809+0000", 2024-09-18T07:43:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829809+0000", 2024-09-18T07:43:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829811+0000", 2024-09-18T07:43:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9360000000000002e-06 2024-09-18T07:43:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829816+0000", 2024-09-18T07:43:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6979999999999997e-06 2024-09-18T07:43:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829824+0000", 2024-09-18T07:43:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6799999999999993e-06 2024-09-18T07:43:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829829+0000", 2024-09-18T07:43:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9159999999999998e-06 2024-09-18T07:43:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829890+0000", 2024-09-18T07:43:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0643000000000002e-05 2024-09-18T07:43:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829898+0000", 2024-09-18T07:43:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.576e-06 2024-09-18T07:43:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.917838+0000", 2024-09-18T07:43:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.087940012 2024-09-18T07:43:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.917970+0000", 2024-09-18T07:43:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013214499999999999 2024-09-18T07:43:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:43:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.918095+0000", 2024-09-18T07:43:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012521 2024-09-18T07:43:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.921001+0000", 2024-09-18T07:43:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029057789999999998 2024-09-18T07:43:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.921054+0000", 2024-09-18T07:43:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3038999999999998e-05 2024-09-18T07:43:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925195+0000", 2024-09-18T07:43:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004140632 2024-09-18T07:43:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925229+0000", 2024-09-18T07:43:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4783000000000002e-05 2024-09-18T07:43:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925274+0000", 2024-09-18T07:43:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5055999999999997e-05 2024-09-18T07:43:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202541 7.f 7:f489b710:::benchmark_data_smithi137_74180_object98125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:43:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.829844+0000", 2024-09-18T07:43:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 453.78824460999999, 2024-09-18T07:43:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0983412960000001, 2024-09-18T07:43:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202541 2024-09-18T07:43:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829844+0000", 2024-09-18T07:43:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829844+0000", 2024-09-18T07:43:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829845+0000", 2024-09-18T07:43:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0100000000000004e-07 2024-09-18T07:43:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829848+0000", 2024-09-18T07:43:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0170000000000001e-06 2024-09-18T07:43:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829854+0000", 2024-09-18T07:43:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9529999999999996e-06 2024-09-18T07:43:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829859+0000", 2024-09-18T07:43:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1499999999999998e-06 2024-09-18T07:43:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.830023+0000", 2024-09-18T07:43:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000163473 2024-09-18T07:43:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.830030+0000", 2024-09-18T07:43:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.396e-06 2024-09-18T07:43:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.921101+0000", 2024-09-18T07:43:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0910708730000001 2024-09-18T07:43:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.921187+0000", 2024-09-18T07:43:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6408000000000003e-05 2024-09-18T07:43:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:43:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.921265+0000", 2024-09-18T07:43:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8019e-05 2024-09-18T07:43:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925399+0000", 2024-09-18T07:43:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004133205 2024-09-18T07:43:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925470+0000", 2024-09-18T07:43:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1420000000000004e-05 2024-09-18T07:43:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928114+0000", 2024-09-18T07:43:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026441580000000002 2024-09-18T07:43:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928155+0000", 2024-09-18T07:43:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0620000000000001e-05 2024-09-18T07:43:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928186+0000", 2024-09-18T07:43:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0902999999999999e-05 2024-09-18T07:43:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202543 7.f 7:f02d8d75:::benchmark_data_smithi137_74180_object98127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:43:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.829915+0000", 2024-09-18T07:43:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 453.78817366200002, 2024-09-18T07:43:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.098406115, 2024-09-18T07:43:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.729 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.729 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.729 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.729 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202543 2024-09-18T07:43:11.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.729 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.729 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.729 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829915+0000", 2024-09-18T07:43:11.729 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.729 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829915+0000", 2024-09-18T07:43:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829916+0000", 2024-09-18T07:43:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3900000000000003e-07 2024-09-18T07:43:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829921+0000", 2024-09-18T07:43:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2250000000000002e-06 2024-09-18T07:43:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829927+0000", 2024-09-18T07:43:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3600000000000001e-06 2024-09-18T07:43:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829933+0000", 2024-09-18T07:43:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0240000000000003e-06 2024-09-18T07:43:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.830051+0000", 2024-09-18T07:43:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118131 2024-09-18T07:43:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.830057+0000", 2024-09-18T07:43:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4929999999999998e-06 2024-09-18T07:43:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.923373+0000", 2024-09-18T07:43:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.093316306 2024-09-18T07:43:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.923465+0000", 2024-09-18T07:43:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2260000000000001e-05 2024-09-18T07:43:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:43:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.923543+0000", 2024-09-18T07:43:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7601999999999993e-05 2024-09-18T07:43:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925460+0000", 2024-09-18T07:43:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001916846 2024-09-18T07:43:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925474+0000", 2024-09-18T07:43:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4708999999999999e-05 2024-09-18T07:43:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928254+0000", 2024-09-18T07:43:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002779662 2024-09-18T07:43:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928293+0000", 2024-09-18T07:43:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9419000000000003e-05 2024-09-18T07:43:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928321+0000", 2024-09-18T07:43:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8138999999999999e-05 2024-09-18T07:43:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202553 7.f 7:f853e395:::benchmark_data_smithi137_74180_object98137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:43:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.836313+0000", 2024-09-18T07:43:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 453.78177595800003, 2024-09-18T07:43:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.099117246, 2024-09-18T07:43:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202553 2024-09-18T07:43:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836313+0000", 2024-09-18T07:43:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836313+0000", 2024-09-18T07:43:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836314+0000", 2024-09-18T07:43:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3519999999999999e-06 2024-09-18T07:43:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836319+0000", 2024-09-18T07:43:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6469999999999997e-06 2024-09-18T07:43:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836329+0000", 2024-09-18T07:43:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0356e-05 2024-09-18T07:43:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836337+0000", 2024-09-18T07:43:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4290000000000004e-06 2024-09-18T07:43:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836372+0000", 2024-09-18T07:43:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5577000000000002e-05 2024-09-18T07:43:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836377+0000", 2024-09-18T07:43:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7199999999999997e-06 2024-09-18T07:43:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925531+0000", 2024-09-18T07:43:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0891534789999999 2024-09-18T07:43:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925628+0000", 2024-09-18T07:43:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7720999999999995e-05 2024-09-18T07:43:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:43:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925719+0000", 2024-09-18T07:43:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0804999999999995e-05 2024-09-18T07:43:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928378+0000", 2024-09-18T07:43:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026585670000000001 2024-09-18T07:43:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935204+0000", 2024-09-18T07:43:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068259189999999997 2024-09-18T07:43:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935218+0000", 2024-09-18T07:43:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4562000000000001e-05 2024-09-18T07:43:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935381+0000", 2024-09-18T07:43:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000162473 2024-09-18T07:43:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935430+0000", 2024-09-18T07:43:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9638999999999997e-05 2024-09-18T07:43:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202576 7.f 7:f8260e72:::benchmark_data_smithi137_74180_object98160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:43:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.852688+0000", 2024-09-18T07:43:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 453.76540062499998, 2024-09-18T07:43:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.082910424, 2024-09-18T07:43:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202576 2024-09-18T07:43:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852688+0000", 2024-09-18T07:43:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852688+0000", 2024-09-18T07:43:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852690+0000", 2024-09-18T07:43:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5039999999999999e-06 2024-09-18T07:43:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852694+0000", 2024-09-18T07:43:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4889999999999997e-06 2024-09-18T07:43:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852704+0000", 2024-09-18T07:43:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1689999999999998e-06 2024-09-18T07:43:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852709+0000", 2024-09-18T07:43:11.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6189999999999999e-06 2024-09-18T07:43:11.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852757+0000", 2024-09-18T07:43:11.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7676000000000002e-05 2024-09-18T07:43:11.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852765+0000", 2024-09-18T07:43:11.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9400000000000002e-06 2024-09-18T07:43:11.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928432+0000", 2024-09-18T07:43:11.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.075667572 2024-09-18T07:43:11.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928561+0000", 2024-09-18T07:43:11.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012874199999999999 2024-09-18T07:43:11.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:43:11.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928674+0000", 2024-09-18T07:43:11.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011271400000000001 2024-09-18T07:43:11.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935474+0000", 2024-09-18T07:43:11.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068001270000000004 2024-09-18T07:43:11.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935484+0000", 2024-09-18T07:43:11.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0515e-05 2024-09-18T07:43:11.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935518+0000", 2024-09-18T07:43:11.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3895e-05 2024-09-18T07:43:11.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935565+0000", 2024-09-18T07:43:11.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6241e-05 2024-09-18T07:43:11.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935599+0000", 2024-09-18T07:43:11.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4221000000000003e-05 2024-09-18T07:43:11.749 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.749 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.749 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.749 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202651 7.f 7:f9b36136:::benchmark_data_smithi137_74180_object98235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:43:11.750 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.895907+0000", 2024-09-18T07:43:11.750 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 453.72218185499997, 2024-09-18T07:43:11.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.071674998, 2024-09-18T07:43:11.750 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.750 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.750 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.750 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.750 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.750 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.750 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202651 2024-09-18T07:43:11.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.750 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895907+0000", 2024-09-18T07:43:11.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895907+0000", 2024-09-18T07:43:11.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895909+0000", 2024-09-18T07:43:11.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8989999999999999e-06 2024-09-18T07:43:11.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895915+0000", 2024-09-18T07:43:11.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6350000000000003e-06 2024-09-18T07:43:11.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895929+0000", 2024-09-18T07:43:11.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4174e-05 2024-09-18T07:43:11.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895936+0000", 2024-09-18T07:43:11.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1450000000000002e-06 2024-09-18T07:43:11.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895992+0000", 2024-09-18T07:43:11.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6255999999999998e-05 2024-09-18T07:43:11.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.896001+0000", 2024-09-18T07:43:11.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4919999999999993e-06 2024-09-18T07:43:11.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.947291+0000", 2024-09-18T07:43:11.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.051289881 2024-09-18T07:43:11.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.947431+0000", 2024-09-18T07:43:11.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014024799999999999 2024-09-18T07:43:11.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:43:11.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.947553+0000", 2024-09-18T07:43:11.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000122032 2024-09-18T07:43:11.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.957558+0000", 2024-09-18T07:43:11.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010004948 2024-09-18T07:43:11.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.957904+0000", 2024-09-18T07:43:11.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:43:11.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034637699999999998 2024-09-18T07:43:11.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967474+0000", 2024-09-18T07:43:11.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095700709999999994 2024-09-18T07:43:11.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967537+0000", 2024-09-18T07:43:11.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2881999999999999e-05 2024-09-18T07:43:11.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967582+0000", 2024-09-18T07:43:11.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4957999999999999e-05 2024-09-18T07:43:11.758 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.758 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.758 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.758 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202659 7.f 7:f246d9b4:::benchmark_data_smithi137_74180_object98243:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:43:11.758 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.901704+0000", 2024-09-18T07:43:11.758 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 453.716385027, 2024-09-18T07:43:11.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0662714099999999, 2024-09-18T07:43:11.758 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.758 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.758 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.758 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.758 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:11.759 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:11.759 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202659 2024-09-18T07:43:11.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.759 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901704+0000", 2024-09-18T07:43:11.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901704+0000", 2024-09-18T07:43:11.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901705+0000", 2024-09-18T07:43:11.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.296e-06 2024-09-18T07:43:11.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901709+0000", 2024-09-18T07:43:11.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9280000000000003e-06 2024-09-18T07:43:11.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901719+0000", 2024-09-18T07:43:11.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3049999999999998e-06 2024-09-18T07:43:11.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901724+0000", 2024-09-18T07:43:11.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2070000000000003e-06 2024-09-18T07:43:11.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901762+0000", 2024-09-18T07:43:11.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8578000000000002e-05 2024-09-18T07:43:11.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901784+0000", 2024-09-18T07:43:11.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1883e-05 2024-09-18T07:43:11.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.950958+0000", 2024-09-18T07:43:11.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.049174042 2024-09-18T07:43:11.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.951088+0000", 2024-09-18T07:43:11.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000129729 2024-09-18T07:43:11.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:43:11.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.951211+0000", 2024-09-18T07:43:11.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012286099999999999 2024-09-18T07:43:11.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.957893+0000", 2024-09-18T07:43:11.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066821709999999998 2024-09-18T07:43:11.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.958138+0000", 2024-09-18T07:43:11.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024456100000000002 2024-09-18T07:43:11.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967712+0000", 2024-09-18T07:43:11.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095747760000000001 2024-09-18T07:43:11.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967934+0000", 2024-09-18T07:43:11.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000221225 2024-09-18T07:43:11.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967975+0000", 2024-09-18T07:43:11.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1848000000000003e-05 2024-09-18T07:43:11.765 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.765 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.765 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.765 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109534 8.6 8:67e394d3:::benchmark_data_smithi137_101942_object4724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-18T07:43:11.765 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.048516+0000", 2024-09-18T07:43:11.765 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 329.56957267199999, 2024-09-18T07:43:11.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.182592823, 2024-09-18T07:43:11.765 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.765 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.766 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.766 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.766 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:43:11.766 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:43:11.766 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109534 2024-09-18T07:43:11.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.766 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.766 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.766 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048516+0000", 2024-09-18T07:43:11.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048516+0000", 2024-09-18T07:43:11.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048517+0000", 2024-09-18T07:43:11.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2e-07 2024-09-18T07:43:11.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048519+0000", 2024-09-18T07:43:11.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.232e-06 2024-09-18T07:43:11.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048523+0000", 2024-09-18T07:43:11.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4479999999999996e-06 2024-09-18T07:43:11.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048528+0000", 2024-09-18T07:43:11.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2169999999999996e-06 2024-09-18T07:43:11.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065112+0000", 2024-09-18T07:43:11.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016583912999999999 2024-09-18T07:43:11.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065121+0000", 2024-09-18T07:43:11.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8640000000000008e-06 2024-09-18T07:43:11.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.219070+0000", 2024-09-18T07:43:11.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1539486139999999 2024-09-18T07:43:11.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.219228+0000", 2024-09-18T07:43:11.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000157751 2024-09-18T07:43:11.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-09-18T07:43:11.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.219376+0000", 2024-09-18T07:43:11.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014781899999999999 2024-09-18T07:43:11.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226242+0000", 2024-09-18T07:43:11.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068661970000000001 2024-09-18T07:43:11.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226253+0000", 2024-09-18T07:43:11.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0929e-05 2024-09-18T07:43:11.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229198+0000", 2024-09-18T07:43:11.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029453790000000001 2024-09-18T07:43:11.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229278+0000", 2024-09-18T07:43:11.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9431000000000006e-05 2024-09-18T07:43:11.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.231109+0000", 2024-09-18T07:43:11.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018316090000000001 2024-09-18T07:43:11.772 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.772 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.772 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.772 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109539 8.6 8:66837289:::benchmark_data_smithi137_101942_object4729:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:43:11.772 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.065140+0000", 2024-09-18T07:43:11.772 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 329.55294920300003, 2024-09-18T07:43:11.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1676551340000001, 2024-09-18T07:43:11.772 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:11.772 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:11.772 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:11.772 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:11.772 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:43:11.773 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:43:11.773 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109539 2024-09-18T07:43:11.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.773 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:11.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:11.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065140+0000", 2024-09-18T07:43:11.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:11.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065140+0000", 2024-09-18T07:43:11.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:11.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.774 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:11.774 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065142+0000", 2024-09-18T07:43:11.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.548e-06 2024-09-18T07:43:11.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.774 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:11.774 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065149+0000", 2024-09-18T07:43:11.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7989999999999996e-06 2024-09-18T07:43:11.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.774 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:11.774 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065162+0000", 2024-09-18T07:43:11.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3027000000000001e-05 2024-09-18T07:43:11.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:11.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065171+0000", 2024-09-18T07:43:11.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1730000000000001e-06 2024-09-18T07:43:11.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065215+0000", 2024-09-18T07:43:11.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3679e-05 2024-09-18T07:43:11.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:11.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065222+0000", 2024-09-18T07:43:11.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9879999999999998e-06 2024-09-18T07:43:11.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:11.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.223751+0000", 2024-09-18T07:43:11.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1585293800000001 2024-09-18T07:43:11.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:11.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.223906+0000", 2024-09-18T07:43:11.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015412600000000001 2024-09-18T07:43:11.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-09-18T07:43:11.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.224068+0000", 2024-09-18T07:43:11.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016266499999999999 2024-09-18T07:43:11.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:11.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226399+0000", 2024-09-18T07:43:11.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002330557 2024-09-18T07:43:11.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226549+0000", 2024-09-18T07:43:11.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000150693 2024-09-18T07:43:11.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:11.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.231185+0000", 2024-09-18T07:43:11.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004635775 2024-09-18T07:43:11.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:11.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.231241+0000", 2024-09-18T07:43:11.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6001999999999997e-05 2024-09-18T07:43:11.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:11.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:11.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:11.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.232795+0000", 2024-09-18T07:43:11.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001553722 2024-09-18T07:43:11.779 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.779 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.779 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.779 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:11.779 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:11.779 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:43:11.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:11.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:11.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:11.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:11.915 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:11.944 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:11.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:11.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:12.030 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:43:12.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:12.052 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 2 16 7142 7126 222.67 212.125 0.00449662 0.00448138 2024-09-18T07:43:12.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:12.094 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:12.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:12.195 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:12.229 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:12.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:12.257 INFO:teuthology.orchestra.run.smithi203.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-18T07:43:12.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:12.371 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:12.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:12.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:12.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:12 smithi203 ceph-mon[32237]: 5.2 scrub starts 2024-09-18T07:43:12.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:12 smithi203 ceph-mon[32237]: 5.2 scrub ok 2024-09-18T07:43:12.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:12 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3759188600' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:43:12.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:12 smithi203 ceph-mon[32237]: from='client.36020 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:43:12.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:12 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:43:12.391 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:43:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:12 smithi137 ceph-mon[29408]: 5.2 scrub starts 2024-09-18T07:43:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:12 smithi137 ceph-mon[29408]: 5.2 scrub ok 2024-09-18T07:43:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:12 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3759188600' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:43:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:12 smithi137 ceph-mon[29408]: from='client.36020 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:43:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:12 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:43:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:43:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:12 smithi137 ceph-mon[39175]: 5.2 scrub starts 2024-09-18T07:43:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:12 smithi137 ceph-mon[39175]: 5.2 scrub ok 2024-09-18T07:43:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:12 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3759188600' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:43:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:12 smithi137 ceph-mon[39175]: from='client.36020 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:43:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:12 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:43:12.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:43:12.474 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:12.509 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:12.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:12.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:12.611 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:12.635 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:12.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:12.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:12.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:12.772 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:12.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:12.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:12.874 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:12.905 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:12.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:13.006 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:13.035 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:13.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:13.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:13.053 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 3 15 10353 10338 215.355 200.75 0.00275508 0.00463504 2024-09-18T07:43:13.137 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:13.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:13.173 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:13.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:13.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:13 smithi203 ceph-mon[32237]: pgmap v650: 158 pgs: 45 active+undersized, 24 active+undersized+degraded, 89 active+clean; 50 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.7 MiB/s wr, 92 op/s; 872/2964 objects degraded (29.420%) 2024-09-18T07:43:13.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:13 smithi203 ceph-mon[32237]: 4.4 scrub starts 2024-09-18T07:43:13.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:13 smithi203 ceph-mon[32237]: 4.4 scrub ok 2024-09-18T07:43:13.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:13 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:13.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:13 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 872/2964 objects degraded (29.420%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:13.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:13 smithi203 ceph-mon[32237]: from='client.36020 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:43:13.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:13 smithi203 ceph-mon[32237]: osdmap e368: 8 total, 7 up, 8 in 2024-09-18T07:43:13.274 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:13.310 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:13.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:13.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:13.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:13 smithi137 ceph-mon[29408]: pgmap v650: 158 pgs: 45 active+undersized, 24 active+undersized+degraded, 89 active+clean; 50 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.7 MiB/s wr, 92 op/s; 872/2964 objects degraded (29.420%) 2024-09-18T07:43:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:13 smithi137 ceph-mon[29408]: 4.4 scrub starts 2024-09-18T07:43:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:13 smithi137 ceph-mon[29408]: 4.4 scrub ok 2024-09-18T07:43:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:13 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:13 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 872/2964 objects degraded (29.420%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:13.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:13 smithi137 ceph-mon[29408]: from='client.36020 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:43:13.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:13 smithi137 ceph-mon[29408]: osdmap e368: 8 total, 7 up, 8 in 2024-09-18T07:43:13.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:13 smithi137 ceph-mon[39175]: pgmap v650: 158 pgs: 45 active+undersized, 24 active+undersized+degraded, 89 active+clean; 50 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.7 MiB/s wr, 92 op/s; 872/2964 objects degraded (29.420%) 2024-09-18T07:43:13.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:13 smithi137 ceph-mon[39175]: 4.4 scrub starts 2024-09-18T07:43:13.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:13 smithi137 ceph-mon[39175]: 4.4 scrub ok 2024-09-18T07:43:13.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:13 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:13.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:13 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 872/2964 objects degraded (29.420%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:13.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:13 smithi137 ceph-mon[39175]: from='client.36020 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:43:13.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:13 smithi137 ceph-mon[39175]: osdmap e368: 8 total, 7 up, 8 in 2024-09-18T07:43:13.447 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:13.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:13.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:13.549 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:13.586 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:13.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:13.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:13.689 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:13.722 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:13.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:13.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T07:43:13.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:13.864 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:13.891 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:13.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:13.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:13.993 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:14.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:14.033 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:14.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:14.053 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 4 16 13499 13483 210.653 196.562 0.0062248 0.004741 2024-09-18T07:43:14.134 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:14.168 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:14.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:14.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:14.271 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:14.304 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:14.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:14.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:14.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:14.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:14.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:14.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:14.508 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:43:14.508 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:43:14.513 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:14.545 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:43:14.545 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:43:14.545 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:14.577 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:43:14.578 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:43:14.578 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:14.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:14.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:14.609 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:43:14.609 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:43:14.610 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-18T07:43:14.643 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:43:14.644 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:43:14.675 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:43:14.675 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:43:14.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:14.707 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:43:14.707 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:43:14.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:14.718 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:14.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:14.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:14.853 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:14.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:14.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:14.954 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:14.988 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:14.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:14.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:15.053 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 5 15 16197 16182 202.257 168.688 0.0133891 0.00493129 2024-09-18T07:43:15.091 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:15.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:43:15.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:15.119 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:15.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:15.220 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:15.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:15.253 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:15.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:15.342 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:15 smithi137 ceph-mon[29408]: pgmap v652: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 432 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 863 op/s; 3690/21306 objects degraded (17.319%) 2024-09-18T07:43:15.343 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:15 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3868946880' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:15.343 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:15 smithi137 ceph-mon[39175]: pgmap v652: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 432 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 863 op/s; 3690/21306 objects degraded (17.319%) 2024-09-18T07:43:15.343 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:15 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3868946880' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:15.353 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:15.399 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:15.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:15.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:15.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:15 smithi203 ceph-mon[32237]: pgmap v652: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 432 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 863 op/s; 3690/21306 objects degraded (17.319%) 2024-09-18T07:43:15.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:15 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3868946880' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:15.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:15.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:15.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:15.647 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:15.681 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:15.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:15.783 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:15.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:15.825 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:15.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:15.920 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:43:15.920 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:43:15.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:15.972 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:15.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:15.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:16.053 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 6 16 18978 18962 197.503 173.75 0.00488294 0.00505795 2024-09-18T07:43:16.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:16.109 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:16.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:16.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:16.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:16.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:16.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:16.339 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:16.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:16.375 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:16.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:16.476 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:16.509 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:16.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:16.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:16.611 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:16.643 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:16.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:16.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:16.745 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:16.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:16.779 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:16.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:16.781 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:43:16.781 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:43:16.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:16.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:16.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:16.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:17.017 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:17.047 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:17.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:17.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:17.053 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 7 16 21682 21666 193.429 169 0.00376412 0.00516465 2024-09-18T07:43:17.148 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:17.181 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:17.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:17.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:17.283 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:17.319 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:17.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:17.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:17.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:17 smithi137 ceph-mon[29408]: pgmap v653: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 432 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 863 op/s; 3690/21306 objects degraded (17.319%) 2024-09-18T07:43:17.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:17 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:43:17.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:17 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:43:17.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:43:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:43:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:43:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:17 smithi137 ceph-mon[39175]: pgmap v653: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 432 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 863 op/s; 3690/21306 objects degraded (17.319%) 2024-09-18T07:43:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:17 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:43:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:17 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:43:17.421 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:17.455 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:17.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:17.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:17.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:17 smithi203 ceph-mon[32237]: pgmap v653: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 432 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 863 op/s; 3690/21306 objects degraded (17.319%) 2024-09-18T07:43:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:17 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:43:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:17 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:43:17.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T07:43:17.557 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:17.567 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:17.610 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:17.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:17.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:17.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:17.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:17.756 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:17.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:17.773 INFO:teuthology.orchestra.run.smithi203.stdout: marking collection for removal 2024-09-18T07:43:17.773 INFO:teuthology.orchestra.run.smithi203.stdout:setting '_remove' omap key 2024-09-18T07:43:17.773 INFO:teuthology.orchestra.run.smithi203.stdout:finish_remove_pgs 1.2_head removing 1.2 2024-09-18T07:43:17.773 INFO:teuthology.orchestra.run.smithi203.stdout:Remove successful 2024-09-18T07:43:17.857 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:17.900 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:17.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:17.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:17.961 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:43:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:43:18.002 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:18.032 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:18.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:18.053 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 8 16 24274 24258 189.499 162 0.0063999 0.00527128 2024-09-18T07:43:18.134 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:18.185 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:18.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:18.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:18.227 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:18.287 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:18.320 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:18.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:18.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:18.362 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:18 smithi137 ceph-mon[29408]: osdmap e369: 8 total, 7 up, 8 in 2024-09-18T07:43:18.362 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:18 smithi137 ceph-mon[39175]: osdmap e369: 8 total, 7 up, 8 in 2024-09-18T07:43:18.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:18.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:18.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:18.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:18.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:18 smithi203 ceph-mon[32237]: osdmap e369: 8 total, 7 up, 8 in 2024-09-18T07:43:18.544 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:18.585 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:18.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:18.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:18.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:18.710 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:18.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:18.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:18.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:43:18.812 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:18.843 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:18.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:18.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:18.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:18.987 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:18.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:18.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:19.053 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 9 16 26426 26410 183.387 134.5 0.00655042 0.0054474 2024-09-18T07:43:19.088 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:43:19.088 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:19.137 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:19.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:19.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:19.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:19.260 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:19.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:19.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:19.362 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:19.387 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:19.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:19.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:19.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:19 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:19.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:19 smithi137 ceph-mon[39175]: pgmap v655: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 168 MiB/s wr, 2.68k op/s; 10943/64983 objects degraded (16.840%) 2024-09-18T07:43:19.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:19 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 10943/64983 objects degraded (16.840%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:19.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:19 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2905131197' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:19.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:19 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:19.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:19 smithi137 ceph-mon[29408]: pgmap v655: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 168 MiB/s wr, 2.68k op/s; 10943/64983 objects degraded (16.840%) 2024-09-18T07:43:19.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:19 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 10943/64983 objects degraded (16.840%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:19.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:19 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2905131197' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:19.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:19 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:19 smithi203 ceph-mon[32237]: pgmap v655: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 168 MiB/s wr, 2.68k op/s; 10943/64983 objects degraded (16.840%) 2024-09-18T07:43:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:19 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 10943/64983 objects degraded (16.840%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:19 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2905131197' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:19.489 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:19.510 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:19.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:19.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:19.612 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:19.641 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:19.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:19.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:19.743 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:19.774 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:19.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:19.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:19.875 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:19.897 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:19.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:19.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:19.999 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:20.032 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:20.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:20.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:20.053 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 10 16 27896 27880 174.235 91.875 0.00626083 0.00573343 2024-09-18T07:43:20.135 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:20.158 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:20.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:20.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:20.260 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:20.293 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:20.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:20.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:20.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:20.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:20.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:20.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:20.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:20 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:20.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:20 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2905131197' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:43:20.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:20 smithi137 ceph-mon[29408]: osdmap e370: 8 total, 7 up, 8 in 2024-09-18T07:43:20.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:20 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:20.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:20 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2905131197' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:43:20.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:20 smithi137 ceph-mon[39175]: osdmap e370: 8 total, 7 up, 8 in 2024-09-18T07:43:20.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:20 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:20 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2905131197' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:43:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:20 smithi203 ceph-mon[32237]: osdmap e370: 8 total, 7 up, 8 in 2024-09-18T07:43:20.518 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:20.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:20.551 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:20.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:20.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:20.678 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:20.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:20.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:20.780 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:20.814 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:20.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:20.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:20.916 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:20.951 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:20.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:20.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:21.053 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 11 16 29887 29871 169.707 124.438 0.00482389 0.00588833 2024-09-18T07:43:21.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:21.091 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:21.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:21.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:21.118 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:43:21.118 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:43:21.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:21.221 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:21.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:21.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:21.323 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:21.357 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:21.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:21.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:21.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:21 smithi137 ceph-mon[29408]: pgmap v657: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 666 B/s rd, 140 MiB/s wr, 2.23k op/s; 10943/64983 objects degraded (16.840%) 2024-09-18T07:43:21.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:21 smithi137 ceph-mon[39175]: pgmap v657: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 666 B/s rd, 140 MiB/s wr, 2.23k op/s; 10943/64983 objects degraded (16.840%) 2024-09-18T07:43:21.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:21.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:21 smithi203 ceph-mon[32237]: pgmap v657: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 1.3 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 666 B/s rd, 140 MiB/s wr, 2.23k op/s; 10943/64983 objects degraded (16.840%) 2024-09-18T07:43:21.495 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:21.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:21.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:21.597 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:21.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:21.629 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:21.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:21.730 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:21.758 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:21.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:21.859 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:21.889 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:21.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:21.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:21.991 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:22.019 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:22.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:22.053 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 12 16 32091 32075 167.043 137.75 0.00653812 0.00598256 2024-09-18T07:43:22.121 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:22.150 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:22.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:22.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:22.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:22.284 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:22.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:22.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:22.386 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:22.417 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:22.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:22.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:22.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:22.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:22.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:22.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:22.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:22.681 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:22.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:22.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:22.783 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:22.820 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:22.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:22.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:22.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:22.956 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:22.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:22.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:23.054 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 13 16 34084 34068 163.771 124.562 0.00695297 0.00609982 2024-09-18T07:43:23.057 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:23.086 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:23.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:23.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:23.189 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:23.226 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:23.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:23.327 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:23.357 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:23.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:23.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:23.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:23 smithi137 ceph-mon[29408]: pgmap v658: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s; 12274/68976 objects degraded (17.795%) 2024-09-18T07:43:23.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:23 smithi137 ceph-mon[39175]: pgmap v658: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s; 12274/68976 objects degraded (17.795%) 2024-09-18T07:43:23.460 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:23 smithi203 ceph-mon[32237]: pgmap v658: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s; 12274/68976 objects degraded (17.795%) 2024-09-18T07:43:23.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:23.496 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:23.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:23.598 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:23.634 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:23.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:23.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:23.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:23.771 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:23.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:23.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:23.827 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -n osd.4 --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.2.4 2024-09-18T07:43:23.872 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:23.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:23.912 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:23.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:24.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:24.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:24.049 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:24.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:24.054 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 14 16 36091 36075 161.032 125.438 0.0168098 0.00620467 2024-09-18T07:43:24.120 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/osd.4/config 2024-09-18T07:43:24.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:24.189 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:24.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:24.290 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:24.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:24.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:24.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:24.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:24 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 16136/96363 objects degraded (16.745%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:24.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:24 smithi137 ceph-mon[29408]: osdmap e371: 8 total, 7 up, 8 in 2024-09-18T07:43:24.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:24 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 16136/96363 objects degraded (16.745%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:24.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:24 smithi137 ceph-mon[39175]: osdmap e371: 8 total, 7 up, 8 in 2024-09-18T07:43:24.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:24.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:24 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 16136/96363 objects degraded (16.745%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:24.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:24 smithi203 ceph-mon[32237]: osdmap e371: 8 total, 7 up, 8 in 2024-09-18T07:43:24.462 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:24.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:24.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:24.553 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T07:43:24.560 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:24.564 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:24.589 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:24.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:24.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:24.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:24.730 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:24.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:24.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:24.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:24.853 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:24.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:24.954 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:24.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:24.998 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:24.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:25.054 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 15 16 37969 37953 158.121 117.375 0.0238608 0.00631315 2024-09-18T07:43:25.099 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:25.150 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:25.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:25.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:25.172 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:25.173 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:43:25.173 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:43:25.209 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:43:25.209 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:43:25.241 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:43:25.241 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:43:25.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:25.256 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:25 smithi203 ceph-mon[32237]: pgmap v659: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 195 MiB/s wr, 3.13k op/s; 16136/96363 objects degraded (16.745%) 2024-09-18T07:43:25.256 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:25 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:43:25.256 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:25 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:43:25.257 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:25 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2737759516' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:25.280 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:43:25.281 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:43:25.281 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-18T07:43:25.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:25.295 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:25.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:25.314 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:43:25.314 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:43:25.346 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:43:25.346 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:43:25.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:25 smithi137 ceph-mon[29408]: pgmap v659: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 195 MiB/s wr, 3.13k op/s; 16136/96363 objects degraded (16.745%) 2024-09-18T07:43:25.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:25 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:43:25.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:25 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:43:25.347 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:25 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2737759516' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:25.348 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:25 smithi137 ceph-mon[39175]: pgmap v659: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 195 MiB/s wr, 3.13k op/s; 16136/96363 objects degraded (16.745%) 2024-09-18T07:43:25.349 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:25 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:43:25.349 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:25 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:43:25.349 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:25 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2737759516' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:25.379 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:43:25.379 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:43:25.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:25.436 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:25.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:25.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:25.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:25.580 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:25.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:25.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:25.681 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:25.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:25.723 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:25.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:25.824 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:25.855 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:25.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:25.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:25.957 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:25.987 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:25.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:26.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:43:26.054 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 16 16 39984 39968 156.108 125.938 0.0120873 0.00639982 2024-09-18T07:43:26.089 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:26.115 INFO:teuthology.orchestra.run.smithi203.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-18T07:43:26.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:26.125 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:26.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:26.226 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:26.266 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:43:26.267 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:43:26.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:26.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:26.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:26.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:26.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:26.406 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:26.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:26.507 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:26.538 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:26.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:26.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:26.592 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:43:26.592 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:43:26.592 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:43:26.592 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:43:26.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.592 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134973 7.15 7:ace4f2fb:::benchmark_data_smithi137_74180_object30557:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e136)", 2024-09-18T07:43:26.593 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:15.967955+0000", 2024-09-18T07:43:26.593 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 550.61854543799996, 2024-09-18T07:43:26.593 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.149059142, 2024-09-18T07:43:26.593 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.593 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.593 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:43:26.593 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:26.593 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:26.593 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:26.594 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134973 2024-09-18T07:43:26.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.594 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.594 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.594 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967955+0000", 2024-09-18T07:43:26.594 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.594 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.594 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967955+0000", 2024-09-18T07:43:26.594 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967957+0000", 2024-09-18T07:43:26.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.154e-06 2024-09-18T07:43:26.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967964+0000", 2024-09-18T07:43:26.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1310000000000003e-06 2024-09-18T07:43:26.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967976+0000", 2024-09-18T07:43:26.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2870999999999999e-05 2024-09-18T07:43:26.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.596 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.967991+0000", 2024-09-18T07:43:26.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4174e-05 2024-09-18T07:43:26.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.596 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.969717+0000", 2024-09-18T07:43:26.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001726694 2024-09-18T07:43:26.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.597 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:43:26.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.969724+0000", 2024-09-18T07:43:26.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4200000000000004e-06 2024-09-18T07:43:26.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.597 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972756+0000", 2024-09-18T07:43:26.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030324610000000002 2024-09-18T07:43:26.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.598 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:26.598 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972766+0000", 2024-09-18T07:43:26.598 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0047999999999999e-05 2024-09-18T07:43:26.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.598 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.598 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976318+0000", 2024-09-18T07:43:26.598 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.003551327 2024-09-18T07:43:26.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.599 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:26.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976329+0000", 2024-09-18T07:43:26.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1413000000000001e-05 2024-09-18T07:43:26.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.599 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.116866+0000", 2024-09-18T07:43:26.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.140536993 2024-09-18T07:43:26.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.600 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.600 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.117014+0000", 2024-09-18T07:43:26.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014845599999999999 2024-09-18T07:43:26.600 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.600 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.600 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.600 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134975 7.15 7:a9f95f32:::benchmark_data_smithi137_74180_object30559:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e136)", 2024-09-18T07:43:26.600 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:15.968007+0000", 2024-09-18T07:43:26.600 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 550.61849373400003, 2024-09-18T07:43:26.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1497606920000001, 2024-09-18T07:43:26.600 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.601 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.601 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:43:26.601 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:26.601 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:26.601 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:26.601 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134975 2024-09-18T07:43:26.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.601 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.601 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.601 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968007+0000", 2024-09-18T07:43:26.601 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968007+0000", 2024-09-18T07:43:26.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968008+0000", 2024-09-18T07:43:26.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9100000000000003e-07 2024-09-18T07:43:26.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968011+0000", 2024-09-18T07:43:26.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0589999999999998e-06 2024-09-18T07:43:26.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968017+0000", 2024-09-18T07:43:26.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5050000000000004e-06 2024-09-18T07:43:26.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.604 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968023+0000", 2024-09-18T07:43:26.604 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6400000000000002e-06 2024-09-18T07:43:26.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.970885+0000", 2024-09-18T07:43:26.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002862505 2024-09-18T07:43:26.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.605 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-18T07:43:26.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.970892+0000", 2024-09-18T07:43:26.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2890000000000003e-06 2024-09-18T07:43:26.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.605 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972803+0000", 2024-09-18T07:43:26.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019118270000000001 2024-09-18T07:43:26.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.606 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.606 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:26.606 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972812+0000", 2024-09-18T07:43:26.606 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.653e-06 2024-09-18T07:43:26.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.606 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.606 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.606 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976352+0000", 2024-09-18T07:43:26.606 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0035398280000001 2024-09-18T07:43:26.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.607 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:26.607 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976359+0000", 2024-09-18T07:43:26.607 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3520000000000001e-06 2024-09-18T07:43:26.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.607 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.607 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.117670+0000", 2024-09-18T07:43:26.607 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.141310979 2024-09-18T07:43:26.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.607 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.608 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.117768+0000", 2024-09-18T07:43:26.608 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7363999999999998e-05 2024-09-18T07:43:26.608 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.608 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.608 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.608 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134979 7.15 7:af63a923:::benchmark_data_smithi137_74180_object30563:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e136)", 2024-09-18T07:43:26.608 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:15.968030+0000", 2024-09-18T07:43:26.608 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 550.61847065699999, 2024-09-18T07:43:26.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1501027640000001, 2024-09-18T07:43:26.609 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.609 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.609 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:43:26.609 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:26.609 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:26.609 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:26.609 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134979 2024-09-18T07:43:26.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.609 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968030+0000", 2024-09-18T07:43:26.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968030+0000", 2024-09-18T07:43:26.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.611 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968031+0000", 2024-09-18T07:43:26.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5000000000000003e-07 2024-09-18T07:43:26.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.611 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.611 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968033+0000", 2024-09-18T07:43:26.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1969999999999999e-06 2024-09-18T07:43:26.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.611 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.611 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968038+0000", 2024-09-18T07:43:26.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0919999999999997e-06 2024-09-18T07:43:26.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.612 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.612 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.968043+0000", 2024-09-18T07:43:26.612 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6619999999999996e-06 2024-09-18T07:43:26.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.612 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.612 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.970922+0000", 2024-09-18T07:43:26.612 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028795100000000001 2024-09-18T07:43:26.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-18T07:43:26.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.970927+0000", 2024-09-18T07:43:26.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0320000000000003e-06 2024-09-18T07:43:26.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972841+0000", 2024-09-18T07:43:26.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001913802 2024-09-18T07:43:26.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:26.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:15.972849+0000", 2024-09-18T07:43:26.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9039999999999992e-06 2024-09-18T07:43:26.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.614 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.614 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976382+0000", 2024-09-18T07:43:26.614 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0035336779999999 2024-09-18T07:43:26.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.614 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:26.614 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976390+0000", 2024-09-18T07:43:26.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6399999999999997e-06 2024-09-18T07:43:26.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.118071+0000", 2024-09-18T07:43:26.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.141680993 2024-09-18T07:43:26.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.118133+0000", 2024-09-18T07:43:26.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1704000000000004e-05 2024-09-18T07:43:26.615 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.616 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.616 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.616 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134988 7.12 7:4ba1d377:::benchmark_data_smithi137_74180_object30572:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:43:26.616 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964095+0000", 2024-09-18T07:43:26.616 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 549.62240618299995, 2024-09-18T07:43:26.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2226311409999999, 2024-09-18T07:43:26.616 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.616 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.616 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:26.617 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:26.617 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:26.617 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:26.617 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134988 2024-09-18T07:43:26.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.617 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.617 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.617 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964095+0000", 2024-09-18T07:43:26.617 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.617 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.617 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964095+0000", 2024-09-18T07:43:26.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964096+0000", 2024-09-18T07:43:26.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.759e-06 2024-09-18T07:43:26.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964103+0000", 2024-09-18T07:43:26.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7950000000000002e-06 2024-09-18T07:43:26.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.619 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964118+0000", 2024-09-18T07:43:26.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5082999999999999e-05 2024-09-18T07:43:26.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.619 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964128+0000", 2024-09-18T07:43:26.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6900000000000004e-06 2024-09-18T07:43:26.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.619 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964200+0000", 2024-09-18T07:43:26.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2656000000000004e-05 2024-09-18T07:43:26.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.620 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-18T07:43:26.620 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964207+0000", 2024-09-18T07:43:26.620 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5320000000000003e-06 2024-09-18T07:43:26.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.620 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.620 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976866+0000", 2024-09-18T07:43:26.620 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012658763 2024-09-18T07:43:26.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.620 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:26.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976874+0000", 2024-09-18T07:43:26.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6459999999999999e-06 2024-09-18T07:43:26.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.174506+0000", 2024-09-18T07:43:26.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1976311799999999 2024-09-18T07:43:26.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:26.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.174658+0000", 2024-09-18T07:43:26.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000152516 2024-09-18T07:43:26.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,3,6", 2024-09-18T07:43:26.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.174933+0000", 2024-09-18T07:43:26.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027465900000000001 2024-09-18T07:43:26.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:26.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184279+0000", 2024-09-18T07:43:26.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093465449999999995 2024-09-18T07:43:26.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:26.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184325+0000", 2024-09-18T07:43:26.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5219999999999997e-05 2024-09-18T07:43:26.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:26.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184838+0000", 2024-09-18T07:43:26.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00051383799999999997 2024-09-18T07:43:26.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:26.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185572+0000", 2024-09-18T07:43:26.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00073320099999999995 2024-09-18T07:43:26.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:26.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186540+0000", 2024-09-18T07:43:26.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00096810500000000003 2024-09-18T07:43:26.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:26.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186603+0000", 2024-09-18T07:43:26.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3411000000000007e-05 2024-09-18T07:43:26.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.625 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.625 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186726+0000", 2024-09-18T07:43:26.625 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012254200000000001 2024-09-18T07:43:26.625 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.625 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.625 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.626 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:134989 7.12 7:4aa85d7f:::benchmark_data_smithi137_74180_object30573:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:43:26.626 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964171+0000", 2024-09-18T07:43:26.626 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 549.622329906, 2024-09-18T07:43:26.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.222781653, 2024-09-18T07:43:26.626 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.626 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.627 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:26.627 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:26.627 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:26.627 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:26.627 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134989 2024-09-18T07:43:26.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.627 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964171+0000", 2024-09-18T07:43:26.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964171+0000", 2024-09-18T07:43:26.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964172+0000", 2024-09-18T07:43:26.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6899999999999996e-07 2024-09-18T07:43:26.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964176+0000", 2024-09-18T07:43:26.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1829999999999996e-06 2024-09-18T07:43:26.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964184+0000", 2024-09-18T07:43:26.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8290000000000008e-06 2024-09-18T07:43:26.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964190+0000", 2024-09-18T07:43:26.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8850000000000004e-06 2024-09-18T07:43:26.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.630 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.630 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964252+0000", 2024-09-18T07:43:26.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2842000000000006e-05 2024-09-18T07:43:26.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.630 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-18T07:43:26.630 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964257+0000", 2024-09-18T07:43:26.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4379999999999997e-06 2024-09-18T07:43:26.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.630 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976922+0000", 2024-09-18T07:43:26.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012665493 2024-09-18T07:43:26.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:26.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976933+0000", 2024-09-18T07:43:26.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0407999999999999e-05 2024-09-18T07:43:26.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.176181+0000", 2024-09-18T07:43:26.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1992481429999999 2024-09-18T07:43:26.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:26.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.176304+0000", 2024-09-18T07:43:26.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012344999999999999 2024-09-18T07:43:26.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,3,6", 2024-09-18T07:43:26.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.176507+0000", 2024-09-18T07:43:26.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020292699999999999 2024-09-18T07:43:26.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:26.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184373+0000", 2024-09-18T07:43:26.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0078657629999999996 2024-09-18T07:43:26.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:26.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184427+0000", 2024-09-18T07:43:26.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3863999999999998e-05 2024-09-18T07:43:26.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:26.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184878+0000", 2024-09-18T07:43:26.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00045142400000000002 2024-09-18T07:43:26.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:26.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185578+0000", 2024-09-18T07:43:26.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00069919800000000005 2024-09-18T07:43:26.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:26.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186770+0000", 2024-09-18T07:43:26.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011924430000000001 2024-09-18T07:43:26.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:26.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186830+0000", 2024-09-18T07:43:26.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0003000000000001e-05 2024-09-18T07:43:26.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.186952+0000", 2024-09-18T07:43:26.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012239099999999999 2024-09-18T07:43:26.635 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.635 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.636 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.636 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:135012 7.12 7:4b131569:::benchmark_data_smithi137_74180_object30596:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-18T07:43:26.636 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:16.964206+0000", 2024-09-18T07:43:26.636 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 549.62229489499998, 2024-09-18T07:43:26.636 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2234539019999999, 2024-09-18T07:43:26.636 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.636 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.636 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:26.636 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:26.636 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:26.636 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:26.637 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135012 2024-09-18T07:43:26.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.637 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964206+0000", 2024-09-18T07:43:26.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964206+0000", 2024-09-18T07:43:26.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964207+0000", 2024-09-18T07:43:26.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6899999999999996e-07 2024-09-18T07:43:26.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964210+0000", 2024-09-18T07:43:26.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4419999999999998e-06 2024-09-18T07:43:26.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964217+0000", 2024-09-18T07:43:26.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4989999999999999e-06 2024-09-18T07:43:26.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964222+0000", 2024-09-18T07:43:26.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5670000000000002e-06 2024-09-18T07:43:26.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964287+0000", 2024-09-18T07:43:26.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5174999999999999e-05 2024-09-18T07:43:26.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-18T07:43:26.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.964293+0000", 2024-09-18T07:43:26.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7209999999999999e-06 2024-09-18T07:43:26.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976971+0000", 2024-09-18T07:43:26.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012677541000000001 2024-09-18T07:43:26.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:43:26.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:16.976991+0000", 2024-09-18T07:43:26.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0231999999999999e-05 2024-09-18T07:43:26.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.641 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.641 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.177610+0000", 2024-09-18T07:43:26.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.200619417 2024-09-18T07:43:26.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.642 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:26.642 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.177725+0000", 2024-09-18T07:43:26.642 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114815 2024-09-18T07:43:26.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,3,6", 2024-09-18T07:43:26.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.177908+0000", 2024-09-18T07:43:26.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000182683 2024-09-18T07:43:26.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:26.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184585+0000", 2024-09-18T07:43:26.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066772230000000004 2024-09-18T07:43:26.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:26.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.184630+0000", 2024-09-18T07:43:26.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4642999999999999e-05 2024-09-18T07:43:26.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:26.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.185558+0000", 2024-09-18T07:43:26.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00092809900000000005 2024-09-18T07:43:26.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:26.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.187003+0000", 2024-09-18T07:43:26.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001445483 2024-09-18T07:43:26.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:26.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.187470+0000", 2024-09-18T07:43:26.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00046721600000000001 2024-09-18T07:43:26.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.646 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:26.646 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.187517+0000", 2024-09-18T07:43:26.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6619000000000002e-05 2024-09-18T07:43:26.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.646 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.646 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:18.187660+0000", 2024-09-18T07:43:26.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014275799999999999 2024-09-18T07:43:26.646 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.646 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.646 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.647 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:142960 7.12 7:4c4afa90:::benchmark_data_smithi137_74180_object38544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-18T07:43:26.647 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:34:25.433818+0000", 2024-09-18T07:43:26.647 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 541.15268301399999, 2024-09-18T07:43:26.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.185941245, 2024-09-18T07:43:26.647 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.647 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.647 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:26.647 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:26.647 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:43:26.647 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:43:26.647 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 142960 2024-09-18T07:43:26.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.648 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433818+0000", 2024-09-18T07:43:26.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433818+0000", 2024-09-18T07:43:26.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433820+0000", 2024-09-18T07:43:26.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3290000000000001e-06 2024-09-18T07:43:26.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433825+0000", 2024-09-18T07:43:26.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4589999999999997e-06 2024-09-18T07:43:26.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433837+0000", 2024-09-18T07:43:26.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1229999999999999e-05 2024-09-18T07:43:26.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433846+0000", 2024-09-18T07:43:26.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2790000000000004e-06 2024-09-18T07:43:26.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.433902+0000", 2024-09-18T07:43:26.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5847999999999998e-05 2024-09-18T07:43:26.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:26.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.434031+0000", 2024-09-18T07:43:26.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012924099999999999 2024-09-18T07:43:26.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2,3,6", 2024-09-18T07:43:26.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.434299+0000", 2024-09-18T07:43:26.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00026747400000000002 2024-09-18T07:43:26.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:26.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.436066+0000", 2024-09-18T07:43:26.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017675519999999999 2024-09-18T07:43:26.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:43:26.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.436687+0000", 2024-09-18T07:43:26.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00062042 2024-09-18T07:43:26.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:26.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.437458+0000", 2024-09-18T07:43:26.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00077107399999999996 2024-09-18T07:43:26.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:26.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.437498+0000", 2024-09-18T07:43:26.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0395999999999997e-05 2024-09-18T07:43:26.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:43:26.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.619551+0000", 2024-09-18T07:43:26.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.182052674 2024-09-18T07:43:26.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:26.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.619612+0000", 2024-09-18T07:43:26.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1678000000000001e-05 2024-09-18T07:43:26.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:34:25.619759+0000", 2024-09-18T07:43:26.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014659100000000001 2024-09-18T07:43:26.654 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.654 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.654 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.655 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172965 9.e e335/289 9:7fd9b0fe:::benchmark_data_smithi137_126072_object40465:head v 335'10641, mlcod=335'10639)", 2024-09-18T07:43:26.655 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.554501+0000", 2024-09-18T07:43:26.655 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 108.031999869, 2024-09-18T07:43:26.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.178309934, 2024-09-18T07:43:26.655 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.655 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.655 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:26.655 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554501+0000", 2024-09-18T07:43:26.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554501+0000", 2024-09-18T07:43:26.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554502+0000", 2024-09-18T07:43:26.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.443e-06 2024-09-18T07:43:26.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554510+0000", 2024-09-18T07:43:26.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0660000000000004e-06 2024-09-18T07:43:26.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554518+0000", 2024-09-18T07:43:26.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0339999999999997e-06 2024-09-18T07:43:26.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554523+0000", 2024-09-18T07:43:26.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8740000000000003e-06 2024-09-18T07:43:26.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554567+0000", 2024-09-18T07:43:26.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3294000000000003e-05 2024-09-18T07:43:26.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:26.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554586+0000", 2024-09-18T07:43:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9419000000000002e-05 2024-09-18T07:43:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732782+0000", 2024-09-18T07:43:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.178196153 2024-09-18T07:43:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732811+0000", 2024-09-18T07:43:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8651000000000001e-05 2024-09-18T07:43:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172968 9.0 e335/289 9:02c69111:::benchmark_data_smithi137_126072_object40468:head v 335'10807, mlcod=335'10804)", 2024-09-18T07:43:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.554702+0000", 2024-09-18T07:43:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 108.031798522, 2024-09-18T07:43:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.179882141, 2024-09-18T07:43:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554702+0000", 2024-09-18T07:43:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554702+0000", 2024-09-18T07:43:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554704+0000", 2024-09-18T07:43:26.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.359e-06 2024-09-18T07:43:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554711+0000", 2024-09-18T07:43:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.199e-06 2024-09-18T07:43:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554720+0000", 2024-09-18T07:43:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5629999999999998e-06 2024-09-18T07:43:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554727+0000", 2024-09-18T07:43:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5889999999999999e-06 2024-09-18T07:43:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554767+0000", 2024-09-18T07:43:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9712000000000002e-05 2024-09-18T07:43:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554782+0000", 2024-09-18T07:43:26.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5639000000000001e-05 2024-09-18T07:43:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734508+0000", 2024-09-18T07:43:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17972560900000001 2024-09-18T07:43:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734584+0000", 2024-09-18T07:43:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6470999999999993e-05 2024-09-18T07:43:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.664 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172969 9.6 e335/289 9:60d9b40c:::benchmark_data_smithi137_126072_object40469:head v 335'10756, mlcod=335'10754)", 2024-09-18T07:43:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.554970+0000", 2024-09-18T07:43:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 108.03153118500001, 2024-09-18T07:43:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17787228899999999, 2024-09-18T07:43:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554970+0000", 2024-09-18T07:43:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554970+0000", 2024-09-18T07:43:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554971+0000", 2024-09-18T07:43:26.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0070000000000001e-06 2024-09-18T07:43:26.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554976+0000", 2024-09-18T07:43:26.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6330000000000001e-06 2024-09-18T07:43:26.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554993+0000", 2024-09-18T07:43:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6949e-05 2024-09-18T07:43:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.555000+0000", 2024-09-18T07:43:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5579999999999997e-06 2024-09-18T07:43:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.555176+0000", 2024-09-18T07:43:26.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017662999999999999 2024-09-18T07:43:26.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:26.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.555193+0000", 2024-09-18T07:43:26.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7212000000000001e-05 2024-09-18T07:43:26.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:26.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732814+0000", 2024-09-18T07:43:26.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.177620315 2024-09-18T07:43:26.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732842+0000", 2024-09-18T07:43:26.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7985000000000001e-05 2024-09-18T07:43:26.670 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.670 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.670 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.670 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172970 9.3 e335/289 9:c7e8a587:::benchmark_data_smithi137_126072_object40470:head v 335'10785, mlcod=335'10782)", 2024-09-18T07:43:26.670 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.557577+0000", 2024-09-18T07:43:26.670 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 108.028923265, 2024-09-18T07:43:26.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.173544268, 2024-09-18T07:43:26.670 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.670 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.670 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:26.671 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557577+0000", 2024-09-18T07:43:26.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557577+0000", 2024-09-18T07:43:26.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557579+0000", 2024-09-18T07:43:26.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5099999999999999e-06 2024-09-18T07:43:26.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557589+0000", 2024-09-18T07:43:26.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8079999999999999e-06 2024-09-18T07:43:26.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557598+0000", 2024-09-18T07:43:26.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2490000000000007e-06 2024-09-18T07:43:26.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557604+0000", 2024-09-18T07:43:26.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2079999999999997e-06 2024-09-18T07:43:26.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726578+0000", 2024-09-18T07:43:26.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.168973331 2024-09-18T07:43:26.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:26.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726596+0000", 2024-09-18T07:43:26.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8159e-05 2024-09-18T07:43:26.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:26.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731084+0000", 2024-09-18T07:43:26.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044883980000000002 2024-09-18T07:43:26.674 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:26.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731122+0000", 2024-09-18T07:43:26.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7605000000000003e-05 2024-09-18T07:43:26.676 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.676 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.676 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.677 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172972 9.e e335/289 9:796b9798:::benchmark_data_smithi137_126072_object40472:head v 335'10642, mlcod=335'10639)", 2024-09-18T07:43:26.677 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.557623+0000", 2024-09-18T07:43:26.677 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 108.02887740600001, 2024-09-18T07:43:26.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.175241702, 2024-09-18T07:43:26.677 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.677 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.677 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:26.677 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557623+0000", 2024-09-18T07:43:26.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557623+0000", 2024-09-18T07:43:26.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557624+0000", 2024-09-18T07:43:26.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4200000000000001e-07 2024-09-18T07:43:26.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557628+0000", 2024-09-18T07:43:26.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8139999999999998e-06 2024-09-18T07:43:26.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557632+0000", 2024-09-18T07:43:26.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6909999999999996e-06 2024-09-18T07:43:26.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557636+0000", 2024-09-18T07:43:26.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6629999999999999e-06 2024-09-18T07:43:26.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726230+0000", 2024-09-18T07:43:26.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.168594563 2024-09-18T07:43:26.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:26.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726286+0000", 2024-09-18T07:43:26.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5702000000000003e-05 2024-09-18T07:43:26.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:26.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732845+0000", 2024-09-18T07:43:26.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065588560000000001 2024-09-18T07:43:26.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732865+0000", 2024-09-18T07:43:26.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9970999999999999e-05 2024-09-18T07:43:26.681 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.681 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.681 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.682 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172971 9.4 e335/289 9:2cb28e72:::benchmark_data_smithi137_126072_object40471:head v 335'10859, mlcod=335'10857)", 2024-09-18T07:43:26.682 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.557654+0000", 2024-09-18T07:43:26.682 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 108.02884638, 2024-09-18T07:43:26.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17145278799999999, 2024-09-18T07:43:26.682 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.682 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.682 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:26.682 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557654+0000", 2024-09-18T07:43:26.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557654+0000", 2024-09-18T07:43:26.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557655+0000", 2024-09-18T07:43:26.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1099999999999995e-07 2024-09-18T07:43:26.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557661+0000", 2024-09-18T07:43:26.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2400000000000004e-06 2024-09-18T07:43:26.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557667+0000", 2024-09-18T07:43:26.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9139999999999996e-06 2024-09-18T07:43:26.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557672+0000", 2024-09-18T07:43:26.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6290000000000001e-06 2024-09-18T07:43:26.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726169+0000", 2024-09-18T07:43:26.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.168497014 2024-09-18T07:43:26.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:26.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726237+0000", 2024-09-18T07:43:26.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8677999999999995e-05 2024-09-18T07:43:26.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:26.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.729066+0000", 2024-09-18T07:43:26.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028290170000000001 2024-09-18T07:43:26.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.729107+0000", 2024-09-18T07:43:26.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0685000000000001e-05 2024-09-18T07:43:26.687 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.687 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.687 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.687 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172976 9.f e335/289 9:f0577982:::benchmark_data_smithi137_126072_object40476:head v 335'10844, mlcod=335'10842)", 2024-09-18T07:43:26.687 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.560675+0000", 2024-09-18T07:43:26.687 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 108.025825275, 2024-09-18T07:43:26.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17558478, 2024-09-18T07:43:26.687 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.688 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.688 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:26.688 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560675+0000", 2024-09-18T07:43:26.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560675+0000", 2024-09-18T07:43:26.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560677+0000", 2024-09-18T07:43:26.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.711e-06 2024-09-18T07:43:26.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560685+0000", 2024-09-18T07:43:26.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1880000000000002e-06 2024-09-18T07:43:26.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726219+0000", 2024-09-18T07:43:26.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16553400300000001 2024-09-18T07:43:26.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726233+0000", 2024-09-18T07:43:26.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3893000000000001e-05 2024-09-18T07:43:26.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726381+0000", 2024-09-18T07:43:26.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014760199999999999 2024-09-18T07:43:26.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:26.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726402+0000", 2024-09-18T07:43:26.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1664000000000001e-05 2024-09-18T07:43:26.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:26.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.736225+0000", 2024-09-18T07:43:26.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0098227820000000004 2024-09-18T07:43:26.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.736260+0000", 2024-09-18T07:43:26.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4937000000000001e-05 2024-09-18T07:43:26.692 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.692 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.692 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.692 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172980 9.3 e335/289 9:ce0a7506:::benchmark_data_smithi137_126072_object40480:head v 335'10786, mlcod=335'10783)", 2024-09-18T07:43:26.693 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.560718+0000", 2024-09-18T07:43:26.693 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 108.025782803, 2024-09-18T07:43:26.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17056611599999999, 2024-09-18T07:43:26.693 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.693 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.693 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:26.693 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560718+0000", 2024-09-18T07:43:26.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560718+0000", 2024-09-18T07:43:26.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560719+0000", 2024-09-18T07:43:26.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8400000000000003e-07 2024-09-18T07:43:26.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560726+0000", 2024-09-18T07:43:26.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4000000000000003e-06 2024-09-18T07:43:26.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726135+0000", 2024-09-18T07:43:26.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16540897700000001 2024-09-18T07:43:26.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726166+0000", 2024-09-18T07:43:26.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.099e-05 2024-09-18T07:43:26.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.728991+0000", 2024-09-18T07:43:26.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028248840000000002 2024-09-18T07:43:26.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:26.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.729009+0000", 2024-09-18T07:43:26.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7804000000000001e-05 2024-09-18T07:43:26.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:26.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731249+0000", 2024-09-18T07:43:26.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022400050000000002 2024-09-18T07:43:26.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731284+0000", 2024-09-18T07:43:26.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5271999999999997e-05 2024-09-18T07:43:26.697 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.697 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.697 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.697 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172984 9.0 e335/289 9:066846aa:::benchmark_data_smithi137_126072_object40484:head v 335'10808, mlcod=335'10805)", 2024-09-18T07:43:26.698 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.567459+0000", 2024-09-18T07:43:26.698 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 108.01904170900001, 2024-09-18T07:43:26.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16718034100000001, 2024-09-18T07:43:26.698 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.698 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.698 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:26.698 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567459+0000", 2024-09-18T07:43:26.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567459+0000", 2024-09-18T07:43:26.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567461+0000", 2024-09-18T07:43:26.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1550000000000001e-06 2024-09-18T07:43:26.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567471+0000", 2024-09-18T07:43:26.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0348e-05 2024-09-18T07:43:26.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726150+0000", 2024-09-18T07:43:26.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15867869100000001 2024-09-18T07:43:26.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726175+0000", 2024-09-18T07:43:26.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5316e-05 2024-09-18T07:43:26.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.728757+0000", 2024-09-18T07:43:26.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002581262 2024-09-18T07:43:26.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:26.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.728778+0000", 2024-09-18T07:43:26.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1739999999999999e-05 2024-09-18T07:43:26.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:26.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734588+0000", 2024-09-18T07:43:26.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005809519 2024-09-18T07:43:26.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734639+0000", 2024-09-18T07:43:26.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1310000000000002e-05 2024-09-18T07:43:26.702 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.702 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.702 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.702 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:244796 9.d e354/289 9:b8ad6f51:::benchmark_data_smithi137_126072_object112296:head v 354'15433, mlcod=354'15430)", 2024-09-18T07:43:26.703 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:42:36.233438+0000", 2024-09-18T07:43:26.703 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 50.353062195, 2024-09-18T07:43:26.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.162856639, 2024-09-18T07:43:26.703 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.703 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.703 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:26.703 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.233438+0000", 2024-09-18T07:43:26.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.233438+0000", 2024-09-18T07:43:26.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.233440+0000", 2024-09-18T07:43:26.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.285e-06 2024-09-18T07:43:26.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.233448+0000", 2024-09-18T07:43:26.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1340000000000004e-06 2024-09-18T07:43:26.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.233485+0000", 2024-09-18T07:43:26.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7151999999999999e-05 2024-09-18T07:43:26.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.233563+0000", 2024-09-18T07:43:26.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7713000000000006e-05 2024-09-18T07:43:26.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.233745+0000", 2024-09-18T07:43:26.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000182277 2024-09-18T07:43:26.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:26.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.233765+0000", 2024-09-18T07:43:26.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0064e-05 2024-09-18T07:43:26.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:26.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396250+0000", 2024-09-18T07:43:26.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.162485294 2024-09-18T07:43:26.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396295+0000", 2024-09-18T07:43:26.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4719999999999999e-05 2024-09-18T07:43:26.708 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.708 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.708 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.708 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:244803 9.d e354/289 9:be42e048:::benchmark_data_smithi137_126072_object112303:head v 354'15434, mlcod=354'15431)", 2024-09-18T07:43:26.708 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:42:36.237930+0000", 2024-09-18T07:43:26.708 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 50.348570373999998, 2024-09-18T07:43:26.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.163469172, 2024-09-18T07:43:26.708 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.708 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.709 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:26.709 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.237930+0000", 2024-09-18T07:43:26.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.237930+0000", 2024-09-18T07:43:26.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.237932+0000", 2024-09-18T07:43:26.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8109999999999999e-06 2024-09-18T07:43:26.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.237944+0000", 2024-09-18T07:43:26.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2159e-05 2024-09-18T07:43:26.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396238+0000", 2024-09-18T07:43:26.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15829367 2024-09-18T07:43:26.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396256+0000", 2024-09-18T07:43:26.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7621e-05 2024-09-18T07:43:26.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396319+0000", 2024-09-18T07:43:26.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3726000000000006e-05 2024-09-18T07:43:26.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:26.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396337+0000", 2024-09-18T07:43:26.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7825e-05 2024-09-18T07:43:26.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:26.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.401117+0000", 2024-09-18T07:43:26.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00478021 2024-09-18T07:43:26.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.401399+0000", 2024-09-18T07:43:26.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028215 2024-09-18T07:43:26.713 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.713 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.713 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.713 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:244806 9.f e354/289 9:f6d427cd:::benchmark_data_smithi137_126072_object112306:head v 354'15363, mlcod=354'15361)", 2024-09-18T07:43:26.713 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:42:36.240579+0000", 2024-09-18T07:43:26.713 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 50.345921885999999, 2024-09-18T07:43:26.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16868968600000001, 2024-09-18T07:43:26.713 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.713 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.713 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:26.714 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240579+0000", 2024-09-18T07:43:26.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240579+0000", 2024-09-18T07:43:26.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240581+0000", 2024-09-18T07:43:26.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9599999999999999e-06 2024-09-18T07:43:26.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240589+0000", 2024-09-18T07:43:26.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4710000000000007e-06 2024-09-18T07:43:26.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396235+0000", 2024-09-18T07:43:26.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.155645916 2024-09-18T07:43:26.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396253+0000", 2024-09-18T07:43:26.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8170000000000001e-05 2024-09-18T07:43:26.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396312+0000", 2024-09-18T07:43:26.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9144000000000002e-05 2024-09-18T07:43:26.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:26.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396337+0000", 2024-09-18T07:43:26.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4771000000000001e-05 2024-09-18T07:43:26.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:26.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.409231+0000", 2024-09-18T07:43:26.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012893745 2024-09-18T07:43:26.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.409268+0000", 2024-09-18T07:43:26.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7509000000000003e-05 2024-09-18T07:43:26.717 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.717 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.718 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.718 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:244808 9.0 e354/289 9:06cfac81:::benchmark_data_smithi137_126072_object112308:head v 354'15323, mlcod=354'15320)", 2024-09-18T07:43:26.718 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:42:36.240688+0000", 2024-09-18T07:43:26.718 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 50.345812449, 2024-09-18T07:43:26.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.164927872, 2024-09-18T07:43:26.718 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:26.719 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:26.719 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:43:26.719 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:26.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:26.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240688+0000", 2024-09-18T07:43:26.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:26.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240688+0000", 2024-09-18T07:43:26.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:26.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:26.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240690+0000", 2024-09-18T07:43:26.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3999999999999999e-06 2024-09-18T07:43:26.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:26.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240697+0000", 2024-09-18T07:43:26.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3549999999999999e-06 2024-09-18T07:43:26.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:26.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396234+0000", 2024-09-18T07:43:26.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15553705900000001 2024-09-18T07:43:26.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:26.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396253+0000", 2024-09-18T07:43:26.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9264000000000001e-05 2024-09-18T07:43:26.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:26.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396315+0000", 2024-09-18T07:43:26.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1450000000000003e-05 2024-09-18T07:43:26.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:26.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396337+0000", 2024-09-18T07:43:26.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2348999999999999e-05 2024-09-18T07:43:26.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:43:26.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.405580+0000", 2024-09-18T07:43:26.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.009242676 2024-09-18T07:43:26.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:26.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:26.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:43:26.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.405616+0000", 2024-09-18T07:43:26.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6319000000000002e-05 2024-09-18T07:43:26.722 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.723 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.723 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.723 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:26.723 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:26.723 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:43:26.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:26.724 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:26.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:26.825 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:26.857 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:26.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:26.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:26.959 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:26.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:26.992 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:26.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:27.022 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:27.055 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 17 16 41783 41767 153.529 112.438 0.00494491 0.00650852 2024-09-18T07:43:27.094 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:27.135 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:27.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:27.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:27 smithi203 ceph-mon[32237]: pgmap v661: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s; 16136/96363 objects degraded (16.745%) 2024-09-18T07:43:27.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:27 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:43:27.170 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:27 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:43:27.237 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:27.268 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:27.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:27.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:27.370 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:27.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:27.405 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:27.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:27.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:27 smithi137 ceph-mon[29408]: pgmap v661: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s; 16136/96363 objects degraded (16.745%) 2024-09-18T07:43:27.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:27 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:43:27.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:27 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:43:27.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:43:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:43:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:43:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:27 smithi137 ceph-mon[39175]: pgmap v661: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s; 16136/96363 objects degraded (16.745%) 2024-09-18T07:43:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:27 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:43:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:27 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:43:27.506 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:27.538 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:27.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:27.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:27.640 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:27.673 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:27.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:27.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:27.674 INFO:teuthology.orchestra.run.smithi203.stdout:get_pg_num_history pg_num_history pg_num_history(e358 pg_nums {1={272=11},2={59=3},3={60=32},4={62=32},5={64=32},6={66=32},9={289=16}} deleted_pools ) 2024-09-18T07:43:27.675 INFO:teuthology.orchestra.run.smithi203.stdout:Importing pgid 1.2 2024-09-18T07:43:27.675 INFO:teuthology.orchestra.run.smithi203.stdout:write_pg epoch 318 info 1.2( v 73'79 (0'0,73'79] local-lis/les=274/275 n=1 ec=272/19 lis/c=274/274 les/c/f=275/275/0 sis=274) 2024-09-18T07:43:27.675 INFO:teuthology.orchestra.run.smithi203.stdout:Import successful 2024-09-18T07:43:27.775 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:27.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T07:43:27.804 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:27.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:27.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:27.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:27.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:27.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:27.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:27.960 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:43:27 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:43:28.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:28.057 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 18 16 43814 43798 152.039 126.938 0.00479811 0.00657214 2024-09-18T07:43:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:28.102 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:28.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:28.203 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:28.256 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:28.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:28.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:28.268 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:43:28.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:28.420 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:28.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:28.425 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:28.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:28.526 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:28.552 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:28.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:28.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:28.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:28.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:28.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:28.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:28.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:28.820 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:28.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:28.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:28.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:29.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:29.017 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:29.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:29.047 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:43:29.047 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:43:29.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.047 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:45376 10.3 10:ce880081:::benchmark_data_smithi137_151160_object45375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-18T07:43:29.047 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:29.036190+0000", 2024-09-18T07:43:29.047 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0047807379999999997, 2024-09-18T07:43:29.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004821283, 2024-09-18T07:43:29.047 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:29.047 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:29.048 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:43:29.048 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:29.048 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:43:29.048 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:43:29.048 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 45376 2024-09-18T07:43:29.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.048 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:29.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.048 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:29.048 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.036190+0000", 2024-09-18T07:43:29.048 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:29.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.048 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:29.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.036190+0000", 2024-09-18T07:43:29.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:29.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.049 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:29.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.036192+0000", 2024-09-18T07:43:29.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.951e-06 2024-09-18T07:43:29.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.049 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:29.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.036216+0000", 2024-09-18T07:43:29.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3811e-05 2024-09-18T07:43:29.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.050 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:29.050 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.036227+0000", 2024-09-18T07:43:29.050 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1450999999999999e-05 2024-09-18T07:43:29.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.050 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:29.050 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.036235+0000", 2024-09-18T07:43:29.050 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0169999999999996e-06 2024-09-18T07:43:29.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.050 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:29.051 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.036426+0000", 2024-09-18T07:43:29.051 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000190619 2024-09-18T07:43:29.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.051 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:29.051 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.036569+0000", 2024-09-18T07:43:29.051 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014276499999999999 2024-09-18T07:43:29.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.051 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-09-18T07:43:29.051 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.036776+0000", 2024-09-18T07:43:29.051 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020689600000000001 2024-09-18T07:43:29.051 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:29.051 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:29.052 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:29.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.052 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:45381 10.3 10:c3dce073:::benchmark_data_smithi137_151160_object45380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-18T07:43:29.052 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:29.036283+0000", 2024-09-18T07:43:29.052 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0046879479999999999, 2024-09-18T07:43:29.052 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047992699999999996, 2024-09-18T07:43:29.052 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:29.052 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:29.052 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:43:29.052 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:43:29.052 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:43:29.053 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:43:29.053 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 45381 2024-09-18T07:43:29.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.053 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:29.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:29.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.036283+0000", 2024-09-18T07:43:29.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:29.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:29.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.036283+0000", 2024-09-18T07:43:29.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:29.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:29.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.036284+0000", 2024-09-18T07:43:29.054 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3600000000000002e-07 2024-09-18T07:43:29.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:29.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.036330+0000", 2024-09-18T07:43:29.054 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6008999999999999e-05 2024-09-18T07:43:29.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:29.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.036337+0000", 2024-09-18T07:43:29.054 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5549999999999997e-06 2024-09-18T07:43:29.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.055 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:29.055 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.036343+0000", 2024-09-18T07:43:29.055 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6699999999999999e-06 2024-09-18T07:43:29.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.055 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:29.055 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.039299+0000", 2024-09-18T07:43:29.055 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029558890000000002 2024-09-18T07:43:29.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.055 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:29.055 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.039469+0000", 2024-09-18T07:43:29.055 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017015699999999999 2024-09-18T07:43:29.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.056 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.056 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-09-18T07:43:29.056 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.039613+0000", 2024-09-18T07:43:29.056 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000144093 2024-09-18T07:43:29.056 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:29.056 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:29.056 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:29.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.056 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.056 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.36014.0:45376 10.3 e371/364)", 2024-09-18T07:43:29.056 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:29.039284+0000", 2024-09-18T07:43:29.056 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.001686896, 2024-09-18T07:43:29.056 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001852942, 2024-09-18T07:43:29.056 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:29.057 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:29.057 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:43:29.057 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:29.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.057 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:29.057 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.039284+0000", 2024-09-18T07:43:29.057 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:29.057 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.057 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:29.057 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.039284+0000", 2024-09-18T07:43:29.057 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:29.057 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.058 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:29.058 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.039285+0000", 2024-09-18T07:43:29.058 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1999999999999999e-06 2024-09-18T07:43:29.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.058 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.058 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:29.058 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.039292+0000", 2024-09-18T07:43:29.058 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9229999999999996e-06 2024-09-18T07:43:29.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.058 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.058 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:29.058 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.039302+0000", 2024-09-18T07:43:29.058 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7480000000000005e-06 2024-09-18T07:43:29.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.058 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.058 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:29.059 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.039308+0000", 2024-09-18T07:43:29.059 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2099999999999998e-06 2024-09-18T07:43:29.059 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:29.059 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:29.059 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:29.059 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.059 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:45384 10.9 e371/364 10:991efe52:::benchmark_data_smithi137_151160_object45383:head v 371'2846, mlcod=371'2844)", 2024-09-18T07:43:29.059 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:29.039802+0000", 2024-09-18T07:43:29.059 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.001168608, 2024-09-18T07:43:29.059 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001373272, 2024-09-18T07:43:29.059 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:29.059 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:29.059 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:43:29.059 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:29.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.060 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:29.060 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.039802+0000", 2024-09-18T07:43:29.060 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:29.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.060 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:29.060 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.039802+0000", 2024-09-18T07:43:29.060 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:29.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.060 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:29.061 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.039803+0000", 2024-09-18T07:43:29.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2139999999999999e-06 2024-09-18T07:43:29.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.061 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:29.061 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.039834+0000", 2024-09-18T07:43:29.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0651000000000002e-05 2024-09-18T07:43:29.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.061 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:29.061 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.039848+0000", 2024-09-18T07:43:29.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4005000000000001e-05 2024-09-18T07:43:29.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.062 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:29.062 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.039855+0000", 2024-09-18T07:43:29.062 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1330000000000001e-06 2024-09-18T07:43:29.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.062 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:29.062 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.039925+0000", 2024-09-18T07:43:29.062 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0308999999999994e-05 2024-09-18T07:43:29.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:29.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:29.062 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:29.062 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:29.039950+0000", 2024-09-18T07:43:29.062 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4932000000000001e-05 2024-09-18T07:43:29.063 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:29.063 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:29.063 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:29.063 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:29.063 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:43:29.063 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 4 2024-09-18T07:43:29.063 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:43:29.063 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 19 16 45408 45392 149.273 99.625 0.0168352 0.00669298 2024-09-18T07:43:29.067 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:29.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:29.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:29.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:29.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:29.162 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/exp.1.2.4 2024-09-18T07:43:29.180 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:43:29.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:29.300 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:29.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:29.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:29.402 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:29.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:29 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:29.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:29 smithi137 ceph-mon[29408]: pgmap v662: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 2.5 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 147 MiB/s wr, 2.35k op/s; 20999/125766 objects degraded (16.697%) 2024-09-18T07:43:29.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:29 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3196760971' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:43:29.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:29 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:29.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:29 smithi137 ceph-mon[39175]: pgmap v662: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 2.5 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 147 MiB/s wr, 2.35k op/s; 20999/125766 objects degraded (16.697%) 2024-09-18T07:43:29.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:29 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3196760971' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:43:29.437 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:29.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:29.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:29 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:29 smithi203 ceph-mon[32237]: pgmap v662: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 2.5 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 147 MiB/s wr, 2.35k op/s; 20999/125766 objects degraded (16.697%) 2024-09-18T07:43:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:29 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3196760971' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:43:29.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:29.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:29.581 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:29.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:29.683 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:29.710 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:29.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:29.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:29.811 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:29.847 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:29.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:29.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:29.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:29.987 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:29.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:29.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:30.058 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:43:30.056602+0000 min lat: 0.00105851 max lat: 0.226985 avg lat: 0.00676049 2024-09-18T07:43:30.058 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:43:30.058 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 20 16 47329 47313 147.812 120.062 0.00672311 0.00676049 2024-09-18T07:43:30.089 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:30.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:30.125 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:30.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:30.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:30.269 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:30.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:30.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:30.371 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:30.407 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:30.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:30.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:30 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:30.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:30 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3196760971' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:43:30.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:30 smithi137 ceph-mon[29408]: osdmap e372: 8 total, 7 up, 8 in 2024-09-18T07:43:30.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:30 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 20999/125766 objects degraded (16.697%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:30.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:30 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:30.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:30 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3196760971' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:43:30.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:30 smithi137 ceph-mon[39175]: osdmap e372: 8 total, 7 up, 8 in 2024-09-18T07:43:30.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:30 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 20999/125766 objects degraded (16.697%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:30 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:30 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3196760971' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:43:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:30 smithi203 ceph-mon[32237]: osdmap e372: 8 total, 7 up, 8 in 2024-09-18T07:43:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:30 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 20999/125766 objects degraded (16.697%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:30.509 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:30.543 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:30.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:30.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:30.645 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:30.674 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:30.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:30.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:30.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T07:43:30.776 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:30.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:30.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:30.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:30.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:30.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:30.963 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:30.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:31.059 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 21 16 49136 49120 146.145 112.938 0.0110694 0.00683832 2024-09-18T07:43:31.064 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:31.097 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:31.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:31.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:31.199 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:31.227 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:31.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:31.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:31.273 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:43:31.273 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:43:31.301 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:31.311 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:43:31.312 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:43:31.329 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:31.348 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:43:31.348 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:43:31.360 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:31.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:31.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:31.380 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:43:31.380 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:43:31.384 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-18T07:43:31.414 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:43:31.414 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:43:31.443 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:31 smithi137 ceph-mon[29408]: pgmap v664: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 2.5 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 148 MiB/s wr, 2.37k op/s; 20999/125766 objects degraded (16.697%) 2024-09-18T07:43:31.443 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:31 smithi137 ceph-mon[39175]: pgmap v664: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 2.5 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 148 MiB/s wr, 2.37k op/s; 20999/125766 objects degraded (16.697%) 2024-09-18T07:43:31.445 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:43:31.446 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:43:31.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:31 smithi203 ceph-mon[32237]: pgmap v664: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 2.5 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 148 MiB/s wr, 2.37k op/s; 20999/125766 objects degraded (16.697%) 2024-09-18T07:43:31.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:31.479 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:43:31.479 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:43:31.497 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:31.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:31.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:31.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:31.631 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:31.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:31.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:31.733 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:31.766 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:31.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:31.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:31.868 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:31.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:31.909 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:31.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:32.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:32.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:32.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:32.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:32.059 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 22 16 50827 50811 144.306 105.688 0.0038064 0.00691151 2024-09-18T07:43:32.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:32.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:32.173 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:32.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:32.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:43:32.274 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:32.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:32.312 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:32.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:32.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:32.440 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:32.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:32.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:32.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:32 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3825407802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:32.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:32 smithi203 ceph-mon[32237]: 3.12 scrub starts 2024-09-18T07:43:32.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:32 smithi203 ceph-mon[32237]: 3.12 scrub ok 2024-09-18T07:43:32.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:32.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:32.587 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:32.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:32.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:32 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3825407802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:32.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:32 smithi137 ceph-mon[29408]: 3.12 scrub starts 2024-09-18T07:43:32.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:32 smithi137 ceph-mon[29408]: 3.12 scrub ok 2024-09-18T07:43:32.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:32 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3825407802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:32.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:32 smithi137 ceph-mon[39175]: 3.12 scrub starts 2024-09-18T07:43:32.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:32 smithi137 ceph-mon[39175]: 3.12 scrub ok 2024-09-18T07:43:32.689 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:32.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:32.711 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:32.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:32.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:32.849 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:32.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:32.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:32.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:32.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:32.987 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:32.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:33.059 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 23 16 52532 52516 142.665 106.562 0.00589138 0.00700456 2024-09-18T07:43:33.088 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:33.115 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:33.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:33.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:33.251 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:33.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:33.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:33.353 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:33.388 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:33.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:33.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:33.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:33 smithi203 ceph-mon[32237]: pgmap v665: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 2.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s; 22105/129084 objects degraded (17.125%) 2024-09-18T07:43:33.490 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:33.520 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:33.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:33.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:33.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:33.659 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:33.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:33.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:33 smithi137 ceph-mon[29408]: pgmap v665: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 2.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s; 22105/129084 objects degraded (17.125%) 2024-09-18T07:43:33.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:33 smithi137 ceph-mon[39175]: pgmap v665: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 2.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s; 22105/129084 objects degraded (17.125%) 2024-09-18T07:43:33.761 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:33.774 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:43:33.774 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:43:33.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:33.803 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:33.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:33.904 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:33.931 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:33.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:33.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:34.033 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:34.061 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 24 16 54211 54195 141.08 104.938 0.0176261 0.00708262 2024-09-18T07:43:34.074 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:34.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:34.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:34.176 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:34.199 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-09-18T07:43:34.200 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:43:34.200 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-18T07:43:34.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-18T07:43:34.203 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:34.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:34.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:34.206 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:43:34.206 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:43:34.225 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:34.305 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:34.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:34.345 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:34.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:34.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:34 smithi137 ceph-mon[29408]: 3.16 scrub starts 2024-09-18T07:43:34.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:34 smithi137 ceph-mon[29408]: 3.16 scrub ok 2024-09-18T07:43:34.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:34 smithi137 ceph-mon[29408]: 6.16 scrub starts 2024-09-18T07:43:34.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:34 smithi137 ceph-mon[29408]: 6.16 scrub ok 2024-09-18T07:43:34.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:34 smithi137 ceph-mon[39175]: 3.16 scrub starts 2024-09-18T07:43:34.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:34 smithi137 ceph-mon[39175]: 3.16 scrub ok 2024-09-18T07:43:34.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:34 smithi137 ceph-mon[39175]: 6.16 scrub starts 2024-09-18T07:43:34.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:34 smithi137 ceph-mon[39175]: 6.16 scrub ok 2024-09-18T07:43:34.446 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:34.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:34.495 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:34.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:34.596 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:34.626 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:34.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:34.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:34.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:34 smithi203 ceph-mon[32237]: 3.16 scrub starts 2024-09-18T07:43:34.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:34 smithi203 ceph-mon[32237]: 3.16 scrub ok 2024-09-18T07:43:34.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:34 smithi203 ceph-mon[32237]: 6.16 scrub starts 2024-09-18T07:43:34.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:34 smithi203 ceph-mon[32237]: 6.16 scrub ok 2024-09-18T07:43:34.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:34.771 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:34.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:34.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:34.873 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:34.907 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:34.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:34.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:35.009 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:35.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:35.049 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:35.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:35.061 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 25 16 55814 55798 139.445 100.188 0.0204273 0.00716606 2024-09-18T07:43:35.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:35.186 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:35.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:35.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:35.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:35.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T07:43:35.324 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:35.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:35.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:35.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:35.481 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:35.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:35.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:35.527 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:35 smithi137 ceph-mon[29408]: pgmap v666: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 122 MiB/s wr, 1.96k op/s; 25465/152742 objects degraded (16.672%) 2024-09-18T07:43:35.527 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:35 smithi137 ceph-mon[29408]: 6.9 scrub starts 2024-09-18T07:43:35.528 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:35 smithi137 ceph-mon[29408]: 6.9 scrub ok 2024-09-18T07:43:35.528 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:35 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 25465/152742 objects degraded (16.672%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:35.528 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:35 smithi137 ceph-mon[39175]: pgmap v666: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 122 MiB/s wr, 1.96k op/s; 25465/152742 objects degraded (16.672%) 2024-09-18T07:43:35.528 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:35 smithi137 ceph-mon[39175]: 6.9 scrub starts 2024-09-18T07:43:35.528 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:35 smithi137 ceph-mon[39175]: 6.9 scrub ok 2024-09-18T07:43:35.528 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:35 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 25465/152742 objects degraded (16.672%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:35.584 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:35.609 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:35.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:35.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:35.610 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:35.710 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:35 smithi203 ceph-mon[32237]: pgmap v666: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 122 MiB/s wr, 1.96k op/s; 25465/152742 objects degraded (16.672%) 2024-09-18T07:43:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:35 smithi203 ceph-mon[32237]: 6.9 scrub starts 2024-09-18T07:43:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:35 smithi203 ceph-mon[32237]: 6.9 scrub ok 2024-09-18T07:43:35.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:35 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 25465/152742 objects degraded (16.672%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:35.737 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:35.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:35.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:35.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:43:35.838 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:35.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:35.884 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:35.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:35.985 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:36.011 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:36.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:36.061 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 26 15 57246 57231 137.526 89.5625 0.00799282 0.00726602 2024-09-18T07:43:36.113 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:36.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:36.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:36.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:36.256 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:43:36.260 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:36.285 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:36.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:36.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:36.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:36.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:36 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/544578348' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:36.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:36 smithi137 ceph-mon[29408]: from='client.36050 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:36.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:36 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/544578348' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:36.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:36 smithi137 ceph-mon[39175]: from='client.36050 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:36.425 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:36.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:36.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:36.527 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:36.553 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:36.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:36.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:36.620 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:43:36.620 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:43:36.620 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 373, 2024-09-18T07:43:36.620 INFO:teuthology.orchestra.run.smithi137.stdout: "fsid": "f7ef099c-758e-11ef-bceb-c7b262605968", 2024-09-18T07:43:36.620 INFO:teuthology.orchestra.run.smithi137.stdout: "created": "2024-09-18T07:25:19.662367+0000", 2024-09-18T07:43:36.620 INFO:teuthology.orchestra.run.smithi137.stdout: "modified": "2024-09-18T07:43:36.218416+0000", 2024-09-18T07:43:36.621 INFO:teuthology.orchestra.run.smithi137.stdout: "last_up_change": "2024-09-18T07:42:53.723776+0000", 2024-09-18T07:43:36.621 INFO:teuthology.orchestra.run.smithi137.stdout: "last_in_change": "2024-09-18T07:37:42.022730+0000", 2024-09-18T07:43:36.621 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T07:43:36.621 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_num": 5799936, 2024-09-18T07:43:36.621 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_set": [ 2024-09-18T07:43:36.621 INFO:teuthology.orchestra.run.smithi137.stdout: "pglog_hardlimit", 2024-09-18T07:43:36.621 INFO:teuthology.orchestra.run.smithi137.stdout: "purged_snapdirs", 2024-09-18T07:43:36.621 INFO:teuthology.orchestra.run.smithi137.stdout: "recovery_deletes", 2024-09-18T07:43:36.621 INFO:teuthology.orchestra.run.smithi137.stdout: "sortbitwise" 2024-09-18T07:43:36.621 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:43:36.621 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_version": 18, 2024-09-18T07:43:36.621 INFO:teuthology.orchestra.run.smithi137.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T07:43:36.622 INFO:teuthology.orchestra.run.smithi137.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T07:43:36.622 INFO:teuthology.orchestra.run.smithi137.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T07:43:36.622 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_snapshot": "", 2024-09-18T07:43:36.622 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_max": 10, 2024-09-18T07:43:36.622 INFO:teuthology.orchestra.run.smithi137.stdout: "max_osd": 8, 2024-09-18T07:43:36.622 INFO:teuthology.orchestra.run.smithi137.stdout: "require_min_compat_client": "luminous", 2024-09-18T07:43:36.622 INFO:teuthology.orchestra.run.smithi137.stdout: "min_compat_client": "luminous", 2024-09-18T07:43:36.622 INFO:teuthology.orchestra.run.smithi137.stdout: "require_osd_release": "squid", 2024-09-18T07:43:36.622 INFO:teuthology.orchestra.run.smithi137.stdout: "allow_crimson": false, 2024-09-18T07:43:36.622 INFO:teuthology.orchestra.run.smithi137.stdout: "pools": [ 2024-09-18T07:43:36.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.622 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 1, 2024-09-18T07:43:36.622 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".mgr", 2024-09-18T07:43:36.623 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:28:20.899655+0000", 2024-09-18T07:43:36.623 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:43:36.623 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:43:36.623 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:43:36.623 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:43:36.623 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:43:36.623 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:43:36.623 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:43:36.623 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:43:36.623 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:43:36.623 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:43:36.623 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:43:36.624 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:43:36.624 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:43:36.624 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 11, 2024-09-18T07:43:36.624 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 11, 2024-09-18T07:43:36.624 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 11, 2024-09-18T07:43:36.624 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 11, 2024-09-18T07:43:36.624 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 11, 2024-09-18T07:43:36.624 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:43:36.624 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:43:36.624 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:43:36.624 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:43:36.624 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:43:36.625 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:43:36.625 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:43:36.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.625 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "274", 2024-09-18T07:43:36.625 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:43:36.625 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:43:36.625 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "272", 2024-09-18T07:43:36.625 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:43:36.625 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:43:36.625 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:43:36.625 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:43:36.625 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:43:36.626 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:43:36.626 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:43:36.626 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:43:36.626 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:43:36.626 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:43:36.626 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:43:36.626 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:43:36.626 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:43:36.626 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:43:36.626 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:43:36.626 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:43:36.626 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:43:36.626 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:43:36.627 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:43:36.627 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:43:36.627 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:43:36.627 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:43:36.627 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:43:36.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.627 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:43:36.627 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:43:36.627 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:43:36.627 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:43:36.627 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:43:36.627 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:43:36.628 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:43:36.628 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:43:36.628 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:43:36.628 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:43:36.628 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:43:36.628 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:43:36.628 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_max": 32, 2024-09-18T07:43:36.628 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_min": 1 2024-09-18T07:43:36.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.628 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:43:36.628 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": {} 2024-09-18T07:43:36.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.628 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:43:36.629 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:43:36.629 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.9099999666213989, 2024-09-18T07:43:36.629 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.9099999666213989, 2024-09-18T07:43:36.629 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:43:36.629 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.9099999666213989, 2024-09-18T07:43:36.629 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.9099999666213989, 2024-09-18T07:43:36.629 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:43:36.629 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:43:36.629 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:43:36.629 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.630 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 2, 2024-09-18T07:43:36.630 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "datapool", 2024-09-18T07:43:36.630 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:53.715373+0000", 2024-09-18T07:43:36.630 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-18T07:43:36.630 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T07:43:36.630 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:43:36.630 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:43:36.630 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:43:36.630 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:43:36.630 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:43:36.630 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:43:36.630 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:43:36.630 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:43:36.630 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:43:36.631 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:43:36.631 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:43:36.631 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 3, 2024-09-18T07:43:36.631 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 3, 2024-09-18T07:43:36.631 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 3, 2024-09-18T07:43:36.631 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 3, 2024-09-18T07:43:36.631 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 3, 2024-09-18T07:43:36.631 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:43:36.631 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:43:36.631 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:43:36.631 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:43:36.631 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:43:36.631 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:43:36.632 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:43:36.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.632 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "63", 2024-09-18T07:43:36.632 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:43:36.632 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:43:36.632 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:43:36.632 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:43:36.632 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:43:36.632 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 2, 2024-09-18T07:43:36.632 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 63, 2024-09-18T07:43:36.632 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:43:36.632 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:43:36.633 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:43:36.633 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:43:36.633 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:43:36.633 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:43:36.633 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:43:36.633 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:43:36.633 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:43:36.633 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:43:36.633 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:43:36.633 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:43:36.633 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:43:36.633 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:43:36.633 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:43:36.634 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:43:36.634 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:43:36.634 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:43:36.634 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:43:36.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.634 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:43:36.634 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:43:36.634 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:43:36.634 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:43:36.634 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:43:36.634 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:43:36.634 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:43:36.634 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:43:36.635 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:43:36.635 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:43:36.635 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:43:36.635 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:43:36.635 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:43:36.635 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd": {} 2024-09-18T07:43:36.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.635 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:43:36.635 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:43:36.635 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.3299999237060547, 2024-09-18T07:43:36.635 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.3299999237060547, 2024-09-18T07:43:36.635 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:43:36.635 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-18T07:43:36.635 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-18T07:43:36.636 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:43:36.636 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:43:36.636 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:43:36.636 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.636 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 3, 2024-09-18T07:43:36.636 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".rgw.root", 2024-09-18T07:43:36.636 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:54.672127+0000", 2024-09-18T07:43:36.636 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:43:36.636 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:43:36.636 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:43:36.636 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:43:36.636 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:43:36.637 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:43:36.637 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:43:36.637 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:43:36.637 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:43:36.637 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:43:36.637 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:43:36.637 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:43:36.637 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:43:36.637 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:43:36.637 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:43:36.637 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:43:36.637 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:43:36.637 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:43:36.638 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:43:36.638 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:43:36.638 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:43:36.638 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:43:36.638 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:43:36.638 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:43:36.638 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:43:36.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.638 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "62", 2024-09-18T07:43:36.638 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:43:36.638 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:43:36.638 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:43:36.638 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:43:36.638 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:43:36.639 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:43:36.639 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:43:36.639 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:43:36.639 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:43:36.639 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:43:36.639 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:43:36.639 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:43:36.639 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:43:36.639 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:43:36.639 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:43:36.639 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:43:36.639 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:43:36.639 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:43:36.639 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:43:36.639 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:43:36.640 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:43:36.640 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:43:36.640 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:43:36.640 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:43:36.640 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:43:36.640 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:43:36.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.640 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:43:36.640 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:43:36.640 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:43:36.640 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:43:36.640 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:43:36.641 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:43:36.641 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:43:36.641 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:43:36.641 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:43:36.641 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:43:36.641 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:43:36.641 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:43:36.641 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:43:36.641 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:43:36.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.641 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:43:36.641 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:43:36.641 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.75, 2024-09-18T07:43:36.642 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.75, 2024-09-18T07:43:36.642 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:43:36.642 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.75, 2024-09-18T07:43:36.642 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.75, 2024-09-18T07:43:36.642 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:43:36.642 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:43:36.642 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:43:36.642 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.642 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 4, 2024-09-18T07:43:36.642 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.log", 2024-09-18T07:43:36.642 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:56.351343+0000", 2024-09-18T07:43:36.643 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:43:36.643 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:43:36.643 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:43:36.643 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:43:36.643 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:43:36.643 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:43:36.643 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:43:36.643 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:43:36.643 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:43:36.643 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:43:36.643 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:43:36.643 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:43:36.643 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:43:36.644 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:43:36.644 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:43:36.644 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:43:36.644 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:43:36.644 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:43:36.644 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:43:36.644 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:43:36.644 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:43:36.644 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:43:36.644 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:43:36.644 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:43:36.644 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:43:36.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.644 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "64", 2024-09-18T07:43:36.645 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:43:36.645 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:43:36.645 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:43:36.645 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:43:36.645 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:43:36.645 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:43:36.645 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:43:36.645 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:43:36.645 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:43:36.645 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:43:36.645 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:43:36.645 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:43:36.645 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:43:36.646 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:43:36.646 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:43:36.646 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:43:36.646 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:43:36.646 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:43:36.646 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:43:36.646 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:43:36.646 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:43:36.646 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:43:36.646 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:43:36.646 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:43:36.646 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:43:36.646 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:43:36.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.646 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:43:36.647 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:43:36.647 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:43:36.647 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:43:36.647 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:43:36.647 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:43:36.647 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:43:36.647 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:43:36.647 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:43:36.647 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:43:36.647 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:43:36.647 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:43:36.647 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:43:36.647 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:43:36.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.647 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:43:36.648 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:43:36.648 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.5299999713897705, 2024-09-18T07:43:36.648 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.5299999713897705, 2024-09-18T07:43:36.648 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:43:36.648 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-18T07:43:36.648 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-18T07:43:36.648 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:43:36.648 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:43:36.648 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:43:36.648 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.648 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 5, 2024-09-18T07:43:36.649 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.control", 2024-09-18T07:43:36.649 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:58.322401+0000", 2024-09-18T07:43:36.649 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:43:36.649 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:43:36.649 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:43:36.649 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:43:36.649 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:43:36.649 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:43:36.649 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:43:36.649 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:43:36.649 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:43:36.649 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:43:36.649 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:43:36.649 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:43:36.650 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:43:36.650 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:43:36.650 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:43:36.650 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:43:36.650 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:43:36.650 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:43:36.650 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:43:36.650 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:43:36.650 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:43:36.650 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:43:36.650 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:43:36.650 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:43:36.650 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:43:36.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.650 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "66", 2024-09-18T07:43:36.651 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:43:36.651 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:43:36.651 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:43:36.651 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:43:36.651 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:43:36.651 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:43:36.651 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:43:36.651 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:43:36.651 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:43:36.651 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:43:36.651 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:43:36.651 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:43:36.651 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:43:36.651 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:43:36.651 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:43:36.652 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:43:36.652 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:43:36.652 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:43:36.652 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:43:36.652 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:43:36.652 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:43:36.652 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:43:36.652 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:43:36.652 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:43:36.652 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:43:36.652 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:43:36.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.652 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:43:36.653 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:43:36.653 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:43:36.653 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:43:36.653 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:43:36.653 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:43:36.653 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:43:36.653 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:43:36.653 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:43:36.653 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:43:36.653 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:43:36.653 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:43:36.653 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:43:36.653 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:43:36.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.654 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:43:36.654 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:43:36.654 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.75, 2024-09-18T07:43:36.654 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.75, 2024-09-18T07:43:36.654 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:43:36.654 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.75, 2024-09-18T07:43:36.654 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.75, 2024-09-18T07:43:36.654 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:43:36.655 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:43:36.655 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:43:36.655 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.655 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 6, 2024-09-18T07:43:36.655 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.meta", 2024-09-18T07:43:36.655 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:31:00.390499+0000", 2024-09-18T07:43:36.655 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:43:36.655 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:43:36.655 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:43:36.655 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:43:36.655 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:43:36.656 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:43:36.656 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:43:36.656 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:43:36.656 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:43:36.656 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:43:36.656 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:43:36.656 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:43:36.656 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:43:36.656 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:43:36.656 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:43:36.656 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:43:36.656 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:43:36.657 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:43:36.657 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:43:36.657 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:43:36.657 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:43:36.657 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:43:36.657 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:43:36.657 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:43:36.657 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:43:36.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.657 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "68", 2024-09-18T07:43:36.657 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:43:36.657 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:43:36.657 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:43:36.657 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:43:36.658 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:43:36.658 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:43:36.658 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:43:36.658 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:43:36.658 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:43:36.658 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:43:36.658 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:43:36.658 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:43:36.658 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:43:36.658 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:43:36.658 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:43:36.658 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:43:36.658 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:43:36.658 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:43:36.659 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:43:36.659 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:43:36.659 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:43:36.659 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:43:36.659 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:43:36.659 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:43:36.659 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:43:36.659 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:43:36.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.659 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:43:36.659 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:43:36.659 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:43:36.659 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:43:36.659 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:43:36.660 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:43:36.660 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:43:36.660 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:43:36.660 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:43:36.660 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:43:36.660 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:43:36.660 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:43:36.660 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_bias": 4 2024-09-18T07:43:36.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.660 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:43:36.660 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:43:36.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.661 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:43:36.661 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:43:36.661 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.5299999713897705, 2024-09-18T07:43:36.661 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.5299999713897705, 2024-09-18T07:43:36.661 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:43:36.661 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-18T07:43:36.661 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-18T07:43:36.661 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:43:36.661 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:43:36.661 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:43:36.661 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.661 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 10, 2024-09-18T07:43:36.662 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "unique_pool_3", 2024-09-18T07:43:36.662 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:43:00.547513+0000", 2024-09-18T07:43:36.662 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:43:36.662 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:43:36.662 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:43:36.662 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:43:36.662 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:43:36.662 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:43:36.662 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:43:36.662 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:43:36.662 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:43:36.662 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:43:36.662 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:43:36.663 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:43:36.663 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:43:36.663 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 16, 2024-09-18T07:43:36.663 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 16, 2024-09-18T07:43:36.663 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 16, 2024-09-18T07:43:36.663 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 16, 2024-09-18T07:43:36.663 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 16, 2024-09-18T07:43:36.663 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:43:36.663 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:43:36.663 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:43:36.663 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:43:36.663 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:43:36.663 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:43:36.664 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:43:36.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.664 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "367", 2024-09-18T07:43:36.664 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:43:36.664 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:43:36.664 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:43:36.664 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:43:36.664 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:43:36.664 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:43:36.664 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:43:36.664 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:43:36.664 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:43:36.664 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:43:36.665 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:43:36.665 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:43:36.665 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:43:36.665 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:43:36.665 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:43:36.665 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:43:36.665 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:43:36.665 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:43:36.665 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:43:36.665 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:43:36.665 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:43:36.665 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:43:36.665 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:43:36.666 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:43:36.666 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:43:36.666 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:43:36.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.666 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:43:36.666 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:43:36.666 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:43:36.666 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:43:36.666 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:43:36.666 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:43:36.666 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:43:36.666 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:43:36.666 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:43:36.667 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:43:36.667 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:43:36.667 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:43:36.667 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:43:36.667 INFO:teuthology.orchestra.run.smithi137.stdout: "rados": {} 2024-09-18T07:43:36.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.667 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:43:36.667 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:43:36.667 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 3.059999942779541, 2024-09-18T07:43:36.667 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 3.059999942779541, 2024-09-18T07:43:36.667 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:43:36.667 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 3.059999942779541, 2024-09-18T07:43:36.667 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 3.059999942779541, 2024-09-18T07:43:36.667 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:43:36.668 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:43:36.668 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:43:36.668 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.668 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.668 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:43:36.668 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-18T07:43:36.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.668 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:43:36.668 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "268cd580-c931-4992-bb93-d6f63c2bfe88", 2024-09-18T07:43:36.668 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:43:36.668 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:43:36.668 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:43:36.668 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:43:36.669 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:43:36.669 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:43:36.669 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 8, 2024-09-18T07:43:36.669 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 364, 2024-09-18T07:43:36.669 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:43:36.669 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:43:36.669 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:43:36.669 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.669 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.669 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6801", 2024-09-18T07:43:36.669 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:43:36.669 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.670 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.670 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:43:36.670 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.670 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.670 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6802", 2024-09-18T07:43:36.670 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:43:36.670 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.670 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.670 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:43:36.670 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.670 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.671 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6804", 2024-09-18T07:43:36.671 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:43:36.671 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.671 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.671 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:43:36.671 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.671 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.671 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6803", 2024-09-18T07:43:36.671 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:43:36.671 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.671 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.672 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6801/2811444477", 2024-09-18T07:43:36.672 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6802/2811444477", 2024-09-18T07:43:36.672 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6804/2811444477", 2024-09-18T07:43:36.672 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6803/2811444477", 2024-09-18T07:43:36.672 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:43:36.672 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:43:36.672 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:43:36.672 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.672 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:43:36.672 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575", 2024-09-18T07:43:36.672 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:43:36.673 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:43:36.673 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:43:36.673 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 0, 2024-09-18T07:43:36.673 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:43:36.673 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:43:36.673 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 13, 2024-09-18T07:43:36.673 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 359, 2024-09-18T07:43:36.673 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:43:36.673 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:43:36.673 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:43:36.673 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.673 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.673 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6805", 2024-09-18T07:43:36.674 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:43:36.674 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.674 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.674 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:43:36.674 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.674 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.674 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6806", 2024-09-18T07:43:36.674 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:43:36.674 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.674 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.674 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:43:36.674 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.675 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.675 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6808", 2024-09-18T07:43:36.675 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:43:36.675 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.675 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.675 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:43:36.675 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.675 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.675 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6807", 2024-09-18T07:43:36.675 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:43:36.675 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.676 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.676 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6805/874102487", 2024-09-18T07:43:36.676 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6806/874102487", 2024-09-18T07:43:36.676 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6808/874102487", 2024-09-18T07:43:36.676 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6807/874102487", 2024-09-18T07:43:36.676 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:43:36.676 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:43:36.676 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:43:36.676 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.676 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:43:36.676 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "480ad6c5-0e0b-4405-bbe2-381682d00bae", 2024-09-18T07:43:36.677 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:43:36.677 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:43:36.677 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:43:36.677 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:43:36.677 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:43:36.677 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:43:36.677 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 18, 2024-09-18T07:43:36.677 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 364, 2024-09-18T07:43:36.677 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:43:36.677 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:43:36.677 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:43:36.677 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.678 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.678 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6809", 2024-09-18T07:43:36.678 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:43:36.678 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.678 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.678 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:43:36.678 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.678 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.678 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6810", 2024-09-18T07:43:36.678 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:43:36.678 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.678 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.679 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:43:36.679 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.679 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.679 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6812", 2024-09-18T07:43:36.679 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:43:36.679 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.679 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.679 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:43:36.679 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.680 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.680 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6811", 2024-09-18T07:43:36.680 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:43:36.680 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.680 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.680 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6809/1754684686", 2024-09-18T07:43:36.680 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6810/1754684686", 2024-09-18T07:43:36.680 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6812/1754684686", 2024-09-18T07:43:36.680 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6811/1754684686", 2024-09-18T07:43:36.680 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:43:36.680 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:43:36.680 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:43:36.681 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.681 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:43:36.681 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14", 2024-09-18T07:43:36.681 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:43:36.681 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:43:36.681 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:43:36.681 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 0.759490966796875, 2024-09-18T07:43:36.681 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:43:36.681 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:43:36.681 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 26, 2024-09-18T07:43:36.681 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 359, 2024-09-18T07:43:36.682 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:43:36.682 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:43:36.682 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:43:36.682 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.682 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.682 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6813", 2024-09-18T07:43:36.682 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:43:36.682 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.682 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.682 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:43:36.682 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.683 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.683 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6814", 2024-09-18T07:43:36.683 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:43:36.683 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.683 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.683 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:43:36.683 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.683 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.683 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6816", 2024-09-18T07:43:36.683 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:43:36.683 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.683 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.684 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:43:36.684 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.684 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.684 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6815", 2024-09-18T07:43:36.684 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:43:36.684 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.684 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.684 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6813/116058392", 2024-09-18T07:43:36.684 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6814/116058392", 2024-09-18T07:43:36.684 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6816/116058392", 2024-09-18T07:43:36.684 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6815/116058392", 2024-09-18T07:43:36.684 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:43:36.685 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:43:36.685 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:43:36.685 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.685 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:43:36.685 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de", 2024-09-18T07:43:36.685 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 0, 2024-09-18T07:43:36.685 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:43:36.685 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:43:36.685 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:43:36.685 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:43:36.685 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:43:36.686 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 32, 2024-09-18T07:43:36.686 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 289, 2024-09-18T07:43:36.686 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 359, 2024-09-18T07:43:36.686 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:43:36.686 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:43:36.686 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.686 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.686 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6800", 2024-09-18T07:43:36.686 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:43:36.686 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.686 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.687 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:43:36.687 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.687 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.687 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6801", 2024-09-18T07:43:36.687 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:43:36.687 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.687 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.687 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:43:36.687 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.687 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.687 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6803", 2024-09-18T07:43:36.688 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:43:36.688 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.688 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.688 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:43:36.688 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.688 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.688 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6802", 2024-09-18T07:43:36.688 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:43:36.688 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.688 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.688 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6800/3654297954", 2024-09-18T07:43:36.689 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6801/3654297954", 2024-09-18T07:43:36.689 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6803/3654297954", 2024-09-18T07:43:36.689 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6802/3654297954", 2024-09-18T07:43:36.689 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:43:36.689 INFO:teuthology.orchestra.run.smithi137.stdout: "exists" 2024-09-18T07:43:36.689 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.689 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:43:36.689 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c", 2024-09-18T07:43:36.689 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:43:36.689 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:43:36.689 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:43:36.690 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:43:36.690 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:43:36.690 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:43:36.690 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 39, 2024-09-18T07:43:36.690 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 364, 2024-09-18T07:43:36.690 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:43:36.690 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:43:36.690 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:43:36.690 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.690 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.690 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6804", 2024-09-18T07:43:36.691 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:36.692 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:43:36.692 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.692 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.692 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:43:36.692 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.692 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.692 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6805", 2024-09-18T07:43:36.692 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:43:36.693 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.693 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.693 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:43:36.693 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.693 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.693 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6807", 2024-09-18T07:43:36.693 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:43:36.693 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.693 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.693 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:43:36.693 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.694 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.694 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6806", 2024-09-18T07:43:36.694 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:43:36.694 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.694 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.694 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6804/837994222", 2024-09-18T07:43:36.694 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6805/837994222", 2024-09-18T07:43:36.694 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6807/837994222", 2024-09-18T07:43:36.694 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6806/837994222", 2024-09-18T07:43:36.694 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:43:36.694 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:43:36.694 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:43:36.695 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.695 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:43:36.695 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6", 2024-09-18T07:43:36.695 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:43:36.695 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:43:36.695 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:43:36.695 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:43:36.695 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:43:36.695 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:43:36.695 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 47, 2024-09-18T07:43:36.695 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 364, 2024-09-18T07:43:36.695 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:43:36.696 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:43:36.696 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:43:36.696 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.696 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.696 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6808", 2024-09-18T07:43:36.696 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:43:36.696 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.696 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.696 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:43:36.696 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.697 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.697 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6809", 2024-09-18T07:43:36.697 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:43:36.697 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.697 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.697 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:43:36.697 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.697 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.697 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6811", 2024-09-18T07:43:36.698 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:43:36.698 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.698 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.698 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:43:36.698 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.698 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.698 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6810", 2024-09-18T07:43:36.698 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:43:36.698 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.698 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.699 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6808/1494175466", 2024-09-18T07:43:36.699 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6809/1494175466", 2024-09-18T07:43:36.699 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6811/1494175466", 2024-09-18T07:43:36.699 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6810/1494175466", 2024-09-18T07:43:36.699 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:43:36.699 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:43:36.699 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:43:36.699 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.699 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:43:36.699 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a", 2024-09-18T07:43:36.700 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:43:36.700 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:43:36.700 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:43:36.700 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:43:36.700 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:43:36.700 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:43:36.700 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 55, 2024-09-18T07:43:36.700 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 364, 2024-09-18T07:43:36.700 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:43:36.700 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:43:36.700 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:43:36.700 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.701 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.701 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6812", 2024-09-18T07:43:36.701 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:43:36.701 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.701 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.701 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:43:36.701 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.701 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.701 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6813", 2024-09-18T07:43:36.701 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:43:36.701 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.702 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.702 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:43:36.702 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.702 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.702 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6815", 2024-09-18T07:43:36.702 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:43:36.702 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.702 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.702 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:43:36.702 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:36.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.703 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:36.703 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6814", 2024-09-18T07:43:36.703 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:43:36.703 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.703 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.703 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6812/2670809315", 2024-09-18T07:43:36.703 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6813/2670809315", 2024-09-18T07:43:36.703 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6815/2670809315", 2024-09-18T07:43:36.703 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6814/2670809315", 2024-09-18T07:43:36.703 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:43:36.703 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:43:36.703 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:43:36.703 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.704 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.704 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:43:36.704 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_xinfo": [ 2024-09-18T07:43:36.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.704 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:43:36.704 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:43:36.704 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:43:36.704 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:43:36.704 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:43:36.704 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:43:36.704 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:43:01.127218+0000", 2024-09-18T07:43:36.704 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:43:36.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.704 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:43:36.705 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:43:36.705 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:43:36.705 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:43:36.705 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:43:36.705 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:43:36.705 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:42:31.595553+0000", 2024-09-18T07:43:36.705 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:43:36.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.705 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:43:36.705 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:43:36.705 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:43:36.705 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:43:36.706 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:43:36.706 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:43:36.706 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:43:24.037996+0000", 2024-09-18T07:43:36.706 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:43:36.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.706 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:43:36.706 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:43:36.706 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:43:36.706 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:43:36.706 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:43:36.706 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:43:36.706 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:43:16.491536+0000", 2024-09-18T07:43:36.706 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:43:36.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.707 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:43:36.707 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:43:36.707 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:43:36.707 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:43:36.707 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:43:36.707 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:43:36.707 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:41:47.479848+0000", 2024-09-18T07:43:36.707 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 358 2024-09-18T07:43:36.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.707 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:43:36.707 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:43:36.707 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:43:36.708 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:43:36.708 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:43:36.708 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:43:36.708 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:42:35.305044+0000", 2024-09-18T07:43:36.708 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:43:36.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.708 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:43:36.708 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:43:36.708 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:43:36.708 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:43:36.708 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:43:36.708 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:43:36.708 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:42:34.238640+0000", 2024-09-18T07:43:36.708 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:43:36.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.709 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:43:36.709 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:43:36.709 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:43:36.709 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:43:36.709 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:43:36.709 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:43:36.709 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:42:59.881262+0000", 2024-09-18T07:43:36.709 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:43:36.709 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.709 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:43:36.709 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap": [], 2024-09-18T07:43:36.709 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_items": [ 2024-09-18T07:43:36.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.710 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.6", 2024-09-18T07:43:36.710 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:36.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.710 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:43:36.710 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:43:36.710 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.710 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.710 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.b", 2024-09-18T07:43:36.710 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:36.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.710 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:43:36.710 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:43:36.710 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.711 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.711 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.e", 2024-09-18T07:43:36.711 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:36.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.711 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:43:36.711 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:43:36.711 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.711 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.711 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.10", 2024-09-18T07:43:36.711 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:36.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.712 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:43:36.712 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:43:36.712 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.712 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.712 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.12", 2024-09-18T07:43:36.712 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:36.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.712 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:43:36.712 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:43:36.712 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.712 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.713 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.13", 2024-09-18T07:43:36.713 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:36.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.713 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 6, 2024-09-18T07:43:36.713 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 7 2024-09-18T07:43:36.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.713 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:43:36.713 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:43:36.713 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.713 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.713 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.18", 2024-09-18T07:43:36.714 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:36.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.714 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:43:36.714 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:43:36.714 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.714 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.714 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.1b", 2024-09-18T07:43:36.714 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:36.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.714 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:43:36.714 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:43:36.714 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.714 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.715 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.1c", 2024-09-18T07:43:36.715 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:36.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.715 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:43:36.715 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:43:36.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.715 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:43:36.715 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 7 2024-09-18T07:43:36.715 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.715 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.716 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.1d", 2024-09-18T07:43:36.716 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:36.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.716 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:43:36.716 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:43:36.716 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.716 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.716 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.b", 2024-09-18T07:43:36.716 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:36.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.716 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 0, 2024-09-18T07:43:36.716 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 3 2024-09-18T07:43:36.717 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.717 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.717 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.4", 2024-09-18T07:43:36.717 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:36.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.717 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:43:36.717 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:43:36.717 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.717 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.717 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.1b", 2024-09-18T07:43:36.718 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:36.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:36.718 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:43:36.718 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:43:36.718 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.718 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:36.718 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.718 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:43:36.718 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_primaries": [], 2024-09-18T07:43:36.718 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_temp": [], 2024-09-18T07:43:36.718 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_temp": [], 2024-09-18T07:43:36.718 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklist": { 2024-09-18T07:43:36.718 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/2096039404": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:43:36.718 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/3636177939": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:43:36.719 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/334967443": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:43:36.719 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2101286111": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:43:36.719 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3387637412": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:43:36.719 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762989503": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:43:36.719 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3490817671": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:43:36.719 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1482147392": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:43:36.719 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1821482181": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:43:36.719 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/4028304023": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:43:36.719 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/621434521": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:43:36.719 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3160704935": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:43:36.719 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2357293499": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:43:36.719 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762352860": "2024-09-19T07:31:25.389124+0000" 2024-09-18T07:43:36.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.720 INFO:teuthology.orchestra.run.smithi137.stdout: "range_blocklist": {}, 2024-09-18T07:43:36.720 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profiles": { 2024-09-18T07:43:36.720 INFO:teuthology.orchestra.run.smithi137.stdout: "default": { 2024-09-18T07:43:36.720 INFO:teuthology.orchestra.run.smithi137.stdout: "crush-failure-domain": "osd", 2024-09-18T07:43:36.720 INFO:teuthology.orchestra.run.smithi137.stdout: "k": "2", 2024-09-18T07:43:36.720 INFO:teuthology.orchestra.run.smithi137.stdout: "m": "1", 2024-09-18T07:43:36.720 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin": "jerasure", 2024-09-18T07:43:36.720 INFO:teuthology.orchestra.run.smithi137.stdout: "technique": "reed_sol_van" 2024-09-18T07:43:36.720 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:36.720 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps_queue": [], 2024-09-18T07:43:36.720 INFO:teuthology.orchestra.run.smithi137.stdout: "new_removed_snaps": [], 2024-09-18T07:43:36.720 INFO:teuthology.orchestra.run.smithi137.stdout: "new_purged_snaps": [], 2024-09-18T07:43:36.720 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_node_flags": {}, 2024-09-18T07:43:36.720 INFO:teuthology.orchestra.run.smithi137.stdout: "device_class_flags": {}, 2024-09-18T07:43:36.721 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode": { 2024-09-18T07:43:36.721 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_enabled": false, 2024-09-18T07:43:36.721 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_bucket_count": 0, 2024-09-18T07:43:36.721 INFO:teuthology.orchestra.run.smithi137.stdout: "degraded_stretch_mode": 0, 2024-09-18T07:43:36.721 INFO:teuthology.orchestra.run.smithi137.stdout: "recovering_stretch_mode": 0, 2024-09-18T07:43:36.721 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_bucket": 0 2024-09-18T07:43:36.721 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:36.721 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:43:36.722 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:36 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/544578348' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:36.722 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:36 smithi203 ceph-mon[32237]: from='client.36050 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:36.723 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:36.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:36.824 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:36.853 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:36.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:36.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:36.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:36.994 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:36.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:36.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:37.062 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 27 16 58837 58821 136.113 99.375 0.00465782 0.00734139 2024-09-18T07:43:37.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:37.130 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:37.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:37.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:37.231 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:37.258 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:37.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:37.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:37.359 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:37.387 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:37.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:37.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:37.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:43:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:43:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:43:37.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:37 smithi137 ceph-mon[29408]: pgmap v667: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.88k op/s; 25465/152742 objects degraded (16.672%) 2024-09-18T07:43:37.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:37 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:43:37.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:37 smithi137 ceph-mon[29408]: from='client.36050 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:43:37.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:37 smithi137 ceph-mon[29408]: osdmap e373: 8 total, 7 up, 8 in 2024-09-18T07:43:37.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:37 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1054343818' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:43:37.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:37 smithi137 ceph-mon[39175]: pgmap v667: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.88k op/s; 25465/152742 objects degraded (16.672%) 2024-09-18T07:43:37.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:37 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:43:37.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:37 smithi137 ceph-mon[39175]: from='client.36050 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:43:37.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:37 smithi137 ceph-mon[39175]: osdmap e373: 8 total, 7 up, 8 in 2024-09-18T07:43:37.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:37 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1054343818' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:43:37.489 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:37.522 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:37.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:37.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:37.598 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:37 smithi203 ceph-mon[32237]: pgmap v667: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.88k op/s; 25465/152742 objects degraded (16.672%) 2024-09-18T07:43:37.598 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:37 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:43:37.598 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:37 smithi203 ceph-mon[32237]: from='client.36050 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:43:37.598 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:37 smithi203 ceph-mon[32237]: osdmap e373: 8 total, 7 up, 8 in 2024-09-18T07:43:37.598 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:37 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1054343818' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:43:37.625 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:37.659 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:37.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:37.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:37.761 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:37.790 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:37.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:37.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:37.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:37.932 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:37.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:37.960 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:43:37 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:43:38.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:38.062 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 28 16 60514 60498 134.995 104.812 0.00924931 0.00740279 2024-09-18T07:43:38.065 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:38.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:38.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:38.095 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:38.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:38.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:38.209 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:38.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:38.310 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:38.340 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:38.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:38.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:38.442 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:38.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:38.469 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:38.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:38.570 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:38.598 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:38.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:38.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:38.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:38 smithi137 ceph-mon[29408]: 5.12 scrub starts 2024-09-18T07:43:38.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:38 smithi137 ceph-mon[29408]: 5.12 scrub ok 2024-09-18T07:43:38.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:38 smithi137 ceph-mon[29408]: 6.18 scrub starts 2024-09-18T07:43:38.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:38 smithi137 ceph-mon[29408]: 6.18 scrub ok 2024-09-18T07:43:38.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:38 smithi137 ceph-mon[29408]: 4.11 scrub starts 2024-09-18T07:43:38.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:38 smithi137 ceph-mon[29408]: 4.11 scrub ok 2024-09-18T07:43:38.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:38 smithi137 ceph-mon[39175]: 5.12 scrub starts 2024-09-18T07:43:38.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:38 smithi137 ceph-mon[39175]: 5.12 scrub ok 2024-09-18T07:43:38.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:38 smithi137 ceph-mon[39175]: 6.18 scrub starts 2024-09-18T07:43:38.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:38 smithi137 ceph-mon[39175]: 6.18 scrub ok 2024-09-18T07:43:38.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:38 smithi137 ceph-mon[39175]: 4.11 scrub starts 2024-09-18T07:43:38.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:38 smithi137 ceph-mon[39175]: 4.11 scrub ok 2024-09-18T07:43:38.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:38.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:38 smithi203 ceph-mon[32237]: 5.12 scrub starts 2024-09-18T07:43:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:38 smithi203 ceph-mon[32237]: 5.12 scrub ok 2024-09-18T07:43:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:38 smithi203 ceph-mon[32237]: 6.18 scrub starts 2024-09-18T07:43:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:38 smithi203 ceph-mon[32237]: 6.18 scrub ok 2024-09-18T07:43:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:38 smithi203 ceph-mon[32237]: 4.11 scrub starts 2024-09-18T07:43:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:38 smithi203 ceph-mon[32237]: 4.11 scrub ok 2024-09-18T07:43:38.735 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:38.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:38.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:38.837 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:38.864 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:38.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:38.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:38.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:38.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:38.998 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:38.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:39.062 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 29 16 62064 62048 133.681 96.875 0.00778655 0.00747562 2024-09-18T07:43:39.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:39.135 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:39.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:39.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:39.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:39.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:43:39.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:39.270 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:39.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:39.371 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:39.393 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:39.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:39.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:39 smithi137 ceph-mon[29408]: 5.16 scrub starts 2024-09-18T07:43:39.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:39 smithi137 ceph-mon[29408]: 5.16 scrub ok 2024-09-18T07:43:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:39 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:39 smithi137 ceph-mon[29408]: pgmap v669: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 28443/172200 objects degraded (16.517%) 2024-09-18T07:43:39.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:39 smithi137 ceph-mon[39175]: 5.16 scrub starts 2024-09-18T07:43:39.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:39 smithi137 ceph-mon[39175]: 5.16 scrub ok 2024-09-18T07:43:39.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:39 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:39.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:39 smithi137 ceph-mon[39175]: pgmap v669: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 28443/172200 objects degraded (16.517%) 2024-09-18T07:43:39.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:39.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:39.537 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:39.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:39.638 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:39.667 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:39.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:39.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:39 smithi203 ceph-mon[32237]: 5.16 scrub starts 2024-09-18T07:43:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:39 smithi203 ceph-mon[32237]: 5.16 scrub ok 2024-09-18T07:43:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:39 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:39 smithi203 ceph-mon[32237]: pgmap v669: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 28443/172200 objects degraded (16.517%) 2024-09-18T07:43:39.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:39.792 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:39.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:39.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:39.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:39.930 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:39.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:39.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:40.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:40.062 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 30 16 63408 63392 132.025 84 0.00603521 0.0075693 2024-09-18T07:43:40.065 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:40.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:40.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:40.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:40.202 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:40.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:40.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:40.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:40.337 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:40.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:40.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:40.440 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:40.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:40.462 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:40.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:40.564 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:40.599 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:40.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:40.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:40 smithi137 ceph-mon[29408]: 6.8 scrub starts 2024-09-18T07:43:40.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:40 smithi137 ceph-mon[29408]: 6.8 scrub ok 2024-09-18T07:43:40.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:40 smithi137 ceph-mon[29408]: pgmap v670: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 106 MiB/s wr, 1.70k op/s; 29448/176748 objects degraded (16.661%) 2024-09-18T07:43:40.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:40 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 28443/172200 objects degraded (16.517%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:40.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:40 smithi137 ceph-mon[39175]: 6.8 scrub starts 2024-09-18T07:43:40.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:40 smithi137 ceph-mon[39175]: 6.8 scrub ok 2024-09-18T07:43:40.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:40 smithi137 ceph-mon[39175]: pgmap v670: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 106 MiB/s wr, 1.70k op/s; 29448/176748 objects degraded (16.661%) 2024-09-18T07:43:40.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:40 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 28443/172200 objects degraded (16.517%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:40.701 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:40.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:40 smithi203 ceph-mon[32237]: 6.8 scrub starts 2024-09-18T07:43:40.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:40 smithi203 ceph-mon[32237]: 6.8 scrub ok 2024-09-18T07:43:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:40 smithi203 ceph-mon[32237]: pgmap v670: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 106 MiB/s wr, 1.70k op/s; 29448/176748 objects degraded (16.661%) 2024-09-18T07:43:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:40 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 28443/172200 objects degraded (16.517%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:40.736 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:40.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:40.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:40.838 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:40.876 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:40.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:40.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:40.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:41.013 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:41.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:41.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:41.062 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 31 16 64670 64654 130.311 78.875 0.00259947 0.0076614 2024-09-18T07:43:41.116 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:41.148 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:41.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:41.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:41.250 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:41.282 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:41.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:41.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:41.384 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:41.413 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:41.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:41.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:41.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:41 smithi203 ceph-mon[32237]: 5.6 scrub starts 2024-09-18T07:43:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:41 smithi203 ceph-mon[32237]: 5.6 scrub ok 2024-09-18T07:43:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:41 smithi203 ceph-mon[32237]: 4.7 scrub starts 2024-09-18T07:43:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:41 smithi203 ceph-mon[32237]: 4.7 scrub ok 2024-09-18T07:43:41.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:41.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:41.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:41.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:41.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:41 smithi137 ceph-mon[29408]: 5.6 scrub starts 2024-09-18T07:43:41.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:41 smithi137 ceph-mon[29408]: 5.6 scrub ok 2024-09-18T07:43:41.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:41 smithi137 ceph-mon[29408]: 4.7 scrub starts 2024-09-18T07:43:41.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:41 smithi137 ceph-mon[29408]: 4.7 scrub ok 2024-09-18T07:43:41.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:41 smithi137 ceph-mon[39175]: 5.6 scrub starts 2024-09-18T07:43:41.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:41 smithi137 ceph-mon[39175]: 5.6 scrub ok 2024-09-18T07:43:41.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:41 smithi137 ceph-mon[39175]: 4.7 scrub starts 2024-09-18T07:43:41.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:41 smithi137 ceph-mon[39175]: 4.7 scrub ok 2024-09-18T07:43:41.677 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:41.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:41.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:41.779 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:41.811 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:41.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:41.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:41.949 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:41.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:41.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:41.999 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:42.027 INFO:tasks.thrashosds.thrasher:j is {'epoch': 373, 'fsid': 'f7ef099c-758e-11ef-bceb-c7b262605968', 'created': '2024-09-18T07:25:19.662367+0000', 'modified': '2024-09-18T07:43:36.218416+0000', 'last_up_change': '2024-09-18T07:42:53.723776+0000', 'last_in_change': '2024-09-18T07:37:42.022730+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T07:28:20.899655+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '274', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '272', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 1, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-18T07:30:53.715373+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 1, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-18T07:30:54.672127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-18T07:30:56.351343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-18T07:30:58.322401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-18T07:31:00.390499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-18T07:43:00.547513+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '367', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.059999942779541, 'score_stable': 3.059999942779541, 'optimal_score': 1, 'raw_score_acting': 3.059999942779541, 'raw_score_stable': 3.059999942779541, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}], 'osds': [{'osd': 0, 'uuid': '268cd580-c931-4992-bb93-d6f63c2bfe88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6801', 'nonce': 2811444477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6802', 'nonce': 2811444477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6804', 'nonce': 2811444477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6803', 'nonce': 2811444477}]}, 'public_addr': '172.21.15.137:6801/2811444477', 'cluster_addr': '172.21.15.137:6802/2811444477', 'heartbeat_back_addr': '172.21.15.137:6804/2811444477', 'heartbeat_front_addr': '172.21.15.137:6803/2811444477', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b06907bc-b499-46c0-8667-c5bdc0d85575', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 359, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6805', 'nonce': 874102487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6806', 'nonce': 874102487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6808', 'nonce': 874102487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6807', 'nonce': 874102487}]}, 'public_addr': '172.21.15.137:6805/874102487', 'cluster_addr': '172.21.15.137:6806/874102487', 'heartbeat_back_addr': '172.21.15.137:6808/874102487', 'heartbeat_front_addr': '172.21.15.137:6807/874102487', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '480ad6c5-0e0b-4405-bbe2-381682d00bae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6809', 'nonce': 1754684686}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6810', 'nonce': 1754684686}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6812', 'nonce': 1754684686}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6811', 'nonce': 1754684686}]}, 'public_addr': '172.21.15.137:6809/1754684686', 'cluster_addr': '172.21.15.137:6810/1754684686', 'heartbeat_back_addr': '172.21.15.137:6812/1754684686', 'heartbeat_front_addr': '172.21.15.137:6811/1754684686', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '178be3e0-f07b-4856-807b-2a1181e48a14', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.759490966796875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 359, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6813', 'nonce': 116058392}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6814', 'nonce': 116058392}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6816', 'nonce': 116058392}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6815', 'nonce': 116058392}]}, 'public_addr': '172.21.15.137:6813/116058392', 'cluster_addr': '172.21.15.137:6814/116058392', 'heartbeat_back_addr': '172.21.15.137:6816/116058392', 'heartbeat_front_addr': '172.21.15.137:6815/116058392', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7547102c-951a-4077-a094-fb8a65bfe6de', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 289, 'down_at': 359, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6800', 'nonce': 3654297954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6801', 'nonce': 3654297954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6803', 'nonce': 3654297954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6802', 'nonce': 3654297954}]}, 'public_addr': '172.21.15.203:6800/3654297954', 'cluster_addr': '172.21.15.203:6801/3654297954', 'heartbeat_back_addr': '172.21.15.203:6803/3654297954', 'heartbeat_front_addr': '172.21.15.203:6802/3654297954', 'state': ['exists']}, {'osd': 5, 'uuid': '49c17901-8102-48a4-9348-fa1bdb63032c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6804', 'nonce': 837994222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6805', 'nonce': 837994222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6807', 'nonce': 837994222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6806', 'nonce': 837994222}]}, 'public_addr': '172.21.15.203:6804/837994222', 'cluster_addr': '172.21.15.203:6805/837994222', 'heartbeat_back_addr': '172.21.15.203:6807/837994222', 'heartbeat_front_addr': '172.21.15.203:6806/837994222', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '75f0ead4-e981-4a15-b1c1-d83dce871ab6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6808', 'nonce': 1494175466}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6809', 'nonce': 1494175466}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6811', 'nonce': 1494175466}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6810', 'nonce': 1494175466}]}, 'public_addr': '172.21.15.203:6808/1494175466', 'cluster_addr': '172.21.15.203:6809/1494175466', 'heartbeat_back_addr': '172.21.15.203:6811/1494175466', 'heartbeat_front_addr': '172.21.15.203:6810/1494175466', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c0dbf299-6798-4b99-9a85-62b123971f2a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 55, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6812', 'nonce': 2670809315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6813', 'nonce': 2670809315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6815', 'nonce': 2670809315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6814', 'nonce': 2670809315}]}, 'public_addr': '172.21.15.203:6812/2670809315', 'cluster_addr': '172.21.15.203:6813/2670809315', 'heartbeat_back_addr': '172.21.15.203:6815/2670809315', 'heartbeat_front_addr': '172.21.15.203:6814/2670809315', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:43:01.127218+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:42:31.595553+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:43:24.037996+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:43:16.491536+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:41:47.479848+0000', 'dead_epoch': 358}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:42:35.305044+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:42:34.238640+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:42:59.881262+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.13', 'mappings': [{'from': 6, 'to': 7}, {'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1b', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '4.1c', 'mappings': [{'from': 3, 'to': 0}, {'from': 1, 'to': 7}]}, {'pgid': '4.1d', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '5.b', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '6.4', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '6.1b', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.137:6800/2096039404': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:6800/3636177939': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/334967443': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/2101286111': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/3387637412': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1762989503': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3490817671': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:6800/1482147392': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1821482181': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/4028304023': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/621434521': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3160704935': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/2357293499': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/1762352860': '2024-09-19T07:31:25.389124+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T07:43:42.027 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 6.4 2024-09-18T07:43:42.027 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd rm-pg-upmap-items 6.4 2024-09-18T07:43:42.051 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:42.062 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 32 16 66064 66048 128.96 87.125 0.00790073 0.00774911 2024-09-18T07:43:42.079 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:42.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:42.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:42.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:42.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:42.239 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:42.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:42.340 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:42.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:42.367 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:42.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:42.468 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:42.497 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:42.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:42.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:42.599 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:42.654 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:42.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:42.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:42.794 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:42.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:42.896 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:42.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:42.918 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:42.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:43.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:43.040 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:43.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:43.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:43.062 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 33 16 67180 67164 127.166 69.75 0.00278508 0.0078136 2024-09-18T07:43:43.142 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:43.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:43.168 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:43.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:42 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:43:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:42 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:43:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:42 smithi137 ceph-mon[29408]: pgmap v671: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 105 MiB/s wr, 1.68k op/s; 30359/179481 objects degraded (16.915%) 2024-09-18T07:43:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:42 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:43:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:42 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:43:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:42 smithi137 ceph-mon[29408]: osdmap e374: 8 total, 7 up, 8 in 2024-09-18T07:43:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:42 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:43:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:42 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:43:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:42 smithi137 ceph-mon[39175]: pgmap v671: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 105 MiB/s wr, 1.68k op/s; 30359/179481 objects degraded (16.915%) 2024-09-18T07:43:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:42 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:43:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:42 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:43:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:42 smithi137 ceph-mon[39175]: osdmap e374: 8 total, 7 up, 8 in 2024-09-18T07:43:43.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:42 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:43:43.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:42 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:43:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:42 smithi203 ceph-mon[32237]: pgmap v671: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 105 MiB/s wr, 1.68k op/s; 30359/179481 objects degraded (16.915%) 2024-09-18T07:43:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:42 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:43:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:42 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:43:43.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:42 smithi203 ceph-mon[32237]: osdmap e374: 8 total, 7 up, 8 in 2024-09-18T07:43:43.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:43.311 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:43.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:43.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:43.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:43.454 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:43.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:43.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:43.475 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:43:43.475 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:43:43.475 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:43:43.475 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:43:43.475 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:43:43.475 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:43:43.476 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:43:43.476 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:43:43.476 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-18T07:43:43.507 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:43:43.508 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:43:43.540 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:43:43.540 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:43:43.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:43.571 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:43:43.571 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:43:43.580 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:43:43.580 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:43:43.584 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:43.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:43.686 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:43.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:43.724 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:43.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:43.825 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:43.850 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:43.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:43.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:43.952 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:43.981 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:43.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:43.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:44.064 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 34 16 68590 68574 126.013 88.125 0.0150434 0.0079311 2024-09-18T07:43:44.084 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:44.104 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:43 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2919962603' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:44.104 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:43 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2919962603' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:44.113 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:44.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:44.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:44.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:43 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2919962603' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:44.283 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:44.319 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:44.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:44.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:44.421 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:44.448 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:44.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:44.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:44.550 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:44.578 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:44.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:44.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:44.680 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:44.712 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:44.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:44.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:44.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:44.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:44.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:44.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T07:43:44.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:44.982 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:44.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:44.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:45.064 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 35 16 70158 70142 125.213 98 0.00951338 0.00798135 2024-09-18T07:43:45.083 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:45.110 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:45.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:45.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:44 smithi137 ceph-mon[29408]: pgmap v673: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 119 MiB/s wr, 1.91k op/s; 33113/198570 objects degraded (16.676%) 2024-09-18T07:43:45.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:44 smithi137 ceph-mon[39175]: pgmap v673: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 119 MiB/s wr, 1.91k op/s; 33113/198570 objects degraded (16.676%) 2024-09-18T07:43:45.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:44 smithi203 ceph-mon[32237]: pgmap v673: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 119 MiB/s wr, 1.91k op/s; 33113/198570 objects degraded (16.676%) 2024-09-18T07:43:45.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:45.231 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:43:45.272 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:43:45.298 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:45.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:45.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:45.314 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:45.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:45.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:45.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:45.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:45.545 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:45.573 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:45.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:45.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:45.675 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:45.701 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:45.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:45.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:45.804 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:45.838 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:45.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:45.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:45.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:45.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:45.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:45.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:46.064 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 36 16 71496 71480 124.057 83.625 0.00908568 0.00805594 2024-09-18T07:43:46.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:46.098 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:45 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 33113/198570 objects degraded (16.676%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:46.098 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:45 smithi137 ceph-mon[29408]: 6.11 scrub starts 2024-09-18T07:43:46.098 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:45 smithi137 ceph-mon[29408]: 6.11 scrub ok 2024-09-18T07:43:46.099 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:45 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 33113/198570 objects degraded (16.676%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:46.099 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:45 smithi137 ceph-mon[39175]: 6.11 scrub starts 2024-09-18T07:43:46.100 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:45 smithi137 ceph-mon[39175]: 6.11 scrub ok 2024-09-18T07:43:46.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:46.117 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:46.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:46.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:45 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 33113/198570 objects degraded (16.676%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:45 smithi203 ceph-mon[32237]: 6.11 scrub starts 2024-09-18T07:43:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:45 smithi203 ceph-mon[32237]: 6.11 scrub ok 2024-09-18T07:43:46.217 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:46.241 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:46.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:46.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:46.343 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:46.369 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:46.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:46.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:46.471 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:46.492 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:46.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:46.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:46.594 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:46.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:46.621 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:46.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:46.722 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:46.755 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:46.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:46.848 INFO:teuthology.orchestra.run.smithi137.stderr:clear 6.4 pg_upmap_items mapping 2024-09-18T07:43:46.857 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:46.878 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:46.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:46.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:46.980 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:47.017 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:47.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:47.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:47.064 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 37 16 72533 72517 122.456 64.8125 0.0149457 0.00816191 2024-09-18T07:43:47.073 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:43:47.074 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:46 smithi137 ceph-mon[29408]: pgmap v674: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.61k op/s; 33113/198570 objects degraded (16.676%) 2024-09-18T07:43:47.074 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:46 smithi137 ceph-mon[29408]: 4.1e scrub starts 2024-09-18T07:43:47.074 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:46 smithi137 ceph-mon[29408]: 4.1e scrub ok 2024-09-18T07:43:47.074 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:46 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4092538809' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:43:47.074 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:46 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2497275876' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "6.4"} : dispatch 2024-09-18T07:43:47.074 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:46 smithi137 ceph-mon[39175]: pgmap v674: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.61k op/s; 33113/198570 objects degraded (16.676%) 2024-09-18T07:43:47.074 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:46 smithi137 ceph-mon[39175]: 4.1e scrub starts 2024-09-18T07:43:47.074 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:46 smithi137 ceph-mon[39175]: 4.1e scrub ok 2024-09-18T07:43:47.075 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:46 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4092538809' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:43:47.075 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:46 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2497275876' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "6.4"} : dispatch 2024-09-18T07:43:47.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:47.152 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:47.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:47.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:46 smithi203 ceph-mon[32237]: pgmap v674: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.61k op/s; 33113/198570 objects degraded (16.676%) 2024-09-18T07:43:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:46 smithi203 ceph-mon[32237]: 4.1e scrub starts 2024-09-18T07:43:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:46 smithi203 ceph-mon[32237]: 4.1e scrub ok 2024-09-18T07:43:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:46 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4092538809' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:43:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:46 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2497275876' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "6.4"} : dispatch 2024-09-18T07:43:47.255 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:47.273 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:43:47.285 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:47.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:47.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:47.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:43:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:43:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:43:47.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:47.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:47.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:47.520 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:47.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:47.580 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:47.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:47.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:47.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:47.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:47.723 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:47.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:47.824 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:47.851 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:47.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:47.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:47.860 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:43:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:43:47.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:47 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:47.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:47 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4092538809' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:43:47.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:47 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2497275876' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.4"}]': finished 2024-09-18T07:43:47.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:47 smithi137 ceph-mon[29408]: osdmap e375: 8 total, 7 up, 8 in 2024-09-18T07:43:47.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:47 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:47.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:47 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4092538809' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:43:47.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:47 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2497275876' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.4"}]': finished 2024-09-18T07:43:47.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:47 smithi137 ceph-mon[39175]: osdmap e375: 8 total, 7 up, 8 in 2024-09-18T07:43:47.954 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:47.987 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:47.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:47.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:48.064 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 38 15 74016 74001 121.675 92.75 0.011528 0.0082142 2024-09-18T07:43:48.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:48.138 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:48.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:48.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:48.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:47 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:48.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:47 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4092538809' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:43:48.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:47 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2497275876' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.4"}]': finished 2024-09-18T07:43:48.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:47 smithi203 ceph-mon[32237]: osdmap e375: 8 total, 7 up, 8 in 2024-09-18T07:43:48.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:48.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:48.282 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:48.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:48.383 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:48.409 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:48.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:48.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:48.514 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:48.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:48.547 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:48.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:48.648 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:48.675 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:48.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:48.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:48.687 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:43:48.688 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:43:48.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.688 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:74859 10.7 e375/364 10:eb6259a8:::benchmark_data_smithi137_151160_object74858:head v 375'4751, mlcod=375'4749)", 2024-09-18T07:43:48.688 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:48.681955+0000", 2024-09-18T07:43:48.688 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00259086, 2024-09-18T07:43:48.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026197479999999999, 2024-09-18T07:43:48.688 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:48.688 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:48.688 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:43:48.688 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:48.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:48.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.681955+0000", 2024-09-18T07:43:48.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:48.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:48.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.681955+0000", 2024-09-18T07:43:48.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:48.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:48.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.681956+0000", 2024-09-18T07:43:48.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3859999999999999e-06 2024-09-18T07:43:48.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:48.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.682019+0000", 2024-09-18T07:43:48.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2439000000000003e-05 2024-09-18T07:43:48.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:48.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.682032+0000", 2024-09-18T07:43:48.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3633e-05 2024-09-18T07:43:48.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:48.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.682040+0000", 2024-09-18T07:43:48.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8509999999999999e-06 2024-09-18T07:43:48.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:48.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.682107+0000", 2024-09-18T07:43:48.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7428999999999994e-05 2024-09-18T07:43:48.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:48.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.682130+0000", 2024-09-18T07:43:48.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2447e-05 2024-09-18T07:43:48.692 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:48.692 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:48.692 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:48.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.692 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:74862 10.f e375/364 10:f2b22408:::benchmark_data_smithi137_151160_object74861:head v 375'4676, mlcod=375'4673)", 2024-09-18T07:43:48.693 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:48.682896+0000", 2024-09-18T07:43:48.693 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0016498579999999999, 2024-09-18T07:43:48.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00173156, 2024-09-18T07:43:48.693 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:48.693 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:48.693 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:43:48.693 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:48.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:48.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.682896+0000", 2024-09-18T07:43:48.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:48.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:48.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.682896+0000", 2024-09-18T07:43:48.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:48.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:48.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.682897+0000", 2024-09-18T07:43:48.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1689999999999999e-06 2024-09-18T07:43:48.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:48.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.682927+0000", 2024-09-18T07:43:48.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0198000000000001e-05 2024-09-18T07:43:48.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:48.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.682939+0000", 2024-09-18T07:43:48.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.204e-05 2024-09-18T07:43:48.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:48.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.682947+0000", 2024-09-18T07:43:48.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7849999999999992e-06 2024-09-18T07:43:48.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:48.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.683462+0000", 2024-09-18T07:43:48.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000515527 2024-09-18T07:43:48.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:48.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.683504+0000", 2024-09-18T07:43:48.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1671000000000001e-05 2024-09-18T07:43:48.696 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:48.697 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:48.697 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:48.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.697 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:74852 10.f e375/364 10:fa369d1b:::benchmark_data_smithi137_151160_object74851:head v 375'4675, mlcod=375'4671)", 2024-09-18T07:43:48.697 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:48.676028+0000", 2024-09-18T07:43:48.697 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0085174559999999996, 2024-09-18T07:43:48.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0086462320000000002, 2024-09-18T07:43:48.697 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:43:48.697 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:43:48.697 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:43:48.697 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:43:48.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:43:48.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.676028+0000", 2024-09-18T07:43:48.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:48.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:43:48.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.676028+0000", 2024-09-18T07:43:48.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:43:48.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:43:48.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.676029+0000", 2024-09-18T07:43:48.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.246e-06 2024-09-18T07:43:48.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:43:48.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.676161+0000", 2024-09-18T07:43:48.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013199899999999999 2024-09-18T07:43:48.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:43:48.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.676175+0000", 2024-09-18T07:43:48.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3298e-05 2024-09-18T07:43:48.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:43:48.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.676182+0000", 2024-09-18T07:43:48.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2849999999999997e-06 2024-09-18T07:43:48.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:43:48.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.676328+0000", 2024-09-18T07:43:48.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014640399999999999 2024-09-18T07:43:48.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:48.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:48.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:43:48.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:48.676377+0000", 2024-09-18T07:43:48.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8288999999999998e-05 2024-09-18T07:43:48.700 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:48.701 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:48.701 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:48.701 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:48.701 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:43:48.701 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 3 2024-09-18T07:43:48.701 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:43:48.777 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:48.806 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:48.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:48.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:48.908 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:48.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:48 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:48.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:48 smithi137 ceph-mon[29408]: pgmap v676: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 35534/214545 objects degraded (16.562%) 2024-09-18T07:43:48.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:48 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:48.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:48 smithi137 ceph-mon[39175]: pgmap v676: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 35534/214545 objects degraded (16.562%) 2024-09-18T07:43:48.937 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:48.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:48.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:49.039 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:49.064 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 39 16 75439 75423 120.834 88.875 0.00777407 0.00827064 2024-09-18T07:43:49.067 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:49.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:49.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:49.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:49.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:49.192 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:49.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:48 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:48 smithi203 ceph-mon[32237]: pgmap v676: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 35534/214545 objects degraded (16.562%) 2024-09-18T07:43:49.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:49.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:49.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:49.429 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:49.461 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:49.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:49.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:49.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:49.604 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:49.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:49.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:49.707 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:49.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:49.746 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:49.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:49.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:49.880 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:49.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:49.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:49.959 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T07:43:49.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:50.019 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:50.027 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:50.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:50.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:50.065 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:43:50.063127+0000 min lat: 0.00105851 max lat: 0.230749 avg lat: 0.00835664 2024-09-18T07:43:50.065 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:43:50.065 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 40 16 76575 76559 119.587 71 0.00926317 0.00835664 2024-09-18T07:43:50.130 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:50.178 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:50.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:50.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:50.280 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:50.312 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:50.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:50.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:50.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:50.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:50.459 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:50.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:50.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:50.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:50.602 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:50.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:50.665 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:43:50.666 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:43:50.699 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:50.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:50.706 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:43:50.706 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:43:50.743 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:43:50.744 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:43:50.746 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:50.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:50.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:50.779 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:43:50.779 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:43:50.780 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-18T07:43:50.812 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:43:50.813 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:43:50.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:50.850 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:43:50.850 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:43:50.885 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:43:50.886 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:43:50.886 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:50.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:50.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:50.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:51.026 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:51.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:51.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:51.064 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 41 16 77853 77837 118.619 79.875 0.00711626 0.00842594 2024-09-18T07:43:51.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:51.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:43:51.154 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:51.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:51.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:51.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:50 smithi137 ceph-mon[29408]: pgmap v677: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 100 MiB/s wr, 1.60k op/s; 36296/218010 objects degraded (16.649%) 2024-09-18T07:43:51.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:50 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 36296/218010 objects degraded (16.649%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:51.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:50 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3826521656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:51.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:50 smithi137 ceph-mon[39175]: pgmap v677: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 100 MiB/s wr, 1.60k op/s; 36296/218010 objects degraded (16.649%) 2024-09-18T07:43:51.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:50 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 36296/218010 objects degraded (16.649%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:51.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:50 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3826521656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:51.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:50 smithi203 ceph-mon[32237]: pgmap v677: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 100 MiB/s wr, 1.60k op/s; 36296/218010 objects degraded (16.649%) 2024-09-18T07:43:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:50 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 36296/218010 objects degraded (16.649%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:50 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3826521656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:51.257 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:51.302 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:51.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:51.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:51.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:51.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:51.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:51.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:51.586 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:51.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:51.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:51.688 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:51.715 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:51.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:51.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:51.817 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:51.845 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:51.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:51.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:51.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:51.980 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:51.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:51.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:52.065 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 42 16 79161 79145 117.741 81.75 0.0165785 0.00848823 2024-09-18T07:43:52.082 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:52.116 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:52.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:52.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:52.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:52.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:52.266 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:52.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:52.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:52.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:52.415 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:52.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:52.421 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:43:52.422 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:43:52.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:52.538 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:52.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:52.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:52.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:52.677 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:52.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:52.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:52.779 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:52.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:52.807 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:52.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:52.908 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:52.937 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:52.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:52.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:53.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:53.065 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 43 16 80151 80135 116.442 61.875 0.0140485 0.00858362 2024-09-18T07:43:53.075 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:53.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:53.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:53.126 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-09-18T07:43:53.126 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:43:53.126 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-18T07:43:53.126 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-18T07:43:53.161 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:52 smithi137 ceph-mon[29408]: pgmap v678: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 917 op/s; 37143/220551 objects degraded (16.841%) 2024-09-18T07:43:53.161 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:52 smithi137 ceph-mon[39175]: pgmap v678: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 917 op/s; 37143/220551 objects degraded (16.841%) 2024-09-18T07:43:53.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:53.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:52 smithi203 ceph-mon[32237]: pgmap v678: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 57 MiB/s wr, 917 op/s; 37143/220551 objects degraded (16.841%) 2024-09-18T07:43:53.219 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:53.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:53.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:53.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:53.348 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:53.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:53.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:53.450 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:53.484 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:43:53.484 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:43:53.514 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:53.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:53.515 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:53.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:53.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:53.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:53.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:53.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:53.759 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:53.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:53.816 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:53.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:53.917 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:53.944 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:53.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:54.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:54.065 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 44 16 81420 81404 115.596 79.3125 0.00942871 0.00864593 2024-09-18T07:43:54.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:54.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:54.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:54.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:54.226 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:54.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:54.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:54.269 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:54.295 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T07:43:54.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:54.356 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:54.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:54.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:54.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:43:54.457 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:54.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:54.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:54.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:54.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:54.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:54.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:54.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:54.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:54.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:54.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:54.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:54.988 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:54 smithi137 ceph-mon[29408]: pgmap v679: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 39672/237876 objects degraded (16.678%) 2024-09-18T07:43:54.988 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:54 smithi137 ceph-mon[39175]: pgmap v679: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 39672/237876 objects degraded (16.678%) 2024-09-18T07:43:55.050 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:55.065 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 45 16 82545 82529 114.59 70.3125 0.0214281 0.00872132 2024-09-18T07:43:55.075 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:55.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:55.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:55.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:55.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:54 smithi203 ceph-mon[32237]: pgmap v679: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 39672/237876 objects degraded (16.678%) 2024-09-18T07:43:55.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:55.229 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:55.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:55.276 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:43:55.277 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:43:55.277 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 375, 2024-09-18T07:43:55.277 INFO:teuthology.orchestra.run.smithi137.stdout: "fsid": "f7ef099c-758e-11ef-bceb-c7b262605968", 2024-09-18T07:43:55.277 INFO:teuthology.orchestra.run.smithi137.stdout: "created": "2024-09-18T07:25:19.662367+0000", 2024-09-18T07:43:55.277 INFO:teuthology.orchestra.run.smithi137.stdout: "modified": "2024-09-18T07:43:46.844250+0000", 2024-09-18T07:43:55.277 INFO:teuthology.orchestra.run.smithi137.stdout: "last_up_change": "2024-09-18T07:42:53.723776+0000", 2024-09-18T07:43:55.277 INFO:teuthology.orchestra.run.smithi137.stdout: "last_in_change": "2024-09-18T07:37:42.022730+0000", 2024-09-18T07:43:55.277 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T07:43:55.277 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_num": 5801984, 2024-09-18T07:43:55.277 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_set": [ 2024-09-18T07:43:55.277 INFO:teuthology.orchestra.run.smithi137.stdout: "noscrub", 2024-09-18T07:43:55.277 INFO:teuthology.orchestra.run.smithi137.stdout: "pglog_hardlimit", 2024-09-18T07:43:55.278 INFO:teuthology.orchestra.run.smithi137.stdout: "purged_snapdirs", 2024-09-18T07:43:55.278 INFO:teuthology.orchestra.run.smithi137.stdout: "recovery_deletes", 2024-09-18T07:43:55.278 INFO:teuthology.orchestra.run.smithi137.stdout: "sortbitwise" 2024-09-18T07:43:55.278 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:43:55.278 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_version": 18, 2024-09-18T07:43:55.278 INFO:teuthology.orchestra.run.smithi137.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T07:43:55.278 INFO:teuthology.orchestra.run.smithi137.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T07:43:55.278 INFO:teuthology.orchestra.run.smithi137.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T07:43:55.278 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_snapshot": "", 2024-09-18T07:43:55.278 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_max": 10, 2024-09-18T07:43:55.278 INFO:teuthology.orchestra.run.smithi137.stdout: "max_osd": 8, 2024-09-18T07:43:55.278 INFO:teuthology.orchestra.run.smithi137.stdout: "require_min_compat_client": "luminous", 2024-09-18T07:43:55.278 INFO:teuthology.orchestra.run.smithi137.stdout: "min_compat_client": "luminous", 2024-09-18T07:43:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: "require_osd_release": "squid", 2024-09-18T07:43:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: "allow_crimson": false, 2024-09-18T07:43:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: "pools": [ 2024-09-18T07:43:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 1, 2024-09-18T07:43:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".mgr", 2024-09-18T07:43:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:28:20.899655+0000", 2024-09-18T07:43:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:43:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:43:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:43:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:43:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:43:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:43:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:43:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:43:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:43:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:43:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:43:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:43:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:43:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 11, 2024-09-18T07:43:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 11, 2024-09-18T07:43:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 11, 2024-09-18T07:43:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 11, 2024-09-18T07:43:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 11, 2024-09-18T07:43:55.280 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:43:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:43:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:43:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:43:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:43:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:43:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:43:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "274", 2024-09-18T07:43:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:43:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:43:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "272", 2024-09-18T07:43:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:43:55.281 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:43:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:43:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:43:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:43:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:43:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:43:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:43:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:43:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:43:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:43:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:43:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:43:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:43:55.282 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:43:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:43:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:43:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:43:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:43:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:43:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:43:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:43:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:43:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:43:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:43:55.283 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:43:55.284 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:43:55.284 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:43:55.284 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:43:55.284 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:43:55.284 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:43:55.284 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:43:55.284 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:43:55.284 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:43:55.284 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:43:55.284 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_max": 32, 2024-09-18T07:43:55.284 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_min": 1 2024-09-18T07:43:55.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.284 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:43:55.285 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": {} 2024-09-18T07:43:55.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.285 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:43:55.285 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:43:55.285 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.9099999666213989, 2024-09-18T07:43:55.285 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.9099999666213989, 2024-09-18T07:43:55.285 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:43:55.285 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.9099999666213989, 2024-09-18T07:43:55.285 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.9099999666213989, 2024-09-18T07:43:55.285 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:43:55.285 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:43:55.285 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:43:55.285 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.286 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 2, 2024-09-18T07:43:55.286 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "datapool", 2024-09-18T07:43:55.286 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:53.715373+0000", 2024-09-18T07:43:55.286 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-18T07:43:55.286 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T07:43:55.286 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:43:55.286 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:43:55.286 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:43:55.286 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:43:55.286 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:43:55.286 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:43:55.286 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:43:55.287 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:43:55.287 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:43:55.287 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:43:55.287 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:43:55.287 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 3, 2024-09-18T07:43:55.287 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 3, 2024-09-18T07:43:55.287 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 3, 2024-09-18T07:43:55.287 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 3, 2024-09-18T07:43:55.287 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 3, 2024-09-18T07:43:55.287 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:43:55.287 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:43:55.287 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:43:55.287 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:43:55.288 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:43:55.288 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:43:55.288 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:43:55.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.288 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "63", 2024-09-18T07:43:55.288 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:43:55.288 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:43:55.288 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:43:55.288 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:43:55.288 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:43:55.288 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 2, 2024-09-18T07:43:55.288 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 63, 2024-09-18T07:43:55.288 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:43:55.289 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:43:55.289 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:43:55.289 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:43:55.289 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:43:55.289 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:43:55.289 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:43:55.289 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:43:55.289 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:43:55.289 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:43:55.289 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:43:55.289 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:43:55.289 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:43:55.289 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:43:55.289 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:43:55.290 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:43:55.290 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:43:55.290 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:43:55.290 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:43:55.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.290 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:43:55.290 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:43:55.290 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:43:55.290 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:43:55.290 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:43:55.290 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:43:55.290 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:43:55.290 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:43:55.290 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:43:55.291 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:43:55.291 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:43:55.291 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:43:55.291 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:43:55.291 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd": {} 2024-09-18T07:43:55.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.291 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:43:55.291 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:43:55.291 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.3299999237060547, 2024-09-18T07:43:55.291 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.3299999237060547, 2024-09-18T07:43:55.291 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:43:55.291 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-18T07:43:55.291 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-18T07:43:55.291 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:43:55.292 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:43:55.292 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:43:55.292 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.292 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 3, 2024-09-18T07:43:55.292 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".rgw.root", 2024-09-18T07:43:55.292 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:54.672127+0000", 2024-09-18T07:43:55.292 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:43:55.292 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:43:55.292 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:43:55.292 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:43:55.292 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:43:55.292 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:43:55.293 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:43:55.293 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:43:55.293 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:43:55.293 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:43:55.293 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:43:55.293 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:43:55.293 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:43:55.293 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:43:55.293 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:43:55.293 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:43:55.293 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:43:55.293 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:43:55.293 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:43:55.293 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:43:55.294 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:43:55.294 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:43:55.294 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:43:55.294 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:43:55.294 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:43:55.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.294 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "62", 2024-09-18T07:43:55.294 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:43:55.294 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:43:55.295 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:43:55.295 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:43:55.295 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:43:55.295 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:43:55.295 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:43:55.295 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:43:55.295 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:43:55.295 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:43:55.295 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:43:55.295 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:43:55.295 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:43:55.295 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:43:55.295 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:43:55.295 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:43:55.296 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:43:55.296 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:43:55.296 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:43:55.296 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:43:55.296 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:43:55.296 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:43:55.296 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:43:55.296 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:43:55.296 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:43:55.296 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:43:55.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.296 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:43:55.296 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:43:55.297 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:43:55.297 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:43:55.297 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:43:55.297 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:43:55.297 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:43:55.297 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:43:55.297 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:43:55.297 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:43:55.297 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:43:55.297 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:43:55.297 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:43:55.297 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:43:55.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.297 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:43:55.298 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:43:55.298 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.75, 2024-09-18T07:43:55.298 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.75, 2024-09-18T07:43:55.298 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:43:55.298 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.75, 2024-09-18T07:43:55.298 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.75, 2024-09-18T07:43:55.298 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:43:55.298 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:43:55.298 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:43:55.298 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.299 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 4, 2024-09-18T07:43:55.299 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.log", 2024-09-18T07:43:55.299 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:56.351343+0000", 2024-09-18T07:43:55.299 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:43:55.299 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:43:55.299 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:43:55.299 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:43:55.299 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:43:55.299 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:43:55.299 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:43:55.299 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:43:55.299 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:43:55.299 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:43:55.300 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:43:55.300 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:43:55.300 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:43:55.300 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:43:55.300 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:43:55.300 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:43:55.300 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:43:55.300 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:43:55.300 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:43:55.300 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:43:55.300 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:43:55.300 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:43:55.300 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:43:55.301 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:43:55.301 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:43:55.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.301 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "64", 2024-09-18T07:43:55.301 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:43:55.301 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:43:55.301 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:43:55.301 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:43:55.301 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:43:55.301 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:43:55.301 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:43:55.301 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:43:55.301 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:43:55.302 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:43:55.302 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:43:55.302 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:43:55.302 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:43:55.302 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:43:55.302 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:43:55.302 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:43:55.302 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:43:55.302 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:43:55.302 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:43:55.302 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:43:55.302 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:43:55.302 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:43:55.303 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:43:55.303 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:43:55.303 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:43:55.303 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:43:55.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.303 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:43:55.303 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:43:55.303 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:43:55.303 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:43:55.303 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:43:55.303 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:43:55.303 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:43:55.303 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:43:55.304 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:43:55.304 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:43:55.304 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:43:55.304 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:43:55.304 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:43:55.304 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:43:55.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.304 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:43:55.304 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:43:55.304 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.5299999713897705, 2024-09-18T07:43:55.304 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.5299999713897705, 2024-09-18T07:43:55.304 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:43:55.304 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-18T07:43:55.305 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-18T07:43:55.305 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:43:55.305 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:43:55.305 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:43:55.305 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.305 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 5, 2024-09-18T07:43:55.305 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.control", 2024-09-18T07:43:55.305 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:58.322401+0000", 2024-09-18T07:43:55.305 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:43:55.305 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:43:55.305 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:43:55.306 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:43:55.306 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:43:55.306 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:43:55.306 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:43:55.306 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:43:55.306 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:43:55.306 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:43:55.306 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:43:55.306 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:43:55.306 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:43:55.306 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:43:55.306 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:43:55.306 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:43:55.306 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:43:55.307 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:43:55.307 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:43:55.307 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:43:55.307 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:43:55.307 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:43:55.307 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:43:55.307 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:43:55.307 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:43:55.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.307 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "66", 2024-09-18T07:43:55.307 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:43:55.307 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:43:55.307 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:43:55.308 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:43:55.308 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:43:55.308 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:43:55.308 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:43:55.308 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:43:55.308 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:43:55.308 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:43:55.308 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:43:55.308 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:43:55.308 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:43:55.308 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:43:55.308 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:43:55.308 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:43:55.309 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:43:55.309 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:43:55.309 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:43:55.309 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:43:55.309 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:43:55.309 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:43:55.309 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:43:55.309 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:43:55.309 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:43:55.309 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:43:55.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.309 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:43:55.309 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:43:55.309 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:43:55.310 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:43:55.310 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:43:55.310 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:43:55.310 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:43:55.310 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:43:55.310 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:43:55.310 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:43:55.310 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:43:55.310 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:43:55.310 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:43:55.310 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:43:55.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.310 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:43:55.311 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:43:55.311 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.75, 2024-09-18T07:43:55.311 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.75, 2024-09-18T07:43:55.311 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:43:55.312 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.75, 2024-09-18T07:43:55.312 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.75, 2024-09-18T07:43:55.312 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:43:55.312 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:43:55.312 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:43:55.312 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.312 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 6, 2024-09-18T07:43:55.312 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.meta", 2024-09-18T07:43:55.312 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:31:00.390499+0000", 2024-09-18T07:43:55.313 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:43:55.313 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:43:55.313 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:43:55.313 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:43:55.313 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:43:55.313 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:43:55.313 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:43:55.313 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:43:55.313 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:43:55.313 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:43:55.313 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:43:55.313 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:43:55.313 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:43:55.314 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:43:55.314 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:43:55.314 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:43:55.314 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:43:55.314 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:43:55.314 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:43:55.314 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:43:55.314 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:43:55.314 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:43:55.314 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:43:55.314 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:43:55.314 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:43:55.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.314 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "68", 2024-09-18T07:43:55.315 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:43:55.315 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:43:55.315 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:43:55.315 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:43:55.315 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:43:55.315 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:43:55.315 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:43:55.315 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:43:55.315 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:43:55.315 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:43:55.315 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:43:55.315 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:43:55.315 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:43:55.315 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:43:55.316 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:43:55.316 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:43:55.316 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:43:55.316 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:43:55.316 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:43:55.316 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:43:55.316 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:43:55.316 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:43:55.316 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:43:55.316 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:43:55.316 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:43:55.316 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:43:55.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.317 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:43:55.317 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:43:55.317 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:43:55.317 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:43:55.317 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:43:55.317 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:43:55.317 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:43:55.317 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:43:55.317 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:43:55.317 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:43:55.317 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:43:55.317 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:43:55.317 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_bias": 4 2024-09-18T07:43:55.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.318 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:43:55.318 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:43:55.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.318 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:43:55.318 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:43:55.318 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.5299999713897705, 2024-09-18T07:43:55.318 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.5299999713897705, 2024-09-18T07:43:55.318 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:43:55.318 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-18T07:43:55.318 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-18T07:43:55.318 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:43:55.318 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:43:55.318 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:43:55.319 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.319 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 10, 2024-09-18T07:43:55.319 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "unique_pool_3", 2024-09-18T07:43:55.319 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:43:00.547513+0000", 2024-09-18T07:43:55.319 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:43:55.319 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:43:55.319 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:43:55.319 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:43:55.319 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:43:55.319 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:43:55.319 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:43:55.320 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:43:55.320 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:43:55.320 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:43:55.320 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:43:55.320 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:43:55.320 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:43:55.320 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 16, 2024-09-18T07:43:55.320 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 16, 2024-09-18T07:43:55.320 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 16, 2024-09-18T07:43:55.320 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 16, 2024-09-18T07:43:55.320 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 16, 2024-09-18T07:43:55.320 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:43:55.320 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:43:55.320 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:43:55.321 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:43:55.321 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:43:55.321 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:43:55.321 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:43:55.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.321 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "367", 2024-09-18T07:43:55.321 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:43:55.321 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:43:55.321 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:43:55.321 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:43:55.321 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:43:55.321 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:43:55.322 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:43:55.322 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:43:55.322 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:43:55.322 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:43:55.322 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:43:55.322 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:43:55.322 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:43:55.322 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:43:55.322 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:43:55.322 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:43:55.322 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:43:55.322 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:43:55.322 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:43:55.322 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:43:55.323 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:43:55.323 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:43:55.323 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:43:55.323 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:43:55.323 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:43:55.323 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:43:55.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.323 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:43:55.323 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:43:55.323 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:43:55.323 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:43:55.323 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:43:55.323 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:43:55.323 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:43:55.324 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:43:55.324 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:43:55.324 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:43:55.324 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:43:55.324 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:43:55.324 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:43:55.324 INFO:teuthology.orchestra.run.smithi137.stdout: "rados": {} 2024-09-18T07:43:55.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.324 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:43:55.324 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:43:55.324 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 3.059999942779541, 2024-09-18T07:43:55.324 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 3.059999942779541, 2024-09-18T07:43:55.324 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:43:55.325 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 3.059999942779541, 2024-09-18T07:43:55.325 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 3.059999942779541, 2024-09-18T07:43:55.325 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:43:55.325 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:43:55.325 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:43:55.325 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.325 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.325 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:43:55.325 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-18T07:43:55.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.325 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:43:55.325 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "268cd580-c931-4992-bb93-d6f63c2bfe88", 2024-09-18T07:43:55.325 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:43:55.325 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:43:55.326 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:43:55.326 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:43:55.326 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:43:55.326 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:43:55.326 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 8, 2024-09-18T07:43:55.326 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 364, 2024-09-18T07:43:55.326 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:43:55.326 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:43:55.326 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:43:55.326 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.326 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.326 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6801", 2024-09-18T07:43:55.326 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:43:55.327 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.327 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.327 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:43:55.327 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.327 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.327 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6802", 2024-09-18T07:43:55.327 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:43:55.327 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.327 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.327 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:43:55.327 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.328 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.328 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6804", 2024-09-18T07:43:55.328 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:43:55.328 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.328 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.328 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:43:55.328 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.328 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.328 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6803", 2024-09-18T07:43:55.328 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:43:55.328 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.329 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.329 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6801/2811444477", 2024-09-18T07:43:55.329 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6802/2811444477", 2024-09-18T07:43:55.329 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6804/2811444477", 2024-09-18T07:43:55.329 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6803/2811444477", 2024-09-18T07:43:55.329 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:43:55.329 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:43:55.329 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:43:55.329 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.329 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:43:55.330 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575", 2024-09-18T07:43:55.330 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:43:55.330 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:43:55.330 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:43:55.330 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 0, 2024-09-18T07:43:55.330 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:43:55.330 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:43:55.330 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 13, 2024-09-18T07:43:55.330 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 359, 2024-09-18T07:43:55.330 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:43:55.330 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:43:55.330 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:43:55.330 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.331 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.331 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6805", 2024-09-18T07:43:55.331 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:43:55.331 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.331 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.331 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:43:55.331 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.331 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.331 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6806", 2024-09-18T07:43:55.331 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:43:55.331 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.331 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.332 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:43:55.332 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.332 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.332 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6808", 2024-09-18T07:43:55.332 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:43:55.332 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.332 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.332 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:43:55.332 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.332 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.333 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6807", 2024-09-18T07:43:55.333 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:43:55.333 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.333 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.333 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6805/874102487", 2024-09-18T07:43:55.333 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6806/874102487", 2024-09-18T07:43:55.333 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6808/874102487", 2024-09-18T07:43:55.333 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6807/874102487", 2024-09-18T07:43:55.333 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:43:55.333 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:43:55.333 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:43:55.333 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.334 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:43:55.334 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "480ad6c5-0e0b-4405-bbe2-381682d00bae", 2024-09-18T07:43:55.334 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:43:55.334 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:43:55.334 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:43:55.334 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:43:55.334 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:43:55.334 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:43:55.334 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 18, 2024-09-18T07:43:55.334 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 364, 2024-09-18T07:43:55.334 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:43:55.335 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:43:55.335 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:43:55.335 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.335 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.335 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6809", 2024-09-18T07:43:55.335 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:43:55.335 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.335 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.335 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:43:55.335 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.336 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.336 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6810", 2024-09-18T07:43:55.336 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:43:55.336 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.336 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.336 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:43:55.336 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.336 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.336 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6812", 2024-09-18T07:43:55.336 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:43:55.336 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.336 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.337 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:43:55.337 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.337 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.337 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6811", 2024-09-18T07:43:55.337 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:43:55.337 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.337 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.337 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6809/1754684686", 2024-09-18T07:43:55.337 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6810/1754684686", 2024-09-18T07:43:55.337 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6812/1754684686", 2024-09-18T07:43:55.338 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6811/1754684686", 2024-09-18T07:43:55.338 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:43:55.338 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:43:55.338 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:43:55.338 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.338 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:43:55.338 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14", 2024-09-18T07:43:55.338 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:43:55.338 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:43:55.338 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:43:55.338 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 0.759490966796875, 2024-09-18T07:43:55.339 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:43:55.339 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:43:55.339 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 26, 2024-09-18T07:43:55.339 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 359, 2024-09-18T07:43:55.339 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:43:55.339 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:43:55.339 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:43:55.339 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.339 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.339 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6813", 2024-09-18T07:43:55.339 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:43:55.339 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.339 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.340 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:43:55.340 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.340 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.340 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6814", 2024-09-18T07:43:55.340 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:43:55.340 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.340 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.340 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:43:55.340 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.341 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.341 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6816", 2024-09-18T07:43:55.341 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:43:55.341 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.341 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.341 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:43:55.341 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.341 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.341 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6815", 2024-09-18T07:43:55.341 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:43:55.341 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.342 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.342 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6813/116058392", 2024-09-18T07:43:55.342 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6814/116058392", 2024-09-18T07:43:55.342 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6816/116058392", 2024-09-18T07:43:55.342 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6815/116058392", 2024-09-18T07:43:55.342 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:43:55.342 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:43:55.342 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:43:55.342 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.342 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:43:55.343 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de", 2024-09-18T07:43:55.343 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 0, 2024-09-18T07:43:55.343 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:43:55.343 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:43:55.343 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:43:55.343 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:43:55.343 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:43:55.343 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 32, 2024-09-18T07:43:55.343 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 289, 2024-09-18T07:43:55.343 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 359, 2024-09-18T07:43:55.343 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:43:55.343 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:43:55.343 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.344 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.344 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6800", 2024-09-18T07:43:55.344 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:43:55.344 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.344 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.344 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:43:55.344 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.344 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.344 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6801", 2024-09-18T07:43:55.344 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:43:55.344 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.345 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.345 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:43:55.345 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.345 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.345 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6803", 2024-09-18T07:43:55.345 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:43:55.345 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.345 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.345 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:43:55.345 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.346 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.346 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6802", 2024-09-18T07:43:55.346 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:43:55.346 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.346 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.346 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6800/3654297954", 2024-09-18T07:43:55.346 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6801/3654297954", 2024-09-18T07:43:55.346 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6803/3654297954", 2024-09-18T07:43:55.346 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6802/3654297954", 2024-09-18T07:43:55.346 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:43:55.346 INFO:teuthology.orchestra.run.smithi137.stdout: "exists" 2024-09-18T07:43:55.346 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:43:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c", 2024-09-18T07:43:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:43:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:43:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:43:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:43:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:43:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:43:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 39, 2024-09-18T07:43:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 364, 2024-09-18T07:43:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:43:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:43:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:43:55.348 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.348 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.348 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:55.349 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6804", 2024-09-18T07:43:55.349 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:43:55.349 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.349 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.349 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:43:55.349 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.349 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.350 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6805", 2024-09-18T07:43:55.350 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:43:55.350 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.350 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.350 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:43:55.350 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.350 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.350 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6807", 2024-09-18T07:43:55.350 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:43:55.350 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.350 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.351 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:43:55.351 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.351 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.351 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6806", 2024-09-18T07:43:55.351 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:43:55.351 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.351 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.351 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6804/837994222", 2024-09-18T07:43:55.351 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6805/837994222", 2024-09-18T07:43:55.351 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6807/837994222", 2024-09-18T07:43:55.351 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6806/837994222", 2024-09-18T07:43:55.352 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:43:55.352 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:43:55.352 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:43:55.352 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.352 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:43:55.352 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6", 2024-09-18T07:43:55.352 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:43:55.352 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:43:55.352 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:43:55.352 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:43:55.352 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:43:55.352 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:43:55.353 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 47, 2024-09-18T07:43:55.353 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 364, 2024-09-18T07:43:55.353 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:43:55.353 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:43:55.353 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:43:55.353 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.353 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.353 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6808", 2024-09-18T07:43:55.353 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:43:55.353 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.353 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.354 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:43:55.354 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.354 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.354 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6809", 2024-09-18T07:43:55.354 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:43:55.354 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.354 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.354 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:43:55.354 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.354 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.354 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6811", 2024-09-18T07:43:55.355 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:43:55.355 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.355 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.355 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:43:55.355 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.355 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.355 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6810", 2024-09-18T07:43:55.355 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:43:55.355 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.355 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.355 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6808/1494175466", 2024-09-18T07:43:55.356 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6809/1494175466", 2024-09-18T07:43:55.356 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6811/1494175466", 2024-09-18T07:43:55.356 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6810/1494175466", 2024-09-18T07:43:55.356 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:43:55.356 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:43:55.356 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:43:55.356 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.356 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:43:55.356 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a", 2024-09-18T07:43:55.357 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:43:55.357 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:43:55.357 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:43:55.357 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:43:55.357 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:43:55.357 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:43:55.357 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 55, 2024-09-18T07:43:55.357 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 364, 2024-09-18T07:43:55.357 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:43:55.357 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:43:55.357 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:43:55.357 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.358 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.358 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6812", 2024-09-18T07:43:55.358 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:43:55.358 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.358 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.358 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:43:55.358 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.358 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.358 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6813", 2024-09-18T07:43:55.358 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:43:55.358 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.359 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.359 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:43:55.359 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.359 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.359 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6815", 2024-09-18T07:43:55.359 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:43:55.359 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.359 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.359 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:43:55.359 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:43:55.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.360 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:43:55.360 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6814", 2024-09-18T07:43:55.360 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:43:55.360 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.360 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.360 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6812/2670809315", 2024-09-18T07:43:55.360 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6813/2670809315", 2024-09-18T07:43:55.360 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6815/2670809315", 2024-09-18T07:43:55.360 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6814/2670809315", 2024-09-18T07:43:55.360 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:43:55.360 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:43:55.360 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:43:55.361 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.361 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.361 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:43:55.361 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_xinfo": [ 2024-09-18T07:43:55.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.361 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:43:55.361 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:43:55.361 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:43:55.361 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:43:55.361 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:43:55.361 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:43:55.361 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:43:01.127218+0000", 2024-09-18T07:43:55.361 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:43:55.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.362 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:43:55.362 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:43:55.362 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:43:55.362 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:43:55.362 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:43:55.362 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:43:55.362 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:43:41.550939+0000", 2024-09-18T07:43:55.362 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:43:55.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.362 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:43:55.362 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:43:55.363 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:43:55.363 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:43:55.363 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:43:55.363 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:43:55.363 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:43:24.037996+0000", 2024-09-18T07:43:55.363 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:43:55.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.363 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:43:55.363 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:43:55.363 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:43:55.363 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:43:55.363 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:43:55.364 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:43:55.364 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:43:16.491536+0000", 2024-09-18T07:43:55.364 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:43:55.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.364 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:43:55.364 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:43:55.364 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:43:55.364 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:43:55.364 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:43:55.364 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:43:55.364 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:41:47.479848+0000", 2024-09-18T07:43:55.364 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 358 2024-09-18T07:43:55.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.365 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:43:55.365 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:43:55.365 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:43:55.365 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:43:55.365 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:43:55.365 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:43:55.365 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:42:35.305044+0000", 2024-09-18T07:43:55.365 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:43:55.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.365 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:43:55.365 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:43:55.366 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:43:55.366 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:43:55.366 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:43:55.366 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:43:55.366 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:42:34.238640+0000", 2024-09-18T07:43:55.366 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:43:55.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.366 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:43:55.366 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:43:55.366 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:43:55.366 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:43:55.366 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:43:55.366 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:43:55.367 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:42:59.881262+0000", 2024-09-18T07:43:55.367 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:43:55.367 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.367 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:43:55.367 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap": [], 2024-09-18T07:43:55.367 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_items": [ 2024-09-18T07:43:55.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.367 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.6", 2024-09-18T07:43:55.367 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:55.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.367 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:43:55.367 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:43:55.367 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.367 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.368 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.b", 2024-09-18T07:43:55.368 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:55.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.368 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:43:55.368 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:43:55.368 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.368 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.368 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.e", 2024-09-18T07:43:55.368 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:55.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.369 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:43:55.369 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:43:55.369 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.369 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.369 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.10", 2024-09-18T07:43:55.369 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:55.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.369 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:43:55.369 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:43:55.369 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.369 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.370 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.12", 2024-09-18T07:43:55.370 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:55.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.370 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:43:55.370 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:43:55.370 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.370 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.370 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.13", 2024-09-18T07:43:55.370 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:55.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.371 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 6, 2024-09-18T07:43:55.371 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 7 2024-09-18T07:43:55.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.371 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:43:55.371 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:43:55.371 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.371 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.371 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.18", 2024-09-18T07:43:55.371 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:55.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.371 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:43:55.372 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:43:55.372 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.372 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.372 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.1b", 2024-09-18T07:43:55.372 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:55.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.372 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:43:55.372 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:43:55.372 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.372 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.373 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.1c", 2024-09-18T07:43:55.373 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:55.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.373 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:43:55.373 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:43:55.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.373 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:43:55.373 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 7 2024-09-18T07:43:55.373 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.373 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.374 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.1d", 2024-09-18T07:43:55.374 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:55.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.374 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:43:55.374 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:43:55.374 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.374 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.374 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.b", 2024-09-18T07:43:55.374 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:55.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.374 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 0, 2024-09-18T07:43:55.374 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 3 2024-09-18T07:43:55.375 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.375 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.375 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.1b", 2024-09-18T07:43:55.375 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:43:55.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:43:55.375 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:43:55.375 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:43:55.375 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.375 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:43:55.375 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.375 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:43:55.376 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_primaries": [], 2024-09-18T07:43:55.376 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_temp": [], 2024-09-18T07:43:55.376 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_temp": [], 2024-09-18T07:43:55.376 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklist": { 2024-09-18T07:43:55.376 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/2096039404": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:43:55.376 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/3636177939": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:43:55.376 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/334967443": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:43:55.376 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2101286111": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:43:55.376 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3387637412": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:43:55.376 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762989503": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:43:55.376 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3490817671": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:43:55.376 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1482147392": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:43:55.376 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1821482181": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:43:55.376 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/4028304023": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:43:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/621434521": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:43:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3160704935": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:43:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2357293499": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:43:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762352860": "2024-09-19T07:31:25.389124+0000" 2024-09-18T07:43:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: "range_blocklist": {}, 2024-09-18T07:43:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profiles": { 2024-09-18T07:43:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: "default": { 2024-09-18T07:43:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: "crush-failure-domain": "osd", 2024-09-18T07:43:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: "k": "2", 2024-09-18T07:43:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: "m": "1", 2024-09-18T07:43:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin": "jerasure", 2024-09-18T07:43:55.377 INFO:teuthology.orchestra.run.smithi137.stdout: "technique": "reed_sol_van" 2024-09-18T07:43:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:43:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps_queue": [], 2024-09-18T07:43:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "new_removed_snaps": [], 2024-09-18T07:43:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "new_purged_snaps": [], 2024-09-18T07:43:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_node_flags": {}, 2024-09-18T07:43:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "device_class_flags": {}, 2024-09-18T07:43:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode": { 2024-09-18T07:43:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_enabled": false, 2024-09-18T07:43:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_bucket_count": 0, 2024-09-18T07:43:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "degraded_stretch_mode": 0, 2024-09-18T07:43:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "recovering_stretch_mode": 0, 2024-09-18T07:43:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_bucket": 0 2024-09-18T07:43:55.378 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:43:55.379 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:43:55.403 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:55.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:55.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:55.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:55.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:55.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:55.646 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:55.672 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:55.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:55.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:55.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:55.808 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:55.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:55.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:55.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:55.945 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:55.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:55.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:56.047 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:56.068 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:43:56.069 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 46 16 83800 83784 113.799 78.4375 0.0136815 0.0087825 2024-09-18T07:43:56.080 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:56.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:56.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:56.166 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:55 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 39672/237876 objects degraded (16.678%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:56.166 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:55 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4020219183' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:43:56.166 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:55 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/225372192' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:56.166 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:55 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 39672/237876 objects degraded (16.678%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:56.167 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:55 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4020219183' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:43:56.167 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:55 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/225372192' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:56.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:56.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:55 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 39672/237876 objects degraded (16.678%), 30 pgs degraded (PG_DEGRADED) 2024-09-18T07:43:56.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:55 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4020219183' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:43:56.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:55 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/225372192' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:43:56.222 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:56.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:56.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:56.324 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:56.350 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:56.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:56.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:56.452 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:56.479 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:56.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:56.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:56.581 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:56.591 INFO:tasks.thrashosds.thrasher:j is {'epoch': 375, 'fsid': 'f7ef099c-758e-11ef-bceb-c7b262605968', 'created': '2024-09-18T07:25:19.662367+0000', 'modified': '2024-09-18T07:43:46.844250+0000', 'last_up_change': '2024-09-18T07:42:53.723776+0000', 'last_in_change': '2024-09-18T07:37:42.022730+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T07:28:20.899655+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '274', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '272', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 1, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-18T07:30:53.715373+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 1, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-18T07:30:54.672127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-18T07:30:56.351343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-18T07:30:58.322401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-18T07:31:00.390499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-18T07:43:00.547513+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '367', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.059999942779541, 'score_stable': 3.059999942779541, 'optimal_score': 1, 'raw_score_acting': 3.059999942779541, 'raw_score_stable': 3.059999942779541, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}], 'osds': [{'osd': 0, 'uuid': '268cd580-c931-4992-bb93-d6f63c2bfe88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6801', 'nonce': 2811444477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6802', 'nonce': 2811444477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6804', 'nonce': 2811444477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6803', 'nonce': 2811444477}]}, 'public_addr': '172.21.15.137:6801/2811444477', 'cluster_addr': '172.21.15.137:6802/2811444477', 'heartbeat_back_addr': '172.21.15.137:6804/2811444477', 'heartbeat_front_addr': '172.21.15.137:6803/2811444477', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b06907bc-b499-46c0-8667-c5bdc0d85575', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 359, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6805', 'nonce': 874102487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6806', 'nonce': 874102487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6808', 'nonce': 874102487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6807', 'nonce': 874102487}]}, 'public_addr': '172.21.15.137:6805/874102487', 'cluster_addr': '172.21.15.137:6806/874102487', 'heartbeat_back_addr': '172.21.15.137:6808/874102487', 'heartbeat_front_addr': '172.21.15.137:6807/874102487', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '480ad6c5-0e0b-4405-bbe2-381682d00bae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6809', 'nonce': 1754684686}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6810', 'nonce': 1754684686}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6812', 'nonce': 1754684686}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6811', 'nonce': 1754684686}]}, 'public_addr': '172.21.15.137:6809/1754684686', 'cluster_addr': '172.21.15.137:6810/1754684686', 'heartbeat_back_addr': '172.21.15.137:6812/1754684686', 'heartbeat_front_addr': '172.21.15.137:6811/1754684686', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '178be3e0-f07b-4856-807b-2a1181e48a14', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.759490966796875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 359, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6813', 'nonce': 116058392}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6814', 'nonce': 116058392}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6816', 'nonce': 116058392}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6815', 'nonce': 116058392}]}, 'public_addr': '172.21.15.137:6813/116058392', 'cluster_addr': '172.21.15.137:6814/116058392', 'heartbeat_back_addr': '172.21.15.137:6816/116058392', 'heartbeat_front_addr': '172.21.15.137:6815/116058392', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7547102c-951a-4077-a094-fb8a65bfe6de', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 289, 'down_at': 359, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6800', 'nonce': 3654297954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6801', 'nonce': 3654297954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6803', 'nonce': 3654297954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6802', 'nonce': 3654297954}]}, 'public_addr': '172.21.15.203:6800/3654297954', 'cluster_addr': '172.21.15.203:6801/3654297954', 'heartbeat_back_addr': '172.21.15.203:6803/3654297954', 'heartbeat_front_addr': '172.21.15.203:6802/3654297954', 'state': ['exists']}, {'osd': 5, 'uuid': '49c17901-8102-48a4-9348-fa1bdb63032c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6804', 'nonce': 837994222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6805', 'nonce': 837994222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6807', 'nonce': 837994222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6806', 'nonce': 837994222}]}, 'public_addr': '172.21.15.203:6804/837994222', 'cluster_addr': '172.21.15.203:6805/837994222', 'heartbeat_back_addr': '172.21.15.203:6807/837994222', 'heartbeat_front_addr': '172.21.15.203:6806/837994222', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '75f0ead4-e981-4a15-b1c1-d83dce871ab6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6808', 'nonce': 1494175466}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6809', 'nonce': 1494175466}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6811', 'nonce': 1494175466}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6810', 'nonce': 1494175466}]}, 'public_addr': '172.21.15.203:6808/1494175466', 'cluster_addr': '172.21.15.203:6809/1494175466', 'heartbeat_back_addr': '172.21.15.203:6811/1494175466', 'heartbeat_front_addr': '172.21.15.203:6810/1494175466', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c0dbf299-6798-4b99-9a85-62b123971f2a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 55, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6812', 'nonce': 2670809315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6813', 'nonce': 2670809315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6815', 'nonce': 2670809315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6814', 'nonce': 2670809315}]}, 'public_addr': '172.21.15.203:6812/2670809315', 'cluster_addr': '172.21.15.203:6813/2670809315', 'heartbeat_back_addr': '172.21.15.203:6815/2670809315', 'heartbeat_front_addr': '172.21.15.203:6814/2670809315', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:43:01.127218+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:43:41.550939+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:43:24.037996+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:43:16.491536+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:41:47.479848+0000', 'dead_epoch': 358}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:42:35.305044+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:42:34.238640+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:42:59.881262+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.13', 'mappings': [{'from': 6, 'to': 7}, {'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1b', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '4.1c', 'mappings': [{'from': 3, 'to': 0}, {'from': 1, 'to': 7}]}, {'pgid': '4.1d', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '5.b', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '6.1b', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.137:6800/2096039404': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:6800/3636177939': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/334967443': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/2101286111': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/3387637412': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1762989503': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3490817671': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:6800/1482147392': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1821482181': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/4028304023': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/621434521': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3160704935': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/2357293499': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/1762352860': '2024-09-19T07:31:25.389124+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T07:43:56.592 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:43:56.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:56.615 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:56.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:56.717 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:56.745 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:56.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:56.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:56.846 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:56.872 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:56.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:56.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:56.974 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:57.013 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:57.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:57.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:57.069 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 47 15 85099 85084 113.106 81.25 0.00841093 0.00883703 2024-09-18T07:43:57.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:57.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:57.159 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:57.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:57.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:56 smithi137 ceph-mon[29408]: pgmap v680: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 39672/237876 objects degraded (16.678%) 2024-09-18T07:43:57.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:56 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:57.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:56 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/225372192' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:43:57.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:56 smithi137 ceph-mon[29408]: osdmap e376: 8 total, 7 up, 8 in 2024-09-18T07:43:57.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:56 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:43:57.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:56 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:43:57.170 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:43:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:43:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:43:57.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:56 smithi137 ceph-mon[39175]: pgmap v680: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 39672/237876 objects degraded (16.678%) 2024-09-18T07:43:57.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:56 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:57.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:56 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/225372192' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:43:57.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:56 smithi137 ceph-mon[39175]: osdmap e376: 8 total, 7 up, 8 in 2024-09-18T07:43:57.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:56 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:43:57.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:56 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:43:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:56 smithi203 ceph-mon[32237]: pgmap v680: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 4.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 39672/237876 objects degraded (16.678%) 2024-09-18T07:43:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:56 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:43:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:56 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/225372192' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:43:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:56 smithi203 ceph-mon[32237]: osdmap e376: 8 total, 7 up, 8 in 2024-09-18T07:43:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:56 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:43:57.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:56 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:43:57.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:57.298 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:57.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:57.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:57.332 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:57.373 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:57.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:57.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:57.449 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:57.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:57.551 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:57.575 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:57.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:57.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:57.677 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:57.702 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:57.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:57.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:57.804 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:57.832 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:57.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:57.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:57.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:57.960 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:43:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:43:57.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:57.974 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:57.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:58.068 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 48 16 86382 86366 112.419 80.125 0.00612928 0.00889071 2024-09-18T07:43:58.076 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:58.117 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:58.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:58.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:58.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:43:58.231 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:43:58.231 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:43:58.257 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:58.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:58.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:43:58.359 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:58.384 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:58.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:58.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:58.470 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:43:58.470 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:43:58.486 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:43:58.509 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:43:58.510 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:43:58.520 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:58.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:58.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:43:58.546 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:43:58.546 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:43:58.578 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:43:58.579 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:43:58.579 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-18T07:43:58.611 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:43:58.611 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:43:58.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:58.643 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:43:58.643 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:43:58.662 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:58.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:58.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:58.678 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:43:58.678 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:43:58.764 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:43:58.796 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:58.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:58.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:43:58.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:58.937 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:58.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:58.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:59.039 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:59.067 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:59.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:59.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:59.069 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 49 16 87660 87644 111.755 79.875 0.00495166 0.00894363 2024-09-18T07:43:59.075 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:58 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:59.076 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:58 smithi137 ceph-mon[29408]: pgmap v682: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.26k op/s; 41911/252243 objects degraded (16.615%) 2024-09-18T07:43:59.076 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:43:58 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3749696836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:59.076 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:58 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:59.076 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:58 smithi137 ceph-mon[39175]: pgmap v682: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.26k op/s; 41911/252243 objects degraded (16.615%) 2024-09-18T07:43:59.076 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:43:58 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3749696836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:59.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T07:43:59.162 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:58 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:43:59.162 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:58 smithi203 ceph-mon[32237]: pgmap v682: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.26k op/s; 41911/252243 objects degraded (16.615%) 2024-09-18T07:43:59.162 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:43:58 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3749696836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:43:59.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:43:59.204 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:59.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:59.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:43:59.306 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:59.335 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:59.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:59.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:59.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:43:59.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:59.492 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:59.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:59.517 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:43:59.596 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:43:59.624 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:59.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:59.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:43:59.726 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:43:59.768 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:43:59.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:59.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:43:59.863 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:43:59.863 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:43:59.870 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:43:59.898 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:43:59.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:43:59.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:43:59.924 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:43:59.979 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:00.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:00.046 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:00.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:00.069 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 50 16 88877 88861 111.039 76.0625 0.00893483 0.00900146 2024-09-18T07:44:00.079 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:44:00.079 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:44:00.080 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-09-18T07:44:00.080 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:44:00.148 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:00.181 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:00.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:00.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:00.283 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:00.310 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:00.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:00.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:00.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:00.442 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:00.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:00.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:00.544 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:00.575 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:00.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:00.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:00.677 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:00.710 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:00.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:00.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:00.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:00.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:00.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:00.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:00.910 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T07:44:00.940 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:00 smithi203 ceph-mon[32237]: pgmap v683: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s; 42681/255744 objects degraded (16.689%) 2024-09-18T07:44:00.941 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:00 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 42681/255744 objects degraded (16.689%), 30 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:00.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:00.993 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:00.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:00.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:01.069 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 51 16 90065 90049 110.318 74.25 0.0125686 0.00905965 2024-09-18T07:44:01.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:01.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:01.136 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:01.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:01.162 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:00 smithi137 ceph-mon[29408]: pgmap v683: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s; 42681/255744 objects degraded (16.689%) 2024-09-18T07:44:01.163 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:00 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 42681/255744 objects degraded (16.689%), 30 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:01.163 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:00 smithi137 ceph-mon[39175]: pgmap v683: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s; 42681/255744 objects degraded (16.689%) 2024-09-18T07:44:01.163 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:00 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 42681/255744 objects degraded (16.689%), 30 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:01.168 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:44:01.171 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:44:01.237 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:01.255 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:44:01.269 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:01.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:01.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:01.371 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:01.401 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:01.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:01.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:01.503 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:01.532 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:01.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:01.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:01.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:01.676 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:01.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:01.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:01.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:01.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:01.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:01.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:01.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:44:01.914 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:01.925 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:44:01.944 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:01.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:01.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:02.017 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":683,"stamp":"2024-09-18T07:43:59.720747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5574421441,"num_objects":85248,"num_object_clones":0,"num_object_copies":255744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85248,"num_whiteouts":0,"num_read":1674,"num_read_kb":1436,"num_write":85781,"num_write_kb":5443329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":112965,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83902,"ondisk_log_size":83902,"up":405,"acting":405,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":469,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17444756,"kb_used_data":13534328,"kb_used_omap":12,"kb_used_meta":3910259,"kb_avail":732516460,"statfs":{"total":767960285184,"available":750096855040,"internally_reserved":0,"allocated":13859151872,"data_stored":13855869625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":4004105776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":824311808,"num_objects":12578,"num_object_clones":0,"num_object_copies":37734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12578,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12578,"num_write_kb":804992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005592"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":676,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.072337+0000","last_change":"2024-09-18T07:41:15.846355+0000","last_active":"2024-09-18T07:43:56.072337+0000","last_peered":"2024-09-18T07:43:56.072337+0000","last_clean":"2024-09-18T07:43:56.072337+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:43:56.072337+0000","last_undegraded":"2024-09-18T07:43:56.072337+0000","last_fullsized":"2024-09-18T07:43:56.072337+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:15.846257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:15.846257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:13.262548+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"323'15","reported_seq":683,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.069840+0000","last_change":"2024-09-18T07:42:54.748140+0000","last_active":"2024-09-18T07:43:56.069840+0000","last_peered":"2024-09-18T07:43:56.069840+0000","last_clean":"2024-09-18T07:41:04.168295+0000","last_became_active":"2024-09-18T07:42:54.748140+0000","last_became_peered":"2024-09-18T07:42:54.748140+0000","last_unstale":"2024-09-18T07:43:56.069840+0000","last_undegraded":"2024-09-18T07:42:54.735181+0000","last_fullsized":"2024-09-18T07:42:54.735072+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:17.269977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:17.269977+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:56.170969+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":619,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.067883+0000","last_change":"2024-09-18T07:41:58.378943+0000","last_active":"2024-09-18T07:43:56.067883+0000","last_peered":"2024-09-18T07:43:56.067883+0000","last_clean":"2024-09-18T07:43:56.067883+0000","last_became_active":"2024-09-18T07:36:13.128805+0000","last_became_peered":"2024-09-18T07:36:13.128805+0000","last_unstale":"2024-09-18T07:43:56.067883+0000","last_undegraded":"2024-09-18T07:43:56.067883+0000","last_fullsized":"2024-09-18T07:43:56.067883+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:58.378571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:58.378571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":43,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.065553+0000","last_change":"2024-09-18T07:42:54.740416+0000","last_active":"2024-09-18T07:43:56.065553+0000","last_peered":"2024-09-18T07:43:56.065553+0000","last_clean":"2024-09-18T07:31:01.319795+0000","last_became_active":"2024-09-18T07:42:54.740416+0000","last_became_peered":"2024-09-18T07:42:54.740416+0000","last_unstale":"2024-09-18T07:43:56.065553+0000","last_undegraded":"2024-09-18T07:43:56.065553+0000","last_fullsized":"2024-09-18T07:42:54.733242+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:47.286927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.286927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:29.659488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":570,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.082456+0000","last_change":"2024-09-18T07:41:12.872510+0000","last_active":"2024-09-18T07:43:56.082456+0000","last_peered":"2024-09-18T07:43:56.082456+0000","last_clean":"2024-09-18T07:43:56.082456+0000","last_became_active":"2024-09-18T07:37:43.056380+0000","last_became_peered":"2024-09-18T07:37:43.056380+0000","last_unstale":"2024-09-18T07:43:56.082456+0000","last_undegraded":"2024-09-18T07:43:56.082456+0000","last_fullsized":"2024-09-18T07:43:56.082456+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:12.872387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:12.872387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:14.272613+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":677,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.075582+0000","last_change":"2024-09-18T07:40:57.319270+0000","last_active":"2024-09-18T07:43:56.075582+0000","last_peered":"2024-09-18T07:43:56.075582+0000","last_clean":"2024-09-18T07:43:56.075582+0000","last_became_active":"2024-09-18T07:37:43.066508+0000","last_became_peered":"2024-09-18T07:37:43.066508+0000","last_unstale":"2024-09-18T07:43:56.075582+0000","last_undegraded":"2024-09-18T07:43:56.075582+0000","last_fullsized":"2024-09-18T07:43:56.075582+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:57.319204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:57.319204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:25.291687+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"323'27","reported_seq":229,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.066818+0000","last_change":"2024-09-18T07:42:46.881571+0000","last_active":"2024-09-18T07:43:56.066818+0000","last_peered":"2024-09-18T07:43:56.066818+0000","last_clean":"2024-09-18T07:43:56.066818+0000","last_became_active":"2024-09-18T07:40:19.064755+0000","last_became_peered":"2024-09-18T07:40:19.064755+0000","last_unstale":"2024-09-18T07:43:56.066818+0000","last_undegraded":"2024-09-18T07:43:56.066818+0000","last_fullsized":"2024-09-18T07:43:56.066818+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:42:46.881480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:46.881480+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:14.012195+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":423,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.070016+0000","last_change":"2024-09-18T07:42:54.739217+0000","last_active":"2024-09-18T07:43:56.070016+0000","last_peered":"2024-09-18T07:43:56.070016+0000","last_clean":"2024-09-18T07:37:38.986586+0000","last_became_active":"2024-09-18T07:42:54.739217+0000","last_became_peered":"2024-09-18T07:42:54.739217+0000","last_unstale":"2024-09-18T07:43:56.070016+0000","last_undegraded":"2024-09-18T07:43:56.070016+0000","last_fullsized":"2024-09-18T07:42:54.734594+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:51.001046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:51.001046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:25.147698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":687,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.067804+0000","last_change":"2024-09-18T07:42:54.744099+0000","last_active":"2024-09-18T07:43:56.067804+0000","last_peered":"2024-09-18T07:43:56.067804+0000","last_clean":"2024-09-18T07:42:51.721624+0000","last_became_active":"2024-09-18T07:42:54.744099+0000","last_became_peered":"2024-09-18T07:42:54.744099+0000","last_unstale":"2024-09-18T07:43:56.067804+0000","last_undegraded":"2024-09-18T07:43:56.067804+0000","last_fullsized":"2024-09-18T07:42:54.735114+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:41.502137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:41.502137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:07.071854+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"323'15","reported_seq":617,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.072907+0000","last_change":"2024-09-18T07:42:54.746959+0000","last_active":"2024-09-18T07:43:56.072907+0000","last_peered":"2024-09-18T07:43:56.072907+0000","last_clean":"2024-09-18T07:41:04.798944+0000","last_became_active":"2024-09-18T07:42:54.746959+0000","last_became_peered":"2024-09-18T07:42:54.746959+0000","last_unstale":"2024-09-18T07:43:56.072907+0000","last_undegraded":"2024-09-18T07:42:54.737030+0000","last_fullsized":"2024-09-18T07:42:54.736921+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:56.021118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:56.021118+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:32.256961+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":675,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.066201+0000","last_change":"2024-09-18T07:42:28.519199+0000","last_active":"2024-09-18T07:43:56.066201+0000","last_peered":"2024-09-18T07:43:56.066201+0000","last_clean":"2024-09-18T07:43:56.066201+0000","last_became_active":"2024-09-18T07:35:01.413443+0000","last_became_peered":"2024-09-18T07:35:01.413443+0000","last_unstale":"2024-09-18T07:43:56.066201+0000","last_undegraded":"2024-09-18T07:43:56.066201+0000","last_fullsized":"2024-09-18T07:43:56.066201+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:28.518968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:28.518968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":669,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.069939+0000","last_change":"2024-09-18T07:43:37.233745+0000","last_active":"2024-09-18T07:43:56.069939+0000","last_peered":"2024-09-18T07:43:56.069939+0000","last_clean":"2024-09-18T07:43:56.069939+0000","last_became_active":"2024-09-18T07:35:01.410284+0000","last_became_peered":"2024-09-18T07:35:01.410284+0000","last_unstale":"2024-09-18T07:43:56.069939+0000","last_undegraded":"2024-09-18T07:43:56.069939+0000","last_fullsized":"2024-09-18T07:43:56.069939+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:37.233589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:37.233589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.694863+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":665,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.065927+0000","last_change":"2024-09-18T07:42:54.745050+0000","last_active":"2024-09-18T07:43:56.065927+0000","last_peered":"2024-09-18T07:43:56.065927+0000","last_clean":"2024-09-18T07:42:51.721032+0000","last_became_active":"2024-09-18T07:42:54.745050+0000","last_became_peered":"2024-09-18T07:42:54.745050+0000","last_unstale":"2024-09-18T07:43:56.065927+0000","last_undegraded":"2024-09-18T07:43:56.065927+0000","last_fullsized":"2024-09-18T07:42:54.733513+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:57.358853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:57.358853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:38.529191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"323'7","reported_seq":618,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.065883+0000","last_change":"2024-09-18T07:42:54.741699+0000","last_active":"2024-09-18T07:43:56.065883+0000","last_peered":"2024-09-18T07:43:56.065883+0000","last_clean":"2024-09-18T07:41:04.680256+0000","last_became_active":"2024-09-18T07:42:54.741699+0000","last_became_peered":"2024-09-18T07:42:54.741699+0000","last_unstale":"2024-09-18T07:43:56.065883+0000","last_undegraded":"2024-09-18T07:42:54.737038+0000","last_fullsized":"2024-09-18T07:42:54.736846+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'7","last_scrub_stamp":"2024-09-18T07:42:46.496916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:46.496916+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:04.557910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":2}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":446,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.072195+0000","last_change":"2024-09-18T07:42:54.741021+0000","last_active":"2024-09-18T07:43:56.072195+0000","last_peered":"2024-09-18T07:43:56.072195+0000","last_clean":"2024-09-18T07:42:51.727903+0000","last_became_active":"2024-09-18T07:42:54.741021+0000","last_became_peered":"2024-09-18T07:42:54.741021+0000","last_unstale":"2024-09-18T07:43:56.072195+0000","last_undegraded":"2024-09-18T07:43:56.072195+0000","last_fullsized":"2024-09-18T07:42:54.738593+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:28.855481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:28.855481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:45.250315+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":681,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.068119+0000","last_change":"2024-09-18T07:43:01.522712+0000","last_active":"2024-09-18T07:43:56.068119+0000","last_peered":"2024-09-18T07:43:56.068119+0000","last_clean":"2024-09-18T07:43:56.068119+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:43:56.068119+0000","last_undegraded":"2024-09-18T07:43:56.068119+0000","last_fullsized":"2024-09-18T07:43:56.068119+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.522665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.522665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:20.744014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":420,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.072951+0000","last_change":"2024-09-18T07:42:54.743937+0000","last_active":"2024-09-18T07:43:56.072951+0000","last_peered":"2024-09-18T07:43:56.072951+0000","last_clean":"2024-09-18T07:37:38.987411+0000","last_became_active":"2024-09-18T07:42:54.743937+0000","last_became_peered":"2024-09-18T07:42:54.743937+0000","last_unstale":"2024-09-18T07:43:56.072951+0000","last_undegraded":"2024-09-18T07:43:56.072951+0000","last_fullsized":"2024-09-18T07:42:54.735028+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:40.432906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:40.432906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:56.952811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":685,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.083233+0000","last_change":"2024-09-18T07:42:54.742535+0000","last_active":"2024-09-18T07:43:56.083233+0000","last_peered":"2024-09-18T07:43:56.083233+0000","last_clean":"2024-09-18T07:42:51.718566+0000","last_became_active":"2024-09-18T07:42:54.742535+0000","last_became_peered":"2024-09-18T07:42:54.742535+0000","last_unstale":"2024-09-18T07:43:56.083233+0000","last_undegraded":"2024-09-18T07:43:56.083233+0000","last_fullsized":"2024-09-18T07:42:54.740869+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:04.117653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.117653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:27.227076+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"323'27","reported_seq":764,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.065467+0000","last_change":"2024-09-18T07:41:48.128836+0000","last_active":"2024-09-18T07:43:56.065467+0000","last_peered":"2024-09-18T07:43:56.065467+0000","last_clean":"2024-09-18T07:43:56.065467+0000","last_became_active":"2024-09-18T07:37:28.002858+0000","last_became_peered":"2024-09-18T07:37:28.002858+0000","last_unstale":"2024-09-18T07:43:56.065467+0000","last_undegraded":"2024-09-18T07:43:56.065467+0000","last_fullsized":"2024-09-18T07:43:56.065467+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:41:48.128756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.128756+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:13.904366+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":44,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.075355+0000","last_change":"2024-09-18T07:42:54.751231+0000","last_active":"2024-09-18T07:43:56.075355+0000","last_peered":"2024-09-18T07:43:56.075355+0000","last_clean":"2024-09-18T07:30:59.305166+0000","last_became_active":"2024-09-18T07:42:54.751231+0000","last_became_peered":"2024-09-18T07:42:54.751231+0000","last_unstale":"2024-09-18T07:43:56.075355+0000","last_undegraded":"2024-09-18T07:43:56.075355+0000","last_fullsized":"2024-09-18T07:42:54.737194+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:26.922506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:26.922506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:21.316063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":535,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.073664+0000","last_change":"2024-09-18T07:42:57.174875+0000","last_active":"2024-09-18T07:43:56.073664+0000","last_peered":"2024-09-18T07:43:56.073664+0000","last_clean":"2024-09-18T07:43:56.073664+0000","last_became_active":"2024-09-18T07:40:19.070463+0000","last_became_peered":"2024-09-18T07:40:19.070463+0000","last_unstale":"2024-09-18T07:43:56.073664+0000","last_undegraded":"2024-09-18T07:43:56.073664+0000","last_fullsized":"2024-09-18T07:43:56.073664+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:57.174827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:57.174827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:23.943515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":42,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.065872+0000","last_change":"2024-09-18T07:42:54.751631+0000","last_active":"2024-09-18T07:43:56.065872+0000","last_peered":"2024-09-18T07:43:56.065872+0000","last_clean":"2024-09-18T07:30:55.271698+0000","last_became_active":"2024-09-18T07:42:54.751631+0000","last_became_peered":"2024-09-18T07:42:54.751631+0000","last_unstale":"2024-09-18T07:43:56.065872+0000","last_undegraded":"2024-09-18T07:43:56.065872+0000","last_fullsized":"2024-09-18T07:42:54.743356+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:31.323963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:31.323963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:41.096756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"323'18","reported_seq":337,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.066296+0000","last_change":"2024-09-18T07:42:54.742187+0000","last_active":"2024-09-18T07:43:56.066296+0000","last_peered":"2024-09-18T07:43:56.066296+0000","last_clean":"2024-09-18T07:42:51.717553+0000","last_became_active":"2024-09-18T07:42:54.742187+0000","last_became_peered":"2024-09-18T07:42:54.742187+0000","last_unstale":"2024-09-18T07:43:56.066296+0000","last_undegraded":"2024-09-18T07:42:54.738159+0000","last_fullsized":"2024-09-18T07:42:54.735312+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:41:59.149629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:59.149629+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:01.900391+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":42,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.076582+0000","last_change":"2024-09-18T07:42:54.753848+0000","last_active":"2024-09-18T07:43:56.076582+0000","last_peered":"2024-09-18T07:43:56.076582+0000","last_clean":"2024-09-18T07:30:59.305166+0000","last_became_active":"2024-09-18T07:42:54.753848+0000","last_became_peered":"2024-09-18T07:42:54.753848+0000","last_unstale":"2024-09-18T07:43:56.076582+0000","last_undegraded":"2024-09-18T07:43:56.076582+0000","last_fullsized":"2024-09-18T07:42:54.743099+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.930435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.930435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:56.271747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":446,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.074318+0000","last_change":"2024-09-18T07:40:10.644587+0000","last_active":"2024-09-18T07:43:56.074318+0000","last_peered":"2024-09-18T07:43:56.074318+0000","last_clean":"2024-09-18T07:43:56.074318+0000","last_became_active":"2024-09-18T07:35:01.413110+0000","last_became_peered":"2024-09-18T07:35:01.413110+0000","last_unstale":"2024-09-18T07:43:56.074318+0000","last_undegraded":"2024-09-18T07:43:56.074318+0000","last_fullsized":"2024-09-18T07:43:56.074318+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:40:10.644461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:10.644461+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:16.109693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":375,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.076180+0000","last_change":"2024-09-18T07:42:54.753800+0000","last_active":"2024-09-18T07:43:56.076180+0000","last_peered":"2024-09-18T07:43:56.076180+0000","last_clean":"2024-09-18T07:42:51.723949+0000","last_became_active":"2024-09-18T07:42:54.753800+0000","last_became_peered":"2024-09-18T07:42:54.753800+0000","last_unstale":"2024-09-18T07:43:56.076180+0000","last_undegraded":"2024-09-18T07:42:54.745541+0000","last_fullsized":"2024-09-18T07:42:54.745455+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:42:42.332950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:42.332950+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"323'20","reported_seq":748,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.072589+0000","last_change":"2024-09-18T07:43:46.102137+0000","last_active":"2024-09-18T07:43:56.072589+0000","last_peered":"2024-09-18T07:43:56.072589+0000","last_clean":"2024-09-18T07:43:56.072589+0000","last_became_active":"2024-09-18T07:35:01.410926+0000","last_became_peered":"2024-09-18T07:35:01.410926+0000","last_unstale":"2024-09-18T07:43:56.072589+0000","last_undegraded":"2024-09-18T07:43:56.072589+0000","last_fullsized":"2024-09-18T07:43:56.072589+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:43:46.102041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:43:46.102041+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:13.396740+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"360'13","reported_seq":385,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.073934+0000","last_change":"2024-09-18T07:42:54.745765+0000","last_active":"2024-09-18T07:43:56.073934+0000","last_peered":"2024-09-18T07:43:56.073934+0000","last_clean":"2024-09-18T07:42:51.730678+0000","last_became_active":"2024-09-18T07:42:54.745765+0000","last_became_peered":"2024-09-18T07:42:54.745765+0000","last_unstale":"2024-09-18T07:43:56.073934+0000","last_undegraded":"2024-09-18T07:42:54.736184+0000","last_fullsized":"2024-09-18T07:42:54.736047+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:40:43.112459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:43.112459+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:04.133303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"376'5236","reported_seq":5268,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.691421+0000","last_change":"2024-09-18T07:43:10.058721+0000","last_active":"2024-09-18T07:43:56.691421+0000","last_peered":"2024-09-18T07:43:56.691421+0000","last_clean":"2024-09-18T07:43:01.032791+0000","last_became_active":"2024-09-18T07:43:02.045039+0000","last_became_peered":"2024-09-18T07:43:02.045039+0000","last_unstale":"2024-09-18T07:43:56.691421+0000","last_undegraded":"2024-09-18T07:43:10.058195+0000","last_fullsized":"2024-09-18T07:43:02.038669+0000","mapping_epoch":364,"log_start":"367'400","ondisk_log_start":"367'400","created":364,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":4836,"log_dups_size":0,"ondisk_log_size":4836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:12.998393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343146496,"num_objects":5236,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5236,"num_write_kb":335104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":5236}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":240,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.065453+0000","last_change":"2024-09-18T07:42:29.515462+0000","last_active":"2024-09-18T07:43:56.065453+0000","last_peered":"2024-09-18T07:43:56.065453+0000","last_clean":"2024-09-18T07:43:56.065453+0000","last_became_active":"2024-09-18T07:39:04.962047+0000","last_became_peered":"2024-09-18T07:39:04.962047+0000","last_unstale":"2024-09-18T07:43:56.065453+0000","last_undegraded":"2024-09-18T07:43:56.065453+0000","last_fullsized":"2024-09-18T07:43:56.065453+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:42:29.515416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:42:29.515416+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"323'27","reported_seq":429,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.076332+0000","last_change":"2024-09-18T07:42:54.746764+0000","last_active":"2024-09-18T07:43:56.076332+0000","last_peered":"2024-09-18T07:43:56.076332+0000","last_clean":"2024-09-18T07:42:51.730768+0000","last_became_active":"2024-09-18T07:42:54.746764+0000","last_became_peered":"2024-09-18T07:42:54.746764+0000","last_unstale":"2024-09-18T07:43:56.076332+0000","last_undegraded":"2024-09-18T07:42:54.742352+0000","last_fullsized":"2024-09-18T07:42:54.742252+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:42:33.391715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:33.391715+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:59.764580+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":693,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.076115+0000","last_change":"2024-09-18T07:39:29.049261+0000","last_active":"2024-09-18T07:43:56.076115+0000","last_peered":"2024-09-18T07:43:56.076115+0000","last_clean":"2024-09-18T07:43:56.076115+0000","last_became_active":"2024-09-18T07:35:01.407339+0000","last_became_peered":"2024-09-18T07:35:01.407339+0000","last_unstale":"2024-09-18T07:43:56.076115+0000","last_undegraded":"2024-09-18T07:43:56.076115+0000","last_fullsized":"2024-09-18T07:43:56.076115+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.049209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.049209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:23.347291+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"360'9","reported_seq":82,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.066200+0000","last_change":"2024-09-18T07:42:54.740567+0000","last_active":"2024-09-18T07:43:56.066200+0000","last_peered":"2024-09-18T07:43:56.066200+0000","last_clean":"2024-09-18T07:31:03.519590+0000","last_became_active":"2024-09-18T07:42:54.740567+0000","last_became_peered":"2024-09-18T07:42:54.740567+0000","last_unstale":"2024-09-18T07:43:56.066200+0000","last_undegraded":"2024-09-18T07:42:54.734181+0000","last_fullsized":"2024-09-18T07:42:54.734092+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:41:27.965205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.965205+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:35.595508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":657,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.067230+0000","last_change":"2024-09-18T07:42:06.507229+0000","last_active":"2024-09-18T07:43:56.067230+0000","last_peered":"2024-09-18T07:43:56.067230+0000","last_clean":"2024-09-18T07:43:56.067230+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:43:56.067230+0000","last_undegraded":"2024-09-18T07:43:56.067230+0000","last_fullsized":"2024-09-18T07:43:56.067230+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.507166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.507166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"376'5297","reported_seq":5329,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.761991+0000","last_change":"2024-09-18T07:43:10.060552+0000","last_active":"2024-09-18T07:43:56.761991+0000","last_peered":"2024-09-18T07:43:56.761991+0000","last_clean":"2024-09-18T07:43:01.032791+0000","last_became_active":"2024-09-18T07:43:02.046628+0000","last_became_peered":"2024-09-18T07:43:02.046628+0000","last_unstale":"2024-09-18T07:43:56.761991+0000","last_undegraded":"2024-09-18T07:43:10.053665+0000","last_fullsized":"2024-09-18T07:43:02.043071+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":364,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5297,"log_dups_size":0,"ondisk_log_size":5297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:27.025706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347144192,"num_objects":5297,"num_object_clones":0,"num_object_copies":15891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5297,"num_write_kb":339008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":5297}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":43,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.066110+0000","last_change":"2024-09-18T07:42:54.748936+0000","last_active":"2024-09-18T07:43:56.066110+0000","last_peered":"2024-09-18T07:43:56.066110+0000","last_clean":"2024-09-18T07:34:59.364705+0000","last_became_active":"2024-09-18T07:42:54.748936+0000","last_became_peered":"2024-09-18T07:42:54.748936+0000","last_unstale":"2024-09-18T07:43:56.066110+0000","last_undegraded":"2024-09-18T07:43:56.066110+0000","last_fullsized":"2024-09-18T07:42:54.738388+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:41:53.004186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:41:53.004186+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T07:44:55.094617+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"323'30","reported_seq":767,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.074991+0000","last_change":"2024-09-18T07:39:00.919075+0000","last_active":"2024-09-18T07:43:56.074991+0000","last_peered":"2024-09-18T07:43:56.074991+0000","last_clean":"2024-09-18T07:43:56.074991+0000","last_became_active":"2024-09-18T07:35:01.418909+0000","last_became_peered":"2024-09-18T07:35:01.418909+0000","last_unstale":"2024-09-18T07:43:56.074991+0000","last_undegraded":"2024-09-18T07:43:56.074991+0000","last_fullsized":"2024-09-18T07:43:56.074991+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-18T07:39:00.918860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:39:00.918860+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":405,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.074793+0000","last_change":"2024-09-18T07:42:39.814649+0000","last_active":"2024-09-18T07:43:56.074793+0000","last_peered":"2024-09-18T07:43:56.074793+0000","last_clean":"2024-09-18T07:43:56.074793+0000","last_became_active":"2024-09-18T07:40:19.068296+0000","last_became_peered":"2024-09-18T07:40:19.068296+0000","last_unstale":"2024-09-18T07:43:56.074793+0000","last_undegraded":"2024-09-18T07:43:56.074793+0000","last_fullsized":"2024-09-18T07:43:56.074793+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:39.814582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:39.814582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:59.295469+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":283,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.074836+0000","last_change":"2024-09-18T07:42:54.753453+0000","last_active":"2024-09-18T07:43:56.074836+0000","last_peered":"2024-09-18T07:43:56.074836+0000","last_clean":"2024-09-18T07:34:59.364667+0000","last_became_active":"2024-09-18T07:42:54.753453+0000","last_became_peered":"2024-09-18T07:42:54.753453+0000","last_unstale":"2024-09-18T07:43:56.074836+0000","last_undegraded":"2024-09-18T07:43:56.074836+0000","last_fullsized":"2024-09-18T07:42:54.746156+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:09.067941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:09.067941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:31.334423+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":43,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.074755+0000","last_change":"2024-09-18T07:42:54.755423+0000","last_active":"2024-09-18T07:43:56.074755+0000","last_peered":"2024-09-18T07:43:56.074755+0000","last_clean":"2024-09-18T07:31:01.319795+0000","last_became_active":"2024-09-18T07:42:54.755423+0000","last_became_peered":"2024-09-18T07:42:54.755423+0000","last_unstale":"2024-09-18T07:43:56.074755+0000","last_undegraded":"2024-09-18T07:43:56.074755+0000","last_fullsized":"2024-09-18T07:42:54.740006+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:44.219547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:44.219547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:05.331595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"376'5233","reported_seq":5268,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.784852+0000","last_change":"2024-09-18T07:43:02.069313+0000","last_active":"2024-09-18T07:43:56.784852+0000","last_peered":"2024-09-18T07:43:56.784852+0000","last_clean":"2024-09-18T07:43:56.784852+0000","last_became_active":"2024-09-18T07:43:02.068957+0000","last_became_peered":"2024-09-18T07:43:02.068957+0000","last_unstale":"2024-09-18T07:43:56.784852+0000","last_undegraded":"2024-09-18T07:43:56.784852+0000","last_fullsized":"2024-09-18T07:43:56.784852+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":364,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5233,"log_dups_size":0,"ondisk_log_size":5233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:12.469916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342949888,"num_objects":5233,"num_object_clones":0,"num_object_copies":15699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5233,"num_write_kb":334912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":237,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.074934+0000","last_change":"2024-09-18T07:41:19.479946+0000","last_active":"2024-09-18T07:43:56.074934+0000","last_peered":"2024-09-18T07:43:56.074934+0000","last_clean":"2024-09-18T07:43:56.074934+0000","last_became_active":"2024-09-18T07:39:04.945385+0000","last_became_peered":"2024-09-18T07:39:04.945385+0000","last_unstale":"2024-09-18T07:43:56.074934+0000","last_undegraded":"2024-09-18T07:43:56.074934+0000","last_fullsized":"2024-09-18T07:43:56.074934+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:41:19.479442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:41:19.479442+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:18.360628+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"323'28","reported_seq":680,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.068215+0000","last_change":"2024-09-18T07:42:54.739669+0000","last_active":"2024-09-18T07:43:56.068215+0000","last_peered":"2024-09-18T07:43:56.068215+0000","last_clean":"2024-09-18T07:41:05.289656+0000","last_became_active":"2024-09-18T07:42:54.739669+0000","last_became_peered":"2024-09-18T07:42:54.739669+0000","last_unstale":"2024-09-18T07:43:56.068215+0000","last_undegraded":"2024-09-18T07:42:54.733414+0000","last_fullsized":"2024-09-18T07:42:54.733291+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-18T07:40:56.183802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.183802+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:16.528654+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":665,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.072872+0000","last_change":"2024-09-18T07:42:54.751066+0000","last_active":"2024-09-18T07:43:56.072872+0000","last_peered":"2024-09-18T07:43:56.072872+0000","last_clean":"2024-09-18T07:42:51.718806+0000","last_became_active":"2024-09-18T07:42:54.751066+0000","last_became_peered":"2024-09-18T07:42:54.751066+0000","last_unstale":"2024-09-18T07:43:56.072872+0000","last_undegraded":"2024-09-18T07:42:54.738145+0000","last_fullsized":"2024-09-18T07:42:54.737872+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:39:28.750720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.750720+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:57.053150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"360'9","reported_seq":833,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.074820+0000","last_change":"2024-09-18T07:42:54.746312+0000","last_active":"2024-09-18T07:43:56.074820+0000","last_peered":"2024-09-18T07:43:56.074820+0000","last_clean":"2024-09-18T07:42:51.726800+0000","last_became_active":"2024-09-18T07:42:54.746312+0000","last_became_peered":"2024-09-18T07:42:54.746312+0000","last_unstale":"2024-09-18T07:43:56.074820+0000","last_undegraded":"2024-09-18T07:42:54.742119+0000","last_fullsized":"2024-09-18T07:42:54.742013+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:42:04.428395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.428395+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:28.345161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":457,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.072943+0000","last_change":"2024-09-18T07:42:45.886132+0000","last_active":"2024-09-18T07:43:56.072943+0000","last_peered":"2024-09-18T07:43:56.072943+0000","last_clean":"2024-09-18T07:43:56.072943+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:43:56.072943+0000","last_undegraded":"2024-09-18T07:43:56.072943+0000","last_fullsized":"2024-09-18T07:43:56.072943+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:45.886069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:45.886069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"376'5317","reported_seq":5349,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.698201+0000","last_change":"2024-09-18T07:43:10.054942+0000","last_active":"2024-09-18T07:43:56.698201+0000","last_peered":"2024-09-18T07:43:56.698201+0000","last_clean":"2024-09-18T07:43:01.032791+0000","last_became_active":"2024-09-18T07:43:02.044295+0000","last_became_peered":"2024-09-18T07:43:02.044295+0000","last_unstale":"2024-09-18T07:43:56.698201+0000","last_undegraded":"2024-09-18T07:43:10.054785+0000","last_fullsized":"2024-09-18T07:43:02.039081+0000","mapping_epoch":364,"log_start":"368'500","ondisk_log_start":"368'500","created":364,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":4817,"log_dups_size":0,"ondisk_log_size":4817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:29.211647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348454912,"num_objects":5317,"num_object_clones":0,"num_object_copies":15951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5317,"num_write_kb":340288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":5317}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":399,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.073521+0000","last_change":"2024-09-18T07:40:38.066508+0000","last_active":"2024-09-18T07:43:56.073521+0000","last_peered":"2024-09-18T07:43:56.073521+0000","last_clean":"2024-09-18T07:43:56.073521+0000","last_became_active":"2024-09-18T07:39:04.941752+0000","last_became_peered":"2024-09-18T07:39:04.941752+0000","last_unstale":"2024-09-18T07:43:56.073521+0000","last_undegraded":"2024-09-18T07:43:56.073521+0000","last_fullsized":"2024-09-18T07:43:56.073521+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:38.066443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.066443+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:20.222095+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"323'20","reported_seq":416,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.066724+0000","last_change":"2024-09-18T07:42:54.742459+0000","last_active":"2024-09-18T07:43:56.066724+0000","last_peered":"2024-09-18T07:43:56.066724+0000","last_clean":"2024-09-18T07:42:51.721094+0000","last_became_active":"2024-09-18T07:42:54.742459+0000","last_became_peered":"2024-09-18T07:42:54.742459+0000","last_unstale":"2024-09-18T07:43:56.066724+0000","last_undegraded":"2024-09-18T07:42:54.733700+0000","last_fullsized":"2024-09-18T07:42:54.733580+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:42:03.427843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:03.427843+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":678,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.067994+0000","last_change":"2024-09-18T07:40:53.350083+0000","last_active":"2024-09-18T07:43:56.067994+0000","last_peered":"2024-09-18T07:43:56.067994+0000","last_clean":"2024-09-18T07:43:56.067994+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:43:56.067994+0000","last_undegraded":"2024-09-18T07:43:56.067994+0000","last_fullsized":"2024-09-18T07:43:56.067994+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:53.350019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:53.350019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:19.601020+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"240'10","reported_seq":847,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.082701+0000","last_change":"2024-09-18T07:40:50.839710+0000","last_active":"2024-09-18T07:43:56.082701+0000","last_peered":"2024-09-18T07:43:56.082701+0000","last_clean":"2024-09-18T07:43:56.082701+0000","last_became_active":"2024-09-18T07:37:43.167850+0000","last_became_peered":"2024-09-18T07:37:43.167850+0000","last_unstale":"2024-09-18T07:43:56.082701+0000","last_undegraded":"2024-09-18T07:43:56.082701+0000","last_fullsized":"2024-09-18T07:43:56.082701+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'10","last_scrub_stamp":"2024-09-18T07:40:50.839504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.839504+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:24.293623+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":648,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.070077+0000","last_change":"2024-09-18T07:42:42.223288+0000","last_active":"2024-09-18T07:43:56.070077+0000","last_peered":"2024-09-18T07:43:56.070077+0000","last_clean":"2024-09-18T07:43:56.070077+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:43:56.070077+0000","last_undegraded":"2024-09-18T07:43:56.070077+0000","last_fullsized":"2024-09-18T07:43:56.070077+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:42.223235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:42.223235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:04.908111+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"376'5319","reported_seq":5350,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.787781+0000","last_change":"2024-09-18T07:43:10.055395+0000","last_active":"2024-09-18T07:43:56.787781+0000","last_peered":"2024-09-18T07:43:56.787781+0000","last_clean":"2024-09-18T07:43:01.032791+0000","last_became_active":"2024-09-18T07:43:02.049117+0000","last_became_peered":"2024-09-18T07:43:02.049117+0000","last_unstale":"2024-09-18T07:43:56.787781+0000","last_undegraded":"2024-09-18T07:43:10.054743+0000","last_fullsized":"2024-09-18T07:43:02.044732+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":364,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5319,"log_dups_size":0,"ondisk_log_size":5319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:28.071161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348585984,"num_objects":5319,"num_object_clones":0,"num_object_copies":15957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":340416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":5319}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":399,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.072803+0000","last_change":"2024-09-18T07:40:18.954153+0000","last_active":"2024-09-18T07:43:56.072803+0000","last_peered":"2024-09-18T07:43:56.072803+0000","last_clean":"2024-09-18T07:43:56.072803+0000","last_became_active":"2024-09-18T07:39:04.948571+0000","last_became_peered":"2024-09-18T07:39:04.948571+0000","last_unstale":"2024-09-18T07:43:56.072803+0000","last_undegraded":"2024-09-18T07:43:56.072803+0000","last_fullsized":"2024-09-18T07:43:56.072803+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:18.954089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:18.954089+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:53.139734+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"323'29","reported_seq":773,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.082921+0000","last_change":"2024-09-18T07:41:16.895792+0000","last_active":"2024-09-18T07:43:56.082921+0000","last_peered":"2024-09-18T07:43:56.082921+0000","last_clean":"2024-09-18T07:43:56.082921+0000","last_became_active":"2024-09-18T07:35:01.415275+0000","last_became_peered":"2024-09-18T07:35:01.415275+0000","last_unstale":"2024-09-18T07:43:56.082921+0000","last_undegraded":"2024-09-18T07:43:56.082921+0000","last_fullsized":"2024-09-18T07:43:56.082921+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'29","last_scrub_stamp":"2024-09-18T07:41:16.895595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:16.895595+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:12.237471+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":362,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.065556+0000","last_change":"2024-09-18T07:42:31.793848+0000","last_active":"2024-09-18T07:43:56.065556+0000","last_peered":"2024-09-18T07:43:56.065556+0000","last_clean":"2024-09-18T07:43:56.065556+0000","last_became_active":"2024-09-18T07:36:13.118553+0000","last_became_peered":"2024-09-18T07:36:13.118553+0000","last_unstale":"2024-09-18T07:43:56.065556+0000","last_undegraded":"2024-09-18T07:43:56.065556+0000","last_fullsized":"2024-09-18T07:43:56.065556+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:31.793760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:31.793760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:58.491919+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":309,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.072097+0000","last_change":"2024-09-18T07:43:11.108075+0000","last_active":"2024-09-18T07:43:56.072097+0000","last_peered":"2024-09-18T07:43:56.072097+0000","last_clean":"2024-09-18T07:43:56.072097+0000","last_became_active":"2024-09-18T07:37:43.070009+0000","last_became_peered":"2024-09-18T07:37:43.070009+0000","last_unstale":"2024-09-18T07:43:56.072097+0000","last_undegraded":"2024-09-18T07:43:56.072097+0000","last_fullsized":"2024-09-18T07:43:56.072097+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:11.108021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:43:11.108021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:31.946331+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":301,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.069852+0000","last_change":"2024-09-18T07:42:47.221767+0000","last_active":"2024-09-18T07:43:56.069852+0000","last_peered":"2024-09-18T07:43:56.069852+0000","last_clean":"2024-09-18T07:43:56.069852+0000","last_became_active":"2024-09-18T07:37:43.051740+0000","last_became_peered":"2024-09-18T07:37:43.051740+0000","last_unstale":"2024-09-18T07:43:56.069852+0000","last_undegraded":"2024-09-18T07:43:56.069852+0000","last_fullsized":"2024-09-18T07:43:56.069852+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:47.221690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:47.221690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:16.188899+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"376'5294","reported_seq":5329,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.791558+0000","last_change":"2024-09-18T07:43:02.048403+0000","last_active":"2024-09-18T07:43:56.791558+0000","last_peered":"2024-09-18T07:43:56.791558+0000","last_clean":"2024-09-18T07:43:56.791558+0000","last_became_active":"2024-09-18T07:43:02.048124+0000","last_became_peered":"2024-09-18T07:43:02.048124+0000","last_unstale":"2024-09-18T07:43:56.791558+0000","last_undegraded":"2024-09-18T07:43:56.791558+0000","last_fullsized":"2024-09-18T07:43:56.791558+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":364,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5294,"log_dups_size":0,"ondisk_log_size":5294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:29.293888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346947584,"num_objects":5294,"num_object_clones":0,"num_object_copies":15882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5294,"num_write_kb":338816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":232,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.067642+0000","last_change":"2024-09-18T07:40:43.346516+0000","last_active":"2024-09-18T07:43:56.067642+0000","last_peered":"2024-09-18T07:43:56.067642+0000","last_clean":"2024-09-18T07:43:56.067642+0000","last_became_active":"2024-09-18T07:39:04.950564+0000","last_became_peered":"2024-09-18T07:39:04.950564+0000","last_unstale":"2024-09-18T07:43:56.067642+0000","last_undegraded":"2024-09-18T07:43:56.067642+0000","last_fullsized":"2024-09-18T07:43:56.067642+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:43.346447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:43.346447+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:24.649295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"323'41","reported_seq":559,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.065419+0000","last_change":"2024-09-18T07:43:11.982067+0000","last_active":"2024-09-18T07:43:56.065419+0000","last_peered":"2024-09-18T07:43:56.065419+0000","last_clean":"2024-09-18T07:43:56.065419+0000","last_became_active":"2024-09-18T07:36:13.115278+0000","last_became_peered":"2024-09-18T07:36:13.115278+0000","last_unstale":"2024-09-18T07:43:56.065419+0000","last_undegraded":"2024-09-18T07:43:56.065419+0000","last_fullsized":"2024-09-18T07:43:56.065419+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'41","last_scrub_stamp":"2024-09-18T07:43:11.981958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:43:11.981958+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:34.511250+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":425,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.083089+0000","last_change":"2024-09-18T07:42:54.748899+0000","last_active":"2024-09-18T07:43:56.083089+0000","last_peered":"2024-09-18T07:43:56.083089+0000","last_clean":"2024-09-18T07:37:38.987203+0000","last_became_active":"2024-09-18T07:42:54.748899+0000","last_became_peered":"2024-09-18T07:42:54.748899+0000","last_unstale":"2024-09-18T07:43:56.083089+0000","last_undegraded":"2024-09-18T07:43:56.083089+0000","last_fullsized":"2024-09-18T07:42:54.740370+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:03.028558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:03.028558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:43.093313+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":702,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.067060+0000","last_change":"2024-09-18T07:42:00.366508+0000","last_active":"2024-09-18T07:43:56.067060+0000","last_peered":"2024-09-18T07:43:56.067060+0000","last_clean":"2024-09-18T07:43:56.067060+0000","last_became_active":"2024-09-18T07:37:43.076595+0000","last_became_peered":"2024-09-18T07:37:43.076595+0000","last_unstale":"2024-09-18T07:43:56.067060+0000","last_undegraded":"2024-09-18T07:43:56.067060+0000","last_fullsized":"2024-09-18T07:43:56.067060+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:42:00.366441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:42:00.366441+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":656,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.075582+0000","last_change":"2024-09-18T07:42:54.751453+0000","last_active":"2024-09-18T07:43:56.075582+0000","last_peered":"2024-09-18T07:43:56.075582+0000","last_clean":"2024-09-18T07:42:51.719583+0000","last_became_active":"2024-09-18T07:42:54.751453+0000","last_became_peered":"2024-09-18T07:42:54.751453+0000","last_unstale":"2024-09-18T07:43:56.075582+0000","last_undegraded":"2024-09-18T07:43:56.075582+0000","last_fullsized":"2024-09-18T07:42:54.737547+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:43.200013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:43.200013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:10.876228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":676,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.074760+0000","last_change":"2024-09-18T07:42:54.750128+0000","last_active":"2024-09-18T07:43:56.074760+0000","last_peered":"2024-09-18T07:43:56.074760+0000","last_clean":"2024-09-18T07:42:51.722974+0000","last_became_active":"2024-09-18T07:42:54.750128+0000","last_became_peered":"2024-09-18T07:42:54.750128+0000","last_unstale":"2024-09-18T07:43:56.074760+0000","last_undegraded":"2024-09-18T07:43:56.074760+0000","last_fullsized":"2024-09-18T07:42:54.741140+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:05.858804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:05.858804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:59.284816+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"376'5344","reported_seq":5379,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:57.119716+0000","last_change":"2024-09-18T07:43:02.050368+0000","last_active":"2024-09-18T07:43:57.119716+0000","last_peered":"2024-09-18T07:43:57.119716+0000","last_clean":"2024-09-18T07:43:57.119716+0000","last_became_active":"2024-09-18T07:43:02.049845+0000","last_became_peered":"2024-09-18T07:43:02.049845+0000","last_unstale":"2024-09-18T07:43:57.119716+0000","last_undegraded":"2024-09-18T07:43:57.119716+0000","last_fullsized":"2024-09-18T07:43:57.119716+0000","mapping_epoch":364,"log_start":"367'300","ondisk_log_start":"367'300","created":364,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5044,"log_dups_size":0,"ondisk_log_size":5044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:08.632035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350224384,"num_objects":5344,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5344,"num_write_kb":342016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":503,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.071203+0000","last_change":"2024-09-18T07:42:54.749373+0000","last_active":"2024-09-18T07:43:56.071203+0000","last_peered":"2024-09-18T07:43:56.071203+0000","last_clean":"2024-09-18T07:42:51.727693+0000","last_became_active":"2024-09-18T07:42:54.749373+0000","last_became_peered":"2024-09-18T07:42:54.749373+0000","last_unstale":"2024-09-18T07:43:56.071203+0000","last_undegraded":"2024-09-18T07:43:56.071203+0000","last_fullsized":"2024-09-18T07:42:54.738094+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:44.476278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:44.476278+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:02.909771+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"323'17","reported_seq":423,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.067510+0000","last_change":"2024-09-18T07:43:40.533349+0000","last_active":"2024-09-18T07:43:56.067510+0000","last_peered":"2024-09-18T07:43:56.067510+0000","last_clean":"2024-09-18T07:43:56.067510+0000","last_became_active":"2024-09-18T07:36:13.117924+0000","last_became_peered":"2024-09-18T07:36:13.117924+0000","last_unstale":"2024-09-18T07:43:56.067510+0000","last_undegraded":"2024-09-18T07:43:56.067510+0000","last_fullsized":"2024-09-18T07:43:56.067510+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'17","last_scrub_stamp":"2024-09-18T07:43:40.533240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:43:40.533240+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:48.046541+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":384,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.064969+0000","last_change":"2024-09-18T07:42:02.127909+0000","last_active":"2024-09-18T07:43:56.064969+0000","last_peered":"2024-09-18T07:43:56.064969+0000","last_clean":"2024-09-18T07:43:56.064969+0000","last_became_active":"2024-09-18T07:37:43.050971+0000","last_became_peered":"2024-09-18T07:37:43.050971+0000","last_unstale":"2024-09-18T07:43:56.064969+0000","last_undegraded":"2024-09-18T07:43:56.064969+0000","last_fullsized":"2024-09-18T07:43:56.064969+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:02.127517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:02.127517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:17.949081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":689,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.066915+0000","last_change":"2024-09-18T07:43:01.065282+0000","last_active":"2024-09-18T07:43:56.066915+0000","last_peered":"2024-09-18T07:43:56.066915+0000","last_clean":"2024-09-18T07:43:56.066915+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:43:56.066915+0000","last_undegraded":"2024-09-18T07:43:56.066915+0000","last_fullsized":"2024-09-18T07:43:56.066915+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:43:01.065193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.065193+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:20.807951+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":685,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.065858+0000","last_change":"2024-09-18T07:43:40.066467+0000","last_active":"2024-09-18T07:43:56.065858+0000","last_peered":"2024-09-18T07:43:56.065858+0000","last_clean":"2024-09-18T07:43:56.065858+0000","last_became_active":"2024-09-18T07:35:01.403680+0000","last_became_peered":"2024-09-18T07:35:01.403680+0000","last_unstale":"2024-09-18T07:43:56.065858+0000","last_undegraded":"2024-09-18T07:43:56.065858+0000","last_fullsized":"2024-09-18T07:43:56.065858+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:40.066356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:43:40.066356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:45.038610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":453,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.073535+0000","last_change":"2024-09-18T07:40:54.574254+0000","last_active":"2024-09-18T07:43:56.073535+0000","last_peered":"2024-09-18T07:43:56.073535+0000","last_clean":"2024-09-18T07:43:56.073535+0000","last_became_active":"2024-09-18T07:37:43.060019+0000","last_became_peered":"2024-09-18T07:37:43.060019+0000","last_unstale":"2024-09-18T07:43:56.073535+0000","last_undegraded":"2024-09-18T07:43:56.073535+0000","last_fullsized":"2024-09-18T07:43:56.073535+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:40:54.574077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:54.574077+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"376'5398","reported_seq":5432,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:57.745321+0000","last_change":"2024-09-18T07:43:02.044389+0000","last_active":"2024-09-18T07:43:57.745321+0000","last_peered":"2024-09-18T07:43:57.745321+0000","last_clean":"2024-09-18T07:43:57.745321+0000","last_became_active":"2024-09-18T07:43:02.044010+0000","last_became_peered":"2024-09-18T07:43:02.044010+0000","last_unstale":"2024-09-18T07:43:57.745321+0000","last_undegraded":"2024-09-18T07:43:57.745321+0000","last_fullsized":"2024-09-18T07:43:57.745321+0000","mapping_epoch":364,"log_start":"367'400","ondisk_log_start":"367'400","created":364,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":4998,"log_dups_size":0,"ondisk_log_size":4998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:13.064096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353763328,"num_objects":5398,"num_object_clones":0,"num_object_copies":16194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5398,"num_write_kb":345472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":403,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.074561+0000","last_change":"2024-09-18T07:42:54.743090+0000","last_active":"2024-09-18T07:43:56.074561+0000","last_peered":"2024-09-18T07:43:56.074561+0000","last_clean":"2024-09-18T07:42:51.730329+0000","last_became_active":"2024-09-18T07:42:54.743090+0000","last_became_peered":"2024-09-18T07:42:54.743090+0000","last_unstale":"2024-09-18T07:43:56.074561+0000","last_undegraded":"2024-09-18T07:43:56.074561+0000","last_fullsized":"2024-09-18T07:42:54.735572+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:42:39.191605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:42:39.191605+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.092770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"323'24","reported_seq":705,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.072727+0000","last_change":"2024-09-18T07:41:27.030793+0000","last_active":"2024-09-18T07:43:56.072727+0000","last_peered":"2024-09-18T07:43:56.072727+0000","last_clean":"2024-09-18T07:43:56.072727+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:43:56.072727+0000","last_undegraded":"2024-09-18T07:43:56.072727+0000","last_fullsized":"2024-09-18T07:43:56.072727+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'24","last_scrub_stamp":"2024-09-18T07:41:27.030705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.030705+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":670,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.072078+0000","last_change":"2024-09-18T07:42:54.753565+0000","last_active":"2024-09-18T07:43:56.072078+0000","last_peered":"2024-09-18T07:43:56.072078+0000","last_clean":"2024-09-18T07:42:51.717122+0000","last_became_active":"2024-09-18T07:42:54.753565+0000","last_became_peered":"2024-09-18T07:42:54.753565+0000","last_unstale":"2024-09-18T07:43:56.072078+0000","last_undegraded":"2024-09-18T07:42:54.738688+0000","last_fullsized":"2024-09-18T07:42:54.738589+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:42:44.401546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:44.401546+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":966,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.071388+0000","last_change":"2024-09-18T07:40:47.554230+0000","last_active":"2024-09-18T07:43:56.071388+0000","last_peered":"2024-09-18T07:43:56.071388+0000","last_clean":"2024-09-18T07:43:56.071388+0000","last_became_active":"2024-09-18T07:35:01.400524+0000","last_became_peered":"2024-09-18T07:35:01.400524+0000","last_unstale":"2024-09-18T07:43:56.071388+0000","last_undegraded":"2024-09-18T07:43:56.071388+0000","last_fullsized":"2024-09-18T07:43:56.071388+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T07:40:47.554161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.554161+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:50.421249+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":541,"num_read_kb":536,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":669,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.067603+0000","last_change":"2024-09-18T07:42:02.399342+0000","last_active":"2024-09-18T07:43:56.067603+0000","last_peered":"2024-09-18T07:43:56.067603+0000","last_clean":"2024-09-18T07:43:56.067603+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:43:56.067603+0000","last_undegraded":"2024-09-18T07:43:56.067603+0000","last_fullsized":"2024-09-18T07:43:56.067603+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:02.399269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:02.399269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:48.466755+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":362,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.066013+0000","last_change":"2024-09-18T07:40:12.269264+0000","last_active":"2024-09-18T07:43:56.066013+0000","last_peered":"2024-09-18T07:43:56.066013+0000","last_clean":"2024-09-18T07:43:56.066013+0000","last_became_active":"2024-09-18T07:36:13.117463+0000","last_became_peered":"2024-09-18T07:36:13.117463+0000","last_unstale":"2024-09-18T07:43:56.066013+0000","last_undegraded":"2024-09-18T07:43:56.066013+0000","last_fullsized":"2024-09-18T07:43:56.066013+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:12.269201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:12.269201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:20.558675+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"376'5248","reported_seq":5283,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.763685+0000","last_change":"2024-09-18T07:43:02.069622+0000","last_active":"2024-09-18T07:43:56.763685+0000","last_peered":"2024-09-18T07:43:56.763685+0000","last_clean":"2024-09-18T07:43:56.763685+0000","last_became_active":"2024-09-18T07:43:02.069360+0000","last_became_peered":"2024-09-18T07:43:02.069360+0000","last_unstale":"2024-09-18T07:43:56.763685+0000","last_undegraded":"2024-09-18T07:43:56.763685+0000","last_fullsized":"2024-09-18T07:43:56.763685+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":364,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5248,"log_dups_size":0,"ondisk_log_size":5248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:18.204978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343932928,"num_objects":5248,"num_object_clones":0,"num_object_copies":15744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5248,"num_write_kb":335872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"360'33","reported_seq":409,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.073579+0000","last_change":"2024-09-18T07:42:54.743919+0000","last_active":"2024-09-18T07:43:56.073579+0000","last_peered":"2024-09-18T07:43:56.073579+0000","last_clean":"2024-09-18T07:42:51.730345+0000","last_became_active":"2024-09-18T07:42:54.743919+0000","last_became_peered":"2024-09-18T07:42:54.743919+0000","last_unstale":"2024-09-18T07:43:56.073579+0000","last_undegraded":"2024-09-18T07:42:54.735862+0000","last_fullsized":"2024-09-18T07:42:54.735753+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-18T07:40:17.950298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:17.950298+0000","objects_scrubbed":8,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:59.184720+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":8}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":540,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.066916+0000","last_change":"2024-09-18T07:42:35.555848+0000","last_active":"2024-09-18T07:43:56.066916+0000","last_peered":"2024-09-18T07:43:56.066916+0000","last_clean":"2024-09-18T07:43:56.066916+0000","last_became_active":"2024-09-18T07:40:19.067624+0000","last_became_peered":"2024-09-18T07:40:19.067624+0000","last_unstale":"2024-09-18T07:43:56.066916+0000","last_undegraded":"2024-09-18T07:43:56.066916+0000","last_fullsized":"2024-09-18T07:43:56.066916+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:35.555785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:35.555785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":493,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.071431+0000","last_change":"2024-09-18T07:40:42.452815+0000","last_active":"2024-09-18T07:43:56.071431+0000","last_peered":"2024-09-18T07:43:56.071431+0000","last_clean":"2024-09-18T07:43:56.071431+0000","last_became_active":"2024-09-18T07:39:03.350419+0000","last_became_peered":"2024-09-18T07:39:03.350419+0000","last_unstale":"2024-09-18T07:43:56.071431+0000","last_undegraded":"2024-09-18T07:43:56.071431+0000","last_fullsized":"2024-09-18T07:43:56.071431+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:42.452744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:42.452744+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:17.111759+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":443,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.074349+0000","last_change":"2024-09-18T07:42:06.726510+0000","last_active":"2024-09-18T07:43:56.074349+0000","last_peered":"2024-09-18T07:43:56.074349+0000","last_clean":"2024-09-18T07:43:56.074349+0000","last_became_active":"2024-09-18T07:35:01.412217+0000","last_became_peered":"2024-09-18T07:35:01.412217+0000","last_unstale":"2024-09-18T07:43:56.074349+0000","last_undegraded":"2024-09-18T07:43:56.074349+0000","last_fullsized":"2024-09-18T07:43:56.074349+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.726445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.726445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:22.156992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":653,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.067273+0000","last_change":"2024-09-18T07:42:54.753417+0000","last_active":"2024-09-18T07:43:56.067273+0000","last_peered":"2024-09-18T07:43:56.067273+0000","last_clean":"2024-09-18T07:42:51.721892+0000","last_became_active":"2024-09-18T07:42:54.753417+0000","last_became_peered":"2024-09-18T07:42:54.753417+0000","last_unstale":"2024-09-18T07:43:56.067273+0000","last_undegraded":"2024-09-18T07:43:56.067273+0000","last_fullsized":"2024-09-18T07:42:54.735411+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:36.592215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:36.592215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"376'5261","reported_seq":5293,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.749040+0000","last_change":"2024-09-18T07:43:10.055584+0000","last_active":"2024-09-18T07:43:56.749040+0000","last_peered":"2024-09-18T07:43:56.749040+0000","last_clean":"2024-09-18T07:43:01.032791+0000","last_became_active":"2024-09-18T07:43:02.064004+0000","last_became_peered":"2024-09-18T07:43:02.064004+0000","last_unstale":"2024-09-18T07:43:56.749040+0000","last_undegraded":"2024-09-18T07:43:10.054922+0000","last_fullsized":"2024-09-18T07:43:02.043033+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":364,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5261,"log_dups_size":0,"ondisk_log_size":5261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:11.482069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344784896,"num_objects":5261,"num_object_clones":0,"num_object_copies":15783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5261,"num_write_kb":336704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5261}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"323'19","reported_seq":706,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.073135+0000","last_change":"2024-09-18T07:42:54.743582+0000","last_active":"2024-09-18T07:43:56.073135+0000","last_peered":"2024-09-18T07:43:56.073135+0000","last_clean":"2024-09-18T07:41:05.256958+0000","last_became_active":"2024-09-18T07:42:54.743582+0000","last_became_peered":"2024-09-18T07:42:54.743582+0000","last_unstale":"2024-09-18T07:43:56.073135+0000","last_undegraded":"2024-09-18T07:42:54.734793+0000","last_fullsized":"2024-09-18T07:42:54.734687+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:42:43.402858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:43.402858+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:48.130660+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":283,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.065188+0000","last_change":"2024-09-18T07:42:54.751650+0000","last_active":"2024-09-18T07:43:56.065188+0000","last_peered":"2024-09-18T07:43:56.065188+0000","last_clean":"2024-09-18T07:34:59.365492+0000","last_became_active":"2024-09-18T07:42:54.751650+0000","last_became_peered":"2024-09-18T07:42:54.751650+0000","last_unstale":"2024-09-18T07:43:56.065188+0000","last_undegraded":"2024-09-18T07:43:56.065188+0000","last_fullsized":"2024-09-18T07:42:54.735937+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.078600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.078600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:57.933003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":679,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.067105+0000","last_change":"2024-09-18T07:42:54.742511+0000","last_active":"2024-09-18T07:43:56.067105+0000","last_peered":"2024-09-18T07:43:56.067105+0000","last_clean":"2024-09-18T07:42:51.721730+0000","last_became_active":"2024-09-18T07:42:54.742511+0000","last_became_peered":"2024-09-18T07:42:54.742511+0000","last_unstale":"2024-09-18T07:43:56.067105+0000","last_undegraded":"2024-09-18T07:43:56.067105+0000","last_fullsized":"2024-09-18T07:42:54.734185+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.406041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.406041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:58.417770+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":534,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.072203+0000","last_change":"2024-09-18T07:41:20.220326+0000","last_active":"2024-09-18T07:43:56.072203+0000","last_peered":"2024-09-18T07:43:56.072203+0000","last_clean":"2024-09-18T07:43:56.072203+0000","last_became_active":"2024-09-18T07:40:19.068998+0000","last_became_peered":"2024-09-18T07:40:19.068998+0000","last_unstale":"2024-09-18T07:43:56.072203+0000","last_undegraded":"2024-09-18T07:43:56.072203+0000","last_fullsized":"2024-09-18T07:43:56.072203+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:20.220262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:20.220262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:23.184298+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3","version":"376'5316","reported_seq":5351,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.901940+0000","last_change":"2024-09-18T07:43:02.064631+0000","last_active":"2024-09-18T07:43:56.901940+0000","last_peered":"2024-09-18T07:43:56.901940+0000","last_clean":"2024-09-18T07:43:56.901940+0000","last_became_active":"2024-09-18T07:43:02.064298+0000","last_became_peered":"2024-09-18T07:43:02.064298+0000","last_unstale":"2024-09-18T07:43:56.901940+0000","last_undegraded":"2024-09-18T07:43:56.901940+0000","last_fullsized":"2024-09-18T07:43:56.901940+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":364,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5316,"log_dups_size":0,"ondisk_log_size":5316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:24.871852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348389376,"num_objects":5316,"num_object_clones":0,"num_object_copies":15948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5316,"num_write_kb":340224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":234,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.065360+0000","last_change":"2024-09-18T07:39:38.447787+0000","last_active":"2024-09-18T07:43:56.065360+0000","last_peered":"2024-09-18T07:43:56.065360+0000","last_clean":"2024-09-18T07:43:56.065360+0000","last_became_active":"2024-09-18T07:39:04.950292+0000","last_became_peered":"2024-09-18T07:39:04.950292+0000","last_unstale":"2024-09-18T07:43:56.065360+0000","last_undegraded":"2024-09-18T07:43:56.065360+0000","last_fullsized":"2024-09-18T07:43:56.065360+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:38.447735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:38.447735+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:52.496773+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"323'31","reported_seq":657,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.066728+0000","last_change":"2024-09-18T07:42:54.742580+0000","last_active":"2024-09-18T07:43:56.066728+0000","last_peered":"2024-09-18T07:43:56.066728+0000","last_clean":"2024-09-18T07:41:05.204075+0000","last_became_active":"2024-09-18T07:42:54.742580+0000","last_became_peered":"2024-09-18T07:42:54.742580+0000","last_unstale":"2024-09-18T07:43:56.066728+0000","last_undegraded":"2024-09-18T07:42:54.738910+0000","last_fullsized":"2024-09-18T07:42:54.738808+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:40:04.281758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:04.281758+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:59.927098+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":313,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.072245+0000","last_change":"2024-09-18T07:42:54.741952+0000","last_active":"2024-09-18T07:43:56.072245+0000","last_peered":"2024-09-18T07:43:56.072245+0000","last_clean":"2024-09-18T07:42:51.729886+0000","last_became_active":"2024-09-18T07:42:54.741952+0000","last_became_peered":"2024-09-18T07:42:54.741952+0000","last_unstale":"2024-09-18T07:43:56.072245+0000","last_undegraded":"2024-09-18T07:43:56.072245+0000","last_fullsized":"2024-09-18T07:42:54.735163+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:54.294307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.294307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:12.111313+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":44,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.073525+0000","last_change":"2024-09-18T07:42:54.750942+0000","last_active":"2024-09-18T07:43:56.073525+0000","last_peered":"2024-09-18T07:43:56.073525+0000","last_clean":"2024-09-18T07:30:59.305166+0000","last_became_active":"2024-09-18T07:42:54.750942+0000","last_became_peered":"2024-09-18T07:42:54.750942+0000","last_unstale":"2024-09-18T07:43:56.073525+0000","last_undegraded":"2024-09-18T07:43:56.073525+0000","last_fullsized":"2024-09-18T07:42:54.740397+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:52.167048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:52.167048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:00.060658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":679,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.065307+0000","last_change":"2024-09-18T07:42:54.749690+0000","last_active":"2024-09-18T07:43:56.065307+0000","last_peered":"2024-09-18T07:43:56.065307+0000","last_clean":"2024-09-18T07:42:51.717771+0000","last_became_active":"2024-09-18T07:42:54.749690+0000","last_became_peered":"2024-09-18T07:42:54.749690+0000","last_unstale":"2024-09-18T07:43:56.065307+0000","last_undegraded":"2024-09-18T07:43:56.065307+0000","last_fullsized":"2024-09-18T07:42:54.737695+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:56.172797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:56.172797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:09.000565+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"376'5417","reported_seq":5447,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:57.752986+0000","last_change":"2024-09-18T07:43:10.056592+0000","last_active":"2024-09-18T07:43:57.752986+0000","last_peered":"2024-09-18T07:43:57.752986+0000","last_clean":"2024-09-18T07:43:01.032791+0000","last_became_active":"2024-09-18T07:43:02.063956+0000","last_became_peered":"2024-09-18T07:43:02.063956+0000","last_unstale":"2024-09-18T07:43:57.752986+0000","last_undegraded":"2024-09-18T07:43:10.052693+0000","last_fullsized":"2024-09-18T07:43:02.038580+0000","mapping_epoch":364,"log_start":"367'400","ondisk_log_start":"367'400","created":364,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5017,"log_dups_size":0,"ondisk_log_size":5017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:30.710726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355008512,"num_objects":5417,"num_object_clones":0,"num_object_copies":16251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5417,"num_write_kb":346688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":5417}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":499,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.073381+0000","last_change":"2024-09-18T07:42:32.807852+0000","last_active":"2024-09-18T07:43:56.073381+0000","last_peered":"2024-09-18T07:43:56.073381+0000","last_clean":"2024-09-18T07:43:56.073381+0000","last_became_active":"2024-09-18T07:39:04.944753+0000","last_became_peered":"2024-09-18T07:39:04.944753+0000","last_unstale":"2024-09-18T07:43:56.073381+0000","last_undegraded":"2024-09-18T07:43:56.073381+0000","last_fullsized":"2024-09-18T07:43:56.073381+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:42:32.807800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:42:32.807800+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"323'20","reported_seq":660,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.076790+0000","last_change":"2024-09-18T07:42:54.753491+0000","last_active":"2024-09-18T07:43:56.076790+0000","last_peered":"2024-09-18T07:43:56.076790+0000","last_clean":"2024-09-18T07:41:05.073165+0000","last_became_active":"2024-09-18T07:42:54.753491+0000","last_became_peered":"2024-09-18T07:42:54.753491+0000","last_unstale":"2024-09-18T07:43:56.076790+0000","last_undegraded":"2024-09-18T07:42:54.745505+0000","last_fullsized":"2024-09-18T07:42:54.745363+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:40:50.209824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.209824+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:07.325297+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":163,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.083040+0000","last_change":"2024-09-18T07:42:54.747701+0000","last_active":"2024-09-18T07:43:56.083040+0000","last_peered":"2024-09-18T07:43:56.083040+0000","last_clean":"2024-09-18T07:42:51.718440+0000","last_became_active":"2024-09-18T07:42:54.747701+0000","last_became_peered":"2024-09-18T07:42:54.747701+0000","last_unstale":"2024-09-18T07:43:56.083040+0000","last_undegraded":"2024-09-18T07:43:56.083040+0000","last_fullsized":"2024-09-18T07:42:54.739078+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:47.799342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.799342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:58.078277+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"173'11","reported_seq":861,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.066677+0000","last_change":"2024-09-18T07:42:40.909417+0000","last_active":"2024-09-18T07:43:56.066677+0000","last_peered":"2024-09-18T07:43:56.066677+0000","last_clean":"2024-09-18T07:43:56.066677+0000","last_became_active":"2024-09-18T07:35:01.403765+0000","last_became_peered":"2024-09-18T07:35:01.403765+0000","last_unstale":"2024-09-18T07:43:56.066677+0000","last_undegraded":"2024-09-18T07:43:56.066677+0000","last_fullsized":"2024-09-18T07:43:56.066677+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'11","last_scrub_stamp":"2024-09-18T07:42:40.909351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:40.909351+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:48.131824+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":43,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.065612+0000","last_change":"2024-09-18T07:42:54.745260+0000","last_active":"2024-09-18T07:43:56.065612+0000","last_peered":"2024-09-18T07:43:56.065612+0000","last_clean":"2024-09-18T07:31:01.319795+0000","last_became_active":"2024-09-18T07:42:54.745260+0000","last_became_peered":"2024-09-18T07:42:54.745260+0000","last_unstale":"2024-09-18T07:43:56.065612+0000","last_undegraded":"2024-09-18T07:43:56.065612+0000","last_fullsized":"2024-09-18T07:42:54.735312+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:33.251093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:33.251093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.5","version":"376'5286","reported_seq":5320,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.923108+0000","last_change":"2024-09-18T07:43:02.071505+0000","last_active":"2024-09-18T07:43:56.923108+0000","last_peered":"2024-09-18T07:43:56.923108+0000","last_clean":"2024-09-18T07:43:56.923108+0000","last_became_active":"2024-09-18T07:43:02.070728+0000","last_became_peered":"2024-09-18T07:43:02.070728+0000","last_unstale":"2024-09-18T07:43:56.923108+0000","last_undegraded":"2024-09-18T07:43:56.923108+0000","last_fullsized":"2024-09-18T07:43:56.923108+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":364,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5286,"log_dups_size":0,"ondisk_log_size":5286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:01.382330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346423296,"num_objects":5286,"num_object_clones":0,"num_object_copies":15858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5286,"num_write_kb":338304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"323'15","reported_seq":709,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.082746+0000","last_change":"2024-09-18T07:42:54.751765+0000","last_active":"2024-09-18T07:43:56.082746+0000","last_peered":"2024-09-18T07:43:56.082746+0000","last_clean":"2024-09-18T07:42:51.718211+0000","last_became_active":"2024-09-18T07:42:54.751765+0000","last_became_peered":"2024-09-18T07:42:54.751765+0000","last_unstale":"2024-09-18T07:43:56.082746+0000","last_undegraded":"2024-09-18T07:42:54.736391+0000","last_fullsized":"2024-09-18T07:42:54.736259+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:21.946066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:21.946066+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:10.233191+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":691,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.068301+0000","last_change":"2024-09-18T07:43:04.518722+0000","last_active":"2024-09-18T07:43:56.068301+0000","last_peered":"2024-09-18T07:43:56.068301+0000","last_clean":"2024-09-18T07:43:56.068301+0000","last_became_active":"2024-09-18T07:37:43.075988+0000","last_became_peered":"2024-09-18T07:37:43.075988+0000","last_unstale":"2024-09-18T07:43:56.068301+0000","last_undegraded":"2024-09-18T07:43:56.068301+0000","last_fullsized":"2024-09-18T07:43:56.068301+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:04.518676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:43:04.518676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:16.665732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":667,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.064594+0000","last_change":"2024-09-18T07:42:54.751297+0000","last_active":"2024-09-18T07:43:56.064594+0000","last_peered":"2024-09-18T07:43:56.064594+0000","last_clean":"2024-09-18T07:42:51.718134+0000","last_became_active":"2024-09-18T07:42:54.751297+0000","last_became_peered":"2024-09-18T07:42:54.751297+0000","last_unstale":"2024-09-18T07:43:56.064594+0000","last_undegraded":"2024-09-18T07:43:56.064594+0000","last_fullsized":"2024-09-18T07:42:54.743454+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:42.922615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:42.922615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:57.724485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":685,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.071970+0000","last_change":"2024-09-18T07:43:34.257816+0000","last_active":"2024-09-18T07:43:56.071970+0000","last_peered":"2024-09-18T07:43:56.071970+0000","last_clean":"2024-09-18T07:43:56.071970+0000","last_became_active":"2024-09-18T07:35:01.410890+0000","last_became_peered":"2024-09-18T07:35:01.410890+0000","last_unstale":"2024-09-18T07:43:56.071970+0000","last_undegraded":"2024-09-18T07:43:56.071970+0000","last_fullsized":"2024-09-18T07:43:56.071970+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:34.257746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:34.257746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:58.955181+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"376'5347","reported_seq":5379,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:57.741388+0000","last_change":"2024-09-18T07:43:10.063264+0000","last_active":"2024-09-18T07:43:57.741388+0000","last_peered":"2024-09-18T07:43:57.741388+0000","last_clean":"2024-09-18T07:43:01.032791+0000","last_became_active":"2024-09-18T07:43:02.071284+0000","last_became_peered":"2024-09-18T07:43:02.071284+0000","last_unstale":"2024-09-18T07:43:57.741388+0000","last_undegraded":"2024-09-18T07:43:10.058462+0000","last_fullsized":"2024-09-18T07:43:02.038779+0000","mapping_epoch":364,"log_start":"367'400","ondisk_log_start":"367'400","created":364,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":4947,"log_dups_size":0,"ondisk_log_size":4947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:20.627488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350420992,"num_objects":5347,"num_object_clones":0,"num_object_copies":16041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5347,"num_write_kb":342208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":5347}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"323'35","reported_seq":345,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.072481+0000","last_change":"2024-09-18T07:42:44.097321+0000","last_active":"2024-09-18T07:43:56.072481+0000","last_peered":"2024-09-18T07:43:56.072481+0000","last_clean":"2024-09-18T07:43:56.072481+0000","last_became_active":"2024-09-18T07:37:43.063319+0000","last_became_peered":"2024-09-18T07:37:43.063319+0000","last_unstale":"2024-09-18T07:43:56.072481+0000","last_undegraded":"2024-09-18T07:43:56.072481+0000","last_fullsized":"2024-09-18T07:43:56.072481+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'35","last_scrub_stamp":"2024-09-18T07:42:44.097218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:44.097218+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:58.864478+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":456,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.073968+0000","last_change":"2024-09-18T07:40:56.546321+0000","last_active":"2024-09-18T07:43:56.073968+0000","last_peered":"2024-09-18T07:43:56.073968+0000","last_clean":"2024-09-18T07:43:56.073968+0000","last_became_active":"2024-09-18T07:37:43.072894+0000","last_became_peered":"2024-09-18T07:37:43.072894+0000","last_unstale":"2024-09-18T07:43:56.073968+0000","last_undegraded":"2024-09-18T07:43:56.073968+0000","last_fullsized":"2024-09-18T07:43:56.073968+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:56.546251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.546251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:54.470740+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":678,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.066505+0000","last_change":"2024-09-18T07:42:04.179746+0000","last_active":"2024-09-18T07:43:56.066505+0000","last_peered":"2024-09-18T07:43:56.066505+0000","last_clean":"2024-09-18T07:43:56.066505+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:43:56.066505+0000","last_undegraded":"2024-09-18T07:43:56.066505+0000","last_fullsized":"2024-09-18T07:43:56.066505+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:04.179670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.179670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:19.929773+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":443,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.071505+0000","last_change":"2024-09-18T07:43:39.239989+0000","last_active":"2024-09-18T07:43:56.071505+0000","last_peered":"2024-09-18T07:43:56.071505+0000","last_clean":"2024-09-18T07:43:56.071505+0000","last_became_active":"2024-09-18T07:35:01.406980+0000","last_became_peered":"2024-09-18T07:35:01.406980+0000","last_unstale":"2024-09-18T07:43:56.071505+0000","last_undegraded":"2024-09-18T07:43:56.071505+0000","last_fullsized":"2024-09-18T07:43:56.071505+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:39.239932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:39.239932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:57.648423+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"376'5332","reported_seq":5367,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.763637+0000","last_change":"2024-09-18T07:43:02.069332+0000","last_active":"2024-09-18T07:43:56.763637+0000","last_peered":"2024-09-18T07:43:56.763637+0000","last_clean":"2024-09-18T07:43:56.763637+0000","last_became_active":"2024-09-18T07:43:02.068979+0000","last_became_peered":"2024-09-18T07:43:02.068979+0000","last_unstale":"2024-09-18T07:43:56.763637+0000","last_undegraded":"2024-09-18T07:43:56.763637+0000","last_fullsized":"2024-09-18T07:43:56.763637+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":364,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5332,"log_dups_size":0,"ondisk_log_size":5332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:04.972175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349437952,"num_objects":5332,"num_object_clones":0,"num_object_copies":15996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5332,"num_write_kb":341248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"323'18","reported_seq":640,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.075207+0000","last_change":"2024-09-18T07:42:54.755198+0000","last_active":"2024-09-18T07:43:56.075207+0000","last_peered":"2024-09-18T07:43:56.075207+0000","last_clean":"2024-09-18T07:41:04.257612+0000","last_became_active":"2024-09-18T07:42:54.755198+0000","last_became_peered":"2024-09-18T07:42:54.755198+0000","last_unstale":"2024-09-18T07:43:56.075207+0000","last_undegraded":"2024-09-18T07:42:54.746711+0000","last_fullsized":"2024-09-18T07:42:54.745765+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:41:47.950068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:47.950068+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:29.362131+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":438,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.074466+0000","last_change":"2024-09-18T07:42:54.741218+0000","last_active":"2024-09-18T07:43:56.074466+0000","last_peered":"2024-09-18T07:43:56.074466+0000","last_clean":"2024-09-18T07:42:51.728402+0000","last_became_active":"2024-09-18T07:42:54.741218+0000","last_became_peered":"2024-09-18T07:42:54.741218+0000","last_unstale":"2024-09-18T07:43:56.074466+0000","last_undegraded":"2024-09-18T07:43:56.074466+0000","last_fullsized":"2024-09-18T07:42:54.737804+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:07.729438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:07.729438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.251922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":687,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.064883+0000","last_change":"2024-09-18T07:41:58.202324+0000","last_active":"2024-09-18T07:43:56.064883+0000","last_peered":"2024-09-18T07:43:56.064883+0000","last_clean":"2024-09-18T07:43:56.064883+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:43:56.064883+0000","last_undegraded":"2024-09-18T07:43:56.064883+0000","last_fullsized":"2024-09-18T07:43:56.064883+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:58.202229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:58.202229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:23.991237+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":672,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.076235+0000","last_change":"2024-09-18T07:43:01.317153+0000","last_active":"2024-09-18T07:43:56.076235+0000","last_peered":"2024-09-18T07:43:56.076235+0000","last_clean":"2024-09-18T07:43:56.076235+0000","last_became_active":"2024-09-18T07:35:01.418735+0000","last_became_peered":"2024-09-18T07:35:01.418735+0000","last_unstale":"2024-09-18T07:43:56.076235+0000","last_undegraded":"2024-09-18T07:43:56.076235+0000","last_fullsized":"2024-09-18T07:43:56.076235+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.317105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.317105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:07.086893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"376'5404","reported_seq":5435,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:57.140235+0000","last_change":"2024-09-18T07:43:10.055586+0000","last_active":"2024-09-18T07:43:57.140235+0000","last_peered":"2024-09-18T07:43:57.140235+0000","last_clean":"2024-09-18T07:43:01.032791+0000","last_became_active":"2024-09-18T07:43:02.048967+0000","last_became_peered":"2024-09-18T07:43:02.048967+0000","last_unstale":"2024-09-18T07:43:57.140235+0000","last_undegraded":"2024-09-18T07:43:10.055058+0000","last_fullsized":"2024-09-18T07:43:02.043209+0000","mapping_epoch":364,"log_start":"367'400","ondisk_log_start":"367'400","created":364,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:17.566081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354156544,"num_objects":5404,"num_object_clones":0,"num_object_copies":16212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5404,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5404,"num_write_kb":345856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":5404}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"323'27","reported_seq":808,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.065587+0000","last_change":"2024-09-18T07:41:51.332873+0000","last_active":"2024-09-18T07:43:56.065587+0000","last_peered":"2024-09-18T07:43:56.065587+0000","last_clean":"2024-09-18T07:43:56.065587+0000","last_became_active":"2024-09-18T07:37:43.075552+0000","last_became_peered":"2024-09-18T07:37:43.075552+0000","last_unstale":"2024-09-18T07:43:56.065587+0000","last_undegraded":"2024-09-18T07:43:56.065587+0000","last_fullsized":"2024-09-18T07:43:56.065587+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:41:51.332791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:51.332791+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":452,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.074908+0000","last_change":"2024-09-18T07:42:54.747864+0000","last_active":"2024-09-18T07:43:56.074908+0000","last_peered":"2024-09-18T07:43:56.074908+0000","last_clean":"2024-09-18T07:42:51.728762+0000","last_became_active":"2024-09-18T07:42:54.747864+0000","last_became_peered":"2024-09-18T07:42:54.747864+0000","last_unstale":"2024-09-18T07:43:56.074908+0000","last_undegraded":"2024-09-18T07:42:54.739112+0000","last_fullsized":"2024-09-18T07:42:54.738998+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:41:27.819269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.819269+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:08.071937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"360'13","reported_seq":582,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.077278+0000","last_change":"2024-09-18T07:42:54.744846+0000","last_active":"2024-09-18T07:43:56.077278+0000","last_peered":"2024-09-18T07:43:56.077278+0000","last_clean":"2024-09-18T07:42:51.728785+0000","last_became_active":"2024-09-18T07:42:54.744846+0000","last_became_peered":"2024-09-18T07:42:54.744846+0000","last_unstale":"2024-09-18T07:43:56.077278+0000","last_undegraded":"2024-09-18T07:42:54.737124+0000","last_fullsized":"2024-09-18T07:42:54.736998+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:42:35.793579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:35.793579+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:48.351749+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":663,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.066503+0000","last_change":"2024-09-18T07:42:07.547399+0000","last_active":"2024-09-18T07:43:56.066503+0000","last_peered":"2024-09-18T07:43:56.066503+0000","last_clean":"2024-09-18T07:43:56.066503+0000","last_became_active":"2024-09-18T07:37:43.078121+0000","last_became_peered":"2024-09-18T07:37:43.078121+0000","last_unstale":"2024-09-18T07:43:56.066503+0000","last_undegraded":"2024-09-18T07:43:56.066503+0000","last_fullsized":"2024-09-18T07:43:56.066503+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:07.547352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:07.547352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:49.424413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":301,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.069762+0000","last_change":"2024-09-18T07:42:08.181612+0000","last_active":"2024-09-18T07:43:56.069762+0000","last_peered":"2024-09-18T07:43:56.069762+0000","last_clean":"2024-09-18T07:43:56.069762+0000","last_became_active":"2024-09-18T07:37:43.087994+0000","last_became_peered":"2024-09-18T07:37:43.087994+0000","last_unstale":"2024-09-18T07:43:56.069762+0000","last_undegraded":"2024-09-18T07:43:56.069762+0000","last_fullsized":"2024-09-18T07:43:56.069762+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:08.181420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:08.181420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:27.181019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"323'12","reported_seq":714,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.076910+0000","last_change":"2024-09-18T07:41:48.359809+0000","last_active":"2024-09-18T07:43:56.076910+0000","last_peered":"2024-09-18T07:43:56.076910+0000","last_clean":"2024-09-18T07:43:56.076910+0000","last_became_active":"2024-09-18T07:35:01.407787+0000","last_became_peered":"2024-09-18T07:35:01.407787+0000","last_unstale":"2024-09-18T07:43:56.076910+0000","last_undegraded":"2024-09-18T07:43:56.076910+0000","last_fullsized":"2024-09-18T07:43:56.076910+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'12","last_scrub_stamp":"2024-09-18T07:41:48.359711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.359711+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:19.370461+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":677,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.065971+0000","last_change":"2024-09-18T07:43:37.515165+0000","last_active":"2024-09-18T07:43:56.065971+0000","last_peered":"2024-09-18T07:43:56.065971+0000","last_clean":"2024-09-18T07:43:56.065971+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:43:56.065971+0000","last_undegraded":"2024-09-18T07:43:56.065971+0000","last_fullsized":"2024-09-18T07:43:56.065971+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:37.515124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:43:37.515124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:47.562291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":444,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.073766+0000","last_change":"2024-09-18T07:42:54.745409+0000","last_active":"2024-09-18T07:43:56.073766+0000","last_peered":"2024-09-18T07:43:56.073766+0000","last_clean":"2024-09-18T07:42:51.728619+0000","last_became_active":"2024-09-18T07:42:54.745409+0000","last_became_peered":"2024-09-18T07:42:54.745409+0000","last_unstale":"2024-09-18T07:43:56.073766+0000","last_undegraded":"2024-09-18T07:43:56.073766+0000","last_fullsized":"2024-09-18T07:42:54.737509+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:51.554073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:51.554073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:57.925766+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"323'15","reported_seq":706,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.072776+0000","last_change":"2024-09-18T07:42:34.313755+0000","last_active":"2024-09-18T07:43:56.072776+0000","last_peered":"2024-09-18T07:43:56.072776+0000","last_clean":"2024-09-18T07:43:56.072776+0000","last_became_active":"2024-09-18T07:35:01.414458+0000","last_became_peered":"2024-09-18T07:35:01.414458+0000","last_unstale":"2024-09-18T07:43:56.072776+0000","last_undegraded":"2024-09-18T07:43:56.072776+0000","last_fullsized":"2024-09-18T07:43:56.072776+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:42:34.313665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:34.313665+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:51.059507+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":460,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.073787+0000","last_change":"2024-09-18T07:42:54.747009+0000","last_active":"2024-09-18T07:43:56.073787+0000","last_peered":"2024-09-18T07:43:56.073787+0000","last_clean":"2024-09-18T07:42:51.728811+0000","last_became_active":"2024-09-18T07:42:54.747009+0000","last_became_peered":"2024-09-18T07:42:54.747009+0000","last_unstale":"2024-09-18T07:43:56.073787+0000","last_undegraded":"2024-09-18T07:43:56.073787+0000","last_fullsized":"2024-09-18T07:42:54.737030+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:52.891811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:52.891811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:23.178220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":684,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.076229+0000","last_change":"2024-09-18T07:41:16.472282+0000","last_active":"2024-09-18T07:43:56.076229+0000","last_peered":"2024-09-18T07:43:56.076229+0000","last_clean":"2024-09-18T07:43:56.076229+0000","last_became_active":"2024-09-18T07:35:01.420133+0000","last_became_peered":"2024-09-18T07:35:01.420133+0000","last_unstale":"2024-09-18T07:43:56.076229+0000","last_undegraded":"2024-09-18T07:43:56.076229+0000","last_fullsized":"2024-09-18T07:43:56.076229+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:16.472210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:16.472210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:13.282511+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":669,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.064863+0000","last_change":"2024-09-18T07:42:54.744738+0000","last_active":"2024-09-18T07:43:56.064863+0000","last_peered":"2024-09-18T07:43:56.064863+0000","last_clean":"2024-09-18T07:42:51.717389+0000","last_became_active":"2024-09-18T07:42:54.744738+0000","last_became_peered":"2024-09-18T07:42:54.744738+0000","last_unstale":"2024-09-18T07:43:56.064863+0000","last_undegraded":"2024-09-18T07:43:56.064863+0000","last_fullsized":"2024-09-18T07:42:54.736826+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:34.876429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:34.876429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:04.416006+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"323'15","reported_seq":724,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.067708+0000","last_change":"2024-09-18T07:40:52.366957+0000","last_active":"2024-09-18T07:43:56.067708+0000","last_peered":"2024-09-18T07:43:56.067708+0000","last_clean":"2024-09-18T07:43:56.067708+0000","last_became_active":"2024-09-18T07:35:01.416744+0000","last_became_peered":"2024-09-18T07:35:01.416744+0000","last_unstale":"2024-09-18T07:43:56.067708+0000","last_undegraded":"2024-09-18T07:43:56.067708+0000","last_fullsized":"2024-09-18T07:43:56.067708+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:52.366672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:52.366672+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:28.664103+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":697,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.073086+0000","last_change":"2024-09-18T07:43:31.255482+0000","last_active":"2024-09-18T07:43:56.073086+0000","last_peered":"2024-09-18T07:43:56.073086+0000","last_clean":"2024-09-18T07:43:56.073086+0000","last_became_active":"2024-09-18T07:35:01.414024+0000","last_became_peered":"2024-09-18T07:35:01.414024+0000","last_unstale":"2024-09-18T07:43:56.073086+0000","last_undegraded":"2024-09-18T07:43:56.073086+0000","last_fullsized":"2024-09-18T07:43:56.073086+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:31.255278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:43:31.255278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:37.083443+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"173'10","reported_seq":849,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.076679+0000","last_change":"2024-09-18T07:42:58.271458+0000","last_active":"2024-09-18T07:43:56.076679+0000","last_peered":"2024-09-18T07:43:56.076679+0000","last_clean":"2024-09-18T07:43:56.076679+0000","last_became_active":"2024-09-18T07:35:01.407568+0000","last_became_peered":"2024-09-18T07:35:01.407568+0000","last_unstale":"2024-09-18T07:43:56.076679+0000","last_undegraded":"2024-09-18T07:43:56.076679+0000","last_fullsized":"2024-09-18T07:43:56.076679+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:42:58.271411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:58.271411+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:04.950201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":43,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.065137+0000","last_change":"2024-09-18T07:42:54.748643+0000","last_active":"2024-09-18T07:43:56.065137+0000","last_peered":"2024-09-18T07:43:56.065137+0000","last_clean":"2024-09-18T07:31:01.319795+0000","last_became_active":"2024-09-18T07:42:54.748643+0000","last_became_peered":"2024-09-18T07:42:54.748643+0000","last_unstale":"2024-09-18T07:43:56.065137+0000","last_undegraded":"2024-09-18T07:43:56.065137+0000","last_fullsized":"2024-09-18T07:42:54.735305+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:48.955463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.955463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:23.004141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"323'20","reported_seq":443,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.074021+0000","last_change":"2024-09-18T07:42:29.867736+0000","last_active":"2024-09-18T07:43:56.074021+0000","last_peered":"2024-09-18T07:43:56.074021+0000","last_clean":"2024-09-18T07:43:56.074021+0000","last_became_active":"2024-09-18T07:40:19.069759+0000","last_became_peered":"2024-09-18T07:40:19.069759+0000","last_unstale":"2024-09-18T07:43:56.074021+0000","last_undegraded":"2024-09-18T07:43:56.074021+0000","last_fullsized":"2024-09-18T07:43:56.074021+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:42:29.867645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:29.867645+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:47.303983+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":446,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.073071+0000","last_change":"2024-09-18T07:42:54.743824+0000","last_active":"2024-09-18T07:43:56.073071+0000","last_peered":"2024-09-18T07:43:56.073071+0000","last_clean":"2024-09-18T07:42:51.728597+0000","last_became_active":"2024-09-18T07:42:54.743824+0000","last_became_peered":"2024-09-18T07:42:54.743824+0000","last_unstale":"2024-09-18T07:43:56.073071+0000","last_undegraded":"2024-09-18T07:43:56.073071+0000","last_fullsized":"2024-09-18T07:42:54.738810+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:04.758396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.758396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:36.134989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":821,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.067429+0000","last_change":"2024-09-18T07:40:40.352088+0000","last_active":"2024-09-18T07:43:56.067429+0000","last_peered":"2024-09-18T07:43:56.067429+0000","last_clean":"2024-09-18T07:43:56.067429+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:43:56.067429+0000","last_undegraded":"2024-09-18T07:43:56.067429+0000","last_fullsized":"2024-09-18T07:43:56.067429+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:40:40.352040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.352040+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:22.552189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":685,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.072290+0000","last_change":"2024-09-18T07:43:33.228092+0000","last_active":"2024-09-18T07:43:56.072290+0000","last_peered":"2024-09-18T07:43:56.072290+0000","last_clean":"2024-09-18T07:43:56.072290+0000","last_became_active":"2024-09-18T07:35:01.410561+0000","last_became_peered":"2024-09-18T07:35:01.410561+0000","last_unstale":"2024-09-18T07:43:56.072290+0000","last_undegraded":"2024-09-18T07:43:56.072290+0000","last_fullsized":"2024-09-18T07:43:56.072290+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:33.228024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:33.228024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:37.940247+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"323'19","reported_seq":692,"reported_epoch":376,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:43:56.072586+0000","last_change":"2024-09-18T07:42:54.743805+0000","last_active":"2024-09-18T07:43:56.072586+0000","last_peered":"2024-09-18T07:43:56.072586+0000","last_clean":"2024-09-18T07:41:05.098917+0000","last_became_active":"2024-09-18T07:42:54.743805+0000","last_became_peered":"2024-09-18T07:42:54.743805+0000","last_unstale":"2024-09-18T07:43:56.072586+0000","last_undegraded":"2024-09-18T07:42:54.738157+0000","last_fullsized":"2024-09-18T07:42:54.738016+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:41:54.022421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.022421+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:24.137981+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":429,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.074127+0000","last_change":"2024-09-18T07:42:54.744327+0000","last_active":"2024-09-18T07:43:56.074127+0000","last_peered":"2024-09-18T07:43:56.074127+0000","last_clean":"2024-09-18T07:37:38.987385+0000","last_became_active":"2024-09-18T07:42:54.744327+0000","last_became_peered":"2024-09-18T07:42:54.744327+0000","last_unstale":"2024-09-18T07:43:56.074127+0000","last_undegraded":"2024-09-18T07:43:56.074127+0000","last_fullsized":"2024-09-18T07:42:54.737730+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:40.149313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.149313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:55.486321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":362,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.066638+0000","last_change":"2024-09-18T07:43:36.563063+0000","last_active":"2024-09-18T07:43:56.066638+0000","last_peered":"2024-09-18T07:43:56.066638+0000","last_clean":"2024-09-18T07:43:56.066638+0000","last_became_active":"2024-09-18T07:36:13.117584+0000","last_became_peered":"2024-09-18T07:36:13.117584+0000","last_unstale":"2024-09-18T07:43:56.066638+0000","last_undegraded":"2024-09-18T07:43:56.066638+0000","last_fullsized":"2024-09-18T07:43:56.066638+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:36.563010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:43:36.563010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:44.842956+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":166,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.071857+0000","last_change":"2024-09-18T07:43:45.121141+0000","last_active":"2024-09-18T07:43:56.071857+0000","last_peered":"2024-09-18T07:43:56.071857+0000","last_clean":"2024-09-18T07:43:56.071857+0000","last_became_active":"2024-09-18T07:40:19.065468+0000","last_became_peered":"2024-09-18T07:40:19.065468+0000","last_unstale":"2024-09-18T07:43:56.071857+0000","last_undegraded":"2024-09-18T07:43:56.071857+0000","last_fullsized":"2024-09-18T07:43:56.071857+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:45.121082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:45.121082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:56.672102+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"323'15","reported_seq":553,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.072774+0000","last_change":"2024-09-18T07:41:20.975210+0000","last_active":"2024-09-18T07:43:56.072774+0000","last_peered":"2024-09-18T07:43:56.072774+0000","last_clean":"2024-09-18T07:43:56.072774+0000","last_became_active":"2024-09-18T07:37:43.067598+0000","last_became_peered":"2024-09-18T07:37:43.067598+0000","last_unstale":"2024-09-18T07:43:56.072774+0000","last_undegraded":"2024-09-18T07:43:56.072774+0000","last_fullsized":"2024-09-18T07:43:56.072774+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:20.975135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:20.975135+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":446,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.073620+0000","last_change":"2024-09-18T07:42:54.748561+0000","last_active":"2024-09-18T07:43:56.073620+0000","last_peered":"2024-09-18T07:43:56.073620+0000","last_clean":"2024-09-18T07:42:51.728528+0000","last_became_active":"2024-09-18T07:42:54.748561+0000","last_became_peered":"2024-09-18T07:42:54.748561+0000","last_unstale":"2024-09-18T07:43:56.073620+0000","last_undegraded":"2024-09-18T07:43:56.073620+0000","last_fullsized":"2024-09-18T07:42:54.738920+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:47.880911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:47.880911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:09.115673+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":158,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.082550+0000","last_change":"2024-09-18T07:42:41.220234+0000","last_active":"2024-09-18T07:43:56.082550+0000","last_peered":"2024-09-18T07:43:56.082550+0000","last_clean":"2024-09-18T07:43:56.082550+0000","last_became_active":"2024-09-18T07:40:19.068237+0000","last_became_peered":"2024-09-18T07:40:19.068237+0000","last_unstale":"2024-09-18T07:43:56.082550+0000","last_undegraded":"2024-09-18T07:43:56.082550+0000","last_fullsized":"2024-09-18T07:43:56.082550+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:41.219755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:41.219755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:47.130390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":671,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.069794+0000","last_change":"2024-09-18T07:41:18.963607+0000","last_active":"2024-09-18T07:43:56.069794+0000","last_peered":"2024-09-18T07:43:56.069794+0000","last_clean":"2024-09-18T07:43:56.069794+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:43:56.069794+0000","last_undegraded":"2024-09-18T07:43:56.069794+0000","last_fullsized":"2024-09-18T07:43:56.069794+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:18.963490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:18.963490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:15.234320+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"323'19","reported_seq":781,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.075108+0000","last_change":"2024-09-18T07:43:37.436589+0000","last_active":"2024-09-18T07:43:56.075108+0000","last_peered":"2024-09-18T07:43:56.075108+0000","last_clean":"2024-09-18T07:43:56.075108+0000","last_became_active":"2024-09-18T07:37:43.066208+0000","last_became_peered":"2024-09-18T07:37:43.066208+0000","last_unstale":"2024-09-18T07:43:56.075108+0000","last_undegraded":"2024-09-18T07:43:56.075108+0000","last_fullsized":"2024-09-18T07:43:56.075108+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:43:37.436336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:43:37.436336+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:39.003498+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":683,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.066285+0000","last_change":"2024-09-18T07:43:32.658023+0000","last_active":"2024-09-18T07:43:56.066285+0000","last_peered":"2024-09-18T07:43:56.066285+0000","last_clean":"2024-09-18T07:43:56.066285+0000","last_became_active":"2024-09-18T07:35:01.415261+0000","last_became_peered":"2024-09-18T07:35:01.415261+0000","last_unstale":"2024-09-18T07:43:56.066285+0000","last_undegraded":"2024-09-18T07:43:56.066285+0000","last_fullsized":"2024-09-18T07:43:56.066285+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:32.657943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:43:32.657943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:00.076443+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":460,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.071577+0000","last_change":"2024-09-18T07:42:54.744492+0000","last_active":"2024-09-18T07:43:56.071577+0000","last_peered":"2024-09-18T07:43:56.071577+0000","last_clean":"2024-09-18T07:42:51.728160+0000","last_became_active":"2024-09-18T07:42:54.744492+0000","last_became_peered":"2024-09-18T07:42:54.744492+0000","last_unstale":"2024-09-18T07:43:56.071577+0000","last_undegraded":"2024-09-18T07:43:56.071577+0000","last_fullsized":"2024-09-18T07:42:54.736952+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:05.775170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:05.775170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:10.099635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":688,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.076416+0000","last_change":"2024-09-18T07:42:34.361617+0000","last_active":"2024-09-18T07:43:56.076416+0000","last_peered":"2024-09-18T07:43:56.076416+0000","last_clean":"2024-09-18T07:43:56.076416+0000","last_became_active":"2024-09-18T07:37:43.067066+0000","last_became_peered":"2024-09-18T07:37:43.067066+0000","last_unstale":"2024-09-18T07:43:56.076416+0000","last_undegraded":"2024-09-18T07:43:56.076416+0000","last_fullsized":"2024-09-18T07:43:56.076416+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:34.361382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:34.361382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:02.969586+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"323'8","reported_seq":717,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.074928+0000","last_change":"2024-09-18T07:37:43.134956+0000","last_active":"2024-09-18T07:43:56.074928+0000","last_peered":"2024-09-18T07:43:56.074928+0000","last_clean":"2024-09-18T07:43:56.074928+0000","last_became_active":"2024-09-18T07:37:43.067503+0000","last_became_peered":"2024-09-18T07:37:43.067503+0000","last_unstale":"2024-09-18T07:43:56.074928+0000","last_undegraded":"2024-09-18T07:43:56.074928+0000","last_fullsized":"2024-09-18T07:43:56.074928+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:36:40.908629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:40.908629+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:17.342666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":667,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.072376+0000","last_change":"2024-09-18T07:42:46.197586+0000","last_active":"2024-09-18T07:43:56.072376+0000","last_peered":"2024-09-18T07:43:56.072376+0000","last_clean":"2024-09-18T07:43:56.072376+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:43:56.072376+0000","last_undegraded":"2024-09-18T07:43:56.072376+0000","last_fullsized":"2024-09-18T07:43:56.072376+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:46.197514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:46.197514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:59.840515+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":297,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.069679+0000","last_change":"2024-09-18T07:42:54.744598+0000","last_active":"2024-09-18T07:43:56.069679+0000","last_peered":"2024-09-18T07:43:56.069679+0000","last_clean":"2024-09-18T07:42:51.729754+0000","last_became_active":"2024-09-18T07:42:54.744598+0000","last_became_peered":"2024-09-18T07:42:54.744598+0000","last_unstale":"2024-09-18T07:43:56.069679+0000","last_undegraded":"2024-09-18T07:43:56.069679+0000","last_fullsized":"2024-09-18T07:42:54.735924+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:40.106921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.106921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:02.138529+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":446,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.072370+0000","last_change":"2024-09-18T07:42:54.751407+0000","last_active":"2024-09-18T07:43:56.072370+0000","last_peered":"2024-09-18T07:43:56.072370+0000","last_clean":"2024-09-18T07:42:51.728069+0000","last_became_active":"2024-09-18T07:42:54.751407+0000","last_became_peered":"2024-09-18T07:42:54.751407+0000","last_unstale":"2024-09-18T07:43:56.072370+0000","last_undegraded":"2024-09-18T07:43:56.072370+0000","last_fullsized":"2024-09-18T07:42:54.738929+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:38.492269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.492269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:03.934385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":350,"reported_epoch":376,"state":"active+undersized","last_fresh":"2024-09-18T07:43:56.067483+0000","last_change":"2024-09-18T07:42:54.746248+0000","last_active":"2024-09-18T07:43:56.067483+0000","last_peered":"2024-09-18T07:43:56.067483+0000","last_clean":"2024-09-18T07:42:51.721338+0000","last_became_active":"2024-09-18T07:42:54.746248+0000","last_became_peered":"2024-09-18T07:42:54.746248+0000","last_unstale":"2024-09-18T07:43:56.067483+0000","last_undegraded":"2024-09-18T07:43:56.067483+0000","last_fullsized":"2024-09-18T07:42:54.734625+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.071629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.071629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:43:57.438488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":684,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.075143+0000","last_change":"2024-09-18T07:41:22.478552+0000","last_active":"2024-09-18T07:43:56.075143+0000","last_peered":"2024-09-18T07:43:56.075143+0000","last_clean":"2024-09-18T07:43:56.075143+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:43:56.075143+0000","last_undegraded":"2024-09-18T07:43:56.075143+0000","last_fullsized":"2024-09-18T07:43:56.075143+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:41:22.478474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:22.478474+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:12.324737+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"323'19","reported_seq":325,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.074039+0000","last_change":"2024-09-18T07:43:10.104168+0000","last_active":"2024-09-18T07:43:56.074039+0000","last_peered":"2024-09-18T07:43:56.074039+0000","last_clean":"2024-09-18T07:43:56.074039+0000","last_became_active":"2024-09-18T07:37:43.070186+0000","last_became_peered":"2024-09-18T07:37:43.070186+0000","last_unstale":"2024-09-18T07:43:56.074039+0000","last_undegraded":"2024-09-18T07:43:56.074039+0000","last_fullsized":"2024-09-18T07:43:56.074039+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:43:10.104058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:43:10.104058+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:35.272876+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":682,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-18T07:43:56.072509+0000","last_change":"2024-09-18T07:41:13.842121+0000","last_active":"2024-09-18T07:43:56.072509+0000","last_peered":"2024-09-18T07:43:56.072509+0000","last_clean":"2024-09-18T07:43:56.072509+0000","last_became_active":"2024-09-18T07:35:01.410469+0000","last_became_peered":"2024-09-18T07:35:01.410469+0000","last_unstale":"2024-09-18T07:43:56.072509+0000","last_undegraded":"2024-09-18T07:43:56.072509+0000","last_fullsized":"2024-09-18T07:43:56.072509+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:13.842068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:13.842068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:49.123726+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5573771264,"num_objects":85049,"num_object_clones":0,"num_object_copies":255147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42598,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85049,"num_write_kb":5443136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13965918208,"data_stored":13965918208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82249,"ondisk_log_size":82249,"up":40,"acting":40,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":84,"acting":84,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":81,"acting":81,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":74,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1109,"num_read_kb":866,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":4664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":685,"ondisk_log_size":685,"up":82,"acting":82,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":79,"acting":79,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":541,"num_read_kb":536,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":30,"acting":30,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201440,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1811580,"kb_used_data":1351400,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":91933572,"statfs":{"total":95995035648,"available":94139977728,"internally_reserved":0,"allocated":1383833600,"data_stored":1383413207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":471202250},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463078,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2762524,"kb_used_data":2344008,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":90982628,"statfs":{"total":95995035648,"available":93166211072,"internally_reserved":0,"allocated":2400264192,"data_stored":2399875152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":428538310},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724715,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2237232,"kb_used_data":1696540,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":91507920,"statfs":{"total":95995035648,"available":93704110080,"internally_reserved":0,"allocated":1737256960,"data_stored":1736848770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953637,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":341332,"kb_used_data":25540,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":93403820,"statfs":{"total":95995035648,"available":95645511680,"internally_reserved":0,"allocated":26152960,"data_stored":25775242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149878,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2287936,"kb_used_data":1688940,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":91457216,"statfs":{"total":95995035648,"available":93652189184,"internally_reserved":0,"allocated":1729474560,"data_stored":1729051190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":613349831},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411516,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2462188,"kb_used_data":2011736,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":91282964,"statfs":{"total":95995035648,"available":93473755136,"internally_reserved":0,"allocated":2060017664,"data_stored":2059613469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575043,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1581448,"kb_used_data":1004660,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92163704,"statfs":{"total":95995035648,"available":94375632896,"internally_reserved":0,"allocated":1028771840,"data_stored":1028339959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738568,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3960516,"kb_used_data":3411504,"kb_used_omap":1,"kb_used_meta":548990,"kb_avail":89784636,"statfs":{"total":95995035648,"available":91939467264,"internally_reserved":0,"allocated":3493380096,"data_stored":3492952636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":562166213},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3523411968,"data_stored":3523411968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1041563648,"data_stored":1041563648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2085093376,"data_stored":2085093376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1749221376,"data_stored":1749221376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1754071040,"data_stored":1754071040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2424111104,"data_stored":2424111104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1388445696,"data_stored":1388445696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:44:02.025 INFO:tasks.thrashosds.thrasher:Setting 3.1 to [2, 5, 7, 3, 6, 0] 2024-09-18T07:44:02.025 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.1', '2', '5', '7', '3', '6', '0'] 2024-09-18T07:44:02.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 3.1 2 5 7 3 6 0 2024-09-18T07:44:02.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:02.069 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 52 16 91251 91235 109.622 74.125 0.00415067 0.00911417 2024-09-18T07:44:02.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:02.118 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:02.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:02.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:01 smithi137 ceph-mon[29408]: from='client.16218 v1:172.21.15.137:0/3658401016' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:44:02.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:01 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/940982451' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:44:02.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:01 smithi137 ceph-mon[29408]: from='client.26164 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:44:02.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:01 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:44:02.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:01 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:44:02.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:01 smithi137 ceph-mon[39175]: from='client.16218 v1:172.21.15.137:0/3658401016' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:44:02.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:01 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/940982451' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:44:02.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:01 smithi137 ceph-mon[39175]: from='client.26164 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:44:02.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:01 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:44:02.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:01 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:44:02.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:01 smithi203 ceph-mon[32237]: from='client.16218 v1:172.21.15.137:0/3658401016' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:44:02.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:01 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/940982451' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:44:02.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:01 smithi203 ceph-mon[32237]: from='client.26164 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:44:02.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:01 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:44:02.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:01 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:44:02.219 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:02.247 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:02.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:02.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:02.349 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:02.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:02.405 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:02.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:02.506 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:02.532 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:02.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:02.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:02.634 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:02.671 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:02.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:02.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:02.774 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:02.800 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:02.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:02.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:02.902 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:02.925 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:02.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:02.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:02.927 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:03.027 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:03.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:03.061 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:03.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:03.070 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 53 16 92447 92431 108.962 74.75 0.00722882 0.00917272 2024-09-18T07:44:03.155 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:44:03.155 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:44:03.155 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:44:03.156 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:44:03.156 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:44:03.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:03.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:02 smithi137 ceph-mon[29408]: pgmap v684: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s; 43409/257928 objects degraded (16.830%) 2024-09-18T07:44:03.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:02 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:44:03.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:02 smithi137 ceph-mon[29408]: from='client.26164 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:44:03.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:02 smithi137 ceph-mon[29408]: osdmap e377: 8 total, 7 up, 8 in 2024-09-18T07:44:03.172 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:02 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:44:03.172 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:02 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:44:03.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:02 smithi137 ceph-mon[39175]: pgmap v684: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s; 43409/257928 objects degraded (16.830%) 2024-09-18T07:44:03.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:02 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:44:03.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:02 smithi137 ceph-mon[39175]: from='client.26164 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:44:03.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:02 smithi137 ceph-mon[39175]: osdmap e377: 8 total, 7 up, 8 in 2024-09-18T07:44:03.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:02 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:44:03.172 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:02 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:44:03.200 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:03.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:03.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:03.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:02 smithi203 ceph-mon[32237]: pgmap v684: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s; 43409/257928 objects degraded (16.830%) 2024-09-18T07:44:03.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:02 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:44:03.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:02 smithi203 ceph-mon[32237]: from='client.26164 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:44:03.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:02 smithi203 ceph-mon[32237]: osdmap e377: 8 total, 7 up, 8 in 2024-09-18T07:44:03.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:02 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:44:03.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:02 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:44:03.242 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:03.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:03.348 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:03.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:03.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:03.450 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:03.478 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:03.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:03.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:03.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:03.606 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:03.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:03.707 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:03.747 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:03.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:03.848 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:03.890 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:03.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:03.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:03.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:04.028 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:03 smithi137 ceph-mon[29408]: 4.0 scrub starts 2024-09-18T07:44:04.028 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:03 smithi137 ceph-mon[29408]: 4.0 scrub ok 2024-09-18T07:44:04.028 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:03 smithi137 ceph-mon[29408]: osdmap e378: 8 total, 7 up, 8 in 2024-09-18T07:44:04.029 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:03 smithi137 ceph-mon[39175]: 4.0 scrub starts 2024-09-18T07:44:04.029 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:03 smithi137 ceph-mon[39175]: 4.0 scrub ok 2024-09-18T07:44:04.029 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:03 smithi137 ceph-mon[39175]: osdmap e378: 8 total, 7 up, 8 in 2024-09-18T07:44:04.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:04.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:04.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:04.070 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 54 16 93585 93569 108.262 71.125 0.0119328 0.0092322 2024-09-18T07:44:04.094 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:44:04.095 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:44:04.126 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:44:04.126 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:44:04.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:04.162 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:44:04.162 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:44:04.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:04.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:04.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:04.195 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:44:04.195 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:44:04.201 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-18T07:44:04.231 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:44:04.231 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:44:04.268 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:44:04.269 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:44:04.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:04.292 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:04.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:04.298 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:44:04.298 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:44:04.361 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:03 smithi203 ceph-mon[32237]: 4.0 scrub starts 2024-09-18T07:44:04.361 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:03 smithi203 ceph-mon[32237]: 4.0 scrub ok 2024-09-18T07:44:04.361 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:03 smithi203 ceph-mon[32237]: osdmap e378: 8 total, 7 up, 8 in 2024-09-18T07:44:04.394 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:04.430 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:04.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:04.532 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:04.562 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:04.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:04.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:04.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:04.701 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:04.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:04.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:04.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:04.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:04.881 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:04.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:04.882 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T07:44:04.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:44:04.968 INFO:teuthology.orchestra.run.smithi137.stderr:set 3.1 pg_upmap_items mapping to [2->5,7->3,6->0] 2024-09-18T07:44:04.982 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:05.010 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:05.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:05.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:05.070 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 55 16 94858 94842 107.74 79.5625 0.00941974 0.00927719 2024-09-18T07:44:05.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:05.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:05.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:05.185 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:44:05.186 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:44:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:04 smithi203 ceph-mon[32237]: pgmap v687: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 99 MiB/s wr, 1.59k op/s; 45702/274347 objects degraded (16.658%) 2024-09-18T07:44:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:04 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3142630300' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.1", "id": [2, 5, 7, 3, 6, 0]} : dispatch 2024-09-18T07:44:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:04 smithi203 ceph-mon[32237]: from='client.26170 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.1", "id": [2, 5, 7, 3, 6, 0]} : dispatch 2024-09-18T07:44:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:04 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1771600973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:04 smithi203 ceph-mon[32237]: 4.12 scrub starts 2024-09-18T07:44:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:04 smithi203 ceph-mon[32237]: 4.12 scrub ok 2024-09-18T07:44:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:04 smithi203 ceph-mon[32237]: from='client.26170 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.1", "id": [2, 5, 7, 3, 6, 0]}]': finished 2024-09-18T07:44:05.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:04 smithi203 ceph-mon[32237]: osdmap e379: 8 total, 7 up, 8 in 2024-09-18T07:44:05.260 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:05.283 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:44:05.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:05.293 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:05.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:05.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:05.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:04 smithi137 ceph-mon[29408]: pgmap v687: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 99 MiB/s wr, 1.59k op/s; 45702/274347 objects degraded (16.658%) 2024-09-18T07:44:05.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:04 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3142630300' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.1", "id": [2, 5, 7, 3, 6, 0]} : dispatch 2024-09-18T07:44:05.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:04 smithi137 ceph-mon[29408]: from='client.26170 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.1", "id": [2, 5, 7, 3, 6, 0]} : dispatch 2024-09-18T07:44:05.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:04 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1771600973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:05.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:04 smithi137 ceph-mon[29408]: 4.12 scrub starts 2024-09-18T07:44:05.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:04 smithi137 ceph-mon[29408]: 4.12 scrub ok 2024-09-18T07:44:05.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:04 smithi137 ceph-mon[29408]: from='client.26170 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.1", "id": [2, 5, 7, 3, 6, 0]}]': finished 2024-09-18T07:44:05.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:04 smithi137 ceph-mon[29408]: osdmap e379: 8 total, 7 up, 8 in 2024-09-18T07:44:05.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:04 smithi137 ceph-mon[39175]: pgmap v687: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 99 MiB/s wr, 1.59k op/s; 45702/274347 objects degraded (16.658%) 2024-09-18T07:44:05.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:04 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3142630300' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.1", "id": [2, 5, 7, 3, 6, 0]} : dispatch 2024-09-18T07:44:05.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:04 smithi137 ceph-mon[39175]: from='client.26170 ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.1", "id": [2, 5, 7, 3, 6, 0]} : dispatch 2024-09-18T07:44:05.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:04 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1771600973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:05.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:04 smithi137 ceph-mon[39175]: 4.12 scrub starts 2024-09-18T07:44:05.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:04 smithi137 ceph-mon[39175]: 4.12 scrub ok 2024-09-18T07:44:05.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:04 smithi137 ceph-mon[39175]: from='client.26170 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.1", "id": [2, 5, 7, 3, 6, 0]}]': finished 2024-09-18T07:44:05.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:04 smithi137 ceph-mon[39175]: osdmap e379: 8 total, 7 up, 8 in 2024-09-18T07:44:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:05.437 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:05.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:05.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:05.579 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:05.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:05.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:05.681 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:05.711 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:05.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:05.813 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:05.854 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:05.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:05.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:05.879 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:44:05.880 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:44:05.956 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:06.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:06.008 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:06.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:06.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:06.073 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 56 16 96095 96079 107.192 77.3125 0.0100904 0.00932474 2024-09-18T07:44:06.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:06.154 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:06.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:06.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:06.184 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:05 smithi137 ceph-mon[29408]: 4.8 scrub starts 2024-09-18T07:44:06.184 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:05 smithi137 ceph-mon[29408]: 4.8 scrub ok 2024-09-18T07:44:06.184 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:05 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 45702/274347 objects degraded (16.658%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:06.184 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:05 smithi137 ceph-mon[29408]: osdmap e380: 8 total, 7 up, 8 in 2024-09-18T07:44:06.184 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:05 smithi137 ceph-mon[39175]: 4.8 scrub starts 2024-09-18T07:44:06.185 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:05 smithi137 ceph-mon[39175]: 4.8 scrub ok 2024-09-18T07:44:06.185 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:05 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 45702/274347 objects degraded (16.658%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:06.185 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:05 smithi137 ceph-mon[39175]: osdmap e380: 8 total, 7 up, 8 in 2024-09-18T07:44:06.257 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:06.286 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:06.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:06.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:06.388 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:06.416 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:06.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:06.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:06.441 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:05 smithi203 ceph-mon[32237]: 4.8 scrub starts 2024-09-18T07:44:06.441 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:05 smithi203 ceph-mon[32237]: 4.8 scrub ok 2024-09-18T07:44:06.441 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:05 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 45702/274347 objects degraded (16.658%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:06.441 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:05 smithi203 ceph-mon[32237]: osdmap e380: 8 total, 7 up, 8 in 2024-09-18T07:44:06.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:06.564 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:06.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:06.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:06.647 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T07:44:06.665 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:06.710 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:06.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:06.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:06.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:06.869 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:06.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:06.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:06.971 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:07.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:07.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:07.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:07.027 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:44:07.073 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 57 16 97168 97152 106.488 67.0625 0.0214074 0.00938579 2024-09-18T07:44:07.113 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:07.152 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:07.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:07.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:07.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:44:07.253 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:07.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:07.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:07.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:07.301 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:07 smithi203 ceph-mon[32237]: pgmap v690: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 85 MiB/s wr, 1.37k op/s; 45702/274347 objects degraded (16.658%) 2024-09-18T07:44:07.301 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:07 smithi203 ceph-mon[32237]: 4.6 scrub starts 2024-09-18T07:44:07.301 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:07 smithi203 ceph-mon[32237]: 4.6 scrub ok 2024-09-18T07:44:07.301 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:07 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:44:07.301 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:07 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:44:07.301 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:07 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:44:07.301 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:07 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:44:07.301 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:07 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/955221239' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:44:07.302 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:07 smithi203 ceph-mon[32237]: from='client.26179 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:44:07.398 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:07.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:44:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:44:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[29408]: pgmap v690: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 85 MiB/s wr, 1.37k op/s; 45702/274347 objects degraded (16.658%) 2024-09-18T07:44:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[29408]: 4.6 scrub starts 2024-09-18T07:44:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[29408]: 4.6 scrub ok 2024-09-18T07:44:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:44:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:44:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:44:07.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:44:07.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/955221239' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:44:07.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[29408]: from='client.26179 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:44:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[39175]: pgmap v690: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 85 MiB/s wr, 1.37k op/s; 45702/274347 objects degraded (16.658%) 2024-09-18T07:44:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[39175]: 4.6 scrub starts 2024-09-18T07:44:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[39175]: 4.6 scrub ok 2024-09-18T07:44:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:44:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:44:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:44:07.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:44:07.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/955221239' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:44:07.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:07 smithi137 ceph-mon[39175]: from='client.26179 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:44:07.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:07.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:07.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:07.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:07.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:07.585 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:07.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:07.586 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:44:07.586 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:44:07.617 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:07.687 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:07.721 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:07.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:07.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:07.823 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:07.852 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:07.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:07.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:07.925 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:44:07.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:44:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:44:07.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:08.017 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:08.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:08.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:08.073 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 58 16 98332 98316 105.906 72.75 0.0100147 0.00943754 2024-09-18T07:44:08.119 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:08.147 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:08.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:08.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:08.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:08 smithi203 ceph-mon[32237]: 2.2 scrub starts 2024-09-18T07:44:08.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:08 smithi203 ceph-mon[32237]: 2.2 scrub ok 2024-09-18T07:44:08.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:08 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:44:08.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:08 smithi203 ceph-mon[32237]: from='client.26179 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:44:08.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:08 smithi203 ceph-mon[32237]: osdmap e381: 8 total, 7 up, 8 in 2024-09-18T07:44:08.249 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:08.285 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:08.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:08.287 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T07:44:08.306 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:08 smithi137 ceph-mon[29408]: 2.2 scrub starts 2024-09-18T07:44:08.306 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:08 smithi137 ceph-mon[29408]: 2.2 scrub ok 2024-09-18T07:44:08.306 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:08 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:44:08.306 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:08 smithi137 ceph-mon[29408]: from='client.26179 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:44:08.306 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:08 smithi137 ceph-mon[29408]: osdmap e381: 8 total, 7 up, 8 in 2024-09-18T07:44:08.307 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:08 smithi137 ceph-mon[39175]: 2.2 scrub starts 2024-09-18T07:44:08.307 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:08 smithi137 ceph-mon[39175]: 2.2 scrub ok 2024-09-18T07:44:08.307 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:08 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:44:08.307 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:08 smithi137 ceph-mon[39175]: from='client.26179 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:44:08.307 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:08 smithi137 ceph-mon[39175]: osdmap e381: 8 total, 7 up, 8 in 2024-09-18T07:44:08.387 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:08.417 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:08.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:08.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:08.428 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:44:08.429 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:44:08.429 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:44:08.429 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:44:08.429 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:44:08.429 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:44:08.429 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:44:08.429 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:44:08.429 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:44:08.429 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-18T07:44:08.429 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:44:08.441 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:44:08.442 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:44:08.473 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:44:08.473 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:44:08.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:08.555 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:08.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:08.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:08.652 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:44:08.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:08.698 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:08.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:08.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:08.799 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:08.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:08.827 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:08.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:08.928 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:08.966 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:08.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:08.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:09.068 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:09.073 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 59 16 99590 99574 105.444 78.625 0.0108761 0.00947893 2024-09-18T07:44:09.092 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:09.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:09.194 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:09.227 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:09.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:09.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:09.275 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:44:09.275 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:44:09.275 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:44:09.275 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:44:09.275 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:44:09.329 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:09.357 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:09.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:09.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:09.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:09 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:09.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:09 smithi137 ceph-mon[29408]: pgmap v691: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 881 B/s rd, 80 MiB/s wr, 1.28k op/s; 47917/288618 objects degraded (16.602%) 2024-09-18T07:44:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:09 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1514753997' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:09 smithi137 ceph-mon[29408]: 3.17 scrub starts 2024-09-18T07:44:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:09 smithi137 ceph-mon[29408]: 3.17 scrub ok 2024-09-18T07:44:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:09 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:09 smithi137 ceph-mon[39175]: pgmap v691: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 881 B/s rd, 80 MiB/s wr, 1.28k op/s; 47917/288618 objects degraded (16.602%) 2024-09-18T07:44:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:09 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1514753997' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:09 smithi137 ceph-mon[39175]: 3.17 scrub starts 2024-09-18T07:44:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:09 smithi137 ceph-mon[39175]: 3.17 scrub ok 2024-09-18T07:44:09.460 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:09.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:09 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:09 smithi203 ceph-mon[32237]: pgmap v691: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 881 B/s rd, 80 MiB/s wr, 1.28k op/s; 47917/288618 objects degraded (16.602%) 2024-09-18T07:44:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:09 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1514753997' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:09 smithi203 ceph-mon[32237]: 3.17 scrub starts 2024-09-18T07:44:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:09 smithi203 ceph-mon[32237]: 3.17 scrub ok 2024-09-18T07:44:09.489 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:09.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:09.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:09.613 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:09.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:09.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:09.714 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:09.753 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:09.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:09.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:09.833 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:44:09.834 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:44:09.855 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:09.880 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:09.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:09.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:09.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:10.020 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:10.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:10.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:10.073 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:44:10.071901+0000 min lat: 0.00105851 max lat: 0.230749 avg lat: 0.00951237 2024-09-18T07:44:10.073 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:44:10.074 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 60 15 100918 100903 105.071 83.0625 0.0319033 0.00951237 2024-09-18T07:44:10.122 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:10.150 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:10.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:10.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:10.252 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:10.283 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:10.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:10.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:10.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:10.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:10 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 47917/288618 objects degraded (16.602%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:10.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:10 smithi137 ceph-mon[29408]: osdmap e382: 8 total, 7 up, 8 in 2024-09-18T07:44:10.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:10 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 47917/288618 objects degraded (16.602%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:10.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:10 smithi137 ceph-mon[39175]: osdmap e382: 8 total, 7 up, 8 in 2024-09-18T07:44:10.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:10.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:10.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:10 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 47917/288618 objects degraded (16.602%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:10 smithi203 ceph-mon[32237]: osdmap e382: 8 total, 7 up, 8 in 2024-09-18T07:44:10.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:10.561 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:10.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:10.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:10.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:10.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:10.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:10.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:10.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:10.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:10.846 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:10.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:10.947 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:10.975 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:10.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:10.989 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-09-18T07:44:10.989 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:44:10.989 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-18T07:44:10.989 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-18T07:44:11.074 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 61 16 102225 102209 104.686 81.625 0.00754793 0.00954791 2024-09-18T07:44:11.077 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:11.082 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:44:11.104 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:11.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:11.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:11.206 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:11.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:11.237 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:11.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:11.338 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:11.365 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:11.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:11.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:11.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:11 smithi137 ceph-mon[29408]: pgmap v693: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 981 op/s; 48705/292017 objects degraded (16.679%) 2024-09-18T07:44:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:11 smithi137 ceph-mon[39175]: pgmap v693: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 981 op/s; 48705/292017 objects degraded (16.679%) 2024-09-18T07:44:11.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:11 smithi203 ceph-mon[32237]: pgmap v693: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 5.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 981 op/s; 48705/292017 objects degraded (16.679%) 2024-09-18T07:44:11.462 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:11.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:11.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:11.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:11.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:11.617 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T07:44:11.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:11.660 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:11.677 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:11.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:11.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:11.779 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:11.815 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:11.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:11.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:11.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:44:11.917 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:11.939 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:44:11.949 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:11.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:12.051 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:12.073 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 62 16 103311 103295 104.093 67.875 0.0141899 0.009602 2024-09-18T07:44:12.076 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:12.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:12.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:12.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:12 smithi137 ceph-mon[29408]: 3.4 scrub starts 2024-09-18T07:44:12.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:12 smithi137 ceph-mon[29408]: 3.4 scrub ok 2024-09-18T07:44:12.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:12 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:44:12.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:44:12.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:12 smithi137 ceph-mon[39175]: 3.4 scrub starts 2024-09-18T07:44:12.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:12 smithi137 ceph-mon[39175]: 3.4 scrub ok 2024-09-18T07:44:12.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:12 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:44:12.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:44:12.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:12.214 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:12.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:12.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:12.349 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:12.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:12.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:12.375 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:12 smithi203 ceph-mon[32237]: 3.4 scrub starts 2024-09-18T07:44:12.375 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:12 smithi203 ceph-mon[32237]: 3.4 scrub ok 2024-09-18T07:44:12.375 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:12 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:44:12.375 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:44:12.451 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:12.495 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:12.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:12.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:12.541 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:44:12.541 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:44:12.541 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:44:12.541 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:44:12.541 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.541 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153406 7.7 7:e5535fc6:::benchmark_data_smithi137_74180_object48990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-18T07:44:12.541 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.169823+0000", 2024-09-18T07:44:12.541 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 570.34556198300004, 2024-09-18T07:44:12.542 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.083783358, 2024-09-18T07:44:12.542 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.542 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.542 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:44:12.542 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.542 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:44:12.542 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:12.542 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153406 2024-09-18T07:44:12.542 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.542 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.542 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.542 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.542 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169823+0000", 2024-09-18T07:44:12.543 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.543 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.543 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.543 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.543 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169823+0000", 2024-09-18T07:44:12.543 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.543 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.543 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.543 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.543 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169826+0000", 2024-09-18T07:44:12.543 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1229999999999998e-06 2024-09-18T07:44:12.543 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.543 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.544 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.544 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169833+0000", 2024-09-18T07:44:12.544 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6869999999999994e-06 2024-09-18T07:44:12.544 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.544 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.544 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.544 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169851+0000", 2024-09-18T07:44:12.544 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7978999999999998e-05 2024-09-18T07:44:12.544 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.544 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.544 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.544 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169867+0000", 2024-09-18T07:44:12.544 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5610000000000001e-05 2024-09-18T07:44:12.545 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.545 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.545 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.545 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.169944+0000", 2024-09-18T07:44:12.545 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6998999999999999e-05 2024-09-18T07:44:12.545 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.545 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.545 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:12.545 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.170029+0000", 2024-09-18T07:44:12.545 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4765999999999994e-05 2024-09-18T07:44:12.545 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.545 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.546 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:44:12.546 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.170285+0000", 2024-09-18T07:44:12.546 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025587199999999998 2024-09-18T07:44:12.546 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.546 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.546 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:12.546 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.171936+0000", 2024-09-18T07:44:12.546 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001651189 2024-09-18T07:44:12.546 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.546 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.546 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.546 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.194310+0000", 2024-09-18T07:44:12.546 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.022374616999999999 2024-09-18T07:44:12.546 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.547 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.547 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.547 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238955+0000", 2024-09-18T07:44:12.547 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.044644716000000001 2024-09-18T07:44:12.547 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.547 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.547 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:44:12.547 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238967+0000", 2024-09-18T07:44:12.547 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1933000000000001e-05 2024-09-18T07:44:12.547 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.547 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.547 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.547 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.253578+0000", 2024-09-18T07:44:12.547 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0146105599999999 2024-09-18T07:44:12.547 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.548 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.548 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.548 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.253607+0000", 2024-09-18T07:44:12.548 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9306999999999999e-05 2024-09-18T07:44:12.548 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.548 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.548 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.548 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.548 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.548 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153419 7.17 7:e887289b:::benchmark_data_smithi137_74180_object49003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-18T07:44:12.548 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.193652+0000", 2024-09-18T07:44:12.548 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 570.32173315900002, 2024-09-18T07:44:12.548 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0605730499999999, 2024-09-18T07:44:12.548 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.549 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.549 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:44:12.549 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.549 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:44:12.549 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:12.549 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153419 2024-09-18T07:44:12.549 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.549 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.549 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.549 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.549 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193652+0000", 2024-09-18T07:44:12.549 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.549 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.549 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.550 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.550 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193652+0000", 2024-09-18T07:44:12.550 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.550 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.550 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.550 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.550 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193654+0000", 2024-09-18T07:44:12.550 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.683e-06 2024-09-18T07:44:12.550 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.550 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.550 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.550 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193658+0000", 2024-09-18T07:44:12.550 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6619999999999998e-06 2024-09-18T07:44:12.550 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.551 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.551 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.551 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193667+0000", 2024-09-18T07:44:12.551 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0389999999999993e-06 2024-09-18T07:44:12.551 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.551 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.551 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.551 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193671+0000", 2024-09-18T07:44:12.551 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6789999999999996e-06 2024-09-18T07:44:12.551 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.551 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.551 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.551 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193715+0000", 2024-09-18T07:44:12.551 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3186000000000003e-05 2024-09-18T07:44:12.552 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.552 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.552 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:12.552 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.193799+0000", 2024-09-18T07:44:12.552 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4172999999999995e-05 2024-09-18T07:44:12.552 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.552 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.552 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:44:12.552 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.194037+0000", 2024-09-18T07:44:12.552 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023850299999999999 2024-09-18T07:44:12.552 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.552 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.552 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:12.552 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.195050+0000", 2024-09-18T07:44:12.553 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010123269999999999 2024-09-18T07:44:12.553 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.553 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.553 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.553 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.198824+0000", 2024-09-18T07:44:12.553 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0037742840000000001 2024-09-18T07:44:12.553 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.553 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.553 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.553 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238955+0000", 2024-09-18T07:44:12.553 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.040130714999999997 2024-09-18T07:44:12.553 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.554 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.554 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:44:12.554 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.238967+0000", 2024-09-18T07:44:12.554 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2665e-05 2024-09-18T07:44:12.554 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.554 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.554 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.554 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254201+0000", 2024-09-18T07:44:12.554 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0152341279999999 2024-09-18T07:44:12.554 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.554 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.554 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.554 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254225+0000", 2024-09-18T07:44:12.555 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4006e-05 2024-09-18T07:44:12.555 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.555 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.555 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.555 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.555 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.555 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153453 7.7 7.146f22e7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:44:12.555 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.282355+0000", 2024-09-18T07:44:12.555 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 570.23303027199995, 2024-09-18T07:44:12.555 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97169607199999997, 2024-09-18T07:44:12.555 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.555 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.556 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:44:12.556 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.556 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:44:12.556 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:12.556 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153453 2024-09-18T07:44:12.556 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.556 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.556 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.556 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.556 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282355+0000", 2024-09-18T07:44:12.556 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.556 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.556 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.557 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.557 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282355+0000", 2024-09-18T07:44:12.557 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.557 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.557 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.557 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.557 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282357+0000", 2024-09-18T07:44:12.557 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2110000000000001e-06 2024-09-18T07:44:12.557 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.557 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.557 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.557 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282363+0000", 2024-09-18T07:44:12.558 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6740000000000002e-06 2024-09-18T07:44:12.558 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.558 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.558 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.558 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282377+0000", 2024-09-18T07:44:12.558 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4039000000000001e-05 2024-09-18T07:44:12.558 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.558 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.558 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.558 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282384+0000", 2024-09-18T07:44:12.558 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5769999999999999e-06 2024-09-18T07:44:12.558 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.558 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.559 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.559 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282438+0000", 2024-09-18T07:44:12.559 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4083999999999999e-05 2024-09-18T07:44:12.559 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.559 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.559 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:44:12.559 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.282445+0000", 2024-09-18T07:44:12.559 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7030000000000006e-06 2024-09-18T07:44:12.559 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.559 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.559 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.559 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254044+0000", 2024-09-18T07:44:12.559 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97159860200000003 2024-09-18T07:44:12.559 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.560 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.560 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.560 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254051+0000", 2024-09-18T07:44:12.560 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.182e-06 2024-09-18T07:44:12.560 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.560 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.560 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.560 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.560 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.560 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153481 7.7 7.33bf19e7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:44:12.560 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.317589+0000", 2024-09-18T07:44:12.560 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 570.19779626800005, 2024-09-18T07:44:12.561 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93650330299999995, 2024-09-18T07:44:12.561 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.561 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.561 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:44:12.561 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.561 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:44:12.561 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:12.561 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153481 2024-09-18T07:44:12.561 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.561 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.561 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.561 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.561 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317589+0000", 2024-09-18T07:44:12.561 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.562 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.562 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.562 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.562 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317589+0000", 2024-09-18T07:44:12.562 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.562 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.562 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.562 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.562 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317590+0000", 2024-09-18T07:44:12.562 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6499999999999999e-07 2024-09-18T07:44:12.562 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.562 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.562 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.563 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317593+0000", 2024-09-18T07:44:12.563 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7870000000000002e-06 2024-09-18T07:44:12.563 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.563 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.563 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.563 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317604+0000", 2024-09-18T07:44:12.563 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0953e-05 2024-09-18T07:44:12.563 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.563 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.563 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.563 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317609+0000", 2024-09-18T07:44:12.563 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4099999999999999e-06 2024-09-18T07:44:12.563 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.564 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.564 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.564 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317654+0000", 2024-09-18T07:44:12.564 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5333000000000001e-05 2024-09-18T07:44:12.564 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.564 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.564 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:44:12.564 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.317661+0000", 2024-09-18T07:44:12.564 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8310000000000003e-06 2024-09-18T07:44:12.564 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.564 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.564 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.564 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254083+0000", 2024-09-18T07:44:12.565 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93642186500000002 2024-09-18T07:44:12.565 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.565 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.565 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.565 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254093+0000", 2024-09-18T07:44:12.565 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4590000000000004e-06 2024-09-18T07:44:12.565 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.565 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.565 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.565 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.565 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.565 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153485 7.7 7.c6dac07 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:44:12.565 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.328201+0000", 2024-09-18T07:44:12.566 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 570.18718433100003, 2024-09-18T07:44:12.566 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.925907068, 2024-09-18T07:44:12.566 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.566 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.566 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:44:12.566 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.566 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:44:12.566 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:12.566 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153485 2024-09-18T07:44:12.566 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.566 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.566 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.566 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.567 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328201+0000", 2024-09-18T07:44:12.567 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.567 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.567 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.567 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.567 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328201+0000", 2024-09-18T07:44:12.567 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.567 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.567 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.567 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.567 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328203+0000", 2024-09-18T07:44:12.567 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3769999999999999e-06 2024-09-18T07:44:12.567 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.567 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.568 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.568 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328207+0000", 2024-09-18T07:44:12.568 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5120000000000002e-06 2024-09-18T07:44:12.568 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.568 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.568 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.568 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328217+0000", 2024-09-18T07:44:12.568 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0438000000000001e-05 2024-09-18T07:44:12.568 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.568 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.568 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.568 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328223+0000", 2024-09-18T07:44:12.568 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7130000000000002e-06 2024-09-18T07:44:12.569 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.569 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.569 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.569 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328270+0000", 2024-09-18T07:44:12.569 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7243000000000002e-05 2024-09-18T07:44:12.569 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.569 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.569 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:44:12.569 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.328277+0000", 2024-09-18T07:44:12.569 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9229999999999996e-06 2024-09-18T07:44:12.569 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.569 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.569 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.570 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254104+0000", 2024-09-18T07:44:12.570 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92582690099999998 2024-09-18T07:44:12.570 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.570 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.570 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.570 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254108+0000", 2024-09-18T07:44:12.570 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9609999999999999e-06 2024-09-18T07:44:12.570 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.570 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.570 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.570 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.570 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.570 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153505 7.7 7.fb8b6b67 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:44:12.570 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.350178+0000", 2024-09-18T07:44:12.571 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 570.16520750200004, 2024-09-18T07:44:12.571 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90398860000000003, 2024-09-18T07:44:12.571 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.571 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.571 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:44:12.571 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.571 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:44:12.571 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:12.571 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153505 2024-09-18T07:44:12.571 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.571 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.571 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.571 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.571 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350178+0000", 2024-09-18T07:44:12.572 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.572 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.572 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.572 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.572 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350178+0000", 2024-09-18T07:44:12.572 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.572 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.572 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.572 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.572 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350179+0000", 2024-09-18T07:44:12.572 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.454e-06 2024-09-18T07:44:12.572 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.572 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.572 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.573 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350183+0000", 2024-09-18T07:44:12.573 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8480000000000003e-06 2024-09-18T07:44:12.573 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.573 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.573 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.573 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350192+0000", 2024-09-18T07:44:12.573 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0899999999999994e-06 2024-09-18T07:44:12.573 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.573 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.573 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.573 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350200+0000", 2024-09-18T07:44:12.573 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.328e-06 2024-09-18T07:44:12.574 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.574 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.574 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.574 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350252+0000", 2024-09-18T07:44:12.574 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2589000000000001e-05 2024-09-18T07:44:12.574 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.574 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.574 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:44:12.574 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.350263+0000", 2024-09-18T07:44:12.574 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0831e-05 2024-09-18T07:44:12.574 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.574 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.574 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.575 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254124+0000", 2024-09-18T07:44:12.575 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90386114100000003 2024-09-18T07:44:12.575 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.575 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.575 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.575 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254167+0000", 2024-09-18T07:44:12.575 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2318999999999999e-05 2024-09-18T07:44:12.575 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.575 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.575 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.575 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.575 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.575 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:153519 7.17 7.5ef16c17 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-18T07:44:12.576 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:34:42.363819+0000", 2024-09-18T07:44:12.576 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 570.15156674900004, 2024-09-18T07:44:12.576 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.89046919899999999, 2024-09-18T07:44:12.576 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.576 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.576 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:44:12.576 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.576 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:44:12.576 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:12.576 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 153519 2024-09-18T07:44:12.576 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.576 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.576 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.576 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.577 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363819+0000", 2024-09-18T07:44:12.577 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.577 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.577 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.577 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.577 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363819+0000", 2024-09-18T07:44:12.577 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.577 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.577 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.577 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.577 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363820+0000", 2024-09-18T07:44:12.577 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.325e-06 2024-09-18T07:44:12.577 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.577 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.578 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.578 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363824+0000", 2024-09-18T07:44:12.578 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2459999999999997e-06 2024-09-18T07:44:12.578 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.578 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.578 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.578 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363834+0000", 2024-09-18T07:44:12.578 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6399999999999992e-06 2024-09-18T07:44:12.578 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.578 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.578 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.578 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363838+0000", 2024-09-18T07:44:12.578 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2509999999999996e-06 2024-09-18T07:44:12.578 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.579 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.579 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.579 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363900+0000", 2024-09-18T07:44:12.579 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1902000000000005e-05 2024-09-18T07:44:12.579 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.579 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.579 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:44:12.579 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:42.363910+0000", 2024-09-18T07:44:12.579 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6879999999999994e-06 2024-09-18T07:44:12.579 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.579 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.579 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.579 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254254+0000", 2024-09-18T07:44:12.580 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.89034411800000002 2024-09-18T07:44:12.580 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.580 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.580 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.580 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:34:43.254288+0000", 2024-09-18T07:44:12.580 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4029000000000002e-05 2024-09-18T07:44:12.580 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.580 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.580 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.580 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.580 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.580 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:162338 7.c 7:39946bc9:::benchmark_data_smithi137_74180_object57922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:44:12.580 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:01.426143+0000", 2024-09-18T07:44:12.581 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 551.08924293799998, 2024-09-18T07:44:12.581 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0750540070000001, 2024-09-18T07:44:12.581 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.581 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.581 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:12.581 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.581 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:44:12.581 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:12.581 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 162338 2024-09-18T07:44:12.581 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.581 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.581 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.581 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.581 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426143+0000", 2024-09-18T07:44:12.582 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.582 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.582 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.582 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.582 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426143+0000", 2024-09-18T07:44:12.582 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.582 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.582 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.582 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.582 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426145+0000", 2024-09-18T07:44:12.582 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4059999999999999e-06 2024-09-18T07:44:12.582 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.582 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.583 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.583 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426151+0000", 2024-09-18T07:44:12.583 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1260000000000003e-06 2024-09-18T07:44:12.583 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.583 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.583 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.583 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426174+0000", 2024-09-18T07:44:12.583 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3255e-05 2024-09-18T07:44:12.583 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.583 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.583 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.583 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426182+0000", 2024-09-18T07:44:12.583 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4189999999999996e-06 2024-09-18T07:44:12.583 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.584 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.584 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.584 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426256+0000", 2024-09-18T07:44:12.584 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4713000000000001e-05 2024-09-18T07:44:12.584 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.584 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.584 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:44:12.584 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.426270+0000", 2024-09-18T07:44:12.584 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3405e-05 2024-09-18T07:44:12.584 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.584 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.584 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.584 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.491167+0000", 2024-09-18T07:44:12.585 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.064897582 2024-09-18T07:44:12.585 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.585 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.585 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:12.585 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.491280+0000", 2024-09-18T07:44:12.585 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011243099999999999 2024-09-18T07:44:12.585 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.585 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.585 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3,7", 2024-09-18T07:44:12.585 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.491357+0000", 2024-09-18T07:44:12.585 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7385999999999994e-05 2024-09-18T07:44:12.585 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.585 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.586 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.586 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496060+0000", 2024-09-18T07:44:12.586 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0047024600000000003 2024-09-18T07:44:12.586 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.586 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.586 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:12.586 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496072+0000", 2024-09-18T07:44:12.586 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2294e-05 2024-09-18T07:44:12.586 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.586 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.586 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.586 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496109+0000", 2024-09-18T07:44:12.586 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7218999999999997e-05 2024-09-18T07:44:12.586 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.587 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.587 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.587 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501071+0000", 2024-09-18T07:44:12.587 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0049614660000000003 2024-09-18T07:44:12.587 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.587 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.587 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:12.587 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501111+0000", 2024-09-18T07:44:12.587 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0776999999999999e-05 2024-09-18T07:44:12.587 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.587 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.587 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.587 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501197+0000", 2024-09-18T07:44:12.587 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5068000000000006e-05 2024-09-18T07:44:12.587 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.588 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.588 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.588 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.588 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.588 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.14865.0:162350 7.c 7:3843094b:::benchmark_data_smithi137_74180_object57934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:44:12.588 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:01.433525+0000", 2024-09-18T07:44:12.588 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 551.08186012800002, 2024-09-18T07:44:12.588 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0678760330000001, 2024-09-18T07:44:12.588 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.588 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.588 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:12.588 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.588 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.14865", 2024-09-18T07:44:12.589 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:12.589 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 162350 2024-09-18T07:44:12.589 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.589 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.589 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.589 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.589 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433525+0000", 2024-09-18T07:44:12.589 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.589 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.589 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.589 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.589 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433525+0000", 2024-09-18T07:44:12.589 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.589 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.590 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.590 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.590 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433527+0000", 2024-09-18T07:44:12.590 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.812e-06 2024-09-18T07:44:12.590 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.590 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.590 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.590 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433533+0000", 2024-09-18T07:44:12.590 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2920000000000001e-06 2024-09-18T07:44:12.590 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.590 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.590 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.590 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433544+0000", 2024-09-18T07:44:12.590 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0339999999999999e-05 2024-09-18T07:44:12.591 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.591 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.591 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.591 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433551+0000", 2024-09-18T07:44:12.591 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6490000000000008e-06 2024-09-18T07:44:12.591 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.591 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.591 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.591 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433620+0000", 2024-09-18T07:44:12.591 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8957999999999999e-05 2024-09-18T07:44:12.591 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.591 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.591 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:44:12.591 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.433631+0000", 2024-09-18T07:44:12.592 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0397000000000001e-05 2024-09-18T07:44:12.592 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.592 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.592 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.592 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.493805+0000", 2024-09-18T07:44:12.592 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0601744559999999 2024-09-18T07:44:12.592 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.592 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.592 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:12.592 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.493907+0000", 2024-09-18T07:44:12.592 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000101377 2024-09-18T07:44:12.592 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.592 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.592 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3,7", 2024-09-18T07:44:12.593 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.493992+0000", 2024-09-18T07:44:12.593 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5672000000000002e-05 2024-09-18T07:44:12.593 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.593 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.593 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.593 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.496146+0000", 2024-09-18T07:44:12.593 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021537470000000001 2024-09-18T07:44:12.593 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.593 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.593 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:12.593 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.498397+0000", 2024-09-18T07:44:12.593 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002250742 2024-09-18T07:44:12.593 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.593 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.594 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.594 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501029+0000", 2024-09-18T07:44:12.594 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026321780000000002 2024-09-18T07:44:12.594 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.594 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.594 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.594 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501274+0000", 2024-09-18T07:44:12.594 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000245472 2024-09-18T07:44:12.594 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.594 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.594 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:12.594 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501321+0000", 2024-09-18T07:44:12.594 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6405e-05 2024-09-18T07:44:12.594 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.595 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.595 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.595 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:02.501401+0000", 2024-09-18T07:44:12.595 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0536000000000003e-05 2024-09-18T07:44:12.595 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.595 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.595 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.595 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.595 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.595 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:71852 10.b 10:dda9ad8f:::benchmark_data_smithi137_151160_object71851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-09-18T07:44:12.595 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:43:46.387229+0000", 2024-09-18T07:44:12.595 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 26.128156830000002, 2024-09-18T07:44:12.595 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.23047490900000001, 2024-09-18T07:44:12.595 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.596 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.596 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:12.596 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.596 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:44:12.596 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:12.596 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 71852 2024-09-18T07:44:12.596 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.596 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.596 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.596 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.596 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.387229+0000", 2024-09-18T07:44:12.596 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.596 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.596 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.597 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.597 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.387229+0000", 2024-09-18T07:44:12.597 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.597 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.597 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.597 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.597 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.387230+0000", 2024-09-18T07:44:12.597 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5230000000000001e-06 2024-09-18T07:44:12.597 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.597 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.597 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.597 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.387287+0000", 2024-09-18T07:44:12.597 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7027000000000001e-05 2024-09-18T07:44:12.598 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.598 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.598 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.598 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.387299+0000", 2024-09-18T07:44:12.598 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2149e-05 2024-09-18T07:44:12.598 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.598 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.598 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.598 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.387307+0000", 2024-09-18T07:44:12.598 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2420000000000003e-06 2024-09-18T07:44:12.598 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.598 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.599 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:12.600 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.600 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.387358+0000", 2024-09-18T07:44:12.600 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0952999999999998e-05 2024-09-18T07:44:12.600 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.600 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.601 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:12.601 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.387478+0000", 2024-09-18T07:44:12.601 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000120114 2024-09-18T07:44:12.601 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.601 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.601 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:44:12.601 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.387600+0000", 2024-09-18T07:44:12.601 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012265999999999999 2024-09-18T07:44:12.601 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.601 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.601 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:12.601 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.389881+0000", 2024-09-18T07:44:12.601 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022809560000000002 2024-09-18T07:44:12.602 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.602 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.602 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.602 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.398844+0000", 2024-09-18T07:44:12.602 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0089622499999999997 2024-09-18T07:44:12.602 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.602 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.602 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.602 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.617611+0000", 2024-09-18T07:44:12.602 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.21876746799999999 2024-09-18T07:44:12.602 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.602 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.602 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:12.603 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.617661+0000", 2024-09-18T07:44:12.603 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0056000000000003e-05 2024-09-18T07:44:12.603 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.603 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.603 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.603 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.617704+0000", 2024-09-18T07:44:12.603 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2511e-05 2024-09-18T07:44:12.603 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.603 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.603 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.603 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.603 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.603 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:71865 10.0 10:019f6f6d:::benchmark_data_smithi137_151160_object71864:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-09-18T07:44:12.603 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:43:46.396463+0000", 2024-09-18T07:44:12.604 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 26.118922843, 2024-09-18T07:44:12.604 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.21735257199999999, 2024-09-18T07:44:12.604 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.604 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.604 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:12.604 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.604 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:44:12.604 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:12.604 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 71865 2024-09-18T07:44:12.604 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.604 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.604 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.604 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.604 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.396463+0000", 2024-09-18T07:44:12.605 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.605 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.605 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.605 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.605 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.396463+0000", 2024-09-18T07:44:12.605 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.605 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.605 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.605 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.605 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.396464+0000", 2024-09-18T07:44:12.605 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3549999999999999e-06 2024-09-18T07:44:12.605 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.605 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.606 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.606 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.396534+0000", 2024-09-18T07:44:12.606 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0439999999999996e-05 2024-09-18T07:44:12.606 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.606 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.606 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.606 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.396543+0000", 2024-09-18T07:44:12.606 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5639999999999997e-06 2024-09-18T07:44:12.606 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.606 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.606 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.606 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.396549+0000", 2024-09-18T07:44:12.606 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3330000000000002e-06 2024-09-18T07:44:12.607 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.607 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.607 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.607 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.396605+0000", 2024-09-18T07:44:12.607 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6107999999999999e-05 2024-09-18T07:44:12.607 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.607 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.607 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:12.607 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.396726+0000", 2024-09-18T07:44:12.607 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012072 2024-09-18T07:44:12.607 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.607 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.607 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1", 2024-09-18T07:44:12.608 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.396826+0000", 2024-09-18T07:44:12.608 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9525999999999995e-05 2024-09-18T07:44:12.608 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.608 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.608 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:12.608 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.398660+0000", 2024-09-18T07:44:12.608 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001834483 2024-09-18T07:44:12.608 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.608 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.608 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.608 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.613723+0000", 2024-09-18T07:44:12.608 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.21506330500000001 2024-09-18T07:44:12.608 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.609 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.609 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:12.609 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.613760+0000", 2024-09-18T07:44:12.609 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6443999999999999e-05 2024-09-18T07:44:12.609 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.609 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.609 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.609 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.613815+0000", 2024-09-18T07:44:12.609 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5294000000000003e-05 2024-09-18T07:44:12.609 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.609 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.609 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.609 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.610 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.610 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:71871 10.b 10:d6314bb9:::benchmark_data_smithi137_151160_object71870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-09-18T07:44:12.610 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:43:46.402491+0000", 2024-09-18T07:44:12.610 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 26.112894021999999, 2024-09-18T07:44:12.610 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.215376241, 2024-09-18T07:44:12.610 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.610 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.610 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:12.610 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.610 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:44:12.610 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:12.610 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 71871 2024-09-18T07:44:12.610 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.610 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.611 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.611 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.611 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.402491+0000", 2024-09-18T07:44:12.611 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.611 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.611 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.611 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.611 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.402491+0000", 2024-09-18T07:44:12.611 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.611 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.611 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.611 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.611 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.402492+0000", 2024-09-18T07:44:12.611 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8299999999999995e-07 2024-09-18T07:44:12.612 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.612 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.612 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.612 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.402545+0000", 2024-09-18T07:44:12.612 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2212000000000001e-05 2024-09-18T07:44:12.612 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.612 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.612 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.612 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.402552+0000", 2024-09-18T07:44:12.612 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6459999999999993e-06 2024-09-18T07:44:12.612 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.612 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.612 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.613 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.402559+0000", 2024-09-18T07:44:12.613 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4840000000000001e-06 2024-09-18T07:44:12.613 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.613 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.613 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.613 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.402642+0000", 2024-09-18T07:44:12.613 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3685999999999998e-05 2024-09-18T07:44:12.613 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.613 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.613 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:12.613 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.402773+0000", 2024-09-18T07:44:12.613 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013051299999999999 2024-09-18T07:44:12.613 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.614 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.614 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:44:12.614 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.402925+0000", 2024-09-18T07:44:12.614 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015202 2024-09-18T07:44:12.614 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.614 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.614 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:12.614 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.405641+0000", 2024-09-18T07:44:12.614 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027160140000000001 2024-09-18T07:44:12.614 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.614 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.614 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.614 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.406176+0000", 2024-09-18T07:44:12.614 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00053474100000000004 2024-09-18T07:44:12.615 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.615 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.615 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.615 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.617778+0000", 2024-09-18T07:44:12.615 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.211601918 2024-09-18T07:44:12.615 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.615 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.615 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:12.615 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.617827+0000", 2024-09-18T07:44:12.615 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.918e-05 2024-09-18T07:44:12.615 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.615 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.615 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.615 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.617868+0000", 2024-09-18T07:44:12.615 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0843999999999997e-05 2024-09-18T07:44:12.616 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.616 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.616 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.616 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.616 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.616 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:71876 10.b 10:d9fa1b9e:::benchmark_data_smithi137_151160_object71875:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-09-18T07:44:12.616 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:43:46.408187+0000", 2024-09-18T07:44:12.616 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 26.107198318999998, 2024-09-18T07:44:12.616 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.20989992299999999, 2024-09-18T07:44:12.616 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.616 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.616 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:12.616 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.616 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:44:12.617 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:12.617 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 71876 2024-09-18T07:44:12.617 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.617 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.617 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.617 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.617 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.408187+0000", 2024-09-18T07:44:12.617 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.617 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.617 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.617 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.617 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.408187+0000", 2024-09-18T07:44:12.618 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.618 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.618 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.618 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.618 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.408189+0000", 2024-09-18T07:44:12.618 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9410000000000001e-06 2024-09-18T07:44:12.618 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.618 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.618 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.618 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.408253+0000", 2024-09-18T07:44:12.618 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.389e-05 2024-09-18T07:44:12.618 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.618 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.619 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.619 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.408264+0000", 2024-09-18T07:44:12.619 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0743e-05 2024-09-18T07:44:12.619 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.619 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.619 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.619 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.408270+0000", 2024-09-18T07:44:12.619 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4420000000000003e-06 2024-09-18T07:44:12.619 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.619 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.619 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.619 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.408309+0000", 2024-09-18T07:44:12.619 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8470000000000003e-05 2024-09-18T07:44:12.619 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.620 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.620 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:12.620 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.408392+0000", 2024-09-18T07:44:12.620 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3385000000000002e-05 2024-09-18T07:44:12.620 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.620 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.620 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:44:12.620 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.408478+0000", 2024-09-18T07:44:12.620 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5630999999999994e-05 2024-09-18T07:44:12.620 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.620 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.620 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:12.620 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.411572+0000", 2024-09-18T07:44:12.620 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003094023 2024-09-18T07:44:12.621 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.621 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.621 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.621 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.416752+0000", 2024-09-18T07:44:12.621 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0051798369999999996 2024-09-18T07:44:12.621 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.621 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.621 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.621 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.617990+0000", 2024-09-18T07:44:12.621 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.20123836000000001 2024-09-18T07:44:12.621 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.621 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.622 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:12.622 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.618046+0000", 2024-09-18T07:44:12.622 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6035999999999997e-05 2024-09-18T07:44:12.622 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.622 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.622 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.622 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.618087+0000", 2024-09-18T07:44:12.622 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1165000000000003e-05 2024-09-18T07:44:12.622 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.622 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.622 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.622 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.622 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.623 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:71877 10.b 10:d52e72bb:::benchmark_data_smithi137_151160_object71876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-09-18T07:44:12.623 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:43:46.410088+0000", 2024-09-18T07:44:12.623 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 26.105297097000001, 2024-09-18T07:44:12.623 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.20825089599999999, 2024-09-18T07:44:12.623 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.623 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.623 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:12.623 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.623 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:44:12.623 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:12.623 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 71877 2024-09-18T07:44:12.623 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.623 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.623 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.624 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.624 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.410088+0000", 2024-09-18T07:44:12.624 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.624 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.624 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.624 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.624 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.410088+0000", 2024-09-18T07:44:12.624 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.624 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.624 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.624 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.624 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.410090+0000", 2024-09-18T07:44:12.624 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1629999999999999e-06 2024-09-18T07:44:12.625 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.625 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.625 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.625 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.410151+0000", 2024-09-18T07:44:12.625 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1032000000000001e-05 2024-09-18T07:44:12.625 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.625 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.625 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.625 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.410163+0000", 2024-09-18T07:44:12.625 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2713e-05 2024-09-18T07:44:12.625 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.625 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.625 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.626 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.410172+0000", 2024-09-18T07:44:12.626 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6349999999999995e-06 2024-09-18T07:44:12.626 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.626 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.626 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.626 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.410224+0000", 2024-09-18T07:44:12.626 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1721000000000002e-05 2024-09-18T07:44:12.626 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.626 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.626 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:12.626 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.410300+0000", 2024-09-18T07:44:12.626 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6655000000000004e-05 2024-09-18T07:44:12.626 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.626 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.627 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:44:12.627 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.410383+0000", 2024-09-18T07:44:12.627 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3001e-05 2024-09-18T07:44:12.627 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.627 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.627 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:12.627 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.411723+0000", 2024-09-18T07:44:12.627 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0013398640000000001 2024-09-18T07:44:12.627 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.627 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.627 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.627 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.416985+0000", 2024-09-18T07:44:12.627 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0052614970000000004 2024-09-18T07:44:12.627 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.628 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.628 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.628 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.618218+0000", 2024-09-18T07:44:12.628 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.201233457 2024-09-18T07:44:12.628 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.628 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.628 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:12.628 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.618271+0000", 2024-09-18T07:44:12.628 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3242999999999998e-05 2024-09-18T07:44:12.628 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.628 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.628 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.628 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.618339+0000", 2024-09-18T07:44:12.629 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7915000000000003e-05 2024-09-18T07:44:12.629 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.629 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.629 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.629 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.629 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.629 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:71891 10.0 10:0b8e9148:::benchmark_data_smithi137_151160_object71890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-09-18T07:44:12.629 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:43:46.420050+0000", 2024-09-18T07:44:12.629 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 26.095335645999999, 2024-09-18T07:44:12.629 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19390434100000001, 2024-09-18T07:44:12.629 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.629 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.629 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:12.630 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.630 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:44:12.630 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:12.630 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 71891 2024-09-18T07:44:12.630 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.630 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.630 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.630 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.630 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.420050+0000", 2024-09-18T07:44:12.630 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.630 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.630 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.630 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.631 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.420050+0000", 2024-09-18T07:44:12.631 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.631 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.631 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.631 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.631 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.420051+0000", 2024-09-18T07:44:12.631 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.362e-06 2024-09-18T07:44:12.631 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.631 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.631 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.631 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.420113+0000", 2024-09-18T07:44:12.631 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2096999999999999e-05 2024-09-18T07:44:12.631 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.631 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.632 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.632 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.420126+0000", 2024-09-18T07:44:12.632 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2996000000000001e-05 2024-09-18T07:44:12.632 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.632 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.632 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.632 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.420132+0000", 2024-09-18T07:44:12.632 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5829999999999997e-06 2024-09-18T07:44:12.632 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.632 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.632 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.632 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.420186+0000", 2024-09-18T07:44:12.632 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3801999999999998e-05 2024-09-18T07:44:12.633 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.633 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.633 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:12.633 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.420318+0000", 2024-09-18T07:44:12.633 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013219600000000001 2024-09-18T07:44:12.633 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.633 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.633 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1", 2024-09-18T07:44:12.633 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.420460+0000", 2024-09-18T07:44:12.633 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014254599999999999 2024-09-18T07:44:12.633 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.633 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.633 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:12.634 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.424672+0000", 2024-09-18T07:44:12.634 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0042111880000000003 2024-09-18T07:44:12.634 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.634 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.634 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.634 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.613857+0000", 2024-09-18T07:44:12.634 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18918581500000001 2024-09-18T07:44:12.634 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.634 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.634 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:12.634 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.613893+0000", 2024-09-18T07:44:12.634 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5816999999999999e-05 2024-09-18T07:44:12.634 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.635 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.635 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.635 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.613954+0000", 2024-09-18T07:44:12.635 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0939e-05 2024-09-18T07:44:12.635 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.635 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.635 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.635 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.635 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.635 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:71892 10.0 10:0ca14cde:::benchmark_data_smithi137_151160_object71891:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-09-18T07:44:12.635 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:43:46.420569+0000", 2024-09-18T07:44:12.635 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 26.094816177999999, 2024-09-18T07:44:12.636 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19348768199999999, 2024-09-18T07:44:12.636 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.636 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.636 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:12.636 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.636 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:44:12.636 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:12.636 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 71892 2024-09-18T07:44:12.636 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.636 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.636 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.636 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.637 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.420569+0000", 2024-09-18T07:44:12.637 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.637 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.637 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.637 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.637 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.420569+0000", 2024-09-18T07:44:12.637 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.637 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.637 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.637 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.637 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.420571+0000", 2024-09-18T07:44:12.637 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2729999999999999e-06 2024-09-18T07:44:12.637 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.637 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.638 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.638 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.420635+0000", 2024-09-18T07:44:12.638 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4288999999999994e-05 2024-09-18T07:44:12.638 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.638 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.638 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.638 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.420646+0000", 2024-09-18T07:44:12.638 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0789e-05 2024-09-18T07:44:12.638 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.638 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.638 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.638 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.420651+0000", 2024-09-18T07:44:12.638 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3600000000000004e-06 2024-09-18T07:44:12.638 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.639 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.639 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.639 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.422731+0000", 2024-09-18T07:44:12.639 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002079544 2024-09-18T07:44:12.639 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.639 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.639 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:12.639 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.422820+0000", 2024-09-18T07:44:12.639 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9139000000000002e-05 2024-09-18T07:44:12.639 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.639 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.639 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1", 2024-09-18T07:44:12.639 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.422901+0000", 2024-09-18T07:44:12.639 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1246000000000001e-05 2024-09-18T07:44:12.640 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.640 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.640 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:12.640 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.424966+0000", 2024-09-18T07:44:12.640 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0020651929999999999 2024-09-18T07:44:12.640 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.640 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.640 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.640 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.613983+0000", 2024-09-18T07:44:12.640 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.18901654400000001 2024-09-18T07:44:12.640 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.640 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.640 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:12.641 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.614013+0000", 2024-09-18T07:44:12.641 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9984999999999998e-05 2024-09-18T07:44:12.641 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.641 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.641 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.641 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.614057+0000", 2024-09-18T07:44:12.641 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4320000000000003e-05 2024-09-18T07:44:12.641 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.641 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.641 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.641 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.641 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.642 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:71903 10.0 10:02e4f753:::benchmark_data_smithi137_151160_object71902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-09-18T07:44:12.642 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:43:46.432200+0000", 2024-09-18T07:44:12.642 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 26.083185811, 2024-09-18T07:44:12.642 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19229929300000001, 2024-09-18T07:44:12.642 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.642 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.642 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:12.642 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.642 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:44:12.642 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:12.642 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 71903 2024-09-18T07:44:12.642 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.642 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.643 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.643 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.643 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.432200+0000", 2024-09-18T07:44:12.643 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.643 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.643 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.643 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.643 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.432200+0000", 2024-09-18T07:44:12.643 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.643 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.643 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.643 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.643 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.432201+0000", 2024-09-18T07:44:12.644 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0279999999999999e-06 2024-09-18T07:44:12.644 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.644 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.644 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.644 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.432263+0000", 2024-09-18T07:44:12.644 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2509999999999996e-05 2024-09-18T07:44:12.644 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.644 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.644 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.644 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.432272+0000", 2024-09-18T07:44:12.644 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9509999999999995e-06 2024-09-18T07:44:12.644 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.644 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.645 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.645 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.432277+0000", 2024-09-18T07:44:12.645 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7849999999999999e-06 2024-09-18T07:44:12.645 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.645 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.645 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.645 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.432321+0000", 2024-09-18T07:44:12.645 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3628000000000003e-05 2024-09-18T07:44:12.645 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.645 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.645 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:12.645 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.432405+0000", 2024-09-18T07:44:12.645 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4227999999999994e-05 2024-09-18T07:44:12.646 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.646 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.646 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1", 2024-09-18T07:44:12.646 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.432493+0000", 2024-09-18T07:44:12.646 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7923999999999994e-05 2024-09-18T07:44:12.646 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.646 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.646 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:12.646 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.434347+0000", 2024-09-18T07:44:12.646 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0018538160000000001 2024-09-18T07:44:12.646 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.646 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.646 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.647 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.624373+0000", 2024-09-18T07:44:12.647 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.19002674999999999 2024-09-18T07:44:12.647 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.647 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.647 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:12.647 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.624417+0000", 2024-09-18T07:44:12.647 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3752999999999999e-05 2024-09-18T07:44:12.647 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.647 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.647 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.647 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:46.624499+0000", 2024-09-18T07:44:12.647 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1920000000000002e-05 2024-09-18T07:44:12.647 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.648 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.648 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.648 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.648 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.648 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:75473 10.d 10:bd3c7af9:::benchmark_data_smithi137_151160_object75472:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-18T07:44:12.648 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:43:49.083631+0000", 2024-09-18T07:44:12.648 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 23.431754534, 2024-09-18T07:44:12.648 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.177731264, 2024-09-18T07:44:12.648 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.648 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.648 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:12.648 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.648 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:44:12.649 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:12.649 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 75473 2024-09-18T07:44:12.649 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.649 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.649 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.649 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.649 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.083631+0000", 2024-09-18T07:44:12.649 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.649 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.649 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.649 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.649 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.083631+0000", 2024-09-18T07:44:12.649 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.650 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.650 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.650 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.650 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.083632+0000", 2024-09-18T07:44:12.650 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9999999999999996e-07 2024-09-18T07:44:12.650 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.650 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.650 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.650 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.083659+0000", 2024-09-18T07:44:12.650 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.694e-05 2024-09-18T07:44:12.650 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.650 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.650 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.650 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.083669+0000", 2024-09-18T07:44:12.651 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0710000000000001e-05 2024-09-18T07:44:12.651 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.651 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.651 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.651 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.083675+0000", 2024-09-18T07:44:12.651 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6679999999999998e-06 2024-09-18T07:44:12.651 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.651 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.651 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.651 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.086469+0000", 2024-09-18T07:44:12.651 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027936879999999999 2024-09-18T07:44:12.651 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.651 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.651 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:12.652 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.086590+0000", 2024-09-18T07:44:12.652 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000121014 2024-09-18T07:44:12.652 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.652 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.652 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0", 2024-09-18T07:44:12.652 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.086760+0000", 2024-09-18T07:44:12.652 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000170447 2024-09-18T07:44:12.652 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.652 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.652 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:12.652 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.095510+0000", 2024-09-18T07:44:12.652 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0087493459999999999 2024-09-18T07:44:12.652 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.653 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.653 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.653 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.261225+0000", 2024-09-18T07:44:12.653 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16571573100000001 2024-09-18T07:44:12.653 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.653 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.653 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:12.653 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.261280+0000", 2024-09-18T07:44:12.653 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4227000000000001e-05 2024-09-18T07:44:12.653 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.653 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.653 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.653 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.261362+0000", 2024-09-18T07:44:12.654 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2693000000000003e-05 2024-09-18T07:44:12.654 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.654 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.654 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.654 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.654 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.654 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:75498 10.0 10:0d1fc94a:::benchmark_data_smithi137_151160_object75497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-18T07:44:12.654 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:43:49.097885+0000", 2024-09-18T07:44:12.654 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 23.417500436000001, 2024-09-18T07:44:12.654 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.206175628, 2024-09-18T07:44:12.654 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.654 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.654 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:12.654 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.655 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:44:12.655 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:12.655 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 75498 2024-09-18T07:44:12.655 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.655 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.655 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.655 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.655 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.097885+0000", 2024-09-18T07:44:12.655 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.655 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.655 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.655 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.655 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.097885+0000", 2024-09-18T07:44:12.655 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.656 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.656 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.656 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.656 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.097886+0000", 2024-09-18T07:44:12.656 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4559999999999999e-06 2024-09-18T07:44:12.656 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.656 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.656 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.656 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.097956+0000", 2024-09-18T07:44:12.656 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9078999999999994e-05 2024-09-18T07:44:12.656 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.656 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.656 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.657 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.097965+0000", 2024-09-18T07:44:12.657 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6579999999999997e-06 2024-09-18T07:44:12.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.657 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.657 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.657 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.097972+0000", 2024-09-18T07:44:12.657 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7660000000000001e-06 2024-09-18T07:44:12.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.657 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.657 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.657 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.261500+0000", 2024-09-18T07:44:12.657 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16352841600000001 2024-09-18T07:44:12.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.658 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.658 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:12.658 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.261658+0000", 2024-09-18T07:44:12.658 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015794600000000001 2024-09-18T07:44:12.658 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.658 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.658 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1", 2024-09-18T07:44:12.658 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.261872+0000", 2024-09-18T07:44:12.658 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000213313 2024-09-18T07:44:12.658 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.658 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.658 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:12.658 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.301151+0000", 2024-09-18T07:44:12.658 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.039279280999999999 2024-09-18T07:44:12.659 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.659 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.659 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.659 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.303898+0000", 2024-09-18T07:44:12.659 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0027474560000000001 2024-09-18T07:44:12.659 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.659 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.659 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:12.659 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.303967+0000", 2024-09-18T07:44:12.659 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8444999999999999e-05 2024-09-18T07:44:12.659 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.659 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.659 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.659 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.304061+0000", 2024-09-18T07:44:12.660 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3812000000000002e-05 2024-09-18T07:44:12.660 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.660 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.660 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.660 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.660 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.660 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:75500 10.0 10:0c77ae21:::benchmark_data_smithi137_151160_object75499:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-18T07:44:12.660 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:43:49.098046+0000", 2024-09-18T07:44:12.660 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 23.417339193, 2024-09-18T07:44:12.660 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.206189661, 2024-09-18T07:44:12.660 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:12.660 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:12.660 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:12.661 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:12.661 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:44:12.661 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:12.661 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 75500 2024-09-18T07:44:12.661 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.661 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:12.661 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.661 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:12.661 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.098046+0000", 2024-09-18T07:44:12.661 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.661 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.661 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.661 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:12.661 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.098046+0000", 2024-09-18T07:44:12.662 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:12.662 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.662 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.662 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:12.662 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.098047+0000", 2024-09-18T07:44:12.662 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1699999999999997e-07 2024-09-18T07:44:12.662 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.662 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.662 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:12.662 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.098110+0000", 2024-09-18T07:44:12.662 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2910999999999994e-05 2024-09-18T07:44:12.662 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.662 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.663 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:12.663 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.261122+0000", 2024-09-18T07:44:12.663 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16301190099999999 2024-09-18T07:44:12.663 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.663 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.663 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:12.663 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.261137+0000", 2024-09-18T07:44:12.663 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4825e-05 2024-09-18T07:44:12.663 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.663 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.663 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:12.663 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.264442+0000", 2024-09-18T07:44:12.663 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0033052099999999998 2024-09-18T07:44:12.663 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.663 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.664 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:12.664 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.264590+0000", 2024-09-18T07:44:12.664 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014804400000000001 2024-09-18T07:44:12.664 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.664 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.664 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1", 2024-09-18T07:44:12.664 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.264706+0000", 2024-09-18T07:44:12.664 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000116297 2024-09-18T07:44:12.664 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.664 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.664 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:12.664 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.304070+0000", 2024-09-18T07:44:12.664 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.039363691999999999 2024-09-18T07:44:12.664 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.664 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.665 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:12.665 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.304107+0000", 2024-09-18T07:44:12.665 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6723000000000001e-05 2024-09-18T07:44:12.665 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.665 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.665 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:12.665 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.304154+0000", 2024-09-18T07:44:12.665 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7077999999999999e-05 2024-09-18T07:44:12.665 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:12.665 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:12.665 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:12.665 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:43:49.304236+0000", 2024-09-18T07:44:12.665 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2262999999999995e-05 2024-09-18T07:44:12.666 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.666 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.666 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.666 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:12.666 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:12.666 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:44:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:12.694 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:12.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:12.795 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:12.825 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:12.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:12.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:12.928 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:12.970 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:12.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:12.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:13.071 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:13.073 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 63 16 104577 104561 103.696 79.125 0.00312226 0.00963372 2024-09-18T07:44:13.092 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:13.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:13.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:13.194 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:13.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:13.230 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:13.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:13.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:13.353 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:13.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:13.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:13.398 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:13 smithi203 ceph-mon[32237]: 5.19 scrub starts 2024-09-18T07:44:13.399 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:13 smithi203 ceph-mon[32237]: 5.19 scrub ok 2024-09-18T07:44:13.399 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:13 smithi203 ceph-mon[32237]: pgmap v695: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 62 MiB/s wr, 996 op/s; 49479/294339 objects degraded (16.810%) 2024-09-18T07:44:13.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:13 smithi137 ceph-mon[29408]: 5.19 scrub starts 2024-09-18T07:44:13.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:13 smithi137 ceph-mon[29408]: 5.19 scrub ok 2024-09-18T07:44:13.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:13 smithi137 ceph-mon[29408]: pgmap v695: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 62 MiB/s wr, 996 op/s; 49479/294339 objects degraded (16.810%) 2024-09-18T07:44:13.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:13 smithi137 ceph-mon[39175]: 5.19 scrub starts 2024-09-18T07:44:13.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:13 smithi137 ceph-mon[39175]: 5.19 scrub ok 2024-09-18T07:44:13.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:13 smithi137 ceph-mon[39175]: pgmap v695: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 62 MiB/s wr, 996 op/s; 49479/294339 objects degraded (16.810%) 2024-09-18T07:44:13.455 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:13.494 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:13.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:13.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:13.596 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:13.628 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:13.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:13.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:13.730 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:13.736 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T07:44:13.751 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:13.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:13.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:13.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:13.874 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:13.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:13.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:13.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:13.997 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:13.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:13.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:14.014 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:44:14.074 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 64 16 105306 105290 102.788 45.5625 0.00491997 0.00972209 2024-09-18T07:44:14.099 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:14.139 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:14.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:14.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:14.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:14.283 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:14.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:14.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:14.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:14.406 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:14.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:14.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:14.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:14 smithi137 ceph-mon[29408]: 5.1b scrub starts 2024-09-18T07:44:14.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:14 smithi137 ceph-mon[29408]: 5.1b scrub ok 2024-09-18T07:44:14.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:14 smithi137 ceph-mon[29408]: 3.18 scrub starts 2024-09-18T07:44:14.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:14 smithi137 ceph-mon[29408]: 3.18 scrub ok 2024-09-18T07:44:14.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:14 smithi137 ceph-mon[29408]: 10.5 scrub starts 2024-09-18T07:44:14.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:14 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:44:14.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:14 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:44:14.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:14 smithi137 ceph-mon[39175]: 5.1b scrub starts 2024-09-18T07:44:14.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:14 smithi137 ceph-mon[39175]: 5.1b scrub ok 2024-09-18T07:44:14.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:14 smithi137 ceph-mon[39175]: 3.18 scrub starts 2024-09-18T07:44:14.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:14 smithi137 ceph-mon[39175]: 3.18 scrub ok 2024-09-18T07:44:14.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:14 smithi137 ceph-mon[39175]: 10.5 scrub starts 2024-09-18T07:44:14.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:14 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:44:14.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:14 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:44:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:14 smithi203 ceph-mon[32237]: 5.1b scrub starts 2024-09-18T07:44:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:14 smithi203 ceph-mon[32237]: 5.1b scrub ok 2024-09-18T07:44:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:14 smithi203 ceph-mon[32237]: 3.18 scrub starts 2024-09-18T07:44:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:14 smithi203 ceph-mon[32237]: 3.18 scrub ok 2024-09-18T07:44:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:14 smithi203 ceph-mon[32237]: 10.5 scrub starts 2024-09-18T07:44:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:14 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:44:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:14 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:44:14.509 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:14.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:14.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:14.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:14.647 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:44:14.647 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:44:14.647 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.647 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:105870 10.1 10:8dad9576:::benchmark_data_smithi137_151160_object105869:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-18T07:44:14.647 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:44:14.620527+0000", 2024-09-18T07:44:14.647 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.023741577999999999, 2024-09-18T07:44:14.648 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.023768092000000001, 2024-09-18T07:44:14.648 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:14.648 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:14.648 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:44:14.648 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:14.648 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:44:14.648 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:14.648 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 105870 2024-09-18T07:44:14.648 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.648 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:14.648 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.648 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:14.648 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.620527+0000", 2024-09-18T07:44:14.649 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:14.649 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.649 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.649 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:14.649 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.620527+0000", 2024-09-18T07:44:14.649 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:14.649 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.649 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.649 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:14.649 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.620535+0000", 2024-09-18T07:44:14.649 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6909999999999997e-06 2024-09-18T07:44:14.649 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.650 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.650 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:14.650 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.620570+0000", 2024-09-18T07:44:14.650 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5083000000000003e-05 2024-09-18T07:44:14.650 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.650 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.650 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:14.650 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.620585+0000", 2024-09-18T07:44:14.650 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.543e-05 2024-09-18T07:44:14.650 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.650 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.650 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:14.650 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.620592+0000", 2024-09-18T07:44:14.650 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7479999999999996e-06 2024-09-18T07:44:14.651 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.651 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.651 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:14.651 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.620645+0000", 2024-09-18T07:44:14.651 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3295999999999999e-05 2024-09-18T07:44:14.651 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.651 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.651 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:14.651 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.620748+0000", 2024-09-18T07:44:14.651 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010231899999999999 2024-09-18T07:44:14.651 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.651 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.651 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0", 2024-09-18T07:44:14.652 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.620893+0000", 2024-09-18T07:44:14.652 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014519700000000001 2024-09-18T07:44:14.652 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.652 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.652 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:14.652 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.622957+0000", 2024-09-18T07:44:14.652 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0020643150000000002 2024-09-18T07:44:14.652 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:14.652 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:14.652 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:14.652 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.652 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.652 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:105875 10.1 10:8ce95fcb:::benchmark_data_smithi137_151160_object105874:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-18T07:44:14.653 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:44:14.624399+0000", 2024-09-18T07:44:14.653 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.019869777000000002, 2024-09-18T07:44:14.653 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.019943592, 2024-09-18T07:44:14.653 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:14.653 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:14.653 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:44:14.653 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:14.653 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:44:14.653 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:14.653 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 105875 2024-09-18T07:44:14.653 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.653 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:14.653 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.654 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:14.654 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.624399+0000", 2024-09-18T07:44:14.654 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:14.654 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.654 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.654 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:14.654 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.624399+0000", 2024-09-18T07:44:14.654 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:14.654 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.654 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.654 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:14.654 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.624400+0000", 2024-09-18T07:44:14.654 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.251e-06 2024-09-18T07:44:14.654 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.655 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.655 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:14.655 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.624457+0000", 2024-09-18T07:44:14.655 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6342999999999999e-05 2024-09-18T07:44:14.655 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.655 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.655 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:14.655 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.624467+0000", 2024-09-18T07:44:14.655 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0203999999999999e-05 2024-09-18T07:44:14.655 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.655 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.655 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:14.655 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.624473+0000", 2024-09-18T07:44:14.655 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2360000000000001e-06 2024-09-18T07:44:14.656 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.656 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.656 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:14.656 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.624531+0000", 2024-09-18T07:44:14.656 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8375999999999998e-05 2024-09-18T07:44:14.656 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.656 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.656 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:14.656 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.624673+0000", 2024-09-18T07:44:14.656 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000142077 2024-09-18T07:44:14.656 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.656 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.656 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0", 2024-09-18T07:44:14.656 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.624845+0000", 2024-09-18T07:44:14.656 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000171416 2024-09-18T07:44:14.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.657 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.657 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:14.657 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.627396+0000", 2024-09-18T07:44:14.657 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025514589999999998 2024-09-18T07:44:14.657 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:14.657 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:14.657 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:14.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.657 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.657 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:105881 10.1 10:8ab7648f:::benchmark_data_smithi137_151160_object105880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-18T07:44:14.657 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:44:14.631148+0000", 2024-09-18T07:44:14.657 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.013120401, 2024-09-18T07:44:14.657 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.01323452, 2024-09-18T07:44:14.658 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:14.658 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:14.658 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:44:14.658 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:14.658 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:44:14.658 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:14.658 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 105881 2024-09-18T07:44:14.658 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.658 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:14.658 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.658 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:14.658 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.631148+0000", 2024-09-18T07:44:14.658 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:14.658 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.659 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.659 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:14.659 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.631148+0000", 2024-09-18T07:44:14.659 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:14.659 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.659 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.659 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:14.659 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.631151+0000", 2024-09-18T07:44:14.659 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2390000000000001e-06 2024-09-18T07:44:14.659 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.659 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.659 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:14.659 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.631196+0000", 2024-09-18T07:44:14.659 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5033000000000001e-05 2024-09-18T07:44:14.659 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.660 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.660 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:14.660 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.631209+0000", 2024-09-18T07:44:14.660 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3421000000000001e-05 2024-09-18T07:44:14.660 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.660 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.660 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:14.660 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.631215+0000", 2024-09-18T07:44:14.660 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.28e-06 2024-09-18T07:44:14.660 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.660 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.660 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:14.660 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.631261+0000", 2024-09-18T07:44:14.660 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5782000000000003e-05 2024-09-18T07:44:14.660 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.661 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.661 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:14.661 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.631355+0000", 2024-09-18T07:44:14.661 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4394000000000005e-05 2024-09-18T07:44:14.661 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.661 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.661 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0", 2024-09-18T07:44:14.661 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.631446+0000", 2024-09-18T07:44:14.661 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0499000000000002e-05 2024-09-18T07:44:14.661 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:14.661 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:14.661 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:14.661 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:14.633716+0000", 2024-09-18T07:44:14.661 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022705030000000001 2024-09-18T07:44:14.661 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:14.662 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:14.662 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:14.662 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:14.662 INFO:teuthology.orchestra.run.smithi203.stdout: ], 2024-09-18T07:44:14.662 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 3 2024-09-18T07:44:14.662 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:44:14.662 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:14.686 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:14.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:14.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:14.788 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:14.818 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:14.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:14.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:14.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:15.074 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:15.074 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 65 16 106256 106240 102.12 59.375 0.021488 0.00978778 2024-09-18T07:44:15.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:15.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:15.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:15.211 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:15.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:15.313 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:15.366 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:15.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:15.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:15.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:15 smithi203 ceph-mon[32237]: pgmap v696: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 51770/310428 objects degraded (16.677%) 2024-09-18T07:44:15.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:15 smithi203 ceph-mon[32237]: osdmap e383: 8 total, 7 up, 8 in 2024-09-18T07:44:15.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:15 smithi203 ceph-mon[32237]: 4.3 scrub starts 2024-09-18T07:44:15.412 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:15 smithi203 ceph-mon[32237]: 4.3 scrub ok 2024-09-18T07:44:15.412 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:15 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 51770/310428 objects degraded (16.677%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:15.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:15 smithi137 ceph-mon[29408]: pgmap v696: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 51770/310428 objects degraded (16.677%) 2024-09-18T07:44:15.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:15 smithi137 ceph-mon[29408]: osdmap e383: 8 total, 7 up, 8 in 2024-09-18T07:44:15.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:15 smithi137 ceph-mon[29408]: 4.3 scrub starts 2024-09-18T07:44:15.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:15 smithi137 ceph-mon[29408]: 4.3 scrub ok 2024-09-18T07:44:15.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:15 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 51770/310428 objects degraded (16.677%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:15 smithi137 ceph-mon[39175]: pgmap v696: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 51770/310428 objects degraded (16.677%) 2024-09-18T07:44:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:15 smithi137 ceph-mon[39175]: osdmap e383: 8 total, 7 up, 8 in 2024-09-18T07:44:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:15 smithi137 ceph-mon[39175]: 4.3 scrub starts 2024-09-18T07:44:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:15 smithi137 ceph-mon[39175]: 4.3 scrub ok 2024-09-18T07:44:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:15 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 51770/310428 objects degraded (16.677%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:15.452 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T07:44:15.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:15.505 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:15.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:15.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:15.606 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:15.633 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:15.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:15.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:15.735 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:15.757 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:44:15.769 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:15.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:15.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:44:15.858 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:44:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 383, 2024-09-18T07:44:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: "fsid": "f7ef099c-758e-11ef-bceb-c7b262605968", 2024-09-18T07:44:15.858 INFO:teuthology.orchestra.run.smithi137.stdout: "created": "2024-09-18T07:25:19.662367+0000", 2024-09-18T07:44:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "modified": "2024-09-18T07:44:14.095902+0000", 2024-09-18T07:44:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "last_up_change": "2024-09-18T07:42:53.723776+0000", 2024-09-18T07:44:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "last_in_change": "2024-09-18T07:37:42.022730+0000", 2024-09-18T07:44:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T07:44:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_num": 5799936, 2024-09-18T07:44:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_set": [ 2024-09-18T07:44:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "pglog_hardlimit", 2024-09-18T07:44:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "purged_snapdirs", 2024-09-18T07:44:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "recovery_deletes", 2024-09-18T07:44:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "sortbitwise" 2024-09-18T07:44:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:44:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_version": 18, 2024-09-18T07:44:15.859 INFO:teuthology.orchestra.run.smithi137.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T07:44:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T07:44:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T07:44:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_snapshot": "", 2024-09-18T07:44:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_max": 10, 2024-09-18T07:44:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "max_osd": 8, 2024-09-18T07:44:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "require_min_compat_client": "luminous", 2024-09-18T07:44:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "min_compat_client": "luminous", 2024-09-18T07:44:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "require_osd_release": "squid", 2024-09-18T07:44:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "allow_crimson": false, 2024-09-18T07:44:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "pools": [ 2024-09-18T07:44:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 1, 2024-09-18T07:44:15.860 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".mgr", 2024-09-18T07:44:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:28:20.899655+0000", 2024-09-18T07:44:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:44:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:44:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:44:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:44:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:44:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:44:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:44:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:44:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:44:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:44:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:44:15.861 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:44:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:44:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 11, 2024-09-18T07:44:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 11, 2024-09-18T07:44:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 11, 2024-09-18T07:44:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 11, 2024-09-18T07:44:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 11, 2024-09-18T07:44:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:44:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:44:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:44:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:44:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:44:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:44:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:44:15.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "274", 2024-09-18T07:44:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:44:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:44:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "272", 2024-09-18T07:44:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:44:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:44:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:44:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:44:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:44:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:44:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:44:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:44:15.863 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:44:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:44:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:44:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:44:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:44:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:44:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:44:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:44:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:44:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:44:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:44:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:44:15.864 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:44:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:44:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:44:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:44:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:44:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:44:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:44:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:44:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:44:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:44:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:44:15.865 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:44:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:44:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:44:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:44:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_max": 32, 2024-09-18T07:44:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_min": 1 2024-09-18T07:44:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:44:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": {} 2024-09-18T07:44:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.866 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:44:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:44:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.9099999666213989, 2024-09-18T07:44:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.9099999666213989, 2024-09-18T07:44:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:44:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.9099999666213989, 2024-09-18T07:44:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.9099999666213989, 2024-09-18T07:44:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:44:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:44:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:44:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 2, 2024-09-18T07:44:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "datapool", 2024-09-18T07:44:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:53.715373+0000", 2024-09-18T07:44:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-18T07:44:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T07:44:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:44:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:44:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:44:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:44:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:44:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:44:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:44:15.868 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:44:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:44:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:44:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:44:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 3, 2024-09-18T07:44:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 3, 2024-09-18T07:44:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 3, 2024-09-18T07:44:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 3, 2024-09-18T07:44:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 3, 2024-09-18T07:44:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:44:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:44:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:44:15.869 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:44:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:44:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:44:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:44:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "63", 2024-09-18T07:44:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:44:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:44:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:44:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:44:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:44:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 2, 2024-09-18T07:44:15.870 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 63, 2024-09-18T07:44:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:44:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:44:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:44:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:44:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:44:15.871 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:44:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:44:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:44:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:44:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:44:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:44:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:44:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:44:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:44:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:44:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:44:15.872 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:44:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:44:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:44:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:44:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:44:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:44:15.873 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:44:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:44:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:44:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:44:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:44:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:44:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:44:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:44:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:44:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:44:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd": {} 2024-09-18T07:44:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:44:15.874 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:44:15.875 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.3299999237060547, 2024-09-18T07:44:15.875 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.3299999237060547, 2024-09-18T07:44:15.875 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:44:15.875 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-18T07:44:15.875 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-18T07:44:15.875 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:44:15.875 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:44:15.875 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:44:15.875 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.875 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 3, 2024-09-18T07:44:15.875 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".rgw.root", 2024-09-18T07:44:15.875 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:54.672127+0000", 2024-09-18T07:44:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:44:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:44:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:44:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:44:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:44:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:44:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:44:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:44:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:44:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:44:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:44:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:44:15.876 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:44:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:44:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:44:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:44:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:44:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:44:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:44:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:44:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:44:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:44:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:44:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:44:15.877 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:44:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "62", 2024-09-18T07:44:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:44:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:44:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:44:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:44:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:44:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:44:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:44:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:44:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:44:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:44:15.878 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:44:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:44:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:44:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:44:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:44:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:44:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:44:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:44:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:44:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:44:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:44:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:44:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:44:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:44:15.879 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:44:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:44:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:44:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:44:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:44:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:44:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:44:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:44:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:44:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:44:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:44:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:44:15.880 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:44:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:44:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:44:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:44:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:44:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:44:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.75, 2024-09-18T07:44:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.75, 2024-09-18T07:44:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:44:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.75, 2024-09-18T07:44:15.881 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.75, 2024-09-18T07:44:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:44:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:44:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:44:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 4, 2024-09-18T07:44:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.log", 2024-09-18T07:44:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:56.351343+0000", 2024-09-18T07:44:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:44:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:44:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:44:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:44:15.882 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:44:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:44:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:44:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:44:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:44:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:44:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:44:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:44:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:44:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:44:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:44:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:44:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:44:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:44:15.883 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:44:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:44:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:44:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:44:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:44:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:44:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:44:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "64", 2024-09-18T07:44:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:44:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:44:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:44:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:44:15.884 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:44:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:44:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:44:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:44:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:44:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:44:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:44:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:44:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:44:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:44:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:44:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:44:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:44:15.885 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:44:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:44:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:44:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:44:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:44:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:44:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:44:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:44:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:44:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:44:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:44:15.886 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:44:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:44:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:44:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:44:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:44:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:44:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:44:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:44:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:44:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:44:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:44:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:44:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.887 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:44:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:44:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.5299999713897705, 2024-09-18T07:44:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.5299999713897705, 2024-09-18T07:44:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:44:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-18T07:44:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-18T07:44:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:44:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:44:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:44:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 5, 2024-09-18T07:44:15.888 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.control", 2024-09-18T07:44:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:58.322401+0000", 2024-09-18T07:44:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:44:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:44:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:44:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:44:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:44:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:44:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:44:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:44:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:44:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:44:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:44:15.889 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:44:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:44:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:44:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:44:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:44:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:44:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:44:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:44:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:44:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:44:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:44:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:44:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:44:15.890 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:44:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "66", 2024-09-18T07:44:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:44:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:44:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:44:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:44:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:44:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:44:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:44:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:44:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:44:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:44:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:44:15.891 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:44:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:44:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:44:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:44:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:44:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:44:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:44:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:44:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:44:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:44:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:44:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:44:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:44:15.892 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:44:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:44:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:44:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:44:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:44:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:44:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:44:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:44:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:44:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:44:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:44:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:44:15.893 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:44:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:44:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:44:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:44:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:44:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:44:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.75, 2024-09-18T07:44:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.75, 2024-09-18T07:44:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:44:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.75, 2024-09-18T07:44:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.75, 2024-09-18T07:44:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:44:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:44:15.894 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:44:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 6, 2024-09-18T07:44:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.meta", 2024-09-18T07:44:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:31:00.390499+0000", 2024-09-18T07:44:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:44:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:44:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:44:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:44:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:44:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:44:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:44:15.895 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:44:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:44:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:44:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:44:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:44:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:44:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:44:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:44:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:44:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:44:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:44:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:44:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:44:15.896 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:44:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:44:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:44:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:44:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:44:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "68", 2024-09-18T07:44:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:44:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:44:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:44:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:44:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:44:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:44:15.897 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:44:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:44:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:44:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:44:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:44:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:44:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:44:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:44:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:44:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:44:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:44:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:44:15.898 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:44:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:44:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:44:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:44:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:44:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:44:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:44:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:44:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:44:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:44:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:44:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:44:15.899 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:44:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:44:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:44:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:44:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:44:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:44:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:44:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:44:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_bias": 4 2024-09-18T07:44:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:44:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:44:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.900 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:44:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:44:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.5299999713897705, 2024-09-18T07:44:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.5299999713897705, 2024-09-18T07:44:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:44:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-18T07:44:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-18T07:44:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:44:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:44:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:44:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.901 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 10, 2024-09-18T07:44:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "unique_pool_3", 2024-09-18T07:44:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:43:00.547513+0000", 2024-09-18T07:44:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:44:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:44:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:44:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:44:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:44:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:44:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:44:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:44:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:44:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:44:15.902 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:44:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:44:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:44:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 16, 2024-09-18T07:44:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 16, 2024-09-18T07:44:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 16, 2024-09-18T07:44:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 16, 2024-09-18T07:44:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 16, 2024-09-18T07:44:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:44:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:44:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:44:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:44:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:44:15.903 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:44:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:44:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "367", 2024-09-18T07:44:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:44:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:44:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:44:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:44:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:44:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:44:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:44:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:44:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:44:15.904 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:44:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:44:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:44:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:44:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:44:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:44:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:44:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:44:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:44:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:44:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:44:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:44:15.905 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:44:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:44:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:44:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:44:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:44:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:44:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:44:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:44:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:44:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:44:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:44:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:44:15.906 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:44:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:44:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:44:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:44:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:44:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:44:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "rados": {} 2024-09-18T07:44:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:44:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:44:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 3.059999942779541, 2024-09-18T07:44:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 3.059999942779541, 2024-09-18T07:44:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:44:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 3.059999942779541, 2024-09-18T07:44:15.907 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 3.059999942779541, 2024-09-18T07:44:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.84493637084960938, 2024-09-18T07:44:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-18T07:44:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-09-18T07:44:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:44:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-18T07:44:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:44:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "268cd580-c931-4992-bb93-d6f63c2bfe88", 2024-09-18T07:44:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:44:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:44:15.908 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:44:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:44:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:44:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:44:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 8, 2024-09-18T07:44:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 364, 2024-09-18T07:44:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:44:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:44:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:44:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6801", 2024-09-18T07:44:15.909 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:44:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:44:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6802", 2024-09-18T07:44:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:44:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.910 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:44:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6804", 2024-09-18T07:44:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:44:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:44:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.911 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6803", 2024-09-18T07:44:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:44:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6801/2811444477", 2024-09-18T07:44:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6802/2811444477", 2024-09-18T07:44:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6804/2811444477", 2024-09-18T07:44:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6803/2811444477", 2024-09-18T07:44:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:44:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:44:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:44:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.913 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:44:15.913 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575", 2024-09-18T07:44:15.913 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:44:15.913 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:44:15.913 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:44:15.913 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 0, 2024-09-18T07:44:15.913 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:44:15.913 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:44:15.913 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 13, 2024-09-18T07:44:15.913 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 359, 2024-09-18T07:44:15.913 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:44:15.914 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:44:15.914 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:44:15.914 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.914 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.914 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6805", 2024-09-18T07:44:15.914 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:44:15.914 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.914 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.914 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:44:15.914 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.914 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.915 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6806", 2024-09-18T07:44:15.915 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:44:15.915 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.915 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.915 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:44:15.915 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.915 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.915 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6808", 2024-09-18T07:44:15.915 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:44:15.915 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.916 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.916 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:44:15.916 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.916 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.916 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6807", 2024-09-18T07:44:15.916 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:44:15.916 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.916 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.916 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6805/874102487", 2024-09-18T07:44:15.916 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6806/874102487", 2024-09-18T07:44:15.917 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6808/874102487", 2024-09-18T07:44:15.917 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6807/874102487", 2024-09-18T07:44:15.917 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:44:15.917 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:44:15.917 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:44:15.917 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.917 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:44:15.917 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "480ad6c5-0e0b-4405-bbe2-381682d00bae", 2024-09-18T07:44:15.917 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:44:15.917 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:44:15.917 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:44:15.918 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:44:15.918 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:44:15.918 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:44:15.918 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 18, 2024-09-18T07:44:15.918 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 364, 2024-09-18T07:44:15.918 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:44:15.918 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:44:15.918 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:44:15.918 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.918 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.918 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6809", 2024-09-18T07:44:15.918 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:44:15.919 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.919 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.919 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:44:15.919 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.919 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.919 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6810", 2024-09-18T07:44:15.919 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:44:15.919 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.919 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.919 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:44:15.920 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.920 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.920 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6812", 2024-09-18T07:44:15.920 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:44:15.920 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.920 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.920 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:44:15.920 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.920 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.921 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6811", 2024-09-18T07:44:15.921 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:44:15.921 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.921 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.921 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6809/1754684686", 2024-09-18T07:44:15.921 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6810/1754684686", 2024-09-18T07:44:15.921 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6812/1754684686", 2024-09-18T07:44:15.921 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6811/1754684686", 2024-09-18T07:44:15.921 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:44:15.922 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:44:15.922 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:44:15.922 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.922 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:44:15.922 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14", 2024-09-18T07:44:15.922 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:44:15.922 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:44:15.922 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:44:15.922 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 0.759490966796875, 2024-09-18T07:44:15.922 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:44:15.922 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:44:15.923 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 26, 2024-09-18T07:44:15.923 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 359, 2024-09-18T07:44:15.923 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:44:15.923 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:44:15.923 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:44:15.923 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.923 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.923 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6813", 2024-09-18T07:44:15.923 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:44:15.923 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.923 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.924 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:44:15.924 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.924 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.924 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6814", 2024-09-18T07:44:15.924 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:44:15.924 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.924 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.924 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:44:15.924 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.924 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.925 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6816", 2024-09-18T07:44:15.925 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:44:15.925 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.925 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.925 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:44:15.925 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.925 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.925 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6815", 2024-09-18T07:44:15.925 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:44:15.925 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.926 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.926 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6813/116058392", 2024-09-18T07:44:15.926 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6814/116058392", 2024-09-18T07:44:15.926 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6816/116058392", 2024-09-18T07:44:15.926 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6815/116058392", 2024-09-18T07:44:15.926 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:44:15.926 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:44:15.926 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:44:15.926 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.926 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:44:15.926 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de", 2024-09-18T07:44:15.927 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 0, 2024-09-18T07:44:15.927 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:44:15.927 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:44:15.927 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:44:15.927 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:44:15.927 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:44:15.927 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 32, 2024-09-18T07:44:15.927 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 289, 2024-09-18T07:44:15.927 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 359, 2024-09-18T07:44:15.927 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:44:15.927 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:44:15.927 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.928 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.928 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6800", 2024-09-18T07:44:15.928 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:44:15.928 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.928 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.928 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:44:15.928 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.928 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.928 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6801", 2024-09-18T07:44:15.928 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:44:15.928 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.929 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.929 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:44:15.929 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.929 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.929 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6803", 2024-09-18T07:44:15.929 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:44:15.929 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.929 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.929 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:44:15.929 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.930 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.930 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6802", 2024-09-18T07:44:15.930 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3654297954 2024-09-18T07:44:15.930 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.930 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.930 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6800/3654297954", 2024-09-18T07:44:15.930 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6801/3654297954", 2024-09-18T07:44:15.930 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6803/3654297954", 2024-09-18T07:44:15.930 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6802/3654297954", 2024-09-18T07:44:15.930 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:44:15.930 INFO:teuthology.orchestra.run.smithi137.stdout: "exists" 2024-09-18T07:44:15.931 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.931 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:44:15.931 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c", 2024-09-18T07:44:15.931 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:44:15.931 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:44:15.931 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:44:15.931 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:44:15.931 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:44:15.931 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:44:15.931 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 39, 2024-09-18T07:44:15.931 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 364, 2024-09-18T07:44:15.932 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:44:15.932 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:44:15.932 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:44:15.932 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.932 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.932 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6804", 2024-09-18T07:44:15.932 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:44:15.932 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.932 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.933 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:44:15.933 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.933 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.933 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6805", 2024-09-18T07:44:15.933 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:44:15.933 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.933 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.933 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:44:15.933 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.934 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.934 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6807", 2024-09-18T07:44:15.934 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:44:15.934 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.934 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.934 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:44:15.934 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.934 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.934 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6806", 2024-09-18T07:44:15.934 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 837994222 2024-09-18T07:44:15.934 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.935 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.935 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6804/837994222", 2024-09-18T07:44:15.935 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6805/837994222", 2024-09-18T07:44:15.935 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6807/837994222", 2024-09-18T07:44:15.935 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6806/837994222", 2024-09-18T07:44:15.935 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:44:15.935 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:44:15.935 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:44:15.935 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.936 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:44:15.936 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6", 2024-09-18T07:44:15.936 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:44:15.936 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:44:15.936 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:44:15.936 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:44:15.936 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:44:15.936 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:44:15.936 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 47, 2024-09-18T07:44:15.936 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 364, 2024-09-18T07:44:15.936 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:44:15.936 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:44:15.936 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:44:15.937 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.937 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.937 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6808", 2024-09-18T07:44:15.937 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:44:15.937 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.937 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.937 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:44:15.937 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.937 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.938 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6809", 2024-09-18T07:44:15.938 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:44:15.938 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.938 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.938 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:44:15.938 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.938 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.938 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6811", 2024-09-18T07:44:15.938 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:44:15.938 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.938 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.939 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:44:15.939 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.939 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.939 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6810", 2024-09-18T07:44:15.939 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:44:15.939 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.939 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.939 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6808/1494175466", 2024-09-18T07:44:15.939 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6809/1494175466", 2024-09-18T07:44:15.939 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6811/1494175466", 2024-09-18T07:44:15.940 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6810/1494175466", 2024-09-18T07:44:15.940 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:44:15.940 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:44:15.940 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:44:15.940 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.940 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:44:15.940 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a", 2024-09-18T07:44:15.940 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:44:15.940 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:44:15.940 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:44:15.941 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:44:15.941 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:44:15.941 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:44:15.941 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 55, 2024-09-18T07:44:15.941 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 364, 2024-09-18T07:44:15.941 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:44:15.941 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:44:15.941 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:44:15.941 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.941 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.941 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6812", 2024-09-18T07:44:15.942 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:44:15.942 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.942 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.942 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:44:15.942 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.942 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.942 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6813", 2024-09-18T07:44:15.942 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:44:15.942 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.942 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.942 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:44:15.943 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.943 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.943 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6815", 2024-09-18T07:44:15.943 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:44:15.943 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.943 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.943 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:44:15.943 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:44:15.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.943 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:44:15.943 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6814", 2024-09-18T07:44:15.944 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:44:15.944 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.944 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.944 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6812/2670809315", 2024-09-18T07:44:15.944 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6813/2670809315", 2024-09-18T07:44:15.944 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6815/2670809315", 2024-09-18T07:44:15.944 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6814/2670809315", 2024-09-18T07:44:15.944 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:44:15.944 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:44:15.944 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:44:15.944 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:44:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_xinfo": [ 2024-09-18T07:44:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:44:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:44:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:44:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:44:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:44:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:44:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:44:13.263276+0000", 2024-09-18T07:44:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:44:15.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:44:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:44:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:44:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:44:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:44:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:44:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:43:41.550939+0000", 2024-09-18T07:44:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:44:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:44:15.946 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:44:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:44:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:44:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:44:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:44:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:43:24.037996+0000", 2024-09-18T07:44:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:44:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:44:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:44:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:44:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:44:15.947 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:44:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:44:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:43:16.491536+0000", 2024-09-18T07:44:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:44:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:44:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:44:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:44:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:44:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:44:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:44:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:41:47.479848+0000", 2024-09-18T07:44:15.948 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 358 2024-09-18T07:44:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:44:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:44:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:44:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:44:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:44:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:44:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:44:02.089826+0000", 2024-09-18T07:44:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:44:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.949 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:44:15.950 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:44:15.950 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:44:15.950 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:44:15.950 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:44:15.950 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:44:15.950 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:44:01.308573+0000", 2024-09-18T07:44:15.950 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:44:15.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.950 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:44:15.950 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:44:15.950 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:44:15.950 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:44:15.951 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:44:15.951 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:44:15.951 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:42:59.881262+0000", 2024-09-18T07:44:15.951 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:44:15.951 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.951 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:44:15.951 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap": [], 2024-09-18T07:44:15.951 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_items": [ 2024-09-18T07:44:15.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.951 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.6", 2024-09-18T07:44:15.951 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:44:15.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.952 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:44:15.952 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:44:15.952 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.952 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.952 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.b", 2024-09-18T07:44:15.952 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:44:15.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.952 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:44:15.952 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:44:15.952 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.952 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.953 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.e", 2024-09-18T07:44:15.953 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:44:15.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.953 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:44:15.953 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:44:15.953 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.953 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.953 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.10", 2024-09-18T07:44:15.953 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:44:15.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.954 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:44:15.954 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:44:15.954 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.954 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.954 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.12", 2024-09-18T07:44:15.954 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:44:15.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.954 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:44:15.954 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:44:15.954 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.955 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.955 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.13", 2024-09-18T07:44:15.955 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:44:15.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.955 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 6, 2024-09-18T07:44:15.955 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 7 2024-09-18T07:44:15.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.955 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:44:15.955 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:44:15.955 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.955 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.956 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.18", 2024-09-18T07:44:15.956 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:44:15.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.956 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:44:15.956 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:44:15.956 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.956 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.956 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.1b", 2024-09-18T07:44:15.956 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:44:15.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.957 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:44:15.957 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-18T07:44:15.957 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.957 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.957 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.1c", 2024-09-18T07:44:15.957 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:44:15.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.957 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:44:15.957 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:44:15.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.958 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 1, 2024-09-18T07:44:15.958 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 7 2024-09-18T07:44:15.958 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.958 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.958 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.1d", 2024-09-18T07:44:15.958 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:44:15.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.958 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:44:15.958 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 2 2024-09-18T07:44:15.958 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.959 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.959 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.b", 2024-09-18T07:44:15.959 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:44:15.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.959 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 0, 2024-09-18T07:44:15.959 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 3 2024-09-18T07:44:15.959 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.959 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.959 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.1b", 2024-09-18T07:44:15.960 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:44:15.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:15.960 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 3, 2024-09-18T07:44:15.960 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 0 2024-09-18T07:44:15.960 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.960 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:15.960 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.960 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:44:15.960 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_primaries": [], 2024-09-18T07:44:15.960 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_temp": [], 2024-09-18T07:44:15.960 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_temp": [], 2024-09-18T07:44:15.960 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklist": { 2024-09-18T07:44:15.960 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/2096039404": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:44:15.961 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/3636177939": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:44:15.961 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/334967443": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:44:15.961 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2101286111": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:44:15.961 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3387637412": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:44:15.961 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762989503": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:44:15.961 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3490817671": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:44:15.961 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1482147392": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:44:15.961 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1821482181": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:44:15.961 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/4028304023": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:44:15.961 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/621434521": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:44:15.961 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3160704935": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:44:15.961 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2357293499": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:44:15.961 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762352860": "2024-09-19T07:31:25.389124+0000" 2024-09-18T07:44:15.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.962 INFO:teuthology.orchestra.run.smithi137.stdout: "range_blocklist": {}, 2024-09-18T07:44:15.962 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profiles": { 2024-09-18T07:44:15.962 INFO:teuthology.orchestra.run.smithi137.stdout: "default": { 2024-09-18T07:44:15.962 INFO:teuthology.orchestra.run.smithi137.stdout: "crush-failure-domain": "osd", 2024-09-18T07:44:15.962 INFO:teuthology.orchestra.run.smithi137.stdout: "k": "2", 2024-09-18T07:44:15.962 INFO:teuthology.orchestra.run.smithi137.stdout: "m": "1", 2024-09-18T07:44:15.962 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin": "jerasure", 2024-09-18T07:44:15.962 INFO:teuthology.orchestra.run.smithi137.stdout: "technique": "reed_sol_van" 2024-09-18T07:44:15.962 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:15.962 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps_queue": [], 2024-09-18T07:44:15.962 INFO:teuthology.orchestra.run.smithi137.stdout: "new_removed_snaps": [], 2024-09-18T07:44:15.962 INFO:teuthology.orchestra.run.smithi137.stdout: "new_purged_snaps": [], 2024-09-18T07:44:15.963 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_node_flags": {}, 2024-09-18T07:44:15.963 INFO:teuthology.orchestra.run.smithi137.stdout: "device_class_flags": {}, 2024-09-18T07:44:15.963 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode": { 2024-09-18T07:44:15.963 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_enabled": false, 2024-09-18T07:44:15.963 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_bucket_count": 0, 2024-09-18T07:44:15.963 INFO:teuthology.orchestra.run.smithi137.stdout: "degraded_stretch_mode": 0, 2024-09-18T07:44:15.963 INFO:teuthology.orchestra.run.smithi137.stdout: "recovering_stretch_mode": 0, 2024-09-18T07:44:15.963 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_bucket": 0 2024-09-18T07:44:15.963 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:15.963 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:44:15.964 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:15.990 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:15.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:15.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:16.074 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 66 16 107275 107259 101.538 63.6875 0.0198203 0.00984354 2024-09-18T07:44:16.092 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:16.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:16.124 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:16.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:16.182 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:16 smithi203 ceph-mon[32237]: 1.a scrub starts 2024-09-18T07:44:16.183 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:16 smithi203 ceph-mon[32237]: 1.a scrub ok 2024-09-18T07:44:16.183 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:16 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2432667605' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:44:16.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:16.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:16.269 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:16.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:16.336 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:44:16.337 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:44:16.337 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:44:16.337 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:44:16.337 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:44:16.370 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:16.404 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:16.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:16.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:16.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:16 smithi137 ceph-mon[29408]: 1.a scrub starts 2024-09-18T07:44:16.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:16 smithi137 ceph-mon[29408]: 1.a scrub ok 2024-09-18T07:44:16.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:16 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2432667605' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:44:16.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:16 smithi137 ceph-mon[39175]: 1.a scrub starts 2024-09-18T07:44:16.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:16 smithi137 ceph-mon[39175]: 1.a scrub ok 2024-09-18T07:44:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:16 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2432667605' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:44:16.506 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:16.544 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:16.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:16.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:16.646 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:16.682 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:16.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:16.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:16.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:16.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:16.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:16.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:16.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:16.966 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:16.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:16.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:17.062 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T07:44:17.068 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:17.074 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 67 16 108522 108506 101.186 77.9375 0.00980198 0.00987788 2024-09-18T07:44:17.109 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:17.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:17.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:17.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:17.250 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:17.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:17.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:17.327 INFO:tasks.thrashosds.thrasher:j is {'epoch': 383, 'fsid': 'f7ef099c-758e-11ef-bceb-c7b262605968', 'created': '2024-09-18T07:25:19.662367+0000', 'modified': '2024-09-18T07:44:14.095902+0000', 'last_up_change': '2024-09-18T07:42:53.723776+0000', 'last_in_change': '2024-09-18T07:37:42.022730+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T07:28:20.899655+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '274', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '272', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 1, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-18T07:30:53.715373+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 1, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-18T07:30:54.672127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-18T07:30:56.351343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-18T07:30:58.322401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-18T07:31:00.390499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-18T07:43:00.547513+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '367', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.059999942779541, 'score_stable': 3.059999942779541, 'optimal_score': 1, 'raw_score_acting': 3.059999942779541, 'raw_score_stable': 3.059999942779541, 'primary_affinity_weighted': 0.8449363708496094, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.8399999737739563}}], 'osds': [{'osd': 0, 'uuid': '268cd580-c931-4992-bb93-d6f63c2bfe88', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6801', 'nonce': 2811444477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6802', 'nonce': 2811444477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6804', 'nonce': 2811444477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6803', 'nonce': 2811444477}]}, 'public_addr': '172.21.15.137:6801/2811444477', 'cluster_addr': '172.21.15.137:6802/2811444477', 'heartbeat_back_addr': '172.21.15.137:6804/2811444477', 'heartbeat_front_addr': '172.21.15.137:6803/2811444477', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b06907bc-b499-46c0-8667-c5bdc0d85575', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 359, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6805', 'nonce': 874102487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6806', 'nonce': 874102487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6808', 'nonce': 874102487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6807', 'nonce': 874102487}]}, 'public_addr': '172.21.15.137:6805/874102487', 'cluster_addr': '172.21.15.137:6806/874102487', 'heartbeat_back_addr': '172.21.15.137:6808/874102487', 'heartbeat_front_addr': '172.21.15.137:6807/874102487', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '480ad6c5-0e0b-4405-bbe2-381682d00bae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6809', 'nonce': 1754684686}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6810', 'nonce': 1754684686}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6812', 'nonce': 1754684686}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6811', 'nonce': 1754684686}]}, 'public_addr': '172.21.15.137:6809/1754684686', 'cluster_addr': '172.21.15.137:6810/1754684686', 'heartbeat_back_addr': '172.21.15.137:6812/1754684686', 'heartbeat_front_addr': '172.21.15.137:6811/1754684686', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '178be3e0-f07b-4856-807b-2a1181e48a14', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.759490966796875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 359, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6813', 'nonce': 116058392}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6814', 'nonce': 116058392}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6816', 'nonce': 116058392}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6815', 'nonce': 116058392}]}, 'public_addr': '172.21.15.137:6813/116058392', 'cluster_addr': '172.21.15.137:6814/116058392', 'heartbeat_back_addr': '172.21.15.137:6816/116058392', 'heartbeat_front_addr': '172.21.15.137:6815/116058392', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7547102c-951a-4077-a094-fb8a65bfe6de', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 289, 'down_at': 359, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6800', 'nonce': 3654297954}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6801', 'nonce': 3654297954}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6803', 'nonce': 3654297954}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6802', 'nonce': 3654297954}]}, 'public_addr': '172.21.15.203:6800/3654297954', 'cluster_addr': '172.21.15.203:6801/3654297954', 'heartbeat_back_addr': '172.21.15.203:6803/3654297954', 'heartbeat_front_addr': '172.21.15.203:6802/3654297954', 'state': ['exists']}, {'osd': 5, 'uuid': '49c17901-8102-48a4-9348-fa1bdb63032c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6804', 'nonce': 837994222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6805', 'nonce': 837994222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6807', 'nonce': 837994222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6806', 'nonce': 837994222}]}, 'public_addr': '172.21.15.203:6804/837994222', 'cluster_addr': '172.21.15.203:6805/837994222', 'heartbeat_back_addr': '172.21.15.203:6807/837994222', 'heartbeat_front_addr': '172.21.15.203:6806/837994222', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '75f0ead4-e981-4a15-b1c1-d83dce871ab6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6808', 'nonce': 1494175466}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6809', 'nonce': 1494175466}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6811', 'nonce': 1494175466}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6810', 'nonce': 1494175466}]}, 'public_addr': '172.21.15.203:6808/1494175466', 'cluster_addr': '172.21.15.203:6809/1494175466', 'heartbeat_back_addr': '172.21.15.203:6811/1494175466', 'heartbeat_front_addr': '172.21.15.203:6810/1494175466', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c0dbf299-6798-4b99-9a85-62b123971f2a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 55, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6812', 'nonce': 2670809315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6813', 'nonce': 2670809315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6815', 'nonce': 2670809315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6814', 'nonce': 2670809315}]}, 'public_addr': '172.21.15.203:6812/2670809315', 'cluster_addr': '172.21.15.203:6813/2670809315', 'heartbeat_back_addr': '172.21.15.203:6815/2670809315', 'heartbeat_front_addr': '172.21.15.203:6814/2670809315', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:44:13.263276+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:43:41.550939+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:43:24.037996+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:43:16.491536+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:41:47.479848+0000', 'dead_epoch': 358}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:44:02.089826+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:44:01.308573+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:42:59.881262+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.13', 'mappings': [{'from': 6, 'to': 7}, {'from': 1, 'to': 2}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1b', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '4.1c', 'mappings': [{'from': 3, 'to': 0}, {'from': 1, 'to': 7}]}, {'pgid': '4.1d', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '5.b', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '6.1b', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.137:6800/2096039404': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:6800/3636177939': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/334967443': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/2101286111': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/3387637412': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1762989503': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3490817671': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:6800/1482147392': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1821482181': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/4028304023': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/621434521': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3160704935': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/2357293499': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/1762352860': '2024-09-19T07:31:25.389124+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T07:44:17.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:44:17.352 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:17.364 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:17 smithi137 ceph-mon[29408]: pgmap v698: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 655 B/s rd, 49 MiB/s wr, 786 op/s; 51770/310428 objects degraded (16.677%) 2024-09-18T07:44:17.364 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:17 smithi137 ceph-mon[29408]: 4.1c scrub starts 2024-09-18T07:44:17.364 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:17 smithi137 ceph-mon[29408]: 4.1c scrub ok 2024-09-18T07:44:17.364 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:17 smithi137 ceph-mon[29408]: 10.5 scrub ok 2024-09-18T07:44:17.364 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:17 smithi137 ceph-mon[39175]: pgmap v698: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 655 B/s rd, 49 MiB/s wr, 786 op/s; 51770/310428 objects degraded (16.677%) 2024-09-18T07:44:17.364 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:17 smithi137 ceph-mon[39175]: 4.1c scrub starts 2024-09-18T07:44:17.365 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:17 smithi137 ceph-mon[39175]: 4.1c scrub ok 2024-09-18T07:44:17.365 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:17 smithi137 ceph-mon[39175]: 10.5 scrub ok 2024-09-18T07:44:17.365 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:44:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:44:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:44:17.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:17.401 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:17.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:17.412 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:44:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:17 smithi203 ceph-mon[32237]: pgmap v698: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 655 B/s rd, 49 MiB/s wr, 786 op/s; 51770/310428 objects degraded (16.677%) 2024-09-18T07:44:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:17 smithi203 ceph-mon[32237]: 4.1c scrub starts 2024-09-18T07:44:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:17 smithi203 ceph-mon[32237]: 4.1c scrub ok 2024-09-18T07:44:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:17 smithi203 ceph-mon[32237]: 10.5 scrub ok 2024-09-18T07:44:17.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:17.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:17.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:17.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:17.646 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:17.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:17.705 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:17.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:17.806 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:17.828 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:44:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:44:17.845 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:17.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:17.946 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:17.970 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:44:17.970 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:44:17.970 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:44:17.970 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:44:17.970 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.970 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.34466.0:233 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2024-09-18T07:44:17.970 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:35:01.270598+0000", 2024-09-18T07:44:17.970 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 556.69409719800001, 2024-09-18T07:44:17.971 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14644679199999999, 2024-09-18T07:44:17.971 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:17.971 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:17.971 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:44:17.971 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:17.971 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.34466", 2024-09-18T07:44:17.971 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.203:0/952514737", 2024-09-18T07:44:17.971 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 233 2024-09-18T07:44:17.971 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.971 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:17.971 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.971 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:17.971 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270598+0000", 2024-09-18T07:44:17.972 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:17.972 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.972 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.972 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:17.972 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270598+0000", 2024-09-18T07:44:17.972 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:17.972 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.972 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.972 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:17.972 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270602+0000", 2024-09-18T07:44:17.972 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4960000000000001e-06 2024-09-18T07:44:17.972 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.972 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.973 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:17.973 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270613+0000", 2024-09-18T07:44:17.973 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0890000000000001e-05 2024-09-18T07:44:17.973 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.973 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.973 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:17.973 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270634+0000", 2024-09-18T07:44:17.973 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1484e-05 2024-09-18T07:44:17.973 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.974 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.974 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:17.974 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270643+0000", 2024-09-18T07:44:17.974 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1249999999999999e-06 2024-09-18T07:44:17.974 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.974 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.974 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:17.974 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270712+0000", 2024-09-18T07:44:17.974 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8733999999999996e-05 2024-09-18T07:44:17.974 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.974 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.974 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:44:17.974 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.270720+0000", 2024-09-18T07:44:17.975 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0209999999999999e-06 2024-09-18T07:44:17.975 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.975 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.975 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:17.975 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.402460+0000", 2024-09-18T07:44:17.975 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13173957 2024-09-18T07:44:17.975 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.975 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.975 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for missing object", 2024-09-18T07:44:17.975 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.402586+0000", 2024-09-18T07:44:17.975 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012606100000000001 2024-09-18T07:44:17.975 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.975 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.975 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:17.976 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.416785+0000", 2024-09-18T07:44:17.976 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.014199145 2024-09-18T07:44:17.976 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.976 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.976 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:17.976 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.416882+0000", 2024-09-18T07:44:17.976 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6990999999999993e-05 2024-09-18T07:44:17.976 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.976 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.976 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:17.976 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:35:01.417045+0000", 2024-09-18T07:44:17.976 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000163275 2024-09-18T07:44:17.976 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:17.977 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:17.977 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:17.977 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.977 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.977 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:170411 8.d e264/201 8:bde4aff4:::benchmark_data_smithi137_101942_object65601:head v 264'10478, mlcod=264'10476)", 2024-09-18T07:44:17.977 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:44.767677+0000", 2024-09-18T07:44:17.977 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 333.19701873899999, 2024-09-18T07:44:17.977 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.143244022, 2024-09-18T07:44:17.977 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:17.977 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:17.977 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:17.977 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:17.978 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.978 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:17.978 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.767677+0000", 2024-09-18T07:44:17.978 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:17.978 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.978 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.978 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:17.978 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.767677+0000", 2024-09-18T07:44:17.978 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:17.978 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.978 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.978 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:17.978 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.767679+0000", 2024-09-18T07:44:17.979 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.798e-06 2024-09-18T07:44:17.979 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.979 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.979 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:17.979 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.767688+0000", 2024-09-18T07:44:17.979 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0170000000000002e-06 2024-09-18T07:44:17.979 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.979 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.979 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:17.979 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.767700+0000", 2024-09-18T07:44:17.979 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1793e-05 2024-09-18T07:44:17.979 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.979 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.980 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:17.980 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.767708+0000", 2024-09-18T07:44:17.980 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1510000000000004e-06 2024-09-18T07:44:17.980 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.980 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.981 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:17.981 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.767764+0000", 2024-09-18T07:44:17.981 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6073999999999999e-05 2024-09-18T07:44:17.981 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.981 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.981 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:17.981 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.767784+0000", 2024-09-18T07:44:17.981 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0483e-05 2024-09-18T07:44:17.981 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.981 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.981 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:17.982 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.910878+0000", 2024-09-18T07:44:17.982 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14309329200000001 2024-09-18T07:44:17.982 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.982 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.982 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:17.982 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.910921+0000", 2024-09-18T07:44:17.982 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3414000000000001e-05 2024-09-18T07:44:17.982 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:17.982 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:17.982 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:17.982 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.982 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.982 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:170414 8.5 e264/255 8:a29be62f:::benchmark_data_smithi137_101942_object65604:head v 264'10576, mlcod=264'10574)", 2024-09-18T07:44:17.982 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:44.768915+0000", 2024-09-18T07:44:17.983 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 333.19578059999998, 2024-09-18T07:44:17.983 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15600961199999999, 2024-09-18T07:44:17.983 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:17.983 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:17.983 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:17.983 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:17.983 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.983 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:17.983 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768915+0000", 2024-09-18T07:44:17.983 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:17.983 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.983 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.983 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:17.983 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768915+0000", 2024-09-18T07:44:17.984 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:17.984 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.984 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.984 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:17.984 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768916+0000", 2024-09-18T07:44:17.984 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.24e-06 2024-09-18T07:44:17.984 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.984 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.984 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:17.984 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768923+0000", 2024-09-18T07:44:17.984 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7410000000000003e-06 2024-09-18T07:44:17.984 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.984 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.985 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:17.985 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768936+0000", 2024-09-18T07:44:17.985 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.345e-05 2024-09-18T07:44:17.985 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.985 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.985 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:17.985 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768944+0000", 2024-09-18T07:44:17.985 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0299999999999996e-06 2024-09-18T07:44:17.985 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.985 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.985 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:17.985 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.770341+0000", 2024-09-18T07:44:17.985 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001397061 2024-09-18T07:44:17.986 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.986 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.986 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:17.986 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.770358+0000", 2024-09-18T07:44:17.986 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7825999999999998e-05 2024-09-18T07:44:17.986 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.986 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.986 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:17.986 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.924876+0000", 2024-09-18T07:44:17.986 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.154517618 2024-09-18T07:44:17.986 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.986 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.986 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:17.987 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.924925+0000", 2024-09-18T07:44:17.987 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8646000000000001e-05 2024-09-18T07:44:17.987 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:17.987 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:17.987 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:17.987 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.987 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.987 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:170415 8.c e264/201 8:341230f3:::benchmark_data_smithi137_101942_object65605:head v 264'10461, mlcod=264'10459)", 2024-09-18T07:44:17.987 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:44.772649+0000", 2024-09-18T07:44:17.987 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 333.19204618700002, 2024-09-18T07:44:17.987 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.154977485, 2024-09-18T07:44:17.987 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:17.987 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:17.987 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:17.988 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:17.988 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.988 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:17.988 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.772649+0000", 2024-09-18T07:44:17.988 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:17.988 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.988 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.988 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:17.988 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.772649+0000", 2024-09-18T07:44:17.988 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:17.988 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.988 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.988 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:17.989 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.772652+0000", 2024-09-18T07:44:17.989 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3300000000000001e-06 2024-09-18T07:44:17.989 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.989 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.989 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:17.989 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.772662+0000", 2024-09-18T07:44:17.989 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0524e-05 2024-09-18T07:44:17.989 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.989 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.989 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:17.989 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.911734+0000", 2024-09-18T07:44:17.989 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13907166200000001 2024-09-18T07:44:17.989 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.990 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.990 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:17.990 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.911745+0000", 2024-09-18T07:44:17.990 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0925e-05 2024-09-18T07:44:17.990 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.990 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.990 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:17.990 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.911798+0000", 2024-09-18T07:44:17.990 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3384000000000003e-05 2024-09-18T07:44:17.990 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.990 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.990 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:17.990 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.911818+0000", 2024-09-18T07:44:17.990 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9242999999999999e-05 2024-09-18T07:44:17.991 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.991 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.991 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:17.991 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.927424+0000", 2024-09-18T07:44:17.991 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.015606201 2024-09-18T07:44:17.991 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.991 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.991 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:17.991 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.927627+0000", 2024-09-18T07:44:17.991 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00020321599999999999 2024-09-18T07:44:17.991 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:17.991 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:17.991 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:17.992 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.992 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.992 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:170425 8.5 e264/255 8:a030dc7e:::benchmark_data_smithi137_101942_object65615:head v 264'10577, mlcod=264'10574)", 2024-09-18T07:44:17.992 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:44.779176+0000", 2024-09-18T07:44:17.992 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 333.18552002199999, 2024-09-18T07:44:17.992 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.145788631, 2024-09-18T07:44:17.992 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:17.992 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:17.992 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:17.992 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:17.992 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.992 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:17.992 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.779176+0000", 2024-09-18T07:44:17.993 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:17.993 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.993 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.993 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:17.993 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.779176+0000", 2024-09-18T07:44:17.993 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:17.993 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.993 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.993 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:17.993 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.779178+0000", 2024-09-18T07:44:17.993 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6759999999999999e-06 2024-09-18T07:44:17.993 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.993 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.993 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:17.994 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.779188+0000", 2024-09-18T07:44:17.994 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9320000000000008e-06 2024-09-18T07:44:17.994 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.994 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.994 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:17.994 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.911967+0000", 2024-09-18T07:44:17.994 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13277883200000001 2024-09-18T07:44:17.994 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.994 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.994 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:17.994 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.911973+0000", 2024-09-18T07:44:17.994 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0209999999999996e-06 2024-09-18T07:44:17.994 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.995 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.995 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:17.995 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.913718+0000", 2024-09-18T07:44:17.995 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0017446479999999999 2024-09-18T07:44:17.995 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.995 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.995 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:17.995 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.913742+0000", 2024-09-18T07:44:17.995 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4189000000000002e-05 2024-09-18T07:44:17.995 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.995 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.995 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:17.995 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.924931+0000", 2024-09-18T07:44:17.995 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.01118863 2024-09-18T07:44:17.996 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.996 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.996 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:17.996 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.924964+0000", 2024-09-18T07:44:17.996 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3702999999999999e-05 2024-09-18T07:44:17.996 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:17.996 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:17.996 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:17.996 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.996 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.996 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:156461 9.d 9:b7fc06f1:::benchmark_data_smithi137_126072_object23961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:44:17.996 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:24.401407+0000", 2024-09-18T07:44:17.996 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 173.56328909699999, 2024-09-18T07:44:17.997 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16122866899999999, 2024-09-18T07:44:17.997 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:17.997 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:17.997 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:17.997 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:17.997 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:44:17.997 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:44:17.997 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 156461 2024-09-18T07:44:17.997 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.997 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:17.997 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.997 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:17.997 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401407+0000", 2024-09-18T07:44:17.997 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:17.998 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.998 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.998 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:17.998 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401407+0000", 2024-09-18T07:44:17.998 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:17.998 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.998 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.998 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:17.998 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401409+0000", 2024-09-18T07:44:17.998 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0150000000000002e-06 2024-09-18T07:44:17.998 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.998 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.998 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:17.999 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401414+0000", 2024-09-18T07:44:17.999 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6810000000000003e-06 2024-09-18T07:44:17.999 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.999 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.999 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:17.999 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401428+0000", 2024-09-18T07:44:17.999 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4049000000000001e-05 2024-09-18T07:44:17.999 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.999 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:17.999 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:17.999 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401436+0000", 2024-09-18T07:44:17.999 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5549999999999997e-06 2024-09-18T07:44:17.999 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:17.999 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.000 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:18.000 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401509+0000", 2024-09-18T07:44:18.000 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3232999999999995e-05 2024-09-18T07:44:18.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.000 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.000 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:18.000 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401613+0000", 2024-09-18T07:44:18.000 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010402400000000001 2024-09-18T07:44:18.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:18.002 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.002 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:44:18.002 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401718+0000", 2024-09-18T07:44:18.002 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010482800000000001 2024-09-18T07:44:18.002 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.002 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.002 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:18.002 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409266+0000", 2024-09-18T07:44:18.002 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0075485639999999998 2024-09-18T07:44:18.002 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.003 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.003 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409351+0000", 2024-09-18T07:44:18.003 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4796000000000006e-05 2024-09-18T07:44:18.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.003 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.003 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.562555+0000", 2024-09-18T07:44:18.003 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.153203325 2024-09-18T07:44:18.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.003 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:18.003 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.562599+0000", 2024-09-18T07:44:18.004 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4431000000000002e-05 2024-09-18T07:44:18.004 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.004 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.004 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:18.004 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.562635+0000", 2024-09-18T07:44:18.004 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6168000000000003e-05 2024-09-18T07:44:18.004 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.004 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:18.004 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.004 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.004 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.004 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:156463 9.d 9:b608ec1e:::benchmark_data_smithi137_126072_object23963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:44:18.005 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:24.403102+0000", 2024-09-18T07:44:18.005 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 173.561593937, 2024-09-18T07:44:18.005 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16236466999999999, 2024-09-18T07:44:18.005 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:18.005 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:18.005 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:18.005 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:18.005 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:44:18.005 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:44:18.005 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 156463 2024-09-18T07:44:18.005 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.005 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:18.005 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.006 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:18.006 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403102+0000", 2024-09-18T07:44:18.006 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.006 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.006 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.006 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:18.006 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403102+0000", 2024-09-18T07:44:18.006 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.006 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.006 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.006 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:18.006 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403103+0000", 2024-09-18T07:44:18.006 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.767e-06 2024-09-18T07:44:18.007 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.007 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.007 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:18.007 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403108+0000", 2024-09-18T07:44:18.007 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5549999999999996e-06 2024-09-18T07:44:18.007 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.007 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.007 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:18.007 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403117+0000", 2024-09-18T07:44:18.007 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9390000000000003e-06 2024-09-18T07:44:18.007 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.007 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.007 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:18.007 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403123+0000", 2024-09-18T07:44:18.008 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4309999999999999e-06 2024-09-18T07:44:18.008 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.008 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.008 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:18.008 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404263+0000", 2024-09-18T07:44:18.008 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001140032 2024-09-18T07:44:18.008 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.008 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.008 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:18.008 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404367+0000", 2024-09-18T07:44:18.008 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000103514 2024-09-18T07:44:18.008 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.008 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.009 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:44:18.009 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404535+0000", 2024-09-18T07:44:18.009 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000167641 2024-09-18T07:44:18.009 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.009 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.009 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:18.009 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409418+0000", 2024-09-18T07:44:18.009 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0048834990000000003 2024-09-18T07:44:18.009 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.009 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.009 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.009 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409552+0000", 2024-09-18T07:44:18.009 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000133688 2024-09-18T07:44:18.010 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.010 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.010 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.564499+0000", 2024-09-18T07:44:18.010 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15494714900000001 2024-09-18T07:44:18.010 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.010 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:18.010 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.565425+0000", 2024-09-18T07:44:18.010 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00092621499999999998 2024-09-18T07:44:18.010 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.010 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:18.011 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.565466+0000", 2024-09-18T07:44:18.011 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1239999999999998e-05 2024-09-18T07:44:18.011 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.011 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:18.011 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.011 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.011 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.011 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:156465 9.e 9:710e39e4:::benchmark_data_smithi137_126072_object23965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:44:18.011 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:24.404394+0000", 2024-09-18T07:44:18.011 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 173.56030211500001, 2024-09-18T07:44:18.012 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.157357105, 2024-09-18T07:44:18.012 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:18.012 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:18.012 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:18.012 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:18.012 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:44:18.012 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:44:18.012 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 156465 2024-09-18T07:44:18.012 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.012 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:18.012 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.012 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:18.012 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404394+0000", 2024-09-18T07:44:18.013 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.013 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.013 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.013 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:18.013 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404394+0000", 2024-09-18T07:44:18.013 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.013 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.013 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.013 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:18.013 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404395+0000", 2024-09-18T07:44:18.013 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.55e-06 2024-09-18T07:44:18.013 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.013 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.013 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:18.014 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404400+0000", 2024-09-18T07:44:18.014 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5299999999999998e-06 2024-09-18T07:44:18.014 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.014 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.014 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:18.014 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404410+0000", 2024-09-18T07:44:18.014 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0139999999999999e-05 2024-09-18T07:44:18.014 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.014 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.014 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:18.014 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404417+0000", 2024-09-18T07:44:18.014 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9979999999999997e-06 2024-09-18T07:44:18.014 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:18.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404520+0000", 2024-09-18T07:44:18.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000103601 2024-09-18T07:44:18.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:18.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404618+0000", 2024-09-18T07:44:18.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7317000000000004e-05 2024-09-18T07:44:18.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:44:18.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404713+0000", 2024-09-18T07:44:18.016 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5427000000000006e-05 2024-09-18T07:44:18.016 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.016 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.016 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.016 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409836+0000", 2024-09-18T07:44:18.016 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0051231840000000002 2024-09-18T07:44:18.016 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.016 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.016 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:18.016 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409962+0000", 2024-09-18T07:44:18.016 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012611299999999999 2024-09-18T07:44:18.016 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.016 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.017 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.017 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.561643+0000", 2024-09-18T07:44:18.017 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15168100000000001 2024-09-18T07:44:18.017 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.017 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.017 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:18.017 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.561701+0000", 2024-09-18T07:44:18.017 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8010000000000002e-05 2024-09-18T07:44:18.017 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.017 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.017 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:18.017 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.561751+0000", 2024-09-18T07:44:18.017 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9234999999999999e-05 2024-09-18T07:44:18.018 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.018 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:18.018 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.018 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.018 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.018 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:156469 9.e 9:772a9531:::benchmark_data_smithi137_126072_object23969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:44:18.018 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:24.410883+0000", 2024-09-18T07:44:18.018 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 173.55381291500001, 2024-09-18T07:44:18.018 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15481848500000001, 2024-09-18T07:44:18.018 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:18.018 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:18.018 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:18.018 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:18.019 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:44:18.019 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:44:18.019 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 156469 2024-09-18T07:44:18.019 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.019 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:18.019 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.019 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:18.019 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.410883+0000", 2024-09-18T07:44:18.019 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.019 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.019 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.019 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:18.019 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.410883+0000", 2024-09-18T07:44:18.020 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.020 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.020 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:18.020 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.410885+0000", 2024-09-18T07:44:18.020 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.979e-06 2024-09-18T07:44:18.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.020 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.020 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:18.020 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.410891+0000", 2024-09-18T07:44:18.020 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1840000000000004e-06 2024-09-18T07:44:18.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.020 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.021 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:18.021 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.410905+0000", 2024-09-18T07:44:18.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4449e-05 2024-09-18T07:44:18.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.021 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:18.021 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.410913+0000", 2024-09-18T07:44:18.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6190000000000002e-06 2024-09-18T07:44:18.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.021 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:18.021 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.410967+0000", 2024-09-18T07:44:18.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4381e-05 2024-09-18T07:44:18.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.022 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:18.022 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.411099+0000", 2024-09-18T07:44:18.022 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013190500000000001 2024-09-18T07:44:18.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.022 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:44:18.022 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.411214+0000", 2024-09-18T07:44:18.022 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000114537 2024-09-18T07:44:18.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.022 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:18.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.413455+0000", 2024-09-18T07:44:18.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022413810000000002 2024-09-18T07:44:18.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.023 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.023 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.414426+0000", 2024-09-18T07:44:18.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00097079800000000004 2024-09-18T07:44:18.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.023 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.023 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.565613+0000", 2024-09-18T07:44:18.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15118668599999999 2024-09-18T07:44:18.024 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.024 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:18.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.565663+0000", 2024-09-18T07:44:18.024 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0096999999999998e-05 2024-09-18T07:44:18.024 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.024 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:18.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.565701+0000", 2024-09-18T07:44:18.024 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8469e-05 2024-09-18T07:44:18.024 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.024 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:18.024 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.024 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.025 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.025 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:156472 9.e 9:78b53fba:::benchmark_data_smithi137_126072_object23972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:44:18.025 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:24.415983+0000", 2024-09-18T07:44:18.025 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 173.54871219500001, 2024-09-18T07:44:18.025 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15303439799999999, 2024-09-18T07:44:18.025 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:18.025 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:18.025 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:18.025 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:18.025 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:44:18.025 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:44:18.025 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 156472 2024-09-18T07:44:18.025 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.026 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:18.026 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.026 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:18.026 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.415983+0000", 2024-09-18T07:44:18.026 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.026 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.026 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.026 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:18.026 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.415983+0000", 2024-09-18T07:44:18.026 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.026 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.026 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.026 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:18.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.415985+0000", 2024-09-18T07:44:18.027 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5409999999999999e-06 2024-09-18T07:44:18.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:18.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.415990+0000", 2024-09-18T07:44:18.027 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2719999999999997e-06 2024-09-18T07:44:18.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:18.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.416001+0000", 2024-09-18T07:44:18.027 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0973e-05 2024-09-18T07:44:18.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.028 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:18.028 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.416007+0000", 2024-09-18T07:44:18.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8259999999999998e-06 2024-09-18T07:44:18.028 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.028 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.028 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:18.028 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.416049+0000", 2024-09-18T07:44:18.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1869000000000002e-05 2024-09-18T07:44:18.028 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.028 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.028 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:18.028 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.416132+0000", 2024-09-18T07:44:18.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3554999999999995e-05 2024-09-18T07:44:18.029 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.029 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.029 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:44:18.029 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.416228+0000", 2024-09-18T07:44:18.029 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.522e-05 2024-09-18T07:44:18.029 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.029 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.029 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:18.029 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.417966+0000", 2024-09-18T07:44:18.029 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0017378579999999999 2024-09-18T07:44:18.029 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.029 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.029 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.030 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.418190+0000", 2024-09-18T07:44:18.030 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00022425 2024-09-18T07:44:18.030 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.030 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.030 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.030 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.568914+0000", 2024-09-18T07:44:18.030 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.150724675 2024-09-18T07:44:18.030 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.030 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.030 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:18.030 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.568976+0000", 2024-09-18T07:44:18.030 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1675999999999997e-05 2024-09-18T07:44:18.030 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.031 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.031 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:18.031 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.569018+0000", 2024-09-18T07:44:18.031 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1683000000000001e-05 2024-09-18T07:44:18.031 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.031 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:18.031 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.031 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.031 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.031 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:172965 9.e 9:7fd9b0fe:::benchmark_data_smithi137_126072_object40465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-18T07:44:18.031 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:38.554115+0000", 2024-09-18T07:44:18.031 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 159.41058097499999, 2024-09-18T07:44:18.031 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.179108299, 2024-09-18T07:44:18.032 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:18.032 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:18.032 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:18.032 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:18.032 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:44:18.032 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:44:18.032 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 172965 2024-09-18T07:44:18.032 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.032 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:18.032 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.032 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:18.032 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554115+0000", 2024-09-18T07:44:18.033 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.033 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.033 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:18.033 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554115+0000", 2024-09-18T07:44:18.033 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.033 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.033 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:18.033 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554116+0000", 2024-09-18T07:44:18.033 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.668e-06 2024-09-18T07:44:18.033 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.033 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:18.034 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554122+0000", 2024-09-18T07:44:18.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9159999999999998e-06 2024-09-18T07:44:18.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:18.034 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554133+0000", 2024-09-18T07:44:18.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1134e-05 2024-09-18T07:44:18.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:18.034 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554139+0000", 2024-09-18T07:44:18.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1889999999999998e-06 2024-09-18T07:44:18.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:18.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554177+0000", 2024-09-18T07:44:18.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8720000000000002e-05 2024-09-18T07:44:18.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:18.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554264+0000", 2024-09-18T07:44:18.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6942999999999997e-05 2024-09-18T07:44:18.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:44:18.036 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554389+0000", 2024-09-18T07:44:18.036 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012495799999999999 2024-09-18T07:44:18.036 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.036 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.036 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.036 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.559699+0000", 2024-09-18T07:44:18.036 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0053099760000000001 2024-09-18T07:44:18.036 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.036 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.036 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:18.036 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.559715+0000", 2024-09-18T07:44:18.036 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5586999999999998e-05 2024-09-18T07:44:18.036 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.037 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733124+0000", 2024-09-18T07:44:18.037 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17340927 2024-09-18T07:44:18.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:18.037 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733181+0000", 2024-09-18T07:44:18.037 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6845000000000002e-05 2024-09-18T07:44:18.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:18.037 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733223+0000", 2024-09-18T07:44:18.038 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2092999999999998e-05 2024-09-18T07:44:18.038 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.038 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:18.038 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.038 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.038 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:172970 9.3 9:c7e8a587:::benchmark_data_smithi137_126072_object40470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-18T07:44:18.038 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:38.554401+0000", 2024-09-18T07:44:18.038 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 159.410294459, 2024-09-18T07:44:18.038 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17690262100000001, 2024-09-18T07:44:18.038 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:18.038 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:18.038 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:18.038 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:18.039 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:44:18.039 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:44:18.039 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 172970 2024-09-18T07:44:18.039 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.039 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:18.039 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.039 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:18.039 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554401+0000", 2024-09-18T07:44:18.039 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.039 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.039 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.039 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:18.039 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554401+0000", 2024-09-18T07:44:18.040 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.040 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.040 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:18.040 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554402+0000", 2024-09-18T07:44:18.040 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.299e-06 2024-09-18T07:44:18.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.040 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.040 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:18.040 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554407+0000", 2024-09-18T07:44:18.040 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6870000000000002e-06 2024-09-18T07:44:18.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.040 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.041 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:18.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554418+0000", 2024-09-18T07:44:18.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0696000000000001e-05 2024-09-18T07:44:18.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.041 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:18.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554426+0000", 2024-09-18T07:44:18.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0989999999999999e-06 2024-09-18T07:44:18.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.041 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:18.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554503+0000", 2024-09-18T07:44:18.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6638999999999996e-05 2024-09-18T07:44:18.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:18.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554615+0000", 2024-09-18T07:44:18.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000112816 2024-09-18T07:44:18.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:44:18.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554762+0000", 2024-09-18T07:44:18.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014699099999999999 2024-09-18T07:44:18.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:18.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.558265+0000", 2024-09-18T07:44:18.043 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0035024539999999999 2024-09-18T07:44:18.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.043 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.043 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.564048+0000", 2024-09-18T07:44:18.043 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0057829969999999998 2024-09-18T07:44:18.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.043 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.043 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731227+0000", 2024-09-18T07:44:18.043 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16717880500000001 2024-09-18T07:44:18.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:18.044 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731260+0000", 2024-09-18T07:44:18.044 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2845000000000001e-05 2024-09-18T07:44:18.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.044 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:18.044 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731304+0000", 2024-09-18T07:44:18.044 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4292999999999998e-05 2024-09-18T07:44:18.044 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.044 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:18.044 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.044 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.045 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:172972 9.e 9:796b9798:::benchmark_data_smithi137_126072_object40472:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-18T07:44:18.045 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:38.556110+0000", 2024-09-18T07:44:18.045 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 159.40858564300001, 2024-09-18T07:44:18.045 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17726146800000001, 2024-09-18T07:44:18.045 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:18.045 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:18.045 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:18.045 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:18.045 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:44:18.045 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:44:18.045 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 172972 2024-09-18T07:44:18.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.045 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:18.046 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:18.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556110+0000", 2024-09-18T07:44:18.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.046 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.046 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:18.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556110+0000", 2024-09-18T07:44:18.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.046 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.046 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:18.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556112+0000", 2024-09-18T07:44:18.047 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.711e-06 2024-09-18T07:44:18.047 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.047 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.047 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:18.047 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556119+0000", 2024-09-18T07:44:18.047 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.644e-06 2024-09-18T07:44:18.047 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.047 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.047 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:18.047 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556129+0000", 2024-09-18T07:44:18.047 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9850000000000001e-06 2024-09-18T07:44:18.047 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.047 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.048 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:18.048 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556136+0000", 2024-09-18T07:44:18.048 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7789999999999998e-06 2024-09-18T07:44:18.048 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.048 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.048 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:18.048 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.557007+0000", 2024-09-18T07:44:18.048 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00087092099999999998 2024-09-18T07:44:18.048 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.048 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.048 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:18.048 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.557105+0000", 2024-09-18T07:44:18.048 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7640999999999996e-05 2024-09-18T07:44:18.049 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.049 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.049 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:44:18.049 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.557203+0000", 2024-09-18T07:44:18.049 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8666999999999996e-05 2024-09-18T07:44:18.049 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.049 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.049 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.049 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.559808+0000", 2024-09-18T07:44:18.049 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026044599999999998 2024-09-18T07:44:18.049 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.049 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.049 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:18.050 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.559832+0000", 2024-09-18T07:44:18.050 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4426e-05 2024-09-18T07:44:18.050 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.050 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.050 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.050 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733280+0000", 2024-09-18T07:44:18.050 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17344823700000001 2024-09-18T07:44:18.050 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.050 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.050 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:18.050 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733334+0000", 2024-09-18T07:44:18.050 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3946999999999997e-05 2024-09-18T07:44:18.050 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.051 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.051 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:18.051 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733371+0000", 2024-09-18T07:44:18.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7049999999999999e-05 2024-09-18T07:44:18.051 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.051 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:18.051 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.051 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.051 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:172980 9.3 9:ce0a7506:::benchmark_data_smithi137_126072_object40480:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-18T07:44:18.051 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:38.560199+0000", 2024-09-18T07:44:18.052 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 159.404496814, 2024-09-18T07:44:18.052 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.171283607, 2024-09-18T07:44:18.052 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:18.052 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:18.052 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:18.052 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:18.052 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:44:18.052 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:44:18.052 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 172980 2024-09-18T07:44:18.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.052 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:18.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:18.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560199+0000", 2024-09-18T07:44:18.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:18.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560199+0000", 2024-09-18T07:44:18.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:18.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560201+0000", 2024-09-18T07:44:18.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9709999999999998e-06 2024-09-18T07:44:18.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.054 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.054 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:18.054 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560206+0000", 2024-09-18T07:44:18.054 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3750000000000002e-06 2024-09-18T07:44:18.054 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.054 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.054 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:18.054 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560221+0000", 2024-09-18T07:44:18.054 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.487e-05 2024-09-18T07:44:18.054 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.054 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.054 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:18.054 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560228+0000", 2024-09-18T07:44:18.055 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4200000000000001e-06 2024-09-18T07:44:18.055 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.055 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.055 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:18.055 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560284+0000", 2024-09-18T07:44:18.055 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5084999999999999e-05 2024-09-18T07:44:18.055 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.055 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.055 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:18.055 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560415+0000", 2024-09-18T07:44:18.055 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013142500000000001 2024-09-18T07:44:18.055 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.055 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.055 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:44:18.056 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560525+0000", 2024-09-18T07:44:18.056 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010991400000000001 2024-09-18T07:44:18.056 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.056 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.056 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.056 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.564084+0000", 2024-09-18T07:44:18.056 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00355953 2024-09-18T07:44:18.056 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.056 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.056 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:18.056 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.566190+0000", 2024-09-18T07:44:18.056 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021053690000000002 2024-09-18T07:44:18.057 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.057 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.057 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.057 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731396+0000", 2024-09-18T07:44:18.057 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16520663299999999 2024-09-18T07:44:18.057 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.057 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.057 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:18.057 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731446+0000", 2024-09-18T07:44:18.057 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9199000000000001e-05 2024-09-18T07:44:18.057 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.057 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.057 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:18.058 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731482+0000", 2024-09-18T07:44:18.058 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6816000000000001e-05 2024-09-18T07:44:18.058 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.058 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:18.058 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.058 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.058 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.058 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:172986 9.d 9:b4a569cc:::benchmark_data_smithi137_126072_object40486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-18T07:44:18.058 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:38.568405+0000", 2024-09-18T07:44:18.058 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 159.396290247, 2024-09-18T07:44:18.058 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16278958299999999, 2024-09-18T07:44:18.058 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:18.058 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:18.059 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:18.059 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:18.059 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:44:18.059 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:44:18.059 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 172986 2024-09-18T07:44:18.059 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.059 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:18.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.059 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:18.059 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568405+0000", 2024-09-18T07:44:18.059 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.059 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.060 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:18.060 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568405+0000", 2024-09-18T07:44:18.060 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.060 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.060 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.060 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:18.060 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568408+0000", 2024-09-18T07:44:18.060 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6929999999999999e-06 2024-09-18T07:44:18.060 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.060 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.060 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:18.060 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568414+0000", 2024-09-18T07:44:18.060 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5219999999999998e-06 2024-09-18T07:44:18.061 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:18.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:18.061 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:18.062 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.062 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.062 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:18.062 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568428+0000", 2024-09-18T07:44:18.062 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4134000000000001e-05 2024-09-18T07:44:18.063 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.063 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.063 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:18.063 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568435+0000", 2024-09-18T07:44:18.063 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.396e-06 2024-09-18T07:44:18.063 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.063 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.063 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:18.063 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568512+0000", 2024-09-18T07:44:18.063 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6614999999999997e-05 2024-09-18T07:44:18.063 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.063 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.063 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:18.064 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568623+0000", 2024-09-18T07:44:18.064 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011123799999999999 2024-09-18T07:44:18.064 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.064 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.064 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:44:18.064 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568744+0000", 2024-09-18T07:44:18.064 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012053900000000001 2024-09-18T07:44:18.064 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.064 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.064 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:18.064 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.571214+0000", 2024-09-18T07:44:18.064 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0024707739999999998 2024-09-18T07:44:18.064 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.065 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.065 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.065 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.571292+0000", 2024-09-18T07:44:18.065 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7903000000000003e-05 2024-09-18T07:44:18.065 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.065 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.065 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.065 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731105+0000", 2024-09-18T07:44:18.065 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.159812378 2024-09-18T07:44:18.065 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.065 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.065 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:18.065 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731154+0000", 2024-09-18T07:44:18.066 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9450000000000003e-05 2024-09-18T07:44:18.066 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.066 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.066 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:18.066 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731195+0000", 2024-09-18T07:44:18.066 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0941e-05 2024-09-18T07:44:18.066 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.066 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:18.066 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.066 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.066 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.066 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:244796 9.d 9:b8ad6f51:::benchmark_data_smithi137_126072_object112296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-18T07:44:18.066 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:36.228692+0000", 2024-09-18T07:44:18.067 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 101.73600364799999, 2024-09-18T07:44:18.067 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16804142, 2024-09-18T07:44:18.067 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:18.067 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:18.067 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:18.067 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:18.067 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:44:18.067 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:44:18.067 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 244796 2024-09-18T07:44:18.067 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.067 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:18.067 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.067 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:18.068 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228692+0000", 2024-09-18T07:44:18.068 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.068 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.068 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.068 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:18.068 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228692+0000", 2024-09-18T07:44:18.068 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.068 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.068 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.068 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:18.068 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228693+0000", 2024-09-18T07:44:18.068 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1930000000000001e-06 2024-09-18T07:44:18.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.069 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.069 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:18.069 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228697+0000", 2024-09-18T07:44:18.069 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.143e-06 2024-09-18T07:44:18.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.069 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.069 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:18.069 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228707+0000", 2024-09-18T07:44:18.069 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3489999999999998e-06 2024-09-18T07:44:18.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.069 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.069 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:18.070 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228713+0000", 2024-09-18T07:44:18.070 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1720000000000004e-06 2024-09-18T07:44:18.070 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.070 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.070 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:18.070 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233077+0000", 2024-09-18T07:44:18.070 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0043645890000000003 2024-09-18T07:44:18.070 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.070 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.070 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:18.070 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233186+0000", 2024-09-18T07:44:18.070 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108081 2024-09-18T07:44:18.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.071 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.071 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:44:18.071 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233291+0000", 2024-09-18T07:44:18.071 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000105025 2024-09-18T07:44:18.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.071 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.071 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:18.071 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.240477+0000", 2024-09-18T07:44:18.071 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0071862430000000001 2024-09-18T07:44:18.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.071 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.072 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.072 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.248094+0000", 2024-09-18T07:44:18.072 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0076167459999999998 2024-09-18T07:44:18.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.072 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.072 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.072 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.396661+0000", 2024-09-18T07:44:18.072 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14856715000000001 2024-09-18T07:44:18.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.072 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.073 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:18.073 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.396703+0000", 2024-09-18T07:44:18.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1850000000000001e-05 2024-09-18T07:44:18.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.073 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:18.073 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.396733+0000", 2024-09-18T07:44:18.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0879e-05 2024-09-18T07:44:18.073 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.073 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:18.073 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.074 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.074 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:244803 9.d 9:be42e048:::benchmark_data_smithi137_126072_object112303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-18T07:44:18.074 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:36.233785+0000", 2024-09-18T07:44:18.074 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 101.730910208, 2024-09-18T07:44:18.074 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16784005699999999, 2024-09-18T07:44:18.074 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:18.074 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:18.074 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:18.074 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:18.074 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:44:18.075 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:44:18.075 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 244803 2024-09-18T07:44:18.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.075 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:18.075 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:18.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233785+0000", 2024-09-18T07:44:18.075 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.075 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:18.076 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233785+0000", 2024-09-18T07:44:18.076 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.076 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.076 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.076 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:18.076 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233787+0000", 2024-09-18T07:44:18.076 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8279999999999999e-06 2024-09-18T07:44:18.076 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.076 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.076 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:18.076 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233793+0000", 2024-09-18T07:44:18.076 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7790000000000001e-06 2024-09-18T07:44:18.077 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.077 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.077 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:18.077 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233803+0000", 2024-09-18T07:44:18.077 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5920000000000007e-06 2024-09-18T07:44:18.077 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.077 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.077 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:18.077 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233810+0000", 2024-09-18T07:44:18.077 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7300000000000005e-06 2024-09-18T07:44:18.077 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.078 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.078 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:18.078 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.237384+0000", 2024-09-18T07:44:18.078 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003573629 2024-09-18T07:44:18.078 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.078 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.078 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:18.078 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.237500+0000", 2024-09-18T07:44:18.078 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000116216 2024-09-18T07:44:18.078 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:44:18.079 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.237637+0000", 2024-09-18T07:44:18.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013664699999999999 2024-09-18T07:44:18.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:18.079 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.243662+0000", 2024-09-18T07:44:18.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0060250429999999999 2024-09-18T07:44:18.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.250643+0000", 2024-09-18T07:44:18.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069812499999999996 2024-09-18T07:44:18.080 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.080 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.080 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401571+0000", 2024-09-18T07:44:18.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15092767600000001 2024-09-18T07:44:18.080 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.081 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.081 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:18.081 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401601+0000", 2024-09-18T07:44:18.081 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9889000000000001e-05 2024-09-18T07:44:18.081 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.081 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.081 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:18.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401625+0000", 2024-09-18T07:44:18.082 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4777999999999999e-05 2024-09-18T07:44:18.082 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.082 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:18.082 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.082 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.082 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.082 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:244820 9.d 9:b40b063b:::benchmark_data_smithi137_126072_object112320:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-18T07:44:18.082 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:36.251409+0000", 2024-09-18T07:44:18.082 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 101.71328627299999, 2024-09-18T07:44:18.083 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.150296187, 2024-09-18T07:44:18.083 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:18.083 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:18.083 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:18.083 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:18.083 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:44:18.083 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:44:18.083 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 244820 2024-09-18T07:44:18.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.083 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:18.083 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.084 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:18.084 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251409+0000", 2024-09-18T07:44:18.084 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.084 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.084 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:18.084 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251409+0000", 2024-09-18T07:44:18.084 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.084 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.085 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:18.085 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251411+0000", 2024-09-18T07:44:18.085 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.84e-06 2024-09-18T07:44:18.085 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.085 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.085 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:18.085 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251417+0000", 2024-09-18T07:44:18.085 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8300000000000001e-06 2024-09-18T07:44:18.085 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.085 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.085 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:18.086 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251428+0000", 2024-09-18T07:44:18.086 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1008e-05 2024-09-18T07:44:18.086 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.086 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.086 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:18.086 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251435+0000", 2024-09-18T07:44:18.086 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5019999999999998e-06 2024-09-18T07:44:18.086 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.086 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.086 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:18.086 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251496+0000", 2024-09-18T07:44:18.087 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1353999999999996e-05 2024-09-18T07:44:18.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.087 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.087 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:18.087 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251578+0000", 2024-09-18T07:44:18.087 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1966999999999996e-05 2024-09-18T07:44:18.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.087 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.087 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:44:18.087 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251645+0000", 2024-09-18T07:44:18.087 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7331999999999998e-05 2024-09-18T07:44:18.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:18.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.256879+0000", 2024-09-18T07:44:18.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.005234069 2024-09-18T07:44:18.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.257762+0000", 2024-09-18T07:44:18.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00088236200000000003 2024-09-18T07:44:18.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401653+0000", 2024-09-18T07:44:18.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14389142399999999 2024-09-18T07:44:18.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:18.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401681+0000", 2024-09-18T07:44:18.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7861e-05 2024-09-18T07:44:18.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:18.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401706+0000", 2024-09-18T07:44:18.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4638e-05 2024-09-18T07:44:18.090 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.090 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:18.090 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.090 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.090 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:244823 9.e 9:7e246909:::benchmark_data_smithi137_126072_object112323:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-18T07:44:18.090 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:36.257973+0000", 2024-09-18T07:44:18.090 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 101.706722184, 2024-09-18T07:44:18.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14363669600000001, 2024-09-18T07:44:18.090 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:18.091 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:18.091 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:18.091 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:18.091 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:44:18.091 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:44:18.091 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 244823 2024-09-18T07:44:18.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.091 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:18.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:18.091 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.257973+0000", 2024-09-18T07:44:18.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.257973+0000", 2024-09-18T07:44:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.257976+0000", 2024-09-18T07:44:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2400000000000002e-06 2024-09-18T07:44:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:18.093 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.257982+0000", 2024-09-18T07:44:18.093 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1190000000000002e-06 2024-09-18T07:44:18.093 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.093 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.093 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:18.093 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.257990+0000", 2024-09-18T07:44:18.093 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2460000000000003e-06 2024-09-18T07:44:18.093 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.093 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.093 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:18.093 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.257995+0000", 2024-09-18T07:44:18.093 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8620000000000002e-06 2024-09-18T07:44:18.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.094 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.094 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:18.094 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.258062+0000", 2024-09-18T07:44:18.094 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6688999999999998e-05 2024-09-18T07:44:18.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.094 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.094 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:18.094 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.258187+0000", 2024-09-18T07:44:18.094 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012529800000000001 2024-09-18T07:44:18.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.094 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.094 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:44:18.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.258308+0000", 2024-09-18T07:44:18.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000120919 2024-09-18T07:44:18.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.095 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.262258+0000", 2024-09-18T07:44:18.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003950294 2024-09-18T07:44:18.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.095 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:18.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.262287+0000", 2024-09-18T07:44:18.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8688e-05 2024-09-18T07:44:18.096 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.096 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.096 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.096 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401550+0000", 2024-09-18T07:44:18.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13926276100000001 2024-09-18T07:44:18.096 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.096 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.096 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:18.096 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401586+0000", 2024-09-18T07:44:18.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5988999999999997e-05 2024-09-18T07:44:18.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.097 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.097 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:18.097 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401610+0000", 2024-09-18T07:44:18.097 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4590999999999999e-05 2024-09-18T07:44:18.097 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.097 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:18.097 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.097 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.098 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:244824 9.e 9:7faa6a78:::benchmark_data_smithi137_126072_object112324:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-09-18T07:44:18.098 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:36.258967+0000", 2024-09-18T07:44:18.098 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 101.705728982, 2024-09-18T07:44:18.098 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.142722036, 2024-09-18T07:44:18.098 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:18.098 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:18.098 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:18.098 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:44:18.098 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:44:18.098 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:44:18.098 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 244824 2024-09-18T07:44:18.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.099 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:18.099 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.099 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:18.099 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.258967+0000", 2024-09-18T07:44:18.099 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.099 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.099 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:18.099 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.258967+0000", 2024-09-18T07:44:18.099 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:18.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.100 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.100 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:18.100 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.258968+0000", 2024-09-18T07:44:18.100 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.065e-06 2024-09-18T07:44:18.100 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.100 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.100 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:18.100 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.258972+0000", 2024-09-18T07:44:18.100 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8240000000000001e-06 2024-09-18T07:44:18.100 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.100 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.100 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:18.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.258981+0000", 2024-09-18T07:44:18.101 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7710000000000001e-06 2024-09-18T07:44:18.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.101 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.101 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:18.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.258987+0000", 2024-09-18T07:44:18.101 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4809999999999997e-06 2024-09-18T07:44:18.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.101 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.101 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:18.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.260325+0000", 2024-09-18T07:44:18.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001337771 2024-09-18T07:44:18.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:18.102 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.260441+0000", 2024-09-18T07:44:18.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000116617 2024-09-18T07:44:18.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:44:18.102 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.260569+0000", 2024-09-18T07:44:18.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012768799999999999 2024-09-18T07:44:18.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.103 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.103 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:44:18.103 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.262351+0000", 2024-09-18T07:44:18.103 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0017825149999999999 2024-09-18T07:44:18.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.103 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.103 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.103 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.263213+0000", 2024-09-18T07:44:18.103 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00086200999999999997 2024-09-18T07:44:18.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.103 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.103 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:18.103 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401638+0000", 2024-09-18T07:44:18.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13842480400000001 2024-09-18T07:44:18.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.104 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:44:18.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401667+0000", 2024-09-18T07:44:18.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8462999999999998e-05 2024-09-18T07:44:18.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:18.104 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:18.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:44:18.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401689+0000", 2024-09-18T07:44:18.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2027000000000001e-05 2024-09-18T07:44:18.104 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.105 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:18.105 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.105 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:18.105 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:18.105 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:44:18.105 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 68 16 109688 109672 100.769 72.875 0.0110421 0.00991826 2024-09-18T07:44:18.111 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:18.162 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:18.192 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:18.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:18.294 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:18.321 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:18.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:18.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:18.423 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:18.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:18.447 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:18.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:18.548 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:18.576 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:18.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:18.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:18.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:18.714 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:18.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:18.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:18.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:18.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:18.857 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:18.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:18.958 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:18.979 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:18.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:18.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:19.074 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 69 16 110687 110671 100.214 62.4375 0.00899342 0.00997351 2024-09-18T07:44:19.081 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:19.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:19.105 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:19.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:19.143 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:44:19.206 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:19.237 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:19.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:19.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:19.338 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:19.375 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:19.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:19.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:19 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:19.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:19 smithi137 ceph-mon[29408]: pgmap v699: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 53653/323031 objects degraded (16.609%) 2024-09-18T07:44:19.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:19 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/926035091' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:44:19.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:19 smithi137 ceph-mon[29408]: 6.1c scrub starts 2024-09-18T07:44:19.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:19 smithi137 ceph-mon[29408]: 6.1c scrub ok 2024-09-18T07:44:19.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:19 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:19.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:19 smithi137 ceph-mon[39175]: pgmap v699: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 53653/323031 objects degraded (16.609%) 2024-09-18T07:44:19.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:19 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/926035091' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:44:19.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:19 smithi137 ceph-mon[39175]: 6.1c scrub starts 2024-09-18T07:44:19.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:19 smithi137 ceph-mon[39175]: 6.1c scrub ok 2024-09-18T07:44:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:19 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:19 smithi203 ceph-mon[32237]: pgmap v699: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 53653/323031 objects degraded (16.609%) 2024-09-18T07:44:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:19 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/926035091' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:44:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:19 smithi203 ceph-mon[32237]: 6.1c scrub starts 2024-09-18T07:44:19.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:19 smithi203 ceph-mon[32237]: 6.1c scrub ok 2024-09-18T07:44:19.477 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:19.506 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:19.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:19.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:19.608 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:19.634 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:19.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:19.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:19.774 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:19.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:19.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:19.876 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:19.898 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:19.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:19.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:20.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:20.042 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:20.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:20.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:20.070 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:44:20.072 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:44:20.074 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 70 16 111678 111662 99.6669 61.9375 0.0181241 0.0100282 2024-09-18T07:44:20.145 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:20.173 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:20.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:20.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:20.275 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:20.305 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:20.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:20.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:20.393 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:44:20.393 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:44:20.393 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:44:20.393 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:44:20.394 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:44:20.394 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:44:20.394 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:44:20.394 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:44:20.394 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:44:20.394 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-18T07:44:20.394 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:44:20.395 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:44:20.395 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:44:20.395 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:44:20.395 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:44:20.396 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:20 smithi137 ceph-mon[29408]: from='client.16254 v1:172.21.15.137:0/2959591854' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:44:20.396 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:20 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:44:20.396 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:20 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/926035091' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:44:20.396 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:20 smithi137 ceph-mon[29408]: osdmap e384: 8 total, 7 up, 8 in 2024-09-18T07:44:20.396 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:20 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/448894330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:20.396 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:20 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 53653/323031 objects degraded (16.609%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:20.397 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:20 smithi137 ceph-mon[29408]: osdmap e385: 8 total, 7 up, 8 in 2024-09-18T07:44:20.397 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:20 smithi137 ceph-mon[39175]: from='client.16254 v1:172.21.15.137:0/2959591854' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:44:20.397 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:20 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:44:20.397 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:20 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/926035091' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:44:20.397 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:20 smithi137 ceph-mon[39175]: osdmap e384: 8 total, 7 up, 8 in 2024-09-18T07:44:20.398 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:20 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/448894330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:20.398 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:20 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 53653/323031 objects degraded (16.609%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:20.398 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:20 smithi137 ceph-mon[39175]: osdmap e385: 8 total, 7 up, 8 in 2024-09-18T07:44:20.406 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:20.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:20.452 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:20.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:20 smithi203 ceph-mon[32237]: from='client.16254 v1:172.21.15.137:0/2959591854' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:44:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:20 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:44:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:20 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/926035091' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:44:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:20 smithi203 ceph-mon[32237]: osdmap e384: 8 total, 7 up, 8 in 2024-09-18T07:44:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:20 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/448894330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:20 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 53653/323031 objects degraded (16.609%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:20 smithi203 ceph-mon[32237]: osdmap e385: 8 total, 7 up, 8 in 2024-09-18T07:44:20.554 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:20.591 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:20.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:20.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:20.693 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:20.720 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:20.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:20.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:20.821 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:20.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:20.858 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:20.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:20.960 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:20.992 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:20.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:21.074 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 71 16 112983 112967 99.4119 81.5625 0.018661 0.010055 2024-09-18T07:44:21.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:21.127 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:21.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:21.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:21.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:21.274 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":699,"stamp":"2024-09-18T07:44:17.731299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7044328385,"num_objects":107677,"num_object_clones":0,"num_object_copies":323031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":53653,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107677,"num_whiteouts":0,"num_read":1694,"num_read_kb":1456,"num_write":108210,"num_write_kb":6878785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":112965,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87331,"ondisk_log_size":87331,"up":405,"acting":405,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":469,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21070588,"kb_used_data":17094840,"kb_used_omap":12,"kb_used_meta":3975539,"kb_avail":728890628,"statfs":{"total":767960285184,"available":746384003072,"internally_reserved":0,"allocated":17505116160,"data_stored":17501587592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":4070952496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":677511168,"num_objects":10338,"num_object_clones":0,"num_object_copies":31014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10338,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10338,"num_write_kb":661632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003762"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":688,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.039961+0000","last_change":"2024-09-18T07:41:15.846355+0000","last_active":"2024-09-18T07:44:10.039961+0000","last_peered":"2024-09-18T07:44:10.039961+0000","last_clean":"2024-09-18T07:44:10.039961+0000","last_became_active":"2024-09-18T07:30:56.282146+0000","last_became_peered":"2024-09-18T07:30:56.282146+0000","last_unstale":"2024-09-18T07:44:10.039961+0000","last_undegraded":"2024-09-18T07:44:10.039961+0000","last_fullsized":"2024-09-18T07:44:10.039961+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:15.846257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:15.846257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:13.262548+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"323'15","reported_seq":697,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.108876+0000","last_change":"2024-09-18T07:42:54.748140+0000","last_active":"2024-09-18T07:44:14.108876+0000","last_peered":"2024-09-18T07:44:14.108876+0000","last_clean":"2024-09-18T07:41:04.168295+0000","last_became_active":"2024-09-18T07:42:54.748140+0000","last_became_peered":"2024-09-18T07:42:54.748140+0000","last_unstale":"2024-09-18T07:44:14.108876+0000","last_undegraded":"2024-09-18T07:42:54.735181+0000","last_fullsized":"2024-09-18T07:42:54.735072+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:17.269977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:17.269977+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:57.188354+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":641,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.115139+0000","last_change":"2024-09-18T07:44:11.422936+0000","last_active":"2024-09-18T07:44:14.115139+0000","last_peered":"2024-09-18T07:44:14.115139+0000","last_clean":"2024-09-18T07:44:14.115139+0000","last_became_active":"2024-09-18T07:36:13.128805+0000","last_became_peered":"2024-09-18T07:36:13.128805+0000","last_unstale":"2024-09-18T07:44:14.115139+0000","last_undegraded":"2024-09-18T07:44:14.115139+0000","last_fullsized":"2024-09-18T07:44:14.115139+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:11.422892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:44:11.422892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:36.022176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":57,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.114966+0000","last_change":"2024-09-18T07:42:54.740416+0000","last_active":"2024-09-18T07:44:14.114966+0000","last_peered":"2024-09-18T07:44:14.114966+0000","last_clean":"2024-09-18T07:31:01.319795+0000","last_became_active":"2024-09-18T07:42:54.740416+0000","last_became_peered":"2024-09-18T07:42:54.740416+0000","last_unstale":"2024-09-18T07:44:14.114966+0000","last_undegraded":"2024-09-18T07:44:14.114966+0000","last_fullsized":"2024-09-18T07:42:54.733242+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:47.286927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.286927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:29.659488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":582,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.037094+0000","last_change":"2024-09-18T07:41:12.872510+0000","last_active":"2024-09-18T07:44:10.037094+0000","last_peered":"2024-09-18T07:44:10.037094+0000","last_clean":"2024-09-18T07:44:10.037094+0000","last_became_active":"2024-09-18T07:37:43.056380+0000","last_became_peered":"2024-09-18T07:37:43.056380+0000","last_unstale":"2024-09-18T07:44:10.037094+0000","last_undegraded":"2024-09-18T07:44:10.037094+0000","last_fullsized":"2024-09-18T07:44:10.037094+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:12.872387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:12.872387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:14.272613+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":691,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.114650+0000","last_change":"2024-09-18T07:40:57.319270+0000","last_active":"2024-09-18T07:44:14.114650+0000","last_peered":"2024-09-18T07:44:14.114650+0000","last_clean":"2024-09-18T07:44:14.114650+0000","last_became_active":"2024-09-18T07:37:43.066508+0000","last_became_peered":"2024-09-18T07:37:43.066508+0000","last_unstale":"2024-09-18T07:44:14.114650+0000","last_undegraded":"2024-09-18T07:44:14.114650+0000","last_fullsized":"2024-09-18T07:44:14.114650+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:57.319204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:57.319204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:25.291687+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"323'27","reported_seq":243,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.116155+0000","last_change":"2024-09-18T07:42:46.881571+0000","last_active":"2024-09-18T07:44:14.116155+0000","last_peered":"2024-09-18T07:44:14.116155+0000","last_clean":"2024-09-18T07:44:14.116155+0000","last_became_active":"2024-09-18T07:40:19.064755+0000","last_became_peered":"2024-09-18T07:40:19.064755+0000","last_unstale":"2024-09-18T07:44:14.116155+0000","last_undegraded":"2024-09-18T07:44:14.116155+0000","last_fullsized":"2024-09-18T07:44:14.116155+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:42:46.881480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:46.881480+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":437,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.109079+0000","last_change":"2024-09-18T07:42:54.739217+0000","last_active":"2024-09-18T07:44:14.109079+0000","last_peered":"2024-09-18T07:44:14.109079+0000","last_clean":"2024-09-18T07:37:38.986586+0000","last_became_active":"2024-09-18T07:42:54.739217+0000","last_became_peered":"2024-09-18T07:42:54.739217+0000","last_unstale":"2024-09-18T07:44:14.109079+0000","last_undegraded":"2024-09-18T07:44:14.109079+0000","last_fullsized":"2024-09-18T07:42:54.734594+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:51.001046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:51.001046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:25.147698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":701,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.115784+0000","last_change":"2024-09-18T07:42:54.744099+0000","last_active":"2024-09-18T07:44:14.115784+0000","last_peered":"2024-09-18T07:44:14.115784+0000","last_clean":"2024-09-18T07:42:51.721624+0000","last_became_active":"2024-09-18T07:42:54.744099+0000","last_became_peered":"2024-09-18T07:42:54.744099+0000","last_unstale":"2024-09-18T07:44:14.115784+0000","last_undegraded":"2024-09-18T07:44:14.115784+0000","last_fullsized":"2024-09-18T07:42:54.735114+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:41.502137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:41.502137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"323'15","reported_seq":629,"reported_epoch":382,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:10.042158+0000","last_change":"2024-09-18T07:42:54.746959+0000","last_active":"2024-09-18T07:44:10.042158+0000","last_peered":"2024-09-18T07:44:10.042158+0000","last_clean":"2024-09-18T07:41:04.798944+0000","last_became_active":"2024-09-18T07:42:54.746959+0000","last_became_peered":"2024-09-18T07:42:54.746959+0000","last_unstale":"2024-09-18T07:44:10.042158+0000","last_undegraded":"2024-09-18T07:42:54.737030+0000","last_fullsized":"2024-09-18T07:42:54.736921+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:56.021118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:56.021118+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:32.256961+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":697,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.115215+0000","last_change":"2024-09-18T07:44:12.437170+0000","last_active":"2024-09-18T07:44:14.115215+0000","last_peered":"2024-09-18T07:44:14.115215+0000","last_clean":"2024-09-18T07:44:14.115215+0000","last_became_active":"2024-09-18T07:35:01.413443+0000","last_became_peered":"2024-09-18T07:35:01.413443+0000","last_unstale":"2024-09-18T07:44:14.115215+0000","last_undegraded":"2024-09-18T07:44:14.115215+0000","last_fullsized":"2024-09-18T07:44:14.115215+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:12.437116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:44:12.437116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:24.782049+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":681,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.044242+0000","last_change":"2024-09-18T07:43:37.233745+0000","last_active":"2024-09-18T07:44:10.044242+0000","last_peered":"2024-09-18T07:44:10.044242+0000","last_clean":"2024-09-18T07:44:10.044242+0000","last_became_active":"2024-09-18T07:35:01.410284+0000","last_became_peered":"2024-09-18T07:35:01.410284+0000","last_unstale":"2024-09-18T07:44:10.044242+0000","last_undegraded":"2024-09-18T07:44:10.044242+0000","last_fullsized":"2024-09-18T07:44:10.044242+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:37.233589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:37.233589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.694863+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":679,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.115250+0000","last_change":"2024-09-18T07:42:54.745050+0000","last_active":"2024-09-18T07:44:14.115250+0000","last_peered":"2024-09-18T07:44:14.115250+0000","last_clean":"2024-09-18T07:42:51.721032+0000","last_became_active":"2024-09-18T07:42:54.745050+0000","last_became_peered":"2024-09-18T07:42:54.745050+0000","last_unstale":"2024-09-18T07:44:14.115250+0000","last_undegraded":"2024-09-18T07:44:14.115250+0000","last_fullsized":"2024-09-18T07:42:54.733513+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:57.358853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:57.358853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:38.529191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"323'7","reported_seq":632,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.115047+0000","last_change":"2024-09-18T07:42:54.741699+0000","last_active":"2024-09-18T07:44:14.115047+0000","last_peered":"2024-09-18T07:44:14.115047+0000","last_clean":"2024-09-18T07:41:04.680256+0000","last_became_active":"2024-09-18T07:42:54.741699+0000","last_became_peered":"2024-09-18T07:42:54.741699+0000","last_unstale":"2024-09-18T07:44:14.115047+0000","last_undegraded":"2024-09-18T07:42:54.737038+0000","last_fullsized":"2024-09-18T07:42:54.736846+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'7","last_scrub_stamp":"2024-09-18T07:42:46.496916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:46.496916+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":2}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":460,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.119661+0000","last_change":"2024-09-18T07:42:54.741021+0000","last_active":"2024-09-18T07:44:14.119661+0000","last_peered":"2024-09-18T07:44:14.119661+0000","last_clean":"2024-09-18T07:42:51.727903+0000","last_became_active":"2024-09-18T07:42:54.741021+0000","last_became_peered":"2024-09-18T07:42:54.741021+0000","last_unstale":"2024-09-18T07:44:14.119661+0000","last_undegraded":"2024-09-18T07:44:14.119661+0000","last_fullsized":"2024-09-18T07:42:54.738593+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:28.855481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:28.855481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:45.250315+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":695,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.115238+0000","last_change":"2024-09-18T07:43:01.522712+0000","last_active":"2024-09-18T07:44:14.115238+0000","last_peered":"2024-09-18T07:44:14.115238+0000","last_clean":"2024-09-18T07:44:14.115238+0000","last_became_active":"2024-09-18T07:31:02.349716+0000","last_became_peered":"2024-09-18T07:31:02.349716+0000","last_unstale":"2024-09-18T07:44:14.115238+0000","last_undegraded":"2024-09-18T07:44:14.115238+0000","last_fullsized":"2024-09-18T07:44:14.115238+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.522665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.522665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:20.744014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":434,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.109355+0000","last_change":"2024-09-18T07:42:54.743937+0000","last_active":"2024-09-18T07:44:14.109355+0000","last_peered":"2024-09-18T07:44:14.109355+0000","last_clean":"2024-09-18T07:37:38.987411+0000","last_became_active":"2024-09-18T07:42:54.743937+0000","last_became_peered":"2024-09-18T07:42:54.743937+0000","last_unstale":"2024-09-18T07:44:14.109355+0000","last_undegraded":"2024-09-18T07:44:14.109355+0000","last_fullsized":"2024-09-18T07:42:54.735028+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:40.432906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:40.432906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":697,"reported_epoch":382,"state":"active+undersized","last_fresh":"2024-09-18T07:44:10.038327+0000","last_change":"2024-09-18T07:42:54.742535+0000","last_active":"2024-09-18T07:44:10.038327+0000","last_peered":"2024-09-18T07:44:10.038327+0000","last_clean":"2024-09-18T07:42:51.718566+0000","last_became_active":"2024-09-18T07:42:54.742535+0000","last_became_peered":"2024-09-18T07:42:54.742535+0000","last_unstale":"2024-09-18T07:44:10.038327+0000","last_undegraded":"2024-09-18T07:44:10.038327+0000","last_fullsized":"2024-09-18T07:42:54.740869+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:04.117653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.117653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:27.227076+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"323'27","reported_seq":786,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:16.167864+0000","last_change":"2024-09-18T07:44:16.167864+0000","last_active":"2024-09-18T07:44:16.167864+0000","last_peered":"2024-09-18T07:44:16.167864+0000","last_clean":"2024-09-18T07:44:16.167864+0000","last_became_active":"2024-09-18T07:37:28.002858+0000","last_became_peered":"2024-09-18T07:37:28.002858+0000","last_unstale":"2024-09-18T07:44:16.167864+0000","last_undegraded":"2024-09-18T07:44:16.167864+0000","last_fullsized":"2024-09-18T07:44:16.167864+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:44:16.167784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:16.167784+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:44.200763+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":56,"reported_epoch":382,"state":"active+undersized","last_fresh":"2024-09-18T07:44:10.044587+0000","last_change":"2024-09-18T07:42:54.751231+0000","last_active":"2024-09-18T07:44:10.044587+0000","last_peered":"2024-09-18T07:44:10.044587+0000","last_clean":"2024-09-18T07:30:59.305166+0000","last_became_active":"2024-09-18T07:42:54.751231+0000","last_became_peered":"2024-09-18T07:42:54.751231+0000","last_unstale":"2024-09-18T07:44:10.044587+0000","last_undegraded":"2024-09-18T07:44:10.044587+0000","last_fullsized":"2024-09-18T07:42:54.737194+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:26.922506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:26.922506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:21.316063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":549,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.109599+0000","last_change":"2024-09-18T07:42:57.174875+0000","last_active":"2024-09-18T07:44:14.109599+0000","last_peered":"2024-09-18T07:44:14.109599+0000","last_clean":"2024-09-18T07:44:14.109599+0000","last_became_active":"2024-09-18T07:40:19.070463+0000","last_became_peered":"2024-09-18T07:40:19.070463+0000","last_unstale":"2024-09-18T07:44:14.109599+0000","last_undegraded":"2024-09-18T07:44:14.109599+0000","last_fullsized":"2024-09-18T07:44:14.109599+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:57.174827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:57.174827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:23.943515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":56,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.115496+0000","last_change":"2024-09-18T07:42:54.751631+0000","last_active":"2024-09-18T07:44:14.115496+0000","last_peered":"2024-09-18T07:44:14.115496+0000","last_clean":"2024-09-18T07:30:55.271698+0000","last_became_active":"2024-09-18T07:42:54.751631+0000","last_became_peered":"2024-09-18T07:42:54.751631+0000","last_unstale":"2024-09-18T07:44:14.115496+0000","last_undegraded":"2024-09-18T07:44:14.115496+0000","last_fullsized":"2024-09-18T07:42:54.743356+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:31.323963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:31.323963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:41.096756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"323'18","reported_seq":351,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.134450+0000","last_change":"2024-09-18T07:42:54.742187+0000","last_active":"2024-09-18T07:44:14.134450+0000","last_peered":"2024-09-18T07:44:14.134450+0000","last_clean":"2024-09-18T07:42:51.717553+0000","last_became_active":"2024-09-18T07:42:54.742187+0000","last_became_peered":"2024-09-18T07:42:54.742187+0000","last_unstale":"2024-09-18T07:44:14.134450+0000","last_undegraded":"2024-09-18T07:42:54.738159+0000","last_fullsized":"2024-09-18T07:42:54.735312+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:41:59.149629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:59.149629+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:07.184759+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":56,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.123988+0000","last_change":"2024-09-18T07:42:54.753848+0000","last_active":"2024-09-18T07:44:14.123988+0000","last_peered":"2024-09-18T07:44:14.123988+0000","last_clean":"2024-09-18T07:30:59.305166+0000","last_became_active":"2024-09-18T07:42:54.753848+0000","last_became_peered":"2024-09-18T07:42:54.753848+0000","last_unstale":"2024-09-18T07:44:14.123988+0000","last_undegraded":"2024-09-18T07:44:14.123988+0000","last_fullsized":"2024-09-18T07:42:54.743099+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.930435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.930435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:03.716914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":460,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.119926+0000","last_change":"2024-09-18T07:40:10.644587+0000","last_active":"2024-09-18T07:44:14.119926+0000","last_peered":"2024-09-18T07:44:14.119926+0000","last_clean":"2024-09-18T07:44:14.119926+0000","last_became_active":"2024-09-18T07:35:01.413110+0000","last_became_peered":"2024-09-18T07:35:01.413110+0000","last_unstale":"2024-09-18T07:44:14.119926+0000","last_undegraded":"2024-09-18T07:44:14.119926+0000","last_fullsized":"2024-09-18T07:44:14.119926+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:40:10.644461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:10.644461+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:16.109693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":389,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.114213+0000","last_change":"2024-09-18T07:42:54.753800+0000","last_active":"2024-09-18T07:44:14.114213+0000","last_peered":"2024-09-18T07:44:14.114213+0000","last_clean":"2024-09-18T07:42:51.723949+0000","last_became_active":"2024-09-18T07:42:54.753800+0000","last_became_peered":"2024-09-18T07:42:54.753800+0000","last_unstale":"2024-09-18T07:44:14.114213+0000","last_undegraded":"2024-09-18T07:42:54.745541+0000","last_fullsized":"2024-09-18T07:42:54.745455+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:42:42.332950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:42.332950+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:08.851607+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"323'20","reported_seq":760,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.042785+0000","last_change":"2024-09-18T07:43:46.102137+0000","last_active":"2024-09-18T07:44:10.042785+0000","last_peered":"2024-09-18T07:44:10.042785+0000","last_clean":"2024-09-18T07:44:10.042785+0000","last_became_active":"2024-09-18T07:35:01.410926+0000","last_became_peered":"2024-09-18T07:35:01.410926+0000","last_unstale":"2024-09-18T07:44:10.042785+0000","last_undegraded":"2024-09-18T07:44:10.042785+0000","last_fullsized":"2024-09-18T07:44:10.042785+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:43:46.102041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:43:46.102041+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:13.396740+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"360'13","reported_seq":404,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.109787+0000","last_change":"2024-09-18T07:42:54.745765+0000","last_active":"2024-09-18T07:44:14.109787+0000","last_peered":"2024-09-18T07:44:14.109787+0000","last_clean":"2024-09-18T07:42:51.730678+0000","last_became_active":"2024-09-18T07:42:54.745765+0000","last_became_peered":"2024-09-18T07:42:54.745765+0000","last_unstale":"2024-09-18T07:44:14.109787+0000","last_undegraded":"2024-09-18T07:42:54.736184+0000","last_fullsized":"2024-09-18T07:42:54.736047+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:40:43.112459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:43.112459+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:15.288005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"383'6633","reported_seq":6679,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:16.675142+0000","last_change":"2024-09-18T07:43:10.058721+0000","last_active":"2024-09-18T07:44:16.675142+0000","last_peered":"2024-09-18T07:44:16.675142+0000","last_clean":"2024-09-18T07:43:01.032791+0000","last_became_active":"2024-09-18T07:43:02.045039+0000","last_became_peered":"2024-09-18T07:43:02.045039+0000","last_unstale":"2024-09-18T07:44:16.675142+0000","last_undegraded":"2024-09-18T07:43:10.058195+0000","last_fullsized":"2024-09-18T07:43:02.038669+0000","mapping_epoch":364,"log_start":"370'1800","ondisk_log_start":"370'1800","created":364,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":4833,"log_dups_size":0,"ondisk_log_size":4833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434700288,"num_objects":6633,"num_object_clones":0,"num_object_copies":19899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6633,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6633,"num_write_kb":424512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":6633}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":262,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.410067+0000","last_change":"2024-09-18T07:44:14.410067+0000","last_active":"2024-09-18T07:44:14.410067+0000","last_peered":"2024-09-18T07:44:14.410067+0000","last_clean":"2024-09-18T07:44:14.410067+0000","last_became_active":"2024-09-18T07:39:04.962047+0000","last_became_peered":"2024-09-18T07:39:04.962047+0000","last_unstale":"2024-09-18T07:44:14.410067+0000","last_undegraded":"2024-09-18T07:44:14.410067+0000","last_fullsized":"2024-09-18T07:44:14.410067+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:44:14.410019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:44:14.410019+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:20.374757+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"323'27","reported_seq":443,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.123709+0000","last_change":"2024-09-18T07:42:54.746764+0000","last_active":"2024-09-18T07:44:14.123709+0000","last_peered":"2024-09-18T07:44:14.123709+0000","last_clean":"2024-09-18T07:42:51.730768+0000","last_became_active":"2024-09-18T07:42:54.746764+0000","last_became_peered":"2024-09-18T07:42:54.746764+0000","last_unstale":"2024-09-18T07:44:14.123709+0000","last_undegraded":"2024-09-18T07:42:54.742352+0000","last_fullsized":"2024-09-18T07:42:54.742252+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:42:33.391715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:33.391715+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:09.815601+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":707,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.114474+0000","last_change":"2024-09-18T07:39:29.049261+0000","last_active":"2024-09-18T07:44:14.114474+0000","last_peered":"2024-09-18T07:44:14.114474+0000","last_clean":"2024-09-18T07:44:14.114474+0000","last_became_active":"2024-09-18T07:35:01.407339+0000","last_became_peered":"2024-09-18T07:35:01.407339+0000","last_unstale":"2024-09-18T07:44:14.114474+0000","last_undegraded":"2024-09-18T07:44:14.114474+0000","last_fullsized":"2024-09-18T07:44:14.114474+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.049209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.049209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:23.347291+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"360'9","reported_seq":100,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.116088+0000","last_change":"2024-09-18T07:42:54.740567+0000","last_active":"2024-09-18T07:44:14.116088+0000","last_peered":"2024-09-18T07:44:14.116088+0000","last_clean":"2024-09-18T07:31:03.519590+0000","last_became_active":"2024-09-18T07:42:54.740567+0000","last_became_peered":"2024-09-18T07:42:54.740567+0000","last_unstale":"2024-09-18T07:44:14.116088+0000","last_undegraded":"2024-09-18T07:42:54.734181+0000","last_fullsized":"2024-09-18T07:42:54.734092+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:41:27.965205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.965205+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:35.595508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":671,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.115124+0000","last_change":"2024-09-18T07:42:06.507229+0000","last_active":"2024-09-18T07:44:14.115124+0000","last_peered":"2024-09-18T07:44:14.115124+0000","last_clean":"2024-09-18T07:44:14.115124+0000","last_became_active":"2024-09-18T07:31:02.351058+0000","last_became_peered":"2024-09-18T07:31:02.351058+0000","last_unstale":"2024-09-18T07:44:14.115124+0000","last_undegraded":"2024-09-18T07:44:14.115124+0000","last_fullsized":"2024-09-18T07:44:14.115124+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.507166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.507166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:59.465213+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"383'6798","reported_seq":6844,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:16.803389+0000","last_change":"2024-09-18T07:43:10.060552+0000","last_active":"2024-09-18T07:44:16.803389+0000","last_peered":"2024-09-18T07:44:16.803389+0000","last_clean":"2024-09-18T07:43:01.032791+0000","last_became_active":"2024-09-18T07:43:02.046628+0000","last_became_peered":"2024-09-18T07:43:02.046628+0000","last_unstale":"2024-09-18T07:44:16.803389+0000","last_undegraded":"2024-09-18T07:43:10.053665+0000","last_fullsized":"2024-09-18T07:43:02.043071+0000","mapping_epoch":364,"log_start":"368'1200","ondisk_log_start":"368'1200","created":364,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5598,"log_dups_size":0,"ondisk_log_size":5598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:27.025706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445513728,"num_objects":6798,"num_object_clones":0,"num_object_copies":20394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6798,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6798,"num_write_kb":435072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":6798}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":57,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.133774+0000","last_change":"2024-09-18T07:42:54.748936+0000","last_active":"2024-09-18T07:44:14.133774+0000","last_peered":"2024-09-18T07:44:14.133774+0000","last_clean":"2024-09-18T07:34:59.364705+0000","last_became_active":"2024-09-18T07:42:54.748936+0000","last_became_peered":"2024-09-18T07:42:54.748936+0000","last_unstale":"2024-09-18T07:44:14.133774+0000","last_undegraded":"2024-09-18T07:44:14.133774+0000","last_fullsized":"2024-09-18T07:42:54.738388+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:41:53.004186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:41:53.004186+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-18T07:44:55.094617+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"323'30","reported_seq":789,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.113607+0000","last_change":"2024-09-18T07:44:02.716229+0000","last_active":"2024-09-18T07:44:14.113607+0000","last_peered":"2024-09-18T07:44:14.113607+0000","last_clean":"2024-09-18T07:44:14.113607+0000","last_became_active":"2024-09-18T07:35:01.418909+0000","last_became_peered":"2024-09-18T07:35:01.418909+0000","last_unstale":"2024-09-18T07:44:14.113607+0000","last_undegraded":"2024-09-18T07:44:14.113607+0000","last_fullsized":"2024-09-18T07:44:14.113607+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'30","last_scrub_stamp":"2024-09-18T07:44:02.716145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:02.716145+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:10.082570+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":419,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.120521+0000","last_change":"2024-09-18T07:42:39.814649+0000","last_active":"2024-09-18T07:44:14.120521+0000","last_peered":"2024-09-18T07:44:14.120521+0000","last_clean":"2024-09-18T07:44:14.120521+0000","last_became_active":"2024-09-18T07:40:19.068296+0000","last_became_peered":"2024-09-18T07:40:19.068296+0000","last_unstale":"2024-09-18T07:44:14.120521+0000","last_undegraded":"2024-09-18T07:44:14.120521+0000","last_fullsized":"2024-09-18T07:44:14.120521+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:39.814582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:39.814582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":297,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.113953+0000","last_change":"2024-09-18T07:42:54.753453+0000","last_active":"2024-09-18T07:44:14.113953+0000","last_peered":"2024-09-18T07:44:14.113953+0000","last_clean":"2024-09-18T07:34:59.364667+0000","last_became_active":"2024-09-18T07:42:54.753453+0000","last_became_peered":"2024-09-18T07:42:54.753453+0000","last_unstale":"2024-09-18T07:44:14.113953+0000","last_undegraded":"2024-09-18T07:44:14.113953+0000","last_fullsized":"2024-09-18T07:42:54.746156+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:09.067941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:09.067941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:31.334423+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":57,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.113578+0000","last_change":"2024-09-18T07:42:54.755423+0000","last_active":"2024-09-18T07:44:14.113578+0000","last_peered":"2024-09-18T07:44:14.113578+0000","last_clean":"2024-09-18T07:31:01.319795+0000","last_became_active":"2024-09-18T07:42:54.755423+0000","last_became_peered":"2024-09-18T07:42:54.755423+0000","last_unstale":"2024-09-18T07:44:14.113578+0000","last_undegraded":"2024-09-18T07:44:14.113578+0000","last_fullsized":"2024-09-18T07:42:54.740006+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:44.219547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:44.219547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:06.807038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"383'6674","reported_seq":6719,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:16.800644+0000","last_change":"2024-09-18T07:43:02.069313+0000","last_active":"2024-09-18T07:44:16.800644+0000","last_peered":"2024-09-18T07:44:16.800644+0000","last_clean":"2024-09-18T07:44:16.800644+0000","last_became_active":"2024-09-18T07:43:02.068957+0000","last_became_peered":"2024-09-18T07:43:02.068957+0000","last_unstale":"2024-09-18T07:44:16.800644+0000","last_undegraded":"2024-09-18T07:44:16.800644+0000","last_fullsized":"2024-09-18T07:44:16.800644+0000","mapping_epoch":364,"log_start":"368'1100","ondisk_log_start":"368'1100","created":364,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5574,"log_dups_size":0,"ondisk_log_size":5574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437387264,"num_objects":6674,"num_object_clones":0,"num_object_copies":20022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6674,"num_write_kb":427136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":251,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.123614+0000","last_change":"2024-09-18T07:41:19.479946+0000","last_active":"2024-09-18T07:44:14.123614+0000","last_peered":"2024-09-18T07:44:14.123614+0000","last_clean":"2024-09-18T07:44:14.123614+0000","last_became_active":"2024-09-18T07:39:04.945385+0000","last_became_peered":"2024-09-18T07:39:04.945385+0000","last_unstale":"2024-09-18T07:44:14.123614+0000","last_undegraded":"2024-09-18T07:44:14.123614+0000","last_fullsized":"2024-09-18T07:44:14.123614+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:41:19.479442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:41:19.479442+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:18.360628+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"323'28","reported_seq":694,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.115320+0000","last_change":"2024-09-18T07:42:54.739669+0000","last_active":"2024-09-18T07:44:14.115320+0000","last_peered":"2024-09-18T07:44:14.115320+0000","last_clean":"2024-09-18T07:41:05.289656+0000","last_became_active":"2024-09-18T07:42:54.739669+0000","last_became_peered":"2024-09-18T07:42:54.739669+0000","last_unstale":"2024-09-18T07:44:14.115320+0000","last_undegraded":"2024-09-18T07:42:54.733414+0000","last_fullsized":"2024-09-18T07:42:54.733291+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-18T07:40:56.183802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.183802+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:16.528654+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":677,"reported_epoch":382,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:10.043661+0000","last_change":"2024-09-18T07:42:54.751066+0000","last_active":"2024-09-18T07:44:10.043661+0000","last_peered":"2024-09-18T07:44:10.043661+0000","last_clean":"2024-09-18T07:42:51.718806+0000","last_became_active":"2024-09-18T07:42:54.751066+0000","last_became_peered":"2024-09-18T07:42:54.751066+0000","last_unstale":"2024-09-18T07:44:10.043661+0000","last_undegraded":"2024-09-18T07:42:54.738145+0000","last_fullsized":"2024-09-18T07:42:54.737872+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:39:28.750720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:28.750720+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:58.160093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"360'9","reported_seq":851,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.114305+0000","last_change":"2024-09-18T07:42:54.746312+0000","last_active":"2024-09-18T07:44:14.114305+0000","last_peered":"2024-09-18T07:44:14.114305+0000","last_clean":"2024-09-18T07:42:51.726800+0000","last_became_active":"2024-09-18T07:42:54.746312+0000","last_became_peered":"2024-09-18T07:42:54.746312+0000","last_unstale":"2024-09-18T07:44:14.114305+0000","last_undegraded":"2024-09-18T07:42:54.742119+0000","last_fullsized":"2024-09-18T07:42:54.742013+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:42:04.428395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.428395+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:28.345161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":471,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.119598+0000","last_change":"2024-09-18T07:42:45.886132+0000","last_active":"2024-09-18T07:44:14.119598+0000","last_peered":"2024-09-18T07:44:14.119598+0000","last_clean":"2024-09-18T07:44:14.119598+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:44:14.119598+0000","last_undegraded":"2024-09-18T07:44:14.119598+0000","last_fullsized":"2024-09-18T07:44:14.119598+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:45.886069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:45.886069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"383'6775","reported_seq":6821,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:16.675093+0000","last_change":"2024-09-18T07:43:10.054942+0000","last_active":"2024-09-18T07:44:16.675093+0000","last_peered":"2024-09-18T07:44:16.675093+0000","last_clean":"2024-09-18T07:43:01.032791+0000","last_became_active":"2024-09-18T07:43:02.044295+0000","last_became_peered":"2024-09-18T07:43:02.044295+0000","last_unstale":"2024-09-18T07:44:16.675093+0000","last_undegraded":"2024-09-18T07:43:10.054785+0000","last_fullsized":"2024-09-18T07:43:02.039081+0000","mapping_epoch":364,"log_start":"370'2000","ondisk_log_start":"370'2000","created":364,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":4775,"log_dups_size":0,"ondisk_log_size":4775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:29.211647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444006400,"num_objects":6775,"num_object_clones":0,"num_object_copies":20325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6775,"num_write_kb":433600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":6775}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":413,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.109562+0000","last_change":"2024-09-18T07:40:38.066508+0000","last_active":"2024-09-18T07:44:14.109562+0000","last_peered":"2024-09-18T07:44:14.109562+0000","last_clean":"2024-09-18T07:44:14.109562+0000","last_became_active":"2024-09-18T07:39:04.941752+0000","last_became_peered":"2024-09-18T07:39:04.941752+0000","last_unstale":"2024-09-18T07:44:14.109562+0000","last_undegraded":"2024-09-18T07:44:14.109562+0000","last_fullsized":"2024-09-18T07:44:14.109562+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:38.066443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.066443+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:20.222095+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"323'20","reported_seq":430,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.116185+0000","last_change":"2024-09-18T07:42:54.742459+0000","last_active":"2024-09-18T07:44:14.116185+0000","last_peered":"2024-09-18T07:44:14.116185+0000","last_clean":"2024-09-18T07:42:51.721094+0000","last_became_active":"2024-09-18T07:42:54.742459+0000","last_became_peered":"2024-09-18T07:42:54.742459+0000","last_unstale":"2024-09-18T07:44:14.116185+0000","last_undegraded":"2024-09-18T07:42:54.733700+0000","last_fullsized":"2024-09-18T07:42:54.733580+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:42:03.427843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:03.427843+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:08.430124+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":692,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.115969+0000","last_change":"2024-09-18T07:40:53.350083+0000","last_active":"2024-09-18T07:44:14.115969+0000","last_peered":"2024-09-18T07:44:14.115969+0000","last_clean":"2024-09-18T07:44:14.115969+0000","last_became_active":"2024-09-18T07:30:56.288824+0000","last_became_peered":"2024-09-18T07:30:56.288824+0000","last_unstale":"2024-09-18T07:44:14.115969+0000","last_undegraded":"2024-09-18T07:44:14.115969+0000","last_fullsized":"2024-09-18T07:44:14.115969+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:53.350019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:53.350019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:19.601020+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"240'10","reported_seq":862,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.037516+0000","last_change":"2024-09-18T07:40:50.839710+0000","last_active":"2024-09-18T07:44:10.037516+0000","last_peered":"2024-09-18T07:44:10.037516+0000","last_clean":"2024-09-18T07:44:10.037516+0000","last_became_active":"2024-09-18T07:37:43.167850+0000","last_became_peered":"2024-09-18T07:37:43.167850+0000","last_unstale":"2024-09-18T07:44:10.037516+0000","last_undegraded":"2024-09-18T07:44:10.037516+0000","last_fullsized":"2024-09-18T07:44:10.037516+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'10","last_scrub_stamp":"2024-09-18T07:40:50.839504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.839504+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:24.293623+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":660,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.046416+0000","last_change":"2024-09-18T07:42:42.223288+0000","last_active":"2024-09-18T07:44:10.046416+0000","last_peered":"2024-09-18T07:44:10.046416+0000","last_clean":"2024-09-18T07:44:10.046416+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:44:10.046416+0000","last_undegraded":"2024-09-18T07:44:10.046416+0000","last_fullsized":"2024-09-18T07:44:10.046416+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:42.223235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:42.223235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"383'6763","reported_seq":6809,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:16.783103+0000","last_change":"2024-09-18T07:43:10.055395+0000","last_active":"2024-09-18T07:44:16.783103+0000","last_peered":"2024-09-18T07:44:16.783103+0000","last_clean":"2024-09-18T07:43:01.032791+0000","last_became_active":"2024-09-18T07:43:02.049117+0000","last_became_peered":"2024-09-18T07:43:02.049117+0000","last_unstale":"2024-09-18T07:44:16.783103+0000","last_undegraded":"2024-09-18T07:43:10.054743+0000","last_fullsized":"2024-09-18T07:43:02.044732+0000","mapping_epoch":364,"log_start":"368'1200","ondisk_log_start":"368'1200","created":364,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5563,"log_dups_size":0,"ondisk_log_size":5563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:28.071161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443219968,"num_objects":6763,"num_object_clones":0,"num_object_copies":20289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6763,"num_write_kb":432832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":6763}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":413,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.109194+0000","last_change":"2024-09-18T07:40:18.954153+0000","last_active":"2024-09-18T07:44:14.109194+0000","last_peered":"2024-09-18T07:44:14.109194+0000","last_clean":"2024-09-18T07:44:14.109194+0000","last_became_active":"2024-09-18T07:39:04.948571+0000","last_became_peered":"2024-09-18T07:39:04.948571+0000","last_unstale":"2024-09-18T07:44:14.109194+0000","last_undegraded":"2024-09-18T07:44:14.109194+0000","last_fullsized":"2024-09-18T07:44:14.109194+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:18.954089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:18.954089+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:53.139734+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"323'29","reported_seq":785,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.037764+0000","last_change":"2024-09-18T07:41:16.895792+0000","last_active":"2024-09-18T07:44:10.037764+0000","last_peered":"2024-09-18T07:44:10.037764+0000","last_clean":"2024-09-18T07:44:10.037764+0000","last_became_active":"2024-09-18T07:35:01.415275+0000","last_became_peered":"2024-09-18T07:35:01.415275+0000","last_unstale":"2024-09-18T07:44:10.037764+0000","last_undegraded":"2024-09-18T07:44:10.037764+0000","last_fullsized":"2024-09-18T07:44:10.037764+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'29","last_scrub_stamp":"2024-09-18T07:41:16.895595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:16.895595+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:12.237471+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":384,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.119746+0000","last_change":"2024-09-18T07:44:11.187033+0000","last_active":"2024-09-18T07:44:14.119746+0000","last_peered":"2024-09-18T07:44:14.119746+0000","last_clean":"2024-09-18T07:44:14.119746+0000","last_became_active":"2024-09-18T07:36:13.118553+0000","last_became_peered":"2024-09-18T07:36:13.118553+0000","last_unstale":"2024-09-18T07:44:14.119746+0000","last_undegraded":"2024-09-18T07:44:14.119746+0000","last_fullsized":"2024-09-18T07:44:14.119746+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:11.186957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:44:11.186957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:23.528711+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":324,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.109475+0000","last_change":"2024-09-18T07:43:11.108075+0000","last_active":"2024-09-18T07:44:14.109475+0000","last_peered":"2024-09-18T07:44:14.109475+0000","last_clean":"2024-09-18T07:44:14.109475+0000","last_became_active":"2024-09-18T07:37:43.070009+0000","last_became_peered":"2024-09-18T07:37:43.070009+0000","last_unstale":"2024-09-18T07:44:14.109475+0000","last_undegraded":"2024-09-18T07:44:14.109475+0000","last_fullsized":"2024-09-18T07:44:14.109475+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:11.108021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:43:11.108021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:31.946331+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":316,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.109083+0000","last_change":"2024-09-18T07:42:47.221767+0000","last_active":"2024-09-18T07:44:14.109083+0000","last_peered":"2024-09-18T07:44:14.109083+0000","last_clean":"2024-09-18T07:44:14.109083+0000","last_became_active":"2024-09-18T07:37:43.051740+0000","last_became_peered":"2024-09-18T07:37:43.051740+0000","last_unstale":"2024-09-18T07:44:14.109083+0000","last_undegraded":"2024-09-18T07:44:14.109083+0000","last_fullsized":"2024-09-18T07:44:14.109083+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:47.221690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:47.221690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:16.188899+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.a","version":"383'6718","reported_seq":6767,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:16.790425+0000","last_change":"2024-09-18T07:43:02.048403+0000","last_active":"2024-09-18T07:44:16.790425+0000","last_peered":"2024-09-18T07:44:16.790425+0000","last_clean":"2024-09-18T07:44:16.790425+0000","last_became_active":"2024-09-18T07:43:02.048124+0000","last_became_peered":"2024-09-18T07:43:02.048124+0000","last_unstale":"2024-09-18T07:44:16.790425+0000","last_undegraded":"2024-09-18T07:44:16.790425+0000","last_fullsized":"2024-09-18T07:44:16.790425+0000","mapping_epoch":364,"log_start":"368'1100","ondisk_log_start":"368'1100","created":364,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5618,"log_dups_size":0,"ondisk_log_size":5618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:29.293888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440270848,"num_objects":6718,"num_object_clones":0,"num_object_copies":20154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6718,"num_write_kb":429952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":246,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.114956+0000","last_change":"2024-09-18T07:40:43.346516+0000","last_active":"2024-09-18T07:44:14.114956+0000","last_peered":"2024-09-18T07:44:14.114956+0000","last_clean":"2024-09-18T07:44:14.114956+0000","last_became_active":"2024-09-18T07:39:04.950564+0000","last_became_peered":"2024-09-18T07:39:04.950564+0000","last_unstale":"2024-09-18T07:44:14.114956+0000","last_undegraded":"2024-09-18T07:44:14.114956+0000","last_fullsized":"2024-09-18T07:44:14.114956+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:43.346447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:43.346447+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:24.649295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"323'41","reported_seq":577,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.119600+0000","last_change":"2024-09-18T07:43:11.982067+0000","last_active":"2024-09-18T07:44:14.119600+0000","last_peered":"2024-09-18T07:44:14.119600+0000","last_clean":"2024-09-18T07:44:14.119600+0000","last_became_active":"2024-09-18T07:36:13.115278+0000","last_became_peered":"2024-09-18T07:36:13.115278+0000","last_unstale":"2024-09-18T07:44:14.119600+0000","last_undegraded":"2024-09-18T07:44:14.119600+0000","last_fullsized":"2024-09-18T07:44:14.119600+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'41","last_scrub_stamp":"2024-09-18T07:43:11.981958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:43:11.981958+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:34.511250+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":437,"reported_epoch":382,"state":"active+undersized","last_fresh":"2024-09-18T07:44:10.038118+0000","last_change":"2024-09-18T07:42:54.748899+0000","last_active":"2024-09-18T07:44:10.038118+0000","last_peered":"2024-09-18T07:44:10.038118+0000","last_clean":"2024-09-18T07:37:38.987203+0000","last_became_active":"2024-09-18T07:42:54.748899+0000","last_became_peered":"2024-09-18T07:42:54.748899+0000","last_unstale":"2024-09-18T07:44:10.038118+0000","last_undegraded":"2024-09-18T07:44:10.038118+0000","last_fullsized":"2024-09-18T07:42:54.740370+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:03.028558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:03.028558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:43.093313+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":724,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.115978+0000","last_change":"2024-09-18T07:44:06.489237+0000","last_active":"2024-09-18T07:44:14.115978+0000","last_peered":"2024-09-18T07:44:14.115978+0000","last_clean":"2024-09-18T07:44:14.115978+0000","last_became_active":"2024-09-18T07:37:43.076595+0000","last_became_peered":"2024-09-18T07:37:43.076595+0000","last_unstale":"2024-09-18T07:44:14.115978+0000","last_undegraded":"2024-09-18T07:44:14.115978+0000","last_fullsized":"2024-09-18T07:44:14.115978+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:44:06.489166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:44:06.489166+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:07.284402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":668,"reported_epoch":382,"state":"active+undersized","last_fresh":"2024-09-18T07:44:10.044792+0000","last_change":"2024-09-18T07:42:54.751453+0000","last_active":"2024-09-18T07:44:10.044792+0000","last_peered":"2024-09-18T07:44:10.044792+0000","last_clean":"2024-09-18T07:42:51.719583+0000","last_became_active":"2024-09-18T07:42:54.751453+0000","last_became_peered":"2024-09-18T07:42:54.751453+0000","last_unstale":"2024-09-18T07:44:10.044792+0000","last_undegraded":"2024-09-18T07:44:10.044792+0000","last_fullsized":"2024-09-18T07:42:54.737547+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:43.200013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:43.200013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:10.876228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":691,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.113996+0000","last_change":"2024-09-18T07:42:54.750128+0000","last_active":"2024-09-18T07:44:14.113996+0000","last_peered":"2024-09-18T07:44:14.113996+0000","last_clean":"2024-09-18T07:42:51.722974+0000","last_became_active":"2024-09-18T07:42:54.750128+0000","last_became_peered":"2024-09-18T07:42:54.750128+0000","last_unstale":"2024-09-18T07:44:14.113996+0000","last_undegraded":"2024-09-18T07:44:14.113996+0000","last_fullsized":"2024-09-18T07:42:54.741140+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:05.858804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:05.858804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:04.751998+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"383'6835","reported_seq":6884,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:17.113929+0000","last_change":"2024-09-18T07:43:02.050368+0000","last_active":"2024-09-18T07:44:17.113929+0000","last_peered":"2024-09-18T07:44:17.113929+0000","last_clean":"2024-09-18T07:44:17.113929+0000","last_became_active":"2024-09-18T07:43:02.049845+0000","last_became_peered":"2024-09-18T07:43:02.049845+0000","last_unstale":"2024-09-18T07:44:17.113929+0000","last_undegraded":"2024-09-18T07:44:17.113929+0000","last_fullsized":"2024-09-18T07:44:17.113929+0000","mapping_epoch":364,"log_start":"370'1800","ondisk_log_start":"370'1800","created":364,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447938560,"num_objects":6835,"num_object_clones":0,"num_object_copies":20505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6835,"num_write_kb":437440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":517,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.119313+0000","last_change":"2024-09-18T07:42:54.749373+0000","last_active":"2024-09-18T07:44:14.119313+0000","last_peered":"2024-09-18T07:44:14.119313+0000","last_clean":"2024-09-18T07:42:51.727693+0000","last_became_active":"2024-09-18T07:42:54.749373+0000","last_became_peered":"2024-09-18T07:42:54.749373+0000","last_unstale":"2024-09-18T07:44:14.119313+0000","last_undegraded":"2024-09-18T07:44:14.119313+0000","last_fullsized":"2024-09-18T07:42:54.738094+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:44.476278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:44.476278+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:03.493848+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"323'17","reported_seq":437,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.116995+0000","last_change":"2024-09-18T07:43:40.533349+0000","last_active":"2024-09-18T07:44:14.116995+0000","last_peered":"2024-09-18T07:44:14.116995+0000","last_clean":"2024-09-18T07:44:14.116995+0000","last_became_active":"2024-09-18T07:36:13.117924+0000","last_became_peered":"2024-09-18T07:36:13.117924+0000","last_unstale":"2024-09-18T07:44:14.116995+0000","last_undegraded":"2024-09-18T07:44:14.116995+0000","last_fullsized":"2024-09-18T07:44:14.116995+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'17","last_scrub_stamp":"2024-09-18T07:43:40.533240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:43:40.533240+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:48.046541+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":399,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.115437+0000","last_change":"2024-09-18T07:42:02.127909+0000","last_active":"2024-09-18T07:44:14.115437+0000","last_peered":"2024-09-18T07:44:14.115437+0000","last_clean":"2024-09-18T07:44:14.115437+0000","last_became_active":"2024-09-18T07:37:43.050971+0000","last_became_peered":"2024-09-18T07:37:43.050971+0000","last_unstale":"2024-09-18T07:44:14.115437+0000","last_undegraded":"2024-09-18T07:44:14.115437+0000","last_fullsized":"2024-09-18T07:44:14.115437+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:02.127517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:02.127517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:17.949081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":703,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.116241+0000","last_change":"2024-09-18T07:43:01.065282+0000","last_active":"2024-09-18T07:44:14.116241+0000","last_peered":"2024-09-18T07:44:14.116241+0000","last_clean":"2024-09-18T07:44:14.116241+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:44:14.116241+0000","last_undegraded":"2024-09-18T07:44:14.116241+0000","last_fullsized":"2024-09-18T07:44:14.116241+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:43:01.065193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.065193+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:20.807951+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":700,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.119832+0000","last_change":"2024-09-18T07:43:40.066467+0000","last_active":"2024-09-18T07:44:14.119832+0000","last_peered":"2024-09-18T07:44:14.119832+0000","last_clean":"2024-09-18T07:44:14.119832+0000","last_became_active":"2024-09-18T07:35:01.403680+0000","last_became_peered":"2024-09-18T07:35:01.403680+0000","last_unstale":"2024-09-18T07:44:14.119832+0000","last_undegraded":"2024-09-18T07:44:14.119832+0000","last_fullsized":"2024-09-18T07:44:14.119832+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:40.066356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:43:40.066356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:45.038610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":467,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.119956+0000","last_change":"2024-09-18T07:40:54.574254+0000","last_active":"2024-09-18T07:44:14.119956+0000","last_peered":"2024-09-18T07:44:14.119956+0000","last_clean":"2024-09-18T07:44:14.119956+0000","last_became_active":"2024-09-18T07:37:43.060019+0000","last_became_peered":"2024-09-18T07:37:43.060019+0000","last_unstale":"2024-09-18T07:44:14.119956+0000","last_undegraded":"2024-09-18T07:44:14.119956+0000","last_fullsized":"2024-09-18T07:44:14.119956+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:40:54.574077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:54.574077+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:57.471913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"382'6565","reported_seq":6612,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:12.784840+0000","last_change":"2024-09-18T07:43:02.044389+0000","last_active":"2024-09-18T07:44:12.784840+0000","last_peered":"2024-09-18T07:44:12.784840+0000","last_clean":"2024-09-18T07:44:12.784840+0000","last_became_active":"2024-09-18T07:43:02.044010+0000","last_became_peered":"2024-09-18T07:43:02.044010+0000","last_unstale":"2024-09-18T07:44:12.784840+0000","last_undegraded":"2024-09-18T07:44:12.784840+0000","last_fullsized":"2024-09-18T07:44:12.784840+0000","mapping_epoch":364,"log_start":"369'1600","ondisk_log_start":"369'1600","created":364,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":4965,"log_dups_size":0,"ondisk_log_size":4965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:13.064096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430243840,"num_objects":6565,"num_object_clones":0,"num_object_copies":19695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6565,"num_write_kb":420160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":417,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.110363+0000","last_change":"2024-09-18T07:42:54.743090+0000","last_active":"2024-09-18T07:44:14.110363+0000","last_peered":"2024-09-18T07:44:14.110363+0000","last_clean":"2024-09-18T07:42:51.730329+0000","last_became_active":"2024-09-18T07:42:54.743090+0000","last_became_peered":"2024-09-18T07:42:54.743090+0000","last_unstale":"2024-09-18T07:44:14.110363+0000","last_undegraded":"2024-09-18T07:44:14.110363+0000","last_fullsized":"2024-09-18T07:42:54.735572+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:42:39.191605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:42:39.191605+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.092770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"323'24","reported_seq":725,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.043501+0000","last_change":"2024-09-18T07:44:06.211747+0000","last_active":"2024-09-18T07:44:10.043501+0000","last_peered":"2024-09-18T07:44:10.043501+0000","last_clean":"2024-09-18T07:44:10.043501+0000","last_became_active":"2024-09-18T07:33:27.017340+0000","last_became_peered":"2024-09-18T07:33:27.017340+0000","last_unstale":"2024-09-18T07:44:10.043501+0000","last_undegraded":"2024-09-18T07:44:10.043501+0000","last_fullsized":"2024-09-18T07:44:10.043501+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'24","last_scrub_stamp":"2024-09-18T07:44:06.211676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:06.211676+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:31.945649+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":682,"reported_epoch":382,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:10.037023+0000","last_change":"2024-09-18T07:42:54.753565+0000","last_active":"2024-09-18T07:44:10.037023+0000","last_peered":"2024-09-18T07:44:10.037023+0000","last_clean":"2024-09-18T07:42:51.717122+0000","last_became_active":"2024-09-18T07:42:54.753565+0000","last_became_peered":"2024-09-18T07:42:54.753565+0000","last_unstale":"2024-09-18T07:44:10.037023+0000","last_undegraded":"2024-09-18T07:42:54.738688+0000","last_fullsized":"2024-09-18T07:42:54.738589+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:42:44.401546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:44.401546+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:07.201639+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":1000,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:15.327216+0000","last_change":"2024-09-18T07:40:47.554230+0000","last_active":"2024-09-18T07:44:15.327216+0000","last_peered":"2024-09-18T07:44:15.327216+0000","last_clean":"2024-09-18T07:44:15.327216+0000","last_became_active":"2024-09-18T07:35:01.400524+0000","last_became_peered":"2024-09-18T07:35:01.400524+0000","last_unstale":"2024-09-18T07:44:15.327216+0000","last_undegraded":"2024-09-18T07:44:15.327216+0000","last_fullsized":"2024-09-18T07:44:15.327216+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T07:40:47.554161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.554161+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:50.421249+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":561,"num_read_kb":556,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":684,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.117011+0000","last_change":"2024-09-18T07:42:02.399342+0000","last_active":"2024-09-18T07:44:14.117011+0000","last_peered":"2024-09-18T07:44:14.117011+0000","last_clean":"2024-09-18T07:44:14.117011+0000","last_became_active":"2024-09-18T07:31:00.331660+0000","last_became_peered":"2024-09-18T07:31:00.331660+0000","last_unstale":"2024-09-18T07:44:14.117011+0000","last_undegraded":"2024-09-18T07:44:14.117011+0000","last_fullsized":"2024-09-18T07:44:14.117011+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:02.399269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:02.399269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:48.466755+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":376,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.115449+0000","last_change":"2024-09-18T07:40:12.269264+0000","last_active":"2024-09-18T07:44:14.115449+0000","last_peered":"2024-09-18T07:44:14.115449+0000","last_clean":"2024-09-18T07:44:14.115449+0000","last_became_active":"2024-09-18T07:36:13.117463+0000","last_became_peered":"2024-09-18T07:36:13.117463+0000","last_unstale":"2024-09-18T07:44:14.115449+0000","last_undegraded":"2024-09-18T07:44:14.115449+0000","last_fullsized":"2024-09-18T07:44:14.115449+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:12.269201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:12.269201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:20.558675+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"383'6694","reported_seq":6743,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:16.790209+0000","last_change":"2024-09-18T07:43:02.069622+0000","last_active":"2024-09-18T07:44:16.790209+0000","last_peered":"2024-09-18T07:44:16.790209+0000","last_clean":"2024-09-18T07:44:16.790209+0000","last_became_active":"2024-09-18T07:43:02.069360+0000","last_became_peered":"2024-09-18T07:43:02.069360+0000","last_unstale":"2024-09-18T07:44:16.790209+0000","last_undegraded":"2024-09-18T07:44:16.790209+0000","last_fullsized":"2024-09-18T07:44:16.790209+0000","mapping_epoch":364,"log_start":"368'1100","ondisk_log_start":"368'1100","created":364,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:18.204978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438697984,"num_objects":6694,"num_object_clones":0,"num_object_copies":20082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6694,"num_write_kb":428416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"360'33","reported_seq":428,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.109425+0000","last_change":"2024-09-18T07:42:54.743919+0000","last_active":"2024-09-18T07:44:14.109425+0000","last_peered":"2024-09-18T07:44:14.109425+0000","last_clean":"2024-09-18T07:42:51.730345+0000","last_became_active":"2024-09-18T07:42:54.743919+0000","last_became_peered":"2024-09-18T07:42:54.743919+0000","last_unstale":"2024-09-18T07:44:14.109425+0000","last_undegraded":"2024-09-18T07:42:54.735862+0000","last_fullsized":"2024-09-18T07:42:54.735753+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-09-18T07:40:17.950298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:17.950298+0000","objects_scrubbed":8,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:10.250575+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":8}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":554,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.115828+0000","last_change":"2024-09-18T07:42:35.555848+0000","last_active":"2024-09-18T07:44:14.115828+0000","last_peered":"2024-09-18T07:44:14.115828+0000","last_clean":"2024-09-18T07:44:14.115828+0000","last_became_active":"2024-09-18T07:40:19.067624+0000","last_became_peered":"2024-09-18T07:40:19.067624+0000","last_unstale":"2024-09-18T07:44:14.115828+0000","last_undegraded":"2024-09-18T07:44:14.115828+0000","last_fullsized":"2024-09-18T07:44:14.115828+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:35.555785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:35.555785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":507,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.119352+0000","last_change":"2024-09-18T07:40:42.452815+0000","last_active":"2024-09-18T07:44:14.119352+0000","last_peered":"2024-09-18T07:44:14.119352+0000","last_clean":"2024-09-18T07:44:14.119352+0000","last_became_active":"2024-09-18T07:39:03.350419+0000","last_became_peered":"2024-09-18T07:39:03.350419+0000","last_unstale":"2024-09-18T07:44:14.119352+0000","last_undegraded":"2024-09-18T07:44:14.119352+0000","last_fullsized":"2024-09-18T07:44:14.119352+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:42.452744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:42.452744+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:17.111759+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":457,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.120032+0000","last_change":"2024-09-18T07:42:06.726510+0000","last_active":"2024-09-18T07:44:14.120032+0000","last_peered":"2024-09-18T07:44:14.120032+0000","last_clean":"2024-09-18T07:44:14.120032+0000","last_became_active":"2024-09-18T07:35:01.412217+0000","last_became_peered":"2024-09-18T07:35:01.412217+0000","last_unstale":"2024-09-18T07:44:14.120032+0000","last_undegraded":"2024-09-18T07:44:14.120032+0000","last_fullsized":"2024-09-18T07:44:14.120032+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.726445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.726445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:22.156992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":668,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.116827+0000","last_change":"2024-09-18T07:42:54.753417+0000","last_active":"2024-09-18T07:44:14.116827+0000","last_peered":"2024-09-18T07:44:14.116827+0000","last_clean":"2024-09-18T07:42:51.721892+0000","last_became_active":"2024-09-18T07:42:54.753417+0000","last_became_peered":"2024-09-18T07:42:54.753417+0000","last_unstale":"2024-09-18T07:44:14.116827+0000","last_undegraded":"2024-09-18T07:44:14.116827+0000","last_fullsized":"2024-09-18T07:42:54.735411+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:36.592215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:36.592215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:13.425611+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"383'6729","reported_seq":6775,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:16.769586+0000","last_change":"2024-09-18T07:43:10.055584+0000","last_active":"2024-09-18T07:44:16.769586+0000","last_peered":"2024-09-18T07:44:16.769586+0000","last_clean":"2024-09-18T07:43:01.032791+0000","last_became_active":"2024-09-18T07:43:02.064004+0000","last_became_peered":"2024-09-18T07:43:02.064004+0000","last_unstale":"2024-09-18T07:44:16.769586+0000","last_undegraded":"2024-09-18T07:43:10.054922+0000","last_fullsized":"2024-09-18T07:43:02.043033+0000","mapping_epoch":364,"log_start":"368'1100","ondisk_log_start":"368'1100","created":364,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5629,"log_dups_size":0,"ondisk_log_size":5629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440991744,"num_objects":6729,"num_object_clones":0,"num_object_copies":20187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6729,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6729,"num_write_kb":430656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":6729}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"323'19","reported_seq":720,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.109371+0000","last_change":"2024-09-18T07:42:54.743582+0000","last_active":"2024-09-18T07:44:14.109371+0000","last_peered":"2024-09-18T07:44:14.109371+0000","last_clean":"2024-09-18T07:41:05.256958+0000","last_became_active":"2024-09-18T07:42:54.743582+0000","last_became_peered":"2024-09-18T07:42:54.743582+0000","last_unstale":"2024-09-18T07:44:14.109371+0000","last_undegraded":"2024-09-18T07:42:54.734793+0000","last_fullsized":"2024-09-18T07:42:54.734687+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:42:43.402858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:43.402858+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:48.130660+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":297,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.115934+0000","last_change":"2024-09-18T07:42:54.751650+0000","last_active":"2024-09-18T07:44:14.115934+0000","last_peered":"2024-09-18T07:44:14.115934+0000","last_clean":"2024-09-18T07:34:59.365492+0000","last_became_active":"2024-09-18T07:42:54.751650+0000","last_became_peered":"2024-09-18T07:42:54.751650+0000","last_unstale":"2024-09-18T07:44:14.115934+0000","last_undegraded":"2024-09-18T07:44:14.115934+0000","last_fullsized":"2024-09-18T07:42:54.735937+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.078600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.078600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:59.149716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":693,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.116783+0000","last_change":"2024-09-18T07:42:54.742511+0000","last_active":"2024-09-18T07:44:14.116783+0000","last_peered":"2024-09-18T07:44:14.116783+0000","last_clean":"2024-09-18T07:42:51.721730+0000","last_became_active":"2024-09-18T07:42:54.742511+0000","last_became_peered":"2024-09-18T07:42:54.742511+0000","last_unstale":"2024-09-18T07:44:14.116783+0000","last_undegraded":"2024-09-18T07:44:14.116783+0000","last_fullsized":"2024-09-18T07:42:54.734185+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:35.406041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:35.406041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:58.434963+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":548,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.108896+0000","last_change":"2024-09-18T07:41:20.220326+0000","last_active":"2024-09-18T07:44:14.108896+0000","last_peered":"2024-09-18T07:44:14.108896+0000","last_clean":"2024-09-18T07:44:14.108896+0000","last_became_active":"2024-09-18T07:40:19.068998+0000","last_became_peered":"2024-09-18T07:40:19.068998+0000","last_unstale":"2024-09-18T07:44:14.108896+0000","last_undegraded":"2024-09-18T07:44:14.108896+0000","last_fullsized":"2024-09-18T07:44:14.108896+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:20.220262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:20.220262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:23.184298+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3","version":"383'6840","reported_seq":6887,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:16.967442+0000","last_change":"2024-09-18T07:43:02.064631+0000","last_active":"2024-09-18T07:44:16.967442+0000","last_peered":"2024-09-18T07:44:16.967442+0000","last_clean":"2024-09-18T07:44:16.967442+0000","last_became_active":"2024-09-18T07:43:02.064298+0000","last_became_peered":"2024-09-18T07:43:02.064298+0000","last_unstale":"2024-09-18T07:44:16.967442+0000","last_undegraded":"2024-09-18T07:44:16.967442+0000","last_fullsized":"2024-09-18T07:44:16.967442+0000","mapping_epoch":364,"log_start":"368'900","ondisk_log_start":"368'900","created":364,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5940,"log_dups_size":0,"ondisk_log_size":5940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:24.871852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448266240,"num_objects":6840,"num_object_clones":0,"num_object_copies":20520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6840,"num_write_kb":437760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":248,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.114766+0000","last_change":"2024-09-18T07:39:38.447787+0000","last_active":"2024-09-18T07:44:14.114766+0000","last_peered":"2024-09-18T07:44:14.114766+0000","last_clean":"2024-09-18T07:44:14.114766+0000","last_became_active":"2024-09-18T07:39:04.950292+0000","last_became_peered":"2024-09-18T07:39:04.950292+0000","last_unstale":"2024-09-18T07:44:14.114766+0000","last_undegraded":"2024-09-18T07:44:14.114766+0000","last_fullsized":"2024-09-18T07:44:14.114766+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:39:38.447735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:39:38.447735+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:52.496773+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"323'31","reported_seq":671,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.134706+0000","last_change":"2024-09-18T07:42:54.742580+0000","last_active":"2024-09-18T07:44:14.134706+0000","last_peered":"2024-09-18T07:44:14.134706+0000","last_clean":"2024-09-18T07:41:05.204075+0000","last_became_active":"2024-09-18T07:42:54.742580+0000","last_became_peered":"2024-09-18T07:42:54.742580+0000","last_unstale":"2024-09-18T07:44:14.134706+0000","last_undegraded":"2024-09-18T07:42:54.738910+0000","last_fullsized":"2024-09-18T07:42:54.738808+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:40:04.281758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:04.281758+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:06.203438+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":328,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.109708+0000","last_change":"2024-09-18T07:42:54.741952+0000","last_active":"2024-09-18T07:44:14.109708+0000","last_peered":"2024-09-18T07:44:14.109708+0000","last_clean":"2024-09-18T07:42:51.729886+0000","last_became_active":"2024-09-18T07:42:54.741952+0000","last_became_peered":"2024-09-18T07:42:54.741952+0000","last_unstale":"2024-09-18T07:44:14.109708+0000","last_undegraded":"2024-09-18T07:44:14.109708+0000","last_fullsized":"2024-09-18T07:42:54.735163+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:54.294307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.294307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":56,"reported_epoch":382,"state":"active+undersized","last_fresh":"2024-09-18T07:44:10.039684+0000","last_change":"2024-09-18T07:42:54.750942+0000","last_active":"2024-09-18T07:44:10.039684+0000","last_peered":"2024-09-18T07:44:10.039684+0000","last_clean":"2024-09-18T07:30:59.305166+0000","last_became_active":"2024-09-18T07:42:54.750942+0000","last_became_peered":"2024-09-18T07:42:54.750942+0000","last_unstale":"2024-09-18T07:44:10.039684+0000","last_undegraded":"2024-09-18T07:44:10.039684+0000","last_fullsized":"2024-09-18T07:42:54.740397+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:52.167048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:52.167048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:02.242444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":693,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.121332+0000","last_change":"2024-09-18T07:42:54.749690+0000","last_active":"2024-09-18T07:44:14.121332+0000","last_peered":"2024-09-18T07:44:14.121332+0000","last_clean":"2024-09-18T07:42:51.717771+0000","last_became_active":"2024-09-18T07:42:54.749690+0000","last_became_peered":"2024-09-18T07:42:54.749690+0000","last_unstale":"2024-09-18T07:44:14.121332+0000","last_undegraded":"2024-09-18T07:44:14.121332+0000","last_fullsized":"2024-09-18T07:42:54.737695+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:56.172797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:56.172797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:10.127929+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"382'6509","reported_seq":6553,"reported_epoch":382,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:12.733063+0000","last_change":"2024-09-18T07:43:10.056592+0000","last_active":"2024-09-18T07:44:12.733063+0000","last_peered":"2024-09-18T07:44:12.733063+0000","last_clean":"2024-09-18T07:43:01.032791+0000","last_became_active":"2024-09-18T07:43:02.063956+0000","last_became_peered":"2024-09-18T07:43:02.063956+0000","last_unstale":"2024-09-18T07:44:12.733063+0000","last_undegraded":"2024-09-18T07:43:10.052693+0000","last_fullsized":"2024-09-18T07:43:02.038580+0000","mapping_epoch":364,"log_start":"369'1500","ondisk_log_start":"369'1500","created":364,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:30.710726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426573824,"num_objects":6509,"num_object_clones":0,"num_object_copies":19527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6509,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6509}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":513,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.119651+0000","last_change":"2024-09-18T07:42:32.807852+0000","last_active":"2024-09-18T07:44:14.119651+0000","last_peered":"2024-09-18T07:44:14.119651+0000","last_clean":"2024-09-18T07:44:14.119651+0000","last_became_active":"2024-09-18T07:39:04.944753+0000","last_became_peered":"2024-09-18T07:39:04.944753+0000","last_unstale":"2024-09-18T07:44:14.119651+0000","last_undegraded":"2024-09-18T07:44:14.119651+0000","last_fullsized":"2024-09-18T07:44:14.119651+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:42:32.807800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:42:32.807800+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"323'20","reported_seq":674,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.124275+0000","last_change":"2024-09-18T07:42:54.753491+0000","last_active":"2024-09-18T07:44:14.124275+0000","last_peered":"2024-09-18T07:44:14.124275+0000","last_clean":"2024-09-18T07:41:05.073165+0000","last_became_active":"2024-09-18T07:42:54.753491+0000","last_became_peered":"2024-09-18T07:42:54.753491+0000","last_unstale":"2024-09-18T07:44:14.124275+0000","last_undegraded":"2024-09-18T07:42:54.745505+0000","last_fullsized":"2024-09-18T07:42:54.745363+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-18T07:40:50.209824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:50.209824+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:07.839589+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":175,"reported_epoch":382,"state":"active+undersized","last_fresh":"2024-09-18T07:44:10.037904+0000","last_change":"2024-09-18T07:42:54.747701+0000","last_active":"2024-09-18T07:44:10.037904+0000","last_peered":"2024-09-18T07:44:10.037904+0000","last_clean":"2024-09-18T07:42:51.718440+0000","last_became_active":"2024-09-18T07:42:54.747701+0000","last_became_peered":"2024-09-18T07:42:54.747701+0000","last_unstale":"2024-09-18T07:44:10.037904+0000","last_undegraded":"2024-09-18T07:44:10.037904+0000","last_fullsized":"2024-09-18T07:42:54.739078+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:47.799342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.799342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:59.139030+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"173'11","reported_seq":879,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.134659+0000","last_change":"2024-09-18T07:42:40.909417+0000","last_active":"2024-09-18T07:44:14.134659+0000","last_peered":"2024-09-18T07:44:14.134659+0000","last_clean":"2024-09-18T07:44:14.134659+0000","last_became_active":"2024-09-18T07:35:01.403765+0000","last_became_peered":"2024-09-18T07:35:01.403765+0000","last_unstale":"2024-09-18T07:44:14.134659+0000","last_undegraded":"2024-09-18T07:44:14.134659+0000","last_fullsized":"2024-09-18T07:44:14.134659+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'11","last_scrub_stamp":"2024-09-18T07:42:40.909351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:40.909351+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:48.131824+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":57,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.119540+0000","last_change":"2024-09-18T07:42:54.745260+0000","last_active":"2024-09-18T07:44:14.119540+0000","last_peered":"2024-09-18T07:44:14.119540+0000","last_clean":"2024-09-18T07:31:01.319795+0000","last_became_active":"2024-09-18T07:42:54.745260+0000","last_became_peered":"2024-09-18T07:42:54.745260+0000","last_unstale":"2024-09-18T07:44:14.119540+0000","last_undegraded":"2024-09-18T07:44:14.119540+0000","last_fullsized":"2024-09-18T07:42:54.735312+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:36:33.251093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:36:33.251093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:56.143249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.5","version":"383'6754","reported_seq":6876,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:16.956333+0000","last_change":"2024-09-18T07:44:16.564271+0000","last_active":"2024-09-18T07:44:16.956333+0000","last_peered":"2024-09-18T07:44:16.956333+0000","last_clean":"2024-09-18T07:44:16.956333+0000","last_became_active":"2024-09-18T07:43:02.070728+0000","last_became_peered":"2024-09-18T07:43:02.070728+0000","last_unstale":"2024-09-18T07:44:16.956333+0000","last_undegraded":"2024-09-18T07:44:16.956333+0000","last_fullsized":"2024-09-18T07:44:16.956333+0000","mapping_epoch":364,"log_start":"368'800","ondisk_log_start":"368'800","created":364,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"383'6722","last_scrub_stamp":"2024-09-18T07:44:16.563963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:44:16.563963+0000","objects_scrubbed":6615,"log_size":5954,"log_dups_size":0,"ondisk_log_size":5954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:22.267657+0000","scrub_duration":3476,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442630144,"num_objects":6754,"num_object_clones":0,"num_object_copies":20262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6754,"num_write_kb":432256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"323'15","reported_seq":721,"reported_epoch":382,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:10.037580+0000","last_change":"2024-09-18T07:42:54.751765+0000","last_active":"2024-09-18T07:44:10.037580+0000","last_peered":"2024-09-18T07:44:10.037580+0000","last_clean":"2024-09-18T07:42:51.718211+0000","last_became_active":"2024-09-18T07:42:54.751765+0000","last_became_peered":"2024-09-18T07:42:54.751765+0000","last_unstale":"2024-09-18T07:44:10.037580+0000","last_undegraded":"2024-09-18T07:42:54.736391+0000","last_fullsized":"2024-09-18T07:42:54.736259+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:21.946066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:21.946066+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:10.233191+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":705,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.115543+0000","last_change":"2024-09-18T07:43:04.518722+0000","last_active":"2024-09-18T07:44:14.115543+0000","last_peered":"2024-09-18T07:44:14.115543+0000","last_clean":"2024-09-18T07:44:14.115543+0000","last_became_active":"2024-09-18T07:37:43.075988+0000","last_became_peered":"2024-09-18T07:37:43.075988+0000","last_unstale":"2024-09-18T07:44:14.115543+0000","last_undegraded":"2024-09-18T07:44:14.115543+0000","last_fullsized":"2024-09-18T07:44:14.115543+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:04.518676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:43:04.518676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:16.665732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":682,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.115280+0000","last_change":"2024-09-18T07:42:54.751297+0000","last_active":"2024-09-18T07:44:14.115280+0000","last_peered":"2024-09-18T07:44:14.115280+0000","last_clean":"2024-09-18T07:42:51.718134+0000","last_became_active":"2024-09-18T07:42:54.751297+0000","last_became_peered":"2024-09-18T07:42:54.751297+0000","last_unstale":"2024-09-18T07:44:14.115280+0000","last_undegraded":"2024-09-18T07:44:14.115280+0000","last_fullsized":"2024-09-18T07:42:54.743454+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:42.922615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:42.922615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:08.184459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":698,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.036914+0000","last_change":"2024-09-18T07:43:34.257816+0000","last_active":"2024-09-18T07:44:10.036914+0000","last_peered":"2024-09-18T07:44:10.036914+0000","last_clean":"2024-09-18T07:44:10.036914+0000","last_became_active":"2024-09-18T07:35:01.410890+0000","last_became_peered":"2024-09-18T07:35:01.410890+0000","last_unstale":"2024-09-18T07:44:10.036914+0000","last_undegraded":"2024-09-18T07:44:10.036914+0000","last_fullsized":"2024-09-18T07:44:10.036914+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:34.257746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:34.257746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:58.955181+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"382'6520","reported_seq":6562,"reported_epoch":382,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:12.768502+0000","last_change":"2024-09-18T07:43:10.063264+0000","last_active":"2024-09-18T07:44:12.768502+0000","last_peered":"2024-09-18T07:44:12.768502+0000","last_clean":"2024-09-18T07:43:01.032791+0000","last_became_active":"2024-09-18T07:43:02.071284+0000","last_became_peered":"2024-09-18T07:43:02.071284+0000","last_unstale":"2024-09-18T07:44:12.768502+0000","last_undegraded":"2024-09-18T07:43:10.058462+0000","last_fullsized":"2024-09-18T07:43:02.038779+0000","mapping_epoch":364,"log_start":"369'1600","ondisk_log_start":"369'1600","created":364,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":4920,"log_dups_size":0,"ondisk_log_size":4920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:20.627488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427294720,"num_objects":6520,"num_object_clones":0,"num_object_copies":19560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6520,"num_write_kb":417280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":6520}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"323'35","reported_seq":360,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.109969+0000","last_change":"2024-09-18T07:42:44.097321+0000","last_active":"2024-09-18T07:44:14.109969+0000","last_peered":"2024-09-18T07:44:14.109969+0000","last_clean":"2024-09-18T07:44:14.109969+0000","last_became_active":"2024-09-18T07:37:43.063319+0000","last_became_peered":"2024-09-18T07:37:43.063319+0000","last_unstale":"2024-09-18T07:44:14.109969+0000","last_undegraded":"2024-09-18T07:44:14.109969+0000","last_fullsized":"2024-09-18T07:44:14.109969+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'35","last_scrub_stamp":"2024-09-18T07:42:44.097218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:44.097218+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":471,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.121081+0000","last_change":"2024-09-18T07:40:56.546321+0000","last_active":"2024-09-18T07:44:14.121081+0000","last_peered":"2024-09-18T07:44:14.121081+0000","last_clean":"2024-09-18T07:44:14.121081+0000","last_became_active":"2024-09-18T07:37:43.072894+0000","last_became_peered":"2024-09-18T07:37:43.072894+0000","last_unstale":"2024-09-18T07:44:14.121081+0000","last_undegraded":"2024-09-18T07:44:14.121081+0000","last_fullsized":"2024-09-18T07:44:14.121081+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:56.546251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.546251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:54.470740+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":692,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.119442+0000","last_change":"2024-09-18T07:42:04.179746+0000","last_active":"2024-09-18T07:44:14.119442+0000","last_peered":"2024-09-18T07:44:14.119442+0000","last_clean":"2024-09-18T07:44:14.119442+0000","last_became_active":"2024-09-18T07:33:27.009692+0000","last_became_peered":"2024-09-18T07:33:27.009692+0000","last_unstale":"2024-09-18T07:44:14.119442+0000","last_undegraded":"2024-09-18T07:44:14.119442+0000","last_fullsized":"2024-09-18T07:44:14.119442+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:04.179670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.179670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:19.929773+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":457,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.119473+0000","last_change":"2024-09-18T07:43:39.239989+0000","last_active":"2024-09-18T07:44:14.119473+0000","last_peered":"2024-09-18T07:44:14.119473+0000","last_clean":"2024-09-18T07:44:14.119473+0000","last_became_active":"2024-09-18T07:35:01.406980+0000","last_became_peered":"2024-09-18T07:35:01.406980+0000","last_unstale":"2024-09-18T07:44:14.119473+0000","last_undegraded":"2024-09-18T07:44:14.119473+0000","last_fullsized":"2024-09-18T07:44:14.119473+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:39.239932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:39.239932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:57.648423+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"383'6828","reported_seq":6877,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:16.791701+0000","last_change":"2024-09-18T07:43:02.069332+0000","last_active":"2024-09-18T07:44:16.791701+0000","last_peered":"2024-09-18T07:44:16.791701+0000","last_clean":"2024-09-18T07:44:16.791701+0000","last_became_active":"2024-09-18T07:43:02.068979+0000","last_became_peered":"2024-09-18T07:43:02.068979+0000","last_unstale":"2024-09-18T07:44:16.791701+0000","last_undegraded":"2024-09-18T07:44:16.791701+0000","last_fullsized":"2024-09-18T07:44:16.791701+0000","mapping_epoch":364,"log_start":"368'1200","ondisk_log_start":"368'1200","created":364,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5628,"log_dups_size":0,"ondisk_log_size":5628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447479808,"num_objects":6828,"num_object_clones":0,"num_object_copies":20484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6828,"num_write_kb":436992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"323'18","reported_seq":654,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.114139+0000","last_change":"2024-09-18T07:42:54.755198+0000","last_active":"2024-09-18T07:44:14.114139+0000","last_peered":"2024-09-18T07:44:14.114139+0000","last_clean":"2024-09-18T07:41:04.257612+0000","last_became_active":"2024-09-18T07:42:54.755198+0000","last_became_peered":"2024-09-18T07:42:54.755198+0000","last_unstale":"2024-09-18T07:44:14.114139+0000","last_undegraded":"2024-09-18T07:42:54.746711+0000","last_fullsized":"2024-09-18T07:42:54.745765+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:41:47.950068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:47.950068+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:29.362131+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":453,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.120667+0000","last_change":"2024-09-18T07:42:54.741218+0000","last_active":"2024-09-18T07:44:14.120667+0000","last_peered":"2024-09-18T07:44:14.120667+0000","last_clean":"2024-09-18T07:42:51.728402+0000","last_became_active":"2024-09-18T07:42:54.741218+0000","last_became_peered":"2024-09-18T07:42:54.741218+0000","last_unstale":"2024-09-18T07:44:14.120667+0000","last_undegraded":"2024-09-18T07:44:14.120667+0000","last_fullsized":"2024-09-18T07:42:54.737804+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:07.729438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:07.729438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.251922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":702,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.115323+0000","last_change":"2024-09-18T07:41:58.202324+0000","last_active":"2024-09-18T07:44:14.115323+0000","last_peered":"2024-09-18T07:44:14.115323+0000","last_clean":"2024-09-18T07:44:14.115323+0000","last_became_active":"2024-09-18T07:31:00.328855+0000","last_became_peered":"2024-09-18T07:31:00.328855+0000","last_unstale":"2024-09-18T07:44:14.115323+0000","last_undegraded":"2024-09-18T07:44:14.115323+0000","last_fullsized":"2024-09-18T07:44:14.115323+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:58.202229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:58.202229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:23.991237+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":686,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.114074+0000","last_change":"2024-09-18T07:43:01.317153+0000","last_active":"2024-09-18T07:44:14.114074+0000","last_peered":"2024-09-18T07:44:14.114074+0000","last_clean":"2024-09-18T07:44:14.114074+0000","last_became_active":"2024-09-18T07:35:01.418735+0000","last_became_peered":"2024-09-18T07:35:01.418735+0000","last_unstale":"2024-09-18T07:44:14.114074+0000","last_undegraded":"2024-09-18T07:44:14.114074+0000","last_fullsized":"2024-09-18T07:44:14.114074+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.317105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.317105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"383'6843","reported_seq":6889,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:17.149162+0000","last_change":"2024-09-18T07:43:10.055586+0000","last_active":"2024-09-18T07:44:17.149162+0000","last_peered":"2024-09-18T07:44:17.149162+0000","last_clean":"2024-09-18T07:43:01.032791+0000","last_became_active":"2024-09-18T07:43:02.048967+0000","last_became_peered":"2024-09-18T07:43:02.048967+0000","last_unstale":"2024-09-18T07:44:17.149162+0000","last_undegraded":"2024-09-18T07:43:10.055058+0000","last_fullsized":"2024-09-18T07:43:02.043209+0000","mapping_epoch":364,"log_start":"370'1800","ondisk_log_start":"370'1800","created":364,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:43:01.032791+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.032791+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:17.566081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448462848,"num_objects":6843,"num_object_clones":0,"num_object_copies":20529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6843,"num_write_kb":437952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":6843}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"323'27","reported_seq":831,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.114862+0000","last_change":"2024-09-18T07:44:04.486924+0000","last_active":"2024-09-18T07:44:14.114862+0000","last_peered":"2024-09-18T07:44:14.114862+0000","last_clean":"2024-09-18T07:44:14.114862+0000","last_became_active":"2024-09-18T07:37:43.075552+0000","last_became_peered":"2024-09-18T07:37:43.075552+0000","last_unstale":"2024-09-18T07:44:14.114862+0000","last_undegraded":"2024-09-18T07:44:14.114862+0000","last_fullsized":"2024-09-18T07:44:14.114862+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:44:04.486867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:04.486867+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:14.305922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":467,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.120582+0000","last_change":"2024-09-18T07:42:54.747864+0000","last_active":"2024-09-18T07:44:14.120582+0000","last_peered":"2024-09-18T07:44:14.120582+0000","last_clean":"2024-09-18T07:42:51.728762+0000","last_became_active":"2024-09-18T07:42:54.747864+0000","last_became_peered":"2024-09-18T07:42:54.747864+0000","last_unstale":"2024-09-18T07:44:14.120582+0000","last_undegraded":"2024-09-18T07:42:54.739112+0000","last_fullsized":"2024-09-18T07:42:54.738998+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:41:27.819269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.819269+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"360'13","reported_seq":600,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.121113+0000","last_change":"2024-09-18T07:42:54.744846+0000","last_active":"2024-09-18T07:44:14.121113+0000","last_peered":"2024-09-18T07:44:14.121113+0000","last_clean":"2024-09-18T07:42:51.728785+0000","last_became_active":"2024-09-18T07:42:54.744846+0000","last_became_peered":"2024-09-18T07:42:54.744846+0000","last_unstale":"2024-09-18T07:44:14.121113+0000","last_undegraded":"2024-09-18T07:42:54.737124+0000","last_fullsized":"2024-09-18T07:42:54.736998+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12","last_scrub_stamp":"2024-09-18T07:42:35.793579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:35.793579+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:48.351749+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":678,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.115442+0000","last_change":"2024-09-18T07:42:07.547399+0000","last_active":"2024-09-18T07:44:14.115442+0000","last_peered":"2024-09-18T07:44:14.115442+0000","last_clean":"2024-09-18T07:44:14.115442+0000","last_became_active":"2024-09-18T07:37:43.078121+0000","last_became_peered":"2024-09-18T07:37:43.078121+0000","last_unstale":"2024-09-18T07:44:14.115442+0000","last_undegraded":"2024-09-18T07:44:14.115442+0000","last_fullsized":"2024-09-18T07:44:14.115442+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:07.547352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:07.547352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:49.424413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":315,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.108789+0000","last_change":"2024-09-18T07:42:08.181612+0000","last_active":"2024-09-18T07:44:14.108789+0000","last_peered":"2024-09-18T07:44:14.108789+0000","last_clean":"2024-09-18T07:44:14.108789+0000","last_became_active":"2024-09-18T07:37:43.087994+0000","last_became_peered":"2024-09-18T07:37:43.087994+0000","last_unstale":"2024-09-18T07:44:14.108789+0000","last_undegraded":"2024-09-18T07:44:14.108789+0000","last_fullsized":"2024-09-18T07:44:14.108789+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:08.181420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:08.181420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:27.181019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"323'12","reported_seq":728,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.123968+0000","last_change":"2024-09-18T07:41:48.359809+0000","last_active":"2024-09-18T07:44:14.123968+0000","last_peered":"2024-09-18T07:44:14.123968+0000","last_clean":"2024-09-18T07:44:14.123968+0000","last_became_active":"2024-09-18T07:35:01.407787+0000","last_became_peered":"2024-09-18T07:35:01.407787+0000","last_unstale":"2024-09-18T07:44:14.123968+0000","last_undegraded":"2024-09-18T07:44:14.123968+0000","last_fullsized":"2024-09-18T07:44:14.123968+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'12","last_scrub_stamp":"2024-09-18T07:41:48.359711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.359711+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:19.370461+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":691,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.115901+0000","last_change":"2024-09-18T07:43:37.515165+0000","last_active":"2024-09-18T07:44:14.115901+0000","last_peered":"2024-09-18T07:44:14.115901+0000","last_clean":"2024-09-18T07:44:14.115901+0000","last_became_active":"2024-09-18T07:31:00.333878+0000","last_became_peered":"2024-09-18T07:31:00.333878+0000","last_unstale":"2024-09-18T07:44:14.115901+0000","last_undegraded":"2024-09-18T07:44:14.115901+0000","last_fullsized":"2024-09-18T07:44:14.115901+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:37.515124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:43:37.515124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:47.562291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":458,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.120092+0000","last_change":"2024-09-18T07:42:54.745409+0000","last_active":"2024-09-18T07:44:14.120092+0000","last_peered":"2024-09-18T07:44:14.120092+0000","last_clean":"2024-09-18T07:42:51.728619+0000","last_became_active":"2024-09-18T07:42:54.745409+0000","last_became_peered":"2024-09-18T07:42:54.745409+0000","last_unstale":"2024-09-18T07:44:14.120092+0000","last_undegraded":"2024-09-18T07:44:14.120092+0000","last_fullsized":"2024-09-18T07:42:54.737509+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:51.554073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:51.554073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:58.488232+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"323'15","reported_seq":718,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.043584+0000","last_change":"2024-09-18T07:42:34.313755+0000","last_active":"2024-09-18T07:44:10.043584+0000","last_peered":"2024-09-18T07:44:10.043584+0000","last_clean":"2024-09-18T07:44:10.043584+0000","last_became_active":"2024-09-18T07:35:01.414458+0000","last_became_peered":"2024-09-18T07:35:01.414458+0000","last_unstale":"2024-09-18T07:44:10.043584+0000","last_undegraded":"2024-09-18T07:44:10.043584+0000","last_fullsized":"2024-09-18T07:44:10.043584+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:42:34.313665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:34.313665+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:51.059507+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":474,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.120083+0000","last_change":"2024-09-18T07:42:54.747009+0000","last_active":"2024-09-18T07:44:14.120083+0000","last_peered":"2024-09-18T07:44:14.120083+0000","last_clean":"2024-09-18T07:42:51.728811+0000","last_became_active":"2024-09-18T07:42:54.747009+0000","last_became_peered":"2024-09-18T07:42:54.747009+0000","last_unstale":"2024-09-18T07:44:14.120083+0000","last_undegraded":"2024-09-18T07:44:14.120083+0000","last_fullsized":"2024-09-18T07:42:54.737030+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:52.891811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:52.891811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:23.178220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":698,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.123610+0000","last_change":"2024-09-18T07:41:16.472282+0000","last_active":"2024-09-18T07:44:14.123610+0000","last_peered":"2024-09-18T07:44:14.123610+0000","last_clean":"2024-09-18T07:44:14.123610+0000","last_became_active":"2024-09-18T07:35:01.420133+0000","last_became_peered":"2024-09-18T07:35:01.420133+0000","last_unstale":"2024-09-18T07:44:14.123610+0000","last_undegraded":"2024-09-18T07:44:14.123610+0000","last_fullsized":"2024-09-18T07:44:14.123610+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:16.472210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:16.472210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":683,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.119134+0000","last_change":"2024-09-18T07:42:54.744738+0000","last_active":"2024-09-18T07:44:14.119134+0000","last_peered":"2024-09-18T07:44:14.119134+0000","last_clean":"2024-09-18T07:42:51.717389+0000","last_became_active":"2024-09-18T07:42:54.744738+0000","last_became_peered":"2024-09-18T07:42:54.744738+0000","last_unstale":"2024-09-18T07:44:14.119134+0000","last_undegraded":"2024-09-18T07:44:14.119134+0000","last_fullsized":"2024-09-18T07:42:54.736826+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:34.876429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:34.876429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"323'15","reported_seq":738,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.115701+0000","last_change":"2024-09-18T07:40:52.366957+0000","last_active":"2024-09-18T07:44:14.115701+0000","last_peered":"2024-09-18T07:44:14.115701+0000","last_clean":"2024-09-18T07:44:14.115701+0000","last_became_active":"2024-09-18T07:35:01.416744+0000","last_became_peered":"2024-09-18T07:35:01.416744+0000","last_unstale":"2024-09-18T07:44:14.115701+0000","last_undegraded":"2024-09-18T07:44:14.115701+0000","last_fullsized":"2024-09-18T07:44:14.115701+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:52.366672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:52.366672+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:28.664103+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":709,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.042340+0000","last_change":"2024-09-18T07:43:31.255482+0000","last_active":"2024-09-18T07:44:10.042340+0000","last_peered":"2024-09-18T07:44:10.042340+0000","last_clean":"2024-09-18T07:44:10.042340+0000","last_became_active":"2024-09-18T07:35:01.414024+0000","last_became_peered":"2024-09-18T07:35:01.414024+0000","last_unstale":"2024-09-18T07:44:10.042340+0000","last_undegraded":"2024-09-18T07:44:10.042340+0000","last_fullsized":"2024-09-18T07:44:10.042340+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:31.255278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:43:31.255278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:37.083443+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"173'10","reported_seq":867,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.124182+0000","last_change":"2024-09-18T07:42:58.271458+0000","last_active":"2024-09-18T07:44:14.124182+0000","last_peered":"2024-09-18T07:44:14.124182+0000","last_clean":"2024-09-18T07:44:14.124182+0000","last_became_active":"2024-09-18T07:35:01.407568+0000","last_became_peered":"2024-09-18T07:35:01.407568+0000","last_unstale":"2024-09-18T07:44:14.124182+0000","last_undegraded":"2024-09-18T07:44:14.124182+0000","last_fullsized":"2024-09-18T07:44:14.124182+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:42:58.271411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:58.271411+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":57,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.120076+0000","last_change":"2024-09-18T07:42:54.748643+0000","last_active":"2024-09-18T07:44:14.120076+0000","last_peered":"2024-09-18T07:44:14.120076+0000","last_clean":"2024-09-18T07:31:01.319795+0000","last_became_active":"2024-09-18T07:42:54.748643+0000","last_became_peered":"2024-09-18T07:42:54.748643+0000","last_unstale":"2024-09-18T07:44:14.120076+0000","last_undegraded":"2024-09-18T07:44:14.120076+0000","last_fullsized":"2024-09-18T07:42:54.735305+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:48.955463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.955463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:23.004141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"323'20","reported_seq":457,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.119527+0000","last_change":"2024-09-18T07:42:29.867736+0000","last_active":"2024-09-18T07:44:14.119527+0000","last_peered":"2024-09-18T07:44:14.119527+0000","last_clean":"2024-09-18T07:44:14.119527+0000","last_became_active":"2024-09-18T07:40:19.069759+0000","last_became_peered":"2024-09-18T07:40:19.069759+0000","last_unstale":"2024-09-18T07:44:14.119527+0000","last_undegraded":"2024-09-18T07:44:14.119527+0000","last_fullsized":"2024-09-18T07:44:14.119527+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:42:29.867645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:29.867645+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:47.303983+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":460,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.119694+0000","last_change":"2024-09-18T07:42:54.743824+0000","last_active":"2024-09-18T07:44:14.119694+0000","last_peered":"2024-09-18T07:44:14.119694+0000","last_clean":"2024-09-18T07:42:51.728597+0000","last_became_active":"2024-09-18T07:42:54.743824+0000","last_became_peered":"2024-09-18T07:42:54.743824+0000","last_unstale":"2024-09-18T07:44:14.119694+0000","last_undegraded":"2024-09-18T07:44:14.119694+0000","last_fullsized":"2024-09-18T07:42:54.738810+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:04.758396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.758396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:36.134989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":839,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.115301+0000","last_change":"2024-09-18T07:40:40.352088+0000","last_active":"2024-09-18T07:44:14.115301+0000","last_peered":"2024-09-18T07:44:14.115301+0000","last_clean":"2024-09-18T07:44:14.115301+0000","last_became_active":"2024-09-18T07:31:00.332456+0000","last_became_peered":"2024-09-18T07:31:00.332456+0000","last_unstale":"2024-09-18T07:44:14.115301+0000","last_undegraded":"2024-09-18T07:44:14.115301+0000","last_fullsized":"2024-09-18T07:44:14.115301+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:40:40.352040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.352040+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:22.552189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":697,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.042566+0000","last_change":"2024-09-18T07:43:33.228092+0000","last_active":"2024-09-18T07:44:10.042566+0000","last_peered":"2024-09-18T07:44:10.042566+0000","last_clean":"2024-09-18T07:44:10.042566+0000","last_became_active":"2024-09-18T07:35:01.410561+0000","last_became_peered":"2024-09-18T07:35:01.410561+0000","last_unstale":"2024-09-18T07:44:10.042566+0000","last_undegraded":"2024-09-18T07:44:10.042566+0000","last_fullsized":"2024-09-18T07:44:10.042566+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:33.228024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:33.228024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:37.940247+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"323'19","reported_seq":706,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2024-09-18T07:44:14.119437+0000","last_change":"2024-09-18T07:42:54.743805+0000","last_active":"2024-09-18T07:44:14.119437+0000","last_peered":"2024-09-18T07:44:14.119437+0000","last_clean":"2024-09-18T07:41:05.098917+0000","last_became_active":"2024-09-18T07:42:54.743805+0000","last_became_peered":"2024-09-18T07:42:54.743805+0000","last_unstale":"2024-09-18T07:44:14.119437+0000","last_undegraded":"2024-09-18T07:42:54.738157+0000","last_fullsized":"2024-09-18T07:42:54.738016+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:41:54.022421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.022421+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:24.137981+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":443,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.119702+0000","last_change":"2024-09-18T07:42:54.744327+0000","last_active":"2024-09-18T07:44:14.119702+0000","last_peered":"2024-09-18T07:44:14.119702+0000","last_clean":"2024-09-18T07:37:38.987385+0000","last_became_active":"2024-09-18T07:42:54.744327+0000","last_became_peered":"2024-09-18T07:42:54.744327+0000","last_unstale":"2024-09-18T07:44:14.119702+0000","last_undegraded":"2024-09-18T07:44:14.119702+0000","last_fullsized":"2024-09-18T07:42:54.737730+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:40.149313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.149313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:55.486321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":376,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.115554+0000","last_change":"2024-09-18T07:43:36.563063+0000","last_active":"2024-09-18T07:44:14.115554+0000","last_peered":"2024-09-18T07:44:14.115554+0000","last_clean":"2024-09-18T07:44:14.115554+0000","last_became_active":"2024-09-18T07:36:13.117584+0000","last_became_peered":"2024-09-18T07:36:13.117584+0000","last_unstale":"2024-09-18T07:44:14.115554+0000","last_undegraded":"2024-09-18T07:44:14.115554+0000","last_fullsized":"2024-09-18T07:44:14.115554+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:36.563010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:43:36.563010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:44.842956+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":178,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.036817+0000","last_change":"2024-09-18T07:43:45.121141+0000","last_active":"2024-09-18T07:44:10.036817+0000","last_peered":"2024-09-18T07:44:10.036817+0000","last_clean":"2024-09-18T07:44:10.036817+0000","last_became_active":"2024-09-18T07:40:19.065468+0000","last_became_peered":"2024-09-18T07:40:19.065468+0000","last_unstale":"2024-09-18T07:44:10.036817+0000","last_undegraded":"2024-09-18T07:44:10.036817+0000","last_fullsized":"2024-09-18T07:44:10.036817+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:45.121082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:45.121082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:56.672102+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"323'15","reported_seq":573,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.041964+0000","last_change":"2024-09-18T07:44:04.220188+0000","last_active":"2024-09-18T07:44:10.041964+0000","last_peered":"2024-09-18T07:44:10.041964+0000","last_clean":"2024-09-18T07:44:10.041964+0000","last_became_active":"2024-09-18T07:37:43.067598+0000","last_became_peered":"2024-09-18T07:37:43.067598+0000","last_unstale":"2024-09-18T07:44:10.041964+0000","last_undegraded":"2024-09-18T07:44:10.041964+0000","last_fullsized":"2024-09-18T07:44:10.041964+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:44:04.219913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:04.219913+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:09.183151+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":460,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.120060+0000","last_change":"2024-09-18T07:42:54.748561+0000","last_active":"2024-09-18T07:44:14.120060+0000","last_peered":"2024-09-18T07:44:14.120060+0000","last_clean":"2024-09-18T07:42:51.728528+0000","last_became_active":"2024-09-18T07:42:54.748561+0000","last_became_peered":"2024-09-18T07:42:54.748561+0000","last_unstale":"2024-09-18T07:44:14.120060+0000","last_undegraded":"2024-09-18T07:44:14.120060+0000","last_fullsized":"2024-09-18T07:42:54.738920+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:47.880911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:47.880911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":170,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.037182+0000","last_change":"2024-09-18T07:42:41.220234+0000","last_active":"2024-09-18T07:44:10.037182+0000","last_peered":"2024-09-18T07:44:10.037182+0000","last_clean":"2024-09-18T07:44:10.037182+0000","last_became_active":"2024-09-18T07:40:19.068237+0000","last_became_peered":"2024-09-18T07:40:19.068237+0000","last_unstale":"2024-09-18T07:44:10.037182+0000","last_undegraded":"2024-09-18T07:44:10.037182+0000","last_fullsized":"2024-09-18T07:44:10.037182+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:41.219755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:41.219755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:47.130390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":683,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.044014+0000","last_change":"2024-09-18T07:41:18.963607+0000","last_active":"2024-09-18T07:44:10.044014+0000","last_peered":"2024-09-18T07:44:10.044014+0000","last_clean":"2024-09-18T07:44:10.044014+0000","last_became_active":"2024-09-18T07:31:02.351609+0000","last_became_peered":"2024-09-18T07:31:02.351609+0000","last_unstale":"2024-09-18T07:44:10.044014+0000","last_undegraded":"2024-09-18T07:44:10.044014+0000","last_fullsized":"2024-09-18T07:44:10.044014+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:18.963490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:18.963490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:15.234320+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"323'19","reported_seq":795,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.114088+0000","last_change":"2024-09-18T07:43:37.436589+0000","last_active":"2024-09-18T07:44:14.114088+0000","last_peered":"2024-09-18T07:44:14.114088+0000","last_clean":"2024-09-18T07:44:14.114088+0000","last_became_active":"2024-09-18T07:37:43.066208+0000","last_became_peered":"2024-09-18T07:37:43.066208+0000","last_unstale":"2024-09-18T07:44:14.114088+0000","last_undegraded":"2024-09-18T07:44:14.114088+0000","last_fullsized":"2024-09-18T07:44:14.114088+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:43:37.436336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:43:37.436336+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:39.003498+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":697,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.116000+0000","last_change":"2024-09-18T07:43:32.658023+0000","last_active":"2024-09-18T07:44:14.116000+0000","last_peered":"2024-09-18T07:44:14.116000+0000","last_clean":"2024-09-18T07:44:14.116000+0000","last_became_active":"2024-09-18T07:35:01.415261+0000","last_became_peered":"2024-09-18T07:35:01.415261+0000","last_unstale":"2024-09-18T07:44:14.116000+0000","last_undegraded":"2024-09-18T07:44:14.116000+0000","last_fullsized":"2024-09-18T07:44:14.116000+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:32.657943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:43:32.657943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:00.076443+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":474,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.119562+0000","last_change":"2024-09-18T07:42:54.744492+0000","last_active":"2024-09-18T07:44:14.119562+0000","last_peered":"2024-09-18T07:44:14.119562+0000","last_clean":"2024-09-18T07:42:51.728160+0000","last_became_active":"2024-09-18T07:42:54.744492+0000","last_became_peered":"2024-09-18T07:42:54.744492+0000","last_unstale":"2024-09-18T07:44:14.119562+0000","last_undegraded":"2024-09-18T07:44:14.119562+0000","last_fullsized":"2024-09-18T07:42:54.736952+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:05.775170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:05.775170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":702,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.114439+0000","last_change":"2024-09-18T07:42:34.361617+0000","last_active":"2024-09-18T07:44:14.114439+0000","last_peered":"2024-09-18T07:44:14.114439+0000","last_clean":"2024-09-18T07:44:14.114439+0000","last_became_active":"2024-09-18T07:37:43.067066+0000","last_became_peered":"2024-09-18T07:37:43.067066+0000","last_unstale":"2024-09-18T07:44:14.114439+0000","last_undegraded":"2024-09-18T07:44:14.114439+0000","last_fullsized":"2024-09-18T07:44:14.114439+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:34.361382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:34.361382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"323'8","reported_seq":731,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.113490+0000","last_change":"2024-09-18T07:37:43.134956+0000","last_active":"2024-09-18T07:44:14.113490+0000","last_peered":"2024-09-18T07:44:14.113490+0000","last_clean":"2024-09-18T07:44:14.113490+0000","last_became_active":"2024-09-18T07:37:43.067503+0000","last_became_peered":"2024-09-18T07:37:43.067503+0000","last_unstale":"2024-09-18T07:44:14.113490+0000","last_undegraded":"2024-09-18T07:44:14.113490+0000","last_fullsized":"2024-09-18T07:44:14.113490+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-18T07:36:40.908629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:36:40.908629+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:17.342666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":687,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.040092+0000","last_change":"2024-09-18T07:44:08.235976+0000","last_active":"2024-09-18T07:44:10.040092+0000","last_peered":"2024-09-18T07:44:10.040092+0000","last_clean":"2024-09-18T07:44:10.040092+0000","last_became_active":"2024-09-18T07:30:56.283375+0000","last_became_peered":"2024-09-18T07:30:56.283375+0000","last_unstale":"2024-09-18T07:44:10.040092+0000","last_undegraded":"2024-09-18T07:44:10.040092+0000","last_fullsized":"2024-09-18T07:44:10.040092+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:08.235926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:44:08.235926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:32.664484+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":311,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.108909+0000","last_change":"2024-09-18T07:42:54.744598+0000","last_active":"2024-09-18T07:44:14.108909+0000","last_peered":"2024-09-18T07:44:14.108909+0000","last_clean":"2024-09-18T07:42:51.729754+0000","last_became_active":"2024-09-18T07:42:54.744598+0000","last_became_peered":"2024-09-18T07:42:54.744598+0000","last_unstale":"2024-09-18T07:44:14.108909+0000","last_undegraded":"2024-09-18T07:44:14.108909+0000","last_fullsized":"2024-09-18T07:42:54.735924+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:40.106921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.106921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":460,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.119821+0000","last_change":"2024-09-18T07:42:54.751407+0000","last_active":"2024-09-18T07:44:14.119821+0000","last_peered":"2024-09-18T07:44:14.119821+0000","last_clean":"2024-09-18T07:42:51.728069+0000","last_became_active":"2024-09-18T07:42:54.751407+0000","last_became_peered":"2024-09-18T07:42:54.751407+0000","last_unstale":"2024-09-18T07:44:14.119821+0000","last_undegraded":"2024-09-18T07:44:14.119821+0000","last_fullsized":"2024-09-18T07:42:54.738929+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:38.492269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.492269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:10.437528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":364,"reported_epoch":383,"state":"active+undersized","last_fresh":"2024-09-18T07:44:14.115345+0000","last_change":"2024-09-18T07:42:54.746248+0000","last_active":"2024-09-18T07:44:14.115345+0000","last_peered":"2024-09-18T07:44:14.115345+0000","last_clean":"2024-09-18T07:42:51.721338+0000","last_became_active":"2024-09-18T07:42:54.746248+0000","last_became_peered":"2024-09-18T07:42:54.746248+0000","last_unstale":"2024-09-18T07:44:14.115345+0000","last_undegraded":"2024-09-18T07:44:14.115345+0000","last_fullsized":"2024-09-18T07:42:54.734625+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:35:02.071629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:35:02.071629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:57.442795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":706,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.114374+0000","last_change":"2024-09-18T07:44:12.888067+0000","last_active":"2024-09-18T07:44:14.114374+0000","last_peered":"2024-09-18T07:44:14.114374+0000","last_clean":"2024-09-18T07:44:14.114374+0000","last_became_active":"2024-09-18T07:30:56.292342+0000","last_became_peered":"2024-09-18T07:30:56.292342+0000","last_unstale":"2024-09-18T07:44:14.114374+0000","last_undegraded":"2024-09-18T07:44:14.114374+0000","last_fullsized":"2024-09-18T07:44:14.114374+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:44:12.887774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:44:12.887774+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:17.870374+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"323'19","reported_seq":340,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-18T07:44:14.109758+0000","last_change":"2024-09-18T07:43:10.104168+0000","last_active":"2024-09-18T07:44:14.109758+0000","last_peered":"2024-09-18T07:44:14.109758+0000","last_clean":"2024-09-18T07:44:14.109758+0000","last_became_active":"2024-09-18T07:37:43.070186+0000","last_became_peered":"2024-09-18T07:37:43.070186+0000","last_unstale":"2024-09-18T07:44:14.109758+0000","last_undegraded":"2024-09-18T07:44:14.109758+0000","last_fullsized":"2024-09-18T07:44:14.109758+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:43:10.104058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:43:10.104058+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:35.272876+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":694,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-09-18T07:44:10.042711+0000","last_change":"2024-09-18T07:41:13.842121+0000","last_active":"2024-09-18T07:44:10.042711+0000","last_peered":"2024-09-18T07:44:10.042711+0000","last_clean":"2024-09-18T07:44:10.042711+0000","last_became_active":"2024-09-18T07:35:01.410469+0000","last_became_peered":"2024-09-18T07:35:01.410469+0000","last_unstale":"2024-09-18T07:44:10.042711+0000","last_undegraded":"2024-09-18T07:44:10.042711+0000","last_fullsized":"2024-09-18T07:44:10.042711+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:13.842068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:13.842068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:49.123726+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":7043678208,"num_objects":107478,"num_object_clones":0,"num_object_copies":322434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":53570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107478,"num_write_kb":6878592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17597923328,"data_stored":17597923328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85678,"ondisk_log_size":85678,"up":40,"acting":40,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":84,"acting":84,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":81,"acting":81,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":74,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1109,"num_read_kb":866,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":4664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":685,"ondisk_log_size":685,"up":82,"acting":82,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":79,"acting":79,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":561,"num_read_kb":556,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":30,"acting":30,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201444,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2197200,"kb_used_data":1728636,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":91547952,"statfs":{"total":95995035648,"available":93745102848,"internally_reserved":0,"allocated":1770123264,"data_stored":1769666953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":479721930},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463082,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3431788,"kb_used_data":3003548,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":90313364,"statfs":{"total":95995035648,"available":92480884736,"internally_reserved":0,"allocated":3075633152,"data_stored":3075208194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":438499782},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724719,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2719040,"kb_used_data":2164464,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":91026112,"statfs":{"total":95995035648,"available":93210738688,"internally_reserved":0,"allocated":2216411136,"data_stored":2215967028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953637,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":341332,"kb_used_data":25540,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":93403820,"statfs":{"total":95995035648,"available":95645511680,"internally_reserved":0,"allocated":26152960,"data_stored":25775242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149882,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2760976,"kb_used_data":2166144,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":90984176,"statfs":{"total":95995035648,"available":93167796224,"internally_reserved":0,"allocated":2218131456,"data_stored":2217672168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":609089991},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411520,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3052732,"kb_used_data":2584236,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":90692420,"statfs":{"total":95995035648,"available":92869038080,"internally_reserved":0,"allocated":2646257664,"data_stored":2645817551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575047,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1862744,"kb_used_data":1285960,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":91882408,"statfs":{"total":95995035648,"available":94087585792,"internally_reserved":0,"allocated":1316823040,"data_stored":1316355241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738571,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4704776,"kb_used_data":4136312,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":89040376,"statfs":{"total":95995035648,"available":91177345024,"internally_reserved":0,"allocated":4235583488,"data_stored":4235125215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4264624128,"data_stored":4264624128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1326579712,"data_stored":1326579712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2670526464,"data_stored":2670526464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2233466880,"data_stored":2233466880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2236481536,"data_stored":2236481536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3093954560,"data_stored":3093954560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1772290048,"data_stored":1772290048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:44:21.281 INFO:tasks.thrashosds.thrasher:Setting 5.1e to [7, 1, 4] 2024-09-18T07:44:21.281 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.1e', '7', '1', '4'] 2024-09-18T07:44:21.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pg-upmap 5.1e 7 1 4 2024-09-18T07:44:21.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:21.285 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:21.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:21.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:21.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:21 smithi137 ceph-mon[29408]: pgmap v701: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.32k op/s; 54287/325932 objects degraded (16.656%) 2024-09-18T07:44:21.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:21 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:44:21.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:21 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:44:21.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:21 smithi137 ceph-mon[39175]: pgmap v701: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.32k op/s; 54287/325932 objects degraded (16.656%) 2024-09-18T07:44:21.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:21 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:44:21.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:21 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:44:21.424 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:21.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:21.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:21.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:21 smithi203 ceph-mon[32237]: pgmap v701: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.32k op/s; 54287/325932 objects degraded (16.656%) 2024-09-18T07:44:21.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:21 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:44:21.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:21 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:44:21.526 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:21.556 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:21.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:21.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:21.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:21.701 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:21.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:21.802 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:21.832 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:21.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:21.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:21.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:21.972 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:21.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:22.038 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T07:44:22.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:22.076 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 72 16 114218 114202 99.1007 77.1875 0.0112266 0.0100862 2024-09-18T07:44:22.111 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:22.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:22.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:22.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:22.269 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:22.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:22.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:22.370 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:22.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:22.404 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:22.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:22.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:22 smithi137 ceph-mon[29408]: osdmap e386: 8 total, 7 up, 8 in 2024-09-18T07:44:22.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:22 smithi137 ceph-mon[39175]: osdmap e386: 8 total, 7 up, 8 in 2024-09-18T07:44:22.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:22 smithi203 ceph-mon[32237]: osdmap e386: 8 total, 7 up, 8 in 2024-09-18T07:44:22.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:22.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:22.538 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:22.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:22.619 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:44:22.619 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:44:22.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:22.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:22.687 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:22.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:22.744 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:22.757 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:22.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:22.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:22.847 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:22.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:22.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:22.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:22.988 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:22.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:23.076 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 73 16 115443 115427 98.792 76.5625 0.0122903 0.0101179 2024-09-18T07:44:23.089 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:23.118 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:23.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:23.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:23.220 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:23.253 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:23.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:23.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:23.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:23.385 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:23.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:23.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:23.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:23 smithi137 ceph-mon[29408]: pgmap v704: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 62 MiB/s wr, 987 op/s; 55038/328185 objects degraded (16.770%) 2024-09-18T07:44:23.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:23 smithi137 ceph-mon[39175]: pgmap v704: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 62 MiB/s wr, 987 op/s; 55038/328185 objects degraded (16.770%) 2024-09-18T07:44:23.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:23 smithi203 ceph-mon[32237]: pgmap v704: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 6.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 62 MiB/s wr, 987 op/s; 55038/328185 objects degraded (16.770%) 2024-09-18T07:44:23.487 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:23.513 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:23.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:23.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:23.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:23.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:23.658 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:23.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:23.721 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:44:23.721 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:44:23.759 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:23.783 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:23.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:23.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:23.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:23.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:23.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:24.012 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:44:24.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:24.077 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 74 16 116643 116627 98.4697 75 0.0175218 0.0101512 2024-09-18T07:44:24.105 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:24.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:24.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:24.183 INFO:teuthology.orchestra.run.smithi137.stderr:set 5.1e pg_upmap mapping to [7,1,4] 2024-09-18T07:44:24.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:24.228 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:24.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:24.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:24.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:24 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/521122162' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.1e", "id": [7, 1, 4]} : dispatch 2024-09-18T07:44:24.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:24 smithi137 ceph-mon[29408]: from='client.26209 ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.1e", "id": [7, 1, 4]} : dispatch 2024-09-18T07:44:24.345 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:24 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/521122162' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.1e", "id": [7, 1, 4]} : dispatch 2024-09-18T07:44:24.345 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:24 smithi137 ceph-mon[39175]: from='client.26209 ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.1e", "id": [7, 1, 4]} : dispatch 2024-09-18T07:44:24.376 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:24.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:24.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:24.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:24 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/521122162' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.1e", "id": [7, 1, 4]} : dispatch 2024-09-18T07:44:24.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:24 smithi203 ceph-mon[32237]: from='client.26209 ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.1e", "id": [7, 1, 4]} : dispatch 2024-09-18T07:44:24.478 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:24.513 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:24.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:24.615 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:24.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:44:24.644 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:24.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:24.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:24.746 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:24.797 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:24.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:24.840 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T07:44:24.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:24.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:24.965 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:24.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:25.066 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:25.077 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 75 15 117738 117723 98.0702 68.5 0.0195755 0.0101911 2024-09-18T07:44:25.093 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:25.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:25.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:25.194 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:25.222 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:25.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:25.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:25.323 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:25.353 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:25.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:25.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:25 smithi137 ceph-mon[29408]: pgmap v705: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.12k op/s; 57121/343155 objects degraded (16.646%) 2024-09-18T07:44:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:25 smithi137 ceph-mon[29408]: from='client.26209 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1e", "id": [7, 1, 4]}]': finished 2024-09-18T07:44:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:25 smithi137 ceph-mon[29408]: osdmap e387: 8 total, 7 up, 8 in 2024-09-18T07:44:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:25 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:44:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:25 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:44:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:25 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 57121/343155 objects degraded (16.646%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:25 smithi137 ceph-mon[29408]: osdmap e388: 8 total, 7 up, 8 in 2024-09-18T07:44:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:25 smithi137 ceph-mon[39175]: pgmap v705: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.12k op/s; 57121/343155 objects degraded (16.646%) 2024-09-18T07:44:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:25 smithi137 ceph-mon[39175]: from='client.26209 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1e", "id": [7, 1, 4]}]': finished 2024-09-18T07:44:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:25 smithi137 ceph-mon[39175]: osdmap e387: 8 total, 7 up, 8 in 2024-09-18T07:44:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:25 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:44:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:25 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:44:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:25 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 57121/343155 objects degraded (16.646%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:25 smithi137 ceph-mon[39175]: osdmap e388: 8 total, 7 up, 8 in 2024-09-18T07:44:25.456 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:25.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:25 smithi203 ceph-mon[32237]: pgmap v705: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.12k op/s; 57121/343155 objects degraded (16.646%) 2024-09-18T07:44:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:25 smithi203 ceph-mon[32237]: from='client.26209 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1e", "id": [7, 1, 4]}]': finished 2024-09-18T07:44:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:25 smithi203 ceph-mon[32237]: osdmap e387: 8 total, 7 up, 8 in 2024-09-18T07:44:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:25 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:44:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:25 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:44:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:25 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 57121/343155 objects degraded (16.646%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:25 smithi203 ceph-mon[32237]: osdmap e388: 8 total, 7 up, 8 in 2024-09-18T07:44:25.484 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:25.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:25.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:25.616 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:25.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:25.638 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:25.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:25.676 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:25.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:25.790 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:25.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:25.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:25.892 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:25.916 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:25.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:25.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:25.940 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:26.018 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:26.064 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:26.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:26.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:26.081 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 76 16 118946 118930 97.7696 75.4375 0.0320624 0.0102239 2024-09-18T07:44:26.166 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:26.194 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:26.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:26.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:26.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:26.328 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:26.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:26.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:26.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:26.470 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:26.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:26.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:26.572 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:26.616 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:26.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:26.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:26.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:26.762 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:26.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:26.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:26.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:26.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:26.909 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:26.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:26.932 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:44:26.932 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:44:26.932 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:44:26.932 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:44:26.933 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:44:26.933 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:44:26.933 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:44:26.933 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:44:26.933 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:44:26.933 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:44:26.933 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:44:26.933 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-18T07:44:26.933 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:44:26.933 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:44:26.933 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:44:27.010 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:27.038 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:27.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:27.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:27.079 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 77 16 120090 120074 97.4284 71.5 0.0079572 0.0102594 2024-09-18T07:44:27.139 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:27.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:44:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:44:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:44:27.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:27 smithi137 ceph-mon[29408]: pgmap v708: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 898 B/s rd, 63 MiB/s wr, 1.01k op/s; 57121/343155 objects degraded (16.646%) 2024-09-18T07:44:27.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:27 smithi137 ceph-mon[29408]: osdmap e389: 8 total, 7 up, 8 in 2024-09-18T07:44:27.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:27 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/998258333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:27.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:27 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1232267004' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:44:27.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:27 smithi137 ceph-mon[29408]: from='client.26215 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:44:27.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:27 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:44:27.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:27 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:44:27.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:27 smithi137 ceph-mon[39175]: pgmap v708: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 898 B/s rd, 63 MiB/s wr, 1.01k op/s; 57121/343155 objects degraded (16.646%) 2024-09-18T07:44:27.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:27 smithi137 ceph-mon[39175]: osdmap e389: 8 total, 7 up, 8 in 2024-09-18T07:44:27.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:27 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/998258333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:27.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:27 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1232267004' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:44:27.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:27 smithi137 ceph-mon[39175]: from='client.26215 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:44:27.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:27 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:44:27.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:27 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:44:27.181 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:27.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:27.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:27.202 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:44:27.283 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:27.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:27.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:27.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:27.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:27 smithi203 ceph-mon[32237]: pgmap v708: 158 pgs: 39 active+undersized, 30 active+undersized+degraded, 89 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 898 B/s rd, 63 MiB/s wr, 1.01k op/s; 57121/343155 objects degraded (16.646%) 2024-09-18T07:44:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:27 smithi203 ceph-mon[32237]: osdmap e389: 8 total, 7 up, 8 in 2024-09-18T07:44:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:27 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/998258333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:27.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:27 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1232267004' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:44:27.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:27 smithi203 ceph-mon[32237]: from='client.26215 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:44:27.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:27 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:44:27.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:27 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:44:27.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:27.464 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:27.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:27.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:27.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:27.605 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:27.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:27.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:27.740 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:27.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:27.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:27.799 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:44:27.799 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:44:27.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:27.881 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:27.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:27.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:27.960 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:44:27 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:44:27.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:28.019 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:28.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:28.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:28.079 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 78 16 121386 121370 97.2178 81 0.0107684 0.0102811 2024-09-18T07:44:28.121 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:28.149 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:28.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:28.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:28.251 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:28.285 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:28.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:28.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:28.387 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:28.416 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:28.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:28.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:28.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:28 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:44:28.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:28 smithi137 ceph-mon[29408]: from='client.26215 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:44:28.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:28 smithi137 ceph-mon[29408]: osdmap e390: 8 total, 7 up, 8 in 2024-09-18T07:44:28.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:28 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:44:28.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:28 smithi137 ceph-mon[39175]: from='client.26215 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:44:28.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:28 smithi137 ceph-mon[39175]: osdmap e390: 8 total, 7 up, 8 in 2024-09-18T07:44:28.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:28 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:44:28.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:28 smithi203 ceph-mon[32237]: from='client.26215 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:44:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:28 smithi203 ceph-mon[32237]: osdmap e390: 8 total, 7 up, 8 in 2024-09-18T07:44:28.518 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:28.543 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:28.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:28.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:28.645 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:28.669 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:28.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:28.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:28.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:28.805 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:44:28.805 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:44:28.806 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:44:28.806 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:44:28.806 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:44:28.809 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:28.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:28.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:28.911 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:28.945 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:28.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:28.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:29.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:29.078 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:29.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:29.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:29.080 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 79 16 122690 122674 97.0188 81.5 0.00819724 0.0103029 2024-09-18T07:44:29.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:29.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:29.221 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:29.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:29.323 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:29.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:29.359 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:29.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:29.460 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:29 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:29 smithi203 ceph-mon[32237]: pgmap v711: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 59355/357513 objects degraded (16.602%) 2024-09-18T07:44:29.496 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:29.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:29.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:29.618 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:29 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:29.618 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:29 smithi137 ceph-mon[29408]: pgmap v711: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 59355/357513 objects degraded (16.602%) 2024-09-18T07:44:29.620 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:29 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:29.620 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:29 smithi137 ceph-mon[39175]: pgmap v711: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 59355/357513 objects degraded (16.602%) 2024-09-18T07:44:29.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:29.638 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:29.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:29.668 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T07:44:29.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:29.770 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:29.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:29.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:29.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:44:29.872 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:29.899 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:29.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:29.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:30.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:30.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:30.074 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:30.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:30.080 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:44:30.078596+0000 min lat: 0.00105851 max lat: 0.247818 avg lat: 0.0103303 2024-09-18T07:44:30.080 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:44:30.080 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 80 16 123908 123892 96.7573 76.125 0.0206525 0.0103303 2024-09-18T07:44:30.176 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:30.182 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:30.203 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:30.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:30.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:30.305 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:30.343 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:30.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:30.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:30.445 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:30.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:30 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 59355/357513 objects degraded (16.602%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:30.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:30 smithi203 ceph-mon[32237]: osdmap e391: 8 total, 7 up, 8 in 2024-09-18T07:44:30.485 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:30.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:30.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:30.487 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:30 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 59355/357513 objects degraded (16.602%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:30.487 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:30 smithi137 ceph-mon[29408]: osdmap e391: 8 total, 7 up, 8 in 2024-09-18T07:44:30.487 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:30 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 59355/357513 objects degraded (16.602%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:30.488 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:30 smithi137 ceph-mon[39175]: osdmap e391: 8 total, 7 up, 8 in 2024-09-18T07:44:30.515 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-09-18T07:44:30.515 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:44:30.515 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-09-18T07:44:30.515 INFO:tasks.cephadm.osd.4:Restarting osd.4 (starting--it wasn't running)... 2024-09-18T07:44:30.516 DEBUG:teuthology.orchestra.run.smithi203:osd.4> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.4.service 2024-09-18T07:44:30.518 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl start ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.4 2024-09-18T07:44:30.588 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:30.650 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:30.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:30.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:30.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:44:30.752 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:30.769 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:30.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:30.796 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:30.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:30.824 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:44:30.825 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:44:30.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:44:30.825 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:44:30.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.825 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162316 7.f 7:f95a8250:::benchmark_data_smithi137_74180_object57900:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:44:30.826 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.409395+0000", 2024-09-18T07:44:30.826 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 569.40378063399999, 2024-09-18T07:44:30.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0987874790000001, 2024-09-18T07:44:30.826 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.826 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.826 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.827 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:30.827 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:44:30.827 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:30.827 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162316 2024-09-18T07:44:30.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.827 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409395+0000", 2024-09-18T07:44:30.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:30.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409395+0000", 2024-09-18T07:44:30.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:30.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409397+0000", 2024-09-18T07:44:30.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3429999999999998e-06 2024-09-18T07:44:30.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:30.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409404+0000", 2024-09-18T07:44:30.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2600000000000002e-06 2024-09-18T07:44:30.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409418+0000", 2024-09-18T07:44:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3983e-05 2024-09-18T07:44:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409427+0000", 2024-09-18T07:44:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9269999999999994e-06 2024-09-18T07:44:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409490+0000", 2024-09-18T07:44:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2868999999999997e-05 2024-09-18T07:44:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.409500+0000", 2024-09-18T07:44:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0621e-05 2024-09-18T07:44:30.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.497693+0000", 2024-09-18T07:44:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.088192405 2024-09-18T07:44:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:30.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.497816+0000", 2024-09-18T07:44:30.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012321500000000001 2024-09-18T07:44:30.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:44:30.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.497964+0000", 2024-09-18T07:44:30.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000148169 2024-09-18T07:44:30.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.500777+0000", 2024-09-18T07:44:30.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028126750000000002 2024-09-18T07:44:30.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.500835+0000", 2024-09-18T07:44:30.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8727000000000002e-05 2024-09-18T07:44:30.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.500878+0000", 2024-09-18T07:44:30.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2682000000000002e-05 2024-09-18T07:44:30.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:30.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507887+0000", 2024-09-18T07:44:30.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0070087789999999997 2024-09-18T07:44:30.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:30.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508067+0000", 2024-09-18T07:44:30.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017973300000000001 2024-09-18T07:44:30.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:30.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508183+0000", 2024-09-18T07:44:30.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116091 2024-09-18T07:44:30.840 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.840 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:30.840 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.841 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162322 7.f 7:f41da5a4:::benchmark_data_smithi137_74180_object57906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:44:30.841 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.420348+0000", 2024-09-18T07:44:30.841 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 569.39282734100004, 2024-09-18T07:44:30.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0968331140000001, 2024-09-18T07:44:30.841 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.841 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.841 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.842 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:30.842 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:44:30.842 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:30.842 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162322 2024-09-18T07:44:30.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.842 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420348+0000", 2024-09-18T07:44:30.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:30.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420348+0000", 2024-09-18T07:44:30.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:30.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420350+0000", 2024-09-18T07:44:30.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8670000000000001e-06 2024-09-18T07:44:30.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:30.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420356+0000", 2024-09-18T07:44:30.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0560000000000001e-06 2024-09-18T07:44:30.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:30.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420369+0000", 2024-09-18T07:44:30.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2507000000000001e-05 2024-09-18T07:44:30.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:30.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420376+0000", 2024-09-18T07:44:30.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.17e-06 2024-09-18T07:44:30.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420444+0000", 2024-09-18T07:44:30.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7801999999999999e-05 2024-09-18T07:44:30.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:30.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420454+0000", 2024-09-18T07:44:30.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0258000000000001e-05 2024-09-18T07:44:30.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.501094+0000", 2024-09-18T07:44:30.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0806402799999999 2024-09-18T07:44:30.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:30.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.501227+0000", 2024-09-18T07:44:30.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013271600000000001 2024-09-18T07:44:30.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:44:30.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.501467+0000", 2024-09-18T07:44:30.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024032599999999999 2024-09-18T07:44:30.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507729+0000", 2024-09-18T07:44:30.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0062618220000000002 2024-09-18T07:44:30.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507768+0000", 2024-09-18T07:44:30.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8668000000000003e-05 2024-09-18T07:44:30.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:30.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.516751+0000", 2024-09-18T07:44:30.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0089832390000000005 2024-09-18T07:44:30.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.516888+0000", 2024-09-18T07:44:30.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137372 2024-09-18T07:44:30.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:30.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517069+0000", 2024-09-18T07:44:30.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000180574 2024-09-18T07:44:30.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:30.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517182+0000", 2024-09-18T07:44:30.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112457 2024-09-18T07:44:30.850 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.850 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:30.850 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.850 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162323 7.f 7:fe31c7fa:::benchmark_data_smithi137_74180_object57907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:44:30.850 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.420396+0000", 2024-09-18T07:44:30.850 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 569.39277987000003, 2024-09-18T07:44:30.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.097250378, 2024-09-18T07:44:30.850 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.850 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.850 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.851 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:30.851 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:44:30.851 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:30.851 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162323 2024-09-18T07:44:30.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.851 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420396+0000", 2024-09-18T07:44:30.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:30.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420396+0000", 2024-09-18T07:44:30.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:30.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420396+0000", 2024-09-18T07:44:30.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9400000000000005e-07 2024-09-18T07:44:30.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:30.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420399+0000", 2024-09-18T07:44:30.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6879999999999999e-06 2024-09-18T07:44:30.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:30.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420404+0000", 2024-09-18T07:44:30.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2680000000000002e-06 2024-09-18T07:44:30.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:30.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420411+0000", 2024-09-18T07:44:30.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2519999999999996e-06 2024-09-18T07:44:30.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420540+0000", 2024-09-18T07:44:30.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012886899999999999 2024-09-18T07:44:30.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:30.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420547+0000", 2024-09-18T07:44:30.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9019999999999999e-06 2024-09-18T07:44:30.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.504020+0000", 2024-09-18T07:44:30.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0834720959999999 2024-09-18T07:44:30.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:30.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.504135+0000", 2024-09-18T07:44:30.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115697 2024-09-18T07:44:30.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:44:30.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.504251+0000", 2024-09-18T07:44:30.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116041 2024-09-18T07:44:30.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507805+0000", 2024-09-18T07:44:30.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035537749999999999 2024-09-18T07:44:30.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.507870+0000", 2024-09-18T07:44:30.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4558000000000001e-05 2024-09-18T07:44:30.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:30.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.516761+0000", 2024-09-18T07:44:30.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0088909690000000003 2024-09-18T07:44:30.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517414+0000", 2024-09-18T07:44:30.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00065372100000000001 2024-09-18T07:44:30.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:30.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517543+0000", 2024-09-18T07:44:30.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012907199999999999 2024-09-18T07:44:30.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:30.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517646+0000", 2024-09-18T07:44:30.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102876 2024-09-18T07:44:30.858 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.858 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:30.858 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.858 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162327 7.f 7:fac5d155:::benchmark_data_smithi137_74180_object57911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:44:30.858 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.420507+0000", 2024-09-18T07:44:30.858 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 569.39266886200005, 2024-09-18T07:44:30.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.097389145, 2024-09-18T07:44:30.858 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.859 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.859 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.859 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:30.859 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:44:30.859 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:30.859 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162327 2024-09-18T07:44:30.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.859 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420507+0000", 2024-09-18T07:44:30.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:30.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420507+0000", 2024-09-18T07:44:30.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:30.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420508+0000", 2024-09-18T07:44:30.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7300000000000005e-07 2024-09-18T07:44:30.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:30.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420511+0000", 2024-09-18T07:44:30.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4089999999999999e-06 2024-09-18T07:44:30.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:30.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420518+0000", 2024-09-18T07:44:30.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1010000000000003e-06 2024-09-18T07:44:30.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:30.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420523+0000", 2024-09-18T07:44:30.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2089999999999996e-06 2024-09-18T07:44:30.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420570+0000", 2024-09-18T07:44:30.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6638000000000003e-05 2024-09-18T07:44:30.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:30.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.420577+0000", 2024-09-18T07:44:30.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2679999999999997e-06 2024-09-18T07:44:30.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508213+0000", 2024-09-18T07:44:30.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0876354130000001 2024-09-18T07:44:30.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:30.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508312+0000", 2024-09-18T07:44:30.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9124999999999997e-05 2024-09-18T07:44:30.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:44:30.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.508859+0000", 2024-09-18T07:44:30.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00054700599999999995 2024-09-18T07:44:30.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.516816+0000", 2024-09-18T07:44:30.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079569410000000004 2024-09-18T07:44:30.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:30.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517264+0000", 2024-09-18T07:44:30.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044864999999999998 2024-09-18T07:44:30.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517309+0000", 2024-09-18T07:44:30.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5033000000000001e-05 2024-09-18T07:44:30.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517755+0000", 2024-09-18T07:44:30.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044556899999999999 2024-09-18T07:44:30.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:30.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517803+0000", 2024-09-18T07:44:30.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7584999999999999e-05 2024-09-18T07:44:30.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:30.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517896+0000", 2024-09-18T07:44:30.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3424999999999994e-05 2024-09-18T07:44:30.866 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.866 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:30.866 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.866 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162336 7.f 7:f2e86194:::benchmark_data_smithi137_74180_object57920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:44:30.866 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.425606+0000", 2024-09-18T07:44:30.866 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 569.38756965599998, 2024-09-18T07:44:30.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0924705029999999, 2024-09-18T07:44:30.867 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.867 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.867 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.867 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:30.867 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:44:30.867 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:30.867 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162336 2024-09-18T07:44:30.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.868 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425606+0000", 2024-09-18T07:44:30.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:30.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425606+0000", 2024-09-18T07:44:30.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:30.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425608+0000", 2024-09-18T07:44:30.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2259999999999999e-06 2024-09-18T07:44:30.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:30.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425614+0000", 2024-09-18T07:44:30.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0549999999999996e-06 2024-09-18T07:44:30.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:30.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425626+0000", 2024-09-18T07:44:30.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1600000000000001e-05 2024-09-18T07:44:30.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:30.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425635+0000", 2024-09-18T07:44:30.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0440000000000001e-06 2024-09-18T07:44:30.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425680+0000", 2024-09-18T07:44:30.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5015999999999997e-05 2024-09-18T07:44:30.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:30.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.425689+0000", 2024-09-18T07:44:30.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3419999999999997e-06 2024-09-18T07:44:30.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.513184+0000", 2024-09-18T07:44:30.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0874942320000001 2024-09-18T07:44:30.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:30.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.513298+0000", 2024-09-18T07:44:30.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114333 2024-09-18T07:44:30.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:44:30.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.513425+0000", 2024-09-18T07:44:30.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126711 2024-09-18T07:44:30.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:30.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517321+0000", 2024-09-18T07:44:30.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038961590000000002 2024-09-18T07:44:30.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517374+0000", 2024-09-18T07:44:30.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3003000000000001e-05 2024-09-18T07:44:30.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517718+0000", 2024-09-18T07:44:30.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034425100000000001 2024-09-18T07:44:30.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517935+0000", 2024-09-18T07:44:30.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021721899999999999 2024-09-18T07:44:30.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:30.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.517986+0000", 2024-09-18T07:44:30.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0627999999999997e-05 2024-09-18T07:44:30.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:30.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.518077+0000", 2024-09-18T07:44:30.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0684000000000001e-05 2024-09-18T07:44:30.874 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.874 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:30.874 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.875 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162345 7.f 7:f017fa1c:::benchmark_data_smithi137_74180_object57929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:44:30.875 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.431398+0000", 2024-09-18T07:44:30.875 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 569.38177816899997, 2024-09-18T07:44:30.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0972635669999999, 2024-09-18T07:44:30.875 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.875 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.875 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.875 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:30.875 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:44:30.875 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:30.875 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162345 2024-09-18T07:44:30.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.875 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431398+0000", 2024-09-18T07:44:30.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:30.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431398+0000", 2024-09-18T07:44:30.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:30.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431399+0000", 2024-09-18T07:44:30.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.361e-06 2024-09-18T07:44:30.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:30.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431403+0000", 2024-09-18T07:44:30.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1520000000000002e-06 2024-09-18T07:44:30.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:30.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431414+0000", 2024-09-18T07:44:30.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1239e-05 2024-09-18T07:44:30.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:30.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431421+0000", 2024-09-18T07:44:30.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.99e-06 2024-09-18T07:44:30.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431537+0000", 2024-09-18T07:44:30.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115278 2024-09-18T07:44:30.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:30.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431547+0000", 2024-09-18T07:44:30.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0543999999999999e-05 2024-09-18T07:44:30.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.518110+0000", 2024-09-18T07:44:30.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.086562942 2024-09-18T07:44:30.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:30.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.518220+0000", 2024-09-18T07:44:30.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010951 2024-09-18T07:44:30.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:44:30.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.518334+0000", 2024-09-18T07:44:30.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114057 2024-09-18T07:44:30.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.527942+0000", 2024-09-18T07:44:30.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0096087029999999997 2024-09-18T07:44:30.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528087+0000", 2024-09-18T07:44:30.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000144871 2024-09-18T07:44:30.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:30.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528103+0000", 2024-09-18T07:44:30.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5531000000000001e-05 2024-09-18T07:44:30.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528148+0000", 2024-09-18T07:44:30.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5667000000000002e-05 2024-09-18T07:44:30.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:30.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528542+0000", 2024-09-18T07:44:30.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00039349200000000001 2024-09-18T07:44:30.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:30.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.528661+0000", 2024-09-18T07:44:30.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011923 2024-09-18T07:44:30.888 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.888 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:30.888 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.889 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202539 7.f 7:fdd02dfa:::benchmark_data_smithi137_74180_object98123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:44:30.889 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.829809+0000", 2024-09-18T07:44:30.889 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 532.983366499, 2024-09-18T07:44:30.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0954651049999999, 2024-09-18T07:44:30.889 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.890 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.890 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.890 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:30.890 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:44:30.890 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:30.890 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202539 2024-09-18T07:44:30.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.891 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829809+0000", 2024-09-18T07:44:30.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:30.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829809+0000", 2024-09-18T07:44:30.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:30.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829811+0000", 2024-09-18T07:44:30.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9360000000000002e-06 2024-09-18T07:44:30.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:30.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829816+0000", 2024-09-18T07:44:30.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6979999999999997e-06 2024-09-18T07:44:30.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:30.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829824+0000", 2024-09-18T07:44:30.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6799999999999993e-06 2024-09-18T07:44:30.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:30.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829829+0000", 2024-09-18T07:44:30.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9159999999999998e-06 2024-09-18T07:44:30.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829890+0000", 2024-09-18T07:44:30.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0643000000000002e-05 2024-09-18T07:44:30.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:30.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829898+0000", 2024-09-18T07:44:30.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.576e-06 2024-09-18T07:44:30.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.917838+0000", 2024-09-18T07:44:30.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.087940012 2024-09-18T07:44:30.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:30.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.917970+0000", 2024-09-18T07:44:30.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013214499999999999 2024-09-18T07:44:30.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:44:30.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.918095+0000", 2024-09-18T07:44:30.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012521 2024-09-18T07:44:30.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:30.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.921001+0000", 2024-09-18T07:44:30.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029057789999999998 2024-09-18T07:44:30.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.921054+0000", 2024-09-18T07:44:30.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3038999999999998e-05 2024-09-18T07:44:30.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925195+0000", 2024-09-18T07:44:30.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004140632 2024-09-18T07:44:30.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:30.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925229+0000", 2024-09-18T07:44:30.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4783000000000002e-05 2024-09-18T07:44:30.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:30.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925274+0000", 2024-09-18T07:44:30.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5055999999999997e-05 2024-09-18T07:44:30.900 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.900 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:30.900 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.900 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202541 7.f 7:f489b710:::benchmark_data_smithi137_74180_object98125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:44:30.900 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.829844+0000", 2024-09-18T07:44:30.900 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 532.98333131799995, 2024-09-18T07:44:30.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0983412960000001, 2024-09-18T07:44:30.900 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.901 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.901 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.901 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:30.901 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:44:30.901 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:30.901 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202541 2024-09-18T07:44:30.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.901 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829844+0000", 2024-09-18T07:44:30.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:30.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829844+0000", 2024-09-18T07:44:30.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:30.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829845+0000", 2024-09-18T07:44:30.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0100000000000004e-07 2024-09-18T07:44:30.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:30.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829848+0000", 2024-09-18T07:44:30.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0170000000000001e-06 2024-09-18T07:44:30.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:30.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829854+0000", 2024-09-18T07:44:30.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9529999999999996e-06 2024-09-18T07:44:30.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:30.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829859+0000", 2024-09-18T07:44:30.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1499999999999998e-06 2024-09-18T07:44:30.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.830023+0000", 2024-09-18T07:44:30.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000163473 2024-09-18T07:44:30.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:30.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.830030+0000", 2024-09-18T07:44:30.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.396e-06 2024-09-18T07:44:30.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.921101+0000", 2024-09-18T07:44:30.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0910708730000001 2024-09-18T07:44:30.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:30.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.921187+0000", 2024-09-18T07:44:30.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6408000000000003e-05 2024-09-18T07:44:30.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:44:30.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.921265+0000", 2024-09-18T07:44:30.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8019e-05 2024-09-18T07:44:30.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925399+0000", 2024-09-18T07:44:30.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004133205 2024-09-18T07:44:30.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:30.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925470+0000", 2024-09-18T07:44:30.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1420000000000004e-05 2024-09-18T07:44:30.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928114+0000", 2024-09-18T07:44:30.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026441580000000002 2024-09-18T07:44:30.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:30.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928155+0000", 2024-09-18T07:44:30.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0620000000000001e-05 2024-09-18T07:44:30.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:30.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928186+0000", 2024-09-18T07:44:30.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0902999999999999e-05 2024-09-18T07:44:30.908 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.908 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:30.908 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.908 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202543 7.f 7:f02d8d75:::benchmark_data_smithi137_74180_object98127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:44:30.909 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.829915+0000", 2024-09-18T07:44:30.909 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 532.98326037000004, 2024-09-18T07:44:30.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.098406115, 2024-09-18T07:44:30.909 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.909 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.909 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.909 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:30.909 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:44:30.909 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:30.909 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202543 2024-09-18T07:44:30.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.909 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829915+0000", 2024-09-18T07:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829915+0000", 2024-09-18T07:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829916+0000", 2024-09-18T07:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3900000000000003e-07 2024-09-18T07:44:30.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:30.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829921+0000", 2024-09-18T07:44:30.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2250000000000002e-06 2024-09-18T07:44:30.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:30.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829927+0000", 2024-09-18T07:44:30.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3600000000000001e-06 2024-09-18T07:44:30.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:30.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.829933+0000", 2024-09-18T07:44:30.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0240000000000003e-06 2024-09-18T07:44:30.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.830051+0000", 2024-09-18T07:44:30.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118131 2024-09-18T07:44:30.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:30.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.830057+0000", 2024-09-18T07:44:30.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4929999999999998e-06 2024-09-18T07:44:30.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.923373+0000", 2024-09-18T07:44:30.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.093316306 2024-09-18T07:44:30.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:30.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.923465+0000", 2024-09-18T07:44:30.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2260000000000001e-05 2024-09-18T07:44:30.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:44:30.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.923543+0000", 2024-09-18T07:44:30.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7601999999999993e-05 2024-09-18T07:44:30.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925460+0000", 2024-09-18T07:44:30.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001916846 2024-09-18T07:44:30.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:30.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925474+0000", 2024-09-18T07:44:30.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4708999999999999e-05 2024-09-18T07:44:30.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928254+0000", 2024-09-18T07:44:30.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002779662 2024-09-18T07:44:30.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:30.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928293+0000", 2024-09-18T07:44:30.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9419000000000003e-05 2024-09-18T07:44:30.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:30.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928321+0000", 2024-09-18T07:44:30.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8138999999999999e-05 2024-09-18T07:44:30.915 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.915 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:30.916 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.916 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202553 7.f 7:f853e395:::benchmark_data_smithi137_74180_object98137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:44:30.916 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.836313+0000", 2024-09-18T07:44:30.916 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 532.97686266599999, 2024-09-18T07:44:30.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.099117246, 2024-09-18T07:44:30.916 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.916 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.916 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.916 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:30.916 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:44:30.916 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:30.917 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202553 2024-09-18T07:44:30.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.917 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836313+0000", 2024-09-18T07:44:30.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:30.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836313+0000", 2024-09-18T07:44:30.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:30.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836314+0000", 2024-09-18T07:44:30.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3519999999999999e-06 2024-09-18T07:44:30.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:30.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836319+0000", 2024-09-18T07:44:30.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6469999999999997e-06 2024-09-18T07:44:30.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:30.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836329+0000", 2024-09-18T07:44:30.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0356e-05 2024-09-18T07:44:30.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:30.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836337+0000", 2024-09-18T07:44:30.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4290000000000004e-06 2024-09-18T07:44:30.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836372+0000", 2024-09-18T07:44:30.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5577000000000002e-05 2024-09-18T07:44:30.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:30.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.836377+0000", 2024-09-18T07:44:30.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7199999999999997e-06 2024-09-18T07:44:30.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925531+0000", 2024-09-18T07:44:30.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0891534789999999 2024-09-18T07:44:30.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:30.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925628+0000", 2024-09-18T07:44:30.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7720999999999995e-05 2024-09-18T07:44:30.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:44:30.922 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.925719+0000", 2024-09-18T07:44:30.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0804999999999995e-05 2024-09-18T07:44:30.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.922 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928378+0000", 2024-09-18T07:44:30.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026585670000000001 2024-09-18T07:44:30.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.922 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935204+0000", 2024-09-18T07:44:30.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068259189999999997 2024-09-18T07:44:30.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:30.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935218+0000", 2024-09-18T07:44:30.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4562000000000001e-05 2024-09-18T07:44:30.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.923 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:30.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935381+0000", 2024-09-18T07:44:30.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000162473 2024-09-18T07:44:30.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.923 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:30.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935430+0000", 2024-09-18T07:44:30.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9638999999999997e-05 2024-09-18T07:44:30.923 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.924 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:30.924 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.924 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202576 7.f 7:f8260e72:::benchmark_data_smithi137_74180_object98160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:44:30.924 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.852688+0000", 2024-09-18T07:44:30.924 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 532.96048733299995, 2024-09-18T07:44:30.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.082910424, 2024-09-18T07:44:30.924 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.924 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.924 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.924 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:30.924 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:44:30.925 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:30.925 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202576 2024-09-18T07:44:30.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.925 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852688+0000", 2024-09-18T07:44:30.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:30.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852688+0000", 2024-09-18T07:44:30.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:30.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852690+0000", 2024-09-18T07:44:30.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5039999999999999e-06 2024-09-18T07:44:30.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:30.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852694+0000", 2024-09-18T07:44:30.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4889999999999997e-06 2024-09-18T07:44:30.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:30.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852704+0000", 2024-09-18T07:44:30.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1689999999999998e-06 2024-09-18T07:44:30.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:30.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852709+0000", 2024-09-18T07:44:30.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6189999999999999e-06 2024-09-18T07:44:30.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852757+0000", 2024-09-18T07:44:30.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7676000000000002e-05 2024-09-18T07:44:30.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:30.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.852765+0000", 2024-09-18T07:44:30.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9400000000000002e-06 2024-09-18T07:44:30.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928432+0000", 2024-09-18T07:44:30.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.075667572 2024-09-18T07:44:30.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:30.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928561+0000", 2024-09-18T07:44:30.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012874199999999999 2024-09-18T07:44:30.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.929 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:44:30.929 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.928674+0000", 2024-09-18T07:44:30.929 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011271400000000001 2024-09-18T07:44:30.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.929 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.929 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935474+0000", 2024-09-18T07:44:30.929 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068001270000000004 2024-09-18T07:44:30.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.929 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:30.929 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935484+0000", 2024-09-18T07:44:30.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0515e-05 2024-09-18T07:44:30.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935518+0000", 2024-09-18T07:44:30.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3895e-05 2024-09-18T07:44:30.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:30.931 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935565+0000", 2024-09-18T07:44:30.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6241e-05 2024-09-18T07:44:30.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.931 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:30.931 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.935599+0000", 2024-09-18T07:44:30.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4221000000000003e-05 2024-09-18T07:44:30.932 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.932 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:30.932 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.933 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202651 7.f 7:f9b36136:::benchmark_data_smithi137_74180_object98235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:44:30.933 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.895907+0000", 2024-09-18T07:44:30.933 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 532.91726856299999, 2024-09-18T07:44:30.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.071674998, 2024-09-18T07:44:30.933 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.934 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.934 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.934 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:30.934 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:44:30.934 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:30.934 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202651 2024-09-18T07:44:30.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.935 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.935 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895907+0000", 2024-09-18T07:44:30.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.936 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.936 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:30.936 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895907+0000", 2024-09-18T07:44:30.936 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.936 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.936 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:30.937 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895909+0000", 2024-09-18T07:44:30.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8989999999999999e-06 2024-09-18T07:44:30.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:30.938 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895915+0000", 2024-09-18T07:44:30.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6350000000000003e-06 2024-09-18T07:44:30.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.938 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:30.938 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895929+0000", 2024-09-18T07:44:30.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4174e-05 2024-09-18T07:44:30.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:30.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895936+0000", 2024-09-18T07:44:30.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1450000000000002e-06 2024-09-18T07:44:30.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.895992+0000", 2024-09-18T07:44:30.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6255999999999998e-05 2024-09-18T07:44:30.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:30.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.896001+0000", 2024-09-18T07:44:30.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4919999999999993e-06 2024-09-18T07:44:30.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.947291+0000", 2024-09-18T07:44:30.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.051289881 2024-09-18T07:44:30.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:30.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.947431+0000", 2024-09-18T07:44:30.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014024799999999999 2024-09-18T07:44:30.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.943 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:44:30.943 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.947553+0000", 2024-09-18T07:44:30.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000122032 2024-09-18T07:44:30.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.957558+0000", 2024-09-18T07:44:30.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010004948 2024-09-18T07:44:30.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.945 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:30.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.957904+0000", 2024-09-18T07:44:30.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034637699999999998 2024-09-18T07:44:30.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.945 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967474+0000", 2024-09-18T07:44:30.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095700709999999994 2024-09-18T07:44:30.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:30.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967537+0000", 2024-09-18T07:44:30.946 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2881999999999999e-05 2024-09-18T07:44:30.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:30.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967582+0000", 2024-09-18T07:44:30.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4957999999999999e-05 2024-09-18T07:44:30.947 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.947 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:30.947 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.947 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:202659 7.f 7:f246d9b4:::benchmark_data_smithi137_74180_object98243:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-18T07:44:30.947 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:37.901704+0000", 2024-09-18T07:44:30.948 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 532.91147173499996, 2024-09-18T07:44:30.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0662714099999999, 2024-09-18T07:44:30.948 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.948 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.948 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.948 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:30.948 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:44:30.948 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:30.949 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 202659 2024-09-18T07:44:30.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.949 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.949 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901704+0000", 2024-09-18T07:44:30.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.950 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:30.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901704+0000", 2024-09-18T07:44:30.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:30.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901705+0000", 2024-09-18T07:44:30.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.296e-06 2024-09-18T07:44:30.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:30.952 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901709+0000", 2024-09-18T07:44:30.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9280000000000003e-06 2024-09-18T07:44:30.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:30.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901719+0000", 2024-09-18T07:44:30.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3049999999999998e-06 2024-09-18T07:44:30.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:30.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901724+0000", 2024-09-18T07:44:30.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2070000000000003e-06 2024-09-18T07:44:30.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901762+0000", 2024-09-18T07:44:30.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8578000000000002e-05 2024-09-18T07:44:30.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:30.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:37.901784+0000", 2024-09-18T07:44:30.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1883e-05 2024-09-18T07:44:30.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.950958+0000", 2024-09-18T07:44:30.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.049174042 2024-09-18T07:44:30.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:30.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.951088+0000", 2024-09-18T07:44:30.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000129729 2024-09-18T07:44:30.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.955 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-18T07:44:30.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.951211+0000", 2024-09-18T07:44:30.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012286099999999999 2024-09-18T07:44:30.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.956 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.956 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.957893+0000", 2024-09-18T07:44:30.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066821709999999998 2024-09-18T07:44:30.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.956 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:30.956 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.958138+0000", 2024-09-18T07:44:30.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024456100000000002 2024-09-18T07:44:30.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967712+0000", 2024-09-18T07:44:30.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095747760000000001 2024-09-18T07:44:30.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:30.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967934+0000", 2024-09-18T07:44:30.958 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000221225 2024-09-18T07:44:30.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:30.959 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:38.967975+0000", 2024-09-18T07:44:30.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1848000000000003e-05 2024-09-18T07:44:30.959 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.959 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:30.959 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.960 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109534 8.6 8:67e394d3:::benchmark_data_smithi137_101942_object4724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-18T07:44:30.960 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.048516+0000", 2024-09-18T07:44:30.960 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 408.76465938000001, 2024-09-18T07:44:30.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.182592823, 2024-09-18T07:44:30.960 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.960 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.961 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.961 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:30.961 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:44:30.961 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:44:30.961 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109534 2024-09-18T07:44:30.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.962 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048516+0000", 2024-09-18T07:44:30.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:30.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048516+0000", 2024-09-18T07:44:30.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:30.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048517+0000", 2024-09-18T07:44:30.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2e-07 2024-09-18T07:44:30.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:30.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048519+0000", 2024-09-18T07:44:30.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.232e-06 2024-09-18T07:44:30.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:30.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048523+0000", 2024-09-18T07:44:30.966 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:30.968 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:30 smithi203 systemd[1]: Starting Ceph osd.4 for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:44:30.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4479999999999996e-06 2024-09-18T07:44:30.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:30.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048528+0000", 2024-09-18T07:44:30.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2169999999999996e-06 2024-09-18T07:44:30.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065112+0000", 2024-09-18T07:44:30.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016583912999999999 2024-09-18T07:44:30.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:30.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065121+0000", 2024-09-18T07:44:30.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8640000000000008e-06 2024-09-18T07:44:30.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.219070+0000", 2024-09-18T07:44:30.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1539486139999999 2024-09-18T07:44:30.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:30.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.219228+0000", 2024-09-18T07:44:30.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000157751 2024-09-18T07:44:30.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-09-18T07:44:30.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.219376+0000", 2024-09-18T07:44:30.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014781899999999999 2024-09-18T07:44:30.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226242+0000", 2024-09-18T07:44:30.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068661970000000001 2024-09-18T07:44:30.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:30.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226253+0000", 2024-09-18T07:44:30.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0929e-05 2024-09-18T07:44:30.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229198+0000", 2024-09-18T07:44:30.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029453790000000001 2024-09-18T07:44:30.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:30.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229278+0000", 2024-09-18T07:44:30.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9431000000000006e-05 2024-09-18T07:44:30.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:30.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.231109+0000", 2024-09-18T07:44:30.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018316090000000001 2024-09-18T07:44:30.976 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.976 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:30.976 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.976 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109539 8.6 8:66837289:::benchmark_data_smithi137_101942_object4729:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:44:30.976 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.065140+0000", 2024-09-18T07:44:30.976 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 408.74803591099999, 2024-09-18T07:44:30.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1676551340000001, 2024-09-18T07:44:30.976 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.977 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.977 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.977 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:30.977 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:44:30.977 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:44:30.977 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109539 2024-09-18T07:44:30.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.977 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065140+0000", 2024-09-18T07:44:30.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:30.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065140+0000", 2024-09-18T07:44:30.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:30.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065142+0000", 2024-09-18T07:44:30.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.548e-06 2024-09-18T07:44:30.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:30.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065149+0000", 2024-09-18T07:44:30.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7989999999999996e-06 2024-09-18T07:44:30.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:30.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065162+0000", 2024-09-18T07:44:30.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3027000000000001e-05 2024-09-18T07:44:30.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:30.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065171+0000", 2024-09-18T07:44:30.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1730000000000001e-06 2024-09-18T07:44:30.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.980 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065215+0000", 2024-09-18T07:44:30.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3679e-05 2024-09-18T07:44:30.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:30.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065222+0000", 2024-09-18T07:44:30.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9879999999999998e-06 2024-09-18T07:44:30.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.223751+0000", 2024-09-18T07:44:30.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1585293800000001 2024-09-18T07:44:30.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:30.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.223906+0000", 2024-09-18T07:44:30.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015412600000000001 2024-09-18T07:44:30.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-09-18T07:44:30.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.224068+0000", 2024-09-18T07:44:30.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016266499999999999 2024-09-18T07:44:30.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:30.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226399+0000", 2024-09-18T07:44:30.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002330557 2024-09-18T07:44:30.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226549+0000", 2024-09-18T07:44:30.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000150693 2024-09-18T07:44:30.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.231185+0000", 2024-09-18T07:44:30.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004635775 2024-09-18T07:44:30.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.984 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:30.984 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.231241+0000", 2024-09-18T07:44:30.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6001999999999997e-05 2024-09-18T07:44:30.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.984 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:30.984 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.232795+0000", 2024-09-18T07:44:30.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001553722 2024-09-18T07:44:30.984 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.985 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:30.985 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.985 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:110500 10.d e383/364 10:bbae7092:::benchmark_data_smithi137_151160_object110499:head v 383'6892, mlcod=383'6890)", 2024-09-18T07:44:30.985 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:18.832997+0000", 2024-09-18T07:44:30.985 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 11.980178613, 2024-09-18T07:44:30.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.070508840000000003, 2024-09-18T07:44:30.985 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.985 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.985 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.986 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.832997+0000", 2024-09-18T07:44:30.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:30.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.832997+0000", 2024-09-18T07:44:30.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:30.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.832998+0000", 2024-09-18T07:44:30.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7100000000000001e-07 2024-09-18T07:44:30.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:30.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.833075+0000", 2024-09-18T07:44:30.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6763999999999999e-05 2024-09-18T07:44:30.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:30.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.833082+0000", 2024-09-18T07:44:30.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9630000000000001e-06 2024-09-18T07:44:30.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:30.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.833086+0000", 2024-09-18T07:44:30.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9570000000000001e-06 2024-09-18T07:44:30.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.833146+0000", 2024-09-18T07:44:30.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9886000000000002e-05 2024-09-18T07:44:30.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:30.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.833245+0000", 2024-09-18T07:44:30.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8215999999999996e-05 2024-09-18T07:44:30.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:30.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.903441+0000", 2024-09-18T07:44:30.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.070196426000000006 2024-09-18T07:44:30.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:30.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.903506+0000", 2024-09-18T07:44:30.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4956999999999995e-05 2024-09-18T07:44:30.989 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.989 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:30.989 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.989 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:110503 10.8 10:17a0d83d:::benchmark_data_smithi137_151160_object110502:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-18T07:44:30.990 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:18.836289+0000", 2024-09-18T07:44:30.990 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 11.976886451, 2024-09-18T07:44:30.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14825258199999999, 2024-09-18T07:44:30.990 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.990 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.990 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.990 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:30.990 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:44:30.990 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:30.990 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 110503 2024-09-18T07:44:30.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.990 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.991 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.991 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.836289+0000", 2024-09-18T07:44:30.991 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.991 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:30.991 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.836289+0000", 2024-09-18T07:44:30.991 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.991 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:30.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.836291+0000", 2024-09-18T07:44:30.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5719999999999999e-06 2024-09-18T07:44:30.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:30.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.836312+0000", 2024-09-18T07:44:30.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1219000000000001e-05 2024-09-18T07:44:30.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:30.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.836322+0000", 2024-09-18T07:44:30.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7089999999999997e-06 2024-09-18T07:44:30.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:30.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.836328+0000", 2024-09-18T07:44:30.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9989999999999997e-06 2024-09-18T07:44:30.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:30.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.836414+0000", 2024-09-18T07:44:30.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6261000000000006e-05 2024-09-18T07:44:30.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:30.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.836558+0000", 2024-09-18T07:44:30.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014353799999999999 2024-09-18T07:44:30.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:44:30.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.836710+0000", 2024-09-18T07:44:30.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015248599999999999 2024-09-18T07:44:30.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.847250+0000", 2024-09-18T07:44:30.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010539662 2024-09-18T07:44:30.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:30.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.848011+0000", 2024-09-18T07:44:30.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00076089899999999999 2024-09-18T07:44:30.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:30.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.984454+0000", 2024-09-18T07:44:30.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13644367800000001 2024-09-18T07:44:30.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:30.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.984494+0000", 2024-09-18T07:44:30.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9437999999999997e-05 2024-09-18T07:44:30.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:30.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:18.984542+0000", 2024-09-18T07:44:30.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8121000000000002e-05 2024-09-18T07:44:30.996 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.997 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:30.997 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:30.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.997 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:111076 10.8 10:144691ae:::benchmark_data_smithi137_151160_object111075:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-18T07:44:30.997 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:19.417178+0000", 2024-09-18T07:44:30.997 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 11.395997592000001, 2024-09-18T07:44:30.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.070628137999999993, 2024-09-18T07:44:30.997 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:30.997 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:30.997 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:30.997 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:30.998 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:44:30.998 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:30.998 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 111076 2024-09-18T07:44:30.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.998 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:30.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.998 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:30.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.417178+0000", 2024-09-18T07:44:30.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:30.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:30.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:30.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:31.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.417178+0000", 2024-09-18T07:44:31.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:31.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:31.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.417180+0000", 2024-09-18T07:44:31.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.556e-06 2024-09-18T07:44:31.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.001 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:31.001 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.417203+0000", 2024-09-18T07:44:31.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3526000000000001e-05 2024-09-18T07:44:31.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:31.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.417213+0000", 2024-09-18T07:44:31.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5680000000000005e-06 2024-09-18T07:44:31.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.003 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:31.003 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.417220+0000", 2024-09-18T07:44:31.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7209999999999997e-06 2024-09-18T07:44:31.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.004 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:31.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.417411+0000", 2024-09-18T07:44:31.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000191623 2024-09-18T07:44:31.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.005 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:31.005 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.417560+0000", 2024-09-18T07:44:31.005 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014894100000000001 2024-09-18T07:44:31.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:44:31.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.418338+0000", 2024-09-18T07:44:31.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00077757800000000001 2024-09-18T07:44:31.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:31.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.422425+0000", 2024-09-18T07:44:31.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040877459999999997 2024-09-18T07:44:31.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:31.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.423028+0000", 2024-09-18T07:44:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00060239199999999999 2024-09-18T07:44:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.487702+0000", 2024-09-18T07:44:31.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.064674385000000001 2024-09-18T07:44:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.487764+0000", 2024-09-18T07:44:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2257999999999999e-05 2024-09-18T07:44:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:31.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.487806+0000", 2024-09-18T07:44:31.010 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1844000000000001e-05 2024-09-18T07:44:31.010 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:31.010 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:31.010 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:31.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.011 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:111087 10.c e384/364 10:30df614a:::benchmark_data_smithi137_151160_object111086:head v 384'6985, mlcod=384'6983)", 2024-09-18T07:44:31.011 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:19.425739+0000", 2024-09-18T07:44:31.011 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 11.387437155000001, 2024-09-18T07:44:31.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.070725184999999996, 2024-09-18T07:44:31.012 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:31.012 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:31.012 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:31.012 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:31.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.013 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:31.013 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.425739+0000", 2024-09-18T07:44:31.013 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:31.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.013 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:31.014 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.425739+0000", 2024-09-18T07:44:31.014 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:31.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.014 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:31.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.425739+0000", 2024-09-18T07:44:31.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3499999999999995e-07 2024-09-18T07:44:31.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:31.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.425777+0000", 2024-09-18T07:44:31.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7663999999999997e-05 2024-09-18T07:44:31.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:31.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.425784+0000", 2024-09-18T07:44:31.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.075e-06 2024-09-18T07:44:31.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.017 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:31.017 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.425789+0000", 2024-09-18T07:44:31.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0440000000000003e-06 2024-09-18T07:44:31.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.017 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:31.017 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.428019+0000", 2024-09-18T07:44:31.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022302810000000002 2024-09-18T07:44:31.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:31.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.428039+0000", 2024-09-18T07:44:31.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9176000000000001e-05 2024-09-18T07:44:31.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:31.019 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.496375+0000", 2024-09-18T07:44:31.019 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.068336851000000004 2024-09-18T07:44:31.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.019 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:31.019 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.496464+0000", 2024-09-18T07:44:31.019 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8358999999999999e-05 2024-09-18T07:44:31.019 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:31.020 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:31.020 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:31.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.020 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.020 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:111090 10.4 10:2fa383e2:::benchmark_data_smithi137_151160_object111089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-18T07:44:31.020 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:19.426004+0000", 2024-09-18T07:44:31.021 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 11.387171622, 2024-09-18T07:44:31.021 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.080489120999999997, 2024-09-18T07:44:31.021 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:31.021 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:31.021 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:31.021 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:31.022 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:44:31.022 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:31.022 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 111090 2024-09-18T07:44:31.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.022 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:31.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:31.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.426004+0000", 2024-09-18T07:44:31.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:31.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:31.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.426004+0000", 2024-09-18T07:44:31.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:31.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:31.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.426005+0000", 2024-09-18T07:44:31.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3379999999999999e-06 2024-09-18T07:44:31.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:31.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.426025+0000", 2024-09-18T07:44:31.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9259999999999999e-05 2024-09-18T07:44:31.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.025 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:31.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.426034+0000", 2024-09-18T07:44:31.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.935e-06 2024-09-18T07:44:31.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.025 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:31.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.426040+0000", 2024-09-18T07:44:31.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1700000000000002e-06 2024-09-18T07:44:31.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.025 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:31.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.430661+0000", 2024-09-18T07:44:31.026 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046213360000000002 2024-09-18T07:44:31.026 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.026 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:31.026 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.487935+0000", 2024-09-18T07:44:31.026 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.057273447999999998 2024-09-18T07:44:31.026 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.026 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-09-18T07:44:31.026 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.488186+0000", 2024-09-18T07:44:31.026 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000251332 2024-09-18T07:44:31.026 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.026 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:31.027 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.501278+0000", 2024-09-18T07:44:31.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013091833000000001 2024-09-18T07:44:31.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.027 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:31.027 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.506352+0000", 2024-09-18T07:44:31.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050740459999999996 2024-09-18T07:44:31.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.027 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:31.027 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.506397+0000", 2024-09-18T07:44:31.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5053999999999999e-05 2024-09-18T07:44:31.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:31.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:31.028 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:31.028 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.506493+0000", 2024-09-18T07:44:31.028 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6368999999999998e-05 2024-09-18T07:44:31.028 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:31.028 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:31.028 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:31.028 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:31.028 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:31.028 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:44:31.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:31.030 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:31.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:31.080 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 81 16 125040 125024 96.4362 70.75 0.0153636 0.0103651 2024-09-18T07:44:31.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:31.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:31.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:31.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:31.196 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:30 smithi203 podman[118312]: 2024-09-18 07:44:30.797992029 +0000 UTC m=+0.025305812 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:44:31.196 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:30 smithi203 podman[118312]: 2024-09-18 07:44:30.931346183 +0000 UTC m=+0.158659962 container create b257d21fcbaaee033b9209e4ae92af8961922a478cb9c5054e4c59d9f3d49ae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=) 2024-09-18T07:44:31.196 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:31 smithi203 podman[118312]: 2024-09-18 07:44:31.193612178 +0000 UTC m=+0.420925958 container init b257d21fcbaaee033b9209e4ae92af8961922a478cb9c5054e4c59d9f3d49ae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-activate, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-18T07:44:31.268 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:31.300 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:31.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:31.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:31.401 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:31.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:31.430 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:31.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:31.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:31 smithi203 ceph-mon[32237]: pgmap v712: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 920 B/s rd, 66 MiB/s wr, 1.06k op/s; 60102/360885 objects degraded (16.654%) 2024-09-18T07:44:31.461 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:31 smithi203 podman[118312]: 2024-09-18 07:44:31.20285648 +0000 UTC m=+0.430170265 container start b257d21fcbaaee033b9209e4ae92af8961922a478cb9c5054e4c59d9f3d49ae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-18T07:44:31.461 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:31 smithi203 podman[118312]: 2024-09-18 07:44:31.29362874 +0000 UTC m=+0.520942545 container attach b257d21fcbaaee033b9209e4ae92af8961922a478cb9c5054e4c59d9f3d49ae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-18T07:44:31.461 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:31 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-activate[118343]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-18T07:44:31.461 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:31 smithi203 bash[118312]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-18T07:44:31.461 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:31 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-activate[118343]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-18T07:44:31.461 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:31 smithi203 bash[118312]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-18T07:44:31.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:31.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:31.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:31.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:31.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:31 smithi137 ceph-mon[29408]: pgmap v712: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 920 B/s rd, 66 MiB/s wr, 1.06k op/s; 60102/360885 objects degraded (16.654%) 2024-09-18T07:44:31.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:31 smithi137 ceph-mon[39175]: pgmap v712: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 920 B/s rd, 66 MiB/s wr, 1.06k op/s; 60102/360885 objects degraded (16.654%) 2024-09-18T07:44:31.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:31.710 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:31.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:31.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:31.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:31.849 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:31.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:31.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:31.951 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:31.978 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:31.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:31.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:32.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:32.081 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 82 16 126291 126275 96.2136 78.1875 0.0185887 0.0103888 2024-09-18T07:44:32.115 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:32.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:32.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:32.217 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:32.249 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:32.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:32.285 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T07:44:32.351 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:32.391 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:32.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:32.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:32.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:32.530 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:32.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:32.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:32.617 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:44:32.618 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:44:32.618 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:32.632 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:32.658 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:44:32.658 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:44:32.661 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:32.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:32.693 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:44:32.694 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:44:32.724 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:44:32.725 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:44:32.726 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-18T07:44:32.754 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:44:32.755 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:44:32.763 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:32.786 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:44:32.786 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:44:32.816 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:44:32.816 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:44:32.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:32.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:32.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:32.923 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:32.955 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:32.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:32.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:33.057 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:33.080 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 83 16 127448 127432 95.9256 72.3125 0.00961502 0.0104205 2024-09-18T07:44:33.091 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:33.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:33.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:33.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:33.223 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:33.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:33.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:33.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:33.361 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:33.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:33.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:33.463 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:33.499 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:33.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:33.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:33.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:33.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:33.637 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:33.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:33.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:33 smithi137 ceph-mon[29408]: pgmap v714: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 7.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 60829/363066 objects degraded (16.754%) 2024-09-18T07:44:33.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:33 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1927223205' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:33.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:33 smithi137 ceph-mon[39175]: pgmap v714: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 7.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 60829/363066 objects degraded (16.754%) 2024-09-18T07:44:33.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:33 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1927223205' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:33.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:33 smithi203 ceph-mon[32237]: pgmap v714: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 7.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 60829/363066 objects degraded (16.754%) 2024-09-18T07:44:33.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:33 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1927223205' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:33.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:33.764 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:44:33.764 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:44:33.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:33.777 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:33.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:33.878 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:33.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:33.916 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:33.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:34.017 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:34.045 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:34.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:34.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:34.081 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 84 15 128744 128729 95.7479 81.0625 0.0151514 0.0104396 2024-09-18T07:44:34.091 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:34.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:34.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:34.192 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:34.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:34.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:34.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:34.342 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:34.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:34.392 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:34 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-activate[118343]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-18T07:44:34.392 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:34 smithi203 bash[118312]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-18T07:44:34.393 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:34 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-activate[118343]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-18T07:44:34.393 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:34 smithi203 bash[118312]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-18T07:44:34.443 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:34.485 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:34.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:34.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:34.587 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:34.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:34.639 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:34.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:34.711 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:34 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-activate[118343]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-18T07:44:34.711 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:34 smithi203 bash[118312]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-18T07:44:34.711 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:34 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-activate[118343]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-18T07:44:34.711 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:34 smithi203 bash[118312]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-18T07:44:34.711 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:34 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-activate[118343]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-18T07:44:34.711 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:34 smithi203 bash[118312]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-18T07:44:34.711 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:34 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-activate[118343]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-18T07:44:34.711 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:34 smithi203 bash[118312]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-18T07:44:34.712 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:34 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-activate[118343]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-18T07:44:34.712 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:34 smithi203 bash[118312]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-18T07:44:34.712 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:34 smithi203 podman[118312]: 2024-09-18 07:44:34.43082784 +0000 UTC m=+3.658141621 container died b257d21fcbaaee033b9209e4ae92af8961922a478cb9c5054e4c59d9f3d49ae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-09-18T07:44:34.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:34.758 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:44:34.758 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:44:34.779 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:34.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:34.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:34.882 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:34.912 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:34.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:35.014 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:35.052 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:35.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:35.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:35.081 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 85 16 129928 129912 95.4912 73.9375 0.029372 0.0104665 2024-09-18T07:44:35.154 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:35.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:35.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:35.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:35.239 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:44:35.293 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:35.331 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:35.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:35.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:35.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:35.504 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:35 smithi203 ceph-mon[32237]: pgmap v715: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 63183/379410 objects degraded (16.653%) 2024-09-18T07:44:35.504 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:35 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/935898816' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:44:35.504 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:35 smithi203 ceph-mon[32237]: from='client.26230 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:44:35.504 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:35 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 63183/379410 objects degraded (16.653%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:35.505 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:35 smithi203 podman[118639]: 2024-09-18 07:44:35.225932342 +0000 UTC m=+0.783842800 container remove b257d21fcbaaee033b9209e4ae92af8961922a478cb9c5054e4c59d9f3d49ae2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-18T07:44:35.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:35.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:35.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:35.643 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:35.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:35 smithi137 ceph-mon[39175]: pgmap v715: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 63183/379410 objects degraded (16.653%) 2024-09-18T07:44:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:35 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/935898816' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:44:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:35 smithi137 ceph-mon[39175]: from='client.26230 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:44:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:35 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 63183/379410 objects degraded (16.653%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:35 smithi137 ceph-mon[29408]: pgmap v715: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 63183/379410 objects degraded (16.653%) 2024-09-18T07:44:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:35 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/935898816' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:44:35.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:35 smithi137 ceph-mon[29408]: from='client.26230 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:44:35.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:35 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 63183/379410 objects degraded (16.653%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:35.672 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:35.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:35.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:44:35.772 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:35.782 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:35 smithi203 podman[118747]: 2024-09-18 07:44:35.391492641 +0000 UTC m=+0.023770883 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:44:35.783 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:35 smithi203 podman[118747]: 2024-09-18 07:44:35.514093735 +0000 UTC m=+0.146371962 container create 7912d2ab502400cac72c976c4d0a49bb8f7c0602cf9d784f0ab703b8bd8127dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-09-18T07:44:35.802 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:35.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:35.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:35.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:35.924 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T07:44:35.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:35.944 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:35.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:36.045 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:36.075 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:36.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:36.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:36.084 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 86 16 131190 131174 95.2948 78.875 0.0182973 0.0104891 2024-09-18T07:44:36.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:36.212 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:35 smithi203 podman[118747]: 2024-09-18 07:44:35.780741043 +0000 UTC m=+0.413019282 container init 7912d2ab502400cac72c976c4d0a49bb8f7c0602cf9d784f0ab703b8bd8127dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4, ceph=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-18T07:44:36.212 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:35 smithi203 podman[118747]: 2024-09-18 07:44:35.785399588 +0000 UTC m=+0.417677824 container start 7912d2ab502400cac72c976c4d0a49bb8f7c0602cf9d784f0ab703b8bd8127dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-18T07:44:36.212 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:35 smithi203 ceph-osd[118796]: -- 172.21.15.203:0/181341708 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1432683336 0 0) 0x55a535ce8000 con 0x55a534ee5c00 2024-09-18T07:44:36.212 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:35 smithi203 ceph-osd[118796]: -- 172.21.15.203:0/181341708 <== mon.1 v1:172.21.15.203:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (unknown 1754086984 0 0) 0x55a534f07a40 con 0x55a534ee5c00 2024-09-18T07:44:36.213 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:35 smithi203 bash[118747]: 7912d2ab502400cac72c976c4d0a49bb8f7c0602cf9d784f0ab703b8bd8127dc 2024-09-18T07:44:36.213 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:35 smithi203 systemd[1]: Started Ceph osd.4 for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:44:36.215 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:36.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:36.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:36.292 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:44:36.318 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:36.336 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T07:44:36.346 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:36.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:36.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:36.448 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:36.480 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:36.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:36.582 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:36.586 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:36 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:44:36.586 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:36 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:44:36.586 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:36 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:44:36.586 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:36 smithi203 ceph-mon[32237]: from='client.26230 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:44:36.586 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:36 smithi203 ceph-mon[32237]: osdmap e392: 8 total, 7 up, 8 in 2024-09-18T07:44:36.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:36.615 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:36.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:36.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:36 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:44:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:36 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:44:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:36 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:44:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:36 smithi137 ceph-mon[29408]: from='client.26230 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:44:36.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:36 smithi137 ceph-mon[29408]: osdmap e392: 8 total, 7 up, 8 in 2024-09-18T07:44:36.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:36 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:44:36.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:36 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:44:36.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:36 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:44:36.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:36 smithi137 ceph-mon[39175]: from='client.26230 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:44:36.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:36 smithi137 ceph-mon[39175]: osdmap e392: 8 total, 7 up, 8 in 2024-09-18T07:44:36.716 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:36.749 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:36.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:36.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:36.851 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:36.880 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:36.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:36.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:36.884 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:36.892 INFO:teuthology.orchestra.run.smithi203.stderr:no valid command found; 10 closest matches: 2024-09-18T07:44:36.892 INFO:teuthology.orchestra.run.smithi203.stderr:0 2024-09-18T07:44:36.892 INFO:teuthology.orchestra.run.smithi203.stderr:1 2024-09-18T07:44:36.892 INFO:teuthology.orchestra.run.smithi203.stderr:2 2024-09-18T07:44:36.892 INFO:teuthology.orchestra.run.smithi203.stderr:abort 2024-09-18T07:44:36.893 INFO:teuthology.orchestra.run.smithi203.stderr:assert 2024-09-18T07:44:36.893 INFO:teuthology.orchestra.run.smithi203.stderr:config diff 2024-09-18T07:44:36.893 INFO:teuthology.orchestra.run.smithi203.stderr:config diff get 2024-09-18T07:44:36.893 INFO:teuthology.orchestra.run.smithi203.stderr:config get 2024-09-18T07:44:36.893 INFO:teuthology.orchestra.run.smithi203.stderr:config help [] 2024-09-18T07:44:36.893 INFO:teuthology.orchestra.run.smithi203.stderr:config set ... 2024-09-18T07:44:36.893 INFO:teuthology.orchestra.run.smithi203.stderr:admin_socket: invalid command 2024-09-18T07:44:36.927 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:36.983 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:37.013 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:37.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:37.085 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 87 16 132400 132384 95.0674 75.625 0.0180739 0.0105141 2024-09-18T07:44:37.115 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:37.148 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:37.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:37.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:37.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:37.273 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:37.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:37.293 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:37 smithi137 ceph-mon[29408]: pgmap v717: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 912 op/s; 63183/379410 objects degraded (16.653%) 2024-09-18T07:44:37.294 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:37 smithi137 ceph-mon[29408]: 4.19 scrub starts 2024-09-18T07:44:37.294 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:37 smithi137 ceph-mon[29408]: 4.19 scrub ok 2024-09-18T07:44:37.294 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:44:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:44:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:44:37.294 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:37 smithi137 ceph-mon[39175]: pgmap v717: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 912 op/s; 63183/379410 objects degraded (16.653%) 2024-09-18T07:44:37.294 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:37 smithi137 ceph-mon[39175]: 4.19 scrub starts 2024-09-18T07:44:37.294 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:37 smithi137 ceph-mon[39175]: 4.19 scrub ok 2024-09-18T07:44:37.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:37.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:37.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:37.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:37.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:37.552 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:37.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:37.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:37.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:37.661 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:37 smithi203 ceph-mon[32237]: pgmap v717: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 912 op/s; 63183/379410 objects degraded (16.653%) 2024-09-18T07:44:37.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:37 smithi203 ceph-mon[32237]: 4.19 scrub starts 2024-09-18T07:44:37.662 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:37 smithi203 ceph-mon[32237]: 4.19 scrub ok 2024-09-18T07:44:37.662 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:37 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:44:37.458+0000 7faf31135640 -1 Falling back to public interface 2024-09-18T07:44:37.693 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:37.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:37.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:37.796 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:37.806 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:44:37.807 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:44:37.820 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:44:37.820 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:44:37.820 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:37.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:37.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:37.856 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:44:37.857 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:44:37.916 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:44:37.916 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:44:37.917 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-18T07:44:37.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:37.926 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:44:37.926 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:44:37.926 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:44:37.926 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:44:37.926 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:44:37.949 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:44:37.949 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:44:37.960 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:44:37 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:44:37.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:37.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:37.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:37.982 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:44:37.982 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:44:38.013 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:44:38.013 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:44:38.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:38.085 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 88 16 133494 133478 94.7641 68.375 0.00344687 0.0105455 2024-09-18T07:44:38.091 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:38.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:38.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:38.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:38.225 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:38.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:38.327 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:38.363 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:38.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:38.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:38.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:38 smithi137 ceph-mon[39175]: 2.1 scrub starts 2024-09-18T07:44:38.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:38 smithi137 ceph-mon[39175]: 2.1 scrub ok 2024-09-18T07:44:38.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:38 smithi137 ceph-mon[39175]: 6.10 scrub starts 2024-09-18T07:44:38.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:38 smithi137 ceph-mon[39175]: 6.10 scrub ok 2024-09-18T07:44:38.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:38 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/191414559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:38.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:38 smithi137 ceph-mon[29408]: 2.1 scrub starts 2024-09-18T07:44:38.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:38 smithi137 ceph-mon[29408]: 2.1 scrub ok 2024-09-18T07:44:38.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:38 smithi137 ceph-mon[29408]: 6.10 scrub starts 2024-09-18T07:44:38.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:38 smithi137 ceph-mon[29408]: 6.10 scrub ok 2024-09-18T07:44:38.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:38 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/191414559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:38.465 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:38.486 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:38.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:38.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:38.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:38.628 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:38.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:38.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:38 smithi203 ceph-mon[32237]: 2.1 scrub starts 2024-09-18T07:44:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:38 smithi203 ceph-mon[32237]: 2.1 scrub ok 2024-09-18T07:44:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:38 smithi203 ceph-mon[32237]: 6.10 scrub starts 2024-09-18T07:44:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:38 smithi203 ceph-mon[32237]: 6.10 scrub ok 2024-09-18T07:44:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:38 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/191414559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:38.730 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:38.770 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:38.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:38.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:38.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:44:38.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:38.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:38.918 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:38.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:38.967 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:44:38.967 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:44:39.019 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:39.044 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:39.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:39.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:39.086 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 89 16 134655 134639 94.5146 72.5625 0.00638538 0.0105758 2024-09-18T07:44:39.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:39.189 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:39.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:39.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:39.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:39.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:39.331 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:39.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:39.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:39.459 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:39.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:39.522 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:39 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:39.522 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:39 smithi137 ceph-mon[29408]: pgmap v718: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 8.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s; 65512/394476 objects degraded (16.607%) 2024-09-18T07:44:39.522 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:39 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:39.522 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:39 smithi137 ceph-mon[39175]: pgmap v718: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 8.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s; 65512/394476 objects degraded (16.607%) 2024-09-18T07:44:39.560 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:39.573 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T07:44:39.593 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:39.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:39.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:39.685 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:39.695 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:39.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:39 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:39 smithi203 ceph-mon[32237]: pgmap v718: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 8.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s; 65512/394476 objects degraded (16.607%) 2024-09-18T07:44:39.718 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:39.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:39.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:39.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:39.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:39.853 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:39.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:39.954 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:39.984 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:39.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:39.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:40.085 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:40.087 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 90 2 135813 135811 94.2784 73.25 0.0188116 0.0105999 2024-09-18T07:44:40.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:40.108 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:40.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:40.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:40.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:40.232 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:40.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:40.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:40.355 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:40.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:40.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:40.457 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:40.479 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:40.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:40.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:40.567 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:40 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 65512/394476 objects degraded (16.607%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:40.568 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:40 smithi137 ceph-mon[29408]: osdmap e393: 8 total, 7 up, 8 in 2024-09-18T07:44:40.568 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:40 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 65512/394476 objects degraded (16.607%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:40.568 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:40 smithi137 ceph-mon[39175]: osdmap e393: 8 total, 7 up, 8 in 2024-09-18T07:44:40.580 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:40.587 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:40 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 65512/394476 objects degraded (16.607%), 30 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:40.587 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:40 smithi203 ceph-mon[32237]: osdmap e393: 8 total, 7 up, 8 in 2024-09-18T07:44:40.602 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:40.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:40.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:40.668 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:40.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:40.725 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:40.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:40.827 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:40.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:40.849 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:40.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:40.946 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T07:44:40.947 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-09-18T07:44:40.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:40.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:44:40.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:40.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:40.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:41.073 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:41.086 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Total time run: 90.13 2024-09-18T07:44:41.086 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Total writes made: 135813 2024-09-18T07:44:41.086 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Write size: 65536 2024-09-18T07:44:41.086 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Object size: 65536 2024-09-18T07:44:41.086 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Bandwidth (MB/sec): 94.1786 2024-09-18T07:44:41.086 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Stddev Bandwidth: 35.708 2024-09-18T07:44:41.086 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Max bandwidth (MB/sec): 233.25 2024-09-18T07:44:41.087 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Min bandwidth (MB/sec): 45.5625 2024-09-18T07:44:41.087 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Average IOPS: 1506 2024-09-18T07:44:41.087 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Stddev IOPS: 571.327 2024-09-18T07:44:41.087 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Max IOPS: 3732 2024-09-18T07:44:41.087 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Min IOPS: 729 2024-09-18T07:44:41.087 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Average Latency(s): 0.0106018 2024-09-18T07:44:41.087 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Stddev Latency(s): 0.00972424 2024-09-18T07:44:41.087 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Max latency(s): 0.247818 2024-09-18T07:44:41.087 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Min latency(s): 0.00105851 2024-09-18T07:44:41.093 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Cleaning up (deleting benchmark objects) 2024-09-18T07:44:41.097 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:41.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:41.199 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:41.210 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:40 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:44:40.976+0000 7faf31135640 -1 osd.4 358 log_to_monitors true 2024-09-18T07:44:41.226 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:41.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:41.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:41.271 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:44:41.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:41.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:41.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:41.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:41.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:41.481 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:41.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:41.582 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:41.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:41.606 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:41.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:41.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:41 smithi137 ceph-mon[29408]: pgmap v719: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 66228/397797 objects degraded (16.649%) 2024-09-18T07:44:41.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:41 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2457608757' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:44:41.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:41 smithi137 ceph-mon[29408]: from='osd.4 v1:172.21.15.203:6800/571224280' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T07:44:41.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:41 smithi137 ceph-mon[29408]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T07:44:41.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:41 smithi137 ceph-mon[39175]: pgmap v719: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 66228/397797 objects degraded (16.649%) 2024-09-18T07:44:41.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:41 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2457608757' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:44:41.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:41 smithi137 ceph-mon[39175]: from='osd.4 v1:172.21.15.203:6800/571224280' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T07:44:41.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:41 smithi137 ceph-mon[39175]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T07:44:41.707 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:41.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:41 smithi203 ceph-mon[32237]: pgmap v719: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 66228/397797 objects degraded (16.649%) 2024-09-18T07:44:41.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:41 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2457608757' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:44:41.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:41 smithi203 ceph-mon[32237]: from='osd.4 v1:172.21.15.203:6800/571224280' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T07:44:41.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:41 smithi203 ceph-mon[32237]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-18T07:44:41.742 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:41.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:41.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:41.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:41.875 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:41.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:41.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:41.977 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:42.013 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:42.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:42.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:42.115 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:42.144 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:42.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:42.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:42.211 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:44:41 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:44:41.756+0000 7faf24ed7640 -1 osd.4 358 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T07:44:42.245 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:42.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:42.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:42.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:42.398 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:42.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:42.425 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:42.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:42.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:42.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:42.563 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:42.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:42.664 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:42.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:44:42.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2457608757' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:44:42.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[29408]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-18T07:44:42.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[29408]: osdmap e394: 8 total, 7 up, 8 in 2024-09-18T07:44:42.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[29408]: from='osd.4 v1:172.21.15.203:6800/571224280' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:44:42.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[29408]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:44:42.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[29408]: 4.a scrub starts 2024-09-18T07:44:42.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[29408]: 4.a scrub ok 2024-09-18T07:44:42.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[29408]: from='osd.4 ' entity='osd.4' 2024-09-18T07:44:42.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:44:42.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:44:42.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:44:42.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2457608757' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:44:42.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[39175]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-18T07:44:42.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[39175]: osdmap e394: 8 total, 7 up, 8 in 2024-09-18T07:44:42.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[39175]: from='osd.4 v1:172.21.15.203:6800/571224280' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:44:42.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[39175]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:44:42.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[39175]: 4.a scrub starts 2024-09-18T07:44:42.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[39175]: 4.a scrub ok 2024-09-18T07:44:42.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[39175]: from='osd.4 ' entity='osd.4' 2024-09-18T07:44:42.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:44:42.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:42 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:44:42.690 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:42.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:42.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:42 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:44:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:42 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2457608757' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:44:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:42 smithi203 ceph-mon[32237]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-18T07:44:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:42 smithi203 ceph-mon[32237]: osdmap e394: 8 total, 7 up, 8 in 2024-09-18T07:44:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:42 smithi203 ceph-mon[32237]: from='osd.4 v1:172.21.15.203:6800/571224280' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:44:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:42 smithi203 ceph-mon[32237]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:44:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:42 smithi203 ceph-mon[32237]: 4.a scrub starts 2024-09-18T07:44:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:42 smithi203 ceph-mon[32237]: 4.a scrub ok 2024-09-18T07:44:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:42 smithi203 ceph-mon[32237]: from='osd.4 ' entity='osd.4' 2024-09-18T07:44:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:42 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:44:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:42 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:44:42.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:42.815 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:42.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:42.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:42.917 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:42.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:42.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:42.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:43.040 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:43.062 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:43.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:43.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:43.164 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:43.185 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:43.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:43.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:43.200 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:44:43.200 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:44:43.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:44:43.200 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:44:43.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.200 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172965 9.e e335/289 9:7fd9b0fe:::benchmark_data_smithi137_126072_object40465:head v 335'10641, mlcod=335'10639)", 2024-09-18T07:44:43.200 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.554501+0000", 2024-09-18T07:44:43.200 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 183.81951371299999, 2024-09-18T07:44:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.178309934, 2024-09-18T07:44:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554501+0000", 2024-09-18T07:44:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554501+0000", 2024-09-18T07:44:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554502+0000", 2024-09-18T07:44:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.443e-06 2024-09-18T07:44:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554510+0000", 2024-09-18T07:44:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0660000000000004e-06 2024-09-18T07:44:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554518+0000", 2024-09-18T07:44:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0339999999999997e-06 2024-09-18T07:44:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554523+0000", 2024-09-18T07:44:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8740000000000003e-06 2024-09-18T07:44:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554567+0000", 2024-09-18T07:44:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3294000000000003e-05 2024-09-18T07:44:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554586+0000", 2024-09-18T07:44:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9419000000000002e-05 2024-09-18T07:44:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732782+0000", 2024-09-18T07:44:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.178196153 2024-09-18T07:44:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732811+0000", 2024-09-18T07:44:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8651000000000001e-05 2024-09-18T07:44:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172968 9.0 e335/289 9:02c69111:::benchmark_data_smithi137_126072_object40468:head v 335'10807, mlcod=335'10804)", 2024-09-18T07:44:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.554702+0000", 2024-09-18T07:44:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 183.81931236599999, 2024-09-18T07:44:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.179882141, 2024-09-18T07:44:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554702+0000", 2024-09-18T07:44:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554702+0000", 2024-09-18T07:44:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554704+0000", 2024-09-18T07:44:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.359e-06 2024-09-18T07:44:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554711+0000", 2024-09-18T07:44:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.199e-06 2024-09-18T07:44:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554720+0000", 2024-09-18T07:44:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5629999999999998e-06 2024-09-18T07:44:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554727+0000", 2024-09-18T07:44:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5889999999999999e-06 2024-09-18T07:44:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554767+0000", 2024-09-18T07:44:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9712000000000002e-05 2024-09-18T07:44:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554782+0000", 2024-09-18T07:44:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5639000000000001e-05 2024-09-18T07:44:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734508+0000", 2024-09-18T07:44:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17972560900000001 2024-09-18T07:44:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734584+0000", 2024-09-18T07:44:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6470999999999993e-05 2024-09-18T07:44:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172969 9.6 e335/289 9:60d9b40c:::benchmark_data_smithi137_126072_object40469:head v 335'10756, mlcod=335'10754)", 2024-09-18T07:44:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.554970+0000", 2024-09-18T07:44:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 183.81904502899999, 2024-09-18T07:44:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17787228899999999, 2024-09-18T07:44:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554970+0000", 2024-09-18T07:44:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554970+0000", 2024-09-18T07:44:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554971+0000", 2024-09-18T07:44:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0070000000000001e-06 2024-09-18T07:44:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554976+0000", 2024-09-18T07:44:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6330000000000001e-06 2024-09-18T07:44:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554993+0000", 2024-09-18T07:44:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6949e-05 2024-09-18T07:44:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.555000+0000", 2024-09-18T07:44:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5579999999999997e-06 2024-09-18T07:44:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.555176+0000", 2024-09-18T07:44:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017662999999999999 2024-09-18T07:44:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.555193+0000", 2024-09-18T07:44:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7212000000000001e-05 2024-09-18T07:44:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732814+0000", 2024-09-18T07:44:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.177620315 2024-09-18T07:44:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732842+0000", 2024-09-18T07:44:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7985000000000001e-05 2024-09-18T07:44:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172970 9.3 e335/289 9:c7e8a587:::benchmark_data_smithi137_126072_object40470:head v 335'10785, mlcod=335'10782)", 2024-09-18T07:44:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.557577+0000", 2024-09-18T07:44:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 183.81643710899999, 2024-09-18T07:44:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.173544268, 2024-09-18T07:44:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557577+0000", 2024-09-18T07:44:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557577+0000", 2024-09-18T07:44:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557579+0000", 2024-09-18T07:44:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5099999999999999e-06 2024-09-18T07:44:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557589+0000", 2024-09-18T07:44:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8079999999999999e-06 2024-09-18T07:44:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557598+0000", 2024-09-18T07:44:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2490000000000007e-06 2024-09-18T07:44:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557604+0000", 2024-09-18T07:44:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2079999999999997e-06 2024-09-18T07:44:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726578+0000", 2024-09-18T07:44:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.168973331 2024-09-18T07:44:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726596+0000", 2024-09-18T07:44:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8159e-05 2024-09-18T07:44:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731084+0000", 2024-09-18T07:44:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044883980000000002 2024-09-18T07:44:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731122+0000", 2024-09-18T07:44:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7605000000000003e-05 2024-09-18T07:44:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172972 9.e e335/289 9:796b9798:::benchmark_data_smithi137_126072_object40472:head v 335'10642, mlcod=335'10639)", 2024-09-18T07:44:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.557623+0000", 2024-09-18T07:44:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 183.81639125000001, 2024-09-18T07:44:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.175241702, 2024-09-18T07:44:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557623+0000", 2024-09-18T07:44:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557623+0000", 2024-09-18T07:44:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557624+0000", 2024-09-18T07:44:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4200000000000001e-07 2024-09-18T07:44:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557628+0000", 2024-09-18T07:44:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8139999999999998e-06 2024-09-18T07:44:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557632+0000", 2024-09-18T07:44:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6909999999999996e-06 2024-09-18T07:44:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557636+0000", 2024-09-18T07:44:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6629999999999999e-06 2024-09-18T07:44:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726230+0000", 2024-09-18T07:44:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.168594563 2024-09-18T07:44:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726286+0000", 2024-09-18T07:44:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5702000000000003e-05 2024-09-18T07:44:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732845+0000", 2024-09-18T07:44:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065588560000000001 2024-09-18T07:44:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732865+0000", 2024-09-18T07:44:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9970999999999999e-05 2024-09-18T07:44:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172971 9.4 e335/289 9:2cb28e72:::benchmark_data_smithi137_126072_object40471:head v 335'10859, mlcod=335'10857)", 2024-09-18T07:44:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.557654+0000", 2024-09-18T07:44:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 183.81636022399999, 2024-09-18T07:44:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17145278799999999, 2024-09-18T07:44:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.225 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557654+0000", 2024-09-18T07:44:43.225 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.225 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.225 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557654+0000", 2024-09-18T07:44:43.225 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.225 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.225 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557655+0000", 2024-09-18T07:44:43.225 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1099999999999995e-07 2024-09-18T07:44:43.225 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557661+0000", 2024-09-18T07:44:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2400000000000004e-06 2024-09-18T07:44:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557667+0000", 2024-09-18T07:44:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9139999999999996e-06 2024-09-18T07:44:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557672+0000", 2024-09-18T07:44:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6290000000000001e-06 2024-09-18T07:44:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726169+0000", 2024-09-18T07:44:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.168497014 2024-09-18T07:44:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726237+0000", 2024-09-18T07:44:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8677999999999995e-05 2024-09-18T07:44:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.729066+0000", 2024-09-18T07:44:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028290170000000001 2024-09-18T07:44:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.729107+0000", 2024-09-18T07:44:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0685000000000001e-05 2024-09-18T07:44:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172976 9.f e335/289 9:f0577982:::benchmark_data_smithi137_126072_object40476:head v 335'10844, mlcod=335'10842)", 2024-09-18T07:44:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.560675+0000", 2024-09-18T07:44:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 183.81333911900001, 2024-09-18T07:44:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17558478, 2024-09-18T07:44:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560675+0000", 2024-09-18T07:44:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560675+0000", 2024-09-18T07:44:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560677+0000", 2024-09-18T07:44:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.711e-06 2024-09-18T07:44:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560685+0000", 2024-09-18T07:44:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1880000000000002e-06 2024-09-18T07:44:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726219+0000", 2024-09-18T07:44:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16553400300000001 2024-09-18T07:44:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726233+0000", 2024-09-18T07:44:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3893000000000001e-05 2024-09-18T07:44:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726381+0000", 2024-09-18T07:44:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014760199999999999 2024-09-18T07:44:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726402+0000", 2024-09-18T07:44:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1664000000000001e-05 2024-09-18T07:44:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.736225+0000", 2024-09-18T07:44:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0098227820000000004 2024-09-18T07:44:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.736260+0000", 2024-09-18T07:44:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4937000000000001e-05 2024-09-18T07:44:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172980 9.3 e335/289 9:ce0a7506:::benchmark_data_smithi137_126072_object40480:head v 335'10786, mlcod=335'10783)", 2024-09-18T07:44:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.560718+0000", 2024-09-18T07:44:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 183.81329664699999, 2024-09-18T07:44:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17056611599999999, 2024-09-18T07:44:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560718+0000", 2024-09-18T07:44:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560718+0000", 2024-09-18T07:44:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560719+0000", 2024-09-18T07:44:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8400000000000003e-07 2024-09-18T07:44:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560726+0000", 2024-09-18T07:44:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4000000000000003e-06 2024-09-18T07:44:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726135+0000", 2024-09-18T07:44:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16540897700000001 2024-09-18T07:44:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726166+0000", 2024-09-18T07:44:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.099e-05 2024-09-18T07:44:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.728991+0000", 2024-09-18T07:44:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028248840000000002 2024-09-18T07:44:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.729009+0000", 2024-09-18T07:44:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7804000000000001e-05 2024-09-18T07:44:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731249+0000", 2024-09-18T07:44:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022400050000000002 2024-09-18T07:44:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731284+0000", 2024-09-18T07:44:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5271999999999997e-05 2024-09-18T07:44:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172984 9.0 e335/289 9:066846aa:::benchmark_data_smithi137_126072_object40484:head v 335'10808, mlcod=335'10805)", 2024-09-18T07:44:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.567459+0000", 2024-09-18T07:44:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 183.80655555300001, 2024-09-18T07:44:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16718034100000001, 2024-09-18T07:44:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567459+0000", 2024-09-18T07:44:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567459+0000", 2024-09-18T07:44:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567461+0000", 2024-09-18T07:44:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1550000000000001e-06 2024-09-18T07:44:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567471+0000", 2024-09-18T07:44:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0348e-05 2024-09-18T07:44:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726150+0000", 2024-09-18T07:44:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15867869100000001 2024-09-18T07:44:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726175+0000", 2024-09-18T07:44:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5316e-05 2024-09-18T07:44:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.728757+0000", 2024-09-18T07:44:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002581262 2024-09-18T07:44:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.728778+0000", 2024-09-18T07:44:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1739999999999999e-05 2024-09-18T07:44:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734588+0000", 2024-09-18T07:44:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005809519 2024-09-18T07:44:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734639+0000", 2024-09-18T07:44:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1310000000000002e-05 2024-09-18T07:44:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:244806 9.f e354/289 9:f6d427cd:::benchmark_data_smithi137_126072_object112306:head v 354'15363, mlcod=354'15361)", 2024-09-18T07:44:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:42:36.240579+0000", 2024-09-18T07:44:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 126.13343573, 2024-09-18T07:44:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16868968600000001, 2024-09-18T07:44:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240579+0000", 2024-09-18T07:44:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240579+0000", 2024-09-18T07:44:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240581+0000", 2024-09-18T07:44:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9599999999999999e-06 2024-09-18T07:44:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240589+0000", 2024-09-18T07:44:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4710000000000007e-06 2024-09-18T07:44:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396235+0000", 2024-09-18T07:44:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.155645916 2024-09-18T07:44:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396253+0000", 2024-09-18T07:44:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8170000000000001e-05 2024-09-18T07:44:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396312+0000", 2024-09-18T07:44:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9144000000000002e-05 2024-09-18T07:44:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396337+0000", 2024-09-18T07:44:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4771000000000001e-05 2024-09-18T07:44:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.409231+0000", 2024-09-18T07:44:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012893745 2024-09-18T07:44:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.409268+0000", 2024-09-18T07:44:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7509000000000003e-05 2024-09-18T07:44:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:71849 10.2 e374/364 10:41855215:::benchmark_data_smithi137_151160_object71848:head v 374'4514, mlcod=374'4512)", 2024-09-18T07:44:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:46.388333+0000", 2024-09-18T07:44:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 55.985681177000004, 2024-09-18T07:44:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.225317779, 2024-09-18T07:44:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388333+0000", 2024-09-18T07:44:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388333+0000", 2024-09-18T07:44:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388335+0000", 2024-09-18T07:44:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7910000000000001e-06 2024-09-18T07:44:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388381+0000", 2024-09-18T07:44:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5909999999999999e-05 2024-09-18T07:44:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388394+0000", 2024-09-18T07:44:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2599000000000001e-05 2024-09-18T07:44:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388402+0000", 2024-09-18T07:44:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2060000000000007e-06 2024-09-18T07:44:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606027+0000", 2024-09-18T07:44:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.21762539 2024-09-18T07:44:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606060+0000", 2024-09-18T07:44:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2991000000000003e-05 2024-09-18T07:44:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.613606+0000", 2024-09-18T07:44:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0075457730000000004 2024-09-18T07:44:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.613651+0000", 2024-09-18T07:44:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5118999999999999e-05 2024-09-18T07:44:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:71852 10.b e374/364 10:dda9ad8f:::benchmark_data_smithi137_151160_object71851:head v 374'4479, mlcod=374'4476)", 2024-09-18T07:44:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:46.390175+0000", 2024-09-18T07:44:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 55.983839447000001, 2024-09-18T07:44:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.22548416099999999, 2024-09-18T07:44:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.390175+0000", 2024-09-18T07:44:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.390175+0000", 2024-09-18T07:44:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.390177+0000", 2024-09-18T07:44:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.548e-06 2024-09-18T07:44:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.390217+0000", 2024-09-18T07:44:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0203000000000001e-05 2024-09-18T07:44:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606008+0000", 2024-09-18T07:44:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.21579153700000001 2024-09-18T07:44:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606020+0000", 2024-09-18T07:44:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1741e-05 2024-09-18T07:44:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606089+0000", 2024-09-18T07:44:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9058000000000002e-05 2024-09-18T07:44:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606112+0000", 2024-09-18T07:44:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2410999999999999e-05 2024-09-18T07:44:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.615499+0000", 2024-09-18T07:44:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093874549999999994 2024-09-18T07:44:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.615659+0000", 2024-09-18T07:44:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016020800000000001 2024-09-18T07:44:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:71850 10.2 e374/364 10:48354359:::benchmark_data_smithi137_151160_object71849:head v 374'4515, mlcod=374'4512)", 2024-09-18T07:44:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:46.392229+0000", 2024-09-18T07:44:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 55.981785862000002, 2024-09-18T07:44:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.221450217, 2024-09-18T07:44:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.392229+0000", 2024-09-18T07:44:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.392229+0000", 2024-09-18T07:44:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.392231+0000", 2024-09-18T07:44:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0109999999999999e-06 2024-09-18T07:44:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.392282+0000", 2024-09-18T07:44:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1019000000000001e-05 2024-09-18T07:44:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.605989+0000", 2024-09-18T07:44:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.213706807 2024-09-18T07:44:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606006+0000", 2024-09-18T07:44:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7289e-05 2024-09-18T07:44:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.608388+0000", 2024-09-18T07:44:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023825399999999998 2024-09-18T07:44:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.608412+0000", 2024-09-18T07:44:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4093000000000001e-05 2024-09-18T07:44:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.613656+0000", 2024-09-18T07:44:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052432060000000003 2024-09-18T07:44:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.613679+0000", 2024-09-18T07:44:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3252e-05 2024-09-18T07:44:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:109516 10.b e383/364 10:de3bb5f2:::benchmark_data_smithi137_151160_object109515:head v 383'6800, mlcod=383'6797)", 2024-09-18T07:44:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:17.911816+0000", 2024-09-18T07:44:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 24.462198862000001, 2024-09-18T07:44:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027846431000000001, 2024-09-18T07:44:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.911816+0000", 2024-09-18T07:44:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.911816+0000", 2024-09-18T07:44:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.911817+0000", 2024-09-18T07:44:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4890000000000001e-06 2024-09-18T07:44:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.911862+0000", 2024-09-18T07:44:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4839000000000002e-05 2024-09-18T07:44:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.911877+0000", 2024-09-18T07:44:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5432999999999999e-05 2024-09-18T07:44:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.911889+0000", 2024-09-18T07:44:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1739000000000001e-05 2024-09-18T07:44:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.919544+0000", 2024-09-18T07:44:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.007654765 2024-09-18T07:44:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.919562+0000", 2024-09-18T07:44:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8022000000000001e-05 2024-09-18T07:44:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.939614+0000", 2024-09-18T07:44:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020052362000000001 2024-09-18T07:44:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.939662+0000", 2024-09-18T07:44:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7781999999999998e-05 2024-09-18T07:44:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:119995 10.b e389/364 10:d71be2e3:::benchmark_data_smithi137_151160_object119994:head v 389'7429, mlcod=389'7425)", 2024-09-18T07:44:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:27.007791+0000", 2024-09-18T07:44:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 15.366223327, 2024-09-18T07:44:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026613194, 2024-09-18T07:44:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.007791+0000", 2024-09-18T07:44:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.007791+0000", 2024-09-18T07:44:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.007793+0000", 2024-09-18T07:44:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6840000000000001e-06 2024-09-18T07:44:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.007831+0000", 2024-09-18T07:44:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7923999999999998e-05 2024-09-18T07:44:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.007847+0000", 2024-09-18T07:44:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6285000000000001e-05 2024-09-18T07:44:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.007854+0000", 2024-09-18T07:44:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3139999999999998e-06 2024-09-18T07:44:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.008659+0000", 2024-09-18T07:44:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00080414799999999997 2024-09-18T07:44:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.008709+0000", 2024-09-18T07:44:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0691e-05 2024-09-18T07:44:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.034243+0000", 2024-09-18T07:44:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025533456999999999 2024-09-18T07:44:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.034404+0000", 2024-09-18T07:44:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016169099999999999 2024-09-18T07:44:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:119996 10.b e389/364 10:de189453:::benchmark_data_smithi137_151160_object119995:head v 389'7430, mlcod=389'7425)", 2024-09-18T07:44:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:27.009437+0000", 2024-09-18T07:44:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 15.364577636, 2024-09-18T07:44:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025171886000000001, 2024-09-18T07:44:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.009437+0000", 2024-09-18T07:44:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.009437+0000", 2024-09-18T07:44:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.009439+0000", 2024-09-18T07:44:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.903e-06 2024-09-18T07:44:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.009474+0000", 2024-09-18T07:44:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5621999999999999e-05 2024-09-18T07:44:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.009489+0000", 2024-09-18T07:44:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4611e-05 2024-09-18T07:44:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.009497+0000", 2024-09-18T07:44:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0569999999999993e-06 2024-09-18T07:44:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.013781+0000", 2024-09-18T07:44:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004283584 2024-09-18T07:44:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.013825+0000", 2024-09-18T07:44:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4737000000000002e-05 2024-09-18T07:44:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.034407+0000", 2024-09-18T07:44:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020581955999999998 2024-09-18T07:44:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.034609+0000", 2024-09-18T07:44:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000201416 2024-09-18T07:44:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.276 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:122152 10.2 e390/364 10:48dcd9dc:::benchmark_data_smithi137_151160_object122151:head v 390'7640, mlcod=390'7635)", 2024-09-18T07:44:43.276 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:28.657416+0000", 2024-09-18T07:44:43.276 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.716598692, 2024-09-18T07:44:43.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032362285999999997, 2024-09-18T07:44:43.276 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.276 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.276 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.276 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657416+0000", 2024-09-18T07:44:43.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657416+0000", 2024-09-18T07:44:43.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657417+0000", 2024-09-18T07:44:43.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3400000000000001e-06 2024-09-18T07:44:43.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657439+0000", 2024-09-18T07:44:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1903999999999999e-05 2024-09-18T07:44:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657449+0000", 2024-09-18T07:44:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.825e-06 2024-09-18T07:44:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657455+0000", 2024-09-18T07:44:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4659999999999996e-06 2024-09-18T07:44:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.664829+0000", 2024-09-18T07:44:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0073740740000000004 2024-09-18T07:44:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.664879+0000", 2024-09-18T07:44:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9308999999999998e-05 2024-09-18T07:44:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.689734+0000", 2024-09-18T07:44:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.024855497000000001 2024-09-18T07:44:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.689778+0000", 2024-09-18T07:44:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3871000000000001e-05 2024-09-18T07:44:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:122153 10.2 e390/364 10:41dfeb35:::benchmark_data_smithi137_151160_object122152:head v 390'7641, mlcod=390'7635)", 2024-09-18T07:44:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:28.664559+0000", 2024-09-18T07:44:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.709455156000001, 2024-09-18T07:44:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025245323, 2024-09-18T07:44:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.664559+0000", 2024-09-18T07:44:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.664559+0000", 2024-09-18T07:44:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.664561+0000", 2024-09-18T07:44:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0990000000000001e-06 2024-09-18T07:44:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.664593+0000", 2024-09-18T07:44:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1102999999999997e-05 2024-09-18T07:44:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.664613+0000", 2024-09-18T07:44:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9996999999999998e-05 2024-09-18T07:44:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.664622+0000", 2024-09-18T07:44:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6719999999999999e-06 2024-09-18T07:44:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.670712+0000", 2024-09-18T07:44:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0060899129999999998 2024-09-18T07:44:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.670767+0000", 2024-09-18T07:44:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5229000000000003e-05 2024-09-18T07:44:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.689784+0000", 2024-09-18T07:44:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019016590999999999 2024-09-18T07:44:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.689805+0000", 2024-09-18T07:44:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0718999999999999e-05 2024-09-18T07:44:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:135810 10.b e392/364 10:da1ccd0a:::benchmark_data_smithi137_151160_object135809:head v 392'8433, mlcod=392'8431)", 2024-09-18T07:44:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:40.045405+0000", 2024-09-18T07:44:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.3286094999999998, 2024-09-18T07:44:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13276168799999999, 2024-09-18T07:44:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045405+0000", 2024-09-18T07:44:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045405+0000", 2024-09-18T07:44:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045406+0000", 2024-09-18T07:44:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3820000000000001e-06 2024-09-18T07:44:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045446+0000", 2024-09-18T07:44:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9371e-05 2024-09-18T07:44:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045457+0000", 2024-09-18T07:44:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1657e-05 2024-09-18T07:44:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045464+0000", 2024-09-18T07:44:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7530000000000004e-06 2024-09-18T07:44:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045533+0000", 2024-09-18T07:44:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.923e-05 2024-09-18T07:44:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045551+0000", 2024-09-18T07:44:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7844999999999999e-05 2024-09-18T07:44:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.178077+0000", 2024-09-18T07:44:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.132526221 2024-09-18T07:44:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.178167+0000", 2024-09-18T07:44:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9228999999999996e-05 2024-09-18T07:44:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:135798 10.2 e392/364 10:4cac0c3f:::benchmark_data_smithi137_151160_object135797:head v 392'8494, mlcod=392'8491)", 2024-09-18T07:44:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:40.048398+0000", 2024-09-18T07:44:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.3256161610000001, 2024-09-18T07:44:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13237333200000001, 2024-09-18T07:44:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.048398+0000", 2024-09-18T07:44:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.048398+0000", 2024-09-18T07:44:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.048400+0000", 2024-09-18T07:44:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.826e-06 2024-09-18T07:44:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.048452+0000", 2024-09-18T07:44:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1360000000000003e-05 2024-09-18T07:44:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.177902+0000", 2024-09-18T07:44:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12945019699999999 2024-09-18T07:44:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.177917+0000", 2024-09-18T07:44:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4766999999999999e-05 2024-09-18T07:44:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.178327+0000", 2024-09-18T07:44:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00041051999999999999 2024-09-18T07:44:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.178347+0000", 2024-09-18T07:44:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9945e-05 2024-09-18T07:44:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.180740+0000", 2024-09-18T07:44:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023925790000000001 2024-09-18T07:44:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.180772+0000", 2024-09-18T07:44:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2138000000000003e-05 2024-09-18T07:44:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:43.295 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:44:43.295 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:43.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:43.317 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:43.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:43.418 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:43.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:43.441 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:43.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:43.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:43.564 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:43.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:43.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:43.666 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:43.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:43 smithi137 ceph-mon[39175]: pgmap v722: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 63 MiB/s wr, 1.03k op/s; 66632/399009 objects degraded (16.699%) 2024-09-18T07:44:43.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:43 smithi137 ceph-mon[39175]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-18T07:44:43.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:43 smithi137 ceph-mon[39175]: osd.4 v1:172.21.15.203:6800/571224280 boot 2024-09-18T07:44:43.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:43 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:44:43.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:43 smithi137 ceph-mon[39175]: osdmap e395: 8 total, 8 up, 8 in 2024-09-18T07:44:43.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:43 smithi137 ceph-mon[29408]: pgmap v722: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 63 MiB/s wr, 1.03k op/s; 66632/399009 objects degraded (16.699%) 2024-09-18T07:44:43.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:43 smithi137 ceph-mon[29408]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-18T07:44:43.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:43 smithi137 ceph-mon[29408]: osd.4 v1:172.21.15.203:6800/571224280 boot 2024-09-18T07:44:43.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:43 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:44:43.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:43 smithi137 ceph-mon[29408]: osdmap e395: 8 total, 8 up, 8 in 2024-09-18T07:44:43.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:43.689 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:43.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:43.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:43 smithi203 ceph-mon[32237]: pgmap v722: 158 pgs: 1 active+clean+remapped, 39 active+undersized, 30 active+undersized+degraded, 88 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 63 MiB/s wr, 1.03k op/s; 66632/399009 objects degraded (16.699%) 2024-09-18T07:44:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:43 smithi203 ceph-mon[32237]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-18T07:44:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:43 smithi203 ceph-mon[32237]: osd.4 v1:172.21.15.203:6800/571224280 boot 2024-09-18T07:44:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:43 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-18T07:44:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:43 smithi203 ceph-mon[32237]: osdmap e395: 8 total, 8 up, 8 in 2024-09-18T07:44:43.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:43.815 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:43.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:43.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:43.917 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:43.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:43.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:43.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:44.040 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:44.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:44.062 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:44.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:44.163 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:44.184 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:44.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:44.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:44.286 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:44.309 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:44.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:44.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:44.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:44.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:44.435 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:44.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:44.536 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:44.561 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:44.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:44.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:44.663 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:44.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:44:44.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:44:44.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[39175]: 10.3 scrub starts 2024-09-18T07:44:44.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[39175]: 6.13 scrub starts 2024-09-18T07:44:44.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[39175]: 6.13 scrub ok 2024-09-18T07:44:44.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[39175]: 6.0 scrub starts 2024-09-18T07:44:44.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[39175]: 6.0 scrub ok 2024-09-18T07:44:44.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[39175]: osdmap e396: 8 total, 8 up, 8 in 2024-09-18T07:44:44.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[39175]: 1.5 scrub starts 2024-09-18T07:44:44.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[39175]: 1.5 scrub ok 2024-09-18T07:44:44.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:44:44.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:44:44.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[29408]: 10.3 scrub starts 2024-09-18T07:44:44.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[29408]: 6.13 scrub starts 2024-09-18T07:44:44.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[29408]: 6.13 scrub ok 2024-09-18T07:44:44.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[29408]: 6.0 scrub starts 2024-09-18T07:44:44.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[29408]: 6.0 scrub ok 2024-09-18T07:44:44.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[29408]: osdmap e396: 8 total, 8 up, 8 in 2024-09-18T07:44:44.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[29408]: 1.5 scrub starts 2024-09-18T07:44:44.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:44 smithi137 ceph-mon[29408]: 1.5 scrub ok 2024-09-18T07:44:44.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:44.687 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:44.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:44.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:44 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:44:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:44 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:44:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:44 smithi203 ceph-mon[32237]: 10.3 scrub starts 2024-09-18T07:44:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:44 smithi203 ceph-mon[32237]: 6.13 scrub starts 2024-09-18T07:44:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:44 smithi203 ceph-mon[32237]: 6.13 scrub ok 2024-09-18T07:44:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:44 smithi203 ceph-mon[32237]: 6.0 scrub starts 2024-09-18T07:44:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:44 smithi203 ceph-mon[32237]: 6.0 scrub ok 2024-09-18T07:44:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:44 smithi203 ceph-mon[32237]: osdmap e396: 8 total, 8 up, 8 in 2024-09-18T07:44:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:44 smithi203 ceph-mon[32237]: 1.5 scrub starts 2024-09-18T07:44:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:44 smithi203 ceph-mon[32237]: 1.5 scrub ok 2024-09-18T07:44:44.788 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:44.815 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:44.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:44.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:44.917 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:44.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:44.953 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:44.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:45.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:45.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:45.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:45.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:45.191 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:45.223 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:45.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:45.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:45.325 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:45.364 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:45.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:45.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[29408]: pgmap v724: 158 pgs: 7 peering, 1 active+clean+remapped, 36 active+undersized, 26 active+undersized+degraded, 88 active+clean; 8.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 855 op/s; 59056/405198 objects degraded (14.575%) 2024-09-18T07:44:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[29408]: 10.3 scrub ok 2024-09-18T07:44:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[29408]: 4.1d scrub starts 2024-09-18T07:44:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[29408]: 4.1d scrub ok 2024-09-18T07:44:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[29408]: 3.6 scrub starts 2024-09-18T07:44:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[29408]: 3.6 scrub ok 2024-09-18T07:44:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[29408]: 6.1d scrub starts 2024-09-18T07:44:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[29408]: 6.1d scrub ok 2024-09-18T07:44:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[29408]: 4.5 scrub starts 2024-09-18T07:44:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[29408]: 4.5 scrub ok 2024-09-18T07:44:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[29408]: 10.1 starting backfill to osd.4 from (0'0,0'0] MAX to 394'8506 2024-09-18T07:44:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[29408]: 10.2 starting backfill to osd.4 from (0'0,0'0] MAX to 394'8624 2024-09-18T07:44:45.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[29408]: 10.c starting backfill to osd.4 from (0'0,0'0] MAX to 394'8609 2024-09-18T07:44:45.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[29408]: osdmap e397: 8 total, 8 up, 8 in 2024-09-18T07:44:45.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[29408]: 10.6 starting backfill to osd.4 from (0'0,0'0] MAX to 394'8668 2024-09-18T07:44:45.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[29408]: 10.4 starting backfill to osd.4 from (0'0,0'0] MAX to 394'8548 2024-09-18T07:44:45.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 59056/405198 objects degraded (14.575%), 26 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:45.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[39175]: pgmap v724: 158 pgs: 7 peering, 1 active+clean+remapped, 36 active+undersized, 26 active+undersized+degraded, 88 active+clean; 8.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 855 op/s; 59056/405198 objects degraded (14.575%) 2024-09-18T07:44:45.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[39175]: 10.3 scrub ok 2024-09-18T07:44:45.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[39175]: 4.1d scrub starts 2024-09-18T07:44:45.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[39175]: 4.1d scrub ok 2024-09-18T07:44:45.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[39175]: 3.6 scrub starts 2024-09-18T07:44:45.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[39175]: 3.6 scrub ok 2024-09-18T07:44:45.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[39175]: 6.1d scrub starts 2024-09-18T07:44:45.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[39175]: 6.1d scrub ok 2024-09-18T07:44:45.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[39175]: 4.5 scrub starts 2024-09-18T07:44:45.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[39175]: 4.5 scrub ok 2024-09-18T07:44:45.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[39175]: 10.1 starting backfill to osd.4 from (0'0,0'0] MAX to 394'8506 2024-09-18T07:44:45.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[39175]: 10.2 starting backfill to osd.4 from (0'0,0'0] MAX to 394'8624 2024-09-18T07:44:45.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[39175]: 10.c starting backfill to osd.4 from (0'0,0'0] MAX to 394'8609 2024-09-18T07:44:45.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[39175]: osdmap e397: 8 total, 8 up, 8 in 2024-09-18T07:44:45.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[39175]: 10.6 starting backfill to osd.4 from (0'0,0'0] MAX to 394'8668 2024-09-18T07:44:45.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[39175]: 10.4 starting backfill to osd.4 from (0'0,0'0] MAX to 394'8548 2024-09-18T07:44:45.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:45 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 59056/405198 objects degraded (14.575%), 26 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:45.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:45.504 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:45.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:45.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:45.606 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:45.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:45.644 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:45.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:45 smithi203 ceph-mon[32237]: pgmap v724: 158 pgs: 7 peering, 1 active+clean+remapped, 36 active+undersized, 26 active+undersized+degraded, 88 active+clean; 8.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 855 op/s; 59056/405198 objects degraded (14.575%) 2024-09-18T07:44:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:45 smithi203 ceph-mon[32237]: 10.3 scrub ok 2024-09-18T07:44:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:45 smithi203 ceph-mon[32237]: 4.1d scrub starts 2024-09-18T07:44:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:45 smithi203 ceph-mon[32237]: 4.1d scrub ok 2024-09-18T07:44:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:45 smithi203 ceph-mon[32237]: 3.6 scrub starts 2024-09-18T07:44:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:45 smithi203 ceph-mon[32237]: 3.6 scrub ok 2024-09-18T07:44:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:45 smithi203 ceph-mon[32237]: 6.1d scrub starts 2024-09-18T07:44:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:45 smithi203 ceph-mon[32237]: 6.1d scrub ok 2024-09-18T07:44:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:45 smithi203 ceph-mon[32237]: 4.5 scrub starts 2024-09-18T07:44:45.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:45 smithi203 ceph-mon[32237]: 4.5 scrub ok 2024-09-18T07:44:45.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:45 smithi203 ceph-mon[32237]: 10.1 starting backfill to osd.4 from (0'0,0'0] MAX to 394'8506 2024-09-18T07:44:45.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:45 smithi203 ceph-mon[32237]: 10.2 starting backfill to osd.4 from (0'0,0'0] MAX to 394'8624 2024-09-18T07:44:45.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:45 smithi203 ceph-mon[32237]: 10.c starting backfill to osd.4 from (0'0,0'0] MAX to 394'8609 2024-09-18T07:44:45.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:45 smithi203 ceph-mon[32237]: osdmap e397: 8 total, 8 up, 8 in 2024-09-18T07:44:45.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:45 smithi203 ceph-mon[32237]: 10.6 starting backfill to osd.4 from (0'0,0'0] MAX to 394'8668 2024-09-18T07:44:45.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:45 smithi203 ceph-mon[32237]: 10.4 starting backfill to osd.4 from (0'0,0'0] MAX to 394'8548 2024-09-18T07:44:45.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:45 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 59056/405198 objects degraded (14.575%), 26 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:45.745 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:45.780 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:45.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:45.882 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:45.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:45.921 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:45.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:45.948 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T07:44:46.022 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:46.051 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:46.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:46.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:46.154 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:46.188 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:46.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:46.290 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:46.336 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:46.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:46.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:46.374 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:44:46.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:46.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:46.469 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:46.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:46.557 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:46 smithi203 ceph-mon[32237]: 10.0 starting backfill to osd.4 from (0'0,0'0] MAX to 394'8566 2024-09-18T07:44:46.558 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:46 smithi203 ceph-mon[32237]: 10.d starting backfill to osd.4 from (0'0,0'0] MAX to 394'8589 2024-09-18T07:44:46.558 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:46 smithi203 ceph-mon[32237]: 10.e starting backfill to osd.4 from (0'0,0'0] MAX to 394'8638 2024-09-18T07:44:46.558 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:46 smithi203 ceph-mon[32237]: 6.f scrub starts 2024-09-18T07:44:46.558 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:46 smithi203 ceph-mon[32237]: 6.f scrub ok 2024-09-18T07:44:46.558 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:46 smithi203 ceph-mon[32237]: 5.f scrub starts 2024-09-18T07:44:46.559 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:46 smithi203 ceph-mon[32237]: 5.f scrub ok 2024-09-18T07:44:46.559 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:46 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:44:46.559 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:46 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:44:46.569 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:46.608 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:46.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:46.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:46.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[29408]: 10.0 starting backfill to osd.4 from (0'0,0'0] MAX to 394'8566 2024-09-18T07:44:46.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[29408]: 10.d starting backfill to osd.4 from (0'0,0'0] MAX to 394'8589 2024-09-18T07:44:46.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[29408]: 10.e starting backfill to osd.4 from (0'0,0'0] MAX to 394'8638 2024-09-18T07:44:46.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[29408]: 6.f scrub starts 2024-09-18T07:44:46.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[29408]: 6.f scrub ok 2024-09-18T07:44:46.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[29408]: 5.f scrub starts 2024-09-18T07:44:46.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[29408]: 5.f scrub ok 2024-09-18T07:44:46.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:44:46.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:44:46.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[39175]: 10.0 starting backfill to osd.4 from (0'0,0'0] MAX to 394'8566 2024-09-18T07:44:46.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[39175]: 10.d starting backfill to osd.4 from (0'0,0'0] MAX to 394'8589 2024-09-18T07:44:46.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[39175]: 10.e starting backfill to osd.4 from (0'0,0'0] MAX to 394'8638 2024-09-18T07:44:46.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[39175]: 6.f scrub starts 2024-09-18T07:44:46.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[39175]: 6.f scrub ok 2024-09-18T07:44:46.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[39175]: 5.f scrub starts 2024-09-18T07:44:46.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[39175]: 5.f scrub ok 2024-09-18T07:44:46.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:44:46.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:46 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:44:46.710 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:46.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:46.765 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:46.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:46.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:46.892 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:46.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:46.994 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:47.032 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:44:47.032 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:44:47.032 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:47.032 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.36014.0:140066 10.e e398/396 10:75f00795:::benchmark_data_smithi137_151160_object4250:head v 398'8790, mlcod=398'8786)", 2024-09-18T07:44:47.032 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:44:47.022938+0000", 2024-09-18T07:44:47.032 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0056937890000000003, 2024-09-18T07:44:47.032 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0057187959999999999, 2024-09-18T07:44:47.032 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:44:47.033 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:44:47.033 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:44:47.033 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:44:47.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:47.033 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:44:47.033 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:47.022938+0000", 2024-09-18T07:44:47.033 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:47.033 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:47.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:47.033 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:44:47.033 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:47.022938+0000", 2024-09-18T07:44:47.033 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:44:47.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:47.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:47.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:44:47.034 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:47.022939+0000", 2024-09-18T07:44:47.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1459999999999999e-06 2024-09-18T07:44:47.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:47.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:47.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:44:47.034 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:47.022955+0000", 2024-09-18T07:44:47.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6277999999999999e-05 2024-09-18T07:44:47.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:47.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:47.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:44:47.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:47.022965+0000", 2024-09-18T07:44:47.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5950000000000005e-06 2024-09-18T07:44:47.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:47.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:47.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:44:47.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:47.022971+0000", 2024-09-18T07:44:47.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2979999999999997e-06 2024-09-18T07:44:47.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:47.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:47.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:44:47.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:47.026393+0000", 2024-09-18T07:44:47.036 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0034221770000000002 2024-09-18T07:44:47.036 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:44:47.036 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:44:47.036 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:44:47.036 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:47.026412+0000", 2024-09-18T07:44:47.036 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8434e-05 2024-09-18T07:44:47.036 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:47.036 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:44:47.036 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:47.036 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:44:47.036 INFO:teuthology.orchestra.run.smithi203.stdout: ], 2024-09-18T07:44:47.036 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 1 2024-09-18T07:44:47.036 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:44:47.039 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:47.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:47.141 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:47.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:47.176 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:47.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:47.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:47.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:47.311 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:47.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:47.411 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:47.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:44:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:44:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:44:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:47 smithi137 ceph-mon[29408]: pgmap v727: 158 pgs: 7 peering, 1 active+clean+remapped, 36 active+undersized, 26 active+undersized+degraded, 88 active+clean; 8.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 45 MiB/s wr, 899 op/s; 59056/405198 objects degraded (14.575%) 2024-09-18T07:44:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:47 smithi137 ceph-mon[29408]: osdmap e398: 8 total, 8 up, 8 in 2024-09-18T07:44:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:47 smithi137 ceph-mon[29408]: 4.2 scrub starts 2024-09-18T07:44:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:47 smithi137 ceph-mon[29408]: 4.2 scrub ok 2024-09-18T07:44:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:47 smithi137 ceph-mon[39175]: pgmap v727: 158 pgs: 7 peering, 1 active+clean+remapped, 36 active+undersized, 26 active+undersized+degraded, 88 active+clean; 8.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 45 MiB/s wr, 899 op/s; 59056/405198 objects degraded (14.575%) 2024-09-18T07:44:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:47 smithi137 ceph-mon[39175]: osdmap e398: 8 total, 8 up, 8 in 2024-09-18T07:44:47.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:47 smithi137 ceph-mon[39175]: 4.2 scrub starts 2024-09-18T07:44:47.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:47 smithi137 ceph-mon[39175]: 4.2 scrub ok 2024-09-18T07:44:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:47.446 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:47.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:47.548 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:47.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:47.593 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:47.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:47.677 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:47 smithi203 ceph-mon[32237]: pgmap v727: 158 pgs: 7 peering, 1 active+clean+remapped, 36 active+undersized, 26 active+undersized+degraded, 88 active+clean; 8.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 45 MiB/s wr, 899 op/s; 59056/405198 objects degraded (14.575%) 2024-09-18T07:44:47.677 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:47 smithi203 ceph-mon[32237]: osdmap e398: 8 total, 8 up, 8 in 2024-09-18T07:44:47.677 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:47 smithi203 ceph-mon[32237]: 4.2 scrub starts 2024-09-18T07:44:47.677 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:47 smithi203 ceph-mon[32237]: 4.2 scrub ok 2024-09-18T07:44:47.694 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:47.728 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:47.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:47.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:47.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:47.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T07:44:47.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:47.865 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:47.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:47.959 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:47.960 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:44:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:44:47.966 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:47.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:47.999 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:47.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:48.100 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:48.132 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:48.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:48.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:48.234 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:48.273 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:48.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:48.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:48.375 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:48.412 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:48.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:48.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:48.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:48 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:48.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:48 smithi137 ceph-mon[29408]: pgmap v729: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 7 peering, 2 active+undersized+degraded, 144 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 4.1 MiB/s wr, 625 op/s; 57244/397467 objects degraded (14.402%); 8.0 MiB/s, 128 objects/s recovering 2024-09-18T07:44:48.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:48 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:48.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:48 smithi137 ceph-mon[39175]: pgmap v729: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 7 peering, 2 active+undersized+degraded, 144 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 4.1 MiB/s wr, 625 op/s; 57244/397467 objects degraded (14.402%); 8.0 MiB/s, 128 objects/s recovering 2024-09-18T07:44:48.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:48.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:48.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:48.631 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:44:48.632 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:44:48.651 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:48.666 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:44:48.666 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:44:48.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:48.692 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:48.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:48.705 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:44:48.705 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:44:48.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:48 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:48.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:48 smithi203 ceph-mon[32237]: pgmap v729: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 7 peering, 2 active+undersized+degraded, 144 active+clean; 8.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 4.1 MiB/s wr, 625 op/s; 57244/397467 objects degraded (14.402%); 8.0 MiB/s, 128 objects/s recovering 2024-09-18T07:44:48.735 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:44:48.736 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:44:48.751 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:48.770 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:44:48.770 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:44:48.793 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:48.809 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:44:48.809 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:44:48.826 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:48.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:48.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:48.849 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:44:48.850 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:44:48.880 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:44:48.881 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:44:48.928 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:48.962 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:48.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:48.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:49.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:49.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:49.121 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:49.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:49.222 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:49.255 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:49.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:49.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:44:49.357 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:49.396 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:49.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:49.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:49.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:49.547 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:49.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:49.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:49.649 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:49.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:49 smithi137 ceph-mon[29408]: 6.6 scrub starts 2024-09-18T07:44:49.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:49 smithi137 ceph-mon[29408]: 6.6 scrub ok 2024-09-18T07:44:49.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:49 smithi137 ceph-mon[29408]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:44:49.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:49 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/787065842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:49.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:49 smithi137 ceph-mon[39175]: 6.6 scrub starts 2024-09-18T07:44:49.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:49 smithi137 ceph-mon[39175]: 6.6 scrub ok 2024-09-18T07:44:49.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:49 smithi137 ceph-mon[39175]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:44:49.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:49 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/787065842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:49.680 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:49.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:49.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:49.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:49 smithi203 ceph-mon[32237]: 6.6 scrub starts 2024-09-18T07:44:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:49 smithi203 ceph-mon[32237]: 6.6 scrub ok 2024-09-18T07:44:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:49 smithi203 ceph-mon[32237]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:44:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:49 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/787065842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:49.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:49.817 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:49.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:49.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:49.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:49.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:49.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:49.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:50.045 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:44:50.045 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:44:50.063 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:50.095 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:50.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:50.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:50.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:50.241 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:50.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:50.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:50.343 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:50.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:50.378 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:50.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:50.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:50.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:50.517 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:50.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:50.618 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:50.651 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:50.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:50.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:50.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:50 smithi137 ceph-mon[29408]: pgmap v730: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 8.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.7 MiB/s wr, 646 op/s; 64438/394533 objects degraded (16.333%); 10 MiB/s, 167 objects/s recovering 2024-09-18T07:44:50.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:50 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 57244/397467 objects degraded (14.402%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:50.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:50 smithi137 ceph-mon[29408]: osdmap e399: 8 total, 8 up, 8 in 2024-09-18T07:44:50.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:50 smithi137 ceph-mon[39175]: pgmap v730: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 8.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.7 MiB/s wr, 646 op/s; 64438/394533 objects degraded (16.333%); 10 MiB/s, 167 objects/s recovering 2024-09-18T07:44:50.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:50 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 57244/397467 objects degraded (14.402%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:50.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:50 smithi137 ceph-mon[39175]: osdmap e399: 8 total, 8 up, 8 in 2024-09-18T07:44:50.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:50 smithi203 ceph-mon[32237]: pgmap v730: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 8.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.7 MiB/s wr, 646 op/s; 64438/394533 objects degraded (16.333%); 10 MiB/s, 167 objects/s recovering 2024-09-18T07:44:50.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:50 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 57244/397467 objects degraded (14.402%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-18T07:44:50.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:50 smithi203 ceph-mon[32237]: osdmap e399: 8 total, 8 up, 8 in 2024-09-18T07:44:50.751 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:50.753 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:44:50.781 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:50.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:50.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:44:50.798 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T07:44:50.883 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:50.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:50.917 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:50.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:51.018 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:51.056 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:51.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:51.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:51.158 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:51.191 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:51.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:51.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:51.293 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:51.327 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:51.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:51.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:51.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:51 smithi137 ceph-mon[29408]: 5.7 scrub starts 2024-09-18T07:44:51.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:51 smithi137 ceph-mon[29408]: 5.7 scrub ok 2024-09-18T07:44:51.345 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:51 smithi137 ceph-mon[39175]: 5.7 scrub starts 2024-09-18T07:44:51.345 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:51 smithi137 ceph-mon[39175]: 5.7 scrub ok 2024-09-18T07:44:51.430 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:51.453 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:51.465 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:51.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:51.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:51.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:51.608 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:51.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:51.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:51.710 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:51 smithi203 ceph-mon[32237]: 5.7 scrub starts 2024-09-18T07:44:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:51 smithi203 ceph-mon[32237]: 5.7 scrub ok 2024-09-18T07:44:51.741 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:51.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:51.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:51.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:44:51.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:51.903 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:51.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:52.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:52.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:52.044 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:52.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:52.145 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:52.180 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:52.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:52.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:52.263 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:44:52.264 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:44:52.281 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:52.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:52.328 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:52.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:52.341 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:44:52.429 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:52.465 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:52.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:52.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:52.490 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:52 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1441780103' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:44:52.490 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:52 smithi137 ceph-mon[29408]: from='client.26257 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:44:52.490 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:52 smithi137 ceph-mon[29408]: pgmap v732: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 8.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 MiB/s wr, 611 op/s; 62984/393351 objects degraded (16.012%); 19 MiB/s, 301 objects/s recovering 2024-09-18T07:44:52.491 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:52 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1441780103' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:44:52.491 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:52 smithi137 ceph-mon[39175]: from='client.26257 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:44:52.491 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:52 smithi137 ceph-mon[39175]: pgmap v732: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 8.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 MiB/s wr, 611 op/s; 62984/393351 objects degraded (16.012%); 19 MiB/s, 301 objects/s recovering 2024-09-18T07:44:52.567 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:52.605 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:52.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:52.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:52.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:52.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:52 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1441780103' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:44:52.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:52 smithi203 ceph-mon[32237]: from='client.26257 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:44:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:52 smithi203 ceph-mon[32237]: pgmap v732: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 8.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 MiB/s wr, 611 op/s; 62984/393351 objects degraded (16.012%); 19 MiB/s, 301 objects/s recovering 2024-09-18T07:44:52.730 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:52.773 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:52.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:52.812 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-18T07:44:52.813 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:44:52.813 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-18T07:44:52.813 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T07:44:52.813 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T07:44:52.813 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-18T07:44:52.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:52.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:52.921 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:52.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:53.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:53.068 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:53.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:53.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:53.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:53.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:53.215 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:53.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:53.316 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:44:53.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:53.352 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:53.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:44:53.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:53.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T07:44:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:53.489 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:53.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:53.591 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:53.619 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:53.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:53 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:44:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:53 smithi137 ceph-mon[29408]: from='client.26257 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:44:53.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:53 smithi137 ceph-mon[29408]: osdmap e400: 8 total, 8 up, 8 in 2024-09-18T07:44:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:53 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:44:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:53 smithi137 ceph-mon[39175]: from='client.26257 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:44:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:53 smithi137 ceph-mon[39175]: osdmap e400: 8 total, 8 up, 8 in 2024-09-18T07:44:53.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:53 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:44:53.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:53 smithi203 ceph-mon[32237]: from='client.26257 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:44:53.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:53 smithi203 ceph-mon[32237]: osdmap e400: 8 total, 8 up, 8 in 2024-09-18T07:44:53.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:53.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:53.760 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:53.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:53.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:53.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:53.917 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:53.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:54.018 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:44:54.055 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:54.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:54.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:44:54.157 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:54.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:54.185 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:54.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:54.286 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:54.320 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:54.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:54.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:54.392 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:54 smithi137 ceph-mon[29408]: pgmap v734: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 MiB/s wr, 1.30k op/s; 58876/377061 objects degraded (15.614%); 31 MiB/s, 500 objects/s recovering 2024-09-18T07:44:54.393 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:54 smithi137 ceph-mon[39175]: pgmap v734: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 MiB/s wr, 1.30k op/s; 58876/377061 objects degraded (15.614%); 31 MiB/s, 500 objects/s recovering 2024-09-18T07:44:54.406 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:54.421 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:54.454 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:54.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:54.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:54.472 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:44:54.473 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:44:54.473 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:54.509 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:44:54.510 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:44:54.541 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:44:54.541 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:44:54.556 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:54.586 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:44:54.586 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:44:54.604 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:54.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:54.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:54.622 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:44:54.622 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:44:54.656 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:44:54.656 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:44:54.690 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:44:54.691 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:44:54.705 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:54.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:54 smithi203 ceph-mon[32237]: pgmap v734: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 MiB/s wr, 1.30k op/s; 58876/377061 objects degraded (15.614%); 31 MiB/s, 500 objects/s recovering 2024-09-18T07:44:54.728 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:44:54.728 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:44:54.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:54.763 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:54.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:54.864 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:54.898 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:54.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:54.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:54.999 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:55.026 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:44:55.027 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:44:55.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:44:55.027 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:44:55.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.027 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162329 7.8 7:14bf872c:::benchmark_data_smithi137_74180_object57913:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:44:55.027 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.422282+0000", 2024-09-18T07:44:55.027 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 593.59923306200005, 2024-09-18T07:44:55.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.010768962, 2024-09-18T07:44:55.027 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.027 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.027 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.028 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:55.028 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:44:55.028 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:55.028 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162329 2024-09-18T07:44:55.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.028 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.028 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.028 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422282+0000", 2024-09-18T07:44:55.028 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.029 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.029 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422282+0000", 2024-09-18T07:44:55.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.029 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.029 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422284+0000", 2024-09-18T07:44:55.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4950000000000001e-06 2024-09-18T07:44:55.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.029 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.029 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422293+0000", 2024-09-18T07:44:55.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7029999999999995e-06 2024-09-18T07:44:55.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.030 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.030 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422302+0000", 2024-09-18T07:44:55.030 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5580000000000006e-06 2024-09-18T07:44:55.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.030 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.030 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422310+0000", 2024-09-18T07:44:55.030 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3010000000000007e-06 2024-09-18T07:44:55.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.030 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.030 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422376+0000", 2024-09-18T07:44:55.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6058999999999999e-05 2024-09-18T07:44:55.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:55.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.422387+0000", 2024-09-18T07:44:55.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0560999999999999e-05 2024-09-18T07:44:55.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.429278+0000", 2024-09-18T07:44:55.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0068906500000001 2024-09-18T07:44:55.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.429407+0000", 2024-09-18T07:44:55.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012889600000000001 2024-09-18T07:44:55.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:44:55.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.429537+0000", 2024-09-18T07:44:55.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013015100000000001 2024-09-18T07:44:55.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431858+0000", 2024-09-18T07:44:55.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023215480000000001 2024-09-18T07:44:55.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.033 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431902+0000", 2024-09-18T07:44:55.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3331999999999998e-05 2024-09-18T07:44:55.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.033 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431988+0000", 2024-09-18T07:44:55.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6834000000000002e-05 2024-09-18T07:44:55.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:55.033 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432895+0000", 2024-09-18T07:44:55.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00090677800000000001 2024-09-18T07:44:55.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.034 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.034 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432943+0000", 2024-09-18T07:44:55.034 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8143000000000003e-05 2024-09-18T07:44:55.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.034 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.034 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433051+0000", 2024-09-18T07:44:55.034 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107953 2024-09-18T07:44:55.034 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.034 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.034 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.035 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.035 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162333 7.8 7:1673a0b1:::benchmark_data_smithi137_74180_object57917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:44:55.035 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.424085+0000", 2024-09-18T07:44:55.035 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 593.59742999100001, 2024-09-18T07:44:55.035 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0091520620000001, 2024-09-18T07:44:55.035 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.035 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.035 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.035 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:55.035 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:44:55.035 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:55.035 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162333 2024-09-18T07:44:55.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.035 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424085+0000", 2024-09-18T07:44:55.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424085+0000", 2024-09-18T07:44:55.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424087+0000", 2024-09-18T07:44:55.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4470000000000001e-06 2024-09-18T07:44:55.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.037 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.037 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424091+0000", 2024-09-18T07:44:55.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9319999999999998e-06 2024-09-18T07:44:55.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.037 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.037 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424098+0000", 2024-09-18T07:44:55.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.187e-06 2024-09-18T07:44:55.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.037 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424104+0000", 2024-09-18T07:44:55.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8730000000000004e-06 2024-09-18T07:44:55.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424145+0000", 2024-09-18T07:44:55.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1445e-05 2024-09-18T07:44:55.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:55.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.424154+0000", 2024-09-18T07:44:55.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0729999999999994e-06 2024-09-18T07:44:55.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.430409+0000", 2024-09-18T07:44:55.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0062551159999999 2024-09-18T07:44:55.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.430534+0000", 2024-09-18T07:44:55.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012458400000000001 2024-09-18T07:44:55.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:44:55.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.430647+0000", 2024-09-18T07:44:55.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112537 2024-09-18T07:44:55.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.040 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431941+0000", 2024-09-18T07:44:55.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001294915 2024-09-18T07:44:55.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.040 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432032+0000", 2024-09-18T07:44:55.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0461e-05 2024-09-18T07:44:55.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.040 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:55.041 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433059+0000", 2024-09-18T07:44:55.041 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010266990000000001 2024-09-18T07:44:55.041 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:55.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.042 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.042 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433100+0000", 2024-09-18T07:44:55.042 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1770000000000002e-05 2024-09-18T07:44:55.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.043 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433146+0000", 2024-09-18T07:44:55.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5204000000000003e-05 2024-09-18T07:44:55.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.043 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433237+0000", 2024-09-18T07:44:55.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1818999999999996e-05 2024-09-18T07:44:55.043 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.043 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.043 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.043 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162348 7.8 7:1d66cfe5:::benchmark_data_smithi137_74180_object57932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:44:55.043 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.431806+0000", 2024-09-18T07:44:55.043 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 593.58970935399998, 2024-09-18T07:44:55.044 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0022084899999999, 2024-09-18T07:44:55.044 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.044 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.044 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.044 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:55.044 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:44:55.044 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:55.044 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162348 2024-09-18T07:44:55.044 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.044 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.044 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.044 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431806+0000", 2024-09-18T07:44:55.044 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.045 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.045 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431806+0000", 2024-09-18T07:44:55.045 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.045 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.045 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431808+0000", 2024-09-18T07:44:55.045 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6300000000000001e-06 2024-09-18T07:44:55.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.045 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.045 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431812+0000", 2024-09-18T07:44:55.045 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6140000000000002e-06 2024-09-18T07:44:55.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.046 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.046 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431823+0000", 2024-09-18T07:44:55.046 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0658e-05 2024-09-18T07:44:55.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.046 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.046 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431830+0000", 2024-09-18T07:44:55.046 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6529999999999997e-06 2024-09-18T07:44:55.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.046 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.046 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431897+0000", 2024-09-18T07:44:55.046 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7286000000000006e-05 2024-09-18T07:44:55.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.047 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:55.047 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.431909+0000", 2024-09-18T07:44:55.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1739999999999999e-05 2024-09-18T07:44:55.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.047 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.047 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431207+0000", 2024-09-18T07:44:55.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99929881799999998 2024-09-18T07:44:55.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.047 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.047 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431322+0000", 2024-09-18T07:44:55.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011495800000000001 2024-09-18T07:44:55.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.048 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:44:55.048 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.431437+0000", 2024-09-18T07:44:55.048 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114752 2024-09-18T07:44:55.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.048 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:55.048 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433064+0000", 2024-09-18T07:44:55.048 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016264280000000001 2024-09-18T07:44:55.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.048 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433276+0000", 2024-09-18T07:44:55.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021219300000000001 2024-09-18T07:44:55.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.049 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433318+0000", 2024-09-18T07:44:55.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1999999999999998e-05 2024-09-18T07:44:55.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.049 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433856+0000", 2024-09-18T07:44:55.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00053855899999999998 2024-09-18T07:44:55.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.050 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.050 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433907+0000", 2024-09-18T07:44:55.050 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0954e-05 2024-09-18T07:44:55.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.050 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.050 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.434015+0000", 2024-09-18T07:44:55.050 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107247 2024-09-18T07:44:55.050 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.050 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.050 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.051 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14865.0:162355 7.8 7:13f232f0:::benchmark_data_smithi137_74180_object57939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-18T07:44:55.051 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:35:01.440402+0000", 2024-09-18T07:44:55.051 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 593.58111361399995, 2024-09-18T07:44:55.051 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99385697299999998, 2024-09-18T07:44:55.051 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.051 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.051 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.051 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:55.051 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14865", 2024-09-18T07:44:55.051 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3164655170", 2024-09-18T07:44:55.051 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 162355 2024-09-18T07:44:55.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.051 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.052 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.052 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440402+0000", 2024-09-18T07:44:55.052 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.052 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.052 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440402+0000", 2024-09-18T07:44:55.052 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.052 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.052 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440403+0000", 2024-09-18T07:44:55.052 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4750000000000001e-06 2024-09-18T07:44:55.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440409+0000", 2024-09-18T07:44:55.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6690000000000003e-06 2024-09-18T07:44:55.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440420+0000", 2024-09-18T07:44:55.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0594000000000001e-05 2024-09-18T07:44:55.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440425+0000", 2024-09-18T07:44:55.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2460000000000003e-06 2024-09-18T07:44:55.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440471+0000", 2024-09-18T07:44:55.054 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6397000000000003e-05 2024-09-18T07:44:55.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:44:55.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:01.440479+0000", 2024-09-18T07:44:55.054 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8220000000000007e-06 2024-09-18T07:44:55.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432099+0000", 2024-09-18T07:44:55.054 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99161995000000003 2024-09-18T07:44:55.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.055 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.055 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432216+0000", 2024-09-18T07:44:55.055 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117404 2024-09-18T07:44:55.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.055 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4,7", 2024-09-18T07:44:55.055 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.432332+0000", 2024-09-18T07:44:55.055 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115939 2024-09-18T07:44:55.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.055 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.055 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433814+0000", 2024-09-18T07:44:55.055 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00148196 2024-09-18T07:44:55.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.056 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.056 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:55.056 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.433827+0000", 2024-09-18T07:44:55.056 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2583e-05 2024-09-18T07:44:55.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.056 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.056 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.056 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.434071+0000", 2024-09-18T07:44:55.056 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024380199999999999 2024-09-18T07:44:55.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.056 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.056 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.056 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.434123+0000", 2024-09-18T07:44:55.056 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2419000000000001e-05 2024-09-18T07:44:55.057 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.057 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.057 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.434167+0000", 2024-09-18T07:44:55.057 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3600000000000003e-05 2024-09-18T07:44:55.057 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.057 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.057 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:35:02.434259+0000", 2024-09-18T07:44:55.057 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2113000000000004e-05 2024-09-18T07:44:55.057 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.057 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.057 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.058 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.058 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:111049 10.3 10:c431df93:::benchmark_data_smithi137_151160_object111048:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-18T07:44:55.058 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:19.395561+0000", 2024-09-18T07:44:55.058 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 35.625954096999997, 2024-09-18T07:44:55.058 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.091648226999999999, 2024-09-18T07:44:55.058 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.058 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.058 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.058 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:55.058 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:44:55.058 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:55.058 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 111049 2024-09-18T07:44:55.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.059 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.059 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.059 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.395561+0000", 2024-09-18T07:44:55.059 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.059 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.059 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.059 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.395561+0000", 2024-09-18T07:44:55.059 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.059 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.059 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.059 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.395564+0000", 2024-09-18T07:44:55.060 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2630000000000002e-06 2024-09-18T07:44:55.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.060 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.060 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.395619+0000", 2024-09-18T07:44:55.060 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4979999999999999e-05 2024-09-18T07:44:55.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.060 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.060 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.395635+0000", 2024-09-18T07:44:55.060 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6563999999999999e-05 2024-09-18T07:44:55.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.061 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.061 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.395642+0000", 2024-09-18T07:44:55.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3819999999999998e-06 2024-09-18T07:44:55.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.061 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.061 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.395999+0000", 2024-09-18T07:44:55.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00035611200000000001 2024-09-18T07:44:55.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.061 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.061 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.396138+0000", 2024-09-18T07:44:55.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013963699999999999 2024-09-18T07:44:55.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.062 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-09-18T07:44:55.062 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.396286+0000", 2024-09-18T07:44:55.062 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014729399999999999 2024-09-18T07:44:55.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.062 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.062 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.403262+0000", 2024-09-18T07:44:55.062 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0069766050000000003 2024-09-18T07:44:55.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.062 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:55.062 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.418043+0000", 2024-09-18T07:44:55.062 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014780636999999999 2024-09-18T07:44:55.063 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.063 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.063 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.063 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.429278+0000", 2024-09-18T07:44:55.063 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011235716 2024-09-18T07:44:55.063 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.063 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.063 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.063 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.487153+0000", 2024-09-18T07:44:55.063 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.057874307 2024-09-18T07:44:55.063 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.063 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.063 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.063 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:19.487210+0000", 2024-09-18T07:44:55.063 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6730000000000001e-05 2024-09-18T07:44:55.063 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.064 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.064 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.064 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:136160 10.3 10:c35fd278:::benchmark_data_smithi137_151160_object344:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-18T07:44:55.064 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:41.504470+0000", 2024-09-18T07:44:55.064 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.517045162000001, 2024-09-18T07:44:55.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.071051911999999995, 2024-09-18T07:44:55.064 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.064 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.064 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.064 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:55.064 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:44:55.064 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:55.065 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136160 2024-09-18T07:44:55.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.065 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.504470+0000", 2024-09-18T07:44:55.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.504470+0000", 2024-09-18T07:44:55.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.504473+0000", 2024-09-18T07:44:55.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3489999999999999e-06 2024-09-18T07:44:55.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.504479+0000", 2024-09-18T07:44:55.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2700000000000001e-06 2024-09-18T07:44:55.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.504492+0000", 2024-09-18T07:44:55.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3251999999999999e-05 2024-09-18T07:44:55.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.504498+0000", 2024-09-18T07:44:55.067 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8950000000000003e-06 2024-09-18T07:44:55.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.067 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.504542+0000", 2024-09-18T07:44:55.067 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4261999999999997e-05 2024-09-18T07:44:55.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.067 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.504629+0000", 2024-09-18T07:44:55.067 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6388e-05 2024-09-18T07:44:55.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-09-18T07:44:55.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.504710+0000", 2024-09-18T07:44:55.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0978999999999998e-05 2024-09-18T07:44:55.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.068 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.510811+0000", 2024-09-18T07:44:55.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.006101647 2024-09-18T07:44:55.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.068 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:55.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.510828+0000", 2024-09-18T07:44:55.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.647e-05 2024-09-18T07:44:55.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.575430+0000", 2024-09-18T07:44:55.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.064602441999999996 2024-09-18T07:44:55.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.575480+0000", 2024-09-18T07:44:55.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0176000000000002e-05 2024-09-18T07:44:55.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.575522+0000", 2024-09-18T07:44:55.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1782000000000001e-05 2024-09-18T07:44:55.069 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.070 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.070 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.070 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:136161 10.3 10:cb09fb95:::benchmark_data_smithi137_151160_object345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-18T07:44:55.070 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:41.504521+0000", 2024-09-18T07:44:55.070 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.516994312, 2024-09-18T07:44:55.070 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.071568081000000006, 2024-09-18T07:44:55.070 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.070 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.070 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.070 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:55.070 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:44:55.070 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:55.071 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136161 2024-09-18T07:44:55.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.071 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.504521+0000", 2024-09-18T07:44:55.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.504521+0000", 2024-09-18T07:44:55.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.504522+0000", 2024-09-18T07:44:55.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8599999999999997e-07 2024-09-18T07:44:55.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.504525+0000", 2024-09-18T07:44:55.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1020000000000001e-06 2024-09-18T07:44:55.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.504532+0000", 2024-09-18T07:44:55.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0180000000000004e-06 2024-09-18T07:44:55.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.073 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.504537+0000", 2024-09-18T07:44:55.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5029999999999997e-06 2024-09-18T07:44:55.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.073 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.508681+0000", 2024-09-18T07:44:55.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041435159999999999 2024-09-18T07:44:55.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.508788+0000", 2024-09-18T07:44:55.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000106564 2024-09-18T07:44:55.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-09-18T07:44:55.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.508868+0000", 2024-09-18T07:44:55.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0637999999999996e-05 2024-09-18T07:44:55.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:55.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.510882+0000", 2024-09-18T07:44:55.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020138019999999999 2024-09-18T07:44:55.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.511310+0000", 2024-09-18T07:44:55.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00042822299999999998 2024-09-18T07:44:55.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.576008+0000", 2024-09-18T07:44:55.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.064697401000000002 2024-09-18T07:44:55.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.576054+0000", 2024-09-18T07:44:55.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6252999999999999e-05 2024-09-18T07:44:55.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.576089+0000", 2024-09-18T07:44:55.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5274999999999997e-05 2024-09-18T07:44:55.076 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.076 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.076 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.076 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:136168 10.3 10:c75fc592:::benchmark_data_smithi137_151160_object352:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-18T07:44:55.076 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:41.519768+0000", 2024-09-18T07:44:55.076 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.501747321, 2024-09-18T07:44:55.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.118074053, 2024-09-18T07:44:55.077 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.077 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.077 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.077 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:55.077 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:44:55.077 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:55.077 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136168 2024-09-18T07:44:55.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.077 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.519768+0000", 2024-09-18T07:44:55.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.519768+0000", 2024-09-18T07:44:55.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.519769+0000", 2024-09-18T07:44:55.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.35e-06 2024-09-18T07:44:55.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.519774+0000", 2024-09-18T07:44:55.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7439999999999998e-06 2024-09-18T07:44:55.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.519785+0000", 2024-09-18T07:44:55.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1192999999999999e-05 2024-09-18T07:44:55.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.519791+0000", 2024-09-18T07:44:55.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.361e-06 2024-09-18T07:44:55.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.519831+0000", 2024-09-18T07:44:55.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0568000000000002e-05 2024-09-18T07:44:55.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.519936+0000", 2024-09-18T07:44:55.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104959 2024-09-18T07:44:55.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-09-18T07:44:55.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.520033+0000", 2024-09-18T07:44:55.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6766000000000002e-05 2024-09-18T07:44:55.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:55.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.522582+0000", 2024-09-18T07:44:55.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025488849999999999 2024-09-18T07:44:55.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.522758+0000", 2024-09-18T07:44:55.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000176403 2024-09-18T07:44:55.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.637726+0000", 2024-09-18T07:44:55.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.114967871 2024-09-18T07:44:55.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.637798+0000", 2024-09-18T07:44:55.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1569000000000005e-05 2024-09-18T07:44:55.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:41.637842+0000", 2024-09-18T07:44:55.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4384000000000001e-05 2024-09-18T07:44:55.082 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.082 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.082 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.083 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:55.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:55.084 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:137016 10.f e394/364 10:f277fab4:::benchmark_data_smithi137_151160_object1200:head v 394'8455, mlcod=394'8453)", 2024-09-18T07:44:55.084 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:42.285824+0000", 2024-09-18T07:44:55.084 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.73569183, 2024-09-18T07:44:55.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.070236167000000002, 2024-09-18T07:44:55.084 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.084 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.084 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.084 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.285824+0000", 2024-09-18T07:44:55.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.285824+0000", 2024-09-18T07:44:55.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.285824+0000", 2024-09-18T07:44:55.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0400000000000005e-07 2024-09-18T07:44:55.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.285830+0000", 2024-09-18T07:44:55.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5280000000000003e-06 2024-09-18T07:44:55.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.285836+0000", 2024-09-18T07:44:55.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.55e-06 2024-09-18T07:44:55.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.285842+0000", 2024-09-18T07:44:55.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6629999999999998e-06 2024-09-18T07:44:55.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.285899+0000", 2024-09-18T07:44:55.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6777000000000002e-05 2024-09-18T07:44:55.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.285917+0000", 2024-09-18T07:44:55.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7978999999999998e-05 2024-09-18T07:44:55.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.355972+0000", 2024-09-18T07:44:55.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.070055111000000003 2024-09-18T07:44:55.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.356060+0000", 2024-09-18T07:44:55.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7755000000000003e-05 2024-09-18T07:44:55.088 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.088 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.088 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.088 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:139968 10.5 10:a7a8a7c0:::benchmark_data_smithi137_151160_object4152:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-18T07:44:55.088 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:46.868641+0000", 2024-09-18T07:44:55.088 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 8.1528739310000002, 2024-09-18T07:44:55.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.076556908000000007, 2024-09-18T07:44:55.089 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.089 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.089 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.089 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:55.089 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:44:55.089 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:55.089 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 139968 2024-09-18T07:44:55.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.089 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:46.868641+0000", 2024-09-18T07:44:55.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:46.868641+0000", 2024-09-18T07:44:55.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:46.868643+0000", 2024-09-18T07:44:55.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0150000000000002e-06 2024-09-18T07:44:55.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:46.868649+0000", 2024-09-18T07:44:55.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6200000000000004e-06 2024-09-18T07:44:55.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:46.868661+0000", 2024-09-18T07:44:55.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2017e-05 2024-09-18T07:44:55.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:46.868669+0000", 2024-09-18T07:44:55.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6569999999999997e-06 2024-09-18T07:44:55.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:46.868732+0000", 2024-09-18T07:44:55.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3442999999999995e-05 2024-09-18T07:44:55.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:46.868857+0000", 2024-09-18T07:44:55.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012497800000000001 2024-09-18T07:44:55.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:44:55.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:46.868985+0000", 2024-09-18T07:44:55.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000127878 2024-09-18T07:44:55.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:46.875170+0000", 2024-09-18T07:44:55.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0061849219999999998 2024-09-18T07:44:55.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:55.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:46.875213+0000", 2024-09-18T07:44:55.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3448000000000001e-05 2024-09-18T07:44:55.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:46.945099+0000", 2024-09-18T07:44:55.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.069885376999999999 2024-09-18T07:44:55.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:46.945153+0000", 2024-09-18T07:44:55.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4468000000000001e-05 2024-09-18T07:44:55.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:46.945198+0000", 2024-09-18T07:44:55.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5084999999999999e-05 2024-09-18T07:44:55.094 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.094 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.094 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.095 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:149049 10.5 10:a91006c2:::benchmark_data_smithi137_151160_object13233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-18T07:44:55.095 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:54.698046+0000", 2024-09-18T07:44:55.095 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.32346923399999999, 2024-09-18T07:44:55.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01983646, 2024-09-18T07:44:55.095 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.095 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.095 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.095 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:55.095 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:44:55.095 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:55.095 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 149049 2024-09-18T07:44:55.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.095 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.698046+0000", 2024-09-18T07:44:55.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.698046+0000", 2024-09-18T07:44:55.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.698048+0000", 2024-09-18T07:44:55.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0590000000000001e-06 2024-09-18T07:44:55.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.097 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.097 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.698055+0000", 2024-09-18T07:44:55.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3369999999999996e-06 2024-09-18T07:44:55.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.097 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.097 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.698068+0000", 2024-09-18T07:44:55.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3294e-05 2024-09-18T07:44:55.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.097 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.698075+0000", 2024-09-18T07:44:55.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3880000000000002e-06 2024-09-18T07:44:55.098 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.698210+0000", 2024-09-18T07:44:55.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0001351 2024-09-18T07:44:55.098 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.698503+0000", 2024-09-18T07:44:55.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029274100000000001 2024-09-18T07:44:55.098 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:44:55.099 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.698822+0000", 2024-09-18T07:44:55.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00031871599999999998 2024-09-18T07:44:55.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.099 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:55.099 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.708880+0000", 2024-09-18T07:44:55.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010057875000000001 2024-09-18T07:44:55.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.099 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.099 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.708929+0000", 2024-09-18T07:44:55.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9187000000000002e-05 2024-09-18T07:44:55.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.717570+0000", 2024-09-18T07:44:55.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0086408040000000002 2024-09-18T07:44:55.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.717672+0000", 2024-09-18T07:44:55.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102481 2024-09-18T07:44:55.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.717883+0000", 2024-09-18T07:44:55.101 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000210478 2024-09-18T07:44:55.101 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.101 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.101 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.101 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.101 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:149061 10.f e400/364 10:f2679a66:::benchmark_data_smithi137_151160_object13245:head v 400'9194, mlcod=400'9192)", 2024-09-18T07:44:55.101 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:54.717042+0000", 2024-09-18T07:44:55.101 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.30447348400000002, 2024-09-18T07:44:55.101 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.024418248, 2024-09-18T07:44:55.101 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.101 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.101 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.102 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.717042+0000", 2024-09-18T07:44:55.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.717042+0000", 2024-09-18T07:44:55.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.717045+0000", 2024-09-18T07:44:55.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1480000000000002e-06 2024-09-18T07:44:55.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.103 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.717058+0000", 2024-09-18T07:44:55.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2581e-05 2024-09-18T07:44:55.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.103 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.717080+0000", 2024-09-18T07:44:55.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1923e-05 2024-09-18T07:44:55.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.104 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.717090+0000", 2024-09-18T07:44:55.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0795000000000001e-05 2024-09-18T07:44:55.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.104 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.717447+0000", 2024-09-18T07:44:55.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00035664699999999998 2024-09-18T07:44:55.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.104 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.717469+0000", 2024-09-18T07:44:55.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2002e-05 2024-09-18T07:44:55.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.105 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.741312+0000", 2024-09-18T07:44:55.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.023842675000000001 2024-09-18T07:44:55.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.105 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.741460+0000", 2024-09-18T07:44:55.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014847699999999999 2024-09-18T07:44:55.105 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.105 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.105 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.106 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:149068 10.f e400/364 10:f5af6417:::benchmark_data_smithi137_151160_object13252:head v 400'9195, mlcod=400'9192)", 2024-09-18T07:44:55.106 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:54.718915+0000", 2024-09-18T07:44:55.106 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.30260011999999997, 2024-09-18T07:44:55.106 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022719881000000001, 2024-09-18T07:44:55.106 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.106 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.106 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.106 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.106 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.106 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.718915+0000", 2024-09-18T07:44:55.106 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.718915+0000", 2024-09-18T07:44:55.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.718917+0000", 2024-09-18T07:44:55.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.376e-06 2024-09-18T07:44:55.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.718924+0000", 2024-09-18T07:44:55.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7389999999999999e-06 2024-09-18T07:44:55.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.718933+0000", 2024-09-18T07:44:55.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8920000000000006e-06 2024-09-18T07:44:55.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.718940+0000", 2024-09-18T07:44:55.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6710000000000002e-06 2024-09-18T07:44:55.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.730727+0000", 2024-09-18T07:44:55.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011786613 2024-09-18T07:44:55.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.730742+0000", 2024-09-18T07:44:55.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5478999999999999e-05 2024-09-18T07:44:55.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.741466+0000", 2024-09-18T07:44:55.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010723642 2024-09-18T07:44:55.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.741635+0000", 2024-09-18T07:44:55.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000169469 2024-09-18T07:44:55.110 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.110 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.110 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.110 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:149091 10.3 10:cd17868e:::benchmark_data_smithi137_151160_object13275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-18T07:44:55.110 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:54.739268+0000", 2024-09-18T07:44:55.110 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.28224758700000002, 2024-09-18T07:44:55.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022084919000000001, 2024-09-18T07:44:55.110 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.110 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.110 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.111 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:55.111 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:44:55.111 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:55.111 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 149091 2024-09-18T07:44:55.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.111 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.739268+0000", 2024-09-18T07:44:55.111 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.739268+0000", 2024-09-18T07:44:55.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.112 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.112 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.739271+0000", 2024-09-18T07:44:55.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6109999999999998e-06 2024-09-18T07:44:55.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.112 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.112 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.739280+0000", 2024-09-18T07:44:55.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1820000000000006e-06 2024-09-18T07:44:55.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.739294+0000", 2024-09-18T07:44:55.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4676e-05 2024-09-18T07:44:55.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.739303+0000", 2024-09-18T07:44:55.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3669999999999997e-06 2024-09-18T07:44:55.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.742731+0000", 2024-09-18T07:44:55.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034279549999999999 2024-09-18T07:44:55.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.114 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.742849+0000", 2024-09-18T07:44:55.114 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118709 2024-09-18T07:44:55.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-09-18T07:44:55.114 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.742963+0000", 2024-09-18T07:44:55.114 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011363599999999999 2024-09-18T07:44:55.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.747441+0000", 2024-09-18T07:44:55.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004478437 2024-09-18T07:44:55.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.115 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:55.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.752198+0000", 2024-09-18T07:44:55.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047562899999999998 2024-09-18T07:44:55.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.115 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.761258+0000", 2024-09-18T07:44:55.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.009060841 2024-09-18T07:44:55.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.116 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.116 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.761310+0000", 2024-09-18T07:44:55.116 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1718999999999997e-05 2024-09-18T07:44:55.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.116 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.116 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.761353+0000", 2024-09-18T07:44:55.116 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2496000000000001e-05 2024-09-18T07:44:55.116 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.116 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.116 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.116 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:149093 10.5 10:a49e2800:::benchmark_data_smithi137_151160_object13277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-18T07:44:55.117 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:54.744638+0000", 2024-09-18T07:44:55.117 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.276877647, 2024-09-18T07:44:55.117 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.034823381, 2024-09-18T07:44:55.117 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.117 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.117 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.117 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:55.117 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:44:55.117 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:55.117 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 149093 2024-09-18T07:44:55.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.117 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.117 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.118 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.118 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.744638+0000", 2024-09-18T07:44:55.118 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.118 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.118 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.118 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.744638+0000", 2024-09-18T07:44:55.118 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.118 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.118 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.118 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.744640+0000", 2024-09-18T07:44:55.118 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7410000000000001e-06 2024-09-18T07:44:55.118 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.119 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.119 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.744648+0000", 2024-09-18T07:44:55.119 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8609999999999998e-06 2024-09-18T07:44:55.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.119 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.119 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.119 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.744666+0000", 2024-09-18T07:44:55.119 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7643e-05 2024-09-18T07:44:55.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.119 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.119 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.119 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.744674+0000", 2024-09-18T07:44:55.119 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4030000000000007e-06 2024-09-18T07:44:55.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.120 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.120 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.120 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.744734+0000", 2024-09-18T07:44:55.120 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9663000000000001e-05 2024-09-18T07:44:55.120 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.120 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.120 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.120 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.744854+0000", 2024-09-18T07:44:55.120 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012033500000000001 2024-09-18T07:44:55.120 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.120 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.120 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:44:55.121 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.744967+0000", 2024-09-18T07:44:55.121 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112936 2024-09-18T07:44:55.121 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.121 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.121 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.121 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.762948+0000", 2024-09-18T07:44:55.121 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017981146 2024-09-18T07:44:55.121 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.121 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.121 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:55.121 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.768402+0000", 2024-09-18T07:44:55.121 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054538870000000001 2024-09-18T07:44:55.121 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.121 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.122 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.122 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.779370+0000", 2024-09-18T07:44:55.122 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010967549 2024-09-18T07:44:55.122 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.122 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.122 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.122 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.779418+0000", 2024-09-18T07:44:55.122 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8547999999999997e-05 2024-09-18T07:44:55.122 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.122 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.122 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.122 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.779461+0000", 2024-09-18T07:44:55.122 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2669000000000001e-05 2024-09-18T07:44:55.122 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.123 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.123 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.123 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.123 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.123 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:149097 10.5 10:a5615ed4:::benchmark_data_smithi137_151160_object13281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-18T07:44:55.123 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:54.748236+0000", 2024-09-18T07:44:55.123 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.273279785, 2024-09-18T07:44:55.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031417698000000001, 2024-09-18T07:44:55.123 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.123 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.123 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.123 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:55.123 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:44:55.124 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:55.124 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 149097 2024-09-18T07:44:55.124 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.124 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.124 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.124 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.748236+0000", 2024-09-18T07:44:55.124 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.124 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.124 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.124 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.748236+0000", 2024-09-18T07:44:55.124 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.125 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.125 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.748239+0000", 2024-09-18T07:44:55.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.96e-06 2024-09-18T07:44:55.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.125 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.125 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.748245+0000", 2024-09-18T07:44:55.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5309999999999998e-06 2024-09-18T07:44:55.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.125 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.125 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.748262+0000", 2024-09-18T07:44:55.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6452000000000001e-05 2024-09-18T07:44:55.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.126 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.126 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.748269+0000", 2024-09-18T07:44:55.126 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6399999999999997e-06 2024-09-18T07:44:55.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.127 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.127 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.749063+0000", 2024-09-18T07:44:55.127 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00079347499999999997 2024-09-18T07:44:55.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.127 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.127 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.749195+0000", 2024-09-18T07:44:55.127 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013242899999999999 2024-09-18T07:44:55.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.127 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:44:55.127 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.749312+0000", 2024-09-18T07:44:55.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116966 2024-09-18T07:44:55.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.128 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.128 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.762986+0000", 2024-09-18T07:44:55.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013674316000000001 2024-09-18T07:44:55.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.128 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:55.128 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.768408+0000", 2024-09-18T07:44:55.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054221349999999998 2024-09-18T07:44:55.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.128 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.129 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.779586+0000", 2024-09-18T07:44:55.129 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011177071 2024-09-18T07:44:55.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.129 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.129 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.129 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.779623+0000", 2024-09-18T07:44:55.129 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7781000000000003e-05 2024-09-18T07:44:55.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.129 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.129 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.129 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.779653+0000", 2024-09-18T07:44:55.129 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9941999999999999e-05 2024-09-18T07:44:55.129 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.130 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.130 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.130 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.130 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:149204 10.5 10:ad2f2a4e:::benchmark_data_smithi137_151160_object13388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-18T07:44:55.130 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:54.840175+0000", 2024-09-18T07:44:55.130 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.18134020000000001, 2024-09-18T07:44:55.130 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020358113000000001, 2024-09-18T07:44:55.130 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.130 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.130 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.130 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:55.130 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:44:55.130 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:55.131 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 149204 2024-09-18T07:44:55.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.131 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.131 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.131 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.840175+0000", 2024-09-18T07:44:55.131 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.131 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.131 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.840175+0000", 2024-09-18T07:44:55.131 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.840177+0000", 2024-09-18T07:44:55.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0559999999999999e-06 2024-09-18T07:44:55.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.840184+0000", 2024-09-18T07:44:55.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3720000000000001e-06 2024-09-18T07:44:55.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.840200+0000", 2024-09-18T07:44:55.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6554000000000001e-05 2024-09-18T07:44:55.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.133 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.133 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.840208+0000", 2024-09-18T07:44:55.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9740000000000002e-06 2024-09-18T07:44:55.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.133 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.133 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.841795+0000", 2024-09-18T07:44:55.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001586716 2024-09-18T07:44:55.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.133 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.134 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.841899+0000", 2024-09-18T07:44:55.134 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010433300000000001 2024-09-18T07:44:55.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.134 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.134 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:44:55.134 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.842011+0000", 2024-09-18T07:44:55.134 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111864 2024-09-18T07:44:55.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.134 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.134 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:55.134 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.846928+0000", 2024-09-18T07:44:55.134 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004917093 2024-09-18T07:44:55.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.135 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.856845+0000", 2024-09-18T07:44:55.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0099167590000000007 2024-09-18T07:44:55.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.135 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.860441+0000", 2024-09-18T07:44:55.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035963129999999999 2024-09-18T07:44:55.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.135 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.860492+0000", 2024-09-18T07:44:55.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0880000000000001e-05 2024-09-18T07:44:55.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.136 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.136 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.860533+0000", 2024-09-18T07:44:55.136 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1199000000000003e-05 2024-09-18T07:44:55.136 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.136 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.136 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.136 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:149278 10.3 10:ca64dc5c:::benchmark_data_smithi137_151160_object13462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-18T07:44:55.136 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:54.901602+0000", 2024-09-18T07:44:55.136 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.11991298, 2024-09-18T07:44:55.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022230364999999998, 2024-09-18T07:44:55.137 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.137 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.137 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.137 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:55.137 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:44:55.137 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:55.137 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 149278 2024-09-18T07:44:55.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.137 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.137 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.901602+0000", 2024-09-18T07:44:55.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.901602+0000", 2024-09-18T07:44:55.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.901622+0000", 2024-09-18T07:44:55.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9145000000000001e-05 2024-09-18T07:44:55.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.901648+0000", 2024-09-18T07:44:55.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6421e-05 2024-09-18T07:44:55.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.901659+0000", 2024-09-18T07:44:55.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0547000000000001e-05 2024-09-18T07:44:55.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.901666+0000", 2024-09-18T07:44:55.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1110000000000002e-06 2024-09-18T07:44:55.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.140 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.912251+0000", 2024-09-18T07:44:55.140 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010585386 2024-09-18T07:44:55.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.140 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.912364+0000", 2024-09-18T07:44:55.140 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112792 2024-09-18T07:44:55.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-09-18T07:44:55.140 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.912477+0000", 2024-09-18T07:44:55.140 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112773 2024-09-18T07:44:55.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.141 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.141 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:55.141 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.918695+0000", 2024-09-18T07:44:55.141 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0062184620000000001 2024-09-18T07:44:55.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.141 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.141 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.141 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.918743+0000", 2024-09-18T07:44:55.141 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7998999999999999e-05 2024-09-18T07:44:55.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.141 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.141 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.141 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.923568+0000", 2024-09-18T07:44:55.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048248120000000004 2024-09-18T07:44:55.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.142 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.923674+0000", 2024-09-18T07:44:55.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105896 2024-09-18T07:44:55.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.142 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.923833+0000", 2024-09-18T07:44:55.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015902099999999999 2024-09-18T07:44:55.142 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.142 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.142 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.143 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:149325 10.5 10:aea94993:::benchmark_data_smithi137_151160_object13509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-18T07:44:55.143 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:54.939407+0000", 2024-09-18T07:44:55.143 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.082108090999999994, 2024-09-18T07:44:55.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.029199979000000001, 2024-09-18T07:44:55.143 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.143 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.143 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.143 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:55.143 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:44:55.143 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:55.143 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 149325 2024-09-18T07:44:55.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.144 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.939407+0000", 2024-09-18T07:44:55.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.939407+0000", 2024-09-18T07:44:55.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.939408+0000", 2024-09-18T07:44:55.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0900000000000001e-07 2024-09-18T07:44:55.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.145 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.939411+0000", 2024-09-18T07:44:55.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3079999999999999e-06 2024-09-18T07:44:55.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.145 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.939417+0000", 2024-09-18T07:44:55.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7420000000000003e-06 2024-09-18T07:44:55.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.939422+0000", 2024-09-18T07:44:55.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5920000000000002e-06 2024-09-18T07:44:55.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.949321+0000", 2024-09-18T07:44:55.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0098990109999999992 2024-09-18T07:44:55.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.949419+0000", 2024-09-18T07:44:55.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7932999999999999e-05 2024-09-18T07:44:55.147 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.147 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.147 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:44:55.147 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.949519+0000", 2024-09-18T07:44:55.147 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100196 2024-09-18T07:44:55.147 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.147 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.147 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:55.147 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.956932+0000", 2024-09-18T07:44:55.147 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0074130000000000003 2024-09-18T07:44:55.147 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.147 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.147 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.147 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.961745+0000", 2024-09-18T07:44:55.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004813019 2024-09-18T07:44:55.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.148 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.148 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.967469+0000", 2024-09-18T07:44:55.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057241870000000004 2024-09-18T07:44:55.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.148 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.148 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.968563+0000", 2024-09-18T07:44:55.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010938149999999999 2024-09-18T07:44:55.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.149 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.149 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.968607+0000", 2024-09-18T07:44:55.149 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4467e-05 2024-09-18T07:44:55.149 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.149 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.149 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.149 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:149338 10.5 10:ad852ff6:::benchmark_data_smithi137_151160_object13522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-18T07:44:55.149 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:54.948363+0000", 2024-09-18T07:44:55.149 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.073152528999999994, 2024-09-18T07:44:55.149 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020493977, 2024-09-18T07:44:55.149 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:55.150 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:55.150 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:44:55.150 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:44:55.150 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:44:55.150 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:44:55.150 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 149338 2024-09-18T07:44:55.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.150 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:55.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.150 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:55.150 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.948363+0000", 2024-09-18T07:44:55.150 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:55.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.948363+0000", 2024-09-18T07:44:55.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:55.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:55.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.948364+0000", 2024-09-18T07:44:55.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0300000000000001e-06 2024-09-18T07:44:55.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:55.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.948368+0000", 2024-09-18T07:44:55.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3059999999999999e-06 2024-09-18T07:44:55.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.152 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:55.152 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.948376+0000", 2024-09-18T07:44:55.152 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1200000000000002e-06 2024-09-18T07:44:55.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.152 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:55.152 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.948382+0000", 2024-09-18T07:44:55.152 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1430000000000003e-06 2024-09-18T07:44:55.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.152 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:55.152 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.953229+0000", 2024-09-18T07:44:55.152 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048460899999999999 2024-09-18T07:44:55.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.153 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.153 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:55.153 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.953346+0000", 2024-09-18T07:44:55.153 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117141 2024-09-18T07:44:55.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.153 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.153 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:44:55.153 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.953458+0000", 2024-09-18T07:44:55.153 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112444 2024-09-18T07:44:55.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.153 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.153 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:44:55.153 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.958005+0000", 2024-09-18T07:44:55.154 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045464980000000004 2024-09-18T07:44:55.154 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.154 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.154 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.154 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.967428+0000", 2024-09-18T07:44:55.154 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.009423776 2024-09-18T07:44:55.154 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.154 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.154 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:44:55.154 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.968766+0000", 2024-09-18T07:44:55.154 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001338007 2024-09-18T07:44:55.154 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.154 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.154 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:44:55.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.968818+0000", 2024-09-18T07:44:55.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2070000000000001e-05 2024-09-18T07:44:55.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:55.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:55.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:44:55.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:54.968857+0000", 2024-09-18T07:44:55.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8352000000000001e-05 2024-09-18T07:44:55.155 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.155 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.155 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.155 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:55.155 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:55.155 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:44:55.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:55.212 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:55.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:55.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:55.314 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:55.347 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:55.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:55.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:55.449 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:55.482 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:55.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:55.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:55.584 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:55.619 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:55.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:55.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:55.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:55 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3138066038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:55.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:55 smithi137 ceph-mon[29408]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:44:55.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:55 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 58876/377061 objects degraded (15.614%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:44:55.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:55 smithi137 ceph-mon[29408]: osdmap e401: 8 total, 8 up, 8 in 2024-09-18T07:44:55.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:55 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3138066038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:55 smithi137 ceph-mon[39175]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:44:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:55 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 58876/377061 objects degraded (15.614%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:44:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:55 smithi137 ceph-mon[39175]: osdmap e401: 8 total, 8 up, 8 in 2024-09-18T07:44:55.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:55 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3138066038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:44:55.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:55 smithi203 ceph-mon[32237]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:44:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:55 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 58876/377061 objects degraded (15.614%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:44:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:55 smithi203 ceph-mon[32237]: osdmap e401: 8 total, 8 up, 8 in 2024-09-18T07:44:55.721 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:44:55.756 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:55.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:55.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:44:55.823 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:44:55.858 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:55.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:55.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:56.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:56.074 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:56.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:56.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:56.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:44:56.214 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:56.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:56.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:44:56.316 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:56.355 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:56.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:56.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:56.457 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:44:56.496 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:56.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:44:56.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T07:44:56.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:44:56.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:56.634 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:56.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:44:56.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:56 smithi137 ceph-mon[29408]: pgmap v736: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 971 op/s; 58876/377061 objects degraded (15.614%); 28 MiB/s, 455 objects/s recovering 2024-09-18T07:44:56.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:56 smithi137 ceph-mon[39175]: pgmap v736: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 971 op/s; 58876/377061 objects degraded (15.614%); 28 MiB/s, 455 objects/s recovering 2024-09-18T07:44:56.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:56 smithi203 ceph-mon[32237]: pgmap v736: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 7.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 971 op/s; 58876/377061 objects degraded (15.614%); 28 MiB/s, 455 objects/s recovering 2024-09-18T07:44:56.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:56.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:56.778 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:56.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:56.879 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:56.911 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:56.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:56.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:57.013 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:57.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:57.049 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:57.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:57.150 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:57.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:44:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:44:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:44:57.186 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:57.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:57.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:57.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:57.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:57.322 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:57.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:57.423 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:57.457 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:57.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:57.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:57.559 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:57.596 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:57.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:57.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:57.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:57.734 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:57.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:57.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:57.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:57.870 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:44:57.870 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "" 2024-09-18T07:44:57.870 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:44:57.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:57.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:57.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:57.960 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:44:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:44:57.989 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:58.022 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:58.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:58.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:58.124 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:44:58.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:58.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:58.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:44:58.181 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:58.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:58.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:58.299 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:58.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:58.301 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:57 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:44:58.302 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:57 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:44:58.302 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:57 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:44:58.302 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:57 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:44:58.401 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:58.436 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:58.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:58.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:57 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:44:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:57 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:44:58.539 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:58.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:58.578 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:58.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:58.679 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:58.710 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:58.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:58.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:58.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:44:58.848 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:58.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:58.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:44:58.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:58.985 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:58.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:58.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:59.009 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:58 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:59.009 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:44:58 smithi137 ceph-mon[29408]: pgmap v737: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.33k op/s; 55064/363699 objects degraded (15.140%); 40 MiB/s, 632 objects/s recovering 2024-09-18T07:44:59.010 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:58 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:59.010 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:44:58 smithi137 ceph-mon[39175]: pgmap v737: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.33k op/s; 55064/363699 objects degraded (15.140%); 40 MiB/s, 632 objects/s recovering 2024-09-18T07:44:59.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:59.109 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:59.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:59.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:44:59.250 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:59.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:59.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:44:59.283 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:44:59.328 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:44:59.328 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:44:59.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:154661 10.3 e401/364 10:cc0680c8:::benchmark_data_smithi137_151160_object18845:head v 401'9759, mlcod=401'9757)", 2024-09-18T07:44:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:59.324572+0000", 2024-09-18T07:44:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00056824000000000002, 2024-09-18T07:44:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000595362, 2024-09-18T07:44:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:44:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:44:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:44:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:44:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:44:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:59.324572+0000", 2024-09-18T07:44:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:59.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:44:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:59.324572+0000", 2024-09-18T07:44:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:44:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:44:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:59.324573+0000", 2024-09-18T07:44:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.429e-06 2024-09-18T07:44:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:44:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:59.324580+0000", 2024-09-18T07:44:59.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9689999999999997e-06 2024-09-18T07:44:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:44:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:59.324593+0000", 2024-09-18T07:44:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2135e-05 2024-09-18T07:44:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:44:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:59.324600+0000", 2024-09-18T07:44:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1860000000000003e-06 2024-09-18T07:44:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:44:59.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:59.324700+0000", 2024-09-18T07:44:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100164 2024-09-18T07:44:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:44:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:44:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:44:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:59.324719+0000", 2024-09-18T07:44:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.925e-05 2024-09-18T07:44:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:44:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:44:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:44:59.332 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 1 2024-09-18T07:44:59.332 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:44:59.352 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:44:59.362 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:58 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:44:59.362 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:44:58 smithi203 ceph-mon[32237]: pgmap v737: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.33k op/s; 55064/363699 objects degraded (15.140%); 40 MiB/s, 632 objects/s recovering 2024-09-18T07:44:59.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:44:59.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:59.397 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:44:59.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:44:59.498 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:44:59.542 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:59.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:59.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:44:59.644 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:44:59.679 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:59.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:59.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:44:59.781 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:44:59.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:59.822 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:59.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:44:59.924 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:44:59.959 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:44:59.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:44:59.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:00.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:00.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:00.104 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:00.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:00.205 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:00.245 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:00.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:00.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:00.347 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:00.383 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:00.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:00.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:00.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:00.522 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:00.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:00.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:00.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T07:45:00.619 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:00.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:00.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:00.690 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:00.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:00.791 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:00.826 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:00.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:00.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:00.928 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:00.960 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:00.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:00.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:01.009 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:45:01.083 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:01.125 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:01.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:01.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:01.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:00 smithi203 ceph-mon[32237]: pgmap v738: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.42k op/s; 53361/359244 objects degraded (14.854%); 34 MiB/s, 551 objects/s recovering 2024-09-18T07:45:01.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:00 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 53361/359244 objects degraded (14.854%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:01.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:00 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3650464628' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:45:01.226 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:01.245 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:00 smithi137 ceph-mon[29408]: pgmap v738: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.42k op/s; 53361/359244 objects degraded (14.854%); 34 MiB/s, 551 objects/s recovering 2024-09-18T07:45:01.245 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:00 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 53361/359244 objects degraded (14.854%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:01.245 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:00 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3650464628' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:45:01.246 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:00 smithi137 ceph-mon[39175]: pgmap v738: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.42k op/s; 53361/359244 objects degraded (14.854%); 34 MiB/s, 551 objects/s recovering 2024-09-18T07:45:01.246 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:00 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 53361/359244 objects degraded (14.854%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:01.246 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:00 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3650464628' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:45:01.265 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:01.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:01.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:01.368 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:01.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:01.405 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:01.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:01.441 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:45:01.441 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:45:01.441 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:45:01.441 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:45:01.441 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:45:01.442 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:45:01.442 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:45:01.442 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:45:01.442 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:45:01.442 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:45:01.442 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:45:01.442 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:45:01.449 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:45:01.449 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:45:01.479 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:45:01.479 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:45:01.507 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:01.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:01.544 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:01.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:01.645 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:01.684 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:01.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:01.786 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:01.829 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:01.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:01.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:01.931 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:01.965 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:01.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:01.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:02.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:02.101 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:02.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:02.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:02.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:01 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:02.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:01 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3650464628' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:45:02.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:01 smithi137 ceph-mon[29408]: osdmap e402: 8 total, 8 up, 8 in 2024-09-18T07:45:02.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:01 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/463430507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:02.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:01 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:02.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:01 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3650464628' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:45:02.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:01 smithi137 ceph-mon[39175]: osdmap e402: 8 total, 8 up, 8 in 2024-09-18T07:45:02.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:01 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/463430507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:02.203 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:02.242 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:02.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:02.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:02.344 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:02.387 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:02.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:02.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:02.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:01 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:02.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:01 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3650464628' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:45:02.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:01 smithi203 ceph-mon[32237]: osdmap e402: 8 total, 8 up, 8 in 2024-09-18T07:45:02.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:01 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/463430507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:02.489 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:02.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:02.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:02.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:02.612 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:02.649 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:02.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:02.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:02.751 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:02.786 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:02.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:02.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:02.887 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:02.919 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:02.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:02.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:03.021 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:03.059 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:03.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:03.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:03.161 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:03.196 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:03.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:03.220 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:03.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:03.340 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:03.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:03.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:03.355 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:03 smithi137 ceph-mon[29408]: pgmap v740: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 789 op/s; 52212/358137 objects degraded (14.579%); 29 MiB/s, 461 objects/s recovering 2024-09-18T07:45:03.356 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:03 smithi137 ceph-mon[39175]: pgmap v740: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 789 op/s; 52212/358137 objects degraded (14.579%); 29 MiB/s, 461 objects/s recovering 2024-09-18T07:45:03.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:03.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:03 smithi203 ceph-mon[32237]: pgmap v740: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 789 op/s; 52212/358137 objects degraded (14.579%); 29 MiB/s, 461 objects/s recovering 2024-09-18T07:45:03.480 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:03.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:03.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:03.582 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:03.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:03.618 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:03.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:03.719 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:03.758 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:03.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:03.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:03.782 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-18T07:45:03.782 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:45:03.782 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-18T07:45:03.782 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T07:45:03.782 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T07:45:03.783 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-18T07:45:03.850 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:45:03.850 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:45:03.859 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:03.905 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:03.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:03.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:04.007 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:04.049 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:04.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:04.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:04.151 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:04.197 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:04.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:04.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:04.298 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:04.330 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:45:04.343 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:45:04.358 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:04.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:04.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:04.460 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:04.497 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:04.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:04.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:04.599 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:04.644 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:04.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:04.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:04.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T07:45:04.746 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:04.789 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:04.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:04.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:04.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:04.941 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:04.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:04.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:05.043 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:05.091 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:05.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:05.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:05.156 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:45:05.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:05.221 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:05.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:05.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:05.251 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:05 smithi137 ceph-mon[39175]: pgmap v741: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.36k op/s; 48419/341601 objects degraded (14.174%); 37 MiB/s, 594 objects/s recovering 2024-09-18T07:45:05.252 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:05 smithi137 ceph-mon[29408]: pgmap v741: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.36k op/s; 48419/341601 objects degraded (14.174%); 37 MiB/s, 594 objects/s recovering 2024-09-18T07:45:05.322 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:05.356 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:05.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:05.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:05.383 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:45:05.384 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:45:05.419 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:45:05.420 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:45:05.428 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:05.455 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:45:05.455 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:45:05.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:05.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:05 smithi203 ceph-mon[32237]: pgmap v741: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.36k op/s; 48419/341601 objects degraded (14.174%); 37 MiB/s, 594 objects/s recovering 2024-09-18T07:45:05.495 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:45:05.495 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:45:05.517 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:05.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:05.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:05.528 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:45:05.528 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:45:05.562 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:45:05.562 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:45:05.565 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:05.601 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:45:05.602 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:45:05.619 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:05.634 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:45:05.634 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:45:05.650 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:05.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:05.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:05.752 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:05.783 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:05.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:05.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:05.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:05.921 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:05.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:05.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:06.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:06.046 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:06.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:06.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:06.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:06.189 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:06.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:06.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:06.277 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:06 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 48419/341601 objects degraded (14.174%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:06.277 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:06 smithi137 ceph-mon[29408]: osdmap e403: 8 total, 8 up, 8 in 2024-09-18T07:45:06.277 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:06 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4206039298' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:06.278 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:06 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 48419/341601 objects degraded (14.174%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:06.278 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:06 smithi137 ceph-mon[39175]: osdmap e403: 8 total, 8 up, 8 in 2024-09-18T07:45:06.278 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:06 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4206039298' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:06.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:06.333 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:06.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:06.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:06.435 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:06.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:06 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 48419/341601 objects degraded (14.174%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:06.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:06 smithi203 ceph-mon[32237]: osdmap e403: 8 total, 8 up, 8 in 2024-09-18T07:45:06.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:06 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4206039298' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:06.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:06.478 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:06.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:06.579 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:06.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:06.614 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:06.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:06.715 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:06.751 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:06.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:06.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:06.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:06.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:06.889 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:06.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:06.990 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:07.029 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:07.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:07.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:07.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:07.168 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:07.168 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:07 smithi203 ceph-mon[32237]: pgmap v743: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 920 op/s; 48419/341601 objects degraded (14.174%); 24 MiB/s, 379 objects/s recovering 2024-09-18T07:45:07.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:07.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:07.221 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:45:07.221 INFO:teuthology.orchestra.run.smithi203.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-18T07:45:07.221 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:45:07.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:07.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:07.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:07.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:07.394 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:07 smithi137 ceph-mon[29408]: pgmap v743: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 920 op/s; 48419/341601 objects degraded (14.174%); 24 MiB/s, 379 objects/s recovering 2024-09-18T07:45:07.395 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:45:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:45:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:45:07.395 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:07 smithi137 ceph-mon[39175]: pgmap v743: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 920 op/s; 48419/341601 objects degraded (14.174%); 24 MiB/s, 379 objects/s recovering 2024-09-18T07:45:07.422 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:07.467 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:07.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:07.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:07.569 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:07.625 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:07.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:07.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:07.726 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:07.766 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:07.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:07.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:07.806 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:45:07.806 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:45:07.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:45:07.806 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:45:07.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.806 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109537 8.7 8:e50487d6:::benchmark_data_smithi137_101942_object4727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:45:07.807 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:45:07.807 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 445.745037056, 2024-09-18T07:45:07.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.185266763, 2024-09-18T07:45:07.807 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.807 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.807 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.807 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:07.807 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:45:07.807 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:45:07.807 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109537 2024-09-18T07:45:07.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.807 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:45:07.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:45:07.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056408+0000", 2024-09-18T07:45:07.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.562e-06 2024-09-18T07:45:07.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056415+0000", 2024-09-18T07:45:07.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9280000000000004e-06 2024-09-18T07:45:07.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056425+0000", 2024-09-18T07:45:07.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0220999999999999e-05 2024-09-18T07:45:07.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056434+0000", 2024-09-18T07:45:07.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7499999999999992e-06 2024-09-18T07:45:07.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.061190+0000", 2024-09-18T07:45:07.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047560620000000001 2024-09-18T07:45:07.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:07.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.061238+0000", 2024-09-18T07:45:07.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8730000000000003e-05 2024-09-18T07:45:07.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229082+0000", 2024-09-18T07:45:07.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.167843878 2024-09-18T07:45:07.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229243+0000", 2024-09-18T07:45:07.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000160426 2024-09-18T07:45:07.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:45:07.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229373+0000", 2024-09-18T07:45:07.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012994999999999999 2024-09-18T07:45:07.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:07.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236041+0000", 2024-09-18T07:45:07.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066680960000000001 2024-09-18T07:45:07.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:07.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236051+0000", 2024-09-18T07:45:07.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8309999999999996e-06 2024-09-18T07:45:07.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:07.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.240107+0000", 2024-09-18T07:45:07.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040564570000000003 2024-09-18T07:45:07.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.240152+0000", 2024-09-18T07:45:07.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4617000000000003e-05 2024-09-18T07:45:07.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.241673+0000", 2024-09-18T07:45:07.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001521255 2024-09-18T07:45:07.814 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.814 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.814 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.814 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109540 8.7 8:e7454470:::benchmark_data_smithi137_101942_object4730:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:45:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:45:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 445.73556506599999, 2024-09-18T07:45:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.186834677, 2024-09-18T07:45:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:45:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:45:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109540 2024-09-18T07:45:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:45:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:45:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065880+0000", 2024-09-18T07:45:07.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8279999999999999e-06 2024-09-18T07:45:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065886+0000", 2024-09-18T07:45:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.117e-06 2024-09-18T07:45:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065899+0000", 2024-09-18T07:45:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2809999999999999e-05 2024-09-18T07:45:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065904+0000", 2024-09-18T07:45:07.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0350000000000001e-06 2024-09-18T07:45:07.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.066028+0000", 2024-09-18T07:45:07.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012424199999999999 2024-09-18T07:45:07.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:07.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.066039+0000", 2024-09-18T07:45:07.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0441999999999999e-05 2024-09-18T07:45:07.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.233503+0000", 2024-09-18T07:45:07.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.167464622 2024-09-18T07:45:07.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.233624+0000", 2024-09-18T07:45:07.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012107 2024-09-18T07:45:07.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:45:07.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.233749+0000", 2024-09-18T07:45:07.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000124418 2024-09-18T07:45:07.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:07.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236078+0000", 2024-09-18T07:45:07.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002328746 2024-09-18T07:45:07.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:07.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.239968+0000", 2024-09-18T07:45:07.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038901220000000002 2024-09-18T07:45:07.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:07.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.250395+0000", 2024-09-18T07:45:07.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010427532999999999 2024-09-18T07:45:07.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.250537+0000", 2024-09-18T07:45:07.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014218100000000001 2024-09-18T07:45:07.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.252713+0000", 2024-09-18T07:45:07.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0021755110000000002 2024-09-18T07:45:07.822 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.822 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.822 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.822 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109545 8.7 8:ecb41cf4:::benchmark_data_smithi137_101942_object4735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:45:07.822 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:45:07.822 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 445.72716028000002, 2024-09-18T07:45:07.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1804624180000001, 2024-09-18T07:45:07.823 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.823 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.823 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.823 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:07.823 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:45:07.823 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:45:07.823 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109545 2024-09-18T07:45:07.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.823 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:45:07.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:45:07.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074285+0000", 2024-09-18T07:45:07.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1169999999999998e-06 2024-09-18T07:45:07.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074292+0000", 2024-09-18T07:45:07.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.153e-06 2024-09-18T07:45:07.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074305+0000", 2024-09-18T07:45:07.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2363e-05 2024-09-18T07:45:07.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074312+0000", 2024-09-18T07:45:07.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3409999999999997e-06 2024-09-18T07:45:07.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074508+0000", 2024-09-18T07:45:07.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000196443 2024-09-18T07:45:07.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:07.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074519+0000", 2024-09-18T07:45:07.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0159999999999999e-05 2024-09-18T07:45:07.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236329+0000", 2024-09-18T07:45:07.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1618104599999999 2024-09-18T07:45:07.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.827 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236416+0000", 2024-09-18T07:45:07.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6670999999999997e-05 2024-09-18T07:45:07.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.827 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:45:07.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236495+0000", 2024-09-18T07:45:07.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9256000000000001e-05 2024-09-18T07:45:07.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.827 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:07.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.241684+0000", 2024-09-18T07:45:07.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051890720000000003 2024-09-18T07:45:07.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:07.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.243812+0000", 2024-09-18T07:45:07.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002127912 2024-09-18T07:45:07.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:07.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.252772+0000", 2024-09-18T07:45:07.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.008960562 2024-09-18T07:45:07.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.252827+0000", 2024-09-18T07:45:07.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4500000000000003e-05 2024-09-18T07:45:07.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.254745+0000", 2024-09-18T07:45:07.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019184079999999999 2024-09-18T07:45:07.829 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.829 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.829 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.829 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109567 8.7 8:e247f5cd:::benchmark_data_smithi137_101942_object4757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:45:07.829 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:45:07.829 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 445.69806111700001, 2024-09-18T07:45:07.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.153246698, 2024-09-18T07:45:07.830 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.830 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.830 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.830 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:07.830 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:45:07.830 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:45:07.830 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109567 2024-09-18T07:45:07.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.830 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:45:07.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:45:07.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103384+0000", 2024-09-18T07:45:07.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.792e-06 2024-09-18T07:45:07.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103390+0000", 2024-09-18T07:45:07.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7139999999999998e-06 2024-09-18T07:45:07.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103401+0000", 2024-09-18T07:45:07.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1559000000000001e-05 2024-09-18T07:45:07.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103408+0000", 2024-09-18T07:45:07.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2110000000000001e-06 2024-09-18T07:45:07.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103464+0000", 2024-09-18T07:45:07.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6032999999999997e-05 2024-09-18T07:45:07.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:07.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103475+0000", 2024-09-18T07:45:07.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0380999999999999e-05 2024-09-18T07:45:07.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.238189+0000", 2024-09-18T07:45:07.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.134713723 2024-09-18T07:45:07.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.238319+0000", 2024-09-18T07:45:07.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013080700000000001 2024-09-18T07:45:07.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:45:07.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.238431+0000", 2024-09-18T07:45:07.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112126 2024-09-18T07:45:07.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:07.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.243830+0000", 2024-09-18T07:45:07.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053980390000000003 2024-09-18T07:45:07.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:07.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.243889+0000", 2024-09-18T07:45:07.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9024000000000003e-05 2024-09-18T07:45:07.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:07.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.254832+0000", 2024-09-18T07:45:07.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0109436 2024-09-18T07:45:07.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.255025+0000", 2024-09-18T07:45:07.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000192926 2024-09-18T07:45:07.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.256629+0000", 2024-09-18T07:45:07.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001603763 2024-09-18T07:45:07.836 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.836 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.836 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.836 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:126833 10.f e391/364 10:f8e7c411:::benchmark_data_smithi137_151160_object126832:head v 391'7824, mlcod=391'7821)", 2024-09-18T07:45:07.836 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:32.541463+0000", 2024-09-18T07:45:07.836 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 35.259979737999998, 2024-09-18T07:45:07.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.042615444000000002, 2024-09-18T07:45:07.837 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.837 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.837 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.837 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.541463+0000", 2024-09-18T07:45:07.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.541463+0000", 2024-09-18T07:45:07.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.541465+0000", 2024-09-18T07:45:07.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.674e-06 2024-09-18T07:45:07.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.541509+0000", 2024-09-18T07:45:07.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4079000000000003e-05 2024-09-18T07:45:07.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.541526+0000", 2024-09-18T07:45:07.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.683e-05 2024-09-18T07:45:07.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.541535+0000", 2024-09-18T07:45:07.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.054e-06 2024-09-18T07:45:07.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.553292+0000", 2024-09-18T07:45:07.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011757327 2024-09-18T07:45:07.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.553313+0000", 2024-09-18T07:45:07.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0783000000000001e-05 2024-09-18T07:45:07.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.583835+0000", 2024-09-18T07:45:07.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.03052148 2024-09-18T07:45:07.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.584079+0000", 2024-09-18T07:45:07.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024421700000000002 2024-09-18T07:45:07.841 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.841 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.841 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.841 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:126912 10.3 e391/364 10:cc218fc4:::benchmark_data_smithi137_151160_object126911:head v 391'8040, mlcod=391'8038)", 2024-09-18T07:45:07.841 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:32.618429+0000", 2024-09-18T07:45:07.841 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 35.183014188000001, 2024-09-18T07:45:07.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.041096705999999997, 2024-09-18T07:45:07.841 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.842 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.842 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.842 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.618429+0000", 2024-09-18T07:45:07.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.618429+0000", 2024-09-18T07:45:07.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.618431+0000", 2024-09-18T07:45:07.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.984e-06 2024-09-18T07:45:07.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.618460+0000", 2024-09-18T07:45:07.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9215000000000001e-05 2024-09-18T07:45:07.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.618475+0000", 2024-09-18T07:45:07.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4415e-05 2024-09-18T07:45:07.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.618484+0000", 2024-09-18T07:45:07.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5389999999999996e-06 2024-09-18T07:45:07.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.623350+0000", 2024-09-18T07:45:07.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048657489999999999 2024-09-18T07:45:07.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.623370+0000", 2024-09-18T07:45:07.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0301000000000001e-05 2024-09-18T07:45:07.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.659487+0000", 2024-09-18T07:45:07.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.036116802000000003 2024-09-18T07:45:07.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.659526+0000", 2024-09-18T07:45:07.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8701000000000001e-05 2024-09-18T07:45:07.846 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.846 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.846 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.846 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:161669 10.f e403/364 10:ff7b857f:::benchmark_data_smithi137_151160_object25853:head v 403'9963, mlcod=403'9960)", 2024-09-18T07:45:07.846 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:05.543990+0000", 2024-09-18T07:45:07.846 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.2574528640000002, 2024-09-18T07:45:07.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016880084, 2024-09-18T07:45:07.846 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.846 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.846 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.846 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.543990+0000", 2024-09-18T07:45:07.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.543990+0000", 2024-09-18T07:45:07.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.543992+0000", 2024-09-18T07:45:07.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0849999999999999e-06 2024-09-18T07:45:07.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.544001+0000", 2024-09-18T07:45:07.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.704e-06 2024-09-18T07:45:07.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.544017+0000", 2024-09-18T07:45:07.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5571000000000001e-05 2024-09-18T07:45:07.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.544024+0000", 2024-09-18T07:45:07.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5469999999999999e-06 2024-09-18T07:45:07.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.544287+0000", 2024-09-18T07:45:07.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00026316600000000001 2024-09-18T07:45:07.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.544311+0000", 2024-09-18T07:45:07.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3473e-05 2024-09-18T07:45:07.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.560783+0000", 2024-09-18T07:45:07.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016472009999999999 2024-09-18T07:45:07.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.560870+0000", 2024-09-18T07:45:07.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7528000000000006e-05 2024-09-18T07:45:07.850 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.850 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.850 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.851 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:161896 10.7 e403/364 10:eda7f0e1:::benchmark_data_smithi137_151160_object26080:head v 403'10225, mlcod=403'10222)", 2024-09-18T07:45:07.851 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:05.755480+0000", 2024-09-18T07:45:07.851 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.045963371, 2024-09-18T07:45:07.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016373232000000001, 2024-09-18T07:45:07.852 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.852 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.852 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.852 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.755480+0000", 2024-09-18T07:45:07.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.755480+0000", 2024-09-18T07:45:07.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.755481+0000", 2024-09-18T07:45:07.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.624e-06 2024-09-18T07:45:07.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.755490+0000", 2024-09-18T07:45:07.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6889999999999993e-06 2024-09-18T07:45:07.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.755509+0000", 2024-09-18T07:45:07.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8451e-05 2024-09-18T07:45:07.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.755529+0000", 2024-09-18T07:45:07.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0418999999999999e-05 2024-09-18T07:45:07.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.755705+0000", 2024-09-18T07:45:07.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000175988 2024-09-18T07:45:07.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.755750+0000", 2024-09-18T07:45:07.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4524999999999998e-05 2024-09-18T07:45:07.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.771802+0000", 2024-09-18T07:45:07.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016052377 2024-09-18T07:45:07.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.771853+0000", 2024-09-18T07:45:07.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1159000000000003e-05 2024-09-18T07:45:07.856 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.856 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.856 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.856 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:162064 10.f e403/364 10:f62aa829:::benchmark_data_smithi137_151160_object26248:head v 403'9989, mlcod=403'9986)", 2024-09-18T07:45:07.856 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:05.891622+0000", 2024-09-18T07:45:07.856 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.9098212160000001, 2024-09-18T07:45:07.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018335800999999999, 2024-09-18T07:45:07.856 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.856 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.857 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.857 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.891622+0000", 2024-09-18T07:45:07.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.891622+0000", 2024-09-18T07:45:07.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.891623+0000", 2024-09-18T07:45:07.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2249999999999999e-06 2024-09-18T07:45:07.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.891630+0000", 2024-09-18T07:45:07.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.674e-06 2024-09-18T07:45:07.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.891640+0000", 2024-09-18T07:45:07.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0161e-05 2024-09-18T07:45:07.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.891647+0000", 2024-09-18T07:45:07.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5089999999999999e-06 2024-09-18T07:45:07.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.891864+0000", 2024-09-18T07:45:07.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000217295 2024-09-18T07:45:07.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.891937+0000", 2024-09-18T07:45:07.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2881999999999998e-05 2024-09-18T07:45:07.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.909815+0000", 2024-09-18T07:45:07.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017877836000000001 2024-09-18T07:45:07.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.909958+0000", 2024-09-18T07:45:07.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014321900000000001 2024-09-18T07:45:07.860 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.861 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.861 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.861 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:162067 10.7 e403/364 10:e0359e32:::benchmark_data_smithi137_151160_object26251:head v 403'10240, mlcod=403'10237)", 2024-09-18T07:45:07.861 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:05.891663+0000", 2024-09-18T07:45:07.861 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.9097800700000001, 2024-09-18T07:45:07.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015112295, 2024-09-18T07:45:07.861 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.861 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.861 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.861 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.891663+0000", 2024-09-18T07:45:07.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.891663+0000", 2024-09-18T07:45:07.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.891664+0000", 2024-09-18T07:45:07.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5400000000000002e-07 2024-09-18T07:45:07.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.891667+0000", 2024-09-18T07:45:07.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6030000000000001e-06 2024-09-18T07:45:07.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.891672+0000", 2024-09-18T07:45:07.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4920000000000004e-06 2024-09-18T07:45:07.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.891676+0000", 2024-09-18T07:45:07.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9709999999999998e-06 2024-09-18T07:45:07.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.891870+0000", 2024-09-18T07:45:07.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019463400000000001 2024-09-18T07:45:07.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.891905+0000", 2024-09-18T07:45:07.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5101000000000002e-05 2024-09-18T07:45:07.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.906700+0000", 2024-09-18T07:45:07.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014794610999999999 2024-09-18T07:45:07.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.906775+0000", 2024-09-18T07:45:07.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5529000000000001e-05 2024-09-18T07:45:07.865 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.865 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.865 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.865 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:162068 10.f e403/364 10:f5ca08e8:::benchmark_data_smithi137_151160_object26252:head v 403'9990, mlcod=403'9986)", 2024-09-18T07:45:07.866 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:05.894383+0000", 2024-09-18T07:45:07.866 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.907060489, 2024-09-18T07:45:07.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015728216, 2024-09-18T07:45:07.866 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.866 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.866 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.866 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.894383+0000", 2024-09-18T07:45:07.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.894383+0000", 2024-09-18T07:45:07.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.894383+0000", 2024-09-18T07:45:07.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2799999999999996e-07 2024-09-18T07:45:07.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.894388+0000", 2024-09-18T07:45:07.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6369999999999998e-06 2024-09-18T07:45:07.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.894393+0000", 2024-09-18T07:45:07.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2109999999999997e-06 2024-09-18T07:45:07.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.894397+0000", 2024-09-18T07:45:07.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.177e-06 2024-09-18T07:45:07.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.895166+0000", 2024-09-18T07:45:07.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00076898199999999998 2024-09-18T07:45:07.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.895184+0000", 2024-09-18T07:45:07.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7357e-05 2024-09-18T07:45:07.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.909964+0000", 2024-09-18T07:45:07.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014779808 2024-09-18T07:45:07.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:05.910111+0000", 2024-09-18T07:45:07.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014741600000000001 2024-09-18T07:45:07.870 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.870 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.870 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.870 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:162411 10.f e403/364 10:f255b541:::benchmark_data_smithi137_151160_object26595:head v 403'10013, mlcod=403'10011)", 2024-09-18T07:45:07.871 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:06.213656+0000", 2024-09-18T07:45:07.871 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.5877875449999999, 2024-09-18T07:45:07.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018469152999999999, 2024-09-18T07:45:07.871 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.871 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.871 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.871 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.213656+0000", 2024-09-18T07:45:07.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.213656+0000", 2024-09-18T07:45:07.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.213657+0000", 2024-09-18T07:45:07.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.533e-06 2024-09-18T07:45:07.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.213664+0000", 2024-09-18T07:45:07.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6359999999999997e-06 2024-09-18T07:45:07.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.213679+0000", 2024-09-18T07:45:07.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5225000000000001e-05 2024-09-18T07:45:07.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.213686+0000", 2024-09-18T07:45:07.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1980000000000004e-06 2024-09-18T07:45:07.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.213857+0000", 2024-09-18T07:45:07.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000170874 2024-09-18T07:45:07.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.213874+0000", 2024-09-18T07:45:07.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7110999999999999e-05 2024-09-18T07:45:07.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.231941+0000", 2024-09-18T07:45:07.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018066858000000002 2024-09-18T07:45:07.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.232125+0000", 2024-09-18T07:45:07.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018371800000000001 2024-09-18T07:45:07.875 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.875 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.875 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.875 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:162865 10.f e403/364 10:f0f9f056:::benchmark_data_smithi137_151160_object27049:head v 403'10039, mlcod=403'10037)", 2024-09-18T07:45:07.875 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:06.621298+0000", 2024-09-18T07:45:07.875 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.1801452960000001, 2024-09-18T07:45:07.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015067588999999999, 2024-09-18T07:45:07.875 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.876 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.876 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.876 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.621298+0000", 2024-09-18T07:45:07.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.621298+0000", 2024-09-18T07:45:07.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.621300+0000", 2024-09-18T07:45:07.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.807e-06 2024-09-18T07:45:07.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.621308+0000", 2024-09-18T07:45:07.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3979999999999999e-06 2024-09-18T07:45:07.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.621324+0000", 2024-09-18T07:45:07.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6294999999999999e-05 2024-09-18T07:45:07.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.621333+0000", 2024-09-18T07:45:07.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4509999999999992e-06 2024-09-18T07:45:07.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.621498+0000", 2024-09-18T07:45:07.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016505799999999999 2024-09-18T07:45:07.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.621541+0000", 2024-09-18T07:45:07.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2932999999999997e-05 2024-09-18T07:45:07.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.636211+0000", 2024-09-18T07:45:07.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014669718999999999 2024-09-18T07:45:07.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.636366+0000", 2024-09-18T07:45:07.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000154928 2024-09-18T07:45:07.880 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.880 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.880 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.880 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:163054 10.3 e403/364 10:c0cb0976:::benchmark_data_smithi137_151160_object27238:head v 403'10297, mlcod=403'10294)", 2024-09-18T07:45:07.880 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:06.769529+0000", 2024-09-18T07:45:07.880 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.0319147339999999, 2024-09-18T07:45:07.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017193942, 2024-09-18T07:45:07.881 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.881 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.881 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.881 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.769529+0000", 2024-09-18T07:45:07.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.769529+0000", 2024-09-18T07:45:07.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.769530+0000", 2024-09-18T07:45:07.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2139999999999999e-06 2024-09-18T07:45:07.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.769537+0000", 2024-09-18T07:45:07.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9269999999999999e-06 2024-09-18T07:45:07.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.769550+0000", 2024-09-18T07:45:07.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3371e-05 2024-09-18T07:45:07.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.769557+0000", 2024-09-18T07:45:07.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0500000000000003e-06 2024-09-18T07:45:07.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.769636+0000", 2024-09-18T07:45:07.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9320000000000006e-05 2024-09-18T07:45:07.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.769658+0000", 2024-09-18T07:45:07.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1526999999999999e-05 2024-09-18T07:45:07.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.786678+0000", 2024-09-18T07:45:07.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017020108999999999 2024-09-18T07:45:07.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:06.786722+0000", 2024-09-18T07:45:07.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4424e-05 2024-09-18T07:45:07.885 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.885 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.885 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.885 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:163710 10.5 e403/364 10:aab39335:::benchmark_data_smithi137_151160_object27894:head v 403'10215, mlcod=403'10211)", 2024-09-18T07:45:07.885 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:07.321875+0000", 2024-09-18T07:45:07.885 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.479568416, 2024-09-18T07:45:07.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015654748999999999, 2024-09-18T07:45:07.885 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.885 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.885 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.886 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.321875+0000", 2024-09-18T07:45:07.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.321875+0000", 2024-09-18T07:45:07.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.321876+0000", 2024-09-18T07:45:07.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.158e-06 2024-09-18T07:45:07.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.321883+0000", 2024-09-18T07:45:07.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5710000000000003e-06 2024-09-18T07:45:07.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.321895+0000", 2024-09-18T07:45:07.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2551000000000001e-05 2024-09-18T07:45:07.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.321902+0000", 2024-09-18T07:45:07.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0809999999999996e-06 2024-09-18T07:45:07.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.325777+0000", 2024-09-18T07:45:07.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038748469999999998 2024-09-18T07:45:07.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.325826+0000", 2024-09-18T07:45:07.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9335000000000001e-05 2024-09-18T07:45:07.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.337420+0000", 2024-09-18T07:45:07.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011593343000000001 2024-09-18T07:45:07.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.337530+0000", 2024-09-18T07:45:07.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000109863 2024-09-18T07:45:07.890 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.890 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.890 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.890 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:163916 10.5 e403/364 10:a8081999:::benchmark_data_smithi137_151160_object28100:head v 403'10231, mlcod=403'10227)", 2024-09-18T07:45:07.890 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:07.496501+0000", 2024-09-18T07:45:07.890 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.30494189500000002, 2024-09-18T07:45:07.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019106304000000001, 2024-09-18T07:45:07.890 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.890 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.890 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.891 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.496501+0000", 2024-09-18T07:45:07.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.496501+0000", 2024-09-18T07:45:07.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.496502+0000", 2024-09-18T07:45:07.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4399999999999999e-07 2024-09-18T07:45:07.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.496508+0000", 2024-09-18T07:45:07.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4870000000000002e-06 2024-09-18T07:45:07.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.496516+0000", 2024-09-18T07:45:07.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9799999999999998e-06 2024-09-18T07:45:07.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.496522+0000", 2024-09-18T07:45:07.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0769999999999996e-06 2024-09-18T07:45:07.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.501373+0000", 2024-09-18T07:45:07.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048509750000000004 2024-09-18T07:45:07.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.501420+0000", 2024-09-18T07:45:07.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7703000000000001e-05 2024-09-18T07:45:07.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.515329+0000", 2024-09-18T07:45:07.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013908716 2024-09-18T07:45:07.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.515608+0000", 2024-09-18T07:45:07.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027862200000000002 2024-09-18T07:45:07.894 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.894 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.894 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.895 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:164039 10.3 e403/364 10:cc41a85f:::benchmark_data_smithi137_151160_object28223:head v 403'10369, mlcod=403'10366)", 2024-09-18T07:45:07.895 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:07.615865+0000", 2024-09-18T07:45:07.895 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.18557834400000001, 2024-09-18T07:45:07.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020915125, 2024-09-18T07:45:07.895 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.895 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.895 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.895 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.615865+0000", 2024-09-18T07:45:07.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.615865+0000", 2024-09-18T07:45:07.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.615866+0000", 2024-09-18T07:45:07.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.432e-06 2024-09-18T07:45:07.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:07.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.615874+0000", 2024-09-18T07:45:07.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0090000000000007e-06 2024-09-18T07:45:07.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.615890+0000", 2024-09-18T07:45:07.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5460000000000001e-05 2024-09-18T07:45:07.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.615897+0000", 2024-09-18T07:45:07.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5730000000000002e-06 2024-09-18T07:45:07.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.615986+0000", 2024-09-18T07:45:07.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8227999999999996e-05 2024-09-18T07:45:07.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.616010+0000", 2024-09-18T07:45:07.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4603000000000001e-05 2024-09-18T07:45:07.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.636643+0000", 2024-09-18T07:45:07.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020632832 2024-09-18T07:45:07.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.636780+0000", 2024-09-18T07:45:07.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000136988 2024-09-18T07:45:07.900 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.900 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.901 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.901 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:164049 10.f e403/364 10:f1a602fb:::benchmark_data_smithi137_151160_object28233:head v 403'10114, mlcod=403'10112)", 2024-09-18T07:45:07.901 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:07.624800+0000", 2024-09-18T07:45:07.901 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.17664322800000001, 2024-09-18T07:45:07.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019159677999999999, 2024-09-18T07:45:07.901 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.901 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.901 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.901 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.624800+0000", 2024-09-18T07:45:07.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.624800+0000", 2024-09-18T07:45:07.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.624801+0000", 2024-09-18T07:45:07.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3149999999999999e-06 2024-09-18T07:45:07.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.624809+0000", 2024-09-18T07:45:07.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4619999999999999e-06 2024-09-18T07:45:07.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.624823+0000", 2024-09-18T07:45:07.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3866000000000001e-05 2024-09-18T07:45:07.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.624830+0000", 2024-09-18T07:45:07.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4619999999999999e-06 2024-09-18T07:45:07.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.624900+0000", 2024-09-18T07:45:07.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9611999999999997e-05 2024-09-18T07:45:07.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.625157+0000", 2024-09-18T07:45:07.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000256968 2024-09-18T07:45:07.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.643795+0000", 2024-09-18T07:45:07.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018638411000000001 2024-09-18T07:45:07.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.643960+0000", 2024-09-18T07:45:07.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000164582 2024-09-18T07:45:07.905 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.905 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.905 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.905 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:164171 10.7 e403/364 10:e4ce444a:::benchmark_data_smithi137_151160_object28355:head v 403'10374, mlcod=403'10371)", 2024-09-18T07:45:07.906 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:07.765579+0000", 2024-09-18T07:45:07.906 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.035864385999999998, 2024-09-18T07:45:07.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016305775000000002, 2024-09-18T07:45:07.906 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.906 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.906 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.906 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.765579+0000", 2024-09-18T07:45:07.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.765579+0000", 2024-09-18T07:45:07.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.765579+0000", 2024-09-18T07:45:07.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6499999999999999e-07 2024-09-18T07:45:07.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.765584+0000", 2024-09-18T07:45:07.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4440000000000001e-06 2024-09-18T07:45:07.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.765604+0000", 2024-09-18T07:45:07.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0635000000000001e-05 2024-09-18T07:45:07.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.765610+0000", 2024-09-18T07:45:07.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.254e-06 2024-09-18T07:45:07.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.766159+0000", 2024-09-18T07:45:07.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000549564 2024-09-18T07:45:07.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.766239+0000", 2024-09-18T07:45:07.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9481999999999996e-05 2024-09-18T07:45:07.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.777921+0000", 2024-09-18T07:45:07.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011681993 2024-09-18T07:45:07.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.781885+0000", 2024-09-18T07:45:07.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039638379999999999 2024-09-18T07:45:07.910 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.910 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.912 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.912 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:164172 10.f e403/364 10:f7d4db0b:::benchmark_data_smithi137_151160_object28356:head v 403'10124, mlcod=403'10121)", 2024-09-18T07:45:07.912 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:07.771507+0000", 2024-09-18T07:45:07.912 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.029935897999999999, 2024-09-18T07:45:07.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.02795454, 2024-09-18T07:45:07.913 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:07.913 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:07.913 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:07.913 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:07.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:07.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.771507+0000", 2024-09-18T07:45:07.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:07.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.771507+0000", 2024-09-18T07:45:07.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:07.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:07.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.771509+0000", 2024-09-18T07:45:07.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.64e-06 2024-09-18T07:45:07.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:07.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.771517+0000", 2024-09-18T07:45:07.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8399999999999995e-06 2024-09-18T07:45:07.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:07.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.771530+0000", 2024-09-18T07:45:07.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3448999999999999e-05 2024-09-18T07:45:07.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:07.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.771538+0000", 2024-09-18T07:45:07.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9729999999999997e-06 2024-09-18T07:45:07.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:07.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.771688+0000", 2024-09-18T07:45:07.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000149326 2024-09-18T07:45:07.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:07.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.771741+0000", 2024-09-18T07:45:07.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3455999999999998e-05 2024-09-18T07:45:07.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:07.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.799208+0000", 2024-09-18T07:45:07.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027467183999999999 2024-09-18T07:45:07.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:07.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:07.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:07.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.799462+0000", 2024-09-18T07:45:07.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00025367199999999998 2024-09-18T07:45:07.917 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.917 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.917 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.917 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:07.917 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:07.917 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:45:07.961 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:45:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:45:07.967 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:07.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:08.069 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:08.104 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:08 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:45:08.104 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:08 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1439913631' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:45:08.117 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:08.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:08.119 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:45:08.219 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:08.266 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:08.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:08.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:08.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:08.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:08.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:08.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:08 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:45:08.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:08 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1439913631' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:45:08.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:08 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:45:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:08 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1439913631' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:45:08.512 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:08.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:08.566 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:08.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:08.667 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:08.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:08.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:08.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:08.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:08.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:08.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:08.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:08.933 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:08.968 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:08.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:08.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:09.070 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:09.123 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:09.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:09.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:09.225 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:09.271 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:09.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:09.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:09.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:45:09.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:09.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:09 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:09.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:09 smithi137 ceph-mon[29408]: pgmap v744: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 45253/328932 objects degraded (13.758%); 33 MiB/s, 522 objects/s recovering 2024-09-18T07:45:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:09 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:09 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1439913631' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:45:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:09 smithi137 ceph-mon[29408]: osdmap e404: 8 total, 8 up, 8 in 2024-09-18T07:45:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:09 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:09 smithi137 ceph-mon[39175]: pgmap v744: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 45253/328932 objects degraded (13.758%); 33 MiB/s, 522 objects/s recovering 2024-09-18T07:45:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:09 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:09 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1439913631' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:45:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:09 smithi137 ceph-mon[39175]: osdmap e404: 8 total, 8 up, 8 in 2024-09-18T07:45:09.432 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:09.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:09.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:09 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:09 smithi203 ceph-mon[32237]: pgmap v744: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 45253/328932 objects degraded (13.758%); 33 MiB/s, 522 objects/s recovering 2024-09-18T07:45:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:09 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:09 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1439913631' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:45:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:09 smithi203 ceph-mon[32237]: osdmap e404: 8 total, 8 up, 8 in 2024-09-18T07:45:09.534 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:09.575 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:09.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:09.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:09.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:09.736 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:09.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:09.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:09.839 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:09.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:09.884 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:09.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:09.960 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T07:45:09.985 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:10.045 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:10.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:10.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:10.056 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:10.147 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:10.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:10.200 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:10.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:10.278 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:10 smithi137 ceph-mon[39175]: 6.b scrub starts 2024-09-18T07:45:10.278 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:10 smithi137 ceph-mon[39175]: 6.b scrub ok 2024-09-18T07:45:10.278 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:10 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 45253/328932 objects degraded (13.758%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:10.278 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:10 smithi137 ceph-mon[39175]: osdmap e405: 8 total, 8 up, 8 in 2024-09-18T07:45:10.279 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:10 smithi137 ceph-mon[29408]: 6.b scrub starts 2024-09-18T07:45:10.279 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:10 smithi137 ceph-mon[29408]: 6.b scrub ok 2024-09-18T07:45:10.279 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:10 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 45253/328932 objects degraded (13.758%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:10.279 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:10 smithi137 ceph-mon[29408]: osdmap e405: 8 total, 8 up, 8 in 2024-09-18T07:45:10.301 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:10.356 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:10.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:10.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:10.400 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:45:10.458 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:10.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:10 smithi203 ceph-mon[32237]: 6.b scrub starts 2024-09-18T07:45:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:10 smithi203 ceph-mon[32237]: 6.b scrub ok 2024-09-18T07:45:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:10 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 45253/328932 objects degraded (13.758%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:10 smithi203 ceph-mon[32237]: osdmap e405: 8 total, 8 up, 8 in 2024-09-18T07:45:10.495 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:10.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:10.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:10.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:10.629 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:10.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:10.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:10.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:10.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:10.765 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:10.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:10.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:10.889 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:10.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:10.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:11.025 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:11.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:11.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:11.126 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:11.153 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:11.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:11.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:11.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:11.287 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:11.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:11.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:11.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:11.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:11.411 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:11.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:11.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:11 smithi137 ceph-mon[29408]: 3.1d scrub starts 2024-09-18T07:45:11.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:11 smithi137 ceph-mon[29408]: 3.1d scrub ok 2024-09-18T07:45:11.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:11 smithi137 ceph-mon[29408]: pgmap v746: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.39k op/s; 43763/324822 objects degraded (13.473%); 30 MiB/s, 480 objects/s recovering 2024-09-18T07:45:11.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:11 smithi137 ceph-mon[39175]: 3.1d scrub starts 2024-09-18T07:45:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:11 smithi137 ceph-mon[39175]: 3.1d scrub ok 2024-09-18T07:45:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:11 smithi137 ceph-mon[39175]: pgmap v746: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.39k op/s; 43763/324822 objects degraded (13.473%); 30 MiB/s, 480 objects/s recovering 2024-09-18T07:45:11.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:11 smithi203 ceph-mon[32237]: 3.1d scrub starts 2024-09-18T07:45:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:11 smithi203 ceph-mon[32237]: 3.1d scrub ok 2024-09-18T07:45:11.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:11 smithi203 ceph-mon[32237]: pgmap v746: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.39k op/s; 43763/324822 objects degraded (13.473%); 30 MiB/s, 480 objects/s recovering 2024-09-18T07:45:11.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:11.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:11.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:11.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:11.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:11.688 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:11.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:11.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:11.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:45:11.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:11.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:11.833 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:11.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:11.935 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:11.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:11.990 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:11.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:12.091 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:12.120 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:12.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:12.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:12.256 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:12.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:12.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:12.358 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:12.389 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:12.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:12.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:12.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:45:12.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:45:12.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:45:12.491 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:12.527 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:12.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:12.629 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:12.665 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:12.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:12.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:12.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:12.797 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:12.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:12.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:12.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:12.933 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:12.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:12.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:13.025 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-18T07:45:13.026 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:45:13.026 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-18T07:45:13.026 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T07:45:13.026 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T07:45:13.026 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-18T07:45:13.036 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:13.124 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:13.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:13.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:13.226 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:13.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:13.257 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:13.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:13.358 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:13.393 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:13.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:13.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:13.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:13 smithi137 ceph-mon[29408]: pgmap v748: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 890 op/s; 42882/323727 objects degraded (13.246%); 29 MiB/s, 467 objects/s recovering 2024-09-18T07:45:13.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:13 smithi137 ceph-mon[39175]: pgmap v748: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 890 op/s; 42882/323727 objects degraded (13.246%); 29 MiB/s, 467 objects/s recovering 2024-09-18T07:45:13.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:13 smithi203 ceph-mon[32237]: pgmap v748: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 890 op/s; 42882/323727 objects degraded (13.246%); 29 MiB/s, 467 objects/s recovering 2024-09-18T07:45:13.495 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:13.523 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:13.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:13.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:13.625 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:13.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:13.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:13.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:13.759 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:13.793 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:13.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:13.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:13.895 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:13.937 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:13.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:13.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:14.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:14.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:14.069 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:14.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:14.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:14.193 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:14.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:14.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:14.195 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:45:14.195 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:45:14.195 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.195 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:170417 10.9 10:9036391a:::benchmark_data_smithi137_151160_object34601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-18T07:45:14.195 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:13.671171+0000", 2024-09-18T07:45:14.195 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.016240345999999999, 2024-09-18T07:45:14.196 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.016269971000000001, 2024-09-18T07:45:14.196 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:45:14.196 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:45:14.196 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:45:14.196 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:45:14.196 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:45:14.196 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:14.196 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 170417 2024-09-18T07:45:14.196 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.196 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:45:14.196 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.196 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:45:14.196 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.671171+0000", 2024-09-18T07:45:14.197 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.197 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.197 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.197 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:45:14.197 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.671171+0000", 2024-09-18T07:45:14.197 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.197 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.197 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.197 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:45:14.197 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.671173+0000", 2024-09-18T07:45:14.197 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1490000000000001e-06 2024-09-18T07:45:14.197 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.197 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.198 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:45:14.198 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.671180+0000", 2024-09-18T07:45:14.198 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7479999999999996e-06 2024-09-18T07:45:14.198 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.198 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.198 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:45:14.198 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.671193+0000", 2024-09-18T07:45:14.198 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3628999999999999e-05 2024-09-18T07:45:14.198 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.198 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.198 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:45:14.198 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.671208+0000", 2024-09-18T07:45:14.198 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4902999999999999e-05 2024-09-18T07:45:14.199 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.199 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.199 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:45:14.199 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.680638+0000", 2024-09-18T07:45:14.199 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0094301920000000004 2024-09-18T07:45:14.199 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.199 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.199 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:45:14.199 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.680740+0000", 2024-09-18T07:45:14.199 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010204699999999999 2024-09-18T07:45:14.199 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.199 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.199 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:45:14.200 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.680853+0000", 2024-09-18T07:45:14.200 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000112806 2024-09-18T07:45:14.200 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.200 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:45:14.200 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.200 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.200 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.200 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:170422 10.9 10:912ae99d:::benchmark_data_smithi137_151160_object34606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-18T07:45:14.200 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:13.672176+0000", 2024-09-18T07:45:14.200 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.015235013, 2024-09-18T07:45:14.200 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.015336618, 2024-09-18T07:45:14.200 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:45:14.200 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:45:14.201 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:45:14.201 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:45:14.201 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:45:14.201 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:14.201 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 170422 2024-09-18T07:45:14.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.201 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:45:14.201 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.201 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:45:14.201 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.672176+0000", 2024-09-18T07:45:14.201 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.202 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.202 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:45:14.202 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.672176+0000", 2024-09-18T07:45:14.202 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.202 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.202 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:45:14.202 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.672177+0000", 2024-09-18T07:45:14.202 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3599999999999999e-06 2024-09-18T07:45:14.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.202 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.202 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:45:14.202 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.672182+0000", 2024-09-18T07:45:14.202 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9699999999999998e-06 2024-09-18T07:45:14.203 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.203 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.203 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:45:14.203 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.672198+0000", 2024-09-18T07:45:14.203 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6093e-05 2024-09-18T07:45:14.203 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.203 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.203 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:45:14.203 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.672206+0000", 2024-09-18T07:45:14.203 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1600000000000001e-06 2024-09-18T07:45:14.203 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.203 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.204 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:45:14.204 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.683984+0000", 2024-09-18T07:45:14.204 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011778274 2024-09-18T07:45:14.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.204 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.204 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:45:14.204 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684099+0000", 2024-09-18T07:45:14.204 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000114592 2024-09-18T07:45:14.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.204 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.204 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:45:14.204 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684217+0000", 2024-09-18T07:45:14.204 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011877799999999999 2024-09-18T07:45:14.204 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.205 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:45:14.205 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.205 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.205 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop_reply(client.36014.0:170399 10.9 e405/364)", 2024-09-18T07:45:14.205 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:13.684328+0000", 2024-09-18T07:45:14.205 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0030834230000000001, 2024-09-18T07:45:14.205 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0032479539999999999, 2024-09-18T07:45:14.205 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:45:14.205 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:45:14.205 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "queued for pg", 2024-09-18T07:45:14.205 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:45:14.206 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.206 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:45:14.206 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684328+0000", 2024-09-18T07:45:14.206 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.206 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.206 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:45:14.206 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684328+0000", 2024-09-18T07:45:14.206 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.206 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.206 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:45:14.207 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684328+0000", 2024-09-18T07:45:14.207 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8299999999999997e-07 2024-09-18T07:45:14.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.207 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.207 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:45:14.207 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684331+0000", 2024-09-18T07:45:14.207 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6620000000000001e-06 2024-09-18T07:45:14.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.207 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.207 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:45:14.207 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684336+0000", 2024-09-18T07:45:14.207 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8670000000000002e-06 2024-09-18T07:45:14.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.207 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.208 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:45:14.208 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684340+0000", 2024-09-18T07:45:14.208 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2679999999999996e-06 2024-09-18T07:45:14.208 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.208 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:45:14.208 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.208 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.208 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.208 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop_reply(client.36014.0:170414 10.9 e405/364)", 2024-09-18T07:45:14.208 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:13.684346+0000", 2024-09-18T07:45:14.208 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0030654549999999999, 2024-09-18T07:45:14.208 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0032683199999999999, 2024-09-18T07:45:14.208 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:45:14.209 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:45:14.209 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "queued for pg", 2024-09-18T07:45:14.209 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:45:14.209 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.209 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:45:14.209 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684346+0000", 2024-09-18T07:45:14.209 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.209 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.209 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.209 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:45:14.209 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684346+0000", 2024-09-18T07:45:14.209 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.209 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.209 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.210 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:45:14.210 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684346+0000", 2024-09-18T07:45:14.210 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6300000000000001e-07 2024-09-18T07:45:14.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.210 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.210 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:45:14.210 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684348+0000", 2024-09-18T07:45:14.210 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.103e-06 2024-09-18T07:45:14.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.210 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.210 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:45:14.210 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684352+0000", 2024-09-18T07:45:14.210 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2599999999999999e-06 2024-09-18T07:45:14.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.211 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:45:14.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684356+0000", 2024-09-18T07:45:14.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.58e-06 2024-09-18T07:45:14.211 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.211 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:45:14.211 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.211 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.211 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop_reply(client.36014.0:170417 10.9 e405/364)", 2024-09-18T07:45:14.211 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:13.684367+0000", 2024-09-18T07:45:14.211 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0030443520000000002, 2024-09-18T07:45:14.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0032792310000000001, 2024-09-18T07:45:14.212 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:45:14.212 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:45:14.212 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "queued for pg", 2024-09-18T07:45:14.212 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:45:14.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.212 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:45:14.212 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684367+0000", 2024-09-18T07:45:14.212 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.212 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:45:14.212 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684367+0000", 2024-09-18T07:45:14.212 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.213 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.213 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:45:14.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684367+0000", 2024-09-18T07:45:14.213 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2899999999999999e-07 2024-09-18T07:45:14.213 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.213 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.213 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:45:14.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684369+0000", 2024-09-18T07:45:14.213 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.368e-06 2024-09-18T07:45:14.213 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.213 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.213 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:45:14.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684374+0000", 2024-09-18T07:45:14.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2919999999999997e-06 2024-09-18T07:45:14.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.214 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:45:14.214 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.684377+0000", 2024-09-18T07:45:14.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4199999999999999e-06 2024-09-18T07:45:14.214 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.214 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:45:14.214 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.214 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.36014.0:170432 10.3 e405/364 10:c8d48b97:::benchmark_data_smithi137_151160_object34616:head v 405'10744, mlcod=405'10742)", 2024-09-18T07:45:14.214 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:13.686139+0000", 2024-09-18T07:45:14.215 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0012717309999999999, 2024-09-18T07:45:14.215 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0015503509999999999, 2024-09-18T07:45:14.215 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:45:14.215 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:45:14.215 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:45:14.215 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:45:14.215 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.215 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:45:14.215 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.686139+0000", 2024-09-18T07:45:14.215 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.215 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.215 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.215 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:45:14.215 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.686139+0000", 2024-09-18T07:45:14.215 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:45:14.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.686141+0000", 2024-09-18T07:45:14.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.316e-06 2024-09-18T07:45:14.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:45:14.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.686151+0000", 2024-09-18T07:45:14.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8779999999999993e-06 2024-09-18T07:45:14.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:45:14.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.686165+0000", 2024-09-18T07:45:14.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.398e-05 2024-09-18T07:45:14.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:45:14.217 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.686173+0000", 2024-09-18T07:45:14.217 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9959999999999993e-06 2024-09-18T07:45:14.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:45:14.217 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.686248+0000", 2024-09-18T07:45:14.217 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5914999999999994e-05 2024-09-18T07:45:14.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:45:14.218 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.686275+0000", 2024-09-18T07:45:14.218 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.667e-05 2024-09-18T07:45:14.218 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.218 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:45:14.218 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.218 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.218 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.218 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.36014.0:170430 10.7 e405/364 10:e7305dd5:::benchmark_data_smithi137_151160_object34614:head v 405'10774, mlcod=405'10772)", 2024-09-18T07:45:14.218 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:13.687235+0000", 2024-09-18T07:45:14.218 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.000175651, 2024-09-18T07:45:14.218 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00050222000000000005, 2024-09-18T07:45:14.218 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:45:14.218 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:45:14.219 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:45:14.219 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:45:14.219 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.219 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:45:14.219 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.687235+0000", 2024-09-18T07:45:14.219 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.219 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.219 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.219 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:45:14.219 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.687235+0000", 2024-09-18T07:45:14.219 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.219 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.219 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.219 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:45:14.220 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.687237+0000", 2024-09-18T07:45:14.220 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.403e-06 2024-09-18T07:45:14.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.220 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:45:14.220 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.687244+0000", 2024-09-18T07:45:14.220 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1790000000000002e-06 2024-09-18T07:45:14.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.220 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:45:14.220 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.687258+0000", 2024-09-18T07:45:14.220 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4292e-05 2024-09-18T07:45:14.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.221 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:45:14.221 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.687265+0000", 2024-09-18T07:45:14.221 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7510000000000002e-06 2024-09-18T07:45:14.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.221 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.221 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:45:14.221 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.687416+0000", 2024-09-18T07:45:14.221 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015045600000000001 2024-09-18T07:45:14.221 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.221 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:45:14.221 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.221 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.221 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:170399 10.9 10:9a125828:::benchmark_data_smithi137_151160_object34583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-18T07:45:14.222 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:13.652045+0000", 2024-09-18T07:45:14.222 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.035365876999999997, 2024-09-18T07:45:14.222 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.035747037000000002, 2024-09-18T07:45:14.222 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:45:14.222 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:45:14.222 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:45:14.222 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:45:14.222 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:45:14.222 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:14.222 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 170399 2024-09-18T07:45:14.222 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.222 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:45:14.222 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.223 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:45:14.223 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.652045+0000", 2024-09-18T07:45:14.223 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.223 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.223 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:45:14.223 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.652045+0000", 2024-09-18T07:45:14.223 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.223 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.223 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:45:14.223 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.652047+0000", 2024-09-18T07:45:14.223 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.451e-06 2024-09-18T07:45:14.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.223 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.224 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:45:14.224 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.652051+0000", 2024-09-18T07:45:14.224 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0470000000000004e-06 2024-09-18T07:45:14.224 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.224 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.224 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:45:14.224 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.652061+0000", 2024-09-18T07:45:14.224 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0081000000000001e-05 2024-09-18T07:45:14.224 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.224 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.224 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:45:14.224 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.652067+0000", 2024-09-18T07:45:14.224 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5320000000000003e-06 2024-09-18T07:45:14.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.225 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.225 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:45:14.225 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.673531+0000", 2024-09-18T07:45:14.225 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.021463836999999999 2024-09-18T07:45:14.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.225 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.225 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:45:14.225 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.673634+0000", 2024-09-18T07:45:14.225 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000103236 2024-09-18T07:45:14.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.225 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.225 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:45:14.225 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.673737+0000", 2024-09-18T07:45:14.226 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000102977 2024-09-18T07:45:14.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.226 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.226 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:45:14.226 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.683820+0000", 2024-09-18T07:45:14.226 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010082555999999999 2024-09-18T07:45:14.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.226 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.226 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:14.226 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.683905+0000", 2024-09-18T07:45:14.226 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4946999999999998e-05 2024-09-18T07:45:14.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.226 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.226 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:14.227 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.687724+0000", 2024-09-18T07:45:14.227 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003819631 2024-09-18T07:45:14.227 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.227 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:45:14.227 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.227 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.227 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:170414 10.9 10:97106711:::benchmark_data_smithi137_151160_object34598:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-18T07:45:14.227 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:13.666151+0000", 2024-09-18T07:45:14.227 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.021259825, 2024-09-18T07:45:14.227 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.021722983000000001, 2024-09-18T07:45:14.227 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:45:14.227 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:45:14.228 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:45:14.228 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:45:14.228 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:45:14.228 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:14.228 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 170414 2024-09-18T07:45:14.228 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.228 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:45:14.228 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.228 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:45:14.228 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.666151+0000", 2024-09-18T07:45:14.228 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.228 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.228 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.228 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:45:14.229 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.666151+0000", 2024-09-18T07:45:14.229 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:14.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.229 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.229 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:45:14.229 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.666153+0000", 2024-09-18T07:45:14.229 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7689999999999999e-06 2024-09-18T07:45:14.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.229 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.229 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:45:14.229 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.666158+0000", 2024-09-18T07:45:14.229 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7299999999999996e-06 2024-09-18T07:45:14.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.229 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.230 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:45:14.230 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.666172+0000", 2024-09-18T07:45:14.230 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4241e-05 2024-09-18T07:45:14.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.230 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.230 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:45:14.230 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.666179+0000", 2024-09-18T07:45:14.230 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1330000000000001e-06 2024-09-18T07:45:14.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.230 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.230 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:45:14.230 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.677679+0000", 2024-09-18T07:45:14.230 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.011499542999999999 2024-09-18T07:45:14.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.231 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.231 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:45:14.231 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.677778+0000", 2024-09-18T07:45:14.231 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9414999999999996e-05 2024-09-18T07:45:14.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.231 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.231 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:45:14.231 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.677880+0000", 2024-09-18T07:45:14.231 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010202800000000001 2024-09-18T07:45:14.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.231 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.231 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:45:14.231 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.683872+0000", 2024-09-18T07:45:14.231 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0059923540000000001 2024-09-18T07:45:14.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:14.232 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:14.232 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:14.232 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:13.683952+0000", 2024-09-18T07:45:14.232 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9561999999999995e-05 2024-09-18T07:45:14.232 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.232 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:45:14.232 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.232 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:14.232 INFO:teuthology.orchestra.run.smithi203.stdout: ], 2024-09-18T07:45:14.232 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 9 2024-09-18T07:45:14.232 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:45:14.295 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:14.328 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:14.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:14.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:14.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:14.470 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:14.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:14.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:14.572 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:14.600 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:14.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:14.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:14.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:14.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:14.737 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:14.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:14.838 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:14.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:14.891 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:14.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:14.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:15.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:15.028 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:15.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:15.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:15.163 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:15.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:15.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:15.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:15.302 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:15.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:15.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:15.404 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:15.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:15 smithi137 ceph-mon[29408]: 10.9 scrub starts 2024-09-18T07:45:15.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:15 smithi137 ceph-mon[29408]: pgmap v749: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 39513/308421 objects degraded (12.811%); 35 MiB/s, 558 objects/s recovering 2024-09-18T07:45:15.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:15 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:45:15.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:15 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:45:15.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:15 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 39513/308421 objects degraded (12.811%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:15 smithi137 ceph-mon[39175]: 10.9 scrub starts 2024-09-18T07:45:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:15 smithi137 ceph-mon[39175]: pgmap v749: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 39513/308421 objects degraded (12.811%); 35 MiB/s, 558 objects/s recovering 2024-09-18T07:45:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:15 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:45:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:15 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:45:15.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:15 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 39513/308421 objects degraded (12.811%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:15.438 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:15.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:15.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:15 smithi203 ceph-mon[32237]: 10.9 scrub starts 2024-09-18T07:45:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:15 smithi203 ceph-mon[32237]: pgmap v749: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 39513/308421 objects degraded (12.811%); 35 MiB/s, 558 objects/s recovering 2024-09-18T07:45:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:15 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:45:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:15 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:45:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:15 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 39513/308421 objects degraded (12.811%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:15.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:15.574 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:15.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:15.672 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T07:45:15.675 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:15.745 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:45:15.777 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:15.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:15.878 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:15.926 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:15.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:15.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:16.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:16.057 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:16.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:16.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:16.159 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:16.189 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:16.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:16.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:16 smithi203 ceph-mon[32237]: osdmap e406: 8 total, 8 up, 8 in 2024-09-18T07:45:16.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:16 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:45:16.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:16 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:45:16.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:16 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1105504380' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:16.290 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:16.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:16.315 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:16.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:16.334 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:45:16.334 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:45:16.334 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:45:16.334 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:45:16.335 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:45:16.335 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:45:16.335 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:45:16.335 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:45:16.335 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:45:16.335 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:45:16.335 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:45:16.335 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:45:16.335 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:45:16.335 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:45:16.335 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:45:16.336 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:45:16.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:16.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:16 smithi137 ceph-mon[29408]: osdmap e406: 8 total, 8 up, 8 in 2024-09-18T07:45:16.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:16 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:45:16.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:16 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:45:16.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:16 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1105504380' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:16 smithi137 ceph-mon[39175]: osdmap e406: 8 total, 8 up, 8 in 2024-09-18T07:45:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:16 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:45:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:16 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:45:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:16 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1105504380' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:16.446 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:16.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:16.548 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:16.573 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:16.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:16.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:16.675 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:16.714 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:16.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:16.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:16.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:16.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:16.851 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:16.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:16.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:17.003 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:17.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:17.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:17.105 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:17.136 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:17.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:17.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:17.239 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:17.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:17.277 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:17.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:17.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:17.414 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:17.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:17.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:17.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:17 smithi137 ceph-mon[29408]: 10.9 scrub ok 2024-09-18T07:45:17.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:17 smithi137 ceph-mon[29408]: pgmap v751: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 669 B/s rd, 0 B/s wr, 715 op/s; 39513/308421 objects degraded (12.811%); 17 MiB/s, 274 objects/s recovering 2024-09-18T07:45:17.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:17 smithi137 ceph-mon[29408]: 4.11 scrub starts 2024-09-18T07:45:17.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:17 smithi137 ceph-mon[29408]: 4.11 scrub ok 2024-09-18T07:45:17.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:17 smithi137 ceph-mon[29408]: osdmap e407: 8 total, 8 up, 8 in 2024-09-18T07:45:17.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:45:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:45:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:45:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:17 smithi137 ceph-mon[39175]: 10.9 scrub ok 2024-09-18T07:45:17.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:17 smithi137 ceph-mon[39175]: pgmap v751: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 669 B/s rd, 0 B/s wr, 715 op/s; 39513/308421 objects degraded (12.811%); 17 MiB/s, 274 objects/s recovering 2024-09-18T07:45:17.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:17 smithi137 ceph-mon[39175]: 4.11 scrub starts 2024-09-18T07:45:17.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:17 smithi137 ceph-mon[39175]: 4.11 scrub ok 2024-09-18T07:45:17.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:17 smithi137 ceph-mon[39175]: osdmap e407: 8 total, 8 up, 8 in 2024-09-18T07:45:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:17 smithi203 ceph-mon[32237]: 10.9 scrub ok 2024-09-18T07:45:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:17 smithi203 ceph-mon[32237]: pgmap v751: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 669 B/s rd, 0 B/s wr, 715 op/s; 39513/308421 objects degraded (12.811%); 17 MiB/s, 274 objects/s recovering 2024-09-18T07:45:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:17 smithi203 ceph-mon[32237]: 4.11 scrub starts 2024-09-18T07:45:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:17 smithi203 ceph-mon[32237]: 4.11 scrub ok 2024-09-18T07:45:17.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:17 smithi203 ceph-mon[32237]: osdmap e407: 8 total, 8 up, 8 in 2024-09-18T07:45:17.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:17.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:17.562 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:17.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:17.597 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:45:17.649 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:45:17.650 INFO:teuthology.orchestra.run.smithi203.stdout: "success": "" 2024-09-18T07:45:17.650 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:45:17.663 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:17.701 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:17.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:17.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:17.730 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:45:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:45:17.803 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:17.828 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:17.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:17.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:17.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:17.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:17.966 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:17.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:18.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:18.103 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:18.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:18.205 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:18.246 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:18.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:18.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:18.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:18.384 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:18.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:18.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:18.486 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:18.513 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:18.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:18.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:18.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:18.652 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:18.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:18.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:18.754 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:18.789 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:18.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:18.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:18.891 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:18.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:18.929 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:18.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:19.030 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:19.073 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:19.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:19.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:19.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:19.225 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:19.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:19.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:19.233 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:19.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:19.369 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:19 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:19.369 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:19 smithi137 ceph-mon[29408]: pgmap v753: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.20k op/s; 36537/297114 objects degraded (12.297%); 32 MiB/s, 518 objects/s recovering 2024-09-18T07:45:19.370 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:19 smithi137 ceph-mon[29408]: 4.0 scrub starts 2024-09-18T07:45:19.370 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:19 smithi137 ceph-mon[29408]: 4.0 scrub ok 2024-09-18T07:45:19.370 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:19 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:19.370 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:19 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:19.370 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:19 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:19.370 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:19 smithi137 ceph-mon[39175]: pgmap v753: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.20k op/s; 36537/297114 objects degraded (12.297%); 32 MiB/s, 518 objects/s recovering 2024-09-18T07:45:19.370 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:19 smithi137 ceph-mon[39175]: 4.0 scrub starts 2024-09-18T07:45:19.370 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:19 smithi137 ceph-mon[39175]: 4.0 scrub ok 2024-09-18T07:45:19.370 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:19 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:19.371 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:19 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:19.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:19.372 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:19.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:19.466 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:19 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:19.466 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:19 smithi203 ceph-mon[32237]: pgmap v753: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.20k op/s; 36537/297114 objects degraded (12.297%); 32 MiB/s, 518 objects/s recovering 2024-09-18T07:45:19.466 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:19 smithi203 ceph-mon[32237]: 4.0 scrub starts 2024-09-18T07:45:19.466 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:19 smithi203 ceph-mon[32237]: 4.0 scrub ok 2024-09-18T07:45:19.466 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:19 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:19.466 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:19 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:19.473 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:19.503 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:19.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:19.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:19.560 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:45:19.560 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:45:19.605 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:19.645 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:19.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:19.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:19.746 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:19.783 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:19.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:19.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:19.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:19.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:19.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:20.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:20.077 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:20.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:20.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:20.179 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:20.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:20 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 36537/297114 objects degraded (12.297%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:20.235 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:20.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:20.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:20.327 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T07:45:20.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:20.369 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:20.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:20.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:20.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:20 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 36537/297114 objects degraded (12.297%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:20.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:20 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 36537/297114 objects degraded (12.297%), 8 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:20.470 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:20.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:20.508 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:20.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:20.609 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:20.654 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:20.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:20.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:20.758 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:45:20.788 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:20.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:20.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:20.890 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:20.941 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:20.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:20.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:21.043 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:21.074 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:21.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:21.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:45:21.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:21.180 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:45:21.232 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:21.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:21.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:21.380 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:21.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:21.405 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:45:21.406 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:45:21.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:21 smithi137 ceph-mon[29408]: pgmap v754: 158 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.26k op/s; 35229/293481 objects degraded (12.004%); 30 MiB/s, 476 objects/s recovering 2024-09-18T07:45:21.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:21 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4140824678' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:45:21.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:21 smithi137 ceph-mon[39175]: pgmap v754: 158 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.26k op/s; 35229/293481 objects degraded (12.004%); 30 MiB/s, 476 objects/s recovering 2024-09-18T07:45:21.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:21 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4140824678' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:45:21.420 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:21 smithi203 ceph-mon[32237]: pgmap v754: 158 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.26k op/s; 35229/293481 objects degraded (12.004%); 30 MiB/s, 476 objects/s recovering 2024-09-18T07:45:21.420 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:21 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4140824678' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:45:21.481 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:21.516 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:21.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:21.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:21.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:21.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:21.658 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:21.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:21.759 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:21.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:21.795 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:21.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:21.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:21.933 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:21.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:21.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:22.035 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:22.066 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:22.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:22.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:22.169 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:22.207 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:22.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:22.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:22.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:22.347 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:22.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:22.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:22.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:22 smithi137 ceph-mon[39175]: 4.10 scrub starts 2024-09-18T07:45:22.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:22 smithi137 ceph-mon[39175]: 4.10 scrub ok 2024-09-18T07:45:22.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:22 smithi137 ceph-mon[39175]: 3.0 scrub starts 2024-09-18T07:45:22.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:22 smithi137 ceph-mon[39175]: 3.0 scrub ok 2024-09-18T07:45:22.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:22 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:45:22.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:22 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4140824678' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:45:22.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:22 smithi137 ceph-mon[39175]: osdmap e408: 8 total, 8 up, 8 in 2024-09-18T07:45:22.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:22 smithi137 ceph-mon[29408]: 4.10 scrub starts 2024-09-18T07:45:22.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:22 smithi137 ceph-mon[29408]: 4.10 scrub ok 2024-09-18T07:45:22.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:22 smithi137 ceph-mon[29408]: 3.0 scrub starts 2024-09-18T07:45:22.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:22 smithi137 ceph-mon[29408]: 3.0 scrub ok 2024-09-18T07:45:22.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:22 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:45:22.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:22 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4140824678' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:45:22.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:22 smithi137 ceph-mon[29408]: osdmap e408: 8 total, 8 up, 8 in 2024-09-18T07:45:22.449 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:22.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:22 smithi203 ceph-mon[32237]: 4.10 scrub starts 2024-09-18T07:45:22.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:22 smithi203 ceph-mon[32237]: 4.10 scrub ok 2024-09-18T07:45:22.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:22 smithi203 ceph-mon[32237]: 3.0 scrub starts 2024-09-18T07:45:22.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:22 smithi203 ceph-mon[32237]: 3.0 scrub ok 2024-09-18T07:45:22.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:22 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:45:22.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:22 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4140824678' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:45:22.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:22 smithi203 ceph-mon[32237]: osdmap e408: 8 total, 8 up, 8 in 2024-09-18T07:45:22.478 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:22.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:22.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:22.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:22.610 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:22.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:22.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:22.637 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:22.712 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:22.740 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:22.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:22.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:22.841 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:22.868 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:22.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:22.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:22.970 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:22.993 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:22.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:22.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:23.094 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:23.119 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:23.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:23.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:23.178 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:45:23.178 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:45:23.199 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:45:23.199 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:45:23.221 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:45:23.221 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:45:23.221 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:23.245 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:45:23.245 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:45:23.250 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:23.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:23.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:23.278 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:45:23.278 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:45:23.311 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:45:23.312 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:45:23.344 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:45:23.345 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:45:23.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[29408]: pgmap v756: 158 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 150 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 803 op/s; 34392/292431 objects degraded (11.761%); 31 MiB/s, 494 objects/s recovering 2024-09-18T07:45:23.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[29408]: 3.18 scrub starts 2024-09-18T07:45:23.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[29408]: 3.18 scrub ok 2024-09-18T07:45:23.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[29408]: osdmap e409: 8 total, 8 up, 8 in 2024-09-18T07:45:23.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:23.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:23.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:45:23.346 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:45:23.346 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:23.346 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1158464627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:23.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[39175]: pgmap v756: 158 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 150 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 803 op/s; 34392/292431 objects degraded (11.761%); 31 MiB/s, 494 objects/s recovering 2024-09-18T07:45:23.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[39175]: 3.18 scrub starts 2024-09-18T07:45:23.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[39175]: 3.18 scrub ok 2024-09-18T07:45:23.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[39175]: osdmap e409: 8 total, 8 up, 8 in 2024-09-18T07:45:23.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:23.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:23.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:45:23.347 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:45:23.347 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:23.347 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:23 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1158464627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:23.353 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:23.375 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:45:23.375 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:45:23.384 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:23.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:23.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:23 smithi203 ceph-mon[32237]: pgmap v756: 158 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 150 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 803 op/s; 34392/292431 objects degraded (11.761%); 31 MiB/s, 494 objects/s recovering 2024-09-18T07:45:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:23 smithi203 ceph-mon[32237]: 3.18 scrub starts 2024-09-18T07:45:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:23 smithi203 ceph-mon[32237]: 3.18 scrub ok 2024-09-18T07:45:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:23 smithi203 ceph-mon[32237]: osdmap e409: 8 total, 8 up, 8 in 2024-09-18T07:45:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:23 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:23 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:23 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:45:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:23 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:45:23.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:23 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:23.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:23 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1158464627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:23.486 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:23.517 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:23.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:23.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:23.619 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:23.645 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:23.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:23.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:23.746 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:23.770 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:23.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:23.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:23.845 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T07:45:23.872 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:23.913 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:23.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:24.014 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:24.042 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:24.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:24.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:24.133 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:45:24.143 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:24.159 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:45:24.171 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:24.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:24.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:24.273 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:24.293 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-18T07:45:24.293 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:45:24.293 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-18T07:45:24.293 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T07:45:24.293 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T07:45:24.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-18T07:45:24.306 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:24.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:24.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:24.408 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:24.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:24 smithi137 ceph-mon[39175]: osdmap e410: 8 total, 8 up, 8 in 2024-09-18T07:45:24.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:24 smithi137 ceph-mon[29408]: osdmap e410: 8 total, 8 up, 8 in 2024-09-18T07:45:24.440 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:24.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:24.542 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:24.553 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:24.571 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:24.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:24.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:24.673 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:24.710 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:24 smithi203 ceph-mon[32237]: osdmap e410: 8 total, 8 up, 8 in 2024-09-18T07:45:24.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:24.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:24.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:24.834 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:24.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:24.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:24.985 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:24.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:24.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:25.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:25.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:25.134 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:25.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:25.196 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:25.235 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:25.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:25.277 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:25.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:25.317 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[29408]: pgmap v759: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s; 30820/277311 objects degraded (11.114%); 32 MiB/s, 516 objects/s recovering 2024-09-18T07:45:25.317 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[29408]: 3.8 scrub starts 2024-09-18T07:45:25.317 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[29408]: 3.8 scrub ok 2024-09-18T07:45:25.317 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[29408]: 5.b scrub starts 2024-09-18T07:45:25.317 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[29408]: 5.b scrub ok 2024-09-18T07:45:25.317 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[29408]: osdmap e411: 8 total, 8 up, 8 in 2024-09-18T07:45:25.318 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[29408]: 1.2 scrub starts 2024-09-18T07:45:25.318 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[29408]: 1.2 scrub ok 2024-09-18T07:45:25.318 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 30820/277311 objects degraded (11.114%), 7 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:25.318 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[39175]: pgmap v759: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s; 30820/277311 objects degraded (11.114%); 32 MiB/s, 516 objects/s recovering 2024-09-18T07:45:25.318 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[39175]: 3.8 scrub starts 2024-09-18T07:45:25.318 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[39175]: 3.8 scrub ok 2024-09-18T07:45:25.318 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[39175]: 5.b scrub starts 2024-09-18T07:45:25.318 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[39175]: 5.b scrub ok 2024-09-18T07:45:25.319 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[39175]: osdmap e411: 8 total, 8 up, 8 in 2024-09-18T07:45:25.319 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[39175]: 1.2 scrub starts 2024-09-18T07:45:25.319 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[39175]: 1.2 scrub ok 2024-09-18T07:45:25.319 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:25 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 30820/277311 objects degraded (11.114%), 7 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:25.379 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:25.412 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:25.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:25.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:25 smithi203 ceph-mon[32237]: pgmap v759: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s; 30820/277311 objects degraded (11.114%); 32 MiB/s, 516 objects/s recovering 2024-09-18T07:45:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:25 smithi203 ceph-mon[32237]: 3.8 scrub starts 2024-09-18T07:45:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:25 smithi203 ceph-mon[32237]: 3.8 scrub ok 2024-09-18T07:45:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:25 smithi203 ceph-mon[32237]: 5.b scrub starts 2024-09-18T07:45:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:25 smithi203 ceph-mon[32237]: 5.b scrub ok 2024-09-18T07:45:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:25 smithi203 ceph-mon[32237]: osdmap e411: 8 total, 8 up, 8 in 2024-09-18T07:45:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:25 smithi203 ceph-mon[32237]: 1.2 scrub starts 2024-09-18T07:45:25.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:25 smithi203 ceph-mon[32237]: 1.2 scrub ok 2024-09-18T07:45:25.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:25 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 30820/277311 objects degraded (11.114%), 7 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:25.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:25.551 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:25.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:25.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:25.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:25.687 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:25.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:25.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:25.798 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:45:25.798 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "" 2024-09-18T07:45:25.798 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:45:25.811 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:25.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:25.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:25.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:25.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:25.950 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:25.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:26.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:26.092 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:26.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:26.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:26.130 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:45:26.195 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:26.216 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:26.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:26.240 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:45:26.319 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:26.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:26.363 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:26.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:26.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:26 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2242447795' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:45:26.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:26 smithi137 ceph-mon[29408]: from='client.26296 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:45:26.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:26 smithi137 ceph-mon[29408]: 1.0 scrub starts 2024-09-18T07:45:26.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:26 smithi137 ceph-mon[29408]: 1.0 scrub ok 2024-09-18T07:45:26.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:26 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2242447795' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:45:26.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:26 smithi137 ceph-mon[39175]: from='client.26296 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:45:26.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:26 smithi137 ceph-mon[39175]: 1.0 scrub starts 2024-09-18T07:45:26.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:26 smithi137 ceph-mon[39175]: 1.0 scrub ok 2024-09-18T07:45:26.463 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:26.498 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:26.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:26.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:26.600 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:26.631 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:26.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:26.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:26.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:26 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2242447795' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:45:26.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:26 smithi203 ceph-mon[32237]: from='client.26296 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:45:26.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:26 smithi203 ceph-mon[32237]: 1.0 scrub starts 2024-09-18T07:45:26.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:26 smithi203 ceph-mon[32237]: 1.0 scrub ok 2024-09-18T07:45:26.733 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:26.766 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:26.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:26.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:26.868 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:26.901 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:26.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:26.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:27.003 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:27.037 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:27.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:27.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:27.139 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:27.164 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:27.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:27.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:27.301 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:27.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:27.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:27.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:27 smithi137 ceph-mon[29408]: pgmap v761: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s; 30820/277311 objects degraded (11.114%); 34 MiB/s, 548 objects/s recovering 2024-09-18T07:45:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:27 smithi137 ceph-mon[29408]: 3.1e scrub starts 2024-09-18T07:45:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:27 smithi137 ceph-mon[29408]: 3.1e scrub ok 2024-09-18T07:45:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:27 smithi137 ceph-mon[29408]: 10.5 scrub starts 2024-09-18T07:45:27.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:27 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:27.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:27 smithi137 ceph-mon[29408]: from='client.26296 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:45:27.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:27 smithi137 ceph-mon[29408]: osdmap e412: 8 total, 8 up, 8 in 2024-09-18T07:45:27.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:27 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:45:27.421 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:45:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:45:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:45:27.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:27 smithi137 ceph-mon[39175]: pgmap v761: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s; 30820/277311 objects degraded (11.114%); 34 MiB/s, 548 objects/s recovering 2024-09-18T07:45:27.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:27 smithi137 ceph-mon[39175]: 3.1e scrub starts 2024-09-18T07:45:27.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:27 smithi137 ceph-mon[39175]: 3.1e scrub ok 2024-09-18T07:45:27.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:27 smithi137 ceph-mon[39175]: 10.5 scrub starts 2024-09-18T07:45:27.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:27 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:27.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:27 smithi137 ceph-mon[39175]: from='client.26296 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:45:27.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:27 smithi137 ceph-mon[39175]: osdmap e412: 8 total, 8 up, 8 in 2024-09-18T07:45:27.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:27 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:45:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:27.437 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:27.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:27 smithi203 ceph-mon[32237]: pgmap v761: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 150 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s; 30820/277311 objects degraded (11.114%); 34 MiB/s, 548 objects/s recovering 2024-09-18T07:45:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:27 smithi203 ceph-mon[32237]: 3.1e scrub starts 2024-09-18T07:45:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:27 smithi203 ceph-mon[32237]: 3.1e scrub ok 2024-09-18T07:45:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:27 smithi203 ceph-mon[32237]: 10.5 scrub starts 2024-09-18T07:45:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:27 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:27 smithi203 ceph-mon[32237]: from='client.26296 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:45:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:27 smithi203 ceph-mon[32237]: osdmap e412: 8 total, 8 up, 8 in 2024-09-18T07:45:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:27 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:45:27.538 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:27.568 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:27.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:27.612 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:27.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:27.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:27.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:27.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:27.793 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:27.825 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:27.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:27.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:27.927 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:27.956 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:27.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:27.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:28.058 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:28.094 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:28.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:28.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:28.196 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:28.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:45:27 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:45:28.232 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:28.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:28.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:28.236 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:45:28.236 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:45:28.278 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:45:28.279 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:45:28.313 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:28 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2178749921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:28.313 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:28 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2178749921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:28.313 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:45:28.314 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:45:28.333 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:28.334 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:45:28.334 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:45:28.354 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:28.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:28.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:28.399 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:45:28.399 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:45:28.428 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:45:28.429 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:45:28.456 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:28.461 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:45:28.461 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:45:28.494 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:28.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:28.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:28.498 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:45:28.498 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:45:28.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:28.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:28.636 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:28.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:28.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:28 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2178749921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:28.737 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:28.776 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:28.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:28.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:28.879 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:28.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:28.911 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:28.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:29.012 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:29.043 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:29.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:29.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:29.145 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:45:29.181 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:29.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:29.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:45:29.283 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:29.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:29.310 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:29.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:29.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:29.446 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:29.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:29.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:29.500 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:45:29.535 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:29 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:29.536 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:29 smithi137 ceph-mon[29408]: pgmap v763: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 917 B/s rd, 0 B/s wr, 835 op/s; 28478/269346 objects degraded (10.573%); 28 MiB/s, 446 objects/s recovering 2024-09-18T07:45:29.536 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:29 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:29.536 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:29 smithi137 ceph-mon[39175]: pgmap v763: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 917 B/s rd, 0 B/s wr, 835 op/s; 28478/269346 objects degraded (10.573%); 28 MiB/s, 446 objects/s recovering 2024-09-18T07:45:29.546 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:45:29.546 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:45:29.546 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.547 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop_reply(client.36014.0:184160 10.a e412/364)", 2024-09-18T07:45:29.547 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:28.302764+0000", 2024-09-18T07:45:29.547 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.057157025, 2024-09-18T07:45:29.547 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.057178807999999998, 2024-09-18T07:45:29.547 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:45:29.547 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:45:29.547 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "queued for pg", 2024-09-18T07:45:29.547 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:45:29.547 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.547 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:45:29.547 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.302764+0000", 2024-09-18T07:45:29.547 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:29.548 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.548 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.548 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:45:29.548 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.302764+0000", 2024-09-18T07:45:29.548 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:29.548 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.548 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.548 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:45:29.548 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.302765+0000", 2024-09-18T07:45:29.548 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.48e-06 2024-09-18T07:45:29.548 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.548 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.548 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:45:29.548 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.302770+0000", 2024-09-18T07:45:29.549 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6210000000000003e-06 2024-09-18T07:45:29.549 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.549 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.549 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:45:29.549 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.302778+0000", 2024-09-18T07:45:29.549 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6410000000000008e-06 2024-09-18T07:45:29.549 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.549 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.549 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:45:29.549 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.302785+0000", 2024-09-18T07:45:29.549 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4910000000000002e-06 2024-09-18T07:45:29.549 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:29.550 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:45:29.550 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:29.550 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.550 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.550 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop_reply(client.36014.0:184148 10.a e412/364)", 2024-09-18T07:45:29.550 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:28.303182+0000", 2024-09-18T07:45:29.550 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.056738332000000002, 2024-09-18T07:45:29.550 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.056814603999999998, 2024-09-18T07:45:29.550 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:45:29.550 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:45:29.550 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "queued for pg", 2024-09-18T07:45:29.550 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:45:29.550 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.551 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:45:29.551 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.303182+0000", 2024-09-18T07:45:29.551 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:29.551 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.551 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.551 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:45:29.551 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.303182+0000", 2024-09-18T07:45:29.551 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:29.551 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.551 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.551 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:45:29.551 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.303183+0000", 2024-09-18T07:45:29.552 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4799999999999997e-07 2024-09-18T07:45:29.552 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.552 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.552 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:45:29.552 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.303187+0000", 2024-09-18T07:45:29.552 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7500000000000001e-06 2024-09-18T07:45:29.552 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.552 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.552 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:45:29.552 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.303198+0000", 2024-09-18T07:45:29.552 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1171e-05 2024-09-18T07:45:29.552 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.552 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.553 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:45:29.553 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.303214+0000", 2024-09-18T07:45:29.553 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.552e-05 2024-09-18T07:45:29.553 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:29.553 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:45:29.553 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:29.553 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.553 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.553 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:184167 10.f 10:f9e9ea59:::benchmark_data_smithi137_151160_object48351:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-18T07:45:29.553 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:28.303306+0000", 2024-09-18T07:45:29.553 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.056614787, 2024-09-18T07:45:29.553 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.056720117, 2024-09-18T07:45:29.553 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:45:29.554 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:45:29.554 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:45:29.554 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:45:29.554 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:45:29.554 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:29.554 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 184167 2024-09-18T07:45:29.554 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.554 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:45:29.554 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.554 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:45:29.554 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.303306+0000", 2024-09-18T07:45:29.554 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:29.554 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.555 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.555 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:45:29.555 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.303306+0000", 2024-09-18T07:45:29.555 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:29.555 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.555 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.555 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:45:29.555 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.303307+0000", 2024-09-18T07:45:29.555 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2359999999999999e-06 2024-09-18T07:45:29.555 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.555 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.555 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:45:29.555 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.303311+0000", 2024-09-18T07:45:29.556 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9530000000000001e-06 2024-09-18T07:45:29.556 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.556 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.556 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:45:29.556 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.303319+0000", 2024-09-18T07:45:29.556 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7849999999999992e-06 2024-09-18T07:45:29.556 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.556 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.556 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:45:29.556 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.303325+0000", 2024-09-18T07:45:29.556 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9279999999999998e-06 2024-09-18T07:45:29.556 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.556 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.557 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:45:29.557 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.303567+0000", 2024-09-18T07:45:29.557 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00024181800000000001 2024-09-18T07:45:29.557 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.557 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.557 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:45:29.557 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.303656+0000", 2024-09-18T07:45:29.557 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8885e-05 2024-09-18T07:45:29.557 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.557 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.557 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:45:29.557 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.303726+0000", 2024-09-18T07:45:29.557 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0439999999999996e-05 2024-09-18T07:45:29.558 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:29.558 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:45:29.558 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:29.558 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.558 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.558 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:184148 10.a 10:56fd3a11:::benchmark_data_smithi137_151160_object48332:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-18T07:45:29.558 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:28.287225+0000", 2024-09-18T07:45:29.558 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.072695281, 2024-09-18T07:45:29.558 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.072839498000000003, 2024-09-18T07:45:29.558 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:45:29.558 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:45:29.558 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:45:29.558 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:45:29.558 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:45:29.558 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:29.559 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 184148 2024-09-18T07:45:29.559 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.559 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:45:29.559 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.559 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:45:29.559 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.287225+0000", 2024-09-18T07:45:29.559 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:29.559 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.559 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.559 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:45:29.559 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.287225+0000", 2024-09-18T07:45:29.559 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:29.559 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.559 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.560 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:45:29.560 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.287226+0000", 2024-09-18T07:45:29.560 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6899999999999997e-07 2024-09-18T07:45:29.560 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.560 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.560 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:45:29.560 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.287229+0000", 2024-09-18T07:45:29.560 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.88e-06 2024-09-18T07:45:29.560 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.560 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.560 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:45:29.560 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.287234+0000", 2024-09-18T07:45:29.561 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0170000000000004e-06 2024-09-18T07:45:29.561 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.561 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.561 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:45:29.561 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.287243+0000", 2024-09-18T07:45:29.561 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6400000000000003e-06 2024-09-18T07:45:29.561 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.561 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.561 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:45:29.561 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.287311+0000", 2024-09-18T07:45:29.561 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8616999999999997e-05 2024-09-18T07:45:29.561 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.561 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.562 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:45:29.562 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.287421+0000", 2024-09-18T07:45:29.562 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010980300000000001 2024-09-18T07:45:29.562 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.562 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.562 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:45:29.562 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.287521+0000", 2024-09-18T07:45:29.562 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9951999999999994e-05 2024-09-18T07:45:29.562 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.562 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.562 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:45:29.562 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.290266+0000", 2024-09-18T07:45:29.562 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002745206 2024-09-18T07:45:29.563 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.563 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.563 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:29.563 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.293365+0000", 2024-09-18T07:45:29.563 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0030989839999999999 2024-09-18T07:45:29.563 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:29.563 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:45:29.563 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:29.563 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.563 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.563 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:184160 10.a 10:519ee8d5:::benchmark_data_smithi137_151160_object48344:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-18T07:45:29.563 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:28.296129+0000", 2024-09-18T07:45:29.563 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.063792250999999994, 2024-09-18T07:45:29.564 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.06397717, 2024-09-18T07:45:29.564 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:45:29.564 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:45:29.564 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:45:29.564 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:45:29.564 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:45:29.564 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:29.564 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 184160 2024-09-18T07:45:29.564 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.564 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:45:29.564 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.564 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:45:29.564 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.296129+0000", 2024-09-18T07:45:29.565 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:29.565 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.565 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.565 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:45:29.565 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.296129+0000", 2024-09-18T07:45:29.565 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:29.565 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.565 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.565 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:45:29.565 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.296130+0000", 2024-09-18T07:45:29.565 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4419999999999999e-06 2024-09-18T07:45:29.565 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.565 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.565 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:45:29.566 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.296135+0000", 2024-09-18T07:45:29.566 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.882e-06 2024-09-18T07:45:29.566 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.566 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.566 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:45:29.566 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.296146+0000", 2024-09-18T07:45:29.566 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0668e-05 2024-09-18T07:45:29.566 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.566 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.566 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:45:29.566 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.296153+0000", 2024-09-18T07:45:29.566 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2799999999999998e-06 2024-09-18T07:45:29.566 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.567 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.567 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:45:29.567 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.296212+0000", 2024-09-18T07:45:29.567 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9085000000000001e-05 2024-09-18T07:45:29.567 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.567 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.567 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:45:29.567 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.296324+0000", 2024-09-18T07:45:29.567 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0001119 2024-09-18T07:45:29.567 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.567 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.567 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:45:29.567 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.296433+0000", 2024-09-18T07:45:29.568 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010969 2024-09-18T07:45:29.568 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.568 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.568 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:45:29.568 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.302560+0000", 2024-09-18T07:45:29.568 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0061262770000000003 2024-09-18T07:45:29.568 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:29.568 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:45:29.568 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:29.568 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.568 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.568 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:184161 10.f 10:f9ff2b60:::benchmark_data_smithi137_151160_object48345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-18T07:45:29.568 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:28.297106+0000", 2024-09-18T07:45:29.569 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.062814775000000003, 2024-09-18T07:45:29.569 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.063034200999999998, 2024-09-18T07:45:29.569 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:45:29.569 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:45:29.569 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:45:29.569 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:45:29.569 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:45:29.569 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:29.569 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 184161 2024-09-18T07:45:29.569 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.569 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:45:29.569 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.569 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:45:29.570 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.297106+0000", 2024-09-18T07:45:29.570 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:29.570 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.570 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.570 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:45:29.570 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.297106+0000", 2024-09-18T07:45:29.570 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:29.570 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.570 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.570 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:45:29.570 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.297107+0000", 2024-09-18T07:45:29.570 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3510000000000001e-06 2024-09-18T07:45:29.570 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.571 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.571 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:45:29.571 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.297115+0000", 2024-09-18T07:45:29.571 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7540000000000006e-06 2024-09-18T07:45:29.571 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.571 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.571 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:45:29.571 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.297126+0000", 2024-09-18T07:45:29.571 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0841e-05 2024-09-18T07:45:29.571 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.571 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.571 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:45:29.571 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.297133+0000", 2024-09-18T07:45:29.572 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4580000000000004e-06 2024-09-18T07:45:29.572 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.572 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.572 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:45:29.572 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.297526+0000", 2024-09-18T07:45:29.572 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00039230899999999997 2024-09-18T07:45:29.572 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.572 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.572 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:45:29.572 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.297620+0000", 2024-09-18T07:45:29.572 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4331000000000002e-05 2024-09-18T07:45:29.572 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.572 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.572 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,3", 2024-09-18T07:45:29.573 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.297720+0000", 2024-09-18T07:45:29.573 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9709000000000004e-05 2024-09-18T07:45:29.573 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.573 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.573 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:45:29.573 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.302493+0000", 2024-09-18T07:45:29.573 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0047731060000000001 2024-09-18T07:45:29.573 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.573 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.573 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:29.573 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.303531+0000", 2024-09-18T07:45:29.573 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010381100000000001 2024-09-18T07:45:29.573 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:29.574 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:45:29.574 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:29.574 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.574 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.574 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:184163 10.d 10:bd0a87f4:::benchmark_data_smithi137_151160_object48347:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-18T07:45:29.574 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:28.299272+0000", 2024-09-18T07:45:29.574 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.060648569999999999, 2024-09-18T07:45:29.574 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.060904789000000001, 2024-09-18T07:45:29.574 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:45:29.574 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:45:29.574 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:45:29.574 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:45:29.575 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:45:29.575 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:29.575 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 184163 2024-09-18T07:45:29.575 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.575 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:45:29.575 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.575 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:45:29.575 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.299272+0000", 2024-09-18T07:45:29.575 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:29.575 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.575 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.575 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:45:29.575 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.299272+0000", 2024-09-18T07:45:29.575 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:29.576 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.576 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.576 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:45:29.576 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.299274+0000", 2024-09-18T07:45:29.576 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.097e-06 2024-09-18T07:45:29.576 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.576 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.576 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:45:29.576 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.299280+0000", 2024-09-18T07:45:29.576 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.879e-06 2024-09-18T07:45:29.576 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.576 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.576 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:45:29.577 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.299291+0000", 2024-09-18T07:45:29.577 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0939e-05 2024-09-18T07:45:29.577 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.577 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.577 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:45:29.577 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.299299+0000", 2024-09-18T07:45:29.577 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8390000000000007e-06 2024-09-18T07:45:29.577 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.577 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.577 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:45:29.577 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.299355+0000", 2024-09-18T07:45:29.577 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6322e-05 2024-09-18T07:45:29.577 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.578 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.578 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:45:29.578 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.299459+0000", 2024-09-18T07:45:29.578 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000104138 2024-09-18T07:45:29.578 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.578 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.578 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,4", 2024-09-18T07:45:29.578 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.299566+0000", 2024-09-18T07:45:29.578 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010634800000000001 2024-09-18T07:45:29.578 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.578 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.578 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:45:29.578 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.302726+0000", 2024-09-18T07:45:29.578 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0031599369999999998 2024-09-18T07:45:29.579 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.579 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.579 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:29.579 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.305424+0000", 2024-09-18T07:45:29.579 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026982579999999998 2024-09-18T07:45:29.579 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:29.579 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:45:29.579 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:29.579 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.579 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.579 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:184164 10.a 10:55fd4a8e:::benchmark_data_smithi137_151160_object48348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-18T07:45:29.579 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:28.300360+0000", 2024-09-18T07:45:29.579 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.059561204, 2024-09-18T07:45:29.579 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.059855919, 2024-09-18T07:45:29.579 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:45:29.580 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:45:29.580 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:45:29.580 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:45:29.580 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:45:29.580 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:29.580 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 184164 2024-09-18T07:45:29.580 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.580 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:45:29.580 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.580 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:45:29.580 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.300360+0000", 2024-09-18T07:45:29.580 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:29.580 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.580 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.581 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:45:29.581 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.300360+0000", 2024-09-18T07:45:29.581 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:45:29.581 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.581 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.581 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:45:29.581 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.300361+0000", 2024-09-18T07:45:29.581 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5060000000000001e-06 2024-09-18T07:45:29.581 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.581 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.581 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:45:29.581 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.300370+0000", 2024-09-18T07:45:29.581 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4440000000000008e-06 2024-09-18T07:45:29.581 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.582 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.582 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:45:29.582 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.300381+0000", 2024-09-18T07:45:29.582 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.134e-05 2024-09-18T07:45:29.582 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.582 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.582 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:45:29.582 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.300388+0000", 2024-09-18T07:45:29.582 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4699999999999996e-06 2024-09-18T07:45:29.582 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.582 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.582 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:45:29.582 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.302662+0000", 2024-09-18T07:45:29.582 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022735239999999999 2024-09-18T07:45:29.582 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.583 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.583 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:45:29.583 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.302774+0000", 2024-09-18T07:45:29.583 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011191100000000001 2024-09-18T07:45:29.583 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:45:29.583 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:45:29.583 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,5", 2024-09-18T07:45:29.583 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:28.302926+0000", 2024-09-18T07:45:29.583 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000152483 2024-09-18T07:45:29.583 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:29.583 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:45:29.583 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:29.583 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:45:29.583 INFO:teuthology.orchestra.run.smithi203.stdout: ], 2024-09-18T07:45:29.583 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 8 2024-09-18T07:45:29.584 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:45:29.584 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:29 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:29.584 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:29 smithi203 ceph-mon[32237]: pgmap v763: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 917 B/s rd, 0 B/s wr, 835 op/s; 28478/269346 objects degraded (10.573%); 28 MiB/s, 446 objects/s recovering 2024-09-18T07:45:29.584 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:29.609 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:29.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:29.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:29.711 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:29.760 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:29.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:29.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:29.862 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:29.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:29.908 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:29.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:29.925 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:30.009 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:30.053 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:30.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:30.154 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:30.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:30.192 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:30.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:30.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:30.317 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:30.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:30.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:30.365 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:30 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:45:30.365 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:30 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:45:30.366 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:30 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 28478/269346 objects degraded (10.573%), 6 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:30.366 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:30 smithi137 ceph-mon[29408]: osdmap e413: 8 total, 8 up, 8 in 2024-09-18T07:45:30.366 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:30 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:45:30.366 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:30 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:45:30.367 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:30 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 28478/269346 objects degraded (10.573%), 6 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:30.367 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:30 smithi137 ceph-mon[39175]: osdmap e413: 8 total, 8 up, 8 in 2024-09-18T07:45:30.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:30.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:30.461 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:30.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:30.562 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:30.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:30.607 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:30.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:30.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:30.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:30 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:45:30.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:30 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:45:30.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:30 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 28478/269346 objects degraded (10.573%), 6 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:30.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:30 smithi203 ceph-mon[32237]: osdmap e413: 8 total, 8 up, 8 in 2024-09-18T07:45:30.745 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:30.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:30.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:30.847 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:30.878 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:30.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:30.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:30.980 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:31.001 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:31.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:31.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:31.103 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:31.144 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:31.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:31.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:31.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:31.279 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:45:31.293 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:31.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:31.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:31.339 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:45:31.366 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:31 smithi137 ceph-mon[29408]: pgmap v764: 158 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 778 B/s rd, 0 B/s wr, 558 op/s; 25991/266316 objects degraded (9.759%); 35 MiB/s, 559 objects/s recovering 2024-09-18T07:45:31.366 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:31 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1873547558' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:45:31.366 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:31 smithi137 ceph-mon[39175]: pgmap v764: 158 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 778 B/s rd, 0 B/s wr, 558 op/s; 25991/266316 objects degraded (9.759%); 35 MiB/s, 559 objects/s recovering 2024-09-18T07:45:31.367 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:31 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1873547558' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:45:31.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:31.431 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:31.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:31.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:31.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:31.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:31.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:31.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:31.673 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:31.685 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-18T07:45:31.685 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:45:31.685 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-18T07:45:31.685 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-18T07:45:31.685 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.5 2024-09-18T07:45:31.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:31 smithi203 ceph-mon[32237]: pgmap v764: 158 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 778 B/s rd, 0 B/s wr, 558 op/s; 25991/266316 objects degraded (9.759%); 35 MiB/s, 559 objects/s recovering 2024-09-18T07:45:31.762 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:31 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1873547558' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:45:31.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:31.763 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:31.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:31.865 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:31.899 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:31.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:31.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:32.001 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:32.040 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:32.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:32.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:32.142 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:32.182 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:32.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:32.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:32.211 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:45:31 smithi203 systemd[1]: Stopping Ceph osd.5 for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:45:32.284 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:32.292 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:32.321 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:32.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:32.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:32 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:32.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:32 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1873547558' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:45:32.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:32 smithi137 ceph-mon[29408]: osdmap e414: 8 total, 8 up, 8 in 2024-09-18T07:45:32.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:32 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:32.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:32 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1873547558' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:45:32.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:32 smithi137 ceph-mon[39175]: osdmap e414: 8 total, 8 up, 8 in 2024-09-18T07:45:32.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:32.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:32.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:32.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:32.559 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:32.590 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:32.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:32.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:32.691 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:32.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:32 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:32.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:32 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1873547558' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:45:32.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:32 smithi203 ceph-mon[32237]: osdmap e414: 8 total, 8 up, 8 in 2024-09-18T07:45:32.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:32.725 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:32.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:32.826 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:32.868 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:32.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:32.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:32.933 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:45:32.934 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:45:32.966 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:45:32.967 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:45:32.970 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:33.002 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:45:33.002 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:45:33.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:33.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:33.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:33.033 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:45:33.033 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:45:33.061 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:45:33.062 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:45:33.095 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:45:33.095 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:45:33.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:33.133 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:45:33.133 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:45:33.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:33.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:33.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:33.169 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:45:33.169 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:45:33.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:33.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:33.293 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:33.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:33.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:33.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:33 smithi137 ceph-mon[29408]: pgmap v767: 158 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 668 op/s; 25991/265296 objects degraded (9.797%); 38 MiB/s, 613 objects/s recovering 2024-09-18T07:45:33.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:33 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2562979216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:33.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:33 smithi137 ceph-mon[39175]: pgmap v767: 158 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 668 op/s; 25991/265296 objects degraded (9.797%); 38 MiB/s, 613 objects/s recovering 2024-09-18T07:45:33.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:33 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2562979216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:33.427 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:33.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:33.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:33.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:33 smithi203 ceph-mon[32237]: pgmap v767: 158 pgs: 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 668 op/s; 25991/265296 objects degraded (9.797%); 38 MiB/s, 613 objects/s recovering 2024-09-18T07:45:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:33 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2562979216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:33.529 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:33.555 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:33.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:33.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:33.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:33.678 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:33.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:33.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:33.780 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:33.806 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:33.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:33.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:33.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:33.930 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:33.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:34.031 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:34.052 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:34.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:34.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:34.122 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T07:45:34.142 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:45:34.154 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:34.193 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:34.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:34.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:34.295 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:34.330 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:34.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:34.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:34.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:34 smithi137 ceph-mon[29408]: osdmap e415: 8 total, 8 up, 8 in 2024-09-18T07:45:34.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:34 smithi137 ceph-mon[29408]: osd.5 marked itself down and dead 2024-09-18T07:45:34.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:34 smithi137 ceph-mon[39175]: osdmap e415: 8 total, 8 up, 8 in 2024-09-18T07:45:34.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:34 smithi137 ceph-mon[39175]: osd.5 marked itself down and dead 2024-09-18T07:45:34.428 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:34.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:34.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:34 smithi203 ceph-mon[32237]: osdmap e415: 8 total, 8 up, 8 in 2024-09-18T07:45:34.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:34 smithi203 ceph-mon[32237]: osd.5 marked itself down and dead 2024-09-18T07:45:34.462 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:45:34 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[41787]: 2024-09-18T07:45:34.281+0000 7f687555a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T07:45:34.462 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:45:34 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[41787]: 2024-09-18T07:45:34.281+0000 7f687555a640 -1 osd.5 415 *** Got signal Terminated *** 2024-09-18T07:45:34.462 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:45:34 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[41787]: 2024-09-18T07:45:34.281+0000 7f687555a640 -1 osd.5 415 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T07:45:34.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:34.463 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:34.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:34.479 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:45:34.564 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:34.591 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:34.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:34.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:34.692 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:34.719 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:34.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:34.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:34.821 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:34.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:34.849 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:34.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:34.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:34.973 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:34.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:34.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:35.075 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:35.116 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:35.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:35.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:35.207 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:45:35.207 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:45:35.207 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:45:35.207 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:45:35.207 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:45:35.218 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:35.250 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:35.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:35.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:35.352 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:35.379 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:35.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:35.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:35.460 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:45:35 smithi203 podman[124026]: 2024-09-18 07:45:35.184957405 +0000 UTC m=+1.039368821 container died 5b4069480d1a256d4d1b10a4ec1b0bca576b8b723a4e06b7faaf3a430423c773 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-18T07:45:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:35 smithi203 ceph-mon[32237]: pgmap v769: 158 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 22494/250317 objects degraded (8.986%); 42 MiB/s, 666 objects/s recovering 2024-09-18T07:45:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:35 smithi203 ceph-mon[32237]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-18T07:45:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:35 smithi203 ceph-mon[32237]: osdmap e416: 8 total, 7 up, 8 in 2024-09-18T07:45:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:35 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 22494/250317 objects degraded (8.986%), 5 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:35 smithi203 ceph-mon[32237]: osdmap e417: 8 total, 7 up, 8 in 2024-09-18T07:45:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:35 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4083368127' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:45:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:35 smithi203 ceph-mon[32237]: from='client.36230 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:45:35.482 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:35.507 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:35.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:35.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:35.610 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:35.633 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:35.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:35.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:35.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:35 smithi137 ceph-mon[39175]: pgmap v769: 158 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 22494/250317 objects degraded (8.986%); 42 MiB/s, 666 objects/s recovering 2024-09-18T07:45:35.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:35 smithi137 ceph-mon[39175]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-18T07:45:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:35 smithi137 ceph-mon[39175]: osdmap e416: 8 total, 7 up, 8 in 2024-09-18T07:45:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:35 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 22494/250317 objects degraded (8.986%), 5 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:35 smithi137 ceph-mon[39175]: osdmap e417: 8 total, 7 up, 8 in 2024-09-18T07:45:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:35 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4083368127' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:45:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:35 smithi137 ceph-mon[39175]: from='client.36230 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:45:35.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:35 smithi137 ceph-mon[29408]: pgmap v769: 158 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 152 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 22494/250317 objects degraded (8.986%); 42 MiB/s, 666 objects/s recovering 2024-09-18T07:45:35.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:35 smithi137 ceph-mon[29408]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-18T07:45:35.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:35 smithi137 ceph-mon[29408]: osdmap e416: 8 total, 7 up, 8 in 2024-09-18T07:45:35.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:35 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 22494/250317 objects degraded (8.986%), 5 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:35.672 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:35 smithi137 ceph-mon[29408]: osdmap e417: 8 total, 7 up, 8 in 2024-09-18T07:45:35.672 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:35 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4083368127' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:45:35.672 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:35 smithi137 ceph-mon[29408]: from='client.36230 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:45:35.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:35.759 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:35.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:35.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:35.860 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:35.887 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:35.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:35.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:35.960 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:45:35 smithi203 podman[124026]: 2024-09-18 07:45:35.628370023 +0000 UTC m=+1.482781424 container cleanup 5b4069480d1a256d4d1b10a4ec1b0bca576b8b723a4e06b7faaf3a430423c773 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-18T07:45:35.961 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:45:35 smithi203 bash[124026]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5 2024-09-18T07:45:35.989 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:36.014 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:36.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:36.086 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:45:36.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:36.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:45:36.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:36.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:36.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:36.242 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:36.267 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:36.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:36.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:36.369 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:36.390 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:45:36 smithi203 podman[124186]: 2024-09-18 07:45:36.08293584 +0000 UTC m=+1.070448384 container remove 5b4069480d1a256d4d1b10a4ec1b0bca576b8b723a4e06b7faaf3a430423c773 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS) 2024-09-18T07:45:36.390 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:36 smithi203 ceph-mon[32237]: 10.6 continuing backfill to osd.4 from (383'6900,415'11989] MIN to 415'11989 2024-09-18T07:45:36.390 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:36 smithi203 ceph-mon[32237]: pgmap v772: 158 pgs: 1 stale+active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 28 stale+active+clean, 1 active+undersized+remapped, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 124 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.19k op/s; 22494/250317 objects degraded (8.986%); 28 MiB/s, 447 objects/s recovering 2024-09-18T07:45:36.390 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:36 smithi203 ceph-mon[32237]: from='client.36230 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:45:36.390 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:36 smithi203 ceph-mon[32237]: osdmap e418: 8 total, 7 up, 8 in 2024-09-18T07:45:36.394 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:36.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:36.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:36.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:36 smithi137 ceph-mon[29408]: 10.6 continuing backfill to osd.4 from (383'6900,415'11989] MIN to 415'11989 2024-09-18T07:45:36.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:36 smithi137 ceph-mon[29408]: pgmap v772: 158 pgs: 1 stale+active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 28 stale+active+clean, 1 active+undersized+remapped, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 124 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.19k op/s; 22494/250317 objects degraded (8.986%); 28 MiB/s, 447 objects/s recovering 2024-09-18T07:45:36.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:36 smithi137 ceph-mon[29408]: from='client.36230 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:45:36.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:36 smithi137 ceph-mon[29408]: osdmap e418: 8 total, 7 up, 8 in 2024-09-18T07:45:36.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:36 smithi137 ceph-mon[39175]: 10.6 continuing backfill to osd.4 from (383'6900,415'11989] MIN to 415'11989 2024-09-18T07:45:36.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:36 smithi137 ceph-mon[39175]: pgmap v772: 158 pgs: 1 stale+active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 28 stale+active+clean, 1 active+undersized+remapped, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 124 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.19k op/s; 22494/250317 objects degraded (8.986%); 28 MiB/s, 447 objects/s recovering 2024-09-18T07:45:36.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:36 smithi137 ceph-mon[39175]: from='client.36230 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:45:36.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:36 smithi137 ceph-mon[39175]: osdmap e418: 8 total, 7 up, 8 in 2024-09-18T07:45:36.496 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:36.521 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:36.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:36.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:36.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:36.633 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T07:45:36.646 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:36.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:36.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:36.748 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:36.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:36.775 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:36.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:36.876 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:36.904 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:36.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:36.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:37.006 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:37.035 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:37.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:37.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:37.043 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:45:37.138 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:37.163 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:37.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:37.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:37.259 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:37.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:37.289 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:37.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:37.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:37.380 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:45:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:45:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:45:37.392 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:37.415 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:37.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:37.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:37.460 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:45:37 smithi203 podman[124480]: 2024-09-18 07:45:36.95989061 +0000 UTC m=+0.025285974 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:45:37.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:37.542 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:37.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:37.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:37.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:37.666 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:37.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:37.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:37.768 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:37.793 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:37.794 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:45:37.794 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:45:37.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:37.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:37.815 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:45:37.815 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:45:37.834 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:45:37.835 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:45:37.854 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:45:37.854 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:45:37.872 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:45:37.872 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:45:37.872 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-18T07:45:37.893 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:45:37.893 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:45:37.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:37.914 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:45:37.915 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:45:37.918 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:37.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:37.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:38.020 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:38.045 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:38.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:38.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:38.146 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:38.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:37 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3961263760' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:38.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:37 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3961263760' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:38.172 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:38.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:38.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:38.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:37 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3961263760' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:38.211 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:45:37 smithi203 podman[124480]: 2024-09-18 07:45:37.955794189 +0000 UTC m=+1.021189547 container create 4b1bbeaf9eb9c1de67b5c39590531e0012c83bb7aff855333a740d6d318782ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T07:45:38.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:45:37 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:45:38.274 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:38.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:38.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:38.398 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:38.424 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:38.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:38.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:38.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:38.549 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:38.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:38.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:38.651 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:38.674 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:38.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:38.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:38.703 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:45:38 smithi203 podman[124480]: 2024-09-18 07:45:38.450253402 +0000 UTC m=+1.515648750 container init 4b1bbeaf9eb9c1de67b5c39590531e0012c83bb7aff855333a740d6d318782ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T07:45:38.703 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:45:38 smithi203 podman[124480]: 2024-09-18 07:45:38.454901405 +0000 UTC m=+1.520296754 container start 4b1bbeaf9eb9c1de67b5c39590531e0012c83bb7aff855333a740d6d318782ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-18T07:45:38.703 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:45:38 smithi203 podman[124480]: 2024-09-18 07:45:38.55035911 +0000 UTC m=+1.615754487 container attach 4b1bbeaf9eb9c1de67b5c39590531e0012c83bb7aff855333a740d6d318782ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-18T07:45:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:45:38.734 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:45:38.776 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:38.802 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:38.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:38.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:38.904 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:38.935 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:38.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:38.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:38.957 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:45:38 smithi203 podman[124480]: 2024-09-18 07:45:38.919078645 +0000 UTC m=+1.984474006 container died 4b1bbeaf9eb9c1de67b5c39590531e0012c83bb7aff855333a740d6d318782ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-18T07:45:38.957 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:38 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:38.957 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:38 smithi203 ceph-mon[32237]: pgmap v774: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 29 active+undersized, 1 stale+active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 6 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 16 active+undersized+degraded, 102 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 9.4 KiB/s rd, 66 KiB/s wr, 312 op/s; 40004/246231 objects degraded (16.247%); 40 MiB/s, 633 objects/s recovering 2024-09-18T07:45:38.957 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:38 smithi203 ceph-mon[32237]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:45:39.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:39.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:39.060 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:39.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:39.161 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:39.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:38 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:39.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:38 smithi137 ceph-mon[29408]: pgmap v774: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 29 active+undersized, 1 stale+active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 6 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 16 active+undersized+degraded, 102 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 9.4 KiB/s rd, 66 KiB/s wr, 312 op/s; 40004/246231 objects degraded (16.247%); 40 MiB/s, 633 objects/s recovering 2024-09-18T07:45:39.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:38 smithi137 ceph-mon[29408]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:45:39.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:38 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:39.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:38 smithi137 ceph-mon[39175]: pgmap v774: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 29 active+undersized, 1 stale+active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 6 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 16 active+undersized+degraded, 102 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 9.4 KiB/s rd, 66 KiB/s wr, 312 op/s; 40004/246231 objects degraded (16.247%); 40 MiB/s, 633 objects/s recovering 2024-09-18T07:45:39.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:38 smithi137 ceph-mon[39175]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:45:39.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:45:39.184 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:39.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:39.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:39.287 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:39.311 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:39.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:39.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:39.413 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:39.437 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:39.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:39.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:39.487 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:39.539 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:39.564 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:39.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:39.666 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.5.service' 2024-09-18T07:45:39.667 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:39.692 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:39.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:39.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:39.711 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T07:45:39.712 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-18T07:45:39.712 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-09-18T07:45:39.712 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi203.front.sepia.ceph.com 2024-09-18T07:45:39.712 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -n osd.5 --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-09-18T07:45:39.788 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:45:39.789 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:45:39.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:39.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:39.817 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:39.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:39.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:39.941 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:39.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:39.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:40.043 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:40.066 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:40.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:40.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:40.169 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/osd.5/config 2024-09-18T07:45:40.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:40.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:40.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:40.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:40.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:40.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:40.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:40.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:40.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:40.438 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:40.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:40.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:40.564 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:40.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:40.666 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:40.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:40.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:40.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:40.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:45:40.791 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:40.816 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:40.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:40.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:40.942 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:40.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:40.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:41.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:41.068 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:41.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:41.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:41.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:40 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:45:41.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:40 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:45:41.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:40 smithi137 ceph-mon[39175]: pgmap v775: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 36 active+undersized, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 94 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 10 KiB/s rd, 74 KiB/s wr, 280 op/s; 40747/245463 objects degraded (16.600%); 74 MiB/s, 1.18k objects/s recovering 2024-09-18T07:45:41.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:40 smithi137 ceph-mon[39175]: osdmap e419: 8 total, 7 up, 8 in 2024-09-18T07:45:41.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:40 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:41.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:40 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 40747/245463 objects degraded (16.600%), 28 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:41.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:40 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:45:41.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:40 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:45:41.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:40 smithi137 ceph-mon[29408]: pgmap v775: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 36 active+undersized, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 94 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 10 KiB/s rd, 74 KiB/s wr, 280 op/s; 40747/245463 objects degraded (16.600%); 74 MiB/s, 1.18k objects/s recovering 2024-09-18T07:45:41.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:40 smithi137 ceph-mon[29408]: osdmap e419: 8 total, 7 up, 8 in 2024-09-18T07:45:41.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:40 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:41.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:40 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 40747/245463 objects degraded (16.600%), 28 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:41.171 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:41.195 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:41.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:41.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:40 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:45:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:40 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:45:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:40 smithi203 ceph-mon[32237]: pgmap v775: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 36 active+undersized, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 94 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 10 KiB/s rd, 74 KiB/s wr, 280 op/s; 40747/245463 objects degraded (16.600%); 74 MiB/s, 1.18k objects/s recovering 2024-09-18T07:45:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:40 smithi203 ceph-mon[32237]: osdmap e419: 8 total, 7 up, 8 in 2024-09-18T07:45:41.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:40 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:41.212 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:40 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 40747/245463 objects degraded (16.600%), 28 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:41.297 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:41.319 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:41.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:41.416 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T07:45:41.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:41.444 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:41.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:41.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:41.547 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:41.575 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:41.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:41.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:41.677 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:41.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:41.702 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:41.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:41.803 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:41.829 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:41.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:41.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:41.857 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:45:41.931 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:41.957 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:41.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:42.058 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:42.081 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:42.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:42.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:42.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:41 smithi137 ceph-mon[29408]: osdmap e420: 8 total, 7 up, 8 in 2024-09-18T07:45:42.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:41 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/547518559' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:45:42.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:41 smithi137 ceph-mon[39175]: osdmap e420: 8 total, 7 up, 8 in 2024-09-18T07:45:42.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:41 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/547518559' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:45:42.183 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:42.208 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:42.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:42.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:42.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:41 smithi203 ceph-mon[32237]: osdmap e420: 8 total, 7 up, 8 in 2024-09-18T07:45:42.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:41 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/547518559' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:45:42.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:42.332 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:42.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:42.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:42.370 INFO:teuthology.orchestra.run.smithi203.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-18T07:45:42.434 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:42.459 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:42.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:42.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:42.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:42.584 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:42.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:42.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:42.686 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:42.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:42.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:42.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:42.808 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:42.837 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:42.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:42.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:42.934 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:42 smithi203 ceph-mon[32237]: pgmap v778: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 36 active+undersized, 1 undersized+degraded+remapped+backfilling+peered, 1 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 95 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 11 KiB/s rd, 74 KiB/s wr, 280 op/s; 39507/245463 objects degraded (16.095%); 87 MiB/s, 1.38k objects/s recovering 2024-09-18T07:45:42.935 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:42 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:45:42.935 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:42 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/547518559' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:45:42.935 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:42 smithi203 ceph-mon[32237]: osdmap e421: 8 total, 7 up, 8 in 2024-09-18T07:45:42.935 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:42 smithi203 ceph-mon[32237]: 4.17 scrub starts 2024-09-18T07:45:42.935 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:42 smithi203 ceph-mon[32237]: 4.17 scrub ok 2024-09-18T07:45:42.935 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:42 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:42.935 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:42 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:45:42.939 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:45:42.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:42.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:42.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:42.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:43.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:43.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:43.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:43.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:42 smithi137 ceph-mon[29408]: pgmap v778: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 36 active+undersized, 1 undersized+degraded+remapped+backfilling+peered, 1 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 95 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 11 KiB/s rd, 74 KiB/s wr, 280 op/s; 39507/245463 objects degraded (16.095%); 87 MiB/s, 1.38k objects/s recovering 2024-09-18T07:45:43.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:42 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:45:43.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:42 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/547518559' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:45:43.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:42 smithi137 ceph-mon[29408]: osdmap e421: 8 total, 7 up, 8 in 2024-09-18T07:45:43.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:42 smithi137 ceph-mon[29408]: 4.17 scrub starts 2024-09-18T07:45:43.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:42 smithi137 ceph-mon[29408]: 4.17 scrub ok 2024-09-18T07:45:43.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:42 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:43.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:42 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:45:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:42 smithi137 ceph-mon[39175]: pgmap v778: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 36 active+undersized, 1 undersized+degraded+remapped+backfilling+peered, 1 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 95 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 11 KiB/s rd, 74 KiB/s wr, 280 op/s; 39507/245463 objects degraded (16.095%); 87 MiB/s, 1.38k objects/s recovering 2024-09-18T07:45:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:42 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:45:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:42 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/547518559' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:45:43.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:42 smithi137 ceph-mon[39175]: osdmap e421: 8 total, 7 up, 8 in 2024-09-18T07:45:43.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:42 smithi137 ceph-mon[39175]: 4.17 scrub starts 2024-09-18T07:45:43.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:42 smithi137 ceph-mon[39175]: 4.17 scrub ok 2024-09-18T07:45:43.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:42 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:43.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:42 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:45:43.186 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:43.214 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:43.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:43.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:43.316 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:43.343 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:43.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:43.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:43.444 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:43.468 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:43.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:43.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:43.502 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:45:43.502 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:45:43.569 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:43.596 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:43.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:43.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:43.698 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:43.720 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:43.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:43.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:43.822 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:43.844 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:43.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:43.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:43.946 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:43.972 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:43.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:43.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:44.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:44.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:44.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:44.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:44.198 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:44.222 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:44.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:44.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:44.226 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T07:45:44.324 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:44.346 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:44.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:44.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:44.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:44 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:45:44.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:44 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:45:44.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:44 smithi137 ceph-mon[29408]: 6.15 scrub starts 2024-09-18T07:45:44.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:44 smithi137 ceph-mon[29408]: 6.15 scrub ok 2024-09-18T07:45:44.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:44 smithi137 ceph-mon[29408]: osdmap e422: 8 total, 7 up, 8 in 2024-09-18T07:45:44.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:44 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:45:44.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:44 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:45:44.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:44 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:45:44.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:44 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:45:44.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:44 smithi137 ceph-mon[39175]: 6.15 scrub starts 2024-09-18T07:45:44.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:44 smithi137 ceph-mon[39175]: 6.15 scrub ok 2024-09-18T07:45:44.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:44 smithi137 ceph-mon[39175]: osdmap e422: 8 total, 7 up, 8 in 2024-09-18T07:45:44.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:44 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:45:44.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:44 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:45:44.447 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:44.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:44 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:45:44.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:44 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:45:44.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:44 smithi203 ceph-mon[32237]: 6.15 scrub starts 2024-09-18T07:45:44.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:44 smithi203 ceph-mon[32237]: 6.15 scrub ok 2024-09-18T07:45:44.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:44 smithi203 ceph-mon[32237]: osdmap e422: 8 total, 7 up, 8 in 2024-09-18T07:45:44.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:44 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:45:44.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:44 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:45:44.474 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:44.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:44.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:44.544 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:45:44.577 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:44.602 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:44.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:44.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:44.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:44.727 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:44.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:44.828 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:44.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:44.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:44.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:44.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:44.972 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:44.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:44.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:45.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:45.098 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:45.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:45.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:45.138 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:45:45.138 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:45:45.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:45 smithi137 ceph-mon[29408]: pgmap v781: 158 pgs: 36 active+undersized, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 96 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 32686/245463 objects degraded (13.316%); 126 MiB/s, 2.01k objects/s recovering 2024-09-18T07:45:45.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:45 smithi137 ceph-mon[29408]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-18T07:45:45.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:45 smithi137 ceph-mon[29408]: osdmap e423: 8 total, 7 up, 8 in 2024-09-18T07:45:45.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:45 smithi137 ceph-mon[39175]: pgmap v781: 158 pgs: 36 active+undersized, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 96 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 32686/245463 objects degraded (13.316%); 126 MiB/s, 2.01k objects/s recovering 2024-09-18T07:45:45.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:45 smithi137 ceph-mon[39175]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-18T07:45:45.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:45 smithi137 ceph-mon[39175]: osdmap e423: 8 total, 7 up, 8 in 2024-09-18T07:45:45.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:45.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:45.222 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:45.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:45.323 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:45.345 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:45.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:45.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:45.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:45 smithi203 ceph-mon[32237]: pgmap v781: 158 pgs: 36 active+undersized, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 96 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 32686/245463 objects degraded (13.316%); 126 MiB/s, 2.01k objects/s recovering 2024-09-18T07:45:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:45 smithi203 ceph-mon[32237]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-18T07:45:45.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:45 smithi203 ceph-mon[32237]: osdmap e423: 8 total, 7 up, 8 in 2024-09-18T07:45:45.468 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:45.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:45.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:45.570 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:45.595 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:45.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:45.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:45.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:45.719 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:45.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:45.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:45.820 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:45.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:45.842 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:45.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:45.874 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T07:45:45.944 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:45.965 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:45.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:45.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:46.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:46.095 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:46.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:46.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:46 smithi203 ceph-mon[32237]: 5.1e scrub starts 2024-09-18T07:45:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:46 smithi203 ceph-mon[32237]: 5.1e scrub ok 2024-09-18T07:45:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:46 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 32686/245463 objects degraded (13.316%), 26 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:46 smithi203 ceph-mon[32237]: osdmap e424: 8 total, 7 up, 8 in 2024-09-18T07:45:46.211 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:45:46.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:46.241 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:46.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:46.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:46.376 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:46.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:46.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:46.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:46 smithi137 ceph-mon[29408]: 5.1e scrub starts 2024-09-18T07:45:46.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:46 smithi137 ceph-mon[29408]: 5.1e scrub ok 2024-09-18T07:45:46.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:46 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 32686/245463 objects degraded (13.316%), 26 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:46.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:46 smithi137 ceph-mon[29408]: osdmap e424: 8 total, 7 up, 8 in 2024-09-18T07:45:46.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:46 smithi137 ceph-mon[39175]: 5.1e scrub starts 2024-09-18T07:45:46.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:46 smithi137 ceph-mon[39175]: 5.1e scrub ok 2024-09-18T07:45:46.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:46 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 32686/245463 objects degraded (13.316%), 26 pgs degraded (PG_DEGRADED) 2024-09-18T07:45:46.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:46 smithi137 ceph-mon[39175]: osdmap e424: 8 total, 7 up, 8 in 2024-09-18T07:45:46.479 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:46.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:46.512 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:46.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:46.613 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:46.645 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:46.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:46.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:46.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:46.789 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:46.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:46.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:46.835 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:45:46.835 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:45:46.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:46.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:46.924 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:46.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:47.025 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:47.063 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:47.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:47.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:47.165 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:47.198 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:47.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:47.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:47.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:47.340 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:47.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:47.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:47.394 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:47 smithi203 ceph-mon[32237]: pgmap v784: 158 pgs: 36 active+undersized, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 96 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 32686/245463 objects degraded (13.316%); 107 MiB/s, 1.70k objects/s recovering 2024-09-18T07:45:47.394 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:47 smithi203 ceph-mon[32237]: osdmap e425: 8 total, 7 up, 8 in 2024-09-18T07:45:47.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:47 smithi137 ceph-mon[29408]: pgmap v784: 158 pgs: 36 active+undersized, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 96 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 32686/245463 objects degraded (13.316%); 107 MiB/s, 1.70k objects/s recovering 2024-09-18T07:45:47.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:47 smithi137 ceph-mon[29408]: osdmap e425: 8 total, 7 up, 8 in 2024-09-18T07:45:47.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:45:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:45:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:45:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:47 smithi137 ceph-mon[39175]: pgmap v784: 158 pgs: 36 active+undersized, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 96 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 32686/245463 objects degraded (13.316%); 107 MiB/s, 1.70k objects/s recovering 2024-09-18T07:45:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:47 smithi137 ceph-mon[39175]: osdmap e425: 8 total, 7 up, 8 in 2024-09-18T07:45:47.443 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:47.479 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:47.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:47.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:47.540 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T07:45:47.581 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:47.627 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:47.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:47.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:47.759 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:47.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:47.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:47.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:47.901 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:47.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:47.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:48.003 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:48.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:48.035 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:48.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:48.095 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:45:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:45:48.136 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:48.163 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:48.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:48.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:48.265 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:48.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:48.299 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:48.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:48.400 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:48.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:48 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:45:48.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:48 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:45:48.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:48 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:45:48.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:48 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:45:48.433 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:48.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:48.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:48.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:48 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:45:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:48 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:45:48.535 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:48.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:48.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:48.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:48.673 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:48.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:48.708 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:48.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:48.809 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:48.837 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:48.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:48.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:48.876 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:48.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:48.978 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:48.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:48.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:49.080 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:49.094 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:49 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:49.095 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:49 smithi137 ceph-mon[39175]: pgmap v786: 158 pgs: 36 active+undersized, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 96 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 143 op/s; 30061/243435 objects degraded (12.349%); 31 MiB/s, 498 objects/s recovering 2024-09-18T07:45:49.095 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:49 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:49.095 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:49 smithi137 ceph-mon[29408]: pgmap v786: 158 pgs: 36 active+undersized, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 96 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 143 op/s; 30061/243435 objects degraded (12.349%); 31 MiB/s, 498 objects/s recovering 2024-09-18T07:45:49.095 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:49 smithi137 ceph-mon[29408]: osdmap e426: 8 total, 7 up, 8 in 2024-09-18T07:45:49.128 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:49.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:49.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:49.230 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:49.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:49.275 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:49.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:49.339 INFO:teuthology.orchestra.run.smithi203.stdout:6.d 2024-09-18T07:45:49.340 INFO:teuthology.orchestra.run.smithi203.stdout:5.6 2024-09-18T07:45:49.340 INFO:teuthology.orchestra.run.smithi203.stdout:3.d 2024-09-18T07:45:49.340 INFO:teuthology.orchestra.run.smithi203.stdout:5.15 2024-09-18T07:45:49.340 INFO:teuthology.orchestra.run.smithi203.stdout:5.7 2024-09-18T07:45:49.340 INFO:teuthology.orchestra.run.smithi203.stdout:4.3 2024-09-18T07:45:49.340 INFO:teuthology.orchestra.run.smithi203.stdout:3.c 2024-09-18T07:45:49.340 INFO:teuthology.orchestra.run.smithi203.stdout:3.2 2024-09-18T07:45:49.340 INFO:teuthology.orchestra.run.smithi203.stdout:3.1d 2024-09-18T07:45:49.341 INFO:teuthology.orchestra.run.smithi203.stdout:1.a 2024-09-18T07:45:49.341 INFO:teuthology.orchestra.run.smithi203.stdout:3.1f 2024-09-18T07:45:49.341 INFO:teuthology.orchestra.run.smithi203.stdout:4.b 2024-09-18T07:45:49.341 INFO:teuthology.orchestra.run.smithi203.stdout:5.3 2024-09-18T07:45:49.341 INFO:teuthology.orchestra.run.smithi203.stdout:3.1c 2024-09-18T07:45:49.341 INFO:teuthology.orchestra.run.smithi203.stdout:1.1 2024-09-18T07:45:49.341 INFO:teuthology.orchestra.run.smithi203.stdout:6.19 2024-09-18T07:45:49.341 INFO:teuthology.orchestra.run.smithi203.stdout:4.1f 2024-09-18T07:45:49.341 INFO:teuthology.orchestra.run.smithi203.stdout:5.16 2024-09-18T07:45:49.341 INFO:teuthology.orchestra.run.smithi203.stdout:3.4 2024-09-18T07:45:49.342 INFO:teuthology.orchestra.run.smithi203.stdout:10.6 2024-09-18T07:45:49.342 INFO:teuthology.orchestra.run.smithi203.stdout:6.17 2024-09-18T07:45:49.342 INFO:teuthology.orchestra.run.smithi203.stdout:10.a 2024-09-18T07:45:49.342 INFO:teuthology.orchestra.run.smithi203.stdout:4.1 2024-09-18T07:45:49.342 INFO:teuthology.orchestra.run.smithi203.stdout:1.4 2024-09-18T07:45:49.342 INFO:teuthology.orchestra.run.smithi203.stdout:5.b 2024-09-18T07:45:49.342 INFO:teuthology.orchestra.run.smithi203.stdout:4.7 2024-09-18T07:45:49.342 INFO:teuthology.orchestra.run.smithi203.stdout:4.6 2024-09-18T07:45:49.342 INFO:teuthology.orchestra.run.smithi203.stdout:5.4 2024-09-18T07:45:49.342 INFO:teuthology.orchestra.run.smithi203.stdout:10.3 2024-09-18T07:45:49.342 INFO:teuthology.orchestra.run.smithi203.stdout:10.7 2024-09-18T07:45:49.343 INFO:teuthology.orchestra.run.smithi203.stdout:4.1b 2024-09-18T07:45:49.343 INFO:teuthology.orchestra.run.smithi203.stdout:3.5 2024-09-18T07:45:49.343 INFO:teuthology.orchestra.run.smithi203.stdout:4.2 2024-09-18T07:45:49.343 INFO:teuthology.orchestra.run.smithi203.stdout:6.10 2024-09-18T07:45:49.343 INFO:teuthology.orchestra.run.smithi203.stdout:10.9 2024-09-18T07:45:49.343 INFO:teuthology.orchestra.run.smithi203.stdout:2.2 2024-09-18T07:45:49.343 INFO:teuthology.orchestra.run.smithi203.stdout:4.e 2024-09-18T07:45:49.343 INFO:teuthology.orchestra.run.smithi203.stdout:5.19 2024-09-18T07:45:49.343 INFO:teuthology.orchestra.run.smithi203.stdout:6.1d 2024-09-18T07:45:49.343 INFO:teuthology.orchestra.run.smithi203.stdout:6.a 2024-09-18T07:45:49.344 INFO:teuthology.orchestra.run.smithi203.stdout:3.10 2024-09-18T07:45:49.344 INFO:teuthology.orchestra.run.smithi203.stdout:3.17 2024-09-18T07:45:49.344 INFO:teuthology.orchestra.run.smithi203.stdout:4.8 2024-09-18T07:45:49.344 INFO:teuthology.orchestra.run.smithi203.stdout:3.16 2024-09-18T07:45:49.344 INFO:teuthology.orchestra.run.smithi203.stdout:5.f 2024-09-18T07:45:49.344 INFO:teuthology.orchestra.run.smithi203.stdout:6.1e 2024-09-18T07:45:49.344 INFO:teuthology.orchestra.run.smithi203.stdout:6.4 2024-09-18T07:45:49.344 INFO:teuthology.orchestra.run.smithi203.stdout:4.15 2024-09-18T07:45:49.344 INFO:teuthology.orchestra.run.smithi203.stdout:5.1b 2024-09-18T07:45:49.344 INFO:teuthology.orchestra.run.smithi203.stdout:5.2 2024-09-18T07:45:49.345 INFO:teuthology.orchestra.run.smithi203.stdout:1.8 2024-09-18T07:45:49.345 INFO:teuthology.orchestra.run.smithi203.stdout:5.d 2024-09-18T07:45:49.345 INFO:teuthology.orchestra.run.smithi203.stdout:5.e 2024-09-18T07:45:49.345 INFO:teuthology.orchestra.run.smithi203.stdout:6.11 2024-09-18T07:45:49.345 INFO:teuthology.orchestra.run.smithi203.stdout:6.1a 2024-09-18T07:45:49.345 INFO:teuthology.orchestra.run.smithi203.stdout:6.c 2024-09-18T07:45:49.345 INFO:teuthology.orchestra.run.smithi203.stdout:6.1f 2024-09-18T07:45:49.345 INFO:teuthology.orchestra.run.smithi203.stdout:6.1c 2024-09-18T07:45:49.345 INFO:teuthology.orchestra.run.smithi203.stdout:5.12 2024-09-18T07:45:49.345 INFO:teuthology.orchestra.run.smithi203.stdout:6.7 2024-09-18T07:45:49.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:49.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:49.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:49.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:49.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:49 smithi137 ceph-mon[39175]: osdmap e426: 8 total, 7 up, 8 in 2024-09-18T07:45:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:49 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:49 smithi203 ceph-mon[32237]: pgmap v786: 158 pgs: 36 active+undersized, 1 undersized+degraded+remapped+backfilling+peered, 2 active+undersized+degraded+remapped+backfilling, 23 active+undersized+degraded, 96 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 143 op/s; 30061/243435 objects degraded (12.349%); 31 MiB/s, 498 objects/s recovering 2024-09-18T07:45:49.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:49 smithi203 ceph-mon[32237]: osdmap e426: 8 total, 7 up, 8 in 2024-09-18T07:45:49.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:49.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:49.559 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:49.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:49.660 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:49.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:49.700 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:49.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:49.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:49.836 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:49.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:49.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:49.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:49.974 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:49.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:49.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:50.076 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:50.117 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:50.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:50.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:50.219 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:50.247 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:50.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:50.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:50.258 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:50 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 30061/243435 objects degraded (12.349%), 26 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-18T07:45:50.258 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:50 smithi137 ceph-mon[29408]: osdmap e427: 8 total, 7 up, 8 in 2024-09-18T07:45:50.259 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:50 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 30061/243435 objects degraded (12.349%), 26 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-18T07:45:50.259 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:50 smithi137 ceph-mon[39175]: osdmap e427: 8 total, 7 up, 8 in 2024-09-18T07:45:50.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:50.372 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:50.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:50.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:50.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:50 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 30061/243435 objects degraded (12.349%), 26 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-18T07:45:50.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:50 smithi203 ceph-mon[32237]: osdmap e427: 8 total, 7 up, 8 in 2024-09-18T07:45:50.473 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:50.495 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:50.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:50.549 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:45:50.549 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:45:50.562 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:45:50.562 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:45:50.578 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:45:50.578 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:45:50.594 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:45:50.595 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:45:50.597 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:50.611 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:45:50.611 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:45:50.611 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-18T07:45:50.619 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:50.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:50.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:50.627 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:50.631 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:45:50.631 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:45:50.649 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:45:50.649 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:45:50.659 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -n osd.5 --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 4.1f --file /var/log/ceph/exp.4.1f.5 2024-09-18T07:45:50.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:50.741 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:50.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:50.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:50.843 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:50.852 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:45:50.867 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:50.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:50.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:50.903 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/osd.5/config 2024-09-18T07:45:50.969 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:50.990 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:50.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:50.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:51.092 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:51.119 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:51.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:51.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:51.134 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:51 smithi137 ceph-mon[29408]: pgmap v788: 158 pgs: 36 active+undersized, 2 active+undersized+degraded+remapped+backfilling, 24 active+undersized+degraded, 96 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 892 B/s rd, 0 B/s wr, 219 op/s; 27320/241701 objects degraded (11.303%); 53 MiB/s, 855 objects/s recovering 2024-09-18T07:45:51.134 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:51 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1384604287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:51.134 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:51 smithi137 ceph-mon[29408]: osdmap e428: 8 total, 7 up, 8 in 2024-09-18T07:45:51.134 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:51 smithi137 ceph-mon[39175]: pgmap v788: 158 pgs: 36 active+undersized, 2 active+undersized+degraded+remapped+backfilling, 24 active+undersized+degraded, 96 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 892 B/s rd, 0 B/s wr, 219 op/s; 27320/241701 objects degraded (11.303%); 53 MiB/s, 855 objects/s recovering 2024-09-18T07:45:51.134 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:51 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1384604287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:51.135 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:51 smithi137 ceph-mon[39175]: osdmap e428: 8 total, 7 up, 8 in 2024-09-18T07:45:51.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:51.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:51.260 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:51.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:51.335 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:51 smithi203 ceph-mon[32237]: pgmap v788: 158 pgs: 36 active+undersized, 2 active+undersized+degraded+remapped+backfilling, 24 active+undersized+degraded, 96 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 892 B/s rd, 0 B/s wr, 219 op/s; 27320/241701 objects degraded (11.303%); 53 MiB/s, 855 objects/s recovering 2024-09-18T07:45:51.335 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:51 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1384604287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:51.336 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:51 smithi203 ceph-mon[32237]: osdmap e428: 8 total, 7 up, 8 in 2024-09-18T07:45:51.360 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:51.392 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:51.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:51.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:51.465 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:45:51.465 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:45:51.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.466 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:195496 10.6 10:60461ba7:::benchmark_data_smithi137_151160_object59680:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-09-18T07:45:51.466 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:51.451218+0000", 2024-09-18T07:45:51.466 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0070527130000000004, 2024-09-18T07:45:51.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.007080907, 2024-09-18T07:45:51.466 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:51.466 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:51.466 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:45:51.466 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:51.466 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:51.466 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 195496 2024-09-18T07:45:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451218+0000", 2024-09-18T07:45:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451218+0000", 2024-09-18T07:45:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451219+0000", 2024-09-18T07:45:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1099999999999995e-07 2024-09-18T07:45:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451222+0000", 2024-09-18T07:45:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9210000000000001e-06 2024-09-18T07:45:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451228+0000", 2024-09-18T07:45:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4949999999999999e-06 2024-09-18T07:45:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451233+0000", 2024-09-18T07:45:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3029999999999999e-06 2024-09-18T07:45:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.457818+0000", 2024-09-18T07:45:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065847579999999996 2024-09-18T07:45:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:51.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.458096+0000", 2024-09-18T07:45:51.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027843799999999999 2024-09-18T07:45:51.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.470 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4", 2024-09-18T07:45:51.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.458220+0000", 2024-09-18T07:45:51.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000123824 2024-09-18T07:45:51.470 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:51.470 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:51.470 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:195484 10.5 e428/364 10:a241fe30:::benchmark_data_smithi137_151160_object59668:head v 428'12174, mlcod=428'12171)", 2024-09-18T07:45:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:51.455263+0000", 2024-09-18T07:45:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.003008007, 2024-09-18T07:45:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031031729999999999, 2024-09-18T07:45:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:45:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:51.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.455263+0000", 2024-09-18T07:45:51.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.455263+0000", 2024-09-18T07:45:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.455264+0000", 2024-09-18T07:45:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.023e-06 2024-09-18T07:45:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.455270+0000", 2024-09-18T07:45:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2389999999999999e-06 2024-09-18T07:45:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.455280+0000", 2024-09-18T07:45:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3319999999999998e-06 2024-09-18T07:45:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.455286+0000", 2024-09-18T07:45:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5629999999999997e-06 2024-09-18T07:45:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.455428+0000", 2024-09-18T07:45:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014180200000000001 2024-09-18T07:45:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.455446+0000", 2024-09-18T07:45:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7621999999999998e-05 2024-09-18T07:45:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:195486 10.5 e428/364)", 2024-09-18T07:45:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:51.457076+0000", 2024-09-18T07:45:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0011948519999999999, 2024-09-18T07:45:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001336779, 2024-09-18T07:45:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:45:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.457076+0000", 2024-09-18T07:45:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.457076+0000", 2024-09-18T07:45:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.457078+0000", 2024-09-18T07:45:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3170000000000001e-06 2024-09-18T07:45:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.457084+0000", 2024-09-18T07:45:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6510000000000004e-06 2024-09-18T07:45:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.457096+0000", 2024-09-18T07:45:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2001000000000001e-05 2024-09-18T07:45:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.457103+0000", 2024-09-18T07:45:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1169999999999998e-06 2024-09-18T07:45:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.36014.0:195494 10.6 e428/424)", 2024-09-18T07:45:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:51.458029+0000", 2024-09-18T07:45:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00024185600000000001, 2024-09-18T07:45:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00042351900000000002, 2024-09-18T07:45:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:45:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.458029+0000", 2024-09-18T07:45:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.458029+0000", 2024-09-18T07:45:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.458030+0000", 2024-09-18T07:45:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0249999999999999e-06 2024-09-18T07:45:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.458034+0000", 2024-09-18T07:45:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0389999999999998e-06 2024-09-18T07:45:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.458042+0000", 2024-09-18T07:45:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4649999999999997e-06 2024-09-18T07:45:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.458048+0000", 2024-09-18T07:45:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8239999999999996e-06 2024-09-18T07:45:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.36014.0:195495 10.6 e428/424)", 2024-09-18T07:45:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:51.458177+0000", 2024-09-18T07:45:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 9.3962999999999995e-05, 2024-09-18T07:45:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00031504299999999999, 2024-09-18T07:45:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:45:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.458177+0000", 2024-09-18T07:45:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.458177+0000", 2024-09-18T07:45:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:51.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.458178+0000", 2024-09-18T07:45:51.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1399999999999996e-07 2024-09-18T07:45:51.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:51.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.458181+0000", 2024-09-18T07:45:51.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0740000000000001e-06 2024-09-18T07:45:51.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.458187+0000", 2024-09-18T07:45:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.443e-06 2024-09-18T07:45:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.458193+0000", 2024-09-18T07:45:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5480000000000001e-06 2024-09-18T07:45:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:195490 10.2 e428/415 10:4f15cd0f:::benchmark_data_smithi137_151160_object59674:head v 428'12264, mlcod=428'12262)", 2024-09-18T07:45:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:51.447067+0000", 2024-09-18T07:45:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.011203967, 2024-09-18T07:45:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011720055, 2024-09-18T07:45:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:45:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.447067+0000", 2024-09-18T07:45:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.447067+0000", 2024-09-18T07:45:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.447068+0000", 2024-09-18T07:45:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.198e-06 2024-09-18T07:45:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:51.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.447075+0000", 2024-09-18T07:45:51.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5629999999999997e-06 2024-09-18T07:45:51.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:51.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.447084+0000", 2024-09-18T07:45:51.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4860000000000003e-06 2024-09-18T07:45:51.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.447092+0000", 2024-09-18T07:45:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2030000000000003e-06 2024-09-18T07:45:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.455223+0000", 2024-09-18T07:45:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081317850000000007 2024-09-18T07:45:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.455241+0000", 2024-09-18T07:45:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7399000000000001e-05 2024-09-18T07:45:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.458019+0000", 2024-09-18T07:45:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027778299999999998 2024-09-18T07:45:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:195494 10.6 10:613d23a0:::benchmark_data_smithi137_151160_object59678:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-09-18T07:45:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:51.451157+0000", 2024-09-18T07:45:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0071143739999999997, 2024-09-18T07:45:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076851580000000001, 2024-09-18T07:45:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:45:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 195494 2024-09-18T07:45:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451157+0000", 2024-09-18T07:45:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451157+0000", 2024-09-18T07:45:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451157+0000", 2024-09-18T07:45:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4499999999999996e-07 2024-09-18T07:45:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451161+0000", 2024-09-18T07:45:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8290000000000001e-06 2024-09-18T07:45:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451168+0000", 2024-09-18T07:45:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3899999999999998e-06 2024-09-18T07:45:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451174+0000", 2024-09-18T07:45:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4019999999999999e-06 2024-09-18T07:45:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451220+0000", 2024-09-18T07:45:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5472000000000001e-05 2024-09-18T07:45:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451333+0000", 2024-09-18T07:45:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011340099999999999 2024-09-18T07:45:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4", 2024-09-18T07:45:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451449+0000", 2024-09-18T07:45:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011580499999999999 2024-09-18T07:45:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:195495 10.6 10:6b162e36:::benchmark_data_smithi137_151160_object59679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-09-18T07:45:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:51.451187+0000", 2024-09-18T07:45:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0070841649999999999, 2024-09-18T07:45:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0077179379999999997, 2024-09-18T07:45:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:45:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 195495 2024-09-18T07:45:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451187+0000", 2024-09-18T07:45:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451187+0000", 2024-09-18T07:45:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451188+0000", 2024-09-18T07:45:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1999999999999999e-07 2024-09-18T07:45:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451190+0000", 2024-09-18T07:45:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.655e-06 2024-09-18T07:45:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451196+0000", 2024-09-18T07:45:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3079999999999998e-06 2024-09-18T07:45:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.451201+0000", 2024-09-18T07:45:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0379999999999999e-06 2024-09-18T07:45:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.454244+0000", 2024-09-18T07:45:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030430129999999998 2024-09-18T07:45:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.454496+0000", 2024-09-18T07:45:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00025200899999999998 2024-09-18T07:45:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4", 2024-09-18T07:45:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.454623+0000", 2024-09-18T07:45:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012761900000000001 2024-09-18T07:45:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:51.500 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:45:51.500 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 8 2024-09-18T07:45:51.500 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:45:51.500 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:51.501 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:51.528 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:51.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:51.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:51.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:51.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:51.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:51.769 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:51.798 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:51.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:51.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:51.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:51.943 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:51.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:52.044 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:52.074 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:52.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:52.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:52.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:52.189 INFO:teuthology.orchestra.run.smithi203.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-18T07:45:52.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:52.216 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:52.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:52.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:52.329 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:52 smithi137 ceph-mon[29408]: 3.f scrub starts 2024-09-18T07:45:52.330 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:52 smithi137 ceph-mon[29408]: 3.f scrub ok 2024-09-18T07:45:52.330 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:52 smithi137 ceph-mon[29408]: 1.6 scrub starts 2024-09-18T07:45:52.330 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:52 smithi137 ceph-mon[29408]: 1.6 scrub ok 2024-09-18T07:45:52.330 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:52 smithi137 ceph-mon[39175]: 3.f scrub starts 2024-09-18T07:45:52.330 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:52 smithi137 ceph-mon[39175]: 3.f scrub ok 2024-09-18T07:45:52.330 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:52 smithi137 ceph-mon[39175]: 1.6 scrub starts 2024-09-18T07:45:52.330 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:52 smithi137 ceph-mon[39175]: 1.6 scrub ok 2024-09-18T07:45:52.357 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:52.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:52.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:52.367 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:52 smithi203 ceph-mon[32237]: 3.f scrub starts 2024-09-18T07:45:52.367 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:52 smithi203 ceph-mon[32237]: 3.f scrub ok 2024-09-18T07:45:52.368 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:52 smithi203 ceph-mon[32237]: 1.6 scrub starts 2024-09-18T07:45:52.368 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:52 smithi203 ceph-mon[32237]: 1.6 scrub ok 2024-09-18T07:45:52.386 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:45:52.386 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:45:52.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:52.495 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:52.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:52.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:52.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:52.632 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:52.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:52.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:52.736 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:52.769 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:52.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:52.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:52.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T07:45:52.871 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:52.897 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:52.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:52.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:52.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:53.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:53.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:53.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:53.138 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:53.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:53.170 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:53.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:53.272 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:53.322 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:53.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:53.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:53.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:53 smithi137 ceph-mon[29408]: 3.11 scrub starts 2024-09-18T07:45:53.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:53 smithi137 ceph-mon[29408]: 3.11 scrub ok 2024-09-18T07:45:53.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:53 smithi137 ceph-mon[29408]: pgmap v791: 158 pgs: 36 active+undersized, 2 active+undersized+degraded+remapped+backfilling, 24 active+undersized+degraded, 96 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 395 op/s; 27320/238722 objects degraded (11.444%); 54 MiB/s, 860 objects/s recovering 2024-09-18T07:45:53.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:53 smithi137 ceph-mon[29408]: 6.6 scrub starts 2024-09-18T07:45:53.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:53 smithi137 ceph-mon[29408]: 6.6 scrub ok 2024-09-18T07:45:53.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:53 smithi137 ceph-mon[39175]: 3.11 scrub starts 2024-09-18T07:45:53.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:53 smithi137 ceph-mon[39175]: 3.11 scrub ok 2024-09-18T07:45:53.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:53 smithi137 ceph-mon[39175]: pgmap v791: 158 pgs: 36 active+undersized, 2 active+undersized+degraded+remapped+backfilling, 24 active+undersized+degraded, 96 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 395 op/s; 27320/238722 objects degraded (11.444%); 54 MiB/s, 860 objects/s recovering 2024-09-18T07:45:53.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:53 smithi137 ceph-mon[39175]: 6.6 scrub starts 2024-09-18T07:45:53.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:53 smithi137 ceph-mon[39175]: 6.6 scrub ok 2024-09-18T07:45:53.424 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:53.455 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:53.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:53.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:53.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:53 smithi203 ceph-mon[32237]: 3.11 scrub starts 2024-09-18T07:45:53.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:53 smithi203 ceph-mon[32237]: 3.11 scrub ok 2024-09-18T07:45:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:53 smithi203 ceph-mon[32237]: pgmap v791: 158 pgs: 36 active+undersized, 2 active+undersized+degraded+remapped+backfilling, 24 active+undersized+degraded, 96 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 395 op/s; 27320/238722 objects degraded (11.444%); 54 MiB/s, 860 objects/s recovering 2024-09-18T07:45:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:53 smithi203 ceph-mon[32237]: 6.6 scrub starts 2024-09-18T07:45:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:53 smithi203 ceph-mon[32237]: 6.6 scrub ok 2024-09-18T07:45:53.553 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:53.557 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:53.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:53.603 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:53.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:53.704 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:53.736 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:53.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:53.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:53.838 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:53.868 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:53.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:53.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:53.970 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:53.999 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:53.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:54.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:54.100 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:54.128 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:54.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:54.138 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:45:54.197 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:45:54.198 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:45:54.198 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:45:54.198 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:45:54.198 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:45:54.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:54.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:54.259 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:54.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:54.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:54.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:45:54.393 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:54.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:54.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:54.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:54 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3819140461' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:45:54.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:54 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:45:54.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:54 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:45:54.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:54 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3819140461' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:45:54.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:54 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:45:54.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:54 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:45:54.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:54 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3819140461' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:45:54.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:54 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:45:54.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:54 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:45:54.494 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:54.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:54.525 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:54.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:54.627 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:54.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:54.664 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:54.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:54.765 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:54.801 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:54.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:54.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:54.903 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:54.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:54.933 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:54.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:55.035 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:55.073 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:55.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:55.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:55.175 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:55.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:55.209 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:55.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:55.310 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:55.332 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:55.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:55.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:55.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[29408]: pgmap v792: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 24 active+undersized+degraded, 97 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 963 op/s; 23843/226101 objects degraded (10.545%); 45 MiB/s, 719 objects/s recovering 2024-09-18T07:45:55.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[29408]: 4.1c scrub starts 2024-09-18T07:45:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[29408]: 4.1c scrub ok 2024-09-18T07:45:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[29408]: 5.1f scrub starts 2024-09-18T07:45:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[29408]: 5.1f scrub ok 2024-09-18T07:45:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3819140461' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:45:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[29408]: osdmap e429: 8 total, 7 up, 8 in 2024-09-18T07:45:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 23843/226101 objects degraded (10.545%), 25 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-18T07:45:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[29408]: osdmap e430: 8 total, 7 up, 8 in 2024-09-18T07:45:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[39175]: pgmap v792: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 24 active+undersized+degraded, 97 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 963 op/s; 23843/226101 objects degraded (10.545%); 45 MiB/s, 719 objects/s recovering 2024-09-18T07:45:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[39175]: 4.1c scrub starts 2024-09-18T07:45:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[39175]: 4.1c scrub ok 2024-09-18T07:45:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[39175]: 5.1f scrub starts 2024-09-18T07:45:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[39175]: 5.1f scrub ok 2024-09-18T07:45:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3819140461' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:45:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[39175]: osdmap e429: 8 total, 7 up, 8 in 2024-09-18T07:45:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 23843/226101 objects degraded (10.545%), 25 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-18T07:45:55.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:55 smithi137 ceph-mon[39175]: osdmap e430: 8 total, 7 up, 8 in 2024-09-18T07:45:55.434 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:55.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:55.456 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:55.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:55.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:55 smithi203 ceph-mon[32237]: pgmap v792: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 24 active+undersized+degraded, 97 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 963 op/s; 23843/226101 objects degraded (10.545%); 45 MiB/s, 719 objects/s recovering 2024-09-18T07:45:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:55 smithi203 ceph-mon[32237]: 4.1c scrub starts 2024-09-18T07:45:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:55 smithi203 ceph-mon[32237]: 4.1c scrub ok 2024-09-18T07:45:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:55 smithi203 ceph-mon[32237]: 5.1f scrub starts 2024-09-18T07:45:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:55 smithi203 ceph-mon[32237]: 5.1f scrub ok 2024-09-18T07:45:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:55 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:45:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:55 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3819140461' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:45:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:55 smithi203 ceph-mon[32237]: osdmap e429: 8 total, 7 up, 8 in 2024-09-18T07:45:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:55 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 23843/226101 objects degraded (10.545%), 25 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-18T07:45:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:55 smithi203 ceph-mon[32237]: osdmap e430: 8 total, 7 up, 8 in 2024-09-18T07:45:55.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T07:45:55.557 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:55.578 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:55.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:55.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:55.680 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:55.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:55.701 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:55.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:55.802 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:55.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:55.824 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:55.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:55.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:55.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:55.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:55.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:56.019 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:56.020 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:56.049 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:56.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:56.073 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:56.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:56.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:56.181 INFO:teuthology.orchestra.run.smithi203.stderr:Exporting 4.1f info 4.1f( v 323'19 (0'0,323'19] local-lis/les=238/239 n=5 ec=62/62 lis/c=238/238 les/c/f=239/239/0 sis=238) 2024-09-18T07:45:56.184 INFO:teuthology.orchestra.run.smithi203.stderr:Read #4:f93251ce:::obj_delete_at_hint.0000000042:head# 2024-09-18T07:45:56.184 INFO:teuthology.orchestra.run.smithi203.stderr:Read #4:fce5a501:::obj_delete_at_hint.0000000075:head# 2024-09-18T07:45:56.184 INFO:teuthology.orchestra.run.smithi203.stderr:Read #4:fd6b5df8:::obj_delete_at_hint.0000000120:head# 2024-09-18T07:45:56.184 INFO:teuthology.orchestra.run.smithi203.stderr:Read #4:fd8d39b6:::obj_delete_at_hint.0000000030:head# 2024-09-18T07:45:56.184 INFO:teuthology.orchestra.run.smithi203.stderr:Read #4:fde4dd55:gc::gc.26:head# 2024-09-18T07:45:56.184 INFO:teuthology.orchestra.run.smithi203.stderr:Export successful 2024-09-18T07:45:56.215 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:56.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:56.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:56.317 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:56.346 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:56.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:56.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:56.448 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:56.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:56.479 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:56.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:56.580 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:56.614 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:56.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:56.715 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:56.747 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:56.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:56.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:56.805 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:45:56.805 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:45:56.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:45:56.805 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:45:56.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.805 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109534 8.6 8:67e394d3:::benchmark_data_smithi137_101942_object4724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-18T07:45:56.805 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.048516+0000", 2024-09-18T07:45:56.805 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 494.751605654, 2024-09-18T07:45:56.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.182592823, 2024-09-18T07:45:56.806 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.806 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.806 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:56.806 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.806 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:45:56.806 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:45:56.806 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109534 2024-09-18T07:45:56.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.806 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048516+0000", 2024-09-18T07:45:56.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048516+0000", 2024-09-18T07:45:56.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048517+0000", 2024-09-18T07:45:56.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2e-07 2024-09-18T07:45:56.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048519+0000", 2024-09-18T07:45:56.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.232e-06 2024-09-18T07:45:56.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048523+0000", 2024-09-18T07:45:56.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4479999999999996e-06 2024-09-18T07:45:56.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048528+0000", 2024-09-18T07:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2169999999999996e-06 2024-09-18T07:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065112+0000", 2024-09-18T07:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016583912999999999 2024-09-18T07:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065121+0000", 2024-09-18T07:45:56.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8640000000000008e-06 2024-09-18T07:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.219070+0000", 2024-09-18T07:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1539486139999999 2024-09-18T07:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.219228+0000", 2024-09-18T07:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000157751 2024-09-18T07:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-09-18T07:45:56.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.219376+0000", 2024-09-18T07:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014781899999999999 2024-09-18T07:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226242+0000", 2024-09-18T07:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068661970000000001 2024-09-18T07:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226253+0000", 2024-09-18T07:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0929e-05 2024-09-18T07:45:56.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229198+0000", 2024-09-18T07:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029453790000000001 2024-09-18T07:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229278+0000", 2024-09-18T07:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9431000000000006e-05 2024-09-18T07:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.231109+0000", 2024-09-18T07:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018316090000000001 2024-09-18T07:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109539 8.6 8:66837289:::benchmark_data_smithi137_101942_object4729:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.065140+0000", 2024-09-18T07:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 494.73498218499998, 2024-09-18T07:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1676551340000001, 2024-09-18T07:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.813 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:56.814 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.814 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:45:56.814 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:45:56.814 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109539 2024-09-18T07:45:56.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.814 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065140+0000", 2024-09-18T07:45:56.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065140+0000", 2024-09-18T07:45:56.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065142+0000", 2024-09-18T07:45:56.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.548e-06 2024-09-18T07:45:56.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065149+0000", 2024-09-18T07:45:56.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7989999999999996e-06 2024-09-18T07:45:56.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065162+0000", 2024-09-18T07:45:56.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3027000000000001e-05 2024-09-18T07:45:56.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065171+0000", 2024-09-18T07:45:56.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1730000000000001e-06 2024-09-18T07:45:56.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065215+0000", 2024-09-18T07:45:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3679e-05 2024-09-18T07:45:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065222+0000", 2024-09-18T07:45:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9879999999999998e-06 2024-09-18T07:45:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.223751+0000", 2024-09-18T07:45:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1585293800000001 2024-09-18T07:45:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.223906+0000", 2024-09-18T07:45:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015412600000000001 2024-09-18T07:45:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-09-18T07:45:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.224068+0000", 2024-09-18T07:45:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016266499999999999 2024-09-18T07:45:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:56.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226399+0000", 2024-09-18T07:45:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002330557 2024-09-18T07:45:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226549+0000", 2024-09-18T07:45:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000150693 2024-09-18T07:45:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.231185+0000", 2024-09-18T07:45:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004635775 2024-09-18T07:45:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:56.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.231241+0000", 2024-09-18T07:45:56.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6001999999999997e-05 2024-09-18T07:45:56.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.232795+0000", 2024-09-18T07:45:56.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001553722 2024-09-18T07:45:56.825 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.825 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.826 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.826 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137615 10.4 10:280114ba:::benchmark_data_smithi137_151160_object1799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-18T07:45:56.826 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:42.771430+0000", 2024-09-18T07:45:56.826 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 74.028691555999998, 2024-09-18T07:45:56.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.155652747, 2024-09-18T07:45:56.826 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.826 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.826 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:45:56.826 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.826 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:56.826 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:56.827 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137615 2024-09-18T07:45:56.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.827 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.827 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771430+0000", 2024-09-18T07:45:56.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.827 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771430+0000", 2024-09-18T07:45:56.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771432+0000", 2024-09-18T07:45:56.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6199999999999999e-06 2024-09-18T07:45:56.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771437+0000", 2024-09-18T07:45:56.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0159999999999999e-06 2024-09-18T07:45:56.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771445+0000", 2024-09-18T07:45:56.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.225e-06 2024-09-18T07:45:56.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771451+0000", 2024-09-18T07:45:56.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.908e-06 2024-09-18T07:45:56.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.772166+0000", 2024-09-18T07:45:56.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000714571 2024-09-18T07:45:56.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:56.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.772292+0000", 2024-09-18T07:45:56.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126322 2024-09-18T07:45:56.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-09-18T07:45:56.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.772478+0000", 2024-09-18T07:45:56.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000185747 2024-09-18T07:45:56.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:56.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.784266+0000", 2024-09-18T07:45:56.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011787824000000001 2024-09-18T07:45:56.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.795223+0000", 2024-09-18T07:45:56.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010956802999999999 2024-09-18T07:45:56.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:56.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.795236+0000", 2024-09-18T07:45:56.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3570999999999999e-05 2024-09-18T07:45:56.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774402+0000", 2024-09-18T07:45:56.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97916638700000003 2024-09-18T07:45:56.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:56.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774413+0000", 2024-09-18T07:45:56.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0035e-05 2024-09-18T07:45:56.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927028+0000", 2024-09-18T07:45:56.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.152615967 2024-09-18T07:45:56.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927083+0000", 2024-09-18T07:45:56.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4750999999999998e-05 2024-09-18T07:45:56.833 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.833 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.833 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.833 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137618 10.4 10:22103fc7:::benchmark_data_smithi137_151160_object1802:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-18T07:45:56.833 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:42.773989+0000", 2024-09-18T07:45:56.833 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 74.026133107000007, 2024-09-18T07:45:56.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1739552340000001, 2024-09-18T07:45:56.834 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.834 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.834 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:45:56.834 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.834 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:56.834 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:56.834 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137618 2024-09-18T07:45:56.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.834 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.773989+0000", 2024-09-18T07:45:56.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.773989+0000", 2024-09-18T07:45:56.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.773991+0000", 2024-09-18T07:45:56.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.657e-06 2024-09-18T07:45:56.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.773997+0000", 2024-09-18T07:45:56.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1789999999999996e-06 2024-09-18T07:45:56.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.774008+0000", 2024-09-18T07:45:56.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0937999999999999e-05 2024-09-18T07:45:56.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.774015+0000", 2024-09-18T07:45:56.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7789999999999996e-06 2024-09-18T07:45:56.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.779798+0000", 2024-09-18T07:45:56.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057823980000000002 2024-09-18T07:45:56.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:56.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.779927+0000", 2024-09-18T07:45:56.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012920100000000001 2024-09-18T07:45:56.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-09-18T07:45:56.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.780075+0000", 2024-09-18T07:45:56.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014824700000000001 2024-09-18T07:45:56.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:56.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.784857+0000", 2024-09-18T07:45:56.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047816070000000002 2024-09-18T07:45:56.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.834874+0000", 2024-09-18T07:45:56.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.050016892 2024-09-18T07:45:56.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:56.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.834885+0000", 2024-09-18T07:45:56.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1080999999999999e-05 2024-09-18T07:45:56.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774441+0000", 2024-09-18T07:45:56.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93955602400000005 2024-09-18T07:45:56.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:56.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774685+0000", 2024-09-18T07:45:56.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024376000000000001 2024-09-18T07:45:56.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927339+0000", 2024-09-18T07:45:56.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1526545050000001 2024-09-18T07:45:56.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-09-18T07:45:56.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927499+0000", 2024-09-18T07:45:56.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016008799999999999 2024-09-18T07:45:56.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.947852+0000", 2024-09-18T07:45:56.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020352456000000001 2024-09-18T07:45:56.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.947944+0000", 2024-09-18T07:45:56.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2422000000000004e-05 2024-09-18T07:45:56.842 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.842 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.843 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.843 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137629 10.4 10:238d70eb:::benchmark_data_smithi137_151160_object1813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-09-18T07:45:56.843 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:42.835110+0000", 2024-09-18T07:45:56.843 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 73.965011532000005, 2024-09-18T07:45:56.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.113021496, 2024-09-18T07:45:56.843 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.843 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.843 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:56.843 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.843 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:56.844 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:56.844 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137629 2024-09-18T07:45:56.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.844 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835110+0000", 2024-09-18T07:45:56.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835110+0000", 2024-09-18T07:45:56.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835113+0000", 2024-09-18T07:45:56.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1150000000000001e-06 2024-09-18T07:45:56.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835117+0000", 2024-09-18T07:45:56.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2359999999999998e-06 2024-09-18T07:45:56.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835124+0000", 2024-09-18T07:45:56.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3629999999999996e-06 2024-09-18T07:45:56.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835131+0000", 2024-09-18T07:45:56.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5899999999999996e-06 2024-09-18T07:45:56.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835166+0000", 2024-09-18T07:45:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5054e-05 2024-09-18T07:45:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835173+0000", 2024-09-18T07:45:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9759999999999998e-06 2024-09-18T07:45:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774713+0000", 2024-09-18T07:45:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93954058799999995 2024-09-18T07:45:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774722+0000", 2024-09-18T07:45:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5320000000000006e-06 2024-09-18T07:45:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927611+0000", 2024-09-18T07:45:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.152888763 2024-09-18T07:45:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927731+0000", 2024-09-18T07:45:56.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012040999999999999 2024-09-18T07:45:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:45:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927854+0000", 2024-09-18T07:45:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012266800000000001 2024-09-18T07:45:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.934460+0000", 2024-09-18T07:45:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066062439999999998 2024-09-18T07:45:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.947486+0000", 2024-09-18T07:45:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013025956999999999 2024-09-18T07:45:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.947988+0000", 2024-09-18T07:45:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050220899999999999 2024-09-18T07:45:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.948040+0000", 2024-09-18T07:45:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1462000000000003e-05 2024-09-18T07:45:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.948132+0000", 2024-09-18T07:45:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2329000000000003e-05 2024-09-18T07:45:56.851 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.852 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.852 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.852 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137617 10.1 10:8638502c:::benchmark_data_smithi137_151160_object1801:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e396)", 2024-09-18T07:45:56.852 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:43.767316+0000", 2024-09-18T07:45:56.852 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 73.032805534000005, 2024-09-18T07:45:56.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1348433360000001, 2024-09-18T07:45:56.852 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.852 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.852 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:45:56.853 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.853 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:56.853 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:56.853 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137617 2024-09-18T07:45:56.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.853 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.767316+0000", 2024-09-18T07:45:56.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.767316+0000", 2024-09-18T07:45:56.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.767318+0000", 2024-09-18T07:45:56.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.889e-06 2024-09-18T07:45:56.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.767324+0000", 2024-09-18T07:45:56.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8429999999999998e-06 2024-09-18T07:45:56.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.768695+0000", 2024-09-18T07:45:56.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001370549 2024-09-18T07:45:56.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.768709+0000", 2024-09-18T07:45:56.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.434e-05 2024-09-18T07:45:56.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.772764+0000", 2024-09-18T07:45:56.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040549729999999999 2024-09-18T07:45:56.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:56.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.772774+0000", 2024-09-18T07:45:56.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4499999999999993e-06 2024-09-18T07:45:56.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.901011+0000", 2024-09-18T07:45:56.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.128237484 2024-09-18T07:45:56.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.902160+0000", 2024-09-18T07:45:56.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011488080000000001 2024-09-18T07:45:56.857 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.857 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.857 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.857 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137632 10.1 10:839066b5:::benchmark_data_smithi137_151160_object1816:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e396)", 2024-09-18T07:45:56.857 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:43.769783+0000", 2024-09-18T07:45:56.857 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 73.030339507999997, 2024-09-18T07:45:56.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1401627699999999, 2024-09-18T07:45:56.858 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.858 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.858 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:56.858 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.858 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:56.858 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:56.858 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137632 2024-09-18T07:45:56.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.858 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769783+0000", 2024-09-18T07:45:56.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769783+0000", 2024-09-18T07:45:56.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769784+0000", 2024-09-18T07:45:56.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7439999999999999e-06 2024-09-18T07:45:56.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769791+0000", 2024-09-18T07:45:56.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6869999999999997e-06 2024-09-18T07:45:56.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769800+0000", 2024-09-18T07:45:56.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0129999999999999e-06 2024-09-18T07:45:56.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769808+0000", 2024-09-18T07:45:56.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7060000000000004e-06 2024-09-18T07:45:56.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.772796+0000", 2024-09-18T07:45:56.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002987899 2024-09-18T07:45:56.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:56.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.772802+0000", 2024-09-18T07:45:56.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9909999999999999e-06 2024-09-18T07:45:56.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.902389+0000", 2024-09-18T07:45:56.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.129587747 2024-09-18T07:45:56.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:56.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.902738+0000", 2024-09-18T07:45:56.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034822599999999998 2024-09-18T07:45:56.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4,7", 2024-09-18T07:45:56.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.902816+0000", 2024-09-18T07:45:56.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8430000000000006e-05 2024-09-18T07:45:56.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.904850+0000", 2024-09-18T07:45:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020342960000000001 2024-09-18T07:45:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.906005+0000", 2024-09-18T07:45:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011550899999999999 2024-09-18T07:45:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.909791+0000", 2024-09-18T07:45:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0037854500000000001 2024-09-18T07:45:56.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:56.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.909846+0000", 2024-09-18T07:45:56.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5565000000000001e-05 2024-09-18T07:45:56.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.909945+0000", 2024-09-18T07:45:56.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8925999999999994e-05 2024-09-18T07:45:56.864 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.864 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.864 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.865 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137633 10.2 10:44856315:::benchmark_data_smithi137_151160_object1817:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e396)", 2024-09-18T07:45:56.865 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:43.769818+0000", 2024-09-18T07:45:56.865 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 73.030304293, 2024-09-18T07:45:56.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.138105184, 2024-09-18T07:45:56.865 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.865 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.865 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:56.865 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.865 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:56.865 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:56.866 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137633 2024-09-18T07:45:56.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.866 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769818+0000", 2024-09-18T07:45:56.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769818+0000", 2024-09-18T07:45:56.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769819+0000", 2024-09-18T07:45:56.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.71e-07 2024-09-18T07:45:56.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769821+0000", 2024-09-18T07:45:56.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8320000000000002e-06 2024-09-18T07:45:56.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769827+0000", 2024-09-18T07:45:56.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8799999999999996e-06 2024-09-18T07:45:56.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769833+0000", 2024-09-18T07:45:56.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5280000000000003e-06 2024-09-18T07:45:56.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774264+0000", 2024-09-18T07:45:56.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044315839999999997 2024-09-18T07:45:56.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:56.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774893+0000", 2024-09-18T07:45:56.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00062851599999999997 2024-09-18T07:45:56.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.897969+0000", 2024-09-18T07:45:56.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.123075681 2024-09-18T07:45:56.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:56.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.898060+0000", 2024-09-18T07:45:56.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1570999999999995e-05 2024-09-18T07:45:56.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:45:56.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.898142+0000", 2024-09-18T07:45:56.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1304000000000006e-05 2024-09-18T07:45:56.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.900535+0000", 2024-09-18T07:45:56.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023931239999999999 2024-09-18T07:45:56.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:56.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.907715+0000", 2024-09-18T07:45:56.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.007180278 2024-09-18T07:45:56.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.907757+0000", 2024-09-18T07:45:56.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1897000000000002e-05 2024-09-18T07:45:56.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:56.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.907819+0000", 2024-09-18T07:45:56.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2139999999999998e-05 2024-09-18T07:45:56.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.872 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:56.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.907923+0000", 2024-09-18T07:45:56.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103978 2024-09-18T07:45:56.874 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.874 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.874 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.874 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:169413 10.4 10:2906f1b0:::benchmark_data_smithi137_151160_object33597:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-18T07:45:56.874 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:12.623237+0000", 2024-09-18T07:45:56.875 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 44.176885445000003, 2024-09-18T07:45:56.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.125565337, 2024-09-18T07:45:56.875 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.875 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.875 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:56.875 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.875 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:56.875 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:56.875 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 169413 2024-09-18T07:45:56.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.875 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623237+0000", 2024-09-18T07:45:56.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623237+0000", 2024-09-18T07:45:56.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623239+0000", 2024-09-18T07:45:56.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9379999999999999e-06 2024-09-18T07:45:56.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623245+0000", 2024-09-18T07:45:56.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1630000000000001e-06 2024-09-18T07:45:56.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623257+0000", 2024-09-18T07:45:56.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2167000000000001e-05 2024-09-18T07:45:56.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623264+0000", 2024-09-18T07:45:56.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.379e-06 2024-09-18T07:45:56.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623314+0000", 2024-09-18T07:45:56.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9749000000000001e-05 2024-09-18T07:45:56.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:56.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623535+0000", 2024-09-18T07:45:56.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000220929 2024-09-18T07:45:56.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:45:56.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623694+0000", 2024-09-18T07:45:56.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000159574 2024-09-18T07:45:56.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:56.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.627299+0000", 2024-09-18T07:45:56.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036042359999999998 2024-09-18T07:45:56.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.632097+0000", 2024-09-18T07:45:56.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047979279999999999 2024-09-18T07:45:56.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.748649+0000", 2024-09-18T07:45:56.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.1165527 2024-09-18T07:45:56.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:56.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.748709+0000", 2024-09-18T07:45:56.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9419000000000001e-05 2024-09-18T07:45:56.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.748802+0000", 2024-09-18T07:45:56.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3154999999999998e-05 2024-09-18T07:45:56.881 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.881 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.881 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.881 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:169434 10.4 10:2739bd15:::benchmark_data_smithi137_151160_object33618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-18T07:45:56.881 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:12.641620+0000", 2024-09-18T07:45:56.881 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 44.158502368000001, 2024-09-18T07:45:56.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.117241649, 2024-09-18T07:45:56.881 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.881 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.882 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:56.882 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.882 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:56.882 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:56.882 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 169434 2024-09-18T07:45:56.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.882 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641620+0000", 2024-09-18T07:45:56.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641620+0000", 2024-09-18T07:45:56.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641621+0000", 2024-09-18T07:45:56.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4339999999999999e-06 2024-09-18T07:45:56.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641626+0000", 2024-09-18T07:45:56.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5109999999999997e-06 2024-09-18T07:45:56.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641636+0000", 2024-09-18T07:45:56.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0889e-05 2024-09-18T07:45:56.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641644+0000", 2024-09-18T07:45:56.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8760000000000005e-06 2024-09-18T07:45:56.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641701+0000", 2024-09-18T07:45:56.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6474999999999997e-05 2024-09-18T07:45:56.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:56.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641780+0000", 2024-09-18T07:45:56.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8894999999999996e-05 2024-09-18T07:45:56.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:45:56.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641857+0000", 2024-09-18T07:45:56.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6932000000000001e-05 2024-09-18T07:45:56.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:56.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.644448+0000", 2024-09-18T07:45:56.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025915539999999998 2024-09-18T07:45:56.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.649282+0000", 2024-09-18T07:45:56.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048333059999999999 2024-09-18T07:45:56.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.758632+0000", 2024-09-18T07:45:56.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10935044200000001 2024-09-18T07:45:56.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:56.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.758700+0000", 2024-09-18T07:45:56.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7780000000000005e-05 2024-09-18T07:45:56.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.758861+0000", 2024-09-18T07:45:56.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016155499999999999 2024-09-18T07:45:56.887 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.887 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.888 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.888 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:181640 10.4 10:2d01346c:::benchmark_data_smithi137_151160_object45824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-09-18T07:45:56.888 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:25.756539+0000", 2024-09-18T07:45:56.888 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 31.043583453, 2024-09-18T07:45:56.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13636992100000001, 2024-09-18T07:45:56.888 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.888 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.888 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:56.888 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.888 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:56.889 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:56.889 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 181640 2024-09-18T07:45:56.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.889 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756539+0000", 2024-09-18T07:45:56.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756539+0000", 2024-09-18T07:45:56.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756541+0000", 2024-09-18T07:45:56.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1600000000000001e-06 2024-09-18T07:45:56.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756547+0000", 2024-09-18T07:45:56.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6880000000000002e-06 2024-09-18T07:45:56.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756559+0000", 2024-09-18T07:45:56.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1667e-05 2024-09-18T07:45:56.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756566+0000", 2024-09-18T07:45:56.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2840000000000001e-06 2024-09-18T07:45:56.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756622+0000", 2024-09-18T07:45:56.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5383999999999997e-05 2024-09-18T07:45:56.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:56.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756754+0000", 2024-09-18T07:45:56.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013199300000000001 2024-09-18T07:45:56.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:45:56.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756895+0000", 2024-09-18T07:45:56.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014090899999999999 2024-09-18T07:45:56.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:56.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.759543+0000", 2024-09-18T07:45:56.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026484299999999998 2024-09-18T07:45:56.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.760252+0000", 2024-09-18T07:45:56.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00070913200000000001 2024-09-18T07:45:56.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.892787+0000", 2024-09-18T07:45:56.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13253521900000001 2024-09-18T07:45:56.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:56.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.892825+0000", 2024-09-18T07:45:56.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7258999999999997e-05 2024-09-18T07:45:56.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.892908+0000", 2024-09-18T07:45:56.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3795999999999995e-05 2024-09-18T07:45:56.894 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.894 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.894 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.895 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189721 10.6 10.a7ff4e76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:45:56.895 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.403483+0000", 2024-09-18T07:45:56.895 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 22.396638878000001, 2024-09-18T07:45:56.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.73509391899999998, 2024-09-18T07:45:56.895 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.895 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.895 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:45:56.895 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.895 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:56.895 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:56.895 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189721 2024-09-18T07:45:56.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.896 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403483+0000", 2024-09-18T07:45:56.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403483+0000", 2024-09-18T07:45:56.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403485+0000", 2024-09-18T07:45:56.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.849e-06 2024-09-18T07:45:56.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403490+0000", 2024-09-18T07:45:56.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.468e-06 2024-09-18T07:45:56.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403502+0000", 2024-09-18T07:45:56.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1865000000000001e-05 2024-09-18T07:45:56.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403509+0000", 2024-09-18T07:45:56.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.601e-06 2024-09-18T07:45:56.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403574+0000", 2024-09-18T07:45:56.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5314000000000005e-05 2024-09-18T07:45:56.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:56.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403585+0000", 2024-09-18T07:45:56.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0688e-05 2024-09-18T07:45:56.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138555+0000", 2024-09-18T07:45:56.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.73496987400000002 2024-09-18T07:45:56.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138577+0000", 2024-09-18T07:45:56.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.226e-05 2024-09-18T07:45:56.899 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.899 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.900 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.900 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189731 10.6 10.884df406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:45:56.900 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.415372+0000", 2024-09-18T07:45:56.900 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 22.384750355000001, 2024-09-18T07:45:56.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.72323327699999995, 2024-09-18T07:45:56.900 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.900 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.900 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:45:56.900 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.900 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:56.900 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:56.901 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189731 2024-09-18T07:45:56.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.901 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415372+0000", 2024-09-18T07:45:56.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415372+0000", 2024-09-18T07:45:56.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415373+0000", 2024-09-18T07:45:56.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3430000000000001e-06 2024-09-18T07:45:56.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415378+0000", 2024-09-18T07:45:56.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5070000000000002e-06 2024-09-18T07:45:56.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415387+0000", 2024-09-18T07:45:56.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7529999999999996e-06 2024-09-18T07:45:56.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415394+0000", 2024-09-18T07:45:56.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4740000000000002e-06 2024-09-18T07:45:56.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415445+0000", 2024-09-18T07:45:56.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1202000000000002e-05 2024-09-18T07:45:56.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:56.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415453+0000", 2024-09-18T07:45:56.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.123e-06 2024-09-18T07:45:56.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138598+0000", 2024-09-18T07:45:56.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.72314508399999999 2024-09-18T07:45:56.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138605+0000", 2024-09-18T07:45:56.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7909999999999999e-06 2024-09-18T07:45:56.905 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.905 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.905 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.905 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189736 10.6 10.2043ac56 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:45:56.905 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.419163+0000", 2024-09-18T07:45:56.905 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 22.380959287, 2024-09-18T07:45:56.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71946739599999998, 2024-09-18T07:45:56.905 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.905 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.905 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:45:56.905 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.906 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:56.906 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:56.906 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189736 2024-09-18T07:45:56.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.906 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419163+0000", 2024-09-18T07:45:56.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419163+0000", 2024-09-18T07:45:56.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419164+0000", 2024-09-18T07:45:56.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4130000000000001e-06 2024-09-18T07:45:56.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419169+0000", 2024-09-18T07:45:56.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8269999999999997e-06 2024-09-18T07:45:56.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419183+0000", 2024-09-18T07:45:56.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3596e-05 2024-09-18T07:45:56.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419191+0000", 2024-09-18T07:45:56.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0239999999999997e-06 2024-09-18T07:45:56.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419296+0000", 2024-09-18T07:45:56.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0001058 2024-09-18T07:45:56.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:45:56.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419306+0000", 2024-09-18T07:45:56.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5109999999999992e-06 2024-09-18T07:45:56.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138625+0000", 2024-09-18T07:45:56.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71931864599999995 2024-09-18T07:45:56.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138630+0000", 2024-09-18T07:45:56.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5790000000000003e-06 2024-09-18T07:45:56.911 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.911 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.911 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.911 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "MOSDPGPushReply(10.1 428/396 [PushReplyOp(10:8e8dbbb4:::benchmark_data_smithi137_151160_object126986:head)])", 2024-09-18T07:45:56.911 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:51.466142+0000", 2024-09-18T07:45:56.911 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 5.3339798849999998, 2024-09-18T07:45:56.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.048643458000000001, 2024-09-18T07:45:56.911 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.911 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.911 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:45:56.912 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.466142+0000", 2024-09-18T07:45:56.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.466142+0000", 2024-09-18T07:45:56.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.466142+0000", 2024-09-18T07:45:56.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.79e-07 2024-09-18T07:45:56.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.466146+0000", 2024-09-18T07:45:56.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.72e-06 2024-09-18T07:45:56.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.466152+0000", 2024-09-18T07:45:56.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8760000000000002e-06 2024-09-18T07:45:56.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.466158+0000", 2024-09-18T07:45:56.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5400000000000003e-06 2024-09-18T07:45:56.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.514567+0000", 2024-09-18T07:45:56.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.048409801000000002 2024-09-18T07:45:56.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.514786+0000", 2024-09-18T07:45:56.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000218242 2024-09-18T07:45:56.915 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.915 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.915 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.915 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:197298 10.4 10:24fd7b8f:::benchmark_data_smithi137_151160_object61482:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-09-18T07:45:56.915 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:52.850006+0000", 2024-09-18T07:45:56.915 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 3.950116108, 2024-09-18T07:45:56.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.046245289000000002, 2024-09-18T07:45:56.915 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.915 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.915 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:56.915 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.916 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:56.916 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:56.916 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 197298 2024-09-18T07:45:56.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.916 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:52.850006+0000", 2024-09-18T07:45:56.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:52.850006+0000", 2024-09-18T07:45:56.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:52.850007+0000", 2024-09-18T07:45:56.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7000000000000004e-07 2024-09-18T07:45:56.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:52.850010+0000", 2024-09-18T07:45:56.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9799999999999998e-06 2024-09-18T07:45:56.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:52.850015+0000", 2024-09-18T07:45:56.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.49e-06 2024-09-18T07:45:56.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:52.850020+0000", 2024-09-18T07:45:56.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.848e-06 2024-09-18T07:45:56.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:52.875298+0000", 2024-09-18T07:45:56.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025277844000000001 2024-09-18T07:45:56.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:56.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:52.875434+0000", 2024-09-18T07:45:56.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013594599999999999 2024-09-18T07:45:56.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:45:56.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:52.875566+0000", 2024-09-18T07:45:56.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013184700000000001 2024-09-18T07:45:56.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:52.882800+0000", 2024-09-18T07:45:56.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.007234636 2024-09-18T07:45:56.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:52.882835+0000", 2024-09-18T07:45:56.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4646e-05 2024-09-18T07:45:56.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:56.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:52.896160+0000", 2024-09-18T07:45:56.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013324686000000001 2024-09-18T07:45:56.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:56.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:52.896206+0000", 2024-09-18T07:45:56.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6272e-05 2024-09-18T07:45:56.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:52.896251+0000", 2024-09-18T07:45:56.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5423999999999997e-05 2024-09-18T07:45:56.922 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.922 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.922 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.922 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:198776 10.6 10:67a2468f:::benchmark_data_smithi137_151160_object62960:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-09-18T07:45:56.922 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:54.098675+0000", 2024-09-18T07:45:56.922 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.7014469110000001, 2024-09-18T07:45:56.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.053537116000000003, 2024-09-18T07:45:56.922 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.922 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.922 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:56.923 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.923 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:56.923 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:56.923 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 198776 2024-09-18T07:45:56.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.923 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.923 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.098675+0000", 2024-09-18T07:45:56.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.923 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.924 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.098675+0000", 2024-09-18T07:45:56.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.924 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.924 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.098676+0000", 2024-09-18T07:45:56.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.113e-06 2024-09-18T07:45:56.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.924 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.924 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.098681+0000", 2024-09-18T07:45:56.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4120000000000003e-06 2024-09-18T07:45:56.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.098688+0000", 2024-09-18T07:45:56.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5340000000000002e-06 2024-09-18T07:45:56.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.098695+0000", 2024-09-18T07:45:56.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5189999999999998e-06 2024-09-18T07:45:56.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.101439+0000", 2024-09-18T07:45:56.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027438879999999999 2024-09-18T07:45:56.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:56.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.101576+0000", 2024-09-18T07:45:56.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013693399999999999 2024-09-18T07:45:56.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4", 2024-09-18T07:45:56.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.101694+0000", 2024-09-18T07:45:56.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118183 2024-09-18T07:45:56.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.108043+0000", 2024-09-18T07:45:56.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063493120000000002 2024-09-18T07:45:56.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:56.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.152075+0000", 2024-09-18T07:45:56.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.044032416999999997 2024-09-18T07:45:56.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:56.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.152134+0000", 2024-09-18T07:45:56.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8826999999999998e-05 2024-09-18T07:45:56.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.152212+0000", 2024-09-18T07:45:56.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7977000000000002e-05 2024-09-18T07:45:56.928 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.928 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.928 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.928 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:198778 10.6 10:61a90a8f:::benchmark_data_smithi137_151160_object62962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-09-18T07:45:56.929 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:54.105885+0000", 2024-09-18T07:45:56.929 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.6942369309999998, 2024-09-18T07:45:56.929 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.046443296000000002, 2024-09-18T07:45:56.929 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.929 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.929 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:56.929 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.929 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:56.929 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:56.929 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 198778 2024-09-18T07:45:56.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.930 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.105885+0000", 2024-09-18T07:45:56.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.105885+0000", 2024-09-18T07:45:56.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.931 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.931 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.105888+0000", 2024-09-18T07:45:56.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4590000000000001e-06 2024-09-18T07:45:56.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.931 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.931 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.105894+0000", 2024-09-18T07:45:56.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4690000000000002e-06 2024-09-18T07:45:56.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.932 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.932 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.105907+0000", 2024-09-18T07:45:56.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2860000000000001e-05 2024-09-18T07:45:56.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.932 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.932 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.105914+0000", 2024-09-18T07:45:56.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1080000000000004e-06 2024-09-18T07:45:56.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.932 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.932 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.108317+0000", 2024-09-18T07:45:56.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024026719999999998 2024-09-18T07:45:56.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:56.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.108437+0000", 2024-09-18T07:45:56.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011993300000000001 2024-09-18T07:45:56.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4", 2024-09-18T07:45:56.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.108575+0000", 2024-09-18T07:45:56.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000138126 2024-09-18T07:45:56.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.118296+0000", 2024-09-18T07:45:56.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0097214439999999992 2024-09-18T07:45:56.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:56.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.152223+0000", 2024-09-18T07:45:56.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033926818999999997 2024-09-18T07:45:56.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:56.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.152260+0000", 2024-09-18T07:45:56.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7230000000000001e-05 2024-09-18T07:45:56.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.935 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.152328+0000", 2024-09-18T07:45:56.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8176000000000006e-05 2024-09-18T07:45:56.935 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.935 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.935 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.936 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:198915 10.8 10:1e9933fa:::benchmark_data_smithi137_151160_object63099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-18T07:45:56.936 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:54.240683+0000", 2024-09-18T07:45:56.936 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.5594388549999998, 2024-09-18T07:45:56.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.057016946999999998, 2024-09-18T07:45:56.937 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.937 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.937 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:56.937 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.937 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:56.937 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:56.937 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 198915 2024-09-18T07:45:56.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.937 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.937 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.240683+0000", 2024-09-18T07:45:56.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.938 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.938 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.240683+0000", 2024-09-18T07:45:56.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.938 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.938 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.240684+0000", 2024-09-18T07:45:56.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3099999999999998e-07 2024-09-18T07:45:56.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.240686+0000", 2024-09-18T07:45:56.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2979999999999999e-06 2024-09-18T07:45:56.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.240691+0000", 2024-09-18T07:45:56.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8489999999999997e-06 2024-09-18T07:45:56.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.240695+0000", 2024-09-18T07:45:56.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5730000000000001e-06 2024-09-18T07:45:56.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.240739+0000", 2024-09-18T07:45:56.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3961000000000001e-05 2024-09-18T07:45:56.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:56.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.258103+0000", 2024-09-18T07:45:56.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017363752 2024-09-18T07:45:56.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:45:56.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.258239+0000", 2024-09-18T07:45:56.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013582899999999999 2024-09-18T07:45:56.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:56.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.260984+0000", 2024-09-18T07:45:56.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027446910000000001 2024-09-18T07:45:56.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.261890+0000", 2024-09-18T07:45:56.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00090584699999999999 2024-09-18T07:45:56.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.297598+0000", 2024-09-18T07:45:56.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.035708186000000003 2024-09-18T07:45:56.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:56.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.297653+0000", 2024-09-18T07:45:56.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5754000000000002e-05 2024-09-18T07:45:56.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.943 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.297700+0000", 2024-09-18T07:45:56.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6675999999999998e-05 2024-09-18T07:45:56.943 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.943 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.943 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.943 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:199970 10.6 10:6477b3dc:::benchmark_data_smithi137_151160_object64154:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-18T07:45:56.943 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:55.066412+0000", 2024-09-18T07:45:56.943 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.733709969, 2024-09-18T07:45:56.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.045047856999999997, 2024-09-18T07:45:56.943 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:45:56.943 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:45:56.944 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:45:56.944 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:45:56.944 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:45:56.944 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:45:56.944 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 199970 2024-09-18T07:45:56.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.944 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:45:56.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:45:56.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:55.066412+0000", 2024-09-18T07:45:56.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.945 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:45:56.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:55.066412+0000", 2024-09-18T07:45:56.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:45:56.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.945 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:45:56.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:55.066413+0000", 2024-09-18T07:45:56.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7199999999999997e-07 2024-09-18T07:45:56.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:45:56.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:55.066416+0000", 2024-09-18T07:45:56.946 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2270000000000002e-06 2024-09-18T07:45:56.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:45:56.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:55.066430+0000", 2024-09-18T07:45:56.946 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3808999999999999e-05 2024-09-18T07:45:56.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:45:56.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:55.066436+0000", 2024-09-18T07:45:56.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7300000000000002e-06 2024-09-18T07:45:56.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:45:56.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:55.075436+0000", 2024-09-18T07:45:56.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0090005329999999998 2024-09-18T07:45:56.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:45:56.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:55.075561+0000", 2024-09-18T07:45:56.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012478599999999999 2024-09-18T07:45:56.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4", 2024-09-18T07:45:56.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:55.075685+0000", 2024-09-18T07:45:56.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012367699999999999 2024-09-18T07:45:56.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:45:56.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:55.081086+0000", 2024-09-18T07:45:56.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054014750000000002 2024-09-18T07:45:56.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:45:56.949 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:55.111323+0000", 2024-09-18T07:45:56.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.030237271999999999 2024-09-18T07:45:56.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:45:56.949 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:55.111379+0000", 2024-09-18T07:45:56.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5900999999999999e-05 2024-09-18T07:45:56.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:45:56.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:45:56.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:45:56.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:55.111460+0000", 2024-09-18T07:45:56.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0574999999999993e-05 2024-09-18T07:45:56.950 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.950 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.950 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.950 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:45:56.950 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:45:56.950 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:45:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:56.951 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:56.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:57.052 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:57.079 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:57.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:57.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:57.164 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:45:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:45:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:45:57.165 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:57 smithi137 ceph-mon[29408]: pgmap v795: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 24 active+undersized+degraded, 97 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 896 B/s rd, 0 B/s wr, 911 op/s; 23843/226101 objects degraded (10.545%); 19 MiB/s, 311 objects/s recovering 2024-09-18T07:45:57.165 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:57 smithi137 ceph-mon[29408]: osdmap e431: 8 total, 7 up, 8 in 2024-09-18T07:45:57.165 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:57 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3915413730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:57.165 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:57 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:57.165 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:57 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:45:57.166 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:57 smithi137 ceph-mon[39175]: pgmap v795: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 24 active+undersized+degraded, 97 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 896 B/s rd, 0 B/s wr, 911 op/s; 23843/226101 objects degraded (10.545%); 19 MiB/s, 311 objects/s recovering 2024-09-18T07:45:57.166 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:57 smithi137 ceph-mon[39175]: osdmap e431: 8 total, 7 up, 8 in 2024-09-18T07:45:57.166 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:57 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3915413730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:57.166 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:57 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:57.166 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:57 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:45:57.181 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:57.207 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:57.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:57.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:57.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:57.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:57.346 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:57.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:57.384 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:45:57.384 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:45:57.384 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:45:57.385 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:45:57.385 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:45:57.385 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:45:57.385 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:45:57.385 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:45:57.385 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:45:57.385 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:45:57.385 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:45:57.385 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:45:57.385 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-18T07:45:57.385 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:45:57.386 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:45:57.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:57.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:57 smithi203 ceph-mon[32237]: pgmap v795: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 36 active+undersized, 24 active+undersized+degraded, 97 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 896 B/s rd, 0 B/s wr, 911 op/s; 23843/226101 objects degraded (10.545%); 19 MiB/s, 311 objects/s recovering 2024-09-18T07:45:57.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:57 smithi203 ceph-mon[32237]: osdmap e431: 8 total, 7 up, 8 in 2024-09-18T07:45:57.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:57 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3915413730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:45:57.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:57 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:45:57.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:57 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:45:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:57.482 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:57.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:57.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:57.605 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:57.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:57.707 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:57.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:57.746 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:57.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:57.849 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:57.878 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:57.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:57.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:57.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:45:57.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:58.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:58.021 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:58.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:58.122 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:45:58.150 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:58.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:58.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:45:58.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:45:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:45:58.253 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:58.281 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:58.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:58.382 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:58.409 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T07:45:58.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:58.447 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:58.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:58.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:58.604 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:58.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:58.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:58.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:58.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:58.753 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:58.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:58.854 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:45:58.901 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:58.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:58.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:58.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:45:58.931 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:45:58.932 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:45:59.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:59.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:59.065 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:59.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:59.166 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:45:59.193 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:59.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:59.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:45:59.295 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:59.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:59.325 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:59.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:59.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:59 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:59.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:45:59 smithi137 ceph-mon[29408]: pgmap v797: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.37k op/s; 22309/213990 objects degraded (10.425%); 21 MiB/s, 331 objects/s recovering 2024-09-18T07:45:59.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:59 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:59.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:45:59 smithi137 ceph-mon[39175]: pgmap v797: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.37k op/s; 22309/213990 objects degraded (10.425%); 21 MiB/s, 331 objects/s recovering 2024-09-18T07:45:59.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:45:59.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:59.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:45:59.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:59 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:45:59.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:45:59 smithi203 ceph-mon[32237]: pgmap v797: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.37k op/s; 22309/213990 objects degraded (10.425%); 21 MiB/s, 331 objects/s recovering 2024-09-18T07:45:59.560 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:45:59.587 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:45:59.606 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:59.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:59.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:45:59.708 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:45:59.740 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:45:59.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:59.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:45:59.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:45:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:45:59.879 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:45:59.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:45:59.980 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:00.012 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:00.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:00.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:00.114 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:00.128 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:46:00.142 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:00.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:00.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:00.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:00 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2195672134' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:00.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:00 smithi203 ceph-mon[32237]: from='client.26344 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:00.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:00 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 22309/213990 objects degraded (10.425%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:00.243 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:00.277 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:00.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:00.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:00.379 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:00.397 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -n osd.5 --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 4.1f 2024-09-18T07:46:00.410 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:00.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:00.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:00 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2195672134' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:00 smithi137 ceph-mon[29408]: from='client.26344 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:00 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 22309/213990 objects degraded (10.425%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:00.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:00 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2195672134' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:00.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:00 smithi137 ceph-mon[39175]: from='client.26344 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:00.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:00 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 22309/213990 objects degraded (10.425%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:00.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:00.556 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:00.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:00.657 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:00.660 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:46:00.661 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:46:00.693 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/osd.5/config 2024-09-18T07:46:00.699 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:00.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:00.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:00.840 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:00.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:00.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:00.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:46:00.942 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:01.017 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:01.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:01.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:01.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:01.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:01.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:01.259 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:01 smithi137 ceph-mon[29408]: pgmap v798: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 825 op/s; 22003/211248 objects degraded (10.416%); 2.2 MiB/s, 35 objects/s recovering 2024-09-18T07:46:01.259 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:01 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:01.259 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:01 smithi137 ceph-mon[29408]: from='client.26344 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:46:01.259 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:01 smithi137 ceph-mon[29408]: osdmap e432: 8 total, 7 up, 8 in 2024-09-18T07:46:01.260 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:01 smithi137 ceph-mon[39175]: pgmap v798: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 825 op/s; 22003/211248 objects degraded (10.416%); 2.2 MiB/s, 35 objects/s recovering 2024-09-18T07:46:01.260 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:01 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:01.260 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:01 smithi137 ceph-mon[39175]: from='client.26344 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:46:01.260 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:01 smithi137 ceph-mon[39175]: osdmap e432: 8 total, 7 up, 8 in 2024-09-18T07:46:01.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:01.310 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:01.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:01.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:01.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:01.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:01.446 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:01.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:01.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:01 smithi203 ceph-mon[32237]: pgmap v798: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 825 op/s; 22003/211248 objects degraded (10.416%); 2.2 MiB/s, 35 objects/s recovering 2024-09-18T07:46:01.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:01 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:01.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:01 smithi203 ceph-mon[32237]: from='client.26344 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:46:01.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:01 smithi203 ceph-mon[32237]: osdmap e432: 8 total, 7 up, 8 in 2024-09-18T07:46:01.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:01.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:01.585 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:01.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:01.686 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:01.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:01.715 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:01.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:01.751 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T07:46:01.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:01.859 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:01.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:01.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:01.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:02.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:02.004 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:02.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:02.105 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:02.133 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:02.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:02.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:02.198 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:02.235 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:02.244 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:02.259 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:02.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:02.361 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:02.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:02.395 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:02.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:02.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:02.558 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:02.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:02.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:02.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:02.701 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:02.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:02.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:02.804 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:02.839 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:02.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:02.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:02.940 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:02.965 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:02.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:02.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:03.071 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:03.076 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:46:03.076 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:46:03.107 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:03.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:03.121 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:46:03.121 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:46:03.161 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:46:03.161 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:46:03.192 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:46:03.193 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:46:03.210 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:03.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:03.241 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:03.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:03.248 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:46:03.249 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:46:03.249 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-18T07:46:03.286 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:46:03.287 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:46:03.289 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:46:03.289 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:46:03.290 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:46:03.290 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:46:03.290 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:46:03.320 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:46:03.320 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:46:03.343 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:03.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:03.382 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:03.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:03.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:03 smithi137 ceph-mon[29408]: pgmap v800: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 958 op/s; 22003/206820 objects degraded (10.639%); 2.0 MiB/s, 31 objects/s recovering 2024-09-18T07:46:03.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:03 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3063714355' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:03.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:03 smithi137 ceph-mon[39175]: pgmap v800: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 958 op/s; 22003/206820 objects degraded (10.639%); 2.0 MiB/s, 31 objects/s recovering 2024-09-18T07:46:03.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:03 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3063714355' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:03.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:03 smithi203 ceph-mon[32237]: pgmap v800: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 958 op/s; 22003/206820 objects degraded (10.639%); 2.0 MiB/s, 31 objects/s recovering 2024-09-18T07:46:03.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:03 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3063714355' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:03.483 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:03.520 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:03.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:03.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:03.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:03.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:03.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:03.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:03.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:03.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:03.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:03.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:03.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:03.929 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:03.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:03.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:04.031 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:04.057 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:04.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:04.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:04.158 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:04.175 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:46:04.191 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:04.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:04.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:04.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:04.339 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:04.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:04.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:04.441 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:46:04.448 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:46:04.468 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:04.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:04.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:04.570 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:04.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:04.594 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:04.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:04.695 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:04.728 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:04.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:04.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:04.831 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:04.866 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:04.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:04.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:04.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T07:46:04.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:05.013 INFO:teuthology.orchestra.run.smithi203.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-18T07:46:05.043 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:05.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:05.049 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:05.145 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:05.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:05 smithi137 ceph-mon[29408]: pgmap v801: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 20083/192858 objects degraded (10.413%); 1.6 MiB/s, 26 objects/s recovering 2024-09-18T07:46:05.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:05 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 20083/192858 objects degraded (10.413%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:05.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:05 smithi137 ceph-mon[29408]: osdmap e433: 8 total, 7 up, 8 in 2024-09-18T07:46:05.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:05 smithi137 ceph-mon[39175]: pgmap v801: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 20083/192858 objects degraded (10.413%); 1.6 MiB/s, 26 objects/s recovering 2024-09-18T07:46:05.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:05 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 20083/192858 objects degraded (10.413%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:05.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:05 smithi137 ceph-mon[39175]: osdmap e433: 8 total, 7 up, 8 in 2024-09-18T07:46:05.176 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:05.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:05.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:05 smithi203 ceph-mon[32237]: pgmap v801: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 20083/192858 objects degraded (10.413%); 1.6 MiB/s, 26 objects/s recovering 2024-09-18T07:46:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:05 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 20083/192858 objects degraded (10.413%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:05.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:05 smithi203 ceph-mon[32237]: osdmap e433: 8 total, 7 up, 8 in 2024-09-18T07:46:05.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:05.318 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:05.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:05.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:05.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:05.469 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:05.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:05.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:05.571 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:05.599 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:05.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:05.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:05.701 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:05.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:05.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:05.765 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:05.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:05.876 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:05.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:05.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:05.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:06.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:06.015 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:06.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:06.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:06.139 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:46:06.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:06.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:06.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:06.161 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:06 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1690953469' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:46:06.161 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:06 smithi137 ceph-mon[29408]: from='client.36278 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:46:06.162 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:06 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1690953469' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:46:06.162 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:06 smithi137 ceph-mon[39175]: from='client.36278 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:46:06.258 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:06.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:06.294 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:06.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:06.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:06.425 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:06.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:06.449 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:46:06.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:06 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1690953469' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:46:06.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:06 smithi203 ceph-mon[32237]: from='client.36278 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:46:06.526 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:06.555 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:06.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:06.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:06.657 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:06.683 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:06.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:06.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:06.785 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:06.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:06.830 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:06.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:06.931 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:06.954 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:06.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:07.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:07.091 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:07.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:07.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:07.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:07.234 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:07.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:07.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:07.336 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:07.367 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:07.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:07.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:07.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:07 smithi137 ceph-mon[29408]: pgmap v803: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 880 op/s; 20083/192858 objects degraded (10.413%) 2024-09-18T07:46:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:07 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:07 smithi137 ceph-mon[29408]: from='client.36278 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:46:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:07 smithi137 ceph-mon[29408]: osdmap e434: 8 total, 7 up, 8 in 2024-09-18T07:46:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:07 smithi137 ceph-mon[29408]: 3.14 scrub starts 2024-09-18T07:46:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:07 smithi137 ceph-mon[29408]: 3.14 scrub ok 2024-09-18T07:46:07.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:46:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:46:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:46:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:07 smithi137 ceph-mon[39175]: pgmap v803: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 880 op/s; 20083/192858 objects degraded (10.413%) 2024-09-18T07:46:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:07 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:07 smithi137 ceph-mon[39175]: from='client.36278 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:46:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:07 smithi137 ceph-mon[39175]: osdmap e434: 8 total, 7 up, 8 in 2024-09-18T07:46:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:07 smithi137 ceph-mon[39175]: 3.14 scrub starts 2024-09-18T07:46:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:07 smithi137 ceph-mon[39175]: 3.14 scrub ok 2024-09-18T07:46:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:07 smithi203 ceph-mon[32237]: pgmap v803: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 880 op/s; 20083/192858 objects degraded (10.413%) 2024-09-18T07:46:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:07 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:07 smithi203 ceph-mon[32237]: from='client.36278 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:46:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:07 smithi203 ceph-mon[32237]: osdmap e434: 8 total, 7 up, 8 in 2024-09-18T07:46:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:07 smithi203 ceph-mon[32237]: 3.14 scrub starts 2024-09-18T07:46:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:07 smithi203 ceph-mon[32237]: 3.14 scrub ok 2024-09-18T07:46:07.470 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:07.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:07.505 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:07.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:07.606 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:07.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:07.647 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:07.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:07.748 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:07.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:07.782 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:07.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:07.883 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:07.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:07.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:07.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:08.042 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:08.075 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:08.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:08.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:08.129 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:46:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:46:08.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:08 smithi137 ceph-mon[29408]: 6.2 scrub starts 2024-09-18T07:46:08.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:08 smithi137 ceph-mon[29408]: 6.2 scrub ok 2024-09-18T07:46:08.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:08 smithi137 ceph-mon[39175]: 6.2 scrub starts 2024-09-18T07:46:08.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:08 smithi137 ceph-mon[39175]: 6.2 scrub ok 2024-09-18T07:46:08.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:08.213 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:08.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:08.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:08.314 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:08.350 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:08.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:08.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:08.430 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:46:08.430 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:46:08.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:46:08.431 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:46:08.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.431 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172965 9.e e335/289 9:7fd9b0fe:::benchmark_data_smithi137_126072_object40465:head v 335'10641, mlcod=335'10639)", 2024-09-18T07:46:08.431 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.554501+0000", 2024-09-18T07:46:08.431 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 269.87127157800001, 2024-09-18T07:46:08.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.178309934, 2024-09-18T07:46:08.431 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.431 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.431 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.431 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554501+0000", 2024-09-18T07:46:08.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554501+0000", 2024-09-18T07:46:08.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554502+0000", 2024-09-18T07:46:08.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.443e-06 2024-09-18T07:46:08.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554510+0000", 2024-09-18T07:46:08.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0660000000000004e-06 2024-09-18T07:46:08.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554518+0000", 2024-09-18T07:46:08.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0339999999999997e-06 2024-09-18T07:46:08.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554523+0000", 2024-09-18T07:46:08.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8740000000000003e-06 2024-09-18T07:46:08.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554567+0000", 2024-09-18T07:46:08.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3294000000000003e-05 2024-09-18T07:46:08.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554586+0000", 2024-09-18T07:46:08.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9419000000000002e-05 2024-09-18T07:46:08.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732782+0000", 2024-09-18T07:46:08.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.178196153 2024-09-18T07:46:08.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732811+0000", 2024-09-18T07:46:08.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8651000000000001e-05 2024-09-18T07:46:08.435 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.435 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.436 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.436 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172968 9.0 e335/289 9:02c69111:::benchmark_data_smithi137_126072_object40468:head v 335'10807, mlcod=335'10804)", 2024-09-18T07:46:08.436 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.554702+0000", 2024-09-18T07:46:08.436 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 269.87107023099998, 2024-09-18T07:46:08.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.179882141, 2024-09-18T07:46:08.436 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.436 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.436 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.436 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554702+0000", 2024-09-18T07:46:08.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554702+0000", 2024-09-18T07:46:08.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554704+0000", 2024-09-18T07:46:08.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.359e-06 2024-09-18T07:46:08.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554711+0000", 2024-09-18T07:46:08.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.199e-06 2024-09-18T07:46:08.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554720+0000", 2024-09-18T07:46:08.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5629999999999998e-06 2024-09-18T07:46:08.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554727+0000", 2024-09-18T07:46:08.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5889999999999999e-06 2024-09-18T07:46:08.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554767+0000", 2024-09-18T07:46:08.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9712000000000002e-05 2024-09-18T07:46:08.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554782+0000", 2024-09-18T07:46:08.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5639000000000001e-05 2024-09-18T07:46:08.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734508+0000", 2024-09-18T07:46:08.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17972560900000001 2024-09-18T07:46:08.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734584+0000", 2024-09-18T07:46:08.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6470999999999993e-05 2024-09-18T07:46:08.440 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.440 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.440 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.440 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172969 9.6 e335/289 9:60d9b40c:::benchmark_data_smithi137_126072_object40469:head v 335'10756, mlcod=335'10754)", 2024-09-18T07:46:08.441 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.554970+0000", 2024-09-18T07:46:08.441 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 269.87080289400001, 2024-09-18T07:46:08.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17787228899999999, 2024-09-18T07:46:08.441 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.441 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.441 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.441 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554970+0000", 2024-09-18T07:46:08.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554970+0000", 2024-09-18T07:46:08.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554971+0000", 2024-09-18T07:46:08.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0070000000000001e-06 2024-09-18T07:46:08.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554976+0000", 2024-09-18T07:46:08.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6330000000000001e-06 2024-09-18T07:46:08.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554993+0000", 2024-09-18T07:46:08.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6949e-05 2024-09-18T07:46:08.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.555000+0000", 2024-09-18T07:46:08.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5579999999999997e-06 2024-09-18T07:46:08.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.555176+0000", 2024-09-18T07:46:08.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017662999999999999 2024-09-18T07:46:08.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.555193+0000", 2024-09-18T07:46:08.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7212000000000001e-05 2024-09-18T07:46:08.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732814+0000", 2024-09-18T07:46:08.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.177620315 2024-09-18T07:46:08.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732842+0000", 2024-09-18T07:46:08.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7985000000000001e-05 2024-09-18T07:46:08.445 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.445 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.445 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.445 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172970 9.3 e335/289 9:c7e8a587:::benchmark_data_smithi137_126072_object40470:head v 335'10785, mlcod=335'10782)", 2024-09-18T07:46:08.445 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.557577+0000", 2024-09-18T07:46:08.445 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 269.86819497400001, 2024-09-18T07:46:08.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.173544268, 2024-09-18T07:46:08.447 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.447 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.447 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.447 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557577+0000", 2024-09-18T07:46:08.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557577+0000", 2024-09-18T07:46:08.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557579+0000", 2024-09-18T07:46:08.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5099999999999999e-06 2024-09-18T07:46:08.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557589+0000", 2024-09-18T07:46:08.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8079999999999999e-06 2024-09-18T07:46:08.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557598+0000", 2024-09-18T07:46:08.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2490000000000007e-06 2024-09-18T07:46:08.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557604+0000", 2024-09-18T07:46:08.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2079999999999997e-06 2024-09-18T07:46:08.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726578+0000", 2024-09-18T07:46:08.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.168973331 2024-09-18T07:46:08.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726596+0000", 2024-09-18T07:46:08.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8159e-05 2024-09-18T07:46:08.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731084+0000", 2024-09-18T07:46:08.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044883980000000002 2024-09-18T07:46:08.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731122+0000", 2024-09-18T07:46:08.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7605000000000003e-05 2024-09-18T07:46:08.451 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.451 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.451 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.451 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172972 9.e e335/289 9:796b9798:::benchmark_data_smithi137_126072_object40472:head v 335'10642, mlcod=335'10639)", 2024-09-18T07:46:08.451 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.557623+0000", 2024-09-18T07:46:08.451 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 269.86814911499999, 2024-09-18T07:46:08.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.175241702, 2024-09-18T07:46:08.451 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.451 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.451 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.452 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557623+0000", 2024-09-18T07:46:08.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557623+0000", 2024-09-18T07:46:08.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557624+0000", 2024-09-18T07:46:08.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4200000000000001e-07 2024-09-18T07:46:08.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557628+0000", 2024-09-18T07:46:08.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8139999999999998e-06 2024-09-18T07:46:08.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557632+0000", 2024-09-18T07:46:08.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6909999999999996e-06 2024-09-18T07:46:08.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557636+0000", 2024-09-18T07:46:08.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6629999999999999e-06 2024-09-18T07:46:08.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726230+0000", 2024-09-18T07:46:08.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.168594563 2024-09-18T07:46:08.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726286+0000", 2024-09-18T07:46:08.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5702000000000003e-05 2024-09-18T07:46:08.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732845+0000", 2024-09-18T07:46:08.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065588560000000001 2024-09-18T07:46:08.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732865+0000", 2024-09-18T07:46:08.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9970999999999999e-05 2024-09-18T07:46:08.455 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.456 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.456 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.456 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172971 9.4 e335/289 9:2cb28e72:::benchmark_data_smithi137_126072_object40471:head v 335'10859, mlcod=335'10857)", 2024-09-18T07:46:08.456 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.557654+0000", 2024-09-18T07:46:08.456 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 269.86811808900001, 2024-09-18T07:46:08.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17145278799999999, 2024-09-18T07:46:08.456 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.456 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.456 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.456 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557654+0000", 2024-09-18T07:46:08.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557654+0000", 2024-09-18T07:46:08.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557655+0000", 2024-09-18T07:46:08.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1099999999999995e-07 2024-09-18T07:46:08.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.458 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.458 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557661+0000", 2024-09-18T07:46:08.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2400000000000004e-06 2024-09-18T07:46:08.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.458 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.458 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557667+0000", 2024-09-18T07:46:08.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9139999999999996e-06 2024-09-18T07:46:08.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.458 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.458 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557672+0000", 2024-09-18T07:46:08.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6290000000000001e-06 2024-09-18T07:46:08.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.459 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726169+0000", 2024-09-18T07:46:08.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.168497014 2024-09-18T07:46:08.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.459 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726237+0000", 2024-09-18T07:46:08.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8677999999999995e-05 2024-09-18T07:46:08.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.729066+0000", 2024-09-18T07:46:08.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028290170000000001 2024-09-18T07:46:08.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.729107+0000", 2024-09-18T07:46:08.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0685000000000001e-05 2024-09-18T07:46:08.460 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.460 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.461 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.461 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172976 9.f e335/289 9:f0577982:::benchmark_data_smithi137_126072_object40476:head v 335'10844, mlcod=335'10842)", 2024-09-18T07:46:08.461 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.560675+0000", 2024-09-18T07:46:08.461 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 269.86509698399999, 2024-09-18T07:46:08.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17558478, 2024-09-18T07:46:08.461 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.461 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.461 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.462 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560675+0000", 2024-09-18T07:46:08.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560675+0000", 2024-09-18T07:46:08.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560677+0000", 2024-09-18T07:46:08.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.711e-06 2024-09-18T07:46:08.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560685+0000", 2024-09-18T07:46:08.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1880000000000002e-06 2024-09-18T07:46:08.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726219+0000", 2024-09-18T07:46:08.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16553400300000001 2024-09-18T07:46:08.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726233+0000", 2024-09-18T07:46:08.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3893000000000001e-05 2024-09-18T07:46:08.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726381+0000", 2024-09-18T07:46:08.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014760199999999999 2024-09-18T07:46:08.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726402+0000", 2024-09-18T07:46:08.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1664000000000001e-05 2024-09-18T07:46:08.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.465 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.736225+0000", 2024-09-18T07:46:08.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0098227820000000004 2024-09-18T07:46:08.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.465 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.736260+0000", 2024-09-18T07:46:08.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4937000000000001e-05 2024-09-18T07:46:08.465 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.465 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.466 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.466 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172980 9.3 e335/289 9:ce0a7506:::benchmark_data_smithi137_126072_object40480:head v 335'10786, mlcod=335'10783)", 2024-09-18T07:46:08.466 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.560718+0000", 2024-09-18T07:46:08.466 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 269.86505451199997, 2024-09-18T07:46:08.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17056611599999999, 2024-09-18T07:46:08.466 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.466 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.466 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.466 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560718+0000", 2024-09-18T07:46:08.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560718+0000", 2024-09-18T07:46:08.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560719+0000", 2024-09-18T07:46:08.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8400000000000003e-07 2024-09-18T07:46:08.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560726+0000", 2024-09-18T07:46:08.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4000000000000003e-06 2024-09-18T07:46:08.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726135+0000", 2024-09-18T07:46:08.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16540897700000001 2024-09-18T07:46:08.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726166+0000", 2024-09-18T07:46:08.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.099e-05 2024-09-18T07:46:08.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.728991+0000", 2024-09-18T07:46:08.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028248840000000002 2024-09-18T07:46:08.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.729009+0000", 2024-09-18T07:46:08.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7804000000000001e-05 2024-09-18T07:46:08.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731249+0000", 2024-09-18T07:46:08.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022400050000000002 2024-09-18T07:46:08.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.470 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731284+0000", 2024-09-18T07:46:08.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5271999999999997e-05 2024-09-18T07:46:08.470 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.470 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.470 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.470 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172984 9.0 e335/289 9:066846aa:::benchmark_data_smithi137_126072_object40484:head v 335'10808, mlcod=335'10805)", 2024-09-18T07:46:08.471 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.567459+0000", 2024-09-18T07:46:08.471 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 269.85831341800002, 2024-09-18T07:46:08.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16718034100000001, 2024-09-18T07:46:08.471 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.471 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.471 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.471 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567459+0000", 2024-09-18T07:46:08.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567459+0000", 2024-09-18T07:46:08.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567461+0000", 2024-09-18T07:46:08.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1550000000000001e-06 2024-09-18T07:46:08.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567471+0000", 2024-09-18T07:46:08.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0348e-05 2024-09-18T07:46:08.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726150+0000", 2024-09-18T07:46:08.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15867869100000001 2024-09-18T07:46:08.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726175+0000", 2024-09-18T07:46:08.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5316e-05 2024-09-18T07:46:08.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.728757+0000", 2024-09-18T07:46:08.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002581262 2024-09-18T07:46:08.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.728778+0000", 2024-09-18T07:46:08.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1739999999999999e-05 2024-09-18T07:46:08.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734588+0000", 2024-09-18T07:46:08.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005809519 2024-09-18T07:46:08.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734639+0000", 2024-09-18T07:46:08.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1310000000000002e-05 2024-09-18T07:46:08.475 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.475 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.475 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.475 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:244806 9.f e354/289 9:f6d427cd:::benchmark_data_smithi137_126072_object112306:head v 354'15363, mlcod=354'15361)", 2024-09-18T07:46:08.475 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:42:36.240579+0000", 2024-09-18T07:46:08.475 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 212.18519359499999, 2024-09-18T07:46:08.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16868968600000001, 2024-09-18T07:46:08.476 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.476 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.476 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.476 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240579+0000", 2024-09-18T07:46:08.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240579+0000", 2024-09-18T07:46:08.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240581+0000", 2024-09-18T07:46:08.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9599999999999999e-06 2024-09-18T07:46:08.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240589+0000", 2024-09-18T07:46:08.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4710000000000007e-06 2024-09-18T07:46:08.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396235+0000", 2024-09-18T07:46:08.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.155645916 2024-09-18T07:46:08.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396253+0000", 2024-09-18T07:46:08.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8170000000000001e-05 2024-09-18T07:46:08.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396312+0000", 2024-09-18T07:46:08.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9144000000000002e-05 2024-09-18T07:46:08.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396337+0000", 2024-09-18T07:46:08.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4771000000000001e-05 2024-09-18T07:46:08.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.409231+0000", 2024-09-18T07:46:08.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012893745 2024-09-18T07:46:08.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.409268+0000", 2024-09-18T07:46:08.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7509000000000003e-05 2024-09-18T07:46:08.480 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.480 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.480 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.480 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:71849 10.2 e374/364 10:41855215:::benchmark_data_smithi137_151160_object71848:head v 374'4514, mlcod=374'4512)", 2024-09-18T07:46:08.480 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:46.388333+0000", 2024-09-18T07:46:08.480 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 142.03743904199999, 2024-09-18T07:46:08.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.225317779, 2024-09-18T07:46:08.480 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.480 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.480 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.481 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388333+0000", 2024-09-18T07:46:08.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388333+0000", 2024-09-18T07:46:08.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388335+0000", 2024-09-18T07:46:08.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7910000000000001e-06 2024-09-18T07:46:08.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388381+0000", 2024-09-18T07:46:08.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5909999999999999e-05 2024-09-18T07:46:08.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388394+0000", 2024-09-18T07:46:08.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2599000000000001e-05 2024-09-18T07:46:08.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388402+0000", 2024-09-18T07:46:08.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2060000000000007e-06 2024-09-18T07:46:08.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606027+0000", 2024-09-18T07:46:08.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.21762539 2024-09-18T07:46:08.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606060+0000", 2024-09-18T07:46:08.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2991000000000003e-05 2024-09-18T07:46:08.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.613606+0000", 2024-09-18T07:46:08.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0075457730000000004 2024-09-18T07:46:08.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.613651+0000", 2024-09-18T07:46:08.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5118999999999999e-05 2024-09-18T07:46:08.484 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.484 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.484 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.485 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:71852 10.b e374/364 10:dda9ad8f:::benchmark_data_smithi137_151160_object71851:head v 374'4479, mlcod=374'4476)", 2024-09-18T07:46:08.485 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:46.390175+0000", 2024-09-18T07:46:08.485 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 142.03559731199999, 2024-09-18T07:46:08.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.22548416099999999, 2024-09-18T07:46:08.485 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.485 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.485 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.485 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.390175+0000", 2024-09-18T07:46:08.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.390175+0000", 2024-09-18T07:46:08.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.390177+0000", 2024-09-18T07:46:08.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.548e-06 2024-09-18T07:46:08.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.390217+0000", 2024-09-18T07:46:08.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0203000000000001e-05 2024-09-18T07:46:08.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606008+0000", 2024-09-18T07:46:08.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.21579153700000001 2024-09-18T07:46:08.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.490 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.490 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606020+0000", 2024-09-18T07:46:08.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1741e-05 2024-09-18T07:46:08.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.490 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.490 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606089+0000", 2024-09-18T07:46:08.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9058000000000002e-05 2024-09-18T07:46:08.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.490 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.490 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606112+0000", 2024-09-18T07:46:08.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2410999999999999e-05 2024-09-18T07:46:08.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.615499+0000", 2024-09-18T07:46:08.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093874549999999994 2024-09-18T07:46:08.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.615659+0000", 2024-09-18T07:46:08.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016020800000000001 2024-09-18T07:46:08.491 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.491 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.491 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.492 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:71850 10.2 e374/364 10:48354359:::benchmark_data_smithi137_151160_object71849:head v 374'4515, mlcod=374'4512)", 2024-09-18T07:46:08.492 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:46.392229+0000", 2024-09-18T07:46:08.492 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 142.03354372699999, 2024-09-18T07:46:08.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.221450217, 2024-09-18T07:46:08.492 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.492 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.492 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.492 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.392229+0000", 2024-09-18T07:46:08.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.392229+0000", 2024-09-18T07:46:08.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.392231+0000", 2024-09-18T07:46:08.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0109999999999999e-06 2024-09-18T07:46:08.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.392282+0000", 2024-09-18T07:46:08.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1019000000000001e-05 2024-09-18T07:46:08.494 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:08.495 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:08 smithi203 ceph-mon[32237]: 6.2 scrub starts 2024-09-18T07:46:08.496 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:08 smithi203 ceph-mon[32237]: 6.2 scrub ok 2024-09-18T07:46:08.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.496 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.605989+0000", 2024-09-18T07:46:08.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.213706807 2024-09-18T07:46:08.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606006+0000", 2024-09-18T07:46:08.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7289e-05 2024-09-18T07:46:08.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.608388+0000", 2024-09-18T07:46:08.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023825399999999998 2024-09-18T07:46:08.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.608412+0000", 2024-09-18T07:46:08.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4093000000000001e-05 2024-09-18T07:46:08.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.613656+0000", 2024-09-18T07:46:08.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052432060000000003 2024-09-18T07:46:08.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.613679+0000", 2024-09-18T07:46:08.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3252e-05 2024-09-18T07:46:08.498 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.499 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.499 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.499 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:109516 10.b e383/364 10:de3bb5f2:::benchmark_data_smithi137_151160_object109515:head v 383'6800, mlcod=383'6797)", 2024-09-18T07:46:08.499 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:17.911816+0000", 2024-09-18T07:46:08.499 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 110.51395672699999, 2024-09-18T07:46:08.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027846431000000001, 2024-09-18T07:46:08.499 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.499 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.499 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.499 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.911816+0000", 2024-09-18T07:46:08.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.911816+0000", 2024-09-18T07:46:08.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.911817+0000", 2024-09-18T07:46:08.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4890000000000001e-06 2024-09-18T07:46:08.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.911862+0000", 2024-09-18T07:46:08.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4839000000000002e-05 2024-09-18T07:46:08.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.911877+0000", 2024-09-18T07:46:08.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5432999999999999e-05 2024-09-18T07:46:08.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.911889+0000", 2024-09-18T07:46:08.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1739000000000001e-05 2024-09-18T07:46:08.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.919544+0000", 2024-09-18T07:46:08.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.007654765 2024-09-18T07:46:08.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.919562+0000", 2024-09-18T07:46:08.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8022000000000001e-05 2024-09-18T07:46:08.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.503 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.503 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.503 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.939614+0000", 2024-09-18T07:46:08.503 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020052362000000001 2024-09-18T07:46:08.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.503 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.503 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.503 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:17.939662+0000", 2024-09-18T07:46:08.504 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7781999999999998e-05 2024-09-18T07:46:08.504 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.504 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.504 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.504 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.504 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:119995 10.b e389/364 10:d71be2e3:::benchmark_data_smithi137_151160_object119994:head v 389'7429, mlcod=389'7425)", 2024-09-18T07:46:08.504 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:27.007791+0000", 2024-09-18T07:46:08.505 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 101.417981192, 2024-09-18T07:46:08.505 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026613194, 2024-09-18T07:46:08.505 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.505 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.505 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.505 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.505 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.007791+0000", 2024-09-18T07:46:08.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.007791+0000", 2024-09-18T07:46:08.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.507 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.507 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.007793+0000", 2024-09-18T07:46:08.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6840000000000001e-06 2024-09-18T07:46:08.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.508 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.508 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.007831+0000", 2024-09-18T07:46:08.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7923999999999998e-05 2024-09-18T07:46:08.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.508 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.508 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.007847+0000", 2024-09-18T07:46:08.509 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6285000000000001e-05 2024-09-18T07:46:08.509 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.509 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.509 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.007854+0000", 2024-09-18T07:46:08.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3139999999999998e-06 2024-09-18T07:46:08.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.510 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.510 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.008659+0000", 2024-09-18T07:46:08.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00080414799999999997 2024-09-18T07:46:08.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.008709+0000", 2024-09-18T07:46:08.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0691e-05 2024-09-18T07:46:08.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.034243+0000", 2024-09-18T07:46:08.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025533456999999999 2024-09-18T07:46:08.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.512 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.512 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.512 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:27.034404+0000", 2024-09-18T07:46:08.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016169099999999999 2024-09-18T07:46:08.512 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.512 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.512 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.512 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.512 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:122152 10.2 e390/364 10:48dcd9dc:::benchmark_data_smithi137_151160_object122151:head v 390'7640, mlcod=390'7635)", 2024-09-18T07:46:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:28.657416+0000", 2024-09-18T07:46:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 99.768356557000004, 2024-09-18T07:46:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032362285999999997, 2024-09-18T07:46:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657416+0000", 2024-09-18T07:46:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657416+0000", 2024-09-18T07:46:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.514 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657417+0000", 2024-09-18T07:46:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3400000000000001e-06 2024-09-18T07:46:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657439+0000", 2024-09-18T07:46:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1903999999999999e-05 2024-09-18T07:46:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657449+0000", 2024-09-18T07:46:08.515 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.825e-06 2024-09-18T07:46:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657455+0000", 2024-09-18T07:46:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4659999999999996e-06 2024-09-18T07:46:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.664829+0000", 2024-09-18T07:46:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0073740740000000004 2024-09-18T07:46:08.516 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.517 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.517 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.517 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.664879+0000", 2024-09-18T07:46:08.517 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9308999999999998e-05 2024-09-18T07:46:08.517 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.517 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.517 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.517 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.689734+0000", 2024-09-18T07:46:08.518 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.024855497000000001 2024-09-18T07:46:08.518 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.518 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.518 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.518 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.689778+0000", 2024-09-18T07:46:08.518 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3871000000000001e-05 2024-09-18T07:46:08.518 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.518 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.519 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.519 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.519 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.519 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:135810 10.b e392/364 10:da1ccd0a:::benchmark_data_smithi137_151160_object135809:head v 392'8433, mlcod=392'8431)", 2024-09-18T07:46:08.519 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:40.045405+0000", 2024-09-18T07:46:08.519 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 88.380367364999998, 2024-09-18T07:46:08.519 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13276168799999999, 2024-09-18T07:46:08.519 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.519 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.519 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.520 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.520 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.520 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.520 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045405+0000", 2024-09-18T07:46:08.520 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.520 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.520 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.520 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.520 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045405+0000", 2024-09-18T07:46:08.520 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.520 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.521 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.521 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.521 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045406+0000", 2024-09-18T07:46:08.521 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3820000000000001e-06 2024-09-18T07:46:08.521 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.521 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.521 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.521 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045446+0000", 2024-09-18T07:46:08.521 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9371e-05 2024-09-18T07:46:08.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.522 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.522 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.522 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045457+0000", 2024-09-18T07:46:08.522 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1657e-05 2024-09-18T07:46:08.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.522 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.522 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.522 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045464+0000", 2024-09-18T07:46:08.522 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7530000000000004e-06 2024-09-18T07:46:08.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.523 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.523 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.523 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045533+0000", 2024-09-18T07:46:08.523 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.923e-05 2024-09-18T07:46:08.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.523 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.523 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.523 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045551+0000", 2024-09-18T07:46:08.523 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7844999999999999e-05 2024-09-18T07:46:08.524 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.524 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.524 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.524 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.178077+0000", 2024-09-18T07:46:08.524 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.132526221 2024-09-18T07:46:08.524 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.524 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.524 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.525 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.178167+0000", 2024-09-18T07:46:08.525 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9228999999999996e-05 2024-09-18T07:46:08.525 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.525 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.525 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.525 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.525 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.525 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:135798 10.2 e392/364 10:4cac0c3f:::benchmark_data_smithi137_151160_object135797:head v 392'8494, mlcod=392'8491)", 2024-09-18T07:46:08.525 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:40.048398+0000", 2024-09-18T07:46:08.525 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 88.377374025999998, 2024-09-18T07:46:08.525 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13237333200000001, 2024-09-18T07:46:08.525 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.526 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.526 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.526 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.526 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.526 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.526 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.048398+0000", 2024-09-18T07:46:08.526 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.526 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.526 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.527 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.527 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.048398+0000", 2024-09-18T07:46:08.527 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.527 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.527 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.527 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.048400+0000", 2024-09-18T07:46:08.527 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.826e-06 2024-09-18T07:46:08.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.528 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.528 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.048452+0000", 2024-09-18T07:46:08.528 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1360000000000003e-05 2024-09-18T07:46:08.528 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.528 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.528 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.177902+0000", 2024-09-18T07:46:08.528 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12945019699999999 2024-09-18T07:46:08.528 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.529 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.529 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.177917+0000", 2024-09-18T07:46:08.529 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4766999999999999e-05 2024-09-18T07:46:08.529 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.529 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.529 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.529 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.178327+0000", 2024-09-18T07:46:08.529 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00041051999999999999 2024-09-18T07:46:08.530 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.530 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.530 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.530 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.178347+0000", 2024-09-18T07:46:08.530 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9945e-05 2024-09-18T07:46:08.530 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.530 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.530 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.530 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.180740+0000", 2024-09-18T07:46:08.530 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023925790000000001 2024-09-18T07:46:08.530 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.531 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.531 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.531 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.180772+0000", 2024-09-18T07:46:08.531 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2138000000000003e-05 2024-09-18T07:46:08.531 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.531 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.531 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.531 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.531 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.531 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:152885 10.2 e401/396 10:47cfffdf:::benchmark_data_smithi137_151160_object17069:head v 401'9622, mlcod=401'9620)", 2024-09-18T07:46:08.531 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:57.849717+0000", 2024-09-18T07:46:08.531 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 70.576055660999998, 2024-09-18T07:46:08.531 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.030134579000000002, 2024-09-18T07:46:08.532 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.532 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.532 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.532 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.532 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.532 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.532 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:57.849717+0000", 2024-09-18T07:46:08.532 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.532 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.532 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.532 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.532 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:57.849717+0000", 2024-09-18T07:46:08.532 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.532 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.533 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.533 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.533 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:57.849719+0000", 2024-09-18T07:46:08.533 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2069999999999998e-06 2024-09-18T07:46:08.533 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.533 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.533 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.533 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:57.849727+0000", 2024-09-18T07:46:08.533 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3590000000000008e-06 2024-09-18T07:46:08.533 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.533 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.533 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.533 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:57.849744+0000", 2024-09-18T07:46:08.534 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6926e-05 2024-09-18T07:46:08.534 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.534 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.534 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.534 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:57.849752+0000", 2024-09-18T07:46:08.534 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0649999999999999e-06 2024-09-18T07:46:08.534 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.534 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.534 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.534 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:57.849930+0000", 2024-09-18T07:46:08.534 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017782599999999999 2024-09-18T07:46:08.534 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.534 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.535 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.535 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:57.849999+0000", 2024-09-18T07:46:08.535 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8621000000000006e-05 2024-09-18T07:46:08.535 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.535 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.535 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.535 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:57.878971+0000", 2024-09-18T07:46:08.535 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028972629 2024-09-18T07:46:08.535 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.535 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.535 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.535 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:57.879851+0000", 2024-09-18T07:46:08.535 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00087994600000000005 2024-09-18T07:46:08.535 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.536 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.536 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.536 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.536 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.536 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:195065 10.0 e428/427 10:0117b3d3:::benchmark_data_smithi137_151160_object59249:head v 428'12144, mlcod=428'12137)", 2024-09-18T07:46:08.536 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:51.129094+0000", 2024-09-18T07:46:08.536 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 17.296678580999998, 2024-09-18T07:46:08.536 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027371426000000001, 2024-09-18T07:46:08.536 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:08.536 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:08.536 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:08.536 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:08.536 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.537 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:08.537 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.129094+0000", 2024-09-18T07:46:08.537 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.537 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.537 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.537 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:08.537 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.129094+0000", 2024-09-18T07:46:08.537 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:08.537 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.537 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.537 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:08.537 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.129096+0000", 2024-09-18T07:46:08.537 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8610000000000001e-06 2024-09-18T07:46:08.538 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.538 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.538 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:08.538 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.129108+0000", 2024-09-18T07:46:08.538 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1959e-05 2024-09-18T07:46:08.538 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.538 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.538 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:08.538 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.129122+0000", 2024-09-18T07:46:08.538 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4329e-05 2024-09-18T07:46:08.538 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.538 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.538 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:08.538 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.129130+0000", 2024-09-18T07:46:08.539 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.605e-06 2024-09-18T07:46:08.539 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.539 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.539 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:08.539 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.143129+0000", 2024-09-18T07:46:08.539 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013999670000000001 2024-09-18T07:46:08.539 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.539 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.539 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:08.539 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.143151+0000", 2024-09-18T07:46:08.539 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1297999999999998e-05 2024-09-18T07:46:08.539 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.539 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.540 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:08.540 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.156035+0000", 2024-09-18T07:46:08.540 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012884081 2024-09-18T07:46:08.540 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:08.540 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:08.540 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:08.540 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:51.156465+0000", 2024-09-18T07:46:08.540 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00043062299999999999 2024-09-18T07:46:08.540 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.540 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.540 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.540 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:08.540 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:08.541 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:46:08.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:08.542 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:08.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:08.643 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:08.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:08.676 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:08.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:08.777 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:08.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:08.801 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:08.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:08.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:08.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:08.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:08.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:09.042 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:09.064 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:09.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:09.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:09.166 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:09.189 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:09.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:09.291 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:09.327 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:09.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:09.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:09 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:09.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:09 smithi137 ceph-mon[29408]: pgmap v805: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.44k op/s; 18619/178158 objects degraded (10.451%) 2024-09-18T07:46:09.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:09 smithi137 ceph-mon[29408]: 5.14 scrub starts 2024-09-18T07:46:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:09 smithi137 ceph-mon[29408]: 5.14 scrub ok 2024-09-18T07:46:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:09 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:09 smithi137 ceph-mon[39175]: pgmap v805: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.44k op/s; 18619/178158 objects degraded (10.451%) 2024-09-18T07:46:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:09 smithi137 ceph-mon[39175]: 5.14 scrub starts 2024-09-18T07:46:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:09 smithi137 ceph-mon[39175]: 5.14 scrub ok 2024-09-18T07:46:09.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:09.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:09 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:09 smithi203 ceph-mon[32237]: pgmap v805: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.44k op/s; 18619/178158 objects degraded (10.451%) 2024-09-18T07:46:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:09 smithi203 ceph-mon[32237]: 5.14 scrub starts 2024-09-18T07:46:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:09 smithi203 ceph-mon[32237]: 5.14 scrub ok 2024-09-18T07:46:09.462 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:09.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:09.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:09.565 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:09.597 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:09.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:09.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:09.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:09.734 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:09.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:09.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:09.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:09.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:09.875 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:09.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:09.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:10.016 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:10.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:10.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:10.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:10.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:10.161 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:10.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:10.262 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:10.301 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:10.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:10.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:10.404 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:10.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:10 smithi137 ceph-mon[29408]: 3.13 scrub starts 2024-09-18T07:46:10.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:10 smithi137 ceph-mon[29408]: 3.13 scrub ok 2024-09-18T07:46:10.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:10 smithi137 ceph-mon[29408]: 5.a scrub starts 2024-09-18T07:46:10.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:10 smithi137 ceph-mon[29408]: 5.a scrub ok 2024-09-18T07:46:10.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:10 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 18619/178158 objects degraded (10.451%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:10.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:10 smithi137 ceph-mon[29408]: osdmap e435: 8 total, 7 up, 8 in 2024-09-18T07:46:10.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:10 smithi137 ceph-mon[39175]: 3.13 scrub starts 2024-09-18T07:46:10.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:10 smithi137 ceph-mon[39175]: 3.13 scrub ok 2024-09-18T07:46:10.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:10 smithi137 ceph-mon[39175]: 5.a scrub starts 2024-09-18T07:46:10.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:10 smithi137 ceph-mon[39175]: 5.a scrub ok 2024-09-18T07:46:10.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:10 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 18619/178158 objects degraded (10.451%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:10.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:10 smithi137 ceph-mon[39175]: osdmap e435: 8 total, 7 up, 8 in 2024-09-18T07:46:10.433 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:10.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:10 smithi203 ceph-mon[32237]: 3.13 scrub starts 2024-09-18T07:46:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:10 smithi203 ceph-mon[32237]: 3.13 scrub ok 2024-09-18T07:46:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:10 smithi203 ceph-mon[32237]: 5.a scrub starts 2024-09-18T07:46:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:10 smithi203 ceph-mon[32237]: 5.a scrub ok 2024-09-18T07:46:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:10 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 18619/178158 objects degraded (10.451%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:10 smithi203 ceph-mon[32237]: osdmap e435: 8 total, 7 up, 8 in 2024-09-18T07:46:10.536 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:10.574 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:10.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:10.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T07:46:10.650 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:10.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:10.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:10.732 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:10.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:10.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:10.880 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:10.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:10.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:10.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:11.046 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:11.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:11.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:11.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:11.193 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:11.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:11.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:11.266 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:11 smithi137 ceph-mon[29408]: pgmap v806: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.34k op/s; 18240/174636 objects degraded (10.445%) 2024-09-18T07:46:11.267 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:11 smithi137 ceph-mon[39175]: pgmap v806: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.34k op/s; 18240/174636 objects degraded (10.445%) 2024-09-18T07:46:11.295 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:11.313 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:46:11.313 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:46:11.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:11.326 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:11.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:11.344 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:46:11.344 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:46:11.371 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:46:11.371 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:46:11.401 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:46:11.401 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:46:11.427 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:11.434 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:46:11.434 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:46:11.437 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:11.438 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-18T07:46:11.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:11 smithi203 ceph-mon[32237]: pgmap v806: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.34k op/s; 18240/174636 objects degraded (10.445%) 2024-09-18T07:46:11.472 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:46:11.472 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:46:11.489 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:11.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:11.504 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:46:11.505 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:46:11.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:11.627 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:11.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:11.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:11.729 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:11.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:11.761 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:11.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:11.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:11.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:11.901 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:11.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:12.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:12.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:12.037 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:12.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:12.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:46:12.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:12.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:12.181 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:12.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:12.282 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:12.324 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:12.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:12.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:12.400 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:12 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/300359186' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:12.400 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:12 smithi137 ceph-mon[29408]: 6.1b scrub starts 2024-09-18T07:46:12.400 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:12 smithi137 ceph-mon[29408]: 6.1b scrub ok 2024-09-18T07:46:12.400 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:46:12.400 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:12 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/300359186' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:12.400 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:12 smithi137 ceph-mon[39175]: 6.1b scrub starts 2024-09-18T07:46:12.400 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:12 smithi137 ceph-mon[39175]: 6.1b scrub ok 2024-09-18T07:46:12.401 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:46:12.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:12.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:12 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/300359186' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:12 smithi203 ceph-mon[32237]: 6.1b scrub starts 2024-09-18T07:46:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:12 smithi203 ceph-mon[32237]: 6.1b scrub ok 2024-09-18T07:46:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:46:12.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:12.474 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:12.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:12.575 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:12.615 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:46:12.615 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:46:12.619 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:12.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:12.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:12.721 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:12.755 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:12.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:12.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:12.785 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:46:12.785 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:46:12.857 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:12.894 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:12.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:12.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:12.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:13.035 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:13.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:13.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:13.083 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:13.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:13.176 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:13.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:13.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:13.278 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:13.310 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:13.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:13.340 INFO:teuthology.orchestra.run.smithi203.stdout: marking collection for removal 2024-09-18T07:46:13.365 INFO:teuthology.orchestra.run.smithi203.stdout:setting '_remove' omap key 2024-09-18T07:46:13.366 INFO:teuthology.orchestra.run.smithi203.stdout:finish_remove_pgs 4.1f_head removing 4.1f 2024-09-18T07:46:13.369 INFO:teuthology.orchestra.run.smithi203.stdout:Remove successful 2024-09-18T07:46:13.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:13.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:13 smithi137 ceph-mon[29408]: pgmap v808: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.13k op/s; 18240/170112 objects degraded (10.722%) 2024-09-18T07:46:13.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:13 smithi137 ceph-mon[29408]: 5.c scrub starts 2024-09-18T07:46:13.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:13 smithi137 ceph-mon[29408]: 5.c scrub ok 2024-09-18T07:46:13.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:13 smithi137 ceph-mon[39175]: pgmap v808: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.13k op/s; 18240/170112 objects degraded (10.722%) 2024-09-18T07:46:13.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:13 smithi137 ceph-mon[39175]: 5.c scrub starts 2024-09-18T07:46:13.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:13 smithi137 ceph-mon[39175]: 5.c scrub ok 2024-09-18T07:46:13.442 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:13.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:13.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:13 smithi203 ceph-mon[32237]: pgmap v808: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.13k op/s; 18240/170112 objects degraded (10.722%) 2024-09-18T07:46:13.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:13 smithi203 ceph-mon[32237]: 5.c scrub starts 2024-09-18T07:46:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:13 smithi203 ceph-mon[32237]: 5.c scrub ok 2024-09-18T07:46:13.543 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:13.572 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:13.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:13.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:13.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:13.689 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T07:46:13.709 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:13.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:13.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:13.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:13.859 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:13.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:13.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:13.961 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:13.990 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:13.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:13.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:14.092 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:14.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:14.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:14.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:14.242 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:14.271 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:14.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:14.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:14.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:14.406 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:14.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:14.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:14 smithi203 ceph-mon[32237]: 4.4 scrub starts 2024-09-18T07:46:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:14 smithi203 ceph-mon[32237]: 4.4 scrub ok 2024-09-18T07:46:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:14 smithi203 ceph-mon[32237]: 5.10 scrub starts 2024-09-18T07:46:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:14 smithi203 ceph-mon[32237]: 5.10 scrub ok 2024-09-18T07:46:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:14 smithi203 ceph-mon[32237]: 4.c scrub starts 2024-09-18T07:46:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:14 smithi203 ceph-mon[32237]: 4.c scrub ok 2024-09-18T07:46:14.508 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:14.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:14.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:14.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:14.581 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:14 smithi137 ceph-mon[29408]: 4.4 scrub starts 2024-09-18T07:46:14.581 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:14 smithi137 ceph-mon[29408]: 4.4 scrub ok 2024-09-18T07:46:14.581 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:14 smithi137 ceph-mon[29408]: 5.10 scrub starts 2024-09-18T07:46:14.581 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:14 smithi137 ceph-mon[29408]: 5.10 scrub ok 2024-09-18T07:46:14.581 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:14 smithi137 ceph-mon[29408]: 4.c scrub starts 2024-09-18T07:46:14.581 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:14 smithi137 ceph-mon[29408]: 4.c scrub ok 2024-09-18T07:46:14.582 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:14 smithi137 ceph-mon[39175]: 4.4 scrub starts 2024-09-18T07:46:14.582 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:14 smithi137 ceph-mon[39175]: 4.4 scrub ok 2024-09-18T07:46:14.582 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:14 smithi137 ceph-mon[39175]: 5.10 scrub starts 2024-09-18T07:46:14.582 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:14 smithi137 ceph-mon[39175]: 5.10 scrub ok 2024-09-18T07:46:14.582 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:14 smithi137 ceph-mon[39175]: 4.c scrub starts 2024-09-18T07:46:14.582 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:14 smithi137 ceph-mon[39175]: 4.c scrub ok 2024-09-18T07:46:14.617 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:46:14.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:14.671 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:14.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:14.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:14.742 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell -n osd.5 --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.1f.5 2024-09-18T07:46:14.774 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:14.790 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:14.812 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:14.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:14.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:14.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:14.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:14.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:15.007 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/osd.5/config 2024-09-18T07:46:15.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:15.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:15.117 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:15.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:15.195 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:46:15.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:15.254 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:15.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:15.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:15.356 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:15.382 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:15.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:15.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:15.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:15 smithi203 ceph-mon[32237]: pgmap v809: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s; 16340/156519 objects degraded (10.440%) 2024-09-18T07:46:15.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:15 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3431056976' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:15.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:15 smithi203 ceph-mon[32237]: from='client.26365 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:15.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:15 smithi203 ceph-mon[32237]: 4.14 scrub starts 2024-09-18T07:46:15.427 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:15 smithi203 ceph-mon[32237]: 4.14 scrub ok 2024-09-18T07:46:15.428 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:15 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 16340/156519 objects degraded (10.440%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:15.484 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:15.512 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:15.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:15.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:15.614 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:15.639 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:15.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:15.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:15.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:15 smithi137 ceph-mon[29408]: pgmap v809: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s; 16340/156519 objects degraded (10.440%) 2024-09-18T07:46:15.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:15 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3431056976' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:15.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:15 smithi137 ceph-mon[29408]: from='client.26365 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:15.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:15 smithi137 ceph-mon[29408]: 4.14 scrub starts 2024-09-18T07:46:15.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:15 smithi137 ceph-mon[29408]: 4.14 scrub ok 2024-09-18T07:46:15.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:15 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 16340/156519 objects degraded (10.440%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:15.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:15 smithi137 ceph-mon[39175]: pgmap v809: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s; 16340/156519 objects degraded (10.440%) 2024-09-18T07:46:15.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:15 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3431056976' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:15.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:15 smithi137 ceph-mon[39175]: from='client.26365 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:15.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:15 smithi137 ceph-mon[39175]: 4.14 scrub starts 2024-09-18T07:46:15.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:15 smithi137 ceph-mon[39175]: 4.14 scrub ok 2024-09-18T07:46:15.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:15 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 16340/156519 objects degraded (10.440%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:15.741 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:15.766 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:15.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:15.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:15.868 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:15.894 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:15.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:15.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:15.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:16.039 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:16.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:16.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:16.141 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:16.172 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:16.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:16.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:16.273 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:16.307 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:16.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:16.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:16.409 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:16.435 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:16.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:16.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:16.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:16 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:46:16.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:16 smithi203 ceph-mon[32237]: from='client.26365 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:46:16.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:16 smithi203 ceph-mon[32237]: osdmap e436: 8 total, 7 up, 8 in 2024-09-18T07:46:16.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:16.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:16.569 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:16.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:16.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:16 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:46:16.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:16 smithi137 ceph-mon[29408]: from='client.26365 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:46:16.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:16 smithi137 ceph-mon[29408]: osdmap e436: 8 total, 7 up, 8 in 2024-09-18T07:46:16.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:16 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:46:16.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:16 smithi137 ceph-mon[39175]: from='client.26365 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:46:16.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:16 smithi137 ceph-mon[39175]: osdmap e436: 8 total, 7 up, 8 in 2024-09-18T07:46:16.671 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:16.702 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:16.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:16.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:16.804 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:16.833 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:16.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:16.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:16.880 INFO:teuthology.orchestra.run.smithi203.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-18T07:46:16.935 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:16.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:16.967 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:16.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:17.068 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:17.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:17.107 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:17.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:17.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:17.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:17.250 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:17.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:17.351 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:17.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:17.381 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:17.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:17.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:17 smithi137 ceph-mon[29408]: pgmap v811: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 901 op/s; 16340/156519 objects degraded (10.440%) 2024-09-18T07:46:17.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:17 smithi137 ceph-mon[29408]: 1.3 scrub starts 2024-09-18T07:46:17.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:17 smithi137 ceph-mon[29408]: 1.3 scrub ok 2024-09-18T07:46:17.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:46:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:46:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:46:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:17 smithi137 ceph-mon[39175]: pgmap v811: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 901 op/s; 16340/156519 objects degraded (10.440%) 2024-09-18T07:46:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:17 smithi137 ceph-mon[39175]: 1.3 scrub starts 2024-09-18T07:46:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:17 smithi137 ceph-mon[39175]: 1.3 scrub ok 2024-09-18T07:46:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:17 smithi203 ceph-mon[32237]: pgmap v811: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 901 op/s; 16340/156519 objects degraded (10.440%) 2024-09-18T07:46:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:17 smithi203 ceph-mon[32237]: 1.3 scrub starts 2024-09-18T07:46:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:17 smithi203 ceph-mon[32237]: 1.3 scrub ok 2024-09-18T07:46:17.485 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:17.518 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:17.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:17.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:17.620 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:17.644 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:17.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:17.745 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:17.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:17.783 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:17.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:17.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:17.954 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:17.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:17.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:17.979 INFO:teuthology.orchestra.run.smithi203.stdout:get_pg_num_history pg_num_history pg_num_history(e415 pg_nums {1={272=11},2={59=3},3={60=32},4={62=32},5={64=32},6={66=32},9={289=16},10={364=16}} deleted_pools 362,9) 2024-09-18T07:46:17.979 INFO:teuthology.orchestra.run.smithi203.stdout:Importing pgid 4.1f 2024-09-18T07:46:17.979 INFO:teuthology.orchestra.run.smithi203.stdout:Write #4:f93251ce:::obj_delete_at_hint.0000000042:head# 2024-09-18T07:46:17.979 INFO:teuthology.orchestra.run.smithi203.stdout:snapset 0=[]:{} 2024-09-18T07:46:17.980 INFO:teuthology.orchestra.run.smithi203.stdout:Write #4:fce5a501:::obj_delete_at_hint.0000000075:head# 2024-09-18T07:46:17.980 INFO:teuthology.orchestra.run.smithi203.stdout:snapset 0=[]:{} 2024-09-18T07:46:17.980 INFO:teuthology.orchestra.run.smithi203.stdout:Write #4:fd6b5df8:::obj_delete_at_hint.0000000120:head# 2024-09-18T07:46:17.980 INFO:teuthology.orchestra.run.smithi203.stdout:snapset 0=[]:{} 2024-09-18T07:46:17.981 INFO:teuthology.orchestra.run.smithi203.stdout:Write #4:fd8d39b6:::obj_delete_at_hint.0000000030:head# 2024-09-18T07:46:17.981 INFO:teuthology.orchestra.run.smithi203.stdout:snapset 0=[]:{} 2024-09-18T07:46:17.981 INFO:teuthology.orchestra.run.smithi203.stdout:Write #4:fde4dd55:gc::gc.26:head# 2024-09-18T07:46:17.981 INFO:teuthology.orchestra.run.smithi203.stdout:snapset 0=[]:{} 2024-09-18T07:46:17.981 INFO:teuthology.orchestra.run.smithi203.stdout:write_pg epoch 408 info 4.1f( v 323'19 (0'0,323'19] local-lis/les=238/239 n=5 ec=62/62 lis/c=238/238 les/c/f=239/239/0 sis=238) 2024-09-18T07:46:17.982 INFO:teuthology.orchestra.run.smithi203.stdout:Import successful 2024-09-18T07:46:18.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:18.100 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:18.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:18.195 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:46:18.195 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:46:18.202 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:18.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:46:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:46:18.229 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:18.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:18.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:18.331 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:18.359 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:18.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:18.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:18.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:18 smithi137 ceph-mon[29408]: 5.1d scrub starts 2024-09-18T07:46:18.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:18 smithi137 ceph-mon[29408]: 5.1d scrub ok 2024-09-18T07:46:18.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:18 smithi137 ceph-mon[39175]: 5.1d scrub starts 2024-09-18T07:46:18.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:18 smithi137 ceph-mon[39175]: 5.1d scrub ok 2024-09-18T07:46:18.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:18.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:18.497 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:18.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:18.503 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:18 smithi203 ceph-mon[32237]: 5.1d scrub starts 2024-09-18T07:46:18.504 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:18 smithi203 ceph-mon[32237]: 5.1d scrub ok 2024-09-18T07:46:18.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:18.628 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:18.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:18.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:18.730 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:18.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:18.768 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:18.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:18.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:18.904 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:18.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:19.006 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:19.037 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:19.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:19.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:19.139 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:19.182 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:19.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:19.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:19.283 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:19.301 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/exp.4.1f.5 2024-09-18T07:46:19.321 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:19.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:19.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:19.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:19 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:19.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:19 smithi137 ceph-mon[29408]: pgmap v812: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 14735/141249 objects degraded (10.432%) 2024-09-18T07:46:19.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:19 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:19.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:19 smithi137 ceph-mon[39175]: pgmap v812: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 14735/141249 objects degraded (10.432%) 2024-09-18T07:46:19.423 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:19.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:19.461 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:19.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:19.562 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T07:46:19.562 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:19.567 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:19.588 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:19.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:19.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:19.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:19 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:19 smithi203 ceph-mon[32237]: pgmap v812: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 14735/141249 objects degraded (10.432%) 2024-09-18T07:46:19.734 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:19.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:19.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:19.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:19.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:19.888 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:19.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:19.989 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:20.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:20.018 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:20.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:20.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:20.154 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:20.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:20.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:20.260 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:20.269 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:46:20.269 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:46:20.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:20.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:20.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:20.307 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:46:20.308 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:46:20.341 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:46:20.342 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:46:20.376 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:46:20.376 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:46:20.406 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:46:20.407 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:46:20.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:20.410 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-18T07:46:20.438 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:46:20.439 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:46:20.439 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:20 smithi137 ceph-mon[29408]: 4.11 scrub starts 2024-09-18T07:46:20.439 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:20 smithi137 ceph-mon[29408]: 4.11 scrub ok 2024-09-18T07:46:20.439 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:20 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 14735/141249 objects degraded (10.432%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:20.439 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:20 smithi137 ceph-mon[29408]: osdmap e437: 8 total, 7 up, 8 in 2024-09-18T07:46:20.440 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:20 smithi137 ceph-mon[39175]: 4.11 scrub starts 2024-09-18T07:46:20.440 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:20 smithi137 ceph-mon[39175]: 4.11 scrub ok 2024-09-18T07:46:20.440 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:20 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 14735/141249 objects degraded (10.432%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:20.440 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:20 smithi137 ceph-mon[39175]: osdmap e437: 8 total, 7 up, 8 in 2024-09-18T07:46:20.453 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:20.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:20.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:20.469 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:46:20.469 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:46:20.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:20.592 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:20.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:20.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:20.694 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:20.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:20 smithi203 ceph-mon[32237]: 4.11 scrub starts 2024-09-18T07:46:20.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:20 smithi203 ceph-mon[32237]: 4.11 scrub ok 2024-09-18T07:46:20.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:20 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 14735/141249 objects degraded (10.432%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:20.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:20 smithi203 ceph-mon[32237]: osdmap e437: 8 total, 7 up, 8 in 2024-09-18T07:46:20.723 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:20.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:20.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:46:20.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:20.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:20.867 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:20.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:20.968 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:20.994 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:20.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:20.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:21.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:21.127 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:21.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:21.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:21.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:21.269 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:21.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:21.364 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:46:21.365 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:46:21.370 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:21.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:21.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:21.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:21.494 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:21 smithi137 ceph-mon[29408]: pgmap v813: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.26k op/s; 14356/137868 objects degraded (10.413%) 2024-09-18T07:46:21.495 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:21 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1605082358' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:21.495 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:21 smithi137 ceph-mon[39175]: pgmap v813: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.26k op/s; 14356/137868 objects degraded (10.413%) 2024-09-18T07:46:21.495 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:21 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1605082358' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:21.522 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:21.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:21.548 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:21.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:21.649 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:21.660 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:46:21.660 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:46:21.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:46:21.661 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:46:21.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.661 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189709 10.3 10:c5ce4044:::benchmark_data_smithi137_151160_object53893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:46:21.661 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.383941+0000", 2024-09-18T07:46:21.661 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 47.270052653999997, 2024-09-18T07:46:21.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.754258172, 2024-09-18T07:46:21.662 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.662 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.662 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.662 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:21.663 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:46:21.663 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:21.663 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189709 2024-09-18T07:46:21.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.663 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.383941+0000", 2024-09-18T07:46:21.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.383941+0000", 2024-09-18T07:46:21.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.665 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.665 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.383945+0000", 2024-09-18T07:46:21.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8519999999999997e-06 2024-09-18T07:46:21.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.665 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.665 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.383955+0000", 2024-09-18T07:46:21.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4190000000000008e-06 2024-09-18T07:46:21.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.383974+0000", 2024-09-18T07:46:21.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9485000000000001e-05 2024-09-18T07:46:21.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.383991+0000", 2024-09-18T07:46:21.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7374999999999999e-05 2024-09-18T07:46:21.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.384053+0000", 2024-09-18T07:46:21.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1968e-05 2024-09-18T07:46:21.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:46:21.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.384063+0000", 2024-09-18T07:46:21.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2879999999999998e-06 2024-09-18T07:46:21.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.129734+0000", 2024-09-18T07:46:21.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.74567087600000004 2024-09-18T07:46:21.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.129843+0000", 2024-09-18T07:46:21.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010930900000000001 2024-09-18T07:46:21.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-09-18T07:46:21.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.129948+0000", 2024-09-18T07:46:21.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105058 2024-09-18T07:46:21.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:21.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.137595+0000", 2024-09-18T07:46:21.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00764688 2024-09-18T07:46:21.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138070+0000", 2024-09-18T07:46:21.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00047476000000000001 2024-09-18T07:46:21.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138119+0000", 2024-09-18T07:46:21.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9376000000000003e-05 2024-09-18T07:46:21.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138199+0000", 2024-09-18T07:46:21.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0525999999999994e-05 2024-09-18T07:46:21.670 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189712 10.3 10:c64415f1:::benchmark_data_smithi137_151160_object53896:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:46:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.389392+0000", 2024-09-18T07:46:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 47.264601900000002, 2024-09-18T07:46:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.75091792300000004, 2024-09-18T07:46:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.671 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.672 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.672 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:21.672 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:46:21.672 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:21.672 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189712 2024-09-18T07:46:21.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.672 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.389392+0000", 2024-09-18T07:46:21.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.389392+0000", 2024-09-18T07:46:21.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.389394+0000", 2024-09-18T07:46:21.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3300000000000001e-06 2024-09-18T07:46:21.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.389401+0000", 2024-09-18T07:46:21.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9920000000000001e-06 2024-09-18T07:46:21.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.389416+0000", 2024-09-18T07:46:21.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4644000000000001e-05 2024-09-18T07:46:21.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.389424+0000", 2024-09-18T07:46:21.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2320000000000001e-06 2024-09-18T07:46:21.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.389493+0000", 2024-09-18T07:46:21.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8683000000000006e-05 2024-09-18T07:46:21.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:46:21.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.389502+0000", 2024-09-18T07:46:21.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5580000000000006e-06 2024-09-18T07:46:21.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.134246+0000", 2024-09-18T07:46:21.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.74474385700000001 2024-09-18T07:46:21.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.134365+0000", 2024-09-18T07:46:21.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118989 2024-09-18T07:46:21.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-09-18T07:46:21.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.134496+0000", 2024-09-18T07:46:21.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000130529 2024-09-18T07:46:21.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138351+0000", 2024-09-18T07:46:21.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038556290000000002 2024-09-18T07:46:21.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:21.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.140058+0000", 2024-09-18T07:46:21.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017069229999999999 2024-09-18T07:46:21.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.140225+0000", 2024-09-18T07:46:21.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000166188 2024-09-18T07:46:21.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.140310+0000", 2024-09-18T07:46:21.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5369000000000002e-05 2024-09-18T07:46:21.678 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.678 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.678 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.679 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189716 10.9 10:9d240f9e:::benchmark_data_smithi137_151160_object53900:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:46:21.679 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.395506+0000", 2024-09-18T07:46:21.679 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 47.258487555000002, 2024-09-18T07:46:21.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.72649347500000006, 2024-09-18T07:46:21.679 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.679 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.679 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.679 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:21.680 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:46:21.680 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:21.680 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189716 2024-09-18T07:46:21.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.680 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.395506+0000", 2024-09-18T07:46:21.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.395506+0000", 2024-09-18T07:46:21.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.395509+0000", 2024-09-18T07:46:21.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.418e-06 2024-09-18T07:46:21.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.395516+0000", 2024-09-18T07:46:21.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7710000000000001e-06 2024-09-18T07:46:21.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.395528+0000", 2024-09-18T07:46:21.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2527999999999999e-05 2024-09-18T07:46:21.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.395536+0000", 2024-09-18T07:46:21.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2689999999999999e-06 2024-09-18T07:46:21.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.395598+0000", 2024-09-18T07:46:21.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1791999999999994e-05 2024-09-18T07:46:21.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:46:21.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.395613+0000", 2024-09-18T07:46:21.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5148e-05 2024-09-18T07:46:21.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.111722+0000", 2024-09-18T07:46:21.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71610891700000001 2024-09-18T07:46:21.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.114924+0000", 2024-09-18T07:46:21.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032013660000000002 2024-09-18T07:46:21.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 7", 2024-09-18T07:46:21.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.115068+0000", 2024-09-18T07:46:21.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000143962 2024-09-18T07:46:21.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.121832+0000", 2024-09-18T07:46:21.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0067644050000000002 2024-09-18T07:46:21.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:21.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.121847+0000", 2024-09-18T07:46:21.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5233e-05 2024-09-18T07:46:21.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.121899+0000", 2024-09-18T07:46:21.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1514000000000002e-05 2024-09-18T07:46:21.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.122000+0000", 2024-09-18T07:46:21.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101152 2024-09-18T07:46:21.686 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.687 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.687 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.687 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189717 10.9 10:91070f94:::benchmark_data_smithi137_151160_object53901:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:46:21.687 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.397857+0000", 2024-09-18T07:46:21.687 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 47.256136495, 2024-09-18T07:46:21.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.73002420099999998, 2024-09-18T07:46:21.688 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.688 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.688 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.688 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:21.688 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:46:21.688 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:21.688 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189717 2024-09-18T07:46:21.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.688 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.397857+0000", 2024-09-18T07:46:21.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.397857+0000", 2024-09-18T07:46:21.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.397860+0000", 2024-09-18T07:46:21.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7920000000000001e-06 2024-09-18T07:46:21.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.397867+0000", 2024-09-18T07:46:21.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1810000000000004e-06 2024-09-18T07:46:21.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.397883+0000", 2024-09-18T07:46:21.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5341000000000001e-05 2024-09-18T07:46:21.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.397892+0000", 2024-09-18T07:46:21.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9029999999999993e-06 2024-09-18T07:46:21.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.397952+0000", 2024-09-18T07:46:21.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0202999999999999e-05 2024-09-18T07:46:21.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:46:21.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.397962+0000", 2024-09-18T07:46:21.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0122999999999999e-05 2024-09-18T07:46:21.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.118561+0000", 2024-09-18T07:46:21.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.720598658 2024-09-18T07:46:21.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.119244+0000", 2024-09-18T07:46:21.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00068316899999999996 2024-09-18T07:46:21.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 7", 2024-09-18T07:46:21.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.119368+0000", 2024-09-18T07:46:21.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000124487 2024-09-18T07:46:21.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:21.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.127642+0000", 2024-09-18T07:46:21.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082739949999999993 2024-09-18T07:46:21.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.127683+0000", 2024-09-18T07:46:21.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0343000000000003e-05 2024-09-18T07:46:21.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.127737+0000", 2024-09-18T07:46:21.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3942e-05 2024-09-18T07:46:21.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.127882+0000", 2024-09-18T07:46:21.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014506400000000001 2024-09-18T07:46:21.696 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.697 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.697 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.697 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189727 10.9 10:996536d9:::benchmark_data_smithi137_151160_object53911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:46:21.697 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.413380+0000", 2024-09-18T07:46:21.697 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 47.240613412000002, 2024-09-18T07:46:21.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71477512799999998, 2024-09-18T07:46:21.697 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.697 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.697 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.698 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:21.698 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:46:21.699 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:21.699 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189727 2024-09-18T07:46:21.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.699 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.413380+0000", 2024-09-18T07:46:21.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.413380+0000", 2024-09-18T07:46:21.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.413383+0000", 2024-09-18T07:46:21.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9009999999999998e-06 2024-09-18T07:46:21.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.413390+0000", 2024-09-18T07:46:21.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7120000000000003e-06 2024-09-18T07:46:21.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.413403+0000", 2024-09-18T07:46:21.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3043e-05 2024-09-18T07:46:21.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.413411+0000", 2024-09-18T07:46:21.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7300000000000005e-06 2024-09-18T07:46:21.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.413477+0000", 2024-09-18T07:46:21.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5900000000000003e-05 2024-09-18T07:46:21.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:46:21.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.413486+0000", 2024-09-18T07:46:21.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9169999999999995e-06 2024-09-18T07:46:21.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.122038+0000", 2024-09-18T07:46:21.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.70855256 2024-09-18T07:46:21.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.122161+0000", 2024-09-18T07:46:21.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012317299999999999 2024-09-18T07:46:21.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 7", 2024-09-18T07:46:21.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.122410+0000", 2024-09-18T07:46:21.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024848099999999999 2024-09-18T07:46:21.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.127938+0000", 2024-09-18T07:46:21.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055279839999999997 2024-09-18T07:46:21.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:21.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.127949+0000", 2024-09-18T07:46:21.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1433999999999999e-05 2024-09-18T07:46:21.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.127999+0000", 2024-09-18T07:46:21.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9249000000000002e-05 2024-09-18T07:46:21.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.128156+0000", 2024-09-18T07:46:21.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015704400000000001 2024-09-18T07:46:21.706 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.706 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.707 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.707 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189735 10.9 10:99a96842:::benchmark_data_smithi137_151160_object53919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:46:21.707 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.418652+0000", 2024-09-18T07:46:21.707 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 47.235341638999998, 2024-09-18T07:46:21.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71745237399999995, 2024-09-18T07:46:21.707 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.707 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.707 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.707 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:21.707 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:46:21.707 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:21.707 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189735 2024-09-18T07:46:21.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.708 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.418652+0000", 2024-09-18T07:46:21.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.418652+0000", 2024-09-18T07:46:21.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.418655+0000", 2024-09-18T07:46:21.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6649999999999999e-06 2024-09-18T07:46:21.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.418661+0000", 2024-09-18T07:46:21.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4139999999999999e-06 2024-09-18T07:46:21.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.418674+0000", 2024-09-18T07:46:21.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2490999999999999e-05 2024-09-18T07:46:21.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.418682+0000", 2024-09-18T07:46:21.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2490000000000001e-06 2024-09-18T07:46:21.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.418743+0000", 2024-09-18T07:46:21.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0599e-05 2024-09-18T07:46:21.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:46:21.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.418752+0000", 2024-09-18T07:46:21.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9460000000000004e-06 2024-09-18T07:46:21.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.125065+0000", 2024-09-18T07:46:21.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.70631298499999995 2024-09-18T07:46:21.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.125182+0000", 2024-09-18T07:46:21.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117773 2024-09-18T07:46:21.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 7", 2024-09-18T07:46:21.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.125284+0000", 2024-09-18T07:46:21.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101947 2024-09-18T07:46:21.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:21.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.128163+0000", 2024-09-18T07:46:21.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028788239999999999 2024-09-18T07:46:21.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.135769+0000", 2024-09-18T07:46:21.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076063349999999997 2024-09-18T07:46:21.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.135858+0000", 2024-09-18T07:46:21.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8555999999999997e-05 2024-09-18T07:46:21.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.136105+0000", 2024-09-18T07:46:21.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024658999999999998 2024-09-18T07:46:21.715 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.715 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.715 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.715 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189741 10.9 10:948cd666:::benchmark_data_smithi137_151160_object53925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:46:21.715 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.425791+0000", 2024-09-18T07:46:21.715 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 47.228202707000001, 2024-09-18T07:46:21.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71069487499999995, 2024-09-18T07:46:21.715 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.716 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.716 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.716 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:21.716 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:46:21.716 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:21.716 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189741 2024-09-18T07:46:21.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.716 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.425791+0000", 2024-09-18T07:46:21.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.425791+0000", 2024-09-18T07:46:21.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.425793+0000", 2024-09-18T07:46:21.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.007e-06 2024-09-18T07:46:21.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.425800+0000", 2024-09-18T07:46:21.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.759e-06 2024-09-18T07:46:21.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.425811+0000", 2024-09-18T07:46:21.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1435e-05 2024-09-18T07:46:21.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.425818+0000", 2024-09-18T07:46:21.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1550000000000004e-06 2024-09-18T07:46:21.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.425880+0000", 2024-09-18T07:46:21.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2362999999999999e-05 2024-09-18T07:46:21.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:46:21.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.425891+0000", 2024-09-18T07:46:21.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1022000000000001e-05 2024-09-18T07:46:21.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.128192+0000", 2024-09-18T07:46:21.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.70230154 2024-09-18T07:46:21.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.128293+0000", 2024-09-18T07:46:21.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100473 2024-09-18T07:46:21.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 7", 2024-09-18T07:46:21.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.128405+0000", 2024-09-18T07:46:21.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011176200000000001 2024-09-18T07:46:21.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:21.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.133495+0000", 2024-09-18T07:46:21.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050899860000000003 2024-09-18T07:46:21.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.136143+0000", 2024-09-18T07:46:21.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002648114 2024-09-18T07:46:21.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.136328+0000", 2024-09-18T07:46:21.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018492199999999999 2024-09-18T07:46:21.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.136486+0000", 2024-09-18T07:46:21.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015833700000000001 2024-09-18T07:46:21.723 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.723 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.723 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.723 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:215858 10.5 10:ab1e836d:::benchmark_data_smithi137_151160_object80042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-18T07:46:21.724 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.932395+0000", 2024-09-18T07:46:21.724 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.721599104999999, 2024-09-18T07:46:21.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.29338155599999999, 2024-09-18T07:46:21.724 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.724 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.724 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.724 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:21.725 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:46:21.725 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:21.725 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 215858 2024-09-18T07:46:21.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.725 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932395+0000", 2024-09-18T07:46:21.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932395+0000", 2024-09-18T07:46:21.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932397+0000", 2024-09-18T07:46:21.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.869e-06 2024-09-18T07:46:21.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932402+0000", 2024-09-18T07:46:21.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0379999999999999e-06 2024-09-18T07:46:21.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932413+0000", 2024-09-18T07:46:21.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1085e-05 2024-09-18T07:46:21.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.728 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.728 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932420+0000", 2024-09-18T07:46:21.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6639999999999998e-06 2024-09-18T07:46:21.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.728 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.728 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932483+0000", 2024-09-18T07:46:21.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2391999999999995e-05 2024-09-18T07:46:21.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.729 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.729 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932590+0000", 2024-09-18T07:46:21.729 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010698 2024-09-18T07:46:21.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.729 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:46:21.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932705+0000", 2024-09-18T07:46:21.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011564499999999999 2024-09-18T07:46:21.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942971+0000", 2024-09-18T07:46:21.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010265560999999999 2024-09-18T07:46:21.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.225656+0000", 2024-09-18T07:46:21.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28268458699999999 2024-09-18T07:46:21.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:21.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.225667+0000", 2024-09-18T07:46:21.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1855000000000001e-05 2024-09-18T07:46:21.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.225715+0000", 2024-09-18T07:46:21.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7574999999999998e-05 2024-09-18T07:46:21.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.225776+0000", 2024-09-18T07:46:21.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1304999999999997e-05 2024-09-18T07:46:21.733 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.733 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.733 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.734 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:215861 10.5 10:a20f721a:::benchmark_data_smithi137_151160_object80045:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-18T07:46:21.734 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.932583+0000", 2024-09-18T07:46:21.734 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.721410519999999, 2024-09-18T07:46:21.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.29341074099999997, 2024-09-18T07:46:21.734 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.734 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.734 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.734 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:21.734 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:46:21.734 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:21.734 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 215861 2024-09-18T07:46:21.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.735 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932583+0000", 2024-09-18T07:46:21.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932583+0000", 2024-09-18T07:46:21.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932585+0000", 2024-09-18T07:46:21.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.415e-06 2024-09-18T07:46:21.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932590+0000", 2024-09-18T07:46:21.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9740000000000001e-06 2024-09-18T07:46:21.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932601+0000", 2024-09-18T07:46:21.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1161999999999999e-05 2024-09-18T07:46:21.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932609+0000", 2024-09-18T07:46:21.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7209999999999994e-06 2024-09-18T07:46:21.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.939153+0000", 2024-09-18T07:46:21.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.006544584 2024-09-18T07:46:21.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.939277+0000", 2024-09-18T07:46:21.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012341199999999999 2024-09-18T07:46:21.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:46:21.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.939396+0000", 2024-09-18T07:46:21.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118949 2024-09-18T07:46:21.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.225609+0000", 2024-09-18T07:46:21.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28621300199999999 2024-09-18T07:46:21.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:21.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.225825+0000", 2024-09-18T07:46:21.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021644000000000001 2024-09-18T07:46:21.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.225868+0000", 2024-09-18T07:46:21.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3235999999999997e-05 2024-09-18T07:46:21.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.225954+0000", 2024-09-18T07:46:21.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5556999999999994e-05 2024-09-18T07:46:21.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.225994+0000", 2024-09-18T07:46:21.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0289e-05 2024-09-18T07:46:21.740 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.740 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.741 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.741 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:215864 10.5 10:af29e229:::benchmark_data_smithi137_151160_object80048:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-18T07:46:21.741 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.932679+0000", 2024-09-18T07:46:21.741 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.721315176999999, 2024-09-18T07:46:21.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.29354628100000002, 2024-09-18T07:46:21.741 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.741 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.741 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.741 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:21.741 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:46:21.742 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:21.742 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 215864 2024-09-18T07:46:21.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.742 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932679+0000", 2024-09-18T07:46:21.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932679+0000", 2024-09-18T07:46:21.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932680+0000", 2024-09-18T07:46:21.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1399999999999995e-07 2024-09-18T07:46:21.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932688+0000", 2024-09-18T07:46:21.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5429999999999994e-06 2024-09-18T07:46:21.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932696+0000", 2024-09-18T07:46:21.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3289999999999996e-06 2024-09-18T07:46:21.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932702+0000", 2024-09-18T07:46:21.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7340000000000002e-06 2024-09-18T07:46:21.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.943008+0000", 2024-09-18T07:46:21.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010305538 2024-09-18T07:46:21.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.943109+0000", 2024-09-18T07:46:21.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101067 2024-09-18T07:46:21.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:46:21.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.943212+0000", 2024-09-18T07:46:21.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103325 2024-09-18T07:46:21.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.225814+0000", 2024-09-18T07:46:21.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28260202499999998 2024-09-18T07:46:21.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:21.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.225830+0000", 2024-09-18T07:46:21.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5526e-05 2024-09-18T07:46:21.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.226096+0000", 2024-09-18T07:46:21.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00026612199999999998 2024-09-18T07:46:21.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.226159+0000", 2024-09-18T07:46:21.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3089000000000005e-05 2024-09-18T07:46:21.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.226225+0000", 2024-09-18T07:46:21.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6068999999999994e-05 2024-09-18T07:46:21.750 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.750 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.750 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.750 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:215867 10.5 10:af3633d5:::benchmark_data_smithi137_151160_object80051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-18T07:46:21.750 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.933890+0000", 2024-09-18T07:46:21.750 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.720103961, 2024-09-18T07:46:21.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.292494488, 2024-09-18T07:46:21.750 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.750 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.751 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.751 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:21.751 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:46:21.751 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:21.751 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 215867 2024-09-18T07:46:21.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.751 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.933890+0000", 2024-09-18T07:46:21.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.933890+0000", 2024-09-18T07:46:21.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.933892+0000", 2024-09-18T07:46:21.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.686e-06 2024-09-18T07:46:21.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.933897+0000", 2024-09-18T07:46:21.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9760000000000003e-06 2024-09-18T07:46:21.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.933912+0000", 2024-09-18T07:46:21.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5413999999999998e-05 2024-09-18T07:46:21.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.933919+0000", 2024-09-18T07:46:21.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9840000000000004e-06 2024-09-18T07:46:21.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.755 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:21 smithi203 ceph-mon[32237]: pgmap v813: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.26k op/s; 14356/137868 objects degraded (10.413%) 2024-09-18T07:46:21.755 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:21 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1605082358' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:21.755 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:21.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:21.757 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:21.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:21.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.946908+0000", 2024-09-18T07:46:21.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012989351 2024-09-18T07:46:21.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.947019+0000", 2024-09-18T07:46:21.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011029199999999999 2024-09-18T07:46:21.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:46:21.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.947118+0000", 2024-09-18T07:46:21.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9687999999999998e-05 2024-09-18T07:46:21.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:21.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.225834+0000", 2024-09-18T07:46:21.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27871537299999999 2024-09-18T07:46:21.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.226038+0000", 2024-09-18T07:46:21.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020450199999999999 2024-09-18T07:46:21.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.226307+0000", 2024-09-18T07:46:21.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000268585 2024-09-18T07:46:21.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.226350+0000", 2024-09-18T07:46:21.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3606000000000002e-05 2024-09-18T07:46:21.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.226384+0000", 2024-09-18T07:46:21.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4031e-05 2024-09-18T07:46:21.762 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.762 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.762 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.762 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:215863 10.8 e434/364 10:1c60eade:::benchmark_data_smithi137_151160_object80047:head v 434'13581, mlcod=434'13576)", 2024-09-18T07:46:21.763 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.938697+0000", 2024-09-18T07:46:21.763 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.715296648000001, 2024-09-18T07:46:21.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28347392999999999, 2024-09-18T07:46:21.763 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.763 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.763 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.763 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.938697+0000", 2024-09-18T07:46:21.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.938697+0000", 2024-09-18T07:46:21.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.938699+0000", 2024-09-18T07:46:21.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4720000000000001e-06 2024-09-18T07:46:21.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.938705+0000", 2024-09-18T07:46:21.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5139999999999998e-06 2024-09-18T07:46:21.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.766 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.766 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.938716+0000", 2024-09-18T07:46:21.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0746e-05 2024-09-18T07:46:21.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.938723+0000", 2024-09-18T07:46:21.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5799999999999997e-06 2024-09-18T07:46:21.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.943095+0000", 2024-09-18T07:46:21.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043724910000000001 2024-09-18T07:46:21.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.943114+0000", 2024-09-18T07:46:21.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9205e-05 2024-09-18T07:46:21.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.221748+0000", 2024-09-18T07:46:21.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27863360799999998 2024-09-18T07:46:21.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222171+0000", 2024-09-18T07:46:21.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00042331399999999999 2024-09-18T07:46:21.769 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.769 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.769 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.769 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:215875 10.5 10:abcae4be:::benchmark_data_smithi137_151160_object80059:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-18T07:46:21.769 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.942531+0000", 2024-09-18T07:46:21.769 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.711462901999999, 2024-09-18T07:46:21.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28889221399999998, 2024-09-18T07:46:21.770 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.770 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.770 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.770 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:21.770 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:46:21.770 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:21.770 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 215875 2024-09-18T07:46:21.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.770 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942531+0000", 2024-09-18T07:46:21.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942531+0000", 2024-09-18T07:46:21.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942533+0000", 2024-09-18T07:46:21.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1490000000000001e-06 2024-09-18T07:46:21.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942539+0000", 2024-09-18T07:46:21.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0299999999999999e-06 2024-09-18T07:46:21.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942550+0000", 2024-09-18T07:46:21.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.11e-05 2024-09-18T07:46:21.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942557+0000", 2024-09-18T07:46:21.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1289999999999998e-06 2024-09-18T07:46:21.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.774 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.950154+0000", 2024-09-18T07:46:21.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0075963269999999999 2024-09-18T07:46:21.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.774 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.774 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.950250+0000", 2024-09-18T07:46:21.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6491999999999997e-05 2024-09-18T07:46:21.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.774 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:46:21.774 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.950348+0000", 2024-09-18T07:46:21.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7726999999999995e-05 2024-09-18T07:46:21.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:21.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.225838+0000", 2024-09-18T07:46:21.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27549008400000002 2024-09-18T07:46:21.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.226265+0000", 2024-09-18T07:46:21.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00042664299999999999 2024-09-18T07:46:21.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.230756+0000", 2024-09-18T07:46:21.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044914270000000001 2024-09-18T07:46:21.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.231376+0000", 2024-09-18T07:46:21.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00061996299999999996 2024-09-18T07:46:21.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.231423+0000", 2024-09-18T07:46:21.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7142999999999999e-05 2024-09-18T07:46:21.776 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.776 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.776 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.777 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:215874 10.8 e434/364 10:1dfe7e41:::benchmark_data_smithi137_151160_object80058:head v 434'13582, mlcod=434'13579)", 2024-09-18T07:46:21.777 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.943918+0000", 2024-09-18T07:46:21.777 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.710076225, 2024-09-18T07:46:21.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27830592900000001, 2024-09-18T07:46:21.777 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.777 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.777 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.777 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.943918+0000", 2024-09-18T07:46:21.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.943918+0000", 2024-09-18T07:46:21.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.943919+0000", 2024-09-18T07:46:21.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.37e-06 2024-09-18T07:46:21.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.943925+0000", 2024-09-18T07:46:21.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1519999999999997e-06 2024-09-18T07:46:21.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.943935+0000", 2024-09-18T07:46:21.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4089999999999992e-06 2024-09-18T07:46:21.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.943941+0000", 2024-09-18T07:46:21.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7100000000000001e-06 2024-09-18T07:46:21.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.946723+0000", 2024-09-18T07:46:21.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027812449999999999 2024-09-18T07:46:21.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.946742+0000", 2024-09-18T07:46:21.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9154999999999999e-05 2024-09-18T07:46:21.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222177+0000", 2024-09-18T07:46:21.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27543490700000001 2024-09-18T07:46:21.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222224+0000", 2024-09-18T07:46:21.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6981000000000003e-05 2024-09-18T07:46:21.782 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.782 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.782 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.782 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:215876 10.9 10:940140eb:::benchmark_data_smithi137_151160_object80060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-18T07:46:21.782 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.943968+0000", 2024-09-18T07:46:21.782 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.710025729, 2024-09-18T07:46:21.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27805472599999997, 2024-09-18T07:46:21.782 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.782 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.782 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.782 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:21.783 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:46:21.783 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:21.783 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 215876 2024-09-18T07:46:21.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.783 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.943968+0000", 2024-09-18T07:46:21.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.943968+0000", 2024-09-18T07:46:21.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.943970+0000", 2024-09-18T07:46:21.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3629999999999999e-06 2024-09-18T07:46:21.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.943974+0000", 2024-09-18T07:46:21.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3449999999999999e-06 2024-09-18T07:46:21.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.943987+0000", 2024-09-18T07:46:21.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3081000000000001e-05 2024-09-18T07:46:21.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.943993+0000", 2024-09-18T07:46:21.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7710000000000003e-06 2024-09-18T07:46:21.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.944575+0000", 2024-09-18T07:46:21.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00058201100000000001 2024-09-18T07:46:21.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.945371+0000", 2024-09-18T07:46:21.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00079619900000000004 2024-09-18T07:46:21.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 7", 2024-09-18T07:46:21.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.945489+0000", 2024-09-18T07:46:21.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118115 2024-09-18T07:46:21.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:21.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.221720+0000", 2024-09-18T07:46:21.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.276231159 2024-09-18T07:46:21.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.221841+0000", 2024-09-18T07:46:21.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000121177 2024-09-18T07:46:21.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.221894+0000", 2024-09-18T07:46:21.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.274e-05 2024-09-18T07:46:21.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222023+0000", 2024-09-18T07:46:21.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000128765 2024-09-18T07:46:21.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.788 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.789 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:215878 10.e e434/419 10:7c2707e7:::benchmark_data_smithi137_151160_object80062:head v 434'13522, mlcod=434'13520)", 2024-09-18T07:46:21.789 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.945309+0000", 2024-09-18T07:46:21.789 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.708685337, 2024-09-18T07:46:21.789 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27683779400000003, 2024-09-18T07:46:21.789 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.789 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.789 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.789 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.945309+0000", 2024-09-18T07:46:21.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.945309+0000", 2024-09-18T07:46:21.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.945310+0000", 2024-09-18T07:46:21.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1599999999999999e-06 2024-09-18T07:46:21.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.945319+0000", 2024-09-18T07:46:21.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0180000000000007e-06 2024-09-18T07:46:21.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.945328+0000", 2024-09-18T07:46:21.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9700000000000005e-06 2024-09-18T07:46:21.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.792 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.792 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.945334+0000", 2024-09-18T07:46:21.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.801e-06 2024-09-18T07:46:21.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.792 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.792 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.945396+0000", 2024-09-18T07:46:21.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2806999999999997e-05 2024-09-18T07:46:21.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.945413+0000", 2024-09-18T07:46:21.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7169000000000001e-05 2024-09-18T07:46:21.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.951151+0000", 2024-09-18T07:46:21.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057375730000000002 2024-09-18T07:46:21.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.794 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222146+0000", 2024-09-18T07:46:21.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.270995296 2024-09-18T07:46:21.794 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.794 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.794 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.794 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:215879 10.5 10:ab42d4cc:::benchmark_data_smithi137_151160_object80063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-18T07:46:21.794 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.945993+0000", 2024-09-18T07:46:21.794 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.70800079, 2024-09-18T07:46:21.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.29180362700000001, 2024-09-18T07:46:21.795 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.795 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.795 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.795 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:21.795 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:46:21.796 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:21.796 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 215879 2024-09-18T07:46:21.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.796 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.945993+0000", 2024-09-18T07:46:21.796 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.945993+0000", 2024-09-18T07:46:21.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.945995+0000", 2024-09-18T07:46:21.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7439999999999999e-06 2024-09-18T07:46:21.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.946000+0000", 2024-09-18T07:46:21.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5230000000000003e-06 2024-09-18T07:46:21.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.946010+0000", 2024-09-18T07:46:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6460000000000005e-06 2024-09-18T07:46:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.946017+0000", 2024-09-18T07:46:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8179999999999998e-06 2024-09-18T07:46:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.226435+0000", 2024-09-18T07:46:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.280417729 2024-09-18T07:46:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.226635+0000", 2024-09-18T07:46:21.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020031500000000001 2024-09-18T07:46:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-09-18T07:46:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.226845+0000", 2024-09-18T07:46:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020990599999999999 2024-09-18T07:46:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.231434+0000", 2024-09-18T07:46:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045891969999999997 2024-09-18T07:46:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.802 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.234481+0000", 2024-09-18T07:46:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030471600000000001 2024-09-18T07:46:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.237249+0000", 2024-09-18T07:46:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027675820000000002 2024-09-18T07:46:21.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.237640+0000", 2024-09-18T07:46:21.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00039125100000000001 2024-09-18T07:46:21.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.237797+0000", 2024-09-18T07:46:21.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000156756 2024-09-18T07:46:21.805 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.805 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.805 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.805 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.36014.0:215861 10.5 e434/364 ondisk, result = 0)", 2024-09-18T07:46:21.805 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.947999+0000", 2024-09-18T07:46:21.805 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.705994412999999, 2024-09-18T07:46:21.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.277934126, 2024-09-18T07:46:21.805 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.805 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.806 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:46:21.806 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.947999+0000", 2024-09-18T07:46:21.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.947999+0000", 2024-09-18T07:46:21.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948001+0000", 2024-09-18T07:46:21.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.626e-06 2024-09-18T07:46:21.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948006+0000", 2024-09-18T07:46:21.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9150000000000004e-06 2024-09-18T07:46:21.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948016+0000", 2024-09-18T07:46:21.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6500000000000008e-06 2024-09-18T07:46:21.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948022+0000", 2024-09-18T07:46:21.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7689999999999999e-06 2024-09-18T07:46:21.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.225595+0000", 2024-09-18T07:46:21.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27757223600000003 2024-09-18T07:46:21.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.225602+0000", 2024-09-18T07:46:21.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5270000000000001e-06 2024-09-18T07:46:21.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.225934+0000", 2024-09-18T07:46:21.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00033140300000000003 2024-09-18T07:46:21.809 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.809 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.809 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.810 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:215880 10.f e434/364 10:f0e2a592:::benchmark_data_smithi137_151160_object80064:head v 434'13330, mlcod=434'13328)", 2024-09-18T07:46:21.810 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.948222+0000", 2024-09-18T07:46:21.810 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.705771725, 2024-09-18T07:46:21.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27389983099999998, 2024-09-18T07:46:21.810 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.810 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.810 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.810 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948222+0000", 2024-09-18T07:46:21.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948222+0000", 2024-09-18T07:46:21.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948224+0000", 2024-09-18T07:46:21.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3480000000000001e-06 2024-09-18T07:46:21.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948230+0000", 2024-09-18T07:46:21.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3929999999999996e-06 2024-09-18T07:46:21.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948239+0000", 2024-09-18T07:46:21.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3829999999999998e-06 2024-09-18T07:46:21.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948246+0000", 2024-09-18T07:46:21.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4130000000000003e-06 2024-09-18T07:46:21.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948313+0000", 2024-09-18T07:46:21.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7244999999999997e-05 2024-09-18T07:46:21.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948332+0000", 2024-09-18T07:46:21.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8819000000000001e-05 2024-09-18T07:46:21.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.221756+0000", 2024-09-18T07:46:21.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27342428800000002 2024-09-18T07:46:21.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222122+0000", 2024-09-18T07:46:21.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00036594200000000001 2024-09-18T07:46:21.814 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.814 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.814 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.815 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:215881 10.3 10:c08c44be:::benchmark_data_smithi137_151160_object80065:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-18T07:46:21.815 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.948701+0000", 2024-09-18T07:46:21.815 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.705293299000001, 2024-09-18T07:46:21.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28919155699999999, 2024-09-18T07:46:21.815 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:21.815 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:21.815 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:21.815 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:21.815 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:46:21.815 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:21.816 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 215881 2024-09-18T07:46:21.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.816 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:21.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:21.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948701+0000", 2024-09-18T07:46:21.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:21.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948701+0000", 2024-09-18T07:46:21.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:21.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:21.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948702+0000", 2024-09-18T07:46:21.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.686e-06 2024-09-18T07:46:21.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:21.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948707+0000", 2024-09-18T07:46:21.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9740000000000001e-06 2024-09-18T07:46:21.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:21.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948716+0000", 2024-09-18T07:46:21.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1290000000000002e-06 2024-09-18T07:46:21.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:21.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948724+0000", 2024-09-18T07:46:21.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6000000000000001e-06 2024-09-18T07:46:21.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:21.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948831+0000", 2024-09-18T07:46:21.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107495 2024-09-18T07:46:21.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.948976+0000", 2024-09-18T07:46:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000144807 2024-09-18T07:46:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-09-18T07:46:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949089+0000", 2024-09-18T07:46:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112242 2024-09-18T07:46:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:21.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.225523+0000", 2024-09-18T07:46:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27643435900000002 2024-09-18T07:46:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.236964+0000", 2024-09-18T07:46:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011441412 2024-09-18T07:46:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.237700+0000", 2024-09-18T07:46:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00073618100000000003 2024-09-18T07:46:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:21.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:21.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:21.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.237892+0000", 2024-09-18T07:46:21.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019167199999999999 2024-09-18T07:46:21.821 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.821 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.821 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.821 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:21.821 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:21.821 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:46:21.858 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:21.888 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:21.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:21.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:21.990 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:22.016 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:22.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:22.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:22.118 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:22.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:22.146 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:22.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:22.247 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:22.278 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:22.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:22.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:22.380 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:22.422 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:22.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:22.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:22.524 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:22.553 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:22.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:22.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:22.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:22.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:22 smithi137 ceph-mon[29408]: 5.9 scrub starts 2024-09-18T07:46:22.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:22 smithi137 ceph-mon[29408]: 5.9 scrub ok 2024-09-18T07:46:22.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:22 smithi137 ceph-mon[29408]: 3.1b scrub starts 2024-09-18T07:46:22.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:22 smithi137 ceph-mon[29408]: 3.1b scrub ok 2024-09-18T07:46:22.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:22 smithi137 ceph-mon[39175]: 5.9 scrub starts 2024-09-18T07:46:22.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:22 smithi137 ceph-mon[39175]: 5.9 scrub ok 2024-09-18T07:46:22.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:22 smithi137 ceph-mon[39175]: 3.1b scrub starts 2024-09-18T07:46:22.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:22 smithi137 ceph-mon[39175]: 3.1b scrub ok 2024-09-18T07:46:22.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:22.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:22.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:22.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:22 smithi203 ceph-mon[32237]: 5.9 scrub starts 2024-09-18T07:46:22.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:22 smithi203 ceph-mon[32237]: 5.9 scrub ok 2024-09-18T07:46:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:22 smithi203 ceph-mon[32237]: 3.1b scrub starts 2024-09-18T07:46:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:22 smithi203 ceph-mon[32237]: 3.1b scrub ok 2024-09-18T07:46:22.793 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:22.823 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:22.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:22.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:22.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T07:46:22.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:22.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:22.976 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:22.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:23.077 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:23.118 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:23.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:23.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:23.220 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:23.254 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:23.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:23.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:23.356 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:23.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:46:23.391 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:23.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:23.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:23.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:23.537 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:23 smithi137 ceph-mon[29408]: pgmap v815: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 977 op/s; 14356/133089 objects degraded (10.787%) 2024-09-18T07:46:23.537 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:23 smithi137 ceph-mon[39175]: pgmap v815: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 977 op/s; 14356/133089 objects degraded (10.787%) 2024-09-18T07:46:23.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:23.555 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:23.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:23.656 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:23.687 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:23.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:23.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:23 smithi203 ceph-mon[32237]: pgmap v815: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 977 op/s; 14356/133089 objects degraded (10.787%) 2024-09-18T07:46:23.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:23.830 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:23.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:23.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:23.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:23.964 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:23.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:23.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:24.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:24.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:24.131 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:24.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:24.182 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:24.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:24.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:24.307 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:24.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:24.340 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 6, 7, 4] 2024-09-18T07:46:24.340 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:46:24.341 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-18T07:46:24.341 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.718290 2024-09-18T07:46:24.341 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 0.7182900977677137 2024-09-18T07:46:24.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:24.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:24.487 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:24.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:24.578 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:24 smithi137 ceph-mon[29408]: 10.d scrub starts 2024-09-18T07:46:24.579 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:24 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:46:24.579 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:24 smithi137 ceph-mon[29408]: 10.d scrub ok 2024-09-18T07:46:24.579 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:24 smithi137 ceph-mon[39175]: 10.d scrub starts 2024-09-18T07:46:24.579 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:24 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:46:24.579 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:24 smithi137 ceph-mon[39175]: 10.d scrub ok 2024-09-18T07:46:24.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:24.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:24.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:24.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:24 smithi203 ceph-mon[32237]: 10.d scrub starts 2024-09-18T07:46:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:24 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:46:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:24 smithi203 ceph-mon[32237]: 10.d scrub ok 2024-09-18T07:46:24.759 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:24.795 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:24.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:24.847 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:24.890 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:46:24.890 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:46:24.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.891 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:235488 10.7 e437/416 10:edb4505c:::benchmark_data_smithi137_151160_object99672:head v 437'14844, mlcod=437'14841)", 2024-09-18T07:46:24.891 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:24.868501+0000", 2024-09-18T07:46:24.891 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.016839646999999999, 2024-09-18T07:46:24.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016866716, 2024-09-18T07:46:24.891 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:24.891 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:24.891 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:46:24.891 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:24.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:24.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.868501+0000", 2024-09-18T07:46:24.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:24.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:24.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.868501+0000", 2024-09-18T07:46:24.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:24.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:24.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.868502+0000", 2024-09-18T07:46:24.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1289999999999999e-06 2024-09-18T07:46:24.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:24.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.868508+0000", 2024-09-18T07:46:24.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7189999999999998e-06 2024-09-18T07:46:24.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:24.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.868521+0000", 2024-09-18T07:46:24.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3151e-05 2024-09-18T07:46:24.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:24.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.868528+0000", 2024-09-18T07:46:24.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9310000000000002e-06 2024-09-18T07:46:24.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:24.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.872203+0000", 2024-09-18T07:46:24.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036752809999999999 2024-09-18T07:46:24.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:24.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.872255+0000", 2024-09-18T07:46:24.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2048e-05 2024-09-18T07:46:24.894 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:24.894 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:24.894 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:24.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.895 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:235489 10.f e437/364 10:fed33ef2:::benchmark_data_smithi137_151160_object99673:head v 437'14518, mlcod=437'14515)", 2024-09-18T07:46:24.895 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:24.868591+0000", 2024-09-18T07:46:24.895 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.016750238000000001, 2024-09-18T07:46:24.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016833233999999999, 2024-09-18T07:46:24.895 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:24.895 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:24.895 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:46:24.895 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:24.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:24.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.868591+0000", 2024-09-18T07:46:24.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:24.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:24.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.868591+0000", 2024-09-18T07:46:24.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:24.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:24.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.868591+0000", 2024-09-18T07:46:24.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0500000000000002e-07 2024-09-18T07:46:24.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:24.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.868596+0000", 2024-09-18T07:46:24.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9590000000000003e-06 2024-09-18T07:46:24.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:24.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.868610+0000", 2024-09-18T07:46:24.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3824e-05 2024-09-18T07:46:24.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:24.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.868622+0000", 2024-09-18T07:46:24.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2211999999999999e-05 2024-09-18T07:46:24.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:24.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.879854+0000", 2024-09-18T07:46:24.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011231362 2024-09-18T07:46:24.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:24.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.879874+0000", 2024-09-18T07:46:24.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9998000000000001e-05 2024-09-18T07:46:24.898 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:24.898 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:24.898 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:24.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.899 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:235495 10.f e437/364 10:f83979aa:::benchmark_data_smithi137_151160_object99679:head v 437'14519, mlcod=437'14515)", 2024-09-18T07:46:24.899 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:24.871538+0000", 2024-09-18T07:46:24.899 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.013802566, 2024-09-18T07:46:24.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013933667, 2024-09-18T07:46:24.899 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:24.899 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:24.899 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:46:24.899 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:24.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:24.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.871538+0000", 2024-09-18T07:46:24.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:24.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:24.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.871538+0000", 2024-09-18T07:46:24.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:24.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:24.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.871540+0000", 2024-09-18T07:46:24.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5689999999999999e-06 2024-09-18T07:46:24.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:24.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.871548+0000", 2024-09-18T07:46:24.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7670000000000003e-06 2024-09-18T07:46:24.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:24.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.871564+0000", 2024-09-18T07:46:24.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6013000000000001e-05 2024-09-18T07:46:24.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:24.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.871572+0000", 2024-09-18T07:46:24.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3699999999999995e-06 2024-09-18T07:46:24.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:24.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.883103+0000", 2024-09-18T07:46:24.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01153159 2024-09-18T07:46:24.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:24.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.883125+0000", 2024-09-18T07:46:24.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1379999999999999e-05 2024-09-18T07:46:24.902 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:24.902 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:24.902 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:24.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.902 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:235487 10.5 e437/364 10:a3cb2c70:::benchmark_data_smithi137_151160_object99671:head v 437'14686, mlcod=437'14683)", 2024-09-18T07:46:24.902 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:24.874654+0000", 2024-09-18T07:46:24.902 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.010686793999999999, 2024-09-18T07:46:24.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010865150000000001, 2024-09-18T07:46:24.903 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:24.903 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:24.903 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:46:24.903 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:24.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:24.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.874654+0000", 2024-09-18T07:46:24.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:24.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:24.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.874654+0000", 2024-09-18T07:46:24.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:24.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:24.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.874655+0000", 2024-09-18T07:46:24.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0900000000000001e-07 2024-09-18T07:46:24.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:24.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.874660+0000", 2024-09-18T07:46:24.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9960000000000001e-06 2024-09-18T07:46:24.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:24.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.874666+0000", 2024-09-18T07:46:24.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5309999999999998e-06 2024-09-18T07:46:24.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:24.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.874672+0000", 2024-09-18T07:46:24.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7200000000000003e-06 2024-09-18T07:46:24.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:24.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.875948+0000", 2024-09-18T07:46:24.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012762299999999999 2024-09-18T07:46:24.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:24.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.876009+0000", 2024-09-18T07:46:24.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0578999999999997e-05 2024-09-18T07:46:24.906 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:24.906 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:24.906 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:24.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.906 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:235497 10.f e437/364)", 2024-09-18T07:46:24.906 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:24.879428+0000", 2024-09-18T07:46:24.906 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0059123709999999996, 2024-09-18T07:46:24.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0061348410000000003, 2024-09-18T07:46:24.906 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:24.906 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:24.907 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:46:24.907 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:24.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:24.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.879428+0000", 2024-09-18T07:46:24.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:24.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:24.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.879428+0000", 2024-09-18T07:46:24.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:24.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:24.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.879430+0000", 2024-09-18T07:46:24.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7680000000000001e-06 2024-09-18T07:46:24.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:24.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.879439+0000", 2024-09-18T07:46:24.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7499999999999992e-06 2024-09-18T07:46:24.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:24.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.879451+0000", 2024-09-18T07:46:24.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2313000000000001e-05 2024-09-18T07:46:24.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:24.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.879459+0000", 2024-09-18T07:46:24.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8620000000000003e-06 2024-09-18T07:46:24.909 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:24.909 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:24.909 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:24.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.909 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:235493 10.5 e437/364 10:aee92054:::benchmark_data_smithi137_151160_object99677:head v 437'14687, mlcod=437'14683)", 2024-09-18T07:46:24.909 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:24.884917+0000", 2024-09-18T07:46:24.910 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.00042344399999999998, 2024-09-18T07:46:24.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00068318700000000003, 2024-09-18T07:46:24.910 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:24.910 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:24.910 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:46:24.910 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:24.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:24.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.884917+0000", 2024-09-18T07:46:24.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:24.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:24.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.884917+0000", 2024-09-18T07:46:24.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:24.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:24.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.884919+0000", 2024-09-18T07:46:24.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6500000000000001e-06 2024-09-18T07:46:24.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:24.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.884928+0000", 2024-09-18T07:46:24.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2180000000000005e-06 2024-09-18T07:46:24.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:24.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.884938+0000", 2024-09-18T07:46:24.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0249e-05 2024-09-18T07:46:24.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:24.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.884958+0000", 2024-09-18T07:46:24.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9238000000000001e-05 2024-09-18T07:46:24.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:24.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.885195+0000", 2024-09-18T07:46:24.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023769399999999999 2024-09-18T07:46:24.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:24.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.885247+0000", 2024-09-18T07:46:24.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1434000000000003e-05 2024-09-18T07:46:24.913 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:24.913 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:24.913 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:24.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.913 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:235484 10.f e437/364 10:f6beefb7:::benchmark_data_smithi137_151160_object99668:head v 437'14517, mlcod=437'14513)", 2024-09-18T07:46:24.913 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:24.864192+0000", 2024-09-18T07:46:24.914 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.02114893, 2024-09-18T07:46:24.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021455094000000001, 2024-09-18T07:46:24.914 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:24.914 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:24.914 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:46:24.914 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:24.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:24.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.864192+0000", 2024-09-18T07:46:24.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:24.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:24.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.864192+0000", 2024-09-18T07:46:24.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:24.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:24.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.864193+0000", 2024-09-18T07:46:24.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0759999999999999e-06 2024-09-18T07:46:24.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:24.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.864199+0000", 2024-09-18T07:46:24.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2169999999999999e-06 2024-09-18T07:46:24.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:24.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.864207+0000", 2024-09-18T07:46:24.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0490000000000004e-06 2024-09-18T07:46:24.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:24.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.864213+0000", 2024-09-18T07:46:24.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1689999999999997e-06 2024-09-18T07:46:24.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:24.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.867916+0000", 2024-09-18T07:46:24.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0037030090000000002 2024-09-18T07:46:24.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:24.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.867957+0000", 2024-09-18T07:46:24.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1142e-05 2024-09-18T07:46:24.917 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:24.917 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:24.917 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:24.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.917 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:235486 10.7 e437/416 10:eec9af1a:::benchmark_data_smithi137_151160_object99670:head v 437'14843, mlcod=437'14840)", 2024-09-18T07:46:24.917 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:24.867712+0000", 2024-09-18T07:46:24.917 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.017629028000000001, 2024-09-18T07:46:24.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01798046, 2024-09-18T07:46:24.918 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:24.918 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:24.918 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:46:24.918 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:24.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:24.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.867712+0000", 2024-09-18T07:46:24.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:24.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:24.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.867712+0000", 2024-09-18T07:46:24.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:24.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:24.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.867713+0000", 2024-09-18T07:46:24.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4449999999999999e-06 2024-09-18T07:46:24.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:24.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.867723+0000", 2024-09-18T07:46:24.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4299999999999995e-06 2024-09-18T07:46:24.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:24.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.867739+0000", 2024-09-18T07:46:24.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5987000000000001e-05 2024-09-18T07:46:24.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:24.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.867748+0000", 2024-09-18T07:46:24.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6579999999999997e-06 2024-09-18T07:46:24.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:24.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.868178+0000", 2024-09-18T07:46:24.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00042989500000000002 2024-09-18T07:46:24.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:24.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:24.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:24.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.868200+0000", 2024-09-18T07:46:24.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1699999999999999e-05 2024-09-18T07:46:24.921 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:24.921 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:24.921 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:24.921 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:24.921 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:46:24.921 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 8 2024-09-18T07:46:24.921 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:46:24.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:24.970 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:24.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:24.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:25.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:25.110 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:25.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:25.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:25.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:25.245 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:25.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:25.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:25.287 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:46:25.347 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:25.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:25.383 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:25.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:25.484 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:25.515 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:25.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:25.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:25.583 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[29408]: pgmap v816: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 12329/118584 objects degraded (10.397%) 2024-09-18T07:46:25.583 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:46:25.583 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:46:25.583 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:46:25.583 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4090630216' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:46:25.583 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[29408]: 1.9 scrub starts 2024-09-18T07:46:25.583 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[29408]: 1.9 scrub ok 2024-09-18T07:46:25.584 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[29408]: 4.9 scrub starts 2024-09-18T07:46:25.584 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[29408]: 4.9 scrub ok 2024-09-18T07:46:25.584 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 12329/118584 objects degraded (10.397%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:25.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[39175]: pgmap v816: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 12329/118584 objects degraded (10.397%) 2024-09-18T07:46:25.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:46:25.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:46:25.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:46:25.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4090630216' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:46:25.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[39175]: 1.9 scrub starts 2024-09-18T07:46:25.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[39175]: 1.9 scrub ok 2024-09-18T07:46:25.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[39175]: 4.9 scrub starts 2024-09-18T07:46:25.585 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[39175]: 4.9 scrub ok 2024-09-18T07:46:25.585 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:25 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 12329/118584 objects degraded (10.397%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:25.617 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:25.651 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:25.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:25.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:25 smithi203 ceph-mon[32237]: pgmap v816: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 12329/118584 objects degraded (10.397%) 2024-09-18T07:46:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:25 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:46:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:25 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:46:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:25 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:46:25.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:25 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4090630216' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:46:25.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:25 smithi203 ceph-mon[32237]: 1.9 scrub starts 2024-09-18T07:46:25.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:25 smithi203 ceph-mon[32237]: 1.9 scrub ok 2024-09-18T07:46:25.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:25 smithi203 ceph-mon[32237]: 4.9 scrub starts 2024-09-18T07:46:25.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:25 smithi203 ceph-mon[32237]: 4.9 scrub ok 2024-09-18T07:46:25.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:25 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 12329/118584 objects degraded (10.397%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:25.753 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:25.783 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:25.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:25.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:25.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T07:46:25.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:25.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:25.957 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:25.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:26.058 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:26.084 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:26.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:26.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:26.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:26.226 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:26.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:26.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:26.285 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:26.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:26.358 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:26.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:26.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:26.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:26.495 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:26.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:26.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:26 smithi137 ceph-mon[29408]: 3.8 scrub starts 2024-09-18T07:46:26.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:26 smithi137 ceph-mon[29408]: 3.8 scrub ok 2024-09-18T07:46:26.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:26 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:26.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:26 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4090630216' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:46:26.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:26 smithi137 ceph-mon[29408]: osdmap e438: 8 total, 7 up, 8 in 2024-09-18T07:46:26.554 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:26 smithi137 ceph-mon[39175]: 3.8 scrub starts 2024-09-18T07:46:26.555 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:26 smithi137 ceph-mon[39175]: 3.8 scrub ok 2024-09-18T07:46:26.555 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:26 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:26.555 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:26 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4090630216' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:46:26.555 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:26 smithi137 ceph-mon[39175]: osdmap e438: 8 total, 7 up, 8 in 2024-09-18T07:46:26.598 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:26.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:26.632 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:26.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:26.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:26 smithi203 ceph-mon[32237]: 3.8 scrub starts 2024-09-18T07:46:26.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:26 smithi203 ceph-mon[32237]: 3.8 scrub ok 2024-09-18T07:46:26.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:26 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:26.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:26 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4090630216' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:46:26.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:26 smithi203 ceph-mon[32237]: osdmap e438: 8 total, 7 up, 8 in 2024-09-18T07:46:26.733 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:26.760 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:26.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:26.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:26.862 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:26.889 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:26.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:26.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:26.971 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:46:26.971 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:46:26.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:27.011 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:46:27.012 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:46:27.025 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:27.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:27.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:27.052 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:46:27.052 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:46:27.084 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:46:27.084 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:46:27.120 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:46:27.121 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:46:27.125 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-18T07:46:27.127 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:27.167 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:46:27.167 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:46:27.172 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:27.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:27.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:27.203 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:46:27.203 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:46:27.274 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:27.301 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:27.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:27.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:27.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:27.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:27 smithi137 ceph-mon[29408]: pgmap v818: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 944 op/s; 12329/118584 objects degraded (10.397%) 2024-09-18T07:46:27.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:27 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1944432359' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:27.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:27 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:46:27.421 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:46:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:46:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:46:27.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:27 smithi137 ceph-mon[39175]: pgmap v818: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 944 op/s; 12329/118584 objects degraded (10.397%) 2024-09-18T07:46:27.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:27 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1944432359' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:27.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:27 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:46:27.441 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:27.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:27.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:27.543 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:27.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:27.589 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:27.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:27.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:27.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:27 smithi203 ceph-mon[32237]: pgmap v818: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 944 op/s; 12329/118584 objects degraded (10.397%) 2024-09-18T07:46:27.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:27 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1944432359' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:27.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:27 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:46:27.726 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:27.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:27.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:27.828 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:27.854 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:27.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:27.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:27.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:27.987 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:27.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:27.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:28.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:28.127 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:28.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:28.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:28.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:46:27 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:46:28.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:28.264 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:28.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:28.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:28.308 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.0 primary-affinity to 0.71829 (8470732) 2024-09-18T07:46:28.366 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:28.393 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:28.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:28.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:28.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:28.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:28.517 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:28.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:28.618 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:28.639 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:28.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:28.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:28.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:28 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/942186131' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.7182900977677137} : dispatch 2024-09-18T07:46:28.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:28 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/942186131' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.7182900977677137} : dispatch 2024-09-18T07:46:28.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:28 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/942186131' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.7182900977677137} : dispatch 2024-09-18T07:46:28.741 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:28.762 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:28.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:28.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:28.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:28.886 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:28.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:28.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:28.896 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:46:28.897 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:46:28.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:29.009 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:29.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:29.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:29.111 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:29.134 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:29.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:29.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:29.235 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:29.257 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:29.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:29.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:29.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:29.383 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:29.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:29.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:29.485 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:29.508 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:29 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:29.508 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:29 smithi137 ceph-mon[29408]: pgmap v819: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 10923/104208 objects degraded (10.482%) 2024-09-18T07:46:29.508 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:29 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/942186131' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.7182900977677137}]': finished 2024-09-18T07:46:29.508 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:29 smithi137 ceph-mon[29408]: osdmap e439: 8 total, 7 up, 8 in 2024-09-18T07:46:29.508 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:29 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:29.508 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:29 smithi137 ceph-mon[39175]: pgmap v819: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 10923/104208 objects degraded (10.482%) 2024-09-18T07:46:29.508 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:29 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/942186131' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.7182900977677137}]': finished 2024-09-18T07:46:29.509 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:29 smithi137 ceph-mon[39175]: osdmap e439: 8 total, 7 up, 8 in 2024-09-18T07:46:29.511 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:29.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:29.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:29.613 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:29.643 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:29.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:29.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:29 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:29 smithi203 ceph-mon[32237]: pgmap v819: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 10923/104208 objects degraded (10.482%) 2024-09-18T07:46:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:29 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/942186131' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.7182900977677137}]': finished 2024-09-18T07:46:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:29 smithi203 ceph-mon[32237]: osdmap e439: 8 total, 7 up, 8 in 2024-09-18T07:46:29.744 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:29.783 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:29.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:29.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:29.885 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:29.913 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:29.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:29.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:29.963 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:30.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:30.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:30.061 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:30.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:30.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:30.201 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:30.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:30.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:30.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:30.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:30.367 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:30.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:30.468 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:30.494 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:30.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:30.584 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:30 smithi137 ceph-mon[29408]: osdmap e440: 8 total, 7 up, 8 in 2024-09-18T07:46:30.584 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:30 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 10923/104208 objects degraded (10.482%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:30.585 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:30 smithi137 ceph-mon[29408]: osdmap e441: 8 total, 7 up, 8 in 2024-09-18T07:46:30.585 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:30 smithi137 ceph-mon[39175]: osdmap e440: 8 total, 7 up, 8 in 2024-09-18T07:46:30.585 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:30 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 10923/104208 objects degraded (10.482%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:30.585 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:30 smithi137 ceph-mon[39175]: osdmap e441: 8 total, 7 up, 8 in 2024-09-18T07:46:30.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:30.619 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:46:30.620 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:46:30.637 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:30.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:30.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:30.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:30 smithi203 ceph-mon[32237]: osdmap e440: 8 total, 7 up, 8 in 2024-09-18T07:46:30.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:30 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 10923/104208 objects degraded (10.482%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:30.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:30 smithi203 ceph-mon[32237]: osdmap e441: 8 total, 7 up, 8 in 2024-09-18T07:46:30.738 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:30.767 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:30.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:30.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:30.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:30.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:46:30.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:30.905 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:30.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:31.006 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:31.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:31.045 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:31.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:31.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:31.189 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:31.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:31.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:31.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:31.333 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:31.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:31.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:31.435 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:31.468 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:31.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:31.540 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T07:46:31.571 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:31.576 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:31 smithi137 ceph-mon[29408]: pgmap v822: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 988 op/s; 10572/100806 objects degraded (10.487%) 2024-09-18T07:46:31.576 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:31 smithi137 ceph-mon[39175]: pgmap v822: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 988 op/s; 10572/100806 objects degraded (10.487%) 2024-09-18T07:46:31.597 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:31.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:31.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:31.684 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:31.699 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:31.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:31 smithi203 ceph-mon[32237]: pgmap v822: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 988 op/s; 10572/100806 objects degraded (10.487%) 2024-09-18T07:46:31.723 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:31.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:31.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:31.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:31.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:31.875 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:31.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:31.965 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:46:31.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:32.061 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:32.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:32.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:32.163 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:32.200 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:32.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:32.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:32.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:32.358 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:46:32.358 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:46:32.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:32.361 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:32.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:32.384 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:32.396 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:46:32.396 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:46:32.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:32 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/680739689' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:32.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:32 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/680739689' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:32.427 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:46:32.428 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:46:32.457 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:46:32.458 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:46:32.462 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:32.472 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:32.489 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:46:32.489 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:46:32.491 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:32.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:32.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:32.493 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-18T07:46:32.526 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:46:32.526 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:46:32.557 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:46:32.557 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:46:32.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:32.627 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:32.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:32.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:32.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:32 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/680739689' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:32.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:32.767 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:32.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:32.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:32.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:32.904 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:32.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:33.008 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:33.033 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:33.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:33.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:33.135 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:33.161 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:33.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:33.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:33.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:33.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:33.296 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:33.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:33.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:33.463 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:33.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:33.564 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:33.596 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:33.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:33.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:33.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:33 smithi137 ceph-mon[29408]: pgmap v824: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.39k op/s; 9927/93573 objects degraded (10.609%); 0 B/s, 0 objects/s recovering 2024-09-18T07:46:33.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:33 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:46:33.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:33 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:46:33.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:33 smithi137 ceph-mon[39175]: pgmap v824: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.39k op/s; 9927/93573 objects degraded (10.609%); 0 B/s, 0 objects/s recovering 2024-09-18T07:46:33.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:33 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:46:33.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:33 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:46:33.698 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:33.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:33 smithi203 ceph-mon[32237]: pgmap v824: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.39k op/s; 9927/93573 objects degraded (10.609%); 0 B/s, 0 objects/s recovering 2024-09-18T07:46:33.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:33 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:46:33.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:33 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:46:33.725 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:33.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:33.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:33.828 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:33.856 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:33.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:33.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:33.958 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:33.984 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:33.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:33.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:34.086 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:34.116 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:34.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:34.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:34.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:34.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:34.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:34.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:34.351 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:34.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:34.387 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:34.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:34.488 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:34.510 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:34.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:34.554 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 6, 7, 4] 2024-09-18T07:46:34.555 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:46:34.555 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-18T07:46:34.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd out 1 2024-09-18T07:46:34.611 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:34.637 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:34.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:34.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:34.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:34 smithi137 ceph-mon[29408]: osdmap e442: 8 total, 7 up, 8 in 2024-09-18T07:46:34.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:34 smithi137 ceph-mon[29408]: pgmap v826: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 897 op/s; 8969/86022 objects degraded (10.426%) 2024-09-18T07:46:34.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:34 smithi137 ceph-mon[39175]: osdmap e442: 8 total, 7 up, 8 in 2024-09-18T07:46:34.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:34 smithi137 ceph-mon[39175]: pgmap v826: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 897 op/s; 8969/86022 objects degraded (10.426%) 2024-09-18T07:46:34.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:34 smithi203 ceph-mon[32237]: osdmap e442: 8 total, 7 up, 8 in 2024-09-18T07:46:34.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:34 smithi203 ceph-mon[32237]: pgmap v826: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 897 op/s; 8969/86022 objects degraded (10.426%) 2024-09-18T07:46:34.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:34.768 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:34.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:34.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:34.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:34.894 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:46:34.894 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:46:34.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:46:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:46:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109537 8.7 8:e50487d6:::benchmark_data_smithi137_101942_object4727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:46:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:46:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 532.01593673000002, 2024-09-18T07:46:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.185266763, 2024-09-18T07:46:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:46:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:46:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109537 2024-09-18T07:46:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:46:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056406+0000", 2024-09-18T07:46:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056408+0000", 2024-09-18T07:46:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.562e-06 2024-09-18T07:46:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056415+0000", 2024-09-18T07:46:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9280000000000004e-06 2024-09-18T07:46:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056425+0000", 2024-09-18T07:46:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0220999999999999e-05 2024-09-18T07:46:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.056434+0000", 2024-09-18T07:46:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7499999999999992e-06 2024-09-18T07:46:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.061190+0000", 2024-09-18T07:46:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047560620000000001 2024-09-18T07:46:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:46:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.061238+0000", 2024-09-18T07:46:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8730000000000003e-05 2024-09-18T07:46:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229082+0000", 2024-09-18T07:46:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.167843878 2024-09-18T07:46:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229243+0000", 2024-09-18T07:46:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000160426 2024-09-18T07:46:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:46:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229373+0000", 2024-09-18T07:46:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012994999999999999 2024-09-18T07:46:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236041+0000", 2024-09-18T07:46:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066680960000000001 2024-09-18T07:46:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236051+0000", 2024-09-18T07:46:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8309999999999996e-06 2024-09-18T07:46:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.240107+0000", 2024-09-18T07:46:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040564570000000003 2024-09-18T07:46:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.240152+0000", 2024-09-18T07:46:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4617000000000003e-05 2024-09-18T07:46:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.241673+0000", 2024-09-18T07:46:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001521255 2024-09-18T07:46:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109540 8.7 8:e7454470:::benchmark_data_smithi137_101942_object4730:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:46:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:46:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 532.00646473999996, 2024-09-18T07:46:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.186834677, 2024-09-18T07:46:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:46:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:46:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109540 2024-09-18T07:46:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:46:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065878+0000", 2024-09-18T07:46:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065880+0000", 2024-09-18T07:46:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8279999999999999e-06 2024-09-18T07:46:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065886+0000", 2024-09-18T07:46:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.117e-06 2024-09-18T07:46:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065899+0000", 2024-09-18T07:46:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2809999999999999e-05 2024-09-18T07:46:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065904+0000", 2024-09-18T07:46:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0350000000000001e-06 2024-09-18T07:46:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.066028+0000", 2024-09-18T07:46:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012424199999999999 2024-09-18T07:46:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:46:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.066039+0000", 2024-09-18T07:46:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0441999999999999e-05 2024-09-18T07:46:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.233503+0000", 2024-09-18T07:46:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.167464622 2024-09-18T07:46:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.233624+0000", 2024-09-18T07:46:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012107 2024-09-18T07:46:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:46:34.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.233749+0000", 2024-09-18T07:46:34.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000124418 2024-09-18T07:46:34.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:34.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236078+0000", 2024-09-18T07:46:34.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002328746 2024-09-18T07:46:34.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:34.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.239968+0000", 2024-09-18T07:46:34.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038901220000000002 2024-09-18T07:46:34.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:34.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.250395+0000", 2024-09-18T07:46:34.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010427532999999999 2024-09-18T07:46:34.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.250537+0000", 2024-09-18T07:46:34.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014218100000000001 2024-09-18T07:46:34.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.252713+0000", 2024-09-18T07:46:34.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0021755110000000002 2024-09-18T07:46:34.914 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.914 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.914 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.915 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109545 8.7 8:ecb41cf4:::benchmark_data_smithi137_101942_object4735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:46:34.915 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:46:34.915 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 531.99805995400004, 2024-09-18T07:46:34.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1804624180000001, 2024-09-18T07:46:34.915 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.915 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.915 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.915 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:34.915 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:46:34.916 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:46:34.916 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109545 2024-09-18T07:46:34.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.916 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:46:34.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074283+0000", 2024-09-18T07:46:34.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074285+0000", 2024-09-18T07:46:34.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1169999999999998e-06 2024-09-18T07:46:34.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074292+0000", 2024-09-18T07:46:34.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.153e-06 2024-09-18T07:46:34.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074305+0000", 2024-09-18T07:46:34.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2363e-05 2024-09-18T07:46:34.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074312+0000", 2024-09-18T07:46:34.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3409999999999997e-06 2024-09-18T07:46:34.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074508+0000", 2024-09-18T07:46:34.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000196443 2024-09-18T07:46:34.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:46:34.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.074519+0000", 2024-09-18T07:46:34.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0159999999999999e-05 2024-09-18T07:46:34.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236329+0000", 2024-09-18T07:46:34.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1618104599999999 2024-09-18T07:46:34.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236416+0000", 2024-09-18T07:46:34.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6670999999999997e-05 2024-09-18T07:46:34.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:46:34.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.236495+0000", 2024-09-18T07:46:34.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9256000000000001e-05 2024-09-18T07:46:34.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:34.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.241684+0000", 2024-09-18T07:46:34.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051890720000000003 2024-09-18T07:46:34.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:34.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.243812+0000", 2024-09-18T07:46:34.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002127912 2024-09-18T07:46:34.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:34.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.252772+0000", 2024-09-18T07:46:34.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.008960562 2024-09-18T07:46:34.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.922 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.252827+0000", 2024-09-18T07:46:34.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4500000000000003e-05 2024-09-18T07:46:34.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.922 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.254745+0000", 2024-09-18T07:46:34.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019184079999999999 2024-09-18T07:46:34.922 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.922 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.922 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.922 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109567 8.7 8:e247f5cd:::benchmark_data_smithi137_101942_object4757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:46:34.923 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:46:34.923 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 531.96896079099997, 2024-09-18T07:46:34.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.153246698, 2024-09-18T07:46:34.923 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.923 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.923 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.923 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:46:34.923 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:46:34.923 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:46:34.923 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109567 2024-09-18T07:46:34.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.923 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.924 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.924 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:46:34.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103382+0000", 2024-09-18T07:46:34.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103384+0000", 2024-09-18T07:46:34.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.792e-06 2024-09-18T07:46:34.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103390+0000", 2024-09-18T07:46:34.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7139999999999998e-06 2024-09-18T07:46:34.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103401+0000", 2024-09-18T07:46:34.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1559000000000001e-05 2024-09-18T07:46:34.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103408+0000", 2024-09-18T07:46:34.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2110000000000001e-06 2024-09-18T07:46:34.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103464+0000", 2024-09-18T07:46:34.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6032999999999997e-05 2024-09-18T07:46:34.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:46:34.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.103475+0000", 2024-09-18T07:46:34.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0380999999999999e-05 2024-09-18T07:46:34.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.238189+0000", 2024-09-18T07:46:34.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.134713723 2024-09-18T07:46:34.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.238319+0000", 2024-09-18T07:46:34.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013080700000000001 2024-09-18T07:46:34.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,6", 2024-09-18T07:46:34.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.238431+0000", 2024-09-18T07:46:34.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112126 2024-09-18T07:46:34.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:46:34.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.243830+0000", 2024-09-18T07:46:34.929 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053980390000000003 2024-09-18T07:46:34.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.929 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:34.929 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.243889+0000", 2024-09-18T07:46:34.929 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9024000000000003e-05 2024-09-18T07:46:34.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.929 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:34.929 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.254832+0000", 2024-09-18T07:46:34.929 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0109436 2024-09-18T07:46:34.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.255025+0000", 2024-09-18T07:46:34.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000192926 2024-09-18T07:46:34.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.256629+0000", 2024-09-18T07:46:34.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001603763 2024-09-18T07:46:34.930 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.930 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.930 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.931 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:126833 10.f e391/364 10:f8e7c411:::benchmark_data_smithi137_151160_object126832:head v 391'7824, mlcod=391'7821)", 2024-09-18T07:46:34.931 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:32.541463+0000", 2024-09-18T07:46:34.931 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 121.530879412, 2024-09-18T07:46:34.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.042615444000000002, 2024-09-18T07:46:34.931 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.931 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.931 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.931 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.931 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.931 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.541463+0000", 2024-09-18T07:46:34.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.932 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.932 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.541463+0000", 2024-09-18T07:46:34.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.932 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.932 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.541465+0000", 2024-09-18T07:46:34.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.674e-06 2024-09-18T07:46:34.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.541509+0000", 2024-09-18T07:46:34.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4079000000000003e-05 2024-09-18T07:46:34.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.541526+0000", 2024-09-18T07:46:34.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.683e-05 2024-09-18T07:46:34.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.541535+0000", 2024-09-18T07:46:34.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.054e-06 2024-09-18T07:46:34.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.553292+0000", 2024-09-18T07:46:34.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011757327 2024-09-18T07:46:34.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.553313+0000", 2024-09-18T07:46:34.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0783000000000001e-05 2024-09-18T07:46:34.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.583835+0000", 2024-09-18T07:46:34.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.03052148 2024-09-18T07:46:34.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.935 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.584079+0000", 2024-09-18T07:46:34.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024421700000000002 2024-09-18T07:46:34.935 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.935 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.935 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.935 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:126912 10.3 e391/364 10:cc218fc4:::benchmark_data_smithi137_151160_object126911:head v 391'8040, mlcod=391'8038)", 2024-09-18T07:46:34.936 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:32.618429+0000", 2024-09-18T07:46:34.936 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 121.45391386199999, 2024-09-18T07:46:34.936 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.041096705999999997, 2024-09-18T07:46:34.936 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.936 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.936 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.936 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.936 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.936 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.936 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.618429+0000", 2024-09-18T07:46:34.936 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.936 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.936 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.936 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.937 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.618429+0000", 2024-09-18T07:46:34.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.937 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.618431+0000", 2024-09-18T07:46:34.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.984e-06 2024-09-18T07:46:34.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.937 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.618460+0000", 2024-09-18T07:46:34.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9215000000000001e-05 2024-09-18T07:46:34.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.938 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.938 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.618475+0000", 2024-09-18T07:46:34.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4415e-05 2024-09-18T07:46:34.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.938 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.938 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.618484+0000", 2024-09-18T07:46:34.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5389999999999996e-06 2024-09-18T07:46:34.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.938 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.938 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.623350+0000", 2024-09-18T07:46:34.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048657489999999999 2024-09-18T07:46:34.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.623370+0000", 2024-09-18T07:46:34.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0301000000000001e-05 2024-09-18T07:46:34.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.659487+0000", 2024-09-18T07:46:34.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.036116802000000003 2024-09-18T07:46:34.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:32.659526+0000", 2024-09-18T07:46:34.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8701000000000001e-05 2024-09-18T07:46:34.940 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.940 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.940 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.940 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:164179 10.f e403/364 10:fb9170dd:::benchmark_data_smithi137_151160_object28363:head v 403'10126, mlcod=403'10122)", 2024-09-18T07:46:34.940 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:07.778976+0000", 2024-09-18T07:46:34.940 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 86.293366817999996, 2024-09-18T07:46:34.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032076422, 2024-09-18T07:46:34.941 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.941 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.941 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.941 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.778976+0000", 2024-09-18T07:46:34.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.778976+0000", 2024-09-18T07:46:34.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.779013+0000", 2024-09-18T07:46:34.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6727000000000003e-05 2024-09-18T07:46:34.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.779021+0000", 2024-09-18T07:46:34.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7030000000000006e-06 2024-09-18T07:46:34.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.943 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.943 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.779031+0000", 2024-09-18T07:46:34.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0676000000000001e-05 2024-09-18T07:46:34.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.943 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.943 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.779038+0000", 2024-09-18T07:46:34.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0190000000000001e-06 2024-09-18T07:46:34.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.943 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.943 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.783995+0000", 2024-09-18T07:46:34.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0049566760000000001 2024-09-18T07:46:34.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.784039+0000", 2024-09-18T07:46:34.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4508000000000001e-05 2024-09-18T07:46:34.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.810886+0000", 2024-09-18T07:46:34.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026847072 2024-09-18T07:46:34.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.945 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.811053+0000", 2024-09-18T07:46:34.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016604099999999999 2024-09-18T07:46:34.945 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.945 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.945 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.946 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:164182 10.7 e403/364 10:e6bdb6b0:::benchmark_data_smithi137_151160_object28366:head v 403'10375, mlcod=403'10372)", 2024-09-18T07:46:34.946 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:07.779050+0000", 2024-09-18T07:46:34.946 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 86.293292923999999, 2024-09-18T07:46:34.946 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031651855, 2024-09-18T07:46:34.946 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.946 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.946 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.946 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.779050+0000", 2024-09-18T07:46:34.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.779050+0000", 2024-09-18T07:46:34.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.779050+0000", 2024-09-18T07:46:34.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.32e-07 2024-09-18T07:46:34.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.779054+0000", 2024-09-18T07:46:34.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5719999999999999e-06 2024-09-18T07:46:34.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.779058+0000", 2024-09-18T07:46:34.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4870000000000004e-06 2024-09-18T07:46:34.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.949 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.779063+0000", 2024-09-18T07:46:34.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3209999999999998e-06 2024-09-18T07:46:34.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.949 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.783800+0000", 2024-09-18T07:46:34.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047370270000000004 2024-09-18T07:46:34.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.783833+0000", 2024-09-18T07:46:34.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3291999999999999e-05 2024-09-18T07:46:34.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.950 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.810501+0000", 2024-09-18T07:46:34.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026667395 2024-09-18T07:46:34.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.950 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.810702+0000", 2024-09-18T07:46:34.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020132900000000001 2024-09-18T07:46:34.950 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.951 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.951 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.951 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:164192 10.f e403/364 10:ffd1a448:::benchmark_data_smithi137_151160_object28376:head v 403'10127, mlcod=403'10122)", 2024-09-18T07:46:34.951 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:07.787527+0000", 2024-09-18T07:46:34.951 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 86.284816149999997, 2024-09-18T07:46:34.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031772090000000003, 2024-09-18T07:46:34.951 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.951 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.951 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.951 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.952 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.787527+0000", 2024-09-18T07:46:34.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.952 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.787527+0000", 2024-09-18T07:46:34.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.787528+0000", 2024-09-18T07:46:34.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1069999999999999e-06 2024-09-18T07:46:34.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.787535+0000", 2024-09-18T07:46:34.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2919999999999998e-06 2024-09-18T07:46:34.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.787545+0000", 2024-09-18T07:46:34.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.026e-05 2024-09-18T07:46:34.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.787552+0000", 2024-09-18T07:46:34.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9190000000000002e-06 2024-09-18T07:46:34.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.799655+0000", 2024-09-18T07:46:34.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012102896 2024-09-18T07:46:34.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.799708+0000", 2024-09-18T07:46:34.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2998000000000003e-05 2024-09-18T07:46:34.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.955 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.819121+0000", 2024-09-18T07:46:34.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019413256 2024-09-18T07:46:34.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.955 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.819299+0000", 2024-09-18T07:46:34.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000177362 2024-09-18T07:46:34.955 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.956 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.956 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.956 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:164194 10.7 e403/364 10:ee04812b:::benchmark_data_smithi137_151160_object28378:head v 403'10377, mlcod=403'10373)", 2024-09-18T07:46:34.956 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:07.790252+0000", 2024-09-18T07:46:34.956 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 86.282090733000004, 2024-09-18T07:46:34.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.037761031, 2024-09-18T07:46:34.956 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.956 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.956 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.956 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.790252+0000", 2024-09-18T07:46:34.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.790252+0000", 2024-09-18T07:46:34.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.790254+0000", 2024-09-18T07:46:34.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6169999999999999e-06 2024-09-18T07:46:34.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.790261+0000", 2024-09-18T07:46:34.958 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.447e-06 2024-09-18T07:46:34.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.790271+0000", 2024-09-18T07:46:34.958 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.009e-05 2024-09-18T07:46:34.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.959 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.959 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.790279+0000", 2024-09-18T07:46:34.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1640000000000001e-06 2024-09-18T07:46:34.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.959 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.959 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.804600+0000", 2024-09-18T07:46:34.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014320220999999999 2024-09-18T07:46:34.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.959 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.959 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.804617+0000", 2024-09-18T07:46:34.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7363e-05 2024-09-18T07:46:34.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.827949+0000", 2024-09-18T07:46:34.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.023332373 2024-09-18T07:46:34.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.828013+0000", 2024-09-18T07:46:34.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3756000000000004e-05 2024-09-18T07:46:34.960 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.960 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.960 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.961 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:164198 10.7 e403/364 10:e8b75f04:::benchmark_data_smithi137_151160_object28382:head v 403'10378, mlcod=403'10373)", 2024-09-18T07:46:34.961 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:07.795529+0000", 2024-09-18T07:46:34.961 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 86.276813481999994, 2024-09-18T07:46:34.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032512640000000002, 2024-09-18T07:46:34.961 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.961 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.961 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.961 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.795529+0000", 2024-09-18T07:46:34.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.795529+0000", 2024-09-18T07:46:34.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.795531+0000", 2024-09-18T07:46:34.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2759999999999999e-06 2024-09-18T07:46:34.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.795538+0000", 2024-09-18T07:46:34.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3699999999999997e-06 2024-09-18T07:46:34.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.795552+0000", 2024-09-18T07:46:34.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3730000000000001e-05 2024-09-18T07:46:34.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.795559+0000", 2024-09-18T07:46:34.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9890000000000003e-06 2024-09-18T07:46:34.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.807682+0000", 2024-09-18T07:46:34.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012123036 2024-09-18T07:46:34.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.807703+0000", 2024-09-18T07:46:34.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1007999999999999e-05 2024-09-18T07:46:34.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.828016+0000", 2024-09-18T07:46:34.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020313614000000001 2024-09-18T07:46:34.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:07.828042+0000", 2024-09-18T07:46:34.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5616999999999999e-05 2024-09-18T07:46:34.965 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.965 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.965 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.965 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:169399 10.f e405/364 10:f17a4787:::benchmark_data_smithi137_151160_object33583:head v 405'10438, mlcod=405'10436)", 2024-09-18T07:46:34.965 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:12.621954+0000", 2024-09-18T07:46:34.965 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 81.450388646999997, 2024-09-18T07:46:34.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12633334099999999, 2024-09-18T07:46:34.966 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.966 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.966 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.966 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.621954+0000", 2024-09-18T07:46:34.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.621954+0000", 2024-09-18T07:46:34.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.967 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.967 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.621956+0000", 2024-09-18T07:46:34.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.37e-06 2024-09-18T07:46:34.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.967 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.967 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.621961+0000", 2024-09-18T07:46:34.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.04e-06 2024-09-18T07:46:34.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.967 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.621975+0000", 2024-09-18T07:46:34.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4348000000000001e-05 2024-09-18T07:46:34.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.621989+0000", 2024-09-18T07:46:34.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3621000000000001e-05 2024-09-18T07:46:34.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.622032+0000", 2024-09-18T07:46:34.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3126e-05 2024-09-18T07:46:34.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.622044+0000", 2024-09-18T07:46:34.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1823e-05 2024-09-18T07:46:34.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.748251+0000", 2024-09-18T07:46:34.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.126207401 2024-09-18T07:46:34.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.748288+0000", 2024-09-18T07:46:34.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6612000000000001e-05 2024-09-18T07:46:34.970 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.970 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.970 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.970 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:169413 10.4 e405/396 10:2906f1b0:::benchmark_data_smithi137_151160_object33597:head v 405'10516, mlcod=405'10514)", 2024-09-18T07:46:34.970 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:12.623768+0000", 2024-09-18T07:46:34.970 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 81.448575016999996, 2024-09-18T07:46:34.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12446594900000001, 2024-09-18T07:46:34.970 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.970 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.970 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.971 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623768+0000", 2024-09-18T07:46:34.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623768+0000", 2024-09-18T07:46:34.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623769+0000", 2024-09-18T07:46:34.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3e-06 2024-09-18T07:46:34.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623775+0000", 2024-09-18T07:46:34.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0780000000000001e-06 2024-09-18T07:46:34.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623785+0000", 2024-09-18T07:46:34.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6439999999999995e-06 2024-09-18T07:46:34.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623791+0000", 2024-09-18T07:46:34.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0100000000000001e-06 2024-09-18T07:46:34.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623839+0000", 2024-09-18T07:46:34.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8424000000000003e-05 2024-09-18T07:46:34.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623861+0000", 2024-09-18T07:46:34.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1675000000000002e-05 2024-09-18T07:46:34.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.748175+0000", 2024-09-18T07:46:34.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12431392400000001 2024-09-18T07:46:34.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.748234+0000", 2024-09-18T07:46:34.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8894000000000002e-05 2024-09-18T07:46:34.974 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.974 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.975 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.975 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:169416 10.3 e405/364 10:c2f63df8:::benchmark_data_smithi137_151160_object33600:head v 405'10686, mlcod=405'10684)", 2024-09-18T07:46:34.975 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:12.624009+0000", 2024-09-18T07:46:34.975 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 81.448333697999999, 2024-09-18T07:46:34.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13276870599999999, 2024-09-18T07:46:34.975 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.975 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.975 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.975 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.624009+0000", 2024-09-18T07:46:34.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.624009+0000", 2024-09-18T07:46:34.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.624027+0000", 2024-09-18T07:46:34.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7625000000000002e-05 2024-09-18T07:46:34.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.624033+0000", 2024-09-18T07:46:34.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1090000000000003e-06 2024-09-18T07:46:34.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.624044+0000", 2024-09-18T07:46:34.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1419e-05 2024-09-18T07:46:34.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.624050+0000", 2024-09-18T07:46:34.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0109999999999997e-06 2024-09-18T07:46:34.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.624114+0000", 2024-09-18T07:46:34.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3829000000000002e-05 2024-09-18T07:46:34.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.624132+0000", 2024-09-18T07:46:34.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8029e-05 2024-09-18T07:46:34.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.756739+0000", 2024-09-18T07:46:34.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13260670599999999 2024-09-18T07:46:34.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.756778+0000", 2024-09-18T07:46:34.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8977999999999998e-05 2024-09-18T07:46:34.979 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.979 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.979 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.979 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:169421 10.3 e405/364 10:c4e21784:::benchmark_data_smithi137_151160_object33605:head v 405'10687, mlcod=405'10684)", 2024-09-18T07:46:34.980 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:12.631800+0000", 2024-09-18T07:46:34.980 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 81.440543371000004, 2024-09-18T07:46:34.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13578131500000001, 2024-09-18T07:46:34.980 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.980 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.980 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.980 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.980 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.631800+0000", 2024-09-18T07:46:34.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.631800+0000", 2024-09-18T07:46:34.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.631801+0000", 2024-09-18T07:46:34.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.804e-06 2024-09-18T07:46:34.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.631811+0000", 2024-09-18T07:46:34.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1889999999999996e-06 2024-09-18T07:46:34.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.748333+0000", 2024-09-18T07:46:34.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.11652246199999999 2024-09-18T07:46:34.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.748341+0000", 2024-09-18T07:46:34.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6699999999999994e-06 2024-09-18T07:46:34.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.751165+0000", 2024-09-18T07:46:34.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028240449999999999 2024-09-18T07:46:34.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.751184+0000", 2024-09-18T07:46:34.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9487000000000002e-05 2024-09-18T07:46:34.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.767552+0000", 2024-09-18T07:46:34.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016368007 2024-09-18T07:46:34.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.984 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.984 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.767581+0000", 2024-09-18T07:46:34.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8651000000000001e-05 2024-09-18T07:46:34.984 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.984 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.984 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.984 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:169427 10.f e405/364 10:f708ecac:::benchmark_data_smithi137_151160_object33611:head v 405'10439, mlcod=405'10436)", 2024-09-18T07:46:34.984 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:12.637779+0000", 2024-09-18T07:46:34.984 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 81.434564370000004, 2024-09-18T07:46:34.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.116250148, 2024-09-18T07:46:34.984 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.985 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.985 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.985 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.637779+0000", 2024-09-18T07:46:34.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.637779+0000", 2024-09-18T07:46:34.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.637781+0000", 2024-09-18T07:46:34.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0449999999999999e-06 2024-09-18T07:46:34.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.637790+0000", 2024-09-18T07:46:34.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2070000000000001e-06 2024-09-18T07:46:34.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.748204+0000", 2024-09-18T07:46:34.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.110414423 2024-09-18T07:46:34.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.748217+0000", 2024-09-18T07:46:34.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2459e-05 2024-09-18T07:46:34.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.748333+0000", 2024-09-18T07:46:34.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011658999999999999 2024-09-18T07:46:34.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.748355+0000", 2024-09-18T07:46:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1437000000000002e-05 2024-09-18T07:46:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.753949+0000", 2024-09-18T07:46:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055944380000000002 2024-09-18T07:46:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.754029+0000", 2024-09-18T07:46:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9548999999999993e-05 2024-09-18T07:46:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.988 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:198876 10.5 e429/364 10:adfc0595:::benchmark_data_smithi137_151160_object63060:head v 429'12388, mlcod=428'12384)", 2024-09-18T07:46:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:54.207626+0000", 2024-09-18T07:46:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 39.864717009000003, 2024-09-18T07:46:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.034579665000000002, 2024-09-18T07:46:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.207626+0000", 2024-09-18T07:46:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.207626+0000", 2024-09-18T07:46:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.207626+0000", 2024-09-18T07:46:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6300000000000001e-07 2024-09-18T07:46:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.990 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.207634+0000", 2024-09-18T07:46:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2559999999999996e-06 2024-09-18T07:46:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.207638+0000", 2024-09-18T07:46:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5249999999999999e-06 2024-09-18T07:46:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.207642+0000", 2024-09-18T07:46:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9360000000000001e-06 2024-09-18T07:46:34.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.213277+0000", 2024-09-18T07:46:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056354710000000004 2024-09-18T07:46:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.213296+0000", 2024-09-18T07:46:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8510999999999999e-05 2024-09-18T07:46:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.242156+0000", 2024-09-18T07:46:34.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028859955 2024-09-18T07:46:34.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:54.242206+0000", 2024-09-18T07:46:34.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9648000000000003e-05 2024-09-18T07:46:34.993 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.993 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.993 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.993 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:211647 10.5 e433/364 10:a50fd815:::benchmark_data_smithi137_151160_object75831:head v 433'13181, mlcod=433'13178)", 2024-09-18T07:46:34.993 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:05.410308+0000", 2024-09-18T07:46:34.993 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 28.662034604999999, 2024-09-18T07:46:34.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032723694999999997, 2024-09-18T07:46:34.994 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.994 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.994 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.994 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:05.410308+0000", 2024-09-18T07:46:34.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:05.410308+0000", 2024-09-18T07:46:34.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:34.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:05.410310+0000", 2024-09-18T07:46:34.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9350000000000001e-06 2024-09-18T07:46:34.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:34.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:05.410319+0000", 2024-09-18T07:46:34.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2800000000000003e-06 2024-09-18T07:46:34.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:34.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:05.410331+0000", 2024-09-18T07:46:34.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2758000000000001e-05 2024-09-18T07:46:34.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:34.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:05.410339+0000", 2024-09-18T07:46:34.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1189999999999997e-06 2024-09-18T07:46:34.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:34.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:05.424693+0000", 2024-09-18T07:46:34.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014353805000000001 2024-09-18T07:46:34.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:34.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:05.424711+0000", 2024-09-18T07:46:34.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8207999999999999e-05 2024-09-18T07:46:34.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:34.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:05.442802+0000", 2024-09-18T07:46:34.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018090108000000001 2024-09-18T07:46:34.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.998 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:34.998 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:05.443032+0000", 2024-09-18T07:46:34.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023048200000000001 2024-09-18T07:46:34.998 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.998 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:34.998 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:34.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.998 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:235034 10.f e437/364 10:f4b2272f:::benchmark_data_smithi137_151160_object99218:head v 437'14495, mlcod=437'14492)", 2024-09-18T07:46:34.998 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:24.406483+0000", 2024-09-18T07:46:34.998 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 9.6658595930000004, 2024-09-18T07:46:34.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033073691000000002, 2024-09-18T07:46:34.999 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:34.999 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:34.999 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:34.999 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:34.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:34.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.406483+0000", 2024-09-18T07:46:34.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:34.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:34.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:34.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.406483+0000", 2024-09-18T07:46:34.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:34.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:35.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.406485+0000", 2024-09-18T07:46:35.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.359e-06 2024-09-18T07:46:35.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:35.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.406492+0000", 2024-09-18T07:46:35.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2099999999999996e-06 2024-09-18T07:46:35.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:35.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.406501+0000", 2024-09-18T07:46:35.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4390000000000006e-06 2024-09-18T07:46:35.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.001 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:35.001 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.406508+0000", 2024-09-18T07:46:35.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7689999999999999e-06 2024-09-18T07:46:35.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.001 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:35.001 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.414678+0000", 2024-09-18T07:46:35.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081694929999999999 2024-09-18T07:46:35.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.001 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:35.001 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.414733+0000", 2024-09-18T07:46:35.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5690999999999999e-05 2024-09-18T07:46:35.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:35.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.439518+0000", 2024-09-18T07:46:35.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.024784947000000002 2024-09-18T07:46:35.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:35.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.439557+0000", 2024-09-18T07:46:35.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8782999999999998e-05 2024-09-18T07:46:35.002 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:35.002 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:35.003 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:35.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.003 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:235073 10.3 e437/416 10:cfb66955:::benchmark_data_smithi137_151160_object99257:head v 437'14835, mlcod=437'14831)", 2024-09-18T07:46:35.003 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:24.455580+0000", 2024-09-18T07:46:35.003 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 9.6167632780000005, 2024-09-18T07:46:35.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.030720730000000002, 2024-09-18T07:46:35.003 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:46:35.003 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:46:35.003 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:35.003 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:46:35.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.003 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:46:35.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.455580+0000", 2024-09-18T07:46:35.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:35.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.004 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:46:35.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.455580+0000", 2024-09-18T07:46:35.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:46:35.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.004 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:46:35.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.455581+0000", 2024-09-18T07:46:35.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2139999999999999e-06 2024-09-18T07:46:35.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.005 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:46:35.005 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.455587+0000", 2024-09-18T07:46:35.005 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.66e-06 2024-09-18T07:46:35.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.005 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:46:35.005 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.455598+0000", 2024-09-18T07:46:35.005 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1078e-05 2024-09-18T07:46:35.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.005 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:46:35.005 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.455604+0000", 2024-09-18T07:46:35.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7689999999999999e-06 2024-09-18T07:46:35.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:46:35.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.464497+0000", 2024-09-18T07:46:35.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0088923690000000007 2024-09-18T07:46:35.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:46:35.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.464521+0000", 2024-09-18T07:46:35.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3805e-05 2024-09-18T07:46:35.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:46:35.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.486108+0000", 2024-09-18T07:46:35.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021587006999999998 2024-09-18T07:46:35.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:46:35.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:46:35.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:46:35.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.486300+0000", 2024-09-18T07:46:35.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000192828 2024-09-18T07:46:35.007 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:35.007 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:35.007 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:35.007 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:46:35.007 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:46:35.008 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:46:35.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:35.009 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:35.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:35.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:35.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:35.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:35.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:35.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:35.294 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:35.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:35.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:35.395 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:35.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:35.423 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:35.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:35.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:35.555 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:35.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:35.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:35.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:35.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:35 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 8969/86022 objects degraded (10.426%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:35.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:35 smithi137 ceph-mon[29408]: osdmap e443: 8 total, 7 up, 8 in 2024-09-18T07:46:35.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:35 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 8969/86022 objects degraded (10.426%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:35.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:35 smithi137 ceph-mon[39175]: osdmap e443: 8 total, 7 up, 8 in 2024-09-18T07:46:35.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:35.696 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:35.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:35.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:35 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 8969/86022 objects degraded (10.426%), 24 pgs degraded (PG_DEGRADED) 2024-09-18T07:46:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:35 smithi203 ceph-mon[32237]: osdmap e443: 8 total, 7 up, 8 in 2024-09-18T07:46:35.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:35.833 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:35.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:35.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:35.935 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:35.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:35.963 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:35.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:36.063 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:36.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:36.098 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:36.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:36.199 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:36.233 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:36.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:36.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:36.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:36.367 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:36.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:36.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:36.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:36.509 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:36.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:36.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:36.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:36.625 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:36 smithi137 ceph-mon[29408]: pgmap v828: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 821 op/s; 8969/86022 objects degraded (10.426%) 2024-09-18T07:46:36.625 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:36 smithi137 ceph-mon[39175]: pgmap v828: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 821 op/s; 8969/86022 objects degraded (10.426%) 2024-09-18T07:46:36.627 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:46:36.627 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:46:36.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:36.665 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:36.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:36.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:36 smithi203 ceph-mon[32237]: pgmap v828: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 821 op/s; 8969/86022 objects degraded (10.426%) 2024-09-18T07:46:36.766 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:36.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:36.795 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:36.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:36.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:36.922 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:36.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:36.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:37.023 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:37.050 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:37.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:37.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:37.151 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:37.184 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:37.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:37.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:37.285 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:37.317 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:37.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:37.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:37.401 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:46:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:46:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:46:37.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:37.445 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:37.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:37.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:37.448 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T07:46:37.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:37.585 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:37.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:37.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:37.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:37.694 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:37.735 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:37.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:37.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:37.838 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:37.845 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:46:37.873 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:37.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:37.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:37.975 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:38.003 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:38.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:38.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:38.106 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:38.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:38.139 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:38.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:38.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:46:37 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:46:38.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:38.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:38.272 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:38.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:38.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:38.401 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:38.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:38.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:38.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:38.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:38.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:38.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:46:38.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:38.682 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:38.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:38.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:38.784 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:38.815 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:38.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:38.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:38.862 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:38 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a[29404]: 2024-09-18T07:46:38.595+0000 7fe5973a1640 -1 mon.a@0(leader).osd e443 definitely_dead 0 2024-09-18T07:46:38.864 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:46:38.864 INFO:teuthology.orchestra.run.smithi137.stderr:marked out osd.1. 2024-09-18T07:46:38.917 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:38.957 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:38.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:38.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:39.059 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:39.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:39.081 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:39.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:39.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:38 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:39.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:38 smithi137 ceph-mon[29408]: pgmap v829: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 1.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 7374/69294 objects degraded (10.642%) 2024-09-18T07:46:39.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:38 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2427960541' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:39.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:38 smithi137 ceph-mon[29408]: from='client.26401 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:39.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:38 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3716812202' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-18T07:46:39.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:38 smithi137 ceph-mon[29408]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-18T07:46:39.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:38 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:39.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:38 smithi137 ceph-mon[39175]: pgmap v829: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 1.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 7374/69294 objects degraded (10.642%) 2024-09-18T07:46:39.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:38 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2427960541' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:39.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:38 smithi137 ceph-mon[39175]: from='client.26401 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:39.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:38 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3716812202' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-18T07:46:39.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:38 smithi137 ceph-mon[39175]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-18T07:46:39.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:38 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:39.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:38 smithi203 ceph-mon[32237]: pgmap v829: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 1.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 7374/69294 objects degraded (10.642%) 2024-09-18T07:46:39.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:38 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2427960541' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:39.175 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:38 smithi203 ceph-mon[32237]: from='client.26401 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:39.176 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:38 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3716812202' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-18T07:46:39.176 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:38 smithi203 ceph-mon[32237]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-18T07:46:39.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:39.203 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:39.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:39.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:39.305 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:39.330 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:39.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:39.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:39.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:39.456 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:39.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:39.557 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:39.581 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:39.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:39.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:39.682 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:39.706 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:39.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:39.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:39.808 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:39.834 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:39.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:39.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:39.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:39.956 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:39.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:39.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:40.058 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:40.080 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:40.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:40.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:40.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:39 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:40.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:39 smithi137 ceph-mon[39175]: from='client.26401 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:46:40.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:39 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3716812202' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-18T07:46:40.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:39 smithi137 ceph-mon[39175]: osdmap e444: 8 total, 7 up, 7 in 2024-09-18T07:46:40.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:39 smithi137 ceph-mon[39175]: osdmap e445: 8 total, 7 up, 7 in 2024-09-18T07:46:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:39 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:39 smithi137 ceph-mon[29408]: from='client.26401 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:46:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:39 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3716812202' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-18T07:46:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:39 smithi137 ceph-mon[29408]: osdmap e444: 8 total, 7 up, 7 in 2024-09-18T07:46:40.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:39 smithi137 ceph-mon[29408]: osdmap e445: 8 total, 7 up, 7 in 2024-09-18T07:46:40.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:40.204 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:40.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:40.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:39 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:39 smithi203 ceph-mon[32237]: from='client.26401 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:46:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:39 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3716812202' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-18T07:46:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:39 smithi203 ceph-mon[32237]: osdmap e444: 8 total, 7 up, 7 in 2024-09-18T07:46:40.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:39 smithi203 ceph-mon[32237]: osdmap e445: 8 total, 7 up, 7 in 2024-09-18T07:46:40.305 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:40.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:40.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:40.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:40.429 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:40.450 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:40.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:40.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:40.516 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:40.552 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:40.579 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:40.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:40.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:40.681 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:40.703 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:40.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:40.804 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:40.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:40.826 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:40.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:40.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:40 smithi137 ceph-mon[29408]: pgmap v831: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 1.3 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 792 B/s rd, 0 B/s wr, 988 op/s; 6956/66876 objects degraded (10.401%) 2024-09-18T07:46:40.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:40 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 6956/66876 objects degraded (10.401%), 24 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-18T07:46:40.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:40 smithi137 ceph-mon[29408]: osdmap e446: 8 total, 7 up, 7 in 2024-09-18T07:46:40.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:40 smithi137 ceph-mon[39175]: pgmap v831: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 1.3 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 792 B/s rd, 0 B/s wr, 988 op/s; 6956/66876 objects degraded (10.401%) 2024-09-18T07:46:40.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:40 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 6956/66876 objects degraded (10.401%), 24 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-18T07:46:40.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:40 smithi137 ceph-mon[39175]: osdmap e446: 8 total, 7 up, 7 in 2024-09-18T07:46:40.927 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:40.955 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:40.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:41.033 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:46:41.033 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:46:41.058 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:41.061 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:46:41.061 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:46:41.095 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:46:41.095 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:46:41.098 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:41.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:41.118 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:46:41.118 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:46:41.160 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:46:41.160 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:46:41.161 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-18T07:46:41.176 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:40 smithi203 ceph-mon[32237]: pgmap v831: 158 pgs: 36 active+undersized, 24 active+undersized+degraded, 98 active+clean; 1.3 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 792 B/s rd, 0 B/s wr, 988 op/s; 6956/66876 objects degraded (10.401%) 2024-09-18T07:46:41.176 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:40 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 6956/66876 objects degraded (10.401%), 24 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-09-18T07:46:41.176 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:40 smithi203 ceph-mon[32237]: osdmap e446: 8 total, 7 up, 7 in 2024-09-18T07:46:41.194 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:46:41.194 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:46:41.199 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:41.227 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:46:41.227 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:46:41.239 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:41.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:41.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:41.339 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:46:41.339 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:46:41.341 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:41.375 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:41.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:41.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:41.477 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:41.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:41.510 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:41.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:41.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:41.652 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:41.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:41.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:41.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:41.787 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:41.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:41.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:41.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:41 smithi137 ceph-mon[29408]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 443'15662 2024-09-18T07:46:41.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:41 smithi137 ceph-mon[29408]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 443'15751 2024-09-18T07:46:41.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:41 smithi137 ceph-mon[29408]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 443'15653 2024-09-18T07:46:41.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:41 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/96628964' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:41.926 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:41 smithi137 ceph-mon[39175]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 443'15662 2024-09-18T07:46:41.926 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:41 smithi137 ceph-mon[39175]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 443'15751 2024-09-18T07:46:41.926 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:41 smithi137 ceph-mon[39175]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 443'15653 2024-09-18T07:46:41.926 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:41 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/96628964' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:41.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:41.929 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:41.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:41.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:41 smithi203 ceph-mon[32237]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 443'15662 2024-09-18T07:46:41.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:41 smithi203 ceph-mon[32237]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 443'15751 2024-09-18T07:46:41.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:41 smithi203 ceph-mon[32237]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 443'15653 2024-09-18T07:46:41.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:41 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/96628964' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:42.030 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:42.064 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:42.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:42.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:42.121 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:46:42.122 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:46:42.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:42.199 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:42.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:42.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:42.202 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T07:46:42.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:42.335 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:42.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:42.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:42.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:46:42.437 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:42.472 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:42.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:42.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:42.574 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:42.623 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:42.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:42.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:42.630 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:46:42.724 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:42.785 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:42.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:42.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:42.844 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:42.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:42.920 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:42.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:42.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:43.022 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:43.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:42 smithi203 ceph-mon[32237]: pgmap v834: 158 pgs: 1 active+remapped+backfilling, 17 active+clean+remapped, 25 active+undersized, 21 active+undersized+degraded, 94 active+clean; 1.2 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 6497/61404 objects degraded (10.581%); 1153/61404 objects misplaced (1.878%); 683 KiB/s, 22 objects/s recovering 2024-09-18T07:46:43.027 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:42 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:46:43.057 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:43.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:43.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:43.148 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:42 smithi137 ceph-mon[39175]: pgmap v834: 158 pgs: 1 active+remapped+backfilling, 17 active+clean+remapped, 25 active+undersized, 21 active+undersized+degraded, 94 active+clean; 1.2 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 6497/61404 objects degraded (10.581%); 1153/61404 objects misplaced (1.878%); 683 KiB/s, 22 objects/s recovering 2024-09-18T07:46:43.149 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:42 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:46:43.149 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:42 smithi137 ceph-mon[29408]: pgmap v834: 158 pgs: 1 active+remapped+backfilling, 17 active+clean+remapped, 25 active+undersized, 21 active+undersized+degraded, 94 active+clean; 1.2 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 6497/61404 objects degraded (10.581%); 1153/61404 objects misplaced (1.878%); 683 KiB/s, 22 objects/s recovering 2024-09-18T07:46:43.149 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:42 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:46:43.159 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:43.197 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:43.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:43.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:43.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:43.322 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:46:43.322 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:46:43.336 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:43.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:43.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:43.439 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:43.469 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:43.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:43.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:43.571 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:43.603 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:43.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:43.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:43.705 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:43.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:43.750 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:43.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:43.852 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:43.890 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:43.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:43.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:43.906 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:46:43.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:44.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:44.023 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:44.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:44.030 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T07:46:44.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:46:44.123 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:44.157 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:43 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/800983930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:46:44.158 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:43 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/800983930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:46:44.163 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:44.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:44.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:44.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:43 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/800983930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:46:44.264 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:44.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:44.311 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:44.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:44.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:44.452 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:44.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:44.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:44.458 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:46:44.554 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:44.595 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:44.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:44.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:44.647 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 6] out_osds: [1] dead_osds: [5] live_osds: [0, 1, 2, 3, 6, 7, 4] 2024-09-18T07:46:44.647 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:46:44.648 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-18T07:46:44.648 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-18T07:46:44.648 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-18T07:46:44.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-18T07:46:44.697 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:44.727 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:44.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:44.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:44.828 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:44.865 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:44.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:44.966 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:45.006 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:45.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:45.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:45.091 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:45.108 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:45.109 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:45.126 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:46:45.126 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:46:45.153 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:45.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:45.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:45.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:44 smithi137 ceph-mon[29408]: pgmap v835: 158 pgs: 3 active+remapped+backfilling, 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 91 active+clean; 1.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 170 B/s rd, 0 B/s wr, 737 op/s; 5805/56019 objects degraded (10.363%); 3061/56019 objects misplaced (5.464%); 4.9 MiB/s, 94 objects/s recovering 2024-09-18T07:46:45.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:44 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:45.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:44 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/800983930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:46:45.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:44 smithi137 ceph-mon[29408]: osdmap e447: 8 total, 7 up, 7 in 2024-09-18T07:46:45.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:44 smithi137 ceph-mon[29408]: 3.4 scrub starts 2024-09-18T07:46:45.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:44 smithi137 ceph-mon[29408]: 3.4 scrub ok 2024-09-18T07:46:45.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:44 smithi137 ceph-mon[39175]: pgmap v835: 158 pgs: 3 active+remapped+backfilling, 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 91 active+clean; 1.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 170 B/s rd, 0 B/s wr, 737 op/s; 5805/56019 objects degraded (10.363%); 3061/56019 objects misplaced (5.464%); 4.9 MiB/s, 94 objects/s recovering 2024-09-18T07:46:45.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:44 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:45.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/800983930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:46:45.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:44 smithi137 ceph-mon[39175]: osdmap e447: 8 total, 7 up, 7 in 2024-09-18T07:46:45.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:44 smithi137 ceph-mon[39175]: 3.4 scrub starts 2024-09-18T07:46:45.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:44 smithi137 ceph-mon[39175]: 3.4 scrub ok 2024-09-18T07:46:45.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:44 smithi203 ceph-mon[32237]: pgmap v835: 158 pgs: 3 active+remapped+backfilling, 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 91 active+clean; 1.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 170 B/s rd, 0 B/s wr, 737 op/s; 5805/56019 objects degraded (10.363%); 3061/56019 objects misplaced (5.464%); 4.9 MiB/s, 94 objects/s recovering 2024-09-18T07:46:45.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:44 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:45.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:44 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/800983930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:46:45.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:44 smithi203 ceph-mon[32237]: osdmap e447: 8 total, 7 up, 7 in 2024-09-18T07:46:45.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:44 smithi203 ceph-mon[32237]: 3.4 scrub starts 2024-09-18T07:46:45.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:44 smithi203 ceph-mon[32237]: 3.4 scrub ok 2024-09-18T07:46:45.254 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:45.293 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:45.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:45.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:45.395 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:45.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:45.438 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:45.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:45.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:45.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:45.584 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:45.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:45.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:45.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:45.724 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:45.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:45.825 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:45.834 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T07:46:45.857 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:45.858 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:46:45.858 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:46:45.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:45.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:45.890 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:46:45.890 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:46:45.923 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:46:45.923 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:46:45.957 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:46:45.958 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:46:45.960 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:45.989 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:46:45.989 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:46:45.990 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-18T07:46:45.995 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:45.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:45.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:46.026 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:46:46.027 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:46:46.058 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:46:46.058 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:45 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 5805/56019 objects degraded (10.363%), 15 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-09-18T07:46:46.058 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:45 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2834201376' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:46.058 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:45 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 5805/56019 objects degraded (10.363%), 15 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-09-18T07:46:46.058 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:45 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2834201376' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:46.059 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:46:46.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:46.136 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:46.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:46.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:45 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 5805/56019 objects degraded (10.363%), 15 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-09-18T07:46:46.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:45 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2834201376' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:46.239 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:46.253 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:46:46.283 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:46.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:46.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:46.385 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:46.414 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:46.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:46.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:46.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:46.556 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:46.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:46.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:46.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:46.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:46.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:46.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:46.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:46.835 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:46.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:46.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:46.931 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:46:46.932 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:46:46.936 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:46.968 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:46.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:46.972 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:46:47.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:47.093 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:47.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:47.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:47.129 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:46:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:46:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:46:47.130 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:46 smithi137 ceph-mon[29408]: 6.d scrub starts 2024-09-18T07:46:47.130 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:46 smithi137 ceph-mon[29408]: 6.d scrub ok 2024-09-18T07:46:47.130 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:46 smithi137 ceph-mon[29408]: pgmap v837: 158 pgs: 3 active+remapped+backfilling, 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 91 active+clean; 1.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 170 B/s rd, 0 B/s wr, 602 op/s; 5805/56019 objects degraded (10.363%); 3061/56019 objects misplaced (5.464%); 4.9 MiB/s, 94 objects/s recovering 2024-09-18T07:46:47.130 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:46 smithi137 ceph-mon[39175]: 6.d scrub starts 2024-09-18T07:46:47.130 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:46 smithi137 ceph-mon[39175]: 6.d scrub ok 2024-09-18T07:46:47.130 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:46 smithi137 ceph-mon[39175]: pgmap v837: 158 pgs: 3 active+remapped+backfilling, 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 91 active+clean; 1.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 170 B/s rd, 0 B/s wr, 602 op/s; 5805/56019 objects degraded (10.363%); 3061/56019 objects misplaced (5.464%); 4.9 MiB/s, 94 objects/s recovering 2024-09-18T07:46:47.161 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:46:47.161 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "" 2024-09-18T07:46:47.161 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:46:47.195 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:47.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:46 smithi203 ceph-mon[32237]: 6.d scrub starts 2024-09-18T07:46:47.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:46 smithi203 ceph-mon[32237]: 6.d scrub ok 2024-09-18T07:46:47.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:46 smithi203 ceph-mon[32237]: pgmap v837: 158 pgs: 3 active+remapped+backfilling, 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 91 active+clean; 1.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 170 B/s rd, 0 B/s wr, 602 op/s; 5805/56019 objects degraded (10.363%); 3061/56019 objects misplaced (5.464%); 4.9 MiB/s, 94 objects/s recovering 2024-09-18T07:46:47.217 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:47.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:47.318 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:47.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:47.341 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:47.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:47.442 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:47.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:47.466 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:47.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:47.569 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:47.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:47.591 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:47.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:47.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:47.714 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:47.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:47.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:47.838 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:47.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:47.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:47.894 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:46:47.894 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:46:47.894 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-09-18T07:46:47.894 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:46:47.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:47.972 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:47.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:47.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:47.974 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:48.075 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:48.106 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:48.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:48.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:48.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:48.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:46:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:46:48.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:47 smithi203 ceph-mon[32237]: 5.5 scrub starts 2024-09-18T07:46:48.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:47 smithi203 ceph-mon[32237]: 5.5 scrub ok 2024-09-18T07:46:48.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:47 smithi203 ceph-mon[32237]: osdmap e448: 8 total, 7 up, 7 in 2024-09-18T07:46:48.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:47 smithi203 ceph-mon[32237]: 3.a scrub starts 2024-09-18T07:46:48.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:47 smithi203 ceph-mon[32237]: 3.a scrub ok 2024-09-18T07:46:48.230 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:48.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:48.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:48.245 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:47 smithi137 ceph-mon[29408]: 5.5 scrub starts 2024-09-18T07:46:48.245 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:47 smithi137 ceph-mon[29408]: 5.5 scrub ok 2024-09-18T07:46:48.245 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:47 smithi137 ceph-mon[29408]: osdmap e448: 8 total, 7 up, 7 in 2024-09-18T07:46:48.245 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:47 smithi137 ceph-mon[29408]: 3.a scrub starts 2024-09-18T07:46:48.245 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:47 smithi137 ceph-mon[29408]: 3.a scrub ok 2024-09-18T07:46:48.246 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:47 smithi137 ceph-mon[39175]: 5.5 scrub starts 2024-09-18T07:46:48.246 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:47 smithi137 ceph-mon[39175]: 5.5 scrub ok 2024-09-18T07:46:48.246 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:47 smithi137 ceph-mon[39175]: osdmap e448: 8 total, 7 up, 7 in 2024-09-18T07:46:48.246 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:47 smithi137 ceph-mon[39175]: 3.a scrub starts 2024-09-18T07:46:48.246 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:47 smithi137 ceph-mon[39175]: 3.a scrub ok 2024-09-18T07:46:48.332 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:48.353 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:48.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:48.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:48.456 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:48.477 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:48.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:48.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:48.552 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T07:46:48.579 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:48.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:48.601 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:48.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:48.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:48.723 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:48.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:48.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:48.810 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:46:48.825 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:48.847 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:48.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:48.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:48.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:46:48.939 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:46:48.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:48.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:48.976 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:48.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:49.078 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:49.118 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:49.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:49.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:49.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:48 smithi203 ceph-mon[32237]: 2.0 scrub starts 2024-09-18T07:46:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:48 smithi203 ceph-mon[32237]: 2.0 scrub ok 2024-09-18T07:46:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:48 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:48 smithi203 ceph-mon[32237]: pgmap v839: 158 pgs: 2 peering, 1 active+remapped+backfilling, 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 91 active+clean; 797 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 884 B/s rd, 0 B/s wr, 1.35k op/s; 4140/38838 objects degraded (10.660%); 175/38838 objects misplaced (0.451%); 26 MiB/s, 422 objects/s recovering 2024-09-18T07:46:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:48 smithi203 ceph-mon[32237]: osdmap e449: 8 total, 7 up, 7 in 2024-09-18T07:46:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:48 smithi203 ceph-mon[32237]: 4.1d scrub starts 2024-09-18T07:46:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:48 smithi203 ceph-mon[32237]: 4.1d scrub ok 2024-09-18T07:46:49.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:48 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1397600216' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:49.220 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:49.250 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:49.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:49.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:49.344 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:48 smithi137 ceph-mon[29408]: 2.0 scrub starts 2024-09-18T07:46:49.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:48 smithi137 ceph-mon[29408]: 2.0 scrub ok 2024-09-18T07:46:49.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:48 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:49.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:48 smithi137 ceph-mon[29408]: pgmap v839: 158 pgs: 2 peering, 1 active+remapped+backfilling, 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 91 active+clean; 797 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 884 B/s rd, 0 B/s wr, 1.35k op/s; 4140/38838 objects degraded (10.660%); 175/38838 objects misplaced (0.451%); 26 MiB/s, 422 objects/s recovering 2024-09-18T07:46:49.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:48 smithi137 ceph-mon[29408]: osdmap e449: 8 total, 7 up, 7 in 2024-09-18T07:46:49.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:48 smithi137 ceph-mon[29408]: 4.1d scrub starts 2024-09-18T07:46:49.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:48 smithi137 ceph-mon[29408]: 4.1d scrub ok 2024-09-18T07:46:49.345 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:48 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1397600216' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:49.345 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:48 smithi137 ceph-mon[39175]: 2.0 scrub starts 2024-09-18T07:46:49.345 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:48 smithi137 ceph-mon[39175]: 2.0 scrub ok 2024-09-18T07:46:49.345 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:48 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:49.345 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:48 smithi137 ceph-mon[39175]: pgmap v839: 158 pgs: 2 peering, 1 active+remapped+backfilling, 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 91 active+clean; 797 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 884 B/s rd, 0 B/s wr, 1.35k op/s; 4140/38838 objects degraded (10.660%); 175/38838 objects misplaced (0.451%); 26 MiB/s, 422 objects/s recovering 2024-09-18T07:46:49.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:48 smithi137 ceph-mon[39175]: osdmap e449: 8 total, 7 up, 7 in 2024-09-18T07:46:49.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:48 smithi137 ceph-mon[39175]: 4.1d scrub starts 2024-09-18T07:46:49.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:48 smithi137 ceph-mon[39175]: 4.1d scrub ok 2024-09-18T07:46:49.346 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:48 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1397600216' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:46:49.352 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:49.393 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:49.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:49.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:49.416 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:46:49.416 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:46:49.416 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:46:49.417 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:46:49.417 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:46:49.495 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:49.525 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:49.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:49.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:49.627 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:49.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:49.662 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:49.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:49.763 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:49.785 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:49.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:49.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:49.887 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:49.921 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:49.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:50.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:50.057 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:50.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:50.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:50.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:50.193 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:50.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:50.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:49 smithi203 ceph-mon[32237]: 6.e scrub starts 2024-09-18T07:46:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:49 smithi203 ceph-mon[32237]: 6.e scrub ok 2024-09-18T07:46:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:49 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:46:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:49 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1397600216' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:46:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:49 smithi203 ceph-mon[32237]: osdmap e450: 8 total, 7 up, 7 in 2024-09-18T07:46:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:49 smithi203 ceph-mon[32237]: 6.1 scrub starts 2024-09-18T07:46:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:49 smithi203 ceph-mon[32237]: 6.1 scrub ok 2024-09-18T07:46:50.239 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T07:46:50.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:50.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:50.333 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:50.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:50.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:49 smithi137 ceph-mon[29408]: 6.e scrub starts 2024-09-18T07:46:50.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:49 smithi137 ceph-mon[29408]: 6.e scrub ok 2024-09-18T07:46:50.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:49 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:46:50.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:49 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1397600216' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:46:50.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:49 smithi137 ceph-mon[29408]: osdmap e450: 8 total, 7 up, 7 in 2024-09-18T07:46:50.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:49 smithi137 ceph-mon[29408]: 6.1 scrub starts 2024-09-18T07:46:50.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:49 smithi137 ceph-mon[29408]: 6.1 scrub ok 2024-09-18T07:46:50.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:49 smithi137 ceph-mon[39175]: 6.e scrub starts 2024-09-18T07:46:50.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:49 smithi137 ceph-mon[39175]: 6.e scrub ok 2024-09-18T07:46:50.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:49 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:46:50.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:49 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1397600216' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:46:50.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:49 smithi137 ceph-mon[39175]: osdmap e450: 8 total, 7 up, 7 in 2024-09-18T07:46:50.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:49 smithi137 ceph-mon[39175]: 6.1 scrub starts 2024-09-18T07:46:50.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:49 smithi137 ceph-mon[39175]: 6.1 scrub ok 2024-09-18T07:46:50.434 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:50.471 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:50.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:50.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:50.572 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:50.606 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:50.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:50.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:50.628 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:46:50.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:50.744 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:50.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:50.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:50.846 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:50.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:50.888 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:50.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:50.989 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:51.026 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:51.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:51.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:51.068 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:51.128 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:51.160 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:51.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:51.192 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:50 smithi203 ceph-mon[32237]: pgmap v842: 158 pgs: 2 peering, 1 active+remapped+backfilling, 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 91 active+clean; 755 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 867 B/s rd, 0 B/s wr, 1.08k op/s; 3808/36831 objects degraded (10.339%); 175/36831 objects misplaced (0.475%); 25 MiB/s, 401 objects/s recovering 2024-09-18T07:46:51.192 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:50 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 3808/36831 objects degraded (10.339%), 15 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-09-18T07:46:51.192 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:50 smithi203 ceph-mon[32237]: osdmap e451: 8 total, 7 up, 7 in 2024-09-18T07:46:51.200 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:50 smithi137 ceph-mon[29408]: pgmap v842: 158 pgs: 2 peering, 1 active+remapped+backfilling, 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 91 active+clean; 755 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 867 B/s rd, 0 B/s wr, 1.08k op/s; 3808/36831 objects degraded (10.339%); 175/36831 objects misplaced (0.475%); 25 MiB/s, 401 objects/s recovering 2024-09-18T07:46:51.200 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:50 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 3808/36831 objects degraded (10.339%), 15 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-09-18T07:46:51.200 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:50 smithi137 ceph-mon[29408]: osdmap e451: 8 total, 7 up, 7 in 2024-09-18T07:46:51.201 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:50 smithi137 ceph-mon[39175]: pgmap v842: 158 pgs: 2 peering, 1 active+remapped+backfilling, 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 91 active+clean; 755 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 867 B/s rd, 0 B/s wr, 1.08k op/s; 3808/36831 objects degraded (10.339%); 175/36831 objects misplaced (0.475%); 25 MiB/s, 401 objects/s recovering 2024-09-18T07:46:51.201 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:50 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 3808/36831 objects degraded (10.339%), 15 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-09-18T07:46:51.201 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:50 smithi137 ceph-mon[39175]: osdmap e451: 8 total, 7 up, 7 in 2024-09-18T07:46:51.262 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:51.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:51.303 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:51.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:51.355 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:46:51.355 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:46:51.355 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:46:51.355 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:46:51.355 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.355 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:170414 8.5 e264/255 8:a29be62f:::benchmark_data_smithi137_101942_object65604:head v 264'10576, mlcod=264'10574)", 2024-09-18T07:46:51.355 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:44.768915+0000", 2024-09-18T07:46:51.356 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 486.58077878300003, 2024-09-18T07:46:51.356 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15600961199999999, 2024-09-18T07:46:51.356 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.356 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.356 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.356 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.356 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.356 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.356 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768915+0000", 2024-09-18T07:46:51.356 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.356 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.356 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.357 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.357 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768915+0000", 2024-09-18T07:46:51.357 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.357 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.357 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.357 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.357 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768916+0000", 2024-09-18T07:46:51.357 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.24e-06 2024-09-18T07:46:51.357 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.357 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.357 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.357 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768923+0000", 2024-09-18T07:46:51.357 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7410000000000003e-06 2024-09-18T07:46:51.358 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.358 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.358 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.358 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768936+0000", 2024-09-18T07:46:51.358 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.345e-05 2024-09-18T07:46:51.358 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.358 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.358 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.358 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768944+0000", 2024-09-18T07:46:51.358 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0299999999999996e-06 2024-09-18T07:46:51.358 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.358 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.358 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.359 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.770341+0000", 2024-09-18T07:46:51.359 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001397061 2024-09-18T07:46:51.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.359 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.359 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.359 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.770358+0000", 2024-09-18T07:46:51.359 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7825999999999998e-05 2024-09-18T07:46:51.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.359 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.359 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.359 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.924876+0000", 2024-09-18T07:46:51.359 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.154517618 2024-09-18T07:46:51.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.359 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.359 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.360 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.924925+0000", 2024-09-18T07:46:51.360 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8646000000000001e-05 2024-09-18T07:46:51.360 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.360 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.360 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.360 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.360 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.360 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:170415 8.c e264/201 8:341230f3:::benchmark_data_smithi137_101942_object65605:head v 264'10461, mlcod=264'10459)", 2024-09-18T07:46:51.360 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:44.772649+0000", 2024-09-18T07:46:51.360 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 486.57704437000001, 2024-09-18T07:46:51.360 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.154977485, 2024-09-18T07:46:51.360 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.360 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.361 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.361 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.361 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.361 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.361 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.772649+0000", 2024-09-18T07:46:51.361 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.361 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.361 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.361 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.361 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.772649+0000", 2024-09-18T07:46:51.361 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.361 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.361 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.362 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.362 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.772652+0000", 2024-09-18T07:46:51.362 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3300000000000001e-06 2024-09-18T07:46:51.362 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.362 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.362 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.362 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.772662+0000", 2024-09-18T07:46:51.362 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0524e-05 2024-09-18T07:46:51.362 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.362 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.362 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.362 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.911734+0000", 2024-09-18T07:46:51.362 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.13907166200000001 2024-09-18T07:46:51.363 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.363 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.363 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.363 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.911745+0000", 2024-09-18T07:46:51.363 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0925e-05 2024-09-18T07:46:51.363 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.363 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.363 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.363 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.911798+0000", 2024-09-18T07:46:51.363 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3384000000000003e-05 2024-09-18T07:46:51.363 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.363 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.364 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.364 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.911818+0000", 2024-09-18T07:46:51.364 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9242999999999999e-05 2024-09-18T07:46:51.364 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.364 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.364 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.364 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.927424+0000", 2024-09-18T07:46:51.364 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.015606201 2024-09-18T07:46:51.364 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.364 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.364 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.364 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.927627+0000", 2024-09-18T07:46:51.364 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00020321599999999999 2024-09-18T07:46:51.365 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.365 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.365 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.365 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.365 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.365 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:156461 9.d 9:b7fc06f1:::benchmark_data_smithi137_126072_object23961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:46:51.365 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:24.401407+0000", 2024-09-18T07:46:51.365 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 326.94828727999999, 2024-09-18T07:46:51.365 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16122866899999999, 2024-09-18T07:46:51.365 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.365 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.365 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.365 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.366 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:46:51.366 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:46:51.366 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 156461 2024-09-18T07:46:51.366 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.366 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.366 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.366 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.366 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401407+0000", 2024-09-18T07:46:51.366 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.366 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.366 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.366 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.366 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401407+0000", 2024-09-18T07:46:51.367 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.367 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.367 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.367 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.367 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401409+0000", 2024-09-18T07:46:51.367 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0150000000000002e-06 2024-09-18T07:46:51.367 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.367 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.367 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.367 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401414+0000", 2024-09-18T07:46:51.367 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6810000000000003e-06 2024-09-18T07:46:51.368 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.368 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.368 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.368 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401428+0000", 2024-09-18T07:46:51.368 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4049000000000001e-05 2024-09-18T07:46:51.368 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.368 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.368 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.368 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401436+0000", 2024-09-18T07:46:51.368 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5549999999999997e-06 2024-09-18T07:46:51.368 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.368 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.369 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.369 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401509+0000", 2024-09-18T07:46:51.369 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3232999999999995e-05 2024-09-18T07:46:51.369 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.369 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.369 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.369 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401613+0000", 2024-09-18T07:46:51.369 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010402400000000001 2024-09-18T07:46:51.369 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.369 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.369 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:46:51.369 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401718+0000", 2024-09-18T07:46:51.369 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010482800000000001 2024-09-18T07:46:51.369 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.370 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.370 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:46:51.370 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409266+0000", 2024-09-18T07:46:51.370 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0075485639999999998 2024-09-18T07:46:51.370 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.370 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.370 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.371 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409351+0000", 2024-09-18T07:46:51.371 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4796000000000006e-05 2024-09-18T07:46:51.371 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.371 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.371 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.371 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.562555+0000", 2024-09-18T07:46:51.371 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.153203325 2024-09-18T07:46:51.371 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.371 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.372 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.372 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.562599+0000", 2024-09-18T07:46:51.372 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4431000000000002e-05 2024-09-18T07:46:51.372 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.372 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.372 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.372 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.562635+0000", 2024-09-18T07:46:51.372 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6168000000000003e-05 2024-09-18T07:46:51.372 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.372 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.372 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.372 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.372 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.373 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:156463 9.d 9:b608ec1e:::benchmark_data_smithi137_126072_object23963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:46:51.373 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:24.403102+0000", 2024-09-18T07:46:51.373 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 326.94659211999999, 2024-09-18T07:46:51.373 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16236466999999999, 2024-09-18T07:46:51.373 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.373 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.373 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.373 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.373 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:46:51.373 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:46:51.373 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 156463 2024-09-18T07:46:51.373 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.373 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.373 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.374 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.374 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403102+0000", 2024-09-18T07:46:51.374 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.374 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.374 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.374 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.374 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403102+0000", 2024-09-18T07:46:51.374 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.374 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.374 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.374 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.374 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403103+0000", 2024-09-18T07:46:51.374 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.767e-06 2024-09-18T07:46:51.374 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.375 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.375 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.375 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403108+0000", 2024-09-18T07:46:51.375 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5549999999999996e-06 2024-09-18T07:46:51.375 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.375 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.375 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.375 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403117+0000", 2024-09-18T07:46:51.375 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9390000000000003e-06 2024-09-18T07:46:51.375 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.375 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.375 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.375 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403123+0000", 2024-09-18T07:46:51.376 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4309999999999999e-06 2024-09-18T07:46:51.376 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.376 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.376 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.376 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404263+0000", 2024-09-18T07:46:51.376 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001140032 2024-09-18T07:46:51.376 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.376 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.376 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.376 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404367+0000", 2024-09-18T07:46:51.376 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000103514 2024-09-18T07:46:51.376 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.376 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.377 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:46:51.377 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404535+0000", 2024-09-18T07:46:51.377 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000167641 2024-09-18T07:46:51.377 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.377 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.377 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:46:51.377 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409418+0000", 2024-09-18T07:46:51.377 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0048834990000000003 2024-09-18T07:46:51.377 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.377 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.377 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.377 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409552+0000", 2024-09-18T07:46:51.377 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000133688 2024-09-18T07:46:51.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.378 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.378 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.378 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.564499+0000", 2024-09-18T07:46:51.378 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15494714900000001 2024-09-18T07:46:51.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.378 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.378 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.378 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.565425+0000", 2024-09-18T07:46:51.378 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00092621499999999998 2024-09-18T07:46:51.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.378 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.378 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.378 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.565466+0000", 2024-09-18T07:46:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1239999999999998e-05 2024-09-18T07:46:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:156465 9.e 9:710e39e4:::benchmark_data_smithi137_126072_object23965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:46:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:24.404394+0000", 2024-09-18T07:46:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 326.94530029800001, 2024-09-18T07:46:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.157357105, 2024-09-18T07:46:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.379 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.380 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:46:51.380 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:46:51.380 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 156465 2024-09-18T07:46:51.380 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.380 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.380 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.380 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.380 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404394+0000", 2024-09-18T07:46:51.380 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.380 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.380 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.380 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.380 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404394+0000", 2024-09-18T07:46:51.380 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.381 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.381 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.381 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.381 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404395+0000", 2024-09-18T07:46:51.381 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.55e-06 2024-09-18T07:46:51.381 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.381 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.381 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.381 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404400+0000", 2024-09-18T07:46:51.381 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5299999999999998e-06 2024-09-18T07:46:51.381 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.381 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.381 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.382 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404410+0000", 2024-09-18T07:46:51.382 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0139999999999999e-05 2024-09-18T07:46:51.382 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.382 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.382 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.382 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404417+0000", 2024-09-18T07:46:51.382 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9979999999999997e-06 2024-09-18T07:46:51.382 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.382 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.382 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.382 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404520+0000", 2024-09-18T07:46:51.382 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000103601 2024-09-18T07:46:51.382 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.383 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.383 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.383 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404618+0000", 2024-09-18T07:46:51.383 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7317000000000004e-05 2024-09-18T07:46:51.383 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.383 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.383 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:46:51.383 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404713+0000", 2024-09-18T07:46:51.383 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5427000000000006e-05 2024-09-18T07:46:51.383 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.383 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.383 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.383 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409836+0000", 2024-09-18T07:46:51.384 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0051231840000000002 2024-09-18T07:46:51.384 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.384 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.384 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:46:51.384 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409962+0000", 2024-09-18T07:46:51.384 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012611299999999999 2024-09-18T07:46:51.384 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.384 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.384 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.384 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.561643+0000", 2024-09-18T07:46:51.384 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15168100000000001 2024-09-18T07:46:51.384 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.384 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.384 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.385 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.561701+0000", 2024-09-18T07:46:51.385 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8010000000000002e-05 2024-09-18T07:46:51.385 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.385 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.385 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.385 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.561751+0000", 2024-09-18T07:46:51.385 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9234999999999999e-05 2024-09-18T07:46:51.385 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.385 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.385 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.385 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.385 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.385 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:156469 9.e 9:772a9531:::benchmark_data_smithi137_126072_object23969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:46:51.386 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:24.410883+0000", 2024-09-18T07:46:51.386 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 326.93881109799997, 2024-09-18T07:46:51.386 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15481848500000001, 2024-09-18T07:46:51.386 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.386 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.386 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.386 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.386 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:46:51.386 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:46:51.386 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 156469 2024-09-18T07:46:51.386 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.386 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.386 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.386 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.387 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.410883+0000", 2024-09-18T07:46:51.387 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.387 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.387 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.387 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.387 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.410883+0000", 2024-09-18T07:46:51.387 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.387 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.387 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.387 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.387 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.410885+0000", 2024-09-18T07:46:51.387 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.979e-06 2024-09-18T07:46:51.387 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.388 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.388 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.388 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.410891+0000", 2024-09-18T07:46:51.388 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1840000000000004e-06 2024-09-18T07:46:51.388 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.388 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.388 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.388 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.410905+0000", 2024-09-18T07:46:51.388 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4449e-05 2024-09-18T07:46:51.388 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.388 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.388 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.388 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.410913+0000", 2024-09-18T07:46:51.388 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6190000000000002e-06 2024-09-18T07:46:51.389 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.389 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.389 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.389 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.410967+0000", 2024-09-18T07:46:51.389 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4381e-05 2024-09-18T07:46:51.389 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.389 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.389 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.389 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.411099+0000", 2024-09-18T07:46:51.389 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013190500000000001 2024-09-18T07:46:51.389 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.389 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.389 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:46:51.389 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.411214+0000", 2024-09-18T07:46:51.390 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000114537 2024-09-18T07:46:51.390 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.390 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.390 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:46:51.390 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.413455+0000", 2024-09-18T07:46:51.390 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022413810000000002 2024-09-18T07:46:51.390 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.390 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.390 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.390 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.414426+0000", 2024-09-18T07:46:51.390 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00097079800000000004 2024-09-18T07:46:51.390 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.390 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.390 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.390 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.565613+0000", 2024-09-18T07:46:51.391 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15118668599999999 2024-09-18T07:46:51.391 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.391 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.391 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.391 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.565663+0000", 2024-09-18T07:46:51.391 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0096999999999998e-05 2024-09-18T07:46:51.391 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.391 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.391 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.391 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.565701+0000", 2024-09-18T07:46:51.391 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8469e-05 2024-09-18T07:46:51.391 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.391 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.392 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.392 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.392 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.392 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:156472 9.e 9:78b53fba:::benchmark_data_smithi137_126072_object23972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:46:51.392 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:24.415983+0000", 2024-09-18T07:46:51.392 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 326.933710378, 2024-09-18T07:46:51.392 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15303439799999999, 2024-09-18T07:46:51.392 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.392 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.392 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.392 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.392 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:46:51.392 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:46:51.392 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 156472 2024-09-18T07:46:51.393 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.393 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.393 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.393 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.393 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.415983+0000", 2024-09-18T07:46:51.393 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.393 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.393 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.393 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.393 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.415983+0000", 2024-09-18T07:46:51.393 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.393 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.393 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.393 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.393 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.415985+0000", 2024-09-18T07:46:51.394 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5409999999999999e-06 2024-09-18T07:46:51.394 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.394 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.394 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.394 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.415990+0000", 2024-09-18T07:46:51.394 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2719999999999997e-06 2024-09-18T07:46:51.394 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.394 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.394 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.394 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.416001+0000", 2024-09-18T07:46:51.394 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0973e-05 2024-09-18T07:46:51.394 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.394 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.395 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.395 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.416007+0000", 2024-09-18T07:46:51.395 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8259999999999998e-06 2024-09-18T07:46:51.395 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.395 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.395 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.395 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.416049+0000", 2024-09-18T07:46:51.395 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1869000000000002e-05 2024-09-18T07:46:51.395 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.395 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.395 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.395 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.416132+0000", 2024-09-18T07:46:51.395 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3554999999999995e-05 2024-09-18T07:46:51.395 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.396 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.396 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:46:51.396 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.416228+0000", 2024-09-18T07:46:51.396 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.522e-05 2024-09-18T07:46:51.396 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.396 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.396 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:46:51.396 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.417966+0000", 2024-09-18T07:46:51.396 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0017378579999999999 2024-09-18T07:46:51.396 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.396 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.396 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.396 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.418190+0000", 2024-09-18T07:46:51.396 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00022425 2024-09-18T07:46:51.397 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.397 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.397 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.397 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.568914+0000", 2024-09-18T07:46:51.397 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.150724675 2024-09-18T07:46:51.397 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.397 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.397 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.397 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.568976+0000", 2024-09-18T07:46:51.397 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1675999999999997e-05 2024-09-18T07:46:51.397 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.397 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.397 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.398 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.569018+0000", 2024-09-18T07:46:51.398 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1683000000000001e-05 2024-09-18T07:46:51.398 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.398 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.398 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.398 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.398 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.398 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:172965 9.e 9:7fd9b0fe:::benchmark_data_smithi137_126072_object40465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-18T07:46:51.398 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:38.554115+0000", 2024-09-18T07:46:51.398 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 312.79557915800001, 2024-09-18T07:46:51.398 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.179108299, 2024-09-18T07:46:51.398 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.399 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.399 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.399 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.399 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:46:51.399 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:46:51.399 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 172965 2024-09-18T07:46:51.399 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.399 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.399 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.399 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.399 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554115+0000", 2024-09-18T07:46:51.399 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.399 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.400 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.400 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.400 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554115+0000", 2024-09-18T07:46:51.400 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.400 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.400 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.400 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.400 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554116+0000", 2024-09-18T07:46:51.400 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.668e-06 2024-09-18T07:46:51.400 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.400 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.400 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.400 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554122+0000", 2024-09-18T07:46:51.401 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9159999999999998e-06 2024-09-18T07:46:51.401 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.401 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.401 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.401 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554133+0000", 2024-09-18T07:46:51.401 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1134e-05 2024-09-18T07:46:51.401 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.401 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.401 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.401 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554139+0000", 2024-09-18T07:46:51.401 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1889999999999998e-06 2024-09-18T07:46:51.401 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.401 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.402 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.402 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554177+0000", 2024-09-18T07:46:51.402 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8720000000000002e-05 2024-09-18T07:46:51.402 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.402 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.402 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.402 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554264+0000", 2024-09-18T07:46:51.402 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6942999999999997e-05 2024-09-18T07:46:51.402 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.402 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.402 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:46:51.402 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554389+0000", 2024-09-18T07:46:51.402 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012495799999999999 2024-09-18T07:46:51.402 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.403 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.403 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.403 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.559699+0000", 2024-09-18T07:46:51.403 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0053099760000000001 2024-09-18T07:46:51.403 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.403 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.403 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:46:51.403 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.559715+0000", 2024-09-18T07:46:51.403 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5586999999999998e-05 2024-09-18T07:46:51.403 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.403 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.403 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.403 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733124+0000", 2024-09-18T07:46:51.404 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17340927 2024-09-18T07:46:51.404 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.404 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.404 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.404 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733181+0000", 2024-09-18T07:46:51.404 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6845000000000002e-05 2024-09-18T07:46:51.404 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.404 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.404 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.404 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733223+0000", 2024-09-18T07:46:51.404 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2092999999999998e-05 2024-09-18T07:46:51.404 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.404 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.404 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.405 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.405 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.405 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:172970 9.3 9:c7e8a587:::benchmark_data_smithi137_126072_object40470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-18T07:46:51.405 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:38.554401+0000", 2024-09-18T07:46:51.405 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 312.79529264199999, 2024-09-18T07:46:51.405 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17690262100000001, 2024-09-18T07:46:51.405 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.405 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.405 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.405 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.405 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:46:51.405 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:46:51.405 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 172970 2024-09-18T07:46:51.406 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.406 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.406 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.406 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.406 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554401+0000", 2024-09-18T07:46:51.406 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.406 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.406 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.406 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.406 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554401+0000", 2024-09-18T07:46:51.406 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.406 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.406 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.407 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.407 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554402+0000", 2024-09-18T07:46:51.407 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.299e-06 2024-09-18T07:46:51.407 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.407 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.407 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.407 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554407+0000", 2024-09-18T07:46:51.407 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6870000000000002e-06 2024-09-18T07:46:51.407 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.407 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.407 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.407 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554418+0000", 2024-09-18T07:46:51.407 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0696000000000001e-05 2024-09-18T07:46:51.408 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.408 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.408 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.408 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554426+0000", 2024-09-18T07:46:51.408 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0989999999999999e-06 2024-09-18T07:46:51.408 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.408 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.408 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.408 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554503+0000", 2024-09-18T07:46:51.408 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6638999999999996e-05 2024-09-18T07:46:51.408 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.408 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.408 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.408 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554615+0000", 2024-09-18T07:46:51.409 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000112816 2024-09-18T07:46:51.409 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.409 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.409 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:46:51.409 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554762+0000", 2024-09-18T07:46:51.409 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014699099999999999 2024-09-18T07:46:51.409 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.409 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.409 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:46:51.409 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.558265+0000", 2024-09-18T07:46:51.409 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0035024539999999999 2024-09-18T07:46:51.409 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.409 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.409 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.410 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.564048+0000", 2024-09-18T07:46:51.410 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0057829969999999998 2024-09-18T07:46:51.410 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.410 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.410 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.410 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731227+0000", 2024-09-18T07:46:51.410 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16717880500000001 2024-09-18T07:46:51.410 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.410 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.410 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.410 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731260+0000", 2024-09-18T07:46:51.410 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2845000000000001e-05 2024-09-18T07:46:51.410 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.411 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.411 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.411 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731304+0000", 2024-09-18T07:46:51.411 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4292999999999998e-05 2024-09-18T07:46:51.411 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.411 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.411 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.411 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.411 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.411 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:172972 9.e 9:796b9798:::benchmark_data_smithi137_126072_object40472:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-18T07:46:51.411 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:38.556110+0000", 2024-09-18T07:46:51.411 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 312.79358382599997, 2024-09-18T07:46:51.411 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17726146800000001, 2024-09-18T07:46:51.411 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.413 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.413 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.413 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.413 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:46:51.413 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:46:51.413 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 172972 2024-09-18T07:46:51.413 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.413 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.413 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.413 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.413 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556110+0000", 2024-09-18T07:46:51.413 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.413 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.414 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.414 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.414 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556110+0000", 2024-09-18T07:46:51.414 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.414 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.414 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.414 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.414 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556112+0000", 2024-09-18T07:46:51.414 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.711e-06 2024-09-18T07:46:51.414 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.414 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:51.416 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.416 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556119+0000", 2024-09-18T07:46:51.416 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.644e-06 2024-09-18T07:46:51.416 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.416 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.416 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.416 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556129+0000", 2024-09-18T07:46:51.416 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9850000000000001e-06 2024-09-18T07:46:51.417 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.417 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.417 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.417 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556136+0000", 2024-09-18T07:46:51.417 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7789999999999998e-06 2024-09-18T07:46:51.417 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.417 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.417 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.417 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.557007+0000", 2024-09-18T07:46:51.417 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00087092099999999998 2024-09-18T07:46:51.417 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.417 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.417 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.418 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.557105+0000", 2024-09-18T07:46:51.418 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7640999999999996e-05 2024-09-18T07:46:51.418 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.418 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.418 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:46:51.418 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.557203+0000", 2024-09-18T07:46:51.418 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8666999999999996e-05 2024-09-18T07:46:51.418 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.418 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.418 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.418 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.559808+0000", 2024-09-18T07:46:51.418 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026044599999999998 2024-09-18T07:46:51.418 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.418 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.419 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:46:51.419 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.559832+0000", 2024-09-18T07:46:51.419 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4426e-05 2024-09-18T07:46:51.419 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.419 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.419 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.419 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733280+0000", 2024-09-18T07:46:51.419 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17344823700000001 2024-09-18T07:46:51.419 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.419 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.419 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.419 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733334+0000", 2024-09-18T07:46:51.419 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3946999999999997e-05 2024-09-18T07:46:51.419 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.420 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.420 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.420 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733371+0000", 2024-09-18T07:46:51.420 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7049999999999999e-05 2024-09-18T07:46:51.420 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.420 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.420 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.420 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.420 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.420 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:172980 9.3 9:ce0a7506:::benchmark_data_smithi137_126072_object40480:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-18T07:46:51.420 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:38.560199+0000", 2024-09-18T07:46:51.420 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 312.78949499700002, 2024-09-18T07:46:51.420 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.171283607, 2024-09-18T07:46:51.421 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.421 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.421 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.421 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.421 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:46:51.421 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:46:51.421 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 172980 2024-09-18T07:46:51.421 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.421 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.421 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.421 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.421 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560199+0000", 2024-09-18T07:46:51.421 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.422 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.422 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.422 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.422 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560199+0000", 2024-09-18T07:46:51.422 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.422 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.422 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.422 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.422 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560201+0000", 2024-09-18T07:46:51.422 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9709999999999998e-06 2024-09-18T07:46:51.422 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.422 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.422 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.422 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560206+0000", 2024-09-18T07:46:51.423 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3750000000000002e-06 2024-09-18T07:46:51.423 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.423 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.423 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.423 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560221+0000", 2024-09-18T07:46:51.423 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.487e-05 2024-09-18T07:46:51.423 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.423 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.423 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.423 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560228+0000", 2024-09-18T07:46:51.423 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4200000000000001e-06 2024-09-18T07:46:51.423 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.423 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.423 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.424 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560284+0000", 2024-09-18T07:46:51.424 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5084999999999999e-05 2024-09-18T07:46:51.424 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.424 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.424 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.424 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560415+0000", 2024-09-18T07:46:51.424 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013142500000000001 2024-09-18T07:46:51.424 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.424 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.424 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:46:51.424 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560525+0000", 2024-09-18T07:46:51.424 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010991400000000001 2024-09-18T07:46:51.424 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.424 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.425 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.425 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.564084+0000", 2024-09-18T07:46:51.425 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00355953 2024-09-18T07:46:51.425 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.425 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.425 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:46:51.425 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.566190+0000", 2024-09-18T07:46:51.425 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021053690000000002 2024-09-18T07:46:51.425 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.425 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.425 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.425 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731396+0000", 2024-09-18T07:46:51.425 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16520663299999999 2024-09-18T07:46:51.426 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.426 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.426 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.426 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731446+0000", 2024-09-18T07:46:51.426 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9199000000000001e-05 2024-09-18T07:46:51.426 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.426 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.426 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.426 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731482+0000", 2024-09-18T07:46:51.426 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6816000000000001e-05 2024-09-18T07:46:51.426 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.426 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.426 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.426 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.426 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.427 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:172986 9.d 9:b4a569cc:::benchmark_data_smithi137_126072_object40486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-18T07:46:51.427 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:38.568405+0000", 2024-09-18T07:46:51.427 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 312.78128843000002, 2024-09-18T07:46:51.427 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16278958299999999, 2024-09-18T07:46:51.427 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.427 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.427 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.427 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.427 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:46:51.427 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:46:51.427 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 172986 2024-09-18T07:46:51.427 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.427 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.427 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.428 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.428 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568405+0000", 2024-09-18T07:46:51.428 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.428 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.428 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.428 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.428 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568405+0000", 2024-09-18T07:46:51.428 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.428 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.428 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.428 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.428 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568408+0000", 2024-09-18T07:46:51.428 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6929999999999999e-06 2024-09-18T07:46:51.429 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.429 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.429 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.429 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568414+0000", 2024-09-18T07:46:51.429 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5219999999999998e-06 2024-09-18T07:46:51.429 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.429 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.429 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.429 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568428+0000", 2024-09-18T07:46:51.429 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4134000000000001e-05 2024-09-18T07:46:51.429 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.429 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.430 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.430 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568435+0000", 2024-09-18T07:46:51.430 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.396e-06 2024-09-18T07:46:51.430 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.430 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.430 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.430 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568512+0000", 2024-09-18T07:46:51.430 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6614999999999997e-05 2024-09-18T07:46:51.430 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.430 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.431 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.431 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568623+0000", 2024-09-18T07:46:51.431 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011123799999999999 2024-09-18T07:46:51.431 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.431 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.431 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:46:51.431 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568744+0000", 2024-09-18T07:46:51.431 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012053900000000001 2024-09-18T07:46:51.431 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.431 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.431 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:46:51.431 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.571214+0000", 2024-09-18T07:46:51.431 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0024707739999999998 2024-09-18T07:46:51.432 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.432 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.432 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.432 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.571292+0000", 2024-09-18T07:46:51.432 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7903000000000003e-05 2024-09-18T07:46:51.432 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.432 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.432 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.432 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731105+0000", 2024-09-18T07:46:51.432 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.159812378 2024-09-18T07:46:51.432 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.432 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.432 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.432 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731154+0000", 2024-09-18T07:46:51.433 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9450000000000003e-05 2024-09-18T07:46:51.433 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.433 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.433 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.433 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731195+0000", 2024-09-18T07:46:51.433 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0941e-05 2024-09-18T07:46:51.433 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.433 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.433 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.433 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.433 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.433 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:244796 9.d 9:b8ad6f51:::benchmark_data_smithi137_126072_object112296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-18T07:46:51.434 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:36.228692+0000", 2024-09-18T07:46:51.434 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 255.121001831, 2024-09-18T07:46:51.434 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16804142, 2024-09-18T07:46:51.434 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.434 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.434 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.434 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.434 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:46:51.434 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:46:51.434 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 244796 2024-09-18T07:46:51.434 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.434 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.434 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.435 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.435 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228692+0000", 2024-09-18T07:46:51.435 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.435 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.435 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.435 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.435 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228692+0000", 2024-09-18T07:46:51.435 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.435 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.435 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.435 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.435 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228693+0000", 2024-09-18T07:46:51.435 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1930000000000001e-06 2024-09-18T07:46:51.435 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.436 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.436 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.436 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228697+0000", 2024-09-18T07:46:51.436 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.143e-06 2024-09-18T07:46:51.436 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.436 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.436 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.436 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228707+0000", 2024-09-18T07:46:51.436 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3489999999999998e-06 2024-09-18T07:46:51.436 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.436 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.436 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.436 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228713+0000", 2024-09-18T07:46:51.436 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1720000000000004e-06 2024-09-18T07:46:51.437 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.437 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.437 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.437 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233077+0000", 2024-09-18T07:46:51.437 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0043645890000000003 2024-09-18T07:46:51.437 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.437 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.437 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.437 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233186+0000", 2024-09-18T07:46:51.437 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108081 2024-09-18T07:46:51.437 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.437 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.437 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:46:51.437 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233291+0000", 2024-09-18T07:46:51.438 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000105025 2024-09-18T07:46:51.438 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.438 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.438 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:46:51.438 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.240477+0000", 2024-09-18T07:46:51.438 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0071862430000000001 2024-09-18T07:46:51.438 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.438 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.438 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.438 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.248094+0000", 2024-09-18T07:46:51.438 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0076167459999999998 2024-09-18T07:46:51.438 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.438 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.438 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.439 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.396661+0000", 2024-09-18T07:46:51.439 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14856715000000001 2024-09-18T07:46:51.439 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.439 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.439 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.439 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.396703+0000", 2024-09-18T07:46:51.439 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1850000000000001e-05 2024-09-18T07:46:51.439 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.439 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.439 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.439 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.396733+0000", 2024-09-18T07:46:51.439 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0879e-05 2024-09-18T07:46:51.439 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.439 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.439 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.440 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.440 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.440 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:244803 9.d 9:be42e048:::benchmark_data_smithi137_126072_object112303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-18T07:46:51.440 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:36.233785+0000", 2024-09-18T07:46:51.440 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 255.115908391, 2024-09-18T07:46:51.440 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16784005699999999, 2024-09-18T07:46:51.440 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.440 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.440 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.440 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.440 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:46:51.440 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:46:51.440 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 244803 2024-09-18T07:46:51.441 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.441 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.441 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.441 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.441 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233785+0000", 2024-09-18T07:46:51.441 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.441 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.441 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.441 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.441 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233785+0000", 2024-09-18T07:46:51.441 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.441 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.441 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.441 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.442 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233787+0000", 2024-09-18T07:46:51.442 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8279999999999999e-06 2024-09-18T07:46:51.442 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.442 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.442 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.442 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233793+0000", 2024-09-18T07:46:51.442 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7790000000000001e-06 2024-09-18T07:46:51.442 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.442 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.442 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.442 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233803+0000", 2024-09-18T07:46:51.442 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5920000000000007e-06 2024-09-18T07:46:51.442 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.442 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.442 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.443 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233810+0000", 2024-09-18T07:46:51.443 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7300000000000005e-06 2024-09-18T07:46:51.443 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.443 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.443 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.443 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.237384+0000", 2024-09-18T07:46:51.443 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003573629 2024-09-18T07:46:51.443 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.443 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.443 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.443 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.237500+0000", 2024-09-18T07:46:51.443 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000116216 2024-09-18T07:46:51.443 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.443 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.444 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:46:51.444 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.237637+0000", 2024-09-18T07:46:51.444 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013664699999999999 2024-09-18T07:46:51.444 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.444 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.444 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:46:51.444 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.243662+0000", 2024-09-18T07:46:51.444 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0060250429999999999 2024-09-18T07:46:51.444 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.444 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.444 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.444 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.250643+0000", 2024-09-18T07:46:51.444 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069812499999999996 2024-09-18T07:46:51.444 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.444 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.445 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.445 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401571+0000", 2024-09-18T07:46:51.445 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15092767600000001 2024-09-18T07:46:51.445 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.445 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.445 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.445 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401601+0000", 2024-09-18T07:46:51.445 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9889000000000001e-05 2024-09-18T07:46:51.445 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.445 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.445 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.445 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401625+0000", 2024-09-18T07:46:51.445 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4777999999999999e-05 2024-09-18T07:46:51.446 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.446 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.446 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.446 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.446 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.446 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:244820 9.d 9:b40b063b:::benchmark_data_smithi137_126072_object112320:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-18T07:46:51.446 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:36.251409+0000", 2024-09-18T07:46:51.446 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 255.09828445599999, 2024-09-18T07:46:51.446 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.150296187, 2024-09-18T07:46:51.446 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.446 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.446 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.446 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.446 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:46:51.447 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:46:51.447 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 244820 2024-09-18T07:46:51.447 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.447 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.447 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.447 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.447 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251409+0000", 2024-09-18T07:46:51.447 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.447 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.447 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.447 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.447 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251409+0000", 2024-09-18T07:46:51.447 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.448 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.448 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.448 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.448 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251411+0000", 2024-09-18T07:46:51.448 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.84e-06 2024-09-18T07:46:51.448 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.448 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.448 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.448 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251417+0000", 2024-09-18T07:46:51.448 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8300000000000001e-06 2024-09-18T07:46:51.448 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.448 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.449 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.449 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251428+0000", 2024-09-18T07:46:51.449 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1008e-05 2024-09-18T07:46:51.449 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.449 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.449 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.449 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251435+0000", 2024-09-18T07:46:51.449 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5019999999999998e-06 2024-09-18T07:46:51.449 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.449 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.449 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.449 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251496+0000", 2024-09-18T07:46:51.449 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1353999999999996e-05 2024-09-18T07:46:51.450 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.450 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.450 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.450 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251578+0000", 2024-09-18T07:46:51.450 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1966999999999996e-05 2024-09-18T07:46:51.450 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.450 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.450 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:46:51.450 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.251645+0000", 2024-09-18T07:46:51.450 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7331999999999998e-05 2024-09-18T07:46:51.450 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.450 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.450 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:46:51.451 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.256879+0000", 2024-09-18T07:46:51.451 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.005234069 2024-09-18T07:46:51.451 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.451 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.451 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.451 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.257762+0000", 2024-09-18T07:46:51.451 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00088236200000000003 2024-09-18T07:46:51.451 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.451 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.451 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.451 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401653+0000", 2024-09-18T07:46:51.451 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14389142399999999 2024-09-18T07:46:51.451 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.452 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.452 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.452 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401681+0000", 2024-09-18T07:46:51.452 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7861e-05 2024-09-18T07:46:51.452 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.452 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.452 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.452 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401706+0000", 2024-09-18T07:46:51.452 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4638e-05 2024-09-18T07:46:51.452 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.452 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.452 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.452 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.453 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.453 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:110495 10.c 10:363a02dd:::benchmark_data_smithi137_151160_object110494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-18T07:46:51.453 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:44:18.824005+0000", 2024-09-18T07:46:51.453 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 152.52568858699999, 2024-09-18T07:46:51.453 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15565354300000001, 2024-09-18T07:46:51.453 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.453 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.453 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.453 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.453 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:46:51.453 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:51.453 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 110495 2024-09-18T07:46:51.454 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.454 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.454 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.454 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.454 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.824005+0000", 2024-09-18T07:46:51.454 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.454 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.454 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.454 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.454 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.824005+0000", 2024-09-18T07:46:51.455 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.455 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.455 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.455 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.455 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.824007+0000", 2024-09-18T07:46:51.455 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.863e-06 2024-09-18T07:46:51.455 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.455 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.455 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.455 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.824055+0000", 2024-09-18T07:46:51.455 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7966000000000001e-05 2024-09-18T07:46:51.455 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.456 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.456 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.456 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.824065+0000", 2024-09-18T07:46:51.456 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0335e-05 2024-09-18T07:46:51.456 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.456 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.456 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.456 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.824071+0000", 2024-09-18T07:46:51.456 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6679999999999998e-06 2024-09-18T07:46:51.456 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.456 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.456 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.457 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.824118+0000", 2024-09-18T07:46:51.457 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6647999999999998e-05 2024-09-18T07:46:51.457 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.457 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.457 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.457 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.824199+0000", 2024-09-18T07:46:51.457 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0798999999999997e-05 2024-09-18T07:46:51.457 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.457 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.457 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0", 2024-09-18T07:46:51.457 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.824285+0000", 2024-09-18T07:46:51.457 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6700000000000007e-05 2024-09-18T07:46:51.457 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.458 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.458 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:46:51.458 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.826613+0000", 2024-09-18T07:46:51.458 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023272969999999999 2024-09-18T07:46:51.458 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.458 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.458 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.458 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.831822+0000", 2024-09-18T07:46:51.458 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0052095500000000003 2024-09-18T07:46:51.458 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.458 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.458 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.458 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.831904+0000", 2024-09-18T07:46:51.458 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1815000000000002e-05 2024-09-18T07:46:51.459 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.459 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.459 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.459 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.979659+0000", 2024-09-18T07:46:51.460 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14775490199999999 2024-09-18T07:46:51.460 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.460 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.461 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.461 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.461 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.461 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:110498 10.1 10:837793f7:::benchmark_data_smithi137_151160_object110497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-18T07:46:51.461 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:44:18.828033+0000", 2024-09-18T07:46:51.461 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 152.52166110300001, 2024-09-18T07:46:51.461 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.151784697, 2024-09-18T07:46:51.461 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.461 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.461 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.461 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.461 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:46:51.461 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:51.462 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 110498 2024-09-18T07:46:51.462 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.462 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.462 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.462 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.462 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.828033+0000", 2024-09-18T07:46:51.462 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.462 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.462 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.462 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.462 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.828033+0000", 2024-09-18T07:46:51.462 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.463 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.463 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.463 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.463 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.828034+0000", 2024-09-18T07:46:51.463 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.424e-06 2024-09-18T07:46:51.463 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.463 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.463 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.463 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.828086+0000", 2024-09-18T07:46:51.463 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2187e-05 2024-09-18T07:46:51.463 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.463 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.463 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.463 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.828095+0000", 2024-09-18T07:46:51.463 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8349999999999993e-06 2024-09-18T07:46:51.464 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.464 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.464 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.464 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.828101+0000", 2024-09-18T07:46:51.464 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8510000000000004e-06 2024-09-18T07:46:51.464 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.464 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.464 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.464 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.828143+0000", 2024-09-18T07:46:51.464 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2466999999999998e-05 2024-09-18T07:46:51.464 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.464 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.464 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.464 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.828221+0000", 2024-09-18T07:46:51.465 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7140999999999999e-05 2024-09-18T07:46:51.465 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.465 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.465 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0", 2024-09-18T07:46:51.465 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.828311+0000", 2024-09-18T07:46:51.465 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0297000000000006e-05 2024-09-18T07:46:51.465 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.465 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.465 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:46:51.465 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.830402+0000", 2024-09-18T07:46:51.465 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002091323 2024-09-18T07:46:51.465 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.465 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.466 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.466 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.979674+0000", 2024-09-18T07:46:51.466 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14927216300000001 2024-09-18T07:46:51.466 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.466 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.466 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.466 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.979734+0000", 2024-09-18T07:46:51.466 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9679000000000002e-05 2024-09-18T07:46:51.466 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.466 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.466 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.466 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:18.979817+0000", 2024-09-18T07:46:51.466 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3330000000000003e-05 2024-09-18T07:46:51.466 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.467 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.467 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.467 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.467 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.467 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:137610 10.c 10:342470b2:::benchmark_data_smithi137_151160_object1794:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-18T07:46:51.467 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:44:42.767132+0000", 2024-09-18T07:46:51.467 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 128.58256144200001, 2024-09-18T07:46:51.467 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0830141860000002, 2024-09-18T07:46:51.467 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.467 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.467 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:46:51.467 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.467 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:46:51.468 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:51.468 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 137610 2024-09-18T07:46:51.468 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.468 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.468 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.468 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.468 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767132+0000", 2024-09-18T07:46:51.468 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.468 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.468 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.468 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.468 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767132+0000", 2024-09-18T07:46:51.468 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.468 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.469 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.469 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.469 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767134+0000", 2024-09-18T07:46:51.469 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.703e-06 2024-09-18T07:46:51.469 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.469 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.469 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.469 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767139+0000", 2024-09-18T07:46:51.469 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7430000000000002e-06 2024-09-18T07:46:51.469 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.469 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.469 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.469 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767152+0000", 2024-09-18T07:46:51.470 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2957e-05 2024-09-18T07:46:51.470 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.470 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.470 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.470 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767160+0000", 2024-09-18T07:46:51.470 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7740000000000004e-06 2024-09-18T07:46:51.470 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.470 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.470 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.470 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767214+0000", 2024-09-18T07:46:51.470 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4163000000000003e-05 2024-09-18T07:46:51.470 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.470 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.470 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.470 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767322+0000", 2024-09-18T07:46:51.471 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108028 2024-09-18T07:46:51.471 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.471 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.471 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0", 2024-09-18T07:46:51.471 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767426+0000", 2024-09-18T07:46:51.471 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000104442 2024-09-18T07:46:51.471 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.471 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.471 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:46:51.471 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.770704+0000", 2024-09-18T07:46:51.471 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0032773479999999998 2024-09-18T07:46:51.471 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.471 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.472 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.472 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.798119+0000", 2024-09-18T07:46:51.472 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.027415001000000001 2024-09-18T07:46:51.472 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.472 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.472 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:46:51.472 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.798131+0000", 2024-09-18T07:46:51.472 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2952999999999999e-05 2024-09-18T07:46:51.472 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.472 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.472 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.472 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:43.770440+0000", 2024-09-18T07:46:51.472 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97230850300000005 2024-09-18T07:46:51.472 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.472 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.473 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:46:51.473 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:43.770754+0000", 2024-09-18T07:46:51.473 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00031391299999999997 2024-09-18T07:46:51.473 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.473 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.473 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.473 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.850064+0000", 2024-09-18T07:46:51.473 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0793099289999999 2024-09-18T07:46:51.473 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.473 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.473 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.473 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.850147+0000", 2024-09-18T07:46:51.473 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2729e-05 2024-09-18T07:46:51.473 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.473 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.474 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.474 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.474 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.474 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:137634 10.c 10:3000b92f:::benchmark_data_smithi137_151160_object1818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-09-18T07:46:51.474 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:44:42.837861+0000", 2024-09-18T07:46:51.474 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 128.511832946, 2024-09-18T07:46:51.474 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0161026510000002, 2024-09-18T07:46:51.474 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.474 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.474 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:46:51.474 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.474 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:46:51.474 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:46:51.474 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 137634 2024-09-18T07:46:51.475 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.475 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.475 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.475 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.475 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.837861+0000", 2024-09-18T07:46:51.475 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.475 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.475 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.475 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.475 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.837861+0000", 2024-09-18T07:46:51.475 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.475 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.476 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.476 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.476 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.837863+0000", 2024-09-18T07:46:51.476 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.86e-06 2024-09-18T07:46:51.476 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.476 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.476 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.476 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.837868+0000", 2024-09-18T07:46:51.476 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5010000000000004e-06 2024-09-18T07:46:51.476 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.476 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.476 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.476 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.837881+0000", 2024-09-18T07:46:51.476 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2317000000000001e-05 2024-09-18T07:46:51.477 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.477 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.477 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.477 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.837887+0000", 2024-09-18T07:46:51.477 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0220000000000001e-06 2024-09-18T07:46:51.477 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.477 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.477 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.477 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.837935+0000", 2024-09-18T07:46:51.477 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8831000000000001e-05 2024-09-18T07:46:51.477 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.477 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.477 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:46:51.478 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.837942+0000", 2024-09-18T07:46:51.478 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2979999999999997e-06 2024-09-18T07:46:51.478 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.478 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.478 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.478 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:43.770955+0000", 2024-09-18T07:46:51.478 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93301373200000004 2024-09-18T07:46:51.478 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.478 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.478 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:46:51.478 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:43.770964+0000", 2024-09-18T07:46:51.478 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0170000000000002e-06 2024-09-18T07:46:51.478 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.479 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.479 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.479 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.850379+0000", 2024-09-18T07:46:51.479 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.079414546 2024-09-18T07:46:51.479 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.479 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.479 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.479 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.850502+0000", 2024-09-18T07:46:51.479 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012337900000000001 2024-09-18T07:46:51.479 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.479 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.479 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,4", 2024-09-18T07:46:51.479 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.850640+0000", 2024-09-18T07:46:51.480 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000138107 2024-09-18T07:46:51.480 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.480 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.480 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.480 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.853493+0000", 2024-09-18T07:46:51.480 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0028527000000000001 2024-09-18T07:46:51.480 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.480 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.480 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:46:51.480 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.853731+0000", 2024-09-18T07:46:51.480 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023793600000000001 2024-09-18T07:46:51.480 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.480 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.481 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:46:51.481 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.853777+0000", 2024-09-18T07:46:51.481 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5531000000000002e-05 2024-09-18T07:46:51.481 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.481 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.481 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:46:51.481 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.853842+0000", 2024-09-18T07:46:51.481 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4917000000000002e-05 2024-09-18T07:46:51.481 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.481 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.481 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.481 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.853964+0000", 2024-09-18T07:46:51.481 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000121957 2024-09-18T07:46:51.482 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.482 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.482 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.482 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.482 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.482 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.34466.0:930 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e443)", 2024-09-18T07:46:51.482 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:46:39.586131+0000", 2024-09-18T07:46:51.482 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 11.763562366, 2024-09-18T07:46:51.482 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.30508526499999999, 2024-09-18T07:46:51.482 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:46:51.482 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:46:51.482 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:46:51.482 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:46:51.482 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.34466", 2024-09-18T07:46:51.483 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.203:0/952514737", 2024-09-18T07:46:51.483 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 930 2024-09-18T07:46:51.483 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.483 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:46:51.483 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.483 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:46:51.483 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.586131+0000", 2024-09-18T07:46:51.483 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.483 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.483 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.483 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:46:51.483 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.586131+0000", 2024-09-18T07:46:51.483 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:46:51.483 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.484 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.484 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:46:51.484 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.586135+0000", 2024-09-18T07:46:51.484 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.721e-06 2024-09-18T07:46:51.484 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.484 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.484 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:46:51.484 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.586149+0000", 2024-09-18T07:46:51.484 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4022000000000001e-05 2024-09-18T07:46:51.484 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.484 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.484 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:46:51.484 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.586174+0000", 2024-09-18T07:46:51.484 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5201000000000001e-05 2024-09-18T07:46:51.485 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.485 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.485 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:46:51.485 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.586185+0000", 2024-09-18T07:46:51.485 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0937e-05 2024-09-18T07:46:51.485 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.485 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.485 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.485 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.586289+0000", 2024-09-18T07:46:51.485 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010391999999999999 2024-09-18T07:46:51.485 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.485 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.485 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:46:51.486 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.586300+0000", 2024-09-18T07:46:51.486 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1219999999999999e-05 2024-09-18T07:46:51.486 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.486 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.486 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:46:51.486 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.890897+0000", 2024-09-18T07:46:51.486 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.30459635200000001 2024-09-18T07:46:51.486 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.486 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.486 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:46:51.486 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.891053+0000", 2024-09-18T07:46:51.486 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015600600000000001 2024-09-18T07:46:51.486 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:46:51.486 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:46:51.487 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:46:51.487 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.891217+0000", 2024-09-18T07:46:51.487 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000163886 2024-09-18T07:46:51.487 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.487 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.487 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.487 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:46:51.487 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:46:51.487 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:46:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:51.488 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:51.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:51.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:51.623 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:51.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:51.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:51.725 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:51.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:51.759 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:51.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:51.775 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:46:51.775 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:46:51.809 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:46:51.810 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:46:51.840 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:46:51.841 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:46:51.860 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:51.876 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:46:51.876 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:46:51.893 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:51.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:51.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:51.908 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:46:51.908 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:46:51.911 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-18T07:46:51.940 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:46:51.941 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:46:51.978 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:46:51.979 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:46:51.995 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:52.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:52.031 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:52.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:52.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:52.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:52.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:52.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:52.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:51 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1032940890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:52.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:51 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1032940890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:52.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:52.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:52.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:52.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:52.408 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:52.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:51 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1032940890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:52.449 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:52.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:52.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:52.468 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T07:46:52.552 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:52.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:52.605 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:52.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:52.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:52.728 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:52.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:52.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:52.825 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:46:52.825 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:46:52.830 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:52.874 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:52.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:52.898 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:46:52.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:52.981 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 6] out_osds: [1] dead_osds: [5] live_osds: [0, 1, 2, 3, 6, 7, 4] 2024-09-18T07:46:52.981 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:46:52.982 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-09-18T07:46:52.982 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-09-18T07:46:52.982 DEBUG:teuthology.orchestra.run.smithi203:osd.5> sudo journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.5.service 2024-09-18T07:46:52.985 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl start ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.5 2024-09-18T07:46:53.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:46:53.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:53.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:53.048 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[29408]: 4.16 scrub starts 2024-09-18T07:46:53.049 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[29408]: 4.16 scrub ok 2024-09-18T07:46:53.049 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[29408]: 3.e scrub starts 2024-09-18T07:46:53.049 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[29408]: 3.e scrub ok 2024-09-18T07:46:53.049 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[29408]: pgmap v844: 158 pgs: 2 peering, 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 92 active+clean; 606 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.11k op/s; 3229/29676 objects degraded (10.881%); 33/29676 objects misplaced (0.111%); 22 MiB/s, 359 objects/s recovering 2024-09-18T07:46:53.049 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:46:53.049 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:46:53.049 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[29408]: 6.4 scrub starts 2024-09-18T07:46:53.049 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[29408]: 6.4 scrub ok 2024-09-18T07:46:53.049 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[39175]: 4.16 scrub starts 2024-09-18T07:46:53.050 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[39175]: 4.16 scrub ok 2024-09-18T07:46:53.050 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[39175]: 3.e scrub starts 2024-09-18T07:46:53.050 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[39175]: 3.e scrub ok 2024-09-18T07:46:53.050 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[39175]: pgmap v844: 158 pgs: 2 peering, 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 92 active+clean; 606 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.11k op/s; 3229/29676 objects degraded (10.881%); 33/29676 objects misplaced (0.111%); 22 MiB/s, 359 objects/s recovering 2024-09-18T07:46:53.050 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:46:53.050 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:46:53.050 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[39175]: 6.4 scrub starts 2024-09-18T07:46:53.050 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:52 smithi137 ceph-mon[39175]: 6.4 scrub ok 2024-09-18T07:46:53.113 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:53.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:53.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:53.257 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:53.306 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:53.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:53.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:53.328 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:53 smithi203 systemd[1]: Starting Ceph osd.5 for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:46:53.328 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:52 smithi203 ceph-mon[32237]: 4.16 scrub starts 2024-09-18T07:46:53.328 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:52 smithi203 ceph-mon[32237]: 4.16 scrub ok 2024-09-18T07:46:53.328 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:52 smithi203 ceph-mon[32237]: 3.e scrub starts 2024-09-18T07:46:53.328 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:52 smithi203 ceph-mon[32237]: 3.e scrub ok 2024-09-18T07:46:53.328 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:52 smithi203 ceph-mon[32237]: pgmap v844: 158 pgs: 2 peering, 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 92 active+clean; 606 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.11k op/s; 3229/29676 objects degraded (10.881%); 33/29676 objects misplaced (0.111%); 22 MiB/s, 359 objects/s recovering 2024-09-18T07:46:53.328 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:52 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:46:53.328 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:52 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:46:53.328 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:52 smithi203 ceph-mon[32237]: 6.4 scrub starts 2024-09-18T07:46:53.329 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:52 smithi203 ceph-mon[32237]: 6.4 scrub ok 2024-09-18T07:46:53.408 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:53.434 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:53.443 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:53.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:53.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:53.545 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:53.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:53.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:53.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:53.673 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:53.712 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:53.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:53.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:53.784 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:46:53.784 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:46:53.813 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:53.848 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:53.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:53.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:53.988 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:53.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:54.090 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:54.131 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:54.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:54.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:54.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:54 smithi137 ceph-mon[29408]: 3.0 scrub starts 2024-09-18T07:46:54.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:54 smithi137 ceph-mon[29408]: 3.0 scrub ok 2024-09-18T07:46:54.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:54 smithi137 ceph-mon[29408]: osdmap e452: 8 total, 7 up, 7 in 2024-09-18T07:46:54.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:54 smithi137 ceph-mon[29408]: 5.17 scrub starts 2024-09-18T07:46:54.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:54 smithi137 ceph-mon[29408]: 5.17 scrub ok 2024-09-18T07:46:54.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:54 smithi137 ceph-mon[39175]: 3.0 scrub starts 2024-09-18T07:46:54.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:54 smithi137 ceph-mon[39175]: 3.0 scrub ok 2024-09-18T07:46:54.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:54 smithi137 ceph-mon[39175]: osdmap e452: 8 total, 7 up, 7 in 2024-09-18T07:46:54.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:54 smithi137 ceph-mon[39175]: 5.17 scrub starts 2024-09-18T07:46:54.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:54 smithi137 ceph-mon[39175]: 5.17 scrub ok 2024-09-18T07:46:54.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:54.276 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:54.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:54.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:54.331 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:54 smithi203 ceph-mon[32237]: 3.0 scrub starts 2024-09-18T07:46:54.332 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:54 smithi203 ceph-mon[32237]: 3.0 scrub ok 2024-09-18T07:46:54.332 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:54 smithi203 ceph-mon[32237]: osdmap e452: 8 total, 7 up, 7 in 2024-09-18T07:46:54.332 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:54 smithi203 ceph-mon[32237]: 5.17 scrub starts 2024-09-18T07:46:54.332 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:54 smithi203 ceph-mon[32237]: 5.17 scrub ok 2024-09-18T07:46:54.377 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:54.415 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:54.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:54.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:54.516 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:54.548 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T07:46:54.551 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:54.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:54.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:54.653 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:54.694 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:54.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:54.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:54.826 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:46:54.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:54.828 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:54.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:54.906 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:54 smithi203 podman[131252]: 2024-09-18 07:46:54.608244284 +0000 UTC m=+0.559644825 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:46:54.906 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:54 smithi203 podman[131252]: 2024-09-18 07:46:54.73505294 +0000 UTC m=+0.686453477 container create 89d66ce80b7125661d618ba8928f35db4ec059aecdfe98a5944e02831ba24671 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-activate, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T07:46:54.928 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:54.968 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:54.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:55.041 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:46:55.070 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:55.103 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:55.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:55.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:55.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:55 smithi137 ceph-mon[29408]: 5.1a scrub starts 2024-09-18T07:46:55.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:55 smithi137 ceph-mon[29408]: 5.1a scrub ok 2024-09-18T07:46:55.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:55 smithi137 ceph-mon[29408]: pgmap v846: 158 pgs: 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 94 active+clean; 439 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 871 B/s rd, 0 B/s wr, 976 op/s; 2189/21633 objects degraded (10.119%); 33/21633 objects misplaced (0.153%); 1.4 MiB/s, 23 objects/s recovering 2024-09-18T07:46:55.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:55 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/616047341' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:46:55.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:55 smithi137 ceph-mon[29408]: from='client.26443 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:46:55.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:55 smithi137 ceph-mon[39175]: 5.1a scrub starts 2024-09-18T07:46:55.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:55 smithi137 ceph-mon[39175]: 5.1a scrub ok 2024-09-18T07:46:55.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:55 smithi137 ceph-mon[39175]: pgmap v846: 158 pgs: 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 94 active+clean; 439 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 871 B/s rd, 0 B/s wr, 976 op/s; 2189/21633 objects degraded (10.119%); 33/21633 objects misplaced (0.153%); 1.4 MiB/s, 23 objects/s recovering 2024-09-18T07:46:55.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:55 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/616047341' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:46:55.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:55 smithi137 ceph-mon[39175]: from='client.26443 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:46:55.205 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:55.216 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:55 smithi203 podman[131252]: 2024-09-18 07:46:55.030347255 +0000 UTC m=+0.981747790 container init 89d66ce80b7125661d618ba8928f35db4ec059aecdfe98a5944e02831ba24671 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T07:46:55.217 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:55 smithi203 podman[131252]: 2024-09-18 07:46:55.039957621 +0000 UTC m=+0.991358155 container start 89d66ce80b7125661d618ba8928f35db4ec059aecdfe98a5944e02831ba24671 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-18T07:46:55.217 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:55 smithi203 podman[131252]: 2024-09-18 07:46:55.1470058 +0000 UTC m=+1.098406336 container attach 89d66ce80b7125661d618ba8928f35db4ec059aecdfe98a5944e02831ba24671 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-18T07:46:55.225 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:55 smithi203 ceph-mon[32237]: 5.1a scrub starts 2024-09-18T07:46:55.225 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:55 smithi203 ceph-mon[32237]: 5.1a scrub ok 2024-09-18T07:46:55.225 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:55 smithi203 ceph-mon[32237]: pgmap v846: 158 pgs: 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 94 active+clean; 439 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 871 B/s rd, 0 B/s wr, 976 op/s; 2189/21633 objects degraded (10.119%); 33/21633 objects misplaced (0.153%); 1.4 MiB/s, 23 objects/s recovering 2024-09-18T07:46:55.226 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:55 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/616047341' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:46:55.226 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:55 smithi203 ceph-mon[32237]: from='client.26443 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:46:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:55.235 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:55.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:55.247 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:46:55.336 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:55.371 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:55.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:55.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:55.473 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:55.506 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:55.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:55.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:55.593 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:55 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-activate[131331]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-18T07:46:55.594 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:55 smithi203 bash[131252]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-18T07:46:55.594 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:55 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-activate[131331]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-18T07:46:55.594 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:55 smithi203 bash[131252]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-18T07:46:55.607 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:55.640 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:55.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:55.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:55.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:55.772 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:55.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:55.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:55.875 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:55.912 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:55.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:55.950 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:46:55.951 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:46:55.964 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:56.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:56.055 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:56.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:56.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:56.157 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:46:56.195 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:56.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:56.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:46:56.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:56 smithi203 ceph-mon[32237]: 4.19 scrub starts 2024-09-18T07:46:56.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:56 smithi203 ceph-mon[32237]: 4.19 scrub ok 2024-09-18T07:46:56.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:56 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:56.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:56 smithi203 ceph-mon[32237]: from='client.26443 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:46:56.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:56 smithi203 ceph-mon[32237]: osdmap e453: 8 total, 7 up, 7 in 2024-09-18T07:46:56.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:56 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 2189/21633 objects degraded (10.119%), 15 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-09-18T07:46:56.297 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:56.332 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:56.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:56.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:56.341 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:56 smithi137 ceph-mon[29408]: 4.19 scrub starts 2024-09-18T07:46:56.341 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:56 smithi137 ceph-mon[29408]: 4.19 scrub ok 2024-09-18T07:46:56.341 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:56 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:56.341 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:56 smithi137 ceph-mon[29408]: from='client.26443 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:46:56.341 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:56 smithi137 ceph-mon[29408]: osdmap e453: 8 total, 7 up, 7 in 2024-09-18T07:46:56.341 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:56 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 2189/21633 objects degraded (10.119%), 15 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-09-18T07:46:56.341 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:56 smithi137 ceph-mon[39175]: 4.19 scrub starts 2024-09-18T07:46:56.341 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:56 smithi137 ceph-mon[39175]: 4.19 scrub ok 2024-09-18T07:46:56.342 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:56 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:46:56.342 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:56 smithi137 ceph-mon[39175]: from='client.26443 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:46:56.342 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:56 smithi137 ceph-mon[39175]: osdmap e453: 8 total, 7 up, 7 in 2024-09-18T07:46:56.342 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:56 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 2189/21633 objects degraded (10.119%), 15 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-09-18T07:46:56.433 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:56.465 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:56.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:56.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:56.566 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:56.602 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:56.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:56.638 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Removed 135813 objects 2024-09-18T07:46:56.638 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Clean up completed and total clean up time :135.546 2024-09-18T07:46:56.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T07:46:56.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:56.724 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:56.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:56.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:56.826 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:56.848 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:56.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:56.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:56.900 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:46:56.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:56.972 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:56.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:56.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:57.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:57.095 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:57.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:57.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:57.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:57.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:57.296 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:57 smithi203 ceph-mon[32237]: pgmap v848: 158 pgs: 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 94 active+clean; 439 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 844 op/s; 2189/21633 objects degraded (10.119%); 33/21633 objects misplaced (0.153%); 1.4 MiB/s, 22 objects/s recovering 2024-09-18T07:46:57.296 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:57 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:46:57.296 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:57 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:46:57.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:57.341 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:57.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:57.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:57.412 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:46:57.412 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:46:57.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:57 smithi137 ceph-mon[29408]: pgmap v848: 158 pgs: 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 94 active+clean; 439 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 844 op/s; 2189/21633 objects degraded (10.119%); 33/21633 objects misplaced (0.153%); 1.4 MiB/s, 22 objects/s recovering 2024-09-18T07:46:57.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:57 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:46:57.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:57 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:46:57.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:46:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:46:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:46:57.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:57 smithi137 ceph-mon[39175]: pgmap v848: 158 pgs: 26 active+clean+remapped, 23 active+undersized, 15 active+undersized+degraded, 94 active+clean; 439 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 844 op/s; 2189/21633 objects degraded (10.119%); 33/21633 objects misplaced (0.153%); 1.4 MiB/s, 22 objects/s recovering 2024-09-18T07:46:57.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:57 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:46:57.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:57 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:46:57.442 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:46:57.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:57.467 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:57.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:46:57.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:57.590 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:57.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:57.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:57.710 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-activate[131331]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-18T07:46:57.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:57 smithi203 bash[131252]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-18T07:46:57.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-activate[131331]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-18T07:46:57.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:57 smithi203 bash[131252]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-18T07:46:57.714 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:57.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:57.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:57.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:46:57.839 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:57.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:57.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:57.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:57.947 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:46:57.947 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:46:57.963 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:57.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:57.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:57.965 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:46:57.965 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:46:57.980 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:46:57.980 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:46:57.997 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:46:57.997 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:46:58.013 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:46:58.013 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:46:58.014 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-18T07:46:58.031 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:46:58.031 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:46:58.048 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:46:58.048 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:46:58.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:58.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:58.087 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:58.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:58.123 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:46:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:46:58.123 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-activate[131331]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-18T07:46:58.123 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:57 smithi203 bash[131252]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-18T07:46:58.123 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-activate[131331]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-18T07:46:58.123 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:57 smithi203 bash[131252]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-18T07:46:58.123 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-activate[131331]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-18T07:46:58.123 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:57 smithi203 bash[131252]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-18T07:46:58.123 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-activate[131331]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-18T07:46:58.124 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:57 smithi203 bash[131252]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-18T07:46:58.124 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-activate[131331]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-18T07:46:58.124 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:57 smithi203 bash[131252]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-18T07:46:58.124 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:46:57 smithi203 podman[131252]: 2024-09-18 07:46:57.77125809 +0000 UTC m=+3.722658627 container died 89d66ce80b7125661d618ba8928f35db4ec059aecdfe98a5944e02831ba24671 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-activate, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-18T07:46:58.188 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:58.212 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:58.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:58.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:58.314 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:58.336 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:58.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:58.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:58.411 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:58 smithi137 ceph-mon[29408]: osdmap e454: 8 total, 7 up, 7 in 2024-09-18T07:46:58.411 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:58 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:46:58.411 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:58 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2965197066' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:58.411 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:58 smithi137 ceph-mon[39175]: osdmap e454: 8 total, 7 up, 7 in 2024-09-18T07:46:58.411 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:58 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:46:58.412 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:58 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2965197066' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:58.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:58.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:58.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:58.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:58 smithi203 ceph-mon[32237]: osdmap e454: 8 total, 7 up, 7 in 2024-09-18T07:46:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:58 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:46:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:58 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2965197066' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:46:58.560 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:58.586 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:58.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:58.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:58.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:58.709 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:58.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:58.811 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:46:58.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:58.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:58.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:46:58.934 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:58.945 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:46:58.946 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:46:58.959 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:46:58.960 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:58.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:59.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:59.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:59.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:59.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:59.173 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:59 smithi137 ceph-mon[29408]: pgmap v850: 158 pgs: 26 active+clean+remapped, 24 active+undersized, 14 active+undersized+degraded, 94 active+clean; 47 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.49k op/s; 384/2826 objects degraded (13.588%); 33/2826 objects misplaced (1.168%); 0 B/s, 0 objects/s recovering 2024-09-18T07:46:59.173 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:46:59 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:59.173 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:59 smithi137 ceph-mon[39175]: pgmap v850: 158 pgs: 26 active+clean+remapped, 24 active+undersized, 14 active+undersized+degraded, 94 active+clean; 47 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.49k op/s; 384/2826 objects degraded (13.588%); 33/2826 objects misplaced (1.168%); 0 B/s, 0 objects/s recovering 2024-09-18T07:46:59.173 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:46:59 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:59.185 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:59.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:59.207 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:59.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:59.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:46:59.329 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:59.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:59.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:46:59.431 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:59.457 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:59.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:59.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:46:59.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:59 smithi203 ceph-mon[32237]: pgmap v850: 158 pgs: 26 active+clean+remapped, 24 active+undersized, 14 active+undersized+degraded, 94 active+clean; 47 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.49k op/s; 384/2826 objects degraded (13.588%); 33/2826 objects misplaced (1.168%); 0 B/s, 0 objects/s recovering 2024-09-18T07:46:59.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:46:59 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:46:59.559 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:46:59.584 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:59.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:59.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:46:59.686 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:59.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:59.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:59.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:59.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:46:59.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:59.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:46:59.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:46:59.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:46:59.958 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:46:59.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:46:59.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:00.060 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:00.082 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:00.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:00.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:00.139 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:47:00.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:00.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:00.206 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:00.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:00.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:00.329 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:00.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:00.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:00 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:47:00.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:00 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:47:00.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:00 smithi137 ceph-mon[29408]: osdmap e455: 8 total, 7 up, 7 in 2024-09-18T07:47:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:00 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2486771342' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:00 smithi137 ceph-mon[29408]: from='client.36368 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:00.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:00 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 384/2826 objects degraded (13.588%), 14 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-09-18T07:47:00.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:00 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:47:00.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:00 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:47:00.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:00 smithi137 ceph-mon[39175]: osdmap e455: 8 total, 7 up, 7 in 2024-09-18T07:47:00.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:00 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2486771342' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:00.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:00 smithi137 ceph-mon[39175]: from='client.36368 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:00.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:00 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 384/2826 objects degraded (13.588%), 14 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-09-18T07:47:00.431 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:00.451 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:00.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:00.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:00.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:00 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:47:00.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:00 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:47:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:00 smithi203 ceph-mon[32237]: osdmap e455: 8 total, 7 up, 7 in 2024-09-18T07:47:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:00 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2486771342' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:00 smithi203 ceph-mon[32237]: from='client.36368 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:00 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 384/2826 objects degraded (13.588%), 14 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-09-18T07:47:00.553 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:00.579 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:00.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:00.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:00.681 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:00.707 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:00.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:00.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:00.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:00.830 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:00.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:00.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:00.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:00.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:47:00.957 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:00.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:00.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:01.059 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:01.080 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:01.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:01.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:01.182 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:01.202 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:01.208 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:01.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:01.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:01.310 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:01.323 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:01 smithi137 ceph-mon[29408]: pgmap v852: 158 pgs: 26 active+clean+remapped, 25 active+undersized, 13 active+undersized+degraded, 94 active+clean; 7.7 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 65/945 objects degraded (6.878%); 33/945 objects misplaced (3.492%) 2024-09-18T07:47:01.323 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:01 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:47:01.323 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:01 smithi137 ceph-mon[29408]: from='client.36368 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:47:01.323 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:01 smithi137 ceph-mon[29408]: osdmap e456: 8 total, 7 up, 7 in 2024-09-18T07:47:01.323 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:01 smithi137 ceph-mon[39175]: pgmap v852: 158 pgs: 26 active+clean+remapped, 25 active+undersized, 13 active+undersized+degraded, 94 active+clean; 7.7 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 65/945 objects degraded (6.878%); 33/945 objects misplaced (3.492%) 2024-09-18T07:47:01.323 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:01 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:47:01.323 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:01 smithi137 ceph-mon[39175]: from='client.36368 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:47:01.323 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:01 smithi137 ceph-mon[39175]: osdmap e456: 8 total, 7 up, 7 in 2024-09-18T07:47:01.337 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:01.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:01.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:01.441 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:01.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:01 smithi203 ceph-mon[32237]: pgmap v852: 158 pgs: 26 active+clean+remapped, 25 active+undersized, 13 active+undersized+degraded, 94 active+clean; 7.7 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 65/945 objects degraded (6.878%); 33/945 objects misplaced (3.492%) 2024-09-18T07:47:01.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:01 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:47:01.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:01 smithi203 ceph-mon[32237]: from='client.36368 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:47:01.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:01 smithi203 ceph-mon[32237]: osdmap e456: 8 total, 7 up, 7 in 2024-09-18T07:47:01.466 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:01.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:01.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:01.568 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:01.594 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:01.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:01.688 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:47:01.688 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:47:01.693 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:47:01.693 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:47:01.696 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:01.709 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:47:01.709 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:47:01.718 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:01.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:01.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:01.725 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-18T07:47:01.726 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-18T07:47:01.726 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:47:01.726 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:47:01.743 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:47:01.743 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:47:01.743 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-18T07:47:01.760 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:47:01.760 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:47:01.776 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:47:01.777 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:47:01.821 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:01.843 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:01.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:01.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:01.966 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:01.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:02.068 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:02.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:02.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:02.106 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:47:01 smithi203 podman[131789]: 2024-09-18 07:47:01.760920705 +0000 UTC m=+3.986190819 container remove 89d66ce80b7125661d618ba8928f35db4ec059aecdfe98a5944e02831ba24671 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-activate, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2) 2024-09-18T07:47:02.135 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T07:47:02.191 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:02.213 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:02.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:02.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:02.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:02.340 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:02.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:02.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:02.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:02 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1240384591' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:02.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:02 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1240384591' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:02.442 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:02.456 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:02 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1240384591' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:02.456 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:47:02 smithi203 podman[132087]: 2024-09-18 07:47:02.203908155 +0000 UTC m=+0.016165090 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:47:02.457 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:47:02 smithi203 podman[132087]: 2024-09-18 07:47:02.347279536 +0000 UTC m=+0.159536527 container create 04b85b2c8ec7bf5eeffebfe29fe3e2d7104d11e22dbb4ca625d84d19bba74f09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-18T07:47:02.464 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:02.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:02.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:02.538 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:47:02.539 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:47:02.566 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:02.590 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:02.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:02.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:02.622 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:02.622 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:02.665 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T07:47:02.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:02.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:47:02 smithi203 podman[132087]: 2024-09-18 07:47:02.558032559 +0000 UTC m=+0.370289492 container init 04b85b2c8ec7bf5eeffebfe29fe3e2d7104d11e22dbb4ca625d84d19bba74f09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-18T07:47:02.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:47:02 smithi203 podman[132087]: 2024-09-18 07:47:02.561276554 +0000 UTC m=+0.373533487 container start 04b85b2c8ec7bf5eeffebfe29fe3e2d7104d11e22dbb4ca625d84d19bba74f09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-18T07:47:02.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:47:02 smithi203 bash[132087]: 04b85b2c8ec7bf5eeffebfe29fe3e2d7104d11e22dbb4ca625d84d19bba74f09 2024-09-18T07:47:02.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:47:02 smithi203 systemd[1]: Started Ceph osd.5 for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:47:02.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:02.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:02.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:02.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:02.837 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:02.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:02.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:02.915 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:47:02.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:02.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:02.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:03.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:47:03.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:03.083 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:03.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:03.185 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:03.208 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:03.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:03.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:03.297 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:03 smithi203 ceph-mon[32237]: pgmap v854: 158 pgs: 26 active+clean+remapped, 27 active+undersized, 11 active+undersized+degraded, 94 active+clean; 483 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.17k op/s; 32/597 objects degraded (5.360%); 33/597 objects misplaced (5.528%) 2024-09-18T07:47:03.311 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:03.335 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:03.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:03.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:03.390 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:03 smithi137 ceph-mon[29408]: pgmap v854: 158 pgs: 26 active+clean+remapped, 27 active+undersized, 11 active+undersized+degraded, 94 active+clean; 483 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.17k op/s; 32/597 objects degraded (5.360%); 33/597 objects misplaced (5.528%) 2024-09-18T07:47:03.390 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:03 smithi137 ceph-mon[39175]: pgmap v854: 158 pgs: 26 active+clean+remapped, 27 active+undersized, 11 active+undersized+degraded, 94 active+clean; 483 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.17k op/s; 32/597 objects degraded (5.360%); 33/597 objects misplaced (5.528%) 2024-09-18T07:47:03.396 INFO:teuthology.orchestra.run.smithi203.stderr:no valid command found; 10 closest matches: 2024-09-18T07:47:03.396 INFO:teuthology.orchestra.run.smithi203.stderr:0 2024-09-18T07:47:03.396 INFO:teuthology.orchestra.run.smithi203.stderr:1 2024-09-18T07:47:03.397 INFO:teuthology.orchestra.run.smithi203.stderr:2 2024-09-18T07:47:03.397 INFO:teuthology.orchestra.run.smithi203.stderr:abort 2024-09-18T07:47:03.397 INFO:teuthology.orchestra.run.smithi203.stderr:assert 2024-09-18T07:47:03.397 INFO:teuthology.orchestra.run.smithi203.stderr:config diff 2024-09-18T07:47:03.397 INFO:teuthology.orchestra.run.smithi203.stderr:config diff get 2024-09-18T07:47:03.397 INFO:teuthology.orchestra.run.smithi203.stderr:config get 2024-09-18T07:47:03.397 INFO:teuthology.orchestra.run.smithi203.stderr:config help [] 2024-09-18T07:47:03.397 INFO:teuthology.orchestra.run.smithi203.stderr:config set ... 2024-09-18T07:47:03.397 INFO:teuthology.orchestra.run.smithi203.stderr:admin_socket: invalid command 2024-09-18T07:47:03.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:03.459 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:03.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:03.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:03.560 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:03.582 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:03.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:03.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:03.636 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:47:03.636 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:47:03.684 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:03.706 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:03.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:03.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:03.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:03.829 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:03.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:03.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:03.885 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:03.930 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:03.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:03.953 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:03.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:04.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:04.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:04.076 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:04.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:04.147 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'unique_pool_3' removed 2024-09-18T07:47:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T07:47:04.171 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-09-18T07:47:04.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:04.235 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:04.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:04.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:04.337 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:04.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:04.362 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:04.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:04.383 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:04 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:47:04.383 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:04 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:47:04.384 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:04 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3333175334' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:47:04.384 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:04 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:47:04.384 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:04 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:47:04.384 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:04 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3333175334' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:47:04.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T07:47:04.461 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:47:04 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:47:04.204+0000 7fb6cddb3640 -1 Falling back to public interface 2024-09-18T07:47:04.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:04 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:47:04.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:04 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:47:04.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:04 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3333175334' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:47:04.463 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:04.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:04.517 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:04.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:04.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:47:04.618 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:04.639 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:04.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:04.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:04.741 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:04.762 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:04.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:04.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:04.864 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:04.886 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:04.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:04.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:04.988 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:05.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:05.011 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:05.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:05.083 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-18T07:47:05.083 INFO:tasks.radosbench:Beginning radosbench... 2024-09-18T07:47:05.084 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-18T07:47:05.084 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-09-18T07:47:05.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:05.154 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:05 smithi137 ceph-mon[29408]: pgmap v855: 158 pgs: 26 active+clean+remapped, 28 active+undersized, 10 active+undersized+degraded, 94 active+clean; 483 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 764 B/s rd, 0 B/s wr, 111 op/s; 32/597 objects degraded (5.360%); 33/597 objects misplaced (5.528%) 2024-09-18T07:47:05.154 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:05 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3333175334' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T07:47:05.154 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:05 smithi137 ceph-mon[29408]: osdmap e457: 8 total, 7 up, 7 in 2024-09-18T07:47:05.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:05.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:05.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:05.281 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:05.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:05.382 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:05.384 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:05 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 32/597 objects degraded (5.360%), 10 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-09-18T07:47:05.385 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:05 smithi137 ceph-mon[39175]: pgmap v855: 158 pgs: 26 active+clean+remapped, 28 active+undersized, 10 active+undersized+degraded, 94 active+clean; 483 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 764 B/s rd, 0 B/s wr, 111 op/s; 32/597 objects degraded (5.360%); 33/597 objects misplaced (5.528%) 2024-09-18T07:47:05.385 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:05 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3333175334' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T07:47:05.385 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:05 smithi137 ceph-mon[39175]: osdmap e457: 8 total, 7 up, 7 in 2024-09-18T07:47:05.385 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:05 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 32/597 objects degraded (5.360%), 10 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-09-18T07:47:05.403 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:05.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:05.413 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:05.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:05 smithi203 ceph-mon[32237]: pgmap v855: 158 pgs: 26 active+clean+remapped, 28 active+undersized, 10 active+undersized+degraded, 94 active+clean; 483 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 764 B/s rd, 0 B/s wr, 111 op/s; 32/597 objects degraded (5.360%); 33/597 objects misplaced (5.528%) 2024-09-18T07:47:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:05 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3333175334' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T07:47:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:05 smithi203 ceph-mon[32237]: osdmap e457: 8 total, 7 up, 7 in 2024-09-18T07:47:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:05 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 32/597 objects degraded (5.360%), 10 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-09-18T07:47:05.514 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:05.535 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:05.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:05.637 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:05.642 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:05.659 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:05.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:05.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:05.761 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:05.785 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:05.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:05.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:05.907 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:05.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:06.009 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:06.031 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:06.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:06.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:06.133 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:06.154 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:06.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:06.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:06.160 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:47:06.256 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:06.277 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:06.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:06.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:06.378 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:06.379 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:06 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2429825680' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:47:06.379 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:06 smithi137 ceph-mon[29408]: from='client.36389 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:47:06.379 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:06 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2429825680' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:47:06.379 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:06 smithi137 ceph-mon[39175]: from='client.36389 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:47:06.393 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:47:06.394 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:47:06.400 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:06.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:06.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:06.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:06 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2429825680' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:47:06.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:06 smithi203 ceph-mon[32237]: from='client.36389 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:47:06.502 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:06.523 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:06.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:06.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:06.625 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:06.648 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:06.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:06.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:06.750 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:06.772 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:06.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:06.809 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:06.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:06.895 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:06.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:06.997 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:07.019 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:07.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:07.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:07.121 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:07.143 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:07.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:07.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:07.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:07 smithi137 ceph-mon[39175]: pgmap v857: 142 pgs: 26 active+clean+remapped, 23 active+undersized, 10 active+undersized+degraded, 83 active+clean; 483 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 611 B/s rd, 0 op/s; 32/597 objects degraded (5.360%); 33/597 objects misplaced (5.528%) 2024-09-18T07:47:07.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:07 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:47:07.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:07 smithi137 ceph-mon[39175]: from='client.36389 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:47:07.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:07 smithi137 ceph-mon[39175]: osdmap e458: 8 total, 7 up, 7 in 2024-09-18T07:47:07.170 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:47:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:47:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:47:07.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:07 smithi137 ceph-mon[29408]: pgmap v857: 142 pgs: 26 active+clean+remapped, 23 active+undersized, 10 active+undersized+degraded, 83 active+clean; 483 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 611 B/s rd, 0 op/s; 32/597 objects degraded (5.360%); 33/597 objects misplaced (5.528%) 2024-09-18T07:47:07.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:07 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:47:07.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:07 smithi137 ceph-mon[29408]: from='client.36389 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:47:07.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:07 smithi137 ceph-mon[29408]: osdmap e458: 8 total, 7 up, 7 in 2024-09-18T07:47:07.245 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:07.266 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:07.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:07.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:07.368 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:07.390 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:07.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:07.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:07.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:07 smithi203 ceph-mon[32237]: pgmap v857: 142 pgs: 26 active+clean+remapped, 23 active+undersized, 10 active+undersized+degraded, 83 active+clean; 483 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 611 B/s rd, 0 op/s; 32/597 objects degraded (5.360%); 33/597 objects misplaced (5.528%) 2024-09-18T07:47:07.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:07 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:47:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:07 smithi203 ceph-mon[32237]: from='client.36389 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:47:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:07 smithi203 ceph-mon[32237]: osdmap e458: 8 total, 7 up, 7 in 2024-09-18T07:47:07.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:07.515 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:07.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:07.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:07.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:07.637 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:07.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:07.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:07.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:07.760 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:07.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:07.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:07.862 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:07.884 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:07.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:07.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:07.986 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:08.007 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:08.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:08.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:08.033 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:47:08.033 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:47:08.046 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:47:08.047 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:47:08.062 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:47:08.062 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:47:08.078 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:47:08.078 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:47:08.096 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:47:08.096 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:47:08.096 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-18T07:47:08.108 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:08.114 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:47:08.114 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:47:08.129 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:08.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:08.131 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:47:08.131 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:47:08.156 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:47:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:47:08.231 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:08.252 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:08.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:08.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:08.354 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:08.375 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:08.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:08.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:08.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:08 smithi137 ceph-mon[29408]: 6.f scrub starts 2024-09-18T07:47:08.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:08 smithi137 ceph-mon[29408]: 6.f scrub ok 2024-09-18T07:47:08.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:08 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/696031764' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:08.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:08 smithi137 ceph-mon[39175]: 6.f scrub starts 2024-09-18T07:47:08.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:08 smithi137 ceph-mon[39175]: 6.f scrub ok 2024-09-18T07:47:08.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:08 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/696031764' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:08 smithi203 ceph-mon[32237]: 6.f scrub starts 2024-09-18T07:47:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:08 smithi203 ceph-mon[32237]: 6.f scrub ok 2024-09-18T07:47:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:08 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/696031764' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:08.461 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:47:08 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:47:08.233+0000 7fb6cddb3640 -1 osd.5 415 log_to_monitors true 2024-09-18T07:47:08.477 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:08.499 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:08.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:08.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:08.600 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:08.623 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:08.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:08.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T07:47:08.725 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:08.755 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:08.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:08.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:08.857 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:08.880 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:08.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:08.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:08.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:09.003 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:09.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:09.104 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:09.130 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:09.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:09.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:09.168 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'unique_pool_4' created 2024-09-18T07:47:09.171 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T07:47:09.233 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:09.255 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:09.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:09.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:09.330 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:47:09.357 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:09.379 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:09.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:09.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:09.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:09 smithi137 ceph-mon[39175]: pgmap v859: 142 pgs: 26 active+clean+remapped, 23 active+undersized, 10 active+undersized+degraded, 83 active+clean; 483 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 32/597 objects degraded (5.360%); 33/597 objects misplaced (5.528%) 2024-09-18T07:47:09.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:09 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:09 smithi137 ceph-mon[39175]: 2.1 scrub starts 2024-09-18T07:47:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:09 smithi137 ceph-mon[39175]: 2.1 scrub ok 2024-09-18T07:47:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:09 smithi137 ceph-mon[39175]: from='osd.5 v1:172.21.15.203:6804/1993911072' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T07:47:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:09 smithi137 ceph-mon[39175]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T07:47:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:09 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/115416506' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-18T07:47:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:09 smithi137 ceph-mon[39175]: from='client.36404 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-18T07:47:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:09 smithi137 ceph-mon[29408]: pgmap v859: 142 pgs: 26 active+clean+remapped, 23 active+undersized, 10 active+undersized+degraded, 83 active+clean; 483 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 32/597 objects degraded (5.360%); 33/597 objects misplaced (5.528%) 2024-09-18T07:47:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:09 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:09 smithi137 ceph-mon[29408]: 2.1 scrub starts 2024-09-18T07:47:09.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:09 smithi137 ceph-mon[29408]: 2.1 scrub ok 2024-09-18T07:47:09.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:09 smithi137 ceph-mon[29408]: from='osd.5 v1:172.21.15.203:6804/1993911072' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T07:47:09.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:09 smithi137 ceph-mon[29408]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T07:47:09.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:09 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/115416506' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-18T07:47:09.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:09 smithi137 ceph-mon[29408]: from='client.36404 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-18T07:47:09.448 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:47:09.460 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:47:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:47:09.228+0000 7fb6c5b5d640 -1 osd.5 459 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T07:47:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:09 smithi203 ceph-mon[32237]: pgmap v859: 142 pgs: 26 active+clean+remapped, 23 active+undersized, 10 active+undersized+degraded, 83 active+clean; 483 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 32/597 objects degraded (5.360%); 33/597 objects misplaced (5.528%) 2024-09-18T07:47:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:09 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:09 smithi203 ceph-mon[32237]: 2.1 scrub starts 2024-09-18T07:47:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:09 smithi203 ceph-mon[32237]: 2.1 scrub ok 2024-09-18T07:47:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:09 smithi203 ceph-mon[32237]: from='osd.5 v1:172.21.15.203:6804/1993911072' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T07:47:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:09 smithi203 ceph-mon[32237]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-18T07:47:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:09 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/115416506' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-18T07:47:09.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:09 smithi203 ceph-mon[32237]: from='client.36404 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-18T07:47:09.463 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-18T07:47:09.464 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-18T07:47:09.481 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:09.502 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:09.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:09.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:09.604 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:09.625 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:09.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:09.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:09.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:09.749 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:09.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:09.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:09.851 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:09.872 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:09.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:09.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:09.974 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:09.995 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:09.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:10.013 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:47:10.013 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:47:10.013 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-09-18T07:47:10.013 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:47:10.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-18T07:47:10.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:10.120 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:10.142 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:10.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:10.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:10.244 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:10.271 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:10.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:10.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:10.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:10.395 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:10.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:10.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:10.405 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[29408]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-18T07:47:10.405 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[29408]: from='client.36404 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-18T07:47:10.405 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[29408]: osdmap e459: 8 total, 7 up, 7 in 2024-09-18T07:47:10.405 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[29408]: from='osd.5 v1:172.21.15.203:6804/1993911072' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:47:10.405 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[29408]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:47:10.405 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[29408]: 3.19 scrub starts 2024-09-18T07:47:10.405 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[29408]: 3.19 scrub ok 2024-09-18T07:47:10.405 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[29408]: 4.18 scrub starts 2024-09-18T07:47:10.405 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[29408]: 4.18 scrub ok 2024-09-18T07:47:10.405 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 32/597 objects degraded (5.360%), 10 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-09-18T07:47:10.406 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[39175]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-18T07:47:10.406 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[39175]: from='client.36404 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-18T07:47:10.406 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[39175]: osdmap e459: 8 total, 7 up, 7 in 2024-09-18T07:47:10.406 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[39175]: from='osd.5 v1:172.21.15.203:6804/1993911072' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:47:10.406 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[39175]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:47:10.406 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[39175]: 3.19 scrub starts 2024-09-18T07:47:10.406 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[39175]: 3.19 scrub ok 2024-09-18T07:47:10.406 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[39175]: 4.18 scrub starts 2024-09-18T07:47:10.406 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[39175]: 4.18 scrub ok 2024-09-18T07:47:10.406 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:10 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 32/597 objects degraded (5.360%), 10 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-09-18T07:47:10.443 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:10 smithi203 ceph-mon[32237]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-18T07:47:10.443 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:10 smithi203 ceph-mon[32237]: from='client.36404 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-18T07:47:10.443 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:10 smithi203 ceph-mon[32237]: osdmap e459: 8 total, 7 up, 7 in 2024-09-18T07:47:10.443 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:10 smithi203 ceph-mon[32237]: from='osd.5 v1:172.21.15.203:6804/1993911072' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:47:10.443 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:10 smithi203 ceph-mon[32237]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]} : dispatch 2024-09-18T07:47:10.443 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:10 smithi203 ceph-mon[32237]: 3.19 scrub starts 2024-09-18T07:47:10.444 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:10 smithi203 ceph-mon[32237]: 3.19 scrub ok 2024-09-18T07:47:10.444 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:10 smithi203 ceph-mon[32237]: 4.18 scrub starts 2024-09-18T07:47:10.444 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:10 smithi203 ceph-mon[32237]: 4.18 scrub ok 2024-09-18T07:47:10.444 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:10 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 32/597 objects degraded (5.360%), 10 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2024-09-18T07:47:10.497 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:10.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:10.522 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:10.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:10.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:10.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:10.645 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:10.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:10.658 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:10.710 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:47:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=sqlstore.transactions t=2024-09-18T07:47:10.440841232Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-18T07:47:10.711 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:47:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=sqlstore.transactions t=2024-09-18T07:47:10.451425683Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-18T07:47:10.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:10.768 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:10.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:10.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:10.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:10.891 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:10.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:10.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:10.929 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:10.994 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:11.016 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:11.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:11.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:11.118 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:11.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:11.140 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:11.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:11.182 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:47:11.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:11.262 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:11.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:11.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:11.343 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:11 smithi137 ceph-mon[29408]: pgmap v861: 158 pgs: 16 unknown, 26 active+clean+remapped, 23 active+undersized, 10 active+undersized+degraded, 83 active+clean; 483 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 32/597 objects degraded (5.360%); 33/597 objects misplaced (5.528%) 2024-09-18T07:47:11.344 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:11 smithi137 ceph-mon[29408]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:47:11.344 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:11 smithi137 ceph-mon[29408]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-18T07:47:11.344 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:11 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:47:11.344 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:11 smithi137 ceph-mon[29408]: osd.5 v1:172.21.15.203:6804/1993911072 boot 2024-09-18T07:47:11.344 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:11 smithi137 ceph-mon[29408]: osdmap e460: 8 total, 8 up, 7 in 2024-09-18T07:47:11.344 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:11 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1752476864' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:11.344 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:11 smithi137 ceph-mon[29408]: from='client.36410 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:11.344 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:11 smithi137 ceph-mon[39175]: pgmap v861: 158 pgs: 16 unknown, 26 active+clean+remapped, 23 active+undersized, 10 active+undersized+degraded, 83 active+clean; 483 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 32/597 objects degraded (5.360%); 33/597 objects misplaced (5.528%) 2024-09-18T07:47:11.345 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:11 smithi137 ceph-mon[39175]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:47:11.345 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:11 smithi137 ceph-mon[39175]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-18T07:47:11.345 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:11 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:47:11.345 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:11 smithi137 ceph-mon[39175]: osd.5 v1:172.21.15.203:6804/1993911072 boot 2024-09-18T07:47:11.345 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:11 smithi137 ceph-mon[39175]: osdmap e460: 8 total, 8 up, 7 in 2024-09-18T07:47:11.345 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:11 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1752476864' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:11.345 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:11 smithi137 ceph-mon[39175]: from='client.36410 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:11.351 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:47:11.351 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:47:11.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:47:11.352 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:47:11.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.352 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109534 8.6 8:67e394d3:::benchmark_data_smithi137_101942_object4724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-18T07:47:11.352 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.048516+0000", 2024-09-18T07:47:11.352 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 569.29155230900005, 2024-09-18T07:47:11.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.182592823, 2024-09-18T07:47:11.352 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.352 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.352 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:11.352 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:11.353 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:47:11.353 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:47:11.353 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109534 2024-09-18T07:47:11.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.353 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048516+0000", 2024-09-18T07:47:11.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048516+0000", 2024-09-18T07:47:11.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048517+0000", 2024-09-18T07:47:11.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2e-07 2024-09-18T07:47:11.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048519+0000", 2024-09-18T07:47:11.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.232e-06 2024-09-18T07:47:11.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048523+0000", 2024-09-18T07:47:11.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4479999999999996e-06 2024-09-18T07:47:11.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.048528+0000", 2024-09-18T07:47:11.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2169999999999996e-06 2024-09-18T07:47:11.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065112+0000", 2024-09-18T07:47:11.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016583912999999999 2024-09-18T07:47:11.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:11.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065121+0000", 2024-09-18T07:47:11.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8640000000000008e-06 2024-09-18T07:47:11.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.219070+0000", 2024-09-18T07:47:11.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1539486139999999 2024-09-18T07:47:11.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:11.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.219228+0000", 2024-09-18T07:47:11.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000157751 2024-09-18T07:47:11.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-09-18T07:47:11.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.219376+0000", 2024-09-18T07:47:11.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014781899999999999 2024-09-18T07:47:11.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226242+0000", 2024-09-18T07:47:11.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068661970000000001 2024-09-18T07:47:11.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:11.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226253+0000", 2024-09-18T07:47:11.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0929e-05 2024-09-18T07:47:11.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229198+0000", 2024-09-18T07:47:11.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029453790000000001 2024-09-18T07:47:11.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:11.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.229278+0000", 2024-09-18T07:47:11.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9431000000000006e-05 2024-09-18T07:47:11.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.231109+0000", 2024-09-18T07:47:11.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018316090000000001 2024-09-18T07:47:11.361 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.361 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.361 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.361 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.35171.0:109539 8.6 8:66837289:::benchmark_data_smithi137_101942_object4729:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-18T07:47:11.361 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:37:42.065140+0000", 2024-09-18T07:47:11.361 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 569.27492884000003, 2024-09-18T07:47:11.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1676551340000001, 2024-09-18T07:47:11.361 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.361 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.362 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:11.362 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:11.362 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.35171", 2024-09-18T07:47:11.362 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/2151983974", 2024-09-18T07:47:11.362 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 109539 2024-09-18T07:47:11.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.362 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065140+0000", 2024-09-18T07:47:11.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065140+0000", 2024-09-18T07:47:11.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065142+0000", 2024-09-18T07:47:11.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.548e-06 2024-09-18T07:47:11.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065149+0000", 2024-09-18T07:47:11.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7989999999999996e-06 2024-09-18T07:47:11.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065162+0000", 2024-09-18T07:47:11.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3027000000000001e-05 2024-09-18T07:47:11.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065171+0000", 2024-09-18T07:47:11.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1730000000000001e-06 2024-09-18T07:47:11.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065215+0000", 2024-09-18T07:47:11.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3679e-05 2024-09-18T07:47:11.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:11.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:42.065222+0000", 2024-09-18T07:47:11.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9879999999999998e-06 2024-09-18T07:47:11.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.223751+0000", 2024-09-18T07:47:11.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1585293800000001 2024-09-18T07:47:11.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:11.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.223906+0000", 2024-09-18T07:47:11.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015412600000000001 2024-09-18T07:47:11.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 6,7", 2024-09-18T07:47:11.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.224068+0000", 2024-09-18T07:47:11.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016266499999999999 2024-09-18T07:47:11.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:11.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226399+0000", 2024-09-18T07:47:11.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002330557 2024-09-18T07:47:11.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.226549+0000", 2024-09-18T07:47:11.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000150693 2024-09-18T07:47:11.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.231185+0000", 2024-09-18T07:47:11.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004635775 2024-09-18T07:47:11.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:11.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.231241+0000", 2024-09-18T07:47:11.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6001999999999997e-05 2024-09-18T07:47:11.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:37:43.232795+0000", 2024-09-18T07:47:11.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001553722 2024-09-18T07:47:11.369 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.369 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.369 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.369 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137615 10.4 10:280114ba:::benchmark_data_smithi137_151160_object1799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-18T07:47:11.369 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:42.771430+0000", 2024-09-18T07:47:11.370 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 148.56863821100001, 2024-09-18T07:47:11.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.155652747, 2024-09-18T07:47:11.370 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.370 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.370 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:47:11.370 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:11.370 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:11.370 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:11.370 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137615 2024-09-18T07:47:11.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.370 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771430+0000", 2024-09-18T07:47:11.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771430+0000", 2024-09-18T07:47:11.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.372 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771432+0000", 2024-09-18T07:47:11.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6199999999999999e-06 2024-09-18T07:47:11.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.372 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771437+0000", 2024-09-18T07:47:11.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0159999999999999e-06 2024-09-18T07:47:11.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771445+0000", 2024-09-18T07:47:11.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.225e-06 2024-09-18T07:47:11.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771451+0000", 2024-09-18T07:47:11.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.908e-06 2024-09-18T07:47:11.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.772166+0000", 2024-09-18T07:47:11.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000714571 2024-09-18T07:47:11.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:11.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.772292+0000", 2024-09-18T07:47:11.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126322 2024-09-18T07:47:11.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-09-18T07:47:11.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.772478+0000", 2024-09-18T07:47:11.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000185747 2024-09-18T07:47:11.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:11.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.784266+0000", 2024-09-18T07:47:11.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011787824000000001 2024-09-18T07:47:11.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.795223+0000", 2024-09-18T07:47:11.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010956802999999999 2024-09-18T07:47:11.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:11.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.795236+0000", 2024-09-18T07:47:11.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3570999999999999e-05 2024-09-18T07:47:11.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774402+0000", 2024-09-18T07:47:11.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97916638700000003 2024-09-18T07:47:11.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:11.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774413+0000", 2024-09-18T07:47:11.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0035e-05 2024-09-18T07:47:11.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927028+0000", 2024-09-18T07:47:11.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.152615967 2024-09-18T07:47:11.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927083+0000", 2024-09-18T07:47:11.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4750999999999998e-05 2024-09-18T07:47:11.378 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.378 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.378 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.378 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137618 10.4 10:22103fc7:::benchmark_data_smithi137_151160_object1802:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-18T07:47:11.378 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:42.773989+0000", 2024-09-18T07:47:11.378 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 148.56607976199999, 2024-09-18T07:47:11.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1739552340000001, 2024-09-18T07:47:11.379 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.379 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.379 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:47:11.379 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:11.379 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:11.379 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:11.379 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137618 2024-09-18T07:47:11.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.379 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.773989+0000", 2024-09-18T07:47:11.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.773989+0000", 2024-09-18T07:47:11.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.773991+0000", 2024-09-18T07:47:11.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.657e-06 2024-09-18T07:47:11.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.773997+0000", 2024-09-18T07:47:11.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1789999999999996e-06 2024-09-18T07:47:11.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.774008+0000", 2024-09-18T07:47:11.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0937999999999999e-05 2024-09-18T07:47:11.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.774015+0000", 2024-09-18T07:47:11.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7789999999999996e-06 2024-09-18T07:47:11.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.779798+0000", 2024-09-18T07:47:11.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057823980000000002 2024-09-18T07:47:11.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:11.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.779927+0000", 2024-09-18T07:47:11.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012920100000000001 2024-09-18T07:47:11.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-09-18T07:47:11.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.780075+0000", 2024-09-18T07:47:11.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014824700000000001 2024-09-18T07:47:11.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:11.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.784857+0000", 2024-09-18T07:47:11.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047816070000000002 2024-09-18T07:47:11.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.834874+0000", 2024-09-18T07:47:11.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.050016892 2024-09-18T07:47:11.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:11.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.834885+0000", 2024-09-18T07:47:11.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1080999999999999e-05 2024-09-18T07:47:11.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774441+0000", 2024-09-18T07:47:11.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93955602400000005 2024-09-18T07:47:11.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:11.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774685+0000", 2024-09-18T07:47:11.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024376000000000001 2024-09-18T07:47:11.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927339+0000", 2024-09-18T07:47:11.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1526545050000001 2024-09-18T07:47:11.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-09-18T07:47:11.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927499+0000", 2024-09-18T07:47:11.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016008799999999999 2024-09-18T07:47:11.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.947852+0000", 2024-09-18T07:47:11.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020352456000000001 2024-09-18T07:47:11.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.947944+0000", 2024-09-18T07:47:11.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2422000000000004e-05 2024-09-18T07:47:11.387 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.387 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.388 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.388 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137629 10.4 10:238d70eb:::benchmark_data_smithi137_151160_object1813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-09-18T07:47:11.388 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:42.835110+0000", 2024-09-18T07:47:11.388 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 148.504958187, 2024-09-18T07:47:11.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.113021496, 2024-09-18T07:47:11.388 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.388 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.388 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:11.389 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:11.389 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:11.389 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:11.389 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137629 2024-09-18T07:47:11.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.389 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835110+0000", 2024-09-18T07:47:11.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835110+0000", 2024-09-18T07:47:11.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835113+0000", 2024-09-18T07:47:11.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1150000000000001e-06 2024-09-18T07:47:11.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835117+0000", 2024-09-18T07:47:11.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2359999999999998e-06 2024-09-18T07:47:11.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835124+0000", 2024-09-18T07:47:11.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3629999999999996e-06 2024-09-18T07:47:11.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835131+0000", 2024-09-18T07:47:11.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5899999999999996e-06 2024-09-18T07:47:11.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835166+0000", 2024-09-18T07:47:11.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5054e-05 2024-09-18T07:47:11.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:11.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835173+0000", 2024-09-18T07:47:11.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9759999999999998e-06 2024-09-18T07:47:11.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774713+0000", 2024-09-18T07:47:11.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93954058799999995 2024-09-18T07:47:11.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:11.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774722+0000", 2024-09-18T07:47:11.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5320000000000006e-06 2024-09-18T07:47:11.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927611+0000", 2024-09-18T07:47:11.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.152888763 2024-09-18T07:47:11.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:11.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927731+0000", 2024-09-18T07:47:11.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012040999999999999 2024-09-18T07:47:11.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:47:11.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927854+0000", 2024-09-18T07:47:11.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012266800000000001 2024-09-18T07:47:11.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:11.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.934460+0000", 2024-09-18T07:47:11.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066062439999999998 2024-09-18T07:47:11.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.947486+0000", 2024-09-18T07:47:11.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013025956999999999 2024-09-18T07:47:11.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.947988+0000", 2024-09-18T07:47:11.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050220899999999999 2024-09-18T07:47:11.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:11.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.948040+0000", 2024-09-18T07:47:11.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1462000000000003e-05 2024-09-18T07:47:11.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.948132+0000", 2024-09-18T07:47:11.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2329000000000003e-05 2024-09-18T07:47:11.396 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.396 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.396 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.397 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137617 10.1 10:8638502c:::benchmark_data_smithi137_151160_object1801:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e396)", 2024-09-18T07:47:11.397 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:43.767316+0000", 2024-09-18T07:47:11.397 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 147.572752189, 2024-09-18T07:47:11.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1348433360000001, 2024-09-18T07:47:11.397 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.397 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.397 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:47:11.397 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:11.397 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:11.397 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:11.398 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137617 2024-09-18T07:47:11.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.398 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.767316+0000", 2024-09-18T07:47:11.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.767316+0000", 2024-09-18T07:47:11.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.767318+0000", 2024-09-18T07:47:11.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.889e-06 2024-09-18T07:47:11.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.767324+0000", 2024-09-18T07:47:11.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8429999999999998e-06 2024-09-18T07:47:11.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.768695+0000", 2024-09-18T07:47:11.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001370549 2024-09-18T07:47:11.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.768709+0000", 2024-09-18T07:47:11.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.434e-05 2024-09-18T07:47:11.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.772764+0000", 2024-09-18T07:47:11.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040549729999999999 2024-09-18T07:47:11.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:11.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.772774+0000", 2024-09-18T07:47:11.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4499999999999993e-06 2024-09-18T07:47:11.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.901011+0000", 2024-09-18T07:47:11.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.128237484 2024-09-18T07:47:11.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.902160+0000", 2024-09-18T07:47:11.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011488080000000001 2024-09-18T07:47:11.402 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.402 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.402 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.402 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137632 10.1 10:839066b5:::benchmark_data_smithi137_151160_object1816:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e396)", 2024-09-18T07:47:11.402 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:43.769783+0000", 2024-09-18T07:47:11.402 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 147.57028616299999, 2024-09-18T07:47:11.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1401627699999999, 2024-09-18T07:47:11.403 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.403 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.403 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:11.403 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:11.403 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:11.403 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:11.403 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137632 2024-09-18T07:47:11.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.403 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769783+0000", 2024-09-18T07:47:11.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769783+0000", 2024-09-18T07:47:11.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769784+0000", 2024-09-18T07:47:11.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7439999999999999e-06 2024-09-18T07:47:11.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769791+0000", 2024-09-18T07:47:11.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6869999999999997e-06 2024-09-18T07:47:11.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769800+0000", 2024-09-18T07:47:11.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0129999999999999e-06 2024-09-18T07:47:11.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769808+0000", 2024-09-18T07:47:11.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7060000000000004e-06 2024-09-18T07:47:11.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.772796+0000", 2024-09-18T07:47:11.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002987899 2024-09-18T07:47:11.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:11.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.772802+0000", 2024-09-18T07:47:11.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9909999999999999e-06 2024-09-18T07:47:11.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.902389+0000", 2024-09-18T07:47:11.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.129587747 2024-09-18T07:47:11.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:11.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.902738+0000", 2024-09-18T07:47:11.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034822599999999998 2024-09-18T07:47:11.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4,7", 2024-09-18T07:47:11.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.902816+0000", 2024-09-18T07:47:11.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8430000000000006e-05 2024-09-18T07:47:11.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:11.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.904850+0000", 2024-09-18T07:47:11.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020342960000000001 2024-09-18T07:47:11.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.906005+0000", 2024-09-18T07:47:11.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011550899999999999 2024-09-18T07:47:11.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.909791+0000", 2024-09-18T07:47:11.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0037854500000000001 2024-09-18T07:47:11.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:11.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.909846+0000", 2024-09-18T07:47:11.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5565000000000001e-05 2024-09-18T07:47:11.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.909945+0000", 2024-09-18T07:47:11.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8925999999999994e-05 2024-09-18T07:47:11.410 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.411 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.411 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.411 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137633 10.2 10:44856315:::benchmark_data_smithi137_151160_object1817:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e396)", 2024-09-18T07:47:11.411 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:43.769818+0000", 2024-09-18T07:47:11.411 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 147.57025094799999, 2024-09-18T07:47:11.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.138105184, 2024-09-18T07:47:11.411 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.411 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.411 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:11.412 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:11.412 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:11.412 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:11.412 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137633 2024-09-18T07:47:11.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.412 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769818+0000", 2024-09-18T07:47:11.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769818+0000", 2024-09-18T07:47:11.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769819+0000", 2024-09-18T07:47:11.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.71e-07 2024-09-18T07:47:11.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769821+0000", 2024-09-18T07:47:11.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8320000000000002e-06 2024-09-18T07:47:11.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769827+0000", 2024-09-18T07:47:11.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8799999999999996e-06 2024-09-18T07:47:11.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769833+0000", 2024-09-18T07:47:11.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5280000000000003e-06 2024-09-18T07:47:11.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774264+0000", 2024-09-18T07:47:11.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044315839999999997 2024-09-18T07:47:11.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:11.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774893+0000", 2024-09-18T07:47:11.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00062851599999999997 2024-09-18T07:47:11.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.897969+0000", 2024-09-18T07:47:11.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.123075681 2024-09-18T07:47:11.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:11.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.898060+0000", 2024-09-18T07:47:11.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1570999999999995e-05 2024-09-18T07:47:11.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:47:11.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.898142+0000", 2024-09-18T07:47:11.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1304000000000006e-05 2024-09-18T07:47:11.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.900535+0000", 2024-09-18T07:47:11.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023931239999999999 2024-09-18T07:47:11.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:11.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.907715+0000", 2024-09-18T07:47:11.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.007180278 2024-09-18T07:47:11.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.907757+0000", 2024-09-18T07:47:11.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1897000000000002e-05 2024-09-18T07:47:11.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:11.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.907819+0000", 2024-09-18T07:47:11.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2139999999999998e-05 2024-09-18T07:47:11.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:11.421 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.907923+0000", 2024-09-18T07:47:11.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103978 2024-09-18T07:47:11.421 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.421 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.421 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.422 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:169413 10.4 10:2906f1b0:::benchmark_data_smithi137_151160_object33597:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-18T07:47:11.422 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:12.623237+0000", 2024-09-18T07:47:11.422 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 118.7168321, 2024-09-18T07:47:11.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.125565337, 2024-09-18T07:47:11.422 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.422 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.422 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:11.422 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:11.422 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:11.422 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:11.423 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 169413 2024-09-18T07:47:11.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.423 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623237+0000", 2024-09-18T07:47:11.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623237+0000", 2024-09-18T07:47:11.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623239+0000", 2024-09-18T07:47:11.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9379999999999999e-06 2024-09-18T07:47:11.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623245+0000", 2024-09-18T07:47:11.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1630000000000001e-06 2024-09-18T07:47:11.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623257+0000", 2024-09-18T07:47:11.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2167000000000001e-05 2024-09-18T07:47:11.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623264+0000", 2024-09-18T07:47:11.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.379e-06 2024-09-18T07:47:11.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623314+0000", 2024-09-18T07:47:11.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9749000000000001e-05 2024-09-18T07:47:11.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:11.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623535+0000", 2024-09-18T07:47:11.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000220929 2024-09-18T07:47:11.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:47:11.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.623694+0000", 2024-09-18T07:47:11.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000159574 2024-09-18T07:47:11.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:11.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.627299+0000", 2024-09-18T07:47:11.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036042359999999998 2024-09-18T07:47:11.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.632097+0000", 2024-09-18T07:47:11.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047979279999999999 2024-09-18T07:47:11.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.748649+0000", 2024-09-18T07:47:11.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.1165527 2024-09-18T07:47:11.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:11.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.748709+0000", 2024-09-18T07:47:11.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9419000000000001e-05 2024-09-18T07:47:11.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.748802+0000", 2024-09-18T07:47:11.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3154999999999998e-05 2024-09-18T07:47:11.429 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.429 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.430 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.430 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:169434 10.4 10:2739bd15:::benchmark_data_smithi137_151160_object33618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-18T07:47:11.430 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:12.641620+0000", 2024-09-18T07:47:11.430 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 118.69844902299999, 2024-09-18T07:47:11.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.117241649, 2024-09-18T07:47:11.430 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.430 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.430 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:11.430 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:11.431 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:11.431 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:11.431 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 169434 2024-09-18T07:47:11.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.431 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641620+0000", 2024-09-18T07:47:11.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641620+0000", 2024-09-18T07:47:11.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641621+0000", 2024-09-18T07:47:11.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4339999999999999e-06 2024-09-18T07:47:11.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641626+0000", 2024-09-18T07:47:11.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5109999999999997e-06 2024-09-18T07:47:11.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641636+0000", 2024-09-18T07:47:11.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0889e-05 2024-09-18T07:47:11.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641644+0000", 2024-09-18T07:47:11.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8760000000000005e-06 2024-09-18T07:47:11.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641701+0000", 2024-09-18T07:47:11.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6474999999999997e-05 2024-09-18T07:47:11.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:11.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641780+0000", 2024-09-18T07:47:11.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8894999999999996e-05 2024-09-18T07:47:11.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:47:11.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.641857+0000", 2024-09-18T07:47:11.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6932000000000001e-05 2024-09-18T07:47:11.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:11.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.644448+0000", 2024-09-18T07:47:11.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025915539999999998 2024-09-18T07:47:11.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.649282+0000", 2024-09-18T07:47:11.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048333059999999999 2024-09-18T07:47:11.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.758632+0000", 2024-09-18T07:47:11.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10935044200000001 2024-09-18T07:47:11.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:11.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.758700+0000", 2024-09-18T07:47:11.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7780000000000005e-05 2024-09-18T07:47:11.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:12.758861+0000", 2024-09-18T07:47:11.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016155499999999999 2024-09-18T07:47:11.438 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.438 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.438 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.438 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:181640 10.4 10:2d01346c:::benchmark_data_smithi137_151160_object45824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-09-18T07:47:11.438 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:25.756539+0000", 2024-09-18T07:47:11.438 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 105.58353010800001, 2024-09-18T07:47:11.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13636992100000001, 2024-09-18T07:47:11.438 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.439 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.439 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:11.439 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:11.439 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:11.439 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:11.439 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 181640 2024-09-18T07:47:11.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.439 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756539+0000", 2024-09-18T07:47:11.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756539+0000", 2024-09-18T07:47:11.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756541+0000", 2024-09-18T07:47:11.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1600000000000001e-06 2024-09-18T07:47:11.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756547+0000", 2024-09-18T07:47:11.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6880000000000002e-06 2024-09-18T07:47:11.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756559+0000", 2024-09-18T07:47:11.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1667e-05 2024-09-18T07:47:11.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756566+0000", 2024-09-18T07:47:11.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2840000000000001e-06 2024-09-18T07:47:11.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756622+0000", 2024-09-18T07:47:11.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5383999999999997e-05 2024-09-18T07:47:11.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:11.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756754+0000", 2024-09-18T07:47:11.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013199300000000001 2024-09-18T07:47:11.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:47:11.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.756895+0000", 2024-09-18T07:47:11.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014090899999999999 2024-09-18T07:47:11.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:11.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.759543+0000", 2024-09-18T07:47:11.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026484299999999998 2024-09-18T07:47:11.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.760252+0000", 2024-09-18T07:47:11.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00070913200000000001 2024-09-18T07:47:11.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.892787+0000", 2024-09-18T07:47:11.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13253521900000001 2024-09-18T07:47:11.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:11.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.892825+0000", 2024-09-18T07:47:11.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7258999999999997e-05 2024-09-18T07:47:11.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:25.892908+0000", 2024-09-18T07:47:11.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3795999999999995e-05 2024-09-18T07:47:11.446 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.446 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.446 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.446 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189721 10.6 10.a7ff4e76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:47:11.446 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.403483+0000", 2024-09-18T07:47:11.446 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 96.936585532999999, 2024-09-18T07:47:11.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.73509391899999998, 2024-09-18T07:47:11.446 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.446 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.446 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:47:11.447 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:11.447 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:11.447 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:11.447 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189721 2024-09-18T07:47:11.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.447 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403483+0000", 2024-09-18T07:47:11.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403483+0000", 2024-09-18T07:47:11.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403485+0000", 2024-09-18T07:47:11.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.849e-06 2024-09-18T07:47:11.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403490+0000", 2024-09-18T07:47:11.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.468e-06 2024-09-18T07:47:11.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403502+0000", 2024-09-18T07:47:11.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1865000000000001e-05 2024-09-18T07:47:11.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403509+0000", 2024-09-18T07:47:11.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.601e-06 2024-09-18T07:47:11.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403574+0000", 2024-09-18T07:47:11.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5314000000000005e-05 2024-09-18T07:47:11.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:11.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403585+0000", 2024-09-18T07:47:11.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0688e-05 2024-09-18T07:47:11.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.451 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.451 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138555+0000", 2024-09-18T07:47:11.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.73496987400000002 2024-09-18T07:47:11.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.451 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.451 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138577+0000", 2024-09-18T07:47:11.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.226e-05 2024-09-18T07:47:11.452 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.452 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.452 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.452 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189731 10.6 10.884df406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:47:11.452 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.415372+0000", 2024-09-18T07:47:11.452 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 96.924697010000003, 2024-09-18T07:47:11.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.72323327699999995, 2024-09-18T07:47:11.452 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.453 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.453 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:47:11.453 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:11.453 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:11.453 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:11.453 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189731 2024-09-18T07:47:11.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.453 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415372+0000", 2024-09-18T07:47:11.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415372+0000", 2024-09-18T07:47:11.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415373+0000", 2024-09-18T07:47:11.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3430000000000001e-06 2024-09-18T07:47:11.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415378+0000", 2024-09-18T07:47:11.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5070000000000002e-06 2024-09-18T07:47:11.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415387+0000", 2024-09-18T07:47:11.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7529999999999996e-06 2024-09-18T07:47:11.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415394+0000", 2024-09-18T07:47:11.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4740000000000002e-06 2024-09-18T07:47:11.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415445+0000", 2024-09-18T07:47:11.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1202000000000002e-05 2024-09-18T07:47:11.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:11.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415453+0000", 2024-09-18T07:47:11.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.123e-06 2024-09-18T07:47:11.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138598+0000", 2024-09-18T07:47:11.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.72314508399999999 2024-09-18T07:47:11.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.458 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138605+0000", 2024-09-18T07:47:11.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7909999999999999e-06 2024-09-18T07:47:11.458 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.458 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.458 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.458 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189736 10.6 10.2043ac56 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:47:11.458 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.419163+0000", 2024-09-18T07:47:11.458 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 96.920905942000005, 2024-09-18T07:47:11.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71946739599999998, 2024-09-18T07:47:11.459 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.459 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.459 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:47:11.459 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:11.459 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:11.459 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:11.459 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189736 2024-09-18T07:47:11.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.459 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419163+0000", 2024-09-18T07:47:11.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419163+0000", 2024-09-18T07:47:11.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419164+0000", 2024-09-18T07:47:11.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4130000000000001e-06 2024-09-18T07:47:11.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419169+0000", 2024-09-18T07:47:11.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8269999999999997e-06 2024-09-18T07:47:11.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419183+0000", 2024-09-18T07:47:11.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3596e-05 2024-09-18T07:47:11.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419191+0000", 2024-09-18T07:47:11.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0239999999999997e-06 2024-09-18T07:47:11.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419296+0000", 2024-09-18T07:47:11.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0001058 2024-09-18T07:47:11.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:11.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419306+0000", 2024-09-18T07:47:11.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5109999999999992e-06 2024-09-18T07:47:11.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138625+0000", 2024-09-18T07:47:11.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71931864599999995 2024-09-18T07:47:11.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138630+0000", 2024-09-18T07:47:11.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5790000000000003e-06 2024-09-18T07:47:11.464 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.464 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.464 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.464 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:212774 10.5 e434/364 10:ad83ab1d:::benchmark_data_smithi137_151160_object76958:head v 434'13240, mlcod=434'13238)", 2024-09-18T07:47:11.464 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:06.405625+0000", 2024-09-18T07:47:11.464 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 64.934444157000001, 2024-09-18T07:47:11.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10916104, 2024-09-18T07:47:11.465 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.465 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.465 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:11.465 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.465 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.405625+0000", 2024-09-18T07:47:11.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.405625+0000", 2024-09-18T07:47:11.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.405626+0000", 2024-09-18T07:47:11.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.46e-06 2024-09-18T07:47:11.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.405633+0000", 2024-09-18T07:47:11.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2450000000000001e-06 2024-09-18T07:47:11.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.405644+0000", 2024-09-18T07:47:11.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0509000000000001e-05 2024-09-18T07:47:11.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.405650+0000", 2024-09-18T07:47:11.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5919999999999997e-06 2024-09-18T07:47:11.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.408461+0000", 2024-09-18T07:47:11.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028107649999999998 2024-09-18T07:47:11.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:11.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.408479+0000", 2024-09-18T07:47:11.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7620000000000001e-05 2024-09-18T07:47:11.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:11.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.514738+0000", 2024-09-18T07:47:11.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.106259193 2024-09-18T07:47:11.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.514786+0000", 2024-09-18T07:47:11.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7655999999999999e-05 2024-09-18T07:47:11.469 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.469 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.469 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.470 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:212787 10.c e434/409 10:3802e8df:::benchmark_data_smithi137_151160_object76971:head v 434'13335, mlcod=434'13333)", 2024-09-18T07:47:11.470 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:06.412569+0000", 2024-09-18T07:47:11.470 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 64.927500021, 2024-09-18T07:47:11.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.099830223999999995, 2024-09-18T07:47:11.470 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.470 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.470 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:11.470 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.470 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.412569+0000", 2024-09-18T07:47:11.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.412569+0000", 2024-09-18T07:47:11.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.412570+0000", 2024-09-18T07:47:11.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6300000000000004e-07 2024-09-18T07:47:11.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.412575+0000", 2024-09-18T07:47:11.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7169999999999996e-06 2024-09-18T07:47:11.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.503139+0000", 2024-09-18T07:47:11.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.090563280999999995 2024-09-18T07:47:11.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.503152+0000", 2024-09-18T07:47:11.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3657e-05 2024-09-18T07:47:11.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.503645+0000", 2024-09-18T07:47:11.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00049273100000000005 2024-09-18T07:47:11.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:11.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.503665+0000", 2024-09-18T07:47:11.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9681e-05 2024-09-18T07:47:11.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:11.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.512337+0000", 2024-09-18T07:47:11.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0086720249999999999 2024-09-18T07:47:11.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.512399+0000", 2024-09-18T07:47:11.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2268999999999996e-05 2024-09-18T07:47:11.474 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:212784 10.d e434/410 10:bf81e37e:::benchmark_data_smithi137_151160_object76968:head v 434'13295, mlcod=434'13293)", 2024-09-18T07:47:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:06.415031+0000", 2024-09-18T07:47:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 64.925037476, 2024-09-18T07:47:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.099864473999999995, 2024-09-18T07:47:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:11.475 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.415031+0000", 2024-09-18T07:47:11.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.415031+0000", 2024-09-18T07:47:11.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.415033+0000", 2024-09-18T07:47:11.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6190000000000001e-06 2024-09-18T07:47:11.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.415041+0000", 2024-09-18T07:47:11.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0709999999999995e-06 2024-09-18T07:47:11.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.503311+0000", 2024-09-18T07:47:11.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.088270150000000006 2024-09-18T07:47:11.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.503318+0000", 2024-09-18T07:47:11.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0469999999999996e-06 2024-09-18T07:47:11.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.503953+0000", 2024-09-18T07:47:11.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00063526399999999997 2024-09-18T07:47:11.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:11.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.503973+0000", 2024-09-18T07:47:11.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9958000000000001e-05 2024-09-18T07:47:11.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:11.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.514790+0000", 2024-09-18T07:47:11.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010816927 2024-09-18T07:47:11.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:06.514896+0000", 2024-09-18T07:47:11.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105438 2024-09-18T07:47:11.480 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.480 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.480 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.480 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:215863 10.8 10:1c60eade:::benchmark_data_smithi137_151160_object80047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-18T07:47:11.480 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.932675+0000", 2024-09-18T07:47:11.481 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 62.407393812999999, 2024-09-18T07:47:11.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28984009900000002, 2024-09-18T07:47:11.481 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.481 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.481 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:11.481 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:11.481 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:11.481 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:11.481 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 215863 2024-09-18T07:47:11.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.481 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932675+0000", 2024-09-18T07:47:11.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932675+0000", 2024-09-18T07:47:11.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932676+0000", 2024-09-18T07:47:11.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5859999999999999e-06 2024-09-18T07:47:11.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932681+0000", 2024-09-18T07:47:11.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9169999999999997e-06 2024-09-18T07:47:11.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932696+0000", 2024-09-18T07:47:11.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4341e-05 2024-09-18T07:47:11.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932703+0000", 2024-09-18T07:47:11.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1369999999999996e-06 2024-09-18T07:47:11.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.938047+0000", 2024-09-18T07:47:11.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005343965 2024-09-18T07:47:11.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:11.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.938162+0000", 2024-09-18T07:47:11.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115383 2024-09-18T07:47:11.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:47:11.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.938546+0000", 2024-09-18T07:47:11.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00038351599999999998 2024-09-18T07:47:11.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.941541+0000", 2024-09-18T07:47:11.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029950369999999999 2024-09-18T07:47:11.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:11.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942037+0000", 2024-09-18T07:47:11.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00049626300000000004 2024-09-18T07:47:11.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222371+0000", 2024-09-18T07:47:11.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28033375100000002 2024-09-18T07:47:11.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:11.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222469+0000", 2024-09-18T07:47:11.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8204999999999999e-05 2024-09-18T07:47:11.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222515+0000", 2024-09-18T07:47:11.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5998000000000002e-05 2024-09-18T07:47:11.487 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.487 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.487 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.488 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:215874 10.8 10:1dfe7e41:::benchmark_data_smithi137_151160_object80058:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-18T07:47:11.488 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.942581+0000", 2024-09-18T07:47:11.488 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 62.397488052, 2024-09-18T07:47:11.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28009725699999999, 2024-09-18T07:47:11.488 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.488 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.488 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:11.488 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:11.488 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:11.488 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:11.488 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 215874 2024-09-18T07:47:11.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.489 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942581+0000", 2024-09-18T07:47:11.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942581+0000", 2024-09-18T07:47:11.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.490 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.490 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942582+0000", 2024-09-18T07:47:11.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.629e-06 2024-09-18T07:47:11.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.490 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.490 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942587+0000", 2024-09-18T07:47:11.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5709999999999999e-06 2024-09-18T07:47:11.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942596+0000", 2024-09-18T07:47:11.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6600000000000007e-06 2024-09-18T07:47:11.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942603+0000", 2024-09-18T07:47:11.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6630000000000004e-06 2024-09-18T07:47:11.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942687+0000", 2024-09-18T07:47:11.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3744000000000003e-05 2024-09-18T07:47:11.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:11.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942797+0000", 2024-09-18T07:47:11.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110093 2024-09-18T07:47:11.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:47:11.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942903+0000", 2024-09-18T07:47:11.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105741 2024-09-18T07:47:11.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949093+0000", 2024-09-18T07:47:11.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0061907419999999999 2024-09-18T07:47:11.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:11.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.951910+0000", 2024-09-18T07:47:11.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028169779999999999 2024-09-18T07:47:11.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222552+0000", 2024-09-18T07:47:11.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27064144600000001 2024-09-18T07:47:11.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:11.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222639+0000", 2024-09-18T07:47:11.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7190999999999999e-05 2024-09-18T07:47:11.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.495 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.495 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222678+0000", 2024-09-18T07:47:11.495 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8798999999999999e-05 2024-09-18T07:47:11.495 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.495 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.495 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.495 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.495 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:215883 10.8 10:10ce8856:::benchmark_data_smithi137_151160_object80067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-18T07:47:11.495 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.949012+0000", 2024-09-18T07:47:11.495 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 62.391056745999997, 2024-09-18T07:47:11.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.281923759, 2024-09-18T07:47:11.496 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:11.496 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:11.496 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:11.496 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:11.496 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:11.496 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:11.496 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 215883 2024-09-18T07:47:11.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.496 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:11.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.496 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:11.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949012+0000", 2024-09-18T07:47:11.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:11.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949012+0000", 2024-09-18T07:47:11.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:11.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:11.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949014+0000", 2024-09-18T07:47:11.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.063e-06 2024-09-18T07:47:11.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:11.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949020+0000", 2024-09-18T07:47:11.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7259999999999999e-06 2024-09-18T07:47:11.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:11.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949031+0000", 2024-09-18T07:47:11.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1748e-05 2024-09-18T07:47:11.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:11.499 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949039+0000", 2024-09-18T07:47:11.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0400000000000004e-06 2024-09-18T07:47:11.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:11.499 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949121+0000", 2024-09-18T07:47:11.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2622999999999998e-05 2024-09-18T07:47:11.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:11.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949221+0000", 2024-09-18T07:47:11.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100164 2024-09-18T07:47:11.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:47:11.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949325+0000", 2024-09-18T07:47:11.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103342 2024-09-18T07:47:11.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:11.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.952116+0000", 2024-09-18T07:47:11.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027912850000000001 2024-09-18T07:47:11.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.953945+0000", 2024-09-18T07:47:11.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018293129999999999 2024-09-18T07:47:11.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:11.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.230849+0000", 2024-09-18T07:47:11.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27690393299999999 2024-09-18T07:47:11.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:11.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.230891+0000", 2024-09-18T07:47:11.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1854999999999998e-05 2024-09-18T07:47:11.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:11.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:11.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:11.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.230936+0000", 2024-09-18T07:47:11.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4666999999999998e-05 2024-09-18T07:47:11.502 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.502 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.502 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.502 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:11.503 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:11.503 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:47:11.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:47:11.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:11.505 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:11.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:11.505 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:11 smithi203 ceph-mon[32237]: pgmap v861: 158 pgs: 16 unknown, 26 active+clean+remapped, 23 active+undersized, 10 active+undersized+degraded, 83 active+clean; 483 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 32/597 objects degraded (5.360%); 33/597 objects misplaced (5.528%) 2024-09-18T07:47:11.506 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:11 smithi203 ceph-mon[32237]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-18T07:47:11.506 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:11 smithi203 ceph-mon[32237]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-18T07:47:11.506 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:11 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-18T07:47:11.506 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:11 smithi203 ceph-mon[32237]: osd.5 v1:172.21.15.203:6804/1993911072 boot 2024-09-18T07:47:11.506 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:11 smithi203 ceph-mon[32237]: osdmap e460: 8 total, 8 up, 7 in 2024-09-18T07:47:11.506 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:11 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1752476864' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:11.506 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:11 smithi203 ceph-mon[32237]: from='client.36410 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:11.606 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:11.627 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:11.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:11.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:11.729 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:11.751 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:11.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:11.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:11.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:11.876 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:11.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:11.977 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:11.999 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:11.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:12.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:12.100 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:12.122 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:12.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:12.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:12.224 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:12.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:12.247 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:12.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:12.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:12.370 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:12.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:12.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:12.437 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:47:12.437 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:47:12.437 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:47:12.437 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:47:12.437 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:47:12.437 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[29408]: from='client.36410 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:47:12.437 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[29408]: osdmap e461: 8 total, 8 up, 7 in 2024-09-18T07:47:12.438 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[29408]: 1.8 scrub starts 2024-09-18T07:47:12.438 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[29408]: 1.8 scrub ok 2024-09-18T07:47:12.438 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:12.438 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:47:12.438 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:47:12.438 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:47:12.439 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:47:12.439 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:47:12.439 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:47:12.439 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[39175]: from='client.36410 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:47:12.439 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[39175]: osdmap e461: 8 total, 8 up, 7 in 2024-09-18T07:47:12.439 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[39175]: 1.8 scrub starts 2024-09-18T07:47:12.439 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[39175]: 1.8 scrub ok 2024-09-18T07:47:12.439 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:12.440 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:47:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:12 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:47:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:12 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:47:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:12 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:47:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:12 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:47:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:12 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:47:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:12 smithi203 ceph-mon[32237]: from='client.36410 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:47:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:12 smithi203 ceph-mon[32237]: osdmap e461: 8 total, 8 up, 7 in 2024-09-18T07:47:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:12 smithi203 ceph-mon[32237]: 1.8 scrub starts 2024-09-18T07:47:12.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:12 smithi203 ceph-mon[32237]: 1.8 scrub ok 2024-09-18T07:47:12.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:12 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:12.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:47:12.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:12.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:12.494 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:12.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:12.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:12.617 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:12.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:12.718 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:12.742 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:12.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:12.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:12.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:12.867 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:12.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:12.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:12.969 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:12.990 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:12.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:13.094 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:13.115 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:13.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:13.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:13.194 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-18T07:47:13.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:13.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:13.240 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:13.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:13.274 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:13 smithi137 ceph-mon[29408]: pgmap v864: 158 pgs: 7 peering, 6 unknown, 19 active+clean+remapped, 17 active+undersized, 5 active+undersized+degraded, 104 active+clean; 483 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s; 21/597 objects degraded (3.518%); 26/597 objects misplaced (4.355%) 2024-09-18T07:47:13.274 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:13 smithi137 ceph-mon[29408]: osdmap e462: 8 total, 8 up, 7 in 2024-09-18T07:47:13.274 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:13 smithi137 ceph-mon[29408]: 1.1 scrub starts 2024-09-18T07:47:13.274 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:13 smithi137 ceph-mon[29408]: 1.1 scrub ok 2024-09-18T07:47:13.274 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:13 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:47:13.275 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:13 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:47:13.275 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:13 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/278039683' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:47:13.275 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:13 smithi137 ceph-mon[29408]: from='client.36416 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:47:13.275 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:13 smithi137 ceph-mon[39175]: pgmap v864: 158 pgs: 7 peering, 6 unknown, 19 active+clean+remapped, 17 active+undersized, 5 active+undersized+degraded, 104 active+clean; 483 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s; 21/597 objects degraded (3.518%); 26/597 objects misplaced (4.355%) 2024-09-18T07:47:13.275 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:13 smithi137 ceph-mon[39175]: osdmap e462: 8 total, 8 up, 7 in 2024-09-18T07:47:13.275 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:13 smithi137 ceph-mon[39175]: 1.1 scrub starts 2024-09-18T07:47:13.275 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:13 smithi137 ceph-mon[39175]: 1.1 scrub ok 2024-09-18T07:47:13.275 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:13 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:47:13.276 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:13 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:47:13.276 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:13 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/278039683' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:47:13.276 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:13 smithi137 ceph-mon[39175]: from='client.36416 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:47:13.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T07:47:13.341 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:13.363 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:13.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:13.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:13 smithi203 ceph-mon[32237]: pgmap v864: 158 pgs: 7 peering, 6 unknown, 19 active+clean+remapped, 17 active+undersized, 5 active+undersized+degraded, 104 active+clean; 483 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s; 21/597 objects degraded (3.518%); 26/597 objects misplaced (4.355%) 2024-09-18T07:47:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:13 smithi203 ceph-mon[32237]: osdmap e462: 8 total, 8 up, 7 in 2024-09-18T07:47:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:13 smithi203 ceph-mon[32237]: 1.1 scrub starts 2024-09-18T07:47:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:13 smithi203 ceph-mon[32237]: 1.1 scrub ok 2024-09-18T07:47:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:13 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:47:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:13 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:47:13.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:13 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/278039683' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:47:13.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:13 smithi203 ceph-mon[32237]: from='client.36416 ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-18T07:47:13.464 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:13.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:13.491 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:13.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:13.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:13.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:13.616 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:13.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:13.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:13.739 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:13.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:13.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:13.841 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:13.866 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:13.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:13.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:13.991 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:13.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:13.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:14.024 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:14.029 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:14.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:14.114 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:14.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:14.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:14.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:14.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:14.239 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:14.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:14.340 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:14.356 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:47:14.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:14.367 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:14.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:14.457 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:14 smithi137 ceph-mon[29408]: 3.1f scrub starts 2024-09-18T07:47:14.458 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:14 smithi137 ceph-mon[29408]: 3.1f scrub ok 2024-09-18T07:47:14.458 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:14 smithi137 ceph-mon[29408]: 1.a scrub starts 2024-09-18T07:47:14.458 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:14 smithi137 ceph-mon[29408]: 1.a scrub ok 2024-09-18T07:47:14.458 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:14 smithi137 ceph-mon[29408]: from='client.36416 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:47:14.458 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:14 smithi137 ceph-mon[29408]: osdmap e463: 8 total, 8 up, 7 in 2024-09-18T07:47:14.458 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:14 smithi137 ceph-mon[39175]: 3.1f scrub starts 2024-09-18T07:47:14.458 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:14 smithi137 ceph-mon[39175]: 3.1f scrub ok 2024-09-18T07:47:14.458 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:14 smithi137 ceph-mon[39175]: 1.a scrub starts 2024-09-18T07:47:14.458 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:14 smithi137 ceph-mon[39175]: 1.a scrub ok 2024-09-18T07:47:14.458 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:14 smithi137 ceph-mon[39175]: from='client.36416 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:47:14.459 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:14 smithi137 ceph-mon[39175]: osdmap e463: 8 total, 8 up, 7 in 2024-09-18T07:47:14.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:14 smithi203 ceph-mon[32237]: 3.1f scrub starts 2024-09-18T07:47:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:14 smithi203 ceph-mon[32237]: 3.1f scrub ok 2024-09-18T07:47:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:14 smithi203 ceph-mon[32237]: 1.a scrub starts 2024-09-18T07:47:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:14 smithi203 ceph-mon[32237]: 1.a scrub ok 2024-09-18T07:47:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:14 smithi203 ceph-mon[32237]: from='client.36416 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-18T07:47:14.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:14 smithi203 ceph-mon[32237]: osdmap e463: 8 total, 8 up, 7 in 2024-09-18T07:47:14.468 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:14.489 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:14.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:14.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:14.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:14.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:14.622 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:14.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:14.675 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:47:14.675 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:47:14.692 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:47:14.692 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:47:14.708 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:47:14.708 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:47:14.723 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:14.724 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:47:14.724 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:47:14.741 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:47:14.741 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:47:14.749 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:14.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:14.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:14.758 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:47:14.758 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:47:14.774 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:47:14.775 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:47:14.791 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:47:14.791 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:47:14.851 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:14.877 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:14.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:14.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:14.950 DEBUG:teuthology.orchestra.run.smithi137:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-18T07:47:14.953 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-18T07:47:14.953 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-18T07:47:14.978 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:15.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:15.005 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:15.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:15.043 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:hints = 1 2024-09-18T07:47:15.043 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-18T07:47:15.043 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Object prefix: benchmark_data_smithi137_181069 2024-09-18T07:47:15.043 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:47:15.043 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 0 0 0 0 0 0 - 0 2024-09-18T07:47:15.106 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:15.141 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:15.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:15.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:15.242 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:15.279 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:15.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:15.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:15.377 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:47:15.377 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:47:15.381 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:15.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:15.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:15.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:15 smithi203 ceph-mon[32237]: pgmap v867: 158 pgs: 10 peering, 1 unknown, 14 active+clean+remapped, 11 active+undersized, 4 active+undersized+degraded, 118 active+clean; 483 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 20/597 objects degraded (3.350%); 12/597 objects misplaced (2.010%) 2024-09-18T07:47:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:15 smithi203 ceph-mon[32237]: 5.8 scrub starts 2024-09-18T07:47:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:15 smithi203 ceph-mon[32237]: 5.8 scrub ok 2024-09-18T07:47:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:15 smithi203 ceph-mon[32237]: 5.13 scrub starts 2024-09-18T07:47:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:15 smithi203 ceph-mon[32237]: 5.13 scrub ok 2024-09-18T07:47:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:15 smithi203 ceph-mon[32237]: 5.3 scrub starts 2024-09-18T07:47:15.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:15 smithi203 ceph-mon[32237]: 5.3 scrub ok 2024-09-18T07:47:15.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:15 smithi203 ceph-mon[32237]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:47:15.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:15 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3127962660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:15.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:15 smithi203 ceph-mon[32237]: 6.1a scrub starts 2024-09-18T07:47:15.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:15 smithi203 ceph-mon[32237]: 6.1a scrub ok 2024-09-18T07:47:15.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:15 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 20/597 objects degraded (3.350%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-09-18T07:47:15.525 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:15.560 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:15.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:15.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:15.662 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:15.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[29408]: pgmap v867: 158 pgs: 10 peering, 1 unknown, 14 active+clean+remapped, 11 active+undersized, 4 active+undersized+degraded, 118 active+clean; 483 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 20/597 objects degraded (3.350%); 12/597 objects misplaced (2.010%) 2024-09-18T07:47:15.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[29408]: 5.8 scrub starts 2024-09-18T07:47:15.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[29408]: 5.8 scrub ok 2024-09-18T07:47:15.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[29408]: 5.13 scrub starts 2024-09-18T07:47:15.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[29408]: 5.13 scrub ok 2024-09-18T07:47:15.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[29408]: 5.3 scrub starts 2024-09-18T07:47:15.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[29408]: 5.3 scrub ok 2024-09-18T07:47:15.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[29408]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:47:15.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3127962660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:15.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[29408]: 6.1a scrub starts 2024-09-18T07:47:15.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[29408]: 6.1a scrub ok 2024-09-18T07:47:15.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 20/597 objects degraded (3.350%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-09-18T07:47:15.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[39175]: pgmap v867: 158 pgs: 10 peering, 1 unknown, 14 active+clean+remapped, 11 active+undersized, 4 active+undersized+degraded, 118 active+clean; 483 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 20/597 objects degraded (3.350%); 12/597 objects misplaced (2.010%) 2024-09-18T07:47:15.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[39175]: 5.8 scrub starts 2024-09-18T07:47:15.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[39175]: 5.8 scrub ok 2024-09-18T07:47:15.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[39175]: 5.13 scrub starts 2024-09-18T07:47:15.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[39175]: 5.13 scrub ok 2024-09-18T07:47:15.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[39175]: 5.3 scrub starts 2024-09-18T07:47:15.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[39175]: 5.3 scrub ok 2024-09-18T07:47:15.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[39175]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-18T07:47:15.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3127962660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:15.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[39175]: 6.1a scrub starts 2024-09-18T07:47:15.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[39175]: 6.1a scrub ok 2024-09-18T07:47:15.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:15 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 20/597 objects degraded (3.350%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-09-18T07:47:15.697 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:15.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:15.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:15.757 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 6] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 6, 7, 4, 5] 2024-09-18T07:47:15.757 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:47:15.757 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-18T07:47:15.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd in 1 2024-09-18T07:47:15.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:15.840 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:15.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:15.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:15.942 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:15.980 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:15.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:15.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:16.044 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 1 16 3753 3737 233.523 233.562 0.00266022 0.00426557 2024-09-18T07:47:16.082 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:16.118 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:16.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:16.220 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:16.247 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:16.253 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:16.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:16.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:16.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:16.391 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:16.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:16.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:16.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:16 smithi137 ceph-mon[29408]: 6.11 scrub starts 2024-09-18T07:47:16.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:16 smithi137 ceph-mon[29408]: 6.11 scrub ok 2024-09-18T07:47:16.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:16 smithi137 ceph-mon[29408]: 1.4 scrub starts 2024-09-18T07:47:16.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:16 smithi137 ceph-mon[29408]: 1.4 scrub ok 2024-09-18T07:47:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:16 smithi137 ceph-mon[39175]: 6.11 scrub starts 2024-09-18T07:47:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:16 smithi137 ceph-mon[39175]: 6.11 scrub ok 2024-09-18T07:47:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:16 smithi137 ceph-mon[39175]: 1.4 scrub starts 2024-09-18T07:47:16.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:16 smithi137 ceph-mon[39175]: 1.4 scrub ok 2024-09-18T07:47:16.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:16 smithi203 ceph-mon[32237]: 6.11 scrub starts 2024-09-18T07:47:16.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:16 smithi203 ceph-mon[32237]: 6.11 scrub ok 2024-09-18T07:47:16.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:16 smithi203 ceph-mon[32237]: 1.4 scrub starts 2024-09-18T07:47:16.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:16 smithi203 ceph-mon[32237]: 1.4 scrub ok 2024-09-18T07:47:16.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:16.528 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:16.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:16.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:16.630 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:16.664 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:16.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:16.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:16.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:16.790 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:16.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:16.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:16.925 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:16.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:16.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:17.027 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:17.043 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 2 16 7166 7150 223.403 213.312 0.00504992 0.00446541 2024-09-18T07:47:17.070 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:17.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:17.172 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:17.207 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:17.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:17.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T07:47:17.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:17.345 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:17.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:17.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:17.360 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:47:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:47:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:47:17.361 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:17 smithi137 ceph-mon[29408]: pgmap v868: 158 pgs: 10 peering, 148 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-18T07:47:17.361 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:17 smithi137 ceph-mon[29408]: 5.d scrub starts 2024-09-18T07:47:17.361 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:17 smithi137 ceph-mon[29408]: 5.d scrub ok 2024-09-18T07:47:17.361 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:17 smithi137 ceph-mon[29408]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/597 objects degraded (3.350%), 4 pgs degraded, 14 pgs undersized) 2024-09-18T07:47:17.361 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:17 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:47:17.361 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:17 smithi137 ceph-mon[39175]: pgmap v868: 158 pgs: 10 peering, 148 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-18T07:47:17.361 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:17 smithi137 ceph-mon[39175]: 5.d scrub starts 2024-09-18T07:47:17.361 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:17 smithi137 ceph-mon[39175]: 5.d scrub ok 2024-09-18T07:47:17.362 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:17 smithi137 ceph-mon[39175]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/597 objects degraded (3.350%), 4 pgs degraded, 14 pgs undersized) 2024-09-18T07:47:17.362 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:17 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:47:17.447 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:17.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:17 smithi203 ceph-mon[32237]: pgmap v868: 158 pgs: 10 peering, 148 active+clean; 583 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-18T07:47:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:17 smithi203 ceph-mon[32237]: 5.d scrub starts 2024-09-18T07:47:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:17 smithi203 ceph-mon[32237]: 5.d scrub ok 2024-09-18T07:47:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:17 smithi203 ceph-mon[32237]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/597 objects degraded (3.350%), 4 pgs degraded, 14 pgs undersized) 2024-09-18T07:47:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:17 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:47:17.481 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:17.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:17.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:17.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:17.618 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:17.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:17.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:17.719 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:17.755 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:17.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:17.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:17.857 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:17.874 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:17.923 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:17.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:17.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:17.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:47:18.026 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:18.043 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 3 16 10291 10275 214.019 195.312 0.00393588 0.00466393 2024-09-18T07:47:18.057 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:18.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:18.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:18.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:18.187 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:18.209 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:47:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:47:18.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:18.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:18.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:18.270 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:47:18.293 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:18 smithi137 ceph-mon[29408]: 5.6 scrub starts 2024-09-18T07:47:18.294 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:18 smithi137 ceph-mon[29408]: 5.6 scrub ok 2024-09-18T07:47:18.294 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:18 smithi137 ceph-mon[29408]: 3.d scrub starts 2024-09-18T07:47:18.294 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:18 smithi137 ceph-mon[29408]: 3.d scrub ok 2024-09-18T07:47:18.294 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:18 smithi137 ceph-mon[39175]: 5.6 scrub starts 2024-09-18T07:47:18.294 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:18 smithi137 ceph-mon[39175]: 5.6 scrub ok 2024-09-18T07:47:18.294 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:18 smithi137 ceph-mon[39175]: 3.d scrub starts 2024-09-18T07:47:18.294 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:18 smithi137 ceph-mon[39175]: 3.d scrub ok 2024-09-18T07:47:18.320 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:18.354 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:18.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:18.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:18.456 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:18.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:18 smithi203 ceph-mon[32237]: 5.6 scrub starts 2024-09-18T07:47:18.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:18 smithi203 ceph-mon[32237]: 5.6 scrub ok 2024-09-18T07:47:18.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:18 smithi203 ceph-mon[32237]: 3.d scrub starts 2024-09-18T07:47:18.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:18 smithi203 ceph-mon[32237]: 3.d scrub ok 2024-09-18T07:47:18.490 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:18.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:18.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:18.592 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:18.634 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:18.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:18.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:18.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:18.774 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:18.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:18.875 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:18.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:18.903 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:18.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:19.004 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:19.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:19.041 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:19.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:19.043 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 4 16 12975 12959 202.45 167.75 0.002884 0.00492968 2024-09-18T07:47:19.141 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:19.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:18 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a[29404]: 2024-09-18T07:47:18.848+0000 7fe5973a1640 -1 mon.a@0(leader).osd e464 definitely_dead 0 2024-09-18T07:47:19.179 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:19.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:19.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:19.260 INFO:teuthology.orchestra.run.smithi137.stderr:marked in osd.1. 2024-09-18T07:47:19.280 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:19.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:19.304 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:19.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:19.405 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:19.427 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:19.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:19.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:19.480 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[29408]: pgmap v869: 158 pgs: 158 active+clean; 334 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 50 MiB/s wr, 796 op/s; 0 B/s, 1 objects/s recovering 2024-09-18T07:47:19.480 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:19.480 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[29408]: 6.c scrub starts 2024-09-18T07:47:19.480 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[29408]: 6.c scrub ok 2024-09-18T07:47:19.481 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3294993985' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:47:19.481 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[29408]: from='client.36434 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:47:19.481 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:47:19.481 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[29408]: from='client.36434 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:47:19.481 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[29408]: osdmap e464: 8 total, 8 up, 7 in 2024-09-18T07:47:19.481 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1462895714' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-18T07:47:19.481 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[39175]: pgmap v869: 158 pgs: 158 active+clean; 334 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 50 MiB/s wr, 796 op/s; 0 B/s, 1 objects/s recovering 2024-09-18T07:47:19.481 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:19.481 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[39175]: 6.c scrub starts 2024-09-18T07:47:19.481 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[39175]: 6.c scrub ok 2024-09-18T07:47:19.481 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3294993985' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:47:19.482 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[39175]: from='client.36434 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:47:19.482 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:47:19.482 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[39175]: from='client.36434 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:47:19.482 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[39175]: osdmap e464: 8 total, 8 up, 7 in 2024-09-18T07:47:19.482 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:19 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1462895714' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-18T07:47:19.528 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:19.550 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:19.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:19.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:19.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:19.673 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:19.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:19.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:19 smithi203 ceph-mon[32237]: pgmap v869: 158 pgs: 158 active+clean; 334 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 50 MiB/s wr, 796 op/s; 0 B/s, 1 objects/s recovering 2024-09-18T07:47:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:19 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:19 smithi203 ceph-mon[32237]: 6.c scrub starts 2024-09-18T07:47:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:19 smithi203 ceph-mon[32237]: 6.c scrub ok 2024-09-18T07:47:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:19 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3294993985' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:47:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:19 smithi203 ceph-mon[32237]: from='client.36434 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:47:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:19 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:47:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:19 smithi203 ceph-mon[32237]: from='client.36434 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:47:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:19 smithi203 ceph-mon[32237]: osdmap e464: 8 total, 8 up, 7 in 2024-09-18T07:47:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:19 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1462895714' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-18T07:47:19.732 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:47:19.732 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:47:19.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:19.797 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:19.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:19.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:19.899 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:19.922 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:19.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:19.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:20.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:20.043 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 5 16 13617 13601 169.986 40.125 0.00190635 0.00496846 2024-09-18T07:47:20.045 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:20.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:20.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:20.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:20.173 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:20.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:20.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:20.275 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:20.296 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:20.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:20.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:20.398 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:20.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:20 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1462895714' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-18T07:47:20.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:20 smithi137 ceph-mon[29408]: osdmap e465: 8 total, 8 up, 8 in 2024-09-18T07:47:20.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:20 smithi137 ceph-mon[29408]: osdmap e466: 8 total, 8 up, 8 in 2024-09-18T07:47:20.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:20 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1462895714' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-18T07:47:20.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:20 smithi137 ceph-mon[39175]: osdmap e465: 8 total, 8 up, 8 in 2024-09-18T07:47:20.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:20 smithi137 ceph-mon[39175]: osdmap e466: 8 total, 8 up, 8 in 2024-09-18T07:47:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:20.421 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:20.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:20.522 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:20.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:20.543 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:20.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:20.552 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-18T07:47:20.644 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:20.665 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:20.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:20.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:20.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:20 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1462895714' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-18T07:47:20.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:20 smithi203 ceph-mon[32237]: osdmap e465: 8 total, 8 up, 8 in 2024-09-18T07:47:20.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:20 smithi203 ceph-mon[32237]: osdmap e466: 8 total, 8 up, 8 in 2024-09-18T07:47:20.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:20.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:20.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:20.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:20.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:20.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:20.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:20.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:21.016 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:21.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T07:47:21.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:21.040 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:21.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:21.043 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 6 16 13617 13601 141.657 0 - 0.00496846 2024-09-18T07:47:21.142 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:21.178 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:21.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:21.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:21.269 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:21 smithi137 ceph-mon[29408]: pgmap v872: 158 pgs: 158 active+clean; 427 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 0 B/s, 1 objects/s recovering 2024-09-18T07:47:21.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:21.323 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:21.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:21.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:21.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:21.460 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:21.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:21.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:21.556 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:21 smithi137 ceph-mon[29408]: osdmap e467: 8 total, 8 up, 8 in 2024-09-18T07:47:21.556 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:21 smithi137 ceph-mon[39175]: pgmap v872: 158 pgs: 158 active+clean; 427 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 0 B/s, 1 objects/s recovering 2024-09-18T07:47:21.556 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:21 smithi137 ceph-mon[39175]: osdmap e467: 8 total, 8 up, 8 in 2024-09-18T07:47:21.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:21.605 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:21.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:21.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:21.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:21 smithi203 ceph-mon[32237]: pgmap v872: 158 pgs: 158 active+clean; 427 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 0 B/s, 1 objects/s recovering 2024-09-18T07:47:21.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:21 smithi203 ceph-mon[32237]: osdmap e467: 8 total, 8 up, 8 in 2024-09-18T07:47:21.731 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:21.754 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:21.771 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:21.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:21.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:21.873 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:21.902 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:21.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:21.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:22.005 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:22.036 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:22.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:22.043 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 7 16 15242 15226 135.928 50.7812 0.00471895 0.00734816 2024-09-18T07:47:22.139 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:22.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:22.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:22.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:22.279 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:22.310 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:22.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:22.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:22.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:22.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:22 smithi137 ceph-mon[29408]: Health check failed: Degraded data redundancy: 1685/37410 objects degraded (4.504%), 3 pgs degraded (PG_DEGRADED) 2024-09-18T07:47:22.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:22 smithi137 ceph-mon[39175]: Health check failed: Degraded data redundancy: 1685/37410 objects degraded (4.504%), 3 pgs degraded (PG_DEGRADED) 2024-09-18T07:47:22.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:22.434 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:22.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:22.535 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:22.574 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:22.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:22.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:22.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:22.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:22 smithi203 ceph-mon[32237]: Health check failed: Degraded data redundancy: 1685/37410 objects degraded (4.504%), 3 pgs degraded (PG_DEGRADED) 2024-09-18T07:47:22.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:22.721 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:22.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:22.822 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:22.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:22.853 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:22.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:22.954 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:22.975 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:22.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:22.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:23.044 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 8 15 16793 16778 131.061 97 0.00503354 0.00762261 2024-09-18T07:47:23.077 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:23.107 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:23.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:23.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:23.210 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:23.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:23.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:23.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:23.312 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:47:23.312 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:47:23.312 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:47:23.312 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:47:23.313 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:47:23.313 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:47:23.313 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:47:23.313 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:47:23.313 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:47:23.314 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:47:23.314 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:47:23.314 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:47:23.314 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:47:23.314 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:47:23.314 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:47:23.314 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:47:23.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:23.391 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:23.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:23.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:23.493 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:23.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:23.526 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:23.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:23.617 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:47:23.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:23.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:23 smithi137 ceph-mon[29408]: pgmap v875: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 768 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 108 MiB/s wr, 1.74k op/s; 1685/37410 objects degraded (4.504%); 2443/37410 objects misplaced (6.530%); 1.1 MiB/s, 35 objects/s recovering 2024-09-18T07:47:23.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:23 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1636016592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:23.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:23 smithi137 ceph-mon[39175]: pgmap v875: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 768 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 108 MiB/s wr, 1.74k op/s; 1685/37410 objects degraded (4.504%); 2443/37410 objects misplaced (6.530%); 1.1 MiB/s, 35 objects/s recovering 2024-09-18T07:47:23.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:23 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1636016592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:23.705 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:23.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:23.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:23.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:23 smithi203 ceph-mon[32237]: pgmap v875: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 768 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 108 MiB/s wr, 1.74k op/s; 1685/37410 objects degraded (4.504%); 2443/37410 objects misplaced (6.530%); 1.1 MiB/s, 35 objects/s recovering 2024-09-18T07:47:23.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:23 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1636016592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:23.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:23.849 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:23.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:23.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:23.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:23.997 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:23.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:24.044 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 9 16 18724 18708 129.9 120.625 0.00600078 0.00769156 2024-09-18T07:47:24.098 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:24.125 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:24.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:24.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:24.228 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:24.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:24.259 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:24.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:24.360 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:24.390 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:24.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:24.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:24.492 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:24.522 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:24.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:24.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:24.690 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:24.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:24.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:24.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:24.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:24.826 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:24.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:24.927 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:24.958 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:24.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:24.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:25.044 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 10 15 20564 20549 128.416 115.062 0.00428679 0.00777961 2024-09-18T07:47:25.060 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:25.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:25.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:25.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:25.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:25.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:25.247 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:25.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:25.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:25.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:25.399 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:25.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:25.500 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:25.537 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:25.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:25.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:25.553 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 6, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 6, 7, 4, 5] 2024-09-18T07:47:25.553 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:47:25.553 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-18T07:47:25.553 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-18T07:47:25.553 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:47:25.593 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:25 smithi137 ceph-mon[39175]: pgmap v876: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 151 active+clean; 917 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 735 B/s rd, 98 MiB/s wr, 1.56k op/s; 3361/44604 objects degraded (7.535%); 2443/44604 objects misplaced (5.477%); 851 KiB/s, 31 objects/s recovering 2024-09-18T07:47:25.593 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:25 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:47:25.593 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:25 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:25.593 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:25 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:25.597 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:25 smithi137 ceph-mon[29408]: pgmap v876: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 151 active+clean; 917 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 735 B/s rd, 98 MiB/s wr, 1.56k op/s; 3361/44604 objects degraded (7.535%); 2443/44604 objects misplaced (5.477%); 851 KiB/s, 31 objects/s recovering 2024-09-18T07:47:25.597 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:25 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:47:25.597 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:25 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:25.597 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:25 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:25.640 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:25.668 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:25.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:25.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:25 smithi203 ceph-mon[32237]: pgmap v876: 158 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 151 active+clean; 917 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 735 B/s rd, 98 MiB/s wr, 1.56k op/s; 3361/44604 objects degraded (7.535%); 2443/44604 objects misplaced (5.477%); 851 KiB/s, 31 objects/s recovering 2024-09-18T07:47:25.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:25 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:47:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:25 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:25 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:25.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:25.800 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:25.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:25.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:25.902 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:25.929 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:25.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:25.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:26.031 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:26.044 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 11 16 22068 22052 125.279 93.9375 0.00522629 0.00797306 2024-09-18T07:47:26.079 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:26.100 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:26.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:26.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:26.106 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:47:26.106 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:47:26.106 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:47:26.106 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:47:26.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.106 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172965 9.e e335/289 9:7fd9b0fe:::benchmark_data_smithi137_126072_object40465:head v 335'10641, mlcod=335'10639)", 2024-09-18T07:47:26.106 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.554501+0000", 2024-09-18T07:47:26.107 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 347.54557636800001, 2024-09-18T07:47:26.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.178309934, 2024-09-18T07:47:26.107 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.107 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.107 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.107 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554501+0000", 2024-09-18T07:47:26.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554501+0000", 2024-09-18T07:47:26.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554502+0000", 2024-09-18T07:47:26.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.443e-06 2024-09-18T07:47:26.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554510+0000", 2024-09-18T07:47:26.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0660000000000004e-06 2024-09-18T07:47:26.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554518+0000", 2024-09-18T07:47:26.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0339999999999997e-06 2024-09-18T07:47:26.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554523+0000", 2024-09-18T07:47:26.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8740000000000003e-06 2024-09-18T07:47:26.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554567+0000", 2024-09-18T07:47:26.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3294000000000003e-05 2024-09-18T07:47:26.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554586+0000", 2024-09-18T07:47:26.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9419000000000002e-05 2024-09-18T07:47:26.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732782+0000", 2024-09-18T07:47:26.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.178196153 2024-09-18T07:47:26.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732811+0000", 2024-09-18T07:47:26.111 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8651000000000001e-05 2024-09-18T07:47:26.111 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.111 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.111 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.111 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172968 9.0 e335/289 9:02c69111:::benchmark_data_smithi137_126072_object40468:head v 335'10807, mlcod=335'10804)", 2024-09-18T07:47:26.111 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.554702+0000", 2024-09-18T07:47:26.111 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 347.54537502099998, 2024-09-18T07:47:26.111 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.179882141, 2024-09-18T07:47:26.111 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.111 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554702+0000", 2024-09-18T07:47:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554702+0000", 2024-09-18T07:47:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554704+0000", 2024-09-18T07:47:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.359e-06 2024-09-18T07:47:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554711+0000", 2024-09-18T07:47:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.199e-06 2024-09-18T07:47:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554720+0000", 2024-09-18T07:47:26.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5629999999999998e-06 2024-09-18T07:47:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554727+0000", 2024-09-18T07:47:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5889999999999999e-06 2024-09-18T07:47:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554767+0000", 2024-09-18T07:47:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9712000000000002e-05 2024-09-18T07:47:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554782+0000", 2024-09-18T07:47:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5639000000000001e-05 2024-09-18T07:47:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734508+0000", 2024-09-18T07:47:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17972560900000001 2024-09-18T07:47:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734584+0000", 2024-09-18T07:47:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6470999999999993e-05 2024-09-18T07:47:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.115 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172969 9.6 e335/289 9:60d9b40c:::benchmark_data_smithi137_126072_object40469:head v 335'10756, mlcod=335'10754)", 2024-09-18T07:47:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.554970+0000", 2024-09-18T07:47:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 347.54510768400002, 2024-09-18T07:47:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17787228899999999, 2024-09-18T07:47:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.116 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554970+0000", 2024-09-18T07:47:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554970+0000", 2024-09-18T07:47:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554971+0000", 2024-09-18T07:47:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0070000000000001e-06 2024-09-18T07:47:26.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.118 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.118 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554976+0000", 2024-09-18T07:47:26.118 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6330000000000001e-06 2024-09-18T07:47:26.118 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.118 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.118 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.554993+0000", 2024-09-18T07:47:26.118 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6949e-05 2024-09-18T07:47:26.118 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.118 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.118 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.118 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.555000+0000", 2024-09-18T07:47:26.118 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5579999999999997e-06 2024-09-18T07:47:26.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.119 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.119 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.119 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.555176+0000", 2024-09-18T07:47:26.119 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017662999999999999 2024-09-18T07:47:26.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.119 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.119 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.119 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.555193+0000", 2024-09-18T07:47:26.119 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7212000000000001e-05 2024-09-18T07:47:26.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.119 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.119 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.120 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732814+0000", 2024-09-18T07:47:26.120 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.177620315 2024-09-18T07:47:26.120 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.120 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.120 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.120 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732842+0000", 2024-09-18T07:47:26.120 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7985000000000001e-05 2024-09-18T07:47:26.120 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.120 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.120 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.120 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.120 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.120 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172970 9.3 e335/289 9:c7e8a587:::benchmark_data_smithi137_126072_object40470:head v 335'10785, mlcod=335'10782)", 2024-09-18T07:47:26.120 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.557577+0000", 2024-09-18T07:47:26.121 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 347.54249976400001, 2024-09-18T07:47:26.121 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.173544268, 2024-09-18T07:47:26.121 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.121 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.121 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.121 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.121 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.121 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.121 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557577+0000", 2024-09-18T07:47:26.121 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.121 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.121 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.121 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.121 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557577+0000", 2024-09-18T07:47:26.122 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.122 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.122 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.122 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.122 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557579+0000", 2024-09-18T07:47:26.122 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5099999999999999e-06 2024-09-18T07:47:26.122 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.122 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.122 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.122 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557589+0000", 2024-09-18T07:47:26.122 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8079999999999999e-06 2024-09-18T07:47:26.122 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.122 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557598+0000", 2024-09-18T07:47:26.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2490000000000007e-06 2024-09-18T07:47:26.123 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.123 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557604+0000", 2024-09-18T07:47:26.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2079999999999997e-06 2024-09-18T07:47:26.123 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.123 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726578+0000", 2024-09-18T07:47:26.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.168973331 2024-09-18T07:47:26.123 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.124 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.124 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726596+0000", 2024-09-18T07:47:26.124 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8159e-05 2024-09-18T07:47:26.124 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.124 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.124 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731084+0000", 2024-09-18T07:47:26.124 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044883980000000002 2024-09-18T07:47:26.124 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.124 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.124 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731122+0000", 2024-09-18T07:47:26.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7605000000000003e-05 2024-09-18T07:47:26.125 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.125 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.125 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.125 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172972 9.e e335/289 9:796b9798:::benchmark_data_smithi137_126072_object40472:head v 335'10642, mlcod=335'10639)", 2024-09-18T07:47:26.125 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.557623+0000", 2024-09-18T07:47:26.125 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 347.542453905, 2024-09-18T07:47:26.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.175241702, 2024-09-18T07:47:26.125 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.125 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.125 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.126 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.126 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.126 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557623+0000", 2024-09-18T07:47:26.126 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.126 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.126 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557623+0000", 2024-09-18T07:47:26.126 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.126 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.126 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557624+0000", 2024-09-18T07:47:26.127 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4200000000000001e-07 2024-09-18T07:47:26.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.127 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.127 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557628+0000", 2024-09-18T07:47:26.127 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8139999999999998e-06 2024-09-18T07:47:26.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.127 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.127 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557632+0000", 2024-09-18T07:47:26.127 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6909999999999996e-06 2024-09-18T07:47:26.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.128 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.128 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557636+0000", 2024-09-18T07:47:26.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6629999999999999e-06 2024-09-18T07:47:26.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.128 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.128 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726230+0000", 2024-09-18T07:47:26.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.168594563 2024-09-18T07:47:26.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.128 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.128 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726286+0000", 2024-09-18T07:47:26.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5702000000000003e-05 2024-09-18T07:47:26.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.129 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.129 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.129 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732845+0000", 2024-09-18T07:47:26.129 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065588560000000001 2024-09-18T07:47:26.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.129 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.129 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.129 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.732865+0000", 2024-09-18T07:47:26.129 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9970999999999999e-05 2024-09-18T07:47:26.129 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.129 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.129 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.130 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.130 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172971 9.4 e335/289 9:2cb28e72:::benchmark_data_smithi137_126072_object40471:head v 335'10859, mlcod=335'10857)", 2024-09-18T07:47:26.130 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.557654+0000", 2024-09-18T07:47:26.130 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 347.54242287900001, 2024-09-18T07:47:26.130 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17145278799999999, 2024-09-18T07:47:26.130 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.130 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.130 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.130 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.130 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.130 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.130 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557654+0000", 2024-09-18T07:47:26.131 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.131 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.131 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557654+0000", 2024-09-18T07:47:26.131 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.131 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.131 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557655+0000", 2024-09-18T07:47:26.131 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1099999999999995e-07 2024-09-18T07:47:26.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.131 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557661+0000", 2024-09-18T07:47:26.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2400000000000004e-06 2024-09-18T07:47:26.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557667+0000", 2024-09-18T07:47:26.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9139999999999996e-06 2024-09-18T07:47:26.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.557672+0000", 2024-09-18T07:47:26.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6290000000000001e-06 2024-09-18T07:47:26.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.133 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726169+0000", 2024-09-18T07:47:26.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.168497014 2024-09-18T07:47:26.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.133 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.133 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726237+0000", 2024-09-18T07:47:26.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8677999999999995e-05 2024-09-18T07:47:26.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.133 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.133 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.729066+0000", 2024-09-18T07:47:26.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028290170000000001 2024-09-18T07:47:26.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.134 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.134 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.134 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.729107+0000", 2024-09-18T07:47:26.134 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0685000000000001e-05 2024-09-18T07:47:26.134 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.134 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.134 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.134 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.134 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172976 9.f e335/289 9:f0577982:::benchmark_data_smithi137_126072_object40476:head v 335'10844, mlcod=335'10842)", 2024-09-18T07:47:26.134 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.560675+0000", 2024-09-18T07:47:26.134 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 347.539401774, 2024-09-18T07:47:26.134 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17558478, 2024-09-18T07:47:26.135 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.135 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.135 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.135 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.135 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560675+0000", 2024-09-18T07:47:26.136 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.136 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.136 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560675+0000", 2024-09-18T07:47:26.136 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.137 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560677+0000", 2024-09-18T07:47:26.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.711e-06 2024-09-18T07:47:26.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.137 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560685+0000", 2024-09-18T07:47:26.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1880000000000002e-06 2024-09-18T07:47:26.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.137 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726219+0000", 2024-09-18T07:47:26.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16553400300000001 2024-09-18T07:47:26.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726233+0000", 2024-09-18T07:47:26.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3893000000000001e-05 2024-09-18T07:47:26.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726381+0000", 2024-09-18T07:47:26.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014760199999999999 2024-09-18T07:47:26.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726402+0000", 2024-09-18T07:47:26.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1664000000000001e-05 2024-09-18T07:47:26.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.736225+0000", 2024-09-18T07:47:26.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0098227820000000004 2024-09-18T07:47:26.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.736260+0000", 2024-09-18T07:47:26.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4937000000000001e-05 2024-09-18T07:47:26.139 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.140 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.140 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.140 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172980 9.3 e335/289 9:ce0a7506:::benchmark_data_smithi137_126072_object40480:head v 335'10786, mlcod=335'10783)", 2024-09-18T07:47:26.140 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.560718+0000", 2024-09-18T07:47:26.140 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 347.53935930199998, 2024-09-18T07:47:26.140 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.17056611599999999, 2024-09-18T07:47:26.140 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.140 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.140 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.140 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.141 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560718+0000", 2024-09-18T07:47:26.141 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.141 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.141 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.141 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560718+0000", 2024-09-18T07:47:26.141 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.141 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.141 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.141 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560719+0000", 2024-09-18T07:47:26.141 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8400000000000003e-07 2024-09-18T07:47:26.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.560726+0000", 2024-09-18T07:47:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4000000000000003e-06 2024-09-18T07:47:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726135+0000", 2024-09-18T07:47:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16540897700000001 2024-09-18T07:47:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726166+0000", 2024-09-18T07:47:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.099e-05 2024-09-18T07:47:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.728991+0000", 2024-09-18T07:47:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028248840000000002 2024-09-18T07:47:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.729009+0000", 2024-09-18T07:47:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7804000000000001e-05 2024-09-18T07:47:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731249+0000", 2024-09-18T07:47:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022400050000000002 2024-09-18T07:47:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.731284+0000", 2024-09-18T07:47:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5271999999999997e-05 2024-09-18T07:47:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:172984 9.0 e335/289 9:066846aa:::benchmark_data_smithi137_126072_object40484:head v 335'10808, mlcod=335'10805)", 2024-09-18T07:47:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:41:38.567459+0000", 2024-09-18T07:47:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 347.53261820799997, 2024-09-18T07:47:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16718034100000001, 2024-09-18T07:47:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567459+0000", 2024-09-18T07:47:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567459+0000", 2024-09-18T07:47:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567461+0000", 2024-09-18T07:47:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1550000000000001e-06 2024-09-18T07:47:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.567471+0000", 2024-09-18T07:47:26.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0348e-05 2024-09-18T07:47:26.147 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.147 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.147 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.147 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726150+0000", 2024-09-18T07:47:26.147 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15867869100000001 2024-09-18T07:47:26.147 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.147 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.147 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.147 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.726175+0000", 2024-09-18T07:47:26.147 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5316e-05 2024-09-18T07:47:26.147 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.147 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.147 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.148 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.728757+0000", 2024-09-18T07:47:26.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002581262 2024-09-18T07:47:26.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.148 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.148 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.728778+0000", 2024-09-18T07:47:26.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1739999999999999e-05 2024-09-18T07:47:26.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.148 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.148 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734588+0000", 2024-09-18T07:47:26.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005809519 2024-09-18T07:47:26.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.149 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.149 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:41:38.734639+0000", 2024-09-18T07:47:26.149 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1310000000000002e-05 2024-09-18T07:47:26.149 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.149 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.149 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.149 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.15699.0:244806 9.f e354/289 9:f6d427cd:::benchmark_data_smithi137_126072_object112306:head v 354'15363, mlcod=354'15361)", 2024-09-18T07:47:26.149 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:42:36.240579+0000", 2024-09-18T07:47:26.149 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 289.85949838499999, 2024-09-18T07:47:26.149 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16868968600000001, 2024-09-18T07:47:26.149 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.150 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.150 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.150 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.150 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.150 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240579+0000", 2024-09-18T07:47:26.150 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.150 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.150 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240579+0000", 2024-09-18T07:47:26.150 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.150 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240581+0000", 2024-09-18T07:47:26.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9599999999999999e-06 2024-09-18T07:47:26.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.240589+0000", 2024-09-18T07:47:26.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4710000000000007e-06 2024-09-18T07:47:26.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396235+0000", 2024-09-18T07:47:26.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.155645916 2024-09-18T07:47:26.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.152 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.152 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396253+0000", 2024-09-18T07:47:26.152 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8170000000000001e-05 2024-09-18T07:47:26.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.152 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.152 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396312+0000", 2024-09-18T07:47:26.152 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9144000000000002e-05 2024-09-18T07:47:26.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.152 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.152 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.396337+0000", 2024-09-18T07:47:26.153 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4771000000000001e-05 2024-09-18T07:47:26.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.153 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.153 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.153 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.409231+0000", 2024-09-18T07:47:26.153 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012893745 2024-09-18T07:47:26.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.153 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.153 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.153 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:42:36.409268+0000", 2024-09-18T07:47:26.153 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7509000000000003e-05 2024-09-18T07:47:26.153 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:71849 10.2 e374/364 10:41855215:::benchmark_data_smithi137_151160_object71848:head v 374'4514, mlcod=374'4512)", 2024-09-18T07:47:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:46.388333+0000", 2024-09-18T07:47:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 219.711743832, 2024-09-18T07:47:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.225317779, 2024-09-18T07:47:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.154 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388333+0000", 2024-09-18T07:47:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388333+0000", 2024-09-18T07:47:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388335+0000", 2024-09-18T07:47:26.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7910000000000001e-06 2024-09-18T07:47:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388381+0000", 2024-09-18T07:47:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5909999999999999e-05 2024-09-18T07:47:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388394+0000", 2024-09-18T07:47:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2599000000000001e-05 2024-09-18T07:47:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388402+0000", 2024-09-18T07:47:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2060000000000007e-06 2024-09-18T07:47:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606027+0000", 2024-09-18T07:47:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.21762539 2024-09-18T07:47:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606060+0000", 2024-09-18T07:47:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2991000000000003e-05 2024-09-18T07:47:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.613606+0000", 2024-09-18T07:47:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0075457730000000004 2024-09-18T07:47:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.613651+0000", 2024-09-18T07:47:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5118999999999999e-05 2024-09-18T07:47:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.158 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.159 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:71852 10.b e374/364 10:dda9ad8f:::benchmark_data_smithi137_151160_object71851:head v 374'4479, mlcod=374'4476)", 2024-09-18T07:47:26.159 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:46.390175+0000", 2024-09-18T07:47:26.159 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 219.709902102, 2024-09-18T07:47:26.159 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.22548416099999999, 2024-09-18T07:47:26.159 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.159 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.159 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.159 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.159 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.159 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.390175+0000", 2024-09-18T07:47:26.159 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.390175+0000", 2024-09-18T07:47:26.160 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.390177+0000", 2024-09-18T07:47:26.160 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.548e-06 2024-09-18T07:47:26.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.390217+0000", 2024-09-18T07:47:26.160 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0203000000000001e-05 2024-09-18T07:47:26.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606008+0000", 2024-09-18T07:47:26.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.21579153700000001 2024-09-18T07:47:26.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606020+0000", 2024-09-18T07:47:26.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1741e-05 2024-09-18T07:47:26.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606089+0000", 2024-09-18T07:47:26.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9058000000000002e-05 2024-09-18T07:47:26.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.162 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606112+0000", 2024-09-18T07:47:26.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2410999999999999e-05 2024-09-18T07:47:26.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.162 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.615499+0000", 2024-09-18T07:47:26.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093874549999999994 2024-09-18T07:47:26.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.163 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.163 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.615659+0000", 2024-09-18T07:47:26.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016020800000000001 2024-09-18T07:47:26.163 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.163 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.163 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.163 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:71850 10.2 e374/364 10:48354359:::benchmark_data_smithi137_151160_object71849:head v 374'4515, mlcod=374'4512)", 2024-09-18T07:47:26.163 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:46.392229+0000", 2024-09-18T07:47:26.163 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 219.707848517, 2024-09-18T07:47:26.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.221450217, 2024-09-18T07:47:26.163 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.164 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.164 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.164 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.392229+0000", 2024-09-18T07:47:26.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.392229+0000", 2024-09-18T07:47:26.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.392231+0000", 2024-09-18T07:47:26.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0109999999999999e-06 2024-09-18T07:47:26.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.392282+0000", 2024-09-18T07:47:26.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1019000000000001e-05 2024-09-18T07:47:26.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.166 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.166 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.605989+0000", 2024-09-18T07:47:26.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.213706807 2024-09-18T07:47:26.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.166 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.166 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606006+0000", 2024-09-18T07:47:26.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7289e-05 2024-09-18T07:47:26.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.166 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.166 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.608388+0000", 2024-09-18T07:47:26.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023825399999999998 2024-09-18T07:47:26.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.167 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.167 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.608412+0000", 2024-09-18T07:47:26.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4093000000000001e-05 2024-09-18T07:47:26.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.167 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.167 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.613656+0000", 2024-09-18T07:47:26.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052432060000000003 2024-09-18T07:47:26.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.167 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.168 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.613679+0000", 2024-09-18T07:47:26.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3252e-05 2024-09-18T07:47:26.168 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.168 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.168 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.168 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:122152 10.2 e390/364 10:48dcd9dc:::benchmark_data_smithi137_151160_object122151:head v 390'7640, mlcod=390'7635)", 2024-09-18T07:47:26.168 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:28.657416+0000", 2024-09-18T07:47:26.168 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 177.44266134700001, 2024-09-18T07:47:26.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032362285999999997, 2024-09-18T07:47:26.168 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.169 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.169 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.169 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.169 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657416+0000", 2024-09-18T07:47:26.169 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.169 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657416+0000", 2024-09-18T07:47:26.169 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.170 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.170 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657417+0000", 2024-09-18T07:47:26.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3400000000000001e-06 2024-09-18T07:47:26.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.170 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.170 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657439+0000", 2024-09-18T07:47:26.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1903999999999999e-05 2024-09-18T07:47:26.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.170 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.170 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657449+0000", 2024-09-18T07:47:26.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.825e-06 2024-09-18T07:47:26.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.657455+0000", 2024-09-18T07:47:26.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4659999999999996e-06 2024-09-18T07:47:26.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.664829+0000", 2024-09-18T07:47:26.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0073740740000000004 2024-09-18T07:47:26.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.664879+0000", 2024-09-18T07:47:26.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9308999999999998e-05 2024-09-18T07:47:26.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.172 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.689734+0000", 2024-09-18T07:47:26.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.024855497000000001 2024-09-18T07:47:26.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.172 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:28.689778+0000", 2024-09-18T07:47:26.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3871000000000001e-05 2024-09-18T07:47:26.172 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.172 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.173 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.173 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.173 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:135810 10.b e392/364 10:da1ccd0a:::benchmark_data_smithi137_151160_object135809:head v 392'8433, mlcod=392'8431)", 2024-09-18T07:47:26.173 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:40.045405+0000", 2024-09-18T07:47:26.173 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 166.05467215499999, 2024-09-18T07:47:26.173 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13276168799999999, 2024-09-18T07:47:26.173 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.173 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.173 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.173 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.173 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.173 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045405+0000", 2024-09-18T07:47:26.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045405+0000", 2024-09-18T07:47:26.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045406+0000", 2024-09-18T07:47:26.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3820000000000001e-06 2024-09-18T07:47:26.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.175 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045446+0000", 2024-09-18T07:47:26.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9371e-05 2024-09-18T07:47:26.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.175 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045457+0000", 2024-09-18T07:47:26.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1657e-05 2024-09-18T07:47:26.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.175 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045464+0000", 2024-09-18T07:47:26.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7530000000000004e-06 2024-09-18T07:47:26.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.176 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.176 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045533+0000", 2024-09-18T07:47:26.176 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.923e-05 2024-09-18T07:47:26.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.176 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.176 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.045551+0000", 2024-09-18T07:47:26.176 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7844999999999999e-05 2024-09-18T07:47:26.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.177 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.177 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.178077+0000", 2024-09-18T07:47:26.177 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.132526221 2024-09-18T07:47:26.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.177 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.177 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.178167+0000", 2024-09-18T07:47:26.177 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9228999999999996e-05 2024-09-18T07:47:26.177 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.177 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.177 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.178 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:135798 10.2 e392/364 10:4cac0c3f:::benchmark_data_smithi137_151160_object135797:head v 392'8494, mlcod=392'8491)", 2024-09-18T07:47:26.178 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:40.048398+0000", 2024-09-18T07:47:26.178 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 166.05167881599999, 2024-09-18T07:47:26.178 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13237333200000001, 2024-09-18T07:47:26.178 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.178 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.178 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.178 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.178 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.178 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.048398+0000", 2024-09-18T07:47:26.178 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.178 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.179 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.179 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.048398+0000", 2024-09-18T07:47:26.179 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.179 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.179 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.179 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.048400+0000", 2024-09-18T07:47:26.179 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.826e-06 2024-09-18T07:47:26.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.179 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.179 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.179 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.048452+0000", 2024-09-18T07:47:26.179 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1360000000000003e-05 2024-09-18T07:47:26.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.180 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.180 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.180 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.177902+0000", 2024-09-18T07:47:26.180 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12945019699999999 2024-09-18T07:47:26.180 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.180 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.180 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.180 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.177917+0000", 2024-09-18T07:47:26.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:47:26.181 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4766999999999999e-05 2024-09-18T07:47:26.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.181 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.182 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.178327+0000", 2024-09-18T07:47:26.182 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00041051999999999999 2024-09-18T07:47:26.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.182 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.182 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.182 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.178347+0000", 2024-09-18T07:47:26.182 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9945e-05 2024-09-18T07:47:26.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.182 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.182 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.182 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.180740+0000", 2024-09-18T07:47:26.182 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023925790000000001 2024-09-18T07:47:26.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:40.180772+0000", 2024-09-18T07:47:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2138000000000003e-05 2024-09-18T07:47:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:234911 10.0 e437/427 10:0d294a58:::benchmark_data_smithi137_151160_object99095:head v 437'14595, mlcod=437'14593)", 2024-09-18T07:47:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:24.285825+0000", 2024-09-18T07:47:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 61.814251736000003, 2024-09-18T07:47:26.183 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031579732999999999, 2024-09-18T07:47:26.184 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.184 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.184 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.184 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.184 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.184 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.184 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.285825+0000", 2024-09-18T07:47:26.184 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.184 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.184 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.184 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.285825+0000", 2024-09-18T07:47:26.184 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.185 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.185 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.185 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.285827+0000", 2024-09-18T07:47:26.185 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.849e-06 2024-09-18T07:47:26.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.185 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.185 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.185 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.285838+0000", 2024-09-18T07:47:26.185 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0479000000000001e-05 2024-09-18T07:47:26.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.185 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.185 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.185 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.285852+0000", 2024-09-18T07:47:26.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4275e-05 2024-09-18T07:47:26.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.186 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.186 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.285860+0000", 2024-09-18T07:47:26.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1799999999999996e-06 2024-09-18T07:47:26.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.186 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.186 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.298469+0000", 2024-09-18T07:47:26.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012608698 2024-09-18T07:47:26.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.186 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.187 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.298514+0000", 2024-09-18T07:47:26.187 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5500999999999997e-05 2024-09-18T07:47:26.187 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.187 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.187 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.187 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.317083+0000", 2024-09-18T07:47:26.187 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018568344000000001 2024-09-18T07:47:26.187 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.187 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.187 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.187 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:24.317405+0000", 2024-09-18T07:47:26.187 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032240699999999998 2024-09-18T07:47:26.187 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.188 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.188 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.188 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.188 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.188 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:236198 10.b e438/364 10:d724674c:::benchmark_data_smithi137_151160_object100382:head v 438'14651, mlcod=438'14647)", 2024-09-18T07:47:26.188 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:25.444745+0000", 2024-09-18T07:47:26.188 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 60.655332401999999, 2024-09-18T07:47:26.188 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.031984378000000001, 2024-09-18T07:47:26.188 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.188 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.188 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.188 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.188 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.189 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.189 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:25.444745+0000", 2024-09-18T07:47:26.189 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.189 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.189 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.189 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:25.444745+0000", 2024-09-18T07:47:26.189 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.189 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.189 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.189 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:25.444746+0000", 2024-09-18T07:47:26.189 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3370000000000001e-06 2024-09-18T07:47:26.189 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.190 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.190 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.190 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:25.444754+0000", 2024-09-18T07:47:26.190 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2009999999999999e-06 2024-09-18T07:47:26.190 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.190 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.190 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.190 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:25.444767+0000", 2024-09-18T07:47:26.190 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2291e-05 2024-09-18T07:47:26.190 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.190 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.190 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.190 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:25.444774+0000", 2024-09-18T07:47:26.190 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0949999999999998e-06 2024-09-18T07:47:26.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.191 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.191 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:25.448471+0000", 2024-09-18T07:47:26.191 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036968230000000001 2024-09-18T07:47:26.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.191 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.192 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:25.448492+0000", 2024-09-18T07:47:26.192 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1642e-05 2024-09-18T07:47:26.192 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.192 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.192 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.192 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:25.476692+0000", 2024-09-18T07:47:26.192 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028199617999999999 2024-09-18T07:47:26.192 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.192 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.193 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.193 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:25.476729+0000", 2024-09-18T07:47:26.193 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7370999999999998e-05 2024-09-18T07:47:26.193 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.193 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.193 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.193 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.193 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.193 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:21892 11.7 e467/466 11:e85aa922:::benchmark_data_smithi137_181069_object21891:head v 467'1356, mlcod=0'0)", 2024-09-18T07:47:26.193 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:25.911884+0000", 2024-09-18T07:47:26.194 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.188193097, 2024-09-18T07:47:26.194 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.034995857999999998, 2024-09-18T07:47:26.194 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.194 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.194 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.194 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.194 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.194 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.194 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.911884+0000", 2024-09-18T07:47:26.194 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.194 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.194 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.194 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.195 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.911884+0000", 2024-09-18T07:47:26.195 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.195 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.195 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.195 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.195 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.911885+0000", 2024-09-18T07:47:26.195 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1799999999999999e-06 2024-09-18T07:47:26.195 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.195 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.195 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.195 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.911931+0000", 2024-09-18T07:47:26.195 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5680999999999999e-05 2024-09-18T07:47:26.195 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.195 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.196 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.196 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.911939+0000", 2024-09-18T07:47:26.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5390000000000007e-06 2024-09-18T07:47:26.196 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.196 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.196 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.196 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.911945+0000", 2024-09-18T07:47:26.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8540000000000002e-06 2024-09-18T07:47:26.196 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.196 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.196 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.196 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.919290+0000", 2024-09-18T07:47:26.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0073443639999999999 2024-09-18T07:47:26.196 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.197 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.197 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.919311+0000", 2024-09-18T07:47:26.197 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1495999999999999e-05 2024-09-18T07:47:26.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.197 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.197 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.946798+0000", 2024-09-18T07:47:26.197 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027486740999999999 2024-09-18T07:47:26.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.197 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.197 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.946880+0000", 2024-09-18T07:47:26.198 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2002999999999994e-05 2024-09-18T07:47:26.198 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.198 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.198 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.198 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.198 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.198 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:21899 11.7 e467/466 11:ec0a2b6d:::benchmark_data_smithi137_181069_object21898:head v 467'1357, mlcod=0'0)", 2024-09-18T07:47:26.198 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:25.916125+0000", 2024-09-18T07:47:26.198 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.183951948, 2024-09-18T07:47:26.198 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033936088000000003, 2024-09-18T07:47:26.198 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:26.198 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:26.198 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:26.199 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:26.199 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.199 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:26.199 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.916125+0000", 2024-09-18T07:47:26.199 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.199 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.199 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.199 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:26.199 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.916125+0000", 2024-09-18T07:47:26.199 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:26.199 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.199 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.199 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:26.199 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.916127+0000", 2024-09-18T07:47:26.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.53e-06 2024-09-18T07:47:26.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.200 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:26.200 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.916158+0000", 2024-09-18T07:47:26.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1318000000000001e-05 2024-09-18T07:47:26.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.200 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:26.200 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.916167+0000", 2024-09-18T07:47:26.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5669999999999995e-06 2024-09-18T07:47:26.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.200 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:26.201 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.916172+0000", 2024-09-18T07:47:26.201 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7699999999999998e-06 2024-09-18T07:47:26.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.201 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:26.201 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.924319+0000", 2024-09-18T07:47:26.201 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081470459999999998 2024-09-18T07:47:26.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.201 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:26.201 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.924343+0000", 2024-09-18T07:47:26.201 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3688000000000001e-05 2024-09-18T07:47:26.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.202 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:26.202 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.948379+0000", 2024-09-18T07:47:26.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.024035562999999999 2024-09-18T07:47:26.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:26.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:26.202 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:26.202 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:25.950061+0000", 2024-09-18T07:47:26.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016826059999999999 2024-09-18T07:47:26.202 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.202 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.202 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.202 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:26.202 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:26.203 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:47:26.203 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:26.229 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:26.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:26.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:26.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:26.396 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:26.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:26.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:26.498 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:26.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:26.527 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:26.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:26.615 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:26.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:26.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:26.674 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:26.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:26.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:26.828 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:26.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:26.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:26.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:26.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:26.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:26 smithi137 ceph-mon[29408]: pgmap v877: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 150 active+clean; 938 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 623 B/s rd, 78 MiB/s wr, 1.25k op/s; 4219/45594 objects degraded (9.253%); 2443/45594 objects misplaced (5.358%); 722 KiB/s, 28 objects/s recovering 2024-09-18T07:47:26.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:47:26.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:26 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:47:26.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:26 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:26.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:26.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:26.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:26 smithi137 ceph-mon[39175]: pgmap v877: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 150 active+clean; 938 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 623 B/s rd, 78 MiB/s wr, 1.25k op/s; 4219/45594 objects degraded (9.253%); 2443/45594 objects misplaced (5.358%); 722 KiB/s, 28 objects/s recovering 2024-09-18T07:47:26.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:47:26.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:26 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:47:26.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:26 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:26.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:26.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:26.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:26.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:26 smithi203 ceph-mon[32237]: pgmap v877: 158 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 150 active+clean; 938 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 623 B/s rd, 78 MiB/s wr, 1.25k op/s; 4219/45594 objects degraded (9.253%); 2443/45594 objects misplaced (5.358%); 722 KiB/s, 28 objects/s recovering 2024-09-18T07:47:26.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:47:26.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:26 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:47:26.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:26 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:26.963 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:26.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:26.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:27.044 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 12 16 23513 23497 122.365 90.3125 0.00805926 0.0081656 2024-09-18T07:47:27.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:27.104 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:27.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:27.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:27.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:27.248 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:27.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:27.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:27.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:27.387 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:27.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:27.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:27.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:47:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:47:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:47:27.489 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:27.518 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:27.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:27.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:27.619 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:27.647 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:27.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:27.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:27.749 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:27.785 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:27.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:27.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:27.887 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:27.918 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:27.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:27.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:28.020 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:28.044 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 13 16 25140 25124 120.774 101.688 0.00256564 0.00827006 2024-09-18T07:47:28.050 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:28.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:28.152 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:28.181 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:28.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:28.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:28.193 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:47:27 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:47:28.283 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:28.317 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:28.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:28.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:28.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:28.443 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:28.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:28.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:28 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:28.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:28 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:47:28.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:28.574 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:28.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:28.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:28.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:28 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:28.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:28 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:47:28.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:28 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:28.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:28 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:47:28.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:28.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:28.717 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:28.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:28.818 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:28.848 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:28.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:28.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:28.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:28.991 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:28.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:28.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:29.044 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 14 16 26807 26791 119.588 104.188 0.0077361 0.00835553 2024-09-18T07:47:29.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:29.133 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:29.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:29.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:29.235 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:29.264 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:29.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:29.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:29.366 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:29.388 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:29.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:29 smithi203 ceph-mon[32237]: pgmap v878: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 150 active+clean; 1.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 122 MiB/s wr, 1.95k op/s; 4219/67821 objects degraded (6.221%); 1192/67821 objects misplaced (1.758%); 10 MiB/s, 179 objects/s recovering 2024-09-18T07:47:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:29 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:29 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 4219/67821 objects degraded (6.221%), 6 pgs degraded (PG_DEGRADED) 2024-09-18T07:47:29.490 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:29.521 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:29.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:29.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:29.623 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:29.651 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:29.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:29.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:29.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:29 smithi137 ceph-mon[29408]: pgmap v878: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 150 active+clean; 1.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 122 MiB/s wr, 1.95k op/s; 4219/67821 objects degraded (6.221%); 1192/67821 objects misplaced (1.758%); 10 MiB/s, 179 objects/s recovering 2024-09-18T07:47:29.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:29 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:29.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:29 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 4219/67821 objects degraded (6.221%), 6 pgs degraded (PG_DEGRADED) 2024-09-18T07:47:29.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:29 smithi137 ceph-mon[39175]: pgmap v878: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 150 active+clean; 1.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 122 MiB/s wr, 1.95k op/s; 4219/67821 objects degraded (6.221%); 1192/67821 objects misplaced (1.758%); 10 MiB/s, 179 objects/s recovering 2024-09-18T07:47:29.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:29 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:29.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:29 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 4219/67821 objects degraded (6.221%), 6 pgs degraded (PG_DEGRADED) 2024-09-18T07:47:29.753 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:29.792 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:29.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:29.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:29.894 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:29.927 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:29.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:30.029 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:30.045 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 15 16 28604 28588 119.099 112.312 0.00692325 0.00838814 2024-09-18T07:47:30.067 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:30.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:30.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:30.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:30.201 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:30.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:30.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:30.305 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:30.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:30.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:30.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:30.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:30.449 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:30.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:30.551 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:30.572 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:30.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:30.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:30.673 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:30.697 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:30.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:30.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:30.799 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:30.822 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:30.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:30.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:30.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:30.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:30.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:30.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:31.045 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 16 16 28991 28975 113.167 24.1875 0.0027242 0.00842065 2024-09-18T07:47:31.049 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:31.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T07:47:31.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:31.073 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:31.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:31.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:31.219 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:31.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:31.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:31.343 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:31.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:31.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:31.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:31 smithi137 ceph-mon[29408]: pgmap v879: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 150 active+clean; 1.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 948 B/s rd, 107 MiB/s wr, 1.71k op/s; 4219/70878 objects degraded (5.952%); 1192/70878 objects misplaced (1.682%); 8.5 MiB/s, 148 objects/s recovering 2024-09-18T07:47:31.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:31 smithi137 ceph-mon[29408]: osdmap e468: 8 total, 8 up, 8 in 2024-09-18T07:47:31.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:31 smithi137 ceph-mon[39175]: pgmap v879: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 150 active+clean; 1.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 948 B/s rd, 107 MiB/s wr, 1.71k op/s; 4219/70878 objects degraded (5.952%); 1192/70878 objects misplaced (1.682%); 8.5 MiB/s, 148 objects/s recovering 2024-09-18T07:47:31.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:31 smithi137 ceph-mon[39175]: osdmap e468: 8 total, 8 up, 8 in 2024-09-18T07:47:31.445 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:31.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:31 smithi203 ceph-mon[32237]: pgmap v879: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 150 active+clean; 1.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 948 B/s rd, 107 MiB/s wr, 1.71k op/s; 4219/70878 objects degraded (5.952%); 1192/70878 objects misplaced (1.682%); 8.5 MiB/s, 148 objects/s recovering 2024-09-18T07:47:31.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:31 smithi203 ceph-mon[32237]: osdmap e468: 8 total, 8 up, 8 in 2024-09-18T07:47:31.469 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:31.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:31.570 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:31.592 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:31.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:31.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:31.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:31.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:31.718 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:31.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:31.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:31.843 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:31.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:31.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:31.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:31.969 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:31.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:31.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:32.045 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 17 16 29116 29100 106.97 7.8125 0.00515175 0.0088736 2024-09-18T07:47:32.071 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:32.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:32.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:32.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:32.199 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:32.221 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:32.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:32.322 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:32.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:32.362 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:32.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:32.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:32 smithi137 ceph-mon[29408]: osdmap e469: 8 total, 8 up, 8 in 2024-09-18T07:47:32.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:32 smithi137 ceph-mon[29408]: osdmap e470: 8 total, 8 up, 8 in 2024-09-18T07:47:32.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:32 smithi137 ceph-mon[39175]: osdmap e469: 8 total, 8 up, 8 in 2024-09-18T07:47:32.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:32 smithi137 ceph-mon[39175]: osdmap e470: 8 total, 8 up, 8 in 2024-09-18T07:47:32.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:32 smithi203 ceph-mon[32237]: osdmap e469: 8 total, 8 up, 8 in 2024-09-18T07:47:32.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:32 smithi203 ceph-mon[32237]: osdmap e470: 8 total, 8 up, 8 in 2024-09-18T07:47:32.463 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:32.491 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:32.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:32.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:32.593 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:32.627 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:32.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:32.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:32.729 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:32.765 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:32.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:32.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:32.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:32.910 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:32.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:33.012 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:33.045 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 18 16 30627 30611 106.274 94.4375 0.0064311 0.0094036 2024-09-18T07:47:33.048 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:33.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:33.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:33.110 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:33.150 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:33.158 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:47:33.178 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:33.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:33.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:33.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:33.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:33.318 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:33.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:33.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:33.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:33 smithi137 ceph-mon[29408]: pgmap v882: 158 pgs: 1 active+recovering+undersized+remapped, 2 peering, 4 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s; 3365/82641 objects degraded (4.072%); 619/82641 objects misplaced (0.749%); 21 MiB/s, 336 objects/s recovering 2024-09-18T07:47:33.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:33 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2559588238' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:33.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:33 smithi137 ceph-mon[29408]: from='client.26539 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:33.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:33 smithi137 ceph-mon[39175]: pgmap v882: 158 pgs: 1 active+recovering+undersized+remapped, 2 peering, 4 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s; 3365/82641 objects degraded (4.072%); 619/82641 objects misplaced (0.749%); 21 MiB/s, 336 objects/s recovering 2024-09-18T07:47:33.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:33 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2559588238' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:33.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:33 smithi137 ceph-mon[39175]: from='client.26539 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:33.443 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:33.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:33.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:33.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:33 smithi203 ceph-mon[32237]: pgmap v882: 158 pgs: 1 active+recovering+undersized+remapped, 2 peering, 4 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s; 3365/82641 objects degraded (4.072%); 619/82641 objects misplaced (0.749%); 21 MiB/s, 336 objects/s recovering 2024-09-18T07:47:33.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:33 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2559588238' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:33 smithi203 ceph-mon[32237]: from='client.26539 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:33.476 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:47:33.479 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:47:33.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:33.570 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:33.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:33.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:33.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:33.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:33.697 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:33.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:33.798 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:33.821 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:33.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:33.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:33.923 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:33.946 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:33.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:33.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:34.045 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 19 16 31123 31107 102.312 31 0.00212707 0.00938264 2024-09-18T07:47:34.047 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:34.071 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:34.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:34.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:34.205 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:34.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:34.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:34.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:34.330 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:34.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:34.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:34.406 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:34 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:47:34.406 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:34 smithi137 ceph-mon[29408]: from='client.26539 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:47:34.406 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:34 smithi137 ceph-mon[29408]: osdmap e471: 8 total, 8 up, 8 in 2024-09-18T07:47:34.406 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:34 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:47:34.407 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:34 smithi137 ceph-mon[39175]: from='client.26539 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:47:34.407 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:34 smithi137 ceph-mon[39175]: osdmap e471: 8 total, 8 up, 8 in 2024-09-18T07:47:34.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:34.454 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:34.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:34.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:34 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:47:34.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:34 smithi203 ceph-mon[32237]: from='client.26539 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:47:34.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:34 smithi203 ceph-mon[32237]: osdmap e471: 8 total, 8 up, 8 in 2024-09-18T07:47:34.514 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:47:34.514 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:47:34.528 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:47:34.528 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:47:34.546 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:47:34.546 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:47:34.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:34.563 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:47:34.564 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:47:34.579 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:34.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:34.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:34.581 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:47:34.581 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:47:34.600 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:47:34.600 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:47:34.618 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:47:34.618 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:47:34.636 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:47:34.636 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:47:34.681 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:34.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:34.704 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:34.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:34.805 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:34.828 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:34.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:34.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:34.930 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:34.952 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:34.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:34.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:35.045 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:47:35.043662+0000 min lat: 0.00129363 max lat: 1.97884 avg lat: 0.0098311 2024-09-18T07:47:35.045 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:47:35.045 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 20 16 31232 31216 97.5373 6.8125 0.00194189 0.0098311 2024-09-18T07:47:35.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:35.076 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:35.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:35.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:35.178 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:35.213 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:35.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:35.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:35.309 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":884,"stamp":"2024-09-18T07:47:33.152076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1884363291,"num_objects":28943,"num_object_clones":0,"num_object_copies":86829,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3365,"num_objects_misplaced":619,"num_objects_unfound":0,"num_objects_dirty":28943,"num_whiteouts":0,"num_read":1870,"num_read_kb":1622,"num_write":29466,"num_write_kb":1839661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3234,"num_bytes_recovered":180368211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30419,"ondisk_log_size":30419,"up":474,"acting":469,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":487,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11564180,"kb_used_data":7178420,"kb_used_omap":20,"kb_used_meta":4385579,"kb_avail":738397036,"statfs":{"total":767960285184,"available":756118564864,"internally_reserved":0,"allocated":7350702080,"data_stored":7344572205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20886,"internal_metadata":4490833514},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,5,4,4,4,0,0,0,0,3,13],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":415236096,"num_objects":6336,"num_object_clones":0,"num_object_copies":19008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-854,"num_objects_misplaced":-573,"num_objects_unfound":0,"num_objects_dirty":6336,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6336,"num_write_kb":405504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1427,"num_bytes_recovered":93519872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.327212"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":878,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.117651+0000","last_change":"2024-09-18T07:47:12.891849+0000","last_active":"2024-09-18T07:47:21.117651+0000","last_peered":"2024-09-18T07:47:21.117651+0000","last_clean":"2024-09-18T07:47:21.117651+0000","last_became_active":"2024-09-18T07:47:11.200534+0000","last_became_peered":"2024-09-18T07:47:11.200534+0000","last_unstale":"2024-09-18T07:47:21.117651+0000","last_undegraded":"2024-09-18T07:47:21.117651+0000","last_fullsized":"2024-09-18T07:47:21.117651+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:12.891802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:12.891802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:25.260952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"323'15","reported_seq":847,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119034+0000","last_change":"2024-09-18T07:47:20.194767+0000","last_active":"2024-09-18T07:47:31.119034+0000","last_peered":"2024-09-18T07:47:31.119034+0000","last_clean":"2024-09-18T07:47:31.119034+0000","last_became_active":"2024-09-18T07:47:20.136254+0000","last_became_peered":"2024-09-18T07:47:20.136254+0000","last_unstale":"2024-09-18T07:47:31.119034+0000","last_undegraded":"2024-09-18T07:47:31.119034+0000","last_fullsized":"2024-09-18T07:47:31.119034+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:47:09.864550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.864550+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:14.398595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":668,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.112719+0000","last_change":"2024-09-18T07:47:20.130967+0000","last_active":"2024-09-18T07:47:21.112719+0000","last_peered":"2024-09-18T07:47:21.112719+0000","last_clean":"2024-09-18T07:47:21.112719+0000","last_became_active":"2024-09-18T07:47:20.130091+0000","last_became_peered":"2024-09-18T07:47:20.130091+0000","last_unstale":"2024-09-18T07:47:21.112719+0000","last_undegraded":"2024-09-18T07:47:21.112719+0000","last_fullsized":"2024-09-18T07:47:21.112719+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:11.422892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:44:11.422892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":726,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119340+0000","last_change":"2024-09-18T07:47:20.145483+0000","last_active":"2024-09-18T07:47:31.119340+0000","last_peered":"2024-09-18T07:47:31.119340+0000","last_clean":"2024-09-18T07:47:31.119340+0000","last_became_active":"2024-09-18T07:47:20.145155+0000","last_became_peered":"2024-09-18T07:47:20.145155+0000","last_unstale":"2024-09-18T07:47:31.119340+0000","last_undegraded":"2024-09-18T07:47:31.119340+0000","last_fullsized":"2024-09-18T07:47:31.119340+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:14.884604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:14.884604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:21.014582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":339,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.124922+0000","last_change":"2024-09-18T07:46:29.320201+0000","last_active":"2024-09-18T07:47:31.124922+0000","last_peered":"2024-09-18T07:47:31.124922+0000","last_clean":"2024-09-18T07:47:31.124922+0000","last_became_active":"2024-09-18T07:46:29.319898+0000","last_became_peered":"2024-09-18T07:46:29.319898+0000","last_unstale":"2024-09-18T07:47:31.124922+0000","last_undegraded":"2024-09-18T07:47:31.124922+0000","last_fullsized":"2024-09-18T07:47:31.124922+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:11.545358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:11.545358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":872,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.120438+0000","last_change":"2024-09-18T07:45:25.982067+0000","last_active":"2024-09-18T07:47:31.120438+0000","last_peered":"2024-09-18T07:47:31.120438+0000","last_clean":"2024-09-18T07:47:31.120438+0000","last_became_active":"2024-09-18T07:37:43.066508+0000","last_became_peered":"2024-09-18T07:37:43.066508+0000","last_unstale":"2024-09-18T07:47:31.120438+0000","last_undegraded":"2024-09-18T07:47:31.120438+0000","last_fullsized":"2024-09-18T07:47:31.120438+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:25.981995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:45:25.981995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"323'27","reported_seq":432,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.121271+0000","last_change":"2024-09-18T07:46:54.956413+0000","last_active":"2024-09-18T07:47:31.121271+0000","last_peered":"2024-09-18T07:47:31.121271+0000","last_clean":"2024-09-18T07:47:31.121271+0000","last_became_active":"2024-09-18T07:40:19.064755+0000","last_became_peered":"2024-09-18T07:40:19.064755+0000","last_unstale":"2024-09-18T07:47:31.121271+0000","last_undegraded":"2024-09-18T07:47:31.121271+0000","last_fullsized":"2024-09-18T07:47:31.121271+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:46:54.956330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:54.956330+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:07.544925+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":788,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119030+0000","last_change":"2024-09-18T07:47:20.137049+0000","last_active":"2024-09-18T07:47:31.119030+0000","last_peered":"2024-09-18T07:47:31.119030+0000","last_clean":"2024-09-18T07:47:31.119030+0000","last_became_active":"2024-09-18T07:47:20.136358+0000","last_became_peered":"2024-09-18T07:47:20.136358+0000","last_unstale":"2024-09-18T07:47:31.119030+0000","last_undegraded":"2024-09-18T07:47:31.119030+0000","last_fullsized":"2024-09-18T07:47:31.119030+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:51.001046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:51.001046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":783,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.113422+0000","last_change":"2024-09-18T07:47:11.202248+0000","last_active":"2024-09-18T07:47:21.113422+0000","last_peered":"2024-09-18T07:47:21.113422+0000","last_clean":"2024-09-18T07:47:21.113422+0000","last_became_active":"2024-09-18T07:47:11.200300+0000","last_became_peered":"2024-09-18T07:47:11.200300+0000","last_unstale":"2024-09-18T07:47:21.113422+0000","last_undegraded":"2024-09-18T07:47:21.113422+0000","last_fullsized":"2024-09-18T07:47:21.113422+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:09.526499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:45:09.526499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"323'15","reported_seq":785,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119073+0000","last_change":"2024-09-18T07:44:43.775510+0000","last_active":"2024-09-18T07:47:31.119073+0000","last_peered":"2024-09-18T07:47:31.119073+0000","last_clean":"2024-09-18T07:47:31.119073+0000","last_became_active":"2024-09-18T07:44:43.775194+0000","last_became_peered":"2024-09-18T07:44:43.775194+0000","last_unstale":"2024-09-18T07:47:31.119073+0000","last_undegraded":"2024-09-18T07:47:31.119073+0000","last_fullsized":"2024-09-18T07:47:31.119073+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:56.021118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:56.021118+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":720,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.113061+0000","last_change":"2024-09-18T07:47:11.192327+0000","last_active":"2024-09-18T07:47:21.113061+0000","last_peered":"2024-09-18T07:47:21.113061+0000","last_clean":"2024-09-18T07:47:21.113061+0000","last_became_active":"2024-09-18T07:47:11.191975+0000","last_became_peered":"2024-09-18T07:47:11.191975+0000","last_unstale":"2024-09-18T07:47:21.113061+0000","last_undegraded":"2024-09-18T07:47:21.113061+0000","last_fullsized":"2024-09-18T07:47:21.113061+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:12.437116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:44:12.437116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":866,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.116411+0000","last_change":"2024-09-18T07:47:20.135642+0000","last_active":"2024-09-18T07:47:21.116411+0000","last_peered":"2024-09-18T07:47:21.116411+0000","last_clean":"2024-09-18T07:47:21.116411+0000","last_became_active":"2024-09-18T07:47:20.135282+0000","last_became_peered":"2024-09-18T07:47:20.135282+0000","last_unstale":"2024-09-18T07:47:21.116411+0000","last_undegraded":"2024-09-18T07:47:21.116411+0000","last_fullsized":"2024-09-18T07:47:21.116411+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:37.233589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:37.233589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":742,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.113373+0000","last_change":"2024-09-18T07:47:20.142711+0000","last_active":"2024-09-18T07:47:21.113373+0000","last_peered":"2024-09-18T07:47:21.113373+0000","last_clean":"2024-09-18T07:47:21.113373+0000","last_became_active":"2024-09-18T07:47:20.142404+0000","last_became_peered":"2024-09-18T07:47:20.142404+0000","last_unstale":"2024-09-18T07:47:21.113373+0000","last_undegraded":"2024-09-18T07:47:21.113373+0000","last_fullsized":"2024-09-18T07:47:21.113373+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:57.358853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:57.358853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"323'7","reported_seq":847,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119413+0000","last_change":"2024-09-18T07:47:20.150030+0000","last_active":"2024-09-18T07:47:31.119413+0000","last_peered":"2024-09-18T07:47:31.119413+0000","last_clean":"2024-09-18T07:47:31.119413+0000","last_became_active":"2024-09-18T07:47:20.135463+0000","last_became_peered":"2024-09-18T07:47:20.135463+0000","last_unstale":"2024-09-18T07:47:31.119413+0000","last_undegraded":"2024-09-18T07:47:31.119413+0000","last_fullsized":"2024-09-18T07:47:31.119413+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'7","last_scrub_stamp":"2024-09-18T07:42:46.496916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:46.496916+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":662,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123871+0000","last_change":"2024-09-18T07:47:20.143380+0000","last_active":"2024-09-18T07:47:31.123871+0000","last_peered":"2024-09-18T07:47:31.123871+0000","last_clean":"2024-09-18T07:47:31.123871+0000","last_became_active":"2024-09-18T07:47:20.142891+0000","last_became_peered":"2024-09-18T07:47:20.142891+0000","last_unstale":"2024-09-18T07:47:31.123871+0000","last_undegraded":"2024-09-18T07:47:31.123871+0000","last_fullsized":"2024-09-18T07:47:31.123871+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:53.754622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:53.754622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:01.005011+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":770,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.112856+0000","last_change":"2024-09-18T07:47:20.134832+0000","last_active":"2024-09-18T07:47:21.112856+0000","last_peered":"2024-09-18T07:47:21.112856+0000","last_clean":"2024-09-18T07:47:21.112856+0000","last_became_active":"2024-09-18T07:47:20.134460+0000","last_became_peered":"2024-09-18T07:47:20.134460+0000","last_unstale":"2024-09-18T07:47:21.112856+0000","last_undegraded":"2024-09-18T07:47:21.112856+0000","last_fullsized":"2024-09-18T07:47:21.112856+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.522665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.522665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":812,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119915+0000","last_change":"2024-09-18T07:47:11.188757+0000","last_active":"2024-09-18T07:47:31.119915+0000","last_peered":"2024-09-18T07:47:31.119915+0000","last_clean":"2024-09-18T07:47:31.119915+0000","last_became_active":"2024-09-18T07:47:11.187916+0000","last_became_peered":"2024-09-18T07:47:11.187916+0000","last_unstale":"2024-09-18T07:47:31.119915+0000","last_undegraded":"2024-09-18T07:47:31.119915+0000","last_fullsized":"2024-09-18T07:47:31.119915+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:40.432906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:40.432906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":883,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.116938+0000","last_change":"2024-09-18T07:46:21.601224+0000","last_active":"2024-09-18T07:47:21.116938+0000","last_peered":"2024-09-18T07:47:21.116938+0000","last_clean":"2024-09-18T07:47:21.116938+0000","last_became_active":"2024-09-18T07:44:43.773339+0000","last_became_peered":"2024-09-18T07:44:43.773339+0000","last_unstale":"2024-09-18T07:47:21.116938+0000","last_undegraded":"2024-09-18T07:47:21.116938+0000","last_fullsized":"2024-09-18T07:47:21.116938+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:21.601160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:21.601160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:24.118079+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"323'27","reported_seq":1013,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.126635+0000","last_change":"2024-09-18T07:47:20.183463+0000","last_active":"2024-09-18T07:47:31.126635+0000","last_peered":"2024-09-18T07:47:31.126635+0000","last_clean":"2024-09-18T07:47:31.126635+0000","last_became_active":"2024-09-18T07:47:20.140525+0000","last_became_peered":"2024-09-18T07:47:20.140525+0000","last_unstale":"2024-09-18T07:47:31.126635+0000","last_undegraded":"2024-09-18T07:47:31.126635+0000","last_fullsized":"2024-09-18T07:47:31.126635+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:45:53.825805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:45:53.825805+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":450,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119803+0000","last_change":"2024-09-18T07:47:20.133202+0000","last_active":"2024-09-18T07:47:31.119803+0000","last_peered":"2024-09-18T07:47:31.119803+0000","last_clean":"2024-09-18T07:47:31.119803+0000","last_became_active":"2024-09-18T07:47:20.131623+0000","last_became_peered":"2024-09-18T07:47:20.131623+0000","last_unstale":"2024-09-18T07:47:31.119803+0000","last_undegraded":"2024-09-18T07:47:31.119803+0000","last_fullsized":"2024-09-18T07:47:31.119803+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:18.008882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:18.008882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:39.019200+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":734,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123843+0000","last_change":"2024-09-18T07:47:11.201194+0000","last_active":"2024-09-18T07:47:31.123843+0000","last_peered":"2024-09-18T07:47:31.123843+0000","last_clean":"2024-09-18T07:47:31.123843+0000","last_became_active":"2024-09-18T07:47:11.200897+0000","last_became_peered":"2024-09-18T07:47:11.200897+0000","last_unstale":"2024-09-18T07:47:31.123843+0000","last_undegraded":"2024-09-18T07:47:31.123843+0000","last_fullsized":"2024-09-18T07:47:31.123843+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:57.174827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:57.174827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":804,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119242+0000","last_change":"2024-09-18T07:47:20.144941+0000","last_active":"2024-09-18T07:47:31.119242+0000","last_peered":"2024-09-18T07:47:31.119242+0000","last_clean":"2024-09-18T07:47:31.119242+0000","last_became_active":"2024-09-18T07:47:20.144620+0000","last_became_peered":"2024-09-18T07:47:20.144620+0000","last_unstale":"2024-09-18T07:47:31.119242+0000","last_undegraded":"2024-09-18T07:47:31.119242+0000","last_fullsized":"2024-09-18T07:47:31.119242+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:31.323963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:31.323963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"323'18","reported_seq":586,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.122123+0000","last_change":"2024-09-18T07:47:20.181142+0000","last_active":"2024-09-18T07:47:31.122123+0000","last_peered":"2024-09-18T07:47:31.122123+0000","last_clean":"2024-09-18T07:47:31.122123+0000","last_became_active":"2024-09-18T07:47:20.139508+0000","last_became_peered":"2024-09-18T07:47:20.139508+0000","last_unstale":"2024-09-18T07:47:31.122123+0000","last_undegraded":"2024-09-18T07:47:31.122123+0000","last_fullsized":"2024-09-18T07:47:31.122123+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:46:47.978409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:47.978409+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:53.523262+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":726,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119588+0000","last_change":"2024-09-18T07:44:43.786027+0000","last_active":"2024-09-18T07:47:31.119588+0000","last_peered":"2024-09-18T07:47:31.119588+0000","last_clean":"2024-09-18T07:47:31.119588+0000","last_became_active":"2024-09-18T07:44:43.785095+0000","last_became_peered":"2024-09-18T07:44:43.785095+0000","last_unstale":"2024-09-18T07:47:31.119588+0000","last_undegraded":"2024-09-18T07:47:31.119588+0000","last_fullsized":"2024-09-18T07:47:31.119588+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.930435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.930435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:03.948295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":652,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.124900+0000","last_change":"2024-09-18T07:47:11.195374+0000","last_active":"2024-09-18T07:47:31.124900+0000","last_peered":"2024-09-18T07:47:31.124900+0000","last_clean":"2024-09-18T07:47:31.124900+0000","last_became_active":"2024-09-18T07:47:11.194447+0000","last_became_peered":"2024-09-18T07:47:11.194447+0000","last_unstale":"2024-09-18T07:47:31.124900+0000","last_undegraded":"2024-09-18T07:47:31.124900+0000","last_fullsized":"2024-09-18T07:47:31.124900+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:44:18.607501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:44:18.607501+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":598,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119266+0000","last_change":"2024-09-18T07:47:20.167246+0000","last_active":"2024-09-18T07:47:31.119266+0000","last_peered":"2024-09-18T07:47:31.119266+0000","last_clean":"2024-09-18T07:47:31.119266+0000","last_became_active":"2024-09-18T07:47:20.139152+0000","last_became_peered":"2024-09-18T07:47:20.139152+0000","last_unstale":"2024-09-18T07:47:31.119266+0000","last_undegraded":"2024-09-18T07:47:31.119266+0000","last_fullsized":"2024-09-18T07:47:31.119266+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:09.496455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.496455+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:23.094728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"323'20","reported_seq":932,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.129313+0000","last_change":"2024-09-18T07:43:46.102137+0000","last_active":"2024-09-18T07:47:21.129313+0000","last_peered":"2024-09-18T07:47:21.129313+0000","last_clean":"2024-09-18T07:47:21.129313+0000","last_became_active":"2024-09-18T07:35:01.410926+0000","last_became_peered":"2024-09-18T07:35:01.410926+0000","last_unstale":"2024-09-18T07:47:21.129313+0000","last_undegraded":"2024-09-18T07:47:21.129313+0000","last_fullsized":"2024-09-18T07:47:21.129313+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:43:46.102041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:43:46.102041+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:20.045500+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"398'14","reported_seq":632,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123877+0000","last_change":"2024-09-18T07:45:53.887882+0000","last_active":"2024-09-18T07:47:31.123877+0000","last_peered":"2024-09-18T07:47:31.123877+0000","last_clean":"2024-09-18T07:47:31.123877+0000","last_became_active":"2024-09-18T07:44:43.786594+0000","last_became_peered":"2024-09-18T07:44:43.786594+0000","last_unstale":"2024-09-18T07:47:31.123877+0000","last_undegraded":"2024-09-18T07:47:31.123877+0000","last_fullsized":"2024-09-18T07:47:31.123877+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:45:53.887755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:45:53.887755+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"468'1838","reported_seq":3608,"reported_epoch":468,"state":"peering","last_fresh":"2024-09-18T07:47:30.665694+0000","last_change":"2024-09-18T07:47:31.121682+0000","last_active":"2024-09-18T07:47:31.121682+0000","last_peered":"2024-09-18T07:47:30.665694+0000","last_clean":"2024-09-18T07:47:19.251795+0000","last_became_active":"2024-09-18T07:47:21.171303+0000","last_became_peered":"2024-09-18T07:47:21.171303+0000","last_unstale":"2024-09-18T07:47:30.665694+0000","last_undegraded":"2024-09-18T07:47:30.665694+0000","last_fullsized":"2024-09-18T07:47:21.117409+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":1838,"log_dups_size":0,"ondisk_log_size":1838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120455168,"num_objects":1838,"num_object_clones":0,"num_object_copies":5514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1838,"num_write_kb":117632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":867,"num_bytes_recovered":56819712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":295,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.113163+0000","last_change":"2024-09-18T07:47:13.167529+0000","last_active":"2024-09-18T07:47:21.113163+0000","last_peered":"2024-09-18T07:47:21.113163+0000","last_clean":"2024-09-18T07:47:21.113163+0000","last_became_active":"2024-09-18T07:47:11.203901+0000","last_became_peered":"2024-09-18T07:47:11.203901+0000","last_unstale":"2024-09-18T07:47:21.113163+0000","last_undegraded":"2024-09-18T07:47:21.113163+0000","last_fullsized":"2024-09-18T07:47:21.113163+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:47:13.167470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:47:13.167470+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:29.279753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"323'27","reported_seq":668,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.120779+0000","last_change":"2024-09-18T07:47:20.190490+0000","last_active":"2024-09-18T07:47:31.120779+0000","last_peered":"2024-09-18T07:47:31.120779+0000","last_clean":"2024-09-18T07:47:31.120779+0000","last_became_active":"2024-09-18T07:47:20.141595+0000","last_became_peered":"2024-09-18T07:47:20.141595+0000","last_unstale":"2024-09-18T07:47:31.120779+0000","last_undegraded":"2024-09-18T07:47:31.120779+0000","last_fullsized":"2024-09-18T07:47:31.120779+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:42:33.391715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:33.391715+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":909,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.121419+0000","last_change":"2024-09-18T07:47:20.143318+0000","last_active":"2024-09-18T07:47:31.121419+0000","last_peered":"2024-09-18T07:47:31.121419+0000","last_clean":"2024-09-18T07:47:31.121419+0000","last_became_active":"2024-09-18T07:47:20.141653+0000","last_became_peered":"2024-09-18T07:47:20.141653+0000","last_unstale":"2024-09-18T07:47:31.121419+0000","last_undegraded":"2024-09-18T07:47:31.121419+0000","last_fullsized":"2024-09-18T07:47:31.121419+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:25.143715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:25.143715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:38.019485+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"461'12","reported_seq":929,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119956+0000","last_change":"2024-09-18T07:47:11.191234+0000","last_active":"2024-09-18T07:47:31.119956+0000","last_peered":"2024-09-18T07:47:31.119956+0000","last_clean":"2024-09-18T07:47:31.119956+0000","last_became_active":"2024-09-18T07:47:11.190670+0000","last_became_peered":"2024-09-18T07:47:11.190670+0000","last_unstale":"2024-09-18T07:47:31.119956+0000","last_undegraded":"2024-09-18T07:47:31.119956+0000","last_fullsized":"2024-09-18T07:47:31.119956+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:41:27.965205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.965205+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:19.937172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":710,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.113646+0000","last_change":"2024-09-18T07:47:20.139676+0000","last_active":"2024-09-18T07:47:21.113646+0000","last_peered":"2024-09-18T07:47:21.113646+0000","last_clean":"2024-09-18T07:47:21.113646+0000","last_became_active":"2024-09-18T07:47:20.138987+0000","last_became_peered":"2024-09-18T07:47:20.138987+0000","last_unstale":"2024-09-18T07:47:21.113646+0000","last_undegraded":"2024-09-18T07:47:21.113646+0000","last_fullsized":"2024-09-18T07:47:21.113646+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:45.670301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:45.670301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:46.096766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"469'1818","reported_seq":1850,"reported_epoch":469,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T07:47:31.208447+0000","last_change":"2024-09-18T07:47:21.194954+0000","last_active":"2024-09-18T07:47:31.208447+0000","last_peered":"2024-09-18T07:47:31.208447+0000","last_clean":"2024-09-18T07:47:19.268115+0000","last_became_active":"2024-09-18T07:47:21.193418+0000","last_became_peered":"2024-09-18T07:47:21.193418+0000","last_unstale":"2024-09-18T07:47:31.208447+0000","last_undegraded":"2024-09-18T07:47:21.122445+0000","last_fullsized":"2024-09-18T07:47:21.115944+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":1818,"log_dups_size":0,"ondisk_log_size":1818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119144448,"num_objects":1818,"num_object_clones":0,"num_object_copies":5454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1818,"num_write_kb":116352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,2],"avail_no_missing":["7","0","2"],"object_location_counts":[{"shards":"0,2,7","objects":831},{"shards":"2,7","objects":987}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":339,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119905+0000","last_change":"2024-09-18T07:44:43.923223+0000","last_active":"2024-09-18T07:47:31.119905+0000","last_peered":"2024-09-18T07:47:31.119905+0000","last_clean":"2024-09-18T07:47:31.119905+0000","last_became_active":"2024-09-18T07:44:43.786396+0000","last_became_peered":"2024-09-18T07:44:43.786396+0000","last_unstale":"2024-09-18T07:47:31.119905+0000","last_undegraded":"2024-09-18T07:47:31.119905+0000","last_fullsized":"2024-09-18T07:47:31.119905+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:44:43.923179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:44:43.923179+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"323'30","reported_seq":970,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119105+0000","last_change":"2024-09-18T07:45:18.084183+0000","last_active":"2024-09-18T07:47:31.119105+0000","last_peered":"2024-09-18T07:47:31.119105+0000","last_clean":"2024-09-18T07:47:31.119105+0000","last_became_active":"2024-09-18T07:35:01.418909+0000","last_became_peered":"2024-09-18T07:35:01.418909+0000","last_unstale":"2024-09-18T07:47:31.119105+0000","last_undegraded":"2024-09-18T07:47:31.119105+0000","last_fullsized":"2024-09-18T07:47:31.119105+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'30","last_scrub_stamp":"2024-09-18T07:45:18.084091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:45:18.084091+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":592,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.125099+0000","last_change":"2024-09-18T07:42:39.814649+0000","last_active":"2024-09-18T07:47:31.125099+0000","last_peered":"2024-09-18T07:47:31.125099+0000","last_clean":"2024-09-18T07:47:31.125099+0000","last_became_active":"2024-09-18T07:40:19.068296+0000","last_became_peered":"2024-09-18T07:40:19.068296+0000","last_unstale":"2024-09-18T07:47:31.125099+0000","last_undegraded":"2024-09-18T07:47:31.125099+0000","last_fullsized":"2024-09-18T07:47:31.125099+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:39.814582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:39.814582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:55.718130+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":777,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119058+0000","last_change":"2024-09-18T07:44:43.788033+0000","last_active":"2024-09-18T07:47:31.119058+0000","last_peered":"2024-09-18T07:47:31.119058+0000","last_clean":"2024-09-18T07:47:31.119058+0000","last_became_active":"2024-09-18T07:44:43.780927+0000","last_became_peered":"2024-09-18T07:44:43.780927+0000","last_unstale":"2024-09-18T07:47:31.119058+0000","last_undegraded":"2024-09-18T07:47:31.119058+0000","last_fullsized":"2024-09-18T07:47:31.119058+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:09.067941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:09.067941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":694,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119100+0000","last_change":"2024-09-18T07:46:07.931110+0000","last_active":"2024-09-18T07:47:31.119100+0000","last_peered":"2024-09-18T07:47:31.119100+0000","last_clean":"2024-09-18T07:47:31.119100+0000","last_became_active":"2024-09-18T07:44:43.785212+0000","last_became_peered":"2024-09-18T07:44:43.785212+0000","last_unstale":"2024-09-18T07:47:31.119100+0000","last_undegraded":"2024-09-18T07:47:31.119100+0000","last_fullsized":"2024-09-18T07:47:31.119100+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:07.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:07.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"469'1840","reported_seq":2343,"reported_epoch":469,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T07:47:31.431553+0000","last_change":"2024-09-18T07:47:30.669939+0000","last_active":"2024-09-18T07:47:31.431553+0000","last_peered":"2024-09-18T07:47:31.431553+0000","last_clean":"2024-09-18T07:47:19.257045+0000","last_became_active":"2024-09-18T07:47:21.191360+0000","last_became_peered":"2024-09-18T07:47:21.191360+0000","last_unstale":"2024-09-18T07:47:31.431553+0000","last_undegraded":"2024-09-18T07:47:31.431553+0000","last_fullsized":"2024-09-18T07:47:21.116805+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":1840,"log_dups_size":0,"ondisk_log_size":1840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120586240,"num_objects":1840,"num_object_clones":0,"num_object_copies":5520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":619,"num_objects_unfound":0,"num_objects_dirty":1840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1840,"num_write_kb":117760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":15335424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,7],"avail_no_missing":["4","0","7"],"object_location_counts":[{"shards":"0,4,7","objects":619},{"shards":"4,7","objects":1221}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":443,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119168+0000","last_change":"2024-09-18T07:47:15.465656+0000","last_active":"2024-09-18T07:47:31.119168+0000","last_peered":"2024-09-18T07:47:31.119168+0000","last_clean":"2024-09-18T07:47:31.119168+0000","last_became_active":"2024-09-18T07:47:11.196995+0000","last_became_peered":"2024-09-18T07:47:11.196995+0000","last_unstale":"2024-09-18T07:47:31.119168+0000","last_undegraded":"2024-09-18T07:47:31.119168+0000","last_fullsized":"2024-09-18T07:47:31.119168+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:47:15.465581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:47:15.465581+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:43.728207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"323'28","reported_seq":812,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119292+0000","last_change":"2024-09-18T07:47:11.188351+0000","last_active":"2024-09-18T07:47:31.119292+0000","last_peered":"2024-09-18T07:47:31.119292+0000","last_clean":"2024-09-18T07:47:31.119292+0000","last_became_active":"2024-09-18T07:47:11.188002+0000","last_became_peered":"2024-09-18T07:47:11.188002+0000","last_unstale":"2024-09-18T07:47:31.119292+0000","last_undegraded":"2024-09-18T07:47:31.119292+0000","last_fullsized":"2024-09-18T07:47:31.119292+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-18T07:40:56.183802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.183802+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:18.938445+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":885,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.129734+0000","last_change":"2024-09-18T07:47:20.151941+0000","last_active":"2024-09-18T07:47:21.129734+0000","last_peered":"2024-09-18T07:47:21.129734+0000","last_clean":"2024-09-18T07:47:21.129734+0000","last_became_active":"2024-09-18T07:47:20.138745+0000","last_became_peered":"2024-09-18T07:47:20.138745+0000","last_unstale":"2024-09-18T07:47:21.129734+0000","last_undegraded":"2024-09-18T07:47:21.129734+0000","last_fullsized":"2024-09-18T07:47:21.129734+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:44:44.266955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:44:44.266955+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"467'12","reported_seq":1092,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.118879+0000","last_change":"2024-09-18T07:47:20.201000+0000","last_active":"2024-09-18T07:47:31.118879+0000","last_peered":"2024-09-18T07:47:31.118879+0000","last_clean":"2024-09-18T07:47:31.118879+0000","last_became_active":"2024-09-18T07:47:20.142972+0000","last_became_peered":"2024-09-18T07:47:20.142972+0000","last_unstale":"2024-09-18T07:47:31.118879+0000","last_undegraded":"2024-09-18T07:47:31.118879+0000","last_fullsized":"2024-09-18T07:47:31.118879+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:42:04.428395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.428395+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:20.535138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":644,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.124875+0000","last_change":"2024-09-18T07:42:45.886132+0000","last_active":"2024-09-18T07:47:31.124875+0000","last_peered":"2024-09-18T07:47:31.124875+0000","last_clean":"2024-09-18T07:47:31.124875+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:47:31.124875+0000","last_undegraded":"2024-09-18T07:47:31.124875+0000","last_fullsized":"2024-09-18T07:47:31.124875+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:45.886069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:45.886069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:36.575612+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"469'1875","reported_seq":1906,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.211084+0000","last_change":"2024-09-18T07:47:10.178170+0000","last_active":"2024-09-18T07:47:31.211084+0000","last_peered":"2024-09-18T07:47:31.211084+0000","last_clean":"2024-09-18T07:47:31.211084+0000","last_became_active":"2024-09-18T07:47:10.177901+0000","last_became_peered":"2024-09-18T07:47:10.177901+0000","last_unstale":"2024-09-18T07:47:31.211084+0000","last_undegraded":"2024-09-18T07:47:31.211084+0000","last_fullsized":"2024-09-18T07:47:31.211084+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":1875,"log_dups_size":0,"ondisk_log_size":1875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:21.545718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122880000,"num_objects":1875,"num_object_clones":0,"num_object_copies":5625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1875,"num_write_kb":120000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":586,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123656+0000","last_change":"2024-09-18T07:40:38.066508+0000","last_active":"2024-09-18T07:47:31.123656+0000","last_peered":"2024-09-18T07:47:31.123656+0000","last_clean":"2024-09-18T07:47:31.123656+0000","last_became_active":"2024-09-18T07:39:04.941752+0000","last_became_peered":"2024-09-18T07:39:04.941752+0000","last_unstale":"2024-09-18T07:47:31.123656+0000","last_undegraded":"2024-09-18T07:47:31.123656+0000","last_fullsized":"2024-09-18T07:47:31.123656+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:38.066443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.066443+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:02.188770+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"323'20","reported_seq":517,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.112977+0000","last_change":"2024-09-18T07:47:20.202968+0000","last_active":"2024-09-18T07:47:21.112977+0000","last_peered":"2024-09-18T07:47:21.112977+0000","last_clean":"2024-09-18T07:47:21.112977+0000","last_became_active":"2024-09-18T07:47:20.144924+0000","last_became_peered":"2024-09-18T07:47:20.144924+0000","last_unstale":"2024-09-18T07:47:21.112977+0000","last_undegraded":"2024-09-18T07:47:21.112977+0000","last_fullsized":"2024-09-18T07:47:21.112977+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:44:46.501666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:46.501666+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":759,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.113338+0000","last_change":"2024-09-18T07:47:11.202895+0000","last_active":"2024-09-18T07:47:21.113338+0000","last_peered":"2024-09-18T07:47:21.113338+0000","last_clean":"2024-09-18T07:47:21.113338+0000","last_became_active":"2024-09-18T07:47:11.202526+0000","last_became_peered":"2024-09-18T07:47:11.202526+0000","last_unstale":"2024-09-18T07:47:21.113338+0000","last_undegraded":"2024-09-18T07:47:21.113338+0000","last_fullsized":"2024-09-18T07:47:21.113338+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:53.350019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:53.350019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:19.135777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"461'13","reported_seq":572,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123222+0000","last_change":"2024-09-18T07:47:14.059129+0000","last_active":"2024-09-18T07:47:31.123222+0000","last_peered":"2024-09-18T07:47:31.123222+0000","last_clean":"2024-09-18T07:47:31.123222+0000","last_became_active":"2024-09-18T07:47:11.189241+0000","last_became_peered":"2024-09-18T07:47:11.189241+0000","last_unstale":"2024-09-18T07:47:31.123222+0000","last_undegraded":"2024-09-18T07:47:31.123222+0000","last_fullsized":"2024-09-18T07:47:31.123222+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'13","last_scrub_stamp":"2024-09-18T07:47:14.059062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:14.059062+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:20.411246+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":840,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.116779+0000","last_change":"2024-09-18T07:44:43.252210+0000","last_active":"2024-09-18T07:47:21.116779+0000","last_peered":"2024-09-18T07:47:21.116779+0000","last_clean":"2024-09-18T07:47:21.116779+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:47:21.116779+0000","last_undegraded":"2024-09-18T07:47:21.116779+0000","last_fullsized":"2024-09-18T07:47:21.116779+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:43.252166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:44:43.252166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"469'1782","reported_seq":3420,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.237120+0000","last_change":"2024-09-18T07:47:31.131323+0000","last_active":"2024-09-18T07:47:31.237120+0000","last_peered":"2024-09-18T07:47:31.237120+0000","last_clean":"2024-09-18T07:47:31.237120+0000","last_became_active":"2024-09-18T07:47:31.128537+0000","last_became_peered":"2024-09-18T07:47:31.128537+0000","last_unstale":"2024-09-18T07:47:31.237120+0000","last_undegraded":"2024-09-18T07:47:31.237120+0000","last_fullsized":"2024-09-18T07:47:31.237120+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":1782,"log_dups_size":0,"ondisk_log_size":1782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:19.120573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116785152,"num_objects":1782,"num_object_clones":0,"num_object_copies":5346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1782,"num_write_kb":114048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":799,"num_bytes_recovered":52363264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":594,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123251+0000","last_change":"2024-09-18T07:45:50.787384+0000","last_active":"2024-09-18T07:47:31.123251+0000","last_peered":"2024-09-18T07:47:31.123251+0000","last_clean":"2024-09-18T07:47:31.123251+0000","last_became_active":"2024-09-18T07:39:04.948571+0000","last_became_peered":"2024-09-18T07:39:04.948571+0000","last_unstale":"2024-09-18T07:47:31.123251+0000","last_undegraded":"2024-09-18T07:47:31.123251+0000","last_fullsized":"2024-09-18T07:47:31.123251+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:45:50.787316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:45:50.787316+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"323'29","reported_seq":682,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.113369+0000","last_change":"2024-09-18T07:47:11.198450+0000","last_active":"2024-09-18T07:47:21.113369+0000","last_peered":"2024-09-18T07:47:21.113369+0000","last_clean":"2024-09-18T07:47:21.113369+0000","last_became_active":"2024-09-18T07:47:11.198126+0000","last_became_peered":"2024-09-18T07:47:11.198126+0000","last_unstale":"2024-09-18T07:47:21.113369+0000","last_undegraded":"2024-09-18T07:47:21.113369+0000","last_fullsized":"2024-09-18T07:47:21.113369+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'29","last_scrub_stamp":"2024-09-18T07:44:14.250974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:14.250974+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":589,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.126730+0000","last_change":"2024-09-18T07:47:20.140174+0000","last_active":"2024-09-18T07:47:31.126730+0000","last_peered":"2024-09-18T07:47:31.126730+0000","last_clean":"2024-09-18T07:47:31.126730+0000","last_became_active":"2024-09-18T07:47:20.139806+0000","last_became_peered":"2024-09-18T07:47:20.139806+0000","last_unstale":"2024-09-18T07:47:31.126730+0000","last_undegraded":"2024-09-18T07:47:31.126730+0000","last_fullsized":"2024-09-18T07:47:31.126730+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:43.985969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:43.985969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:08.438184+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":508,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.124176+0000","last_change":"2024-09-18T07:47:11.194476+0000","last_active":"2024-09-18T07:47:31.124176+0000","last_peered":"2024-09-18T07:47:31.124176+0000","last_clean":"2024-09-18T07:47:31.124176+0000","last_became_active":"2024-09-18T07:47:11.193892+0000","last_became_peered":"2024-09-18T07:47:11.193892+0000","last_unstale":"2024-09-18T07:47:31.124176+0000","last_undegraded":"2024-09-18T07:47:31.124176+0000","last_fullsized":"2024-09-18T07:47:31.124176+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:11.108021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:43:11.108021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":511,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123186+0000","last_change":"2024-09-18T07:47:20.138912+0000","last_active":"2024-09-18T07:47:31.123186+0000","last_peered":"2024-09-18T07:47:31.123186+0000","last_clean":"2024-09-18T07:47:31.123186+0000","last_became_active":"2024-09-18T07:47:20.138390+0000","last_became_peered":"2024-09-18T07:47:20.138390+0000","last_unstale":"2024-09-18T07:47:31.123186+0000","last_undegraded":"2024-09-18T07:47:31.123186+0000","last_fullsized":"2024-09-18T07:47:31.123186+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:49.018222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:49.018222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:52.717999+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.b","version":"468'1841","reported_seq":3573,"reported_epoch":468,"state":"peering","last_fresh":"2024-09-18T07:47:30.846485+0000","last_change":"2024-09-18T07:47:31.122305+0000","last_active":"2024-09-18T07:47:31.122305+0000","last_peered":"2024-09-18T07:47:30.846485+0000","last_clean":"2024-09-18T07:47:19.262561+0000","last_became_active":"2024-09-18T07:47:21.161740+0000","last_became_peered":"2024-09-18T07:47:21.161740+0000","last_unstale":"2024-09-18T07:47:30.846485+0000","last_undegraded":"2024-09-18T07:47:30.846485+0000","last_fullsized":"2024-09-18T07:47:21.117383+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":1841,"log_dups_size":0,"ondisk_log_size":1841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120651776,"num_objects":1841,"num_object_clones":0,"num_object_copies":5523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1841,"num_write_kb":117824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":849,"num_bytes_recovered":55640064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":315,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.112917+0000","last_change":"2024-09-18T07:47:12.191482+0000","last_active":"2024-09-18T07:47:21.112917+0000","last_peered":"2024-09-18T07:47:21.112917+0000","last_clean":"2024-09-18T07:47:21.112917+0000","last_became_active":"2024-09-18T07:47:11.195706+0000","last_became_peered":"2024-09-18T07:47:11.195706+0000","last_unstale":"2024-09-18T07:47:21.112917+0000","last_undegraded":"2024-09-18T07:47:21.112917+0000","last_fullsized":"2024-09-18T07:47:21.112917+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:47:12.191419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:47:12.191419+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:38.375897+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"467'43","reported_seq":855,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.126545+0000","last_change":"2024-09-18T07:47:20.189498+0000","last_active":"2024-09-18T07:47:31.126545+0000","last_peered":"2024-09-18T07:47:31.126545+0000","last_clean":"2024-09-18T07:47:31.126545+0000","last_became_active":"2024-09-18T07:47:20.143283+0000","last_became_peered":"2024-09-18T07:47:20.143283+0000","last_unstale":"2024-09-18T07:47:31.126545+0000","last_undegraded":"2024-09-18T07:47:31.126545+0000","last_fullsized":"2024-09-18T07:47:31.126545+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'41","last_scrub_stamp":"2024-09-18T07:46:12.858570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:12.858570+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":783,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119528+0000","last_change":"2024-09-18T07:44:43.780418+0000","last_active":"2024-09-18T07:47:31.119528+0000","last_peered":"2024-09-18T07:47:31.119528+0000","last_clean":"2024-09-18T07:47:31.119528+0000","last_became_active":"2024-09-18T07:44:43.780114+0000","last_became_peered":"2024-09-18T07:44:43.780114+0000","last_unstale":"2024-09-18T07:47:31.119528+0000","last_undegraded":"2024-09-18T07:47:31.119528+0000","last_fullsized":"2024-09-18T07:47:31.119528+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:03.028558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:03.028558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":751,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.113045+0000","last_change":"2024-09-18T07:47:20.190556+0000","last_active":"2024-09-18T07:47:21.113045+0000","last_peered":"2024-09-18T07:47:21.113045+0000","last_clean":"2024-09-18T07:47:21.113045+0000","last_became_active":"2024-09-18T07:47:20.144312+0000","last_became_peered":"2024-09-18T07:47:20.144312+0000","last_unstale":"2024-09-18T07:47:21.113045+0000","last_undegraded":"2024-09-18T07:47:21.113045+0000","last_fullsized":"2024-09-18T07:47:21.113045+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:44:06.489166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:44:06.489166+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":868,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.116428+0000","last_change":"2024-09-18T07:47:20.131135+0000","last_active":"2024-09-18T07:47:21.116428+0000","last_peered":"2024-09-18T07:47:21.116428+0000","last_clean":"2024-09-18T07:47:21.116428+0000","last_became_active":"2024-09-18T07:47:20.130809+0000","last_became_peered":"2024-09-18T07:47:20.130809+0000","last_unstale":"2024-09-18T07:47:21.116428+0000","last_undegraded":"2024-09-18T07:47:21.116428+0000","last_fullsized":"2024-09-18T07:47:21.116428+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:46.629196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:46.629196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:15.302326+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":886,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.120150+0000","last_change":"2024-09-18T07:45:52.882759+0000","last_active":"2024-09-18T07:47:31.120150+0000","last_peered":"2024-09-18T07:47:31.120150+0000","last_clean":"2024-09-18T07:47:31.120150+0000","last_became_active":"2024-09-18T07:44:43.780294+0000","last_became_peered":"2024-09-18T07:44:43.780294+0000","last_unstale":"2024-09-18T07:47:31.120150+0000","last_undegraded":"2024-09-18T07:47:31.120150+0000","last_fullsized":"2024-09-18T07:47:31.120150+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:52.882700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:45:52.882700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"469'1890","reported_seq":1920,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.248452+0000","last_change":"2024-09-18T07:47:11.190416+0000","last_active":"2024-09-18T07:47:31.248452+0000","last_peered":"2024-09-18T07:47:31.248452+0000","last_clean":"2024-09-18T07:47:31.248452+0000","last_became_active":"2024-09-18T07:47:11.190117+0000","last_became_peered":"2024-09-18T07:47:11.190117+0000","last_unstale":"2024-09-18T07:47:31.248452+0000","last_undegraded":"2024-09-18T07:47:31.248452+0000","last_fullsized":"2024-09-18T07:47:31.248452+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":1890,"log_dups_size":0,"ondisk_log_size":1890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:15.924044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123863040,"num_objects":1890,"num_object_clones":0,"num_object_copies":5670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1890,"num_write_kb":120960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":705,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123618+0000","last_change":"2024-09-18T07:45:24.617065+0000","last_active":"2024-09-18T07:47:31.123618+0000","last_peered":"2024-09-18T07:47:31.123618+0000","last_clean":"2024-09-18T07:47:31.123618+0000","last_became_active":"2024-09-18T07:44:43.783295+0000","last_became_peered":"2024-09-18T07:44:43.783295+0000","last_unstale":"2024-09-18T07:47:31.123618+0000","last_undegraded":"2024-09-18T07:47:31.123618+0000","last_fullsized":"2024-09-18T07:47:31.123618+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:45:24.616992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:45:24.616992+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"323'17","reported_seq":543,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.114062+0000","last_change":"2024-09-18T07:47:20.202629+0000","last_active":"2024-09-18T07:47:21.114062+0000","last_peered":"2024-09-18T07:47:21.114062+0000","last_clean":"2024-09-18T07:47:21.114062+0000","last_became_active":"2024-09-18T07:47:20.144799+0000","last_became_peered":"2024-09-18T07:47:20.144799+0000","last_unstale":"2024-09-18T07:47:21.114062+0000","last_undegraded":"2024-09-18T07:47:21.114062+0000","last_fullsized":"2024-09-18T07:47:21.114062+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'17","last_scrub_stamp":"2024-09-18T07:43:40.533240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:43:40.533240+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":23,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":602,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.120855+0000","last_change":"2024-09-18T07:47:20.140678+0000","last_active":"2024-09-18T07:47:31.120855+0000","last_peered":"2024-09-18T07:47:31.120855+0000","last_clean":"2024-09-18T07:47:31.120855+0000","last_became_active":"2024-09-18T07:47:20.140376+0000","last_became_peered":"2024-09-18T07:47:20.140376+0000","last_unstale":"2024-09-18T07:47:31.120855+0000","last_undegraded":"2024-09-18T07:47:31.120855+0000","last_fullsized":"2024-09-18T07:47:31.120855+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:52.917296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:52.917296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:20.683925+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":892,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.121364+0000","last_change":"2024-09-18T07:47:07.841463+0000","last_active":"2024-09-18T07:47:31.121364+0000","last_peered":"2024-09-18T07:47:31.121364+0000","last_clean":"2024-09-18T07:47:31.121364+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:47:31.121364+0000","last_undegraded":"2024-09-18T07:47:31.121364+0000","last_fullsized":"2024-09-18T07:47:31.121364+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:07.841406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:47:07.841406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:36.552524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":892,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.125704+0000","last_change":"2024-09-18T07:47:16.834107+0000","last_active":"2024-09-18T07:47:31.125704+0000","last_peered":"2024-09-18T07:47:31.125704+0000","last_clean":"2024-09-18T07:47:31.125704+0000","last_became_active":"2024-09-18T07:47:11.201379+0000","last_became_peered":"2024-09-18T07:47:11.201379+0000","last_unstale":"2024-09-18T07:47:31.125704+0000","last_undegraded":"2024-09-18T07:47:31.125704+0000","last_fullsized":"2024-09-18T07:47:31.125704+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:16.834043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:16.834043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:46.448197+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":640,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.124406+0000","last_change":"2024-09-18T07:40:54.574254+0000","last_active":"2024-09-18T07:47:31.124406+0000","last_peered":"2024-09-18T07:47:31.124406+0000","last_clean":"2024-09-18T07:47:31.124406+0000","last_became_active":"2024-09-18T07:37:43.060019+0000","last_became_peered":"2024-09-18T07:37:43.060019+0000","last_unstale":"2024-09-18T07:47:31.124406+0000","last_undegraded":"2024-09-18T07:47:31.124406+0000","last_fullsized":"2024-09-18T07:47:31.124406+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:40:54.574077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:54.574077+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:35.620297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"469'1787","reported_seq":1818,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.212836+0000","last_change":"2024-09-18T07:47:10.191710+0000","last_active":"2024-09-18T07:47:31.212836+0000","last_peered":"2024-09-18T07:47:31.212836+0000","last_clean":"2024-09-18T07:47:31.212836+0000","last_became_active":"2024-09-18T07:47:10.191380+0000","last_became_peered":"2024-09-18T07:47:10.191380+0000","last_unstale":"2024-09-18T07:47:31.212836+0000","last_undegraded":"2024-09-18T07:47:31.212836+0000","last_fullsized":"2024-09-18T07:47:31.212836+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":1787,"log_dups_size":0,"ondisk_log_size":1787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:26.431119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117112832,"num_objects":1787,"num_object_clones":0,"num_object_copies":5361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1787,"num_write_kb":114368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":625,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.122986+0000","last_change":"2024-09-18T07:47:20.138473+0000","last_active":"2024-09-18T07:47:31.122986+0000","last_peered":"2024-09-18T07:47:31.122986+0000","last_clean":"2024-09-18T07:47:31.122986+0000","last_became_active":"2024-09-18T07:47:20.137990+0000","last_became_peered":"2024-09-18T07:47:20.137990+0000","last_unstale":"2024-09-18T07:47:31.122986+0000","last_undegraded":"2024-09-18T07:47:31.122986+0000","last_fullsized":"2024-09-18T07:47:31.122986+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:46:15.989579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:46:15.989579+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"323'24","reported_seq":942,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.129423+0000","last_change":"2024-09-18T07:47:20.159485+0000","last_active":"2024-09-18T07:47:21.129423+0000","last_peered":"2024-09-18T07:47:21.129423+0000","last_clean":"2024-09-18T07:47:21.129423+0000","last_became_active":"2024-09-18T07:47:20.138961+0000","last_became_peered":"2024-09-18T07:47:20.138961+0000","last_unstale":"2024-09-18T07:47:21.129423+0000","last_undegraded":"2024-09-18T07:47:21.129423+0000","last_fullsized":"2024-09-18T07:47:21.129423+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'24","last_scrub_stamp":"2024-09-18T07:44:06.211676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:06.211676+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":860,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.117083+0000","last_change":"2024-09-18T07:44:43.784965+0000","last_active":"2024-09-18T07:47:21.117083+0000","last_peered":"2024-09-18T07:47:21.117083+0000","last_clean":"2024-09-18T07:47:21.117083+0000","last_became_active":"2024-09-18T07:44:43.784621+0000","last_became_peered":"2024-09-18T07:44:43.784621+0000","last_unstale":"2024-09-18T07:47:21.117083+0000","last_undegraded":"2024-09-18T07:47:21.117083+0000","last_fullsized":"2024-09-18T07:47:21.117083+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:42:44.401546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:44.401546+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:38.653573+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":1397,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123577+0000","last_change":"2024-09-18T07:47:20.162839+0000","last_active":"2024-09-18T07:47:31.123577+0000","last_peered":"2024-09-18T07:47:31.123577+0000","last_clean":"2024-09-18T07:47:31.123577+0000","last_became_active":"2024-09-18T07:47:20.138901+0000","last_became_peered":"2024-09-18T07:47:20.138901+0000","last_unstale":"2024-09-18T07:47:31.123577+0000","last_undegraded":"2024-09-18T07:47:31.123577+0000","last_fullsized":"2024-09-18T07:47:31.123577+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T07:46:47.771151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:46:47.771151+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:54.068379+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":752,"num_read_kb":747,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":753,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.114302+0000","last_change":"2024-09-18T07:47:20.145165+0000","last_active":"2024-09-18T07:47:21.114302+0000","last_peered":"2024-09-18T07:47:21.114302+0000","last_clean":"2024-09-18T07:47:21.114302+0000","last_became_active":"2024-09-18T07:47:20.144413+0000","last_became_peered":"2024-09-18T07:47:20.144413+0000","last_unstale":"2024-09-18T07:47:21.114302+0000","last_undegraded":"2024-09-18T07:47:21.114302+0000","last_fullsized":"2024-09-18T07:47:21.114302+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:50.587498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:44:50.587498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":413,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.113454+0000","last_change":"2024-09-18T07:47:20.143704+0000","last_active":"2024-09-18T07:47:21.113454+0000","last_peered":"2024-09-18T07:47:21.113454+0000","last_clean":"2024-09-18T07:47:21.113454+0000","last_became_active":"2024-09-18T07:47:20.143383+0000","last_became_peered":"2024-09-18T07:47:20.143383+0000","last_unstale":"2024-09-18T07:47:21.113454+0000","last_undegraded":"2024-09-18T07:47:21.113454+0000","last_fullsized":"2024-09-18T07:47:21.113454+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:52.440332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:52.440332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:00.226264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.a","version":"469'1835","reported_seq":1865,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.240884+0000","last_change":"2024-09-18T07:47:11.194180+0000","last_active":"2024-09-18T07:47:31.240884+0000","last_peered":"2024-09-18T07:47:31.240884+0000","last_clean":"2024-09-18T07:47:31.240884+0000","last_became_active":"2024-09-18T07:47:11.193156+0000","last_became_peered":"2024-09-18T07:47:11.193156+0000","last_unstale":"2024-09-18T07:47:31.240884+0000","last_undegraded":"2024-09-18T07:47:31.240884+0000","last_fullsized":"2024-09-18T07:47:31.240884+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":1835,"log_dups_size":0,"ondisk_log_size":1835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:15.069852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120258560,"num_objects":1835,"num_object_clones":0,"num_object_copies":5505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1835,"num_write_kb":117440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"398'34","reported_seq":656,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123651+0000","last_change":"2024-09-18T07:44:44.523660+0000","last_active":"2024-09-18T07:47:31.123651+0000","last_peered":"2024-09-18T07:47:31.123651+0000","last_clean":"2024-09-18T07:47:31.123651+0000","last_became_active":"2024-09-18T07:44:43.786827+0000","last_became_peered":"2024-09-18T07:44:43.786827+0000","last_unstale":"2024-09-18T07:47:31.123651+0000","last_undegraded":"2024-09-18T07:47:31.123651+0000","last_fullsized":"2024-09-18T07:47:31.123651+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'33","last_scrub_stamp":"2024-09-18T07:44:44.523601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:44.523601+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":606,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.113306+0000","last_change":"2024-09-18T07:47:11.199888+0000","last_active":"2024-09-18T07:47:21.113306+0000","last_peered":"2024-09-18T07:47:21.113306+0000","last_clean":"2024-09-18T07:47:21.113306+0000","last_became_active":"2024-09-18T07:47:11.198809+0000","last_became_peered":"2024-09-18T07:47:11.198809+0000","last_unstale":"2024-09-18T07:47:21.113306+0000","last_undegraded":"2024-09-18T07:47:21.113306+0000","last_fullsized":"2024-09-18T07:47:21.113306+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:35.555785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:35.555785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":688,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123626+0000","last_change":"2024-09-18T07:45:25.666309+0000","last_active":"2024-09-18T07:47:31.123626+0000","last_peered":"2024-09-18T07:47:31.123626+0000","last_clean":"2024-09-18T07:47:31.123626+0000","last_became_active":"2024-09-18T07:39:03.350419+0000","last_became_peered":"2024-09-18T07:39:03.350419+0000","last_unstale":"2024-09-18T07:47:31.123626+0000","last_undegraded":"2024-09-18T07:47:31.123626+0000","last_fullsized":"2024-09-18T07:47:31.123626+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:45:25.666036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:45:25.666036+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":641,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.125002+0000","last_change":"2024-09-18T07:47:11.195667+0000","last_active":"2024-09-18T07:47:31.125002+0000","last_peered":"2024-09-18T07:47:31.125002+0000","last_clean":"2024-09-18T07:47:31.125002+0000","last_became_active":"2024-09-18T07:47:11.195084+0000","last_became_peered":"2024-09-18T07:47:11.195084+0000","last_unstale":"2024-09-18T07:47:31.125002+0000","last_undegraded":"2024-09-18T07:47:31.125002+0000","last_fullsized":"2024-09-18T07:47:31.125002+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.726445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.726445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:20.870125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":727,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.114184+0000","last_change":"2024-09-18T07:47:11.198595+0000","last_active":"2024-09-18T07:47:21.114184+0000","last_peered":"2024-09-18T07:47:21.114184+0000","last_clean":"2024-09-18T07:47:21.114184+0000","last_became_active":"2024-09-18T07:47:11.196928+0000","last_became_peered":"2024-09-18T07:47:11.196928+0000","last_unstale":"2024-09-18T07:47:21.114184+0000","last_undegraded":"2024-09-18T07:47:21.114184+0000","last_fullsized":"2024-09-18T07:47:21.114184+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:36.592215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:36.592215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"469'1834","reported_seq":1865,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.197449+0000","last_change":"2024-09-18T07:47:10.190697+0000","last_active":"2024-09-18T07:47:31.197449+0000","last_peered":"2024-09-18T07:47:31.197449+0000","last_clean":"2024-09-18T07:47:31.197449+0000","last_became_active":"2024-09-18T07:47:10.190402+0000","last_became_peered":"2024-09-18T07:47:10.190402+0000","last_unstale":"2024-09-18T07:47:31.197449+0000","last_undegraded":"2024-09-18T07:47:31.197449+0000","last_fullsized":"2024-09-18T07:47:31.197449+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":1834,"log_dups_size":0,"ondisk_log_size":1834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:27.449274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120193024,"num_objects":1834,"num_object_clones":0,"num_object_copies":5502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1834,"num_write_kb":117376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"323'19","reported_seq":944,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.120059+0000","last_change":"2024-09-18T07:47:20.196636+0000","last_active":"2024-09-18T07:47:31.120059+0000","last_peered":"2024-09-18T07:47:31.120059+0000","last_clean":"2024-09-18T07:47:31.120059+0000","last_became_active":"2024-09-18T07:47:20.137370+0000","last_became_peered":"2024-09-18T07:47:20.137370+0000","last_unstale":"2024-09-18T07:47:31.120059+0000","last_undegraded":"2024-09-18T07:47:31.120059+0000","last_fullsized":"2024-09-18T07:47:31.120059+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:42:43.402858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:43.402858+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":754,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119215+0000","last_change":"2024-09-18T07:44:43.793621+0000","last_active":"2024-09-18T07:47:31.119215+0000","last_peered":"2024-09-18T07:47:31.119215+0000","last_clean":"2024-09-18T07:47:31.119215+0000","last_became_active":"2024-09-18T07:44:43.790160+0000","last_became_peered":"2024-09-18T07:44:43.790160+0000","last_unstale":"2024-09-18T07:47:31.119215+0000","last_undegraded":"2024-09-18T07:47:31.119215+0000","last_fullsized":"2024-09-18T07:47:31.119215+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.078600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.078600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:57.964570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":762,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.113819+0000","last_change":"2024-09-18T07:47:11.202169+0000","last_active":"2024-09-18T07:47:21.113819+0000","last_peered":"2024-09-18T07:47:21.113819+0000","last_clean":"2024-09-18T07:47:21.113819+0000","last_became_active":"2024-09-18T07:47:11.198103+0000","last_became_peered":"2024-09-18T07:47:11.198103+0000","last_unstale":"2024-09-18T07:47:21.113819+0000","last_undegraded":"2024-09-18T07:47:21.113819+0000","last_fullsized":"2024-09-18T07:47:21.113819+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:45.479624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:44:45.479624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":741,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123048+0000","last_change":"2024-09-18T07:47:18.117749+0000","last_active":"2024-09-18T07:47:31.123048+0000","last_peered":"2024-09-18T07:47:31.123048+0000","last_clean":"2024-09-18T07:47:31.123048+0000","last_became_active":"2024-09-18T07:47:11.189605+0000","last_became_peered":"2024-09-18T07:47:11.189605+0000","last_unstale":"2024-09-18T07:47:31.123048+0000","last_undegraded":"2024-09-18T07:47:31.123048+0000","last_fullsized":"2024-09-18T07:47:31.123048+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:18.117581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:18.117581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:26.308182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"469'1750","reported_seq":1781,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.238615+0000","last_change":"2024-09-18T07:47:10.192621+0000","last_active":"2024-09-18T07:47:31.238615+0000","last_peered":"2024-09-18T07:47:31.238615+0000","last_clean":"2024-09-18T07:47:31.238615+0000","last_became_active":"2024-09-18T07:47:10.192273+0000","last_became_peered":"2024-09-18T07:47:10.192273+0000","last_unstale":"2024-09-18T07:47:31.238615+0000","last_undegraded":"2024-09-18T07:47:31.238615+0000","last_fullsized":"2024-09-18T07:47:31.238615+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":1750,"log_dups_size":0,"ondisk_log_size":1750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:29.305991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114688000,"num_objects":1750,"num_object_clones":0,"num_object_copies":5250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1750,"num_write_kb":112000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":269,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.112794+0000","last_change":"2024-09-18T07:47:11.204479+0000","last_active":"2024-09-18T07:47:21.112794+0000","last_peered":"2024-09-18T07:47:21.112794+0000","last_clean":"2024-09-18T07:47:21.112794+0000","last_became_active":"2024-09-18T07:47:11.195101+0000","last_became_peered":"2024-09-18T07:47:11.195101+0000","last_unstale":"2024-09-18T07:47:21.112794+0000","last_undegraded":"2024-09-18T07:47:21.112794+0000","last_fullsized":"2024-09-18T07:47:21.112794+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:47:11.204436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:47:11.204436+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:34.065822+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"323'31","reported_seq":828,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119981+0000","last_change":"2024-09-18T07:47:20.154723+0000","last_active":"2024-09-18T07:47:31.119981+0000","last_peered":"2024-09-18T07:47:31.119981+0000","last_clean":"2024-09-18T07:47:31.119981+0000","last_became_active":"2024-09-18T07:47:20.132912+0000","last_became_peered":"2024-09-18T07:47:20.132912+0000","last_unstale":"2024-09-18T07:47:31.119981+0000","last_undegraded":"2024-09-18T07:47:31.119981+0000","last_fullsized":"2024-09-18T07:47:31.119981+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:40:04.281758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:04.281758+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:20.957420+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":530,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.124289+0000","last_change":"2024-09-18T07:47:20.141372+0000","last_active":"2024-09-18T07:47:31.124289+0000","last_peered":"2024-09-18T07:47:31.124289+0000","last_clean":"2024-09-18T07:47:31.124289+0000","last_became_active":"2024-09-18T07:47:20.141038+0000","last_became_peered":"2024-09-18T07:47:20.141038+0000","last_unstale":"2024-09-18T07:47:31.124289+0000","last_undegraded":"2024-09-18T07:47:31.124289+0000","last_fullsized":"2024-09-18T07:47:31.124289+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:47.023220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:47.023220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:05.811084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":425,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119577+0000","last_change":"2024-09-18T07:47:20.131352+0000","last_active":"2024-09-18T07:47:31.119577+0000","last_peered":"2024-09-18T07:47:31.119577+0000","last_clean":"2024-09-18T07:47:31.119577+0000","last_became_active":"2024-09-18T07:47:20.130994+0000","last_became_peered":"2024-09-18T07:47:20.130994+0000","last_unstale":"2024-09-18T07:47:31.119577+0000","last_undegraded":"2024-09-18T07:47:31.119577+0000","last_fullsized":"2024-09-18T07:47:31.119577+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:13.019711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:13.019711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":890,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.125971+0000","last_change":"2024-09-18T07:47:06.889207+0000","last_active":"2024-09-18T07:47:31.125971+0000","last_peered":"2024-09-18T07:47:31.125971+0000","last_clean":"2024-09-18T07:47:31.125971+0000","last_became_active":"2024-09-18T07:44:43.785056+0000","last_became_peered":"2024-09-18T07:44:43.785056+0000","last_unstale":"2024-09-18T07:47:31.125971+0000","last_undegraded":"2024-09-18T07:47:31.125971+0000","last_fullsized":"2024-09-18T07:47:31.125971+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:06.889139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:06.889139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:16.178878+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"469'1790","reported_seq":1830,"reported_epoch":469,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T07:47:31.224457+0000","last_change":"2024-09-18T07:47:21.228811+0000","last_active":"2024-09-18T07:47:31.224457+0000","last_peered":"2024-09-18T07:47:31.224457+0000","last_clean":"2024-09-18T07:47:19.256977+0000","last_became_active":"2024-09-18T07:47:21.226764+0000","last_became_peered":"2024-09-18T07:47:21.226764+0000","last_unstale":"2024-09-18T07:47:31.224457+0000","last_undegraded":"2024-09-18T07:47:21.212212+0000","last_fullsized":"2024-09-18T07:47:21.114379+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":1790,"log_dups_size":0,"ondisk_log_size":1790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117309440,"num_objects":1790,"num_object_clones":0,"num_object_copies":5370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":845,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1790,"num_write_kb":114560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7],"avail_no_missing":["6","5","7"],"object_location_counts":[{"shards":"5,6,7","objects":845},{"shards":"6,7","objects":945}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":693,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.122938+0000","last_change":"2024-09-18T07:46:24.585165+0000","last_active":"2024-09-18T07:47:31.122938+0000","last_peered":"2024-09-18T07:47:31.122938+0000","last_clean":"2024-09-18T07:47:31.122938+0000","last_became_active":"2024-09-18T07:39:04.944753+0000","last_became_peered":"2024-09-18T07:39:04.944753+0000","last_unstale":"2024-09-18T07:47:31.122938+0000","last_undegraded":"2024-09-18T07:47:31.122938+0000","last_fullsized":"2024-09-18T07:47:31.122938+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:46:24.585120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:46:24.585120+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:41.374983+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"323'20","reported_seq":789,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119690+0000","last_change":"2024-09-18T07:46:14.047994+0000","last_active":"2024-09-18T07:47:31.119690+0000","last_peered":"2024-09-18T07:47:31.119690+0000","last_clean":"2024-09-18T07:47:31.119690+0000","last_became_active":"2024-09-18T07:44:43.784244+0000","last_became_peered":"2024-09-18T07:44:43.784244+0000","last_unstale":"2024-09-18T07:47:31.119690+0000","last_undegraded":"2024-09-18T07:47:31.119690+0000","last_fullsized":"2024-09-18T07:47:31.119690+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:46:14.047907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:14.047907+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":271,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119289+0000","last_change":"2024-09-18T07:46:29.320743+0000","last_active":"2024-09-18T07:47:31.119289+0000","last_peered":"2024-09-18T07:47:31.119289+0000","last_clean":"2024-09-18T07:47:31.119289+0000","last_became_active":"2024-09-18T07:46:29.320499+0000","last_became_peered":"2024-09-18T07:46:29.320499+0000","last_unstale":"2024-09-18T07:47:31.119289+0000","last_undegraded":"2024-09-18T07:47:31.119289+0000","last_fullsized":"2024-09-18T07:47:31.119289+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:47.799342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.799342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:36.100670+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"461'13","reported_seq":1112,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.122142+0000","last_change":"2024-09-18T07:47:15.865446+0000","last_active":"2024-09-18T07:47:31.122142+0000","last_peered":"2024-09-18T07:47:31.122142+0000","last_clean":"2024-09-18T07:47:31.122142+0000","last_became_active":"2024-09-18T07:47:11.203256+0000","last_became_peered":"2024-09-18T07:47:11.203256+0000","last_unstale":"2024-09-18T07:47:31.122142+0000","last_undegraded":"2024-09-18T07:47:31.122142+0000","last_fullsized":"2024-09-18T07:47:31.122142+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'13","last_scrub_stamp":"2024-09-18T07:47:15.865373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:15.865373+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:21.101204+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":763,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.120371+0000","last_change":"2024-09-18T07:47:20.132483+0000","last_active":"2024-09-18T07:47:31.120371+0000","last_peered":"2024-09-18T07:47:31.120371+0000","last_clean":"2024-09-18T07:47:31.120371+0000","last_became_active":"2024-09-18T07:47:20.132179+0000","last_became_peered":"2024-09-18T07:47:20.132179+0000","last_unstale":"2024-09-18T07:47:31.120371+0000","last_undegraded":"2024-09-18T07:47:31.120371+0000","last_fullsized":"2024-09-18T07:47:31.120371+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:48.207484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:48.207484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:03.286685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"469'1817","reported_seq":1848,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.217579+0000","last_change":"2024-09-18T07:47:10.191967+0000","last_active":"2024-09-18T07:47:31.217579+0000","last_peered":"2024-09-18T07:47:31.217579+0000","last_clean":"2024-09-18T07:47:31.217579+0000","last_became_active":"2024-09-18T07:47:10.190921+0000","last_became_peered":"2024-09-18T07:47:10.190921+0000","last_unstale":"2024-09-18T07:47:31.217579+0000","last_undegraded":"2024-09-18T07:47:31.217579+0000","last_fullsized":"2024-09-18T07:47:31.217579+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":1817,"log_dups_size":0,"ondisk_log_size":1817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:19.287004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119078912,"num_objects":1817,"num_object_clones":0,"num_object_copies":5451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1817,"num_write_kb":116288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"323'15","reported_seq":938,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.116510+0000","last_change":"2024-09-18T07:47:20.156876+0000","last_active":"2024-09-18T07:47:21.116510+0000","last_peered":"2024-09-18T07:47:21.116510+0000","last_clean":"2024-09-18T07:47:21.116510+0000","last_became_active":"2024-09-18T07:47:20.138834+0000","last_became_peered":"2024-09-18T07:47:20.138834+0000","last_unstale":"2024-09-18T07:47:21.116510+0000","last_undegraded":"2024-09-18T07:47:21.116510+0000","last_fullsized":"2024-09-18T07:47:21.116510+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:21.946066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:21.946066+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:21.009046+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":781,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.113533+0000","last_change":"2024-09-18T07:47:11.195107+0000","last_active":"2024-09-18T07:47:21.113533+0000","last_peered":"2024-09-18T07:47:21.113533+0000","last_clean":"2024-09-18T07:47:21.113533+0000","last_became_active":"2024-09-18T07:47:11.194699+0000","last_became_peered":"2024-09-18T07:47:11.194699+0000","last_unstale":"2024-09-18T07:47:21.113533+0000","last_undegraded":"2024-09-18T07:47:21.113533+0000","last_fullsized":"2024-09-18T07:47:21.113533+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:04.518676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:43:04.518676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":870,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.121606+0000","last_change":"2024-09-18T07:46:08.854995+0000","last_active":"2024-09-18T07:47:31.121606+0000","last_peered":"2024-09-18T07:47:31.121606+0000","last_clean":"2024-09-18T07:47:31.121606+0000","last_became_active":"2024-09-18T07:44:43.784507+0000","last_became_peered":"2024-09-18T07:44:43.784507+0000","last_unstale":"2024-09-18T07:47:31.121606+0000","last_undegraded":"2024-09-18T07:47:31.121606+0000","last_fullsized":"2024-09-18T07:47:31.121606+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:08.854920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:08.854920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:19.894319+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":306,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123258+0000","last_change":"2024-09-18T07:46:29.323007+0000","last_active":"2024-09-18T07:47:31.123258+0000","last_peered":"2024-09-18T07:47:31.123258+0000","last_clean":"2024-09-18T07:47:31.123258+0000","last_became_active":"2024-09-18T07:46:29.322779+0000","last_became_peered":"2024-09-18T07:46:29.322779+0000","last_unstale":"2024-09-18T07:47:31.123258+0000","last_undegraded":"2024-09-18T07:47:31.123258+0000","last_fullsized":"2024-09-18T07:47:31.123258+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:34.257746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:34.257746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:05.673685+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"469'1818","reported_seq":1848,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.220739+0000","last_change":"2024-09-18T07:47:11.201597+0000","last_active":"2024-09-18T07:47:31.220739+0000","last_peered":"2024-09-18T07:47:31.220739+0000","last_clean":"2024-09-18T07:47:31.220739+0000","last_became_active":"2024-09-18T07:47:11.201275+0000","last_became_peered":"2024-09-18T07:47:11.201275+0000","last_unstale":"2024-09-18T07:47:31.220739+0000","last_undegraded":"2024-09-18T07:47:31.220739+0000","last_fullsized":"2024-09-18T07:47:31.220739+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":1818,"log_dups_size":0,"ondisk_log_size":1818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:36.150226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119144448,"num_objects":1818,"num_object_clones":0,"num_object_copies":5454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1818,"num_write_kb":116352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"323'35","reported_seq":600,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.124508+0000","last_change":"2024-09-18T07:47:20.166667+0000","last_active":"2024-09-18T07:47:31.124508+0000","last_peered":"2024-09-18T07:47:31.124508+0000","last_clean":"2024-09-18T07:47:31.124508+0000","last_became_active":"2024-09-18T07:47:20.141942+0000","last_became_peered":"2024-09-18T07:47:20.141942+0000","last_unstale":"2024-09-18T07:47:31.124508+0000","last_undegraded":"2024-09-18T07:47:31.124508+0000","last_fullsized":"2024-09-18T07:47:31.124508+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'35","last_scrub_stamp":"2024-09-18T07:44:41.447137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:41.447137+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":662,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.124581+0000","last_change":"2024-09-18T07:47:17.794908+0000","last_active":"2024-09-18T07:47:31.124581+0000","last_peered":"2024-09-18T07:47:31.124581+0000","last_clean":"2024-09-18T07:47:31.124581+0000","last_became_active":"2024-09-18T07:47:11.194873+0000","last_became_peered":"2024-09-18T07:47:11.194873+0000","last_unstale":"2024-09-18T07:47:31.124581+0000","last_undegraded":"2024-09-18T07:47:31.124581+0000","last_fullsized":"2024-09-18T07:47:31.124581+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:17.794850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:17.794850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:37.381809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":885,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.121724+0000","last_change":"2024-09-18T07:47:11.202717+0000","last_active":"2024-09-18T07:47:31.121724+0000","last_peered":"2024-09-18T07:47:31.121724+0000","last_clean":"2024-09-18T07:47:31.121724+0000","last_became_active":"2024-09-18T07:47:11.202437+0000","last_became_peered":"2024-09-18T07:47:11.202437+0000","last_unstale":"2024-09-18T07:47:31.121724+0000","last_undegraded":"2024-09-18T07:47:31.121724+0000","last_fullsized":"2024-09-18T07:47:31.121724+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:23.985496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:45:23.985496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:18.898670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":629,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123716+0000","last_change":"2024-09-18T07:43:39.239989+0000","last_active":"2024-09-18T07:47:31.123716+0000","last_peered":"2024-09-18T07:47:31.123716+0000","last_clean":"2024-09-18T07:47:31.123716+0000","last_became_active":"2024-09-18T07:35:01.406980+0000","last_became_peered":"2024-09-18T07:35:01.406980+0000","last_unstale":"2024-09-18T07:47:31.123716+0000","last_undegraded":"2024-09-18T07:47:31.123716+0000","last_fullsized":"2024-09-18T07:47:31.123716+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:39.239932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:39.239932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"467'1522","reported_seq":1552,"reported_epoch":467,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T07:47:28.154411+0000","last_change":"2024-09-18T07:47:21.194096+0000","last_active":"2024-09-18T07:47:28.154411+0000","last_peered":"2024-09-18T07:47:28.154411+0000","last_clean":"2024-09-18T07:47:19.254009+0000","last_became_active":"2024-09-18T07:47:21.191281+0000","last_became_peered":"2024-09-18T07:47:21.191281+0000","last_unstale":"2024-09-18T07:47:28.154411+0000","last_undegraded":"2024-09-18T07:47:21.128615+0000","last_fullsized":"2024-09-18T07:47:21.115839+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":1522,"log_dups_size":0,"ondisk_log_size":1522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99745792,"num_objects":1522,"num_object_clones":0,"num_object_copies":4566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1522,"num_write_kb":97408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2],"avail_no_missing":["0","2","4"],"object_location_counts":[{"shards":"0,2","objects":691},{"shards":"0,2,4","objects":831}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"323'18","reported_seq":848,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119130+0000","last_change":"2024-09-18T07:47:20.151153+0000","last_active":"2024-09-18T07:47:31.119130+0000","last_peered":"2024-09-18T07:47:31.119130+0000","last_clean":"2024-09-18T07:47:31.119130+0000","last_became_active":"2024-09-18T07:47:20.132026+0000","last_became_peered":"2024-09-18T07:47:20.132026+0000","last_unstale":"2024-09-18T07:47:31.119130+0000","last_undegraded":"2024-09-18T07:47:31.119130+0000","last_fullsized":"2024-09-18T07:47:31.119130+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:46:25.012172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:25.012172+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:49.030383+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":654,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.124909+0000","last_change":"2024-09-18T07:47:20.139266+0000","last_active":"2024-09-18T07:47:31.124909+0000","last_peered":"2024-09-18T07:47:31.124909+0000","last_clean":"2024-09-18T07:47:31.124909+0000","last_became_active":"2024-09-18T07:47:20.138910+0000","last_became_peered":"2024-09-18T07:47:20.138910+0000","last_unstale":"2024-09-18T07:47:31.124909+0000","last_undegraded":"2024-09-18T07:47:31.124909+0000","last_fullsized":"2024-09-18T07:47:31.124909+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:51.752578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:51.752578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:20.078827+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":902,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.120766+0000","last_change":"2024-09-18T07:47:20.140408+0000","last_active":"2024-09-18T07:47:31.120766+0000","last_peered":"2024-09-18T07:47:31.120766+0000","last_clean":"2024-09-18T07:47:31.120766+0000","last_became_active":"2024-09-18T07:47:20.139842+0000","last_became_peered":"2024-09-18T07:47:20.139842+0000","last_unstale":"2024-09-18T07:47:31.120766+0000","last_undegraded":"2024-09-18T07:47:31.120766+0000","last_fullsized":"2024-09-18T07:47:31.120766+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:13.830958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:13.830958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:14.510479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":881,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119132+0000","last_change":"2024-09-18T07:47:20.141687+0000","last_active":"2024-09-18T07:47:31.119132+0000","last_peered":"2024-09-18T07:47:31.119132+0000","last_clean":"2024-09-18T07:47:31.119132+0000","last_became_active":"2024-09-18T07:47:20.141391+0000","last_became_peered":"2024-09-18T07:47:20.141391+0000","last_unstale":"2024-09-18T07:47:31.119132+0000","last_undegraded":"2024-09-18T07:47:31.119132+0000","last_fullsized":"2024-09-18T07:47:31.119132+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:09.113374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:45:09.113374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"467'1707","reported_seq":1735,"reported_epoch":467,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T07:47:29.170004+0000","last_change":"2024-09-18T07:47:21.171645+0000","last_active":"2024-09-18T07:47:29.170004+0000","last_peered":"2024-09-18T07:47:29.170004+0000","last_clean":"2024-09-18T07:47:19.257298+0000","last_became_active":"2024-09-18T07:47:21.169901+0000","last_became_peered":"2024-09-18T07:47:21.169901+0000","last_unstale":"2024-09-18T07:47:29.170004+0000","last_undegraded":"2024-09-18T07:47:21.119219+0000","last_fullsized":"2024-09-18T07:47:21.113807+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":1707,"log_dups_size":0,"ondisk_log_size":1707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111869952,"num_objects":1707,"num_object_clones":0,"num_object_copies":5121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1707,"num_write_kb":109248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,4],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":858},{"shards":"4,5","objects":849}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"323'27","reported_seq":846,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.112677+0000","last_change":"2024-09-18T07:47:11.197814+0000","last_active":"2024-09-18T07:47:21.112677+0000","last_peered":"2024-09-18T07:47:21.112677+0000","last_clean":"2024-09-18T07:47:21.112677+0000","last_became_active":"2024-09-18T07:47:11.196288+0000","last_became_peered":"2024-09-18T07:47:11.196288+0000","last_unstale":"2024-09-18T07:47:21.112677+0000","last_undegraded":"2024-09-18T07:47:21.112677+0000","last_fullsized":"2024-09-18T07:47:21.112677+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:44:04.486867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:04.486867+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":654,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.125217+0000","last_change":"2024-09-18T07:45:50.588718+0000","last_active":"2024-09-18T07:47:31.125217+0000","last_peered":"2024-09-18T07:47:31.125217+0000","last_clean":"2024-09-18T07:47:31.125217+0000","last_became_active":"2024-09-18T07:44:43.778284+0000","last_became_peered":"2024-09-18T07:44:43.778284+0000","last_unstale":"2024-09-18T07:47:31.125217+0000","last_undegraded":"2024-09-18T07:47:31.125217+0000","last_fullsized":"2024-09-18T07:47:31.125217+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:45:50.588657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:45:50.588657+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"398'14","reported_seq":827,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123550+0000","last_change":"2024-09-18T07:46:21.582783+0000","last_active":"2024-09-18T07:47:31.123550+0000","last_peered":"2024-09-18T07:47:31.123550+0000","last_clean":"2024-09-18T07:47:31.123550+0000","last_became_active":"2024-09-18T07:44:43.782355+0000","last_became_peered":"2024-09-18T07:44:43.782355+0000","last_unstale":"2024-09-18T07:47:31.123550+0000","last_undegraded":"2024-09-18T07:47:31.123550+0000","last_fullsized":"2024-09-18T07:47:31.123550+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:46:21.582737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:21.582737+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:48.610505+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":713,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.112928+0000","last_change":"2024-09-18T07:47:11.200863+0000","last_active":"2024-09-18T07:47:21.112928+0000","last_peered":"2024-09-18T07:47:21.112928+0000","last_clean":"2024-09-18T07:47:21.112928+0000","last_became_active":"2024-09-18T07:47:11.197583+0000","last_became_peered":"2024-09-18T07:47:11.197583+0000","last_unstale":"2024-09-18T07:47:21.112928+0000","last_undegraded":"2024-09-18T07:47:21.112928+0000","last_fullsized":"2024-09-18T07:47:21.112928+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:07.547352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:07.547352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":500,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.124050+0000","last_change":"2024-09-18T07:47:11.188646+0000","last_active":"2024-09-18T07:47:31.124050+0000","last_peered":"2024-09-18T07:47:31.124050+0000","last_clean":"2024-09-18T07:47:31.124050+0000","last_became_active":"2024-09-18T07:47:11.188332+0000","last_became_peered":"2024-09-18T07:47:11.188332+0000","last_unstale":"2024-09-18T07:47:31.124050+0000","last_undegraded":"2024-09-18T07:47:31.124050+0000","last_fullsized":"2024-09-18T07:47:31.124050+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:08.181420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:08.181420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"323'12","reported_seq":938,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.120927+0000","last_change":"2024-09-18T07:47:20.188023+0000","last_active":"2024-09-18T07:47:31.120927+0000","last_peered":"2024-09-18T07:47:31.120927+0000","last_clean":"2024-09-18T07:47:31.120927+0000","last_became_active":"2024-09-18T07:47:20.142056+0000","last_became_peered":"2024-09-18T07:47:20.142056+0000","last_unstale":"2024-09-18T07:47:31.120927+0000","last_undegraded":"2024-09-18T07:47:31.120927+0000","last_fullsized":"2024-09-18T07:47:31.120927+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'12","last_scrub_stamp":"2024-09-18T07:45:41.914239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:45:41.914239+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":784,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.113787+0000","last_change":"2024-09-18T07:47:20.141214+0000","last_active":"2024-09-18T07:47:21.113787+0000","last_peered":"2024-09-18T07:47:21.113787+0000","last_clean":"2024-09-18T07:47:21.113787+0000","last_became_active":"2024-09-18T07:47:20.140847+0000","last_became_peered":"2024-09-18T07:47:20.140847+0000","last_unstale":"2024-09-18T07:47:21.113787+0000","last_undegraded":"2024-09-18T07:47:21.113787+0000","last_fullsized":"2024-09-18T07:47:21.113787+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:37.515124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:43:37.515124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":646,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.124461+0000","last_change":"2024-09-18T07:45:42.533680+0000","last_active":"2024-09-18T07:47:31.124461+0000","last_peered":"2024-09-18T07:47:31.124461+0000","last_clean":"2024-09-18T07:47:31.124461+0000","last_became_active":"2024-09-18T07:44:43.782348+0000","last_became_peered":"2024-09-18T07:44:43.782348+0000","last_unstale":"2024-09-18T07:47:31.124461+0000","last_undegraded":"2024-09-18T07:47:31.124461+0000","last_fullsized":"2024-09-18T07:47:31.124461+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:42.533634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:45:42.533634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"323'15","reported_seq":897,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.129642+0000","last_change":"2024-09-18T07:46:51.712735+0000","last_active":"2024-09-18T07:47:21.129642+0000","last_peered":"2024-09-18T07:47:21.129642+0000","last_clean":"2024-09-18T07:47:21.129642+0000","last_became_active":"2024-09-18T07:35:01.414458+0000","last_became_peered":"2024-09-18T07:35:01.414458+0000","last_unstale":"2024-09-18T07:47:21.129642+0000","last_undegraded":"2024-09-18T07:47:21.129642+0000","last_fullsized":"2024-09-18T07:47:21.129642+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:46:51.712666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:51.712666+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:52.984099+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":662,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123389+0000","last_change":"2024-09-18T07:45:51.559901+0000","last_active":"2024-09-18T07:47:31.123389+0000","last_peered":"2024-09-18T07:47:31.123389+0000","last_clean":"2024-09-18T07:47:31.123389+0000","last_became_active":"2024-09-18T07:44:43.782930+0000","last_became_peered":"2024-09-18T07:44:43.782930+0000","last_unstale":"2024-09-18T07:47:31.123389+0000","last_undegraded":"2024-09-18T07:47:31.123389+0000","last_fullsized":"2024-09-18T07:47:31.123389+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:51.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:45:51.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":892,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.120694+0000","last_change":"2024-09-18T07:47:20.141939+0000","last_active":"2024-09-18T07:47:31.120694+0000","last_peered":"2024-09-18T07:47:31.120694+0000","last_clean":"2024-09-18T07:47:31.120694+0000","last_became_active":"2024-09-18T07:47:20.141420+0000","last_became_peered":"2024-09-18T07:47:20.141420+0000","last_unstale":"2024-09-18T07:47:31.120694+0000","last_undegraded":"2024-09-18T07:47:31.120694+0000","last_fullsized":"2024-09-18T07:47:31.120694+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:53.481418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:53.481418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:09.936935+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":864,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.126388+0000","last_change":"2024-09-18T07:44:43.790870+0000","last_active":"2024-09-18T07:47:31.126388+0000","last_peered":"2024-09-18T07:47:31.126388+0000","last_clean":"2024-09-18T07:47:31.126388+0000","last_became_active":"2024-09-18T07:44:43.790582+0000","last_became_peered":"2024-09-18T07:44:43.790582+0000","last_unstale":"2024-09-18T07:47:31.126388+0000","last_undegraded":"2024-09-18T07:47:31.126388+0000","last_fullsized":"2024-09-18T07:47:31.126388+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:34.876429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:34.876429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:02.938984+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"323'15","reported_seq":809,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.113397+0000","last_change":"2024-09-18T07:47:11.201346+0000","last_active":"2024-09-18T07:47:21.113397+0000","last_peered":"2024-09-18T07:47:21.113397+0000","last_clean":"2024-09-18T07:47:21.113397+0000","last_became_active":"2024-09-18T07:47:11.198306+0000","last_became_peered":"2024-09-18T07:47:11.198306+0000","last_unstale":"2024-09-18T07:47:21.113397+0000","last_undegraded":"2024-09-18T07:47:21.113397+0000","last_fullsized":"2024-09-18T07:47:21.113397+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:52.366672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:52.366672+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:20.107354+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":318,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123706+0000","last_change":"2024-09-18T07:46:29.319452+0000","last_active":"2024-09-18T07:47:31.123706+0000","last_peered":"2024-09-18T07:47:31.123706+0000","last_clean":"2024-09-18T07:47:31.123706+0000","last_became_active":"2024-09-18T07:46:29.319112+0000","last_became_peered":"2024-09-18T07:46:29.319112+0000","last_unstale":"2024-09-18T07:47:31.123706+0000","last_undegraded":"2024-09-18T07:47:31.123706+0000","last_fullsized":"2024-09-18T07:47:31.123706+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:31.255278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:43:31.255278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"173'10","reported_seq":1086,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119866+0000","last_change":"2024-09-18T07:46:08.086507+0000","last_active":"2024-09-18T07:47:31.119866+0000","last_peered":"2024-09-18T07:47:31.119866+0000","last_clean":"2024-09-18T07:47:31.119866+0000","last_became_active":"2024-09-18T07:35:01.407568+0000","last_became_peered":"2024-09-18T07:35:01.407568+0000","last_unstale":"2024-09-18T07:47:31.119866+0000","last_undegraded":"2024-09-18T07:47:31.119866+0000","last_fullsized":"2024-09-18T07:47:31.119866+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:46:08.086437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:08.086437+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:32.468227+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":759,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119939+0000","last_change":"2024-09-18T07:47:11.193555+0000","last_active":"2024-09-18T07:47:31.119939+0000","last_peered":"2024-09-18T07:47:31.119939+0000","last_clean":"2024-09-18T07:47:31.119939+0000","last_became_active":"2024-09-18T07:47:11.193273+0000","last_became_peered":"2024-09-18T07:47:11.193273+0000","last_unstale":"2024-09-18T07:47:31.119939+0000","last_undegraded":"2024-09-18T07:47:31.119939+0000","last_fullsized":"2024-09-18T07:47:31.119939+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:48.955463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.955463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"323'20","reported_seq":676,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.124557+0000","last_change":"2024-09-18T07:47:20.165350+0000","last_active":"2024-09-18T07:47:31.124557+0000","last_peered":"2024-09-18T07:47:31.124557+0000","last_clean":"2024-09-18T07:47:31.124557+0000","last_became_active":"2024-09-18T07:47:20.138883+0000","last_became_peered":"2024-09-18T07:47:20.138883+0000","last_unstale":"2024-09-18T07:47:31.124557+0000","last_undegraded":"2024-09-18T07:47:31.124557+0000","last_fullsized":"2024-09-18T07:47:31.124557+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:46:14.650883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:14.650883+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":648,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.125045+0000","last_change":"2024-09-18T07:46:08.731460+0000","last_active":"2024-09-18T07:47:31.125045+0000","last_peered":"2024-09-18T07:47:31.125045+0000","last_clean":"2024-09-18T07:47:31.125045+0000","last_became_active":"2024-09-18T07:44:43.779305+0000","last_became_peered":"2024-09-18T07:44:43.779305+0000","last_unstale":"2024-09-18T07:47:31.125045+0000","last_undegraded":"2024-09-18T07:47:31.125045+0000","last_fullsized":"2024-09-18T07:47:31.125045+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:08.731385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:08.731385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:34.080154+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"467'14","reported_seq":917,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.569753+0000","last_change":"2024-09-18T07:47:20.169193+0000","last_active":"2024-09-18T07:47:21.569753+0000","last_peered":"2024-09-18T07:47:21.569753+0000","last_clean":"2024-09-18T07:47:21.569753+0000","last_became_active":"2024-09-18T07:47:20.141913+0000","last_became_peered":"2024-09-18T07:47:20.141913+0000","last_unstale":"2024-09-18T07:47:21.569753+0000","last_undegraded":"2024-09-18T07:47:21.569753+0000","last_fullsized":"2024-09-18T07:47:21.569753+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:40:40.352040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.352040+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":306,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.125026+0000","last_change":"2024-09-18T07:46:29.319811+0000","last_active":"2024-09-18T07:47:31.125026+0000","last_peered":"2024-09-18T07:47:31.125026+0000","last_clean":"2024-09-18T07:47:31.125026+0000","last_became_active":"2024-09-18T07:46:29.319164+0000","last_became_peered":"2024-09-18T07:46:29.319164+0000","last_unstale":"2024-09-18T07:47:31.125026+0000","last_undegraded":"2024-09-18T07:47:31.125026+0000","last_fullsized":"2024-09-18T07:47:31.125026+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:33.228024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:33.228024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"323'19","reported_seq":902,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119757+0000","last_change":"2024-09-18T07:47:20.196648+0000","last_active":"2024-09-18T07:47:31.119757+0000","last_peered":"2024-09-18T07:47:31.119757+0000","last_clean":"2024-09-18T07:47:31.119757+0000","last_became_active":"2024-09-18T07:47:20.140070+0000","last_became_peered":"2024-09-18T07:47:20.140070+0000","last_unstale":"2024-09-18T07:47:31.119757+0000","last_undegraded":"2024-09-18T07:47:31.119757+0000","last_fullsized":"2024-09-18T07:47:31.119757+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:41:54.022421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.022421+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":816,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119690+0000","last_change":"2024-09-18T07:46:06.964899+0000","last_active":"2024-09-18T07:47:31.119690+0000","last_peered":"2024-09-18T07:47:31.119690+0000","last_clean":"2024-09-18T07:47:31.119690+0000","last_became_active":"2024-09-18T07:44:43.780305+0000","last_became_peered":"2024-09-18T07:44:43.780305+0000","last_unstale":"2024-09-18T07:47:31.119690+0000","last_undegraded":"2024-09-18T07:47:31.119690+0000","last_fullsized":"2024-09-18T07:47:31.119690+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:06.964847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:06.964847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":469,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.112792+0000","last_change":"2024-09-18T07:47:20.145972+0000","last_active":"2024-09-18T07:47:21.112792+0000","last_peered":"2024-09-18T07:47:21.112792+0000","last_clean":"2024-09-18T07:47:21.112792+0000","last_became_active":"2024-09-18T07:47:20.145450+0000","last_became_peered":"2024-09-18T07:47:20.145450+0000","last_unstale":"2024-09-18T07:47:21.112792+0000","last_undegraded":"2024-09-18T07:47:21.112792+0000","last_fullsized":"2024-09-18T07:47:21.112792+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:36.563010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:43:36.563010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":369,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.116781+0000","last_change":"2024-09-18T07:47:14.913645+0000","last_active":"2024-09-18T07:47:21.116781+0000","last_peered":"2024-09-18T07:47:21.116781+0000","last_clean":"2024-09-18T07:47:21.116781+0000","last_became_active":"2024-09-18T07:47:11.188162+0000","last_became_peered":"2024-09-18T07:47:11.188162+0000","last_unstale":"2024-09-18T07:47:21.116781+0000","last_undegraded":"2024-09-18T07:47:21.116781+0000","last_fullsized":"2024-09-18T07:47:21.116781+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:14.913596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:14.913596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:22.308133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"323'15","reported_seq":514,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.124169+0000","last_change":"2024-09-18T07:47:20.167453+0000","last_active":"2024-09-18T07:47:31.124169+0000","last_peered":"2024-09-18T07:47:31.124169+0000","last_clean":"2024-09-18T07:47:31.124169+0000","last_became_active":"2024-09-18T07:47:20.141411+0000","last_became_peered":"2024-09-18T07:47:20.141411+0000","last_unstale":"2024-09-18T07:47:31.124169+0000","last_undegraded":"2024-09-18T07:47:31.124169+0000","last_fullsized":"2024-09-18T07:47:31.124169+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:44:04.219913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:04.219913+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":640,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.124401+0000","last_change":"2024-09-18T07:44:43.783005+0000","last_active":"2024-09-18T07:47:31.124401+0000","last_peered":"2024-09-18T07:47:31.124401+0000","last_clean":"2024-09-18T07:47:31.124401+0000","last_became_active":"2024-09-18T07:44:43.782709+0000","last_became_peered":"2024-09-18T07:44:43.782709+0000","last_unstale":"2024-09-18T07:47:31.124401+0000","last_undegraded":"2024-09-18T07:47:31.124401+0000","last_fullsized":"2024-09-18T07:47:31.124401+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:47.880911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:47.880911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:01.682226+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":369,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.116364+0000","last_change":"2024-09-18T07:47:20.135826+0000","last_active":"2024-09-18T07:47:21.116364+0000","last_peered":"2024-09-18T07:47:21.116364+0000","last_clean":"2024-09-18T07:47:21.116364+0000","last_became_active":"2024-09-18T07:47:20.135517+0000","last_became_peered":"2024-09-18T07:47:20.135517+0000","last_unstale":"2024-09-18T07:47:21.116364+0000","last_undegraded":"2024-09-18T07:47:21.116364+0000","last_fullsized":"2024-09-18T07:47:21.116364+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:13.911901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:13.911901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:35.013343+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":886,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.116315+0000","last_change":"2024-09-18T07:47:20.136279+0000","last_active":"2024-09-18T07:47:21.116315+0000","last_peered":"2024-09-18T07:47:21.116315+0000","last_clean":"2024-09-18T07:47:21.116315+0000","last_became_active":"2024-09-18T07:47:20.135880+0000","last_became_peered":"2024-09-18T07:47:20.135880+0000","last_unstale":"2024-09-18T07:47:21.116315+0000","last_undegraded":"2024-09-18T07:47:21.116315+0000","last_fullsized":"2024-09-18T07:47:21.116315+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:37.209823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:44:37.209823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"323'19","reported_seq":984,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119115+0000","last_change":"2024-09-18T07:46:19.041222+0000","last_active":"2024-09-18T07:47:31.119115+0000","last_peered":"2024-09-18T07:47:31.119115+0000","last_clean":"2024-09-18T07:47:31.119115+0000","last_became_active":"2024-09-18T07:37:43.066208+0000","last_became_peered":"2024-09-18T07:37:43.066208+0000","last_unstale":"2024-09-18T07:47:31.119115+0000","last_undegraded":"2024-09-18T07:47:31.119115+0000","last_fullsized":"2024-09-18T07:47:31.119115+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:46:19.041153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:19.041153+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":789,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.113690+0000","last_change":"2024-09-18T07:47:20.140839+0000","last_active":"2024-09-18T07:47:21.113690+0000","last_peered":"2024-09-18T07:47:21.113690+0000","last_clean":"2024-09-18T07:47:21.113690+0000","last_became_active":"2024-09-18T07:47:20.140495+0000","last_became_peered":"2024-09-18T07:47:20.140495+0000","last_unstale":"2024-09-18T07:47:21.113690+0000","last_undegraded":"2024-09-18T07:47:21.113690+0000","last_fullsized":"2024-09-18T07:47:21.113690+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:32.657943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:43:32.657943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":662,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123802+0000","last_change":"2024-09-18T07:46:13.660302+0000","last_active":"2024-09-18T07:47:31.123802+0000","last_peered":"2024-09-18T07:47:31.123802+0000","last_clean":"2024-09-18T07:47:31.123802+0000","last_became_active":"2024-09-18T07:44:43.778318+0000","last_became_peered":"2024-09-18T07:44:43.778318+0000","last_unstale":"2024-09-18T07:47:31.123802+0000","last_undegraded":"2024-09-18T07:47:31.123802+0000","last_fullsized":"2024-09-18T07:47:31.123802+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:13.660254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:13.660254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:42.326916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":883,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119495+0000","last_change":"2024-09-18T07:44:43.090466+0000","last_active":"2024-09-18T07:47:31.119495+0000","last_peered":"2024-09-18T07:47:31.119495+0000","last_clean":"2024-09-18T07:47:31.119495+0000","last_became_active":"2024-09-18T07:37:43.067066+0000","last_became_peered":"2024-09-18T07:37:43.067066+0000","last_unstale":"2024-09-18T07:47:31.119495+0000","last_undegraded":"2024-09-18T07:47:31.119495+0000","last_fullsized":"2024-09-18T07:47:31.119495+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:43.090421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:44:43.090421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"323'8","reported_seq":938,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119002+0000","last_change":"2024-09-18T07:47:20.199058+0000","last_active":"2024-09-18T07:47:31.119002+0000","last_peered":"2024-09-18T07:47:31.119002+0000","last_clean":"2024-09-18T07:47:31.119002+0000","last_became_active":"2024-09-18T07:47:20.142156+0000","last_became_peered":"2024-09-18T07:47:20.142156+0000","last_unstale":"2024-09-18T07:47:31.119002+0000","last_undegraded":"2024-09-18T07:47:31.119002+0000","last_fullsized":"2024-09-18T07:47:31.119002+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'8","last_scrub_stamp":"2024-09-18T07:45:20.998406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:45:20.998406+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":28,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.113908+0000","last_change":"2024-09-18T07:47:11.200765+0000","last_active":"2024-09-18T07:47:21.113908+0000","last_peered":"2024-09-18T07:47:21.113908+0000","last_clean":"2024-09-18T07:47:21.113908+0000","last_became_active":"2024-09-18T07:47:11.197716+0000","last_became_peered":"2024-09-18T07:47:11.197716+0000","last_unstale":"2024-09-18T07:47:21.113908+0000","last_undegraded":"2024-09-18T07:47:21.113908+0000","last_fullsized":"2024-09-18T07:47:21.113908+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:08.235926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:44:08.235926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":492,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123109+0000","last_change":"2024-09-18T07:44:43.793061+0000","last_active":"2024-09-18T07:47:31.123109+0000","last_peered":"2024-09-18T07:47:31.123109+0000","last_clean":"2024-09-18T07:47:31.123109+0000","last_became_active":"2024-09-18T07:44:43.792641+0000","last_became_peered":"2024-09-18T07:44:43.792641+0000","last_unstale":"2024-09-18T07:47:31.123109+0000","last_undegraded":"2024-09-18T07:47:31.123109+0000","last_fullsized":"2024-09-18T07:47:31.123109+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:40.106921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.106921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:55.125341+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":640,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123912+0000","last_change":"2024-09-18T07:44:43.783022+0000","last_active":"2024-09-18T07:47:31.123912+0000","last_peered":"2024-09-18T07:47:31.123912+0000","last_clean":"2024-09-18T07:47:31.123912+0000","last_became_active":"2024-09-18T07:44:43.782378+0000","last_became_peered":"2024-09-18T07:44:43.782378+0000","last_unstale":"2024-09-18T07:47:31.123912+0000","last_undegraded":"2024-09-18T07:47:31.123912+0000","last_fullsized":"2024-09-18T07:47:31.123912+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:38.492269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.492269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":434,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-18T07:47:21.113557+0000","last_change":"2024-09-18T07:47:20.139313+0000","last_active":"2024-09-18T07:47:21.113557+0000","last_peered":"2024-09-18T07:47:21.113557+0000","last_clean":"2024-09-18T07:47:21.113557+0000","last_became_active":"2024-09-18T07:47:20.138598+0000","last_became_peered":"2024-09-18T07:47:20.138598+0000","last_unstale":"2024-09-18T07:47:21.113557+0000","last_undegraded":"2024-09-18T07:47:21.113557+0000","last_fullsized":"2024-09-18T07:47:21.113557+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:44.492181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:44:44.492181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":908,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.119276+0000","last_change":"2024-09-18T07:47:20.202612+0000","last_active":"2024-09-18T07:47:31.119276+0000","last_peered":"2024-09-18T07:47:31.119276+0000","last_clean":"2024-09-18T07:47:31.119276+0000","last_became_active":"2024-09-18T07:47:20.143572+0000","last_became_peered":"2024-09-18T07:47:20.143572+0000","last_unstale":"2024-09-18T07:47:31.119276+0000","last_undegraded":"2024-09-18T07:47:31.119276+0000","last_fullsized":"2024-09-18T07:47:31.119276+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:45:21.944553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:45:21.944553+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"323'19","reported_seq":560,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.123936+0000","last_change":"2024-09-18T07:47:20.160909+0000","last_active":"2024-09-18T07:47:31.123936+0000","last_peered":"2024-09-18T07:47:31.123936+0000","last_clean":"2024-09-18T07:47:31.123936+0000","last_became_active":"2024-09-18T07:47:20.142007+0000","last_became_peered":"2024-09-18T07:47:20.142007+0000","last_unstale":"2024-09-18T07:47:31.123936+0000","last_undegraded":"2024-09-18T07:47:31.123936+0000","last_fullsized":"2024-09-18T07:47:31.123936+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:43:10.104058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:43:10.104058+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":394,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-18T07:47:31.124582+0000","last_change":"2024-09-18T07:47:20.140033+0000","last_active":"2024-09-18T07:47:31.124582+0000","last_peered":"2024-09-18T07:47:31.124582+0000","last_clean":"2024-09-18T07:47:31.124582+0000","last_became_active":"2024-09-18T07:47:20.139668+0000","last_became_peered":"2024-09-18T07:47:20.139668+0000","last_unstale":"2024-09-18T07:47:31.124582+0000","last_undegraded":"2024-09-18T07:47:31.124582+0000","last_fullsized":"2024-09-18T07:47:31.124582+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:44.585789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:45:44.585789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1883766784,"num_objects":28744,"num_object_clones":0,"num_object_copies":86232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3365,"num_objects_misplaced":619,"num_objects_unfound":0,"num_objects_dirty":28744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28744,"num_write_kb":1839616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2752,"num_bytes_recovered":180355072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5887426560,"data_stored":5887426560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28744,"ondisk_log_size":28744,"up":48,"acting":43,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1104,"num_read_kb":861,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":9230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},"log_size":688,"ondisk_log_size":688,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":752,"num_read_kb":747,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201483,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1346936,"kb_used_data":747936,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92398216,"statfs":{"total":95995035648,"available":94615773184,"internally_reserved":0,"allocated":765886464,"data_stored":765115648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":613349834},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463121,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1574236,"kb_used_data":993284,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92170916,"statfs":{"total":95995035648,"available":94383017984,"internally_reserved":0,"allocated":1017122816,"data_stored":1016379367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594868678},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":460,"seq":1975684956165,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2046088,"kb_used_data":1955320,"kb_used_omap":6,"kb_used_meta":90745,"kb_avail":91699064,"statfs":{"total":95995035648,"available":93899841536,"internally_reserved":0,"allocated":2002247680,"data_stored":2001503242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6877,"internal_metadata":92923171},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":395,"seq":1696512081955,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":962492,"kb_used_data":573860,"kb_used_omap":4,"kb_used_meta":388603,"kb_avail":92782660,"statfs":{"total":95995035648,"available":95009443840,"internally_reserved":0,"allocated":587632640,"data_stored":586857561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4447,"internal_metadata":397930145},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149921,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1502832,"kb_used_data":817756,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92242320,"statfs":{"total":95995035648,"available":94456135680,"internally_reserved":0,"allocated":837382144,"data_stored":836609474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":701495751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411559,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1320724,"kb_used_data":689792,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92424428,"statfs":{"total":95995035648,"available":94642614272,"internally_reserved":0,"allocated":706347008,"data_stored":705601485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,3,13],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575086,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1410712,"kb_used_data":739528,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92334440,"statfs":{"total":95995035648,"available":94550466560,"internally_reserved":0,"allocated":757276672,"data_stored":756466466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738610,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1400160,"kb_used_data":660944,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":92344992,"statfs":{"total":95995035648,"available":94561271808,"internally_reserved":0,"allocated":676806656,"data_stored":676038962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756939205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":677249024,"data_stored":677249024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":785055744,"data_stored":785055744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":708444160,"data_stored":708444160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":837222400,"data_stored":837222400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":589430784,"data_stored":589430784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":507248640,"data_stored":507248640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1016659968,"data_stored":1016659968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":766115840,"data_stored":766115840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:47:35.317 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-09-18T07:47:35.317 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:35.352 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:35.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:35.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:35.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:35.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:35 smithi203 ceph-mon[32237]: from='client.16608 v1:172.21.15.137:0/3995293049' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:47:35.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:35 smithi203 ceph-mon[32237]: pgmap v885: 158 pgs: 2 active+recovering+undersized+remapped, 2 peering, 3 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s; 2534/87909 objects degraded (2.883%); 960/87909 objects misplaced (1.092%); 30 MiB/s, 479 objects/s recovering 2024-09-18T07:47:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:35 smithi203 ceph-mon[32237]: Health check update: Degraded data redundancy: 2534/87909 objects degraded (2.883%), 3 pgs degraded (PG_DEGRADED) 2024-09-18T07:47:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:35 smithi203 ceph-mon[32237]: osdmap e472: 8 total, 8 up, 8 in 2024-09-18T07:47:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:35 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1767296993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:35.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:35 smithi203 ceph-mon[32237]: osdmap e473: 8 total, 8 up, 8 in 2024-09-18T07:47:35.489 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:35.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:35.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:35.591 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:35.630 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:35.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:35.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:35.637 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:35 smithi137 ceph-mon[29408]: from='client.16608 v1:172.21.15.137:0/3995293049' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:47:35.637 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:35 smithi137 ceph-mon[29408]: pgmap v885: 158 pgs: 2 active+recovering+undersized+remapped, 2 peering, 3 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s; 2534/87909 objects degraded (2.883%); 960/87909 objects misplaced (1.092%); 30 MiB/s, 479 objects/s recovering 2024-09-18T07:47:35.637 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:35 smithi137 ceph-mon[29408]: Health check update: Degraded data redundancy: 2534/87909 objects degraded (2.883%), 3 pgs degraded (PG_DEGRADED) 2024-09-18T07:47:35.637 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:35 smithi137 ceph-mon[29408]: osdmap e472: 8 total, 8 up, 8 in 2024-09-18T07:47:35.637 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:35 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1767296993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:35.637 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:35 smithi137 ceph-mon[29408]: osdmap e473: 8 total, 8 up, 8 in 2024-09-18T07:47:35.637 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:35 smithi137 ceph-mon[39175]: from='client.16608 v1:172.21.15.137:0/3995293049' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:47:35.637 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:35 smithi137 ceph-mon[39175]: pgmap v885: 158 pgs: 2 active+recovering+undersized+remapped, 2 peering, 3 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s; 2534/87909 objects degraded (2.883%); 960/87909 objects misplaced (1.092%); 30 MiB/s, 479 objects/s recovering 2024-09-18T07:47:35.637 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:35 smithi137 ceph-mon[39175]: Health check update: Degraded data redundancy: 2534/87909 objects degraded (2.883%), 3 pgs degraded (PG_DEGRADED) 2024-09-18T07:47:35.638 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:35 smithi137 ceph-mon[39175]: osdmap e472: 8 total, 8 up, 8 in 2024-09-18T07:47:35.638 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:35 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1767296993' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:35.638 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:35 smithi137 ceph-mon[39175]: osdmap e473: 8 total, 8 up, 8 in 2024-09-18T07:47:35.733 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:35.768 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:35.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:35.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:35.870 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:35.901 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:35.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:35.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:36.003 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:36.024 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:36.032 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:36.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:36.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:36.045 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 21 16 33106 33090 98.4695 117.125 0.00790622 0.0101483 2024-09-18T07:47:36.133 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:36.179 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:36.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:36.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:36.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:47:36.281 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:36.309 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:36.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:36.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:36.411 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:36.432 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:36.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:36.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:36.534 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:36.556 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:36.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:36.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:36.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:36.680 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:36.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:36.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:36.782 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:36.804 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:36.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:36.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:36.905 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:36.927 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:36.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:36.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:37.022 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:47:37.023 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:47:37.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:37.045 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 22 16 33665 33649 95.5812 34.9375 0.00225639 0.0100914 2024-09-18T07:47:37.050 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:37.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:37.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:37.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:37.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:37.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:37.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:37.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:37.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:37.306 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:37.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:37.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:37.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:37 smithi137 ceph-mon[29408]: pgmap v888: 158 pgs: 2 active+recovering+undersized+remapped, 2 peering, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 31 MiB/s wr, 496 op/s; 1676/88596 objects degraded (1.892%); 960/88596 objects misplaced (1.084%); 21 MiB/s, 336 objects/s recovering 2024-09-18T07:47:37.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:37 smithi137 ceph-mon[29408]: osdmap e474: 8 total, 8 up, 8 in 2024-09-18T07:47:37.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:37 smithi137 ceph-mon[39175]: pgmap v888: 158 pgs: 2 active+recovering+undersized+remapped, 2 peering, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 31 MiB/s wr, 496 op/s; 1676/88596 objects degraded (1.892%); 960/88596 objects misplaced (1.084%); 21 MiB/s, 336 objects/s recovering 2024-09-18T07:47:37.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:37 smithi137 ceph-mon[39175]: osdmap e474: 8 total, 8 up, 8 in 2024-09-18T07:47:37.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:47:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:47:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:47:37.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:37.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:37.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:37.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:37 smithi203 ceph-mon[32237]: pgmap v888: 158 pgs: 2 active+recovering+undersized+remapped, 2 peering, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 31 MiB/s wr, 496 op/s; 1676/88596 objects degraded (1.892%); 960/88596 objects misplaced (1.084%); 21 MiB/s, 336 objects/s recovering 2024-09-18T07:47:37.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:37 smithi203 ceph-mon[32237]: osdmap e474: 8 total, 8 up, 8 in 2024-09-18T07:47:37.534 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:37.555 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:37.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:37.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:37.657 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:37.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:37.680 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:37.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:37.781 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:37.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:37.804 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:37.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:37.905 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:37.926 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:37.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:37.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:38.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:38.045 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 23 16 33960 33944 92.2273 18.4375 0.00250865 0.0105025 2024-09-18T07:47:38.050 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:38.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:38.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:38.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:47:38.152 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:38.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T07:47:38.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:38.174 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:38.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:38.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:47:37 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:47:38.258 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:38.276 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:38.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:38.304 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:38.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:38.398 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:38 smithi137 ceph-mon[39175]: osdmap e475: 8 total, 8 up, 8 in 2024-09-18T07:47:38.398 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:38 smithi137 ceph-mon[29408]: osdmap e475: 8 total, 8 up, 8 in 2024-09-18T07:47:38.405 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:38.434 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:38.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:38.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:38.537 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:38.577 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:38.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:38.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:38.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:38.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:38 smithi203 ceph-mon[32237]: osdmap e475: 8 total, 8 up, 8 in 2024-09-18T07:47:38.714 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:38.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:38.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:38.817 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:38.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:38.875 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:38.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:38.967 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:38.976 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:38.995 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:39.002 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:39.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:39.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:39.046 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 24 16 35425 35409 92.1957 91.5625 0.00728698 0.0108406 2024-09-18T07:47:39.104 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:39.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:39.140 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:39.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:39.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:39.275 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:39.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:39.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:39.371 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:39 smithi137 ceph-mon[29408]: pgmap v891: 158 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 155 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.05k op/s; 341/100473 objects misplaced (0.339%); 49 MiB/s, 788 objects/s recovering 2024-09-18T07:47:39.371 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:39 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:39.371 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:39 smithi137 ceph-mon[29408]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1676/88596 objects degraded (1.892%), 2 pgs degraded) 2024-09-18T07:47:39.371 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:39 smithi137 ceph-mon[29408]: osdmap e476: 8 total, 8 up, 8 in 2024-09-18T07:47:39.372 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:39 smithi137 ceph-mon[39175]: pgmap v891: 158 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 155 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.05k op/s; 341/100473 objects misplaced (0.339%); 49 MiB/s, 788 objects/s recovering 2024-09-18T07:47:39.372 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:39 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:39.372 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:39 smithi137 ceph-mon[39175]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1676/88596 objects degraded (1.892%), 2 pgs degraded) 2024-09-18T07:47:39.372 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:39 smithi137 ceph-mon[39175]: osdmap e476: 8 total, 8 up, 8 in 2024-09-18T07:47:39.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:39.414 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:39.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:39.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:39.516 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:39.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:39.553 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:39.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:39.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:39.685 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:39.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:39.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:39 smithi203 ceph-mon[32237]: pgmap v891: 158 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 155 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.05k op/s; 341/100473 objects misplaced (0.339%); 49 MiB/s, 788 objects/s recovering 2024-09-18T07:47:39.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:39 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:39 smithi203 ceph-mon[32237]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1676/88596 objects degraded (1.892%), 2 pgs degraded) 2024-09-18T07:47:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:39 smithi203 ceph-mon[32237]: osdmap e476: 8 total, 8 up, 8 in 2024-09-18T07:47:39.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:39.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:39.828 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:39.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:39.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:39.943 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:47:39.943 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:47:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:39.968 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:39.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:39.976 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:47:39.994 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:47:40.013 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:47:40.014 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:47:40.047 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 25 15 37350 37335 93.3223 120.375 0.0034522 0.0107102 2024-09-18T07:47:40.049 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:47:40.049 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:47:40.069 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:40.085 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:47:40.085 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:47:40.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:40.101 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:40.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:40.118 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:47:40.119 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:47:40.134 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:47:40.155 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:47:40.155 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:47:40.190 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:47:40.191 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:47:40.202 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:40.245 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:47:40.246 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:47:40.246 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:47:40.246 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:47:40.246 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:47:40.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:40.247 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:40.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:40.318 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 6, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 6, 7, 4, 5] 2024-09-18T07:47:40.318 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:47:40.318 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-18T07:47:40.318 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-18T07:47:40.319 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:47:40.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:40.356 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:40 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2410053672' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:47:40.357 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:40 smithi137 ceph-mon[29408]: from='client.36461 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:47:40.357 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:40 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1103611382' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:40.357 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:40 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:47:40.357 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:40 smithi137 ceph-mon[29408]: from='client.36461 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:47:40.357 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:40 smithi137 ceph-mon[29408]: osdmap e477: 8 total, 8 up, 8 in 2024-09-18T07:47:40.357 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:40 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2410053672' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:47:40.357 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:40 smithi137 ceph-mon[39175]: from='client.36461 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:47:40.358 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:40 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1103611382' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:40.358 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:40 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:47:40.358 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:40 smithi137 ceph-mon[39175]: from='client.36461 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:47:40.358 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:40 smithi137 ceph-mon[39175]: osdmap e477: 8 total, 8 up, 8 in 2024-09-18T07:47:40.398 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:40.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:40.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:40.500 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:40.533 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:40.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:40.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:40.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:40.695 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:40.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:40.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:40 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2410053672' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:47:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:40 smithi203 ceph-mon[32237]: from='client.36461 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:47:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:40 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1103611382' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:40 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:47:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:40 smithi203 ceph-mon[32237]: from='client.36461 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:47:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:40 smithi203 ceph-mon[32237]: osdmap e477: 8 total, 8 up, 8 in 2024-09-18T07:47:40.796 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:40.832 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:40.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:40.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:40.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:40.955 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:40.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:41.046 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 26 16 38869 38853 93.3816 94.875 0.00263158 0.010648 2024-09-18T07:47:41.057 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:41.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:41.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:41.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:41.186 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:41.214 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:41.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:41.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:41.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:41.357 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:41.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:41.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:41.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:41 smithi137 ceph-mon[29408]: pgmap v893: 158 pgs: 1 peering, 1 active+undersized+remapped, 156 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 949 op/s; 35 MiB/s, 559 objects/s recovering 2024-09-18T07:47:41.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:41 smithi137 ceph-mon[29408]: 6.b scrub starts 2024-09-18T07:47:41.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:41 smithi137 ceph-mon[29408]: 6.b scrub ok 2024-09-18T07:47:41.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:41 smithi137 ceph-mon[39175]: pgmap v893: 158 pgs: 1 peering, 1 active+undersized+remapped, 156 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 949 op/s; 35 MiB/s, 559 objects/s recovering 2024-09-18T07:47:41.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:41 smithi137 ceph-mon[39175]: 6.b scrub starts 2024-09-18T07:47:41.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:41 smithi137 ceph-mon[39175]: 6.b scrub ok 2024-09-18T07:47:41.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:41.488 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:41.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:41.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:41.581 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T07:47:41.590 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:41.620 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:41.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:41.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:41.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:41 smithi203 ceph-mon[32237]: pgmap v893: 158 pgs: 1 peering, 1 active+undersized+remapped, 156 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 949 op/s; 35 MiB/s, 559 objects/s recovering 2024-09-18T07:47:41.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:41 smithi203 ceph-mon[32237]: 6.b scrub starts 2024-09-18T07:47:41.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:41 smithi203 ceph-mon[32237]: 6.b scrub ok 2024-09-18T07:47:41.722 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:41.756 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:41.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:41.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:41.859 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:41.892 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:41.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:41.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:41.994 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:42.027 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:42.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:42.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:42.047 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 27 16 40718 40702 94.2027 115.562 0.0161382 0.0106096 2024-09-18T07:47:42.084 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:42.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:42.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:42.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:42.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:42.271 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:42.308 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:42.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:42.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:42.409 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:42.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:42 smithi137 ceph-mon[29408]: 5.2 scrub starts 2024-09-18T07:47:42.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:42 smithi137 ceph-mon[29408]: 5.2 scrub ok 2024-09-18T07:47:42.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:42 smithi137 ceph-mon[29408]: 3.18 scrub starts 2024-09-18T07:47:42.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:42 smithi137 ceph-mon[29408]: 3.18 scrub ok 2024-09-18T07:47:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:42 smithi137 ceph-mon[29408]: 6.5 scrub starts 2024-09-18T07:47:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:42 smithi137 ceph-mon[29408]: 6.5 scrub ok 2024-09-18T07:47:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:42 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:42 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:47:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:42 smithi137 ceph-mon[39175]: 5.2 scrub starts 2024-09-18T07:47:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:42 smithi137 ceph-mon[39175]: 5.2 scrub ok 2024-09-18T07:47:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:42 smithi137 ceph-mon[39175]: 3.18 scrub starts 2024-09-18T07:47:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:42 smithi137 ceph-mon[39175]: 3.18 scrub ok 2024-09-18T07:47:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:42 smithi137 ceph-mon[39175]: 6.5 scrub starts 2024-09-18T07:47:42.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:42 smithi137 ceph-mon[39175]: 6.5 scrub ok 2024-09-18T07:47:42.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:42 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:42.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:42 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:47:42.449 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:42.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:42.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:42.551 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:42.589 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:42.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:42.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:42.690 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:42 smithi203 ceph-mon[32237]: 5.2 scrub starts 2024-09-18T07:47:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:42 smithi203 ceph-mon[32237]: 5.2 scrub ok 2024-09-18T07:47:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:42 smithi203 ceph-mon[32237]: 3.18 scrub starts 2024-09-18T07:47:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:42 smithi203 ceph-mon[32237]: 3.18 scrub ok 2024-09-18T07:47:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:42 smithi203 ceph-mon[32237]: 6.5 scrub starts 2024-09-18T07:47:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:42 smithi203 ceph-mon[32237]: 6.5 scrub ok 2024-09-18T07:47:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:42 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:42 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:47:42.726 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:42.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:42.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:42.828 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:42.861 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:42.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:42.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:42.963 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:42.993 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:42.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:42.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:43.047 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 28 16 42582 42566 94.9986 116.5 0.0219046 0.0105215 2024-09-18T07:47:43.095 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:43.122 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:43.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:43.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:43.224 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:43.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:43.255 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:43.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:43.323 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:43.356 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:43.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:43.406 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:43.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:43.503 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:43 smithi137 ceph-mon[29408]: pgmap v895: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.8 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 29 MiB/s, 467 objects/s recovering 2024-09-18T07:47:43.503 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:43 smithi137 ceph-mon[29408]: 6.a scrub starts 2024-09-18T07:47:43.503 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:43 smithi137 ceph-mon[29408]: 6.a scrub ok 2024-09-18T07:47:43.503 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:43 smithi137 ceph-mon[29408]: 3.b scrub starts 2024-09-18T07:47:43.503 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:43 smithi137 ceph-mon[29408]: 3.b scrub ok 2024-09-18T07:47:43.503 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:43 smithi137 ceph-mon[29408]: 6.3 scrub starts 2024-09-18T07:47:43.503 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:43 smithi137 ceph-mon[29408]: 6.3 scrub ok 2024-09-18T07:47:43.504 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:43 smithi137 ceph-mon[39175]: pgmap v895: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.8 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 29 MiB/s, 467 objects/s recovering 2024-09-18T07:47:43.504 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:43 smithi137 ceph-mon[39175]: 6.a scrub starts 2024-09-18T07:47:43.504 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:43 smithi137 ceph-mon[39175]: 6.a scrub ok 2024-09-18T07:47:43.504 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:43 smithi137 ceph-mon[39175]: 3.b scrub starts 2024-09-18T07:47:43.504 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:43 smithi137 ceph-mon[39175]: 3.b scrub ok 2024-09-18T07:47:43.504 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:43 smithi137 ceph-mon[39175]: 6.3 scrub starts 2024-09-18T07:47:43.504 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:43 smithi137 ceph-mon[39175]: 6.3 scrub ok 2024-09-18T07:47:43.507 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:43.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:43.557 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:43.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:43.636 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:47:43.638 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:47:43.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:43.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:43 smithi203 ceph-mon[32237]: pgmap v895: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.8 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 29 MiB/s, 467 objects/s recovering 2024-09-18T07:47:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:43 smithi203 ceph-mon[32237]: 6.a scrub starts 2024-09-18T07:47:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:43 smithi203 ceph-mon[32237]: 6.a scrub ok 2024-09-18T07:47:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:43 smithi203 ceph-mon[32237]: 3.b scrub starts 2024-09-18T07:47:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:43 smithi203 ceph-mon[32237]: 3.b scrub ok 2024-09-18T07:47:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:43 smithi203 ceph-mon[32237]: 6.3 scrub starts 2024-09-18T07:47:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:43 smithi203 ceph-mon[32237]: 6.3 scrub ok 2024-09-18T07:47:43.714 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:43.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:43.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:43.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:43.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:43.858 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:43.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:43.959 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:43.986 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:43.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:43.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:44.047 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 29 15 44263 44248 95.3474 105.125 0.00644517 0.0104823 2024-09-18T07:47:44.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:47:44.088 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:44.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:44.121 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:44.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:44.219 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:47:44.219 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:47:44.219 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:47:44.219 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:47:44.220 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.220 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189709 10.3 10:c5ce4044:::benchmark_data_smithi137_151160_object53893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:47:44.220 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.383941+0000", 2024-09-18T07:47:44.220 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.83034607100001, 2024-09-18T07:47:44.220 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.754258172, 2024-09-18T07:47:44.220 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.220 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.220 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.220 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.220 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:44.220 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:44.220 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189709 2024-09-18T07:47:44.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.221 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.221 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.383941+0000", 2024-09-18T07:47:44.221 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.221 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.383941+0000", 2024-09-18T07:47:44.221 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.222 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.383945+0000", 2024-09-18T07:47:44.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8519999999999997e-06 2024-09-18T07:47:44.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.222 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.383955+0000", 2024-09-18T07:47:44.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4190000000000008e-06 2024-09-18T07:47:44.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.222 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.383974+0000", 2024-09-18T07:47:44.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9485000000000001e-05 2024-09-18T07:47:44.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.223 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.383991+0000", 2024-09-18T07:47:44.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7374999999999999e-05 2024-09-18T07:47:44.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.223 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.384053+0000", 2024-09-18T07:47:44.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1968e-05 2024-09-18T07:47:44.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.224 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.224 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.384063+0000", 2024-09-18T07:47:44.224 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2879999999999998e-06 2024-09-18T07:47:44.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.224 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.224 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.129734+0000", 2024-09-18T07:47:44.224 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.74567087600000004 2024-09-18T07:47:44.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.224 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.224 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.129843+0000", 2024-09-18T07:47:44.224 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010930900000000001 2024-09-18T07:47:44.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-09-18T07:47:44.225 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.129948+0000", 2024-09-18T07:47:44.225 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105058 2024-09-18T07:47:44.225 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.225 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.137595+0000", 2024-09-18T07:47:44.225 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00764688 2024-09-18T07:47:44.225 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138070+0000", 2024-09-18T07:47:44.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00047476000000000001 2024-09-18T07:47:44.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138119+0000", 2024-09-18T07:47:44.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9376000000000003e-05 2024-09-18T07:47:44.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138199+0000", 2024-09-18T07:47:44.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0525999999999994e-05 2024-09-18T07:47:44.226 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.227 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.227 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.227 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189712 10.3 10:c64415f1:::benchmark_data_smithi137_151160_object53896:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:47:44.227 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.389392+0000", 2024-09-18T07:47:44.227 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.824895317, 2024-09-18T07:47:44.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.75091792300000004, 2024-09-18T07:47:44.227 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.227 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.227 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.227 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.227 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:44.228 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:44.228 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189712 2024-09-18T07:47:44.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.228 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.228 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.228 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.389392+0000", 2024-09-18T07:47:44.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.228 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.228 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.389392+0000", 2024-09-18T07:47:44.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.229 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.229 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.389394+0000", 2024-09-18T07:47:44.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3300000000000001e-06 2024-09-18T07:47:44.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.229 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.229 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.389401+0000", 2024-09-18T07:47:44.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9920000000000001e-06 2024-09-18T07:47:44.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.229 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.229 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.389416+0000", 2024-09-18T07:47:44.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4644000000000001e-05 2024-09-18T07:47:44.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.389424+0000", 2024-09-18T07:47:44.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2320000000000001e-06 2024-09-18T07:47:44.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.389493+0000", 2024-09-18T07:47:44.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8683000000000006e-05 2024-09-18T07:47:44.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.389502+0000", 2024-09-18T07:47:44.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5580000000000006e-06 2024-09-18T07:47:44.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.134246+0000", 2024-09-18T07:47:44.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.74474385700000001 2024-09-18T07:47:44.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.134365+0000", 2024-09-18T07:47:44.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118989 2024-09-18T07:47:44.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-09-18T07:47:44.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.134496+0000", 2024-09-18T07:47:44.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000130529 2024-09-18T07:47:44.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138351+0000", 2024-09-18T07:47:44.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038556290000000002 2024-09-18T07:47:44.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.140058+0000", 2024-09-18T07:47:44.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017069229999999999 2024-09-18T07:47:44.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.140225+0000", 2024-09-18T07:47:44.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000166188 2024-09-18T07:47:44.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.140310+0000", 2024-09-18T07:47:44.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5369000000000002e-05 2024-09-18T07:47:44.233 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.233 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.233 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.234 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189716 10.9 10:9d240f9e:::benchmark_data_smithi137_151160_object53900:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:47:44.234 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:44.235 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.395506+0000", 2024-09-18T07:47:44.235 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.81878097200001, 2024-09-18T07:47:44.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.72649347500000006, 2024-09-18T07:47:44.236 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.236 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.236 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.236 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.236 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:44.236 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:44.236 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189716 2024-09-18T07:47:44.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.236 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.395506+0000", 2024-09-18T07:47:44.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.395506+0000", 2024-09-18T07:47:44.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.395509+0000", 2024-09-18T07:47:44.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.418e-06 2024-09-18T07:47:44.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.238 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.238 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.395516+0000", 2024-09-18T07:47:44.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7710000000000001e-06 2024-09-18T07:47:44.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.238 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.238 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.238 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.395528+0000", 2024-09-18T07:47:44.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2527999999999999e-05 2024-09-18T07:47:44.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.238 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.238 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.238 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.395536+0000", 2024-09-18T07:47:44.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2689999999999999e-06 2024-09-18T07:47:44.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.395598+0000", 2024-09-18T07:47:44.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1791999999999994e-05 2024-09-18T07:47:44.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.395613+0000", 2024-09-18T07:47:44.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5148e-05 2024-09-18T07:47:44.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.239 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.111722+0000", 2024-09-18T07:47:44.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71610891700000001 2024-09-18T07:47:44.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.114924+0000", 2024-09-18T07:47:44.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032013660000000002 2024-09-18T07:47:44.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 7", 2024-09-18T07:47:44.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.115068+0000", 2024-09-18T07:47:44.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000143962 2024-09-18T07:47:44.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.121832+0000", 2024-09-18T07:47:44.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0067644050000000002 2024-09-18T07:47:44.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.121847+0000", 2024-09-18T07:47:44.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5233e-05 2024-09-18T07:47:44.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.121899+0000", 2024-09-18T07:47:44.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1514000000000002e-05 2024-09-18T07:47:44.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.122000+0000", 2024-09-18T07:47:44.242 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101152 2024-09-18T07:47:44.242 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.242 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.242 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.242 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189717 10.9 10:91070f94:::benchmark_data_smithi137_151160_object53901:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:47:44.242 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.397857+0000", 2024-09-18T07:47:44.242 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.81642991199999, 2024-09-18T07:47:44.242 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.73002420099999998, 2024-09-18T07:47:44.243 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.243 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.243 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.243 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.243 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:44.243 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:44.243 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189717 2024-09-18T07:47:44.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.243 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.243 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.243 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.243 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.397857+0000", 2024-09-18T07:47:44.243 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.397857+0000", 2024-09-18T07:47:44.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.397860+0000", 2024-09-18T07:47:44.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7920000000000001e-06 2024-09-18T07:47:44.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.397867+0000", 2024-09-18T07:47:44.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1810000000000004e-06 2024-09-18T07:47:44.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.397883+0000", 2024-09-18T07:47:44.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5341000000000001e-05 2024-09-18T07:47:44.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.397892+0000", 2024-09-18T07:47:44.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9029999999999993e-06 2024-09-18T07:47:44.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.397952+0000", 2024-09-18T07:47:44.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0202999999999999e-05 2024-09-18T07:47:44.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.397962+0000", 2024-09-18T07:47:44.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0122999999999999e-05 2024-09-18T07:47:44.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.118561+0000", 2024-09-18T07:47:44.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.720598658 2024-09-18T07:47:44.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.119244+0000", 2024-09-18T07:47:44.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00068316899999999996 2024-09-18T07:47:44.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 7", 2024-09-18T07:47:44.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.119368+0000", 2024-09-18T07:47:44.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000124487 2024-09-18T07:47:44.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.127642+0000", 2024-09-18T07:47:44.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082739949999999993 2024-09-18T07:47:44.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.127683+0000", 2024-09-18T07:47:44.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0343000000000003e-05 2024-09-18T07:47:44.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.127737+0000", 2024-09-18T07:47:44.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3942e-05 2024-09-18T07:47:44.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.127882+0000", 2024-09-18T07:47:44.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014506400000000001 2024-09-18T07:47:44.249 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.249 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.249 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.249 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189727 10.9 10:996536d9:::benchmark_data_smithi137_151160_object53911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:47:44.249 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.413380+0000", 2024-09-18T07:47:44.249 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.80090682900001, 2024-09-18T07:47:44.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71477512799999998, 2024-09-18T07:47:44.249 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.249 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.249 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.250 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.250 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:44.250 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:44.250 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189727 2024-09-18T07:47:44.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.250 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.413380+0000", 2024-09-18T07:47:44.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.413380+0000", 2024-09-18T07:47:44.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.413383+0000", 2024-09-18T07:47:44.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9009999999999998e-06 2024-09-18T07:47:44.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.413390+0000", 2024-09-18T07:47:44.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7120000000000003e-06 2024-09-18T07:47:44.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.413403+0000", 2024-09-18T07:47:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3043e-05 2024-09-18T07:47:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.413411+0000", 2024-09-18T07:47:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7300000000000005e-06 2024-09-18T07:47:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.413477+0000", 2024-09-18T07:47:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5900000000000003e-05 2024-09-18T07:47:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.413486+0000", 2024-09-18T07:47:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9169999999999995e-06 2024-09-18T07:47:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.122038+0000", 2024-09-18T07:47:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.70855256 2024-09-18T07:47:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.122161+0000", 2024-09-18T07:47:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012317299999999999 2024-09-18T07:47:44.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 7", 2024-09-18T07:47:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.122410+0000", 2024-09-18T07:47:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024848099999999999 2024-09-18T07:47:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.127938+0000", 2024-09-18T07:47:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055279839999999997 2024-09-18T07:47:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.127949+0000", 2024-09-18T07:47:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1433999999999999e-05 2024-09-18T07:47:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.127999+0000", 2024-09-18T07:47:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9249000000000002e-05 2024-09-18T07:47:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.128156+0000", 2024-09-18T07:47:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015704400000000001 2024-09-18T07:47:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.255 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.256 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189735 10.9 10:99a96842:::benchmark_data_smithi137_151160_object53919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:47:44.256 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.418652+0000", 2024-09-18T07:47:44.256 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.79563505600001, 2024-09-18T07:47:44.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71745237399999995, 2024-09-18T07:47:44.256 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.256 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.256 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.256 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.256 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:44.256 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:44.256 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189735 2024-09-18T07:47:44.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.257 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.418652+0000", 2024-09-18T07:47:44.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.418652+0000", 2024-09-18T07:47:44.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.418655+0000", 2024-09-18T07:47:44.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6649999999999999e-06 2024-09-18T07:47:44.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.418661+0000", 2024-09-18T07:47:44.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4139999999999999e-06 2024-09-18T07:47:44.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.418674+0000", 2024-09-18T07:47:44.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2490999999999999e-05 2024-09-18T07:47:44.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.418682+0000", 2024-09-18T07:47:44.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2490000000000001e-06 2024-09-18T07:47:44.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.418743+0000", 2024-09-18T07:47:44.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0599e-05 2024-09-18T07:47:44.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.418752+0000", 2024-09-18T07:47:44.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9460000000000004e-06 2024-09-18T07:47:44.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.125065+0000", 2024-09-18T07:47:44.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.70631298499999995 2024-09-18T07:47:44.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.125182+0000", 2024-09-18T07:47:44.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117773 2024-09-18T07:47:44.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 7", 2024-09-18T07:47:44.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.125284+0000", 2024-09-18T07:47:44.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101947 2024-09-18T07:47:44.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.128163+0000", 2024-09-18T07:47:44.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028788239999999999 2024-09-18T07:47:44.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.135769+0000", 2024-09-18T07:47:44.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076063349999999997 2024-09-18T07:47:44.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.135858+0000", 2024-09-18T07:47:44.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8555999999999997e-05 2024-09-18T07:47:44.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.136105+0000", 2024-09-18T07:47:44.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024658999999999998 2024-09-18T07:47:44.262 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.262 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.262 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.262 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189741 10.9 10:948cd666:::benchmark_data_smithi137_151160_object53925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:47:44.262 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.425791+0000", 2024-09-18T07:47:44.262 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.78849612400001, 2024-09-18T07:47:44.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71069487499999995, 2024-09-18T07:47:44.263 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.263 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.263 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.263 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.263 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:47:44.263 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:47:44.263 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189741 2024-09-18T07:47:44.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.263 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.425791+0000", 2024-09-18T07:47:44.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.425791+0000", 2024-09-18T07:47:44.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.425793+0000", 2024-09-18T07:47:44.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.007e-06 2024-09-18T07:47:44.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.425800+0000", 2024-09-18T07:47:44.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.759e-06 2024-09-18T07:47:44.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.425811+0000", 2024-09-18T07:47:44.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1435e-05 2024-09-18T07:47:44.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.425818+0000", 2024-09-18T07:47:44.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1550000000000004e-06 2024-09-18T07:47:44.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.425880+0000", 2024-09-18T07:47:44.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2362999999999999e-05 2024-09-18T07:47:44.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.425891+0000", 2024-09-18T07:47:44.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1022000000000001e-05 2024-09-18T07:47:44.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.128192+0000", 2024-09-18T07:47:44.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.70230154 2024-09-18T07:47:44.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.128293+0000", 2024-09-18T07:47:44.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100473 2024-09-18T07:47:44.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 7", 2024-09-18T07:47:44.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.128405+0000", 2024-09-18T07:47:44.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011176200000000001 2024-09-18T07:47:44.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.133495+0000", 2024-09-18T07:47:44.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050899860000000003 2024-09-18T07:47:44.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.136143+0000", 2024-09-18T07:47:44.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002648114 2024-09-18T07:47:44.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.136328+0000", 2024-09-18T07:47:44.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018492199999999999 2024-09-18T07:47:44.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.136486+0000", 2024-09-18T07:47:44.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015833700000000001 2024-09-18T07:47:44.269 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.269 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.269 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.269 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:28725 11.c 11:3119780a:::benchmark_data_smithi137_181069_object28724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-09-18T07:47:44.269 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:30.111723+0000", 2024-09-18T07:47:44.269 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 14.102564359, 2024-09-18T07:47:44.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.019158674, 2024-09-18T07:47:44.269 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.269 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.269 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:47:44.269 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.270 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:47:44.270 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:47:44.270 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 28725 2024-09-18T07:47:44.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.270 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.111723+0000", 2024-09-18T07:47:44.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.111723+0000", 2024-09-18T07:47:44.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.111725+0000", 2024-09-18T07:47:44.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9410000000000001e-06 2024-09-18T07:47:44.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.111756+0000", 2024-09-18T07:47:44.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0599000000000003e-05 2024-09-18T07:47:44.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.111767+0000", 2024-09-18T07:47:44.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1630999999999999e-05 2024-09-18T07:47:44.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.111775+0000", 2024-09-18T07:47:44.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5989999999999996e-06 2024-09-18T07:47:44.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.112132+0000", 2024-09-18T07:47:44.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000357709 2024-09-18T07:47:44.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.112272+0000", 2024-09-18T07:47:44.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000139765 2024-09-18T07:47:44.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:47:44.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.112607+0000", 2024-09-18T07:47:44.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00033469899999999999 2024-09-18T07:47:44.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.117704+0000", 2024-09-18T07:47:44.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050970080000000001 2024-09-18T07:47:44.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.117728+0000", 2024-09-18T07:47:44.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4360999999999999e-05 2024-09-18T07:47:44.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.123700+0000", 2024-09-18T07:47:44.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0059719259999999998 2024-09-18T07:47:44.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.123712+0000", 2024-09-18T07:47:44.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1334e-05 2024-09-18T07:47:44.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.130678+0000", 2024-09-18T07:47:44.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0069661969999999 2024-09-18T07:47:44.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.130882+0000", 2024-09-18T07:47:44.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020390499999999999 2024-09-18T07:47:44.275 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.275 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.275 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.276 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:28743 11.c 11:3fdafd2c:::benchmark_data_smithi137_181069_object28742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-18T07:47:44.276 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:30.121880+0000", 2024-09-18T07:47:44.276 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 14.092407578, 2024-09-18T07:47:44.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.016381838, 2024-09-18T07:47:44.276 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.276 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.276 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.276 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.276 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:47:44.276 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:47:44.276 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 28743 2024-09-18T07:47:44.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.276 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.121880+0000", 2024-09-18T07:47:44.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.121880+0000", 2024-09-18T07:47:44.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.121882+0000", 2024-09-18T07:47:44.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4710000000000001e-06 2024-09-18T07:47:44.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.121908+0000", 2024-09-18T07:47:44.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5420999999999999e-05 2024-09-18T07:47:44.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.123043+0000", 2024-09-18T07:47:44.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001135853 2024-09-18T07:47:44.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.123057+0000", 2024-09-18T07:47:44.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3079e-05 2024-09-18T07:47:44.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.124180+0000", 2024-09-18T07:47:44.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011232620000000001 2024-09-18T07:47:44.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.124189+0000", 2024-09-18T07:47:44.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8389999999999996e-06 2024-09-18T07:47:44.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.131702+0000", 2024-09-18T07:47:44.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0075134400000001 2024-09-18T07:47:44.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.131929+0000", 2024-09-18T07:47:44.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022657499999999999 2024-09-18T07:47:44.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:47:44.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.132077+0000", 2024-09-18T07:47:44.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000148364 2024-09-18T07:47:44.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.135109+0000", 2024-09-18T07:47:44.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003032313 2024-09-18T07:47:44.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138088+0000", 2024-09-18T07:47:44.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029782160000000001 2024-09-18T07:47:44.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138192+0000", 2024-09-18T07:47:44.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104498 2024-09-18T07:47:44.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138234+0000", 2024-09-18T07:47:44.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1717000000000001e-05 2024-09-18T07:47:44.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138262+0000", 2024-09-18T07:47:44.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.779e-05 2024-09-18T07:47:44.284 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.284 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.284 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.284 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:28764 11.c 11:3ed26a0f:::benchmark_data_smithi137_181069_object28763:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-18T07:47:44.284 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:30.139635+0000", 2024-09-18T07:47:44.284 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 14.074652216, 2024-09-18T07:47:44.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99880435199999995, 2024-09-18T07:47:44.284 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.285 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.285 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.285 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.285 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:47:44.285 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:47:44.285 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 28764 2024-09-18T07:47:44.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.285 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.139635+0000", 2024-09-18T07:47:44.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.139635+0000", 2024-09-18T07:47:44.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.139637+0000", 2024-09-18T07:47:44.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9089999999999998e-06 2024-09-18T07:47:44.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.139660+0000", 2024-09-18T07:47:44.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2668e-05 2024-09-18T07:47:44.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.139673+0000", 2024-09-18T07:47:44.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3497e-05 2024-09-18T07:47:44.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.139680+0000", 2024-09-18T07:47:44.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5830000000000003e-06 2024-09-18T07:47:44.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.139735+0000", 2024-09-18T07:47:44.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5628999999999999e-05 2024-09-18T07:47:44.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.139743+0000", 2024-09-18T07:47:44.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0460000000000006e-06 2024-09-18T07:47:44.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.133867+0000", 2024-09-18T07:47:44.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99412396000000003 2024-09-18T07:47:44.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.134003+0000", 2024-09-18T07:47:44.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013544300000000001 2024-09-18T07:47:44.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:47:44.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.134121+0000", 2024-09-18T07:47:44.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118528 2024-09-18T07:47:44.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138000+0000", 2024-09-18T07:47:44.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038784480000000001 2024-09-18T07:47:44.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138271+0000", 2024-09-18T07:47:44.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027133399999999997 2024-09-18T07:47:44.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138349+0000", 2024-09-18T07:47:44.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8020000000000002e-05 2024-09-18T07:47:44.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138402+0000", 2024-09-18T07:47:44.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2491999999999998e-05 2024-09-18T07:47:44.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138439+0000", 2024-09-18T07:47:44.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7795e-05 2024-09-18T07:47:44.291 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.291 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.291 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.291 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:28776 11.c 11:33c7dc59:::benchmark_data_smithi137_181069_object28775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-18T07:47:44.291 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:30.146668+0000", 2024-09-18T07:47:44.291 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 14.067619234, 2024-09-18T07:47:44.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99196297899999997, 2024-09-18T07:47:44.291 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.291 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.291 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.292 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.292 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:47:44.292 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:47:44.292 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 28776 2024-09-18T07:47:44.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.292 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.146668+0000", 2024-09-18T07:47:44.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.146668+0000", 2024-09-18T07:47:44.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.146670+0000", 2024-09-18T07:47:44.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2900000000000001e-06 2024-09-18T07:47:44.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.146778+0000", 2024-09-18T07:47:44.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010768300000000001 2024-09-18T07:47:44.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.146791+0000", 2024-09-18T07:47:44.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2636000000000001e-05 2024-09-18T07:47:44.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.146800+0000", 2024-09-18T07:47:44.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9139999999999997e-06 2024-09-18T07:47:44.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.147544+0000", 2024-09-18T07:47:44.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00074485800000000004 2024-09-18T07:47:44.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.147555+0000", 2024-09-18T07:47:44.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.066e-05 2024-09-18T07:47:44.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.135142+0000", 2024-09-18T07:47:44.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.987587349 2024-09-18T07:47:44.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.135250+0000", 2024-09-18T07:47:44.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010728 2024-09-18T07:47:44.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:47:44.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.135372+0000", 2024-09-18T07:47:44.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000122144 2024-09-18T07:47:44.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138277+0000", 2024-09-18T07:47:44.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002904808 2024-09-18T07:47:44.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138313+0000", 2024-09-18T07:47:44.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6179e-05 2024-09-18T07:47:44.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138545+0000", 2024-09-18T07:47:44.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023164799999999999 2024-09-18T07:47:44.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138595+0000", 2024-09-18T07:47:44.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0680000000000003e-05 2024-09-18T07:47:44.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138631+0000", 2024-09-18T07:47:44.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5849999999999997e-05 2024-09-18T07:47:44.298 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.298 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.298 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.298 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:28781 11.c 11:3b4a5081:::benchmark_data_smithi137_181069_object28780:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-18T07:47:44.298 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:30.152726+0000", 2024-09-18T07:47:44.298 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 14.061560922, 2024-09-18T07:47:44.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98602892499999995, 2024-09-18T07:47:44.298 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.298 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.298 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.299 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.299 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:47:44.299 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:47:44.299 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 28781 2024-09-18T07:47:44.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.299 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.152726+0000", 2024-09-18T07:47:44.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.152726+0000", 2024-09-18T07:47:44.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.152730+0000", 2024-09-18T07:47:44.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3349999999999998e-06 2024-09-18T07:47:44.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.152760+0000", 2024-09-18T07:47:44.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0417e-05 2024-09-18T07:47:44.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.152781+0000", 2024-09-18T07:47:44.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0681000000000001e-05 2024-09-18T07:47:44.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.152791+0000", 2024-09-18T07:47:44.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0339999999999999e-05 2024-09-18T07:47:44.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.153085+0000", 2024-09-18T07:47:44.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029371600000000002 2024-09-18T07:47:44.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.153096+0000", 2024-09-18T07:47:44.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1354e-05 2024-09-18T07:47:44.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.136498+0000", 2024-09-18T07:47:44.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98340159999999999 2024-09-18T07:47:44.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.136639+0000", 2024-09-18T07:47:44.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000141574 2024-09-18T07:47:44.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:47:44.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.136780+0000", 2024-09-18T07:47:44.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014075099999999999 2024-09-18T07:47:44.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138494+0000", 2024-09-18T07:47:44.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017134489999999999 2024-09-18T07:47:44.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138509+0000", 2024-09-18T07:47:44.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5129e-05 2024-09-18T07:47:44.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138674+0000", 2024-09-18T07:47:44.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000165339 2024-09-18T07:47:44.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138720+0000", 2024-09-18T07:47:44.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5420000000000002e-05 2024-09-18T07:47:44.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138755+0000", 2024-09-18T07:47:44.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5819999999999999e-05 2024-09-18T07:47:44.304 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.304 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.305 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.305 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:28782 11.c 11:3e2be750:::benchmark_data_smithi137_181069_object28781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-18T07:47:44.305 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:30.153173+0000", 2024-09-18T07:47:44.305 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 14.061114429, 2024-09-18T07:47:44.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99105674899999996, 2024-09-18T07:47:44.305 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.305 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.305 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.305 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.305 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:47:44.305 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:47:44.306 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 28782 2024-09-18T07:47:44.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.306 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.153173+0000", 2024-09-18T07:47:44.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.153173+0000", 2024-09-18T07:47:44.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.153174+0000", 2024-09-18T07:47:44.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4109999999999999e-06 2024-09-18T07:47:44.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.153200+0000", 2024-09-18T07:47:44.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5882000000000001e-05 2024-09-18T07:47:44.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.153215+0000", 2024-09-18T07:47:44.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4395e-05 2024-09-18T07:47:44.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.153222+0000", 2024-09-18T07:47:44.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4240000000000004e-06 2024-09-18T07:47:44.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.153280+0000", 2024-09-18T07:47:44.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7768e-05 2024-09-18T07:47:44.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.153287+0000", 2024-09-18T07:47:44.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0489999999999998e-06 2024-09-18T07:47:44.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138790+0000", 2024-09-18T07:47:44.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98550275499999995 2024-09-18T07:47:44.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138919+0000", 2024-09-18T07:47:44.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012959500000000001 2024-09-18T07:47:44.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:47:44.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.139044+0000", 2024-09-18T07:47:44.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012521200000000001 2024-09-18T07:47:44.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144117+0000", 2024-09-18T07:47:44.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005072684 2024-09-18T07:47:44.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144129+0000", 2024-09-18T07:47:44.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1731e-05 2024-09-18T07:47:44.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144149+0000", 2024-09-18T07:47:44.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0432999999999999e-05 2024-09-18T07:47:44.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144186+0000", 2024-09-18T07:47:44.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6928999999999998e-05 2024-09-18T07:47:44.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144230+0000", 2024-09-18T07:47:44.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3480999999999999e-05 2024-09-18T07:47:44.311 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.311 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.311 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.312 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:28841 11.c 11:3ffe0172:::benchmark_data_smithi137_181069_object28840:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-18T07:47:44.312 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:30.207150+0000", 2024-09-18T07:47:44.312 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 14.007137334999999, 2024-09-18T07:47:44.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93726686199999998, 2024-09-18T07:47:44.312 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.312 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.312 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.312 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.312 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:47:44.312 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:47:44.312 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 28841 2024-09-18T07:47:44.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.312 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.207150+0000", 2024-09-18T07:47:44.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.207150+0000", 2024-09-18T07:47:44.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.207152+0000", 2024-09-18T07:47:44.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9e-06 2024-09-18T07:47:44.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.207174+0000", 2024-09-18T07:47:44.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1999999999999999e-05 2024-09-18T07:47:44.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.207188+0000", 2024-09-18T07:47:44.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3917000000000001e-05 2024-09-18T07:47:44.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.207195+0000", 2024-09-18T07:47:44.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5390000000000002e-06 2024-09-18T07:47:44.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.207254+0000", 2024-09-18T07:47:44.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8430000000000001e-05 2024-09-18T07:47:44.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.207263+0000", 2024-09-18T07:47:44.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6020000000000006e-06 2024-09-18T07:47:44.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.140595+0000", 2024-09-18T07:47:44.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93333201899999996 2024-09-18T07:47:44.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.140710+0000", 2024-09-18T07:47:44.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114898 2024-09-18T07:47:44.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:47:44.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.141010+0000", 2024-09-18T07:47:44.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029964400000000001 2024-09-18T07:47:44.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144244+0000", 2024-09-18T07:47:44.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032338559999999998 2024-09-18T07:47:44.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144281+0000", 2024-09-18T07:47:44.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7088999999999997e-05 2024-09-18T07:47:44.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144332+0000", 2024-09-18T07:47:44.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1107999999999999e-05 2024-09-18T07:47:44.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144380+0000", 2024-09-18T07:47:44.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8467000000000002e-05 2024-09-18T07:47:44.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.318 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144417+0000", 2024-09-18T07:47:44.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6393000000000002e-05 2024-09-18T07:47:44.318 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.318 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.318 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.318 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:28872 11.c 11:3049cef6:::benchmark_data_smithi137_181069_object28871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-18T07:47:44.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:44.319 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:44.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:44.320 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:30.224860+0000", 2024-09-18T07:47:44.320 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.989427812000001, 2024-09-18T07:47:44.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.92334746400000001, 2024-09-18T07:47:44.320 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.320 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.320 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.320 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.321 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:47:44.321 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:47:44.321 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 28872 2024-09-18T07:47:44.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.321 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.224860+0000", 2024-09-18T07:47:44.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.224860+0000", 2024-09-18T07:47:44.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.224862+0000", 2024-09-18T07:47:44.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.063e-06 2024-09-18T07:47:44.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.224888+0000", 2024-09-18T07:47:44.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6594e-05 2024-09-18T07:47:44.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.224899+0000", 2024-09-18T07:47:44.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1161e-05 2024-09-18T07:47:44.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.224914+0000", 2024-09-18T07:47:44.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4577e-05 2024-09-18T07:47:44.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.224966+0000", 2024-09-18T07:47:44.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1897999999999997e-05 2024-09-18T07:47:44.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.224976+0000", 2024-09-18T07:47:44.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0163000000000001e-05 2024-09-18T07:47:44.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.142547+0000", 2024-09-18T07:47:44.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91757123200000001 2024-09-18T07:47:44.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.142696+0000", 2024-09-18T07:47:44.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014891200000000001 2024-09-18T07:47:44.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:47:44.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.142849+0000", 2024-09-18T07:47:44.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000152869 2024-09-18T07:47:44.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144292+0000", 2024-09-18T07:47:44.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014434980000000001 2024-09-18T07:47:44.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148017+0000", 2024-09-18T07:47:44.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0037241739999999998 2024-09-18T07:47:44.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148102+0000", 2024-09-18T07:47:44.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5428999999999998e-05 2024-09-18T07:47:44.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148157+0000", 2024-09-18T07:47:44.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5353999999999999e-05 2024-09-18T07:47:44.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148207+0000", 2024-09-18T07:47:44.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9539999999999997e-05 2024-09-18T07:47:44.327 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.327 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.327 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.327 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:28876 11.c 11:3482938c:::benchmark_data_smithi137_181069_object28875:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-18T07:47:44.327 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:30.225802+0000", 2024-09-18T07:47:44.327 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.98848516, 2024-09-18T07:47:44.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.92268435299999996, 2024-09-18T07:47:44.327 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.327 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.327 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.327 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.327 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:47:44.328 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:47:44.328 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 28876 2024-09-18T07:47:44.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.328 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.225802+0000", 2024-09-18T07:47:44.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.225802+0000", 2024-09-18T07:47:44.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.225804+0000", 2024-09-18T07:47:44.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.482e-06 2024-09-18T07:47:44.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.225826+0000", 2024-09-18T07:47:44.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2379e-05 2024-09-18T07:47:44.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.225837+0000", 2024-09-18T07:47:44.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0574000000000001e-05 2024-09-18T07:47:44.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.225844+0000", 2024-09-18T07:47:44.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8660000000000006e-06 2024-09-18T07:47:44.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.225898+0000", 2024-09-18T07:47:44.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3165999999999999e-05 2024-09-18T07:47:44.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.225906+0000", 2024-09-18T07:47:44.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1580000000000005e-06 2024-09-18T07:47:44.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144466+0000", 2024-09-18T07:47:44.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91856010799999999 2024-09-18T07:47:44.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144594+0000", 2024-09-18T07:47:44.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012782599999999999 2024-09-18T07:47:44.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:47:44.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144789+0000", 2024-09-18T07:47:44.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000195303 2024-09-18T07:47:44.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148032+0000", 2024-09-18T07:47:44.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032432419999999999 2024-09-18T07:47:44.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148067+0000", 2024-09-18T07:47:44.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4644999999999998e-05 2024-09-18T07:47:44.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148379+0000", 2024-09-18T07:47:44.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00031204200000000001 2024-09-18T07:47:44.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148429+0000", 2024-09-18T07:47:44.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9972999999999997e-05 2024-09-18T07:47:44.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148487+0000", 2024-09-18T07:47:44.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7589e-05 2024-09-18T07:47:44.334 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.334 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.334 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.334 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:28902 11.c 11:353c1da3:::benchmark_data_smithi137_181069_object28901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-18T07:47:44.334 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:30.282616+0000", 2024-09-18T07:47:44.334 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.931671406, 2024-09-18T07:47:44.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.86601945000000002, 2024-09-18T07:47:44.336 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.336 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.336 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.336 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.337 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:47:44.337 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:47:44.337 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 28902 2024-09-18T07:47:44.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.337 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.282616+0000", 2024-09-18T07:47:44.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.282616+0000", 2024-09-18T07:47:44.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.282619+0000", 2024-09-18T07:47:44.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.332e-06 2024-09-18T07:47:44.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.282648+0000", 2024-09-18T07:47:44.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8665000000000001e-05 2024-09-18T07:47:44.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.282664+0000", 2024-09-18T07:47:44.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6124999999999999e-05 2024-09-18T07:47:44.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.282674+0000", 2024-09-18T07:47:44.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0441e-05 2024-09-18T07:47:44.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.282745+0000", 2024-09-18T07:47:44.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0278999999999996e-05 2024-09-18T07:47:44.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.282752+0000", 2024-09-18T07:47:44.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0879999999999997e-06 2024-09-18T07:47:44.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.146241+0000", 2024-09-18T07:47:44.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.86348948999999997 2024-09-18T07:47:44.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.146534+0000", 2024-09-18T07:47:44.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000292517 2024-09-18T07:47:44.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:47:44.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.146730+0000", 2024-09-18T07:47:44.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019661 2024-09-18T07:47:44.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148221+0000", 2024-09-18T07:47:44.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001490395 2024-09-18T07:47:44.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148283+0000", 2024-09-18T07:47:44.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2520000000000004e-05 2024-09-18T07:47:44.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148529+0000", 2024-09-18T07:47:44.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024527199999999999 2024-09-18T07:47:44.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148596+0000", 2024-09-18T07:47:44.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7624000000000002e-05 2024-09-18T07:47:44.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148635+0000", 2024-09-18T07:47:44.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9091999999999998e-05 2024-09-18T07:47:44.344 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.344 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.344 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.344 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:28923 11.c 11:3188b8d2:::benchmark_data_smithi137_181069_object28922:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-18T07:47:44.344 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:30.467163+0000", 2024-09-18T07:47:44.344 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.747124600999999, 2024-09-18T07:47:44.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.68975161600000001, 2024-09-18T07:47:44.344 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.344 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.344 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.345 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.345 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:47:44.345 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:47:44.345 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 28923 2024-09-18T07:47:44.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.345 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.467163+0000", 2024-09-18T07:47:44.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.467163+0000", 2024-09-18T07:47:44.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.467165+0000", 2024-09-18T07:47:44.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.39e-06 2024-09-18T07:47:44.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.467191+0000", 2024-09-18T07:47:44.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5984000000000001e-05 2024-09-18T07:47:44.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.467209+0000", 2024-09-18T07:47:44.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7757e-05 2024-09-18T07:47:44.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.467218+0000", 2024-09-18T07:47:44.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2189999999999993e-06 2024-09-18T07:47:44.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.467286+0000", 2024-09-18T07:47:44.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7508999999999993e-05 2024-09-18T07:47:44.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.467295+0000", 2024-09-18T07:47:44.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9220000000000003e-06 2024-09-18T07:47:44.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148667+0000", 2024-09-18T07:47:44.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.68137276800000002 2024-09-18T07:47:44.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148802+0000", 2024-09-18T07:47:44.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013487699999999999 2024-09-18T07:47:44.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:47:44.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148906+0000", 2024-09-18T07:47:44.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104002 2024-09-18T07:47:44.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.153679+0000", 2024-09-18T07:47:44.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047723679999999999 2024-09-18T07:47:44.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.153746+0000", 2024-09-18T07:47:44.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7160000000000001e-05 2024-09-18T07:47:44.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.156821+0000", 2024-09-18T07:47:44.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030749839999999998 2024-09-18T07:47:44.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.156869+0000", 2024-09-18T07:47:44.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7982000000000002e-05 2024-09-18T07:47:44.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.156914+0000", 2024-09-18T07:47:44.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5694e-05 2024-09-18T07:47:44.351 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.351 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.351 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.351 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:28931 11.c 11:3b181186:::benchmark_data_smithi137_181069_object28930:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-18T07:47:44.351 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:30.472238+0000", 2024-09-18T07:47:44.351 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.742049096000001, 2024-09-18T07:47:44.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.68480135399999997, 2024-09-18T07:47:44.351 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.351 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.351 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.351 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.351 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:47:44.352 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:47:44.352 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 28931 2024-09-18T07:47:44.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.352 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.472238+0000", 2024-09-18T07:47:44.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.472238+0000", 2024-09-18T07:47:44.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.472242+0000", 2024-09-18T07:47:44.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3079999999999999e-06 2024-09-18T07:47:44.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.472265+0000", 2024-09-18T07:47:44.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3530999999999999e-05 2024-09-18T07:47:44.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.472282+0000", 2024-09-18T07:47:44.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6484999999999999e-05 2024-09-18T07:47:44.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.472291+0000", 2024-09-18T07:47:44.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2189999999999993e-06 2024-09-18T07:47:44.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.472349+0000", 2024-09-18T07:47:44.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8545999999999998e-05 2024-09-18T07:47:44.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.472357+0000", 2024-09-18T07:47:44.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0139999999999998e-06 2024-09-18T07:47:44.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.150174+0000", 2024-09-18T07:47:44.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.67781644699999999 2024-09-18T07:47:44.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.150291+0000", 2024-09-18T07:47:44.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011728 2024-09-18T07:47:44.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:47:44.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.150420+0000", 2024-09-18T07:47:44.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012847800000000001 2024-09-18T07:47:44.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.153713+0000", 2024-09-18T07:47:44.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003293734 2024-09-18T07:47:44.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.156732+0000", 2024-09-18T07:47:44.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030186340000000001 2024-09-18T07:47:44.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.156921+0000", 2024-09-18T07:47:44.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018935499999999999 2024-09-18T07:47:44.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157004+0000", 2024-09-18T07:47:44.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2981999999999999e-05 2024-09-18T07:47:44.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157040+0000", 2024-09-18T07:47:44.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5340999999999999e-05 2024-09-18T07:47:44.357 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.357 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.358 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.358 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:28940 11.c 11:31fb142f:::benchmark_data_smithi137_181069_object28939:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-18T07:47:44.358 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:30.478716+0000", 2024-09-18T07:47:44.358 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.735571112000001, 2024-09-18T07:47:44.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.67843525900000001, 2024-09-18T07:47:44.358 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:47:44.358 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:47:44.358 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:47:44.358 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:47:44.358 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:47:44.359 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:47:44.359 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 28940 2024-09-18T07:47:44.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.359 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:47:44.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:47:44.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.478716+0000", 2024-09-18T07:47:44.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:47:44.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.478716+0000", 2024-09-18T07:47:44.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:47:44.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:47:44.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.478718+0000", 2024-09-18T07:47:44.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7519999999999999e-06 2024-09-18T07:47:44.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:47:44.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.478742+0000", 2024-09-18T07:47:44.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4009999999999999e-05 2024-09-18T07:47:44.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:47:44.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.478753+0000", 2024-09-18T07:47:44.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.132e-05 2024-09-18T07:47:44.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:47:44.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.478759+0000", 2024-09-18T07:47:44.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6110000000000001e-06 2024-09-18T07:47:44.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.478806+0000", 2024-09-18T07:47:44.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6628999999999997e-05 2024-09-18T07:47:44.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:47:44.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:30.478813+0000", 2024-09-18T07:47:44.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1049999999999997e-06 2024-09-18T07:47:44.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:47:44.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.151742+0000", 2024-09-18T07:47:44.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.67292918000000002 2024-09-18T07:47:44.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:47:44.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.151853+0000", 2024-09-18T07:47:44.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011134299999999999 2024-09-18T07:47:44.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:47:44.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.151977+0000", 2024-09-18T07:47:44.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012393500000000001 2024-09-18T07:47:44.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.153778+0000", 2024-09-18T07:47:44.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00180127 2024-09-18T07:47:44.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:47:44.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.156811+0000", 2024-09-18T07:47:44.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030327660000000001 2024-09-18T07:47:44.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:47:44.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157046+0000", 2024-09-18T07:47:44.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00023473299999999999 2024-09-18T07:47:44.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:47:44.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157120+0000", 2024-09-18T07:47:44.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4399000000000004e-05 2024-09-18T07:47:44.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:47:44.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:47:44.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:47:44.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157151+0000", 2024-09-18T07:47:44.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1205999999999999e-05 2024-09-18T07:47:44.365 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.365 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.365 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.365 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:47:44.365 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:47:44.365 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:47:44.420 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:44.448 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:44.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:44.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:44.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:44.588 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:44.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:44.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:47:44.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:44.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:44.731 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:44.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:44.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:44.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:44.875 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:44.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:44.976 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:45.007 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:45.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:45.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:45.047 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 30 16 45877 45861 95.5285 100.812 0.00730348 0.0104625 2024-09-18T07:47:45.109 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:45.138 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:45.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:45.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:45.143 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":895,"stamp":"2024-09-18T07:47:41.829443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2473073179,"num_objects":37926,"num_object_clones":0,"num_object_copies":113778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37926,"num_whiteouts":0,"num_read":1880,"num_read_kb":1632,"num_write":38449,"num_write_kb":2414573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7218,"num_bytes_recovered":441463635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39402,"ondisk_log_size":39402,"up":474,"acting":473,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":484,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13013416,"kb_used_data":8616516,"kb_used_omap":20,"kb_used_meta":4396715,"kb_avail":736947800,"statfs":{"total":767960285184,"available":754634547200,"internally_reserved":0,"allocated":8823312384,"data_stored":8816869722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20886,"internal_metadata":4502236778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":550109184,"num_objects":8394,"num_object_clones":0,"num_object_copies":25182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1676,"num_objects_misplaced":-960,"num_objects_unfound":0,"num_objects_dirty":8394,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8394,"num_write_kb":537216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2636,"num_bytes_recovered":172752896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.633842"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":895,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-18T07:47:37.200574+0000","last_change":"2024-09-18T07:47:12.891849+0000","last_active":"2024-09-18T07:47:37.200574+0000","last_peered":"2024-09-18T07:47:37.200574+0000","last_clean":"2024-09-18T07:47:37.200574+0000","last_became_active":"2024-09-18T07:47:11.200534+0000","last_became_peered":"2024-09-18T07:47:11.200534+0000","last_unstale":"2024-09-18T07:47:37.200574+0000","last_undegraded":"2024-09-18T07:47:37.200574+0000","last_fullsized":"2024-09-18T07:47:37.200574+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:12.891802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:12.891802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:25.260952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"323'15","reported_seq":863,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.120971+0000","last_change":"2024-09-18T07:47:20.194767+0000","last_active":"2024-09-18T07:47:40.120971+0000","last_peered":"2024-09-18T07:47:40.120971+0000","last_clean":"2024-09-18T07:47:40.120971+0000","last_became_active":"2024-09-18T07:47:20.136254+0000","last_became_peered":"2024-09-18T07:47:20.136254+0000","last_unstale":"2024-09-18T07:47:40.120971+0000","last_undegraded":"2024-09-18T07:47:40.120971+0000","last_fullsized":"2024-09-18T07:47:40.120971+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:47:09.864550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.864550+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:14.398595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":686,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.233805+0000","last_change":"2024-09-18T07:47:20.130967+0000","last_active":"2024-09-18T07:47:38.233805+0000","last_peered":"2024-09-18T07:47:38.233805+0000","last_clean":"2024-09-18T07:47:38.233805+0000","last_became_active":"2024-09-18T07:47:20.130091+0000","last_became_peered":"2024-09-18T07:47:20.130091+0000","last_unstale":"2024-09-18T07:47:38.233805+0000","last_undegraded":"2024-09-18T07:47:38.233805+0000","last_fullsized":"2024-09-18T07:47:38.233805+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:11.422892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:44:11.422892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":742,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121350+0000","last_change":"2024-09-18T07:47:20.145483+0000","last_active":"2024-09-18T07:47:40.121350+0000","last_peered":"2024-09-18T07:47:40.121350+0000","last_clean":"2024-09-18T07:47:40.121350+0000","last_became_active":"2024-09-18T07:47:20.145155+0000","last_became_peered":"2024-09-18T07:47:20.145155+0000","last_unstale":"2024-09-18T07:47:40.121350+0000","last_undegraded":"2024-09-18T07:47:40.121350+0000","last_fullsized":"2024-09-18T07:47:40.121350+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:14.884604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:14.884604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:21.014582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":355,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133360+0000","last_change":"2024-09-18T07:46:29.320201+0000","last_active":"2024-09-18T07:47:40.133360+0000","last_peered":"2024-09-18T07:47:40.133360+0000","last_clean":"2024-09-18T07:47:40.133360+0000","last_became_active":"2024-09-18T07:46:29.319898+0000","last_became_peered":"2024-09-18T07:46:29.319898+0000","last_unstale":"2024-09-18T07:47:40.133360+0000","last_undegraded":"2024-09-18T07:47:40.133360+0000","last_fullsized":"2024-09-18T07:47:40.133360+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:11.545358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:11.545358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":888,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.139852+0000","last_change":"2024-09-18T07:45:25.982067+0000","last_active":"2024-09-18T07:47:40.139852+0000","last_peered":"2024-09-18T07:47:40.139852+0000","last_clean":"2024-09-18T07:47:40.139852+0000","last_became_active":"2024-09-18T07:37:43.066508+0000","last_became_peered":"2024-09-18T07:37:43.066508+0000","last_unstale":"2024-09-18T07:47:40.139852+0000","last_undegraded":"2024-09-18T07:47:40.139852+0000","last_fullsized":"2024-09-18T07:47:40.139852+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:25.981995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:45:25.981995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"323'27","reported_seq":449,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.135522+0000","last_change":"2024-09-18T07:46:54.956413+0000","last_active":"2024-09-18T07:47:40.135522+0000","last_peered":"2024-09-18T07:47:40.135522+0000","last_clean":"2024-09-18T07:47:40.135522+0000","last_became_active":"2024-09-18T07:40:19.064755+0000","last_became_peered":"2024-09-18T07:40:19.064755+0000","last_unstale":"2024-09-18T07:47:40.135522+0000","last_undegraded":"2024-09-18T07:47:40.135522+0000","last_fullsized":"2024-09-18T07:47:40.135522+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:46:54.956330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:54.956330+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:07.544925+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":804,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121016+0000","last_change":"2024-09-18T07:47:20.137049+0000","last_active":"2024-09-18T07:47:40.121016+0000","last_peered":"2024-09-18T07:47:40.121016+0000","last_clean":"2024-09-18T07:47:40.121016+0000","last_became_active":"2024-09-18T07:47:20.136358+0000","last_became_peered":"2024-09-18T07:47:20.136358+0000","last_unstale":"2024-09-18T07:47:40.121016+0000","last_undegraded":"2024-09-18T07:47:40.121016+0000","last_fullsized":"2024-09-18T07:47:40.121016+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:51.001046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:51.001046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":801,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234440+0000","last_change":"2024-09-18T07:47:11.202248+0000","last_active":"2024-09-18T07:47:38.234440+0000","last_peered":"2024-09-18T07:47:38.234440+0000","last_clean":"2024-09-18T07:47:38.234440+0000","last_became_active":"2024-09-18T07:47:11.200300+0000","last_became_peered":"2024-09-18T07:47:11.200300+0000","last_unstale":"2024-09-18T07:47:38.234440+0000","last_undegraded":"2024-09-18T07:47:38.234440+0000","last_fullsized":"2024-09-18T07:47:38.234440+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:09.526499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:45:09.526499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"323'15","reported_seq":801,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.120985+0000","last_change":"2024-09-18T07:44:43.775510+0000","last_active":"2024-09-18T07:47:40.120985+0000","last_peered":"2024-09-18T07:47:40.120985+0000","last_clean":"2024-09-18T07:47:40.120985+0000","last_became_active":"2024-09-18T07:44:43.775194+0000","last_became_peered":"2024-09-18T07:44:43.775194+0000","last_unstale":"2024-09-18T07:47:40.120985+0000","last_undegraded":"2024-09-18T07:47:40.120985+0000","last_fullsized":"2024-09-18T07:47:40.120985+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:56.021118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:56.021118+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":738,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.233875+0000","last_change":"2024-09-18T07:47:11.192327+0000","last_active":"2024-09-18T07:47:38.233875+0000","last_peered":"2024-09-18T07:47:38.233875+0000","last_clean":"2024-09-18T07:47:38.233875+0000","last_became_active":"2024-09-18T07:47:11.191975+0000","last_became_peered":"2024-09-18T07:47:11.191975+0000","last_unstale":"2024-09-18T07:47:38.233875+0000","last_undegraded":"2024-09-18T07:47:38.233875+0000","last_fullsized":"2024-09-18T07:47:38.233875+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:12.437116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:44:12.437116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":882,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-18T07:47:37.199617+0000","last_change":"2024-09-18T07:47:20.135642+0000","last_active":"2024-09-18T07:47:37.199617+0000","last_peered":"2024-09-18T07:47:37.199617+0000","last_clean":"2024-09-18T07:47:37.199617+0000","last_became_active":"2024-09-18T07:47:20.135282+0000","last_became_peered":"2024-09-18T07:47:20.135282+0000","last_unstale":"2024-09-18T07:47:37.199617+0000","last_undegraded":"2024-09-18T07:47:37.199617+0000","last_fullsized":"2024-09-18T07:47:37.199617+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:37.233589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:37.233589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:34.140753+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":760,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234532+0000","last_change":"2024-09-18T07:47:20.142711+0000","last_active":"2024-09-18T07:47:38.234532+0000","last_peered":"2024-09-18T07:47:38.234532+0000","last_clean":"2024-09-18T07:47:38.234532+0000","last_became_active":"2024-09-18T07:47:20.142404+0000","last_became_peered":"2024-09-18T07:47:20.142404+0000","last_unstale":"2024-09-18T07:47:38.234532+0000","last_undegraded":"2024-09-18T07:47:38.234532+0000","last_fullsized":"2024-09-18T07:47:38.234532+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:57.358853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:57.358853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"323'7","reported_seq":863,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121301+0000","last_change":"2024-09-18T07:47:20.150030+0000","last_active":"2024-09-18T07:47:40.121301+0000","last_peered":"2024-09-18T07:47:40.121301+0000","last_clean":"2024-09-18T07:47:40.121301+0000","last_became_active":"2024-09-18T07:47:20.135463+0000","last_became_peered":"2024-09-18T07:47:20.135463+0000","last_unstale":"2024-09-18T07:47:40.121301+0000","last_undegraded":"2024-09-18T07:47:40.121301+0000","last_fullsized":"2024-09-18T07:47:40.121301+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'7","last_scrub_stamp":"2024-09-18T07:42:46.496916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:46.496916+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":678,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.132818+0000","last_change":"2024-09-18T07:47:20.143380+0000","last_active":"2024-09-18T07:47:40.132818+0000","last_peered":"2024-09-18T07:47:40.132818+0000","last_clean":"2024-09-18T07:47:40.132818+0000","last_became_active":"2024-09-18T07:47:20.142891+0000","last_became_peered":"2024-09-18T07:47:20.142891+0000","last_unstale":"2024-09-18T07:47:40.132818+0000","last_undegraded":"2024-09-18T07:47:40.132818+0000","last_fullsized":"2024-09-18T07:47:40.132818+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:53.754622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:53.754622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:01.005011+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":788,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.233959+0000","last_change":"2024-09-18T07:47:20.134832+0000","last_active":"2024-09-18T07:47:38.233959+0000","last_peered":"2024-09-18T07:47:38.233959+0000","last_clean":"2024-09-18T07:47:38.233959+0000","last_became_active":"2024-09-18T07:47:20.134460+0000","last_became_peered":"2024-09-18T07:47:20.134460+0000","last_unstale":"2024-09-18T07:47:38.233959+0000","last_undegraded":"2024-09-18T07:47:38.233959+0000","last_fullsized":"2024-09-18T07:47:38.233959+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:01.522665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:01.522665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":828,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121612+0000","last_change":"2024-09-18T07:47:11.188757+0000","last_active":"2024-09-18T07:47:40.121612+0000","last_peered":"2024-09-18T07:47:40.121612+0000","last_clean":"2024-09-18T07:47:40.121612+0000","last_became_active":"2024-09-18T07:47:11.187916+0000","last_became_peered":"2024-09-18T07:47:11.187916+0000","last_unstale":"2024-09-18T07:47:40.121612+0000","last_undegraded":"2024-09-18T07:47:40.121612+0000","last_fullsized":"2024-09-18T07:47:40.121612+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:40.432906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:40.432906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":899,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-18T07:47:37.200674+0000","last_change":"2024-09-18T07:46:21.601224+0000","last_active":"2024-09-18T07:47:37.200674+0000","last_peered":"2024-09-18T07:47:37.200674+0000","last_clean":"2024-09-18T07:47:37.200674+0000","last_became_active":"2024-09-18T07:44:43.773339+0000","last_became_peered":"2024-09-18T07:44:43.773339+0000","last_unstale":"2024-09-18T07:47:37.200674+0000","last_undegraded":"2024-09-18T07:47:37.200674+0000","last_fullsized":"2024-09-18T07:47:37.200674+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:21.601160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:21.601160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"323'27","reported_seq":1029,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.136013+0000","last_change":"2024-09-18T07:47:20.183463+0000","last_active":"2024-09-18T07:47:40.136013+0000","last_peered":"2024-09-18T07:47:40.136013+0000","last_clean":"2024-09-18T07:47:40.136013+0000","last_became_active":"2024-09-18T07:47:20.140525+0000","last_became_peered":"2024-09-18T07:47:20.140525+0000","last_unstale":"2024-09-18T07:47:40.136013+0000","last_undegraded":"2024-09-18T07:47:40.136013+0000","last_fullsized":"2024-09-18T07:47:40.136013+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:45:53.825805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:45:53.825805+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:36.926196+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":466,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121624+0000","last_change":"2024-09-18T07:47:20.133202+0000","last_active":"2024-09-18T07:47:40.121624+0000","last_peered":"2024-09-18T07:47:40.121624+0000","last_clean":"2024-09-18T07:47:40.121624+0000","last_became_active":"2024-09-18T07:47:20.131623+0000","last_became_peered":"2024-09-18T07:47:20.131623+0000","last_unstale":"2024-09-18T07:47:40.121624+0000","last_undegraded":"2024-09-18T07:47:40.121624+0000","last_fullsized":"2024-09-18T07:47:40.121624+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:18.008882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:18.008882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":744,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.204912+0000","last_change":"2024-09-18T07:47:11.201194+0000","last_active":"2024-09-18T07:47:36.204912+0000","last_peered":"2024-09-18T07:47:36.204912+0000","last_clean":"2024-09-18T07:47:36.204912+0000","last_became_active":"2024-09-18T07:47:11.200897+0000","last_became_peered":"2024-09-18T07:47:11.200897+0000","last_unstale":"2024-09-18T07:47:36.204912+0000","last_undegraded":"2024-09-18T07:47:36.204912+0000","last_fullsized":"2024-09-18T07:47:36.204912+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:57.174827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:57.174827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":820,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121285+0000","last_change":"2024-09-18T07:47:20.144941+0000","last_active":"2024-09-18T07:47:40.121285+0000","last_peered":"2024-09-18T07:47:40.121285+0000","last_clean":"2024-09-18T07:47:40.121285+0000","last_became_active":"2024-09-18T07:47:20.144620+0000","last_became_peered":"2024-09-18T07:47:20.144620+0000","last_unstale":"2024-09-18T07:47:40.121285+0000","last_undegraded":"2024-09-18T07:47:40.121285+0000","last_fullsized":"2024-09-18T07:47:40.121285+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:31.323963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:31.323963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"323'18","reported_seq":602,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.135247+0000","last_change":"2024-09-18T07:47:20.181142+0000","last_active":"2024-09-18T07:47:40.135247+0000","last_peered":"2024-09-18T07:47:40.135247+0000","last_clean":"2024-09-18T07:47:40.135247+0000","last_became_active":"2024-09-18T07:47:20.139508+0000","last_became_peered":"2024-09-18T07:47:20.139508+0000","last_unstale":"2024-09-18T07:47:40.135247+0000","last_undegraded":"2024-09-18T07:47:40.135247+0000","last_fullsized":"2024-09-18T07:47:40.135247+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:46:47.978409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:47.978409+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:53.523262+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":742,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121697+0000","last_change":"2024-09-18T07:44:43.786027+0000","last_active":"2024-09-18T07:47:40.121697+0000","last_peered":"2024-09-18T07:47:40.121697+0000","last_clean":"2024-09-18T07:47:40.121697+0000","last_became_active":"2024-09-18T07:44:43.785095+0000","last_became_peered":"2024-09-18T07:44:43.785095+0000","last_unstale":"2024-09-18T07:47:40.121697+0000","last_undegraded":"2024-09-18T07:47:40.121697+0000","last_fullsized":"2024-09-18T07:47:40.121697+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.930435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.930435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:03.948295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":668,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133777+0000","last_change":"2024-09-18T07:47:11.195374+0000","last_active":"2024-09-18T07:47:40.133777+0000","last_peered":"2024-09-18T07:47:40.133777+0000","last_clean":"2024-09-18T07:47:40.133777+0000","last_became_active":"2024-09-18T07:47:11.194447+0000","last_became_peered":"2024-09-18T07:47:11.194447+0000","last_unstale":"2024-09-18T07:47:40.133777+0000","last_undegraded":"2024-09-18T07:47:40.133777+0000","last_fullsized":"2024-09-18T07:47:40.133777+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:44:18.607501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:44:18.607501+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":614,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.138959+0000","last_change":"2024-09-18T07:47:20.167246+0000","last_active":"2024-09-18T07:47:40.138959+0000","last_peered":"2024-09-18T07:47:40.138959+0000","last_clean":"2024-09-18T07:47:40.138959+0000","last_became_active":"2024-09-18T07:47:20.139152+0000","last_became_peered":"2024-09-18T07:47:20.139152+0000","last_unstale":"2024-09-18T07:47:40.138959+0000","last_undegraded":"2024-09-18T07:47:40.138959+0000","last_fullsized":"2024-09-18T07:47:40.138959+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:09.496455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.496455+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:23.094728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"323'20","reported_seq":948,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-18T07:47:37.200592+0000","last_change":"2024-09-18T07:43:46.102137+0000","last_active":"2024-09-18T07:47:37.200592+0000","last_peered":"2024-09-18T07:47:37.200592+0000","last_clean":"2024-09-18T07:47:37.200592+0000","last_became_active":"2024-09-18T07:35:01.410926+0000","last_became_peered":"2024-09-18T07:35:01.410926+0000","last_unstale":"2024-09-18T07:47:37.200592+0000","last_undegraded":"2024-09-18T07:47:37.200592+0000","last_fullsized":"2024-09-18T07:47:37.200592+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:43:46.102041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:43:46.102041+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:20.045500+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"398'14","reported_seq":643,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.205058+0000","last_change":"2024-09-18T07:45:53.887882+0000","last_active":"2024-09-18T07:47:36.205058+0000","last_peered":"2024-09-18T07:47:36.205058+0000","last_clean":"2024-09-18T07:47:36.205058+0000","last_became_active":"2024-09-18T07:44:43.786594+0000","last_became_peered":"2024-09-18T07:44:43.786594+0000","last_unstale":"2024-09-18T07:47:36.205058+0000","last_undegraded":"2024-09-18T07:47:36.205058+0000","last_fullsized":"2024-09-18T07:47:36.205058+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:45:53.887755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:45:53.887755+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"477'2474","reported_seq":4268,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:41.457790+0000","last_change":"2024-09-18T07:47:32.133816+0000","last_active":"2024-09-18T07:47:41.457790+0000","last_peered":"2024-09-18T07:47:41.457790+0000","last_clean":"2024-09-18T07:47:41.457790+0000","last_became_active":"2024-09-18T07:47:32.128751+0000","last_became_peered":"2024-09-18T07:47:32.128751+0000","last_unstale":"2024-09-18T07:47:41.457790+0000","last_undegraded":"2024-09-18T07:47:41.457790+0000","last_fullsized":"2024-09-18T07:47:41.457790+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":2474,"log_dups_size":0,"ondisk_log_size":2474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:22.032629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162136064,"num_objects":2474,"num_object_clones":0,"num_object_copies":7422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2474,"num_write_kb":158336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":867,"num_bytes_recovered":56819712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":313,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234357+0000","last_change":"2024-09-18T07:47:13.167529+0000","last_active":"2024-09-18T07:47:38.234357+0000","last_peered":"2024-09-18T07:47:38.234357+0000","last_clean":"2024-09-18T07:47:38.234357+0000","last_became_active":"2024-09-18T07:47:11.203901+0000","last_became_peered":"2024-09-18T07:47:11.203901+0000","last_unstale":"2024-09-18T07:47:38.234357+0000","last_undegraded":"2024-09-18T07:47:38.234357+0000","last_fullsized":"2024-09-18T07:47:38.234357+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:47:13.167470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:47:13.167470+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:29.279753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"323'27","reported_seq":685,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.141301+0000","last_change":"2024-09-18T07:47:20.190490+0000","last_active":"2024-09-18T07:47:40.141301+0000","last_peered":"2024-09-18T07:47:40.141301+0000","last_clean":"2024-09-18T07:47:40.141301+0000","last_became_active":"2024-09-18T07:47:20.141595+0000","last_became_peered":"2024-09-18T07:47:20.141595+0000","last_unstale":"2024-09-18T07:47:40.141301+0000","last_undegraded":"2024-09-18T07:47:40.141301+0000","last_fullsized":"2024-09-18T07:47:40.141301+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:42:33.391715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:33.391715+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:31.626482+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":925,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.134691+0000","last_change":"2024-09-18T07:47:20.143318+0000","last_active":"2024-09-18T07:47:40.134691+0000","last_peered":"2024-09-18T07:47:40.134691+0000","last_clean":"2024-09-18T07:47:40.134691+0000","last_became_active":"2024-09-18T07:47:20.141653+0000","last_became_peered":"2024-09-18T07:47:20.141653+0000","last_unstale":"2024-09-18T07:47:40.134691+0000","last_undegraded":"2024-09-18T07:47:40.134691+0000","last_fullsized":"2024-09-18T07:47:40.134691+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:25.143715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:25.143715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"461'12","reported_seq":947,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121696+0000","last_change":"2024-09-18T07:47:11.191234+0000","last_active":"2024-09-18T07:47:40.121696+0000","last_peered":"2024-09-18T07:47:40.121696+0000","last_clean":"2024-09-18T07:47:40.121696+0000","last_became_active":"2024-09-18T07:47:11.190670+0000","last_became_peered":"2024-09-18T07:47:11.190670+0000","last_unstale":"2024-09-18T07:47:40.121696+0000","last_undegraded":"2024-09-18T07:47:40.121696+0000","last_fullsized":"2024-09-18T07:47:40.121696+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:41:27.965205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:41:27.965205+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:19.937172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":728,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234690+0000","last_change":"2024-09-18T07:47:20.139676+0000","last_active":"2024-09-18T07:47:38.234690+0000","last_peered":"2024-09-18T07:47:38.234690+0000","last_clean":"2024-09-18T07:47:38.234690+0000","last_became_active":"2024-09-18T07:47:20.138987+0000","last_became_peered":"2024-09-18T07:47:20.138987+0000","last_unstale":"2024-09-18T07:47:38.234690+0000","last_undegraded":"2024-09-18T07:47:38.234690+0000","last_fullsized":"2024-09-18T07:47:38.234690+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:45.670301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:45.670301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:46.096766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"477'2528","reported_seq":4247,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:41.796711+0000","last_change":"2024-09-18T07:47:37.208096+0000","last_active":"2024-09-18T07:47:41.796711+0000","last_peered":"2024-09-18T07:47:41.796711+0000","last_clean":"2024-09-18T07:47:41.796711+0000","last_became_active":"2024-09-18T07:47:37.204971+0000","last_became_peered":"2024-09-18T07:47:37.204971+0000","last_unstale":"2024-09-18T07:47:41.796711+0000","last_undegraded":"2024-09-18T07:47:41.796711+0000","last_fullsized":"2024-09-18T07:47:41.796711+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":2528,"log_dups_size":0,"ondisk_log_size":2528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:16.452878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165675008,"num_objects":2528,"num_object_clones":0,"num_object_copies":7584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2528,"num_write_kb":161792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":831,"num_bytes_recovered":54460416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":355,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121686+0000","last_change":"2024-09-18T07:44:43.923223+0000","last_active":"2024-09-18T07:47:40.121686+0000","last_peered":"2024-09-18T07:47:40.121686+0000","last_clean":"2024-09-18T07:47:40.121686+0000","last_became_active":"2024-09-18T07:44:43.786396+0000","last_became_peered":"2024-09-18T07:44:43.786396+0000","last_unstale":"2024-09-18T07:47:40.121686+0000","last_undegraded":"2024-09-18T07:47:40.121686+0000","last_fullsized":"2024-09-18T07:47:40.121686+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:44:43.923179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:44:43.923179+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"323'30","reported_seq":986,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.134356+0000","last_change":"2024-09-18T07:45:18.084183+0000","last_active":"2024-09-18T07:47:40.134356+0000","last_peered":"2024-09-18T07:47:40.134356+0000","last_clean":"2024-09-18T07:47:40.134356+0000","last_became_active":"2024-09-18T07:35:01.418909+0000","last_became_peered":"2024-09-18T07:35:01.418909+0000","last_unstale":"2024-09-18T07:47:40.134356+0000","last_undegraded":"2024-09-18T07:47:40.134356+0000","last_fullsized":"2024-09-18T07:47:40.134356+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'30","last_scrub_stamp":"2024-09-18T07:45:18.084091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:45:18.084091+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":608,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133558+0000","last_change":"2024-09-18T07:42:39.814649+0000","last_active":"2024-09-18T07:47:40.133558+0000","last_peered":"2024-09-18T07:47:40.133558+0000","last_clean":"2024-09-18T07:47:40.133558+0000","last_became_active":"2024-09-18T07:40:19.068296+0000","last_became_peered":"2024-09-18T07:40:19.068296+0000","last_unstale":"2024-09-18T07:47:40.133558+0000","last_undegraded":"2024-09-18T07:47:40.133558+0000","last_fullsized":"2024-09-18T07:47:40.133558+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:39.814582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:39.814582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:55.718130+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":793,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.120990+0000","last_change":"2024-09-18T07:44:43.788033+0000","last_active":"2024-09-18T07:47:40.120990+0000","last_peered":"2024-09-18T07:47:40.120990+0000","last_clean":"2024-09-18T07:47:40.120990+0000","last_became_active":"2024-09-18T07:44:43.780927+0000","last_became_peered":"2024-09-18T07:44:43.780927+0000","last_unstale":"2024-09-18T07:47:40.120990+0000","last_undegraded":"2024-09-18T07:47:40.120990+0000","last_fullsized":"2024-09-18T07:47:40.120990+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:09.067941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:09.067941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":710,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121127+0000","last_change":"2024-09-18T07:46:07.931110+0000","last_active":"2024-09-18T07:47:40.121127+0000","last_peered":"2024-09-18T07:47:40.121127+0000","last_clean":"2024-09-18T07:47:40.121127+0000","last_became_active":"2024-09-18T07:44:43.785212+0000","last_became_peered":"2024-09-18T07:44:43.785212+0000","last_unstale":"2024-09-18T07:47:40.121127+0000","last_undegraded":"2024-09-18T07:47:40.121127+0000","last_fullsized":"2024-09-18T07:47:40.121127+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:07.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:07.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"477'2494","reported_seq":4259,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:41.432322+0000","last_change":"2024-09-18T07:47:35.123117+0000","last_active":"2024-09-18T07:47:41.432322+0000","last_peered":"2024-09-18T07:47:41.432322+0000","last_clean":"2024-09-18T07:47:41.432322+0000","last_became_active":"2024-09-18T07:47:35.119437+0000","last_became_peered":"2024-09-18T07:47:35.119437+0000","last_unstale":"2024-09-18T07:47:41.432322+0000","last_undegraded":"2024-09-18T07:47:41.432322+0000","last_fullsized":"2024-09-18T07:47:41.432322+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":2494,"log_dups_size":0,"ondisk_log_size":2494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:19.863382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163446784,"num_objects":2494,"num_object_clones":0,"num_object_copies":7482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2494,"num_write_kb":159616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":55902208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":459,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.140786+0000","last_change":"2024-09-18T07:47:15.465656+0000","last_active":"2024-09-18T07:47:40.140786+0000","last_peered":"2024-09-18T07:47:40.140786+0000","last_clean":"2024-09-18T07:47:40.140786+0000","last_became_active":"2024-09-18T07:47:11.196995+0000","last_became_peered":"2024-09-18T07:47:11.196995+0000","last_unstale":"2024-09-18T07:47:40.140786+0000","last_undegraded":"2024-09-18T07:47:40.140786+0000","last_fullsized":"2024-09-18T07:47:40.140786+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:47:15.465581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:47:15.465581+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:43.728207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"323'28","reported_seq":828,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121259+0000","last_change":"2024-09-18T07:47:11.188351+0000","last_active":"2024-09-18T07:47:40.121259+0000","last_peered":"2024-09-18T07:47:40.121259+0000","last_clean":"2024-09-18T07:47:40.121259+0000","last_became_active":"2024-09-18T07:47:11.188002+0000","last_became_peered":"2024-09-18T07:47:11.188002+0000","last_unstale":"2024-09-18T07:47:40.121259+0000","last_undegraded":"2024-09-18T07:47:40.121259+0000","last_fullsized":"2024-09-18T07:47:40.121259+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-18T07:40:56.183802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:56.183802+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:18.938445+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":901,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-18T07:47:37.200992+0000","last_change":"2024-09-18T07:47:20.151941+0000","last_active":"2024-09-18T07:47:37.200992+0000","last_peered":"2024-09-18T07:47:37.200992+0000","last_clean":"2024-09-18T07:47:37.200992+0000","last_became_active":"2024-09-18T07:47:20.138745+0000","last_became_peered":"2024-09-18T07:47:20.138745+0000","last_unstale":"2024-09-18T07:47:37.200992+0000","last_undegraded":"2024-09-18T07:47:37.200992+0000","last_fullsized":"2024-09-18T07:47:37.200992+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:44:44.266955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:44:44.266955+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"467'12","reported_seq":1110,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.134155+0000","last_change":"2024-09-18T07:47:20.201000+0000","last_active":"2024-09-18T07:47:40.134155+0000","last_peered":"2024-09-18T07:47:40.134155+0000","last_clean":"2024-09-18T07:47:40.134155+0000","last_became_active":"2024-09-18T07:47:20.142972+0000","last_became_peered":"2024-09-18T07:47:20.142972+0000","last_unstale":"2024-09-18T07:47:40.134155+0000","last_undegraded":"2024-09-18T07:47:40.134155+0000","last_fullsized":"2024-09-18T07:47:40.134155+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:42:04.428395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:04.428395+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:20.535138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":660,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133359+0000","last_change":"2024-09-18T07:42:45.886132+0000","last_active":"2024-09-18T07:47:40.133359+0000","last_peered":"2024-09-18T07:47:40.133359+0000","last_clean":"2024-09-18T07:47:40.133359+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:47:40.133359+0000","last_undegraded":"2024-09-18T07:47:40.133359+0000","last_fullsized":"2024-09-18T07:47:40.133359+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:45.886069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:45.886069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"477'2622","reported_seq":2669,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:41.452870+0000","last_change":"2024-09-18T07:47:10.178170+0000","last_active":"2024-09-18T07:47:41.452870+0000","last_peered":"2024-09-18T07:47:41.452870+0000","last_clean":"2024-09-18T07:47:41.452870+0000","last_became_active":"2024-09-18T07:47:10.177901+0000","last_became_peered":"2024-09-18T07:47:10.177901+0000","last_unstale":"2024-09-18T07:47:41.452870+0000","last_undegraded":"2024-09-18T07:47:41.452870+0000","last_fullsized":"2024-09-18T07:47:41.452870+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":2622,"log_dups_size":0,"ondisk_log_size":2622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:21.545718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171835392,"num_objects":2622,"num_object_clones":0,"num_object_copies":7866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2622,"num_write_kb":167808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":596,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.204675+0000","last_change":"2024-09-18T07:40:38.066508+0000","last_active":"2024-09-18T07:47:36.204675+0000","last_peered":"2024-09-18T07:47:36.204675+0000","last_clean":"2024-09-18T07:47:36.204675+0000","last_became_active":"2024-09-18T07:39:04.941752+0000","last_became_peered":"2024-09-18T07:39:04.941752+0000","last_unstale":"2024-09-18T07:47:36.204675+0000","last_undegraded":"2024-09-18T07:47:36.204675+0000","last_fullsized":"2024-09-18T07:47:36.204675+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:40:38.066443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.066443+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:02.188770+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"323'20","reported_seq":535,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234186+0000","last_change":"2024-09-18T07:47:20.202968+0000","last_active":"2024-09-18T07:47:38.234186+0000","last_peered":"2024-09-18T07:47:38.234186+0000","last_clean":"2024-09-18T07:47:38.234186+0000","last_became_active":"2024-09-18T07:47:20.144924+0000","last_became_peered":"2024-09-18T07:47:20.144924+0000","last_unstale":"2024-09-18T07:47:38.234186+0000","last_undegraded":"2024-09-18T07:47:38.234186+0000","last_fullsized":"2024-09-18T07:47:38.234186+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:44:46.501666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:46.501666+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":777,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234329+0000","last_change":"2024-09-18T07:47:11.202895+0000","last_active":"2024-09-18T07:47:38.234329+0000","last_peered":"2024-09-18T07:47:38.234329+0000","last_clean":"2024-09-18T07:47:38.234329+0000","last_became_active":"2024-09-18T07:47:11.202526+0000","last_became_peered":"2024-09-18T07:47:11.202526+0000","last_unstale":"2024-09-18T07:47:38.234329+0000","last_undegraded":"2024-09-18T07:47:38.234329+0000","last_fullsized":"2024-09-18T07:47:38.234329+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:53.350019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:53.350019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:19.135777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"461'13","reported_seq":583,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.206381+0000","last_change":"2024-09-18T07:47:14.059129+0000","last_active":"2024-09-18T07:47:36.206381+0000","last_peered":"2024-09-18T07:47:36.206381+0000","last_clean":"2024-09-18T07:47:36.206381+0000","last_became_active":"2024-09-18T07:47:11.189241+0000","last_became_peered":"2024-09-18T07:47:11.189241+0000","last_unstale":"2024-09-18T07:47:36.206381+0000","last_undegraded":"2024-09-18T07:47:36.206381+0000","last_fullsized":"2024-09-18T07:47:36.206381+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'13","last_scrub_stamp":"2024-09-18T07:47:14.059062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:14.059062+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:20.411246+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":856,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-18T07:47:37.199732+0000","last_change":"2024-09-18T07:44:43.252210+0000","last_active":"2024-09-18T07:47:37.199732+0000","last_peered":"2024-09-18T07:47:37.199732+0000","last_clean":"2024-09-18T07:47:37.199732+0000","last_became_active":"2024-09-18T07:31:02.348162+0000","last_became_peered":"2024-09-18T07:31:02.348162+0000","last_unstale":"2024-09-18T07:47:37.199732+0000","last_undegraded":"2024-09-18T07:47:37.199732+0000","last_fullsized":"2024-09-18T07:47:37.199732+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:43.252166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:44:43.252166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"477'2404","reported_seq":4057,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:41.442363+0000","last_change":"2024-09-18T07:47:31.131323+0000","last_active":"2024-09-18T07:47:41.442363+0000","last_peered":"2024-09-18T07:47:41.442363+0000","last_clean":"2024-09-18T07:47:41.442363+0000","last_became_active":"2024-09-18T07:47:31.128537+0000","last_became_peered":"2024-09-18T07:47:31.128537+0000","last_unstale":"2024-09-18T07:47:41.442363+0000","last_undegraded":"2024-09-18T07:47:41.442363+0000","last_fullsized":"2024-09-18T07:47:41.442363+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":2404,"log_dups_size":0,"ondisk_log_size":2404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:19.120573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157548544,"num_objects":2404,"num_object_clones":0,"num_object_copies":7212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2404,"num_write_kb":153856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":799,"num_bytes_recovered":52363264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":604,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.202244+0000","last_change":"2024-09-18T07:45:50.787384+0000","last_active":"2024-09-18T07:47:36.202244+0000","last_peered":"2024-09-18T07:47:36.202244+0000","last_clean":"2024-09-18T07:47:36.202244+0000","last_became_active":"2024-09-18T07:39:04.948571+0000","last_became_peered":"2024-09-18T07:39:04.948571+0000","last_unstale":"2024-09-18T07:47:36.202244+0000","last_undegraded":"2024-09-18T07:47:36.202244+0000","last_fullsized":"2024-09-18T07:47:36.202244+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:45:50.787316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:45:50.787316+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"323'29","reported_seq":700,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234180+0000","last_change":"2024-09-18T07:47:11.198450+0000","last_active":"2024-09-18T07:47:38.234180+0000","last_peered":"2024-09-18T07:47:38.234180+0000","last_clean":"2024-09-18T07:47:38.234180+0000","last_became_active":"2024-09-18T07:47:11.198126+0000","last_became_peered":"2024-09-18T07:47:11.198126+0000","last_unstale":"2024-09-18T07:47:38.234180+0000","last_undegraded":"2024-09-18T07:47:38.234180+0000","last_fullsized":"2024-09-18T07:47:38.234180+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'29","last_scrub_stamp":"2024-09-18T07:44:14.250974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:14.250974+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":605,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.136171+0000","last_change":"2024-09-18T07:47:20.140174+0000","last_active":"2024-09-18T07:47:40.136171+0000","last_peered":"2024-09-18T07:47:40.136171+0000","last_clean":"2024-09-18T07:47:40.136171+0000","last_became_active":"2024-09-18T07:47:20.139806+0000","last_became_peered":"2024-09-18T07:47:20.139806+0000","last_unstale":"2024-09-18T07:47:40.136171+0000","last_undegraded":"2024-09-18T07:47:40.136171+0000","last_fullsized":"2024-09-18T07:47:40.136171+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:43.985969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:43.985969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:08.438184+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":518,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.202168+0000","last_change":"2024-09-18T07:47:11.194476+0000","last_active":"2024-09-18T07:47:36.202168+0000","last_peered":"2024-09-18T07:47:36.202168+0000","last_clean":"2024-09-18T07:47:36.202168+0000","last_became_active":"2024-09-18T07:47:11.193892+0000","last_became_peered":"2024-09-18T07:47:11.193892+0000","last_unstale":"2024-09-18T07:47:36.202168+0000","last_undegraded":"2024-09-18T07:47:36.202168+0000","last_fullsized":"2024-09-18T07:47:36.202168+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:11.108021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:43:11.108021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":521,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.203016+0000","last_change":"2024-09-18T07:47:20.138912+0000","last_active":"2024-09-18T07:47:36.203016+0000","last_peered":"2024-09-18T07:47:36.203016+0000","last_clean":"2024-09-18T07:47:36.203016+0000","last_became_active":"2024-09-18T07:47:20.138390+0000","last_became_peered":"2024-09-18T07:47:20.138390+0000","last_unstale":"2024-09-18T07:47:36.203016+0000","last_undegraded":"2024-09-18T07:47:36.203016+0000","last_fullsized":"2024-09-18T07:47:36.203016+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:49.018222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:49.018222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:52.717999+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.b","version":"477'2493","reported_seq":4248,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:41.471032+0000","last_change":"2024-09-18T07:47:32.130415+0000","last_active":"2024-09-18T07:47:41.471032+0000","last_peered":"2024-09-18T07:47:41.471032+0000","last_clean":"2024-09-18T07:47:41.471032+0000","last_became_active":"2024-09-18T07:47:32.128340+0000","last_became_peered":"2024-09-18T07:47:32.128340+0000","last_unstale":"2024-09-18T07:47:41.471032+0000","last_undegraded":"2024-09-18T07:47:41.471032+0000","last_fullsized":"2024-09-18T07:47:41.471032+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":2493,"log_dups_size":0,"ondisk_log_size":2493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:35.984390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163381248,"num_objects":2493,"num_object_clones":0,"num_object_copies":7479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":159552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":849,"num_bytes_recovered":55640064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":333,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234069+0000","last_change":"2024-09-18T07:47:12.191482+0000","last_active":"2024-09-18T07:47:38.234069+0000","last_peered":"2024-09-18T07:47:38.234069+0000","last_clean":"2024-09-18T07:47:38.234069+0000","last_became_active":"2024-09-18T07:47:11.195706+0000","last_became_peered":"2024-09-18T07:47:11.195706+0000","last_unstale":"2024-09-18T07:47:38.234069+0000","last_undegraded":"2024-09-18T07:47:38.234069+0000","last_fullsized":"2024-09-18T07:47:38.234069+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:47:12.191419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:47:12.191419+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:38.375897+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"467'43","reported_seq":873,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.135922+0000","last_change":"2024-09-18T07:47:20.189498+0000","last_active":"2024-09-18T07:47:40.135922+0000","last_peered":"2024-09-18T07:47:40.135922+0000","last_clean":"2024-09-18T07:47:40.135922+0000","last_became_active":"2024-09-18T07:47:20.143283+0000","last_became_peered":"2024-09-18T07:47:20.143283+0000","last_unstale":"2024-09-18T07:47:40.135922+0000","last_undegraded":"2024-09-18T07:47:40.135922+0000","last_fullsized":"2024-09-18T07:47:40.135922+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'41","last_scrub_stamp":"2024-09-18T07:46:12.858570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:12.858570+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:39.934683+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":799,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121398+0000","last_change":"2024-09-18T07:44:43.780418+0000","last_active":"2024-09-18T07:47:40.121398+0000","last_peered":"2024-09-18T07:47:40.121398+0000","last_clean":"2024-09-18T07:47:40.121398+0000","last_became_active":"2024-09-18T07:44:43.780114+0000","last_became_peered":"2024-09-18T07:44:43.780114+0000","last_unstale":"2024-09-18T07:47:40.121398+0000","last_undegraded":"2024-09-18T07:47:40.121398+0000","last_fullsized":"2024-09-18T07:47:40.121398+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:03.028558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:03.028558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":769,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234248+0000","last_change":"2024-09-18T07:47:20.190556+0000","last_active":"2024-09-18T07:47:38.234248+0000","last_peered":"2024-09-18T07:47:38.234248+0000","last_clean":"2024-09-18T07:47:38.234248+0000","last_became_active":"2024-09-18T07:47:20.144312+0000","last_became_peered":"2024-09-18T07:47:20.144312+0000","last_unstale":"2024-09-18T07:47:38.234248+0000","last_undegraded":"2024-09-18T07:47:38.234248+0000","last_fullsized":"2024-09-18T07:47:38.234248+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:44:06.489166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:44:06.489166+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":884,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-18T07:47:37.200184+0000","last_change":"2024-09-18T07:47:20.131135+0000","last_active":"2024-09-18T07:47:37.200184+0000","last_peered":"2024-09-18T07:47:37.200184+0000","last_clean":"2024-09-18T07:47:37.200184+0000","last_became_active":"2024-09-18T07:47:20.130809+0000","last_became_peered":"2024-09-18T07:47:20.130809+0000","last_unstale":"2024-09-18T07:47:37.200184+0000","last_undegraded":"2024-09-18T07:47:37.200184+0000","last_fullsized":"2024-09-18T07:47:37.200184+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:46.629196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:46.629196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:15.302326+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":902,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.139587+0000","last_change":"2024-09-18T07:45:52.882759+0000","last_active":"2024-09-18T07:47:40.139587+0000","last_peered":"2024-09-18T07:47:40.139587+0000","last_clean":"2024-09-18T07:47:40.139587+0000","last_became_active":"2024-09-18T07:44:43.780294+0000","last_became_peered":"2024-09-18T07:44:43.780294+0000","last_unstale":"2024-09-18T07:47:40.139587+0000","last_undegraded":"2024-09-18T07:47:40.139587+0000","last_fullsized":"2024-09-18T07:47:40.139587+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:52.882700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:45:52.882700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"474'2181","reported_seq":2221,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.371319+0000","last_change":"2024-09-18T07:47:11.190416+0000","last_active":"2024-09-18T07:47:36.371319+0000","last_peered":"2024-09-18T07:47:36.371319+0000","last_clean":"2024-09-18T07:47:36.371319+0000","last_became_active":"2024-09-18T07:47:11.190117+0000","last_became_peered":"2024-09-18T07:47:11.190117+0000","last_unstale":"2024-09-18T07:47:36.371319+0000","last_undegraded":"2024-09-18T07:47:36.371319+0000","last_fullsized":"2024-09-18T07:47:36.371319+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":2181,"log_dups_size":0,"ondisk_log_size":2181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:15.924044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142934016,"num_objects":2181,"num_object_clones":0,"num_object_copies":6543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2181,"num_write_kb":139584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":721,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.132705+0000","last_change":"2024-09-18T07:45:24.617065+0000","last_active":"2024-09-18T07:47:40.132705+0000","last_peered":"2024-09-18T07:47:40.132705+0000","last_clean":"2024-09-18T07:47:40.132705+0000","last_became_active":"2024-09-18T07:44:43.783295+0000","last_became_peered":"2024-09-18T07:44:43.783295+0000","last_unstale":"2024-09-18T07:47:40.132705+0000","last_undegraded":"2024-09-18T07:47:40.132705+0000","last_fullsized":"2024-09-18T07:47:40.132705+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:45:24.616992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:45:24.616992+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"323'17","reported_seq":561,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234477+0000","last_change":"2024-09-18T07:47:20.202629+0000","last_active":"2024-09-18T07:47:38.234477+0000","last_peered":"2024-09-18T07:47:38.234477+0000","last_clean":"2024-09-18T07:47:38.234477+0000","last_became_active":"2024-09-18T07:47:20.144799+0000","last_became_peered":"2024-09-18T07:47:20.144799+0000","last_unstale":"2024-09-18T07:47:38.234477+0000","last_undegraded":"2024-09-18T07:47:38.234477+0000","last_fullsized":"2024-09-18T07:47:38.234477+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'17","last_scrub_stamp":"2024-09-18T07:43:40.533240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:43:40.533240+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":23,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":618,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.134891+0000","last_change":"2024-09-18T07:47:20.140678+0000","last_active":"2024-09-18T07:47:40.134891+0000","last_peered":"2024-09-18T07:47:40.134891+0000","last_clean":"2024-09-18T07:47:40.134891+0000","last_became_active":"2024-09-18T07:47:20.140376+0000","last_became_peered":"2024-09-18T07:47:20.140376+0000","last_unstale":"2024-09-18T07:47:40.134891+0000","last_undegraded":"2024-09-18T07:47:40.134891+0000","last_fullsized":"2024-09-18T07:47:40.134891+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:52.917296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:52.917296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:20.683925+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":909,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.135569+0000","last_change":"2024-09-18T07:47:07.841463+0000","last_active":"2024-09-18T07:47:40.135569+0000","last_peered":"2024-09-18T07:47:40.135569+0000","last_clean":"2024-09-18T07:47:40.135569+0000","last_became_active":"2024-09-18T07:30:55.283844+0000","last_became_peered":"2024-09-18T07:30:55.283844+0000","last_unstale":"2024-09-18T07:47:40.135569+0000","last_undegraded":"2024-09-18T07:47:40.135569+0000","last_fullsized":"2024-09-18T07:47:40.135569+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:07.841406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:47:07.841406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:36.552524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":908,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.135929+0000","last_change":"2024-09-18T07:47:16.834107+0000","last_active":"2024-09-18T07:47:40.135929+0000","last_peered":"2024-09-18T07:47:40.135929+0000","last_clean":"2024-09-18T07:47:40.135929+0000","last_became_active":"2024-09-18T07:47:11.201379+0000","last_became_peered":"2024-09-18T07:47:11.201379+0000","last_unstale":"2024-09-18T07:47:40.135929+0000","last_undegraded":"2024-09-18T07:47:40.135929+0000","last_fullsized":"2024-09-18T07:47:40.135929+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:16.834043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:16.834043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:46.448197+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":656,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133005+0000","last_change":"2024-09-18T07:40:54.574254+0000","last_active":"2024-09-18T07:47:40.133005+0000","last_peered":"2024-09-18T07:47:40.133005+0000","last_clean":"2024-09-18T07:47:40.133005+0000","last_became_active":"2024-09-18T07:37:43.060019+0000","last_became_peered":"2024-09-18T07:37:43.060019+0000","last_unstale":"2024-09-18T07:47:40.133005+0000","last_undegraded":"2024-09-18T07:47:40.133005+0000","last_fullsized":"2024-09-18T07:47:40.133005+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:40:54.574077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:54.574077+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"477'2453","reported_seq":2500,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:41.475385+0000","last_change":"2024-09-18T07:47:10.191710+0000","last_active":"2024-09-18T07:47:41.475385+0000","last_peered":"2024-09-18T07:47:41.475385+0000","last_clean":"2024-09-18T07:47:41.475385+0000","last_became_active":"2024-09-18T07:47:10.191380+0000","last_became_peered":"2024-09-18T07:47:10.191380+0000","last_unstale":"2024-09-18T07:47:41.475385+0000","last_undegraded":"2024-09-18T07:47:41.475385+0000","last_fullsized":"2024-09-18T07:47:41.475385+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":2453,"log_dups_size":0,"ondisk_log_size":2453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:26.431119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160759808,"num_objects":2453,"num_object_clones":0,"num_object_copies":7359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2453,"num_write_kb":156992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":635,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.205489+0000","last_change":"2024-09-18T07:47:20.138473+0000","last_active":"2024-09-18T07:47:36.205489+0000","last_peered":"2024-09-18T07:47:36.205489+0000","last_clean":"2024-09-18T07:47:36.205489+0000","last_became_active":"2024-09-18T07:47:20.137990+0000","last_became_peered":"2024-09-18T07:47:20.137990+0000","last_unstale":"2024-09-18T07:47:36.205489+0000","last_undegraded":"2024-09-18T07:47:36.205489+0000","last_fullsized":"2024-09-18T07:47:36.205489+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:46:15.989579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:46:15.989579+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"323'24","reported_seq":958,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-18T07:47:37.200720+0000","last_change":"2024-09-18T07:47:20.159485+0000","last_active":"2024-09-18T07:47:37.200720+0000","last_peered":"2024-09-18T07:47:37.200720+0000","last_clean":"2024-09-18T07:47:37.200720+0000","last_became_active":"2024-09-18T07:47:20.138961+0000","last_became_peered":"2024-09-18T07:47:20.138961+0000","last_unstale":"2024-09-18T07:47:37.200720+0000","last_undegraded":"2024-09-18T07:47:37.200720+0000","last_fullsized":"2024-09-18T07:47:37.200720+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'24","last_scrub_stamp":"2024-09-18T07:44:06.211676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:06.211676+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:36.138635+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":877,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-18T07:47:37.199839+0000","last_change":"2024-09-18T07:44:43.784965+0000","last_active":"2024-09-18T07:47:37.199839+0000","last_peered":"2024-09-18T07:47:37.199839+0000","last_clean":"2024-09-18T07:47:37.199839+0000","last_became_active":"2024-09-18T07:44:43.784621+0000","last_became_peered":"2024-09-18T07:44:43.784621+0000","last_unstale":"2024-09-18T07:47:37.199839+0000","last_undegraded":"2024-09-18T07:47:37.199839+0000","last_fullsized":"2024-09-18T07:47:37.199839+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:42:44.401546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:44.401546+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:38.653573+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":1423,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.189383+0000","last_change":"2024-09-18T07:47:20.162839+0000","last_active":"2024-09-18T07:47:40.189383+0000","last_peered":"2024-09-18T07:47:40.189383+0000","last_clean":"2024-09-18T07:47:40.189383+0000","last_became_active":"2024-09-18T07:47:20.138901+0000","last_became_peered":"2024-09-18T07:47:20.138901+0000","last_unstale":"2024-09-18T07:47:40.189383+0000","last_undegraded":"2024-09-18T07:47:40.189383+0000","last_fullsized":"2024-09-18T07:47:40.189383+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T07:46:47.771151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:46:47.771151+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:54.068379+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":762,"num_read_kb":757,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":771,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234808+0000","last_change":"2024-09-18T07:47:20.145165+0000","last_active":"2024-09-18T07:47:38.234808+0000","last_peered":"2024-09-18T07:47:38.234808+0000","last_clean":"2024-09-18T07:47:38.234808+0000","last_became_active":"2024-09-18T07:47:20.144413+0000","last_became_peered":"2024-09-18T07:47:20.144413+0000","last_unstale":"2024-09-18T07:47:38.234808+0000","last_undegraded":"2024-09-18T07:47:38.234808+0000","last_fullsized":"2024-09-18T07:47:38.234808+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:50.587498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:44:50.587498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":431,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234630+0000","last_change":"2024-09-18T07:47:20.143704+0000","last_active":"2024-09-18T07:47:38.234630+0000","last_peered":"2024-09-18T07:47:38.234630+0000","last_clean":"2024-09-18T07:47:38.234630+0000","last_became_active":"2024-09-18T07:47:20.143383+0000","last_became_peered":"2024-09-18T07:47:20.143383+0000","last_unstale":"2024-09-18T07:47:38.234630+0000","last_undegraded":"2024-09-18T07:47:38.234630+0000","last_fullsized":"2024-09-18T07:47:38.234630+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:52.440332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:52.440332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:00.226264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.a","version":"474'2147","reported_seq":2187,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.355504+0000","last_change":"2024-09-18T07:47:11.194180+0000","last_active":"2024-09-18T07:47:36.355504+0000","last_peered":"2024-09-18T07:47:36.355504+0000","last_clean":"2024-09-18T07:47:36.355504+0000","last_became_active":"2024-09-18T07:47:11.193156+0000","last_became_peered":"2024-09-18T07:47:11.193156+0000","last_unstale":"2024-09-18T07:47:36.355504+0000","last_undegraded":"2024-09-18T07:47:36.355504+0000","last_fullsized":"2024-09-18T07:47:36.355504+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":2147,"log_dups_size":0,"ondisk_log_size":2147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:15.069852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140705792,"num_objects":2147,"num_object_clones":0,"num_object_copies":6441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2147,"num_write_kb":137408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"398'34","reported_seq":667,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.202669+0000","last_change":"2024-09-18T07:44:44.523660+0000","last_active":"2024-09-18T07:47:36.202669+0000","last_peered":"2024-09-18T07:47:36.202669+0000","last_clean":"2024-09-18T07:47:36.202669+0000","last_became_active":"2024-09-18T07:44:43.786827+0000","last_became_peered":"2024-09-18T07:44:43.786827+0000","last_unstale":"2024-09-18T07:47:36.202669+0000","last_undegraded":"2024-09-18T07:47:36.202669+0000","last_fullsized":"2024-09-18T07:47:36.202669+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'33","last_scrub_stamp":"2024-09-18T07:44:44.523601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:44.523601+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":624,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234486+0000","last_change":"2024-09-18T07:47:11.199888+0000","last_active":"2024-09-18T07:47:38.234486+0000","last_peered":"2024-09-18T07:47:38.234486+0000","last_clean":"2024-09-18T07:47:38.234486+0000","last_became_active":"2024-09-18T07:47:11.198809+0000","last_became_peered":"2024-09-18T07:47:11.198809+0000","last_unstale":"2024-09-18T07:47:38.234486+0000","last_undegraded":"2024-09-18T07:47:38.234486+0000","last_fullsized":"2024-09-18T07:47:38.234486+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:35.555785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:35.555785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":704,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133058+0000","last_change":"2024-09-18T07:45:25.666309+0000","last_active":"2024-09-18T07:47:40.133058+0000","last_peered":"2024-09-18T07:47:40.133058+0000","last_clean":"2024-09-18T07:47:40.133058+0000","last_became_active":"2024-09-18T07:39:03.350419+0000","last_became_peered":"2024-09-18T07:39:03.350419+0000","last_unstale":"2024-09-18T07:47:40.133058+0000","last_undegraded":"2024-09-18T07:47:40.133058+0000","last_fullsized":"2024-09-18T07:47:40.133058+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:45:25.666036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:45:25.666036+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":657,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133812+0000","last_change":"2024-09-18T07:47:11.195667+0000","last_active":"2024-09-18T07:47:40.133812+0000","last_peered":"2024-09-18T07:47:40.133812+0000","last_clean":"2024-09-18T07:47:40.133812+0000","last_became_active":"2024-09-18T07:47:11.195084+0000","last_became_peered":"2024-09-18T07:47:11.195084+0000","last_unstale":"2024-09-18T07:47:40.133812+0000","last_undegraded":"2024-09-18T07:47:40.133812+0000","last_fullsized":"2024-09-18T07:47:40.133812+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:06.726445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:42:06.726445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:20.870125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":745,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234626+0000","last_change":"2024-09-18T07:47:11.198595+0000","last_active":"2024-09-18T07:47:38.234626+0000","last_peered":"2024-09-18T07:47:38.234626+0000","last_clean":"2024-09-18T07:47:38.234626+0000","last_became_active":"2024-09-18T07:47:11.196928+0000","last_became_peered":"2024-09-18T07:47:11.196928+0000","last_unstale":"2024-09-18T07:47:38.234626+0000","last_undegraded":"2024-09-18T07:47:38.234626+0000","last_fullsized":"2024-09-18T07:47:38.234626+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:36.592215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:36.592215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"474'2115","reported_seq":2156,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.313309+0000","last_change":"2024-09-18T07:47:10.190697+0000","last_active":"2024-09-18T07:47:36.313309+0000","last_peered":"2024-09-18T07:47:36.313309+0000","last_clean":"2024-09-18T07:47:36.313309+0000","last_became_active":"2024-09-18T07:47:10.190402+0000","last_became_peered":"2024-09-18T07:47:10.190402+0000","last_unstale":"2024-09-18T07:47:36.313309+0000","last_undegraded":"2024-09-18T07:47:36.313309+0000","last_fullsized":"2024-09-18T07:47:36.313309+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":2115,"log_dups_size":0,"ondisk_log_size":2115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:27.449274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138608640,"num_objects":2115,"num_object_clones":0,"num_object_copies":6345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2115,"num_write_kb":135360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"323'19","reported_seq":960,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121797+0000","last_change":"2024-09-18T07:47:20.196636+0000","last_active":"2024-09-18T07:47:40.121797+0000","last_peered":"2024-09-18T07:47:40.121797+0000","last_clean":"2024-09-18T07:47:40.121797+0000","last_became_active":"2024-09-18T07:47:20.137370+0000","last_became_peered":"2024-09-18T07:47:20.137370+0000","last_unstale":"2024-09-18T07:47:40.121797+0000","last_undegraded":"2024-09-18T07:47:40.121797+0000","last_fullsized":"2024-09-18T07:47:40.121797+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:42:43.402858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:43.402858+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":770,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121156+0000","last_change":"2024-09-18T07:44:43.793621+0000","last_active":"2024-09-18T07:47:40.121156+0000","last_peered":"2024-09-18T07:47:40.121156+0000","last_clean":"2024-09-18T07:47:40.121156+0000","last_became_active":"2024-09-18T07:44:43.790160+0000","last_became_peered":"2024-09-18T07:44:43.790160+0000","last_unstale":"2024-09-18T07:47:40.121156+0000","last_undegraded":"2024-09-18T07:47:40.121156+0000","last_fullsized":"2024-09-18T07:47:40.121156+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.078600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.078600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:57.964570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":780,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234332+0000","last_change":"2024-09-18T07:47:11.202169+0000","last_active":"2024-09-18T07:47:38.234332+0000","last_peered":"2024-09-18T07:47:38.234332+0000","last_clean":"2024-09-18T07:47:38.234332+0000","last_became_active":"2024-09-18T07:47:11.198103+0000","last_became_peered":"2024-09-18T07:47:11.198103+0000","last_unstale":"2024-09-18T07:47:38.234332+0000","last_undegraded":"2024-09-18T07:47:38.234332+0000","last_fullsized":"2024-09-18T07:47:38.234332+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:45.479624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:44:45.479624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":751,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.204752+0000","last_change":"2024-09-18T07:47:18.117749+0000","last_active":"2024-09-18T07:47:36.204752+0000","last_peered":"2024-09-18T07:47:36.204752+0000","last_clean":"2024-09-18T07:47:36.204752+0000","last_became_active":"2024-09-18T07:47:11.189605+0000","last_became_peered":"2024-09-18T07:47:11.189605+0000","last_unstale":"2024-09-18T07:47:36.204752+0000","last_undegraded":"2024-09-18T07:47:36.204752+0000","last_fullsized":"2024-09-18T07:47:36.204752+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:18.117581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:18.117581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:26.308182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"477'2413","reported_seq":2460,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:41.439690+0000","last_change":"2024-09-18T07:47:10.192621+0000","last_active":"2024-09-18T07:47:41.439690+0000","last_peered":"2024-09-18T07:47:41.439690+0000","last_clean":"2024-09-18T07:47:41.439690+0000","last_became_active":"2024-09-18T07:47:10.192273+0000","last_became_peered":"2024-09-18T07:47:10.192273+0000","last_unstale":"2024-09-18T07:47:41.439690+0000","last_undegraded":"2024-09-18T07:47:41.439690+0000","last_fullsized":"2024-09-18T07:47:41.439690+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":2413,"log_dups_size":0,"ondisk_log_size":2413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:29.305991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158138368,"num_objects":2413,"num_object_clones":0,"num_object_copies":7239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2413,"num_write_kb":154432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":287,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.233895+0000","last_change":"2024-09-18T07:47:11.204479+0000","last_active":"2024-09-18T07:47:38.233895+0000","last_peered":"2024-09-18T07:47:38.233895+0000","last_clean":"2024-09-18T07:47:38.233895+0000","last_became_active":"2024-09-18T07:47:11.195101+0000","last_became_peered":"2024-09-18T07:47:11.195101+0000","last_unstale":"2024-09-18T07:47:38.233895+0000","last_undegraded":"2024-09-18T07:47:38.233895+0000","last_fullsized":"2024-09-18T07:47:38.233895+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:47:11.204436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:47:11.204436+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:34.065822+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"323'31","reported_seq":844,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121757+0000","last_change":"2024-09-18T07:47:20.154723+0000","last_active":"2024-09-18T07:47:40.121757+0000","last_peered":"2024-09-18T07:47:40.121757+0000","last_clean":"2024-09-18T07:47:40.121757+0000","last_became_active":"2024-09-18T07:47:20.132912+0000","last_became_peered":"2024-09-18T07:47:20.132912+0000","last_unstale":"2024-09-18T07:47:40.121757+0000","last_undegraded":"2024-09-18T07:47:40.121757+0000","last_fullsized":"2024-09-18T07:47:40.121757+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-18T07:40:04.281758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:04.281758+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:20.957420+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":540,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.202386+0000","last_change":"2024-09-18T07:47:20.141372+0000","last_active":"2024-09-18T07:47:36.202386+0000","last_peered":"2024-09-18T07:47:36.202386+0000","last_clean":"2024-09-18T07:47:36.202386+0000","last_became_active":"2024-09-18T07:47:20.141038+0000","last_became_peered":"2024-09-18T07:47:20.141038+0000","last_unstale":"2024-09-18T07:47:36.202386+0000","last_undegraded":"2024-09-18T07:47:36.202386+0000","last_fullsized":"2024-09-18T07:47:36.202386+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:47.023220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:47.023220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:05.811084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":441,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121646+0000","last_change":"2024-09-18T07:47:20.131352+0000","last_active":"2024-09-18T07:47:40.121646+0000","last_peered":"2024-09-18T07:47:40.121646+0000","last_clean":"2024-09-18T07:47:40.121646+0000","last_became_active":"2024-09-18T07:47:20.130994+0000","last_became_peered":"2024-09-18T07:47:20.130994+0000","last_unstale":"2024-09-18T07:47:40.121646+0000","last_undegraded":"2024-09-18T07:47:40.121646+0000","last_fullsized":"2024-09-18T07:47:40.121646+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:13.019711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:13.019711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":906,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.137133+0000","last_change":"2024-09-18T07:47:06.889207+0000","last_active":"2024-09-18T07:47:40.137133+0000","last_peered":"2024-09-18T07:47:40.137133+0000","last_clean":"2024-09-18T07:47:40.137133+0000","last_became_active":"2024-09-18T07:44:43.785056+0000","last_became_peered":"2024-09-18T07:44:43.785056+0000","last_unstale":"2024-09-18T07:47:40.137133+0000","last_undegraded":"2024-09-18T07:47:40.137133+0000","last_fullsized":"2024-09-18T07:47:40.137133+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:06.889139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:06.889139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:16.178878+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"474'2088","reported_seq":3830,"reported_epoch":474,"state":"active+undersized+remapped","last_fresh":"2024-09-18T07:47:36.856854+0000","last_change":"2024-09-18T07:47:36.857058+0000","last_active":"2024-09-18T07:47:36.856854+0000","last_peered":"2024-09-18T07:47:36.856854+0000","last_clean":"2024-09-18T07:47:19.256977+0000","last_became_active":"2024-09-18T07:47:21.226764+0000","last_became_peered":"2024-09-18T07:47:21.226764+0000","last_unstale":"2024-09-18T07:47:36.856854+0000","last_undegraded":"2024-09-18T07:47:36.856854+0000","last_fullsized":"2024-09-18T07:47:21.114379+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":2088,"log_dups_size":0,"ondisk_log_size":2088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136839168,"num_objects":2088,"num_object_clones":0,"num_object_copies":6264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2088,"num_write_kb":133632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":848,"num_bytes_recovered":55574528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7],"avail_no_missing":["6","1","5","7"],"object_location_counts":[{"shards":"6,7","objects":2088}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":709,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.132633+0000","last_change":"2024-09-18T07:46:24.585165+0000","last_active":"2024-09-18T07:47:40.132633+0000","last_peered":"2024-09-18T07:47:40.132633+0000","last_clean":"2024-09-18T07:47:40.132633+0000","last_became_active":"2024-09-18T07:39:04.944753+0000","last_became_peered":"2024-09-18T07:39:04.944753+0000","last_unstale":"2024-09-18T07:47:40.132633+0000","last_undegraded":"2024-09-18T07:47:40.132633+0000","last_fullsized":"2024-09-18T07:47:40.132633+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:46:24.585120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:46:24.585120+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:41.374983+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"323'20","reported_seq":805,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121744+0000","last_change":"2024-09-18T07:46:14.047994+0000","last_active":"2024-09-18T07:47:40.121744+0000","last_peered":"2024-09-18T07:47:40.121744+0000","last_clean":"2024-09-18T07:47:40.121744+0000","last_became_active":"2024-09-18T07:44:43.784244+0000","last_became_peered":"2024-09-18T07:44:43.784244+0000","last_unstale":"2024-09-18T07:47:40.121744+0000","last_undegraded":"2024-09-18T07:47:40.121744+0000","last_fullsized":"2024-09-18T07:47:40.121744+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:46:14.047907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:14.047907+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":287,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121201+0000","last_change":"2024-09-18T07:46:29.320743+0000","last_active":"2024-09-18T07:47:40.121201+0000","last_peered":"2024-09-18T07:47:40.121201+0000","last_clean":"2024-09-18T07:47:40.121201+0000","last_became_active":"2024-09-18T07:46:29.320499+0000","last_became_peered":"2024-09-18T07:46:29.320499+0000","last_unstale":"2024-09-18T07:47:40.121201+0000","last_undegraded":"2024-09-18T07:47:40.121201+0000","last_fullsized":"2024-09-18T07:47:40.121201+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:47.799342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:40:47.799342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"461'13","reported_seq":1130,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.137283+0000","last_change":"2024-09-18T07:47:15.865446+0000","last_active":"2024-09-18T07:47:40.137283+0000","last_peered":"2024-09-18T07:47:40.137283+0000","last_clean":"2024-09-18T07:47:40.137283+0000","last_became_active":"2024-09-18T07:47:11.203256+0000","last_became_peered":"2024-09-18T07:47:11.203256+0000","last_unstale":"2024-09-18T07:47:40.137283+0000","last_undegraded":"2024-09-18T07:47:40.137283+0000","last_fullsized":"2024-09-18T07:47:40.137283+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'13","last_scrub_stamp":"2024-09-18T07:47:15.865373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:15.865373+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:21.101204+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":779,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.122035+0000","last_change":"2024-09-18T07:47:20.132483+0000","last_active":"2024-09-18T07:47:40.122035+0000","last_peered":"2024-09-18T07:47:40.122035+0000","last_clean":"2024-09-18T07:47:40.122035+0000","last_became_active":"2024-09-18T07:47:20.132179+0000","last_became_peered":"2024-09-18T07:47:20.132179+0000","last_unstale":"2024-09-18T07:47:40.122035+0000","last_undegraded":"2024-09-18T07:47:40.122035+0000","last_fullsized":"2024-09-18T07:47:40.122035+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:48.207484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:48.207484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:03.286685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"477'2502","reported_seq":2549,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:41.778061+0000","last_change":"2024-09-18T07:47:10.191967+0000","last_active":"2024-09-18T07:47:41.778061+0000","last_peered":"2024-09-18T07:47:41.778061+0000","last_clean":"2024-09-18T07:47:41.778061+0000","last_became_active":"2024-09-18T07:47:10.190921+0000","last_became_peered":"2024-09-18T07:47:10.190921+0000","last_unstale":"2024-09-18T07:47:41.778061+0000","last_undegraded":"2024-09-18T07:47:41.778061+0000","last_fullsized":"2024-09-18T07:47:41.778061+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":2502,"log_dups_size":0,"ondisk_log_size":2502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:19.287004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163971072,"num_objects":2502,"num_object_clones":0,"num_object_copies":7506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2502,"num_write_kb":160128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"323'15","reported_seq":954,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-18T07:47:37.200231+0000","last_change":"2024-09-18T07:47:20.156876+0000","last_active":"2024-09-18T07:47:37.200231+0000","last_peered":"2024-09-18T07:47:37.200231+0000","last_clean":"2024-09-18T07:47:37.200231+0000","last_became_active":"2024-09-18T07:47:20.138834+0000","last_became_peered":"2024-09-18T07:47:20.138834+0000","last_unstale":"2024-09-18T07:47:37.200231+0000","last_undegraded":"2024-09-18T07:47:37.200231+0000","last_fullsized":"2024-09-18T07:47:37.200231+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:41:21.946066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:21.946066+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:21.009046+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":799,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234721+0000","last_change":"2024-09-18T07:47:11.195107+0000","last_active":"2024-09-18T07:47:38.234721+0000","last_peered":"2024-09-18T07:47:38.234721+0000","last_clean":"2024-09-18T07:47:38.234721+0000","last_became_active":"2024-09-18T07:47:11.194699+0000","last_became_peered":"2024-09-18T07:47:11.194699+0000","last_unstale":"2024-09-18T07:47:38.234721+0000","last_undegraded":"2024-09-18T07:47:38.234721+0000","last_fullsized":"2024-09-18T07:47:38.234721+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:04.518676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:43:04.518676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":886,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.135284+0000","last_change":"2024-09-18T07:46:08.854995+0000","last_active":"2024-09-18T07:47:40.135284+0000","last_peered":"2024-09-18T07:47:40.135284+0000","last_clean":"2024-09-18T07:47:40.135284+0000","last_became_active":"2024-09-18T07:44:43.784507+0000","last_became_peered":"2024-09-18T07:44:43.784507+0000","last_unstale":"2024-09-18T07:47:40.135284+0000","last_undegraded":"2024-09-18T07:47:40.135284+0000","last_fullsized":"2024-09-18T07:47:40.135284+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:08.854920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:08.854920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:19.894319+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":322,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.132901+0000","last_change":"2024-09-18T07:46:29.323007+0000","last_active":"2024-09-18T07:47:40.132901+0000","last_peered":"2024-09-18T07:47:40.132901+0000","last_clean":"2024-09-18T07:47:40.132901+0000","last_became_active":"2024-09-18T07:46:29.322779+0000","last_became_peered":"2024-09-18T07:46:29.322779+0000","last_unstale":"2024-09-18T07:47:40.132901+0000","last_undegraded":"2024-09-18T07:47:40.132901+0000","last_fullsized":"2024-09-18T07:47:40.132901+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:34.257746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:34.257746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:05.673685+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"477'2482","reported_seq":2528,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:41.791065+0000","last_change":"2024-09-18T07:47:11.201597+0000","last_active":"2024-09-18T07:47:41.791065+0000","last_peered":"2024-09-18T07:47:41.791065+0000","last_clean":"2024-09-18T07:47:41.791065+0000","last_became_active":"2024-09-18T07:47:11.201275+0000","last_became_peered":"2024-09-18T07:47:11.201275+0000","last_unstale":"2024-09-18T07:47:41.791065+0000","last_undegraded":"2024-09-18T07:47:41.791065+0000","last_fullsized":"2024-09-18T07:47:41.791065+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":2482,"log_dups_size":0,"ondisk_log_size":2482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:36.150226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162660352,"num_objects":2482,"num_object_clones":0,"num_object_copies":7446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2482,"num_write_kb":158848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"323'35","reported_seq":610,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.202633+0000","last_change":"2024-09-18T07:47:20.166667+0000","last_active":"2024-09-18T07:47:36.202633+0000","last_peered":"2024-09-18T07:47:36.202633+0000","last_clean":"2024-09-18T07:47:36.202633+0000","last_became_active":"2024-09-18T07:47:20.141942+0000","last_became_peered":"2024-09-18T07:47:20.141942+0000","last_unstale":"2024-09-18T07:47:36.202633+0000","last_undegraded":"2024-09-18T07:47:36.202633+0000","last_fullsized":"2024-09-18T07:47:36.202633+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'35","last_scrub_stamp":"2024-09-18T07:44:41.447137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:41.447137+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":678,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133280+0000","last_change":"2024-09-18T07:47:17.794908+0000","last_active":"2024-09-18T07:47:40.133280+0000","last_peered":"2024-09-18T07:47:40.133280+0000","last_clean":"2024-09-18T07:47:40.133280+0000","last_became_active":"2024-09-18T07:47:11.194873+0000","last_became_peered":"2024-09-18T07:47:11.194873+0000","last_unstale":"2024-09-18T07:47:40.133280+0000","last_undegraded":"2024-09-18T07:47:40.133280+0000","last_fullsized":"2024-09-18T07:47:40.133280+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:17.794850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:17.794850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:37.381809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":901,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.135194+0000","last_change":"2024-09-18T07:47:11.202717+0000","last_active":"2024-09-18T07:47:40.135194+0000","last_peered":"2024-09-18T07:47:40.135194+0000","last_clean":"2024-09-18T07:47:40.135194+0000","last_became_active":"2024-09-18T07:47:11.202437+0000","last_became_peered":"2024-09-18T07:47:11.202437+0000","last_unstale":"2024-09-18T07:47:40.135194+0000","last_undegraded":"2024-09-18T07:47:40.135194+0000","last_fullsized":"2024-09-18T07:47:40.135194+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:23.985496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:45:23.985496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:18.898670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":645,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133411+0000","last_change":"2024-09-18T07:43:39.239989+0000","last_active":"2024-09-18T07:47:40.133411+0000","last_peered":"2024-09-18T07:47:40.133411+0000","last_clean":"2024-09-18T07:47:40.133411+0000","last_became_active":"2024-09-18T07:35:01.406980+0000","last_became_peered":"2024-09-18T07:35:01.406980+0000","last_unstale":"2024-09-18T07:47:40.133411+0000","last_undegraded":"2024-09-18T07:47:40.133411+0000","last_fullsized":"2024-09-18T07:47:40.133411+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:39.239932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:39.239932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"475'2081","reported_seq":3798,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-18T07:47:37.417730+0000","last_change":"2024-09-18T07:47:35.127276+0000","last_active":"2024-09-18T07:47:37.417730+0000","last_peered":"2024-09-18T07:47:37.417730+0000","last_clean":"2024-09-18T07:47:37.417730+0000","last_became_active":"2024-09-18T07:47:35.123351+0000","last_became_peered":"2024-09-18T07:47:35.123351+0000","last_unstale":"2024-09-18T07:47:37.417730+0000","last_undegraded":"2024-09-18T07:47:37.417730+0000","last_fullsized":"2024-09-18T07:47:37.417730+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":2081,"log_dups_size":0,"ondisk_log_size":2081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:18.506239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136380416,"num_objects":2081,"num_object_clones":0,"num_object_copies":6243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2081,"num_write_kb":133184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":831,"num_bytes_recovered":54460416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"323'18","reported_seq":864,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121077+0000","last_change":"2024-09-18T07:47:20.151153+0000","last_active":"2024-09-18T07:47:40.121077+0000","last_peered":"2024-09-18T07:47:40.121077+0000","last_clean":"2024-09-18T07:47:40.121077+0000","last_became_active":"2024-09-18T07:47:20.132026+0000","last_became_peered":"2024-09-18T07:47:20.132026+0000","last_unstale":"2024-09-18T07:47:40.121077+0000","last_undegraded":"2024-09-18T07:47:40.121077+0000","last_fullsized":"2024-09-18T07:47:40.121077+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:46:25.012172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:25.012172+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:49.030383+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":670,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133422+0000","last_change":"2024-09-18T07:47:20.139266+0000","last_active":"2024-09-18T07:47:40.133422+0000","last_peered":"2024-09-18T07:47:40.133422+0000","last_clean":"2024-09-18T07:47:40.133422+0000","last_became_active":"2024-09-18T07:47:20.138910+0000","last_became_peered":"2024-09-18T07:47:20.138910+0000","last_unstale":"2024-09-18T07:47:40.133422+0000","last_undegraded":"2024-09-18T07:47:40.133422+0000","last_fullsized":"2024-09-18T07:47:40.133422+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:51.752578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:51.752578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:20.078827+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":918,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.134815+0000","last_change":"2024-09-18T07:47:20.140408+0000","last_active":"2024-09-18T07:47:40.134815+0000","last_peered":"2024-09-18T07:47:40.134815+0000","last_clean":"2024-09-18T07:47:40.134815+0000","last_became_active":"2024-09-18T07:47:20.139842+0000","last_became_peered":"2024-09-18T07:47:20.139842+0000","last_unstale":"2024-09-18T07:47:40.134815+0000","last_undegraded":"2024-09-18T07:47:40.134815+0000","last_fullsized":"2024-09-18T07:47:40.134815+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:13.830958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:13.830958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:14.510479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":905,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.439846+0000","last_change":"2024-09-18T07:47:40.439846+0000","last_active":"2024-09-18T07:47:40.439846+0000","last_peered":"2024-09-18T07:47:40.439846+0000","last_clean":"2024-09-18T07:47:40.439846+0000","last_became_active":"2024-09-18T07:47:20.141391+0000","last_became_peered":"2024-09-18T07:47:20.141391+0000","last_unstale":"2024-09-18T07:47:40.439846+0000","last_undegraded":"2024-09-18T07:47:40.439846+0000","last_fullsized":"2024-09-18T07:47:40.439846+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:40.439672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:40.439672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:51.944288+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"476'2250","reported_seq":4021,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:39.184739+0000","last_change":"2024-09-18T07:47:34.185546+0000","last_active":"2024-09-18T07:47:39.184739+0000","last_peered":"2024-09-18T07:47:39.184739+0000","last_clean":"2024-09-18T07:47:39.184739+0000","last_became_active":"2024-09-18T07:47:34.181063+0000","last_became_peered":"2024-09-18T07:47:34.181063+0000","last_unstale":"2024-09-18T07:47:39.184739+0000","last_undegraded":"2024-09-18T07:47:39.184739+0000","last_fullsized":"2024-09-18T07:47:39.184739+0000","mapping_epoch":471,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":2250,"log_dups_size":0,"ondisk_log_size":2250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:36.932427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147456000,"num_objects":2250,"num_object_clones":0,"num_object_copies":6750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2250,"num_write_kb":144000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":858,"num_bytes_recovered":56229888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"323'27","reported_seq":864,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.233734+0000","last_change":"2024-09-18T07:47:11.197814+0000","last_active":"2024-09-18T07:47:38.233734+0000","last_peered":"2024-09-18T07:47:38.233734+0000","last_clean":"2024-09-18T07:47:38.233734+0000","last_became_active":"2024-09-18T07:47:11.196288+0000","last_became_peered":"2024-09-18T07:47:11.196288+0000","last_unstale":"2024-09-18T07:47:38.233734+0000","last_undegraded":"2024-09-18T07:47:38.233734+0000","last_fullsized":"2024-09-18T07:47:38.233734+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:44:04.486867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:04.486867+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":670,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133641+0000","last_change":"2024-09-18T07:45:50.588718+0000","last_active":"2024-09-18T07:47:40.133641+0000","last_peered":"2024-09-18T07:47:40.133641+0000","last_clean":"2024-09-18T07:47:40.133641+0000","last_became_active":"2024-09-18T07:44:43.778284+0000","last_became_peered":"2024-09-18T07:44:43.778284+0000","last_unstale":"2024-09-18T07:47:40.133641+0000","last_undegraded":"2024-09-18T07:47:40.133641+0000","last_fullsized":"2024-09-18T07:47:40.133641+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:45:50.588657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:45:50.588657+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"398'14","reported_seq":845,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133151+0000","last_change":"2024-09-18T07:46:21.582783+0000","last_active":"2024-09-18T07:47:40.133151+0000","last_peered":"2024-09-18T07:47:40.133151+0000","last_clean":"2024-09-18T07:47:40.133151+0000","last_became_active":"2024-09-18T07:44:43.782355+0000","last_became_peered":"2024-09-18T07:44:43.782355+0000","last_unstale":"2024-09-18T07:47:40.133151+0000","last_undegraded":"2024-09-18T07:47:40.133151+0000","last_fullsized":"2024-09-18T07:47:40.133151+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:46:21.582737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:21.582737+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:48.610505+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":731,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234162+0000","last_change":"2024-09-18T07:47:11.200863+0000","last_active":"2024-09-18T07:47:38.234162+0000","last_peered":"2024-09-18T07:47:38.234162+0000","last_clean":"2024-09-18T07:47:38.234162+0000","last_became_active":"2024-09-18T07:47:11.197583+0000","last_became_peered":"2024-09-18T07:47:11.197583+0000","last_unstale":"2024-09-18T07:47:38.234162+0000","last_undegraded":"2024-09-18T07:47:38.234162+0000","last_fullsized":"2024-09-18T07:47:38.234162+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:07.547352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:07.547352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":510,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.203171+0000","last_change":"2024-09-18T07:47:11.188646+0000","last_active":"2024-09-18T07:47:36.203171+0000","last_peered":"2024-09-18T07:47:36.203171+0000","last_clean":"2024-09-18T07:47:36.203171+0000","last_became_active":"2024-09-18T07:47:11.188332+0000","last_became_peered":"2024-09-18T07:47:11.188332+0000","last_unstale":"2024-09-18T07:47:36.203171+0000","last_undegraded":"2024-09-18T07:47:36.203171+0000","last_fullsized":"2024-09-18T07:47:36.203171+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:08.181420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:08.181420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"323'12","reported_seq":954,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.141387+0000","last_change":"2024-09-18T07:47:20.188023+0000","last_active":"2024-09-18T07:47:40.141387+0000","last_peered":"2024-09-18T07:47:40.141387+0000","last_clean":"2024-09-18T07:47:40.141387+0000","last_became_active":"2024-09-18T07:47:20.142056+0000","last_became_peered":"2024-09-18T07:47:20.142056+0000","last_unstale":"2024-09-18T07:47:40.141387+0000","last_undegraded":"2024-09-18T07:47:40.141387+0000","last_fullsized":"2024-09-18T07:47:40.141387+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'12","last_scrub_stamp":"2024-09-18T07:45:41.914239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:45:41.914239+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":802,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234308+0000","last_change":"2024-09-18T07:47:20.141214+0000","last_active":"2024-09-18T07:47:38.234308+0000","last_peered":"2024-09-18T07:47:38.234308+0000","last_clean":"2024-09-18T07:47:38.234308+0000","last_became_active":"2024-09-18T07:47:20.140847+0000","last_became_peered":"2024-09-18T07:47:20.140847+0000","last_unstale":"2024-09-18T07:47:38.234308+0000","last_undegraded":"2024-09-18T07:47:38.234308+0000","last_fullsized":"2024-09-18T07:47:38.234308+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:37.515124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:43:37.515124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":662,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133140+0000","last_change":"2024-09-18T07:45:42.533680+0000","last_active":"2024-09-18T07:47:40.133140+0000","last_peered":"2024-09-18T07:47:40.133140+0000","last_clean":"2024-09-18T07:47:40.133140+0000","last_became_active":"2024-09-18T07:44:43.782348+0000","last_became_peered":"2024-09-18T07:44:43.782348+0000","last_unstale":"2024-09-18T07:47:40.133140+0000","last_undegraded":"2024-09-18T07:47:40.133140+0000","last_fullsized":"2024-09-18T07:47:40.133140+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:42.533634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:45:42.533634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"323'15","reported_seq":913,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-18T07:47:37.200887+0000","last_change":"2024-09-18T07:46:51.712735+0000","last_active":"2024-09-18T07:47:37.200887+0000","last_peered":"2024-09-18T07:47:37.200887+0000","last_clean":"2024-09-18T07:47:37.200887+0000","last_became_active":"2024-09-18T07:35:01.414458+0000","last_became_peered":"2024-09-18T07:35:01.414458+0000","last_unstale":"2024-09-18T07:47:37.200887+0000","last_undegraded":"2024-09-18T07:47:37.200887+0000","last_fullsized":"2024-09-18T07:47:37.200887+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:46:51.712666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:51.712666+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:52.984099+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":678,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133024+0000","last_change":"2024-09-18T07:45:51.559901+0000","last_active":"2024-09-18T07:47:40.133024+0000","last_peered":"2024-09-18T07:47:40.133024+0000","last_clean":"2024-09-18T07:47:40.133024+0000","last_became_active":"2024-09-18T07:44:43.782930+0000","last_became_peered":"2024-09-18T07:44:43.782930+0000","last_unstale":"2024-09-18T07:47:40.133024+0000","last_undegraded":"2024-09-18T07:47:40.133024+0000","last_fullsized":"2024-09-18T07:47:40.133024+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:51.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:45:51.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":908,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.140672+0000","last_change":"2024-09-18T07:47:20.141939+0000","last_active":"2024-09-18T07:47:40.140672+0000","last_peered":"2024-09-18T07:47:40.140672+0000","last_clean":"2024-09-18T07:47:40.140672+0000","last_became_active":"2024-09-18T07:47:20.141420+0000","last_became_peered":"2024-09-18T07:47:20.141420+0000","last_unstale":"2024-09-18T07:47:40.140672+0000","last_undegraded":"2024-09-18T07:47:40.140672+0000","last_fullsized":"2024-09-18T07:47:40.140672+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:53.481418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:53.481418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:09.936935+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":880,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.135656+0000","last_change":"2024-09-18T07:44:43.790870+0000","last_active":"2024-09-18T07:47:40.135656+0000","last_peered":"2024-09-18T07:47:40.135656+0000","last_clean":"2024-09-18T07:47:40.135656+0000","last_became_active":"2024-09-18T07:44:43.790582+0000","last_became_peered":"2024-09-18T07:44:43.790582+0000","last_unstale":"2024-09-18T07:47:40.135656+0000","last_undegraded":"2024-09-18T07:47:40.135656+0000","last_fullsized":"2024-09-18T07:47:40.135656+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:34.876429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:42:34.876429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:02.938984+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"323'15","reported_seq":827,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234435+0000","last_change":"2024-09-18T07:47:11.201346+0000","last_active":"2024-09-18T07:47:38.234435+0000","last_peered":"2024-09-18T07:47:38.234435+0000","last_clean":"2024-09-18T07:47:38.234435+0000","last_became_active":"2024-09-18T07:47:11.198306+0000","last_became_peered":"2024-09-18T07:47:11.198306+0000","last_unstale":"2024-09-18T07:47:38.234435+0000","last_undegraded":"2024-09-18T07:47:38.234435+0000","last_fullsized":"2024-09-18T07:47:38.234435+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-18T07:40:52.366672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:40:52.366672+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:20.107354+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":334,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.132709+0000","last_change":"2024-09-18T07:46:29.319452+0000","last_active":"2024-09-18T07:47:40.132709+0000","last_peered":"2024-09-18T07:47:40.132709+0000","last_clean":"2024-09-18T07:47:40.132709+0000","last_became_active":"2024-09-18T07:46:29.319112+0000","last_became_peered":"2024-09-18T07:46:29.319112+0000","last_unstale":"2024-09-18T07:47:40.132709+0000","last_undegraded":"2024-09-18T07:47:40.132709+0000","last_fullsized":"2024-09-18T07:47:40.132709+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:31.255278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:43:31.255278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"173'10","reported_seq":1104,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.140942+0000","last_change":"2024-09-18T07:46:08.086507+0000","last_active":"2024-09-18T07:47:40.140942+0000","last_peered":"2024-09-18T07:47:40.140942+0000","last_clean":"2024-09-18T07:47:40.140942+0000","last_became_active":"2024-09-18T07:35:01.407568+0000","last_became_peered":"2024-09-18T07:35:01.407568+0000","last_unstale":"2024-09-18T07:47:40.140942+0000","last_undegraded":"2024-09-18T07:47:40.140942+0000","last_fullsized":"2024-09-18T07:47:40.140942+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:46:08.086437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:08.086437+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":775,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.122310+0000","last_change":"2024-09-18T07:47:11.193555+0000","last_active":"2024-09-18T07:47:40.122310+0000","last_peered":"2024-09-18T07:47:40.122310+0000","last_clean":"2024-09-18T07:47:40.122310+0000","last_became_active":"2024-09-18T07:47:11.193273+0000","last_became_peered":"2024-09-18T07:47:11.193273+0000","last_unstale":"2024-09-18T07:47:40.122310+0000","last_undegraded":"2024-09-18T07:47:40.122310+0000","last_fullsized":"2024-09-18T07:47:40.122310+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:48.955463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.955463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:35.099307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"323'20","reported_seq":692,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133407+0000","last_change":"2024-09-18T07:47:20.165350+0000","last_active":"2024-09-18T07:47:40.133407+0000","last_peered":"2024-09-18T07:47:40.133407+0000","last_clean":"2024-09-18T07:47:40.133407+0000","last_became_active":"2024-09-18T07:47:20.138883+0000","last_became_peered":"2024-09-18T07:47:20.138883+0000","last_unstale":"2024-09-18T07:47:40.133407+0000","last_undegraded":"2024-09-18T07:47:40.133407+0000","last_fullsized":"2024-09-18T07:47:40.133407+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:46:14.650883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:14.650883+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":664,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133482+0000","last_change":"2024-09-18T07:46:08.731460+0000","last_active":"2024-09-18T07:47:40.133482+0000","last_peered":"2024-09-18T07:47:40.133482+0000","last_clean":"2024-09-18T07:47:40.133482+0000","last_became_active":"2024-09-18T07:44:43.779305+0000","last_became_peered":"2024-09-18T07:44:43.779305+0000","last_unstale":"2024-09-18T07:47:40.133482+0000","last_undegraded":"2024-09-18T07:47:40.133482+0000","last_fullsized":"2024-09-18T07:47:40.133482+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:08.731385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:08.731385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"467'14","reported_seq":937,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234543+0000","last_change":"2024-09-18T07:47:20.169193+0000","last_active":"2024-09-18T07:47:38.234543+0000","last_peered":"2024-09-18T07:47:38.234543+0000","last_clean":"2024-09-18T07:47:38.234543+0000","last_became_active":"2024-09-18T07:47:20.141913+0000","last_became_peered":"2024-09-18T07:47:20.141913+0000","last_unstale":"2024-09-18T07:47:38.234543+0000","last_undegraded":"2024-09-18T07:47:38.234543+0000","last_fullsized":"2024-09-18T07:47:38.234543+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:40:40.352040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.352040+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:36.076794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":322,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133159+0000","last_change":"2024-09-18T07:46:29.319811+0000","last_active":"2024-09-18T07:47:40.133159+0000","last_peered":"2024-09-18T07:47:40.133159+0000","last_clean":"2024-09-18T07:47:40.133159+0000","last_became_active":"2024-09-18T07:46:29.319164+0000","last_became_peered":"2024-09-18T07:46:29.319164+0000","last_unstale":"2024-09-18T07:47:40.133159+0000","last_undegraded":"2024-09-18T07:47:40.133159+0000","last_fullsized":"2024-09-18T07:47:40.133159+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:33.228024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:33.228024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"323'19","reported_seq":918,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121501+0000","last_change":"2024-09-18T07:47:20.196648+0000","last_active":"2024-09-18T07:47:40.121501+0000","last_peered":"2024-09-18T07:47:40.121501+0000","last_clean":"2024-09-18T07:47:40.121501+0000","last_became_active":"2024-09-18T07:47:20.140070+0000","last_became_peered":"2024-09-18T07:47:20.140070+0000","last_unstale":"2024-09-18T07:47:40.121501+0000","last_undegraded":"2024-09-18T07:47:40.121501+0000","last_fullsized":"2024-09-18T07:47:40.121501+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:41:54.022421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.022421+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:34.097428+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":832,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.121794+0000","last_change":"2024-09-18T07:46:06.964899+0000","last_active":"2024-09-18T07:47:40.121794+0000","last_peered":"2024-09-18T07:47:40.121794+0000","last_clean":"2024-09-18T07:47:40.121794+0000","last_became_active":"2024-09-18T07:44:43.780305+0000","last_became_peered":"2024-09-18T07:44:43.780305+0000","last_unstale":"2024-09-18T07:47:40.121794+0000","last_undegraded":"2024-09-18T07:47:40.121794+0000","last_fullsized":"2024-09-18T07:47:40.121794+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:06.964847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:06.964847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":487,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.233970+0000","last_change":"2024-09-18T07:47:20.145972+0000","last_active":"2024-09-18T07:47:38.233970+0000","last_peered":"2024-09-18T07:47:38.233970+0000","last_clean":"2024-09-18T07:47:38.233970+0000","last_became_active":"2024-09-18T07:47:20.145450+0000","last_became_peered":"2024-09-18T07:47:20.145450+0000","last_unstale":"2024-09-18T07:47:38.233970+0000","last_undegraded":"2024-09-18T07:47:38.233970+0000","last_fullsized":"2024-09-18T07:47:38.233970+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:36.563010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:43:36.563010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":385,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-18T07:47:37.199679+0000","last_change":"2024-09-18T07:47:14.913645+0000","last_active":"2024-09-18T07:47:37.199679+0000","last_peered":"2024-09-18T07:47:37.199679+0000","last_clean":"2024-09-18T07:47:37.199679+0000","last_became_active":"2024-09-18T07:47:11.188162+0000","last_became_peered":"2024-09-18T07:47:11.188162+0000","last_unstale":"2024-09-18T07:47:37.199679+0000","last_undegraded":"2024-09-18T07:47:37.199679+0000","last_fullsized":"2024-09-18T07:47:37.199679+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:14.913596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:14.913596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:22.308133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"323'15","reported_seq":524,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.202283+0000","last_change":"2024-09-18T07:47:20.167453+0000","last_active":"2024-09-18T07:47:36.202283+0000","last_peered":"2024-09-18T07:47:36.202283+0000","last_clean":"2024-09-18T07:47:36.202283+0000","last_became_active":"2024-09-18T07:47:20.141411+0000","last_became_peered":"2024-09-18T07:47:20.141411+0000","last_unstale":"2024-09-18T07:47:36.202283+0000","last_undegraded":"2024-09-18T07:47:36.202283+0000","last_fullsized":"2024-09-18T07:47:36.202283+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:44:04.219913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:04.219913+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":656,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133118+0000","last_change":"2024-09-18T07:44:43.783005+0000","last_active":"2024-09-18T07:47:40.133118+0000","last_peered":"2024-09-18T07:47:40.133118+0000","last_clean":"2024-09-18T07:47:40.133118+0000","last_became_active":"2024-09-18T07:44:43.782709+0000","last_became_peered":"2024-09-18T07:44:43.782709+0000","last_unstale":"2024-09-18T07:47:40.133118+0000","last_undegraded":"2024-09-18T07:47:40.133118+0000","last_fullsized":"2024-09-18T07:47:40.133118+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:47.880911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:47.880911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:01.682226+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":385,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-18T07:47:37.199966+0000","last_change":"2024-09-18T07:47:20.135826+0000","last_active":"2024-09-18T07:47:37.199966+0000","last_peered":"2024-09-18T07:47:37.199966+0000","last_clean":"2024-09-18T07:47:37.199966+0000","last_became_active":"2024-09-18T07:47:20.135517+0000","last_became_peered":"2024-09-18T07:47:20.135517+0000","last_unstale":"2024-09-18T07:47:37.199966+0000","last_undegraded":"2024-09-18T07:47:37.199966+0000","last_fullsized":"2024-09-18T07:47:37.199966+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:13.911901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:13.911901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:35.013343+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":902,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-18T07:47:37.199537+0000","last_change":"2024-09-18T07:47:20.136279+0000","last_active":"2024-09-18T07:47:37.199537+0000","last_peered":"2024-09-18T07:47:37.199537+0000","last_clean":"2024-09-18T07:47:37.199537+0000","last_became_active":"2024-09-18T07:47:20.135880+0000","last_became_peered":"2024-09-18T07:47:20.135880+0000","last_unstale":"2024-09-18T07:47:37.199537+0000","last_undegraded":"2024-09-18T07:47:37.199537+0000","last_fullsized":"2024-09-18T07:47:37.199537+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:37.209823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:44:37.209823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"323'19","reported_seq":1000,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.138670+0000","last_change":"2024-09-18T07:46:19.041222+0000","last_active":"2024-09-18T07:47:40.138670+0000","last_peered":"2024-09-18T07:47:40.138670+0000","last_clean":"2024-09-18T07:47:40.138670+0000","last_became_active":"2024-09-18T07:37:43.066208+0000","last_became_peered":"2024-09-18T07:37:43.066208+0000","last_unstale":"2024-09-18T07:47:40.138670+0000","last_undegraded":"2024-09-18T07:47:40.138670+0000","last_fullsized":"2024-09-18T07:47:40.138670+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:46:19.041153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:19.041153+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":807,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234131+0000","last_change":"2024-09-18T07:47:20.140839+0000","last_active":"2024-09-18T07:47:38.234131+0000","last_peered":"2024-09-18T07:47:38.234131+0000","last_clean":"2024-09-18T07:47:38.234131+0000","last_became_active":"2024-09-18T07:47:20.140495+0000","last_became_peered":"2024-09-18T07:47:20.140495+0000","last_unstale":"2024-09-18T07:47:38.234131+0000","last_undegraded":"2024-09-18T07:47:38.234131+0000","last_fullsized":"2024-09-18T07:47:38.234131+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:32.657943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:43:32.657943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":678,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133593+0000","last_change":"2024-09-18T07:46:13.660302+0000","last_active":"2024-09-18T07:47:40.133593+0000","last_peered":"2024-09-18T07:47:40.133593+0000","last_clean":"2024-09-18T07:47:40.133593+0000","last_became_active":"2024-09-18T07:44:43.778318+0000","last_became_peered":"2024-09-18T07:44:43.778318+0000","last_unstale":"2024-09-18T07:47:40.133593+0000","last_undegraded":"2024-09-18T07:47:40.133593+0000","last_fullsized":"2024-09-18T07:47:40.133593+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:13.660254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:13.660254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:42.326916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":899,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.142692+0000","last_change":"2024-09-18T07:44:43.090466+0000","last_active":"2024-09-18T07:47:40.142692+0000","last_peered":"2024-09-18T07:47:40.142692+0000","last_clean":"2024-09-18T07:47:40.142692+0000","last_became_active":"2024-09-18T07:37:43.067066+0000","last_became_peered":"2024-09-18T07:37:43.067066+0000","last_unstale":"2024-09-18T07:47:40.142692+0000","last_undegraded":"2024-09-18T07:47:40.142692+0000","last_fullsized":"2024-09-18T07:47:40.142692+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:43.090421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:44:43.090421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:35.481417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"323'8","reported_seq":954,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.134265+0000","last_change":"2024-09-18T07:47:20.199058+0000","last_active":"2024-09-18T07:47:40.134265+0000","last_peered":"2024-09-18T07:47:40.134265+0000","last_clean":"2024-09-18T07:47:40.134265+0000","last_became_active":"2024-09-18T07:47:20.142156+0000","last_became_peered":"2024-09-18T07:47:20.142156+0000","last_unstale":"2024-09-18T07:47:40.134265+0000","last_undegraded":"2024-09-18T07:47:40.134265+0000","last_fullsized":"2024-09-18T07:47:40.134265+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'8","last_scrub_stamp":"2024-09-18T07:45:20.998406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:45:20.998406+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":46,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234338+0000","last_change":"2024-09-18T07:47:11.200765+0000","last_active":"2024-09-18T07:47:38.234338+0000","last_peered":"2024-09-18T07:47:38.234338+0000","last_clean":"2024-09-18T07:47:38.234338+0000","last_became_active":"2024-09-18T07:47:11.197716+0000","last_became_peered":"2024-09-18T07:47:11.197716+0000","last_unstale":"2024-09-18T07:47:38.234338+0000","last_undegraded":"2024-09-18T07:47:38.234338+0000","last_fullsized":"2024-09-18T07:47:38.234338+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:08.235926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:44:08.235926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":502,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.202909+0000","last_change":"2024-09-18T07:44:43.793061+0000","last_active":"2024-09-18T07:47:36.202909+0000","last_peered":"2024-09-18T07:47:36.202909+0000","last_clean":"2024-09-18T07:47:36.202909+0000","last_became_active":"2024-09-18T07:44:43.792641+0000","last_became_peered":"2024-09-18T07:44:43.792641+0000","last_unstale":"2024-09-18T07:47:36.202909+0000","last_undegraded":"2024-09-18T07:47:36.202909+0000","last_fullsized":"2024-09-18T07:47:36.202909+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:40.106921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.106921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:55.125341+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":656,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.132988+0000","last_change":"2024-09-18T07:44:43.783022+0000","last_active":"2024-09-18T07:47:40.132988+0000","last_peered":"2024-09-18T07:47:40.132988+0000","last_clean":"2024-09-18T07:47:40.132988+0000","last_became_active":"2024-09-18T07:44:43.782378+0000","last_became_peered":"2024-09-18T07:44:43.782378+0000","last_unstale":"2024-09-18T07:47:40.132988+0000","last_undegraded":"2024-09-18T07:47:40.132988+0000","last_fullsized":"2024-09-18T07:47:40.132988+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:38.492269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.492269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:37.092248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":452,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-09-18T07:47:38.234594+0000","last_change":"2024-09-18T07:47:20.139313+0000","last_active":"2024-09-18T07:47:38.234594+0000","last_peered":"2024-09-18T07:47:38.234594+0000","last_clean":"2024-09-18T07:47:38.234594+0000","last_became_active":"2024-09-18T07:47:20.138598+0000","last_became_peered":"2024-09-18T07:47:20.138598+0000","last_unstale":"2024-09-18T07:47:38.234594+0000","last_undegraded":"2024-09-18T07:47:38.234594+0000","last_fullsized":"2024-09-18T07:47:38.234594+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:44.492181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:44:44.492181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":932,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:41.464601+0000","last_change":"2024-09-18T07:47:41.464601+0000","last_active":"2024-09-18T07:47:41.464601+0000","last_peered":"2024-09-18T07:47:41.464601+0000","last_clean":"2024-09-18T07:47:41.464601+0000","last_became_active":"2024-09-18T07:47:20.143572+0000","last_became_peered":"2024-09-18T07:47:20.143572+0000","last_unstale":"2024-09-18T07:47:41.464601+0000","last_undegraded":"2024-09-18T07:47:41.464601+0000","last_fullsized":"2024-09-18T07:47:41.464601+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:41.464337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:41.464337+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:58.467470+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"323'19","reported_seq":570,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-18T07:47:36.205101+0000","last_change":"2024-09-18T07:47:20.160909+0000","last_active":"2024-09-18T07:47:36.205101+0000","last_peered":"2024-09-18T07:47:36.205101+0000","last_clean":"2024-09-18T07:47:36.205101+0000","last_became_active":"2024-09-18T07:47:20.142007+0000","last_became_peered":"2024-09-18T07:47:20.142007+0000","last_unstale":"2024-09-18T07:47:36.205101+0000","last_undegraded":"2024-09-18T07:47:36.205101+0000","last_fullsized":"2024-09-18T07:47:36.205101+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:43:10.104058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:43:10.104058+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":410,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-09-18T07:47:40.133217+0000","last_change":"2024-09-18T07:47:20.140033+0000","last_active":"2024-09-18T07:47:40.133217+0000","last_peered":"2024-09-18T07:47:40.133217+0000","last_clean":"2024-09-18T07:47:40.133217+0000","last_became_active":"2024-09-18T07:47:20.139668+0000","last_became_peered":"2024-09-18T07:47:20.139668+0000","last_unstale":"2024-09-18T07:47:40.133217+0000","last_undegraded":"2024-09-18T07:47:40.133217+0000","last_fullsized":"2024-09-18T07:47:40.133217+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:44.585789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:45:44.585789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":2472476672,"num_objects":37727,"num_object_clones":0,"num_object_copies":113181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37727,"num_write_kb":2414528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6736,"num_bytes_recovered":441450496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7476019200,"data_stored":7476019200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37727,"ondisk_log_size":37727,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":442,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1104,"num_read_kb":861,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":9230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},"log_size":688,"ondisk_log_size":688,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":762,"num_read_kb":757,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201485,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1576916,"kb_used_data":964032,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92168236,"statfs":{"total":95995035648,"available":94380273664,"internally_reserved":0,"allocated":987168768,"data_stored":986356055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":627571146},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463122,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1658580,"kb_used_data":1085952,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92086572,"statfs":{"total":95995035648,"available":94296649728,"internally_reserved":0,"allocated":1112014848,"data_stored":1111245393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":586348998},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":460,"seq":1975684956167,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2078456,"kb_used_data":1996004,"kb_used_omap":6,"kb_used_meta":82425,"kb_avail":91666696,"statfs":{"total":95995035648,"available":93866696704,"internally_reserved":0,"allocated":2043908096,"data_stored":2043116884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6877,"internal_metadata":84403491},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":395,"seq":1696512081957,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1033112,"kb_used_data":636164,"kb_used_omap":4,"kb_used_meta":396923,"kb_avail":92712040,"statfs":{"total":95995035648,"available":94937128960,"internally_reserved":0,"allocated":651431936,"data_stored":650614960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4447,"internal_metadata":406449825},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149923,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1816344,"kb_used_data":1113224,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":91928808,"statfs":{"total":95995035648,"available":94135099392,"internally_reserved":0,"allocated":1139941376,"data_stored":1139121700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":719976903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411561,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1558080,"kb_used_data":931308,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92187072,"statfs":{"total":95995035648,"available":94399561728,"internally_reserved":0,"allocated":953659392,"data_stored":952866863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575087,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1741340,"kb_used_data":1070148,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92003812,"statfs":{"total":95995035648,"available":94211903488,"internally_reserved":0,"allocated":1095831552,"data_stored":1094995340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738612,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1550588,"kb_used_data":819684,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92194564,"statfs":{"total":95995035648,"available":94407233536,"internally_reserved":0,"allocated":839356416,"data_stored":838552527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":837550080,"data_stored":837550080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1096744960,"data_stored":1096744960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":970653696,"data_stored":970653696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1138688000,"data_stored":1138688000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":656801792,"data_stored":656801792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":646578176,"data_stored":646578176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1112342528,"data_stored":1112342528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1016659968,"data_stored":1016659968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:47:45.150 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-09-18T07:47:45.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:45.279 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:45.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:45.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:45.381 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:45 smithi137 ceph-mon[29408]: from='client.16629 v1:172.21.15.137:0/3144204293' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:47:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:45 smithi137 ceph-mon[29408]: pgmap v896: 158 pgs: 158 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 771 B/s rd, 63 MiB/s wr, 1.01k op/s; 3.2 MiB/s, 51 objects/s recovering 2024-09-18T07:47:45.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:45 smithi137 ceph-mon[29408]: 6.10 scrub starts 2024-09-18T07:47:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:45 smithi137 ceph-mon[29408]: 6.10 scrub ok 2024-09-18T07:47:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:45 smithi137 ceph-mon[29408]: 6.1f scrub starts 2024-09-18T07:47:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:45 smithi137 ceph-mon[29408]: 6.1f scrub ok 2024-09-18T07:47:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:45 smithi137 ceph-mon[29408]: 6.7 scrub starts 2024-09-18T07:47:45.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:45 smithi137 ceph-mon[29408]: 6.7 scrub ok 2024-09-18T07:47:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:45 smithi137 ceph-mon[39175]: from='client.16629 v1:172.21.15.137:0/3144204293' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:47:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:45 smithi137 ceph-mon[39175]: pgmap v896: 158 pgs: 158 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 771 B/s rd, 63 MiB/s wr, 1.01k op/s; 3.2 MiB/s, 51 objects/s recovering 2024-09-18T07:47:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:45 smithi137 ceph-mon[39175]: 6.10 scrub starts 2024-09-18T07:47:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:45 smithi137 ceph-mon[39175]: 6.10 scrub ok 2024-09-18T07:47:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:45 smithi137 ceph-mon[39175]: 6.1f scrub starts 2024-09-18T07:47:45.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:45 smithi137 ceph-mon[39175]: 6.1f scrub ok 2024-09-18T07:47:45.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:45 smithi137 ceph-mon[39175]: 6.7 scrub starts 2024-09-18T07:47:45.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:45 smithi137 ceph-mon[39175]: 6.7 scrub ok 2024-09-18T07:47:45.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:45.422 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:45.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:45.523 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:45.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:45.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:45.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:45.664 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:45.695 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:45.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:45.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:45.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:45 smithi203 ceph-mon[32237]: from='client.16629 v1:172.21.15.137:0/3144204293' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:47:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:45 smithi203 ceph-mon[32237]: pgmap v896: 158 pgs: 158 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 771 B/s rd, 63 MiB/s wr, 1.01k op/s; 3.2 MiB/s, 51 objects/s recovering 2024-09-18T07:47:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:45 smithi203 ceph-mon[32237]: 6.10 scrub starts 2024-09-18T07:47:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:45 smithi203 ceph-mon[32237]: 6.10 scrub ok 2024-09-18T07:47:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:45 smithi203 ceph-mon[32237]: 6.1f scrub starts 2024-09-18T07:47:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:45 smithi203 ceph-mon[32237]: 6.1f scrub ok 2024-09-18T07:47:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:45 smithi203 ceph-mon[32237]: 6.7 scrub starts 2024-09-18T07:47:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:45 smithi203 ceph-mon[32237]: 6.7 scrub ok 2024-09-18T07:47:45.798 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:45.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:45.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:45.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:45.933 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:45.954 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T07:47:45.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:45.965 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:45.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:46.049 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 31 16 47575 47559 95.8639 106.125 0.0107509 0.0104255 2024-09-18T07:47:46.066 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:46.067 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:46.076 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:46.093 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:46.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:46.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:46.194 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:46.223 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:46.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:46.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:46.324 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:46.367 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:46.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:46.469 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:46.504 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:46.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:46.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:46.538 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:46 smithi137 ceph-mon[29408]: 6.16 scrub starts 2024-09-18T07:47:46.538 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:46 smithi137 ceph-mon[29408]: 6.16 scrub ok 2024-09-18T07:47:46.538 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:46 smithi137 ceph-mon[29408]: 6.0 scrub starts 2024-09-18T07:47:46.538 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:46 smithi137 ceph-mon[29408]: 6.0 scrub ok 2024-09-18T07:47:46.538 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:46 smithi137 ceph-mon[39175]: 6.16 scrub starts 2024-09-18T07:47:46.539 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:46 smithi137 ceph-mon[39175]: 6.16 scrub ok 2024-09-18T07:47:46.539 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:46 smithi137 ceph-mon[39175]: 6.0 scrub starts 2024-09-18T07:47:46.539 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:46 smithi137 ceph-mon[39175]: 6.0 scrub ok 2024-09-18T07:47:46.606 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:46.646 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:46.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:46.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:46.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:46 smithi203 ceph-mon[32237]: 6.16 scrub starts 2024-09-18T07:47:46.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:46 smithi203 ceph-mon[32237]: 6.16 scrub ok 2024-09-18T07:47:46.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:46 smithi203 ceph-mon[32237]: 6.0 scrub starts 2024-09-18T07:47:46.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:46 smithi203 ceph-mon[32237]: 6.0 scrub ok 2024-09-18T07:47:46.748 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:46.783 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:46.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:46.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:46.884 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:46.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:46.915 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:46.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:47.016 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:47.048 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:47.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:47.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:47.050 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 32 16 49127 49111 95.8991 97 0.00733763 0.010422 2024-09-18T07:47:47.151 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:47.177 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:47.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:47.222 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:47:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:47:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:47:47.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:47.289 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:47:47.303 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:47.314 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:47:47.314 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:47:47.336 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:47.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:47.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:47.356 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:47:47.357 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:47:47.395 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:47:47.395 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:47:47.426 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:47:47.426 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:47:47.440 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:47.456 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:47:47.456 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:47:47.483 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:47.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:47.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:47.489 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:47:47.490 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:47:47.490 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:47 smithi137 ceph-mon[29408]: pgmap v897: 158 pgs: 158 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 905 op/s; 2.7 MiB/s, 42 objects/s recovering 2024-09-18T07:47:47.490 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:47 smithi137 ceph-mon[29408]: 4.10 scrub starts 2024-09-18T07:47:47.490 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:47 smithi137 ceph-mon[29408]: 4.10 scrub ok 2024-09-18T07:47:47.490 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:47 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1877838104' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:47.490 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:47 smithi137 ceph-mon[29408]: from='client.26569 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:47.491 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:47 smithi137 ceph-mon[39175]: pgmap v897: 158 pgs: 158 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 905 op/s; 2.7 MiB/s, 42 objects/s recovering 2024-09-18T07:47:47.491 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:47 smithi137 ceph-mon[39175]: 4.10 scrub starts 2024-09-18T07:47:47.491 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:47 smithi137 ceph-mon[39175]: 4.10 scrub ok 2024-09-18T07:47:47.491 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:47 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1877838104' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:47.491 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:47 smithi137 ceph-mon[39175]: from='client.26569 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:47.522 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:47:47.522 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:47:47.556 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:47:47.556 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:47:47.586 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:47.613 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:47.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:47.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:47.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:47 smithi203 ceph-mon[32237]: pgmap v897: 158 pgs: 158 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 905 op/s; 2.7 MiB/s, 42 objects/s recovering 2024-09-18T07:47:47.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:47 smithi203 ceph-mon[32237]: 4.10 scrub starts 2024-09-18T07:47:47.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:47 smithi203 ceph-mon[32237]: 4.10 scrub ok 2024-09-18T07:47:47.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:47 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1877838104' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:47.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:47 smithi203 ceph-mon[32237]: from='client.26569 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:47:47.716 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:47.749 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:47.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:47.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:47.851 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:47.884 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:47.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:47.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:47.986 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:48.022 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:48.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:48.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:48.049 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 33 15 50629 50614 95.8394 93.9375 0.015636 0.0104294 2024-09-18T07:47:48.124 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:48.157 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:48.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:48.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:48.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:47:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:47:48.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:48.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:48.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:48.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:48.407 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:48.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:48.439 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:48.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:48.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:48.572 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:48.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:48.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:48.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[29408]: 4.1a scrub starts 2024-09-18T07:47:48.656 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[29408]: 4.1a scrub ok 2024-09-18T07:47:48.656 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3477515344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:48.656 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:47:48.656 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:47:48.656 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[29408]: from='client.26569 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:47:48.656 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[29408]: osdmap e478: 8 total, 8 up, 8 in 2024-09-18T07:47:48.656 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[29408]: 4.1f scrub starts 2024-09-18T07:47:48.656 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[29408]: 4.1f scrub ok 2024-09-18T07:47:48.656 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[29408]: 3.2 scrub starts 2024-09-18T07:47:48.656 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[29408]: 3.2 scrub ok 2024-09-18T07:47:48.656 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[29408]: 3.6 scrub starts 2024-09-18T07:47:48.656 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[29408]: 3.6 scrub ok 2024-09-18T07:47:48.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[39175]: 4.1a scrub starts 2024-09-18T07:47:48.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[39175]: 4.1a scrub ok 2024-09-18T07:47:48.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3477515344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:48.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:47:48.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:47:48.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[39175]: from='client.26569 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:47:48.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[39175]: osdmap e478: 8 total, 8 up, 8 in 2024-09-18T07:47:48.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[39175]: 4.1f scrub starts 2024-09-18T07:47:48.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[39175]: 4.1f scrub ok 2024-09-18T07:47:48.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[39175]: 3.2 scrub starts 2024-09-18T07:47:48.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[39175]: 3.2 scrub ok 2024-09-18T07:47:48.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[39175]: 3.6 scrub starts 2024-09-18T07:47:48.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:48 smithi137 ceph-mon[39175]: 3.6 scrub ok 2024-09-18T07:47:48.673 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:48.701 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:48.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:48.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:48.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:48 smithi203 ceph-mon[32237]: 4.1a scrub starts 2024-09-18T07:47:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:48 smithi203 ceph-mon[32237]: 4.1a scrub ok 2024-09-18T07:47:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:48 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3477515344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:48 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:47:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:48 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:47:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:48 smithi203 ceph-mon[32237]: from='client.26569 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:47:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:48 smithi203 ceph-mon[32237]: osdmap e478: 8 total, 8 up, 8 in 2024-09-18T07:47:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:48 smithi203 ceph-mon[32237]: 4.1f scrub starts 2024-09-18T07:47:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:48 smithi203 ceph-mon[32237]: 4.1f scrub ok 2024-09-18T07:47:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:48 smithi203 ceph-mon[32237]: 3.2 scrub starts 2024-09-18T07:47:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:48 smithi203 ceph-mon[32237]: 3.2 scrub ok 2024-09-18T07:47:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:48 smithi203 ceph-mon[32237]: 3.6 scrub starts 2024-09-18T07:47:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:48 smithi203 ceph-mon[32237]: 3.6 scrub ok 2024-09-18T07:47:48.804 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:48.834 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:48.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:48.935 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:48.961 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:48.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:48.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:49.050 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 34 16 52241 52225 95.9817 100.688 0.00885756 0.0104135 2024-09-18T07:47:49.063 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:49.095 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:49.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:49.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:49.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:49.230 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:49.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:49.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:49.333 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:49.368 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:49.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:49.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:49.511 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:49.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:49.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:49.613 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:49.617 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:49 smithi137 ceph-mon[29408]: pgmap v899: 158 pgs: 158 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-09-18T07:47:49.617 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:49 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:49.617 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:49 smithi137 ceph-mon[29408]: 6.19 scrub starts 2024-09-18T07:47:49.617 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:49 smithi137 ceph-mon[29408]: 6.19 scrub ok 2024-09-18T07:47:49.617 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:49 smithi137 ceph-mon[29408]: 3.1b scrub starts 2024-09-18T07:47:49.617 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:49 smithi137 ceph-mon[29408]: 3.1b scrub ok 2024-09-18T07:47:49.617 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:49 smithi137 ceph-mon[39175]: pgmap v899: 158 pgs: 158 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-09-18T07:47:49.617 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:49 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:49.618 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:49 smithi137 ceph-mon[39175]: 6.19 scrub starts 2024-09-18T07:47:49.618 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:49 smithi137 ceph-mon[39175]: 6.19 scrub ok 2024-09-18T07:47:49.618 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:49 smithi137 ceph-mon[39175]: 3.1b scrub starts 2024-09-18T07:47:49.618 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:49 smithi137 ceph-mon[39175]: 3.1b scrub ok 2024-09-18T07:47:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:49.656 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:49.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:49 smithi203 ceph-mon[32237]: pgmap v899: 158 pgs: 158 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-09-18T07:47:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:49 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:49 smithi203 ceph-mon[32237]: 6.19 scrub starts 2024-09-18T07:47:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:49 smithi203 ceph-mon[32237]: 6.19 scrub ok 2024-09-18T07:47:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:49 smithi203 ceph-mon[32237]: 3.1b scrub starts 2024-09-18T07:47:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:49 smithi203 ceph-mon[32237]: 3.1b scrub ok 2024-09-18T07:47:49.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:49.787 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:47:49.787 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:47:49.792 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:49.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:49.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:49.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:49.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:49.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:50.042 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:50.050 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 35 16 53871 53855 96.1499 101.875 0.00480094 0.0103948 2024-09-18T07:47:50.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:50.068 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:50.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:50.150 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 6, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 6, 7, 4, 5] 2024-09-18T07:47:50.150 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:47:50.150 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-18T07:47:50.150 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.510739 2024-09-18T07:47:50.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 0.5107391301161092 2024-09-18T07:47:50.169 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:50.198 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:50.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:50.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:50.300 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:50.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:50.329 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:50.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:50.430 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:50.458 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:50.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:50.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:50.541 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:50 smithi137 ceph-mon[39175]: 4.12 scrub starts 2024-09-18T07:47:50.541 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:50 smithi137 ceph-mon[39175]: 4.12 scrub ok 2024-09-18T07:47:50.541 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:50 smithi137 ceph-mon[39175]: 6.8 scrub starts 2024-09-18T07:47:50.541 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:50 smithi137 ceph-mon[39175]: 6.8 scrub ok 2024-09-18T07:47:50.541 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:50 smithi137 ceph-mon[39175]: osdmap e479: 8 total, 8 up, 8 in 2024-09-18T07:47:50.543 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:50 smithi137 ceph-mon[29408]: 4.12 scrub starts 2024-09-18T07:47:50.543 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:50 smithi137 ceph-mon[29408]: 4.12 scrub ok 2024-09-18T07:47:50.543 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:50 smithi137 ceph-mon[29408]: 6.8 scrub starts 2024-09-18T07:47:50.543 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:50 smithi137 ceph-mon[29408]: 6.8 scrub ok 2024-09-18T07:47:50.543 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:50 smithi137 ceph-mon[29408]: osdmap e479: 8 total, 8 up, 8 in 2024-09-18T07:47:50.560 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:50.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T07:47:50.612 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:50.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:50.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:50.704 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:50.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:50 smithi203 ceph-mon[32237]: 4.12 scrub starts 2024-09-18T07:47:50.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:50 smithi203 ceph-mon[32237]: 4.12 scrub ok 2024-09-18T07:47:50.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:50 smithi203 ceph-mon[32237]: 6.8 scrub starts 2024-09-18T07:47:50.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:50 smithi203 ceph-mon[32237]: 6.8 scrub ok 2024-09-18T07:47:50.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:50 smithi203 ceph-mon[32237]: osdmap e479: 8 total, 8 up, 8 in 2024-09-18T07:47:50.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:47:50.714 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:50.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:50.792 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:50.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:50.893 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:50.918 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:50.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:50.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:51.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:51.050 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 36 16 55181 55165 95.7532 81.875 0.00904169 0.0104381 2024-09-18T07:47:51.060 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:51.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:51.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:51.163 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:51.196 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:51.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:51.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:51.298 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:51.342 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:51.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:51.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:51.446 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:51.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:51.476 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:51.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:51.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:47:51.530 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:51 smithi137 ceph-mon[29408]: pgmap v900: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-18T07:47:51.530 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:51 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:47:51.530 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:51 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:47:51.531 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:51 smithi137 ceph-mon[29408]: 3.c scrub starts 2024-09-18T07:47:51.531 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:51 smithi137 ceph-mon[29408]: 3.c scrub ok 2024-09-18T07:47:51.531 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:51 smithi137 ceph-mon[39175]: pgmap v900: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-18T07:47:51.531 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:51 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:47:51.531 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:51 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:47:51.531 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:51 smithi137 ceph-mon[39175]: 3.c scrub starts 2024-09-18T07:47:51.531 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:51 smithi137 ceph-mon[39175]: 3.c scrub ok 2024-09-18T07:47:51.577 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:51.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:51.607 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:51.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:51.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:51.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:51 smithi203 ceph-mon[32237]: pgmap v900: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-18T07:47:51.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:51 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:47:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:51 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:47:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:51 smithi203 ceph-mon[32237]: 3.c scrub starts 2024-09-18T07:47:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:51 smithi203 ceph-mon[32237]: 3.c scrub ok 2024-09-18T07:47:51.734 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:51.740 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:47:51.810 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:51.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:51.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:51.911 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:51.967 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:51.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:51.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:52.050 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 37 16 56499 56483 95.3914 82.375 0.00608958 0.0104784 2024-09-18T07:47:52.069 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:52.100 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:52.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:52.202 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:52.232 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:52.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:52.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:52.331 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.0 primary-affinity to 0.510739 (8334712) 2024-09-18T07:47:52.334 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:52.356 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:52.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:52.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[29408]: 3.3 scrub starts 2024-09-18T07:47:52.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[29408]: 3.3 scrub ok 2024-09-18T07:47:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[29408]: osdmap e480: 8 total, 8 up, 8 in 2024-09-18T07:47:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3303295633' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.5107391301161092} : dispatch 2024-09-18T07:47:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[29408]: from='client.36482 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.5107391301161092} : dispatch 2024-09-18T07:47:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[29408]: 3.12 scrub starts 2024-09-18T07:47:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[29408]: 3.12 scrub ok 2024-09-18T07:47:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[29408]: 5.18 scrub starts 2024-09-18T07:47:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[29408]: 5.18 scrub ok 2024-09-18T07:47:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[29408]: 3.16 scrub starts 2024-09-18T07:47:52.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[29408]: 3.16 scrub ok 2024-09-18T07:47:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[39175]: 3.3 scrub starts 2024-09-18T07:47:52.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[39175]: 3.3 scrub ok 2024-09-18T07:47:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[39175]: osdmap e480: 8 total, 8 up, 8 in 2024-09-18T07:47:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3303295633' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.5107391301161092} : dispatch 2024-09-18T07:47:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[39175]: from='client.36482 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.5107391301161092} : dispatch 2024-09-18T07:47:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[39175]: 3.12 scrub starts 2024-09-18T07:47:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[39175]: 3.12 scrub ok 2024-09-18T07:47:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[39175]: 5.18 scrub starts 2024-09-18T07:47:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[39175]: 5.18 scrub ok 2024-09-18T07:47:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[39175]: 3.16 scrub starts 2024-09-18T07:47:52.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:52 smithi137 ceph-mon[39175]: 3.16 scrub ok 2024-09-18T07:47:52.434 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:47:52.435 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:47:52.458 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:52.486 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:52.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:52.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:52.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:52.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:52.613 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:52.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:52 smithi203 ceph-mon[32237]: 3.3 scrub starts 2024-09-18T07:47:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:52 smithi203 ceph-mon[32237]: 3.3 scrub ok 2024-09-18T07:47:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:52 smithi203 ceph-mon[32237]: osdmap e480: 8 total, 8 up, 8 in 2024-09-18T07:47:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:52 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3303295633' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.5107391301161092} : dispatch 2024-09-18T07:47:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:52 smithi203 ceph-mon[32237]: from='client.36482 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.5107391301161092} : dispatch 2024-09-18T07:47:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:52 smithi203 ceph-mon[32237]: 3.12 scrub starts 2024-09-18T07:47:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:52 smithi203 ceph-mon[32237]: 3.12 scrub ok 2024-09-18T07:47:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:52 smithi203 ceph-mon[32237]: 5.18 scrub starts 2024-09-18T07:47:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:52 smithi203 ceph-mon[32237]: 5.18 scrub ok 2024-09-18T07:47:52.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:52 smithi203 ceph-mon[32237]: 3.16 scrub starts 2024-09-18T07:47:52.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:52 smithi203 ceph-mon[32237]: 3.16 scrub ok 2024-09-18T07:47:52.714 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:52.735 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:52.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:52.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:52.837 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:52.859 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:52.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:52.961 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:52.983 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:52.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:52.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:53.050 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 38 16 57040 57024 93.7709 33.8125 0.00255527 0.0104774 2024-09-18T07:47:53.085 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:53.106 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:53.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:53.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:53.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:53.231 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:53.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:53.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:53.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:53.365 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:53.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:53.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:53.505 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:53.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:53.606 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:53.628 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:53.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:53.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:53.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[29408]: pgmap v903: 158 pgs: 158 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 137 MiB/s wr, 2.20k op/s 2024-09-18T07:47:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[29408]: from='client.36482 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.5107391301161092}]': finished 2024-09-18T07:47:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[29408]: osdmap e481: 8 total, 8 up, 8 in 2024-09-18T07:47:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[29408]: 4.a scrub starts 2024-09-18T07:47:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[29408]: 4.a scrub ok 2024-09-18T07:47:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[29408]: 6.1c scrub starts 2024-09-18T07:47:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[29408]: 6.1c scrub ok 2024-09-18T07:47:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[29408]: 5.1 scrub starts 2024-09-18T07:47:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[29408]: 5.1 scrub ok 2024-09-18T07:47:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[29408]: 5.16 scrub starts 2024-09-18T07:47:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[29408]: 5.16 scrub ok 2024-09-18T07:47:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[29408]: osdmap e482: 8 total, 8 up, 8 in 2024-09-18T07:47:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[39175]: pgmap v903: 158 pgs: 158 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 137 MiB/s wr, 2.20k op/s 2024-09-18T07:47:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[39175]: from='client.36482 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.5107391301161092}]': finished 2024-09-18T07:47:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[39175]: osdmap e481: 8 total, 8 up, 8 in 2024-09-18T07:47:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[39175]: 4.a scrub starts 2024-09-18T07:47:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[39175]: 4.a scrub ok 2024-09-18T07:47:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[39175]: 6.1c scrub starts 2024-09-18T07:47:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[39175]: 6.1c scrub ok 2024-09-18T07:47:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[39175]: 5.1 scrub starts 2024-09-18T07:47:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[39175]: 5.1 scrub ok 2024-09-18T07:47:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[39175]: 5.16 scrub starts 2024-09-18T07:47:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[39175]: 5.16 scrub ok 2024-09-18T07:47:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:53 smithi137 ceph-mon[39175]: osdmap e482: 8 total, 8 up, 8 in 2024-09-18T07:47:53.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:53 smithi203 ceph-mon[32237]: pgmap v903: 158 pgs: 158 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 137 MiB/s wr, 2.20k op/s 2024-09-18T07:47:53.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:53 smithi203 ceph-mon[32237]: from='client.36482 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.5107391301161092}]': finished 2024-09-18T07:47:53.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:53 smithi203 ceph-mon[32237]: osdmap e481: 8 total, 8 up, 8 in 2024-09-18T07:47:53.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:53 smithi203 ceph-mon[32237]: 4.a scrub starts 2024-09-18T07:47:53.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:53 smithi203 ceph-mon[32237]: 4.a scrub ok 2024-09-18T07:47:53.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:53 smithi203 ceph-mon[32237]: 6.1c scrub starts 2024-09-18T07:47:53.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:53 smithi203 ceph-mon[32237]: 6.1c scrub ok 2024-09-18T07:47:53.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:53 smithi203 ceph-mon[32237]: 5.1 scrub starts 2024-09-18T07:47:53.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:53 smithi203 ceph-mon[32237]: 5.1 scrub ok 2024-09-18T07:47:53.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:53 smithi203 ceph-mon[32237]: 5.16 scrub starts 2024-09-18T07:47:53.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:53 smithi203 ceph-mon[32237]: 5.16 scrub ok 2024-09-18T07:47:53.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:53 smithi203 ceph-mon[32237]: osdmap e482: 8 total, 8 up, 8 in 2024-09-18T07:47:53.730 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:53.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:53.767 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:53.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:53.868 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:53.900 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:53.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:53.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:54.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:54.039 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:54.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:54.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:54.050 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 39 16 58027 58011 92.9482 61.6875 0.00909912 0.0107531 2024-09-18T07:47:54.141 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:54.165 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:54.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:54.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:54.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:54.301 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:54.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:54.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:54.402 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:54.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:54.432 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:54.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:54.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:54.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:54.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:54.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:54.673 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:54.702 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:54.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:54.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:54.804 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:54.834 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:54.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:54.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:54.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:54.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:54.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:54.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:55.050 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:47:55.048751+0000 min lat: 0.00129363 max lat: 1.97884 avg lat: 0.0107434 2024-09-18T07:47:55.050 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:47:55.050 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 40 16 59572 59556 93.0383 96.5625 0.00575369 0.0107434 2024-09-18T07:47:55.073 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:55.103 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:55.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:55.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:55.205 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:55.242 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:55.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:55.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:55.344 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:55.408 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:55.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:55.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:55.510 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:55.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:55.544 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:55.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:55.645 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:55.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:55 smithi137 ceph-mon[29408]: pgmap v906: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-09-18T07:47:55.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:55 smithi137 ceph-mon[29408]: 4.1d scrub starts 2024-09-18T07:47:55.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:55 smithi137 ceph-mon[29408]: 4.1d scrub ok 2024-09-18T07:47:55.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:55 smithi137 ceph-mon[29408]: 6.18 scrub starts 2024-09-18T07:47:55.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:55 smithi137 ceph-mon[29408]: 6.18 scrub ok 2024-09-18T07:47:55.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:55 smithi137 ceph-mon[29408]: osdmap e483: 8 total, 8 up, 8 in 2024-09-18T07:47:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:55 smithi137 ceph-mon[39175]: pgmap v906: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-09-18T07:47:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:55 smithi137 ceph-mon[39175]: 4.1d scrub starts 2024-09-18T07:47:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:55 smithi137 ceph-mon[39175]: 4.1d scrub ok 2024-09-18T07:47:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:55 smithi137 ceph-mon[39175]: 6.18 scrub starts 2024-09-18T07:47:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:55 smithi137 ceph-mon[39175]: 6.18 scrub ok 2024-09-18T07:47:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:55 smithi137 ceph-mon[39175]: osdmap e483: 8 total, 8 up, 8 in 2024-09-18T07:47:55.676 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:55.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:55.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:55 smithi203 ceph-mon[32237]: pgmap v906: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-09-18T07:47:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:55 smithi203 ceph-mon[32237]: 4.1d scrub starts 2024-09-18T07:47:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:55 smithi203 ceph-mon[32237]: 4.1d scrub ok 2024-09-18T07:47:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:55 smithi203 ceph-mon[32237]: 6.18 scrub starts 2024-09-18T07:47:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:55 smithi203 ceph-mon[32237]: 6.18 scrub ok 2024-09-18T07:47:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:55 smithi203 ceph-mon[32237]: osdmap e483: 8 total, 8 up, 8 in 2024-09-18T07:47:55.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:55.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:55.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:55.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:55.915 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:55.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:55.947 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:55.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:56.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:56.051 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 41 16 60965 60949 92.8924 87.0625 0.00277861 0.0107566 2024-09-18T07:47:56.070 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:56.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:56.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:56.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:56.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:56.210 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:56.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:56.312 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:56.333 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:56.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:56.434 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:56.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:56.466 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:56.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:56.566 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:56.599 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:56.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:56.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:56 smithi137 ceph-mon[29408]: 5.11 scrub starts 2024-09-18T07:47:56.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:56 smithi137 ceph-mon[29408]: 5.11 scrub ok 2024-09-18T07:47:56.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:56 smithi137 ceph-mon[29408]: pgmap v908: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-18T07:47:56.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:56 smithi137 ceph-mon[29408]: 4.7 scrub starts 2024-09-18T07:47:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:56 smithi137 ceph-mon[29408]: 4.7 scrub ok 2024-09-18T07:47:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:56 smithi137 ceph-mon[29408]: 3.7 scrub starts 2024-09-18T07:47:56.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:56 smithi137 ceph-mon[29408]: 3.7 scrub ok 2024-09-18T07:47:56.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:56 smithi137 ceph-mon[39175]: 5.11 scrub starts 2024-09-18T07:47:56.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:56 smithi137 ceph-mon[39175]: 5.11 scrub ok 2024-09-18T07:47:56.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:56 smithi137 ceph-mon[39175]: pgmap v908: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-18T07:47:56.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:56 smithi137 ceph-mon[39175]: 4.7 scrub starts 2024-09-18T07:47:56.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:56 smithi137 ceph-mon[39175]: 4.7 scrub ok 2024-09-18T07:47:56.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:56 smithi137 ceph-mon[39175]: 3.7 scrub starts 2024-09-18T07:47:56.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:56 smithi137 ceph-mon[39175]: 3.7 scrub ok 2024-09-18T07:47:56.701 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:56 smithi203 ceph-mon[32237]: 5.11 scrub starts 2024-09-18T07:47:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:56 smithi203 ceph-mon[32237]: 5.11 scrub ok 2024-09-18T07:47:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:56 smithi203 ceph-mon[32237]: pgmap v908: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-18T07:47:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:56 smithi203 ceph-mon[32237]: 4.7 scrub starts 2024-09-18T07:47:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:56 smithi203 ceph-mon[32237]: 4.7 scrub ok 2024-09-18T07:47:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:56 smithi203 ceph-mon[32237]: 3.7 scrub starts 2024-09-18T07:47:56.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:56 smithi203 ceph-mon[32237]: 3.7 scrub ok 2024-09-18T07:47:56.735 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:56.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:56.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:56.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:56.872 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:56.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:56.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:56.974 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:57.009 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:57.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:57.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:57.050 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 42 16 62300 62284 92.6671 83.4375 0.0131422 0.0107851 2024-09-18T07:47:57.111 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:57.150 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:57.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:57.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:57.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:57.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:57.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:57.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:57.352 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:47:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:47:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:47:57.398 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:57.431 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:57.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:57.532 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:57.565 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:57.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:57.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:57.668 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:57.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:57 smithi137 ceph-mon[29408]: 3.1a scrub starts 2024-09-18T07:47:57.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:57 smithi137 ceph-mon[29408]: 3.1a scrub ok 2024-09-18T07:47:57.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:57 smithi137 ceph-mon[29408]: 4.0 scrub starts 2024-09-18T07:47:57.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:57 smithi137 ceph-mon[29408]: 4.0 scrub ok 2024-09-18T07:47:57.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:57 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:57.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:57 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:47:57.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:57 smithi137 ceph-mon[39175]: 3.1a scrub starts 2024-09-18T07:47:57.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:57 smithi137 ceph-mon[39175]: 3.1a scrub ok 2024-09-18T07:47:57.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:57 smithi137 ceph-mon[39175]: 4.0 scrub starts 2024-09-18T07:47:57.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:57 smithi137 ceph-mon[39175]: 4.0 scrub ok 2024-09-18T07:47:57.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:57 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:57.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:57 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:47:57.697 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:57.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:57.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:57.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:57 smithi203 ceph-mon[32237]: 3.1a scrub starts 2024-09-18T07:47:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:57 smithi203 ceph-mon[32237]: 3.1a scrub ok 2024-09-18T07:47:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:57 smithi203 ceph-mon[32237]: 4.0 scrub starts 2024-09-18T07:47:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:57 smithi203 ceph-mon[32237]: 4.0 scrub ok 2024-09-18T07:47:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:57 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:47:57.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:57 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:47:57.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:57.837 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:57.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:57.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:57.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:47:57.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:57.978 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:57.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:47:58.051 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 43 16 63510 63494 92.2706 75.625 0.00878559 0.0108321 2024-09-18T07:47:58.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:58.115 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:58.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:58.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:47:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:47:58.216 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:58.252 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:58.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:58.354 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:58.395 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:58.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:58.497 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:47:58.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:58.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:58.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:47:58.631 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:47:58.663 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:58.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:58.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:47:58.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:58 smithi137 ceph-mon[29408]: 3.1e scrub starts 2024-09-18T07:47:58.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:58 smithi137 ceph-mon[29408]: 3.1e scrub ok 2024-09-18T07:47:58.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:58 smithi137 ceph-mon[29408]: pgmap v909: 158 pgs: 158 active+clean; 3.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 511 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-18T07:47:58.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:58 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/814088389' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:58.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:58 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:58.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:58 smithi137 ceph-mon[39175]: 3.1e scrub starts 2024-09-18T07:47:58.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:58 smithi137 ceph-mon[39175]: 3.1e scrub ok 2024-09-18T07:47:58.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:58 smithi137 ceph-mon[39175]: pgmap v909: 158 pgs: 158 active+clean; 3.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 511 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-18T07:47:58.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:58 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/814088389' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:58.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:58 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:58.675 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:47:58.676 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:47:58.676 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:47:58.676 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:47:58.676 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:47:58.676 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:47:58.676 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:47:58.676 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:47:58.676 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:47:58.676 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:47:58.677 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:47:58.677 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:47:58.677 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:47:58.677 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:47:58.677 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:47:58.677 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:47:58.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:58 smithi203 ceph-mon[32237]: 3.1e scrub starts 2024-09-18T07:47:58.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:58 smithi203 ceph-mon[32237]: 3.1e scrub ok 2024-09-18T07:47:58.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:58 smithi203 ceph-mon[32237]: pgmap v909: 158 pgs: 158 active+clean; 3.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 511 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-18T07:47:58.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:58 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/814088389' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:47:58.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:58 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:47:58.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:58.800 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:58.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:58.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:58.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:58.931 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:58.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:58.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:59.033 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:59.051 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 44 16 64925 64909 92.1831 88.4375 0.0147859 0.0108418 2024-09-18T07:47:59.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:59.069 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:59.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:59.170 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:47:59.192 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:59.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:59.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:47:59.294 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:59.321 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:59.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:59.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:59.423 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:47:59.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:59.465 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:59.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:47:59.566 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:47:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:59.599 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:59.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:47:59.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:59 smithi203 ceph-mon[32237]: 4.5 scrub starts 2024-09-18T07:47:59.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:59 smithi203 ceph-mon[32237]: 4.5 scrub ok 2024-09-18T07:47:59.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:59 smithi203 ceph-mon[32237]: 5.12 scrub starts 2024-09-18T07:47:59.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:47:59 smithi203 ceph-mon[32237]: 5.12 scrub ok 2024-09-18T07:47:59.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:59 smithi137 ceph-mon[29408]: 4.5 scrub starts 2024-09-18T07:47:59.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:59 smithi137 ceph-mon[29408]: 4.5 scrub ok 2024-09-18T07:47:59.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:59 smithi137 ceph-mon[29408]: 5.12 scrub starts 2024-09-18T07:47:59.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:47:59 smithi137 ceph-mon[29408]: 5.12 scrub ok 2024-09-18T07:47:59.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:59 smithi137 ceph-mon[39175]: 4.5 scrub starts 2024-09-18T07:47:59.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:59 smithi137 ceph-mon[39175]: 4.5 scrub ok 2024-09-18T07:47:59.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:59 smithi137 ceph-mon[39175]: 5.12 scrub starts 2024-09-18T07:47:59.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:47:59 smithi137 ceph-mon[39175]: 5.12 scrub ok 2024-09-18T07:47:59.699 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:47:59.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:59.730 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:47:59.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:47:59.831 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:47:59.873 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:47:59.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:47:59.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:47:59.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:00.006 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:00.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:00.051 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 45 16 66244 66228 91.9664 82.4375 0.00313092 0.0108656 2024-09-18T07:48:00.108 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:00.144 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:00.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:00.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:00.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:00.280 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:00.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:00.381 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:00.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:00.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:00.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:00.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:00.557 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:00.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:00.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:00.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:00 smithi137 ceph-mon[29408]: 1.6 scrub starts 2024-09-18T07:48:00.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:00 smithi137 ceph-mon[29408]: 1.6 scrub ok 2024-09-18T07:48:00.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:00 smithi137 ceph-mon[29408]: pgmap v910: 158 pgs: 158 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 408 B/s rd, 54 MiB/s wr, 862 op/s 2024-09-18T07:48:00.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:00 smithi137 ceph-mon[39175]: 1.6 scrub starts 2024-09-18T07:48:00.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:00 smithi137 ceph-mon[39175]: 1.6 scrub ok 2024-09-18T07:48:00.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:00 smithi137 ceph-mon[39175]: pgmap v910: 158 pgs: 158 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 408 B/s rd, 54 MiB/s wr, 862 op/s 2024-09-18T07:48:00.697 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:00.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:00.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:00.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:00 smithi203 ceph-mon[32237]: 1.6 scrub starts 2024-09-18T07:48:00.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:00 smithi203 ceph-mon[32237]: 1.6 scrub ok 2024-09-18T07:48:00.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:00 smithi203 ceph-mon[32237]: pgmap v910: 158 pgs: 158 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 408 B/s rd, 54 MiB/s wr, 862 op/s 2024-09-18T07:48:00.799 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:00.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:00.840 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:00.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:00.941 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:00.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:00.964 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:00.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:01.051 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 46 16 67297 67281 91.3977 65.8125 0.140317 0.0109102 2024-09-18T07:48:01.066 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:01.102 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:01.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:01.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:01.204 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:01.235 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:01.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:01.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:01.249 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T07:48:01.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:01.374 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:01.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:01.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:01.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:01.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:01.522 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:01.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:01.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:01 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:48:01.606 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:01 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:48:01.606 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:01 smithi137 ceph-mon[29408]: 4.3 scrub starts 2024-09-18T07:48:01.606 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:01 smithi137 ceph-mon[29408]: 4.3 scrub ok 2024-09-18T07:48:01.606 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:01 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:48:01.606 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:01 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:48:01.606 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:01 smithi137 ceph-mon[39175]: 4.3 scrub starts 2024-09-18T07:48:01.606 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:01 smithi137 ceph-mon[39175]: 4.3 scrub ok 2024-09-18T07:48:01.623 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:01.659 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:01.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:01.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:01 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:48:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:01 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:48:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:01 smithi203 ceph-mon[32237]: 4.3 scrub starts 2024-09-18T07:48:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:01 smithi203 ceph-mon[32237]: 4.3 scrub ok 2024-09-18T07:48:01.761 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:01.801 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:01.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:01.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:01.903 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:01.940 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:01.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:01.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:02.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:02.051 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 47 16 68578 68562 91.1555 80.0625 0.0102714 0.0109655 2024-09-18T07:48:02.080 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:02.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:02.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:02.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:02.229 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:02.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:02.330 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:02.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:02.368 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:02.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:02.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:02.505 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:02.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:02.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:02.607 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:02.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:02.640 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:02.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:02.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:02 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:48:02.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:02 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:48:02.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:02 smithi137 ceph-mon[29408]: osdmap e484: 8 total, 8 up, 8 in 2024-09-18T07:48:02.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:02 smithi137 ceph-mon[29408]: pgmap v912: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-18T07:48:02.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:02 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:48:02.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:02 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:48:02.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:02 smithi137 ceph-mon[39175]: osdmap e484: 8 total, 8 up, 8 in 2024-09-18T07:48:02.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:02 smithi137 ceph-mon[39175]: pgmap v912: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-18T07:48:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:02 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:48:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:02 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:48:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:02 smithi203 ceph-mon[32237]: osdmap e484: 8 total, 8 up, 8 in 2024-09-18T07:48:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:02 smithi203 ceph-mon[32237]: pgmap v912: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-18T07:48:02.741 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:02.776 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:02.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:02.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:02.878 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:02.911 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:02.917 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:02.932 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:02.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:02.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:03.034 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:03.052 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 48 16 69827 69811 90.8825 78.0625 0.0159116 0.0109978 2024-09-18T07:48:03.062 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:03.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:03.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:03.164 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:03.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:03.200 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:03.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:03.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:03.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:03.356 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:03.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:03.457 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:03.489 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:03.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:03.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:03.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:03.627 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:03.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:03 smithi137 ceph-mon[29408]: 1.7 scrub starts 2024-09-18T07:48:03.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:03 smithi137 ceph-mon[29408]: 1.7 scrub ok 2024-09-18T07:48:03.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:03 smithi137 ceph-mon[29408]: 4.8 scrub starts 2024-09-18T07:48:03.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:03 smithi137 ceph-mon[29408]: 4.8 scrub ok 2024-09-18T07:48:03.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:03 smithi137 ceph-mon[39175]: 1.7 scrub starts 2024-09-18T07:48:03.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:03 smithi137 ceph-mon[39175]: 1.7 scrub ok 2024-09-18T07:48:03.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:03 smithi137 ceph-mon[39175]: 4.8 scrub starts 2024-09-18T07:48:03.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:03 smithi137 ceph-mon[39175]: 4.8 scrub ok 2024-09-18T07:48:03.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:03 smithi203 ceph-mon[32237]: 1.7 scrub starts 2024-09-18T07:48:03.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:03 smithi203 ceph-mon[32237]: 1.7 scrub ok 2024-09-18T07:48:03.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:03 smithi203 ceph-mon[32237]: 4.8 scrub starts 2024-09-18T07:48:03.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:03 smithi203 ceph-mon[32237]: 4.8 scrub ok 2024-09-18T07:48:03.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:03.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:03.753 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:03.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:03.854 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:03.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:03.886 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:03.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:03.987 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:04.021 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:04.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:04.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:04.052 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 49 16 71020 71004 90.5493 74.5625 0.0174272 0.0110389 2024-09-18T07:48:04.123 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:04.154 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:04.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:04.255 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:04.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:04.291 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:04.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:04.392 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:04.427 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:04.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:04.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:04.529 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:04.563 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:04.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:04.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:04.669 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:04.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:04 smithi137 ceph-mon[29408]: 4.17 scrub starts 2024-09-18T07:48:04.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:04 smithi137 ceph-mon[29408]: 4.17 scrub ok 2024-09-18T07:48:04.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:04 smithi137 ceph-mon[29408]: pgmap v913: 158 pgs: 158 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 939 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-18T07:48:04.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:04 smithi137 ceph-mon[29408]: 3.17 scrub starts 2024-09-18T07:48:04.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:04 smithi137 ceph-mon[29408]: 3.17 scrub ok 2024-09-18T07:48:04.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:04 smithi137 ceph-mon[39175]: 4.17 scrub starts 2024-09-18T07:48:04.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:04 smithi137 ceph-mon[39175]: 4.17 scrub ok 2024-09-18T07:48:04.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:04 smithi137 ceph-mon[39175]: pgmap v913: 158 pgs: 158 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 939 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-18T07:48:04.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:04 smithi137 ceph-mon[39175]: 3.17 scrub starts 2024-09-18T07:48:04.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:04 smithi137 ceph-mon[39175]: 3.17 scrub ok 2024-09-18T07:48:04.698 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:04.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:04.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:04.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:04 smithi203 ceph-mon[32237]: 4.17 scrub starts 2024-09-18T07:48:04.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:04 smithi203 ceph-mon[32237]: 4.17 scrub ok 2024-09-18T07:48:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:04 smithi203 ceph-mon[32237]: pgmap v913: 158 pgs: 158 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 939 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-18T07:48:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:04 smithi203 ceph-mon[32237]: 3.17 scrub starts 2024-09-18T07:48:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:04 smithi203 ceph-mon[32237]: 3.17 scrub ok 2024-09-18T07:48:04.800 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:04.823 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:48:04.823 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:48:04.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:48:04.823 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:48:04.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.823 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:28995 11.b 11:ddfbb2a5:::benchmark_data_smithi137_181069_object28994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:48:04.824 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.138702+0000", 2024-09-18T07:48:04.824 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 33.537289536999999, 2024-09-18T07:48:04.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0007341729999999, 2024-09-18T07:48:04.824 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.824 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.824 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.824 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:04.824 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:04.824 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:04.824 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 28995 2024-09-18T07:48:04.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.824 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138702+0000", 2024-09-18T07:48:04.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138702+0000", 2024-09-18T07:48:04.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138706+0000", 2024-09-18T07:48:04.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5410000000000001e-06 2024-09-18T07:48:04.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138753+0000", 2024-09-18T07:48:04.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7506000000000002e-05 2024-09-18T07:48:04.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138792+0000", 2024-09-18T07:48:04.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8992000000000002e-05 2024-09-18T07:48:04.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138897+0000", 2024-09-18T07:48:04.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105167 2024-09-18T07:48:04.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.827 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138974+0000", 2024-09-18T07:48:04.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7225999999999995e-05 2024-09-18T07:48:04.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.827 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:04.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138997+0000", 2024-09-18T07:48:04.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2711999999999999e-05 2024-09-18T07:48:04.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.130768+0000", 2024-09-18T07:48:04.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99177061899999996 2024-09-18T07:48:04.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.130889+0000", 2024-09-18T07:48:04.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000121519 2024-09-18T07:48:04.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:48:04.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.131038+0000", 2024-09-18T07:48:04.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014834700000000001 2024-09-18T07:48:04.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.135620+0000", 2024-09-18T07:48:04.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045826139999999996 2024-09-18T07:48:04.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:04.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139286+0000", 2024-09-18T07:48:04.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036660220000000001 2024-09-18T07:48:04.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139338+0000", 2024-09-18T07:48:04.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2077999999999998e-05 2024-09-18T07:48:04.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139392+0000", 2024-09-18T07:48:04.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4129000000000003e-05 2024-09-18T07:48:04.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139436+0000", 2024-09-18T07:48:04.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3701000000000001e-05 2024-09-18T07:48:04.831 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.831 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.831 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.831 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29002 11.0 11:09879661:::benchmark_data_smithi137_181069_object29001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:48:04.831 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.144657+0000", 2024-09-18T07:48:04.831 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 33.531334418999997, 2024-09-18T07:48:04.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99799427299999999, 2024-09-18T07:48:04.832 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.832 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.832 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.832 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:04.832 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:04.832 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:04.832 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29002 2024-09-18T07:48:04.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.832 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144657+0000", 2024-09-18T07:48:04.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144657+0000", 2024-09-18T07:48:04.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144659+0000", 2024-09-18T07:48:04.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4249999999999999e-06 2024-09-18T07:48:04.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144683+0000", 2024-09-18T07:48:04.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4658e-05 2024-09-18T07:48:04.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144699+0000", 2024-09-18T07:48:04.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5821999999999999e-05 2024-09-18T07:48:04.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144707+0000", 2024-09-18T07:48:04.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3790000000000006e-06 2024-09-18T07:48:04.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144801+0000", 2024-09-18T07:48:04.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3278999999999999e-05 2024-09-18T07:48:04.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:04.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144811+0000", 2024-09-18T07:48:04.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8130000000000007e-06 2024-09-18T07:48:04.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.132138+0000", 2024-09-18T07:48:04.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98732766500000002 2024-09-18T07:48:04.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.132269+0000", 2024-09-18T07:48:04.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013058699999999999 2024-09-18T07:48:04.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:48:04.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.132403+0000", 2024-09-18T07:48:04.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013382300000000001 2024-09-18T07:48:04.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.136660+0000", 2024-09-18T07:48:04.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004257762 2024-09-18T07:48:04.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:04.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142430+0000", 2024-09-18T07:48:04.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057698050000000002 2024-09-18T07:48:04.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142552+0000", 2024-09-18T07:48:04.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012166400000000001 2024-09-18T07:48:04.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142611+0000", 2024-09-18T07:48:04.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8901999999999999e-05 2024-09-18T07:48:04.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142651+0000", 2024-09-18T07:48:04.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0689000000000003e-05 2024-09-18T07:48:04.839 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.839 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.839 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.839 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29006 11.0 11:0045c8b6:::benchmark_data_smithi137_181069_object29005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:48:04.839 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.148410+0000", 2024-09-18T07:48:04.839 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 33.52758137, 2024-09-18T07:48:04.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99787961400000003, 2024-09-18T07:48:04.840 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.840 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.840 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.840 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:04.840 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:04.840 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:04.840 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29006 2024-09-18T07:48:04.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.840 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148410+0000", 2024-09-18T07:48:04.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148410+0000", 2024-09-18T07:48:04.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148413+0000", 2024-09-18T07:48:04.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7580000000000001e-06 2024-09-18T07:48:04.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148442+0000", 2024-09-18T07:48:04.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8632999999999998e-05 2024-09-18T07:48:04.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148462+0000", 2024-09-18T07:48:04.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0166999999999999e-05 2024-09-18T07:48:04.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148471+0000", 2024-09-18T07:48:04.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3819999999999993e-06 2024-09-18T07:48:04.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148541+0000", 2024-09-18T07:48:04.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0277000000000005e-05 2024-09-18T07:48:04.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:04.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148552+0000", 2024-09-18T07:48:04.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0584999999999999e-05 2024-09-18T07:48:04.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.134043+0000", 2024-09-18T07:48:04.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98549055699999999 2024-09-18T07:48:04.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.134770+0000", 2024-09-18T07:48:04.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00072761600000000005 2024-09-18T07:48:04.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:48:04.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.134899+0000", 2024-09-18T07:48:04.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012857100000000001 2024-09-18T07:48:04.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142732+0000", 2024-09-18T07:48:04.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0078337389999999993 2024-09-18T07:48:04.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142773+0000", 2024-09-18T07:48:04.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0132999999999997e-05 2024-09-18T07:48:04.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:04.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.146198+0000", 2024-09-18T07:48:04.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034254709999999998 2024-09-18T07:48:04.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.146249+0000", 2024-09-18T07:48:04.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0636000000000001e-05 2024-09-18T07:48:04.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.146290+0000", 2024-09-18T07:48:04.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1088999999999999e-05 2024-09-18T07:48:04.847 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.847 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.847 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.847 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29012 11.0 11:09fb93d5:::benchmark_data_smithi137_181069_object29011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:48:04.847 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.154931+0000", 2024-09-18T07:48:04.847 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 33.521060984000002, 2024-09-18T07:48:04.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.993148433, 2024-09-18T07:48:04.848 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.848 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.848 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.848 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:04.848 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:04.848 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:04.848 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29012 2024-09-18T07:48:04.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.848 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.154931+0000", 2024-09-18T07:48:04.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.154931+0000", 2024-09-18T07:48:04.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.154934+0000", 2024-09-18T07:48:04.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0570000000000001e-06 2024-09-18T07:48:04.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155009+0000", 2024-09-18T07:48:04.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5451000000000007e-05 2024-09-18T07:48:04.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155030+0000", 2024-09-18T07:48:04.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1262000000000001e-05 2024-09-18T07:48:04.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155041+0000", 2024-09-18T07:48:04.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0509999999999999e-05 2024-09-18T07:48:04.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155122+0000", 2024-09-18T07:48:04.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0785999999999995e-05 2024-09-18T07:48:04.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:04.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155133+0000", 2024-09-18T07:48:04.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1314e-05 2024-09-18T07:48:04.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.136710+0000", 2024-09-18T07:48:04.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98157681100000005 2024-09-18T07:48:04.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.136842+0000", 2024-09-18T07:48:04.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013199699999999999 2024-09-18T07:48:04.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:48:04.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.136966+0000", 2024-09-18T07:48:04.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000123757 2024-09-18T07:48:04.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142811+0000", 2024-09-18T07:48:04.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058454559999999997 2024-09-18T07:48:04.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:04.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.146298+0000", 2024-09-18T07:48:04.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034865690000000001 2024-09-18T07:48:04.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.147987+0000", 2024-09-18T07:48:04.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001689636 2024-09-18T07:48:04.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.148039+0000", 2024-09-18T07:48:04.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1335999999999998e-05 2024-09-18T07:48:04.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.148079+0000", 2024-09-18T07:48:04.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0491000000000002e-05 2024-09-18T07:48:04.855 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.855 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.855 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.856 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29013 11.0 11:0f785d5e:::benchmark_data_smithi137_181069_object29012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:48:04.856 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.155066+0000", 2024-09-18T07:48:04.856 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 33.520925945000002, 2024-09-18T07:48:04.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99598627500000003, 2024-09-18T07:48:04.856 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.856 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.857 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.857 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:04.857 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:04.857 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:04.857 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29013 2024-09-18T07:48:04.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.857 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155066+0000", 2024-09-18T07:48:04.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155066+0000", 2024-09-18T07:48:04.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155066+0000", 2024-09-18T07:48:04.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8000000000000004e-07 2024-09-18T07:48:04.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155115+0000", 2024-09-18T07:48:04.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8026999999999999e-05 2024-09-18T07:48:04.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155123+0000", 2024-09-18T07:48:04.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.507e-06 2024-09-18T07:48:04.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155129+0000", 2024-09-18T07:48:04.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4119999999999998e-06 2024-09-18T07:48:04.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155190+0000", 2024-09-18T07:48:04.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0618000000000001e-05 2024-09-18T07:48:04.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:04.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155200+0000", 2024-09-18T07:48:04.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0360000000000001e-05 2024-09-18T07:48:04.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.138108+0000", 2024-09-18T07:48:04.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98290805299999995 2024-09-18T07:48:04.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.138207+0000", 2024-09-18T07:48:04.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8399000000000004e-05 2024-09-18T07:48:04.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:48:04.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.138323+0000", 2024-09-18T07:48:04.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115858 2024-09-18T07:48:04.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142850+0000", 2024-09-18T07:48:04.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045276090000000001 2024-09-18T07:48:04.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:04.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.146302+0000", 2024-09-18T07:48:04.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003451759 2024-09-18T07:48:04.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.150948+0000", 2024-09-18T07:48:04.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046454440000000003 2024-09-18T07:48:04.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.151009+0000", 2024-09-18T07:48:04.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1697000000000002e-05 2024-09-18T07:48:04.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.151052+0000", 2024-09-18T07:48:04.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2651999999999998e-05 2024-09-18T07:48:04.866 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.866 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.866 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.867 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29017 11.0 11:0743585d:::benchmark_data_smithi137_181069_object29016:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:48:04.867 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.157700+0000", 2024-09-18T07:48:04.867 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 33.518291765000001, 2024-09-18T07:48:04.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0031416529999999, 2024-09-18T07:48:04.867 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.867 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.867 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.867 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:04.867 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:04.867 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:04.867 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29017 2024-09-18T07:48:04.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.867 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157700+0000", 2024-09-18T07:48:04.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157700+0000", 2024-09-18T07:48:04.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157703+0000", 2024-09-18T07:48:04.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.954e-06 2024-09-18T07:48:04.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157758+0000", 2024-09-18T07:48:04.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4923000000000003e-05 2024-09-18T07:48:04.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157775+0000", 2024-09-18T07:48:04.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7334999999999999e-05 2024-09-18T07:48:04.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157785+0000", 2024-09-18T07:48:04.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0131e-05 2024-09-18T07:48:04.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157860+0000", 2024-09-18T07:48:04.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4964999999999998e-05 2024-09-18T07:48:04.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:04.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157870+0000", 2024-09-18T07:48:04.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0047999999999999e-05 2024-09-18T07:48:04.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139724+0000", 2024-09-18T07:48:04.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98185376000000002 2024-09-18T07:48:04.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139843+0000", 2024-09-18T07:48:04.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011950700000000001 2024-09-18T07:48:04.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:48:04.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.141219+0000", 2024-09-18T07:48:04.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013756409999999999 2024-09-18T07:48:04.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:04.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.147813+0000", 2024-09-18T07:48:04.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065937160000000003 2024-09-18T07:48:04.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.147878+0000", 2024-09-18T07:48:04.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5513999999999996e-05 2024-09-18T07:48:04.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160742+0000", 2024-09-18T07:48:04.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012863638 2024-09-18T07:48:04.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160804+0000", 2024-09-18T07:48:04.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1717000000000006e-05 2024-09-18T07:48:04.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160841+0000", 2024-09-18T07:48:04.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7803999999999999e-05 2024-09-18T07:48:04.874 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.874 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.874 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.874 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29027 11.b 11:d70c9013:::benchmark_data_smithi137_181069_object29026:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:48:04.874 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.170795+0000", 2024-09-18T07:48:04.874 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 33.505196324000003, 2024-09-18T07:48:04.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96875128899999996, 2024-09-18T07:48:04.874 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.874 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.874 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.874 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:04.875 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:04.875 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:04.875 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29027 2024-09-18T07:48:04.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.875 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170795+0000", 2024-09-18T07:48:04.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170795+0000", 2024-09-18T07:48:04.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170798+0000", 2024-09-18T07:48:04.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4880000000000001e-06 2024-09-18T07:48:04.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170853+0000", 2024-09-18T07:48:04.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4836999999999997e-05 2024-09-18T07:48:04.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170867+0000", 2024-09-18T07:48:04.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4615e-05 2024-09-18T07:48:04.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170876+0000", 2024-09-18T07:48:04.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2299999999999997e-06 2024-09-18T07:48:04.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170945+0000", 2024-09-18T07:48:04.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8979999999999993e-05 2024-09-18T07:48:04.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:04.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170956+0000", 2024-09-18T07:48:04.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0716999999999999e-05 2024-09-18T07:48:04.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.132286+0000", 2024-09-18T07:48:04.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96132993099999997 2024-09-18T07:48:04.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.132397+0000", 2024-09-18T07:48:04.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011101 2024-09-18T07:48:04.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:48:04.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.132547+0000", 2024-09-18T07:48:04.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014972 2024-09-18T07:48:04.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.135658+0000", 2024-09-18T07:48:04.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031108949999999998 2024-09-18T07:48:04.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:04.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139296+0000", 2024-09-18T07:48:04.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003638208 2024-09-18T07:48:04.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139472+0000", 2024-09-18T07:48:04.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017624900000000001 2024-09-18T07:48:04.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139515+0000", 2024-09-18T07:48:04.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2787000000000002e-05 2024-09-18T07:48:04.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139547+0000", 2024-09-18T07:48:04.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1622000000000003e-05 2024-09-18T07:48:04.881 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.881 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.881 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.881 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29029 11.b 11:dd1c139d:::benchmark_data_smithi137_181069_object29028:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:48:04.881 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.170902+0000", 2024-09-18T07:48:04.882 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 33.505089984000001, 2024-09-18T07:48:04.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96875504400000001, 2024-09-18T07:48:04.882 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.882 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.882 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.882 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:04.882 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:04.882 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:04.882 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29029 2024-09-18T07:48:04.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.882 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170902+0000", 2024-09-18T07:48:04.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170902+0000", 2024-09-18T07:48:04.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170902+0000", 2024-09-18T07:48:04.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2999999999999999e-07 2024-09-18T07:48:04.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170948+0000", 2024-09-18T07:48:04.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5804999999999999e-05 2024-09-18T07:48:04.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170957+0000", 2024-09-18T07:48:04.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1290000000000002e-06 2024-09-18T07:48:04.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170964+0000", 2024-09-18T07:48:04.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3999999999999997e-06 2024-09-18T07:48:04.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.172474+0000", 2024-09-18T07:48:04.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001510286 2024-09-18T07:48:04.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:04.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.172485+0000", 2024-09-18T07:48:04.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0621999999999999e-05 2024-09-18T07:48:04.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.134224+0000", 2024-09-18T07:48:04.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96173945699999996 2024-09-18T07:48:04.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.134333+0000", 2024-09-18T07:48:04.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000108731 2024-09-18T07:48:04.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:48:04.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.134455+0000", 2024-09-18T07:48:04.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012238799999999999 2024-09-18T07:48:04.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.138920+0000", 2024-09-18T07:48:04.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044642049999999997 2024-09-18T07:48:04.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:04.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139301+0000", 2024-09-18T07:48:04.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00038108599999999999 2024-09-18T07:48:04.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139582+0000", 2024-09-18T07:48:04.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000281754 2024-09-18T07:48:04.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139625+0000", 2024-09-18T07:48:04.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2472000000000003e-05 2024-09-18T07:48:04.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139657+0000", 2024-09-18T07:48:04.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1779000000000002e-05 2024-09-18T07:48:04.889 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.889 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.889 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.889 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29048 11.0 11:02c48360:::benchmark_data_smithi137_181069_object29047:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:48:04.889 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.185570+0000", 2024-09-18T07:48:04.889 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 33.490421658000002, 2024-09-18T07:48:04.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97538213600000001, 2024-09-18T07:48:04.889 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.890 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.890 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.890 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:04.890 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:04.890 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:04.890 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29048 2024-09-18T07:48:04.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.890 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.185570+0000", 2024-09-18T07:48:04.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.185570+0000", 2024-09-18T07:48:04.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.185573+0000", 2024-09-18T07:48:04.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0699999999999998e-06 2024-09-18T07:48:04.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.185635+0000", 2024-09-18T07:48:04.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1809000000000004e-05 2024-09-18T07:48:04.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.185652+0000", 2024-09-18T07:48:04.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7543000000000001e-05 2024-09-18T07:48:04.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.185661+0000", 2024-09-18T07:48:04.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9479999999999997e-06 2024-09-18T07:48:04.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.185751+0000", 2024-09-18T07:48:04.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9740000000000005e-05 2024-09-18T07:48:04.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:04.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.185761+0000", 2024-09-18T07:48:04.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0017000000000001e-05 2024-09-18T07:48:04.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142881+0000", 2024-09-18T07:48:04.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95711977999999998 2024-09-18T07:48:04.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.143134+0000", 2024-09-18T07:48:04.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00025368400000000001 2024-09-18T07:48:04.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:48:04.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.143275+0000", 2024-09-18T07:48:04.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014023900000000001 2024-09-18T07:48:04.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:04.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.147818+0000", 2024-09-18T07:48:04.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045433619999999996 2024-09-18T07:48:04.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.147932+0000", 2024-09-18T07:48:04.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114017 2024-09-18T07:48:04.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160876+0000", 2024-09-18T07:48:04.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012943875000000001 2024-09-18T07:48:04.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160920+0000", 2024-09-18T07:48:04.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3668000000000003e-05 2024-09-18T07:48:04.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160952+0000", 2024-09-18T07:48:04.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2384e-05 2024-09-18T07:48:04.897 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.897 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.897 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.898 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29051 11.0 11:0623b464:::benchmark_data_smithi137_181069_object29050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:48:04.898 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.189389+0000", 2024-09-18T07:48:04.898 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 33.486602425000001, 2024-09-18T07:48:04.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97168827700000004, 2024-09-18T07:48:04.898 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.898 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.898 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.899 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:04.899 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:04.899 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:04.899 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29051 2024-09-18T07:48:04.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.899 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.189389+0000", 2024-09-18T07:48:04.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.189389+0000", 2024-09-18T07:48:04.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.189391+0000", 2024-09-18T07:48:04.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3499999999999999e-06 2024-09-18T07:48:04.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.189446+0000", 2024-09-18T07:48:04.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4775000000000003e-05 2024-09-18T07:48:04.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.189459+0000", 2024-09-18T07:48:04.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2826000000000001e-05 2024-09-18T07:48:04.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.189468+0000", 2024-09-18T07:48:04.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7930000000000003e-06 2024-09-18T07:48:04.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.189545+0000", 2024-09-18T07:48:04.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6785000000000004e-05 2024-09-18T07:48:04.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:04.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.189556+0000", 2024-09-18T07:48:04.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1343e-05 2024-09-18T07:48:04.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.144652+0000", 2024-09-18T07:48:04.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95509612300000002 2024-09-18T07:48:04.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.144785+0000", 2024-09-18T07:48:04.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013303999999999999 2024-09-18T07:48:04.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:48:04.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.144931+0000", 2024-09-18T07:48:04.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014624399999999999 2024-09-18T07:48:04.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:04.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.147822+0000", 2024-09-18T07:48:04.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028909439999999999 2024-09-18T07:48:04.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.148128+0000", 2024-09-18T07:48:04.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00030586799999999999 2024-09-18T07:48:04.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160999+0000", 2024-09-18T07:48:04.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012870967000000001 2024-09-18T07:48:04.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161044+0000", 2024-09-18T07:48:04.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5222000000000002e-05 2024-09-18T07:48:04.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161077+0000", 2024-09-18T07:48:04.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2997000000000003e-05 2024-09-18T07:48:04.906 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.906 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.907 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.907 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29075 11.b 11:dfbfee08:::benchmark_data_smithi137_181069_object29074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:48:04.907 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.207141+0000", 2024-09-18T07:48:04.907 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 33.468850402999998, 2024-09-18T07:48:04.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.936065696, 2024-09-18T07:48:04.907 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.907 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.907 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.907 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:04.907 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:04.907 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:04.908 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29075 2024-09-18T07:48:04.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.908 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.207141+0000", 2024-09-18T07:48:04.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.207141+0000", 2024-09-18T07:48:04.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.207143+0000", 2024-09-18T07:48:04.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8589999999999999e-06 2024-09-18T07:48:04.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.207193+0000", 2024-09-18T07:48:04.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9669999999999997e-05 2024-09-18T07:48:04.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.207209+0000", 2024-09-18T07:48:04.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5909e-05 2024-09-18T07:48:04.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.207217+0000", 2024-09-18T07:48:04.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0139999999999998e-06 2024-09-18T07:48:04.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.207277+0000", 2024-09-18T07:48:04.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0816000000000002e-05 2024-09-18T07:48:04.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:04.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.207289+0000", 2024-09-18T07:48:04.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1226999999999999e-05 2024-09-18T07:48:04.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.135725+0000", 2024-09-18T07:48:04.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.92843600999999998 2024-09-18T07:48:04.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.135854+0000", 2024-09-18T07:48:04.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000129305 2024-09-18T07:48:04.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:48:04.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.135996+0000", 2024-09-18T07:48:04.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000141722 2024-09-18T07:48:04.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:04.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139305+0000", 2024-09-18T07:48:04.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003309317 2024-09-18T07:48:04.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.141294+0000", 2024-09-18T07:48:04.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00198854 2024-09-18T07:48:04.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.143116+0000", 2024-09-18T07:48:04.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001822167 2024-09-18T07:48:04.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.143166+0000", 2024-09-18T07:48:04.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9947999999999997e-05 2024-09-18T07:48:04.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.143207+0000", 2024-09-18T07:48:04.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1192000000000001e-05 2024-09-18T07:48:04.914 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.914 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.914 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.914 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29079 11.0 11:0c34a040:::benchmark_data_smithi137_181069_object29078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:48:04.914 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.209414+0000", 2024-09-18T07:48:04.914 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 33.466577915000002, 2024-09-18T07:48:04.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95177025599999998, 2024-09-18T07:48:04.915 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.915 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.915 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.915 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:04.915 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:04.915 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:04.915 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29079 2024-09-18T07:48:04.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.915 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.209414+0000", 2024-09-18T07:48:04.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.209414+0000", 2024-09-18T07:48:04.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.209415+0000", 2024-09-18T07:48:04.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0210000000000001e-06 2024-09-18T07:48:04.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.209438+0000", 2024-09-18T07:48:04.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3682000000000001e-05 2024-09-18T07:48:04.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.209447+0000", 2024-09-18T07:48:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4160000000000004e-06 2024-09-18T07:48:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.209454+0000", 2024-09-18T07:48:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9319999999999999e-06 2024-09-18T07:48:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.209529+0000", 2024-09-18T07:48:04.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5168999999999998e-05 2024-09-18T07:48:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.209538+0000", 2024-09-18T07:48:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5519999999999993e-06 2024-09-18T07:48:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.146333+0000", 2024-09-18T07:48:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93679421500000004 2024-09-18T07:48:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.146464+0000", 2024-09-18T07:48:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000131633 2024-09-18T07:48:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:48:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.146606+0000", 2024-09-18T07:48:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014206200000000001 2024-09-18T07:48:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.148144+0000", 2024-09-18T07:48:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001537561 2024-09-18T07:48:04.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.151135+0000", 2024-09-18T07:48:04.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029910929999999998 2024-09-18T07:48:04.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161110+0000", 2024-09-18T07:48:04.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0099754189999999993 2024-09-18T07:48:04.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161153+0000", 2024-09-18T07:48:04.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2589999999999997e-05 2024-09-18T07:48:04.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161184+0000", 2024-09-18T07:48:04.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0911000000000003e-05 2024-09-18T07:48:04.921 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.921 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.921 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.921 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29092 11.0 11:02dae925:::benchmark_data_smithi137_181069_object29091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:48:04.921 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.221277+0000", 2024-09-18T07:48:04.921 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 33.454714207999999, 2024-09-18T07:48:04.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94000973799999998, 2024-09-18T07:48:04.922 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.922 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.922 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.922 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:04.922 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:04.922 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:04.922 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29092 2024-09-18T07:48:04.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.922 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.922 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.221277+0000", 2024-09-18T07:48:04.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.923 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.221277+0000", 2024-09-18T07:48:04.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.923 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.221280+0000", 2024-09-18T07:48:04.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1919999999999999e-06 2024-09-18T07:48:04.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.924 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.924 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.221323+0000", 2024-09-18T07:48:04.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3514999999999999e-05 2024-09-18T07:48:04.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.924 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.924 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.221350+0000", 2024-09-18T07:48:04.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7441999999999999e-05 2024-09-18T07:48:04.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.924 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.924 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.221359+0000", 2024-09-18T07:48:04.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3669999999999997e-06 2024-09-18T07:48:04.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.221429+0000", 2024-09-18T07:48:04.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9970999999999998e-05 2024-09-18T07:48:04.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:04.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.221440+0000", 2024-09-18T07:48:04.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0937e-05 2024-09-18T07:48:04.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.148187+0000", 2024-09-18T07:48:04.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.92674730800000005 2024-09-18T07:48:04.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.148317+0000", 2024-09-18T07:48:04.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012991200000000001 2024-09-18T07:48:04.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:48:04.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.148438+0000", 2024-09-18T07:48:04.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000121148 2024-09-18T07:48:04.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.151189+0000", 2024-09-18T07:48:04.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027507339999999999 2024-09-18T07:48:04.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:04.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160226+0000", 2024-09-18T07:48:04.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0090371380000000001 2024-09-18T07:48:04.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161215+0000", 2024-09-18T07:48:04.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00098912200000000005 2024-09-18T07:48:04.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161257+0000", 2024-09-18T07:48:04.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1613000000000003e-05 2024-09-18T07:48:04.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161287+0000", 2024-09-18T07:48:04.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0338999999999999e-05 2024-09-18T07:48:04.928 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.928 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.928 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.928 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29095 11.0 11:0f755343:::benchmark_data_smithi137_181069_object29094:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:48:04.929 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.224759+0000", 2024-09-18T07:48:04.929 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 33.451232302999998, 2024-09-18T07:48:04.929 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93664074100000005, 2024-09-18T07:48:04.929 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.929 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.929 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.929 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:04.929 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:04.929 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:04.929 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29095 2024-09-18T07:48:04.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.929 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.224759+0000", 2024-09-18T07:48:04.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.224759+0000", 2024-09-18T07:48:04.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.224762+0000", 2024-09-18T07:48:04.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4719999999999998e-06 2024-09-18T07:48:04.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.931 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.931 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.224788+0000", 2024-09-18T07:48:04.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6205999999999999e-05 2024-09-18T07:48:04.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.931 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.932 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.224806+0000", 2024-09-18T07:48:04.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8414e-05 2024-09-18T07:48:04.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.932 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.932 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.224816+0000", 2024-09-18T07:48:04.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0149000000000001e-05 2024-09-18T07:48:04.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.224889+0000", 2024-09-18T07:48:04.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2083999999999996e-05 2024-09-18T07:48:04.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:04.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.224902+0000", 2024-09-18T07:48:04.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3179999999999999e-05 2024-09-18T07:48:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.149507+0000", 2024-09-18T07:48:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.92460489300000004 2024-09-18T07:48:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.149624+0000", 2024-09-18T07:48:04.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116968 2024-09-18T07:48:04.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.935 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:48:04.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.149800+0000", 2024-09-18T07:48:04.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017676000000000001 2024-09-18T07:48:04.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.936 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.936 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160210+0000", 2024-09-18T07:48:04.936 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010410034 2024-09-18T07:48:04.936 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.936 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.936 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:04.936 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160231+0000", 2024-09-18T07:48:04.936 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0441999999999998e-05 2024-09-18T07:48:04.936 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.937 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161328+0000", 2024-09-18T07:48:04.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001097413 2024-09-18T07:48:04.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.938 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161369+0000", 2024-09-18T07:48:04.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1208999999999998e-05 2024-09-18T07:48:04.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.938 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.938 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161400+0000", 2024-09-18T07:48:04.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0516999999999999e-05 2024-09-18T07:48:04.938 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.939 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.939 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.939 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29107 11.0 11:0c46d5ad:::benchmark_data_smithi137_181069_object29106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:48:04.939 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.241089+0000", 2024-09-18T07:48:04.939 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 33.434902379, 2024-09-18T07:48:04.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.92074101200000003, 2024-09-18T07:48:04.940 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.940 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.940 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.940 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:04.940 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:04.940 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:04.940 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29107 2024-09-18T07:48:04.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.941 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.241089+0000", 2024-09-18T07:48:04.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.241089+0000", 2024-09-18T07:48:04.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.241091+0000", 2024-09-18T07:48:04.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1100000000000001e-06 2024-09-18T07:48:04.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.943 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.943 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.241113+0000", 2024-09-18T07:48:04.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1509e-05 2024-09-18T07:48:04.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.943 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.943 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.241125+0000", 2024-09-18T07:48:04.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1932e-05 2024-09-18T07:48:04.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.241132+0000", 2024-09-18T07:48:04.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2980000000000003e-06 2024-09-18T07:48:04.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.241194+0000", 2024-09-18T07:48:04.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1828999999999994e-05 2024-09-18T07:48:04.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.945 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:04.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.241202+0000", 2024-09-18T07:48:04.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1519999999999992e-06 2024-09-18T07:48:04.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.945 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.151364+0000", 2024-09-18T07:48:04.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91016249100000002 2024-09-18T07:48:04.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.151486+0000", 2024-09-18T07:48:04.946 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000121925 2024-09-18T07:48:04.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:48:04.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.151618+0000", 2024-09-18T07:48:04.946 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013172300000000001 2024-09-18T07:48:04.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:04.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160235+0000", 2024-09-18T07:48:04.946 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0086173059999999999 2024-09-18T07:48:04.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161432+0000", 2024-09-18T07:48:04.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011964409999999999 2024-09-18T07:48:04.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161737+0000", 2024-09-18T07:48:04.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00030523600000000002 2024-09-18T07:48:04.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161783+0000", 2024-09-18T07:48:04.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6230000000000003e-05 2024-09-18T07:48:04.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161830+0000", 2024-09-18T07:48:04.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6829999999999997e-05 2024-09-18T07:48:04.948 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.948 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.948 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.948 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29116 11.b 11:de07af6b:::benchmark_data_smithi137_181069_object29115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:48:04.948 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.268587+0000", 2024-09-18T07:48:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 33.407404362000001, 2024-09-18T07:48:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.87712996300000001, 2024-09-18T07:48:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29116 2024-09-18T07:48:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.268587+0000", 2024-09-18T07:48:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.268587+0000", 2024-09-18T07:48:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.268590+0000", 2024-09-18T07:48:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3190000000000002e-06 2024-09-18T07:48:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.950 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.268614+0000", 2024-09-18T07:48:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.406e-05 2024-09-18T07:48:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.268635+0000", 2024-09-18T07:48:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1658000000000002e-05 2024-09-18T07:48:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.268643+0000", 2024-09-18T07:48:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.875e-06 2024-09-18T07:48:04.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.268707+0000", 2024-09-18T07:48:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3862999999999995e-05 2024-09-18T07:48:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.268715+0000", 2024-09-18T07:48:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0989999999999999e-06 2024-09-18T07:48:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.952 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.137340+0000", 2024-09-18T07:48:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.86862488500000001 2024-09-18T07:48:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.137444+0000", 2024-09-18T07:48:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103609 2024-09-18T07:48:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:48:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.137563+0000", 2024-09-18T07:48:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011926000000000001 2024-09-18T07:48:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:04.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.141380+0000", 2024-09-18T07:48:04.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038170439999999999 2024-09-18T07:48:04.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142079+0000", 2024-09-18T07:48:04.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00069944099999999997 2024-09-18T07:48:04.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.145604+0000", 2024-09-18T07:48:04.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035245039999999999 2024-09-18T07:48:04.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.955 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.145674+0000", 2024-09-18T07:48:04.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0593000000000007e-05 2024-09-18T07:48:04.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.955 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.145717+0000", 2024-09-18T07:48:04.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2753000000000002e-05 2024-09-18T07:48:04.955 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.955 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.955 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.956 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:67251 11.b 11:d877251f:::benchmark_data_smithi137_181069_object67250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e483)", 2024-09-18T07:48:04.956 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:00.906275+0000", 2024-09-18T07:48:04.956 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 3.7697167029999998, 2024-09-18T07:48:04.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14005775000000001, 2024-09-18T07:48:04.956 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.956 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.956 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.956 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:04.956 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:04.956 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:04.956 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 67251 2024-09-18T07:48:04.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.956 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906275+0000", 2024-09-18T07:48:04.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906275+0000", 2024-09-18T07:48:04.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906277+0000", 2024-09-18T07:48:04.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8619999999999999e-06 2024-09-18T07:48:04.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906332+0000", 2024-09-18T07:48:04.958 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5778e-05 2024-09-18T07:48:04.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906344+0000", 2024-09-18T07:48:04.958 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1027e-05 2024-09-18T07:48:04.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906351+0000", 2024-09-18T07:48:04.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3950000000000003e-06 2024-09-18T07:48:04.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.959 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.959 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906403+0000", 2024-09-18T07:48:04.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1786000000000002e-05 2024-09-18T07:48:04.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.959 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.959 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906513+0000", 2024-09-18T07:48:04.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110438 2024-09-18T07:48:04.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.959 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:48:04.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906604+0000", 2024-09-18T07:48:04.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0493000000000002e-05 2024-09-18T07:48:04.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:04.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.909046+0000", 2024-09-18T07:48:04.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024423209999999999 2024-09-18T07:48:04.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.909331+0000", 2024-09-18T07:48:04.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000284935 2024-09-18T07:48:04.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:04.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046260+0000", 2024-09-18T07:48:04.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13692871500000001 2024-09-18T07:48:04.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046300+0000", 2024-09-18T07:48:04.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0187e-05 2024-09-18T07:48:04.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046333+0000", 2024-09-18T07:48:04.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2812999999999999e-05 2024-09-18T07:48:04.962 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.962 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.962 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.962 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop_reply(client.16581.0:67251 11.b e483/469 ondisk, result = 0)", 2024-09-18T07:48:04.962 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:00.909329+0000", 2024-09-18T07:48:04.962 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 3.7666628520000001, 2024-09-18T07:48:04.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.137009873, 2024-09-18T07:48:04.962 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.962 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.962 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:48:04.962 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.909329+0000", 2024-09-18T07:48:04.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.909329+0000", 2024-09-18T07:48:04.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.909330+0000", 2024-09-18T07:48:04.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2139999999999999e-06 2024-09-18T07:48:04.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.909335+0000", 2024-09-18T07:48:04.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7010000000000004e-06 2024-09-18T07:48:04.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.909342+0000", 2024-09-18T07:48:04.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5739999999999998e-06 2024-09-18T07:48:04.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.909348+0000", 2024-09-18T07:48:04.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8130000000000001e-06 2024-09-18T07:48:04.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046243+0000", 2024-09-18T07:48:04.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13689459000000001 2024-09-18T07:48:04.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046252+0000", 2024-09-18T07:48:04.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9089999999999995e-06 2024-09-18T07:48:04.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046339+0000", 2024-09-18T07:48:04.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6071999999999999e-05 2024-09-18T07:48:04.966 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.966 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.966 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.966 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:67234 11.2 e483/459 11:4edd578a:::benchmark_data_smithi137_181069_object67233:head v 483'4135, mlcod=483'4133)", 2024-09-18T07:48:04.966 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:00.914389+0000", 2024-09-18T07:48:04.967 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 3.7616029979999999, 2024-09-18T07:48:04.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14120948, 2024-09-18T07:48:04.967 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.967 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.967 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.967 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.967 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.967 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.914389+0000", 2024-09-18T07:48:04.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.967 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.967 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.914389+0000", 2024-09-18T07:48:04.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.914390+0000", 2024-09-18T07:48:04.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9269999999999999e-06 2024-09-18T07:48:04.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.914438+0000", 2024-09-18T07:48:04.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7681e-05 2024-09-18T07:48:04.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046208+0000", 2024-09-18T07:48:04.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13176966700000001 2024-09-18T07:48:04.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046220+0000", 2024-09-18T07:48:04.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2041e-05 2024-09-18T07:48:04.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046283+0000", 2024-09-18T07:48:04.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3003999999999995e-05 2024-09-18T07:48:04.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046305+0000", 2024-09-18T07:48:04.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2385e-05 2024-09-18T07:48:04.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.055546+0000", 2024-09-18T07:48:04.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0092402820000000007 2024-09-18T07:48:04.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.055598+0000", 2024-09-18T07:48:04.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2493e-05 2024-09-18T07:48:04.971 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.971 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.971 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.971 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:67256 11.8 e483/460 11:114e4703:::benchmark_data_smithi137_181069_object67255:head v 483'4193, mlcod=483'4191)", 2024-09-18T07:48:04.971 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:00.914722+0000", 2024-09-18T07:48:04.971 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 3.7612696850000003, 2024-09-18T07:48:04.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13614241199999999, 2024-09-18T07:48:04.971 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:04.971 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:04.971 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:04.972 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:04.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:04.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.914722+0000", 2024-09-18T07:48:04.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:04.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.914722+0000", 2024-09-18T07:48:04.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:04.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:04.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.914723+0000", 2024-09-18T07:48:04.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.477e-06 2024-09-18T07:48:04.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:04.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.914756+0000", 2024-09-18T07:48:04.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2359e-05 2024-09-18T07:48:04.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:04.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046210+0000", 2024-09-18T07:48:04.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13145411400000001 2024-09-18T07:48:04.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:04.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046221+0000", 2024-09-18T07:48:04.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0927999999999999e-05 2024-09-18T07:48:04.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:04.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046300+0000", 2024-09-18T07:48:04.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8991000000000003e-05 2024-09-18T07:48:04.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:04.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046318+0000", 2024-09-18T07:48:04.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7954000000000001e-05 2024-09-18T07:48:04.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:04.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.050815+0000", 2024-09-18T07:48:04.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044970560000000001 2024-09-18T07:48:04.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:04.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:04.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:04.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.050864+0000", 2024-09-18T07:48:04.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9533000000000002e-05 2024-09-18T07:48:04.975 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.975 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.975 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.975 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:04.976 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:04.976 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:48:04.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:48:04.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:04.978 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:04.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:05.053 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 50 16 72369 72353 90.4244 84.3125 0.00935518 0.0110516 2024-09-18T07:48:05.078 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:05.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:05.122 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:05.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:05.223 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:05.251 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:05.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:05.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:05.353 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:05.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:05.390 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:05.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:05.491 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:05.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:05.517 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:05.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:05.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:05.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:05.668 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:05.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:05.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:05 smithi137 ceph-mon[29408]: 6.14 scrub starts 2024-09-18T07:48:05.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:05 smithi137 ceph-mon[29408]: 6.14 scrub ok 2024-09-18T07:48:05.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:05 smithi137 ceph-mon[29408]: 1.3 scrub starts 2024-09-18T07:48:05.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:05 smithi137 ceph-mon[29408]: 1.3 scrub ok 2024-09-18T07:48:05.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:05 smithi137 ceph-mon[29408]: 5.19 scrub starts 2024-09-18T07:48:05.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:05 smithi137 ceph-mon[29408]: 5.19 scrub ok 2024-09-18T07:48:05.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:05 smithi137 ceph-mon[29408]: osdmap e485: 8 total, 8 up, 8 in 2024-09-18T07:48:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:05 smithi137 ceph-mon[39175]: 6.14 scrub starts 2024-09-18T07:48:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:05 smithi137 ceph-mon[39175]: 6.14 scrub ok 2024-09-18T07:48:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:05 smithi137 ceph-mon[39175]: 1.3 scrub starts 2024-09-18T07:48:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:05 smithi137 ceph-mon[39175]: 1.3 scrub ok 2024-09-18T07:48:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:05 smithi137 ceph-mon[39175]: 5.19 scrub starts 2024-09-18T07:48:05.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:05 smithi137 ceph-mon[39175]: 5.19 scrub ok 2024-09-18T07:48:05.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:05 smithi137 ceph-mon[39175]: osdmap e485: 8 total, 8 up, 8 in 2024-09-18T07:48:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:05 smithi203 ceph-mon[32237]: 6.14 scrub starts 2024-09-18T07:48:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:05 smithi203 ceph-mon[32237]: 6.14 scrub ok 2024-09-18T07:48:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:05 smithi203 ceph-mon[32237]: 1.3 scrub starts 2024-09-18T07:48:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:05 smithi203 ceph-mon[32237]: 1.3 scrub ok 2024-09-18T07:48:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:05 smithi203 ceph-mon[32237]: 5.19 scrub starts 2024-09-18T07:48:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:05 smithi203 ceph-mon[32237]: 5.19 scrub ok 2024-09-18T07:48:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:05 smithi203 ceph-mon[32237]: osdmap e485: 8 total, 8 up, 8 in 2024-09-18T07:48:05.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:05.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:05.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:05.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:05.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:05.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:05.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:05.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:06.049 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:06.052 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 51 16 73505 73489 90.0435 71 0.00636408 0.0111001 2024-09-18T07:48:06.087 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:06.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:06.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:06.135 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T07:48:06.189 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:06.207 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:06.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:06.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:06.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:06.339 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:06.376 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:06.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:06.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:06.436 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:48:06.478 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:06.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:06.510 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:06.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:06.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:06.646 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:06.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:06.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:06.649 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:48:06.654 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:06 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/913438518' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:48:06.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:06 smithi137 ceph-mon[29408]: from='client.26593 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:48:06.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:06 smithi137 ceph-mon[29408]: pgmap v915: 158 pgs: 158 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 847 op/s 2024-09-18T07:48:06.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:06 smithi137 ceph-mon[29408]: 2.2 scrub starts 2024-09-18T07:48:06.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:06 smithi137 ceph-mon[29408]: 2.2 scrub ok 2024-09-18T07:48:06.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:06 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/913438518' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:48:06.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:06 smithi137 ceph-mon[39175]: from='client.26593 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:48:06.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:06 smithi137 ceph-mon[39175]: pgmap v915: 158 pgs: 158 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 847 op/s 2024-09-18T07:48:06.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:06 smithi137 ceph-mon[39175]: 2.2 scrub starts 2024-09-18T07:48:06.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:06 smithi137 ceph-mon[39175]: 2.2 scrub ok 2024-09-18T07:48:06.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:06 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/913438518' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:48:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:06 smithi203 ceph-mon[32237]: from='client.26593 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:48:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:06 smithi203 ceph-mon[32237]: pgmap v915: 158 pgs: 158 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 847 op/s 2024-09-18T07:48:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:06 smithi203 ceph-mon[32237]: 2.2 scrub starts 2024-09-18T07:48:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:06 smithi203 ceph-mon[32237]: 2.2 scrub ok 2024-09-18T07:48:06.748 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:06.784 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:06.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:06.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:06.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:06.915 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:06.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:06.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:07.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:07.052 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 52 16 74677 74661 89.7204 73.25 0.00938818 0.0111401 2024-09-18T07:48:07.061 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:07.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:07.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:07.146 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 6, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 6, 7, 4, 5] 2024-09-18T07:48:07.146 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:48:07.146 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-18T07:48:07.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-09-18T07:48:07.165 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:07.195 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:07.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:07.297 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:07.324 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:07.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:07.355 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:48:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:48:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:48:07.426 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:07.458 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:07.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:07.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:07.560 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:07.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:07.609 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:07.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:07.652 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:07 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:07.653 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:07 smithi137 ceph-mon[29408]: from='client.26593 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:48:07.653 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:07 smithi137 ceph-mon[29408]: osdmap e486: 8 total, 8 up, 8 in 2024-09-18T07:48:07.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:07 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:07.654 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:07 smithi137 ceph-mon[39175]: from='client.26593 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:48:07.654 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:07 smithi137 ceph-mon[39175]: osdmap e486: 8 total, 8 up, 8 in 2024-09-18T07:48:07.707 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:48:07.707 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:48:07.710 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:07 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:07 smithi203 ceph-mon[32237]: from='client.26593 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:48:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:07 smithi203 ceph-mon[32237]: osdmap e486: 8 total, 8 up, 8 in 2024-09-18T07:48:07.737 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:48:07.737 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:48:07.743 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:07.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:07.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:07.766 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:48:07.767 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:48:07.787 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:48:07.787 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:48:07.793 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:48:07.816 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:48:07.835 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:48:07.835 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:48:07.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:07.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:07.867 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:07.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:07.968 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:07.987 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:48:07.987 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:48:08.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:08.004 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:08.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:08.024 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:48:08.024 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:48:08.052 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 53 16 75619 75603 89.1384 58.875 0.00838924 0.0112131 2024-09-18T07:48:08.105 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:08.135 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:08.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:08.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:08.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:48:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:48:08.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:08.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:08.261 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:08.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:08.362 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:08.393 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:08.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:08.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:08.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:08.533 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:08.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:08.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:08.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:08.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:08 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1660395065' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:08.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:08 smithi137 ceph-mon[29408]: pgmap v917: 158 pgs: 158 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-09-18T07:48:08.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:08 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:08.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:08 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1660395065' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:08.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:08 smithi137 ceph-mon[39175]: pgmap v917: 158 pgs: 158 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-09-18T07:48:08.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:08 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:08.681 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:08.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:08.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:08.691 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:08 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1660395065' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:08.691 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:08 smithi203 ceph-mon[32237]: pgmap v917: 158 pgs: 158 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-09-18T07:48:08.691 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:08 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:08.783 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:08.822 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:08.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:08.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:08.838 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:48:08.838 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:48:08.924 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:08.945 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:08.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:08.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:09.047 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:09.052 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 54 16 76736 76720 88.7804 69.8125 0.0117679 0.0112589 2024-09-18T07:48:09.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:09.085 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:09.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:09.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:09.226 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:09.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:09.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:09.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:09.363 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:09.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:09.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:09.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:48:09.465 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:09.466 INFO:teuthology.orchestra.run.smithi137.stdout:moved 21 / 474 (4.43038%) 2024-09-18T07:48:09.466 INFO:teuthology.orchestra.run.smithi137.stdout:avg 59.25 2024-09-18T07:48:09.466 INFO:teuthology.orchestra.run.smithi137.stdout:stddev 6.77772 -> 4.84123 (expected baseline 7.20026) 2024-09-18T07:48:09.466 INFO:teuthology.orchestra.run.smithi137.stdout:min osd.2 with 47 -> 55 pgs (0.793249 -> 0.92827 * mean) 2024-09-18T07:48:09.466 INFO:teuthology.orchestra.run.smithi137.stdout:max osd.1 with 67 -> 56 pgs (1.1308 -> 0.945148 * mean) 2024-09-18T07:48:09.466 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:48:09.466 INFO:teuthology.orchestra.run.smithi137.stdout:oload 110 2024-09-18T07:48:09.466 INFO:teuthology.orchestra.run.smithi137.stdout:max_change 3 2024-09-18T07:48:09.466 INFO:teuthology.orchestra.run.smithi137.stdout:max_change_osds 4 2024-09-18T07:48:09.467 INFO:teuthology.orchestra.run.smithi137.stdout:average_utilization 0.0265 2024-09-18T07:48:09.467 INFO:teuthology.orchestra.run.smithi137.stdout:overload_utilization 0.0291 2024-09-18T07:48:09.467 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 weight 1.0000 -> 0.8067 2024-09-18T07:48:09.467 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 weight 1.0000 -> 0.8307 2024-09-18T07:48:09.467 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 weight 1.0000 -> 0.8851 2024-09-18T07:48:09.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:09.533 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:09.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:09.634 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:09.661 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T07:48:09.667 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:09.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:09.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:09.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:09.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:09.812 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:09.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:09.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:09.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:09 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:48:09.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:09 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:48:09.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:09 smithi137 ceph-mon[29408]: osdmap e487: 8 total, 8 up, 8 in 2024-09-18T07:48:09.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:09 smithi137 ceph-mon[29408]: from='client.26605 v1:172.21.15.137:0/2699005524' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:48:09.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:09 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'52869','3':'58006','6':'54441'}"} : dispatch 2024-09-18T07:48:09.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:09 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'52869','3':'58006','6':'54441'}"} : dispatch 2024-09-18T07:48:09.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:09 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:48:09.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:09 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:48:09.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:09 smithi137 ceph-mon[39175]: osdmap e487: 8 total, 8 up, 8 in 2024-09-18T07:48:09.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:09 smithi137 ceph-mon[39175]: from='client.26605 v1:172.21.15.137:0/2699005524' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:48:09.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:09 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'52869','3':'58006','6':'54441'}"} : dispatch 2024-09-18T07:48:09.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:09 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'52869','3':'58006','6':'54441'}"} : dispatch 2024-09-18T07:48:09.949 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:09.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:09.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:09.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:09 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:48:09.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:09 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:48:09.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:09 smithi203 ceph-mon[32237]: osdmap e487: 8 total, 8 up, 8 in 2024-09-18T07:48:09.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:09 smithi203 ceph-mon[32237]: from='client.26605 v1:172.21.15.137:0/2699005524' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:48:09.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:09 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'52869','3':'58006','6':'54441'}"} : dispatch 2024-09-18T07:48:09.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:09 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'52869','3':'58006','6':'54441'}"} : dispatch 2024-09-18T07:48:10.050 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:10.052 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 55 16 77934 77918 88.5275 74.875 0.016982 0.011291 2024-09-18T07:48:10.066 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:48:10.088 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:10.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:10.189 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:10.224 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:10.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:10.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:10.359 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:10.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:10.460 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:10.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:10.507 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:10.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:10.608 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:10.640 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:10.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:10.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:10.725 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:48:10.752 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:48:10.752 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:10.753 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:10 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'52869','3':'58006','6':'54441'}"}]': finished 2024-09-18T07:48:10.753 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:10 smithi203 ceph-mon[32237]: osdmap e488: 8 total, 8 up, 8 in 2024-09-18T07:48:10.754 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:10 smithi203 ceph-mon[32237]: pgmap v920: 158 pgs: 158 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-18T07:48:10.754 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:10 smithi203 ceph-mon[32237]: osdmap e489: 8 total, 8 up, 8 in 2024-09-18T07:48:10.800 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:10.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:10.813 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:10 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'52869','3':'58006','6':'54441'}"}]': finished 2024-09-18T07:48:10.813 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:10 smithi137 ceph-mon[29408]: osdmap e488: 8 total, 8 up, 8 in 2024-09-18T07:48:10.813 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:10 smithi137 ceph-mon[29408]: pgmap v920: 158 pgs: 158 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-18T07:48:10.813 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:10 smithi137 ceph-mon[29408]: osdmap e489: 8 total, 8 up, 8 in 2024-09-18T07:48:10.816 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:10 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'52869','3':'58006','6':'54441'}"}]': finished 2024-09-18T07:48:10.816 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:10 smithi137 ceph-mon[39175]: osdmap e488: 8 total, 8 up, 8 in 2024-09-18T07:48:10.816 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:10 smithi137 ceph-mon[39175]: pgmap v920: 158 pgs: 158 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-18T07:48:10.816 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:10 smithi137 ceph-mon[39175]: osdmap e489: 8 total, 8 up, 8 in 2024-09-18T07:48:10.869 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-18T07:48:10.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 110 1.0 2024-09-18T07:48:10.897 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:10.902 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:10.930 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:10.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:10.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:11.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:11.055 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 56 16 79120 79104 88.266 74.125 0.0213612 0.0113227 2024-09-18T07:48:11.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:11.072 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:11.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:11.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:11.216 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:11.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:11.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:11.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:11.359 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:11.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:11.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:11.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:11.506 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:11.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:11.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:11.534 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:11 smithi203 ceph-mon[32237]: 1.3 scrub starts 2024-09-18T07:48:11.579 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T07:48:11.580 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:11.608 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:11.666 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:11.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:11.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:11.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:11 smithi137 ceph-mon[29408]: 1.3 scrub starts 2024-09-18T07:48:11.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:11 smithi137 ceph-mon[39175]: 1.3 scrub starts 2024-09-18T07:48:11.768 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:11.811 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:11.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:11.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:11.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:11.952 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:48:11.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:11.965 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:11.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:12.055 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 57 16 80223 80207 87.9268 68.9375 0.0111087 0.0113677 2024-09-18T07:48:12.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:12.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:12.102 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:12.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:12.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:48:12.203 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:12.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:12.250 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:12.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:12.351 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:12.377 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:12.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:12.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:12.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:12.496 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:48:12.528 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:48:12.528 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:48:12.555 INFO:teuthology.orchestra.run.smithi137.stdout:moved 27 / 474 (5.6962%) 2024-09-18T07:48:12.555 INFO:teuthology.orchestra.run.smithi137.stdout:avg 59.25 2024-09-18T07:48:12.555 INFO:teuthology.orchestra.run.smithi137.stdout:stddev 4.84123 -> 9.36416 (expected baseline 7.20026) 2024-09-18T07:48:12.556 INFO:teuthology.orchestra.run.smithi137.stdout:min osd.6 with 53 -> 41 pgs (0.894515 -> 0.691983 * mean) 2024-09-18T07:48:12.556 INFO:teuthology.orchestra.run.smithi137.stdout:max osd.7 with 67 -> 70 pgs (1.1308 -> 1.18143 * mean) 2024-09-18T07:48:12.556 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:48:12.556 INFO:teuthology.orchestra.run.smithi137.stdout:oload 110 2024-09-18T07:48:12.556 INFO:teuthology.orchestra.run.smithi137.stdout:max_change 1 2024-09-18T07:48:12.556 INFO:teuthology.orchestra.run.smithi137.stdout:max_change_osds 4 2024-09-18T07:48:12.556 INFO:teuthology.orchestra.run.smithi137.stdout:average_utilization 0.0278 2024-09-18T07:48:12.556 INFO:teuthology.orchestra.run.smithi137.stdout:overload_utilization 0.0305 2024-09-18T07:48:12.556 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 weight 0.8067 -> 0.6418 2024-09-18T07:48:12.557 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 weight 0.8307 -> 0.6830 2024-09-18T07:48:12.557 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 weight 0.8851 -> 0.7764 2024-09-18T07:48:12.557 INFO:teuthology.orchestra.run.smithi137.stderr:no change 2024-09-18T07:48:12.558 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:12.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:12.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:12.661 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:12.669 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:48:12 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: 2024-09-18T07:48:12.547+0000 7f460a846640 -1 mgr.server reply reply (3) No such process no change 2024-09-18T07:48:12.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:12 smithi137 ceph-mon[29408]: pgmap v922: 158 pgs: 158 active+clean; 4.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 5.4 KiB/s rd, 116 MiB/s wr, 1.85k op/s; 39 KiB/s, 3 objects/s recovering 2024-09-18T07:48:12.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:12 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4136133421' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:48:12.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:12 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:48:12.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:48:12.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:12 smithi137 ceph-mon[39175]: pgmap v922: 158 pgs: 158 active+clean; 4.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 5.4 KiB/s rd, 116 MiB/s wr, 1.85k op/s; 39 KiB/s, 3 objects/s recovering 2024-09-18T07:48:12.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:12 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4136133421' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:48:12.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:12 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:48:12.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:48:12.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:12.694 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:12.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:12.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:12 smithi203 ceph-mon[32237]: pgmap v922: 158 pgs: 158 active+clean; 4.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 5.4 KiB/s rd, 116 MiB/s wr, 1.85k op/s; 39 KiB/s, 3 objects/s recovering 2024-09-18T07:48:12.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:12 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4136133421' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:48:12.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:12 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:48:12.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:48:12.795 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:12.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:12.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:12.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:12.933 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:12.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:12.968 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:12.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:13.056 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 58 15 81339 81324 87.6144 69.8125 0.0143904 0.011409 2024-09-18T07:48:13.070 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:13.105 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:13.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:13.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:13.227 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T07:48:13.244 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:13.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:13.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:13.346 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:13.381 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:13.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:13.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:13.519 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:13.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:13.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:13.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:13.633 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:48:13.647 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:13.681 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:13.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:13.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:13.760 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:13 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:13.761 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:13 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4136133421' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:48:13.761 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:13 smithi137 ceph-mon[29408]: osdmap e490: 8 total, 8 up, 8 in 2024-09-18T07:48:13.761 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:13 smithi137 ceph-mon[29408]: from='client.36518 v1:172.21.15.137:0/3988603158' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:48:13.762 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:13 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:13.762 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:13 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4136133421' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:48:13.762 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:13 smithi203 ceph-mon[32237]: osdmap e490: 8 total, 8 up, 8 in 2024-09-18T07:48:13.762 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:13 smithi203 ceph-mon[32237]: from='client.36518 v1:172.21.15.137:0/3988603158' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:48:13.762 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:13 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:13.762 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:13 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4136133421' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:48:13.762 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:13 smithi137 ceph-mon[39175]: osdmap e490: 8 total, 8 up, 8 in 2024-09-18T07:48:13.762 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:13 smithi137 ceph-mon[39175]: from='client.36518 v1:172.21.15.137:0/3988603158' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:48:13.783 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:13.824 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:13.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:13.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:13.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:13.968 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:13.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:13.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:14.056 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 59 15 82616 82601 87.482 79.8125 0.00644269 0.011426 2024-09-18T07:48:14.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:14.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:14.115 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:14.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:14.157 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-18T07:48:14.157 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-09-18T07:48:14.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:14.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:14.257 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:14.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:14.358 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:14.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:14.407 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:14.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:14.508 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:14.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:14.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:14.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:14.651 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:14.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:14 smithi137 ceph-mon[29408]: pgmap v924: 158 pgs: 158 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 4.5 KiB/s rd, 65 MiB/s wr, 1.05k op/s; 39 KiB/s, 4 objects/s recovering 2024-09-18T07:48:14.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:14 smithi137 ceph-mon[39175]: pgmap v924: 158 pgs: 158 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 4.5 KiB/s rd, 65 MiB/s wr, 1.05k op/s; 39 KiB/s, 4 objects/s recovering 2024-09-18T07:48:14.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:14.690 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:14.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:14.782 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:48:14.782 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:48:14.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:14.793 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:14.808 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:14 smithi203 ceph-mon[32237]: pgmap v924: 158 pgs: 158 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 4.5 KiB/s rd, 65 MiB/s wr, 1.05k op/s; 39 KiB/s, 4 objects/s recovering 2024-09-18T07:48:14.820 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:14.820 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:48:14.820 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:48:14.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:14.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:14.852 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:48:14.852 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:48:14.883 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:48:14.883 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:48:14.919 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:48:14.919 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:48:14.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:14.952 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:48:14.952 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:48:14.955 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:48:14.955 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:48:14.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:14.975 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:14.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:14.987 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:48:14.987 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:48:15.020 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:48:15.020 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:48:15.055 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:48:15.053949+0000 min lat: 0.00129363 max lat: 1.97884 avg lat: 0.0114695 2024-09-18T07:48:15.056 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:48:15.056 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 60 15 83695 83680 87.1479 67.4375 0.0171769 0.0114695 2024-09-18T07:48:15.076 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:15.108 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:15.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:15.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:15.210 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:15.260 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:15.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:15.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:15.362 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:15.399 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:15.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:15.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:15.501 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:15.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:15.534 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:15.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:15.608 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:48:15.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:15.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T07:48:15.698 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:15.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:15.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:15.800 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:15.837 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:15.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:15.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:15.857 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:15 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/264089501' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:15.857 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:15 smithi137 ceph-mon[29408]: osdmap e491: 8 total, 8 up, 8 in 2024-09-18T07:48:15.857 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:15 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/264089501' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:15.857 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:15 smithi137 ceph-mon[39175]: osdmap e491: 8 total, 8 up, 8 in 2024-09-18T07:48:15.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:15.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:15 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/264089501' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:15.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:15 smithi203 ceph-mon[32237]: osdmap e491: 8 total, 8 up, 8 in 2024-09-18T07:48:15.985 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:15.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:16.038 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:48:16.055 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 61 16 84873 84857 86.9251 73.5625 0.0054017 0.0114987 2024-09-18T07:48:16.086 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:16.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:16.127 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:16.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:16.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:16.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:16.286 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:16.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:16.292 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:16.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:16.437 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:16.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:16.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:16.522 INFO:teuthology.orchestra.run.smithi137.stdout:moved 0 / 474 (0%) 2024-09-18T07:48:16.523 INFO:teuthology.orchestra.run.smithi137.stdout:avg 59.25 2024-09-18T07:48:16.523 INFO:teuthology.orchestra.run.smithi137.stdout:stddev 4.84123 -> 4.84123 (expected baseline 7.20026) 2024-09-18T07:48:16.523 INFO:teuthology.orchestra.run.smithi137.stdout:min osd.6 with 53 -> 53 pgs (0.894515 -> 0.894515 * mean) 2024-09-18T07:48:16.523 INFO:teuthology.orchestra.run.smithi137.stdout:max osd.7 with 67 -> 67 pgs (1.1308 -> 1.1308 * mean) 2024-09-18T07:48:16.523 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:48:16.523 INFO:teuthology.orchestra.run.smithi137.stdout:oload 1000 2024-09-18T07:48:16.523 INFO:teuthology.orchestra.run.smithi137.stdout:max_change 1 2024-09-18T07:48:16.523 INFO:teuthology.orchestra.run.smithi137.stdout:max_change_osds 4 2024-09-18T07:48:16.523 INFO:teuthology.orchestra.run.smithi137.stdout:average_utilization 0.0283 2024-09-18T07:48:16.523 INFO:teuthology.orchestra.run.smithi137.stdout:overload_utilization 0.2830 2024-09-18T07:48:16.524 INFO:teuthology.orchestra.run.smithi137.stderr:no change 2024-09-18T07:48:16.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:16.580 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:16.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:16.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:16.641 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:48:16.641 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:48:16.641 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:48:16.641 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:48:16.642 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:48:16.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:16.701 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:16 smithi137 ceph-mon[29408]: pgmap v926: 158 pgs: 158 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 4.0 KiB/s rd, 59 MiB/s wr, 947 op/s; 35 KiB/s, 5 objects/s recovering 2024-09-18T07:48:16.701 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:16 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:48:16.701 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:16 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:48:16.702 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:16 smithi137 ceph-mon[39175]: pgmap v926: 158 pgs: 158 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 4.0 KiB/s rd, 59 MiB/s wr, 947 op/s; 35 KiB/s, 5 objects/s recovering 2024-09-18T07:48:16.702 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:16 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:48:16.702 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:16 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:48:16.725 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:16.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:16.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:16.827 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:16.862 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:16.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:16.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:16.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:16 smithi203 ceph-mon[32237]: pgmap v926: 158 pgs: 158 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 4.0 KiB/s rd, 59 MiB/s wr, 947 op/s; 35 KiB/s, 5 objects/s recovering 2024-09-18T07:48:16.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:16 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:48:16.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:16 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:48:16.963 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:16.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:16.995 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:16.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:17.056 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 62 16 85949 85933 86.6077 67.25 0.0169834 0.0115418 2024-09-18T07:48:17.096 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:17.133 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:17.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:17.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:17.234 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:17.268 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:17.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:17.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:17.370 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:17.389 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:48:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:48:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:48:17.405 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:17.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:17.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:17.417 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T07:48:17.441 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-18T07:48:17.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-09-18T07:48:17.508 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:17.539 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:17.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:17.641 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:17.650 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:17 smithi137 ceph-mon[29408]: osdmap e492: 8 total, 8 up, 8 in 2024-09-18T07:48:17.650 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:17 smithi137 ceph-mon[29408]: from='client.36527 v1:172.21.15.137:0/3944988745' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:48:17.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:17 smithi137 ceph-mon[39175]: osdmap e492: 8 total, 8 up, 8 in 2024-09-18T07:48:17.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:17 smithi137 ceph-mon[39175]: from='client.36527 v1:172.21.15.137:0/3944988745' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:48:17.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:17.687 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:17.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:17.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:17 smithi203 ceph-mon[32237]: osdmap e492: 8 total, 8 up, 8 in 2024-09-18T07:48:17.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:17 smithi203 ceph-mon[32237]: from='client.36527 v1:172.21.15.137:0/3944988745' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:48:17.788 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:17.820 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:48:17.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:17.838 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:17.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:17.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:48:17.931 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:17.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:18.004 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:18.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:18.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:18.057 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 63 16 87026 87010 86.2989 67.3125 0.00561469 0.011582 2024-09-18T07:48:18.106 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:18.147 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:18.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:18.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:18.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:48:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:48:18.249 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:18.283 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:18.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:18.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:18.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:18.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:18.422 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:18.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:18.523 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:18.530 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:48:18.552 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:18.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:18.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:18.562 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:18 smithi137 ceph-mon[29408]: pgmap v928: 158 pgs: 158 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.18k op/s; 145 B/s, 2 objects/s recovering 2024-09-18T07:48:18.562 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:18 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:18.562 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:18 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2002867408' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:48:18.563 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:18 smithi137 ceph-mon[39175]: pgmap v928: 158 pgs: 158 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.18k op/s; 145 B/s, 2 objects/s recovering 2024-09-18T07:48:18.563 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:18 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:18.563 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:18 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2002867408' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:48:18.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:18.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:18.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:18.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:18.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:18 smithi203 ceph-mon[32237]: pgmap v928: 158 pgs: 158 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.18k op/s; 145 B/s, 2 objects/s recovering 2024-09-18T07:48:18.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:18 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:18.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:18 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2002867408' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:48:18.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:18.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:18.851 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:18.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:18.865 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:18.952 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:18.961 INFO:teuthology.orchestra.run.smithi137.stdout:moved 0 / 474 (0%) 2024-09-18T07:48:18.961 INFO:teuthology.orchestra.run.smithi137.stdout:avg 59.25 2024-09-18T07:48:18.961 INFO:teuthology.orchestra.run.smithi137.stdout:stddev 4.84123 -> 4.84123 (expected baseline 7.20026) 2024-09-18T07:48:18.961 INFO:teuthology.orchestra.run.smithi137.stdout:min osd.6 with 53 -> 53 pgs (0.894515 -> 0.894515 * mean) 2024-09-18T07:48:18.961 INFO:teuthology.orchestra.run.smithi137.stdout:max osd.7 with 67 -> 67 pgs (1.1308 -> 1.1308 * mean) 2024-09-18T07:48:18.962 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:48:18.962 INFO:teuthology.orchestra.run.smithi137.stdout:oload 1000 2024-09-18T07:48:18.962 INFO:teuthology.orchestra.run.smithi137.stdout:max_change 3 2024-09-18T07:48:18.962 INFO:teuthology.orchestra.run.smithi137.stdout:max_change_osds 4 2024-09-18T07:48:18.962 INFO:teuthology.orchestra.run.smithi137.stdout:average_utilization 0.0293 2024-09-18T07:48:18.962 INFO:teuthology.orchestra.run.smithi137.stdout:overload_utilization 0.2932 2024-09-18T07:48:18.962 INFO:teuthology.orchestra.run.smithi137.stderr:no change 2024-09-18T07:48:18.985 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:18.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:18.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:18.990 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:48:18.990 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:48:18.991 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:48:18.991 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:48:18.991 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:18.991 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.35171.0:170414 8.5 e264/255 8:a29be62f:::benchmark_data_smithi137_101942_object65604:head v 264'10576, mlcod=264'10574)", 2024-09-18T07:48:18.991 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:38:44.768915+0000", 2024-09-18T07:48:18.991 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 573.66764719299999, 2024-09-18T07:48:18.991 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15600961199999999, 2024-09-18T07:48:18.991 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:18.991 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:18.991 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:18.991 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:18.991 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:18.991 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:18.992 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768915+0000", 2024-09-18T07:48:18.992 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:18.992 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:18.992 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:18.992 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:18.992 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768915+0000", 2024-09-18T07:48:18.992 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:18.992 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:18.992 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:18.992 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:18.992 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768916+0000", 2024-09-18T07:48:18.992 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.24e-06 2024-09-18T07:48:18.993 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:18.993 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:18.993 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:18.993 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768923+0000", 2024-09-18T07:48:18.993 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7410000000000003e-06 2024-09-18T07:48:18.994 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:18.994 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:18.994 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:18.994 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768936+0000", 2024-09-18T07:48:18.994 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.345e-05 2024-09-18T07:48:18.994 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:18.994 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:18.994 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:18.994 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.768944+0000", 2024-09-18T07:48:18.994 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0299999999999996e-06 2024-09-18T07:48:18.994 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:18.994 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:18.995 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:18.995 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.770341+0000", 2024-09-18T07:48:18.995 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001397061 2024-09-18T07:48:18.995 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:18.995 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:18.995 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:18.995 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.770358+0000", 2024-09-18T07:48:18.995 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7825999999999998e-05 2024-09-18T07:48:18.995 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:18.995 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:18.995 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:18.995 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.924876+0000", 2024-09-18T07:48:18.996 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.154517618 2024-09-18T07:48:18.996 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:18.996 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:18.996 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:18.996 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:38:44.924925+0000", 2024-09-18T07:48:18.996 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8646000000000001e-05 2024-09-18T07:48:18.996 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:18.996 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:18.996 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:18.996 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:18.997 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:18.997 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:156461 9.d 9:b7fc06f1:::benchmark_data_smithi137_126072_object23961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:48:18.997 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:24.401407+0000", 2024-09-18T07:48:18.997 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 414.03515569000001, 2024-09-18T07:48:18.997 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16122866899999999, 2024-09-18T07:48:18.997 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:18.997 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:18.998 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:18.998 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:18.998 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:48:18.998 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:48:18.998 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 156461 2024-09-18T07:48:18.998 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:18.998 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:18.998 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:18.998 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:18.998 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401407+0000", 2024-09-18T07:48:18.998 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:18.998 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:18.999 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:18.999 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:18.999 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401407+0000", 2024-09-18T07:48:18.999 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:18.999 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:18.999 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:18.999 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:18.999 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401409+0000", 2024-09-18T07:48:19.000 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0150000000000002e-06 2024-09-18T07:48:19.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.000 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.000 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.000 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401414+0000", 2024-09-18T07:48:19.000 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6810000000000003e-06 2024-09-18T07:48:19.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.000 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.000 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.000 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401428+0000", 2024-09-18T07:48:19.000 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4049000000000001e-05 2024-09-18T07:48:19.000 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.001 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.001 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.001 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401436+0000", 2024-09-18T07:48:19.001 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5549999999999997e-06 2024-09-18T07:48:19.001 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.001 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.001 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.001 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401509+0000", 2024-09-18T07:48:19.002 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3232999999999995e-05 2024-09-18T07:48:19.002 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.002 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.002 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.002 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401613+0000", 2024-09-18T07:48:19.002 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010402400000000001 2024-09-18T07:48:19.002 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.002 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.002 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:48:19.002 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.401718+0000", 2024-09-18T07:48:19.002 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010482800000000001 2024-09-18T07:48:19.002 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.002 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.003 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409266+0000", 2024-09-18T07:48:19.003 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0075485639999999998 2024-09-18T07:48:19.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.003 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.003 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.003 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409351+0000", 2024-09-18T07:48:19.003 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4796000000000006e-05 2024-09-18T07:48:19.003 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.003 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.004 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.004 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.562555+0000", 2024-09-18T07:48:19.004 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.153203325 2024-09-18T07:48:19.004 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.004 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.004 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:19.004 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.562599+0000", 2024-09-18T07:48:19.004 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4431000000000002e-05 2024-09-18T07:48:19.004 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.004 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.005 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.005 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.562635+0000", 2024-09-18T07:48:19.005 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6168000000000003e-05 2024-09-18T07:48:19.005 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.005 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.005 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.005 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.005 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.005 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:156463 9.d 9:b608ec1e:::benchmark_data_smithi137_126072_object23963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:48:19.005 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:24.403102+0000", 2024-09-18T07:48:19.005 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 414.03346053000001, 2024-09-18T07:48:19.005 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16236466999999999, 2024-09-18T07:48:19.006 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.006 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.006 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:19.006 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.006 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:48:19.006 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:48:19.006 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 156463 2024-09-18T07:48:19.006 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.006 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.006 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.006 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.006 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403102+0000", 2024-09-18T07:48:19.006 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.007 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.007 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.007 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.007 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403102+0000", 2024-09-18T07:48:19.007 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.007 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.007 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.007 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.007 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403103+0000", 2024-09-18T07:48:19.007 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.767e-06 2024-09-18T07:48:19.007 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.007 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.007 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.008 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403108+0000", 2024-09-18T07:48:19.008 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5549999999999996e-06 2024-09-18T07:48:19.008 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.008 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.008 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.008 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403117+0000", 2024-09-18T07:48:19.008 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9390000000000003e-06 2024-09-18T07:48:19.008 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.008 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.008 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.008 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.403123+0000", 2024-09-18T07:48:19.008 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4309999999999999e-06 2024-09-18T07:48:19.008 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.009 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.009 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.009 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404263+0000", 2024-09-18T07:48:19.009 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001140032 2024-09-18T07:48:19.009 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.009 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.009 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.009 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404367+0000", 2024-09-18T07:48:19.009 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000103514 2024-09-18T07:48:19.009 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.009 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.009 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:48:19.009 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404535+0000", 2024-09-18T07:48:19.009 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000167641 2024-09-18T07:48:19.010 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.010 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.010 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409418+0000", 2024-09-18T07:48:19.010 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0048834990000000003 2024-09-18T07:48:19.010 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.010 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.010 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409552+0000", 2024-09-18T07:48:19.010 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000133688 2024-09-18T07:48:19.010 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.010 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.010 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.011 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.564499+0000", 2024-09-18T07:48:19.011 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15494714900000001 2024-09-18T07:48:19.011 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.011 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.011 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:19.011 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.565425+0000", 2024-09-18T07:48:19.011 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00092621499999999998 2024-09-18T07:48:19.011 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.011 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.011 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.011 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.565466+0000", 2024-09-18T07:48:19.011 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1239999999999998e-05 2024-09-18T07:48:19.011 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.011 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.012 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.012 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.012 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.012 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:156465 9.e 9:710e39e4:::benchmark_data_smithi137_126072_object23965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-18T07:48:19.012 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:24.404394+0000", 2024-09-18T07:48:19.012 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 414.03216870799997, 2024-09-18T07:48:19.012 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.157357105, 2024-09-18T07:48:19.012 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.012 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.012 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:19.012 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.012 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:48:19.012 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:48:19.013 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 156465 2024-09-18T07:48:19.013 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.013 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.013 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.013 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.013 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404394+0000", 2024-09-18T07:48:19.013 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.013 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.013 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.013 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.013 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404394+0000", 2024-09-18T07:48:19.013 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.014 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.014 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.014 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.014 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404395+0000", 2024-09-18T07:48:19.014 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.55e-06 2024-09-18T07:48:19.014 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.014 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.014 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.014 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404400+0000", 2024-09-18T07:48:19.014 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5299999999999998e-06 2024-09-18T07:48:19.014 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.014 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.014 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.014 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404410+0000", 2024-09-18T07:48:19.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0139999999999999e-05 2024-09-18T07:48:19.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404417+0000", 2024-09-18T07:48:19.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9979999999999997e-06 2024-09-18T07:48:19.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.015 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.015 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404520+0000", 2024-09-18T07:48:19.015 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000103601 2024-09-18T07:48:19.015 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.015 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.016 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.016 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404618+0000", 2024-09-18T07:48:19.016 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7317000000000004e-05 2024-09-18T07:48:19.016 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.016 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.016 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:19.016 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.404713+0000", 2024-09-18T07:48:19.016 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5427000000000006e-05 2024-09-18T07:48:19.016 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.016 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.016 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.016 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409836+0000", 2024-09-18T07:48:19.017 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0051231840000000002 2024-09-18T07:48:19.017 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.017 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.017 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.017 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.409962+0000", 2024-09-18T07:48:19.017 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012611299999999999 2024-09-18T07:48:19.017 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.017 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.017 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.017 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.561643+0000", 2024-09-18T07:48:19.017 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15168100000000001 2024-09-18T07:48:19.017 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.018 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.018 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:19.018 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.561701+0000", 2024-09-18T07:48:19.018 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8010000000000002e-05 2024-09-18T07:48:19.018 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.018 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.018 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.018 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:24.561751+0000", 2024-09-18T07:48:19.018 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9234999999999999e-05 2024-09-18T07:48:19.018 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.019 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.019 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.019 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.019 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.019 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:172965 9.e 9:7fd9b0fe:::benchmark_data_smithi137_126072_object40465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-18T07:48:19.019 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:38.554115+0000", 2024-09-18T07:48:19.019 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 399.88244756799998, 2024-09-18T07:48:19.019 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.179108299, 2024-09-18T07:48:19.019 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.019 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.019 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:19.019 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.019 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:48:19.020 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:48:19.020 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 172965 2024-09-18T07:48:19.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.020 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.020 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.020 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.020 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554115+0000", 2024-09-18T07:48:19.020 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.020 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.020 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.020 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.020 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554115+0000", 2024-09-18T07:48:19.020 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.021 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.021 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554116+0000", 2024-09-18T07:48:19.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.668e-06 2024-09-18T07:48:19.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.021 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.021 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554122+0000", 2024-09-18T07:48:19.021 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9159999999999998e-06 2024-09-18T07:48:19.021 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.021 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.021 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.022 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554133+0000", 2024-09-18T07:48:19.022 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1134e-05 2024-09-18T07:48:19.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.022 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.022 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554139+0000", 2024-09-18T07:48:19.022 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1889999999999998e-06 2024-09-18T07:48:19.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.022 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.022 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554177+0000", 2024-09-18T07:48:19.022 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8720000000000002e-05 2024-09-18T07:48:19.022 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.022 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.023 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554264+0000", 2024-09-18T07:48:19.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6942999999999997e-05 2024-09-18T07:48:19.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.023 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.023 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:19.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554389+0000", 2024-09-18T07:48:19.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012495799999999999 2024-09-18T07:48:19.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.023 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.023 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.023 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.559699+0000", 2024-09-18T07:48:19.023 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0053099760000000001 2024-09-18T07:48:19.023 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.024 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.559715+0000", 2024-09-18T07:48:19.024 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5586999999999998e-05 2024-09-18T07:48:19.024 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.024 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733124+0000", 2024-09-18T07:48:19.024 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17340927 2024-09-18T07:48:19.024 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.024 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.024 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:19.024 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733181+0000", 2024-09-18T07:48:19.025 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6845000000000002e-05 2024-09-18T07:48:19.025 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.025 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.025 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.025 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733223+0000", 2024-09-18T07:48:19.025 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2092999999999998e-05 2024-09-18T07:48:19.025 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.025 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.025 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.025 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.025 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.026 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:172970 9.3 9:c7e8a587:::benchmark_data_smithi137_126072_object40470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-18T07:48:19.026 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:38.554401+0000", 2024-09-18T07:48:19.026 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 399.88216105200001, 2024-09-18T07:48:19.026 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17690262100000001, 2024-09-18T07:48:19.026 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.026 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.026 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:19.026 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.026 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:48:19.026 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:48:19.026 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 172970 2024-09-18T07:48:19.026 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.027 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554401+0000", 2024-09-18T07:48:19.027 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554401+0000", 2024-09-18T07:48:19.027 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.027 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.027 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.027 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.027 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554402+0000", 2024-09-18T07:48:19.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.299e-06 2024-09-18T07:48:19.028 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.028 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.028 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.028 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554407+0000", 2024-09-18T07:48:19.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6870000000000002e-06 2024-09-18T07:48:19.028 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.028 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.028 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.028 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554418+0000", 2024-09-18T07:48:19.028 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0696000000000001e-05 2024-09-18T07:48:19.028 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.028 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.028 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.029 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554426+0000", 2024-09-18T07:48:19.029 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0989999999999999e-06 2024-09-18T07:48:19.029 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.029 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.029 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.029 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554503+0000", 2024-09-18T07:48:19.029 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6638999999999996e-05 2024-09-18T07:48:19.029 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.029 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.029 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.029 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554615+0000", 2024-09-18T07:48:19.029 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000112816 2024-09-18T07:48:19.029 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.030 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.030 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:48:19.030 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.554762+0000", 2024-09-18T07:48:19.030 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014699099999999999 2024-09-18T07:48:19.030 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.030 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.030 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.030 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.558265+0000", 2024-09-18T07:48:19.030 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0035024539999999999 2024-09-18T07:48:19.030 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.030 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.030 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.030 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.564048+0000", 2024-09-18T07:48:19.030 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0057829969999999998 2024-09-18T07:48:19.031 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.031 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.031 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.031 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731227+0000", 2024-09-18T07:48:19.031 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16717880500000001 2024-09-18T07:48:19.031 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.031 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.031 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:19.031 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731260+0000", 2024-09-18T07:48:19.031 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2845000000000001e-05 2024-09-18T07:48:19.031 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.031 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.031 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.031 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731304+0000", 2024-09-18T07:48:19.032 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4292999999999998e-05 2024-09-18T07:48:19.032 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.032 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.032 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.032 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.032 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.032 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:172972 9.e 9:796b9798:::benchmark_data_smithi137_126072_object40472:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-18T07:48:19.032 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:38.556110+0000", 2024-09-18T07:48:19.032 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 399.880452236, 2024-09-18T07:48:19.032 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17726146800000001, 2024-09-18T07:48:19.032 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.032 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.033 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:19.033 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.033 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:48:19.033 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:48:19.033 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 172972 2024-09-18T07:48:19.033 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.033 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.033 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.033 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556110+0000", 2024-09-18T07:48:19.033 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.033 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.033 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.034 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556110+0000", 2024-09-18T07:48:19.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.034 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556112+0000", 2024-09-18T07:48:19.034 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.711e-06 2024-09-18T07:48:19.034 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.034 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.034 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556119+0000", 2024-09-18T07:48:19.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.644e-06 2024-09-18T07:48:19.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556129+0000", 2024-09-18T07:48:19.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9850000000000001e-06 2024-09-18T07:48:19.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.035 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.035 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.556136+0000", 2024-09-18T07:48:19.035 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7789999999999998e-06 2024-09-18T07:48:19.035 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.035 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.036 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.036 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.557007+0000", 2024-09-18T07:48:19.036 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00087092099999999998 2024-09-18T07:48:19.036 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.036 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.036 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.036 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.557105+0000", 2024-09-18T07:48:19.036 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7640999999999996e-05 2024-09-18T07:48:19.036 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.036 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.036 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:19.036 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.557203+0000", 2024-09-18T07:48:19.037 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8666999999999996e-05 2024-09-18T07:48:19.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.037 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.559808+0000", 2024-09-18T07:48:19.037 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026044599999999998 2024-09-18T07:48:19.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.037 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.559832+0000", 2024-09-18T07:48:19.037 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4426e-05 2024-09-18T07:48:19.037 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.037 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.037 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.038 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733280+0000", 2024-09-18T07:48:19.038 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17344823700000001 2024-09-18T07:48:19.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.038 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.038 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:19.038 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733334+0000", 2024-09-18T07:48:19.038 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3946999999999997e-05 2024-09-18T07:48:19.038 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.038 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.038 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.038 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.733371+0000", 2024-09-18T07:48:19.038 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7049999999999999e-05 2024-09-18T07:48:19.038 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.039 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.039 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.039 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.039 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.039 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:172980 9.3 9:ce0a7506:::benchmark_data_smithi137_126072_object40480:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-18T07:48:19.039 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:38.560199+0000", 2024-09-18T07:48:19.039 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 399.87636340699999, 2024-09-18T07:48:19.039 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.171283607, 2024-09-18T07:48:19.039 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.039 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.039 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:19.039 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.039 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:48:19.039 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:48:19.040 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 172980 2024-09-18T07:48:19.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.040 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.040 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.040 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.040 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560199+0000", 2024-09-18T07:48:19.040 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.040 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.040 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.040 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.040 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560199+0000", 2024-09-18T07:48:19.040 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.041 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560201+0000", 2024-09-18T07:48:19.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9709999999999998e-06 2024-09-18T07:48:19.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.041 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560206+0000", 2024-09-18T07:48:19.041 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3750000000000002e-06 2024-09-18T07:48:19.041 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.041 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.041 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.041 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560221+0000", 2024-09-18T07:48:19.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.487e-05 2024-09-18T07:48:19.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560228+0000", 2024-09-18T07:48:19.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4200000000000001e-06 2024-09-18T07:48:19.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.042 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560284+0000", 2024-09-18T07:48:19.042 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5084999999999999e-05 2024-09-18T07:48:19.042 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.042 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.042 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.043 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560415+0000", 2024-09-18T07:48:19.043 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013142500000000001 2024-09-18T07:48:19.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.043 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,3", 2024-09-18T07:48:19.043 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.560525+0000", 2024-09-18T07:48:19.043 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010991400000000001 2024-09-18T07:48:19.043 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.043 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.044 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.564084+0000", 2024-09-18T07:48:19.044 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00355953 2024-09-18T07:48:19.044 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.044 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.044 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.045 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.566190+0000", 2024-09-18T07:48:19.045 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021053690000000002 2024-09-18T07:48:19.045 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.045 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.045 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.045 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731396+0000", 2024-09-18T07:48:19.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16520663299999999 2024-09-18T07:48:19.046 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.046 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:19.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731446+0000", 2024-09-18T07:48:19.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9199000000000001e-05 2024-09-18T07:48:19.046 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.046 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.046 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.046 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731482+0000", 2024-09-18T07:48:19.046 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6816000000000001e-05 2024-09-18T07:48:19.046 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.046 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.047 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.047 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.047 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.047 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:172986 9.d 9:b4a569cc:::benchmark_data_smithi137_126072_object40486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-18T07:48:19.047 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:41:38.568405+0000", 2024-09-18T07:48:19.047 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 399.86815683999998, 2024-09-18T07:48:19.047 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16278958299999999, 2024-09-18T07:48:19.047 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.047 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.047 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:19.047 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.047 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:48:19.048 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:48:19.048 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 172986 2024-09-18T07:48:19.048 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.048 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.048 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.048 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.048 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568405+0000", 2024-09-18T07:48:19.048 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.048 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.048 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.048 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.048 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568405+0000", 2024-09-18T07:48:19.049 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.049 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.049 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.049 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.049 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568408+0000", 2024-09-18T07:48:19.049 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6929999999999999e-06 2024-09-18T07:48:19.049 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.049 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.049 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.049 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568414+0000", 2024-09-18T07:48:19.049 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5219999999999998e-06 2024-09-18T07:48:19.049 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.049 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.049 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.050 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568428+0000", 2024-09-18T07:48:19.050 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4134000000000001e-05 2024-09-18T07:48:19.050 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.050 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.050 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.050 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568435+0000", 2024-09-18T07:48:19.050 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.396e-06 2024-09-18T07:48:19.050 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.050 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.050 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.050 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568512+0000", 2024-09-18T07:48:19.050 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6614999999999997e-05 2024-09-18T07:48:19.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.051 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.051 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.051 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568623+0000", 2024-09-18T07:48:19.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011123799999999999 2024-09-18T07:48:19.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.051 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.051 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:48:19.051 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.568744+0000", 2024-09-18T07:48:19.051 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012053900000000001 2024-09-18T07:48:19.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.051 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.051 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.571214+0000", 2024-09-18T07:48:19.052 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0024707739999999998 2024-09-18T07:48:19.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.052 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.571292+0000", 2024-09-18T07:48:19.052 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7903000000000003e-05 2024-09-18T07:48:19.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.052 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.052 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731105+0000", 2024-09-18T07:48:19.052 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.159812378 2024-09-18T07:48:19.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.052 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:19.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731154+0000", 2024-09-18T07:48:19.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9450000000000003e-05 2024-09-18T07:48:19.053 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.053 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.053 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.053 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:41:38.731195+0000", 2024-09-18T07:48:19.053 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0941e-05 2024-09-18T07:48:19.053 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.053 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.053 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.054 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.054 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.054 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:244796 9.d 9:b8ad6f51:::benchmark_data_smithi137_126072_object112296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-18T07:48:19.054 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:36.228692+0000", 2024-09-18T07:48:19.054 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 342.20787024100002, 2024-09-18T07:48:19.054 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16804142, 2024-09-18T07:48:19.054 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.054 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.054 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:19.054 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.054 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:48:19.055 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:48:19.055 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 244796 2024-09-18T07:48:19.055 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.055 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.055 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.055 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.055 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228692+0000", 2024-09-18T07:48:19.055 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.055 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.055 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.055 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.055 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228692+0000", 2024-09-18T07:48:19.055 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.056 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.056 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.056 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.056 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228693+0000", 2024-09-18T07:48:19.056 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1930000000000001e-06 2024-09-18T07:48:19.056 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.056 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.056 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.056 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228697+0000", 2024-09-18T07:48:19.056 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.143e-06 2024-09-18T07:48:19.056 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.056 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.057 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.057 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228707+0000", 2024-09-18T07:48:19.057 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3489999999999998e-06 2024-09-18T07:48:19.057 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.057 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.057 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.057 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.228713+0000", 2024-09-18T07:48:19.057 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1720000000000004e-06 2024-09-18T07:48:19.057 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.057 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.058 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.058 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233077+0000", 2024-09-18T07:48:19.058 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0043645890000000003 2024-09-18T07:48:19.058 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.058 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.058 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.058 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233186+0000", 2024-09-18T07:48:19.058 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108081 2024-09-18T07:48:19.058 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.059 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:48:19.059 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233291+0000", 2024-09-18T07:48:19.059 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000105025 2024-09-18T07:48:19.059 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.059 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.059 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.240477+0000", 2024-09-18T07:48:19.059 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0071862430000000001 2024-09-18T07:48:19.059 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.059 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.059 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.059 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.248094+0000", 2024-09-18T07:48:19.060 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0076167459999999998 2024-09-18T07:48:19.060 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.060 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.060 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.060 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.396661+0000", 2024-09-18T07:48:19.060 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.14856715000000001 2024-09-18T07:48:19.060 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.060 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.060 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:19.060 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.396703+0000", 2024-09-18T07:48:19.061 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1850000000000001e-05 2024-09-18T07:48:19.061 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.061 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.061 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.061 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.396733+0000", 2024-09-18T07:48:19.061 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0879e-05 2024-09-18T07:48:19.061 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.061 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.061 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.062 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.062 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.062 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.15699.0:244803 9.d 9:be42e048:::benchmark_data_smithi137_126072_object112303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-18T07:48:19.062 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:42:36.233785+0000", 2024-09-18T07:48:19.062 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 342.20277680100003, 2024-09-18T07:48:19.062 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16784005699999999, 2024-09-18T07:48:19.062 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.062 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.062 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:19.062 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.062 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.15699", 2024-09-18T07:48:19.062 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/3054003456", 2024-09-18T07:48:19.063 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 244803 2024-09-18T07:48:19.063 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.063 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.063 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.063 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.063 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233785+0000", 2024-09-18T07:48:19.063 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.063 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.063 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.063 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.063 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233785+0000", 2024-09-18T07:48:19.063 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.063 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.064 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.064 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.064 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233787+0000", 2024-09-18T07:48:19.064 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8279999999999999e-06 2024-09-18T07:48:19.064 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.064 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.064 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.064 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233793+0000", 2024-09-18T07:48:19.065 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7790000000000001e-06 2024-09-18T07:48:19.065 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.065 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.065 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.065 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233803+0000", 2024-09-18T07:48:19.065 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5920000000000007e-06 2024-09-18T07:48:19.066 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.066 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.066 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.066 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.233810+0000", 2024-09-18T07:48:19.066 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7300000000000005e-06 2024-09-18T07:48:19.066 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.066 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.066 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.066 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.237384+0000", 2024-09-18T07:48:19.066 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003573629 2024-09-18T07:48:19.066 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.066 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.067 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.067 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.237500+0000", 2024-09-18T07:48:19.067 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000116216 2024-09-18T07:48:19.067 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.067 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.067 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,5", 2024-09-18T07:48:19.068 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.237637+0000", 2024-09-18T07:48:19.068 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013664699999999999 2024-09-18T07:48:19.068 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.068 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.068 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.068 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.243662+0000", 2024-09-18T07:48:19.068 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0060250429999999999 2024-09-18T07:48:19.068 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.068 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.068 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.068 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.250643+0000", 2024-09-18T07:48:19.068 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0069812499999999996 2024-09-18T07:48:19.068 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.069 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.069 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.069 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401571+0000", 2024-09-18T07:48:19.069 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15092767600000001 2024-09-18T07:48:19.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.069 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.069 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:19.069 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401601+0000", 2024-09-18T07:48:19.069 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9889000000000001e-05 2024-09-18T07:48:19.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.069 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.069 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.069 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:42:36.401625+0000", 2024-09-18T07:48:19.069 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.4777999999999999e-05 2024-09-18T07:48:19.070 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.070 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.070 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.070 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.070 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.070 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:137610 10.c 10:342470b2:::benchmark_data_smithi137_151160_object1794:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-18T07:48:19.070 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:44:42.767132+0000", 2024-09-18T07:48:19.070 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 215.66942985200001, 2024-09-18T07:48:19.070 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0830141860000002, 2024-09-18T07:48:19.070 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.071 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.071 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:48:19.071 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.071 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:19.071 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:19.071 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 137610 2024-09-18T07:48:19.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.071 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.071 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.071 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.071 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767132+0000", 2024-09-18T07:48:19.071 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.071 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.071 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.072 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767132+0000", 2024-09-18T07:48:19.072 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.072 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.072 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.072 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767134+0000", 2024-09-18T07:48:19.072 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.703e-06 2024-09-18T07:48:19.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.072 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.072 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.072 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767139+0000", 2024-09-18T07:48:19.072 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7430000000000002e-06 2024-09-18T07:48:19.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.072 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.073 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.073 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767152+0000", 2024-09-18T07:48:19.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2957e-05 2024-09-18T07:48:19.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.073 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.073 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767160+0000", 2024-09-18T07:48:19.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7740000000000004e-06 2024-09-18T07:48:19.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.073 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.073 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.073 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767214+0000", 2024-09-18T07:48:19.073 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4163000000000003e-05 2024-09-18T07:48:19.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.074 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.074 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.074 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767322+0000", 2024-09-18T07:48:19.074 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108028 2024-09-18T07:48:19.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.074 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.074 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0", 2024-09-18T07:48:19.074 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.767426+0000", 2024-09-18T07:48:19.074 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000104442 2024-09-18T07:48:19.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.074 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.074 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.770704+0000", 2024-09-18T07:48:19.075 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0032773479999999998 2024-09-18T07:48:19.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.075 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.798119+0000", 2024-09-18T07:48:19.075 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.027415001000000001 2024-09-18T07:48:19.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.075 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.075 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:19.075 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.798131+0000", 2024-09-18T07:48:19.075 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2952999999999999e-05 2024-09-18T07:48:19.076 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.076 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.076 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.076 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:43.770440+0000", 2024-09-18T07:48:19.076 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97230850300000005 2024-09-18T07:48:19.076 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.076 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.076 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:19.076 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:43.770754+0000", 2024-09-18T07:48:19.076 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00031391299999999997 2024-09-18T07:48:19.076 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.076 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.076 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.076 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.850064+0000", 2024-09-18T07:48:19.077 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0793099289999999 2024-09-18T07:48:19.077 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.077 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.077 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.077 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.850147+0000", 2024-09-18T07:48:19.077 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2729e-05 2024-09-18T07:48:19.077 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.077 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.077 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.077 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.077 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.077 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:137634 10.c 10:3000b92f:::benchmark_data_smithi137_151160_object1818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-09-18T07:48:19.077 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:44:42.837861+0000", 2024-09-18T07:48:19.078 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 215.59870135599999, 2024-09-18T07:48:19.078 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0161026510000002, 2024-09-18T07:48:19.078 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.078 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.078 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:19.078 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.078 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:19.078 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:19.078 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 137634 2024-09-18T07:48:19.078 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.078 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.078 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.079 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.837861+0000", 2024-09-18T07:48:19.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.079 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.837861+0000", 2024-09-18T07:48:19.079 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.079 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.079 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.079 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.079 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.837863+0000", 2024-09-18T07:48:19.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.86e-06 2024-09-18T07:48:19.080 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.080 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.080 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.837868+0000", 2024-09-18T07:48:19.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5010000000000004e-06 2024-09-18T07:48:19.080 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.080 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.080 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.080 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.837881+0000", 2024-09-18T07:48:19.080 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2317000000000001e-05 2024-09-18T07:48:19.081 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.081 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.081 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.081 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.837887+0000", 2024-09-18T07:48:19.081 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0220000000000001e-06 2024-09-18T07:48:19.081 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.081 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.081 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.081 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.837935+0000", 2024-09-18T07:48:19.081 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8831000000000001e-05 2024-09-18T07:48:19.081 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.081 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.081 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:19.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:42.837942+0000", 2024-09-18T07:48:19.082 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2979999999999997e-06 2024-09-18T07:48:19.082 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.082 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.082 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:43.770955+0000", 2024-09-18T07:48:19.082 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93301373200000004 2024-09-18T07:48:19.082 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.082 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.082 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:19.082 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:43.770964+0000", 2024-09-18T07:48:19.083 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0170000000000002e-06 2024-09-18T07:48:19.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.083 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.083 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.083 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.850379+0000", 2024-09-18T07:48:19.083 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.079414546 2024-09-18T07:48:19.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.083 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.083 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.083 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.850502+0000", 2024-09-18T07:48:19.083 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012337900000000001 2024-09-18T07:48:19.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.083 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.083 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,4", 2024-09-18T07:48:19.084 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.850640+0000", 2024-09-18T07:48:19.084 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000138107 2024-09-18T07:48:19.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.084 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.084 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.084 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.853493+0000", 2024-09-18T07:48:19.084 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0028527000000000001 2024-09-18T07:48:19.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.084 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.084 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.084 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.853731+0000", 2024-09-18T07:48:19.084 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023793600000000001 2024-09-18T07:48:19.084 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.084 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.085 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.085 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.853777+0000", 2024-09-18T07:48:19.085 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5531000000000002e-05 2024-09-18T07:48:19.085 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.085 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.085 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:19.085 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.853842+0000", 2024-09-18T07:48:19.085 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4917000000000002e-05 2024-09-18T07:48:19.085 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.085 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.085 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.085 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:44:44.853964+0000", 2024-09-18T07:48:19.086 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000121957 2024-09-18T07:48:19.086 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.086 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.086 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.086 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.086 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.087 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.34466.0:930 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e443)", 2024-09-18T07:48:19.087 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:46:39.586131+0000", 2024-09-18T07:48:19.087 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 98.850430775999996, 2024-09-18T07:48:19.087 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.30508526499999999, 2024-09-18T07:48:19.087 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.087 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.087 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:48:19.087 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.088 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.34466", 2024-09-18T07:48:19.088 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.203:0/952514737", 2024-09-18T07:48:19.088 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 930 2024-09-18T07:48:19.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.088 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.088 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.586131+0000", 2024-09-18T07:48:19.088 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.088 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.088 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.586131+0000", 2024-09-18T07:48:19.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.586135+0000", 2024-09-18T07:48:19.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.721e-06 2024-09-18T07:48:19.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.089 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.089 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.586149+0000", 2024-09-18T07:48:19.089 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4022000000000001e-05 2024-09-18T07:48:19.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.089 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.090 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.090 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.586174+0000", 2024-09-18T07:48:19.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5201000000000001e-05 2024-09-18T07:48:19.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.090 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.090 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.090 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.586185+0000", 2024-09-18T07:48:19.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0937e-05 2024-09-18T07:48:19.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.090 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.090 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.090 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.586289+0000", 2024-09-18T07:48:19.090 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010391999999999999 2024-09-18T07:48:19.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:19.091 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.586300+0000", 2024-09-18T07:48:19.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1219999999999999e-05 2024-09-18T07:48:19.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.091 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.091 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.890897+0000", 2024-09-18T07:48:19.091 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.30459635200000001 2024-09-18T07:48:19.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.091 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.891053+0000", 2024-09-18T07:48:19.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015600600000000001 2024-09-18T07:48:19.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.092 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.092 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.092 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.891217+0000", 2024-09-18T07:48:19.092 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000163886 2024-09-18T07:48:19.092 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.092 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.092 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.092 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.093 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.093 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:38866 11.5 11:aa7d8bc1:::benchmark_data_smithi137_181069_object38865:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-09-18T07:48:19.093 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:40.911677+0000", 2024-09-18T07:48:19.093 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 37.524885394999998, 2024-09-18T07:48:19.093 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.163743535, 2024-09-18T07:48:19.093 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.093 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.093 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:19.093 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.093 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:19.093 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:19.093 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 38866 2024-09-18T07:48:19.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.094 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.094 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.094 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.094 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.911677+0000", 2024-09-18T07:48:19.094 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.094 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.094 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.094 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.911677+0000", 2024-09-18T07:48:19.094 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.094 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.095 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.911678+0000", 2024-09-18T07:48:19.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.576e-06 2024-09-18T07:48:19.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.095 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.911706+0000", 2024-09-18T07:48:19.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8022999999999999e-05 2024-09-18T07:48:19.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.095 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.095 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.911715+0000", 2024-09-18T07:48:19.095 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.9760000000000001e-06 2024-09-18T07:48:19.095 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.095 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.096 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.096 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.911720+0000", 2024-09-18T07:48:19.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9479999999999999e-06 2024-09-18T07:48:19.096 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.096 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.096 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.096 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.911761+0000", 2024-09-18T07:48:19.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0581999999999999e-05 2024-09-18T07:48:19.096 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.096 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.096 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.096 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.911844+0000", 2024-09-18T07:48:19.096 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3206000000000003e-05 2024-09-18T07:48:19.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.097 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.097 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:48:19.097 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.911927+0000", 2024-09-18T07:48:19.100 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2508000000000004e-05 2024-09-18T07:48:19.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.101 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.101 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.913108+0000", 2024-09-18T07:48:19.101 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0011808420000000001 2024-09-18T07:48:19.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.101 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.101 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.913269+0000", 2024-09-18T07:48:19.101 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00016121700000000001 2024-09-18T07:48:19.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.101 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.101 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:41.075306+0000", 2024-09-18T07:48:19.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.162037082 2024-09-18T07:48:19.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:19.102 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:41.075377+0000", 2024-09-18T07:48:19.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1225999999999999e-05 2024-09-18T07:48:19.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.102 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:41.075420+0000", 2024-09-18T07:48:19.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.3349000000000001e-05 2024-09-18T07:48:19.102 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.102 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.102 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.103 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.103 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:38868 11.4 11:2cc92da2:::benchmark_data_smithi137_181069_object38867:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-09-18T07:48:19.103 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:40.912259+0000", 2024-09-18T07:48:19.103 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 37.524303635000003, 2024-09-18T07:48:19.103 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16215756000000001, 2024-09-18T07:48:19.103 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.103 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.103 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:19.103 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.103 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:19.103 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:19.103 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 38868 2024-09-18T07:48:19.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.104 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.104 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.912259+0000", 2024-09-18T07:48:19.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.104 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.912259+0000", 2024-09-18T07:48:19.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.104 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.105 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.912259+0000", 2024-09-18T07:48:19.105 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6099999999999999e-07 2024-09-18T07:48:19.105 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.105 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.105 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.105 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.912317+0000", 2024-09-18T07:48:19.105 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7856999999999998e-05 2024-09-18T07:48:19.105 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.105 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.105 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.105 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.912325+0000", 2024-09-18T07:48:19.106 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2609999999999996e-06 2024-09-18T07:48:19.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.106 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.106 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.106 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.912329+0000", 2024-09-18T07:48:19.106 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2359999999999998e-06 2024-09-18T07:48:19.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.106 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.106 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.106 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.912367+0000", 2024-09-18T07:48:19.106 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7846000000000003e-05 2024-09-18T07:48:19.106 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.106 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.107 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.107 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.912445+0000", 2024-09-18T07:48:19.107 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8257000000000006e-05 2024-09-18T07:48:19.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.107 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.107 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,6", 2024-09-18T07:48:19.107 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.912522+0000", 2024-09-18T07:48:19.107 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7464000000000002e-05 2024-09-18T07:48:19.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.107 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.107 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.107 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.913727+0000", 2024-09-18T07:48:19.107 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001205107 2024-09-18T07:48:19.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.108 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.108 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.108 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:40.914536+0000", 2024-09-18T07:48:19.108 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000808791 2024-09-18T07:48:19.108 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.108 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.108 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.108 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:41.074240+0000", 2024-09-18T07:48:19.108 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.159704032 2024-09-18T07:48:19.108 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.108 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.108 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:19.108 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:41.074371+0000", 2024-09-18T07:48:19.109 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000130226 2024-09-18T07:48:19.109 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.109 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.109 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.109 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:41.074416+0000", 2024-09-18T07:48:19.109 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5621999999999998e-05 2024-09-18T07:48:19.109 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.109 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.109 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.109 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.109 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.109 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:75482 11.5 11:a090401b:::benchmark_data_smithi137_181069_object75481:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-09-18T07:48:19.109 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:48:07.793440+0000", 2024-09-18T07:48:19.110 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 10.643122268000001, 2024-09-18T07:48:19.110 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17050164700000001, 2024-09-18T07:48:19.110 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.110 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.110 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:19.110 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.110 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:19.110 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:19.110 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 75482 2024-09-18T07:48:19.110 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.110 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.110 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.110 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.110 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.793440+0000", 2024-09-18T07:48:19.111 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.111 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.111 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.111 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.111 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.793440+0000", 2024-09-18T07:48:19.111 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.111 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.111 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.111 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.111 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.793441+0000", 2024-09-18T07:48:19.111 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4619999999999999e-06 2024-09-18T07:48:19.111 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.111 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.112 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.112 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.793520+0000", 2024-09-18T07:48:19.112 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8228999999999999e-05 2024-09-18T07:48:19.112 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.112 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.112 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.112 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.793531+0000", 2024-09-18T07:48:19.112 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1355000000000001e-05 2024-09-18T07:48:19.112 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.112 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.112 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.112 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.793539+0000", 2024-09-18T07:48:19.112 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5140000000000004e-06 2024-09-18T07:48:19.113 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.113 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.113 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.113 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.796102+0000", 2024-09-18T07:48:19.113 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002563638 2024-09-18T07:48:19.113 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.113 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.113 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.113 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.796233+0000", 2024-09-18T07:48:19.113 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000131116 2024-09-18T07:48:19.113 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.113 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.113 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:48:19.114 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.796441+0000", 2024-09-18T07:48:19.114 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00020799699999999999 2024-09-18T07:48:19.114 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.114 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.114 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.114 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.801669+0000", 2024-09-18T07:48:19.114 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0052273939999999998 2024-09-18T07:48:19.114 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.114 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.114 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.114 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.801721+0000", 2024-09-18T07:48:19.114 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2468e-05 2024-09-18T07:48:19.114 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.114 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.114 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.115 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.963822+0000", 2024-09-18T07:48:19.115 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.16210090799999999 2024-09-18T07:48:19.115 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.115 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.115 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:19.115 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.963894+0000", 2024-09-18T07:48:19.115 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2058999999999996e-05 2024-09-18T07:48:19.115 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.115 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.115 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.115 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.963942+0000", 2024-09-18T07:48:19.115 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7506999999999998e-05 2024-09-18T07:48:19.115 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.116 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.116 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.116 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.116 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.116 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:75483 11.5 11:a762554a:::benchmark_data_smithi137_181069_object75482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-09-18T07:48:19.116 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:48:07.793549+0000", 2024-09-18T07:48:19.116 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 10.643013007, 2024-09-18T07:48:19.116 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17876815400000001, 2024-09-18T07:48:19.116 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.116 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.116 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:19.116 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.116 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:19.116 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 75483 2024-09-18T07:48:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.793549+0000", 2024-09-18T07:48:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.793549+0000", 2024-09-18T07:48:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.117 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.118 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.118 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.793550+0000", 2024-09-18T07:48:19.118 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5099999999999999e-07 2024-09-18T07:48:19.118 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.118 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.118 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.118 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.793598+0000", 2024-09-18T07:48:19.118 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8476999999999997e-05 2024-09-18T07:48:19.118 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.118 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.118 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.118 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.793605+0000", 2024-09-18T07:48:19.118 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.049e-06 2024-09-18T07:48:19.119 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.119 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.119 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.119 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.793610+0000", 2024-09-18T07:48:19.119 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0089999999999998e-06 2024-09-18T07:48:19.119 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.119 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.119 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.119 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.799020+0000", 2024-09-18T07:48:19.119 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0054107959999999998 2024-09-18T07:48:19.119 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.119 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.119 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.120 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.799135+0000", 2024-09-18T07:48:19.120 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000114555 2024-09-18T07:48:19.120 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.120 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.120 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 5,6", 2024-09-18T07:48:19.120 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.799265+0000", 2024-09-18T07:48:19.120 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012978800000000001 2024-09-18T07:48:19.120 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.120 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.120 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.120 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.801868+0000", 2024-09-18T07:48:19.120 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026030020000000001 2024-09-18T07:48:19.120 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.120 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.121 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.121 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.801910+0000", 2024-09-18T07:48:19.121 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2502000000000001e-05 2024-09-18T07:48:19.121 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.121 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.121 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.121 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.972239+0000", 2024-09-18T07:48:19.121 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.17032877799999999 2024-09-18T07:48:19.121 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.121 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.121 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:19.121 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.972283+0000", 2024-09-18T07:48:19.121 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4175000000000003e-05 2024-09-18T07:48:19.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.122 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.122 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.122 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.972317+0000", 2024-09-18T07:48:19.122 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4272e-05 2024-09-18T07:48:19.122 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.122 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.122 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.122 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.122 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:75500 11.4 11:23b40a34:::benchmark_data_smithi137_181069_object75499:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-09-18T07:48:19.122 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:48:07.809430+0000", 2024-09-18T07:48:19.122 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 10.627132316000001, 2024-09-18T07:48:19.123 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.162399615, 2024-09-18T07:48:19.123 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.123 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.123 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:19.123 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.123 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:19.123 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:19.123 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 75500 2024-09-18T07:48:19.123 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.123 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.123 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.123 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.123 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.809430+0000", 2024-09-18T07:48:19.123 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.124 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.124 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.809430+0000", 2024-09-18T07:48:19.124 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.124 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.124 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.809432+0000", 2024-09-18T07:48:19.124 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0159999999999998e-06 2024-09-18T07:48:19.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.124 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.124 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.124 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.809482+0000", 2024-09-18T07:48:19.124 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0472000000000001e-05 2024-09-18T07:48:19.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.125 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.125 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.125 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.809496+0000", 2024-09-18T07:48:19.125 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3199e-05 2024-09-18T07:48:19.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.125 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.125 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.125 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.809505+0000", 2024-09-18T07:48:19.125 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2040000000000003e-06 2024-09-18T07:48:19.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.125 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.125 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.125 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.809556+0000", 2024-09-18T07:48:19.126 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1328000000000001e-05 2024-09-18T07:48:19.126 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.126 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.126 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.126 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.809697+0000", 2024-09-18T07:48:19.126 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014057499999999999 2024-09-18T07:48:19.126 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.126 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.126 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,6", 2024-09-18T07:48:19.126 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.809851+0000", 2024-09-18T07:48:19.126 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015445199999999999 2024-09-18T07:48:19.126 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.127 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.127 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.127 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.812101+0000", 2024-09-18T07:48:19.127 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002250087 2024-09-18T07:48:19.127 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.127 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.127 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.127 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.813608+0000", 2024-09-18T07:48:19.127 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001506734 2024-09-18T07:48:19.127 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.127 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.127 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.127 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.971737+0000", 2024-09-18T07:48:19.127 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15812885400000001 2024-09-18T07:48:19.128 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.128 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.128 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:19.128 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.971791+0000", 2024-09-18T07:48:19.128 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4564999999999997e-05 2024-09-18T07:48:19.128 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.128 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.128 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.128 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.971830+0000", 2024-09-18T07:48:19.128 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8129e-05 2024-09-18T07:48:19.128 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.128 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.128 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.128 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.128 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.129 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:75506 11.4 11:2371ad2d:::benchmark_data_smithi137_181069_object75505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-09-18T07:48:19.129 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:48:07.814787+0000", 2024-09-18T07:48:19.129 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 10.621774800000001, 2024-09-18T07:48:19.129 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.15714710800000001, 2024-09-18T07:48:19.129 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:19.129 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:19.129 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:19.129 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:19.129 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:19.129 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:19.129 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 75506 2024-09-18T07:48:19.129 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.129 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:19.129 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.130 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:19.130 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.814787+0000", 2024-09-18T07:48:19.130 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.130 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:19.130 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.814787+0000", 2024-09-18T07:48:19.130 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:19.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.130 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:19.130 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.814789+0000", 2024-09-18T07:48:19.130 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.234e-06 2024-09-18T07:48:19.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.130 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.130 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:19.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.814852+0000", 2024-09-18T07:48:19.131 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3448999999999995e-05 2024-09-18T07:48:19.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.131 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.131 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:19.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.814861+0000", 2024-09-18T07:48:19.131 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1889999999999996e-06 2024-09-18T07:48:19.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.131 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.131 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:19.131 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.814867+0000", 2024-09-18T07:48:19.131 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6969999999999998e-06 2024-09-18T07:48:19.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.131 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.132 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:19.132 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.814908+0000", 2024-09-18T07:48:19.132 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0602000000000002e-05 2024-09-18T07:48:19.132 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.132 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.132 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:19.132 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.815010+0000", 2024-09-18T07:48:19.132 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010191699999999999 2024-09-18T07:48:19.132 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.132 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.132 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 2,6", 2024-09-18T07:48:19.132 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.815101+0000", 2024-09-18T07:48:19.132 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1629e-05 2024-09-18T07:48:19.133 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.133 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.133 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:19.133 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.817226+0000", 2024-09-18T07:48:19.133 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021244200000000001 2024-09-18T07:48:19.133 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.133 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.133 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.133 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.818122+0000", 2024-09-18T07:48:19.133 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00089647900000000005 2024-09-18T07:48:19.133 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.133 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.133 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:19.133 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.971867+0000", 2024-09-18T07:48:19.134 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.153744562 2024-09-18T07:48:19.134 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.134 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.134 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:19.134 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.971906+0000", 2024-09-18T07:48:19.134 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8893999999999997e-05 2024-09-18T07:48:19.134 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:19.134 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:19.134 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:19.134 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:07.971935+0000", 2024-09-18T07:48:19.134 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9036000000000001e-05 2024-09-18T07:48:19.134 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.134 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.134 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.134 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:19.135 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:19.135 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:48:19.135 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:19.136 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 64 16 88072 88056 85.9719 65.375 0.0223288 0.0116267 2024-09-18T07:48:19.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:19.169 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:19.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:19.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:19.308 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:19.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:19.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:19.410 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:19.444 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:19.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:19.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:19.546 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:19.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:19.577 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:19.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:19.623 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:19 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:19.623 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:19 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2002867408' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:48:19.623 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:19 smithi137 ceph-mon[29408]: osdmap e493: 8 total, 8 up, 8 in 2024-09-18T07:48:19.623 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:19 smithi137 ceph-mon[29408]: from='client.26635 v1:172.21.15.137:0/2907441599' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:48:19.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:19 smithi137 ceph-mon[29408]: 3.5 scrub starts 2024-09-18T07:48:19.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:19 smithi137 ceph-mon[29408]: 3.5 scrub ok 2024-09-18T07:48:19.624 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:19 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:19.624 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:19 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2002867408' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:48:19.625 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:19 smithi137 ceph-mon[39175]: osdmap e493: 8 total, 8 up, 8 in 2024-09-18T07:48:19.625 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:19 smithi137 ceph-mon[39175]: from='client.26635 v1:172.21.15.137:0/2907441599' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:48:19.625 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:19 smithi137 ceph-mon[39175]: 3.5 scrub starts 2024-09-18T07:48:19.625 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:19 smithi137 ceph-mon[39175]: 3.5 scrub ok 2024-09-18T07:48:19.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:19.711 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:19.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:19.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:19.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:19.846 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:19.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:19.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:19.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:19.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:19 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:19.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:19 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2002867408' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:48:19.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:19 smithi203 ceph-mon[32237]: osdmap e493: 8 total, 8 up, 8 in 2024-09-18T07:48:19.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:19 smithi203 ceph-mon[32237]: from='client.26635 v1:172.21.15.137:0/2907441599' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:48:19.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:19 smithi203 ceph-mon[32237]: 3.5 scrub starts 2024-09-18T07:48:19.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:19 smithi203 ceph-mon[32237]: 3.5 scrub ok 2024-09-18T07:48:19.988 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:19.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:19.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:20.058 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 65 16 89230 89214 85.7627 72.375 0.0142471 0.0116549 2024-09-18T07:48:20.090 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:20.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:20.130 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:20.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:20.178 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:48:20.178 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:48:20.210 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:48:20.210 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:48:20.232 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:20.244 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:48:20.244 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:48:20.259 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:20.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:20.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:20.280 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:48:20.280 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:48:20.296 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:48:20.296 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:48:20.312 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:48:20.312 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:48:20.356 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:48:20.356 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:48:20.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:20.386 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:48:20.387 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:48:20.396 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:20.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:20.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:20.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:20.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:20.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:20.641 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:20.666 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:20.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:20.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:20.768 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:20.802 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:20.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:20.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:20.811 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-18T07:48:20.811 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 110 0.05 2024-09-18T07:48:20.904 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:20.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:20.930 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:20.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:21.031 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:21.058 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 66 16 90209 90193 85.3903 61.1875 0.015103 0.0117054 2024-09-18T07:48:21.059 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:21.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:21.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:21.140 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[39175]: pgmap v930: 158 pgs: 158 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.01k op/s; 54 B/s, 1 objects/s recovering 2024-09-18T07:48:21.140 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[39175]: 4.1e scrub starts 2024-09-18T07:48:21.140 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[39175]: 4.1e scrub ok 2024-09-18T07:48:21.140 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2981651966' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:21.140 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[39175]: 11.a scrub starts 2024-09-18T07:48:21.140 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[39175]: 5.b scrub starts 2024-09-18T07:48:21.140 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[39175]: 5.b scrub ok 2024-09-18T07:48:21.140 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[39175]: 5.0 scrub starts 2024-09-18T07:48:21.141 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[39175]: 5.0 scrub ok 2024-09-18T07:48:21.161 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:21.213 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:21.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:21.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:21.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:21.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:21.350 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:21.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:21.410 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:48:21.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[29408]: pgmap v930: 158 pgs: 158 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.01k op/s; 54 B/s, 1 objects/s recovering 2024-09-18T07:48:21.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[29408]: 4.1e scrub starts 2024-09-18T07:48:21.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[29408]: 4.1e scrub ok 2024-09-18T07:48:21.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2981651966' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:21.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[29408]: 11.a scrub starts 2024-09-18T07:48:21.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[29408]: 5.b scrub starts 2024-09-18T07:48:21.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[29408]: 5.b scrub ok 2024-09-18T07:48:21.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[29408]: 5.0 scrub starts 2024-09-18T07:48:21.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:21 smithi137 ceph-mon[29408]: 5.0 scrub ok 2024-09-18T07:48:21.422 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:21 smithi203 ceph-mon[32237]: pgmap v930: 158 pgs: 158 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.01k op/s; 54 B/s, 1 objects/s recovering 2024-09-18T07:48:21.422 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:21 smithi203 ceph-mon[32237]: 4.1e scrub starts 2024-09-18T07:48:21.423 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:21 smithi203 ceph-mon[32237]: 4.1e scrub ok 2024-09-18T07:48:21.423 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:21 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2981651966' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:21.423 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:21 smithi203 ceph-mon[32237]: 11.a scrub starts 2024-09-18T07:48:21.423 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:21 smithi203 ceph-mon[32237]: 5.b scrub starts 2024-09-18T07:48:21.423 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:21 smithi203 ceph-mon[32237]: 5.b scrub ok 2024-09-18T07:48:21.423 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:21 smithi203 ceph-mon[32237]: 5.0 scrub starts 2024-09-18T07:48:21.423 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:21 smithi203 ceph-mon[32237]: 5.0 scrub ok 2024-09-18T07:48:21.451 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:21.486 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:21.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:21.588 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:21.615 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:21.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:21.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:21.717 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:21.753 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:21.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:21.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:21.812 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:21.854 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:21.863 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:21.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:21.892 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:21.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:21.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:22.033 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:22.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:22.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:22.058 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 67 16 91280 91264 85.1145 66.9375 0.0140242 0.0117442 2024-09-18T07:48:22.135 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:22.144 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T07:48:22.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:22.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:22.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:22.291 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:22 smithi137 ceph-mon[29408]: 4.d scrub starts 2024-09-18T07:48:22.292 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:22 smithi137 ceph-mon[29408]: 4.d scrub ok 2024-09-18T07:48:22.292 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:22 smithi137 ceph-mon[39175]: 4.d scrub starts 2024-09-18T07:48:22.292 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:22 smithi137 ceph-mon[39175]: 4.d scrub ok 2024-09-18T07:48:22.294 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:22.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:22.334 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:22.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:22.435 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:22.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:22 smithi203 ceph-mon[32237]: 4.d scrub starts 2024-09-18T07:48:22.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:22 smithi203 ceph-mon[32237]: 4.d scrub ok 2024-09-18T07:48:22.480 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:22.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:22.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:22.529 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:48:22.582 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:22.609 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:22.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:22.711 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:22.743 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:22.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:22.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:22.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:22.871 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:22.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:22.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:22.973 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:23.014 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:23.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:23.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:23.058 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 68 16 92359 92343 84.8545 67.4375 0.00948362 0.0117779 2024-09-18T07:48:23.064 INFO:teuthology.orchestra.run.smithi137.stdout:moved 15 / 474 (3.16456%) 2024-09-18T07:48:23.065 INFO:teuthology.orchestra.run.smithi137.stdout:avg 59.25 2024-09-18T07:48:23.065 INFO:teuthology.orchestra.run.smithi137.stdout:stddev 4.84123 -> 6.95971 (expected baseline 7.20026) 2024-09-18T07:48:23.065 INFO:teuthology.orchestra.run.smithi137.stdout:min osd.6 with 53 -> 47 pgs (0.894515 -> 0.793249 * mean) 2024-09-18T07:48:23.065 INFO:teuthology.orchestra.run.smithi137.stdout:max osd.7 with 67 -> 68 pgs (1.1308 -> 1.14768 * mean) 2024-09-18T07:48:23.065 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:48:23.065 INFO:teuthology.orchestra.run.smithi137.stdout:oload 110 2024-09-18T07:48:23.065 INFO:teuthology.orchestra.run.smithi137.stdout:max_change 0.05 2024-09-18T07:48:23.065 INFO:teuthology.orchestra.run.smithi137.stdout:max_change_osds 4 2024-09-18T07:48:23.065 INFO:teuthology.orchestra.run.smithi137.stdout:average_utilization 0.0300 2024-09-18T07:48:23.065 INFO:teuthology.orchestra.run.smithi137.stdout:overload_utilization 0.0330 2024-09-18T07:48:23.065 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 weight 0.8067 -> 0.7567 2024-09-18T07:48:23.065 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 weight 0.8307 -> 0.7807 2024-09-18T07:48:23.066 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 weight 0.8851 -> 0.8351 2024-09-18T07:48:23.066 INFO:teuthology.orchestra.run.smithi137.stderr:no change 2024-09-18T07:48:23.116 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:23.128 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:48:23.128 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:48:23.144 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:23.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:23.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:23.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:48:23 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: 2024-09-18T07:48:23.061+0000 7f460a846640 -1 mgr.server reply reply (3) No such process no change 2024-09-18T07:48:23.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:23 smithi137 ceph-mon[29408]: pgmap v931: 158 pgs: 158 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-18T07:48:23.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:23 smithi137 ceph-mon[29408]: 5.4 scrub starts 2024-09-18T07:48:23.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:23 smithi137 ceph-mon[29408]: 5.4 scrub ok 2024-09-18T07:48:23.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:23 smithi137 ceph-mon[29408]: 11.a scrub ok 2024-09-18T07:48:23.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:23 smithi137 ceph-mon[29408]: 4.b scrub starts 2024-09-18T07:48:23.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:23 smithi137 ceph-mon[29408]: 4.b scrub ok 2024-09-18T07:48:23.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:23 smithi137 ceph-mon[29408]: 5.a scrub starts 2024-09-18T07:48:23.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:23 smithi137 ceph-mon[29408]: 5.a scrub ok 2024-09-18T07:48:23.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:23 smithi137 ceph-mon[39175]: pgmap v931: 158 pgs: 158 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-18T07:48:23.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:23 smithi137 ceph-mon[39175]: 5.4 scrub starts 2024-09-18T07:48:23.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:23 smithi137 ceph-mon[39175]: 5.4 scrub ok 2024-09-18T07:48:23.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:23 smithi137 ceph-mon[39175]: 11.a scrub ok 2024-09-18T07:48:23.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:23 smithi137 ceph-mon[39175]: 4.b scrub starts 2024-09-18T07:48:23.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:23 smithi137 ceph-mon[39175]: 4.b scrub ok 2024-09-18T07:48:23.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:23 smithi137 ceph-mon[39175]: 5.a scrub starts 2024-09-18T07:48:23.171 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:23 smithi137 ceph-mon[39175]: 5.a scrub ok 2024-09-18T07:48:23.246 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:23.273 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:23.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:23.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:23.375 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:23.402 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:23.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:23.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:23 smithi203 ceph-mon[32237]: pgmap v931: 158 pgs: 158 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-18T07:48:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:23 smithi203 ceph-mon[32237]: 5.4 scrub starts 2024-09-18T07:48:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:23 smithi203 ceph-mon[32237]: 5.4 scrub ok 2024-09-18T07:48:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:23 smithi203 ceph-mon[32237]: 11.a scrub ok 2024-09-18T07:48:23.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:23 smithi203 ceph-mon[32237]: 4.b scrub starts 2024-09-18T07:48:23.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:23 smithi203 ceph-mon[32237]: 4.b scrub ok 2024-09-18T07:48:23.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:23 smithi203 ceph-mon[32237]: 5.a scrub starts 2024-09-18T07:48:23.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:23 smithi203 ceph-mon[32237]: 5.a scrub ok 2024-09-18T07:48:23.504 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:23.535 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:23.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:23.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:23.625 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:48:23.636 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:23.666 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:23.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:23.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:23.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:23.788 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:23.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:23.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:23.890 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:23.917 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:23.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:23.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:24.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:24.057 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:24.060 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 69 16 93333 93317 84.505 60.875 0.00696837 0.0118285 2024-09-18T07:48:24.068 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:24.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:24.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:24.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:24.171 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:48:24.173 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:24 smithi137 ceph-mon[29408]: from='client.36545 v1:172.21.15.137:0/2600571756' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:48:24.174 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:24 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/825544112' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:48:24.174 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:24 smithi137 ceph-mon[29408]: from='client.36551 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:48:24.174 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:24 smithi137 ceph-mon[29408]: 11.8 scrub starts 2024-09-18T07:48:24.174 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:24 smithi137 ceph-mon[29408]: 5.8 scrub starts 2024-09-18T07:48:24.174 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:24 smithi137 ceph-mon[29408]: 5.8 scrub ok 2024-09-18T07:48:24.174 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:24 smithi137 ceph-mon[39175]: from='client.36545 v1:172.21.15.137:0/2600571756' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:48:24.174 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:24 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/825544112' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:48:24.174 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:24 smithi137 ceph-mon[39175]: from='client.36551 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:48:24.174 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:24 smithi137 ceph-mon[39175]: 11.8 scrub starts 2024-09-18T07:48:24.174 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:24 smithi137 ceph-mon[39175]: 5.8 scrub starts 2024-09-18T07:48:24.174 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:24 smithi137 ceph-mon[39175]: 5.8 scrub ok 2024-09-18T07:48:24.206 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:24.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:24.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:24.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:24.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:24.339 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:24.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:24.365 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T07:48:24.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:24.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:24 smithi203 ceph-mon[32237]: from='client.36545 v1:172.21.15.137:0/2600571756' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-18T07:48:24.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:24 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/825544112' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:48:24.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:24 smithi203 ceph-mon[32237]: from='client.36551 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:48:24.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:24 smithi203 ceph-mon[32237]: 11.8 scrub starts 2024-09-18T07:48:24.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:24 smithi203 ceph-mon[32237]: 5.8 scrub starts 2024-09-18T07:48:24.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:24 smithi203 ceph-mon[32237]: 5.8 scrub ok 2024-09-18T07:48:24.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:24.476 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:24.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:24.577 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:24.622 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:24.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:24.724 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:24.765 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:24.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:24.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:24.766 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:48:24.867 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:24.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:24.899 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:24.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:25.000 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:25.042 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:25.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:25.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:25.061 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 70 16 94330 94314 84.1865 62.3125 0.0149164 0.0118734 2024-09-18T07:48:25.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:25.180 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:25.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:25.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:25.282 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:25.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:25.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:25.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:25.372 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:48:25.372 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:48:25.415 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:25 smithi137 ceph-mon[29408]: pgmap v932: 158 pgs: 158 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-18T07:48:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:25 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:48:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:25 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:48:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:25 smithi137 ceph-mon[29408]: from='client.36551 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:48:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:25 smithi137 ceph-mon[29408]: osdmap e494: 8 total, 8 up, 8 in 2024-09-18T07:48:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:25 smithi137 ceph-mon[29408]: 5.3 scrub starts 2024-09-18T07:48:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:25 smithi137 ceph-mon[29408]: 5.3 scrub ok 2024-09-18T07:48:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:25 smithi137 ceph-mon[29408]: osdmap e495: 8 total, 8 up, 8 in 2024-09-18T07:48:25.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:25 smithi137 ceph-mon[39175]: pgmap v932: 158 pgs: 158 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-18T07:48:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:25 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:48:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:25 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:48:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:25 smithi137 ceph-mon[39175]: from='client.36551 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:48:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:25 smithi137 ceph-mon[39175]: osdmap e494: 8 total, 8 up, 8 in 2024-09-18T07:48:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:25 smithi137 ceph-mon[39175]: 5.3 scrub starts 2024-09-18T07:48:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:25 smithi137 ceph-mon[39175]: 5.3 scrub ok 2024-09-18T07:48:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:25 smithi137 ceph-mon[39175]: osdmap e495: 8 total, 8 up, 8 in 2024-09-18T07:48:25.451 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:25.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:25.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:25.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:25 smithi203 ceph-mon[32237]: pgmap v932: 158 pgs: 158 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-18T07:48:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:25 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:48:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:25 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:48:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:25 smithi203 ceph-mon[32237]: from='client.36551 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:48:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:25 smithi203 ceph-mon[32237]: osdmap e494: 8 total, 8 up, 8 in 2024-09-18T07:48:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:25 smithi203 ceph-mon[32237]: 5.3 scrub starts 2024-09-18T07:48:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:25 smithi203 ceph-mon[32237]: 5.3 scrub ok 2024-09-18T07:48:25.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:25 smithi203 ceph-mon[32237]: osdmap e495: 8 total, 8 up, 8 in 2024-09-18T07:48:25.553 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:25.585 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:25.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:25.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:25.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:25.724 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:25.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:25.826 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:25.858 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:25.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:25.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:25.960 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:25.997 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:25.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:25.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:26.061 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 71 16 95457 95441 83.9928 70.4375 0.00782053 0.0119011 2024-09-18T07:48:26.103 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:26.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:26.137 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:26.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:26.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:26 smithi203 ceph-mon[32237]: 11.8 scrub ok 2024-09-18T07:48:26.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:26 smithi203 ceph-mon[32237]: 11.3 scrub starts 2024-09-18T07:48:26.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:26.271 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:26.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:26.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:26.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:26.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:26.395 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:26.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:26.496 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:26.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:26.533 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:26.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:26.605 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T07:48:26.634 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:26.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:26 smithi137 ceph-mon[29408]: 11.8 scrub ok 2024-09-18T07:48:26.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:26 smithi137 ceph-mon[29408]: 11.3 scrub starts 2024-09-18T07:48:26.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:26 smithi137 ceph-mon[39175]: 11.8 scrub ok 2024-09-18T07:48:26.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:26 smithi137 ceph-mon[39175]: 11.3 scrub starts 2024-09-18T07:48:26.683 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:26.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:26.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:26.785 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:26.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:26.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:26.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:26.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:26.970 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:26.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:26.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:27.026 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:48:27.062 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 72 16 96559 96543 83.7819 68.875 0.0169805 0.0119298 2024-09-18T07:48:27.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:27.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:27.142 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:27.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:27.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:48:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:48:27.243 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:27.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:27.294 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:27.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:27.395 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:27.438 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:27.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:27.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:27 smithi203 ceph-mon[32237]: 6.11 scrub starts 2024-09-18T07:48:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:27 smithi203 ceph-mon[32237]: 6.11 scrub ok 2024-09-18T07:48:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:27 smithi203 ceph-mon[32237]: 4.15 scrub starts 2024-09-18T07:48:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:27 smithi203 ceph-mon[32237]: 4.15 scrub ok 2024-09-18T07:48:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:27 smithi203 ceph-mon[32237]: pgmap v935: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 699 B/s rd, 49 MiB/s wr, 782 op/s 2024-09-18T07:48:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:27 smithi203 ceph-mon[32237]: 6.c scrub starts 2024-09-18T07:48:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:27 smithi203 ceph-mon[32237]: 6.c scrub ok 2024-09-18T07:48:27.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:27 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:48:27.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:27 smithi203 ceph-mon[32237]: 4.1 scrub starts 2024-09-18T07:48:27.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:27 smithi203 ceph-mon[32237]: 4.1 scrub ok 2024-09-18T07:48:27.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:27 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3649735131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:27.540 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:27.576 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:27.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:27.648 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:48:27.648 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:48:27.648 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:48:27.648 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:48:27.648 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.648 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189537 10.2 10:41c597a1:::benchmark_data_smithi137_151160_object53721:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:48:27.648 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.301971+0000", 2024-09-18T07:48:27.648 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.34166674299999, 2024-09-18T07:48:27.649 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0664383530000001, 2024-09-18T07:48:27.649 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.649 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.649 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:48:27.649 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.649 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.649 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.649 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189537 2024-09-18T07:48:27.649 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.649 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.649 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.649 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.649 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.301971+0000", 2024-09-18T07:48:27.650 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.650 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.650 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.650 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.650 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.301971+0000", 2024-09-18T07:48:27.650 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.650 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.650 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.650 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.650 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.301973+0000", 2024-09-18T07:48:27.650 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.99e-06 2024-09-18T07:48:27.650 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.650 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.651 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.651 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.301979+0000", 2024-09-18T07:48:27.651 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4389999999999999e-06 2024-09-18T07:48:27.651 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.651 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.651 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.651 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.301995+0000", 2024-09-18T07:48:27.651 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6220000000000001e-05 2024-09-18T07:48:27.651 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.651 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.651 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.651 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302003+0000", 2024-09-18T07:48:27.651 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0020000000000006e-06 2024-09-18T07:48:27.651 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.652 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.652 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.652 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302091+0000", 2024-09-18T07:48:27.652 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7639000000000006e-05 2024-09-18T07:48:27.652 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.652 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.652 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.652 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302098+0000", 2024-09-18T07:48:27.652 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5989999999999996e-06 2024-09-18T07:48:27.652 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.652 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.652 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.652 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.368329+0000", 2024-09-18T07:48:27.653 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0662302459999999 2024-09-18T07:48:27.653 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.653 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.653 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.653 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.368410+0000", 2024-09-18T07:48:27.653 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1217999999999994e-05 2024-09-18T07:48:27.653 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.653 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.653 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.653 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.653 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.653 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189549 10.2 10:4a928452:::benchmark_data_smithi137_151160_object53733:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:48:27.653 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.302028+0000", 2024-09-18T07:48:27.654 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.34161000399999, 2024-09-18T07:48:27.654 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.100603623, 2024-09-18T07:48:27.654 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.654 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.654 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:48:27.654 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.654 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.654 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.654 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189549 2024-09-18T07:48:27.654 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.654 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.654 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.654 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.655 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302028+0000", 2024-09-18T07:48:27.655 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.655 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.655 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.655 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.655 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302028+0000", 2024-09-18T07:48:27.655 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.655 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.655 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.655 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.655 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302029+0000", 2024-09-18T07:48:27.655 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3399999999999998e-07 2024-09-18T07:48:27.655 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.656 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.656 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.656 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302033+0000", 2024-09-18T07:48:27.656 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4710000000000001e-06 2024-09-18T07:48:27.656 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.656 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.656 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.656 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302041+0000", 2024-09-18T07:48:27.656 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9279999999999993e-06 2024-09-18T07:48:27.656 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.656 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.656 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.656 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302047+0000", 2024-09-18T07:48:27.657 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0789999999999997e-06 2024-09-18T07:48:27.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.657 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.657 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.657 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302133+0000", 2024-09-18T07:48:27.657 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5400000000000002e-05 2024-09-18T07:48:27.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.657 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.657 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.657 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302138+0000", 2024-09-18T07:48:27.657 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5419999999999996e-06 2024-09-18T07:48:27.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.658 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.658 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.658 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.368614+0000", 2024-09-18T07:48:27.658 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0664751509999999 2024-09-18T07:48:27.658 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.658 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.658 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for missing object", 2024-09-18T07:48:27.658 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.368706+0000", 2024-09-18T07:48:27.658 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2783999999999998e-05 2024-09-18T07:48:27.658 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.658 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.658 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.658 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.402529+0000", 2024-09-18T07:48:27.658 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.033822673999999997 2024-09-18T07:48:27.658 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.659 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.659 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.659 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.402632+0000", 2024-09-18T07:48:27.659 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010276 2024-09-18T07:48:27.659 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.659 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.659 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.659 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.659 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.659 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189555 10.2 10:4c34833c:::benchmark_data_smithi137_151160_object53739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:48:27.659 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.305399+0000", 2024-09-18T07:48:27.659 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.33823960500001, 2024-09-18T07:48:27.659 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.075830737, 2024-09-18T07:48:27.660 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.660 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.660 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:27.660 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.660 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.660 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.660 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189555 2024-09-18T07:48:27.660 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.660 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.660 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.660 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.660 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.305399+0000", 2024-09-18T07:48:27.660 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.661 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.661 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.661 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.661 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.305399+0000", 2024-09-18T07:48:27.661 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.661 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.661 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.661 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.661 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.305400+0000", 2024-09-18T07:48:27.661 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4890000000000001e-06 2024-09-18T07:48:27.661 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.661 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.661 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.661 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.305405+0000", 2024-09-18T07:48:27.662 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0760000000000002e-06 2024-09-18T07:48:27.662 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.662 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.662 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.662 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.305415+0000", 2024-09-18T07:48:27.662 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0077e-05 2024-09-18T07:48:27.662 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.662 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.662 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.662 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.305420+0000", 2024-09-18T07:48:27.662 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0819999999999998e-06 2024-09-18T07:48:27.662 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.662 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.662 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.662 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.305472+0000", 2024-09-18T07:48:27.663 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2182999999999998e-05 2024-09-18T07:48:27.663 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.663 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.663 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.663 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.305482+0000", 2024-09-18T07:48:27.663 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0860000000000008e-06 2024-09-18T07:48:27.663 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.663 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.663 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.663 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.368768+0000", 2024-09-18T07:48:27.663 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0632862219999999 2024-09-18T07:48:27.663 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.663 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.663 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:27.664 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.368997+0000", 2024-09-18T07:48:27.664 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000229535 2024-09-18T07:48:27.664 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.664 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.664 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:48:27.664 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.369369+0000", 2024-09-18T07:48:27.664 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00037124899999999998 2024-09-18T07:48:27.664 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.664 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.664 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.664 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.379199+0000", 2024-09-18T07:48:27.664 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0098309239999999996 2024-09-18T07:48:27.664 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.664 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.665 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.665 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.379274+0000", 2024-09-18T07:48:27.665 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4609000000000003e-05 2024-09-18T07:48:27.665 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.665 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.665 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:27.665 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.381069+0000", 2024-09-18T07:48:27.665 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0017952619999999999 2024-09-18T07:48:27.665 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.665 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.665 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:27.665 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.381124+0000", 2024-09-18T07:48:27.665 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5084999999999999e-05 2024-09-18T07:48:27.666 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.666 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.666 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.666 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.381229+0000", 2024-09-18T07:48:27.666 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000104858 2024-09-18T07:48:27.666 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.666 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.666 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.666 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.666 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.666 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189558 10.2 10:4577464b:::benchmark_data_smithi137_151160_object53742:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:48:27.666 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.307579+0000", 2024-09-18T07:48:27.666 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.336059616, 2024-09-18T07:48:27.666 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0761420020000001, 2024-09-18T07:48:27.667 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.667 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.667 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:27.667 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.667 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.667 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.667 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189558 2024-09-18T07:48:27.667 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.667 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.667 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.667 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.667 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.307579+0000", 2024-09-18T07:48:27.667 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.668 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.668 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.668 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.668 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.307579+0000", 2024-09-18T07:48:27.668 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.668 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.668 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.668 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.668 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.307580+0000", 2024-09-18T07:48:27.668 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7400000000000001e-06 2024-09-18T07:48:27.668 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.668 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.669 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.669 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.307585+0000", 2024-09-18T07:48:27.669 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5950000000000001e-06 2024-09-18T07:48:27.669 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.669 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.669 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.669 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.307601+0000", 2024-09-18T07:48:27.669 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5971999999999999e-05 2024-09-18T07:48:27.669 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.669 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.669 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.669 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.307608+0000", 2024-09-18T07:48:27.669 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2470000000000002e-06 2024-09-18T07:48:27.670 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.670 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.670 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.670 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.307767+0000", 2024-09-18T07:48:27.670 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015924099999999999 2024-09-18T07:48:27.670 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.670 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.670 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.670 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.307775+0000", 2024-09-18T07:48:27.670 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7090000000000002e-06 2024-09-18T07:48:27.670 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.670 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.670 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.670 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.376610+0000", 2024-09-18T07:48:27.671 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0688345130000001 2024-09-18T07:48:27.671 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.671 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.671 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:27.671 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.376818+0000", 2024-09-18T07:48:27.671 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00020837199999999999 2024-09-18T07:48:27.671 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.671 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.671 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:48:27.671 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.376968+0000", 2024-09-18T07:48:27.671 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000149741 2024-09-18T07:48:27.671 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.671 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.672 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:27.672 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.383501+0000", 2024-09-18T07:48:27.672 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0065335790000000003 2024-09-18T07:48:27.672 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.672 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.672 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.672 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.383544+0000", 2024-09-18T07:48:27.672 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2759999999999997e-05 2024-09-18T07:48:27.672 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.672 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.672 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.672 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.383587+0000", 2024-09-18T07:48:27.672 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2932000000000002e-05 2024-09-18T07:48:27.672 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.673 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.673 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:27.673 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.383632+0000", 2024-09-18T07:48:27.673 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4734000000000002e-05 2024-09-18T07:48:27.673 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.673 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.673 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.673 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.383721+0000", 2024-09-18T07:48:27.673 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8867000000000001e-05 2024-09-18T07:48:27.673 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.673 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.673 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.674 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.674 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.674 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189576 10.2 10:403b32d3:::benchmark_data_smithi137_151160_object53760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:48:27.674 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.323665+0000", 2024-09-18T07:48:27.674 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.319972979, 2024-09-18T07:48:27.674 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0652314759999999, 2024-09-18T07:48:27.674 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.674 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.674 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:27.674 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.674 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.674 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.674 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189576 2024-09-18T07:48:27.674 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.675 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.675 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.675 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.675 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.323665+0000", 2024-09-18T07:48:27.675 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.675 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.675 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.675 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.675 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.323665+0000", 2024-09-18T07:48:27.675 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.675 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.675 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.675 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.676 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.323667+0000", 2024-09-18T07:48:27.676 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1610000000000001e-06 2024-09-18T07:48:27.676 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.676 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.676 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.676 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.323673+0000", 2024-09-18T07:48:27.676 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0469999999999999e-06 2024-09-18T07:48:27.676 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.676 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.676 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.676 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.323684+0000", 2024-09-18T07:48:27.676 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0168e-05 2024-09-18T07:48:27.677 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.677 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.677 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[29408]: 6.11 scrub starts 2024-09-18T07:48:27.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[29408]: 6.11 scrub ok 2024-09-18T07:48:27.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[29408]: 4.15 scrub starts 2024-09-18T07:48:27.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[29408]: 4.15 scrub ok 2024-09-18T07:48:27.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[29408]: pgmap v935: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 699 B/s rd, 49 MiB/s wr, 782 op/s 2024-09-18T07:48:27.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[29408]: 6.c scrub starts 2024-09-18T07:48:27.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[29408]: 6.c scrub ok 2024-09-18T07:48:27.679 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:48:27.679 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[29408]: 4.1 scrub starts 2024-09-18T07:48:27.679 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[29408]: 4.1 scrub ok 2024-09-18T07:48:27.680 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3649735131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:27.680 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[39175]: 6.11 scrub starts 2024-09-18T07:48:27.680 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[39175]: 6.11 scrub ok 2024-09-18T07:48:27.680 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[39175]: 4.15 scrub starts 2024-09-18T07:48:27.680 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[39175]: 4.15 scrub ok 2024-09-18T07:48:27.680 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[39175]: pgmap v935: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 699 B/s rd, 49 MiB/s wr, 782 op/s 2024-09-18T07:48:27.680 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[39175]: 6.c scrub starts 2024-09-18T07:48:27.681 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[39175]: 6.c scrub ok 2024-09-18T07:48:27.681 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:48:27.681 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[39175]: 4.1 scrub starts 2024-09-18T07:48:27.681 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[39175]: 4.1 scrub ok 2024-09-18T07:48:27.681 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:27 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3649735131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:27.681 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.681 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.323691+0000", 2024-09-18T07:48:27.681 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1949999999999997e-06 2024-09-18T07:48:27.681 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.681 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.681 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.682 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.331894+0000", 2024-09-18T07:48:27.682 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0082035219999999996 2024-09-18T07:48:27.682 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.682 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.682 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.682 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.331902+0000", 2024-09-18T07:48:27.682 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0260000000000007e-06 2024-09-18T07:48:27.682 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.682 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.682 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.682 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.381275+0000", 2024-09-18T07:48:27.682 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0493726240000001 2024-09-18T07:48:27.682 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.682 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.683 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:27.683 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.381376+0000", 2024-09-18T07:48:27.683 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010126799999999999 2024-09-18T07:48:27.683 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.683 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.683 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:48:27.683 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.381451+0000", 2024-09-18T07:48:27.683 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5108e-05 2024-09-18T07:48:27.683 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.683 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.683 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:27.683 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.388536+0000", 2024-09-18T07:48:27.683 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0070845819999999999 2024-09-18T07:48:27.683 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.684 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.684 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.684 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.388688+0000", 2024-09-18T07:48:27.684 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015197100000000001 2024-09-18T07:48:27.684 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.684 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.684 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.684 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.388748+0000", 2024-09-18T07:48:27.684 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0303999999999997e-05 2024-09-18T07:48:27.684 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.684 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.684 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:27.684 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.388804+0000", 2024-09-18T07:48:27.685 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6070999999999999e-05 2024-09-18T07:48:27.685 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.685 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.685 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.685 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.388897+0000", 2024-09-18T07:48:27.685 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2429000000000006e-05 2024-09-18T07:48:27.685 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.685 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.685 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.685 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.685 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.685 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189578 10.2 10:467f99cb:::benchmark_data_smithi137_151160_object53762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:48:27.686 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.324033+0000", 2024-09-18T07:48:27.686 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.31960558700001, 2024-09-18T07:48:27.686 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0651037249999999, 2024-09-18T07:48:27.686 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.686 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.686 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:27.686 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.686 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.686 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.686 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189578 2024-09-18T07:48:27.686 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.686 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.686 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.687 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.687 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.324033+0000", 2024-09-18T07:48:27.687 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.687 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.687 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.687 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.687 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.324033+0000", 2024-09-18T07:48:27.687 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.687 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.687 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.687 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.687 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.324034+0000", 2024-09-18T07:48:27.687 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.156e-06 2024-09-18T07:48:27.688 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.688 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.688 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.688 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.324038+0000", 2024-09-18T07:48:27.688 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1999999999999996e-06 2024-09-18T07:48:27.688 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.688 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.688 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.688 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.324047+0000", 2024-09-18T07:48:27.688 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8659999999999995e-06 2024-09-18T07:48:27.688 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.688 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.688 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.689 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.324054+0000", 2024-09-18T07:48:27.689 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8279999999999997e-06 2024-09-18T07:48:27.689 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.689 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.689 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.689 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.331930+0000", 2024-09-18T07:48:27.689 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0078765510000000007 2024-09-18T07:48:27.689 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.689 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.689 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.689 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.331936+0000", 2024-09-18T07:48:27.689 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4280000000000004e-06 2024-09-18T07:48:27.689 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.690 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.690 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.690 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.383774+0000", 2024-09-18T07:48:27.690 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0518385020000001 2024-09-18T07:48:27.690 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.690 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.690 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:27.690 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.383905+0000", 2024-09-18T07:48:27.690 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013089699999999999 2024-09-18T07:48:27.690 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.690 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.690 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:48:27.690 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.384063+0000", 2024-09-18T07:48:27.690 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015821800000000001 2024-09-18T07:48:27.691 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.691 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.691 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:27.691 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.388706+0000", 2024-09-18T07:48:27.691 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.004643309 2024-09-18T07:48:27.691 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.691 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.691 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.691 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.388959+0000", 2024-09-18T07:48:27.691 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025269099999999998 2024-09-18T07:48:27.691 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.691 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.691 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.692 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.389006+0000", 2024-09-18T07:48:27.692 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6391999999999999e-05 2024-09-18T07:48:27.692 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.692 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.692 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:27.692 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.389052+0000", 2024-09-18T07:48:27.692 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6768999999999999e-05 2024-09-18T07:48:27.692 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.692 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.692 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.692 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.389136+0000", 2024-09-18T07:48:27.692 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3918000000000005e-05 2024-09-18T07:48:27.692 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.692 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.693 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.693 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.693 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.693 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189583 10.2 10:4ef4bc3a:::benchmark_data_smithi137_151160_object53767:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:48:27.693 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.325300+0000", 2024-09-18T07:48:27.693 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.31833835899999, 2024-09-18T07:48:27.693 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.068459346, 2024-09-18T07:48:27.693 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.694 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.694 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:27.694 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.694 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.695 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.695 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189583 2024-09-18T07:48:27.695 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.695 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.695 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.695 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.695 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.325300+0000", 2024-09-18T07:48:27.695 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.695 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.695 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.695 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.695 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.325300+0000", 2024-09-18T07:48:27.695 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.696 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.696 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.696 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.696 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.325301+0000", 2024-09-18T07:48:27.696 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4389999999999999e-06 2024-09-18T07:48:27.696 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.696 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.696 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.696 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.325306+0000", 2024-09-18T07:48:27.696 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4530000000000004e-06 2024-09-18T07:48:27.696 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.696 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.696 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.697 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.325316+0000", 2024-09-18T07:48:27.697 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.008e-05 2024-09-18T07:48:27.697 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.697 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.697 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.697 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.325323+0000", 2024-09-18T07:48:27.697 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3900000000000004e-06 2024-09-18T07:48:27.697 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.697 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.697 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.697 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.331986+0000", 2024-09-18T07:48:27.698 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0066628069999999998 2024-09-18T07:48:27.698 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.698 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.698 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.698 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.332079+0000", 2024-09-18T07:48:27.698 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2844999999999996e-05 2024-09-18T07:48:27.698 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.698 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.698 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.698 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.386405+0000", 2024-09-18T07:48:27.698 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0543261900000001 2024-09-18T07:48:27.698 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.698 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.699 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:27.699 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.386505+0000", 2024-09-18T07:48:27.699 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000100076 2024-09-18T07:48:27.699 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.699 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.699 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:48:27.699 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.386616+0000", 2024-09-18T07:48:27.699 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011094099999999999 2024-09-18T07:48:27.699 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.699 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.699 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:27.699 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.391271+0000", 2024-09-18T07:48:27.699 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0046548819999999999 2024-09-18T07:48:27.700 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.700 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.700 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.700 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.393590+0000", 2024-09-18T07:48:27.700 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023194869999999999 2024-09-18T07:48:27.700 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.700 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.700 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.700 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.393656+0000", 2024-09-18T07:48:27.700 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5579000000000004e-05 2024-09-18T07:48:27.700 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.700 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.700 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:27.701 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.393691+0000", 2024-09-18T07:48:27.701 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5144000000000001e-05 2024-09-18T07:48:27.701 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.701 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.701 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.701 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.393759+0000", 2024-09-18T07:48:27.701 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8033000000000004e-05 2024-09-18T07:48:27.701 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.701 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.701 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.701 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.701 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.701 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189620 10.2 10:45188eeb:::benchmark_data_smithi137_151160_object53804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:48:27.702 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.359144+0000", 2024-09-18T07:48:27.702 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.284493892, 2024-09-18T07:48:27.702 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.043621656, 2024-09-18T07:48:27.702 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.702 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.702 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:27.702 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.702 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.702 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.702 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189620 2024-09-18T07:48:27.702 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.702 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.702 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.703 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.703 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.359144+0000", 2024-09-18T07:48:27.703 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.703 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.703 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.703 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.703 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.359144+0000", 2024-09-18T07:48:27.703 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.703 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.703 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.703 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.703 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.359146+0000", 2024-09-18T07:48:27.704 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.125e-06 2024-09-18T07:48:27.704 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.704 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.704 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.704 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.359153+0000", 2024-09-18T07:48:27.704 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4099999999999996e-06 2024-09-18T07:48:27.704 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.704 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.704 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.704 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.359164+0000", 2024-09-18T07:48:27.704 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1104000000000001e-05 2024-09-18T07:48:27.704 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.704 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.705 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.705 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.359171+0000", 2024-09-18T07:48:27.705 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3950000000000003e-06 2024-09-18T07:48:27.705 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.705 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.705 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.705 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.359228+0000", 2024-09-18T07:48:27.705 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6341000000000001e-05 2024-09-18T07:48:27.705 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.705 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.705 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.705 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.359236+0000", 2024-09-18T07:48:27.706 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0960000000000001e-06 2024-09-18T07:48:27.706 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.706 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.706 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.706 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.389179+0000", 2024-09-18T07:48:27.706 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0299431889999999 2024-09-18T07:48:27.706 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.706 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.706 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:27.706 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.389287+0000", 2024-09-18T07:48:27.706 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000107902 2024-09-18T07:48:27.706 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.706 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.707 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:48:27.707 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.389427+0000", 2024-09-18T07:48:27.707 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000139763 2024-09-18T07:48:27.707 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.707 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.707 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:27.707 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.393770+0000", 2024-09-18T07:48:27.707 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0043434440000000001 2024-09-18T07:48:27.707 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.707 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.707 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.707 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.397844+0000", 2024-09-18T07:48:27.707 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0040739160000000003 2024-09-18T07:48:27.707 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.708 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.708 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.708 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.402660+0000", 2024-09-18T07:48:27.708 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0048165150000000004 2024-09-18T07:48:27.708 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.708 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.708 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:27.708 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.402693+0000", 2024-09-18T07:48:27.708 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2381e-05 2024-09-18T07:48:27.708 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.708 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.708 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.708 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.402766+0000", 2024-09-18T07:48:27.709 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3075000000000001e-05 2024-09-18T07:48:27.709 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.709 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.709 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.709 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.709 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.709 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189626 10.2 10:461f64b9:::benchmark_data_smithi137_151160_object53810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:48:27.709 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.364564+0000", 2024-09-18T07:48:27.709 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.27907391700001, 2024-09-18T07:48:27.709 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0384643680000001, 2024-09-18T07:48:27.709 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.710 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.710 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:27.710 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.710 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.710 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.710 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189626 2024-09-18T07:48:27.710 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.710 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.710 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.710 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.710 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.364564+0000", 2024-09-18T07:48:27.710 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.710 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.710 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.711 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.711 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.364564+0000", 2024-09-18T07:48:27.711 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.711 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.711 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.711 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.711 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.364566+0000", 2024-09-18T07:48:27.711 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8080000000000001e-06 2024-09-18T07:48:27.711 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.711 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.711 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.711 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.364573+0000", 2024-09-18T07:48:27.711 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7850000000000003e-06 2024-09-18T07:48:27.711 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.712 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.712 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.712 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.364588+0000", 2024-09-18T07:48:27.712 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4806e-05 2024-09-18T07:48:27.712 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.712 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.712 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.712 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.364596+0000", 2024-09-18T07:48:27.712 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1689999999999992e-06 2024-09-18T07:48:27.712 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.712 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.712 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.712 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.364654+0000", 2024-09-18T07:48:27.712 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8658e-05 2024-09-18T07:48:27.713 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.713 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.713 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.713 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.364662+0000", 2024-09-18T07:48:27.713 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7409999999999992e-06 2024-09-18T07:48:27.713 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.713 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.713 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.713 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.391301+0000", 2024-09-18T07:48:27.713 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0266390919999999 2024-09-18T07:48:27.713 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.713 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.714 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:27.714 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.391419+0000", 2024-09-18T07:48:27.714 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000117724 2024-09-18T07:48:27.714 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.714 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.714 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:48:27.714 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.391536+0000", 2024-09-18T07:48:27.714 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000117162 2024-09-18T07:48:27.714 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.714 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.714 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:27.714 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.397781+0000", 2024-09-18T07:48:27.714 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0062446330000000003 2024-09-18T07:48:27.715 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.715 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.715 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.715 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.397917+0000", 2024-09-18T07:48:27.715 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013668600000000001 2024-09-18T07:48:27.715 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.715 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.715 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.715 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.402881+0000", 2024-09-18T07:48:27.715 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0049634040000000003 2024-09-18T07:48:27.715 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.715 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.715 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:27.716 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.402936+0000", 2024-09-18T07:48:27.716 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5284999999999997e-05 2024-09-18T07:48:27.716 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.716 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.716 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.716 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.403029+0000", 2024-09-18T07:48:27.716 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2415000000000002e-05 2024-09-18T07:48:27.716 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.716 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.716 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.716 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.716 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.716 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189638 10.2 10:423cbe89:::benchmark_data_smithi137_151160_object53822:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:48:27.717 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.372890+0000", 2024-09-18T07:48:27.717 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.27074809499999, 2024-09-18T07:48:27.717 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0303354929999999, 2024-09-18T07:48:27.717 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.717 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.717 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:27.717 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.717 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.717 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.717 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189638 2024-09-18T07:48:27.717 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.717 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.717 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.717 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.718 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.372890+0000", 2024-09-18T07:48:27.718 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.718 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.718 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.718 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.718 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.372890+0000", 2024-09-18T07:48:27.718 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.718 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.718 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.718 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.718 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.372892+0000", 2024-09-18T07:48:27.718 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.88e-06 2024-09-18T07:48:27.718 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.719 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.719 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.719 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.372897+0000", 2024-09-18T07:48:27.719 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3419999999999998e-06 2024-09-18T07:48:27.719 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.719 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.719 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.719 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.372908+0000", 2024-09-18T07:48:27.719 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0421999999999999e-05 2024-09-18T07:48:27.719 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.719 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.719 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.719 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.372914+0000", 2024-09-18T07:48:27.720 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6699999999999997e-06 2024-09-18T07:48:27.720 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.720 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.720 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.720 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.372977+0000", 2024-09-18T07:48:27.720 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2943999999999999e-05 2024-09-18T07:48:27.720 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.720 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.720 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.720 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.372985+0000", 2024-09-18T07:48:27.720 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7759999999999997e-06 2024-09-18T07:48:27.720 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.720 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.721 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.721 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.393800+0000", 2024-09-18T07:48:27.721 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.020814707 2024-09-18T07:48:27.721 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.721 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.721 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:27.721 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.393866+0000", 2024-09-18T07:48:27.721 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6685000000000003e-05 2024-09-18T07:48:27.721 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.721 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.721 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:48:27.721 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.393999+0000", 2024-09-18T07:48:27.721 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013285 2024-09-18T07:48:27.722 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.722 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.722 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:27.722 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.397931+0000", 2024-09-18T07:48:27.722 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0039321260000000002 2024-09-18T07:48:27.722 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.722 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.722 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.722 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.402167+0000", 2024-09-18T07:48:27.722 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0042359850000000003 2024-09-18T07:48:27.722 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.722 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.722 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.723 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.403082+0000", 2024-09-18T07:48:27.723 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00091465000000000003 2024-09-18T07:48:27.723 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.723 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.723 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:27.723 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.403134+0000", 2024-09-18T07:48:27.723 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2346000000000003e-05 2024-09-18T07:48:27.723 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.723 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.723 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.723 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.403226+0000", 2024-09-18T07:48:27.723 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.111e-05 2024-09-18T07:48:27.724 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.724 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.724 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.724 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.724 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.724 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189676 10.2 10:446916f9:::benchmark_data_smithi137_151160_object53860:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:48:27.724 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.416477+0000", 2024-09-18T07:48:27.724 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 174.227160774, 2024-09-18T07:48:27.724 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.99152496400000001, 2024-09-18T07:48:27.724 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.724 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.724 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:27.725 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.725 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.725 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.725 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189676 2024-09-18T07:48:27.725 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.725 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.725 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.725 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.725 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.416477+0000", 2024-09-18T07:48:27.725 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.725 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.725 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.725 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.725 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.416477+0000", 2024-09-18T07:48:27.726 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.726 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.726 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.726 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.726 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.416480+0000", 2024-09-18T07:48:27.726 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:27.727 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.39e-06 2024-09-18T07:48:27.727 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.728 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.728 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.728 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.416487+0000", 2024-09-18T07:48:27.728 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.576e-06 2024-09-18T07:48:27.728 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.728 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.728 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.728 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.416501+0000", 2024-09-18T07:48:27.728 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3954999999999999e-05 2024-09-18T07:48:27.728 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.728 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.728 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.728 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.416509+0000", 2024-09-18T07:48:27.729 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4080000000000001e-06 2024-09-18T07:48:27.729 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.729 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.729 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.729 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.416568+0000", 2024-09-18T07:48:27.729 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9821000000000002e-05 2024-09-18T07:48:27.729 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.729 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.729 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.729 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.416578+0000", 2024-09-18T07:48:27.729 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0929999999999992e-06 2024-09-18T07:48:27.729 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.729 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.730 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.730 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.395905+0000", 2024-09-18T07:48:27.730 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.97932717400000002 2024-09-18T07:48:27.730 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.730 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.730 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:27.730 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.395985+0000", 2024-09-18T07:48:27.730 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9772999999999997e-05 2024-09-18T07:48:27.730 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.730 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.730 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:48:27.730 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.396056+0000", 2024-09-18T07:48:27.731 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1829000000000006e-05 2024-09-18T07:48:27.731 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.731 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.731 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:27.731 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.402107+0000", 2024-09-18T07:48:27.731 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0060509980000000001 2024-09-18T07:48:27.731 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.731 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.731 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.731 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.403279+0000", 2024-09-18T07:48:27.731 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0011720000000000001 2024-09-18T07:48:27.731 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.731 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.732 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.732 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.407848+0000", 2024-09-18T07:48:27.732 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0045689349999999997 2024-09-18T07:48:27.732 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.732 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.732 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:27.732 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.407904+0000", 2024-09-18T07:48:27.732 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6113000000000003e-05 2024-09-18T07:48:27.732 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.732 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.732 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.732 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.408002+0000", 2024-09-18T07:48:27.733 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7899000000000006e-05 2024-09-18T07:48:27.733 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.733 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.733 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.733 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.733 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.733 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:199982 10.1 10:8621fea9:::benchmark_data_smithi137_151160_object64166:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e430)", 2024-09-18T07:48:27.733 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:55.078327+0000", 2024-09-18T07:48:27.733 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 152.56531090999999, 2024-09-18T07:48:27.733 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.047601421, 2024-09-18T07:48:27.733 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.733 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.733 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:48:27.734 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.734 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.734 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.734 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 199982 2024-09-18T07:48:27.734 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.734 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.734 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.734 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.734 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.078327+0000", 2024-09-18T07:48:27.734 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.734 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.734 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.734 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.735 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.078327+0000", 2024-09-18T07:48:27.735 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.735 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.735 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.735 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.735 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.078329+0000", 2024-09-18T07:48:27.735 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.018e-06 2024-09-18T07:48:27.735 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.735 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.735 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.735 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.078335+0000", 2024-09-18T07:48:27.735 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4539999999999998e-06 2024-09-18T07:48:27.735 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.736 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.736 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.736 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.078346+0000", 2024-09-18T07:48:27.736 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.164e-05 2024-09-18T07:48:27.736 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.736 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.736 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.736 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.078354+0000", 2024-09-18T07:48:27.736 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0790000000000001e-06 2024-09-18T07:48:27.736 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.736 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.736 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.737 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.079784+0000", 2024-09-18T07:48:27.737 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014295880000000001 2024-09-18T07:48:27.737 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.737 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.737 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.737 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.079792+0000", 2024-09-18T07:48:27.737 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1829999999999994e-06 2024-09-18T07:48:27.737 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.737 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.737 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.737 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.115555+0000", 2024-09-18T07:48:27.737 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0357623970000001 2024-09-18T07:48:27.737 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.738 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.738 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for missing object", 2024-09-18T07:48:27.738 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.115633+0000", 2024-09-18T07:48:27.738 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8830000000000002e-05 2024-09-18T07:48:27.738 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.738 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.738 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.738 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125890+0000", 2024-09-18T07:48:27.738 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010256131999999999 2024-09-18T07:48:27.738 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.738 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.738 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.739 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125929+0000", 2024-09-18T07:48:27.739 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9100000000000002e-05 2024-09-18T07:48:27.739 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.739 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.739 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.739 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.739 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.739 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:199992 10.1 10:8f0b8e8f:::benchmark_data_smithi137_151160_object64176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-09-18T07:48:27.739 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:55.089372+0000", 2024-09-18T07:48:27.739 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 152.55426566099999, 2024-09-18T07:48:27.739 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0362496459999999, 2024-09-18T07:48:27.739 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.740 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.740 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:27.740 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.740 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.740 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.740 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 199992 2024-09-18T07:48:27.740 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.740 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.740 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.740 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.740 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.089372+0000", 2024-09-18T07:48:27.740 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.740 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.741 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.741 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.741 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.089372+0000", 2024-09-18T07:48:27.741 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.741 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.741 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.741 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.741 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.089374+0000", 2024-09-18T07:48:27.741 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8169999999999999e-06 2024-09-18T07:48:27.741 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.741 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.741 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.741 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.089380+0000", 2024-09-18T07:48:27.741 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7830000000000004e-06 2024-09-18T07:48:27.742 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.742 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.742 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.742 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.089392+0000", 2024-09-18T07:48:27.742 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2041999999999999e-05 2024-09-18T07:48:27.742 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.742 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.742 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.742 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.089398+0000", 2024-09-18T07:48:27.742 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4070000000000001e-06 2024-09-18T07:48:27.742 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.742 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.742 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.743 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.089453+0000", 2024-09-18T07:48:27.743 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5881999999999998e-05 2024-09-18T07:48:27.743 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.743 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.743 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.743 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.089459+0000", 2024-09-18T07:48:27.743 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.587e-06 2024-09-18T07:48:27.743 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.743 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.743 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.743 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.115822+0000", 2024-09-18T07:48:27.743 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.026363409 2024-09-18T07:48:27.743 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.743 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.744 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:27.744 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.115891+0000", 2024-09-18T07:48:27.744 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8275999999999995e-05 2024-09-18T07:48:27.744 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.744 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.744 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,7", 2024-09-18T07:48:27.744 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.116206+0000", 2024-09-18T07:48:27.744 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00031504100000000001 2024-09-18T07:48:27.744 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.744 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.744 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.744 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125445+0000", 2024-09-18T07:48:27.744 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0092397869999999993 2024-09-18T07:48:27.745 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.745 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.745 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:27.745 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125457+0000", 2024-09-18T07:48:27.745 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.119e-05 2024-09-18T07:48:27.745 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.745 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.745 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.745 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125536+0000", 2024-09-18T07:48:27.745 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8971e-05 2024-09-18T07:48:27.745 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.745 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.746 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:27.746 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125569+0000", 2024-09-18T07:48:27.746 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3135e-05 2024-09-18T07:48:27.746 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.746 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.746 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.746 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125622+0000", 2024-09-18T07:48:27.746 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3319000000000002e-05 2024-09-18T07:48:27.746 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.746 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.746 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.746 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.746 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.747 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:200000 10.1 10:8f465e9f:::benchmark_data_smithi137_151160_object64184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-09-18T07:48:27.747 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:55.098734+0000", 2024-09-18T07:48:27.747 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 152.54490368, 2024-09-18T07:48:27.747 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.026996365, 2024-09-18T07:48:27.747 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.747 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.747 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:27.747 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.747 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.747 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.747 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 200000 2024-09-18T07:48:27.747 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.747 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.748 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.748 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.748 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.098734+0000", 2024-09-18T07:48:27.748 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.748 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.748 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.748 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.748 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.098734+0000", 2024-09-18T07:48:27.748 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.748 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.748 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.748 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.748 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.098736+0000", 2024-09-18T07:48:27.749 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9860000000000001e-06 2024-09-18T07:48:27.749 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.749 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.749 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.749 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.098742+0000", 2024-09-18T07:48:27.749 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5849999999999999e-06 2024-09-18T07:48:27.749 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.749 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.749 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.749 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.098754+0000", 2024-09-18T07:48:27.749 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1882000000000001e-05 2024-09-18T07:48:27.749 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.750 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.750 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.750 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.098762+0000", 2024-09-18T07:48:27.750 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6450000000000005e-06 2024-09-18T07:48:27.750 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.750 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.750 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.750 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.098822+0000", 2024-09-18T07:48:27.750 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0618000000000001e-05 2024-09-18T07:48:27.750 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.750 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.750 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.750 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.098832+0000", 2024-09-18T07:48:27.751 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5580000000000006e-06 2024-09-18T07:48:27.751 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.751 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.751 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.751 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.121193+0000", 2024-09-18T07:48:27.751 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.022361694 2024-09-18T07:48:27.751 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.751 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.751 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:27.751 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.121336+0000", 2024-09-18T07:48:27.751 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014290900000000001 2024-09-18T07:48:27.751 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.751 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.751 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,7", 2024-09-18T07:48:27.752 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.121485+0000", 2024-09-18T07:48:27.752 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000148706 2024-09-18T07:48:27.752 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.752 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.752 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:27.752 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125465+0000", 2024-09-18T07:48:27.752 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0039797749999999996 2024-09-18T07:48:27.752 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.752 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.752 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.752 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125507+0000", 2024-09-18T07:48:27.752 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2333999999999998e-05 2024-09-18T07:48:27.752 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.753 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.753 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.753 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125659+0000", 2024-09-18T07:48:27.753 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000152029 2024-09-18T07:48:27.753 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.753 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.753 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:27.753 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125684+0000", 2024-09-18T07:48:27.753 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5306000000000001e-05 2024-09-18T07:48:27.753 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.753 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.753 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.753 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125731+0000", 2024-09-18T07:48:27.754 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6338000000000003e-05 2024-09-18T07:48:27.754 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.754 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.754 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.754 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.754 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.754 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:200014 10.1 10:8d4a215b:::benchmark_data_smithi137_151160_object64198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-09-18T07:48:27.754 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:55.114672+0000", 2024-09-18T07:48:27.754 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 152.52896634699999, 2024-09-18T07:48:27.754 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.011365474, 2024-09-18T07:48:27.754 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.754 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.754 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:27.755 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.755 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.755 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.755 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 200014 2024-09-18T07:48:27.755 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.755 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.755 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.755 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.755 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.114672+0000", 2024-09-18T07:48:27.755 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.755 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.755 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.755 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.755 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.114672+0000", 2024-09-18T07:48:27.756 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.756 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.756 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.756 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.756 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.114674+0000", 2024-09-18T07:48:27.756 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.34e-06 2024-09-18T07:48:27.756 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.756 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.756 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.756 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.114680+0000", 2024-09-18T07:48:27.756 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6420000000000004e-06 2024-09-18T07:48:27.756 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.756 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.757 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.757 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.114695+0000", 2024-09-18T07:48:27.757 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5255999999999999e-05 2024-09-18T07:48:27.757 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.757 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.757 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.757 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.114703+0000", 2024-09-18T07:48:27.757 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9650000000000008e-06 2024-09-18T07:48:27.757 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.757 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.757 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.757 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.115999+0000", 2024-09-18T07:48:27.757 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001295907 2024-09-18T07:48:27.758 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.758 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.758 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.758 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.116007+0000", 2024-09-18T07:48:27.758 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2570000000000007e-06 2024-09-18T07:48:27.758 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.758 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.758 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.758 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.123175+0000", 2024-09-18T07:48:27.758 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.007167897 2024-09-18T07:48:27.758 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.758 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.758 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:27.759 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.123291+0000", 2024-09-18T07:48:27.759 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000116278 2024-09-18T07:48:27.759 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.759 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.759 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,7", 2024-09-18T07:48:27.759 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.123384+0000", 2024-09-18T07:48:27.759 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2707000000000005e-05 2024-09-18T07:48:27.759 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.759 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.759 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:27.759 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125471+0000", 2024-09-18T07:48:27.759 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0020867210000000001 2024-09-18T07:48:27.759 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.759 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.759 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.760 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125762+0000", 2024-09-18T07:48:27.760 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00029164300000000001 2024-09-18T07:48:27.760 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.760 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.760 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.760 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125960+0000", 2024-09-18T07:48:27.760 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019801500000000001 2024-09-18T07:48:27.760 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.760 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.760 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:27.760 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125995+0000", 2024-09-18T07:48:27.760 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4347999999999997e-05 2024-09-18T07:48:27.761 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.761 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.761 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.761 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.126037+0000", 2024-09-18T07:48:27.761 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2497999999999999e-05 2024-09-18T07:48:27.761 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.761 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.761 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.761 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.761 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.761 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:200025 10.1 10:841dfac0:::benchmark_data_smithi137_151160_object64209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-09-18T07:48:27.761 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:55.123654+0000", 2024-09-18T07:48:27.761 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 152.51998418700001, 2024-09-18T07:48:27.762 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.014501978, 2024-09-18T07:48:27.762 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.762 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.762 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:27.762 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.762 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.762 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.762 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 200025 2024-09-18T07:48:27.762 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.762 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.762 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.762 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.762 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.123654+0000", 2024-09-18T07:48:27.763 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.763 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.763 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.763 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.123654+0000", 2024-09-18T07:48:27.763 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.763 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.763 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.763 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.123655+0000", 2024-09-18T07:48:27.763 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1429999999999999e-06 2024-09-18T07:48:27.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.763 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.763 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.763 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.123659+0000", 2024-09-18T07:48:27.764 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9600000000000002e-06 2024-09-18T07:48:27.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.764 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.764 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.764 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.123668+0000", 2024-09-18T07:48:27.764 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2869999999999993e-06 2024-09-18T07:48:27.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.764 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.764 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.764 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.123675+0000", 2024-09-18T07:48:27.764 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8120000000000002e-06 2024-09-18T07:48:27.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.764 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.765 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.765 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.123785+0000", 2024-09-18T07:48:27.765 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000110006 2024-09-18T07:48:27.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.765 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.765 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.765 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.123792+0000", 2024-09-18T07:48:27.765 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2099999999999996e-06 2024-09-18T07:48:27.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.765 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.765 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.765 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.126103+0000", 2024-09-18T07:48:27.765 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0023103499999999 2024-09-18T07:48:27.766 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.766 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.766 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:27.766 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.126200+0000", 2024-09-18T07:48:27.766 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7106000000000002e-05 2024-09-18T07:48:27.766 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.766 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.766 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,7", 2024-09-18T07:48:27.766 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.126318+0000", 2024-09-18T07:48:27.766 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000118439 2024-09-18T07:48:27.766 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.766 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.766 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.767 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.133064+0000", 2024-09-18T07:48:27.767 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0067462420000000004 2024-09-18T07:48:27.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.767 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.767 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.767 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.133128+0000", 2024-09-18T07:48:27.767 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3774000000000003e-05 2024-09-18T07:48:27.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.767 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.767 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:27.767 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.137958+0000", 2024-09-18T07:48:27.767 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0048296800000000003 2024-09-18T07:48:27.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.768 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.768 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:27.768 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.138049+0000", 2024-09-18T07:48:27.768 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0587000000000005e-05 2024-09-18T07:48:27.768 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.768 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.768 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.768 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.138156+0000", 2024-09-18T07:48:27.768 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000107382 2024-09-18T07:48:27.768 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.768 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.768 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.768 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.769 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.769 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:251939 10.2 10:4579883a:::benchmark_data_smithi137_151160_object116123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-18T07:48:27.769 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:46:38.919114+0000", 2024-09-18T07:48:27.769 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 108.724524272, 2024-09-18T07:48:27.769 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0558060870000001, 2024-09-18T07:48:27.769 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.769 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.769 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:27.769 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.769 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.769 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.769 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 251939 2024-09-18T07:48:27.769 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.769 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.770 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.770 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.770 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.919114+0000", 2024-09-18T07:48:27.770 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.770 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.770 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.770 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.770 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.919114+0000", 2024-09-18T07:48:27.770 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.770 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.770 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.770 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.770 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.919115+0000", 2024-09-18T07:48:27.770 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3170000000000001e-06 2024-09-18T07:48:27.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.771 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.771 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.771 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.919120+0000", 2024-09-18T07:48:27.771 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.425e-06 2024-09-18T07:48:27.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.771 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.771 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.771 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.919128+0000", 2024-09-18T07:48:27.771 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0690000000000002e-06 2024-09-18T07:48:27.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.771 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.772 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.772 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.919132+0000", 2024-09-18T07:48:27.772 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4889999999999997e-06 2024-09-18T07:48:27.772 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.772 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.772 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.772 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.919190+0000", 2024-09-18T07:48:27.772 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8239999999999998e-05 2024-09-18T07:48:27.772 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.772 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.772 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.772 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.919202+0000", 2024-09-18T07:48:27.772 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1155000000000001e-05 2024-09-18T07:48:27.773 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.773 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.773 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.773 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.875920+0000", 2024-09-18T07:48:27.773 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.956717975 2024-09-18T07:48:27.773 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.773 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.773 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.773 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.875934+0000", 2024-09-18T07:48:27.773 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4823e-05 2024-09-18T07:48:27.773 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.773 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.773 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.774 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.966003+0000", 2024-09-18T07:48:27.774 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.090068364 2024-09-18T07:48:27.774 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.774 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.774 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:27.774 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.966103+0000", 2024-09-18T07:48:27.774 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010009900000000001 2024-09-18T07:48:27.774 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.774 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.774 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1,3", 2024-09-18T07:48:27.774 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.966454+0000", 2024-09-18T07:48:27.774 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00035081199999999999 2024-09-18T07:48:27.774 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.774 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.775 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.775 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.971862+0000", 2024-09-18T07:48:27.775 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0054080459999999997 2024-09-18T07:48:27.775 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.775 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.775 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:27.775 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.972094+0000", 2024-09-18T07:48:27.775 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023215000000000001 2024-09-18T07:48:27.775 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.775 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.775 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.775 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.974670+0000", 2024-09-18T07:48:27.775 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025764410000000001 2024-09-18T07:48:27.776 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.776 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.776 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.776 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.974753+0000", 2024-09-18T07:48:27.776 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3001e-05 2024-09-18T07:48:27.776 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.776 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.776 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:27.776 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.974811+0000", 2024-09-18T07:48:27.776 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8119999999999999e-05 2024-09-18T07:48:27.776 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.776 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.776 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.777 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.974920+0000", 2024-09-18T07:48:27.777 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010856100000000001 2024-09-18T07:48:27.777 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.777 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.777 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.777 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.777 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.777 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:251948 10.2 10:4604b5ef:::benchmark_data_smithi137_151160_object116132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-18T07:48:27.777 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:46:38.933161+0000", 2024-09-18T07:48:27.777 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 108.710477006, 2024-09-18T07:48:27.777 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0420709339999998, 2024-09-18T07:48:27.777 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.777 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.778 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:27.778 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.778 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.778 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.778 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 251948 2024-09-18T07:48:27.778 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.778 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.778 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.778 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.778 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.933161+0000", 2024-09-18T07:48:27.778 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.778 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.778 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.779 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.779 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.933161+0000", 2024-09-18T07:48:27.779 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.779 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.779 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.779 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.779 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.933163+0000", 2024-09-18T07:48:27.779 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4470000000000001e-06 2024-09-18T07:48:27.779 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.779 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.779 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.779 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.933167+0000", 2024-09-18T07:48:27.779 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.685e-06 2024-09-18T07:48:27.779 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.780 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.780 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.780 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.933177+0000", 2024-09-18T07:48:27.780 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8500000000000006e-06 2024-09-18T07:48:27.780 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.780 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.780 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.780 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.933182+0000", 2024-09-18T07:48:27.780 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0540000000000002e-06 2024-09-18T07:48:27.780 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.780 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.780 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.781 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.933228+0000", 2024-09-18T07:48:27.781 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5850000000000003e-05 2024-09-18T07:48:27.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.781 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.781 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.781 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.933234+0000", 2024-09-18T07:48:27.781 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.913e-06 2024-09-18T07:48:27.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.781 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.781 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.781 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.875969+0000", 2024-09-18T07:48:27.781 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.94273543699999995 2024-09-18T07:48:27.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.782 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.782 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.782 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.875978+0000", 2024-09-18T07:48:27.782 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2900000000000002e-06 2024-09-18T07:48:27.782 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.782 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.782 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.782 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.969585+0000", 2024-09-18T07:48:27.782 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.093607574 2024-09-18T07:48:27.782 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.782 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.782 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:27.782 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.969678+0000", 2024-09-18T07:48:27.782 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2670999999999994e-05 2024-09-18T07:48:27.783 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.783 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.783 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1,3", 2024-09-18T07:48:27.783 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.969767+0000", 2024-09-18T07:48:27.783 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8710999999999998e-05 2024-09-18T07:48:27.783 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.783 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.783 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:27.783 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.972100+0000", 2024-09-18T07:48:27.783 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023329539999999999 2024-09-18T07:48:27.783 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.783 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.783 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.784 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.974612+0000", 2024-09-18T07:48:27.784 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002512182 2024-09-18T07:48:27.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.784 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.784 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.784 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.974711+0000", 2024-09-18T07:48:27.784 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9077e-05 2024-09-18T07:48:27.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.784 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.784 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.784 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975085+0000", 2024-09-18T07:48:27.784 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00037395000000000001 2024-09-18T07:48:27.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.785 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.785 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:27.785 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975134+0000", 2024-09-18T07:48:27.785 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9251e-05 2024-09-18T07:48:27.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.785 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.785 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.785 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975232+0000", 2024-09-18T07:48:27.785 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8037999999999999e-05 2024-09-18T07:48:27.785 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.785 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.785 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.785 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.786 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:251954 10.2 10:48d2efb2:::benchmark_data_smithi137_151160_object116138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-18T07:48:27.787 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:46:38.937932+0000", 2024-09-18T07:48:27.787 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 108.705706229, 2024-09-18T07:48:27.787 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0375994749999999, 2024-09-18T07:48:27.787 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.787 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.787 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:27.787 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.787 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.787 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.787 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 251954 2024-09-18T07:48:27.787 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.787 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.787 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.788 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.788 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.937932+0000", 2024-09-18T07:48:27.788 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.788 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.788 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.788 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.788 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.937932+0000", 2024-09-18T07:48:27.788 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.788 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.788 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.788 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.788 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.937934+0000", 2024-09-18T07:48:27.788 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.77e-06 2024-09-18T07:48:27.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.789 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.789 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.789 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.937939+0000", 2024-09-18T07:48:27.789 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1749999999999996e-06 2024-09-18T07:48:27.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.789 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.789 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.789 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.937960+0000", 2024-09-18T07:48:27.789 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1426000000000001e-05 2024-09-18T07:48:27.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.789 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.789 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.790 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.937968+0000", 2024-09-18T07:48:27.790 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3209999999999999e-06 2024-09-18T07:48:27.790 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.790 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.790 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.790 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.938028+0000", 2024-09-18T07:48:27.790 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0856999999999997e-05 2024-09-18T07:48:27.790 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.790 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.790 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.790 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.938038+0000", 2024-09-18T07:48:27.790 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0003000000000001e-05 2024-09-18T07:48:27.791 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.791 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.791 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.791 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.876003+0000", 2024-09-18T07:48:27.791 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93796477700000003 2024-09-18T07:48:27.791 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.791 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.791 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.791 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.876011+0000", 2024-09-18T07:48:27.791 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9149999999999996e-06 2024-09-18T07:48:27.791 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.791 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.791 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.792 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.972190+0000", 2024-09-18T07:48:27.792 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0961789040000001 2024-09-18T07:48:27.792 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.792 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.792 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:27.792 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.972346+0000", 2024-09-18T07:48:27.792 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015571500000000001 2024-09-18T07:48:27.792 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.792 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.792 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1,3", 2024-09-18T07:48:27.792 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.972492+0000", 2024-09-18T07:48:27.792 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014632499999999999 2024-09-18T07:48:27.792 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.793 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.793 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:27.793 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975039+0000", 2024-09-18T07:48:27.793 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002546825 2024-09-18T07:48:27.793 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.793 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.793 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.793 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975299+0000", 2024-09-18T07:48:27.793 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00026037199999999998 2024-09-18T07:48:27.793 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.793 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.793 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.794 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975357+0000", 2024-09-18T07:48:27.794 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7661000000000002e-05 2024-09-18T07:48:27.794 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.794 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.794 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.794 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975398+0000", 2024-09-18T07:48:27.794 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1362000000000001e-05 2024-09-18T07:48:27.794 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.794 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.794 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:27.794 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975437+0000", 2024-09-18T07:48:27.794 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8729000000000001e-05 2024-09-18T07:48:27.794 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.794 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.795 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.795 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975531+0000", 2024-09-18T07:48:27.795 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4338000000000004e-05 2024-09-18T07:48:27.795 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.795 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.795 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.795 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.795 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.795 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:251992 10.2 10:447243fa:::benchmark_data_smithi137_151160_object116176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-18T07:48:27.795 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:46:38.970964+0000", 2024-09-18T07:48:27.795 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 108.67267388800001, 2024-09-18T07:48:27.795 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.016393844, 2024-09-18T07:48:27.795 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:27.796 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:27.796 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:27.796 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:27.796 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:48:27.796 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:27.796 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 251992 2024-09-18T07:48:27.796 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.796 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:27.796 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.796 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:27.796 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.970964+0000", 2024-09-18T07:48:27.796 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.796 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.797 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.797 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:27.797 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.970964+0000", 2024-09-18T07:48:27.797 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:27.797 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.797 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.797 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:27.797 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.970966+0000", 2024-09-18T07:48:27.797 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.274e-06 2024-09-18T07:48:27.797 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.797 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.797 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:27.797 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.970969+0000", 2024-09-18T07:48:27.798 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9589999999999997e-06 2024-09-18T07:48:27.798 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.798 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.798 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:27.798 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.970979+0000", 2024-09-18T07:48:27.798 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6069999999999997e-06 2024-09-18T07:48:27.798 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.798 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.798 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:27.798 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.970986+0000", 2024-09-18T07:48:27.798 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3509999999999996e-06 2024-09-18T07:48:27.798 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.798 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.798 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.799 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.971050+0000", 2024-09-18T07:48:27.799 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3492999999999997e-05 2024-09-18T07:48:27.799 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.799 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.799 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.799 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.971059+0000", 2024-09-18T07:48:27.799 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4800000000000007e-06 2024-09-18T07:48:27.799 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.799 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.799 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.799 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.876039+0000", 2024-09-18T07:48:27.799 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90497923199999997 2024-09-18T07:48:27.799 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.800 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.800 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:27.800 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.876046+0000", 2024-09-18T07:48:27.800 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7780000000000007e-06 2024-09-18T07:48:27.800 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.800 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.800 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:27.800 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975668+0000", 2024-09-18T07:48:27.800 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0996218470000001 2024-09-18T07:48:27.800 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.800 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.800 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:27.800 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975815+0000", 2024-09-18T07:48:27.800 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014654300000000001 2024-09-18T07:48:27.801 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.801 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.801 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1,3", 2024-09-18T07:48:27.801 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975960+0000", 2024-09-18T07:48:27.801 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014475600000000001 2024-09-18T07:48:27.801 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.801 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.801 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.801 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.986819+0000", 2024-09-18T07:48:27.801 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010859089000000001 2024-09-18T07:48:27.801 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.801 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.802 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:27.802 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.986832+0000", 2024-09-18T07:48:27.802 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2951e-05 2024-09-18T07:48:27.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.802 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.802 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.802 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.986870+0000", 2024-09-18T07:48:27.802 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8519999999999997e-05 2024-09-18T07:48:27.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.802 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.802 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:27.802 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.987163+0000", 2024-09-18T07:48:27.802 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00029275999999999998 2024-09-18T07:48:27.802 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.803 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.803 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:27.803 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.987243+0000", 2024-09-18T07:48:27.803 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0457999999999994e-05 2024-09-18T07:48:27.803 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:27.803 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:27.803 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:27.803 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.987358+0000", 2024-09-18T07:48:27.803 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000114746 2024-09-18T07:48:27.803 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.803 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.803 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.803 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:27.803 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:27.804 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:48:27.820 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:27.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:27.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:27.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:27.962 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:27.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:27.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:48:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:48:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:48:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:48:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:48:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:48:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:48:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:48:28.002 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:48:28.002 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:48:28.002 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:48:28.002 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:48:28.002 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:48:28.002 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:48:28.002 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:48:28.002 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:48:28.062 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 73 16 97591 97575 83.5177 64.5 0.0127053 0.0119687 2024-09-18T07:48:28.065 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:28.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:28.101 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:28.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:28.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:48:27 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:48:28.202 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:28.240 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:28.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:28.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:28.344 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:28.385 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:28.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:28.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:28.388 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[29408]: 1.0 scrub starts 2024-09-18T07:48:28.388 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[29408]: 1.0 scrub ok 2024-09-18T07:48:28.388 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:48:28.388 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:48:28.388 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[29408]: osdmap e496: 8 total, 8 up, 8 in 2024-09-18T07:48:28.388 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:48:28.388 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:48:28.389 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[29408]: 11.3 scrub ok 2024-09-18T07:48:28.389 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:48:28.389 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:48:28.389 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[29408]: 4.16 scrub starts 2024-09-18T07:48:28.389 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[29408]: 4.16 scrub ok 2024-09-18T07:48:28.389 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[39175]: 1.0 scrub starts 2024-09-18T07:48:28.389 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[39175]: 1.0 scrub ok 2024-09-18T07:48:28.389 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:48:28.389 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:48:28.389 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[39175]: osdmap e496: 8 total, 8 up, 8 in 2024-09-18T07:48:28.390 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:48:28.390 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:48:28.390 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[39175]: 11.3 scrub ok 2024-09-18T07:48:28.390 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:48:28.390 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:48:28.390 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[39175]: 4.16 scrub starts 2024-09-18T07:48:28.390 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:28 smithi137 ceph-mon[39175]: 4.16 scrub ok 2024-09-18T07:48:28.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:28 smithi203 ceph-mon[32237]: 1.0 scrub starts 2024-09-18T07:48:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:28 smithi203 ceph-mon[32237]: 1.0 scrub ok 2024-09-18T07:48:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:28 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:48:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:28 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:48:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:28 smithi203 ceph-mon[32237]: osdmap e496: 8 total, 8 up, 8 in 2024-09-18T07:48:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:28 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:48:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:28 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:48:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:28 smithi203 ceph-mon[32237]: 11.3 scrub ok 2024-09-18T07:48:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:28 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:48:28.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:28 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:48:28.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:28 smithi203 ceph-mon[32237]: 4.16 scrub starts 2024-09-18T07:48:28.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:28 smithi203 ceph-mon[32237]: 4.16 scrub ok 2024-09-18T07:48:28.487 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:28.515 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:28.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:28.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:28.617 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:28.642 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:28.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:28.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:28.744 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:28.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:28.777 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:28.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:28.879 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:28.912 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:28.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:28.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:28.982 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T07:48:28.984 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 6, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 6, 7, 4, 5] 2024-09-18T07:48:28.985 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:48:28.985 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 2, 3, 4, 5, 7, 6, 1] 2024-09-18T07:48:28.985 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd out 3 2024-09-18T07:48:29.013 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:29.062 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:29.063 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 74 16 98687 98671 83.3145 68.5 0.0149942 0.0119974 2024-09-18T07:48:29.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:29.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:29.164 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:29.203 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:29.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:29.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:29.305 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:29.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:29.360 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:29.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:29.365 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:48:29.402 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:29.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[29408]: pgmap v937: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-18T07:48:29.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:29.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[29408]: 1.2 scrub starts 2024-09-18T07:48:29.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[29408]: 1.2 scrub ok 2024-09-18T07:48:29.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[29408]: 3.1f scrub starts 2024-09-18T07:48:29.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[29408]: 3.1f scrub ok 2024-09-18T07:48:29.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[29408]: osdmap e497: 8 total, 8 up, 8 in 2024-09-18T07:48:29.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:48:29.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:48:29.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:48:29.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[39175]: pgmap v937: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-18T07:48:29.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:29.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[39175]: 1.2 scrub starts 2024-09-18T07:48:29.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[39175]: 1.2 scrub ok 2024-09-18T07:48:29.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[39175]: 3.1f scrub starts 2024-09-18T07:48:29.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[39175]: 3.1f scrub ok 2024-09-18T07:48:29.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[39175]: osdmap e497: 8 total, 8 up, 8 in 2024-09-18T07:48:29.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:48:29.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:48:29.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:29 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:48:29.460 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:29 smithi203 ceph-mon[32237]: pgmap v937: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-18T07:48:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:29 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:29.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:29 smithi203 ceph-mon[32237]: 1.2 scrub starts 2024-09-18T07:48:29.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:29 smithi203 ceph-mon[32237]: 1.2 scrub ok 2024-09-18T07:48:29.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:29 smithi203 ceph-mon[32237]: 3.1f scrub starts 2024-09-18T07:48:29.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:29 smithi203 ceph-mon[32237]: 3.1f scrub ok 2024-09-18T07:48:29.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:29 smithi203 ceph-mon[32237]: osdmap e497: 8 total, 8 up, 8 in 2024-09-18T07:48:29.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:29 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:48:29.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:29 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:48:29.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:29 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:48:29.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:29.511 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:29.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:29.612 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:29.643 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:29.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:29.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:29.747 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:29.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:29.787 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:29.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:29.888 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:29.924 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:29.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:29.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:30.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:30.063 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 75 16 99769 99753 83.1052 67.625 0.014146 0.0120277 2024-09-18T07:48:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:30.076 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:30.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:30.096 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:48:30.096 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:48:30.096 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.096 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.16581.0:99744 11.5 e497/460 11:a79f5bfe:::benchmark_data_smithi137_181069_object99743:head v 497'6166, mlcod=497'6164)", 2024-09-18T07:48:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:48:30.036620+0000", 2024-09-18T07:48:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0013338460000000001, 2024-09-18T07:48:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0013707070000000001, 2024-09-18T07:48:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:48:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:30.036620+0000", 2024-09-18T07:48:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:30.036620+0000", 2024-09-18T07:48:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:30.036622+0000", 2024-09-18T07:48:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9489999999999999e-06 2024-09-18T07:48:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:30.098 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:30.036652+0000", 2024-09-18T07:48:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9147999999999999e-05 2024-09-18T07:48:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:30.036670+0000", 2024-09-18T07:48:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8490000000000001e-05 2024-09-18T07:48:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:30.036680+0000", 2024-09-18T07:48:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9450000000000005e-06 2024-09-18T07:48:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:30.099 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:30.036751+0000", 2024-09-18T07:48:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0580999999999994e-05 2024-09-18T07:48:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:30.036774+0000", 2024-09-18T07:48:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3632e-05 2024-09-18T07:48:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:30.100 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.16581.0:99748 11.8 e497/460 11:15780277:::benchmark_data_smithi137_181069_object99747:head v 497'6186, mlcod=497'6183)", 2024-09-18T07:48:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:48:30.037866+0000", 2024-09-18T07:48:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 8.8036999999999998e-05, 2024-09-18T07:48:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000179853, 2024-09-18T07:48:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:48:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:30.037866+0000", 2024-09-18T07:48:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:30.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:30.037866+0000", 2024-09-18T07:48:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:30.037868+0000", 2024-09-18T07:48:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2619999999999999e-06 2024-09-18T07:48:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.102 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:30.037890+0000", 2024-09-18T07:48:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2687000000000001e-05 2024-09-18T07:48:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:30.037905+0000", 2024-09-18T07:48:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4908e-05 2024-09-18T07:48:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:30.037912+0000", 2024-09-18T07:48:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.472e-06 2024-09-18T07:48:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:30.103 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:30.037973+0000", 2024-09-18T07:48:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1377999999999994e-05 2024-09-18T07:48:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:30.037992+0000", 2024-09-18T07:48:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9221000000000001e-05 2024-09-18T07:48:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: ], 2024-09-18T07:48:30.104 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 2 2024-09-18T07:48:30.105 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:48:30.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:30.227 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:30.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:30.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:30.329 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:30.364 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:30.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:30.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:30.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:30 smithi137 ceph-mon[29408]: 5.1e scrub starts 2024-09-18T07:48:30.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:30 smithi137 ceph-mon[29408]: 5.1e scrub ok 2024-09-18T07:48:30.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:30 smithi137 ceph-mon[39175]: 5.1e scrub starts 2024-09-18T07:48:30.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:30 smithi137 ceph-mon[39175]: 5.1e scrub ok 2024-09-18T07:48:30.436 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:48:30.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:30 smithi203 ceph-mon[32237]: 5.1e scrub starts 2024-09-18T07:48:30.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:30 smithi203 ceph-mon[32237]: 5.1e scrub ok 2024-09-18T07:48:30.466 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:30.497 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:30.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:30.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:30.599 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:30.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:30.637 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:30.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:30.738 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:30.768 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:30.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:30.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:30.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:30.886 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T07:48:30.887 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:30.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:30 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a[29404]: 2024-09-18T07:48:30.492+0000 7fe5973a1640 -1 mon.a@0(leader).osd e498 definitely_dead 0 2024-09-18T07:48:30.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:30.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:30.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:31.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:31.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:48:31.060 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:31.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:31.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:31.063 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 76 15 100710 100695 82.7864 58.875 0.00453294 0.0120714 2024-09-18T07:48:31.162 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:31.192 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:31.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:31.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:31.227 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:48:31.242 INFO:teuthology.orchestra.run.smithi137.stderr:marked out osd.3. 2024-09-18T07:48:31.294 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:31.315 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:31.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:31.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:31.386 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:31 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:48:31.387 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:31 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:48:31.387 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:31 smithi203 ceph-mon[32237]: pgmap v939: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 898 B/s rd, 61 MiB/s wr, 983 op/s 2024-09-18T07:48:31.387 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:31 smithi203 ceph-mon[32237]: osdmap e498: 8 total, 8 up, 8 in 2024-09-18T07:48:31.387 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:31 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4141429144' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-18T07:48:31.387 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:31 smithi203 ceph-mon[32237]: from='client.36563 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-18T07:48:31.387 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:31 smithi203 ceph-mon[32237]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-18T07:48:31.387 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:31 smithi203 ceph-mon[32237]: 5.17 scrub starts 2024-09-18T07:48:31.387 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:31 smithi203 ceph-mon[32237]: 5.17 scrub ok 2024-09-18T07:48:31.387 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:31 smithi203 ceph-mon[32237]: 5.e scrub starts 2024-09-18T07:48:31.387 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:31 smithi203 ceph-mon[32237]: 5.e scrub ok 2024-09-18T07:48:31.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:31.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:48:31.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:48:31.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[29408]: pgmap v939: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 898 B/s rd, 61 MiB/s wr, 983 op/s 2024-09-18T07:48:31.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[29408]: osdmap e498: 8 total, 8 up, 8 in 2024-09-18T07:48:31.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4141429144' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-18T07:48:31.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[29408]: from='client.36563 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-18T07:48:31.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[29408]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-18T07:48:31.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[29408]: 5.17 scrub starts 2024-09-18T07:48:31.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[29408]: 5.17 scrub ok 2024-09-18T07:48:31.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[29408]: 5.e scrub starts 2024-09-18T07:48:31.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[29408]: 5.e scrub ok 2024-09-18T07:48:31.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:48:31.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:48:31.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[39175]: pgmap v939: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 898 B/s rd, 61 MiB/s wr, 983 op/s 2024-09-18T07:48:31.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[39175]: osdmap e498: 8 total, 8 up, 8 in 2024-09-18T07:48:31.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4141429144' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-18T07:48:31.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[39175]: from='client.36563 ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-18T07:48:31.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[39175]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-18T07:48:31.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[39175]: 5.17 scrub starts 2024-09-18T07:48:31.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[39175]: 5.17 scrub ok 2024-09-18T07:48:31.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[39175]: 5.e scrub starts 2024-09-18T07:48:31.423 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:31 smithi137 ceph-mon[39175]: 5.e scrub ok 2024-09-18T07:48:31.438 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:31.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:31.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:31.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:31.562 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:31.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:31.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:31.664 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:31.685 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:31.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:31.787 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:31.808 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:31.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:31.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:31.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:31.935 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:31.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:31.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:32.037 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:32.058 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:32.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:32.063 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 77 16 100788 100772 81.7739 4.8125 0.00347426 0.0120953 2024-09-18T07:48:32.115 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:48:32.116 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:48:32.116 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:48:32.116 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:48:32.116 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:48:32.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:32.184 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:32.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:32.286 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:32.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:32.317 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:32.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:32.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:32.440 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:32.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:32.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:32.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:32.563 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:32.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:32.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:32.665 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:32.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:32 smithi137 ceph-mon[29408]: from='client.36563 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-18T07:48:32.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:32 smithi137 ceph-mon[29408]: osdmap e499: 8 total, 8 up, 7 in 2024-09-18T07:48:32.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:32 smithi137 ceph-mon[29408]: 6.1e scrub starts 2024-09-18T07:48:32.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:32 smithi137 ceph-mon[29408]: 6.1e scrub ok 2024-09-18T07:48:32.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:32 smithi137 ceph-mon[39175]: from='client.36563 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-18T07:48:32.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:32 smithi137 ceph-mon[39175]: osdmap e499: 8 total, 8 up, 7 in 2024-09-18T07:48:32.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:32 smithi137 ceph-mon[39175]: 6.1e scrub starts 2024-09-18T07:48:32.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:32 smithi137 ceph-mon[39175]: 6.1e scrub ok 2024-09-18T07:48:32.688 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:32.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:32.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:32.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:32 smithi203 ceph-mon[32237]: from='client.36563 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-18T07:48:32.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:32 smithi203 ceph-mon[32237]: osdmap e499: 8 total, 8 up, 7 in 2024-09-18T07:48:32.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:32 smithi203 ceph-mon[32237]: 6.1e scrub starts 2024-09-18T07:48:32.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:32 smithi203 ceph-mon[32237]: 6.1e scrub ok 2024-09-18T07:48:32.778 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T07:48:32.790 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:32.840 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:32.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:32.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:32.942 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:32.965 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:32.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:32.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:33.063 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 78 16 100788 100772 80.7257 0 - 0.0120953 2024-09-18T07:48:33.066 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:33.070 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:48:33.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:33.089 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:33.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:33.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:33.219 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:33.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:33.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:33.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:33.369 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:33.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:33.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:33 smithi203 ceph-mon[32237]: pgmap v942: 158 pgs: 2 unknown, 38 peering, 1 active+clean+scrubbing, 117 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 219 B/s rd, 0 op/s 2024-09-18T07:48:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:33 smithi203 ceph-mon[32237]: Health check failed: Reduced data availability: 7 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-18T07:48:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:33 smithi203 ceph-mon[32237]: osdmap e500: 8 total, 8 up, 7 in 2024-09-18T07:48:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:33 smithi203 ceph-mon[32237]: 3.a scrub starts 2024-09-18T07:48:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:33 smithi203 ceph-mon[32237]: 3.a scrub ok 2024-09-18T07:48:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:33 smithi203 ceph-mon[32237]: 3.15 scrub starts 2024-09-18T07:48:33.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:33 smithi203 ceph-mon[32237]: 3.15 scrub ok 2024-09-18T07:48:33.472 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:33.495 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:33.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:33.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:33.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:33.607 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:48:33.607 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:48:33.608 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:48:33.608 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:48:33.608 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.608 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24412.0:3402 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 94763678224896 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-09-18T07:48:33.608 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:11.552301+0000", 2024-09-18T07:48:33.608 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 82.050192988000006, 2024-09-18T07:48:33.608 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.65149643400000001, 2024-09-18T07:48:33.608 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.608 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.608 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:48:33.608 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.609 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24412", 2024-09-18T07:48:33.609 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/4233491205", 2024-09-18T07:48:33.609 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 3402 2024-09-18T07:48:33.609 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.609 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.609 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.609 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.609 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552301+0000", 2024-09-18T07:48:33.609 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.609 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.609 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.610 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.610 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552301+0000", 2024-09-18T07:48:33.610 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.610 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.610 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.610 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.610 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552303+0000", 2024-09-18T07:48:33.610 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2230000000000001e-06 2024-09-18T07:48:33.610 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.610 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.610 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.610 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552308+0000", 2024-09-18T07:48:33.611 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8840000000000002e-06 2024-09-18T07:48:33.611 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.611 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.611 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.611 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552328+0000", 2024-09-18T07:48:33.611 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0360000000000002e-05 2024-09-18T07:48:33.611 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.611 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.611 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.611 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552337+0000", 2024-09-18T07:48:33.611 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3899999999999993e-06 2024-09-18T07:48:33.611 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.611 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.612 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.612 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552419+0000", 2024-09-18T07:48:33.612 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2566999999999997e-05 2024-09-18T07:48:33.612 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.612 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.612 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.612 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552429+0000", 2024-09-18T07:48:33.612 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1840000000000005e-06 2024-09-18T07:48:33.612 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.612 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.612 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.612 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:12.202182+0000", 2024-09-18T07:48:33.613 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.64975312100000004 2024-09-18T07:48:33.613 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.613 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.613 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.613 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:12.202354+0000", 2024-09-18T07:48:33.613 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017184399999999999 2024-09-18T07:48:33.613 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.613 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.613 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.613 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:12.203797+0000", 2024-09-18T07:48:33.613 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014438610000000001 2024-09-18T07:48:33.613 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.614 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.614 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.614 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.614 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.614 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24412.0:3406 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 94763678224896 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-09-18T07:48:33.614 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:11.552354+0000", 2024-09-18T07:48:33.614 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 82.050139666000007, 2024-09-18T07:48:33.614 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.65190388799999999, 2024-09-18T07:48:33.614 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.614 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.615 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:48:33.615 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.615 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24412", 2024-09-18T07:48:33.615 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/4233491205", 2024-09-18T07:48:33.615 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 3406 2024-09-18T07:48:33.615 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.615 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.615 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.615 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.615 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552354+0000", 2024-09-18T07:48:33.615 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.615 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.615 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.616 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.616 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552354+0000", 2024-09-18T07:48:33.616 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.616 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.616 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.616 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.616 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552355+0000", 2024-09-18T07:48:33.616 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0800000000000004e-07 2024-09-18T07:48:33.616 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.616 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.616 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.616 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552359+0000", 2024-09-18T07:48:33.616 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.49e-06 2024-09-18T07:48:33.617 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.617 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.617 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.617 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552365+0000", 2024-09-18T07:48:33.617 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.742e-06 2024-09-18T07:48:33.617 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.617 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.617 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.618 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552371+0000", 2024-09-18T07:48:33.618 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9290000000000003e-06 2024-09-18T07:48:33.618 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.618 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.618 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.618 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552454+0000", 2024-09-18T07:48:33.618 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3115000000000006e-05 2024-09-18T07:48:33.618 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.618 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.618 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.618 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552462+0000", 2024-09-18T07:48:33.618 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9820000000000008e-06 2024-09-18T07:48:33.618 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.619 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.619 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.619 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:12.203343+0000", 2024-09-18T07:48:33.619 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.65088101700000001 2024-09-18T07:48:33.619 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.619 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.619 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.619 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:12.203427+0000", 2024-09-18T07:48:33.619 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3855000000000003e-05 2024-09-18T07:48:33.619 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.619 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.619 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.619 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:12.204258+0000", 2024-09-18T07:48:33.619 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00083095 2024-09-18T07:48:33.620 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.620 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.620 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.620 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.620 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.620 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:29176 11.7 11:ee7a84c3:::benchmark_data_smithi137_181069_object29175:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e470)", 2024-09-18T07:48:33.620 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:32.184268+0000", 2024-09-18T07:48:33.620 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 61.418225929000002, 2024-09-18T07:48:33.620 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.096316757000000003, 2024-09-18T07:48:33.620 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.620 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.620 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.620 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.621 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.621 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.621 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 29176 2024-09-18T07:48:33.621 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.621 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.621 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.621 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.621 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.184268+0000", 2024-09-18T07:48:33.621 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.621 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.621 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.621 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.622 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.184268+0000", 2024-09-18T07:48:33.622 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.622 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.622 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.622 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.622 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.184270+0000", 2024-09-18T07:48:33.622 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9400000000000001e-06 2024-09-18T07:48:33.622 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.622 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.622 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.622 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.184375+0000", 2024-09-18T07:48:33.622 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000105454 2024-09-18T07:48:33.622 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.622 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.623 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.623 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.184387+0000", 2024-09-18T07:48:33.623 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1243000000000001e-05 2024-09-18T07:48:33.623 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.623 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.623 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.623 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.184394+0000", 2024-09-18T07:48:33.623 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.024e-06 2024-09-18T07:48:33.623 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.623 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.623 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.623 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.187324+0000", 2024-09-18T07:48:33.623 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0029305630000000002 2024-09-18T07:48:33.623 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.623 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.624 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.624 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.187462+0000", 2024-09-18T07:48:33.624 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013773 2024-09-18T07:48:33.624 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.624 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.624 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.624 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.187638+0000", 2024-09-18T07:48:33.624 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000176093 2024-09-18T07:48:33.624 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.624 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.624 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.624 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.189798+0000", 2024-09-18T07:48:33.625 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021596660000000002 2024-09-18T07:48:33.625 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.625 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.625 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.625 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.190116+0000", 2024-09-18T07:48:33.625 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00031792000000000002 2024-09-18T07:48:33.625 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.625 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.625 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.625 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.280327+0000", 2024-09-18T07:48:33.625 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.090211343999999999 2024-09-18T07:48:33.626 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.626 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.626 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.626 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.280425+0000", 2024-09-18T07:48:33.626 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8088e-05 2024-09-18T07:48:33.626 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.626 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.626 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.626 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.280585+0000", 2024-09-18T07:48:33.627 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015969199999999999 2024-09-18T07:48:33.627 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.627 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.627 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.627 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.627 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.627 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:29178 11.7 11:e1ada224:::benchmark_data_smithi137_181069_object29177:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e470)", 2024-09-18T07:48:33.627 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:32.186125+0000", 2024-09-18T07:48:33.627 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 61.416368532, 2024-09-18T07:48:33.627 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.098747373999999999, 2024-09-18T07:48:33.627 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.627 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.628 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.628 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.628 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.628 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.628 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 29178 2024-09-18T07:48:33.628 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.628 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.628 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.628 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.628 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.186125+0000", 2024-09-18T07:48:33.629 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.629 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.629 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.629 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.629 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.186125+0000", 2024-09-18T07:48:33.629 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.629 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.629 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.629 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.629 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.186127+0000", 2024-09-18T07:48:33.629 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8279999999999999e-06 2024-09-18T07:48:33.629 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.629 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.630 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.630 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.186207+0000", 2024-09-18T07:48:33.630 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0074999999999995e-05 2024-09-18T07:48:33.630 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.630 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.630 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.630 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.186222+0000", 2024-09-18T07:48:33.630 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4351e-05 2024-09-18T07:48:33.630 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.630 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.630 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.630 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.186230+0000", 2024-09-18T07:48:33.630 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0199999999999994e-06 2024-09-18T07:48:33.631 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.631 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.631 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.631 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.188657+0000", 2024-09-18T07:48:33.631 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0024274560000000001 2024-09-18T07:48:33.631 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.631 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.631 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.631 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.188766+0000", 2024-09-18T07:48:33.631 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108995 2024-09-18T07:48:33.631 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.631 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.631 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.631 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.188919+0000", 2024-09-18T07:48:33.632 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000153082 2024-09-18T07:48:33.632 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.632 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.632 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.632 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.189942+0000", 2024-09-18T07:48:33.632 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0010226739999999999 2024-09-18T07:48:33.632 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.632 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.632 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.632 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.190342+0000", 2024-09-18T07:48:33.632 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00040024700000000001 2024-09-18T07:48:33.632 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.632 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.633 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.633 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.284661+0000", 2024-09-18T07:48:33.633 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.094318418000000001 2024-09-18T07:48:33.633 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.633 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.633 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.633 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.284727+0000", 2024-09-18T07:48:33.633 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6933000000000004e-05 2024-09-18T07:48:33.633 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.634 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.634 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.634 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:32.284873+0000", 2024-09-18T07:48:33.634 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014529500000000001 2024-09-18T07:48:33.634 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.634 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.634 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.634 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.635 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.635 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30810 11.7 11:e3ec4fcf:::benchmark_data_smithi137_181069_object30809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:48:33.635 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.172317+0000", 2024-09-18T07:48:33.635 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 60.430177352999998, 2024-09-18T07:48:33.635 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0154700940000001, 2024-09-18T07:48:33.635 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.635 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.635 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.635 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.635 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.635 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.635 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30810 2024-09-18T07:48:33.636 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.636 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.636 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.636 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.636 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.172317+0000", 2024-09-18T07:48:33.636 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.636 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.636 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.636 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.636 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.172317+0000", 2024-09-18T07:48:33.636 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.636 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.637 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.637 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.637 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.172319+0000", 2024-09-18T07:48:33.637 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.836e-06 2024-09-18T07:48:33.637 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.637 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.637 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.637 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.172378+0000", 2024-09-18T07:48:33.637 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8761999999999997e-05 2024-09-18T07:48:33.637 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.637 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.637 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.637 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.172396+0000", 2024-09-18T07:48:33.638 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8122999999999999e-05 2024-09-18T07:48:33.638 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.638 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.638 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.638 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.172406+0000", 2024-09-18T07:48:33.638 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4150000000000005e-06 2024-09-18T07:48:33.638 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.638 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.638 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.638 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.172469+0000", 2024-09-18T07:48:33.638 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3324000000000006e-05 2024-09-18T07:48:33.638 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.639 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.639 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.639 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.172477+0000", 2024-09-18T07:48:33.639 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3950000000000001e-06 2024-09-18T07:48:33.639 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.639 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.639 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.640 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.183147+0000", 2024-09-18T07:48:33.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:33.642 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:33.642 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.010669912 2024-09-18T07:48:33.642 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.642 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.642 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.642 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.183289+0000", 2024-09-18T07:48:33.642 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014143700000000001 2024-09-18T07:48:33.643 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.643 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.643 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.643 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.183444+0000", 2024-09-18T07:48:33.643 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015522600000000001 2024-09-18T07:48:33.643 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.643 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.643 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.643 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.185800+0000", 2024-09-18T07:48:33.643 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023560489999999998 2024-09-18T07:48:33.643 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.643 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.644 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.644 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.185869+0000", 2024-09-18T07:48:33.644 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9043999999999998e-05 2024-09-18T07:48:33.644 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.644 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.644 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.644 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.187718+0000", 2024-09-18T07:48:33.644 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001848567 2024-09-18T07:48:33.645 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.645 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.645 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.645 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.187755+0000", 2024-09-18T07:48:33.645 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6962000000000003e-05 2024-09-18T07:48:33.645 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.645 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.645 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.645 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.187787+0000", 2024-09-18T07:48:33.645 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2042000000000003e-05 2024-09-18T07:48:33.645 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.645 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.646 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.646 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.646 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.646 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30838 11.7 11:e2d1080e:::benchmark_data_smithi137_181069_object30837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:48:33.646 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.201700+0000", 2024-09-18T07:48:33.646 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 60.400794382000001, 2024-09-18T07:48:33.646 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.99197580799999996, 2024-09-18T07:48:33.646 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.646 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.646 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.646 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.646 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.647 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.647 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30838 2024-09-18T07:48:33.647 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.647 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.647 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.647 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.647 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201700+0000", 2024-09-18T07:48:33.647 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.647 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.647 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.647 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.648 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201700+0000", 2024-09-18T07:48:33.648 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.648 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.648 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.648 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.648 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201702+0000", 2024-09-18T07:48:33.648 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1730000000000002e-06 2024-09-18T07:48:33.648 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.648 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.648 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.648 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201754+0000", 2024-09-18T07:48:33.648 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2125e-05 2024-09-18T07:48:33.649 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.649 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.649 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.649 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201771+0000", 2024-09-18T07:48:33.649 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6708e-05 2024-09-18T07:48:33.649 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.649 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.649 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.649 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201779+0000", 2024-09-18T07:48:33.649 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4409999999999993e-06 2024-09-18T07:48:33.649 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.649 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.649 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.650 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201847+0000", 2024-09-18T07:48:33.650 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8196999999999998e-05 2024-09-18T07:48:33.650 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.650 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.650 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.650 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201856+0000", 2024-09-18T07:48:33.650 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2720000000000003e-06 2024-09-18T07:48:33.650 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.650 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.650 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.650 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.185900+0000", 2024-09-18T07:48:33.650 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.98404355899999996 2024-09-18T07:48:33.650 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.651 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.651 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.651 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.186018+0000", 2024-09-18T07:48:33.651 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000117685 2024-09-18T07:48:33.651 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.651 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.651 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.651 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.186171+0000", 2024-09-18T07:48:33.651 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000153187 2024-09-18T07:48:33.651 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.651 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.651 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.652 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.189010+0000", 2024-09-18T07:48:33.652 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0028390619999999998 2024-09-18T07:48:33.652 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.652 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.652 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.652 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.189046+0000", 2024-09-18T07:48:33.652 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5655999999999999e-05 2024-09-18T07:48:33.652 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.652 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.653 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.653 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193582+0000", 2024-09-18T07:48:33.653 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0045365589999999999 2024-09-18T07:48:33.653 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.653 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.653 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.654 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193631+0000", 2024-09-18T07:48:33.654 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8959999999999999e-05 2024-09-18T07:48:33.654 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.654 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.654 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.654 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193675+0000", 2024-09-18T07:48:33.654 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4224000000000002e-05 2024-09-18T07:48:33.654 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.654 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.654 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.654 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.655 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.655 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30840 11.7 11:e6e0bbbb:::benchmark_data_smithi137_181069_object30839:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:48:33.655 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.201799+0000", 2024-09-18T07:48:33.655 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 60.400694836, 2024-09-18T07:48:33.655 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.99198059800000005, 2024-09-18T07:48:33.655 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.655 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.655 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.655 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.655 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.655 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.655 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30840 2024-09-18T07:48:33.656 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.656 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.656 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.656 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.656 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201799+0000", 2024-09-18T07:48:33.656 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.656 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.656 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.656 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.656 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201799+0000", 2024-09-18T07:48:33.656 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.656 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.656 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201800+0000", 2024-09-18T07:48:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4200000000000004e-07 2024-09-18T07:48:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201834+0000", 2024-09-18T07:48:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3600999999999999e-05 2024-09-18T07:48:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201841+0000", 2024-09-18T07:48:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1389999999999998e-06 2024-09-18T07:48:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.658 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.658 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.658 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201847+0000", 2024-09-18T07:48:33.658 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8889999999999999e-06 2024-09-18T07:48:33.658 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.658 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.658 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.658 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201877+0000", 2024-09-18T07:48:33.658 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0195000000000001e-05 2024-09-18T07:48:33.658 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.658 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.658 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.659 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201882+0000", 2024-09-18T07:48:33.659 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5450000000000003e-06 2024-09-18T07:48:33.659 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.659 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.659 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.659 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.187829+0000", 2024-09-18T07:48:33.659 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.98594618099999998 2024-09-18T07:48:33.659 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.659 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.659 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.660 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.187951+0000", 2024-09-18T07:48:33.660 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000121957 2024-09-18T07:48:33.660 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.660 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.660 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.660 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.188097+0000", 2024-09-18T07:48:33.660 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000146055 2024-09-18T07:48:33.660 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.660 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.660 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.660 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.191952+0000", 2024-09-18T07:48:33.660 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0038551369999999998 2024-09-18T07:48:33.660 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.660 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.661 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.661 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.192101+0000", 2024-09-18T07:48:33.661 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000149173 2024-09-18T07:48:33.661 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.661 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.661 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.661 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193684+0000", 2024-09-18T07:48:33.661 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001583416 2024-09-18T07:48:33.661 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.661 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.661 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.662 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193742+0000", 2024-09-18T07:48:33.662 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7515000000000001e-05 2024-09-18T07:48:33.662 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.662 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.662 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.662 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193780+0000", 2024-09-18T07:48:33.662 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7852999999999998e-05 2024-09-18T07:48:33.662 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.662 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.662 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.662 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.662 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.662 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30843 11.7 11:eb1a4352:::benchmark_data_smithi137_181069_object30842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:48:33.663 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.208267+0000", 2024-09-18T07:48:33.663 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 60.394226955999997, 2024-09-18T07:48:33.663 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.98756237099999999, 2024-09-18T07:48:33.663 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.663 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.663 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.663 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.663 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.663 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.663 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30843 2024-09-18T07:48:33.663 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.663 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.663 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.664 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.664 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.208267+0000", 2024-09-18T07:48:33.664 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.664 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.664 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.664 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.664 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.208267+0000", 2024-09-18T07:48:33.664 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.664 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.664 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.664 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.664 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.208269+0000", 2024-09-18T07:48:33.665 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0020000000000001e-06 2024-09-18T07:48:33.665 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.665 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.665 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.665 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.208324+0000", 2024-09-18T07:48:33.665 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5457000000000001e-05 2024-09-18T07:48:33.665 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.665 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.665 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.665 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.208338+0000", 2024-09-18T07:48:33.665 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3631999999999999e-05 2024-09-18T07:48:33.665 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.666 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.666 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.666 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.208347+0000", 2024-09-18T07:48:33.666 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5019999999999992e-06 2024-09-18T07:48:33.666 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.666 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.666 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.666 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.208411+0000", 2024-09-18T07:48:33.666 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3935000000000004e-05 2024-09-18T07:48:33.666 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.666 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.666 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.666 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.208422+0000", 2024-09-18T07:48:33.667 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1834000000000001e-05 2024-09-18T07:48:33.667 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.667 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.667 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.667 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.189072+0000", 2024-09-18T07:48:33.667 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.98064952599999999 2024-09-18T07:48:33.667 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.667 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.667 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.667 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.189189+0000", 2024-09-18T07:48:33.667 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000117236 2024-09-18T07:48:33.667 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.668 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.668 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.668 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.189307+0000", 2024-09-18T07:48:33.668 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000117862 2024-09-18T07:48:33.668 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.668 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.668 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.668 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.191991+0000", 2024-09-18T07:48:33.668 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026844799999999999 2024-09-18T07:48:33.668 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.668 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.668 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.668 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193786+0000", 2024-09-18T07:48:33.669 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001794992 2024-09-18T07:48:33.669 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.669 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.669 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.669 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195746+0000", 2024-09-18T07:48:33.669 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0019596940000000001 2024-09-18T07:48:33.669 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.669 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.669 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.669 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195792+0000", 2024-09-18T07:48:33.669 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6360000000000003e-05 2024-09-18T07:48:33.669 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.670 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.670 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.670 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195829+0000", 2024-09-18T07:48:33.670 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6859000000000001e-05 2024-09-18T07:48:33.670 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.670 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.670 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.670 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.670 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.670 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30877 11.7 11:e09287fc:::benchmark_data_smithi137_181069_object30876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:48:33.670 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.232483+0000", 2024-09-18T07:48:33.670 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 60.37001076, 2024-09-18T07:48:33.671 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.96350860000000005, 2024-09-18T07:48:33.671 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.671 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.671 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.671 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.671 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.671 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.671 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30877 2024-09-18T07:48:33.671 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.671 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.671 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.671 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.672 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.232483+0000", 2024-09-18T07:48:33.672 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.672 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.672 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.672 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.672 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.232483+0000", 2024-09-18T07:48:33.672 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.672 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.672 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.672 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.672 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.232486+0000", 2024-09-18T07:48:33.672 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3740000000000001e-06 2024-09-18T07:48:33.672 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.672 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.673 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.673 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.232568+0000", 2024-09-18T07:48:33.673 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2531e-05 2024-09-18T07:48:33.673 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.673 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.673 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.673 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.232581+0000", 2024-09-18T07:48:33.673 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3210000000000001e-05 2024-09-18T07:48:33.673 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.673 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.673 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.673 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.232588+0000", 2024-09-18T07:48:33.673 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9129999999999997e-06 2024-09-18T07:48:33.674 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.674 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.674 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.674 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.232649+0000", 2024-09-18T07:48:33.674 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1192000000000006e-05 2024-09-18T07:48:33.674 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.674 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.674 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.674 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.232659+0000", 2024-09-18T07:48:33.674 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1379999999999996e-06 2024-09-18T07:48:33.674 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.674 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.675 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.675 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.190261+0000", 2024-09-18T07:48:33.675 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.95760236600000004 2024-09-18T07:48:33.675 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.675 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.675 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.675 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.190368+0000", 2024-09-18T07:48:33.675 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000107292 2024-09-18T07:48:33.675 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.675 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.675 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.675 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.190492+0000", 2024-09-18T07:48:33.675 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012389700000000001 2024-09-18T07:48:33.675 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.676 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.676 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.676 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.192043+0000", 2024-09-18T07:48:33.676 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001550645 2024-09-18T07:48:33.676 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.676 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.676 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.676 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195649+0000", 2024-09-18T07:48:33.676 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003606068 2024-09-18T07:48:33.676 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.676 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.676 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.676 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195915+0000", 2024-09-18T07:48:33.676 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00026607699999999998 2024-09-18T07:48:33.677 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.677 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.677 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.677 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195961+0000", 2024-09-18T07:48:33.677 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5877000000000001e-05 2024-09-18T07:48:33.677 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.677 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.677 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.677 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195992+0000", 2024-09-18T07:48:33.677 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1019999999999998e-05 2024-09-18T07:48:33.677 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.677 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.677 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.678 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.678 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.678 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30915 11.7 11:e1818ecf:::benchmark_data_smithi137_181069_object30914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:48:33.678 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.260052+0000", 2024-09-18T07:48:33.678 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 60.342441590999996, 2024-09-18T07:48:33.678 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93627841599999995, 2024-09-18T07:48:33.678 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.678 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.678 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.678 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.678 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.678 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.678 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30915 2024-09-18T07:48:33.679 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.679 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.679 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.679 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.679 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.260052+0000", 2024-09-18T07:48:33.679 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.679 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.679 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.679 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.679 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.260052+0000", 2024-09-18T07:48:33.679 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.680 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.680 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.680 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.680 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.260054+0000", 2024-09-18T07:48:33.680 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1179999999999999e-06 2024-09-18T07:48:33.680 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.680 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.680 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.680 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.260094+0000", 2024-09-18T07:48:33.680 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9935999999999998e-05 2024-09-18T07:48:33.680 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.680 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.680 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.681 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.260109+0000", 2024-09-18T07:48:33.681 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4521e-05 2024-09-18T07:48:33.681 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.681 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.681 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.681 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.260117+0000", 2024-09-18T07:48:33.681 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8699999999999992e-06 2024-09-18T07:48:33.681 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.681 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.681 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.681 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.260174+0000", 2024-09-18T07:48:33.681 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7018000000000002e-05 2024-09-18T07:48:33.681 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.682 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.682 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.682 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.260183+0000", 2024-09-18T07:48:33.682 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7299999999999994e-06 2024-09-18T07:48:33.682 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.682 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.682 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.682 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.192246+0000", 2024-09-18T07:48:33.682 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.932063737 2024-09-18T07:48:33.682 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.682 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.682 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.682 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.192368+0000", 2024-09-18T07:48:33.683 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012155999999999999 2024-09-18T07:48:33.683 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.683 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.683 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.683 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.192511+0000", 2024-09-18T07:48:33.683 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014365800000000001 2024-09-18T07:48:33.683 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.683 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.683 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.683 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195843+0000", 2024-09-18T07:48:33.683 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003331442 2024-09-18T07:48:33.683 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.683 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.684 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.684 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195875+0000", 2024-09-18T07:48:33.684 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1890000000000001e-05 2024-09-18T07:48:33.684 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.684 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.684 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.684 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196056+0000", 2024-09-18T07:48:33.684 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018159599999999999 2024-09-18T07:48:33.684 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.684 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.684 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.684 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196289+0000", 2024-09-18T07:48:33.684 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000232859 2024-09-18T07:48:33.685 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.685 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.685 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.685 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196331+0000", 2024-09-18T07:48:33.685 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1480999999999998e-05 2024-09-18T07:48:33.685 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.685 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.685 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.685 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.685 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.685 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30925 11.7 11:ec9c9156:::benchmark_data_smithi137_181069_object30924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:48:33.685 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.264403+0000", 2024-09-18T07:48:33.686 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 60.338090653000002, 2024-09-18T07:48:33.686 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93234712799999997, 2024-09-18T07:48:33.686 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.686 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.686 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.686 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.686 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.686 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.686 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30925 2024-09-18T07:48:33.686 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.686 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.686 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.687 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.687 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.264403+0000", 2024-09-18T07:48:33.687 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.687 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.687 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.687 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.687 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.264403+0000", 2024-09-18T07:48:33.687 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.687 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.687 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.687 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.687 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.264406+0000", 2024-09-18T07:48:33.688 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.446e-06 2024-09-18T07:48:33.688 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.688 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.688 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.688 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.264478+0000", 2024-09-18T07:48:33.688 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2671999999999998e-05 2024-09-18T07:48:33.688 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.688 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.688 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.688 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.264498+0000", 2024-09-18T07:48:33.688 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9497e-05 2024-09-18T07:48:33.688 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.689 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.689 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.689 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.264507+0000", 2024-09-18T07:48:33.689 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0019999999999995e-06 2024-09-18T07:48:33.689 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.689 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.689 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.689 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.264575+0000", 2024-09-18T07:48:33.689 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8256999999999994e-05 2024-09-18T07:48:33.689 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.689 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.689 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.690 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.264586+0000", 2024-09-18T07:48:33.690 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0481e-05 2024-09-18T07:48:33.690 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.690 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.690 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.690 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193815+0000", 2024-09-18T07:48:33.690 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92922892999999995 2024-09-18T07:48:33.690 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.690 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.690 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.690 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193923+0000", 2024-09-18T07:48:33.690 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108276 2024-09-18T07:48:33.691 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.691 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.691 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.691 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.194043+0000", 2024-09-18T07:48:33.691 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000119974 2024-09-18T07:48:33.691 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.691 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.691 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.691 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195888+0000", 2024-09-18T07:48:33.691 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0018447719999999999 2024-09-18T07:48:33.691 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.691 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.691 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.692 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196025+0000", 2024-09-18T07:48:33.692 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013774600000000001 2024-09-18T07:48:33.692 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.692 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.692 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.692 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196569+0000", 2024-09-18T07:48:33.692 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00054400399999999995 2024-09-18T07:48:33.692 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.692 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.692 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.692 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196710+0000", 2024-09-18T07:48:33.692 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014022899999999999 2024-09-18T07:48:33.692 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.693 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.693 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.693 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196750+0000", 2024-09-18T07:48:33.693 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0842e-05 2024-09-18T07:48:33.693 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.693 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.693 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.693 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.693 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.693 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30928 11.7 11:e64e9d9c:::benchmark_data_smithi137_181069_object30927:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:48:33.693 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.268666+0000", 2024-09-18T07:48:33.693 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 60.333828261999997, 2024-09-18T07:48:33.694 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93909738700000001, 2024-09-18T07:48:33.694 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.694 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.694 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.694 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.694 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.694 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.694 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30928 2024-09-18T07:48:33.694 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.694 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.694 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.694 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.695 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.268666+0000", 2024-09-18T07:48:33.695 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.695 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.695 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.695 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.695 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.268666+0000", 2024-09-18T07:48:33.695 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.695 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.695 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.695 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.695 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.268668+0000", 2024-09-18T07:48:33.695 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3989999999999998e-06 2024-09-18T07:48:33.696 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.696 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.696 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.696 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.268722+0000", 2024-09-18T07:48:33.696 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4008000000000002e-05 2024-09-18T07:48:33.696 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.696 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.696 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.696 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.268733+0000", 2024-09-18T07:48:33.696 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1391e-05 2024-09-18T07:48:33.697 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.697 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.697 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.697 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.268739+0000", 2024-09-18T07:48:33.697 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.609e-06 2024-09-18T07:48:33.697 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.697 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.697 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.697 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.268799+0000", 2024-09-18T07:48:33.697 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9793000000000002e-05 2024-09-18T07:48:33.697 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.697 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.697 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.698 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.268806+0000", 2024-09-18T07:48:33.698 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1980000000000004e-06 2024-09-18T07:48:33.698 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.698 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.698 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.698 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196787+0000", 2024-09-18T07:48:33.698 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.927980574 2024-09-18T07:48:33.698 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.698 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.698 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.698 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196879+0000", 2024-09-18T07:48:33.698 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2306999999999996e-05 2024-09-18T07:48:33.698 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.699 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.699 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.699 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.197001+0000", 2024-09-18T07:48:33.699 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012252199999999999 2024-09-18T07:48:33.699 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.699 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.699 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.699 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.202913+0000", 2024-09-18T07:48:33.699 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0059114700000000003 2024-09-18T07:48:33.699 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.699 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.699 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.699 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.202960+0000", 2024-09-18T07:48:33.700 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6758000000000002e-05 2024-09-18T07:48:33.700 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.700 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.700 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.700 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207626+0000", 2024-09-18T07:48:33.700 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0046666570000000003 2024-09-18T07:48:33.700 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.700 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.700 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.700 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207715+0000", 2024-09-18T07:48:33.700 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8434000000000001e-05 2024-09-18T07:48:33.700 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.701 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.701 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.701 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207763+0000", 2024-09-18T07:48:33.701 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8266999999999997e-05 2024-09-18T07:48:33.701 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.701 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.702 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.702 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.702 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.702 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30932 11.7 11:ee7a1da2:::benchmark_data_smithi137_181069_object30931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:48:33.702 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.270974+0000", 2024-09-18T07:48:33.702 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 60.331519796000002, 2024-09-18T07:48:33.702 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93698104800000004, 2024-09-18T07:48:33.703 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.703 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.703 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.703 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.703 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.703 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.703 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30932 2024-09-18T07:48:33.704 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.704 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.704 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.704 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.704 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.270974+0000", 2024-09-18T07:48:33.704 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.704 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.705 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.705 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.705 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.270974+0000", 2024-09-18T07:48:33.705 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.705 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.705 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.705 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.705 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.270976+0000", 2024-09-18T07:48:33.706 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9750000000000001e-06 2024-09-18T07:48:33.706 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.706 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.706 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.706 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271032+0000", 2024-09-18T07:48:33.707 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5878000000000003e-05 2024-09-18T07:48:33.707 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.707 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.707 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.707 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271044+0000", 2024-09-18T07:48:33.707 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2408e-05 2024-09-18T07:48:33.708 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.708 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.708 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.708 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271052+0000", 2024-09-18T07:48:33.708 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6520000000000006e-06 2024-09-18T07:48:33.708 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.709 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.709 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.709 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271128+0000", 2024-09-18T07:48:33.709 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6322000000000006e-05 2024-09-18T07:48:33.709 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.709 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.709 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.709 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271135+0000", 2024-09-18T07:48:33.710 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5919999999999997e-06 2024-09-18T07:48:33.710 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.710 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.710 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.710 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.198221+0000", 2024-09-18T07:48:33.710 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92708616799999999 2024-09-18T07:48:33.710 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.710 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.710 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.710 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.198326+0000", 2024-09-18T07:48:33.710 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010516799999999999 2024-09-18T07:48:33.710 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.710 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.711 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.711 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.198447+0000", 2024-09-18T07:48:33.711 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012055799999999999 2024-09-18T07:48:33.711 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.711 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.711 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.711 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.202996+0000", 2024-09-18T07:48:33.711 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0045494050000000003 2024-09-18T07:48:33.711 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.711 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.711 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.711 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.203032+0000", 2024-09-18T07:48:33.711 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5299000000000002e-05 2024-09-18T07:48:33.711 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.712 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.712 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.712 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207882+0000", 2024-09-18T07:48:33.712 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0048507250000000002 2024-09-18T07:48:33.712 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.712 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.712 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.712 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207923+0000", 2024-09-18T07:48:33.712 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0599999999999998e-05 2024-09-18T07:48:33.712 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.712 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.712 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.712 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207955+0000", 2024-09-18T07:48:33.712 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2298000000000001e-05 2024-09-18T07:48:33.713 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.713 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.713 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.713 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.713 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.713 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30933 11.7 11:e9faa571:::benchmark_data_smithi137_181069_object30932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:48:33.713 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.271203+0000", 2024-09-18T07:48:33.713 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 60.331291067999999, 2024-09-18T07:48:33.713 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93684991900000003, 2024-09-18T07:48:33.713 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.713 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.713 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.713 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.714 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.714 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.714 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30933 2024-09-18T07:48:33.714 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.714 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.714 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.714 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.714 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271203+0000", 2024-09-18T07:48:33.714 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.714 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.715 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.715 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.715 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271203+0000", 2024-09-18T07:48:33.715 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.715 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.715 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.715 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.715 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271204+0000", 2024-09-18T07:48:33.715 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5999999999999999e-06 2024-09-18T07:48:33.715 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.715 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.715 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.715 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271254+0000", 2024-09-18T07:48:33.715 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9212999999999998e-05 2024-09-18T07:48:33.716 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.716 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.716 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.716 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271266+0000", 2024-09-18T07:48:33.716 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2571999999999999e-05 2024-09-18T07:48:33.716 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.716 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.716 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.716 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271274+0000", 2024-09-18T07:48:33.716 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5120000000000002e-06 2024-09-18T07:48:33.716 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.716 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.716 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.717 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271332+0000", 2024-09-18T07:48:33.717 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8199000000000003e-05 2024-09-18T07:48:33.717 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.717 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.717 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.717 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271342+0000", 2024-09-18T07:48:33.717 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7219999999999994e-06 2024-09-18T07:48:33.717 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.717 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.717 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.717 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.199551+0000", 2024-09-18T07:48:33.717 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.928209536 2024-09-18T07:48:33.717 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.718 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.718 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.718 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.199673+0000", 2024-09-18T07:48:33.718 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012221899999999999 2024-09-18T07:48:33.718 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.718 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.718 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.718 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.199806+0000", 2024-09-18T07:48:33.718 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000132328 2024-09-18T07:48:33.718 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.718 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.718 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.718 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.203066+0000", 2024-09-18T07:48:33.718 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003259942 2024-09-18T07:48:33.719 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.719 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.719 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.719 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.205648+0000", 2024-09-18T07:48:33.719 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025826790000000001 2024-09-18T07:48:33.719 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.719 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.719 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.719 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207965+0000", 2024-09-18T07:48:33.719 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023169990000000001 2024-09-18T07:48:33.719 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.719 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.719 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.719 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208011+0000", 2024-09-18T07:48:33.720 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5568000000000001e-05 2024-09-18T07:48:33.720 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.720 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.720 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.720 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208053+0000", 2024-09-18T07:48:33.720 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1829999999999998e-05 2024-09-18T07:48:33.720 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.720 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.720 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.720 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.720 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.721 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30958 11.7 11:eebed827:::benchmark_data_smithi137_181069_object30957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:48:33.721 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.295207+0000", 2024-09-18T07:48:33.721 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 60.307286761, 2024-09-18T07:48:33.721 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.91294648099999998, 2024-09-18T07:48:33.721 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.721 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.721 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.721 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.721 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.721 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.721 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30958 2024-09-18T07:48:33.721 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.721 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.722 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.722 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.722 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.295207+0000", 2024-09-18T07:48:33.722 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.722 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.722 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.722 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.722 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.295207+0000", 2024-09-18T07:48:33.722 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.722 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.722 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.722 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.722 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.295210+0000", 2024-09-18T07:48:33.722 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7999999999999999e-06 2024-09-18T07:48:33.722 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.723 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.723 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.723 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.295276+0000", 2024-09-18T07:48:33.723 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6266999999999994e-05 2024-09-18T07:48:33.723 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.723 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.723 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.723 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.295294+0000", 2024-09-18T07:48:33.723 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7739000000000001e-05 2024-09-18T07:48:33.723 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.723 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.723 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.723 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.295301+0000", 2024-09-18T07:48:33.723 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2830000000000004e-06 2024-09-18T07:48:33.723 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.724 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.724 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.724 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.295363+0000", 2024-09-18T07:48:33.724 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2084999999999999e-05 2024-09-18T07:48:33.724 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.724 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.724 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.724 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.295371+0000", 2024-09-18T07:48:33.724 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2590000000000003e-06 2024-09-18T07:48:33.724 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.724 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.724 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.724 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.201681+0000", 2024-09-18T07:48:33.724 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90630996799999997 2024-09-18T07:48:33.725 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.725 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.725 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.725 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.201805+0000", 2024-09-18T07:48:33.725 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000124253 2024-09-18T07:48:33.725 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.725 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.725 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.726 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.201961+0000", 2024-09-18T07:48:33.726 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015598000000000001 2024-09-18T07:48:33.726 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.726 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.726 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.726 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.205685+0000", 2024-09-18T07:48:33.726 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003724163 2024-09-18T07:48:33.726 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.726 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.726 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.728 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.205718+0000", 2024-09-18T07:48:33.728 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2991000000000003e-05 2024-09-18T07:48:33.728 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.728 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.728 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.728 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208061+0000", 2024-09-18T07:48:33.728 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002342645 2024-09-18T07:48:33.728 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.729 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.729 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.729 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208100+0000", 2024-09-18T07:48:33.729 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9376999999999999e-05 2024-09-18T07:48:33.729 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.729 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.729 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.729 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208154+0000", 2024-09-18T07:48:33.729 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3671000000000002e-05 2024-09-18T07:48:33.729 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.730 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.730 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.730 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.730 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.730 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30969 11.7 11:ef647aa1:::benchmark_data_smithi137_181069_object30968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:48:33.730 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.302772+0000", 2024-09-18T07:48:33.730 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 60.299722285999998, 2024-09-18T07:48:33.730 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90547923200000002, 2024-09-18T07:48:33.731 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.731 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.731 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.731 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.731 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.731 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.731 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30969 2024-09-18T07:48:33.732 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.732 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.732 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.732 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.732 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.302772+0000", 2024-09-18T07:48:33.732 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.733 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.733 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.733 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.733 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.302772+0000", 2024-09-18T07:48:33.733 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.734 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.734 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.734 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.734 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.302774+0000", 2024-09-18T07:48:33.734 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8810000000000001e-06 2024-09-18T07:48:33.734 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.735 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.735 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.735 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.302847+0000", 2024-09-18T07:48:33.735 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3746999999999997e-05 2024-09-18T07:48:33.736 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.736 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.736 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.736 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.302859+0000", 2024-09-18T07:48:33.736 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1977000000000001e-05 2024-09-18T07:48:33.737 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.737 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.737 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.737 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.302869+0000", 2024-09-18T07:48:33.737 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2909999999999996e-06 2024-09-18T07:48:33.737 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.737 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.737 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.738 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.302926+0000", 2024-09-18T07:48:33.738 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7362999999999999e-05 2024-09-18T07:48:33.738 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.738 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.738 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.738 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.302933+0000", 2024-09-18T07:48:33.738 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7209999999999997e-06 2024-09-18T07:48:33.738 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.739 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.739 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.739 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.203094+0000", 2024-09-18T07:48:33.739 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90016099900000002 2024-09-18T07:48:33.739 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.739 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.739 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.739 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.203200+0000", 2024-09-18T07:48:33.739 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000106183 2024-09-18T07:48:33.739 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.739 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.739 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.739 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.203325+0000", 2024-09-18T07:48:33.740 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012539300000000001 2024-09-18T07:48:33.740 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.740 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.740 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.740 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.205751+0000", 2024-09-18T07:48:33.740 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0024260729999999999 2024-09-18T07:48:33.740 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.740 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.740 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.740 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207814+0000", 2024-09-18T07:48:33.741 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0020626189999999999 2024-09-18T07:48:33.741 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.741 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.741 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.741 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208163+0000", 2024-09-18T07:48:33.741 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00034904800000000002 2024-09-18T07:48:33.741 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.741 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.741 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.741 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208215+0000", 2024-09-18T07:48:33.741 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2516000000000003e-05 2024-09-18T07:48:33.741 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.741 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.742 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.742 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208251+0000", 2024-09-18T07:48:33.742 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5420999999999998e-05 2024-09-18T07:48:33.742 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.742 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.742 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.742 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.742 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.742 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30981 11.7 11:e0ad3924:::benchmark_data_smithi137_181069_object30980:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:48:33.742 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.318119+0000", 2024-09-18T07:48:33.742 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 60.284375187000002, 2024-09-18T07:48:33.742 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.89035505999999998, 2024-09-18T07:48:33.743 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.743 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.743 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.743 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.743 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.743 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.743 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30981 2024-09-18T07:48:33.743 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.743 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.743 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.743 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:33.744 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:33 smithi137 ceph-mon[29408]: pgmap v942: 158 pgs: 2 unknown, 38 peering, 1 active+clean+scrubbing, 117 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 219 B/s rd, 0 op/s 2024-09-18T07:48:33.744 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:33 smithi137 ceph-mon[29408]: Health check failed: Reduced data availability: 7 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-18T07:48:33.745 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:33 smithi137 ceph-mon[29408]: osdmap e500: 8 total, 8 up, 7 in 2024-09-18T07:48:33.745 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:33 smithi137 ceph-mon[29408]: 3.a scrub starts 2024-09-18T07:48:33.745 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:33 smithi137 ceph-mon[29408]: 3.a scrub ok 2024-09-18T07:48:33.745 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:33 smithi137 ceph-mon[29408]: 3.15 scrub starts 2024-09-18T07:48:33.745 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:33 smithi137 ceph-mon[29408]: 3.15 scrub ok 2024-09-18T07:48:33.745 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:33 smithi137 ceph-mon[39175]: pgmap v942: 158 pgs: 2 unknown, 38 peering, 1 active+clean+scrubbing, 117 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 219 B/s rd, 0 op/s 2024-09-18T07:48:33.745 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:33 smithi137 ceph-mon[39175]: Health check failed: Reduced data availability: 7 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-18T07:48:33.745 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:33 smithi137 ceph-mon[39175]: osdmap e500: 8 total, 8 up, 7 in 2024-09-18T07:48:33.745 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:33 smithi137 ceph-mon[39175]: 3.a scrub starts 2024-09-18T07:48:33.745 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:33 smithi137 ceph-mon[39175]: 3.a scrub ok 2024-09-18T07:48:33.746 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:33 smithi137 ceph-mon[39175]: 3.15 scrub starts 2024-09-18T07:48:33.746 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:33 smithi137 ceph-mon[39175]: 3.15 scrub ok 2024-09-18T07:48:33.746 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.318119+0000", 2024-09-18T07:48:33.746 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.746 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.746 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.746 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.746 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.318119+0000", 2024-09-18T07:48:33.746 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.747 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.747 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.747 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.747 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.318121+0000", 2024-09-18T07:48:33.747 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3300000000000001e-06 2024-09-18T07:48:33.747 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.747 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.747 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.747 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.318183+0000", 2024-09-18T07:48:33.748 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2277e-05 2024-09-18T07:48:33.748 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.748 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.748 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.748 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.318199+0000", 2024-09-18T07:48:33.748 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5617e-05 2024-09-18T07:48:33.748 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.748 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.748 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.748 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.318207+0000", 2024-09-18T07:48:33.748 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2239999999999995e-06 2024-09-18T07:48:33.748 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.749 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.749 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.749 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.318269+0000", 2024-09-18T07:48:33.749 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2193999999999994e-05 2024-09-18T07:48:33.749 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.749 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.749 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.749 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.318278+0000", 2024-09-18T07:48:33.750 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1920000000000005e-06 2024-09-18T07:48:33.750 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.750 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.750 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.750 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.204376+0000", 2024-09-18T07:48:33.750 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.88609823200000004 2024-09-18T07:48:33.750 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.750 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.750 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.750 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.204511+0000", 2024-09-18T07:48:33.750 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013500400000000001 2024-09-18T07:48:33.751 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.751 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.751 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.751 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.204639+0000", 2024-09-18T07:48:33.751 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012823399999999999 2024-09-18T07:48:33.751 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.751 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.751 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.751 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207867+0000", 2024-09-18T07:48:33.752 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00322829 2024-09-18T07:48:33.752 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.752 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.752 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.752 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208259+0000", 2024-09-18T07:48:33.752 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00039183000000000002 2024-09-18T07:48:33.752 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.752 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.752 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.753 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208372+0000", 2024-09-18T07:48:33.753 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000113037 2024-09-18T07:48:33.753 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.753 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.753 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.753 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208428+0000", 2024-09-18T07:48:33.754 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6072000000000001e-05 2024-09-18T07:48:33.754 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.754 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.754 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.754 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208474+0000", 2024-09-18T07:48:33.755 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5527e-05 2024-09-18T07:48:33.755 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.755 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.755 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.755 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.755 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.755 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:31065 11.7 11:ef02e3d5:::benchmark_data_smithi137_181069_object31064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:48:33.756 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.402987+0000", 2024-09-18T07:48:33.756 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 60.199507259000001, 2024-09-18T07:48:33.756 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.80566682499999998, 2024-09-18T07:48:33.756 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.756 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.756 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.757 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.757 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.757 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.757 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 31065 2024-09-18T07:48:33.757 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.758 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.758 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.758 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.758 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.402987+0000", 2024-09-18T07:48:33.758 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.758 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.758 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.758 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.759 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.402987+0000", 2024-09-18T07:48:33.759 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.759 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.759 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.759 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.759 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.402989+0000", 2024-09-18T07:48:33.759 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7089999999999998e-06 2024-09-18T07:48:33.759 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.759 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.759 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.759 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.403044+0000", 2024-09-18T07:48:33.760 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4138999999999998e-05 2024-09-18T07:48:33.760 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.760 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.760 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.760 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.403060+0000", 2024-09-18T07:48:33.760 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6189e-05 2024-09-18T07:48:33.760 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.760 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.760 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.760 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.403069+0000", 2024-09-18T07:48:33.760 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7589999999999992e-06 2024-09-18T07:48:33.760 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.761 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.761 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.761 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.403136+0000", 2024-09-18T07:48:33.761 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6792999999999996e-05 2024-09-18T07:48:33.761 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.761 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.761 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.761 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.403144+0000", 2024-09-18T07:48:33.761 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2330000000000006e-06 2024-09-18T07:48:33.761 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.761 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.761 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.761 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.205783+0000", 2024-09-18T07:48:33.762 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.80263816499999996 2024-09-18T07:48:33.762 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.762 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.762 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.762 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.205876+0000", 2024-09-18T07:48:33.762 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3547000000000004e-05 2024-09-18T07:48:33.762 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.762 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.762 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.763 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.206004+0000", 2024-09-18T07:48:33.763 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012819099999999999 2024-09-18T07:48:33.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.763 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.763 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.763 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208264+0000", 2024-09-18T07:48:33.763 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022599540000000002 2024-09-18T07:48:33.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.763 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.763 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.764 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208523+0000", 2024-09-18T07:48:33.764 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025910100000000003 2024-09-18T07:48:33.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.764 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.764 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.764 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208563+0000", 2024-09-18T07:48:33.764 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9569e-05 2024-09-18T07:48:33.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.764 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.764 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.764 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208618+0000", 2024-09-18T07:48:33.764 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4991000000000003e-05 2024-09-18T07:48:33.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.765 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.765 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.765 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208654+0000", 2024-09-18T07:48:33.765 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5485000000000003e-05 2024-09-18T07:48:33.765 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.765 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.765 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.765 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.765 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:31081 11.7 11:eb7b016c:::benchmark_data_smithi137_181069_object31080:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:48:33.765 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.429960+0000", 2024-09-18T07:48:33.766 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 60.172534405999997, 2024-09-18T07:48:33.766 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.77885249999999995, 2024-09-18T07:48:33.766 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.766 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.766 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.766 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.766 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.766 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.766 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 31081 2024-09-18T07:48:33.766 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.766 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.766 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.766 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.766 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.429960+0000", 2024-09-18T07:48:33.767 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.767 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.767 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.767 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.429960+0000", 2024-09-18T07:48:33.767 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.767 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.767 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.767 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.429962+0000", 2024-09-18T07:48:33.767 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8030000000000001e-06 2024-09-18T07:48:33.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.767 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.768 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.768 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.429998+0000", 2024-09-18T07:48:33.768 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5843999999999998e-05 2024-09-18T07:48:33.768 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.768 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.768 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.768 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.430014+0000", 2024-09-18T07:48:33.768 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5931e-05 2024-09-18T07:48:33.768 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.768 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.768 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.768 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.430023+0000", 2024-09-18T07:48:33.768 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2560000000000008e-06 2024-09-18T07:48:33.769 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.769 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.769 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.769 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.430099+0000", 2024-09-18T07:48:33.769 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5411e-05 2024-09-18T07:48:33.769 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.769 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.769 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.769 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.430111+0000", 2024-09-18T07:48:33.769 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2136e-05 2024-09-18T07:48:33.769 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.769 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.770 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.770 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.206697+0000", 2024-09-18T07:48:33.770 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.77658628200000002 2024-09-18T07:48:33.770 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.770 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.770 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.770 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.206788+0000", 2024-09-18T07:48:33.770 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0576999999999997e-05 2024-09-18T07:48:33.770 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.770 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.770 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.770 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.206898+0000", 2024-09-18T07:48:33.770 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000110671 2024-09-18T07:48:33.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.771 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.771 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.771 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208270+0000", 2024-09-18T07:48:33.771 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0013715209999999999 2024-09-18T07:48:33.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.771 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.771 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.771 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208697+0000", 2024-09-18T07:48:33.771 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00042693199999999998 2024-09-18T07:48:33.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.771 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.772 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.772 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208736+0000", 2024-09-18T07:48:33.772 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9561000000000003e-05 2024-09-18T07:48:33.772 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.772 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.772 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.772 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208781+0000", 2024-09-18T07:48:33.772 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4603e-05 2024-09-18T07:48:33.772 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.772 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.772 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.772 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208812+0000", 2024-09-18T07:48:33.773 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0972000000000001e-05 2024-09-18T07:48:33.773 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.773 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.773 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.773 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.773 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.773 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:31123 11.7 11:eff82b5a:::benchmark_data_smithi137_181069_object31122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:48:33.773 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.538919+0000", 2024-09-18T07:48:33.773 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 60.063574756999998, 2024-09-18T07:48:33.773 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.67483459700000004, 2024-09-18T07:48:33.773 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:48:33.773 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:48:33.773 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:33.774 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:48:33.774 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:48:33.774 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:33.774 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 31123 2024-09-18T07:48:33.774 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.774 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:48:33.774 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.774 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:48:33.774 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.538919+0000", 2024-09-18T07:48:33.774 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.774 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.774 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.774 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:48:33.775 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.538919+0000", 2024-09-18T07:48:33.775 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:48:33.775 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.775 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.775 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:48:33.775 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.538921+0000", 2024-09-18T07:48:33.775 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8109999999999999e-06 2024-09-18T07:48:33.775 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.775 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.775 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:48:33.775 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.538978+0000", 2024-09-18T07:48:33.775 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6709999999999997e-05 2024-09-18T07:48:33.775 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.776 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.776 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:48:33.776 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.538988+0000", 2024-09-18T07:48:33.776 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0482000000000001e-05 2024-09-18T07:48:33.776 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.776 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.776 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:48:33.776 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.538994+0000", 2024-09-18T07:48:33.776 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7649999999999999e-06 2024-09-18T07:48:33.776 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.777 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.777 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.777 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.539042+0000", 2024-09-18T07:48:33.777 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8461000000000002e-05 2024-09-18T07:48:33.777 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.777 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.778 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:48:33.778 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.539049+0000", 2024-09-18T07:48:33.778 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5010000000000001e-06 2024-09-18T07:48:33.778 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.778 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.778 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:48:33.779 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208929+0000", 2024-09-18T07:48:33.779 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.66988051199999998 2024-09-18T07:48:33.779 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.779 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.779 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:48:33.779 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.209047+0000", 2024-09-18T07:48:33.780 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011739399999999999 2024-09-18T07:48:33.780 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.780 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.780 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:33.780 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.209164+0000", 2024-09-18T07:48:33.780 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000117126 2024-09-18T07:48:33.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.781 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.781 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.781 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.211312+0000", 2024-09-18T07:48:33.781 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021483320000000002 2024-09-18T07:48:33.781 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.782 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.782 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:48:33.782 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.212733+0000", 2024-09-18T07:48:33.782 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014205509999999999 2024-09-18T07:48:33.782 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.782 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.783 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:33.783 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.213640+0000", 2024-09-18T07:48:33.783 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00090767599999999995 2024-09-18T07:48:33.783 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.783 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.783 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:48:33.784 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.213701+0000", 2024-09-18T07:48:33.784 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0096999999999997e-05 2024-09-18T07:48:33.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:48:33.784 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:48:33.784 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:48:33.785 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.213754+0000", 2024-09-18T07:48:33.785 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3179e-05 2024-09-18T07:48:33.785 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.785 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.785 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.785 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:48:33.786 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:48:33.786 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:48:33.845 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:33.880 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:33.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:33.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:33.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:34.022 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:34.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:34.063 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 79 16 101210 101194 80.0378 13.1875 0.0238791 0.0124897 2024-09-18T07:48:34.124 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:34.163 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:34.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:34.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:34.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:34.304 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:34.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:34.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:34.406 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:34.454 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:34.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:34.507 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:34 smithi203 ceph-mon[32237]: 1.4 scrub starts 2024-09-18T07:48:34.507 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:34 smithi203 ceph-mon[32237]: 1.4 scrub ok 2024-09-18T07:48:34.507 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:34 smithi203 ceph-mon[32237]: osdmap e501: 8 total, 8 up, 7 in 2024-09-18T07:48:34.507 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:34 smithi203 ceph-mon[32237]: 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 498'6290 2024-09-18T07:48:34.507 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:34 smithi203 ceph-mon[32237]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 498'6249 2024-09-18T07:48:34.507 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:34 smithi203 ceph-mon[32237]: 11.a starting backfill to osd.2 from (0'0,0'0] MAX to 498'6247 2024-09-18T07:48:34.507 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:34 smithi203 ceph-mon[32237]: 11.0 starting backfill to osd.0 from (0'0,0'0] MAX to 498'6436 2024-09-18T07:48:34.507 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:34 smithi203 ceph-mon[32237]: 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 498'6356 2024-09-18T07:48:34.507 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:34 smithi203 ceph-mon[32237]: 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 498'6141 2024-09-18T07:48:34.508 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:34 smithi203 ceph-mon[32237]: 11.b starting backfill to osd.4 from (0'0,0'0] MAX to 498'6236 2024-09-18T07:48:34.508 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:34 smithi203 ceph-mon[32237]: 6.13 scrub starts 2024-09-18T07:48:34.508 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:34 smithi203 ceph-mon[32237]: 6.13 scrub ok 2024-09-18T07:48:34.508 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:34 smithi203 ceph-mon[32237]: 3.f scrub starts 2024-09-18T07:48:34.508 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:34 smithi203 ceph-mon[32237]: 3.f scrub ok 2024-09-18T07:48:34.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:34.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T07:48:34.582 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[29408]: 1.4 scrub starts 2024-09-18T07:48:34.582 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[29408]: 1.4 scrub ok 2024-09-18T07:48:34.582 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[29408]: osdmap e501: 8 total, 8 up, 7 in 2024-09-18T07:48:34.582 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[29408]: 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 498'6290 2024-09-18T07:48:34.583 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[29408]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 498'6249 2024-09-18T07:48:34.583 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[29408]: 11.a starting backfill to osd.2 from (0'0,0'0] MAX to 498'6247 2024-09-18T07:48:34.583 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[29408]: 11.0 starting backfill to osd.0 from (0'0,0'0] MAX to 498'6436 2024-09-18T07:48:34.583 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[29408]: 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 498'6356 2024-09-18T07:48:34.583 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[29408]: 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 498'6141 2024-09-18T07:48:34.583 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[29408]: 11.b starting backfill to osd.4 from (0'0,0'0] MAX to 498'6236 2024-09-18T07:48:34.583 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[29408]: 6.13 scrub starts 2024-09-18T07:48:34.583 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[29408]: 6.13 scrub ok 2024-09-18T07:48:34.583 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[29408]: 3.f scrub starts 2024-09-18T07:48:34.583 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[29408]: 3.f scrub ok 2024-09-18T07:48:34.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[39175]: 1.4 scrub starts 2024-09-18T07:48:34.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[39175]: 1.4 scrub ok 2024-09-18T07:48:34.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[39175]: osdmap e501: 8 total, 8 up, 7 in 2024-09-18T07:48:34.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[39175]: 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 498'6290 2024-09-18T07:48:34.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[39175]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 498'6249 2024-09-18T07:48:34.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[39175]: 11.a starting backfill to osd.2 from (0'0,0'0] MAX to 498'6247 2024-09-18T07:48:34.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[39175]: 11.0 starting backfill to osd.0 from (0'0,0'0] MAX to 498'6436 2024-09-18T07:48:34.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[39175]: 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 498'6356 2024-09-18T07:48:34.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[39175]: 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 498'6141 2024-09-18T07:48:34.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[39175]: 11.b starting backfill to osd.4 from (0'0,0'0] MAX to 498'6236 2024-09-18T07:48:34.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[39175]: 6.13 scrub starts 2024-09-18T07:48:34.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[39175]: 6.13 scrub ok 2024-09-18T07:48:34.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[39175]: 3.f scrub starts 2024-09-18T07:48:34.585 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:34 smithi137 ceph-mon[39175]: 3.f scrub ok 2024-09-18T07:48:34.596 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:34.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:34.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:34.697 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:34.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:34.727 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:34.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:34.828 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:34.869 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:34.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:34.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:34.972 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:35.003 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:35.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:35.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:35.063 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:2024-09-18T07:48:35.061639+0000 min lat: 0.00129363 max lat: 2.42906 avg lat: 0.0125273 2024-09-18T07:48:35.063 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T07:48:35.063 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 80 16 102174 102158 79.7904 60.25 0.0209964 0.0125273 2024-09-18T07:48:35.105 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:35.142 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:35.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:35.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:35.244 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:35.271 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:35.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:35.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:35.281 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:48:35.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:35.428 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:35.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:35.529 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:35.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:35.560 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:35.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:35.661 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:35.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:35 smithi137 ceph-mon[29408]: 4.2 scrub starts 2024-09-18T07:48:35.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:35 smithi137 ceph-mon[29408]: 4.2 scrub ok 2024-09-18T07:48:35.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:35 smithi137 ceph-mon[29408]: pgmap v945: 158 pgs: 1 remapped+peering, 2 unknown, 47 peering, 108 active+clean; 6.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-18T07:48:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:35 smithi137 ceph-mon[29408]: 3.1d scrub starts 2024-09-18T07:48:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:35 smithi137 ceph-mon[29408]: 3.1d scrub ok 2024-09-18T07:48:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:35 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4204041322' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:48:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:35 smithi137 ceph-mon[39175]: 4.2 scrub starts 2024-09-18T07:48:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:35 smithi137 ceph-mon[39175]: 4.2 scrub ok 2024-09-18T07:48:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:35 smithi137 ceph-mon[39175]: pgmap v945: 158 pgs: 1 remapped+peering, 2 unknown, 47 peering, 108 active+clean; 6.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-18T07:48:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:35 smithi137 ceph-mon[39175]: 3.1d scrub starts 2024-09-18T07:48:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:35 smithi137 ceph-mon[39175]: 3.1d scrub ok 2024-09-18T07:48:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:35 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4204041322' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:48:35.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:35.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:35.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:35.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:35 smithi203 ceph-mon[32237]: 4.2 scrub starts 2024-09-18T07:48:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:35 smithi203 ceph-mon[32237]: 4.2 scrub ok 2024-09-18T07:48:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:35 smithi203 ceph-mon[32237]: pgmap v945: 158 pgs: 1 remapped+peering, 2 unknown, 47 peering, 108 active+clean; 6.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-18T07:48:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:35 smithi203 ceph-mon[32237]: 3.1d scrub starts 2024-09-18T07:48:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:35 smithi203 ceph-mon[32237]: 3.1d scrub ok 2024-09-18T07:48:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:35 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4204041322' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:48:35.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:35.838 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:35.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:35.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:35.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:35.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:35.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:35.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:36.063 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 81 16 103133 103117 79.5453 59.9375 0.00972504 0.0125668 2024-09-18T07:48:36.080 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:36.108 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:36.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:36.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:36.183 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:36.210 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:36.254 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:36.264 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:36 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:36.264 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:36 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4204041322' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:48:36.264 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:36 smithi137 ceph-mon[29408]: osdmap e502: 8 total, 8 up, 7 in 2024-09-18T07:48:36.265 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:36 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:36.265 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:36 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4204041322' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:48:36.265 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:36 smithi137 ceph-mon[39175]: osdmap e502: 8 total, 8 up, 7 in 2024-09-18T07:48:36.269 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:36.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:36.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:36.372 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:36.414 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:36.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:36.516 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:36.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:36.560 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:36.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:36.661 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:36.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:36.693 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:36.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:36.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:36 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:36 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4204041322' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:48:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:36 smithi203 ceph-mon[32237]: osdmap e502: 8 total, 8 up, 7 in 2024-09-18T07:48:36.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:36.851 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:36.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:36.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:36.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:36.988 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:36.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:36.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:37.063 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 82 15 104121 104106 79.329 61.8125 0.0135932 0.0126001 2024-09-18T07:48:37.089 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:37.118 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:37.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:37.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:37.136 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:48:37.136 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:48:37.203 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:48:37.203 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:48:37.203 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:48:37.203 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:48:37.204 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:48:37.204 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:48:37.210 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:48:37.211 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:48:37.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:37.249 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:48:37.250 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:48:37.263 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:37.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:37.281 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:48:37.281 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:48:37.315 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:48:37.315 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:48:37.315 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:48:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:48:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:48:37.316 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:37 smithi137 ceph-mon[29408]: pgmap v947: 158 pgs: 1 remapped+peering, 2 unknown, 47 peering, 108 active+clean; 6.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 962 op/s; 55 KiB/s, 0 objects/s recovering 2024-09-18T07:48:37.316 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:37 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3899467855' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:37.317 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:37 smithi137 ceph-mon[39175]: pgmap v947: 158 pgs: 1 remapped+peering, 2 unknown, 47 peering, 108 active+clean; 6.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 962 op/s; 55 KiB/s, 0 objects/s recovering 2024-09-18T07:48:37.317 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:37 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3899467855' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:37.348 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:48:37.349 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:48:37.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:37.408 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:37.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:37.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:37.510 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:37.550 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:37.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:37.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:37.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:37.683 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:37.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:37.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:37.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:37 smithi203 ceph-mon[32237]: pgmap v947: 158 pgs: 1 remapped+peering, 2 unknown, 47 peering, 108 active+clean; 6.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 962 op/s; 55 KiB/s, 0 objects/s recovering 2024-09-18T07:48:37.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:37 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3899467855' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:37.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:37.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:37.822 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:37.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:37.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:37.965 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:37.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:37.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:38.063 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 83 16 105207 105191 79.1902 67.8125 0.00493054 0.0126229 2024-09-18T07:48:38.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:38.097 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:38.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:38.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:38.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:48:38.199 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:38.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:48:37 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:48:38.259 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:38.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:38.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:38.361 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:38.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:38.392 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:38.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:38.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:38 smithi137 ceph-mon[29408]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:48:38.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:38 smithi137 ceph-mon[29408]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 15 pgs peering) 2024-09-18T07:48:38.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:38 smithi137 ceph-mon[39175]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:48:38.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:38 smithi137 ceph-mon[39175]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 15 pgs peering) 2024-09-18T07:48:38.493 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:38.524 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:38.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:38.625 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:38.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:38.663 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:38.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:38.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:38 smithi203 ceph-mon[32237]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:48:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:38 smithi203 ceph-mon[32237]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 15 pgs peering) 2024-09-18T07:48:38.764 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:38.795 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:38.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:38.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:38.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T07:48:38.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:38.967 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:38.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:38.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:38.974 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:39.064 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 84 16 106203 106187 78.9879 62.25 0.0286259 0.0126542 2024-09-18T07:48:39.068 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:39.098 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:39.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:39.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:39.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:39.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:39.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:39.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:39.350 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:39.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:39.382 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:39.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:39.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:39 smithi137 ceph-mon[39175]: pgmap v948: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 remapped+peering, 2 active+remapped+backfill_toofull, 151 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s; 37544/311094 objects misplaced (12.068%); 23 MiB/s, 379 objects/s recovering 2024-09-18T07:48:39.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:39 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:39 smithi137 ceph-mon[29408]: pgmap v948: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 remapped+peering, 2 active+remapped+backfill_toofull, 151 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s; 37544/311094 objects misplaced (12.068%); 23 MiB/s, 379 objects/s recovering 2024-09-18T07:48:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:39 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:39.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:39.518 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:39.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:39.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:39.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:39.660 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:39.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:39.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:39.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:39 smithi203 ceph-mon[32237]: pgmap v948: 158 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 remapped+peering, 2 active+remapped+backfill_toofull, 151 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s; 37544/311094 objects misplaced (12.068%); 23 MiB/s, 379 objects/s recovering 2024-09-18T07:48:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:39 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:39.762 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:39.797 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:39.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:39.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:39.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:39.942 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:39.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:39.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:40.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:40.064 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 85 16 107295 107279 78.8614 68.25 0.00697392 0.0126753 2024-09-18T07:48:40.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:40.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:40.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:40.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:40.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:40.228 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:40.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:40.236 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 4, 5, 7, 6, 1] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 6, 7, 4, 5] 2024-09-18T07:48:40.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-18T07:48:40.256 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:48:40.330 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:40.358 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:40.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:40.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:40.460 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:40.485 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:40.488 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:40.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:40.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:40.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:40.671 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:40.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:40.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:40.773 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:40.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:40.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:40.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:40.923 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:40.952 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:40.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:41.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:41.064 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 86 16 108138 108122 78.557 52.6875 0.0159831 0.0127241 2024-09-18T07:48:41.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:41.107 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:41.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:41.192 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:48:41.192 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:48:41.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:41.257 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:41.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:41.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:41.359 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:41.364 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:48:41.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:41.395 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:41.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:41.496 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:41.530 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:41.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:41.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:41.632 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:41.663 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:41.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:41.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:41.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:41 smithi137 ceph-mon[39175]: pgmap v949: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 151 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s; 43501/312918 objects misplaced (13.902%); 23 MiB/s, 373 objects/s recovering 2024-09-18T07:48:41.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:41 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1817568082' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:48:41.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:41 smithi137 ceph-mon[39175]: from='client.26677 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:48:41.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:41 smithi137 ceph-mon[29408]: pgmap v949: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 151 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s; 43501/312918 objects misplaced (13.902%); 23 MiB/s, 373 objects/s recovering 2024-09-18T07:48:41.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:41 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1817568082' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:48:41.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:41 smithi137 ceph-mon[29408]: from='client.26677 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:48:41.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:41 smithi203 ceph-mon[32237]: pgmap v949: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 151 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s; 43501/312918 objects misplaced (13.902%); 23 MiB/s, 373 objects/s recovering 2024-09-18T07:48:41.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:41 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1817568082' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:48:41.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:41 smithi203 ceph-mon[32237]: from='client.26677 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:48:41.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:41.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:41.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:41.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:41.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:41.954 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:41.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:42.055 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:42.064 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 87 16 109194 109178 78.4126 66 0.00928257 0.0127487 2024-09-18T07:48:42.092 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:42.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:42.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:42.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:42.225 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:42.228 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:42.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:42.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:42.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:42.350 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:42.379 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:42.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:42.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:42.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:42 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:42 smithi137 ceph-mon[29408]: from='client.26677 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:48:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:42 smithi137 ceph-mon[29408]: osdmap e503: 8 total, 8 up, 7 in 2024-09-18T07:48:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:42 smithi137 ceph-mon[29408]: 1.3 scrub starts 2024-09-18T07:48:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:42 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:48:42.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:42 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:48:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:42 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:42 smithi137 ceph-mon[39175]: from='client.26677 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:48:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:42 smithi137 ceph-mon[39175]: osdmap e503: 8 total, 8 up, 7 in 2024-09-18T07:48:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:42 smithi137 ceph-mon[39175]: 1.3 scrub starts 2024-09-18T07:48:42.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:42 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:48:42.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:42 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:48:42.481 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:42.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:42.513 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:42.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:42.614 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:42.645 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:42.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:42.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:42 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:42 smithi203 ceph-mon[32237]: from='client.26677 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:48:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:42 smithi203 ceph-mon[32237]: osdmap e503: 8 total, 8 up, 7 in 2024-09-18T07:48:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:42 smithi203 ceph-mon[32237]: 1.3 scrub starts 2024-09-18T07:48:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:42 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:48:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:42 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:48:42.727 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T07:48:42.749 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:42.780 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:42.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:42.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:42.882 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:42.914 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:42.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:42.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:43.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:43.066 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 88 16 110199 110183 78.2346 62.8125 0.0165014 0.012776 2024-09-18T07:48:43.103 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:43.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:43.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:43.205 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:43.241 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:43.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:43.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:43.343 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:43.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:43.379 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:43.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:43.481 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:43.512 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:43.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:43.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:43.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:43 smithi137 ceph-mon[29408]: pgmap v951: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 151 active+clean; 6.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 783 op/s; 44126/321366 objects misplaced (13.731%); 15 MiB/s, 249 objects/s recovering 2024-09-18T07:48:43.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:43 smithi137 ceph-mon[39175]: pgmap v951: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 151 active+clean; 6.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 783 op/s; 44126/321366 objects misplaced (13.731%); 15 MiB/s, 249 objects/s recovering 2024-09-18T07:48:43.613 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:43.664 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:43.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:43.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:43.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:43 smithi203 ceph-mon[32237]: pgmap v951: 158 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 151 active+clean; 6.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 783 op/s; 44126/321366 objects misplaced (13.731%); 15 MiB/s, 249 objects/s recovering 2024-09-18T07:48:43.766 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:43.797 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:43.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:43.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:43.845 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:43.867 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:48:43.867 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:48:43.899 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:43.919 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:48:43.919 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:48:43.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:43.929 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:43.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:43.957 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:48:43.957 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:48:43.992 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:48:43.992 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:48:44.023 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:48:44.024 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:48:44.030 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:44.057 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:44.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:44.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:44.060 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:48:44.060 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:48:44.065 INFO:tasks.radosbench.radosbench.0.smithi137.stdout: 89 16 111157 111141 78.0282 59.875 0.0160544 0.0128106 2024-09-18T07:48:44.091 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:48:44.091 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:48:44.125 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:48:44.126 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:48:44.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:44.195 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:48:44.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:44.200 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:44.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:44.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:44.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:44.346 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:44.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:44.362 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.0 to 1 (10000) 2024-09-18T07:48:44.385 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:44 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1664669490' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:44.385 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:44 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1953812712' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:48:44.385 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1664669490' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:44.385 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1953812712' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:48:44.448 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:44.486 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:44.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:44.587 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:44.638 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:44.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:44.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:44 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1664669490' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:48:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:44 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1953812712' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:48:44.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:44.782 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:44.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:44.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:44.917 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:44.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:44.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:45.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:45.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:45.055 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:45.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:45.065 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Total time run: 90.0082 2024-09-18T07:48:45.065 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Total writes made: 112151 2024-09-18T07:48:45.066 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Write size: 65536 2024-09-18T07:48:45.066 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Object size: 65536 2024-09-18T07:48:45.066 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Bandwidth (MB/sec): 77.8755 2024-09-18T07:48:45.066 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Stddev Bandwidth: 39.1228 2024-09-18T07:48:45.066 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Max bandwidth (MB/sec): 233.562 2024-09-18T07:48:45.066 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Min bandwidth (MB/sec): 0 2024-09-18T07:48:45.066 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Average IOPS: 1246 2024-09-18T07:48:45.066 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Stddev IOPS: 625.969 2024-09-18T07:48:45.066 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Max IOPS: 3737 2024-09-18T07:48:45.066 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Min IOPS: 0 2024-09-18T07:48:45.066 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Average Latency(s): 0.0128367 2024-09-18T07:48:45.066 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Stddev Latency(s): 0.0467584 2024-09-18T07:48:45.066 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Max latency(s): 2.42906 2024-09-18T07:48:45.067 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Min latency(s): 0.00129363 2024-09-18T07:48:45.072 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Cleaning up (deleting benchmark objects) 2024-09-18T07:48:45.156 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:45.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:45.189 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:45.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:45.238 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:48:45.238 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:48:45.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:48:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:48:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137615 10.4 10:280114ba:::benchmark_data_smithi137_151160_object1799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-18T07:48:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:42.771430+0000", 2024-09-18T07:48:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 242.462227692, 2024-09-18T07:48:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.155652747, 2024-09-18T07:48:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:48:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:48:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:45.239 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137615 2024-09-18T07:48:45.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.240 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771430+0000", 2024-09-18T07:48:45.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771430+0000", 2024-09-18T07:48:45.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771432+0000", 2024-09-18T07:48:45.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6199999999999999e-06 2024-09-18T07:48:45.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771437+0000", 2024-09-18T07:48:45.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0159999999999999e-06 2024-09-18T07:48:45.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771445+0000", 2024-09-18T07:48:45.242 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.225e-06 2024-09-18T07:48:45.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771451+0000", 2024-09-18T07:48:45.242 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.908e-06 2024-09-18T07:48:45.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.243 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.772166+0000", 2024-09-18T07:48:45.243 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000714571 2024-09-18T07:48:45.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.243 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.243 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:45.243 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.772292+0000", 2024-09-18T07:48:45.243 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126322 2024-09-18T07:48:45.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.243 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.243 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-09-18T07:48:45.243 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.772478+0000", 2024-09-18T07:48:45.243 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000185747 2024-09-18T07:48:45.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:45.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.784266+0000", 2024-09-18T07:48:45.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011787824000000001 2024-09-18T07:48:45.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.795223+0000", 2024-09-18T07:48:45.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010956802999999999 2024-09-18T07:48:45.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.795236+0000", 2024-09-18T07:48:45.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3570999999999999e-05 2024-09-18T07:48:45.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774402+0000", 2024-09-18T07:48:45.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97916638700000003 2024-09-18T07:48:45.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774413+0000", 2024-09-18T07:48:45.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0035e-05 2024-09-18T07:48:45.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927028+0000", 2024-09-18T07:48:45.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.152615967 2024-09-18T07:48:45.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927083+0000", 2024-09-18T07:48:45.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4750999999999998e-05 2024-09-18T07:48:45.246 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.246 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.246 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.246 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137618 10.4 10:22103fc7:::benchmark_data_smithi137_151160_object1802:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-18T07:48:45.247 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:42.773989+0000", 2024-09-18T07:48:45.247 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 242.45966924300001, 2024-09-18T07:48:45.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1739552340000001, 2024-09-18T07:48:45.247 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.247 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.247 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:48:45.247 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.247 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:48:45.247 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:45.247 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137618 2024-09-18T07:48:45.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.248 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.773989+0000", 2024-09-18T07:48:45.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.773989+0000", 2024-09-18T07:48:45.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.773991+0000", 2024-09-18T07:48:45.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.657e-06 2024-09-18T07:48:45.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.773997+0000", 2024-09-18T07:48:45.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1789999999999996e-06 2024-09-18T07:48:45.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.774008+0000", 2024-09-18T07:48:45.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0937999999999999e-05 2024-09-18T07:48:45.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.774015+0000", 2024-09-18T07:48:45.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7789999999999996e-06 2024-09-18T07:48:45.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.779798+0000", 2024-09-18T07:48:45.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057823980000000002 2024-09-18T07:48:45.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:45.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.779927+0000", 2024-09-18T07:48:45.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012920100000000001 2024-09-18T07:48:45.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-09-18T07:48:45.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.780075+0000", 2024-09-18T07:48:45.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014824700000000001 2024-09-18T07:48:45.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:45.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.784857+0000", 2024-09-18T07:48:45.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047816070000000002 2024-09-18T07:48:45.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.834874+0000", 2024-09-18T07:48:45.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.050016892 2024-09-18T07:48:45.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.834885+0000", 2024-09-18T07:48:45.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1080999999999999e-05 2024-09-18T07:48:45.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774441+0000", 2024-09-18T07:48:45.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93955602400000005 2024-09-18T07:48:45.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774685+0000", 2024-09-18T07:48:45.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024376000000000001 2024-09-18T07:48:45.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927339+0000", 2024-09-18T07:48:45.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1526545050000001 2024-09-18T07:48:45.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-09-18T07:48:45.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927499+0000", 2024-09-18T07:48:45.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016008799999999999 2024-09-18T07:48:45.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.947852+0000", 2024-09-18T07:48:45.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020352456000000001 2024-09-18T07:48:45.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.947944+0000", 2024-09-18T07:48:45.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2422000000000004e-05 2024-09-18T07:48:45.255 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.255 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.255 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.255 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137629 10.4 10:238d70eb:::benchmark_data_smithi137_151160_object1813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-09-18T07:48:45.255 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:42.835110+0000", 2024-09-18T07:48:45.256 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 242.39854766799999, 2024-09-18T07:48:45.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.113021496, 2024-09-18T07:48:45.256 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.256 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.256 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:45.256 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.256 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:48:45.256 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:45.256 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137629 2024-09-18T07:48:45.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.256 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835110+0000", 2024-09-18T07:48:45.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835110+0000", 2024-09-18T07:48:45.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835113+0000", 2024-09-18T07:48:45.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1150000000000001e-06 2024-09-18T07:48:45.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835117+0000", 2024-09-18T07:48:45.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2359999999999998e-06 2024-09-18T07:48:45.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835124+0000", 2024-09-18T07:48:45.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3629999999999996e-06 2024-09-18T07:48:45.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835131+0000", 2024-09-18T07:48:45.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5899999999999996e-06 2024-09-18T07:48:45.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835166+0000", 2024-09-18T07:48:45.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5054e-05 2024-09-18T07:48:45.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835173+0000", 2024-09-18T07:48:45.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9759999999999998e-06 2024-09-18T07:48:45.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774713+0000", 2024-09-18T07:48:45.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93954058799999995 2024-09-18T07:48:45.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774722+0000", 2024-09-18T07:48:45.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5320000000000006e-06 2024-09-18T07:48:45.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927611+0000", 2024-09-18T07:48:45.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.152888763 2024-09-18T07:48:45.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:45.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927731+0000", 2024-09-18T07:48:45.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012040999999999999 2024-09-18T07:48:45.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:48:45.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927854+0000", 2024-09-18T07:48:45.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012266800000000001 2024-09-18T07:48:45.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:45.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.934460+0000", 2024-09-18T07:48:45.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066062439999999998 2024-09-18T07:48:45.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.947486+0000", 2024-09-18T07:48:45.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013025956999999999 2024-09-18T07:48:45.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.947988+0000", 2024-09-18T07:48:45.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050220899999999999 2024-09-18T07:48:45.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:45.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.948040+0000", 2024-09-18T07:48:45.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1462000000000003e-05 2024-09-18T07:48:45.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.948132+0000", 2024-09-18T07:48:45.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2329000000000003e-05 2024-09-18T07:48:45.264 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.264 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.264 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.264 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137617 10.1 10:8638502c:::benchmark_data_smithi137_151160_object1801:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e396)", 2024-09-18T07:48:45.264 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:43.767316+0000", 2024-09-18T07:48:45.264 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 241.46634166999999, 2024-09-18T07:48:45.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1348433360000001, 2024-09-18T07:48:45.265 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.265 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.265 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:48:45.265 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.265 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:48:45.265 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:45.265 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137617 2024-09-18T07:48:45.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.265 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.767316+0000", 2024-09-18T07:48:45.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.767316+0000", 2024-09-18T07:48:45.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.767318+0000", 2024-09-18T07:48:45.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.889e-06 2024-09-18T07:48:45.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.767324+0000", 2024-09-18T07:48:45.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8429999999999998e-06 2024-09-18T07:48:45.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.768695+0000", 2024-09-18T07:48:45.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001370549 2024-09-18T07:48:45.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.768709+0000", 2024-09-18T07:48:45.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.434e-05 2024-09-18T07:48:45.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.772764+0000", 2024-09-18T07:48:45.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040549729999999999 2024-09-18T07:48:45.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.772774+0000", 2024-09-18T07:48:45.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4499999999999993e-06 2024-09-18T07:48:45.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.901011+0000", 2024-09-18T07:48:45.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.128237484 2024-09-18T07:48:45.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.902160+0000", 2024-09-18T07:48:45.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011488080000000001 2024-09-18T07:48:45.269 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.269 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.269 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.270 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137632 10.1 10:839066b5:::benchmark_data_smithi137_151160_object1816:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e396)", 2024-09-18T07:48:45.270 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:43.769783+0000", 2024-09-18T07:48:45.270 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 241.46387564400001, 2024-09-18T07:48:45.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1401627699999999, 2024-09-18T07:48:45.270 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.270 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.270 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:45.270 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.270 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:48:45.270 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:45.270 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137632 2024-09-18T07:48:45.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.271 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769783+0000", 2024-09-18T07:48:45.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769783+0000", 2024-09-18T07:48:45.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769784+0000", 2024-09-18T07:48:45.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7439999999999999e-06 2024-09-18T07:48:45.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769791+0000", 2024-09-18T07:48:45.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6869999999999997e-06 2024-09-18T07:48:45.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769800+0000", 2024-09-18T07:48:45.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0129999999999999e-06 2024-09-18T07:48:45.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769808+0000", 2024-09-18T07:48:45.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7060000000000004e-06 2024-09-18T07:48:45.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.772796+0000", 2024-09-18T07:48:45.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002987899 2024-09-18T07:48:45.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.772802+0000", 2024-09-18T07:48:45.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9909999999999999e-06 2024-09-18T07:48:45.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.902389+0000", 2024-09-18T07:48:45.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.129587747 2024-09-18T07:48:45.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:45.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.902738+0000", 2024-09-18T07:48:45.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034822599999999998 2024-09-18T07:48:45.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4,7", 2024-09-18T07:48:45.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.902816+0000", 2024-09-18T07:48:45.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8430000000000006e-05 2024-09-18T07:48:45.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:45.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.904850+0000", 2024-09-18T07:48:45.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020342960000000001 2024-09-18T07:48:45.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.906005+0000", 2024-09-18T07:48:45.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011550899999999999 2024-09-18T07:48:45.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.909791+0000", 2024-09-18T07:48:45.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0037854500000000001 2024-09-18T07:48:45.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:45.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.909846+0000", 2024-09-18T07:48:45.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5565000000000001e-05 2024-09-18T07:48:45.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.909945+0000", 2024-09-18T07:48:45.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8925999999999994e-05 2024-09-18T07:48:45.281 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.282 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.282 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.282 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137633 10.2 10:44856315:::benchmark_data_smithi137_151160_object1817:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e396)", 2024-09-18T07:48:45.282 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:43.769818+0000", 2024-09-18T07:48:45.282 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 241.46384042899999, 2024-09-18T07:48:45.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.138105184, 2024-09-18T07:48:45.282 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.282 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.282 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:45.283 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.283 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:48:45.283 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:45.283 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137633 2024-09-18T07:48:45.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.283 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769818+0000", 2024-09-18T07:48:45.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769818+0000", 2024-09-18T07:48:45.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769819+0000", 2024-09-18T07:48:45.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.71e-07 2024-09-18T07:48:45.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769821+0000", 2024-09-18T07:48:45.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8320000000000002e-06 2024-09-18T07:48:45.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769827+0000", 2024-09-18T07:48:45.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8799999999999996e-06 2024-09-18T07:48:45.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769833+0000", 2024-09-18T07:48:45.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5280000000000003e-06 2024-09-18T07:48:45.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774264+0000", 2024-09-18T07:48:45.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044315839999999997 2024-09-18T07:48:45.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774893+0000", 2024-09-18T07:48:45.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00062851599999999997 2024-09-18T07:48:45.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.897969+0000", 2024-09-18T07:48:45.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.123075681 2024-09-18T07:48:45.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:45.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.898060+0000", 2024-09-18T07:48:45.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1570999999999995e-05 2024-09-18T07:48:45.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:48:45.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.898142+0000", 2024-09-18T07:48:45.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1304000000000006e-05 2024-09-18T07:48:45.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.900535+0000", 2024-09-18T07:48:45.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023931239999999999 2024-09-18T07:48:45.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:45.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.907715+0000", 2024-09-18T07:48:45.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.007180278 2024-09-18T07:48:45.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.907757+0000", 2024-09-18T07:48:45.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1897000000000002e-05 2024-09-18T07:48:45.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:45.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.907819+0000", 2024-09-18T07:48:45.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2139999999999998e-05 2024-09-18T07:48:45.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.907923+0000", 2024-09-18T07:48:45.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103978 2024-09-18T07:48:45.290 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:45.291 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.291 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.292 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.292 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189721 10.6 10.a7ff4e76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:48:45.292 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.403483+0000", 2024-09-18T07:48:45.292 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 190.83017501399999, 2024-09-18T07:48:45.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.73509391899999998, 2024-09-18T07:48:45.292 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.292 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.292 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:48:45.292 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.292 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:48:45.293 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:45.293 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189721 2024-09-18T07:48:45.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.293 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403483+0000", 2024-09-18T07:48:45.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403483+0000", 2024-09-18T07:48:45.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403485+0000", 2024-09-18T07:48:45.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.849e-06 2024-09-18T07:48:45.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403490+0000", 2024-09-18T07:48:45.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.468e-06 2024-09-18T07:48:45.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403502+0000", 2024-09-18T07:48:45.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1865000000000001e-05 2024-09-18T07:48:45.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403509+0000", 2024-09-18T07:48:45.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.601e-06 2024-09-18T07:48:45.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403574+0000", 2024-09-18T07:48:45.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5314000000000005e-05 2024-09-18T07:48:45.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403585+0000", 2024-09-18T07:48:45.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0688e-05 2024-09-18T07:48:45.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138555+0000", 2024-09-18T07:48:45.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.73496987400000002 2024-09-18T07:48:45.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138577+0000", 2024-09-18T07:48:45.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.226e-05 2024-09-18T07:48:45.297 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.297 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.297 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.297 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189731 10.6 10.884df406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:48:45.297 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.415372+0000", 2024-09-18T07:48:45.297 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 190.81828649100001, 2024-09-18T07:48:45.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.72323327699999995, 2024-09-18T07:48:45.297 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.298 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.298 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:48:45.298 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.298 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:48:45.298 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:45.298 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189731 2024-09-18T07:48:45.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.298 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415372+0000", 2024-09-18T07:48:45.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415372+0000", 2024-09-18T07:48:45.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415373+0000", 2024-09-18T07:48:45.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3430000000000001e-06 2024-09-18T07:48:45.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415378+0000", 2024-09-18T07:48:45.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5070000000000002e-06 2024-09-18T07:48:45.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415387+0000", 2024-09-18T07:48:45.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7529999999999996e-06 2024-09-18T07:48:45.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415394+0000", 2024-09-18T07:48:45.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4740000000000002e-06 2024-09-18T07:48:45.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415445+0000", 2024-09-18T07:48:45.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1202000000000002e-05 2024-09-18T07:48:45.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415453+0000", 2024-09-18T07:48:45.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.123e-06 2024-09-18T07:48:45.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138598+0000", 2024-09-18T07:48:45.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.72314508399999999 2024-09-18T07:48:45.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138605+0000", 2024-09-18T07:48:45.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7909999999999999e-06 2024-09-18T07:48:45.303 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.303 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.303 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.303 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189736 10.6 10.2043ac56 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:48:45.303 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.419163+0000", 2024-09-18T07:48:45.303 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 190.81449542300001, 2024-09-18T07:48:45.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71946739599999998, 2024-09-18T07:48:45.303 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.303 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.303 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:48:45.303 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.304 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:48:45.304 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:45.304 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189736 2024-09-18T07:48:45.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.304 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419163+0000", 2024-09-18T07:48:45.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419163+0000", 2024-09-18T07:48:45.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.305 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.305 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419164+0000", 2024-09-18T07:48:45.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4130000000000001e-06 2024-09-18T07:48:45.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.305 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.305 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419169+0000", 2024-09-18T07:48:45.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8269999999999997e-06 2024-09-18T07:48:45.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419183+0000", 2024-09-18T07:48:45.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3596e-05 2024-09-18T07:48:45.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419191+0000", 2024-09-18T07:48:45.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0239999999999997e-06 2024-09-18T07:48:45.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419296+0000", 2024-09-18T07:48:45.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0001058 2024-09-18T07:48:45.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419306+0000", 2024-09-18T07:48:45.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5109999999999992e-06 2024-09-18T07:48:45.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138625+0000", 2024-09-18T07:48:45.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71931864599999995 2024-09-18T07:48:45.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138630+0000", 2024-09-18T07:48:45.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5790000000000003e-06 2024-09-18T07:48:45.310 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.310 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.310 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.310 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:215863 10.8 10:1c60eade:::benchmark_data_smithi137_151160_object80047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-18T07:48:45.310 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.932675+0000", 2024-09-18T07:48:45.310 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 156.30098329399999, 2024-09-18T07:48:45.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28984009900000002, 2024-09-18T07:48:45.310 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.310 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.310 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:48:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 215863 2024-09-18T07:48:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932675+0000", 2024-09-18T07:48:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932675+0000", 2024-09-18T07:48:45.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932676+0000", 2024-09-18T07:48:45.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5859999999999999e-06 2024-09-18T07:48:45.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932681+0000", 2024-09-18T07:48:45.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9169999999999997e-06 2024-09-18T07:48:45.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932696+0000", 2024-09-18T07:48:45.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4341e-05 2024-09-18T07:48:45.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932703+0000", 2024-09-18T07:48:45.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1369999999999996e-06 2024-09-18T07:48:45.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.938047+0000", 2024-09-18T07:48:45.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005343965 2024-09-18T07:48:45.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:45.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.938162+0000", 2024-09-18T07:48:45.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115383 2024-09-18T07:48:45.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:48:45.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.938546+0000", 2024-09-18T07:48:45.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00038351599999999998 2024-09-18T07:48:45.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.941541+0000", 2024-09-18T07:48:45.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029950369999999999 2024-09-18T07:48:45.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.318 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:45.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942037+0000", 2024-09-18T07:48:45.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00049626300000000004 2024-09-18T07:48:45.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222371+0000", 2024-09-18T07:48:45.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28033375100000002 2024-09-18T07:48:45.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:45.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222469+0000", 2024-09-18T07:48:45.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8204999999999999e-05 2024-09-18T07:48:45.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222515+0000", 2024-09-18T07:48:45.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5998000000000002e-05 2024-09-18T07:48:45.321 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.322 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.322 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.322 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:215874 10.8 10:1dfe7e41:::benchmark_data_smithi137_151160_object80058:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-18T07:48:45.322 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.942581+0000", 2024-09-18T07:48:45.323 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 156.29107753299999, 2024-09-18T07:48:45.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28009725699999999, 2024-09-18T07:48:45.323 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.323 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.323 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:45.324 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.324 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:48:45.324 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:45.324 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 215874 2024-09-18T07:48:45.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.324 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942581+0000", 2024-09-18T07:48:45.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942581+0000", 2024-09-18T07:48:45.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942582+0000", 2024-09-18T07:48:45.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.629e-06 2024-09-18T07:48:45.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942587+0000", 2024-09-18T07:48:45.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5709999999999999e-06 2024-09-18T07:48:45.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942596+0000", 2024-09-18T07:48:45.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6600000000000007e-06 2024-09-18T07:48:45.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942603+0000", 2024-09-18T07:48:45.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6630000000000004e-06 2024-09-18T07:48:45.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942687+0000", 2024-09-18T07:48:45.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3744000000000003e-05 2024-09-18T07:48:45.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:45.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942797+0000", 2024-09-18T07:48:45.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110093 2024-09-18T07:48:45.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:48:45.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942903+0000", 2024-09-18T07:48:45.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105741 2024-09-18T07:48:45.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949093+0000", 2024-09-18T07:48:45.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0061907419999999999 2024-09-18T07:48:45.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:45.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.951910+0000", 2024-09-18T07:48:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028169779999999999 2024-09-18T07:48:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222552+0000", 2024-09-18T07:48:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27064144600000001 2024-09-18T07:48:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222639+0000", 2024-09-18T07:48:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7190999999999999e-05 2024-09-18T07:48:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222678+0000", 2024-09-18T07:48:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8798999999999999e-05 2024-09-18T07:48:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:215883 10.8 10:10ce8856:::benchmark_data_smithi137_151160_object80067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-18T07:48:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.949012+0000", 2024-09-18T07:48:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 156.284646227, 2024-09-18T07:48:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.281923759, 2024-09-18T07:48:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:48:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:48:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 215883 2024-09-18T07:48:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949012+0000", 2024-09-18T07:48:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949012+0000", 2024-09-18T07:48:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949014+0000", 2024-09-18T07:48:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.063e-06 2024-09-18T07:48:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949020+0000", 2024-09-18T07:48:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7259999999999999e-06 2024-09-18T07:48:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949031+0000", 2024-09-18T07:48:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1748e-05 2024-09-18T07:48:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949039+0000", 2024-09-18T07:48:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0400000000000004e-06 2024-09-18T07:48:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949121+0000", 2024-09-18T07:48:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2622999999999998e-05 2024-09-18T07:48:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949221+0000", 2024-09-18T07:48:45.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100164 2024-09-18T07:48:45.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:48:45.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949325+0000", 2024-09-18T07:48:45.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103342 2024-09-18T07:48:45.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:45.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.952116+0000", 2024-09-18T07:48:45.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027912850000000001 2024-09-18T07:48:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.953945+0000", 2024-09-18T07:48:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018293129999999999 2024-09-18T07:48:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.230849+0000", 2024-09-18T07:48:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27690393299999999 2024-09-18T07:48:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.230891+0000", 2024-09-18T07:48:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1854999999999998e-05 2024-09-18T07:48:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.230936+0000", 2024-09-18T07:48:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4666999999999998e-05 2024-09-18T07:48:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:100758 11.b 11:d317733f:::benchmark_data_smithi137_181069_object100757:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-18T07:48:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:32.242152+0000", 2024-09-18T07:48:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.991505998999999, 2024-09-18T07:48:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1179330860000001, 2024-09-18T07:48:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:48:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 100758 2024-09-18T07:48:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242152+0000", 2024-09-18T07:48:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242152+0000", 2024-09-18T07:48:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242155+0000", 2024-09-18T07:48:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7750000000000001e-06 2024-09-18T07:48:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242261+0000", 2024-09-18T07:48:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010591599999999999 2024-09-18T07:48:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242276+0000", 2024-09-18T07:48:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5593000000000001e-05 2024-09-18T07:48:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242289+0000", 2024-09-18T07:48:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2439000000000001e-05 2024-09-18T07:48:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249077+0000", 2024-09-18T07:48:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0067876439999999998 2024-09-18T07:48:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249087+0000", 2024-09-18T07:48:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0796e-05 2024-09-18T07:48:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.358235+0000", 2024-09-18T07:48:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1091477510000001 2024-09-18T07:48:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.360085+0000", 2024-09-18T07:48:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001850172 2024-09-18T07:48:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:100766 11.b 11:d6b001bf:::benchmark_data_smithi137_181069_object100765:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-18T07:48:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:32.242309+0000", 2024-09-18T07:48:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.991348938, 2024-09-18T07:48:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1181298340000001, 2024-09-18T07:48:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:48:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 100766 2024-09-18T07:48:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242309+0000", 2024-09-18T07:48:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242309+0000", 2024-09-18T07:48:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242310+0000", 2024-09-18T07:48:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2160000000000001e-06 2024-09-18T07:48:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242414+0000", 2024-09-18T07:48:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103411 2024-09-18T07:48:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242426+0000", 2024-09-18T07:48:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2334000000000001e-05 2024-09-18T07:48:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242434+0000", 2024-09-18T07:48:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9340000000000006e-06 2024-09-18T07:48:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249113+0000", 2024-09-18T07:48:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066785239999999999 2024-09-18T07:48:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249120+0000", 2024-09-18T07:48:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9619999999999996e-06 2024-09-18T07:48:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.360349+0000", 2024-09-18T07:48:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.111229037 2024-09-18T07:48:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.360439+0000", 2024-09-18T07:48:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0415999999999996e-05 2024-09-18T07:48:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:100768 11.b 11:dde0046a:::benchmark_data_smithi137_181069_object100767:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-18T07:48:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:32.242445+0000", 2024-09-18T07:48:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.991212692, 2024-09-18T07:48:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1180812979999999, 2024-09-18T07:48:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:48:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 100768 2024-09-18T07:48:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242445+0000", 2024-09-18T07:48:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242445+0000", 2024-09-18T07:48:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242446+0000", 2024-09-18T07:48:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3099999999999996e-07 2024-09-18T07:48:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242482+0000", 2024-09-18T07:48:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5871999999999998e-05 2024-09-18T07:48:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242490+0000", 2024-09-18T07:48:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9740000000000002e-06 2024-09-18T07:48:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242497+0000", 2024-09-18T07:48:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2529999999999998e-06 2024-09-18T07:48:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249144+0000", 2024-09-18T07:48:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066462190000000001 2024-09-18T07:48:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249150+0000", 2024-09-18T07:48:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4269999999999996e-06 2024-09-18T07:48:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.360462+0000", 2024-09-18T07:48:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1113119440000001 2024-09-18T07:48:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.360527+0000", 2024-09-18T07:48:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4777999999999995e-05 2024-09-18T07:48:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:100772 11.b 11:d9b390b9:::benchmark_data_smithi137_181069_object100771:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-18T07:48:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:32.242507+0000", 2024-09-18T07:48:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.991151499000001, 2024-09-18T07:48:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1180945229999999, 2024-09-18T07:48:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:48:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 100772 2024-09-18T07:48:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242507+0000", 2024-09-18T07:48:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242507+0000", 2024-09-18T07:48:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242508+0000", 2024-09-18T07:48:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3500000000000005e-07 2024-09-18T07:48:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242542+0000", 2024-09-18T07:48:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4329999999999998e-05 2024-09-18T07:48:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242559+0000", 2024-09-18T07:48:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6694e-05 2024-09-18T07:48:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242565+0000", 2024-09-18T07:48:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4110000000000001e-06 2024-09-18T07:48:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249172+0000", 2024-09-18T07:48:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066068919999999996 2024-09-18T07:48:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249178+0000", 2024-09-18T07:48:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2570000000000004e-06 2024-09-18T07:48:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.360548+0000", 2024-09-18T07:48:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.111369914 2024-09-18T07:48:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.360601+0000", 2024-09-18T07:48:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3189999999999997e-05 2024-09-18T07:48:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:100779 11.b 11:da6a2060:::benchmark_data_smithi137_181069_object100778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-18T07:48:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:32.242573+0000", 2024-09-18T07:48:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.991085021, 2024-09-18T07:48:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.134857102, 2024-09-18T07:48:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 100779 2024-09-18T07:48:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242573+0000", 2024-09-18T07:48:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242573+0000", 2024-09-18T07:48:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242574+0000", 2024-09-18T07:48:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6700000000000003e-07 2024-09-18T07:48:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242598+0000", 2024-09-18T07:48:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3621e-05 2024-09-18T07:48:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242604+0000", 2024-09-18T07:48:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4069999999999998e-06 2024-09-18T07:48:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242610+0000", 2024-09-18T07:48:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6540000000000004e-06 2024-09-18T07:48:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249201+0000", 2024-09-18T07:48:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065912619999999996 2024-09-18T07:48:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249207+0000", 2024-09-18T07:48:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.455e-06 2024-09-18T07:48:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.360621+0000", 2024-09-18T07:48:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1114135629999999 2024-09-18T07:48:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.361315+0000", 2024-09-18T07:48:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00069365400000000004 2024-09-18T07:48:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,3,4", 2024-09-18T07:48:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.361795+0000", 2024-09-18T07:48:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000480278 2024-09-18T07:48:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.376052+0000", 2024-09-18T07:48:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014257626000000001 2024-09-18T07:48:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.376236+0000", 2024-09-18T07:48:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000183108 2024-09-18T07:48:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.376445+0000", 2024-09-18T07:48:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020976999999999999 2024-09-18T07:48:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.376805+0000", 2024-09-18T07:48:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00035929499999999998 2024-09-18T07:48:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.377049+0000", 2024-09-18T07:48:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000243918 2024-09-18T07:48:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.377430+0000", 2024-09-18T07:48:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00038172400000000001 2024-09-18T07:48:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:100783 11.b 11:d0344236:::benchmark_data_smithi137_181069_object100782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-18T07:48:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:32.242618+0000", 2024-09-18T07:48:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.991040286, 2024-09-18T07:48:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.144450019, 2024-09-18T07:48:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:48:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:48:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:48:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 100783 2024-09-18T07:48:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242618+0000", 2024-09-18T07:48:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242618+0000", 2024-09-18T07:48:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242619+0000", 2024-09-18T07:48:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7199999999999998e-07 2024-09-18T07:48:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242861+0000", 2024-09-18T07:48:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024185999999999999 2024-09-18T07:48:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.243988+0000", 2024-09-18T07:48:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001127402 2024-09-18T07:48:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.244000+0000", 2024-09-18T07:48:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2136999999999999e-05 2024-09-18T07:48:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249230+0000", 2024-09-18T07:48:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052303360000000004 2024-09-18T07:48:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:48:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249237+0000", 2024-09-18T07:48:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6510000000000004e-06 2024-09-18T07:48:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.371777+0000", 2024-09-18T07:48:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1225399549999999 2024-09-18T07:48:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.371923+0000", 2024-09-18T07:48:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014589099999999999 2024-09-18T07:48:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,3,4", 2024-09-18T07:48:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.372092+0000", 2024-09-18T07:48:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000168856 2024-09-18T07:48:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.376395+0000", 2024-09-18T07:48:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043029959999999999 2024-09-18T07:48:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:48:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.376466+0000", 2024-09-18T07:48:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0900000000000002e-05 2024-09-18T07:48:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.380530+0000", 2024-09-18T07:48:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040639589999999998 2024-09-18T07:48:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:48:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.386868+0000", 2024-09-18T07:48:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063385050000000004 2024-09-18T07:48:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.386940+0000", 2024-09-18T07:48:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1828000000000004e-05 2024-09-18T07:48:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.387068+0000", 2024-09-18T07:48:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000127971 2024-09-18T07:48:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "MOSDPGPush(11.1 501/500 [PushOp(11:850e205e:::benchmark_data_smithi137_181069_object100773:head, version: 498'6356, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(11:850e205e:::benchmark_data_smithi137_181069_object100773:head@498'6356, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-18T07:48:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:33.406225+0000", 2024-09-18T07:48:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 11.827433423, 2024-09-18T07:48:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.23373427199999999, 2024-09-18T07:48:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:48:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.406225+0000", 2024-09-18T07:48:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.406225+0000", 2024-09-18T07:48:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.406226+0000", 2024-09-18T07:48:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9800000000000002e-07 2024-09-18T07:48:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.406353+0000", 2024-09-18T07:48:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000127162 2024-09-18T07:48:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.406367+0000", 2024-09-18T07:48:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4079e-05 2024-09-18T07:48:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.406377+0000", 2024-09-18T07:48:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8900000000000002e-06 2024-09-18T07:48:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.406442+0000", 2024-09-18T07:48:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4995999999999999e-05 2024-09-18T07:48:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.406455+0000", 2024-09-18T07:48:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.292e-05 2024-09-18T07:48:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.639959+0000", 2024-09-18T07:48:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.23350422700000001 2024-09-18T07:48:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:100778 11.1 e501/500 11:854730f9:::benchmark_data_smithi137_181069_object100777:head v 501'6357, mlcod=0'0)", 2024-09-18T07:48:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:33.406993+0000", 2024-09-18T07:48:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 11.826665623, 2024-09-18T07:48:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.26161929, 2024-09-18T07:48:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:48:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.406993+0000", 2024-09-18T07:48:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.406993+0000", 2024-09-18T07:48:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.406993+0000", 2024-09-18T07:48:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8199999999999998e-07 2024-09-18T07:48:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.409785+0000", 2024-09-18T07:48:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002791651 2024-09-18T07:48:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.639984+0000", 2024-09-18T07:48:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.23019884700000001 2024-09-18T07:48:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.639993+0000", 2024-09-18T07:48:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8729999999999996e-06 2024-09-18T07:48:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.640035+0000", 2024-09-18T07:48:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2116000000000001e-05 2024-09-18T07:48:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.640059+0000", 2024-09-18T07:48:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3935999999999999e-05 2024-09-18T07:48:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:48:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.668533+0000", 2024-09-18T07:48:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028474322 2024-09-18T07:48:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:48:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.668612+0000", 2024-09-18T07:48:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8663000000000002e-05 2024-09-18T07:48:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:45.405 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:48:45.434 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:45.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:45.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:45.536 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:45.579 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:45.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:45.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:45 smithi137 ceph-mon[29408]: pgmap v952: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 935 op/s; 43167/327102 objects misplaced (13.197%); 31 MiB/s, 512 objects/s recovering 2024-09-18T07:48:45.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:45 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1953812712' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:48:45.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:45 smithi137 ceph-mon[29408]: osdmap e504: 8 total, 8 up, 7 in 2024-09-18T07:48:45.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:45 smithi137 ceph-mon[39175]: pgmap v952: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 935 op/s; 43167/327102 objects misplaced (13.197%); 31 MiB/s, 512 objects/s recovering 2024-09-18T07:48:45.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:45 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1953812712' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:48:45.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:45 smithi137 ceph-mon[39175]: osdmap e504: 8 total, 8 up, 7 in 2024-09-18T07:48:45.681 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:45.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:45 smithi203 ceph-mon[32237]: pgmap v952: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 935 op/s; 43167/327102 objects misplaced (13.197%); 31 MiB/s, 512 objects/s recovering 2024-09-18T07:48:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:45 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1953812712' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:48:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:45 smithi203 ceph-mon[32237]: osdmap e504: 8 total, 8 up, 7 in 2024-09-18T07:48:45.721 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:45.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:45.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:45.824 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:45.862 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:45.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:45.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:45.964 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:46.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:46.016 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:46.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:46.117 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:46.148 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:46.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:46.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:46.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:46.296 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:46.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:46.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:46.398 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:46.426 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:46.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:46.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:46.532 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:46.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:46.567 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:46.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:46.668 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:46.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:46 smithi137 ceph-mon[29408]: pgmap v954: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 44 MiB/s wr, 703 op/s; 43167/327960 objects misplaced (13.162%); 23 MiB/s, 370 objects/s recovering 2024-09-18T07:48:46.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:46 smithi137 ceph-mon[39175]: pgmap v954: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 44 MiB/s wr, 703 op/s; 43167/327960 objects misplaced (13.162%); 23 MiB/s, 370 objects/s recovering 2024-09-18T07:48:46.706 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:46.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:46.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:46.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:46 smithi203 ceph-mon[32237]: pgmap v954: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 44 MiB/s wr, 703 op/s; 43167/327960 objects misplaced (13.162%); 23 MiB/s, 370 objects/s recovering 2024-09-18T07:48:46.809 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:46.840 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:46.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:46.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:46.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-18T07:48:46.941 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:46.969 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:46.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:46.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:47.071 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:47.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:47.104 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:47.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:47.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:48:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:48:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:48:47.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:47.242 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:47.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:47.344 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:47.376 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:47.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:47.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:47.478 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:47.510 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:47.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:47.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:47.612 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:47.643 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:47.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:47.745 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:47.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T07:48:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:47.767 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:47.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:47.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:47.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:47.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:47.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:48.024 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:48.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:48.083 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:48.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:48.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:48.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:48:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:48:48.245 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:48.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:48.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:48.286 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:48.347 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:48.368 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:48.375 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:48.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:48.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:48.383 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:48.477 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:48.515 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:48.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:48.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:48.617 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:48.645 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:48.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:48.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:48.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:48.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:48.793 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:48.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:48.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:48.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:48.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:48.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:49.010 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:48 smithi137 ceph-mon[29408]: pgmap v955: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.4 KiB/s rd, 61 MiB/s wr, 1.19k op/s; 41639/331494 objects misplaced (12.561%); 34 MiB/s, 537 objects/s recovering 2024-09-18T07:48:49.010 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:48 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:49.010 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:48 smithi137 ceph-mon[39175]: pgmap v955: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.4 KiB/s rd, 61 MiB/s wr, 1.19k op/s; 41639/331494 objects misplaced (12.561%); 34 MiB/s, 537 objects/s recovering 2024-09-18T07:48:49.010 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:48 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:49.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:49.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:49.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:49.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:49.192 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:49.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:48 smithi203 ceph-mon[32237]: pgmap v955: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.4 KiB/s rd, 61 MiB/s wr, 1.19k op/s; 41639/331494 objects misplaced (12.561%); 34 MiB/s, 537 objects/s recovering 2024-09-18T07:48:49.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:48 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:49.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:49.227 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:49.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:49.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:49.359 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:49.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:49.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:49.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:49.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:49.503 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:49.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:49.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:49.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:49.645 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:49.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:49.746 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:49.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:49.780 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:49.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:49.881 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:49.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:49.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:49.940 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:48:49.944 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.2 to 1 (10000) 2024-09-18T07:48:49.999 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:49 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2690851735' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:48:49.999 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:49 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3501996378' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:48:49.999 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:49 smithi137 ceph-mon[29408]: from='client.26692 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:48:49.999 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:49 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2690851735' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:48:49.999 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:49 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3501996378' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:48:49.999 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:49 smithi137 ceph-mon[39175]: from='client.26692 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:48:50.024 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:50.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:50.058 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:50.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:50.070 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:48:50.070 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:48:50.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.071 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:117380 11.7 e505/471 11:e98bb994:::benchmark_data_smithi137_181069_object5226:head v 505'7379, mlcod=505'7376)", 2024-09-18T07:48:50.071 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:50.066035+0000", 2024-09-18T07:48:50.071 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0010841270000000001, 2024-09-18T07:48:50.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011093640000000001, 2024-09-18T07:48:50.071 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:50.071 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:50.071 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:48:50.071 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:50.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:50.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.066035+0000", 2024-09-18T07:48:50.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:50.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:50.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.066035+0000", 2024-09-18T07:48:50.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:50.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:50.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.066037+0000", 2024-09-18T07:48:50.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.313e-06 2024-09-18T07:48:50.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.073 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:50.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.066043+0000", 2024-09-18T07:48:50.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3160000000000002e-06 2024-09-18T07:48:50.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.073 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:50.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.066056+0000", 2024-09-18T07:48:50.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3511e-05 2024-09-18T07:48:50.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:50.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.066064+0000", 2024-09-18T07:48:50.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3939999999999998e-06 2024-09-18T07:48:50.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:50.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.066816+0000", 2024-09-18T07:48:50.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00075257199999999996 2024-09-18T07:48:50.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:50.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.066841+0000", 2024-09-18T07:48:50.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4179e-05 2024-09-18T07:48:50.075 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:50.075 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:50.075 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:50.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.075 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:117364 11.6 e505/481 11:654811b0:::benchmark_data_smithi137_181069_object5210:head v 505'7236, mlcod=504'7234)", 2024-09-18T07:48:50.075 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:50.053089+0000", 2024-09-18T07:48:50.075 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.014030134, 2024-09-18T07:48:50.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014109926, 2024-09-18T07:48:50.076 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:50.076 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:50.076 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:48:50.076 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:50.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:50.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.053089+0000", 2024-09-18T07:48:50.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:50.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:50.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.053089+0000", 2024-09-18T07:48:50.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:50.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:50.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.053091+0000", 2024-09-18T07:48:50.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2750000000000001e-06 2024-09-18T07:48:50.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:50.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.053097+0000", 2024-09-18T07:48:50.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1789999999999996e-06 2024-09-18T07:48:50.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:50.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.053105+0000", 2024-09-18T07:48:50.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2900000000000002e-06 2024-09-18T07:48:50.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:50.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.053111+0000", 2024-09-18T07:48:50.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1940000000000003e-06 2024-09-18T07:48:50.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:50.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.053668+0000", 2024-09-18T07:48:50.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00055720399999999995 2024-09-18T07:48:50.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:50.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.058299+0000", 2024-09-18T07:48:50.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046308299999999998 2024-09-18T07:48:50.079 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:50.079 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:50.079 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:50.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.079 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:117368 11.9 e505/500 11:97253006:::benchmark_data_smithi137_181069_object5214:head v 505'7321, mlcod=505'7318)", 2024-09-18T07:48:50.079 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:50.060026+0000", 2024-09-18T07:48:50.079 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0070937250000000004, 2024-09-18T07:48:50.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0072203199999999997, 2024-09-18T07:48:50.079 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:50.080 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:50.080 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:48:50.080 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:50.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:50.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060026+0000", 2024-09-18T07:48:50.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:50.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:50.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060026+0000", 2024-09-18T07:48:50.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:50.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060027+0000", 2024-09-18T07:48:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1429999999999999e-06 2024-09-18T07:48:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060038+0000", 2024-09-18T07:48:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0988000000000001e-05 2024-09-18T07:48:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:50.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060048+0000", 2024-09-18T07:48:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0518e-05 2024-09-18T07:48:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060055+0000", 2024-09-18T07:48:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9800000000000001e-06 2024-09-18T07:48:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.062093+0000", 2024-09-18T07:48:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020377609999999999 2024-09-18T07:48:50.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.062114+0000", 2024-09-18T07:48:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0772999999999999e-05 2024-09-18T07:48:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:117366 11.6 e505/481 11:6e9ccde7:::benchmark_data_smithi137_181069_object5212:head v 505'7237, mlcod=504'7234)", 2024-09-18T07:48:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:50.060752+0000", 2024-09-18T07:48:50.083 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0063673269999999999, 2024-09-18T07:48:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065392519999999997, 2024-09-18T07:48:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:48:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060752+0000", 2024-09-18T07:48:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:50.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060752+0000", 2024-09-18T07:48:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060753+0000", 2024-09-18T07:48:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0599999999999999e-07 2024-09-18T07:48:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060759+0000", 2024-09-18T07:48:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6579999999999999e-06 2024-09-18T07:48:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060766+0000", 2024-09-18T07:48:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2980000000000003e-06 2024-09-18T07:48:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060772+0000", 2024-09-18T07:48:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6219999999999997e-06 2024-09-18T07:48:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:48:50.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.061822+0000", 2024-09-18T07:48:50.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00105054 2024-09-18T07:48:50.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:48:50.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.062586+0000", 2024-09-18T07:48:50.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00076353399999999996 2024-09-18T07:48:50.087 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:50.087 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:50.087 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:50.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.087 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:117370 11.6 e505/481)", 2024-09-18T07:48:50.087 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:50.060830+0000", 2024-09-18T07:48:50.088 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0062898709999999998, 2024-09-18T07:48:50.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065044719999999999, 2024-09-18T07:48:50.088 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:50.088 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:50.088 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:48:50.088 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:50.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:50.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060830+0000", 2024-09-18T07:48:50.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:50.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:50.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060830+0000", 2024-09-18T07:48:50.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:50.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:50.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060830+0000", 2024-09-18T07:48:50.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0500000000000004e-07 2024-09-18T07:48:50.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:50.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060837+0000", 2024-09-18T07:48:50.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0720000000000002e-06 2024-09-18T07:48:50.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:50.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060852+0000", 2024-09-18T07:48:50.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4992e-05 2024-09-18T07:48:50.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:50.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060857+0000", 2024-09-18T07:48:50.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8969999999999999e-06 2024-09-18T07:48:50.090 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:50.091 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:50.091 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:50.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.091 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:117371 11.e e505/472)", 2024-09-18T07:48:50.091 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:50.060874+0000", 2024-09-18T07:48:50.091 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0062456229999999996, 2024-09-18T07:48:50.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0064942840000000003, 2024-09-18T07:48:50.091 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:50.091 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:50.091 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:48:50.091 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:50.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:50.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060874+0000", 2024-09-18T07:48:50.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:50.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:50.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060874+0000", 2024-09-18T07:48:50.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:50.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:50.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060875+0000", 2024-09-18T07:48:50.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9599999999999999e-07 2024-09-18T07:48:50.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:50.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060879+0000", 2024-09-18T07:48:50.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9030000000000003e-06 2024-09-18T07:48:50.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:50.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060885+0000", 2024-09-18T07:48:50.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0360000000000003e-06 2024-09-18T07:48:50.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:50.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.060891+0000", 2024-09-18T07:48:50.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4040000000000003e-06 2024-09-18T07:48:50.094 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:50.094 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:50.094 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:50.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.094 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:117373 11.6 e505/481)", 2024-09-18T07:48:50.094 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:50.062273+0000", 2024-09-18T07:48:50.094 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0048460559999999996, 2024-09-18T07:48:50.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051284219999999997, 2024-09-18T07:48:50.095 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:48:50.095 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:48:50.095 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:48:50.095 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:48:50.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:48:50.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.062273+0000", 2024-09-18T07:48:50.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:50.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:48:50.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.062273+0000", 2024-09-18T07:48:50.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:48:50.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:48:50.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.062274+0000", 2024-09-18T07:48:50.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5600000000000004e-07 2024-09-18T07:48:50.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.097 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:48:50.097 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.062280+0000", 2024-09-18T07:48:50.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8649999999999998e-06 2024-09-18T07:48:50.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.097 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:48:50.097 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.062288+0000", 2024-09-18T07:48:50.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1340000000000004e-06 2024-09-18T07:48:50.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:48:50.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:48:50.097 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:48:50.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:50.062294+0000", 2024-09-18T07:48:50.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0360000000000003e-06 2024-09-18T07:48:50.098 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:50.098 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:48:50.098 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:50.098 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:48:50.098 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:48:50.098 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 7 2024-09-18T07:48:50.098 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:48:50.158 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:50.199 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:50.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:50.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:50.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:49 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2690851735' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:48:50.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:49 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3501996378' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:48:50.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:49 smithi203 ceph-mon[32237]: from='client.26692 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:48:50.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:48:50.301 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:50.331 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:50.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:50.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:50.433 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:50.461 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:50.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:50.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:50.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:50.603 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:50.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:50.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:50.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:50.736 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:50.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:50.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:50.838 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:50.874 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:50.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:50.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:50.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:51.015 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:51.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:51.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:51.117 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:51.147 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:51.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:51.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:50 smithi203 ceph-mon[32237]: pgmap v956: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.3 KiB/s rd, 59 MiB/s wr, 1.20k op/s; 40903/330549 objects misplaced (12.374%); 37 MiB/s, 585 objects/s recovering 2024-09-18T07:48:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:50 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:50 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2690851735' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:48:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:50 smithi203 ceph-mon[32237]: from='client.26692 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:48:51.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:50 smithi203 ceph-mon[32237]: osdmap e505: 8 total, 8 up, 7 in 2024-09-18T07:48:51.249 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:51.283 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:51.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:51.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:51.385 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:51.417 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:51.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:51.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:50 smithi137 ceph-mon[29408]: pgmap v956: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.3 KiB/s rd, 59 MiB/s wr, 1.20k op/s; 40903/330549 objects misplaced (12.374%); 37 MiB/s, 585 objects/s recovering 2024-09-18T07:48:51.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:50 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:51.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:50 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2690851735' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:48:51.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:50 smithi137 ceph-mon[29408]: from='client.26692 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:48:51.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:50 smithi137 ceph-mon[29408]: osdmap e505: 8 total, 8 up, 7 in 2024-09-18T07:48:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:50 smithi137 ceph-mon[39175]: pgmap v956: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.3 KiB/s rd, 59 MiB/s wr, 1.20k op/s; 40903/330549 objects misplaced (12.374%); 37 MiB/s, 585 objects/s recovering 2024-09-18T07:48:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:50 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:48:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:50 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2690851735' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:48:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:50 smithi137 ceph-mon[39175]: from='client.26692 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:48:51.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:50 smithi137 ceph-mon[39175]: osdmap e505: 8 total, 8 up, 7 in 2024-09-18T07:48:51.518 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:51.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:51.541 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:51.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:51.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:51.678 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:51.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:51.785 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:51.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:51.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:51.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:51.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:51.943 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:51.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:52.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:52.083 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:52.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:52.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:52.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:52.219 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:52.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:52.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:52.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:52.349 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:52.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:52.450 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:52.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:52.486 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:52.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:52.587 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:52.621 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:52.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:52.723 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:52.764 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:52.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:52.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:52.866 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:52.902 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:52.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:52.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:53.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:53.039 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:53.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:53.141 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:53.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:53.174 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:53.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:53.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:52 smithi203 ceph-mon[32237]: 11.7 scrub starts 2024-09-18T07:48:53.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:52 smithi203 ceph-mon[32237]: pgmap v958: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 26 MiB/s wr, 1.05k op/s; 40278/321888 objects misplaced (12.513%); 20 MiB/s, 322 objects/s recovering 2024-09-18T07:48:53.275 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:53.312 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:53.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:53.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:53.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:53.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:52 smithi137 ceph-mon[39175]: 11.7 scrub starts 2024-09-18T07:48:53.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:52 smithi137 ceph-mon[39175]: pgmap v958: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 26 MiB/s wr, 1.05k op/s; 40278/321888 objects misplaced (12.513%); 20 MiB/s, 322 objects/s recovering 2024-09-18T07:48:53.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:52 smithi137 ceph-mon[29408]: 11.7 scrub starts 2024-09-18T07:48:53.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:52 smithi137 ceph-mon[29408]: pgmap v958: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 26 MiB/s wr, 1.05k op/s; 40278/321888 objects misplaced (12.513%); 20 MiB/s, 322 objects/s recovering 2024-09-18T07:48:53.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:53.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:53.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:53.557 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:53.596 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:53.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:53.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:53.698 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:53.733 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:53.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:53.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:53.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:53.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:53.877 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:53.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:53.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:54.013 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:54.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:54.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:54.115 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:54.136 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:54.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:54.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:54.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:54.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:54.270 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:54.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:54.371 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:54.406 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:54.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:54.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:54.508 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:54.540 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:54.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:54.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:54.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:54.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:54.682 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:54.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:54.783 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:54.820 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:54.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:54.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:54.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:54.964 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:54.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:55.065 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:55.086 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:55.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:55.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:55.188 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:55.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:54 smithi203 ceph-mon[32237]: 11.7 scrub ok 2024-09-18T07:48:55.210 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:54 smithi203 ceph-mon[32237]: pgmap v959: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 20 MiB/s wr, 1.05k op/s; 36807/316362 objects misplaced (11.634%); 32 MiB/s, 515 objects/s recovering 2024-09-18T07:48:55.213 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:55.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:55.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:55.315 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:55.343 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:55.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:55.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:55.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:54 smithi137 ceph-mon[29408]: 11.7 scrub ok 2024-09-18T07:48:55.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:54 smithi137 ceph-mon[29408]: pgmap v959: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 20 MiB/s wr, 1.05k op/s; 36807/316362 objects misplaced (11.634%); 32 MiB/s, 515 objects/s recovering 2024-09-18T07:48:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:54 smithi137 ceph-mon[39175]: 11.7 scrub ok 2024-09-18T07:48:55.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:54 smithi137 ceph-mon[39175]: pgmap v959: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 20 MiB/s wr, 1.05k op/s; 36807/316362 objects misplaced (11.634%); 32 MiB/s, 515 objects/s recovering 2024-09-18T07:48:55.444 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:55.478 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:55.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:55.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:55.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:55.611 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:55.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:55.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:55.713 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:48:55.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:55.749 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:55.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:48:55.850 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:55.876 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:55.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:55.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:55.978 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:56.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:56.007 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:56.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:56.108 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:56.147 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:56.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:56.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:56.249 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:56.276 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:56.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:56.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:56.402 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:56.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:56.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:56.504 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:56.544 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:56.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:56.646 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:56.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:56.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:56.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:56.769 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:56.800 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:56.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:56.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:56.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:56.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:56.946 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:56.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:57.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:57.088 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:57.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:57.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:57.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:48:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:48:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:48:57.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:57 smithi137 ceph-mon[29408]: pgmap v960: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 19 MiB/s wr, 1.03k op/s; 36807/315306 objects misplaced (11.673%); 31 MiB/s, 489 objects/s recovering 2024-09-18T07:48:57.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:57 smithi137 ceph-mon[39175]: pgmap v960: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 19 MiB/s wr, 1.03k op/s; 36807/315306 objects misplaced (11.673%); 31 MiB/s, 489 objects/s recovering 2024-09-18T07:48:57.190 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:57.221 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:57.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:57.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:57.323 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:57.353 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:57.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:57.454 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:48:57.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:57 smithi203 ceph-mon[32237]: pgmap v960: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 19 MiB/s wr, 1.03k op/s; 36807/315306 objects misplaced (11.673%); 31 MiB/s, 489 objects/s recovering 2024-09-18T07:48:57.484 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:57.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:57.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:48:57.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:57.622 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:57.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:57.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:57.724 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:57.764 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:57.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:57.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:57.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-18T07:48:57.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:57.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:57.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:57.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:58.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:58.073 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:58.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:58.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:58.175 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:58.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:58 smithi203 ceph-mon[32237]: 6.15 scrub starts 2024-09-18T07:48:58.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:58 smithi203 ceph-mon[32237]: 6.15 scrub ok 2024-09-18T07:48:58.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:58 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:48:58.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:48:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:48:58.227 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:58.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:58.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:48:58.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T07:48:58.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:58.418 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:58.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:48:58.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:58 smithi137 ceph-mon[39175]: 6.15 scrub starts 2024-09-18T07:48:58.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:58 smithi137 ceph-mon[39175]: 6.15 scrub ok 2024-09-18T07:48:58.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:58 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:48:58.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:58 smithi137 ceph-mon[29408]: 6.15 scrub starts 2024-09-18T07:48:58.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:58 smithi137 ceph-mon[29408]: 6.15 scrub ok 2024-09-18T07:48:58.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:58 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:48:58.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:58.568 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:58.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:58.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:58.670 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:48:58.711 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:58.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:58.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:48:58.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:58.839 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:58.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:58.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:48:58.863 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:58.928 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:58.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:58.963 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:48:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:58.997 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:58.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:59.098 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:48:59.132 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:59.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:48:59.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:59.310 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:59.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:59.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:59.363 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:59 smithi137 ceph-mon[29408]: pgmap v961: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 1.1 MiB/s wr, 987 op/s; 33630/302952 objects misplaced (11.101%); 30 MiB/s, 472 objects/s recovering 2024-09-18T07:48:59.363 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:48:59 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:59.363 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:59 smithi137 ceph-mon[39175]: pgmap v961: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 1.1 MiB/s wr, 987 op/s; 33630/302952 objects misplaced (11.101%); 30 MiB/s, 472 objects/s recovering 2024-09-18T07:48:59.363 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:48:59 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:59.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:48:59.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:59.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:59.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:48:59.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:59 smithi203 ceph-mon[32237]: pgmap v961: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 1.1 MiB/s wr, 987 op/s; 33630/302952 objects misplaced (11.101%); 30 MiB/s, 472 objects/s recovering 2024-09-18T07:48:59.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:48:59 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:48:59.557 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:59.634 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:59.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:59.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:59.737 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:48:59.780 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:48:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:59.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:48:59.882 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:48:59.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:48:59.911 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:48:59.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:00.012 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:00.045 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:00.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:00.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:00.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:00.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:00.187 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:00.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:00.267 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:49:00.268 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:49:00.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:00 smithi137 ceph-mon[29408]: 4.11 scrub starts 2024-09-18T07:49:00.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:00 smithi137 ceph-mon[29408]: 4.11 scrub ok 2024-09-18T07:49:00.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:00 smithi137 ceph-mon[39175]: 4.11 scrub starts 2024-09-18T07:49:00.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:00 smithi137 ceph-mon[39175]: 4.11 scrub ok 2024-09-18T07:49:00.288 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:00.306 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:49:00.307 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:49:00.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:00.323 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:00.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:00.342 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:49:00.343 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:49:00.378 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:49:00.378 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:49:00.412 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:49:00.412 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:49:00.425 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:00.442 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:49:00.443 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:49:00.453 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:00.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:00.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:00.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:00 smithi203 ceph-mon[32237]: 4.11 scrub starts 2024-09-18T07:49:00.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:00 smithi203 ceph-mon[32237]: 4.11 scrub ok 2024-09-18T07:49:00.481 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:49:00.481 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:49:00.512 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:49:00.512 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:49:00.555 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:00.581 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:00.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:00.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:00.683 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:00.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:00.728 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:00.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:00.829 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:00.855 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:49:00.859 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:00.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:00.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:00.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:01.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:01.014 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:01.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:01.115 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:01.139 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:01.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:01.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:01.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:01.269 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:01.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:01.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:01.318 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:49:01.319 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:49:01.370 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:01.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:01.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:01.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:01.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:01 smithi137 ceph-mon[29408]: 1.1 scrub starts 2024-09-18T07:49:01.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:01 smithi137 ceph-mon[29408]: 1.1 scrub ok 2024-09-18T07:49:01.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:01 smithi137 ceph-mon[29408]: pgmap v962: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 978 op/s; 32735/301218 objects misplaced (10.868%); 30 MiB/s, 476 objects/s recovering 2024-09-18T07:49:01.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:01 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4276623687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:01.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:01 smithi137 ceph-mon[39175]: 1.1 scrub starts 2024-09-18T07:49:01.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:01 smithi137 ceph-mon[39175]: 1.1 scrub ok 2024-09-18T07:49:01.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:01 smithi137 ceph-mon[39175]: pgmap v962: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 978 op/s; 32735/301218 objects misplaced (10.868%); 30 MiB/s, 476 objects/s recovering 2024-09-18T07:49:01.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:01 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4276623687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:01.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:01 smithi203 ceph-mon[32237]: 1.1 scrub starts 2024-09-18T07:49:01.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:01 smithi203 ceph-mon[32237]: 1.1 scrub ok 2024-09-18T07:49:01.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:01 smithi203 ceph-mon[32237]: pgmap v962: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 978 op/s; 32735/301218 objects misplaced (10.868%); 30 MiB/s, 476 objects/s recovering 2024-09-18T07:49:01.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:01 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4276623687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:01.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:01.556 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:01.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:01.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:01.658 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:01.691 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:01.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:01.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:01.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:01.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:01.820 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:01.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:01.921 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:01.945 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:01.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:01.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:02.047 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:02.077 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:02.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:02.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:02.178 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:02.211 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:02.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:02.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:02.223 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.4 to 1 (10000) 2024-09-18T07:49:02.230 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T07:49:02.313 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:02.361 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:02.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:02.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:02 smithi203 ceph-mon[32237]: 3.13 scrub starts 2024-09-18T07:49:02.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:02 smithi203 ceph-mon[32237]: 3.13 scrub ok 2024-09-18T07:49:02.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:02 smithi203 ceph-mon[32237]: 1.8 scrub starts 2024-09-18T07:49:02.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:02 smithi203 ceph-mon[32237]: 1.8 scrub ok 2024-09-18T07:49:02.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:02 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3934456835' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:49:02.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:02 smithi203 ceph-mon[32237]: from='client.36593 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:49:02.462 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:02.491 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:02.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:02.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:02.593 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:02.613 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:02.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:02.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:02.627 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:02 smithi137 ceph-mon[29408]: 3.13 scrub starts 2024-09-18T07:49:02.627 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:02 smithi137 ceph-mon[29408]: 3.13 scrub ok 2024-09-18T07:49:02.627 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:02 smithi137 ceph-mon[29408]: 1.8 scrub starts 2024-09-18T07:49:02.627 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:02 smithi137 ceph-mon[29408]: 1.8 scrub ok 2024-09-18T07:49:02.627 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:02 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3934456835' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:49:02.627 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:02 smithi137 ceph-mon[29408]: from='client.36593 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:49:02.628 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:02 smithi137 ceph-mon[39175]: 3.13 scrub starts 2024-09-18T07:49:02.628 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:02 smithi137 ceph-mon[39175]: 3.13 scrub ok 2024-09-18T07:49:02.628 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:02 smithi137 ceph-mon[39175]: 1.8 scrub starts 2024-09-18T07:49:02.628 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:02 smithi137 ceph-mon[39175]: 1.8 scrub ok 2024-09-18T07:49:02.628 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:02 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3934456835' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:49:02.628 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:02 smithi137 ceph-mon[39175]: from='client.36593 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:49:02.717 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:02.753 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:02.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:02.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:02.855 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:02.880 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:02.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:02.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:02.982 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:02.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-18T07:49:03.007 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:03.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:03.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:03.109 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:03.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:03.136 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:03.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:03.212 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:03 smithi137 ceph-mon[29408]: pgmap v963: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.06k op/s; 32113/292755 objects misplaced (10.969%); 25 MiB/s, 399 objects/s recovering 2024-09-18T07:49:03.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:03.269 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:03.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:03.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:03.371 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:03.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:03.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:03.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:03.512 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:03.521 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:03 smithi137 ceph-mon[29408]: from='client.36593 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:49:03.521 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:03 smithi137 ceph-mon[29408]: osdmap e506: 8 total, 8 up, 7 in 2024-09-18T07:49:03.521 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:03 smithi137 ceph-mon[39175]: pgmap v963: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.06k op/s; 32113/292755 objects misplaced (10.969%); 25 MiB/s, 399 objects/s recovering 2024-09-18T07:49:03.521 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:03 smithi137 ceph-mon[39175]: from='client.36593 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:49:03.521 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:03 smithi137 ceph-mon[39175]: osdmap e506: 8 total, 8 up, 7 in 2024-09-18T07:49:03.543 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:03.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:03.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:03.645 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:03.645 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:03.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:03.703 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:03.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:03.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:03 smithi203 ceph-mon[32237]: pgmap v963: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.06k op/s; 32113/292755 objects misplaced (10.969%); 25 MiB/s, 399 objects/s recovering 2024-09-18T07:49:03.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:03 smithi203 ceph-mon[32237]: from='client.36593 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:49:03.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:03 smithi203 ceph-mon[32237]: osdmap e506: 8 total, 8 up, 7 in 2024-09-18T07:49:03.804 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:03.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:03.854 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:03.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:03.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:04.000 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:04.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:04.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:04.102 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:04.150 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:04.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:04.251 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:04.291 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:04.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:04.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:04.330 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:04.331 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:04.392 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:04.428 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:04.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:04.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:04.530 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:04.567 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:04.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:04.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:04.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:04.719 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:04.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:04.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:04.821 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:04.851 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:04.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:04.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:04.954 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:04.981 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:04.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:04.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:05.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:05.123 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:05.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:05.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:05.176 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:49:05.177 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:49:05.224 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:05.240 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:49:05.261 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:05.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:05.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:05.363 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:05.400 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:05.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:05.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:05.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:05.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:05.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:05.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:05.641 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:05.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:49:05.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:05 smithi137 ceph-mon[29408]: pgmap v965: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 5.8 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 968 op/s; 28435/287313 objects misplaced (9.897%); 32 MiB/s, 518 objects/s recovering 2024-09-18T07:49:05.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:05 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2020536463' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:05.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:05 smithi137 ceph-mon[29408]: from='client.26707 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:05 smithi137 ceph-mon[39175]: pgmap v965: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 5.8 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 968 op/s; 28435/287313 objects misplaced (9.897%); 32 MiB/s, 518 objects/s recovering 2024-09-18T07:49:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:05 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2020536463' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:05 smithi137 ceph-mon[39175]: from='client.26707 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:05.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:05.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:05.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:05 smithi203 ceph-mon[32237]: pgmap v965: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 5.8 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 968 op/s; 28435/287313 objects misplaced (9.897%); 32 MiB/s, 518 objects/s recovering 2024-09-18T07:49:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:05 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2020536463' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:05 smithi203 ceph-mon[32237]: from='client.26707 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:05.790 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:05.829 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:05.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:05.931 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:05.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:05.962 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:05.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:06.063 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:06.099 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:06.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:06.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:06.201 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:06.235 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.5 to 1 (10000) 2024-09-18T07:49:06.240 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:06.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:06.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:06.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:06.380 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:06.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:06.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:06.481 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:06.511 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:06.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:06.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:06.613 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:06.641 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:06 smithi137 ceph-mon[29408]: 6.7 scrub starts 2024-09-18T07:49:06.642 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:06 smithi137 ceph-mon[29408]: 6.7 scrub ok 2024-09-18T07:49:06.642 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:06 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:49:06.642 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:06 smithi137 ceph-mon[29408]: from='client.26707 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:49:06.642 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:06 smithi137 ceph-mon[29408]: osdmap e507: 8 total, 8 up, 7 in 2024-09-18T07:49:06.642 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:06 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2653980143' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:49:06.642 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:06 smithi137 ceph-mon[39175]: 6.7 scrub starts 2024-09-18T07:49:06.642 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:06 smithi137 ceph-mon[39175]: 6.7 scrub ok 2024-09-18T07:49:06.642 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:06 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:49:06.642 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:06 smithi137 ceph-mon[39175]: from='client.26707 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:49:06.642 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:06 smithi137 ceph-mon[39175]: osdmap e507: 8 total, 8 up, 7 in 2024-09-18T07:49:06.643 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:06 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2653980143' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:49:06.647 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:06.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:06.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:06.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:06 smithi203 ceph-mon[32237]: 6.7 scrub starts 2024-09-18T07:49:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:06 smithi203 ceph-mon[32237]: 6.7 scrub ok 2024-09-18T07:49:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:06 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:49:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:06 smithi203 ceph-mon[32237]: from='client.26707 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:49:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:06 smithi203 ceph-mon[32237]: osdmap e507: 8 total, 8 up, 7 in 2024-09-18T07:49:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:06 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2653980143' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:49:06.749 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:06.786 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:06.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:06.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:06.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:06.923 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:06.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:06.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:07.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:07.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:07.085 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:07.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:07.186 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:07.224 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:07.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:07.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:07.363 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:07.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:07 smithi137 ceph-mon[29408]: pgmap v967: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 5.8 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 680 op/s; 28435/286623 objects misplaced (9.921%); 27 MiB/s, 432 objects/s recovering 2024-09-18T07:49:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:07 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2653980143' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:49:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:07 smithi137 ceph-mon[29408]: osdmap e508: 8 total, 8 up, 7 in 2024-09-18T07:49:07.421 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:49:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:49:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:49:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:07 smithi137 ceph-mon[39175]: pgmap v967: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 5.8 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 680 op/s; 28435/286623 objects misplaced (9.921%); 27 MiB/s, 432 objects/s recovering 2024-09-18T07:49:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:07 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2653980143' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:49:07.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:07 smithi137 ceph-mon[39175]: osdmap e508: 8 total, 8 up, 7 in 2024-09-18T07:49:07.465 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:07.503 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:07.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:07.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:07.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:07.641 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:07.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:07 smithi203 ceph-mon[32237]: pgmap v967: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 151 active+clean; 5.8 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 680 op/s; 28435/286623 objects misplaced (9.921%); 27 MiB/s, 432 objects/s recovering 2024-09-18T07:49:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:07 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2653980143' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:49:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:07 smithi203 ceph-mon[32237]: osdmap e508: 8 total, 8 up, 7 in 2024-09-18T07:49:07.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:07.763 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:07.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:07.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:07.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:07.903 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:07.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:07.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:08.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:08.043 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:08.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:08.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:08.145 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:08.174 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:08.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:08.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:49:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:49:08.276 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:08.308 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:08.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:08.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:08.410 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:08.444 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:08.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:08.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:08.588 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:08.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:08.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:08.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:08 smithi137 ceph-mon[29408]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:49:08.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:08 smithi137 ceph-mon[39175]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:49:08.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:08.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:08 smithi203 ceph-mon[32237]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:49:08.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:08.730 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:08.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:08.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:08.865 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:08.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:08.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:08.967 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:09.002 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:09.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:09.104 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:09.125 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:09.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:09.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:09.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:09.268 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:09.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:09.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:09.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:09.401 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:09.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:09.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:09.503 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:09.537 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:09.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:09.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:09.639 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:09.668 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:09.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:09.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:09.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:09 smithi137 ceph-mon[29408]: pgmap v969: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 923 op/s; 24858/276138 objects misplaced (9.002%); 55 MiB/s, 872 objects/s recovering 2024-09-18T07:49:09.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:09 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:09.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:09 smithi137 ceph-mon[29408]: 3.2 scrub starts 2024-09-18T07:49:09.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:09 smithi137 ceph-mon[29408]: 3.2 scrub ok 2024-09-18T07:49:09.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:09 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:49:09.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:09 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:49:09.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:09 smithi137 ceph-mon[39175]: pgmap v969: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 923 op/s; 24858/276138 objects misplaced (9.002%); 55 MiB/s, 872 objects/s recovering 2024-09-18T07:49:09.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:09 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:09.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:09 smithi137 ceph-mon[39175]: 3.2 scrub starts 2024-09-18T07:49:09.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:09 smithi137 ceph-mon[39175]: 3.2 scrub ok 2024-09-18T07:49:09.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:09 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:49:09.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:09 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:49:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:09 smithi203 ceph-mon[32237]: pgmap v969: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 923 op/s; 24858/276138 objects misplaced (9.002%); 55 MiB/s, 872 objects/s recovering 2024-09-18T07:49:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:09 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:09 smithi203 ceph-mon[32237]: 3.2 scrub starts 2024-09-18T07:49:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:09 smithi203 ceph-mon[32237]: 3.2 scrub ok 2024-09-18T07:49:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:09 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:49:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:09 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:49:09.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:09.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:09.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:09.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:09.908 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:09.936 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:09.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:09.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:10.038 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:10.072 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:10.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:10.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:10.206 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:10.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:10.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:10.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:10.345 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:10.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:10.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:10.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:10.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:10.501 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:10.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:10.602 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:10.628 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:10 smithi137 ceph-mon[29408]: osdmap e509: 8 total, 8 up, 7 in 2024-09-18T07:49:10.628 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:10 smithi137 ceph-mon[39175]: osdmap e509: 8 total, 8 up, 7 in 2024-09-18T07:49:10.629 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:10.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:10.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:10.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:10 smithi203 ceph-mon[32237]: osdmap e509: 8 total, 8 up, 7 in 2024-09-18T07:49:10.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:10.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:10.771 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:10.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:10.872 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:10.903 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:10.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:10.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:11.005 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:11.038 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:11.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:11.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:11.140 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:11.169 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:11.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:11.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:11.270 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:11.301 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:11.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:11.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:11.403 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:11.449 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T07:49:11.464 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:11.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:11.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:11.566 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:11.597 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:11.597 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:11 smithi137 ceph-mon[29408]: pgmap v971: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 720 op/s; 24060/274356 objects misplaced (8.770%); 33 MiB/s, 534 objects/s recovering 2024-09-18T07:49:11.598 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:11 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:49:11.598 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:11 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:49:11.598 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:11 smithi137 ceph-mon[39175]: pgmap v971: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 720 op/s; 24060/274356 objects misplaced (8.770%); 33 MiB/s, 534 objects/s recovering 2024-09-18T07:49:11.598 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:11 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:49:11.598 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:11 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:49:11.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:11.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:11.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:11.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:11 smithi203 ceph-mon[32237]: pgmap v971: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 720 op/s; 24060/274356 objects misplaced (8.770%); 33 MiB/s, 534 objects/s recovering 2024-09-18T07:49:11.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:11 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:49:11.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:11 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:49:11.746 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:11.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:11.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:11.849 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:11.889 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:11.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:11.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:11.991 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:12.018 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:12.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:12.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:12.021 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-18T07:49:12.057 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:12.095 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:12.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:12.171 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:12.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:12.273 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:12.308 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:12.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:12.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:12.390 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:49:12.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:12.495 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:12 smithi137 ceph-mon[29408]: osdmap e510: 8 total, 8 up, 7 in 2024-09-18T07:49:12.496 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:12 smithi137 ceph-mon[39175]: osdmap e510: 8 total, 8 up, 7 in 2024-09-18T07:49:12.504 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:12.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:12.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:12.606 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:12.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:12.677 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:12.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:12.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:12 smithi203 ceph-mon[32237]: osdmap e510: 8 total, 8 up, 7 in 2024-09-18T07:49:12.778 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:12.804 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:12.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:12.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:12.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:12.965 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:12.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:13.035 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:49:13.035 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:49:13.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:13.073 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:49:13.073 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:49:13.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:13.110 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:13.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:13.111 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:49:13.111 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:49:13.152 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:49:13.153 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:49:13.190 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:49:13.190 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:49:13.211 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:13.231 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:49:13.231 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:49:13.238 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:13.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:13.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:13.264 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:49:13.264 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:49:13.286 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:49:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:49:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:139275 11.c 11:324b2854:::benchmark_data_smithi137_181069_object27121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e510)", 2024-09-18T07:49:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:49:13.263771+0000", 2024-09-18T07:49:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.017502354000000001, 2024-09-18T07:49:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017533917, 2024-09-18T07:49:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:49:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:13.286 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 139275 2024-09-18T07:49:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.263771+0000", 2024-09-18T07:49:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.263771+0000", 2024-09-18T07:49:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.263773+0000", 2024-09-18T07:49:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5639999999999999e-06 2024-09-18T07:49:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.263781+0000", 2024-09-18T07:49:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5109999999999997e-06 2024-09-18T07:49:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:13.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.263794+0000", 2024-09-18T07:49:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3550000000000001e-05 2024-09-18T07:49:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.263802+0000", 2024-09-18T07:49:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6969999999999993e-06 2024-09-18T07:49:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.265001+0000", 2024-09-18T07:49:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011990130000000001 2024-09-18T07:49:13.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.265127+0000", 2024-09-18T07:49:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012582399999999999 2024-09-18T07:49:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:49:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.265252+0000", 2024-09-18T07:49:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000124518 2024-09-18T07:49:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.271748+0000", 2024-09-18T07:49:13.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0064965379999999996 2024-09-18T07:49:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.273050+0000", 2024-09-18T07:49:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001301459 2024-09-18T07:49:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:139278 11.a e510/500 11:588c45aa:::benchmark_data_smithi137_181069_object27124:head v 510'8642, mlcod=510'8638)", 2024-09-18T07:49:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:49:13.267695+0000", 2024-09-18T07:49:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.013578224999999999, 2024-09-18T07:49:13.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013692149000000001, 2024-09-18T07:49:13.292 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:13.292 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:13.292 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:49:13.292 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:13.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:13.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.267695+0000", 2024-09-18T07:49:13.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:13.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:13.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.267695+0000", 2024-09-18T07:49:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.267696+0000", 2024-09-18T07:49:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.423e-06 2024-09-18T07:49:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.267704+0000", 2024-09-18T07:49:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.379e-06 2024-09-18T07:49:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.267714+0000", 2024-09-18T07:49:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.044e-05 2024-09-18T07:49:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.267721+0000", 2024-09-18T07:49:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9489999999999999e-06 2024-09-18T07:49:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.267789+0000", 2024-09-18T07:49:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7335999999999994e-05 2024-09-18T07:49:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.267809+0000", 2024-09-18T07:49:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0217e-05 2024-09-18T07:49:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:139280 11.c 11:39dacbaa:::benchmark_data_smithi137_181069_object27126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e510)", 2024-09-18T07:49:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:49:13.276203+0000", 2024-09-18T07:49:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0050706379999999997, 2024-09-18T07:49:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052401059999999996, 2024-09-18T07:49:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:13.296 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-18T07:49:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 139280 2024-09-18T07:49:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276203+0000", 2024-09-18T07:49:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:13.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276203+0000", 2024-09-18T07:49:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276205+0000", 2024-09-18T07:49:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.165e-06 2024-09-18T07:49:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276210+0000", 2024-09-18T07:49:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5990000000000001e-06 2024-09-18T07:49:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:13.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276223+0000", 2024-09-18T07:49:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2864000000000001e-05 2024-09-18T07:49:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276232+0000", 2024-09-18T07:49:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2959999999999999e-06 2024-09-18T07:49:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.281209+0000", 2024-09-18T07:49:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0049779189999999999 2024-09-18T07:49:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.281335+0000", 2024-09-18T07:49:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012604099999999999 2024-09-18T07:49:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:49:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.281453+0000", 2024-09-18T07:49:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117312 2024-09-18T07:49:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.300 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:139282 11.c 11.1066abfc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e510)", 2024-09-18T07:49:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:49:13.276253+0000", 2024-09-18T07:49:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0050204769999999998, 2024-09-18T07:49:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052525599999999999, 2024-09-18T07:49:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:49:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 139282 2024-09-18T07:49:13.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276253+0000", 2024-09-18T07:49:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276253+0000", 2024-09-18T07:49:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276253+0000", 2024-09-18T07:49:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5700000000000002e-07 2024-09-18T07:49:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276256+0000", 2024-09-18T07:49:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5220000000000002e-06 2024-09-18T07:49:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276261+0000", 2024-09-18T07:49:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0919999999999997e-06 2024-09-18T07:49:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276265+0000", 2024-09-18T07:49:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5770000000000004e-06 2024-09-18T07:49:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:139286 11.c 11.b7cff9cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e510)", 2024-09-18T07:49:13.304 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:49:13.276271+0000", 2024-09-18T07:49:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0050020610000000004, 2024-09-18T07:49:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052829629999999999, 2024-09-18T07:49:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "queued for pg", 2024-09-18T07:49:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 139286 2024-09-18T07:49:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:13.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276271+0000", 2024-09-18T07:49:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276271+0000", 2024-09-18T07:49:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276272+0000", 2024-09-18T07:49:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9000000000000002e-07 2024-09-18T07:49:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276274+0000", 2024-09-18T07:49:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.973e-06 2024-09-18T07:49:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276278+0000", 2024-09-18T07:49:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1840000000000001e-06 2024-09-18T07:49:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276282+0000", 2024-09-18T07:49:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9299999999999996e-06 2024-09-18T07:49:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:13.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:139289 11.d 11:bb587e15:::benchmark_data_smithi137_181069_object27135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e510)", 2024-09-18T07:49:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:49:13.276287+0000", 2024-09-18T07:49:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.0049866270000000004, 2024-09-18T07:49:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053185389999999997, 2024-09-18T07:49:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:49:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 139289 2024-09-18T07:49:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.310 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276287+0000", 2024-09-18T07:49:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276287+0000", 2024-09-18T07:49:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276287+0000", 2024-09-18T07:49:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7099999999999997e-07 2024-09-18T07:49:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276289+0000", 2024-09-18T07:49:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0080000000000001e-06 2024-09-18T07:49:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276293+0000", 2024-09-18T07:49:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1230000000000002e-06 2024-09-18T07:49:13.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276297+0000", 2024-09-18T07:49:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9600000000000002e-06 2024-09-18T07:49:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276358+0000", 2024-09-18T07:49:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1084e-05 2024-09-18T07:49:13.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276469+0000", 2024-09-18T07:49:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011100200000000001 2024-09-18T07:49:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4,6", 2024-09-18T07:49:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:49:13.276573+0000", 2024-09-18T07:49:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010425999999999999 2024-09-18T07:49:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:13.314 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:13.315 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:13.315 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:49:13.315 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 6 2024-09-18T07:49:13.315 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:49:13.315 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:49:13.315 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:49:13.339 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:13.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:13.389 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:13.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:13 smithi137 ceph-mon[29408]: pgmap v973: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.16k op/s; 22947/265752 objects misplaced (8.635%); 39 MiB/s, 618 objects/s recovering 2024-09-18T07:49:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:13 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:49:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:13 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:49:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:13 smithi137 ceph-mon[29408]: 1.3 scrub starts 2024-09-18T07:49:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:13 smithi137 ceph-mon[29408]: 1.3 scrub ok 2024-09-18T07:49:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:13 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/110647098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:13.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:13 smithi137 ceph-mon[39175]: pgmap v973: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.16k op/s; 22947/265752 objects misplaced (8.635%); 39 MiB/s, 618 objects/s recovering 2024-09-18T07:49:13.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:13 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:49:13.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:13 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:49:13.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:13 smithi137 ceph-mon[39175]: 1.3 scrub starts 2024-09-18T07:49:13.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:13 smithi137 ceph-mon[39175]: 1.3 scrub ok 2024-09-18T07:49:13.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:13 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/110647098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:13.489 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:13.523 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:13.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:13.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:13.625 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:13.663 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:13.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:13.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:13.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:13 smithi203 ceph-mon[32237]: pgmap v973: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.16k op/s; 22947/265752 objects misplaced (8.635%); 39 MiB/s, 618 objects/s recovering 2024-09-18T07:49:13.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:13 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:49:13.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:13 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:49:13.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:13 smithi203 ceph-mon[32237]: 1.3 scrub starts 2024-09-18T07:49:13.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:13 smithi203 ceph-mon[32237]: 1.3 scrub ok 2024-09-18T07:49:13.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:13 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/110647098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:13.764 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:13.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:13.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:13.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:13.903 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:13.934 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:13.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:13.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:14.035 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:14.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:14.070 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:14.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:14.113 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T07:49:14.171 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:14.246 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:14.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:14.348 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:14.377 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:14.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:14.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:14.479 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:14.512 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:14.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:14.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:14.614 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:14.640 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:14.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:14.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:14.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:14.770 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:14.794 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:14.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:14.811 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:14.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:14.912 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:14.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:14.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:14.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:15.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:15.099 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:15.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:15.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:15.201 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:15.230 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:15.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:15.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:15.332 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:15.364 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:15.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:15.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:15 smithi203 ceph-mon[32237]: pgmap v974: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.17k op/s; 20083/259854 objects misplaced (7.729%); 48 MiB/s, 770 objects/s recovering 2024-09-18T07:49:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:15 smithi203 ceph-mon[32237]: 5.1b scrub starts 2024-09-18T07:49:15.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:15 smithi203 ceph-mon[32237]: 5.1b scrub ok 2024-09-18T07:49:15.466 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:15.491 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:15.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:15.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:15.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:15.629 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:15.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:15.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:15.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:15 smithi137 ceph-mon[29408]: pgmap v974: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.17k op/s; 20083/259854 objects misplaced (7.729%); 48 MiB/s, 770 objects/s recovering 2024-09-18T07:49:15.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:15 smithi137 ceph-mon[29408]: 5.1b scrub starts 2024-09-18T07:49:15.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:15 smithi137 ceph-mon[29408]: 5.1b scrub ok 2024-09-18T07:49:15.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:15 smithi137 ceph-mon[39175]: pgmap v974: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.17k op/s; 20083/259854 objects misplaced (7.729%); 48 MiB/s, 770 objects/s recovering 2024-09-18T07:49:15.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:15 smithi137 ceph-mon[39175]: 5.1b scrub starts 2024-09-18T07:49:15.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:15 smithi137 ceph-mon[39175]: 5.1b scrub ok 2024-09-18T07:49:15.731 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:15.761 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:15.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:15.863 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:15.886 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:15.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:15.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:15.988 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:16.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:16.022 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:16.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:16.123 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:16.153 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:16.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:16.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:16.254 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:16.275 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:16.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:16.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:16.377 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:16.411 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:16.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:16.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:16.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:16.539 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:16.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:16.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:16.641 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:16.674 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:16.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:16.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:16.753 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:49:16.776 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:16.809 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:16.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:16.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:16.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:16.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:16.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:16.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:17.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:17.107 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:49:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:49:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:49:17.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:17.118 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:17.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:17.219 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:17.243 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:17.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:17.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:17.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:17.390 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:17.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:17.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:17.423 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:17.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:17.530 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:17.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:17.632 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:17.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:17.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:17 smithi137 ceph-mon[29408]: pgmap v975: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 721 op/s; 20083/258840 objects misplaced (7.759%); 26 MiB/s, 423 objects/s recovering 2024-09-18T07:49:17.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:17 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:49:17.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:17 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:49:17.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:17.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:17.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:17 smithi137 ceph-mon[39175]: pgmap v975: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 721 op/s; 20083/258840 objects misplaced (7.759%); 26 MiB/s, 423 objects/s recovering 2024-09-18T07:49:17.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:17 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:49:17.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:17 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:49:17.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:17 smithi203 ceph-mon[32237]: pgmap v975: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 721 op/s; 20083/258840 objects misplaced (7.759%); 26 MiB/s, 423 objects/s recovering 2024-09-18T07:49:17.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:17 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:49:17.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:17 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:49:17.771 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:17.803 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:17.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:17.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:17.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:17.945 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:17.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:17.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:18.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:18.083 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:18.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:18.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:18.185 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:18.219 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:18.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:18.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:18.320 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:18.356 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:18.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:18.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:18.390 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:49:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:49:18.402 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.7 to 1 (10000) 2024-09-18T07:49:18.457 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:18.487 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:18.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:18.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:18.553 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:18 smithi137 ceph-mon[29408]: osdmap e511: 8 total, 8 up, 7 in 2024-09-18T07:49:18.553 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:18 smithi137 ceph-mon[29408]: 6.1d scrub starts 2024-09-18T07:49:18.553 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:18 smithi137 ceph-mon[29408]: 6.1d scrub ok 2024-09-18T07:49:18.553 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:18 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1631307057' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:49:18.553 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:18 smithi137 ceph-mon[29408]: from='client.36611 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:49:18.553 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:18 smithi137 ceph-mon[29408]: pgmap v977: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.18k op/s; 16955/246042 objects misplaced (6.891%); 37 MiB/s, 598 objects/s recovering 2024-09-18T07:49:18.553 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:18 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:18.553 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:18 smithi137 ceph-mon[39175]: osdmap e511: 8 total, 8 up, 7 in 2024-09-18T07:49:18.554 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:18 smithi137 ceph-mon[39175]: 6.1d scrub starts 2024-09-18T07:49:18.554 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:18 smithi137 ceph-mon[39175]: 6.1d scrub ok 2024-09-18T07:49:18.554 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:18 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1631307057' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:49:18.554 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:18 smithi137 ceph-mon[39175]: from='client.36611 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:49:18.554 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:18 smithi137 ceph-mon[39175]: pgmap v977: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.18k op/s; 16955/246042 objects misplaced (6.891%); 37 MiB/s, 598 objects/s recovering 2024-09-18T07:49:18.554 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:18 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:18.589 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:18.614 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:18.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:18.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:18.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:18 smithi203 ceph-mon[32237]: osdmap e511: 8 total, 8 up, 7 in 2024-09-18T07:49:18.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:18 smithi203 ceph-mon[32237]: 6.1d scrub starts 2024-09-18T07:49:18.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:18 smithi203 ceph-mon[32237]: 6.1d scrub ok 2024-09-18T07:49:18.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:18 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1631307057' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:49:18.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:18 smithi203 ceph-mon[32237]: from='client.36611 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:49:18.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:18 smithi203 ceph-mon[32237]: pgmap v977: 158 pgs: 1 active+remapped+backfill_wait, 5 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.18k op/s; 16955/246042 objects misplaced (6.891%); 37 MiB/s, 598 objects/s recovering 2024-09-18T07:49:18.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:18 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:18.716 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:18.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:18.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:18.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:18.868 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:18.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:18.970 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:18.995 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:18.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:18.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:19.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:19.124 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:19.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:19.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:19.185 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:49:19.185 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:49:19.185 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:49:19.185 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:49:19.185 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:49:19.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:19.253 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:19.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:19.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:19.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:19.380 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:19.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:19.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:19.386 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:49:19.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:19.518 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:19.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:19.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:19.620 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:19.652 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:19.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:19.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:19.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:19 smithi137 ceph-mon[29408]: from='client.36611 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T07:49:19.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:19 smithi137 ceph-mon[29408]: osdmap e512: 8 total, 8 up, 7 in 2024-09-18T07:49:19.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:19 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1001617597' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:49:19.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:19 smithi137 ceph-mon[39175]: from='client.36611 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T07:49:19.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:19 smithi137 ceph-mon[39175]: osdmap e512: 8 total, 8 up, 7 in 2024-09-18T07:49:19.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:19 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1001617597' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:49:19.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:19 smithi203 ceph-mon[32237]: from='client.36611 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T07:49:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:19 smithi203 ceph-mon[32237]: osdmap e512: 8 total, 8 up, 7 in 2024-09-18T07:49:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:19 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1001617597' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:49:19.753 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:19.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-18T07:49:19.786 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:19.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:19.888 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:19.919 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:19.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:19.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:20.021 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:20.051 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:20.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:20.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:20.153 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:20.176 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:20.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:20.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:20.278 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:20.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:20.320 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:20.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:20.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:20.464 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:20.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:20.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:20.566 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:20.598 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:20.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:20.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:20.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:20 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:49:20.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:20 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1001617597' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:49:20.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:20 smithi137 ceph-mon[29408]: osdmap e513: 8 total, 8 up, 7 in 2024-09-18T07:49:20.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:20 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:49:20.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:20 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:49:20.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:20 smithi137 ceph-mon[29408]: pgmap v980: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 16346/244275 objects misplaced (6.692%); 28 MiB/s, 450 objects/s recovering 2024-09-18T07:49:20.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:20 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:49:20.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:20 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1001617597' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:49:20.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:20 smithi137 ceph-mon[39175]: osdmap e513: 8 total, 8 up, 7 in 2024-09-18T07:49:20.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:20 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:49:20.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:20 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:49:20.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:20 smithi137 ceph-mon[39175]: pgmap v980: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 16346/244275 objects misplaced (6.692%); 28 MiB/s, 450 objects/s recovering 2024-09-18T07:49:20.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:20.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:20 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:49:20.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:20 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1001617597' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:49:20.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:20 smithi203 ceph-mon[32237]: osdmap e513: 8 total, 8 up, 7 in 2024-09-18T07:49:20.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:20 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:49:20.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:20 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:49:20.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:20 smithi203 ceph-mon[32237]: pgmap v980: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 151 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 16346/244275 objects misplaced (6.692%); 28 MiB/s, 450 objects/s recovering 2024-09-18T07:49:20.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:20.738 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:20.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:20.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T07:49:20.840 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:20.886 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:20.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:20.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:20.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:21.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:21.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:21.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:21.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:21.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:21.222 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:21.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:21.241 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:21.270 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:21.323 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:21.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:21.364 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:21.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:21.465 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:21.494 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:21.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:21.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:21.550 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:21.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:21.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:21.618 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:21.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:21.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:21 smithi137 ceph-mon[29408]: osdmap e514: 8 total, 8 up, 7 in 2024-09-18T07:49:21.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:21 smithi137 ceph-mon[39175]: osdmap e514: 8 total, 8 up, 7 in 2024-09-18T07:49:21.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:21 smithi203 ceph-mon[32237]: osdmap e514: 8 total, 8 up, 7 in 2024-09-18T07:49:21.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:21.744 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:21.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:21.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:21.846 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:21.867 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:21.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:21.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:21.969 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:21.993 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:21.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:21.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:22.095 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:22.118 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:22.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:22.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:22.220 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:22.244 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:22.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:22.316 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:49:22.316 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:49:22.336 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:49:22.337 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:49:22.346 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:22.357 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:49:22.357 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:49:22.370 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:22.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:22.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:22.377 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:49:22.377 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:49:22.397 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:49:22.397 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:49:22.417 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:49:22.417 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:49:22.441 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:49:22.441 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:49:22.462 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:49:22.462 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:49:22.472 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:22.500 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:22.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:22.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:22.582 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:22 smithi137 ceph-mon[29408]: osdmap e515: 8 total, 8 up, 7 in 2024-09-18T07:49:22.582 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:22 smithi137 ceph-mon[29408]: pgmap v983: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 152 active+clean; 4.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 750 op/s; 15006/237054 objects misplaced (6.330%); 24 MiB/s, 381 objects/s recovering 2024-09-18T07:49:22.582 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:22 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2505598995' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:22.582 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:22 smithi137 ceph-mon[39175]: osdmap e515: 8 total, 8 up, 7 in 2024-09-18T07:49:22.582 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:22 smithi137 ceph-mon[39175]: pgmap v983: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 152 active+clean; 4.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 750 op/s; 15006/237054 objects misplaced (6.330%); 24 MiB/s, 381 objects/s recovering 2024-09-18T07:49:22.582 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:22 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2505598995' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:22.602 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:22.629 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:22.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:22.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:22 smithi203 ceph-mon[32237]: osdmap e515: 8 total, 8 up, 7 in 2024-09-18T07:49:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:22 smithi203 ceph-mon[32237]: pgmap v983: 158 pgs: 1 active+remapped+backfill_wait, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 152 active+clean; 4.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 750 op/s; 15006/237054 objects misplaced (6.330%); 24 MiB/s, 381 objects/s recovering 2024-09-18T07:49:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:22 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2505598995' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:22.732 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:22.772 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:22.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:22.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:22.874 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:22.902 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:22.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:22.906 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:49:22.906 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:49:23.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:23.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:23.045 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:23.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:23.146 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:23.177 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:23.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:23.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:23.205 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:49:23.278 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:23.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:23.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:23.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:23.427 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:23.474 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:23.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:23.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:23.576 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:23.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:23.613 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:23.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:23.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:23 smithi137 ceph-mon[29408]: osdmap e516: 8 total, 8 up, 7 in 2024-09-18T07:49:23.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:23 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1352598911' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:49:23.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:23 smithi137 ceph-mon[39175]: osdmap e516: 8 total, 8 up, 7 in 2024-09-18T07:49:23.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:23 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1352598911' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:49:23.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:23 smithi203 ceph-mon[32237]: osdmap e516: 8 total, 8 up, 7 in 2024-09-18T07:49:23.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:23 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1352598911' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:49:23.714 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:23.745 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:23.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:23.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:23.847 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:23.873 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:23.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:23.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:23.975 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:23.996 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:23.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:23.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:24.098 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:24.121 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:24.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:24.223 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:24.245 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:24.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:24.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:24.347 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:24.370 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:24.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:24.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:24.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:24.495 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:24.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:24.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:24.597 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:24.632 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:24.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:24.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:24.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:24 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:49:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:24 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:49:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:24 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1352598911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:49:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:24 smithi203 ceph-mon[32237]: osdmap e517: 8 total, 8 up, 7 in 2024-09-18T07:49:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:24 smithi203 ceph-mon[32237]: pgmap v986: 158 pgs: 1 peering, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 889 op/s; 11684/233607 objects misplaced (5.002%); 62 MiB/s, 994 objects/s recovering 2024-09-18T07:49:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:24 smithi203 ceph-mon[32237]: osdmap e518: 8 total, 8 up, 7 in 2024-09-18T07:49:24.734 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:24.761 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:24.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:24.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:24.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:24.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:24.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:24.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:24.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:24 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:49:24.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:24 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:49:24.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:24 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1352598911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:49:24.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:24 smithi137 ceph-mon[29408]: osdmap e517: 8 total, 8 up, 7 in 2024-09-18T07:49:24.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:24 smithi137 ceph-mon[29408]: pgmap v986: 158 pgs: 1 peering, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 889 op/s; 11684/233607 objects misplaced (5.002%); 62 MiB/s, 994 objects/s recovering 2024-09-18T07:49:24.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:24 smithi137 ceph-mon[29408]: osdmap e518: 8 total, 8 up, 7 in 2024-09-18T07:49:24.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:24 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:49:24.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:24 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:49:24.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:24 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1352598911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:49:24.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:24 smithi137 ceph-mon[39175]: osdmap e517: 8 total, 8 up, 7 in 2024-09-18T07:49:24.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:24 smithi137 ceph-mon[39175]: pgmap v986: 158 pgs: 1 peering, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 153 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 889 op/s; 11684/233607 objects misplaced (5.002%); 62 MiB/s, 994 objects/s recovering 2024-09-18T07:49:24.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:24 smithi137 ceph-mon[39175]: osdmap e518: 8 total, 8 up, 7 in 2024-09-18T07:49:24.989 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:25.014 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:25.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:25.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:25.117 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:25.142 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:25.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:25.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:25.244 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:25.269 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:25.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:25.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:25.371 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:25.396 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:25.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:25.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:25.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:25.534 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:25.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:25.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:25.549 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T07:49:25.636 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:25.678 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.6 to 1 (10000) 2024-09-18T07:49:25.690 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:25.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:25.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:25.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:25 smithi203 ceph-mon[32237]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 516'9394 2024-09-18T07:49:25.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:25.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:25.829 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:25.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:25.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:25 smithi137 ceph-mon[39175]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 516'9394 2024-09-18T07:49:25.921 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:25 smithi137 ceph-mon[29408]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 516'9394 2024-09-18T07:49:25.930 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:25.973 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:25.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:25.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:26.075 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:26.108 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:26.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:26.210 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:26.246 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:26.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:26.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:26.348 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:26.379 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:26.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:26.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:26.481 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:26.486 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-18T07:49:26.506 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:26.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:26.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:26.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:26.589 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:26.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:26.695 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:26 smithi137 ceph-mon[29408]: osdmap e519: 8 total, 8 up, 7 in 2024-09-18T07:49:26.695 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:26 smithi137 ceph-mon[29408]: pgmap v989: 158 pgs: 2 peering, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 354 op/s; 11684/232803 objects misplaced (5.019%); 48 MiB/s, 762 objects/s recovering 2024-09-18T07:49:26.696 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:26.696 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:26 smithi137 ceph-mon[39175]: osdmap e519: 8 total, 8 up, 7 in 2024-09-18T07:49:26.696 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:26 smithi137 ceph-mon[39175]: pgmap v989: 158 pgs: 2 peering, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 354 op/s; 11684/232803 objects misplaced (5.019%); 48 MiB/s, 762 objects/s recovering 2024-09-18T07:49:26.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:26.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:26.798 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:26.860 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:26.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:26.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:26.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:26 smithi203 ceph-mon[32237]: osdmap e519: 8 total, 8 up, 7 in 2024-09-18T07:49:26.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:26 smithi203 ceph-mon[32237]: pgmap v989: 158 pgs: 2 peering, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 354 op/s; 11684/232803 objects misplaced (5.019%); 48 MiB/s, 762 objects/s recovering 2024-09-18T07:49:26.962 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:27.002 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:27.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:27.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:27.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:49:27.104 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:27.149 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:27.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:27.213 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:49:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:49:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:49:27.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:27.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:27.287 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:27.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:27.388 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:27.420 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:27.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:27.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:27.522 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:27.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:27.552 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:27.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:27.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:27.676 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:27.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:27.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:27.778 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:27.803 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:27.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:27.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:27.905 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:27.937 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:27.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:27.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:28.039 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:28.069 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:28.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:28.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:28.140 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:49:28.140 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:49:28.171 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:28.207 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:28.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:28.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:28.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:28.347 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:28.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:28.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:28.417 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:49:27 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:49:28.432 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:49:28.449 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:28.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:28.482 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:28.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:28.583 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:28.617 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:28.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:28.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:28.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:28 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:49:28.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:28 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:49:28.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:28 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4075694641' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:28.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:28 smithi137 ceph-mon[29408]: pgmap v990: 158 pgs: 1 peering, 3 active+remapped+backfilling, 154 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 939 B/s rd, 0 B/s wr, 663 op/s; 7276/222774 objects misplaced (3.266%); 46 MiB/s, 737 objects/s recovering 2024-09-18T07:49:28.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:28 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:28.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:28 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:49:28.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:28 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:49:28.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:28 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4075694641' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:28.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:28 smithi137 ceph-mon[39175]: pgmap v990: 158 pgs: 1 peering, 3 active+remapped+backfilling, 154 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 939 B/s rd, 0 B/s wr, 663 op/s; 7276/222774 objects misplaced (3.266%); 46 MiB/s, 737 objects/s recovering 2024-09-18T07:49:28.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:28 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:28.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:28 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:49:28.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:28 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:49:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:28 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4075694641' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:28 smithi203 ceph-mon[32237]: pgmap v990: 158 pgs: 1 peering, 3 active+remapped+backfilling, 154 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 939 B/s rd, 0 B/s wr, 663 op/s; 7276/222774 objects misplaced (3.266%); 46 MiB/s, 737 objects/s recovering 2024-09-18T07:49:28.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:28 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:28.719 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:28.754 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:28.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:28.855 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:28.895 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:28.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:28.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:28.997 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:29.035 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:29.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:29.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:29.137 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:29.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:29.175 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:29.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:29.276 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:29.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:29.320 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:29.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:29.420 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:29.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:29 smithi137 ceph-mon[29408]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-18T07:49:29.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:29 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:49:29.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:29 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4075694641' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:49:29.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:29 smithi137 ceph-mon[29408]: osdmap e520: 8 total, 8 up, 7 in 2024-09-18T07:49:29.422 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:29 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:49:29.459 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:29.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:29.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:29.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:29.602 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:29.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:29.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:29.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:29.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:29 smithi203 ceph-mon[32237]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-18T07:49:29.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:29 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:49:29.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:29 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4075694641' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:49:29.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:29 smithi203 ceph-mon[32237]: osdmap e520: 8 total, 8 up, 7 in 2024-09-18T07:49:29.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:29 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:49:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:29.744 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:29.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:29.784 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:29 smithi137 ceph-mon[39175]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-18T07:49:29.784 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:29 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:49:29.785 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:29 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4075694641' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:49:29.785 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:29 smithi137 ceph-mon[39175]: osdmap e520: 8 total, 8 up, 7 in 2024-09-18T07:49:29.785 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:29 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:49:29.845 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:29.858 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:29.859 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T07:49:29.873 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:29.874 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:29.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:29.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:29.976 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:30.009 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:30.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:30.111 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:30.140 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:30.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:30.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:30.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:30.287 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:30.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:30.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:30.389 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:30.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:30.422 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:30.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:30.523 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:30.558 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:30.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:30.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:30.650 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:30 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:49:30.650 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:30 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:49:30.650 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:30 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:49:30.650 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:30 smithi137 ceph-mon[29408]: pgmap v992: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 651 op/s; 7276/221880 objects misplaced (3.279%); 42 MiB/s, 669 objects/s recovering 2024-09-18T07:49:30.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:30 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:49:30.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:30 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:49:30.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:30 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:49:30.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:30 smithi137 ceph-mon[39175]: pgmap v992: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 651 op/s; 7276/221880 objects misplaced (3.279%); 42 MiB/s, 669 objects/s recovering 2024-09-18T07:49:30.661 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:30.696 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:30.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:30.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:30.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:30 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:49:30.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:30 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:49:30.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:30 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:49:30.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:30 smithi203 ceph-mon[32237]: pgmap v992: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 651 op/s; 7276/221880 objects misplaced (3.279%); 42 MiB/s, 669 objects/s recovering 2024-09-18T07:49:30.798 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:30.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:30.833 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:30.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:30.935 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:30.980 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:30.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:30.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:31.082 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:31.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:31.114 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:31.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:31.164 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:31.176 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:49:31.176 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:49:31.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:31.219 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:49:31.219 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:49:31.248 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:49:31.248 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:49:31.252 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:31.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:31.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:31.282 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:49:31.282 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:49:31.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:49:31.316 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:49:31.316 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:49:31.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:31.356 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:49:31.357 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:49:31.386 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:49:31.387 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:49:31.407 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:31.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:31.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:31.423 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:49:31.423 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:49:31.509 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:31.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:31.543 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:31.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:31.631 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.1 to 1 (10000) 2024-09-18T07:49:31.643 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:31.673 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:31.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:31.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:31.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:31.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:31.797 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:31.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:31.898 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:31.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:31 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2172601853' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:49:31.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:31 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2464551374' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:31.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:31 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2172601853' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:49:31.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:31 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2464551374' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:31.927 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:31.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:31.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:31 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2172601853' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:49:31.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:31 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2464551374' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:32.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:32.053 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:32.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:32.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:32.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:32.178 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:32.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:32.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:32.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:32.302 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:32.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:32.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:32.404 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:32.429 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:32.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:32.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:32.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:32.553 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:32.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:32.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:32.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:32.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:32.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:32.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:32.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:32.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:32.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:32.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:32 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2172601853' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:49:32.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:32 smithi137 ceph-mon[29408]: osdmap e521: 8 total, 8 up, 7 in 2024-09-18T07:49:32.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:32 smithi137 ceph-mon[29408]: pgmap v994: 158 pgs: 1 peering, 3 active+remapped+backfilling, 154 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.11k op/s; 5173/211371 objects misplaced (2.447%); 57 MiB/s, 906 objects/s recovering 2024-09-18T07:49:32.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:32 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2172601853' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:49:32.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:32 smithi137 ceph-mon[39175]: osdmap e521: 8 total, 8 up, 7 in 2024-09-18T07:49:32.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:32 smithi137 ceph-mon[39175]: pgmap v994: 158 pgs: 1 peering, 3 active+remapped+backfilling, 154 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.11k op/s; 5173/211371 objects misplaced (2.447%); 57 MiB/s, 906 objects/s recovering 2024-09-18T07:49:32.933 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:32.957 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:32.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:32.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:32.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:32 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2172601853' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:49:32.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:32 smithi203 ceph-mon[32237]: osdmap e521: 8 total, 8 up, 7 in 2024-09-18T07:49:32.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:32 smithi203 ceph-mon[32237]: pgmap v994: 158 pgs: 1 peering, 3 active+remapped+backfilling, 154 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.11k op/s; 5173/211371 objects misplaced (2.447%); 57 MiB/s, 906 objects/s recovering 2024-09-18T07:49:33.059 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:33.082 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:33.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:33.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:33.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:33.205 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:33.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:33.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:33.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:33.329 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:33.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:33.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:33.431 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:33.453 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:33.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:33.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:33.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:33.576 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:33.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:33.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:33.603 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:49:33.603 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:49:33.603 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:49:33.603 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:49:33.604 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:49:33.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:33.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:33.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:33.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:33.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:33.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:33.824 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:33.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:33.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:33 smithi137 ceph-mon[29408]: osdmap e522: 8 total, 8 up, 7 in 2024-09-18T07:49:33.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:33 smithi137 ceph-mon[39175]: osdmap e522: 8 total, 8 up, 7 in 2024-09-18T07:49:33.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:33.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:33.954 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:33.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:33.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:33 smithi203 ceph-mon[32237]: osdmap e522: 8 total, 8 up, 7 in 2024-09-18T07:49:34.054 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:34.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:34.087 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:34.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:34.188 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:34.210 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:34.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:34.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:34.312 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:34.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:34.351 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:34.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:34.452 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:34.487 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:34.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:34.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:34.589 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:34.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:34.622 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:34.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:34.723 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:34.759 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:34.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:34.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:34.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:34.906 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:34.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:34.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:34 smithi137 ceph-mon[29408]: osdmap e523: 8 total, 8 up, 7 in 2024-09-18T07:49:34.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:34 smithi137 ceph-mon[29408]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 520'9676 2024-09-18T07:49:34.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:34 smithi137 ceph-mon[29408]: pgmap v997: 158 pgs: 5 peering, 3 active+remapped+backfilling, 150 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 939 B/s rd, 0 B/s wr, 988 op/s; 4266/206622 objects misplaced (2.065%); 30 MiB/s, 478 objects/s recovering 2024-09-18T07:49:34.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:34 smithi137 ceph-mon[39175]: osdmap e523: 8 total, 8 up, 7 in 2024-09-18T07:49:34.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:34 smithi137 ceph-mon[39175]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 520'9676 2024-09-18T07:49:34.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:34 smithi137 ceph-mon[39175]: pgmap v997: 158 pgs: 5 peering, 3 active+remapped+backfilling, 150 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 939 B/s rd, 0 B/s wr, 988 op/s; 4266/206622 objects misplaced (2.065%); 30 MiB/s, 478 objects/s recovering 2024-09-18T07:49:34.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:34 smithi203 ceph-mon[32237]: osdmap e523: 8 total, 8 up, 7 in 2024-09-18T07:49:34.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:34 smithi203 ceph-mon[32237]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 520'9676 2024-09-18T07:49:34.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:34 smithi203 ceph-mon[32237]: pgmap v997: 158 pgs: 5 peering, 3 active+remapped+backfilling, 150 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 939 B/s rd, 0 B/s wr, 988 op/s; 4266/206622 objects misplaced (2.065%); 30 MiB/s, 478 objects/s recovering 2024-09-18T07:49:35.007 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:35.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:35.042 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:35.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:35.143 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:35.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:35.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:35.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:35.278 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:35.300 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:35.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:35.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:35.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:35.432 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:35.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:35.534 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:35.567 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:35.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:35.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:35.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:35.708 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:35.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:35.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:35.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:35.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:35.846 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:35.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:35.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:35.983 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:35.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:36.085 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:36.119 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:36.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:36.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:36.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:36.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:36.255 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:36.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:36.356 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:36.388 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:36.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:36.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:36.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:36 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:49:36.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:36 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:49:36.490 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:36.528 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:36.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:36.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:36.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:36.656 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:36.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:36.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:36.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:36 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:49:36.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:36 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:49:36.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:36 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:49:36.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:36 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:49:36.758 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:36.792 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:36.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:36.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:36.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:36.928 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:36.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:37.030 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:37.066 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:37.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:37.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:37.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:37.203 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:37.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:37.305 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:37.329 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:37.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:37.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:37.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:37 smithi137 ceph-mon[39175]: pgmap v998: 158 pgs: 5 peering, 3 active+remapped+backfilling, 150 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 878 op/s; 4266/206073 objects misplaced (2.070%); 27 MiB/s, 436 objects/s recovering 2024-09-18T07:49:37.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:37 smithi137 ceph-mon[39175]: osdmap e524: 8 total, 8 up, 7 in 2024-09-18T07:49:37.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:37 smithi137 ceph-mon[29408]: pgmap v998: 158 pgs: 5 peering, 3 active+remapped+backfilling, 150 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 878 op/s; 4266/206073 objects misplaced (2.070%); 27 MiB/s, 436 objects/s recovering 2024-09-18T07:49:37.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:37 smithi137 ceph-mon[29408]: osdmap e524: 8 total, 8 up, 7 in 2024-09-18T07:49:37.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:49:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:49:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:49:37.430 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:37.452 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:37.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:37.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:37.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:37 smithi203 ceph-mon[32237]: pgmap v998: 158 pgs: 5 peering, 3 active+remapped+backfilling, 150 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 878 op/s; 4266/206073 objects misplaced (2.070%); 27 MiB/s, 436 objects/s recovering 2024-09-18T07:49:37.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:37 smithi203 ceph-mon[32237]: osdmap e524: 8 total, 8 up, 7 in 2024-09-18T07:49:37.554 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:37.575 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:37.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:37.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:37.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T07:49:37.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:37.698 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:37.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:37.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:37.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:37.820 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:37.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:37.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:37.922 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:37.945 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:37.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:37.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:38.046 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:38.069 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:38.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:38.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:38.171 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:38.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:38.194 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:38.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:38.295 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:38.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:38.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:38.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:38.412 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:38 smithi137 ceph-mon[29408]: osdmap e525: 8 total, 8 up, 7 in 2024-09-18T07:49:38.412 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:38 smithi137 ceph-mon[39175]: osdmap e525: 8 total, 8 up, 7 in 2024-09-18T07:49:38.429 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:38.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:38 smithi203 ceph-mon[32237]: osdmap e525: 8 total, 8 up, 7 in 2024-09-18T07:49:38.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:49:37 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:49:38.464 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-18T07:49:38.464 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:49:38.477 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:38.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:38.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:38.583 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:38.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:38.613 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:38.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:38.714 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:38.749 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:38.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:38.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:38.851 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:38.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:38.891 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:38.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:38.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:39.025 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:39.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:39.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:39.126 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:39.128 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:39.183 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:39.190 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:39.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:39.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:39.330 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:39.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:39.431 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:39.465 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:39.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:39.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:39.469 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:39 smithi137 ceph-mon[29408]: pgmap v1001: 158 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 975 B/s rd, 0 B/s wr, 741 op/s; 4883/194958 objects misplaced (2.505%); 38 MiB/s, 615 objects/s recovering 2024-09-18T07:49:39.469 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:39 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:39.469 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:39 smithi137 ceph-mon[29408]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:49:39.469 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:39 smithi137 ceph-mon[29408]: osdmap e526: 8 total, 8 up, 7 in 2024-09-18T07:49:39.469 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:39 smithi137 ceph-mon[39175]: pgmap v1001: 158 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 975 B/s rd, 0 B/s wr, 741 op/s; 4883/194958 objects misplaced (2.505%); 38 MiB/s, 615 objects/s recovering 2024-09-18T07:49:39.470 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:39 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:39.470 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:39 smithi137 ceph-mon[39175]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:49:39.470 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:39 smithi137 ceph-mon[39175]: osdmap e526: 8 total, 8 up, 7 in 2024-09-18T07:49:39.567 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:39.604 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:39.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:39.705 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:39.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:39 smithi203 ceph-mon[32237]: pgmap v1001: 158 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 975 B/s rd, 0 B/s wr, 741 op/s; 4883/194958 objects misplaced (2.505%); 38 MiB/s, 615 objects/s recovering 2024-09-18T07:49:39.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:39 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:39 smithi203 ceph-mon[32237]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:49:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:39 smithi203 ceph-mon[32237]: osdmap e526: 8 total, 8 up, 7 in 2024-09-18T07:49:39.752 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:39.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:39.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:39.854 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:39.889 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:39.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:39.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:39.976 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:39.991 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:40.021 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:40.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:40.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:40.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:40.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:40.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:40.264 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:40.271 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:49:40.271 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:49:40.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:49:40.271 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:49:40.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.271 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:28995 11.b 11:ddfbb2a5:::benchmark_data_smithi137_181069_object28994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:49:40.271 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.138702+0000", 2024-09-18T07:49:40.272 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.12583755599999, 2024-09-18T07:49:40.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0007341729999999, 2024-09-18T07:49:40.272 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.272 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.272 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.272 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.272 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.272 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.272 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 28995 2024-09-18T07:49:40.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.272 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138702+0000", 2024-09-18T07:49:40.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138702+0000", 2024-09-18T07:49:40.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138706+0000", 2024-09-18T07:49:40.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5410000000000001e-06 2024-09-18T07:49:40.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138753+0000", 2024-09-18T07:49:40.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7506000000000002e-05 2024-09-18T07:49:40.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138792+0000", 2024-09-18T07:49:40.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8992000000000002e-05 2024-09-18T07:49:40.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138897+0000", 2024-09-18T07:49:40.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105167 2024-09-18T07:49:40.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138974+0000", 2024-09-18T07:49:40.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7225999999999995e-05 2024-09-18T07:49:40.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.275 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:49:40.275 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.138997+0000", 2024-09-18T07:49:40.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2711999999999999e-05 2024-09-18T07:49:40.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.130768+0000", 2024-09-18T07:49:40.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99177061899999996 2024-09-18T07:49:40.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.130889+0000", 2024-09-18T07:49:40.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000121519 2024-09-18T07:49:40.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:49:40.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.131038+0000", 2024-09-18T07:49:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014834700000000001 2024-09-18T07:49:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.135620+0000", 2024-09-18T07:49:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045826139999999996 2024-09-18T07:49:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139286+0000", 2024-09-18T07:49:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036660220000000001 2024-09-18T07:49:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139338+0000", 2024-09-18T07:49:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2077999999999998e-05 2024-09-18T07:49:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139392+0000", 2024-09-18T07:49:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4129000000000003e-05 2024-09-18T07:49:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139436+0000", 2024-09-18T07:49:40.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3701000000000001e-05 2024-09-18T07:49:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29002 11.0 11:09879661:::benchmark_data_smithi137_181069_object29001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:49:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.144657+0000", 2024-09-18T07:49:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.11988243799999, 2024-09-18T07:49:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99799427299999999, 2024-09-18T07:49:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29002 2024-09-18T07:49:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144657+0000", 2024-09-18T07:49:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144657+0000", 2024-09-18T07:49:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144659+0000", 2024-09-18T07:49:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4249999999999999e-06 2024-09-18T07:49:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144683+0000", 2024-09-18T07:49:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4658e-05 2024-09-18T07:49:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144699+0000", 2024-09-18T07:49:40.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5821999999999999e-05 2024-09-18T07:49:40.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144707+0000", 2024-09-18T07:49:40.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3790000000000006e-06 2024-09-18T07:49:40.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144801+0000", 2024-09-18T07:49:40.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3278999999999999e-05 2024-09-18T07:49:40.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:49:40.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.144811+0000", 2024-09-18T07:49:40.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8130000000000007e-06 2024-09-18T07:49:40.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.132138+0000", 2024-09-18T07:49:40.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98732766500000002 2024-09-18T07:49:40.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.132269+0000", 2024-09-18T07:49:40.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013058699999999999 2024-09-18T07:49:40.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:49:40.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.132403+0000", 2024-09-18T07:49:40.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013382300000000001 2024-09-18T07:49:40.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.136660+0000", 2024-09-18T07:49:40.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004257762 2024-09-18T07:49:40.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142430+0000", 2024-09-18T07:49:40.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057698050000000002 2024-09-18T07:49:40.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142552+0000", 2024-09-18T07:49:40.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012166400000000001 2024-09-18T07:49:40.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142611+0000", 2024-09-18T07:49:40.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8901999999999999e-05 2024-09-18T07:49:40.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142651+0000", 2024-09-18T07:49:40.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0689000000000003e-05 2024-09-18T07:49:40.286 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.286 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.286 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.286 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29006 11.0 11:0045c8b6:::benchmark_data_smithi137_181069_object29005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:49:40.287 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.148410+0000", 2024-09-18T07:49:40.287 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.11612938900001, 2024-09-18T07:49:40.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99787961400000003, 2024-09-18T07:49:40.287 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.287 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.287 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.287 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.287 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.287 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.287 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29006 2024-09-18T07:49:40.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.287 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148410+0000", 2024-09-18T07:49:40.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148410+0000", 2024-09-18T07:49:40.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148413+0000", 2024-09-18T07:49:40.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7580000000000001e-06 2024-09-18T07:49:40.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148442+0000", 2024-09-18T07:49:40.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8632999999999998e-05 2024-09-18T07:49:40.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148462+0000", 2024-09-18T07:49:40.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0166999999999999e-05 2024-09-18T07:49:40.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148471+0000", 2024-09-18T07:49:40.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3819999999999993e-06 2024-09-18T07:49:40.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148541+0000", 2024-09-18T07:49:40.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0277000000000005e-05 2024-09-18T07:49:40.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:49:40.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.148552+0000", 2024-09-18T07:49:40.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0584999999999999e-05 2024-09-18T07:49:40.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.134043+0000", 2024-09-18T07:49:40.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98549055699999999 2024-09-18T07:49:40.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.134770+0000", 2024-09-18T07:49:40.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00072761600000000005 2024-09-18T07:49:40.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:49:40.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.134899+0000", 2024-09-18T07:49:40.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012857100000000001 2024-09-18T07:49:40.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142732+0000", 2024-09-18T07:49:40.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0078337389999999993 2024-09-18T07:49:40.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142773+0000", 2024-09-18T07:49:40.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0132999999999997e-05 2024-09-18T07:49:40.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.146198+0000", 2024-09-18T07:49:40.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034254709999999998 2024-09-18T07:49:40.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.146249+0000", 2024-09-18T07:49:40.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0636000000000001e-05 2024-09-18T07:49:40.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.146290+0000", 2024-09-18T07:49:40.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1088999999999999e-05 2024-09-18T07:49:40.293 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.294 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.294 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.294 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29012 11.0 11:09fb93d5:::benchmark_data_smithi137_181069_object29011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:49:40.294 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.154931+0000", 2024-09-18T07:49:40.294 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.109609003, 2024-09-18T07:49:40.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.993148433, 2024-09-18T07:49:40.294 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.294 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.294 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.294 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.294 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.295 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.295 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29012 2024-09-18T07:49:40.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.295 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.154931+0000", 2024-09-18T07:49:40.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.154931+0000", 2024-09-18T07:49:40.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.154934+0000", 2024-09-18T07:49:40.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0570000000000001e-06 2024-09-18T07:49:40.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155009+0000", 2024-09-18T07:49:40.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5451000000000007e-05 2024-09-18T07:49:40.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155030+0000", 2024-09-18T07:49:40.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1262000000000001e-05 2024-09-18T07:49:40.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155041+0000", 2024-09-18T07:49:40.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0509999999999999e-05 2024-09-18T07:49:40.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155122+0000", 2024-09-18T07:49:40.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0785999999999995e-05 2024-09-18T07:49:40.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:49:40.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155133+0000", 2024-09-18T07:49:40.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1314e-05 2024-09-18T07:49:40.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.136710+0000", 2024-09-18T07:49:40.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98157681100000005 2024-09-18T07:49:40.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.136842+0000", 2024-09-18T07:49:40.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013199699999999999 2024-09-18T07:49:40.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:49:40.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.136966+0000", 2024-09-18T07:49:40.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000123757 2024-09-18T07:49:40.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142811+0000", 2024-09-18T07:49:40.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058454559999999997 2024-09-18T07:49:40.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.146298+0000", 2024-09-18T07:49:40.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034865690000000001 2024-09-18T07:49:40.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:40.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.147987+0000", 2024-09-18T07:49:40.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001689636 2024-09-18T07:49:40.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.148039+0000", 2024-09-18T07:49:40.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1335999999999998e-05 2024-09-18T07:49:40.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.148079+0000", 2024-09-18T07:49:40.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0491000000000002e-05 2024-09-18T07:49:40.303 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.303 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.303 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.303 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29013 11.0 11:0f785d5e:::benchmark_data_smithi137_181069_object29012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:49:40.303 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.155066+0000", 2024-09-18T07:49:40.303 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.10947396399999, 2024-09-18T07:49:40.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.99598627500000003, 2024-09-18T07:49:40.303 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.303 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.304 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.304 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.304 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.304 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.304 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29013 2024-09-18T07:49:40.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.304 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155066+0000", 2024-09-18T07:49:40.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.305 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.305 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155066+0000", 2024-09-18T07:49:40.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.305 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.305 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155066+0000", 2024-09-18T07:49:40.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8000000000000004e-07 2024-09-18T07:49:40.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.305 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.305 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155115+0000", 2024-09-18T07:49:40.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8026999999999999e-05 2024-09-18T07:49:40.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155123+0000", 2024-09-18T07:49:40.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.507e-06 2024-09-18T07:49:40.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155129+0000", 2024-09-18T07:49:40.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4119999999999998e-06 2024-09-18T07:49:40.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155190+0000", 2024-09-18T07:49:40.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0618000000000001e-05 2024-09-18T07:49:40.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:49:40.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.155200+0000", 2024-09-18T07:49:40.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0360000000000001e-05 2024-09-18T07:49:40.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.138108+0000", 2024-09-18T07:49:40.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98290805299999995 2024-09-18T07:49:40.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.138207+0000", 2024-09-18T07:49:40.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8399000000000004e-05 2024-09-18T07:49:40.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:49:40.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.138323+0000", 2024-09-18T07:49:40.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115858 2024-09-18T07:49:40.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142850+0000", 2024-09-18T07:49:40.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045276090000000001 2024-09-18T07:49:40.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.146302+0000", 2024-09-18T07:49:40.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003451759 2024-09-18T07:49:40.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.150948+0000", 2024-09-18T07:49:40.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046454440000000003 2024-09-18T07:49:40.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.151009+0000", 2024-09-18T07:49:40.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1697000000000002e-05 2024-09-18T07:49:40.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.151052+0000", 2024-09-18T07:49:40.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2651999999999998e-05 2024-09-18T07:49:40.310 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.310 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.310 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.311 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29017 11.0 11:0743585d:::benchmark_data_smithi137_181069_object29016:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:49:40.311 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.157700+0000", 2024-09-18T07:49:40.311 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.10683978399999, 2024-09-18T07:49:40.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0031416529999999, 2024-09-18T07:49:40.311 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.311 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.311 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.311 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.311 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.311 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.311 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29017 2024-09-18T07:49:40.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.311 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157700+0000", 2024-09-18T07:49:40.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157700+0000", 2024-09-18T07:49:40.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157703+0000", 2024-09-18T07:49:40.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.954e-06 2024-09-18T07:49:40.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157758+0000", 2024-09-18T07:49:40.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4923000000000003e-05 2024-09-18T07:49:40.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157775+0000", 2024-09-18T07:49:40.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7334999999999999e-05 2024-09-18T07:49:40.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157785+0000", 2024-09-18T07:49:40.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0131e-05 2024-09-18T07:49:40.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157860+0000", 2024-09-18T07:49:40.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4964999999999998e-05 2024-09-18T07:49:40.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:49:40.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.157870+0000", 2024-09-18T07:49:40.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0047999999999999e-05 2024-09-18T07:49:40.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139724+0000", 2024-09-18T07:49:40.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.98185376000000002 2024-09-18T07:49:40.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139843+0000", 2024-09-18T07:49:40.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011950700000000001 2024-09-18T07:49:40.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:49:40.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.141219+0000", 2024-09-18T07:49:40.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013756409999999999 2024-09-18T07:49:40.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.147813+0000", 2024-09-18T07:49:40.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065937160000000003 2024-09-18T07:49:40.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.147878+0000", 2024-09-18T07:49:40.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5513999999999996e-05 2024-09-18T07:49:40.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160742+0000", 2024-09-18T07:49:40.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012863638 2024-09-18T07:49:40.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160804+0000", 2024-09-18T07:49:40.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1717000000000006e-05 2024-09-18T07:49:40.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160841+0000", 2024-09-18T07:49:40.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7803999999999999e-05 2024-09-18T07:49:40.317 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.317 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.318 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.318 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29027 11.b 11:d70c9013:::benchmark_data_smithi137_181069_object29026:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:49:40.318 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.170795+0000", 2024-09-18T07:49:40.318 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.093744343, 2024-09-18T07:49:40.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96875128899999996, 2024-09-18T07:49:40.318 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.318 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.318 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.318 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.318 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.318 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.318 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29027 2024-09-18T07:49:40.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.319 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170795+0000", 2024-09-18T07:49:40.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170795+0000", 2024-09-18T07:49:40.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170798+0000", 2024-09-18T07:49:40.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4880000000000001e-06 2024-09-18T07:49:40.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170853+0000", 2024-09-18T07:49:40.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4836999999999997e-05 2024-09-18T07:49:40.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170867+0000", 2024-09-18T07:49:40.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4615e-05 2024-09-18T07:49:40.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170876+0000", 2024-09-18T07:49:40.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2299999999999997e-06 2024-09-18T07:49:40.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170945+0000", 2024-09-18T07:49:40.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8979999999999993e-05 2024-09-18T07:49:40.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:49:40.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170956+0000", 2024-09-18T07:49:40.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0716999999999999e-05 2024-09-18T07:49:40.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.132286+0000", 2024-09-18T07:49:40.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96132993099999997 2024-09-18T07:49:40.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.132397+0000", 2024-09-18T07:49:40.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011101 2024-09-18T07:49:40.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:49:40.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.132547+0000", 2024-09-18T07:49:40.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014972 2024-09-18T07:49:40.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.135658+0000", 2024-09-18T07:49:40.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031108949999999998 2024-09-18T07:49:40.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139296+0000", 2024-09-18T07:49:40.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003638208 2024-09-18T07:49:40.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139472+0000", 2024-09-18T07:49:40.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017624900000000001 2024-09-18T07:49:40.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139515+0000", 2024-09-18T07:49:40.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2787000000000002e-05 2024-09-18T07:49:40.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139547+0000", 2024-09-18T07:49:40.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1622000000000003e-05 2024-09-18T07:49:40.325 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.325 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.325 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.325 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29029 11.b 11:dd1c139d:::benchmark_data_smithi137_181069_object29028:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:49:40.325 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.170902+0000", 2024-09-18T07:49:40.325 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.093638003, 2024-09-18T07:49:40.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96875504400000001, 2024-09-18T07:49:40.326 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.326 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.326 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.326 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.326 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.326 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.326 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29029 2024-09-18T07:49:40.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.326 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170902+0000", 2024-09-18T07:49:40.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170902+0000", 2024-09-18T07:49:40.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170902+0000", 2024-09-18T07:49:40.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2999999999999999e-07 2024-09-18T07:49:40.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170948+0000", 2024-09-18T07:49:40.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5804999999999999e-05 2024-09-18T07:49:40.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170957+0000", 2024-09-18T07:49:40.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1290000000000002e-06 2024-09-18T07:49:40.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.170964+0000", 2024-09-18T07:49:40.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3999999999999997e-06 2024-09-18T07:49:40.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.172474+0000", 2024-09-18T07:49:40.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001510286 2024-09-18T07:49:40.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:49:40.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.172485+0000", 2024-09-18T07:49:40.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0621999999999999e-05 2024-09-18T07:49:40.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.134224+0000", 2024-09-18T07:49:40.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.96173945699999996 2024-09-18T07:49:40.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.134333+0000", 2024-09-18T07:49:40.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000108731 2024-09-18T07:49:40.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:49:40.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.134455+0000", 2024-09-18T07:49:40.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012238799999999999 2024-09-18T07:49:40.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.138920+0000", 2024-09-18T07:49:40.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044642049999999997 2024-09-18T07:49:40.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139301+0000", 2024-09-18T07:49:40.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00038108599999999999 2024-09-18T07:49:40.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139582+0000", 2024-09-18T07:49:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000281754 2024-09-18T07:49:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139625+0000", 2024-09-18T07:49:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2472000000000003e-05 2024-09-18T07:49:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139657+0000", 2024-09-18T07:49:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1779000000000002e-05 2024-09-18T07:49:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.332 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29048 11.0 11:02c48360:::benchmark_data_smithi137_181069_object29047:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:49:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.185570+0000", 2024-09-18T07:49:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.078969677, 2024-09-18T07:49:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97538213600000001, 2024-09-18T07:49:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.333 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29048 2024-09-18T07:49:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.185570+0000", 2024-09-18T07:49:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.185570+0000", 2024-09-18T07:49:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.185573+0000", 2024-09-18T07:49:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0699999999999998e-06 2024-09-18T07:49:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.185635+0000", 2024-09-18T07:49:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1809000000000004e-05 2024-09-18T07:49:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.185652+0000", 2024-09-18T07:49:40.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7543000000000001e-05 2024-09-18T07:49:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.185661+0000", 2024-09-18T07:49:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9479999999999997e-06 2024-09-18T07:49:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.185751+0000", 2024-09-18T07:49:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9740000000000005e-05 2024-09-18T07:49:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:49:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.185761+0000", 2024-09-18T07:49:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0017000000000001e-05 2024-09-18T07:49:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142881+0000", 2024-09-18T07:49:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95711977999999998 2024-09-18T07:49:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.143134+0000", 2024-09-18T07:49:40.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00025368400000000001 2024-09-18T07:49:40.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:49:40.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.143275+0000", 2024-09-18T07:49:40.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014023900000000001 2024-09-18T07:49:40.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.147818+0000", 2024-09-18T07:49:40.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045433619999999996 2024-09-18T07:49:40.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.147932+0000", 2024-09-18T07:49:40.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114017 2024-09-18T07:49:40.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160876+0000", 2024-09-18T07:49:40.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012943875000000001 2024-09-18T07:49:40.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160920+0000", 2024-09-18T07:49:40.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3668000000000003e-05 2024-09-18T07:49:40.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160952+0000", 2024-09-18T07:49:40.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2384e-05 2024-09-18T07:49:40.340 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.340 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.340 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.340 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29051 11.0 11:0623b464:::benchmark_data_smithi137_181069_object29050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:49:40.340 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.189389+0000", 2024-09-18T07:49:40.340 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.075150444, 2024-09-18T07:49:40.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97168827700000004, 2024-09-18T07:49:40.340 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.340 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.341 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.341 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.341 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.341 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.341 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29051 2024-09-18T07:49:40.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.341 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.189389+0000", 2024-09-18T07:49:40.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.189389+0000", 2024-09-18T07:49:40.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.189391+0000", 2024-09-18T07:49:40.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3499999999999999e-06 2024-09-18T07:49:40.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.189446+0000", 2024-09-18T07:49:40.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4775000000000003e-05 2024-09-18T07:49:40.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.189459+0000", 2024-09-18T07:49:40.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2826000000000001e-05 2024-09-18T07:49:40.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.189468+0000", 2024-09-18T07:49:40.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7930000000000003e-06 2024-09-18T07:49:40.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.189545+0000", 2024-09-18T07:49:40.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6785000000000004e-05 2024-09-18T07:49:40.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:49:40.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.189556+0000", 2024-09-18T07:49:40.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1343e-05 2024-09-18T07:49:40.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.144652+0000", 2024-09-18T07:49:40.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95509612300000002 2024-09-18T07:49:40.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.345 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:40.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:40.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.144785+0000", 2024-09-18T07:49:40.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013303999999999999 2024-09-18T07:49:40.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:49:40.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.144931+0000", 2024-09-18T07:49:40.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014624399999999999 2024-09-18T07:49:40.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.147822+0000", 2024-09-18T07:49:40.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028909439999999999 2024-09-18T07:49:40.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.148128+0000", 2024-09-18T07:49:40.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00030586799999999999 2024-09-18T07:49:40.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160999+0000", 2024-09-18T07:49:40.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012870967000000001 2024-09-18T07:49:40.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161044+0000", 2024-09-18T07:49:40.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5222000000000002e-05 2024-09-18T07:49:40.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161077+0000", 2024-09-18T07:49:40.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2997000000000003e-05 2024-09-18T07:49:40.348 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.348 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.348 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.349 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29075 11.b 11:dfbfee08:::benchmark_data_smithi137_181069_object29074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:49:40.349 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.207141+0000", 2024-09-18T07:49:40.349 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.05739842200001, 2024-09-18T07:49:40.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.936065696, 2024-09-18T07:49:40.349 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.349 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.349 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.349 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.349 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.349 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.349 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29075 2024-09-18T07:49:40.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.349 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.207141+0000", 2024-09-18T07:49:40.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.207141+0000", 2024-09-18T07:49:40.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.207143+0000", 2024-09-18T07:49:40.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8589999999999999e-06 2024-09-18T07:49:40.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.207193+0000", 2024-09-18T07:49:40.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9669999999999997e-05 2024-09-18T07:49:40.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.207209+0000", 2024-09-18T07:49:40.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5909e-05 2024-09-18T07:49:40.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.207217+0000", 2024-09-18T07:49:40.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0139999999999998e-06 2024-09-18T07:49:40.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.207277+0000", 2024-09-18T07:49:40.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0816000000000002e-05 2024-09-18T07:49:40.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:49:40.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.207289+0000", 2024-09-18T07:49:40.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1226999999999999e-05 2024-09-18T07:49:40.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.135725+0000", 2024-09-18T07:49:40.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.92843600999999998 2024-09-18T07:49:40.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.135854+0000", 2024-09-18T07:49:40.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000129305 2024-09-18T07:49:40.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:49:40.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.135996+0000", 2024-09-18T07:49:40.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000141722 2024-09-18T07:49:40.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.139305+0000", 2024-09-18T07:49:40.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003309317 2024-09-18T07:49:40.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.141294+0000", 2024-09-18T07:49:40.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00198854 2024-09-18T07:49:40.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.143116+0000", 2024-09-18T07:49:40.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001822167 2024-09-18T07:49:40.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.143166+0000", 2024-09-18T07:49:40.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9947999999999997e-05 2024-09-18T07:49:40.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.143207+0000", 2024-09-18T07:49:40.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1192000000000001e-05 2024-09-18T07:49:40.356 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.356 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.356 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.356 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29079 11.0 11:0c34a040:::benchmark_data_smithi137_181069_object29078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:49:40.356 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.209414+0000", 2024-09-18T07:49:40.357 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.05512593399999, 2024-09-18T07:49:40.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95177025599999998, 2024-09-18T07:49:40.357 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.357 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.357 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.357 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.357 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.357 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.357 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29079 2024-09-18T07:49:40.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.357 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.209414+0000", 2024-09-18T07:49:40.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.209414+0000", 2024-09-18T07:49:40.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.209415+0000", 2024-09-18T07:49:40.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0210000000000001e-06 2024-09-18T07:49:40.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.209438+0000", 2024-09-18T07:49:40.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3682000000000001e-05 2024-09-18T07:49:40.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.209447+0000", 2024-09-18T07:49:40.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4160000000000004e-06 2024-09-18T07:49:40.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.209454+0000", 2024-09-18T07:49:40.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9319999999999999e-06 2024-09-18T07:49:40.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.209529+0000", 2024-09-18T07:49:40.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5168999999999998e-05 2024-09-18T07:49:40.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:49:40.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.209538+0000", 2024-09-18T07:49:40.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5519999999999993e-06 2024-09-18T07:49:40.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.146333+0000", 2024-09-18T07:49:40.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93679421500000004 2024-09-18T07:49:40.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.146464+0000", 2024-09-18T07:49:40.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000131633 2024-09-18T07:49:40.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:49:40.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.146606+0000", 2024-09-18T07:49:40.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014206200000000001 2024-09-18T07:49:40.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.148144+0000", 2024-09-18T07:49:40.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001537561 2024-09-18T07:49:40.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.151135+0000", 2024-09-18T07:49:40.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029910929999999998 2024-09-18T07:49:40.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161110+0000", 2024-09-18T07:49:40.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0099754189999999993 2024-09-18T07:49:40.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161153+0000", 2024-09-18T07:49:40.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2589999999999997e-05 2024-09-18T07:49:40.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161184+0000", 2024-09-18T07:49:40.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0911000000000003e-05 2024-09-18T07:49:40.363 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.363 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.364 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.364 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29092 11.0 11:02dae925:::benchmark_data_smithi137_181069_object29091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:49:40.364 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.221277+0000", 2024-09-18T07:49:40.364 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.04326222700001, 2024-09-18T07:49:40.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.94000973799999998, 2024-09-18T07:49:40.364 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.364 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.364 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.364 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.364 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.364 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.365 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29092 2024-09-18T07:49:40.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.365 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.221277+0000", 2024-09-18T07:49:40.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.221277+0000", 2024-09-18T07:49:40.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.221280+0000", 2024-09-18T07:49:40.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1919999999999999e-06 2024-09-18T07:49:40.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.221323+0000", 2024-09-18T07:49:40.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3514999999999999e-05 2024-09-18T07:49:40.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.221350+0000", 2024-09-18T07:49:40.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7441999999999999e-05 2024-09-18T07:49:40.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.221359+0000", 2024-09-18T07:49:40.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3669999999999997e-06 2024-09-18T07:49:40.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.221429+0000", 2024-09-18T07:49:40.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9970999999999998e-05 2024-09-18T07:49:40.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:49:40.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.221440+0000", 2024-09-18T07:49:40.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0937e-05 2024-09-18T07:49:40.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.148187+0000", 2024-09-18T07:49:40.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.92674730800000005 2024-09-18T07:49:40.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.148317+0000", 2024-09-18T07:49:40.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012991200000000001 2024-09-18T07:49:40.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:49:40.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.148438+0000", 2024-09-18T07:49:40.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000121148 2024-09-18T07:49:40.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.151189+0000", 2024-09-18T07:49:40.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027507339999999999 2024-09-18T07:49:40.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160226+0000", 2024-09-18T07:49:40.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0090371380000000001 2024-09-18T07:49:40.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161215+0000", 2024-09-18T07:49:40.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00098912200000000005 2024-09-18T07:49:40.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161257+0000", 2024-09-18T07:49:40.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1613000000000003e-05 2024-09-18T07:49:40.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161287+0000", 2024-09-18T07:49:40.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0338999999999999e-05 2024-09-18T07:49:40.371 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.371 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.371 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.371 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29095 11.0 11:0f755343:::benchmark_data_smithi137_181069_object29094:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:49:40.371 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.224759+0000", 2024-09-18T07:49:40.371 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.03978032200001, 2024-09-18T07:49:40.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93664074100000005, 2024-09-18T07:49:40.371 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.371 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.372 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.372 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.372 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.372 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.372 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29095 2024-09-18T07:49:40.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.372 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.372 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.224759+0000", 2024-09-18T07:49:40.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.224759+0000", 2024-09-18T07:49:40.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.224762+0000", 2024-09-18T07:49:40.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4719999999999998e-06 2024-09-18T07:49:40.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.224788+0000", 2024-09-18T07:49:40.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6205999999999999e-05 2024-09-18T07:49:40.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.224806+0000", 2024-09-18T07:49:40.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8414e-05 2024-09-18T07:49:40.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.224816+0000", 2024-09-18T07:49:40.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0149000000000001e-05 2024-09-18T07:49:40.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.224889+0000", 2024-09-18T07:49:40.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2083999999999996e-05 2024-09-18T07:49:40.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:49:40.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.224902+0000", 2024-09-18T07:49:40.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3179999999999999e-05 2024-09-18T07:49:40.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.149507+0000", 2024-09-18T07:49:40.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.92460489300000004 2024-09-18T07:49:40.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.149624+0000", 2024-09-18T07:49:40.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116968 2024-09-18T07:49:40.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:49:40.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.149800+0000", 2024-09-18T07:49:40.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017676000000000001 2024-09-18T07:49:40.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160210+0000", 2024-09-18T07:49:40.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010410034 2024-09-18T07:49:40.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160231+0000", 2024-09-18T07:49:40.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0441999999999998e-05 2024-09-18T07:49:40.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161328+0000", 2024-09-18T07:49:40.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001097413 2024-09-18T07:49:40.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161369+0000", 2024-09-18T07:49:40.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1208999999999998e-05 2024-09-18T07:49:40.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161400+0000", 2024-09-18T07:49:40.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0516999999999999e-05 2024-09-18T07:49:40.378 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.378 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.378 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.379 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29107 11.0 11:0c46d5ad:::benchmark_data_smithi137_181069_object29106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:49:40.379 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.241089+0000", 2024-09-18T07:49:40.379 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 129.02345039799999, 2024-09-18T07:49:40.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.92074101200000003, 2024-09-18T07:49:40.379 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.379 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.379 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.379 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.379 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.379 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.379 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29107 2024-09-18T07:49:40.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.380 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.241089+0000", 2024-09-18T07:49:40.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.241089+0000", 2024-09-18T07:49:40.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.241091+0000", 2024-09-18T07:49:40.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1100000000000001e-06 2024-09-18T07:49:40.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.241113+0000", 2024-09-18T07:49:40.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1509e-05 2024-09-18T07:49:40.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.241125+0000", 2024-09-18T07:49:40.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1932e-05 2024-09-18T07:49:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.241132+0000", 2024-09-18T07:49:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2980000000000003e-06 2024-09-18T07:49:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.241194+0000", 2024-09-18T07:49:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1828999999999994e-05 2024-09-18T07:49:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:49:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.241202+0000", 2024-09-18T07:49:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1519999999999992e-06 2024-09-18T07:49:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.151364+0000", 2024-09-18T07:49:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91016249100000002 2024-09-18T07:49:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.151486+0000", 2024-09-18T07:49:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000121925 2024-09-18T07:49:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:49:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.151618+0000", 2024-09-18T07:49:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013172300000000001 2024-09-18T07:49:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.160235+0000", 2024-09-18T07:49:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0086173059999999999 2024-09-18T07:49:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161432+0000", 2024-09-18T07:49:40.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011964409999999999 2024-09-18T07:49:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161737+0000", 2024-09-18T07:49:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00030523600000000002 2024-09-18T07:49:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161783+0000", 2024-09-18T07:49:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6230000000000003e-05 2024-09-18T07:49:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.161830+0000", 2024-09-18T07:49:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6829999999999997e-05 2024-09-18T07:49:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:29116 11.b 11:de07af6b:::benchmark_data_smithi137_181069_object29115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-09-18T07:49:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:47:31.268587+0000", 2024-09-18T07:49:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 128.995952381, 2024-09-18T07:49:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.87712996300000001, 2024-09-18T07:49:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.386 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.387 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.387 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.387 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.387 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 29116 2024-09-18T07:49:40.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.387 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.268587+0000", 2024-09-18T07:49:40.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.268587+0000", 2024-09-18T07:49:40.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.268590+0000", 2024-09-18T07:49:40.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3190000000000002e-06 2024-09-18T07:49:40.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.268614+0000", 2024-09-18T07:49:40.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.406e-05 2024-09-18T07:49:40.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.268635+0000", 2024-09-18T07:49:40.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1658000000000002e-05 2024-09-18T07:49:40.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.268643+0000", 2024-09-18T07:49:40.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.875e-06 2024-09-18T07:49:40.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.268707+0000", 2024-09-18T07:49:40.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3862999999999995e-05 2024-09-18T07:49:40.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:49:40.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:31.268715+0000", 2024-09-18T07:49:40.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0989999999999999e-06 2024-09-18T07:49:40.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.137340+0000", 2024-09-18T07:49:40.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.86862488500000001 2024-09-18T07:49:40.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.137444+0000", 2024-09-18T07:49:40.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103609 2024-09-18T07:49:40.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:49:40.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.137563+0000", 2024-09-18T07:49:40.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011926000000000001 2024-09-18T07:49:40.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.141380+0000", 2024-09-18T07:49:40.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038170439999999999 2024-09-18T07:49:40.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.142079+0000", 2024-09-18T07:49:40.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00069944099999999997 2024-09-18T07:49:40.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.145604+0000", 2024-09-18T07:49:40.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035245039999999999 2024-09-18T07:49:40.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.145674+0000", 2024-09-18T07:49:40.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0593000000000007e-05 2024-09-18T07:49:40.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:47:32.145717+0000", 2024-09-18T07:49:40.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2753000000000002e-05 2024-09-18T07:49:40.394 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.394 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.394 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.394 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:67251 11.b 11:d877251f:::benchmark_data_smithi137_181069_object67250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e483)", 2024-09-18T07:49:40.394 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:00.906275+0000", 2024-09-18T07:49:40.394 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 99.358264722000001, 2024-09-18T07:49:40.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14005775000000001, 2024-09-18T07:49:40.394 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.395 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.395 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.395 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.395 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.395 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.395 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 67251 2024-09-18T07:49:40.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.395 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906275+0000", 2024-09-18T07:49:40.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906275+0000", 2024-09-18T07:49:40.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906277+0000", 2024-09-18T07:49:40.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8619999999999999e-06 2024-09-18T07:49:40.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906332+0000", 2024-09-18T07:49:40.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5778e-05 2024-09-18T07:49:40.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906344+0000", 2024-09-18T07:49:40.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1027e-05 2024-09-18T07:49:40.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906351+0000", 2024-09-18T07:49:40.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3950000000000003e-06 2024-09-18T07:49:40.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906403+0000", 2024-09-18T07:49:40.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1786000000000002e-05 2024-09-18T07:49:40.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906513+0000", 2024-09-18T07:49:40.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110438 2024-09-18T07:49:40.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:49:40.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.906604+0000", 2024-09-18T07:49:40.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0493000000000002e-05 2024-09-18T07:49:40.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.909046+0000", 2024-09-18T07:49:40.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024423209999999999 2024-09-18T07:49:40.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.909331+0000", 2024-09-18T07:49:40.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000284935 2024-09-18T07:49:40.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046260+0000", 2024-09-18T07:49:40.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13692871500000001 2024-09-18T07:49:40.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046300+0000", 2024-09-18T07:49:40.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0187e-05 2024-09-18T07:49:40.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046333+0000", 2024-09-18T07:49:40.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2812999999999999e-05 2024-09-18T07:49:40.401 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.401 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.401 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.401 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:67234 11.2 e483/459 11:4edd578a:::benchmark_data_smithi137_181069_object67233:head v 483'4135, mlcod=483'4133)", 2024-09-18T07:49:40.401 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:00.914389+0000", 2024-09-18T07:49:40.401 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 99.350151017000002, 2024-09-18T07:49:40.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14120948, 2024-09-18T07:49:40.401 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.402 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.402 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.402 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.914389+0000", 2024-09-18T07:49:40.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.914389+0000", 2024-09-18T07:49:40.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.914390+0000", 2024-09-18T07:49:40.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9269999999999999e-06 2024-09-18T07:49:40.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:00.914438+0000", 2024-09-18T07:49:40.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7681e-05 2024-09-18T07:49:40.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046208+0000", 2024-09-18T07:49:40.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.13176966700000001 2024-09-18T07:49:40.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046220+0000", 2024-09-18T07:49:40.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2041e-05 2024-09-18T07:49:40.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046283+0000", 2024-09-18T07:49:40.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3003999999999995e-05 2024-09-18T07:49:40.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.046305+0000", 2024-09-18T07:49:40.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2385e-05 2024-09-18T07:49:40.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.055546+0000", 2024-09-18T07:49:40.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0092402820000000007 2024-09-18T07:49:40.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:01.055598+0000", 2024-09-18T07:49:40.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2493e-05 2024-09-18T07:49:40.406 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.406 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.406 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.406 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:100666 11.b 11:d57d81c3:::benchmark_data_smithi137_181069_object100665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-09-18T07:49:40.406 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:31.004690+0000", 2024-09-18T07:49:40.406 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 69.259849181999996, 2024-09-18T07:49:40.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15533425000000001, 2024-09-18T07:49:40.406 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.406 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.406 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.406 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.407 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.407 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.407 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 100666 2024-09-18T07:49:40.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.407 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.004690+0000", 2024-09-18T07:49:40.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.004690+0000", 2024-09-18T07:49:40.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.004691+0000", 2024-09-18T07:49:40.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5999999999999991e-07 2024-09-18T07:49:40.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.004713+0000", 2024-09-18T07:49:40.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1466000000000001e-05 2024-09-18T07:49:40.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.004720+0000", 2024-09-18T07:49:40.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4030000000000001e-06 2024-09-18T07:49:40.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.004726+0000", 2024-09-18T07:49:40.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5740000000000003e-06 2024-09-18T07:49:40.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.005170+0000", 2024-09-18T07:49:40.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044430400000000002 2024-09-18T07:49:40.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.005315+0000", 2024-09-18T07:49:40.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014535700000000001 2024-09-18T07:49:40.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:49:40.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.005485+0000", 2024-09-18T07:49:40.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016952300000000001 2024-09-18T07:49:40.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.007859+0000", 2024-09-18T07:49:40.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023742189999999999 2024-09-18T07:49:40.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.008681+0000", 2024-09-18T07:49:40.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00082156700000000004 2024-09-18T07:49:40.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.159934+0000", 2024-09-18T07:49:40.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.15125297200000001 2024-09-18T07:49:40.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.159991+0000", 2024-09-18T07:49:40.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7399999999999999e-05 2024-09-18T07:49:40.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.160025+0000", 2024-09-18T07:49:40.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3504999999999999e-05 2024-09-18T07:49:40.412 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.412 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.412 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.413 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:100681 11.b 11:da997bc0:::benchmark_data_smithi137_181069_object100680:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-09-18T07:49:40.413 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:31.018446+0000", 2024-09-18T07:49:40.413 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 69.246093848000001, 2024-09-18T07:49:40.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.141729829, 2024-09-18T07:49:40.413 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:49:40.413 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:49:40.413 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:49:40.413 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:49:40.413 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:49:40.413 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:40.413 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 100681 2024-09-18T07:49:40.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.413 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:49:40.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:49:40.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.018446+0000", 2024-09-18T07:49:40.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:49:40.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.018446+0000", 2024-09-18T07:49:40.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:49:40.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:49:40.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.018448+0000", 2024-09-18T07:49:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9479999999999998e-06 2024-09-18T07:49:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:49:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.018470+0000", 2024-09-18T07:49:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2008999999999999e-05 2024-09-18T07:49:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:49:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.018482+0000", 2024-09-18T07:49:40.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2308e-05 2024-09-18T07:49:40.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:49:40.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.018489+0000", 2024-09-18T07:49:40.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1130000000000003e-06 2024-09-18T07:49:40.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:49:40.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.018552+0000", 2024-09-18T07:49:40.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3114999999999995e-05 2024-09-18T07:49:40.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:49:40.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.018684+0000", 2024-09-18T07:49:40.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000131342 2024-09-18T07:49:40.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:49:40.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.018824+0000", 2024-09-18T07:49:40.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000140865 2024-09-18T07:49:40.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:49:40.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.021881+0000", 2024-09-18T07:49:40.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030567200000000002 2024-09-18T07:49:40.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.035100+0000", 2024-09-18T07:49:40.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013218473999999999 2024-09-18T07:49:40.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:40.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.160081+0000", 2024-09-18T07:49:40.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.12498174300000001 2024-09-18T07:49:40.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:49:40.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.160135+0000", 2024-09-18T07:49:40.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3798000000000002e-05 2024-09-18T07:49:40.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:49:40.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:49:40.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:49:40.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:31.160176+0000", 2024-09-18T07:49:40.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0394e-05 2024-09-18T07:49:40.419 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.419 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.419 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.419 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:49:40.419 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:49:40.419 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:49:40.446 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:40.471 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:40.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:40.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:40.573 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:40.594 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:40.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:40.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:40.696 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:40.717 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:40.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:40.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:40.819 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:40.840 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:40.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:40.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:40.942 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:40.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:40.965 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:40.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:41.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:41.087 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:41.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:41.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:41.098 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:49:41.173 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:49:41.189 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:41.224 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:41.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:41.325 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:41.357 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:41.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:41.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:41.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:41 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:49:41.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:41 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:49:41.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:41 smithi137 ceph-mon[29408]: pgmap v1003: 158 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 702 op/s; 4883/193992 objects misplaced (2.517%); 33 MiB/s, 537 objects/s recovering 2024-09-18T07:49:41.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:41 smithi137 ceph-mon[29408]: osdmap e527: 8 total, 8 up, 7 in 2024-09-18T07:49:41.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:41 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4152438899' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:49:41.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:41 smithi137 ceph-mon[29408]: from='client.26755 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:49:41.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:41 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:49:41.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:41 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:49:41.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:41 smithi137 ceph-mon[39175]: pgmap v1003: 158 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 702 op/s; 4883/193992 objects misplaced (2.517%); 33 MiB/s, 537 objects/s recovering 2024-09-18T07:49:41.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:41 smithi137 ceph-mon[39175]: osdmap e527: 8 total, 8 up, 7 in 2024-09-18T07:49:41.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:41 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4152438899' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:49:41.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:41 smithi137 ceph-mon[39175]: from='client.26755 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:49:41.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:41.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:41 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:49:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:41 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:49:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:41 smithi203 ceph-mon[32237]: pgmap v1003: 158 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 702 op/s; 4883/193992 objects misplaced (2.517%); 33 MiB/s, 537 objects/s recovering 2024-09-18T07:49:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:41 smithi203 ceph-mon[32237]: osdmap e527: 8 total, 8 up, 7 in 2024-09-18T07:49:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:41 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4152438899' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:49:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:41 smithi203 ceph-mon[32237]: from='client.26755 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:49:41.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:41.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:41.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:41.610 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:41.645 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:41.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:41.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:41.710 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T07:49:41.746 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:41.799 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:41.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:41.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:41.901 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:41.936 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:41.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:41.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:42.038 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:42.072 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:42.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:42.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:42.134 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:49:42.175 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:42.214 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:42.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:42.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:42.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:42.336 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:42 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:49:42.336 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:42 smithi137 ceph-mon[29408]: from='client.26755 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:49:42.336 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:42 smithi137 ceph-mon[29408]: osdmap e528: 8 total, 8 up, 7 in 2024-09-18T07:49:42.336 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:42 smithi137 ceph-mon[29408]: 5.1f scrub starts 2024-09-18T07:49:42.336 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:42 smithi137 ceph-mon[29408]: 5.1f scrub ok 2024-09-18T07:49:42.337 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:42 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:49:42.337 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:42 smithi137 ceph-mon[39175]: from='client.26755 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:49:42.337 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:42 smithi137 ceph-mon[39175]: osdmap e528: 8 total, 8 up, 7 in 2024-09-18T07:49:42.337 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:42 smithi137 ceph-mon[39175]: 5.1f scrub starts 2024-09-18T07:49:42.337 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:42 smithi137 ceph-mon[39175]: 5.1f scrub ok 2024-09-18T07:49:42.351 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:42.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:42.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:42.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:42.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:42 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:49:42.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:42 smithi203 ceph-mon[32237]: from='client.26755 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:49:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:42 smithi203 ceph-mon[32237]: osdmap e528: 8 total, 8 up, 7 in 2024-09-18T07:49:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:42 smithi203 ceph-mon[32237]: 5.1f scrub starts 2024-09-18T07:49:42.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:42 smithi203 ceph-mon[32237]: 5.1f scrub ok 2024-09-18T07:49:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:42.488 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:42.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:42.555 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:49:42.564 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:49:42.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:42.622 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:42.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:42.725 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:42.764 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:42.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:42.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:42.866 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:42.900 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:42.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:42.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:43.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:43.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:43.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:43.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:43.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:43.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:43.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:43.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:43.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:43.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:43.307 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:43.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:43.313 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:43 smithi203 ceph-mon[32237]: 3.1c scrub starts 2024-09-18T07:49:43.313 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:43 smithi203 ceph-mon[32237]: 3.1c scrub ok 2024-09-18T07:49:43.313 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:43 smithi203 ceph-mon[32237]: pgmap v1006: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 156 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 949 op/s; 3912/185631 objects misplaced (2.107%); 29 MiB/s, 472 objects/s recovering 2024-09-18T07:49:43.313 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:43 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:49:43.407 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:43 smithi137 ceph-mon[29408]: 3.1c scrub starts 2024-09-18T07:49:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:43 smithi137 ceph-mon[29408]: 3.1c scrub ok 2024-09-18T07:49:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:43 smithi137 ceph-mon[29408]: pgmap v1006: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 156 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 949 op/s; 3912/185631 objects misplaced (2.107%); 29 MiB/s, 472 objects/s recovering 2024-09-18T07:49:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:43 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:49:43.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:43 smithi137 ceph-mon[39175]: 3.1c scrub starts 2024-09-18T07:49:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:43 smithi137 ceph-mon[39175]: 3.1c scrub ok 2024-09-18T07:49:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:43 smithi137 ceph-mon[39175]: pgmap v1006: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 156 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 949 op/s; 3912/185631 objects misplaced (2.107%); 29 MiB/s, 472 objects/s recovering 2024-09-18T07:49:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:43 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:49:43.445 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:43.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:43.489 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:49:43.490 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:49:43.490 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.490 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:166279 11.9 11:9460ff28:::benchmark_data_smithi137_181069_object54125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-09-18T07:49:43.490 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:49:43.477296+0000", 2024-09-18T07:49:43.490 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0071975320000000004, 2024-09-18T07:49:43.490 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0072178399999999997, 2024-09-18T07:49:43.490 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:49:43.490 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:49:43.490 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:49:43.490 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:49:43.490 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:49:43.491 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:43.491 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 166279 2024-09-18T07:49:43.491 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.491 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:49:43.491 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.491 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:49:43.491 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.477296+0000", 2024-09-18T07:49:43.491 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.491 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.491 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.491 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:49:43.491 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.477296+0000", 2024-09-18T07:49:43.492 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.492 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.492 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.492 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:49:43.492 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.477298+0000", 2024-09-18T07:49:43.492 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3960000000000001e-06 2024-09-18T07:49:43.492 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.492 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.492 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:49:43.492 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.477302+0000", 2024-09-18T07:49:43.492 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.724e-06 2024-09-18T07:49:43.492 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.492 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.493 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:49:43.493 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.477311+0000", 2024-09-18T07:49:43.493 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5620000000000004e-06 2024-09-18T07:49:43.493 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.493 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.493 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:49:43.493 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.477318+0000", 2024-09-18T07:49:43.493 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7730000000000002e-06 2024-09-18T07:49:43.493 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.493 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.493 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:49:43.493 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.477379+0000", 2024-09-18T07:49:43.493 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1119999999999998e-05 2024-09-18T07:49:43.493 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.494 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.494 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:49:43.494 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.477476+0000", 2024-09-18T07:49:43.494 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6898000000000006e-05 2024-09-18T07:49:43.494 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.494 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.494 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:49:43.494 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.477573+0000", 2024-09-18T07:49:43.494 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7368999999999996e-05 2024-09-18T07:49:43.494 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.494 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.494 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:49:43.494 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483957+0000", 2024-09-18T07:49:43.495 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0063837599999999996 2024-09-18T07:49:43.495 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.495 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:49:43.495 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.495 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.495 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.495 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:166282 11.3 11:c1cf6baf:::benchmark_data_smithi137_181069_object54128:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-09-18T07:49:43.495 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:49:43.478358+0000", 2024-09-18T07:49:43.495 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0061362889999999996, 2024-09-18T07:49:43.495 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0062011949999999996, 2024-09-18T07:49:43.495 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:49:43.495 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:49:43.495 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:49:43.496 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:49:43.496 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:49:43.496 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:43.496 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 166282 2024-09-18T07:49:43.496 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.496 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:49:43.496 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.496 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:49:43.496 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.478358+0000", 2024-09-18T07:49:43.496 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.496 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.496 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.496 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:49:43.497 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.478358+0000", 2024-09-18T07:49:43.497 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.497 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.497 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.497 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:49:43.497 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.478359+0000", 2024-09-18T07:49:43.497 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.232e-06 2024-09-18T07:49:43.497 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.497 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.497 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:49:43.497 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.478363+0000", 2024-09-18T07:49:43.497 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5589999999999999e-06 2024-09-18T07:49:43.497 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.497 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.498 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:49:43.498 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.478373+0000", 2024-09-18T07:49:43.498 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3670000000000003e-06 2024-09-18T07:49:43.498 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.498 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.498 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:49:43.498 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.478379+0000", 2024-09-18T07:49:43.498 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7599999999999997e-06 2024-09-18T07:49:43.498 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.498 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.498 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:49:43.498 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.478435+0000", 2024-09-18T07:49:43.498 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5754999999999997e-05 2024-09-18T07:49:43.499 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.499 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.499 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:49:43.499 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.478537+0000", 2024-09-18T07:49:43.499 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010184200000000001 2024-09-18T07:49:43.499 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.499 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.499 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:49:43.499 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.478636+0000", 2024-09-18T07:49:43.499 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8425999999999996e-05 2024-09-18T07:49:43.499 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.499 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.499 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:49:43.499 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.480998+0000", 2024-09-18T07:49:43.500 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023628070000000002 2024-09-18T07:49:43.500 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.500 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:49:43.500 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.500 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.500 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.500 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:166285 11.a 11:5c5c92c6:::benchmark_data_smithi137_181069_object54131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-09-18T07:49:43.500 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:49:43.480893+0000", 2024-09-18T07:49:43.500 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0036010740000000001, 2024-09-18T07:49:43.500 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003702833, 2024-09-18T07:49:43.500 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:49:43.501 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:49:43.501 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:49:43.501 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:49:43.501 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:49:43.501 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:43.501 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 166285 2024-09-18T07:49:43.501 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.501 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:49:43.501 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.501 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:49:43.501 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.480893+0000", 2024-09-18T07:49:43.501 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.502 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.502 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.502 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:49:43.502 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.480893+0000", 2024-09-18T07:49:43.502 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.502 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.502 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.502 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:49:43.502 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.480894+0000", 2024-09-18T07:49:43.502 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.269e-06 2024-09-18T07:49:43.502 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.502 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.502 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:49:43.502 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.480899+0000", 2024-09-18T07:49:43.503 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5889999999999996e-06 2024-09-18T07:49:43.503 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.503 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.503 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:49:43.503 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.480966+0000", 2024-09-18T07:49:43.503 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6905999999999999e-05 2024-09-18T07:49:43.503 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.503 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.503 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:49:43.503 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.480973+0000", 2024-09-18T07:49:43.503 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0040000000000002e-06 2024-09-18T07:49:43.503 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.503 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.503 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:49:43.503 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.481034+0000", 2024-09-18T07:49:43.504 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1545999999999997e-05 2024-09-18T07:49:43.504 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.504 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.504 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:49:43.504 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.481142+0000", 2024-09-18T07:49:43.504 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000107427 2024-09-18T07:49:43.504 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.504 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.504 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,2,5", 2024-09-18T07:49:43.504 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.481261+0000", 2024-09-18T07:49:43.504 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000119399 2024-09-18T07:49:43.504 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.504 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:49:43.504 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.505 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.505 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.505 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:166286 11.1 11:85993387:::benchmark_data_smithi137_181069_object54132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-09-18T07:49:43.505 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:49:43.482912+0000", 2024-09-18T07:49:43.505 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.001581456, 2024-09-18T07:49:43.505 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0017269119999999999, 2024-09-18T07:49:43.505 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:49:43.505 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:49:43.505 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:49:43.505 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:49:43.505 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:49:43.505 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:43.505 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 166286 2024-09-18T07:49:43.506 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.506 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:49:43.506 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.506 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:49:43.506 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.482912+0000", 2024-09-18T07:49:43.506 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.506 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.506 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.506 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:49:43.506 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.482912+0000", 2024-09-18T07:49:43.506 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.506 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.506 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.507 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:49:43.507 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.482913+0000", 2024-09-18T07:49:43.507 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0109999999999999e-06 2024-09-18T07:49:43.507 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.507 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.507 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:49:43.507 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.482918+0000", 2024-09-18T07:49:43.507 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7849999999999999e-06 2024-09-18T07:49:43.507 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.507 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.507 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:49:43.507 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.482931+0000", 2024-09-18T07:49:43.507 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2853999999999999e-05 2024-09-18T07:49:43.508 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.508 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.508 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:49:43.508 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.482938+0000", 2024-09-18T07:49:43.508 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7449999999999998e-06 2024-09-18T07:49:43.508 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.508 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.508 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:49:43.508 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483830+0000", 2024-09-18T07:49:43.508 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00089245700000000004 2024-09-18T07:49:43.508 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.508 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.508 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:49:43.509 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483925+0000", 2024-09-18T07:49:43.509 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5100999999999996e-05 2024-09-18T07:49:43.509 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.509 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.509 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,2", 2024-09-18T07:49:43.509 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.484028+0000", 2024-09-18T07:49:43.509 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000102237 2024-09-18T07:49:43.509 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.509 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:49:43.509 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.509 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.509 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:166287 11.a 11:57f84613:::benchmark_data_smithi137_181069_object54133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-09-18T07:49:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:49:43.482958+0000", 2024-09-18T07:49:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0015358049999999999, 2024-09-18T07:49:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001745325, 2024-09-18T07:49:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:49:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:49:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:49:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:49:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:49:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 166287 2024-09-18T07:49:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:49:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.511 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:49:43.511 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.482958+0000", 2024-09-18T07:49:43.511 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.511 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.511 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.511 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:49:43.511 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.482958+0000", 2024-09-18T07:49:43.511 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.511 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.511 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.511 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:49:43.511 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.482959+0000", 2024-09-18T07:49:43.511 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4799999999999997e-07 2024-09-18T07:49:43.511 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.512 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.512 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:49:43.512 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.482962+0000", 2024-09-18T07:49:43.512 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0740000000000001e-06 2024-09-18T07:49:43.512 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.512 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.512 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:49:43.512 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.482968+0000", 2024-09-18T07:49:43.512 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.711e-06 2024-09-18T07:49:43.512 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.512 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.512 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:49:43.512 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.482973+0000", 2024-09-18T07:49:43.512 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0509999999999996e-06 2024-09-18T07:49:43.513 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.513 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.513 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:49:43.513 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483455+0000", 2024-09-18T07:49:43.513 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000482106 2024-09-18T07:49:43.513 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.513 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.513 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:49:43.513 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483573+0000", 2024-09-18T07:49:43.513 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000118136 2024-09-18T07:49:43.513 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.513 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.513 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,2,5", 2024-09-18T07:49:43.513 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483706+0000", 2024-09-18T07:49:43.514 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000133106 2024-09-18T07:49:43.514 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.514 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:49:43.514 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.514 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.514 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.514 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:166288 11.a 11.1f73235a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-09-18T07:49:43.514 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:49:43.483364+0000", 2024-09-18T07:49:43.514 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0011296940000000001, 2024-09-18T07:49:43.514 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0013986580000000001, 2024-09-18T07:49:43.514 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:49:43.514 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:49:43.514 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "queued for pg", 2024-09-18T07:49:43.514 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:49:43.514 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:49:43.515 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:43.515 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 166288 2024-09-18T07:49:43.515 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.515 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:49:43.515 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.515 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:49:43.515 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483364+0000", 2024-09-18T07:49:43.515 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.515 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.515 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.515 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:49:43.515 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483364+0000", 2024-09-18T07:49:43.515 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.515 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.516 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.516 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:49:43.516 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483365+0000", 2024-09-18T07:49:43.516 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.018e-06 2024-09-18T07:49:43.516 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.516 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.516 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:49:43.516 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483369+0000", 2024-09-18T07:49:43.516 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9509999999999999e-06 2024-09-18T07:49:43.516 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.516 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.516 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:49:43.516 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483379+0000", 2024-09-18T07:49:43.516 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8519999999999999e-06 2024-09-18T07:49:43.517 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.517 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.517 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:49:43.517 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483388+0000", 2024-09-18T07:49:43.517 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0259999999999996e-06 2024-09-18T07:49:43.517 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.517 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:49:43.517 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.517 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.517 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.517 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:166290 11.3 11:c310245c:::benchmark_data_smithi137_181069_object54136:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-09-18T07:49:43.518 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:49:43.483439+0000", 2024-09-18T07:49:43.518 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0010550970000000001, 2024-09-18T07:49:43.518 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0013669190000000001, 2024-09-18T07:49:43.518 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:49:43.518 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:49:43.518 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "waiting for sub ops", 2024-09-18T07:49:43.518 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:49:43.518 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:49:43.518 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:43.518 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 166290 2024-09-18T07:49:43.518 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.518 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:49:43.519 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.519 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:49:43.519 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483439+0000", 2024-09-18T07:49:43.520 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.520 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.520 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.520 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:49:43.520 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483439+0000", 2024-09-18T07:49:43.520 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.520 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.520 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.520 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:49:43.520 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483440+0000", 2024-09-18T07:49:43.520 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.6899999999999996e-07 2024-09-18T07:49:43.520 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.520 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.521 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:49:43.521 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483443+0000", 2024-09-18T07:49:43.521 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7309999999999999e-06 2024-09-18T07:49:43.521 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.521 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.521 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:49:43.521 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483450+0000", 2024-09-18T07:49:43.521 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0010000000000004e-06 2024-09-18T07:49:43.521 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.521 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.521 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:49:43.521 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483456+0000", 2024-09-18T07:49:43.522 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.711e-06 2024-09-18T07:49:43.522 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.522 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.522 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:49:43.522 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483503+0000", 2024-09-18T07:49:43.522 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.7131e-05 2024-09-18T07:49:43.522 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.522 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.522 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:49:43.522 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483615+0000", 2024-09-18T07:49:43.522 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000112056 2024-09-18T07:49:43.522 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.522 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.522 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,7", 2024-09-18T07:49:43.523 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483733+0000", 2024-09-18T07:49:43.523 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011801 2024-09-18T07:49:43.523 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.523 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:49:43.523 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.523 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.523 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.523 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:166291 11.1 11.ca0564e1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-09-18T07:49:43.523 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:49:43.483475+0000", 2024-09-18T07:49:43.523 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.001019018, 2024-09-18T07:49:43.523 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0013637409999999999, 2024-09-18T07:49:43.523 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:49:43.523 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:49:43.524 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "queued for pg", 2024-09-18T07:49:43.524 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:49:43.524 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:49:43.524 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:43.524 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 166291 2024-09-18T07:49:43.524 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.524 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:49:43.524 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.524 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:49:43.524 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483475+0000", 2024-09-18T07:49:43.524 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.524 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.524 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.525 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:49:43.525 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483475+0000", 2024-09-18T07:49:43.525 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.525 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.525 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.525 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:49:43.525 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483476+0000", 2024-09-18T07:49:43.525 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3300000000000001e-07 2024-09-18T07:49:43.525 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.525 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.525 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:49:43.525 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483479+0000", 2024-09-18T07:49:43.525 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.9799999999999998e-06 2024-09-18T07:49:43.526 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.526 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.526 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:49:43.526 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483484+0000", 2024-09-18T07:49:43.526 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9799999999999998e-06 2024-09-18T07:49:43.526 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.526 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.526 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:49:43.526 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483490+0000", 2024-09-18T07:49:43.526 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9780000000000002e-06 2024-09-18T07:49:43.526 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.526 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:49:43.526 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.526 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.527 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.527 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:166292 11.9 11.8981a7e9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e528)", 2024-09-18T07:49:43.527 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:49:43.483502+0000", 2024-09-18T07:49:43.527 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.00099233599999999991, 2024-09-18T07:49:43.527 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0013602659999999999, 2024-09-18T07:49:43.527 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:49:43.527 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:49:43.527 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "queued for pg", 2024-09-18T07:49:43.527 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:49:43.527 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:49:43.527 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:49:43.527 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 166292 2024-09-18T07:49:43.527 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.528 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:49:43.528 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.528 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:49:43.528 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483502+0000", 2024-09-18T07:49:43.528 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.528 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.528 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.528 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:49:43.528 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483502+0000", 2024-09-18T07:49:43.528 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.528 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.529 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.529 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:49:43.529 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483502+0000", 2024-09-18T07:49:43.529 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4300000000000003e-07 2024-09-18T07:49:43.529 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.529 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.529 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:49:43.529 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483505+0000", 2024-09-18T07:49:43.529 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5299999999999999e-06 2024-09-18T07:49:43.529 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.529 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.529 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:49:43.529 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483510+0000", 2024-09-18T07:49:43.530 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3770000000000004e-06 2024-09-18T07:49:43.530 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.530 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.530 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:49:43.530 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483515+0000", 2024-09-18T07:49:43.530 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.198e-06 2024-09-18T07:49:43.530 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.530 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:49:43.530 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.530 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.530 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.530 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop_reply(client.16581.0:166282 11.3 e528/475)", 2024-09-18T07:49:43.530 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:49:43.483733+0000", 2024-09-18T07:49:43.530 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.00076117899999999996, 2024-09-18T07:49:43.531 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0011510820000000001, 2024-09-18T07:49:43.531 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:49:43.531 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:49:43.531 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "queued for pg", 2024-09-18T07:49:43.531 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:49:43.531 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.531 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:49:43.531 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483733+0000", 2024-09-18T07:49:43.531 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.531 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.531 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.531 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:49:43.531 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483733+0000", 2024-09-18T07:49:43.531 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:43.532 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.532 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.532 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:49:43.532 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483734+0000", 2024-09-18T07:49:43.532 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1900000000000001e-07 2024-09-18T07:49:43.532 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.532 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.532 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:49:43.532 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483738+0000", 2024-09-18T07:49:43.532 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0790000000000002e-06 2024-09-18T07:49:43.532 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.532 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.532 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:49:43.533 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483744+0000", 2024-09-18T07:49:43.533 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6989999999999997e-06 2024-09-18T07:49:43.533 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:43.533 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:43.533 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:49:43.533 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:43.483750+0000", 2024-09-18T07:49:43.533 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9399999999999999e-06 2024-09-18T07:49:43.533 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.533 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:49:43.533 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.533 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:43.533 INFO:teuthology.orchestra.run.smithi203.stdout: ], 2024-09-18T07:49:43.533 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 10 2024-09-18T07:49:43.534 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:49:43.534 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":1006,"stamp":"2024-09-18T07:49:41.884373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4042660403,"num_objects":61877,"num_object_clones":0,"num_object_copies":185631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3912,"num_objects_unfound":0,"num_objects_dirty":61877,"num_whiteouts":0,"num_read":1986,"num_read_kb":1738,"num_write":163348,"num_write_kb":7177710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45083,"num_bytes_recovered":2915517604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74171,"ondisk_log_size":74171,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":481,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16778504,"kb_used_data":12282296,"kb_used_omap":18,"kb_used_meta":4496045,"kb_avail":639437560,"statfs":{"total":671965249536,"available":654784061440,"internally_reserved":0,"allocated":12577071104,"data_stored":12574289481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19293,"internal_metadata":4603950243},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2,5,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-291176448,"num_objects":-4443,"num_object_clones":0,"num_object_copies":-13329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1662,"num_objects_unfound":0,"num_objects_dirty":-4443,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4443,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2216,"num_bytes_recovered":144900206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.686193"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":1006,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.214480+0000","last_change":"2024-09-18T07:48:28.181443+0000","last_active":"2024-09-18T07:49:38.214480+0000","last_peered":"2024-09-18T07:49:38.214480+0000","last_clean":"2024-09-18T07:49:38.214480+0000","last_became_active":"2024-09-18T07:47:11.200534+0000","last_became_peered":"2024-09-18T07:47:11.200534+0000","last_unstale":"2024-09-18T07:49:38.214480+0000","last_undegraded":"2024-09-18T07:49:38.214480+0000","last_fullsized":"2024-09-18T07:49:38.214480+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:28.181383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:28.181383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:46.295639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"323'15","reported_seq":966,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169095+0000","last_change":"2024-09-18T07:47:20.194767+0000","last_active":"2024-09-18T07:49:41.169095+0000","last_peered":"2024-09-18T07:49:41.169095+0000","last_clean":"2024-09-18T07:49:41.169095+0000","last_became_active":"2024-09-18T07:47:20.136254+0000","last_became_peered":"2024-09-18T07:47:20.136254+0000","last_unstale":"2024-09-18T07:49:41.169095+0000","last_undegraded":"2024-09-18T07:49:41.169095+0000","last_fullsized":"2024-09-18T07:49:41.169095+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:47:09.864550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.864550+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":795,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.242917+0000","last_change":"2024-09-18T07:48:05.093858+0000","last_active":"2024-09-18T07:49:38.242917+0000","last_peered":"2024-09-18T07:49:38.242917+0000","last_clean":"2024-09-18T07:49:38.242917+0000","last_became_active":"2024-09-18T07:47:20.130091+0000","last_became_peered":"2024-09-18T07:47:20.130091+0000","last_unstale":"2024-09-18T07:49:38.242917+0000","last_undegraded":"2024-09-18T07:49:38.242917+0000","last_fullsized":"2024-09-18T07:49:38.242917+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:05.093789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:05.093789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":859,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169516+0000","last_change":"2024-09-18T07:49:32.689133+0000","last_active":"2024-09-18T07:49:41.169516+0000","last_peered":"2024-09-18T07:49:41.169516+0000","last_clean":"2024-09-18T07:49:41.169516+0000","last_became_active":"2024-09-18T07:49:32.688553+0000","last_became_peered":"2024-09-18T07:49:32.688553+0000","last_unstale":"2024-09-18T07:49:41.169516+0000","last_undegraded":"2024-09-18T07:49:41.169516+0000","last_fullsized":"2024-09-18T07:49:41.169516+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:14.884604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:14.884604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":465,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172047+0000","last_change":"2024-09-18T07:48:32.256620+0000","last_active":"2024-09-18T07:49:41.172047+0000","last_peered":"2024-09-18T07:49:41.172047+0000","last_clean":"2024-09-18T07:49:41.172047+0000","last_became_active":"2024-09-18T07:48:32.255219+0000","last_became_peered":"2024-09-18T07:48:32.255219+0000","last_unstale":"2024-09-18T07:49:41.172047+0000","last_undegraded":"2024-09-18T07:49:41.172047+0000","last_fullsized":"2024-09-18T07:49:41.172047+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:11.545358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:11.545358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":445,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.203214+0000","last_change":"2024-09-18T07:48:32.257934+0000","last_active":"2024-09-18T07:49:37.203214+0000","last_peered":"2024-09-18T07:49:37.203214+0000","last_clean":"2024-09-18T07:49:37.203214+0000","last_became_active":"2024-09-18T07:48:32.257676+0000","last_became_peered":"2024-09-18T07:48:32.257676+0000","last_unstale":"2024-09-18T07:49:37.203214+0000","last_undegraded":"2024-09-18T07:49:37.203214+0000","last_fullsized":"2024-09-18T07:49:37.203214+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:57.674688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:57.674688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"323'27","reported_seq":574,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169709+0000","last_change":"2024-09-18T07:48:32.299123+0000","last_active":"2024-09-18T07:49:41.169709+0000","last_peered":"2024-09-18T07:49:41.169709+0000","last_clean":"2024-09-18T07:49:41.169709+0000","last_became_active":"2024-09-18T07:48:32.261006+0000","last_became_peered":"2024-09-18T07:48:32.261006+0000","last_unstale":"2024-09-18T07:49:41.169709+0000","last_undegraded":"2024-09-18T07:49:41.169709+0000","last_fullsized":"2024-09-18T07:49:41.169709+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:46:54.956330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:54.956330+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":915,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169029+0000","last_change":"2024-09-18T07:47:52.060299+0000","last_active":"2024-09-18T07:49:41.169029+0000","last_peered":"2024-09-18T07:49:41.169029+0000","last_clean":"2024-09-18T07:49:41.169029+0000","last_became_active":"2024-09-18T07:47:20.136358+0000","last_became_peered":"2024-09-18T07:47:20.136358+0000","last_unstale":"2024-09-18T07:49:41.169029+0000","last_undegraded":"2024-09-18T07:49:41.169029+0000","last_fullsized":"2024-09-18T07:49:41.169029+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:52.060246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.060246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":910,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243317+0000","last_change":"2024-09-18T07:48:34.470535+0000","last_active":"2024-09-18T07:49:38.243317+0000","last_peered":"2024-09-18T07:49:38.243317+0000","last_clean":"2024-09-18T07:49:38.243317+0000","last_became_active":"2024-09-18T07:47:11.200300+0000","last_became_peered":"2024-09-18T07:47:11.200300+0000","last_unstale":"2024-09-18T07:49:38.243317+0000","last_undegraded":"2024-09-18T07:49:38.243317+0000","last_fullsized":"2024-09-18T07:49:38.243317+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:34.470475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:34.470475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:59.691917+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"323'15","reported_seq":929,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169115+0000","last_change":"2024-09-18T07:48:32.305165+0000","last_active":"2024-09-18T07:49:41.169115+0000","last_peered":"2024-09-18T07:49:41.169115+0000","last_clean":"2024-09-18T07:49:41.169115+0000","last_became_active":"2024-09-18T07:48:32.264535+0000","last_became_peered":"2024-09-18T07:48:32.264535+0000","last_unstale":"2024-09-18T07:49:41.169115+0000","last_undegraded":"2024-09-18T07:49:41.169115+0000","last_fullsized":"2024-09-18T07:49:41.169115+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:47:47.157636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:47.157636+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":847,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243220+0000","last_change":"2024-09-18T07:49:14.687448+0000","last_active":"2024-09-18T07:49:38.243220+0000","last_peered":"2024-09-18T07:49:38.243220+0000","last_clean":"2024-09-18T07:49:38.243220+0000","last_became_active":"2024-09-18T07:47:11.191975+0000","last_became_peered":"2024-09-18T07:47:11.191975+0000","last_unstale":"2024-09-18T07:49:38.243220+0000","last_undegraded":"2024-09-18T07:49:38.243220+0000","last_fullsized":"2024-09-18T07:49:38.243220+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:14.687386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:49:14.687386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:39.954301+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":967,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171449+0000","last_change":"2024-09-18T07:47:55.019436+0000","last_active":"2024-09-18T07:49:41.171449+0000","last_peered":"2024-09-18T07:49:41.171449+0000","last_clean":"2024-09-18T07:49:41.171449+0000","last_became_active":"2024-09-18T07:47:53.329870+0000","last_became_peered":"2024-09-18T07:47:53.329870+0000","last_unstale":"2024-09-18T07:49:41.171449+0000","last_undegraded":"2024-09-18T07:49:41.171449+0000","last_fullsized":"2024-09-18T07:49:41.171449+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:55.019244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:55.019244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":861,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243779+0000","last_change":"2024-09-18T07:47:20.142711+0000","last_active":"2024-09-18T07:49:38.243779+0000","last_peered":"2024-09-18T07:49:38.243779+0000","last_clean":"2024-09-18T07:49:38.243779+0000","last_became_active":"2024-09-18T07:47:20.142404+0000","last_became_peered":"2024-09-18T07:47:20.142404+0000","last_unstale":"2024-09-18T07:49:38.243779+0000","last_undegraded":"2024-09-18T07:49:38.243779+0000","last_fullsized":"2024-09-18T07:49:38.243779+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:57.358853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:57.358853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:40.591263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"323'7","reported_seq":1005,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169426+0000","last_change":"2024-09-18T07:49:32.699096+0000","last_active":"2024-09-18T07:49:41.169426+0000","last_peered":"2024-09-18T07:49:41.169426+0000","last_clean":"2024-09-18T07:49:41.169426+0000","last_became_active":"2024-09-18T07:49:32.689928+0000","last_became_peered":"2024-09-18T07:49:32.689928+0000","last_unstale":"2024-09-18T07:49:41.169426+0000","last_undegraded":"2024-09-18T07:49:41.169426+0000","last_fullsized":"2024-09-18T07:49:41.169426+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'7","last_scrub_stamp":"2024-09-18T07:42:46.496916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:46.496916+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:40.792920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":780,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171738+0000","last_change":"2024-09-18T07:47:20.143380+0000","last_active":"2024-09-18T07:49:41.171738+0000","last_peered":"2024-09-18T07:49:41.171738+0000","last_clean":"2024-09-18T07:49:41.171738+0000","last_became_active":"2024-09-18T07:47:20.142891+0000","last_became_peered":"2024-09-18T07:47:20.142891+0000","last_unstale":"2024-09-18T07:49:41.171738+0000","last_undegraded":"2024-09-18T07:49:41.171738+0000","last_fullsized":"2024-09-18T07:49:41.171738+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:53.754622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:53.754622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":904,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243049+0000","last_change":"2024-09-18T07:48:32.255121+0000","last_active":"2024-09-18T07:49:38.243049+0000","last_peered":"2024-09-18T07:49:38.243049+0000","last_clean":"2024-09-18T07:49:38.243049+0000","last_became_active":"2024-09-18T07:48:32.254682+0000","last_became_peered":"2024-09-18T07:48:32.254682+0000","last_unstale":"2024-09-18T07:49:38.243049+0000","last_undegraded":"2024-09-18T07:49:38.243049+0000","last_fullsized":"2024-09-18T07:49:38.243049+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:48.995794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:48.995794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":939,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169968+0000","last_change":"2024-09-18T07:48:31.926895+0000","last_active":"2024-09-18T07:49:41.169968+0000","last_peered":"2024-09-18T07:49:41.169968+0000","last_clean":"2024-09-18T07:49:41.169968+0000","last_became_active":"2024-09-18T07:47:11.187916+0000","last_became_peered":"2024-09-18T07:47:11.187916+0000","last_unstale":"2024-09-18T07:49:41.169968+0000","last_undegraded":"2024-09-18T07:49:41.169968+0000","last_fullsized":"2024-09-18T07:49:41.169968+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:31.926847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:31.926847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:47.604755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1010,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.213728+0000","last_change":"2024-09-18T07:47:49.150861+0000","last_active":"2024-09-18T07:49:38.213728+0000","last_peered":"2024-09-18T07:49:38.213728+0000","last_clean":"2024-09-18T07:49:38.213728+0000","last_became_active":"2024-09-18T07:44:43.773339+0000","last_became_peered":"2024-09-18T07:44:43.773339+0000","last_unstale":"2024-09-18T07:49:38.213728+0000","last_undegraded":"2024-09-18T07:49:38.213728+0000","last_fullsized":"2024-09-18T07:49:38.213728+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:49.150795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:49.150795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:37.524851+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"323'27","reported_seq":1132,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171143+0000","last_change":"2024-09-18T07:47:20.183463+0000","last_active":"2024-09-18T07:49:41.171143+0000","last_peered":"2024-09-18T07:49:41.171143+0000","last_clean":"2024-09-18T07:49:41.171143+0000","last_became_active":"2024-09-18T07:47:20.140525+0000","last_became_peered":"2024-09-18T07:47:20.140525+0000","last_unstale":"2024-09-18T07:49:41.171143+0000","last_undegraded":"2024-09-18T07:49:41.171143+0000","last_fullsized":"2024-09-18T07:49:41.171143+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:45:53.825805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:45:53.825805+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":569,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169770+0000","last_change":"2024-09-18T07:47:20.133202+0000","last_active":"2024-09-18T07:49:41.169770+0000","last_peered":"2024-09-18T07:49:41.169770+0000","last_clean":"2024-09-18T07:49:41.169770+0000","last_became_active":"2024-09-18T07:47:20.131623+0000","last_became_peered":"2024-09-18T07:47:20.131623+0000","last_unstale":"2024-09-18T07:49:41.169770+0000","last_undegraded":"2024-09-18T07:49:41.169770+0000","last_fullsized":"2024-09-18T07:49:41.169770+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:18.008882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:18.008882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":869,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.203876+0000","last_change":"2024-09-18T07:49:32.685656+0000","last_active":"2024-09-18T07:49:37.203876+0000","last_peered":"2024-09-18T07:49:37.203876+0000","last_clean":"2024-09-18T07:49:37.203876+0000","last_became_active":"2024-09-18T07:49:32.685347+0000","last_became_peered":"2024-09-18T07:49:32.685347+0000","last_unstale":"2024-09-18T07:49:37.203876+0000","last_undegraded":"2024-09-18T07:49:37.203876+0000","last_fullsized":"2024-09-18T07:49:37.203876+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:44.397262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:44.397262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":931,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169409+0000","last_change":"2024-09-18T07:47:56.099010+0000","last_active":"2024-09-18T07:49:41.169409+0000","last_peered":"2024-09-18T07:49:41.169409+0000","last_clean":"2024-09-18T07:49:41.169409+0000","last_became_active":"2024-09-18T07:47:20.144620+0000","last_became_peered":"2024-09-18T07:47:20.144620+0000","last_unstale":"2024-09-18T07:49:41.169409+0000","last_undegraded":"2024-09-18T07:49:41.169409+0000","last_fullsized":"2024-09-18T07:49:41.169409+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:56.098927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.098927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"323'18","reported_seq":297,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169747+0000","last_change":"2024-09-18T07:48:32.293504+0000","last_active":"2024-09-18T07:49:41.169747+0000","last_peered":"2024-09-18T07:49:41.169747+0000","last_clean":"2024-09-18T07:49:41.169747+0000","last_became_active":"2024-09-18T07:48:32.266466+0000","last_became_peered":"2024-09-18T07:48:32.266466+0000","last_unstale":"2024-09-18T07:49:41.169747+0000","last_undegraded":"2024-09-18T07:49:41.169747+0000","last_fullsized":"2024-09-18T07:49:41.169747+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:47:53.905251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.905251+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":852,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169695+0000","last_change":"2024-09-18T07:48:32.264906+0000","last_active":"2024-09-18T07:49:41.169695+0000","last_peered":"2024-09-18T07:49:41.169695+0000","last_clean":"2024-09-18T07:49:41.169695+0000","last_became_active":"2024-09-18T07:48:32.264578+0000","last_became_peered":"2024-09-18T07:48:32.264578+0000","last_unstale":"2024-09-18T07:49:41.169695+0000","last_undegraded":"2024-09-18T07:49:41.169695+0000","last_fullsized":"2024-09-18T07:49:41.169695+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.930435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.930435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:20.674047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":778,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172116+0000","last_change":"2024-09-18T07:47:53.034043+0000","last_active":"2024-09-18T07:49:41.172116+0000","last_peered":"2024-09-18T07:49:41.172116+0000","last_clean":"2024-09-18T07:49:41.172116+0000","last_became_active":"2024-09-18T07:47:11.194447+0000","last_became_peered":"2024-09-18T07:47:11.194447+0000","last_unstale":"2024-09-18T07:49:41.172116+0000","last_undegraded":"2024-09-18T07:49:41.172116+0000","last_fullsized":"2024-09-18T07:49:41.172116+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:47:53.033975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.033975+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":82,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169072+0000","last_change":"2024-09-18T07:48:32.274975+0000","last_active":"2024-09-18T07:49:41.169072+0000","last_peered":"2024-09-18T07:49:41.169072+0000","last_clean":"2024-09-18T07:49:41.169072+0000","last_became_active":"2024-09-18T07:48:32.262299+0000","last_became_peered":"2024-09-18T07:48:32.262299+0000","last_unstale":"2024-09-18T07:49:41.169072+0000","last_undegraded":"2024-09-18T07:49:41.169072+0000","last_fullsized":"2024-09-18T07:49:41.169072+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:09.496455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.496455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"323'20","reported_seq":755,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172571+0000","last_change":"2024-09-18T07:48:19.962428+0000","last_active":"2024-09-18T07:49:41.172571+0000","last_peered":"2024-09-18T07:49:41.172571+0000","last_clean":"2024-09-18T07:49:41.172571+0000","last_became_active":"2024-09-18T07:48:10.157273+0000","last_became_peered":"2024-09-18T07:48:10.157273+0000","last_unstale":"2024-09-18T07:49:41.172571+0000","last_undegraded":"2024-09-18T07:49:41.172571+0000","last_fullsized":"2024-09-18T07:49:41.172571+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:48:19.962346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:19.962346+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"398'14","reported_seq":770,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.203979+0000","last_change":"2024-09-18T07:45:53.887882+0000","last_active":"2024-09-18T07:49:37.203979+0000","last_peered":"2024-09-18T07:49:37.203979+0000","last_clean":"2024-09-18T07:49:37.203979+0000","last_became_active":"2024-09-18T07:44:43.786594+0000","last_became_peered":"2024-09-18T07:44:43.786594+0000","last_unstale":"2024-09-18T07:49:37.203979+0000","last_undegraded":"2024-09-18T07:49:37.203979+0000","last_fullsized":"2024-09-18T07:49:37.203979+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:45:53.887755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:45:53.887755+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"528'10406","reported_seq":21188,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.860304+0000","last_change":"2024-09-18T07:49:41.190449+0000","last_active":"2024-09-18T07:49:41.860304+0000","last_peered":"2024-09-18T07:49:41.860304+0000","last_clean":"2024-09-18T07:49:41.860304+0000","last_became_active":"2024-09-18T07:49:41.174729+0000","last_became_peered":"2024-09-18T07:49:41.174729+0000","last_unstale":"2024-09-18T07:49:41.860304+0000","last_undegraded":"2024-09-18T07:49:41.860304+0000","last_fullsized":"2024-09-18T07:49:41.860304+0000","mapping_epoch":527,"log_start":"495'6036","ondisk_log_start":"495'6036","created":459,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4370,"log_dups_size":0,"ondisk_log_size":4370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254803968,"num_objects":3888,"num_object_clones":0,"num_object_copies":11664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10406,"num_write_kb":457408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5294,"num_bytes_recovered":346947584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":414,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.244099+0000","last_change":"2024-09-18T07:47:13.167529+0000","last_active":"2024-09-18T07:49:38.244099+0000","last_peered":"2024-09-18T07:49:38.244099+0000","last_clean":"2024-09-18T07:49:38.244099+0000","last_became_active":"2024-09-18T07:47:11.203901+0000","last_became_peered":"2024-09-18T07:47:11.203901+0000","last_unstale":"2024-09-18T07:49:38.244099+0000","last_undegraded":"2024-09-18T07:49:38.244099+0000","last_fullsized":"2024-09-18T07:49:38.244099+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:47:13.167470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:47:13.167470+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:46.555145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"323'27","reported_seq":393,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170051+0000","last_change":"2024-09-18T07:48:32.312362+0000","last_active":"2024-09-18T07:49:41.170051+0000","last_peered":"2024-09-18T07:49:41.170051+0000","last_clean":"2024-09-18T07:49:41.170051+0000","last_became_active":"2024-09-18T07:48:32.268036+0000","last_became_peered":"2024-09-18T07:48:32.268036+0000","last_unstale":"2024-09-18T07:49:41.170051+0000","last_undegraded":"2024-09-18T07:49:41.170051+0000","last_fullsized":"2024-09-18T07:49:41.170051+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:42:33.391715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:33.391715+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:25.785407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":962,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171782+0000","last_change":"2024-09-18T07:48:32.250884+0000","last_active":"2024-09-18T07:49:41.171782+0000","last_peered":"2024-09-18T07:49:41.171782+0000","last_clean":"2024-09-18T07:49:41.171782+0000","last_became_active":"2024-09-18T07:48:32.250451+0000","last_became_peered":"2024-09-18T07:48:32.250451+0000","last_unstale":"2024-09-18T07:49:41.171782+0000","last_undegraded":"2024-09-18T07:49:41.171782+0000","last_fullsized":"2024-09-18T07:49:41.171782+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:25.143715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:25.143715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"461'12","reported_seq":1082,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170048+0000","last_change":"2024-09-18T07:48:30.963331+0000","last_active":"2024-09-18T07:49:41.170048+0000","last_peered":"2024-09-18T07:49:41.170048+0000","last_clean":"2024-09-18T07:49:41.170048+0000","last_became_active":"2024-09-18T07:47:11.190670+0000","last_became_peered":"2024-09-18T07:47:11.190670+0000","last_unstale":"2024-09-18T07:49:41.170048+0000","last_undegraded":"2024-09-18T07:49:41.170048+0000","last_fullsized":"2024-09-18T07:49:41.170048+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'12","last_scrub_stamp":"2024-09-18T07:48:30.963258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:30.963258+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:56.466762+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":843,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243549+0000","last_change":"2024-09-18T07:49:32.687377+0000","last_active":"2024-09-18T07:49:38.243549+0000","last_peered":"2024-09-18T07:49:38.243549+0000","last_clean":"2024-09-18T07:49:38.243549+0000","last_became_active":"2024-09-18T07:49:32.687063+0000","last_became_peered":"2024-09-18T07:49:32.687063+0000","last_unstale":"2024-09-18T07:49:38.243549+0000","last_undegraded":"2024-09-18T07:49:38.243549+0000","last_fullsized":"2024-09-18T07:49:38.243549+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:45.670301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:45.670301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:34.512574+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"528'10305","reported_seq":12127,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.861605+0000","last_change":"2024-09-18T07:47:37.208096+0000","last_active":"2024-09-18T07:49:41.861605+0000","last_peered":"2024-09-18T07:49:41.861605+0000","last_clean":"2024-09-18T07:49:41.861605+0000","last_became_active":"2024-09-18T07:47:37.204971+0000","last_became_peered":"2024-09-18T07:47:37.204971+0000","last_unstale":"2024-09-18T07:49:41.861605+0000","last_undegraded":"2024-09-18T07:49:41.861605+0000","last_fullsized":"2024-09-18T07:49:41.861605+0000","mapping_epoch":474,"log_start":"495'5941","ondisk_log_start":"495'5941","created":459,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4364,"log_dups_size":0,"ondisk_log_size":4364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248840192,"num_objects":3797,"num_object_clones":0,"num_object_copies":11391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10305,"num_write_kb":451264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":831,"num_bytes_recovered":54460416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":458,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169833+0000","last_change":"2024-09-18T07:44:43.923223+0000","last_active":"2024-09-18T07:49:41.169833+0000","last_peered":"2024-09-18T07:49:41.169833+0000","last_clean":"2024-09-18T07:49:41.169833+0000","last_became_active":"2024-09-18T07:44:43.786396+0000","last_became_peered":"2024-09-18T07:44:43.786396+0000","last_unstale":"2024-09-18T07:49:41.169833+0000","last_undegraded":"2024-09-18T07:49:41.169833+0000","last_fullsized":"2024-09-18T07:49:41.169833+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:44:43.923179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:44:43.923179+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"323'30","reported_seq":745,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171331+0000","last_change":"2024-09-18T07:48:32.316007+0000","last_active":"2024-09-18T07:49:41.171331+0000","last_peered":"2024-09-18T07:49:41.171331+0000","last_clean":"2024-09-18T07:49:41.171331+0000","last_became_active":"2024-09-18T07:48:32.255887+0000","last_became_peered":"2024-09-18T07:48:32.255887+0000","last_unstale":"2024-09-18T07:49:41.171331+0000","last_undegraded":"2024-09-18T07:49:41.171331+0000","last_fullsized":"2024-09-18T07:49:41.171331+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'30","last_scrub_stamp":"2024-09-18T07:47:56.645991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.645991+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":726,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172605+0000","last_change":"2024-09-18T07:48:32.260193+0000","last_active":"2024-09-18T07:49:41.172605+0000","last_peered":"2024-09-18T07:49:41.172605+0000","last_clean":"2024-09-18T07:49:41.172605+0000","last_became_active":"2024-09-18T07:48:32.259860+0000","last_became_peered":"2024-09-18T07:48:32.259860+0000","last_unstale":"2024-09-18T07:49:41.172605+0000","last_undegraded":"2024-09-18T07:49:41.172605+0000","last_fullsized":"2024-09-18T07:49:41.172605+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:56.077323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.077323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":911,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169030+0000","last_change":"2024-09-18T07:48:32.265565+0000","last_active":"2024-09-18T07:49:41.169030+0000","last_peered":"2024-09-18T07:49:41.169030+0000","last_clean":"2024-09-18T07:49:41.169030+0000","last_became_active":"2024-09-18T07:48:32.265285+0000","last_became_peered":"2024-09-18T07:48:32.265285+0000","last_unstale":"2024-09-18T07:49:41.169030+0000","last_undegraded":"2024-09-18T07:49:41.169030+0000","last_fullsized":"2024-09-18T07:49:41.169030+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:53.056484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.056484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":820,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169208+0000","last_change":"2024-09-18T07:48:32.265789+0000","last_active":"2024-09-18T07:49:41.169208+0000","last_peered":"2024-09-18T07:49:41.169208+0000","last_clean":"2024-09-18T07:49:41.169208+0000","last_became_active":"2024-09-18T07:48:32.264093+0000","last_became_peered":"2024-09-18T07:48:32.264093+0000","last_unstale":"2024-09-18T07:49:41.169208+0000","last_undegraded":"2024-09-18T07:49:41.169208+0000","last_fullsized":"2024-09-18T07:49:41.169208+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:07.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:07.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"528'10304","reported_seq":12172,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.477714+0000","last_change":"2024-09-18T07:47:35.123117+0000","last_active":"2024-09-18T07:49:41.477714+0000","last_peered":"2024-09-18T07:49:41.477714+0000","last_clean":"2024-09-18T07:49:41.477714+0000","last_became_active":"2024-09-18T07:47:35.119437+0000","last_became_peered":"2024-09-18T07:47:35.119437+0000","last_unstale":"2024-09-18T07:49:41.477714+0000","last_undegraded":"2024-09-18T07:49:41.477714+0000","last_fullsized":"2024-09-18T07:49:41.477714+0000","mapping_epoch":472,"log_start":"495'6056","ondisk_log_start":"495'6056","created":459,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4248,"log_dups_size":0,"ondisk_log_size":4248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247595008,"num_objects":3778,"num_object_clones":0,"num_object_copies":11334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10304,"num_write_kb":450624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":55902208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":78,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243705+0000","last_change":"2024-09-18T07:48:32.459297+0000","last_active":"2024-09-18T07:49:38.243705+0000","last_peered":"2024-09-18T07:49:38.243705+0000","last_clean":"2024-09-18T07:49:38.243705+0000","last_became_active":"2024-09-18T07:48:32.271495+0000","last_became_peered":"2024-09-18T07:48:32.271495+0000","last_unstale":"2024-09-18T07:49:38.243705+0000","last_undegraded":"2024-09-18T07:49:38.243705+0000","last_fullsized":"2024-09-18T07:49:38.243705+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:32.459229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:32.459229+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:01.535886+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"323'28","reported_seq":939,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169285+0000","last_change":"2024-09-18T07:48:26.993988+0000","last_active":"2024-09-18T07:49:41.169285+0000","last_peered":"2024-09-18T07:49:41.169285+0000","last_clean":"2024-09-18T07:49:41.169285+0000","last_became_active":"2024-09-18T07:47:11.188002+0000","last_became_peered":"2024-09-18T07:47:11.188002+0000","last_unstale":"2024-09-18T07:49:41.169285+0000","last_undegraded":"2024-09-18T07:49:41.169285+0000","last_fullsized":"2024-09-18T07:49:41.169285+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'28","last_scrub_stamp":"2024-09-18T07:48:26.993871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:26.993871+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:44.242031+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":1012,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.215185+0000","last_change":"2024-09-18T07:47:48.117798+0000","last_active":"2024-09-18T07:49:38.215185+0000","last_peered":"2024-09-18T07:49:38.215185+0000","last_clean":"2024-09-18T07:49:38.215185+0000","last_became_active":"2024-09-18T07:47:20.138745+0000","last_became_peered":"2024-09-18T07:47:20.138745+0000","last_unstale":"2024-09-18T07:49:38.215185+0000","last_undegraded":"2024-09-18T07:49:38.215185+0000","last_fullsized":"2024-09-18T07:49:38.215185+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:48.117629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:48.117629+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"500'13","reported_seq":967,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.168985+0000","last_change":"2024-09-18T07:48:32.276078+0000","last_active":"2024-09-18T07:49:41.168985+0000","last_peered":"2024-09-18T07:49:41.168985+0000","last_clean":"2024-09-18T07:49:41.168985+0000","last_became_active":"2024-09-18T07:48:32.254811+0000","last_became_peered":"2024-09-18T07:48:32.254811+0000","last_unstale":"2024-09-18T07:49:41.168985+0000","last_undegraded":"2024-09-18T07:49:41.168985+0000","last_fullsized":"2024-09-18T07:49:41.168985+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'12","last_scrub_stamp":"2024-09-18T07:48:20.859791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:20.859791+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":771,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171905+0000","last_change":"2024-09-18T07:47:42.984418+0000","last_active":"2024-09-18T07:49:41.171905+0000","last_peered":"2024-09-18T07:49:41.171905+0000","last_clean":"2024-09-18T07:49:41.171905+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:49:41.171905+0000","last_undegraded":"2024-09-18T07:49:41.171905+0000","last_fullsized":"2024-09-18T07:49:41.171905+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:42.984298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.984298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"528'10586","reported_seq":10737,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.756060+0000","last_change":"2024-09-18T07:47:10.178170+0000","last_active":"2024-09-18T07:49:41.756060+0000","last_peered":"2024-09-18T07:49:41.756060+0000","last_clean":"2024-09-18T07:49:41.756060+0000","last_became_active":"2024-09-18T07:47:10.177901+0000","last_became_peered":"2024-09-18T07:47:10.177901+0000","last_unstale":"2024-09-18T07:49:41.756060+0000","last_undegraded":"2024-09-18T07:49:41.756060+0000","last_fullsized":"2024-09-18T07:49:41.756060+0000","mapping_epoch":459,"log_start":"490'5364","ondisk_log_start":"490'5364","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":5222,"log_dups_size":0,"ondisk_log_size":5222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252248088,"num_objects":3850,"num_object_clones":0,"num_object_copies":11550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3850,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10586,"num_write_kb":461889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":714,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.203538+0000","last_change":"2024-09-18T07:48:10.151652+0000","last_active":"2024-09-18T07:49:37.203538+0000","last_peered":"2024-09-18T07:49:37.203538+0000","last_clean":"2024-09-18T07:49:37.203538+0000","last_became_active":"2024-09-18T07:48:10.151343+0000","last_became_peered":"2024-09-18T07:48:10.151343+0000","last_unstale":"2024-09-18T07:49:37.203538+0000","last_undegraded":"2024-09-18T07:49:37.203538+0000","last_fullsized":"2024-09-18T07:49:37.203538+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:02.444272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:02.444272+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"323'20","reported_seq":682,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243007+0000","last_change":"2024-09-18T07:49:32.698546+0000","last_active":"2024-09-18T07:49:38.243007+0000","last_peered":"2024-09-18T07:49:38.243007+0000","last_clean":"2024-09-18T07:49:38.243007+0000","last_became_active":"2024-09-18T07:49:32.688605+0000","last_became_peered":"2024-09-18T07:49:32.688605+0000","last_unstale":"2024-09-18T07:49:38.243007+0000","last_undegraded":"2024-09-18T07:49:38.243007+0000","last_fullsized":"2024-09-18T07:49:38.243007+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:48:33.642815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:33.642815+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:44.608266+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":893,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243107+0000","last_change":"2024-09-18T07:48:32.268542+0000","last_active":"2024-09-18T07:49:38.243107+0000","last_peered":"2024-09-18T07:49:38.243107+0000","last_clean":"2024-09-18T07:49:38.243107+0000","last_became_active":"2024-09-18T07:48:32.267802+0000","last_became_peered":"2024-09-18T07:48:32.267802+0000","last_unstale":"2024-09-18T07:49:38.243107+0000","last_undegraded":"2024-09-18T07:49:38.243107+0000","last_fullsized":"2024-09-18T07:49:38.243107+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:19.210077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:19.210077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"461'13","reported_seq":718,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.202658+0000","last_change":"2024-09-18T07:48:24.606091+0000","last_active":"2024-09-18T07:49:37.202658+0000","last_peered":"2024-09-18T07:49:37.202658+0000","last_clean":"2024-09-18T07:49:37.202658+0000","last_became_active":"2024-09-18T07:47:11.189241+0000","last_became_peered":"2024-09-18T07:47:11.189241+0000","last_unstale":"2024-09-18T07:49:37.202658+0000","last_undegraded":"2024-09-18T07:49:37.202658+0000","last_fullsized":"2024-09-18T07:49:37.202658+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'13","last_scrub_stamp":"2024-09-18T07:48:24.605876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:24.605876+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":71,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169084+0000","last_change":"2024-09-18T07:48:32.258814+0000","last_active":"2024-09-18T07:49:41.169084+0000","last_peered":"2024-09-18T07:49:41.169084+0000","last_clean":"2024-09-18T07:49:41.169084+0000","last_became_active":"2024-09-18T07:48:32.258477+0000","last_became_peered":"2024-09-18T07:48:32.258477+0000","last_unstale":"2024-09-18T07:49:41.169084+0000","last_undegraded":"2024-09-18T07:49:41.169084+0000","last_fullsized":"2024-09-18T07:49:41.169084+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:46.172671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:46.172671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.c","version":"528'10082","reported_seq":11839,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.779351+0000","last_change":"2024-09-18T07:47:31.131323+0000","last_active":"2024-09-18T07:49:41.779351+0000","last_peered":"2024-09-18T07:49:41.779351+0000","last_clean":"2024-09-18T07:49:41.779351+0000","last_became_active":"2024-09-18T07:47:31.128537+0000","last_became_peered":"2024-09-18T07:47:31.128537+0000","last_unstale":"2024-09-18T07:49:41.779351+0000","last_undegraded":"2024-09-18T07:49:41.779351+0000","last_fullsized":"2024-09-18T07:49:41.779351+0000","mapping_epoch":468,"log_start":"489'4815","ondisk_log_start":"489'4815","created":459,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":5267,"log_dups_size":0,"ondisk_log_size":5267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248774656,"num_objects":3796,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10082,"num_write_kb":444096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":799,"num_bytes_recovered":52363264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":722,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.203127+0000","last_change":"2024-09-18T07:48:32.273778+0000","last_active":"2024-09-18T07:49:37.203127+0000","last_peered":"2024-09-18T07:49:37.203127+0000","last_clean":"2024-09-18T07:49:37.203127+0000","last_became_active":"2024-09-18T07:48:32.273227+0000","last_became_peered":"2024-09-18T07:48:32.273227+0000","last_unstale":"2024-09-18T07:49:37.203127+0000","last_undegraded":"2024-09-18T07:49:37.203127+0000","last_fullsized":"2024-09-18T07:49:37.203127+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:47:59.422927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:47:59.422927+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"323'29","reported_seq":809,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243458+0000","last_change":"2024-09-18T07:48:01.062906+0000","last_active":"2024-09-18T07:49:38.243458+0000","last_peered":"2024-09-18T07:49:38.243458+0000","last_clean":"2024-09-18T07:49:38.243458+0000","last_became_active":"2024-09-18T07:47:11.198126+0000","last_became_peered":"2024-09-18T07:47:11.198126+0000","last_unstale":"2024-09-18T07:49:38.243458+0000","last_undegraded":"2024-09-18T07:49:38.243458+0000","last_fullsized":"2024-09-18T07:49:38.243458+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'29","last_scrub_stamp":"2024-09-18T07:48:01.062822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:01.062822+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":708,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171242+0000","last_change":"2024-09-18T07:47:20.140174+0000","last_active":"2024-09-18T07:49:41.171242+0000","last_peered":"2024-09-18T07:49:41.171242+0000","last_clean":"2024-09-18T07:49:41.171242+0000","last_became_active":"2024-09-18T07:47:20.139806+0000","last_became_peered":"2024-09-18T07:47:20.139806+0000","last_unstale":"2024-09-18T07:49:41.171242+0000","last_undegraded":"2024-09-18T07:49:41.171242+0000","last_fullsized":"2024-09-18T07:49:41.171242+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:43.985969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:43.985969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":629,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.202674+0000","last_change":"2024-09-18T07:47:41.319363+0000","last_active":"2024-09-18T07:49:37.202674+0000","last_peered":"2024-09-18T07:49:37.202674+0000","last_clean":"2024-09-18T07:49:37.202674+0000","last_became_active":"2024-09-18T07:47:11.193892+0000","last_became_peered":"2024-09-18T07:47:11.193892+0000","last_unstale":"2024-09-18T07:49:37.202674+0000","last_undegraded":"2024-09-18T07:49:37.202674+0000","last_fullsized":"2024-09-18T07:49:37.202674+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:41.319318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:41.319318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":624,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.203904+0000","last_change":"2024-09-18T07:47:20.138912+0000","last_active":"2024-09-18T07:49:37.203904+0000","last_peered":"2024-09-18T07:49:37.203904+0000","last_clean":"2024-09-18T07:49:37.203904+0000","last_became_active":"2024-09-18T07:47:20.138390+0000","last_became_peered":"2024-09-18T07:47:20.138390+0000","last_unstale":"2024-09-18T07:49:37.203904+0000","last_undegraded":"2024-09-18T07:49:37.203904+0000","last_fullsized":"2024-09-18T07:49:37.203904+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:49.018222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:49.018222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.b","version":"528'10221","reported_seq":24247,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.449274+0000","last_change":"2024-09-18T07:49:19.420062+0000","last_active":"2024-09-18T07:49:41.449274+0000","last_peered":"2024-09-18T07:49:41.449274+0000","last_clean":"2024-09-18T07:49:41.449274+0000","last_became_active":"2024-09-18T07:49:19.393415+0000","last_became_peered":"2024-09-18T07:49:19.393415+0000","last_unstale":"2024-09-18T07:49:41.449274+0000","last_undegraded":"2024-09-18T07:49:41.449274+0000","last_fullsized":"2024-09-18T07:49:41.449274+0000","mapping_epoch":512,"log_start":"496'6014","ondisk_log_start":"496'6014","created":459,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4207,"log_dups_size":0,"ondisk_log_size":4207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242941952,"num_objects":3707,"num_object_clones":0,"num_object_copies":11121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10221,"num_write_kb":445696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6927,"num_bytes_recovered":453967872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":442,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243084+0000","last_change":"2024-09-18T07:48:59.662366+0000","last_active":"2024-09-18T07:49:38.243084+0000","last_peered":"2024-09-18T07:49:38.243084+0000","last_clean":"2024-09-18T07:49:38.243084+0000","last_became_active":"2024-09-18T07:47:11.195706+0000","last_became_peered":"2024-09-18T07:47:11.195706+0000","last_unstale":"2024-09-18T07:49:38.243084+0000","last_undegraded":"2024-09-18T07:49:38.243084+0000","last_fullsized":"2024-09-18T07:49:38.243084+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:59.662286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:59.662286+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:04.207309+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"519'45","reported_seq":1059,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171086+0000","last_change":"2024-09-18T07:49:24.465774+0000","last_active":"2024-09-18T07:49:41.171086+0000","last_peered":"2024-09-18T07:49:41.171086+0000","last_clean":"2024-09-18T07:49:41.171086+0000","last_became_active":"2024-09-18T07:49:24.452427+0000","last_became_peered":"2024-09-18T07:49:24.452427+0000","last_unstale":"2024-09-18T07:49:41.171086+0000","last_undegraded":"2024-09-18T07:49:41.171086+0000","last_fullsized":"2024-09-18T07:49:41.171086+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'41","last_scrub_stamp":"2024-09-18T07:46:12.858570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:12.858570+0000","objects_scrubbed":10,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":910,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169490+0000","last_change":"2024-09-18T07:47:51.085464+0000","last_active":"2024-09-18T07:49:41.169490+0000","last_peered":"2024-09-18T07:49:41.169490+0000","last_clean":"2024-09-18T07:49:41.169490+0000","last_became_active":"2024-09-18T07:44:43.780114+0000","last_became_peered":"2024-09-18T07:44:43.780114+0000","last_unstale":"2024-09-18T07:49:41.169490+0000","last_undegraded":"2024-09-18T07:49:41.169490+0000","last_fullsized":"2024-09-18T07:49:41.169490+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:51.085400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:51.085400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":878,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243115+0000","last_change":"2024-09-18T07:48:06.127800+0000","last_active":"2024-09-18T07:49:38.243115+0000","last_peered":"2024-09-18T07:49:38.243115+0000","last_clean":"2024-09-18T07:49:38.243115+0000","last_became_active":"2024-09-18T07:47:20.144312+0000","last_became_peered":"2024-09-18T07:47:20.144312+0000","last_unstale":"2024-09-18T07:49:38.243115+0000","last_undegraded":"2024-09-18T07:49:38.243115+0000","last_fullsized":"2024-09-18T07:49:38.243115+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:48:06.127745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:48:06.127745+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":987,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.214721+0000","last_change":"2024-09-18T07:47:20.131135+0000","last_active":"2024-09-18T07:49:38.214721+0000","last_peered":"2024-09-18T07:49:38.214721+0000","last_clean":"2024-09-18T07:49:38.214721+0000","last_became_active":"2024-09-18T07:47:20.130809+0000","last_became_peered":"2024-09-18T07:47:20.130809+0000","last_unstale":"2024-09-18T07:49:38.214721+0000","last_undegraded":"2024-09-18T07:49:38.214721+0000","last_fullsized":"2024-09-18T07:49:38.214721+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:46.629196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:46.629196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:27.458621+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":790,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170264+0000","last_change":"2024-09-18T07:49:24.445613+0000","last_active":"2024-09-18T07:49:41.170264+0000","last_peered":"2024-09-18T07:49:41.170264+0000","last_clean":"2024-09-18T07:49:41.170264+0000","last_became_active":"2024-09-18T07:49:24.445300+0000","last_became_peered":"2024-09-18T07:49:24.445300+0000","last_unstale":"2024-09-18T07:49:41.170264+0000","last_undegraded":"2024-09-18T07:49:41.170264+0000","last_fullsized":"2024-09-18T07:49:41.170264+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:52.882700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:45:52.882700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.8","version":"524'10045","reported_seq":22128,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.203692+0000","last_change":"2024-09-18T07:49:22.516716+0000","last_active":"2024-09-18T07:49:37.203692+0000","last_peered":"2024-09-18T07:49:37.203692+0000","last_clean":"2024-09-18T07:49:37.203692+0000","last_became_active":"2024-09-18T07:49:22.442874+0000","last_became_peered":"2024-09-18T07:49:22.442874+0000","last_unstale":"2024-09-18T07:49:37.203692+0000","last_undegraded":"2024-09-18T07:49:37.203692+0000","last_fullsized":"2024-09-18T07:49:37.203692+0000","mapping_epoch":515,"log_start":"493'5791","ondisk_log_start":"493'5791","created":459,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'5883","last_scrub_stamp":"2024-09-18T07:48:25.402648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:48:25.402648+0000","objects_scrubbed":5819,"log_size":4254,"log_dups_size":0,"ondisk_log_size":4254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:44.883793+0000","scrub_duration":1753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259850240,"num_objects":3965,"num_object_clones":0,"num_object_copies":11895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10045,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5929,"num_bytes_recovered":388431872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":832,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171491+0000","last_change":"2024-09-18T07:48:27.916178+0000","last_active":"2024-09-18T07:49:41.171491+0000","last_peered":"2024-09-18T07:49:41.171491+0000","last_clean":"2024-09-18T07:49:41.171491+0000","last_became_active":"2024-09-18T07:44:43.783295+0000","last_became_peered":"2024-09-18T07:44:43.783295+0000","last_unstale":"2024-09-18T07:49:41.171491+0000","last_undegraded":"2024-09-18T07:49:41.171491+0000","last_fullsized":"2024-09-18T07:49:41.171491+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:27.916095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:27.916095+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"323'17","reported_seq":708,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243619+0000","last_change":"2024-09-18T07:49:32.704380+0000","last_active":"2024-09-18T07:49:38.243619+0000","last_peered":"2024-09-18T07:49:38.243619+0000","last_clean":"2024-09-18T07:49:38.243619+0000","last_became_active":"2024-09-18T07:49:32.690652+0000","last_became_peered":"2024-09-18T07:49:32.690652+0000","last_unstale":"2024-09-18T07:49:38.243619+0000","last_undegraded":"2024-09-18T07:49:38.243619+0000","last_fullsized":"2024-09-18T07:49:38.243619+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'17","last_scrub_stamp":"2024-09-18T07:47:56.037047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.037047+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":23,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":721,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169359+0000","last_change":"2024-09-18T07:47:20.140678+0000","last_active":"2024-09-18T07:49:41.169359+0000","last_peered":"2024-09-18T07:49:41.169359+0000","last_clean":"2024-09-18T07:49:41.169359+0000","last_became_active":"2024-09-18T07:47:20.140376+0000","last_became_peered":"2024-09-18T07:47:20.140376+0000","last_unstale":"2024-09-18T07:49:41.169359+0000","last_undegraded":"2024-09-18T07:49:41.169359+0000","last_fullsized":"2024-09-18T07:49:41.169359+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:52.917296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:52.917296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":1023,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169775+0000","last_change":"2024-09-18T07:48:32.288661+0000","last_active":"2024-09-18T07:49:41.169775+0000","last_peered":"2024-09-18T07:49:41.169775+0000","last_clean":"2024-09-18T07:49:41.169775+0000","last_became_active":"2024-09-18T07:48:32.257016+0000","last_became_peered":"2024-09-18T07:48:32.257016+0000","last_unstale":"2024-09-18T07:49:41.169775+0000","last_undegraded":"2024-09-18T07:49:41.169775+0000","last_fullsized":"2024-09-18T07:49:41.169775+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:07.841406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:47:07.841406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1011,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170668+0000","last_change":"2024-09-18T07:47:16.834107+0000","last_active":"2024-09-18T07:49:41.170668+0000","last_peered":"2024-09-18T07:49:41.170668+0000","last_clean":"2024-09-18T07:49:41.170668+0000","last_became_active":"2024-09-18T07:47:11.201379+0000","last_became_peered":"2024-09-18T07:47:11.201379+0000","last_unstale":"2024-09-18T07:49:41.170668+0000","last_undegraded":"2024-09-18T07:49:41.170668+0000","last_fullsized":"2024-09-18T07:49:41.170668+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:16.834043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:16.834043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":800,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172125+0000","last_change":"2024-09-18T07:49:24.457052+0000","last_active":"2024-09-18T07:49:41.172125+0000","last_peered":"2024-09-18T07:49:41.172125+0000","last_clean":"2024-09-18T07:49:41.172125+0000","last_became_active":"2024-09-18T07:49:24.446434+0000","last_became_peered":"2024-09-18T07:49:24.446434+0000","last_unstale":"2024-09-18T07:49:41.172125+0000","last_undegraded":"2024-09-18T07:49:41.172125+0000","last_fullsized":"2024-09-18T07:49:41.172125+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:47:42.037791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.037791+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":78,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"524'9949","reported_seq":22223,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.203300+0000","last_change":"2024-09-18T07:49:19.413579+0000","last_active":"2024-09-18T07:49:37.203300+0000","last_peered":"2024-09-18T07:49:37.203300+0000","last_clean":"2024-09-18T07:49:37.203300+0000","last_became_active":"2024-09-18T07:49:19.392861+0000","last_became_peered":"2024-09-18T07:49:19.392861+0000","last_unstale":"2024-09-18T07:49:37.203300+0000","last_undegraded":"2024-09-18T07:49:37.203300+0000","last_fullsized":"2024-09-18T07:49:37.203300+0000","mapping_epoch":512,"log_start":"493'5727","ondisk_log_start":"493'5727","created":459,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4222,"log_dups_size":0,"ondisk_log_size":4222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264044544,"num_objects":4029,"num_object_clones":0,"num_object_copies":12087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9949,"num_write_kb":447296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6057,"num_bytes_recovered":396951552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":26,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.202532+0000","last_change":"2024-09-18T07:49:24.453650+0000","last_active":"2024-09-18T07:49:37.202532+0000","last_peered":"2024-09-18T07:49:37.202532+0000","last_clean":"2024-09-18T07:49:37.202532+0000","last_became_active":"2024-09-18T07:49:24.453318+0000","last_became_peered":"2024-09-18T07:49:24.453318+0000","last_unstale":"2024-09-18T07:49:37.202532+0000","last_undegraded":"2024-09-18T07:49:37.202532+0000","last_fullsized":"2024-09-18T07:49:37.202532+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:49:12.773317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:49:12.773317+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:38.973101+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"323'24","reported_seq":1061,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.215101+0000","last_change":"2024-09-18T07:47:20.159485+0000","last_active":"2024-09-18T07:49:38.215101+0000","last_peered":"2024-09-18T07:49:38.215101+0000","last_clean":"2024-09-18T07:49:38.215101+0000","last_became_active":"2024-09-18T07:47:20.138961+0000","last_became_peered":"2024-09-18T07:47:20.138961+0000","last_unstale":"2024-09-18T07:49:38.215101+0000","last_undegraded":"2024-09-18T07:49:38.215101+0000","last_fullsized":"2024-09-18T07:49:38.215101+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'24","last_scrub_stamp":"2024-09-18T07:44:06.211676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:06.211676+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:22.448696+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":991,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.214335+0000","last_change":"2024-09-18T07:48:10.165907+0000","last_active":"2024-09-18T07:49:38.214335+0000","last_peered":"2024-09-18T07:49:38.214335+0000","last_clean":"2024-09-18T07:49:38.214335+0000","last_became_active":"2024-09-18T07:48:10.142633+0000","last_became_peered":"2024-09-18T07:48:10.142633+0000","last_unstale":"2024-09-18T07:49:38.214335+0000","last_undegraded":"2024-09-18T07:49:38.214335+0000","last_fullsized":"2024-09-18T07:49:38.214335+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:42:44.401546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:44.401546+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:21.405419+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":1646,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171999+0000","last_change":"2024-09-18T07:47:20.162839+0000","last_active":"2024-09-18T07:49:41.171999+0000","last_peered":"2024-09-18T07:49:41.171999+0000","last_clean":"2024-09-18T07:49:41.171999+0000","last_became_active":"2024-09-18T07:47:20.138901+0000","last_became_peered":"2024-09-18T07:47:20.138901+0000","last_unstale":"2024-09-18T07:49:41.171999+0000","last_undegraded":"2024-09-18T07:49:41.171999+0000","last_fullsized":"2024-09-18T07:49:41.171999+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T07:46:47.771151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:46:47.771151+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":882,"num_read_kb":877,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":872,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243829+0000","last_change":"2024-09-18T07:47:20.145165+0000","last_active":"2024-09-18T07:49:38.243829+0000","last_peered":"2024-09-18T07:49:38.243829+0000","last_clean":"2024-09-18T07:49:38.243829+0000","last_became_active":"2024-09-18T07:47:20.144413+0000","last_became_peered":"2024-09-18T07:47:20.144413+0000","last_unstale":"2024-09-18T07:49:38.243829+0000","last_undegraded":"2024-09-18T07:49:38.243829+0000","last_fullsized":"2024-09-18T07:49:38.243829+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:50.587498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:44:50.587498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":553,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243801+0000","last_change":"2024-09-18T07:49:32.691732+0000","last_active":"2024-09-18T07:49:38.243801+0000","last_peered":"2024-09-18T07:49:38.243801+0000","last_clean":"2024-09-18T07:49:38.243801+0000","last_became_active":"2024-09-18T07:49:32.691427+0000","last_became_peered":"2024-09-18T07:49:32.691427+0000","last_unstale":"2024-09-18T07:49:38.243801+0000","last_undegraded":"2024-09-18T07:49:38.243801+0000","last_fullsized":"2024-09-18T07:49:38.243801+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:52.440332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:52.440332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.a","version":"524'9952","reported_seq":21703,"reported_epoch":525,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-18T07:49:37.204378+0000","last_change":"2024-09-18T07:49:33.964097+0000","last_active":"2024-09-18T07:49:37.204378+0000","last_peered":"2024-09-18T07:49:37.204378+0000","last_clean":"2024-09-18T07:49:31.606072+0000","last_became_active":"2024-09-18T07:49:33.895295+0000","last_became_peered":"2024-09-18T07:49:33.895295+0000","last_unstale":"2024-09-18T07:49:37.204378+0000","last_undegraded":"2024-09-18T07:49:37.204378+0000","last_fullsized":"2024-09-18T07:49:37.204378+0000","mapping_epoch":522,"log_start":"493'5772","ondisk_log_start":"493'5772","created":459,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'5664","last_scrub_stamp":"2024-09-18T07:48:22.026341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:48:22.026341+0000","objects_scrubbed":5623,"log_size":4180,"log_dups_size":0,"ondisk_log_size":4180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256376832,"num_objects":3912,"num_object_clones":0,"num_object_copies":11736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3912,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9952,"num_write_kb":443648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5758,"num_bytes_recovered":377290752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,2],"avail_no_missing":["6","2","3","5"],"object_location_counts":[{"shards":"2,5,6","objects":3912}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"398'34","reported_seq":802,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.203488+0000","last_change":"2024-09-18T07:47:58.415964+0000","last_active":"2024-09-18T07:49:37.203488+0000","last_peered":"2024-09-18T07:49:37.203488+0000","last_clean":"2024-09-18T07:49:37.203488+0000","last_became_active":"2024-09-18T07:44:43.786827+0000","last_became_peered":"2024-09-18T07:44:43.786827+0000","last_unstale":"2024-09-18T07:49:37.203488+0000","last_undegraded":"2024-09-18T07:49:37.203488+0000","last_fullsized":"2024-09-18T07:49:37.203488+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'34","last_scrub_stamp":"2024-09-18T07:47:58.415734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:58.415734+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":748,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.244188+0000","last_change":"2024-09-18T07:49:08.604265+0000","last_active":"2024-09-18T07:49:38.244188+0000","last_peered":"2024-09-18T07:49:38.244188+0000","last_clean":"2024-09-18T07:49:38.244188+0000","last_became_active":"2024-09-18T07:48:32.266432+0000","last_became_peered":"2024-09-18T07:48:32.266432+0000","last_unstale":"2024-09-18T07:49:38.244188+0000","last_undegraded":"2024-09-18T07:49:38.244188+0000","last_fullsized":"2024-09-18T07:49:38.244188+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:08.604203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:08.604203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:22.110767+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":815,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171986+0000","last_change":"2024-09-18T07:48:26.963331+0000","last_active":"2024-09-18T07:49:41.171986+0000","last_peered":"2024-09-18T07:49:41.171986+0000","last_clean":"2024-09-18T07:49:41.171986+0000","last_became_active":"2024-09-18T07:39:03.350419+0000","last_became_peered":"2024-09-18T07:39:03.350419+0000","last_unstale":"2024-09-18T07:49:41.171986+0000","last_undegraded":"2024-09-18T07:49:41.171986+0000","last_fullsized":"2024-09-18T07:49:41.171986+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:26.963223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:26.963223+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":768,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172313+0000","last_change":"2024-09-18T07:48:21.934191+0000","last_active":"2024-09-18T07:49:41.172313+0000","last_peered":"2024-09-18T07:49:41.172313+0000","last_clean":"2024-09-18T07:49:41.172313+0000","last_became_active":"2024-09-18T07:47:11.195084+0000","last_became_peered":"2024-09-18T07:47:11.195084+0000","last_unstale":"2024-09-18T07:49:41.172313+0000","last_undegraded":"2024-09-18T07:49:41.172313+0000","last_fullsized":"2024-09-18T07:49:41.172313+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:21.934130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:21.934130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:49.622120+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":876,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243725+0000","last_change":"2024-09-18T07:49:32.688396+0000","last_active":"2024-09-18T07:49:38.243725+0000","last_peered":"2024-09-18T07:49:38.243725+0000","last_clean":"2024-09-18T07:49:38.243725+0000","last_became_active":"2024-09-18T07:49:32.687989+0000","last_became_peered":"2024-09-18T07:49:32.687989+0000","last_unstale":"2024-09-18T07:49:38.243725+0000","last_undegraded":"2024-09-18T07:49:38.243725+0000","last_fullsized":"2024-09-18T07:49:38.243725+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:04.610883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:04.610883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:30.506002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"524'10047","reported_seq":19488,"reported_epoch":524,"state":"peering","last_fresh":"2024-09-18T07:49:37.199989+0000","last_change":"2024-09-18T07:49:37.206494+0000","last_active":"2024-09-18T07:49:37.206494+0000","last_peered":"2024-09-18T07:49:37.199989+0000","last_clean":"2024-09-18T07:49:37.199989+0000","last_became_active":"2024-09-18T07:48:33.400291+0000","last_became_peered":"2024-09-18T07:48:33.400291+0000","last_unstale":"2024-09-18T07:49:37.199989+0000","last_undegraded":"2024-09-18T07:49:37.199989+0000","last_fullsized":"2024-09-18T07:49:37.199989+0000","mapping_epoch":525,"log_start":"493'5868","ondisk_log_start":"493'5868","created":459,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4179,"log_dups_size":0,"ondisk_log_size":4179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267059200,"num_objects":4075,"num_object_clones":0,"num_object_copies":12225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10047,"num_write_kb":451904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4643,"num_bytes_recovered":304283648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"323'19","reported_seq":1120,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170072+0000","last_change":"2024-09-18T07:49:32.696886+0000","last_active":"2024-09-18T07:49:41.170072+0000","last_peered":"2024-09-18T07:49:41.170072+0000","last_clean":"2024-09-18T07:49:41.170072+0000","last_became_active":"2024-09-18T07:49:32.684857+0000","last_became_peered":"2024-09-18T07:49:32.684857+0000","last_unstale":"2024-09-18T07:49:41.170072+0000","last_undegraded":"2024-09-18T07:49:41.170072+0000","last_fullsized":"2024-09-18T07:49:41.170072+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:42:43.402858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:43.402858+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:39.835405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":873,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169212+0000","last_change":"2024-09-18T07:44:43.793621+0000","last_active":"2024-09-18T07:49:41.169212+0000","last_peered":"2024-09-18T07:49:41.169212+0000","last_clean":"2024-09-18T07:49:41.169212+0000","last_became_active":"2024-09-18T07:44:43.790160+0000","last_became_peered":"2024-09-18T07:44:43.790160+0000","last_unstale":"2024-09-18T07:49:41.169212+0000","last_undegraded":"2024-09-18T07:49:41.169212+0000","last_fullsized":"2024-09-18T07:49:41.169212+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.078600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.078600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:19.692544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":881,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243445+0000","last_change":"2024-09-18T07:47:11.202169+0000","last_active":"2024-09-18T07:49:38.243445+0000","last_peered":"2024-09-18T07:49:38.243445+0000","last_clean":"2024-09-18T07:49:38.243445+0000","last_became_active":"2024-09-18T07:47:11.198103+0000","last_became_peered":"2024-09-18T07:47:11.198103+0000","last_unstale":"2024-09-18T07:49:38.243445+0000","last_undegraded":"2024-09-18T07:49:38.243445+0000","last_fullsized":"2024-09-18T07:49:38.243445+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:45.479624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:44:45.479624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:21.577272+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":869,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.204920+0000","last_change":"2024-09-18T07:48:32.259237+0000","last_active":"2024-09-18T07:49:37.204920+0000","last_peered":"2024-09-18T07:49:37.204920+0000","last_clean":"2024-09-18T07:49:37.204920+0000","last_became_active":"2024-09-18T07:48:32.258901+0000","last_became_peered":"2024-09-18T07:48:32.258901+0000","last_unstale":"2024-09-18T07:49:37.204920+0000","last_undegraded":"2024-09-18T07:49:37.204920+0000","last_fullsized":"2024-09-18T07:49:37.204920+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:26.691414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:26.691414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"528'10075","reported_seq":19954,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.840856+0000","last_change":"2024-09-18T07:49:33.657485+0000","last_active":"2024-09-18T07:49:41.840856+0000","last_peered":"2024-09-18T07:49:41.840856+0000","last_clean":"2024-09-18T07:49:41.840856+0000","last_became_active":"2024-09-18T07:49:33.641738+0000","last_became_peered":"2024-09-18T07:49:33.641738+0000","last_unstale":"2024-09-18T07:49:41.840856+0000","last_undegraded":"2024-09-18T07:49:41.840856+0000","last_fullsized":"2024-09-18T07:49:41.840856+0000","mapping_epoch":522,"log_start":"489'4841","ondisk_log_start":"489'4841","created":459,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":5234,"log_dups_size":0,"ondisk_log_size":5234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240582656,"num_objects":3671,"num_object_clones":0,"num_object_copies":11013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10075,"num_write_kb":439872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4854,"num_bytes_recovered":318111744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":418,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243009+0000","last_change":"2024-09-18T07:49:24.456347+0000","last_active":"2024-09-18T07:49:38.243009+0000","last_peered":"2024-09-18T07:49:38.243009+0000","last_clean":"2024-09-18T07:49:38.243009+0000","last_became_active":"2024-09-18T07:49:24.448174+0000","last_became_peered":"2024-09-18T07:49:24.448174+0000","last_unstale":"2024-09-18T07:49:38.243009+0000","last_undegraded":"2024-09-18T07:49:38.243009+0000","last_fullsized":"2024-09-18T07:49:38.243009+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:49:00.644450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:49:00.644450+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:11.512414+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"323'31","reported_seq":955,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169916+0000","last_change":"2024-09-18T07:48:22.002514+0000","last_active":"2024-09-18T07:49:41.169916+0000","last_peered":"2024-09-18T07:49:41.169916+0000","last_clean":"2024-09-18T07:49:41.169916+0000","last_became_active":"2024-09-18T07:47:20.132912+0000","last_became_peered":"2024-09-18T07:47:20.132912+0000","last_unstale":"2024-09-18T07:49:41.169916+0000","last_undegraded":"2024-09-18T07:49:41.169916+0000","last_fullsized":"2024-09-18T07:49:41.169916+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'31","last_scrub_stamp":"2024-09-18T07:48:22.002416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:22.002416+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":672,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.202849+0000","last_change":"2024-09-18T07:49:32.688728+0000","last_active":"2024-09-18T07:49:37.202849+0000","last_peered":"2024-09-18T07:49:37.202849+0000","last_clean":"2024-09-18T07:49:37.202849+0000","last_became_active":"2024-09-18T07:49:32.688352+0000","last_became_peered":"2024-09-18T07:49:32.688352+0000","last_unstale":"2024-09-18T07:49:37.202849+0000","last_undegraded":"2024-09-18T07:49:37.202849+0000","last_fullsized":"2024-09-18T07:49:37.202849+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:32.491919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:32.491919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:46.469204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":544,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169540+0000","last_change":"2024-09-18T07:47:20.131352+0000","last_active":"2024-09-18T07:49:41.169540+0000","last_peered":"2024-09-18T07:49:41.169540+0000","last_clean":"2024-09-18T07:49:41.169540+0000","last_became_active":"2024-09-18T07:47:20.130994+0000","last_became_peered":"2024-09-18T07:47:20.130994+0000","last_unstale":"2024-09-18T07:49:41.169540+0000","last_undegraded":"2024-09-18T07:49:41.169540+0000","last_fullsized":"2024-09-18T07:49:41.169540+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:13.019711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:13.019711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1016,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170635+0000","last_change":"2024-09-18T07:48:32.263327+0000","last_active":"2024-09-18T07:49:41.170635+0000","last_peered":"2024-09-18T07:49:41.170635+0000","last_clean":"2024-09-18T07:49:41.170635+0000","last_became_active":"2024-09-18T07:48:32.263024+0000","last_became_peered":"2024-09-18T07:48:32.263024+0000","last_unstale":"2024-09-18T07:49:41.170635+0000","last_undegraded":"2024-09-18T07:49:41.170635+0000","last_fullsized":"2024-09-18T07:49:41.170635+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:06.889139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:06.889139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"524'9977","reported_seq":11897,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.202752+0000","last_change":"2024-09-18T07:48:27.491090+0000","last_active":"2024-09-18T07:49:37.202752+0000","last_peered":"2024-09-18T07:49:37.202752+0000","last_clean":"2024-09-18T07:49:37.202752+0000","last_became_active":"2024-09-18T07:47:38.218858+0000","last_became_peered":"2024-09-18T07:47:38.218858+0000","last_unstale":"2024-09-18T07:49:37.202752+0000","last_undegraded":"2024-09-18T07:49:37.202752+0000","last_fullsized":"2024-09-18T07:49:37.202752+0000","mapping_epoch":475,"log_start":"493'5756","ondisk_log_start":"493'5756","created":459,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'6054","last_scrub_stamp":"2024-09-18T07:48:27.490707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:48:27.490707+0000","objects_scrubbed":6013,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:38.273123+0000","scrub_duration":1857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262471680,"num_objects":4005,"num_object_clones":0,"num_object_copies":12015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9977,"num_write_kb":447424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":848,"num_bytes_recovered":55574528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":819,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171319+0000","last_change":"2024-09-18T07:48:32.261433+0000","last_active":"2024-09-18T07:49:41.171319+0000","last_peered":"2024-09-18T07:49:41.171319+0000","last_clean":"2024-09-18T07:49:41.171319+0000","last_became_active":"2024-09-18T07:48:32.259457+0000","last_became_peered":"2024-09-18T07:48:32.259457+0000","last_unstale":"2024-09-18T07:49:41.171319+0000","last_undegraded":"2024-09-18T07:49:41.171319+0000","last_fullsized":"2024-09-18T07:49:41.171319+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:46:24.585120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:46:24.585120+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"323'20","reported_seq":927,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169753+0000","last_change":"2024-09-18T07:48:32.303165+0000","last_active":"2024-09-18T07:49:41.169753+0000","last_peered":"2024-09-18T07:49:41.169753+0000","last_clean":"2024-09-18T07:49:41.169753+0000","last_became_active":"2024-09-18T07:48:32.269075+0000","last_became_peered":"2024-09-18T07:48:32.269075+0000","last_unstale":"2024-09-18T07:49:41.169753+0000","last_undegraded":"2024-09-18T07:49:41.169753+0000","last_fullsized":"2024-09-18T07:49:41.169753+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:46:14.047907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:14.047907+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":405,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169286+0000","last_change":"2024-09-18T07:48:32.255423+0000","last_active":"2024-09-18T07:49:41.169286+0000","last_peered":"2024-09-18T07:49:41.169286+0000","last_clean":"2024-09-18T07:49:41.169286+0000","last_became_active":"2024-09-18T07:48:32.255130+0000","last_became_peered":"2024-09-18T07:48:32.255130+0000","last_unstale":"2024-09-18T07:49:41.169286+0000","last_undegraded":"2024-09-18T07:49:41.169286+0000","last_fullsized":"2024-09-18T07:49:41.169286+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:42.225418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.225418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"461'13","reported_seq":1258,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171006+0000","last_change":"2024-09-18T07:47:15.865446+0000","last_active":"2024-09-18T07:49:41.171006+0000","last_peered":"2024-09-18T07:49:41.171006+0000","last_clean":"2024-09-18T07:49:41.171006+0000","last_became_active":"2024-09-18T07:47:11.203256+0000","last_became_peered":"2024-09-18T07:47:11.203256+0000","last_unstale":"2024-09-18T07:49:41.171006+0000","last_undegraded":"2024-09-18T07:49:41.171006+0000","last_fullsized":"2024-09-18T07:49:41.171006+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'13","last_scrub_stamp":"2024-09-18T07:47:15.865373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:15.865373+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":896,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170314+0000","last_change":"2024-09-18T07:49:32.690291+0000","last_active":"2024-09-18T07:49:41.170314+0000","last_peered":"2024-09-18T07:49:41.170314+0000","last_clean":"2024-09-18T07:49:41.170314+0000","last_became_active":"2024-09-18T07:49:32.689562+0000","last_became_peered":"2024-09-18T07:49:32.689562+0000","last_unstale":"2024-09-18T07:49:41.170314+0000","last_undegraded":"2024-09-18T07:49:41.170314+0000","last_fullsized":"2024-09-18T07:49:41.170314+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:48.207484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:48.207484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"528'10277","reported_seq":10427,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.864933+0000","last_change":"2024-09-18T07:47:10.191967+0000","last_active":"2024-09-18T07:49:41.864933+0000","last_peered":"2024-09-18T07:49:41.864933+0000","last_clean":"2024-09-18T07:49:41.864933+0000","last_became_active":"2024-09-18T07:47:10.190921+0000","last_became_peered":"2024-09-18T07:47:10.190921+0000","last_unstale":"2024-09-18T07:49:41.864933+0000","last_undegraded":"2024-09-18T07:49:41.864933+0000","last_fullsized":"2024-09-18T07:49:41.864933+0000","mapping_epoch":459,"log_start":"493'5845","ondisk_log_start":"493'5845","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4432,"log_dups_size":0,"ondisk_log_size":4432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248709120,"num_objects":3795,"num_object_clones":0,"num_object_copies":11385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10277,"num_write_kb":450304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"323'15","reported_seq":1065,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.213454+0000","last_change":"2024-09-18T07:48:22.197957+0000","last_active":"2024-09-18T07:49:38.213454+0000","last_peered":"2024-09-18T07:49:38.213454+0000","last_clean":"2024-09-18T07:49:38.213454+0000","last_became_active":"2024-09-18T07:47:20.138834+0000","last_became_peered":"2024-09-18T07:47:20.138834+0000","last_unstale":"2024-09-18T07:49:38.213454+0000","last_undegraded":"2024-09-18T07:49:38.213454+0000","last_fullsized":"2024-09-18T07:49:38.213454+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:48:22.197888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:22.197888+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:43.300890+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":915,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243831+0000","last_change":"2024-09-18T07:48:32.262270+0000","last_active":"2024-09-18T07:49:38.243831+0000","last_peered":"2024-09-18T07:49:38.243831+0000","last_clean":"2024-09-18T07:49:38.243831+0000","last_became_active":"2024-09-18T07:48:32.261507+0000","last_became_peered":"2024-09-18T07:48:32.261507+0000","last_unstale":"2024-09-18T07:49:38.243831+0000","last_undegraded":"2024-09-18T07:49:38.243831+0000","last_fullsized":"2024-09-18T07:49:38.243831+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:51.021530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:51.021530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:23.522646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1004,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169210+0000","last_change":"2024-09-18T07:48:32.265137+0000","last_active":"2024-09-18T07:49:41.169210+0000","last_peered":"2024-09-18T07:49:41.169210+0000","last_clean":"2024-09-18T07:49:41.169210+0000","last_became_active":"2024-09-18T07:48:32.261737+0000","last_became_peered":"2024-09-18T07:48:32.261737+0000","last_unstale":"2024-09-18T07:49:41.169210+0000","last_undegraded":"2024-09-18T07:49:41.169210+0000","last_fullsized":"2024-09-18T07:49:41.169210+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:22.799585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:22.799585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":425,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171519+0000","last_change":"2024-09-18T07:46:29.323007+0000","last_active":"2024-09-18T07:49:41.171519+0000","last_peered":"2024-09-18T07:49:41.171519+0000","last_clean":"2024-09-18T07:49:41.171519+0000","last_became_active":"2024-09-18T07:46:29.322779+0000","last_became_peered":"2024-09-18T07:46:29.322779+0000","last_unstale":"2024-09-18T07:49:41.171519+0000","last_undegraded":"2024-09-18T07:49:41.171519+0000","last_fullsized":"2024-09-18T07:49:41.171519+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:34.257746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:34.257746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"528'10160","reported_seq":10309,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.863439+0000","last_change":"2024-09-18T07:47:11.201597+0000","last_active":"2024-09-18T07:49:41.863439+0000","last_peered":"2024-09-18T07:49:41.863439+0000","last_clean":"2024-09-18T07:49:41.863439+0000","last_became_active":"2024-09-18T07:47:11.201275+0000","last_became_peered":"2024-09-18T07:47:11.201275+0000","last_unstale":"2024-09-18T07:49:41.863439+0000","last_undegraded":"2024-09-18T07:49:41.863439+0000","last_fullsized":"2024-09-18T07:49:41.863439+0000","mapping_epoch":460,"log_start":"493'5732","ondisk_log_start":"493'5732","created":459,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242876416,"num_objects":3706,"num_object_clones":0,"num_object_copies":11118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10160,"num_write_kb":443712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"323'35","reported_seq":721,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.203925+0000","last_change":"2024-09-18T07:47:52.386750+0000","last_active":"2024-09-18T07:49:37.203925+0000","last_peered":"2024-09-18T07:49:37.203925+0000","last_clean":"2024-09-18T07:49:37.203925+0000","last_became_active":"2024-09-18T07:47:20.141942+0000","last_became_peered":"2024-09-18T07:47:20.141942+0000","last_unstale":"2024-09-18T07:49:37.203925+0000","last_undegraded":"2024-09-18T07:49:37.203925+0000","last_fullsized":"2024-09-18T07:49:37.203925+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'35","last_scrub_stamp":"2024-09-18T07:47:52.386688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.386688+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":781,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172403+0000","last_change":"2024-09-18T07:47:17.794908+0000","last_active":"2024-09-18T07:49:41.172403+0000","last_peered":"2024-09-18T07:49:41.172403+0000","last_clean":"2024-09-18T07:49:41.172403+0000","last_became_active":"2024-09-18T07:47:11.194873+0000","last_became_peered":"2024-09-18T07:47:11.194873+0000","last_unstale":"2024-09-18T07:49:41.172403+0000","last_undegraded":"2024-09-18T07:49:41.172403+0000","last_fullsized":"2024-09-18T07:49:41.172403+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:17.794850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:17.794850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1019,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170172+0000","last_change":"2024-09-18T07:48:32.257353+0000","last_active":"2024-09-18T07:49:41.170172+0000","last_peered":"2024-09-18T07:49:41.170172+0000","last_clean":"2024-09-18T07:49:41.170172+0000","last_became_active":"2024-09-18T07:48:32.257023+0000","last_became_peered":"2024-09-18T07:48:32.257023+0000","last_unstale":"2024-09-18T07:49:41.170172+0000","last_undegraded":"2024-09-18T07:49:41.170172+0000","last_fullsized":"2024-09-18T07:49:41.170172+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:20.767517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:20.767517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":763,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172071+0000","last_change":"2024-09-18T07:48:32.265587+0000","last_active":"2024-09-18T07:49:41.172071+0000","last_peered":"2024-09-18T07:49:41.172071+0000","last_clean":"2024-09-18T07:49:41.172071+0000","last_became_active":"2024-09-18T07:48:32.265086+0000","last_became_peered":"2024-09-18T07:48:32.265086+0000","last_unstale":"2024-09-18T07:49:41.172071+0000","last_undegraded":"2024-09-18T07:49:41.172071+0000","last_fullsized":"2024-09-18T07:49:41.172071+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:50.079275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:50.079275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"528'10107","reported_seq":11936,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.820479+0000","last_change":"2024-09-18T07:47:53.337653+0000","last_active":"2024-09-18T07:49:41.820479+0000","last_peered":"2024-09-18T07:49:41.820479+0000","last_clean":"2024-09-18T07:49:41.820479+0000","last_became_active":"2024-09-18T07:47:53.331769+0000","last_became_peered":"2024-09-18T07:47:53.331769+0000","last_unstale":"2024-09-18T07:49:41.820479+0000","last_undegraded":"2024-09-18T07:49:41.820479+0000","last_fullsized":"2024-09-18T07:49:41.820479+0000","mapping_epoch":481,"log_start":"489'4926","ondisk_log_start":"489'4926","created":459,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":5181,"log_dups_size":0,"ondisk_log_size":5181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245825536,"num_objects":3751,"num_object_clones":0,"num_object_copies":11253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10107,"num_write_kb":443456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":831,"num_bytes_recovered":54460416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"323'18","reported_seq":986,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169146+0000","last_change":"2024-09-18T07:48:32.288842+0000","last_active":"2024-09-18T07:49:41.169146+0000","last_peered":"2024-09-18T07:49:41.169146+0000","last_clean":"2024-09-18T07:49:41.169146+0000","last_became_active":"2024-09-18T07:48:32.262707+0000","last_became_peered":"2024-09-18T07:48:32.262707+0000","last_unstale":"2024-09-18T07:49:41.169146+0000","last_undegraded":"2024-09-18T07:49:41.169146+0000","last_fullsized":"2024-09-18T07:49:41.169146+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:46:25.012172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:25.012172+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":773,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172643+0000","last_change":"2024-09-18T07:47:20.139266+0000","last_active":"2024-09-18T07:49:41.172643+0000","last_peered":"2024-09-18T07:49:41.172643+0000","last_clean":"2024-09-18T07:49:41.172643+0000","last_became_active":"2024-09-18T07:47:20.138910+0000","last_became_peered":"2024-09-18T07:47:20.138910+0000","last_unstale":"2024-09-18T07:49:41.172643+0000","last_undegraded":"2024-09-18T07:49:41.172643+0000","last_fullsized":"2024-09-18T07:49:41.172643+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:51.752578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:51.752578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1029,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169280+0000","last_change":"2024-09-18T07:48:23.757438+0000","last_active":"2024-09-18T07:49:41.169280+0000","last_peered":"2024-09-18T07:49:41.169280+0000","last_clean":"2024-09-18T07:49:41.169280+0000","last_became_active":"2024-09-18T07:47:20.139842+0000","last_became_peered":"2024-09-18T07:47:20.139842+0000","last_unstale":"2024-09-18T07:49:41.169280+0000","last_undegraded":"2024-09-18T07:49:41.169280+0000","last_fullsized":"2024-09-18T07:49:41.169280+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:23.757372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:23.757372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":934,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171836+0000","last_change":"2024-09-18T07:48:32.255818+0000","last_active":"2024-09-18T07:49:41.171836+0000","last_peered":"2024-09-18T07:49:41.171836+0000","last_clean":"2024-09-18T07:49:41.171836+0000","last_became_active":"2024-09-18T07:48:32.253112+0000","last_became_peered":"2024-09-18T07:48:32.253112+0000","last_unstale":"2024-09-18T07:49:41.171836+0000","last_undegraded":"2024-09-18T07:49:41.171836+0000","last_fullsized":"2024-09-18T07:49:41.171836+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:40.439672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:40.439672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"526'10133","reported_seq":12079,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:39.220359+0000","last_change":"2024-09-18T07:48:53.603702+0000","last_active":"2024-09-18T07:49:39.220359+0000","last_peered":"2024-09-18T07:49:39.220359+0000","last_clean":"2024-09-18T07:49:39.220359+0000","last_became_active":"2024-09-18T07:47:34.181063+0000","last_became_peered":"2024-09-18T07:47:34.181063+0000","last_unstale":"2024-09-18T07:49:39.220359+0000","last_undegraded":"2024-09-18T07:49:39.220359+0000","last_fullsized":"2024-09-18T07:49:39.220359+0000","mapping_epoch":471,"log_start":"493'5650","ondisk_log_start":"493'5650","created":459,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'7617","last_scrub_stamp":"2024-09-18T07:48:53.603594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:48:53.603594+0000","objects_scrubbed":6521,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:06.768809+0000","scrub_duration":1906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259063808,"num_objects":3953,"num_object_clones":0,"num_object_copies":11859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10133,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":858,"num_bytes_recovered":56229888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"323'27","reported_seq":973,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.242717+0000","last_change":"2024-09-18T07:48:03.057334+0000","last_active":"2024-09-18T07:49:38.242717+0000","last_peered":"2024-09-18T07:49:38.242717+0000","last_clean":"2024-09-18T07:49:38.242717+0000","last_became_active":"2024-09-18T07:47:11.196288+0000","last_became_peered":"2024-09-18T07:47:11.196288+0000","last_unstale":"2024-09-18T07:49:38.242717+0000","last_undegraded":"2024-09-18T07:49:38.242717+0000","last_fullsized":"2024-09-18T07:49:38.242717+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:48:03.057182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:03.057182+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":781,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172683+0000","last_change":"2024-09-18T07:48:34.003445+0000","last_active":"2024-09-18T07:49:41.172683+0000","last_peered":"2024-09-18T07:49:41.172683+0000","last_clean":"2024-09-18T07:49:41.172683+0000","last_became_active":"2024-09-18T07:44:43.778284+0000","last_became_peered":"2024-09-18T07:44:43.778284+0000","last_unstale":"2024-09-18T07:49:41.172683+0000","last_undegraded":"2024-09-18T07:49:41.172683+0000","last_fullsized":"2024-09-18T07:49:41.172683+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:48:34.003134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:34.003134+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:55.154854+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"398'14","reported_seq":972,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171688+0000","last_change":"2024-09-18T07:46:21.582783+0000","last_active":"2024-09-18T07:49:41.171688+0000","last_peered":"2024-09-18T07:49:41.171688+0000","last_clean":"2024-09-18T07:49:41.171688+0000","last_became_active":"2024-09-18T07:44:43.782355+0000","last_became_peered":"2024-09-18T07:44:43.782355+0000","last_unstale":"2024-09-18T07:49:41.171688+0000","last_undegraded":"2024-09-18T07:49:41.171688+0000","last_fullsized":"2024-09-18T07:49:41.171688+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:46:21.582737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:21.582737+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":840,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.242957+0000","last_change":"2024-09-18T07:47:42.111034+0000","last_active":"2024-09-18T07:49:38.242957+0000","last_peered":"2024-09-18T07:49:38.242957+0000","last_clean":"2024-09-18T07:49:38.242957+0000","last_became_active":"2024-09-18T07:47:11.197583+0000","last_became_peered":"2024-09-18T07:47:11.197583+0000","last_unstale":"2024-09-18T07:49:38.242957+0000","last_undegraded":"2024-09-18T07:49:38.242957+0000","last_fullsized":"2024-09-18T07:49:38.242957+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:42.110980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.110980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:31.565452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":613,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.203234+0000","last_change":"2024-09-18T07:47:11.188646+0000","last_active":"2024-09-18T07:49:37.203234+0000","last_peered":"2024-09-18T07:49:37.203234+0000","last_clean":"2024-09-18T07:49:37.203234+0000","last_became_active":"2024-09-18T07:47:11.188332+0000","last_became_peered":"2024-09-18T07:47:11.188332+0000","last_unstale":"2024-09-18T07:49:37.203234+0000","last_undegraded":"2024-09-18T07:49:37.203234+0000","last_fullsized":"2024-09-18T07:49:37.203234+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:08.181420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:08.181420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"323'12","reported_seq":696,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172490+0000","last_change":"2024-09-18T07:48:10.211661+0000","last_active":"2024-09-18T07:49:41.172490+0000","last_peered":"2024-09-18T07:49:41.172490+0000","last_clean":"2024-09-18T07:49:41.172490+0000","last_became_active":"2024-09-18T07:48:10.160725+0000","last_became_peered":"2024-09-18T07:48:10.160725+0000","last_unstale":"2024-09-18T07:49:41.172490+0000","last_undegraded":"2024-09-18T07:49:41.172490+0000","last_fullsized":"2024-09-18T07:49:41.172490+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'12","last_scrub_stamp":"2024-09-18T07:48:03.699388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:03.699388+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":918,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243634+0000","last_change":"2024-09-18T07:48:32.261425+0000","last_active":"2024-09-18T07:49:38.243634+0000","last_peered":"2024-09-18T07:49:38.243634+0000","last_clean":"2024-09-18T07:49:38.243634+0000","last_became_active":"2024-09-18T07:48:32.261094+0000","last_became_peered":"2024-09-18T07:48:32.261094+0000","last_unstale":"2024-09-18T07:49:38.243634+0000","last_undegraded":"2024-09-18T07:49:38.243634+0000","last_fullsized":"2024-09-18T07:49:38.243634+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:53.089903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.089903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:32.526030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":773,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172184+0000","last_change":"2024-09-18T07:48:56.269806+0000","last_active":"2024-09-18T07:49:41.172184+0000","last_peered":"2024-09-18T07:49:41.172184+0000","last_clean":"2024-09-18T07:49:41.172184+0000","last_became_active":"2024-09-18T07:44:43.782348+0000","last_became_peered":"2024-09-18T07:44:43.782348+0000","last_unstale":"2024-09-18T07:49:41.172184+0000","last_undegraded":"2024-09-18T07:49:41.172184+0000","last_fullsized":"2024-09-18T07:49:41.172184+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:56.269745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:56.269745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:03.146403+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"323'15","reported_seq":668,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172320+0000","last_change":"2024-09-18T07:48:32.298683+0000","last_active":"2024-09-18T07:49:41.172320+0000","last_peered":"2024-09-18T07:49:41.172320+0000","last_clean":"2024-09-18T07:49:41.172320+0000","last_became_active":"2024-09-18T07:48:32.259977+0000","last_became_peered":"2024-09-18T07:48:32.259977+0000","last_unstale":"2024-09-18T07:49:41.172320+0000","last_undegraded":"2024-09-18T07:49:41.172320+0000","last_fullsized":"2024-09-18T07:49:41.172320+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:48:27.802966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:27.802966+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:53.895081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":781,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171609+0000","last_change":"2024-09-18T07:45:51.559901+0000","last_active":"2024-09-18T07:49:41.171609+0000","last_peered":"2024-09-18T07:49:41.171609+0000","last_clean":"2024-09-18T07:49:41.171609+0000","last_became_active":"2024-09-18T07:44:43.782930+0000","last_became_peered":"2024-09-18T07:44:43.782930+0000","last_unstale":"2024-09-18T07:49:41.171609+0000","last_undegraded":"2024-09-18T07:49:41.171609+0000","last_fullsized":"2024-09-18T07:49:41.171609+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:51.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:45:51.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":945,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172403+0000","last_change":"2024-09-18T07:48:32.258194+0000","last_active":"2024-09-18T07:49:41.172403+0000","last_peered":"2024-09-18T07:49:41.172403+0000","last_clean":"2024-09-18T07:49:41.172403+0000","last_became_active":"2024-09-18T07:48:32.255775+0000","last_became_peered":"2024-09-18T07:48:32.255775+0000","last_unstale":"2024-09-18T07:49:41.172403+0000","last_undegraded":"2024-09-18T07:49:41.172403+0000","last_fullsized":"2024-09-18T07:49:41.172403+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:30.750167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:30.750167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":990,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170871+0000","last_change":"2024-09-18T07:48:03.866565+0000","last_active":"2024-09-18T07:49:41.170871+0000","last_peered":"2024-09-18T07:49:41.170871+0000","last_clean":"2024-09-18T07:49:41.170871+0000","last_became_active":"2024-09-18T07:44:43.790582+0000","last_became_peered":"2024-09-18T07:44:43.790582+0000","last_unstale":"2024-09-18T07:49:41.170871+0000","last_undegraded":"2024-09-18T07:49:41.170871+0000","last_fullsized":"2024-09-18T07:49:41.170871+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:03.866499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:03.866499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"323'15","reported_seq":953,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243200+0000","last_change":"2024-09-18T07:48:32.286075+0000","last_active":"2024-09-18T07:49:38.243200+0000","last_peered":"2024-09-18T07:49:38.243200+0000","last_clean":"2024-09-18T07:49:38.243200+0000","last_became_active":"2024-09-18T07:48:32.267360+0000","last_became_peered":"2024-09-18T07:48:32.267360+0000","last_unstale":"2024-09-18T07:49:38.243200+0000","last_undegraded":"2024-09-18T07:49:38.243200+0000","last_fullsized":"2024-09-18T07:49:38.243200+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:48:25.381861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:25.381861+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":452,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171629+0000","last_change":"2024-09-18T07:48:32.254574+0000","last_active":"2024-09-18T07:49:41.171629+0000","last_peered":"2024-09-18T07:49:41.171629+0000","last_clean":"2024-09-18T07:49:41.171629+0000","last_became_active":"2024-09-18T07:48:32.254219+0000","last_became_peered":"2024-09-18T07:48:32.254219+0000","last_unstale":"2024-09-18T07:49:41.171629+0000","last_undegraded":"2024-09-18T07:49:41.171629+0000","last_fullsized":"2024-09-18T07:49:41.171629+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:52.023913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.023913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"500'11","reported_seq":381,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171867+0000","last_change":"2024-09-18T07:48:32.311073+0000","last_active":"2024-09-18T07:49:41.171867+0000","last_peered":"2024-09-18T07:49:41.171867+0000","last_clean":"2024-09-18T07:49:41.171867+0000","last_became_active":"2024-09-18T07:48:32.264561+0000","last_became_peered":"2024-09-18T07:48:32.264561+0000","last_unstale":"2024-09-18T07:49:41.171867+0000","last_undegraded":"2024-09-18T07:49:41.171867+0000","last_fullsized":"2024-09-18T07:49:41.171867+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:46:08.086437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:08.086437+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":878,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169896+0000","last_change":"2024-09-18T07:47:11.193555+0000","last_active":"2024-09-18T07:49:41.169896+0000","last_peered":"2024-09-18T07:49:41.169896+0000","last_clean":"2024-09-18T07:49:41.169896+0000","last_became_active":"2024-09-18T07:47:11.193273+0000","last_became_peered":"2024-09-18T07:47:11.193273+0000","last_unstale":"2024-09-18T07:49:41.169896+0000","last_undegraded":"2024-09-18T07:49:41.169896+0000","last_fullsized":"2024-09-18T07:49:41.169896+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:48.955463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.955463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:23.715894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"323'20","reported_seq":852,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171859+0000","last_change":"2024-09-18T07:49:32.702824+0000","last_active":"2024-09-18T07:49:41.171859+0000","last_peered":"2024-09-18T07:49:41.171859+0000","last_clean":"2024-09-18T07:49:41.171859+0000","last_became_active":"2024-09-18T07:49:32.690835+0000","last_became_peered":"2024-09-18T07:49:32.690835+0000","last_unstale":"2024-09-18T07:49:41.171859+0000","last_undegraded":"2024-09-18T07:49:41.171859+0000","last_fullsized":"2024-09-18T07:49:41.171859+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:46:14.650883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:14.650883+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":775,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172142+0000","last_change":"2024-09-18T07:49:00.272282+0000","last_active":"2024-09-18T07:49:41.172142+0000","last_peered":"2024-09-18T07:49:41.172142+0000","last_clean":"2024-09-18T07:49:41.172142+0000","last_became_active":"2024-09-18T07:44:43.779305+0000","last_became_peered":"2024-09-18T07:44:43.779305+0000","last_unstale":"2024-09-18T07:49:41.172142+0000","last_undegraded":"2024-09-18T07:49:41.172142+0000","last_fullsized":"2024-09-18T07:49:41.172142+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:00.272210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:00.272210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:16.088683+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"467'14","reported_seq":1062,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243365+0000","last_change":"2024-09-18T07:47:20.169193+0000","last_active":"2024-09-18T07:49:38.243365+0000","last_peered":"2024-09-18T07:49:38.243365+0000","last_clean":"2024-09-18T07:49:38.243365+0000","last_became_active":"2024-09-18T07:47:20.141913+0000","last_became_peered":"2024-09-18T07:47:20.141913+0000","last_unstale":"2024-09-18T07:49:38.243365+0000","last_undegraded":"2024-09-18T07:49:38.243365+0000","last_fullsized":"2024-09-18T07:49:38.243365+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:40:40.352040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.352040+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":440,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172416+0000","last_change":"2024-09-18T07:48:32.259527+0000","last_active":"2024-09-18T07:49:41.172416+0000","last_peered":"2024-09-18T07:49:41.172416+0000","last_clean":"2024-09-18T07:49:41.172416+0000","last_became_active":"2024-09-18T07:48:32.259107+0000","last_became_peered":"2024-09-18T07:48:32.259107+0000","last_unstale":"2024-09-18T07:49:41.172416+0000","last_undegraded":"2024-09-18T07:49:41.172416+0000","last_fullsized":"2024-09-18T07:49:41.172416+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:45.971311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:45.971311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"323'19","reported_seq":1021,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169643+0000","last_change":"2024-09-18T07:47:20.196648+0000","last_active":"2024-09-18T07:49:41.169643+0000","last_peered":"2024-09-18T07:49:41.169643+0000","last_clean":"2024-09-18T07:49:41.169643+0000","last_became_active":"2024-09-18T07:47:20.140070+0000","last_became_peered":"2024-09-18T07:47:20.140070+0000","last_unstale":"2024-09-18T07:49:41.169643+0000","last_undegraded":"2024-09-18T07:49:41.169643+0000","last_fullsized":"2024-09-18T07:49:41.169643+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:41:54.022421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.022421+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:22.741685+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":935,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169858+0000","last_change":"2024-09-18T07:46:06.964899+0000","last_active":"2024-09-18T07:49:41.169858+0000","last_peered":"2024-09-18T07:49:41.169858+0000","last_clean":"2024-09-18T07:49:41.169858+0000","last_became_active":"2024-09-18T07:44:43.780305+0000","last_became_peered":"2024-09-18T07:44:43.780305+0000","last_unstale":"2024-09-18T07:49:41.169858+0000","last_undegraded":"2024-09-18T07:49:41.169858+0000","last_fullsized":"2024-09-18T07:49:41.169858+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:06.964847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:06.964847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":603,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.242846+0000","last_change":"2024-09-18T07:48:10.163971+0000","last_active":"2024-09-18T07:49:38.242846+0000","last_peered":"2024-09-18T07:49:38.242846+0000","last_clean":"2024-09-18T07:49:38.242846+0000","last_became_active":"2024-09-18T07:48:10.163110+0000","last_became_peered":"2024-09-18T07:48:10.163110+0000","last_unstale":"2024-09-18T07:49:38.242846+0000","last_undegraded":"2024-09-18T07:49:38.242846+0000","last_fullsized":"2024-09-18T07:49:38.242846+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:59.022075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:59.022075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":502,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.214075+0000","last_change":"2024-09-18T07:48:25.249174+0000","last_active":"2024-09-18T07:49:38.214075+0000","last_peered":"2024-09-18T07:49:38.214075+0000","last_clean":"2024-09-18T07:49:38.214075+0000","last_became_active":"2024-09-18T07:48:10.154399+0000","last_became_peered":"2024-09-18T07:48:10.154399+0000","last_unstale":"2024-09-18T07:49:38.214075+0000","last_undegraded":"2024-09-18T07:49:38.214075+0000","last_fullsized":"2024-09-18T07:49:38.214075+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:25.249102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:25.249102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:50.673733+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"323'15","reported_seq":669,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.202760+0000","last_change":"2024-09-18T07:49:32.696134+0000","last_active":"2024-09-18T07:49:37.202760+0000","last_peered":"2024-09-18T07:49:37.202760+0000","last_clean":"2024-09-18T07:49:37.202760+0000","last_became_active":"2024-09-18T07:49:32.687721+0000","last_became_peered":"2024-09-18T07:49:32.687721+0000","last_unstale":"2024-09-18T07:49:37.202760+0000","last_undegraded":"2024-09-18T07:49:37.202760+0000","last_fullsized":"2024-09-18T07:49:37.202760+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:47:49.411931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:49.411931+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":774,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172054+0000","last_change":"2024-09-18T07:48:33.013859+0000","last_active":"2024-09-18T07:49:41.172054+0000","last_peered":"2024-09-18T07:49:41.172054+0000","last_clean":"2024-09-18T07:49:41.172054+0000","last_became_active":"2024-09-18T07:48:32.272079+0000","last_became_peered":"2024-09-18T07:48:32.272079+0000","last_unstale":"2024-09-18T07:49:41.172054+0000","last_undegraded":"2024-09-18T07:49:41.172054+0000","last_fullsized":"2024-09-18T07:49:41.172054+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:33.013808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:33.013808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:02.298988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":494,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.213344+0000","last_change":"2024-09-18T07:48:32.259701+0000","last_active":"2024-09-18T07:49:38.213344+0000","last_peered":"2024-09-18T07:49:38.213344+0000","last_clean":"2024-09-18T07:49:38.213344+0000","last_became_active":"2024-09-18T07:48:32.258708+0000","last_became_peered":"2024-09-18T07:48:32.258708+0000","last_unstale":"2024-09-18T07:49:38.213344+0000","last_undegraded":"2024-09-18T07:49:38.213344+0000","last_fullsized":"2024-09-18T07:49:38.213344+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:13.911901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:13.911901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:31.499215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1012,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.213240+0000","last_change":"2024-09-18T07:47:44.214937+0000","last_active":"2024-09-18T07:49:38.213240+0000","last_peered":"2024-09-18T07:49:38.213240+0000","last_clean":"2024-09-18T07:49:38.213240+0000","last_became_active":"2024-09-18T07:47:20.135880+0000","last_became_peered":"2024-09-18T07:47:20.135880+0000","last_unstale":"2024-09-18T07:49:38.213240+0000","last_undegraded":"2024-09-18T07:49:38.213240+0000","last_fullsized":"2024-09-18T07:49:38.213240+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:44.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:44.214865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:35.490659+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"323'19","reported_seq":745,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171331+0000","last_change":"2024-09-18T07:48:58.276742+0000","last_active":"2024-09-18T07:49:41.171331+0000","last_peered":"2024-09-18T07:49:41.171331+0000","last_clean":"2024-09-18T07:49:41.171331+0000","last_became_active":"2024-09-18T07:48:32.259858+0000","last_became_peered":"2024-09-18T07:48:32.259858+0000","last_unstale":"2024-09-18T07:49:41.171331+0000","last_undegraded":"2024-09-18T07:49:41.171331+0000","last_fullsized":"2024-09-18T07:49:41.171331+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:48:58.276659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:58.276659+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:10.277979+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":916,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243461+0000","last_change":"2024-09-18T07:47:52.070685+0000","last_active":"2024-09-18T07:49:38.243461+0000","last_peered":"2024-09-18T07:49:38.243461+0000","last_clean":"2024-09-18T07:49:38.243461+0000","last_became_active":"2024-09-18T07:47:20.140495+0000","last_became_peered":"2024-09-18T07:47:20.140495+0000","last_unstale":"2024-09-18T07:49:38.243461+0000","last_undegraded":"2024-09-18T07:49:38.243461+0000","last_fullsized":"2024-09-18T07:49:38.243461+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:52.070602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.070602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":781,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172139+0000","last_change":"2024-09-18T07:46:13.660302+0000","last_active":"2024-09-18T07:49:41.172139+0000","last_peered":"2024-09-18T07:49:41.172139+0000","last_clean":"2024-09-18T07:49:41.172139+0000","last_became_active":"2024-09-18T07:44:43.778318+0000","last_became_peered":"2024-09-18T07:44:43.778318+0000","last_unstale":"2024-09-18T07:49:41.172139+0000","last_undegraded":"2024-09-18T07:49:41.172139+0000","last_fullsized":"2024-09-18T07:49:41.172139+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:13.660254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:13.660254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1000,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.213518+0000","last_change":"2024-09-18T07:49:24.443767+0000","last_active":"2024-09-18T07:49:38.213518+0000","last_peered":"2024-09-18T07:49:38.213518+0000","last_clean":"2024-09-18T07:49:38.213518+0000","last_became_active":"2024-09-18T07:49:24.443399+0000","last_became_peered":"2024-09-18T07:49:24.443399+0000","last_unstale":"2024-09-18T07:49:38.213518+0000","last_undegraded":"2024-09-18T07:49:38.213518+0000","last_fullsized":"2024-09-18T07:49:38.213518+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:33.311735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:33.311735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:02.660401+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"323'8","reported_seq":677,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.203009+0000","last_change":"2024-09-18T07:48:32.275394+0000","last_active":"2024-09-18T07:49:37.203009+0000","last_peered":"2024-09-18T07:49:37.203009+0000","last_clean":"2024-09-18T07:49:37.203009+0000","last_became_active":"2024-09-18T07:48:32.256691+0000","last_became_peered":"2024-09-18T07:48:32.256691+0000","last_unstale":"2024-09-18T07:49:37.203009+0000","last_undegraded":"2024-09-18T07:49:37.203009+0000","last_fullsized":"2024-09-18T07:49:37.203009+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'8","last_scrub_stamp":"2024-09-18T07:47:46.405595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:46.405595+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":162,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243597+0000","last_change":"2024-09-18T07:48:32.266044+0000","last_active":"2024-09-18T07:49:38.243597+0000","last_peered":"2024-09-18T07:49:38.243597+0000","last_clean":"2024-09-18T07:49:38.243597+0000","last_became_active":"2024-09-18T07:48:32.265622+0000","last_became_peered":"2024-09-18T07:48:32.265622+0000","last_unstale":"2024-09-18T07:49:38.243597+0000","last_undegraded":"2024-09-18T07:49:38.243597+0000","last_fullsized":"2024-09-18T07:49:38.243597+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:04.066418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:04.066418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":613,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.203822+0000","last_change":"2024-09-18T07:47:55.363130+0000","last_active":"2024-09-18T07:49:37.203822+0000","last_peered":"2024-09-18T07:49:37.203822+0000","last_clean":"2024-09-18T07:49:37.203822+0000","last_became_active":"2024-09-18T07:44:43.792641+0000","last_became_peered":"2024-09-18T07:44:43.792641+0000","last_unstale":"2024-09-18T07:49:37.203822+0000","last_undegraded":"2024-09-18T07:49:37.203822+0000","last_fullsized":"2024-09-18T07:49:37.203822+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:55.363087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:55.363087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":759,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171909+0000","last_change":"2024-09-18T07:44:43.783022+0000","last_active":"2024-09-18T07:49:41.171909+0000","last_peered":"2024-09-18T07:49:41.171909+0000","last_clean":"2024-09-18T07:49:41.171909+0000","last_became_active":"2024-09-18T07:44:43.782378+0000","last_became_peered":"2024-09-18T07:44:43.782378+0000","last_unstale":"2024-09-18T07:49:41.171909+0000","last_undegraded":"2024-09-18T07:49:41.171909+0000","last_fullsized":"2024-09-18T07:49:41.171909+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:38.492269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.492269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:48.284932+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":561,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243533+0000","last_change":"2024-09-18T07:49:17.654443+0000","last_active":"2024-09-18T07:49:38.243533+0000","last_peered":"2024-09-18T07:49:38.243533+0000","last_clean":"2024-09-18T07:49:38.243533+0000","last_became_active":"2024-09-18T07:47:20.138598+0000","last_became_peered":"2024-09-18T07:47:20.138598+0000","last_unstale":"2024-09-18T07:49:38.243533+0000","last_undegraded":"2024-09-18T07:49:38.243533+0000","last_fullsized":"2024-09-18T07:49:38.243533+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:17.654398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:17.654398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:37.045609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":90,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.213698+0000","last_change":"2024-09-18T07:49:32.691139+0000","last_active":"2024-09-18T07:49:38.213698+0000","last_peered":"2024-09-18T07:49:38.213698+0000","last_clean":"2024-09-18T07:49:38.213698+0000","last_became_active":"2024-09-18T07:49:32.685882+0000","last_became_peered":"2024-09-18T07:49:32.685882+0000","last_unstale":"2024-09-18T07:49:38.213698+0000","last_undegraded":"2024-09-18T07:49:38.213698+0000","last_fullsized":"2024-09-18T07:49:38.213698+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:41.464337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:41.464337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"323'19","reported_seq":681,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-18T07:49:37.204072+0000","last_change":"2024-09-18T07:47:47.469967+0000","last_active":"2024-09-18T07:49:37.204072+0000","last_peered":"2024-09-18T07:49:37.204072+0000","last_clean":"2024-09-18T07:49:37.204072+0000","last_became_active":"2024-09-18T07:47:20.142007+0000","last_became_peered":"2024-09-18T07:47:20.142007+0000","last_unstale":"2024-09-18T07:49:37.204072+0000","last_undegraded":"2024-09-18T07:49:37.204072+0000","last_fullsized":"2024-09-18T07:49:37.204072+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:47:47.469902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:47.469902+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":521,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172374+0000","last_change":"2024-09-18T07:48:28.910567+0000","last_active":"2024-09-18T07:49:41.172374+0000","last_peered":"2024-09-18T07:49:41.172374+0000","last_clean":"2024-09-18T07:49:41.172374+0000","last_became_active":"2024-09-18T07:47:20.139668+0000","last_became_peered":"2024-09-18T07:47:20.139668+0000","last_unstale":"2024-09-18T07:49:41.172374+0000","last_undegraded":"2024-09-18T07:49:41.172374+0000","last_fullsized":"2024-09-18T07:49:41.172374+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:28.910419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:28.910419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:54.424567+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4042063896,"num_objects":61678,"num_object_clones":0,"num_object_copies":185034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3912,"num_objects_unfound":0,"num_objects_dirty":61678,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162626,"num_write_kb":7177665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44482,"num_bytes_recovered":2914975744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14110437376,"data_stored":14110425160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72492,"ondisk_log_size":72492,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1096,"num_read_kb":853,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":565,"num_bytes_recovered":12146,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":479232,"data_stored":12674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},"log_size":690,"ondisk_log_size":690,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2652,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":5878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":882,"num_read_kb":877,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201509,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2388540,"kb_used_data":1707620,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":91356612,"statfs":{"total":95995035648,"available":93549170688,"internally_reserved":0,"allocated":1748602880,"data_stored":1748200488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":697235914},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463146,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2992788,"kb_used_data":2275780,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":90752364,"statfs":{"total":95995035648,"available":92930420736,"internally_reserved":0,"allocated":2330398720,"data_stored":2330001557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":734198214},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":460,"seq":1975684956191,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1338256,"kb_used_data":1223872,"kb_used_omap":6,"kb_used_meta":114361,"kb_avail":92406896,"statfs":{"total":95995035648,"available":94624661504,"internally_reserved":0,"allocated":1253244928,"data_stored":1252865826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6877,"internal_metadata":117105955},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":395,"seq":1696512081981,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1735152,"kb_used_data":1225756,"kb_used_omap":4,"kb_used_meta":509371,"kb_avail":92010000,"statfs":{"total":95995035648,"available":94218240000,"internally_reserved":0,"allocated":1255174144,"data_stored":1254772819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4447,"internal_metadata":521596577},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669149947,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411585,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2716712,"kb_used_data":1959440,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":91028440,"statfs":{"total":95995035648,"available":93213122560,"internally_reserved":0,"allocated":2006466560,"data_stored":2006078510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575111,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3197304,"kb_used_data":2363680,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":90547848,"statfs":{"total":95995035648,"available":92720996352,"internally_reserved":0,"allocated":2420408320,"data_stored":2419984058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":853604805},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738636,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2409752,"kb_used_data":1526148,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":91335400,"statfs":{"total":95995035648,"available":93527449600,"internally_reserved":0,"allocated":1562775552,"data_stored":1562386223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904788421},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1558380544,"data_stored":1558380544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2371092480,"data_stored":2371092480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1975783424,"data_stored":1975779352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1902575616,"data_stored":1902575616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1245974528,"data_stored":1245970456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1016463360,"data_stored":1016463360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2312704000,"data_stored":2312699928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1727463424,"data_stored":1727463424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:49:43.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:49:43.543 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:43.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:43.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:43.641 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:43.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:43.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:43.786 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:43.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:43.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:43.888 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:43.923 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:43.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:43.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:44.025 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:44.061 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:44.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:44.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:44.155 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:49:44.155 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:49:44.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:44.197 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:49:44.197 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:49:44.206 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:44.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:44.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:44.234 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:49:44.234 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:49:44.267 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:49:44.267 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:49:44.298 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:49:44.298 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:49:44.307 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:44.328 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:49:44.329 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:49:44.337 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:44.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:44.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:44.338 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:44.359 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:49:44.359 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:49:44.391 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:44 smithi137 ceph-mon[29408]: 6.9 scrub starts 2024-09-18T07:49:44.391 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:44 smithi137 ceph-mon[29408]: 6.9 scrub ok 2024-09-18T07:49:44.391 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:44 smithi137 ceph-mon[29408]: from='client.26761 v1:172.21.15.137:0/2648520421' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:49:44.391 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:44 smithi137 ceph-mon[29408]: 5.7 scrub starts 2024-09-18T07:49:44.392 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:44 smithi137 ceph-mon[29408]: 5.7 scrub ok 2024-09-18T07:49:44.392 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:44 smithi137 ceph-mon[29408]: 6.1 scrub starts 2024-09-18T07:49:44.392 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:44 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1961133321' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:44.392 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:49:44.392 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:49:44.393 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:44 smithi137 ceph-mon[39175]: 6.9 scrub starts 2024-09-18T07:49:44.393 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:44 smithi137 ceph-mon[39175]: 6.9 scrub ok 2024-09-18T07:49:44.393 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:44 smithi137 ceph-mon[39175]: from='client.26761 v1:172.21.15.137:0/2648520421' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:49:44.393 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:44 smithi137 ceph-mon[39175]: 5.7 scrub starts 2024-09-18T07:49:44.393 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:44 smithi137 ceph-mon[39175]: 5.7 scrub ok 2024-09-18T07:49:44.393 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:44 smithi137 ceph-mon[39175]: 6.1 scrub starts 2024-09-18T07:49:44.393 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1961133321' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:44.404 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:44 smithi203 ceph-mon[32237]: 6.9 scrub starts 2024-09-18T07:49:44.404 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:44 smithi203 ceph-mon[32237]: 6.9 scrub ok 2024-09-18T07:49:44.404 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:44 smithi203 ceph-mon[32237]: from='client.26761 v1:172.21.15.137:0/2648520421' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:49:44.404 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:44 smithi203 ceph-mon[32237]: 5.7 scrub starts 2024-09-18T07:49:44.404 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:44 smithi203 ceph-mon[32237]: 5.7 scrub ok 2024-09-18T07:49:44.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:44 smithi203 ceph-mon[32237]: 6.1 scrub starts 2024-09-18T07:49:44.405 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:44 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1961133321' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:44.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:44.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T07:49:44.470 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:44.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:44.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:44.572 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:44.614 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:44.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:44.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:44.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:49:44.716 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:44.758 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:44.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:44.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:44.860 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:44.878 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:49:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:44.901 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:44.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:45.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:45.045 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:45.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:45.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:45.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:45.185 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:45.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:45.208 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:45.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:45.304 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:45 smithi137 ceph-mon[29408]: 6.4 scrub starts 2024-09-18T07:49:45.304 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:45 smithi137 ceph-mon[29408]: 6.4 scrub ok 2024-09-18T07:49:45.304 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:45 smithi137 ceph-mon[29408]: 1.5 scrub starts 2024-09-18T07:49:45.304 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:45 smithi137 ceph-mon[29408]: 1.5 scrub ok 2024-09-18T07:49:45.304 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:45 smithi137 ceph-mon[29408]: pgmap v1007: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 753 op/s; 3659/181404 objects misplaced (2.017%); 9.9 MiB/s, 158 objects/s recovering 2024-09-18T07:49:45.304 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:45 smithi137 ceph-mon[29408]: 6.1 scrub ok 2024-09-18T07:49:45.304 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:45 smithi137 ceph-mon[29408]: 4.1c scrub starts 2024-09-18T07:49:45.304 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:45 smithi137 ceph-mon[29408]: 4.1c scrub ok 2024-09-18T07:49:45.304 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:45 smithi137 ceph-mon[39175]: 6.4 scrub starts 2024-09-18T07:49:45.305 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:45 smithi137 ceph-mon[39175]: 6.4 scrub ok 2024-09-18T07:49:45.305 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:45 smithi137 ceph-mon[39175]: 1.5 scrub starts 2024-09-18T07:49:45.305 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:45 smithi137 ceph-mon[39175]: 1.5 scrub ok 2024-09-18T07:49:45.305 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:45 smithi137 ceph-mon[39175]: pgmap v1007: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 753 op/s; 3659/181404 objects misplaced (2.017%); 9.9 MiB/s, 158 objects/s recovering 2024-09-18T07:49:45.305 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:45 smithi137 ceph-mon[39175]: 6.1 scrub ok 2024-09-18T07:49:45.305 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:45 smithi137 ceph-mon[39175]: 4.1c scrub starts 2024-09-18T07:49:45.305 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:45 smithi137 ceph-mon[39175]: 4.1c scrub ok 2024-09-18T07:49:45.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:45.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:45 smithi203 ceph-mon[32237]: 6.4 scrub starts 2024-09-18T07:49:45.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:45 smithi203 ceph-mon[32237]: 6.4 scrub ok 2024-09-18T07:49:45.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:45 smithi203 ceph-mon[32237]: 1.5 scrub starts 2024-09-18T07:49:45.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:45 smithi203 ceph-mon[32237]: 1.5 scrub ok 2024-09-18T07:49:45.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:45 smithi203 ceph-mon[32237]: pgmap v1007: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 753 op/s; 3659/181404 objects misplaced (2.017%); 9.9 MiB/s, 158 objects/s recovering 2024-09-18T07:49:45.338 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:45 smithi203 ceph-mon[32237]: 6.1 scrub ok 2024-09-18T07:49:45.339 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:45 smithi203 ceph-mon[32237]: 4.1c scrub starts 2024-09-18T07:49:45.339 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:45 smithi203 ceph-mon[32237]: 4.1c scrub ok 2024-09-18T07:49:45.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:45.340 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:45.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:45.441 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:45.461 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:49:45.464 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:49:45.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:45.479 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:45.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:45.494 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:49:45.494 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:49:45.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:45.615 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:45.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:45.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:45.717 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:45.748 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:45.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:45.850 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:45.882 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:45.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:45.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:45.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:46.022 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:46.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:46.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:46.124 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:46.156 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:46.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:46.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:46.242 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T07:49:46.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:46.291 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:46.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:46.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:46.392 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:46.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:46 smithi137 ceph-mon[29408]: 3.6 scrub starts 2024-09-18T07:49:46.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:46 smithi137 ceph-mon[29408]: 3.6 scrub ok 2024-09-18T07:49:46.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:46 smithi137 ceph-mon[39175]: 3.6 scrub starts 2024-09-18T07:49:46.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:46 smithi137 ceph-mon[39175]: 3.6 scrub ok 2024-09-18T07:49:46.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:46.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:46.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:46.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:46 smithi203 ceph-mon[32237]: 3.6 scrub starts 2024-09-18T07:49:46.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:46 smithi203 ceph-mon[32237]: 3.6 scrub ok 2024-09-18T07:49:46.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:46.572 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:46.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:46.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:46.658 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:49:46.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:46.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:46.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:46.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:46.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:46.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:46.856 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:46.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:46.957 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:46.992 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:46.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:46.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:47.094 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:47.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:47.129 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:47.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:47.201 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":1007,"stamp":"2024-09-18T07:49:43.885025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3950320179,"num_objects":60468,"num_object_clones":0,"num_object_copies":181404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3659,"num_objects_unfound":0,"num_objects_dirty":60468,"num_whiteouts":0,"num_read":1986,"num_read_kb":1738,"num_write":164757,"num_write_kb":7177710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45083,"num_bytes_recovered":2915517604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74180,"ondisk_log_size":74180,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":480,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16356696,"kb_used_data":11860476,"kb_used_omap":18,"kb_used_meta":4496045,"kb_avail":639859368,"statfs":{"total":671965249536,"available":655215992832,"internally_reserved":0,"allocated":12145127424,"data_stored":12142286645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19293,"internal_metadata":4603950243},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-296091648,"num_objects":-4518,"num_object_clones":0,"num_object_copies":-13554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1224,"num_objects_unfound":0,"num_objects_dirty":-4518,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4518,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":951,"num_bytes_recovered":62324736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001932"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":1010,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.197691+0000","last_change":"2024-09-18T07:48:28.181443+0000","last_active":"2024-09-18T07:49:41.197691+0000","last_peered":"2024-09-18T07:49:41.197691+0000","last_clean":"2024-09-18T07:49:41.197691+0000","last_became_active":"2024-09-18T07:47:11.200534+0000","last_became_peered":"2024-09-18T07:47:11.200534+0000","last_unstale":"2024-09-18T07:49:41.197691+0000","last_undegraded":"2024-09-18T07:49:41.197691+0000","last_fullsized":"2024-09-18T07:49:41.197691+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:28.181383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:28.181383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:46.295639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"323'15","reported_seq":966,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169095+0000","last_change":"2024-09-18T07:47:20.194767+0000","last_active":"2024-09-18T07:49:41.169095+0000","last_peered":"2024-09-18T07:49:41.169095+0000","last_clean":"2024-09-18T07:49:41.169095+0000","last_became_active":"2024-09-18T07:47:20.136254+0000","last_became_peered":"2024-09-18T07:47:20.136254+0000","last_unstale":"2024-09-18T07:49:41.169095+0000","last_undegraded":"2024-09-18T07:49:41.169095+0000","last_fullsized":"2024-09-18T07:49:41.169095+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:47:09.864550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.864550+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":795,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.242917+0000","last_change":"2024-09-18T07:48:05.093858+0000","last_active":"2024-09-18T07:49:38.242917+0000","last_peered":"2024-09-18T07:49:38.242917+0000","last_clean":"2024-09-18T07:49:38.242917+0000","last_became_active":"2024-09-18T07:47:20.130091+0000","last_became_peered":"2024-09-18T07:47:20.130091+0000","last_unstale":"2024-09-18T07:49:38.242917+0000","last_undegraded":"2024-09-18T07:49:38.242917+0000","last_fullsized":"2024-09-18T07:49:38.242917+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:05.093789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:05.093789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":859,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169516+0000","last_change":"2024-09-18T07:49:32.689133+0000","last_active":"2024-09-18T07:49:41.169516+0000","last_peered":"2024-09-18T07:49:41.169516+0000","last_clean":"2024-09-18T07:49:41.169516+0000","last_became_active":"2024-09-18T07:49:32.688553+0000","last_became_peered":"2024-09-18T07:49:32.688553+0000","last_unstale":"2024-09-18T07:49:41.169516+0000","last_undegraded":"2024-09-18T07:49:41.169516+0000","last_fullsized":"2024-09-18T07:49:41.169516+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:14.884604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:14.884604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":465,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172047+0000","last_change":"2024-09-18T07:48:32.256620+0000","last_active":"2024-09-18T07:49:41.172047+0000","last_peered":"2024-09-18T07:49:41.172047+0000","last_clean":"2024-09-18T07:49:41.172047+0000","last_became_active":"2024-09-18T07:48:32.255219+0000","last_became_peered":"2024-09-18T07:48:32.255219+0000","last_unstale":"2024-09-18T07:49:41.172047+0000","last_undegraded":"2024-09-18T07:49:41.172047+0000","last_fullsized":"2024-09-18T07:49:41.172047+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:11.545358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:11.545358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":451,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.173940+0000","last_change":"2024-09-18T07:48:32.257934+0000","last_active":"2024-09-18T07:49:41.173940+0000","last_peered":"2024-09-18T07:49:41.173940+0000","last_clean":"2024-09-18T07:49:41.173940+0000","last_became_active":"2024-09-18T07:48:32.257676+0000","last_became_peered":"2024-09-18T07:48:32.257676+0000","last_unstale":"2024-09-18T07:49:41.173940+0000","last_undegraded":"2024-09-18T07:49:41.173940+0000","last_fullsized":"2024-09-18T07:49:41.173940+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:57.674688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:57.674688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"323'27","reported_seq":574,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169709+0000","last_change":"2024-09-18T07:48:32.299123+0000","last_active":"2024-09-18T07:49:41.169709+0000","last_peered":"2024-09-18T07:49:41.169709+0000","last_clean":"2024-09-18T07:49:41.169709+0000","last_became_active":"2024-09-18T07:48:32.261006+0000","last_became_peered":"2024-09-18T07:48:32.261006+0000","last_unstale":"2024-09-18T07:49:41.169709+0000","last_undegraded":"2024-09-18T07:49:41.169709+0000","last_fullsized":"2024-09-18T07:49:41.169709+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:46:54.956330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:54.956330+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":915,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169029+0000","last_change":"2024-09-18T07:47:52.060299+0000","last_active":"2024-09-18T07:49:41.169029+0000","last_peered":"2024-09-18T07:49:41.169029+0000","last_clean":"2024-09-18T07:49:41.169029+0000","last_became_active":"2024-09-18T07:47:20.136358+0000","last_became_peered":"2024-09-18T07:47:20.136358+0000","last_unstale":"2024-09-18T07:49:41.169029+0000","last_undegraded":"2024-09-18T07:49:41.169029+0000","last_fullsized":"2024-09-18T07:49:41.169029+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:52.060246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.060246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":910,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243317+0000","last_change":"2024-09-18T07:48:34.470535+0000","last_active":"2024-09-18T07:49:38.243317+0000","last_peered":"2024-09-18T07:49:38.243317+0000","last_clean":"2024-09-18T07:49:38.243317+0000","last_became_active":"2024-09-18T07:47:11.200300+0000","last_became_peered":"2024-09-18T07:47:11.200300+0000","last_unstale":"2024-09-18T07:49:38.243317+0000","last_undegraded":"2024-09-18T07:49:38.243317+0000","last_fullsized":"2024-09-18T07:49:38.243317+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:34.470475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:34.470475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:59.691917+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"323'15","reported_seq":929,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169115+0000","last_change":"2024-09-18T07:48:32.305165+0000","last_active":"2024-09-18T07:49:41.169115+0000","last_peered":"2024-09-18T07:49:41.169115+0000","last_clean":"2024-09-18T07:49:41.169115+0000","last_became_active":"2024-09-18T07:48:32.264535+0000","last_became_peered":"2024-09-18T07:48:32.264535+0000","last_unstale":"2024-09-18T07:49:41.169115+0000","last_undegraded":"2024-09-18T07:49:41.169115+0000","last_fullsized":"2024-09-18T07:49:41.169115+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:47:47.157636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:47.157636+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":847,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243220+0000","last_change":"2024-09-18T07:49:14.687448+0000","last_active":"2024-09-18T07:49:38.243220+0000","last_peered":"2024-09-18T07:49:38.243220+0000","last_clean":"2024-09-18T07:49:38.243220+0000","last_became_active":"2024-09-18T07:47:11.191975+0000","last_became_peered":"2024-09-18T07:47:11.191975+0000","last_unstale":"2024-09-18T07:49:38.243220+0000","last_undegraded":"2024-09-18T07:49:38.243220+0000","last_fullsized":"2024-09-18T07:49:38.243220+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:14.687386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:49:14.687386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:39.954301+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":967,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171449+0000","last_change":"2024-09-18T07:47:55.019436+0000","last_active":"2024-09-18T07:49:41.171449+0000","last_peered":"2024-09-18T07:49:41.171449+0000","last_clean":"2024-09-18T07:49:41.171449+0000","last_became_active":"2024-09-18T07:47:53.329870+0000","last_became_peered":"2024-09-18T07:47:53.329870+0000","last_unstale":"2024-09-18T07:49:41.171449+0000","last_undegraded":"2024-09-18T07:49:41.171449+0000","last_fullsized":"2024-09-18T07:49:41.171449+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:55.019244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:55.019244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":861,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243779+0000","last_change":"2024-09-18T07:47:20.142711+0000","last_active":"2024-09-18T07:49:38.243779+0000","last_peered":"2024-09-18T07:49:38.243779+0000","last_clean":"2024-09-18T07:49:38.243779+0000","last_became_active":"2024-09-18T07:47:20.142404+0000","last_became_peered":"2024-09-18T07:47:20.142404+0000","last_unstale":"2024-09-18T07:49:38.243779+0000","last_undegraded":"2024-09-18T07:49:38.243779+0000","last_fullsized":"2024-09-18T07:49:38.243779+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:57.358853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:41:57.358853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:40.591263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"323'7","reported_seq":1005,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169426+0000","last_change":"2024-09-18T07:49:32.699096+0000","last_active":"2024-09-18T07:49:41.169426+0000","last_peered":"2024-09-18T07:49:41.169426+0000","last_clean":"2024-09-18T07:49:41.169426+0000","last_became_active":"2024-09-18T07:49:32.689928+0000","last_became_peered":"2024-09-18T07:49:32.689928+0000","last_unstale":"2024-09-18T07:49:41.169426+0000","last_undegraded":"2024-09-18T07:49:41.169426+0000","last_fullsized":"2024-09-18T07:49:41.169426+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'7","last_scrub_stamp":"2024-09-18T07:42:46.496916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:46.496916+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:40.792920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":780,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171738+0000","last_change":"2024-09-18T07:47:20.143380+0000","last_active":"2024-09-18T07:49:41.171738+0000","last_peered":"2024-09-18T07:49:41.171738+0000","last_clean":"2024-09-18T07:49:41.171738+0000","last_became_active":"2024-09-18T07:47:20.142891+0000","last_became_peered":"2024-09-18T07:47:20.142891+0000","last_unstale":"2024-09-18T07:49:41.171738+0000","last_undegraded":"2024-09-18T07:49:41.171738+0000","last_fullsized":"2024-09-18T07:49:41.171738+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:53.754622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:53.754622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":904,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243049+0000","last_change":"2024-09-18T07:48:32.255121+0000","last_active":"2024-09-18T07:49:38.243049+0000","last_peered":"2024-09-18T07:49:38.243049+0000","last_clean":"2024-09-18T07:49:38.243049+0000","last_became_active":"2024-09-18T07:48:32.254682+0000","last_became_peered":"2024-09-18T07:48:32.254682+0000","last_unstale":"2024-09-18T07:49:38.243049+0000","last_undegraded":"2024-09-18T07:49:38.243049+0000","last_fullsized":"2024-09-18T07:49:38.243049+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:48.995794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:48.995794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":939,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169968+0000","last_change":"2024-09-18T07:48:31.926895+0000","last_active":"2024-09-18T07:49:41.169968+0000","last_peered":"2024-09-18T07:49:41.169968+0000","last_clean":"2024-09-18T07:49:41.169968+0000","last_became_active":"2024-09-18T07:47:11.187916+0000","last_became_peered":"2024-09-18T07:47:11.187916+0000","last_unstale":"2024-09-18T07:49:41.169968+0000","last_undegraded":"2024-09-18T07:49:41.169968+0000","last_fullsized":"2024-09-18T07:49:41.169968+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:31.926847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:31.926847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:47.604755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1014,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.198156+0000","last_change":"2024-09-18T07:47:49.150861+0000","last_active":"2024-09-18T07:49:41.198156+0000","last_peered":"2024-09-18T07:49:41.198156+0000","last_clean":"2024-09-18T07:49:41.198156+0000","last_became_active":"2024-09-18T07:44:43.773339+0000","last_became_peered":"2024-09-18T07:44:43.773339+0000","last_unstale":"2024-09-18T07:49:41.198156+0000","last_undegraded":"2024-09-18T07:49:41.198156+0000","last_fullsized":"2024-09-18T07:49:41.198156+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:49.150795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:49.150795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:37.524851+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"323'27","reported_seq":1132,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171143+0000","last_change":"2024-09-18T07:47:20.183463+0000","last_active":"2024-09-18T07:49:41.171143+0000","last_peered":"2024-09-18T07:49:41.171143+0000","last_clean":"2024-09-18T07:49:41.171143+0000","last_became_active":"2024-09-18T07:47:20.140525+0000","last_became_peered":"2024-09-18T07:47:20.140525+0000","last_unstale":"2024-09-18T07:49:41.171143+0000","last_undegraded":"2024-09-18T07:49:41.171143+0000","last_fullsized":"2024-09-18T07:49:41.171143+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:45:53.825805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:45:53.825805+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":569,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169770+0000","last_change":"2024-09-18T07:47:20.133202+0000","last_active":"2024-09-18T07:49:41.169770+0000","last_peered":"2024-09-18T07:49:41.169770+0000","last_clean":"2024-09-18T07:49:41.169770+0000","last_became_active":"2024-09-18T07:47:20.131623+0000","last_became_peered":"2024-09-18T07:47:20.131623+0000","last_unstale":"2024-09-18T07:49:41.169770+0000","last_undegraded":"2024-09-18T07:49:41.169770+0000","last_fullsized":"2024-09-18T07:49:41.169770+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:18.008882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:18.008882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":875,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.174177+0000","last_change":"2024-09-18T07:49:32.685656+0000","last_active":"2024-09-18T07:49:41.174177+0000","last_peered":"2024-09-18T07:49:41.174177+0000","last_clean":"2024-09-18T07:49:41.174177+0000","last_became_active":"2024-09-18T07:49:32.685347+0000","last_became_peered":"2024-09-18T07:49:32.685347+0000","last_unstale":"2024-09-18T07:49:41.174177+0000","last_undegraded":"2024-09-18T07:49:41.174177+0000","last_fullsized":"2024-09-18T07:49:41.174177+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:44.397262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:44.397262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":931,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169409+0000","last_change":"2024-09-18T07:47:56.099010+0000","last_active":"2024-09-18T07:49:41.169409+0000","last_peered":"2024-09-18T07:49:41.169409+0000","last_clean":"2024-09-18T07:49:41.169409+0000","last_became_active":"2024-09-18T07:47:20.144620+0000","last_became_peered":"2024-09-18T07:47:20.144620+0000","last_unstale":"2024-09-18T07:49:41.169409+0000","last_undegraded":"2024-09-18T07:49:41.169409+0000","last_fullsized":"2024-09-18T07:49:41.169409+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:56.098927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.098927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"323'18","reported_seq":297,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169747+0000","last_change":"2024-09-18T07:48:32.293504+0000","last_active":"2024-09-18T07:49:41.169747+0000","last_peered":"2024-09-18T07:49:41.169747+0000","last_clean":"2024-09-18T07:49:41.169747+0000","last_became_active":"2024-09-18T07:48:32.266466+0000","last_became_peered":"2024-09-18T07:48:32.266466+0000","last_unstale":"2024-09-18T07:49:41.169747+0000","last_undegraded":"2024-09-18T07:49:41.169747+0000","last_fullsized":"2024-09-18T07:49:41.169747+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:47:53.905251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.905251+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":852,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169695+0000","last_change":"2024-09-18T07:48:32.264906+0000","last_active":"2024-09-18T07:49:41.169695+0000","last_peered":"2024-09-18T07:49:41.169695+0000","last_clean":"2024-09-18T07:49:41.169695+0000","last_became_active":"2024-09-18T07:48:32.264578+0000","last_became_peered":"2024-09-18T07:48:32.264578+0000","last_unstale":"2024-09-18T07:49:41.169695+0000","last_undegraded":"2024-09-18T07:49:41.169695+0000","last_fullsized":"2024-09-18T07:49:41.169695+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.930435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.930435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:20.674047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":778,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172116+0000","last_change":"2024-09-18T07:47:53.034043+0000","last_active":"2024-09-18T07:49:41.172116+0000","last_peered":"2024-09-18T07:49:41.172116+0000","last_clean":"2024-09-18T07:49:41.172116+0000","last_became_active":"2024-09-18T07:47:11.194447+0000","last_became_peered":"2024-09-18T07:47:11.194447+0000","last_unstale":"2024-09-18T07:49:41.172116+0000","last_undegraded":"2024-09-18T07:49:41.172116+0000","last_fullsized":"2024-09-18T07:49:41.172116+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:47:53.033975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.033975+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":82,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169072+0000","last_change":"2024-09-18T07:48:32.274975+0000","last_active":"2024-09-18T07:49:41.169072+0000","last_peered":"2024-09-18T07:49:41.169072+0000","last_clean":"2024-09-18T07:49:41.169072+0000","last_became_active":"2024-09-18T07:48:32.262299+0000","last_became_peered":"2024-09-18T07:48:32.262299+0000","last_unstale":"2024-09-18T07:49:41.169072+0000","last_undegraded":"2024-09-18T07:49:41.169072+0000","last_fullsized":"2024-09-18T07:49:41.169072+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:09.496455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.496455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"323'20","reported_seq":755,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172571+0000","last_change":"2024-09-18T07:48:19.962428+0000","last_active":"2024-09-18T07:49:41.172571+0000","last_peered":"2024-09-18T07:49:41.172571+0000","last_clean":"2024-09-18T07:49:41.172571+0000","last_became_active":"2024-09-18T07:48:10.157273+0000","last_became_peered":"2024-09-18T07:48:10.157273+0000","last_unstale":"2024-09-18T07:49:41.172571+0000","last_undegraded":"2024-09-18T07:49:41.172571+0000","last_fullsized":"2024-09-18T07:49:41.172571+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:48:19.962346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:19.962346+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"398'14","reported_seq":785,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.853017+0000","last_change":"2024-09-18T07:49:41.853017+0000","last_active":"2024-09-18T07:49:41.853017+0000","last_peered":"2024-09-18T07:49:41.853017+0000","last_clean":"2024-09-18T07:49:41.853017+0000","last_became_active":"2024-09-18T07:44:43.786594+0000","last_became_peered":"2024-09-18T07:44:43.786594+0000","last_unstale":"2024-09-18T07:49:41.853017+0000","last_undegraded":"2024-09-18T07:49:41.853017+0000","last_fullsized":"2024-09-18T07:49:41.853017+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:49:41.852939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:49:41.852939+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:46.677099+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"528'10406","reported_seq":21188,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.860304+0000","last_change":"2024-09-18T07:49:41.190449+0000","last_active":"2024-09-18T07:49:41.860304+0000","last_peered":"2024-09-18T07:49:41.860304+0000","last_clean":"2024-09-18T07:49:41.860304+0000","last_became_active":"2024-09-18T07:49:41.174729+0000","last_became_peered":"2024-09-18T07:49:41.174729+0000","last_unstale":"2024-09-18T07:49:41.860304+0000","last_undegraded":"2024-09-18T07:49:41.860304+0000","last_fullsized":"2024-09-18T07:49:41.860304+0000","mapping_epoch":527,"log_start":"495'6036","ondisk_log_start":"495'6036","created":459,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4370,"log_dups_size":0,"ondisk_log_size":4370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254803968,"num_objects":3888,"num_object_clones":0,"num_object_copies":11664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10406,"num_write_kb":457408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5294,"num_bytes_recovered":346947584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":414,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.244099+0000","last_change":"2024-09-18T07:47:13.167529+0000","last_active":"2024-09-18T07:49:38.244099+0000","last_peered":"2024-09-18T07:49:38.244099+0000","last_clean":"2024-09-18T07:49:38.244099+0000","last_became_active":"2024-09-18T07:47:11.203901+0000","last_became_peered":"2024-09-18T07:47:11.203901+0000","last_unstale":"2024-09-18T07:49:38.244099+0000","last_undegraded":"2024-09-18T07:49:38.244099+0000","last_fullsized":"2024-09-18T07:49:38.244099+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:47:13.167470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:47:13.167470+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:46.555145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"323'27","reported_seq":393,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170051+0000","last_change":"2024-09-18T07:48:32.312362+0000","last_active":"2024-09-18T07:49:41.170051+0000","last_peered":"2024-09-18T07:49:41.170051+0000","last_clean":"2024-09-18T07:49:41.170051+0000","last_became_active":"2024-09-18T07:48:32.268036+0000","last_became_peered":"2024-09-18T07:48:32.268036+0000","last_unstale":"2024-09-18T07:49:41.170051+0000","last_undegraded":"2024-09-18T07:49:41.170051+0000","last_fullsized":"2024-09-18T07:49:41.170051+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:42:33.391715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:33.391715+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:25.785407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":962,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171782+0000","last_change":"2024-09-18T07:48:32.250884+0000","last_active":"2024-09-18T07:49:41.171782+0000","last_peered":"2024-09-18T07:49:41.171782+0000","last_clean":"2024-09-18T07:49:41.171782+0000","last_became_active":"2024-09-18T07:48:32.250451+0000","last_became_peered":"2024-09-18T07:48:32.250451+0000","last_unstale":"2024-09-18T07:49:41.171782+0000","last_undegraded":"2024-09-18T07:49:41.171782+0000","last_fullsized":"2024-09-18T07:49:41.171782+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:25.143715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:25.143715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"461'12","reported_seq":1082,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170048+0000","last_change":"2024-09-18T07:48:30.963331+0000","last_active":"2024-09-18T07:49:41.170048+0000","last_peered":"2024-09-18T07:49:41.170048+0000","last_clean":"2024-09-18T07:49:41.170048+0000","last_became_active":"2024-09-18T07:47:11.190670+0000","last_became_peered":"2024-09-18T07:47:11.190670+0000","last_unstale":"2024-09-18T07:49:41.170048+0000","last_undegraded":"2024-09-18T07:49:41.170048+0000","last_fullsized":"2024-09-18T07:49:41.170048+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'12","last_scrub_stamp":"2024-09-18T07:48:30.963258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:30.963258+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:56.466762+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":843,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243549+0000","last_change":"2024-09-18T07:49:32.687377+0000","last_active":"2024-09-18T07:49:38.243549+0000","last_peered":"2024-09-18T07:49:38.243549+0000","last_clean":"2024-09-18T07:49:38.243549+0000","last_became_active":"2024-09-18T07:49:32.687063+0000","last_became_peered":"2024-09-18T07:49:32.687063+0000","last_unstale":"2024-09-18T07:49:38.243549+0000","last_undegraded":"2024-09-18T07:49:38.243549+0000","last_fullsized":"2024-09-18T07:49:38.243549+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:45.670301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:45.670301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:34.512574+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"528'10305","reported_seq":12127,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.861605+0000","last_change":"2024-09-18T07:47:37.208096+0000","last_active":"2024-09-18T07:49:41.861605+0000","last_peered":"2024-09-18T07:49:41.861605+0000","last_clean":"2024-09-18T07:49:41.861605+0000","last_became_active":"2024-09-18T07:47:37.204971+0000","last_became_peered":"2024-09-18T07:47:37.204971+0000","last_unstale":"2024-09-18T07:49:41.861605+0000","last_undegraded":"2024-09-18T07:49:41.861605+0000","last_fullsized":"2024-09-18T07:49:41.861605+0000","mapping_epoch":474,"log_start":"495'5941","ondisk_log_start":"495'5941","created":459,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4364,"log_dups_size":0,"ondisk_log_size":4364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248840192,"num_objects":3797,"num_object_clones":0,"num_object_copies":11391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10305,"num_write_kb":451264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":831,"num_bytes_recovered":54460416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":458,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169833+0000","last_change":"2024-09-18T07:44:43.923223+0000","last_active":"2024-09-18T07:49:41.169833+0000","last_peered":"2024-09-18T07:49:41.169833+0000","last_clean":"2024-09-18T07:49:41.169833+0000","last_became_active":"2024-09-18T07:44:43.786396+0000","last_became_peered":"2024-09-18T07:44:43.786396+0000","last_unstale":"2024-09-18T07:49:41.169833+0000","last_undegraded":"2024-09-18T07:49:41.169833+0000","last_fullsized":"2024-09-18T07:49:41.169833+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:44:43.923179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:44:43.923179+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"323'30","reported_seq":745,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171331+0000","last_change":"2024-09-18T07:48:32.316007+0000","last_active":"2024-09-18T07:49:41.171331+0000","last_peered":"2024-09-18T07:49:41.171331+0000","last_clean":"2024-09-18T07:49:41.171331+0000","last_became_active":"2024-09-18T07:48:32.255887+0000","last_became_peered":"2024-09-18T07:48:32.255887+0000","last_unstale":"2024-09-18T07:49:41.171331+0000","last_undegraded":"2024-09-18T07:49:41.171331+0000","last_fullsized":"2024-09-18T07:49:41.171331+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'30","last_scrub_stamp":"2024-09-18T07:47:56.645991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.645991+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":726,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172605+0000","last_change":"2024-09-18T07:48:32.260193+0000","last_active":"2024-09-18T07:49:41.172605+0000","last_peered":"2024-09-18T07:49:41.172605+0000","last_clean":"2024-09-18T07:49:41.172605+0000","last_became_active":"2024-09-18T07:48:32.259860+0000","last_became_peered":"2024-09-18T07:48:32.259860+0000","last_unstale":"2024-09-18T07:49:41.172605+0000","last_undegraded":"2024-09-18T07:49:41.172605+0000","last_fullsized":"2024-09-18T07:49:41.172605+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:56.077323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.077323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":911,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169030+0000","last_change":"2024-09-18T07:48:32.265565+0000","last_active":"2024-09-18T07:49:41.169030+0000","last_peered":"2024-09-18T07:49:41.169030+0000","last_clean":"2024-09-18T07:49:41.169030+0000","last_became_active":"2024-09-18T07:48:32.265285+0000","last_became_peered":"2024-09-18T07:48:32.265285+0000","last_unstale":"2024-09-18T07:49:41.169030+0000","last_undegraded":"2024-09-18T07:49:41.169030+0000","last_fullsized":"2024-09-18T07:49:41.169030+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:53.056484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.056484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":820,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169208+0000","last_change":"2024-09-18T07:48:32.265789+0000","last_active":"2024-09-18T07:49:41.169208+0000","last_peered":"2024-09-18T07:49:41.169208+0000","last_clean":"2024-09-18T07:49:41.169208+0000","last_became_active":"2024-09-18T07:48:32.264093+0000","last_became_peered":"2024-09-18T07:48:32.264093+0000","last_unstale":"2024-09-18T07:49:41.169208+0000","last_undegraded":"2024-09-18T07:49:41.169208+0000","last_fullsized":"2024-09-18T07:49:41.169208+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:07.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:07.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"528'10304","reported_seq":12172,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.477714+0000","last_change":"2024-09-18T07:47:35.123117+0000","last_active":"2024-09-18T07:49:41.477714+0000","last_peered":"2024-09-18T07:49:41.477714+0000","last_clean":"2024-09-18T07:49:41.477714+0000","last_became_active":"2024-09-18T07:47:35.119437+0000","last_became_peered":"2024-09-18T07:47:35.119437+0000","last_unstale":"2024-09-18T07:49:41.477714+0000","last_undegraded":"2024-09-18T07:49:41.477714+0000","last_fullsized":"2024-09-18T07:49:41.477714+0000","mapping_epoch":472,"log_start":"495'6056","ondisk_log_start":"495'6056","created":459,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4248,"log_dups_size":0,"ondisk_log_size":4248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247595008,"num_objects":3778,"num_object_clones":0,"num_object_copies":11334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10304,"num_write_kb":450624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":55902208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":78,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243705+0000","last_change":"2024-09-18T07:48:32.459297+0000","last_active":"2024-09-18T07:49:38.243705+0000","last_peered":"2024-09-18T07:49:38.243705+0000","last_clean":"2024-09-18T07:49:38.243705+0000","last_became_active":"2024-09-18T07:48:32.271495+0000","last_became_peered":"2024-09-18T07:48:32.271495+0000","last_unstale":"2024-09-18T07:49:38.243705+0000","last_undegraded":"2024-09-18T07:49:38.243705+0000","last_fullsized":"2024-09-18T07:49:38.243705+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:32.459229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:32.459229+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:01.535886+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"323'28","reported_seq":939,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169285+0000","last_change":"2024-09-18T07:48:26.993988+0000","last_active":"2024-09-18T07:49:41.169285+0000","last_peered":"2024-09-18T07:49:41.169285+0000","last_clean":"2024-09-18T07:49:41.169285+0000","last_became_active":"2024-09-18T07:47:11.188002+0000","last_became_peered":"2024-09-18T07:47:11.188002+0000","last_unstale":"2024-09-18T07:49:41.169285+0000","last_undegraded":"2024-09-18T07:49:41.169285+0000","last_fullsized":"2024-09-18T07:49:41.169285+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'28","last_scrub_stamp":"2024-09-18T07:48:26.993871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:26.993871+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:44.242031+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":1016,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.198664+0000","last_change":"2024-09-18T07:47:48.117798+0000","last_active":"2024-09-18T07:49:41.198664+0000","last_peered":"2024-09-18T07:49:41.198664+0000","last_clean":"2024-09-18T07:49:41.198664+0000","last_became_active":"2024-09-18T07:47:20.138745+0000","last_became_peered":"2024-09-18T07:47:20.138745+0000","last_unstale":"2024-09-18T07:49:41.198664+0000","last_undegraded":"2024-09-18T07:49:41.198664+0000","last_fullsized":"2024-09-18T07:49:41.198664+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:48.117629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:48.117629+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"500'13","reported_seq":967,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.168985+0000","last_change":"2024-09-18T07:48:32.276078+0000","last_active":"2024-09-18T07:49:41.168985+0000","last_peered":"2024-09-18T07:49:41.168985+0000","last_clean":"2024-09-18T07:49:41.168985+0000","last_became_active":"2024-09-18T07:48:32.254811+0000","last_became_peered":"2024-09-18T07:48:32.254811+0000","last_unstale":"2024-09-18T07:49:41.168985+0000","last_undegraded":"2024-09-18T07:49:41.168985+0000","last_fullsized":"2024-09-18T07:49:41.168985+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'12","last_scrub_stamp":"2024-09-18T07:48:20.859791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:20.859791+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":771,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171905+0000","last_change":"2024-09-18T07:47:42.984418+0000","last_active":"2024-09-18T07:49:41.171905+0000","last_peered":"2024-09-18T07:49:41.171905+0000","last_clean":"2024-09-18T07:49:41.171905+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:49:41.171905+0000","last_undegraded":"2024-09-18T07:49:41.171905+0000","last_fullsized":"2024-09-18T07:49:41.171905+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:42.984298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.984298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"528'10586","reported_seq":10737,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.756060+0000","last_change":"2024-09-18T07:47:10.178170+0000","last_active":"2024-09-18T07:49:41.756060+0000","last_peered":"2024-09-18T07:49:41.756060+0000","last_clean":"2024-09-18T07:49:41.756060+0000","last_became_active":"2024-09-18T07:47:10.177901+0000","last_became_peered":"2024-09-18T07:47:10.177901+0000","last_unstale":"2024-09-18T07:49:41.756060+0000","last_undegraded":"2024-09-18T07:49:41.756060+0000","last_fullsized":"2024-09-18T07:49:41.756060+0000","mapping_epoch":459,"log_start":"490'5364","ondisk_log_start":"490'5364","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":5222,"log_dups_size":0,"ondisk_log_size":5222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252248088,"num_objects":3850,"num_object_clones":0,"num_object_copies":11550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3850,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10586,"num_write_kb":461889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":720,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.174049+0000","last_change":"2024-09-18T07:48:10.151652+0000","last_active":"2024-09-18T07:49:41.174049+0000","last_peered":"2024-09-18T07:49:41.174049+0000","last_clean":"2024-09-18T07:49:41.174049+0000","last_became_active":"2024-09-18T07:48:10.151343+0000","last_became_peered":"2024-09-18T07:48:10.151343+0000","last_unstale":"2024-09-18T07:49:41.174049+0000","last_undegraded":"2024-09-18T07:49:41.174049+0000","last_fullsized":"2024-09-18T07:49:41.174049+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:02.444272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:02.444272+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"323'20","reported_seq":682,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243007+0000","last_change":"2024-09-18T07:49:32.698546+0000","last_active":"2024-09-18T07:49:38.243007+0000","last_peered":"2024-09-18T07:49:38.243007+0000","last_clean":"2024-09-18T07:49:38.243007+0000","last_became_active":"2024-09-18T07:49:32.688605+0000","last_became_peered":"2024-09-18T07:49:32.688605+0000","last_unstale":"2024-09-18T07:49:38.243007+0000","last_undegraded":"2024-09-18T07:49:38.243007+0000","last_fullsized":"2024-09-18T07:49:38.243007+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:48:33.642815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:33.642815+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:44.608266+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":893,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243107+0000","last_change":"2024-09-18T07:48:32.268542+0000","last_active":"2024-09-18T07:49:38.243107+0000","last_peered":"2024-09-18T07:49:38.243107+0000","last_clean":"2024-09-18T07:49:38.243107+0000","last_became_active":"2024-09-18T07:48:32.267802+0000","last_became_peered":"2024-09-18T07:48:32.267802+0000","last_unstale":"2024-09-18T07:49:38.243107+0000","last_undegraded":"2024-09-18T07:49:38.243107+0000","last_fullsized":"2024-09-18T07:49:38.243107+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:19.210077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:19.210077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"461'13","reported_seq":725,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.173919+0000","last_change":"2024-09-18T07:48:24.606091+0000","last_active":"2024-09-18T07:49:41.173919+0000","last_peered":"2024-09-18T07:49:41.173919+0000","last_clean":"2024-09-18T07:49:41.173919+0000","last_became_active":"2024-09-18T07:47:11.189241+0000","last_became_peered":"2024-09-18T07:47:11.189241+0000","last_unstale":"2024-09-18T07:49:41.173919+0000","last_undegraded":"2024-09-18T07:49:41.173919+0000","last_fullsized":"2024-09-18T07:49:41.173919+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'13","last_scrub_stamp":"2024-09-18T07:48:24.605876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:24.605876+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":71,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169084+0000","last_change":"2024-09-18T07:48:32.258814+0000","last_active":"2024-09-18T07:49:41.169084+0000","last_peered":"2024-09-18T07:49:41.169084+0000","last_clean":"2024-09-18T07:49:41.169084+0000","last_became_active":"2024-09-18T07:48:32.258477+0000","last_became_peered":"2024-09-18T07:48:32.258477+0000","last_unstale":"2024-09-18T07:49:41.169084+0000","last_undegraded":"2024-09-18T07:49:41.169084+0000","last_fullsized":"2024-09-18T07:49:41.169084+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:46.172671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:46.172671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.c","version":"528'10082","reported_seq":11839,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.779351+0000","last_change":"2024-09-18T07:47:31.131323+0000","last_active":"2024-09-18T07:49:41.779351+0000","last_peered":"2024-09-18T07:49:41.779351+0000","last_clean":"2024-09-18T07:49:41.779351+0000","last_became_active":"2024-09-18T07:47:31.128537+0000","last_became_peered":"2024-09-18T07:47:31.128537+0000","last_unstale":"2024-09-18T07:49:41.779351+0000","last_undegraded":"2024-09-18T07:49:41.779351+0000","last_fullsized":"2024-09-18T07:49:41.779351+0000","mapping_epoch":468,"log_start":"489'4815","ondisk_log_start":"489'4815","created":459,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":5267,"log_dups_size":0,"ondisk_log_size":5267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248774656,"num_objects":3796,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10082,"num_write_kb":444096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":799,"num_bytes_recovered":52363264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":728,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.173840+0000","last_change":"2024-09-18T07:48:32.273778+0000","last_active":"2024-09-18T07:49:41.173840+0000","last_peered":"2024-09-18T07:49:41.173840+0000","last_clean":"2024-09-18T07:49:41.173840+0000","last_became_active":"2024-09-18T07:48:32.273227+0000","last_became_peered":"2024-09-18T07:48:32.273227+0000","last_unstale":"2024-09-18T07:49:41.173840+0000","last_undegraded":"2024-09-18T07:49:41.173840+0000","last_fullsized":"2024-09-18T07:49:41.173840+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:47:59.422927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:47:59.422927+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"323'29","reported_seq":809,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243458+0000","last_change":"2024-09-18T07:48:01.062906+0000","last_active":"2024-09-18T07:49:38.243458+0000","last_peered":"2024-09-18T07:49:38.243458+0000","last_clean":"2024-09-18T07:49:38.243458+0000","last_became_active":"2024-09-18T07:47:11.198126+0000","last_became_peered":"2024-09-18T07:47:11.198126+0000","last_unstale":"2024-09-18T07:49:38.243458+0000","last_undegraded":"2024-09-18T07:49:38.243458+0000","last_fullsized":"2024-09-18T07:49:38.243458+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'29","last_scrub_stamp":"2024-09-18T07:48:01.062822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:01.062822+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":708,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171242+0000","last_change":"2024-09-18T07:47:20.140174+0000","last_active":"2024-09-18T07:49:41.171242+0000","last_peered":"2024-09-18T07:49:41.171242+0000","last_clean":"2024-09-18T07:49:41.171242+0000","last_became_active":"2024-09-18T07:47:20.139806+0000","last_became_peered":"2024-09-18T07:47:20.139806+0000","last_unstale":"2024-09-18T07:49:41.171242+0000","last_undegraded":"2024-09-18T07:49:41.171242+0000","last_fullsized":"2024-09-18T07:49:41.171242+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:43.985969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:43.985969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":635,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.173404+0000","last_change":"2024-09-18T07:47:41.319363+0000","last_active":"2024-09-18T07:49:41.173404+0000","last_peered":"2024-09-18T07:49:41.173404+0000","last_clean":"2024-09-18T07:49:41.173404+0000","last_became_active":"2024-09-18T07:47:11.193892+0000","last_became_peered":"2024-09-18T07:47:11.193892+0000","last_unstale":"2024-09-18T07:49:41.173404+0000","last_undegraded":"2024-09-18T07:49:41.173404+0000","last_fullsized":"2024-09-18T07:49:41.173404+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:41.319318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:41.319318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":630,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.174038+0000","last_change":"2024-09-18T07:47:20.138912+0000","last_active":"2024-09-18T07:49:41.174038+0000","last_peered":"2024-09-18T07:49:41.174038+0000","last_clean":"2024-09-18T07:49:41.174038+0000","last_became_active":"2024-09-18T07:47:20.138390+0000","last_became_peered":"2024-09-18T07:47:20.138390+0000","last_unstale":"2024-09-18T07:49:41.174038+0000","last_undegraded":"2024-09-18T07:49:41.174038+0000","last_fullsized":"2024-09-18T07:49:41.174038+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:49.018222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:49.018222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.b","version":"528'10221","reported_seq":24247,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.449274+0000","last_change":"2024-09-18T07:49:19.420062+0000","last_active":"2024-09-18T07:49:41.449274+0000","last_peered":"2024-09-18T07:49:41.449274+0000","last_clean":"2024-09-18T07:49:41.449274+0000","last_became_active":"2024-09-18T07:49:19.393415+0000","last_became_peered":"2024-09-18T07:49:19.393415+0000","last_unstale":"2024-09-18T07:49:41.449274+0000","last_undegraded":"2024-09-18T07:49:41.449274+0000","last_fullsized":"2024-09-18T07:49:41.449274+0000","mapping_epoch":512,"log_start":"496'6014","ondisk_log_start":"496'6014","created":459,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4207,"log_dups_size":0,"ondisk_log_size":4207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242941952,"num_objects":3707,"num_object_clones":0,"num_object_copies":11121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10221,"num_write_kb":445696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6927,"num_bytes_recovered":453967872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":442,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243084+0000","last_change":"2024-09-18T07:48:59.662366+0000","last_active":"2024-09-18T07:49:38.243084+0000","last_peered":"2024-09-18T07:49:38.243084+0000","last_clean":"2024-09-18T07:49:38.243084+0000","last_became_active":"2024-09-18T07:47:11.195706+0000","last_became_peered":"2024-09-18T07:47:11.195706+0000","last_unstale":"2024-09-18T07:49:38.243084+0000","last_undegraded":"2024-09-18T07:49:38.243084+0000","last_fullsized":"2024-09-18T07:49:38.243084+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:59.662286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:59.662286+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:04.207309+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"519'45","reported_seq":1059,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171086+0000","last_change":"2024-09-18T07:49:24.465774+0000","last_active":"2024-09-18T07:49:41.171086+0000","last_peered":"2024-09-18T07:49:41.171086+0000","last_clean":"2024-09-18T07:49:41.171086+0000","last_became_active":"2024-09-18T07:49:24.452427+0000","last_became_peered":"2024-09-18T07:49:24.452427+0000","last_unstale":"2024-09-18T07:49:41.171086+0000","last_undegraded":"2024-09-18T07:49:41.171086+0000","last_fullsized":"2024-09-18T07:49:41.171086+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'41","last_scrub_stamp":"2024-09-18T07:46:12.858570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:12.858570+0000","objects_scrubbed":10,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":910,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169490+0000","last_change":"2024-09-18T07:47:51.085464+0000","last_active":"2024-09-18T07:49:41.169490+0000","last_peered":"2024-09-18T07:49:41.169490+0000","last_clean":"2024-09-18T07:49:41.169490+0000","last_became_active":"2024-09-18T07:44:43.780114+0000","last_became_peered":"2024-09-18T07:44:43.780114+0000","last_unstale":"2024-09-18T07:49:41.169490+0000","last_undegraded":"2024-09-18T07:49:41.169490+0000","last_fullsized":"2024-09-18T07:49:41.169490+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:51.085400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:51.085400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":878,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243115+0000","last_change":"2024-09-18T07:48:06.127800+0000","last_active":"2024-09-18T07:49:38.243115+0000","last_peered":"2024-09-18T07:49:38.243115+0000","last_clean":"2024-09-18T07:49:38.243115+0000","last_became_active":"2024-09-18T07:47:20.144312+0000","last_became_peered":"2024-09-18T07:47:20.144312+0000","last_unstale":"2024-09-18T07:49:38.243115+0000","last_undegraded":"2024-09-18T07:49:38.243115+0000","last_fullsized":"2024-09-18T07:49:38.243115+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:48:06.127745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:48:06.127745+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":991,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.197369+0000","last_change":"2024-09-18T07:47:20.131135+0000","last_active":"2024-09-18T07:49:41.197369+0000","last_peered":"2024-09-18T07:49:41.197369+0000","last_clean":"2024-09-18T07:49:41.197369+0000","last_became_active":"2024-09-18T07:47:20.130809+0000","last_became_peered":"2024-09-18T07:47:20.130809+0000","last_unstale":"2024-09-18T07:49:41.197369+0000","last_undegraded":"2024-09-18T07:49:41.197369+0000","last_fullsized":"2024-09-18T07:49:41.197369+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:46.629196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:46.629196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:27.458621+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":790,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170264+0000","last_change":"2024-09-18T07:49:24.445613+0000","last_active":"2024-09-18T07:49:41.170264+0000","last_peered":"2024-09-18T07:49:41.170264+0000","last_clean":"2024-09-18T07:49:41.170264+0000","last_became_active":"2024-09-18T07:49:24.445300+0000","last_became_peered":"2024-09-18T07:49:24.445300+0000","last_unstale":"2024-09-18T07:49:41.170264+0000","last_undegraded":"2024-09-18T07:49:41.170264+0000","last_fullsized":"2024-09-18T07:49:41.170264+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:52.882700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:45:52.882700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.8","version":"528'10318","reported_seq":22407,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:42.190385+0000","last_change":"2024-09-18T07:49:22.516716+0000","last_active":"2024-09-18T07:49:42.190385+0000","last_peered":"2024-09-18T07:49:42.190385+0000","last_clean":"2024-09-18T07:49:42.190385+0000","last_became_active":"2024-09-18T07:49:22.442874+0000","last_became_peered":"2024-09-18T07:49:22.442874+0000","last_unstale":"2024-09-18T07:49:42.190385+0000","last_undegraded":"2024-09-18T07:49:42.190385+0000","last_fullsized":"2024-09-18T07:49:42.190385+0000","mapping_epoch":515,"log_start":"497'6091","ondisk_log_start":"497'6091","created":459,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'5883","last_scrub_stamp":"2024-09-18T07:48:25.402648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:48:25.402648+0000","objects_scrubbed":5819,"log_size":4227,"log_dups_size":0,"ondisk_log_size":4227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:44.883793+0000","scrub_duration":1753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241958912,"num_objects":3692,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10318,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5929,"num_bytes_recovered":388431872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":832,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171491+0000","last_change":"2024-09-18T07:48:27.916178+0000","last_active":"2024-09-18T07:49:41.171491+0000","last_peered":"2024-09-18T07:49:41.171491+0000","last_clean":"2024-09-18T07:49:41.171491+0000","last_became_active":"2024-09-18T07:44:43.783295+0000","last_became_peered":"2024-09-18T07:44:43.783295+0000","last_unstale":"2024-09-18T07:49:41.171491+0000","last_undegraded":"2024-09-18T07:49:41.171491+0000","last_fullsized":"2024-09-18T07:49:41.171491+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:27.916095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:27.916095+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"323'17","reported_seq":708,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243619+0000","last_change":"2024-09-18T07:49:32.704380+0000","last_active":"2024-09-18T07:49:38.243619+0000","last_peered":"2024-09-18T07:49:38.243619+0000","last_clean":"2024-09-18T07:49:38.243619+0000","last_became_active":"2024-09-18T07:49:32.690652+0000","last_became_peered":"2024-09-18T07:49:32.690652+0000","last_unstale":"2024-09-18T07:49:38.243619+0000","last_undegraded":"2024-09-18T07:49:38.243619+0000","last_fullsized":"2024-09-18T07:49:38.243619+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'17","last_scrub_stamp":"2024-09-18T07:47:56.037047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.037047+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":23,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":721,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169359+0000","last_change":"2024-09-18T07:47:20.140678+0000","last_active":"2024-09-18T07:49:41.169359+0000","last_peered":"2024-09-18T07:49:41.169359+0000","last_clean":"2024-09-18T07:49:41.169359+0000","last_became_active":"2024-09-18T07:47:20.140376+0000","last_became_peered":"2024-09-18T07:47:20.140376+0000","last_unstale":"2024-09-18T07:49:41.169359+0000","last_undegraded":"2024-09-18T07:49:41.169359+0000","last_fullsized":"2024-09-18T07:49:41.169359+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:52.917296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:52.917296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":1023,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169775+0000","last_change":"2024-09-18T07:48:32.288661+0000","last_active":"2024-09-18T07:49:41.169775+0000","last_peered":"2024-09-18T07:49:41.169775+0000","last_clean":"2024-09-18T07:49:41.169775+0000","last_became_active":"2024-09-18T07:48:32.257016+0000","last_became_peered":"2024-09-18T07:48:32.257016+0000","last_unstale":"2024-09-18T07:49:41.169775+0000","last_undegraded":"2024-09-18T07:49:41.169775+0000","last_fullsized":"2024-09-18T07:49:41.169775+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:07.841406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:47:07.841406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1011,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170668+0000","last_change":"2024-09-18T07:47:16.834107+0000","last_active":"2024-09-18T07:49:41.170668+0000","last_peered":"2024-09-18T07:49:41.170668+0000","last_clean":"2024-09-18T07:49:41.170668+0000","last_became_active":"2024-09-18T07:47:11.201379+0000","last_became_peered":"2024-09-18T07:47:11.201379+0000","last_unstale":"2024-09-18T07:49:41.170668+0000","last_undegraded":"2024-09-18T07:49:41.170668+0000","last_fullsized":"2024-09-18T07:49:41.170668+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:16.834043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:16.834043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":800,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172125+0000","last_change":"2024-09-18T07:49:24.457052+0000","last_active":"2024-09-18T07:49:41.172125+0000","last_peered":"2024-09-18T07:49:41.172125+0000","last_clean":"2024-09-18T07:49:41.172125+0000","last_became_active":"2024-09-18T07:49:24.446434+0000","last_became_peered":"2024-09-18T07:49:24.446434+0000","last_unstale":"2024-09-18T07:49:41.172125+0000","last_undegraded":"2024-09-18T07:49:41.172125+0000","last_fullsized":"2024-09-18T07:49:41.172125+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:47:42.037791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.037791+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":78,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"528'10259","reported_seq":22539,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:42.222675+0000","last_change":"2024-09-18T07:49:19.413579+0000","last_active":"2024-09-18T07:49:42.222675+0000","last_peered":"2024-09-18T07:49:42.222675+0000","last_clean":"2024-09-18T07:49:42.222675+0000","last_became_active":"2024-09-18T07:49:19.392861+0000","last_became_peered":"2024-09-18T07:49:19.392861+0000","last_unstale":"2024-09-18T07:49:42.222675+0000","last_undegraded":"2024-09-18T07:49:42.222675+0000","last_fullsized":"2024-09-18T07:49:42.222675+0000","mapping_epoch":512,"log_start":"495'6027","ondisk_log_start":"495'6027","created":459,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4232,"log_dups_size":0,"ondisk_log_size":4232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243728384,"num_objects":3719,"num_object_clones":0,"num_object_copies":11157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10259,"num_write_kb":447296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6057,"num_bytes_recovered":396951552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":32,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.173657+0000","last_change":"2024-09-18T07:49:24.453650+0000","last_active":"2024-09-18T07:49:41.173657+0000","last_peered":"2024-09-18T07:49:41.173657+0000","last_clean":"2024-09-18T07:49:41.173657+0000","last_became_active":"2024-09-18T07:49:24.453318+0000","last_became_peered":"2024-09-18T07:49:24.453318+0000","last_unstale":"2024-09-18T07:49:41.173657+0000","last_undegraded":"2024-09-18T07:49:41.173657+0000","last_fullsized":"2024-09-18T07:49:41.173657+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:49:12.773317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:49:12.773317+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:38.973101+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"323'24","reported_seq":1065,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.198513+0000","last_change":"2024-09-18T07:47:20.159485+0000","last_active":"2024-09-18T07:49:41.198513+0000","last_peered":"2024-09-18T07:49:41.198513+0000","last_clean":"2024-09-18T07:49:41.198513+0000","last_became_active":"2024-09-18T07:47:20.138961+0000","last_became_peered":"2024-09-18T07:47:20.138961+0000","last_unstale":"2024-09-18T07:49:41.198513+0000","last_undegraded":"2024-09-18T07:49:41.198513+0000","last_fullsized":"2024-09-18T07:49:41.198513+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'24","last_scrub_stamp":"2024-09-18T07:44:06.211676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:06.211676+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:22.448696+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":995,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.197297+0000","last_change":"2024-09-18T07:48:10.165907+0000","last_active":"2024-09-18T07:49:41.197297+0000","last_peered":"2024-09-18T07:49:41.197297+0000","last_clean":"2024-09-18T07:49:41.197297+0000","last_became_active":"2024-09-18T07:48:10.142633+0000","last_became_peered":"2024-09-18T07:48:10.142633+0000","last_unstale":"2024-09-18T07:49:41.197297+0000","last_undegraded":"2024-09-18T07:49:41.197297+0000","last_fullsized":"2024-09-18T07:49:41.197297+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:42:44.401546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:44.401546+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:21.405419+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":1646,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171999+0000","last_change":"2024-09-18T07:47:20.162839+0000","last_active":"2024-09-18T07:49:41.171999+0000","last_peered":"2024-09-18T07:49:41.171999+0000","last_clean":"2024-09-18T07:49:41.171999+0000","last_became_active":"2024-09-18T07:47:20.138901+0000","last_became_peered":"2024-09-18T07:47:20.138901+0000","last_unstale":"2024-09-18T07:49:41.171999+0000","last_undegraded":"2024-09-18T07:49:41.171999+0000","last_fullsized":"2024-09-18T07:49:41.171999+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T07:46:47.771151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:46:47.771151+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":882,"num_read_kb":877,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":872,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243829+0000","last_change":"2024-09-18T07:47:20.145165+0000","last_active":"2024-09-18T07:49:38.243829+0000","last_peered":"2024-09-18T07:49:38.243829+0000","last_clean":"2024-09-18T07:49:38.243829+0000","last_became_active":"2024-09-18T07:47:20.144413+0000","last_became_peered":"2024-09-18T07:47:20.144413+0000","last_unstale":"2024-09-18T07:49:38.243829+0000","last_undegraded":"2024-09-18T07:49:38.243829+0000","last_fullsized":"2024-09-18T07:49:38.243829+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:50.587498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:44:50.587498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":553,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243801+0000","last_change":"2024-09-18T07:49:32.691732+0000","last_active":"2024-09-18T07:49:38.243801+0000","last_peered":"2024-09-18T07:49:38.243801+0000","last_clean":"2024-09-18T07:49:38.243801+0000","last_became_active":"2024-09-18T07:49:32.691427+0000","last_became_peered":"2024-09-18T07:49:32.691427+0000","last_unstale":"2024-09-18T07:49:38.243801+0000","last_undegraded":"2024-09-18T07:49:38.243801+0000","last_fullsized":"2024-09-18T07:49:38.243801+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:52.440332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:52.440332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.a","version":"528'10205","reported_seq":21962,"reported_epoch":528,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-18T07:49:42.180869+0000","last_change":"2024-09-18T07:49:33.964097+0000","last_active":"2024-09-18T07:49:42.180869+0000","last_peered":"2024-09-18T07:49:42.180869+0000","last_clean":"2024-09-18T07:49:31.606072+0000","last_became_active":"2024-09-18T07:49:33.895295+0000","last_became_peered":"2024-09-18T07:49:33.895295+0000","last_unstale":"2024-09-18T07:49:42.180869+0000","last_undegraded":"2024-09-18T07:49:42.180869+0000","last_fullsized":"2024-09-18T07:49:42.180869+0000","mapping_epoch":522,"log_start":"495'5972","ondisk_log_start":"495'5972","created":459,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'5664","last_scrub_stamp":"2024-09-18T07:48:22.026341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:48:22.026341+0000","objects_scrubbed":5623,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239796224,"num_objects":3659,"num_object_clones":0,"num_object_copies":10977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3659,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10205,"num_write_kb":443648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5758,"num_bytes_recovered":377290752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,2],"avail_no_missing":["6","2","3","5"],"object_location_counts":[{"shards":"2,5,6","objects":3659}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"398'34","reported_seq":809,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.174376+0000","last_change":"2024-09-18T07:47:58.415964+0000","last_active":"2024-09-18T07:49:41.174376+0000","last_peered":"2024-09-18T07:49:41.174376+0000","last_clean":"2024-09-18T07:49:41.174376+0000","last_became_active":"2024-09-18T07:44:43.786827+0000","last_became_peered":"2024-09-18T07:44:43.786827+0000","last_unstale":"2024-09-18T07:49:41.174376+0000","last_undegraded":"2024-09-18T07:49:41.174376+0000","last_fullsized":"2024-09-18T07:49:41.174376+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'34","last_scrub_stamp":"2024-09-18T07:47:58.415734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:58.415734+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":748,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.244188+0000","last_change":"2024-09-18T07:49:08.604265+0000","last_active":"2024-09-18T07:49:38.244188+0000","last_peered":"2024-09-18T07:49:38.244188+0000","last_clean":"2024-09-18T07:49:38.244188+0000","last_became_active":"2024-09-18T07:48:32.266432+0000","last_became_peered":"2024-09-18T07:48:32.266432+0000","last_unstale":"2024-09-18T07:49:38.244188+0000","last_undegraded":"2024-09-18T07:49:38.244188+0000","last_fullsized":"2024-09-18T07:49:38.244188+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:08.604203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:08.604203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:22.110767+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":815,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171986+0000","last_change":"2024-09-18T07:48:26.963331+0000","last_active":"2024-09-18T07:49:41.171986+0000","last_peered":"2024-09-18T07:49:41.171986+0000","last_clean":"2024-09-18T07:49:41.171986+0000","last_became_active":"2024-09-18T07:39:03.350419+0000","last_became_peered":"2024-09-18T07:39:03.350419+0000","last_unstale":"2024-09-18T07:49:41.171986+0000","last_undegraded":"2024-09-18T07:49:41.171986+0000","last_fullsized":"2024-09-18T07:49:41.171986+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:26.963223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:26.963223+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":768,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172313+0000","last_change":"2024-09-18T07:48:21.934191+0000","last_active":"2024-09-18T07:49:41.172313+0000","last_peered":"2024-09-18T07:49:41.172313+0000","last_clean":"2024-09-18T07:49:41.172313+0000","last_became_active":"2024-09-18T07:47:11.195084+0000","last_became_peered":"2024-09-18T07:47:11.195084+0000","last_unstale":"2024-09-18T07:49:41.172313+0000","last_undegraded":"2024-09-18T07:49:41.172313+0000","last_fullsized":"2024-09-18T07:49:41.172313+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:21.934130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:21.934130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:49.622120+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":876,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243725+0000","last_change":"2024-09-18T07:49:32.688396+0000","last_active":"2024-09-18T07:49:38.243725+0000","last_peered":"2024-09-18T07:49:38.243725+0000","last_clean":"2024-09-18T07:49:38.243725+0000","last_became_active":"2024-09-18T07:49:32.687989+0000","last_became_peered":"2024-09-18T07:49:32.687989+0000","last_unstale":"2024-09-18T07:49:38.243725+0000","last_undegraded":"2024-09-18T07:49:38.243725+0000","last_fullsized":"2024-09-18T07:49:38.243725+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:04.610883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:04.610883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:30.506002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"528'10352","reported_seq":19807,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:42.201157+0000","last_change":"2024-09-18T07:49:38.235683+0000","last_active":"2024-09-18T07:49:42.201157+0000","last_peered":"2024-09-18T07:49:42.201157+0000","last_clean":"2024-09-18T07:49:42.201157+0000","last_became_active":"2024-09-18T07:49:38.216903+0000","last_became_peered":"2024-09-18T07:49:38.216903+0000","last_unstale":"2024-09-18T07:49:42.201157+0000","last_undegraded":"2024-09-18T07:49:42.201157+0000","last_fullsized":"2024-09-18T07:49:42.201157+0000","mapping_epoch":525,"log_start":"497'6168","ondisk_log_start":"497'6168","created":459,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4184,"log_dups_size":0,"ondisk_log_size":4184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247070720,"num_objects":3770,"num_object_clones":0,"num_object_copies":11310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10352,"num_write_kb":451904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4643,"num_bytes_recovered":304283648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"323'19","reported_seq":1120,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170072+0000","last_change":"2024-09-18T07:49:32.696886+0000","last_active":"2024-09-18T07:49:41.170072+0000","last_peered":"2024-09-18T07:49:41.170072+0000","last_clean":"2024-09-18T07:49:41.170072+0000","last_became_active":"2024-09-18T07:49:32.684857+0000","last_became_peered":"2024-09-18T07:49:32.684857+0000","last_unstale":"2024-09-18T07:49:41.170072+0000","last_undegraded":"2024-09-18T07:49:41.170072+0000","last_fullsized":"2024-09-18T07:49:41.170072+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:42:43.402858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:43.402858+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:39.835405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":873,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169212+0000","last_change":"2024-09-18T07:44:43.793621+0000","last_active":"2024-09-18T07:49:41.169212+0000","last_peered":"2024-09-18T07:49:41.169212+0000","last_clean":"2024-09-18T07:49:41.169212+0000","last_became_active":"2024-09-18T07:44:43.790160+0000","last_became_peered":"2024-09-18T07:44:43.790160+0000","last_unstale":"2024-09-18T07:49:41.169212+0000","last_undegraded":"2024-09-18T07:49:41.169212+0000","last_fullsized":"2024-09-18T07:49:41.169212+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.078600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.078600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:19.692544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":881,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243445+0000","last_change":"2024-09-18T07:47:11.202169+0000","last_active":"2024-09-18T07:49:38.243445+0000","last_peered":"2024-09-18T07:49:38.243445+0000","last_clean":"2024-09-18T07:49:38.243445+0000","last_became_active":"2024-09-18T07:47:11.198103+0000","last_became_peered":"2024-09-18T07:47:11.198103+0000","last_unstale":"2024-09-18T07:49:38.243445+0000","last_undegraded":"2024-09-18T07:49:38.243445+0000","last_fullsized":"2024-09-18T07:49:38.243445+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:45.479624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:44:45.479624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:21.577272+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":875,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.174458+0000","last_change":"2024-09-18T07:48:32.259237+0000","last_active":"2024-09-18T07:49:41.174458+0000","last_peered":"2024-09-18T07:49:41.174458+0000","last_clean":"2024-09-18T07:49:41.174458+0000","last_became_active":"2024-09-18T07:48:32.258901+0000","last_became_peered":"2024-09-18T07:48:32.258901+0000","last_unstale":"2024-09-18T07:49:41.174458+0000","last_undegraded":"2024-09-18T07:49:41.174458+0000","last_fullsized":"2024-09-18T07:49:41.174458+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:26.691414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:26.691414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"528'10075","reported_seq":19954,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.840856+0000","last_change":"2024-09-18T07:49:33.657485+0000","last_active":"2024-09-18T07:49:41.840856+0000","last_peered":"2024-09-18T07:49:41.840856+0000","last_clean":"2024-09-18T07:49:41.840856+0000","last_became_active":"2024-09-18T07:49:33.641738+0000","last_became_peered":"2024-09-18T07:49:33.641738+0000","last_unstale":"2024-09-18T07:49:41.840856+0000","last_undegraded":"2024-09-18T07:49:41.840856+0000","last_fullsized":"2024-09-18T07:49:41.840856+0000","mapping_epoch":522,"log_start":"489'4841","ondisk_log_start":"489'4841","created":459,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":5234,"log_dups_size":0,"ondisk_log_size":5234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240582656,"num_objects":3671,"num_object_clones":0,"num_object_copies":11013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10075,"num_write_kb":439872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4854,"num_bytes_recovered":318111744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":418,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243009+0000","last_change":"2024-09-18T07:49:24.456347+0000","last_active":"2024-09-18T07:49:38.243009+0000","last_peered":"2024-09-18T07:49:38.243009+0000","last_clean":"2024-09-18T07:49:38.243009+0000","last_became_active":"2024-09-18T07:49:24.448174+0000","last_became_peered":"2024-09-18T07:49:24.448174+0000","last_unstale":"2024-09-18T07:49:38.243009+0000","last_undegraded":"2024-09-18T07:49:38.243009+0000","last_fullsized":"2024-09-18T07:49:38.243009+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:49:00.644450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:49:00.644450+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:11.512414+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"323'31","reported_seq":955,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169916+0000","last_change":"2024-09-18T07:48:22.002514+0000","last_active":"2024-09-18T07:49:41.169916+0000","last_peered":"2024-09-18T07:49:41.169916+0000","last_clean":"2024-09-18T07:49:41.169916+0000","last_became_active":"2024-09-18T07:47:20.132912+0000","last_became_peered":"2024-09-18T07:47:20.132912+0000","last_unstale":"2024-09-18T07:49:41.169916+0000","last_undegraded":"2024-09-18T07:49:41.169916+0000","last_fullsized":"2024-09-18T07:49:41.169916+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'31","last_scrub_stamp":"2024-09-18T07:48:22.002416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:22.002416+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":678,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.173497+0000","last_change":"2024-09-18T07:49:32.688728+0000","last_active":"2024-09-18T07:49:41.173497+0000","last_peered":"2024-09-18T07:49:41.173497+0000","last_clean":"2024-09-18T07:49:41.173497+0000","last_became_active":"2024-09-18T07:49:32.688352+0000","last_became_peered":"2024-09-18T07:49:32.688352+0000","last_unstale":"2024-09-18T07:49:41.173497+0000","last_undegraded":"2024-09-18T07:49:41.173497+0000","last_fullsized":"2024-09-18T07:49:41.173497+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:32.491919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:32.491919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:46.469204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":544,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169540+0000","last_change":"2024-09-18T07:47:20.131352+0000","last_active":"2024-09-18T07:49:41.169540+0000","last_peered":"2024-09-18T07:49:41.169540+0000","last_clean":"2024-09-18T07:49:41.169540+0000","last_became_active":"2024-09-18T07:47:20.130994+0000","last_became_peered":"2024-09-18T07:47:20.130994+0000","last_unstale":"2024-09-18T07:49:41.169540+0000","last_undegraded":"2024-09-18T07:49:41.169540+0000","last_fullsized":"2024-09-18T07:49:41.169540+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:13.019711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:13.019711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1016,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170635+0000","last_change":"2024-09-18T07:48:32.263327+0000","last_active":"2024-09-18T07:49:41.170635+0000","last_peered":"2024-09-18T07:49:41.170635+0000","last_clean":"2024-09-18T07:49:41.170635+0000","last_became_active":"2024-09-18T07:48:32.263024+0000","last_became_peered":"2024-09-18T07:48:32.263024+0000","last_unstale":"2024-09-18T07:49:41.170635+0000","last_undegraded":"2024-09-18T07:49:41.170635+0000","last_fullsized":"2024-09-18T07:49:41.170635+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:06.889139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:06.889139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"528'10245","reported_seq":12171,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:42.187833+0000","last_change":"2024-09-18T07:48:27.491090+0000","last_active":"2024-09-18T07:49:42.187833+0000","last_peered":"2024-09-18T07:49:42.187833+0000","last_clean":"2024-09-18T07:49:42.187833+0000","last_became_active":"2024-09-18T07:47:38.218858+0000","last_became_peered":"2024-09-18T07:47:38.218858+0000","last_unstale":"2024-09-18T07:49:42.187833+0000","last_undegraded":"2024-09-18T07:49:42.187833+0000","last_fullsized":"2024-09-18T07:49:42.187833+0000","mapping_epoch":475,"log_start":"496'6056","ondisk_log_start":"496'6056","created":459,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'6054","last_scrub_stamp":"2024-09-18T07:48:27.490707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:48:27.490707+0000","objects_scrubbed":6013,"log_size":4189,"log_dups_size":0,"ondisk_log_size":4189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244908032,"num_objects":3737,"num_object_clones":0,"num_object_copies":11211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10245,"num_write_kb":447424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":848,"num_bytes_recovered":55574528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":819,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171319+0000","last_change":"2024-09-18T07:48:32.261433+0000","last_active":"2024-09-18T07:49:41.171319+0000","last_peered":"2024-09-18T07:49:41.171319+0000","last_clean":"2024-09-18T07:49:41.171319+0000","last_became_active":"2024-09-18T07:48:32.259457+0000","last_became_peered":"2024-09-18T07:48:32.259457+0000","last_unstale":"2024-09-18T07:49:41.171319+0000","last_undegraded":"2024-09-18T07:49:41.171319+0000","last_fullsized":"2024-09-18T07:49:41.171319+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:46:24.585120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:46:24.585120+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"323'20","reported_seq":927,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169753+0000","last_change":"2024-09-18T07:48:32.303165+0000","last_active":"2024-09-18T07:49:41.169753+0000","last_peered":"2024-09-18T07:49:41.169753+0000","last_clean":"2024-09-18T07:49:41.169753+0000","last_became_active":"2024-09-18T07:48:32.269075+0000","last_became_peered":"2024-09-18T07:48:32.269075+0000","last_unstale":"2024-09-18T07:49:41.169753+0000","last_undegraded":"2024-09-18T07:49:41.169753+0000","last_fullsized":"2024-09-18T07:49:41.169753+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:46:14.047907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:14.047907+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":405,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169286+0000","last_change":"2024-09-18T07:48:32.255423+0000","last_active":"2024-09-18T07:49:41.169286+0000","last_peered":"2024-09-18T07:49:41.169286+0000","last_clean":"2024-09-18T07:49:41.169286+0000","last_became_active":"2024-09-18T07:48:32.255130+0000","last_became_peered":"2024-09-18T07:48:32.255130+0000","last_unstale":"2024-09-18T07:49:41.169286+0000","last_undegraded":"2024-09-18T07:49:41.169286+0000","last_fullsized":"2024-09-18T07:49:41.169286+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:42.225418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.225418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"461'13","reported_seq":1258,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171006+0000","last_change":"2024-09-18T07:47:15.865446+0000","last_active":"2024-09-18T07:49:41.171006+0000","last_peered":"2024-09-18T07:49:41.171006+0000","last_clean":"2024-09-18T07:49:41.171006+0000","last_became_active":"2024-09-18T07:47:11.203256+0000","last_became_peered":"2024-09-18T07:47:11.203256+0000","last_unstale":"2024-09-18T07:49:41.171006+0000","last_undegraded":"2024-09-18T07:49:41.171006+0000","last_fullsized":"2024-09-18T07:49:41.171006+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'13","last_scrub_stamp":"2024-09-18T07:47:15.865373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:15.865373+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":896,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170314+0000","last_change":"2024-09-18T07:49:32.690291+0000","last_active":"2024-09-18T07:49:41.170314+0000","last_peered":"2024-09-18T07:49:41.170314+0000","last_clean":"2024-09-18T07:49:41.170314+0000","last_became_active":"2024-09-18T07:49:32.689562+0000","last_became_peered":"2024-09-18T07:49:32.689562+0000","last_unstale":"2024-09-18T07:49:41.170314+0000","last_undegraded":"2024-09-18T07:49:41.170314+0000","last_fullsized":"2024-09-18T07:49:41.170314+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:48.207484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:48.207484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"528'10277","reported_seq":10427,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.864933+0000","last_change":"2024-09-18T07:47:10.191967+0000","last_active":"2024-09-18T07:49:41.864933+0000","last_peered":"2024-09-18T07:49:41.864933+0000","last_clean":"2024-09-18T07:49:41.864933+0000","last_became_active":"2024-09-18T07:47:10.190921+0000","last_became_peered":"2024-09-18T07:47:10.190921+0000","last_unstale":"2024-09-18T07:49:41.864933+0000","last_undegraded":"2024-09-18T07:49:41.864933+0000","last_fullsized":"2024-09-18T07:49:41.864933+0000","mapping_epoch":459,"log_start":"493'5845","ondisk_log_start":"493'5845","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4432,"log_dups_size":0,"ondisk_log_size":4432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248709120,"num_objects":3795,"num_object_clones":0,"num_object_copies":11385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10277,"num_write_kb":450304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"323'15","reported_seq":1069,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.196857+0000","last_change":"2024-09-18T07:48:22.197957+0000","last_active":"2024-09-18T07:49:41.196857+0000","last_peered":"2024-09-18T07:49:41.196857+0000","last_clean":"2024-09-18T07:49:41.196857+0000","last_became_active":"2024-09-18T07:47:20.138834+0000","last_became_peered":"2024-09-18T07:47:20.138834+0000","last_unstale":"2024-09-18T07:49:41.196857+0000","last_undegraded":"2024-09-18T07:49:41.196857+0000","last_fullsized":"2024-09-18T07:49:41.196857+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:48:22.197888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:22.197888+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:43.300890+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":915,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243831+0000","last_change":"2024-09-18T07:48:32.262270+0000","last_active":"2024-09-18T07:49:38.243831+0000","last_peered":"2024-09-18T07:49:38.243831+0000","last_clean":"2024-09-18T07:49:38.243831+0000","last_became_active":"2024-09-18T07:48:32.261507+0000","last_became_peered":"2024-09-18T07:48:32.261507+0000","last_unstale":"2024-09-18T07:49:38.243831+0000","last_undegraded":"2024-09-18T07:49:38.243831+0000","last_fullsized":"2024-09-18T07:49:38.243831+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:51.021530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:51.021530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:23.522646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1004,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169210+0000","last_change":"2024-09-18T07:48:32.265137+0000","last_active":"2024-09-18T07:49:41.169210+0000","last_peered":"2024-09-18T07:49:41.169210+0000","last_clean":"2024-09-18T07:49:41.169210+0000","last_became_active":"2024-09-18T07:48:32.261737+0000","last_became_peered":"2024-09-18T07:48:32.261737+0000","last_unstale":"2024-09-18T07:49:41.169210+0000","last_undegraded":"2024-09-18T07:49:41.169210+0000","last_fullsized":"2024-09-18T07:49:41.169210+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:22.799585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:22.799585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":425,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171519+0000","last_change":"2024-09-18T07:46:29.323007+0000","last_active":"2024-09-18T07:49:41.171519+0000","last_peered":"2024-09-18T07:49:41.171519+0000","last_clean":"2024-09-18T07:49:41.171519+0000","last_became_active":"2024-09-18T07:46:29.322779+0000","last_became_peered":"2024-09-18T07:46:29.322779+0000","last_unstale":"2024-09-18T07:49:41.171519+0000","last_undegraded":"2024-09-18T07:49:41.171519+0000","last_fullsized":"2024-09-18T07:49:41.171519+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:43:34.257746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:43:34.257746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"528'10160","reported_seq":10309,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.863439+0000","last_change":"2024-09-18T07:47:11.201597+0000","last_active":"2024-09-18T07:49:41.863439+0000","last_peered":"2024-09-18T07:49:41.863439+0000","last_clean":"2024-09-18T07:49:41.863439+0000","last_became_active":"2024-09-18T07:47:11.201275+0000","last_became_peered":"2024-09-18T07:47:11.201275+0000","last_unstale":"2024-09-18T07:49:41.863439+0000","last_undegraded":"2024-09-18T07:49:41.863439+0000","last_fullsized":"2024-09-18T07:49:41.863439+0000","mapping_epoch":460,"log_start":"493'5732","ondisk_log_start":"493'5732","created":459,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242876416,"num_objects":3706,"num_object_clones":0,"num_object_copies":11118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10160,"num_write_kb":443712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"323'35","reported_seq":727,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.173638+0000","last_change":"2024-09-18T07:47:52.386750+0000","last_active":"2024-09-18T07:49:41.173638+0000","last_peered":"2024-09-18T07:49:41.173638+0000","last_clean":"2024-09-18T07:49:41.173638+0000","last_became_active":"2024-09-18T07:47:20.141942+0000","last_became_peered":"2024-09-18T07:47:20.141942+0000","last_unstale":"2024-09-18T07:49:41.173638+0000","last_undegraded":"2024-09-18T07:49:41.173638+0000","last_fullsized":"2024-09-18T07:49:41.173638+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'35","last_scrub_stamp":"2024-09-18T07:47:52.386688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.386688+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":781,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172403+0000","last_change":"2024-09-18T07:47:17.794908+0000","last_active":"2024-09-18T07:49:41.172403+0000","last_peered":"2024-09-18T07:49:41.172403+0000","last_clean":"2024-09-18T07:49:41.172403+0000","last_became_active":"2024-09-18T07:47:11.194873+0000","last_became_peered":"2024-09-18T07:47:11.194873+0000","last_unstale":"2024-09-18T07:49:41.172403+0000","last_undegraded":"2024-09-18T07:49:41.172403+0000","last_fullsized":"2024-09-18T07:49:41.172403+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:17.794850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:17.794850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1019,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170172+0000","last_change":"2024-09-18T07:48:32.257353+0000","last_active":"2024-09-18T07:49:41.170172+0000","last_peered":"2024-09-18T07:49:41.170172+0000","last_clean":"2024-09-18T07:49:41.170172+0000","last_became_active":"2024-09-18T07:48:32.257023+0000","last_became_peered":"2024-09-18T07:48:32.257023+0000","last_unstale":"2024-09-18T07:49:41.170172+0000","last_undegraded":"2024-09-18T07:49:41.170172+0000","last_fullsized":"2024-09-18T07:49:41.170172+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:20.767517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:20.767517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":763,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172071+0000","last_change":"2024-09-18T07:48:32.265587+0000","last_active":"2024-09-18T07:49:41.172071+0000","last_peered":"2024-09-18T07:49:41.172071+0000","last_clean":"2024-09-18T07:49:41.172071+0000","last_became_active":"2024-09-18T07:48:32.265086+0000","last_became_peered":"2024-09-18T07:48:32.265086+0000","last_unstale":"2024-09-18T07:49:41.172071+0000","last_undegraded":"2024-09-18T07:49:41.172071+0000","last_fullsized":"2024-09-18T07:49:41.172071+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:50.079275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:50.079275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"528'10107","reported_seq":11936,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.820479+0000","last_change":"2024-09-18T07:47:53.337653+0000","last_active":"2024-09-18T07:49:41.820479+0000","last_peered":"2024-09-18T07:49:41.820479+0000","last_clean":"2024-09-18T07:49:41.820479+0000","last_became_active":"2024-09-18T07:47:53.331769+0000","last_became_peered":"2024-09-18T07:47:53.331769+0000","last_unstale":"2024-09-18T07:49:41.820479+0000","last_undegraded":"2024-09-18T07:49:41.820479+0000","last_fullsized":"2024-09-18T07:49:41.820479+0000","mapping_epoch":481,"log_start":"489'4926","ondisk_log_start":"489'4926","created":459,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":5181,"log_dups_size":0,"ondisk_log_size":5181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245825536,"num_objects":3751,"num_object_clones":0,"num_object_copies":11253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10107,"num_write_kb":443456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":831,"num_bytes_recovered":54460416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"323'18","reported_seq":986,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169146+0000","last_change":"2024-09-18T07:48:32.288842+0000","last_active":"2024-09-18T07:49:41.169146+0000","last_peered":"2024-09-18T07:49:41.169146+0000","last_clean":"2024-09-18T07:49:41.169146+0000","last_became_active":"2024-09-18T07:48:32.262707+0000","last_became_peered":"2024-09-18T07:48:32.262707+0000","last_unstale":"2024-09-18T07:49:41.169146+0000","last_undegraded":"2024-09-18T07:49:41.169146+0000","last_fullsized":"2024-09-18T07:49:41.169146+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:46:25.012172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:25.012172+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":773,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172643+0000","last_change":"2024-09-18T07:47:20.139266+0000","last_active":"2024-09-18T07:49:41.172643+0000","last_peered":"2024-09-18T07:49:41.172643+0000","last_clean":"2024-09-18T07:49:41.172643+0000","last_became_active":"2024-09-18T07:47:20.138910+0000","last_became_peered":"2024-09-18T07:47:20.138910+0000","last_unstale":"2024-09-18T07:49:41.172643+0000","last_undegraded":"2024-09-18T07:49:41.172643+0000","last_fullsized":"2024-09-18T07:49:41.172643+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:51.752578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:51.752578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1029,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169280+0000","last_change":"2024-09-18T07:48:23.757438+0000","last_active":"2024-09-18T07:49:41.169280+0000","last_peered":"2024-09-18T07:49:41.169280+0000","last_clean":"2024-09-18T07:49:41.169280+0000","last_became_active":"2024-09-18T07:47:20.139842+0000","last_became_peered":"2024-09-18T07:47:20.139842+0000","last_unstale":"2024-09-18T07:49:41.169280+0000","last_undegraded":"2024-09-18T07:49:41.169280+0000","last_fullsized":"2024-09-18T07:49:41.169280+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:23.757372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:23.757372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":934,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171836+0000","last_change":"2024-09-18T07:48:32.255818+0000","last_active":"2024-09-18T07:49:41.171836+0000","last_peered":"2024-09-18T07:49:41.171836+0000","last_clean":"2024-09-18T07:49:41.171836+0000","last_became_active":"2024-09-18T07:48:32.253112+0000","last_became_peered":"2024-09-18T07:48:32.253112+0000","last_unstale":"2024-09-18T07:49:41.171836+0000","last_undegraded":"2024-09-18T07:49:41.171836+0000","last_fullsized":"2024-09-18T07:49:41.171836+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:40.439672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:40.439672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"526'10133","reported_seq":12079,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:39.220359+0000","last_change":"2024-09-18T07:48:53.603702+0000","last_active":"2024-09-18T07:49:39.220359+0000","last_peered":"2024-09-18T07:49:39.220359+0000","last_clean":"2024-09-18T07:49:39.220359+0000","last_became_active":"2024-09-18T07:47:34.181063+0000","last_became_peered":"2024-09-18T07:47:34.181063+0000","last_unstale":"2024-09-18T07:49:39.220359+0000","last_undegraded":"2024-09-18T07:49:39.220359+0000","last_fullsized":"2024-09-18T07:49:39.220359+0000","mapping_epoch":471,"log_start":"493'5650","ondisk_log_start":"493'5650","created":459,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'7617","last_scrub_stamp":"2024-09-18T07:48:53.603594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:48:53.603594+0000","objects_scrubbed":6521,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:06.768809+0000","scrub_duration":1906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259063808,"num_objects":3953,"num_object_clones":0,"num_object_copies":11859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10133,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":858,"num_bytes_recovered":56229888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"323'27","reported_seq":973,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.242717+0000","last_change":"2024-09-18T07:48:03.057334+0000","last_active":"2024-09-18T07:49:38.242717+0000","last_peered":"2024-09-18T07:49:38.242717+0000","last_clean":"2024-09-18T07:49:38.242717+0000","last_became_active":"2024-09-18T07:47:11.196288+0000","last_became_peered":"2024-09-18T07:47:11.196288+0000","last_unstale":"2024-09-18T07:49:38.242717+0000","last_undegraded":"2024-09-18T07:49:38.242717+0000","last_fullsized":"2024-09-18T07:49:38.242717+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:48:03.057182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:03.057182+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":781,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172683+0000","last_change":"2024-09-18T07:48:34.003445+0000","last_active":"2024-09-18T07:49:41.172683+0000","last_peered":"2024-09-18T07:49:41.172683+0000","last_clean":"2024-09-18T07:49:41.172683+0000","last_became_active":"2024-09-18T07:44:43.778284+0000","last_became_peered":"2024-09-18T07:44:43.778284+0000","last_unstale":"2024-09-18T07:49:41.172683+0000","last_undegraded":"2024-09-18T07:49:41.172683+0000","last_fullsized":"2024-09-18T07:49:41.172683+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:48:34.003134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:34.003134+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:55.154854+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"398'14","reported_seq":972,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171688+0000","last_change":"2024-09-18T07:46:21.582783+0000","last_active":"2024-09-18T07:49:41.171688+0000","last_peered":"2024-09-18T07:49:41.171688+0000","last_clean":"2024-09-18T07:49:41.171688+0000","last_became_active":"2024-09-18T07:44:43.782355+0000","last_became_peered":"2024-09-18T07:44:43.782355+0000","last_unstale":"2024-09-18T07:49:41.171688+0000","last_undegraded":"2024-09-18T07:49:41.171688+0000","last_fullsized":"2024-09-18T07:49:41.171688+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:46:21.582737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:21.582737+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":840,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.242957+0000","last_change":"2024-09-18T07:47:42.111034+0000","last_active":"2024-09-18T07:49:38.242957+0000","last_peered":"2024-09-18T07:49:38.242957+0000","last_clean":"2024-09-18T07:49:38.242957+0000","last_became_active":"2024-09-18T07:47:11.197583+0000","last_became_peered":"2024-09-18T07:47:11.197583+0000","last_unstale":"2024-09-18T07:49:38.242957+0000","last_undegraded":"2024-09-18T07:49:38.242957+0000","last_fullsized":"2024-09-18T07:49:38.242957+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:42.110980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.110980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:31.565452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":619,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.173920+0000","last_change":"2024-09-18T07:47:11.188646+0000","last_active":"2024-09-18T07:49:41.173920+0000","last_peered":"2024-09-18T07:49:41.173920+0000","last_clean":"2024-09-18T07:49:41.173920+0000","last_became_active":"2024-09-18T07:47:11.188332+0000","last_became_peered":"2024-09-18T07:47:11.188332+0000","last_unstale":"2024-09-18T07:49:41.173920+0000","last_undegraded":"2024-09-18T07:49:41.173920+0000","last_fullsized":"2024-09-18T07:49:41.173920+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:08.181420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:08.181420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:40.813652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"323'12","reported_seq":696,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172490+0000","last_change":"2024-09-18T07:48:10.211661+0000","last_active":"2024-09-18T07:49:41.172490+0000","last_peered":"2024-09-18T07:49:41.172490+0000","last_clean":"2024-09-18T07:49:41.172490+0000","last_became_active":"2024-09-18T07:48:10.160725+0000","last_became_peered":"2024-09-18T07:48:10.160725+0000","last_unstale":"2024-09-18T07:49:41.172490+0000","last_undegraded":"2024-09-18T07:49:41.172490+0000","last_fullsized":"2024-09-18T07:49:41.172490+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'12","last_scrub_stamp":"2024-09-18T07:48:03.699388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:03.699388+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":918,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243634+0000","last_change":"2024-09-18T07:48:32.261425+0000","last_active":"2024-09-18T07:49:38.243634+0000","last_peered":"2024-09-18T07:49:38.243634+0000","last_clean":"2024-09-18T07:49:38.243634+0000","last_became_active":"2024-09-18T07:48:32.261094+0000","last_became_peered":"2024-09-18T07:48:32.261094+0000","last_unstale":"2024-09-18T07:49:38.243634+0000","last_undegraded":"2024-09-18T07:49:38.243634+0000","last_fullsized":"2024-09-18T07:49:38.243634+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:53.089903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.089903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:32.526030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":773,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172184+0000","last_change":"2024-09-18T07:48:56.269806+0000","last_active":"2024-09-18T07:49:41.172184+0000","last_peered":"2024-09-18T07:49:41.172184+0000","last_clean":"2024-09-18T07:49:41.172184+0000","last_became_active":"2024-09-18T07:44:43.782348+0000","last_became_peered":"2024-09-18T07:44:43.782348+0000","last_unstale":"2024-09-18T07:49:41.172184+0000","last_undegraded":"2024-09-18T07:49:41.172184+0000","last_fullsized":"2024-09-18T07:49:41.172184+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:56.269745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:56.269745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:03.146403+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"323'15","reported_seq":668,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172320+0000","last_change":"2024-09-18T07:48:32.298683+0000","last_active":"2024-09-18T07:49:41.172320+0000","last_peered":"2024-09-18T07:49:41.172320+0000","last_clean":"2024-09-18T07:49:41.172320+0000","last_became_active":"2024-09-18T07:48:32.259977+0000","last_became_peered":"2024-09-18T07:48:32.259977+0000","last_unstale":"2024-09-18T07:49:41.172320+0000","last_undegraded":"2024-09-18T07:49:41.172320+0000","last_fullsized":"2024-09-18T07:49:41.172320+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:48:27.802966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:27.802966+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:53.895081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":781,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171609+0000","last_change":"2024-09-18T07:45:51.559901+0000","last_active":"2024-09-18T07:49:41.171609+0000","last_peered":"2024-09-18T07:49:41.171609+0000","last_clean":"2024-09-18T07:49:41.171609+0000","last_became_active":"2024-09-18T07:44:43.782930+0000","last_became_peered":"2024-09-18T07:44:43.782930+0000","last_unstale":"2024-09-18T07:49:41.171609+0000","last_undegraded":"2024-09-18T07:49:41.171609+0000","last_fullsized":"2024-09-18T07:49:41.171609+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:51.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:45:51.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":945,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172403+0000","last_change":"2024-09-18T07:48:32.258194+0000","last_active":"2024-09-18T07:49:41.172403+0000","last_peered":"2024-09-18T07:49:41.172403+0000","last_clean":"2024-09-18T07:49:41.172403+0000","last_became_active":"2024-09-18T07:48:32.255775+0000","last_became_peered":"2024-09-18T07:48:32.255775+0000","last_unstale":"2024-09-18T07:49:41.172403+0000","last_undegraded":"2024-09-18T07:49:41.172403+0000","last_fullsized":"2024-09-18T07:49:41.172403+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:30.750167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:30.750167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":990,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170871+0000","last_change":"2024-09-18T07:48:03.866565+0000","last_active":"2024-09-18T07:49:41.170871+0000","last_peered":"2024-09-18T07:49:41.170871+0000","last_clean":"2024-09-18T07:49:41.170871+0000","last_became_active":"2024-09-18T07:44:43.790582+0000","last_became_peered":"2024-09-18T07:44:43.790582+0000","last_unstale":"2024-09-18T07:49:41.170871+0000","last_undegraded":"2024-09-18T07:49:41.170871+0000","last_fullsized":"2024-09-18T07:49:41.170871+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:03.866499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:03.866499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"323'15","reported_seq":953,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243200+0000","last_change":"2024-09-18T07:48:32.286075+0000","last_active":"2024-09-18T07:49:38.243200+0000","last_peered":"2024-09-18T07:49:38.243200+0000","last_clean":"2024-09-18T07:49:38.243200+0000","last_became_active":"2024-09-18T07:48:32.267360+0000","last_became_peered":"2024-09-18T07:48:32.267360+0000","last_unstale":"2024-09-18T07:49:38.243200+0000","last_undegraded":"2024-09-18T07:49:38.243200+0000","last_fullsized":"2024-09-18T07:49:38.243200+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:48:25.381861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:25.381861+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":452,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171629+0000","last_change":"2024-09-18T07:48:32.254574+0000","last_active":"2024-09-18T07:49:41.171629+0000","last_peered":"2024-09-18T07:49:41.171629+0000","last_clean":"2024-09-18T07:49:41.171629+0000","last_became_active":"2024-09-18T07:48:32.254219+0000","last_became_peered":"2024-09-18T07:48:32.254219+0000","last_unstale":"2024-09-18T07:49:41.171629+0000","last_undegraded":"2024-09-18T07:49:41.171629+0000","last_fullsized":"2024-09-18T07:49:41.171629+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:52.023913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.023913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"500'11","reported_seq":381,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171867+0000","last_change":"2024-09-18T07:48:32.311073+0000","last_active":"2024-09-18T07:49:41.171867+0000","last_peered":"2024-09-18T07:49:41.171867+0000","last_clean":"2024-09-18T07:49:41.171867+0000","last_became_active":"2024-09-18T07:48:32.264561+0000","last_became_peered":"2024-09-18T07:48:32.264561+0000","last_unstale":"2024-09-18T07:49:41.171867+0000","last_undegraded":"2024-09-18T07:49:41.171867+0000","last_fullsized":"2024-09-18T07:49:41.171867+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:46:08.086437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:08.086437+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":878,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169896+0000","last_change":"2024-09-18T07:47:11.193555+0000","last_active":"2024-09-18T07:49:41.169896+0000","last_peered":"2024-09-18T07:49:41.169896+0000","last_clean":"2024-09-18T07:49:41.169896+0000","last_became_active":"2024-09-18T07:47:11.193273+0000","last_became_peered":"2024-09-18T07:47:11.193273+0000","last_unstale":"2024-09-18T07:49:41.169896+0000","last_undegraded":"2024-09-18T07:49:41.169896+0000","last_fullsized":"2024-09-18T07:49:41.169896+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:48.955463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.955463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:23.715894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"323'20","reported_seq":852,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171859+0000","last_change":"2024-09-18T07:49:32.702824+0000","last_active":"2024-09-18T07:49:41.171859+0000","last_peered":"2024-09-18T07:49:41.171859+0000","last_clean":"2024-09-18T07:49:41.171859+0000","last_became_active":"2024-09-18T07:49:32.690835+0000","last_became_peered":"2024-09-18T07:49:32.690835+0000","last_unstale":"2024-09-18T07:49:41.171859+0000","last_undegraded":"2024-09-18T07:49:41.171859+0000","last_fullsized":"2024-09-18T07:49:41.171859+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:46:14.650883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:14.650883+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":775,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172142+0000","last_change":"2024-09-18T07:49:00.272282+0000","last_active":"2024-09-18T07:49:41.172142+0000","last_peered":"2024-09-18T07:49:41.172142+0000","last_clean":"2024-09-18T07:49:41.172142+0000","last_became_active":"2024-09-18T07:44:43.779305+0000","last_became_peered":"2024-09-18T07:44:43.779305+0000","last_unstale":"2024-09-18T07:49:41.172142+0000","last_undegraded":"2024-09-18T07:49:41.172142+0000","last_fullsized":"2024-09-18T07:49:41.172142+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:00.272210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:00.272210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:16.088683+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"467'14","reported_seq":1062,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243365+0000","last_change":"2024-09-18T07:47:20.169193+0000","last_active":"2024-09-18T07:49:38.243365+0000","last_peered":"2024-09-18T07:49:38.243365+0000","last_clean":"2024-09-18T07:49:38.243365+0000","last_became_active":"2024-09-18T07:47:20.141913+0000","last_became_peered":"2024-09-18T07:47:20.141913+0000","last_unstale":"2024-09-18T07:49:38.243365+0000","last_undegraded":"2024-09-18T07:49:38.243365+0000","last_fullsized":"2024-09-18T07:49:38.243365+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:40:40.352040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.352040+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":440,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172416+0000","last_change":"2024-09-18T07:48:32.259527+0000","last_active":"2024-09-18T07:49:41.172416+0000","last_peered":"2024-09-18T07:49:41.172416+0000","last_clean":"2024-09-18T07:49:41.172416+0000","last_became_active":"2024-09-18T07:48:32.259107+0000","last_became_peered":"2024-09-18T07:48:32.259107+0000","last_unstale":"2024-09-18T07:49:41.172416+0000","last_undegraded":"2024-09-18T07:49:41.172416+0000","last_fullsized":"2024-09-18T07:49:41.172416+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:45.971311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:45.971311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"323'19","reported_seq":1021,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169643+0000","last_change":"2024-09-18T07:47:20.196648+0000","last_active":"2024-09-18T07:49:41.169643+0000","last_peered":"2024-09-18T07:49:41.169643+0000","last_clean":"2024-09-18T07:49:41.169643+0000","last_became_active":"2024-09-18T07:47:20.140070+0000","last_became_peered":"2024-09-18T07:47:20.140070+0000","last_unstale":"2024-09-18T07:49:41.169643+0000","last_undegraded":"2024-09-18T07:49:41.169643+0000","last_fullsized":"2024-09-18T07:49:41.169643+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:41:54.022421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.022421+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:22.741685+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":935,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169858+0000","last_change":"2024-09-18T07:46:06.964899+0000","last_active":"2024-09-18T07:49:41.169858+0000","last_peered":"2024-09-18T07:49:41.169858+0000","last_clean":"2024-09-18T07:49:41.169858+0000","last_became_active":"2024-09-18T07:44:43.780305+0000","last_became_peered":"2024-09-18T07:44:43.780305+0000","last_unstale":"2024-09-18T07:49:41.169858+0000","last_undegraded":"2024-09-18T07:49:41.169858+0000","last_fullsized":"2024-09-18T07:49:41.169858+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:06.964847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:06.964847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":603,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.242846+0000","last_change":"2024-09-18T07:48:10.163971+0000","last_active":"2024-09-18T07:49:38.242846+0000","last_peered":"2024-09-18T07:49:38.242846+0000","last_clean":"2024-09-18T07:49:38.242846+0000","last_became_active":"2024-09-18T07:48:10.163110+0000","last_became_peered":"2024-09-18T07:48:10.163110+0000","last_unstale":"2024-09-18T07:49:38.242846+0000","last_undegraded":"2024-09-18T07:49:38.242846+0000","last_fullsized":"2024-09-18T07:49:38.242846+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:59.022075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:59.022075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":507,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.197143+0000","last_change":"2024-09-18T07:48:25.249174+0000","last_active":"2024-09-18T07:49:41.197143+0000","last_peered":"2024-09-18T07:49:41.197143+0000","last_clean":"2024-09-18T07:49:41.197143+0000","last_became_active":"2024-09-18T07:48:10.154399+0000","last_became_peered":"2024-09-18T07:48:10.154399+0000","last_unstale":"2024-09-18T07:49:41.197143+0000","last_undegraded":"2024-09-18T07:49:41.197143+0000","last_fullsized":"2024-09-18T07:49:41.197143+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:25.249102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:25.249102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:50.673733+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"323'15","reported_seq":675,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.173443+0000","last_change":"2024-09-18T07:49:32.696134+0000","last_active":"2024-09-18T07:49:41.173443+0000","last_peered":"2024-09-18T07:49:41.173443+0000","last_clean":"2024-09-18T07:49:41.173443+0000","last_became_active":"2024-09-18T07:49:32.687721+0000","last_became_peered":"2024-09-18T07:49:32.687721+0000","last_unstale":"2024-09-18T07:49:41.173443+0000","last_undegraded":"2024-09-18T07:49:41.173443+0000","last_fullsized":"2024-09-18T07:49:41.173443+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:47:49.411931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:49.411931+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":774,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172054+0000","last_change":"2024-09-18T07:48:33.013859+0000","last_active":"2024-09-18T07:49:41.172054+0000","last_peered":"2024-09-18T07:49:41.172054+0000","last_clean":"2024-09-18T07:49:41.172054+0000","last_became_active":"2024-09-18T07:48:32.272079+0000","last_became_peered":"2024-09-18T07:48:32.272079+0000","last_unstale":"2024-09-18T07:49:41.172054+0000","last_undegraded":"2024-09-18T07:49:41.172054+0000","last_fullsized":"2024-09-18T07:49:41.172054+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:33.013808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:33.013808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:02.298988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":499,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.196644+0000","last_change":"2024-09-18T07:48:32.259701+0000","last_active":"2024-09-18T07:49:41.196644+0000","last_peered":"2024-09-18T07:49:41.196644+0000","last_clean":"2024-09-18T07:49:41.196644+0000","last_became_active":"2024-09-18T07:48:32.258708+0000","last_became_peered":"2024-09-18T07:48:32.258708+0000","last_unstale":"2024-09-18T07:49:41.196644+0000","last_undegraded":"2024-09-18T07:49:41.196644+0000","last_fullsized":"2024-09-18T07:49:41.196644+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:13.911901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:13.911901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:31.499215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1017,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.196527+0000","last_change":"2024-09-18T07:47:44.214937+0000","last_active":"2024-09-18T07:49:41.196527+0000","last_peered":"2024-09-18T07:49:41.196527+0000","last_clean":"2024-09-18T07:49:41.196527+0000","last_became_active":"2024-09-18T07:47:20.135880+0000","last_became_peered":"2024-09-18T07:47:20.135880+0000","last_unstale":"2024-09-18T07:49:41.196527+0000","last_undegraded":"2024-09-18T07:49:41.196527+0000","last_fullsized":"2024-09-18T07:49:41.196527+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:44.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:44.214865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:35.490659+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"323'19","reported_seq":745,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171331+0000","last_change":"2024-09-18T07:48:58.276742+0000","last_active":"2024-09-18T07:49:41.171331+0000","last_peered":"2024-09-18T07:49:41.171331+0000","last_clean":"2024-09-18T07:49:41.171331+0000","last_became_active":"2024-09-18T07:48:32.259858+0000","last_became_peered":"2024-09-18T07:48:32.259858+0000","last_unstale":"2024-09-18T07:49:41.171331+0000","last_undegraded":"2024-09-18T07:49:41.171331+0000","last_fullsized":"2024-09-18T07:49:41.171331+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:48:58.276659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:58.276659+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:10.277979+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":916,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243461+0000","last_change":"2024-09-18T07:47:52.070685+0000","last_active":"2024-09-18T07:49:38.243461+0000","last_peered":"2024-09-18T07:49:38.243461+0000","last_clean":"2024-09-18T07:49:38.243461+0000","last_became_active":"2024-09-18T07:47:20.140495+0000","last_became_peered":"2024-09-18T07:47:20.140495+0000","last_unstale":"2024-09-18T07:49:38.243461+0000","last_undegraded":"2024-09-18T07:49:38.243461+0000","last_fullsized":"2024-09-18T07:49:38.243461+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:52.070602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.070602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":781,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172139+0000","last_change":"2024-09-18T07:46:13.660302+0000","last_active":"2024-09-18T07:49:41.172139+0000","last_peered":"2024-09-18T07:49:41.172139+0000","last_clean":"2024-09-18T07:49:41.172139+0000","last_became_active":"2024-09-18T07:44:43.778318+0000","last_became_peered":"2024-09-18T07:44:43.778318+0000","last_unstale":"2024-09-18T07:49:41.172139+0000","last_undegraded":"2024-09-18T07:49:41.172139+0000","last_fullsized":"2024-09-18T07:49:41.172139+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:13.660254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:13.660254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1004,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.198230+0000","last_change":"2024-09-18T07:49:24.443767+0000","last_active":"2024-09-18T07:49:41.198230+0000","last_peered":"2024-09-18T07:49:41.198230+0000","last_clean":"2024-09-18T07:49:41.198230+0000","last_became_active":"2024-09-18T07:49:24.443399+0000","last_became_peered":"2024-09-18T07:49:24.443399+0000","last_unstale":"2024-09-18T07:49:41.198230+0000","last_undegraded":"2024-09-18T07:49:41.198230+0000","last_fullsized":"2024-09-18T07:49:41.198230+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:33.311735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:33.311735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:02.660401+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"323'8","reported_seq":683,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.173753+0000","last_change":"2024-09-18T07:48:32.275394+0000","last_active":"2024-09-18T07:49:41.173753+0000","last_peered":"2024-09-18T07:49:41.173753+0000","last_clean":"2024-09-18T07:49:41.173753+0000","last_became_active":"2024-09-18T07:48:32.256691+0000","last_became_peered":"2024-09-18T07:48:32.256691+0000","last_unstale":"2024-09-18T07:49:41.173753+0000","last_undegraded":"2024-09-18T07:49:41.173753+0000","last_fullsized":"2024-09-18T07:49:41.173753+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'8","last_scrub_stamp":"2024-09-18T07:47:46.405595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:46.405595+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":162,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243597+0000","last_change":"2024-09-18T07:48:32.266044+0000","last_active":"2024-09-18T07:49:38.243597+0000","last_peered":"2024-09-18T07:49:38.243597+0000","last_clean":"2024-09-18T07:49:38.243597+0000","last_became_active":"2024-09-18T07:48:32.265622+0000","last_became_peered":"2024-09-18T07:48:32.265622+0000","last_unstale":"2024-09-18T07:49:38.243597+0000","last_undegraded":"2024-09-18T07:49:38.243597+0000","last_fullsized":"2024-09-18T07:49:38.243597+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:04.066418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:04.066418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":619,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.173585+0000","last_change":"2024-09-18T07:47:55.363130+0000","last_active":"2024-09-18T07:49:41.173585+0000","last_peered":"2024-09-18T07:49:41.173585+0000","last_clean":"2024-09-18T07:49:41.173585+0000","last_became_active":"2024-09-18T07:44:43.792641+0000","last_became_peered":"2024-09-18T07:44:43.792641+0000","last_unstale":"2024-09-18T07:49:41.173585+0000","last_undegraded":"2024-09-18T07:49:41.173585+0000","last_fullsized":"2024-09-18T07:49:41.173585+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:55.363087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:55.363087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":759,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171909+0000","last_change":"2024-09-18T07:44:43.783022+0000","last_active":"2024-09-18T07:49:41.171909+0000","last_peered":"2024-09-18T07:49:41.171909+0000","last_clean":"2024-09-18T07:49:41.171909+0000","last_became_active":"2024-09-18T07:44:43.782378+0000","last_became_peered":"2024-09-18T07:44:43.782378+0000","last_unstale":"2024-09-18T07:49:41.171909+0000","last_undegraded":"2024-09-18T07:49:41.171909+0000","last_fullsized":"2024-09-18T07:49:41.171909+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:38.492269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.492269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:48.284932+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":561,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-09-18T07:49:38.243533+0000","last_change":"2024-09-18T07:49:17.654443+0000","last_active":"2024-09-18T07:49:38.243533+0000","last_peered":"2024-09-18T07:49:38.243533+0000","last_clean":"2024-09-18T07:49:38.243533+0000","last_became_active":"2024-09-18T07:47:20.138598+0000","last_became_peered":"2024-09-18T07:47:20.138598+0000","last_unstale":"2024-09-18T07:49:38.243533+0000","last_undegraded":"2024-09-18T07:49:38.243533+0000","last_fullsized":"2024-09-18T07:49:38.243533+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:17.654398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:17.654398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:37.045609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":94,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.196883+0000","last_change":"2024-09-18T07:49:32.691139+0000","last_active":"2024-09-18T07:49:41.196883+0000","last_peered":"2024-09-18T07:49:41.196883+0000","last_clean":"2024-09-18T07:49:41.196883+0000","last_became_active":"2024-09-18T07:49:32.685882+0000","last_became_peered":"2024-09-18T07:49:32.685882+0000","last_unstale":"2024-09-18T07:49:41.196883+0000","last_undegraded":"2024-09-18T07:49:41.196883+0000","last_fullsized":"2024-09-18T07:49:41.196883+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:41.464337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:41.464337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:38.480805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"323'19","reported_seq":687,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.174284+0000","last_change":"2024-09-18T07:47:47.469967+0000","last_active":"2024-09-18T07:49:41.174284+0000","last_peered":"2024-09-18T07:49:41.174284+0000","last_clean":"2024-09-18T07:49:41.174284+0000","last_became_active":"2024-09-18T07:47:20.142007+0000","last_became_peered":"2024-09-18T07:47:20.142007+0000","last_unstale":"2024-09-18T07:49:41.174284+0000","last_undegraded":"2024-09-18T07:49:41.174284+0000","last_fullsized":"2024-09-18T07:49:41.174284+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:47:47.469902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:47.469902+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":521,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172374+0000","last_change":"2024-09-18T07:48:28.910567+0000","last_active":"2024-09-18T07:49:41.172374+0000","last_peered":"2024-09-18T07:49:41.172374+0000","last_clean":"2024-09-18T07:49:41.172374+0000","last_became_active":"2024-09-18T07:47:20.139668+0000","last_became_peered":"2024-09-18T07:47:20.139668+0000","last_unstale":"2024-09-18T07:49:41.172374+0000","last_undegraded":"2024-09-18T07:49:41.172374+0000","last_fullsized":"2024-09-18T07:49:41.172374+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:28.910419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:28.910419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:54.424567+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3949723672,"num_objects":60269,"num_object_clones":0,"num_object_copies":180807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3659,"num_objects_unfound":0,"num_objects_dirty":60269,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164035,"num_write_kb":7177665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44482,"num_bytes_recovered":2914975744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13699198976,"data_stored":13699186760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72501,"ondisk_log_size":72501,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1096,"num_read_kb":853,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":565,"num_bytes_recovered":12146,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":479232,"data_stored":12674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},"log_size":690,"ondisk_log_size":690,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2652,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":5878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":882,"num_read_kb":877,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201509,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2388540,"kb_used_data":1707620,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":91356612,"statfs":{"total":95995035648,"available":93549170688,"internally_reserved":0,"allocated":1748602880,"data_stored":1748200488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":697235914},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463147,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2892428,"kb_used_data":2175412,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":90852724,"statfs":{"total":95995035648,"available":93033189376,"internally_reserved":0,"allocated":2227621888,"data_stored":2227203219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":734198214},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":460,"seq":1975684956191,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1338256,"kb_used_data":1223872,"kb_used_omap":6,"kb_used_meta":114361,"kb_avail":92406896,"statfs":{"total":95995035648,"available":94624661504,"internally_reserved":0,"allocated":1253244928,"data_stored":1252865826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6877,"internal_metadata":117105955},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":395,"seq":1696512081981,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1735152,"kb_used_data":1225756,"kb_used_omap":4,"kb_used_meta":509371,"kb_avail":92010000,"statfs":{"total":95995035648,"available":94218240000,"internally_reserved":0,"allocated":1255174144,"data_stored":1254772819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4447,"internal_metadata":521596577},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669149947,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411585,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2716712,"kb_used_data":1959440,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":91028440,"statfs":{"total":95995035648,"available":93213122560,"internally_reserved":0,"allocated":2006466560,"data_stored":2006078510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575112,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2985512,"kb_used_data":2156048,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":90759640,"statfs":{"total":95995035648,"available":92937871360,"internally_reserved":0,"allocated":2207793152,"data_stored":2207347384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":849344965},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738637,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2300096,"kb_used_data":1412328,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":91445056,"statfs":{"total":95995035648,"available":93639737344,"internally_reserved":0,"allocated":1446223872,"data_stored":1445818399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1416429568,"data_stored":1416429568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2206728192,"data_stored":2206728192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1975783424,"data_stored":1975779352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1902575616,"data_stored":1902575616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1245974528,"data_stored":1245970456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1016463360,"data_stored":1016463360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2207780864,"data_stored":2207776792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1727463424,"data_stored":1727463424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:49:47.207 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-18T07:49:47.207 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:49:47.208 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-18T07:49:47.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:49:47.218 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:49:47.230 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:47.253 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:47.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:47.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:47.355 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:47.383 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:47.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:47.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:49:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:49:47.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[29408]: from='client.16839 v1:172.21.15.137:0/1094771575' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:49:47.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[29408]: 4.b scrub starts 2024-09-18T07:49:47.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[29408]: 3.11 scrub starts 2024-09-18T07:49:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[29408]: 3.11 scrub ok 2024-09-18T07:49:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[29408]: 4.b scrub ok 2024-09-18T07:49:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[29408]: pgmap v1008: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 667 B/s rd, 0 B/s wr, 634 op/s; 3659/180375 objects misplaced (2.029%); 7.7 MiB/s, 123 objects/s recovering 2024-09-18T07:49:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[29408]: 11.9 scrub starts 2024-09-18T07:49:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/430761052' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:47.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[29408]: from='client.36653 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[39175]: from='client.16839 v1:172.21.15.137:0/1094771575' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:49:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[39175]: 4.b scrub starts 2024-09-18T07:49:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[39175]: 3.11 scrub starts 2024-09-18T07:49:47.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[39175]: 3.11 scrub ok 2024-09-18T07:49:47.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[39175]: 4.b scrub ok 2024-09-18T07:49:47.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[39175]: pgmap v1008: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 667 B/s rd, 0 B/s wr, 634 op/s; 3659/180375 objects misplaced (2.029%); 7.7 MiB/s, 123 objects/s recovering 2024-09-18T07:49:47.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[39175]: 11.9 scrub starts 2024-09-18T07:49:47.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/430761052' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:47.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:47 smithi137 ceph-mon[39175]: from='client.36653 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:47 smithi203 ceph-mon[32237]: from='client.16839 v1:172.21.15.137:0/1094771575' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:49:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:47 smithi203 ceph-mon[32237]: 4.b scrub starts 2024-09-18T07:49:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:47 smithi203 ceph-mon[32237]: 3.11 scrub starts 2024-09-18T07:49:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:47 smithi203 ceph-mon[32237]: 3.11 scrub ok 2024-09-18T07:49:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:47 smithi203 ceph-mon[32237]: 4.b scrub ok 2024-09-18T07:49:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:47 smithi203 ceph-mon[32237]: pgmap v1008: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 667 B/s rd, 0 B/s wr, 634 op/s; 3659/180375 objects misplaced (2.029%); 7.7 MiB/s, 123 objects/s recovering 2024-09-18T07:49:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:47 smithi203 ceph-mon[32237]: 11.9 scrub starts 2024-09-18T07:49:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:47 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/430761052' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:47.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:47 smithi203 ceph-mon[32237]: from='client.36653 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:47.486 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:47.518 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:47.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:47.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:47.620 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:47.656 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:47.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:47.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:47.757 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:47.790 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:47.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:47.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:47.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:49:47.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:47.943 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:47.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:47.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:48.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:48.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:48.072 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:48.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:48.097 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:48.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:48.205 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:48.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:48.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:48.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:48.339 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:48.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:48.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:48.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:48.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:49:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:49:48.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:48 smithi203 ceph-mon[32237]: 3.1f scrub starts 2024-09-18T07:49:48.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:48 smithi203 ceph-mon[32237]: 3.1f scrub ok 2024-09-18T07:49:48.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:48 smithi203 ceph-mon[32237]: 11.9 scrub ok 2024-09-18T07:49:48.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:48 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:49:48.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:48 smithi203 ceph-mon[32237]: from='client.36653 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:49:48.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:48 smithi203 ceph-mon[32237]: osdmap e529: 8 total, 8 up, 7 in 2024-09-18T07:49:48.467 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:48.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:48.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:48.569 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:48.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:48 smithi137 ceph-mon[39175]: 3.1f scrub starts 2024-09-18T07:49:48.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:48 smithi137 ceph-mon[39175]: 3.1f scrub ok 2024-09-18T07:49:48.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:48 smithi137 ceph-mon[39175]: 11.9 scrub ok 2024-09-18T07:49:48.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:48 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:49:48.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:48 smithi137 ceph-mon[39175]: from='client.36653 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:49:48.585 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:48 smithi137 ceph-mon[39175]: osdmap e529: 8 total, 8 up, 7 in 2024-09-18T07:49:48.585 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:48 smithi137 ceph-mon[29408]: 3.1f scrub starts 2024-09-18T07:49:48.585 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:48 smithi137 ceph-mon[29408]: 3.1f scrub ok 2024-09-18T07:49:48.585 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:48 smithi137 ceph-mon[29408]: 11.9 scrub ok 2024-09-18T07:49:48.585 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:48 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:49:48.585 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:48 smithi137 ceph-mon[29408]: from='client.36653 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:49:48.585 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:48 smithi137 ceph-mon[29408]: osdmap e529: 8 total, 8 up, 7 in 2024-09-18T07:49:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:48.610 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:48.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:48.673 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:49:48.674 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:49:48.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:48.751 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:48.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:48.852 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:48.880 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:48.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:48.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:48.981 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:49.001 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:49:49.003 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:49:49.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:49.005 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:49.006 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:49.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:49.107 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:49.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:49.146 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:49.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:49.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:49.290 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:49.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:49.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:49.392 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:49.426 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:49.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:49.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:49.527 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:49.548 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:49.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:49.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:49.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:49.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:49 smithi137 ceph-mon[29408]: 1.9 scrub starts 2024-09-18T07:49:49.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:49 smithi137 ceph-mon[29408]: 1.9 scrub ok 2024-09-18T07:49:49.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:49 smithi137 ceph-mon[29408]: pgmap v1010: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.40k op/s; 3275/161595 objects misplaced (2.027%); 7.7 MiB/s, 123 objects/s recovering 2024-09-18T07:49:49.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:49 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:49.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:49 smithi137 ceph-mon[39175]: 1.9 scrub starts 2024-09-18T07:49:49.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:49 smithi137 ceph-mon[39175]: 1.9 scrub ok 2024-09-18T07:49:49.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:49 smithi137 ceph-mon[39175]: pgmap v1010: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.40k op/s; 3275/161595 objects misplaced (2.027%); 7.7 MiB/s, 123 objects/s recovering 2024-09-18T07:49:49.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:49 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:49.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:49.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:49.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:49.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:49 smithi203 ceph-mon[32237]: 1.9 scrub starts 2024-09-18T07:49:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:49 smithi203 ceph-mon[32237]: 1.9 scrub ok 2024-09-18T07:49:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:49 smithi203 ceph-mon[32237]: pgmap v1010: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.40k op/s; 3275/161595 objects misplaced (2.027%); 7.7 MiB/s, 123 objects/s recovering 2024-09-18T07:49:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:49 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:49.790 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:49.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:49.824 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:49.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:49.859 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":1010,"stamp":"2024-09-18T07:49:47.886190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3517585971,"num_objects":53865,"num_object_clones":0,"num_object_copies":161595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3275,"num_objects_unfound":0,"num_objects_dirty":53865,"num_whiteouts":0,"num_read":1991,"num_read_kb":1743,"num_write":171360,"num_write_kb":7177710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45083,"num_bytes_recovered":2915517604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74183,"ondisk_log_size":74183,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":480,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":15200100,"kb_used_data":10724460,"kb_used_omap":18,"kb_used_meta":4475245,"kb_avail":641015964,"statfs":{"total":671965249536,"available":656400347136,"internally_reserved":0,"allocated":10981847040,"data_stored":10978984668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19293,"internal_metadata":4582651043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-707723264,"num_objects":-10799,"num_object_clones":0,"num_object_copies":-32397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1608,"num_objects_unfound":0,"num_objects_dirty":-10799,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10799,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":951,"num_bytes_recovered":62324736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.721303"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":1010,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.197691+0000","last_change":"2024-09-18T07:48:28.181443+0000","last_active":"2024-09-18T07:49:41.197691+0000","last_peered":"2024-09-18T07:49:41.197691+0000","last_clean":"2024-09-18T07:49:41.197691+0000","last_became_active":"2024-09-18T07:47:11.200534+0000","last_became_peered":"2024-09-18T07:47:11.200534+0000","last_unstale":"2024-09-18T07:49:41.197691+0000","last_undegraded":"2024-09-18T07:49:41.197691+0000","last_fullsized":"2024-09-18T07:49:41.197691+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:28.181383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:28.181383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:46.295639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"323'15","reported_seq":966,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169095+0000","last_change":"2024-09-18T07:47:20.194767+0000","last_active":"2024-09-18T07:49:41.169095+0000","last_peered":"2024-09-18T07:49:41.169095+0000","last_clean":"2024-09-18T07:49:41.169095+0000","last_became_active":"2024-09-18T07:47:20.136254+0000","last_became_peered":"2024-09-18T07:47:20.136254+0000","last_unstale":"2024-09-18T07:49:41.169095+0000","last_undegraded":"2024-09-18T07:49:41.169095+0000","last_fullsized":"2024-09-18T07:49:41.169095+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:47:09.864550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.864550+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":799,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.205008+0000","last_change":"2024-09-18T07:48:05.093858+0000","last_active":"2024-09-18T07:49:41.205008+0000","last_peered":"2024-09-18T07:49:41.205008+0000","last_clean":"2024-09-18T07:49:41.205008+0000","last_became_active":"2024-09-18T07:47:20.130091+0000","last_became_peered":"2024-09-18T07:47:20.130091+0000","last_unstale":"2024-09-18T07:49:41.205008+0000","last_undegraded":"2024-09-18T07:49:41.205008+0000","last_fullsized":"2024-09-18T07:49:41.205008+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:05.093789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:05.093789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":859,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169516+0000","last_change":"2024-09-18T07:49:32.689133+0000","last_active":"2024-09-18T07:49:41.169516+0000","last_peered":"2024-09-18T07:49:41.169516+0000","last_clean":"2024-09-18T07:49:41.169516+0000","last_became_active":"2024-09-18T07:49:32.688553+0000","last_became_peered":"2024-09-18T07:49:32.688553+0000","last_unstale":"2024-09-18T07:49:41.169516+0000","last_undegraded":"2024-09-18T07:49:41.169516+0000","last_fullsized":"2024-09-18T07:49:41.169516+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:14.884604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:14.884604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":465,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172047+0000","last_change":"2024-09-18T07:48:32.256620+0000","last_active":"2024-09-18T07:49:41.172047+0000","last_peered":"2024-09-18T07:49:41.172047+0000","last_clean":"2024-09-18T07:49:41.172047+0000","last_became_active":"2024-09-18T07:48:32.255219+0000","last_became_peered":"2024-09-18T07:48:32.255219+0000","last_unstale":"2024-09-18T07:49:41.172047+0000","last_undegraded":"2024-09-18T07:49:41.172047+0000","last_fullsized":"2024-09-18T07:49:41.172047+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:11.545358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:11.545358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":453,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.265322+0000","last_change":"2024-09-18T07:48:32.257934+0000","last_active":"2024-09-18T07:49:47.265322+0000","last_peered":"2024-09-18T07:49:47.265322+0000","last_clean":"2024-09-18T07:49:47.265322+0000","last_became_active":"2024-09-18T07:48:32.257676+0000","last_became_peered":"2024-09-18T07:48:32.257676+0000","last_unstale":"2024-09-18T07:49:47.265322+0000","last_undegraded":"2024-09-18T07:49:47.265322+0000","last_fullsized":"2024-09-18T07:49:47.265322+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:57.674688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:57.674688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"323'27","reported_seq":574,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169709+0000","last_change":"2024-09-18T07:48:32.299123+0000","last_active":"2024-09-18T07:49:41.169709+0000","last_peered":"2024-09-18T07:49:41.169709+0000","last_clean":"2024-09-18T07:49:41.169709+0000","last_became_active":"2024-09-18T07:48:32.261006+0000","last_became_peered":"2024-09-18T07:48:32.261006+0000","last_unstale":"2024-09-18T07:49:41.169709+0000","last_undegraded":"2024-09-18T07:49:41.169709+0000","last_fullsized":"2024-09-18T07:49:41.169709+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:46:54.956330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:54.956330+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":915,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169029+0000","last_change":"2024-09-18T07:47:52.060299+0000","last_active":"2024-09-18T07:49:41.169029+0000","last_peered":"2024-09-18T07:49:41.169029+0000","last_clean":"2024-09-18T07:49:41.169029+0000","last_became_active":"2024-09-18T07:47:20.136358+0000","last_became_peered":"2024-09-18T07:47:20.136358+0000","last_unstale":"2024-09-18T07:49:41.169029+0000","last_undegraded":"2024-09-18T07:49:41.169029+0000","last_fullsized":"2024-09-18T07:49:41.169029+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:52.060246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.060246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":914,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.206768+0000","last_change":"2024-09-18T07:48:34.470535+0000","last_active":"2024-09-18T07:49:41.206768+0000","last_peered":"2024-09-18T07:49:41.206768+0000","last_clean":"2024-09-18T07:49:41.206768+0000","last_became_active":"2024-09-18T07:47:11.200300+0000","last_became_peered":"2024-09-18T07:47:11.200300+0000","last_unstale":"2024-09-18T07:49:41.206768+0000","last_undegraded":"2024-09-18T07:49:41.206768+0000","last_fullsized":"2024-09-18T07:49:41.206768+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:34.470475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:34.470475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:59.691917+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"323'15","reported_seq":929,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169115+0000","last_change":"2024-09-18T07:48:32.305165+0000","last_active":"2024-09-18T07:49:41.169115+0000","last_peered":"2024-09-18T07:49:41.169115+0000","last_clean":"2024-09-18T07:49:41.169115+0000","last_became_active":"2024-09-18T07:48:32.264535+0000","last_became_peered":"2024-09-18T07:48:32.264535+0000","last_unstale":"2024-09-18T07:49:41.169115+0000","last_undegraded":"2024-09-18T07:49:41.169115+0000","last_fullsized":"2024-09-18T07:49:41.169115+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:47:47.157636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:47.157636+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":851,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.205808+0000","last_change":"2024-09-18T07:49:14.687448+0000","last_active":"2024-09-18T07:49:41.205808+0000","last_peered":"2024-09-18T07:49:41.205808+0000","last_clean":"2024-09-18T07:49:41.205808+0000","last_became_active":"2024-09-18T07:47:11.191975+0000","last_became_peered":"2024-09-18T07:47:11.191975+0000","last_unstale":"2024-09-18T07:49:41.205808+0000","last_undegraded":"2024-09-18T07:49:41.205808+0000","last_fullsized":"2024-09-18T07:49:41.205808+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:14.687386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:49:14.687386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:39.954301+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":967,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171449+0000","last_change":"2024-09-18T07:47:55.019436+0000","last_active":"2024-09-18T07:49:41.171449+0000","last_peered":"2024-09-18T07:49:41.171449+0000","last_clean":"2024-09-18T07:49:41.171449+0000","last_became_active":"2024-09-18T07:47:53.329870+0000","last_became_peered":"2024-09-18T07:47:53.329870+0000","last_unstale":"2024-09-18T07:49:41.171449+0000","last_undegraded":"2024-09-18T07:49:41.171449+0000","last_fullsized":"2024-09-18T07:49:41.171449+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:55.019244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:55.019244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":873,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.617104+0000","last_change":"2024-09-18T07:49:41.617104+0000","last_active":"2024-09-18T07:49:41.617104+0000","last_peered":"2024-09-18T07:49:41.617104+0000","last_clean":"2024-09-18T07:49:41.617104+0000","last_became_active":"2024-09-18T07:47:20.142404+0000","last_became_peered":"2024-09-18T07:47:20.142404+0000","last_unstale":"2024-09-18T07:49:41.617104+0000","last_undegraded":"2024-09-18T07:49:41.617104+0000","last_fullsized":"2024-09-18T07:49:41.617104+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:41.617037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:41.617037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:55.639234+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"323'7","reported_seq":1005,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169426+0000","last_change":"2024-09-18T07:49:32.699096+0000","last_active":"2024-09-18T07:49:41.169426+0000","last_peered":"2024-09-18T07:49:41.169426+0000","last_clean":"2024-09-18T07:49:41.169426+0000","last_became_active":"2024-09-18T07:49:32.689928+0000","last_became_peered":"2024-09-18T07:49:32.689928+0000","last_unstale":"2024-09-18T07:49:41.169426+0000","last_undegraded":"2024-09-18T07:49:41.169426+0000","last_fullsized":"2024-09-18T07:49:41.169426+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'7","last_scrub_stamp":"2024-09-18T07:42:46.496916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:46.496916+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:40.792920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":780,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171738+0000","last_change":"2024-09-18T07:47:20.143380+0000","last_active":"2024-09-18T07:49:41.171738+0000","last_peered":"2024-09-18T07:49:41.171738+0000","last_clean":"2024-09-18T07:49:41.171738+0000","last_became_active":"2024-09-18T07:47:20.142891+0000","last_became_peered":"2024-09-18T07:47:20.142891+0000","last_unstale":"2024-09-18T07:49:41.171738+0000","last_undegraded":"2024-09-18T07:49:41.171738+0000","last_fullsized":"2024-09-18T07:49:41.171738+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:53.754622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:53.754622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":908,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.205254+0000","last_change":"2024-09-18T07:48:32.255121+0000","last_active":"2024-09-18T07:49:41.205254+0000","last_peered":"2024-09-18T07:49:41.205254+0000","last_clean":"2024-09-18T07:49:41.205254+0000","last_became_active":"2024-09-18T07:48:32.254682+0000","last_became_peered":"2024-09-18T07:48:32.254682+0000","last_unstale":"2024-09-18T07:49:41.205254+0000","last_undegraded":"2024-09-18T07:49:41.205254+0000","last_fullsized":"2024-09-18T07:49:41.205254+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:48.995794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:48.995794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":939,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169968+0000","last_change":"2024-09-18T07:48:31.926895+0000","last_active":"2024-09-18T07:49:41.169968+0000","last_peered":"2024-09-18T07:49:41.169968+0000","last_clean":"2024-09-18T07:49:41.169968+0000","last_became_active":"2024-09-18T07:47:11.187916+0000","last_became_peered":"2024-09-18T07:47:11.187916+0000","last_unstale":"2024-09-18T07:49:41.169968+0000","last_undegraded":"2024-09-18T07:49:41.169968+0000","last_fullsized":"2024-09-18T07:49:41.169968+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:31.926847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:31.926847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:47.604755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1014,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.198156+0000","last_change":"2024-09-18T07:47:49.150861+0000","last_active":"2024-09-18T07:49:41.198156+0000","last_peered":"2024-09-18T07:49:41.198156+0000","last_clean":"2024-09-18T07:49:41.198156+0000","last_became_active":"2024-09-18T07:44:43.773339+0000","last_became_peered":"2024-09-18T07:44:43.773339+0000","last_unstale":"2024-09-18T07:49:41.198156+0000","last_undegraded":"2024-09-18T07:49:41.198156+0000","last_fullsized":"2024-09-18T07:49:41.198156+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:49.150795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:49.150795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:37.524851+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"323'27","reported_seq":1140,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:43.979210+0000","last_change":"2024-09-18T07:49:43.979210+0000","last_active":"2024-09-18T07:49:43.979210+0000","last_peered":"2024-09-18T07:49:43.979210+0000","last_clean":"2024-09-18T07:49:43.979210+0000","last_became_active":"2024-09-18T07:47:20.140525+0000","last_became_peered":"2024-09-18T07:47:20.140525+0000","last_unstale":"2024-09-18T07:49:43.979210+0000","last_undegraded":"2024-09-18T07:49:43.979210+0000","last_fullsized":"2024-09-18T07:49:43.979210+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:49:43.979116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:49:43.979116+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:07.138647+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":569,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169770+0000","last_change":"2024-09-18T07:47:20.133202+0000","last_active":"2024-09-18T07:49:41.169770+0000","last_peered":"2024-09-18T07:49:41.169770+0000","last_clean":"2024-09-18T07:49:41.169770+0000","last_became_active":"2024-09-18T07:47:20.131623+0000","last_became_peered":"2024-09-18T07:47:20.131623+0000","last_unstale":"2024-09-18T07:49:41.169770+0000","last_undegraded":"2024-09-18T07:49:41.169770+0000","last_fullsized":"2024-09-18T07:49:41.169770+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:18.008882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:18.008882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":877,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.268173+0000","last_change":"2024-09-18T07:49:32.685656+0000","last_active":"2024-09-18T07:49:47.268173+0000","last_peered":"2024-09-18T07:49:47.268173+0000","last_clean":"2024-09-18T07:49:47.268173+0000","last_became_active":"2024-09-18T07:49:32.685347+0000","last_became_peered":"2024-09-18T07:49:32.685347+0000","last_unstale":"2024-09-18T07:49:47.268173+0000","last_undegraded":"2024-09-18T07:49:47.268173+0000","last_fullsized":"2024-09-18T07:49:47.268173+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:44.397262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:44.397262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":931,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169409+0000","last_change":"2024-09-18T07:47:56.099010+0000","last_active":"2024-09-18T07:49:41.169409+0000","last_peered":"2024-09-18T07:49:41.169409+0000","last_clean":"2024-09-18T07:49:41.169409+0000","last_became_active":"2024-09-18T07:47:20.144620+0000","last_became_peered":"2024-09-18T07:47:20.144620+0000","last_unstale":"2024-09-18T07:49:41.169409+0000","last_undegraded":"2024-09-18T07:49:41.169409+0000","last_fullsized":"2024-09-18T07:49:41.169409+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:56.098927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.098927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"323'18","reported_seq":297,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169747+0000","last_change":"2024-09-18T07:48:32.293504+0000","last_active":"2024-09-18T07:49:41.169747+0000","last_peered":"2024-09-18T07:49:41.169747+0000","last_clean":"2024-09-18T07:49:41.169747+0000","last_became_active":"2024-09-18T07:48:32.266466+0000","last_became_peered":"2024-09-18T07:48:32.266466+0000","last_unstale":"2024-09-18T07:49:41.169747+0000","last_undegraded":"2024-09-18T07:49:41.169747+0000","last_fullsized":"2024-09-18T07:49:41.169747+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:47:53.905251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.905251+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":852,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169695+0000","last_change":"2024-09-18T07:48:32.264906+0000","last_active":"2024-09-18T07:49:41.169695+0000","last_peered":"2024-09-18T07:49:41.169695+0000","last_clean":"2024-09-18T07:49:41.169695+0000","last_became_active":"2024-09-18T07:48:32.264578+0000","last_became_peered":"2024-09-18T07:48:32.264578+0000","last_unstale":"2024-09-18T07:49:41.169695+0000","last_undegraded":"2024-09-18T07:49:41.169695+0000","last_fullsized":"2024-09-18T07:49:41.169695+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.930435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.930435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:20.674047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":778,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172116+0000","last_change":"2024-09-18T07:47:53.034043+0000","last_active":"2024-09-18T07:49:41.172116+0000","last_peered":"2024-09-18T07:49:41.172116+0000","last_clean":"2024-09-18T07:49:41.172116+0000","last_became_active":"2024-09-18T07:47:11.194447+0000","last_became_peered":"2024-09-18T07:47:11.194447+0000","last_unstale":"2024-09-18T07:49:41.172116+0000","last_undegraded":"2024-09-18T07:49:41.172116+0000","last_fullsized":"2024-09-18T07:49:41.172116+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:47:53.033975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.033975+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":82,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169072+0000","last_change":"2024-09-18T07:48:32.274975+0000","last_active":"2024-09-18T07:49:41.169072+0000","last_peered":"2024-09-18T07:49:41.169072+0000","last_clean":"2024-09-18T07:49:41.169072+0000","last_became_active":"2024-09-18T07:48:32.262299+0000","last_became_peered":"2024-09-18T07:48:32.262299+0000","last_unstale":"2024-09-18T07:49:41.169072+0000","last_undegraded":"2024-09-18T07:49:41.169072+0000","last_fullsized":"2024-09-18T07:49:41.169072+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:09.496455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.496455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"323'20","reported_seq":755,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172571+0000","last_change":"2024-09-18T07:48:19.962428+0000","last_active":"2024-09-18T07:49:41.172571+0000","last_peered":"2024-09-18T07:49:41.172571+0000","last_clean":"2024-09-18T07:49:41.172571+0000","last_became_active":"2024-09-18T07:48:10.157273+0000","last_became_peered":"2024-09-18T07:48:10.157273+0000","last_unstale":"2024-09-18T07:49:41.172571+0000","last_undegraded":"2024-09-18T07:49:41.172571+0000","last_fullsized":"2024-09-18T07:49:41.172571+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:48:19.962346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:19.962346+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"398'14","reported_seq":788,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.268254+0000","last_change":"2024-09-18T07:49:41.853017+0000","last_active":"2024-09-18T07:49:47.268254+0000","last_peered":"2024-09-18T07:49:47.268254+0000","last_clean":"2024-09-18T07:49:47.268254+0000","last_became_active":"2024-09-18T07:44:43.786594+0000","last_became_peered":"2024-09-18T07:44:43.786594+0000","last_unstale":"2024-09-18T07:49:47.268254+0000","last_undegraded":"2024-09-18T07:49:47.268254+0000","last_fullsized":"2024-09-18T07:49:47.268254+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:49:41.852939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:49:41.852939+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:46.677099+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"528'10838","reported_seq":21619,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:46.870776+0000","last_change":"2024-09-18T07:49:41.190449+0000","last_active":"2024-09-18T07:49:46.870776+0000","last_peered":"2024-09-18T07:49:46.870776+0000","last_clean":"2024-09-18T07:49:46.870776+0000","last_became_active":"2024-09-18T07:49:41.174729+0000","last_became_peered":"2024-09-18T07:49:41.174729+0000","last_unstale":"2024-09-18T07:49:46.870776+0000","last_undegraded":"2024-09-18T07:49:46.870776+0000","last_fullsized":"2024-09-18T07:49:46.870776+0000","mapping_epoch":527,"log_start":"498'6436","ondisk_log_start":"498'6436","created":459,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4402,"log_dups_size":0,"ondisk_log_size":4402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":3456,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10838,"num_write_kb":457408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5294,"num_bytes_recovered":346947584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":418,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.206876+0000","last_change":"2024-09-18T07:47:13.167529+0000","last_active":"2024-09-18T07:49:41.206876+0000","last_peered":"2024-09-18T07:49:41.206876+0000","last_clean":"2024-09-18T07:49:41.206876+0000","last_became_active":"2024-09-18T07:47:11.203901+0000","last_became_peered":"2024-09-18T07:47:11.203901+0000","last_unstale":"2024-09-18T07:49:41.206876+0000","last_undegraded":"2024-09-18T07:49:41.206876+0000","last_fullsized":"2024-09-18T07:49:41.206876+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:47:13.167470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:47:13.167470+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:46.555145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"323'27","reported_seq":393,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170051+0000","last_change":"2024-09-18T07:48:32.312362+0000","last_active":"2024-09-18T07:49:41.170051+0000","last_peered":"2024-09-18T07:49:41.170051+0000","last_clean":"2024-09-18T07:49:41.170051+0000","last_became_active":"2024-09-18T07:48:32.268036+0000","last_became_peered":"2024-09-18T07:48:32.268036+0000","last_unstale":"2024-09-18T07:49:41.170051+0000","last_undegraded":"2024-09-18T07:49:41.170051+0000","last_fullsized":"2024-09-18T07:49:41.170051+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:42:33.391715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:33.391715+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:25.785407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":962,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171782+0000","last_change":"2024-09-18T07:48:32.250884+0000","last_active":"2024-09-18T07:49:41.171782+0000","last_peered":"2024-09-18T07:49:41.171782+0000","last_clean":"2024-09-18T07:49:41.171782+0000","last_became_active":"2024-09-18T07:48:32.250451+0000","last_became_peered":"2024-09-18T07:48:32.250451+0000","last_unstale":"2024-09-18T07:49:41.171782+0000","last_undegraded":"2024-09-18T07:49:41.171782+0000","last_fullsized":"2024-09-18T07:49:41.171782+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:25.143715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:25.143715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"461'12","reported_seq":1083,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170135+0000","last_change":"2024-09-18T07:48:30.963331+0000","last_active":"2024-09-18T07:49:41.170135+0000","last_peered":"2024-09-18T07:49:41.170135+0000","last_clean":"2024-09-18T07:49:41.170135+0000","last_became_active":"2024-09-18T07:47:11.190670+0000","last_became_peered":"2024-09-18T07:47:11.190670+0000","last_unstale":"2024-09-18T07:49:41.170135+0000","last_undegraded":"2024-09-18T07:49:41.170135+0000","last_fullsized":"2024-09-18T07:49:41.170135+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'12","last_scrub_stamp":"2024-09-18T07:48:30.963258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:30.963258+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:56.466762+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":847,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.206986+0000","last_change":"2024-09-18T07:49:32.687377+0000","last_active":"2024-09-18T07:49:41.206986+0000","last_peered":"2024-09-18T07:49:41.206986+0000","last_clean":"2024-09-18T07:49:41.206986+0000","last_became_active":"2024-09-18T07:49:32.687063+0000","last_became_peered":"2024-09-18T07:49:32.687063+0000","last_unstale":"2024-09-18T07:49:41.206986+0000","last_undegraded":"2024-09-18T07:49:41.206986+0000","last_fullsized":"2024-09-18T07:49:41.206986+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:45.670301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:45.670301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:34.512574+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"528'10723","reported_seq":12545,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:46.870717+0000","last_change":"2024-09-18T07:47:37.208096+0000","last_active":"2024-09-18T07:49:46.870717+0000","last_peered":"2024-09-18T07:49:46.870717+0000","last_clean":"2024-09-18T07:49:46.870717+0000","last_became_active":"2024-09-18T07:47:37.204971+0000","last_became_peered":"2024-09-18T07:47:37.204971+0000","last_unstale":"2024-09-18T07:49:46.870717+0000","last_undegraded":"2024-09-18T07:49:46.870717+0000","last_fullsized":"2024-09-18T07:49:46.870717+0000","mapping_epoch":474,"log_start":"499'6341","ondisk_log_start":"499'6341","created":459,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4382,"log_dups_size":0,"ondisk_log_size":4382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221446144,"num_objects":3379,"num_object_clones":0,"num_object_copies":10137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10723,"num_write_kb":451264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":831,"num_bytes_recovered":54460416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":466,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:43.801107+0000","last_change":"2024-09-18T07:49:43.801107+0000","last_active":"2024-09-18T07:49:43.801107+0000","last_peered":"2024-09-18T07:49:43.801107+0000","last_clean":"2024-09-18T07:49:43.801107+0000","last_became_active":"2024-09-18T07:44:43.786396+0000","last_became_peered":"2024-09-18T07:44:43.786396+0000","last_unstale":"2024-09-18T07:49:43.801107+0000","last_undegraded":"2024-09-18T07:49:43.801107+0000","last_fullsized":"2024-09-18T07:49:43.801107+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:49:43.801039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:49:43.801039+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:53.586078+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"323'30","reported_seq":745,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171331+0000","last_change":"2024-09-18T07:48:32.316007+0000","last_active":"2024-09-18T07:49:41.171331+0000","last_peered":"2024-09-18T07:49:41.171331+0000","last_clean":"2024-09-18T07:49:41.171331+0000","last_became_active":"2024-09-18T07:48:32.255887+0000","last_became_peered":"2024-09-18T07:48:32.255887+0000","last_unstale":"2024-09-18T07:49:41.171331+0000","last_undegraded":"2024-09-18T07:49:41.171331+0000","last_fullsized":"2024-09-18T07:49:41.171331+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'30","last_scrub_stamp":"2024-09-18T07:47:56.645991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.645991+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":726,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172605+0000","last_change":"2024-09-18T07:48:32.260193+0000","last_active":"2024-09-18T07:49:41.172605+0000","last_peered":"2024-09-18T07:49:41.172605+0000","last_clean":"2024-09-18T07:49:41.172605+0000","last_became_active":"2024-09-18T07:48:32.259860+0000","last_became_peered":"2024-09-18T07:48:32.259860+0000","last_unstale":"2024-09-18T07:49:41.172605+0000","last_undegraded":"2024-09-18T07:49:41.172605+0000","last_fullsized":"2024-09-18T07:49:41.172605+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:56.077323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.077323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":911,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169030+0000","last_change":"2024-09-18T07:48:32.265565+0000","last_active":"2024-09-18T07:49:41.169030+0000","last_peered":"2024-09-18T07:49:41.169030+0000","last_clean":"2024-09-18T07:49:41.169030+0000","last_became_active":"2024-09-18T07:48:32.265285+0000","last_became_peered":"2024-09-18T07:48:32.265285+0000","last_unstale":"2024-09-18T07:49:41.169030+0000","last_undegraded":"2024-09-18T07:49:41.169030+0000","last_fullsized":"2024-09-18T07:49:41.169030+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:53.056484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.056484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":820,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169208+0000","last_change":"2024-09-18T07:48:32.265789+0000","last_active":"2024-09-18T07:49:41.169208+0000","last_peered":"2024-09-18T07:49:41.169208+0000","last_clean":"2024-09-18T07:49:41.169208+0000","last_became_active":"2024-09-18T07:48:32.264093+0000","last_became_peered":"2024-09-18T07:48:32.264093+0000","last_unstale":"2024-09-18T07:49:41.169208+0000","last_undegraded":"2024-09-18T07:49:41.169208+0000","last_fullsized":"2024-09-18T07:49:41.169208+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:07.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:07.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"528'10687","reported_seq":12555,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:46.474840+0000","last_change":"2024-09-18T07:47:35.123117+0000","last_active":"2024-09-18T07:49:46.474840+0000","last_peered":"2024-09-18T07:49:46.474840+0000","last_clean":"2024-09-18T07:49:46.474840+0000","last_became_active":"2024-09-18T07:47:35.119437+0000","last_became_peered":"2024-09-18T07:47:35.119437+0000","last_unstale":"2024-09-18T07:49:46.474840+0000","last_undegraded":"2024-09-18T07:49:46.474840+0000","last_fullsized":"2024-09-18T07:49:46.474840+0000","mapping_epoch":472,"log_start":"501'6456","ondisk_log_start":"501'6456","created":459,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4231,"log_dups_size":0,"ondisk_log_size":4231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222494720,"num_objects":3395,"num_object_clones":0,"num_object_copies":10185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10687,"num_write_kb":450624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":55902208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":82,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.205403+0000","last_change":"2024-09-18T07:48:32.459297+0000","last_active":"2024-09-18T07:49:41.205403+0000","last_peered":"2024-09-18T07:49:41.205403+0000","last_clean":"2024-09-18T07:49:41.205403+0000","last_became_active":"2024-09-18T07:48:32.271495+0000","last_became_peered":"2024-09-18T07:48:32.271495+0000","last_unstale":"2024-09-18T07:49:41.205403+0000","last_undegraded":"2024-09-18T07:49:41.205403+0000","last_fullsized":"2024-09-18T07:49:41.205403+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:32.459229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:32.459229+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:01.535886+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"323'28","reported_seq":939,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169285+0000","last_change":"2024-09-18T07:48:26.993988+0000","last_active":"2024-09-18T07:49:41.169285+0000","last_peered":"2024-09-18T07:49:41.169285+0000","last_clean":"2024-09-18T07:49:41.169285+0000","last_became_active":"2024-09-18T07:47:11.188002+0000","last_became_peered":"2024-09-18T07:47:11.188002+0000","last_unstale":"2024-09-18T07:49:41.169285+0000","last_undegraded":"2024-09-18T07:49:41.169285+0000","last_fullsized":"2024-09-18T07:49:41.169285+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'28","last_scrub_stamp":"2024-09-18T07:48:26.993871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:26.993871+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:44.242031+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":1016,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.198664+0000","last_change":"2024-09-18T07:47:48.117798+0000","last_active":"2024-09-18T07:49:41.198664+0000","last_peered":"2024-09-18T07:49:41.198664+0000","last_clean":"2024-09-18T07:49:41.198664+0000","last_became_active":"2024-09-18T07:47:20.138745+0000","last_became_peered":"2024-09-18T07:47:20.138745+0000","last_unstale":"2024-09-18T07:49:41.198664+0000","last_undegraded":"2024-09-18T07:49:41.198664+0000","last_fullsized":"2024-09-18T07:49:41.198664+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:48.117629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:48.117629+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"500'13","reported_seq":968,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169039+0000","last_change":"2024-09-18T07:48:32.276078+0000","last_active":"2024-09-18T07:49:41.169039+0000","last_peered":"2024-09-18T07:49:41.169039+0000","last_clean":"2024-09-18T07:49:41.169039+0000","last_became_active":"2024-09-18T07:48:32.254811+0000","last_became_peered":"2024-09-18T07:48:32.254811+0000","last_unstale":"2024-09-18T07:49:41.169039+0000","last_undegraded":"2024-09-18T07:49:41.169039+0000","last_fullsized":"2024-09-18T07:49:41.169039+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'12","last_scrub_stamp":"2024-09-18T07:48:20.859791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:20.859791+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":771,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171905+0000","last_change":"2024-09-18T07:47:42.984418+0000","last_active":"2024-09-18T07:49:41.171905+0000","last_peered":"2024-09-18T07:49:41.171905+0000","last_clean":"2024-09-18T07:49:41.171905+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:49:41.171905+0000","last_undegraded":"2024-09-18T07:49:41.171905+0000","last_fullsized":"2024-09-18T07:49:41.171905+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:42.984298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.984298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"528'11020","reported_seq":11171,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:46.818713+0000","last_change":"2024-09-18T07:47:10.178170+0000","last_active":"2024-09-18T07:49:46.818713+0000","last_peered":"2024-09-18T07:49:46.818713+0000","last_clean":"2024-09-18T07:49:46.818713+0000","last_became_active":"2024-09-18T07:47:10.177901+0000","last_became_peered":"2024-09-18T07:47:10.177901+0000","last_unstale":"2024-09-18T07:49:46.818713+0000","last_undegraded":"2024-09-18T07:49:46.818713+0000","last_fullsized":"2024-09-18T07:49:46.818713+0000","mapping_epoch":459,"log_start":"493'5764","ondisk_log_start":"493'5764","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":5256,"log_dups_size":0,"ondisk_log_size":5256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223805464,"num_objects":3416,"num_object_clones":0,"num_object_copies":10248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11020,"num_write_kb":461889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":722,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.268028+0000","last_change":"2024-09-18T07:48:10.151652+0000","last_active":"2024-09-18T07:49:47.268028+0000","last_peered":"2024-09-18T07:49:47.268028+0000","last_clean":"2024-09-18T07:49:47.268028+0000","last_became_active":"2024-09-18T07:48:10.151343+0000","last_became_peered":"2024-09-18T07:48:10.151343+0000","last_unstale":"2024-09-18T07:49:47.268028+0000","last_undegraded":"2024-09-18T07:49:47.268028+0000","last_fullsized":"2024-09-18T07:49:47.268028+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:02.444272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:02.444272+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"323'20","reported_seq":686,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.205282+0000","last_change":"2024-09-18T07:49:32.698546+0000","last_active":"2024-09-18T07:49:41.205282+0000","last_peered":"2024-09-18T07:49:41.205282+0000","last_clean":"2024-09-18T07:49:41.205282+0000","last_became_active":"2024-09-18T07:49:32.688605+0000","last_became_peered":"2024-09-18T07:49:32.688605+0000","last_unstale":"2024-09-18T07:49:41.205282+0000","last_undegraded":"2024-09-18T07:49:41.205282+0000","last_fullsized":"2024-09-18T07:49:41.205282+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:48:33.642815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:33.642815+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:44.608266+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":897,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.206662+0000","last_change":"2024-09-18T07:48:32.268542+0000","last_active":"2024-09-18T07:49:41.206662+0000","last_peered":"2024-09-18T07:49:41.206662+0000","last_clean":"2024-09-18T07:49:41.206662+0000","last_became_active":"2024-09-18T07:48:32.267802+0000","last_became_peered":"2024-09-18T07:48:32.267802+0000","last_unstale":"2024-09-18T07:49:41.206662+0000","last_undegraded":"2024-09-18T07:49:41.206662+0000","last_fullsized":"2024-09-18T07:49:41.206662+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:19.210077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:19.210077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"461'13","reported_seq":728,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.276705+0000","last_change":"2024-09-18T07:48:24.606091+0000","last_active":"2024-09-18T07:49:47.276705+0000","last_peered":"2024-09-18T07:49:47.276705+0000","last_clean":"2024-09-18T07:49:47.276705+0000","last_became_active":"2024-09-18T07:47:11.189241+0000","last_became_peered":"2024-09-18T07:47:11.189241+0000","last_unstale":"2024-09-18T07:49:47.276705+0000","last_undegraded":"2024-09-18T07:49:47.276705+0000","last_fullsized":"2024-09-18T07:49:47.276705+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'13","last_scrub_stamp":"2024-09-18T07:48:24.605876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:24.605876+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":71,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169084+0000","last_change":"2024-09-18T07:48:32.258814+0000","last_active":"2024-09-18T07:49:41.169084+0000","last_peered":"2024-09-18T07:49:41.169084+0000","last_clean":"2024-09-18T07:49:41.169084+0000","last_became_active":"2024-09-18T07:48:32.258477+0000","last_became_peered":"2024-09-18T07:48:32.258477+0000","last_unstale":"2024-09-18T07:49:41.169084+0000","last_undegraded":"2024-09-18T07:49:41.169084+0000","last_fullsized":"2024-09-18T07:49:41.169084+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:46.172671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:46.172671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.c","version":"528'10501","reported_seq":12258,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:46.836407+0000","last_change":"2024-09-18T07:47:31.131323+0000","last_active":"2024-09-18T07:49:46.836407+0000","last_peered":"2024-09-18T07:49:46.836407+0000","last_clean":"2024-09-18T07:49:46.836407+0000","last_became_active":"2024-09-18T07:47:31.128537+0000","last_became_peered":"2024-09-18T07:47:31.128537+0000","last_unstale":"2024-09-18T07:49:46.836407+0000","last_undegraded":"2024-09-18T07:49:46.836407+0000","last_fullsized":"2024-09-18T07:49:46.836407+0000","mapping_epoch":468,"log_start":"492'5315","ondisk_log_start":"492'5315","created":459,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":5186,"log_dups_size":0,"ondisk_log_size":5186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221315072,"num_objects":3377,"num_object_clones":0,"num_object_copies":10131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10501,"num_write_kb":444096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":799,"num_bytes_recovered":52363264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":730,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.264619+0000","last_change":"2024-09-18T07:48:32.273778+0000","last_active":"2024-09-18T07:49:47.264619+0000","last_peered":"2024-09-18T07:49:47.264619+0000","last_clean":"2024-09-18T07:49:47.264619+0000","last_became_active":"2024-09-18T07:48:32.273227+0000","last_became_peered":"2024-09-18T07:48:32.273227+0000","last_unstale":"2024-09-18T07:49:47.264619+0000","last_undegraded":"2024-09-18T07:49:47.264619+0000","last_fullsized":"2024-09-18T07:49:47.264619+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:47:59.422927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:47:59.422927+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"323'29","reported_seq":813,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.206256+0000","last_change":"2024-09-18T07:48:01.062906+0000","last_active":"2024-09-18T07:49:41.206256+0000","last_peered":"2024-09-18T07:49:41.206256+0000","last_clean":"2024-09-18T07:49:41.206256+0000","last_became_active":"2024-09-18T07:47:11.198126+0000","last_became_peered":"2024-09-18T07:47:11.198126+0000","last_unstale":"2024-09-18T07:49:41.206256+0000","last_undegraded":"2024-09-18T07:49:41.206256+0000","last_fullsized":"2024-09-18T07:49:41.206256+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'29","last_scrub_stamp":"2024-09-18T07:48:01.062822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:01.062822+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":708,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171242+0000","last_change":"2024-09-18T07:47:20.140174+0000","last_active":"2024-09-18T07:49:41.171242+0000","last_peered":"2024-09-18T07:49:41.171242+0000","last_clean":"2024-09-18T07:49:41.171242+0000","last_became_active":"2024-09-18T07:47:20.139806+0000","last_became_peered":"2024-09-18T07:47:20.139806+0000","last_unstale":"2024-09-18T07:49:41.171242+0000","last_undegraded":"2024-09-18T07:49:41.171242+0000","last_fullsized":"2024-09-18T07:49:41.171242+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:43.985969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:43.985969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":637,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.264581+0000","last_change":"2024-09-18T07:47:41.319363+0000","last_active":"2024-09-18T07:49:47.264581+0000","last_peered":"2024-09-18T07:49:47.264581+0000","last_clean":"2024-09-18T07:49:47.264581+0000","last_became_active":"2024-09-18T07:47:11.193892+0000","last_became_peered":"2024-09-18T07:47:11.193892+0000","last_unstale":"2024-09-18T07:49:47.264581+0000","last_undegraded":"2024-09-18T07:49:47.264581+0000","last_fullsized":"2024-09-18T07:49:47.264581+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:41.319318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:41.319318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":640,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.268421+0000","last_change":"2024-09-18T07:49:43.912594+0000","last_active":"2024-09-18T07:49:47.268421+0000","last_peered":"2024-09-18T07:49:47.268421+0000","last_clean":"2024-09-18T07:49:47.268421+0000","last_became_active":"2024-09-18T07:47:20.138390+0000","last_became_peered":"2024-09-18T07:47:20.138390+0000","last_unstale":"2024-09-18T07:49:47.268421+0000","last_undegraded":"2024-09-18T07:49:47.268421+0000","last_fullsized":"2024-09-18T07:49:47.268421+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:43.912475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:43.912475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:51.322173+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.b","version":"528'10663","reported_seq":24689,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:46.471415+0000","last_change":"2024-09-18T07:49:19.420062+0000","last_active":"2024-09-18T07:49:46.471415+0000","last_peered":"2024-09-18T07:49:46.471415+0000","last_clean":"2024-09-18T07:49:46.471415+0000","last_became_active":"2024-09-18T07:49:19.393415+0000","last_became_peered":"2024-09-18T07:49:19.393415+0000","last_unstale":"2024-09-18T07:49:46.471415+0000","last_undegraded":"2024-09-18T07:49:46.471415+0000","last_fullsized":"2024-09-18T07:49:46.471415+0000","mapping_epoch":512,"log_start":"502'6414","ondisk_log_start":"502'6414","created":459,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4249,"log_dups_size":0,"ondisk_log_size":4249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213975040,"num_objects":3265,"num_object_clones":0,"num_object_copies":9795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":445696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6927,"num_bytes_recovered":453967872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":446,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.205341+0000","last_change":"2024-09-18T07:48:59.662366+0000","last_active":"2024-09-18T07:49:41.205341+0000","last_peered":"2024-09-18T07:49:41.205341+0000","last_clean":"2024-09-18T07:49:41.205341+0000","last_became_active":"2024-09-18T07:47:11.195706+0000","last_became_peered":"2024-09-18T07:47:11.195706+0000","last_unstale":"2024-09-18T07:49:41.205341+0000","last_undegraded":"2024-09-18T07:49:41.205341+0000","last_fullsized":"2024-09-18T07:49:41.205341+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:59.662286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:59.662286+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:04.207309+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"519'45","reported_seq":1060,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.586484+0000","last_change":"2024-09-18T07:49:24.465774+0000","last_active":"2024-09-18T07:49:41.586484+0000","last_peered":"2024-09-18T07:49:41.586484+0000","last_clean":"2024-09-18T07:49:41.586484+0000","last_became_active":"2024-09-18T07:49:24.452427+0000","last_became_peered":"2024-09-18T07:49:24.452427+0000","last_unstale":"2024-09-18T07:49:41.586484+0000","last_undegraded":"2024-09-18T07:49:41.586484+0000","last_fullsized":"2024-09-18T07:49:41.586484+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'41","last_scrub_stamp":"2024-09-18T07:46:12.858570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:12.858570+0000","objects_scrubbed":10,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":910,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169490+0000","last_change":"2024-09-18T07:47:51.085464+0000","last_active":"2024-09-18T07:49:41.169490+0000","last_peered":"2024-09-18T07:49:41.169490+0000","last_clean":"2024-09-18T07:49:41.169490+0000","last_became_active":"2024-09-18T07:44:43.780114+0000","last_became_peered":"2024-09-18T07:44:43.780114+0000","last_unstale":"2024-09-18T07:49:41.169490+0000","last_undegraded":"2024-09-18T07:49:41.169490+0000","last_fullsized":"2024-09-18T07:49:41.169490+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:51.085400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:51.085400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":882,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.205373+0000","last_change":"2024-09-18T07:48:06.127800+0000","last_active":"2024-09-18T07:49:41.205373+0000","last_peered":"2024-09-18T07:49:41.205373+0000","last_clean":"2024-09-18T07:49:41.205373+0000","last_became_active":"2024-09-18T07:47:20.144312+0000","last_became_peered":"2024-09-18T07:47:20.144312+0000","last_unstale":"2024-09-18T07:49:41.205373+0000","last_undegraded":"2024-09-18T07:49:41.205373+0000","last_fullsized":"2024-09-18T07:49:41.205373+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:48:06.127745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:48:06.127745+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":991,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.197369+0000","last_change":"2024-09-18T07:47:20.131135+0000","last_active":"2024-09-18T07:49:41.197369+0000","last_peered":"2024-09-18T07:49:41.197369+0000","last_clean":"2024-09-18T07:49:41.197369+0000","last_became_active":"2024-09-18T07:47:20.130809+0000","last_became_peered":"2024-09-18T07:47:20.130809+0000","last_unstale":"2024-09-18T07:49:41.197369+0000","last_undegraded":"2024-09-18T07:49:41.197369+0000","last_fullsized":"2024-09-18T07:49:41.197369+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:46.629196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:46.629196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:27.458621+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":790,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170264+0000","last_change":"2024-09-18T07:49:24.445613+0000","last_active":"2024-09-18T07:49:41.170264+0000","last_peered":"2024-09-18T07:49:41.170264+0000","last_clean":"2024-09-18T07:49:41.170264+0000","last_became_active":"2024-09-18T07:49:24.445300+0000","last_became_peered":"2024-09-18T07:49:24.445300+0000","last_unstale":"2024-09-18T07:49:41.170264+0000","last_undegraded":"2024-09-18T07:49:41.170264+0000","last_fullsized":"2024-09-18T07:49:41.170264+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:45:52.882700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:45:52.882700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.8","version":"529'10701","reported_seq":22792,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.273589+0000","last_change":"2024-09-18T07:49:22.516716+0000","last_active":"2024-09-18T07:49:47.273589+0000","last_peered":"2024-09-18T07:49:47.273589+0000","last_clean":"2024-09-18T07:49:47.273589+0000","last_became_active":"2024-09-18T07:49:22.442874+0000","last_became_peered":"2024-09-18T07:49:22.442874+0000","last_unstale":"2024-09-18T07:49:47.273589+0000","last_undegraded":"2024-09-18T07:49:47.273589+0000","last_fullsized":"2024-09-18T07:49:47.273589+0000","mapping_epoch":515,"log_start":"502'6491","ondisk_log_start":"502'6491","created":459,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'5883","last_scrub_stamp":"2024-09-18T07:48:25.402648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:48:25.402648+0000","objects_scrubbed":5819,"log_size":4210,"log_dups_size":0,"ondisk_log_size":4210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216858624,"num_objects":3309,"num_object_clones":0,"num_object_copies":9927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10701,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5929,"num_bytes_recovered":388431872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":832,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171491+0000","last_change":"2024-09-18T07:48:27.916178+0000","last_active":"2024-09-18T07:49:41.171491+0000","last_peered":"2024-09-18T07:49:41.171491+0000","last_clean":"2024-09-18T07:49:41.171491+0000","last_became_active":"2024-09-18T07:44:43.783295+0000","last_became_peered":"2024-09-18T07:44:43.783295+0000","last_unstale":"2024-09-18T07:49:41.171491+0000","last_undegraded":"2024-09-18T07:49:41.171491+0000","last_fullsized":"2024-09-18T07:49:41.171491+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:27.916095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:27.916095+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"323'17","reported_seq":712,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.207774+0000","last_change":"2024-09-18T07:49:32.704380+0000","last_active":"2024-09-18T07:49:41.207774+0000","last_peered":"2024-09-18T07:49:41.207774+0000","last_clean":"2024-09-18T07:49:41.207774+0000","last_became_active":"2024-09-18T07:49:32.690652+0000","last_became_peered":"2024-09-18T07:49:32.690652+0000","last_unstale":"2024-09-18T07:49:41.207774+0000","last_undegraded":"2024-09-18T07:49:41.207774+0000","last_fullsized":"2024-09-18T07:49:41.207774+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'17","last_scrub_stamp":"2024-09-18T07:47:56.037047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.037047+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":23,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":721,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169359+0000","last_change":"2024-09-18T07:47:20.140678+0000","last_active":"2024-09-18T07:49:41.169359+0000","last_peered":"2024-09-18T07:49:41.169359+0000","last_clean":"2024-09-18T07:49:41.169359+0000","last_became_active":"2024-09-18T07:47:20.140376+0000","last_became_peered":"2024-09-18T07:47:20.140376+0000","last_unstale":"2024-09-18T07:49:41.169359+0000","last_undegraded":"2024-09-18T07:49:41.169359+0000","last_fullsized":"2024-09-18T07:49:41.169359+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:52.917296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:52.917296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":1023,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169775+0000","last_change":"2024-09-18T07:48:32.288661+0000","last_active":"2024-09-18T07:49:41.169775+0000","last_peered":"2024-09-18T07:49:41.169775+0000","last_clean":"2024-09-18T07:49:41.169775+0000","last_became_active":"2024-09-18T07:48:32.257016+0000","last_became_peered":"2024-09-18T07:48:32.257016+0000","last_unstale":"2024-09-18T07:49:41.169775+0000","last_undegraded":"2024-09-18T07:49:41.169775+0000","last_fullsized":"2024-09-18T07:49:41.169775+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:07.841406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:47:07.841406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1011,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170668+0000","last_change":"2024-09-18T07:47:16.834107+0000","last_active":"2024-09-18T07:49:41.170668+0000","last_peered":"2024-09-18T07:49:41.170668+0000","last_clean":"2024-09-18T07:49:41.170668+0000","last_became_active":"2024-09-18T07:47:11.201379+0000","last_became_peered":"2024-09-18T07:47:11.201379+0000","last_unstale":"2024-09-18T07:49:41.170668+0000","last_undegraded":"2024-09-18T07:49:41.170668+0000","last_fullsized":"2024-09-18T07:49:41.170668+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:16.834043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:16.834043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":800,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172125+0000","last_change":"2024-09-18T07:49:24.457052+0000","last_active":"2024-09-18T07:49:41.172125+0000","last_peered":"2024-09-18T07:49:41.172125+0000","last_clean":"2024-09-18T07:49:41.172125+0000","last_became_active":"2024-09-18T07:49:24.446434+0000","last_became_peered":"2024-09-18T07:49:24.446434+0000","last_unstale":"2024-09-18T07:49:41.172125+0000","last_undegraded":"2024-09-18T07:49:41.172125+0000","last_fullsized":"2024-09-18T07:49:41.172125+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:47:42.037791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.037791+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":78,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"529'10688","reported_seq":23011,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.277373+0000","last_change":"2024-09-18T07:49:47.098590+0000","last_active":"2024-09-18T07:49:47.277373+0000","last_peered":"2024-09-18T07:49:47.277373+0000","last_clean":"2024-09-18T07:49:47.277373+0000","last_became_active":"2024-09-18T07:49:19.392861+0000","last_became_peered":"2024-09-18T07:49:19.392861+0000","last_unstale":"2024-09-18T07:49:47.277373+0000","last_undegraded":"2024-09-18T07:49:47.277373+0000","last_fullsized":"2024-09-18T07:49:47.277373+0000","mapping_epoch":512,"log_start":"502'6427","ondisk_log_start":"502'6427","created":459,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'10674","last_scrub_stamp":"2024-09-18T07:49:47.098373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:49:47.098373+0000","objects_scrubbed":3360,"log_size":4261,"log_dups_size":0,"ondisk_log_size":4261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:49.766358+0000","scrub_duration":1162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215613440,"num_objects":3290,"num_object_clones":0,"num_object_copies":9870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10688,"num_write_kb":447296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6057,"num_bytes_recovered":396951552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":34,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.276187+0000","last_change":"2024-09-18T07:49:24.453650+0000","last_active":"2024-09-18T07:49:47.276187+0000","last_peered":"2024-09-18T07:49:47.276187+0000","last_clean":"2024-09-18T07:49:47.276187+0000","last_became_active":"2024-09-18T07:49:24.453318+0000","last_became_peered":"2024-09-18T07:49:24.453318+0000","last_unstale":"2024-09-18T07:49:47.276187+0000","last_undegraded":"2024-09-18T07:49:47.276187+0000","last_fullsized":"2024-09-18T07:49:47.276187+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:49:12.773317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:49:12.773317+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:38.973101+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"323'24","reported_seq":1065,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.198513+0000","last_change":"2024-09-18T07:47:20.159485+0000","last_active":"2024-09-18T07:49:41.198513+0000","last_peered":"2024-09-18T07:49:41.198513+0000","last_clean":"2024-09-18T07:49:41.198513+0000","last_became_active":"2024-09-18T07:47:20.138961+0000","last_became_peered":"2024-09-18T07:47:20.138961+0000","last_unstale":"2024-09-18T07:49:41.198513+0000","last_undegraded":"2024-09-18T07:49:41.198513+0000","last_fullsized":"2024-09-18T07:49:41.198513+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'24","last_scrub_stamp":"2024-09-18T07:44:06.211676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:44:06.211676+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:22.448696+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":995,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.197297+0000","last_change":"2024-09-18T07:48:10.165907+0000","last_active":"2024-09-18T07:49:41.197297+0000","last_peered":"2024-09-18T07:49:41.197297+0000","last_clean":"2024-09-18T07:49:41.197297+0000","last_became_active":"2024-09-18T07:48:10.142633+0000","last_became_peered":"2024-09-18T07:48:10.142633+0000","last_unstale":"2024-09-18T07:49:41.197297+0000","last_undegraded":"2024-09-18T07:49:41.197297+0000","last_fullsized":"2024-09-18T07:49:41.197297+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:42:44.401546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:44.401546+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:21.405419+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":1651,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:45.462414+0000","last_change":"2024-09-18T07:47:20.162839+0000","last_active":"2024-09-18T07:49:45.462414+0000","last_peered":"2024-09-18T07:49:45.462414+0000","last_clean":"2024-09-18T07:49:45.462414+0000","last_became_active":"2024-09-18T07:47:20.138901+0000","last_became_peered":"2024-09-18T07:47:20.138901+0000","last_unstale":"2024-09-18T07:49:45.462414+0000","last_undegraded":"2024-09-18T07:49:45.462414+0000","last_fullsized":"2024-09-18T07:49:45.462414+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T07:46:47.771151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:46:47.771151+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":887,"num_read_kb":882,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":884,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:42.663860+0000","last_change":"2024-09-18T07:49:42.663860+0000","last_active":"2024-09-18T07:49:42.663860+0000","last_peered":"2024-09-18T07:49:42.663860+0000","last_clean":"2024-09-18T07:49:42.663860+0000","last_became_active":"2024-09-18T07:47:20.144413+0000","last_became_peered":"2024-09-18T07:47:20.144413+0000","last_unstale":"2024-09-18T07:49:42.663860+0000","last_undegraded":"2024-09-18T07:49:42.663860+0000","last_fullsized":"2024-09-18T07:49:42.663860+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:42.663800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:49:42.663800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:51.780741+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":565,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:43.622155+0000","last_change":"2024-09-18T07:49:43.622155+0000","last_active":"2024-09-18T07:49:43.622155+0000","last_peered":"2024-09-18T07:49:43.622155+0000","last_clean":"2024-09-18T07:49:43.622155+0000","last_became_active":"2024-09-18T07:49:32.691427+0000","last_became_peered":"2024-09-18T07:49:32.691427+0000","last_unstale":"2024-09-18T07:49:43.622155+0000","last_undegraded":"2024-09-18T07:49:43.622155+0000","last_fullsized":"2024-09-18T07:49:43.622155+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:43.622102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:43.622102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:49.970792+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.a","version":"528'10589","reported_seq":22348,"reported_epoch":529,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-18T07:49:47.264745+0000","last_change":"2024-09-18T07:49:33.964097+0000","last_active":"2024-09-18T07:49:47.264745+0000","last_peered":"2024-09-18T07:49:47.264745+0000","last_clean":"2024-09-18T07:49:31.606072+0000","last_became_active":"2024-09-18T07:49:33.895295+0000","last_became_peered":"2024-09-18T07:49:33.895295+0000","last_unstale":"2024-09-18T07:49:47.264745+0000","last_undegraded":"2024-09-18T07:49:47.264745+0000","last_fullsized":"2024-09-18T07:49:47.264745+0000","mapping_epoch":522,"log_start":"502'6372","ondisk_log_start":"502'6372","created":459,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'5664","last_scrub_stamp":"2024-09-18T07:48:22.026341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:48:22.026341+0000","objects_scrubbed":5623,"log_size":4217,"log_dups_size":0,"ondisk_log_size":4217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214630400,"num_objects":3275,"num_object_clones":0,"num_object_copies":9825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3275,"num_objects_unfound":0,"num_objects_dirty":3275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10589,"num_write_kb":443648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5758,"num_bytes_recovered":377290752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,2],"avail_no_missing":["6","2","3","5"],"object_location_counts":[{"shards":"2,5,6","objects":3275}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"398'34","reported_seq":812,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.265327+0000","last_change":"2024-09-18T07:47:58.415964+0000","last_active":"2024-09-18T07:49:47.265327+0000","last_peered":"2024-09-18T07:49:47.265327+0000","last_clean":"2024-09-18T07:49:47.265327+0000","last_became_active":"2024-09-18T07:44:43.786827+0000","last_became_peered":"2024-09-18T07:44:43.786827+0000","last_unstale":"2024-09-18T07:49:47.265327+0000","last_undegraded":"2024-09-18T07:49:47.265327+0000","last_fullsized":"2024-09-18T07:49:47.265327+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'34","last_scrub_stamp":"2024-09-18T07:47:58.415734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:58.415734+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":752,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.207078+0000","last_change":"2024-09-18T07:49:08.604265+0000","last_active":"2024-09-18T07:49:41.207078+0000","last_peered":"2024-09-18T07:49:41.207078+0000","last_clean":"2024-09-18T07:49:41.207078+0000","last_became_active":"2024-09-18T07:48:32.266432+0000","last_became_peered":"2024-09-18T07:48:32.266432+0000","last_unstale":"2024-09-18T07:49:41.207078+0000","last_undegraded":"2024-09-18T07:49:41.207078+0000","last_fullsized":"2024-09-18T07:49:41.207078+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:08.604203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:08.604203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:22.110767+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":815,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171986+0000","last_change":"2024-09-18T07:48:26.963331+0000","last_active":"2024-09-18T07:49:41.171986+0000","last_peered":"2024-09-18T07:49:41.171986+0000","last_clean":"2024-09-18T07:49:41.171986+0000","last_became_active":"2024-09-18T07:39:03.350419+0000","last_became_peered":"2024-09-18T07:39:03.350419+0000","last_unstale":"2024-09-18T07:49:41.171986+0000","last_undegraded":"2024-09-18T07:49:41.171986+0000","last_fullsized":"2024-09-18T07:49:41.171986+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:26.963223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:26.963223+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":768,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172313+0000","last_change":"2024-09-18T07:48:21.934191+0000","last_active":"2024-09-18T07:49:41.172313+0000","last_peered":"2024-09-18T07:49:41.172313+0000","last_clean":"2024-09-18T07:49:41.172313+0000","last_became_active":"2024-09-18T07:47:11.195084+0000","last_became_peered":"2024-09-18T07:47:11.195084+0000","last_unstale":"2024-09-18T07:49:41.172313+0000","last_undegraded":"2024-09-18T07:49:41.172313+0000","last_fullsized":"2024-09-18T07:49:41.172313+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:21.934130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:21.934130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:49.622120+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":880,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.207934+0000","last_change":"2024-09-18T07:49:32.688396+0000","last_active":"2024-09-18T07:49:41.207934+0000","last_peered":"2024-09-18T07:49:41.207934+0000","last_clean":"2024-09-18T07:49:41.207934+0000","last_became_active":"2024-09-18T07:49:32.687989+0000","last_became_peered":"2024-09-18T07:49:32.687989+0000","last_unstale":"2024-09-18T07:49:41.207934+0000","last_undegraded":"2024-09-18T07:49:41.207934+0000","last_fullsized":"2024-09-18T07:49:41.207934+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:04.610883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:04.610883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:30.506002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"529'10780","reported_seq":20236,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.268107+0000","last_change":"2024-09-18T07:49:38.235683+0000","last_active":"2024-09-18T07:49:47.268107+0000","last_peered":"2024-09-18T07:49:47.268107+0000","last_clean":"2024-09-18T07:49:47.268107+0000","last_became_active":"2024-09-18T07:49:38.216903+0000","last_became_peered":"2024-09-18T07:49:38.216903+0000","last_unstale":"2024-09-18T07:49:47.268107+0000","last_undegraded":"2024-09-18T07:49:47.268107+0000","last_fullsized":"2024-09-18T07:49:47.268107+0000","mapping_epoch":525,"log_start":"502'6568","ondisk_log_start":"502'6568","created":459,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4212,"log_dups_size":0,"ondisk_log_size":4212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219021312,"num_objects":3342,"num_object_clones":0,"num_object_copies":10026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10780,"num_write_kb":451904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4643,"num_bytes_recovered":304283648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"323'19","reported_seq":1120,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170072+0000","last_change":"2024-09-18T07:49:32.696886+0000","last_active":"2024-09-18T07:49:41.170072+0000","last_peered":"2024-09-18T07:49:41.170072+0000","last_clean":"2024-09-18T07:49:41.170072+0000","last_became_active":"2024-09-18T07:49:32.684857+0000","last_became_peered":"2024-09-18T07:49:32.684857+0000","last_unstale":"2024-09-18T07:49:41.170072+0000","last_undegraded":"2024-09-18T07:49:41.170072+0000","last_fullsized":"2024-09-18T07:49:41.170072+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:42:43.402858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:43.402858+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:39.835405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":873,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169212+0000","last_change":"2024-09-18T07:44:43.793621+0000","last_active":"2024-09-18T07:49:41.169212+0000","last_peered":"2024-09-18T07:49:41.169212+0000","last_clean":"2024-09-18T07:49:41.169212+0000","last_became_active":"2024-09-18T07:44:43.790160+0000","last_became_peered":"2024-09-18T07:44:43.790160+0000","last_unstale":"2024-09-18T07:49:41.169212+0000","last_undegraded":"2024-09-18T07:49:41.169212+0000","last_fullsized":"2024-09-18T07:49:41.169212+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.078600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.078600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:19.692544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":885,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.207549+0000","last_change":"2024-09-18T07:47:11.202169+0000","last_active":"2024-09-18T07:49:41.207549+0000","last_peered":"2024-09-18T07:49:41.207549+0000","last_clean":"2024-09-18T07:49:41.207549+0000","last_became_active":"2024-09-18T07:47:11.198103+0000","last_became_peered":"2024-09-18T07:47:11.198103+0000","last_unstale":"2024-09-18T07:49:41.207549+0000","last_undegraded":"2024-09-18T07:49:41.207549+0000","last_fullsized":"2024-09-18T07:49:41.207549+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:44:45.479624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:44:45.479624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:21.577272+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":877,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.265865+0000","last_change":"2024-09-18T07:48:32.259237+0000","last_active":"2024-09-18T07:49:47.265865+0000","last_peered":"2024-09-18T07:49:47.265865+0000","last_clean":"2024-09-18T07:49:47.265865+0000","last_became_active":"2024-09-18T07:48:32.258901+0000","last_became_peered":"2024-09-18T07:48:32.258901+0000","last_unstale":"2024-09-18T07:49:47.265865+0000","last_undegraded":"2024-09-18T07:49:47.265865+0000","last_fullsized":"2024-09-18T07:49:47.265865+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:26.691414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:26.691414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"528'10500","reported_seq":20379,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:46.847124+0000","last_change":"2024-09-18T07:49:33.657485+0000","last_active":"2024-09-18T07:49:46.847124+0000","last_peered":"2024-09-18T07:49:46.847124+0000","last_clean":"2024-09-18T07:49:46.847124+0000","last_became_active":"2024-09-18T07:49:33.641738+0000","last_became_peered":"2024-09-18T07:49:33.641738+0000","last_unstale":"2024-09-18T07:49:46.847124+0000","last_undegraded":"2024-09-18T07:49:46.847124+0000","last_fullsized":"2024-09-18T07:49:46.847124+0000","mapping_epoch":522,"log_start":"492'5241","ondisk_log_start":"492'5241","created":459,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":5259,"log_dups_size":0,"ondisk_log_size":5259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212729856,"num_objects":3246,"num_object_clones":0,"num_object_copies":9738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10500,"num_write_kb":439872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4854,"num_bytes_recovered":318111744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":422,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.206087+0000","last_change":"2024-09-18T07:49:24.456347+0000","last_active":"2024-09-18T07:49:41.206087+0000","last_peered":"2024-09-18T07:49:41.206087+0000","last_clean":"2024-09-18T07:49:41.206087+0000","last_became_active":"2024-09-18T07:49:24.448174+0000","last_became_peered":"2024-09-18T07:49:24.448174+0000","last_unstale":"2024-09-18T07:49:41.206087+0000","last_undegraded":"2024-09-18T07:49:41.206087+0000","last_fullsized":"2024-09-18T07:49:41.206087+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:49:00.644450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:49:00.644450+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:11.512414+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"323'31","reported_seq":955,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169916+0000","last_change":"2024-09-18T07:48:22.002514+0000","last_active":"2024-09-18T07:49:41.169916+0000","last_peered":"2024-09-18T07:49:41.169916+0000","last_clean":"2024-09-18T07:49:41.169916+0000","last_became_active":"2024-09-18T07:47:20.132912+0000","last_became_peered":"2024-09-18T07:47:20.132912+0000","last_unstale":"2024-09-18T07:49:41.169916+0000","last_undegraded":"2024-09-18T07:49:41.169916+0000","last_fullsized":"2024-09-18T07:49:41.169916+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'31","last_scrub_stamp":"2024-09-18T07:48:22.002416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:22.002416+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":680,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.264698+0000","last_change":"2024-09-18T07:49:32.688728+0000","last_active":"2024-09-18T07:49:47.264698+0000","last_peered":"2024-09-18T07:49:47.264698+0000","last_clean":"2024-09-18T07:49:47.264698+0000","last_became_active":"2024-09-18T07:49:32.688352+0000","last_became_peered":"2024-09-18T07:49:32.688352+0000","last_unstale":"2024-09-18T07:49:47.264698+0000","last_undegraded":"2024-09-18T07:49:47.264698+0000","last_fullsized":"2024-09-18T07:49:47.264698+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:32.491919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:32.491919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":544,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169540+0000","last_change":"2024-09-18T07:47:20.131352+0000","last_active":"2024-09-18T07:49:41.169540+0000","last_peered":"2024-09-18T07:49:41.169540+0000","last_clean":"2024-09-18T07:49:41.169540+0000","last_became_active":"2024-09-18T07:47:20.130994+0000","last_became_peered":"2024-09-18T07:47:20.130994+0000","last_unstale":"2024-09-18T07:49:41.169540+0000","last_undegraded":"2024-09-18T07:49:41.169540+0000","last_fullsized":"2024-09-18T07:49:41.169540+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:13.019711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:13.019711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1016,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170635+0000","last_change":"2024-09-18T07:48:32.263327+0000","last_active":"2024-09-18T07:49:41.170635+0000","last_peered":"2024-09-18T07:49:41.170635+0000","last_clean":"2024-09-18T07:49:41.170635+0000","last_became_active":"2024-09-18T07:48:32.263024+0000","last_became_peered":"2024-09-18T07:48:32.263024+0000","last_unstale":"2024-09-18T07:49:41.170635+0000","last_undegraded":"2024-09-18T07:49:41.170635+0000","last_fullsized":"2024-09-18T07:49:41.170635+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:06.889139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:06.889139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"529'10648","reported_seq":12572,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.276653+0000","last_change":"2024-09-18T07:48:27.491090+0000","last_active":"2024-09-18T07:49:47.276653+0000","last_peered":"2024-09-18T07:49:47.276653+0000","last_clean":"2024-09-18T07:49:47.276653+0000","last_became_active":"2024-09-18T07:47:38.218858+0000","last_became_peered":"2024-09-18T07:47:38.218858+0000","last_unstale":"2024-09-18T07:49:47.276653+0000","last_undegraded":"2024-09-18T07:49:47.276653+0000","last_fullsized":"2024-09-18T07:49:47.276653+0000","mapping_epoch":475,"log_start":"502'6456","ondisk_log_start":"502'6456","created":459,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'6054","last_scrub_stamp":"2024-09-18T07:48:27.490707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:48:27.490707+0000","objects_scrubbed":6013,"log_size":4192,"log_dups_size":0,"ondisk_log_size":4192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218497024,"num_objects":3334,"num_object_clones":0,"num_object_copies":10002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10648,"num_write_kb":447424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":848,"num_bytes_recovered":55574528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":819,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171319+0000","last_change":"2024-09-18T07:48:32.261433+0000","last_active":"2024-09-18T07:49:41.171319+0000","last_peered":"2024-09-18T07:49:41.171319+0000","last_clean":"2024-09-18T07:49:41.171319+0000","last_became_active":"2024-09-18T07:48:32.259457+0000","last_became_peered":"2024-09-18T07:48:32.259457+0000","last_unstale":"2024-09-18T07:49:41.171319+0000","last_undegraded":"2024-09-18T07:49:41.171319+0000","last_fullsized":"2024-09-18T07:49:41.171319+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:46:24.585120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:46:24.585120+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"323'20","reported_seq":927,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169753+0000","last_change":"2024-09-18T07:48:32.303165+0000","last_active":"2024-09-18T07:49:41.169753+0000","last_peered":"2024-09-18T07:49:41.169753+0000","last_clean":"2024-09-18T07:49:41.169753+0000","last_became_active":"2024-09-18T07:48:32.269075+0000","last_became_peered":"2024-09-18T07:48:32.269075+0000","last_unstale":"2024-09-18T07:49:41.169753+0000","last_undegraded":"2024-09-18T07:49:41.169753+0000","last_fullsized":"2024-09-18T07:49:41.169753+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:46:14.047907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:14.047907+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":405,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169286+0000","last_change":"2024-09-18T07:48:32.255423+0000","last_active":"2024-09-18T07:49:41.169286+0000","last_peered":"2024-09-18T07:49:41.169286+0000","last_clean":"2024-09-18T07:49:41.169286+0000","last_became_active":"2024-09-18T07:48:32.255130+0000","last_became_peered":"2024-09-18T07:48:32.255130+0000","last_unstale":"2024-09-18T07:49:41.169286+0000","last_undegraded":"2024-09-18T07:49:41.169286+0000","last_fullsized":"2024-09-18T07:49:41.169286+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:42.225418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.225418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"461'13","reported_seq":1259,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.586250+0000","last_change":"2024-09-18T07:47:15.865446+0000","last_active":"2024-09-18T07:49:41.586250+0000","last_peered":"2024-09-18T07:49:41.586250+0000","last_clean":"2024-09-18T07:49:41.586250+0000","last_became_active":"2024-09-18T07:47:11.203256+0000","last_became_peered":"2024-09-18T07:47:11.203256+0000","last_unstale":"2024-09-18T07:49:41.586250+0000","last_undegraded":"2024-09-18T07:49:41.586250+0000","last_fullsized":"2024-09-18T07:49:41.586250+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'13","last_scrub_stamp":"2024-09-18T07:47:15.865373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:15.865373+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":896,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170314+0000","last_change":"2024-09-18T07:49:32.690291+0000","last_active":"2024-09-18T07:49:41.170314+0000","last_peered":"2024-09-18T07:49:41.170314+0000","last_clean":"2024-09-18T07:49:41.170314+0000","last_became_active":"2024-09-18T07:49:32.689562+0000","last_became_peered":"2024-09-18T07:49:32.689562+0000","last_unstale":"2024-09-18T07:49:41.170314+0000","last_undegraded":"2024-09-18T07:49:41.170314+0000","last_fullsized":"2024-09-18T07:49:41.170314+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:48.207484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:48.207484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"528'10718","reported_seq":10868,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:46.855270+0000","last_change":"2024-09-18T07:47:10.191967+0000","last_active":"2024-09-18T07:49:46.855270+0000","last_peered":"2024-09-18T07:49:46.855270+0000","last_clean":"2024-09-18T07:49:46.855270+0000","last_became_active":"2024-09-18T07:47:10.190921+0000","last_became_peered":"2024-09-18T07:47:10.190921+0000","last_unstale":"2024-09-18T07:49:46.855270+0000","last_undegraded":"2024-09-18T07:49:46.855270+0000","last_fullsized":"2024-09-18T07:49:46.855270+0000","mapping_epoch":459,"log_start":"498'6345","ondisk_log_start":"498'6345","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219807744,"num_objects":3354,"num_object_clones":0,"num_object_copies":10062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10718,"num_write_kb":450304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"323'15","reported_seq":1069,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.196857+0000","last_change":"2024-09-18T07:48:22.197957+0000","last_active":"2024-09-18T07:49:41.196857+0000","last_peered":"2024-09-18T07:49:41.196857+0000","last_clean":"2024-09-18T07:49:41.196857+0000","last_became_active":"2024-09-18T07:47:20.138834+0000","last_became_peered":"2024-09-18T07:47:20.138834+0000","last_unstale":"2024-09-18T07:49:41.196857+0000","last_undegraded":"2024-09-18T07:49:41.196857+0000","last_fullsized":"2024-09-18T07:49:41.196857+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:48:22.197888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:22.197888+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:43.300890+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":919,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.206612+0000","last_change":"2024-09-18T07:48:32.262270+0000","last_active":"2024-09-18T07:49:41.206612+0000","last_peered":"2024-09-18T07:49:41.206612+0000","last_clean":"2024-09-18T07:49:41.206612+0000","last_became_active":"2024-09-18T07:48:32.261507+0000","last_became_peered":"2024-09-18T07:48:32.261507+0000","last_unstale":"2024-09-18T07:49:41.206612+0000","last_undegraded":"2024-09-18T07:49:41.206612+0000","last_fullsized":"2024-09-18T07:49:41.206612+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:51.021530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:51.021530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:23.522646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1004,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169210+0000","last_change":"2024-09-18T07:48:32.265137+0000","last_active":"2024-09-18T07:49:41.169210+0000","last_peered":"2024-09-18T07:49:41.169210+0000","last_clean":"2024-09-18T07:49:41.169210+0000","last_became_active":"2024-09-18T07:48:32.261737+0000","last_became_peered":"2024-09-18T07:48:32.261737+0000","last_unstale":"2024-09-18T07:49:41.169210+0000","last_undegraded":"2024-09-18T07:49:41.169210+0000","last_fullsized":"2024-09-18T07:49:41.169210+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:22.799585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:22.799585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":433,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:42.436609+0000","last_change":"2024-09-18T07:49:42.436609+0000","last_active":"2024-09-18T07:49:42.436609+0000","last_peered":"2024-09-18T07:49:42.436609+0000","last_clean":"2024-09-18T07:49:42.436609+0000","last_became_active":"2024-09-18T07:46:29.322779+0000","last_became_peered":"2024-09-18T07:46:29.322779+0000","last_unstale":"2024-09-18T07:49:42.436609+0000","last_undegraded":"2024-09-18T07:49:42.436609+0000","last_fullsized":"2024-09-18T07:49:42.436609+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:42.436545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:42.436545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:54.114621+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"528'10553","reported_seq":10702,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:46.875718+0000","last_change":"2024-09-18T07:47:11.201597+0000","last_active":"2024-09-18T07:49:46.875718+0000","last_peered":"2024-09-18T07:49:46.875718+0000","last_clean":"2024-09-18T07:49:46.875718+0000","last_became_active":"2024-09-18T07:47:11.201275+0000","last_became_peered":"2024-09-18T07:47:11.201275+0000","last_unstale":"2024-09-18T07:49:46.875718+0000","last_undegraded":"2024-09-18T07:49:46.875718+0000","last_fullsized":"2024-09-18T07:49:46.875718+0000","mapping_epoch":460,"log_start":"497'6132","ondisk_log_start":"497'6132","created":459,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4421,"log_dups_size":0,"ondisk_log_size":4421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217120768,"num_objects":3313,"num_object_clones":0,"num_object_copies":9939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10553,"num_write_kb":443712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"323'35","reported_seq":729,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.264890+0000","last_change":"2024-09-18T07:47:52.386750+0000","last_active":"2024-09-18T07:49:47.264890+0000","last_peered":"2024-09-18T07:49:47.264890+0000","last_clean":"2024-09-18T07:49:47.264890+0000","last_became_active":"2024-09-18T07:47:20.141942+0000","last_became_peered":"2024-09-18T07:47:20.141942+0000","last_unstale":"2024-09-18T07:49:47.264890+0000","last_undegraded":"2024-09-18T07:49:47.264890+0000","last_fullsized":"2024-09-18T07:49:47.264890+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'35","last_scrub_stamp":"2024-09-18T07:47:52.386688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.386688+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":781,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172403+0000","last_change":"2024-09-18T07:47:17.794908+0000","last_active":"2024-09-18T07:49:41.172403+0000","last_peered":"2024-09-18T07:49:41.172403+0000","last_clean":"2024-09-18T07:49:41.172403+0000","last_became_active":"2024-09-18T07:47:11.194873+0000","last_became_peered":"2024-09-18T07:47:11.194873+0000","last_unstale":"2024-09-18T07:49:41.172403+0000","last_undegraded":"2024-09-18T07:49:41.172403+0000","last_fullsized":"2024-09-18T07:49:41.172403+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:17.794850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:17.794850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1019,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170172+0000","last_change":"2024-09-18T07:48:32.257353+0000","last_active":"2024-09-18T07:49:41.170172+0000","last_peered":"2024-09-18T07:49:41.170172+0000","last_clean":"2024-09-18T07:49:41.170172+0000","last_became_active":"2024-09-18T07:48:32.257023+0000","last_became_peered":"2024-09-18T07:48:32.257023+0000","last_unstale":"2024-09-18T07:49:41.170172+0000","last_undegraded":"2024-09-18T07:49:41.170172+0000","last_fullsized":"2024-09-18T07:49:41.170172+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:20.767517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:20.767517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":763,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172071+0000","last_change":"2024-09-18T07:48:32.265587+0000","last_active":"2024-09-18T07:49:41.172071+0000","last_peered":"2024-09-18T07:49:41.172071+0000","last_clean":"2024-09-18T07:49:41.172071+0000","last_became_active":"2024-09-18T07:48:32.265086+0000","last_became_peered":"2024-09-18T07:48:32.265086+0000","last_unstale":"2024-09-18T07:49:41.172071+0000","last_undegraded":"2024-09-18T07:49:41.172071+0000","last_fullsized":"2024-09-18T07:49:41.172071+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:50.079275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:50.079275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"528'10553","reported_seq":12382,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:46.841443+0000","last_change":"2024-09-18T07:47:53.337653+0000","last_active":"2024-09-18T07:49:46.841443+0000","last_peered":"2024-09-18T07:49:46.841443+0000","last_clean":"2024-09-18T07:49:46.841443+0000","last_became_active":"2024-09-18T07:47:53.331769+0000","last_became_peered":"2024-09-18T07:47:53.331769+0000","last_unstale":"2024-09-18T07:49:46.841443+0000","last_undegraded":"2024-09-18T07:49:46.841443+0000","last_fullsized":"2024-09-18T07:49:46.841443+0000","mapping_epoch":481,"log_start":"492'5326","ondisk_log_start":"492'5326","created":459,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216596480,"num_objects":3305,"num_object_clones":0,"num_object_copies":9915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10553,"num_write_kb":443456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":831,"num_bytes_recovered":54460416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"323'18","reported_seq":986,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169146+0000","last_change":"2024-09-18T07:48:32.288842+0000","last_active":"2024-09-18T07:49:41.169146+0000","last_peered":"2024-09-18T07:49:41.169146+0000","last_clean":"2024-09-18T07:49:41.169146+0000","last_became_active":"2024-09-18T07:48:32.262707+0000","last_became_peered":"2024-09-18T07:48:32.262707+0000","last_unstale":"2024-09-18T07:49:41.169146+0000","last_undegraded":"2024-09-18T07:49:41.169146+0000","last_fullsized":"2024-09-18T07:49:41.169146+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:46:25.012172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:25.012172+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":773,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172643+0000","last_change":"2024-09-18T07:47:20.139266+0000","last_active":"2024-09-18T07:49:41.172643+0000","last_peered":"2024-09-18T07:49:41.172643+0000","last_clean":"2024-09-18T07:49:41.172643+0000","last_became_active":"2024-09-18T07:47:20.138910+0000","last_became_peered":"2024-09-18T07:47:20.138910+0000","last_unstale":"2024-09-18T07:49:41.172643+0000","last_undegraded":"2024-09-18T07:49:41.172643+0000","last_fullsized":"2024-09-18T07:49:41.172643+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:51.752578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:51.752578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1029,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169280+0000","last_change":"2024-09-18T07:48:23.757438+0000","last_active":"2024-09-18T07:49:41.169280+0000","last_peered":"2024-09-18T07:49:41.169280+0000","last_clean":"2024-09-18T07:49:41.169280+0000","last_became_active":"2024-09-18T07:47:20.139842+0000","last_became_peered":"2024-09-18T07:47:20.139842+0000","last_unstale":"2024-09-18T07:49:41.169280+0000","last_undegraded":"2024-09-18T07:49:41.169280+0000","last_fullsized":"2024-09-18T07:49:41.169280+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:23.757372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:23.757372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":934,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171836+0000","last_change":"2024-09-18T07:48:32.255818+0000","last_active":"2024-09-18T07:49:41.171836+0000","last_peered":"2024-09-18T07:49:41.171836+0000","last_clean":"2024-09-18T07:49:41.171836+0000","last_became_active":"2024-09-18T07:48:32.253112+0000","last_became_peered":"2024-09-18T07:48:32.253112+0000","last_unstale":"2024-09-18T07:49:41.171836+0000","last_undegraded":"2024-09-18T07:49:41.171836+0000","last_fullsized":"2024-09-18T07:49:41.171836+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:40.439672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:40.439672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"528'10476","reported_seq":12426,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:44.210052+0000","last_change":"2024-09-18T07:48:53.603702+0000","last_active":"2024-09-18T07:49:44.210052+0000","last_peered":"2024-09-18T07:49:44.210052+0000","last_clean":"2024-09-18T07:49:44.210052+0000","last_became_active":"2024-09-18T07:47:34.181063+0000","last_became_peered":"2024-09-18T07:47:34.181063+0000","last_unstale":"2024-09-18T07:49:44.210052+0000","last_undegraded":"2024-09-18T07:49:44.210052+0000","last_fullsized":"2024-09-18T07:49:44.210052+0000","mapping_epoch":471,"log_start":"495'6050","ondisk_log_start":"495'6050","created":459,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'7617","last_scrub_stamp":"2024-09-18T07:48:53.603594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:48:53.603594+0000","objects_scrubbed":6521,"log_size":4426,"log_dups_size":0,"ondisk_log_size":4426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:06.768809+0000","scrub_duration":1906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236584960,"num_objects":3610,"num_object_clones":0,"num_object_copies":10830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10476,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":858,"num_bytes_recovered":56229888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"323'27","reported_seq":977,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.206250+0000","last_change":"2024-09-18T07:48:03.057334+0000","last_active":"2024-09-18T07:49:41.206250+0000","last_peered":"2024-09-18T07:49:41.206250+0000","last_clean":"2024-09-18T07:49:41.206250+0000","last_became_active":"2024-09-18T07:47:11.196288+0000","last_became_peered":"2024-09-18T07:47:11.196288+0000","last_unstale":"2024-09-18T07:49:41.206250+0000","last_undegraded":"2024-09-18T07:49:41.206250+0000","last_fullsized":"2024-09-18T07:49:41.206250+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:48:03.057182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:03.057182+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":781,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172683+0000","last_change":"2024-09-18T07:48:34.003445+0000","last_active":"2024-09-18T07:49:41.172683+0000","last_peered":"2024-09-18T07:49:41.172683+0000","last_clean":"2024-09-18T07:49:41.172683+0000","last_became_active":"2024-09-18T07:44:43.778284+0000","last_became_peered":"2024-09-18T07:44:43.778284+0000","last_unstale":"2024-09-18T07:49:41.172683+0000","last_undegraded":"2024-09-18T07:49:41.172683+0000","last_fullsized":"2024-09-18T07:49:41.172683+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:48:34.003134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:34.003134+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:55.154854+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"398'14","reported_seq":973,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.587159+0000","last_change":"2024-09-18T07:46:21.582783+0000","last_active":"2024-09-18T07:49:41.587159+0000","last_peered":"2024-09-18T07:49:41.587159+0000","last_clean":"2024-09-18T07:49:41.587159+0000","last_became_active":"2024-09-18T07:44:43.782355+0000","last_became_peered":"2024-09-18T07:44:43.782355+0000","last_unstale":"2024-09-18T07:49:41.587159+0000","last_undegraded":"2024-09-18T07:49:41.587159+0000","last_fullsized":"2024-09-18T07:49:41.587159+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:46:21.582737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:21.582737+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":844,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.205191+0000","last_change":"2024-09-18T07:47:42.111034+0000","last_active":"2024-09-18T07:49:41.205191+0000","last_peered":"2024-09-18T07:49:41.205191+0000","last_clean":"2024-09-18T07:49:41.205191+0000","last_became_active":"2024-09-18T07:47:11.197583+0000","last_became_peered":"2024-09-18T07:47:11.197583+0000","last_unstale":"2024-09-18T07:49:41.205191+0000","last_undegraded":"2024-09-18T07:49:41.205191+0000","last_fullsized":"2024-09-18T07:49:41.205191+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:42.110980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.110980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:31.565452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":621,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.264813+0000","last_change":"2024-09-18T07:47:11.188646+0000","last_active":"2024-09-18T07:49:47.264813+0000","last_peered":"2024-09-18T07:49:47.264813+0000","last_clean":"2024-09-18T07:49:47.264813+0000","last_became_active":"2024-09-18T07:47:11.188332+0000","last_became_peered":"2024-09-18T07:47:11.188332+0000","last_unstale":"2024-09-18T07:49:47.264813+0000","last_undegraded":"2024-09-18T07:49:47.264813+0000","last_fullsized":"2024-09-18T07:49:47.264813+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:08.181420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:08.181420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:40.813652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"323'12","reported_seq":696,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172490+0000","last_change":"2024-09-18T07:48:10.211661+0000","last_active":"2024-09-18T07:49:41.172490+0000","last_peered":"2024-09-18T07:49:41.172490+0000","last_clean":"2024-09-18T07:49:41.172490+0000","last_became_active":"2024-09-18T07:48:10.160725+0000","last_became_peered":"2024-09-18T07:48:10.160725+0000","last_unstale":"2024-09-18T07:49:41.172490+0000","last_undegraded":"2024-09-18T07:49:41.172490+0000","last_fullsized":"2024-09-18T07:49:41.172490+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'12","last_scrub_stamp":"2024-09-18T07:48:03.699388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:03.699388+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":922,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.207155+0000","last_change":"2024-09-18T07:48:32.261425+0000","last_active":"2024-09-18T07:49:41.207155+0000","last_peered":"2024-09-18T07:49:41.207155+0000","last_clean":"2024-09-18T07:49:41.207155+0000","last_became_active":"2024-09-18T07:48:32.261094+0000","last_became_peered":"2024-09-18T07:48:32.261094+0000","last_unstale":"2024-09-18T07:49:41.207155+0000","last_undegraded":"2024-09-18T07:49:41.207155+0000","last_fullsized":"2024-09-18T07:49:41.207155+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:53.089903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.089903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:32.526030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":773,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172184+0000","last_change":"2024-09-18T07:48:56.269806+0000","last_active":"2024-09-18T07:49:41.172184+0000","last_peered":"2024-09-18T07:49:41.172184+0000","last_clean":"2024-09-18T07:49:41.172184+0000","last_became_active":"2024-09-18T07:44:43.782348+0000","last_became_peered":"2024-09-18T07:44:43.782348+0000","last_unstale":"2024-09-18T07:49:41.172184+0000","last_undegraded":"2024-09-18T07:49:41.172184+0000","last_fullsized":"2024-09-18T07:49:41.172184+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:56.269745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:56.269745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:03.146403+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"323'15","reported_seq":668,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172320+0000","last_change":"2024-09-18T07:48:32.298683+0000","last_active":"2024-09-18T07:49:41.172320+0000","last_peered":"2024-09-18T07:49:41.172320+0000","last_clean":"2024-09-18T07:49:41.172320+0000","last_became_active":"2024-09-18T07:48:32.259977+0000","last_became_peered":"2024-09-18T07:48:32.259977+0000","last_unstale":"2024-09-18T07:49:41.172320+0000","last_undegraded":"2024-09-18T07:49:41.172320+0000","last_fullsized":"2024-09-18T07:49:41.172320+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:48:27.802966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:27.802966+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:53.895081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":789,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:45.461696+0000","last_change":"2024-09-18T07:49:45.461696+0000","last_active":"2024-09-18T07:49:45.461696+0000","last_peered":"2024-09-18T07:49:45.461696+0000","last_clean":"2024-09-18T07:49:45.461696+0000","last_became_active":"2024-09-18T07:44:43.782930+0000","last_became_peered":"2024-09-18T07:44:43.782930+0000","last_unstale":"2024-09-18T07:49:45.461696+0000","last_undegraded":"2024-09-18T07:49:45.461696+0000","last_fullsized":"2024-09-18T07:49:45.461696+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:45.461629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:45.461629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:52.011721+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":945,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172403+0000","last_change":"2024-09-18T07:48:32.258194+0000","last_active":"2024-09-18T07:49:41.172403+0000","last_peered":"2024-09-18T07:49:41.172403+0000","last_clean":"2024-09-18T07:49:41.172403+0000","last_became_active":"2024-09-18T07:48:32.255775+0000","last_became_peered":"2024-09-18T07:48:32.255775+0000","last_unstale":"2024-09-18T07:49:41.172403+0000","last_undegraded":"2024-09-18T07:49:41.172403+0000","last_fullsized":"2024-09-18T07:49:41.172403+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:30.750167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:30.750167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":990,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.170871+0000","last_change":"2024-09-18T07:48:03.866565+0000","last_active":"2024-09-18T07:49:41.170871+0000","last_peered":"2024-09-18T07:49:41.170871+0000","last_clean":"2024-09-18T07:49:41.170871+0000","last_became_active":"2024-09-18T07:44:43.790582+0000","last_became_peered":"2024-09-18T07:44:43.790582+0000","last_unstale":"2024-09-18T07:49:41.170871+0000","last_undegraded":"2024-09-18T07:49:41.170871+0000","last_fullsized":"2024-09-18T07:49:41.170871+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:03.866499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:03.866499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"323'15","reported_seq":957,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.206697+0000","last_change":"2024-09-18T07:48:32.286075+0000","last_active":"2024-09-18T07:49:41.206697+0000","last_peered":"2024-09-18T07:49:41.206697+0000","last_clean":"2024-09-18T07:49:41.206697+0000","last_became_active":"2024-09-18T07:48:32.267360+0000","last_became_peered":"2024-09-18T07:48:32.267360+0000","last_unstale":"2024-09-18T07:49:41.206697+0000","last_undegraded":"2024-09-18T07:49:41.206697+0000","last_fullsized":"2024-09-18T07:49:41.206697+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:48:25.381861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:25.381861+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":452,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171629+0000","last_change":"2024-09-18T07:48:32.254574+0000","last_active":"2024-09-18T07:49:41.171629+0000","last_peered":"2024-09-18T07:49:41.171629+0000","last_clean":"2024-09-18T07:49:41.171629+0000","last_became_active":"2024-09-18T07:48:32.254219+0000","last_became_peered":"2024-09-18T07:48:32.254219+0000","last_unstale":"2024-09-18T07:49:41.171629+0000","last_undegraded":"2024-09-18T07:49:41.171629+0000","last_fullsized":"2024-09-18T07:49:41.171629+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:52.023913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.023913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"500'11","reported_seq":382,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.587107+0000","last_change":"2024-09-18T07:48:32.311073+0000","last_active":"2024-09-18T07:49:41.587107+0000","last_peered":"2024-09-18T07:49:41.587107+0000","last_clean":"2024-09-18T07:49:41.587107+0000","last_became_active":"2024-09-18T07:48:32.264561+0000","last_became_peered":"2024-09-18T07:48:32.264561+0000","last_unstale":"2024-09-18T07:49:41.587107+0000","last_undegraded":"2024-09-18T07:49:41.587107+0000","last_fullsized":"2024-09-18T07:49:41.587107+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'10","last_scrub_stamp":"2024-09-18T07:46:08.086437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:08.086437+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":878,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169896+0000","last_change":"2024-09-18T07:47:11.193555+0000","last_active":"2024-09-18T07:49:41.169896+0000","last_peered":"2024-09-18T07:49:41.169896+0000","last_clean":"2024-09-18T07:49:41.169896+0000","last_became_active":"2024-09-18T07:47:11.193273+0000","last_became_peered":"2024-09-18T07:47:11.193273+0000","last_unstale":"2024-09-18T07:49:41.169896+0000","last_undegraded":"2024-09-18T07:49:41.169896+0000","last_fullsized":"2024-09-18T07:49:41.169896+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:48.955463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.955463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:23.715894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"323'20","reported_seq":852,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171859+0000","last_change":"2024-09-18T07:49:32.702824+0000","last_active":"2024-09-18T07:49:41.171859+0000","last_peered":"2024-09-18T07:49:41.171859+0000","last_clean":"2024-09-18T07:49:41.171859+0000","last_became_active":"2024-09-18T07:49:32.690835+0000","last_became_peered":"2024-09-18T07:49:32.690835+0000","last_unstale":"2024-09-18T07:49:41.171859+0000","last_undegraded":"2024-09-18T07:49:41.171859+0000","last_fullsized":"2024-09-18T07:49:41.171859+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:46:14.650883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:14.650883+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":775,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172142+0000","last_change":"2024-09-18T07:49:00.272282+0000","last_active":"2024-09-18T07:49:41.172142+0000","last_peered":"2024-09-18T07:49:41.172142+0000","last_clean":"2024-09-18T07:49:41.172142+0000","last_became_active":"2024-09-18T07:44:43.779305+0000","last_became_peered":"2024-09-18T07:44:43.779305+0000","last_unstale":"2024-09-18T07:49:41.172142+0000","last_undegraded":"2024-09-18T07:49:41.172142+0000","last_fullsized":"2024-09-18T07:49:41.172142+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:00.272210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:00.272210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:16.088683+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"467'14","reported_seq":1067,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.206834+0000","last_change":"2024-09-18T07:47:20.169193+0000","last_active":"2024-09-18T07:49:41.206834+0000","last_peered":"2024-09-18T07:49:41.206834+0000","last_clean":"2024-09-18T07:49:41.206834+0000","last_became_active":"2024-09-18T07:47:20.141913+0000","last_became_peered":"2024-09-18T07:47:20.141913+0000","last_unstale":"2024-09-18T07:49:41.206834+0000","last_undegraded":"2024-09-18T07:49:41.206834+0000","last_fullsized":"2024-09-18T07:49:41.206834+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-18T07:40:40.352040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:40:40.352040+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:40.623629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":440,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172416+0000","last_change":"2024-09-18T07:48:32.259527+0000","last_active":"2024-09-18T07:49:41.172416+0000","last_peered":"2024-09-18T07:49:41.172416+0000","last_clean":"2024-09-18T07:49:41.172416+0000","last_became_active":"2024-09-18T07:48:32.259107+0000","last_became_peered":"2024-09-18T07:48:32.259107+0000","last_unstale":"2024-09-18T07:49:41.172416+0000","last_undegraded":"2024-09-18T07:49:41.172416+0000","last_fullsized":"2024-09-18T07:49:41.172416+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:45.971311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:45.971311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"323'19","reported_seq":1021,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169643+0000","last_change":"2024-09-18T07:47:20.196648+0000","last_active":"2024-09-18T07:49:41.169643+0000","last_peered":"2024-09-18T07:49:41.169643+0000","last_clean":"2024-09-18T07:49:41.169643+0000","last_became_active":"2024-09-18T07:47:20.140070+0000","last_became_peered":"2024-09-18T07:47:20.140070+0000","last_unstale":"2024-09-18T07:49:41.169643+0000","last_undegraded":"2024-09-18T07:49:41.169643+0000","last_fullsized":"2024-09-18T07:49:41.169643+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:41:54.022421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.022421+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:22.741685+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":935,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.169858+0000","last_change":"2024-09-18T07:46:06.964899+0000","last_active":"2024-09-18T07:49:41.169858+0000","last_peered":"2024-09-18T07:49:41.169858+0000","last_clean":"2024-09-18T07:49:41.169858+0000","last_became_active":"2024-09-18T07:44:43.780305+0000","last_became_peered":"2024-09-18T07:44:43.780305+0000","last_unstale":"2024-09-18T07:49:41.169858+0000","last_undegraded":"2024-09-18T07:49:41.169858+0000","last_fullsized":"2024-09-18T07:49:41.169858+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:06.964847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:06.964847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":607,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.204991+0000","last_change":"2024-09-18T07:48:10.163971+0000","last_active":"2024-09-18T07:49:41.204991+0000","last_peered":"2024-09-18T07:49:41.204991+0000","last_clean":"2024-09-18T07:49:41.204991+0000","last_became_active":"2024-09-18T07:48:10.163110+0000","last_became_peered":"2024-09-18T07:48:10.163110+0000","last_unstale":"2024-09-18T07:49:41.204991+0000","last_undegraded":"2024-09-18T07:49:41.204991+0000","last_fullsized":"2024-09-18T07:49:41.204991+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:59.022075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:59.022075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":507,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.197143+0000","last_change":"2024-09-18T07:48:25.249174+0000","last_active":"2024-09-18T07:49:41.197143+0000","last_peered":"2024-09-18T07:49:41.197143+0000","last_clean":"2024-09-18T07:49:41.197143+0000","last_became_active":"2024-09-18T07:48:10.154399+0000","last_became_peered":"2024-09-18T07:48:10.154399+0000","last_unstale":"2024-09-18T07:49:41.197143+0000","last_undegraded":"2024-09-18T07:49:41.197143+0000","last_fullsized":"2024-09-18T07:49:41.197143+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:25.249102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:25.249102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:50.673733+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"323'15","reported_seq":677,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.264635+0000","last_change":"2024-09-18T07:49:32.696134+0000","last_active":"2024-09-18T07:49:47.264635+0000","last_peered":"2024-09-18T07:49:47.264635+0000","last_clean":"2024-09-18T07:49:47.264635+0000","last_became_active":"2024-09-18T07:49:32.687721+0000","last_became_peered":"2024-09-18T07:49:32.687721+0000","last_unstale":"2024-09-18T07:49:47.264635+0000","last_undegraded":"2024-09-18T07:49:47.264635+0000","last_fullsized":"2024-09-18T07:49:47.264635+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:47:49.411931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:49.411931+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":774,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172054+0000","last_change":"2024-09-18T07:48:33.013859+0000","last_active":"2024-09-18T07:49:41.172054+0000","last_peered":"2024-09-18T07:49:41.172054+0000","last_clean":"2024-09-18T07:49:41.172054+0000","last_became_active":"2024-09-18T07:48:32.272079+0000","last_became_peered":"2024-09-18T07:48:32.272079+0000","last_unstale":"2024-09-18T07:49:41.172054+0000","last_undegraded":"2024-09-18T07:49:41.172054+0000","last_fullsized":"2024-09-18T07:49:41.172054+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:33.013808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:33.013808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:02.298988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":499,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.196644+0000","last_change":"2024-09-18T07:48:32.259701+0000","last_active":"2024-09-18T07:49:41.196644+0000","last_peered":"2024-09-18T07:49:41.196644+0000","last_clean":"2024-09-18T07:49:41.196644+0000","last_became_active":"2024-09-18T07:48:32.258708+0000","last_became_peered":"2024-09-18T07:48:32.258708+0000","last_unstale":"2024-09-18T07:49:41.196644+0000","last_undegraded":"2024-09-18T07:49:41.196644+0000","last_fullsized":"2024-09-18T07:49:41.196644+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:13.911901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:13.911901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:31.499215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1017,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.196527+0000","last_change":"2024-09-18T07:47:44.214937+0000","last_active":"2024-09-18T07:49:41.196527+0000","last_peered":"2024-09-18T07:49:41.196527+0000","last_clean":"2024-09-18T07:49:41.196527+0000","last_became_active":"2024-09-18T07:47:20.135880+0000","last_became_peered":"2024-09-18T07:47:20.135880+0000","last_unstale":"2024-09-18T07:49:41.196527+0000","last_undegraded":"2024-09-18T07:49:41.196527+0000","last_fullsized":"2024-09-18T07:49:41.196527+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:44.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:44.214865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:35.490659+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"323'19","reported_seq":745,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171331+0000","last_change":"2024-09-18T07:48:58.276742+0000","last_active":"2024-09-18T07:49:41.171331+0000","last_peered":"2024-09-18T07:49:41.171331+0000","last_clean":"2024-09-18T07:49:41.171331+0000","last_became_active":"2024-09-18T07:48:32.259858+0000","last_became_peered":"2024-09-18T07:48:32.259858+0000","last_unstale":"2024-09-18T07:49:41.171331+0000","last_undegraded":"2024-09-18T07:49:41.171331+0000","last_fullsized":"2024-09-18T07:49:41.171331+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:48:58.276659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:58.276659+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:10.277979+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":920,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.207022+0000","last_change":"2024-09-18T07:47:52.070685+0000","last_active":"2024-09-18T07:49:41.207022+0000","last_peered":"2024-09-18T07:49:41.207022+0000","last_clean":"2024-09-18T07:49:41.207022+0000","last_became_active":"2024-09-18T07:47:20.140495+0000","last_became_peered":"2024-09-18T07:47:20.140495+0000","last_unstale":"2024-09-18T07:49:41.207022+0000","last_undegraded":"2024-09-18T07:49:41.207022+0000","last_fullsized":"2024-09-18T07:49:41.207022+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:52.070602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.070602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":781,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172139+0000","last_change":"2024-09-18T07:46:13.660302+0000","last_active":"2024-09-18T07:49:41.172139+0000","last_peered":"2024-09-18T07:49:41.172139+0000","last_clean":"2024-09-18T07:49:41.172139+0000","last_became_active":"2024-09-18T07:44:43.778318+0000","last_became_peered":"2024-09-18T07:44:43.778318+0000","last_unstale":"2024-09-18T07:49:41.172139+0000","last_undegraded":"2024-09-18T07:49:41.172139+0000","last_fullsized":"2024-09-18T07:49:41.172139+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:13.660254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:13.660254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1004,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.198230+0000","last_change":"2024-09-18T07:49:24.443767+0000","last_active":"2024-09-18T07:49:41.198230+0000","last_peered":"2024-09-18T07:49:41.198230+0000","last_clean":"2024-09-18T07:49:41.198230+0000","last_became_active":"2024-09-18T07:49:24.443399+0000","last_became_peered":"2024-09-18T07:49:24.443399+0000","last_unstale":"2024-09-18T07:49:41.198230+0000","last_undegraded":"2024-09-18T07:49:41.198230+0000","last_fullsized":"2024-09-18T07:49:41.198230+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:33.311735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:33.311735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:02.660401+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"323'8","reported_seq":685,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.264550+0000","last_change":"2024-09-18T07:48:32.275394+0000","last_active":"2024-09-18T07:49:47.264550+0000","last_peered":"2024-09-18T07:49:47.264550+0000","last_clean":"2024-09-18T07:49:47.264550+0000","last_became_active":"2024-09-18T07:48:32.256691+0000","last_became_peered":"2024-09-18T07:48:32.256691+0000","last_unstale":"2024-09-18T07:49:47.264550+0000","last_undegraded":"2024-09-18T07:49:47.264550+0000","last_fullsized":"2024-09-18T07:49:47.264550+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'8","last_scrub_stamp":"2024-09-18T07:47:46.405595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:46.405595+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":166,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.207772+0000","last_change":"2024-09-18T07:48:32.266044+0000","last_active":"2024-09-18T07:49:41.207772+0000","last_peered":"2024-09-18T07:49:41.207772+0000","last_clean":"2024-09-18T07:49:41.207772+0000","last_became_active":"2024-09-18T07:48:32.265622+0000","last_became_peered":"2024-09-18T07:48:32.265622+0000","last_unstale":"2024-09-18T07:49:41.207772+0000","last_undegraded":"2024-09-18T07:49:41.207772+0000","last_fullsized":"2024-09-18T07:49:41.207772+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:04.066418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:04.066418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":621,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.268345+0000","last_change":"2024-09-18T07:47:55.363130+0000","last_active":"2024-09-18T07:49:47.268345+0000","last_peered":"2024-09-18T07:49:47.268345+0000","last_clean":"2024-09-18T07:49:47.268345+0000","last_became_active":"2024-09-18T07:44:43.792641+0000","last_became_peered":"2024-09-18T07:44:43.792641+0000","last_unstale":"2024-09-18T07:49:47.268345+0000","last_undegraded":"2024-09-18T07:49:47.268345+0000","last_fullsized":"2024-09-18T07:49:47.268345+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:55.363087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:55.363087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":759,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.171909+0000","last_change":"2024-09-18T07:44:43.783022+0000","last_active":"2024-09-18T07:49:41.171909+0000","last_peered":"2024-09-18T07:49:41.171909+0000","last_clean":"2024-09-18T07:49:41.171909+0000","last_became_active":"2024-09-18T07:44:43.782378+0000","last_became_peered":"2024-09-18T07:44:43.782378+0000","last_unstale":"2024-09-18T07:49:41.171909+0000","last_undegraded":"2024-09-18T07:49:41.171909+0000","last_fullsized":"2024-09-18T07:49:41.171909+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:40:38.492269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:40:38.492269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:48.284932+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":565,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.206880+0000","last_change":"2024-09-18T07:49:17.654443+0000","last_active":"2024-09-18T07:49:41.206880+0000","last_peered":"2024-09-18T07:49:41.206880+0000","last_clean":"2024-09-18T07:49:41.206880+0000","last_became_active":"2024-09-18T07:47:20.138598+0000","last_became_peered":"2024-09-18T07:47:20.138598+0000","last_unstale":"2024-09-18T07:49:41.206880+0000","last_undegraded":"2024-09-18T07:49:41.206880+0000","last_fullsized":"2024-09-18T07:49:41.206880+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:17.654398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:17.654398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:37.045609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":94,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.196883+0000","last_change":"2024-09-18T07:49:32.691139+0000","last_active":"2024-09-18T07:49:41.196883+0000","last_peered":"2024-09-18T07:49:41.196883+0000","last_clean":"2024-09-18T07:49:41.196883+0000","last_became_active":"2024-09-18T07:49:32.685882+0000","last_became_peered":"2024-09-18T07:49:32.685882+0000","last_unstale":"2024-09-18T07:49:41.196883+0000","last_undegraded":"2024-09-18T07:49:41.196883+0000","last_fullsized":"2024-09-18T07:49:41.196883+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:41.464337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:41.464337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:38.480805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"323'19","reported_seq":689,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-09-18T07:49:47.268337+0000","last_change":"2024-09-18T07:47:47.469967+0000","last_active":"2024-09-18T07:49:47.268337+0000","last_peered":"2024-09-18T07:49:47.268337+0000","last_clean":"2024-09-18T07:49:47.268337+0000","last_became_active":"2024-09-18T07:47:20.142007+0000","last_became_peered":"2024-09-18T07:47:20.142007+0000","last_unstale":"2024-09-18T07:49:47.268337+0000","last_undegraded":"2024-09-18T07:49:47.268337+0000","last_fullsized":"2024-09-18T07:49:47.268337+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:47:47.469902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:47.469902+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":521,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-18T07:49:41.172374+0000","last_change":"2024-09-18T07:48:28.910567+0000","last_active":"2024-09-18T07:49:41.172374+0000","last_peered":"2024-09-18T07:49:41.172374+0000","last_clean":"2024-09-18T07:49:41.172374+0000","last_became_active":"2024-09-18T07:47:20.139668+0000","last_became_peered":"2024-09-18T07:47:20.139668+0000","last_unstale":"2024-09-18T07:49:41.172374+0000","last_undegraded":"2024-09-18T07:49:41.172374+0000","last_fullsized":"2024-09-18T07:49:41.172374+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:28.910419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:28.910419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:54.424567+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3516989464,"num_objects":53666,"num_object_clones":0,"num_object_copies":160998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3275,"num_objects_unfound":0,"num_objects_dirty":53666,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170638,"num_write_kb":7177665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44482,"num_bytes_recovered":2914975744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12522172416,"data_stored":12522160200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72504,"ondisk_log_size":72504,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1096,"num_read_kb":853,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":565,"num_bytes_recovered":12146,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":479232,"data_stored":12674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},"log_size":690,"ondisk_log_size":690,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2652,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":5878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":887,"num_read_kb":882,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201510,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2195892,"kb_used_data":1519140,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":91549260,"statfs":{"total":95995035648,"available":93746442240,"internally_reserved":0,"allocated":1555599360,"data_stored":1555196968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":692976074},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463148,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2646788,"kb_used_data":1933812,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":91098364,"statfs":{"total":95995035648,"available":93284724736,"internally_reserved":0,"allocated":1980223488,"data_stored":1979804819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":729938374},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":460,"seq":1975684956192,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1206952,"kb_used_data":1088408,"kb_used_omap":6,"kb_used_meta":118521,"kb_avail":92538200,"statfs":{"total":95995035648,"available":94759116800,"internally_reserved":0,"allocated":1114529792,"data_stored":1114139929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6877,"internal_metadata":121365795},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":395,"seq":1696512081982,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1600824,"kb_used_data":1103912,"kb_used_omap":4,"kb_used_meta":496891,"kb_avail":92144328,"statfs":{"total":95995035648,"available":94355791872,"internally_reserved":0,"allocated":1130405888,"data_stored":1129999147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4447,"internal_metadata":508817057},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669149948,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411586,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2501292,"kb_used_data":1748188,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":91243860,"statfs":{"total":95995035648,"available":93433712640,"internally_reserved":0,"allocated":1790144512,"data_stored":1789751046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":771160517},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575113,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2748256,"kb_used_data":1918672,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":90996896,"statfs":{"total":95995035648,"available":93180821504,"internally_reserved":0,"allocated":1964720128,"data_stored":1964274360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":849344965},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738637,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2300096,"kb_used_data":1412328,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":91445056,"statfs":{"total":95995035648,"available":93639737344,"internally_reserved":0,"allocated":1446223872,"data_stored":1445818399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1416429568,"data_stored":1416429568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1962803200,"data_stored":1962803200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1753419776,"data_stored":1753415704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1845624832,"data_stored":1845624832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1109921792,"data_stored":1109917720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":929759232,"data_stored":929759232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1966018560,"data_stored":1966014488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1538195456,"data_stored":1538195456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:49:49.866 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-09-18T07:49:49.924 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:49.956 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:49.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:49.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:49.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:49:50.058 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:50.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:50.108 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:50.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:50.210 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:50.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:50.236 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:50.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:50.337 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:50.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:50.371 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:50.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:50.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:50 smithi137 ceph-mon[29408]: from='client.26779 v1:172.21.15.137:0/562821490' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:49:50.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:50 smithi137 ceph-mon[39175]: from='client.26779 v1:172.21.15.137:0/562821490' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:49:50.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:50.497 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:49:50.497 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:49:50.498 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:50.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:50.532 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:49:50.532 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:49:50.571 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:49:50.571 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:49:50.592 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:50.600 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:50.606 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:49:50.606 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:49:50.632 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:50.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:50.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:50.636 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:49:50.636 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:49:50.643 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:50.670 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:49:50.670 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:49:50.705 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:49:50.705 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:49:50.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:50 smithi203 ceph-mon[32237]: from='client.26779 v1:172.21.15.137:0/562821490' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:49:50.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:50.741 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:49:50.741 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:49:50.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:50.786 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:50.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:50.887 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:50.924 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:50.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:50.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:51.026 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:51.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:51.073 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:51.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:51.174 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:51.198 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:51.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:51.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:51.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:51.342 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:51.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:51.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:51.444 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:51.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:51 smithi203 ceph-mon[32237]: 6.12 scrub starts 2024-09-18T07:49:51.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:51 smithi203 ceph-mon[32237]: 6.12 scrub ok 2024-09-18T07:49:51.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:51 smithi203 ceph-mon[32237]: pgmap v1011: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.28k op/s; 3275/160413 objects misplaced (2.042%); 6.8 MiB/s, 109 objects/s recovering 2024-09-18T07:49:51.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:51 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1182017113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:51.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:51 smithi203 ceph-mon[32237]: 11.1 scrub starts 2024-09-18T07:49:51.486 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:51.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:51.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:51.521 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:51 smithi137 ceph-mon[39175]: 6.12 scrub starts 2024-09-18T07:49:51.521 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:51 smithi137 ceph-mon[39175]: 6.12 scrub ok 2024-09-18T07:49:51.521 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:51 smithi137 ceph-mon[39175]: pgmap v1011: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.28k op/s; 3275/160413 objects misplaced (2.042%); 6.8 MiB/s, 109 objects/s recovering 2024-09-18T07:49:51.522 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:51 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1182017113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:51.522 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:51 smithi137 ceph-mon[39175]: 11.1 scrub starts 2024-09-18T07:49:51.522 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:51 smithi137 ceph-mon[29408]: 6.12 scrub starts 2024-09-18T07:49:51.522 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:51 smithi137 ceph-mon[29408]: 6.12 scrub ok 2024-09-18T07:49:51.522 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:51 smithi137 ceph-mon[29408]: pgmap v1011: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.28k op/s; 3275/160413 objects misplaced (2.042%); 6.8 MiB/s, 109 objects/s recovering 2024-09-18T07:49:51.522 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:51 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1182017113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:51.522 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:51 smithi137 ceph-mon[29408]: 11.1 scrub starts 2024-09-18T07:49:51.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:51.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:51.623 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:51.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:51.724 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:51.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:51.759 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:51.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:51.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:51.890 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:51.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:51.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:51.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:52.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:52.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:52.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:52.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:52.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:52.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:52.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:52.258 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:49:52.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:52.306 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:52.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:52.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:52.408 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:52.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:52.445 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:52.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:52.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:52 smithi203 ceph-mon[32237]: 5.14 scrub starts 2024-09-18T07:49:52.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:52 smithi203 ceph-mon[32237]: 5.14 scrub ok 2024-09-18T07:49:52.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:52 smithi203 ceph-mon[32237]: 4.4 scrub starts 2024-09-18T07:49:52.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:52 smithi203 ceph-mon[32237]: 4.4 scrub ok 2024-09-18T07:49:52.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:52 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1918956996' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:49:52.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:52 smithi203 ceph-mon[32237]: from='client.36668 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:49:52.480 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:52 smithi137 ceph-mon[29408]: 5.14 scrub starts 2024-09-18T07:49:52.480 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:52 smithi137 ceph-mon[29408]: 5.14 scrub ok 2024-09-18T07:49:52.480 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:52 smithi137 ceph-mon[29408]: 4.4 scrub starts 2024-09-18T07:49:52.480 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:52 smithi137 ceph-mon[29408]: 4.4 scrub ok 2024-09-18T07:49:52.480 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:52 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1918956996' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:49:52.480 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:52 smithi137 ceph-mon[29408]: from='client.36668 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:49:52.481 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:52 smithi137 ceph-mon[39175]: 5.14 scrub starts 2024-09-18T07:49:52.481 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:52 smithi137 ceph-mon[39175]: 5.14 scrub ok 2024-09-18T07:49:52.481 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:52 smithi137 ceph-mon[39175]: 4.4 scrub starts 2024-09-18T07:49:52.481 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:52 smithi137 ceph-mon[39175]: 4.4 scrub ok 2024-09-18T07:49:52.481 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:52 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1918956996' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:49:52.481 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:52 smithi137 ceph-mon[39175]: from='client.36668 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:49:52.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:52.586 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:52.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:52.687 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:52.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:52.718 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:52.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:52.819 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:52.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:52.857 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:52.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:52.958 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:52.990 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:52.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:52.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:53.092 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:53.125 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:53.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:53.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:53.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T07:49:53.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:53.251 INFO:teuthology.orchestra.run.smithi137.stderr:set pool 1 pgp_num to 11 2024-09-18T07:49:53.257 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:53.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:53.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:53.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:53.397 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:53.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:53.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:53 smithi203 ceph-mon[32237]: 3.8 scrub starts 2024-09-18T07:49:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:53 smithi203 ceph-mon[32237]: 3.8 scrub ok 2024-09-18T07:49:53.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:53 smithi203 ceph-mon[32237]: pgmap v1012: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s; 3275/149721 objects misplaced (2.187%) 2024-09-18T07:49:53.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:53 smithi203 ceph-mon[32237]: 6.6 scrub starts 2024-09-18T07:49:53.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:53 smithi203 ceph-mon[32237]: 6.6 scrub ok 2024-09-18T07:49:53.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:53 smithi203 ceph-mon[32237]: 11.1 scrub ok 2024-09-18T07:49:53.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:53 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:49:53.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:53 smithi203 ceph-mon[32237]: from='client.36668 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:49:53.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:53 smithi203 ceph-mon[32237]: osdmap e530: 8 total, 8 up, 7 in 2024-09-18T07:49:53.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:53 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1923210392' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-18T07:49:53.499 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:53.526 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:53.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:53.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:53.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:53.633 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:49:53.664 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:53.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:53.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[29408]: 3.8 scrub starts 2024-09-18T07:49:53.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[29408]: 3.8 scrub ok 2024-09-18T07:49:53.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[29408]: pgmap v1012: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s; 3275/149721 objects misplaced (2.187%) 2024-09-18T07:49:53.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[29408]: 6.6 scrub starts 2024-09-18T07:49:53.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[29408]: 6.6 scrub ok 2024-09-18T07:49:53.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[29408]: 11.1 scrub ok 2024-09-18T07:49:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:49:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[29408]: from='client.36668 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:49:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[29408]: osdmap e530: 8 total, 8 up, 7 in 2024-09-18T07:49:53.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1923210392' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-18T07:49:53.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[39175]: 3.8 scrub starts 2024-09-18T07:49:53.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[39175]: 3.8 scrub ok 2024-09-18T07:49:53.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[39175]: pgmap v1012: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s; 3275/149721 objects misplaced (2.187%) 2024-09-18T07:49:53.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[39175]: 6.6 scrub starts 2024-09-18T07:49:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[39175]: 6.6 scrub ok 2024-09-18T07:49:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[39175]: 11.1 scrub ok 2024-09-18T07:49:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:49:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[39175]: from='client.36668 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:49:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[39175]: osdmap e530: 8 total, 8 up, 7 in 2024-09-18T07:49:53.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:53 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1923210392' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-18T07:49:53.766 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:53.802 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:53.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:53.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:53.904 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:53.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:53.954 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:53.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:54.055 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:54.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:54.097 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:54.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:54.198 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:54.239 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:54.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:54.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:54.255 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:49:54.255 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:49:54.255 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.255 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.16581.0:179618 11.2 e531/522 11:443f8824:::benchmark_data_smithi137_181069_object67464:head v 531'11025, mlcod=531'11021)", 2024-09-18T07:49:54.255 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:49:54.244696+0000", 2024-09-18T07:49:54.255 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0072907520000000002, 2024-09-18T07:49:54.255 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0073163739999999996, 2024-09-18T07:49:54.255 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:49:54.255 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:49:54.255 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:49:54.256 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:49:54.256 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.256 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:49:54.256 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.244696+0000", 2024-09-18T07:49:54.256 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:54.256 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.256 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.256 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:49:54.256 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.244696+0000", 2024-09-18T07:49:54.256 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:54.256 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.256 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.256 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:49:54.257 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.244697+0000", 2024-09-18T07:49:54.257 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6330000000000001e-06 2024-09-18T07:49:54.257 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.257 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.257 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:49:54.257 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.244704+0000", 2024-09-18T07:49:54.257 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.973e-06 2024-09-18T07:49:54.257 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.257 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.257 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:49:54.257 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.244715+0000", 2024-09-18T07:49:54.257 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0169999999999999e-05 2024-09-18T07:49:54.257 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.258 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.258 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:49:54.258 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.244722+0000", 2024-09-18T07:49:54.258 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.0929999999999997e-06 2024-09-18T07:49:54.258 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.258 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.258 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:49:54.258 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.244798+0000", 2024-09-18T07:49:54.258 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6867999999999997e-05 2024-09-18T07:49:54.258 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.258 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.258 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:49:54.258 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.244818+0000", 2024-09-18T07:49:54.259 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9638000000000001e-05 2024-09-18T07:49:54.259 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:54.259 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:49:54.259 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:54.259 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.259 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.259 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.16581.0:179626 11.2 e531/522 11:467a100b:::benchmark_data_smithi137_181069_object67472:head v 531'11026, mlcod=531'11021)", 2024-09-18T07:49:54.259 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:49:54.247034+0000", 2024-09-18T07:49:54.259 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0049525130000000004, 2024-09-18T07:49:54.259 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0050180520000000003, 2024-09-18T07:49:54.259 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:49:54.259 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:49:54.259 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:49:54.260 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:49:54.260 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.260 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:49:54.260 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.247034+0000", 2024-09-18T07:49:54.260 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:54.260 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.260 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.260 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:49:54.260 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.247034+0000", 2024-09-18T07:49:54.260 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:54.260 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.260 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.261 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:49:54.261 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.247035+0000", 2024-09-18T07:49:54.261 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.074e-06 2024-09-18T07:49:54.261 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.261 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.261 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:49:54.261 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.247041+0000", 2024-09-18T07:49:54.261 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1149999999999999e-06 2024-09-18T07:49:54.261 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.261 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.261 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:49:54.261 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.247049+0000", 2024-09-18T07:49:54.261 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2479999999999996e-06 2024-09-18T07:49:54.262 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.262 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.262 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:49:54.262 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.247055+0000", 2024-09-18T07:49:54.262 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8810000000000001e-06 2024-09-18T07:49:54.262 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.262 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.262 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:49:54.262 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.247286+0000", 2024-09-18T07:49:54.262 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023117499999999999 2024-09-18T07:49:54.262 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.262 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.262 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:49:54.263 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.247308+0000", 2024-09-18T07:49:54.263 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1659e-05 2024-09-18T07:49:54.263 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:54.263 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:49:54.263 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:54.263 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.263 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.263 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.16581.0:179625 11.3 e531/475 11:c907b638:::benchmark_data_smithi137_181069_object67471:head v 531'11165, mlcod=531'11161)", 2024-09-18T07:49:54.263 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:49:54.249021+0000", 2024-09-18T07:49:54.263 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.002965479, 2024-09-18T07:49:54.263 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0030812779999999998, 2024-09-18T07:49:54.263 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:49:54.263 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:49:54.263 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:49:54.264 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:49:54.264 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.264 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:49:54.264 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.249021+0000", 2024-09-18T07:49:54.264 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:54.264 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.264 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.264 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:49:54.264 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.249021+0000", 2024-09-18T07:49:54.264 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:54.264 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.264 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.264 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:49:54.264 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.249022+0000", 2024-09-18T07:49:54.265 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3850000000000001e-06 2024-09-18T07:49:54.265 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.265 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.265 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:49:54.265 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.249029+0000", 2024-09-18T07:49:54.265 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8410000000000002e-06 2024-09-18T07:49:54.265 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.265 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.265 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:49:54.265 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.249043+0000", 2024-09-18T07:49:54.265 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3988000000000001e-05 2024-09-18T07:49:54.265 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.265 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.265 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:49:54.266 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.249051+0000", 2024-09-18T07:49:54.266 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3159999999999999e-06 2024-09-18T07:49:54.266 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.266 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.266 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:49:54.266 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.249313+0000", 2024-09-18T07:49:54.266 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00026238200000000002 2024-09-18T07:49:54.266 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.266 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.266 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:49:54.266 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.249330+0000", 2024-09-18T07:49:54.266 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7348000000000001e-05 2024-09-18T07:49:54.266 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:54.266 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:49:54.267 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:54.267 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.267 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.267 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_repop(client.16581.0:179630 11.2 e531/522 11:48cb4ac1:::benchmark_data_smithi137_181069_object67476:head v 531'11027, mlcod=531'11021)", 2024-09-18T07:49:54.267 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:49:54.249822+0000", 2024-09-18T07:49:54.267 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 0.0021648489999999999, 2024-09-18T07:49:54.267 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023265849999999999, 2024-09-18T07:49:54.267 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:49:54.267 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:49:54.267 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:49:54.267 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:49:54.267 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.267 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:49:54.267 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.249822+0000", 2024-09-18T07:49:54.268 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:54.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.268 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.268 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:49:54.268 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.249822+0000", 2024-09-18T07:49:54.268 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:49:54.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.268 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.268 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:49:54.268 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.249823+0000", 2024-09-18T07:49:54.268 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.47e-07 2024-09-18T07:49:54.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.268 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.268 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:49:54.268 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.249828+0000", 2024-09-18T07:49:54.269 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6339999999999998e-06 2024-09-18T07:49:54.269 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.269 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.269 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:49:54.269 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.249837+0000", 2024-09-18T07:49:54.269 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8270000000000004e-06 2024-09-18T07:49:54.269 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.269 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.269 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:49:54.269 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.249843+0000", 2024-09-18T07:49:54.269 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8529999999999997e-06 2024-09-18T07:49:54.269 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.269 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.269 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:49:54.270 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.249893+0000", 2024-09-18T07:49:54.270 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9620000000000003e-05 2024-09-18T07:49:54.270 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:49:54.270 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:49:54.270 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:49:54.270 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:49:54.249924+0000", 2024-09-18T07:49:54.270 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1535000000000002e-05 2024-09-18T07:49:54.270 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:54.270 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:49:54.270 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:54.270 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:49:54.270 INFO:teuthology.orchestra.run.smithi203.stdout: ], 2024-09-18T07:49:54.270 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 4 2024-09-18T07:49:54.270 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:49:54.271 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:49:54.341 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:54.374 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:54.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:54.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:54.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:54 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1923210392' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-18T07:49:54.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:54 smithi203 ceph-mon[32237]: osdmap e531: 8 total, 8 up, 7 in 2024-09-18T07:49:54.476 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:54.513 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:54.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:54.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:54.615 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:54.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:54.649 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:54.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:54.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:54 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1923210392' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-18T07:49:54.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:54 smithi137 ceph-mon[29408]: osdmap e531: 8 total, 8 up, 7 in 2024-09-18T07:49:54.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:54 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1923210392' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-18T07:49:54.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:54 smithi137 ceph-mon[39175]: osdmap e531: 8 total, 8 up, 7 in 2024-09-18T07:49:54.750 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:54.778 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:54.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:54.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:54.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:54.913 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:54.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:54.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:55.015 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:55.047 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:55.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:55.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:55.138 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T07:49:55.149 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:55.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:55.219 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:55.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:55.320 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:55.361 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:55.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:55.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:55.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:49:55.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:55 smithi203 ceph-mon[32237]: pgmap v1015: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 766 B/s rd, 0 B/s wr, 1.16k op/s; 2918/144390 objects misplaced (2.021%) 2024-09-18T07:49:55.463 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:55.509 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:55.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:55.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:55.577 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:49:55.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:55.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:55.654 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:55.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:55.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:55 smithi137 ceph-mon[29408]: pgmap v1015: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 766 B/s rd, 0 B/s wr, 1.16k op/s; 2918/144390 objects misplaced (2.021%) 2024-09-18T07:49:55.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:55 smithi137 ceph-mon[39175]: pgmap v1015: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 766 B/s rd, 0 B/s wr, 1.16k op/s; 2918/144390 objects misplaced (2.021%) 2024-09-18T07:49:55.755 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:55.766 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:55.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:55.795 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:55.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:55.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:55.931 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:55.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:55.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:56.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:56.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:56.070 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:56.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:56.080 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:49:56.156 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:49:56.156 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:49:56.156 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:49:56.156 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:49:56.156 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:49:56.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:56.205 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:56.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:56.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:56.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:56.345 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:56.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:56.412 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:49:56.412 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:49:56.442 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:49:56.443 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:49:56.446 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:56.474 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:49:56.474 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:49:56.482 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:56.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:56.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:56.506 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:49:56.506 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:49:56.538 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:49:56.538 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:49:56.571 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:49:56.571 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:49:56.584 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:56.607 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:49:56.607 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:49:56.630 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:56.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:56.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:56.640 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:49:56.640 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:49:56.732 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:56.772 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:56.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:56.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:56.874 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:49:56.917 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:56.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:56.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:49:57.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:57.051 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T07:49:57.064 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:57.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:57.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:57.165 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:57.207 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:57.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:57.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:57.296 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:49:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:49:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:49:57.309 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:49:57.345 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:57.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:57.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:49:57.448 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:57.476 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:49:57.482 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:57.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:57.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:57.585 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:57.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:57 smithi203 ceph-mon[32237]: pgmap v1016: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 763 op/s; 2918/143277 objects misplaced (2.037%) 2024-09-18T07:49:57.612 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:57 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2600768208' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:57.613 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:57 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:49:57.613 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:57 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:49:57.622 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:57.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:57.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:57.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:57 smithi137 ceph-mon[29408]: pgmap v1016: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 763 op/s; 2918/143277 objects misplaced (2.037%) 2024-09-18T07:49:57.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:57 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2600768208' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:57.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:57 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:49:57.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:57 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:49:57.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:57 smithi137 ceph-mon[39175]: pgmap v1016: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 763 op/s; 2918/143277 objects misplaced (2.037%) 2024-09-18T07:49:57.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:57 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2600768208' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:49:57.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:57 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:49:57.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:57 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:49:57.723 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:57.765 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:57.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:57.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:57.867 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:57.907 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:57.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:57.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:58.009 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:58.054 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:58.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:58.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:58.141 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:49:58.141 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:49:58.156 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:58.187 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:58.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:58.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:58.289 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:58.311 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:49:58 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:49:58.317 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:49:58.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:58.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:58.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:58.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:58.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:58.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:58.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:58.559 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:58.591 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:58.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:58.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:58.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:58 smithi137 ceph-mon[29408]: osdmap e532: 8 total, 8 up, 7 in 2024-09-18T07:49:58.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:58 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:49:58.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:58 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/521737139' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:58.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:58 smithi137 ceph-mon[29408]: from='client.26809 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:58.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:58 smithi137 ceph-mon[39175]: osdmap e532: 8 total, 8 up, 7 in 2024-09-18T07:49:58.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:58 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:49:58.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:58 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/521737139' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:58.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:58 smithi137 ceph-mon[39175]: from='client.26809 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:58.693 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:58.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:58 smithi203 ceph-mon[32237]: osdmap e532: 8 total, 8 up, 7 in 2024-09-18T07:49:58.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:58 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:49:58.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:58 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/521737139' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:58.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:58 smithi203 ceph-mon[32237]: from='client.26809 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:49:58.727 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:58.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:58.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:58.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:58.852 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T07:49:58.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:58.864 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:58.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:58.965 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:58.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:58.996 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:58.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:59.097 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:49:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:59.133 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:59.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:49:59.234 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:49:59.269 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:59.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:59.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:49:59.271 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:49:59.370 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:49:59.402 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:59.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:59.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:49:59.464 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:59 smithi203 ceph-mon[32237]: pgmap v1018: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.42k op/s; 2464/124227 objects misplaced (1.983%) 2024-09-18T07:49:59.464 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:59 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:59.464 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:59 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:49:59.464 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:59 smithi203 ceph-mon[32237]: from='client.26809 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:49:59.464 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:49:59 smithi203 ceph-mon[32237]: osdmap e533: 8 total, 8 up, 7 in 2024-09-18T07:49:59.504 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:49:59.539 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:49:59.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:59.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:49:59.641 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:59.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:59.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:49:59.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:59 smithi137 ceph-mon[29408]: pgmap v1018: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.42k op/s; 2464/124227 objects misplaced (1.983%) 2024-09-18T07:49:59.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:59 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:59.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:59 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:49:59.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:59 smithi137 ceph-mon[29408]: from='client.26809 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:49:59.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:49:59 smithi137 ceph-mon[29408]: osdmap e533: 8 total, 8 up, 7 in 2024-09-18T07:49:59.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:59 smithi137 ceph-mon[39175]: pgmap v1018: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.42k op/s; 2464/124227 objects misplaced (1.983%) 2024-09-18T07:49:59.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:59 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:49:59.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:59 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:49:59.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:59 smithi137 ceph-mon[39175]: from='client.26809 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:49:59.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:49:59 smithi137 ceph-mon[39175]: osdmap e533: 8 total, 8 up, 7 in 2024-09-18T07:49:59.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:49:59.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:49:59.811 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:59.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:59.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:49:59.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:49:59.919 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:49:59.919 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:49:59.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:49:59.954 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:49:59.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:00.055 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:00.097 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:00.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:00.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:00.119 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:00.198 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:00.242 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:00.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:00.344 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:00.377 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:00.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:00.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:00 smithi203 ceph-mon[32237]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-18T07:50:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:00 smithi203 ceph-mon[32237]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-18T07:50:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:00 smithi203 ceph-mon[32237]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-18T07:50:00.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:00 smithi203 ceph-mon[32237]: pg 11.a is active+remapped+backfill_toofull, acting [6,5,2] 2024-09-18T07:50:00.478 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:00.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:00.511 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:00.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:00.521 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:00 smithi137 ceph-mon[29408]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-18T07:50:00.522 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:00 smithi137 ceph-mon[29408]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-18T07:50:00.522 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:00 smithi137 ceph-mon[29408]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-18T07:50:00.522 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:00 smithi137 ceph-mon[29408]: pg 11.a is active+remapped+backfill_toofull, acting [6,5,2] 2024-09-18T07:50:00.522 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:00 smithi137 ceph-mon[39175]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-18T07:50:00.522 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:00 smithi137 ceph-mon[39175]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-18T07:50:00.522 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:00 smithi137 ceph-mon[39175]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-18T07:50:00.522 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:00 smithi137 ceph-mon[39175]: pg 11.a is active+remapped+backfill_toofull, acting [6,5,2] 2024-09-18T07:50:00.613 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:00.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:00.655 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:00.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:00.746 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 4, 5, 7, 6, 1] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 6, 7, 4, 5] 2024-09-18T07:50:00.747 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:50:00.747 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 2, 4, 5, 7, 6, 1] 2024-09-18T07:50:00.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd out 0 2024-09-18T07:50:00.749 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T07:50:00.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:00.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:00.825 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:00.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:00.826 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:50:00.826 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:50:00.869 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:50:00.869 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:50:00.906 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:50:00.906 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:50:00.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:00.937 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:50:00.938 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:50:00.967 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:00.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:00.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:00.978 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:50:00.978 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:50:01.013 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:50:01.013 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:50:01.056 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:50:01.057 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:50:01.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:01.100 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:50:01.101 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:50:01.132 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:01.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:01.166 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:50:01.234 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:01.279 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:01.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:01.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:01.383 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:01.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:50:01.414 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:01.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:01.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:01.516 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:01.538 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:01 smithi137 ceph-mon[39175]: pgmap v1020: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 771 B/s rd, 0 B/s wr, 1.08k op/s; 2464/122922 objects misplaced (2.005%) 2024-09-18T07:50:01.538 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:01 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2474411208' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:01.538 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:01 smithi137 ceph-mon[29408]: pgmap v1020: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 771 B/s rd, 0 B/s wr, 1.08k op/s; 2464/122922 objects misplaced (2.005%) 2024-09-18T07:50:01.538 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:01 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2474411208' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:01.557 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:01.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:01.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:01.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:01 smithi203 ceph-mon[32237]: pgmap v1020: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 771 B/s rd, 0 B/s wr, 1.08k op/s; 2464/122922 objects misplaced (2.005%) 2024-09-18T07:50:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:01 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2474411208' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:01.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:01.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:01.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:01.815 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:01.842 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:01.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:01.944 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:01.972 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:01.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:01.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:02.077 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:02.107 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:02.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:02.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:02.253 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:02.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:02.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:02.355 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:02.370 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:50:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:50:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:50:02.371 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:50:02.371 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:50:02.401 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:02.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:02.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:02.503 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:02.537 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:02.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:02.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:02.639 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:02.657 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:02.670 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:02.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:02.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:02.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:02.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:02.818 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:02.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:02.919 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:02.951 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:02.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:02.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:03.053 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:03.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:03.090 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:03.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:03.100 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T07:50:03.192 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:03.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:03.237 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:03.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:03.271 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:03.338 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:03.530 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:03.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:03.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:03.538 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:50:03.562 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:03 smithi137 ceph-mon[29408]: pgmap v1021: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s; 2464/115434 objects misplaced (2.135%) 2024-09-18T07:50:03.562 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:03 smithi137 ceph-mon[39175]: pgmap v1021: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s; 2464/115434 objects misplaced (2.135%) 2024-09-18T07:50:03.632 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:03.663 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:03.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:03.672 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:03 smithi203 ceph-mon[32237]: pgmap v1021: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s; 2464/115434 objects misplaced (2.135%) 2024-09-18T07:50:03.765 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:03.798 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:03.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:03.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:03.856 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:03 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a[29404]: 2024-09-18T07:50:03.655+0000 7fe5973a1640 -1 mon.a@0(leader).osd e533 definitely_dead 0 2024-09-18T07:50:03.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:03.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:03.938 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:03.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:04.039 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:04.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:04.074 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:04.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:04.149 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:50:04.149 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:50:04.149 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:50:04.149 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:50:04.149 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.149 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189537 10.2 10:41c597a1:::benchmark_data_smithi137_151160_object53721:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:50:04.149 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.301971+0000", 2024-09-18T07:50:04.150 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 270.84189294800001, 2024-09-18T07:50:04.150 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0664383530000001, 2024-09-18T07:50:04.150 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.150 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.150 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:50:04.150 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.150 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.150 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.150 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189537 2024-09-18T07:50:04.150 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.150 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.150 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.150 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.151 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.301971+0000", 2024-09-18T07:50:04.151 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.151 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.151 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.151 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.151 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.301971+0000", 2024-09-18T07:50:04.151 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.151 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.151 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.151 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.151 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.301973+0000", 2024-09-18T07:50:04.151 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.99e-06 2024-09-18T07:50:04.151 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.152 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.152 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.152 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.301979+0000", 2024-09-18T07:50:04.152 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4389999999999999e-06 2024-09-18T07:50:04.152 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.152 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.152 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.152 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.301995+0000", 2024-09-18T07:50:04.152 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6220000000000001e-05 2024-09-18T07:50:04.152 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.152 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.152 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.153 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302003+0000", 2024-09-18T07:50:04.153 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0020000000000006e-06 2024-09-18T07:50:04.153 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.153 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.153 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.153 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302091+0000", 2024-09-18T07:50:04.153 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7639000000000006e-05 2024-09-18T07:50:04.153 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.153 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.153 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.154 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302098+0000", 2024-09-18T07:50:04.154 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5989999999999996e-06 2024-09-18T07:50:04.154 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.154 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.154 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.154 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.368329+0000", 2024-09-18T07:50:04.154 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0662302459999999 2024-09-18T07:50:04.154 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.154 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.154 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.154 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.368410+0000", 2024-09-18T07:50:04.154 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1217999999999994e-05 2024-09-18T07:50:04.154 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.154 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.155 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.155 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.155 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.155 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189549 10.2 10:4a928452:::benchmark_data_smithi137_151160_object53733:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:50:04.155 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.302028+0000", 2024-09-18T07:50:04.155 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 270.84183620900001, 2024-09-18T07:50:04.155 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.100603623, 2024-09-18T07:50:04.155 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.155 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.155 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:50:04.155 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.155 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.155 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.156 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189549 2024-09-18T07:50:04.156 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.156 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.156 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.156 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.156 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302028+0000", 2024-09-18T07:50:04.156 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.156 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.156 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.156 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.156 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302028+0000", 2024-09-18T07:50:04.156 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.157 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.157 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.157 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302029+0000", 2024-09-18T07:50:04.157 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3399999999999998e-07 2024-09-18T07:50:04.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.157 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.157 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.157 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302033+0000", 2024-09-18T07:50:04.157 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4710000000000001e-06 2024-09-18T07:50:04.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.157 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.157 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.158 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302041+0000", 2024-09-18T07:50:04.158 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9279999999999993e-06 2024-09-18T07:50:04.158 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.158 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.158 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.158 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302047+0000", 2024-09-18T07:50:04.158 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0789999999999997e-06 2024-09-18T07:50:04.158 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.158 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.158 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.158 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302133+0000", 2024-09-18T07:50:04.158 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5400000000000002e-05 2024-09-18T07:50:04.158 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.159 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.159 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.159 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.302138+0000", 2024-09-18T07:50:04.159 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5419999999999996e-06 2024-09-18T07:50:04.159 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.159 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.159 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.159 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.368614+0000", 2024-09-18T07:50:04.159 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0664751509999999 2024-09-18T07:50:04.159 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.159 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.159 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for missing object", 2024-09-18T07:50:04.159 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.368706+0000", 2024-09-18T07:50:04.160 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2783999999999998e-05 2024-09-18T07:50:04.160 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.160 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.160 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.160 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.402529+0000", 2024-09-18T07:50:04.160 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.033822673999999997 2024-09-18T07:50:04.160 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.160 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.160 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.160 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.402632+0000", 2024-09-18T07:50:04.160 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010276 2024-09-18T07:50:04.160 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.160 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.161 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.161 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.161 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.161 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189555 10.2 10:4c34833c:::benchmark_data_smithi137_151160_object53739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:50:04.161 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.305399+0000", 2024-09-18T07:50:04.161 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 270.83846581, 2024-09-18T07:50:04.161 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.075830737, 2024-09-18T07:50:04.161 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.161 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.161 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:04.161 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.161 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.161 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.162 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189555 2024-09-18T07:50:04.162 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.162 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.162 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.162 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.162 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.305399+0000", 2024-09-18T07:50:04.162 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.162 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.162 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.162 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.162 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.305399+0000", 2024-09-18T07:50:04.162 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.162 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.163 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.163 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.163 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.305400+0000", 2024-09-18T07:50:04.163 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4890000000000001e-06 2024-09-18T07:50:04.163 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.163 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.163 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.163 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.305405+0000", 2024-09-18T07:50:04.163 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0760000000000002e-06 2024-09-18T07:50:04.163 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.163 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.163 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.163 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.305415+0000", 2024-09-18T07:50:04.163 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0077e-05 2024-09-18T07:50:04.164 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.164 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.164 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.164 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.305420+0000", 2024-09-18T07:50:04.164 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0819999999999998e-06 2024-09-18T07:50:04.164 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.165 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.165 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.165 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.305472+0000", 2024-09-18T07:50:04.165 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2182999999999998e-05 2024-09-18T07:50:04.165 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.165 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.165 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.165 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.305482+0000", 2024-09-18T07:50:04.165 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0860000000000008e-06 2024-09-18T07:50:04.165 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.166 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.166 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.166 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.368768+0000", 2024-09-18T07:50:04.166 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0632862219999999 2024-09-18T07:50:04.166 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.166 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.166 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:04.166 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.368997+0000", 2024-09-18T07:50:04.166 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000229535 2024-09-18T07:50:04.166 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.166 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.166 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:50:04.166 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.369369+0000", 2024-09-18T07:50:04.167 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00037124899999999998 2024-09-18T07:50:04.167 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.167 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.167 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.167 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.379199+0000", 2024-09-18T07:50:04.167 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0098309239999999996 2024-09-18T07:50:04.167 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.167 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.167 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.167 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.379274+0000", 2024-09-18T07:50:04.167 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4609000000000003e-05 2024-09-18T07:50:04.167 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.168 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.168 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:04.168 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.381069+0000", 2024-09-18T07:50:04.168 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0017952619999999999 2024-09-18T07:50:04.168 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.168 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.168 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:04.168 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.381124+0000", 2024-09-18T07:50:04.168 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5084999999999999e-05 2024-09-18T07:50:04.168 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.168 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.168 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.169 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.381229+0000", 2024-09-18T07:50:04.169 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000104858 2024-09-18T07:50:04.169 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.169 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.169 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.169 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.169 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.169 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189558 10.2 10:4577464b:::benchmark_data_smithi137_151160_object53742:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:50:04.169 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.307579+0000", 2024-09-18T07:50:04.169 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 270.83628582099999, 2024-09-18T07:50:04.169 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0761420020000001, 2024-09-18T07:50:04.169 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.169 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.170 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:04.170 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.170 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.170 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.170 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189558 2024-09-18T07:50:04.170 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.170 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.170 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.170 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.170 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.307579+0000", 2024-09-18T07:50:04.170 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.170 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.170 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.170 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.171 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.307579+0000", 2024-09-18T07:50:04.171 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.171 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.171 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.171 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.171 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.307580+0000", 2024-09-18T07:50:04.171 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7400000000000001e-06 2024-09-18T07:50:04.171 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.171 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.171 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.171 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.307585+0000", 2024-09-18T07:50:04.172 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5950000000000001e-06 2024-09-18T07:50:04.172 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.172 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.172 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.172 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.307601+0000", 2024-09-18T07:50:04.172 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5971999999999999e-05 2024-09-18T07:50:04.172 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.172 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.172 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.172 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.307608+0000", 2024-09-18T07:50:04.172 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2470000000000002e-06 2024-09-18T07:50:04.172 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.173 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.173 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.173 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.307767+0000", 2024-09-18T07:50:04.173 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015924099999999999 2024-09-18T07:50:04.173 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.173 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.173 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.173 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.307775+0000", 2024-09-18T07:50:04.173 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7090000000000002e-06 2024-09-18T07:50:04.173 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.173 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.173 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.173 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.376610+0000", 2024-09-18T07:50:04.174 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0688345130000001 2024-09-18T07:50:04.174 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.174 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.174 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:04.174 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.376818+0000", 2024-09-18T07:50:04.174 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00020837199999999999 2024-09-18T07:50:04.174 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.174 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.174 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:50:04.174 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.376968+0000", 2024-09-18T07:50:04.174 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000149741 2024-09-18T07:50:04.174 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.174 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.174 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:04.175 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.383501+0000", 2024-09-18T07:50:04.175 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0065335790000000003 2024-09-18T07:50:04.175 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.175 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.175 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.175 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.383544+0000", 2024-09-18T07:50:04.175 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2759999999999997e-05 2024-09-18T07:50:04.175 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.175 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.175 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.175 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.383587+0000", 2024-09-18T07:50:04.175 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2932000000000002e-05 2024-09-18T07:50:04.175 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.176 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.176 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:04.176 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.383632+0000", 2024-09-18T07:50:04.176 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4734000000000002e-05 2024-09-18T07:50:04.176 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.176 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.176 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.176 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.383721+0000", 2024-09-18T07:50:04.176 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8867000000000001e-05 2024-09-18T07:50:04.176 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.176 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.176 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.176 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.176 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.177 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189576 10.2 10:403b32d3:::benchmark_data_smithi137_151160_object53760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:50:04.177 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.323665+0000", 2024-09-18T07:50:04.177 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 270.82019918399999, 2024-09-18T07:50:04.177 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0652314759999999, 2024-09-18T07:50:04.177 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.177 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.177 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:04.177 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.177 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.177 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.177 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189576 2024-09-18T07:50:04.177 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.177 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.178 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.178 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.178 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.323665+0000", 2024-09-18T07:50:04.178 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.178 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.178 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.178 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.178 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.323665+0000", 2024-09-18T07:50:04.178 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.178 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.178 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.178 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.178 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.323667+0000", 2024-09-18T07:50:04.179 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1610000000000001e-06 2024-09-18T07:50:04.179 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.179 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.179 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.179 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.323673+0000", 2024-09-18T07:50:04.179 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0469999999999999e-06 2024-09-18T07:50:04.179 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.179 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.179 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.179 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.323684+0000", 2024-09-18T07:50:04.179 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0168e-05 2024-09-18T07:50:04.179 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.179 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.180 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.180 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.323691+0000", 2024-09-18T07:50:04.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1949999999999997e-06 2024-09-18T07:50:04.180 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.180 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.180 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.180 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.331894+0000", 2024-09-18T07:50:04.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0082035219999999996 2024-09-18T07:50:04.180 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.180 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.180 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.180 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.331902+0000", 2024-09-18T07:50:04.180 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0260000000000007e-06 2024-09-18T07:50:04.180 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.181 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.181 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.181 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.381275+0000", 2024-09-18T07:50:04.181 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0493726240000001 2024-09-18T07:50:04.181 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.181 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.181 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:04.181 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.381376+0000", 2024-09-18T07:50:04.181 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010126799999999999 2024-09-18T07:50:04.181 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.181 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.181 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:50:04.181 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.381451+0000", 2024-09-18T07:50:04.181 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5108e-05 2024-09-18T07:50:04.182 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.182 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.182 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:04.182 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.388536+0000", 2024-09-18T07:50:04.182 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0070845819999999999 2024-09-18T07:50:04.182 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.182 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.182 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.182 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.388688+0000", 2024-09-18T07:50:04.182 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015197100000000001 2024-09-18T07:50:04.182 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.182 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.182 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.182 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.388748+0000", 2024-09-18T07:50:04.183 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0303999999999997e-05 2024-09-18T07:50:04.183 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.183 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.183 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:04.183 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.388804+0000", 2024-09-18T07:50:04.183 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6070999999999999e-05 2024-09-18T07:50:04.183 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.183 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.183 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.183 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.388897+0000", 2024-09-18T07:50:04.183 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2429000000000006e-05 2024-09-18T07:50:04.183 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.184 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.184 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.184 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.184 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.184 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189578 10.2 10:467f99cb:::benchmark_data_smithi137_151160_object53762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:50:04.184 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.324033+0000", 2024-09-18T07:50:04.184 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 270.819831792, 2024-09-18T07:50:04.184 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0651037249999999, 2024-09-18T07:50:04.184 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.184 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.184 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:04.184 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.184 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.185 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.185 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189578 2024-09-18T07:50:04.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.185 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.185 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.185 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.185 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.324033+0000", 2024-09-18T07:50:04.185 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.185 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.185 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.185 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.324033+0000", 2024-09-18T07:50:04.186 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.186 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.186 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.186 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.324034+0000", 2024-09-18T07:50:04.186 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.156e-06 2024-09-18T07:50:04.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.186 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.186 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.186 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.324038+0000", 2024-09-18T07:50:04.186 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1999999999999996e-06 2024-09-18T07:50:04.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.186 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.186 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.187 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.324047+0000", 2024-09-18T07:50:04.187 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8659999999999995e-06 2024-09-18T07:50:04.187 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.187 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.187 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.187 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.324054+0000", 2024-09-18T07:50:04.187 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8279999999999997e-06 2024-09-18T07:50:04.187 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.187 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.188 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.188 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.331930+0000", 2024-09-18T07:50:04.188 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0078765510000000007 2024-09-18T07:50:04.188 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.188 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.188 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.188 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.331936+0000", 2024-09-18T07:50:04.188 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4280000000000004e-06 2024-09-18T07:50:04.188 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.188 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.188 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.188 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.383774+0000", 2024-09-18T07:50:04.188 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0518385020000001 2024-09-18T07:50:04.188 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.189 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.189 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:04.189 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.383905+0000", 2024-09-18T07:50:04.189 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013089699999999999 2024-09-18T07:50:04.189 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.189 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.189 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:50:04.189 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.384063+0000", 2024-09-18T07:50:04.189 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015821800000000001 2024-09-18T07:50:04.189 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.189 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.189 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:04.189 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.388706+0000", 2024-09-18T07:50:04.189 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.004643309 2024-09-18T07:50:04.190 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.190 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.190 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.190 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.388959+0000", 2024-09-18T07:50:04.190 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025269099999999998 2024-09-18T07:50:04.190 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.190 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.190 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.190 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.389006+0000", 2024-09-18T07:50:04.190 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6391999999999999e-05 2024-09-18T07:50:04.190 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.190 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.190 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:04.190 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.389052+0000", 2024-09-18T07:50:04.190 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6768999999999999e-05 2024-09-18T07:50:04.191 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.191 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.191 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.191 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.389136+0000", 2024-09-18T07:50:04.191 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3918000000000005e-05 2024-09-18T07:50:04.191 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.191 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.191 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.191 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.191 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.191 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189583 10.2 10:4ef4bc3a:::benchmark_data_smithi137_151160_object53767:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:50:04.191 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.325300+0000", 2024-09-18T07:50:04.191 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 270.81856456399998, 2024-09-18T07:50:04.192 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.068459346, 2024-09-18T07:50:04.192 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.192 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.192 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:04.192 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.192 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.192 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.192 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189583 2024-09-18T07:50:04.192 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.192 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.192 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.192 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.192 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.325300+0000", 2024-09-18T07:50:04.193 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.193 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.193 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.193 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.193 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.325300+0000", 2024-09-18T07:50:04.193 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.193 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.193 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.193 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.193 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.325301+0000", 2024-09-18T07:50:04.193 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4389999999999999e-06 2024-09-18T07:50:04.193 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.193 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.193 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.194 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.325306+0000", 2024-09-18T07:50:04.194 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4530000000000004e-06 2024-09-18T07:50:04.194 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.194 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:04.195 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.195 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.325316+0000", 2024-09-18T07:50:04.195 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.008e-05 2024-09-18T07:50:04.195 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.195 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.195 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.195 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.325323+0000", 2024-09-18T07:50:04.196 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3900000000000004e-06 2024-09-18T07:50:04.196 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.196 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.196 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.196 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.331986+0000", 2024-09-18T07:50:04.196 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0066628069999999998 2024-09-18T07:50:04.196 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.196 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.196 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.196 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.332079+0000", 2024-09-18T07:50:04.196 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2844999999999996e-05 2024-09-18T07:50:04.196 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.197 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.197 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.197 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.386405+0000", 2024-09-18T07:50:04.197 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0543261900000001 2024-09-18T07:50:04.197 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.197 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.197 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:04.197 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.386505+0000", 2024-09-18T07:50:04.197 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000100076 2024-09-18T07:50:04.197 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.197 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.197 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:50:04.197 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.386616+0000", 2024-09-18T07:50:04.197 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011094099999999999 2024-09-18T07:50:04.198 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.198 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.198 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:04.198 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.391271+0000", 2024-09-18T07:50:04.198 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0046548819999999999 2024-09-18T07:50:04.198 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.198 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.198 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.198 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.393590+0000", 2024-09-18T07:50:04.198 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023194869999999999 2024-09-18T07:50:04.198 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.198 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.198 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.198 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.393656+0000", 2024-09-18T07:50:04.199 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5579000000000004e-05 2024-09-18T07:50:04.199 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.199 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.199 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:04.199 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.393691+0000", 2024-09-18T07:50:04.199 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5144000000000001e-05 2024-09-18T07:50:04.199 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.199 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.199 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.199 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.393759+0000", 2024-09-18T07:50:04.199 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8033000000000004e-05 2024-09-18T07:50:04.199 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.199 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.200 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.200 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.200 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.200 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189620 10.2 10:45188eeb:::benchmark_data_smithi137_151160_object53804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:50:04.200 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.359144+0000", 2024-09-18T07:50:04.200 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 270.78472009699999, 2024-09-18T07:50:04.200 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.043621656, 2024-09-18T07:50:04.200 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.200 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.201 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:04.201 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.201 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.201 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.201 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189620 2024-09-18T07:50:04.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.201 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.201 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.201 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.201 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.359144+0000", 2024-09-18T07:50:04.201 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.201 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.201 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.202 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.359144+0000", 2024-09-18T07:50:04.202 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.202 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.202 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.202 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.359146+0000", 2024-09-18T07:50:04.202 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.125e-06 2024-09-18T07:50:04.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.202 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.202 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.202 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.359153+0000", 2024-09-18T07:50:04.202 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4099999999999996e-06 2024-09-18T07:50:04.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.203 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.203 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.203 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.359164+0000", 2024-09-18T07:50:04.203 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1104000000000001e-05 2024-09-18T07:50:04.203 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.203 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.203 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.203 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.359171+0000", 2024-09-18T07:50:04.203 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3950000000000003e-06 2024-09-18T07:50:04.203 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.203 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.203 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.203 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.359228+0000", 2024-09-18T07:50:04.203 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6341000000000001e-05 2024-09-18T07:50:04.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.204 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.204 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.204 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.359236+0000", 2024-09-18T07:50:04.204 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0960000000000001e-06 2024-09-18T07:50:04.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.204 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.204 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.204 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.389179+0000", 2024-09-18T07:50:04.204 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0299431889999999 2024-09-18T07:50:04.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.204 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.204 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:04.205 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.389287+0000", 2024-09-18T07:50:04.205 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000107902 2024-09-18T07:50:04.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.205 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.205 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:50:04.205 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.389427+0000", 2024-09-18T07:50:04.205 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000139763 2024-09-18T07:50:04.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.205 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.205 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:04.205 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.393770+0000", 2024-09-18T07:50:04.205 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0043434440000000001 2024-09-18T07:50:04.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.206 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.206 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.206 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.397844+0000", 2024-09-18T07:50:04.206 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0040739160000000003 2024-09-18T07:50:04.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.206 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.206 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.206 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.402660+0000", 2024-09-18T07:50:04.206 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0048165150000000004 2024-09-18T07:50:04.206 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.206 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.206 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:04.206 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.402693+0000", 2024-09-18T07:50:04.206 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2381e-05 2024-09-18T07:50:04.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.207 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.207 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.207 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.402766+0000", 2024-09-18T07:50:04.207 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3075000000000001e-05 2024-09-18T07:50:04.207 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.207 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.207 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.207 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.207 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.207 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189626 10.2 10:461f64b9:::benchmark_data_smithi137_151160_object53810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:50:04.207 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.364564+0000", 2024-09-18T07:50:04.207 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 270.779300122, 2024-09-18T07:50:04.207 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0384643680000001, 2024-09-18T07:50:04.208 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.208 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.208 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:04.208 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.208 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.208 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.208 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189626 2024-09-18T07:50:04.208 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.208 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.208 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.208 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.208 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.364564+0000", 2024-09-18T07:50:04.208 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.209 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.209 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.209 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.209 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.364564+0000", 2024-09-18T07:50:04.209 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.209 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.209 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.209 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.209 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.364566+0000", 2024-09-18T07:50:04.209 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8080000000000001e-06 2024-09-18T07:50:04.209 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.209 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.209 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.210 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.364573+0000", 2024-09-18T07:50:04.210 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7850000000000003e-06 2024-09-18T07:50:04.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.210 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.210 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.210 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.364588+0000", 2024-09-18T07:50:04.210 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4806e-05 2024-09-18T07:50:04.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.210 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.210 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.210 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.364596+0000", 2024-09-18T07:50:04.210 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1689999999999992e-06 2024-09-18T07:50:04.210 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.210 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.364654+0000", 2024-09-18T07:50:04.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8658e-05 2024-09-18T07:50:04.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.211 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.364662+0000", 2024-09-18T07:50:04.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7409999999999992e-06 2024-09-18T07:50:04.211 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.211 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.211 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.211 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.391301+0000", 2024-09-18T07:50:04.211 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0266390919999999 2024-09-18T07:50:04.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.212 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:04.212 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.391419+0000", 2024-09-18T07:50:04.212 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000117724 2024-09-18T07:50:04.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.212 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:50:04.212 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.391536+0000", 2024-09-18T07:50:04.212 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000117162 2024-09-18T07:50:04.212 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.212 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.212 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:04.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.397781+0000", 2024-09-18T07:50:04.213 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0062446330000000003 2024-09-18T07:50:04.213 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.213 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.213 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.397917+0000", 2024-09-18T07:50:04.213 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013668600000000001 2024-09-18T07:50:04.213 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.213 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.213 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.213 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.402881+0000", 2024-09-18T07:50:04.213 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0049634040000000003 2024-09-18T07:50:04.213 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.214 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:04.214 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.402936+0000", 2024-09-18T07:50:04.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5284999999999997e-05 2024-09-18T07:50:04.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.214 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.214 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.403029+0000", 2024-09-18T07:50:04.214 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2415000000000002e-05 2024-09-18T07:50:04.214 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.214 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.214 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.214 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.215 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:189638 10.2 10:423cbe89:::benchmark_data_smithi137_151160_object53822:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-18T07:50:04.215 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:33.372890+0000", 2024-09-18T07:50:04.215 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 270.77097429999998, 2024-09-18T07:50:04.215 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0303354929999999, 2024-09-18T07:50:04.215 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.215 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.215 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:04.215 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.215 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.215 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.215 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 189638 2024-09-18T07:50:04.215 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.215 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.372890+0000", 2024-09-18T07:50:04.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.372890+0000", 2024-09-18T07:50:04.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.216 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.216 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.216 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.372892+0000", 2024-09-18T07:50:04.216 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.88e-06 2024-09-18T07:50:04.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.217 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.372897+0000", 2024-09-18T07:50:04.217 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3419999999999998e-06 2024-09-18T07:50:04.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.217 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.217 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.372908+0000", 2024-09-18T07:50:04.217 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0421999999999999e-05 2024-09-18T07:50:04.217 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.217 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.218 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.218 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.372914+0000", 2024-09-18T07:50:04.218 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6699999999999997e-06 2024-09-18T07:50:04.218 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.218 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.218 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.218 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.372977+0000", 2024-09-18T07:50:04.218 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2943999999999999e-05 2024-09-18T07:50:04.218 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.218 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.218 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.218 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:33.372985+0000", 2024-09-18T07:50:04.218 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7759999999999997e-06 2024-09-18T07:50:04.218 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.219 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.219 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.219 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.393800+0000", 2024-09-18T07:50:04.219 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.020814707 2024-09-18T07:50:04.219 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.219 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.219 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:04.219 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.393866+0000", 2024-09-18T07:50:04.219 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6685000000000003e-05 2024-09-18T07:50:04.219 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.219 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.219 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1", 2024-09-18T07:50:04.219 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.393999+0000", 2024-09-18T07:50:04.219 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013285 2024-09-18T07:50:04.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.220 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:04.220 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.397931+0000", 2024-09-18T07:50:04.220 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0039321260000000002 2024-09-18T07:50:04.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.220 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.220 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.402167+0000", 2024-09-18T07:50:04.220 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0042359850000000003 2024-09-18T07:50:04.220 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.220 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.220 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.220 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.403082+0000", 2024-09-18T07:50:04.221 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00091465000000000003 2024-09-18T07:50:04.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.221 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.221 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:04.221 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.403134+0000", 2024-09-18T07:50:04.221 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2346000000000003e-05 2024-09-18T07:50:04.221 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.221 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.221 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.221 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:34.403226+0000", 2024-09-18T07:50:04.221 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.111e-05 2024-09-18T07:50:04.221 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.221 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.221 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.222 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.222 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.222 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:199982 10.1 10:8621fea9:::benchmark_data_smithi137_151160_object64166:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e430)", 2024-09-18T07:50:04.222 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:55.078327+0000", 2024-09-18T07:50:04.222 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 249.06553711500001, 2024-09-18T07:50:04.222 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.047601421, 2024-09-18T07:50:04.222 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.222 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.222 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:50:04.222 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.222 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.222 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.222 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 199982 2024-09-18T07:50:04.222 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.223 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.223 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.223 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.223 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.078327+0000", 2024-09-18T07:50:04.223 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.223 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.223 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.223 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.078327+0000", 2024-09-18T07:50:04.223 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.223 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.223 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.223 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.078329+0000", 2024-09-18T07:50:04.224 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.018e-06 2024-09-18T07:50:04.224 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.224 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.224 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.224 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.078335+0000", 2024-09-18T07:50:04.224 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4539999999999998e-06 2024-09-18T07:50:04.224 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.224 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.224 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.224 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.078346+0000", 2024-09-18T07:50:04.224 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.164e-05 2024-09-18T07:50:04.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.225 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.225 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.225 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.078354+0000", 2024-09-18T07:50:04.225 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0790000000000001e-06 2024-09-18T07:50:04.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.225 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.225 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.225 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.079784+0000", 2024-09-18T07:50:04.225 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014295880000000001 2024-09-18T07:50:04.225 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.225 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.225 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.226 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.079792+0000", 2024-09-18T07:50:04.226 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1829999999999994e-06 2024-09-18T07:50:04.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.226 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.226 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.226 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.115555+0000", 2024-09-18T07:50:04.226 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0357623970000001 2024-09-18T07:50:04.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.226 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.226 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for missing object", 2024-09-18T07:50:04.226 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.115633+0000", 2024-09-18T07:50:04.226 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8830000000000002e-05 2024-09-18T07:50:04.226 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.227 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.227 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.227 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125890+0000", 2024-09-18T07:50:04.227 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010256131999999999 2024-09-18T07:50:04.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.227 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.227 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.227 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125929+0000", 2024-09-18T07:50:04.228 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9100000000000002e-05 2024-09-18T07:50:04.228 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.228 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.228 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.228 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.228 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.228 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:199992 10.1 10:8f0b8e8f:::benchmark_data_smithi137_151160_object64176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-09-18T07:50:04.228 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:55.089372+0000", 2024-09-18T07:50:04.228 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 249.05449186600001, 2024-09-18T07:50:04.228 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0362496459999999, 2024-09-18T07:50:04.228 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.228 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.228 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:04.229 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.229 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.229 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.229 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 199992 2024-09-18T07:50:04.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.229 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.229 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.229 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.229 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.089372+0000", 2024-09-18T07:50:04.229 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.229 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.229 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.230 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.089372+0000", 2024-09-18T07:50:04.230 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.230 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.230 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.230 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.089374+0000", 2024-09-18T07:50:04.230 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8169999999999999e-06 2024-09-18T07:50:04.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.230 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.230 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.230 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.089380+0000", 2024-09-18T07:50:04.230 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7830000000000004e-06 2024-09-18T07:50:04.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.231 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.231 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.231 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.089392+0000", 2024-09-18T07:50:04.231 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2041999999999999e-05 2024-09-18T07:50:04.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.231 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.231 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.231 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.089398+0000", 2024-09-18T07:50:04.231 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4070000000000001e-06 2024-09-18T07:50:04.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.231 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.231 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.231 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.089453+0000", 2024-09-18T07:50:04.231 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5881999999999998e-05 2024-09-18T07:50:04.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.232 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.232 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.232 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.089459+0000", 2024-09-18T07:50:04.232 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.587e-06 2024-09-18T07:50:04.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.232 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.232 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.232 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.115822+0000", 2024-09-18T07:50:04.232 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.026363409 2024-09-18T07:50:04.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.232 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.232 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:04.233 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.115891+0000", 2024-09-18T07:50:04.233 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8275999999999995e-05 2024-09-18T07:50:04.233 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.233 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.233 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,7", 2024-09-18T07:50:04.233 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.116206+0000", 2024-09-18T07:50:04.233 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00031504100000000001 2024-09-18T07:50:04.233 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.233 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.233 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.233 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125445+0000", 2024-09-18T07:50:04.233 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0092397869999999993 2024-09-18T07:50:04.233 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.233 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.234 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:04.234 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125457+0000", 2024-09-18T07:50:04.234 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.119e-05 2024-09-18T07:50:04.234 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.234 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.234 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.234 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125536+0000", 2024-09-18T07:50:04.234 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8971e-05 2024-09-18T07:50:04.234 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.234 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.234 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:04.234 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125569+0000", 2024-09-18T07:50:04.234 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3135e-05 2024-09-18T07:50:04.235 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.235 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.235 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.235 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125622+0000", 2024-09-18T07:50:04.235 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3319000000000002e-05 2024-09-18T07:50:04.235 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.235 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.235 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.235 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.235 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.235 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:200000 10.1 10:8f465e9f:::benchmark_data_smithi137_151160_object64184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-09-18T07:50:04.235 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:55.098734+0000", 2024-09-18T07:50:04.235 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 249.04512988499999, 2024-09-18T07:50:04.236 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.026996365, 2024-09-18T07:50:04.236 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.236 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.236 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:04.236 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.236 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.236 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.236 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 200000 2024-09-18T07:50:04.236 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.236 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.236 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.236 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.236 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.098734+0000", 2024-09-18T07:50:04.237 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.237 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.237 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.237 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.237 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.098734+0000", 2024-09-18T07:50:04.237 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.237 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.237 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.237 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.237 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.098736+0000", 2024-09-18T07:50:04.237 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9860000000000001e-06 2024-09-18T07:50:04.237 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.237 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.238 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.238 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.098742+0000", 2024-09-18T07:50:04.238 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5849999999999999e-06 2024-09-18T07:50:04.238 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.238 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.238 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.238 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.098754+0000", 2024-09-18T07:50:04.238 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1882000000000001e-05 2024-09-18T07:50:04.238 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.238 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.238 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.238 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.098762+0000", 2024-09-18T07:50:04.238 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6450000000000005e-06 2024-09-18T07:50:04.238 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.240 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.240 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.240 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.098822+0000", 2024-09-18T07:50:04.240 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0618000000000001e-05 2024-09-18T07:50:04.240 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.240 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.240 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.240 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.098832+0000", 2024-09-18T07:50:04.240 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.5580000000000006e-06 2024-09-18T07:50:04.240 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.241 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.241 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.241 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.121193+0000", 2024-09-18T07:50:04.241 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.022361694 2024-09-18T07:50:04.241 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.241 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.241 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:04.241 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.121336+0000", 2024-09-18T07:50:04.241 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014290900000000001 2024-09-18T07:50:04.241 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.241 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.241 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,7", 2024-09-18T07:50:04.241 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.121485+0000", 2024-09-18T07:50:04.242 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000148706 2024-09-18T07:50:04.242 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.242 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.242 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:04.242 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125465+0000", 2024-09-18T07:50:04.242 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0039797749999999996 2024-09-18T07:50:04.242 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.242 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.242 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.242 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125507+0000", 2024-09-18T07:50:04.242 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2333999999999998e-05 2024-09-18T07:50:04.242 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.242 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.242 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.243 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125659+0000", 2024-09-18T07:50:04.243 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000152029 2024-09-18T07:50:04.243 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.243 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.243 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:04.243 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125684+0000", 2024-09-18T07:50:04.243 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.5306000000000001e-05 2024-09-18T07:50:04.243 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.243 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.243 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.243 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125731+0000", 2024-09-18T07:50:04.243 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6338000000000003e-05 2024-09-18T07:50:04.243 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.243 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.243 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.244 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.244 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.244 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:200014 10.1 10:8d4a215b:::benchmark_data_smithi137_151160_object64198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-09-18T07:50:04.244 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:55.114672+0000", 2024-09-18T07:50:04.244 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 249.02919255200001, 2024-09-18T07:50:04.244 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.011365474, 2024-09-18T07:50:04.244 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.244 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.244 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:04.244 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.244 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.244 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.244 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 200014 2024-09-18T07:50:04.245 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.245 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.245 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.245 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.245 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.114672+0000", 2024-09-18T07:50:04.245 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.245 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.245 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.245 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.245 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.114672+0000", 2024-09-18T07:50:04.245 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.245 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.245 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.245 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.246 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.114674+0000", 2024-09-18T07:50:04.246 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.34e-06 2024-09-18T07:50:04.246 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.246 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.246 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.246 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.114680+0000", 2024-09-18T07:50:04.246 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6420000000000004e-06 2024-09-18T07:50:04.246 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.246 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.246 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.246 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.114695+0000", 2024-09-18T07:50:04.246 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5255999999999999e-05 2024-09-18T07:50:04.246 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.247 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.247 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.247 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.114703+0000", 2024-09-18T07:50:04.247 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9650000000000008e-06 2024-09-18T07:50:04.247 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.247 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.247 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.247 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.115999+0000", 2024-09-18T07:50:04.247 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001295907 2024-09-18T07:50:04.247 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.247 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.247 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.247 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.116007+0000", 2024-09-18T07:50:04.248 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2570000000000007e-06 2024-09-18T07:50:04.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.248 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.248 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.248 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.123175+0000", 2024-09-18T07:50:04.248 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.007167897 2024-09-18T07:50:04.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.248 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.248 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:04.248 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.123291+0000", 2024-09-18T07:50:04.248 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000116278 2024-09-18T07:50:04.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.248 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.248 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,7", 2024-09-18T07:50:04.249 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.123384+0000", 2024-09-18T07:50:04.249 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2707000000000005e-05 2024-09-18T07:50:04.249 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.249 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.249 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:04.249 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125471+0000", 2024-09-18T07:50:04.249 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0020867210000000001 2024-09-18T07:50:04.249 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.249 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.249 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.249 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125762+0000", 2024-09-18T07:50:04.249 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00029164300000000001 2024-09-18T07:50:04.249 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.249 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.250 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.250 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125960+0000", 2024-09-18T07:50:04.250 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00019801500000000001 2024-09-18T07:50:04.250 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.250 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.250 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:04.250 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.125995+0000", 2024-09-18T07:50:04.250 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.4347999999999997e-05 2024-09-18T07:50:04.250 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.250 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.250 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.250 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.126037+0000", 2024-09-18T07:50:04.250 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.2497999999999999e-05 2024-09-18T07:50:04.250 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.251 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.251 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.251 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.251 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.251 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:200025 10.1 10:841dfac0:::benchmark_data_smithi137_151160_object64209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-09-18T07:50:04.251 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:45:55.123654+0000", 2024-09-18T07:50:04.251 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 249.020210392, 2024-09-18T07:50:04.251 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.014501978, 2024-09-18T07:50:04.251 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.251 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.251 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:04.251 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.251 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.251 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.252 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 200025 2024-09-18T07:50:04.252 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.252 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.252 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.252 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.252 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.123654+0000", 2024-09-18T07:50:04.252 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.252 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.252 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.252 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.252 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.123654+0000", 2024-09-18T07:50:04.252 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.252 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.253 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.253 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.253 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.123655+0000", 2024-09-18T07:50:04.253 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1429999999999999e-06 2024-09-18T07:50:04.253 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.253 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.253 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.253 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.123659+0000", 2024-09-18T07:50:04.253 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9600000000000002e-06 2024-09-18T07:50:04.253 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.253 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.254 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.254 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.123668+0000", 2024-09-18T07:50:04.254 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2869999999999993e-06 2024-09-18T07:50:04.254 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.254 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.254 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.254 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.123675+0000", 2024-09-18T07:50:04.254 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8120000000000002e-06 2024-09-18T07:50:04.254 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.254 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.254 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.254 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.123785+0000", 2024-09-18T07:50:04.254 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000110006 2024-09-18T07:50:04.255 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.255 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.255 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.255 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:55.123792+0000", 2024-09-18T07:50:04.255 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2099999999999996e-06 2024-09-18T07:50:04.255 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.255 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.255 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.255 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.126103+0000", 2024-09-18T07:50:04.255 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0023103499999999 2024-09-18T07:50:04.255 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.255 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.255 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:04.255 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.126200+0000", 2024-09-18T07:50:04.256 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7106000000000002e-05 2024-09-18T07:50:04.256 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.256 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.256 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,7", 2024-09-18T07:50:04.256 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.126318+0000", 2024-09-18T07:50:04.256 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000118439 2024-09-18T07:50:04.256 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.256 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.256 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.256 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.133064+0000", 2024-09-18T07:50:04.256 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0067462420000000004 2024-09-18T07:50:04.256 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.256 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.256 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.257 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.133128+0000", 2024-09-18T07:50:04.257 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3774000000000003e-05 2024-09-18T07:50:04.257 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.257 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.257 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:04.257 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.137958+0000", 2024-09-18T07:50:04.257 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0048296800000000003 2024-09-18T07:50:04.257 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.257 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.257 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:04.257 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.138049+0000", 2024-09-18T07:50:04.257 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0587000000000005e-05 2024-09-18T07:50:04.257 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.258 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.258 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.258 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:45:56.138156+0000", 2024-09-18T07:50:04.258 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000107382 2024-09-18T07:50:04.258 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.258 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.258 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.258 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.258 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.258 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:251939 10.2 10:4579883a:::benchmark_data_smithi137_151160_object116123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-18T07:50:04.258 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:46:38.919114+0000", 2024-09-18T07:50:04.258 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 205.22475047699999, 2024-09-18T07:50:04.258 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0558060870000001, 2024-09-18T07:50:04.258 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.259 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.259 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:04.259 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.259 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.259 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.259 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 251939 2024-09-18T07:50:04.259 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.259 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.259 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.259 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.259 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.919114+0000", 2024-09-18T07:50:04.259 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.259 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.259 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.260 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.260 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.919114+0000", 2024-09-18T07:50:04.260 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.260 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.260 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.260 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.260 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.919115+0000", 2024-09-18T07:50:04.260 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3170000000000001e-06 2024-09-18T07:50:04.260 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.260 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.260 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.260 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.919120+0000", 2024-09-18T07:50:04.260 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.425e-06 2024-09-18T07:50:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.919128+0000", 2024-09-18T07:50:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0690000000000002e-06 2024-09-18T07:50:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.919132+0000", 2024-09-18T07:50:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4889999999999997e-06 2024-09-18T07:50:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.919190+0000", 2024-09-18T07:50:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8239999999999998e-05 2024-09-18T07:50:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.919202+0000", 2024-09-18T07:50:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1155000000000001e-05 2024-09-18T07:50:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.875920+0000", 2024-09-18T07:50:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.956717975 2024-09-18T07:50:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.875934+0000", 2024-09-18T07:50:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4823e-05 2024-09-18T07:50:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.966003+0000", 2024-09-18T07:50:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.090068364 2024-09-18T07:50:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.966103+0000", 2024-09-18T07:50:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010009900000000001 2024-09-18T07:50:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1,3", 2024-09-18T07:50:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.966454+0000", 2024-09-18T07:50:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00035081199999999999 2024-09-18T07:50:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.971862+0000", 2024-09-18T07:50:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0054080459999999997 2024-09-18T07:50:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.972094+0000", 2024-09-18T07:50:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00023215000000000001 2024-09-18T07:50:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.974670+0000", 2024-09-18T07:50:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025764410000000001 2024-09-18T07:50:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.974753+0000", 2024-09-18T07:50:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3001e-05 2024-09-18T07:50:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.974811+0000", 2024-09-18T07:50:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8119999999999999e-05 2024-09-18T07:50:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.974920+0000", 2024-09-18T07:50:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010856100000000001 2024-09-18T07:50:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:251948 10.2 10:4604b5ef:::benchmark_data_smithi137_151160_object116132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-18T07:50:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:46:38.933161+0000", 2024-09-18T07:50:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 205.21070321100001, 2024-09-18T07:50:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0420709339999998, 2024-09-18T07:50:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 251948 2024-09-18T07:50:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.933161+0000", 2024-09-18T07:50:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.933161+0000", 2024-09-18T07:50:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.933163+0000", 2024-09-18T07:50:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4470000000000001e-06 2024-09-18T07:50:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.933167+0000", 2024-09-18T07:50:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.685e-06 2024-09-18T07:50:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.933177+0000", 2024-09-18T07:50:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8500000000000006e-06 2024-09-18T07:50:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.933182+0000", 2024-09-18T07:50:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.0540000000000002e-06 2024-09-18T07:50:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.933228+0000", 2024-09-18T07:50:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5850000000000003e-05 2024-09-18T07:50:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.933234+0000", 2024-09-18T07:50:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.913e-06 2024-09-18T07:50:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.271 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.875969+0000", 2024-09-18T07:50:04.271 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.94273543699999995 2024-09-18T07:50:04.271 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.271 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.271 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.271 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.875978+0000", 2024-09-18T07:50:04.271 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2900000000000002e-06 2024-09-18T07:50:04.271 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.271 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.271 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.271 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.969585+0000", 2024-09-18T07:50:04.271 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.093607574 2024-09-18T07:50:04.271 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.271 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.272 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:04.272 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.969678+0000", 2024-09-18T07:50:04.272 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2670999999999994e-05 2024-09-18T07:50:04.272 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.272 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.272 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1,3", 2024-09-18T07:50:04.272 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.969767+0000", 2024-09-18T07:50:04.272 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8710999999999998e-05 2024-09-18T07:50:04.272 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.272 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.272 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:04.272 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.972100+0000", 2024-09-18T07:50:04.272 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023329539999999999 2024-09-18T07:50:04.272 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.273 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.273 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.273 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.974612+0000", 2024-09-18T07:50:04.273 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002512182 2024-09-18T07:50:04.273 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.273 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.273 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.273 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.974711+0000", 2024-09-18T07:50:04.273 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9077e-05 2024-09-18T07:50:04.273 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.273 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.273 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.273 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975085+0000", 2024-09-18T07:50:04.274 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00037395000000000001 2024-09-18T07:50:04.274 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.274 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.274 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:04.274 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975134+0000", 2024-09-18T07:50:04.274 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9251e-05 2024-09-18T07:50:04.274 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.274 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.274 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.274 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975232+0000", 2024-09-18T07:50:04.274 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.8037999999999999e-05 2024-09-18T07:50:04.274 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.274 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.274 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.275 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.275 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.275 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:251954 10.2 10:48d2efb2:::benchmark_data_smithi137_151160_object116138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-18T07:50:04.275 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:46:38.937932+0000", 2024-09-18T07:50:04.275 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 205.205932434, 2024-09-18T07:50:04.275 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0375994749999999, 2024-09-18T07:50:04.275 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.275 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.275 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:04.275 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.275 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.275 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.276 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 251954 2024-09-18T07:50:04.276 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.276 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.276 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.276 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.276 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.937932+0000", 2024-09-18T07:50:04.276 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.276 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.276 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.276 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.276 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.937932+0000", 2024-09-18T07:50:04.276 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.276 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.277 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.277 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.277 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.937934+0000", 2024-09-18T07:50:04.277 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.77e-06 2024-09-18T07:50:04.277 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.277 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.277 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.277 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.937939+0000", 2024-09-18T07:50:04.277 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.1749999999999996e-06 2024-09-18T07:50:04.277 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.277 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.277 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.277 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.937960+0000", 2024-09-18T07:50:04.278 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1426000000000001e-05 2024-09-18T07:50:04.278 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.278 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.278 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.278 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.937968+0000", 2024-09-18T07:50:04.278 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3209999999999999e-06 2024-09-18T07:50:04.278 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.278 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.278 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.278 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.938028+0000", 2024-09-18T07:50:04.278 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0856999999999997e-05 2024-09-18T07:50:04.278 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.278 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.279 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.279 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.938038+0000", 2024-09-18T07:50:04.279 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0003000000000001e-05 2024-09-18T07:50:04.279 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.279 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.279 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.279 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.876003+0000", 2024-09-18T07:50:04.279 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93796477700000003 2024-09-18T07:50:04.279 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.279 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.279 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.279 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.876011+0000", 2024-09-18T07:50:04.279 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9149999999999996e-06 2024-09-18T07:50:04.279 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.280 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.280 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.280 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.972190+0000", 2024-09-18T07:50:04.280 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0961789040000001 2024-09-18T07:50:04.280 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.280 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.280 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:04.280 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.972346+0000", 2024-09-18T07:50:04.280 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015571500000000001 2024-09-18T07:50:04.280 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.280 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.280 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1,3", 2024-09-18T07:50:04.280 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.972492+0000", 2024-09-18T07:50:04.280 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014632499999999999 2024-09-18T07:50:04.281 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.281 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.281 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:04.281 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975039+0000", 2024-09-18T07:50:04.281 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002546825 2024-09-18T07:50:04.281 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.281 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.281 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.281 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975299+0000", 2024-09-18T07:50:04.281 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00026037199999999998 2024-09-18T07:50:04.281 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.281 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.281 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.282 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975357+0000", 2024-09-18T07:50:04.282 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7661000000000002e-05 2024-09-18T07:50:04.282 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.282 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.282 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.282 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975398+0000", 2024-09-18T07:50:04.282 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1362000000000001e-05 2024-09-18T07:50:04.282 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.282 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.282 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:04.282 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975437+0000", 2024-09-18T07:50:04.282 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8729000000000001e-05 2024-09-18T07:50:04.282 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975531+0000", 2024-09-18T07:50:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4338000000000004e-05 2024-09-18T07:50:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.36014.0:251992 10.2 10:447243fa:::benchmark_data_smithi137_151160_object116176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-18T07:50:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:46:38.970964+0000", 2024-09-18T07:50:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 205.17290009300001, 2024-09-18T07:50:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.016393844, 2024-09-18T07:50:04.283 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.36014", 2024-09-18T07:50:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 251992 2024-09-18T07:50:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.970964+0000", 2024-09-18T07:50:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.284 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.970964+0000", 2024-09-18T07:50:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.970966+0000", 2024-09-18T07:50:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.274e-06 2024-09-18T07:50:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.970969+0000", 2024-09-18T07:50:04.285 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9589999999999997e-06 2024-09-18T07:50:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.970979+0000", 2024-09-18T07:50:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.6069999999999997e-06 2024-09-18T07:50:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.970986+0000", 2024-09-18T07:50:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3509999999999996e-06 2024-09-18T07:50:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.286 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.971050+0000", 2024-09-18T07:50:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3492999999999997e-05 2024-09-18T07:50:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:38.971059+0000", 2024-09-18T07:50:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4800000000000007e-06 2024-09-18T07:50:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.287 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.876039+0000", 2024-09-18T07:50:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90497923199999997 2024-09-18T07:50:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:39.876046+0000", 2024-09-18T07:50:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.7780000000000007e-06 2024-09-18T07:50:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975668+0000", 2024-09-18T07:50:04.288 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0996218470000001 2024-09-18T07:50:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975815+0000", 2024-09-18T07:50:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014654300000000001 2024-09-18T07:50:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 0,1,3", 2024-09-18T07:50:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.975960+0000", 2024-09-18T07:50:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014475600000000001 2024-09-18T07:50:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.289 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.986819+0000", 2024-09-18T07:50:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.010859089000000001 2024-09-18T07:50:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.986832+0000", 2024-09-18T07:50:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2951e-05 2024-09-18T07:50:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.986870+0000", 2024-09-18T07:50:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.8519999999999997e-05 2024-09-18T07:50:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.290 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.987163+0000", 2024-09-18T07:50:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00029275999999999998 2024-09-18T07:50:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.987243+0000", 2024-09-18T07:50:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0457999999999994e-05 2024-09-18T07:50:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:46:40.987358+0000", 2024-09-18T07:50:04.291 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000114746 2024-09-18T07:50:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:100766 11.b 11.fd800d6b (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e499)", 2024-09-18T07:50:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:48:31.250927+0000", 2024-09-18T07:50:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 92.892937356000004, 2024-09-18T07:50:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0025936440000001, 2024-09-18T07:50:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "reached pg", 2024-09-18T07:50:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 100766 2024-09-18T07:50:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:31.250927+0000", 2024-09-18T07:50:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:31.250927+0000", 2024-09-18T07:50:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:04.293 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:31.250928+0000", 2024-09-18T07:50:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8000000000000005e-07 2024-09-18T07:50:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:31.251003+0000", 2024-09-18T07:50:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.4930000000000003e-05 2024-09-18T07:50:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:31.251011+0000", 2024-09-18T07:50:04.294 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9189999999999999e-06 2024-09-18T07:50:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:31.251017+0000", 2024-09-18T07:50:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6100000000000002e-06 2024-09-18T07:50:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:31.259836+0000", 2024-09-18T07:50:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0088190089999999992 2024-09-18T07:50:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:04.295 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:31.259839+0000", 2024-09-18T07:50:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.6919999999999998e-06 2024-09-18T07:50:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:32.253512+0000", 2024-09-18T07:50:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.99367333700000005 2024-09-18T07:50:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:32.253521+0000", 2024-09-18T07:50:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3669999999999997e-06 2024-09-18T07:50:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.296 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.297 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:04.297 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:04.297 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:50:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:04.298 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:04.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:04.338 INFO:teuthology.orchestra.run.smithi137.stderr:marked out osd.0. 2024-09-18T07:50:04.346 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:50:04.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:04.420 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:04.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:04.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:04.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:04.543 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:04.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:04.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:04.645 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:04.666 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:04.667 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:50:04.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:04.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:04.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:04 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3744258016' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-18T07:50:04.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:04 smithi137 ceph-mon[29408]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-18T07:50:04.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:04 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2585988866' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:50:04.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:04 smithi137 ceph-mon[29408]: from='client.36704 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:50:04.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:04 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3744258016' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-18T07:50:04.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:04 smithi137 ceph-mon[39175]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-18T07:50:04.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:04 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2585988866' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:50:04.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:04 smithi137 ceph-mon[39175]: from='client.36704 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:50:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:04 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3744258016' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-18T07:50:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:04 smithi203 ceph-mon[32237]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-18T07:50:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:04 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2585988866' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:50:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:04 smithi203 ceph-mon[32237]: from='client.36704 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:50:04.769 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:04.792 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:04.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:04.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:04.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:04.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:04.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:05.016 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:05.038 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:05.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:05.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:05.140 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:05.162 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:05.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:05.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:05.180 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T07:50:05.265 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:05.286 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:05.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:05.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:05.388 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:05.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:05.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:05.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:05.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:05 smithi203 ceph-mon[32237]: pgmap v1022: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 2094/104478 objects misplaced (2.004%) 2024-09-18T07:50:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:05 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:50:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:05 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3744258016' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-18T07:50:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:05 smithi203 ceph-mon[32237]: from='client.36704 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:50:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:05 smithi203 ceph-mon[32237]: osdmap e534: 8 total, 8 up, 6 in 2024-09-18T07:50:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:05 smithi203 ceph-mon[32237]: 4.15 scrub starts 2024-09-18T07:50:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:05 smithi203 ceph-mon[32237]: 4.15 scrub ok 2024-09-18T07:50:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:05 smithi203 ceph-mon[32237]: 4.9 scrub starts 2024-09-18T07:50:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:05 smithi203 ceph-mon[32237]: 4.9 scrub ok 2024-09-18T07:50:05.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:05 smithi203 ceph-mon[32237]: osdmap e535: 8 total, 8 up, 6 in 2024-09-18T07:50:05.476 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:50:05.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:05.534 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:05.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:05.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:05.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:05.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:05.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:05.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:05.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[39175]: pgmap v1022: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 2094/104478 objects misplaced (2.004%) 2024-09-18T07:50:05.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:50:05.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3744258016' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-18T07:50:05.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[39175]: from='client.36704 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:50:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[39175]: osdmap e534: 8 total, 8 up, 6 in 2024-09-18T07:50:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[39175]: 4.15 scrub starts 2024-09-18T07:50:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[39175]: 4.15 scrub ok 2024-09-18T07:50:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[39175]: 4.9 scrub starts 2024-09-18T07:50:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[39175]: 4.9 scrub ok 2024-09-18T07:50:05.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[39175]: osdmap e535: 8 total, 8 up, 6 in 2024-09-18T07:50:05.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[29408]: pgmap v1022: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 2094/104478 objects misplaced (2.004%) 2024-09-18T07:50:05.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:50:05.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3744258016' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-18T07:50:05.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[29408]: from='client.36704 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:50:05.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[29408]: osdmap e534: 8 total, 8 up, 6 in 2024-09-18T07:50:05.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[29408]: 4.15 scrub starts 2024-09-18T07:50:05.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[29408]: 4.15 scrub ok 2024-09-18T07:50:05.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[29408]: 4.9 scrub starts 2024-09-18T07:50:05.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[29408]: 4.9 scrub ok 2024-09-18T07:50:05.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:05 smithi137 ceph-mon[29408]: osdmap e535: 8 total, 8 up, 6 in 2024-09-18T07:50:05.759 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:05.777 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:05.781 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:05.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:05.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:05.883 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:05.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:05.905 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:05.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:05.989 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:50:05.989 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:50:05.989 INFO:teuthology.orchestra.run.smithi203.stdout: "num_ops": 0 2024-09-18T07:50:05.989 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:50:06.006 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:06.028 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:06.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:06.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:06.129 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:06.150 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:06.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:06.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:06.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:06.270 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:50:06.270 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:50:06.276 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:06.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:06.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:06.294 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:50:06.294 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:50:06.332 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:50:06.332 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:50:06.356 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:50:06.357 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:50:06.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:06.383 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:50:06.384 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:50:06.411 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:06.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:06.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:06.412 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:50:06.413 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:50:06.448 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:50:06.449 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:50:06.487 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:50:06.487 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:50:06.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:06.558 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:06.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:06.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:06.660 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:06.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:06 smithi137 ceph-mon[29408]: 5.9 scrub starts 2024-09-18T07:50:06.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:06 smithi137 ceph-mon[29408]: 5.9 scrub ok 2024-09-18T07:50:06.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:06 smithi137 ceph-mon[29408]: 4.6 scrub starts 2024-09-18T07:50:06.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:06 smithi137 ceph-mon[29408]: 4.6 scrub ok 2024-09-18T07:50:06.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:06 smithi137 ceph-mon[29408]: osdmap e536: 8 total, 8 up, 6 in 2024-09-18T07:50:06.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:06 smithi137 ceph-mon[29408]: 11.b starting backfill to osd.5 from (0'0,0'0] MAX to 533'12030 2024-09-18T07:50:06.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:06 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/209834606' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:06.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:06 smithi137 ceph-mon[39175]: 5.9 scrub starts 2024-09-18T07:50:06.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:06 smithi137 ceph-mon[39175]: 5.9 scrub ok 2024-09-18T07:50:06.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:06 smithi137 ceph-mon[39175]: 4.6 scrub starts 2024-09-18T07:50:06.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:06 smithi137 ceph-mon[39175]: 4.6 scrub ok 2024-09-18T07:50:06.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:06 smithi137 ceph-mon[39175]: osdmap e536: 8 total, 8 up, 6 in 2024-09-18T07:50:06.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:06 smithi137 ceph-mon[39175]: 11.b starting backfill to osd.5 from (0'0,0'0] MAX to 533'12030 2024-09-18T07:50:06.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:06 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/209834606' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:06.697 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:06.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:06.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:06 smithi203 ceph-mon[32237]: 5.9 scrub starts 2024-09-18T07:50:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:06 smithi203 ceph-mon[32237]: 5.9 scrub ok 2024-09-18T07:50:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:06 smithi203 ceph-mon[32237]: 4.6 scrub starts 2024-09-18T07:50:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:06 smithi203 ceph-mon[32237]: 4.6 scrub ok 2024-09-18T07:50:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:06 smithi203 ceph-mon[32237]: osdmap e536: 8 total, 8 up, 6 in 2024-09-18T07:50:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:06 smithi203 ceph-mon[32237]: 11.b starting backfill to osd.5 from (0'0,0'0] MAX to 533'12030 2024-09-18T07:50:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:06 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/209834606' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:06.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:06.834 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:06.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:06.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:06.936 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:06.969 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:06.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:06.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:07.071 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:07.106 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:07.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:07.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:07.208 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:07.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:07.236 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:07.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:07.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:07.373 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:07.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:07.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:07.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:50:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:50:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:50:07.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:07 smithi137 ceph-mon[29408]: pgmap v1025: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.1 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 675 B/s rd, 0 B/s wr, 920 op/s; 2094/103305 objects misplaced (2.027%) 2024-09-18T07:50:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:07 smithi137 ceph-mon[29408]: 11.2 starting backfill to osd.1 from (0'0,0'0] MAX to 533'11802 2024-09-18T07:50:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:07 smithi137 ceph-mon[29408]: 11.6 starting backfill to osd.4 from (0'0,0'0] MAX to 533'11906 2024-09-18T07:50:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:07 smithi137 ceph-mon[29408]: 11.1 starting backfill to osd.4 from (0'0,0'0] MAX to 533'12110 2024-09-18T07:50:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:07 smithi137 ceph-mon[29408]: 11.8 starting backfill to osd.2 from (0'0,0'0] MAX to 533'12018 2024-09-18T07:50:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:07 smithi137 ceph-mon[29408]: 11.9 starting backfill to osd.7 from (0'0,0'0] MAX to 533'12002 2024-09-18T07:50:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:07 smithi137 ceph-mon[29408]: 5.15 scrub starts 2024-09-18T07:50:07.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:07 smithi137 ceph-mon[29408]: 5.15 scrub ok 2024-09-18T07:50:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:07 smithi137 ceph-mon[39175]: pgmap v1025: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.1 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 675 B/s rd, 0 B/s wr, 920 op/s; 2094/103305 objects misplaced (2.027%) 2024-09-18T07:50:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:07 smithi137 ceph-mon[39175]: 11.2 starting backfill to osd.1 from (0'0,0'0] MAX to 533'11802 2024-09-18T07:50:07.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:07 smithi137 ceph-mon[39175]: 11.6 starting backfill to osd.4 from (0'0,0'0] MAX to 533'11906 2024-09-18T07:50:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:07 smithi137 ceph-mon[39175]: 11.1 starting backfill to osd.4 from (0'0,0'0] MAX to 533'12110 2024-09-18T07:50:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:07 smithi137 ceph-mon[39175]: 11.8 starting backfill to osd.2 from (0'0,0'0] MAX to 533'12018 2024-09-18T07:50:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:07 smithi137 ceph-mon[39175]: 11.9 starting backfill to osd.7 from (0'0,0'0] MAX to 533'12002 2024-09-18T07:50:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:07 smithi137 ceph-mon[39175]: 5.15 scrub starts 2024-09-18T07:50:07.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:07 smithi137 ceph-mon[39175]: 5.15 scrub ok 2024-09-18T07:50:07.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:07.498 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:07.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:07.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:07.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:07.622 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:07.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:07 smithi203 ceph-mon[32237]: pgmap v1025: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.1 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 675 B/s rd, 0 B/s wr, 920 op/s; 2094/103305 objects misplaced (2.027%) 2024-09-18T07:50:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:07 smithi203 ceph-mon[32237]: 11.2 starting backfill to osd.1 from (0'0,0'0] MAX to 533'11802 2024-09-18T07:50:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:07 smithi203 ceph-mon[32237]: 11.6 starting backfill to osd.4 from (0'0,0'0] MAX to 533'11906 2024-09-18T07:50:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:07 smithi203 ceph-mon[32237]: 11.1 starting backfill to osd.4 from (0'0,0'0] MAX to 533'12110 2024-09-18T07:50:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:07 smithi203 ceph-mon[32237]: 11.8 starting backfill to osd.2 from (0'0,0'0] MAX to 533'12018 2024-09-18T07:50:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:07 smithi203 ceph-mon[32237]: 11.9 starting backfill to osd.7 from (0'0,0'0] MAX to 533'12002 2024-09-18T07:50:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:07 smithi203 ceph-mon[32237]: 5.15 scrub starts 2024-09-18T07:50:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:07 smithi203 ceph-mon[32237]: 5.15 scrub ok 2024-09-18T07:50:07.723 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:07.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:50:07.758 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:07.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:07.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:07.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:07.898 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:07.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:08.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:08.037 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:08.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:08.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:08.139 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:08.161 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:08.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:08.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:08.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:08.285 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:08.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:08.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:08.345 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:50:08 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:50:08.386 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:08.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:08.417 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:08.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:08.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:08.555 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:08.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:08.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:08.657 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:08.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:08 smithi137 ceph-mon[29408]: 2.0 scrub starts 2024-09-18T07:50:08.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:08 smithi137 ceph-mon[29408]: 2.0 scrub ok 2024-09-18T07:50:08.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:08 smithi137 ceph-mon[39175]: 2.0 scrub starts 2024-09-18T07:50:08.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:08 smithi137 ceph-mon[39175]: 2.0 scrub ok 2024-09-18T07:50:08.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:08.695 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:08.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:08.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:08 smithi203 ceph-mon[32237]: 2.0 scrub starts 2024-09-18T07:50:08.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:08 smithi203 ceph-mon[32237]: 2.0 scrub ok 2024-09-18T07:50:08.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:08.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:08.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:08.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:08.931 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:08.967 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:08.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:08.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:09.069 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:09.098 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:09.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:09.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:09.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:09.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:09.232 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:09.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:09.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:09.366 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:09.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:09.468 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:09.488 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:09.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:09.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:09.590 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:09.623 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:09.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:09.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:09.661 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:09 smithi137 ceph-mon[29408]: pgmap v1027: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.9 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 511 B/s rd, 0 B/s wr, 1.27k op/s; 13140/92568 objects misplaced (14.195%); 1.2 MiB/s, 24 objects/s recovering 2024-09-18T07:50:09.661 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:09 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:09.661 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:09 smithi137 ceph-mon[29408]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:50:09.661 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:09 smithi137 ceph-mon[29408]: 6.d scrub starts 2024-09-18T07:50:09.661 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:09 smithi137 ceph-mon[29408]: 6.d scrub ok 2024-09-18T07:50:09.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:09 smithi137 ceph-mon[39175]: pgmap v1027: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.9 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 511 B/s rd, 0 B/s wr, 1.27k op/s; 13140/92568 objects misplaced (14.195%); 1.2 MiB/s, 24 objects/s recovering 2024-09-18T07:50:09.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:09 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:09.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:09 smithi137 ceph-mon[39175]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:50:09.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:09 smithi137 ceph-mon[39175]: 6.d scrub starts 2024-09-18T07:50:09.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:09 smithi137 ceph-mon[39175]: 6.d scrub ok 2024-09-18T07:50:09.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:09 smithi203 ceph-mon[32237]: pgmap v1027: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.9 GiB data, 8.6 GiB used, 528 GiB / 536 GiB avail; 511 B/s rd, 0 B/s wr, 1.27k op/s; 13140/92568 objects misplaced (14.195%); 1.2 MiB/s, 24 objects/s recovering 2024-09-18T07:50:09.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:09 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:09 smithi203 ceph-mon[32237]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:50:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:09 smithi203 ceph-mon[32237]: 6.d scrub starts 2024-09-18T07:50:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:09 smithi203 ceph-mon[32237]: 6.d scrub ok 2024-09-18T07:50:09.725 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:09.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:09.747 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:09.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:09.848 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:09.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:09.881 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:09.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:09.982 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:10.019 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:10.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:10.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:10.082 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 5, 7, 6, 1] out_osds: [3, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 6, 7, 4, 5] 2024-09-18T07:50:10.083 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:50:10.083 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-18T07:50:10.083 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.577406 2024-09-18T07:50:10.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 0.5774056832141756 2024-09-18T07:50:10.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:10.147 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:10.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:10.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:10.249 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:10.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:10.301 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:10.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:10.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:10.441 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:10.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:10.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:10.563 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:10.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:10.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:10.665 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:10.705 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:10.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:10.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:10.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:10.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:10.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:10.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:10.933 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:10.972 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:10.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:10.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:11.074 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:11.097 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:11.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:11.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:11.199 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:11.220 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:11.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:11.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:11.257 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:11.322 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:11.357 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:11.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:11.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:11.433 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:11 smithi137 ceph-mon[29408]: pgmap v1028: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.9 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 695 op/s; 13140/91947 objects misplaced (14.291%); 1.2 MiB/s, 26 objects/s recovering 2024-09-18T07:50:11.433 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:11 smithi137 ceph-mon[29408]: osdmap e537: 8 total, 8 up, 6 in 2024-09-18T07:50:11.433 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:11 smithi137 ceph-mon[39175]: pgmap v1028: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.9 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 695 op/s; 13140/91947 objects misplaced (14.291%); 1.2 MiB/s, 26 objects/s recovering 2024-09-18T07:50:11.434 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:11 smithi137 ceph-mon[39175]: osdmap e537: 8 total, 8 up, 6 in 2024-09-18T07:50:11.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:11.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:11 smithi203 ceph-mon[32237]: pgmap v1028: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.9 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 695 op/s; 13140/91947 objects misplaced (14.291%); 1.2 MiB/s, 26 objects/s recovering 2024-09-18T07:50:11.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:11 smithi203 ceph-mon[32237]: osdmap e537: 8 total, 8 up, 6 in 2024-09-18T07:50:11.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:11.494 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:11.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:11.595 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:11.624 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:11.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:11.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:11.726 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:11.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:11.766 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:11.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:11.867 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:11.899 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:11.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:11.908 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:12.001 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:12.037 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:12.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:12.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:12.139 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:12.167 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:12.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:12.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:12.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:12.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:12.307 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:12.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:12.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:12.446 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:12.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:12.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:12.547 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:12.580 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:12.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:12.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:12.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:12.718 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:12.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:12.819 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:12.855 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:12.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:12.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:12.957 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:12.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:12.993 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:12.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:13.094 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:13.121 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:13.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:13.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:13.196 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:50:13.200 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.5 primary-affinity to 0.577406 (8378402) 2024-09-18T07:50:13.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:50:13.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:13.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:13.302 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:13.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:13.376 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T07:50:13.403 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:13.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:13 smithi137 ceph-mon[29408]: pgmap v1030: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.7 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 854 op/s; 12186/86064 objects misplaced (14.159%); 1.0 MiB/s, 23 objects/s recovering 2024-09-18T07:50:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:13 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/7006739' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:50:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:13 smithi137 ceph-mon[29408]: from='client.26836 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:50:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:13 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:50:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:13 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1684507526' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.5774056832141756} : dispatch 2024-09-18T07:50:13.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:13 smithi137 ceph-mon[29408]: from='client.26839 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.5774056832141756} : dispatch 2024-09-18T07:50:13.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:13 smithi137 ceph-mon[39175]: pgmap v1030: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.7 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 854 op/s; 12186/86064 objects misplaced (14.159%); 1.0 MiB/s, 23 objects/s recovering 2024-09-18T07:50:13.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:13 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/7006739' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:50:13.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:13 smithi137 ceph-mon[39175]: from='client.26836 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:50:13.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:13 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:50:13.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:13 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1684507526' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.5774056832141756} : dispatch 2024-09-18T07:50:13.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:13 smithi137 ceph-mon[39175]: from='client.26839 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.5774056832141756} : dispatch 2024-09-18T07:50:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:13 smithi203 ceph-mon[32237]: pgmap v1030: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.7 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 0 B/s wr, 854 op/s; 12186/86064 objects misplaced (14.159%); 1.0 MiB/s, 23 objects/s recovering 2024-09-18T07:50:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:13 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/7006739' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:50:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:13 smithi203 ceph-mon[32237]: from='client.26836 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:50:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:13 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:50:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:13 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1684507526' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.5774056832141756} : dispatch 2024-09-18T07:50:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:13 smithi203 ceph-mon[32237]: from='client.26839 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.5774056832141756} : dispatch 2024-09-18T07:50:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:13.469 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:13.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:13.570 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:13.617 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:13.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:13.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:13.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:13.750 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:13.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:13.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:13.814 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:50:13.852 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:13.908 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:13.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:14.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:14.032 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:14.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:14.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:14.134 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:14.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:14.168 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:14.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:14.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:14 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:50:14.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:14 smithi203 ceph-mon[32237]: from='client.26836 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:50:14.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:14 smithi203 ceph-mon[32237]: from='client.26839 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5774056832141756}]': finished 2024-09-18T07:50:14.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:14 smithi203 ceph-mon[32237]: osdmap e538: 8 total, 8 up, 6 in 2024-09-18T07:50:14.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:14.304 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:14.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:14.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:14.406 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:14.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:14 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:50:14.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:14 smithi137 ceph-mon[29408]: from='client.26836 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:50:14.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:14 smithi137 ceph-mon[29408]: from='client.26839 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5774056832141756}]': finished 2024-09-18T07:50:14.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:14 smithi137 ceph-mon[29408]: osdmap e538: 8 total, 8 up, 6 in 2024-09-18T07:50:14.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:14 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:50:14.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:14 smithi137 ceph-mon[39175]: from='client.26836 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:50:14.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:14 smithi137 ceph-mon[39175]: from='client.26839 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5774056832141756}]': finished 2024-09-18T07:50:14.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:14 smithi137 ceph-mon[39175]: osdmap e538: 8 total, 8 up, 6 in 2024-09-18T07:50:14.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:14.428 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:14.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:14.533 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:14.566 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:14.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:14.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:14.668 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:14.702 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:14.703 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:14.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:14.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:14.804 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:14.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:14.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:14.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:14.934 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:14.955 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:14.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:14.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:15.057 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:15.080 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:15.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:15.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:15.208 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:15.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:15.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:15.270 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:50:15.270 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:50:15.305 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:50:15.305 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:50:15.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:15.339 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:50:15.339 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:50:15.346 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:15.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:15.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:15.369 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:50:15.369 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:50:15.400 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:50:15.401 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:50:15.401 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:15 smithi137 ceph-mon[29408]: pgmap v1032: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.6 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 398 B/s rd, 0 B/s wr, 1.13k op/s; 10481/77073 objects misplaced (13.599%); 4.7 MiB/s, 81 objects/s recovering 2024-09-18T07:50:15.401 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:15 smithi137 ceph-mon[29408]: osdmap e539: 8 total, 8 up, 6 in 2024-09-18T07:50:15.401 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:15 smithi137 ceph-mon[29408]: osdmap e540: 8 total, 8 up, 6 in 2024-09-18T07:50:15.402 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:15 smithi137 ceph-mon[39175]: pgmap v1032: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.6 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 398 B/s rd, 0 B/s wr, 1.13k op/s; 10481/77073 objects misplaced (13.599%); 4.7 MiB/s, 81 objects/s recovering 2024-09-18T07:50:15.402 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:15 smithi137 ceph-mon[39175]: osdmap e539: 8 total, 8 up, 6 in 2024-09-18T07:50:15.402 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:15 smithi137 ceph-mon[39175]: osdmap e540: 8 total, 8 up, 6 in 2024-09-18T07:50:15.431 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:50:15.431 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:50:15.448 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:15.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:15 smithi203 ceph-mon[32237]: pgmap v1032: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.6 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 398 B/s rd, 0 B/s wr, 1.13k op/s; 10481/77073 objects misplaced (13.599%); 4.7 MiB/s, 81 objects/s recovering 2024-09-18T07:50:15.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:15 smithi203 ceph-mon[32237]: osdmap e539: 8 total, 8 up, 6 in 2024-09-18T07:50:15.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:15 smithi203 ceph-mon[32237]: osdmap e540: 8 total, 8 up, 6 in 2024-09-18T07:50:15.469 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:15.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:15.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:15.486 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:50:15.487 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:50:15.519 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:50:15.519 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:50:15.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:15.607 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:15.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:15.709 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:15.745 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:15.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:15.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:15.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:15.877 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:15.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:15.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:15.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:16.008 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:16.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:16.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:16.109 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:16.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:16.139 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:16.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:16.151 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:50:16.151 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [], 2024-09-18T07:50:16.151 INFO:teuthology.orchestra.run.smithi203.stdout: "complaint_time": 30, 2024-09-18T07:50:16.151 INFO:teuthology.orchestra.run.smithi203.stdout: "num_blocked_ops": 0 2024-09-18T07:50:16.151 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:50:16.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:16.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:16.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:16.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:16.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:16.412 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:16 smithi203 ceph-mon[32237]: 1.4 scrub starts 2024-09-18T07:50:16.413 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:16 smithi203 ceph-mon[32237]: 1.4 scrub ok 2024-09-18T07:50:16.413 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:16 smithi203 ceph-mon[32237]: 1.1 scrub starts 2024-09-18T07:50:16.413 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:16 smithi203 ceph-mon[32237]: 1.1 scrub ok 2024-09-18T07:50:16.413 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:16 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/210896836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:16.414 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:16.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:16.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:16.516 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:16.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:16.547 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:16.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:16.632 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:50:16.648 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:16.665 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:16 smithi137 ceph-mon[29408]: 1.4 scrub starts 2024-09-18T07:50:16.665 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:16 smithi137 ceph-mon[29408]: 1.4 scrub ok 2024-09-18T07:50:16.665 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:16 smithi137 ceph-mon[29408]: 1.1 scrub starts 2024-09-18T07:50:16.665 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:16 smithi137 ceph-mon[29408]: 1.1 scrub ok 2024-09-18T07:50:16.665 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:16 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/210896836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:16.665 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:16 smithi137 ceph-mon[39175]: 1.4 scrub starts 2024-09-18T07:50:16.665 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:16 smithi137 ceph-mon[39175]: 1.4 scrub ok 2024-09-18T07:50:16.665 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:16 smithi137 ceph-mon[39175]: 1.1 scrub starts 2024-09-18T07:50:16.665 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:16 smithi137 ceph-mon[39175]: 1.1 scrub ok 2024-09-18T07:50:16.666 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:16 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/210896836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:16.684 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:16.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:16.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:16.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:16.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:16.825 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:16.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:16.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:16.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:16.966 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:16.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:17.043 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:17.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:17.113 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T07:50:17.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:17.118 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:17.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:17.219 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:17.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:17.267 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:17.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:17.355 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:50:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:50:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:50:17.355 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:17 smithi137 ceph-mon[29408]: 3.17 scrub starts 2024-09-18T07:50:17.356 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:17 smithi137 ceph-mon[29408]: 3.17 scrub ok 2024-09-18T07:50:17.356 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:17 smithi137 ceph-mon[29408]: pgmap v1035: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 894 B/s rd, 0 B/s wr, 935 op/s; 10481/75888 objects misplaced (13.811%); 5.1 MiB/s, 81 objects/s recovering 2024-09-18T07:50:17.356 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:17 smithi137 ceph-mon[29408]: 1.8 scrub starts 2024-09-18T07:50:17.356 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:17 smithi137 ceph-mon[29408]: 1.8 scrub ok 2024-09-18T07:50:17.356 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:17 smithi137 ceph-mon[39175]: 3.17 scrub starts 2024-09-18T07:50:17.356 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:17 smithi137 ceph-mon[39175]: 3.17 scrub ok 2024-09-18T07:50:17.356 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:17 smithi137 ceph-mon[39175]: pgmap v1035: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 894 B/s rd, 0 B/s wr, 935 op/s; 10481/75888 objects misplaced (13.811%); 5.1 MiB/s, 81 objects/s recovering 2024-09-18T07:50:17.356 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:17 smithi137 ceph-mon[39175]: 1.8 scrub starts 2024-09-18T07:50:17.356 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:17 smithi137 ceph-mon[39175]: 1.8 scrub ok 2024-09-18T07:50:17.369 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:17.410 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:17.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:17.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:17 smithi203 ceph-mon[32237]: 3.17 scrub starts 2024-09-18T07:50:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:17 smithi203 ceph-mon[32237]: 3.17 scrub ok 2024-09-18T07:50:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:17 smithi203 ceph-mon[32237]: pgmap v1035: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.5 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 894 B/s rd, 0 B/s wr, 935 op/s; 10481/75888 objects misplaced (13.811%); 5.1 MiB/s, 81 objects/s recovering 2024-09-18T07:50:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:17 smithi203 ceph-mon[32237]: 1.8 scrub starts 2024-09-18T07:50:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:17 smithi203 ceph-mon[32237]: 1.8 scrub ok 2024-09-18T07:50:17.512 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:17.560 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:17.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:17.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:17.578 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:50:17.664 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:17.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:17.713 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:17.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:17.814 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:17.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:17.853 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:17.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:17.954 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:17.982 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:17.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:17.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:18.084 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:18.113 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:18.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:18.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:18.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:18.235 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:50:18.237 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-09-18T07:50:18.238 INFO:teuthology.orchestra.run.smithi203.stdout: "size": 20, 2024-09-18T07:50:18.238 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 600, 2024-09-18T07:50:18.238 INFO:teuthology.orchestra.run.smithi203.stdout: "ops": [ 2024-09-18T07:50:18.238 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.238 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24412.0:3402 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 94763678224896 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-09-18T07:50:18.238 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:11.552301+0000", 2024-09-18T07:50:18.238 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 186.67974144499999, 2024-09-18T07:50:18.238 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.65149643400000001, 2024-09-18T07:50:18.238 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.238 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.239 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:50:18.239 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.239 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24412", 2024-09-18T07:50:18.239 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/4233491205", 2024-09-18T07:50:18.239 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 3402 2024-09-18T07:50:18.239 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.239 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.239 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.239 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.239 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552301+0000", 2024-09-18T07:50:18.239 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.239 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.239 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.239 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.240 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552301+0000", 2024-09-18T07:50:18.240 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.240 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.240 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.240 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.240 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552303+0000", 2024-09-18T07:50:18.240 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2230000000000001e-06 2024-09-18T07:50:18.240 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.240 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.240 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.240 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552308+0000", 2024-09-18T07:50:18.241 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8840000000000002e-06 2024-09-18T07:50:18.241 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.241 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.241 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.241 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552328+0000", 2024-09-18T07:50:18.241 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0360000000000002e-05 2024-09-18T07:50:18.241 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.241 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.241 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.241 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552337+0000", 2024-09-18T07:50:18.241 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3899999999999993e-06 2024-09-18T07:50:18.241 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.241 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.242 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.242 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552419+0000", 2024-09-18T07:50:18.242 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2566999999999997e-05 2024-09-18T07:50:18.242 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.242 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.242 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.242 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552429+0000", 2024-09-18T07:50:18.242 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1840000000000005e-06 2024-09-18T07:50:18.242 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.242 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.242 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.242 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:12.202182+0000", 2024-09-18T07:50:18.243 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.64975312100000004 2024-09-18T07:50:18.243 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.243 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.243 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.243 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:12.202354+0000", 2024-09-18T07:50:18.243 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00017184399999999999 2024-09-18T07:50:18.243 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.243 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.243 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.243 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:12.203797+0000", 2024-09-18T07:50:18.244 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014438610000000001 2024-09-18T07:50:18.244 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.244 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.244 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.244 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.244 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.244 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.24412.0:3406 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 94763678224896 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-09-18T07:50:18.244 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:11.552354+0000", 2024-09-18T07:50:18.244 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 186.67968812300001, 2024-09-18T07:50:18.244 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.65190388799999999, 2024-09-18T07:50:18.244 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.244 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.244 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "started", 2024-09-18T07:50:18.244 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.245 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.24412", 2024-09-18T07:50:18.245 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/4233491205", 2024-09-18T07:50:18.245 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 3406 2024-09-18T07:50:18.245 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.245 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.245 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.245 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.245 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552354+0000", 2024-09-18T07:50:18.245 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.245 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.245 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.245 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.245 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552354+0000", 2024-09-18T07:50:18.246 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.246 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.246 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.246 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.246 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552355+0000", 2024-09-18T07:50:18.246 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.0800000000000004e-07 2024-09-18T07:50:18.246 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.246 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.246 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.246 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552359+0000", 2024-09-18T07:50:18.246 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.49e-06 2024-09-18T07:50:18.246 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.247 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.247 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.247 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552365+0000", 2024-09-18T07:50:18.247 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.742e-06 2024-09-18T07:50:18.247 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.247 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.247 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.247 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552371+0000", 2024-09-18T07:50:18.247 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9290000000000003e-06 2024-09-18T07:50:18.247 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.247 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.247 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.247 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552454+0000", 2024-09-18T07:50:18.248 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3115000000000006e-05 2024-09-18T07:50:18.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.248 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.248 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.248 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:11.552462+0000", 2024-09-18T07:50:18.248 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.9820000000000008e-06 2024-09-18T07:50:18.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.248 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.248 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.248 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:12.203343+0000", 2024-09-18T07:50:18.248 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.65088101700000001 2024-09-18T07:50:18.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.248 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.249 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.249 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:12.203427+0000", 2024-09-18T07:50:18.249 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3855000000000003e-05 2024-09-18T07:50:18.249 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.249 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.249 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.249 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:12.204258+0000", 2024-09-18T07:50:18.249 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00083095 2024-09-18T07:50:18.249 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.249 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.249 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.249 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.249 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.250 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30810 11.7 11:e3ec4fcf:::benchmark_data_smithi137_181069_object30809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:50:18.250 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.172317+0000", 2024-09-18T07:50:18.250 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 165.05972581, 2024-09-18T07:50:18.250 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0154700940000001, 2024-09-18T07:50:18.250 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.250 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.250 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.250 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.250 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.250 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.250 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30810 2024-09-18T07:50:18.250 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.250 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.250 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.251 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.251 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.172317+0000", 2024-09-18T07:50:18.251 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.251 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.251 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.251 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.251 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.172317+0000", 2024-09-18T07:50:18.251 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.251 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.251 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.251 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.251 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.172319+0000", 2024-09-18T07:50:18.251 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.836e-06 2024-09-18T07:50:18.251 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.252 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.252 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.252 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.172378+0000", 2024-09-18T07:50:18.252 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8761999999999997e-05 2024-09-18T07:50:18.252 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.252 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.252 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.252 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.172396+0000", 2024-09-18T07:50:18.252 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8122999999999999e-05 2024-09-18T07:50:18.252 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.252 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.253 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.253 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.172406+0000", 2024-09-18T07:50:18.253 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4150000000000005e-06 2024-09-18T07:50:18.253 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.254 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.254 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.254 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.172469+0000", 2024-09-18T07:50:18.254 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3324000000000006e-05 2024-09-18T07:50:18.254 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.254 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.254 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.255 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.172477+0000", 2024-09-18T07:50:18.255 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3950000000000001e-06 2024-09-18T07:50:18.255 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.255 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.255 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.255 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.183147+0000", 2024-09-18T07:50:18.255 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.010669912 2024-09-18T07:50:18.255 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.255 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.255 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.255 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.183289+0000", 2024-09-18T07:50:18.255 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014143700000000001 2024-09-18T07:50:18.255 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.255 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.256 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.256 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.183444+0000", 2024-09-18T07:50:18.256 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015522600000000001 2024-09-18T07:50:18.256 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.256 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.256 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.256 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.185800+0000", 2024-09-18T07:50:18.256 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023560489999999998 2024-09-18T07:50:18.256 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.256 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.256 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.256 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.185869+0000", 2024-09-18T07:50:18.257 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9043999999999998e-05 2024-09-18T07:50:18.257 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.257 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.257 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.257 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.187718+0000", 2024-09-18T07:50:18.257 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001848567 2024-09-18T07:50:18.257 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.257 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.257 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.257 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.187755+0000", 2024-09-18T07:50:18.258 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6962000000000003e-05 2024-09-18T07:50:18.258 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.258 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.258 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.258 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.187787+0000", 2024-09-18T07:50:18.258 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2042000000000003e-05 2024-09-18T07:50:18.258 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.258 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.258 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.258 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.258 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.258 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30838 11.7 11:e2d1080e:::benchmark_data_smithi137_181069_object30837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:50:18.258 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.201700+0000", 2024-09-18T07:50:18.259 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 165.03034283900001, 2024-09-18T07:50:18.259 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.99197580799999996, 2024-09-18T07:50:18.259 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.259 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.259 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.259 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.259 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.259 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.259 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30838 2024-09-18T07:50:18.259 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.259 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.259 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.259 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.260 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201700+0000", 2024-09-18T07:50:18.260 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.260 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.260 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.260 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.260 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201700+0000", 2024-09-18T07:50:18.260 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.260 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.260 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.260 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.260 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201702+0000", 2024-09-18T07:50:18.260 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1730000000000002e-06 2024-09-18T07:50:18.260 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.261 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.261 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.261 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201754+0000", 2024-09-18T07:50:18.261 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2125e-05 2024-09-18T07:50:18.261 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.261 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.261 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.261 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201771+0000", 2024-09-18T07:50:18.261 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6708e-05 2024-09-18T07:50:18.261 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.261 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.261 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.262 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201779+0000", 2024-09-18T07:50:18.262 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.4409999999999993e-06 2024-09-18T07:50:18.262 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.262 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.262 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.262 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201847+0000", 2024-09-18T07:50:18.262 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8196999999999998e-05 2024-09-18T07:50:18.262 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.262 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.262 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.262 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201856+0000", 2024-09-18T07:50:18.262 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2720000000000003e-06 2024-09-18T07:50:18.263 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.263 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.263 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.263 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.185900+0000", 2024-09-18T07:50:18.263 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.98404355899999996 2024-09-18T07:50:18.263 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.263 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.263 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.263 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.186018+0000", 2024-09-18T07:50:18.263 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000117685 2024-09-18T07:50:18.263 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.263 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.263 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.263 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.186171+0000", 2024-09-18T07:50:18.264 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000153187 2024-09-18T07:50:18.264 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.264 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.264 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.264 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.189010+0000", 2024-09-18T07:50:18.264 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0028390619999999998 2024-09-18T07:50:18.264 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.264 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.264 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.264 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.189046+0000", 2024-09-18T07:50:18.264 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5655999999999999e-05 2024-09-18T07:50:18.264 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.264 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.264 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.265 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193582+0000", 2024-09-18T07:50:18.265 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0045365589999999999 2024-09-18T07:50:18.265 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.265 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.265 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.265 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193631+0000", 2024-09-18T07:50:18.265 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8959999999999999e-05 2024-09-18T07:50:18.265 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.265 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.265 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.265 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193675+0000", 2024-09-18T07:50:18.265 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4224000000000002e-05 2024-09-18T07:50:18.265 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.265 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.266 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.266 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.266 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.266 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30840 11.7 11:e6e0bbbb:::benchmark_data_smithi137_181069_object30839:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:50:18.266 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.201799+0000", 2024-09-18T07:50:18.266 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 165.03024329300001, 2024-09-18T07:50:18.266 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.99198059800000005, 2024-09-18T07:50:18.266 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.266 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.266 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.266 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.266 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.266 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.266 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30840 2024-09-18T07:50:18.267 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.267 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.267 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.267 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.267 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201799+0000", 2024-09-18T07:50:18.267 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.267 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.267 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.267 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.267 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201799+0000", 2024-09-18T07:50:18.267 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.267 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.268 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.268 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.268 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201800+0000", 2024-09-18T07:50:18.268 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.4200000000000004e-07 2024-09-18T07:50:18.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.268 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.268 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.268 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201834+0000", 2024-09-18T07:50:18.268 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.3600999999999999e-05 2024-09-18T07:50:18.268 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.268 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.268 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.268 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201841+0000", 2024-09-18T07:50:18.268 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1389999999999998e-06 2024-09-18T07:50:18.269 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.269 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.269 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.269 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201847+0000", 2024-09-18T07:50:18.269 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8889999999999999e-06 2024-09-18T07:50:18.269 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.269 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.269 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.269 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201877+0000", 2024-09-18T07:50:18.269 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0195000000000001e-05 2024-09-18T07:50:18.269 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.270 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.270 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.270 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.201882+0000", 2024-09-18T07:50:18.270 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5450000000000003e-06 2024-09-18T07:50:18.270 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.270 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.270 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.270 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.187829+0000", 2024-09-18T07:50:18.270 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.98594618099999998 2024-09-18T07:50:18.270 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.270 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.271 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.271 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.187951+0000", 2024-09-18T07:50:18.271 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000121957 2024-09-18T07:50:18.271 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.271 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.271 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.271 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.188097+0000", 2024-09-18T07:50:18.271 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000146055 2024-09-18T07:50:18.271 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.271 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.271 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.271 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.191952+0000", 2024-09-18T07:50:18.271 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0038551369999999998 2024-09-18T07:50:18.272 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.272 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.272 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.272 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.192101+0000", 2024-09-18T07:50:18.272 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000149173 2024-09-18T07:50:18.272 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.272 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.272 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.272 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193684+0000", 2024-09-18T07:50:18.272 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001583416 2024-09-18T07:50:18.272 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.272 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.272 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.273 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193742+0000", 2024-09-18T07:50:18.273 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7515000000000001e-05 2024-09-18T07:50:18.273 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.273 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.273 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.273 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193780+0000", 2024-09-18T07:50:18.273 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.7852999999999998e-05 2024-09-18T07:50:18.273 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.273 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.273 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.273 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.273 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.274 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30843 11.7 11:eb1a4352:::benchmark_data_smithi137_181069_object30842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:50:18.274 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.208267+0000", 2024-09-18T07:50:18.274 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 165.02377541300001, 2024-09-18T07:50:18.274 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.98756237099999999, 2024-09-18T07:50:18.274 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.274 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.274 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.274 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.274 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.274 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.274 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30843 2024-09-18T07:50:18.274 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.274 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.274 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.275 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.275 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.208267+0000", 2024-09-18T07:50:18.275 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.275 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.275 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.275 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.275 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.208267+0000", 2024-09-18T07:50:18.275 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.275 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.275 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.275 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.275 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.208269+0000", 2024-09-18T07:50:18.275 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.0020000000000001e-06 2024-09-18T07:50:18.275 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.276 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.276 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.276 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.208324+0000", 2024-09-18T07:50:18.276 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5457000000000001e-05 2024-09-18T07:50:18.276 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.276 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.276 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.276 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.208338+0000", 2024-09-18T07:50:18.276 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3631999999999999e-05 2024-09-18T07:50:18.276 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.276 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.276 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.276 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.208347+0000", 2024-09-18T07:50:18.276 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.5019999999999992e-06 2024-09-18T07:50:18.277 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.277 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.277 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.277 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.208411+0000", 2024-09-18T07:50:18.277 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.3935000000000004e-05 2024-09-18T07:50:18.277 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.277 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.277 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.277 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.208422+0000", 2024-09-18T07:50:18.277 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1834000000000001e-05 2024-09-18T07:50:18.277 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.277 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.278 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.278 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.189072+0000", 2024-09-18T07:50:18.278 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.98064952599999999 2024-09-18T07:50:18.278 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.278 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.278 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.278 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.189189+0000", 2024-09-18T07:50:18.278 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000117236 2024-09-18T07:50:18.278 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.278 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.278 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.278 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.189307+0000", 2024-09-18T07:50:18.279 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000117862 2024-09-18T07:50:18.279 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.279 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.279 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.279 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.191991+0000", 2024-09-18T07:50:18.279 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0026844799999999999 2024-09-18T07:50:18.279 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.279 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.279 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.279 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193786+0000", 2024-09-18T07:50:18.279 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001794992 2024-09-18T07:50:18.279 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.279 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.280 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.280 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195746+0000", 2024-09-18T07:50:18.280 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0019596940000000001 2024-09-18T07:50:18.280 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.280 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.280 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.280 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195792+0000", 2024-09-18T07:50:18.280 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6360000000000003e-05 2024-09-18T07:50:18.280 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.280 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.280 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.281 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195829+0000", 2024-09-18T07:50:18.281 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.6859000000000001e-05 2024-09-18T07:50:18.281 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.281 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.281 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.281 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.281 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.281 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30877 11.7 11:e09287fc:::benchmark_data_smithi137_181069_object30876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:50:18.281 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.232483+0000", 2024-09-18T07:50:18.281 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 164.99955921700001, 2024-09-18T07:50:18.281 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.96350860000000005, 2024-09-18T07:50:18.281 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.281 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.281 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.282 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.282 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.282 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.282 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30877 2024-09-18T07:50:18.282 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.282 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.282 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.282 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.282 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.232483+0000", 2024-09-18T07:50:18.282 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.282 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.282 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.283 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.283 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.232483+0000", 2024-09-18T07:50:18.283 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.283 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.283 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.283 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.283 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.232486+0000", 2024-09-18T07:50:18.283 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3740000000000001e-06 2024-09-18T07:50:18.283 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.283 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.283 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:18.284 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:18.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:18.285 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.232568+0000", 2024-09-18T07:50:18.285 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2531e-05 2024-09-18T07:50:18.285 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.286 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.286 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.286 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.232581+0000", 2024-09-18T07:50:18.286 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.3210000000000001e-05 2024-09-18T07:50:18.286 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.286 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.286 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.286 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.232588+0000", 2024-09-18T07:50:18.286 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.9129999999999997e-06 2024-09-18T07:50:18.287 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.287 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.287 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.287 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.232649+0000", 2024-09-18T07:50:18.287 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1192000000000006e-05 2024-09-18T07:50:18.287 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.287 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.287 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.287 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.232659+0000", 2024-09-18T07:50:18.287 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.1379999999999996e-06 2024-09-18T07:50:18.287 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.287 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.288 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.288 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.190261+0000", 2024-09-18T07:50:18.288 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.95760236600000004 2024-09-18T07:50:18.288 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.288 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.288 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.288 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.190368+0000", 2024-09-18T07:50:18.288 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000107292 2024-09-18T07:50:18.288 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.288 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.288 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.288 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.190492+0000", 2024-09-18T07:50:18.288 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012389700000000001 2024-09-18T07:50:18.289 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.289 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.289 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.289 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.192043+0000", 2024-09-18T07:50:18.289 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.001550645 2024-09-18T07:50:18.289 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.289 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.289 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.290 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195649+0000", 2024-09-18T07:50:18.290 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003606068 2024-09-18T07:50:18.290 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.290 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.290 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.290 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195915+0000", 2024-09-18T07:50:18.290 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00026607699999999998 2024-09-18T07:50:18.290 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.290 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.290 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.290 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195961+0000", 2024-09-18T07:50:18.290 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5877000000000001e-05 2024-09-18T07:50:18.290 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.291 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.291 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.291 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195992+0000", 2024-09-18T07:50:18.291 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1019999999999998e-05 2024-09-18T07:50:18.291 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.291 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.291 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.291 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.291 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.291 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30915 11.7 11:e1818ecf:::benchmark_data_smithi137_181069_object30914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:50:18.291 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.260052+0000", 2024-09-18T07:50:18.291 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 164.97199004800001, 2024-09-18T07:50:18.291 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93627841599999995, 2024-09-18T07:50:18.292 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.292 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.292 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.292 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.292 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.292 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.292 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30915 2024-09-18T07:50:18.292 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.292 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.292 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.292 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.292 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.260052+0000", 2024-09-18T07:50:18.292 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.292 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.293 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.293 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.293 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.260052+0000", 2024-09-18T07:50:18.293 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.293 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.293 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.293 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.293 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.260054+0000", 2024-09-18T07:50:18.293 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.1179999999999999e-06 2024-09-18T07:50:18.293 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.293 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.293 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.293 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.260094+0000", 2024-09-18T07:50:18.294 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9935999999999998e-05 2024-09-18T07:50:18.294 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.294 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.294 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.294 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.260109+0000", 2024-09-18T07:50:18.294 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.4521e-05 2024-09-18T07:50:18.294 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.294 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.294 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.294 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.260117+0000", 2024-09-18T07:50:18.294 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.8699999999999992e-06 2024-09-18T07:50:18.294 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.294 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.295 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.295 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.260174+0000", 2024-09-18T07:50:18.295 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7018000000000002e-05 2024-09-18T07:50:18.295 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.295 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.295 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.295 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.260183+0000", 2024-09-18T07:50:18.295 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.7299999999999994e-06 2024-09-18T07:50:18.295 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.295 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.295 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.295 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.192246+0000", 2024-09-18T07:50:18.295 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.932063737 2024-09-18T07:50:18.295 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.296 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.296 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.296 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.192368+0000", 2024-09-18T07:50:18.296 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012155999999999999 2024-09-18T07:50:18.296 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.296 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.296 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.296 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.192511+0000", 2024-09-18T07:50:18.296 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014365800000000001 2024-09-18T07:50:18.297 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.297 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.297 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.297 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195843+0000", 2024-09-18T07:50:18.297 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003331442 2024-09-18T07:50:18.297 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.297 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.297 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.297 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195875+0000", 2024-09-18T07:50:18.297 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.1890000000000001e-05 2024-09-18T07:50:18.297 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.297 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.297 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.298 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196056+0000", 2024-09-18T07:50:18.298 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00018159599999999999 2024-09-18T07:50:18.298 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.298 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.298 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.298 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196289+0000", 2024-09-18T07:50:18.298 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000232859 2024-09-18T07:50:18.298 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.298 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.298 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.299 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196331+0000", 2024-09-18T07:50:18.299 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1480999999999998e-05 2024-09-18T07:50:18.299 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.299 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.299 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.299 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.299 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.299 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30925 11.7 11:ec9c9156:::benchmark_data_smithi137_181069_object30924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:50:18.299 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.264403+0000", 2024-09-18T07:50:18.299 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 164.96763910999999, 2024-09-18T07:50:18.299 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93234712799999997, 2024-09-18T07:50:18.299 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.299 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.300 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.300 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.300 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.300 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.300 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30925 2024-09-18T07:50:18.300 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.300 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.300 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.300 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.300 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.264403+0000", 2024-09-18T07:50:18.300 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.300 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.300 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.301 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.301 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.264403+0000", 2024-09-18T07:50:18.301 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.301 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.301 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.301 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.301 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.264406+0000", 2024-09-18T07:50:18.301 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.446e-06 2024-09-18T07:50:18.301 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.301 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.301 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.301 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.264478+0000", 2024-09-18T07:50:18.302 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2671999999999998e-05 2024-09-18T07:50:18.302 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.302 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.302 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.302 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.264498+0000", 2024-09-18T07:50:18.302 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9497e-05 2024-09-18T07:50:18.302 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.302 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.302 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.302 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.264507+0000", 2024-09-18T07:50:18.302 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0019999999999995e-06 2024-09-18T07:50:18.302 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.302 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.303 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.303 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.264575+0000", 2024-09-18T07:50:18.303 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.8256999999999994e-05 2024-09-18T07:50:18.303 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.303 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.303 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.303 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.264586+0000", 2024-09-18T07:50:18.303 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0481e-05 2024-09-18T07:50:18.303 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.303 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.303 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.303 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193815+0000", 2024-09-18T07:50:18.303 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92922892999999995 2024-09-18T07:50:18.304 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.304 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.304 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.304 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.193923+0000", 2024-09-18T07:50:18.304 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000108276 2024-09-18T07:50:18.304 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.304 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.304 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.304 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.194043+0000", 2024-09-18T07:50:18.304 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000119974 2024-09-18T07:50:18.304 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.304 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.304 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.304 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.195888+0000", 2024-09-18T07:50:18.305 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0018447719999999999 2024-09-18T07:50:18.305 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.305 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.305 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.305 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196025+0000", 2024-09-18T07:50:18.305 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013774600000000001 2024-09-18T07:50:18.305 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.305 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.305 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.305 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196569+0000", 2024-09-18T07:50:18.305 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00054400399999999995 2024-09-18T07:50:18.305 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.305 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.306 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.306 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196710+0000", 2024-09-18T07:50:18.306 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00014022899999999999 2024-09-18T07:50:18.306 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.306 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.306 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.306 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196750+0000", 2024-09-18T07:50:18.306 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0842e-05 2024-09-18T07:50:18.306 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.306 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.306 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.306 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.307 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.307 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30928 11.7 11:e64e9d9c:::benchmark_data_smithi137_181069_object30927:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:50:18.307 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.268666+0000", 2024-09-18T07:50:18.307 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 164.963376719, 2024-09-18T07:50:18.307 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93909738700000001, 2024-09-18T07:50:18.307 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.307 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.307 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.307 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.307 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.307 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.307 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30928 2024-09-18T07:50:18.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.308 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.308 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.308 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.308 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.268666+0000", 2024-09-18T07:50:18.308 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.308 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.308 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.308 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.268666+0000", 2024-09-18T07:50:18.308 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.309 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.309 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.309 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.268668+0000", 2024-09-18T07:50:18.309 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3989999999999998e-06 2024-09-18T07:50:18.309 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.309 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.309 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.309 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.268722+0000", 2024-09-18T07:50:18.309 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4008000000000002e-05 2024-09-18T07:50:18.309 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.309 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.309 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.310 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.268733+0000", 2024-09-18T07:50:18.310 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1391e-05 2024-09-18T07:50:18.310 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.310 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.310 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.310 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.268739+0000", 2024-09-18T07:50:18.310 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.609e-06 2024-09-18T07:50:18.310 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.310 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.310 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.310 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.268799+0000", 2024-09-18T07:50:18.310 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.9793000000000002e-05 2024-09-18T07:50:18.310 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.311 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.311 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.311 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.268806+0000", 2024-09-18T07:50:18.311 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.1980000000000004e-06 2024-09-18T07:50:18.311 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.311 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.311 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.311 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196787+0000", 2024-09-18T07:50:18.311 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.927980574 2024-09-18T07:50:18.311 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.311 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.311 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.311 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.196879+0000", 2024-09-18T07:50:18.311 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2306999999999996e-05 2024-09-18T07:50:18.312 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.312 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.312 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.312 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.197001+0000", 2024-09-18T07:50:18.312 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012252199999999999 2024-09-18T07:50:18.312 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.312 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.312 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.312 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.202913+0000", 2024-09-18T07:50:18.312 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0059114700000000003 2024-09-18T07:50:18.312 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.312 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.312 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.313 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.202960+0000", 2024-09-18T07:50:18.313 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.6758000000000002e-05 2024-09-18T07:50:18.313 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.313 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.313 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.313 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207626+0000", 2024-09-18T07:50:18.313 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0046666570000000003 2024-09-18T07:50:18.313 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.313 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.313 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.313 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207715+0000", 2024-09-18T07:50:18.313 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.8434000000000001e-05 2024-09-18T07:50:18.313 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.314 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.314 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.314 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207763+0000", 2024-09-18T07:50:18.314 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8266999999999997e-05 2024-09-18T07:50:18.314 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.314 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.314 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.314 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.314 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.314 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30932 11.7 11:ee7a1da2:::benchmark_data_smithi137_181069_object30931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:50:18.314 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.270974+0000", 2024-09-18T07:50:18.314 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 164.96106825300001, 2024-09-18T07:50:18.314 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93698104800000004, 2024-09-18T07:50:18.315 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.315 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.315 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.315 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.315 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.315 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.315 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30932 2024-09-18T07:50:18.315 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.315 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.315 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.315 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.315 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.270974+0000", 2024-09-18T07:50:18.316 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.316 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.316 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.316 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.316 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.270974+0000", 2024-09-18T07:50:18.316 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.316 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.316 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.316 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.316 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.270976+0000", 2024-09-18T07:50:18.316 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.9750000000000001e-06 2024-09-18T07:50:18.316 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.316 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.317 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.317 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271032+0000", 2024-09-18T07:50:18.317 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.5878000000000003e-05 2024-09-18T07:50:18.317 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.317 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.317 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.317 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271044+0000", 2024-09-18T07:50:18.317 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2408e-05 2024-09-18T07:50:18.317 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.317 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.317 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.317 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271052+0000", 2024-09-18T07:50:18.318 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6520000000000006e-06 2024-09-18T07:50:18.318 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.318 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.318 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.318 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271128+0000", 2024-09-18T07:50:18.318 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.6322000000000006e-05 2024-09-18T07:50:18.318 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.318 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.318 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.318 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271135+0000", 2024-09-18T07:50:18.318 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5919999999999997e-06 2024-09-18T07:50:18.318 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.319 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.319 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.319 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.198221+0000", 2024-09-18T07:50:18.319 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.92708616799999999 2024-09-18T07:50:18.319 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.319 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.319 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.319 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.198326+0000", 2024-09-18T07:50:18.319 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00010516799999999999 2024-09-18T07:50:18.319 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.319 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.319 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.319 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.198447+0000", 2024-09-18T07:50:18.320 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012055799999999999 2024-09-18T07:50:18.320 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.320 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.320 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.320 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.202996+0000", 2024-09-18T07:50:18.320 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0045494050000000003 2024-09-18T07:50:18.320 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.320 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.320 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.320 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.203032+0000", 2024-09-18T07:50:18.320 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5299000000000002e-05 2024-09-18T07:50:18.320 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.321 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.321 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.321 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207882+0000", 2024-09-18T07:50:18.321 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0048507250000000002 2024-09-18T07:50:18.321 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.321 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.321 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.321 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207923+0000", 2024-09-18T07:50:18.321 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.0599999999999998e-05 2024-09-18T07:50:18.321 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.321 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.321 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.321 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207955+0000", 2024-09-18T07:50:18.322 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2298000000000001e-05 2024-09-18T07:50:18.322 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.322 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.322 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.322 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.322 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.322 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30933 11.7 11:e9faa571:::benchmark_data_smithi137_181069_object30932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:50:18.322 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.271203+0000", 2024-09-18T07:50:18.322 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 164.96083952500001, 2024-09-18T07:50:18.322 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.93684991900000003, 2024-09-18T07:50:18.322 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.322 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.322 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.322 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.323 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.323 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.323 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30933 2024-09-18T07:50:18.323 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.323 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.323 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.323 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.323 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271203+0000", 2024-09-18T07:50:18.323 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.323 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.323 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.323 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.323 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271203+0000", 2024-09-18T07:50:18.323 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.324 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.324 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.324 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.324 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271204+0000", 2024-09-18T07:50:18.324 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5999999999999999e-06 2024-09-18T07:50:18.324 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.324 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.324 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.324 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271254+0000", 2024-09-18T07:50:18.324 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9212999999999998e-05 2024-09-18T07:50:18.324 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.324 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.324 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.325 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271266+0000", 2024-09-18T07:50:18.325 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2571999999999999e-05 2024-09-18T07:50:18.325 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.325 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.325 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.325 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271274+0000", 2024-09-18T07:50:18.325 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5120000000000002e-06 2024-09-18T07:50:18.325 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.325 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.325 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.325 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271332+0000", 2024-09-18T07:50:18.325 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.8199000000000003e-05 2024-09-18T07:50:18.325 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.326 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.326 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.326 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.271342+0000", 2024-09-18T07:50:18.326 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7219999999999994e-06 2024-09-18T07:50:18.326 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.326 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.326 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.326 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.199551+0000", 2024-09-18T07:50:18.326 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.928209536 2024-09-18T07:50:18.326 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.326 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.326 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.327 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.199673+0000", 2024-09-18T07:50:18.327 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012221899999999999 2024-09-18T07:50:18.327 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.327 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.327 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.327 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.199806+0000", 2024-09-18T07:50:18.327 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000132328 2024-09-18T07:50:18.327 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.327 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.327 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.327 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.203066+0000", 2024-09-18T07:50:18.327 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003259942 2024-09-18T07:50:18.327 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.328 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.328 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.328 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.205648+0000", 2024-09-18T07:50:18.328 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0025826790000000001 2024-09-18T07:50:18.328 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.328 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.328 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.328 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207965+0000", 2024-09-18T07:50:18.328 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0023169990000000001 2024-09-18T07:50:18.328 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.328 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.328 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.328 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208011+0000", 2024-09-18T07:50:18.329 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5568000000000001e-05 2024-09-18T07:50:18.329 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.329 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.329 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.329 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208053+0000", 2024-09-18T07:50:18.329 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.1829999999999998e-05 2024-09-18T07:50:18.329 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.329 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.329 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.329 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.329 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.329 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30958 11.7 11:eebed827:::benchmark_data_smithi137_181069_object30957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:50:18.330 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.295207+0000", 2024-09-18T07:50:18.330 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 164.936835218, 2024-09-18T07:50:18.330 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.91294648099999998, 2024-09-18T07:50:18.330 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.330 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.330 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.330 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.330 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:18 smithi203 ceph-mon[32237]: 4.3 scrub starts 2024-09-18T07:50:18.331 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:18 smithi203 ceph-mon[32237]: 4.3 scrub ok 2024-09-18T07:50:18.331 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:18 smithi203 ceph-mon[32237]: 3.1 scrub starts 2024-09-18T07:50:18.331 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:18 smithi203 ceph-mon[32237]: 3.1 scrub ok 2024-09-18T07:50:18.331 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:18 smithi203 ceph-mon[32237]: 1.a scrub starts 2024-09-18T07:50:18.331 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:18 smithi203 ceph-mon[32237]: 1.a scrub ok 2024-09-18T07:50:18.331 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:18 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2589431954' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:50:18.331 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:18 smithi203 ceph-mon[32237]: from='client.36728 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:50:18.331 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:50:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:50:18.332 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.332 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.332 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30958 2024-09-18T07:50:18.332 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.332 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.332 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.333 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.333 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.295207+0000", 2024-09-18T07:50:18.333 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.333 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.333 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.333 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.333 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.295207+0000", 2024-09-18T07:50:18.333 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.333 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.333 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.333 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.333 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.295210+0000", 2024-09-18T07:50:18.333 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7999999999999999e-06 2024-09-18T07:50:18.334 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.334 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.334 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.334 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.295276+0000", 2024-09-18T07:50:18.334 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6266999999999994e-05 2024-09-18T07:50:18.334 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.334 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.334 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.334 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.295294+0000", 2024-09-18T07:50:18.334 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.7739000000000001e-05 2024-09-18T07:50:18.334 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.334 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.334 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.335 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.295301+0000", 2024-09-18T07:50:18.335 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2830000000000004e-06 2024-09-18T07:50:18.335 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.335 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.335 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.335 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.295363+0000", 2024-09-18T07:50:18.335 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2084999999999999e-05 2024-09-18T07:50:18.335 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.335 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.335 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.335 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.295371+0000", 2024-09-18T07:50:18.335 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.2590000000000003e-06 2024-09-18T07:50:18.336 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.336 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.336 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.336 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.201681+0000", 2024-09-18T07:50:18.336 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90630996799999997 2024-09-18T07:50:18.336 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.336 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.336 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.336 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.201805+0000", 2024-09-18T07:50:18.336 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000124253 2024-09-18T07:50:18.336 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.336 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.336 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.337 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.201961+0000", 2024-09-18T07:50:18.337 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00015598000000000001 2024-09-18T07:50:18.337 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.337 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.337 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.337 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.205685+0000", 2024-09-18T07:50:18.337 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.003724163 2024-09-18T07:50:18.337 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.337 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.337 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.337 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.205718+0000", 2024-09-18T07:50:18.337 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.2991000000000003e-05 2024-09-18T07:50:18.337 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.337 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.338 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.338 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208061+0000", 2024-09-18T07:50:18.338 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002342645 2024-09-18T07:50:18.338 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.338 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.338 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.338 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208100+0000", 2024-09-18T07:50:18.338 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9376999999999999e-05 2024-09-18T07:50:18.338 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.338 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.338 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.338 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208154+0000", 2024-09-18T07:50:18.338 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3671000000000002e-05 2024-09-18T07:50:18.339 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.339 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.339 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.339 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.339 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.339 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30969 11.7 11:ef647aa1:::benchmark_data_smithi137_181069_object30968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:50:18.339 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.302772+0000", 2024-09-18T07:50:18.339 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 164.92927074299999, 2024-09-18T07:50:18.339 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90547923200000002, 2024-09-18T07:50:18.339 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.339 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.339 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.339 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.340 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.340 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.340 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30969 2024-09-18T07:50:18.340 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.340 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.340 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.340 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.340 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.302772+0000", 2024-09-18T07:50:18.340 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.340 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.340 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.340 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.340 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.302772+0000", 2024-09-18T07:50:18.340 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.341 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.341 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.341 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.341 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.302774+0000", 2024-09-18T07:50:18.341 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8810000000000001e-06 2024-09-18T07:50:18.341 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.341 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.341 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.341 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.302847+0000", 2024-09-18T07:50:18.341 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3746999999999997e-05 2024-09-18T07:50:18.341 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.341 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.341 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.342 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.302859+0000", 2024-09-18T07:50:18.342 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.1977000000000001e-05 2024-09-18T07:50:18.342 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.342 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.342 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.342 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.302869+0000", 2024-09-18T07:50:18.342 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2909999999999996e-06 2024-09-18T07:50:18.342 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.342 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.342 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.342 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.302926+0000", 2024-09-18T07:50:18.342 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7362999999999999e-05 2024-09-18T07:50:18.342 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.343 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.343 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.343 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.302933+0000", 2024-09-18T07:50:18.343 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.7209999999999997e-06 2024-09-18T07:50:18.343 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.343 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.343 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.343 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.203094+0000", 2024-09-18T07:50:18.343 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.90016099900000002 2024-09-18T07:50:18.343 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.343 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.343 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.343 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.203200+0000", 2024-09-18T07:50:18.343 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000106183 2024-09-18T07:50:18.344 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.344 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.344 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.344 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.203325+0000", 2024-09-18T07:50:18.344 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012539300000000001 2024-09-18T07:50:18.344 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.344 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.344 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.344 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.205751+0000", 2024-09-18T07:50:18.344 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0024260729999999999 2024-09-18T07:50:18.344 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.344 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.344 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.344 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207814+0000", 2024-09-18T07:50:18.345 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0020626189999999999 2024-09-18T07:50:18.345 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.345 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.345 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.345 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208163+0000", 2024-09-18T07:50:18.345 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00034904800000000002 2024-09-18T07:50:18.345 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.345 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.345 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.345 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208215+0000", 2024-09-18T07:50:18.345 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.2516000000000003e-05 2024-09-18T07:50:18.345 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.345 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.346 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.346 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208251+0000", 2024-09-18T07:50:18.346 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5420999999999998e-05 2024-09-18T07:50:18.346 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.346 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.346 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.346 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.346 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.346 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:30981 11.7 11:e0ad3924:::benchmark_data_smithi137_181069_object30980:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:50:18.346 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.318119+0000", 2024-09-18T07:50:18.346 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 164.91392364399999, 2024-09-18T07:50:18.346 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.89035505999999998, 2024-09-18T07:50:18.346 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.347 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.347 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.347 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.347 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.347 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.347 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 30981 2024-09-18T07:50:18.347 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.347 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.347 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.347 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.347 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.318119+0000", 2024-09-18T07:50:18.347 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.347 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.348 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.348 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.348 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.318119+0000", 2024-09-18T07:50:18.348 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.348 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.348 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.348 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.348 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.318121+0000", 2024-09-18T07:50:18.348 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.3300000000000001e-06 2024-09-18T07:50:18.348 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.348 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.348 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.348 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.318183+0000", 2024-09-18T07:50:18.349 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2277e-05 2024-09-18T07:50:18.349 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.349 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.349 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.349 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.318199+0000", 2024-09-18T07:50:18.349 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5617e-05 2024-09-18T07:50:18.349 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.349 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.349 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.349 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.318207+0000", 2024-09-18T07:50:18.349 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2239999999999995e-06 2024-09-18T07:50:18.349 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.349 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.350 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.350 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.318269+0000", 2024-09-18T07:50:18.350 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.2193999999999994e-05 2024-09-18T07:50:18.350 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.350 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.350 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.350 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.318278+0000", 2024-09-18T07:50:18.350 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.1920000000000005e-06 2024-09-18T07:50:18.350 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.350 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.350 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.350 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.204376+0000", 2024-09-18T07:50:18.350 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.88609823200000004 2024-09-18T07:50:18.350 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.351 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.351 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.351 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.204511+0000", 2024-09-18T07:50:18.351 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013500400000000001 2024-09-18T07:50:18.351 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.351 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.351 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.351 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.204639+0000", 2024-09-18T07:50:18.351 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012823399999999999 2024-09-18T07:50:18.351 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.351 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.351 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.351 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.207867+0000", 2024-09-18T07:50:18.352 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00322829 2024-09-18T07:50:18.352 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.352 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.352 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.352 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208259+0000", 2024-09-18T07:50:18.352 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00039183000000000002 2024-09-18T07:50:18.352 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.352 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.352 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.352 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208372+0000", 2024-09-18T07:50:18.352 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000113037 2024-09-18T07:50:18.352 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.352 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.353 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.353 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208428+0000", 2024-09-18T07:50:18.353 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6072000000000001e-05 2024-09-18T07:50:18.353 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.353 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.353 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.353 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208474+0000", 2024-09-18T07:50:18.353 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.5527e-05 2024-09-18T07:50:18.353 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.353 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.353 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.353 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.353 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.354 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:31065 11.7 11:ef02e3d5:::benchmark_data_smithi137_181069_object31064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:50:18.354 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.402987+0000", 2024-09-18T07:50:18.354 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 164.829055716, 2024-09-18T07:50:18.354 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.80566682499999998, 2024-09-18T07:50:18.354 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.354 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.354 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.354 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.354 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.354 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.354 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 31065 2024-09-18T07:50:18.354 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.354 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.355 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.355 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.355 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.402987+0000", 2024-09-18T07:50:18.355 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.355 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.355 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.355 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.355 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.402987+0000", 2024-09-18T07:50:18.355 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.355 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.355 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.355 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.355 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.402989+0000", 2024-09-18T07:50:18.356 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.7089999999999998e-06 2024-09-18T07:50:18.356 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.356 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.356 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.356 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.403044+0000", 2024-09-18T07:50:18.356 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4138999999999998e-05 2024-09-18T07:50:18.356 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.356 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.356 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.356 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.403060+0000", 2024-09-18T07:50:18.356 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.6189e-05 2024-09-18T07:50:18.356 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.356 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.357 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.357 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.403069+0000", 2024-09-18T07:50:18.357 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7589999999999992e-06 2024-09-18T07:50:18.357 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.357 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.357 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.357 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.403136+0000", 2024-09-18T07:50:18.357 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.6792999999999996e-05 2024-09-18T07:50:18.357 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.357 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.357 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.357 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.403144+0000", 2024-09-18T07:50:18.357 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.2330000000000006e-06 2024-09-18T07:50:18.358 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.358 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.358 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.358 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.205783+0000", 2024-09-18T07:50:18.358 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.80263816499999996 2024-09-18T07:50:18.358 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.358 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.358 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.358 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.205876+0000", 2024-09-18T07:50:18.358 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.3547000000000004e-05 2024-09-18T07:50:18.358 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.358 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.359 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.359 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.206004+0000", 2024-09-18T07:50:18.359 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00012819099999999999 2024-09-18T07:50:18.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.359 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.359 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.359 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208264+0000", 2024-09-18T07:50:18.359 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0022599540000000002 2024-09-18T07:50:18.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.359 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.359 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.359 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208523+0000", 2024-09-18T07:50:18.359 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00025910100000000003 2024-09-18T07:50:18.359 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.360 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.360 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.360 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208563+0000", 2024-09-18T07:50:18.360 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9569e-05 2024-09-18T07:50:18.360 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.360 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.360 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.360 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208618+0000", 2024-09-18T07:50:18.360 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4991000000000003e-05 2024-09-18T07:50:18.360 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.360 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.360 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.360 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208654+0000", 2024-09-18T07:50:18.360 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5485000000000003e-05 2024-09-18T07:50:18.361 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.361 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.361 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.361 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.361 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.361 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:31081 11.7 11:eb7b016c:::benchmark_data_smithi137_181069_object31080:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:50:18.361 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.429960+0000", 2024-09-18T07:50:18.361 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 164.80208286300001, 2024-09-18T07:50:18.361 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.77885249999999995, 2024-09-18T07:50:18.361 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.361 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.361 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.361 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.361 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.362 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.362 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 31081 2024-09-18T07:50:18.362 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.362 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.362 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.362 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.362 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.429960+0000", 2024-09-18T07:50:18.362 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.362 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.362 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.362 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.362 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.429960+0000", 2024-09-18T07:50:18.362 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.362 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.363 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.363 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.363 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.429962+0000", 2024-09-18T07:50:18.363 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.8030000000000001e-06 2024-09-18T07:50:18.363 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.363 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.363 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.363 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.429998+0000", 2024-09-18T07:50:18.363 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.5843999999999998e-05 2024-09-18T07:50:18.363 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.363 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.363 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.363 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.430014+0000", 2024-09-18T07:50:18.363 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5931e-05 2024-09-18T07:50:18.364 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.364 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.364 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.364 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.430023+0000", 2024-09-18T07:50:18.364 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.2560000000000008e-06 2024-09-18T07:50:18.364 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.364 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.364 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.364 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.430099+0000", 2024-09-18T07:50:18.364 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.5411e-05 2024-09-18T07:50:18.364 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.364 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.364 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.365 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.430111+0000", 2024-09-18T07:50:18.365 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.2136e-05 2024-09-18T07:50:18.365 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.365 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.365 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.365 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.206697+0000", 2024-09-18T07:50:18.365 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.77658628200000002 2024-09-18T07:50:18.365 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.365 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.365 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.365 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.206788+0000", 2024-09-18T07:50:18.365 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.0576999999999997e-05 2024-09-18T07:50:18.365 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.366 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.366 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.366 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.206898+0000", 2024-09-18T07:50:18.366 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000110671 2024-09-18T07:50:18.366 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.366 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.366 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.366 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208270+0000", 2024-09-18T07:50:18.366 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0013715209999999999 2024-09-18T07:50:18.366 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.366 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.366 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.366 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208697+0000", 2024-09-18T07:50:18.366 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00042693199999999998 2024-09-18T07:50:18.366 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.367 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.367 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.367 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208736+0000", 2024-09-18T07:50:18.367 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.9561000000000003e-05 2024-09-18T07:50:18.367 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.367 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.367 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.367 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208781+0000", 2024-09-18T07:50:18.367 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4603e-05 2024-09-18T07:50:18.367 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.367 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.367 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.367 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208812+0000", 2024-09-18T07:50:18.368 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 3.0972000000000001e-05 2024-09-18T07:50:18.368 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.368 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.368 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.368 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.368 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.368 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:31123 11.7 11:eff82b5a:::benchmark_data_smithi137_181069_object31122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-09-18T07:50:18.368 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:47:33.538919+0000", 2024-09-18T07:50:18.368 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 164.693123214, 2024-09-18T07:50:18.368 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.67483459700000004, 2024-09-18T07:50:18.368 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.368 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.368 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.369 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.369 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.369 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.369 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 31123 2024-09-18T07:50:18.369 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.369 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.369 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.369 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.369 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.538919+0000", 2024-09-18T07:50:18.369 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.369 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.369 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.369 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.369 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.538919+0000", 2024-09-18T07:50:18.370 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.370 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.370 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.370 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.370 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.538921+0000", 2024-09-18T07:50:18.370 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.8109999999999999e-06 2024-09-18T07:50:18.370 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.370 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.370 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.370 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.538978+0000", 2024-09-18T07:50:18.370 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.6709999999999997e-05 2024-09-18T07:50:18.370 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.370 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.370 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.371 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.538988+0000", 2024-09-18T07:50:18.371 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.0482000000000001e-05 2024-09-18T07:50:18.371 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.371 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.371 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.371 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.538994+0000", 2024-09-18T07:50:18.371 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.7649999999999999e-06 2024-09-18T07:50:18.371 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.371 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.371 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.371 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.539042+0000", 2024-09-18T07:50:18.371 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.8461000000000002e-05 2024-09-18T07:50:18.371 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.371 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.371 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for peered", 2024-09-18T07:50:18.372 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:33.539049+0000", 2024-09-18T07:50:18.372 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.5010000000000001e-06 2024-09-18T07:50:18.372 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.372 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.372 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.372 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.208929+0000", 2024-09-18T07:50:18.372 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.66988051199999998 2024-09-18T07:50:18.372 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.372 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.372 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.372 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.209047+0000", 2024-09-18T07:50:18.372 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00011739399999999999 2024-09-18T07:50:18.372 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.373 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.373 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.373 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.209164+0000", 2024-09-18T07:50:18.373 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000117126 2024-09-18T07:50:18.373 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.373 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.373 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.373 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.211312+0000", 2024-09-18T07:50:18.373 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0021483320000000002 2024-09-18T07:50:18.373 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.373 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.373 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.373 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.212733+0000", 2024-09-18T07:50:18.374 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0014205509999999999 2024-09-18T07:50:18.374 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.374 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.374 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.374 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.213640+0000", 2024-09-18T07:50:18.374 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00090767599999999995 2024-09-18T07:50:18.374 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.374 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.374 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.374 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.213701+0000", 2024-09-18T07:50:18.374 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.0096999999999997e-05 2024-09-18T07:50:18.374 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.374 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.375 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.375 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:47:34.213754+0000", 2024-09-18T07:50:18.375 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.3179e-05 2024-09-18T07:50:18.375 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.375 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.375 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.375 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.375 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.375 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:123524 11.7 11:eb5f8cff:::benchmark_data_smithi137_181069_object11370:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e505)", 2024-09-18T07:50:18.375 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:48:56.199735+0000", 2024-09-18T07:50:18.375 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 82.032307724999995, 2024-09-18T07:50:18.375 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.27179915999999998, 2024-09-18T07:50:18.375 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.375 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.376 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.376 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.376 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.376 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.376 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 123524 2024-09-18T07:50:18.376 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.376 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.376 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.376 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.376 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.199735+0000", 2024-09-18T07:50:18.376 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.376 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.376 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.376 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.377 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.199735+0000", 2024-09-18T07:50:18.377 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.377 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.377 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.377 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.377 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.199737+0000", 2024-09-18T07:50:18.377 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.362e-06 2024-09-18T07:50:18.377 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.377 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.377 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.377 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.199743+0000", 2024-09-18T07:50:18.377 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.4069999999999998e-06 2024-09-18T07:50:18.377 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.377 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.378 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.378 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.199761+0000", 2024-09-18T07:50:18.378 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.747e-05 2024-09-18T07:50:18.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.378 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.378 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.378 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.199771+0000", 2024-09-18T07:50:18.378 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.7620000000000007e-06 2024-09-18T07:50:18.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.378 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.378 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.378 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.199844+0000", 2024-09-18T07:50:18.378 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 7.3310000000000001e-05 2024-09-18T07:50:18.378 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.379 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.379 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.379 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.199944+0000", 2024-09-18T07:50:18.379 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 9.9967e-05 2024-09-18T07:50:18.379 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.379 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.379 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.379 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.200027+0000", 2024-09-18T07:50:18.379 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 8.3154999999999999e-05 2024-09-18T07:50:18.379 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.379 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.379 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.379 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.202225+0000", 2024-09-18T07:50:18.379 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.002198003 2024-09-18T07:50:18.379 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.379 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.380 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.380 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.202615+0000", 2024-09-18T07:50:18.380 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00039025799999999999 2024-09-18T07:50:18.380 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.380 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.380 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.380 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.471382+0000", 2024-09-18T07:50:18.380 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.26876626100000001 2024-09-18T07:50:18.380 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.380 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.380 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.380 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.471484+0000", 2024-09-18T07:50:18.380 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000102378 2024-09-18T07:50:18.381 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.381 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.381 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.381 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.471534+0000", 2024-09-18T07:50:18.381 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.9827000000000003e-05 2024-09-18T07:50:18.381 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.381 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.381 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.381 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.381 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.381 INFO:teuthology.orchestra.run.smithi203.stdout: "description": "osd_op(client.16581.0:123530 11.7 11:e5df1a10:::benchmark_data_smithi137_181069_object11376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e505)", 2024-09-18T07:50:18.381 INFO:teuthology.orchestra.run.smithi203.stdout: "initiated_at": "2024-09-18T07:48:56.207014+0000", 2024-09-18T07:50:18.381 INFO:teuthology.orchestra.run.smithi203.stdout: "age": 82.025028828999993, 2024-09-18T07:50:18.381 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.271573656, 2024-09-18T07:50:18.382 INFO:teuthology.orchestra.run.smithi203.stdout: "continuous": false, 2024-09-18T07:50:18.382 INFO:teuthology.orchestra.run.smithi203.stdout: "type_data": { 2024-09-18T07:50:18.382 INFO:teuthology.orchestra.run.smithi203.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:18.382 INFO:teuthology.orchestra.run.smithi203.stdout: "client_info": { 2024-09-18T07:50:18.382 INFO:teuthology.orchestra.run.smithi203.stdout: "client": "client.16581", 2024-09-18T07:50:18.382 INFO:teuthology.orchestra.run.smithi203.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:18.382 INFO:teuthology.orchestra.run.smithi203.stdout: "tid": 123530 2024-09-18T07:50:18.382 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.382 INFO:teuthology.orchestra.run.smithi203.stdout: "events": [ 2024-09-18T07:50:18.382 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.382 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "initiated", 2024-09-18T07:50:18.382 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.207014+0000", 2024-09-18T07:50:18.382 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.383 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.383 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.383 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "header_read", 2024-09-18T07:50:18.383 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.207014+0000", 2024-09-18T07:50:18.383 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0 2024-09-18T07:50:18.383 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.383 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.383 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "throttled", 2024-09-18T07:50:18.383 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.207016+0000", 2024-09-18T07:50:18.383 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 2.2689999999999998e-06 2024-09-18T07:50:18.383 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.383 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.383 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "all_read", 2024-09-18T07:50:18.384 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.207023+0000", 2024-09-18T07:50:18.384 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.866e-06 2024-09-18T07:50:18.384 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.384 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.384 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "dispatched", 2024-09-18T07:50:18.384 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.207038+0000", 2024-09-18T07:50:18.384 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 1.5242000000000001e-05 2024-09-18T07:50:18.384 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.384 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.384 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "queued_for_pg", 2024-09-18T07:50:18.384 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.207045+0000", 2024-09-18T07:50:18.384 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.613e-06 2024-09-18T07:50:18.384 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.385 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.385 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "reached_pg", 2024-09-18T07:50:18.385 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.207099+0000", 2024-09-18T07:50:18.385 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 5.4323000000000001e-05 2024-09-18T07:50:18.385 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.385 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.385 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "started", 2024-09-18T07:50:18.385 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.207215+0000", 2024-09-18T07:50:18.385 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.000116179 2024-09-18T07:50:18.385 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.385 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.385 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:18.385 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.207349+0000", 2024-09-18T07:50:18.386 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00013351899999999999 2024-09-18T07:50:18.386 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.386 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.386 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "op_commit", 2024-09-18T07:50:18.386 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.210727+0000", 2024-09-18T07:50:18.386 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.0033783799999999998 2024-09-18T07:50:18.386 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.386 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.386 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.386 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.211280+0000", 2024-09-18T07:50:18.386 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.00055288199999999998 2024-09-18T07:50:18.386 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.387 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.387 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:18.387 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.478481+0000", 2024-09-18T07:50:18.387 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 0.26720107399999998 2024-09-18T07:50:18.387 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.387 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.387 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "commit_sent", 2024-09-18T07:50:18.387 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.478542+0000", 2024-09-18T07:50:18.387 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 6.1470999999999995e-05 2024-09-18T07:50:18.387 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-09-18T07:50:18.387 INFO:teuthology.orchestra.run.smithi203.stdout: { 2024-09-18T07:50:18.387 INFO:teuthology.orchestra.run.smithi203.stdout: "event": "done", 2024-09-18T07:50:18.387 INFO:teuthology.orchestra.run.smithi203.stdout: "time": "2024-09-18T07:48:56.478587+0000", 2024-09-18T07:50:18.388 INFO:teuthology.orchestra.run.smithi203.stdout: "duration": 4.4838e-05 2024-09-18T07:50:18.388 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.388 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.388 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.388 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-09-18T07:50:18.388 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-09-18T07:50:18.388 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-09-18T07:50:18.388 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:18.409 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:50:18.429 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:18.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:18.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:18.531 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:18.567 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:18.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:18.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:18.643 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:18 smithi137 ceph-mon[29408]: 4.3 scrub starts 2024-09-18T07:50:18.643 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:18 smithi137 ceph-mon[29408]: 4.3 scrub ok 2024-09-18T07:50:18.643 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:18 smithi137 ceph-mon[29408]: 3.1 scrub starts 2024-09-18T07:50:18.643 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:18 smithi137 ceph-mon[29408]: 3.1 scrub ok 2024-09-18T07:50:18.643 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:18 smithi137 ceph-mon[29408]: 1.a scrub starts 2024-09-18T07:50:18.644 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:18 smithi137 ceph-mon[29408]: 1.a scrub ok 2024-09-18T07:50:18.644 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:18 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2589431954' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:50:18.644 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:18 smithi137 ceph-mon[29408]: from='client.36728 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:50:18.644 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:18 smithi137 ceph-mon[39175]: 4.3 scrub starts 2024-09-18T07:50:18.644 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:18 smithi137 ceph-mon[39175]: 4.3 scrub ok 2024-09-18T07:50:18.644 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:18 smithi137 ceph-mon[39175]: 3.1 scrub starts 2024-09-18T07:50:18.644 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:18 smithi137 ceph-mon[39175]: 3.1 scrub ok 2024-09-18T07:50:18.644 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:18 smithi137 ceph-mon[39175]: 1.a scrub starts 2024-09-18T07:50:18.644 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:18 smithi137 ceph-mon[39175]: 1.a scrub ok 2024-09-18T07:50:18.644 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:18 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2589431954' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:50:18.645 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:18 smithi137 ceph-mon[39175]: from='client.36728 ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:50:18.669 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:18.706 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:18.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:18.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:18.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:18.834 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:18.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:18.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:18.936 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:18.979 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:18.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:18.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:19.081 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:19.104 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 5, 7, 6, 1] out_osds: [3, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 6, 7, 4, 5] 2024-09-18T07:50:19.105 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:50:19.105 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-09-18T07:50:19.105 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd in 3 2024-09-18T07:50:19.116 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:19.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:19.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:19.142 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:19.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:19.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:19.262 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:19.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:19.363 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:19.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:19.393 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:19.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:19.452 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-18T07:50:19.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:19.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:19.519 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:19.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:19.604 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:19 smithi137 ceph-mon[39175]: 5.f scrub starts 2024-09-18T07:50:19.604 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:19 smithi137 ceph-mon[39175]: 5.f scrub ok 2024-09-18T07:50:19.604 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:19 smithi137 ceph-mon[39175]: pgmap v1036: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.2 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 5.0 KiB/s rd, 25 KiB/s wr, 1.43k op/s; 7475/60342 objects misplaced (12.388%); 13 MiB/s, 204 objects/s recovering 2024-09-18T07:50:19.604 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:19 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:19.605 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:19 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:50:19.605 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:19 smithi137 ceph-mon[39175]: from='client.36728 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:50:19.605 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:19 smithi137 ceph-mon[39175]: osdmap e541: 8 total, 8 up, 6 in 2024-09-18T07:50:19.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:19 smithi137 ceph-mon[29408]: 5.f scrub starts 2024-09-18T07:50:19.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:19 smithi137 ceph-mon[29408]: 5.f scrub ok 2024-09-18T07:50:19.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:19 smithi137 ceph-mon[29408]: pgmap v1036: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.2 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 5.0 KiB/s rd, 25 KiB/s wr, 1.43k op/s; 7475/60342 objects misplaced (12.388%); 13 MiB/s, 204 objects/s recovering 2024-09-18T07:50:19.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:19 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:19.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:19 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:50:19.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:19 smithi137 ceph-mon[29408]: from='client.36728 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:50:19.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:19 smithi137 ceph-mon[29408]: osdmap e541: 8 total, 8 up, 6 in 2024-09-18T07:50:19.620 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:19.641 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:19.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:19.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:19 smithi203 ceph-mon[32237]: 5.f scrub starts 2024-09-18T07:50:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:19 smithi203 ceph-mon[32237]: 5.f scrub ok 2024-09-18T07:50:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:19 smithi203 ceph-mon[32237]: pgmap v1036: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.2 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 5.0 KiB/s rd, 25 KiB/s wr, 1.43k op/s; 7475/60342 objects misplaced (12.388%); 13 MiB/s, 204 objects/s recovering 2024-09-18T07:50:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:19 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:19 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:50:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:19 smithi203 ceph-mon[32237]: from='client.36728 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:50:19.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:19 smithi203 ceph-mon[32237]: osdmap e541: 8 total, 8 up, 6 in 2024-09-18T07:50:19.720 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:50:19.720 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:50:19.721 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:50:19.722 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:50:19.740 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:50:19.740 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:50:19.743 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:19.758 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:50:19.758 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:50:19.765 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:19.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:19.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:19.775 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:50:19.775 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:50:19.793 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:50:19.793 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:50:19.809 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:50:19.809 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:50:19.826 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:50:19.827 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:50:19.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:19.892 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:19.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:19.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:19.994 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:20.016 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:20.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:20.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:20.094 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:20.094 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:20.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:20.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:20.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:20.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:20.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:20.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:20.279 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:20.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:20.381 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:20.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:20.403 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:20.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:20.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:20.530 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:20 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:50:20.530 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:20 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:50:20.530 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:20 smithi137 ceph-mon[29408]: osdmap e542: 8 total, 8 up, 6 in 2024-09-18T07:50:20.531 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:20 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3252551671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:20.531 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:20 smithi137 ceph-mon[29408]: osdmap e543: 8 total, 8 up, 6 in 2024-09-18T07:50:20.531 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:20 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:50:20.531 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:20 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:50:20.531 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:20 smithi137 ceph-mon[39175]: osdmap e542: 8 total, 8 up, 6 in 2024-09-18T07:50:20.531 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:20 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3252551671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:20.531 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:20 smithi137 ceph-mon[39175]: osdmap e543: 8 total, 8 up, 6 in 2024-09-18T07:50:20.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:20.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:20.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:20.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:20.674 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:20.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:20.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:20.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:20 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:50:20.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:20 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:50:20.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:20 smithi203 ceph-mon[32237]: osdmap e542: 8 total, 8 up, 6 in 2024-09-18T07:50:20.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:20 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3252551671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:20.713 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:20 smithi203 ceph-mon[32237]: osdmap e543: 8 total, 8 up, 6 in 2024-09-18T07:50:20.776 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:20.813 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:20.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:20.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:20.915 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:20.951 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:20.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:20.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:21.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-18T07:50:21.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:21.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:21.111 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:21.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:21.212 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:21.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:21.241 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:21.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:21.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:21.380 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:21.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:21.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:21 smithi137 ceph-mon[29408]: pgmap v1039: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.2 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 4.4 KiB/s rd, 26 KiB/s wr, 1.04k op/s; 7475/59301 objects misplaced (12.605%); 8.4 MiB/s, 133 objects/s recovering 2024-09-18T07:50:21.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:21 smithi137 ceph-mon[39175]: pgmap v1039: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.2 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 4.4 KiB/s rd, 26 KiB/s wr, 1.04k op/s; 7475/59301 objects misplaced (12.605%); 8.4 MiB/s, 133 objects/s recovering 2024-09-18T07:50:21.482 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:21.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:21.518 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:21.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:21.619 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:21.639 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:50:21.639 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:50:21.650 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:21.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:21.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:21.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:21 smithi203 ceph-mon[32237]: pgmap v1039: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.2 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 4.4 KiB/s rd, 26 KiB/s wr, 1.04k op/s; 7475/59301 objects misplaced (12.605%); 8.4 MiB/s, 133 objects/s recovering 2024-09-18T07:50:21.751 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:21.787 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:21.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:21.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:21.889 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:21.920 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:21.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:21.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:22.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:22.049 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:21 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a[29404]: 2024-09-18T07:50:21.718+0000 7fe5973a1640 -1 mon.a@0(leader).osd e543 definitely_dead 0 2024-09-18T07:50:22.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:22.055 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:22.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:22.156 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:22.193 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:22.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:22.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:22.267 INFO:teuthology.orchestra.run.smithi137.stderr:marked in osd.3. 2024-09-18T07:50:22.294 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:22.318 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:22.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:22.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:22.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:22 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/352907085' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-18T07:50:22.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:22 smithi137 ceph-mon[29408]: from='client.36740 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-18T07:50:22.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:22 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/352907085' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-18T07:50:22.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:22 smithi137 ceph-mon[39175]: from='client.36740 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-18T07:50:22.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:22.441 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:22.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:22.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:22.474 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-18T07:50:22.543 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:22.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:22.565 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:22.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:22.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:50:22.666 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:22.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:22.688 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:22.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:22 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/352907085' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-18T07:50:22.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:22 smithi203 ceph-mon[32237]: from='client.36740 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-18T07:50:22.722 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:22.789 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:22.811 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:22.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:22.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:22.913 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:22.934 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:22.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:22.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:23.036 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:23.057 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:23.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:23.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:23.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:23.181 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:23.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:23.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:23.283 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:23.319 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:23.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:23.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:23.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:23.449 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:23.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:23.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:23.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:23 smithi137 ceph-mon[29408]: pgmap v1041: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.1 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.26k op/s; 6479/53238 objects misplaced (12.170%); 7.9 MiB/s, 126 objects/s recovering 2024-09-18T07:50:23.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:23 smithi137 ceph-mon[29408]: from='client.36740 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-18T07:50:23.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:23 smithi137 ceph-mon[29408]: osdmap e544: 8 total, 8 up, 7 in 2024-09-18T07:50:23.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:23 smithi137 ceph-mon[29408]: 11.b starting backfill to osd.3 from (483'4175,511'9014] MIN to 543'13071 2024-09-18T07:50:23.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:23 smithi137 ceph-mon[29408]: osdmap e545: 8 total, 8 up, 7 in 2024-09-18T07:50:23.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:23 smithi137 ceph-mon[39175]: pgmap v1041: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.1 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.26k op/s; 6479/53238 objects misplaced (12.170%); 7.9 MiB/s, 126 objects/s recovering 2024-09-18T07:50:23.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:23 smithi137 ceph-mon[39175]: from='client.36740 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-18T07:50:23.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:23 smithi137 ceph-mon[39175]: osdmap e544: 8 total, 8 up, 7 in 2024-09-18T07:50:23.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:23 smithi137 ceph-mon[39175]: 11.b starting backfill to osd.3 from (483'4175,511'9014] MIN to 543'13071 2024-09-18T07:50:23.475 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:23 smithi137 ceph-mon[39175]: osdmap e545: 8 total, 8 up, 7 in 2024-09-18T07:50:23.510 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-09-18T07:50:23.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:23.573 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:23.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:23.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:23.697 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:23.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:23.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:23.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:23 smithi203 ceph-mon[32237]: pgmap v1041: 158 pgs: 1 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 151 active+clean; 1.1 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.26k op/s; 6479/53238 objects misplaced (12.170%); 7.9 MiB/s, 126 objects/s recovering 2024-09-18T07:50:23.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:23 smithi203 ceph-mon[32237]: from='client.36740 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-18T07:50:23.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:23 smithi203 ceph-mon[32237]: osdmap e544: 8 total, 8 up, 7 in 2024-09-18T07:50:23.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:23 smithi203 ceph-mon[32237]: 11.b starting backfill to osd.3 from (483'4175,511'9014] MIN to 543'13071 2024-09-18T07:50:23.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:23 smithi203 ceph-mon[32237]: osdmap e545: 8 total, 8 up, 7 in 2024-09-18T07:50:23.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:23.822 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:23.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:23.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:23.924 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:23.948 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:23.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:23.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:24.050 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:24.071 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:24.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:24.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:24.163 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:24.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:24.198 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:24.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:24.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:24.201 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:24.300 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:24.312 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:24 smithi137 ceph-mon[29408]: 11.2 starting backfill to osd.3 from (485'4441,521'9536] MIN to 543'12845 2024-09-18T07:50:24.312 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:24 smithi137 ceph-mon[29408]: 11.1 starting backfill to osd.3 from (493'5868,524'10047] MIN to 543'13243 2024-09-18T07:50:24.312 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:24 smithi137 ceph-mon[29408]: 11.a starting backfill to osd.3 from (486'4665,517'9238] MIN to 543'13032 2024-09-18T07:50:24.312 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:24 smithi137 ceph-mon[29408]: 11.9 starting backfill to osd.3 from (484'4275,511'8975] MIN to 543'13118 2024-09-18T07:50:24.313 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:24 smithi137 ceph-mon[29408]: osdmap e546: 8 total, 8 up, 7 in 2024-09-18T07:50:24.313 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:24 smithi137 ceph-mon[39175]: 11.2 starting backfill to osd.3 from (485'4441,521'9536] MIN to 543'12845 2024-09-18T07:50:24.313 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:24 smithi137 ceph-mon[39175]: 11.1 starting backfill to osd.3 from (493'5868,524'10047] MIN to 543'13243 2024-09-18T07:50:24.313 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:24 smithi137 ceph-mon[39175]: 11.a starting backfill to osd.3 from (486'4665,517'9238] MIN to 543'13032 2024-09-18T07:50:24.313 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:24 smithi137 ceph-mon[39175]: 11.9 starting backfill to osd.3 from (484'4275,511'8975] MIN to 543'13118 2024-09-18T07:50:24.313 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:24 smithi137 ceph-mon[39175]: osdmap e546: 8 total, 8 up, 7 in 2024-09-18T07:50:24.323 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:24.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:24.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:24.461 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:24.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:24.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:24.562 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:24.585 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:24.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:24.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:24.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:24.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:24 smithi203 ceph-mon[32237]: 11.2 starting backfill to osd.3 from (485'4441,521'9536] MIN to 543'12845 2024-09-18T07:50:24.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:24 smithi203 ceph-mon[32237]: 11.1 starting backfill to osd.3 from (493'5868,524'10047] MIN to 543'13243 2024-09-18T07:50:24.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:24 smithi203 ceph-mon[32237]: 11.a starting backfill to osd.3 from (486'4665,517'9238] MIN to 543'13032 2024-09-18T07:50:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:24 smithi203 ceph-mon[32237]: 11.9 starting backfill to osd.3 from (484'4275,511'8975] MIN to 543'13118 2024-09-18T07:50:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:24 smithi203 ceph-mon[32237]: osdmap e546: 8 total, 8 up, 7 in 2024-09-18T07:50:24.711 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:24.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:24.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:24.834 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:24.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:24.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:24.936 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:24.963 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:24.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:24.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:25.039 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:50:25.039 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:50:25.064 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:25.068 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:50:25.068 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:50:25.092 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:25.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:25.093 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:50:25.094 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:50:25.113 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:50:25.113 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:50:25.133 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:50:25.133 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:50:25.158 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:50:25.159 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:50:25.175 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-18T07:50:25.188 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:50:25.189 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:50:25.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:25.217 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:50:25.218 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:50:25.225 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:25.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:25.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:25.263 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:50:25.264 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-18T07:50:25.264 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-18T07:50:25.264 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-18T07:50:25.264 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:50:25.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:25.355 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:25.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:25.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[29408]: pgmap v1044: 158 pgs: 3 remapped+peering, 6 peering, 3 active+remapped+backfill_toofull, 146 active+clean; 910 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 1.15k op/s; 2748/44229 objects misplaced (6.213%); 1.9 MiB/s, 30 objects/s recovering 2024-09-18T07:50:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[29408]: 11.0 starting backfill to osd.3 from (494'5936,526'10317] MIN to 543'13400 2024-09-18T07:50:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[29408]: 11.8 starting backfill to osd.3 from (485'4534,514'9240] MIN to 543'13113 2024-09-18T07:50:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[29408]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:50:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/786139808' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:50:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3835754347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:25.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[29408]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:50:25.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/786139808' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:50:25.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[29408]: osdmap e547: 8 total, 8 up, 7 in 2024-09-18T07:50:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[39175]: pgmap v1044: 158 pgs: 3 remapped+peering, 6 peering, 3 active+remapped+backfill_toofull, 146 active+clean; 910 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 1.15k op/s; 2748/44229 objects misplaced (6.213%); 1.9 MiB/s, 30 objects/s recovering 2024-09-18T07:50:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[39175]: 11.0 starting backfill to osd.3 from (494'5936,526'10317] MIN to 543'13400 2024-09-18T07:50:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[39175]: 11.8 starting backfill to osd.3 from (485'4534,514'9240] MIN to 543'13113 2024-09-18T07:50:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[39175]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:50:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/786139808' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:50:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3835754347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[39175]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:50:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/786139808' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:50:25.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:25 smithi137 ceph-mon[39175]: osdmap e547: 8 total, 8 up, 7 in 2024-09-18T07:50:25.457 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:25.478 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:25.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:25.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:25.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:25.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:25.603 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:25.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:25.704 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:25.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:25 smithi203 ceph-mon[32237]: pgmap v1044: 158 pgs: 3 remapped+peering, 6 peering, 3 active+remapped+backfill_toofull, 146 active+clean; 910 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 1.15k op/s; 2748/44229 objects misplaced (6.213%); 1.9 MiB/s, 30 objects/s recovering 2024-09-18T07:50:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:25 smithi203 ceph-mon[32237]: 11.0 starting backfill to osd.3 from (494'5936,526'10317] MIN to 543'13400 2024-09-18T07:50:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:25 smithi203 ceph-mon[32237]: 11.8 starting backfill to osd.3 from (485'4534,514'9240] MIN to 543'13113 2024-09-18T07:50:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:25 smithi203 ceph-mon[32237]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:50:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:25 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/786139808' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:50:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:25 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3835754347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:25 smithi203 ceph-mon[32237]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:50:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:25 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/786139808' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-18T07:50:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:25 smithi203 ceph-mon[32237]: osdmap e547: 8 total, 8 up, 7 in 2024-09-18T07:50:25.727 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:25.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:25.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:25.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:25.855 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:25.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:25.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:25.957 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:25.978 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:25.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:25.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:26.080 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:26.101 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:26.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:26.203 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:26.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:26.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:26.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:26.339 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:26.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:26.361 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:26.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:26.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:26.483 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:26.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:26.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:26.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:26.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:26.606 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:26.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:26.707 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:26.731 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:26.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:26.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:26.833 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:26.857 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:26.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:26.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:26.959 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:26.985 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:26.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:26.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:27.086 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:27.111 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:27.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:27.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:27.213 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:27.252 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:27.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:27.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:27.354 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:27.386 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:27.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:27.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:27.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:27 smithi137 ceph-mon[29408]: pgmap v1047: 158 pgs: 3 remapped+peering, 6 peering, 3 active+remapped+backfill_toofull, 146 active+clean; 895 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 808 op/s; 2748/43548 objects misplaced (6.310%); 2.2 MiB/s, 35 objects/s recovering 2024-09-18T07:50:27.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:27 smithi137 ceph-mon[29408]: osdmap e548: 8 total, 8 up, 7 in 2024-09-18T07:50:27.420 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:50:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:50:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:50:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:27 smithi137 ceph-mon[39175]: pgmap v1047: 158 pgs: 3 remapped+peering, 6 peering, 3 active+remapped+backfill_toofull, 146 active+clean; 895 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 808 op/s; 2748/43548 objects misplaced (6.310%); 2.2 MiB/s, 35 objects/s recovering 2024-09-18T07:50:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:27 smithi137 ceph-mon[39175]: osdmap e548: 8 total, 8 up, 7 in 2024-09-18T07:50:27.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:27 smithi203 ceph-mon[32237]: pgmap v1047: 158 pgs: 3 remapped+peering, 6 peering, 3 active+remapped+backfill_toofull, 146 active+clean; 895 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 808 op/s; 2748/43548 objects misplaced (6.310%); 2.2 MiB/s, 35 objects/s recovering 2024-09-18T07:50:27.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:27 smithi203 ceph-mon[32237]: osdmap e548: 8 total, 8 up, 7 in 2024-09-18T07:50:27.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:27.512 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:27.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:27.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:27.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-18T07:50:27.614 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:27.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:27.636 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:27.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:27.737 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:27.758 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:27.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:27.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:27.835 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:27.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:27.881 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:27.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:27.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:27.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:28.004 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:28.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:28.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:28.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:50:28.106 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:28.127 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:28.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:28.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:28.208 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:28 smithi137 ceph-mon[29408]: osdmap e549: 8 total, 8 up, 7 in 2024-09-18T07:50:28.208 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:28 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:50:28.209 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:28 smithi137 ceph-mon[39175]: osdmap e549: 8 total, 8 up, 7 in 2024-09-18T07:50:28.209 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:28 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:50:28.229 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:28.260 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:28.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:28.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:28.361 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:28.363 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:50:28.363 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:50:28.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:50:28.363 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:50:28.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.363 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137615 10.4 10:280114ba:::benchmark_data_smithi137_151160_object1799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-18T07:50:28.363 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:42.771430+0000", 2024-09-18T07:50:28.364 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 345.58448792199999, 2024-09-18T07:50:28.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.155652747, 2024-09-18T07:50:28.364 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.364 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.364 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:50:28.364 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.364 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:50:28.364 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:28.364 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137615 2024-09-18T07:50:28.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.364 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771430+0000", 2024-09-18T07:50:28.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771430+0000", 2024-09-18T07:50:28.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771432+0000", 2024-09-18T07:50:28.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6199999999999999e-06 2024-09-18T07:50:28.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771437+0000", 2024-09-18T07:50:28.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0159999999999999e-06 2024-09-18T07:50:28.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771445+0000", 2024-09-18T07:50:28.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.225e-06 2024-09-18T07:50:28.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.771451+0000", 2024-09-18T07:50:28.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.908e-06 2024-09-18T07:50:28.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.772166+0000", 2024-09-18T07:50:28.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000714571 2024-09-18T07:50:28.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:28.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.772292+0000", 2024-09-18T07:50:28.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126322 2024-09-18T07:50:28.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-09-18T07:50:28.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.772478+0000", 2024-09-18T07:50:28.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000185747 2024-09-18T07:50:28.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:28.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.784266+0000", 2024-09-18T07:50:28.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011787824000000001 2024-09-18T07:50:28.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.795223+0000", 2024-09-18T07:50:28.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010956802999999999 2024-09-18T07:50:28.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.795236+0000", 2024-09-18T07:50:28.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3570999999999999e-05 2024-09-18T07:50:28.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774402+0000", 2024-09-18T07:50:28.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.97916638700000003 2024-09-18T07:50:28.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774413+0000", 2024-09-18T07:50:28.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0035e-05 2024-09-18T07:50:28.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927028+0000", 2024-09-18T07:50:28.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.152615967 2024-09-18T07:50:28.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927083+0000", 2024-09-18T07:50:28.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4750999999999998e-05 2024-09-18T07:50:28.372 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.372 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.372 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.372 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137618 10.4 10:22103fc7:::benchmark_data_smithi137_151160_object1802:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-18T07:50:28.372 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:42.773989+0000", 2024-09-18T07:50:28.372 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 345.581929473, 2024-09-18T07:50:28.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1739552340000001, 2024-09-18T07:50:28.372 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.372 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.373 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:50:28.373 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.373 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:50:28.373 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:28.373 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137618 2024-09-18T07:50:28.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.373 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.773989+0000", 2024-09-18T07:50:28.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.773989+0000", 2024-09-18T07:50:28.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.773991+0000", 2024-09-18T07:50:28.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.657e-06 2024-09-18T07:50:28.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.773997+0000", 2024-09-18T07:50:28.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1789999999999996e-06 2024-09-18T07:50:28.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.774008+0000", 2024-09-18T07:50:28.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0937999999999999e-05 2024-09-18T07:50:28.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.774015+0000", 2024-09-18T07:50:28.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7789999999999996e-06 2024-09-18T07:50:28.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.779798+0000", 2024-09-18T07:50:28.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057823980000000002 2024-09-18T07:50:28.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:28.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.779927+0000", 2024-09-18T07:50:28.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012920100000000001 2024-09-18T07:50:28.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3", 2024-09-18T07:50:28.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.780075+0000", 2024-09-18T07:50:28.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014824700000000001 2024-09-18T07:50:28.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:28.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.784857+0000", 2024-09-18T07:50:28.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047816070000000002 2024-09-18T07:50:28.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.834874+0000", 2024-09-18T07:50:28.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.050016892 2024-09-18T07:50:28.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.834885+0000", 2024-09-18T07:50:28.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1080999999999999e-05 2024-09-18T07:50:28.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774441+0000", 2024-09-18T07:50:28.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93955602400000005 2024-09-18T07:50:28.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774685+0000", 2024-09-18T07:50:28.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024376000000000001 2024-09-18T07:50:28.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927339+0000", 2024-09-18T07:50:28.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1526545050000001 2024-09-18T07:50:28.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-09-18T07:50:28.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927499+0000", 2024-09-18T07:50:28.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016008799999999999 2024-09-18T07:50:28.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.947852+0000", 2024-09-18T07:50:28.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020352456000000001 2024-09-18T07:50:28.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.947944+0000", 2024-09-18T07:50:28.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2422000000000004e-05 2024-09-18T07:50:28.381 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.381 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.381 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.381 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137629 10.4 10:238d70eb:::benchmark_data_smithi137_151160_object1813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-09-18T07:50:28.381 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:42.835110+0000", 2024-09-18T07:50:28.382 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 345.52080789799999, 2024-09-18T07:50:28.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.113021496, 2024-09-18T07:50:28.382 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.382 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.382 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:28.382 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.382 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:50:28.382 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:28.382 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137629 2024-09-18T07:50:28.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.383 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835110+0000", 2024-09-18T07:50:28.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835110+0000", 2024-09-18T07:50:28.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835113+0000", 2024-09-18T07:50:28.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1150000000000001e-06 2024-09-18T07:50:28.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835117+0000", 2024-09-18T07:50:28.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2359999999999998e-06 2024-09-18T07:50:28.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835124+0000", 2024-09-18T07:50:28.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3629999999999996e-06 2024-09-18T07:50:28.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835131+0000", 2024-09-18T07:50:28.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5899999999999996e-06 2024-09-18T07:50:28.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835166+0000", 2024-09-18T07:50:28.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5054e-05 2024-09-18T07:50:28.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:42.835173+0000", 2024-09-18T07:50:28.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9759999999999998e-06 2024-09-18T07:50:28.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774713+0000", 2024-09-18T07:50:28.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.93954058799999995 2024-09-18T07:50:28.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774722+0000", 2024-09-18T07:50:28.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5320000000000006e-06 2024-09-18T07:50:28.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927611+0000", 2024-09-18T07:50:28.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.152888763 2024-09-18T07:50:28.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:28.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927731+0000", 2024-09-18T07:50:28.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012040999999999999 2024-09-18T07:50:28.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,4", 2024-09-18T07:50:28.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.927854+0000", 2024-09-18T07:50:28.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012266800000000001 2024-09-18T07:50:28.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:28.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.934460+0000", 2024-09-18T07:50:28.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066062439999999998 2024-09-18T07:50:28.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.947486+0000", 2024-09-18T07:50:28.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013025956999999999 2024-09-18T07:50:28.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.947988+0000", 2024-09-18T07:50:28.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050220899999999999 2024-09-18T07:50:28.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:28.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.948040+0000", 2024-09-18T07:50:28.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1462000000000003e-05 2024-09-18T07:50:28.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.948132+0000", 2024-09-18T07:50:28.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2329000000000003e-05 2024-09-18T07:50:28.391 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.391 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.391 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.392 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137617 10.1 10:8638502c:::benchmark_data_smithi137_151160_object1801:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e396)", 2024-09-18T07:50:28.392 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:43.767316+0000", 2024-09-18T07:50:28.392 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 344.58860190000001, 2024-09-18T07:50:28.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1348433360000001, 2024-09-18T07:50:28.392 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.392 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.392 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:50:28.392 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.392 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:50:28.392 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:28.392 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137617 2024-09-18T07:50:28.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.393 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.767316+0000", 2024-09-18T07:50:28.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.767316+0000", 2024-09-18T07:50:28.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.767318+0000", 2024-09-18T07:50:28.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.889e-06 2024-09-18T07:50:28.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.767324+0000", 2024-09-18T07:50:28.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8429999999999998e-06 2024-09-18T07:50:28.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.768695+0000", 2024-09-18T07:50:28.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001370549 2024-09-18T07:50:28.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.768709+0000", 2024-09-18T07:50:28.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.434e-05 2024-09-18T07:50:28.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.772764+0000", 2024-09-18T07:50:28.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040549729999999999 2024-09-18T07:50:28.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.772774+0000", 2024-09-18T07:50:28.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4499999999999993e-06 2024-09-18T07:50:28.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.901011+0000", 2024-09-18T07:50:28.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.128237484 2024-09-18T07:50:28.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.902160+0000", 2024-09-18T07:50:28.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011488080000000001 2024-09-18T07:50:28.397 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.397 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.397 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.399 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137632 10.1 10:839066b5:::benchmark_data_smithi137_151160_object1816:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e396)", 2024-09-18T07:50:28.399 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:43.769783+0000", 2024-09-18T07:50:28.399 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 344.58613587399998, 2024-09-18T07:50:28.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1401627699999999, 2024-09-18T07:50:28.399 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.399 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.399 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:28.399 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.400 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:50:28.400 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:28.400 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137632 2024-09-18T07:50:28.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.400 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769783+0000", 2024-09-18T07:50:28.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769783+0000", 2024-09-18T07:50:28.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769784+0000", 2024-09-18T07:50:28.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7439999999999999e-06 2024-09-18T07:50:28.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769791+0000", 2024-09-18T07:50:28.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6869999999999997e-06 2024-09-18T07:50:28.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769800+0000", 2024-09-18T07:50:28.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0129999999999999e-06 2024-09-18T07:50:28.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769808+0000", 2024-09-18T07:50:28.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7060000000000004e-06 2024-09-18T07:50:28.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.772796+0000", 2024-09-18T07:50:28.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002987899 2024-09-18T07:50:28.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.772802+0000", 2024-09-18T07:50:28.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9909999999999999e-06 2024-09-18T07:50:28.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.902389+0000", 2024-09-18T07:50:28.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.129587747 2024-09-18T07:50:28.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:28.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.902738+0000", 2024-09-18T07:50:28.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034822599999999998 2024-09-18T07:50:28.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 4,7", 2024-09-18T07:50:28.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.902816+0000", 2024-09-18T07:50:28.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8430000000000006e-05 2024-09-18T07:50:28.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:28.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.904850+0000", 2024-09-18T07:50:28.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020342960000000001 2024-09-18T07:50:28.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.906005+0000", 2024-09-18T07:50:28.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011550899999999999 2024-09-18T07:50:28.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.909791+0000", 2024-09-18T07:50:28.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0037854500000000001 2024-09-18T07:50:28.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:28.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.909846+0000", 2024-09-18T07:50:28.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5565000000000001e-05 2024-09-18T07:50:28.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.909945+0000", 2024-09-18T07:50:28.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8925999999999994e-05 2024-09-18T07:50:28.407 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.407 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.407 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.407 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:137633 10.2 10:44856315:::benchmark_data_smithi137_151160_object1817:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e396)", 2024-09-18T07:50:28.407 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:44:43.769818+0000", 2024-09-18T07:50:28.408 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 344.58610065900001, 2024-09-18T07:50:28.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.138105184, 2024-09-18T07:50:28.408 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.408 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.408 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:28.408 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.408 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:50:28.408 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:28.408 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137633 2024-09-18T07:50:28.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.408 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769818+0000", 2024-09-18T07:50:28.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769818+0000", 2024-09-18T07:50:28.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769819+0000", 2024-09-18T07:50:28.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.71e-07 2024-09-18T07:50:28.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769821+0000", 2024-09-18T07:50:28.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8320000000000002e-06 2024-09-18T07:50:28.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769827+0000", 2024-09-18T07:50:28.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8799999999999996e-06 2024-09-18T07:50:28.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.769833+0000", 2024-09-18T07:50:28.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5280000000000003e-06 2024-09-18T07:50:28.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774264+0000", 2024-09-18T07:50:28.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044315839999999997 2024-09-18T07:50:28.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:43.774893+0000", 2024-09-18T07:50:28.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00062851599999999997 2024-09-18T07:50:28.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.897969+0000", 2024-09-18T07:50:28.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.123075681 2024-09-18T07:50:28.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:28.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.898060+0000", 2024-09-18T07:50:28.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1570999999999995e-05 2024-09-18T07:50:28.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-18T07:50:28.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.898142+0000", 2024-09-18T07:50:28.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1304000000000006e-05 2024-09-18T07:50:28.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.900535+0000", 2024-09-18T07:50:28.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023931239999999999 2024-09-18T07:50:28.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:28.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.907715+0000", 2024-09-18T07:50:28.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.007180278 2024-09-18T07:50:28.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.907757+0000", 2024-09-18T07:50:28.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1897000000000002e-05 2024-09-18T07:50:28.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:28.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.907819+0000", 2024-09-18T07:50:28.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2139999999999998e-05 2024-09-18T07:50:28.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:44:44.907923+0000", 2024-09-18T07:50:28.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103978 2024-09-18T07:50:28.415 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.415 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.415 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.416 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189721 10.6 10.a7ff4e76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:50:28.416 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.403483+0000", 2024-09-18T07:50:28.416 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 293.95243524400001, 2024-09-18T07:50:28.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.73509391899999998, 2024-09-18T07:50:28.416 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.416 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.416 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:50:28.416 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.416 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:50:28.417 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:28.417 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189721 2024-09-18T07:50:28.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.417 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403483+0000", 2024-09-18T07:50:28.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403483+0000", 2024-09-18T07:50:28.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403485+0000", 2024-09-18T07:50:28.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.849e-06 2024-09-18T07:50:28.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403490+0000", 2024-09-18T07:50:28.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.468e-06 2024-09-18T07:50:28.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403502+0000", 2024-09-18T07:50:28.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1865000000000001e-05 2024-09-18T07:50:28.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403509+0000", 2024-09-18T07:50:28.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.601e-06 2024-09-18T07:50:28.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403574+0000", 2024-09-18T07:50:28.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5314000000000005e-05 2024-09-18T07:50:28.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.420 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.403585+0000", 2024-09-18T07:50:28.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0688e-05 2024-09-18T07:50:28.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.420 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138555+0000", 2024-09-18T07:50:28.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.73496987400000002 2024-09-18T07:50:28.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.421 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.421 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138577+0000", 2024-09-18T07:50:28.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.226e-05 2024-09-18T07:50:28.421 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.421 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.421 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.421 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189731 10.6 10.884df406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:50:28.422 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.415372+0000", 2024-09-18T07:50:28.422 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 293.94054672099998, 2024-09-18T07:50:28.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.72323327699999995, 2024-09-18T07:50:28.422 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.422 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.422 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:50:28.422 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.422 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:50:28.422 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:28.422 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189731 2024-09-18T07:50:28.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.422 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415372+0000", 2024-09-18T07:50:28.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415372+0000", 2024-09-18T07:50:28.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415373+0000", 2024-09-18T07:50:28.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3430000000000001e-06 2024-09-18T07:50:28.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415378+0000", 2024-09-18T07:50:28.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5070000000000002e-06 2024-09-18T07:50:28.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415387+0000", 2024-09-18T07:50:28.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7529999999999996e-06 2024-09-18T07:50:28.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415394+0000", 2024-09-18T07:50:28.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4740000000000002e-06 2024-09-18T07:50:28.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415445+0000", 2024-09-18T07:50:28.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1202000000000002e-05 2024-09-18T07:50:28.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.415453+0000", 2024-09-18T07:50:28.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.123e-06 2024-09-18T07:50:28.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138598+0000", 2024-09-18T07:50:28.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.72314508399999999 2024-09-18T07:50:28.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138605+0000", 2024-09-18T07:50:28.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7909999999999999e-06 2024-09-18T07:50:28.427 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.427 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.427 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.427 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:189736 10.6 10.2043ac56 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-18T07:50:28.427 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:45:34.419163+0000", 2024-09-18T07:50:28.427 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 293.93675565299998, 2024-09-18T07:50:28.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71946739599999998, 2024-09-18T07:50:28.428 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.428 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.428 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:50:28.428 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.428 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:50:28.428 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:28.428 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 189736 2024-09-18T07:50:28.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.428 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419163+0000", 2024-09-18T07:50:28.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419163+0000", 2024-09-18T07:50:28.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419164+0000", 2024-09-18T07:50:28.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4130000000000001e-06 2024-09-18T07:50:28.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419169+0000", 2024-09-18T07:50:28.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8269999999999997e-06 2024-09-18T07:50:28.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419183+0000", 2024-09-18T07:50:28.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3596e-05 2024-09-18T07:50:28.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419191+0000", 2024-09-18T07:50:28.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0239999999999997e-06 2024-09-18T07:50:28.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419296+0000", 2024-09-18T07:50:28.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0001058 2024-09-18T07:50:28.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:34.419306+0000", 2024-09-18T07:50:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5109999999999992e-06 2024-09-18T07:50:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138625+0000", 2024-09-18T07:50:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.71931864599999995 2024-09-18T07:50:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:45:35.138630+0000", 2024-09-18T07:50:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5790000000000003e-06 2024-09-18T07:50:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:215863 10.8 10:1c60eade:::benchmark_data_smithi137_151160_object80047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-18T07:50:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.932675+0000", 2024-09-18T07:50:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 259.42324352399999, 2024-09-18T07:50:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28984009900000002, 2024-09-18T07:50:28.433 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:50:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 215863 2024-09-18T07:50:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932675+0000", 2024-09-18T07:50:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932675+0000", 2024-09-18T07:50:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932676+0000", 2024-09-18T07:50:28.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5859999999999999e-06 2024-09-18T07:50:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932681+0000", 2024-09-18T07:50:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9169999999999997e-06 2024-09-18T07:50:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932696+0000", 2024-09-18T07:50:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4341e-05 2024-09-18T07:50:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.932703+0000", 2024-09-18T07:50:28.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1369999999999996e-06 2024-09-18T07:50:28.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.938047+0000", 2024-09-18T07:50:28.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005343965 2024-09-18T07:50:28.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:28.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.938162+0000", 2024-09-18T07:50:28.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115383 2024-09-18T07:50:28.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:50:28.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.938546+0000", 2024-09-18T07:50:28.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00038351599999999998 2024-09-18T07:50:28.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.941541+0000", 2024-09-18T07:50:28.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029950369999999999 2024-09-18T07:50:28.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:28.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942037+0000", 2024-09-18T07:50:28.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00049626300000000004 2024-09-18T07:50:28.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222371+0000", 2024-09-18T07:50:28.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28033375100000002 2024-09-18T07:50:28.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:28.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222469+0000", 2024-09-18T07:50:28.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8204999999999999e-05 2024-09-18T07:50:28.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222515+0000", 2024-09-18T07:50:28.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5998000000000002e-05 2024-09-18T07:50:28.440 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.440 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.441 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.441 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:215874 10.8 10:1dfe7e41:::benchmark_data_smithi137_151160_object80058:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-18T07:50:28.441 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.942581+0000", 2024-09-18T07:50:28.441 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 259.41333776300002, 2024-09-18T07:50:28.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28009725699999999, 2024-09-18T07:50:28.441 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.441 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.441 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:28.441 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.441 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:50:28.442 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:28.442 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 215874 2024-09-18T07:50:28.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.442 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942581+0000", 2024-09-18T07:50:28.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942581+0000", 2024-09-18T07:50:28.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942582+0000", 2024-09-18T07:50:28.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.629e-06 2024-09-18T07:50:28.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942587+0000", 2024-09-18T07:50:28.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5709999999999999e-06 2024-09-18T07:50:28.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942596+0000", 2024-09-18T07:50:28.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6600000000000007e-06 2024-09-18T07:50:28.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942603+0000", 2024-09-18T07:50:28.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6630000000000004e-06 2024-09-18T07:50:28.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942687+0000", 2024-09-18T07:50:28.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3744000000000003e-05 2024-09-18T07:50:28.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:28.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942797+0000", 2024-09-18T07:50:28.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110093 2024-09-18T07:50:28.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:50:28.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.942903+0000", 2024-09-18T07:50:28.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105741 2024-09-18T07:50:28.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949093+0000", 2024-09-18T07:50:28.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0061907419999999999 2024-09-18T07:50:28.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:28.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.951910+0000", 2024-09-18T07:50:28.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028169779999999999 2024-09-18T07:50:28.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222552+0000", 2024-09-18T07:50:28.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27064144600000001 2024-09-18T07:50:28.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:28.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222639+0000", 2024-09-18T07:50:28.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7190999999999999e-05 2024-09-18T07:50:28.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.222678+0000", 2024-09-18T07:50:28.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8798999999999999e-05 2024-09-18T07:50:28.448 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.448 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.448 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.448 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.36014.0:215883 10.8 10:10ce8856:::benchmark_data_smithi137_151160_object80067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-18T07:50:28.448 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:46:08.949012+0000", 2024-09-18T07:50:28.448 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 259.40690645699999, 2024-09-18T07:50:28.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.281923759, 2024-09-18T07:50:28.449 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.449 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.449 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:28.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:28.450 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:28.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:28.451 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.451 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.36014", 2024-09-18T07:50:28.451 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/359446658", 2024-09-18T07:50:28.451 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 215883 2024-09-18T07:50:28.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.451 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.451 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.451 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949012+0000", 2024-09-18T07:50:28.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949012+0000", 2024-09-18T07:50:28.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949014+0000", 2024-09-18T07:50:28.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.063e-06 2024-09-18T07:50:28.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949020+0000", 2024-09-18T07:50:28.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7259999999999999e-06 2024-09-18T07:50:28.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949031+0000", 2024-09-18T07:50:28.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1748e-05 2024-09-18T07:50:28.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949039+0000", 2024-09-18T07:50:28.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0400000000000004e-06 2024-09-18T07:50:28.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949121+0000", 2024-09-18T07:50:28.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2622999999999998e-05 2024-09-18T07:50:28.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:28.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949221+0000", 2024-09-18T07:50:28.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100164 2024-09-18T07:50:28.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-18T07:50:28.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.949325+0000", 2024-09-18T07:50:28.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103342 2024-09-18T07:50:28.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:28.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.952116+0000", 2024-09-18T07:50:28.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027912850000000001 2024-09-18T07:50:28.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:08.953945+0000", 2024-09-18T07:50:28.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018293129999999999 2024-09-18T07:50:28.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.230849+0000", 2024-09-18T07:50:28.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27690393299999999 2024-09-18T07:50:28.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:28.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.230891+0000", 2024-09-18T07:50:28.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1854999999999998e-05 2024-09-18T07:50:28.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:46:09.230936+0000", 2024-09-18T07:50:28.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4666999999999998e-05 2024-09-18T07:50:28.457 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.457 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.457 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.458 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:100758 11.b 11:d317733f:::benchmark_data_smithi137_181069_object100757:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-18T07:50:28.458 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:32.242152+0000", 2024-09-18T07:50:28.458 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 116.11376622900001, 2024-09-18T07:50:28.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1179330860000001, 2024-09-18T07:50:28.458 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.458 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.458 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:50:28.458 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.458 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:50:28.459 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:28.459 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 100758 2024-09-18T07:50:28.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.459 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.459 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242152+0000", 2024-09-18T07:50:28.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242152+0000", 2024-09-18T07:50:28.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242155+0000", 2024-09-18T07:50:28.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7750000000000001e-06 2024-09-18T07:50:28.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242261+0000", 2024-09-18T07:50:28.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010591599999999999 2024-09-18T07:50:28.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242276+0000", 2024-09-18T07:50:28.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5593000000000001e-05 2024-09-18T07:50:28.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242289+0000", 2024-09-18T07:50:28.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2439000000000001e-05 2024-09-18T07:50:28.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249077+0000", 2024-09-18T07:50:28.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0067876439999999998 2024-09-18T07:50:28.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249087+0000", 2024-09-18T07:50:28.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0796e-05 2024-09-18T07:50:28.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.358235+0000", 2024-09-18T07:50:28.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1091477510000001 2024-09-18T07:50:28.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.360085+0000", 2024-09-18T07:50:28.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001850172 2024-09-18T07:50:28.463 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.464 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.464 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.464 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:100766 11.b 11:d6b001bf:::benchmark_data_smithi137_181069_object100765:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-18T07:50:28.464 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:32.242309+0000", 2024-09-18T07:50:28.464 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 116.113609168, 2024-09-18T07:50:28.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1181298340000001, 2024-09-18T07:50:28.464 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.464 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.464 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:50:28.465 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.465 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:50:28.465 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:28.465 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 100766 2024-09-18T07:50:28.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.465 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.465 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242309+0000", 2024-09-18T07:50:28.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242309+0000", 2024-09-18T07:50:28.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242310+0000", 2024-09-18T07:50:28.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2160000000000001e-06 2024-09-18T07:50:28.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242414+0000", 2024-09-18T07:50:28.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103411 2024-09-18T07:50:28.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242426+0000", 2024-09-18T07:50:28.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2334000000000001e-05 2024-09-18T07:50:28.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242434+0000", 2024-09-18T07:50:28.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9340000000000006e-06 2024-09-18T07:50:28.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249113+0000", 2024-09-18T07:50:28.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066785239999999999 2024-09-18T07:50:28.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249120+0000", 2024-09-18T07:50:28.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9619999999999996e-06 2024-09-18T07:50:28.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.360349+0000", 2024-09-18T07:50:28.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.111229037 2024-09-18T07:50:28.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.360439+0000", 2024-09-18T07:50:28.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0415999999999996e-05 2024-09-18T07:50:28.470 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.470 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.470 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.470 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:100768 11.b 11:dde0046a:::benchmark_data_smithi137_181069_object100767:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-18T07:50:28.470 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:32.242445+0000", 2024-09-18T07:50:28.470 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 116.113472922, 2024-09-18T07:50:28.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1180812979999999, 2024-09-18T07:50:28.470 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.470 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.470 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:50:28.471 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.471 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:50:28.471 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:28.471 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 100768 2024-09-18T07:50:28.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.471 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242445+0000", 2024-09-18T07:50:28.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242445+0000", 2024-09-18T07:50:28.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242446+0000", 2024-09-18T07:50:28.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3099999999999996e-07 2024-09-18T07:50:28.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242482+0000", 2024-09-18T07:50:28.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5871999999999998e-05 2024-09-18T07:50:28.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242490+0000", 2024-09-18T07:50:28.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9740000000000002e-06 2024-09-18T07:50:28.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242497+0000", 2024-09-18T07:50:28.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2529999999999998e-06 2024-09-18T07:50:28.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249144+0000", 2024-09-18T07:50:28.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066462190000000001 2024-09-18T07:50:28.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249150+0000", 2024-09-18T07:50:28.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4269999999999996e-06 2024-09-18T07:50:28.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.360462+0000", 2024-09-18T07:50:28.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1113119440000001 2024-09-18T07:50:28.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.360527+0000", 2024-09-18T07:50:28.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4777999999999995e-05 2024-09-18T07:50:28.476 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.476 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.476 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.476 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:100772 11.b 11:d9b390b9:::benchmark_data_smithi137_181069_object100771:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-18T07:50:28.476 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:32.242507+0000", 2024-09-18T07:50:28.476 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 116.11341172900001, 2024-09-18T07:50:28.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1180945229999999, 2024-09-18T07:50:28.476 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.476 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.477 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-18T07:50:28.477 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.477 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:50:28.477 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:28.477 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 100772 2024-09-18T07:50:28.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.477 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242507+0000", 2024-09-18T07:50:28.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242507+0000", 2024-09-18T07:50:28.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242508+0000", 2024-09-18T07:50:28.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3500000000000005e-07 2024-09-18T07:50:28.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242542+0000", 2024-09-18T07:50:28.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4329999999999998e-05 2024-09-18T07:50:28.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242559+0000", 2024-09-18T07:50:28.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6694e-05 2024-09-18T07:50:28.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242565+0000", 2024-09-18T07:50:28.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4110000000000001e-06 2024-09-18T07:50:28.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249172+0000", 2024-09-18T07:50:28.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066068919999999996 2024-09-18T07:50:28.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249178+0000", 2024-09-18T07:50:28.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2570000000000004e-06 2024-09-18T07:50:28.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.360548+0000", 2024-09-18T07:50:28.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.111369914 2024-09-18T07:50:28.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.360601+0000", 2024-09-18T07:50:28.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3189999999999997e-05 2024-09-18T07:50:28.481 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.481 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.482 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.482 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:100779 11.b 11:da6a2060:::benchmark_data_smithi137_181069_object100778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-18T07:50:28.482 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:32.242573+0000", 2024-09-18T07:50:28.482 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 116.113345251, 2024-09-18T07:50:28.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.134857102, 2024-09-18T07:50:28.482 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.482 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.482 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:28.482 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.482 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:50:28.483 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:28.483 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 100779 2024-09-18T07:50:28.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.483 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242573+0000", 2024-09-18T07:50:28.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242573+0000", 2024-09-18T07:50:28.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242574+0000", 2024-09-18T07:50:28.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6700000000000003e-07 2024-09-18T07:50:28.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242598+0000", 2024-09-18T07:50:28.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3621e-05 2024-09-18T07:50:28.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242604+0000", 2024-09-18T07:50:28.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4069999999999998e-06 2024-09-18T07:50:28.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242610+0000", 2024-09-18T07:50:28.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6540000000000004e-06 2024-09-18T07:50:28.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249201+0000", 2024-09-18T07:50:28.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065912619999999996 2024-09-18T07:50:28.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249207+0000", 2024-09-18T07:50:28.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.455e-06 2024-09-18T07:50:28.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.360621+0000", 2024-09-18T07:50:28.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1114135629999999 2024-09-18T07:50:28.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:28.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.361315+0000", 2024-09-18T07:50:28.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00069365400000000004 2024-09-18T07:50:28.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,3,4", 2024-09-18T07:50:28.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.361795+0000", 2024-09-18T07:50:28.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000480278 2024-09-18T07:50:28.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.376052+0000", 2024-09-18T07:50:28.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014257626000000001 2024-09-18T07:50:28.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.376236+0000", 2024-09-18T07:50:28.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000183108 2024-09-18T07:50:28.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:28.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.376445+0000", 2024-09-18T07:50:28.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020976999999999999 2024-09-18T07:50:28.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.490 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.490 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.376805+0000", 2024-09-18T07:50:28.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00035929499999999998 2024-09-18T07:50:28.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.490 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:28.490 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.377049+0000", 2024-09-18T07:50:28.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000243918 2024-09-18T07:50:28.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.490 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.377430+0000", 2024-09-18T07:50:28.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00038172400000000001 2024-09-18T07:50:28.491 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.491 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.491 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.491 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:100783 11.b 11:d0344236:::benchmark_data_smithi137_181069_object100782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e500)", 2024-09-18T07:50:28.491 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:32.242618+0000", 2024-09-18T07:50:28.491 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 116.113300516, 2024-09-18T07:50:28.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.144450019, 2024-09-18T07:50:28.492 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.492 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.492 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:28.492 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.492 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:50:28.492 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:28.492 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 100783 2024-09-18T07:50:28.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.492 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242618+0000", 2024-09-18T07:50:28.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242618+0000", 2024-09-18T07:50:28.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242619+0000", 2024-09-18T07:50:28.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7199999999999998e-07 2024-09-18T07:50:28.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.242861+0000", 2024-09-18T07:50:28.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024185999999999999 2024-09-18T07:50:28.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.243988+0000", 2024-09-18T07:50:28.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001127402 2024-09-18T07:50:28.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.495 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.495 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.244000+0000", 2024-09-18T07:50:28.495 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2136999999999999e-05 2024-09-18T07:50:28.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.495 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.495 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.495 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249230+0000", 2024-09-18T07:50:28.495 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052303360000000004 2024-09-18T07:50:28.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.495 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.495 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-18T07:50:28.495 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:32.249237+0000", 2024-09-18T07:50:28.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6510000000000004e-06 2024-09-18T07:50:28.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.496 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.371777+0000", 2024-09-18T07:50:28.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1225399549999999 2024-09-18T07:50:28.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.496 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:28.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.371923+0000", 2024-09-18T07:50:28.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014589099999999999 2024-09-18T07:50:28.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,3,4", 2024-09-18T07:50:28.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.372092+0000", 2024-09-18T07:50:28.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000168856 2024-09-18T07:50:28.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.376395+0000", 2024-09-18T07:50:28.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043029959999999999 2024-09-18T07:50:28.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:28.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.376466+0000", 2024-09-18T07:50:28.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0900000000000002e-05 2024-09-18T07:50:28.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.380530+0000", 2024-09-18T07:50:28.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040639589999999998 2024-09-18T07:50:28.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.386868+0000", 2024-09-18T07:50:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063385050000000004 2024-09-18T07:50:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.386940+0000", 2024-09-18T07:50:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1828000000000004e-05 2024-09-18T07:50:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.387068+0000", 2024-09-18T07:50:28.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000127971 2024-09-18T07:50:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:100778 11.1 e501/500 11:854730f9:::benchmark_data_smithi137_181069_object100777:head v 501'6357, mlcod=0'0)", 2024-09-18T07:50:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:33.406993+0000", 2024-09-18T07:50:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 114.94892585300001, 2024-09-18T07:50:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.26161929, 2024-09-18T07:50:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.500 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.406993+0000", 2024-09-18T07:50:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.406993+0000", 2024-09-18T07:50:28.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.406993+0000", 2024-09-18T07:50:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8199999999999998e-07 2024-09-18T07:50:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.409785+0000", 2024-09-18T07:50:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002791651 2024-09-18T07:50:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.639984+0000", 2024-09-18T07:50:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.23019884700000001 2024-09-18T07:50:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.639993+0000", 2024-09-18T07:50:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8729999999999996e-06 2024-09-18T07:50:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.503 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.640035+0000", 2024-09-18T07:50:28.504 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2116000000000001e-05 2024-09-18T07:50:28.504 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.504 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:28.504 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.640059+0000", 2024-09-18T07:50:28.504 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3935999999999999e-05 2024-09-18T07:50:28.504 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.504 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:28.504 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.668533+0000", 2024-09-18T07:50:28.504 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028474322 2024-09-18T07:50:28.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.505 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.505 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.505 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:33.668612+0000", 2024-09-18T07:50:28.505 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8663000000000002e-05 2024-09-18T07:50:28.505 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.505 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.505 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.505 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.505 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:123520 11.b 11:dd2ca72f:::benchmark_data_smithi137_181069_object11366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e505)", 2024-09-18T07:50:28.506 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:56.199969+0000", 2024-09-18T07:50:28.506 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 92.155949172000007, 2024-09-18T07:50:28.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.26828468100000002, 2024-09-18T07:50:28.506 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:28.506 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:28.506 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:28.506 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:28.506 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:50:28.506 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:28.506 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 123520 2024-09-18T07:50:28.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.507 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:28.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.507 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:28.507 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.199969+0000", 2024-09-18T07:50:28.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.507 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:28.507 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.199969+0000", 2024-09-18T07:50:28.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:28.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.508 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:28.508 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.199971+0000", 2024-09-18T07:50:28.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7519999999999999e-06 2024-09-18T07:50:28.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.508 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:28.508 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.199976+0000", 2024-09-18T07:50:28.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2569999999999998e-06 2024-09-18T07:50:28.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.509 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:28.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.199997+0000", 2024-09-18T07:50:28.509 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0530000000000002e-05 2024-09-18T07:50:28.509 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.509 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.509 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:28.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.200004+0000", 2024-09-18T07:50:28.509 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0389999999999999e-06 2024-09-18T07:50:28.509 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.509 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.509 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:28.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.200986+0000", 2024-09-18T07:50:28.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00098180199999999994 2024-09-18T07:50:28.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.510 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:28.510 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.201095+0000", 2024-09-18T07:50:28.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000109212 2024-09-18T07:50:28.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.510 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,3,4", 2024-09-18T07:50:28.510 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.201211+0000", 2024-09-18T07:50:28.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115937 2024-09-18T07:50:28.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:28.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.204445+0000", 2024-09-18T07:50:28.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032343630000000001 2024-09-18T07:50:28.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.205086+0000", 2024-09-18T07:50:28.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00064089099999999999 2024-09-18T07:50:28.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.512 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.512 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.207178+0000", 2024-09-18T07:50:28.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020915529999999999 2024-09-18T07:50:28.512 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.512 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.512 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:28.512 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.468090+0000", 2024-09-18T07:50:28.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.26091239100000002 2024-09-18T07:50:28.512 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.512 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.513 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:28.513 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.468149+0000", 2024-09-18T07:50:28.513 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9040999999999999e-05 2024-09-18T07:50:28.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:28.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:28.513 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:28.513 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.468254+0000", 2024-09-18T07:50:28.513 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104913 2024-09-18T07:50:28.513 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.513 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.513 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.513 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:28.514 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:28.514 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:50:28.514 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 5, 7, 6, 1, 3] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 6, 7, 4, 5] 2024-09-18T07:50:28.514 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:50:28.514 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-09-18T07:50:28.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:50:28.515 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:28 smithi203 ceph-mon[32237]: osdmap e549: 8 total, 8 up, 7 in 2024-09-18T07:50:28.515 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:28 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:50:28.516 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:50:28 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:50:28.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:28.610 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:28.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:28.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:28.712 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:28.753 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:28.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:28.785 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:50:28.855 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:28.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:28.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:28.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:29.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:29.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:29.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:29.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:29.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:29.229 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:29.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:29.330 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:29.368 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:29.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:29.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:29.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:29 smithi203 ceph-mon[32237]: pgmap v1050: 158 pgs: 3 peering, 4 active+remapped+backfill_toofull, 151 active+clean; 822 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 351 op/s; 3248/40008 objects misplaced (8.118%); 15 MiB/s, 252 objects/s recovering 2024-09-18T07:50:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:29 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:29.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:29 smithi203 ceph-mon[32237]: osdmap e550: 8 total, 8 up, 7 in 2024-09-18T07:50:29.470 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:29.502 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:29.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:29.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:29.638 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:29.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:29.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:29.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:29 smithi137 ceph-mon[29408]: pgmap v1050: 158 pgs: 3 peering, 4 active+remapped+backfill_toofull, 151 active+clean; 822 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 351 op/s; 3248/40008 objects misplaced (8.118%); 15 MiB/s, 252 objects/s recovering 2024-09-18T07:50:29.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:29 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:29 smithi137 ceph-mon[29408]: osdmap e550: 8 total, 8 up, 7 in 2024-09-18T07:50:29.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:29 smithi137 ceph-mon[39175]: pgmap v1050: 158 pgs: 3 peering, 4 active+remapped+backfill_toofull, 151 active+clean; 822 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 351 op/s; 3248/40008 objects misplaced (8.118%); 15 MiB/s, 252 objects/s recovering 2024-09-18T07:50:29.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:29 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:29.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:29 smithi137 ceph-mon[39175]: osdmap e550: 8 total, 8 up, 7 in 2024-09-18T07:50:29.741 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:29.783 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:29.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:29.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:29.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:29.911 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:29.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:29.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:29.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:30.041 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:30.082 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:30.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:30.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:30.185 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:30.220 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:30.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:30.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:30.323 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:30.368 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:30.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:30.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:30.470 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:30.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:30.516 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:30.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:30.617 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:30.661 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:30.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:30.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:30.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:30 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:50:30.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:30 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:50:30.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:30 smithi137 ceph-mon[39175]: osdmap e551: 8 total, 8 up, 7 in 2024-09-18T07:50:30.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:30 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:50:30.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:30 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:50:30.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:30 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:50:30.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:30 smithi137 ceph-mon[39175]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:50:30.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:30 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:50:30.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:30 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:50:30.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:30 smithi137 ceph-mon[29408]: osdmap e551: 8 total, 8 up, 7 in 2024-09-18T07:50:30.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:30 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:50:30.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:30 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:50:30.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:30 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:50:30.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:30 smithi137 ceph-mon[29408]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:50:30.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:30 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:50:30.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:30 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:50:30.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:30 smithi203 ceph-mon[32237]: osdmap e551: 8 total, 8 up, 7 in 2024-09-18T07:50:30.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:30 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:50:30.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:30 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:50:30.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:30 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:50:30.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:30 smithi203 ceph-mon[32237]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:50:30.763 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:30.795 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:30.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:30.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:30.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:30.932 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:30.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:30.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:31.034 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:31.071 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:31.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:31.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:31.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:31.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:31.208 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:31.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:31.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:31.341 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:31.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:31.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:31.443 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:31.481 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:31.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:31.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:31.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:31.607 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:31.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:31.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:31.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:31 smithi137 ceph-mon[29408]: pgmap v1053: 158 pgs: 3 peering, 4 active+remapped+backfill_toofull, 151 active+clean; 812 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 331 op/s; 3248/39567 objects misplaced (8.209%); 15 MiB/s, 251 objects/s recovering 2024-09-18T07:50:31.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:31 smithi137 ceph-mon[29408]: osdmap e552: 8 total, 8 up, 7 in 2024-09-18T07:50:31.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:31 smithi137 ceph-mon[39175]: pgmap v1053: 158 pgs: 3 peering, 4 active+remapped+backfill_toofull, 151 active+clean; 812 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 331 op/s; 3248/39567 objects misplaced (8.209%); 15 MiB/s, 251 objects/s recovering 2024-09-18T07:50:31.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:31 smithi137 ceph-mon[39175]: osdmap e552: 8 total, 8 up, 7 in 2024-09-18T07:50:31.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:31.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:31 smithi203 ceph-mon[32237]: pgmap v1053: 158 pgs: 3 peering, 4 active+remapped+backfill_toofull, 151 active+clean; 812 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 331 op/s; 3248/39567 objects misplaced (8.209%); 15 MiB/s, 251 objects/s recovering 2024-09-18T07:50:31.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:31 smithi203 ceph-mon[32237]: osdmap e552: 8 total, 8 up, 7 in 2024-09-18T07:50:31.730 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:31.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:31.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:31.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:31.861 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:31.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:31.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:31.963 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:31.998 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:31.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:31.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:32.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:32.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:32.141 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:32.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:32.242 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:32.269 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:32.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:32.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:32.371 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:32.400 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:32.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:32.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:32.502 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:32.531 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:32.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:32.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:32.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:32.673 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:32.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:32.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:32.775 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:32.801 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:32.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:32.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:32.903 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:32.926 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:32.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:32.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:33.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:33.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:33.067 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:33.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:33.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:33.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:33.208 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:33.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:33.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:33.339 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:33.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:33.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:33.441 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:33.475 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:33.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:33.576 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:33.609 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:33.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:33.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:33.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:33 smithi137 ceph-mon[29408]: pgmap v1055: 158 pgs: 1 peering, 4 active+remapped+backfill_toofull, 153 active+clean; 740 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 326 op/s; 2962/36069 objects misplaced (8.212%); 5.4 MiB/s, 86 objects/s recovering 2024-09-18T07:50:33.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:33 smithi137 ceph-mon[39175]: pgmap v1055: 158 pgs: 1 peering, 4 active+remapped+backfill_toofull, 153 active+clean; 740 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 326 op/s; 2962/36069 objects misplaced (8.212%); 5.4 MiB/s, 86 objects/s recovering 2024-09-18T07:50:33.710 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:33.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:33 smithi203 ceph-mon[32237]: pgmap v1055: 158 pgs: 1 peering, 4 active+remapped+backfill_toofull, 153 active+clean; 740 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 326 op/s; 2962/36069 objects misplaced (8.212%); 5.4 MiB/s, 86 objects/s recovering 2024-09-18T07:50:33.741 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:33.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:33.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:33.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:33.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:33.880 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:33.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:33.981 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:34.016 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:34.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:34.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:34.118 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:34.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:34.153 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:34.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:34.254 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:34.289 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:34.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:34.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:34.391 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:34.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:34.423 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:34.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:34.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:34.528 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:34 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:50:34.528 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:34 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:50:34.528 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:34 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:50:34.529 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:34 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:50:34.529 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:34 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:50:34.529 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:34 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:50:34.530 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-18T07:50:34.572 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:34.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:34.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:34.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:34.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:34 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:50:34.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:34 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:50:34.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:34 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:50:34.727 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:34.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:34.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:34.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:34.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:34.878 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:34.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:34.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:35.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:35.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:35.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:35.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:35.174 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:35.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:35.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:35.243 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:35.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:35.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:35.303 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:35.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:35.328 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:50:35.329 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:50:35.361 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:50:35.362 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:50:35.394 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:50:35.394 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:50:35.404 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:35.428 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:50:35.428 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:50:35.439 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:35.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:35.460 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:50:35.460 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:50:35.492 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:50:35.492 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:50:35.492 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:35 smithi137 ceph-mon[29408]: pgmap v1056: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 526 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 851 B/s rd, 0 B/s wr, 788 op/s; 2045/25815 objects misplaced (7.922%) 2024-09-18T07:50:35.493 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:35 smithi137 ceph-mon[39175]: pgmap v1056: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 526 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 851 B/s rd, 0 B/s wr, 788 op/s; 2045/25815 objects misplaced (7.922%) 2024-09-18T07:50:35.530 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:50:35.530 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:50:35.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:35.563 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:50:35.563 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:50:35.566 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:35.582 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:35.592 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:35.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:35.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:35.694 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:35.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:35 smithi203 ceph-mon[32237]: pgmap v1056: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 526 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 851 B/s rd, 0 B/s wr, 788 op/s; 2045/25815 objects misplaced (7.922%) 2024-09-18T07:50:35.723 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:35.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:35.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:35.824 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:35.861 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:35.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:35.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:35.963 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:36.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:36.008 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:36.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:36.109 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:36.139 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:36.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:36.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:36.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:36.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:36.277 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:36.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:36.379 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:36.404 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:36.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:36.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:36.436 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:50:36.441 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:50:36.463 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:36 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2822259644' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:36.463 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:36 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2822259644' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:36.506 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:36.533 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:36.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:36.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:36.634 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:36.664 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:36.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:36.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:36.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:36 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2822259644' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:36.766 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:36.795 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:36.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:36.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:36.897 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:36.926 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:36.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:36.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:37.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:37.065 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:37.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:37.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:37.167 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:37.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:50:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:50:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:50:37.198 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:37.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:37.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:37.287 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:50:37.299 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:37.328 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:37.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:37.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:37.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:37.453 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:37.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:37.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:37.555 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:37.587 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:37.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:37.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:37.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:37 smithi137 ceph-mon[39175]: pgmap v1057: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 502 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 663 B/s rd, 0 B/s wr, 662 op/s; 2045/24681 objects misplaced (8.286%) 2024-09-18T07:50:37.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:37 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:50:37.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:37 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:50:37.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:37 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/646221428' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:50:37.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:37 smithi137 ceph-mon[39175]: from='client.26878 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:50:37.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:37 smithi137 ceph-mon[29408]: pgmap v1057: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 502 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 663 B/s rd, 0 B/s wr, 662 op/s; 2045/24681 objects misplaced (8.286%) 2024-09-18T07:50:37.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:37 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:50:37.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:37 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:50:37.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:37 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/646221428' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:50:37.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:37 smithi137 ceph-mon[29408]: from='client.26878 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:50:37.689 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:37.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:37 smithi203 ceph-mon[32237]: pgmap v1057: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 502 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 663 B/s rd, 0 B/s wr, 662 op/s; 2045/24681 objects misplaced (8.286%) 2024-09-18T07:50:37.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:37 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:50:37.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:37 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:50:37.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:37 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/646221428' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:50:37.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:37 smithi203 ceph-mon[32237]: from='client.26878 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:50:37.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:37.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:37.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:37.815 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:37.846 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:37.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:37.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:37.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:37.983 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:37.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:38.085 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:38.118 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:38.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:38.220 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:38.255 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:38.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:38.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:38.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:50:38 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:50:38.358 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:38.394 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:38.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:38.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:38.468 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:38 smithi137 ceph-mon[29408]: from='client.16947 v1:172.21.15.137:0/445069952' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:50:38.468 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:38 smithi137 ceph-mon[29408]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:50:38.468 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:38 smithi137 ceph-mon[29408]: from='client.26878 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:50:38.468 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:38 smithi137 ceph-mon[29408]: osdmap e553: 8 total, 8 up, 7 in 2024-09-18T07:50:38.468 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:38 smithi137 ceph-mon[29408]: 5.13 scrub starts 2024-09-18T07:50:38.468 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:38 smithi137 ceph-mon[29408]: 5.13 scrub ok 2024-09-18T07:50:38.468 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:38 smithi137 ceph-mon[29408]: 5.c scrub starts 2024-09-18T07:50:38.469 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:38 smithi137 ceph-mon[29408]: 5.c scrub ok 2024-09-18T07:50:38.469 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:38 smithi137 ceph-mon[39175]: from='client.16947 v1:172.21.15.137:0/445069952' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:50:38.469 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:38 smithi137 ceph-mon[39175]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:50:38.469 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:38 smithi137 ceph-mon[39175]: from='client.26878 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:50:38.469 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:38 smithi137 ceph-mon[39175]: osdmap e553: 8 total, 8 up, 7 in 2024-09-18T07:50:38.469 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:38 smithi137 ceph-mon[39175]: 5.13 scrub starts 2024-09-18T07:50:38.469 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:38 smithi137 ceph-mon[39175]: 5.13 scrub ok 2024-09-18T07:50:38.469 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:38 smithi137 ceph-mon[39175]: 5.c scrub starts 2024-09-18T07:50:38.469 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:38 smithi137 ceph-mon[39175]: 5.c scrub ok 2024-09-18T07:50:38.496 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:38.517 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:38.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:38.619 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:38.640 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:38.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:38.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:38 smithi203 ceph-mon[32237]: from='client.16947 v1:172.21.15.137:0/445069952' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:50:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:38 smithi203 ceph-mon[32237]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:50:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:38 smithi203 ceph-mon[32237]: from='client.26878 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:50:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:38 smithi203 ceph-mon[32237]: osdmap e553: 8 total, 8 up, 7 in 2024-09-18T07:50:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:38 smithi203 ceph-mon[32237]: 5.13 scrub starts 2024-09-18T07:50:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:38 smithi203 ceph-mon[32237]: 5.13 scrub ok 2024-09-18T07:50:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:38 smithi203 ceph-mon[32237]: 5.c scrub starts 2024-09-18T07:50:38.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:38 smithi203 ceph-mon[32237]: 5.c scrub ok 2024-09-18T07:50:38.742 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:38.763 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:38.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:38.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:38.846 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:50:38.846 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:50:38.865 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:38.886 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:38.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:38.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:38.987 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:39.009 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:39.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:39.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:39.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:39.133 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:39.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:39.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:39.236 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:39.257 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:39.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:39.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:39.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:39.393 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:39.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:39.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:39.473 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:39 smithi137 ceph-mon[29408]: 6.16 scrub starts 2024-09-18T07:50:39.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:39 smithi137 ceph-mon[29408]: 6.16 scrub ok 2024-09-18T07:50:39.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:39 smithi137 ceph-mon[29408]: pgmap v1059: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 150 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 491/7752 objects misplaced (6.334%); 512 KiB/s, 7 objects/s recovering 2024-09-18T07:50:39.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:39 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:39.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:39 smithi137 ceph-mon[29408]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:50:39.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:39 smithi137 ceph-mon[29408]: osdmap e554: 8 total, 8 up, 7 in 2024-09-18T07:50:39.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:39 smithi137 ceph-mon[29408]: 3.19 scrub starts 2024-09-18T07:50:39.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:39 smithi137 ceph-mon[29408]: 3.19 scrub ok 2024-09-18T07:50:39.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:39 smithi137 ceph-mon[39175]: 6.16 scrub starts 2024-09-18T07:50:39.475 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:39 smithi137 ceph-mon[39175]: 6.16 scrub ok 2024-09-18T07:50:39.475 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:39 smithi137 ceph-mon[39175]: pgmap v1059: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 150 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 491/7752 objects misplaced (6.334%); 512 KiB/s, 7 objects/s recovering 2024-09-18T07:50:39.475 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:39 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:39.475 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:39 smithi137 ceph-mon[39175]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:50:39.475 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:39 smithi137 ceph-mon[39175]: osdmap e554: 8 total, 8 up, 7 in 2024-09-18T07:50:39.475 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:39 smithi137 ceph-mon[39175]: 3.19 scrub starts 2024-09-18T07:50:39.475 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:39 smithi137 ceph-mon[39175]: 3.19 scrub ok 2024-09-18T07:50:39.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:39.527 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-18T07:50:39.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:39.533 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:39.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:39.550 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Removed 112151 objects 2024-09-18T07:50:39.550 INFO:tasks.radosbench.radosbench.0.smithi137.stdout:Clean up completed and total clean up time :114.479 2024-09-18T07:50:39.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:39.656 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:39.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:39.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:39.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:39 smithi203 ceph-mon[32237]: 6.16 scrub starts 2024-09-18T07:50:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:39 smithi203 ceph-mon[32237]: 6.16 scrub ok 2024-09-18T07:50:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:39 smithi203 ceph-mon[32237]: pgmap v1059: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 150 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 491/7752 objects misplaced (6.334%); 512 KiB/s, 7 objects/s recovering 2024-09-18T07:50:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:39 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:39 smithi203 ceph-mon[32237]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-18T07:50:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:39 smithi203 ceph-mon[32237]: osdmap e554: 8 total, 8 up, 7 in 2024-09-18T07:50:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:39 smithi203 ceph-mon[32237]: 3.19 scrub starts 2024-09-18T07:50:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:39 smithi203 ceph-mon[32237]: 3.19 scrub ok 2024-09-18T07:50:39.758 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:39.780 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:39.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:39.881 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:39.906 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:39.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:39.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:39.932 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":1057,"stamp":"2024-09-18T07:50:35.911016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":526654003,"num_objects":8227,"num_object_clones":0,"num_object_copies":24681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2045,"num_objects_unfound":0,"num_objects_dirty":8227,"num_whiteouts":0,"num_read":2022,"num_read_kb":1774,"num_write":216998,"num_write_kb":7177710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47620,"num_bytes_recovered":3075231183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70615,"ondisk_log_size":70615,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":479,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6901320,"kb_used_data":2288352,"kb_used_omap":18,"kb_used_meta":4612653,"kb_avail":649314744,"statfs":{"total":671965249536,"available":664898297856,"internally_reserved":0,"allocated":2343272448,"data_stored":2340099710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19291,"internal_metadata":4723356837},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-334823424,"num_objects":-5109,"num_object_clones":0,"num_object_copies":-15327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1203,"num_objects_unfound":0,"num_objects_dirty":-5109,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5109,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.717222"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":889,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.247652+0000","last_change":"2024-09-18T07:50:14.213839+0000","last_active":"2024-09-18T07:50:30.247652+0000","last_peered":"2024-09-18T07:50:30.247652+0000","last_clean":"2024-09-18T07:50:30.247652+0000","last_became_active":"2024-09-18T07:50:14.213544+0000","last_became_peered":"2024-09-18T07:50:14.213544+0000","last_unstale":"2024-09-18T07:50:30.247652+0000","last_undegraded":"2024-09-18T07:50:30.247652+0000","last_fullsized":"2024-09-18T07:50:30.247652+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:46.455865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:46.455865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:11.769962+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"323'15","reported_seq":1014,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.237336+0000","last_change":"2024-09-18T07:47:20.194767+0000","last_active":"2024-09-18T07:50:30.237336+0000","last_peered":"2024-09-18T07:50:30.237336+0000","last_clean":"2024-09-18T07:50:30.237336+0000","last_became_active":"2024-09-18T07:47:20.136254+0000","last_became_peered":"2024-09-18T07:47:20.136254+0000","last_unstale":"2024-09-18T07:50:30.237336+0000","last_undegraded":"2024-09-18T07:50:30.237336+0000","last_fullsized":"2024-09-18T07:50:30.237336+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:47:09.864550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.864550+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":695,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249694+0000","last_change":"2024-09-18T07:50:14.215869+0000","last_active":"2024-09-18T07:50:30.249694+0000","last_peered":"2024-09-18T07:50:30.249694+0000","last_clean":"2024-09-18T07:50:30.249694+0000","last_became_active":"2024-09-18T07:50:14.215498+0000","last_became_peered":"2024-09-18T07:50:14.215498+0000","last_unstale":"2024-09-18T07:50:30.249694+0000","last_undegraded":"2024-09-18T07:50:30.249694+0000","last_fullsized":"2024-09-18T07:50:30.249694+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:05.093789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:05.093789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":907,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.238382+0000","last_change":"2024-09-18T07:49:32.689133+0000","last_active":"2024-09-18T07:50:30.238382+0000","last_peered":"2024-09-18T07:50:30.238382+0000","last_clean":"2024-09-18T07:50:30.238382+0000","last_became_active":"2024-09-18T07:49:32.688553+0000","last_became_peered":"2024-09-18T07:49:32.688553+0000","last_unstale":"2024-09-18T07:50:30.238382+0000","last_undegraded":"2024-09-18T07:50:30.238382+0000","last_fullsized":"2024-09-18T07:50:30.238382+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:14.884604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:14.884604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":26,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249798+0000","last_change":"2024-09-18T07:50:23.285610+0000","last_active":"2024-09-18T07:50:30.249798+0000","last_peered":"2024-09-18T07:50:30.249798+0000","last_clean":"2024-09-18T07:50:30.249798+0000","last_became_active":"2024-09-18T07:50:23.285220+0000","last_became_peered":"2024-09-18T07:50:23.285220+0000","last_unstale":"2024-09-18T07:50:30.249798+0000","last_undegraded":"2024-09-18T07:50:30.249798+0000","last_fullsized":"2024-09-18T07:50:30.249798+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:11.545358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:11.545358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:28.594000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":26,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.251818+0000","last_change":"2024-09-18T07:50:23.292723+0000","last_active":"2024-09-18T07:50:30.251818+0000","last_peered":"2024-09-18T07:50:30.251818+0000","last_clean":"2024-09-18T07:50:30.251818+0000","last_became_active":"2024-09-18T07:50:23.292434+0000","last_became_peered":"2024-09-18T07:50:23.292434+0000","last_unstale":"2024-09-18T07:50:30.251818+0000","last_undegraded":"2024-09-18T07:50:30.251818+0000","last_fullsized":"2024-09-18T07:50:30.251818+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:57.674688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:57.674688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"323'27","reported_seq":669,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.242611+0000","last_change":"2024-09-18T07:50:23.319419+0000","last_active":"2024-09-18T07:50:30.242611+0000","last_peered":"2024-09-18T07:50:30.242611+0000","last_clean":"2024-09-18T07:50:30.242611+0000","last_became_active":"2024-09-18T07:50:23.269846+0000","last_became_peered":"2024-09-18T07:50:23.269846+0000","last_unstale":"2024-09-18T07:50:30.242611+0000","last_undegraded":"2024-09-18T07:50:30.242611+0000","last_fullsized":"2024-09-18T07:50:30.242611+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:46:54.956330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:54.956330+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:25.107782+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":963,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.237249+0000","last_change":"2024-09-18T07:47:52.060299+0000","last_active":"2024-09-18T07:50:30.237249+0000","last_peered":"2024-09-18T07:50:30.237249+0000","last_clean":"2024-09-18T07:50:30.237249+0000","last_became_active":"2024-09-18T07:47:20.136358+0000","last_became_peered":"2024-09-18T07:47:20.136358+0000","last_unstale":"2024-09-18T07:50:30.237249+0000","last_undegraded":"2024-09-18T07:50:30.237249+0000","last_fullsized":"2024-09-18T07:50:30.237249+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:52.060246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.060246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":856,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.239713+0000","last_change":"2024-09-18T07:50:14.226012+0000","last_active":"2024-09-18T07:50:30.239713+0000","last_peered":"2024-09-18T07:50:30.239713+0000","last_clean":"2024-09-18T07:50:30.239713+0000","last_became_active":"2024-09-18T07:50:14.225616+0000","last_became_peered":"2024-09-18T07:50:14.225616+0000","last_unstale":"2024-09-18T07:50:30.239713+0000","last_undegraded":"2024-09-18T07:50:30.239713+0000","last_fullsized":"2024-09-18T07:50:30.239713+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:34.470475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:34.470475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"323'15","reported_seq":1011,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.237732+0000","last_change":"2024-09-18T07:50:23.399381+0000","last_active":"2024-09-18T07:50:30.237732+0000","last_peered":"2024-09-18T07:50:30.237732+0000","last_clean":"2024-09-18T07:50:30.237732+0000","last_became_active":"2024-09-18T07:50:23.279365+0000","last_became_peered":"2024-09-18T07:50:23.279365+0000","last_unstale":"2024-09-18T07:50:30.237732+0000","last_undegraded":"2024-09-18T07:50:30.237732+0000","last_fullsized":"2024-09-18T07:50:30.237732+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:47:47.157636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:47.157636+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":906,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.248922+0000","last_change":"2024-09-18T07:50:05.209718+0000","last_active":"2024-09-18T07:50:30.248922+0000","last_peered":"2024-09-18T07:50:30.248922+0000","last_clean":"2024-09-18T07:50:30.248922+0000","last_became_active":"2024-09-18T07:50:05.209393+0000","last_became_peered":"2024-09-18T07:50:05.209393+0000","last_unstale":"2024-09-18T07:50:30.248922+0000","last_undegraded":"2024-09-18T07:50:30.248922+0000","last_fullsized":"2024-09-18T07:50:30.248922+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:14.687386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:49:14.687386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:33.417246+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1023,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.248778+0000","last_change":"2024-09-18T07:50:05.201867+0000","last_active":"2024-09-18T07:50:30.248778+0000","last_peered":"2024-09-18T07:50:30.248778+0000","last_clean":"2024-09-18T07:50:30.248778+0000","last_became_active":"2024-09-18T07:50:05.199748+0000","last_became_peered":"2024-09-18T07:50:05.199748+0000","last_unstale":"2024-09-18T07:50:30.248778+0000","last_undegraded":"2024-09-18T07:50:30.248778+0000","last_fullsized":"2024-09-18T07:50:30.248778+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:55.019244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:55.019244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":841,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.239281+0000","last_change":"2024-09-18T07:50:14.212267+0000","last_active":"2024-09-18T07:50:30.239281+0000","last_peered":"2024-09-18T07:50:30.239281+0000","last_clean":"2024-09-18T07:50:30.239281+0000","last_became_active":"2024-09-18T07:50:14.211977+0000","last_became_peered":"2024-09-18T07:50:14.211977+0000","last_unstale":"2024-09-18T07:50:30.239281+0000","last_undegraded":"2024-09-18T07:50:30.239281+0000","last_fullsized":"2024-09-18T07:50:30.239281+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:41.617037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:41.617037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:03.818785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"323'7","reported_seq":1066,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.238559+0000","last_change":"2024-09-18T07:50:23.333287+0000","last_active":"2024-09-18T07:50:30.238559+0000","last_peered":"2024-09-18T07:50:30.238559+0000","last_clean":"2024-09-18T07:50:30.238559+0000","last_became_active":"2024-09-18T07:50:23.276419+0000","last_became_peered":"2024-09-18T07:50:23.276419+0000","last_unstale":"2024-09-18T07:50:30.238559+0000","last_undegraded":"2024-09-18T07:50:30.238559+0000","last_fullsized":"2024-09-18T07:50:30.238559+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'7","last_scrub_stamp":"2024-09-18T07:42:46.496916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:46.496916+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":829,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.252256+0000","last_change":"2024-09-18T07:47:20.143380+0000","last_active":"2024-09-18T07:50:30.252256+0000","last_peered":"2024-09-18T07:50:30.252256+0000","last_clean":"2024-09-18T07:50:30.252256+0000","last_became_active":"2024-09-18T07:47:20.142891+0000","last_became_peered":"2024-09-18T07:47:20.142891+0000","last_unstale":"2024-09-18T07:50:30.252256+0000","last_undegraded":"2024-09-18T07:50:30.252256+0000","last_fullsized":"2024-09-18T07:50:30.252256+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:53.754622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:53.754622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:18.411088+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":963,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.248612+0000","last_change":"2024-09-18T07:50:23.271566+0000","last_active":"2024-09-18T07:50:30.248612+0000","last_peered":"2024-09-18T07:50:30.248612+0000","last_clean":"2024-09-18T07:50:30.248612+0000","last_became_active":"2024-09-18T07:50:23.271118+0000","last_became_peered":"2024-09-18T07:50:23.271118+0000","last_unstale":"2024-09-18T07:50:30.248612+0000","last_undegraded":"2024-09-18T07:50:30.248612+0000","last_fullsized":"2024-09-18T07:50:30.248612+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:48.995794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:48.995794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:28.610063+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":987,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.239933+0000","last_change":"2024-09-18T07:48:31.926895+0000","last_active":"2024-09-18T07:50:30.239933+0000","last_peered":"2024-09-18T07:50:30.239933+0000","last_clean":"2024-09-18T07:50:30.239933+0000","last_became_active":"2024-09-18T07:47:11.187916+0000","last_became_peered":"2024-09-18T07:47:11.187916+0000","last_unstale":"2024-09-18T07:50:30.239933+0000","last_undegraded":"2024-09-18T07:50:30.239933+0000","last_fullsized":"2024-09-18T07:50:30.239933+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:31.926847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:31.926847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":769,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.238503+0000","last_change":"2024-09-18T07:50:05.207363+0000","last_active":"2024-09-18T07:50:30.238503+0000","last_peered":"2024-09-18T07:50:30.238503+0000","last_clean":"2024-09-18T07:50:30.238503+0000","last_became_active":"2024-09-18T07:50:05.206513+0000","last_became_peered":"2024-09-18T07:50:05.206513+0000","last_unstale":"2024-09-18T07:50:30.238503+0000","last_undegraded":"2024-09-18T07:50:30.238503+0000","last_fullsized":"2024-09-18T07:50:30.238503+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:49.150795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:49.150795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"323'27","reported_seq":1234,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.246431+0000","last_change":"2024-09-18T07:50:23.360874+0000","last_active":"2024-09-18T07:50:30.246431+0000","last_peered":"2024-09-18T07:50:30.246431+0000","last_clean":"2024-09-18T07:50:30.246431+0000","last_became_active":"2024-09-18T07:50:23.296232+0000","last_became_peered":"2024-09-18T07:50:23.296232+0000","last_unstale":"2024-09-18T07:50:30.246431+0000","last_undegraded":"2024-09-18T07:50:30.246431+0000","last_fullsized":"2024-09-18T07:50:30.246431+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:49:43.979116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:49:43.979116+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:09.363799+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":624,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.239486+0000","last_change":"2024-09-18T07:50:05.205666+0000","last_active":"2024-09-18T07:50:30.239486+0000","last_peered":"2024-09-18T07:50:30.239486+0000","last_clean":"2024-09-18T07:50:30.239486+0000","last_became_active":"2024-09-18T07:50:05.205190+0000","last_became_peered":"2024-09-18T07:50:05.205190+0000","last_unstale":"2024-09-18T07:50:30.239486+0000","last_undegraded":"2024-09-18T07:50:30.239486+0000","last_fullsized":"2024-09-18T07:50:30.239486+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:18.008882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:18.008882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":930,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.245054+0000","last_change":"2024-09-18T07:50:23.287546+0000","last_active":"2024-09-18T07:50:30.245054+0000","last_peered":"2024-09-18T07:50:30.245054+0000","last_clean":"2024-09-18T07:50:30.245054+0000","last_became_active":"2024-09-18T07:50:23.286001+0000","last_became_peered":"2024-09-18T07:50:23.286001+0000","last_unstale":"2024-09-18T07:50:30.245054+0000","last_undegraded":"2024-09-18T07:50:30.245054+0000","last_fullsized":"2024-09-18T07:50:30.245054+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:44.397262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:44.397262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":979,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.238053+0000","last_change":"2024-09-18T07:47:56.099010+0000","last_active":"2024-09-18T07:50:30.238053+0000","last_peered":"2024-09-18T07:50:30.238053+0000","last_clean":"2024-09-18T07:50:30.238053+0000","last_became_active":"2024-09-18T07:47:20.144620+0000","last_became_peered":"2024-09-18T07:47:20.144620+0000","last_unstale":"2024-09-18T07:50:30.238053+0000","last_undegraded":"2024-09-18T07:50:30.238053+0000","last_fullsized":"2024-09-18T07:50:30.238053+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:56.098927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.098927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"323'18","reported_seq":364,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.239580+0000","last_change":"2024-09-18T07:50:23.304479+0000","last_active":"2024-09-18T07:50:30.239580+0000","last_peered":"2024-09-18T07:50:30.239580+0000","last_clean":"2024-09-18T07:50:30.239580+0000","last_became_active":"2024-09-18T07:50:23.271770+0000","last_became_peered":"2024-09-18T07:50:23.271770+0000","last_unstale":"2024-09-18T07:50:30.239580+0000","last_undegraded":"2024-09-18T07:50:30.239580+0000","last_fullsized":"2024-09-18T07:50:30.239580+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:47:53.905251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.905251+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":907,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.239012+0000","last_change":"2024-09-18T07:50:23.273104+0000","last_active":"2024-09-18T07:50:30.239012+0000","last_peered":"2024-09-18T07:50:30.239012+0000","last_clean":"2024-09-18T07:50:30.239012+0000","last_became_active":"2024-09-18T07:50:23.271208+0000","last_became_peered":"2024-09-18T07:50:23.271208+0000","last_unstale":"2024-09-18T07:50:30.239012+0000","last_undegraded":"2024-09-18T07:50:30.239012+0000","last_fullsized":"2024-09-18T07:50:30.239012+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.930435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.930435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:26.920511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":827,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.252331+0000","last_change":"2024-09-18T07:47:53.034043+0000","last_active":"2024-09-18T07:50:30.252331+0000","last_peered":"2024-09-18T07:50:30.252331+0000","last_clean":"2024-09-18T07:50:30.252331+0000","last_became_active":"2024-09-18T07:47:11.194447+0000","last_became_peered":"2024-09-18T07:47:11.194447+0000","last_unstale":"2024-09-18T07:50:30.252331+0000","last_undegraded":"2024-09-18T07:50:30.252331+0000","last_fullsized":"2024-09-18T07:50:30.252331+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:47:53.033975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.033975+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":30,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.250422+0000","last_change":"2024-09-18T07:50:23.390469+0000","last_active":"2024-09-18T07:50:30.250422+0000","last_peered":"2024-09-18T07:50:30.250422+0000","last_clean":"2024-09-18T07:50:30.250422+0000","last_became_active":"2024-09-18T07:50:23.283549+0000","last_became_peered":"2024-09-18T07:50:23.283549+0000","last_unstale":"2024-09-18T07:50:30.250422+0000","last_undegraded":"2024-09-18T07:50:30.250422+0000","last_fullsized":"2024-09-18T07:50:30.250422+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:09.496455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.496455+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"323'20","reported_seq":823,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.251372+0000","last_change":"2024-09-18T07:50:05.228529+0000","last_active":"2024-09-18T07:50:30.251372+0000","last_peered":"2024-09-18T07:50:30.251372+0000","last_clean":"2024-09-18T07:50:30.251372+0000","last_became_active":"2024-09-18T07:50:05.199488+0000","last_became_peered":"2024-09-18T07:50:05.199488+0000","last_unstale":"2024-09-18T07:50:30.251372+0000","last_undegraded":"2024-09-18T07:50:30.251372+0000","last_fullsized":"2024-09-18T07:50:30.251372+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:48:19.962346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:19.962346+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"398'14","reported_seq":844,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.245168+0000","last_change":"2024-09-18T07:49:41.853017+0000","last_active":"2024-09-18T07:50:30.245168+0000","last_peered":"2024-09-18T07:50:30.245168+0000","last_clean":"2024-09-18T07:50:30.245168+0000","last_became_active":"2024-09-18T07:44:43.786594+0000","last_became_peered":"2024-09-18T07:44:43.786594+0000","last_unstale":"2024-09-18T07:50:30.245168+0000","last_undegraded":"2024-09-18T07:50:30.245168+0000","last_fullsized":"2024-09-18T07:50:30.245168+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:49:41.852939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:49:41.852939+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:46.677099+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"552'13755","reported_seq":25157,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:31.575395+0000","last_change":"2024-09-18T07:50:27.206926+0000","last_active":"2024-09-18T07:50:31.575395+0000","last_peered":"2024-09-18T07:50:31.575395+0000","last_clean":"2024-09-18T07:50:31.575395+0000","last_became_active":"2024-09-18T07:50:27.185665+0000","last_became_peered":"2024-09-18T07:50:27.185665+0000","last_unstale":"2024-09-18T07:50:31.575395+0000","last_undegraded":"2024-09-18T07:50:31.575395+0000","last_fullsized":"2024-09-18T07:50:31.575395+0000","mapping_epoch":548,"log_start":"520'9650","ondisk_log_start":"520'9650","created":459,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35323904,"num_objects":539,"num_object_clones":0,"num_object_copies":1617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13755,"num_write_kb":457408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5574,"num_bytes_recovered":365297664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":141,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.246851+0000","last_change":"2024-09-18T07:50:17.061109+0000","last_active":"2024-09-18T07:50:30.246851+0000","last_peered":"2024-09-18T07:50:30.246851+0000","last_clean":"2024-09-18T07:50:30.246851+0000","last_became_active":"2024-09-18T07:50:14.220311+0000","last_became_peered":"2024-09-18T07:50:14.220311+0000","last_unstale":"2024-09-18T07:50:30.246851+0000","last_undegraded":"2024-09-18T07:50:30.246851+0000","last_fullsized":"2024-09-18T07:50:30.246851+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:50:17.061041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:50:17.061041+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:21.651657+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"323'27","reported_seq":42,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.251305+0000","last_change":"2024-09-18T07:50:23.322623+0000","last_active":"2024-09-18T07:50:30.251305+0000","last_peered":"2024-09-18T07:50:30.251305+0000","last_clean":"2024-09-18T07:50:30.251305+0000","last_became_active":"2024-09-18T07:50:23.284875+0000","last_became_peered":"2024-09-18T07:50:23.284875+0000","last_unstale":"2024-09-18T07:50:30.251305+0000","last_undegraded":"2024-09-18T07:50:30.251305+0000","last_fullsized":"2024-09-18T07:50:30.251305+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:42:33.391715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:33.391715+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:26.558888+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":26,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.255209+0000","last_change":"2024-09-18T07:50:23.299970+0000","last_active":"2024-09-18T07:50:30.255209+0000","last_peered":"2024-09-18T07:50:30.255209+0000","last_clean":"2024-09-18T07:50:30.255209+0000","last_became_active":"2024-09-18T07:50:23.298465+0000","last_became_peered":"2024-09-18T07:50:23.298465+0000","last_unstale":"2024-09-18T07:50:30.255209+0000","last_undegraded":"2024-09-18T07:50:30.255209+0000","last_fullsized":"2024-09-18T07:50:30.255209+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:51.435576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:51.435576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:59.020688+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"536'13","reported_seq":1151,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.240021+0000","last_change":"2024-09-18T07:50:05.217905+0000","last_active":"2024-09-18T07:50:30.240021+0000","last_peered":"2024-09-18T07:50:30.240021+0000","last_clean":"2024-09-18T07:50:30.240021+0000","last_became_active":"2024-09-18T07:50:05.195271+0000","last_became_peered":"2024-09-18T07:50:05.195271+0000","last_unstale":"2024-09-18T07:50:30.240021+0000","last_undegraded":"2024-09-18T07:50:30.240021+0000","last_fullsized":"2024-09-18T07:50:30.240021+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'12","last_scrub_stamp":"2024-09-18T07:48:30.963258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:30.963258+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":897,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.245622+0000","last_change":"2024-09-18T07:50:14.212562+0000","last_active":"2024-09-18T07:50:30.245622+0000","last_peered":"2024-09-18T07:50:30.245622+0000","last_clean":"2024-09-18T07:50:30.245622+0000","last_became_active":"2024-09-18T07:50:14.212320+0000","last_became_peered":"2024-09-18T07:50:14.212320+0000","last_unstale":"2024-09-18T07:50:30.245622+0000","last_undegraded":"2024-09-18T07:50:30.245622+0000","last_fullsized":"2024-09-18T07:50:30.245622+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:08.520361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:50:08.520361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:18.334906+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"552'13586","reported_seq":15456,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:31.915000+0000","last_change":"2024-09-18T07:47:37.208096+0000","last_active":"2024-09-18T07:50:31.915000+0000","last_peered":"2024-09-18T07:50:31.915000+0000","last_clean":"2024-09-18T07:50:31.915000+0000","last_became_active":"2024-09-18T07:47:37.204971+0000","last_became_peered":"2024-09-18T07:47:37.204971+0000","last_unstale":"2024-09-18T07:50:31.915000+0000","last_undegraded":"2024-09-18T07:50:31.915000+0000","last_fullsized":"2024-09-18T07:50:31.915000+0000","mapping_epoch":474,"log_start":"517'9285","ondisk_log_start":"517'9285","created":459,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4301,"log_dups_size":0,"ondisk_log_size":4301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:24.472279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33816576,"num_objects":516,"num_object_clones":0,"num_object_copies":1548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13586,"num_write_kb":451264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":831,"num_bytes_recovered":54460416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":515,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.239682+0000","last_change":"2024-09-18T07:49:43.801107+0000","last_active":"2024-09-18T07:50:30.239682+0000","last_peered":"2024-09-18T07:50:30.239682+0000","last_clean":"2024-09-18T07:50:30.239682+0000","last_became_active":"2024-09-18T07:44:43.786396+0000","last_became_peered":"2024-09-18T07:44:43.786396+0000","last_unstale":"2024-09-18T07:50:30.239682+0000","last_undegraded":"2024-09-18T07:50:30.239682+0000","last_fullsized":"2024-09-18T07:50:30.239682+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:49:43.801039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:49:43.801039+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:56.698863+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"323'30","reported_seq":44,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.254702+0000","last_change":"2024-09-18T07:50:23.410091+0000","last_active":"2024-09-18T07:50:30.254702+0000","last_peered":"2024-09-18T07:50:30.254702+0000","last_clean":"2024-09-18T07:50:30.254702+0000","last_became_active":"2024-09-18T07:50:23.299608+0000","last_became_peered":"2024-09-18T07:50:23.299608+0000","last_unstale":"2024-09-18T07:50:30.254702+0000","last_undegraded":"2024-09-18T07:50:30.254702+0000","last_fullsized":"2024-09-18T07:50:30.254702+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'30","last_scrub_stamp":"2024-09-18T07:47:56.645991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.645991+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":788,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249394+0000","last_change":"2024-09-18T07:50:23.297915+0000","last_active":"2024-09-18T07:50:30.249394+0000","last_peered":"2024-09-18T07:50:30.249394+0000","last_clean":"2024-09-18T07:50:30.249394+0000","last_became_active":"2024-09-18T07:50:23.296257+0000","last_became_peered":"2024-09-18T07:50:23.296257+0000","last_unstale":"2024-09-18T07:50:30.249394+0000","last_undegraded":"2024-09-18T07:50:30.249394+0000","last_fullsized":"2024-09-18T07:50:30.249394+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:56.077323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.077323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":973,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.237267+0000","last_change":"2024-09-18T07:50:23.272848+0000","last_active":"2024-09-18T07:50:30.237267+0000","last_peered":"2024-09-18T07:50:30.237267+0000","last_clean":"2024-09-18T07:50:30.237267+0000","last_became_active":"2024-09-18T07:50:23.271640+0000","last_became_peered":"2024-09-18T07:50:23.271640+0000","last_unstale":"2024-09-18T07:50:30.237267+0000","last_undegraded":"2024-09-18T07:50:30.237267+0000","last_fullsized":"2024-09-18T07:50:30.237267+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:53.056484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.056484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":875,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.237835+0000","last_change":"2024-09-18T07:50:23.279639+0000","last_active":"2024-09-18T07:50:30.237835+0000","last_peered":"2024-09-18T07:50:30.237835+0000","last_clean":"2024-09-18T07:50:30.237835+0000","last_became_active":"2024-09-18T07:50:23.279337+0000","last_became_peered":"2024-09-18T07:50:23.279337+0000","last_unstale":"2024-09-18T07:50:30.237835+0000","last_undegraded":"2024-09-18T07:50:30.237835+0000","last_fullsized":"2024-09-18T07:50:30.237835+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:07.931065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:07.931065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"552'13567","reported_seq":15483,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:31.484111+0000","last_change":"2024-09-18T07:47:35.123117+0000","last_active":"2024-09-18T07:50:31.484111+0000","last_peered":"2024-09-18T07:50:31.484111+0000","last_clean":"2024-09-18T07:50:31.484111+0000","last_became_active":"2024-09-18T07:47:35.119437+0000","last_became_peered":"2024-09-18T07:47:35.119437+0000","last_unstale":"2024-09-18T07:50:31.484111+0000","last_undegraded":"2024-09-18T07:50:31.484111+0000","last_fullsized":"2024-09-18T07:50:31.484111+0000","mapping_epoch":472,"log_start":"520'9569","ondisk_log_start":"520'9569","created":459,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751040,"num_objects":515,"num_object_clones":0,"num_object_copies":1545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13567,"num_write_kb":450624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":55902208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":30,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.250174+0000","last_change":"2024-09-18T07:50:23.307333+0000","last_active":"2024-09-18T07:50:30.250174+0000","last_peered":"2024-09-18T07:50:30.250174+0000","last_clean":"2024-09-18T07:50:30.250174+0000","last_became_active":"2024-09-18T07:50:23.276444+0000","last_became_peered":"2024-09-18T07:50:23.276444+0000","last_unstale":"2024-09-18T07:50:30.250174+0000","last_undegraded":"2024-09-18T07:50:30.250174+0000","last_fullsized":"2024-09-18T07:50:30.250174+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:50:14.450061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:50:14.450061+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:33.842830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"323'28","reported_seq":987,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.237753+0000","last_change":"2024-09-18T07:48:26.993988+0000","last_active":"2024-09-18T07:50:30.237753+0000","last_peered":"2024-09-18T07:50:30.237753+0000","last_clean":"2024-09-18T07:50:30.237753+0000","last_became_active":"2024-09-18T07:47:11.188002+0000","last_became_peered":"2024-09-18T07:47:11.188002+0000","last_unstale":"2024-09-18T07:50:30.237753+0000","last_undegraded":"2024-09-18T07:50:30.237753+0000","last_fullsized":"2024-09-18T07:50:30.237753+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'28","last_scrub_stamp":"2024-09-18T07:48:26.993871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:26.993871+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":61,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.240190+0000","last_change":"2024-09-18T07:50:05.229977+0000","last_active":"2024-09-18T07:50:30.240190+0000","last_peered":"2024-09-18T07:50:30.240190+0000","last_clean":"2024-09-18T07:50:30.240190+0000","last_became_active":"2024-09-18T07:50:05.202682+0000","last_became_peered":"2024-09-18T07:50:05.202682+0000","last_unstale":"2024-09-18T07:50:30.240190+0000","last_undegraded":"2024-09-18T07:50:30.240190+0000","last_fullsized":"2024-09-18T07:50:30.240190+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:49:44.482296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:44.482296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:57.492382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"548'15","reported_seq":32,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.255015+0000","last_change":"2024-09-18T07:50:23.403994+0000","last_active":"2024-09-18T07:50:30.255015+0000","last_peered":"2024-09-18T07:50:30.255015+0000","last_clean":"2024-09-18T07:50:30.255015+0000","last_became_active":"2024-09-18T07:50:23.300057+0000","last_became_peered":"2024-09-18T07:50:23.300057+0000","last_unstale":"2024-09-18T07:50:30.255015+0000","last_undegraded":"2024-09-18T07:50:30.255015+0000","last_fullsized":"2024-09-18T07:50:30.255015+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'12","last_scrub_stamp":"2024-09-18T07:48:20.859791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:20.859791+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":819,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.251768+0000","last_change":"2024-09-18T07:47:42.984418+0000","last_active":"2024-09-18T07:50:30.251768+0000","last_peered":"2024-09-18T07:50:30.251768+0000","last_clean":"2024-09-18T07:50:30.251768+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:50:30.251768+0000","last_undegraded":"2024-09-18T07:50:30.251768+0000","last_fullsized":"2024-09-18T07:50:30.251768+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:42.984298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.984298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"552'13895","reported_seq":14094,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:31.890353+0000","last_change":"2024-09-18T07:47:10.178170+0000","last_active":"2024-09-18T07:50:31.890353+0000","last_peered":"2024-09-18T07:50:31.890353+0000","last_clean":"2024-09-18T07:50:31.890353+0000","last_became_active":"2024-09-18T07:47:10.177901+0000","last_became_peered":"2024-09-18T07:47:10.177901+0000","last_unstale":"2024-09-18T07:50:31.890353+0000","last_undegraded":"2024-09-18T07:50:31.890353+0000","last_fullsized":"2024-09-18T07:50:31.890353+0000","mapping_epoch":459,"log_start":"510'9170","ondisk_log_start":"510'9170","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4725,"log_dups_size":0,"ondisk_log_size":4725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35389464,"num_objects":541,"num_object_clones":0,"num_object_copies":1623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":541,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13895,"num_write_kb":461889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":770,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.244912+0000","last_change":"2024-09-18T07:48:10.151652+0000","last_active":"2024-09-18T07:50:30.244912+0000","last_peered":"2024-09-18T07:50:30.244912+0000","last_clean":"2024-09-18T07:50:30.244912+0000","last_became_active":"2024-09-18T07:48:10.151343+0000","last_became_peered":"2024-09-18T07:48:10.151343+0000","last_unstale":"2024-09-18T07:50:30.244912+0000","last_undegraded":"2024-09-18T07:50:30.244912+0000","last_fullsized":"2024-09-18T07:50:30.244912+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:02.444272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:02.444272+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"323'20","reported_seq":734,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.248563+0000","last_change":"2024-09-18T07:49:32.698546+0000","last_active":"2024-09-18T07:50:30.248563+0000","last_peered":"2024-09-18T07:50:30.248563+0000","last_clean":"2024-09-18T07:50:30.248563+0000","last_became_active":"2024-09-18T07:49:32.688605+0000","last_became_peered":"2024-09-18T07:49:32.688605+0000","last_unstale":"2024-09-18T07:50:30.248563+0000","last_undegraded":"2024-09-18T07:50:30.248563+0000","last_fullsized":"2024-09-18T07:50:30.248563+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:48:33.642815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:33.642815+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:20.641560+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":952,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.250303+0000","last_change":"2024-09-18T07:50:23.268974+0000","last_active":"2024-09-18T07:50:30.250303+0000","last_peered":"2024-09-18T07:50:30.250303+0000","last_clean":"2024-09-18T07:50:30.250303+0000","last_became_active":"2024-09-18T07:50:23.268446+0000","last_became_peered":"2024-09-18T07:50:23.268446+0000","last_unstale":"2024-09-18T07:50:30.250303+0000","last_undegraded":"2024-09-18T07:50:30.250303+0000","last_fullsized":"2024-09-18T07:50:30.250303+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:19.210077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:19.210077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:29.643987+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"548'15","reported_seq":805,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.246488+0000","last_change":"2024-09-18T07:50:23.333125+0000","last_active":"2024-09-18T07:50:30.246488+0000","last_peered":"2024-09-18T07:50:30.246488+0000","last_clean":"2024-09-18T07:50:30.246488+0000","last_became_active":"2024-09-18T07:50:23.286461+0000","last_became_peered":"2024-09-18T07:50:23.286461+0000","last_unstale":"2024-09-18T07:50:30.246488+0000","last_undegraded":"2024-09-18T07:50:30.246488+0000","last_fullsized":"2024-09-18T07:50:30.246488+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'13","last_scrub_stamp":"2024-09-18T07:48:24.605876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:24.605876+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":26,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.254867+0000","last_change":"2024-09-18T07:50:23.293839+0000","last_active":"2024-09-18T07:50:30.254867+0000","last_peered":"2024-09-18T07:50:30.254867+0000","last_clean":"2024-09-18T07:50:30.254867+0000","last_became_active":"2024-09-18T07:50:23.293581+0000","last_became_peered":"2024-09-18T07:50:23.293581+0000","last_unstale":"2024-09-18T07:50:30.254867+0000","last_undegraded":"2024-09-18T07:50:30.254867+0000","last_fullsized":"2024-09-18T07:50:30.254867+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:46.172671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:46.172671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"552'13367","reported_seq":15172,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:31.879187+0000","last_change":"2024-09-18T07:47:31.131323+0000","last_active":"2024-09-18T07:50:31.879187+0000","last_peered":"2024-09-18T07:50:31.879187+0000","last_clean":"2024-09-18T07:50:31.879187+0000","last_became_active":"2024-09-18T07:47:31.128537+0000","last_became_peered":"2024-09-18T07:47:31.128537+0000","last_unstale":"2024-09-18T07:50:31.879187+0000","last_undegraded":"2024-09-18T07:50:31.879187+0000","last_fullsized":"2024-09-18T07:50:31.879187+0000","mapping_epoch":468,"log_start":"510'8638","ondisk_log_start":"510'8638","created":459,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4729,"log_dups_size":0,"ondisk_log_size":4729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:03.944505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33488896,"num_objects":511,"num_object_clones":0,"num_object_copies":1533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":444096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":799,"num_bytes_recovered":52363264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":792,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.244538+0000","last_change":"2024-09-18T07:50:23.286117+0000","last_active":"2024-09-18T07:50:30.244538+0000","last_peered":"2024-09-18T07:50:30.244538+0000","last_clean":"2024-09-18T07:50:30.244538+0000","last_became_active":"2024-09-18T07:50:23.283668+0000","last_became_peered":"2024-09-18T07:50:23.283668+0000","last_unstale":"2024-09-18T07:50:30.244538+0000","last_undegraded":"2024-09-18T07:50:30.244538+0000","last_fullsized":"2024-09-18T07:50:30.244538+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:47:59.422927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:47:59.422927+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"323'29","reported_seq":894,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249162+0000","last_change":"2024-09-18T07:50:16.593489+0000","last_active":"2024-09-18T07:50:30.249162+0000","last_peered":"2024-09-18T07:50:30.249162+0000","last_clean":"2024-09-18T07:50:30.249162+0000","last_became_active":"2024-09-18T07:50:05.198854+0000","last_became_peered":"2024-09-18T07:50:05.198854+0000","last_unstale":"2024-09-18T07:50:30.249162+0000","last_undegraded":"2024-09-18T07:50:30.249162+0000","last_fullsized":"2024-09-18T07:50:30.249162+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'29","last_scrub_stamp":"2024-09-18T07:50:16.593349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:50:16.593349+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:43.411102+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":756,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.246631+0000","last_change":"2024-09-18T07:47:20.140174+0000","last_active":"2024-09-18T07:50:30.246631+0000","last_peered":"2024-09-18T07:50:30.246631+0000","last_clean":"2024-09-18T07:50:30.246631+0000","last_became_active":"2024-09-18T07:47:20.139806+0000","last_became_peered":"2024-09-18T07:47:20.139806+0000","last_unstale":"2024-09-18T07:50:30.246631+0000","last_undegraded":"2024-09-18T07:50:30.246631+0000","last_fullsized":"2024-09-18T07:50:30.246631+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:43.985969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:43.985969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:55.976994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":691,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.244557+0000","last_change":"2024-09-18T07:50:05.192481+0000","last_active":"2024-09-18T07:50:30.244557+0000","last_peered":"2024-09-18T07:50:30.244557+0000","last_clean":"2024-09-18T07:50:30.244557+0000","last_became_active":"2024-09-18T07:50:05.192195+0000","last_became_peered":"2024-09-18T07:50:05.192195+0000","last_unstale":"2024-09-18T07:50:30.244557+0000","last_undegraded":"2024-09-18T07:50:30.244557+0000","last_fullsized":"2024-09-18T07:50:30.244557+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:41.319318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:41.319318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":687,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.247687+0000","last_change":"2024-09-18T07:49:43.912594+0000","last_active":"2024-09-18T07:50:30.247687+0000","last_peered":"2024-09-18T07:50:30.247687+0000","last_clean":"2024-09-18T07:50:30.247687+0000","last_became_active":"2024-09-18T07:47:20.138390+0000","last_became_peered":"2024-09-18T07:47:20.138390+0000","last_unstale":"2024-09-18T07:50:30.247687+0000","last_undegraded":"2024-09-18T07:50:30.247687+0000","last_fullsized":"2024-09-18T07:50:30.247687+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:43.912475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:43.912475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:51.322173+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.b","version":"552'13389","reported_seq":27486,"reported_epoch":552,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-18T07:50:31.493140+0000","last_change":"2024-09-18T07:50:23.403232+0000","last_active":"2024-09-18T07:50:31.493140+0000","last_peered":"2024-09-18T07:50:31.493140+0000","last_clean":"2024-09-18T07:50:04.311095+0000","last_became_active":"2024-09-18T07:50:23.370312+0000","last_became_peered":"2024-09-18T07:50:23.370312+0000","last_unstale":"2024-09-18T07:50:31.493140+0000","last_undegraded":"2024-09-18T07:50:31.493140+0000","last_fullsized":"2024-09-18T07:50:31.493140+0000","mapping_epoch":544,"log_start":"519'9372","ondisk_log_start":"519'9372","created":459,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35323904,"num_objects":539,"num_object_clones":0,"num_object_copies":1617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":539,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13389,"num_write_kb":445696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6928,"num_bytes_recovered":453967872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[4,1,0],"avail_no_missing":["4","0","1"],"object_location_counts":[{"shards":"0,1,4","objects":539}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":48,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.242240+0000","last_change":"2024-09-18T07:50:15.048190+0000","last_active":"2024-09-18T07:50:30.242240+0000","last_peered":"2024-09-18T07:50:30.242240+0000","last_clean":"2024-09-18T07:50:30.242240+0000","last_became_active":"2024-09-18T07:50:14.210684+0000","last_became_peered":"2024-09-18T07:50:14.210684+0000","last_unstale":"2024-09-18T07:50:30.242240+0000","last_undegraded":"2024-09-18T07:50:30.242240+0000","last_fullsized":"2024-09-18T07:50:30.242240+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:50:15.048136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:50:15.048136+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:24.356275+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"548'46","reported_seq":1154,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.246382+0000","last_change":"2024-09-18T07:50:23.400624+0000","last_active":"2024-09-18T07:50:30.246382+0000","last_peered":"2024-09-18T07:50:30.246382+0000","last_clean":"2024-09-18T07:50:30.246382+0000","last_became_active":"2024-09-18T07:50:23.296840+0000","last_became_peered":"2024-09-18T07:50:23.296840+0000","last_unstale":"2024-09-18T07:50:30.246382+0000","last_undegraded":"2024-09-18T07:50:30.246382+0000","last_fullsized":"2024-09-18T07:50:30.246382+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'45","last_scrub_stamp":"2024-09-18T07:49:51.025843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:49:51.025843+0000","objects_scrubbed":10,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:05.378056+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":1790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":965,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.238803+0000","last_change":"2024-09-18T07:50:05.207077+0000","last_active":"2024-09-18T07:50:30.238803+0000","last_peered":"2024-09-18T07:50:30.238803+0000","last_clean":"2024-09-18T07:50:30.238803+0000","last_became_active":"2024-09-18T07:50:05.206221+0000","last_became_peered":"2024-09-18T07:50:05.206221+0000","last_unstale":"2024-09-18T07:50:30.238803+0000","last_undegraded":"2024-09-18T07:50:30.238803+0000","last_fullsized":"2024-09-18T07:50:30.238803+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:51.085400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:51.085400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":546,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.244519+0000","last_change":"2024-09-18T07:50:14.227393+0000","last_active":"2024-09-18T07:50:30.244519+0000","last_peered":"2024-09-18T07:50:30.244519+0000","last_clean":"2024-09-18T07:50:30.244519+0000","last_became_active":"2024-09-18T07:50:14.227023+0000","last_became_peered":"2024-09-18T07:50:14.227023+0000","last_unstale":"2024-09-18T07:50:30.244519+0000","last_undegraded":"2024-09-18T07:50:30.244519+0000","last_fullsized":"2024-09-18T07:50:30.244519+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:48:06.127745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:48:06.127745+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":903,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.239389+0000","last_change":"2024-09-18T07:50:05.197988+0000","last_active":"2024-09-18T07:50:30.239389+0000","last_peered":"2024-09-18T07:50:30.239389+0000","last_clean":"2024-09-18T07:50:30.239389+0000","last_became_active":"2024-09-18T07:50:05.195463+0000","last_became_peered":"2024-09-18T07:50:05.195463+0000","last_unstale":"2024-09-18T07:50:30.239389+0000","last_undegraded":"2024-09-18T07:50:30.239389+0000","last_fullsized":"2024-09-18T07:50:30.239389+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:46.629196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:46.629196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":26,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.251656+0000","last_change":"2024-09-18T07:50:23.286830+0000","last_active":"2024-09-18T07:50:30.251656+0000","last_peered":"2024-09-18T07:50:30.251656+0000","last_clean":"2024-09-18T07:50:30.251656+0000","last_became_active":"2024-09-18T07:50:23.286499+0000","last_became_peered":"2024-09-18T07:50:23.286499+0000","last_unstale":"2024-09-18T07:50:30.251656+0000","last_undegraded":"2024-09-18T07:50:30.251656+0000","last_fullsized":"2024-09-18T07:50:30.251656+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:51.995541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:51.995541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:03.582186+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"552'13488","reported_seq":29148,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:32.319532+0000","last_change":"2024-09-18T07:50:28.232622+0000","last_active":"2024-09-18T07:50:32.319532+0000","last_peered":"2024-09-18T07:50:32.319532+0000","last_clean":"2024-09-18T07:50:32.319532+0000","last_became_active":"2024-09-18T07:50:28.200679+0000","last_became_peered":"2024-09-18T07:50:28.200679+0000","last_unstale":"2024-09-18T07:50:32.319532+0000","last_undegraded":"2024-09-18T07:50:32.319532+0000","last_fullsized":"2024-09-18T07:50:32.319532+0000","mapping_epoch":549,"log_start":"518'9360","ondisk_log_start":"518'9360","created":459,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'5883","last_scrub_stamp":"2024-09-18T07:48:25.402648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:48:25.402648+0000","objects_scrubbed":5819,"log_size":4128,"log_dups_size":0,"ondisk_log_size":4128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34209792,"num_objects":522,"num_object_clones":0,"num_object_copies":1566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13488,"num_write_kb":448320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7675,"num_bytes_recovered":502857728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":888,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.252013+0000","last_change":"2024-09-18T07:50:05.196938+0000","last_active":"2024-09-18T07:50:30.252013+0000","last_peered":"2024-09-18T07:50:30.252013+0000","last_clean":"2024-09-18T07:50:30.252013+0000","last_became_active":"2024-09-18T07:50:05.196606+0000","last_became_peered":"2024-09-18T07:50:05.196606+0000","last_unstale":"2024-09-18T07:50:30.252013+0000","last_undegraded":"2024-09-18T07:50:30.252013+0000","last_fullsized":"2024-09-18T07:50:30.252013+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:27.916095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:27.916095+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"323'17","reported_seq":762,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249014+0000","last_change":"2024-09-18T07:50:14.214658+0000","last_active":"2024-09-18T07:50:30.249014+0000","last_peered":"2024-09-18T07:50:30.249014+0000","last_clean":"2024-09-18T07:50:30.249014+0000","last_became_active":"2024-09-18T07:50:14.213033+0000","last_became_peered":"2024-09-18T07:50:14.213033+0000","last_unstale":"2024-09-18T07:50:30.249014+0000","last_undegraded":"2024-09-18T07:50:30.249014+0000","last_fullsized":"2024-09-18T07:50:30.249014+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'17","last_scrub_stamp":"2024-09-18T07:47:56.037047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.037047+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":23,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":770,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.243086+0000","last_change":"2024-09-18T07:47:20.140678+0000","last_active":"2024-09-18T07:50:30.243086+0000","last_peered":"2024-09-18T07:50:30.243086+0000","last_clean":"2024-09-18T07:50:30.243086+0000","last_became_active":"2024-09-18T07:47:20.140376+0000","last_became_peered":"2024-09-18T07:47:20.140376+0000","last_unstale":"2024-09-18T07:50:30.243086+0000","last_undegraded":"2024-09-18T07:50:30.243086+0000","last_fullsized":"2024-09-18T07:50:30.243086+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:52.917296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:52.917296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":1093,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.242696+0000","last_change":"2024-09-18T07:50:23.389266+0000","last_active":"2024-09-18T07:50:30.242696+0000","last_peered":"2024-09-18T07:50:30.242696+0000","last_clean":"2024-09-18T07:50:30.242696+0000","last_became_active":"2024-09-18T07:50:23.276401+0000","last_became_peered":"2024-09-18T07:50:23.276401+0000","last_unstale":"2024-09-18T07:50:30.242696+0000","last_undegraded":"2024-09-18T07:50:30.242696+0000","last_fullsized":"2024-09-18T07:50:30.242696+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:07.841406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:47:07.841406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1060,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.250407+0000","last_change":"2024-09-18T07:47:16.834107+0000","last_active":"2024-09-18T07:50:30.250407+0000","last_peered":"2024-09-18T07:50:30.250407+0000","last_clean":"2024-09-18T07:50:30.250407+0000","last_became_active":"2024-09-18T07:47:11.201379+0000","last_became_peered":"2024-09-18T07:47:11.201379+0000","last_unstale":"2024-09-18T07:50:30.250407+0000","last_undegraded":"2024-09-18T07:50:30.250407+0000","last_fullsized":"2024-09-18T07:50:30.250407+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:16.834043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:16.834043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":859,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249437+0000","last_change":"2024-09-18T07:50:23.318559+0000","last_active":"2024-09-18T07:50:30.249437+0000","last_peered":"2024-09-18T07:50:30.249437+0000","last_clean":"2024-09-18T07:50:30.249437+0000","last_became_active":"2024-09-18T07:50:23.296775+0000","last_became_peered":"2024-09-18T07:50:23.296775+0000","last_unstale":"2024-09-18T07:50:30.249437+0000","last_undegraded":"2024-09-18T07:50:30.249437+0000","last_fullsized":"2024-09-18T07:50:30.249437+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:47:42.037791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.037791+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":91,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"552'13446","reported_seq":26359,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:31.555066+0000","last_change":"2024-09-18T07:50:27.204083+0000","last_active":"2024-09-18T07:50:31.555066+0000","last_peered":"2024-09-18T07:50:31.555066+0000","last_clean":"2024-09-18T07:50:31.555066+0000","last_became_active":"2024-09-18T07:50:27.184616+0000","last_became_peered":"2024-09-18T07:50:27.184616+0000","last_unstale":"2024-09-18T07:50:31.555066+0000","last_undegraded":"2024-09-18T07:50:31.555066+0000","last_fullsized":"2024-09-18T07:50:31.555066+0000","mapping_epoch":548,"log_start":"519'9444","ondisk_log_start":"519'9444","created":459,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":0,"last_scrub":"528'10674","last_scrub_stamp":"2024-09-18T07:49:47.098373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:49:47.098373+0000","objects_scrubbed":3360,"log_size":4002,"log_dups_size":0,"ondisk_log_size":4002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:47.350849+0000","scrub_duration":1162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34865152,"num_objects":532,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13446,"num_write_kb":447296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6315,"num_bytes_recovered":413794304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":81,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.246182+0000","last_change":"2024-09-18T07:49:24.453650+0000","last_active":"2024-09-18T07:50:30.246182+0000","last_peered":"2024-09-18T07:50:30.246182+0000","last_clean":"2024-09-18T07:50:30.246182+0000","last_became_active":"2024-09-18T07:49:24.453318+0000","last_became_peered":"2024-09-18T07:49:24.453318+0000","last_unstale":"2024-09-18T07:50:30.246182+0000","last_undegraded":"2024-09-18T07:50:30.246182+0000","last_fullsized":"2024-09-18T07:50:30.246182+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:49:12.773317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:49:12.773317+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"323'24","reported_seq":656,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.250303+0000","last_change":"2024-09-18T07:50:05.958641+0000","last_active":"2024-09-18T07:50:30.250303+0000","last_peered":"2024-09-18T07:50:30.250303+0000","last_clean":"2024-09-18T07:50:30.250303+0000","last_became_active":"2024-09-18T07:50:05.201657+0000","last_became_peered":"2024-09-18T07:50:05.201657+0000","last_unstale":"2024-09-18T07:50:30.250303+0000","last_undegraded":"2024-09-18T07:50:30.250303+0000","last_fullsized":"2024-09-18T07:50:30.250303+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'24","last_scrub_stamp":"2024-09-18T07:50:05.958566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:50:05.958566+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:24.486287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":69,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.237505+0000","last_change":"2024-09-18T07:50:16.881850+0000","last_active":"2024-09-18T07:50:30.237505+0000","last_peered":"2024-09-18T07:50:30.237505+0000","last_clean":"2024-09-18T07:50:30.237505+0000","last_became_active":"2024-09-18T07:50:05.191515+0000","last_became_peered":"2024-09-18T07:50:05.191515+0000","last_unstale":"2024-09-18T07:50:30.237505+0000","last_undegraded":"2024-09-18T07:50:30.237505+0000","last_fullsized":"2024-09-18T07:50:30.237505+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:50:16.881751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:50:16.881751+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:34.588198+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":1773,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.308934+0000","last_change":"2024-09-18T07:50:23.351560+0000","last_active":"2024-09-18T07:50:30.308934+0000","last_peered":"2024-09-18T07:50:30.308934+0000","last_clean":"2024-09-18T07:50:30.308934+0000","last_became_active":"2024-09-18T07:50:23.297717+0000","last_became_peered":"2024-09-18T07:50:23.297717+0000","last_unstale":"2024-09-18T07:50:30.308934+0000","last_undegraded":"2024-09-18T07:50:30.308934+0000","last_fullsized":"2024-09-18T07:50:30.308934+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T07:50:07.355097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:50:07.355097+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:29.323794+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":925,"num_read_kb":920,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1945,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":939,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.253126+0000","last_change":"2024-09-18T07:50:05.197752+0000","last_active":"2024-09-18T07:50:30.253126+0000","last_peered":"2024-09-18T07:50:30.253126+0000","last_clean":"2024-09-18T07:50:30.253126+0000","last_became_active":"2024-09-18T07:50:05.196676+0000","last_became_peered":"2024-09-18T07:50:05.196676+0000","last_unstale":"2024-09-18T07:50:30.253126+0000","last_undegraded":"2024-09-18T07:50:30.253126+0000","last_fullsized":"2024-09-18T07:50:30.253126+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:42.663800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:49:42.663800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:55.598699+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":620,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249588+0000","last_change":"2024-09-18T07:50:23.275031+0000","last_active":"2024-09-18T07:50:30.249588+0000","last_peered":"2024-09-18T07:50:30.249588+0000","last_clean":"2024-09-18T07:50:30.249588+0000","last_became_active":"2024-09-18T07:50:23.274722+0000","last_became_peered":"2024-09-18T07:50:23.274722+0000","last_unstale":"2024-09-18T07:50:30.249588+0000","last_undegraded":"2024-09-18T07:50:30.249588+0000","last_fullsized":"2024-09-18T07:50:30.249588+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:43.622102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:43.622102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:54.255403+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.a","version":"552'13403","reported_seq":25225,"reported_epoch":552,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-18T07:50:32.310873+0000","last_change":"2024-09-18T07:50:23.428921+0000","last_active":"2024-09-18T07:50:32.310873+0000","last_peered":"2024-09-18T07:50:32.310873+0000","last_clean":"2024-09-18T07:49:31.606072+0000","last_became_active":"2024-09-18T07:50:23.392157+0000","last_became_peered":"2024-09-18T07:50:23.392157+0000","last_unstale":"2024-09-18T07:50:32.310873+0000","last_undegraded":"2024-09-18T07:50:32.310873+0000","last_fullsized":"2024-09-18T07:50:32.310873+0000","mapping_epoch":544,"log_start":"519'9289","ondisk_log_start":"519'9289","created":459,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'5664","last_scrub_stamp":"2024-09-18T07:48:22.026341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:48:22.026341+0000","objects_scrubbed":5623,"log_size":4114,"log_dups_size":0,"ondisk_log_size":4114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":461,"num_object_clones":0,"num_object_copies":1383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":461,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13403,"num_write_kb":443648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5759,"num_bytes_recovered":377290752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,5,2],"avail_no_missing":["6","1","2","5"],"object_location_counts":[{"shards":"2,5,6","objects":461}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"398'34","reported_seq":868,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.247267+0000","last_change":"2024-09-18T07:47:58.415964+0000","last_active":"2024-09-18T07:50:30.247267+0000","last_peered":"2024-09-18T07:50:30.247267+0000","last_clean":"2024-09-18T07:50:30.247267+0000","last_became_active":"2024-09-18T07:44:43.786827+0000","last_became_peered":"2024-09-18T07:44:43.786827+0000","last_unstale":"2024-09-18T07:50:30.247267+0000","last_undegraded":"2024-09-18T07:50:30.247267+0000","last_fullsized":"2024-09-18T07:50:30.247267+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'34","last_scrub_stamp":"2024-09-18T07:47:58.415734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:58.415734+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":807,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.248817+0000","last_change":"2024-09-18T07:50:23.306674+0000","last_active":"2024-09-18T07:50:30.248817+0000","last_peered":"2024-09-18T07:50:30.248817+0000","last_clean":"2024-09-18T07:50:30.248817+0000","last_became_active":"2024-09-18T07:50:23.306340+0000","last_became_peered":"2024-09-18T07:50:23.306340+0000","last_unstale":"2024-09-18T07:50:30.248817+0000","last_undegraded":"2024-09-18T07:50:30.248817+0000","last_fullsized":"2024-09-18T07:50:30.248817+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:08.604203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:08.604203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":871,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249088+0000","last_change":"2024-09-18T07:50:05.203665+0000","last_active":"2024-09-18T07:50:30.249088+0000","last_peered":"2024-09-18T07:50:30.249088+0000","last_clean":"2024-09-18T07:50:30.249088+0000","last_became_active":"2024-09-18T07:50:05.202222+0000","last_became_peered":"2024-09-18T07:50:05.202222+0000","last_unstale":"2024-09-18T07:50:30.249088+0000","last_undegraded":"2024-09-18T07:50:30.249088+0000","last_fullsized":"2024-09-18T07:50:30.249088+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:26.963223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:26.963223+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":816,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.252427+0000","last_change":"2024-09-18T07:48:21.934191+0000","last_active":"2024-09-18T07:50:30.252427+0000","last_peered":"2024-09-18T07:50:30.252427+0000","last_clean":"2024-09-18T07:50:30.252427+0000","last_became_active":"2024-09-18T07:47:11.195084+0000","last_became_peered":"2024-09-18T07:47:11.195084+0000","last_unstale":"2024-09-18T07:50:30.252427+0000","last_undegraded":"2024-09-18T07:50:30.252427+0000","last_fullsized":"2024-09-18T07:50:30.252427+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:21.934130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:21.934130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":935,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.253028+0000","last_change":"2024-09-18T07:50:23.275089+0000","last_active":"2024-09-18T07:50:30.253028+0000","last_peered":"2024-09-18T07:50:30.253028+0000","last_clean":"2024-09-18T07:50:30.253028+0000","last_became_active":"2024-09-18T07:50:23.274710+0000","last_became_peered":"2024-09-18T07:50:23.274710+0000","last_unstale":"2024-09-18T07:50:30.253028+0000","last_undegraded":"2024-09-18T07:50:30.253028+0000","last_fullsized":"2024-09-18T07:50:30.253028+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:04.610883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:04.610883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"552'13639","reported_seq":23515,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:32.326302+0000","last_change":"2024-09-18T07:50:26.200515+0000","last_active":"2024-09-18T07:50:32.326302+0000","last_peered":"2024-09-18T07:50:32.326302+0000","last_clean":"2024-09-18T07:50:32.326302+0000","last_became_active":"2024-09-18T07:50:26.182838+0000","last_became_peered":"2024-09-18T07:50:26.182838+0000","last_unstale":"2024-09-18T07:50:32.326302+0000","last_undegraded":"2024-09-18T07:50:32.326302+0000","last_fullsized":"2024-09-18T07:50:32.326302+0000","mapping_epoch":547,"log_start":"520'9552","ondisk_log_start":"520'9552","created":459,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"529'11126","last_scrub_stamp":"2024-09-18T07:49:52.119967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:49:52.119967+0000","objects_scrubbed":3032,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:04.869397+0000","scrub_duration":1238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31653888,"num_objects":483,"num_object_clones":0,"num_object_copies":1449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13639,"num_write_kb":451904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4797,"num_bytes_recovered":314310656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"323'19","reported_seq":1168,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.240074+0000","last_change":"2024-09-18T07:49:32.696886+0000","last_active":"2024-09-18T07:50:30.240074+0000","last_peered":"2024-09-18T07:50:30.240074+0000","last_clean":"2024-09-18T07:50:30.240074+0000","last_became_active":"2024-09-18T07:49:32.684857+0000","last_became_peered":"2024-09-18T07:49:32.684857+0000","last_unstale":"2024-09-18T07:50:30.240074+0000","last_undegraded":"2024-09-18T07:50:30.240074+0000","last_fullsized":"2024-09-18T07:50:30.240074+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:42:43.402858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:43.402858+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:39.835405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":921,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.237534+0000","last_change":"2024-09-18T07:44:43.793621+0000","last_active":"2024-09-18T07:50:30.237534+0000","last_peered":"2024-09-18T07:50:30.237534+0000","last_clean":"2024-09-18T07:50:30.237534+0000","last_became_active":"2024-09-18T07:44:43.790160+0000","last_became_peered":"2024-09-18T07:44:43.790160+0000","last_unstale":"2024-09-18T07:50:30.237534+0000","last_undegraded":"2024-09-18T07:50:30.237534+0000","last_fullsized":"2024-09-18T07:50:30.237534+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.078600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.078600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:25.912776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":857,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.239273+0000","last_change":"2024-09-18T07:50:17.844593+0000","last_active":"2024-09-18T07:50:30.239273+0000","last_peered":"2024-09-18T07:50:30.239273+0000","last_clean":"2024-09-18T07:50:30.239273+0000","last_became_active":"2024-09-18T07:50:14.223548+0000","last_became_peered":"2024-09-18T07:50:14.223548+0000","last_unstale":"2024-09-18T07:50:30.239273+0000","last_undegraded":"2024-09-18T07:50:30.239273+0000","last_fullsized":"2024-09-18T07:50:30.239273+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:17.844473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:50:17.844473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:22.705281+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":930,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.245512+0000","last_change":"2024-09-18T07:50:23.287269+0000","last_active":"2024-09-18T07:50:30.245512+0000","last_peered":"2024-09-18T07:50:30.245512+0000","last_clean":"2024-09-18T07:50:30.245512+0000","last_became_active":"2024-09-18T07:50:23.285080+0000","last_became_peered":"2024-09-18T07:50:23.285080+0000","last_unstale":"2024-09-18T07:50:30.245512+0000","last_undegraded":"2024-09-18T07:50:30.245512+0000","last_fullsized":"2024-09-18T07:50:30.245512+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:26.691414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:26.691414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"552'13212","reported_seq":23151,"reported_epoch":552,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-18T07:50:31.904140+0000","last_change":"2024-09-18T07:50:23.416491+0000","last_active":"2024-09-18T07:50:31.904140+0000","last_peered":"2024-09-18T07:50:31.904140+0000","last_clean":"2024-09-18T07:50:04.310249+0000","last_became_active":"2024-09-18T07:50:23.379158+0000","last_became_peered":"2024-09-18T07:50:23.379158+0000","last_unstale":"2024-09-18T07:50:31.904140+0000","last_undegraded":"2024-09-18T07:50:31.904140+0000","last_fullsized":"2024-09-18T07:50:31.904140+0000","mapping_epoch":544,"log_start":"510'8455","ondisk_log_start":"510'8455","created":459,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4757,"log_dups_size":0,"ondisk_log_size":4757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34996224,"num_objects":534,"num_object_clones":0,"num_object_copies":1602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":534,"num_objects_unfound":0,"num_objects_dirty":534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13212,"num_write_kb":439872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4854,"num_bytes_recovered":318111744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,7,0],"avail_no_missing":["2","0","7"],"object_location_counts":[{"shards":"0,2,7","objects":534}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":152,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.242629+0000","last_change":"2024-09-18T07:50:23.312027+0000","last_active":"2024-09-18T07:50:30.242629+0000","last_peered":"2024-09-18T07:50:30.242629+0000","last_clean":"2024-09-18T07:50:30.242629+0000","last_became_active":"2024-09-18T07:50:23.275682+0000","last_became_peered":"2024-09-18T07:50:23.275682+0000","last_unstale":"2024-09-18T07:50:30.242629+0000","last_undegraded":"2024-09-18T07:50:30.242629+0000","last_fullsized":"2024-09-18T07:50:30.242629+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:50:16.037345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:50:16.037345+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:43.188373+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"323'31","reported_seq":1003,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.239807+0000","last_change":"2024-09-18T07:48:22.002514+0000","last_active":"2024-09-18T07:50:30.239807+0000","last_peered":"2024-09-18T07:50:30.239807+0000","last_clean":"2024-09-18T07:50:30.239807+0000","last_became_active":"2024-09-18T07:47:20.132912+0000","last_became_peered":"2024-09-18T07:47:20.132912+0000","last_unstale":"2024-09-18T07:50:30.239807+0000","last_undegraded":"2024-09-18T07:50:30.239807+0000","last_fullsized":"2024-09-18T07:50:30.239807+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'31","last_scrub_stamp":"2024-09-18T07:48:22.002416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:22.002416+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":726,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.244673+0000","last_change":"2024-09-18T07:49:32.688728+0000","last_active":"2024-09-18T07:50:30.244673+0000","last_peered":"2024-09-18T07:50:30.244673+0000","last_clean":"2024-09-18T07:50:30.244673+0000","last_became_active":"2024-09-18T07:49:32.688352+0000","last_became_peered":"2024-09-18T07:49:32.688352+0000","last_unstale":"2024-09-18T07:50:30.244673+0000","last_undegraded":"2024-09-18T07:50:30.244673+0000","last_fullsized":"2024-09-18T07:50:30.244673+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:32.491919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:32.491919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":599,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.238658+0000","last_change":"2024-09-18T07:50:05.197643+0000","last_active":"2024-09-18T07:50:30.238658+0000","last_peered":"2024-09-18T07:50:30.238658+0000","last_clean":"2024-09-18T07:50:30.238658+0000","last_became_active":"2024-09-18T07:50:05.196640+0000","last_became_peered":"2024-09-18T07:50:05.196640+0000","last_unstale":"2024-09-18T07:50:30.238658+0000","last_undegraded":"2024-09-18T07:50:30.238658+0000","last_fullsized":"2024-09-18T07:50:30.238658+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:13.019711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:13.019711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1071,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.247566+0000","last_change":"2024-09-18T07:50:23.277183+0000","last_active":"2024-09-18T07:50:30.247566+0000","last_peered":"2024-09-18T07:50:30.247566+0000","last_clean":"2024-09-18T07:50:30.247566+0000","last_became_active":"2024-09-18T07:50:23.276876+0000","last_became_peered":"2024-09-18T07:50:23.276876+0000","last_unstale":"2024-09-18T07:50:30.247566+0000","last_undegraded":"2024-09-18T07:50:30.247566+0000","last_fullsized":"2024-09-18T07:50:30.247566+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:06.889139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:06.889139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:30.073517+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"552'13505","reported_seq":15479,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:32.338431+0000","last_change":"2024-09-18T07:48:27.491090+0000","last_active":"2024-09-18T07:50:32.338431+0000","last_peered":"2024-09-18T07:50:32.338431+0000","last_clean":"2024-09-18T07:50:32.338431+0000","last_became_active":"2024-09-18T07:47:38.218858+0000","last_became_peered":"2024-09-18T07:47:38.218858+0000","last_unstale":"2024-09-18T07:50:32.338431+0000","last_undegraded":"2024-09-18T07:50:32.338431+0000","last_fullsized":"2024-09-18T07:50:32.338431+0000","mapping_epoch":475,"log_start":"519'9407","ondisk_log_start":"519'9407","created":459,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'6054","last_scrub_stamp":"2024-09-18T07:48:27.490707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:48:27.490707+0000","objects_scrubbed":6013,"log_size":4098,"log_dups_size":0,"ondisk_log_size":4098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31260672,"num_objects":477,"num_object_clones":0,"num_object_copies":1431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13505,"num_write_kb":447424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":848,"num_bytes_recovered":55574528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":884,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249020+0000","last_change":"2024-09-18T07:50:23.296215+0000","last_active":"2024-09-18T07:50:30.249020+0000","last_peered":"2024-09-18T07:50:30.249020+0000","last_clean":"2024-09-18T07:50:30.249020+0000","last_became_active":"2024-09-18T07:50:23.295783+0000","last_became_peered":"2024-09-18T07:50:23.295783+0000","last_unstale":"2024-09-18T07:50:30.249020+0000","last_undegraded":"2024-09-18T07:50:30.249020+0000","last_fullsized":"2024-09-18T07:50:30.249020+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:49:47.404771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:49:47.404771+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:51.076110+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"323'20","reported_seq":994,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.239105+0000","last_change":"2024-09-18T07:50:23.329138+0000","last_active":"2024-09-18T07:50:30.239105+0000","last_peered":"2024-09-18T07:50:30.239105+0000","last_clean":"2024-09-18T07:50:30.239105+0000","last_became_active":"2024-09-18T07:50:23.278411+0000","last_became_peered":"2024-09-18T07:50:23.278411+0000","last_unstale":"2024-09-18T07:50:30.239105+0000","last_undegraded":"2024-09-18T07:50:30.239105+0000","last_fullsized":"2024-09-18T07:50:30.239105+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:46:14.047907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:14.047907+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":467,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.237775+0000","last_change":"2024-09-18T07:50:23.278102+0000","last_active":"2024-09-18T07:50:30.237775+0000","last_peered":"2024-09-18T07:50:30.237775+0000","last_clean":"2024-09-18T07:50:30.237775+0000","last_became_active":"2024-09-18T07:50:23.277790+0000","last_became_peered":"2024-09-18T07:50:23.277790+0000","last_unstale":"2024-09-18T07:50:30.237775+0000","last_undegraded":"2024-09-18T07:50:30.237775+0000","last_fullsized":"2024-09-18T07:50:30.237775+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:42.225418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.225418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"461'13","reported_seq":1316,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.245843+0000","last_change":"2024-09-18T07:47:15.865446+0000","last_active":"2024-09-18T07:50:30.245843+0000","last_peered":"2024-09-18T07:50:30.245843+0000","last_clean":"2024-09-18T07:50:30.245843+0000","last_became_active":"2024-09-18T07:47:11.203256+0000","last_became_peered":"2024-09-18T07:47:11.203256+0000","last_unstale":"2024-09-18T07:50:30.245843+0000","last_undegraded":"2024-09-18T07:50:30.245843+0000","last_fullsized":"2024-09-18T07:50:30.245843+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'13","last_scrub_stamp":"2024-09-18T07:47:15.865373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:15.865373+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":944,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.240393+0000","last_change":"2024-09-18T07:49:32.690291+0000","last_active":"2024-09-18T07:50:30.240393+0000","last_peered":"2024-09-18T07:50:30.240393+0000","last_clean":"2024-09-18T07:50:30.240393+0000","last_became_active":"2024-09-18T07:49:32.689562+0000","last_became_peered":"2024-09-18T07:49:32.689562+0000","last_unstale":"2024-09-18T07:50:30.240393+0000","last_undegraded":"2024-09-18T07:50:30.240393+0000","last_fullsized":"2024-09-18T07:50:30.240393+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:48.207484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:48.207484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"552'13556","reported_seq":13754,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:31.919825+0000","last_change":"2024-09-18T07:47:10.191967+0000","last_active":"2024-09-18T07:50:31.919825+0000","last_peered":"2024-09-18T07:50:31.919825+0000","last_clean":"2024-09-18T07:50:31.919825+0000","last_became_active":"2024-09-18T07:47:10.190921+0000","last_became_peered":"2024-09-18T07:47:10.190921+0000","last_unstale":"2024-09-18T07:50:31.919825+0000","last_undegraded":"2024-09-18T07:50:31.919825+0000","last_fullsized":"2024-09-18T07:50:31.919825+0000","mapping_epoch":459,"log_start":"518'9296","ondisk_log_start":"518'9296","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:25.522892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33816576,"num_objects":516,"num_object_clones":0,"num_object_copies":1548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13556,"num_write_kb":450304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"323'15","reported_seq":662,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.238881+0000","last_change":"2024-09-18T07:50:23.406391+0000","last_active":"2024-09-18T07:50:30.238881+0000","last_peered":"2024-09-18T07:50:30.238881+0000","last_clean":"2024-09-18T07:50:30.238881+0000","last_became_active":"2024-09-18T07:50:23.288752+0000","last_became_peered":"2024-09-18T07:50:23.288752+0000","last_unstale":"2024-09-18T07:50:30.238881+0000","last_undegraded":"2024-09-18T07:50:30.238881+0000","last_fullsized":"2024-09-18T07:50:30.238881+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:49:45.465070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:49:45.465070+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:08.907615+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":26,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.250087+0000","last_change":"2024-09-18T07:50:23.285852+0000","last_active":"2024-09-18T07:50:30.250087+0000","last_peered":"2024-09-18T07:50:30.250087+0000","last_clean":"2024-09-18T07:50:30.250087+0000","last_became_active":"2024-09-18T07:50:23.285473+0000","last_became_peered":"2024-09-18T07:50:23.285473+0000","last_unstale":"2024-09-18T07:50:30.250087+0000","last_undegraded":"2024-09-18T07:50:30.250087+0000","last_fullsized":"2024-09-18T07:50:30.250087+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:51.021530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:51.021530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1066,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.247161+0000","last_change":"2024-09-18T07:50:23.274125+0000","last_active":"2024-09-18T07:50:30.247161+0000","last_peered":"2024-09-18T07:50:30.247161+0000","last_clean":"2024-09-18T07:50:30.247161+0000","last_became_active":"2024-09-18T07:50:23.273703+0000","last_became_peered":"2024-09-18T07:50:23.273703+0000","last_unstale":"2024-09-18T07:50:30.247161+0000","last_undegraded":"2024-09-18T07:50:30.247161+0000","last_fullsized":"2024-09-18T07:50:30.247161+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:22.799585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:22.799585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":495,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249402+0000","last_change":"2024-09-18T07:50:23.275281+0000","last_active":"2024-09-18T07:50:30.249402+0000","last_peered":"2024-09-18T07:50:30.249402+0000","last_clean":"2024-09-18T07:50:30.249402+0000","last_became_active":"2024-09-18T07:50:23.274910+0000","last_became_peered":"2024-09-18T07:50:23.274910+0000","last_unstale":"2024-09-18T07:50:30.249402+0000","last_undegraded":"2024-09-18T07:50:30.249402+0000","last_fullsized":"2024-09-18T07:50:30.249402+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:42.436545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:42.436545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:03.369633+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"552'13353","reported_seq":13550,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:31.900353+0000","last_change":"2024-09-18T07:47:11.201597+0000","last_active":"2024-09-18T07:50:31.900353+0000","last_peered":"2024-09-18T07:50:31.900353+0000","last_clean":"2024-09-18T07:50:31.900353+0000","last_became_active":"2024-09-18T07:47:11.201275+0000","last_became_peered":"2024-09-18T07:47:11.201275+0000","last_unstale":"2024-09-18T07:50:31.900353+0000","last_undegraded":"2024-09-18T07:50:31.900353+0000","last_fullsized":"2024-09-18T07:50:31.900353+0000","mapping_epoch":460,"log_start":"516'9090","ondisk_log_start":"516'9090","created":459,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4263,"log_dups_size":0,"ondisk_log_size":4263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33619968,"num_objects":513,"num_object_clones":0,"num_object_copies":1539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13353,"num_write_kb":443712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"323'35","reported_seq":775,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.244846+0000","last_change":"2024-09-18T07:47:52.386750+0000","last_active":"2024-09-18T07:50:30.244846+0000","last_peered":"2024-09-18T07:50:30.244846+0000","last_clean":"2024-09-18T07:50:30.244846+0000","last_became_active":"2024-09-18T07:47:20.141942+0000","last_became_peered":"2024-09-18T07:47:20.141942+0000","last_unstale":"2024-09-18T07:50:30.244846+0000","last_undegraded":"2024-09-18T07:50:30.244846+0000","last_fullsized":"2024-09-18T07:50:30.244846+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'35","last_scrub_stamp":"2024-09-18T07:47:52.386688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.386688+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":829,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249857+0000","last_change":"2024-09-18T07:47:17.794908+0000","last_active":"2024-09-18T07:50:30.249857+0000","last_peered":"2024-09-18T07:50:30.249857+0000","last_clean":"2024-09-18T07:50:30.249857+0000","last_became_active":"2024-09-18T07:47:11.194873+0000","last_became_peered":"2024-09-18T07:47:11.194873+0000","last_unstale":"2024-09-18T07:50:30.249857+0000","last_undegraded":"2024-09-18T07:50:30.249857+0000","last_fullsized":"2024-09-18T07:50:30.249857+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:17.794850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:17.794850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1074,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.248633+0000","last_change":"2024-09-18T07:50:05.193285+0000","last_active":"2024-09-18T07:50:30.248633+0000","last_peered":"2024-09-18T07:50:30.248633+0000","last_clean":"2024-09-18T07:50:30.248633+0000","last_became_active":"2024-09-18T07:50:05.192813+0000","last_became_peered":"2024-09-18T07:50:05.192813+0000","last_unstale":"2024-09-18T07:50:30.248633+0000","last_undegraded":"2024-09-18T07:50:30.248633+0000","last_fullsized":"2024-09-18T07:50:30.248633+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:20.767517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:20.767517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":819,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249179+0000","last_change":"2024-09-18T07:50:23.297645+0000","last_active":"2024-09-18T07:50:30.249179+0000","last_peered":"2024-09-18T07:50:30.249179+0000","last_clean":"2024-09-18T07:50:30.249179+0000","last_became_active":"2024-09-18T07:50:23.296866+0000","last_became_peered":"2024-09-18T07:50:23.296866+0000","last_unstale":"2024-09-18T07:50:30.249179+0000","last_undegraded":"2024-09-18T07:50:30.249179+0000","last_fullsized":"2024-09-18T07:50:30.249179+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:50.079275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:50.079275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"552'13347","reported_seq":15232,"reported_epoch":552,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-18T07:50:31.908964+0000","last_change":"2024-09-18T07:50:06.357457+0000","last_active":"2024-09-18T07:50:31.908964+0000","last_peered":"2024-09-18T07:50:31.908964+0000","last_clean":"2024-09-18T07:50:04.320709+0000","last_became_active":"2024-09-18T07:50:06.277856+0000","last_became_peered":"2024-09-18T07:50:06.277856+0000","last_unstale":"2024-09-18T07:50:31.908964+0000","last_undegraded":"2024-09-18T07:50:31.908964+0000","last_fullsized":"2024-09-18T07:50:31.908964+0000","mapping_epoch":535,"log_start":"510'8659","ondisk_log_start":"510'8659","created":459,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.154476+0000","objects_scrubbed":0,"log_size":4688,"log_dups_size":0,"ondisk_log_size":4688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33488896,"num_objects":511,"num_object_clones":0,"num_object_copies":1533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":511,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13347,"num_write_kb":443456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":831,"num_bytes_recovered":54460416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,0],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"0,1,2","objects":511}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"323'18","reported_seq":1061,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.237451+0000","last_change":"2024-09-18T07:50:23.305432+0000","last_active":"2024-09-18T07:50:30.237451+0000","last_peered":"2024-09-18T07:50:30.237451+0000","last_clean":"2024-09-18T07:50:30.237451+0000","last_became_active":"2024-09-18T07:50:23.272111+0000","last_became_peered":"2024-09-18T07:50:23.272111+0000","last_unstale":"2024-09-18T07:50:30.237451+0000","last_undegraded":"2024-09-18T07:50:30.237451+0000","last_fullsized":"2024-09-18T07:50:30.237451+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:50:04.943731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:50:04.943731+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:15.112228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":821,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.251470+0000","last_change":"2024-09-18T07:47:20.139266+0000","last_active":"2024-09-18T07:50:30.251470+0000","last_peered":"2024-09-18T07:50:30.251470+0000","last_clean":"2024-09-18T07:50:30.251470+0000","last_became_active":"2024-09-18T07:47:20.138910+0000","last_became_peered":"2024-09-18T07:47:20.138910+0000","last_unstale":"2024-09-18T07:50:30.251470+0000","last_undegraded":"2024-09-18T07:50:30.251470+0000","last_fullsized":"2024-09-18T07:50:30.251470+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:51.752578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:51.752578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:26.489350+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1085,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.242946+0000","last_change":"2024-09-18T07:50:05.180815+0000","last_active":"2024-09-18T07:50:30.242946+0000","last_peered":"2024-09-18T07:50:30.242946+0000","last_clean":"2024-09-18T07:50:30.242946+0000","last_became_active":"2024-09-18T07:50:05.180071+0000","last_became_peered":"2024-09-18T07:50:05.180071+0000","last_unstale":"2024-09-18T07:50:30.242946+0000","last_undegraded":"2024-09-18T07:50:30.242946+0000","last_fullsized":"2024-09-18T07:50:30.242946+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:23.757372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:23.757372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":26,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.250139+0000","last_change":"2024-09-18T07:50:23.299961+0000","last_active":"2024-09-18T07:50:30.250139+0000","last_peered":"2024-09-18T07:50:30.250139+0000","last_clean":"2024-09-18T07:50:30.250139+0000","last_became_active":"2024-09-18T07:50:23.299077+0000","last_became_peered":"2024-09-18T07:50:23.299077+0000","last_unstale":"2024-09-18T07:50:30.250139+0000","last_undegraded":"2024-09-18T07:50:30.250139+0000","last_fullsized":"2024-09-18T07:50:30.250139+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:40.439672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:40.439672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"552'13768","reported_seq":15766,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:34.234210+0000","last_change":"2024-09-18T07:48:53.603702+0000","last_active":"2024-09-18T07:50:34.234210+0000","last_peered":"2024-09-18T07:50:34.234210+0000","last_clean":"2024-09-18T07:50:34.234210+0000","last_became_active":"2024-09-18T07:47:34.181063+0000","last_became_peered":"2024-09-18T07:47:34.181063+0000","last_unstale":"2024-09-18T07:50:34.234210+0000","last_undegraded":"2024-09-18T07:50:34.234210+0000","last_fullsized":"2024-09-18T07:50:34.234210+0000","mapping_epoch":471,"log_start":"513'9113","ondisk_log_start":"513'9113","created":459,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'7617","last_scrub_stamp":"2024-09-18T07:48:53.603594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:47:09.154476+0000","last_clean_scrub_stamp":"2024-09-18T07:48:53.603594+0000","objects_scrubbed":6521,"log_size":4655,"log_dups_size":0,"ondisk_log_size":4655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20840448,"num_objects":318,"num_object_clones":0,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13768,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":858,"num_bytes_recovered":56229888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"323'27","reported_seq":1025,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.248217+0000","last_change":"2024-09-18T07:48:03.057334+0000","last_active":"2024-09-18T07:50:30.248217+0000","last_peered":"2024-09-18T07:50:30.248217+0000","last_clean":"2024-09-18T07:50:30.248217+0000","last_became_active":"2024-09-18T07:47:11.196288+0000","last_became_peered":"2024-09-18T07:47:11.196288+0000","last_unstale":"2024-09-18T07:50:30.248217+0000","last_undegraded":"2024-09-18T07:50:30.248217+0000","last_fullsized":"2024-09-18T07:50:30.248217+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:48:03.057182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:03.057182+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:01.465573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":842,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.251329+0000","last_change":"2024-09-18T07:50:05.231617+0000","last_active":"2024-09-18T07:50:30.251329+0000","last_peered":"2024-09-18T07:50:30.251329+0000","last_clean":"2024-09-18T07:50:30.251329+0000","last_became_active":"2024-09-18T07:50:05.200873+0000","last_became_peered":"2024-09-18T07:50:05.200873+0000","last_unstale":"2024-09-18T07:50:30.251329+0000","last_undegraded":"2024-09-18T07:50:30.251329+0000","last_fullsized":"2024-09-18T07:50:30.251329+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:48:34.003134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:34.003134+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"398'14","reported_seq":1039,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249646+0000","last_change":"2024-09-18T07:50:05.354087+0000","last_active":"2024-09-18T07:50:30.249646+0000","last_peered":"2024-09-18T07:50:30.249646+0000","last_clean":"2024-09-18T07:50:30.249646+0000","last_became_active":"2024-09-18T07:44:43.782355+0000","last_became_peered":"2024-09-18T07:44:43.782355+0000","last_unstale":"2024-09-18T07:50:30.249646+0000","last_undegraded":"2024-09-18T07:50:30.249646+0000","last_fullsized":"2024-09-18T07:50:30.249646+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:50:05.354034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:50:05.354034+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:08.048196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":894,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.244414+0000","last_change":"2024-09-18T07:50:14.217105+0000","last_active":"2024-09-18T07:50:30.244414+0000","last_peered":"2024-09-18T07:50:30.244414+0000","last_clean":"2024-09-18T07:50:30.244414+0000","last_became_active":"2024-09-18T07:50:14.216827+0000","last_became_peered":"2024-09-18T07:50:14.216827+0000","last_unstale":"2024-09-18T07:50:30.244414+0000","last_undegraded":"2024-09-18T07:50:30.244414+0000","last_fullsized":"2024-09-18T07:50:30.244414+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:42.110980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.110980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":681,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.247442+0000","last_change":"2024-09-18T07:50:23.287774+0000","last_active":"2024-09-18T07:50:30.247442+0000","last_peered":"2024-09-18T07:50:30.247442+0000","last_clean":"2024-09-18T07:50:30.247442+0000","last_became_active":"2024-09-18T07:50:23.286252+0000","last_became_peered":"2024-09-18T07:50:23.286252+0000","last_unstale":"2024-09-18T07:50:30.247442+0000","last_undegraded":"2024-09-18T07:50:30.247442+0000","last_fullsized":"2024-09-18T07:50:30.247442+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:08.181420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:08.181420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:29.905624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"323'12","reported_seq":744,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249213+0000","last_change":"2024-09-18T07:48:10.211661+0000","last_active":"2024-09-18T07:50:30.249213+0000","last_peered":"2024-09-18T07:50:30.249213+0000","last_clean":"2024-09-18T07:50:30.249213+0000","last_became_active":"2024-09-18T07:48:10.160725+0000","last_became_peered":"2024-09-18T07:48:10.160725+0000","last_unstale":"2024-09-18T07:50:30.249213+0000","last_undegraded":"2024-09-18T07:50:30.249213+0000","last_fullsized":"2024-09-18T07:50:30.249213+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'12","last_scrub_stamp":"2024-09-18T07:48:03.699388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:03.699388+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":977,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.250764+0000","last_change":"2024-09-18T07:50:23.274124+0000","last_active":"2024-09-18T07:50:30.250764+0000","last_peered":"2024-09-18T07:50:30.250764+0000","last_clean":"2024-09-18T07:50:30.250764+0000","last_became_active":"2024-09-18T07:50:23.273803+0000","last_became_peered":"2024-09-18T07:50:23.273803+0000","last_unstale":"2024-09-18T07:50:30.250764+0000","last_undegraded":"2024-09-18T07:50:30.250764+0000","last_fullsized":"2024-09-18T07:50:30.250764+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:53.089903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.089903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:27.587459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":821,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249615+0000","last_change":"2024-09-18T07:48:56.269806+0000","last_active":"2024-09-18T07:50:30.249615+0000","last_peered":"2024-09-18T07:50:30.249615+0000","last_clean":"2024-09-18T07:50:30.249615+0000","last_became_active":"2024-09-18T07:44:43.782348+0000","last_became_peered":"2024-09-18T07:44:43.782348+0000","last_unstale":"2024-09-18T07:50:30.249615+0000","last_undegraded":"2024-09-18T07:50:30.249615+0000","last_fullsized":"2024-09-18T07:50:30.249615+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:56.269745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:56.269745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"323'15","reported_seq":36,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.251833+0000","last_change":"2024-09-18T07:50:23.335646+0000","last_active":"2024-09-18T07:50:30.251833+0000","last_peered":"2024-09-18T07:50:30.251833+0000","last_clean":"2024-09-18T07:50:30.251833+0000","last_became_active":"2024-09-18T07:50:23.290478+0000","last_became_peered":"2024-09-18T07:50:23.290478+0000","last_unstale":"2024-09-18T07:50:30.251833+0000","last_undegraded":"2024-09-18T07:50:30.251833+0000","last_fullsized":"2024-09-18T07:50:30.251833+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:48:27.802966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:27.802966+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":837,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249512+0000","last_change":"2024-09-18T07:49:45.461696+0000","last_active":"2024-09-18T07:50:30.249512+0000","last_peered":"2024-09-18T07:50:30.249512+0000","last_clean":"2024-09-18T07:50:30.249512+0000","last_became_active":"2024-09-18T07:44:43.782930+0000","last_became_peered":"2024-09-18T07:44:43.782930+0000","last_unstale":"2024-09-18T07:50:30.249512+0000","last_undegraded":"2024-09-18T07:50:30.249512+0000","last_fullsized":"2024-09-18T07:50:30.249512+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:45.461629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:45.461629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:52.011721+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":26,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.251468+0000","last_change":"2024-09-18T07:50:23.274328+0000","last_active":"2024-09-18T07:50:30.251468+0000","last_peered":"2024-09-18T07:50:30.251468+0000","last_clean":"2024-09-18T07:50:30.251468+0000","last_became_active":"2024-09-18T07:50:23.274007+0000","last_became_peered":"2024-09-18T07:50:23.274007+0000","last_unstale":"2024-09-18T07:50:30.251468+0000","last_undegraded":"2024-09-18T07:50:30.251468+0000","last_fullsized":"2024-09-18T07:50:30.251468+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:30.750167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:30.750167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1039,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.246001+0000","last_change":"2024-09-18T07:48:03.866565+0000","last_active":"2024-09-18T07:50:30.246001+0000","last_peered":"2024-09-18T07:50:30.246001+0000","last_clean":"2024-09-18T07:50:30.246001+0000","last_became_active":"2024-09-18T07:44:43.790582+0000","last_became_peered":"2024-09-18T07:44:43.790582+0000","last_unstale":"2024-09-18T07:50:30.246001+0000","last_undegraded":"2024-09-18T07:50:30.246001+0000","last_fullsized":"2024-09-18T07:50:30.246001+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:03.866499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:03.866499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"323'15","reported_seq":763,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249285+0000","last_change":"2024-09-18T07:50:23.338747+0000","last_active":"2024-09-18T07:50:30.249285+0000","last_peered":"2024-09-18T07:50:30.249285+0000","last_clean":"2024-09-18T07:50:30.249285+0000","last_became_active":"2024-09-18T07:50:23.298004+0000","last_became_peered":"2024-09-18T07:50:23.298004+0000","last_unstale":"2024-09-18T07:50:30.249285+0000","last_undegraded":"2024-09-18T07:50:30.249285+0000","last_fullsized":"2024-09-18T07:50:30.249285+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:50:04.470601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:50:04.470601+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:30.138856+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":514,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.252111+0000","last_change":"2024-09-18T07:50:23.297867+0000","last_active":"2024-09-18T07:50:30.252111+0000","last_peered":"2024-09-18T07:50:30.252111+0000","last_clean":"2024-09-18T07:50:30.252111+0000","last_became_active":"2024-09-18T07:50:23.296118+0000","last_became_peered":"2024-09-18T07:50:23.296118+0000","last_unstale":"2024-09-18T07:50:30.252111+0000","last_undegraded":"2024-09-18T07:50:30.252111+0000","last_fullsized":"2024-09-18T07:50:30.252111+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:52.023913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.023913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"548'12","reported_seq":32,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249977+0000","last_change":"2024-09-18T07:50:23.327908+0000","last_active":"2024-09-18T07:50:30.249977+0000","last_peered":"2024-09-18T07:50:30.249977+0000","last_clean":"2024-09-18T07:50:30.249977+0000","last_became_active":"2024-09-18T07:50:23.294281+0000","last_became_peered":"2024-09-18T07:50:23.294281+0000","last_unstale":"2024-09-18T07:50:30.249977+0000","last_undegraded":"2024-09-18T07:50:30.249977+0000","last_fullsized":"2024-09-18T07:50:30.249977+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'11","last_scrub_stamp":"2024-09-18T07:49:50.502186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:49:50.502186+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:09.383930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":926,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.239350+0000","last_change":"2024-09-18T07:47:11.193555+0000","last_active":"2024-09-18T07:50:30.239350+0000","last_peered":"2024-09-18T07:50:30.239350+0000","last_clean":"2024-09-18T07:50:30.239350+0000","last_became_active":"2024-09-18T07:47:11.193273+0000","last_became_peered":"2024-09-18T07:47:11.193273+0000","last_unstale":"2024-09-18T07:50:30.239350+0000","last_undegraded":"2024-09-18T07:50:30.239350+0000","last_fullsized":"2024-09-18T07:50:30.239350+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:48.955463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.955463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"323'20","reported_seq":919,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.250042+0000","last_change":"2024-09-18T07:50:23.343896+0000","last_active":"2024-09-18T07:50:30.250042+0000","last_peered":"2024-09-18T07:50:30.250042+0000","last_clean":"2024-09-18T07:50:30.250042+0000","last_became_active":"2024-09-18T07:50:23.296164+0000","last_became_peered":"2024-09-18T07:50:23.296164+0000","last_unstale":"2024-09-18T07:50:30.250042+0000","last_undegraded":"2024-09-18T07:50:30.250042+0000","last_fullsized":"2024-09-18T07:50:30.250042+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:46:14.650883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:14.650883+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:23.472632+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":823,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.251905+0000","last_change":"2024-09-18T07:49:00.272282+0000","last_active":"2024-09-18T07:50:30.251905+0000","last_peered":"2024-09-18T07:50:30.251905+0000","last_clean":"2024-09-18T07:50:30.251905+0000","last_became_active":"2024-09-18T07:44:43.779305+0000","last_became_peered":"2024-09-18T07:44:43.779305+0000","last_unstale":"2024-09-18T07:50:30.251905+0000","last_undegraded":"2024-09-18T07:50:30.251905+0000","last_fullsized":"2024-09-18T07:50:30.251905+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:00.272210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:00.272210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"548'16","reported_seq":1155,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.251056+0000","last_change":"2024-09-18T07:50:23.304787+0000","last_active":"2024-09-18T07:50:30.251056+0000","last_peered":"2024-09-18T07:50:30.251056+0000","last_clean":"2024-09-18T07:50:30.251056+0000","last_became_active":"2024-09-18T07:50:23.270200+0000","last_became_peered":"2024-09-18T07:50:23.270200+0000","last_unstale":"2024-09-18T07:50:30.251056+0000","last_undegraded":"2024-09-18T07:50:30.251056+0000","last_fullsized":"2024-09-18T07:50:30.251056+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'14","last_scrub_stamp":"2024-09-18T07:50:06.474533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:50:06.474533+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:08.100014+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":502,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.251155+0000","last_change":"2024-09-18T07:50:23.295756+0000","last_active":"2024-09-18T07:50:30.251155+0000","last_peered":"2024-09-18T07:50:30.251155+0000","last_clean":"2024-09-18T07:50:30.251155+0000","last_became_active":"2024-09-18T07:50:23.295431+0000","last_became_peered":"2024-09-18T07:50:23.295431+0000","last_unstale":"2024-09-18T07:50:30.251155+0000","last_undegraded":"2024-09-18T07:50:30.251155+0000","last_fullsized":"2024-09-18T07:50:30.251155+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:45.971311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:45.971311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"323'19","reported_seq":1069,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.239023+0000","last_change":"2024-09-18T07:47:20.196648+0000","last_active":"2024-09-18T07:50:30.239023+0000","last_peered":"2024-09-18T07:50:30.239023+0000","last_clean":"2024-09-18T07:50:30.239023+0000","last_became_active":"2024-09-18T07:47:20.140070+0000","last_became_peered":"2024-09-18T07:47:20.140070+0000","last_unstale":"2024-09-18T07:50:30.239023+0000","last_undegraded":"2024-09-18T07:50:30.239023+0000","last_fullsized":"2024-09-18T07:50:30.239023+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:41:54.022421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.022421+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":983,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.239200+0000","last_change":"2024-09-18T07:46:06.964899+0000","last_active":"2024-09-18T07:50:30.239200+0000","last_peered":"2024-09-18T07:50:30.239200+0000","last_clean":"2024-09-18T07:50:30.239200+0000","last_became_active":"2024-09-18T07:44:43.780305+0000","last_became_peered":"2024-09-18T07:44:43.780305+0000","last_unstale":"2024-09-18T07:50:30.239200+0000","last_undegraded":"2024-09-18T07:50:30.239200+0000","last_fullsized":"2024-09-18T07:50:30.239200+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:06.964847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:06.964847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:54.987085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":655,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.248330+0000","last_change":"2024-09-18T07:48:10.163971+0000","last_active":"2024-09-18T07:50:30.248330+0000","last_peered":"2024-09-18T07:50:30.248330+0000","last_clean":"2024-09-18T07:50:30.248330+0000","last_became_active":"2024-09-18T07:48:10.163110+0000","last_became_peered":"2024-09-18T07:48:10.163110+0000","last_unstale":"2024-09-18T07:50:30.248330+0000","last_undegraded":"2024-09-18T07:50:30.248330+0000","last_fullsized":"2024-09-18T07:50:30.248330+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:59.022075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:59.022075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:18.612591+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":451,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.242487+0000","last_change":"2024-09-18T07:50:14.220470+0000","last_active":"2024-09-18T07:50:30.242487+0000","last_peered":"2024-09-18T07:50:30.242487+0000","last_clean":"2024-09-18T07:50:30.242487+0000","last_became_active":"2024-09-18T07:50:14.220062+0000","last_became_peered":"2024-09-18T07:50:14.220062+0000","last_unstale":"2024-09-18T07:50:30.242487+0000","last_undegraded":"2024-09-18T07:50:30.242487+0000","last_fullsized":"2024-09-18T07:50:30.242487+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:25.249102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:25.249102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"323'15","reported_seq":723,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.244627+0000","last_change":"2024-09-18T07:49:32.696134+0000","last_active":"2024-09-18T07:50:30.244627+0000","last_peered":"2024-09-18T07:50:30.244627+0000","last_clean":"2024-09-18T07:50:30.244627+0000","last_became_active":"2024-09-18T07:49:32.687721+0000","last_became_peered":"2024-09-18T07:49:32.687721+0000","last_unstale":"2024-09-18T07:50:30.244627+0000","last_undegraded":"2024-09-18T07:50:30.244627+0000","last_fullsized":"2024-09-18T07:50:30.244627+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:47:49.411931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:49.411931+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:58.912624+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":836,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249532+0000","last_change":"2024-09-18T07:50:23.298146+0000","last_active":"2024-09-18T07:50:30.249532+0000","last_peered":"2024-09-18T07:50:30.249532+0000","last_clean":"2024-09-18T07:50:30.249532+0000","last_became_active":"2024-09-18T07:50:23.296874+0000","last_became_peered":"2024-09-18T07:50:23.296874+0000","last_unstale":"2024-09-18T07:50:30.249532+0000","last_undegraded":"2024-09-18T07:50:30.249532+0000","last_fullsized":"2024-09-18T07:50:30.249532+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:33.013808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:33.013808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":26,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.250057+0000","last_change":"2024-09-18T07:50:23.299695+0000","last_active":"2024-09-18T07:50:30.250057+0000","last_peered":"2024-09-18T07:50:30.250057+0000","last_clean":"2024-09-18T07:50:30.250057+0000","last_became_active":"2024-09-18T07:50:23.298745+0000","last_became_peered":"2024-09-18T07:50:23.298745+0000","last_unstale":"2024-09-18T07:50:30.250057+0000","last_undegraded":"2024-09-18T07:50:30.250057+0000","last_fullsized":"2024-09-18T07:50:30.250057+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:13.911901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:13.911901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":914,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.247105+0000","last_change":"2024-09-18T07:50:14.217318+0000","last_active":"2024-09-18T07:50:30.247105+0000","last_peered":"2024-09-18T07:50:30.247105+0000","last_clean":"2024-09-18T07:50:30.247105+0000","last_became_active":"2024-09-18T07:50:14.217052+0000","last_became_peered":"2024-09-18T07:50:14.217052+0000","last_unstale":"2024-09-18T07:50:30.247105+0000","last_undegraded":"2024-09-18T07:50:30.247105+0000","last_fullsized":"2024-09-18T07:50:30.247105+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:44.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:44.214865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"323'19","reported_seq":38,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.250546+0000","last_change":"2024-09-18T07:50:23.451303+0000","last_active":"2024-09-18T07:50:30.250546+0000","last_peered":"2024-09-18T07:50:30.250546+0000","last_clean":"2024-09-18T07:50:30.250546+0000","last_became_active":"2024-09-18T07:50:23.389095+0000","last_became_peered":"2024-09-18T07:50:23.389095+0000","last_unstale":"2024-09-18T07:50:30.250546+0000","last_undegraded":"2024-09-18T07:50:30.250546+0000","last_fullsized":"2024-09-18T07:50:30.250546+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:48:58.276659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:58.276659+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":968,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.250819+0000","last_change":"2024-09-18T07:47:52.070685+0000","last_active":"2024-09-18T07:50:30.250819+0000","last_peered":"2024-09-18T07:50:30.250819+0000","last_clean":"2024-09-18T07:50:30.250819+0000","last_became_active":"2024-09-18T07:47:20.140495+0000","last_became_peered":"2024-09-18T07:47:20.140495+0000","last_unstale":"2024-09-18T07:50:30.250819+0000","last_undegraded":"2024-09-18T07:50:30.250819+0000","last_fullsized":"2024-09-18T07:50:30.250819+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:52.070602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.070602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:54.598914+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":829,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.252544+0000","last_change":"2024-09-18T07:46:13.660302+0000","last_active":"2024-09-18T07:50:30.252544+0000","last_peered":"2024-09-18T07:50:30.252544+0000","last_clean":"2024-09-18T07:50:30.252544+0000","last_became_active":"2024-09-18T07:44:43.778318+0000","last_became_peered":"2024-09-18T07:44:43.778318+0000","last_unstale":"2024-09-18T07:50:30.252544+0000","last_undegraded":"2024-09-18T07:50:30.252544+0000","last_fullsized":"2024-09-18T07:50:30.252544+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:13.660254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:13.660254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:01.470078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":26,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.249956+0000","last_change":"2024-09-18T07:50:23.295263+0000","last_active":"2024-09-18T07:50:30.249956+0000","last_peered":"2024-09-18T07:50:30.249956+0000","last_clean":"2024-09-18T07:50:30.249956+0000","last_became_active":"2024-09-18T07:50:23.294923+0000","last_became_peered":"2024-09-18T07:50:23.294923+0000","last_unstale":"2024-09-18T07:50:30.249956+0000","last_undegraded":"2024-09-18T07:50:30.249956+0000","last_fullsized":"2024-09-18T07:50:30.249956+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:33.311735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:33.311735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"323'8","reported_seq":32,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.255066+0000","last_change":"2024-09-18T07:50:23.399317+0000","last_active":"2024-09-18T07:50:30.255066+0000","last_peered":"2024-09-18T07:50:30.255066+0000","last_clean":"2024-09-18T07:50:30.255066+0000","last_became_active":"2024-09-18T07:50:23.298033+0000","last_became_peered":"2024-09-18T07:50:23.298033+0000","last_unstale":"2024-09-18T07:50:30.255066+0000","last_undegraded":"2024-09-18T07:50:30.255066+0000","last_fullsized":"2024-09-18T07:50:30.255066+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'8","last_scrub_stamp":"2024-09-18T07:47:46.405595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:46.405595+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":236,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.252900+0000","last_change":"2024-09-18T07:50:23.285262+0000","last_active":"2024-09-18T07:50:30.252900+0000","last_peered":"2024-09-18T07:50:30.252900+0000","last_clean":"2024-09-18T07:50:30.252900+0000","last_became_active":"2024-09-18T07:50:23.284906+0000","last_became_peered":"2024-09-18T07:50:23.284906+0000","last_unstale":"2024-09-18T07:50:30.252900+0000","last_undegraded":"2024-09-18T07:50:30.252900+0000","last_fullsized":"2024-09-18T07:50:30.252900+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:15.549558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:50:15.549558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:15.993742+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":667,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.247602+0000","last_change":"2024-09-18T07:47:55.363130+0000","last_active":"2024-09-18T07:50:30.247602+0000","last_peered":"2024-09-18T07:50:30.247602+0000","last_clean":"2024-09-18T07:50:30.247602+0000","last_became_active":"2024-09-18T07:44:43.792641+0000","last_became_peered":"2024-09-18T07:44:43.792641+0000","last_unstale":"2024-09-18T07:50:30.247602+0000","last_undegraded":"2024-09-18T07:50:30.247602+0000","last_fullsized":"2024-09-18T07:50:30.247602+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:55.363087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:55.363087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":815,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.252340+0000","last_change":"2024-09-18T07:49:49.436754+0000","last_active":"2024-09-18T07:50:30.252340+0000","last_peered":"2024-09-18T07:50:30.252340+0000","last_clean":"2024-09-18T07:50:30.252340+0000","last_became_active":"2024-09-18T07:44:43.782378+0000","last_became_peered":"2024-09-18T07:44:43.782378+0000","last_unstale":"2024-09-18T07:50:30.252340+0000","last_undegraded":"2024-09-18T07:50:30.252340+0000","last_fullsized":"2024-09-18T07:50:30.252340+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:49.436635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:49.436635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:52.792800+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":613,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.251158+0000","last_change":"2024-09-18T07:49:17.654443+0000","last_active":"2024-09-18T07:50:30.251158+0000","last_peered":"2024-09-18T07:50:30.251158+0000","last_clean":"2024-09-18T07:50:30.251158+0000","last_became_active":"2024-09-18T07:47:20.138598+0000","last_became_peered":"2024-09-18T07:47:20.138598+0000","last_unstale":"2024-09-18T07:50:30.251158+0000","last_undegraded":"2024-09-18T07:50:30.251158+0000","last_fullsized":"2024-09-18T07:50:30.251158+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:17.654398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:17.654398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:37.045609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":30,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.254241+0000","last_change":"2024-09-18T07:50:23.402013+0000","last_active":"2024-09-18T07:50:30.254241+0000","last_peered":"2024-09-18T07:50:30.254241+0000","last_clean":"2024-09-18T07:50:30.254241+0000","last_became_active":"2024-09-18T07:50:23.298899+0000","last_became_peered":"2024-09-18T07:50:23.298899+0000","last_unstale":"2024-09-18T07:50:30.254241+0000","last_undegraded":"2024-09-18T07:50:30.254241+0000","last_fullsized":"2024-09-18T07:50:30.254241+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:41.464337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:41.464337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"323'19","reported_seq":736,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.245211+0000","last_change":"2024-09-18T07:47:47.469967+0000","last_active":"2024-09-18T07:50:30.245211+0000","last_peered":"2024-09-18T07:50:30.245211+0000","last_clean":"2024-09-18T07:50:30.245211+0000","last_became_active":"2024-09-18T07:47:20.142007+0000","last_became_peered":"2024-09-18T07:47:20.142007+0000","last_unstale":"2024-09-18T07:50:30.245211+0000","last_undegraded":"2024-09-18T07:50:30.245211+0000","last_fullsized":"2024-09-18T07:50:30.245211+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:47:47.469902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:47.469902+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":570,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-09-18T07:50:30.251068+0000","last_change":"2024-09-18T07:48:28.910567+0000","last_active":"2024-09-18T07:50:30.251068+0000","last_peered":"2024-09-18T07:50:30.251068+0000","last_clean":"2024-09-18T07:50:30.251068+0000","last_became_active":"2024-09-18T07:47:20.139668+0000","last_became_peered":"2024-09-18T07:47:20.139668+0000","last_unstale":"2024-09-18T07:50:30.251068+0000","last_undegraded":"2024-09-18T07:50:30.251068+0000","last_fullsized":"2024-09-18T07:50:30.251068+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:28.910419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:28.910419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":526057496,"num_objects":8028,"num_object_clones":0,"num_object_copies":24084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2045,"num_objects_unfound":0,"num_objects_dirty":8028,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":216276,"num_write_kb":7177665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46922,"num_bytes_recovered":3074621440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2444505088,"data_stored":2444492872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68927,"ondisk_log_size":68927,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1096,"num_read_kb":853,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":649,"num_bytes_recovered":14154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},"log_size":691,"ondisk_log_size":691,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":6406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":925,"num_read_kb":920,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1983,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201519,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":966000,"kb_used_data":240672,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92779152,"statfs":{"total":95995035648,"available":95005851648,"internally_reserved":0,"allocated":246448128,"data_stored":245985716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":742717898},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463157,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1006148,"kb_used_data":266932,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":92739004,"statfs":{"total":95995035648,"available":94964740096,"internally_reserved":0,"allocated":273338368,"data_stored":272918919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":756939206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":460,"seq":1975684956202,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":260368,"kb_used_data":97408,"kb_used_omap":6,"kb_used_meta":162937,"kb_avail":93484784,"statfs":{"total":95995035648,"available":95728418816,"internally_reserved":0,"allocated":99745792,"data_stored":99315577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6877,"internal_metadata":166847779},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":395,"seq":1696512081991,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":700004,"kb_used_data":185044,"kb_used_omap":4,"kb_used_meta":514939,"kb_avail":93045148,"statfs":{"total":95995035648,"available":95278231552,"internally_reserved":0,"allocated":189485056,"data_stored":189033977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4447,"internal_metadata":527298209},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149957,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1712952,"kb_used_data":861288,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92032200,"statfs":{"total":95995035648,"available":94240972800,"internally_reserved":0,"allocated":881958912,"data_stored":881489228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":872085959},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411595,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1108616,"kb_used_data":341440,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92636536,"statfs":{"total":95995035648,"available":94859812864,"internally_reserved":0,"allocated":349634560,"data_stored":349204825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":785381829},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575122,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1147232,"kb_used_data":295568,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92597920,"statfs":{"total":95995035648,"available":94820270080,"internally_reserved":0,"allocated":302661632,"data_stored":302151468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738647,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":212926464,"data_stored":212926464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282525696,"data_stored":282525696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319033344,"data_stored":319029272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":879886336,"data_stored":879886336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":173150208,"data_stored":173146136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":86048768,"data_stored":86048768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":255528960,"data_stored":255524888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":235405312,"data_stored":235405312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:50:39.940 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-18T07:50:39.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_4 pg_num 26 2024-09-18T07:50:40.009 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:40.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:40.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:40.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:40.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:40.159 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:40.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:40.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:40.261 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:40.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:40.285 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:40.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:40.370 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-18T07:50:40.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:40.458 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:40.466 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:40.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:40.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:40.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:40.581 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:40 smithi137 ceph-mon[29408]: 11.7 scrub starts 2024-09-18T07:50:40.581 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:40 smithi137 ceph-mon[29408]: 11.7 scrub ok 2024-09-18T07:50:40.581 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:40 smithi137 ceph-mon[29408]: 5.d scrub starts 2024-09-18T07:50:40.581 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:40 smithi137 ceph-mon[29408]: 5.d scrub ok 2024-09-18T07:50:40.582 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:40 smithi137 ceph-mon[29408]: osdmap e555: 8 total, 8 up, 7 in 2024-09-18T07:50:40.582 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:40 smithi137 ceph-mon[39175]: 11.7 scrub starts 2024-09-18T07:50:40.582 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:40 smithi137 ceph-mon[39175]: 11.7 scrub ok 2024-09-18T07:50:40.582 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:40 smithi137 ceph-mon[39175]: 5.d scrub starts 2024-09-18T07:50:40.582 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:40 smithi137 ceph-mon[39175]: 5.d scrub ok 2024-09-18T07:50:40.582 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:40 smithi137 ceph-mon[39175]: osdmap e555: 8 total, 8 up, 7 in 2024-09-18T07:50:40.589 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:40.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:40.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:40.691 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:40.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:40 smithi203 ceph-mon[32237]: 11.7 scrub starts 2024-09-18T07:50:40.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:40 smithi203 ceph-mon[32237]: 11.7 scrub ok 2024-09-18T07:50:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:40 smithi203 ceph-mon[32237]: 5.d scrub starts 2024-09-18T07:50:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:40 smithi203 ceph-mon[32237]: 5.d scrub ok 2024-09-18T07:50:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:40 smithi203 ceph-mon[32237]: osdmap e555: 8 total, 8 up, 7 in 2024-09-18T07:50:40.717 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:40.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:40.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:40.819 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:40.845 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:40.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:40.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:40.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:40.969 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:40.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:40.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:40.986 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:41.022 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:41.071 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:41.097 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:41.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:41.199 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:41.223 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:41.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:41.324 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:41.336 INFO:teuthology.orchestra.run.smithi137.stderr:set pool 11 pg_num to 26 2024-09-18T07:50:41.347 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:41.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:41.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:41.383 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:41 smithi137 ceph-mon[29408]: 3.2 scrub starts 2024-09-18T07:50:41.383 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:41 smithi137 ceph-mon[29408]: 3.2 scrub ok 2024-09-18T07:50:41.383 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:41 smithi137 ceph-mon[29408]: pgmap v1062: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 131 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 491/6843 objects misplaced (7.175%); 682 KiB/s, 10 objects/s recovering 2024-09-18T07:50:41.383 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:41 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4037402839' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-18T07:50:41.383 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:41 smithi137 ceph-mon[39175]: 3.2 scrub starts 2024-09-18T07:50:41.384 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:41 smithi137 ceph-mon[39175]: 3.2 scrub ok 2024-09-18T07:50:41.384 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:41 smithi137 ceph-mon[39175]: pgmap v1062: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 131 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 491/6843 objects misplaced (7.175%); 682 KiB/s, 10 objects/s recovering 2024-09-18T07:50:41.384 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:41 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4037402839' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-18T07:50:41.449 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:41.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:41 smithi203 ceph-mon[32237]: 3.2 scrub starts 2024-09-18T07:50:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:41 smithi203 ceph-mon[32237]: 3.2 scrub ok 2024-09-18T07:50:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:41 smithi203 ceph-mon[32237]: pgmap v1062: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 131 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 491/6843 objects misplaced (7.175%); 682 KiB/s, 10 objects/s recovering 2024-09-18T07:50:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:41 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4037402839' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-18T07:50:41.471 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:41.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:41.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:41.573 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:41.596 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:41.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:41.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:41.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:41.719 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:41.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:41.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:41.843 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:41.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:41.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:41.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:41.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:41.968 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:41.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:42.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:42.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:42.092 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:42.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:42.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:42.215 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:42.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:42.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:42.305 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:50:42.305 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:50:42.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:42.341 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:42.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:42.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:42.396 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-18T07:50:42.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-18T07:50:42.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:50:42.443 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:42.488 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:42.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:42.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:42.515 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:42 smithi137 ceph-mon[29408]: 11.d scrub starts 2024-09-18T07:50:42.515 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:42 smithi137 ceph-mon[29408]: 11.d scrub ok 2024-09-18T07:50:42.515 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:42 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4037402839' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-18T07:50:42.515 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:42 smithi137 ceph-mon[29408]: osdmap e556: 8 total, 8 up, 7 in 2024-09-18T07:50:42.515 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:42 smithi137 ceph-mon[29408]: 6.a scrub starts 2024-09-18T07:50:42.515 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:42 smithi137 ceph-mon[29408]: 6.a scrub ok 2024-09-18T07:50:42.515 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:42 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-18T07:50:42.516 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:42 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-18T07:50:42.516 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:42 smithi137 ceph-mon[39175]: 11.d scrub starts 2024-09-18T07:50:42.516 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:42 smithi137 ceph-mon[39175]: 11.d scrub ok 2024-09-18T07:50:42.516 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:42 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4037402839' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-18T07:50:42.516 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:42 smithi137 ceph-mon[39175]: osdmap e556: 8 total, 8 up, 7 in 2024-09-18T07:50:42.516 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:42 smithi137 ceph-mon[39175]: 6.a scrub starts 2024-09-18T07:50:42.516 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:42 smithi137 ceph-mon[39175]: 6.a scrub ok 2024-09-18T07:50:42.516 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:42 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-18T07:50:42.516 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:42 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-18T07:50:42.590 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:42.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:42.614 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:42.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:42 smithi203 ceph-mon[32237]: 11.d scrub starts 2024-09-18T07:50:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:42 smithi203 ceph-mon[32237]: 11.d scrub ok 2024-09-18T07:50:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:42 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4037402839' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-18T07:50:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:42 smithi203 ceph-mon[32237]: osdmap e556: 8 total, 8 up, 7 in 2024-09-18T07:50:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:42 smithi203 ceph-mon[32237]: 6.a scrub starts 2024-09-18T07:50:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:42 smithi203 ceph-mon[32237]: 6.a scrub ok 2024-09-18T07:50:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:42 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-18T07:50:42.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:42 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-18T07:50:42.715 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:42.737 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:42.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:42.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:42.839 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:42.861 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:42.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:42.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:42.963 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:42.987 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:42.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:42.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:43.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:43.112 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:43.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:43.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:43.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:43.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:43.236 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:43.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:43.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:43.368 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:43.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:43.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:43 smithi137 ceph-mon[29408]: pgmap v1064: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 89 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 537 op/s; 319/4821 objects misplaced (6.617%) 2024-09-18T07:50:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:43 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-18T07:50:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:43 smithi137 ceph-mon[29408]: osdmap e557: 8 total, 8 up, 7 in 2024-09-18T07:50:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:43 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:50:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:43 smithi137 ceph-mon[39175]: pgmap v1064: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 89 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 537 op/s; 319/4821 objects misplaced (6.617%) 2024-09-18T07:50:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:43 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-18T07:50:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:43 smithi137 ceph-mon[39175]: osdmap e557: 8 total, 8 up, 7 in 2024-09-18T07:50:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:43 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:50:43.471 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:43.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:43.493 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:43.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:43.594 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:43.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:43.618 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:43.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:43.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:43 smithi203 ceph-mon[32237]: pgmap v1064: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 89 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 537 op/s; 319/4821 objects misplaced (6.617%) 2024-09-18T07:50:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:43 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-18T07:50:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:43 smithi203 ceph-mon[32237]: osdmap e557: 8 total, 8 up, 7 in 2024-09-18T07:50:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:43 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:50:43.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:43.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:43.743 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:43.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:43.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-18T07:50:43.845 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:43.869 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:43.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:43.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:43.971 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:43.993 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:43.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:43.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:44.094 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:44.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:44.135 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:44.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:44.236 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:44.246 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:44.258 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:44.259 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:50:44.259 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:50:44.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:44.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:44.277 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:50:44.277 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:50:44.296 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:50:44.296 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:50:44.299 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:44.314 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:50:44.314 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:50:44.334 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:50:44.334 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:50:44.351 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:50:44.352 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:50:44.360 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:44.377 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:50:44.377 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:50:44.385 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:44.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:44.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:44.394 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:50:44.394 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:50:44.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:44.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:44.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:44.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:44.558 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[29408]: osdmap e558: 8 total, 8 up, 7 in 2024-09-18T07:50:44.558 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[29408]: 11.2 continuing backfill to osd.3 from (511'8755,555'13746] MIN to 555'13746 2024-09-18T07:50:44.558 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[29408]: 11.4 scrub starts 2024-09-18T07:50:44.558 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[29408]: 11.4 scrub ok 2024-09-18T07:50:44.559 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[29408]: 11.7 scrub starts 2024-09-18T07:50:44.559 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[29408]: 11.7 scrub ok 2024-09-18T07:50:44.559 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[29408]: 11.8 scrub starts 2024-09-18T07:50:44.559 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[29408]: 11.8 scrub ok 2024-09-18T07:50:44.559 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[29408]: 11.0 scrub starts 2024-09-18T07:50:44.559 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[29408]: 11.0 scrub ok 2024-09-18T07:50:44.559 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-18T07:50:44.559 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-18T07:50:44.559 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4185862434' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:44.559 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[39175]: osdmap e558: 8 total, 8 up, 7 in 2024-09-18T07:50:44.560 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[39175]: 11.2 continuing backfill to osd.3 from (511'8755,555'13746] MIN to 555'13746 2024-09-18T07:50:44.560 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[39175]: 11.4 scrub starts 2024-09-18T07:50:44.560 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[39175]: 11.4 scrub ok 2024-09-18T07:50:44.560 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[39175]: 11.7 scrub starts 2024-09-18T07:50:44.560 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[39175]: 11.7 scrub ok 2024-09-18T07:50:44.560 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[39175]: 11.8 scrub starts 2024-09-18T07:50:44.560 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[39175]: 11.8 scrub ok 2024-09-18T07:50:44.560 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[39175]: 11.0 scrub starts 2024-09-18T07:50:44.560 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[39175]: 11.0 scrub ok 2024-09-18T07:50:44.560 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-18T07:50:44.560 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-18T07:50:44.560 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4185862434' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:44.611 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:44.632 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:44.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:44.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:44 smithi203 ceph-mon[32237]: osdmap e558: 8 total, 8 up, 7 in 2024-09-18T07:50:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:44 smithi203 ceph-mon[32237]: 11.2 continuing backfill to osd.3 from (511'8755,555'13746] MIN to 555'13746 2024-09-18T07:50:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:44 smithi203 ceph-mon[32237]: 11.4 scrub starts 2024-09-18T07:50:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:44 smithi203 ceph-mon[32237]: 11.4 scrub ok 2024-09-18T07:50:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:44 smithi203 ceph-mon[32237]: 11.7 scrub starts 2024-09-18T07:50:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:44 smithi203 ceph-mon[32237]: 11.7 scrub ok 2024-09-18T07:50:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:44 smithi203 ceph-mon[32237]: 11.8 scrub starts 2024-09-18T07:50:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:44 smithi203 ceph-mon[32237]: 11.8 scrub ok 2024-09-18T07:50:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:44 smithi203 ceph-mon[32237]: 11.0 scrub starts 2024-09-18T07:50:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:44 smithi203 ceph-mon[32237]: 11.0 scrub ok 2024-09-18T07:50:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:44 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-18T07:50:44.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:44 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-18T07:50:44.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:44 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4185862434' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:44.734 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:44.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:44.757 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:44.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:44.858 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:44.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:44.884 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:44.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:44.985 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:45.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:45.008 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:45.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:45.109 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:45.130 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:45.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:45.232 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:45.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:45.255 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:45.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:45.357 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:45.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:45.380 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:45.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:45.421 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:45.481 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:45.502 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:45.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:45.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:45.542 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[29408]: pgmap v1067: 168 pgs: 3 peering, 10 unknown, 3 active+remapped+backfill_toofull, 152 active+clean; 1.5 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:50:45.542 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[29408]: 11.6 scrub starts 2024-09-18T07:50:45.542 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[29408]: 11.6 scrub ok 2024-09-18T07:50:45.542 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-18T07:50:45.542 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[29408]: osdmap e559: 8 total, 8 up, 7 in 2024-09-18T07:50:45.542 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[29408]: 11.5 scrub starts 2024-09-18T07:50:45.543 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[29408]: 11.5 scrub ok 2024-09-18T07:50:45.543 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[29408]: 11.1 scrub starts 2024-09-18T07:50:45.543 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[29408]: 11.1 scrub ok 2024-09-18T07:50:45.543 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[29408]: 11.9 scrub starts 2024-09-18T07:50:45.543 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[29408]: 11.9 scrub ok 2024-09-18T07:50:45.543 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[29408]: osdmap e560: 8 total, 8 up, 7 in 2024-09-18T07:50:45.543 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[39175]: pgmap v1067: 168 pgs: 3 peering, 10 unknown, 3 active+remapped+backfill_toofull, 152 active+clean; 1.5 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:50:45.544 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[39175]: 11.6 scrub starts 2024-09-18T07:50:45.544 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[39175]: 11.6 scrub ok 2024-09-18T07:50:45.544 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-18T07:50:45.544 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[39175]: osdmap e559: 8 total, 8 up, 7 in 2024-09-18T07:50:45.544 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[39175]: 11.5 scrub starts 2024-09-18T07:50:45.544 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[39175]: 11.5 scrub ok 2024-09-18T07:50:45.544 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[39175]: 11.1 scrub starts 2024-09-18T07:50:45.544 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[39175]: 11.1 scrub ok 2024-09-18T07:50:45.544 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[39175]: 11.9 scrub starts 2024-09-18T07:50:45.544 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[39175]: 11.9 scrub ok 2024-09-18T07:50:45.544 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:45 smithi137 ceph-mon[39175]: osdmap e560: 8 total, 8 up, 7 in 2024-09-18T07:50:45.604 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:45.625 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:45.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:45.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:45 smithi203 ceph-mon[32237]: pgmap v1067: 168 pgs: 3 peering, 10 unknown, 3 active+remapped+backfill_toofull, 152 active+clean; 1.5 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:50:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:45 smithi203 ceph-mon[32237]: 11.6 scrub starts 2024-09-18T07:50:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:45 smithi203 ceph-mon[32237]: 11.6 scrub ok 2024-09-18T07:50:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:45 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-18T07:50:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:45 smithi203 ceph-mon[32237]: osdmap e559: 8 total, 8 up, 7 in 2024-09-18T07:50:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:45 smithi203 ceph-mon[32237]: 11.5 scrub starts 2024-09-18T07:50:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:45 smithi203 ceph-mon[32237]: 11.5 scrub ok 2024-09-18T07:50:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:45 smithi203 ceph-mon[32237]: 11.1 scrub starts 2024-09-18T07:50:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:45 smithi203 ceph-mon[32237]: 11.1 scrub ok 2024-09-18T07:50:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:45 smithi203 ceph-mon[32237]: 11.9 scrub starts 2024-09-18T07:50:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:45 smithi203 ceph-mon[32237]: 11.9 scrub ok 2024-09-18T07:50:45.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:45 smithi203 ceph-mon[32237]: osdmap e560: 8 total, 8 up, 7 in 2024-09-18T07:50:45.727 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:45.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:45.769 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:45.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:45.871 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:45.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:45.893 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:45.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:45.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:46.015 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:46.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:46.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:46.118 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:46.140 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:46.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:46.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:46.193 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:50:46.207 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'unique_pool_4' removed 2024-09-18T07:50:46.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:46.275 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:46.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:46.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:46.399 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:46.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:46.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:46.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[29408]: 11.2 scrub starts 2024-09-18T07:50:46.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[29408]: 11.2 scrub ok 2024-09-18T07:50:46.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3607675038' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:50:46.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[29408]: 11.3 scrub starts 2024-09-18T07:50:46.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[29408]: 11.3 scrub ok 2024-09-18T07:50:46.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[29408]: 4.e scrub starts 2024-09-18T07:50:46.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[29408]: 4.e scrub ok 2024-09-18T07:50:46.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1404526460' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:50:46.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[29408]: from='client.36785 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:50:46.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:50:46.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3607675038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:50:46.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[29408]: from='client.36785 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T07:50:46.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[29408]: osdmap e561: 8 total, 8 up, 7 in 2024-09-18T07:50:46.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[39175]: 11.2 scrub starts 2024-09-18T07:50:46.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[39175]: 11.2 scrub ok 2024-09-18T07:50:46.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3607675038' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:50:46.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[39175]: 11.3 scrub starts 2024-09-18T07:50:46.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[39175]: 11.3 scrub ok 2024-09-18T07:50:46.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[39175]: 4.e scrub starts 2024-09-18T07:50:46.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[39175]: 4.e scrub ok 2024-09-18T07:50:46.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1404526460' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:50:46.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[39175]: from='client.36785 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:50:46.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:50:46.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3607675038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:50:46.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[39175]: from='client.36785 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T07:50:46.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:46 smithi137 ceph-mon[39175]: osdmap e561: 8 total, 8 up, 7 in 2024-09-18T07:50:46.501 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:46.523 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:46.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:46.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:46.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:46.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:46.648 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:46.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:46.651 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:50:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-18T07:50:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-18T07:50:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-18T07:50:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:71849 10.2 e374/364 10:41855215:::benchmark_data_smithi137_151160_object71848:head v 374'4514, mlcod=374'4512)", 2024-09-18T07:50:46.651 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:46.388333+0000", 2024-09-18T07:50:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 420.259612903, 2024-09-18T07:50:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.225317779, 2024-09-18T07:50:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388333+0000", 2024-09-18T07:50:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388333+0000", 2024-09-18T07:50:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388335+0000", 2024-09-18T07:50:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7910000000000001e-06 2024-09-18T07:50:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388381+0000", 2024-09-18T07:50:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5909999999999999e-05 2024-09-18T07:50:46.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388394+0000", 2024-09-18T07:50:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2599000000000001e-05 2024-09-18T07:50:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.388402+0000", 2024-09-18T07:50:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2060000000000007e-06 2024-09-18T07:50:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606027+0000", 2024-09-18T07:50:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.21762539 2024-09-18T07:50:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606060+0000", 2024-09-18T07:50:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2991000000000003e-05 2024-09-18T07:50:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.613606+0000", 2024-09-18T07:50:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0075457730000000004 2024-09-18T07:50:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.613651+0000", 2024-09-18T07:50:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5118999999999999e-05 2024-09-18T07:50:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:71852 10.b e374/364 10:dda9ad8f:::benchmark_data_smithi137_151160_object71851:head v 374'4479, mlcod=374'4476)", 2024-09-18T07:50:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:46.390175+0000", 2024-09-18T07:50:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 420.25777117299998, 2024-09-18T07:50:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.22548416099999999, 2024-09-18T07:50:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.656 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.390175+0000", 2024-09-18T07:50:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.390175+0000", 2024-09-18T07:50:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.390177+0000", 2024-09-18T07:50:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.548e-06 2024-09-18T07:50:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.390217+0000", 2024-09-18T07:50:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0203000000000001e-05 2024-09-18T07:50:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606008+0000", 2024-09-18T07:50:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.21579153700000001 2024-09-18T07:50:46.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606020+0000", 2024-09-18T07:50:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1741e-05 2024-09-18T07:50:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606089+0000", 2024-09-18T07:50:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9058000000000002e-05 2024-09-18T07:50:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.659 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606112+0000", 2024-09-18T07:50:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2410999999999999e-05 2024-09-18T07:50:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.615499+0000", 2024-09-18T07:50:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093874549999999994 2024-09-18T07:50:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.615659+0000", 2024-09-18T07:50:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016020800000000001 2024-09-18T07:50:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.660 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.36014.0:71850 10.2 e374/364 10:48354359:::benchmark_data_smithi137_151160_object71849:head v 374'4515, mlcod=374'4512)", 2024-09-18T07:50:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:43:46.392229+0000", 2024-09-18T07:50:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 420.25571758799998, 2024-09-18T07:50:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.221450217, 2024-09-18T07:50:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.392229+0000", 2024-09-18T07:50:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.392229+0000", 2024-09-18T07:50:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.392231+0000", 2024-09-18T07:50:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0109999999999999e-06 2024-09-18T07:50:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.392282+0000", 2024-09-18T07:50:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1019000000000001e-05 2024-09-18T07:50:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.605989+0000", 2024-09-18T07:50:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.213706807 2024-09-18T07:50:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.606006+0000", 2024-09-18T07:50:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7289e-05 2024-09-18T07:50:46.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.608388+0000", 2024-09-18T07:50:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023825399999999998 2024-09-18T07:50:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.608412+0000", 2024-09-18T07:50:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4093000000000001e-05 2024-09-18T07:50:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.613656+0000", 2024-09-18T07:50:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052432060000000003 2024-09-18T07:50:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:43:46.613679+0000", 2024-09-18T07:50:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3252e-05 2024-09-18T07:50:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:123513 11.f e505/474 11:f8362444:::benchmark_data_smithi137_181069_object11359:head v 505'7753, mlcod=505'7751)", 2024-09-18T07:50:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:56.195354+0000", 2024-09-18T07:50:46.665 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 110.45259223799999, 2024-09-18T07:50:46.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27443678300000002, 2024-09-18T07:50:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.195354+0000", 2024-09-18T07:50:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.195354+0000", 2024-09-18T07:50:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.195356+0000", 2024-09-18T07:50:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6610000000000001e-06 2024-09-18T07:50:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.195363+0000", 2024-09-18T07:50:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7519999999999996e-06 2024-09-18T07:50:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.197743+0000", 2024-09-18T07:50:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023797219999999999 2024-09-18T07:50:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.197751+0000", 2024-09-18T07:50:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7260000000000002e-06 2024-09-18T07:50:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.197953+0000", 2024-09-18T07:50:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000202086 2024-09-18T07:50:46.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.197972+0000", 2024-09-18T07:50:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9351000000000002e-05 2024-09-18T07:50:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.469761+0000", 2024-09-18T07:50:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.271789102 2024-09-18T07:50:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.469791+0000", 2024-09-18T07:50:46.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9383e-05 2024-09-18T07:50:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:123515 11.f e505/474 11:fdda5828:::benchmark_data_smithi137_181069_object11361:head v 505'7754, mlcod=505'7751)", 2024-09-18T07:50:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:56.197773+0000", 2024-09-18T07:50:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 110.45017288699999, 2024-09-18T07:50:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27204197299999999, 2024-09-18T07:50:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.671 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.197773+0000", 2024-09-18T07:50:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.197773+0000", 2024-09-18T07:50:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.197775+0000", 2024-09-18T07:50:46.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4529999999999999e-06 2024-09-18T07:50:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.197781+0000", 2024-09-18T07:50:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4869999999999999e-06 2024-09-18T07:50:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.197787+0000", 2024-09-18T07:50:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.643e-06 2024-09-18T07:50:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.197791+0000", 2024-09-18T07:50:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1989999999999999e-06 2024-09-18T07:50:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.200634+0000", 2024-09-18T07:50:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002842846 2024-09-18T07:50:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.200654+0000", 2024-09-18T07:50:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.986e-05 2024-09-18T07:50:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.469793+0000", 2024-09-18T07:50:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.26913936900000002 2024-09-18T07:50:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.469815+0000", 2024-09-18T07:50:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2116e-05 2024-09-18T07:50:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:123514 11.c e505/468 11:36fa2ea6:::benchmark_data_smithi137_181069_object11360:head v 505'7619, mlcod=505'7616)", 2024-09-18T07:50:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:56.197816+0000", 2024-09-18T07:50:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 110.45013017799999, 2024-09-18T07:50:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27132467599999999, 2024-09-18T07:50:46.676 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.197816+0000", 2024-09-18T07:50:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.197816+0000", 2024-09-18T07:50:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.197817+0000", 2024-09-18T07:50:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5300000000000004e-07 2024-09-18T07:50:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.197821+0000", 2024-09-18T07:50:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7530000000000001e-06 2024-09-18T07:50:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.197827+0000", 2024-09-18T07:50:46.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4890000000000003e-06 2024-09-18T07:50:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.197831+0000", 2024-09-18T07:50:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4510000000000002e-06 2024-09-18T07:50:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.198529+0000", 2024-09-18T07:50:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00069771199999999996 2024-09-18T07:50:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.198547+0000", 2024-09-18T07:50:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8352e-05 2024-09-18T07:50:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.469106+0000", 2024-09-18T07:50:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.270558568 2024-09-18T07:50:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.469141+0000", 2024-09-18T07:50:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4697999999999999e-05 2024-09-18T07:50:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.680 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:123518 11.f e505/474 11:f4105960:::benchmark_data_smithi137_181069_object11364:head v 505'7755, mlcod=505'7751)", 2024-09-18T07:50:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:56.198847+0000", 2024-09-18T07:50:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 110.449099225, 2024-09-18T07:50:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27102953899999999, 2024-09-18T07:50:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.198847+0000", 2024-09-18T07:50:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.198847+0000", 2024-09-18T07:50:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.198848+0000", 2024-09-18T07:50:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9099999999999991e-07 2024-09-18T07:50:46.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.198854+0000", 2024-09-18T07:50:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1859999999999997e-06 2024-09-18T07:50:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.198863+0000", 2024-09-18T07:50:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5569999999999996e-06 2024-09-18T07:50:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.198869+0000", 2024-09-18T07:50:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1319999999999999e-06 2024-09-18T07:50:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.463348+0000", 2024-09-18T07:50:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.26447899800000002 2024-09-18T07:50:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.463368+0000", 2024-09-18T07:50:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0060000000000001e-05 2024-09-18T07:50:46.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.469864+0000", 2024-09-18T07:50:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0064958689999999996 2024-09-18T07:50:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.469877+0000", 2024-09-18T07:50:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2746e-05 2024-09-18T07:50:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:123519 11.c e505/468 11:3ed71fe1:::benchmark_data_smithi137_181069_object11365:head v 505'7620, mlcod=505'7617)", 2024-09-18T07:50:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:56.199724+0000", 2024-09-18T07:50:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 110.448222426, 2024-09-18T07:50:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27219565000000001, 2024-09-18T07:50:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.199724+0000", 2024-09-18T07:50:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.199724+0000", 2024-09-18T07:50:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.199725+0000", 2024-09-18T07:50:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.19e-06 2024-09-18T07:50:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.199731+0000", 2024-09-18T07:50:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5979999999999996e-06 2024-09-18T07:50:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.199739+0000", 2024-09-18T07:50:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6549999999999993e-06 2024-09-18T07:50:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.199745+0000", 2024-09-18T07:50:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7640000000000002e-06 2024-09-18T07:50:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.463319+0000", 2024-09-18T07:50:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.26357383600000001 2024-09-18T07:50:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.463343+0000", 2024-09-18T07:50:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4436000000000001e-05 2024-09-18T07:50:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.471883+0000", 2024-09-18T07:50:46.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0085399240000000008 2024-09-18T07:50:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.471919+0000", 2024-09-18T07:50:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6247e-05 2024-09-18T07:50:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:123524 11.7 e505/471 11:eb5f8cff:::benchmark_data_smithi137_181069_object11370:head v 505'7784, mlcod=505'7782)", 2024-09-18T07:50:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:56.200123+0000", 2024-09-18T07:50:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 110.44782317000001, 2024-09-18T07:50:46.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27041105799999998, 2024-09-18T07:50:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.200123+0000", 2024-09-18T07:50:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.200123+0000", 2024-09-18T07:50:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.200124+0000", 2024-09-18T07:50:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1400000000000001e-06 2024-09-18T07:50:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.200131+0000", 2024-09-18T07:50:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5169999999999996e-06 2024-09-18T07:50:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.200140+0000", 2024-09-18T07:50:46.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2909999999999996e-06 2024-09-18T07:50:46.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.200146+0000", 2024-09-18T07:50:46.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.874e-06 2024-09-18T07:50:46.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.202653+0000", 2024-09-18T07:50:46.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025072860000000001 2024-09-18T07:50:46.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.202670+0000", 2024-09-18T07:50:46.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6949e-05 2024-09-18T07:50:46.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.470474+0000", 2024-09-18T07:50:46.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.26780431100000002 2024-09-18T07:50:46.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.470534+0000", 2024-09-18T07:50:46.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9689999999999999e-05 2024-09-18T07:50:46.694 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.695 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.695 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.695 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:123527 11.3 e505/475 11:c9598565:::benchmark_data_smithi137_181069_object11373:head v 505'7697, mlcod=505'7695)", 2024-09-18T07:50:46.695 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:56.201005+0000", 2024-09-18T07:50:46.695 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 110.446941067, 2024-09-18T07:50:46.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.26238615799999998, 2024-09-18T07:50:46.695 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.695 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.695 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.695 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.201005+0000", 2024-09-18T07:50:46.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.201005+0000", 2024-09-18T07:50:46.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.201006+0000", 2024-09-18T07:50:46.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1620000000000001e-06 2024-09-18T07:50:46.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.201012+0000", 2024-09-18T07:50:46.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1059999999999996e-06 2024-09-18T07:50:46.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.201022+0000", 2024-09-18T07:50:46.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2019999999999993e-06 2024-09-18T07:50:46.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.201028+0000", 2024-09-18T07:50:46.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1480000000000003e-06 2024-09-18T07:50:46.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.201359+0000", 2024-09-18T07:50:46.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00033123099999999997 2024-09-18T07:50:46.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.201377+0000", 2024-09-18T07:50:46.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7663999999999999e-05 2024-09-18T07:50:46.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.463343+0000", 2024-09-18T07:50:46.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.26196676200000002 2024-09-18T07:50:46.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.463391+0000", 2024-09-18T07:50:46.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7883000000000002e-05 2024-09-18T07:50:46.699 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.700 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.700 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.700 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:123521 11.f e505/474 11:f52d7c62:::benchmark_data_smithi137_181069_object11367:head v 505'7756, mlcod=505'7751)", 2024-09-18T07:50:46.700 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:56.202999+0000", 2024-09-18T07:50:46.700 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 110.44494763, 2024-09-18T07:50:46.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.267565634, 2024-09-18T07:50:46.700 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.700 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.700 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.701 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.202999+0000", 2024-09-18T07:50:46.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.202999+0000", 2024-09-18T07:50:46.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.203001+0000", 2024-09-18T07:50:46.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.894e-06 2024-09-18T07:50:46.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.203009+0000", 2024-09-18T07:50:46.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.208e-06 2024-09-18T07:50:46.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.463285+0000", 2024-09-18T07:50:46.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.26027662200000001 2024-09-18T07:50:46.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.463298+0000", 2024-09-18T07:50:46.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2199e-05 2024-09-18T07:50:46.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.465510+0000", 2024-09-18T07:50:46.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022122299999999999 2024-09-18T07:50:46.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.465526+0000", 2024-09-18T07:50:46.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6705999999999999e-05 2024-09-18T07:50:46.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.470538+0000", 2024-09-18T07:50:46.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050116689999999998 2024-09-18T07:50:46.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.470564+0000", 2024-09-18T07:50:46.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6106e-05 2024-09-18T07:50:46.704 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.704 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.704 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.705 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.16581.0:123530 11.7 e505/471 11:e5df1a10:::benchmark_data_smithi137_181069_object11376:head v 505'7785, mlcod=505'7782)", 2024-09-18T07:50:46.705 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:48:56.207488+0000", 2024-09-18T07:50:46.705 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 110.44045798800001, 2024-09-18T07:50:46.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.26677282899999999, 2024-09-18T07:50:46.705 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.705 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.705 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.705 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.207488+0000", 2024-09-18T07:50:46.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.207488+0000", 2024-09-18T07:50:46.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.207490+0000", 2024-09-18T07:50:46.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8300000000000001e-06 2024-09-18T07:50:46.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.207500+0000", 2024-09-18T07:50:46.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8800000000000003e-06 2024-09-18T07:50:46.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.463296+0000", 2024-09-18T07:50:46.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.25579593900000003 2024-09-18T07:50:46.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.463306+0000", 2024-09-18T07:50:46.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9089999999999995e-06 2024-09-18T07:50:46.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.467437+0000", 2024-09-18T07:50:46.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041307879999999998 2024-09-18T07:50:46.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.467474+0000", 2024-09-18T07:50:46.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7548e-05 2024-09-18T07:50:46.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.474104+0000", 2024-09-18T07:50:46.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066299540000000004 2024-09-18T07:50:46.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:48:56.474261+0000", 2024-09-18T07:50:46.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015698099999999999 2024-09-18T07:50:46.709 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.709 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.709 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.709 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:210900 11.0 11:0b5a00e9:::benchmark_data_smithi137_181069_object98746:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e547)", 2024-09-18T07:50:46.709 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:50:25.310121+0000", 2024-09-18T07:50:46.709 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 21.337825612, 2024-09-18T07:50:46.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.19358066199999999, 2024-09-18T07:50:46.710 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.710 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.710 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.710 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:46.710 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:50:46.710 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:46.710 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 210900 2024-09-18T07:50:46.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.710 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.310121+0000", 2024-09-18T07:50:46.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.310121+0000", 2024-09-18T07:50:46.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.310123+0000", 2024-09-18T07:50:46.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0619999999999999e-06 2024-09-18T07:50:46.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.310128+0000", 2024-09-18T07:50:46.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5289999999999999e-06 2024-09-18T07:50:46.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.310140+0000", 2024-09-18T07:50:46.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1554e-05 2024-09-18T07:50:46.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.310147+0000", 2024-09-18T07:50:46.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1289999999999998e-06 2024-09-18T07:50:46.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.393913+0000", 2024-09-18T07:50:46.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.083765822000000004 2024-09-18T07:50:46.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.394067+0000", 2024-09-18T07:50:46.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015421599999999999 2024-09-18T07:50:46.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,6,7", 2024-09-18T07:50:46.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.394206+0000", 2024-09-18T07:50:46.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000139238 2024-09-18T07:50:46.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.395569+0000", 2024-09-18T07:50:46.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001363078 2024-09-18T07:50:46.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.395609+0000", 2024-09-18T07:50:46.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9390000000000001e-05 2024-09-18T07:50:46.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:46.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.396673+0000", 2024-09-18T07:50:46.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010642620000000001 2024-09-18T07:50:46.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.503482+0000", 2024-09-18T07:50:46.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.106809219 2024-09-18T07:50:46.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.503555+0000", 2024-09-18T07:50:46.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3044000000000001e-05 2024-09-18T07:50:46.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.503701+0000", 2024-09-18T07:50:46.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000146119 2024-09-18T07:50:46.716 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.716 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.716 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.716 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:210904 11.0 11:0749e2b6:::benchmark_data_smithi137_181069_object98750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e547)", 2024-09-18T07:50:46.716 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:50:25.310866+0000", 2024-09-18T07:50:46.716 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 21.337080481000001, 2024-09-18T07:50:46.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.19303707000000001, 2024-09-18T07:50:46.717 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.717 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.717 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.717 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:46.717 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:50:46.717 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:46.717 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 210904 2024-09-18T07:50:46.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.717 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.310866+0000", 2024-09-18T07:50:46.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.310866+0000", 2024-09-18T07:50:46.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.310867+0000", 2024-09-18T07:50:46.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1790000000000001e-06 2024-09-18T07:50:46.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.310871+0000", 2024-09-18T07:50:46.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2690000000000001e-06 2024-09-18T07:50:46.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.310884+0000", 2024-09-18T07:50:46.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2369e-05 2024-09-18T07:50:46.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.310890+0000", 2024-09-18T07:50:46.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6109999999999999e-06 2024-09-18T07:50:46.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.395635+0000", 2024-09-18T07:50:46.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.084744687999999999 2024-09-18T07:50:46.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.395710+0000", 2024-09-18T07:50:46.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5253999999999995e-05 2024-09-18T07:50:46.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,6,7", 2024-09-18T07:50:46.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.395781+0000", 2024-09-18T07:50:46.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0783000000000003e-05 2024-09-18T07:50:46.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.399107+0000", 2024-09-18T07:50:46.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003326028 2024-09-18T07:50:46.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.399140+0000", 2024-09-18T07:50:46.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3513999999999998e-05 2024-09-18T07:50:46.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:46.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.399153+0000", 2024-09-18T07:50:46.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2377e-05 2024-09-18T07:50:46.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.503766+0000", 2024-09-18T07:50:46.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.10461335099999999 2024-09-18T07:50:46.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.503803+0000", 2024-09-18T07:50:46.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6498000000000002e-05 2024-09-18T07:50:46.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.503903+0000", 2024-09-18T07:50:46.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010014900000000001 2024-09-18T07:50:46.723 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.723 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.723 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.723 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:210965 11.0 11:035c0361:::benchmark_data_smithi137_181069_object98811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e547)", 2024-09-18T07:50:46.724 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:50:25.420178+0000", 2024-09-18T07:50:46.724 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 21.227768186999999, 2024-09-18T07:50:46.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.25511863200000001, 2024-09-18T07:50:46.724 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.724 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.724 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.724 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:46.724 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:50:46.724 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:46.724 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 210965 2024-09-18T07:50:46.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.724 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.420178+0000", 2024-09-18T07:50:46.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.420178+0000", 2024-09-18T07:50:46.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.420180+0000", 2024-09-18T07:50:46.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.849e-06 2024-09-18T07:50:46.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.420186+0000", 2024-09-18T07:50:46.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4969999999999998e-06 2024-09-18T07:50:46.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:46 smithi203 ceph-mon[32237]: 11.2 scrub starts 2024-09-18T07:50:46.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:46 smithi203 ceph-mon[32237]: 11.2 scrub ok 2024-09-18T07:50:46.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:46 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3607675038' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:50:46.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:46 smithi203 ceph-mon[32237]: 11.3 scrub starts 2024-09-18T07:50:46.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:46 smithi203 ceph-mon[32237]: 11.3 scrub ok 2024-09-18T07:50:46.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:46 smithi203 ceph-mon[32237]: 4.e scrub starts 2024-09-18T07:50:46.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:46 smithi203 ceph-mon[32237]: 4.e scrub ok 2024-09-18T07:50:46.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:46 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1404526460' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:50:46.727 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:46 smithi203 ceph-mon[32237]: from='client.36785 ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-18T07:50:46.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:46 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-18T07:50:46.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:46 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3607675038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:50:46.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:46 smithi203 ceph-mon[32237]: from='client.36785 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-18T07:50:46.728 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:46 smithi203 ceph-mon[32237]: osdmap e561: 8 total, 8 up, 7 in 2024-09-18T07:50:46.728 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.420290+0000", 2024-09-18T07:50:46.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0001039 2024-09-18T07:50:46.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.729 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.729 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.420319+0000", 2024-09-18T07:50:46.729 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8408999999999999e-05 2024-09-18T07:50:46.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.729 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.729 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.505683+0000", 2024-09-18T07:50:46.729 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.085364548999999998 2024-09-18T07:50:46.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.729 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.729 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.505803+0000", 2024-09-18T07:50:46.729 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011987 2024-09-18T07:50:46.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,6,7", 2024-09-18T07:50:46.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.505933+0000", 2024-09-18T07:50:46.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013015600000000001 2024-09-18T07:50:46.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.511801+0000", 2024-09-18T07:50:46.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058676709999999997 2024-09-18T07:50:46.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.511896+0000", 2024-09-18T07:50:46.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4858000000000006e-05 2024-09-18T07:50:46.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:46.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.511915+0000", 2024-09-18T07:50:46.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9491e-05 2024-09-18T07:50:46.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.675148+0000", 2024-09-18T07:50:46.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.16323294599999999 2024-09-18T07:50:46.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.675190+0000", 2024-09-18T07:50:46.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1349e-05 2024-09-18T07:50:46.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.675297+0000", 2024-09-18T07:50:46.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107087 2024-09-18T07:50:46.732 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.732 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.732 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.732 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:210975 11.0 11:009c9873:::benchmark_data_smithi137_181069_object98821:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e547)", 2024-09-18T07:50:46.733 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:50:25.444365+0000", 2024-09-18T07:50:46.733 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 21.203581172, 2024-09-18T07:50:46.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.23107968600000001, 2024-09-18T07:50:46.733 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.733 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.733 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.733 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:46.733 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:50:46.733 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:46.733 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 210975 2024-09-18T07:50:46.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.733 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.444365+0000", 2024-09-18T07:50:46.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.444365+0000", 2024-09-18T07:50:46.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.444367+0000", 2024-09-18T07:50:46.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.886e-06 2024-09-18T07:50:46.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.444373+0000", 2024-09-18T07:50:46.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6879999999999996e-06 2024-09-18T07:50:46.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.444386+0000", 2024-09-18T07:50:46.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3086e-05 2024-09-18T07:50:46.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.444395+0000", 2024-09-18T07:50:46.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8759999999999994e-06 2024-09-18T07:50:46.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.509713+0000", 2024-09-18T07:50:46.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.065318072000000005 2024-09-18T07:50:46.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.509840+0000", 2024-09-18T07:50:46.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012701100000000001 2024-09-18T07:50:46.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,6,7", 2024-09-18T07:50:46.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.509925+0000", 2024-09-18T07:50:46.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5581999999999995e-05 2024-09-18T07:50:46.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:46.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.511923+0000", 2024-09-18T07:50:46.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019973880000000001 2024-09-18T07:50:46.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.675074+0000", 2024-09-18T07:50:46.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.163151034 2024-09-18T07:50:46.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.675118+0000", 2024-09-18T07:50:46.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3974999999999998e-05 2024-09-18T07:50:46.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.675337+0000", 2024-09-18T07:50:46.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021918799999999999 2024-09-18T07:50:46.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.675372+0000", 2024-09-18T07:50:46.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4958e-05 2024-09-18T07:50:46.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.675445+0000", 2024-09-18T07:50:46.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2941999999999994e-05 2024-09-18T07:50:46.739 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.739 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.740 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.740 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:210998 11.0 11:071a16a9:::benchmark_data_smithi137_181069_object98844:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e547)", 2024-09-18T07:50:46.740 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:50:25.529988+0000", 2024-09-18T07:50:46.740 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 21.117958649999998, 2024-09-18T07:50:46.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.29284167799999999, 2024-09-18T07:50:46.740 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.740 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.740 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.740 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:46.740 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:50:46.740 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:46.741 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 210998 2024-09-18T07:50:46.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.741 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.529988+0000", 2024-09-18T07:50:46.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.529988+0000", 2024-09-18T07:50:46.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.529990+0000", 2024-09-18T07:50:46.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1799999999999999e-06 2024-09-18T07:50:46.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.529996+0000", 2024-09-18T07:50:46.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2230000000000004e-06 2024-09-18T07:50:46.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.530023+0000", 2024-09-18T07:50:46.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7245000000000001e-05 2024-09-18T07:50:46.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.530030+0000", 2024-09-18T07:50:46.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0870000000000001e-06 2024-09-18T07:50:46.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.675478+0000", 2024-09-18T07:50:46.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.145447558 2024-09-18T07:50:46.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.675562+0000", 2024-09-18T07:50:46.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4338999999999993e-05 2024-09-18T07:50:46.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,6,7", 2024-09-18T07:50:46.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.675650+0000", 2024-09-18T07:50:46.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7844999999999997e-05 2024-09-18T07:50:46.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.680112+0000", 2024-09-18T07:50:46.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044615929999999998 2024-09-18T07:50:46.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.680164+0000", 2024-09-18T07:50:46.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2565999999999998e-05 2024-09-18T07:50:46.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:46.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.822545+0000", 2024-09-18T07:50:46.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.142380704 2024-09-18T07:50:46.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.822615+0000", 2024-09-18T07:50:46.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9728999999999996e-05 2024-09-18T07:50:46.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.822689+0000", 2024-09-18T07:50:46.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4802000000000007e-05 2024-09-18T07:50:46.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.822829+0000", 2024-09-18T07:50:46.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013980700000000001 2024-09-18T07:50:46.747 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.747 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.747 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.747 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:211002 11.0 11:0261748c:::benchmark_data_smithi137_181069_object98848:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e547)", 2024-09-18T07:50:46.747 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:50:25.539023+0000", 2024-09-18T07:50:46.747 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 21.108923701999998, 2024-09-18T07:50:46.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.28404019699999999, 2024-09-18T07:50:46.748 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.748 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.748 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.748 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:46.748 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:50:46.748 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:46.748 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211002 2024-09-18T07:50:46.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.748 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.539023+0000", 2024-09-18T07:50:46.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.539023+0000", 2024-09-18T07:50:46.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.539025+0000", 2024-09-18T07:50:46.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4789999999999999e-06 2024-09-18T07:50:46.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.539031+0000", 2024-09-18T07:50:46.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5799999999999999e-06 2024-09-18T07:50:46.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.539041+0000", 2024-09-18T07:50:46.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0854e-05 2024-09-18T07:50:46.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.539048+0000", 2024-09-18T07:50:46.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3810000000000004e-06 2024-09-18T07:50:46.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.677706+0000", 2024-09-18T07:50:46.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.138657752 2024-09-18T07:50:46.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.677826+0000", 2024-09-18T07:50:46.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120123 2024-09-18T07:50:46.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,6,7", 2024-09-18T07:50:46.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.677964+0000", 2024-09-18T07:50:46.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013821600000000001 2024-09-18T07:50:46.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.680228+0000", 2024-09-18T07:50:46.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002264407 2024-09-18T07:50:46.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.680283+0000", 2024-09-18T07:50:46.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4147000000000002e-05 2024-09-18T07:50:46.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.822893+0000", 2024-09-18T07:50:46.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.14261020299999999 2024-09-18T07:50:46.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:46.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.822908+0000", 2024-09-18T07:50:46.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5563e-05 2024-09-18T07:50:46.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.822943+0000", 2024-09-18T07:50:46.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5204999999999999e-05 2024-09-18T07:50:46.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.823063+0000", 2024-09-18T07:50:46.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000119287 2024-09-18T07:50:46.754 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.754 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.754 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.755 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:211016 11.0 11:06711054:::benchmark_data_smithi137_181069_object98862:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e547)", 2024-09-18T07:50:46.755 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:50:25.702014+0000", 2024-09-18T07:50:46.755 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 20.945931778999999, 2024-09-18T07:50:46.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.23675436899999999, 2024-09-18T07:50:46.755 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.755 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.755 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.755 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:46.755 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:50:46.755 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:46.755 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211016 2024-09-18T07:50:46.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.755 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.702014+0000", 2024-09-18T07:50:46.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.702014+0000", 2024-09-18T07:50:46.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.702016+0000", 2024-09-18T07:50:46.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5719999999999999e-06 2024-09-18T07:50:46.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.702022+0000", 2024-09-18T07:50:46.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7130000000000002e-06 2024-09-18T07:50:46.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.702031+0000", 2024-09-18T07:50:46.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0580000000000004e-06 2024-09-18T07:50:46.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.702037+0000", 2024-09-18T07:50:46.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6170000000000003e-06 2024-09-18T07:50:46.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.823528+0000", 2024-09-18T07:50:46.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.121490498 2024-09-18T07:50:46.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.823665+0000", 2024-09-18T07:50:46.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137224 2024-09-18T07:50:46.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,6,7", 2024-09-18T07:50:46.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.823790+0000", 2024-09-18T07:50:46.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012443900000000001 2024-09-18T07:50:46.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:46.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.938426+0000", 2024-09-18T07:50:46.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.114636847 2024-09-18T07:50:46.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.938503+0000", 2024-09-18T07:50:46.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6831999999999999e-05 2024-09-18T07:50:46.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.938582+0000", 2024-09-18T07:50:46.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8850000000000006e-05 2024-09-18T07:50:46.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.938645+0000", 2024-09-18T07:50:46.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2489000000000004e-05 2024-09-18T07:50:46.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.938687+0000", 2024-09-18T07:50:46.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2799000000000001e-05 2024-09-18T07:50:46.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.938769+0000", 2024-09-18T07:50:46.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1431e-05 2024-09-18T07:50:46.761 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.761 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.761 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.761 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.16581.0:211028 11.0 11:0d965eb0:::benchmark_data_smithi137_181069_object98874:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e547)", 2024-09-18T07:50:46.761 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-18T07:50:25.733814+0000", 2024-09-18T07:50:46.761 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 20.914132155000001, 2024-09-18T07:50:46.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.20517032700000001, 2024-09-18T07:50:46.762 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-18T07:50:46.762 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-18T07:50:46.762 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-18T07:50:46.762 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-18T07:50:46.762 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.16581", 2024-09-18T07:50:46.762 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/927474977", 2024-09-18T07:50:46.762 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 211028 2024-09-18T07:50:46.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.762 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-18T07:50:46.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-18T07:50:46.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.733814+0000", 2024-09-18T07:50:46.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-18T07:50:46.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.733814+0000", 2024-09-18T07:50:46.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-18T07:50:46.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-18T07:50:46.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.733816+0000", 2024-09-18T07:50:46.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.34e-06 2024-09-18T07:50:46.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-18T07:50:46.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.733825+0000", 2024-09-18T07:50:46.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2689999999999999e-06 2024-09-18T07:50:46.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-18T07:50:46.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.733838+0000", 2024-09-18T07:50:46.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.325e-05 2024-09-18T07:50:46.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-18T07:50:46.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.733846+0000", 2024-09-18T07:50:46.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5290000000000008e-06 2024-09-18T07:50:46.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-18T07:50:46.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.825971+0000", 2024-09-18T07:50:46.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.092124286999999999 2024-09-18T07:50:46.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-18T07:50:46.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.826124+0000", 2024-09-18T07:50:46.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015368700000000001 2024-09-18T07:50:46.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,6,7", 2024-09-18T07:50:46.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.826286+0000", 2024-09-18T07:50:46.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016197800000000001 2024-09-18T07:50:46.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.766 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.766 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.938542+0000", 2024-09-18T07:50:46.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.112255383 2024-09-18T07:50:46.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.766 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-18T07:50:46.766 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.938551+0000", 2024-09-18T07:50:46.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2820000000000002e-06 2024-09-18T07:50:46.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.766 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.766 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.938614+0000", 2024-09-18T07:50:46.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3026000000000003e-05 2024-09-18T07:50:46.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-18T07:50:46.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.938820+0000", 2024-09-18T07:50:46.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0002063 2024-09-18T07:50:46.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-18T07:50:46.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.938864+0000", 2024-09-18T07:50:46.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3090000000000002e-05 2024-09-18T07:50:46.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:46.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:46.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-18T07:50:46.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-18T07:50:25.938984+0000", 2024-09-18T07:50:46.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120906 2024-09-18T07:50:46.768 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.768 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.768 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.768 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:46.768 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:46.768 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:50:46.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:46.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:46.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:46.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:46.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:46.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:46.913 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:46.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:47.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:47.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:47.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:47.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:47.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:47.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:47.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:47.224 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-18T07:50:47.225 INFO:tasks.thrashosds.thrasher:in_osds: [2, 4, 5, 7, 6, 1, 3] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 6, 7, 4, 5] 2024-09-18T07:50:47.225 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-18T07:50:47.225 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-18T07:50:47.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-18T07:50:47.260 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:47.284 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:47.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:47.367 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:50:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:50:47.385 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:47.407 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:47.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:47.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:47.509 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:47.531 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:47.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:47.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:47.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:47.654 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:47.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:47.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:47.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[29408]: pgmap v1070: 168 pgs: 3 peering, 9 unknown, 3 active+remapped+backfill_toofull, 153 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:50:47.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[29408]: 6.3 scrub starts 2024-09-18T07:50:47.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[29408]: 6.3 scrub ok 2024-09-18T07:50:47.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[29408]: 4.5 scrub starts 2024-09-18T07:50:47.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[29408]: 5.1b scrub starts 2024-09-18T07:50:47.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[29408]: 5.1b scrub ok 2024-09-18T07:50:47.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[29408]: 4.5 scrub ok 2024-09-18T07:50:47.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[29408]: 4.c scrub starts 2024-09-18T07:50:47.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[29408]: 4.c scrub ok 2024-09-18T07:50:47.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[39175]: pgmap v1070: 168 pgs: 3 peering, 9 unknown, 3 active+remapped+backfill_toofull, 153 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:50:47.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[39175]: 6.3 scrub starts 2024-09-18T07:50:47.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[39175]: 6.3 scrub ok 2024-09-18T07:50:47.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[39175]: 4.5 scrub starts 2024-09-18T07:50:47.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[39175]: 5.1b scrub starts 2024-09-18T07:50:47.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[39175]: 5.1b scrub ok 2024-09-18T07:50:47.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[39175]: 4.5 scrub ok 2024-09-18T07:50:47.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[39175]: 4.c scrub starts 2024-09-18T07:50:47.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:47 smithi137 ceph-mon[39175]: 4.c scrub ok 2024-09-18T07:50:47.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:47 smithi203 ceph-mon[32237]: pgmap v1070: 168 pgs: 3 peering, 9 unknown, 3 active+remapped+backfill_toofull, 153 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:50:47.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:47 smithi203 ceph-mon[32237]: 6.3 scrub starts 2024-09-18T07:50:47.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:47 smithi203 ceph-mon[32237]: 6.3 scrub ok 2024-09-18T07:50:47.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:47 smithi203 ceph-mon[32237]: 4.5 scrub starts 2024-09-18T07:50:47.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:47 smithi203 ceph-mon[32237]: 5.1b scrub starts 2024-09-18T07:50:47.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:47 smithi203 ceph-mon[32237]: 5.1b scrub ok 2024-09-18T07:50:47.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:47 smithi203 ceph-mon[32237]: 4.5 scrub ok 2024-09-18T07:50:47.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:47 smithi203 ceph-mon[32237]: 4.c scrub starts 2024-09-18T07:50:47.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:47 smithi203 ceph-mon[32237]: 4.c scrub ok 2024-09-18T07:50:47.757 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:47.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:47.779 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:47.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:47.880 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:47.884 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:47.889 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:47.906 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:47.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:48.008 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:48.034 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:48.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:48.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:48.136 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3 2024-09-18T07:50:48.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:48.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.3' 2024-09-18T07:50:48.261 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2 2024-09-18T07:50:48.283 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:48.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.2' 2024-09-18T07:50:48.376 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:50:48 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:50:48.385 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:48.411 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:48.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:48.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:48.513 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:48.538 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:48.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:48.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:48.640 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:48.665 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:48.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:48.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:48.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:48 smithi137 ceph-mon[29408]: 5.2 scrub starts 2024-09-18T07:50:48.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:48 smithi137 ceph-mon[29408]: 5.2 scrub ok 2024-09-18T07:50:48.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:48 smithi137 ceph-mon[29408]: 6.2 scrub starts 2024-09-18T07:50:48.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:48 smithi137 ceph-mon[29408]: 6.2 scrub ok 2024-09-18T07:50:48.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:48 smithi137 ceph-mon[39175]: 5.2 scrub starts 2024-09-18T07:50:48.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:48 smithi137 ceph-mon[39175]: 5.2 scrub ok 2024-09-18T07:50:48.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:48 smithi137 ceph-mon[39175]: 6.2 scrub starts 2024-09-18T07:50:48.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:48 smithi137 ceph-mon[39175]: 6.2 scrub ok 2024-09-18T07:50:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:48 smithi203 ceph-mon[32237]: 5.2 scrub starts 2024-09-18T07:50:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:48 smithi203 ceph-mon[32237]: 5.2 scrub ok 2024-09-18T07:50:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:48 smithi203 ceph-mon[32237]: 6.2 scrub starts 2024-09-18T07:50:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:48 smithi203 ceph-mon[32237]: 6.2 scrub ok 2024-09-18T07:50:48.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:48.790 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:48.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:48.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:48.892 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:48.917 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:48.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:48.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:49.019 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:49.045 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:49.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:49.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:49.147 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:49.173 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:49.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:49.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:49.198 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-18T07:50:49.275 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:49.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:49.299 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:49.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:49.400 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:49.414 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-18T07:50:49.415 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-18T07:50:49.424 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:49.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:49.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:49.431 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-18T07:50:49.431 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-18T07:50:49.447 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-18T07:50:49.447 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-18T07:50:49.463 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-18T07:50:49.463 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-18T07:50:49.479 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-18T07:50:49.479 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-18T07:50:49.479 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:49 smithi137 ceph-mon[29408]: pgmap v1072: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 896 B/s rd, 0 op/s 2024-09-18T07:50:49.479 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:49 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:49.480 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:49 smithi137 ceph-mon[29408]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-09-18T07:50:49.480 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:49 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:50:49.480 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:49 smithi137 ceph-mon[29408]: 1.7 scrub starts 2024-09-18T07:50:49.480 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:49 smithi137 ceph-mon[29408]: 1.7 scrub ok 2024-09-18T07:50:49.480 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:49 smithi137 ceph-mon[39175]: pgmap v1072: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 896 B/s rd, 0 op/s 2024-09-18T07:50:49.480 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:49 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:49.480 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:49 smithi137 ceph-mon[39175]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-09-18T07:50:49.480 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:49 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:50:49.480 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:49 smithi137 ceph-mon[39175]: 1.7 scrub starts 2024-09-18T07:50:49.480 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:49 smithi137 ceph-mon[39175]: 1.7 scrub ok 2024-09-18T07:50:49.496 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-18T07:50:49.496 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-18T07:50:49.513 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-18T07:50:49.514 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-18T07:50:49.526 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5 2024-09-18T07:50:49.532 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-18T07:50:49.532 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-18T07:50:49.549 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-18T07:50:49.550 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:49.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:49.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.5' 2024-09-18T07:50:49.652 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1 2024-09-18T07:50:49.675 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:49.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:49.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.1' 2024-09-18T07:50:49.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:49 smithi203 ceph-mon[32237]: pgmap v1072: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 896 B/s rd, 0 op/s 2024-09-18T07:50:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:49 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:49 smithi203 ceph-mon[32237]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-09-18T07:50:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:49 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:50:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:49 smithi203 ceph-mon[32237]: 1.7 scrub starts 2024-09-18T07:50:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:49 smithi203 ceph-mon[32237]: 1.7 scrub ok 2024-09-18T07:50:49.776 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6 2024-09-18T07:50:49.799 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:49.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:49.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.6' 2024-09-18T07:50:49.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:49.924 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:49.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:50.026 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4 2024-09-18T07:50:50.050 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:50.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:50.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.4' 2024-09-18T07:50:50.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0 2024-09-18T07:50:50.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:50.174 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-18T07:50:50.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.0' 2024-09-18T07:50:50.275 DEBUG:teuthology.orchestra.run.smithi203:> sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7 2024-09-18T07:50:50.300 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: docker: command not found 2024-09-18T07:50:50.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T07:50:50.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi203 with status 1: 'sudo docker kill -s 1 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd.7' 2024-09-18T07:50:50.325 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-18T07:50:50.334 INFO:tasks.thrashosds:joining thrashosds 2024-09-18T07:50:50.594 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:50 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1156212825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:50.594 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:50 smithi137 ceph-mon[29408]: 6.1d scrub starts 2024-09-18T07:50:50.594 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:50 smithi137 ceph-mon[29408]: 6.1d scrub ok 2024-09-18T07:50:50.595 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:50 smithi137 ceph-mon[29408]: osdmap e562: 8 total, 8 up, 7 in 2024-09-18T07:50:50.595 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:50 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1156212825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:50.595 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:50 smithi137 ceph-mon[39175]: 6.1d scrub starts 2024-09-18T07:50:50.595 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:50 smithi137 ceph-mon[39175]: 6.1d scrub ok 2024-09-18T07:50:50.595 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:50 smithi137 ceph-mon[39175]: osdmap e562: 8 total, 8 up, 7 in 2024-09-18T07:50:50.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:50 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1156212825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-18T07:50:50.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:50 smithi203 ceph-mon[32237]: 6.1d scrub starts 2024-09-18T07:50:50.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:50 smithi203 ceph-mon[32237]: 6.1d scrub ok 2024-09-18T07:50:50.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:50 smithi203 ceph-mon[32237]: osdmap e562: 8 total, 8 up, 7 in 2024-09-18T07:50:51.403 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:51.403 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:51.530 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:51 smithi137 ceph-mon[29408]: pgmap v1073: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-18T07:50:51.530 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:51 smithi137 ceph-mon[29408]: 6.1f scrub starts 2024-09-18T07:50:51.531 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:51 smithi137 ceph-mon[29408]: 6.1f scrub ok 2024-09-18T07:50:51.531 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:51 smithi137 ceph-mon[29408]: 5.1d scrub starts 2024-09-18T07:50:51.531 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:51 smithi137 ceph-mon[29408]: 5.1d scrub ok 2024-09-18T07:50:51.531 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:51 smithi137 ceph-mon[39175]: pgmap v1073: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-18T07:50:51.531 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:51 smithi137 ceph-mon[39175]: 6.1f scrub starts 2024-09-18T07:50:51.531 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:51 smithi137 ceph-mon[39175]: 6.1f scrub ok 2024-09-18T07:50:51.531 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:51 smithi137 ceph-mon[39175]: 5.1d scrub starts 2024-09-18T07:50:51.531 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:51 smithi137 ceph-mon[39175]: 5.1d scrub ok 2024-09-18T07:50:51.532 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:50:51.533 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-18T07:50:51.533 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 563, 2024-09-18T07:50:51.533 INFO:teuthology.orchestra.run.smithi137.stdout: "fsid": "f7ef099c-758e-11ef-bceb-c7b262605968", 2024-09-18T07:50:51.533 INFO:teuthology.orchestra.run.smithi137.stdout: "created": "2024-09-18T07:25:19.662367+0000", 2024-09-18T07:50:51.533 INFO:teuthology.orchestra.run.smithi137.stdout: "modified": "2024-09-18T07:50:51.380796+0000", 2024-09-18T07:50:51.533 INFO:teuthology.orchestra.run.smithi137.stdout: "last_up_change": "2024-09-18T07:47:10.160431+0000", 2024-09-18T07:50:51.533 INFO:teuthology.orchestra.run.smithi137.stdout: "last_in_change": "2024-09-18T07:50:22.246016+0000", 2024-09-18T07:50:51.533 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-18T07:50:51.533 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_num": 5799936, 2024-09-18T07:50:51.533 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_set": [ 2024-09-18T07:50:51.533 INFO:teuthology.orchestra.run.smithi137.stdout: "pglog_hardlimit", 2024-09-18T07:50:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: "purged_snapdirs", 2024-09-18T07:50:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: "recovery_deletes", 2024-09-18T07:50:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: "sortbitwise" 2024-09-18T07:50:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:50:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_version": 20, 2024-09-18T07:50:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: "full_ratio": 0.94999998807907104, 2024-09-18T07:50:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-18T07:50:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-18T07:50:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_snapshot": "", 2024-09-18T07:50:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_max": 11, 2024-09-18T07:50:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: "max_osd": 8, 2024-09-18T07:50:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: "require_min_compat_client": "luminous", 2024-09-18T07:50:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: "min_compat_client": "luminous", 2024-09-18T07:50:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: "require_osd_release": "squid", 2024-09-18T07:50:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: "allow_crimson": false, 2024-09-18T07:50:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: "pools": [ 2024-09-18T07:50:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 1, 2024-09-18T07:50:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".mgr", 2024-09-18T07:50:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:28:20.899655+0000", 2024-09-18T07:50:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:50:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:50:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:50:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:50:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:50:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:50:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:50:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:50:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:50:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:50:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:50:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:50:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:50:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 11, 2024-09-18T07:50:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 11, 2024-09-18T07:50:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 11, 2024-09-18T07:50:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 11, 2024-09-18T07:50:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 11, 2024-09-18T07:50:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:50:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:50:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:50:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:50:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:50:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:50:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:50:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "531", 2024-09-18T07:50:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:50:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:50:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "272", 2024-09-18T07:50:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:50:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:50:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:50:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:50:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:50:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:50:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:50:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:50:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:50:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:50:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:50:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:50:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:50:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:50:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:50:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:50:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:50:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:50:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:50:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:50:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:50:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:50:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:50:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:50:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:50:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:50:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:50:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:50:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:50:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:50:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:50:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:50:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:50:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:50:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:50:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_max": 32, 2024-09-18T07:50:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_min": 1 2024-09-18T07:50:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:50:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": {} 2024-09-18T07:50:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:50:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:50:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.9099999666213989, 2024-09-18T07:50:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.9099999666213989, 2024-09-18T07:50:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:50:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.9099999666213989, 2024-09-18T07:50:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.9099999666213989, 2024-09-18T07:50:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.73095130920410156, 2024-09-18T07:50:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.8399999737739563, 2024-09-18T07:50:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.73000001907348633 2024-09-18T07:50:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 2, 2024-09-18T07:50:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "datapool", 2024-09-18T07:50:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:53.715373+0000", 2024-09-18T07:50:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-18T07:50:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-18T07:50:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:50:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:50:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:50:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:50:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:50:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:50:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:50:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:50:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:50:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:50:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:50:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 3, 2024-09-18T07:50:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 3, 2024-09-18T07:50:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 3, 2024-09-18T07:50:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 3, 2024-09-18T07:50:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 3, 2024-09-18T07:50:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:50:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:50:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:50:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:50:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:50:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:50:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:50:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "63", 2024-09-18T07:50:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:50:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:50:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:50:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:50:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:50:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 2, 2024-09-18T07:50:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 63, 2024-09-18T07:50:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:50:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:50:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:50:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:50:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:50:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:50:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:50:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:50:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:50:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:50:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:50:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:50:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:50:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:50:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:50:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:50:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:50:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:50:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:50:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:50:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:50:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:50:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:50:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:50:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:50:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:50:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:50:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:50:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:50:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:50:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:50:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:50:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd": {} 2024-09-18T07:50:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:50:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:50:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.3299999237060547, 2024-09-18T07:50:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.3299999237060547, 2024-09-18T07:50:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.86000001430511475, 2024-09-18T07:50:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2, 2024-09-18T07:50:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2, 2024-09-18T07:50:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.73095130920410156, 2024-09-18T07:50:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.8399999737739563, 2024-09-18T07:50:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.73000001907348633 2024-09-18T07:50:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 3, 2024-09-18T07:50:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".rgw.root", 2024-09-18T07:50:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:54.672127+0000", 2024-09-18T07:50:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:50:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:50:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:50:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:50:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:50:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:50:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:50:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:50:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:50:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:50:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:50:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:50:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:50:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:50:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:50:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:50:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:50:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:50:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:50:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:50:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:50:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:50:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:50:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:50:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:50:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "62", 2024-09-18T07:50:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:50:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:50:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:50:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:50:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:50:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:50:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:50:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:50:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:50:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:50:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:50:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:50:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:50:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:50:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:50:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:50:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:50:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:50:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:50:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:50:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:50:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:50:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:50:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:50:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:50:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:50:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:50:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:50:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:50:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:50:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:50:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:50:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:50:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:50:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:50:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:50:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:50:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:50:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:50:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:50:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:50:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:50:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.190000057220459, 2024-09-18T07:50:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.190000057220459, 2024-09-18T07:50:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:50:51.555 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.190000057220459, 2024-09-18T07:50:51.555 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.190000057220459, 2024-09-18T07:50:51.555 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.73095130920410156, 2024-09-18T07:50:51.555 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.8399999737739563, 2024-09-18T07:50:51.555 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.73000001907348633 2024-09-18T07:50:51.555 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.555 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.555 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.555 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 4, 2024-09-18T07:50:51.555 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.log", 2024-09-18T07:50:51.555 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:56.351343+0000", 2024-09-18T07:50:51.555 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:50:51.555 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:50:51.556 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:50:51.556 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:50:51.556 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:50:51.556 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:50:51.556 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:50:51.556 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:50:51.556 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:50:51.556 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:50:51.556 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:50:51.556 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:50:51.556 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:50:51.556 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:50:51.556 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:50:51.557 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:50:51.557 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:50:51.557 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:50:51.557 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:50:51.557 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:50:51.557 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:50:51.557 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:50:51.557 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:50:51.557 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:50:51.557 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:50:51.557 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.557 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "64", 2024-09-18T07:50:51.557 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:50:51.558 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:50:51.558 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:50:51.558 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:50:51.558 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:50:51.558 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:50:51.558 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:50:51.558 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:50:51.558 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:50:51.558 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:50:51.558 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:50:51.558 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:50:51.558 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:50:51.558 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:50:51.558 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:50:51.559 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:50:51.559 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:50:51.559 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:50:51.559 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:50:51.559 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:50:51.559 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:50:51.559 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:50:51.559 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:50:51.559 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:50:51.559 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:50:51.559 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:50:51.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.559 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:50:51.560 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:50:51.560 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:50:51.560 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:50:51.560 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:50:51.560 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:50:51.560 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:50:51.560 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:50:51.560 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:50:51.560 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:50:51.560 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:50:51.560 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:50:51.560 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:50:51.560 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:50:51.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.561 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:50:51.561 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:50:51.561 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.4100000858306885, 2024-09-18T07:50:51.561 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.4100000858306885, 2024-09-18T07:50:51.561 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:50:51.561 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.4100000858306885, 2024-09-18T07:50:51.561 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.4100000858306885, 2024-09-18T07:50:51.561 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.73095130920410156, 2024-09-18T07:50:51.561 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.8399999737739563, 2024-09-18T07:50:51.561 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.73000001907348633 2024-09-18T07:50:51.561 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.562 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.562 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 5, 2024-09-18T07:50:51.562 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.control", 2024-09-18T07:50:51.562 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:30:58.322401+0000", 2024-09-18T07:50:51.562 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:50:51.562 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:50:51.562 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:50:51.562 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:50:51.562 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:50:51.562 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:50:51.562 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:50:51.562 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:50:51.562 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:50:51.563 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:50:51.563 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:50:51.563 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:50:51.563 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:50:51.563 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:50:51.563 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:50:51.563 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:50:51.563 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:50:51.563 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:50:51.563 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:50:51.563 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:50:51.563 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:50:51.563 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:50:51.564 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:50:51.564 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:50:51.564 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:50:51.564 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.564 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "66", 2024-09-18T07:50:51.564 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:50:51.564 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:50:51.564 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:50:51.564 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:50:51.564 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:50:51.564 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:50:51.564 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:50:51.564 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:50:51.564 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:50:51.565 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:50:51.565 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:50:51.565 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:50:51.565 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:50:51.565 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:50:51.565 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:50:51.565 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:50:51.565 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:50:51.565 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:50:51.566 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:50:51.566 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:50:51.566 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:50:51.566 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:50:51.566 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:50:51.566 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:50:51.566 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:50:51.566 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:50:51.566 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.567 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:50:51.567 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:50:51.567 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:50:51.567 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:50:51.567 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:50:51.567 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:50:51.567 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:50:51.567 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:50:51.567 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:50:51.567 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:50:51.567 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:50:51.567 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-18T07:50:51.567 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:50:51.567 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:50:51.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.568 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:50:51.568 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:50:51.568 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.75, 2024-09-18T07:50:51.568 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.75, 2024-09-18T07:50:51.569 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:50:51.569 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.75, 2024-09-18T07:50:51.569 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.75, 2024-09-18T07:50:51.569 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.73095130920410156, 2024-09-18T07:50:51.569 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.8399999737739563, 2024-09-18T07:50:51.570 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.73000001907348633 2024-09-18T07:50:51.570 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.570 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.570 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 6, 2024-09-18T07:50:51.570 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.meta", 2024-09-18T07:50:51.570 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-18T07:31:00.390499+0000", 2024-09-18T07:50:51.570 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-18T07:50:51.570 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-18T07:50:51.570 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-18T07:50:51.570 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-18T07:50:51.570 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-18T07:50:51.570 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-18T07:50:51.571 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-18T07:50:51.571 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-18T07:50:51.571 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-18T07:50:51.571 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-18T07:50:51.571 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-18T07:50:51.571 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-18T07:50:51.571 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-18T07:50:51.571 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-18T07:50:51.571 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-18T07:50:51.571 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-18T07:50:51.571 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-18T07:50:51.571 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-18T07:50:51.571 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-18T07:50:51.572 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-18T07:50:51.572 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-18T07:50:51.572 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-18T07:50:51.572 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-18T07:50:51.572 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-18T07:50:51.572 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-18T07:50:51.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.572 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "68", 2024-09-18T07:50:51.572 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-18T07:50:51.572 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-18T07:50:51.572 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-18T07:50:51.572 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-18T07:50:51.573 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-18T07:50:51.573 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-18T07:50:51.573 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-18T07:50:51.573 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-18T07:50:51.573 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-18T07:50:51.573 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-18T07:50:51.573 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-18T07:50:51.573 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-18T07:50:51.573 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-18T07:50:51.573 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-18T07:50:51.573 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-18T07:50:51.573 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-18T07:50:51.573 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-18T07:50:51.574 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-18T07:50:51.574 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-18T07:50:51.574 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-18T07:50:51.574 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-18T07:50:51.574 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-18T07:50:51.574 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-18T07:50:51.574 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-18T07:50:51.574 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-18T07:50:51.574 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-18T07:50:51.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.574 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-18T07:50:51.574 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-18T07:50:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-18T07:50:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-18T07:50:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-18T07:50:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-18T07:50:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-18T07:50:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-18T07:50:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-18T07:50:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-18T07:50:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-18T07:50:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-18T07:50:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_bias": 4 2024-09-18T07:50:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-18T07:50:51.576 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-18T07:50:51.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.576 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-18T07:50:51.576 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-18T07:50:51.576 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.9700000286102295, 2024-09-18T07:50:51.576 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.9700000286102295, 2024-09-18T07:50:51.576 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 1, 2024-09-18T07:50:51.576 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-18T07:50:51.576 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-18T07:50:51.576 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.73095130920410156, 2024-09-18T07:50:51.576 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.8399999737739563, 2024-09-18T07:50:51.576 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.73000001907348633 2024-09-18T07:50:51.577 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.577 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.577 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:50:51.577 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-18T07:50:51.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.577 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:50:51.577 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "268cd580-c931-4992-bb93-d6f63c2bfe88", 2024-09-18T07:50:51.577 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:50:51.577 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-18T07:50:51.577 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-18T07:50:51.577 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 0.5107269287109375, 2024-09-18T07:50:51.577 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:50:51.578 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:50:51.578 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 8, 2024-09-18T07:50:51.578 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 521, 2024-09-18T07:50:51.578 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:50:51.578 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:50:51.578 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:50:51.578 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.578 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.578 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6801", 2024-09-18T07:50:51.578 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:50:51.578 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.578 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.579 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:50:51.579 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.579 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.579 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6802", 2024-09-18T07:50:51.579 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:50:51.579 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.579 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.579 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:50:51.579 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.579 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.580 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6804", 2024-09-18T07:50:51.580 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:50:51.580 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.580 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.580 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:50:51.580 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.580 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.580 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6803", 2024-09-18T07:50:51.580 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2811444477 2024-09-18T07:50:51.580 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.580 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.581 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6801/2811444477", 2024-09-18T07:50:51.581 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6802/2811444477", 2024-09-18T07:50:51.581 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6804/2811444477", 2024-09-18T07:50:51.581 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6803/2811444477", 2024-09-18T07:50:51.581 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:50:51.581 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:50:51.581 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:50:51.581 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.581 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:50:51.581 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "b06907bc-b499-46c0-8667-c5bdc0d85575", 2024-09-18T07:50:51.582 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:50:51.582 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:50:51.582 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:50:51.582 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 0, 2024-09-18T07:50:51.582 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:50:51.582 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:50:51.582 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 13, 2024-09-18T07:50:51.582 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 560, 2024-09-18T07:50:51.582 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:50:51.582 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:50:51.582 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:50:51.582 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.583 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.583 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6805", 2024-09-18T07:50:51.583 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:50:51.583 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.583 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.583 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:50:51.583 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.583 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.583 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6806", 2024-09-18T07:50:51.583 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:50:51.583 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.584 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.584 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:50:51.584 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.584 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.584 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6808", 2024-09-18T07:50:51.584 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:50:51.584 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.584 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.584 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:50:51.584 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.585 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.585 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6807", 2024-09-18T07:50:51.585 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 874102487 2024-09-18T07:50:51.585 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.585 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.585 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6805/874102487", 2024-09-18T07:50:51.585 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6806/874102487", 2024-09-18T07:50:51.585 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6808/874102487", 2024-09-18T07:50:51.585 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6807/874102487", 2024-09-18T07:50:51.585 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:50:51.585 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:50:51.585 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:50:51.586 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.586 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:50:51.586 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "480ad6c5-0e0b-4405-bbe2-381682d00bae", 2024-09-18T07:50:51.586 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:50:51.586 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:50:51.586 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:50:51.586 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:50:51.586 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:50:51.586 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:50:51.586 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 18, 2024-09-18T07:50:51.586 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 560, 2024-09-18T07:50:51.587 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:50:51.587 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:50:51.587 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:50:51.587 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.587 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.587 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.587 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6809", 2024-09-18T07:50:51.587 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:50:51.587 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.587 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.587 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:50:51.587 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.587 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.588 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.588 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6810", 2024-09-18T07:50:51.588 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:50:51.588 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.588 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.588 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:50:51.588 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.588 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.588 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6812", 2024-09-18T07:50:51.588 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:50:51.588 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.589 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.589 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:50:51.589 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.589 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.589 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6811", 2024-09-18T07:50:51.589 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1754684686 2024-09-18T07:50:51.589 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.589 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.589 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6809/1754684686", 2024-09-18T07:50:51.589 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6810/1754684686", 2024-09-18T07:50:51.590 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6812/1754684686", 2024-09-18T07:50:51.590 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6811/1754684686", 2024-09-18T07:50:51.590 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:50:51.590 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:50:51.590 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:50:51.590 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.590 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:50:51.590 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "178be3e0-f07b-4856-807b-2a1181e48a14", 2024-09-18T07:50:51.590 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:50:51.590 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:50:51.590 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0.885101318359375, 2024-09-18T07:50:51.591 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 0.759490966796875, 2024-09-18T07:50:51.591 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:50:51.591 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:50:51.591 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 26, 2024-09-18T07:50:51.591 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 560, 2024-09-18T07:50:51.591 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:50:51.591 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:50:51.591 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:50:51.591 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.591 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.591 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6813", 2024-09-18T07:50:51.591 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:50:51.592 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.592 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.592 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:50:51.592 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.592 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.592 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6814", 2024-09-18T07:50:51.592 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:50:51.592 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.592 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.592 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:50:51.593 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.593 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.593 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6816", 2024-09-18T07:50:51.593 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:50:51.593 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.593 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.593 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:50:51.593 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.594 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.594 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6815", 2024-09-18T07:50:51.594 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 116058392 2024-09-18T07:50:51.594 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.594 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.594 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6813/116058392", 2024-09-18T07:50:51.594 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6814/116058392", 2024-09-18T07:50:51.594 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6816/116058392", 2024-09-18T07:50:51.594 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6815/116058392", 2024-09-18T07:50:51.594 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:50:51.594 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:50:51.594 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:50:51.595 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.595 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:50:51.595 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "7547102c-951a-4077-a094-fb8a65bfe6de", 2024-09-18T07:50:51.595 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:50:51.595 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:50:51.595 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:50:51.595 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:50:51.595 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 32, 2024-09-18T07:50:51.595 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 358, 2024-09-18T07:50:51.595 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 395, 2024-09-18T07:50:51.595 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 559, 2024-09-18T07:50:51.596 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 359, 2024-09-18T07:50:51.596 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:50:51.596 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:50:51.596 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.596 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.596 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6800", 2024-09-18T07:50:51.596 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 571224280 2024-09-18T07:50:51.596 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.596 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.596 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:50:51.597 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.597 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.597 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6801", 2024-09-18T07:50:51.597 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 571224280 2024-09-18T07:50:51.597 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.597 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.597 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:50:51.597 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.597 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.597 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6803", 2024-09-18T07:50:51.598 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 571224280 2024-09-18T07:50:51.598 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.598 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.598 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:50:51.598 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.598 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.598 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6802", 2024-09-18T07:50:51.598 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 571224280 2024-09-18T07:50:51.598 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.598 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.598 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6800/571224280", 2024-09-18T07:50:51.599 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6801/571224280", 2024-09-18T07:50:51.599 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6803/571224280", 2024-09-18T07:50:51.599 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6802/571224280", 2024-09-18T07:50:51.599 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:50:51.599 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:50:51.599 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:50:51.599 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.599 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:50:51.599 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "49c17901-8102-48a4-9348-fa1bdb63032c", 2024-09-18T07:50:51.599 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:50:51.599 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:50:51.600 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:50:51.600 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 0.577392578125, 2024-09-18T07:50:51.600 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 39, 2024-09-18T07:50:51.600 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 415, 2024-09-18T07:50:51.600 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 460, 2024-09-18T07:50:51.600 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 560, 2024-09-18T07:50:51.600 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 416, 2024-09-18T07:50:51.600 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:50:51.600 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:50:51.600 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.600 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.600 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6804", 2024-09-18T07:50:51.601 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1993911072 2024-09-18T07:50:51.601 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.601 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.601 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:50:51.601 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.601 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.601 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6805", 2024-09-18T07:50:51.601 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1993911072 2024-09-18T07:50:51.601 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.601 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.602 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:50:51.602 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.602 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.602 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6807", 2024-09-18T07:50:51.602 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1993911072 2024-09-18T07:50:51.602 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.602 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.602 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:50:51.602 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.602 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.603 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6806", 2024-09-18T07:50:51.603 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1993911072 2024-09-18T07:50:51.603 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.603 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.603 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6804/1993911072", 2024-09-18T07:50:51.603 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6805/1993911072", 2024-09-18T07:50:51.603 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6807/1993911072", 2024-09-18T07:50:51.603 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6806/1993911072", 2024-09-18T07:50:51.603 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:50:51.603 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:50:51.603 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:50:51.603 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.604 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:50:51.604 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "75f0ead4-e981-4a15-b1c1-d83dce871ab6", 2024-09-18T07:50:51.604 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:50:51.604 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:50:51.604 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:50:51.604 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:50:51.604 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:50:51.604 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:50:51.604 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 47, 2024-09-18T07:50:51.604 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 560, 2024-09-18T07:50:51.604 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:50:51.604 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:50:51.605 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:50:51.605 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.605 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.605 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6808", 2024-09-18T07:50:51.605 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:50:51.605 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.605 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.605 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:50:51.605 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.605 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.605 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6809", 2024-09-18T07:50:51.606 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:50:51.606 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.606 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.606 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:50:51.606 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.606 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.606 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.606 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6811", 2024-09-18T07:50:51.606 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:50:51.606 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.606 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.607 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:50:51.607 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.607 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.607 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6810", 2024-09-18T07:50:51.607 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1494175466 2024-09-18T07:50:51.607 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.607 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.607 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6808/1494175466", 2024-09-18T07:50:51.607 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6809/1494175466", 2024-09-18T07:50:51.607 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6811/1494175466", 2024-09-18T07:50:51.607 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6810/1494175466", 2024-09-18T07:50:51.608 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:50:51.608 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:50:51.608 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:50:51.608 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.608 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:50:51.608 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "c0dbf299-6798-4b99-9a85-62b123971f2a", 2024-09-18T07:50:51.608 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-18T07:50:51.608 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-18T07:50:51.608 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-18T07:50:51.608 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-18T07:50:51.609 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-18T07:50:51.609 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-18T07:50:51.609 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 55, 2024-09-18T07:50:51.609 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 560, 2024-09-18T07:50:51.609 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-18T07:50:51.609 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-18T07:50:51.609 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-18T07:50:51.609 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.609 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.609 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6812", 2024-09-18T07:50:51.609 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:50:51.609 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.609 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.610 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-18T07:50:51.610 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.610 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.610 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6813", 2024-09-18T07:50:51.610 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:50:51.610 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.610 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.610 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-18T07:50:51.610 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.611 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.611 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6815", 2024-09-18T07:50:51.611 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:50:51.611 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.611 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.611 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-18T07:50:51.611 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-18T07:50:51.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.611 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-18T07:50:51.611 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.203:6814", 2024-09-18T07:50:51.611 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2670809315 2024-09-18T07:50:51.611 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.612 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.612 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.203:6812/2670809315", 2024-09-18T07:50:51.612 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.203:6813/2670809315", 2024-09-18T07:50:51.612 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.203:6815/2670809315", 2024-09-18T07:50:51.612 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.203:6814/2670809315", 2024-09-18T07:50:51.612 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-18T07:50:51.612 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-18T07:50:51.612 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-18T07:50:51.612 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.612 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.612 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:50:51.612 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_xinfo": [ 2024-09-18T07:50:51.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.613 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-18T07:50:51.613 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:50:51.613 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:50:51.613 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:50:51.613 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:50:51.613 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-18T07:50:51.613 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:50:28.985118+0000", 2024-09-18T07:50:51.613 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:50:51.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.613 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-18T07:50:51.613 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:50:51.614 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:50:51.614 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:50:51.614 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:50:51.614 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:50:51.614 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:50:50.669669+0000", 2024-09-18T07:50:51.614 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:50:51.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.614 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-18T07:50:51.614 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:50:51.614 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:50:51.614 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:50:51.614 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:50:51.614 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:50:51.615 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:49:56.734590+0000", 2024-09-18T07:50:51.615 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:50:51.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.615 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-18T07:50:51.615 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:50:51.615 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:50:51.615 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:50:51.615 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:50:51.615 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:50:51.615 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:50:29.752954+0000", 2024-09-18T07:50:51.615 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:50:51.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.616 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-18T07:50:51.616 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:50:51.616 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:50:51.616 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:50:51.616 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:50:51.616 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:50:51.616 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:50:36.329500+0000", 2024-09-18T07:50:51.616 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 358 2024-09-18T07:50:51.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.616 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-18T07:50:51.616 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:50:51.616 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:50:51.617 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:50:51.617 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:50:51.617 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:50:51.617 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:49:35.188048+0000", 2024-09-18T07:50:51.617 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 415 2024-09-18T07:50:51.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.617 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-18T07:50:51.617 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:50:51.617 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:50:51.617 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:50:51.617 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:50:51.617 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:50:51.618 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:50:18.964683+0000", 2024-09-18T07:50:51.618 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:50:51.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.618 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-18T07:50:51.618 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-18T07:50:51.618 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-18T07:50:51.618 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-18T07:50:51.618 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540719139924082687, 2024-09-18T07:50:51.618 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-18T07:50:51.618 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-18T07:49:39.288824+0000", 2024-09-18T07:50:51.618 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-18T07:50:51.618 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.618 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:50:51.619 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap": [ 2024-09-18T07:50:51.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.619 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.1e", 2024-09-18T07:50:51.619 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-18T07:50:51.619 INFO:teuthology.orchestra.run.smithi137.stdout: 7, 2024-09-18T07:50:51.619 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-18T07:50:51.619 INFO:teuthology.orchestra.run.smithi137.stdout: 4 2024-09-18T07:50:51.619 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.619 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.619 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:50:51.619 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_items": [ 2024-09-18T07:50:51.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.619 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.13", 2024-09-18T07:50:51.619 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-18T07:50:51.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-18T07:50:51.620 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 6, 2024-09-18T07:50:51.620 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 7 2024-09-18T07:50:51.620 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.620 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-18T07:50:51.620 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.620 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-18T07:50:51.620 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_primaries": [], 2024-09-18T07:50:51.620 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_temp": [], 2024-09-18T07:50:51.620 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_temp": [], 2024-09-18T07:50:51.620 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklist": { 2024-09-18T07:50:51.620 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/2096039404": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:50:51.620 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/3636177939": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:50:51.621 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/334967443": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:50:51.621 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2101286111": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:50:51.621 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3387637412": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:50:51.621 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762989503": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:50:51.621 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3490817671": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:50:51.621 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1482147392": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:50:51.621 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1821482181": "2024-09-19T07:25:41.661328+0000", 2024-09-18T07:50:51.621 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/4028304023": "2024-09-19T07:26:06.595443+0000", 2024-09-18T07:50:51.621 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/621434521": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:50:51.621 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3160704935": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:50:51.621 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2357293499": "2024-09-19T07:31:25.389124+0000", 2024-09-18T07:50:51.621 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1762352860": "2024-09-19T07:31:25.389124+0000" 2024-09-18T07:50:51.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.622 INFO:teuthology.orchestra.run.smithi137.stdout: "range_blocklist": {}, 2024-09-18T07:50:51.622 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profiles": { 2024-09-18T07:50:51.622 INFO:teuthology.orchestra.run.smithi137.stdout: "default": { 2024-09-18T07:50:51.622 INFO:teuthology.orchestra.run.smithi137.stdout: "crush-failure-domain": "osd", 2024-09-18T07:50:51.622 INFO:teuthology.orchestra.run.smithi137.stdout: "k": "2", 2024-09-18T07:50:51.622 INFO:teuthology.orchestra.run.smithi137.stdout: "m": "1", 2024-09-18T07:50:51.622 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin": "jerasure", 2024-09-18T07:50:51.622 INFO:teuthology.orchestra.run.smithi137.stdout: "technique": "reed_sol_van" 2024-09-18T07:50:51.622 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-18T07:50:51.622 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps_queue": [], 2024-09-18T07:50:51.622 INFO:teuthology.orchestra.run.smithi137.stdout: "new_removed_snaps": [], 2024-09-18T07:50:51.622 INFO:teuthology.orchestra.run.smithi137.stdout: "new_purged_snaps": [], 2024-09-18T07:50:51.622 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_node_flags": {}, 2024-09-18T07:50:51.623 INFO:teuthology.orchestra.run.smithi137.stdout: "device_class_flags": {}, 2024-09-18T07:50:51.623 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode": { 2024-09-18T07:50:51.623 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_enabled": false, 2024-09-18T07:50:51.623 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_bucket_count": 0, 2024-09-18T07:50:51.623 INFO:teuthology.orchestra.run.smithi137.stdout: "degraded_stretch_mode": 0, 2024-09-18T07:50:51.623 INFO:teuthology.orchestra.run.smithi137.stdout: "recovering_stretch_mode": 0, 2024-09-18T07:50:51.623 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_bucket": 0 2024-09-18T07:50:51.623 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-18T07:50:51.623 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-18T07:50:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:51 smithi203 ceph-mon[32237]: pgmap v1073: 142 pgs: 142 active+clean; 583 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-18T07:50:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:51 smithi203 ceph-mon[32237]: 6.1f scrub starts 2024-09-18T07:50:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:51 smithi203 ceph-mon[32237]: 6.1f scrub ok 2024-09-18T07:50:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:51 smithi203 ceph-mon[32237]: 5.1d scrub starts 2024-09-18T07:50:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:51 smithi203 ceph-mon[32237]: 5.1d scrub ok 2024-09-18T07:50:52.336 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:50:52.336 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:50:52.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:52 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:50:52.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:52 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:50:52.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:52 smithi137 ceph-mon[29408]: osdmap e563: 8 total, 8 up, 7 in 2024-09-18T07:50:52.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:52 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4128042022' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:50:52.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:52 smithi137 ceph-mon[29408]: pgmap v1076: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-18T07:50:52.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:52 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:50:52.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:52 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:50:52.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:52 smithi137 ceph-mon[39175]: osdmap e563: 8 total, 8 up, 7 in 2024-09-18T07:50:52.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:52 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4128042022' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:50:52.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:52 smithi137 ceph-mon[39175]: pgmap v1076: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-18T07:50:52.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:52 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:50:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:52 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:50:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:52 smithi203 ceph-mon[32237]: osdmap e563: 8 total, 8 up, 7 in 2024-09-18T07:50:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:52 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4128042022' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-18T07:50:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:52 smithi203 ceph-mon[32237]: pgmap v1076: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-18T07:50:53.480 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-18T07:50:53.653 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:53 smithi137 ceph-mon[29408]: 5.11 scrub starts 2024-09-18T07:50:53.653 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:53 smithi137 ceph-mon[29408]: 5.11 scrub ok 2024-09-18T07:50:53.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:53 smithi137 ceph-mon[39175]: 5.11 scrub starts 2024-09-18T07:50:53.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:53 smithi137 ceph-mon[39175]: 5.11 scrub ok 2024-09-18T07:50:53.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:53 smithi203 ceph-mon[32237]: 5.11 scrub starts 2024-09-18T07:50:53.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:53 smithi203 ceph-mon[32237]: 5.11 scrub ok 2024-09-18T07:50:54.127 INFO:tasks.thrashosds.thrasher:j is {'epoch': 563, 'fsid': 'f7ef099c-758e-11ef-bceb-c7b262605968', 'created': '2024-09-18T07:25:19.662367+0000', 'modified': '2024-09-18T07:50:51.380796+0000', 'last_up_change': '2024-09-18T07:47:10.160431+0000', 'last_in_change': '2024-09-18T07:50:22.246016+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T07:28:20.899655+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '531', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '272', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 1, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 0.7309513092041016, 'average_primary_affinity': 0.8399999737739563, 'average_primary_affinity_weighted': 0.7300000190734863}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-18T07:30:53.715373+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.7309513092041016, 'average_primary_affinity': 0.8399999737739563, 'average_primary_affinity_weighted': 0.7300000190734863}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-18T07:30:54.672127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.7309513092041016, 'average_primary_affinity': 0.8399999737739563, 'average_primary_affinity_weighted': 0.7300000190734863}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-18T07:30:56.351343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4100000858306885, 'score_stable': 2.4100000858306885, 'optimal_score': 1, 'raw_score_acting': 2.4100000858306885, 'raw_score_stable': 2.4100000858306885, 'primary_affinity_weighted': 0.7309513092041016, 'average_primary_affinity': 0.8399999737739563, 'average_primary_affinity_weighted': 0.7300000190734863}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-18T07:30:58.322401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.7309513092041016, 'average_primary_affinity': 0.8399999737739563, 'average_primary_affinity_weighted': 0.7300000190734863}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-18T07:31:00.390499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.7309513092041016, 'average_primary_affinity': 0.8399999737739563, 'average_primary_affinity_weighted': 0.7300000190734863}}], 'osds': [{'osd': 0, 'uuid': '268cd580-c931-4992-bb93-d6f63c2bfe88', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.5107269287109375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 521, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6801', 'nonce': 2811444477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6802', 'nonce': 2811444477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6804', 'nonce': 2811444477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6803', 'nonce': 2811444477}]}, 'public_addr': '172.21.15.137:6801/2811444477', 'cluster_addr': '172.21.15.137:6802/2811444477', 'heartbeat_back_addr': '172.21.15.137:6804/2811444477', 'heartbeat_front_addr': '172.21.15.137:6803/2811444477', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b06907bc-b499-46c0-8667-c5bdc0d85575', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 560, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6805', 'nonce': 874102487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6806', 'nonce': 874102487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6808', 'nonce': 874102487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6807', 'nonce': 874102487}]}, 'public_addr': '172.21.15.137:6805/874102487', 'cluster_addr': '172.21.15.137:6806/874102487', 'heartbeat_back_addr': '172.21.15.137:6808/874102487', 'heartbeat_front_addr': '172.21.15.137:6807/874102487', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '480ad6c5-0e0b-4405-bbe2-381682d00bae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 560, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6809', 'nonce': 1754684686}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6810', 'nonce': 1754684686}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6812', 'nonce': 1754684686}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6811', 'nonce': 1754684686}]}, 'public_addr': '172.21.15.137:6809/1754684686', 'cluster_addr': '172.21.15.137:6810/1754684686', 'heartbeat_back_addr': '172.21.15.137:6812/1754684686', 'heartbeat_front_addr': '172.21.15.137:6811/1754684686', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '178be3e0-f07b-4856-807b-2a1181e48a14', 'up': 1, 'in': 1, 'weight': 0.885101318359375, 'primary_affinity': 0.759490966796875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 560, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6813', 'nonce': 116058392}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6814', 'nonce': 116058392}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6816', 'nonce': 116058392}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6815', 'nonce': 116058392}]}, 'public_addr': '172.21.15.137:6813/116058392', 'cluster_addr': '172.21.15.137:6814/116058392', 'heartbeat_back_addr': '172.21.15.137:6816/116058392', 'heartbeat_front_addr': '172.21.15.137:6815/116058392', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7547102c-951a-4077-a094-fb8a65bfe6de', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 32, 'last_clean_end': 358, 'up_from': 395, 'up_thru': 559, 'down_at': 359, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6800', 'nonce': 571224280}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6801', 'nonce': 571224280}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6803', 'nonce': 571224280}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6802', 'nonce': 571224280}]}, 'public_addr': '172.21.15.203:6800/571224280', 'cluster_addr': '172.21.15.203:6801/571224280', 'heartbeat_back_addr': '172.21.15.203:6803/571224280', 'heartbeat_front_addr': '172.21.15.203:6802/571224280', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '49c17901-8102-48a4-9348-fa1bdb63032c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.577392578125, 'last_clean_begin': 39, 'last_clean_end': 415, 'up_from': 460, 'up_thru': 560, 'down_at': 416, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6804', 'nonce': 1993911072}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6805', 'nonce': 1993911072}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6807', 'nonce': 1993911072}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6806', 'nonce': 1993911072}]}, 'public_addr': '172.21.15.203:6804/1993911072', 'cluster_addr': '172.21.15.203:6805/1993911072', 'heartbeat_back_addr': '172.21.15.203:6807/1993911072', 'heartbeat_front_addr': '172.21.15.203:6806/1993911072', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '75f0ead4-e981-4a15-b1c1-d83dce871ab6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 560, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6808', 'nonce': 1494175466}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6809', 'nonce': 1494175466}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6811', 'nonce': 1494175466}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6810', 'nonce': 1494175466}]}, 'public_addr': '172.21.15.203:6808/1494175466', 'cluster_addr': '172.21.15.203:6809/1494175466', 'heartbeat_back_addr': '172.21.15.203:6811/1494175466', 'heartbeat_front_addr': '172.21.15.203:6810/1494175466', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c0dbf299-6798-4b99-9a85-62b123971f2a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 55, 'up_thru': 560, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6812', 'nonce': 2670809315}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6813', 'nonce': 2670809315}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6815', 'nonce': 2670809315}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.203:6814', 'nonce': 2670809315}]}, 'public_addr': '172.21.15.203:6812/2670809315', 'cluster_addr': '172.21.15.203:6813/2670809315', 'heartbeat_back_addr': '172.21.15.203:6815/2670809315', 'heartbeat_front_addr': '172.21.15.203:6814/2670809315', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-18T07:50:28.985118+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:50:50.669669+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:49:56.734590+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:50:29.752954+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:50:36.329500+0000', 'dead_epoch': 358}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:49:35.188048+0000', 'dead_epoch': 415}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:50:18.964683+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-18T07:49:39.288824+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.1e', 'osds': [7, 1, 4]}], 'pg_upmap_items': [{'pgid': '4.13', 'mappings': [{'from': 6, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.137:6800/2096039404': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:6800/3636177939': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/334967443': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/2101286111': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/3387637412': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1762989503': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3490817671': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:6800/1482147392': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/1821482181': '2024-09-19T07:25:41.661328+0000', '172.21.15.137:0/4028304023': '2024-09-19T07:26:06.595443+0000', '172.21.15.137:0/621434521': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/3160704935': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/2357293499': '2024-09-19T07:31:25.389124+0000', '172.21.15.137:0/1762352860': '2024-09-19T07:31:25.389124+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-18T07:50:54.127 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:50:54.165 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:54.407 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-18T07:50:54.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:54 smithi137 ceph-mon[39175]: 2.2 scrub starts 2024-09-18T07:50:54.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:54 smithi137 ceph-mon[39175]: 2.2 scrub ok 2024-09-18T07:50:54.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:54 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1568414073' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:50:54.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:54 smithi137 ceph-mon[39175]: pgmap v1077: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-18T07:50:54.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:54 smithi137 ceph-mon[29408]: 2.2 scrub starts 2024-09-18T07:50:54.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:54 smithi137 ceph-mon[29408]: 2.2 scrub ok 2024-09-18T07:50:54.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:54 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1568414073' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:50:54.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:54 smithi137 ceph-mon[29408]: pgmap v1077: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-18T07:50:54.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:54 smithi203 ceph-mon[32237]: 2.2 scrub starts 2024-09-18T07:50:54.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:54 smithi203 ceph-mon[32237]: 2.2 scrub ok 2024-09-18T07:50:54.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:54 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1568414073' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-18T07:50:54.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:54 smithi203 ceph-mon[32237]: pgmap v1077: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-18T07:50:54.801 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:50:54.801 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:50:55.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-18T07:50:55.570 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:55.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:55 smithi137 ceph-mon[29408]: 5.6 scrub starts 2024-09-18T07:50:55.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:55 smithi137 ceph-mon[29408]: 5.6 scrub ok 2024-09-18T07:50:55.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:55 smithi137 ceph-mon[29408]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:50:55.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:55 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1568414073' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:50:55.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:55 smithi137 ceph-mon[29408]: osdmap e564: 8 total, 8 up, 7 in 2024-09-18T07:50:55.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:55 smithi137 ceph-mon[29408]: 1.6 scrub starts 2024-09-18T07:50:55.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:55 smithi137 ceph-mon[29408]: 1.6 scrub ok 2024-09-18T07:50:55.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:55 smithi137 ceph-mon[29408]: osdmap e565: 8 total, 8 up, 7 in 2024-09-18T07:50:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:55 smithi137 ceph-mon[39175]: 5.6 scrub starts 2024-09-18T07:50:55.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:55 smithi137 ceph-mon[39175]: 5.6 scrub ok 2024-09-18T07:50:55.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:55 smithi137 ceph-mon[39175]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:50:55.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:55 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1568414073' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:50:55.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:55 smithi137 ceph-mon[39175]: osdmap e564: 8 total, 8 up, 7 in 2024-09-18T07:50:55.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:55 smithi137 ceph-mon[39175]: 1.6 scrub starts 2024-09-18T07:50:55.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:55 smithi137 ceph-mon[39175]: 1.6 scrub ok 2024-09-18T07:50:55.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:55 smithi137 ceph-mon[39175]: osdmap e565: 8 total, 8 up, 7 in 2024-09-18T07:50:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:55 smithi203 ceph-mon[32237]: 5.6 scrub starts 2024-09-18T07:50:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:55 smithi203 ceph-mon[32237]: 5.6 scrub ok 2024-09-18T07:50:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:55 smithi203 ceph-mon[32237]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-18T07:50:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:55 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1568414073' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-18T07:50:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:55 smithi203 ceph-mon[32237]: osdmap e564: 8 total, 8 up, 7 in 2024-09-18T07:50:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:55 smithi203 ceph-mon[32237]: 1.6 scrub starts 2024-09-18T07:50:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:55 smithi203 ceph-mon[32237]: 1.6 scrub ok 2024-09-18T07:50:55.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:55 smithi203 ceph-mon[32237]: osdmap e565: 8 total, 8 up, 7 in 2024-09-18T07:50:56.521 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:56.643 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:56 smithi137 ceph-mon[29408]: pgmap v1080: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 892 B/s rd, 0 op/s 2024-09-18T07:50:56.643 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:56 smithi137 ceph-mon[39175]: pgmap v1080: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 892 B/s rd, 0 op/s 2024-09-18T07:50:56.690 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:50:56.692 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:50:56.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:56 smithi203 ceph-mon[32237]: pgmap v1080: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 892 B/s rd, 0 op/s 2024-09-18T07:50:57.409 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:50:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:50:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:50:57.500 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":1080,"stamp":"2024-09-18T07:50:55.916383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2041,"num_read_kb":1793,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":698,"num_bytes_recovered":609743,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1688,"ondisk_log_size":1688,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":430,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5388212,"kb_used_data":635168,"kb_used_omap":18,"kb_used_meta":4752877,"kb_avail":650827852,"statfs":{"total":671965249536,"available":666447720448,"internally_reserved":0,"allocated":650412032,"data_stored":647203873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19291,"internal_metadata":4866946213},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.733499"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":911,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503652+0000","last_change":"2024-09-18T07:50:14.213839+0000","last_active":"2024-09-18T07:50:51.503652+0000","last_peered":"2024-09-18T07:50:51.503652+0000","last_clean":"2024-09-18T07:50:51.503652+0000","last_became_active":"2024-09-18T07:50:14.213544+0000","last_became_peered":"2024-09-18T07:50:14.213544+0000","last_unstale":"2024-09-18T07:50:51.503652+0000","last_undegraded":"2024-09-18T07:50:51.503652+0000","last_fullsized":"2024-09-18T07:50:51.503652+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:46.455865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:46.455865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:11.769962+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"323'15","reported_seq":1036,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503517+0000","last_change":"2024-09-18T07:47:20.194767+0000","last_active":"2024-09-18T07:50:51.503517+0000","last_peered":"2024-09-18T07:50:51.503517+0000","last_clean":"2024-09-18T07:50:51.503517+0000","last_became_active":"2024-09-18T07:47:20.136254+0000","last_became_peered":"2024-09-18T07:47:20.136254+0000","last_unstale":"2024-09-18T07:50:51.503517+0000","last_undegraded":"2024-09-18T07:50:51.503517+0000","last_fullsized":"2024-09-18T07:50:51.503517+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:47:09.864550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:09.864550+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":717,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392289+0000","last_change":"2024-09-18T07:50:14.215869+0000","last_active":"2024-09-18T07:50:51.392289+0000","last_peered":"2024-09-18T07:50:51.392289+0000","last_clean":"2024-09-18T07:50:51.392289+0000","last_became_active":"2024-09-18T07:50:14.215498+0000","last_became_peered":"2024-09-18T07:50:14.215498+0000","last_unstale":"2024-09-18T07:50:51.392289+0000","last_undegraded":"2024-09-18T07:50:51.392289+0000","last_fullsized":"2024-09-18T07:50:51.392289+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:05.093789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:05.093789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":929,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505320+0000","last_change":"2024-09-18T07:49:32.689133+0000","last_active":"2024-09-18T07:50:51.505320+0000","last_peered":"2024-09-18T07:50:51.505320+0000","last_clean":"2024-09-18T07:50:51.505320+0000","last_became_active":"2024-09-18T07:49:32.688553+0000","last_became_peered":"2024-09-18T07:49:32.688553+0000","last_unstale":"2024-09-18T07:50:51.505320+0000","last_undegraded":"2024-09-18T07:50:51.505320+0000","last_fullsized":"2024-09-18T07:50:51.505320+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:14.884604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:14.884604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":47,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.397438+0000","last_change":"2024-09-18T07:50:23.285610+0000","last_active":"2024-09-18T07:50:51.397438+0000","last_peered":"2024-09-18T07:50:51.397438+0000","last_clean":"2024-09-18T07:50:51.397438+0000","last_became_active":"2024-09-18T07:50:23.285220+0000","last_became_peered":"2024-09-18T07:50:23.285220+0000","last_unstale":"2024-09-18T07:50:51.397438+0000","last_undegraded":"2024-09-18T07:50:51.397438+0000","last_fullsized":"2024-09-18T07:50:51.397438+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:11.545358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:11.545358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:28.594000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":47,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.397010+0000","last_change":"2024-09-18T07:50:23.292723+0000","last_active":"2024-09-18T07:50:51.397010+0000","last_peered":"2024-09-18T07:50:51.397010+0000","last_clean":"2024-09-18T07:50:51.397010+0000","last_became_active":"2024-09-18T07:50:23.292434+0000","last_became_peered":"2024-09-18T07:50:23.292434+0000","last_unstale":"2024-09-18T07:50:51.397010+0000","last_undegraded":"2024-09-18T07:50:51.397010+0000","last_fullsized":"2024-09-18T07:50:51.397010+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:57.674688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:57.674688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"323'27","reported_seq":691,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503293+0000","last_change":"2024-09-18T07:50:23.319419+0000","last_active":"2024-09-18T07:50:51.503293+0000","last_peered":"2024-09-18T07:50:51.503293+0000","last_clean":"2024-09-18T07:50:51.503293+0000","last_became_active":"2024-09-18T07:50:23.269846+0000","last_became_peered":"2024-09-18T07:50:23.269846+0000","last_unstale":"2024-09-18T07:50:51.503293+0000","last_undegraded":"2024-09-18T07:50:51.503293+0000","last_fullsized":"2024-09-18T07:50:51.503293+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:46:54.956330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:54.956330+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:25.107782+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":985,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503507+0000","last_change":"2024-09-18T07:47:52.060299+0000","last_active":"2024-09-18T07:50:51.503507+0000","last_peered":"2024-09-18T07:50:51.503507+0000","last_clean":"2024-09-18T07:50:51.503507+0000","last_became_active":"2024-09-18T07:47:20.136358+0000","last_became_peered":"2024-09-18T07:47:20.136358+0000","last_unstale":"2024-09-18T07:50:51.503507+0000","last_undegraded":"2024-09-18T07:50:51.503507+0000","last_fullsized":"2024-09-18T07:50:51.503507+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:52.060246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.060246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":878,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.506543+0000","last_change":"2024-09-18T07:50:14.226012+0000","last_active":"2024-09-18T07:50:51.506543+0000","last_peered":"2024-09-18T07:50:51.506543+0000","last_clean":"2024-09-18T07:50:51.506543+0000","last_became_active":"2024-09-18T07:50:14.225616+0000","last_became_peered":"2024-09-18T07:50:14.225616+0000","last_unstale":"2024-09-18T07:50:51.506543+0000","last_undegraded":"2024-09-18T07:50:51.506543+0000","last_fullsized":"2024-09-18T07:50:51.506543+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:34.470475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:34.470475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"323'15","reported_seq":1033,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505672+0000","last_change":"2024-09-18T07:50:23.399381+0000","last_active":"2024-09-18T07:50:51.505672+0000","last_peered":"2024-09-18T07:50:51.505672+0000","last_clean":"2024-09-18T07:50:51.505672+0000","last_became_active":"2024-09-18T07:50:23.279365+0000","last_became_peered":"2024-09-18T07:50:23.279365+0000","last_unstale":"2024-09-18T07:50:51.505672+0000","last_undegraded":"2024-09-18T07:50:51.505672+0000","last_fullsized":"2024-09-18T07:50:51.505672+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:47:47.157636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:47.157636+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":936,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504059+0000","last_change":"2024-09-18T07:50:46.629310+0000","last_active":"2024-09-18T07:50:51.504059+0000","last_peered":"2024-09-18T07:50:51.504059+0000","last_clean":"2024-09-18T07:50:51.504059+0000","last_became_active":"2024-09-18T07:50:05.209393+0000","last_became_peered":"2024-09-18T07:50:05.209393+0000","last_unstale":"2024-09-18T07:50:51.504059+0000","last_undegraded":"2024-09-18T07:50:51.504059+0000","last_fullsized":"2024-09-18T07:50:51.504059+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:46.629242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:50:46.629242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:05.584437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1045,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.391915+0000","last_change":"2024-09-18T07:50:05.201867+0000","last_active":"2024-09-18T07:50:51.391915+0000","last_peered":"2024-09-18T07:50:51.391915+0000","last_clean":"2024-09-18T07:50:51.391915+0000","last_became_active":"2024-09-18T07:50:05.199748+0000","last_became_peered":"2024-09-18T07:50:05.199748+0000","last_unstale":"2024-09-18T07:50:51.391915+0000","last_undegraded":"2024-09-18T07:50:51.391915+0000","last_fullsized":"2024-09-18T07:50:51.391915+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:55.019244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:55.019244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":863,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505037+0000","last_change":"2024-09-18T07:50:14.212267+0000","last_active":"2024-09-18T07:50:51.505037+0000","last_peered":"2024-09-18T07:50:51.505037+0000","last_clean":"2024-09-18T07:50:51.505037+0000","last_became_active":"2024-09-18T07:50:14.211977+0000","last_became_peered":"2024-09-18T07:50:14.211977+0000","last_unstale":"2024-09-18T07:50:51.505037+0000","last_undegraded":"2024-09-18T07:50:51.505037+0000","last_fullsized":"2024-09-18T07:50:51.505037+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:41.617037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:41.617037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:03.818785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"323'7","reported_seq":1088,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505783+0000","last_change":"2024-09-18T07:50:23.333287+0000","last_active":"2024-09-18T07:50:51.505783+0000","last_peered":"2024-09-18T07:50:51.505783+0000","last_clean":"2024-09-18T07:50:51.505783+0000","last_became_active":"2024-09-18T07:50:23.276419+0000","last_became_peered":"2024-09-18T07:50:23.276419+0000","last_unstale":"2024-09-18T07:50:51.505783+0000","last_undegraded":"2024-09-18T07:50:51.505783+0000","last_fullsized":"2024-09-18T07:50:51.505783+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'7","last_scrub_stamp":"2024-09-18T07:42:46.496916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:46.496916+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:33.893060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":851,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392364+0000","last_change":"2024-09-18T07:47:20.143380+0000","last_active":"2024-09-18T07:50:51.392364+0000","last_peered":"2024-09-18T07:50:51.392364+0000","last_clean":"2024-09-18T07:50:51.392364+0000","last_became_active":"2024-09-18T07:47:20.142891+0000","last_became_peered":"2024-09-18T07:47:20.142891+0000","last_unstale":"2024-09-18T07:50:51.392364+0000","last_undegraded":"2024-09-18T07:50:51.392364+0000","last_fullsized":"2024-09-18T07:50:51.392364+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:53.754622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:53.754622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:18.411088+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":985,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503979+0000","last_change":"2024-09-18T07:50:23.271566+0000","last_active":"2024-09-18T07:50:51.503979+0000","last_peered":"2024-09-18T07:50:51.503979+0000","last_clean":"2024-09-18T07:50:51.503979+0000","last_became_active":"2024-09-18T07:50:23.271118+0000","last_became_peered":"2024-09-18T07:50:23.271118+0000","last_unstale":"2024-09-18T07:50:51.503979+0000","last_undegraded":"2024-09-18T07:50:51.503979+0000","last_fullsized":"2024-09-18T07:50:51.503979+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:48.995794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:48.995794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:28.610063+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1010,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505703+0000","last_change":"2024-09-18T07:48:31.926895+0000","last_active":"2024-09-18T07:50:51.505703+0000","last_peered":"2024-09-18T07:50:51.505703+0000","last_clean":"2024-09-18T07:50:51.505703+0000","last_became_active":"2024-09-18T07:47:11.187916+0000","last_became_peered":"2024-09-18T07:47:11.187916+0000","last_unstale":"2024-09-18T07:50:51.505703+0000","last_undegraded":"2024-09-18T07:50:51.505703+0000","last_fullsized":"2024-09-18T07:50:51.505703+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:31.926847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:31.926847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":791,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505587+0000","last_change":"2024-09-18T07:50:05.207363+0000","last_active":"2024-09-18T07:50:51.505587+0000","last_peered":"2024-09-18T07:50:51.505587+0000","last_clean":"2024-09-18T07:50:51.505587+0000","last_became_active":"2024-09-18T07:50:05.206513+0000","last_became_peered":"2024-09-18T07:50:05.206513+0000","last_unstale":"2024-09-18T07:50:51.505587+0000","last_undegraded":"2024-09-18T07:50:51.505587+0000","last_fullsized":"2024-09-18T07:50:51.505587+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:49.150795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:49.150795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"323'27","reported_seq":1256,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503831+0000","last_change":"2024-09-18T07:50:23.360874+0000","last_active":"2024-09-18T07:50:51.503831+0000","last_peered":"2024-09-18T07:50:51.503831+0000","last_clean":"2024-09-18T07:50:51.503831+0000","last_became_active":"2024-09-18T07:50:23.296232+0000","last_became_peered":"2024-09-18T07:50:23.296232+0000","last_unstale":"2024-09-18T07:50:51.503831+0000","last_undegraded":"2024-09-18T07:50:51.503831+0000","last_fullsized":"2024-09-18T07:50:51.503831+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:49:43.979116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:49:43.979116+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:09.363799+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":654,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.506238+0000","last_change":"2024-09-18T07:50:50.850780+0000","last_active":"2024-09-18T07:50:51.506238+0000","last_peered":"2024-09-18T07:50:51.506238+0000","last_clean":"2024-09-18T07:50:51.506238+0000","last_became_active":"2024-09-18T07:50:05.205190+0000","last_became_peered":"2024-09-18T07:50:05.205190+0000","last_unstale":"2024-09-18T07:50:51.506238+0000","last_undegraded":"2024-09-18T07:50:51.506238+0000","last_fullsized":"2024-09-18T07:50:51.506238+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:50.850725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:50:50.850725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:18.069400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":960,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.507171+0000","last_change":"2024-09-18T07:50:50.672987+0000","last_active":"2024-09-18T07:50:51.507171+0000","last_peered":"2024-09-18T07:50:51.507171+0000","last_clean":"2024-09-18T07:50:51.507171+0000","last_became_active":"2024-09-18T07:50:23.286001+0000","last_became_peered":"2024-09-18T07:50:23.286001+0000","last_unstale":"2024-09-18T07:50:51.507171+0000","last_undegraded":"2024-09-18T07:50:51.507171+0000","last_fullsized":"2024-09-18T07:50:51.507171+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:50.672909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:50:50.672909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:02.138407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1001,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503902+0000","last_change":"2024-09-18T07:47:56.099010+0000","last_active":"2024-09-18T07:50:51.503902+0000","last_peered":"2024-09-18T07:50:51.503902+0000","last_clean":"2024-09-18T07:50:51.503902+0000","last_became_active":"2024-09-18T07:47:20.144620+0000","last_became_peered":"2024-09-18T07:47:20.144620+0000","last_unstale":"2024-09-18T07:50:51.503902+0000","last_undegraded":"2024-09-18T07:50:51.503902+0000","last_fullsized":"2024-09-18T07:50:51.503902+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:56.098927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.098927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"323'18","reported_seq":386,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.506332+0000","last_change":"2024-09-18T07:50:23.304479+0000","last_active":"2024-09-18T07:50:51.506332+0000","last_peered":"2024-09-18T07:50:51.506332+0000","last_clean":"2024-09-18T07:50:51.506332+0000","last_became_active":"2024-09-18T07:50:23.271770+0000","last_became_peered":"2024-09-18T07:50:23.271770+0000","last_unstale":"2024-09-18T07:50:51.506332+0000","last_undegraded":"2024-09-18T07:50:51.506332+0000","last_fullsized":"2024-09-18T07:50:51.506332+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:47:53.905251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.905251+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":929,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504731+0000","last_change":"2024-09-18T07:50:23.273104+0000","last_active":"2024-09-18T07:50:51.504731+0000","last_peered":"2024-09-18T07:50:51.504731+0000","last_clean":"2024-09-18T07:50:51.504731+0000","last_became_active":"2024-09-18T07:50:23.271208+0000","last_became_peered":"2024-09-18T07:50:23.271208+0000","last_unstale":"2024-09-18T07:50:51.504731+0000","last_undegraded":"2024-09-18T07:50:51.504731+0000","last_fullsized":"2024-09-18T07:50:51.504731+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:37.930435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:39:37.930435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:26.920511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":849,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392710+0000","last_change":"2024-09-18T07:47:53.034043+0000","last_active":"2024-09-18T07:50:51.392710+0000","last_peered":"2024-09-18T07:50:51.392710+0000","last_clean":"2024-09-18T07:50:51.392710+0000","last_became_active":"2024-09-18T07:47:11.194447+0000","last_became_peered":"2024-09-18T07:47:11.194447+0000","last_unstale":"2024-09-18T07:50:51.392710+0000","last_undegraded":"2024-09-18T07:50:51.392710+0000","last_fullsized":"2024-09-18T07:50:51.392710+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:47:53.033975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.033975+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":59,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.398212+0000","last_change":"2024-09-18T07:50:38.599737+0000","last_active":"2024-09-18T07:50:51.398212+0000","last_peered":"2024-09-18T07:50:51.398212+0000","last_clean":"2024-09-18T07:50:51.398212+0000","last_became_active":"2024-09-18T07:50:23.283549+0000","last_became_peered":"2024-09-18T07:50:23.283549+0000","last_unstale":"2024-09-18T07:50:51.398212+0000","last_undegraded":"2024-09-18T07:50:51.398212+0000","last_fullsized":"2024-09-18T07:50:51.398212+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:50:38.599674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:50:38.599674+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:47.809323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"323'20","reported_seq":845,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.393169+0000","last_change":"2024-09-18T07:50:05.228529+0000","last_active":"2024-09-18T07:50:51.393169+0000","last_peered":"2024-09-18T07:50:51.393169+0000","last_clean":"2024-09-18T07:50:51.393169+0000","last_became_active":"2024-09-18T07:50:05.199488+0000","last_became_peered":"2024-09-18T07:50:05.199488+0000","last_unstale":"2024-09-18T07:50:51.393169+0000","last_undegraded":"2024-09-18T07:50:51.393169+0000","last_fullsized":"2024-09-18T07:50:51.393169+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:48:19.962346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:19.962346+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"398'14","reported_seq":869,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.507233+0000","last_change":"2024-09-18T07:49:41.853017+0000","last_active":"2024-09-18T07:50:51.507233+0000","last_peered":"2024-09-18T07:50:51.507233+0000","last_clean":"2024-09-18T07:50:51.507233+0000","last_became_active":"2024-09-18T07:44:43.786594+0000","last_became_peered":"2024-09-18T07:44:43.786594+0000","last_unstale":"2024-09-18T07:50:51.507233+0000","last_undegraded":"2024-09-18T07:50:51.507233+0000","last_fullsized":"2024-09-18T07:50:51.507233+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:49:41.852939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:49:41.852939+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":163,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503195+0000","last_change":"2024-09-18T07:50:17.061109+0000","last_active":"2024-09-18T07:50:51.503195+0000","last_peered":"2024-09-18T07:50:51.503195+0000","last_clean":"2024-09-18T07:50:51.503195+0000","last_became_active":"2024-09-18T07:50:14.220311+0000","last_became_peered":"2024-09-18T07:50:14.220311+0000","last_unstale":"2024-09-18T07:50:51.503195+0000","last_undegraded":"2024-09-18T07:50:51.503195+0000","last_fullsized":"2024-09-18T07:50:51.503195+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:50:17.061041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:50:17.061041+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:21.651657+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"323'27","reported_seq":63,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.396328+0000","last_change":"2024-09-18T07:50:23.322623+0000","last_active":"2024-09-18T07:50:51.396328+0000","last_peered":"2024-09-18T07:50:51.396328+0000","last_clean":"2024-09-18T07:50:51.396328+0000","last_became_active":"2024-09-18T07:50:23.284875+0000","last_became_peered":"2024-09-18T07:50:23.284875+0000","last_unstale":"2024-09-18T07:50:51.396328+0000","last_undegraded":"2024-09-18T07:50:51.396328+0000","last_fullsized":"2024-09-18T07:50:51.396328+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:42:33.391715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:42:33.391715+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:26.558888+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":47,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.396719+0000","last_change":"2024-09-18T07:50:23.299970+0000","last_active":"2024-09-18T07:50:51.396719+0000","last_peered":"2024-09-18T07:50:51.396719+0000","last_clean":"2024-09-18T07:50:51.396719+0000","last_became_active":"2024-09-18T07:50:23.298465+0000","last_became_peered":"2024-09-18T07:50:23.298465+0000","last_unstale":"2024-09-18T07:50:51.396719+0000","last_undegraded":"2024-09-18T07:50:51.396719+0000","last_fullsized":"2024-09-18T07:50:51.396719+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:51.435576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:51.435576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:59.020688+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"536'13","reported_seq":1177,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505797+0000","last_change":"2024-09-18T07:50:05.217905+0000","last_active":"2024-09-18T07:50:51.505797+0000","last_peered":"2024-09-18T07:50:51.505797+0000","last_clean":"2024-09-18T07:50:51.505797+0000","last_became_active":"2024-09-18T07:50:05.195271+0000","last_became_peered":"2024-09-18T07:50:05.195271+0000","last_unstale":"2024-09-18T07:50:51.505797+0000","last_undegraded":"2024-09-18T07:50:51.505797+0000","last_fullsized":"2024-09-18T07:50:51.505797+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'12","last_scrub_stamp":"2024-09-18T07:48:30.963258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:30.963258+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":919,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503307+0000","last_change":"2024-09-18T07:50:14.212562+0000","last_active":"2024-09-18T07:50:51.503307+0000","last_peered":"2024-09-18T07:50:51.503307+0000","last_clean":"2024-09-18T07:50:51.503307+0000","last_became_active":"2024-09-18T07:50:14.212320+0000","last_became_peered":"2024-09-18T07:50:14.212320+0000","last_unstale":"2024-09-18T07:50:51.503307+0000","last_undegraded":"2024-09-18T07:50:51.503307+0000","last_fullsized":"2024-09-18T07:50:51.503307+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:08.520361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:50:08.520361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:18.334906+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":537,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.506449+0000","last_change":"2024-09-18T07:49:43.801107+0000","last_active":"2024-09-18T07:50:51.506449+0000","last_peered":"2024-09-18T07:50:51.506449+0000","last_clean":"2024-09-18T07:50:51.506449+0000","last_became_active":"2024-09-18T07:44:43.786396+0000","last_became_peered":"2024-09-18T07:44:43.786396+0000","last_unstale":"2024-09-18T07:50:51.506449+0000","last_undegraded":"2024-09-18T07:50:51.506449+0000","last_fullsized":"2024-09-18T07:50:51.506449+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:49:43.801039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:49:43.801039+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:56.698863+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"323'30","reported_seq":65,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.397320+0000","last_change":"2024-09-18T07:50:23.410091+0000","last_active":"2024-09-18T07:50:51.397320+0000","last_peered":"2024-09-18T07:50:51.397320+0000","last_clean":"2024-09-18T07:50:51.397320+0000","last_became_active":"2024-09-18T07:50:23.299608+0000","last_became_peered":"2024-09-18T07:50:23.299608+0000","last_unstale":"2024-09-18T07:50:51.397320+0000","last_undegraded":"2024-09-18T07:50:51.397320+0000","last_fullsized":"2024-09-18T07:50:51.397320+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'30","last_scrub_stamp":"2024-09-18T07:47:56.645991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.645991+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":810,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.393374+0000","last_change":"2024-09-18T07:50:23.297915+0000","last_active":"2024-09-18T07:50:51.393374+0000","last_peered":"2024-09-18T07:50:51.393374+0000","last_clean":"2024-09-18T07:50:51.393374+0000","last_became_active":"2024-09-18T07:50:23.296257+0000","last_became_peered":"2024-09-18T07:50:23.296257+0000","last_unstale":"2024-09-18T07:50:51.393374+0000","last_undegraded":"2024-09-18T07:50:51.393374+0000","last_fullsized":"2024-09-18T07:50:51.393374+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:56.077323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.077323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":995,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503525+0000","last_change":"2024-09-18T07:50:23.272848+0000","last_active":"2024-09-18T07:50:51.503525+0000","last_peered":"2024-09-18T07:50:51.503525+0000","last_clean":"2024-09-18T07:50:51.503525+0000","last_became_active":"2024-09-18T07:50:23.271640+0000","last_became_peered":"2024-09-18T07:50:23.271640+0000","last_unstale":"2024-09-18T07:50:51.503525+0000","last_undegraded":"2024-09-18T07:50:51.503525+0000","last_fullsized":"2024-09-18T07:50:51.503525+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:53.056484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.056484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":905,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503679+0000","last_change":"2024-09-18T07:50:47.845279+0000","last_active":"2024-09-18T07:50:51.503679+0000","last_peered":"2024-09-18T07:50:51.503679+0000","last_clean":"2024-09-18T07:50:51.503679+0000","last_became_active":"2024-09-18T07:50:23.279337+0000","last_became_peered":"2024-09-18T07:50:23.279337+0000","last_unstale":"2024-09-18T07:50:51.503679+0000","last_undegraded":"2024-09-18T07:50:51.503679+0000","last_fullsized":"2024-09-18T07:50:51.503679+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:47.845219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:50:47.845219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:10.259502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":51,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.395188+0000","last_change":"2024-09-18T07:50:23.307333+0000","last_active":"2024-09-18T07:50:51.395188+0000","last_peered":"2024-09-18T07:50:51.395188+0000","last_clean":"2024-09-18T07:50:51.395188+0000","last_became_active":"2024-09-18T07:50:23.276444+0000","last_became_peered":"2024-09-18T07:50:23.276444+0000","last_unstale":"2024-09-18T07:50:51.395188+0000","last_undegraded":"2024-09-18T07:50:51.395188+0000","last_fullsized":"2024-09-18T07:50:51.395188+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:50:14.450061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:50:14.450061+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:33.842830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"323'28","reported_seq":1009,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504210+0000","last_change":"2024-09-18T07:48:26.993988+0000","last_active":"2024-09-18T07:50:51.504210+0000","last_peered":"2024-09-18T07:50:51.504210+0000","last_clean":"2024-09-18T07:50:51.504210+0000","last_became_active":"2024-09-18T07:47:11.188002+0000","last_became_peered":"2024-09-18T07:47:11.188002+0000","last_unstale":"2024-09-18T07:50:51.504210+0000","last_undegraded":"2024-09-18T07:50:51.504210+0000","last_fullsized":"2024-09-18T07:50:51.504210+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'28","last_scrub_stamp":"2024-09-18T07:48:26.993871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:26.993871+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":83,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505997+0000","last_change":"2024-09-18T07:50:05.229977+0000","last_active":"2024-09-18T07:50:51.505997+0000","last_peered":"2024-09-18T07:50:51.505997+0000","last_clean":"2024-09-18T07:50:51.505997+0000","last_became_active":"2024-09-18T07:50:05.202682+0000","last_became_peered":"2024-09-18T07:50:05.202682+0000","last_unstale":"2024-09-18T07:50:51.505997+0000","last_undegraded":"2024-09-18T07:50:51.505997+0000","last_fullsized":"2024-09-18T07:50:51.505997+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:49:44.482296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:44.482296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:57.492382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"548'15","reported_seq":57,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.397026+0000","last_change":"2024-09-18T07:50:23.403994+0000","last_active":"2024-09-18T07:50:51.397026+0000","last_peered":"2024-09-18T07:50:51.397026+0000","last_clean":"2024-09-18T07:50:51.397026+0000","last_became_active":"2024-09-18T07:50:23.300057+0000","last_became_peered":"2024-09-18T07:50:23.300057+0000","last_unstale":"2024-09-18T07:50:51.397026+0000","last_undegraded":"2024-09-18T07:50:51.397026+0000","last_fullsized":"2024-09-18T07:50:51.397026+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'12","last_scrub_stamp":"2024-09-18T07:48:20.859791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:20.859791+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":849,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392836+0000","last_change":"2024-09-18T07:50:46.444318+0000","last_active":"2024-09-18T07:50:51.392836+0000","last_peered":"2024-09-18T07:50:51.392836+0000","last_clean":"2024-09-18T07:50:51.392836+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:50:51.392836+0000","last_undegraded":"2024-09-18T07:50:51.392836+0000","last_fullsized":"2024-09-18T07:50:51.392836+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:46.444268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:50:46.444268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:58.609003+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":799,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.506881+0000","last_change":"2024-09-18T07:50:48.664703+0000","last_active":"2024-09-18T07:50:51.506881+0000","last_peered":"2024-09-18T07:50:51.506881+0000","last_clean":"2024-09-18T07:50:51.506881+0000","last_became_active":"2024-09-18T07:48:10.151343+0000","last_became_peered":"2024-09-18T07:48:10.151343+0000","last_unstale":"2024-09-18T07:50:51.506881+0000","last_undegraded":"2024-09-18T07:50:51.506881+0000","last_fullsized":"2024-09-18T07:50:51.506881+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:50:48.664637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:50:48.664637+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:56.270460+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"323'20","reported_seq":756,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504099+0000","last_change":"2024-09-18T07:49:32.698546+0000","last_active":"2024-09-18T07:50:51.504099+0000","last_peered":"2024-09-18T07:50:51.504099+0000","last_clean":"2024-09-18T07:50:51.504099+0000","last_became_active":"2024-09-18T07:49:32.688605+0000","last_became_peered":"2024-09-18T07:49:32.688605+0000","last_unstale":"2024-09-18T07:50:51.504099+0000","last_undegraded":"2024-09-18T07:50:51.504099+0000","last_fullsized":"2024-09-18T07:50:51.504099+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:48:33.642815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:33.642815+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:20.641560+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":974,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504261+0000","last_change":"2024-09-18T07:50:23.268974+0000","last_active":"2024-09-18T07:50:51.504261+0000","last_peered":"2024-09-18T07:50:51.504261+0000","last_clean":"2024-09-18T07:50:51.504261+0000","last_became_active":"2024-09-18T07:50:23.268446+0000","last_became_peered":"2024-09-18T07:50:23.268446+0000","last_unstale":"2024-09-18T07:50:51.504261+0000","last_undegraded":"2024-09-18T07:50:51.504261+0000","last_fullsized":"2024-09-18T07:50:51.504261+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:19.210077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:19.210077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:29.643987+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"548'15","reported_seq":831,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.506243+0000","last_change":"2024-09-18T07:50:23.333125+0000","last_active":"2024-09-18T07:50:51.506243+0000","last_peered":"2024-09-18T07:50:51.506243+0000","last_clean":"2024-09-18T07:50:51.506243+0000","last_became_active":"2024-09-18T07:50:23.286461+0000","last_became_peered":"2024-09-18T07:50:23.286461+0000","last_unstale":"2024-09-18T07:50:51.506243+0000","last_undegraded":"2024-09-18T07:50:51.506243+0000","last_fullsized":"2024-09-18T07:50:51.506243+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'13","last_scrub_stamp":"2024-09-18T07:48:24.605876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:24.605876+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":47,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.397116+0000","last_change":"2024-09-18T07:50:23.293839+0000","last_active":"2024-09-18T07:50:51.397116+0000","last_peered":"2024-09-18T07:50:51.397116+0000","last_clean":"2024-09-18T07:50:51.397116+0000","last_became_active":"2024-09-18T07:50:23.293581+0000","last_became_peered":"2024-09-18T07:50:23.293581+0000","last_unstale":"2024-09-18T07:50:51.397116+0000","last_undegraded":"2024-09-18T07:50:51.397116+0000","last_fullsized":"2024-09-18T07:50:51.397116+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:46.172671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:46.172671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":813,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505486+0000","last_change":"2024-09-18T07:50:23.286117+0000","last_active":"2024-09-18T07:50:51.505486+0000","last_peered":"2024-09-18T07:50:51.505486+0000","last_clean":"2024-09-18T07:50:51.505486+0000","last_became_active":"2024-09-18T07:50:23.283668+0000","last_became_peered":"2024-09-18T07:50:23.283668+0000","last_unstale":"2024-09-18T07:50:51.505486+0000","last_undegraded":"2024-09-18T07:50:51.505486+0000","last_fullsized":"2024-09-18T07:50:51.505486+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:47:59.422927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:47:59.422927+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"323'29","reported_seq":916,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504710+0000","last_change":"2024-09-18T07:50:16.593489+0000","last_active":"2024-09-18T07:50:51.504710+0000","last_peered":"2024-09-18T07:50:51.504710+0000","last_clean":"2024-09-18T07:50:51.504710+0000","last_became_active":"2024-09-18T07:50:05.198854+0000","last_became_peered":"2024-09-18T07:50:05.198854+0000","last_unstale":"2024-09-18T07:50:51.504710+0000","last_undegraded":"2024-09-18T07:50:51.504710+0000","last_fullsized":"2024-09-18T07:50:51.504710+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'29","last_scrub_stamp":"2024-09-18T07:50:16.593349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:50:16.593349+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:43.411102+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":779,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503930+0000","last_change":"2024-09-18T07:47:20.140174+0000","last_active":"2024-09-18T07:50:51.503930+0000","last_peered":"2024-09-18T07:50:51.503930+0000","last_clean":"2024-09-18T07:50:51.503930+0000","last_became_active":"2024-09-18T07:47:20.139806+0000","last_became_peered":"2024-09-18T07:47:20.139806+0000","last_unstale":"2024-09-18T07:50:51.503930+0000","last_undegraded":"2024-09-18T07:50:51.503930+0000","last_fullsized":"2024-09-18T07:50:51.503930+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:43.985969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:43.985969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:55.976994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":720,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504674+0000","last_change":"2024-09-18T07:50:47.648131+0000","last_active":"2024-09-18T07:50:51.504674+0000","last_peered":"2024-09-18T07:50:51.504674+0000","last_clean":"2024-09-18T07:50:51.504674+0000","last_became_active":"2024-09-18T07:50:05.192195+0000","last_became_peered":"2024-09-18T07:50:05.192195+0000","last_unstale":"2024-09-18T07:50:51.504674+0000","last_undegraded":"2024-09-18T07:50:51.504674+0000","last_fullsized":"2024-09-18T07:50:51.504674+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:47.648079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:50:47.648079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:13.045741+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":708,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.507141+0000","last_change":"2024-09-18T07:49:43.912594+0000","last_active":"2024-09-18T07:50:51.507141+0000","last_peered":"2024-09-18T07:50:51.507141+0000","last_clean":"2024-09-18T07:50:51.507141+0000","last_became_active":"2024-09-18T07:47:20.138390+0000","last_became_peered":"2024-09-18T07:47:20.138390+0000","last_unstale":"2024-09-18T07:50:51.507141+0000","last_undegraded":"2024-09-18T07:50:51.507141+0000","last_fullsized":"2024-09-18T07:50:51.507141+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:43.912475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:43.912475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":70,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503053+0000","last_change":"2024-09-18T07:50:15.048190+0000","last_active":"2024-09-18T07:50:51.503053+0000","last_peered":"2024-09-18T07:50:51.503053+0000","last_clean":"2024-09-18T07:50:51.503053+0000","last_became_active":"2024-09-18T07:50:14.210684+0000","last_became_peered":"2024-09-18T07:50:14.210684+0000","last_unstale":"2024-09-18T07:50:51.503053+0000","last_undegraded":"2024-09-18T07:50:51.503053+0000","last_fullsized":"2024-09-18T07:50:51.503053+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:50:15.048136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:50:15.048136+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:24.356275+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"548'46","reported_seq":1181,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.618650+0000","last_change":"2024-09-18T07:50:23.400624+0000","last_active":"2024-09-18T07:50:51.618650+0000","last_peered":"2024-09-18T07:50:51.618650+0000","last_clean":"2024-09-18T07:50:51.618650+0000","last_became_active":"2024-09-18T07:50:23.296840+0000","last_became_peered":"2024-09-18T07:50:23.296840+0000","last_unstale":"2024-09-18T07:50:51.618650+0000","last_undegraded":"2024-09-18T07:50:51.618650+0000","last_fullsized":"2024-09-18T07:50:51.618650+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'45","last_scrub_stamp":"2024-09-18T07:49:51.025843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:49:51.025843+0000","objects_scrubbed":10,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:05.378056+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":1790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":987,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505729+0000","last_change":"2024-09-18T07:50:05.207077+0000","last_active":"2024-09-18T07:50:51.505729+0000","last_peered":"2024-09-18T07:50:51.505729+0000","last_clean":"2024-09-18T07:50:51.505729+0000","last_became_active":"2024-09-18T07:50:05.206221+0000","last_became_peered":"2024-09-18T07:50:05.206221+0000","last_unstale":"2024-09-18T07:50:51.505729+0000","last_undegraded":"2024-09-18T07:50:51.505729+0000","last_fullsized":"2024-09-18T07:50:51.505729+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:51.085400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:51.085400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":567,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504539+0000","last_change":"2024-09-18T07:50:14.227393+0000","last_active":"2024-09-18T07:50:51.504539+0000","last_peered":"2024-09-18T07:50:51.504539+0000","last_clean":"2024-09-18T07:50:51.504539+0000","last_became_active":"2024-09-18T07:50:14.227023+0000","last_became_peered":"2024-09-18T07:50:14.227023+0000","last_unstale":"2024-09-18T07:50:51.504539+0000","last_undegraded":"2024-09-18T07:50:51.504539+0000","last_fullsized":"2024-09-18T07:50:51.504539+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:48:06.127745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:48:06.127745+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":925,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.506137+0000","last_change":"2024-09-18T07:50:05.197988+0000","last_active":"2024-09-18T07:50:51.506137+0000","last_peered":"2024-09-18T07:50:51.506137+0000","last_clean":"2024-09-18T07:50:51.506137+0000","last_became_active":"2024-09-18T07:50:05.195463+0000","last_became_peered":"2024-09-18T07:50:05.195463+0000","last_unstale":"2024-09-18T07:50:51.506137+0000","last_undegraded":"2024-09-18T07:50:51.506137+0000","last_fullsized":"2024-09-18T07:50:51.506137+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:46.629196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:46.629196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":47,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.396793+0000","last_change":"2024-09-18T07:50:23.286830+0000","last_active":"2024-09-18T07:50:51.396793+0000","last_peered":"2024-09-18T07:50:51.396793+0000","last_clean":"2024-09-18T07:50:51.396793+0000","last_became_active":"2024-09-18T07:50:23.286499+0000","last_became_peered":"2024-09-18T07:50:23.286499+0000","last_unstale":"2024-09-18T07:50:51.396793+0000","last_undegraded":"2024-09-18T07:50:51.396793+0000","last_fullsized":"2024-09-18T07:50:51.396793+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:51.995541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:51.995541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:03.582186+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":910,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392094+0000","last_change":"2024-09-18T07:50:05.196938+0000","last_active":"2024-09-18T07:50:51.392094+0000","last_peered":"2024-09-18T07:50:51.392094+0000","last_clean":"2024-09-18T07:50:51.392094+0000","last_became_active":"2024-09-18T07:50:05.196606+0000","last_became_peered":"2024-09-18T07:50:05.196606+0000","last_unstale":"2024-09-18T07:50:51.392094+0000","last_undegraded":"2024-09-18T07:50:51.392094+0000","last_fullsized":"2024-09-18T07:50:51.392094+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:27.916095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:27.916095+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"323'17","reported_seq":784,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.393339+0000","last_change":"2024-09-18T07:50:14.214658+0000","last_active":"2024-09-18T07:50:51.393339+0000","last_peered":"2024-09-18T07:50:51.393339+0000","last_clean":"2024-09-18T07:50:51.393339+0000","last_became_active":"2024-09-18T07:50:14.213033+0000","last_became_peered":"2024-09-18T07:50:14.213033+0000","last_unstale":"2024-09-18T07:50:51.393339+0000","last_undegraded":"2024-09-18T07:50:51.393339+0000","last_fullsized":"2024-09-18T07:50:51.393339+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'17","last_scrub_stamp":"2024-09-18T07:47:56.037047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:56.037047+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":23,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":792,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503159+0000","last_change":"2024-09-18T07:47:20.140678+0000","last_active":"2024-09-18T07:50:51.503159+0000","last_peered":"2024-09-18T07:50:51.503159+0000","last_clean":"2024-09-18T07:50:51.503159+0000","last_became_active":"2024-09-18T07:47:20.140376+0000","last_became_peered":"2024-09-18T07:47:20.140376+0000","last_unstale":"2024-09-18T07:50:51.503159+0000","last_undegraded":"2024-09-18T07:50:51.503159+0000","last_fullsized":"2024-09-18T07:50:51.503159+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:52.917296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:52.917296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:33.150519+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":1115,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503387+0000","last_change":"2024-09-18T07:50:23.389266+0000","last_active":"2024-09-18T07:50:51.503387+0000","last_peered":"2024-09-18T07:50:51.503387+0000","last_clean":"2024-09-18T07:50:51.503387+0000","last_became_active":"2024-09-18T07:50:23.276401+0000","last_became_peered":"2024-09-18T07:50:23.276401+0000","last_unstale":"2024-09-18T07:50:51.503387+0000","last_undegraded":"2024-09-18T07:50:51.503387+0000","last_fullsized":"2024-09-18T07:50:51.503387+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:07.841406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:47:07.841406+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:32.130396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1082,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503582+0000","last_change":"2024-09-18T07:47:16.834107+0000","last_active":"2024-09-18T07:50:51.503582+0000","last_peered":"2024-09-18T07:50:51.503582+0000","last_clean":"2024-09-18T07:50:51.503582+0000","last_became_active":"2024-09-18T07:47:11.201379+0000","last_became_peered":"2024-09-18T07:47:11.201379+0000","last_unstale":"2024-09-18T07:50:51.503582+0000","last_undegraded":"2024-09-18T07:50:51.503582+0000","last_fullsized":"2024-09-18T07:50:51.503582+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:16.834043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:16.834043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":881,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392753+0000","last_change":"2024-09-18T07:50:23.318559+0000","last_active":"2024-09-18T07:50:51.392753+0000","last_peered":"2024-09-18T07:50:51.392753+0000","last_clean":"2024-09-18T07:50:51.392753+0000","last_became_active":"2024-09-18T07:50:23.296775+0000","last_became_peered":"2024-09-18T07:50:23.296775+0000","last_unstale":"2024-09-18T07:50:51.392753+0000","last_undegraded":"2024-09-18T07:50:51.392753+0000","last_fullsized":"2024-09-18T07:50:51.392753+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:47:42.037791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.037791+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":91,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":102,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505941+0000","last_change":"2024-09-18T07:49:24.453650+0000","last_active":"2024-09-18T07:50:51.505941+0000","last_peered":"2024-09-18T07:50:51.505941+0000","last_clean":"2024-09-18T07:50:51.505941+0000","last_became_active":"2024-09-18T07:49:24.453318+0000","last_became_peered":"2024-09-18T07:49:24.453318+0000","last_unstale":"2024-09-18T07:50:51.505941+0000","last_undegraded":"2024-09-18T07:50:51.505941+0000","last_fullsized":"2024-09-18T07:50:51.505941+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:49:12.773317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:49:12.773317+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"323'24","reported_seq":678,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503505+0000","last_change":"2024-09-18T07:50:05.958641+0000","last_active":"2024-09-18T07:50:51.503505+0000","last_peered":"2024-09-18T07:50:51.503505+0000","last_clean":"2024-09-18T07:50:51.503505+0000","last_became_active":"2024-09-18T07:50:05.201657+0000","last_became_peered":"2024-09-18T07:50:05.201657+0000","last_unstale":"2024-09-18T07:50:51.503505+0000","last_undegraded":"2024-09-18T07:50:51.503505+0000","last_fullsized":"2024-09-18T07:50:51.503505+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'24","last_scrub_stamp":"2024-09-18T07:50:05.958566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:50:05.958566+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:24.486287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":91,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503921+0000","last_change":"2024-09-18T07:50:16.881850+0000","last_active":"2024-09-18T07:50:51.503921+0000","last_peered":"2024-09-18T07:50:51.503921+0000","last_clean":"2024-09-18T07:50:51.503921+0000","last_became_active":"2024-09-18T07:50:05.191515+0000","last_became_peered":"2024-09-18T07:50:05.191515+0000","last_unstale":"2024-09-18T07:50:51.503921+0000","last_undegraded":"2024-09-18T07:50:51.503921+0000","last_fullsized":"2024-09-18T07:50:51.503921+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:50:16.881751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:50:16.881751+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:34.588198+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":1815,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392000+0000","last_change":"2024-09-18T07:50:23.351560+0000","last_active":"2024-09-18T07:50:51.392000+0000","last_peered":"2024-09-18T07:50:51.392000+0000","last_clean":"2024-09-18T07:50:51.392000+0000","last_became_active":"2024-09-18T07:50:23.297717+0000","last_became_peered":"2024-09-18T07:50:23.297717+0000","last_unstale":"2024-09-18T07:50:51.392000+0000","last_undegraded":"2024-09-18T07:50:51.392000+0000","last_fullsized":"2024-09-18T07:50:51.392000+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T07:50:07.355097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:50:07.355097+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:29.323794+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":945,"num_read_kb":940,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1945,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":961,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505382+0000","last_change":"2024-09-18T07:50:05.197752+0000","last_active":"2024-09-18T07:50:51.505382+0000","last_peered":"2024-09-18T07:50:51.505382+0000","last_clean":"2024-09-18T07:50:51.505382+0000","last_became_active":"2024-09-18T07:50:05.196676+0000","last_became_peered":"2024-09-18T07:50:05.196676+0000","last_unstale":"2024-09-18T07:50:51.505382+0000","last_undegraded":"2024-09-18T07:50:51.505382+0000","last_fullsized":"2024-09-18T07:50:51.505382+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:42.663800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:49:42.663800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:55.598699+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":642,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504536+0000","last_change":"2024-09-18T07:50:23.275031+0000","last_active":"2024-09-18T07:50:51.504536+0000","last_peered":"2024-09-18T07:50:51.504536+0000","last_clean":"2024-09-18T07:50:51.504536+0000","last_became_active":"2024-09-18T07:50:23.274722+0000","last_became_peered":"2024-09-18T07:50:23.274722+0000","last_unstale":"2024-09-18T07:50:51.504536+0000","last_undegraded":"2024-09-18T07:50:51.504536+0000","last_fullsized":"2024-09-18T07:50:51.504536+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:43.622102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:43.622102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:54.255403+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"398'34","reported_seq":901,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.507184+0000","last_change":"2024-09-18T07:50:46.629405+0000","last_active":"2024-09-18T07:50:51.507184+0000","last_peered":"2024-09-18T07:50:51.507184+0000","last_clean":"2024-09-18T07:50:51.507184+0000","last_became_active":"2024-09-18T07:44:43.786827+0000","last_became_peered":"2024-09-18T07:44:43.786827+0000","last_unstale":"2024-09-18T07:50:51.507184+0000","last_undegraded":"2024-09-18T07:50:51.507184+0000","last_fullsized":"2024-09-18T07:50:51.507184+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'34","last_scrub_stamp":"2024-09-18T07:50:46.629318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:50:46.629318+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:47.126410+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":837,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504385+0000","last_change":"2024-09-18T07:50:39.627156+0000","last_active":"2024-09-18T07:50:51.504385+0000","last_peered":"2024-09-18T07:50:51.504385+0000","last_clean":"2024-09-18T07:50:51.504385+0000","last_became_active":"2024-09-18T07:50:23.306340+0000","last_became_peered":"2024-09-18T07:50:23.306340+0000","last_unstale":"2024-09-18T07:50:51.504385+0000","last_undegraded":"2024-09-18T07:50:51.504385+0000","last_fullsized":"2024-09-18T07:50:51.504385+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:39.627115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:50:39.627115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:45.806778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":893,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392135+0000","last_change":"2024-09-18T07:50:05.203665+0000","last_active":"2024-09-18T07:50:51.392135+0000","last_peered":"2024-09-18T07:50:51.392135+0000","last_clean":"2024-09-18T07:50:51.392135+0000","last_became_active":"2024-09-18T07:50:05.202222+0000","last_became_peered":"2024-09-18T07:50:05.202222+0000","last_unstale":"2024-09-18T07:50:51.392135+0000","last_undegraded":"2024-09-18T07:50:51.392135+0000","last_fullsized":"2024-09-18T07:50:51.392135+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:48:26.963223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:48:26.963223+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":838,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392717+0000","last_change":"2024-09-18T07:48:21.934191+0000","last_active":"2024-09-18T07:50:51.392717+0000","last_peered":"2024-09-18T07:50:51.392717+0000","last_clean":"2024-09-18T07:50:51.392717+0000","last_became_active":"2024-09-18T07:47:11.195084+0000","last_became_peered":"2024-09-18T07:47:11.195084+0000","last_unstale":"2024-09-18T07:50:51.392717+0000","last_undegraded":"2024-09-18T07:50:51.392717+0000","last_fullsized":"2024-09-18T07:50:51.392717+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:21.934130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:21.934130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":957,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505171+0000","last_change":"2024-09-18T07:50:23.275089+0000","last_active":"2024-09-18T07:50:51.505171+0000","last_peered":"2024-09-18T07:50:51.505171+0000","last_clean":"2024-09-18T07:50:51.505171+0000","last_became_active":"2024-09-18T07:50:23.274710+0000","last_became_peered":"2024-09-18T07:50:23.274710+0000","last_unstale":"2024-09-18T07:50:51.505171+0000","last_undegraded":"2024-09-18T07:50:51.505171+0000","last_fullsized":"2024-09-18T07:50:51.505171+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:04.610883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:04.610883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:31.656708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"323'19","reported_seq":1198,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505891+0000","last_change":"2024-09-18T07:50:45.896229+0000","last_active":"2024-09-18T07:50:51.505891+0000","last_peered":"2024-09-18T07:50:51.505891+0000","last_clean":"2024-09-18T07:50:51.505891+0000","last_became_active":"2024-09-18T07:49:32.684857+0000","last_became_peered":"2024-09-18T07:49:32.684857+0000","last_unstale":"2024-09-18T07:50:51.505891+0000","last_undegraded":"2024-09-18T07:50:51.505891+0000","last_fullsized":"2024-09-18T07:50:51.505891+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:50:45.896175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:50:45.896175+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:56.375990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":943,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504052+0000","last_change":"2024-09-18T07:44:43.793621+0000","last_active":"2024-09-18T07:50:51.504052+0000","last_peered":"2024-09-18T07:50:51.504052+0000","last_clean":"2024-09-18T07:50:51.504052+0000","last_became_active":"2024-09-18T07:44:43.790160+0000","last_became_peered":"2024-09-18T07:44:43.790160+0000","last_unstale":"2024-09-18T07:50:51.504052+0000","last_undegraded":"2024-09-18T07:50:51.504052+0000","last_fullsized":"2024-09-18T07:50:51.504052+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:39:29.078600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:39:29.078600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:25.912776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":879,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.506165+0000","last_change":"2024-09-18T07:50:17.844593+0000","last_active":"2024-09-18T07:50:51.506165+0000","last_peered":"2024-09-18T07:50:51.506165+0000","last_clean":"2024-09-18T07:50:51.506165+0000","last_became_active":"2024-09-18T07:50:14.223548+0000","last_became_peered":"2024-09-18T07:50:14.223548+0000","last_unstale":"2024-09-18T07:50:51.506165+0000","last_undegraded":"2024-09-18T07:50:51.506165+0000","last_fullsized":"2024-09-18T07:50:51.506165+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:17.844473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:50:17.844473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:22.705281+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":951,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.506871+0000","last_change":"2024-09-18T07:50:23.287269+0000","last_active":"2024-09-18T07:50:51.506871+0000","last_peered":"2024-09-18T07:50:51.506871+0000","last_clean":"2024-09-18T07:50:51.506871+0000","last_became_active":"2024-09-18T07:50:23.285080+0000","last_became_peered":"2024-09-18T07:50:23.285080+0000","last_unstale":"2024-09-18T07:50:51.506871+0000","last_undegraded":"2024-09-18T07:50:51.506871+0000","last_fullsized":"2024-09-18T07:50:51.506871+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:26.691414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:26.691414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":174,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.502960+0000","last_change":"2024-09-18T07:50:23.312027+0000","last_active":"2024-09-18T07:50:51.502960+0000","last_peered":"2024-09-18T07:50:51.502960+0000","last_clean":"2024-09-18T07:50:51.502960+0000","last_became_active":"2024-09-18T07:50:23.275682+0000","last_became_peered":"2024-09-18T07:50:23.275682+0000","last_unstale":"2024-09-18T07:50:51.502960+0000","last_undegraded":"2024-09-18T07:50:51.502960+0000","last_fullsized":"2024-09-18T07:50:51.502960+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:50:16.037345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:50:16.037345+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:43.188373+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"323'31","reported_seq":1025,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.506631+0000","last_change":"2024-09-18T07:48:22.002514+0000","last_active":"2024-09-18T07:50:51.506631+0000","last_peered":"2024-09-18T07:50:51.506631+0000","last_clean":"2024-09-18T07:50:51.506631+0000","last_became_active":"2024-09-18T07:47:20.132912+0000","last_became_peered":"2024-09-18T07:47:20.132912+0000","last_unstale":"2024-09-18T07:50:51.506631+0000","last_undegraded":"2024-09-18T07:50:51.506631+0000","last_fullsized":"2024-09-18T07:50:51.506631+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'31","last_scrub_stamp":"2024-09-18T07:48:22.002416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:22.002416+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":748,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505008+0000","last_change":"2024-09-18T07:49:32.688728+0000","last_active":"2024-09-18T07:50:51.505008+0000","last_peered":"2024-09-18T07:50:51.505008+0000","last_clean":"2024-09-18T07:50:51.505008+0000","last_became_active":"2024-09-18T07:49:32.688352+0000","last_became_peered":"2024-09-18T07:49:32.688352+0000","last_unstale":"2024-09-18T07:50:51.505008+0000","last_undegraded":"2024-09-18T07:50:51.505008+0000","last_fullsized":"2024-09-18T07:50:51.505008+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:32.491919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:32.491919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":629,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504629+0000","last_change":"2024-09-18T07:50:37.952224+0000","last_active":"2024-09-18T07:50:51.504629+0000","last_peered":"2024-09-18T07:50:51.504629+0000","last_clean":"2024-09-18T07:50:51.504629+0000","last_became_active":"2024-09-18T07:50:05.196640+0000","last_became_peered":"2024-09-18T07:50:05.196640+0000","last_unstale":"2024-09-18T07:50:51.504629+0000","last_undegraded":"2024-09-18T07:50:51.504629+0000","last_fullsized":"2024-09-18T07:50:51.504629+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:37.952164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:50:37.952164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:53.757330+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1093,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503601+0000","last_change":"2024-09-18T07:50:23.277183+0000","last_active":"2024-09-18T07:50:51.503601+0000","last_peered":"2024-09-18T07:50:51.503601+0000","last_clean":"2024-09-18T07:50:51.503601+0000","last_became_active":"2024-09-18T07:50:23.276876+0000","last_became_peered":"2024-09-18T07:50:23.276876+0000","last_unstale":"2024-09-18T07:50:51.503601+0000","last_undegraded":"2024-09-18T07:50:51.503601+0000","last_fullsized":"2024-09-18T07:50:51.503601+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:06.889139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:06.889139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:30.073517+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":906,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.391917+0000","last_change":"2024-09-18T07:50:23.296215+0000","last_active":"2024-09-18T07:50:51.391917+0000","last_peered":"2024-09-18T07:50:51.391917+0000","last_clean":"2024-09-18T07:50:51.391917+0000","last_became_active":"2024-09-18T07:50:23.295783+0000","last_became_peered":"2024-09-18T07:50:23.295783+0000","last_unstale":"2024-09-18T07:50:51.391917+0000","last_undegraded":"2024-09-18T07:50:51.391917+0000","last_fullsized":"2024-09-18T07:50:51.391917+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:49:47.404771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:49:47.404771+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"323'20","reported_seq":1024,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504833+0000","last_change":"2024-09-18T07:50:46.869864+0000","last_active":"2024-09-18T07:50:51.504833+0000","last_peered":"2024-09-18T07:50:51.504833+0000","last_clean":"2024-09-18T07:50:51.504833+0000","last_became_active":"2024-09-18T07:50:23.278411+0000","last_became_peered":"2024-09-18T07:50:23.278411+0000","last_unstale":"2024-09-18T07:50:51.504833+0000","last_undegraded":"2024-09-18T07:50:51.504833+0000","last_fullsized":"2024-09-18T07:50:51.504833+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:50:46.869811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:50:46.869811+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:12.812560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":489,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505497+0000","last_change":"2024-09-18T07:50:23.278102+0000","last_active":"2024-09-18T07:50:51.505497+0000","last_peered":"2024-09-18T07:50:51.505497+0000","last_clean":"2024-09-18T07:50:51.505497+0000","last_became_active":"2024-09-18T07:50:23.277790+0000","last_became_peered":"2024-09-18T07:50:23.277790+0000","last_unstale":"2024-09-18T07:50:51.505497+0000","last_undegraded":"2024-09-18T07:50:51.505497+0000","last_fullsized":"2024-09-18T07:50:51.505497+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:42.225418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:42.225418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"461'13","reported_seq":1350,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503648+0000","last_change":"2024-09-18T07:50:39.220479+0000","last_active":"2024-09-18T07:50:51.503648+0000","last_peered":"2024-09-18T07:50:51.503648+0000","last_clean":"2024-09-18T07:50:51.503648+0000","last_became_active":"2024-09-18T07:47:11.203256+0000","last_became_peered":"2024-09-18T07:47:11.203256+0000","last_unstale":"2024-09-18T07:50:51.503648+0000","last_undegraded":"2024-09-18T07:50:51.503648+0000","last_fullsized":"2024-09-18T07:50:51.503648+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'13","last_scrub_stamp":"2024-09-18T07:50:39.220421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:50:39.220421+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:45.446878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":966,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.506228+0000","last_change":"2024-09-18T07:49:32.690291+0000","last_active":"2024-09-18T07:50:51.506228+0000","last_peered":"2024-09-18T07:50:51.506228+0000","last_clean":"2024-09-18T07:50:51.506228+0000","last_became_active":"2024-09-18T07:49:32.689562+0000","last_became_peered":"2024-09-18T07:49:32.689562+0000","last_unstale":"2024-09-18T07:50:51.506228+0000","last_undegraded":"2024-09-18T07:50:51.506228+0000","last_fullsized":"2024-09-18T07:50:51.506228+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:48.207484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:46:48.207484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"323'15","reported_seq":684,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505824+0000","last_change":"2024-09-18T07:50:23.406391+0000","last_active":"2024-09-18T07:50:51.505824+0000","last_peered":"2024-09-18T07:50:51.505824+0000","last_clean":"2024-09-18T07:50:51.505824+0000","last_became_active":"2024-09-18T07:50:23.288752+0000","last_became_peered":"2024-09-18T07:50:23.288752+0000","last_unstale":"2024-09-18T07:50:51.505824+0000","last_undegraded":"2024-09-18T07:50:51.505824+0000","last_fullsized":"2024-09-18T07:50:51.505824+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:49:45.465070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:49:45.465070+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:08.907615+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":47,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.395061+0000","last_change":"2024-09-18T07:50:23.285852+0000","last_active":"2024-09-18T07:50:51.395061+0000","last_peered":"2024-09-18T07:50:51.395061+0000","last_clean":"2024-09-18T07:50:51.395061+0000","last_became_active":"2024-09-18T07:50:23.285473+0000","last_became_peered":"2024-09-18T07:50:23.285473+0000","last_unstale":"2024-09-18T07:50:51.395061+0000","last_undegraded":"2024-09-18T07:50:51.395061+0000","last_fullsized":"2024-09-18T07:50:51.395061+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:51.021530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:51.021530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1089,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503492+0000","last_change":"2024-09-18T07:50:23.274125+0000","last_active":"2024-09-18T07:50:51.503492+0000","last_peered":"2024-09-18T07:50:51.503492+0000","last_clean":"2024-09-18T07:50:51.503492+0000","last_became_active":"2024-09-18T07:50:23.273703+0000","last_became_peered":"2024-09-18T07:50:23.273703+0000","last_unstale":"2024-09-18T07:50:51.503492+0000","last_undegraded":"2024-09-18T07:50:51.503492+0000","last_fullsized":"2024-09-18T07:50:51.503492+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:22.799585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:22.799585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":518,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392342+0000","last_change":"2024-09-18T07:50:23.275281+0000","last_active":"2024-09-18T07:50:51.392342+0000","last_peered":"2024-09-18T07:50:51.392342+0000","last_clean":"2024-09-18T07:50:51.392342+0000","last_became_active":"2024-09-18T07:50:23.274910+0000","last_became_peered":"2024-09-18T07:50:23.274910+0000","last_unstale":"2024-09-18T07:50:51.392342+0000","last_undegraded":"2024-09-18T07:50:51.392342+0000","last_fullsized":"2024-09-18T07:50:51.392342+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:42.436545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:42.436545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:03.369633+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"323'35","reported_seq":797,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505285+0000","last_change":"2024-09-18T07:47:52.386750+0000","last_active":"2024-09-18T07:50:51.505285+0000","last_peered":"2024-09-18T07:50:51.505285+0000","last_clean":"2024-09-18T07:50:51.505285+0000","last_became_active":"2024-09-18T07:47:20.141942+0000","last_became_peered":"2024-09-18T07:47:20.141942+0000","last_unstale":"2024-09-18T07:50:51.505285+0000","last_undegraded":"2024-09-18T07:50:51.505285+0000","last_fullsized":"2024-09-18T07:50:51.505285+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'35","last_scrub_stamp":"2024-09-18T07:47:52.386688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.386688+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:36.803954+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":852,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.393122+0000","last_change":"2024-09-18T07:47:17.794908+0000","last_active":"2024-09-18T07:50:51.393122+0000","last_peered":"2024-09-18T07:50:51.393122+0000","last_clean":"2024-09-18T07:50:51.393122+0000","last_became_active":"2024-09-18T07:47:11.194873+0000","last_became_peered":"2024-09-18T07:47:11.194873+0000","last_unstale":"2024-09-18T07:50:51.393122+0000","last_undegraded":"2024-09-18T07:50:51.393122+0000","last_fullsized":"2024-09-18T07:50:51.393122+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:17.794850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:17.794850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:33.560477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1096,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503452+0000","last_change":"2024-09-18T07:50:05.193285+0000","last_active":"2024-09-18T07:50:51.503452+0000","last_peered":"2024-09-18T07:50:51.503452+0000","last_clean":"2024-09-18T07:50:51.503452+0000","last_became_active":"2024-09-18T07:50:05.192813+0000","last_became_peered":"2024-09-18T07:50:05.192813+0000","last_unstale":"2024-09-18T07:50:51.503452+0000","last_undegraded":"2024-09-18T07:50:51.503452+0000","last_fullsized":"2024-09-18T07:50:51.503452+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:20.767517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:20.767517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":841,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392164+0000","last_change":"2024-09-18T07:50:23.297645+0000","last_active":"2024-09-18T07:50:51.392164+0000","last_peered":"2024-09-18T07:50:51.392164+0000","last_clean":"2024-09-18T07:50:51.392164+0000","last_became_active":"2024-09-18T07:50:23.296866+0000","last_became_peered":"2024-09-18T07:50:23.296866+0000","last_unstale":"2024-09-18T07:50:51.392164+0000","last_undegraded":"2024-09-18T07:50:51.392164+0000","last_fullsized":"2024-09-18T07:50:51.392164+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:50.079275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:50.079275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"323'18","reported_seq":1083,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503811+0000","last_change":"2024-09-18T07:50:23.305432+0000","last_active":"2024-09-18T07:50:51.503811+0000","last_peered":"2024-09-18T07:50:51.503811+0000","last_clean":"2024-09-18T07:50:51.503811+0000","last_became_active":"2024-09-18T07:50:23.272111+0000","last_became_peered":"2024-09-18T07:50:23.272111+0000","last_unstale":"2024-09-18T07:50:51.503811+0000","last_undegraded":"2024-09-18T07:50:51.503811+0000","last_fullsized":"2024-09-18T07:50:51.503811+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'18","last_scrub_stamp":"2024-09-18T07:50:04.943731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:50:04.943731+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:15.112228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":844,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.393233+0000","last_change":"2024-09-18T07:47:20.139266+0000","last_active":"2024-09-18T07:50:51.393233+0000","last_peered":"2024-09-18T07:50:51.393233+0000","last_clean":"2024-09-18T07:50:51.393233+0000","last_became_active":"2024-09-18T07:47:20.138910+0000","last_became_peered":"2024-09-18T07:47:20.138910+0000","last_unstale":"2024-09-18T07:50:51.393233+0000","last_undegraded":"2024-09-18T07:50:51.393233+0000","last_fullsized":"2024-09-18T07:50:51.393233+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:51.752578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:51.752578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:26.489350+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1107,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503158+0000","last_change":"2024-09-18T07:50:05.180815+0000","last_active":"2024-09-18T07:50:51.503158+0000","last_peered":"2024-09-18T07:50:51.503158+0000","last_clean":"2024-09-18T07:50:51.503158+0000","last_became_active":"2024-09-18T07:50:05.180071+0000","last_became_peered":"2024-09-18T07:50:05.180071+0000","last_unstale":"2024-09-18T07:50:51.503158+0000","last_undegraded":"2024-09-18T07:50:51.503158+0000","last_fullsized":"2024-09-18T07:50:51.503158+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:23.757372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:23.757372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":47,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.397877+0000","last_change":"2024-09-18T07:50:23.299961+0000","last_active":"2024-09-18T07:50:51.397877+0000","last_peered":"2024-09-18T07:50:51.397877+0000","last_clean":"2024-09-18T07:50:51.397877+0000","last_became_active":"2024-09-18T07:50:23.299077+0000","last_became_peered":"2024-09-18T07:50:23.299077+0000","last_unstale":"2024-09-18T07:50:51.397877+0000","last_undegraded":"2024-09-18T07:50:51.397877+0000","last_fullsized":"2024-09-18T07:50:51.397877+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:40.439672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:40.439672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"323'27","reported_seq":1047,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503422+0000","last_change":"2024-09-18T07:48:03.057334+0000","last_active":"2024-09-18T07:50:51.503422+0000","last_peered":"2024-09-18T07:50:51.503422+0000","last_clean":"2024-09-18T07:50:51.503422+0000","last_became_active":"2024-09-18T07:47:11.196288+0000","last_became_peered":"2024-09-18T07:47:11.196288+0000","last_unstale":"2024-09-18T07:50:51.503422+0000","last_undegraded":"2024-09-18T07:50:51.503422+0000","last_fullsized":"2024-09-18T07:50:51.503422+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'27","last_scrub_stamp":"2024-09-18T07:48:03.057182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:03.057182+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:01.465573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":865,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.393442+0000","last_change":"2024-09-18T07:50:05.231617+0000","last_active":"2024-09-18T07:50:51.393442+0000","last_peered":"2024-09-18T07:50:51.393442+0000","last_clean":"2024-09-18T07:50:51.393442+0000","last_became_active":"2024-09-18T07:50:05.200873+0000","last_became_peered":"2024-09-18T07:50:05.200873+0000","last_unstale":"2024-09-18T07:50:51.393442+0000","last_undegraded":"2024-09-18T07:50:51.393442+0000","last_fullsized":"2024-09-18T07:50:51.393442+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:48:34.003134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:34.003134+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"398'14","reported_seq":1065,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392534+0000","last_change":"2024-09-18T07:50:05.354087+0000","last_active":"2024-09-18T07:50:51.392534+0000","last_peered":"2024-09-18T07:50:51.392534+0000","last_clean":"2024-09-18T07:50:51.392534+0000","last_became_active":"2024-09-18T07:44:43.782355+0000","last_became_peered":"2024-09-18T07:44:43.782355+0000","last_unstale":"2024-09-18T07:50:51.392534+0000","last_undegraded":"2024-09-18T07:50:51.392534+0000","last_fullsized":"2024-09-18T07:50:51.392534+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:50:05.354034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:50:05.354034+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:08.048196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":923,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504402+0000","last_change":"2024-09-18T07:50:41.641389+0000","last_active":"2024-09-18T07:50:51.504402+0000","last_peered":"2024-09-18T07:50:51.504402+0000","last_clean":"2024-09-18T07:50:51.504402+0000","last_became_active":"2024-09-18T07:50:14.216827+0000","last_became_peered":"2024-09-18T07:50:14.216827+0000","last_unstale":"2024-09-18T07:50:51.504402+0000","last_undegraded":"2024-09-18T07:50:51.504402+0000","last_fullsized":"2024-09-18T07:50:51.504402+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:41.641268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:50:41.641268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:42.478259+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":702,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505658+0000","last_change":"2024-09-18T07:50:23.287774+0000","last_active":"2024-09-18T07:50:51.505658+0000","last_peered":"2024-09-18T07:50:51.505658+0000","last_clean":"2024-09-18T07:50:51.505658+0000","last_became_active":"2024-09-18T07:50:23.286252+0000","last_became_peered":"2024-09-18T07:50:23.286252+0000","last_unstale":"2024-09-18T07:50:51.505658+0000","last_undegraded":"2024-09-18T07:50:51.505658+0000","last_fullsized":"2024-09-18T07:50:51.505658+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:42:08.181420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:42:08.181420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:29.905624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"323'12","reported_seq":767,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.393299+0000","last_change":"2024-09-18T07:48:10.211661+0000","last_active":"2024-09-18T07:50:51.393299+0000","last_peered":"2024-09-18T07:50:51.393299+0000","last_clean":"2024-09-18T07:50:51.393299+0000","last_became_active":"2024-09-18T07:48:10.160725+0000","last_became_peered":"2024-09-18T07:48:10.160725+0000","last_unstale":"2024-09-18T07:50:51.393299+0000","last_undegraded":"2024-09-18T07:50:51.393299+0000","last_fullsized":"2024-09-18T07:50:51.393299+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'12","last_scrub_stamp":"2024-09-18T07:48:03.699388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:03.699388+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":999,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504817+0000","last_change":"2024-09-18T07:50:23.274124+0000","last_active":"2024-09-18T07:50:51.504817+0000","last_peered":"2024-09-18T07:50:51.504817+0000","last_clean":"2024-09-18T07:50:51.504817+0000","last_became_active":"2024-09-18T07:50:23.273803+0000","last_became_peered":"2024-09-18T07:50:23.273803+0000","last_unstale":"2024-09-18T07:50:51.504817+0000","last_undegraded":"2024-09-18T07:50:51.504817+0000","last_fullsized":"2024-09-18T07:50:51.504817+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:53.089903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:53.089903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:27.587459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":843,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.393032+0000","last_change":"2024-09-18T07:48:56.269806+0000","last_active":"2024-09-18T07:50:51.393032+0000","last_peered":"2024-09-18T07:50:51.393032+0000","last_clean":"2024-09-18T07:50:51.393032+0000","last_became_active":"2024-09-18T07:44:43.782348+0000","last_became_peered":"2024-09-18T07:44:43.782348+0000","last_unstale":"2024-09-18T07:50:51.393032+0000","last_undegraded":"2024-09-18T07:50:51.393032+0000","last_fullsized":"2024-09-18T07:50:51.393032+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:56.269745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:56.269745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"323'15","reported_seq":57,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.396662+0000","last_change":"2024-09-18T07:50:23.335646+0000","last_active":"2024-09-18T07:50:51.396662+0000","last_peered":"2024-09-18T07:50:51.396662+0000","last_clean":"2024-09-18T07:50:51.396662+0000","last_became_active":"2024-09-18T07:50:23.290478+0000","last_became_peered":"2024-09-18T07:50:23.290478+0000","last_unstale":"2024-09-18T07:50:51.396662+0000","last_undegraded":"2024-09-18T07:50:51.396662+0000","last_fullsized":"2024-09-18T07:50:51.396662+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:48:27.802966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:27.802966+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":859,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392409+0000","last_change":"2024-09-18T07:49:45.461696+0000","last_active":"2024-09-18T07:50:51.392409+0000","last_peered":"2024-09-18T07:50:51.392409+0000","last_clean":"2024-09-18T07:50:51.392409+0000","last_became_active":"2024-09-18T07:44:43.782930+0000","last_became_peered":"2024-09-18T07:44:43.782930+0000","last_unstale":"2024-09-18T07:50:51.392409+0000","last_undegraded":"2024-09-18T07:50:51.392409+0000","last_fullsized":"2024-09-18T07:50:51.392409+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:45.461629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:45.461629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:52.011721+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":47,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.396203+0000","last_change":"2024-09-18T07:50:23.274328+0000","last_active":"2024-09-18T07:50:51.396203+0000","last_peered":"2024-09-18T07:50:51.396203+0000","last_clean":"2024-09-18T07:50:51.396203+0000","last_became_active":"2024-09-18T07:50:23.274007+0000","last_became_peered":"2024-09-18T07:50:23.274007+0000","last_unstale":"2024-09-18T07:50:51.396203+0000","last_undegraded":"2024-09-18T07:50:51.396203+0000","last_fullsized":"2024-09-18T07:50:51.396203+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:30.750167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:30.750167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1061,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503269+0000","last_change":"2024-09-18T07:48:03.866565+0000","last_active":"2024-09-18T07:50:51.503269+0000","last_peered":"2024-09-18T07:50:51.503269+0000","last_clean":"2024-09-18T07:50:51.503269+0000","last_became_active":"2024-09-18T07:44:43.790582+0000","last_became_peered":"2024-09-18T07:44:43.790582+0000","last_unstale":"2024-09-18T07:50:51.503269+0000","last_undegraded":"2024-09-18T07:50:51.503269+0000","last_fullsized":"2024-09-18T07:50:51.503269+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:03.866499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:03.866499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"323'15","reported_seq":785,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392693+0000","last_change":"2024-09-18T07:50:23.338747+0000","last_active":"2024-09-18T07:50:51.392693+0000","last_peered":"2024-09-18T07:50:51.392693+0000","last_clean":"2024-09-18T07:50:51.392693+0000","last_became_active":"2024-09-18T07:50:23.298004+0000","last_became_peered":"2024-09-18T07:50:23.298004+0000","last_unstale":"2024-09-18T07:50:51.392693+0000","last_undegraded":"2024-09-18T07:50:51.392693+0000","last_fullsized":"2024-09-18T07:50:51.392693+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:50:04.470601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:50:04.470601+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:30.138856+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":536,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392144+0000","last_change":"2024-09-18T07:50:23.297867+0000","last_active":"2024-09-18T07:50:51.392144+0000","last_peered":"2024-09-18T07:50:51.392144+0000","last_clean":"2024-09-18T07:50:51.392144+0000","last_became_active":"2024-09-18T07:50:23.296118+0000","last_became_peered":"2024-09-18T07:50:23.296118+0000","last_unstale":"2024-09-18T07:50:51.392144+0000","last_undegraded":"2024-09-18T07:50:51.392144+0000","last_fullsized":"2024-09-18T07:50:51.392144+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:52.023913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.023913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"548'12","reported_seq":57,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.394887+0000","last_change":"2024-09-18T07:50:23.327908+0000","last_active":"2024-09-18T07:50:51.394887+0000","last_peered":"2024-09-18T07:50:51.394887+0000","last_clean":"2024-09-18T07:50:51.394887+0000","last_became_active":"2024-09-18T07:50:23.294281+0000","last_became_peered":"2024-09-18T07:50:23.294281+0000","last_unstale":"2024-09-18T07:50:51.394887+0000","last_undegraded":"2024-09-18T07:50:51.394887+0000","last_fullsized":"2024-09-18T07:50:51.394887+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'11","last_scrub_stamp":"2024-09-18T07:49:50.502186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:49:50.502186+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:09.383930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":948,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.506201+0000","last_change":"2024-09-18T07:47:11.193555+0000","last_active":"2024-09-18T07:50:51.506201+0000","last_peered":"2024-09-18T07:50:51.506201+0000","last_clean":"2024-09-18T07:50:51.506201+0000","last_became_active":"2024-09-18T07:47:11.193273+0000","last_became_peered":"2024-09-18T07:47:11.193273+0000","last_unstale":"2024-09-18T07:50:51.506201+0000","last_undegraded":"2024-09-18T07:50:51.506201+0000","last_fullsized":"2024-09-18T07:50:51.506201+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:41:48.955463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:41:48.955463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:32.906781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"323'20","reported_seq":941,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392398+0000","last_change":"2024-09-18T07:50:23.343896+0000","last_active":"2024-09-18T07:50:51.392398+0000","last_peered":"2024-09-18T07:50:51.392398+0000","last_clean":"2024-09-18T07:50:51.392398+0000","last_became_active":"2024-09-18T07:50:23.296164+0000","last_became_peered":"2024-09-18T07:50:23.296164+0000","last_unstale":"2024-09-18T07:50:51.392398+0000","last_undegraded":"2024-09-18T07:50:51.392398+0000","last_fullsized":"2024-09-18T07:50:51.392398+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:46:14.650883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:46:14.650883+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:23.472632+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":845,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392986+0000","last_change":"2024-09-18T07:49:00.272282+0000","last_active":"2024-09-18T07:50:51.392986+0000","last_peered":"2024-09-18T07:50:51.392986+0000","last_clean":"2024-09-18T07:50:51.392986+0000","last_became_active":"2024-09-18T07:44:43.779305+0000","last_became_peered":"2024-09-18T07:44:43.779305+0000","last_unstale":"2024-09-18T07:50:51.392986+0000","last_undegraded":"2024-09-18T07:50:51.392986+0000","last_fullsized":"2024-09-18T07:50:51.392986+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:00.272210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:49:00.272210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"548'16","reported_seq":1181,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504988+0000","last_change":"2024-09-18T07:50:23.304787+0000","last_active":"2024-09-18T07:50:51.504988+0000","last_peered":"2024-09-18T07:50:51.504988+0000","last_clean":"2024-09-18T07:50:51.504988+0000","last_became_active":"2024-09-18T07:50:23.270200+0000","last_became_peered":"2024-09-18T07:50:23.270200+0000","last_unstale":"2024-09-18T07:50:51.504988+0000","last_undegraded":"2024-09-18T07:50:51.504988+0000","last_fullsized":"2024-09-18T07:50:51.504988+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'14","last_scrub_stamp":"2024-09-18T07:50:06.474533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:50:06.474533+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:08.100014+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":532,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.393044+0000","last_change":"2024-09-18T07:50:37.578139+0000","last_active":"2024-09-18T07:50:51.393044+0000","last_peered":"2024-09-18T07:50:51.393044+0000","last_clean":"2024-09-18T07:50:51.393044+0000","last_became_active":"2024-09-18T07:50:23.295431+0000","last_became_peered":"2024-09-18T07:50:23.295431+0000","last_unstale":"2024-09-18T07:50:51.393044+0000","last_undegraded":"2024-09-18T07:50:51.393044+0000","last_fullsized":"2024-09-18T07:50:51.393044+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:37.578068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:50:37.578068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:41.865349+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"323'19","reported_seq":1091,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505935+0000","last_change":"2024-09-18T07:47:20.196648+0000","last_active":"2024-09-18T07:50:51.505935+0000","last_peered":"2024-09-18T07:50:51.505935+0000","last_clean":"2024-09-18T07:50:51.505935+0000","last_became_active":"2024-09-18T07:47:20.140070+0000","last_became_peered":"2024-09-18T07:47:20.140070+0000","last_unstale":"2024-09-18T07:50:51.505935+0000","last_undegraded":"2024-09-18T07:50:51.505935+0000","last_fullsized":"2024-09-18T07:50:51.505935+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:41:54.022421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:41:54.022421+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:31.916690+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1005,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504936+0000","last_change":"2024-09-18T07:46:06.964899+0000","last_active":"2024-09-18T07:50:51.504936+0000","last_peered":"2024-09-18T07:50:51.504936+0000","last_clean":"2024-09-18T07:50:51.504936+0000","last_became_active":"2024-09-18T07:44:43.780305+0000","last_became_peered":"2024-09-18T07:44:43.780305+0000","last_unstale":"2024-09-18T07:50:51.504936+0000","last_undegraded":"2024-09-18T07:50:51.504936+0000","last_fullsized":"2024-09-18T07:50:51.504936+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:06.964847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:46:06.964847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:54.987085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":677,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503695+0000","last_change":"2024-09-18T07:48:10.163971+0000","last_active":"2024-09-18T07:50:51.503695+0000","last_peered":"2024-09-18T07:50:51.503695+0000","last_clean":"2024-09-18T07:50:51.503695+0000","last_became_active":"2024-09-18T07:48:10.163110+0000","last_became_peered":"2024-09-18T07:48:10.163110+0000","last_unstale":"2024-09-18T07:50:51.503695+0000","last_undegraded":"2024-09-18T07:50:51.503695+0000","last_fullsized":"2024-09-18T07:50:51.503695+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:59.022075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:59.022075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:18.612591+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":473,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.503222+0000","last_change":"2024-09-18T07:50:14.220470+0000","last_active":"2024-09-18T07:50:51.503222+0000","last_peered":"2024-09-18T07:50:51.503222+0000","last_clean":"2024-09-18T07:50:51.503222+0000","last_became_active":"2024-09-18T07:50:14.220062+0000","last_became_peered":"2024-09-18T07:50:14.220062+0000","last_unstale":"2024-09-18T07:50:51.503222+0000","last_undegraded":"2024-09-18T07:50:51.503222+0000","last_fullsized":"2024-09-18T07:50:51.503222+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:25.249102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:25.249102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"323'15","reported_seq":744,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504811+0000","last_change":"2024-09-18T07:49:32.696134+0000","last_active":"2024-09-18T07:50:51.504811+0000","last_peered":"2024-09-18T07:50:51.504811+0000","last_clean":"2024-09-18T07:50:51.504811+0000","last_became_active":"2024-09-18T07:49:32.687721+0000","last_became_peered":"2024-09-18T07:49:32.687721+0000","last_unstale":"2024-09-18T07:50:51.504811+0000","last_undegraded":"2024-09-18T07:50:51.504811+0000","last_fullsized":"2024-09-18T07:50:51.504811+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:47:49.411931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:49.411931+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:58.912624+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":858,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392862+0000","last_change":"2024-09-18T07:50:23.298146+0000","last_active":"2024-09-18T07:50:51.392862+0000","last_peered":"2024-09-18T07:50:51.392862+0000","last_clean":"2024-09-18T07:50:51.392862+0000","last_became_active":"2024-09-18T07:50:23.296874+0000","last_became_peered":"2024-09-18T07:50:23.296874+0000","last_unstale":"2024-09-18T07:50:51.392862+0000","last_undegraded":"2024-09-18T07:50:51.392862+0000","last_fullsized":"2024-09-18T07:50:51.392862+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:33.013808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:48:33.013808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":55,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.397757+0000","last_change":"2024-09-18T07:50:37.618410+0000","last_active":"2024-09-18T07:50:51.397757+0000","last_peered":"2024-09-18T07:50:51.397757+0000","last_clean":"2024-09-18T07:50:51.397757+0000","last_became_active":"2024-09-18T07:50:23.298745+0000","last_became_peered":"2024-09-18T07:50:23.298745+0000","last_unstale":"2024-09-18T07:50:51.397757+0000","last_undegraded":"2024-09-18T07:50:51.397757+0000","last_fullsized":"2024-09-18T07:50:51.397757+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:37.618234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:50:37.618234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:41.036168+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":935,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505043+0000","last_change":"2024-09-18T07:50:14.217318+0000","last_active":"2024-09-18T07:50:51.505043+0000","last_peered":"2024-09-18T07:50:51.505043+0000","last_clean":"2024-09-18T07:50:51.505043+0000","last_became_active":"2024-09-18T07:50:14.217052+0000","last_became_peered":"2024-09-18T07:50:14.217052+0000","last_unstale":"2024-09-18T07:50:51.505043+0000","last_undegraded":"2024-09-18T07:50:51.505043+0000","last_fullsized":"2024-09-18T07:50:51.505043+0000","mapping_epoch":538,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":539,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:44.214865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:47:44.214865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:34.860626+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"323'19","reported_seq":59,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.398416+0000","last_change":"2024-09-18T07:50:23.451303+0000","last_active":"2024-09-18T07:50:51.398416+0000","last_peered":"2024-09-18T07:50:51.398416+0000","last_clean":"2024-09-18T07:50:51.398416+0000","last_became_active":"2024-09-18T07:50:23.389095+0000","last_became_peered":"2024-09-18T07:50:23.389095+0000","last_unstale":"2024-09-18T07:50:51.398416+0000","last_undegraded":"2024-09-18T07:50:51.398416+0000","last_fullsized":"2024-09-18T07:50:51.398416+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:48:58.276659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:48:58.276659+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":990,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504725+0000","last_change":"2024-09-18T07:47:52.070685+0000","last_active":"2024-09-18T07:50:51.504725+0000","last_peered":"2024-09-18T07:50:51.504725+0000","last_clean":"2024-09-18T07:50:51.504725+0000","last_became_active":"2024-09-18T07:47:20.140495+0000","last_became_peered":"2024-09-18T07:47:20.140495+0000","last_unstale":"2024-09-18T07:50:51.504725+0000","last_undegraded":"2024-09-18T07:50:51.504725+0000","last_fullsized":"2024-09-18T07:50:51.504725+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:52.070602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:52.070602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:54.598914+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":851,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392262+0000","last_change":"2024-09-18T07:46:13.660302+0000","last_active":"2024-09-18T07:50:51.392262+0000","last_peered":"2024-09-18T07:50:51.392262+0000","last_clean":"2024-09-18T07:50:51.392262+0000","last_became_active":"2024-09-18T07:44:43.778318+0000","last_became_peered":"2024-09-18T07:44:43.778318+0000","last_unstale":"2024-09-18T07:50:51.392262+0000","last_undegraded":"2024-09-18T07:50:51.392262+0000","last_fullsized":"2024-09-18T07:50:51.392262+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:46:13.660254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:46:13.660254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:01.470078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":47,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.397632+0000","last_change":"2024-09-18T07:50:23.295263+0000","last_active":"2024-09-18T07:50:51.397632+0000","last_peered":"2024-09-18T07:50:51.397632+0000","last_clean":"2024-09-18T07:50:51.397632+0000","last_became_active":"2024-09-18T07:50:23.294923+0000","last_became_peered":"2024-09-18T07:50:23.294923+0000","last_unstale":"2024-09-18T07:50:51.397632+0000","last_undegraded":"2024-09-18T07:50:51.397632+0000","last_fullsized":"2024-09-18T07:50:51.397632+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:33.311735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:48:33.311735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"323'8","reported_seq":53,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.396851+0000","last_change":"2024-09-18T07:50:23.399317+0000","last_active":"2024-09-18T07:50:51.396851+0000","last_peered":"2024-09-18T07:50:51.396851+0000","last_clean":"2024-09-18T07:50:51.396851+0000","last_became_active":"2024-09-18T07:50:23.298033+0000","last_became_peered":"2024-09-18T07:50:23.298033+0000","last_unstale":"2024-09-18T07:50:51.396851+0000","last_undegraded":"2024-09-18T07:50:51.396851+0000","last_fullsized":"2024-09-18T07:50:51.396851+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'8","last_scrub_stamp":"2024-09-18T07:47:46.405595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:46.405595+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":258,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.504461+0000","last_change":"2024-09-18T07:50:23.285262+0000","last_active":"2024-09-18T07:50:51.504461+0000","last_peered":"2024-09-18T07:50:51.504461+0000","last_clean":"2024-09-18T07:50:51.504461+0000","last_became_active":"2024-09-18T07:50:23.284906+0000","last_became_peered":"2024-09-18T07:50:23.284906+0000","last_unstale":"2024-09-18T07:50:51.504461+0000","last_undegraded":"2024-09-18T07:50:51.504461+0000","last_fullsized":"2024-09-18T07:50:51.504461+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:15.549558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:50:15.549558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:15.993742+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":688,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.506939+0000","last_change":"2024-09-18T07:47:55.363130+0000","last_active":"2024-09-18T07:50:51.506939+0000","last_peered":"2024-09-18T07:50:51.506939+0000","last_clean":"2024-09-18T07:50:51.506939+0000","last_became_active":"2024-09-18T07:44:43.792641+0000","last_became_peered":"2024-09-18T07:44:43.792641+0000","last_unstale":"2024-09-18T07:50:51.506939+0000","last_undegraded":"2024-09-18T07:50:51.506939+0000","last_fullsized":"2024-09-18T07:50:51.506939+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:47:55.363087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:47:55.363087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":837,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392400+0000","last_change":"2024-09-18T07:49:49.436754+0000","last_active":"2024-09-18T07:50:51.392400+0000","last_peered":"2024-09-18T07:50:51.392400+0000","last_clean":"2024-09-18T07:50:51.392400+0000","last_became_active":"2024-09-18T07:44:43.782378+0000","last_became_peered":"2024-09-18T07:44:43.782378+0000","last_unstale":"2024-09-18T07:50:51.392400+0000","last_undegraded":"2024-09-18T07:50:51.392400+0000","last_fullsized":"2024-09-18T07:50:51.392400+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:49:49.436635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:49:49.436635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:52.792800+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":643,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.505040+0000","last_change":"2024-09-18T07:50:49.580460+0000","last_active":"2024-09-18T07:50:51.505040+0000","last_peered":"2024-09-18T07:50:51.505040+0000","last_clean":"2024-09-18T07:50:51.505040+0000","last_became_active":"2024-09-18T07:47:20.138598+0000","last_became_peered":"2024-09-18T07:47:20.138598+0000","last_unstale":"2024-09-18T07:50:51.505040+0000","last_undegraded":"2024-09-18T07:50:51.505040+0000","last_fullsized":"2024-09-18T07:50:51.505040+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:49.580400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:50:49.580400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:19.168896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":51,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.397521+0000","last_change":"2024-09-18T07:50:23.402013+0000","last_active":"2024-09-18T07:50:51.397521+0000","last_peered":"2024-09-18T07:50:51.397521+0000","last_clean":"2024-09-18T07:50:51.397521+0000","last_became_active":"2024-09-18T07:50:23.298899+0000","last_became_peered":"2024-09-18T07:50:23.298899+0000","last_unstale":"2024-09-18T07:50:51.397521+0000","last_undegraded":"2024-09-18T07:50:51.397521+0000","last_fullsized":"2024-09-18T07:50:51.397521+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:47:41.464337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:47:41.464337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"323'19","reported_seq":757,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.507288+0000","last_change":"2024-09-18T07:47:47.469967+0000","last_active":"2024-09-18T07:50:51.507288+0000","last_peered":"2024-09-18T07:50:51.507288+0000","last_clean":"2024-09-18T07:50:51.507288+0000","last_became_active":"2024-09-18T07:47:20.142007+0000","last_became_peered":"2024-09-18T07:47:20.142007+0000","last_unstale":"2024-09-18T07:50:51.507288+0000","last_undegraded":"2024-09-18T07:50:51.507288+0000","last_fullsized":"2024-09-18T07:50:51.507288+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'19","last_scrub_stamp":"2024-09-18T07:47:47.469902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:47:47.469902+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:33.819016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":592,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-18T07:50:51.392958+0000","last_change":"2024-09-18T07:48:28.910567+0000","last_active":"2024-09-18T07:50:51.392958+0000","last_peered":"2024-09-18T07:50:51.392958+0000","last_clean":"2024-09-18T07:50:51.392958+0000","last_became_active":"2024-09-18T07:47:20.139668+0000","last_became_peered":"2024-09-18T07:47:20.139668+0000","last_unstale":"2024-09-18T07:50:51.392958+0000","last_undegraded":"2024-09-18T07:50:51.392958+0000","last_fullsized":"2024-09-18T07:50:51.392958+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:48:28.910419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:48:28.910419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1096,"num_read_kb":853,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":649,"num_bytes_recovered":14154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},"log_size":691,"ondisk_log_size":691,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":6406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":945,"num_read_kb":940,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1983,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201523,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":740472,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93004680,"statfs":{"total":95995035648,"available":95236792320,"internally_reserved":0,"allocated":1282048,"data_stored":816129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":756939210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463161,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758164,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92986988,"statfs":{"total":95995035648,"available":95218675712,"internally_reserved":0,"allocated":917504,"data_stored":492848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":775420358},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":460,"seq":1975684956206,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":170024,"kb_used_data":1488,"kb_used_omap":6,"kb_used_meta":168505,"kb_avail":93575128,"statfs":{"total":95995035648,"available":95820931072,"internally_reserved":0,"allocated":1523712,"data_stored":1084218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6877,"internal_metadata":172549411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":395,"seq":1696512081995,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":546408,"kb_used_data":916,"kb_used_omap":4,"kb_used_meta":545467,"kb_avail":93198744,"statfs":{"total":95995035648,"available":95435513856,"internally_reserved":0,"allocated":937984,"data_stored":487470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4447,"internal_metadata":558558881},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149961,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1485644,"kb_used_data":615928,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92259508,"statfs":{"total":95995035648,"available":94473736192,"internally_reserved":0,"allocated":630710272,"data_stored":630231309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":890567111},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411599,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":798740,"kb_used_data":13696,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92946412,"statfs":{"total":95995035648,"available":95177125888,"internally_reserved":0,"allocated":14024704,"data_stored":13595534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575126,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888760,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92856392,"statfs":{"total":95995035648,"available":95084945408,"internally_reserved":0,"allocated":1015808,"data_stored":496365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738651,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:50:57.506 INFO:tasks.thrashosds.thrasher:Setting 3.0 to [7, 1, 4] 2024-09-18T07:50:57.506 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.0', '7', '1', '4'] 2024-09-18T07:50:57.507 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd pg-upmap 3.0 7 1 4 2024-09-18T07:50:57.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:57 smithi137 ceph-mon[29408]: from='client.17001 v1:172.21.15.137:0/3538718630' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:50:57.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:57 smithi137 ceph-mon[39175]: from='client.17001 v1:172.21.15.137:0/3538718630' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:50:57.698 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:50:57.698 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:50:57.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:57 smithi203 ceph-mon[32237]: from='client.17001 v1:172.21.15.137:0/3538718630' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:50:58.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-18T07:50:58.314 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-18T07:50:58.315 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:58.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:50:58 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:50:58.890 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:58 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:50:58.890 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:58 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:50:58.890 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:58 smithi137 ceph-mon[29408]: pgmap v1081: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-18T07:50:58.890 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:58 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:58.891 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:58 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:50:58.891 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:58 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:50:58.891 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:58 smithi137 ceph-mon[39175]: pgmap v1081: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-18T07:50:58.891 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:58 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:58.896 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:58.933 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:50:58.960 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:58 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:50:58.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:58 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:50:58.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:58 smithi203 ceph-mon[32237]: pgmap v1081: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-18T07:50:58.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:58 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:50:59.550 INFO:teuthology.orchestra.run.smithi137.stderr:set 3.0 pg_upmap mapping to [7,1,4] 2024-09-18T07:50:59.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:59 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2332042816' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.0", "id": [7, 1, 4]} : dispatch 2024-09-18T07:50:59.788 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:59 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:50:59.788 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:50:59 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:50:59.789 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:59 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2332042816' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.0", "id": [7, 1, 4]} : dispatch 2024-09-18T07:50:59.789 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:59 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:50:59.789 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:50:59 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:50:59.789 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:50:59.789 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:50:59.862 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:59 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2332042816' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.0", "id": [7, 1, 4]} : dispatch 2024-09-18T07:50:59.862 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:59 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:50:59.862 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:50:59 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:51:00.190 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-18T07:51:00.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2332042816' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.0", "id": [7, 1, 4]}]': finished 2024-09-18T07:51:00.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[29408]: osdmap e566: 8 total, 8 up, 7 in 2024-09-18T07:51:00.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:51:00.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:51:00.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:51:00.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:51:00.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/798397210' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:51:00.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[29408]: from='client.36821 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:51:00.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[29408]: pgmap v1083: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:51:00.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[29408]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-18T07:51:00.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:51:00.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[29408]: from='client.36821 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:51:00.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[29408]: osdmap e567: 8 total, 8 up, 7 in 2024-09-18T07:51:00.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2332042816' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.0", "id": [7, 1, 4]}]': finished 2024-09-18T07:51:00.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[39175]: osdmap e566: 8 total, 8 up, 7 in 2024-09-18T07:51:00.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:51:00.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:51:00.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:51:00.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:51:00.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/798397210' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:51:00.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[39175]: from='client.36821 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:51:00.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[39175]: pgmap v1083: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:51:00.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[39175]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-18T07:51:00.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:51:00.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[39175]: from='client.36821 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:51:00.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:00 smithi137 ceph-mon[39175]: osdmap e567: 8 total, 8 up, 7 in 2024-09-18T07:51:00.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:00 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2332042816' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.0", "id": [7, 1, 4]}]': finished 2024-09-18T07:51:00.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:00 smithi203 ceph-mon[32237]: osdmap e566: 8 total, 8 up, 7 in 2024-09-18T07:51:00.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:00 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:51:00.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:00 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:51:00.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:00 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:51:00.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:00 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:51:00.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:00 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/798397210' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:51:00.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:00 smithi203 ceph-mon[32237]: from='client.36821 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-18T07:51:00.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:00 smithi203 ceph-mon[32237]: pgmap v1083: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:51:00.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:00 smithi203 ceph-mon[32237]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-18T07:51:00.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:00 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:51:00.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:00 smithi203 ceph-mon[32237]: from='client.36821 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-18T07:51:00.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:00 smithi203 ceph-mon[32237]: osdmap e567: 8 total, 8 up, 7 in 2024-09-18T07:51:01.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-18T07:51:01.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-18T07:51:01.869 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:01 smithi137 ceph-mon[29408]: 6.8 scrub starts 2024-09-18T07:51:01.869 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:01 smithi137 ceph-mon[29408]: 6.8 scrub ok 2024-09-18T07:51:01.870 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:01 smithi137 ceph-mon[29408]: 6.b scrub starts 2024-09-18T07:51:01.870 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:01 smithi137 ceph-mon[29408]: 6.b scrub ok 2024-09-18T07:51:01.870 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:01 smithi137 ceph-mon[39175]: 6.8 scrub starts 2024-09-18T07:51:01.870 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:01 smithi137 ceph-mon[39175]: 6.8 scrub ok 2024-09-18T07:51:01.870 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:01 smithi137 ceph-mon[39175]: 6.b scrub starts 2024-09-18T07:51:01.870 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:01 smithi137 ceph-mon[39175]: 6.b scrub ok 2024-09-18T07:51:01.960 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:01.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:01 smithi203 ceph-mon[32237]: 6.8 scrub starts 2024-09-18T07:51:01.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:01 smithi203 ceph-mon[32237]: 6.8 scrub ok 2024-09-18T07:51:01.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:01 smithi203 ceph-mon[32237]: 6.b scrub starts 2024-09-18T07:51:01.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:01 smithi203 ceph-mon[32237]: 6.b scrub ok 2024-09-18T07:51:01.982 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:02.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:02 smithi137 ceph-mon[29408]: 5.10 scrub starts 2024-09-18T07:51:02.861 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:02 smithi137 ceph-mon[29408]: 5.10 scrub ok 2024-09-18T07:51:02.861 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:02 smithi137 ceph-mon[29408]: 6.c scrub starts 2024-09-18T07:51:02.861 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:02 smithi137 ceph-mon[29408]: 6.c scrub ok 2024-09-18T07:51:02.861 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:02 smithi137 ceph-mon[29408]: pgmap v1085: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 760 B/s rd, 0 op/s 2024-09-18T07:51:02.861 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:02 smithi137 ceph-mon[29408]: 4.18 scrub starts 2024-09-18T07:51:02.861 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:02 smithi137 ceph-mon[29408]: 4.18 scrub ok 2024-09-18T07:51:02.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:02 smithi137 ceph-mon[39175]: 5.10 scrub starts 2024-09-18T07:51:02.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:02 smithi137 ceph-mon[39175]: 5.10 scrub ok 2024-09-18T07:51:02.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:02 smithi137 ceph-mon[39175]: 6.c scrub starts 2024-09-18T07:51:02.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:02 smithi137 ceph-mon[39175]: 6.c scrub ok 2024-09-18T07:51:02.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:02 smithi137 ceph-mon[39175]: pgmap v1085: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 760 B/s rd, 0 op/s 2024-09-18T07:51:02.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:02 smithi137 ceph-mon[39175]: 4.18 scrub starts 2024-09-18T07:51:02.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:02 smithi137 ceph-mon[39175]: 4.18 scrub ok 2024-09-18T07:51:02.884 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:51:02.884 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:51:02.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:02 smithi203 ceph-mon[32237]: 5.10 scrub starts 2024-09-18T07:51:02.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:02 smithi203 ceph-mon[32237]: 5.10 scrub ok 2024-09-18T07:51:02.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:02 smithi203 ceph-mon[32237]: 6.c scrub starts 2024-09-18T07:51:02.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:02 smithi203 ceph-mon[32237]: 6.c scrub ok 2024-09-18T07:51:02.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:02 smithi203 ceph-mon[32237]: pgmap v1085: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 760 B/s rd, 0 op/s 2024-09-18T07:51:02.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:02 smithi203 ceph-mon[32237]: 4.18 scrub starts 2024-09-18T07:51:02.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:02 smithi203 ceph-mon[32237]: 4.18 scrub ok 2024-09-18T07:51:03.569 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-18T07:51:03.890 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:03 smithi137 ceph-mon[29408]: 3.16 scrub starts 2024-09-18T07:51:03.890 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:03 smithi137 ceph-mon[29408]: 3.16 scrub ok 2024-09-18T07:51:03.890 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:03 smithi137 ceph-mon[29408]: 3.a scrub starts 2024-09-18T07:51:03.890 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:03 smithi137 ceph-mon[29408]: 3.a scrub ok 2024-09-18T07:51:03.891 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:03 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2045852309' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:51:03.891 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:03 smithi137 ceph-mon[29408]: from='client.36827 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:51:03.891 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:03 smithi137 ceph-mon[29408]: 5.5 scrub starts 2024-09-18T07:51:03.891 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:03 smithi137 ceph-mon[29408]: 5.5 scrub ok 2024-09-18T07:51:03.891 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:03 smithi137 ceph-mon[39175]: 3.16 scrub starts 2024-09-18T07:51:03.891 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:03 smithi137 ceph-mon[39175]: 3.16 scrub ok 2024-09-18T07:51:03.891 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:03 smithi137 ceph-mon[39175]: 3.a scrub starts 2024-09-18T07:51:03.891 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:03 smithi137 ceph-mon[39175]: 3.a scrub ok 2024-09-18T07:51:03.891 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:03 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2045852309' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:51:03.891 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:03 smithi137 ceph-mon[39175]: from='client.36827 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:51:03.891 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:03 smithi137 ceph-mon[39175]: 5.5 scrub starts 2024-09-18T07:51:03.891 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:03 smithi137 ceph-mon[39175]: 5.5 scrub ok 2024-09-18T07:51:03.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-18T07:51:03.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:03 smithi203 ceph-mon[32237]: 3.16 scrub starts 2024-09-18T07:51:03.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:03 smithi203 ceph-mon[32237]: 3.16 scrub ok 2024-09-18T07:51:03.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:03 smithi203 ceph-mon[32237]: 3.a scrub starts 2024-09-18T07:51:03.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:03 smithi203 ceph-mon[32237]: 3.a scrub ok 2024-09-18T07:51:03.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:03 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2045852309' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:51:03.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:03 smithi203 ceph-mon[32237]: from='client.36827 ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-18T07:51:03.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:03 smithi203 ceph-mon[32237]: 5.5 scrub starts 2024-09-18T07:51:03.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:03 smithi203 ceph-mon[32237]: 5.5 scrub ok 2024-09-18T07:51:04.221 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:04.654 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:04 smithi137 ceph-mon[39175]: 3.4 scrub starts 2024-09-18T07:51:04.654 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:04 smithi137 ceph-mon[39175]: 3.4 scrub ok 2024-09-18T07:51:04.654 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:04 smithi137 ceph-mon[39175]: from='client.36827 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:51:04.654 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:04 smithi137 ceph-mon[39175]: osdmap e568: 8 total, 8 up, 7 in 2024-09-18T07:51:04.654 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:04 smithi137 ceph-mon[39175]: 3.18 scrub starts 2024-09-18T07:51:04.654 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:04 smithi137 ceph-mon[39175]: 3.18 scrub ok 2024-09-18T07:51:04.654 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:04 smithi137 ceph-mon[39175]: pgmap v1087: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:51:04.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:04 smithi137 ceph-mon[29408]: 3.4 scrub starts 2024-09-18T07:51:04.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:04 smithi137 ceph-mon[29408]: 3.4 scrub ok 2024-09-18T07:51:04.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:04 smithi137 ceph-mon[29408]: from='client.36827 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:51:04.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:04 smithi137 ceph-mon[29408]: osdmap e568: 8 total, 8 up, 7 in 2024-09-18T07:51:04.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:04 smithi137 ceph-mon[29408]: 3.18 scrub starts 2024-09-18T07:51:04.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:04 smithi137 ceph-mon[29408]: 3.18 scrub ok 2024-09-18T07:51:04.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:04 smithi137 ceph-mon[29408]: pgmap v1087: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:51:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:04 smithi203 ceph-mon[32237]: 3.4 scrub starts 2024-09-18T07:51:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:04 smithi203 ceph-mon[32237]: 3.4 scrub ok 2024-09-18T07:51:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:04 smithi203 ceph-mon[32237]: from='client.36827 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-18T07:51:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:04 smithi203 ceph-mon[32237]: osdmap e568: 8 total, 8 up, 7 in 2024-09-18T07:51:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:04 smithi203 ceph-mon[32237]: 3.18 scrub starts 2024-09-18T07:51:04.961 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:04 smithi203 ceph-mon[32237]: 3.18 scrub ok 2024-09-18T07:51:04.962 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:04 smithi203 ceph-mon[32237]: pgmap v1087: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:51:05.132 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-18T07:51:05.132 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:51:05.714 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-18T07:51:05.976 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:51:06.396 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:06 smithi203 ceph-mon[32237]: 4.8 scrub starts 2024-09-18T07:51:06.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:06 smithi203 ceph-mon[32237]: 4.8 scrub ok 2024-09-18T07:51:06.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:06 smithi203 ceph-mon[32237]: osdmap e569: 8 total, 8 up, 7 in 2024-09-18T07:51:06.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:06 smithi203 ceph-mon[32237]: 5.b scrub starts 2024-09-18T07:51:06.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:06 smithi203 ceph-mon[32237]: 5.b scrub ok 2024-09-18T07:51:06.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:06 smithi203 ceph-mon[32237]: 1.3 scrub starts 2024-09-18T07:51:06.397 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:06 smithi203 ceph-mon[32237]: 1.3 scrub ok 2024-09-18T07:51:06.629 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:51:06.629 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:51:06.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:06 smithi137 ceph-mon[29408]: 4.8 scrub starts 2024-09-18T07:51:06.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:06 smithi137 ceph-mon[29408]: 4.8 scrub ok 2024-09-18T07:51:06.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:06 smithi137 ceph-mon[29408]: osdmap e569: 8 total, 8 up, 7 in 2024-09-18T07:51:06.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:06 smithi137 ceph-mon[29408]: 5.b scrub starts 2024-09-18T07:51:06.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:06 smithi137 ceph-mon[29408]: 5.b scrub ok 2024-09-18T07:51:06.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:06 smithi137 ceph-mon[29408]: 1.3 scrub starts 2024-09-18T07:51:06.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:06 smithi137 ceph-mon[29408]: 1.3 scrub ok 2024-09-18T07:51:06.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:06 smithi137 ceph-mon[39175]: 4.8 scrub starts 2024-09-18T07:51:06.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:06 smithi137 ceph-mon[39175]: 4.8 scrub ok 2024-09-18T07:51:06.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:06 smithi137 ceph-mon[39175]: osdmap e569: 8 total, 8 up, 7 in 2024-09-18T07:51:06.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:06 smithi137 ceph-mon[39175]: 5.b scrub starts 2024-09-18T07:51:06.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:06 smithi137 ceph-mon[39175]: 5.b scrub ok 2024-09-18T07:51:06.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:06 smithi137 ceph-mon[39175]: 1.3 scrub starts 2024-09-18T07:51:06.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:06 smithi137 ceph-mon[39175]: 1.3 scrub ok 2024-09-18T07:51:06.897 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-18T07:51:06.897 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-09-18T07:51:06.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd in 0 2024-09-18T07:51:07.158 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:07.196 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:51:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:51:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:51:07.364 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-18T07:51:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:07 smithi203 ceph-mon[32237]: pgmap v1089: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-18T07:51:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:07 smithi203 ceph-mon[32237]: 3.1e scrub starts 2024-09-18T07:51:07.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:07 smithi203 ceph-mon[32237]: 3.1e scrub ok 2024-09-18T07:51:07.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:07 smithi137 ceph-mon[29408]: pgmap v1089: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-18T07:51:07.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:07 smithi137 ceph-mon[29408]: 3.1e scrub starts 2024-09-18T07:51:07.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:07 smithi137 ceph-mon[29408]: 3.1e scrub ok 2024-09-18T07:51:07.462 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:07 smithi137 ceph-mon[39175]: pgmap v1089: 142 pgs: 142 active+clean; 583 KiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-18T07:51:07.462 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:07 smithi137 ceph-mon[39175]: 3.1e scrub starts 2024-09-18T07:51:07.462 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:07 smithi137 ceph-mon[39175]: 3.1e scrub ok 2024-09-18T07:51:07.618 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:51:08.156 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:51:08.156 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:51:08.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a[29404]: 2024-09-18T07:51:07.820+0000 7fe5973a1640 -1 mon.a@0(leader).osd e569 definitely_dead 0 2024-09-18T07:51:08.206 INFO:teuthology.orchestra.run.smithi137.stderr:marked in osd.0. 2024-09-18T07:51:08.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:51:08 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:51:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:08 smithi203 ceph-mon[32237]: 6.4 scrub starts 2024-09-18T07:51:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:08 smithi203 ceph-mon[32237]: 6.4 scrub ok 2024-09-18T07:51:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:08 smithi203 ceph-mon[32237]: 5.1f scrub starts 2024-09-18T07:51:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:08 smithi203 ceph-mon[32237]: 5.1f scrub ok 2024-09-18T07:51:08.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:08 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4183183727' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-18T07:51:08.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:08 smithi203 ceph-mon[32237]: from='client.36833 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-18T07:51:08.594 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:08 smithi137 ceph-mon[29408]: 6.4 scrub starts 2024-09-18T07:51:08.594 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:08 smithi137 ceph-mon[29408]: 6.4 scrub ok 2024-09-18T07:51:08.594 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:08 smithi137 ceph-mon[29408]: 5.1f scrub starts 2024-09-18T07:51:08.594 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:08 smithi137 ceph-mon[29408]: 5.1f scrub ok 2024-09-18T07:51:08.594 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:08 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4183183727' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-18T07:51:08.594 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:08 smithi137 ceph-mon[29408]: from='client.36833 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-18T07:51:08.595 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:08 smithi137 ceph-mon[39175]: 6.4 scrub starts 2024-09-18T07:51:08.595 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:08 smithi137 ceph-mon[39175]: 6.4 scrub ok 2024-09-18T07:51:08.595 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:08 smithi137 ceph-mon[39175]: 5.1f scrub starts 2024-09-18T07:51:08.595 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:08 smithi137 ceph-mon[39175]: 5.1f scrub ok 2024-09-18T07:51:08.595 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:08 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4183183727' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-18T07:51:08.595 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:08 smithi137 ceph-mon[39175]: from='client.36833 ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-18T07:51:08.798 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-18T07:51:09.037 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-09-18T07:51:09.038 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-18T07:51:09.048 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:51:09.321 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:09.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:09 smithi137 ceph-mon[29408]: pgmap v1090: 142 pgs: 142 active+clean; 583 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-18T07:51:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:09 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:51:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:09 smithi137 ceph-mon[29408]: from='client.36833 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-18T07:51:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:09 smithi137 ceph-mon[29408]: osdmap e570: 8 total, 8 up, 8 in 2024-09-18T07:51:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:09 smithi137 ceph-mon[29408]: 6.14 scrub starts 2024-09-18T07:51:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:09 smithi137 ceph-mon[29408]: 6.14 scrub ok 2024-09-18T07:51:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:09 smithi137 ceph-mon[29408]: 6.1 scrub starts 2024-09-18T07:51:09.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:09 smithi137 ceph-mon[29408]: 6.1 scrub ok 2024-09-18T07:51:09.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:09 smithi137 ceph-mon[39175]: pgmap v1090: 142 pgs: 142 active+clean; 583 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-18T07:51:09.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:09 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:51:09.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:09 smithi137 ceph-mon[39175]: from='client.36833 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-18T07:51:09.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:09 smithi137 ceph-mon[39175]: osdmap e570: 8 total, 8 up, 8 in 2024-09-18T07:51:09.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:09 smithi137 ceph-mon[39175]: 6.14 scrub starts 2024-09-18T07:51:09.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:09 smithi137 ceph-mon[39175]: 6.14 scrub ok 2024-09-18T07:51:09.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:09 smithi137 ceph-mon[39175]: 6.1 scrub starts 2024-09-18T07:51:09.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:09 smithi137 ceph-mon[39175]: 6.1 scrub ok 2024-09-18T07:51:09.422 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:09 smithi203 ceph-mon[32237]: pgmap v1090: 142 pgs: 142 active+clean; 583 KiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-09-18T07:51:09.422 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:09 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:51:09.423 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:09 smithi203 ceph-mon[32237]: from='client.36833 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-18T07:51:09.423 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:09 smithi203 ceph-mon[32237]: osdmap e570: 8 total, 8 up, 8 in 2024-09-18T07:51:09.423 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:09 smithi203 ceph-mon[32237]: 6.14 scrub starts 2024-09-18T07:51:09.423 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:09 smithi203 ceph-mon[32237]: 6.14 scrub ok 2024-09-18T07:51:09.423 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:09 smithi203 ceph-mon[32237]: 6.1 scrub starts 2024-09-18T07:51:09.423 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:09 smithi203 ceph-mon[32237]: 6.1 scrub ok 2024-09-18T07:51:09.523 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:51:09.523 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:51:09.796 INFO:teuthology.orchestra.run.smithi137.stdout:mon.a: {} 2024-09-18T07:51:09.796 INFO:teuthology.orchestra.run.smithi137.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T07:51:09.805 INFO:teuthology.orchestra.run.smithi137.stdout:mon.b: {} 2024-09-18T07:51:09.805 INFO:teuthology.orchestra.run.smithi137.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T07:51:09.813 INFO:teuthology.orchestra.run.smithi137.stdout:mon.c: {} 2024-09-18T07:51:09.813 INFO:teuthology.orchestra.run.smithi137.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-18T07:51:10.132 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-18T07:51:10.388 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:51:10.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:10 smithi137 ceph-mon[39175]: osdmap e571: 8 total, 8 up, 8 in 2024-09-18T07:51:10.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:10 smithi137 ceph-mon[39175]: 6.0 scrub starts 2024-09-18T07:51:10.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:10 smithi137 ceph-mon[39175]: 6.0 scrub ok 2024-09-18T07:51:10.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:10 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2766093383' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:51:10.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:10 smithi137 ceph-mon[29408]: osdmap e571: 8 total, 8 up, 8 in 2024-09-18T07:51:10.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:10 smithi137 ceph-mon[29408]: 6.0 scrub starts 2024-09-18T07:51:10.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:10 smithi137 ceph-mon[29408]: 6.0 scrub ok 2024-09-18T07:51:10.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:10 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2766093383' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:51:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:10 smithi203 ceph-mon[32237]: osdmap e571: 8 total, 8 up, 8 in 2024-09-18T07:51:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:10 smithi203 ceph-mon[32237]: 6.0 scrub starts 2024-09-18T07:51:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:10 smithi203 ceph-mon[32237]: 6.0 scrub ok 2024-09-18T07:51:10.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:10 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2766093383' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-18T07:51:10.497 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-18T07:51:10.752 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:10.973 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:51:10.973 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:51:11.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:11 smithi137 ceph-mon[29408]: pgmap v1093: 142 pgs: 1 active+recovering+degraded, 11 peering, 1 active+recovering, 129 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 8/597 objects misplaced (1.340%); 0 B/s, 0 objects/s recovering 2024-09-18T07:51:11.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:11 smithi137 ceph-mon[29408]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-18T07:51:11.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:11 smithi137 ceph-mon[29408]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-09-18T07:51:11.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:11 smithi137 ceph-mon[29408]: 4.19 scrub starts 2024-09-18T07:51:11.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:11 smithi137 ceph-mon[29408]: 4.19 scrub ok 2024-09-18T07:51:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:11 smithi137 ceph-mon[39175]: pgmap v1093: 142 pgs: 1 active+recovering+degraded, 11 peering, 1 active+recovering, 129 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 8/597 objects misplaced (1.340%); 0 B/s, 0 objects/s recovering 2024-09-18T07:51:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:11 smithi137 ceph-mon[39175]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-18T07:51:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:11 smithi137 ceph-mon[39175]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-09-18T07:51:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:11 smithi137 ceph-mon[39175]: 4.19 scrub starts 2024-09-18T07:51:11.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:11 smithi137 ceph-mon[39175]: 4.19 scrub ok 2024-09-18T07:51:11.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:11 smithi203 ceph-mon[32237]: pgmap v1093: 142 pgs: 1 active+recovering+degraded, 11 peering, 1 active+recovering, 129 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 8/597 objects misplaced (1.340%); 0 B/s, 0 objects/s recovering 2024-09-18T07:51:11.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:11 smithi203 ceph-mon[32237]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-18T07:51:11.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:11 smithi203 ceph-mon[32237]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-09-18T07:51:11.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:11 smithi203 ceph-mon[32237]: 4.19 scrub starts 2024-09-18T07:51:11.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:11 smithi203 ceph-mon[32237]: 4.19 scrub ok 2024-09-18T07:51:11.622 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-18T07:51:11.873 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:51:12.218 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.0 to 1 (10000) 2024-09-18T07:51:12.290 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:12 smithi203 ceph-mon[32237]: 5.7 scrub starts 2024-09-18T07:51:12.291 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:12 smithi203 ceph-mon[32237]: 5.7 scrub ok 2024-09-18T07:51:12.291 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:12 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2200814541' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:51:12.390 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:51:12.390 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:51:12.621 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:12 smithi137 ceph-mon[29408]: 5.7 scrub starts 2024-09-18T07:51:12.621 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:12 smithi137 ceph-mon[29408]: 5.7 scrub ok 2024-09-18T07:51:12.621 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:12 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2200814541' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:51:12.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:12 smithi137 ceph-mon[39175]: 5.7 scrub starts 2024-09-18T07:51:12.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:12 smithi137 ceph-mon[39175]: 5.7 scrub ok 2024-09-18T07:51:12.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:12 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2200814541' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:51:13.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-18T07:51:13.109 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-18T07:51:13.266 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:13.360 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:51:13.387 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[29408]: 3.0 scrub starts 2024-09-18T07:51:13.387 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[29408]: 3.0 scrub ok 2024-09-18T07:51:13.387 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[29408]: pgmap v1094: 142 pgs: 1 active+recovering+degraded, 11 peering, 1 active+recovering, 129 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 105 op/s; 8/597 objects misplaced (1.340%); 212 B/s, 2 objects/s recovering 2024-09-18T07:51:13.388 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[29408]: 4.7 scrub starts 2024-09-18T07:51:13.388 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[29408]: 4.7 scrub ok 2024-09-18T07:51:13.388 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2200814541' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:51:13.388 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[29408]: osdmap e572: 8 total, 8 up, 8 in 2024-09-18T07:51:13.388 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:51:13.388 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:51:13.388 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[39175]: 3.0 scrub starts 2024-09-18T07:51:13.388 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[39175]: 3.0 scrub ok 2024-09-18T07:51:13.388 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[39175]: pgmap v1094: 142 pgs: 1 active+recovering+degraded, 11 peering, 1 active+recovering, 129 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 105 op/s; 8/597 objects misplaced (1.340%); 212 B/s, 2 objects/s recovering 2024-09-18T07:51:13.388 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[39175]: 4.7 scrub starts 2024-09-18T07:51:13.389 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[39175]: 4.7 scrub ok 2024-09-18T07:51:13.389 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2200814541' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:51:13.389 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[39175]: osdmap e572: 8 total, 8 up, 8 in 2024-09-18T07:51:13.389 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:51:13.389 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:13 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:51:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:13 smithi203 ceph-mon[32237]: 3.0 scrub starts 2024-09-18T07:51:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:13 smithi203 ceph-mon[32237]: 3.0 scrub ok 2024-09-18T07:51:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:13 smithi203 ceph-mon[32237]: pgmap v1094: 142 pgs: 1 active+recovering+degraded, 11 peering, 1 active+recovering, 129 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 105 op/s; 8/597 objects misplaced (1.340%); 212 B/s, 2 objects/s recovering 2024-09-18T07:51:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:13 smithi203 ceph-mon[32237]: 4.7 scrub starts 2024-09-18T07:51:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:13 smithi203 ceph-mon[32237]: 4.7 scrub ok 2024-09-18T07:51:13.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:13 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2200814541' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:51:13.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:13 smithi203 ceph-mon[32237]: osdmap e572: 8 total, 8 up, 8 in 2024-09-18T07:51:13.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:13 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:51:13.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:13 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:51:13.883 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:51:13.883 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:51:14.225 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-18T07:51:14.537 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:14 smithi137 ceph-mon[29408]: 3.1b scrub starts 2024-09-18T07:51:14.537 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:14 smithi137 ceph-mon[29408]: 3.1b scrub ok 2024-09-18T07:51:14.537 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:14 smithi137 ceph-mon[29408]: 6.e scrub starts 2024-09-18T07:51:14.537 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:14 smithi137 ceph-mon[29408]: 6.e scrub ok 2024-09-18T07:51:14.537 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:14 smithi137 ceph-mon[29408]: 2.1 scrub starts 2024-09-18T07:51:14.537 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:14 smithi137 ceph-mon[29408]: 2.1 scrub ok 2024-09-18T07:51:14.537 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:14 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3791474732' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:51:14.537 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:14 smithi137 ceph-mon[39175]: 3.1b scrub starts 2024-09-18T07:51:14.538 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:14 smithi137 ceph-mon[39175]: 3.1b scrub ok 2024-09-18T07:51:14.538 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:14 smithi137 ceph-mon[39175]: 6.e scrub starts 2024-09-18T07:51:14.538 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:14 smithi137 ceph-mon[39175]: 6.e scrub ok 2024-09-18T07:51:14.538 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:14 smithi137 ceph-mon[39175]: 2.1 scrub starts 2024-09-18T07:51:14.538 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:14 smithi137 ceph-mon[39175]: 2.1 scrub ok 2024-09-18T07:51:14.538 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:14 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3791474732' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:51:14.697 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:14 smithi203 ceph-mon[32237]: 3.1b scrub starts 2024-09-18T07:51:14.697 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:14 smithi203 ceph-mon[32237]: 3.1b scrub ok 2024-09-18T07:51:14.697 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:14 smithi203 ceph-mon[32237]: 6.e scrub starts 2024-09-18T07:51:14.698 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:14 smithi203 ceph-mon[32237]: 6.e scrub ok 2024-09-18T07:51:14.698 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:14 smithi203 ceph-mon[32237]: 2.1 scrub starts 2024-09-18T07:51:14.698 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:14 smithi203 ceph-mon[32237]: 2.1 scrub ok 2024-09-18T07:51:14.698 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:14 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3791474732' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-18T07:51:14.730 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-18T07:51:14.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-18T07:51:14.980 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:51:15.148 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:15.410 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:15 smithi203 ceph-mon[32237]: 5.19 scrub starts 2024-09-18T07:51:15.410 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:15 smithi203 ceph-mon[32237]: 5.19 scrub ok 2024-09-18T07:51:15.410 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:15 smithi203 ceph-mon[32237]: 6.10 scrub starts 2024-09-18T07:51:15.410 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:15 smithi203 ceph-mon[32237]: 6.10 scrub ok 2024-09-18T07:51:15.410 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:15 smithi203 ceph-mon[32237]: pgmap v1096: 142 pgs: 1 active+recovering+degraded, 1 active+recovering, 140 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 11 op/s; 8/597 objects misplaced (1.340%); 490 B/s, 8 objects/s recovering 2024-09-18T07:51:15.410 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:15 smithi203 ceph-mon[32237]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-09-18T07:51:15.410 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:15 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3791474732' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:51:15.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:15 smithi203 ceph-mon[32237]: osdmap e573: 8 total, 8 up, 8 in 2024-09-18T07:51:15.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:15 smithi203 ceph-mon[32237]: 3.c scrub starts 2024-09-18T07:51:15.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:15 smithi203 ceph-mon[32237]: 3.c scrub ok 2024-09-18T07:51:15.411 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:15 smithi203 ceph-mon[32237]: osdmap e574: 8 total, 8 up, 8 in 2024-09-18T07:51:15.411 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:51:15.411 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:51:15.583 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[39175]: 5.19 scrub starts 2024-09-18T07:51:15.583 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[39175]: 5.19 scrub ok 2024-09-18T07:51:15.583 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[39175]: 6.10 scrub starts 2024-09-18T07:51:15.583 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[39175]: 6.10 scrub ok 2024-09-18T07:51:15.583 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[39175]: pgmap v1096: 142 pgs: 1 active+recovering+degraded, 1 active+recovering, 140 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 11 op/s; 8/597 objects misplaced (1.340%); 490 B/s, 8 objects/s recovering 2024-09-18T07:51:15.583 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[39175]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-09-18T07:51:15.583 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3791474732' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:51:15.583 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[39175]: osdmap e573: 8 total, 8 up, 8 in 2024-09-18T07:51:15.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[39175]: 3.c scrub starts 2024-09-18T07:51:15.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[39175]: 3.c scrub ok 2024-09-18T07:51:15.584 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[39175]: osdmap e574: 8 total, 8 up, 8 in 2024-09-18T07:51:15.584 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[29408]: 5.19 scrub starts 2024-09-18T07:51:15.584 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[29408]: 5.19 scrub ok 2024-09-18T07:51:15.584 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[29408]: 6.10 scrub starts 2024-09-18T07:51:15.584 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[29408]: 6.10 scrub ok 2024-09-18T07:51:15.584 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[29408]: pgmap v1096: 142 pgs: 1 active+recovering+degraded, 1 active+recovering, 140 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 11 op/s; 8/597 objects misplaced (1.340%); 490 B/s, 8 objects/s recovering 2024-09-18T07:51:15.584 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[29408]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-09-18T07:51:15.584 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3791474732' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-18T07:51:15.584 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[29408]: osdmap e573: 8 total, 8 up, 8 in 2024-09-18T07:51:15.585 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[29408]: 3.c scrub starts 2024-09-18T07:51:15.585 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[29408]: 3.c scrub ok 2024-09-18T07:51:15.585 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:15 smithi137 ceph-mon[29408]: osdmap e574: 8 total, 8 up, 8 in 2024-09-18T07:51:16.240 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.1 to 1 (10000) 2024-09-18T07:51:16.388 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-18T07:51:16.620 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:16 smithi137 ceph-mon[29408]: 4.17 scrub starts 2024-09-18T07:51:16.620 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:16 smithi137 ceph-mon[29408]: 4.17 scrub ok 2024-09-18T07:51:16.621 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:16 smithi137 ceph-mon[29408]: 4.0 scrub starts 2024-09-18T07:51:16.621 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:16 smithi137 ceph-mon[29408]: 4.0 scrub ok 2024-09-18T07:51:16.621 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:16 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2497835195' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:51:16.621 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:16 smithi137 ceph-mon[39175]: 4.17 scrub starts 2024-09-18T07:51:16.621 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:16 smithi137 ceph-mon[39175]: 4.17 scrub ok 2024-09-18T07:51:16.621 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:16 smithi137 ceph-mon[39175]: 4.0 scrub starts 2024-09-18T07:51:16.621 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:16 smithi137 ceph-mon[39175]: 4.0 scrub ok 2024-09-18T07:51:16.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:16 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2497835195' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:51:16.639 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:51:16.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:16 smithi203 ceph-mon[32237]: 4.17 scrub starts 2024-09-18T07:51:16.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:16 smithi203 ceph-mon[32237]: 4.17 scrub ok 2024-09-18T07:51:16.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:16 smithi203 ceph-mon[32237]: 4.0 scrub starts 2024-09-18T07:51:16.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:16 smithi203 ceph-mon[32237]: 4.0 scrub ok 2024-09-18T07:51:16.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:16 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2497835195' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:51:17.071 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-18T07:51:17.134 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:51:17.135 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:51:17.324 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:17.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:51:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:51:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[39175]: 3.12 scrub starts 2024-09-18T07:51:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[39175]: 3.12 scrub ok 2024-09-18T07:51:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[39175]: pgmap v1099: 142 pgs: 1 active+recovering+degraded, 141 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 8/597 objects misplaced (1.340%); 490 B/s, 8 objects/s recovering 2024-09-18T07:51:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2497835195' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:51:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[39175]: osdmap e575: 8 total, 8 up, 8 in 2024-09-18T07:51:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[39175]: 5.a scrub starts 2024-09-18T07:51:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[39175]: 5.a scrub ok 2024-09-18T07:51:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[39175]: 3.10 scrub starts 2024-09-18T07:51:17.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[39175]: 3.10 scrub ok 2024-09-18T07:51:17.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[29408]: 3.12 scrub starts 2024-09-18T07:51:17.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[29408]: 3.12 scrub ok 2024-09-18T07:51:17.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[29408]: pgmap v1099: 142 pgs: 1 active+recovering+degraded, 141 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 8/597 objects misplaced (1.340%); 490 B/s, 8 objects/s recovering 2024-09-18T07:51:17.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2497835195' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:51:17.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[29408]: osdmap e575: 8 total, 8 up, 8 in 2024-09-18T07:51:17.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[29408]: 5.a scrub starts 2024-09-18T07:51:17.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[29408]: 5.a scrub ok 2024-09-18T07:51:17.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[29408]: 3.10 scrub starts 2024-09-18T07:51:17.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:17 smithi137 ceph-mon[29408]: 3.10 scrub ok 2024-09-18T07:51:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:17 smithi203 ceph-mon[32237]: 3.12 scrub starts 2024-09-18T07:51:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:17 smithi203 ceph-mon[32237]: 3.12 scrub ok 2024-09-18T07:51:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:17 smithi203 ceph-mon[32237]: pgmap v1099: 142 pgs: 1 active+recovering+degraded, 141 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 8/597 objects misplaced (1.340%); 490 B/s, 8 objects/s recovering 2024-09-18T07:51:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:17 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2497835195' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:51:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:17 smithi203 ceph-mon[32237]: osdmap e575: 8 total, 8 up, 8 in 2024-09-18T07:51:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:17 smithi203 ceph-mon[32237]: 5.a scrub starts 2024-09-18T07:51:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:17 smithi203 ceph-mon[32237]: 5.a scrub ok 2024-09-18T07:51:17.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:17 smithi203 ceph-mon[32237]: 3.10 scrub starts 2024-09-18T07:51:17.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:17 smithi203 ceph-mon[32237]: 3.10 scrub ok 2024-09-18T07:51:17.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-18T07:51:18.011 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:51:18.252 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-18T07:51:18.381 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:51:18 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:51:18.381 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:18 smithi203 ceph-mon[32237]: 6.1c scrub starts 2024-09-18T07:51:18.381 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:18 smithi203 ceph-mon[32237]: 6.1c scrub ok 2024-09-18T07:51:18.381 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:18 smithi203 ceph-mon[32237]: 6.18 scrub starts 2024-09-18T07:51:18.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:18 smithi203 ceph-mon[32237]: 6.18 scrub ok 2024-09-18T07:51:18.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:18 smithi203 ceph-mon[32237]: 5.8 scrub starts 2024-09-18T07:51:18.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:18 smithi203 ceph-mon[32237]: 5.8 scrub ok 2024-09-18T07:51:18.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:18 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3284551756' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:51:18.382 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:18 smithi203 ceph-mon[32237]: from='client.36869 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:51:18.481 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:51:18.481 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:51:18.621 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:18 smithi137 ceph-mon[29408]: 6.1c scrub starts 2024-09-18T07:51:18.621 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:18 smithi137 ceph-mon[29408]: 6.1c scrub ok 2024-09-18T07:51:18.622 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:18 smithi137 ceph-mon[29408]: 6.18 scrub starts 2024-09-18T07:51:18.622 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:18 smithi137 ceph-mon[29408]: 6.18 scrub ok 2024-09-18T07:51:18.622 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:18 smithi137 ceph-mon[29408]: 5.8 scrub starts 2024-09-18T07:51:18.622 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:18 smithi137 ceph-mon[29408]: 5.8 scrub ok 2024-09-18T07:51:18.622 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:18 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3284551756' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:51:18.622 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:18 smithi137 ceph-mon[29408]: from='client.36869 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:51:18.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:18 smithi137 ceph-mon[39175]: 6.1c scrub starts 2024-09-18T07:51:18.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:18 smithi137 ceph-mon[39175]: 6.1c scrub ok 2024-09-18T07:51:18.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:18 smithi137 ceph-mon[39175]: 6.18 scrub starts 2024-09-18T07:51:18.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:18 smithi137 ceph-mon[39175]: 6.18 scrub ok 2024-09-18T07:51:18.623 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:18 smithi137 ceph-mon[39175]: 5.8 scrub starts 2024-09-18T07:51:18.623 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:18 smithi137 ceph-mon[39175]: 5.8 scrub ok 2024-09-18T07:51:18.623 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:18 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3284551756' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:51:18.623 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:18 smithi137 ceph-mon[39175]: from='client.36869 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-18T07:51:19.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-18T07:51:19.262 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-18T07:51:19.319 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:19.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[29408]: 3.b scrub starts 2024-09-18T07:51:19.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[29408]: 3.b scrub ok 2024-09-18T07:51:19.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[29408]: pgmap v1101: 142 pgs: 1 active+recovering+degraded, 141 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 717 B/s rd, 0 op/s; 8/597 objects misplaced (1.340%); 161 B/s, 3 objects/s recovering 2024-09-18T07:51:19.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[29408]: 6.1a scrub starts 2024-09-18T07:51:19.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[29408]: 6.1a scrub ok 2024-09-18T07:51:19.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:51:19.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[29408]: from='client.36869 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:51:19.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[29408]: osdmap e576: 8 total, 8 up, 8 in 2024-09-18T07:51:19.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[29408]: osdmap e577: 8 total, 8 up, 8 in 2024-09-18T07:51:19.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[39175]: 3.b scrub starts 2024-09-18T07:51:19.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[39175]: 3.b scrub ok 2024-09-18T07:51:19.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[39175]: pgmap v1101: 142 pgs: 1 active+recovering+degraded, 141 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 717 B/s rd, 0 op/s; 8/597 objects misplaced (1.340%); 161 B/s, 3 objects/s recovering 2024-09-18T07:51:19.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[39175]: 6.1a scrub starts 2024-09-18T07:51:19.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[39175]: 6.1a scrub ok 2024-09-18T07:51:19.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:51:19.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[39175]: from='client.36869 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:51:19.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[39175]: osdmap e576: 8 total, 8 up, 8 in 2024-09-18T07:51:19.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:19 smithi137 ceph-mon[39175]: osdmap e577: 8 total, 8 up, 8 in 2024-09-18T07:51:19.512 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:51:19.633 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:19 smithi203 ceph-mon[32237]: 3.b scrub starts 2024-09-18T07:51:19.634 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:19 smithi203 ceph-mon[32237]: 3.b scrub ok 2024-09-18T07:51:19.634 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:19 smithi203 ceph-mon[32237]: pgmap v1101: 142 pgs: 1 active+recovering+degraded, 141 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 717 B/s rd, 0 op/s; 8/597 objects misplaced (1.340%); 161 B/s, 3 objects/s recovering 2024-09-18T07:51:19.634 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:19 smithi203 ceph-mon[32237]: 6.1a scrub starts 2024-09-18T07:51:19.634 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:19 smithi203 ceph-mon[32237]: 6.1a scrub ok 2024-09-18T07:51:19.634 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:19 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:51:19.634 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:19 smithi203 ceph-mon[32237]: from='client.36869 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-18T07:51:19.634 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:19 smithi203 ceph-mon[32237]: osdmap e576: 8 total, 8 up, 8 in 2024-09-18T07:51:19.634 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:19 smithi203 ceph-mon[32237]: osdmap e577: 8 total, 8 up, 8 in 2024-09-18T07:51:20.156 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:51:20.156 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:51:20.270 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.2 to 1 (10000) 2024-09-18T07:51:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:20 smithi203 ceph-mon[32237]: 4.1c scrub starts 2024-09-18T07:51:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:20 smithi203 ceph-mon[32237]: 4.1c scrub ok 2024-09-18T07:51:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:20 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/96176250' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:51:20.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:20 smithi203 ceph-mon[32237]: from='client.36875 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:51:20.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:20 smithi137 ceph-mon[29408]: 4.1c scrub starts 2024-09-18T07:51:20.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:20 smithi137 ceph-mon[29408]: 4.1c scrub ok 2024-09-18T07:51:20.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:20 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/96176250' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:51:20.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:20 smithi137 ceph-mon[29408]: from='client.36875 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:51:20.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:20 smithi137 ceph-mon[39175]: 4.1c scrub starts 2024-09-18T07:51:20.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:20 smithi137 ceph-mon[39175]: 4.1c scrub ok 2024-09-18T07:51:20.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:20 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/96176250' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:51:20.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:20 smithi137 ceph-mon[39175]: from='client.36875 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:51:20.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-18T07:51:21.061 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:51:21.128 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-18T07:51:21.378 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:21.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[29408]: 4.1e scrub starts 2024-09-18T07:51:21.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[29408]: 4.1e scrub ok 2024-09-18T07:51:21.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[29408]: 4.f scrub starts 2024-09-18T07:51:21.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[29408]: 4.f scrub ok 2024-09-18T07:51:21.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[29408]: pgmap v1104: 142 pgs: 142 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:51:21.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[29408]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-09-18T07:51:21.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:51:21.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[29408]: from='client.36875 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:51:21.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[29408]: osdmap e578: 8 total, 8 up, 8 in 2024-09-18T07:51:21.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[29408]: 5.0 scrub starts 2024-09-18T07:51:21.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[29408]: 5.0 scrub ok 2024-09-18T07:51:21.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[39175]: 4.1e scrub starts 2024-09-18T07:51:21.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[39175]: 4.1e scrub ok 2024-09-18T07:51:21.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[39175]: 4.f scrub starts 2024-09-18T07:51:21.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[39175]: 4.f scrub ok 2024-09-18T07:51:21.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[39175]: pgmap v1104: 142 pgs: 142 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:51:21.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[39175]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-09-18T07:51:21.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:51:21.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[39175]: from='client.36875 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:51:21.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[39175]: osdmap e578: 8 total, 8 up, 8 in 2024-09-18T07:51:21.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[39175]: 5.0 scrub starts 2024-09-18T07:51:21.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:21 smithi137 ceph-mon[39175]: 5.0 scrub ok 2024-09-18T07:51:21.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:21 smithi203 ceph-mon[32237]: 4.1e scrub starts 2024-09-18T07:51:21.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:21 smithi203 ceph-mon[32237]: 4.1e scrub ok 2024-09-18T07:51:21.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:21 smithi203 ceph-mon[32237]: 4.f scrub starts 2024-09-18T07:51:21.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:21 smithi203 ceph-mon[32237]: 4.f scrub ok 2024-09-18T07:51:21.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:21 smithi203 ceph-mon[32237]: pgmap v1104: 142 pgs: 142 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-18T07:51:21.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:21 smithi203 ceph-mon[32237]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-09-18T07:51:21.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:21 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:51:21.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:21 smithi203 ceph-mon[32237]: from='client.36875 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:51:21.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:21 smithi203 ceph-mon[32237]: osdmap e578: 8 total, 8 up, 8 in 2024-09-18T07:51:21.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:21 smithi203 ceph-mon[32237]: 5.0 scrub starts 2024-09-18T07:51:21.440 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:21 smithi203 ceph-mon[32237]: 5.0 scrub ok 2024-09-18T07:51:21.538 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:51:21.539 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:51:22.206 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-18T07:51:22.270 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-18T07:51:22.455 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.b/config 2024-09-18T07:51:22.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:22 smithi203 ceph-mon[32237]: 5.3 scrub starts 2024-09-18T07:51:22.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:22 smithi203 ceph-mon[32237]: 5.3 scrub ok 2024-09-18T07:51:22.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:22 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1103079038' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:51:22.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:22 smithi203 ceph-mon[32237]: from='client.36881 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:51:22.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:22 smithi137 ceph-mon[29408]: 5.3 scrub starts 2024-09-18T07:51:22.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:22 smithi137 ceph-mon[29408]: 5.3 scrub ok 2024-09-18T07:51:22.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:22 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1103079038' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:51:22.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:22 smithi137 ceph-mon[29408]: from='client.36881 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:51:22.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:22 smithi137 ceph-mon[39175]: 5.3 scrub starts 2024-09-18T07:51:22.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:22 smithi137 ceph-mon[39175]: 5.3 scrub ok 2024-09-18T07:51:22.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:22 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1103079038' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:51:22.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:22 smithi137 ceph-mon[39175]: from='client.36881 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-18T07:51:22.900 INFO:teuthology.orchestra.run.smithi203.stdout:ERROR: (22) Invalid argument 2024-09-18T07:51:22.900 INFO:teuthology.orchestra.run.smithi203.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-18T07:51:23.102 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-18T07:51:23.353 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:23.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[29408]: 4.16 scrub starts 2024-09-18T07:51:23.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[29408]: 4.16 scrub ok 2024-09-18T07:51:23.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[29408]: 3.1a scrub starts 2024-09-18T07:51:23.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[29408]: 3.1a scrub ok 2024-09-18T07:51:23.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[29408]: 4.14 scrub starts 2024-09-18T07:51:23.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[29408]: 4.14 scrub ok 2024-09-18T07:51:23.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[29408]: pgmap v1106: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-09-18T07:51:23.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[29408]: from='client.36881 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:51:23.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[29408]: osdmap e579: 8 total, 8 up, 8 in 2024-09-18T07:51:23.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[39175]: 4.16 scrub starts 2024-09-18T07:51:23.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[39175]: 4.16 scrub ok 2024-09-18T07:51:23.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[39175]: 3.1a scrub starts 2024-09-18T07:51:23.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[39175]: 3.1a scrub ok 2024-09-18T07:51:23.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[39175]: 4.14 scrub starts 2024-09-18T07:51:23.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[39175]: 4.14 scrub ok 2024-09-18T07:51:23.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[39175]: pgmap v1106: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-09-18T07:51:23.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[39175]: from='client.36881 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:51:23.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:23 smithi137 ceph-mon[39175]: osdmap e579: 8 total, 8 up, 8 in 2024-09-18T07:51:23.597 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:23 smithi203 ceph-mon[32237]: 4.16 scrub starts 2024-09-18T07:51:23.597 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:23 smithi203 ceph-mon[32237]: 4.16 scrub ok 2024-09-18T07:51:23.597 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:23 smithi203 ceph-mon[32237]: 3.1a scrub starts 2024-09-18T07:51:23.597 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:23 smithi203 ceph-mon[32237]: 3.1a scrub ok 2024-09-18T07:51:23.598 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:23 smithi203 ceph-mon[32237]: 4.14 scrub starts 2024-09-18T07:51:23.598 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:23 smithi203 ceph-mon[32237]: 4.14 scrub ok 2024-09-18T07:51:23.598 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:23 smithi203 ceph-mon[32237]: pgmap v1106: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-09-18T07:51:23.598 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:23 smithi203 ceph-mon[32237]: from='client.36881 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-18T07:51:23.598 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:23 smithi203 ceph-mon[32237]: osdmap e579: 8 total, 8 up, 8 in 2024-09-18T07:51:24.282 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.3 to 1 (10000) 2024-09-18T07:51:24.608 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:24 smithi137 ceph-mon[29408]: 4.1a scrub starts 2024-09-18T07:51:24.608 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:24 smithi137 ceph-mon[29408]: 4.1a scrub ok 2024-09-18T07:51:24.608 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:24 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:51:24.608 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:24 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:51:24.608 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:24 smithi137 ceph-mon[29408]: 6.13 scrub starts 2024-09-18T07:51:24.608 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:24 smithi137 ceph-mon[29408]: 6.13 scrub ok 2024-09-18T07:51:24.608 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:24 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2838700225' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:51:24.609 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:24 smithi137 ceph-mon[29408]: from='client.36887 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:51:24.609 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:24 smithi137 ceph-mon[39175]: 4.1a scrub starts 2024-09-18T07:51:24.609 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:24 smithi137 ceph-mon[39175]: 4.1a scrub ok 2024-09-18T07:51:24.609 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:24 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:51:24.609 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:24 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:51:24.609 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:24 smithi137 ceph-mon[39175]: 6.13 scrub starts 2024-09-18T07:51:24.609 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:24 smithi137 ceph-mon[39175]: 6.13 scrub ok 2024-09-18T07:51:24.609 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:24 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2838700225' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:51:24.609 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:24 smithi137 ceph-mon[39175]: from='client.36887 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:51:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:24 smithi203 ceph-mon[32237]: 4.1a scrub starts 2024-09-18T07:51:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:24 smithi203 ceph-mon[32237]: 4.1a scrub ok 2024-09-18T07:51:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:24 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:51:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:24 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:51:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:24 smithi203 ceph-mon[32237]: 6.13 scrub starts 2024-09-18T07:51:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:24 smithi203 ceph-mon[32237]: 6.13 scrub ok 2024-09-18T07:51:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:24 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2838700225' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:51:24.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:24 smithi203 ceph-mon[32237]: from='client.36887 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:51:25.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-18T07:51:25.263 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:25.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[29408]: 5.1a scrub starts 2024-09-18T07:51:25.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[29408]: 5.1a scrub ok 2024-09-18T07:51:25.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[29408]: 4.12 scrub starts 2024-09-18T07:51:25.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[29408]: 4.12 scrub ok 2024-09-18T07:51:25.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[29408]: pgmap v1108: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail 2024-09-18T07:51:25.555 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[29408]: from='client.36887 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T07:51:25.555 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[29408]: osdmap e580: 8 total, 8 up, 8 in 2024-09-18T07:51:25.555 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[29408]: 1.a scrub starts 2024-09-18T07:51:25.555 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[29408]: 1.a scrub ok 2024-09-18T07:51:25.555 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[29408]: osdmap e581: 8 total, 8 up, 8 in 2024-09-18T07:51:25.555 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[39175]: 5.1a scrub starts 2024-09-18T07:51:25.555 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[39175]: 5.1a scrub ok 2024-09-18T07:51:25.555 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[39175]: 4.12 scrub starts 2024-09-18T07:51:25.555 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[39175]: 4.12 scrub ok 2024-09-18T07:51:25.555 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[39175]: pgmap v1108: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail 2024-09-18T07:51:25.555 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[39175]: from='client.36887 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T07:51:25.556 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[39175]: osdmap e580: 8 total, 8 up, 8 in 2024-09-18T07:51:25.556 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[39175]: 1.a scrub starts 2024-09-18T07:51:25.556 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[39175]: 1.a scrub ok 2024-09-18T07:51:25.556 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:25 smithi137 ceph-mon[39175]: osdmap e581: 8 total, 8 up, 8 in 2024-09-18T07:51:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:25 smithi203 ceph-mon[32237]: 5.1a scrub starts 2024-09-18T07:51:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:25 smithi203 ceph-mon[32237]: 5.1a scrub ok 2024-09-18T07:51:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:25 smithi203 ceph-mon[32237]: 4.12 scrub starts 2024-09-18T07:51:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:25 smithi203 ceph-mon[32237]: 4.12 scrub ok 2024-09-18T07:51:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:25 smithi203 ceph-mon[32237]: pgmap v1108: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail 2024-09-18T07:51:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:25 smithi203 ceph-mon[32237]: from='client.36887 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-18T07:51:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:25 smithi203 ceph-mon[32237]: osdmap e580: 8 total, 8 up, 8 in 2024-09-18T07:51:25.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:25 smithi203 ceph-mon[32237]: 1.a scrub starts 2024-09-18T07:51:25.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:25 smithi203 ceph-mon[32237]: 1.a scrub ok 2024-09-18T07:51:25.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:25 smithi203 ceph-mon[32237]: osdmap e581: 8 total, 8 up, 8 in 2024-09-18T07:51:26.279 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-18T07:51:26.658 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:26 smithi137 ceph-mon[29408]: 3.7 scrub starts 2024-09-18T07:51:26.658 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:26 smithi137 ceph-mon[29408]: 3.7 scrub ok 2024-09-18T07:51:26.658 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:26 smithi137 ceph-mon[29408]: 4.1d scrub starts 2024-09-18T07:51:26.658 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:26 smithi137 ceph-mon[29408]: 4.1d scrub ok 2024-09-18T07:51:26.658 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:26 smithi137 ceph-mon[29408]: 5.17 scrub starts 2024-09-18T07:51:26.658 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:26 smithi137 ceph-mon[29408]: 5.17 scrub ok 2024-09-18T07:51:26.658 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:26 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2985056847' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:51:26.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:26 smithi137 ceph-mon[39175]: 3.7 scrub starts 2024-09-18T07:51:26.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:26 smithi137 ceph-mon[39175]: 3.7 scrub ok 2024-09-18T07:51:26.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:26 smithi137 ceph-mon[39175]: 4.1d scrub starts 2024-09-18T07:51:26.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:26 smithi137 ceph-mon[39175]: 4.1d scrub ok 2024-09-18T07:51:26.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:26 smithi137 ceph-mon[39175]: 5.17 scrub starts 2024-09-18T07:51:26.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:26 smithi137 ceph-mon[39175]: 5.17 scrub ok 2024-09-18T07:51:26.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:26 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2985056847' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:51:26.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:26 smithi203 ceph-mon[32237]: 3.7 scrub starts 2024-09-18T07:51:26.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:26 smithi203 ceph-mon[32237]: 3.7 scrub ok 2024-09-18T07:51:26.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:26 smithi203 ceph-mon[32237]: 4.1d scrub starts 2024-09-18T07:51:26.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:26 smithi203 ceph-mon[32237]: 4.1d scrub ok 2024-09-18T07:51:26.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:26 smithi203 ceph-mon[32237]: 5.17 scrub starts 2024-09-18T07:51:26.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:26 smithi203 ceph-mon[32237]: 5.17 scrub ok 2024-09-18T07:51:26.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:26 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2985056847' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-18T07:51:27.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-18T07:51:27.363 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:27.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:51:27 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:51:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:51:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:27 smithi137 ceph-mon[29408]: pgmap v1111: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail 2024-09-18T07:51:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:27 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2985056847' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-18T07:51:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:27 smithi137 ceph-mon[29408]: osdmap e582: 8 total, 8 up, 8 in 2024-09-18T07:51:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:27 smithi137 ceph-mon[29408]: 3.9 scrub starts 2024-09-18T07:51:27.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:27 smithi137 ceph-mon[29408]: 3.9 scrub ok 2024-09-18T07:51:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:27 smithi137 ceph-mon[39175]: pgmap v1111: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail 2024-09-18T07:51:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:27 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2985056847' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-18T07:51:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:27 smithi137 ceph-mon[39175]: osdmap e582: 8 total, 8 up, 8 in 2024-09-18T07:51:27.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:27 smithi137 ceph-mon[39175]: 3.9 scrub starts 2024-09-18T07:51:27.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:27 smithi137 ceph-mon[39175]: 3.9 scrub ok 2024-09-18T07:51:27.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:27 smithi203 ceph-mon[32237]: pgmap v1111: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail 2024-09-18T07:51:27.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:27 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2985056847' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-18T07:51:27.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:27 smithi203 ceph-mon[32237]: osdmap e582: 8 total, 8 up, 8 in 2024-09-18T07:51:27.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:27 smithi203 ceph-mon[32237]: 3.9 scrub starts 2024-09-18T07:51:27.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:27 smithi203 ceph-mon[32237]: 3.9 scrub ok 2024-09-18T07:51:28.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:51:28 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:51:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:28 smithi203 ceph-mon[32237]: 3.1f scrub starts 2024-09-18T07:51:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:28 smithi203 ceph-mon[32237]: 3.1f scrub ok 2024-09-18T07:51:28.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:28 smithi203 ceph-mon[32237]: 5.12 scrub starts 2024-09-18T07:51:28.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:28 smithi203 ceph-mon[32237]: 5.12 scrub ok 2024-09-18T07:51:28.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:28 smithi203 ceph-mon[32237]: osdmap e583: 8 total, 8 up, 8 in 2024-09-18T07:51:28.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:28 smithi203 ceph-mon[32237]: 6.11 scrub starts 2024-09-18T07:51:28.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:28 smithi203 ceph-mon[32237]: 6.11 scrub ok 2024-09-18T07:51:28.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:28 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:51:28.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:28 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:51:28.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:28 smithi203 ceph-mon[32237]: 5.1c scrub starts 2024-09-18T07:51:28.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:28 smithi203 ceph-mon[32237]: 5.1c scrub ok 2024-09-18T07:51:28.462 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:28 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2905891367' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:51:28.572 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.6 to 1 (10000) 2024-09-18T07:51:28.572 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[29408]: 3.1f scrub starts 2024-09-18T07:51:28.572 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[29408]: 3.1f scrub ok 2024-09-18T07:51:28.572 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[29408]: 5.12 scrub starts 2024-09-18T07:51:28.573 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[29408]: 5.12 scrub ok 2024-09-18T07:51:28.573 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[29408]: osdmap e583: 8 total, 8 up, 8 in 2024-09-18T07:51:28.573 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[29408]: 6.11 scrub starts 2024-09-18T07:51:28.573 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[29408]: 6.11 scrub ok 2024-09-18T07:51:28.573 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:51:28.573 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:51:28.573 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[29408]: 5.1c scrub starts 2024-09-18T07:51:28.573 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[29408]: 5.1c scrub ok 2024-09-18T07:51:28.573 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2905891367' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:51:28.573 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[39175]: 3.1f scrub starts 2024-09-18T07:51:28.573 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[39175]: 3.1f scrub ok 2024-09-18T07:51:28.573 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[39175]: 5.12 scrub starts 2024-09-18T07:51:28.574 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[39175]: 5.12 scrub ok 2024-09-18T07:51:28.574 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[39175]: osdmap e583: 8 total, 8 up, 8 in 2024-09-18T07:51:28.574 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[39175]: 6.11 scrub starts 2024-09-18T07:51:28.574 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[39175]: 6.11 scrub ok 2024-09-18T07:51:28.574 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:51:28.574 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:51:28.574 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[39175]: 5.1c scrub starts 2024-09-18T07:51:28.574 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[39175]: 5.1c scrub ok 2024-09-18T07:51:28.574 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:28 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2905891367' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:51:29.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-18T07:51:29.574 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:29.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[29408]: 3.e scrub starts 2024-09-18T07:51:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[29408]: 3.e scrub ok 2024-09-18T07:51:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[29408]: pgmap v1114: 142 pgs: 9 peering, 133 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-18T07:51:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:51:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[29408]: 4.11 scrub starts 2024-09-18T07:51:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[29408]: 4.11 scrub ok 2024-09-18T07:51:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[29408]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-18T07:51:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2905891367' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:51:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[29408]: osdmap e584: 8 total, 8 up, 8 in 2024-09-18T07:51:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[29408]: 5.18 scrub starts 2024-09-18T07:51:29.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[29408]: 5.18 scrub ok 2024-09-18T07:51:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[39175]: 3.e scrub starts 2024-09-18T07:51:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[39175]: 3.e scrub ok 2024-09-18T07:51:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[39175]: pgmap v1114: 142 pgs: 9 peering, 133 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-18T07:51:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:51:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[39175]: 4.11 scrub starts 2024-09-18T07:51:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[39175]: 4.11 scrub ok 2024-09-18T07:51:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[39175]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-18T07:51:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2905891367' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:51:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[39175]: osdmap e584: 8 total, 8 up, 8 in 2024-09-18T07:51:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[39175]: 5.18 scrub starts 2024-09-18T07:51:29.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:29 smithi137 ceph-mon[39175]: 5.18 scrub ok 2024-09-18T07:51:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:29 smithi203 ceph-mon[32237]: 3.e scrub starts 2024-09-18T07:51:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:29 smithi203 ceph-mon[32237]: 3.e scrub ok 2024-09-18T07:51:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:29 smithi203 ceph-mon[32237]: pgmap v1114: 142 pgs: 9 peering, 133 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-18T07:51:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:29 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:51:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:29 smithi203 ceph-mon[32237]: 4.11 scrub starts 2024-09-18T07:51:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:29 smithi203 ceph-mon[32237]: 4.11 scrub ok 2024-09-18T07:51:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:29 smithi203 ceph-mon[32237]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-18T07:51:29.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:29 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2905891367' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:51:29.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:29 smithi203 ceph-mon[32237]: osdmap e584: 8 total, 8 up, 8 in 2024-09-18T07:51:29.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:29 smithi203 ceph-mon[32237]: 5.18 scrub starts 2024-09-18T07:51:29.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:29 smithi203 ceph-mon[32237]: 5.18 scrub ok 2024-09-18T07:51:30.633 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:30 smithi203 ceph-mon[32237]: 3.6 scrub starts 2024-09-18T07:51:30.633 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:30 smithi203 ceph-mon[32237]: 3.6 scrub ok 2024-09-18T07:51:30.633 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:30 smithi203 ceph-mon[32237]: 4.2 scrub starts 2024-09-18T07:51:30.633 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:30 smithi203 ceph-mon[32237]: 4.2 scrub ok 2024-09-18T07:51:30.633 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:30 smithi203 ceph-mon[32237]: osdmap e585: 8 total, 8 up, 8 in 2024-09-18T07:51:30.634 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:30 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/85047087' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:51:30.634 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:30 smithi203 ceph-mon[32237]: from='client.36905 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:51:30.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:30 smithi137 ceph-mon[29408]: 3.6 scrub starts 2024-09-18T07:51:30.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:30 smithi137 ceph-mon[29408]: 3.6 scrub ok 2024-09-18T07:51:30.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:30 smithi137 ceph-mon[29408]: 4.2 scrub starts 2024-09-18T07:51:30.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:30 smithi137 ceph-mon[29408]: 4.2 scrub ok 2024-09-18T07:51:30.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:30 smithi137 ceph-mon[29408]: osdmap e585: 8 total, 8 up, 8 in 2024-09-18T07:51:30.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:30 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/85047087' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:51:30.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:30 smithi137 ceph-mon[29408]: from='client.36905 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:51:30.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:30 smithi137 ceph-mon[39175]: 3.6 scrub starts 2024-09-18T07:51:30.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:30 smithi137 ceph-mon[39175]: 3.6 scrub ok 2024-09-18T07:51:30.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:30 smithi137 ceph-mon[39175]: 4.2 scrub starts 2024-09-18T07:51:30.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:30 smithi137 ceph-mon[39175]: 4.2 scrub ok 2024-09-18T07:51:30.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:30 smithi137 ceph-mon[39175]: osdmap e585: 8 total, 8 up, 8 in 2024-09-18T07:51:30.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:30 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/85047087' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:51:30.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:30 smithi137 ceph-mon[39175]: from='client.36905 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-18T07:51:31.205 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-18T07:51:31.594 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:31 smithi203 ceph-mon[32237]: 1.0 scrub starts 2024-09-18T07:51:31.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:31 smithi203 ceph-mon[32237]: 1.0 scrub ok 2024-09-18T07:51:31.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:31 smithi203 ceph-mon[32237]: pgmap v1116: 142 pgs: 9 peering, 133 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 649 B/s rd, 0 op/s; 68 B/s, 0 objects/s recovering 2024-09-18T07:51:31.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:31 smithi203 ceph-mon[32237]: 6.f scrub starts 2024-09-18T07:51:31.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:31 smithi203 ceph-mon[32237]: 6.f scrub ok 2024-09-18T07:51:31.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:31 smithi203 ceph-mon[32237]: 3.3 scrub starts 2024-09-18T07:51:31.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:31 smithi203 ceph-mon[32237]: 3.3 scrub ok 2024-09-18T07:51:31.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:31 smithi203 ceph-mon[32237]: from='client.36905 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:51:31.595 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:31 smithi203 ceph-mon[32237]: osdmap e586: 8 total, 8 up, 8 in 2024-09-18T07:51:31.602 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[29408]: 1.0 scrub starts 2024-09-18T07:51:31.602 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[29408]: 1.0 scrub ok 2024-09-18T07:51:31.602 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[29408]: pgmap v1116: 142 pgs: 9 peering, 133 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 649 B/s rd, 0 op/s; 68 B/s, 0 objects/s recovering 2024-09-18T07:51:31.602 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[29408]: 6.f scrub starts 2024-09-18T07:51:31.602 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[29408]: 6.f scrub ok 2024-09-18T07:51:31.602 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[29408]: 3.3 scrub starts 2024-09-18T07:51:31.602 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[29408]: 3.3 scrub ok 2024-09-18T07:51:31.602 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[29408]: from='client.36905 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:51:31.602 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[29408]: osdmap e586: 8 total, 8 up, 8 in 2024-09-18T07:51:31.603 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[39175]: 1.0 scrub starts 2024-09-18T07:51:31.603 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[39175]: 1.0 scrub ok 2024-09-18T07:51:31.603 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[39175]: pgmap v1116: 142 pgs: 9 peering, 133 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 649 B/s rd, 0 op/s; 68 B/s, 0 objects/s recovering 2024-09-18T07:51:31.603 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[39175]: 6.f scrub starts 2024-09-18T07:51:31.603 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[39175]: 6.f scrub ok 2024-09-18T07:51:31.603 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[39175]: 3.3 scrub starts 2024-09-18T07:51:31.603 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[39175]: 3.3 scrub ok 2024-09-18T07:51:31.603 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[39175]: from='client.36905 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-18T07:51:31.603 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:31 smithi137 ceph-mon[39175]: osdmap e586: 8 total, 8 up, 8 in 2024-09-18T07:51:32.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-18T07:51:32.277 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:32.557 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:32 smithi137 ceph-mon[29408]: 1.2 scrub starts 2024-09-18T07:51:32.557 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:32 smithi137 ceph-mon[29408]: 1.2 scrub ok 2024-09-18T07:51:32.557 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:32 smithi137 ceph-mon[29408]: 6.1b scrub starts 2024-09-18T07:51:32.557 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:32 smithi137 ceph-mon[29408]: 6.1b scrub ok 2024-09-18T07:51:32.558 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:32 smithi137 ceph-mon[39175]: 1.2 scrub starts 2024-09-18T07:51:32.558 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:32 smithi137 ceph-mon[39175]: 1.2 scrub ok 2024-09-18T07:51:32.558 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:32 smithi137 ceph-mon[39175]: 6.1b scrub starts 2024-09-18T07:51:32.558 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:32 smithi137 ceph-mon[39175]: 6.1b scrub ok 2024-09-18T07:51:32.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:32 smithi203 ceph-mon[32237]: 1.2 scrub starts 2024-09-18T07:51:32.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:32 smithi203 ceph-mon[32237]: 1.2 scrub ok 2024-09-18T07:51:32.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:32 smithi203 ceph-mon[32237]: 6.1b scrub starts 2024-09-18T07:51:32.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:32 smithi203 ceph-mon[32237]: 6.1b scrub ok 2024-09-18T07:51:33.312 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.7 to 1 (10000) 2024-09-18T07:51:33.622 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:33 smithi137 ceph-mon[29408]: pgmap v1119: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 93 B/s, 1 objects/s recovering 2024-09-18T07:51:33.622 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:33 smithi137 ceph-mon[29408]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-18T07:51:33.622 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:33 smithi137 ceph-mon[29408]: Cluster is now healthy 2024-09-18T07:51:33.622 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:33 smithi137 ceph-mon[29408]: 3.8 scrub starts 2024-09-18T07:51:33.622 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:33 smithi137 ceph-mon[29408]: 3.8 scrub ok 2024-09-18T07:51:33.622 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:33 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1559845401' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:51:33.622 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:33 smithi137 ceph-mon[29408]: from='client.36911 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:51:33.623 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:33 smithi137 ceph-mon[39175]: pgmap v1119: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 93 B/s, 1 objects/s recovering 2024-09-18T07:51:33.623 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:33 smithi137 ceph-mon[39175]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-18T07:51:33.623 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:33 smithi137 ceph-mon[39175]: Cluster is now healthy 2024-09-18T07:51:33.623 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:33 smithi137 ceph-mon[39175]: 3.8 scrub starts 2024-09-18T07:51:33.623 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:33 smithi137 ceph-mon[39175]: 3.8 scrub ok 2024-09-18T07:51:33.623 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:33 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1559845401' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:51:33.623 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:33 smithi137 ceph-mon[39175]: from='client.36911 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:51:33.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:33 smithi203 ceph-mon[32237]: pgmap v1119: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 93 B/s, 1 objects/s recovering 2024-09-18T07:51:33.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:33 smithi203 ceph-mon[32237]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-18T07:51:33.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:33 smithi203 ceph-mon[32237]: Cluster is now healthy 2024-09-18T07:51:33.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:33 smithi203 ceph-mon[32237]: 3.8 scrub starts 2024-09-18T07:51:33.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:33 smithi203 ceph-mon[32237]: 3.8 scrub ok 2024-09-18T07:51:33.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:33 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1559845401' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:51:33.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:33 smithi203 ceph-mon[32237]: from='client.36911 ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:51:34.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-18T07:51:34.496 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:34.618 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:34 smithi137 ceph-mon[29408]: from='client.36911 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T07:51:34.618 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:34 smithi137 ceph-mon[29408]: osdmap e587: 8 total, 8 up, 8 in 2024-09-18T07:51:34.618 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:34 smithi137 ceph-mon[29408]: 6.6 scrub starts 2024-09-18T07:51:34.618 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:34 smithi137 ceph-mon[29408]: 6.6 scrub ok 2024-09-18T07:51:34.618 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:34 smithi137 ceph-mon[39175]: from='client.36911 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T07:51:34.618 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:34 smithi137 ceph-mon[39175]: osdmap e587: 8 total, 8 up, 8 in 2024-09-18T07:51:34.619 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:34 smithi137 ceph-mon[39175]: 6.6 scrub starts 2024-09-18T07:51:34.619 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:34 smithi137 ceph-mon[39175]: 6.6 scrub ok 2024-09-18T07:51:34.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:34 smithi203 ceph-mon[32237]: from='client.36911 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-18T07:51:34.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:34 smithi203 ceph-mon[32237]: osdmap e587: 8 total, 8 up, 8 in 2024-09-18T07:51:34.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:34 smithi203 ceph-mon[32237]: 6.6 scrub starts 2024-09-18T07:51:34.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:34 smithi203 ceph-mon[32237]: 6.6 scrub ok 2024-09-18T07:51:35.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[39175]: 4.b scrub starts 2024-09-18T07:51:35.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[39175]: 4.b scrub ok 2024-09-18T07:51:35.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[39175]: pgmap v1121: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 956 B/s rd, 0 op/s 2024-09-18T07:51:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:51:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[39175]: 4.1f scrub starts 2024-09-18T07:51:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[39175]: 4.1f scrub ok 2024-09-18T07:51:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:51:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:51:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:51:35.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3177399193' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:51:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[29408]: 4.b scrub starts 2024-09-18T07:51:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[29408]: 4.b scrub ok 2024-09-18T07:51:35.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[29408]: pgmap v1121: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 956 B/s rd, 0 op/s 2024-09-18T07:51:35.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:51:35.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[29408]: 4.1f scrub starts 2024-09-18T07:51:35.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[29408]: 4.1f scrub ok 2024-09-18T07:51:35.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:51:35.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:51:35.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:51:35.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:35 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3177399193' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:51:35.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:35 smithi203 ceph-mon[32237]: 4.b scrub starts 2024-09-18T07:51:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:35 smithi203 ceph-mon[32237]: 4.b scrub ok 2024-09-18T07:51:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:35 smithi203 ceph-mon[32237]: pgmap v1121: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 956 B/s rd, 0 op/s 2024-09-18T07:51:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:35 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:51:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:35 smithi203 ceph-mon[32237]: 4.1f scrub starts 2024-09-18T07:51:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:35 smithi203 ceph-mon[32237]: 4.1f scrub ok 2024-09-18T07:51:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:35 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:51:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:35 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:51:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:35 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:51:35.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:35 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3177399193' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-18T07:51:36.197 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-18T07:51:36.575 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:36 smithi137 ceph-mon[29408]: 6.9 scrub starts 2024-09-18T07:51:36.575 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:36 smithi137 ceph-mon[29408]: 6.9 scrub ok 2024-09-18T07:51:36.575 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:36 smithi137 ceph-mon[29408]: 4.4 scrub starts 2024-09-18T07:51:36.576 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:36 smithi137 ceph-mon[29408]: 4.4 scrub ok 2024-09-18T07:51:36.576 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:36 smithi137 ceph-mon[29408]: 4.13 scrub starts 2024-09-18T07:51:36.576 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:36 smithi137 ceph-mon[29408]: 4.13 scrub ok 2024-09-18T07:51:36.576 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:36 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3177399193' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-18T07:51:36.576 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:36 smithi137 ceph-mon[29408]: osdmap e588: 8 total, 8 up, 8 in 2024-09-18T07:51:36.576 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:36 smithi137 ceph-mon[39175]: 6.9 scrub starts 2024-09-18T07:51:36.576 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:36 smithi137 ceph-mon[39175]: 6.9 scrub ok 2024-09-18T07:51:36.576 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:36 smithi137 ceph-mon[39175]: 4.4 scrub starts 2024-09-18T07:51:36.576 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:36 smithi137 ceph-mon[39175]: 4.4 scrub ok 2024-09-18T07:51:36.576 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:36 smithi137 ceph-mon[39175]: 4.13 scrub starts 2024-09-18T07:51:36.577 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:36 smithi137 ceph-mon[39175]: 4.13 scrub ok 2024-09-18T07:51:36.577 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:36 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3177399193' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-18T07:51:36.577 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:36 smithi137 ceph-mon[39175]: osdmap e588: 8 total, 8 up, 8 in 2024-09-18T07:51:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:36 smithi203 ceph-mon[32237]: 6.9 scrub starts 2024-09-18T07:51:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:36 smithi203 ceph-mon[32237]: 6.9 scrub ok 2024-09-18T07:51:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:36 smithi203 ceph-mon[32237]: 4.4 scrub starts 2024-09-18T07:51:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:36 smithi203 ceph-mon[32237]: 4.4 scrub ok 2024-09-18T07:51:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:36 smithi203 ceph-mon[32237]: 4.13 scrub starts 2024-09-18T07:51:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:36 smithi203 ceph-mon[32237]: 4.13 scrub ok 2024-09-18T07:51:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:36 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3177399193' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-18T07:51:36.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:36 smithi203 ceph-mon[32237]: osdmap e588: 8 total, 8 up, 8 in 2024-09-18T07:51:37.021 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-18T07:51:37.275 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:37.325 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:51:37 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:51:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:51:37.579 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:37 smithi137 ceph-mon[29408]: pgmap v1122: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:51:37.580 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:37 smithi137 ceph-mon[29408]: 6.17 scrub starts 2024-09-18T07:51:37.580 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:37 smithi137 ceph-mon[29408]: 6.17 scrub ok 2024-09-18T07:51:37.580 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:37 smithi137 ceph-mon[39175]: pgmap v1122: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:51:37.580 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:37 smithi137 ceph-mon[39175]: 6.17 scrub starts 2024-09-18T07:51:37.580 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:37 smithi137 ceph-mon[39175]: 6.17 scrub ok 2024-09-18T07:51:37.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:37 smithi203 ceph-mon[32237]: pgmap v1122: 142 pgs: 142 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:51:37.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:37 smithi203 ceph-mon[32237]: 6.17 scrub starts 2024-09-18T07:51:37.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:37 smithi203 ceph-mon[32237]: 6.17 scrub ok 2024-09-18T07:51:38.326 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.4 to 1 (10000) 2024-09-18T07:51:38.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:38 smithi203 ceph-mon[32237]: 3.d scrub starts 2024-09-18T07:51:38.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:38 smithi203 ceph-mon[32237]: 3.d scrub ok 2024-09-18T07:51:38.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:38 smithi203 ceph-mon[32237]: 4.a scrub starts 2024-09-18T07:51:38.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:38 smithi203 ceph-mon[32237]: 4.a scrub ok 2024-09-18T07:51:38.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:38 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4097764338' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:51:38.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:51:38 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:51:38.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:38 smithi137 ceph-mon[29408]: 3.d scrub starts 2024-09-18T07:51:38.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:38 smithi137 ceph-mon[29408]: 3.d scrub ok 2024-09-18T07:51:38.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:38 smithi137 ceph-mon[29408]: 4.a scrub starts 2024-09-18T07:51:38.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:38 smithi137 ceph-mon[29408]: 4.a scrub ok 2024-09-18T07:51:38.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:38 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4097764338' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:51:38.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:38 smithi137 ceph-mon[39175]: 3.d scrub starts 2024-09-18T07:51:38.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:38 smithi137 ceph-mon[39175]: 3.d scrub ok 2024-09-18T07:51:38.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:38 smithi137 ceph-mon[39175]: 4.a scrub starts 2024-09-18T07:51:38.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:38 smithi137 ceph-mon[39175]: 4.a scrub ok 2024-09-18T07:51:38.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:38 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4097764338' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:51:39.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-18T07:51:39.375 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:39.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[29408]: 6.d scrub starts 2024-09-18T07:51:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[29408]: 6.d scrub ok 2024-09-18T07:51:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[29408]: 4.6 scrub starts 2024-09-18T07:51:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[29408]: 4.6 scrub ok 2024-09-18T07:51:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[29408]: pgmap v1124: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-18T07:51:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:51:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4097764338' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:51:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[29408]: osdmap e589: 8 total, 8 up, 8 in 2024-09-18T07:51:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[29408]: 5.14 scrub starts 2024-09-18T07:51:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[29408]: 5.14 scrub ok 2024-09-18T07:51:39.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[29408]: 5.16 scrub starts 2024-09-18T07:51:39.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[39175]: 6.d scrub starts 2024-09-18T07:51:39.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[39175]: 6.d scrub ok 2024-09-18T07:51:39.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[39175]: 4.6 scrub starts 2024-09-18T07:51:39.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[39175]: 4.6 scrub ok 2024-09-18T07:51:39.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[39175]: pgmap v1124: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-18T07:51:39.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:51:39.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4097764338' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:51:39.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[39175]: osdmap e589: 8 total, 8 up, 8 in 2024-09-18T07:51:39.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[39175]: 5.14 scrub starts 2024-09-18T07:51:39.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[39175]: 5.14 scrub ok 2024-09-18T07:51:39.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:39 smithi137 ceph-mon[39175]: 5.16 scrub starts 2024-09-18T07:51:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:39 smithi203 ceph-mon[32237]: 6.d scrub starts 2024-09-18T07:51:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:39 smithi203 ceph-mon[32237]: 6.d scrub ok 2024-09-18T07:51:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:39 smithi203 ceph-mon[32237]: 4.6 scrub starts 2024-09-18T07:51:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:39 smithi203 ceph-mon[32237]: 4.6 scrub ok 2024-09-18T07:51:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:39 smithi203 ceph-mon[32237]: pgmap v1124: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-18T07:51:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:39 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:51:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:39 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4097764338' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:51:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:39 smithi203 ceph-mon[32237]: osdmap e589: 8 total, 8 up, 8 in 2024-09-18T07:51:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:39 smithi203 ceph-mon[32237]: 5.14 scrub starts 2024-09-18T07:51:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:39 smithi203 ceph-mon[32237]: 5.14 scrub ok 2024-09-18T07:51:39.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:39 smithi203 ceph-mon[32237]: 5.16 scrub starts 2024-09-18T07:51:40.339 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-18T07:51:40.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:40 smithi137 ceph-mon[39175]: 4.3 scrub starts 2024-09-18T07:51:40.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:40 smithi137 ceph-mon[39175]: 4.3 scrub ok 2024-09-18T07:51:40.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:40 smithi137 ceph-mon[39175]: 5.16 scrub ok 2024-09-18T07:51:40.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:40 smithi137 ceph-mon[39175]: 3.15 scrub starts 2024-09-18T07:51:40.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:40 smithi137 ceph-mon[39175]: 3.15 scrub ok 2024-09-18T07:51:40.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:40 smithi137 ceph-mon[39175]: 1.4 scrub starts 2024-09-18T07:51:40.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:40 smithi137 ceph-mon[39175]: 1.4 scrub ok 2024-09-18T07:51:40.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:40 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1383573620' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:51:40.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:40 smithi137 ceph-mon[29408]: 4.3 scrub starts 2024-09-18T07:51:40.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:40 smithi137 ceph-mon[29408]: 4.3 scrub ok 2024-09-18T07:51:40.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:40 smithi137 ceph-mon[29408]: 5.16 scrub ok 2024-09-18T07:51:40.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:40 smithi137 ceph-mon[29408]: 3.15 scrub starts 2024-09-18T07:51:40.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:40 smithi137 ceph-mon[29408]: 3.15 scrub ok 2024-09-18T07:51:40.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:40 smithi137 ceph-mon[29408]: 1.4 scrub starts 2024-09-18T07:51:40.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:40 smithi137 ceph-mon[29408]: 1.4 scrub ok 2024-09-18T07:51:40.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:40 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1383573620' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:51:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:40 smithi203 ceph-mon[32237]: 4.3 scrub starts 2024-09-18T07:51:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:40 smithi203 ceph-mon[32237]: 4.3 scrub ok 2024-09-18T07:51:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:40 smithi203 ceph-mon[32237]: 5.16 scrub ok 2024-09-18T07:51:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:40 smithi203 ceph-mon[32237]: 3.15 scrub starts 2024-09-18T07:51:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:40 smithi203 ceph-mon[32237]: 3.15 scrub ok 2024-09-18T07:51:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:40 smithi203 ceph-mon[32237]: 1.4 scrub starts 2024-09-18T07:51:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:40 smithi203 ceph-mon[32237]: 1.4 scrub ok 2024-09-18T07:51:40.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:40 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1383573620' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-18T07:51:41.134 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-18T07:51:41.388 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:41.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:41 smithi137 ceph-mon[29408]: pgmap v1126: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 773 B/s rd, 0 op/s 2024-09-18T07:51:41.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:41 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1383573620' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:51:41.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:41 smithi137 ceph-mon[29408]: osdmap e590: 8 total, 8 up, 8 in 2024-09-18T07:51:41.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:41 smithi137 ceph-mon[29408]: 5.4 scrub starts 2024-09-18T07:51:41.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:41 smithi137 ceph-mon[29408]: 5.4 scrub ok 2024-09-18T07:51:41.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:41 smithi137 ceph-mon[39175]: pgmap v1126: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 773 B/s rd, 0 op/s 2024-09-18T07:51:41.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:41 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1383573620' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:51:41.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:41 smithi137 ceph-mon[39175]: osdmap e590: 8 total, 8 up, 8 in 2024-09-18T07:51:41.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:41 smithi137 ceph-mon[39175]: 5.4 scrub starts 2024-09-18T07:51:41.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:41 smithi137 ceph-mon[39175]: 5.4 scrub ok 2024-09-18T07:51:41.460 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:41 smithi203 ceph-mon[32237]: pgmap v1126: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 773 B/s rd, 0 op/s 2024-09-18T07:51:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:41 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1383573620' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-18T07:51:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:41 smithi203 ceph-mon[32237]: osdmap e590: 8 total, 8 up, 8 in 2024-09-18T07:51:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:41 smithi203 ceph-mon[32237]: 5.4 scrub starts 2024-09-18T07:51:41.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:41 smithi203 ceph-mon[32237]: 5.4 scrub ok 2024-09-18T07:51:42.353 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.5 to 1 (10000) 2024-09-18T07:51:42.469 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:42 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2586110789' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:51:42.469 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:42 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2586110789' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:51:42.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:42 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2586110789' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:51:43.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-18T07:51:43.419 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:43 smithi137 ceph-mon[29408]: 3.1 scrub starts 2024-09-18T07:51:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:43 smithi137 ceph-mon[29408]: 3.1 scrub ok 2024-09-18T07:51:43.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:43 smithi137 ceph-mon[29408]: pgmap v1128: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:51:43.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:43 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/2586110789' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:51:43.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:43 smithi137 ceph-mon[29408]: osdmap e591: 8 total, 8 up, 8 in 2024-09-18T07:51:43.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:43 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:51:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:43 smithi137 ceph-mon[39175]: 3.1 scrub starts 2024-09-18T07:51:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:43 smithi137 ceph-mon[39175]: 3.1 scrub ok 2024-09-18T07:51:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:43 smithi137 ceph-mon[39175]: pgmap v1128: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:51:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:43 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/2586110789' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:51:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:43 smithi137 ceph-mon[39175]: osdmap e591: 8 total, 8 up, 8 in 2024-09-18T07:51:43.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:43 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:51:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:43 smithi203 ceph-mon[32237]: 3.1 scrub starts 2024-09-18T07:51:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:43 smithi203 ceph-mon[32237]: 3.1 scrub ok 2024-09-18T07:51:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:43 smithi203 ceph-mon[32237]: pgmap v1128: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:51:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:43 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/2586110789' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:51:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:43 smithi203 ceph-mon[32237]: osdmap e591: 8 total, 8 up, 8 in 2024-09-18T07:51:43.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:43 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:51:44.359 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-18T07:51:44.654 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:44 smithi137 ceph-mon[29408]: 6.19 scrub starts 2024-09-18T07:51:44.654 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:44 smithi137 ceph-mon[29408]: 6.19 scrub ok 2024-09-18T07:51:44.654 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:44 smithi137 ceph-mon[29408]: 4.1b scrub starts 2024-09-18T07:51:44.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:44 smithi137 ceph-mon[29408]: 4.1b scrub ok 2024-09-18T07:51:44.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:44 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1325080197' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:51:44.655 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:44 smithi137 ceph-mon[29408]: from='client.36941 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:51:44.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:44 smithi137 ceph-mon[39175]: 6.19 scrub starts 2024-09-18T07:51:44.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:44 smithi137 ceph-mon[39175]: 6.19 scrub ok 2024-09-18T07:51:44.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:44 smithi137 ceph-mon[39175]: 4.1b scrub starts 2024-09-18T07:51:44.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:44 smithi137 ceph-mon[39175]: 4.1b scrub ok 2024-09-18T07:51:44.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:44 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1325080197' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:51:44.655 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:44 smithi137 ceph-mon[39175]: from='client.36941 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:51:44.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:44 smithi203 ceph-mon[32237]: 6.19 scrub starts 2024-09-18T07:51:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:44 smithi203 ceph-mon[32237]: 6.19 scrub ok 2024-09-18T07:51:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:44 smithi203 ceph-mon[32237]: 4.1b scrub starts 2024-09-18T07:51:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:44 smithi203 ceph-mon[32237]: 4.1b scrub ok 2024-09-18T07:51:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:44 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1325080197' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:51:44.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:44 smithi203 ceph-mon[32237]: from='client.36941 ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-18T07:51:45.256 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-18T07:51:45.256 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-18T07:51:45.256 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-18T07:51:45.256 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-18T07:51:45.257 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-18T07:51:45.257 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-18T07:51:45.513 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:45.634 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:45 smithi137 ceph-mon[29408]: 3.17 scrub starts 2024-09-18T07:51:45.634 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:45 smithi137 ceph-mon[29408]: 3.17 scrub ok 2024-09-18T07:51:45.635 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:45 smithi137 ceph-mon[29408]: pgmap v1130: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:51:45.635 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:45 smithi137 ceph-mon[29408]: from='client.36941 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:51:45.635 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:45 smithi137 ceph-mon[29408]: osdmap e592: 8 total, 8 up, 8 in 2024-09-18T07:51:45.635 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:45 smithi137 ceph-mon[29408]: osdmap e593: 8 total, 8 up, 8 in 2024-09-18T07:51:45.635 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:45 smithi137 ceph-mon[39175]: 3.17 scrub starts 2024-09-18T07:51:45.635 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:45 smithi137 ceph-mon[39175]: 3.17 scrub ok 2024-09-18T07:51:45.635 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:45 smithi137 ceph-mon[39175]: pgmap v1130: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:51:45.635 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:45 smithi137 ceph-mon[39175]: from='client.36941 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:51:45.635 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:45 smithi137 ceph-mon[39175]: osdmap e592: 8 total, 8 up, 8 in 2024-09-18T07:51:45.635 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:45 smithi137 ceph-mon[39175]: osdmap e593: 8 total, 8 up, 8 in 2024-09-18T07:51:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:45 smithi203 ceph-mon[32237]: 3.17 scrub starts 2024-09-18T07:51:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:45 smithi203 ceph-mon[32237]: 3.17 scrub ok 2024-09-18T07:51:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:45 smithi203 ceph-mon[32237]: pgmap v1130: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:51:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:45 smithi203 ceph-mon[32237]: from='client.36941 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-18T07:51:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:45 smithi203 ceph-mon[32237]: osdmap e592: 8 total, 8 up, 8 in 2024-09-18T07:51:45.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:45 smithi203 ceph-mon[32237]: osdmap e593: 8 total, 8 up, 8 in 2024-09-18T07:51:46.229 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:51:46.230 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":593,"fsid":"f7ef099c-758e-11ef-bceb-c7b262605968","created":"2024-09-18T07:25:19.662367+0000","modified":"2024-09-18T07:51:45.194656+0000","last_up_change":"2024-09-18T07:47:10.160431+0000","last_in_change":"2024-09-18T07:51:08.194222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T07:28:20.899655+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"531","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"272","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-18T07:30:53.715373+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-18T07:30:54.672127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-18T07:30:56.351343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-18T07:30:58.322401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-18T07:31:00.390499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268cd580-c931-4992-bb93-d6f63c2bfe88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":580,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6801","nonce":2811444477}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6802","nonce":2811444477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6804","nonce":2811444477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6803","nonce":2811444477}]},"public_addr":"172.21.15.137:6801/2811444477","cluster_addr":"172.21.15.137:6802/2811444477","heartbeat_back_addr":"172.21.15.137:6804/2811444477","heartbeat_front_addr":"172.21.15.137:6803/2811444477","state":["exists","up"]},{"osd":1,"uuid":"b06907bc-b499-46c0-8667-c5bdc0d85575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":580,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6805","nonce":874102487}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6806","nonce":874102487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6808","nonce":874102487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6807","nonce":874102487}]},"public_addr":"172.21.15.137:6805/874102487","cluster_addr":"172.21.15.137:6806/874102487","heartbeat_back_addr":"172.21.15.137:6808/874102487","heartbeat_front_addr":"172.21.15.137:6807/874102487","state":["exists","up"]},{"osd":2,"uuid":"480ad6c5-0e0b-4405-bbe2-381682d00bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":570,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6809","nonce":1754684686}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6810","nonce":1754684686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6812","nonce":1754684686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6811","nonce":1754684686}]},"public_addr":"172.21.15.137:6809/1754684686","cluster_addr":"172.21.15.137:6810/1754684686","heartbeat_back_addr":"172.21.15.137:6812/1754684686","heartbeat_front_addr":"172.21.15.137:6811/1754684686","state":["exists","up"]},{"osd":3,"uuid":"178be3e0-f07b-4856-807b-2a1181e48a14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":582,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6813","nonce":116058392}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6814","nonce":116058392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6816","nonce":116058392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6815","nonce":116058392}]},"public_addr":"172.21.15.137:6813/116058392","cluster_addr":"172.21.15.137:6814/116058392","heartbeat_back_addr":"172.21.15.137:6816/116058392","heartbeat_front_addr":"172.21.15.137:6815/116058392","state":["exists","up"]},{"osd":4,"uuid":"7547102c-951a-4077-a094-fb8a65bfe6de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":32,"last_clean_end":358,"up_from":395,"up_thru":570,"down_at":359,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":571224280}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":571224280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":571224280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":571224280}]},"public_addr":"172.21.15.203:6800/571224280","cluster_addr":"172.21.15.203:6801/571224280","heartbeat_back_addr":"172.21.15.203:6803/571224280","heartbeat_front_addr":"172.21.15.203:6802/571224280","state":["exists","up"]},{"osd":5,"uuid":"49c17901-8102-48a4-9348-fa1bdb63032c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":415,"up_from":460,"up_thru":592,"down_at":416,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":1993911072}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":1993911072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":1993911072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":1993911072}]},"public_addr":"172.21.15.203:6804/1993911072","cluster_addr":"172.21.15.203:6805/1993911072","heartbeat_back_addr":"172.21.15.203:6807/1993911072","heartbeat_front_addr":"172.21.15.203:6806/1993911072","state":["exists","up"]},{"osd":6,"uuid":"75f0ead4-e981-4a15-b1c1-d83dce871ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":580,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":1494175466}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":1494175466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":1494175466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":1494175466}]},"public_addr":"172.21.15.203:6808/1494175466","cluster_addr":"172.21.15.203:6809/1494175466","heartbeat_back_addr":"172.21.15.203:6811/1494175466","heartbeat_front_addr":"172.21.15.203:6810/1494175466","state":["exists","up"]},{"osd":7,"uuid":"c0dbf299-6798-4b99-9a85-62b123971f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":570,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6812","nonce":2670809315}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6813","nonce":2670809315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6815","nonce":2670809315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6814","nonce":2670809315}]},"public_addr":"172.21.15.203:6812/2670809315","cluster_addr":"172.21.15.203:6813/2670809315","heartbeat_back_addr":"172.21.15.203:6815/2670809315","heartbeat_front_addr":"172.21.15.203:6814/2670809315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:50:28.985118+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:50:50.669669+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:50:59.752815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:50:29.752954+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:50:36.329500+0000","dead_epoch":358},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:50:59.223133+0000","dead_epoch":415},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:51:23.279669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:50:59.706397+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.0","osds":[7,1,4]},{"pgid":"5.1e","osds":[7,1,4]}],"pg_upmap_items":[{"pgid":"4.13","mappings":[{"from":6,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:6800/2096039404":"2024-09-19T07:31:25.389124+0000","172.21.15.137:6800/3636177939":"2024-09-19T07:25:41.661328+0000","172.21.15.137:0/334967443":"2024-09-19T07:25:41.661328+0000","172.21.15.137:0/2101286111":"2024-09-19T07:25:41.661328+0000","172.21.15.137:0/3387637412":"2024-09-19T07:26:06.595443+0000","172.21.15.137:0/1762989503":"2024-09-19T07:31:25.389124+0000","172.21.15.137:0/3490817671":"2024-09-19T07:26:06.595443+0000","172.21.15.137:6800/1482147392":"2024-09-19T07:26:06.595443+0000","172.21.15.137:0/1821482181":"2024-09-19T07:25:41.661328+0000","172.21.15.137:0/4028304023":"2024-09-19T07:26:06.595443+0000","172.21.15.137:0/621434521":"2024-09-19T07:31:25.389124+0000","172.21.15.137:0/3160704935":"2024-09-19T07:31:25.389124+0000","172.21.15.137:0/2357293499":"2024-09-19T07:31:25.389124+0000","172.21.15.137:0/1762352860":"2024-09-19T07:31:25.389124+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T07:51:46.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:46 smithi137 ceph-mon[29408]: 5.d scrub starts 2024-09-18T07:51:46.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:46 smithi137 ceph-mon[29408]: 5.d scrub ok 2024-09-18T07:51:46.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:46 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4121004329' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T07:51:46.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:46 smithi137 ceph-mon[29408]: 5.1e scrub starts 2024-09-18T07:51:46.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:46 smithi137 ceph-mon[29408]: 5.1e scrub ok 2024-09-18T07:51:46.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:46 smithi137 ceph-mon[39175]: 5.d scrub starts 2024-09-18T07:51:46.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:46 smithi137 ceph-mon[39175]: 5.d scrub ok 2024-09-18T07:51:46.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:46 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4121004329' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T07:51:46.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:46 smithi137 ceph-mon[39175]: 5.1e scrub starts 2024-09-18T07:51:46.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:46 smithi137 ceph-mon[39175]: 5.1e scrub ok 2024-09-18T07:51:46.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:46 smithi203 ceph-mon[32237]: 5.d scrub starts 2024-09-18T07:51:46.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:46 smithi203 ceph-mon[32237]: 5.d scrub ok 2024-09-18T07:51:46.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:46 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4121004329' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T07:51:46.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:46 smithi203 ceph-mon[32237]: 5.1e scrub starts 2024-09-18T07:51:46.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:46 smithi203 ceph-mon[32237]: 5.1e scrub ok 2024-09-18T07:51:47.018 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-18T07:51:47.018 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-18T07:51:47.267 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:47.370 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:51:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:51:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:51:47.660 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:47 smithi137 ceph-mon[29408]: pgmap v1133: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 915 B/s rd, 0 op/s 2024-09-18T07:51:47.660 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:47 smithi137 ceph-mon[29408]: 3.5 scrub starts 2024-09-18T07:51:47.660 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:47 smithi137 ceph-mon[29408]: 3.5 scrub ok 2024-09-18T07:51:47.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:47 smithi137 ceph-mon[39175]: pgmap v1133: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 915 B/s rd, 0 op/s 2024-09-18T07:51:47.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:47 smithi137 ceph-mon[39175]: 3.5 scrub starts 2024-09-18T07:51:47.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:47 smithi137 ceph-mon[39175]: 3.5 scrub ok 2024-09-18T07:51:47.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:47 smithi203 ceph-mon[32237]: pgmap v1133: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 915 B/s rd, 0 op/s 2024-09-18T07:51:47.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:47 smithi203 ceph-mon[32237]: 3.5 scrub starts 2024-09-18T07:51:47.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:47 smithi203 ceph-mon[32237]: 3.5 scrub ok 2024-09-18T07:51:47.923 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:51:47.924 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":593,"fsid":"f7ef099c-758e-11ef-bceb-c7b262605968","created":"2024-09-18T07:25:19.662367+0000","modified":"2024-09-18T07:51:45.194656+0000","last_up_change":"2024-09-18T07:47:10.160431+0000","last_in_change":"2024-09-18T07:51:08.194222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T07:28:20.899655+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"531","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"272","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-18T07:30:53.715373+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-18T07:30:54.672127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-18T07:30:56.351343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-18T07:30:58.322401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-18T07:31:00.390499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"268cd580-c931-4992-bb93-d6f63c2bfe88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":580,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6801","nonce":2811444477}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6802","nonce":2811444477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6804","nonce":2811444477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6803","nonce":2811444477}]},"public_addr":"172.21.15.137:6801/2811444477","cluster_addr":"172.21.15.137:6802/2811444477","heartbeat_back_addr":"172.21.15.137:6804/2811444477","heartbeat_front_addr":"172.21.15.137:6803/2811444477","state":["exists","up"]},{"osd":1,"uuid":"b06907bc-b499-46c0-8667-c5bdc0d85575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":580,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6805","nonce":874102487}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6806","nonce":874102487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6808","nonce":874102487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6807","nonce":874102487}]},"public_addr":"172.21.15.137:6805/874102487","cluster_addr":"172.21.15.137:6806/874102487","heartbeat_back_addr":"172.21.15.137:6808/874102487","heartbeat_front_addr":"172.21.15.137:6807/874102487","state":["exists","up"]},{"osd":2,"uuid":"480ad6c5-0e0b-4405-bbe2-381682d00bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":570,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6809","nonce":1754684686}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6810","nonce":1754684686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6812","nonce":1754684686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6811","nonce":1754684686}]},"public_addr":"172.21.15.137:6809/1754684686","cluster_addr":"172.21.15.137:6810/1754684686","heartbeat_back_addr":"172.21.15.137:6812/1754684686","heartbeat_front_addr":"172.21.15.137:6811/1754684686","state":["exists","up"]},{"osd":3,"uuid":"178be3e0-f07b-4856-807b-2a1181e48a14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":582,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6813","nonce":116058392}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6814","nonce":116058392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6816","nonce":116058392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6815","nonce":116058392}]},"public_addr":"172.21.15.137:6813/116058392","cluster_addr":"172.21.15.137:6814/116058392","heartbeat_back_addr":"172.21.15.137:6816/116058392","heartbeat_front_addr":"172.21.15.137:6815/116058392","state":["exists","up"]},{"osd":4,"uuid":"7547102c-951a-4077-a094-fb8a65bfe6de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":32,"last_clean_end":358,"up_from":395,"up_thru":570,"down_at":359,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":571224280}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":571224280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":571224280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":571224280}]},"public_addr":"172.21.15.203:6800/571224280","cluster_addr":"172.21.15.203:6801/571224280","heartbeat_back_addr":"172.21.15.203:6803/571224280","heartbeat_front_addr":"172.21.15.203:6802/571224280","state":["exists","up"]},{"osd":5,"uuid":"49c17901-8102-48a4-9348-fa1bdb63032c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":415,"up_from":460,"up_thru":592,"down_at":416,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":1993911072}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":1993911072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":1993911072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":1993911072}]},"public_addr":"172.21.15.203:6804/1993911072","cluster_addr":"172.21.15.203:6805/1993911072","heartbeat_back_addr":"172.21.15.203:6807/1993911072","heartbeat_front_addr":"172.21.15.203:6806/1993911072","state":["exists","up"]},{"osd":6,"uuid":"75f0ead4-e981-4a15-b1c1-d83dce871ab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":580,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":1494175466}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":1494175466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":1494175466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":1494175466}]},"public_addr":"172.21.15.203:6808/1494175466","cluster_addr":"172.21.15.203:6809/1494175466","heartbeat_back_addr":"172.21.15.203:6811/1494175466","heartbeat_front_addr":"172.21.15.203:6810/1494175466","state":["exists","up"]},{"osd":7,"uuid":"c0dbf299-6798-4b99-9a85-62b123971f2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":570,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6812","nonce":2670809315}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6813","nonce":2670809315}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6815","nonce":2670809315}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6814","nonce":2670809315}]},"public_addr":"172.21.15.203:6812/2670809315","cluster_addr":"172.21.15.203:6813/2670809315","heartbeat_back_addr":"172.21.15.203:6815/2670809315","heartbeat_front_addr":"172.21.15.203:6814/2670809315","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:50:28.985118+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:50:50.669669+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:50:59.752815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:50:29.752954+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:50:36.329500+0000","dead_epoch":358},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:50:59.223133+0000","dead_epoch":415},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:51:23.279669+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T07:50:59.706397+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.0","osds":[7,1,4]},{"pgid":"5.1e","osds":[7,1,4]}],"pg_upmap_items":[{"pgid":"4.13","mappings":[{"from":6,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:6800/2096039404":"2024-09-19T07:31:25.389124+0000","172.21.15.137:6800/3636177939":"2024-09-19T07:25:41.661328+0000","172.21.15.137:0/334967443":"2024-09-19T07:25:41.661328+0000","172.21.15.137:0/2101286111":"2024-09-19T07:25:41.661328+0000","172.21.15.137:0/3387637412":"2024-09-19T07:26:06.595443+0000","172.21.15.137:0/1762989503":"2024-09-19T07:31:25.389124+0000","172.21.15.137:0/3490817671":"2024-09-19T07:26:06.595443+0000","172.21.15.137:6800/1482147392":"2024-09-19T07:26:06.595443+0000","172.21.15.137:0/1821482181":"2024-09-19T07:25:41.661328+0000","172.21.15.137:0/4028304023":"2024-09-19T07:26:06.595443+0000","172.21.15.137:0/621434521":"2024-09-19T07:31:25.389124+0000","172.21.15.137:0/3160704935":"2024-09-19T07:31:25.389124+0000","172.21.15.137:0/2357293499":"2024-09-19T07:31:25.389124+0000","172.21.15.137:0/1762352860":"2024-09-19T07:31:25.389124+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T07:51:48.376 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:51:48 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:51:48.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:48 smithi137 ceph-mon[39175]: 3.1d scrub starts 2024-09-18T07:51:48.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:48 smithi137 ceph-mon[39175]: 3.1d scrub ok 2024-09-18T07:51:48.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:48 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1252951439' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T07:51:48.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:48 smithi137 ceph-mon[29408]: 3.1d scrub starts 2024-09-18T07:51:48.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:48 smithi137 ceph-mon[29408]: 3.1d scrub ok 2024-09-18T07:51:48.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:48 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1252951439' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T07:51:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:48 smithi203 ceph-mon[32237]: 3.1d scrub starts 2024-09-18T07:51:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:48 smithi203 ceph-mon[32237]: 3.1d scrub ok 2024-09-18T07:51:48.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:48 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1252951439' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-18T07:51:48.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-18T07:51:48.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-18T07:51:48.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-18T07:51:48.730 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-18T07:51:48.730 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-18T07:51:48.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-18T07:51:48.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-18T07:51:48.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-18T07:51:49.258 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:49.258 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:49.259 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:49.260 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:49.262 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:49.266 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:49.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[29408]: pgmap v1134: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-18T07:51:49.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:51:49.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:51:49.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:51:49.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[29408]: osdmap e594: 8 total, 8 up, 8 in 2024-09-18T07:51:49.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[29408]: 6.7 scrub starts 2024-09-18T07:51:49.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[29408]: 6.7 scrub ok 2024-09-18T07:51:49.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[29408]: 4.5 scrub starts 2024-09-18T07:51:49.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[29408]: 4.5 scrub ok 2024-09-18T07:51:49.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[39175]: pgmap v1134: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-18T07:51:49.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:51:49.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:51:49.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:51:49.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[39175]: osdmap e594: 8 total, 8 up, 8 in 2024-09-18T07:51:49.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[39175]: 6.7 scrub starts 2024-09-18T07:51:49.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[39175]: 6.7 scrub ok 2024-09-18T07:51:49.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[39175]: 4.5 scrub starts 2024-09-18T07:51:49.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:49 smithi137 ceph-mon[39175]: 4.5 scrub ok 2024-09-18T07:51:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:49 smithi203 ceph-mon[32237]: pgmap v1134: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-18T07:51:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:49 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:51:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:49 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:51:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:49 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:51:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:49 smithi203 ceph-mon[32237]: osdmap e594: 8 total, 8 up, 8 in 2024-09-18T07:51:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:49 smithi203 ceph-mon[32237]: 6.7 scrub starts 2024-09-18T07:51:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:49 smithi203 ceph-mon[32237]: 6.7 scrub ok 2024-09-18T07:51:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:49 smithi203 ceph-mon[32237]: 4.5 scrub starts 2024-09-18T07:51:49.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:49 smithi203 ceph-mon[32237]: 4.5 scrub ok 2024-09-18T07:51:50.149 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:50.149 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:50.483 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:50 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:51:50.483 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:50 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:51:50.483 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:50 smithi137 ceph-mon[29408]: osdmap e595: 8 total, 8 up, 8 in 2024-09-18T07:51:50.484 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:50 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:51:50.484 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:50 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:51:50.484 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:50 smithi137 ceph-mon[39175]: osdmap e595: 8 total, 8 up, 8 in 2024-09-18T07:51:50.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:50 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:51:50.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:50 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:51:50.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:50 smithi203 ceph-mon[32237]: osdmap e595: 8 total, 8 up, 8 in 2024-09-18T07:51:51.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:51 smithi137 ceph-mon[29408]: 6.16 scrub starts 2024-09-18T07:51:51.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:51 smithi137 ceph-mon[29408]: 6.16 scrub ok 2024-09-18T07:51:51.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:51 smithi137 ceph-mon[29408]: pgmap v1137: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 734 B/s rd, 0 op/s; 46 KiB/s, 0 objects/s recovering 2024-09-18T07:51:51.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:51 smithi137 ceph-mon[29408]: 5.1 scrub starts 2024-09-18T07:51:51.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:51 smithi137 ceph-mon[29408]: 5.1 scrub ok 2024-09-18T07:51:51.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:51 smithi137 ceph-mon[29408]: 3.f scrub starts 2024-09-18T07:51:51.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:51 smithi137 ceph-mon[29408]: 3.f scrub ok 2024-09-18T07:51:51.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:51 smithi137 ceph-mon[39175]: 6.16 scrub starts 2024-09-18T07:51:51.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:51 smithi137 ceph-mon[39175]: 6.16 scrub ok 2024-09-18T07:51:51.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:51 smithi137 ceph-mon[39175]: pgmap v1137: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 734 B/s rd, 0 op/s; 46 KiB/s, 0 objects/s recovering 2024-09-18T07:51:51.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:51 smithi137 ceph-mon[39175]: 5.1 scrub starts 2024-09-18T07:51:51.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:51 smithi137 ceph-mon[39175]: 5.1 scrub ok 2024-09-18T07:51:51.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:51 smithi137 ceph-mon[39175]: 3.f scrub starts 2024-09-18T07:51:51.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:51 smithi137 ceph-mon[39175]: 3.f scrub ok 2024-09-18T07:51:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:51 smithi203 ceph-mon[32237]: 6.16 scrub starts 2024-09-18T07:51:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:51 smithi203 ceph-mon[32237]: 6.16 scrub ok 2024-09-18T07:51:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:51 smithi203 ceph-mon[32237]: pgmap v1137: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 734 B/s rd, 0 op/s; 46 KiB/s, 0 objects/s recovering 2024-09-18T07:51:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:51 smithi203 ceph-mon[32237]: 5.1 scrub starts 2024-09-18T07:51:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:51 smithi203 ceph-mon[32237]: 5.1 scrub ok 2024-09-18T07:51:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:51 smithi203 ceph-mon[32237]: 3.f scrub starts 2024-09-18T07:51:51.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:51 smithi203 ceph-mon[32237]: 3.f scrub ok 2024-09-18T07:51:51.848 INFO:teuthology.orchestra.run.smithi137.stdout:111669149973 2024-09-18T07:51:51.848 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-18T07:51:52.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:52 smithi137 ceph-mon[29408]: 3.1c scrub starts 2024-09-18T07:51:52.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:52 smithi137 ceph-mon[29408]: 3.1c scrub ok 2024-09-18T07:51:52.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:52 smithi137 ceph-mon[29408]: 4.d scrub starts 2024-09-18T07:51:52.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:52 smithi137 ceph-mon[29408]: 4.d scrub ok 2024-09-18T07:51:52.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:52 smithi137 ceph-mon[29408]: pgmap v1138: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 608 B/s rd, 0 op/s; 38 KiB/s, 0 objects/s recovering 2024-09-18T07:51:52.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:52 smithi137 ceph-mon[39175]: 3.1c scrub starts 2024-09-18T07:51:52.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:52 smithi137 ceph-mon[39175]: 3.1c scrub ok 2024-09-18T07:51:52.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:52 smithi137 ceph-mon[39175]: 4.d scrub starts 2024-09-18T07:51:52.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:52 smithi137 ceph-mon[39175]: 4.d scrub ok 2024-09-18T07:51:52.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:52 smithi137 ceph-mon[39175]: pgmap v1138: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 608 B/s rd, 0 op/s; 38 KiB/s, 0 objects/s recovering 2024-09-18T07:51:52.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:52 smithi203 ceph-mon[32237]: 3.1c scrub starts 2024-09-18T07:51:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:52 smithi203 ceph-mon[32237]: 3.1c scrub ok 2024-09-18T07:51:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:52 smithi203 ceph-mon[32237]: 4.d scrub starts 2024-09-18T07:51:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:52 smithi203 ceph-mon[32237]: 4.d scrub ok 2024-09-18T07:51:52.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:52 smithi203 ceph-mon[32237]: pgmap v1138: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 608 B/s rd, 0 op/s; 38 KiB/s, 0 objects/s recovering 2024-09-18T07:51:52.905 INFO:teuthology.orchestra.run.smithi137.stdout:1975684956218 2024-09-18T07:51:52.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-18T07:51:53.912 INFO:teuthology.orchestra.run.smithi137.stdout:236223201536 2024-09-18T07:51:53.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-18T07:51:54.778 INFO:teuthology.orchestra.run.smithi137.stdout:55834575139 2024-09-18T07:51:54.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-18T07:51:55.002 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[29408]: 4.15 scrub starts 2024-09-18T07:51:55.002 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[29408]: 4.15 scrub ok 2024-09-18T07:51:55.002 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[29408]: 5.e scrub starts 2024-09-18T07:51:55.002 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[29408]: 5.e scrub ok 2024-09-18T07:51:55.002 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[29408]: pgmap v1139: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 32 KiB/s, 0 objects/s recovering 2024-09-18T07:51:55.002 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[29408]: 6.15 scrub starts 2024-09-18T07:51:55.003 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[29408]: 6.15 scrub ok 2024-09-18T07:51:55.003 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:51:55.003 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:51:55.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[39175]: 4.15 scrub starts 2024-09-18T07:51:55.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[39175]: 4.15 scrub ok 2024-09-18T07:51:55.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[39175]: 5.e scrub starts 2024-09-18T07:51:55.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[39175]: 5.e scrub ok 2024-09-18T07:51:55.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[39175]: pgmap v1139: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 32 KiB/s, 0 objects/s recovering 2024-09-18T07:51:55.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[39175]: 6.15 scrub starts 2024-09-18T07:51:55.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[39175]: 6.15 scrub ok 2024-09-18T07:51:55.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:51:55.004 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:54 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:51:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:54 smithi203 ceph-mon[32237]: 4.15 scrub starts 2024-09-18T07:51:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:54 smithi203 ceph-mon[32237]: 4.15 scrub ok 2024-09-18T07:51:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:54 smithi203 ceph-mon[32237]: 5.e scrub starts 2024-09-18T07:51:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:54 smithi203 ceph-mon[32237]: 5.e scrub ok 2024-09-18T07:51:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:54 smithi203 ceph-mon[32237]: pgmap v1139: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 32 KiB/s, 0 objects/s recovering 2024-09-18T07:51:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:54 smithi203 ceph-mon[32237]: 6.15 scrub starts 2024-09-18T07:51:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:54 smithi203 ceph-mon[32237]: 6.15 scrub ok 2024-09-18T07:51:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:54 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:51:55.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:54 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:51:56.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:55 smithi137 ceph-mon[39175]: 5.13 scrub starts 2024-09-18T07:51:56.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:55 smithi137 ceph-mon[39175]: 5.13 scrub ok 2024-09-18T07:51:56.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:55 smithi137 ceph-mon[39175]: 4.1 scrub starts 2024-09-18T07:51:56.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:55 smithi137 ceph-mon[39175]: 4.1 scrub ok 2024-09-18T07:51:56.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:55 smithi137 ceph-mon[39175]: osdmap e596: 8 total, 8 up, 8 in 2024-09-18T07:51:56.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:55 smithi137 ceph-mon[39175]: osdmap e597: 8 total, 8 up, 8 in 2024-09-18T07:51:56.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:55 smithi137 ceph-mon[39175]: 5.2 scrub starts 2024-09-18T07:51:56.170 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:55 smithi137 ceph-mon[39175]: 5.2 scrub ok 2024-09-18T07:51:56.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:55 smithi137 ceph-mon[29408]: 5.13 scrub starts 2024-09-18T07:51:56.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:55 smithi137 ceph-mon[29408]: 5.13 scrub ok 2024-09-18T07:51:56.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:55 smithi137 ceph-mon[29408]: 4.1 scrub starts 2024-09-18T07:51:56.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:55 smithi137 ceph-mon[29408]: 4.1 scrub ok 2024-09-18T07:51:56.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:55 smithi137 ceph-mon[29408]: osdmap e596: 8 total, 8 up, 8 in 2024-09-18T07:51:56.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:55 smithi137 ceph-mon[29408]: osdmap e597: 8 total, 8 up, 8 in 2024-09-18T07:51:56.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:55 smithi137 ceph-mon[29408]: 5.2 scrub starts 2024-09-18T07:51:56.171 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:55 smithi137 ceph-mon[29408]: 5.2 scrub ok 2024-09-18T07:51:56.225 INFO:teuthology.orchestra.run.smithi137.stdout:77309411612 2024-09-18T07:51:56.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-18T07:51:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:55 smithi203 ceph-mon[32237]: 5.13 scrub starts 2024-09-18T07:51:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:55 smithi203 ceph-mon[32237]: 5.13 scrub ok 2024-09-18T07:51:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:55 smithi203 ceph-mon[32237]: 4.1 scrub starts 2024-09-18T07:51:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:55 smithi203 ceph-mon[32237]: 4.1 scrub ok 2024-09-18T07:51:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:55 smithi203 ceph-mon[32237]: osdmap e596: 8 total, 8 up, 8 in 2024-09-18T07:51:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:55 smithi203 ceph-mon[32237]: osdmap e597: 8 total, 8 up, 8 in 2024-09-18T07:51:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:55 smithi203 ceph-mon[32237]: 5.2 scrub starts 2024-09-18T07:51:56.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:55 smithi203 ceph-mon[32237]: 5.2 scrub ok 2024-09-18T07:51:57.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:51:57 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:51:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:51:57.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:57 smithi203 ceph-mon[32237]: 1.1 scrub starts 2024-09-18T07:51:57.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:57 smithi203 ceph-mon[32237]: 1.1 scrub ok 2024-09-18T07:51:57.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:57 smithi203 ceph-mon[32237]: pgmap v1142: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 781 B/s rd, 0 op/s 2024-09-18T07:51:57.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:57 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:51:57.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:57 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:51:57.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:57 smithi203 ceph-mon[32237]: 3.13 scrub starts 2024-09-18T07:51:57.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:57 smithi203 ceph-mon[32237]: 3.13 scrub ok 2024-09-18T07:51:57.500 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:57 smithi137 ceph-mon[29408]: 1.1 scrub starts 2024-09-18T07:51:57.500 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:57 smithi137 ceph-mon[29408]: 1.1 scrub ok 2024-09-18T07:51:57.500 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:57 smithi137 ceph-mon[29408]: pgmap v1142: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 781 B/s rd, 0 op/s 2024-09-18T07:51:57.500 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:57 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:51:57.501 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:57 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:51:57.501 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:57 smithi137 ceph-mon[29408]: 3.13 scrub starts 2024-09-18T07:51:57.501 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:57 smithi137 ceph-mon[29408]: 3.13 scrub ok 2024-09-18T07:51:57.501 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:57 smithi137 ceph-mon[39175]: 1.1 scrub starts 2024-09-18T07:51:57.501 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:57 smithi137 ceph-mon[39175]: 1.1 scrub ok 2024-09-18T07:51:57.501 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:57 smithi137 ceph-mon[39175]: pgmap v1142: 142 pgs: 142 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 781 B/s rd, 0 op/s 2024-09-18T07:51:57.501 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:57 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:51:57.501 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:57 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:51:57.501 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:57 smithi137 ceph-mon[39175]: 3.13 scrub starts 2024-09-18T07:51:57.501 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:57 smithi137 ceph-mon[39175]: 3.13 scrub ok 2024-09-18T07:51:57.528 INFO:teuthology.orchestra.run.smithi137.stdout:201863463174 2024-09-18T07:51:57.528 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-18T07:51:57.956 INFO:teuthology.orchestra.run.smithi137.stdout:34359738664 2024-09-18T07:51:57.957 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-18T07:51:58.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:51:58 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:51:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:58 smithi203 ceph-mon[32237]: 5.15 scrub starts 2024-09-18T07:51:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:58 smithi203 ceph-mon[32237]: 5.15 scrub ok 2024-09-18T07:51:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:58 smithi203 ceph-mon[32237]: osdmap e598: 8 total, 8 up, 8 in 2024-09-18T07:51:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:58 smithi203 ceph-mon[32237]: 1.9 scrub starts 2024-09-18T07:51:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:58 smithi203 ceph-mon[32237]: 1.9 scrub ok 2024-09-18T07:51:58.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:58 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:51:58.490 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:58 smithi137 ceph-mon[29408]: 5.15 scrub starts 2024-09-18T07:51:58.491 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:58 smithi137 ceph-mon[29408]: 5.15 scrub ok 2024-09-18T07:51:58.491 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:58 smithi137 ceph-mon[29408]: osdmap e598: 8 total, 8 up, 8 in 2024-09-18T07:51:58.491 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:58 smithi137 ceph-mon[29408]: 1.9 scrub starts 2024-09-18T07:51:58.491 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:58 smithi137 ceph-mon[29408]: 1.9 scrub ok 2024-09-18T07:51:58.491 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:58 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:51:58.491 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:58 smithi137 ceph-mon[39175]: 5.15 scrub starts 2024-09-18T07:51:58.491 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:58 smithi137 ceph-mon[39175]: 5.15 scrub ok 2024-09-18T07:51:58.491 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:58 smithi137 ceph-mon[39175]: osdmap e598: 8 total, 8 up, 8 in 2024-09-18T07:51:58.491 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:58 smithi137 ceph-mon[39175]: 1.9 scrub starts 2024-09-18T07:51:58.491 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:58 smithi137 ceph-mon[39175]: 1.9 scrub ok 2024-09-18T07:51:58.491 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:58 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:51:58.518 INFO:teuthology.orchestra.run.smithi137.stdout:1696512082008 2024-09-18T07:51:58.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-18T07:51:58.562 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:58.562 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:58.563 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:51:59.381 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:59 smithi137 ceph-mon[29408]: pgmap v1144: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-18T07:51:59.381 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:51:59 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:51:59.382 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:59 smithi137 ceph-mon[39175]: pgmap v1144: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-18T07:51:59.382 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:51:59 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:51:59.539 INFO:teuthology.orchestra.run.smithi137.stdout:111669149975 2024-09-18T07:51:59.667 INFO:teuthology.orchestra.run.smithi137.stdout:201863463175 2024-09-18T07:51:59.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:59 smithi203 ceph-mon[32237]: pgmap v1144: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-18T07:51:59.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:51:59 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:52:00.395 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:00 smithi137 ceph-mon[29408]: 5.f scrub starts 2024-09-18T07:52:00.395 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:00 smithi137 ceph-mon[29408]: 5.f scrub ok 2024-09-18T07:52:00.396 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:00 smithi137 ceph-mon[29408]: 4.10 scrub starts 2024-09-18T07:52:00.396 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:00 smithi137 ceph-mon[29408]: 4.10 scrub ok 2024-09-18T07:52:00.396 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:00 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1598378695' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-18T07:52:00.396 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:00 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3126452367' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-18T07:52:00.396 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:00 smithi137 ceph-mon[29408]: osdmap e599: 8 total, 8 up, 8 in 2024-09-18T07:52:00.396 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:00 smithi137 ceph-mon[39175]: 5.f scrub starts 2024-09-18T07:52:00.396 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:00 smithi137 ceph-mon[39175]: 5.f scrub ok 2024-09-18T07:52:00.396 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:00 smithi137 ceph-mon[39175]: 4.10 scrub starts 2024-09-18T07:52:00.396 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:00 smithi137 ceph-mon[39175]: 4.10 scrub ok 2024-09-18T07:52:00.396 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:00 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1598378695' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-18T07:52:00.397 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:00 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3126452367' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-18T07:52:00.397 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:00 smithi137 ceph-mon[39175]: osdmap e599: 8 total, 8 up, 8 in 2024-09-18T07:52:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:00 smithi203 ceph-mon[32237]: 5.f scrub starts 2024-09-18T07:52:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:00 smithi203 ceph-mon[32237]: 5.f scrub ok 2024-09-18T07:52:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:00 smithi203 ceph-mon[32237]: 4.10 scrub starts 2024-09-18T07:52:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:00 smithi203 ceph-mon[32237]: 4.10 scrub ok 2024-09-18T07:52:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:00 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1598378695' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-18T07:52:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:00 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3126452367' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-18T07:52:00.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:00 smithi203 ceph-mon[32237]: osdmap e599: 8 total, 8 up, 8 in 2024-09-18T07:52:00.928 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:52:00.929 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:52:00.930 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:52:00.973 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149973 got 111669149975 for osd.3 2024-09-18T07:52:00.973 DEBUG:teuthology.parallel:result is None 2024-09-18T07:52:01.010 INFO:teuthology.orchestra.run.smithi137.stdout:236223201537 2024-09-18T07:52:01.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[39175]: 3.11 scrub starts 2024-09-18T07:52:01.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[39175]: 3.11 scrub ok 2024-09-18T07:52:01.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[39175]: 5.c scrub starts 2024-09-18T07:52:01.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[39175]: 5.c scrub ok 2024-09-18T07:52:01.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[39175]: 6.1e scrub starts 2024-09-18T07:52:01.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[39175]: 6.1e scrub ok 2024-09-18T07:52:01.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[39175]: pgmap v1145: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:52:01.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[39175]: 6.5 scrub starts 2024-09-18T07:52:01.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[39175]: 6.5 scrub ok 2024-09-18T07:52:01.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/73110899' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-18T07:52:01.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[29408]: 3.11 scrub starts 2024-09-18T07:52:01.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[29408]: 3.11 scrub ok 2024-09-18T07:52:01.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[29408]: 5.c scrub starts 2024-09-18T07:52:01.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[29408]: 5.c scrub ok 2024-09-18T07:52:01.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[29408]: 6.1e scrub starts 2024-09-18T07:52:01.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[29408]: 6.1e scrub ok 2024-09-18T07:52:01.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[29408]: pgmap v1145: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:52:01.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[29408]: 6.5 scrub starts 2024-09-18T07:52:01.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[29408]: 6.5 scrub ok 2024-09-18T07:52:01.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:01 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/73110899' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-18T07:52:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:01 smithi203 ceph-mon[32237]: 3.11 scrub starts 2024-09-18T07:52:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:01 smithi203 ceph-mon[32237]: 3.11 scrub ok 2024-09-18T07:52:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:01 smithi203 ceph-mon[32237]: 5.c scrub starts 2024-09-18T07:52:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:01 smithi203 ceph-mon[32237]: 5.c scrub ok 2024-09-18T07:52:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:01 smithi203 ceph-mon[32237]: 6.1e scrub starts 2024-09-18T07:52:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:01 smithi203 ceph-mon[32237]: 6.1e scrub ok 2024-09-18T07:52:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:01 smithi203 ceph-mon[32237]: pgmap v1145: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-18T07:52:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:01 smithi203 ceph-mon[32237]: 6.5 scrub starts 2024-09-18T07:52:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:01 smithi203 ceph-mon[32237]: 6.5 scrub ok 2024-09-18T07:52:01.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:01 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/73110899' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-18T07:52:01.978 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201536 got 236223201537 for osd.7 2024-09-18T07:52:01.978 DEBUG:teuthology.parallel:result is None 2024-09-18T07:52:02.111 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:52:02.112 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:52:02.122 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863463174 got 201863463175 for osd.6 2024-09-18T07:52:02.122 DEBUG:teuthology.parallel:result is None 2024-09-18T07:52:02.384 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:02 smithi137 ceph-mon[29408]: 3.19 scrub starts 2024-09-18T07:52:02.384 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:02 smithi137 ceph-mon[29408]: 3.19 scrub ok 2024-09-18T07:52:02.384 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:02 smithi137 ceph-mon[29408]: 1.5 scrub starts 2024-09-18T07:52:02.384 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:02 smithi137 ceph-mon[29408]: 1.5 scrub ok 2024-09-18T07:52:02.385 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:02 smithi137 ceph-mon[39175]: 3.19 scrub starts 2024-09-18T07:52:02.385 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:02 smithi137 ceph-mon[39175]: 3.19 scrub ok 2024-09-18T07:52:02.385 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:02 smithi137 ceph-mon[39175]: 1.5 scrub starts 2024-09-18T07:52:02.385 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:02 smithi137 ceph-mon[39175]: 1.5 scrub ok 2024-09-18T07:52:02.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:02 smithi203 ceph-mon[32237]: 3.19 scrub starts 2024-09-18T07:52:02.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:02 smithi203 ceph-mon[32237]: 3.19 scrub ok 2024-09-18T07:52:02.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:02 smithi203 ceph-mon[32237]: 1.5 scrub starts 2024-09-18T07:52:02.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:02 smithi203 ceph-mon[32237]: 1.5 scrub ok 2024-09-18T07:52:03.038 INFO:teuthology.orchestra.run.smithi137.stdout:77309411613 2024-09-18T07:52:03.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:03 smithi137 ceph-mon[39175]: 6.12 scrub starts 2024-09-18T07:52:03.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:03 smithi137 ceph-mon[39175]: 6.12 scrub ok 2024-09-18T07:52:03.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:03 smithi137 ceph-mon[39175]: 1.8 scrub starts 2024-09-18T07:52:03.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:03 smithi137 ceph-mon[39175]: 1.8 scrub ok 2024-09-18T07:52:03.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:03 smithi137 ceph-mon[39175]: pgmap v1147: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 760 B/s rd, 0 op/s 2024-09-18T07:52:03.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:03 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1785795994' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-18T07:52:03.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:03 smithi137 ceph-mon[29408]: 6.12 scrub starts 2024-09-18T07:52:03.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:03 smithi137 ceph-mon[29408]: 6.12 scrub ok 2024-09-18T07:52:03.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:03 smithi137 ceph-mon[29408]: 1.8 scrub starts 2024-09-18T07:52:03.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:03 smithi137 ceph-mon[29408]: 1.8 scrub ok 2024-09-18T07:52:03.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:03 smithi137 ceph-mon[29408]: pgmap v1147: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 760 B/s rd, 0 op/s 2024-09-18T07:52:03.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:03 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1785795994' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-18T07:52:03.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:03 smithi203 ceph-mon[32237]: 6.12 scrub starts 2024-09-18T07:52:03.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:03 smithi203 ceph-mon[32237]: 6.12 scrub ok 2024-09-18T07:52:03.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:03 smithi203 ceph-mon[32237]: 1.8 scrub starts 2024-09-18T07:52:03.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:03 smithi203 ceph-mon[32237]: 1.8 scrub ok 2024-09-18T07:52:03.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:03 smithi203 ceph-mon[32237]: pgmap v1147: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 760 B/s rd, 0 op/s 2024-09-18T07:52:03.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:03 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1785795994' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-18T07:52:04.058 INFO:teuthology.orchestra.run.smithi137.stdout:34359738665 2024-09-18T07:52:04.264 INFO:teuthology.orchestra.run.smithi137.stdout:55834575141 2024-09-18T07:52:04.419 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[39175]: 5.9 scrub starts 2024-09-18T07:52:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[39175]: 5.9 scrub ok 2024-09-18T07:52:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[39175]: 6.a scrub starts 2024-09-18T07:52:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[39175]: 6.a scrub ok 2024-09-18T07:52:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[39175]: 6.1d scrub starts 2024-09-18T07:52:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[39175]: 6.1d scrub ok 2024-09-18T07:52:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[39175]: 3.14 scrub starts 2024-09-18T07:52:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[39175]: 3.14 scrub ok 2024-09-18T07:52:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[39175]: 2.0 scrub starts 2024-09-18T07:52:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[39175]: 2.0 scrub ok 2024-09-18T07:52:04.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3443303163' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-18T07:52:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[29408]: 5.9 scrub starts 2024-09-18T07:52:04.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[29408]: 5.9 scrub ok 2024-09-18T07:52:04.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[29408]: 6.a scrub starts 2024-09-18T07:52:04.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[29408]: 6.a scrub ok 2024-09-18T07:52:04.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[29408]: 6.1d scrub starts 2024-09-18T07:52:04.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[29408]: 6.1d scrub ok 2024-09-18T07:52:04.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[29408]: 3.14 scrub starts 2024-09-18T07:52:04.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[29408]: 3.14 scrub ok 2024-09-18T07:52:04.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[29408]: 2.0 scrub starts 2024-09-18T07:52:04.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[29408]: 2.0 scrub ok 2024-09-18T07:52:04.421 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:04 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3443303163' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-18T07:52:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:04 smithi203 ceph-mon[32237]: 5.9 scrub starts 2024-09-18T07:52:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:04 smithi203 ceph-mon[32237]: 5.9 scrub ok 2024-09-18T07:52:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:04 smithi203 ceph-mon[32237]: 6.a scrub starts 2024-09-18T07:52:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:04 smithi203 ceph-mon[32237]: 6.a scrub ok 2024-09-18T07:52:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:04 smithi203 ceph-mon[32237]: 6.1d scrub starts 2024-09-18T07:52:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:04 smithi203 ceph-mon[32237]: 6.1d scrub ok 2024-09-18T07:52:04.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:04 smithi203 ceph-mon[32237]: 3.14 scrub starts 2024-09-18T07:52:04.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:04 smithi203 ceph-mon[32237]: 3.14 scrub ok 2024-09-18T07:52:04.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:04 smithi203 ceph-mon[32237]: 2.0 scrub starts 2024-09-18T07:52:04.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:04 smithi203 ceph-mon[32237]: 2.0 scrub ok 2024-09-18T07:52:04.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:04 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3443303163' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-18T07:52:04.847 INFO:teuthology.orchestra.run.smithi137.stdout:1975684956220 2024-09-18T07:52:05.193 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738664 got 34359738665 for osd.0 2024-09-18T07:52:05.194 DEBUG:teuthology.parallel:result is None 2024-09-18T07:52:05.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:05 smithi137 ceph-mon[29408]: 5.1d scrub starts 2024-09-18T07:52:05.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:05 smithi137 ceph-mon[29408]: 5.1d scrub ok 2024-09-18T07:52:05.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:05 smithi137 ceph-mon[29408]: 4.9 scrub starts 2024-09-18T07:52:05.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:05 smithi137 ceph-mon[29408]: 4.9 scrub ok 2024-09-18T07:52:05.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:05 smithi137 ceph-mon[29408]: pgmap v1148: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:52:05.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:05 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/1617437350' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-18T07:52:05.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:05 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/4215289060' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-18T07:52:05.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:05 smithi137 ceph-mon[39175]: 5.1d scrub starts 2024-09-18T07:52:05.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:05 smithi137 ceph-mon[39175]: 5.1d scrub ok 2024-09-18T07:52:05.420 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:05 smithi137 ceph-mon[39175]: 4.9 scrub starts 2024-09-18T07:52:05.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:05 smithi137 ceph-mon[39175]: 4.9 scrub ok 2024-09-18T07:52:05.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:05 smithi137 ceph-mon[39175]: pgmap v1148: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:52:05.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:05 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/1617437350' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-18T07:52:05.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:05 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/4215289060' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-18T07:52:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:05 smithi203 ceph-mon[32237]: 5.1d scrub starts 2024-09-18T07:52:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:05 smithi203 ceph-mon[32237]: 5.1d scrub ok 2024-09-18T07:52:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:05 smithi203 ceph-mon[32237]: 4.9 scrub starts 2024-09-18T07:52:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:05 smithi203 ceph-mon[32237]: 4.9 scrub ok 2024-09-18T07:52:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:05 smithi203 ceph-mon[32237]: pgmap v1148: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:52:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:05 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/1617437350' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-18T07:52:05.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:05 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/4215289060' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-18T07:52:06.064 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1975684956218 got 1975684956220 for osd.5 2024-09-18T07:52:06.064 DEBUG:teuthology.parallel:result is None 2024-09-18T07:52:06.419 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[29408]: 6.3 scrub starts 2024-09-18T07:52:06.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[29408]: 6.3 scrub ok 2024-09-18T07:52:06.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[29408]: 5.1b scrub starts 2024-09-18T07:52:06.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[29408]: 5.1b scrub ok 2024-09-18T07:52:06.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[29408]: purged_snaps scrub starts 2024-09-18T07:52:06.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[29408]: purged_snaps scrub ok 2024-09-18T07:52:06.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[29408]: 4.e scrub starts 2024-09-18T07:52:06.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[29408]: 4.e scrub ok 2024-09-18T07:52:06.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[29408]: osdmap e600: 8 total, 8 up, 8 in 2024-09-18T07:52:06.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[39175]: 6.3 scrub starts 2024-09-18T07:52:06.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[39175]: 6.3 scrub ok 2024-09-18T07:52:06.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[39175]: 5.1b scrub starts 2024-09-18T07:52:06.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[39175]: 5.1b scrub ok 2024-09-18T07:52:06.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[39175]: purged_snaps scrub starts 2024-09-18T07:52:06.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[39175]: purged_snaps scrub ok 2024-09-18T07:52:06.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[39175]: 4.e scrub starts 2024-09-18T07:52:06.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[39175]: 4.e scrub ok 2024-09-18T07:52:06.422 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:06 smithi137 ceph-mon[39175]: osdmap e600: 8 total, 8 up, 8 in 2024-09-18T07:52:06.451 INFO:teuthology.orchestra.run.smithi137.stdout:1696512082010 2024-09-18T07:52:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:06 smithi203 ceph-mon[32237]: 6.3 scrub starts 2024-09-18T07:52:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:06 smithi203 ceph-mon[32237]: 6.3 scrub ok 2024-09-18T07:52:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:06 smithi203 ceph-mon[32237]: 5.1b scrub starts 2024-09-18T07:52:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:06 smithi203 ceph-mon[32237]: 5.1b scrub ok 2024-09-18T07:52:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:06 smithi203 ceph-mon[32237]: purged_snaps scrub starts 2024-09-18T07:52:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:06 smithi203 ceph-mon[32237]: purged_snaps scrub ok 2024-09-18T07:52:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:06 smithi203 ceph-mon[32237]: 4.e scrub starts 2024-09-18T07:52:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:06 smithi203 ceph-mon[32237]: 4.e scrub ok 2024-09-18T07:52:06.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:06 smithi203 ceph-mon[32237]: osdmap e600: 8 total, 8 up, 8 in 2024-09-18T07:52:06.885 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575139 got 55834575141 for osd.1 2024-09-18T07:52:06.885 DEBUG:teuthology.parallel:result is None 2024-09-18T07:52:07.169 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:52:07 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:52:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:52:07.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:07 smithi137 ceph-mon[29408]: pgmap v1150: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-18T07:52:07.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:07 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/940334766' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-18T07:52:07.605 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:07 smithi137 ceph-mon[39175]: pgmap v1150: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-18T07:52:07.606 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:07 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/940334766' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-18T07:52:07.634 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1696512082008 got 1696512082010 for osd.4 2024-09-18T07:52:07.634 DEBUG:teuthology.parallel:result is None 2024-09-18T07:52:07.710 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:07 smithi203 ceph-mon[32237]: pgmap v1150: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-18T07:52:07.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:07 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/940334766' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-18T07:52:07.945 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411612 got 77309411613 for osd.2 2024-09-18T07:52:07.945 DEBUG:teuthology.parallel:result is None 2024-09-18T07:52:07.945 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-18T07:52:07.946 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:52:08.198 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:52:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:08 smithi203 ceph-mon[32237]: 3.18 scrub starts 2024-09-18T07:52:08.461 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:08 smithi203 ceph-mon[32237]: 3.18 scrub ok 2024-09-18T07:52:08.461 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:52:08 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug there is no tcmu-runner data available 2024-09-18T07:52:08.611 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:08 smithi137 ceph-mon[29408]: 3.18 scrub starts 2024-09-18T07:52:08.611 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:08 smithi137 ceph-mon[29408]: 3.18 scrub ok 2024-09-18T07:52:08.611 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:08 smithi137 ceph-mon[39175]: 3.18 scrub starts 2024-09-18T07:52:08.612 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:08 smithi137 ceph-mon[39175]: 3.18 scrub ok 2024-09-18T07:52:08.880 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:52:08.882 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:52:09.566 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:09 smithi137 ceph-mon[29408]: pgmap v1151: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:52:09.566 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:09 smithi137 ceph-mon[29408]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:52:09.566 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:09 smithi137 ceph-mon[29408]: 3.2 scrub starts 2024-09-18T07:52:09.566 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:09 smithi137 ceph-mon[29408]: 3.2 scrub ok 2024-09-18T07:52:09.567 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:09 smithi137 ceph-mon[39175]: pgmap v1151: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:52:09.567 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:09 smithi137 ceph-mon[39175]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:52:09.567 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:09 smithi137 ceph-mon[39175]: 3.2 scrub starts 2024-09-18T07:52:09.567 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:09 smithi137 ceph-mon[39175]: 3.2 scrub ok 2024-09-18T07:52:09.595 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":1151,"stamp":"2024-09-18T07:52:07.934188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2530,"num_read_kb":2282,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":743,"num_bytes_recovered":875547,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1979,"ondisk_log_size":1979,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":428,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5859140,"kb_used_data":211692,"kb_used_omap":20,"kb_used_meta":5646187,"kb_avail":744102076,"statfs":{"total":767960285184,"available":761960525824,"internally_reserved":0,"allocated":216772608,"data_stored":212691303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20886,"internal_metadata":5781696106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002062"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":76,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.204303+0000","last_change":"2024-09-18T07:51:26.792552+0000","last_active":"2024-09-18T07:52:00.204303+0000","last_peered":"2024-09-18T07:52:00.204303+0000","last_clean":"2024-09-18T07:52:00.204303+0000","last_became_active":"2024-09-18T07:51:09.218894+0000","last_became_peered":"2024-09-18T07:51:09.218894+0000","last_unstale":"2024-09-18T07:52:00.204303+0000","last_undegraded":"2024-09-18T07:52:00.204303+0000","last_fullsized":"2024-09-18T07:52:00.204303+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:26.792494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:26.792494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:55.122561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"568'21","reported_seq":1128,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274118+0000","last_change":"2024-09-18T07:51:01.977506+0000","last_active":"2024-09-18T07:52:05.274118+0000","last_peered":"2024-09-18T07:52:05.274118+0000","last_clean":"2024-09-18T07:52:05.274118+0000","last_became_active":"2024-09-18T07:47:20.136254+0000","last_became_peered":"2024-09-18T07:47:20.136254+0000","last_unstale":"2024-09-18T07:52:05.274118+0000","last_undegraded":"2024-09-18T07:52:05.274118+0000","last_fullsized":"2024-09-18T07:52:05.274118+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:51:01.977421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:01.977421+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:19.711863+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":717,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.270648+0000","last_change":"2024-09-18T07:51:19.257480+0000","last_active":"2024-09-18T07:52:05.270648+0000","last_peered":"2024-09-18T07:52:05.270648+0000","last_clean":"2024-09-18T07:52:05.270648+0000","last_became_active":"2024-09-18T07:51:19.257114+0000","last_became_peered":"2024-09-18T07:51:19.257114+0000","last_unstale":"2024-09-18T07:52:05.270648+0000","last_undegraded":"2024-09-18T07:52:05.270648+0000","last_fullsized":"2024-09-18T07:52:05.270648+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:13.608556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:13.608556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:32.147463+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1011,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274428+0000","last_change":"2024-09-18T07:51:17.961214+0000","last_active":"2024-09-18T07:52:05.274428+0000","last_peered":"2024-09-18T07:52:05.274428+0000","last_clean":"2024-09-18T07:52:05.274428+0000","last_became_active":"2024-09-18T07:49:32.688553+0000","last_became_peered":"2024-09-18T07:49:32.688553+0000","last_unstale":"2024-09-18T07:52:05.274428+0000","last_undegraded":"2024-09-18T07:52:05.274428+0000","last_fullsized":"2024-09-18T07:52:05.274428+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:17.961167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:17.961167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:31.291311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":129,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272159+0000","last_change":"2024-09-18T07:51:31.473729+0000","last_active":"2024-09-18T07:52:05.272159+0000","last_peered":"2024-09-18T07:52:05.272159+0000","last_clean":"2024-09-18T07:52:05.272159+0000","last_became_active":"2024-09-18T07:50:23.285220+0000","last_became_peered":"2024-09-18T07:50:23.285220+0000","last_unstale":"2024-09-18T07:52:05.272159+0000","last_undegraded":"2024-09-18T07:52:05.272159+0000","last_fullsized":"2024-09-18T07:52:05.272159+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:31.473663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:31.473663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:34.340579+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":129,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272750+0000","last_change":"2024-09-18T07:51:06.625821+0000","last_active":"2024-09-18T07:52:05.272750+0000","last_peered":"2024-09-18T07:52:05.272750+0000","last_clean":"2024-09-18T07:52:05.272750+0000","last_became_active":"2024-09-18T07:50:23.292434+0000","last_became_peered":"2024-09-18T07:50:23.292434+0000","last_unstale":"2024-09-18T07:52:05.272750+0000","last_undegraded":"2024-09-18T07:52:05.272750+0000","last_fullsized":"2024-09-18T07:52:05.272750+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:06.625767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:06.625767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:31.857130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"568'39","reported_seq":764,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271762+0000","last_change":"2024-09-18T07:51:27.295746+0000","last_active":"2024-09-18T07:52:05.271762+0000","last_peered":"2024-09-18T07:52:05.271762+0000","last_clean":"2024-09-18T07:52:05.271762+0000","last_became_active":"2024-09-18T07:51:27.295067+0000","last_became_peered":"2024-09-18T07:51:27.295067+0000","last_unstale":"2024-09-18T07:52:05.271762+0000","last_undegraded":"2024-09-18T07:52:05.271762+0000","last_fullsized":"2024-09-18T07:52:05.271762+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'39","last_scrub_stamp":"2024-09-18T07:51:10.541004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:10.541004+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:26.190605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1068,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274072+0000","last_change":"2024-09-18T07:51:28.913904+0000","last_active":"2024-09-18T07:52:05.274072+0000","last_peered":"2024-09-18T07:52:05.274072+0000","last_clean":"2024-09-18T07:52:05.274072+0000","last_became_active":"2024-09-18T07:47:20.136358+0000","last_became_peered":"2024-09-18T07:47:20.136358+0000","last_unstale":"2024-09-18T07:52:05.274072+0000","last_undegraded":"2024-09-18T07:52:05.274072+0000","last_fullsized":"2024-09-18T07:52:05.274072+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:28.913848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:28.913848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:34.992822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":963,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.206514+0000","last_change":"2024-09-18T07:51:47.388662+0000","last_active":"2024-09-18T07:52:00.206514+0000","last_peered":"2024-09-18T07:52:00.206514+0000","last_clean":"2024-09-18T07:52:00.206514+0000","last_became_active":"2024-09-18T07:51:45.206543+0000","last_became_peered":"2024-09-18T07:51:45.206543+0000","last_unstale":"2024-09-18T07:52:00.206514+0000","last_undegraded":"2024-09-18T07:52:00.206514+0000","last_fullsized":"2024-09-18T07:52:00.206514+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:47.388600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:47.388600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:02.712629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"568'21","reported_seq":1141,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274114+0000","last_change":"2024-09-18T07:51:22.852512+0000","last_active":"2024-09-18T07:52:05.274114+0000","last_peered":"2024-09-18T07:52:05.274114+0000","last_clean":"2024-09-18T07:52:05.274114+0000","last_became_active":"2024-09-18T07:51:09.220573+0000","last_became_peered":"2024-09-18T07:51:09.220573+0000","last_unstale":"2024-09-18T07:52:05.274114+0000","last_undegraded":"2024-09-18T07:52:05.274114+0000","last_fullsized":"2024-09-18T07:52:05.274114+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'21","last_scrub_stamp":"2024-09-18T07:51:22.852453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:22.852453+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:42.365900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1015,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.206088+0000","last_change":"2024-09-18T07:51:09.215982+0000","last_active":"2024-09-18T07:52:00.206088+0000","last_peered":"2024-09-18T07:52:00.206088+0000","last_clean":"2024-09-18T07:52:00.206088+0000","last_became_active":"2024-09-18T07:51:09.215682+0000","last_became_peered":"2024-09-18T07:51:09.215682+0000","last_unstale":"2024-09-18T07:52:00.206088+0000","last_undegraded":"2024-09-18T07:52:00.206088+0000","last_fullsized":"2024-09-18T07:52:00.206088+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:46.629242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:50:46.629242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:04.033793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1129,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.204037+0000","last_change":"2024-09-18T07:51:16.704038+0000","last_active":"2024-09-18T07:52:00.204037+0000","last_peered":"2024-09-18T07:52:00.204037+0000","last_clean":"2024-09-18T07:52:00.204037+0000","last_became_active":"2024-09-18T07:51:15.237333+0000","last_became_peered":"2024-09-18T07:51:15.237333+0000","last_unstale":"2024-09-18T07:52:00.204037+0000","last_undegraded":"2024-09-18T07:52:00.204037+0000","last_fullsized":"2024-09-18T07:52:00.204037+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:16.703992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:16.703992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:19.370569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":922,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.206347+0000","last_change":"2024-09-18T07:51:51.490569+0000","last_active":"2024-09-18T07:52:00.206347+0000","last_peered":"2024-09-18T07:52:00.206347+0000","last_clean":"2024-09-18T07:52:00.206347+0000","last_became_active":"2024-09-18T07:51:45.207136+0000","last_became_peered":"2024-09-18T07:51:45.207136+0000","last_unstale":"2024-09-18T07:52:00.206347+0000","last_undegraded":"2024-09-18T07:52:00.206347+0000","last_fullsized":"2024-09-18T07:52:00.206347+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:51.490504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:51.490504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:59.862474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"568'9","reported_seq":1173,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274339+0000","last_change":"2024-09-18T07:51:43.726184+0000","last_active":"2024-09-18T07:52:05.274339+0000","last_peered":"2024-09-18T07:52:05.274339+0000","last_clean":"2024-09-18T07:52:05.274339+0000","last_became_active":"2024-09-18T07:50:23.276419+0000","last_became_peered":"2024-09-18T07:50:23.276419+0000","last_unstale":"2024-09-18T07:52:05.274339+0000","last_undegraded":"2024-09-18T07:52:05.274339+0000","last_fullsized":"2024-09-18T07:52:05.274339+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'9","last_scrub_stamp":"2024-09-18T07:51:43.726114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:43.726114+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:50.468711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":933,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590676+0000","last_change":"2024-09-18T07:51:23.552640+0000","last_active":"2024-09-18T07:52:05.590676+0000","last_peered":"2024-09-18T07:52:05.590676+0000","last_clean":"2024-09-18T07:52:05.590676+0000","last_became_active":"2024-09-18T07:47:20.142891+0000","last_became_peered":"2024-09-18T07:47:20.142891+0000","last_unstale":"2024-09-18T07:52:05.590676+0000","last_undegraded":"2024-09-18T07:52:05.590676+0000","last_fullsized":"2024-09-18T07:52:05.590676+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:23.552591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:23.552591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:39.476156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1066,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.205872+0000","last_change":"2024-09-18T07:51:43.372570+0000","last_active":"2024-09-18T07:52:00.205872+0000","last_peered":"2024-09-18T07:52:00.205872+0000","last_clean":"2024-09-18T07:52:00.205872+0000","last_became_active":"2024-09-18T07:50:23.271118+0000","last_became_peered":"2024-09-18T07:50:23.271118+0000","last_unstale":"2024-09-18T07:52:00.205872+0000","last_undegraded":"2024-09-18T07:52:00.205872+0000","last_fullsized":"2024-09-18T07:52:00.205872+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:43.372521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:43.372521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:06.391031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1092,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274752+0000","last_change":"2024-09-18T07:51:59.823768+0000","last_active":"2024-09-18T07:52:05.274752+0000","last_peered":"2024-09-18T07:52:05.274752+0000","last_clean":"2024-09-18T07:52:05.274752+0000","last_became_active":"2024-09-18T07:47:11.187916+0000","last_became_peered":"2024-09-18T07:47:11.187916+0000","last_unstale":"2024-09-18T07:52:05.274752+0000","last_undegraded":"2024-09-18T07:52:05.274752+0000","last_fullsized":"2024-09-18T07:52:05.274752+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:59.823701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:59.823701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:11.641203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":76,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.205059+0000","last_change":"2024-09-18T07:51:12.629631+0000","last_active":"2024-09-18T07:52:00.205059+0000","last_peered":"2024-09-18T07:52:00.205059+0000","last_clean":"2024-09-18T07:52:00.205059+0000","last_became_active":"2024-09-18T07:51:09.224720+0000","last_became_peered":"2024-09-18T07:51:09.224720+0000","last_unstale":"2024-09-18T07:52:00.205059+0000","last_undegraded":"2024-09-18T07:52:00.205059+0000","last_fullsized":"2024-09-18T07:52:00.205059+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:12.629579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:12.629579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:13.778419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"568'39","reported_seq":1357,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590014+0000","last_change":"2024-09-18T07:51:19.451785+0000","last_active":"2024-09-18T07:52:05.590014+0000","last_peered":"2024-09-18T07:52:05.590014+0000","last_clean":"2024-09-18T07:52:05.590014+0000","last_became_active":"2024-09-18T07:50:23.296232+0000","last_became_peered":"2024-09-18T07:50:23.296232+0000","last_unstale":"2024-09-18T07:52:05.590014+0000","last_undegraded":"2024-09-18T07:52:05.590014+0000","last_fullsized":"2024-09-18T07:52:05.590014+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'39","last_scrub_stamp":"2024-09-18T07:51:19.451724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:19.451724+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:34.161935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":734,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272225+0000","last_change":"2024-09-18T07:52:03.801574+0000","last_active":"2024-09-18T07:52:05.272225+0000","last_peered":"2024-09-18T07:52:05.272225+0000","last_clean":"2024-09-18T07:52:05.272225+0000","last_became_active":"2024-09-18T07:51:19.263415+0000","last_became_peered":"2024-09-18T07:51:19.263415+0000","last_unstale":"2024-09-18T07:52:05.272225+0000","last_undegraded":"2024-09-18T07:52:05.272225+0000","last_fullsized":"2024-09-18T07:52:05.272225+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:52:03.801509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:52:03.801509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:21.652578+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":953,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272726+0000","last_change":"2024-09-18T07:51:27.530215+0000","last_active":"2024-09-18T07:52:05.272726+0000","last_peered":"2024-09-18T07:52:05.272726+0000","last_clean":"2024-09-18T07:52:05.272726+0000","last_became_active":"2024-09-18T07:51:27.529908+0000","last_became_peered":"2024-09-18T07:51:27.529908+0000","last_unstale":"2024-09-18T07:52:05.272726+0000","last_undegraded":"2024-09-18T07:52:05.272726+0000","last_fullsized":"2024-09-18T07:52:05.272726+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:50.672909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:50:50.672909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:09.260524+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1083,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274324+0000","last_change":"2024-09-18T07:51:21.826034+0000","last_active":"2024-09-18T07:52:05.274324+0000","last_peered":"2024-09-18T07:52:05.274324+0000","last_clean":"2024-09-18T07:52:05.274324+0000","last_became_active":"2024-09-18T07:47:20.144620+0000","last_became_peered":"2024-09-18T07:47:20.144620+0000","last_unstale":"2024-09-18T07:52:05.274324+0000","last_undegraded":"2024-09-18T07:52:05.274324+0000","last_fullsized":"2024-09-18T07:52:05.274324+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:21.825986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:21.825986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:37.969195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"568'24","reported_seq":451,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272409+0000","last_change":"2024-09-18T07:51:27.297308+0000","last_active":"2024-09-18T07:52:05.272409+0000","last_peered":"2024-09-18T07:52:05.272409+0000","last_clean":"2024-09-18T07:52:05.272409+0000","last_became_active":"2024-09-18T07:51:27.296261+0000","last_became_peered":"2024-09-18T07:51:27.296261+0000","last_unstale":"2024-09-18T07:52:05.272409+0000","last_undegraded":"2024-09-18T07:52:05.272409+0000","last_fullsized":"2024-09-18T07:52:05.272409+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'24","last_scrub_stamp":"2024-09-18T07:51:24.882921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:24.882921+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:46.460599+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":31,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1011,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274513+0000","last_change":"2024-09-18T07:51:27.898744+0000","last_active":"2024-09-18T07:52:05.274513+0000","last_peered":"2024-09-18T07:52:05.274513+0000","last_clean":"2024-09-18T07:52:05.274513+0000","last_became_active":"2024-09-18T07:50:23.271208+0000","last_became_peered":"2024-09-18T07:50:23.271208+0000","last_unstale":"2024-09-18T07:52:05.274513+0000","last_undegraded":"2024-09-18T07:52:05.274513+0000","last_fullsized":"2024-09-18T07:52:05.274513+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:27.898672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:27.898672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:44.646916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":931,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590915+0000","last_change":"2024-09-18T07:51:16.577134+0000","last_active":"2024-09-18T07:52:05.590915+0000","last_peered":"2024-09-18T07:52:05.590915+0000","last_clean":"2024-09-18T07:52:05.590915+0000","last_became_active":"2024-09-18T07:47:11.194447+0000","last_became_peered":"2024-09-18T07:47:11.194447+0000","last_unstale":"2024-09-18T07:52:05.590915+0000","last_undegraded":"2024-09-18T07:52:05.590915+0000","last_fullsized":"2024-09-18T07:52:05.590915+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:51:16.577081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:16.577081+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:30.192910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":645,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.270697+0000","last_change":"2024-09-18T07:52:00.793949+0000","last_active":"2024-09-18T07:52:05.270697+0000","last_peered":"2024-09-18T07:52:05.270697+0000","last_clean":"2024-09-18T07:52:05.270697+0000","last_became_active":"2024-09-18T07:51:19.258923+0000","last_became_peered":"2024-09-18T07:51:19.258923+0000","last_unstale":"2024-09-18T07:52:05.270697+0000","last_undegraded":"2024-09-18T07:52:05.270697+0000","last_fullsized":"2024-09-18T07:52:05.270697+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:52:00.793895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:52:00.793895+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:10.220383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"568'30","reported_seq":963,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.204189+0000","last_change":"2024-09-18T07:51:25.208580+0000","last_active":"2024-09-18T07:52:00.204189+0000","last_peered":"2024-09-18T07:52:00.204189+0000","last_clean":"2024-09-18T07:52:00.204189+0000","last_became_active":"2024-09-18T07:51:25.201124+0000","last_became_peered":"2024-09-18T07:51:25.201124+0000","last_unstale":"2024-09-18T07:52:00.204189+0000","last_undegraded":"2024-09-18T07:52:00.204189+0000","last_fullsized":"2024-09-18T07:52:00.204189+0000","mapping_epoch":580,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":581,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'30","last_scrub_stamp":"2024-09-18T07:51:19.775996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:19.775996+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:36.992772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"398'14","reported_seq":966,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591819+0000","last_change":"2024-09-18T07:51:07.654295+0000","last_active":"2024-09-18T07:52:05.591819+0000","last_peered":"2024-09-18T07:52:05.591819+0000","last_clean":"2024-09-18T07:52:05.591819+0000","last_became_active":"2024-09-18T07:44:43.786594+0000","last_became_peered":"2024-09-18T07:44:43.786594+0000","last_unstale":"2024-09-18T07:52:05.591819+0000","last_undegraded":"2024-09-18T07:52:05.591819+0000","last_fullsized":"2024-09-18T07:52:05.591819+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:51:07.654244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:07.654244+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:28.946936+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":461,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.206130+0000","last_change":"2024-09-18T07:51:45.209776+0000","last_active":"2024-09-18T07:52:00.206130+0000","last_peered":"2024-09-18T07:52:00.206130+0000","last_clean":"2024-09-18T07:52:00.206130+0000","last_became_active":"2024-09-18T07:51:45.209417+0000","last_became_peered":"2024-09-18T07:51:45.209417+0000","last_unstale":"2024-09-18T07:52:00.206130+0000","last_undegraded":"2024-09-18T07:52:00.206130+0000","last_fullsized":"2024-09-18T07:52:00.206130+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:51:24.460975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:51:24.460975+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:42.624370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"568'39","reported_seq":156,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271488+0000","last_change":"2024-09-18T07:51:19.890800+0000","last_active":"2024-09-18T07:52:05.271488+0000","last_peered":"2024-09-18T07:52:05.271488+0000","last_clean":"2024-09-18T07:52:05.271488+0000","last_became_active":"2024-09-18T07:51:19.261732+0000","last_became_peered":"2024-09-18T07:51:19.261732+0000","last_unstale":"2024-09-18T07:52:05.271488+0000","last_undegraded":"2024-09-18T07:52:05.271488+0000","last_fullsized":"2024-09-18T07:52:05.271488+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'39","last_scrub_stamp":"2024-09-18T07:51:19.890746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:19.890746+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:24.913065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":129,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271434+0000","last_change":"2024-09-18T07:51:32.524378+0000","last_active":"2024-09-18T07:52:05.271434+0000","last_peered":"2024-09-18T07:52:05.271434+0000","last_clean":"2024-09-18T07:52:05.271434+0000","last_became_active":"2024-09-18T07:50:23.298465+0000","last_became_peered":"2024-09-18T07:50:23.298465+0000","last_unstale":"2024-09-18T07:52:05.271434+0000","last_undegraded":"2024-09-18T07:52:05.271434+0000","last_fullsized":"2024-09-18T07:52:05.271434+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:32.524333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:32.524333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:50.995593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"571'14","reported_seq":1286,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274817+0000","last_change":"2024-09-18T07:51:53.776692+0000","last_active":"2024-09-18T07:52:05.274817+0000","last_peered":"2024-09-18T07:52:05.274817+0000","last_clean":"2024-09-18T07:52:05.274817+0000","last_became_active":"2024-09-18T07:51:09.215979+0000","last_became_peered":"2024-09-18T07:51:09.215979+0000","last_unstale":"2024-09-18T07:52:05.274817+0000","last_undegraded":"2024-09-18T07:52:05.274817+0000","last_fullsized":"2024-09-18T07:52:05.274817+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"571'14","last_scrub_stamp":"2024-09-18T07:51:53.776625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:53.776625+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:13.321194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":917,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.206713+0000","last_change":"2024-09-18T07:51:45.209609+0000","last_active":"2024-09-18T07:52:00.206713+0000","last_peered":"2024-09-18T07:52:00.206713+0000","last_clean":"2024-09-18T07:52:00.206713+0000","last_became_active":"2024-09-18T07:51:45.209222+0000","last_became_peered":"2024-09-18T07:51:45.209222+0000","last_unstale":"2024-09-18T07:52:00.206713+0000","last_undegraded":"2024-09-18T07:52:00.206713+0000","last_fullsized":"2024-09-18T07:52:00.206713+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:37.831287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:37.831287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:00.161728+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":620,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274749+0000","last_change":"2024-09-18T07:52:00.834631+0000","last_active":"2024-09-18T07:52:05.274749+0000","last_peered":"2024-09-18T07:52:05.274749+0000","last_clean":"2024-09-18T07:52:05.274749+0000","last_became_active":"2024-09-18T07:44:43.786396+0000","last_became_peered":"2024-09-18T07:44:43.786396+0000","last_unstale":"2024-09-18T07:52:05.274749+0000","last_undegraded":"2024-09-18T07:52:05.274749+0000","last_fullsized":"2024-09-18T07:52:05.274749+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:52:00.834568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:52:00.834568+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:29.617773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"568'42","reported_seq":190,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271786+0000","last_change":"2024-09-18T07:51:15.596205+0000","last_active":"2024-09-18T07:52:05.271786+0000","last_peered":"2024-09-18T07:52:05.271786+0000","last_clean":"2024-09-18T07:52:05.271786+0000","last_became_active":"2024-09-18T07:51:09.235289+0000","last_became_peered":"2024-09-18T07:51:09.235289+0000","last_unstale":"2024-09-18T07:52:05.271786+0000","last_undegraded":"2024-09-18T07:52:05.271786+0000","last_fullsized":"2024-09-18T07:52:05.271786+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'42","last_scrub_stamp":"2024-09-18T07:51:15.596152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:15.596152+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:25.888748+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":870,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272731+0000","last_change":"2024-09-18T07:51:27.295782+0000","last_active":"2024-09-18T07:52:05.272731+0000","last_peered":"2024-09-18T07:52:05.272731+0000","last_clean":"2024-09-18T07:52:05.272731+0000","last_became_active":"2024-09-18T07:51:27.295491+0000","last_became_peered":"2024-09-18T07:51:27.295491+0000","last_unstale":"2024-09-18T07:52:05.272731+0000","last_undegraded":"2024-09-18T07:52:05.272731+0000","last_fullsized":"2024-09-18T07:52:05.272731+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:24.552207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:24.552207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:49.695764+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1078,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274075+0000","last_change":"2024-09-18T07:51:50.820516+0000","last_active":"2024-09-18T07:52:05.274075+0000","last_peered":"2024-09-18T07:52:05.274075+0000","last_clean":"2024-09-18T07:52:05.274075+0000","last_became_active":"2024-09-18T07:50:23.271640+0000","last_became_peered":"2024-09-18T07:50:23.271640+0000","last_unstale":"2024-09-18T07:52:05.274075+0000","last_undegraded":"2024-09-18T07:52:05.274075+0000","last_fullsized":"2024-09-18T07:52:05.274075+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:50.820446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:50.820446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:03.485644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":980,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274128+0000","last_change":"2024-09-18T07:50:47.845279+0000","last_active":"2024-09-18T07:52:05.274128+0000","last_peered":"2024-09-18T07:52:05.274128+0000","last_clean":"2024-09-18T07:52:05.274128+0000","last_became_active":"2024-09-18T07:50:23.279337+0000","last_became_peered":"2024-09-18T07:50:23.279337+0000","last_unstale":"2024-09-18T07:52:05.274128+0000","last_undegraded":"2024-09-18T07:52:05.274128+0000","last_fullsized":"2024-09-18T07:52:05.274128+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:47.845219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:50:47.845219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:10.259502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":133,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272211+0000","last_change":"2024-09-18T07:51:39.469646+0000","last_active":"2024-09-18T07:52:05.272211+0000","last_peered":"2024-09-18T07:52:05.272211+0000","last_clean":"2024-09-18T07:52:05.272211+0000","last_became_active":"2024-09-18T07:50:23.276444+0000","last_became_peered":"2024-09-18T07:50:23.276444+0000","last_unstale":"2024-09-18T07:52:05.272211+0000","last_undegraded":"2024-09-18T07:52:05.272211+0000","last_fullsized":"2024-09-18T07:52:05.272211+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:51:39.469567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:51:39.469567+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:01.652053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"568'42","reported_seq":1113,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274379+0000","last_change":"2024-09-18T07:51:54.765844+0000","last_active":"2024-09-18T07:52:05.274379+0000","last_peered":"2024-09-18T07:52:05.274379+0000","last_clean":"2024-09-18T07:52:05.274379+0000","last_became_active":"2024-09-18T07:47:11.188002+0000","last_became_peered":"2024-09-18T07:47:11.188002+0000","last_unstale":"2024-09-18T07:52:05.274379+0000","last_undegraded":"2024-09-18T07:52:05.274379+0000","last_fullsized":"2024-09-18T07:52:05.274379+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'42","last_scrub_stamp":"2024-09-18T07:51:54.765772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:54.765772+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:15.164539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":80,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.204475+0000","last_change":"2024-09-18T07:51:28.854565+0000","last_active":"2024-09-18T07:52:00.204475+0000","last_peered":"2024-09-18T07:52:00.204475+0000","last_clean":"2024-09-18T07:52:00.204475+0000","last_became_active":"2024-09-18T07:51:09.231383+0000","last_became_peered":"2024-09-18T07:51:09.231383+0000","last_unstale":"2024-09-18T07:52:00.204475+0000","last_undegraded":"2024-09-18T07:52:00.204475+0000","last_fullsized":"2024-09-18T07:52:00.204475+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:51:28.854514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:28.854514+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:44.856258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"548'15","reported_seq":154,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271689+0000","last_change":"2024-09-18T07:51:20.565974+0000","last_active":"2024-09-18T07:52:05.271689+0000","last_peered":"2024-09-18T07:52:05.271689+0000","last_clean":"2024-09-18T07:52:05.271689+0000","last_became_active":"2024-09-18T07:50:23.300057+0000","last_became_peered":"2024-09-18T07:50:23.300057+0000","last_unstale":"2024-09-18T07:52:05.271689+0000","last_undegraded":"2024-09-18T07:52:05.271689+0000","last_fullsized":"2024-09-18T07:52:05.271689+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'15","last_scrub_stamp":"2024-09-18T07:51:20.565926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:20.565926+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:49.557826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":932,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591907+0000","last_change":"2024-09-18T07:52:04.367414+0000","last_active":"2024-09-18T07:52:05.591907+0000","last_peered":"2024-09-18T07:52:05.591907+0000","last_clean":"2024-09-18T07:52:05.591907+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:52:05.591907+0000","last_undegraded":"2024-09-18T07:52:05.591907+0000","last_fullsized":"2024-09-18T07:52:05.591907+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:52:04.367352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:52:04.367352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:06.642826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":881,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591600+0000","last_change":"2024-09-18T07:51:25.200415+0000","last_active":"2024-09-18T07:52:05.591600+0000","last_peered":"2024-09-18T07:52:05.591600+0000","last_clean":"2024-09-18T07:52:05.591600+0000","last_became_active":"2024-09-18T07:51:25.200028+0000","last_became_peered":"2024-09-18T07:51:25.200028+0000","last_unstale":"2024-09-18T07:52:05.591600+0000","last_undegraded":"2024-09-18T07:52:05.591600+0000","last_fullsized":"2024-09-18T07:52:05.591600+0000","mapping_epoch":580,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":581,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:50:48.664637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:50:48.664637+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:14.535564+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"568'30","reported_seq":846,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.270726+0000","last_change":"2024-09-18T07:51:28.936119+0000","last_active":"2024-09-18T07:52:05.270726+0000","last_peered":"2024-09-18T07:52:05.270726+0000","last_clean":"2024-09-18T07:52:05.270726+0000","last_became_active":"2024-09-18T07:51:19.260618+0000","last_became_peered":"2024-09-18T07:51:19.260618+0000","last_unstale":"2024-09-18T07:52:05.270726+0000","last_undegraded":"2024-09-18T07:52:05.270726+0000","last_fullsized":"2024-09-18T07:52:05.270726+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'30","last_scrub_stamp":"2024-09-18T07:51:28.936068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:28.936068+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:38.606541+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":37,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1055,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.206410+0000","last_change":"2024-09-18T07:51:46.357562+0000","last_active":"2024-09-18T07:52:00.206410+0000","last_peered":"2024-09-18T07:52:00.206410+0000","last_clean":"2024-09-18T07:52:00.206410+0000","last_became_active":"2024-09-18T07:50:23.268446+0000","last_became_peered":"2024-09-18T07:50:23.268446+0000","last_unstale":"2024-09-18T07:52:00.206410+0000","last_undegraded":"2024-09-18T07:52:00.206410+0000","last_fullsized":"2024-09-18T07:52:00.206410+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:46.357499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:46.357499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:14.017075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"575'17","reported_seq":928,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.204254+0000","last_change":"2024-09-18T07:51:20.747209+0000","last_active":"2024-09-18T07:52:00.204254+0000","last_peered":"2024-09-18T07:52:00.204254+0000","last_clean":"2024-09-18T07:52:00.204254+0000","last_became_active":"2024-09-18T07:51:15.235642+0000","last_became_peered":"2024-09-18T07:51:15.235642+0000","last_unstale":"2024-09-18T07:52:00.204254+0000","last_undegraded":"2024-09-18T07:52:00.204254+0000","last_fullsized":"2024-09-18T07:52:00.204254+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'17","last_scrub_stamp":"2024-09-18T07:51:20.747157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:20.747157+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:38.859938+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":123,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.204263+0000","last_change":"2024-09-18T07:51:15.236408+0000","last_active":"2024-09-18T07:52:00.204263+0000","last_peered":"2024-09-18T07:52:00.204263+0000","last_clean":"2024-09-18T07:52:00.204263+0000","last_became_active":"2024-09-18T07:51:15.235148+0000","last_became_peered":"2024-09-18T07:51:15.235148+0000","last_unstale":"2024-09-18T07:52:00.204263+0000","last_undegraded":"2024-09-18T07:52:00.204263+0000","last_fullsized":"2024-09-18T07:52:00.204263+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:09.666223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:09.666223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:34.536632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":903,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591296+0000","last_change":"2024-09-18T07:51:09.233169+0000","last_active":"2024-09-18T07:52:05.591296+0000","last_peered":"2024-09-18T07:52:05.591296+0000","last_clean":"2024-09-18T07:52:05.591296+0000","last_became_active":"2024-09-18T07:51:09.232811+0000","last_became_peered":"2024-09-18T07:51:09.232811+0000","last_unstale":"2024-09-18T07:52:05.591296+0000","last_undegraded":"2024-09-18T07:52:05.591296+0000","last_fullsized":"2024-09-18T07:52:05.591296+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:50:54.661812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:50:54.661812+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:23.439730+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"568'39","reported_seq":1015,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.204299+0000","last_change":"2024-09-18T07:51:38.873869+0000","last_active":"2024-09-18T07:52:00.204299+0000","last_peered":"2024-09-18T07:52:00.204299+0000","last_clean":"2024-09-18T07:52:00.204299+0000","last_became_active":"2024-09-18T07:51:15.234676+0000","last_became_peered":"2024-09-18T07:51:15.234676+0000","last_unstale":"2024-09-18T07:52:00.204299+0000","last_undegraded":"2024-09-18T07:52:00.204299+0000","last_fullsized":"2024-09-18T07:52:00.204299+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'39","last_scrub_stamp":"2024-09-18T07:51:38.873813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:38.873813+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:50.939603+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":632,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271213+0000","last_change":"2024-09-18T07:51:19.258009+0000","last_active":"2024-09-18T07:52:05.271213+0000","last_peered":"2024-09-18T07:52:05.271213+0000","last_clean":"2024-09-18T07:52:05.271213+0000","last_became_active":"2024-09-18T07:51:19.257463+0000","last_became_peered":"2024-09-18T07:51:19.257463+0000","last_unstale":"2024-09-18T07:52:05.271213+0000","last_undegraded":"2024-09-18T07:52:05.271213+0000","last_fullsized":"2024-09-18T07:52:05.271213+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:03.448570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:03.448570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:17.333594+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":809,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592495+0000","last_change":"2024-09-18T07:51:55.536453+0000","last_active":"2024-09-18T07:52:05.592495+0000","last_peered":"2024-09-18T07:52:05.592495+0000","last_clean":"2024-09-18T07:52:05.592495+0000","last_became_active":"2024-09-18T07:51:09.231928+0000","last_became_peered":"2024-09-18T07:51:09.231928+0000","last_unstale":"2024-09-18T07:52:05.592495+0000","last_undegraded":"2024-09-18T07:52:05.592495+0000","last_fullsized":"2024-09-18T07:52:05.592495+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:55.536393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:55.536393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:04.636419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":554,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.270587+0000","last_change":"2024-09-18T07:51:19.255681+0000","last_active":"2024-09-18T07:52:05.270587+0000","last_peered":"2024-09-18T07:52:05.270587+0000","last_clean":"2024-09-18T07:52:05.270587+0000","last_became_active":"2024-09-18T07:51:19.255390+0000","last_became_peered":"2024-09-18T07:51:19.255390+0000","last_unstale":"2024-09-18T07:52:05.270587+0000","last_undegraded":"2024-09-18T07:52:05.270587+0000","last_fullsized":"2024-09-18T07:52:05.270587+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:08.633382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:08.633382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:36.058930+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":504,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.205988+0000","last_change":"2024-09-18T07:51:55.456646+0000","last_active":"2024-09-18T07:52:00.205988+0000","last_peered":"2024-09-18T07:52:00.205988+0000","last_clean":"2024-09-18T07:52:00.205988+0000","last_became_active":"2024-09-18T07:51:45.204912+0000","last_became_peered":"2024-09-18T07:51:45.204912+0000","last_unstale":"2024-09-18T07:52:00.205988+0000","last_undegraded":"2024-09-18T07:52:00.205988+0000","last_fullsized":"2024-09-18T07:52:00.205988+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:51:55.456551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:51:55.456551+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:56.445886+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"578'59","reported_seq":1286,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271107+0000","last_change":"2024-09-18T07:51:34.854066+0000","last_active":"2024-09-18T07:52:05.271107+0000","last_peered":"2024-09-18T07:52:05.271107+0000","last_clean":"2024-09-18T07:52:05.271107+0000","last_became_active":"2024-09-18T07:51:19.255807+0000","last_became_peered":"2024-09-18T07:51:19.255807+0000","last_unstale":"2024-09-18T07:52:05.271107+0000","last_undegraded":"2024-09-18T07:52:05.271107+0000","last_fullsized":"2024-09-18T07:52:05.271107+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'59","last_scrub_stamp":"2024-09-18T07:51:34.853966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:34.853966+0000","objects_scrubbed":10,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:02.277729+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":82,"num_read_kb":67,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":1790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1077,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274472+0000","last_change":"2024-09-18T07:51:30.905155+0000","last_active":"2024-09-18T07:52:05.274472+0000","last_peered":"2024-09-18T07:52:05.274472+0000","last_clean":"2024-09-18T07:52:05.274472+0000","last_became_active":"2024-09-18T07:51:09.216307+0000","last_became_peered":"2024-09-18T07:51:09.216307+0000","last_unstale":"2024-09-18T07:52:05.274472+0000","last_undegraded":"2024-09-18T07:52:05.274472+0000","last_fullsized":"2024-09-18T07:52:05.274472+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:30.905100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:30.905100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:59.784123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":924,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.206119+0000","last_change":"2024-09-18T07:51:45.208985+0000","last_active":"2024-09-18T07:52:00.206119+0000","last_peered":"2024-09-18T07:52:00.206119+0000","last_clean":"2024-09-18T07:52:00.206119+0000","last_became_active":"2024-09-18T07:51:45.208101+0000","last_became_peered":"2024-09-18T07:51:45.208101+0000","last_unstale":"2024-09-18T07:52:00.206119+0000","last_undegraded":"2024-09-18T07:52:00.206119+0000","last_fullsized":"2024-09-18T07:52:00.206119+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:50:53.662326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:50:53.662326+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:13.723996+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":68,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.204006+0000","last_change":"2024-09-18T07:51:09.217314+0000","last_active":"2024-09-18T07:52:00.204006+0000","last_peered":"2024-09-18T07:52:00.204006+0000","last_clean":"2024-09-18T07:52:00.204006+0000","last_became_active":"2024-09-18T07:51:09.217019+0000","last_became_peered":"2024-09-18T07:51:09.217019+0000","last_unstale":"2024-09-18T07:52:00.204006+0000","last_undegraded":"2024-09-18T07:52:00.204006+0000","last_fullsized":"2024-09-18T07:52:00.204006+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:02.936812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:02.936812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:16.894474+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":129,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272712+0000","last_change":"2024-09-18T07:51:33.531280+0000","last_active":"2024-09-18T07:52:05.272712+0000","last_peered":"2024-09-18T07:52:05.272712+0000","last_clean":"2024-09-18T07:52:05.272712+0000","last_became_active":"2024-09-18T07:50:23.286499+0000","last_became_peered":"2024-09-18T07:50:23.286499+0000","last_unstale":"2024-09-18T07:52:05.272712+0000","last_undegraded":"2024-09-18T07:52:05.272712+0000","last_fullsized":"2024-09-18T07:52:05.272712+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:33.531237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:33.531237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:50.397177+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":1000,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590344+0000","last_change":"2024-09-18T07:51:30.424756+0000","last_active":"2024-09-18T07:52:05.590344+0000","last_peered":"2024-09-18T07:52:05.590344+0000","last_clean":"2024-09-18T07:52:05.590344+0000","last_became_active":"2024-09-18T07:51:09.218702+0000","last_became_peered":"2024-09-18T07:51:09.218702+0000","last_unstale":"2024-09-18T07:52:05.590344+0000","last_undegraded":"2024-09-18T07:52:05.590344+0000","last_fullsized":"2024-09-18T07:52:05.590344+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:51:30.424697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:51:30.424697+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:44.306670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"568'21","reported_seq":890,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271203+0000","last_change":"2024-09-18T07:51:19.264838+0000","last_active":"2024-09-18T07:52:05.271203+0000","last_peered":"2024-09-18T07:52:05.271203+0000","last_clean":"2024-09-18T07:52:05.271203+0000","last_became_active":"2024-09-18T07:51:19.263668+0000","last_became_peered":"2024-09-18T07:51:19.263668+0000","last_unstale":"2024-09-18T07:52:05.271203+0000","last_undegraded":"2024-09-18T07:52:05.271203+0000","last_fullsized":"2024-09-18T07:52:05.271203+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'21","last_scrub_stamp":"2024-09-18T07:51:12.053730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:12.053730+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:39.962516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":86,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590164+0000","last_change":"2024-09-18T07:51:11.569922+0000","last_active":"2024-09-18T07:52:05.590164+0000","last_peered":"2024-09-18T07:52:05.590164+0000","last_clean":"2024-09-18T07:52:05.590164+0000","last_became_active":"2024-09-18T07:51:01.009728+0000","last_became_peered":"2024-09-18T07:51:01.009728+0000","last_unstale":"2024-09-18T07:52:05.590164+0000","last_undegraded":"2024-09-18T07:52:05.590164+0000","last_fullsized":"2024-09-18T07:52:05.590164+0000","mapping_epoch":566,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":567,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:11.569873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:11.569873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:20.836170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":1209,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.589710+0000","last_change":"2024-09-18T07:51:13.550053+0000","last_active":"2024-09-18T07:52:05.589710+0000","last_peered":"2024-09-18T07:52:05.589710+0000","last_clean":"2024-09-18T07:52:05.589710+0000","last_became_active":"2024-09-18T07:51:09.239633+0000","last_became_peered":"2024-09-18T07:51:09.239633+0000","last_unstale":"2024-09-18T07:52:05.589710+0000","last_undegraded":"2024-09-18T07:52:05.589710+0000","last_fullsized":"2024-09-18T07:52:05.589710+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:51:13.550001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:51:13.550001+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:39.469350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1164,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.589958+0000","last_change":"2024-09-18T07:50:54.371523+0000","last_active":"2024-09-18T07:52:05.589958+0000","last_peered":"2024-09-18T07:52:05.589958+0000","last_clean":"2024-09-18T07:52:05.589958+0000","last_became_active":"2024-09-18T07:47:11.201379+0000","last_became_peered":"2024-09-18T07:47:11.201379+0000","last_unstale":"2024-09-18T07:52:05.589958+0000","last_undegraded":"2024-09-18T07:52:05.589958+0000","last_fullsized":"2024-09-18T07:52:05.589958+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:54.371475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:50:54.371475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:21.594190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":964,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590962+0000","last_change":"2024-09-18T07:52:00.316365+0000","last_active":"2024-09-18T07:52:05.590962+0000","last_peered":"2024-09-18T07:52:05.590962+0000","last_clean":"2024-09-18T07:52:05.590962+0000","last_became_active":"2024-09-18T07:50:23.296775+0000","last_became_peered":"2024-09-18T07:50:23.296775+0000","last_unstale":"2024-09-18T07:52:05.590962+0000","last_undegraded":"2024-09-18T07:52:05.590962+0000","last_fullsized":"2024-09-18T07:52:05.590962+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:52:00.316303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:52:00.316303+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:08.941866+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":91,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":667,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271235+0000","last_change":"2024-09-18T07:51:19.262419+0000","last_active":"2024-09-18T07:52:05.271235+0000","last_peered":"2024-09-18T07:52:05.271235+0000","last_clean":"2024-09-18T07:52:05.271235+0000","last_became_active":"2024-09-18T07:51:19.262081+0000","last_became_peered":"2024-09-18T07:51:19.262081+0000","last_unstale":"2024-09-18T07:52:05.271235+0000","last_undegraded":"2024-09-18T07:52:05.271235+0000","last_fullsized":"2024-09-18T07:52:05.271235+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:51:05.654368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:51:05.654368+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:30.880142+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"568'36","reported_seq":90,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.204530+0000","last_change":"2024-09-18T07:51:37.851879+0000","last_active":"2024-09-18T07:52:00.204530+0000","last_peered":"2024-09-18T07:52:00.204530+0000","last_clean":"2024-09-18T07:52:00.204530+0000","last_became_active":"2024-09-18T07:51:09.237471+0000","last_became_peered":"2024-09-18T07:51:09.237471+0000","last_unstale":"2024-09-18T07:52:00.204530+0000","last_undegraded":"2024-09-18T07:52:00.204530+0000","last_fullsized":"2024-09-18T07:52:00.204530+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'36","last_scrub_stamp":"2024-09-18T07:51:37.851824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:37.851824+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:50.507555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":91,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.203908+0000","last_change":"2024-09-18T07:51:41.811345+0000","last_active":"2024-09-18T07:52:00.203908+0000","last_peered":"2024-09-18T07:52:00.203908+0000","last_clean":"2024-09-18T07:52:00.203908+0000","last_became_active":"2024-09-18T07:51:25.199645+0000","last_became_peered":"2024-09-18T07:51:25.199645+0000","last_unstale":"2024-09-18T07:52:00.203908+0000","last_undegraded":"2024-09-18T07:52:00.203908+0000","last_fullsized":"2024-09-18T07:52:00.203908+0000","mapping_epoch":580,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":581,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:51:41.811295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:41.811295+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:48.574640+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":1983,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591645+0000","last_change":"2024-09-18T07:52:03.328063+0000","last_active":"2024-09-18T07:52:05.591645+0000","last_peered":"2024-09-18T07:52:05.591645+0000","last_clean":"2024-09-18T07:52:05.591645+0000","last_became_active":"2024-09-18T07:51:09.231463+0000","last_became_peered":"2024-09-18T07:51:09.231463+0000","last_unstale":"2024-09-18T07:52:05.591645+0000","last_undegraded":"2024-09-18T07:52:05.591645+0000","last_fullsized":"2024-09-18T07:52:05.591645+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T07:52:03.327997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:52:03.327997+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:11.091518+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1017,"num_read_kb":1012,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2334,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1049,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.206884+0000","last_change":"2024-09-18T07:51:10.580404+0000","last_active":"2024-09-18T07:52:00.206884+0000","last_peered":"2024-09-18T07:52:00.206884+0000","last_clean":"2024-09-18T07:52:00.206884+0000","last_became_active":"2024-09-18T07:51:09.219290+0000","last_became_peered":"2024-09-18T07:51:09.219290+0000","last_unstale":"2024-09-18T07:52:00.206884+0000","last_undegraded":"2024-09-18T07:52:00.206884+0000","last_fullsized":"2024-09-18T07:52:00.206884+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:10.580357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:10.580357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:38.242597+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":655,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.270923+0000","last_change":"2024-09-18T07:51:19.257727+0000","last_active":"2024-09-18T07:52:05.270923+0000","last_peered":"2024-09-18T07:52:05.270923+0000","last_clean":"2024-09-18T07:52:05.270923+0000","last_became_active":"2024-09-18T07:51:19.257164+0000","last_became_peered":"2024-09-18T07:51:19.257164+0000","last_unstale":"2024-09-18T07:52:05.270923+0000","last_undegraded":"2024-09-18T07:52:05.270923+0000","last_fullsized":"2024-09-18T07:52:05.270923+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:06.543375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:06.543375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:16.657934+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"568'46","reported_seq":1016,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592557+0000","last_change":"2024-09-18T07:51:48.500801+0000","last_active":"2024-09-18T07:52:05.592557+0000","last_peered":"2024-09-18T07:52:05.592557+0000","last_clean":"2024-09-18T07:52:05.592557+0000","last_became_active":"2024-09-18T07:44:43.786827+0000","last_became_peered":"2024-09-18T07:44:43.786827+0000","last_unstale":"2024-09-18T07:52:05.592557+0000","last_undegraded":"2024-09-18T07:52:05.592557+0000","last_fullsized":"2024-09-18T07:52:05.592557+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'46","last_scrub_stamp":"2024-09-18T07:51:48.500713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:48.500713+0000","objects_scrubbed":8,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:05.288214+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":830,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272463+0000","last_change":"2024-09-18T07:51:27.530400+0000","last_active":"2024-09-18T07:52:05.272463+0000","last_peered":"2024-09-18T07:52:05.272463+0000","last_clean":"2024-09-18T07:52:05.272463+0000","last_became_active":"2024-09-18T07:51:27.530139+0000","last_became_peered":"2024-09-18T07:51:27.530139+0000","last_unstale":"2024-09-18T07:52:05.272463+0000","last_undegraded":"2024-09-18T07:52:05.272463+0000","last_fullsized":"2024-09-18T07:52:05.272463+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:39.627115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:50:39.627115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":983,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592286+0000","last_change":"2024-09-18T07:51:29.455174+0000","last_active":"2024-09-18T07:52:05.592286+0000","last_peered":"2024-09-18T07:52:05.592286+0000","last_clean":"2024-09-18T07:52:05.592286+0000","last_became_active":"2024-09-18T07:51:09.231028+0000","last_became_peered":"2024-09-18T07:51:09.231028+0000","last_unstale":"2024-09-18T07:52:05.592286+0000","last_undegraded":"2024-09-18T07:52:05.592286+0000","last_fullsized":"2024-09-18T07:52:05.592286+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:51:29.455113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:51:29.455113+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:49.429278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":921,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591657+0000","last_change":"2024-09-18T07:51:40.374754+0000","last_active":"2024-09-18T07:52:05.591657+0000","last_peered":"2024-09-18T07:52:05.591657+0000","last_clean":"2024-09-18T07:52:05.591657+0000","last_became_active":"2024-09-18T07:47:11.195084+0000","last_became_peered":"2024-09-18T07:47:11.195084+0000","last_unstale":"2024-09-18T07:52:05.591657+0000","last_undegraded":"2024-09-18T07:52:05.591657+0000","last_fullsized":"2024-09-18T07:52:05.591657+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:40.374696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:40.374696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:02.144236+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1038,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.206808+0000","last_change":"2024-09-18T07:51:48.405578+0000","last_active":"2024-09-18T07:52:00.206808+0000","last_peered":"2024-09-18T07:52:00.206808+0000","last_clean":"2024-09-18T07:52:00.206808+0000","last_became_active":"2024-09-18T07:50:23.274710+0000","last_became_peered":"2024-09-18T07:50:23.274710+0000","last_unstale":"2024-09-18T07:52:00.206808+0000","last_undegraded":"2024-09-18T07:52:00.206808+0000","last_fullsized":"2024-09-18T07:52:00.206808+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:48.405459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:48.405459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:56.151375+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"568'27","reported_seq":1293,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274853+0000","last_change":"2024-09-18T07:52:04.815187+0000","last_active":"2024-09-18T07:52:05.274853+0000","last_peered":"2024-09-18T07:52:05.274853+0000","last_clean":"2024-09-18T07:52:05.274853+0000","last_became_active":"2024-09-18T07:49:32.684857+0000","last_became_peered":"2024-09-18T07:49:32.684857+0000","last_unstale":"2024-09-18T07:52:05.274853+0000","last_undegraded":"2024-09-18T07:52:05.274853+0000","last_fullsized":"2024-09-18T07:52:05.274853+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'27","last_scrub_stamp":"2024-09-18T07:52:04.815087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:52:04.815087+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:30.795151+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1026,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274274+0000","last_change":"2024-09-18T07:51:26.866971+0000","last_active":"2024-09-18T07:52:05.274274+0000","last_peered":"2024-09-18T07:52:05.274274+0000","last_clean":"2024-09-18T07:52:05.274274+0000","last_became_active":"2024-09-18T07:44:43.790160+0000","last_became_peered":"2024-09-18T07:44:43.790160+0000","last_unstale":"2024-09-18T07:52:05.274274+0000","last_undegraded":"2024-09-18T07:52:05.274274+0000","last_fullsized":"2024-09-18T07:52:05.274274+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:26.866900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:26.866900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:27.691180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":935,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.206597+0000","last_change":"2024-09-18T07:51:58.503215+0000","last_active":"2024-09-18T07:52:00.206597+0000","last_peered":"2024-09-18T07:52:00.206597+0000","last_clean":"2024-09-18T07:52:00.206597+0000","last_became_active":"2024-09-18T07:51:45.206603+0000","last_became_peered":"2024-09-18T07:51:45.206603+0000","last_unstale":"2024-09-18T07:52:00.206597+0000","last_undegraded":"2024-09-18T07:52:00.206597+0000","last_fullsized":"2024-09-18T07:52:00.206597+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:58.503150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:58.503150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:23.150726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":953,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271637+0000","last_change":"2024-09-18T07:51:27.530235+0000","last_active":"2024-09-18T07:52:05.271637+0000","last_peered":"2024-09-18T07:52:05.271637+0000","last_clean":"2024-09-18T07:52:05.271637+0000","last_became_active":"2024-09-18T07:51:27.529923+0000","last_became_peered":"2024-09-18T07:51:27.529923+0000","last_unstale":"2024-09-18T07:52:05.271637+0000","last_undegraded":"2024-09-18T07:52:05.271637+0000","last_fullsized":"2024-09-18T07:52:05.271637+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:01.659774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:01.659774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:19.747393+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":473,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:01.575294+0000","last_change":"2024-09-18T07:52:01.575294+0000","last_active":"2024-09-18T07:52:01.575294+0000","last_peered":"2024-09-18T07:52:01.575294+0000","last_clean":"2024-09-18T07:52:01.575294+0000","last_became_active":"2024-09-18T07:51:45.207398+0000","last_became_peered":"2024-09-18T07:51:45.207398+0000","last_unstale":"2024-09-18T07:52:01.575294+0000","last_undegraded":"2024-09-18T07:52:01.575294+0000","last_fullsized":"2024-09-18T07:52:01.575294+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:52:01.575223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:52:01.575223+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:11.256659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"568'45","reported_seq":1129,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274848+0000","last_change":"2024-09-18T07:51:51.869099+0000","last_active":"2024-09-18T07:52:05.274848+0000","last_peered":"2024-09-18T07:52:05.274848+0000","last_clean":"2024-09-18T07:52:05.274848+0000","last_became_active":"2024-09-18T07:47:20.132912+0000","last_became_peered":"2024-09-18T07:47:20.132912+0000","last_unstale":"2024-09-18T07:52:05.274848+0000","last_undegraded":"2024-09-18T07:52:05.274848+0000","last_fullsized":"2024-09-18T07:52:05.274848+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'45","last_scrub_stamp":"2024-09-18T07:51:51.869009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:51.869009+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:12.241662+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":830,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592488+0000","last_change":"2024-09-18T07:51:02.691728+0000","last_active":"2024-09-18T07:52:05.592488+0000","last_peered":"2024-09-18T07:52:05.592488+0000","last_clean":"2024-09-18T07:52:05.592488+0000","last_became_active":"2024-09-18T07:49:32.688352+0000","last_became_peered":"2024-09-18T07:49:32.688352+0000","last_unstale":"2024-09-18T07:52:05.592488+0000","last_undegraded":"2024-09-18T07:52:05.592488+0000","last_fullsized":"2024-09-18T07:52:05.592488+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:02.691674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:02.691674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:18.998767+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":709,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271348+0000","last_change":"2024-09-18T07:51:59.797142+0000","last_active":"2024-09-18T07:52:05.271348+0000","last_peered":"2024-09-18T07:52:05.271348+0000","last_clean":"2024-09-18T07:52:05.271348+0000","last_became_active":"2024-09-18T07:51:19.263461+0000","last_became_peered":"2024-09-18T07:51:19.263461+0000","last_unstale":"2024-09-18T07:52:05.271348+0000","last_undegraded":"2024-09-18T07:52:05.271348+0000","last_fullsized":"2024-09-18T07:52:05.271348+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:59.797094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:59.797094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:10.149459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1176,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.589993+0000","last_change":"2024-09-18T07:51:30.488108+0000","last_active":"2024-09-18T07:52:05.589993+0000","last_peered":"2024-09-18T07:52:05.589993+0000","last_clean":"2024-09-18T07:52:05.589993+0000","last_became_active":"2024-09-18T07:50:23.276876+0000","last_became_peered":"2024-09-18T07:50:23.276876+0000","last_unstale":"2024-09-18T07:52:05.589993+0000","last_undegraded":"2024-09-18T07:52:05.589993+0000","last_fullsized":"2024-09-18T07:52:05.589993+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:30.488061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:30.488061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:46.403015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":989,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590503+0000","last_change":"2024-09-18T07:51:57.288930+0000","last_active":"2024-09-18T07:52:05.590503+0000","last_peered":"2024-09-18T07:52:05.590503+0000","last_clean":"2024-09-18T07:52:05.590503+0000","last_became_active":"2024-09-18T07:50:23.295783+0000","last_became_peered":"2024-09-18T07:50:23.295783+0000","last_unstale":"2024-09-18T07:52:05.590503+0000","last_undegraded":"2024-09-18T07:52:05.590503+0000","last_fullsized":"2024-09-18T07:52:05.590503+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:51:57.288877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:51:57.288877+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:15.479957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"568'30","reported_seq":1114,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274507+0000","last_change":"2024-09-18T07:50:46.869864+0000","last_active":"2024-09-18T07:52:05.274507+0000","last_peered":"2024-09-18T07:52:05.274507+0000","last_clean":"2024-09-18T07:52:05.274507+0000","last_became_active":"2024-09-18T07:50:23.278411+0000","last_became_peered":"2024-09-18T07:50:23.278411+0000","last_unstale":"2024-09-18T07:52:05.274507+0000","last_undegraded":"2024-09-18T07:52:05.274507+0000","last_fullsized":"2024-09-18T07:52:05.274507+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:50:46.869811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:50:46.869811+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:12.812560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":552,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272239+0000","last_change":"2024-09-18T07:51:27.296039+0000","last_active":"2024-09-18T07:52:05.272239+0000","last_peered":"2024-09-18T07:52:05.272239+0000","last_clean":"2024-09-18T07:52:05.272239+0000","last_became_active":"2024-09-18T07:51:27.295752+0000","last_became_peered":"2024-09-18T07:51:27.295752+0000","last_unstale":"2024-09-18T07:52:05.272239+0000","last_undegraded":"2024-09-18T07:52:05.272239+0000","last_fullsized":"2024-09-18T07:52:05.272239+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:40.432715+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"461'13","reported_seq":1448,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.589883+0000","last_change":"2024-09-18T07:51:45.574837+0000","last_active":"2024-09-18T07:52:05.589883+0000","last_peered":"2024-09-18T07:52:05.589883+0000","last_clean":"2024-09-18T07:52:05.589883+0000","last_became_active":"2024-09-18T07:47:11.203256+0000","last_became_peered":"2024-09-18T07:47:11.203256+0000","last_unstale":"2024-09-18T07:52:05.589883+0000","last_undegraded":"2024-09-18T07:52:05.589883+0000","last_fullsized":"2024-09-18T07:52:05.589883+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'13","last_scrub_stamp":"2024-09-18T07:51:45.574786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:45.574786+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:52.918180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1049,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.275040+0000","last_change":"2024-09-18T07:51:12.902793+0000","last_active":"2024-09-18T07:52:05.275040+0000","last_peered":"2024-09-18T07:52:05.275040+0000","last_clean":"2024-09-18T07:52:05.275040+0000","last_became_active":"2024-09-18T07:49:32.689562+0000","last_became_peered":"2024-09-18T07:49:32.689562+0000","last_unstale":"2024-09-18T07:52:05.275040+0000","last_undegraded":"2024-09-18T07:52:05.275040+0000","last_fullsized":"2024-09-18T07:52:05.275040+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:12.902740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:12.902740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:13.202651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"568'21","reported_seq":86,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.204962+0000","last_change":"2024-09-18T07:51:33.773423+0000","last_active":"2024-09-18T07:52:00.204962+0000","last_peered":"2024-09-18T07:52:00.204962+0000","last_clean":"2024-09-18T07:52:00.204962+0000","last_became_active":"2024-09-18T07:51:09.223848+0000","last_became_peered":"2024-09-18T07:51:09.223848+0000","last_unstale":"2024-09-18T07:52:00.204962+0000","last_undegraded":"2024-09-18T07:52:00.204962+0000","last_fullsized":"2024-09-18T07:52:00.204962+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'21","last_scrub_stamp":"2024-09-18T07:51:33.773353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:33.773353+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:00.548154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":961,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.206450+0000","last_change":"2024-09-18T07:51:45.209235+0000","last_active":"2024-09-18T07:52:00.206450+0000","last_peered":"2024-09-18T07:52:00.206450+0000","last_clean":"2024-09-18T07:52:00.206450+0000","last_became_active":"2024-09-18T07:51:45.208865+0000","last_became_peered":"2024-09-18T07:51:45.208865+0000","last_unstale":"2024-09-18T07:52:00.206450+0000","last_undegraded":"2024-09-18T07:52:00.206450+0000","last_fullsized":"2024-09-18T07:52:00.206450+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:14.600447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:14.600447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:15.017387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1171,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.589677+0000","last_change":"2024-09-18T07:51:16.435171+0000","last_active":"2024-09-18T07:52:05.589677+0000","last_peered":"2024-09-18T07:52:05.589677+0000","last_clean":"2024-09-18T07:52:05.589677+0000","last_became_active":"2024-09-18T07:50:23.273703+0000","last_became_peered":"2024-09-18T07:50:23.273703+0000","last_unstale":"2024-09-18T07:52:05.589677+0000","last_undegraded":"2024-09-18T07:52:05.589677+0000","last_fullsized":"2024-09-18T07:52:05.589677+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:16.435124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:16.435124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:27.652986+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":602,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.203810+0000","last_change":"2024-09-18T07:51:34.818926+0000","last_active":"2024-09-18T07:52:00.203810+0000","last_peered":"2024-09-18T07:52:00.203810+0000","last_clean":"2024-09-18T07:52:00.203810+0000","last_became_active":"2024-09-18T07:51:15.234271+0000","last_became_peered":"2024-09-18T07:51:15.234271+0000","last_unstale":"2024-09-18T07:52:00.203810+0000","last_undegraded":"2024-09-18T07:52:00.203810+0000","last_fullsized":"2024-09-18T07:52:00.203810+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:34.818869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:34.818869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:51.813395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"568'51","reported_seq":903,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592634+0000","last_change":"2024-09-18T07:51:37.539716+0000","last_active":"2024-09-18T07:52:05.592634+0000","last_peered":"2024-09-18T07:52:05.592634+0000","last_clean":"2024-09-18T07:52:05.592634+0000","last_became_active":"2024-09-18T07:47:20.141942+0000","last_became_peered":"2024-09-18T07:47:20.141942+0000","last_unstale":"2024-09-18T07:52:05.592634+0000","last_undegraded":"2024-09-18T07:52:05.592634+0000","last_fullsized":"2024-09-18T07:52:05.592634+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'51","last_scrub_stamp":"2024-09-18T07:51:37.539625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:37.539625+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:41.599784+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":934,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591231+0000","last_change":"2024-09-18T07:51:36.438107+0000","last_active":"2024-09-18T07:52:05.591231+0000","last_peered":"2024-09-18T07:52:05.591231+0000","last_clean":"2024-09-18T07:52:05.591231+0000","last_became_active":"2024-09-18T07:47:11.194873+0000","last_became_peered":"2024-09-18T07:47:11.194873+0000","last_unstale":"2024-09-18T07:52:05.591231+0000","last_undegraded":"2024-09-18T07:52:05.591231+0000","last_fullsized":"2024-09-18T07:52:05.591231+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:36.438038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:36.438038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:05.058323+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1186,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.589715+0000","last_change":"2024-09-18T07:51:09.234135+0000","last_active":"2024-09-18T07:52:05.589715+0000","last_peered":"2024-09-18T07:52:05.589715+0000","last_clean":"2024-09-18T07:52:05.589715+0000","last_became_active":"2024-09-18T07:51:09.233826+0000","last_became_peered":"2024-09-18T07:51:09.233826+0000","last_unstale":"2024-09-18T07:52:05.589715+0000","last_undegraded":"2024-09-18T07:52:05.589715+0000","last_fullsized":"2024-09-18T07:52:05.589715+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:05.491144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:05.491144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:12.937636+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":923,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592275+0000","last_change":"2024-09-18T07:51:00.509832+0000","last_active":"2024-09-18T07:52:05.592275+0000","last_peered":"2024-09-18T07:52:05.592275+0000","last_clean":"2024-09-18T07:52:05.592275+0000","last_became_active":"2024-09-18T07:50:23.296866+0000","last_became_peered":"2024-09-18T07:50:23.296866+0000","last_unstale":"2024-09-18T07:52:05.592275+0000","last_undegraded":"2024-09-18T07:52:05.592275+0000","last_fullsized":"2024-09-18T07:52:05.592275+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:00.509766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:00.509766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:22.907420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"568'24","reported_seq":1175,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274155+0000","last_change":"2024-09-18T07:52:03.810149+0000","last_active":"2024-09-18T07:52:05.274155+0000","last_peered":"2024-09-18T07:52:05.274155+0000","last_clean":"2024-09-18T07:52:05.274155+0000","last_became_active":"2024-09-18T07:50:23.272111+0000","last_became_peered":"2024-09-18T07:50:23.272111+0000","last_unstale":"2024-09-18T07:52:05.274155+0000","last_undegraded":"2024-09-18T07:52:05.274155+0000","last_fullsized":"2024-09-18T07:52:05.274155+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'24","last_scrub_stamp":"2024-09-18T07:52:03.810083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:52:03.810083+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:20.792783+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":926,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592215+0000","last_change":"2024-09-18T07:51:27.507544+0000","last_active":"2024-09-18T07:52:05.592215+0000","last_peered":"2024-09-18T07:52:05.592215+0000","last_clean":"2024-09-18T07:52:05.592215+0000","last_became_active":"2024-09-18T07:47:20.138910+0000","last_became_peered":"2024-09-18T07:47:20.138910+0000","last_unstale":"2024-09-18T07:52:05.592215+0000","last_undegraded":"2024-09-18T07:52:05.592215+0000","last_fullsized":"2024-09-18T07:52:05.592215+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:27.507488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:27.507488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:50.641141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1196,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.589572+0000","last_change":"2024-09-18T07:51:17.405413+0000","last_active":"2024-09-18T07:52:05.589572+0000","last_peered":"2024-09-18T07:52:05.589572+0000","last_clean":"2024-09-18T07:52:05.589572+0000","last_became_active":"2024-09-18T07:51:09.231725+0000","last_became_peered":"2024-09-18T07:51:09.231725+0000","last_unstale":"2024-09-18T07:52:05.589572+0000","last_undegraded":"2024-09-18T07:52:05.589572+0000","last_fullsized":"2024-09-18T07:52:05.589572+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:17.405363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:17.405363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:24.592333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":129,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272470+0000","last_change":"2024-09-18T07:51:00.677134+0000","last_active":"2024-09-18T07:52:05.272470+0000","last_peered":"2024-09-18T07:52:05.272470+0000","last_clean":"2024-09-18T07:52:05.272470+0000","last_became_active":"2024-09-18T07:50:23.299077+0000","last_became_peered":"2024-09-18T07:50:23.299077+0000","last_unstale":"2024-09-18T07:52:05.272470+0000","last_undegraded":"2024-09-18T07:52:05.272470+0000","last_fullsized":"2024-09-18T07:52:05.272470+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:00.677086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:00.677086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:11.141258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"568'39","reported_seq":1146,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.205764+0000","last_change":"2024-09-18T07:51:04.569896+0000","last_active":"2024-09-18T07:52:00.205764+0000","last_peered":"2024-09-18T07:52:00.205764+0000","last_clean":"2024-09-18T07:52:00.205764+0000","last_became_active":"2024-09-18T07:47:11.196288+0000","last_became_peered":"2024-09-18T07:47:11.196288+0000","last_unstale":"2024-09-18T07:52:00.205764+0000","last_undegraded":"2024-09-18T07:52:00.205764+0000","last_fullsized":"2024-09-18T07:52:00.205764+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'37","last_scrub_stamp":"2024-09-18T07:51:04.569839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:04.569839+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:11.165888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":960,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590925+0000","last_change":"2024-09-18T07:51:51.219166+0000","last_active":"2024-09-18T07:52:05.590925+0000","last_peered":"2024-09-18T07:52:05.590925+0000","last_clean":"2024-09-18T07:52:05.590925+0000","last_became_active":"2024-09-18T07:51:09.223041+0000","last_became_peered":"2024-09-18T07:51:09.223041+0000","last_unstale":"2024-09-18T07:52:05.590925+0000","last_undegraded":"2024-09-18T07:52:05.590925+0000","last_fullsized":"2024-09-18T07:52:05.590925+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:51:51.219100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:51.219100+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:14.396523+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"398'14","reported_seq":1162,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591610+0000","last_change":"2024-09-18T07:52:02.368992+0000","last_active":"2024-09-18T07:52:05.591610+0000","last_peered":"2024-09-18T07:52:05.591610+0000","last_clean":"2024-09-18T07:52:05.591610+0000","last_became_active":"2024-09-18T07:44:43.782355+0000","last_became_peered":"2024-09-18T07:44:43.782355+0000","last_unstale":"2024-09-18T07:52:05.591610+0000","last_undegraded":"2024-09-18T07:52:05.591610+0000","last_fullsized":"2024-09-18T07:52:05.591610+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:52:02.368911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:52:02.368911+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:26.979454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":969,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:02.555409+0000","last_change":"2024-09-18T07:52:02.555409+0000","last_active":"2024-09-18T07:52:02.555409+0000","last_peered":"2024-09-18T07:52:02.555409+0000","last_clean":"2024-09-18T07:52:02.555409+0000","last_became_active":"2024-09-18T07:51:45.207416+0000","last_became_peered":"2024-09-18T07:51:45.207416+0000","last_unstale":"2024-09-18T07:52:02.555409+0000","last_undegraded":"2024-09-18T07:52:02.555409+0000","last_fullsized":"2024-09-18T07:52:02.555409+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:52:02.555356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:52:02.555356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:26.036297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":792,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590195+0000","last_change":"2024-09-18T07:51:16.664570+0000","last_active":"2024-09-18T07:52:05.590195+0000","last_peered":"2024-09-18T07:52:05.590195+0000","last_clean":"2024-09-18T07:52:05.590195+0000","last_became_active":"2024-09-18T07:51:09.231456+0000","last_became_peered":"2024-09-18T07:51:09.231456+0000","last_unstale":"2024-09-18T07:52:05.590195+0000","last_undegraded":"2024-09-18T07:52:05.590195+0000","last_fullsized":"2024-09-18T07:52:05.590195+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:16.664498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:16.664498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:43.054012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"568'18","reported_seq":58,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272624+0000","last_change":"2024-09-18T07:51:25.206195+0000","last_active":"2024-09-18T07:52:05.272624+0000","last_peered":"2024-09-18T07:52:05.272624+0000","last_clean":"2024-09-18T07:52:05.272624+0000","last_became_active":"2024-09-18T07:51:25.200814+0000","last_became_peered":"2024-09-18T07:51:25.200814+0000","last_unstale":"2024-09-18T07:52:05.272624+0000","last_undegraded":"2024-09-18T07:52:05.272624+0000","last_fullsized":"2024-09-18T07:52:05.272624+0000","mapping_epoch":580,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":581,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'18","last_scrub_stamp":"2024-09-18T07:51:14.578829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:14.578829+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:24.169560+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1080,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.206703+0000","last_change":"2024-09-18T07:51:39.334521+0000","last_active":"2024-09-18T07:52:00.206703+0000","last_peered":"2024-09-18T07:52:00.206703+0000","last_clean":"2024-09-18T07:52:00.206703+0000","last_became_active":"2024-09-18T07:50:23.273803+0000","last_became_peered":"2024-09-18T07:50:23.273803+0000","last_unstale":"2024-09-18T07:52:00.206703+0000","last_undegraded":"2024-09-18T07:52:00.206703+0000","last_fullsized":"2024-09-18T07:52:00.206703+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:39.334465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:39.334465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:05.600879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":926,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591082+0000","last_change":"2024-09-18T07:51:54.206541+0000","last_active":"2024-09-18T07:52:05.591082+0000","last_peered":"2024-09-18T07:52:05.591082+0000","last_clean":"2024-09-18T07:52:05.591082+0000","last_became_active":"2024-09-18T07:44:43.782348+0000","last_became_peered":"2024-09-18T07:44:43.782348+0000","last_unstale":"2024-09-18T07:52:05.591082+0000","last_undegraded":"2024-09-18T07:52:05.591082+0000","last_fullsized":"2024-09-18T07:52:05.591082+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:54.206480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:54.206480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:02.492004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"568'21","reported_seq":150,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.204441+0000","last_change":"2024-09-18T07:51:21.795622+0000","last_active":"2024-09-18T07:52:00.204441+0000","last_peered":"2024-09-18T07:52:00.204441+0000","last_clean":"2024-09-18T07:52:00.204441+0000","last_became_active":"2024-09-18T07:51:15.238300+0000","last_became_peered":"2024-09-18T07:51:15.238300+0000","last_unstale":"2024-09-18T07:52:00.204441+0000","last_undegraded":"2024-09-18T07:52:00.204441+0000","last_fullsized":"2024-09-18T07:52:00.204441+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'21","last_scrub_stamp":"2024-09-18T07:51:21.795567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:21.795567+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:33.896402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":942,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591238+0000","last_change":"2024-09-18T07:51:59.339423+0000","last_active":"2024-09-18T07:52:05.591238+0000","last_peered":"2024-09-18T07:52:05.591238+0000","last_clean":"2024-09-18T07:52:05.591238+0000","last_became_active":"2024-09-18T07:44:43.782930+0000","last_became_peered":"2024-09-18T07:44:43.782930+0000","last_unstale":"2024-09-18T07:52:05.591238+0000","last_undegraded":"2024-09-18T07:52:05.591238+0000","last_fullsized":"2024-09-18T07:52:05.591238+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:59.339361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:59.339361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:17.990759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":129,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272898+0000","last_change":"2024-09-18T07:51:25.497520+0000","last_active":"2024-09-18T07:52:05.272898+0000","last_peered":"2024-09-18T07:52:05.272898+0000","last_clean":"2024-09-18T07:52:05.272898+0000","last_became_active":"2024-09-18T07:50:23.274007+0000","last_became_peered":"2024-09-18T07:50:23.274007+0000","last_unstale":"2024-09-18T07:52:05.272898+0000","last_undegraded":"2024-09-18T07:52:05.272898+0000","last_fullsized":"2024-09-18T07:52:05.272898+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:25.497473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:25.497473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:30.968653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1143,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.589795+0000","last_change":"2024-09-18T07:51:08.465112+0000","last_active":"2024-09-18T07:52:05.589795+0000","last_peered":"2024-09-18T07:52:05.589795+0000","last_clean":"2024-09-18T07:52:05.589795+0000","last_became_active":"2024-09-18T07:44:43.790582+0000","last_became_peered":"2024-09-18T07:44:43.790582+0000","last_unstale":"2024-09-18T07:52:05.589795+0000","last_undegraded":"2024-09-18T07:52:05.589795+0000","last_fullsized":"2024-09-18T07:52:05.589795+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:08.465062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:08.465062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:25.231438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"568'21","reported_seq":835,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.206461+0000","last_change":"2024-09-18T07:51:53.484445+0000","last_active":"2024-09-18T07:52:00.206461+0000","last_peered":"2024-09-18T07:52:00.206461+0000","last_clean":"2024-09-18T07:52:00.206461+0000","last_became_active":"2024-09-18T07:51:45.208309+0000","last_became_peered":"2024-09-18T07:51:45.208309+0000","last_unstale":"2024-09-18T07:52:00.206461+0000","last_undegraded":"2024-09-18T07:52:00.206461+0000","last_fullsized":"2024-09-18T07:52:00.206461+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'21","last_scrub_stamp":"2024-09-18T07:51:53.484377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:53.484377+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:02.233769+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":620,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.203759+0000","last_change":"2024-09-18T07:51:15.659362+0000","last_active":"2024-09-18T07:52:00.203759+0000","last_peered":"2024-09-18T07:52:00.203759+0000","last_clean":"2024-09-18T07:52:00.203759+0000","last_became_active":"2024-09-18T07:51:15.235143+0000","last_became_peered":"2024-09-18T07:51:15.235143+0000","last_unstale":"2024-09-18T07:52:00.203759+0000","last_undegraded":"2024-09-18T07:52:00.203759+0000","last_fullsized":"2024-09-18T07:52:00.203759+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:15.659312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:15.659312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:24.825353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"548'12","reported_seq":154,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272074+0000","last_change":"2024-09-18T07:51:38.507467+0000","last_active":"2024-09-18T07:52:05.272074+0000","last_peered":"2024-09-18T07:52:05.272074+0000","last_clean":"2024-09-18T07:52:05.272074+0000","last_became_active":"2024-09-18T07:50:23.294281+0000","last_became_peered":"2024-09-18T07:50:23.294281+0000","last_unstale":"2024-09-18T07:52:05.272074+0000","last_undegraded":"2024-09-18T07:52:05.272074+0000","last_fullsized":"2024-09-18T07:52:05.272074+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'12","last_scrub_stamp":"2024-09-18T07:51:38.507421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:38.507421+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:56.194027+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1031,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274923+0000","last_change":"2024-09-18T07:51:36.857221+0000","last_active":"2024-09-18T07:52:05.274923+0000","last_peered":"2024-09-18T07:52:05.274923+0000","last_clean":"2024-09-18T07:52:05.274923+0000","last_became_active":"2024-09-18T07:47:11.193273+0000","last_became_peered":"2024-09-18T07:47:11.193273+0000","last_unstale":"2024-09-18T07:52:05.274923+0000","last_undegraded":"2024-09-18T07:52:05.274923+0000","last_fullsized":"2024-09-18T07:52:05.274923+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:36.857151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:36.857151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:54.434263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"568'30","reported_seq":1011,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271731+0000","last_change":"2024-09-18T07:51:27.295557+0000","last_active":"2024-09-18T07:52:05.271731+0000","last_peered":"2024-09-18T07:52:05.271731+0000","last_clean":"2024-09-18T07:52:05.271731+0000","last_became_active":"2024-09-18T07:51:27.295220+0000","last_became_peered":"2024-09-18T07:51:27.295220+0000","last_unstale":"2024-09-18T07:52:05.271731+0000","last_undegraded":"2024-09-18T07:52:05.271731+0000","last_fullsized":"2024-09-18T07:52:05.271731+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'30","last_scrub_stamp":"2024-09-18T07:51:21.907205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:21.907205+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:24.368145+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":928,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592083+0000","last_change":"2024-09-18T07:51:56.270686+0000","last_active":"2024-09-18T07:52:05.592083+0000","last_peered":"2024-09-18T07:52:05.592083+0000","last_clean":"2024-09-18T07:52:05.592083+0000","last_became_active":"2024-09-18T07:44:43.779305+0000","last_became_peered":"2024-09-18T07:44:43.779305+0000","last_unstale":"2024-09-18T07:52:05.592083+0000","last_undegraded":"2024-09-18T07:52:05.592083+0000","last_fullsized":"2024-09-18T07:52:05.592083+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:56.270617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:56.270617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:20.479706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"571'17","reported_seq":1287,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.206628+0000","last_change":"2024-09-18T07:51:56.454606+0000","last_active":"2024-09-18T07:52:00.206628+0000","last_peered":"2024-09-18T07:52:00.206628+0000","last_clean":"2024-09-18T07:52:00.206628+0000","last_became_active":"2024-09-18T07:51:09.215676+0000","last_became_peered":"2024-09-18T07:51:09.215676+0000","last_unstale":"2024-09-18T07:52:00.206628+0000","last_undegraded":"2024-09-18T07:52:00.206628+0000","last_fullsized":"2024-09-18T07:52:00.206628+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"571'17","last_scrub_stamp":"2024-09-18T07:51:56.454540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:56.454540+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:13.534025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":616,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.204106+0000","last_change":"2024-09-18T07:51:49.695350+0000","last_active":"2024-09-18T07:52:00.204106+0000","last_peered":"2024-09-18T07:52:00.204106+0000","last_clean":"2024-09-18T07:52:00.204106+0000","last_became_active":"2024-09-18T07:51:15.237465+0000","last_became_peered":"2024-09-18T07:51:15.237465+0000","last_unstale":"2024-09-18T07:52:00.204106+0000","last_undegraded":"2024-09-18T07:52:00.204106+0000","last_fullsized":"2024-09-18T07:52:00.204106+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:49.695309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:49.695309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:03.928177+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"568'27","reported_seq":1186,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274583+0000","last_change":"2024-09-18T07:51:35.897044+0000","last_active":"2024-09-18T07:52:05.274583+0000","last_peered":"2024-09-18T07:52:05.274583+0000","last_clean":"2024-09-18T07:52:05.274583+0000","last_became_active":"2024-09-18T07:47:20.140070+0000","last_became_peered":"2024-09-18T07:47:20.140070+0000","last_unstale":"2024-09-18T07:52:05.274583+0000","last_undegraded":"2024-09-18T07:52:05.274583+0000","last_fullsized":"2024-09-18T07:52:05.274583+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'27","last_scrub_stamp":"2024-09-18T07:51:35.896983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:35.896983+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:57.610961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1088,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274655+0000","last_change":"2024-09-18T07:52:02.850609+0000","last_active":"2024-09-18T07:52:05.274655+0000","last_peered":"2024-09-18T07:52:05.274655+0000","last_clean":"2024-09-18T07:52:05.274655+0000","last_became_active":"2024-09-18T07:44:43.780305+0000","last_became_peered":"2024-09-18T07:44:43.780305+0000","last_unstale":"2024-09-18T07:52:05.274655+0000","last_undegraded":"2024-09-18T07:52:05.274655+0000","last_fullsized":"2024-09-18T07:52:05.274655+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:52:02.850552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:52:02.850552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:03.753388+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":592,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.270923+0000","last_change":"2024-09-18T07:51:26.856394+0000","last_active":"2024-09-18T07:52:05.270923+0000","last_peered":"2024-09-18T07:52:05.270923+0000","last_clean":"2024-09-18T07:52:05.270923+0000","last_became_active":"2024-09-18T07:51:25.200221+0000","last_became_peered":"2024-09-18T07:51:25.200221+0000","last_unstale":"2024-09-18T07:52:05.270923+0000","last_undegraded":"2024-09-18T07:52:05.270923+0000","last_fullsized":"2024-09-18T07:52:05.270923+0000","mapping_epoch":580,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":581,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:26.856334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:26.856334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:38.312784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":84,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271538+0000","last_change":"2024-09-18T07:51:27.567502+0000","last_active":"2024-09-18T07:52:05.271538+0000","last_peered":"2024-09-18T07:52:05.271538+0000","last_clean":"2024-09-18T07:52:05.271538+0000","last_became_active":"2024-09-18T07:51:27.345060+0000","last_became_peered":"2024-09-18T07:51:27.345060+0000","last_unstale":"2024-09-18T07:52:05.271538+0000","last_undegraded":"2024-09-18T07:52:05.271538+0000","last_fullsized":"2024-09-18T07:52:05.271538+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:27.567448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:27.567448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:48.530139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"568'21","reported_seq":828,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.270738+0000","last_change":"2024-09-18T07:51:23.889518+0000","last_active":"2024-09-18T07:52:05.270738+0000","last_peered":"2024-09-18T07:52:05.270738+0000","last_clean":"2024-09-18T07:52:05.270738+0000","last_became_active":"2024-09-18T07:51:19.256222+0000","last_became_peered":"2024-09-18T07:51:19.256222+0000","last_unstale":"2024-09-18T07:52:05.270738+0000","last_undegraded":"2024-09-18T07:52:05.270738+0000","last_fullsized":"2024-09-18T07:52:05.270738+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'21","last_scrub_stamp":"2024-09-18T07:51:23.889468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:23.889468+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:41.647872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":941,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591060+0000","last_change":"2024-09-18T07:51:39.418212+0000","last_active":"2024-09-18T07:52:05.591060+0000","last_peered":"2024-09-18T07:52:05.591060+0000","last_clean":"2024-09-18T07:52:05.591060+0000","last_became_active":"2024-09-18T07:50:23.296874+0000","last_became_peered":"2024-09-18T07:50:23.296874+0000","last_unstale":"2024-09-18T07:52:05.591060+0000","last_undegraded":"2024-09-18T07:52:05.591060+0000","last_fullsized":"2024-09-18T07:52:05.591060+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:39.418156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:39.418156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:03.930110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":120,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272395+0000","last_change":"2024-09-18T07:51:54.291807+0000","last_active":"2024-09-18T07:52:05.272395+0000","last_peered":"2024-09-18T07:52:05.272395+0000","last_clean":"2024-09-18T07:52:05.272395+0000","last_became_active":"2024-09-18T07:51:27.295722+0000","last_became_peered":"2024-09-18T07:51:27.295722+0000","last_unstale":"2024-09-18T07:52:05.272395+0000","last_undegraded":"2024-09-18T07:52:05.272395+0000","last_fullsized":"2024-09-18T07:52:05.272395+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:54.291761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:54.291761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:02.158477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":76,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.204415+0000","last_change":"2024-09-18T07:51:13.652044+0000","last_active":"2024-09-18T07:52:00.204415+0000","last_peered":"2024-09-18T07:52:00.204415+0000","last_clean":"2024-09-18T07:52:00.204415+0000","last_became_active":"2024-09-18T07:51:09.221267+0000","last_became_peered":"2024-09-18T07:51:09.221267+0000","last_unstale":"2024-09-18T07:52:00.204415+0000","last_undegraded":"2024-09-18T07:52:00.204415+0000","last_fullsized":"2024-09-18T07:52:00.204415+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:13.651996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:13.651996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:42.470038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"568'27","reported_seq":153,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272146+0000","last_change":"2024-09-18T07:51:28.550261+0000","last_active":"2024-09-18T07:52:05.272146+0000","last_peered":"2024-09-18T07:52:05.272146+0000","last_clean":"2024-09-18T07:52:05.272146+0000","last_became_active":"2024-09-18T07:50:23.389095+0000","last_became_peered":"2024-09-18T07:50:23.389095+0000","last_unstale":"2024-09-18T07:52:05.272146+0000","last_undegraded":"2024-09-18T07:52:05.272146+0000","last_fullsized":"2024-09-18T07:52:05.272146+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'27","last_scrub_stamp":"2024-09-18T07:51:28.550205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:28.550205+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:44.527319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1071,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.206582+0000","last_change":"2024-09-18T07:51:02.632440+0000","last_active":"2024-09-18T07:52:00.206582+0000","last_peered":"2024-09-18T07:52:00.206582+0000","last_clean":"2024-09-18T07:52:00.206582+0000","last_became_active":"2024-09-18T07:47:20.140495+0000","last_became_peered":"2024-09-18T07:47:20.140495+0000","last_unstale":"2024-09-18T07:52:00.206582+0000","last_undegraded":"2024-09-18T07:52:00.206582+0000","last_fullsized":"2024-09-18T07:52:00.206582+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:02.632361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:02.632361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:32.171602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":934,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592353+0000","last_change":"2024-09-18T07:51:01.482805+0000","last_active":"2024-09-18T07:52:05.592353+0000","last_peered":"2024-09-18T07:52:05.592353+0000","last_clean":"2024-09-18T07:52:05.592353+0000","last_became_active":"2024-09-18T07:44:43.778318+0000","last_became_peered":"2024-09-18T07:44:43.778318+0000","last_unstale":"2024-09-18T07:52:05.592353+0000","last_undegraded":"2024-09-18T07:52:05.592353+0000","last_fullsized":"2024-09-18T07:52:05.592353+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:01.482740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:01.482740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:19.740487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":136,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272313+0000","last_change":"2024-09-18T07:51:23.518872+0000","last_active":"2024-09-18T07:52:05.272313+0000","last_peered":"2024-09-18T07:52:05.272313+0000","last_clean":"2024-09-18T07:52:05.272313+0000","last_became_active":"2024-09-18T07:51:09.232703+0000","last_became_peered":"2024-09-18T07:51:09.232703+0000","last_unstale":"2024-09-18T07:52:05.272313+0000","last_undegraded":"2024-09-18T07:52:05.272313+0000","last_fullsized":"2024-09-18T07:52:05.272313+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:23.518811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:23.518811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:39.109440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"568'12","reported_seq":141,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271538+0000","last_change":"2024-09-18T07:51:59.244312+0000","last_active":"2024-09-18T07:52:05.271538+0000","last_peered":"2024-09-18T07:52:05.271538+0000","last_clean":"2024-09-18T07:52:05.271538+0000","last_became_active":"2024-09-18T07:50:23.298033+0000","last_became_peered":"2024-09-18T07:50:23.298033+0000","last_unstale":"2024-09-18T07:52:05.271538+0000","last_undegraded":"2024-09-18T07:52:05.271538+0000","last_fullsized":"2024-09-18T07:52:05.271538+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'12","last_scrub_stamp":"2024-09-18T07:51:59.244260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:59.244260+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:01.683157+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":342,"reported_epoch":599,"state":"active+clean","last_fresh":"2024-09-18T07:52:00.204159+0000","last_change":"2024-09-18T07:51:43.766097+0000","last_active":"2024-09-18T07:52:00.204159+0000","last_peered":"2024-09-18T07:52:00.204159+0000","last_clean":"2024-09-18T07:52:00.204159+0000","last_became_active":"2024-09-18T07:51:15.235647+0000","last_became_peered":"2024-09-18T07:51:15.235647+0000","last_unstale":"2024-09-18T07:52:00.204159+0000","last_undegraded":"2024-09-18T07:52:00.204159+0000","last_fullsized":"2024-09-18T07:52:00.204159+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:43.766055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:43.766055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:02.535199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":771,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590503+0000","last_change":"2024-09-18T07:50:52.711210+0000","last_active":"2024-09-18T07:52:05.590503+0000","last_peered":"2024-09-18T07:52:05.590503+0000","last_clean":"2024-09-18T07:52:05.590503+0000","last_became_active":"2024-09-18T07:44:43.792641+0000","last_became_peered":"2024-09-18T07:44:43.792641+0000","last_unstale":"2024-09-18T07:52:05.590503+0000","last_undegraded":"2024-09-18T07:52:05.590503+0000","last_fullsized":"2024-09-18T07:52:05.590503+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:52.711146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:50:52.711146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:22.035108+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":920,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590684+0000","last_change":"2024-09-18T07:52:01.355114+0000","last_active":"2024-09-18T07:52:05.590684+0000","last_peered":"2024-09-18T07:52:05.590684+0000","last_clean":"2024-09-18T07:52:05.590684+0000","last_became_active":"2024-09-18T07:44:43.782378+0000","last_became_peered":"2024-09-18T07:44:43.782378+0000","last_unstale":"2024-09-18T07:52:05.590684+0000","last_undegraded":"2024-09-18T07:52:05.590684+0000","last_fullsized":"2024-09-18T07:52:05.590684+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:52:01.355061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:52:01.355061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:03.058506+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":489,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272020+0000","last_change":"2024-09-18T07:52:02.787203+0000","last_active":"2024-09-18T07:52:05.272020+0000","last_peered":"2024-09-18T07:52:05.272020+0000","last_clean":"2024-09-18T07:52:05.272020+0000","last_became_active":"2024-09-18T07:51:19.260630+0000","last_became_peered":"2024-09-18T07:51:19.260630+0000","last_unstale":"2024-09-18T07:52:05.272020+0000","last_undegraded":"2024-09-18T07:52:05.272020+0000","last_fullsized":"2024-09-18T07:52:05.272020+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:52:02.787155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:52:02.787155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:14.258870+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":144,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271924+0000","last_change":"2024-09-18T07:51:09.250179+0000","last_active":"2024-09-18T07:52:05.271924+0000","last_peered":"2024-09-18T07:52:05.271924+0000","last_clean":"2024-09-18T07:52:05.271924+0000","last_became_active":"2024-09-18T07:51:09.234244+0000","last_became_peered":"2024-09-18T07:51:09.234244+0000","last_unstale":"2024-09-18T07:52:05.271924+0000","last_undegraded":"2024-09-18T07:52:05.271924+0000","last_fullsized":"2024-09-18T07:52:05.271924+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:51:03.659174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:03.659174+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2979,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"568'27","reported_seq":851,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591874+0000","last_change":"2024-09-18T07:51:34.554484+0000","last_active":"2024-09-18T07:52:05.591874+0000","last_peered":"2024-09-18T07:52:05.591874+0000","last_clean":"2024-09-18T07:52:05.591874+0000","last_became_active":"2024-09-18T07:47:20.142007+0000","last_became_peered":"2024-09-18T07:47:20.142007+0000","last_unstale":"2024-09-18T07:52:05.591874+0000","last_undegraded":"2024-09-18T07:52:05.591874+0000","last_fullsized":"2024-09-18T07:52:05.591874+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'27","last_scrub_stamp":"2024-09-18T07:51:34.554423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:34.554423+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:43.542889+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":674,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590855+0000","last_change":"2024-09-18T07:51:46.239219+0000","last_active":"2024-09-18T07:52:05.590855+0000","last_peered":"2024-09-18T07:52:05.590855+0000","last_clean":"2024-09-18T07:52:05.590855+0000","last_became_active":"2024-09-18T07:47:20.139668+0000","last_became_peered":"2024-09-18T07:47:20.139668+0000","last_unstale":"2024-09-18T07:52:05.590855+0000","last_undegraded":"2024-09-18T07:52:05.590855+0000","last_fullsized":"2024-09-18T07:52:05.590855+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:46.239161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:46.239161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:11.718206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1513,"num_read_kb":1270,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":684,"num_bytes_recovered":14924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},"log_size":978,"ondisk_log_size":978,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":4701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1017,"num_read_kb":1012,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2372,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":853056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201539,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744852,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93000300,"statfs":{"total":95995035648,"available":95232307200,"internally_reserved":0,"allocated":1323008,"data_stored":830478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":761199050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463177,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758392,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92986760,"statfs":{"total":95995035648,"available":95218442240,"internally_reserved":0,"allocated":966656,"data_stored":501493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":775420358},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":460,"seq":1975684956221,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":170228,"kb_used_data":1696,"kb_used_omap":6,"kb_used_meta":168505,"kb_avail":93574924,"statfs":{"total":95995035648,"available":95820722176,"internally_reserved":0,"allocated":1736704,"data_stored":1190788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6877,"internal_metadata":172549411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":395,"seq":1696512082011,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":534168,"kb_used_data":976,"kb_used_omap":4,"kb_used_meta":532987,"kb_avail":93210984,"statfs":{"total":95995035648,"available":95448047616,"internally_reserved":0,"allocated":999424,"data_stored":502631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4447,"internal_metadata":545779361},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149977,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1073324,"kb_used_data":203428,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92671828,"statfs":{"total":95995035648,"available":94895951872,"internally_reserved":0,"allocated":208310272,"data_stored":207794835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":890567111},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411615,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":786424,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92958728,"statfs":{"total":95995035648,"available":95189737472,"internally_reserved":0,"allocated":1228800,"data_stored":764426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575142,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888956,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92856196,"statfs":{"total":95995035648,"available":95084744704,"internally_reserved":0,"allocated":1036288,"data_stored":504368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738666,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902796,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92842356,"statfs":{"total":95995035648,"available":95070572544,"internally_reserved":0,"allocated":1171456,"data_stored":602284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:52:09.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-18T07:52:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:09 smithi203 ceph-mon[32237]: pgmap v1151: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-18T07:52:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:09 smithi203 ceph-mon[32237]: from='client.34463 v1:172.21.15.203:0/4102717062' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-18T07:52:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:09 smithi203 ceph-mon[32237]: 3.2 scrub starts 2024-09-18T07:52:09.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:09 smithi203 ceph-mon[32237]: 3.2 scrub ok 2024-09-18T07:52:09.847 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:52:10.514 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-18T07:52:10.514 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:10 smithi137 ceph-mon[29408]: from='client.37055 v1:172.21.15.137:0/240123310' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:52:10.515 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:10 smithi137 ceph-mon[39175]: from='client.37055 v1:172.21.15.137:0/240123310' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:52:10.519 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-18T07:52:10.620 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:10 smithi203 ceph-mon[32237]: from='client.37055 v1:172.21.15.137:0/240123310' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:52:10.620 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:52:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=cleanup t=2024-09-18T07:52:10.49202914Z level=info msg="Completed cleanup jobs" duration=33.007569ms 2024-09-18T07:52:10.961 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:52:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=plugins.update.checker t=2024-09-18T07:52:10.618201346Z level=info msg="Update check succeeded" duration=77.5862ms 2024-09-18T07:52:10.961 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:52:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=grafana.update.checker t=2024-09-18T07:52:10.627902386Z level=info msg="Update check succeeded" duration=90.734701ms 2024-09-18T07:52:11.181 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":1152,"stamp":"2024-09-18T07:52:09.935102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2530,"num_read_kb":2282,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":743,"num_bytes_recovered":875547,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1979,"ondisk_log_size":1979,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":428,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5859164,"kb_used_data":211356,"kb_used_omap":20,"kb_used_meta":5646187,"kb_avail":744102052,"statfs":{"total":767960285184,"available":761960501248,"internally_reserved":0,"allocated":216428544,"data_stored":212494361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20886,"internal_metadata":5781696106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.730400"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":78,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.409293+0000","last_change":"2024-09-18T07:51:26.792552+0000","last_active":"2024-09-18T07:52:05.409293+0000","last_peered":"2024-09-18T07:52:05.409293+0000","last_clean":"2024-09-18T07:52:05.409293+0000","last_became_active":"2024-09-18T07:51:09.218894+0000","last_became_peered":"2024-09-18T07:51:09.218894+0000","last_unstale":"2024-09-18T07:52:05.409293+0000","last_undegraded":"2024-09-18T07:52:05.409293+0000","last_fullsized":"2024-09-18T07:52:05.409293+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:26.792494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:26.792494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:55.122561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"568'21","reported_seq":1128,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274118+0000","last_change":"2024-09-18T07:51:01.977506+0000","last_active":"2024-09-18T07:52:05.274118+0000","last_peered":"2024-09-18T07:52:05.274118+0000","last_clean":"2024-09-18T07:52:05.274118+0000","last_became_active":"2024-09-18T07:47:20.136254+0000","last_became_peered":"2024-09-18T07:47:20.136254+0000","last_unstale":"2024-09-18T07:52:05.274118+0000","last_undegraded":"2024-09-18T07:52:05.274118+0000","last_fullsized":"2024-09-18T07:52:05.274118+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'15","last_scrub_stamp":"2024-09-18T07:51:01.977421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:01.977421+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:19.711863+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":717,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.270648+0000","last_change":"2024-09-18T07:51:19.257480+0000","last_active":"2024-09-18T07:52:05.270648+0000","last_peered":"2024-09-18T07:52:05.270648+0000","last_clean":"2024-09-18T07:52:05.270648+0000","last_became_active":"2024-09-18T07:51:19.257114+0000","last_became_peered":"2024-09-18T07:51:19.257114+0000","last_unstale":"2024-09-18T07:52:05.270648+0000","last_undegraded":"2024-09-18T07:52:05.270648+0000","last_fullsized":"2024-09-18T07:52:05.270648+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:13.608556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:13.608556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:32.147463+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1011,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274428+0000","last_change":"2024-09-18T07:51:17.961214+0000","last_active":"2024-09-18T07:52:05.274428+0000","last_peered":"2024-09-18T07:52:05.274428+0000","last_clean":"2024-09-18T07:52:05.274428+0000","last_became_active":"2024-09-18T07:49:32.688553+0000","last_became_peered":"2024-09-18T07:49:32.688553+0000","last_unstale":"2024-09-18T07:52:05.274428+0000","last_undegraded":"2024-09-18T07:52:05.274428+0000","last_fullsized":"2024-09-18T07:52:05.274428+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:17.961167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:17.961167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:31.291311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":129,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272159+0000","last_change":"2024-09-18T07:51:31.473729+0000","last_active":"2024-09-18T07:52:05.272159+0000","last_peered":"2024-09-18T07:52:05.272159+0000","last_clean":"2024-09-18T07:52:05.272159+0000","last_became_active":"2024-09-18T07:50:23.285220+0000","last_became_peered":"2024-09-18T07:50:23.285220+0000","last_unstale":"2024-09-18T07:52:05.272159+0000","last_undegraded":"2024-09-18T07:52:05.272159+0000","last_fullsized":"2024-09-18T07:52:05.272159+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:31.473663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:31.473663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:34.340579+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":129,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272750+0000","last_change":"2024-09-18T07:51:06.625821+0000","last_active":"2024-09-18T07:52:05.272750+0000","last_peered":"2024-09-18T07:52:05.272750+0000","last_clean":"2024-09-18T07:52:05.272750+0000","last_became_active":"2024-09-18T07:50:23.292434+0000","last_became_peered":"2024-09-18T07:50:23.292434+0000","last_unstale":"2024-09-18T07:52:05.272750+0000","last_undegraded":"2024-09-18T07:52:05.272750+0000","last_fullsized":"2024-09-18T07:52:05.272750+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:06.625767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:06.625767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:31.857130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"568'39","reported_seq":764,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271762+0000","last_change":"2024-09-18T07:51:27.295746+0000","last_active":"2024-09-18T07:52:05.271762+0000","last_peered":"2024-09-18T07:52:05.271762+0000","last_clean":"2024-09-18T07:52:05.271762+0000","last_became_active":"2024-09-18T07:51:27.295067+0000","last_became_peered":"2024-09-18T07:51:27.295067+0000","last_unstale":"2024-09-18T07:52:05.271762+0000","last_undegraded":"2024-09-18T07:52:05.271762+0000","last_fullsized":"2024-09-18T07:52:05.271762+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'39","last_scrub_stamp":"2024-09-18T07:51:10.541004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:10.541004+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:26.190605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1068,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274072+0000","last_change":"2024-09-18T07:51:28.913904+0000","last_active":"2024-09-18T07:52:05.274072+0000","last_peered":"2024-09-18T07:52:05.274072+0000","last_clean":"2024-09-18T07:52:05.274072+0000","last_became_active":"2024-09-18T07:47:20.136358+0000","last_became_peered":"2024-09-18T07:47:20.136358+0000","last_unstale":"2024-09-18T07:52:05.274072+0000","last_undegraded":"2024-09-18T07:52:05.274072+0000","last_fullsized":"2024-09-18T07:52:05.274072+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:28.913848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:28.913848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:34.992822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":966,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591962+0000","last_change":"2024-09-18T07:51:47.388662+0000","last_active":"2024-09-18T07:52:05.591962+0000","last_peered":"2024-09-18T07:52:05.591962+0000","last_clean":"2024-09-18T07:52:05.591962+0000","last_became_active":"2024-09-18T07:51:45.206543+0000","last_became_peered":"2024-09-18T07:51:45.206543+0000","last_unstale":"2024-09-18T07:52:05.591962+0000","last_undegraded":"2024-09-18T07:52:05.591962+0000","last_fullsized":"2024-09-18T07:52:05.591962+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:47.388600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:47.388600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:02.712629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"568'21","reported_seq":1141,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274114+0000","last_change":"2024-09-18T07:51:22.852512+0000","last_active":"2024-09-18T07:52:05.274114+0000","last_peered":"2024-09-18T07:52:05.274114+0000","last_clean":"2024-09-18T07:52:05.274114+0000","last_became_active":"2024-09-18T07:51:09.220573+0000","last_became_peered":"2024-09-18T07:51:09.220573+0000","last_unstale":"2024-09-18T07:52:05.274114+0000","last_undegraded":"2024-09-18T07:52:05.274114+0000","last_fullsized":"2024-09-18T07:52:05.274114+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'21","last_scrub_stamp":"2024-09-18T07:51:22.852453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:22.852453+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:42.365900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1026,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590629+0000","last_change":"2024-09-18T07:52:04.543093+0000","last_active":"2024-09-18T07:52:05.590629+0000","last_peered":"2024-09-18T07:52:05.590629+0000","last_clean":"2024-09-18T07:52:05.590629+0000","last_became_active":"2024-09-18T07:51:09.215682+0000","last_became_peered":"2024-09-18T07:51:09.215682+0000","last_unstale":"2024-09-18T07:52:05.590629+0000","last_undegraded":"2024-09-18T07:52:05.590629+0000","last_fullsized":"2024-09-18T07:52:05.590629+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:52:04.543033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:52:04.543033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:07.543469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1131,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.408654+0000","last_change":"2024-09-18T07:51:16.704038+0000","last_active":"2024-09-18T07:52:05.408654+0000","last_peered":"2024-09-18T07:52:05.408654+0000","last_clean":"2024-09-18T07:52:05.408654+0000","last_became_active":"2024-09-18T07:51:15.237333+0000","last_became_peered":"2024-09-18T07:51:15.237333+0000","last_unstale":"2024-09-18T07:52:05.408654+0000","last_undegraded":"2024-09-18T07:52:05.408654+0000","last_fullsized":"2024-09-18T07:52:05.408654+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:16.703992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:16.703992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:19.370569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":925,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591632+0000","last_change":"2024-09-18T07:51:51.490569+0000","last_active":"2024-09-18T07:52:05.591632+0000","last_peered":"2024-09-18T07:52:05.591632+0000","last_clean":"2024-09-18T07:52:05.591632+0000","last_became_active":"2024-09-18T07:51:45.207136+0000","last_became_peered":"2024-09-18T07:51:45.207136+0000","last_unstale":"2024-09-18T07:52:05.591632+0000","last_undegraded":"2024-09-18T07:52:05.591632+0000","last_fullsized":"2024-09-18T07:52:05.591632+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:51.490504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:51.490504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:59.862474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"568'9","reported_seq":1173,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274339+0000","last_change":"2024-09-18T07:51:43.726184+0000","last_active":"2024-09-18T07:52:05.274339+0000","last_peered":"2024-09-18T07:52:05.274339+0000","last_clean":"2024-09-18T07:52:05.274339+0000","last_became_active":"2024-09-18T07:50:23.276419+0000","last_became_peered":"2024-09-18T07:50:23.276419+0000","last_unstale":"2024-09-18T07:52:05.274339+0000","last_undegraded":"2024-09-18T07:52:05.274339+0000","last_fullsized":"2024-09-18T07:52:05.274339+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'9","last_scrub_stamp":"2024-09-18T07:51:43.726114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:43.726114+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:50.468711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":933,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590676+0000","last_change":"2024-09-18T07:51:23.552640+0000","last_active":"2024-09-18T07:52:05.590676+0000","last_peered":"2024-09-18T07:52:05.590676+0000","last_clean":"2024-09-18T07:52:05.590676+0000","last_became_active":"2024-09-18T07:47:20.142891+0000","last_became_peered":"2024-09-18T07:47:20.142891+0000","last_unstale":"2024-09-18T07:52:05.590676+0000","last_undegraded":"2024-09-18T07:52:05.590676+0000","last_fullsized":"2024-09-18T07:52:05.590676+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:23.552591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:23.552591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:39.476156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1068,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590457+0000","last_change":"2024-09-18T07:51:43.372570+0000","last_active":"2024-09-18T07:52:05.590457+0000","last_peered":"2024-09-18T07:52:05.590457+0000","last_clean":"2024-09-18T07:52:05.590457+0000","last_became_active":"2024-09-18T07:50:23.271118+0000","last_became_peered":"2024-09-18T07:50:23.271118+0000","last_unstale":"2024-09-18T07:52:05.590457+0000","last_undegraded":"2024-09-18T07:52:05.590457+0000","last_fullsized":"2024-09-18T07:52:05.590457+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:43.372521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:43.372521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:06.391031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1092,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274752+0000","last_change":"2024-09-18T07:51:59.823768+0000","last_active":"2024-09-18T07:52:05.274752+0000","last_peered":"2024-09-18T07:52:05.274752+0000","last_clean":"2024-09-18T07:52:05.274752+0000","last_became_active":"2024-09-18T07:47:11.187916+0000","last_became_peered":"2024-09-18T07:47:11.187916+0000","last_unstale":"2024-09-18T07:52:05.274752+0000","last_undegraded":"2024-09-18T07:52:05.274752+0000","last_fullsized":"2024-09-18T07:52:05.274752+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:59.823701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:59.823701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:11.641203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":78,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.409252+0000","last_change":"2024-09-18T07:51:12.629631+0000","last_active":"2024-09-18T07:52:05.409252+0000","last_peered":"2024-09-18T07:52:05.409252+0000","last_clean":"2024-09-18T07:52:05.409252+0000","last_became_active":"2024-09-18T07:51:09.224720+0000","last_became_peered":"2024-09-18T07:51:09.224720+0000","last_unstale":"2024-09-18T07:52:05.409252+0000","last_undegraded":"2024-09-18T07:52:05.409252+0000","last_fullsized":"2024-09-18T07:52:05.409252+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:12.629579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:12.629579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:13.778419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"568'39","reported_seq":1357,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590014+0000","last_change":"2024-09-18T07:51:19.451785+0000","last_active":"2024-09-18T07:52:05.590014+0000","last_peered":"2024-09-18T07:52:05.590014+0000","last_clean":"2024-09-18T07:52:05.590014+0000","last_became_active":"2024-09-18T07:50:23.296232+0000","last_became_peered":"2024-09-18T07:50:23.296232+0000","last_unstale":"2024-09-18T07:52:05.590014+0000","last_undegraded":"2024-09-18T07:52:05.590014+0000","last_fullsized":"2024-09-18T07:52:05.590014+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'39","last_scrub_stamp":"2024-09-18T07:51:19.451724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:19.451724+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:34.161935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":734,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272225+0000","last_change":"2024-09-18T07:52:03.801574+0000","last_active":"2024-09-18T07:52:05.272225+0000","last_peered":"2024-09-18T07:52:05.272225+0000","last_clean":"2024-09-18T07:52:05.272225+0000","last_became_active":"2024-09-18T07:51:19.263415+0000","last_became_peered":"2024-09-18T07:51:19.263415+0000","last_unstale":"2024-09-18T07:52:05.272225+0000","last_undegraded":"2024-09-18T07:52:05.272225+0000","last_fullsized":"2024-09-18T07:52:05.272225+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:52:03.801509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:52:03.801509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:21.652578+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":953,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272726+0000","last_change":"2024-09-18T07:51:27.530215+0000","last_active":"2024-09-18T07:52:05.272726+0000","last_peered":"2024-09-18T07:52:05.272726+0000","last_clean":"2024-09-18T07:52:05.272726+0000","last_became_active":"2024-09-18T07:51:27.529908+0000","last_became_peered":"2024-09-18T07:51:27.529908+0000","last_unstale":"2024-09-18T07:52:05.272726+0000","last_undegraded":"2024-09-18T07:52:05.272726+0000","last_fullsized":"2024-09-18T07:52:05.272726+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:50.672909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:50:50.672909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:09.260524+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1083,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274324+0000","last_change":"2024-09-18T07:51:21.826034+0000","last_active":"2024-09-18T07:52:05.274324+0000","last_peered":"2024-09-18T07:52:05.274324+0000","last_clean":"2024-09-18T07:52:05.274324+0000","last_became_active":"2024-09-18T07:47:20.144620+0000","last_became_peered":"2024-09-18T07:47:20.144620+0000","last_unstale":"2024-09-18T07:52:05.274324+0000","last_undegraded":"2024-09-18T07:52:05.274324+0000","last_fullsized":"2024-09-18T07:52:05.274324+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:21.825986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:21.825986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:37.969195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"568'24","reported_seq":451,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272409+0000","last_change":"2024-09-18T07:51:27.297308+0000","last_active":"2024-09-18T07:52:05.272409+0000","last_peered":"2024-09-18T07:52:05.272409+0000","last_clean":"2024-09-18T07:52:05.272409+0000","last_became_active":"2024-09-18T07:51:27.296261+0000","last_became_peered":"2024-09-18T07:51:27.296261+0000","last_unstale":"2024-09-18T07:52:05.272409+0000","last_undegraded":"2024-09-18T07:52:05.272409+0000","last_fullsized":"2024-09-18T07:52:05.272409+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'24","last_scrub_stamp":"2024-09-18T07:51:24.882921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:24.882921+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:46.460599+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":31,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1011,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274513+0000","last_change":"2024-09-18T07:51:27.898744+0000","last_active":"2024-09-18T07:52:05.274513+0000","last_peered":"2024-09-18T07:52:05.274513+0000","last_clean":"2024-09-18T07:52:05.274513+0000","last_became_active":"2024-09-18T07:50:23.271208+0000","last_became_peered":"2024-09-18T07:50:23.271208+0000","last_unstale":"2024-09-18T07:52:05.274513+0000","last_undegraded":"2024-09-18T07:52:05.274513+0000","last_fullsized":"2024-09-18T07:52:05.274513+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:27.898672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:27.898672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:44.646916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":931,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590915+0000","last_change":"2024-09-18T07:51:16.577134+0000","last_active":"2024-09-18T07:52:05.590915+0000","last_peered":"2024-09-18T07:52:05.590915+0000","last_clean":"2024-09-18T07:52:05.590915+0000","last_became_active":"2024-09-18T07:47:11.194447+0000","last_became_peered":"2024-09-18T07:47:11.194447+0000","last_unstale":"2024-09-18T07:52:05.590915+0000","last_undegraded":"2024-09-18T07:52:05.590915+0000","last_fullsized":"2024-09-18T07:52:05.590915+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:51:16.577081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:16.577081+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:30.192910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":645,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.270697+0000","last_change":"2024-09-18T07:52:00.793949+0000","last_active":"2024-09-18T07:52:05.270697+0000","last_peered":"2024-09-18T07:52:05.270697+0000","last_clean":"2024-09-18T07:52:05.270697+0000","last_became_active":"2024-09-18T07:51:19.258923+0000","last_became_peered":"2024-09-18T07:51:19.258923+0000","last_unstale":"2024-09-18T07:52:05.270697+0000","last_undegraded":"2024-09-18T07:52:05.270697+0000","last_fullsized":"2024-09-18T07:52:05.270697+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:52:00.793895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:52:00.793895+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:10.220383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"568'30","reported_seq":965,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.409184+0000","last_change":"2024-09-18T07:51:25.208580+0000","last_active":"2024-09-18T07:52:05.409184+0000","last_peered":"2024-09-18T07:52:05.409184+0000","last_clean":"2024-09-18T07:52:05.409184+0000","last_became_active":"2024-09-18T07:51:25.201124+0000","last_became_peered":"2024-09-18T07:51:25.201124+0000","last_unstale":"2024-09-18T07:52:05.409184+0000","last_undegraded":"2024-09-18T07:52:05.409184+0000","last_fullsized":"2024-09-18T07:52:05.409184+0000","mapping_epoch":580,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":581,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'30","last_scrub_stamp":"2024-09-18T07:51:19.775996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:19.775996+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:36.992772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"398'14","reported_seq":966,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591819+0000","last_change":"2024-09-18T07:51:07.654295+0000","last_active":"2024-09-18T07:52:05.591819+0000","last_peered":"2024-09-18T07:52:05.591819+0000","last_clean":"2024-09-18T07:52:05.591819+0000","last_became_active":"2024-09-18T07:44:43.786594+0000","last_became_peered":"2024-09-18T07:44:43.786594+0000","last_unstale":"2024-09-18T07:52:05.591819+0000","last_undegraded":"2024-09-18T07:52:05.591819+0000","last_fullsized":"2024-09-18T07:52:05.591819+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:51:07.654244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:07.654244+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:28.946936+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"73'79","reported_seq":463,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591476+0000","last_change":"2024-09-18T07:51:45.209776+0000","last_active":"2024-09-18T07:52:05.591476+0000","last_peered":"2024-09-18T07:52:05.591476+0000","last_clean":"2024-09-18T07:52:05.591476+0000","last_became_active":"2024-09-18T07:51:45.209417+0000","last_became_peered":"2024-09-18T07:51:45.209417+0000","last_unstale":"2024-09-18T07:52:05.591476+0000","last_undegraded":"2024-09-18T07:52:05.591476+0000","last_fullsized":"2024-09-18T07:52:05.591476+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:51:24.460975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:51:24.460975+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:42.624370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"568'39","reported_seq":156,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271488+0000","last_change":"2024-09-18T07:51:19.890800+0000","last_active":"2024-09-18T07:52:05.271488+0000","last_peered":"2024-09-18T07:52:05.271488+0000","last_clean":"2024-09-18T07:52:05.271488+0000","last_became_active":"2024-09-18T07:51:19.261732+0000","last_became_peered":"2024-09-18T07:51:19.261732+0000","last_unstale":"2024-09-18T07:52:05.271488+0000","last_undegraded":"2024-09-18T07:52:05.271488+0000","last_fullsized":"2024-09-18T07:52:05.271488+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'39","last_scrub_stamp":"2024-09-18T07:51:19.890746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:19.890746+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:24.913065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":129,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271434+0000","last_change":"2024-09-18T07:51:32.524378+0000","last_active":"2024-09-18T07:52:05.271434+0000","last_peered":"2024-09-18T07:52:05.271434+0000","last_clean":"2024-09-18T07:52:05.271434+0000","last_became_active":"2024-09-18T07:50:23.298465+0000","last_became_peered":"2024-09-18T07:50:23.298465+0000","last_unstale":"2024-09-18T07:52:05.271434+0000","last_undegraded":"2024-09-18T07:52:05.271434+0000","last_fullsized":"2024-09-18T07:52:05.271434+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:32.524333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:32.524333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:50.995593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"571'14","reported_seq":1286,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274817+0000","last_change":"2024-09-18T07:51:53.776692+0000","last_active":"2024-09-18T07:52:05.274817+0000","last_peered":"2024-09-18T07:52:05.274817+0000","last_clean":"2024-09-18T07:52:05.274817+0000","last_became_active":"2024-09-18T07:51:09.215979+0000","last_became_peered":"2024-09-18T07:51:09.215979+0000","last_unstale":"2024-09-18T07:52:05.274817+0000","last_undegraded":"2024-09-18T07:52:05.274817+0000","last_fullsized":"2024-09-18T07:52:05.274817+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"571'14","last_scrub_stamp":"2024-09-18T07:51:53.776625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:53.776625+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:13.321194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":919,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592147+0000","last_change":"2024-09-18T07:51:45.209609+0000","last_active":"2024-09-18T07:52:05.592147+0000","last_peered":"2024-09-18T07:52:05.592147+0000","last_clean":"2024-09-18T07:52:05.592147+0000","last_became_active":"2024-09-18T07:51:45.209222+0000","last_became_peered":"2024-09-18T07:51:45.209222+0000","last_unstale":"2024-09-18T07:52:05.592147+0000","last_undegraded":"2024-09-18T07:52:05.592147+0000","last_fullsized":"2024-09-18T07:52:05.592147+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:37.831287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:37.831287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:00.161728+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"73'79","reported_seq":620,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274749+0000","last_change":"2024-09-18T07:52:00.834631+0000","last_active":"2024-09-18T07:52:05.274749+0000","last_peered":"2024-09-18T07:52:05.274749+0000","last_clean":"2024-09-18T07:52:05.274749+0000","last_became_active":"2024-09-18T07:44:43.786396+0000","last_became_peered":"2024-09-18T07:44:43.786396+0000","last_unstale":"2024-09-18T07:52:05.274749+0000","last_undegraded":"2024-09-18T07:52:05.274749+0000","last_fullsized":"2024-09-18T07:52:05.274749+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:52:00.834568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:52:00.834568+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:29.617773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"568'42","reported_seq":190,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271786+0000","last_change":"2024-09-18T07:51:15.596205+0000","last_active":"2024-09-18T07:52:05.271786+0000","last_peered":"2024-09-18T07:52:05.271786+0000","last_clean":"2024-09-18T07:52:05.271786+0000","last_became_active":"2024-09-18T07:51:09.235289+0000","last_became_peered":"2024-09-18T07:51:09.235289+0000","last_unstale":"2024-09-18T07:52:05.271786+0000","last_undegraded":"2024-09-18T07:52:05.271786+0000","last_fullsized":"2024-09-18T07:52:05.271786+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'42","last_scrub_stamp":"2024-09-18T07:51:15.596152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:15.596152+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:25.888748+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":870,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272731+0000","last_change":"2024-09-18T07:51:27.295782+0000","last_active":"2024-09-18T07:52:05.272731+0000","last_peered":"2024-09-18T07:52:05.272731+0000","last_clean":"2024-09-18T07:52:05.272731+0000","last_became_active":"2024-09-18T07:51:27.295491+0000","last_became_peered":"2024-09-18T07:51:27.295491+0000","last_unstale":"2024-09-18T07:52:05.272731+0000","last_undegraded":"2024-09-18T07:52:05.272731+0000","last_fullsized":"2024-09-18T07:52:05.272731+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:24.552207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:24.552207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:49.695764+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1078,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274075+0000","last_change":"2024-09-18T07:51:50.820516+0000","last_active":"2024-09-18T07:52:05.274075+0000","last_peered":"2024-09-18T07:52:05.274075+0000","last_clean":"2024-09-18T07:52:05.274075+0000","last_became_active":"2024-09-18T07:50:23.271640+0000","last_became_peered":"2024-09-18T07:50:23.271640+0000","last_unstale":"2024-09-18T07:52:05.274075+0000","last_undegraded":"2024-09-18T07:52:05.274075+0000","last_fullsized":"2024-09-18T07:52:05.274075+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:50.820446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:50.820446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:03.485644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":980,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274128+0000","last_change":"2024-09-18T07:50:47.845279+0000","last_active":"2024-09-18T07:52:05.274128+0000","last_peered":"2024-09-18T07:52:05.274128+0000","last_clean":"2024-09-18T07:52:05.274128+0000","last_became_active":"2024-09-18T07:50:23.279337+0000","last_became_peered":"2024-09-18T07:50:23.279337+0000","last_unstale":"2024-09-18T07:52:05.274128+0000","last_undegraded":"2024-09-18T07:52:05.274128+0000","last_fullsized":"2024-09-18T07:52:05.274128+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:47.845219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:50:47.845219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:10.259502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"73'79","reported_seq":133,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272211+0000","last_change":"2024-09-18T07:51:39.469646+0000","last_active":"2024-09-18T07:52:05.272211+0000","last_peered":"2024-09-18T07:52:05.272211+0000","last_clean":"2024-09-18T07:52:05.272211+0000","last_became_active":"2024-09-18T07:50:23.276444+0000","last_became_peered":"2024-09-18T07:50:23.276444+0000","last_unstale":"2024-09-18T07:52:05.272211+0000","last_undegraded":"2024-09-18T07:52:05.272211+0000","last_fullsized":"2024-09-18T07:52:05.272211+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:51:39.469567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:51:39.469567+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:01.652053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"568'42","reported_seq":1113,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274379+0000","last_change":"2024-09-18T07:51:54.765844+0000","last_active":"2024-09-18T07:52:05.274379+0000","last_peered":"2024-09-18T07:52:05.274379+0000","last_clean":"2024-09-18T07:52:05.274379+0000","last_became_active":"2024-09-18T07:47:11.188002+0000","last_became_peered":"2024-09-18T07:47:11.188002+0000","last_unstale":"2024-09-18T07:52:05.274379+0000","last_undegraded":"2024-09-18T07:52:05.274379+0000","last_fullsized":"2024-09-18T07:52:05.274379+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'42","last_scrub_stamp":"2024-09-18T07:51:54.765772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:54.765772+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:15.164539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":82,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.409293+0000","last_change":"2024-09-18T07:51:28.854565+0000","last_active":"2024-09-18T07:52:05.409293+0000","last_peered":"2024-09-18T07:52:05.409293+0000","last_clean":"2024-09-18T07:52:05.409293+0000","last_became_active":"2024-09-18T07:51:09.231383+0000","last_became_peered":"2024-09-18T07:51:09.231383+0000","last_unstale":"2024-09-18T07:52:05.409293+0000","last_undegraded":"2024-09-18T07:52:05.409293+0000","last_fullsized":"2024-09-18T07:52:05.409293+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:51:28.854514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:28.854514+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:44.856258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"548'15","reported_seq":154,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271689+0000","last_change":"2024-09-18T07:51:20.565974+0000","last_active":"2024-09-18T07:52:05.271689+0000","last_peered":"2024-09-18T07:52:05.271689+0000","last_clean":"2024-09-18T07:52:05.271689+0000","last_became_active":"2024-09-18T07:50:23.300057+0000","last_became_peered":"2024-09-18T07:50:23.300057+0000","last_unstale":"2024-09-18T07:52:05.271689+0000","last_undegraded":"2024-09-18T07:52:05.271689+0000","last_fullsized":"2024-09-18T07:52:05.271689+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'15","last_scrub_stamp":"2024-09-18T07:51:20.565926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:20.565926+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:49.557826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":932,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591907+0000","last_change":"2024-09-18T07:52:04.367414+0000","last_active":"2024-09-18T07:52:05.591907+0000","last_peered":"2024-09-18T07:52:05.591907+0000","last_clean":"2024-09-18T07:52:05.591907+0000","last_became_active":"2024-09-18T07:37:43.057060+0000","last_became_peered":"2024-09-18T07:37:43.057060+0000","last_unstale":"2024-09-18T07:52:05.591907+0000","last_undegraded":"2024-09-18T07:52:05.591907+0000","last_fullsized":"2024-09-18T07:52:05.591907+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:52:04.367352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:52:04.367352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:06.642826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"73'79","reported_seq":881,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591600+0000","last_change":"2024-09-18T07:51:25.200415+0000","last_active":"2024-09-18T07:52:05.591600+0000","last_peered":"2024-09-18T07:52:05.591600+0000","last_clean":"2024-09-18T07:52:05.591600+0000","last_became_active":"2024-09-18T07:51:25.200028+0000","last_became_peered":"2024-09-18T07:51:25.200028+0000","last_unstale":"2024-09-18T07:52:05.591600+0000","last_undegraded":"2024-09-18T07:52:05.591600+0000","last_fullsized":"2024-09-18T07:52:05.591600+0000","mapping_epoch":580,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":581,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:50:48.664637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:50:48.664637+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:14.535564+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"568'30","reported_seq":846,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.270726+0000","last_change":"2024-09-18T07:51:28.936119+0000","last_active":"2024-09-18T07:52:05.270726+0000","last_peered":"2024-09-18T07:52:05.270726+0000","last_clean":"2024-09-18T07:52:05.270726+0000","last_became_active":"2024-09-18T07:51:19.260618+0000","last_became_peered":"2024-09-18T07:51:19.260618+0000","last_unstale":"2024-09-18T07:52:05.270726+0000","last_undegraded":"2024-09-18T07:52:05.270726+0000","last_fullsized":"2024-09-18T07:52:05.270726+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'30","last_scrub_stamp":"2024-09-18T07:51:28.936068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:28.936068+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:38.606541+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":37,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1057,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591598+0000","last_change":"2024-09-18T07:51:46.357562+0000","last_active":"2024-09-18T07:52:05.591598+0000","last_peered":"2024-09-18T07:52:05.591598+0000","last_clean":"2024-09-18T07:52:05.591598+0000","last_became_active":"2024-09-18T07:50:23.268446+0000","last_became_peered":"2024-09-18T07:50:23.268446+0000","last_unstale":"2024-09-18T07:52:05.591598+0000","last_undegraded":"2024-09-18T07:52:05.591598+0000","last_fullsized":"2024-09-18T07:52:05.591598+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:46.357499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:46.357499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:14.017075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"575'17","reported_seq":931,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.408973+0000","last_change":"2024-09-18T07:51:20.747209+0000","last_active":"2024-09-18T07:52:05.408973+0000","last_peered":"2024-09-18T07:52:05.408973+0000","last_clean":"2024-09-18T07:52:05.408973+0000","last_became_active":"2024-09-18T07:51:15.235642+0000","last_became_peered":"2024-09-18T07:51:15.235642+0000","last_unstale":"2024-09-18T07:52:05.408973+0000","last_undegraded":"2024-09-18T07:52:05.408973+0000","last_fullsized":"2024-09-18T07:52:05.408973+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'17","last_scrub_stamp":"2024-09-18T07:51:20.747157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:20.747157+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:38.859938+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":125,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.408763+0000","last_change":"2024-09-18T07:51:15.236408+0000","last_active":"2024-09-18T07:52:05.408763+0000","last_peered":"2024-09-18T07:52:05.408763+0000","last_clean":"2024-09-18T07:52:05.408763+0000","last_became_active":"2024-09-18T07:51:15.235148+0000","last_became_peered":"2024-09-18T07:51:15.235148+0000","last_unstale":"2024-09-18T07:52:05.408763+0000","last_undegraded":"2024-09-18T07:52:05.408763+0000","last_fullsized":"2024-09-18T07:52:05.408763+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:09.666223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:09.666223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:34.536632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"73'79","reported_seq":903,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591296+0000","last_change":"2024-09-18T07:51:09.233169+0000","last_active":"2024-09-18T07:52:05.591296+0000","last_peered":"2024-09-18T07:52:05.591296+0000","last_clean":"2024-09-18T07:52:05.591296+0000","last_became_active":"2024-09-18T07:51:09.232811+0000","last_became_peered":"2024-09-18T07:51:09.232811+0000","last_unstale":"2024-09-18T07:52:05.591296+0000","last_undegraded":"2024-09-18T07:52:05.591296+0000","last_fullsized":"2024-09-18T07:52:05.591296+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:50:54.661812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:50:54.661812+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:23.439730+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"568'39","reported_seq":1017,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.408998+0000","last_change":"2024-09-18T07:51:38.873869+0000","last_active":"2024-09-18T07:52:05.408998+0000","last_peered":"2024-09-18T07:52:05.408998+0000","last_clean":"2024-09-18T07:52:05.408998+0000","last_became_active":"2024-09-18T07:51:15.234676+0000","last_became_peered":"2024-09-18T07:51:15.234676+0000","last_unstale":"2024-09-18T07:52:05.408998+0000","last_undegraded":"2024-09-18T07:52:05.408998+0000","last_fullsized":"2024-09-18T07:52:05.408998+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'39","last_scrub_stamp":"2024-09-18T07:51:38.873813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:38.873813+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:50.939603+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":632,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271213+0000","last_change":"2024-09-18T07:51:19.258009+0000","last_active":"2024-09-18T07:52:05.271213+0000","last_peered":"2024-09-18T07:52:05.271213+0000","last_clean":"2024-09-18T07:52:05.271213+0000","last_became_active":"2024-09-18T07:51:19.257463+0000","last_became_peered":"2024-09-18T07:51:19.257463+0000","last_unstale":"2024-09-18T07:52:05.271213+0000","last_undegraded":"2024-09-18T07:52:05.271213+0000","last_fullsized":"2024-09-18T07:52:05.271213+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:03.448570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:03.448570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:17.333594+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":809,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592495+0000","last_change":"2024-09-18T07:51:55.536453+0000","last_active":"2024-09-18T07:52:05.592495+0000","last_peered":"2024-09-18T07:52:05.592495+0000","last_clean":"2024-09-18T07:52:05.592495+0000","last_became_active":"2024-09-18T07:51:09.231928+0000","last_became_peered":"2024-09-18T07:51:09.231928+0000","last_unstale":"2024-09-18T07:52:05.592495+0000","last_undegraded":"2024-09-18T07:52:05.592495+0000","last_fullsized":"2024-09-18T07:52:05.592495+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:55.536393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:55.536393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:04.636419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":554,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.270587+0000","last_change":"2024-09-18T07:51:19.255681+0000","last_active":"2024-09-18T07:52:05.270587+0000","last_peered":"2024-09-18T07:52:05.270587+0000","last_clean":"2024-09-18T07:52:05.270587+0000","last_became_active":"2024-09-18T07:51:19.255390+0000","last_became_peered":"2024-09-18T07:51:19.255390+0000","last_unstale":"2024-09-18T07:52:05.270587+0000","last_undegraded":"2024-09-18T07:52:05.270587+0000","last_fullsized":"2024-09-18T07:52:05.270587+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:08.633382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:08.633382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:36.058930+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"73'79","reported_seq":506,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590525+0000","last_change":"2024-09-18T07:51:55.456646+0000","last_active":"2024-09-18T07:52:05.590525+0000","last_peered":"2024-09-18T07:52:05.590525+0000","last_clean":"2024-09-18T07:52:05.590525+0000","last_became_active":"2024-09-18T07:51:45.204912+0000","last_became_peered":"2024-09-18T07:51:45.204912+0000","last_unstale":"2024-09-18T07:52:05.590525+0000","last_undegraded":"2024-09-18T07:52:05.590525+0000","last_fullsized":"2024-09-18T07:52:05.590525+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:51:55.456551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:51:55.456551+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:56.445886+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"578'59","reported_seq":1286,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271107+0000","last_change":"2024-09-18T07:51:34.854066+0000","last_active":"2024-09-18T07:52:05.271107+0000","last_peered":"2024-09-18T07:52:05.271107+0000","last_clean":"2024-09-18T07:52:05.271107+0000","last_became_active":"2024-09-18T07:51:19.255807+0000","last_became_peered":"2024-09-18T07:51:19.255807+0000","last_unstale":"2024-09-18T07:52:05.271107+0000","last_undegraded":"2024-09-18T07:52:05.271107+0000","last_fullsized":"2024-09-18T07:52:05.271107+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'59","last_scrub_stamp":"2024-09-18T07:51:34.853966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:34.853966+0000","objects_scrubbed":10,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:02.277729+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":82,"num_read_kb":67,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":1790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1077,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274472+0000","last_change":"2024-09-18T07:51:30.905155+0000","last_active":"2024-09-18T07:52:05.274472+0000","last_peered":"2024-09-18T07:52:05.274472+0000","last_clean":"2024-09-18T07:52:05.274472+0000","last_became_active":"2024-09-18T07:51:09.216307+0000","last_became_peered":"2024-09-18T07:51:09.216307+0000","last_unstale":"2024-09-18T07:52:05.274472+0000","last_undegraded":"2024-09-18T07:52:05.274472+0000","last_fullsized":"2024-09-18T07:52:05.274472+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:30.905100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:30.905100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:59.784123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":926,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591346+0000","last_change":"2024-09-18T07:51:45.208985+0000","last_active":"2024-09-18T07:52:05.591346+0000","last_peered":"2024-09-18T07:52:05.591346+0000","last_clean":"2024-09-18T07:52:05.591346+0000","last_became_active":"2024-09-18T07:51:45.208101+0000","last_became_peered":"2024-09-18T07:51:45.208101+0000","last_unstale":"2024-09-18T07:52:05.591346+0000","last_undegraded":"2024-09-18T07:52:05.591346+0000","last_fullsized":"2024-09-18T07:52:05.591346+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-18T07:50:53.662326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:50:53.662326+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:13.723996+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00058211199999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":70,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.409075+0000","last_change":"2024-09-18T07:51:09.217314+0000","last_active":"2024-09-18T07:52:05.409075+0000","last_peered":"2024-09-18T07:52:05.409075+0000","last_clean":"2024-09-18T07:52:05.409075+0000","last_became_active":"2024-09-18T07:51:09.217019+0000","last_became_peered":"2024-09-18T07:51:09.217019+0000","last_unstale":"2024-09-18T07:52:05.409075+0000","last_undegraded":"2024-09-18T07:52:05.409075+0000","last_fullsized":"2024-09-18T07:52:05.409075+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:02.936812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:02.936812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:16.894474+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":129,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272712+0000","last_change":"2024-09-18T07:51:33.531280+0000","last_active":"2024-09-18T07:52:05.272712+0000","last_peered":"2024-09-18T07:52:05.272712+0000","last_clean":"2024-09-18T07:52:05.272712+0000","last_became_active":"2024-09-18T07:50:23.286499+0000","last_became_peered":"2024-09-18T07:50:23.286499+0000","last_unstale":"2024-09-18T07:52:05.272712+0000","last_undegraded":"2024-09-18T07:52:05.272712+0000","last_fullsized":"2024-09-18T07:52:05.272712+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:33.531237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:33.531237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:50.397177+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"73'79","reported_seq":1000,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590344+0000","last_change":"2024-09-18T07:51:30.424756+0000","last_active":"2024-09-18T07:52:05.590344+0000","last_peered":"2024-09-18T07:52:05.590344+0000","last_clean":"2024-09-18T07:52:05.590344+0000","last_became_active":"2024-09-18T07:51:09.218702+0000","last_became_peered":"2024-09-18T07:51:09.218702+0000","last_unstale":"2024-09-18T07:52:05.590344+0000","last_undegraded":"2024-09-18T07:52:05.590344+0000","last_fullsized":"2024-09-18T07:52:05.590344+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:51:30.424697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:51:30.424697+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:44.306670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"568'21","reported_seq":890,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271203+0000","last_change":"2024-09-18T07:51:19.264838+0000","last_active":"2024-09-18T07:52:05.271203+0000","last_peered":"2024-09-18T07:52:05.271203+0000","last_clean":"2024-09-18T07:52:05.271203+0000","last_became_active":"2024-09-18T07:51:19.263668+0000","last_became_peered":"2024-09-18T07:51:19.263668+0000","last_unstale":"2024-09-18T07:52:05.271203+0000","last_undegraded":"2024-09-18T07:52:05.271203+0000","last_fullsized":"2024-09-18T07:52:05.271203+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'21","last_scrub_stamp":"2024-09-18T07:51:12.053730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:12.053730+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:39.962516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":86,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590164+0000","last_change":"2024-09-18T07:51:11.569922+0000","last_active":"2024-09-18T07:52:05.590164+0000","last_peered":"2024-09-18T07:52:05.590164+0000","last_clean":"2024-09-18T07:52:05.590164+0000","last_became_active":"2024-09-18T07:51:01.009728+0000","last_became_peered":"2024-09-18T07:51:01.009728+0000","last_unstale":"2024-09-18T07:52:05.590164+0000","last_undegraded":"2024-09-18T07:52:05.590164+0000","last_fullsized":"2024-09-18T07:52:05.590164+0000","mapping_epoch":566,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":567,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:11.569873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:11.569873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:20.836170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":1209,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.589710+0000","last_change":"2024-09-18T07:51:13.550053+0000","last_active":"2024-09-18T07:52:05.589710+0000","last_peered":"2024-09-18T07:52:05.589710+0000","last_clean":"2024-09-18T07:52:05.589710+0000","last_became_active":"2024-09-18T07:51:09.239633+0000","last_became_peered":"2024-09-18T07:51:09.239633+0000","last_unstale":"2024-09-18T07:52:05.589710+0000","last_undegraded":"2024-09-18T07:52:05.589710+0000","last_fullsized":"2024-09-18T07:52:05.589710+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:51:13.550001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:51:13.550001+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:39.469350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0009943319999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1164,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.589958+0000","last_change":"2024-09-18T07:50:54.371523+0000","last_active":"2024-09-18T07:52:05.589958+0000","last_peered":"2024-09-18T07:52:05.589958+0000","last_clean":"2024-09-18T07:52:05.589958+0000","last_became_active":"2024-09-18T07:47:11.201379+0000","last_became_peered":"2024-09-18T07:47:11.201379+0000","last_unstale":"2024-09-18T07:52:05.589958+0000","last_undegraded":"2024-09-18T07:52:05.589958+0000","last_fullsized":"2024-09-18T07:52:05.589958+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:54.371475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:50:54.371475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:21.594190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"68'1","reported_seq":964,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590962+0000","last_change":"2024-09-18T07:52:00.316365+0000","last_active":"2024-09-18T07:52:05.590962+0000","last_peered":"2024-09-18T07:52:05.590962+0000","last_clean":"2024-09-18T07:52:05.590962+0000","last_became_active":"2024-09-18T07:50:23.296775+0000","last_became_peered":"2024-09-18T07:50:23.296775+0000","last_unstale":"2024-09-18T07:52:05.590962+0000","last_undegraded":"2024-09-18T07:52:05.590962+0000","last_fullsized":"2024-09-18T07:52:05.590962+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-18T07:52:00.316303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:52:00.316303+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:08.941866+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":91,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"73'79","reported_seq":667,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271235+0000","last_change":"2024-09-18T07:51:19.262419+0000","last_active":"2024-09-18T07:52:05.271235+0000","last_peered":"2024-09-18T07:52:05.271235+0000","last_clean":"2024-09-18T07:52:05.271235+0000","last_became_active":"2024-09-18T07:51:19.262081+0000","last_became_peered":"2024-09-18T07:51:19.262081+0000","last_unstale":"2024-09-18T07:52:05.271235+0000","last_undegraded":"2024-09-18T07:52:05.271235+0000","last_fullsized":"2024-09-18T07:52:05.271235+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":3,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:51:05.654368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:51:05.654368+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:30.880142+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"568'36","reported_seq":92,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.409361+0000","last_change":"2024-09-18T07:51:37.851879+0000","last_active":"2024-09-18T07:52:05.409361+0000","last_peered":"2024-09-18T07:52:05.409361+0000","last_clean":"2024-09-18T07:52:05.409361+0000","last_became_active":"2024-09-18T07:51:09.237471+0000","last_became_peered":"2024-09-18T07:51:09.237471+0000","last_unstale":"2024-09-18T07:52:05.409361+0000","last_undegraded":"2024-09-18T07:52:05.409361+0000","last_fullsized":"2024-09-18T07:52:05.409361+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'36","last_scrub_stamp":"2024-09-18T07:51:37.851824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:37.851824+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:50.507555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":93,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.408717+0000","last_change":"2024-09-18T07:51:41.811345+0000","last_active":"2024-09-18T07:52:05.408717+0000","last_peered":"2024-09-18T07:52:05.408717+0000","last_clean":"2024-09-18T07:52:05.408717+0000","last_became_active":"2024-09-18T07:51:25.199645+0000","last_became_peered":"2024-09-18T07:51:25.199645+0000","last_unstale":"2024-09-18T07:52:05.408717+0000","last_undegraded":"2024-09-18T07:52:05.408717+0000","last_fullsized":"2024-09-18T07:52:05.408717+0000","mapping_epoch":580,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":581,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:51:41.811295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:41.811295+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:48.574640+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":1983,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591645+0000","last_change":"2024-09-18T07:52:03.328063+0000","last_active":"2024-09-18T07:52:05.591645+0000","last_peered":"2024-09-18T07:52:05.591645+0000","last_clean":"2024-09-18T07:52:05.591645+0000","last_became_active":"2024-09-18T07:51:09.231463+0000","last_became_peered":"2024-09-18T07:51:09.231463+0000","last_unstale":"2024-09-18T07:52:05.591645+0000","last_undegraded":"2024-09-18T07:52:05.591645+0000","last_fullsized":"2024-09-18T07:52:05.591645+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-09-18T07:52:03.327997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:54.270008+0000","last_clean_scrub_stamp":"2024-09-18T07:52:03.327997+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:11.091518+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00063612499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1017,"num_read_kb":1012,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2334,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1051,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591874+0000","last_change":"2024-09-18T07:51:10.580404+0000","last_active":"2024-09-18T07:52:05.591874+0000","last_peered":"2024-09-18T07:52:05.591874+0000","last_clean":"2024-09-18T07:52:05.591874+0000","last_became_active":"2024-09-18T07:51:09.219290+0000","last_became_peered":"2024-09-18T07:51:09.219290+0000","last_unstale":"2024-09-18T07:52:05.591874+0000","last_undegraded":"2024-09-18T07:52:05.591874+0000","last_fullsized":"2024-09-18T07:52:05.591874+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:10.580357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:10.580357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:38.242597+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":655,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.270923+0000","last_change":"2024-09-18T07:51:19.257727+0000","last_active":"2024-09-18T07:52:05.270923+0000","last_peered":"2024-09-18T07:52:05.270923+0000","last_clean":"2024-09-18T07:52:05.270923+0000","last_became_active":"2024-09-18T07:51:19.257164+0000","last_became_peered":"2024-09-18T07:51:19.257164+0000","last_unstale":"2024-09-18T07:52:05.270923+0000","last_undegraded":"2024-09-18T07:52:05.270923+0000","last_fullsized":"2024-09-18T07:52:05.270923+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:06.543375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:06.543375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:16.657934+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"568'46","reported_seq":1016,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592557+0000","last_change":"2024-09-18T07:51:48.500801+0000","last_active":"2024-09-18T07:52:05.592557+0000","last_peered":"2024-09-18T07:52:05.592557+0000","last_clean":"2024-09-18T07:52:05.592557+0000","last_became_active":"2024-09-18T07:44:43.786827+0000","last_became_peered":"2024-09-18T07:44:43.786827+0000","last_unstale":"2024-09-18T07:52:05.592557+0000","last_undegraded":"2024-09-18T07:52:05.592557+0000","last_fullsized":"2024-09-18T07:52:05.592557+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'46","last_scrub_stamp":"2024-09-18T07:51:48.500713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:48.500713+0000","objects_scrubbed":8,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:05.288214+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":830,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272463+0000","last_change":"2024-09-18T07:51:27.530400+0000","last_active":"2024-09-18T07:52:05.272463+0000","last_peered":"2024-09-18T07:52:05.272463+0000","last_clean":"2024-09-18T07:52:05.272463+0000","last_became_active":"2024-09-18T07:51:27.530139+0000","last_became_peered":"2024-09-18T07:51:27.530139+0000","last_unstale":"2024-09-18T07:52:05.272463+0000","last_undegraded":"2024-09-18T07:52:05.272463+0000","last_fullsized":"2024-09-18T07:52:05.272463+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:39.627115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:50:39.627115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"73'79","reported_seq":983,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592286+0000","last_change":"2024-09-18T07:51:29.455174+0000","last_active":"2024-09-18T07:52:05.592286+0000","last_peered":"2024-09-18T07:52:05.592286+0000","last_clean":"2024-09-18T07:52:05.592286+0000","last_became_active":"2024-09-18T07:51:09.231028+0000","last_became_peered":"2024-09-18T07:51:09.231028+0000","last_unstale":"2024-09-18T07:52:05.592286+0000","last_undegraded":"2024-09-18T07:52:05.592286+0000","last_fullsized":"2024-09-18T07:52:05.592286+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:51:29.455113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:51:29.455113+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:49.429278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":921,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591657+0000","last_change":"2024-09-18T07:51:40.374754+0000","last_active":"2024-09-18T07:52:05.591657+0000","last_peered":"2024-09-18T07:52:05.591657+0000","last_clean":"2024-09-18T07:52:05.591657+0000","last_became_active":"2024-09-18T07:47:11.195084+0000","last_became_peered":"2024-09-18T07:47:11.195084+0000","last_unstale":"2024-09-18T07:52:05.591657+0000","last_undegraded":"2024-09-18T07:52:05.591657+0000","last_fullsized":"2024-09-18T07:52:05.591657+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:40.374696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:40.374696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:02.144236+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1040,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591703+0000","last_change":"2024-09-18T07:51:48.405578+0000","last_active":"2024-09-18T07:52:05.591703+0000","last_peered":"2024-09-18T07:52:05.591703+0000","last_clean":"2024-09-18T07:52:05.591703+0000","last_became_active":"2024-09-18T07:50:23.274710+0000","last_became_peered":"2024-09-18T07:50:23.274710+0000","last_unstale":"2024-09-18T07:52:05.591703+0000","last_undegraded":"2024-09-18T07:52:05.591703+0000","last_fullsized":"2024-09-18T07:52:05.591703+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:48.405459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:48.405459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:56.151375+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"568'27","reported_seq":1293,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274853+0000","last_change":"2024-09-18T07:52:04.815187+0000","last_active":"2024-09-18T07:52:05.274853+0000","last_peered":"2024-09-18T07:52:05.274853+0000","last_clean":"2024-09-18T07:52:05.274853+0000","last_became_active":"2024-09-18T07:49:32.684857+0000","last_became_peered":"2024-09-18T07:49:32.684857+0000","last_unstale":"2024-09-18T07:52:05.274853+0000","last_undegraded":"2024-09-18T07:52:05.274853+0000","last_fullsized":"2024-09-18T07:52:05.274853+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'27","last_scrub_stamp":"2024-09-18T07:52:04.815087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:52:04.815087+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:30.795151+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1026,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274274+0000","last_change":"2024-09-18T07:51:26.866971+0000","last_active":"2024-09-18T07:52:05.274274+0000","last_peered":"2024-09-18T07:52:05.274274+0000","last_clean":"2024-09-18T07:52:05.274274+0000","last_became_active":"2024-09-18T07:44:43.790160+0000","last_became_peered":"2024-09-18T07:44:43.790160+0000","last_unstale":"2024-09-18T07:52:05.274274+0000","last_undegraded":"2024-09-18T07:52:05.274274+0000","last_fullsized":"2024-09-18T07:52:05.274274+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:26.866900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:26.866900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:27.691180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":937,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591231+0000","last_change":"2024-09-18T07:51:58.503215+0000","last_active":"2024-09-18T07:52:05.591231+0000","last_peered":"2024-09-18T07:52:05.591231+0000","last_clean":"2024-09-18T07:52:05.591231+0000","last_became_active":"2024-09-18T07:51:45.206603+0000","last_became_peered":"2024-09-18T07:51:45.206603+0000","last_unstale":"2024-09-18T07:52:05.591231+0000","last_undegraded":"2024-09-18T07:52:05.591231+0000","last_fullsized":"2024-09-18T07:52:05.591231+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:58.503150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:58.503150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:23.150726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":953,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271637+0000","last_change":"2024-09-18T07:51:27.530235+0000","last_active":"2024-09-18T07:52:05.271637+0000","last_peered":"2024-09-18T07:52:05.271637+0000","last_clean":"2024-09-18T07:52:05.271637+0000","last_became_active":"2024-09-18T07:51:27.529923+0000","last_became_peered":"2024-09-18T07:51:27.529923+0000","last_unstale":"2024-09-18T07:52:05.271637+0000","last_undegraded":"2024-09-18T07:52:05.271637+0000","last_fullsized":"2024-09-18T07:52:05.271637+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:01.659774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:01.659774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:19.747393+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"73'79","reported_seq":475,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590240+0000","last_change":"2024-09-18T07:52:01.575294+0000","last_active":"2024-09-18T07:52:05.590240+0000","last_peered":"2024-09-18T07:52:05.590240+0000","last_clean":"2024-09-18T07:52:05.590240+0000","last_became_active":"2024-09-18T07:51:45.207398+0000","last_became_peered":"2024-09-18T07:51:45.207398+0000","last_unstale":"2024-09-18T07:52:05.590240+0000","last_undegraded":"2024-09-18T07:52:05.590240+0000","last_fullsized":"2024-09-18T07:52:05.590240+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:52:01.575223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:52:01.575223+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:11.256659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"568'45","reported_seq":1129,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274848+0000","last_change":"2024-09-18T07:51:51.869099+0000","last_active":"2024-09-18T07:52:05.274848+0000","last_peered":"2024-09-18T07:52:05.274848+0000","last_clean":"2024-09-18T07:52:05.274848+0000","last_became_active":"2024-09-18T07:47:20.132912+0000","last_became_peered":"2024-09-18T07:47:20.132912+0000","last_unstale":"2024-09-18T07:52:05.274848+0000","last_undegraded":"2024-09-18T07:52:05.274848+0000","last_fullsized":"2024-09-18T07:52:05.274848+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'45","last_scrub_stamp":"2024-09-18T07:51:51.869009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:51.869009+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:12.241662+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":830,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592488+0000","last_change":"2024-09-18T07:51:02.691728+0000","last_active":"2024-09-18T07:52:05.592488+0000","last_peered":"2024-09-18T07:52:05.592488+0000","last_clean":"2024-09-18T07:52:05.592488+0000","last_became_active":"2024-09-18T07:49:32.688352+0000","last_became_peered":"2024-09-18T07:49:32.688352+0000","last_unstale":"2024-09-18T07:52:05.592488+0000","last_undegraded":"2024-09-18T07:52:05.592488+0000","last_fullsized":"2024-09-18T07:52:05.592488+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:02.691674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:02.691674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:18.998767+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":709,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271348+0000","last_change":"2024-09-18T07:51:59.797142+0000","last_active":"2024-09-18T07:52:05.271348+0000","last_peered":"2024-09-18T07:52:05.271348+0000","last_clean":"2024-09-18T07:52:05.271348+0000","last_became_active":"2024-09-18T07:51:19.263461+0000","last_became_peered":"2024-09-18T07:51:19.263461+0000","last_unstale":"2024-09-18T07:52:05.271348+0000","last_undegraded":"2024-09-18T07:52:05.271348+0000","last_fullsized":"2024-09-18T07:52:05.271348+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:59.797094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:59.797094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:10.149459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1176,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.589993+0000","last_change":"2024-09-18T07:51:30.488108+0000","last_active":"2024-09-18T07:52:05.589993+0000","last_peered":"2024-09-18T07:52:05.589993+0000","last_clean":"2024-09-18T07:52:05.589993+0000","last_became_active":"2024-09-18T07:50:23.276876+0000","last_became_peered":"2024-09-18T07:50:23.276876+0000","last_unstale":"2024-09-18T07:52:05.589993+0000","last_undegraded":"2024-09-18T07:52:05.589993+0000","last_fullsized":"2024-09-18T07:52:05.589993+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:30.488061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:30.488061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:46.403015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"73'79","reported_seq":989,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590503+0000","last_change":"2024-09-18T07:51:57.288930+0000","last_active":"2024-09-18T07:52:05.590503+0000","last_peered":"2024-09-18T07:52:05.590503+0000","last_clean":"2024-09-18T07:52:05.590503+0000","last_became_active":"2024-09-18T07:50:23.295783+0000","last_became_peered":"2024-09-18T07:50:23.295783+0000","last_unstale":"2024-09-18T07:52:05.590503+0000","last_undegraded":"2024-09-18T07:52:05.590503+0000","last_fullsized":"2024-09-18T07:52:05.590503+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":272,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":4,"last_scrub":"73'79","last_scrub_stamp":"2024-09-18T07:51:57.288877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:28:21.738070+0000","last_clean_scrub_stamp":"2024-09-18T07:51:57.288877+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:15.479957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"568'30","reported_seq":1114,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274507+0000","last_change":"2024-09-18T07:50:46.869864+0000","last_active":"2024-09-18T07:52:05.274507+0000","last_peered":"2024-09-18T07:52:05.274507+0000","last_clean":"2024-09-18T07:52:05.274507+0000","last_became_active":"2024-09-18T07:50:23.278411+0000","last_became_peered":"2024-09-18T07:50:23.278411+0000","last_unstale":"2024-09-18T07:52:05.274507+0000","last_undegraded":"2024-09-18T07:52:05.274507+0000","last_fullsized":"2024-09-18T07:52:05.274507+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'20","last_scrub_stamp":"2024-09-18T07:50:46.869811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:50:46.869811+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:12.812560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":552,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272239+0000","last_change":"2024-09-18T07:51:27.296039+0000","last_active":"2024-09-18T07:52:05.272239+0000","last_peered":"2024-09-18T07:52:05.272239+0000","last_clean":"2024-09-18T07:52:05.272239+0000","last_became_active":"2024-09-18T07:51:27.295752+0000","last_became_peered":"2024-09-18T07:51:27.295752+0000","last_unstale":"2024-09-18T07:52:05.272239+0000","last_undegraded":"2024-09-18T07:52:05.272239+0000","last_fullsized":"2024-09-18T07:52:05.272239+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:17.733505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:17.733505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:40.432715+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"461'13","reported_seq":1448,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.589883+0000","last_change":"2024-09-18T07:51:45.574837+0000","last_active":"2024-09-18T07:52:05.589883+0000","last_peered":"2024-09-18T07:52:05.589883+0000","last_clean":"2024-09-18T07:52:05.589883+0000","last_became_active":"2024-09-18T07:47:11.203256+0000","last_became_peered":"2024-09-18T07:47:11.203256+0000","last_unstale":"2024-09-18T07:52:05.589883+0000","last_undegraded":"2024-09-18T07:52:05.589883+0000","last_fullsized":"2024-09-18T07:52:05.589883+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'13","last_scrub_stamp":"2024-09-18T07:51:45.574786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:45.574786+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:52.918180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1049,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.275040+0000","last_change":"2024-09-18T07:51:12.902793+0000","last_active":"2024-09-18T07:52:05.275040+0000","last_peered":"2024-09-18T07:52:05.275040+0000","last_clean":"2024-09-18T07:52:05.275040+0000","last_became_active":"2024-09-18T07:49:32.689562+0000","last_became_peered":"2024-09-18T07:49:32.689562+0000","last_unstale":"2024-09-18T07:52:05.275040+0000","last_undegraded":"2024-09-18T07:52:05.275040+0000","last_fullsized":"2024-09-18T07:52:05.275040+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:12.902740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:12.902740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:13.202651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"568'21","reported_seq":88,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.409215+0000","last_change":"2024-09-18T07:51:33.773423+0000","last_active":"2024-09-18T07:52:05.409215+0000","last_peered":"2024-09-18T07:52:05.409215+0000","last_clean":"2024-09-18T07:52:05.409215+0000","last_became_active":"2024-09-18T07:51:09.223848+0000","last_became_peered":"2024-09-18T07:51:09.223848+0000","last_unstale":"2024-09-18T07:52:05.409215+0000","last_undegraded":"2024-09-18T07:52:05.409215+0000","last_fullsized":"2024-09-18T07:52:05.409215+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'21","last_scrub_stamp":"2024-09-18T07:51:33.773353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:33.773353+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:00.548154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":963,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591555+0000","last_change":"2024-09-18T07:51:45.209235+0000","last_active":"2024-09-18T07:52:05.591555+0000","last_peered":"2024-09-18T07:52:05.591555+0000","last_clean":"2024-09-18T07:52:05.591555+0000","last_became_active":"2024-09-18T07:51:45.208865+0000","last_became_peered":"2024-09-18T07:51:45.208865+0000","last_unstale":"2024-09-18T07:52:05.591555+0000","last_undegraded":"2024-09-18T07:52:05.591555+0000","last_fullsized":"2024-09-18T07:52:05.591555+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:14.600447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:14.600447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:15.017387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1171,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.589677+0000","last_change":"2024-09-18T07:51:16.435171+0000","last_active":"2024-09-18T07:52:05.589677+0000","last_peered":"2024-09-18T07:52:05.589677+0000","last_clean":"2024-09-18T07:52:05.589677+0000","last_became_active":"2024-09-18T07:50:23.273703+0000","last_became_peered":"2024-09-18T07:50:23.273703+0000","last_unstale":"2024-09-18T07:52:05.589677+0000","last_undegraded":"2024-09-18T07:52:05.589677+0000","last_fullsized":"2024-09-18T07:52:05.589677+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:16.435124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:16.435124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:27.652986+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":604,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.408709+0000","last_change":"2024-09-18T07:51:34.818926+0000","last_active":"2024-09-18T07:52:05.408709+0000","last_peered":"2024-09-18T07:52:05.408709+0000","last_clean":"2024-09-18T07:52:05.408709+0000","last_became_active":"2024-09-18T07:51:15.234271+0000","last_became_peered":"2024-09-18T07:51:15.234271+0000","last_unstale":"2024-09-18T07:52:05.408709+0000","last_undegraded":"2024-09-18T07:52:05.408709+0000","last_fullsized":"2024-09-18T07:52:05.408709+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:34.818869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:34.818869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:51.813395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"568'51","reported_seq":903,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592634+0000","last_change":"2024-09-18T07:51:37.539716+0000","last_active":"2024-09-18T07:52:05.592634+0000","last_peered":"2024-09-18T07:52:05.592634+0000","last_clean":"2024-09-18T07:52:05.592634+0000","last_became_active":"2024-09-18T07:47:20.141942+0000","last_became_peered":"2024-09-18T07:47:20.141942+0000","last_unstale":"2024-09-18T07:52:05.592634+0000","last_undegraded":"2024-09-18T07:52:05.592634+0000","last_fullsized":"2024-09-18T07:52:05.592634+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'51","last_scrub_stamp":"2024-09-18T07:51:37.539625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:37.539625+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:41.599784+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":934,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591231+0000","last_change":"2024-09-18T07:51:36.438107+0000","last_active":"2024-09-18T07:52:05.591231+0000","last_peered":"2024-09-18T07:52:05.591231+0000","last_clean":"2024-09-18T07:52:05.591231+0000","last_became_active":"2024-09-18T07:47:11.194873+0000","last_became_peered":"2024-09-18T07:47:11.194873+0000","last_unstale":"2024-09-18T07:52:05.591231+0000","last_undegraded":"2024-09-18T07:52:05.591231+0000","last_fullsized":"2024-09-18T07:52:05.591231+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:36.438038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:36.438038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:05.058323+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1186,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.589715+0000","last_change":"2024-09-18T07:51:09.234135+0000","last_active":"2024-09-18T07:52:05.589715+0000","last_peered":"2024-09-18T07:52:05.589715+0000","last_clean":"2024-09-18T07:52:05.589715+0000","last_became_active":"2024-09-18T07:51:09.233826+0000","last_became_peered":"2024-09-18T07:51:09.233826+0000","last_unstale":"2024-09-18T07:52:05.589715+0000","last_undegraded":"2024-09-18T07:52:05.589715+0000","last_fullsized":"2024-09-18T07:52:05.589715+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:05.491144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:05.491144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:12.937636+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":923,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592275+0000","last_change":"2024-09-18T07:51:00.509832+0000","last_active":"2024-09-18T07:52:05.592275+0000","last_peered":"2024-09-18T07:52:05.592275+0000","last_clean":"2024-09-18T07:52:05.592275+0000","last_became_active":"2024-09-18T07:50:23.296866+0000","last_became_peered":"2024-09-18T07:50:23.296866+0000","last_unstale":"2024-09-18T07:52:05.592275+0000","last_undegraded":"2024-09-18T07:52:05.592275+0000","last_fullsized":"2024-09-18T07:52:05.592275+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:00.509766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:00.509766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:22.907420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"568'24","reported_seq":1175,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274155+0000","last_change":"2024-09-18T07:52:03.810149+0000","last_active":"2024-09-18T07:52:05.274155+0000","last_peered":"2024-09-18T07:52:05.274155+0000","last_clean":"2024-09-18T07:52:05.274155+0000","last_became_active":"2024-09-18T07:50:23.272111+0000","last_became_peered":"2024-09-18T07:50:23.272111+0000","last_unstale":"2024-09-18T07:52:05.274155+0000","last_undegraded":"2024-09-18T07:52:05.274155+0000","last_fullsized":"2024-09-18T07:52:05.274155+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'24","last_scrub_stamp":"2024-09-18T07:52:03.810083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:52:03.810083+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:20.792783+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":926,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592215+0000","last_change":"2024-09-18T07:51:27.507544+0000","last_active":"2024-09-18T07:52:05.592215+0000","last_peered":"2024-09-18T07:52:05.592215+0000","last_clean":"2024-09-18T07:52:05.592215+0000","last_became_active":"2024-09-18T07:47:20.138910+0000","last_became_peered":"2024-09-18T07:47:20.138910+0000","last_unstale":"2024-09-18T07:52:05.592215+0000","last_undegraded":"2024-09-18T07:52:05.592215+0000","last_fullsized":"2024-09-18T07:52:05.592215+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:27.507488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:27.507488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:50.641141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1196,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.589572+0000","last_change":"2024-09-18T07:51:17.405413+0000","last_active":"2024-09-18T07:52:05.589572+0000","last_peered":"2024-09-18T07:52:05.589572+0000","last_clean":"2024-09-18T07:52:05.589572+0000","last_became_active":"2024-09-18T07:51:09.231725+0000","last_became_peered":"2024-09-18T07:51:09.231725+0000","last_unstale":"2024-09-18T07:52:05.589572+0000","last_undegraded":"2024-09-18T07:52:05.589572+0000","last_fullsized":"2024-09-18T07:52:05.589572+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:17.405363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:17.405363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:24.592333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":129,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272470+0000","last_change":"2024-09-18T07:51:00.677134+0000","last_active":"2024-09-18T07:52:05.272470+0000","last_peered":"2024-09-18T07:52:05.272470+0000","last_clean":"2024-09-18T07:52:05.272470+0000","last_became_active":"2024-09-18T07:50:23.299077+0000","last_became_peered":"2024-09-18T07:50:23.299077+0000","last_unstale":"2024-09-18T07:52:05.272470+0000","last_undegraded":"2024-09-18T07:52:05.272470+0000","last_fullsized":"2024-09-18T07:52:05.272470+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:00.677086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:00.677086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:11.141258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"568'39","reported_seq":1148,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590108+0000","last_change":"2024-09-18T07:51:04.569896+0000","last_active":"2024-09-18T07:52:05.590108+0000","last_peered":"2024-09-18T07:52:05.590108+0000","last_clean":"2024-09-18T07:52:05.590108+0000","last_became_active":"2024-09-18T07:47:11.196288+0000","last_became_peered":"2024-09-18T07:47:11.196288+0000","last_unstale":"2024-09-18T07:52:05.590108+0000","last_undegraded":"2024-09-18T07:52:05.590108+0000","last_fullsized":"2024-09-18T07:52:05.590108+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'37","last_scrub_stamp":"2024-09-18T07:51:04.569839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:04.569839+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:11.165888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":960,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590925+0000","last_change":"2024-09-18T07:51:51.219166+0000","last_active":"2024-09-18T07:52:05.590925+0000","last_peered":"2024-09-18T07:52:05.590925+0000","last_clean":"2024-09-18T07:52:05.590925+0000","last_became_active":"2024-09-18T07:51:09.223041+0000","last_became_peered":"2024-09-18T07:51:09.223041+0000","last_unstale":"2024-09-18T07:52:05.590925+0000","last_undegraded":"2024-09-18T07:52:05.590925+0000","last_fullsized":"2024-09-18T07:52:05.590925+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-18T07:51:51.219100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:51.219100+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:14.396523+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"398'14","reported_seq":1162,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591610+0000","last_change":"2024-09-18T07:52:02.368992+0000","last_active":"2024-09-18T07:52:05.591610+0000","last_peered":"2024-09-18T07:52:05.591610+0000","last_clean":"2024-09-18T07:52:05.591610+0000","last_became_active":"2024-09-18T07:44:43.782355+0000","last_became_peered":"2024-09-18T07:44:43.782355+0000","last_unstale":"2024-09-18T07:52:05.591610+0000","last_undegraded":"2024-09-18T07:52:05.591610+0000","last_fullsized":"2024-09-18T07:52:05.591610+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14","last_scrub_stamp":"2024-09-18T07:52:02.368911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:52:02.368911+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:26.979454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":971,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591160+0000","last_change":"2024-09-18T07:52:02.555409+0000","last_active":"2024-09-18T07:52:05.591160+0000","last_peered":"2024-09-18T07:52:05.591160+0000","last_clean":"2024-09-18T07:52:05.591160+0000","last_became_active":"2024-09-18T07:51:45.207416+0000","last_became_peered":"2024-09-18T07:51:45.207416+0000","last_unstale":"2024-09-18T07:52:05.591160+0000","last_undegraded":"2024-09-18T07:52:05.591160+0000","last_fullsized":"2024-09-18T07:52:05.591160+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:52:02.555356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:52:02.555356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:26.036297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":792,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590195+0000","last_change":"2024-09-18T07:51:16.664570+0000","last_active":"2024-09-18T07:52:05.590195+0000","last_peered":"2024-09-18T07:52:05.590195+0000","last_clean":"2024-09-18T07:52:05.590195+0000","last_became_active":"2024-09-18T07:51:09.231456+0000","last_became_peered":"2024-09-18T07:51:09.231456+0000","last_unstale":"2024-09-18T07:52:05.590195+0000","last_undegraded":"2024-09-18T07:52:05.590195+0000","last_fullsized":"2024-09-18T07:52:05.590195+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:16.664498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:16.664498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:43.054012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"568'18","reported_seq":58,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272624+0000","last_change":"2024-09-18T07:51:25.206195+0000","last_active":"2024-09-18T07:52:05.272624+0000","last_peered":"2024-09-18T07:52:05.272624+0000","last_clean":"2024-09-18T07:52:05.272624+0000","last_became_active":"2024-09-18T07:51:25.200814+0000","last_became_peered":"2024-09-18T07:51:25.200814+0000","last_unstale":"2024-09-18T07:52:05.272624+0000","last_undegraded":"2024-09-18T07:52:05.272624+0000","last_fullsized":"2024-09-18T07:52:05.272624+0000","mapping_epoch":580,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":581,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'18","last_scrub_stamp":"2024-09-18T07:51:14.578829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:14.578829+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:24.169560+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1082,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591388+0000","last_change":"2024-09-18T07:51:39.334521+0000","last_active":"2024-09-18T07:52:05.591388+0000","last_peered":"2024-09-18T07:52:05.591388+0000","last_clean":"2024-09-18T07:52:05.591388+0000","last_became_active":"2024-09-18T07:50:23.273803+0000","last_became_peered":"2024-09-18T07:50:23.273803+0000","last_unstale":"2024-09-18T07:52:05.591388+0000","last_undegraded":"2024-09-18T07:52:05.591388+0000","last_fullsized":"2024-09-18T07:52:05.591388+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:39.334465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:39.334465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:05.600879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":926,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591082+0000","last_change":"2024-09-18T07:51:54.206541+0000","last_active":"2024-09-18T07:52:05.591082+0000","last_peered":"2024-09-18T07:52:05.591082+0000","last_clean":"2024-09-18T07:52:05.591082+0000","last_became_active":"2024-09-18T07:44:43.782348+0000","last_became_peered":"2024-09-18T07:44:43.782348+0000","last_unstale":"2024-09-18T07:52:05.591082+0000","last_undegraded":"2024-09-18T07:52:05.591082+0000","last_fullsized":"2024-09-18T07:52:05.591082+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:54.206480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:54.206480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:02.492004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"568'21","reported_seq":152,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.409261+0000","last_change":"2024-09-18T07:51:21.795622+0000","last_active":"2024-09-18T07:52:05.409261+0000","last_peered":"2024-09-18T07:52:05.409261+0000","last_clean":"2024-09-18T07:52:05.409261+0000","last_became_active":"2024-09-18T07:51:15.238300+0000","last_became_peered":"2024-09-18T07:51:15.238300+0000","last_unstale":"2024-09-18T07:52:05.409261+0000","last_undegraded":"2024-09-18T07:52:05.409261+0000","last_fullsized":"2024-09-18T07:52:05.409261+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'21","last_scrub_stamp":"2024-09-18T07:51:21.795567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:21.795567+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:33.896402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":942,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591238+0000","last_change":"2024-09-18T07:51:59.339423+0000","last_active":"2024-09-18T07:52:05.591238+0000","last_peered":"2024-09-18T07:52:05.591238+0000","last_clean":"2024-09-18T07:52:05.591238+0000","last_became_active":"2024-09-18T07:44:43.782930+0000","last_became_peered":"2024-09-18T07:44:43.782930+0000","last_unstale":"2024-09-18T07:52:05.591238+0000","last_undegraded":"2024-09-18T07:52:05.591238+0000","last_fullsized":"2024-09-18T07:52:05.591238+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:59.339361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:59.339361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:17.990759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":129,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272898+0000","last_change":"2024-09-18T07:51:25.497520+0000","last_active":"2024-09-18T07:52:05.272898+0000","last_peered":"2024-09-18T07:52:05.272898+0000","last_clean":"2024-09-18T07:52:05.272898+0000","last_became_active":"2024-09-18T07:50:23.274007+0000","last_became_peered":"2024-09-18T07:50:23.274007+0000","last_unstale":"2024-09-18T07:52:05.272898+0000","last_undegraded":"2024-09-18T07:52:05.272898+0000","last_fullsized":"2024-09-18T07:52:05.272898+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:25.497473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:25.497473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:30.968653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1143,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.589795+0000","last_change":"2024-09-18T07:51:08.465112+0000","last_active":"2024-09-18T07:52:05.589795+0000","last_peered":"2024-09-18T07:52:05.589795+0000","last_clean":"2024-09-18T07:52:05.589795+0000","last_became_active":"2024-09-18T07:44:43.790582+0000","last_became_peered":"2024-09-18T07:44:43.790582+0000","last_unstale":"2024-09-18T07:52:05.589795+0000","last_undegraded":"2024-09-18T07:52:05.589795+0000","last_fullsized":"2024-09-18T07:52:05.589795+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:08.465062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:08.465062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:25.231438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"568'21","reported_seq":837,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591523+0000","last_change":"2024-09-18T07:51:53.484445+0000","last_active":"2024-09-18T07:52:05.591523+0000","last_peered":"2024-09-18T07:52:05.591523+0000","last_clean":"2024-09-18T07:52:05.591523+0000","last_became_active":"2024-09-18T07:51:45.208309+0000","last_became_peered":"2024-09-18T07:51:45.208309+0000","last_unstale":"2024-09-18T07:52:05.591523+0000","last_undegraded":"2024-09-18T07:52:05.591523+0000","last_fullsized":"2024-09-18T07:52:05.591523+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'21","last_scrub_stamp":"2024-09-18T07:51:53.484377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:53.484377+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:02.233769+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":622,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.408855+0000","last_change":"2024-09-18T07:51:15.659362+0000","last_active":"2024-09-18T07:52:05.408855+0000","last_peered":"2024-09-18T07:52:05.408855+0000","last_clean":"2024-09-18T07:52:05.408855+0000","last_became_active":"2024-09-18T07:51:15.235143+0000","last_became_peered":"2024-09-18T07:51:15.235143+0000","last_unstale":"2024-09-18T07:52:05.408855+0000","last_undegraded":"2024-09-18T07:52:05.408855+0000","last_fullsized":"2024-09-18T07:52:05.408855+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:15.659312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:15.659312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:24.825353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"548'12","reported_seq":154,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272074+0000","last_change":"2024-09-18T07:51:38.507467+0000","last_active":"2024-09-18T07:52:05.272074+0000","last_peered":"2024-09-18T07:52:05.272074+0000","last_clean":"2024-09-18T07:52:05.272074+0000","last_became_active":"2024-09-18T07:50:23.294281+0000","last_became_peered":"2024-09-18T07:50:23.294281+0000","last_unstale":"2024-09-18T07:52:05.272074+0000","last_undegraded":"2024-09-18T07:52:05.272074+0000","last_fullsized":"2024-09-18T07:52:05.272074+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'12","last_scrub_stamp":"2024-09-18T07:51:38.507421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:38.507421+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:56.194027+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1031,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274923+0000","last_change":"2024-09-18T07:51:36.857221+0000","last_active":"2024-09-18T07:52:05.274923+0000","last_peered":"2024-09-18T07:52:05.274923+0000","last_clean":"2024-09-18T07:52:05.274923+0000","last_became_active":"2024-09-18T07:47:11.193273+0000","last_became_peered":"2024-09-18T07:47:11.193273+0000","last_unstale":"2024-09-18T07:52:05.274923+0000","last_undegraded":"2024-09-18T07:52:05.274923+0000","last_fullsized":"2024-09-18T07:52:05.274923+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:36.857151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:36.857151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:54.434263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"568'30","reported_seq":1011,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271731+0000","last_change":"2024-09-18T07:51:27.295557+0000","last_active":"2024-09-18T07:52:05.271731+0000","last_peered":"2024-09-18T07:52:05.271731+0000","last_clean":"2024-09-18T07:52:05.271731+0000","last_became_active":"2024-09-18T07:51:27.295220+0000","last_became_peered":"2024-09-18T07:51:27.295220+0000","last_unstale":"2024-09-18T07:52:05.271731+0000","last_undegraded":"2024-09-18T07:52:05.271731+0000","last_fullsized":"2024-09-18T07:52:05.271731+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'30","last_scrub_stamp":"2024-09-18T07:51:21.907205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:21.907205+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:24.368145+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":928,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592083+0000","last_change":"2024-09-18T07:51:56.270686+0000","last_active":"2024-09-18T07:52:05.592083+0000","last_peered":"2024-09-18T07:52:05.592083+0000","last_clean":"2024-09-18T07:52:05.592083+0000","last_became_active":"2024-09-18T07:44:43.779305+0000","last_became_peered":"2024-09-18T07:44:43.779305+0000","last_unstale":"2024-09-18T07:52:05.592083+0000","last_undegraded":"2024-09-18T07:52:05.592083+0000","last_fullsized":"2024-09-18T07:52:05.592083+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:56.270617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:56.270617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:20.479706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"571'17","reported_seq":1290,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592066+0000","last_change":"2024-09-18T07:51:56.454606+0000","last_active":"2024-09-18T07:52:05.592066+0000","last_peered":"2024-09-18T07:52:05.592066+0000","last_clean":"2024-09-18T07:52:05.592066+0000","last_became_active":"2024-09-18T07:51:09.215676+0000","last_became_peered":"2024-09-18T07:51:09.215676+0000","last_unstale":"2024-09-18T07:52:05.592066+0000","last_undegraded":"2024-09-18T07:52:05.592066+0000","last_fullsized":"2024-09-18T07:52:05.592066+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"571'17","last_scrub_stamp":"2024-09-18T07:51:56.454540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:56.454540+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:13.534025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":618,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.409046+0000","last_change":"2024-09-18T07:51:49.695350+0000","last_active":"2024-09-18T07:52:05.409046+0000","last_peered":"2024-09-18T07:52:05.409046+0000","last_clean":"2024-09-18T07:52:05.409046+0000","last_became_active":"2024-09-18T07:51:15.237465+0000","last_became_peered":"2024-09-18T07:51:15.237465+0000","last_unstale":"2024-09-18T07:52:05.409046+0000","last_undegraded":"2024-09-18T07:52:05.409046+0000","last_fullsized":"2024-09-18T07:52:05.409046+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:49.695309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:49.695309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:03.928177+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"568'27","reported_seq":1186,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274583+0000","last_change":"2024-09-18T07:51:35.897044+0000","last_active":"2024-09-18T07:52:05.274583+0000","last_peered":"2024-09-18T07:52:05.274583+0000","last_clean":"2024-09-18T07:52:05.274583+0000","last_became_active":"2024-09-18T07:47:20.140070+0000","last_became_peered":"2024-09-18T07:47:20.140070+0000","last_unstale":"2024-09-18T07:52:05.274583+0000","last_undegraded":"2024-09-18T07:52:05.274583+0000","last_fullsized":"2024-09-18T07:52:05.274583+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'27","last_scrub_stamp":"2024-09-18T07:51:35.896983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:35.896983+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:57.610961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1088,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.274655+0000","last_change":"2024-09-18T07:52:02.850609+0000","last_active":"2024-09-18T07:52:05.274655+0000","last_peered":"2024-09-18T07:52:05.274655+0000","last_clean":"2024-09-18T07:52:05.274655+0000","last_became_active":"2024-09-18T07:44:43.780305+0000","last_became_peered":"2024-09-18T07:44:43.780305+0000","last_unstale":"2024-09-18T07:52:05.274655+0000","last_undegraded":"2024-09-18T07:52:05.274655+0000","last_fullsized":"2024-09-18T07:52:05.274655+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:52:02.850552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:52:02.850552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:03.753388+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":592,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.270923+0000","last_change":"2024-09-18T07:51:26.856394+0000","last_active":"2024-09-18T07:52:05.270923+0000","last_peered":"2024-09-18T07:52:05.270923+0000","last_clean":"2024-09-18T07:52:05.270923+0000","last_became_active":"2024-09-18T07:51:25.200221+0000","last_became_peered":"2024-09-18T07:51:25.200221+0000","last_unstale":"2024-09-18T07:52:05.270923+0000","last_undegraded":"2024-09-18T07:52:05.270923+0000","last_fullsized":"2024-09-18T07:52:05.270923+0000","mapping_epoch":580,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":581,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:26.856334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:26.856334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:38.312784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":84,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271538+0000","last_change":"2024-09-18T07:51:27.567502+0000","last_active":"2024-09-18T07:52:05.271538+0000","last_peered":"2024-09-18T07:52:05.271538+0000","last_clean":"2024-09-18T07:52:05.271538+0000","last_became_active":"2024-09-18T07:51:27.345060+0000","last_became_peered":"2024-09-18T07:51:27.345060+0000","last_unstale":"2024-09-18T07:52:05.271538+0000","last_undegraded":"2024-09-18T07:52:05.271538+0000","last_fullsized":"2024-09-18T07:52:05.271538+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:27.567448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:27.567448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:48.530139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"568'21","reported_seq":828,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.270738+0000","last_change":"2024-09-18T07:51:23.889518+0000","last_active":"2024-09-18T07:52:05.270738+0000","last_peered":"2024-09-18T07:52:05.270738+0000","last_clean":"2024-09-18T07:52:05.270738+0000","last_became_active":"2024-09-18T07:51:19.256222+0000","last_became_peered":"2024-09-18T07:51:19.256222+0000","last_unstale":"2024-09-18T07:52:05.270738+0000","last_undegraded":"2024-09-18T07:52:05.270738+0000","last_fullsized":"2024-09-18T07:52:05.270738+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'21","last_scrub_stamp":"2024-09-18T07:51:23.889468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:23.889468+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:41.647872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":941,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591060+0000","last_change":"2024-09-18T07:51:39.418212+0000","last_active":"2024-09-18T07:52:05.591060+0000","last_peered":"2024-09-18T07:52:05.591060+0000","last_clean":"2024-09-18T07:52:05.591060+0000","last_became_active":"2024-09-18T07:50:23.296874+0000","last_became_peered":"2024-09-18T07:50:23.296874+0000","last_unstale":"2024-09-18T07:52:05.591060+0000","last_undegraded":"2024-09-18T07:52:05.591060+0000","last_fullsized":"2024-09-18T07:52:05.591060+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:39.418156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:39.418156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:03.930110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":120,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272395+0000","last_change":"2024-09-18T07:51:54.291807+0000","last_active":"2024-09-18T07:52:05.272395+0000","last_peered":"2024-09-18T07:52:05.272395+0000","last_clean":"2024-09-18T07:52:05.272395+0000","last_became_active":"2024-09-18T07:51:27.295722+0000","last_became_peered":"2024-09-18T07:51:27.295722+0000","last_unstale":"2024-09-18T07:52:05.272395+0000","last_undegraded":"2024-09-18T07:52:05.272395+0000","last_fullsized":"2024-09-18T07:52:05.272395+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:54.291761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:54.291761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:02.158477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":78,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.408920+0000","last_change":"2024-09-18T07:51:13.652044+0000","last_active":"2024-09-18T07:52:05.408920+0000","last_peered":"2024-09-18T07:52:05.408920+0000","last_clean":"2024-09-18T07:52:05.408920+0000","last_became_active":"2024-09-18T07:51:09.221267+0000","last_became_peered":"2024-09-18T07:51:09.221267+0000","last_unstale":"2024-09-18T07:52:05.408920+0000","last_undegraded":"2024-09-18T07:52:05.408920+0000","last_fullsized":"2024-09-18T07:52:05.408920+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:13.651996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:13.651996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:42.470038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"568'27","reported_seq":153,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272146+0000","last_change":"2024-09-18T07:51:28.550261+0000","last_active":"2024-09-18T07:52:05.272146+0000","last_peered":"2024-09-18T07:52:05.272146+0000","last_clean":"2024-09-18T07:52:05.272146+0000","last_became_active":"2024-09-18T07:50:23.389095+0000","last_became_peered":"2024-09-18T07:50:23.389095+0000","last_unstale":"2024-09-18T07:52:05.272146+0000","last_undegraded":"2024-09-18T07:52:05.272146+0000","last_fullsized":"2024-09-18T07:52:05.272146+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'27","last_scrub_stamp":"2024-09-18T07:51:28.550205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:28.550205+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:44.527319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1073,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591293+0000","last_change":"2024-09-18T07:51:02.632440+0000","last_active":"2024-09-18T07:52:05.591293+0000","last_peered":"2024-09-18T07:52:05.591293+0000","last_clean":"2024-09-18T07:52:05.591293+0000","last_became_active":"2024-09-18T07:47:20.140495+0000","last_became_peered":"2024-09-18T07:47:20.140495+0000","last_unstale":"2024-09-18T07:52:05.591293+0000","last_undegraded":"2024-09-18T07:52:05.591293+0000","last_fullsized":"2024-09-18T07:52:05.591293+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:02.632361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:02.632361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:32.171602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":934,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.592353+0000","last_change":"2024-09-18T07:51:01.482805+0000","last_active":"2024-09-18T07:52:05.592353+0000","last_peered":"2024-09-18T07:52:05.592353+0000","last_clean":"2024-09-18T07:52:05.592353+0000","last_became_active":"2024-09-18T07:44:43.778318+0000","last_became_peered":"2024-09-18T07:44:43.778318+0000","last_unstale":"2024-09-18T07:52:05.592353+0000","last_undegraded":"2024-09-18T07:52:05.592353+0000","last_fullsized":"2024-09-18T07:52:05.592353+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:01.482740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:01.482740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:19.740487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":136,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272313+0000","last_change":"2024-09-18T07:51:23.518872+0000","last_active":"2024-09-18T07:52:05.272313+0000","last_peered":"2024-09-18T07:52:05.272313+0000","last_clean":"2024-09-18T07:52:05.272313+0000","last_became_active":"2024-09-18T07:51:09.232703+0000","last_became_peered":"2024-09-18T07:51:09.232703+0000","last_unstale":"2024-09-18T07:52:05.272313+0000","last_undegraded":"2024-09-18T07:52:05.272313+0000","last_fullsized":"2024-09-18T07:52:05.272313+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:23.518811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:51:23.518811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:39.109440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"568'12","reported_seq":141,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271538+0000","last_change":"2024-09-18T07:51:59.244312+0000","last_active":"2024-09-18T07:52:05.271538+0000","last_peered":"2024-09-18T07:52:05.271538+0000","last_clean":"2024-09-18T07:52:05.271538+0000","last_became_active":"2024-09-18T07:50:23.298033+0000","last_became_peered":"2024-09-18T07:50:23.298033+0000","last_unstale":"2024-09-18T07:52:05.271538+0000","last_undegraded":"2024-09-18T07:52:05.271538+0000","last_fullsized":"2024-09-18T07:52:05.271538+0000","mapping_epoch":544,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'12","last_scrub_stamp":"2024-09-18T07:51:59.244260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:59.244260+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:01.683157+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":344,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.409207+0000","last_change":"2024-09-18T07:51:43.766097+0000","last_active":"2024-09-18T07:52:05.409207+0000","last_peered":"2024-09-18T07:52:05.409207+0000","last_clean":"2024-09-18T07:52:05.409207+0000","last_became_active":"2024-09-18T07:51:15.235647+0000","last_became_peered":"2024-09-18T07:51:15.235647+0000","last_unstale":"2024-09-18T07:52:05.409207+0000","last_undegraded":"2024-09-18T07:52:05.409207+0000","last_fullsized":"2024-09-18T07:52:05.409207+0000","mapping_epoch":573,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":574,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:43.766055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:43.766055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:02.535199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":771,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590503+0000","last_change":"2024-09-18T07:50:52.711210+0000","last_active":"2024-09-18T07:52:05.590503+0000","last_peered":"2024-09-18T07:52:05.590503+0000","last_clean":"2024-09-18T07:52:05.590503+0000","last_became_active":"2024-09-18T07:44:43.792641+0000","last_became_peered":"2024-09-18T07:44:43.792641+0000","last_unstale":"2024-09-18T07:52:05.590503+0000","last_undegraded":"2024-09-18T07:52:05.590503+0000","last_fullsized":"2024-09-18T07:52:05.590503+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:50:52.711146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:50:52.711146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:22.035108+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":920,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590684+0000","last_change":"2024-09-18T07:52:01.355114+0000","last_active":"2024-09-18T07:52:05.590684+0000","last_peered":"2024-09-18T07:52:05.590684+0000","last_clean":"2024-09-18T07:52:05.590684+0000","last_became_active":"2024-09-18T07:44:43.782378+0000","last_became_peered":"2024-09-18T07:44:43.782378+0000","last_unstale":"2024-09-18T07:52:05.590684+0000","last_undegraded":"2024-09-18T07:52:05.590684+0000","last_fullsized":"2024-09-18T07:52:05.590684+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:52:01.355061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:52:01.355061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:03.058506+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":489,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.272020+0000","last_change":"2024-09-18T07:52:02.787203+0000","last_active":"2024-09-18T07:52:05.272020+0000","last_peered":"2024-09-18T07:52:05.272020+0000","last_clean":"2024-09-18T07:52:05.272020+0000","last_became_active":"2024-09-18T07:51:19.260630+0000","last_became_peered":"2024-09-18T07:51:19.260630+0000","last_unstale":"2024-09-18T07:52:05.272020+0000","last_undegraded":"2024-09-18T07:52:05.272020+0000","last_fullsized":"2024-09-18T07:52:05.272020+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:52:02.787155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:31:01.319795+0000","last_clean_scrub_stamp":"2024-09-18T07:52:02.787155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:14.258870+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"61'1","reported_seq":144,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.271924+0000","last_change":"2024-09-18T07:51:09.250179+0000","last_active":"2024-09-18T07:52:05.271924+0000","last_peered":"2024-09-18T07:52:05.271924+0000","last_clean":"2024-09-18T07:52:05.271924+0000","last_became_active":"2024-09-18T07:51:09.234244+0000","last_became_peered":"2024-09-18T07:51:09.234244+0000","last_unstale":"2024-09-18T07:52:05.271924+0000","last_undegraded":"2024-09-18T07:52:05.271924+0000","last_fullsized":"2024-09-18T07:52:05.271924+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-18T07:51:03.659174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:55.271698+0000","last_clean_scrub_stamp":"2024-09-18T07:51:03.659174+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2979,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"568'27","reported_seq":851,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.591874+0000","last_change":"2024-09-18T07:51:34.554484+0000","last_active":"2024-09-18T07:52:05.591874+0000","last_peered":"2024-09-18T07:52:05.591874+0000","last_clean":"2024-09-18T07:52:05.591874+0000","last_became_active":"2024-09-18T07:47:20.142007+0000","last_became_peered":"2024-09-18T07:47:20.142007+0000","last_unstale":"2024-09-18T07:52:05.591874+0000","last_undegraded":"2024-09-18T07:52:05.591874+0000","last_fullsized":"2024-09-18T07:52:05.591874+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'27","last_scrub_stamp":"2024-09-18T07:51:34.554423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:57.286504+0000","last_clean_scrub_stamp":"2024-09-18T07:51:34.554423+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:43.542889+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":674,"reported_epoch":600,"state":"active+clean","last_fresh":"2024-09-18T07:52:05.590855+0000","last_change":"2024-09-18T07:51:46.239219+0000","last_active":"2024-09-18T07:52:05.590855+0000","last_peered":"2024-09-18T07:52:05.590855+0000","last_clean":"2024-09-18T07:52:05.590855+0000","last_became_active":"2024-09-18T07:47:20.139668+0000","last_became_peered":"2024-09-18T07:47:20.139668+0000","last_unstale":"2024-09-18T07:52:05.590855+0000","last_undegraded":"2024-09-18T07:52:05.590855+0000","last_fullsized":"2024-09-18T07:52:05.590855+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T07:51:46.239161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T07:30:59.305166+0000","last_clean_scrub_stamp":"2024-09-18T07:51:46.239161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:11.718206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1513,"num_read_kb":1270,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":684,"num_bytes_recovered":14924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},"log_size":978,"ondisk_log_size":978,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":4701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1017,"num_read_kb":1012,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2372,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":853056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201539,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744852,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93000300,"statfs":{"total":95995035648,"available":95232307200,"internally_reserved":0,"allocated":1323008,"data_stored":830478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":761199050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463177,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758392,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92986760,"statfs":{"total":95995035648,"available":95218442240,"internally_reserved":0,"allocated":966656,"data_stored":501493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":775420358},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":460,"seq":1975684956222,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":170240,"kb_used_data":1528,"kb_used_omap":6,"kb_used_meta":168505,"kb_avail":93574912,"statfs":{"total":95995035648,"available":95820709888,"internally_reserved":0,"allocated":1564672,"data_stored":1092317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6877,"internal_metadata":172549411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":395,"seq":1696512082011,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":534168,"kb_used_data":976,"kb_used_omap":4,"kb_used_meta":532987,"kb_avail":93210984,"statfs":{"total":95995035648,"available":95448047616,"internally_reserved":0,"allocated":999424,"data_stored":502631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4447,"internal_metadata":545779361},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149977,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1073324,"kb_used_data":203428,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92671828,"statfs":{"total":95995035648,"available":94895951872,"internally_reserved":0,"allocated":208310272,"data_stored":207794835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":890567111},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411615,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":786424,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92958728,"statfs":{"total":95995035648,"available":95189737472,"internally_reserved":0,"allocated":1228800,"data_stored":764426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575142,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888956,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92856196,"statfs":{"total":95995035648,"available":95084744704,"internally_reserved":0,"allocated":1036288,"data_stored":504368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738667,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902808,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92842344,"statfs":{"total":95995035648,"available":95070560256,"internally_reserved":0,"allocated":999424,"data_stored":503813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T07:52:11.188 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-18T07:52:11.188 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e shell --fsid f7ef099c-758e-11ef-bceb-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-18T07:52:11.441 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/mon.a/config 2024-09-18T07:52:11.562 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:11 smithi137 ceph-mon[29408]: pgmap v1152: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-18T07:52:11.563 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:11 smithi137 ceph-mon[39175]: pgmap v1152: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-18T07:52:11.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:11 smithi203 ceph-mon[32237]: pgmap v1152: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-18T07:52:12.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:52:12] ENGINE Bus STOPPING 2024-09-18T07:52:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[29408]: from='client.37061 v1:172.21.15.137:0/3163644913' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:52:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[29408]: 6.2 scrub starts 2024-09-18T07:52:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[29408]: 6.2 scrub ok 2024-09-18T07:52:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3022480201' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-18T07:52:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[29408]: from='client.? v1:172.21.15.137:0/3022480201' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-18T07:52:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:52:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:52:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:52:12.420 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[29408]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:52:12.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[39175]: from='client.37061 v1:172.21.15.137:0/3163644913' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:52:12.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[39175]: 6.2 scrub starts 2024-09-18T07:52:12.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[39175]: 6.2 scrub ok 2024-09-18T07:52:12.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3022480201' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-18T07:52:12.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[39175]: from='client.? v1:172.21.15.137:0/3022480201' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-18T07:52:12.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:52:12.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:52:12.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:52:12.421 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-mon[39175]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:52:12.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:12 smithi203 ceph-mon[32237]: from='client.37061 v1:172.21.15.137:0/3163644913' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-18T07:52:12.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:12 smithi203 ceph-mon[32237]: 6.2 scrub starts 2024-09-18T07:52:12.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:12 smithi203 ceph-mon[32237]: 6.2 scrub ok 2024-09-18T07:52:12.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:12 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3022480201' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-18T07:52:12.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:12 smithi203 ceph-mon[32237]: from='client.? v1:172.21.15.137:0/3022480201' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-18T07:52:12.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-18T07:52:12.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-18T07:52:12.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:12 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-18T07:52:12.712 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:12 smithi203 ceph-mon[32237]: from='mgr.14637 ' entity='mgr.y' 2024-09-18T07:52:12.714 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:52:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-18T07:52:12.714 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:52:12] ENGINE Bus STOPPED 2024-09-18T07:52:12.714 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:52:12] ENGINE Bus STARTING 2024-09-18T07:52:12.945 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-18T07:52:12.956 INFO:tasks.cephadm:Teardown begin 2024-09-18T07:52:12.957 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-18T07:52:12.978 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:52:12] ENGINE Serving on http://:::9283 2024-09-18T07:52:12.979 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:52:12 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: [18/Sep/2024:07:52:12] ENGINE Bus STARTED 2024-09-18T07:52:13.011 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-18T07:52:13.063 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-18T07:52:13.064 DEBUG:teuthology.orchestra.run.smithi137:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-18T07:52:13.090 DEBUG:teuthology.orchestra.run.smithi203:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-18T07:52:13.109 INFO:tasks.cephadm:Stopping all daemons... 2024-09-18T07:52:13.109 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-18T07:52:13.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mon.a 2024-09-18T07:52:13.296 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:13 smithi137 systemd[1]: Stopping Ceph mon.a for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:52:13.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:13 smithi137 ceph-mon[39175]: pgmap v1153: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:52:13.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:13 smithi137 ceph-mon[39175]: 6.1f scrub starts 2024-09-18T07:52:13.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:13 smithi137 ceph-mon[39175]: 6.1f scrub ok 2024-09-18T07:52:13.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:13 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:52:13.670 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:13 smithi137 ceph-mon[39175]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:52:13.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:13 smithi137 ceph-mon[29408]: pgmap v1153: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:52:13.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:13 smithi137 ceph-mon[29408]: 6.1f scrub starts 2024-09-18T07:52:13.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:13 smithi137 ceph-mon[29408]: 6.1f scrub ok 2024-09-18T07:52:13.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:13 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:52:13.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:13 smithi137 ceph-mon[29408]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:52:13.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a[29404]: 2024-09-18T07:52:13.468+0000 7fe59cbac640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T07:52:13.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:13 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a[29404]: 2024-09-18T07:52:13.468+0000 7fe59cbac640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-18T07:52:13.670 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:13 smithi137 podman[211338]: 2024-09-18 07:52:13.509523347 +0000 UTC m=+0.158335058 container died f2b0cbaedbeb412e09afc64593eb785cc0967881b3cb091f002c9e314a866268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-09-18T07:52:13.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:13 smithi203 ceph-mon[32237]: pgmap v1153: 142 pgs: 142 active+clean; 583 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-18T07:52:13.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:13 smithi203 ceph-mon[32237]: 6.1f scrub starts 2024-09-18T07:52:13.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:13 smithi203 ceph-mon[32237]: 6.1f scrub ok 2024-09-18T07:52:13.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:13 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-18T07:52:13.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:13 smithi203 ceph-mon[32237]: from='mgr.14637 v1:172.21.15.137:0/535977180' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-18T07:52:14.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:14 smithi137 podman[211338]: 2024-09-18 07:52:14.0187461 +0000 UTC m=+0.667557843 container cleanup f2b0cbaedbeb412e09afc64593eb785cc0967881b3cb091f002c9e314a866268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-09-18T07:52:14.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:14 smithi137 bash[211338]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a 2024-09-18T07:52:14.669 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 18 07:52:14 smithi137 podman[211351]: 2024-09-18 07:52:14.395835287 +0000 UTC m=+0.881235507 container remove f2b0cbaedbeb412e09afc64593eb785cc0967881b3cb091f002c9e314a866268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-a, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-18T07:52:14.711 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mon.a.service' 2024-09-18T07:52:14.763 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T07:52:14.764 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-18T07:52:14.764 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-18T07:52:14.764 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mon.c 2024-09-18T07:52:15.059 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:14 smithi137 systemd[1]: Stopping Ceph mon.c for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:52:15.402 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-c[39171]: 2024-09-18T07:52:15.056+0000 7f783d802640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T07:52:15.402 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:15 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-c[39171]: 2024-09-18T07:52:15.056+0000 7f783d802640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-18T07:52:15.402 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:15 smithi137 podman[211460]: 2024-09-18 07:52:15.098247622 +0000 UTC m=+0.150397125 container died 46d7d7550bc469526b0b5bd656c23b4b5dbd40c2b9c4e05aa6b09069b70ab5bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-18T07:52:15.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:15 smithi137 podman[211460]: 2024-09-18 07:52:15.499058255 +0000 UTC m=+0.551207759 container cleanup 46d7d7550bc469526b0b5bd656c23b4b5dbd40c2b9c4e05aa6b09069b70ab5bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-18T07:52:15.669 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 18 07:52:15 smithi137 bash[211460]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-c 2024-09-18T07:52:15.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mon.c.service' 2024-09-18T07:52:15.954 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T07:52:15.954 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-18T07:52:15.954 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-18T07:52:15.954 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mon.b 2024-09-18T07:52:16.266 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:15 smithi203 systemd[1]: Stopping Ceph mon.b for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:52:16.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:16 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-b[32233]: 2024-09-18T07:52:16.262+0000 7f8a72d76640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T07:52:16.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:16 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-b[32233]: 2024-09-18T07:52:16.262+0000 7f8a72d76640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-18T07:52:16.711 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:16 smithi203 podman[150854]: 2024-09-18 07:52:16.477620331 +0000 UTC m=+0.330267708 container died b63ad8ee863c70070b7130c5119f6279c2f73077d55481306af42c31719b243d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD) 2024-09-18T07:52:17.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:16 smithi203 podman[150854]: 2024-09-18 07:52:16.981223884 +0000 UTC m=+0.833871251 container cleanup b63ad8ee863c70070b7130c5119f6279c2f73077d55481306af42c31719b243d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-18T07:52:17.211 INFO:journalctl@ceph.mon.b.smithi203.stdout:Sep 18 07:52:16 smithi203 bash[150854]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mon-b 2024-09-18T07:52:17.406 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mon.b.service' 2024-09-18T07:52:17.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:52:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: ::ffff:172.21.15.203 - - [18/Sep/2024:07:52:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-18T07:52:17.443 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T07:52:17.443 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-18T07:52:17.443 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-18T07:52:17.444 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mgr.y 2024-09-18T07:52:17.755 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:52:17 smithi137 systemd[1]: Stopping Ceph mgr.y for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:52:18.082 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:52:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: 2024-09-18T07:52:17.753+0000 7f4668ae1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T07:52:18.082 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:52:17 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y[29635]: 2024-09-18T07:52:17.753+0000 7f4668ae1640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-18T07:52:18.082 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:52:17 smithi137 podman[211579]: 2024-09-18 07:52:17.784594085 +0000 UTC m=+0.145427005 container died a576bfc627b59d86cef5ef079d5e240e4528fb684c8f4c9f22e9c0461f2ecd66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-18T07:52:18.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:52:18 smithi137 podman[211579]: 2024-09-18 07:52:18.279367296 +0000 UTC m=+0.640200204 container cleanup a576bfc627b59d86cef5ef079d5e240e4528fb684c8f4c9f22e9c0461f2ecd66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7fb7b56dbae911c82c9a0310bb8cf8c37e8363e, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y, ceph=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-18T07:52:18.419 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 18 07:52:18 smithi137 bash[211579]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-y 2024-09-18T07:52:18.748 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-18T07:52:18.783 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T07:52:18.783 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-18T07:52:18.783 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-18T07:52:18.783 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mgr.x 2024-09-18T07:52:19.119 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:52:18 smithi203 systemd[1]: Stopping Ceph mgr.x for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:52:19.389 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:52:19 smithi203 podman[150976]: 2024-09-18 07:52:19.117359671 +0000 UTC m=+0.137507821 container died 1be2f12a125f85ae0c5740cdfc25bf253fd90a2f19fadda22684fbf8913df4a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-x, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T07:52:19.711 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:52:19 smithi203 podman[150976]: 2024-09-18 07:52:19.486307082 +0000 UTC m=+0.506455218 container cleanup 1be2f12a125f85ae0c5740cdfc25bf253fd90a2f19fadda22684fbf8913df4a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-18T07:52:19.711 INFO:journalctl@ceph.mgr.x.smithi203.stdout:Sep 18 07:52:19 smithi203 bash[150976]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-mgr-x 2024-09-18T07:52:20.157 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-18T07:52:20.193 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T07:52:20.193 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-18T07:52:20.193 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-18T07:52:20.193 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.0 2024-09-18T07:52:20.495 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:52:20 smithi137 systemd[1]: Stopping Ceph osd.0 for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:52:20.919 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:52:20 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-0[42480]: 2024-09-18T07:52:20.491+0000 7fb0e7c4e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T07:52:20.919 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:52:20 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-0[42480]: 2024-09-18T07:52:20.491+0000 7fb0e7c4e640 -1 osd.0 600 *** Got signal Terminated *** 2024-09-18T07:52:20.920 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:52:20 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-0[42480]: 2024-09-18T07:52:20.491+0000 7fb0e7c4e640 -1 osd.0 600 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T07:52:26.312 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:52:26 smithi137 podman[211699]: 2024-09-18 07:52:26.040972947 +0000 UTC m=+5.653935404 container died fc29a4ed444bfc139dc59d9c23c2b2ffe92d392e64bd628ec737d590b9f64e78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-18T07:52:26.669 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:52:26 smithi137 podman[211699]: 2024-09-18 07:52:26.501066889 +0000 UTC m=+6.114029377 container cleanup fc29a4ed444bfc139dc59d9c23c2b2ffe92d392e64bd628ec737d590b9f64e78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-0, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T07:52:26.669 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:52:26 smithi137 bash[211699]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-0 2024-09-18T07:52:27.113 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:52:26 smithi137 podman[211714]: 2024-09-18 07:52:26.802751072 +0000 UTC m=+0.756426263 container remove fc29a4ed444bfc139dc59d9c23c2b2ffe92d392e64bd628ec737d590b9f64e78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T07:52:27.419 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:52:27 smithi137 podman[211804]: 2024-09-18 07:52:27.014431062 +0000 UTC m=+0.015987828 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:52:27.419 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:52:27 smithi137 podman[211804]: 2024-09-18 07:52:27.120185187 +0000 UTC m=+0.121741990 container create 4b951094f5de849b6004aef52a2fe2ed43c0abfaff941fd6d380f05fb1727407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True) 2024-09-18T07:52:27.420 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:52:27 smithi137 podman[211804]: 2024-09-18 07:52:27.339307656 +0000 UTC m=+0.340864423 container init 4b951094f5de849b6004aef52a2fe2ed43c0abfaff941fd6d380f05fb1727407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-0-deactivate, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=) 2024-09-18T07:52:27.420 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:52:27 smithi137 podman[211804]: 2024-09-18 07:52:27.342575846 +0000 UTC m=+0.344132617 container start 4b951094f5de849b6004aef52a2fe2ed43c0abfaff941fd6d380f05fb1727407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-18T07:52:27.705 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:52:27 smithi137 podman[211804]: 2024-09-18 07:52:27.439192549 +0000 UTC m=+0.440749310 container attach 4b951094f5de849b6004aef52a2fe2ed43c0abfaff941fd6d380f05fb1727407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T07:52:27.955 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 18 07:52:27 smithi137 podman[211804]: 2024-09-18 07:52:27.703531683 +0000 UTC m=+0.705088454 container died 4b951094f5de849b6004aef52a2fe2ed43c0abfaff941fd6d380f05fb1727407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True) 2024-09-18T07:52:28.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.0.service' 2024-09-18T07:52:28.365 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T07:52:28.365 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-18T07:52:28.366 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-18T07:52:28.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.1 2024-09-18T07:52:28.669 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:52:28 smithi137 systemd[1]: Stopping Ceph osd.1 for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:52:29.169 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:52:28 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-1[48136]: 2024-09-18T07:52:28.703+0000 7fbad98ef640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T07:52:29.169 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:52:28 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-1[48136]: 2024-09-18T07:52:28.703+0000 7fbad98ef640 -1 osd.1 600 *** Got signal Terminated *** 2024-09-18T07:52:29.169 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:52:28 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-1[48136]: 2024-09-18T07:52:28.703+0000 7fbad98ef640 -1 osd.1 600 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T07:52:34.533 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:52:34 smithi137 podman[211908]: 2024-09-18 07:52:34.201046598 +0000 UTC m=+5.613331444 container stop 09802b98c316c6ccffbceb69f72b87909d4b4dbcf59355c0ad32a5cf1622db2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-1, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-18T07:52:34.533 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:52:34 smithi137 podman[211908]: 2024-09-18 07:52:34.230214097 +0000 UTC m=+5.642499016 container died 09802b98c316c6ccffbceb69f72b87909d4b4dbcf59355c0ad32a5cf1622db2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-18T07:52:34.919 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:52:34 smithi137 podman[211908]: 2024-09-18 07:52:34.746175691 +0000 UTC m=+6.158460524 container cleanup 09802b98c316c6ccffbceb69f72b87909d4b4dbcf59355c0ad32a5cf1622db2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-1, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-18T07:52:34.919 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:52:34 smithi137 bash[211908]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-1 2024-09-18T07:52:35.352 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:52:34 smithi137 podman[211936]: 2024-09-18 07:52:34.947690642 +0000 UTC m=+0.739852762 container remove 09802b98c316c6ccffbceb69f72b87909d4b4dbcf59355c0ad32a5cf1622db2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T07:52:35.628 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:52:35 smithi137 podman[212028]: 2024-09-18 07:52:35.252442754 +0000 UTC m=+0.016112110 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:52:35.628 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:52:35 smithi137 podman[212028]: 2024-09-18 07:52:35.373560178 +0000 UTC m=+0.137229535 container create 48793bef99410f5a994c166f388e41e6a0c98c3161ad84e01e5343744bf31a7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-18T07:52:35.919 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:52:35 smithi137 podman[212028]: 2024-09-18 07:52:35.626097823 +0000 UTC m=+0.389767187 container init 48793bef99410f5a994c166f388e41e6a0c98c3161ad84e01e5343744bf31a7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-18T07:52:35.920 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:52:35 smithi137 podman[212028]: 2024-09-18 07:52:35.629533258 +0000 UTC m=+0.393202615 container start 48793bef99410f5a994c166f388e41e6a0c98c3161ad84e01e5343744bf31a7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-09-18T07:52:35.920 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:52:35 smithi137 podman[212028]: 2024-09-18 07:52:35.718076136 +0000 UTC m=+0.481745501 container attach 48793bef99410f5a994c166f388e41e6a0c98c3161ad84e01e5343744bf31a7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-18T07:52:35.920 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 18 07:52:35 smithi137 podman[212028]: 2024-09-18 07:52:35.814515191 +0000 UTC m=+0.578184548 container died 48793bef99410f5a994c166f388e41e6a0c98c3161ad84e01e5343744bf31a7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-18T07:52:36.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.1.service' 2024-09-18T07:52:36.565 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T07:52:36.566 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-18T07:52:36.566 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-18T07:52:36.566 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.2 2024-09-18T07:52:36.887 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:52:36 smithi137 systemd[1]: Stopping Ceph osd.2 for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:52:37.169 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:52:36 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-2[53968]: 2024-09-18T07:52:36.882+0000 7f8d42dff640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T07:52:37.170 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:52:36 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-2[53968]: 2024-09-18T07:52:36.882+0000 7f8d42dff640 -1 osd.2 600 *** Got signal Terminated *** 2024-09-18T07:52:37.170 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:52:36 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-2[53968]: 2024-09-18T07:52:36.882+0000 7f8d42dff640 -1 osd.2 600 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T07:52:42.780 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:52:42 smithi137 podman[212129]: 2024-09-18 07:52:42.504133017 +0000 UTC m=+5.716882808 container died 413ec05f7695a510b263096dcadd12dfd23c682de77f40f42da6b34de7544f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-18T07:52:43.155 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:52:42 smithi137 podman[212129]: 2024-09-18 07:52:42.950525739 +0000 UTC m=+6.163275522 container cleanup 413ec05f7695a510b263096dcadd12dfd23c682de77f40f42da6b34de7544f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-18T07:52:43.155 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:52:42 smithi137 bash[212129]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-2 2024-09-18T07:52:43.419 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:52:43 smithi137 podman[212144]: 2024-09-18 07:52:43.151855937 +0000 UTC m=+0.642577373 container remove 413ec05f7695a510b263096dcadd12dfd23c682de77f40f42da6b34de7544f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-18T07:52:43.857 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:52:43 smithi137 podman[212234]: 2024-09-18 07:52:43.502081503 +0000 UTC m=+0.016173653 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:52:43.857 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:52:43 smithi137 podman[212234]: 2024-09-18 07:52:43.619382245 +0000 UTC m=+0.133474378 container create 5fe0dd20380059aadc0e71132d0a84d17f98dbc672932bb1622d44de6cad03a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-18T07:52:43.857 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:52:43 smithi137 podman[212234]: 2024-09-18 07:52:43.855138577 +0000 UTC m=+0.369230722 container init 5fe0dd20380059aadc0e71132d0a84d17f98dbc672932bb1622d44de6cad03a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True) 2024-09-18T07:52:44.169 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:52:43 smithi137 podman[212234]: 2024-09-18 07:52:43.858488285 +0000 UTC m=+0.372580422 container start 5fe0dd20380059aadc0e71132d0a84d17f98dbc672932bb1622d44de6cad03a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-09-18T07:52:44.170 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:52:43 smithi137 podman[212234]: 2024-09-18 07:52:43.938513882 +0000 UTC m=+0.452606017 container attach 5fe0dd20380059aadc0e71132d0a84d17f98dbc672932bb1622d44de6cad03a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-18T07:52:44.170 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 18 07:52:44 smithi137 podman[212234]: 2024-09-18 07:52:44.043726381 +0000 UTC m=+0.557818528 container died 5fe0dd20380059aadc0e71132d0a84d17f98dbc672932bb1622d44de6cad03a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T07:52:44.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.2.service' 2024-09-18T07:52:44.835 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T07:52:44.835 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-18T07:52:44.835 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-18T07:52:44.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.3 2024-09-18T07:52:45.169 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:52:44 smithi137 systemd[1]: Stopping Ceph osd.3 for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:52:45.669 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:52:45 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-3[59869]: 2024-09-18T07:52:45.170+0000 7f4fed984640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T07:52:45.669 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:52:45 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-3[59869]: 2024-09-18T07:52:45.170+0000 7f4fed984640 -1 osd.3 600 *** Got signal Terminated *** 2024-09-18T07:52:45.669 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:52:45 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-3[59869]: 2024-09-18T07:52:45.170+0000 7f4fed984640 -1 osd.3 600 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T07:52:46.961 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:46 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:46.700+0000 7faf266da640 -1 osd.4 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:20.998443+0000 front 2024-09-18T07:52:20.998411+0000 (oldest deadline 2024-09-18T07:52:46.298343+0000) 2024-09-18T07:52:46.961 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:46 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:46.677+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:52:47.669 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:52:47 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-3[59869]: 2024-09-18T07:52:47.401+0000 7f4fe9f99640 -1 osd.3 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:22.490578+0000 front 2024-09-18T07:52:22.490688+0000 (oldest deadline 2024-09-18T07:52:47.190663+0000) 2024-09-18T07:52:48.188 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:47.748+0000 7faf266da640 -1 osd.4 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:20.998443+0000 front 2024-09-18T07:52:20.998411+0000 (oldest deadline 2024-09-18T07:52:46.298343+0000) 2024-09-18T07:52:48.188 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:47 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:47.724+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:52:48.919 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:52:48 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-3[59869]: 2024-09-18T07:52:48.437+0000 7f4fe9f99640 -1 osd.3 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:22.490578+0000 front 2024-09-18T07:52:22.490688+0000 (oldest deadline 2024-09-18T07:52:47.190663+0000) 2024-09-18T07:52:48.960 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:48 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:48.704+0000 7faf266da640 -1 osd.4 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:20.998443+0000 front 2024-09-18T07:52:20.998411+0000 (oldest deadline 2024-09-18T07:52:46.298343+0000) 2024-09-18T07:52:48.961 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:48 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:48.716+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:52:49.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:52:49 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:52:49.303+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:52:49.919 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:52:49 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-3[59869]: 2024-09-18T07:52:49.447+0000 7f4fe9f99640 -1 osd.3 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:22.490578+0000 front 2024-09-18T07:52:22.490688+0000 (oldest deadline 2024-09-18T07:52:47.190663+0000) 2024-09-18T07:52:50.211 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:49 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:49.725+0000 7faf266da640 -1 osd.4 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:20.998443+0000 front 2024-09-18T07:52:20.998411+0000 (oldest deadline 2024-09-18T07:52:46.298343+0000) 2024-09-18T07:52:50.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:49 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:49.721+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:52:50.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:52:50 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:52:50.282+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:52:51.169 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:52:50 smithi137 podman[212347]: 2024-09-18 07:52:50.882406276 +0000 UTC m=+5.815541135 container died c43de787f262a1e0c291553f7bcb473a7686a8f7f7c341859aca6f48bb6cf1cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T07:52:51.210 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:50 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:50.710+0000 7faf266da640 -1 osd.4 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:20.998443+0000 front 2024-09-18T07:52:20.998411+0000 (oldest deadline 2024-09-18T07:52:46.298343+0000) 2024-09-18T07:52:51.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:50 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:50.751+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:52:51.615 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:52:51 smithi137 podman[212347]: 2024-09-18 07:52:51.345656414 +0000 UTC m=+6.278791302 container cleanup c43de787f262a1e0c291553f7bcb473a7686a8f7f7c341859aca6f48bb6cf1cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-18T07:52:51.616 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:52:51 smithi137 bash[212347]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-3 2024-09-18T07:52:51.697 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:52:51 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:52:51.240+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:52:51.919 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:52:51 smithi137 podman[212363]: 2024-09-18 07:52:51.613539444 +0000 UTC m=+0.721340490 container remove c43de787f262a1e0c291553f7bcb473a7686a8f7f7c341859aca6f48bb6cf1cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-3, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True) 2024-09-18T07:52:51.961 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:51 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:51.694+0000 7faf266da640 -1 osd.4 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:20.998443+0000 front 2024-09-18T07:52:20.998411+0000 (oldest deadline 2024-09-18T07:52:46.298343+0000) 2024-09-18T07:52:51.961 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:51 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:51.725+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:52:51.961 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:52:51 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:52:51.930+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:52:52.260 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:52:51 smithi137 podman[212450]: 2024-09-18 07:52:51.850737344 +0000 UTC m=+0.015646798 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:52:52.260 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:52:51 smithi137 podman[212450]: 2024-09-18 07:52:51.956062869 +0000 UTC m=+0.120972318 container create 683abbeefd73d581d8a01bc6b492ae204c283aed860af1ad3a44e6767a11f8d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T07:52:52.260 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:52:52 smithi137 podman[212450]: 2024-09-18 07:52:52.175041909 +0000 UTC m=+0.339951360 container init 683abbeefd73d581d8a01bc6b492ae204c283aed860af1ad3a44e6767a11f8d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-18T07:52:52.260 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:52:52 smithi137 podman[212450]: 2024-09-18 07:52:52.178768071 +0000 UTC m=+0.343677520 container start 683abbeefd73d581d8a01bc6b492ae204c283aed860af1ad3a44e6767a11f8d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T07:52:52.260 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:52:52 smithi137 podman[212450]: 2024-09-18 07:52:52.258170436 +0000 UTC m=+0.423079902 container attach 683abbeefd73d581d8a01bc6b492ae204c283aed860af1ad3a44e6767a11f8d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.schema-version=1.0) 2024-09-18T07:52:52.669 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 18 07:52:52 smithi137 podman[212450]: 2024-09-18 07:52:52.367593356 +0000 UTC m=+0.532502806 container died 683abbeefd73d581d8a01bc6b492ae204c283aed860af1ad3a44e6767a11f8d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-18T07:52:52.677 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:52:52 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:52:52.240+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:52:52.961 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:52:52 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:52:52.910+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:52:52.961 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:52 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:52.769+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:52:52.961 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:52 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:52.674+0000 7faf266da640 -1 osd.4 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:20.998443+0000 front 2024-09-18T07:52:20.998411+0000 (oldest deadline 2024-09-18T07:52:46.298343+0000) 2024-09-18T07:52:53.092 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.3.service' 2024-09-18T07:52:53.132 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T07:52:53.132 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-18T07:52:53.133 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-18T07:52:53.133 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.4 2024-09-18T07:52:53.433 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:53 smithi203 systemd[1]: Stopping Ceph osd.4 for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:52:53.434 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:52:53 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:52:53.262+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:52:53.711 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:53 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:53.430+0000 7faf2e0cd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T07:52:53.711 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:53 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:53.430+0000 7faf2e0cd640 -1 osd.4 600 *** Got signal Terminated *** 2024-09-18T07:52:53.711 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:53 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:53.430+0000 7faf2e0cd640 -1 osd.4 600 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T07:52:54.211 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:53 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:53.711+0000 7faf266da640 -1 osd.4 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:20.998443+0000 front 2024-09-18T07:52:20.998411+0000 (oldest deadline 2024-09-18T07:52:46.298343+0000) 2024-09-18T07:52:54.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:53 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:53.816+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:52:54.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:52:53 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:52:53.955+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:52:54.710 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:52:54 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:52:54.227+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:52:54.988 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:54 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:54.717+0000 7faf266da640 -1 osd.4 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:20.998443+0000 front 2024-09-18T07:52:20.998411+0000 (oldest deadline 2024-09-18T07:52:46.298343+0000) 2024-09-18T07:52:54.989 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:54 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:54.792+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:52:54.989 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:54 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:54.792+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:52:55.461 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:52:55 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:52:55.216+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:52:55.461 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:52:54 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:52:54.985+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:52:56.202 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:55 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:55.763+0000 7faf266da640 -1 osd.4 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:20.998443+0000 front 2024-09-18T07:52:20.998411+0000 (oldest deadline 2024-09-18T07:52:46.298343+0000) 2024-09-18T07:52:56.202 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:55 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:55.763+0000 7faf266da640 -1 osd.4 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:32.699710+0000 front 2024-09-18T07:52:32.699570+0000 (oldest deadline 2024-09-18T07:52:55.599550+0000) 2024-09-18T07:52:56.202 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:52:55 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:52:55.955+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:52:56.202 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:52:55 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:52:55.955+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:52:56.203 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:55 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:55.770+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:52:56.203 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:55 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:55.770+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:52:56.461 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:52:56 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:52:56.199+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:52:57.211 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:56 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:56.806+0000 7faf266da640 -1 osd.4 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:20.998443+0000 front 2024-09-18T07:52:20.998411+0000 (oldest deadline 2024-09-18T07:52:46.298343+0000) 2024-09-18T07:52:57.211 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:56 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:56.806+0000 7faf266da640 -1 osd.4 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:32.699710+0000 front 2024-09-18T07:52:32.699570+0000 (oldest deadline 2024-09-18T07:52:55.599550+0000) 2024-09-18T07:52:57.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:56 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:56.792+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:52:57.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:56 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:56.792+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:52:57.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:52:56 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:52:56.950+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:52:57.212 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:52:56 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:52:56.950+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:52:57.704 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:52:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:52:57.243+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:52:57.705 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:52:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:52:57.243+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.191059+0000 front 2024-09-18T07:52:33.191071+0000 (oldest deadline 2024-09-18T07:52:56.690941+0000) 2024-09-18T07:52:57.961 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:52:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:52:57.905+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:52:57.961 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:52:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:52:57.905+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:52:57.961 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:57.825+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:52:57.961 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:57.825+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:52:57.962 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:57.763+0000 7faf266da640 -1 osd.4 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:20.998443+0000 front 2024-09-18T07:52:20.998411+0000 (oldest deadline 2024-09-18T07:52:46.298343+0000) 2024-09-18T07:52:57.962 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4[118761]: 2024-09-18T07:52:57.763+0000 7faf266da640 -1 osd.4 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:32.699710+0000 front 2024-09-18T07:52:32.699570+0000 (oldest deadline 2024-09-18T07:52:55.599550+0000) 2024-09-18T07:52:58.461 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:52:58 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:52:58.206+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:52:58.461 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:52:58 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:52:58.206+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.191059+0000 front 2024-09-18T07:52:33.191071+0000 (oldest deadline 2024-09-18T07:52:56.690941+0000) 2024-09-18T07:52:59.201 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:59 smithi203 podman[151112]: 2024-09-18 07:52:59.048250678 +0000 UTC m=+5.715908328 container died 7912d2ab502400cac72c976c4d0a49bb8f7c0602cf9d784f0ab703b8bd8127dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T07:52:59.201 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:52:58 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:52:58.928+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:52:59.201 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:52:58 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:52:58.928+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:52:59.202 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:58 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:58.830+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:52:59.202 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:58 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:58.830+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:52:59.461 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:59 smithi203 podman[151112]: 2024-09-18 07:52:59.389613972 +0000 UTC m=+6.057271622 container cleanup 7912d2ab502400cac72c976c4d0a49bb8f7c0602cf9d784f0ab703b8bd8127dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5) 2024-09-18T07:52:59.461 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:59 smithi203 bash[151112]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4 2024-09-18T07:52:59.461 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:52:59 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:52:59.198+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:52:59.461 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:52:59 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:52:59.198+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.191059+0000 front 2024-09-18T07:52:33.191071+0000 (oldest deadline 2024-09-18T07:52:56.690941+0000) 2024-09-18T07:52:59.823 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:52:59 smithi203 podman[151128]: 2024-09-18 07:52:59.566191578 +0000 UTC m=+0.511762272 container remove 7912d2ab502400cac72c976c4d0a49bb8f7c0602cf9d784f0ab703b8bd8127dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_BRANCH=HEAD) 2024-09-18T07:53:00.136 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:52:59 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:52:59.908+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:53:00.136 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:52:59 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:52:59.908+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:53:00.136 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:59 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:59.820+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:00.137 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:52:59 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:52:59.820+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:00.137 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:53:00 smithi203 podman[151216]: 2024-09-18 07:52:59.904436808 +0000 UTC m=+0.016203229 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:53:00.137 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:53:00 smithi203 podman[151216]: 2024-09-18 07:53:00.008893313 +0000 UTC m=+0.120659723 container create 9aa199bd3c4e3370baaa4921e5d0f983bdab34d04e1bb44a7697aec95b1eba21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-deactivate, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-18T07:53:00.461 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:00 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:00.242+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:53:00.461 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:00 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:00.242+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.191059+0000 front 2024-09-18T07:52:33.191071+0000 (oldest deadline 2024-09-18T07:52:56.690941+0000) 2024-09-18T07:53:00.461 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:53:00 smithi203 podman[151216]: 2024-09-18 07:53:00.261732185 +0000 UTC m=+0.373498592 container init 9aa199bd3c4e3370baaa4921e5d0f983bdab34d04e1bb44a7697aec95b1eba21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-deactivate, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-18T07:53:00.461 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:53:00 smithi203 podman[151216]: 2024-09-18 07:53:00.265024134 +0000 UTC m=+0.376790541 container start 9aa199bd3c4e3370baaa4921e5d0f983bdab34d04e1bb44a7697aec95b1eba21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=) 2024-09-18T07:53:00.461 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:53:00 smithi203 podman[151216]: 2024-09-18 07:53:00.345021865 +0000 UTC m=+0.456788283 container attach 9aa199bd3c4e3370baaa4921e5d0f983bdab34d04e1bb44a7697aec95b1eba21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T07:53:00.817 INFO:journalctl@ceph.osd.4.smithi203.stdout:Sep 18 07:53:00 smithi203 podman[151216]: 2024-09-18 07:53:00.519107873 +0000 UTC m=+0.630874297 container died 9aa199bd3c4e3370baaa4921e5d0f983bdab34d04e1bb44a7697aec95b1eba21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0) 2024-09-18T07:53:01.093 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:00 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:00.867+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:53:01.093 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:00 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:00.867+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:53:01.093 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:00 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:00.830+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:01.093 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:00 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:00.830+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:01.115 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.4.service' 2024-09-18T07:53:01.155 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T07:53:01.155 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-18T07:53:01.156 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-18T07:53:01.156 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.5 2024-09-18T07:53:01.376 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:01 smithi203 systemd[1]: Stopping Ceph osd.5 for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:53:01.377 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:01 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:01.259+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:53:01.377 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:01 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:01.259+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.191059+0000 front 2024-09-18T07:52:33.191071+0000 (oldest deadline 2024-09-18T07:52:56.690941+0000) 2024-09-18T07:53:01.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:01 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:01.518+0000 7fb6cad4b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T07:53:01.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:01 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:01.518+0000 7fb6cad4b640 -1 osd.5 600 *** Got signal Terminated *** 2024-09-18T07:53:01.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:01 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:01.518+0000 7fb6cad4b640 -1 osd.5 600 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T07:53:02.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:01 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:01.912+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:53:02.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:01 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:01.912+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:53:02.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:01 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:01.865+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:02.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:01 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:01.865+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:02.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:02 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:02.307+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:53:02.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:02 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:02.307+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.191059+0000 front 2024-09-18T07:52:33.191071+0000 (oldest deadline 2024-09-18T07:52:56.690941+0000) 2024-09-18T07:53:03.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:02 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:02.882+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:03.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:02 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:02.882+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:03.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:02 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:02.886+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:53:03.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:02 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:02.886+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:53:03.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:03 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:03.300+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:53:03.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:03 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:03.300+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.191059+0000 front 2024-09-18T07:52:33.191071+0000 (oldest deadline 2024-09-18T07:52:56.690941+0000) 2024-09-18T07:53:04.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:03 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:03.935+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:53:04.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:03 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:03.935+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:53:04.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:03 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:03.922+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:04.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:03 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:03.922+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:04.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:04 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:04.328+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:53:04.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:04 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:04.328+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.191059+0000 front 2024-09-18T07:52:33.191071+0000 (oldest deadline 2024-09-18T07:52:56.690941+0000) 2024-09-18T07:53:05.210 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:04 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:04.900+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:53:05.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:04 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:04.900+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:53:05.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:04 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:04.929+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:05.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:04 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:04.929+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:05.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:04 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:04.929+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:05.710 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:05 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:05.284+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:53:05.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:05 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:05.284+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.191059+0000 front 2024-09-18T07:52:33.191071+0000 (oldest deadline 2024-09-18T07:52:56.690941+0000) 2024-09-18T07:53:06.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:05 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:05.909+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:53:06.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:05 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:05.909+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:53:06.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:05 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:05.950+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:06.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:05 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:05.950+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:06.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:05 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:05.950+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:06.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:06 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:06.299+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:23.289179+0000 front 2024-09-18T07:52:23.289346+0000 (oldest deadline 2024-09-18T07:52:48.589318+0000) 2024-09-18T07:53:06.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:06 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:06.299+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.191059+0000 front 2024-09-18T07:52:33.191071+0000 (oldest deadline 2024-09-18T07:52:56.690941+0000) 2024-09-18T07:53:06.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:06 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5[132119]: 2024-09-18T07:53:06.299+0000 7fb6c7360640 -1 osd.5 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.592971+0000 front 2024-09-18T07:52:40.593076+0000 (oldest deadline 2024-09-18T07:53:05.892881+0000) 2024-09-18T07:53:07.211 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:06 smithi203 podman[151319]: 2024-09-18 07:53:06.998472607 +0000 UTC m=+5.612823213 container died 04b85b2c8ec7bf5eeffebfe29fe3e2d7104d11e22dbb4ca625d84d19bba74f09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T07:53:07.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:06 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:06.859+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:53:07.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:06 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:06.859+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:53:07.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:06 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:06.859+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:41.179677+0000 front 2024-09-18T07:52:41.179667+0000 (oldest deadline 2024-09-18T07:53:06.479531+0000) 2024-09-18T07:53:07.212 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:06 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:06.928+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:07.212 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:06 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:06.928+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:07.212 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:06 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:06.928+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:07.656 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:07 smithi203 podman[151319]: 2024-09-18 07:53:07.386639509 +0000 UTC m=+6.000990119 container cleanup 04b85b2c8ec7bf5eeffebfe29fe3e2d7104d11e22dbb4ca625d84d19bba74f09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-18T07:53:07.668 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:07 smithi203 bash[151319]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5 2024-09-18T07:53:07.930 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:07 smithi203 podman[151349]: 2024-09-18 07:53:07.654104123 +0000 UTC m=+0.644032558 container remove 04b85b2c8ec7bf5eeffebfe29fe3e2d7104d11e22dbb4ca625d84d19bba74f09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-18T07:53:07.930 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:07.869+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:53:07.930 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:07.869+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:53:07.930 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:07.869+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:41.179677+0000 front 2024-09-18T07:52:41.179667+0000 (oldest deadline 2024-09-18T07:53:06.479531+0000) 2024-09-18T07:53:08.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:07.927+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:08.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:07.927+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:08.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:07 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:07.927+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:08.211 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:07 smithi203 podman[151435]: 2024-09-18 07:53:07.867568768 +0000 UTC m=+0.016077831 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:53:08.212 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:07 smithi203 podman[151435]: 2024-09-18 07:53:07.980649486 +0000 UTC m=+0.129158550 container create 563ab2672449ab68309d871304df155b88e3769d2a0a700f648917844216f779 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-18T07:53:08.711 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:08 smithi203 podman[151435]: 2024-09-18 07:53:08.21667301 +0000 UTC m=+0.365182073 container init 563ab2672449ab68309d871304df155b88e3769d2a0a700f648917844216f779 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-18T07:53:08.712 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:08 smithi203 podman[151435]: 2024-09-18 07:53:08.220635356 +0000 UTC m=+0.369144428 container start 563ab2672449ab68309d871304df155b88e3769d2a0a700f648917844216f779 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T07:53:08.712 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:08 smithi203 podman[151435]: 2024-09-18 07:53:08.299959124 +0000 UTC m=+0.448468206 container attach 563ab2672449ab68309d871304df155b88e3769d2a0a700f648917844216f779 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-18T07:53:08.712 INFO:journalctl@ceph.osd.5.smithi203.stdout:Sep 18 07:53:08 smithi203 podman[151435]: 2024-09-18 07:53:08.412268749 +0000 UTC m=+0.560777814 container died 563ab2672449ab68309d871304df155b88e3769d2a0a700f648917844216f779 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-18T07:53:09.023 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:08 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:08.899+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:53:09.023 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:08 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:08.899+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:53:09.023 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:08 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:08.899+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:41.179677+0000 front 2024-09-18T07:52:41.179667+0000 (oldest deadline 2024-09-18T07:53:06.479531+0000) 2024-09-18T07:53:09.023 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:08 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:08.892+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:09.023 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:08 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:08.892+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:09.023 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:08 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:08.892+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:09.043 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.5.service' 2024-09-18T07:53:09.084 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T07:53:09.084 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-18T07:53:09.084 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-18T07:53:09.084 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.6 2024-09-18T07:53:09.461 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:09 smithi203 systemd[1]: Stopping Ceph osd.6 for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:53:09.894 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:09.478+0000 7fe3e1f8d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T07:53:09.894 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:09.478+0000 7fe3e1f8d640 -1 osd.6 600 *** Got signal Terminated *** 2024-09-18T07:53:09.894 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:09.478+0000 7fe3e1f8d640 -1 osd.6 600 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T07:53:09.895 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:09.891+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:53:09.895 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:09.891+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:53:09.895 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:09.891+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:41.179677+0000 front 2024-09-18T07:52:41.179667+0000 (oldest deadline 2024-09-18T07:53:06.479531+0000) 2024-09-18T07:53:10.210 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:09.898+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:10.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:09.898+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:10.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:09 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:09.898+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:11.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:10.879+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:53:11.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:10.879+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:53:11.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:10.879+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:41.179677+0000 front 2024-09-18T07:52:41.179667+0000 (oldest deadline 2024-09-18T07:53:06.479531+0000) 2024-09-18T07:53:11.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:10.858+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:11.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:10.858+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:11.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:10 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:10.858+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:12.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:11 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:11.899+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:12.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:11 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:11.899+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:12.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:11 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:11.899+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:12.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:11 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:11.924+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:53:12.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:11 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:11.924+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:53:12.212 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:11 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:11.924+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:41.179677+0000 front 2024-09-18T07:52:41.179667+0000 (oldest deadline 2024-09-18T07:53:06.479531+0000) 2024-09-18T07:53:12.212 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:11 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:11.924+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6815 osd.3 since back 2024-09-18T07:52:46.480095+0000 front 2024-09-18T07:52:46.480011+0000 (oldest deadline 2024-09-18T07:53:11.779935+0000) 2024-09-18T07:53:12.711 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:12.337Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.137:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.137:8765: connect: connection refused" 2024-09-18T07:53:12.711 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:12.337Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.137:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.137:8765: connect: connection refused" 2024-09-18T07:53:12.711 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:12.338Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.137:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.137:8765: connect: connection refused" 2024-09-18T07:53:12.711 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:12.340Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.137:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.137:8765: connect: connection refused" 2024-09-18T07:53:12.711 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:12.340Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.137:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.137:8765: connect: connection refused" 2024-09-18T07:53:12.712 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:12.342Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.137:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.137:8765: connect: connection refused" 2024-09-18T07:53:12.712 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:12.343Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.137:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.137:8765: connect: connection refused" 2024-09-18T07:53:13.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:12.869+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:13.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:12.869+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:13.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:12.869+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:13.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:12.939+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:53:13.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:12.939+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:53:13.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:12.939+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:41.179677+0000 front 2024-09-18T07:52:41.179667+0000 (oldest deadline 2024-09-18T07:53:06.479531+0000) 2024-09-18T07:53:13.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:12 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:12.939+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6815 osd.3 since back 2024-09-18T07:52:46.480095+0000 front 2024-09-18T07:52:46.480011+0000 (oldest deadline 2024-09-18T07:53:11.779935+0000) 2024-09-18T07:53:14.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:13 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:13.910+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:14.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:13 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:13.910+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:14.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:13 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:13.910+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:14.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:13 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:13.978+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:25.477106+0000 front 2024-09-18T07:52:25.477064+0000 (oldest deadline 2024-09-18T07:52:51.377026+0000) 2024-09-18T07:53:14.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:13 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:13.978+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:33.578799+0000 front 2024-09-18T07:52:33.578786+0000 (oldest deadline 2024-09-18T07:52:55.878648+0000) 2024-09-18T07:53:14.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:13 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:13.978+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:41.179677+0000 front 2024-09-18T07:52:41.179667+0000 (oldest deadline 2024-09-18T07:53:06.479531+0000) 2024-09-18T07:53:14.212 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:13 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6[47108]: 2024-09-18T07:53:13.978+0000 7fe3da59a640 -1 osd.6 600 heartbeat_check: no reply from 172.21.15.137:6815 osd.3 since back 2024-09-18T07:52:46.480095+0000 front 2024-09-18T07:52:46.480011+0000 (oldest deadline 2024-09-18T07:53:11.779935+0000) 2024-09-18T07:53:15.211 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:15 smithi203 podman[151539]: 2024-09-18 07:53:15.112115255 +0000 UTC m=+5.809289930 container died b860df82fd385a4a27b99901d7f75f6ed0608293779253824a78715f65afb913 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , ceph=True) 2024-09-18T07:53:15.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:14 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:14.880+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:15.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:14 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:14.880+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:15.211 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:14 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:14.880+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:15.710 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:15 smithi203 podman[151539]: 2024-09-18 07:53:15.566403453 +0000 UTC m=+6.263578170 container cleanup b860df82fd385a4a27b99901d7f75f6ed0608293779253824a78715f65afb913 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T07:53:15.711 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:15 smithi203 bash[151539]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6 2024-09-18T07:53:16.061 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:15 smithi203 podman[151554]: 2024-09-18 07:53:15.743002305 +0000 UTC m=+0.634665282 container remove b860df82fd385a4a27b99901d7f75f6ed0608293779253824a78715f65afb913 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-18T07:53:16.062 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:15 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:15.910+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:16.062 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:15 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:15.910+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:16.062 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:15 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:15.910+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:16.062 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:15 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:15.910+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6815 osd.3 since back 2024-09-18T07:52:49.197936+0000 front 2024-09-18T07:52:49.197892+0000 (oldest deadline 2024-09-18T07:53:15.097842+0000) 2024-09-18T07:53:16.349 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:16 smithi203 podman[151643]: 2024-09-18 07:53:15.962054714 +0000 UTC m=+0.015857555 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:53:16.349 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:16 smithi203 podman[151643]: 2024-09-18 07:53:16.060487189 +0000 UTC m=+0.114290016 container create b2106cf4c5d62ff4c055ef32497cf3327b62a55a3e4e6f9817d3c979b832623c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, ceph=True) 2024-09-18T07:53:16.349 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:16 smithi203 podman[151643]: 2024-09-18 07:53:16.27184903 +0000 UTC m=+0.325651868 container init b2106cf4c5d62ff4c055ef32497cf3327b62a55a3e4e6f9817d3c979b832623c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-18T07:53:16.349 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:16 smithi203 podman[151643]: 2024-09-18 07:53:16.275186921 +0000 UTC m=+0.328989749 container start b2106cf4c5d62ff4c055ef32497cf3327b62a55a3e4e6f9817d3c979b832623c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6-deactivate, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-18T07:53:16.349 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:16 smithi203 podman[151643]: 2024-09-18 07:53:16.346886188 +0000 UTC m=+0.400689023 container attach b2106cf4c5d62ff4c055ef32497cf3327b62a55a3e4e6f9817d3c979b832623c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-18T07:53:16.711 INFO:journalctl@ceph.osd.6.smithi203.stdout:Sep 18 07:53:16 smithi203 podman[151643]: 2024-09-18 07:53:16.460672224 +0000 UTC m=+0.514475053 container died b2106cf4c5d62ff4c055ef32497cf3327b62a55a3e4e6f9817d3c979b832623c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-18T07:53:17.103 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:16 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:16.956+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:17.104 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:16 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:16.956+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:17.104 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:16 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:16.956+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:17.104 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:16 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:16.956+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6815 osd.3 since back 2024-09-18T07:52:49.197936+0000 front 2024-09-18T07:52:49.197892+0000 (oldest deadline 2024-09-18T07:53:15.097842+0000) 2024-09-18T07:53:17.128 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.6.service' 2024-09-18T07:53:17.167 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T07:53:17.168 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-18T07:53:17.168 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-18T07:53:17.168 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.7 2024-09-18T07:53:17.387 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:17 smithi203 systemd[1]: Stopping Ceph osd.7 for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:53:17.711 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:17.537+0000 7f9b8ccba640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T07:53:17.711 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:17.537+0000 7f9b8ccba640 -1 osd.7 600 *** Got signal Terminated *** 2024-09-18T07:53:17.711 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:17.537+0000 7f9b8ccba640 -1 osd.7 600 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-18T07:53:18.356 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:17.973+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:18.356 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:17.973+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:18.357 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:17.973+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:18.357 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:17 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:17.974+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6815 osd.3 since back 2024-09-18T07:52:49.197936+0000 front 2024-09-18T07:52:49.197892+0000 (oldest deadline 2024-09-18T07:53:15.097842+0000) 2024-09-18T07:53:19.461 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:19 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:19.022+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:19.461 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:19 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:19.022+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:19.461 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:19 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:19.022+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:19.461 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:19 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:19.022+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6815 osd.3 since back 2024-09-18T07:52:49.197936+0000 front 2024-09-18T07:52:49.197892+0000 (oldest deadline 2024-09-18T07:53:15.097842+0000) 2024-09-18T07:53:20.461 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:20 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:20.017+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:20.461 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:20 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:20.017+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:20.461 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:20 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:20.017+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:20.461 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:20 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:20.017+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6815 osd.3 since back 2024-09-18T07:52:49.197936+0000 front 2024-09-18T07:52:49.197892+0000 (oldest deadline 2024-09-18T07:53:15.097842+0000) 2024-09-18T07:53:21.461 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:21 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:21.031+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:21.461 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:21 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:21.031+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:21.461 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:21 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:21.031+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:21.461 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:21 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:21.031+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6815 osd.3 since back 2024-09-18T07:52:49.197936+0000 front 2024-09-18T07:52:49.197892+0000 (oldest deadline 2024-09-18T07:53:15.097842+0000) 2024-09-18T07:53:22.461 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:22 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:22.063+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6803 osd.0 since back 2024-09-18T07:52:24.694738+0000 front 2024-09-18T07:52:24.694681+0000 (oldest deadline 2024-09-18T07:52:46.394531+0000) 2024-09-18T07:53:22.461 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:22 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:22.063+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6807 osd.1 since back 2024-09-18T07:52:29.895429+0000 front 2024-09-18T07:52:29.895471+0000 (oldest deadline 2024-09-18T07:52:54.595403+0000) 2024-09-18T07:53:22.461 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:22 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:22.063+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6811 osd.2 since back 2024-09-18T07:52:40.396737+0000 front 2024-09-18T07:52:40.396668+0000 (oldest deadline 2024-09-18T07:53:04.496713+0000) 2024-09-18T07:53:22.461 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:22 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:22.063+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.137:6815 osd.3 since back 2024-09-18T07:52:49.197936+0000 front 2024-09-18T07:52:49.197892+0000 (oldest deadline 2024-09-18T07:53:15.097842+0000) 2024-09-18T07:53:22.461 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:22 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7[52723]: 2024-09-18T07:53:22.063+0000 7f9b892cf640 -1 osd.7 600 heartbeat_check: no reply from 172.21.15.203:6802 osd.4 since back 2024-09-18T07:52:57.998971+0000 front 2024-09-18T07:52:57.998834+0000 (oldest deadline 2024-09-18T07:53:21.498836+0000) 2024-09-18T07:53:23.281 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:22 smithi203 podman[151748]: 2024-09-18 07:53:22.97726938 +0000 UTC m=+5.572986060 container died 8eaef7effc059b791cef991c7146ea38235d9e754c5da50fcf8ee7298920fb0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-18T07:53:23.701 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:23 smithi203 podman[151748]: 2024-09-18 07:53:23.496876983 +0000 UTC m=+6.092593663 container cleanup 8eaef7effc059b791cef991c7146ea38235d9e754c5da50fcf8ee7298920fb0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-18T07:53:23.701 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:23 smithi203 bash[151748]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7 2024-09-18T07:53:23.960 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:23 smithi203 podman[151774]: 2024-09-18 07:53:23.698339825 +0000 UTC m=+0.719627085 container remove 8eaef7effc059b791cef991c7146ea38235d9e754c5da50fcf8ee7298920fb0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-18T07:53:24.430 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:24 smithi203 podman[151865]: 2024-09-18 07:53:24.020777658 +0000 UTC m=+0.016144513 image pull b22e1e37ddf1ca6e22136cd6540a9ab61e7cafb8bfbc7a91a2f20f7f9941bd9a quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9 2024-09-18T07:53:24.430 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:24 smithi203 podman[151865]: 2024-09-18 07:53:24.124257977 +0000 UTC m=+0.119624832 container create b11e5c4f509afc71b04be1b6c41df2748e961b3b8a2fc08788a1cc206b409a78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7-deactivate, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-18T07:53:24.430 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:24 smithi203 podman[151865]: 2024-09-18 07:53:24.343494907 +0000 UTC m=+0.338861763 container init b11e5c4f509afc71b04be1b6c41df2748e961b3b8a2fc08788a1cc206b409a78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5) 2024-09-18T07:53:24.430 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:24 smithi203 podman[151865]: 2024-09-18 07:53:24.34668821 +0000 UTC m=+0.342055068 container start b11e5c4f509afc71b04be1b6c41df2748e961b3b8a2fc08788a1cc206b409a78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-18T07:53:24.711 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:24 smithi203 podman[151865]: 2024-09-18 07:53:24.426821858 +0000 UTC m=+0.422188722 container attach b11e5c4f509afc71b04be1b6c41df2748e961b3b8a2fc08788a1cc206b409a78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-18T07:53:24.711 INFO:journalctl@ceph.osd.7.smithi203.stdout:Sep 18 07:53:24 smithi203 podman[151865]: 2024-09-18 07:53:24.543338626 +0000 UTC m=+0.538705481 container died b11e5c4f509afc71b04be1b6c41df2748e961b3b8a2fc08788a1cc206b409a78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0) 2024-09-18T07:53:25.162 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@osd.7.service' 2024-09-18T07:53:25.202 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T07:53:25.202 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-18T07:53:25.202 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-18T07:53:25.203 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-f7ef099c-758e-11ef-bceb-c7b262605968@rgw.foo.a 2024-09-18T07:53:25.497 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 18 07:53:25 smithi137 systemd[1]: Stopping Ceph rgw.foo.a for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:53:25.869 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 18 07:53:25 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-rgw-foo-a[64604]: 2024-09-18T07:53:25.493+0000 7fdd2a30d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-18T07:53:25.869 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 18 07:53:25 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-rgw-foo-a[64604]: 2024-09-18T07:53:25.493+0000 7fdd2db79880 -1 shutting down 2024-09-18T07:53:35.780 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 18 07:53:35 smithi137 bash[212552]: time="2024-09-18T07:53:35Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-f7ef099c-758e-11ef-bceb-c7b262605968-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-09-18T07:53:35.780 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 18 07:53:35 smithi137 podman[212552]: 2024-09-18 07:53:35.513071843 +0000 UTC m=+10.118870289 container died cb85a1a7c9de62e094f9bd54956ca7d943c0f85562d0ad2abce5b3ceff0b26a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-rgw-foo-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-18T07:53:36.129 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 18 07:53:35 smithi137 podman[212552]: 2024-09-18 07:53:35.981028324 +0000 UTC m=+10.586826807 container cleanup cb85a1a7c9de62e094f9bd54956ca7d943c0f85562d0ad2abce5b3ceff0b26a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-rgw-foo-a, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-18T07:53:36.129 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 18 07:53:35 smithi137 bash[212552]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-rgw-foo-a 2024-09-18T07:53:36.389 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 18 07:53:36 smithi137 podman[212579]: 2024-09-18 07:53:36.277216399 +0000 UTC m=+0.756257216 container remove cb85a1a7c9de62e094f9bd54956ca7d943c0f85562d0ad2abce5b3ceff0b26a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-rgw-foo-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-18T07:53:36.389 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 18 07:53:36 smithi137 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-09-18T07:53:36.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@rgw.foo.a.service' 2024-09-18T07:53:36.447 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T07:53:36.448 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-18T07:53:36.448 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-18T07:53:36.448 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-f7ef099c-758e-11ef-bceb-c7b262605968@prometheus.a 2024-09-18T07:53:36.751 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:36 smithi203 systemd[1]: Stopping Ceph prometheus.a for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:53:37.011 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:36 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:36.749Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-18T07:53:37.011 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:36 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:36.749Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-18T07:53:37.011 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:36 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:36.749Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-18T07:53:37.011 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:36 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:36.749Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-18T07:53:37.011 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:36 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:36.749Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-18T07:53:37.011 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:36 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:36.749Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-18T07:53:37.012 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:36 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:36.749Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-18T07:53:37.012 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:36 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:36.749Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-18T07:53:37.012 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:36 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:36.749Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-18T07:53:37.012 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:36 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:36.750Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-18T07:53:37.012 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:36 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:36.750Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-18T07:53:37.012 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:36 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a[65096]: ts=2024-09-18T07:53:36.750Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-18T07:53:37.012 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:36 smithi203 podman[151966]: 2024-09-18 07:53:36.759996163 +0000 UTC m=+0.109915650 container died 40a774eaf7fecea80fb1af75d5688351fbad311e6b076215cac95284558d16bd (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T07:53:37.386 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:37 smithi203 podman[151966]: 2024-09-18 07:53:37.199734058 +0000 UTC m=+0.549653585 container cleanup 40a774eaf7fecea80fb1af75d5688351fbad311e6b076215cac95284558d16bd (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-18T07:53:37.387 INFO:journalctl@ceph.prometheus.a.smithi203.stdout:Sep 18 07:53:37 smithi203 bash[151966]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-prometheus-a 2024-09-18T07:53:37.600 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f7ef099c-758e-11ef-bceb-c7b262605968@prometheus.a.service' 2024-09-18T07:53:37.634 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-18T07:53:37.634 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-18T07:53:37.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f7ef099c-758e-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-18T07:53:37.836 INFO:teuthology.orchestra.run.smithi137.stdout:Deleting cluster with fsid: f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:53:39.758 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:53:39 smithi137 systemd[1]: Stopping Ceph alertmanager.a for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:53:39.758 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:53:39 smithi137 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a[80142]: ts=2024-09-18T07:53:39.714Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-18T07:53:39.758 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:53:39 smithi137 podman[213051]: 2024-09-18 07:53:39.755609466 +0000 UTC m=+0.146945454 container died b9bd9f49497c2c68d11bab42772f084db06529c5afd0bb679e6d6d5c4a7e1d13 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T07:53:40.419 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:53:40 smithi137 podman[213051]: 2024-09-18 07:53:40.20654441 +0000 UTC m=+0.597880437 container cleanup b9bd9f49497c2c68d11bab42772f084db06529c5afd0bb679e6d6d5c4a7e1d13 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T07:53:40.419 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:53:40 smithi137 bash[213051]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a 2024-09-18T07:53:40.912 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:53:40 smithi137 podman[213062]: 2024-09-18 07:53:40.516138753 +0000 UTC m=+0.755084218 container remove b9bd9f49497c2c68d11bab42772f084db06529c5afd0bb679e6d6d5c4a7e1d13 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-18T07:53:40.912 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:53:40 smithi137 podman[213062]: 2024-09-18 07:53:40.592031351 +0000 UTC m=+0.830976768 volume remove 5d6bfb21fbab9079fde370126939652fba56fcf9d542827c00ee40995f6d77bc 2024-09-18T07:53:40.912 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:53:40 smithi137 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-18T07:53:40.912 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:53:40 smithi137 systemd[1]: Stopped Ceph alertmanager.a for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:53:40.912 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 18 07:53:40 smithi137 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.581s CPU time. 2024-09-18T07:53:41.169 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:53:40 smithi137 systemd[1]: Stopping Ceph node-exporter.a for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:53:41.580 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:53:41 smithi137 podman[213177]: 2024-09-18 07:53:41.313619531 +0000 UTC m=+0.264802593 container died 99cfcc69738e4b7bf82b21c6b9b23e60dc2e1afa8f0ca597e377a13f6390add7 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-18T07:53:41.919 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:53:41 smithi137 podman[213177]: 2024-09-18 07:53:41.769718305 +0000 UTC m=+0.720901405 container cleanup 99cfcc69738e4b7bf82b21c6b9b23e60dc2e1afa8f0ca597e377a13f6390add7 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-18T07:53:41.919 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:53:41 smithi137 bash[213177]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a 2024-09-18T07:53:42.333 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:53:42 smithi137 podman[213192]: 2024-09-18 07:53:42.045909727 +0000 UTC m=+0.735737678 container remove 99cfcc69738e4b7bf82b21c6b9b23e60dc2e1afa8f0ca597e377a13f6390add7 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-18T07:53:42.333 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:53:42 smithi137 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-18T07:53:42.333 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:53:42 smithi137 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-18T07:53:42.333 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:53:42 smithi137 systemd[1]: Stopped Ceph node-exporter.a for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:53:42.334 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 18 07:53:42 smithi137 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 4.517s CPU time. 2024-09-18T07:53:42.890 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f7ef099c-758e-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-18T07:53:43.095 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T07:53:44.711 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:53:44 smithi203 systemd[1]: Stopping Ceph iscsi.iscsi.a for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:53:44.711 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:53:44 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a[57264]: debug Shutdown received 2024-09-18T07:53:54.961 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:53:54 smithi203 bash[152377]: time="2024-09-18T07:53:54Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-09-18T07:53:54.961 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:53:54 smithi203 podman[152377]: 2024-09-18 07:53:54.636399491 +0000 UTC m=+10.130147939 container died 4924466ab0b425557b7062c993691dd30bc4bf2d2aae4ba8ea31f3b7f25b3fef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.vendor=CentOS) 2024-09-18T07:53:55.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:53:55 smithi203 podman[152377]: 2024-09-18 07:53:55.070124535 +0000 UTC m=+10.563872971 container cleanup 4924466ab0b425557b7062c993691dd30bc4bf2d2aae4ba8ea31f3b7f25b3fef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-18T07:53:55.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:53:55 smithi203 bash[152377]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a 2024-09-18T07:53:55.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:53:55 smithi203 podman[152392]: 2024-09-18 07:53:55.254837868 +0000 UTC m=+0.614603515 container remove 4924466ab0b425557b7062c993691dd30bc4bf2d2aae4ba8ea31f3b7f25b3fef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1f5b5a3734f665ef333266f39143ac3ef1eb1951b21258f2307e10332e2d7ac9, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-iscsi-iscsi-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-17-1213-e7fb7b5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-18T07:53:55.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:53:55 smithi203 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-18T07:53:55.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:53:55 smithi203 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-18T07:53:55.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:53:55 smithi203 systemd[1]: Stopped Ceph iscsi.iscsi.a for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:53:55.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi203.stdout:Sep 18 07:53:55 smithi203 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@iscsi.iscsi.a.service: Consumed 2.616s CPU time. 2024-09-18T07:53:57.211 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:53:56 smithi203 systemd[1]: Stopping Ceph grafana.a for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:53:57.557 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:53:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=server t=2024-09-18T07:53:57.263214345Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-09-18T07:53:57.557 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:53:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=tracing t=2024-09-18T07:53:57.263536302Z level=info msg="Closing tracing" 2024-09-18T07:53:57.557 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:53:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=grafana-apiserver t=2024-09-18T07:53:57.263567541Z level=info msg="StorageObjectCountTracker pruner is exiting" 2024-09-18T07:53:57.557 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:53:57 smithi203 ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a[59984]: logger=ticker t=2024-09-18T07:53:57.263588593Z level=info msg=stopped last_tick=2024-09-18T07:53:50Z 2024-09-18T07:53:57.557 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:53:57 smithi203 podman[152666]: 2024-09-18 07:53:57.293786587 +0000 UTC m=+0.140762649 container died 9d9dd8054cbc2930b1dc715438c326408ff78c2a949101b453685de4f05ee7f8 (image=quay.io/ceph/grafana:10.4.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-18T07:53:57.824 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:53:57 smithi203 podman[152666]: 2024-09-18 07:53:57.629898135 +0000 UTC m=+0.476874229 container cleanup 9d9dd8054cbc2930b1dc715438c326408ff78c2a949101b453685de4f05ee7f8 (image=quay.io/ceph/grafana:10.4.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-18T07:53:57.825 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:53:57 smithi203 bash[152666]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a 2024-09-18T07:53:58.210 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:53:57 smithi203 podman[152677]: 2024-09-18 07:53:57.82280065 +0000 UTC m=+0.531490370 container remove 9d9dd8054cbc2930b1dc715438c326408ff78c2a949101b453685de4f05ee7f8 (image=quay.io/ceph/grafana:10.4.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-18T07:53:58.211 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:53:58 smithi203 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@grafana.a.service: Deactivated successfully. 2024-09-18T07:53:58.211 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:53:58 smithi203 systemd[1]: Stopped Ceph grafana.a for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:53:58.211 INFO:journalctl@ceph.grafana.a.smithi203.stdout:Sep 18 07:53:58 smithi203 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@grafana.a.service: Consumed 14.843s CPU time. 2024-09-18T07:53:58.710 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:53:58 smithi203 systemd[1]: Stopping Ceph node-exporter.b for f7ef099c-758e-11ef-bceb-c7b262605968... 2024-09-18T07:53:58.711 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:53:58 smithi203 podman[152791]: 2024-09-18 07:53:58.48709104 +0000 UTC m=+0.106544022 container died f2e1665435ce9d4ebbc72b082a8b99eedebbf498e8bf930711fbf0306505ef68 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-18T07:53:59.121 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:53:58 smithi203 podman[152791]: 2024-09-18 07:53:58.851416968 +0000 UTC m=+0.470869939 container cleanup f2e1665435ce9d4ebbc72b082a8b99eedebbf498e8bf930711fbf0306505ef68 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-18T07:53:59.121 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:53:58 smithi203 bash[152791]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b 2024-09-18T07:53:59.376 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:53:59 smithi203 podman[152805]: 2024-09-18 07:53:59.119443598 +0000 UTC m=+0.631749100 container remove f2e1665435ce9d4ebbc72b082a8b99eedebbf498e8bf930711fbf0306505ef68 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-f7ef099c-758e-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-18T07:53:59.376 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:53:59 smithi203 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-18T07:53:59.376 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:53:59 smithi203 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-18T07:53:59.377 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:53:59 smithi203 systemd[1]: Stopped Ceph node-exporter.b for f7ef099c-758e-11ef-bceb-c7b262605968. 2024-09-18T07:53:59.377 INFO:journalctl@ceph.node-exporter.b.smithi203.stdout:Sep 18 07:53:59 smithi203 systemd[1]: ceph-f7ef099c-758e-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 4.499s CPU time. 2024-09-18T07:53:59.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-18T07:53:59.943 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-18T07:53:59.973 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-18T07:53:59.976 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910293/remote/smithi137/crash 2024-09-18T07:53:59.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/crash -- . 2024-09-18T07:54:00.028 INFO:teuthology.orchestra.run.smithi137.stderr:tar: /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-18T07:54:00.029 INFO:teuthology.orchestra.run.smithi137.stderr:tar: Error is not recoverable: exiting now 2024-09-18T07:54:00.030 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910293/remote/smithi203/crash 2024-09-18T07:54:00.031 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/crash -- . 2024-09-18T07:54:00.069 INFO:teuthology.orchestra.run.smithi203.stderr:tar: /var/lib/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-18T07:54:00.069 INFO:teuthology.orchestra.run.smithi203.stderr:tar: Error is not recoverable: exiting now 2024-09-18T07:54:00.070 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-18T07:54:00.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-18T07:54:00.157 INFO:tasks.cephadm:Compressing logs... 2024-09-18T07:54:00.158 DEBUG:teuthology.orchestra.run.smithi137:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T07:54:00.160 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T07:54:00.211 INFO:teuthology.orchestra.run.smithi203.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/tcmu-runner.log: No such file or directory 2024-09-18T07:54:00.212 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-18T07:54:00.212 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/objectstore_tool.2.log 2024-09-18T07:54:00.212 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-18T07:54:00.212 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/tcmu-runner.log: /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/objectstore_tool.2.log: 63.3% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/tcmu-runner.log.gz 2024-09-18T07:54:00.212 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-18T07:54:00.212 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-18T07:54:00.212 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-18T07:54:00.212 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-18T07:54:00.212 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-18T07:54:00.212 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-18T07:54:00.212 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.log 2024-09-18T07:54:00.213 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-18T07:54:00.213 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-18T07:54:00.213 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-18T07:54:00.217 INFO:teuthology.orchestra.run.smithi137.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.0.log: No such file or directory 2024-09-18T07:54:00.218 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-18T07:54:00.218 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-18T07:54:00.218 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-18T07:54:00.218 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-18T07:54:00.218 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-18T07:54:00.218 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-18T07:54:00.218 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-18T07:54:00.218 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-18T07:54:00.218 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-18T07:54:00.218 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.log 2024-09-18T07:54:00.218 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log 2024-09-18T07:54:00.218 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-18T07:54:00.218 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-18T07:54:00.219 INFO:teuthology.orchestra.run.smithi137.stderr: 58.9% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-09-18T07:54:00.227 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 91.8% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-18T07:54:00.285 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 89.8% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-18T07:54:00.356 INFO:teuthology.orchestra.run.smithi137.stderr: 91.7% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-18T07:54:00.429 INFO:teuthology.orchestra.run.smithi137.stderr: 88.9% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-18T07:54:00.429 INFO:teuthology.orchestra.run.smithi137.stderr: 93.9% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-18T07:54:00.431 INFO:teuthology.orchestra.run.smithi203.stderr: 91.4% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-18T07:54:00.437 INFO:teuthology.orchestra.run.smithi203.stderr: 91.7% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-18T07:54:00.460 INFO:teuthology.orchestra.run.smithi137.stderr: 95.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-18T07:54:00.549 INFO:teuthology.orchestra.run.smithi203.stderr: 85.3% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-18T07:54:00.659 INFO:teuthology.orchestra.run.smithi203.stderr: 79.0% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-18T07:54:00.734 INFO:teuthology.orchestra.run.smithi203.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-18T07:54:01.121 INFO:teuthology.orchestra.run.smithi203.stderr: 94.4% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/objectstore_tool.2.log.gz 2024-09-18T07:54:04.642 INFO:teuthology.orchestra.run.smithi137.stderr: 90.1% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-18T07:54:05.693 INFO:teuthology.orchestra.run.smithi137.stderr: 89.6% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-18T07:54:05.735 INFO:teuthology.orchestra.run.smithi203.stderr: 90.5% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-18T07:54:05.857 INFO:teuthology.orchestra.run.smithi137.stderr: 90.5% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-18T07:59:55.093 INFO:teuthology.orchestra.run.smithi137.stderr: 93.2% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-18T08:00:00.138 INFO:teuthology.orchestra.run.smithi137.stderr: 94.1% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-18T08:04:08.884 INFO:teuthology.orchestra.run.smithi137.stderr: 93.2% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-18T08:04:47.241 INFO:teuthology.orchestra.run.smithi137.stderr: 92.5% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-18T08:04:47.243 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-18T08:04:47.243 INFO:teuthology.orchestra.run.smithi137.stderr:real 10m47.069s 2024-09-18T08:04:47.243 INFO:teuthology.orchestra.run.smithi137.stderr:user 5m39.861s 2024-09-18T08:04:47.244 INFO:teuthology.orchestra.run.smithi137.stderr:sys 0m16.417s 2024-09-18T08:06:23.349 INFO:teuthology.orchestra.run.smithi203.stderr: 93.1% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-18T08:09:31.857 INFO:teuthology.orchestra.run.smithi203.stderr: 93.4% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-18T08:13:10.041 INFO:teuthology.orchestra.run.smithi203.stderr: 92.6% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-18T08:13:49.452 INFO:teuthology.orchestra.run.smithi203.stderr: 92.6% -- replaced with /var/log/ceph/f7ef099c-758e-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-18T08:13:49.454 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-18T08:13:49.454 INFO:teuthology.orchestra.run.smithi203.stderr:real 19m49.281s 2024-09-18T08:13:49.455 INFO:teuthology.orchestra.run.smithi203.stderr:user 6m26.373s 2024-09-18T08:13:49.455 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m20.914s 2024-09-18T08:13:49.455 INFO:tasks.cephadm:Archiving logs... 2024-09-18T08:13:49.456 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910293/remote/smithi137/log 2024-09-18T08:13:49.457 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T08:14:42.344 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910293/remote/smithi203/log 2024-09-18T08:14:42.345 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T08:15:42.077 INFO:tasks.cephadm:Removing cluster... 2024-09-18T08:15:42.078 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f7ef099c-758e-11ef-bceb-c7b262605968 --force 2024-09-18T08:15:42.379 INFO:teuthology.orchestra.run.smithi137.stdout:Deleting cluster with fsid: f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T08:15:43.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f7ef099c-758e-11ef-bceb-c7b262605968 --force 2024-09-18T08:15:43.312 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: f7ef099c-758e-11ef-bceb-c7b262605968 2024-09-18T08:15:44.068 INFO:tasks.cephadm:Removing cephadm ... 2024-09-18T08:15:44.068 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-18T08:15:44.085 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-18T08:15:44.101 INFO:tasks.cephadm:Teardown complete 2024-09-18T08:15:44.101 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-18T08:15:44.110 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-18T08:15:44.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T08:15:44.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T08:15:44.224 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-18T08:15:44.233 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-18T08:15:44.233 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T08:15:44.235 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T08:15:44.247 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-09-18T08:15:44.249 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-09-18T08:15:44.271 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T08:15:44.271 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-09-18T08:15:44.271 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv01.front.sepia.ceph.com 3 6 377 1 -1275us[-1265us] +/- 55ms 2024-09-18T08:15:44.271 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 8 +644us[ +654us] +/- 52ms 2024-09-18T08:15:44.271 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +922us[ +932us] +/- 81ms 2024-09-18T08:15:44.271 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T08:15:44.272 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T08:15:44.273 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-09-18T08:15:44.273 INFO:teuthology.orchestra.run.smithi137.stdout:^* hv01.front.sepia.ceph.com 3 6 377 5 -1271us[-1269us] +/- 55ms 2024-09-18T08:15:44.273 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 1 +632us[ +632us] +/- 52ms 2024-09-18T08:15:44.273 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +935us[ +935us] +/- 81ms 2024-09-18T08:15:44.273 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T08:15:44.274 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-18T08:15:44.290 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-18T08:15:44.291 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-18T08:15:44.341 DEBUG:teuthology.orchestra.run.smithi137:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-18T08:15:44.418 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-18T08:15:44.496 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-18T08:15:44.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:44.523 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-18T08:15:44.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:44.549 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-18T08:15:44.558 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-18T08:15:44.600 INFO:teuthology.task.internal:Duration was 3468.244260 seconds 2024-09-18T08:15:44.600 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-18T08:15:44.650 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-18T08:15:44.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T08:15:44.653 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T08:15:44.704 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T08:15:44.713 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T08:15:44.980 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-18T08:15:44.980 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2024-09-18T08:15:44.981 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T08:15:45.000 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-09-18T08:15:45.001 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T08:15:45.034 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-18T08:15:45.034 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T08:15:45.044 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T08:15:45.101 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-18T08:15:45.101 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T08:15:45.109 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T08:15:47.397 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-18T08:15:47.407 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-18T08:15:47.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T08:15:47.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T08:15:47.489 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-18T08:15:47.500 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T08:15:47.502 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T08:15:47.529 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-09-18T08:15:47.532 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2024-09-18T08:15:47.578 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T08:15:47.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:47.634 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T08:15:47.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T08:15:47.650 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-18T08:15:47.660 INFO:teuthology.task.internal:Transferring archived files... 2024-09-18T08:15:47.661 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910293/remote/smithi137 2024-09-18T08:15:47.662 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T08:15:47.746 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-18_06:59:44-orch:cephadm-wip-guits-main-2024-09-17-1213-distro-default-smithi/7910293/remote/smithi203 2024-09-18T08:15:47.746 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T08:15:47.788 INFO:teuthology.task.internal:Removing archive directory... 2024-09-18T08:15:47.788 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T08:15:47.791 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T08:15:47.838 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-18T08:15:47.849 INFO:teuthology.task.internal:Not uploading archives. 2024-09-18T08:15:47.849 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-18T08:15:47.875 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-18T08:15:47.875 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T08:15:47.877 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T08:15:47.890 INFO:teuthology.orchestra.run.smithi137.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 08:15 /home/ubuntu/cephtest 2024-09-18T08:15:47.894 INFO:teuthology.orchestra.run.smithi203.stdout: 265644 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 08:15 /home/ubuntu/cephtest 2024-09-18T08:15:47.895 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-18T08:15:47.925 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-18T08:15:48.027 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} duration: 3468.2442603111267 flavor: default owner: scheduled_gabrioux@teuthology success: true 2024-09-18T08:15:48.027 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T08:15:48.159 INFO:teuthology.run:pass